Hi all,
i have spent the last weeks setting up a website that integrates a shopify shop into a kirby site via the Shopify Plugin. That all worked really well (last successful products.cache was the 29th of october), untill last night, when some products got updated in the shopify shop and the webhook started to return an almost empty json file. It consists of this:
{“created”:1605472371,“minutes”:0,“value”:null}
This causes the kirby website to stop working (no php error messages, just no responses anymore) anywhere where i target the shop. i can’t even ask if there are product subpages existing without the page to crash.
I have tested this with two different shopify shops and on two different domains. The problem is consistent.
Since some of you may use the plug in and shopify as well… Have you experienced any problems with that recently?
@possible I changed the category on purpose, the plugins category is only for advertising plugins, not for plugin related questions. Maybe we should have a different category, but for the moment please leave it as is. Thank you.
The plugin is incompatable with the newest version of kirby. I ran into this too. Something changed between Kirby version 3.2.5 and the current version. I dont know what changed but if you roll back to 3.2.5 you should find that it works again. A newer version than that may work, but i didnt have time to keep trying versions to the find the one that broke it.
Version 3.3.0 was a pretty huge update so i guessing its that one 
I guess its something that @tristantbg needs to look into.
hello @jimbobrjames,
thanks for the reply. unfortunately, going back to version 3.2.5 doesn’t work in my case. I am still receiving these empty json cache files… also i never changed the kirby version (3.3.0) from when it worked to when it stopped. i even tried v 3.4.3 with no luck.
interestingly though, the collections.cache (in v3.4.3) is working fine…
Is there a way to see or know if the json file arrived fine and was interpreted wrong or if it already shipped out at shopify with “value”: null ?
Did your cache files were empty as well?
We almost launched the site already… now i am happy we still got some time to fix this…
p.s. if i try to often to reload the products page in the panel, i’ll get a “Error 503 - Service Unavailable The server is currently unavailable.” message…
That’s odd. I am using the last version of the plugin with the last version of Kirby (3.4.4) and don’t have any problem.
I guess this is related to your webhooks or the way they are verified. Can you check if the webhooks are still ok on Shopify ?
hi @tristantbg,
i just checked, seems all fine. i can do a “Send test notification” and get another value null file… I tested it with two shops (one official and my developer) on 2 different domains and have the same results…
i will try today a clean kirby / shopify plugin version and see if it persists.
So here are the results of my investigation:
- New kirby install (3.4.4) and the shopify plugin won’t work in my case. I still get 47byte big cache files.
- I erased all webhooks and started new. And i found some odd behaviour:
If i open the collections page in the backend with no collections.cache file in the cache folder present, i get the correct file with the current collections.
If i change anything in the collections at shopify, i am back again with the 47byte (almost) empty file… Seems like opening the collections page in the backend starts a different process than the actual webhook? - the products webhook doesn’t work. i get a file, but it is empty…
Also as an additional info: i am using the 2020-10 webhook-api
I also found the same when i ran into this - i seemed to be getting a correct collections cache file, but kept ending up with the same 47byte product file as you got. The only thing that fixed it for me was rolling back to 3.2.5.
Now i tried this with my developer shop (the original has 300+ products, this one just 5) and oddly enough, this happens:
when i update a product in shopify, i get the 47 byte file. but when i then go to the panel and reload the products page, i get the correct products listed and a new file in the cash folder…
still pretty unclear why and how this works…
Well one way is pushing the data, the other is pulling, so this suggests an issue with Shopify communicating with Kirby, rather than Kirby connecting to shopify and getting data in the other direction. From what i understand, a Shopify update will trigger the webhook to push the data to kirby. When you reload in the panel, it triggers an API request to Shopify and get’s JSON back, which seems to work. So im guessing problem lies somwhere in the webhook stuff.
I am still scratching my head… i have now tried some things back on the development website that i wanted to launch soon:
- connecting to my developer shop (4 items) seems to work, although here i just can’t get the collections to show up
- if i reconnect to the original shopify store (320 items) it just doesn’t work. so my guess is now, that it might be a filesize problem. the previous file (315 items) had 1.5MB, so maybe there are some time-outs and such in place now. (sidenote: here the collections are loaded…)
also: the whole things keeps crashing with a “Error 503 - Service Unavailable”.
so weird… i can’t really figure it out and my best guess is, that i shouldn’t rely too much on that bridge in the future. right now, the site is quering the shopify pages in several places, but i can’t even prevent the crashes(?) with an if clause…
maybe its better to rebuild the whole shop in kirby and just pass the id for the shopify buttons.
but: untill this problem, the plugin was a wonderful thing, i must say.
Thank you for keeping us up to date with this. I am trying to find where could be the error.
If the products.cache is created successfully while loading the “Products” page in the back-end but not through the webhook, this is quite odd as it is the same behaviour for both trigger…
Hello Tristan,
right now, my best bet is the size of the file… in the meantime i have transferred all the json data from the last working products.cache into kirby pages, so that the website can go online regardless…
To figure out what is causing my problems with the webhooks, i will do one more test later and delete exactly the products that have been added between the webhooks working and not working.
If this proves successful, we’ll add more items and see when it stops working…
On a sidenote: the first time i connected to the shopify shop, there where ca. 20 items missing from the shopify shop. we didn’t know why. we then deleted them and added them again and it worked.
i didn’t think about this much, but in the light of the recent events, it may add to my suspicion that there is either a maximum in filesize or in amount of items that the webhooks can deliver…
So, i copied the whole shopify product catalog (303 items) into my developer account and doubled the amount of items there (i basically duplicated the 303 items with a new handle).
This way, i could start checking for a pattern regarding the product count.
I set around 270 items as active and the rest as draft, so i could make some items active and see what happens.
First of all, the behaviour is strangely erratic. If i set a certain amount of draft items to active, the product count at kirby sometimes just goes backwards and if i activate just a few more items suddenly jumps to a higher (correct) number… It seems as if the data is either not complete at arrival or incompletely processed.
Around 300+ items it starts to crash (filesize is now around 1.8MB) so that i think it might be either a filesize limit or some kind of time-out problem. I think its probably the latter, because i had problems with different amounts of items. Since it always takes some time for the website to come back from the crash, i can’t really pinpoint the exact amount.
I can only ask you replicate this in a shop that has enough items with enough data. I think this is a problem of complexity… If you want, i could ask my client if i can send you an products-database export, so you can import it into a dummy shop, just like i did. That is of course, if you have the resources to look into this matter…
One major problem of this is, that when the threshold is reached, the whole page loops into oblivion and the site becomes unusable. Maybe there is a way to check if the .cache file sports a {“created”:1605472371,“minutes”:0,“value”:null} that everything should just stop trying…
With some more testing i am also more confused…
At shopify, i have 313 items active and 297 items as draft.
The .cache file lists (its a bit hard for me to run this through my sheet programs) around 328 items (close enough).
Kirby though lists only 250 items. (The count was 280 before i changed the status on one item in the shop from draft to active and it jumped to 250 from there).
None of these numbers make any sense to me… It seems to me, like there is something lost in the process, which is strange. Unfortunately, i have no tools or knowledge to investigate any further…
My best guess is, that having a shop with a limited amount of products should be no problem for the plugin. But around 300 products, it gets really wobbly…
Are you sure its 300 products thats the limit? It happens on my store which has around 40 products on it, split into 4 collections.
In my case it was fine around 300 items. But thats also where we started. We didn’t start integrate the shop when it was just 40 items big… So, maybe there could be also a factor of a growing shop… I really don’t know actually, what the problem might be.
I think the first thing should be a safety measure, so that kirby doesn’t crashes when the file has Value Null…
Thank you @possible for reporting all of this, it’s very helpful.
I guess there is something related to the 250 products limit by the Shopify API.
The plugin needs to make 2 request call for more than 250 products.
If the server TIMEOUT before the end of the two calls, the cache file might end empty.
I am going to try with a website I did with a lot of products (over 3000) to see how it reacts and maybe prevent the website to be broken if the cache file is empty.
hello @tristantbg,
thanks for the infos. i didn’t know about the 250 products limit. I tested some more and have some new findings:
- i need to revise my comment regarding the number of items changing weirdly. this was due to the fact that i changed the status of the items at shopify, but never realized that it actually takes some time for them to complete. sometimes i just reloaded the kirby site too early.
- when i reach 250 active products in shopify, only 246 items get transferred to kirby.
- when i add 50 more at shopify, it crashes.
an empty shop sends out a .cache like this:
{"created":1605868949,"minutes":0,"value":[]}
the crashing one looks like this:
{“created”:1605472371,“minutes”:0,“value”:null}
now, my guess is, that value: null means some kind of in between status, when the cache is erased but the api hasn’t received the informations yet. maybe that puts the system in limbo…
now i don’t understand how i was able to have 300 items already in the other shop…
the problem around 250 items seems now persistent, if i go back from 300 items to 250, the site loads again (but also again only 246 items).
Hi @possible,
I just tried with a website with 1270 products (and fully Shopify hosted).
My products.cache is 8.9mo and it took forever to load all the data…
But it is working as expected.
I think your server is having an overload, timeout or getting an API limit from Shopify at fetching too many times the products.
Can you upgrade the max-execution-time ?
When you change the status of 250 products, Shopify sends 250 times to Kirby a cache flush and products.cache rebuild.
This Kirby plugin might not be a perfect fit for big eshops but more for smaller ones with less than 250 products, sometimes it is better to go fully hosted on Shopify instead to prevent these sync problems.
Let me know if I can help in something else.