False 404 pages after stache:warm #10410
Comments
|
Would you be able to provide a repo where this behavior happens? |
|
I think ours are separate issues @heidkaemper. It looks like #10412 solves the issue I was running into. As this project I'm noticing it on features both structured collections and static caching (also locally because I was testing stuff). |
|
Yea that's right. We do use static caching (half) in production but certainly not in development. @jasonvarga We have to prepare a repo without client data. But we can manage that, I'll get back to you on that! |
|
We are experiencing this same random behaviour (on v5). Also half page caching using the redis application cache driver. |
|
An update from me as well: The 404 pages in Production could have something to do with #10412 for us as well. Cuz we run cache:clear and static:warm one after the other in deployment pipelines. So that would at least explain some of it. The other part of the issue is probably related to our development environment. Because I did:
When I tried to open the page at first the I dived in a little deeper and looked at the stache files. After a stache:warm, they appear to be identical in both of my development environments. Here for example the After I opened that page in the browser the stache key in my Laravel Herd environment remained unchanged. The file in our regular development environment does not: I have to say that the stache behavior goes a bit over my head. :-) |
heidkaemper
changed the title
|
We also have this same problem, locally (with no caching) and on production (with "half"-measure caching). The issue seems to appear a few hours after the last deployment but we couldn't reliably reproduce it. |
|
Just had this issue after the last release. Had to revert to I'm using Happy yo test things if needed! |
|
Okay, I think I found the problem for our setup. And it's quite clearly our own fault.. We use a docker setup for local development. And we often run console commands from outside the container. But things like the Stache Watcher obviously run inside the docker container. And although both work, there is one important difference: the For example, when I run stache:warm from outside the container, the Stache file looks something like this: The same command from inside the container generates a Stache file like this: I'm pretty sure this causes empty stache files. And it also explains why we could not reliably reproduce the error in production. This could possibly be fixed by not saving absolute paths in the stache. However, in my opinion this is an edge case and probably not worth the time and effort. Or what do you think @jasonvarga? As far as I'm concerned, we can close this issue. However, I am unsure whether the problems mentioned in the other comments are still valid. They sound more like issues with static caching?! |
|
@heidkaemper We are seeing this issue locally (using docker) as well as on production (not using docker). So I don't think the issue is docker. It might be one of the ways to cause this but not the only one. |
|
Okay, it's still valid than. Do you get the same |
|
I think we were getting a |
|
I think this issue is still valid in general, but the path issue @heidkaemper outlined I wouldn't consider an issue. You should run the commands in the docker container. 👍 |
|
Chiming in that we appear to be running into this on almost every deploy and it's not a good time! I'd thought I resolved it by updating our pipeline to use the more aggressive cache clear of PHP 8.3.9 |
|
I think its also related to this one, i have the same problem. |
|
We separated the stache from the application cache so we won't have to clear the stache and warm it again on each deployment. After this change, our application did not have the 404 problems for about 2 weeks, over multiple deploys. Today, however, I had to run php please stache:clear and the problem immediately returned. Last time this happened, it was caused by php artisan optimize:clear and fixed by reloading PHP FPM Workers. This time, it was caused by stache:clear and not fixed by reloading FPM workers but by running artisan optimize:clear twice. Very strange! When the 404s happened, I noticed, that a segment was missing from the URL. The URL is something like domain/en/projects/project_name/about and it was missing the "projects" part. The entire sub-navigation was also gone and came back after running optimize:clear twice. We only see the 404s on pages that are nested 2 levels in the collection tree. My wild guess is, that clearing the stache messes up the URL structure somehow. I hope this helps with debugging this mysterious issues. Would be great, if we didn't have to worry anymore about broken production sites. |
|
Another observation on this behaviour is that it appears to be time based in some way? I need to do more testing to confirm but we get 404's across entire sites on the first deploy in a certain period but subsequent deploys within an hour or two do not exhibit the same behaviour even though they go through the exact same pipeline commands. I'll also note that we're seeing these site wide 404s on all of our Statamic V5 projects (didn't happen on V4) with manually clearing the Application Cache being the only resolution which is problematic when we have various people triggering deploys at times, including content editors. |
|
Is there any update on this? One of our production sites still breaks on every deploy until we manually refresh the stache multiple times. |
|
Sorry, we don't have any updates on this since we haven't been able to reproduce it ourselves. Is anyone able to reproduce the issue reliably and is able to provide acess to their site for debugging? |
|
Thanks, @duncanmcclean. I can give you access to our staging server or our repo to test it locally. I'm not sure if this is related, but it might be. Lately I am getting this error when run You would think this is a data issue (and maybe it is), but when I run the same command again a second time, it passes with no errors. So maybe there is some sort of race condition going on. |
|
While I can't open up our sites for debugging purposes I have managed to work around this for now. From Statamic 3-4 our pipeline looked like this (simplified for clarity): - composer install
- npm ci --cache .npm && npm run build
- php artisan route:clear
- php artisan cache:clear
- rsync files to server
- ssh user@server cd /application && sed -i cache_timestamp .env &&
- && php please static:warmThis worked reliably for a couple of years, but after Statamic 5 this resulted in a 404 on every single page post deploy which would be fixed after clearing cache via the CP. After various attempts to resolve this I found the following process works reliably across all of our sites (simplified again): - composer install
- npm ci --cache .npm && npm run build
- rsync files to server
- ssh user@server cd /application && sed -i cache_timestamp .env &&
- php artisan optimize:clear &&
- php artisan optimize &&
- php please stache:refresh &&
- php please static:warm"TL;DR since Statamic 5 we have needed to refresh all Laravel caches, and the stache after the rsync to the server, and then finally warm the static cache. Hopefully this helps other people at least avoid this issue/bug for now and potentially gives a reproduction path. |
|
@schwemmer Access to a repository would be great, thanks! |
|
@duncanmcclean I'll send you the details. |
|
@duncanmcclean
Note: I can reproduce the bug for Statamic v5.02 and v5.25.0 |
|
I added some fix before it will be fixed. (Note: it can hit performance, as it calls multiply times per collection tree saving) |
|
In one project, we now have problems with 404 pages several times a week. Setup:
In the project a queued job runs every morning and updates many entries. We use And at the very end, just before that job completes, we run Sometimes this works just fine. And sometimes the website only has 404 pages afterwards. There are no errors in log files. I noticed one particular thing: stache:refresh seems not work in a queued job. In a synced job or a command, however, it works without any problems. I am currently considering switching from flat file to elequent to avoid the problem. |
We're also still experiencing the same issue. Can't put a finger on when it happens, but looks like it is sometimes after a deployment. The deployment script runs
Setup is the same as @heidkaemper, except for Statamic being version Hopefully this information can help debugging the issue. |
|
@duncanmcclean Have you been able to look into this? We are still having this issue, every once in a while after a deploy, random pages don't work anymore and it's a pretty big problem. Production sites stop functioning properly. |
|
Sorry, I haven't yet. I'll see if I can find some time in the next week or so to take a look. |
Bug description
This is a strange one, sorry! :-)
After the update to 5.x, everything looked okay at first. After a few hours, however, 404 pages for published entries appeared in production. Clearing the cache helped at first, but after a while the 404 pages were back again.
I have since been able to reproduce the behavior with two projects in development as well. PHP version and collection settings do not seem to matter.
Rob also has this problem randonly with his projects, but cannot reproduce it reliably.
Any ideas? Unintentional behavior of the stache perhaps?!
How to reproduce
php please stache:warmCall to a member function requiresSlugs() on nullerror occursLogs
Environment
Installation
Other (please explain)
Additional details
Plain statamic/statamic site, update from 4.x to 5.x
The text was updated successfully, but these errors were encountered: