I fail to reproduce from Europe. Is this reproducible or a one-off?

In T331820#8685842, @Aklapper wrote:

I fail to reproduce from Europe. Is this reproducible or a one-off?

Apparently, just one-off.

I think I have the same problem on several files. Example one:
https://commons.wikimedia.org/wiki/File:NASA’s_SpaceX_Crew-5_Splashdown_(NHQ202303110021).jpg => image is not displayed
https://upload.wikimedia.org/wikipedia/commons/thumb/c/cf/NASA’s_SpaceX_Crew-5_Splashdown_(NHQ202303110021).jpg/1280px-NASA’s_SpaceX_Crew-5_Splashdown_(NHQ202303110021).jpg => HTTP 404

Request from 2a02:8428:7cba:5a01:f4a4:8d46:2615:ae62 via cp6008 cp6008, Varnish XID 555059990
Upstream caches: cp6008 int
Error: 404, Not Found at Tue, 14 Mar 2023 12:40:29 GMT

Firefox 110.0.1 (64 bits) on Windows 11, French ISP

Again here https://upload.wikimedia.org/wikipedia/commons/thumb/f/f1/Rolland_-_Beethoven,_tr._Hull,_1927.pdf/page9-375px-Rolland_-_Beethoven,_tr._Hull,_1927.pdf.jpg

Request from 90.112.34.87 via cp6006 cp6006, Varnish XID 675386332
Upstream caches: cp6006 int
Error: 404, Not Found at Tue, 14 Mar 2023 17:15:04 GMT

Chrome on Windows 10, French ISP

Other people got the same error: https://commons.wikimedia.org/wiki/Commons:Village_pump#Files_are_not_appearing.

Another example:
https://commons.wikimedia.org/wiki/File:The_Sky_is_Not_the_Limit;_There_Are_Footprints_on_the_Moon_(5932386).jpeg
https://upload.wikimedia.org/wikipedia/commons/thumb/c/ca/The_Sky_is_Not_the_Limit;_There_Are_Footprints_on_the_Moon_(5932386).jpeg/1280px-The_Sky_is_Not_the_Limit;_There_Are_Footprints_on_the_Moon_(5932386).jpeg

Request from 2a02:8428:7cba:5a01:d81d:1b72:f29c:3f02 via cp6005 cp6005, Varnish XID 764095573
Upstream caches: cp6005 int
Error: 404, Not Found at Tue, 14 Mar 2023 21:06:02 GMT

Firefox 111.0 (64 bits) on Windows 11, French ISP

I've had this intermittently today as well — https://upload.wikimedia.org/wikipedia/commons/thumb/f/f9/Octicons-gift.svg/12px-Octicons-gift.svg.png being the most recent example

In T331820#8695475, @TheresNoTime wrote:

I've had this intermittently today as well — https://upload.wikimedia.org/wikipedia/commons/thumb/f/f9/Octicons-gift.svg/12px-Octicons-gift.svg.png being the most recent example

Mmm, whereas that WFM now.

Typically swift (the media storage system) would not return a 404 unless the source file actually does not exist. When it looks for a thumbnail and cannot find it swift should call Thumbor to create the thumb from the original and then store and return the thumb. This all sort of looks like the 404 is actually going out to the CDN edge as a response to the original caller, but also that the thumb is generated and stored for the next requestor. I haven't yet been able to recreate the 404 for any of the reported URLs.

another example I've found on an article is https://upload.wikimedia.org/wikipedia/commons/thumb/c/c4/Pile_up_(8196476217).jpg/440px-Pile_up_(8196476217).jpg (file page)

(10:38 PM) Update: Fixed

Some logs (NDA, they have IPs in) for a particular request above - P45869.

It's notable that if thumbor gets an AUTH failure, it will return a 404. I suspect that thumbor doesn't cache its auth sessions very well, and so may be somewhat inclined to overload the swift auth service; I would generally expect thumbs to be regenerated next time round (if you see what I mean), modulo the edge caching the 404 for 10 minutes.

Hello, this problem causes difficulties on Wikisource, due to the impossibility to access page image while proofreading...

I'm experiencing some problems with files not displayed and broken thumbnails too, here are my screenshots (from discussion on Village pump "Files are not appearing.") I have a good internet connection, in general I have no problems with thumbnails, upload and download. Windows 10 Home, Chrome version 111.0.5563.65 (64 bits) and Firefox version 111.0 (64 bits).

https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)#Thumbnails_not_generating

further three are missing when searching: https://de.wikipedia.org/w/index.php?fulltext=1&search="Auguste Bock"&title=Spezial:Suche&ns0=1

Picture links are missing in source code:

In T331820#8698824, @doctaxon wrote:

further three are missing when searching: https://de.wikipedia.org/w/index.php?fulltext=1&search="Auguste Bock"&title=Spezial:Suche&ns0=1

2 things can be true at the same time.

maybe it's unclear as rendered but some articles just don't have any images associated with them. or whatever algorithm is excluding some from being chosen as the "primary" image for an article. see e.g. https://de.wikipedia.org/wiki/Liste_der_Gedenktafeln_in_Berlin-Mitte?action=info&uselang=en

anyway if you want to pursue this further I think that's not relevant here, please open a new ticket.

It appears that there was a lingering switchover issue related to communication between thumbor and swift. We're still investigating what caused this but it appears that new files are being thumbnailled correctly now.

@Aklapper which bug task belongs to this 429 error issue?

@doctaxon: Not sure; please search.