I am encountering an increased amount of user-facing 429 responses while trying to contribute on both the French and the Esperanto Wikisource.
The issue seems similar to the one encountered in T337649, with similarly increased amounts of 429 errors emitted by codfw for Thumbor jobs.
See this grafana chart: https://grafana.wikimedia.org/d/Pukjw6cWk/thumbor?orgId=1&refresh=1m&from=1723467291972&to=1723640091972&viewPanel=62 (warning: the scale is logarithmic).
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | hnowlan | T372470 Elevated 429 responses from Thumbor on codfw starting 2024-08-14 00:00 UTC | |||
| Open | hnowlan | T376538 Per-format monitoring for Thumbor |
I'm investigating this issue, but it does not appear to be the same issue as T337649 on first glance. Restarts and release of locks does not immediately drop the throttle rate, and a great deal of the elevated 429 errors are legitimate response to aggressive scraping (of course, that doesn't mean _all_ are legitimate errors)
Yup. I'm still encountering 429s when contributing and have other contributors reporting me the same issue.
Change #1063217 had a related patch set uploaded (by Hnowlan; author: Hnowlan):
[operations/software/thumbor-plugins@master] poolcounter: introduce allowlist to skip rate limit
Change #1063228 had a related patch set uploaded (by Hnowlan; author: Hnowlan):
[operations/deployment-charts@master] thumbor: add allowlist to thumbor to address internal rate limiting
Almost no new PDF file thumbnails can be generated from codfw: https://commons.wikimedia.org/wiki/Special:NewFiles?user=&mediatype[]=OFFICE&start=&end=&wpFormIdentifier=specialnewimages&limit=50&offset=
The situation is a lot better when switching to eqiad. But the thumbnails rendered in eqiad somehow aren't shared with codfw, so when switching back to codfw, the thumbnails are still not loading.
I've disabled a throttling feature that might be impacting this behaviour, I see format based throttling has stopped so this should no longer be an issue.
found T376509 while investigating 429 for https://upload.wikimedia.org/wikipedia/commons/thumb/7/74/Feedback_form_Odia_Wikipedia_outreach.pdf/page1-180px-Feedback_form_Odia_Wikipedia_outreach.pdf.jpg
https://commons.wikimedia.org/wiki/File:Feedback_form_Odia_Wikipedia_outreach.pdf
ahhhhh, now I found T372470#10113572.
Change #1078043 had a related patch set uploaded (by Hnowlan; author: Hnowlan):
[operations/deployment-charts@master] thumbor: disable expensive counter
It appears that there was an underlying issue with a rate limiting service used by thumbor. I've applied a fix to work around this for the short-term and that PDF now renders.
Just to explain the issue - a while ago a rate-limiting feature that was known to be problematic was reenabled in an emergency due to a harmful surge in traffic. This was left enabled and caused this issue to recur. I've since disabled this feature and we'll be removing it to prevent it being erroneously triggered again. However, the fact that this required manual reporting and wasn't noticed on the SRE-side isn't really acceptable so next week I'll be working on adding per-format alerting so that if there is an increase in errors for a single format we'll catch these before they can have a wide impact which will be tracked in T376538.
Can confirm recovery occuring at English Wikisoruce on files mentioned previously, eslewhere.
Change #1078043 merged by jenkins-bot:
[operations/deployment-charts@master] thumbor: disable expensive counter