I'm not sure what happened, but as of about two weeks, the queries ran by GUC seem to hit a snag somewhere. It now quite often takes anywhere from 80s to over 250s for a request to get a response where previously the same requests came back in just a second or two.
Example: https://tools.wmflabs.org/guc/?user=Krinkle&debug=true
Using GUC's debug mode to print information about connections and queries, I see that in most cases the biggest part of the delay is from a single query, usually the first "user" query.
This is odd, for a number of reasons.
- The first database listed doesn't appear be technically different from the others (relatively small wiki and less than 50 rows in total for that username).
- The first "user" query is not the first query overall on that connection, so it's not about being the first query.
- The given query is relatively simple compared to the much bigger query before it. See below, "Query all wikis for matching revisions" is a huge union query across 100 wikis at once for the revision tables, taking only <70ms. Then after that is a user table query for one database, taking 88 seconds?
Debug sample 1/2:
0s: Got input, start search 0s: Get list of all wikis 0s: Create connection to s1.web.db.svc.eqiad.wmflabs 0.07s: Query all wikis for matching revisions 88.97s: Query user data for ab.wikipedia.org 0.03s: Query contributions on ab.wikipedia.org 0.05s: Query user data for ace.wikipedia.org 0.03s: Query contributions on ace.wikipedia.org 0.03s: Query user data for ady.wikipedia.org 0.03s: Query contributions on ady.wikipedia.org 0.03s: Query user data for af.wikipedia.org 0.04s: Query contributions on af.wikipedia.org ... ... etc., etc. – 800 times more for other wikis
Debug sample 2/2:
0s: Got input, start search 0s: Get list of all wikis 0s: Create connection to s1.web.db.svc.eqiad.wmflabs 0.05s: Query all wikis for matching revisions 20.95s: Query user data for ab.wikipedia.org 0.03s: Query contributions on ab.wikipedia.org 0.08s: Query user data for ace.wikipedia.org 0.02s: Query contributions on ace.wikipedia.org 0.03s: Query user data for ady.wikipedia.org ... ... etc., etc.