I couldn't find anything that might connect this issue to the risky patch (T281152#7144507) and it might be an issue with the l10n cache of beta cluster. I think we should wait until deployment to group0 and see if it happens there too

Specially since I can't reproduce the issue locally at all.

In T285125#7163042, @Ladsgroup wrote:

it might be an issue with the l10n cache of beta cluster.

Could be, as e.g. https://meta.wikimedia.beta.wmflabs.org/wiki/Special:CentralAuth/Test shows ⧼centralauth-admin-notblocked⧽.

Yeah, that looks like something broken with beta itself. The relevant message keys are present in deployment-mediawiki11:/srv/mediawiki/php-master/cache/l10n/upstream/l10n_cache-en.cdb.json, I'm trying to figure out if there are any relevant logs

T285119: beta: Error: invalid magic word 'ns' and T285118: beta: Error: Unsupported operand types - though they're more symptoms rather than the cause (pointing at some l10n cache related issue)

@dancy Hi, do you imagine the recent scap python3 migration might be causing these?

https://logstash-beta.wmcloud.org (works now, if not clean up /var/log/logstash/logstash-json.log and restart logstash.service on deployment-logstash[04-06]) is full of php unserialize errors in localization related code, so something is definitely going wrong when generating the localization cache files, and given the files itself appear to be fully invalid (instead of having the wrong contents or something like that) my best guess is something in scap changed

Scap just calls a MW maintenance script to make them…

In T285125#7163537, @Majavah wrote:

@dancy Hi, do you imagine the recent scap python3 migration might be causing these?

@dancy will be out next week as will I. @LarsWirzenius may be able to take a look early next week. Otherwise, it'll be the following week before we're in a position to take a look.

We'v been porting Scap to Python3 latetly, and this _might_ be Unicode string confusion problem somewhere. I don't know that it is, though, nor can I see where it might be.

Mentioned in SAL (#wikimedia-operations) [2021-06-22T17:42:59Z] <dduvall> testwikis to 1.37.0-wmf.11 (cc open blockers T285125 T285118 T271011)

I tried downgrading scap in beta to match the version in prod (by running sudo apt-get install scap=3.17.1-1 on deployment-deploy01.deployment-prep) to see if that narrows down the issue.

a similar issue is happening here
https://en.wikipedia.beta.wmflabs.org/wiki/Special:Version
version: 1.37.0-alpha (1f66af0) 09:37, 24 June 2021

on test wiki
version: 1.37.0-wmf.11 (rMW32a113bceb44) 22:42, 23 June 2021

Mentioned in SAL (#wikimedia-releng) [2021-06-28T11:59:22Z] <majavah> downgrade scap to 3.17.1-1 (matching production) on deployment-mediawiki11, testing for T285125

In T285125#7174054, @dpifke wrote:

I tried downgrading scap in beta to match the version in prod (by running sudo apt-get install scap=3.17.1-1 on deployment-deploy01.deployment-prep) to see if that narrows down the issue.

I downgraded an appserver (deployment-mediawiki11) too and force regenerated English language caches like this:

taavi@deployment-mediawiki11:/srv/mediawiki/php-master/cache/l10n$ sudo rm l10n_cache-en.cdb 
taavi@deployment-mediawiki11:/srv/mediawiki/php-master/cache/l10n$ sudo -u mwdeploy /usr/bin/scap cdb-rebuild
l10n merge: 100% (ok: 439; fail: 0; left: 0)                                    
12:05:31 Updated 1 CDB files(s) in /srv/mediawiki/php-master/cache/l10n

The issue was fixed after this. To confirm it was really a problem with the latest scap versions I re-upgraded scap (sudo apt-get upgrade) and force regenerated English caches once again. The issues was back, so that confirms it's a Scap issue.