**Note: this is a fallback task, written as more or less a verbatim copy of T264991 on grounds of doubts on the feasibility of the approach adopted in T263437 (see T329491#8895863). **
# Goal
In order to upgrade for all our MediaWiki clusters (including #mw-on-k8s ) to Debian Bullseye, we need to upgrade the current Debian Buster installation to use ICU 67. The reason is that Debian ships Bullseye with version 67. This process requires all servers to be upgraded within a short period of time, and after that running the `updateCollation.php` script on all wikis with a non-standard collation.
# Dependencies
**Migration day TBD **
The transition plan will roughly be as follows:
[x] Rebuild related packages for stretch by linking them to `libicu67` and add them to the `component/icu67` repo
[x] php7.4-bcmath
[x] php7.4-bz2
[x] php7.4-cli
[x] php7.4-common
[x] php7.4-curl
[x] php7.4-dba
[x] php7.4-fpm
[x] php7.4-gd
[x] php7.4-gmp
[x] php7.4-intl
[x] php7.4-json
[x] php7.4-mbstring
[x] php7.4-mysql
[x] php7.4-opcache
[x] php7.4-phpdbg
[x] php7.4-readline
[x] php7.4-xml
[x] dh-php
[x] libxml2
[x] php-acpu
[x] php-excimer
[x] php-geoip
[x] php-igbinary
[x] php-imagick
[x] php-luasandbox
[x] php-memcached
[x] php-msgpack
[x] php-redis
[x] php-wmerrors
[x] tideways
[x] wikidiff2
# Process
[x] Use a feature flag to enable `component/icu67` in puppet
[x] Build container base images containing the new packages (https://gerrit.wikimedia.org/r/954700)
[x] Define a migration date - **2023-11-02**
[x] Notify the communities, with commrel's help
* https://meta.wikimedia.org/wiki/Tech/News/2023/44
* https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)#Heads-up:_Update_to_ICU_Unicode_library
[x] Test packages on mwdebug servers
[x] Update packages on deployment-prep
[ ] On Migration day:
[ ] Upgrade to libicu67 everywhere
[ ] app, api, parsoid, jobrunners
[ ] snapshot hosts
[ ] kubernetes deploymens (`scap sync-world --k8s-only -D full_image_build:true`)
[ ] deploy hosts (can be done later)
[ ] Run `updateCollation.php` on all wikis where it's needed (~10 days, this is the point of no return, **use screen/tmux)
```
# each of those in parallel
foreachwikiindblist s1.dblist updateCollation.php --verbose-stats
foreachwikiindblist s2.dblist updateCollation.php --verbose-stats
foreachwikiindblist s3.dblist updateCollation.php --verbose-stats
foreachwikiindblist s5.dblist updateCollation.php --verbose-stats
foreachwikiindblist s6.dblist updateCollation.php --verbose-stats
foreachwikiindblist s7.dblist updateCollation.php --verbose-stats
```
[ ] Include the icu67 upgrade into the general php7.4 images / retag the php7.4-icu67 ones
Notes:
`updateCollation.php` needs to run for: **s1, s2, s3, s5, s6, s7** (according to [[ https://phabricator.wikimedia.org/T264991#6609917 | find_collations.py ]])
```
s1:
enwiki
s2:
cswiki fiwiki itwiki nlwiki nowiki plwiki ptwiki svwiki thwiki trwiki
s3:
mediawikiwiki abwiki azwiki bawiki bawikibooks be_x_oldwiki bewiki bewikisource bnwiki bnwikisource bswiki bswiktionary ckbwiki cswiktionary cywiki cywikibooks cywikiquote cywikisource cywiktionary dewikisource eswikiversity etwiki etwikibooks etwikimedia etwikiquote etwikisource etwiktionary fawikisource fawiktionary fawikibooks fawikinews fawikiquote fiwikibooks fiwikimedia fiwikinews fiwikiquote fiwikisource fiwikiversity fiwikivoyage frwikibooks frwikinews frwikiversity gawiki gdwiki gewikimedia glwiki grwikimedia hewikisource hrwiki hsbwiki idwikimedia id_internalwikimedia ilowiki iswiki iswiktionary ltwiki lvwiki mkwiki napwikisource nnwiki nowikimedia olowiki plwikisource plwikivoyage plwiktionary ptwikibooks romdwikimedia rowikibooks rowikinews rowikiquote rowikisource rowikivoyage rowiktionary rswikimedia ruwikibooks ruwikinews ruwikiquote ruwikisource ruwikiversity ruwikivoyage ruwiktionary sewiki simplewiki slwiki slwikibooks slwikiquote slwikisource slwikiversity slwiktionary sqwiki sqwikibooks sqwikinews sqwikiquote srwikibooks srwikinews srwikiquote srwikisource srwiktionary skwiki svwikisource tawiki tawikibooks tawikinews tawikiquote tawikisource tawiktionary testwiki thwikibooks thwikinews thwikiquote thwikisource thwiktionary trwikiquote trwiktionary uawikimedia ukwikibooks ukwikinews ukwikiquote ukwikisource ukwikivoyage ukwiktionary viwikibooks viwikiquote viwikisource viwikivoyage viwiktionary wbwikimedia
s5:
azwikimedia shwiki srwiki trwikivoyage
s6:
frwiki ruwiki
s7:
eswiki fawiki frwiktionary hewiki huwiki rowiki ukwiki viwiki
```
For updating app, api, parsoid, jobrunners, we 'll use this command. For updating snapshot* and deploy* we 'll make sure to remove packages that are not installed there.
```
cumin mwdebug1001* 'depool; sleep 1; run-puppet-agent -e "upgrading ICU67" ; apt-get -q update;
export DEBIAN_FRONTEND=noninteractive; apt-get install $PACKAGES -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold"; systemctl
restart php7.4-fpm ; pool'
```
I've checked all binary packages in the icu67 component with `sudo -i reprepro -C component/icu67 -A amd64 list buster-wikimedia | sed 's/.*amd64: //;s/ .*//' | grep -v '\-dbg'` and checked if they are installed with `cat icu67_packages.txt | while read pkg; do dpkg-query -W --showformat='${db:Status-Status}' "$pkg" 2>&1| grep ^installed -q && echo -n "$pkg "; done; echo`
This looks like:
```
cloudweb1003.wikimedia.org:
libicu67 libxml2 php-common php-ldap php7.4-apcu php7.4-bcmath php7.4-bz2 php7.4-cli php7.4-common php7.4-curl php7.4-dba php7.4-excimer php7.4-fpm php7.4-gd php7.4-geoip php7.4-gmp php7.4-igbinary php7.4-intl php7.4-json php7.4-ldap php7.4-luasandbox php7.4-mbstring php7.4-memcached php7.4-msgpack php7.4-mysql php7.4-opcache php7.4-readline php7.4-redis php7.4-wikidiff2 php7.4-wmerrors php7.4-xml php7.4-yaml
deploy1002.eqiad.wmnet:
libicu67 libxml2 php-common php-readline php7.4-apcu php7.4-bcmath php7.4-bz2 php7.4-cli php7.4-common php7.4-curl php7.4-dba php7.4-excimer php7.4-gd php7.4-geoip php7.4-gmp php7.4-igbinary php7.4-intl php7.4-json php7.4-luasandbox php7.4-mbstring php7.4-memcached php7.4-msgpack php7.4-mysql php7.4-opcache php7.4-readline php7.4-redis php7.4-wikidiff2 php7.4-xml php7.4-yaml
dumpsdata1001.eqiad.wmnet:
libicu67 libxml2
htmldumper1001.eqiad.wmnet:
libicu67 libxml2
mw1414.eqiad.wmnet (appserver):
libicu67 libxml2 php-common php7.4-apcu php7.4-bcmath php7.4-bz2 php7.4-cli php7.4-common php7.4-curl php7.4-dba php7.4-excimer php7.4-fpm php7.4-gd php7.4-geoip php7.4-gmp php7.4-igbinary php7.4-intl php7.4-json php7.4-luasandbox php7.4-mbstring php7.4-memcached php7.4-msgpack php7.4-mysql php7.4-opcache php7.4-readline php7.4-redis php7.4-wikidiff2 php7.4-wmerrors php7.4-xml php7.4-yaml
mw1437.eqiad.wmnet (jobrunner)
libicu67 libxml2 php-common php7.4-apcu php7.4-bcmath php7.4-bz2 php7.4-cli php7.4-common php7.4-curl php7.4-dba php7.4-excimer php7.4-fpm php7.4-gd php7.4-geoip php7.4-gmp php7.4-igbinary php7.4-intl php7.4-json php7.4-luasandbox php7.4-mbstring php7.4-memcached php7.4-msgpack php7.4-mysql php7.4-opcache php7.4-readline php7.4-redis php7.4-wikidiff2 php7.4-wmerrors php7.4-xml php7.4-yaml
mwdebug1001.eqiad.wmnet:
libicu67 libxml2 php-common php7.4-apcu php7.4-bcmath php7.4-bz2 php7.4-cli php7.4-common php7.4-curl php7.4-dba php7.4-excimer php7.4-fpm php7.4-gd php7.4-geoip php7.4-gmp php7.4-igbinary php7.4-intl php7.4-json php7.4-luasandbox php7.4-mbstring php7.4-memcached php7.4-msgpack php7.4-mysql php7.4-opcache php7.4-phpdbg php7.4-readline php7.4-redis php7.4-tideways php7.4-wikidiff2 php7.4-wmerrors php7.4-xml php7.4-yaml
mwmaint1002.eqiad.wmnet:
libicu67 libapache2-mod-php libapache2-mod-php7.4 libxml2 php-common php-readline php7.4-apcu php7.4-bcmath php7.4-bz2 php7.4-cli php7.4-common php7.4-curl php7.4-dba php7.4-excimer php7.4-fpm php7.4-gd php7.4-geoip php7.4-gmp php7.4-igbinary php7.4-intl php7.4-json php7.4-luasandbox php7.4-mbstring php7.4-memcached php7.4-msgpack php7.4-mysql php7.4-opcache php7.4-readline php7.4-redis php7.4-wikidiff2 php7.4-wmerrors php7.4-xml php7.4-yaml
snapshot1008.eqiad.wmnet:
libicu67 libxml2 php-common php7.4-apcu php7.4-bcmath php7.4-bz2 php7.4-cli php7.4-common php7.4-curl php7.4-dba php7.4-excimer php7.4-gd php7.4-geoip php7.4-gmp php7.4-igbinary php7.4-intl php7.4-json php7.4-luasandbox php7.4-mbstring php7.4-memcached php7.4-msgpack php7.4-mysql php7.4-opcache php7.4-readline php7.4-redis php7.4-wikidiff2 php7.4-xml php7.4-yaml
```