Page MenuHomePhabricator

Migrate WMF production from PHP 7.2 to PHP 7.4
Closed, ResolvedPublic

Assigned To
Authored By
Reedy
Jan 11 2021, 6:01 PM
Referenced Files
None
Tokens
"Love" token, awarded by abi_."Love" token, awarded by Jdforrester-WMF."Love" token, awarded by Daimona."Love" token, awarded by Ladsgroup."Love" token, awarded by kostajh."Heartbreak" token, awarded by thiemowmde.

Description

Now that the MediaWiki appservers are migrated to Buster, T245757: Upgrade MediaWiki clusters to Debian Buster (debian 10), (whilst still on PHP 7.2, but ICU from Buster), we should move WMF production to PHP 7.4 (packaged in Buster) or possibly later (PHP 7.3 went fully EOL in December 2021) so that we can drop PHP 7.2 support in MW master (like 1.35-1.37) as per T261872: Drop PHP 7.2 & 7.3 support from MediaWiki master branch, once Wikimedia production is on 7.4.

Progress:

  • App servers
    • MW servers
      • 100% of cookie-based traffic
    • Appservers - patch
    • API servers – patch
    • Job runners – patch
    • Parsoid
  • Dumps – patch
  • Deployment hosts - patch
  • Maintenance hosts – patch
  • Wikitech

Details

SubjectRepoBranchLines /-
mediawiki/vagrantmaster 50 -49
operations/mediawiki-configmaster 0 -9
operations/puppetproduction 0 -12
operations/puppetproduction 11 -10
mediawiki/vagrantmaster 50 -49
operations/puppetproduction 6 -3
mediawiki/tools/releasemaster 1 -1
operations/puppetproduction 1 -1
operations/puppetproduction 7 -0
operations/puppetproduction 3 -0
operations/puppetproduction 1 -1
operations/puppetproduction 2 -2
operations/puppetproduction 2 -2
operations/mediawiki-configmaster 1 -1
operations/puppetproduction 2 -2
operations/mediawiki-configmaster 1 -1
operations/mediawiki-configmaster 1 -1
operations/puppetproduction 1 -1
operations/puppetproduction 1 -1
operations/mediawiki-configmaster 1 -1
operations/mediawiki-configmaster 1 -1
operations/mediawiki-configmaster 1 -1
operations/mediawiki-configmaster 1 -1
operations/puppetproduction 12 -0
operations/mediawiki-configmaster 9 -0
operations/puppetproduction 8 -1
operations/mediawiki-configmaster 240 -0
Show related patches Customize query in gerrit

Related Objects

StatusSubtypeAssignedTask
ResolvedNone
ResolvedJdforrester-WMF
ResolvedJdforrester-WMF
ResolvedJdforrester-WMF
ResolvedJdforrester-WMF
Resolved toan
ResolvedLucas_Werkmeister_WMDE
ResolvedJoe
ResolvedJdforrester-WMF
ResolvedLadsgroup
InvalidNone
ResolvedReedy
OpenNone
Resolvedtstarling
ResolvedJdforrester-WMF
ResolvedPRODUCTION ERRORLegoktm
Resolvedtstarling
ResolvedJoe
ResolvedDzahn
Resolvedhashar
ResolvedJdforrester-WMF
ResolvedLadsgroup
ResolvedMoritzMuehlenhoff
Resolvedjijiki
ResolvedMoritzMuehlenhoff
ResolvedTrizek-WMF
ResolvedDzahn
Resolved Gilles
ResolvedDzahn
ResolvedRequestPapaul
Resolvedjijiki
DeclinedNone
ResolvedDzahn
ResolvedDzahn
ResolvedPapaul
Resolved Cmjohnson
ResolvedRequest Cmjohnson
ResolvedRequestPapaul
ResolvedAndrew
ResolvedArielGlenn
ResolvedDzahn
ResolvedLegoktm
ResolvedPapaul
ResolvedDzahn
Declined Gilles
ResolvedVolans
ResolvedDzahn
ResolvedLegoktm
ResolvedPleaseStand
ResolvedJoe
Resolvedtstarling
ResolvedArielGlenn
ResolvedJoe
Resolvedtstarling
ResolvedJdforrester-WMF
ResolvedJdforrester-WMF
ResolvedLegoktm
ResolvedJdforrester-WMF
ResolvedDaimona
ResolvedDaimona
ResolvedJdforrester-WMF
ResolvedJoe
ResolvedJMeybohm
ResolvedJoe
ResolvedJoe
ResolvedJoe
ResolvedJoe
ResolvedKrinkle
ResolvedJoe
ResolvedClement_Goubert
ResolvedClement_Goubert
ResolvedClement_Goubert
ResolvedMainframe98
ResolvedJoe

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 824690 merged by ArielGlenn:

[operations/puppet@production] add php7.4 install to the snapshot hosts

https://gerrit.wikimedia.org/r/824690

I'll corrdinate with @Joe in the first part of next week to see about the process for switching the snapshot hosts over to use just php7.4. We'll want that to get done before the new dumps run starts on the 1st of the month.

Change 823675 merged by Giuseppe Lavagetto:

[operations/mediawiki-config@master] Move 0.1% of user traffic to php 7.4

https://gerrit.wikimedia.org/r/823675

Change 823676 merged by Giuseppe Lavagetto:

[operations/mediawiki-config@master] Move 1% of traffic to php 7.4

https://gerrit.wikimedia.org/r/823676

My current plan is to proceed as follows:

  • Move 0.1% of the traffic on monday 22nd
  • Ramp up to […]

Note that the client-side code currently in prod doesn't work.

In T311388#8161816 on 17 August 2022:

Change 824204 merged:

[mediawiki/extensions/WikimediaEvents@master] phpEngine: Actually include the phpEngine.js file

https://gerrit.wikimedia.org/r/824204

This stack will need backporting as it landed 17th, thus out by train next Thursday 26th.

Oh yeah I planned to backport the change, but then I've been out for a week, so the traffic is being switched now :/

Change 827954 had a related patch set uploaded (by ArielGlenn; author: ArielGlenn):

[operations/puppet@production] switch snapshot hosts to use php7.4

https://gerrit.wikimedia.org/r/827954

Change 823677 merged by jenkins-bot:

[operations/mediawiki-config@master] Move 5% of traffic to php 7.4

https://gerrit.wikimedia.org/r/823677

Mentioned in SAL (#wikimedia-operations) [2022-09-01T13:09:51Z] <oblivian@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:823677|Move 5% of traffic to php 7.4 (T271736)]] (duration: 03m 45s)

Change 829217 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/puppet@production] api_appserver: convert all canaries to php 7.4 by default

https://gerrit.wikimedia.org/r/829217

Given other delays, I am now planning to move the snapshot hosts to php7.4 when the current dump run completes, about the 16th or 17th of the month.

Change 829550 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/puppet@production] canary_appserver: use php 7.4 by default

https://gerrit.wikimedia.org/r/829550

Change 829551 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/puppet@production] jobrunner: convert to use php 7.4 by default

https://gerrit.wikimedia.org/r/829551

Change 829552 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/puppet@production] api appserver: convert to php 7.4 by default

https://gerrit.wikimedia.org/r/829552

Change 823678 merged by jenkins-bot:

[operations/mediawiki-config@master] Move 10% of traffic to php 7.4

https://gerrit.wikimedia.org/r/823678

Mentioned in SAL (#wikimedia-operations) [2022-09-05T07:07:48Z] <oblivian@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:823678|Move 10% of traffic to php 7.4 (T271736)]] (duration: 03m 50s)

Change 829217 merged by Giuseppe Lavagetto:

[operations/puppet@production] api_appserver: convert all canaries to php 7.4 by default

https://gerrit.wikimedia.org/r/829217

Change 829550 merged by Giuseppe Lavagetto:

[operations/puppet@production] canary_appserver: use php 7.4 by default

https://gerrit.wikimedia.org/r/829550

Change 823679 merged by jenkins-bot:

[operations/mediawiki-config@master] Move 1 of 6 users to php 7.4

https://gerrit.wikimedia.org/r/823679

Mentioned in SAL (#wikimedia-operations) [2022-09-06T07:11:48Z] <oblivian@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:823679|Move 1 of 6 users to php 7.4 (T271736)]] (duration: 04m 06s)

Change 823680 merged by Giuseppe Lavagetto:

[operations/mediawiki-config@master] Move 50% of traffic to php 7.4

https://gerrit.wikimedia.org/r/823680

Mentioned in SAL (#wikimedia-operations) [2022-09-07T08:42:20Z] <oblivian@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:823680|Move 50% of traffic to php 7.4 (T271736)]] (duration: 04m 00s)

Change 827954 merged by ArielGlenn:

[operations/puppet@production] switch snapshot hosts to use php7.4

https://gerrit.wikimedia.org/r/827954

Change 823681 merged by jenkins-bot:

[operations/mediawiki-config@master] Move 100% of cookie-accepting clients to php 7.4

https://gerrit.wikimedia.org/r/823681

Mentioned in SAL (#wikimedia-operations) [2022-09-26T07:26:08Z] <oblivian@deploy1002> Started scap: Backport for [[gerrit:823681|Move 100% of cookie-accepting clients to php 7.4 (T271736)]]

Mentioned in SAL (#wikimedia-operations) [2022-09-26T07:26:31Z] <oblivian@deploy1002> oblivian and oblivian: Backport for [[gerrit:823681|Move 100% of cookie-accepting clients to php 7.4 (T271736)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet

Mentioned in SAL (#wikimedia-operations) [2022-09-26T07:31:40Z] <oblivian@deploy1002> Finished scap: Backport for [[gerrit:823681|Move 100% of cookie-accepting clients to php 7.4 (T271736)]] (duration: 05m 31s)

Change 829551 merged by Giuseppe Lavagetto:

[operations/puppet@production] jobrunner: convert to use php 7.4 by default

https://gerrit.wikimedia.org/r/829551

Change 835629 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/puppet@production] role::mediawiki::maintenance: switch to php 7.4 by default

https://gerrit.wikimedia.org/r/835629

Change 835629 merged by Giuseppe Lavagetto:

[operations/puppet@production] role::mediawiki::maintenance: switch to php 7.4 by default

https://gerrit.wikimedia.org/r/835629

Change 829552 merged by Giuseppe Lavagetto:

[operations/puppet@production] api appserver: convert to php 7.4 by default

https://gerrit.wikimedia.org/r/829552

Change 836094 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/puppet@production] deployment_server: switch to php 7.4 by default

https://gerrit.wikimedia.org/r/836094

Change 836143 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/puppet@production] cloudweb: install php 7.4

https://gerrit.wikimedia.org/r/836143

Change 836144 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/puppet@production] wikitech: switch to php 7.4

https://gerrit.wikimedia.org/r/836144

Change 836094 merged by Giuseppe Lavagetto:

[operations/puppet@production] deployment_server: switch to php 7.4 by default

https://gerrit.wikimedia.org/r/836094

Change 836143 merged by Giuseppe Lavagetto:

[operations/puppet@production] cloudweb: install php 7.4

https://gerrit.wikimedia.org/r/836143

Change 836144 merged by Giuseppe Lavagetto:

[operations/puppet@production] wikitech: switch to php 7.4

https://gerrit.wikimedia.org/r/836144

Production is effectively migrated, so this task can be considered "Resolved" as far as CI etc are concerned.

To clean up php 7.2 from production, we'll do as follows:

  • Remove php 7.2 from puppet's profile::mediawiki::php::php_versions; this will not remove packages or configurations, minus the configuration in apache and rsyslog regarding php 7.2, and the observability stack
  • Stop and mask the systemd services related to php 7.2
  • Switch pybal back to check a single url

A few weeks later we can proceed and also clean up the packages / files related to php, but that will be a separate task.

awarding token is not enough: <3 <3 <3 <3 <3 <3

Change 836932 had a related patch set uploaded (by Ahmon Dancy; author: Ahmon Dancy):

[operations/puppet@production] scap.cfg.erb: 7.2 -> 7.4

https://gerrit.wikimedia.org/r/836932

Change 836933 had a related patch set uploaded (by Ahmon Dancy; author: Ahmon Dancy):

[mediawiki/tools/release@master] auto-stage: Use PHP 7.4

https://gerrit.wikimedia.org/r/836933

Change 836933 merged by jenkins-bot:

[mediawiki/tools/release@master] auto-stage: Use PHP 7.4

https://gerrit.wikimedia.org/r/836933

Change 836932 merged by Giuseppe Lavagetto:

[operations/puppet@production] scap.cfg.erb: 7.2 -> 7.4

https://gerrit.wikimedia.org/r/836932

Change 837166 had a related patch set uploaded (by Mainframe98; author: Mainframe98):

[mediawiki/vagrant@master] Switch to PHP 7.4

https://gerrit.wikimedia.org/r/837166

Change 837166 abandoned by Mainframe98:

[mediawiki/vagrant@master] Switch to PHP 7.4

Reason:

This patch is targeting the wrong branch and needs to be redone after buster-migration has been rebased on top of master

https://gerrit.wikimedia.org/r/837166

Change 838083 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/puppet@production] logging: switch to php 7.4

https://gerrit.wikimedia.org/r/838083

Change 838083 merged by Giuseppe Lavagetto:

[operations/puppet@production] logging: switch to php 7.4

https://gerrit.wikimedia.org/r/838083

Change 838843 had a related patch set uploaded (by RLazarus; author: RLazarus):

[operations/puppet@production] httpbb: Remove PHP7.2/7.4 routing tests, now that force_php72=1 gets 7.4

https://gerrit.wikimedia.org/r/838843

Change 838843 abandoned by RLazarus:

[operations/puppet@production] httpbb: Remove PHP7.2/7.4 routing tests, now that force_php72=1 gets 7.4

Reason:

https://gerrit.wikimedia.org/r/c/operations/puppet/ /838112

https://gerrit.wikimedia.org/r/838843

Change 839499 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/mediawiki-config@master] Stop assigning the PHP_ENGINE cookie

https://gerrit.wikimedia.org/r/839499

With one last outstanding patch to cleanup mediawiki-config, I think we can anyways declare this task resolved.

Change 842964 had a related patch set uploaded (by Mainframe98; author: Mainframe98):

[mediawiki/vagrant@master] Switch to PHP 7.4

https://gerrit.wikimedia.org/r/842964

Change 839499 merged by jenkins-bot:

[operations/mediawiki-config@master] Stop assigning the PHP_ENGINE cookie

https://gerrit.wikimedia.org/r/839499

Mentioned in SAL (#wikimedia-operations) [2022-10-24T09:18:14Z] <oblivian@deploy1002> Started scap: Backport for [[gerrit:839499|Stop assigning the PHP_ENGINE cookie (T271736)]]

Mentioned in SAL (#wikimedia-operations) [2022-10-24T09:18:34Z] <oblivian@deploy1002> oblivian and oblivian: Backport for [[gerrit:839499|Stop assigning the PHP_ENGINE cookie (T271736)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet

Mentioned in SAL (#wikimedia-operations) [2022-10-24T09:23:15Z] <oblivian@deploy1002> Finished scap: Backport for [[gerrit:839499|Stop assigning the PHP_ENGINE cookie (T271736)]] (duration: 04m 59s)

Change 842964 merged by jenkins-bot:

[mediawiki/vagrant@master] Switch to PHP 7.4

https://gerrit.wikimedia.org/r/842964