Page MenuHomePhabricator
Create Task
Maniphest T269377

Upgrade snapshot hosts to Buster
Closed, ResolvedPublic
Actions

Description

Dumps (snapshot*) hosts should be migrated to buster around the same time we upgrade our other mediawiki clusters.

Details

Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedNoneT248925 Make MediaWiki release tarball compatible with PHP 8.0
 ResolvedJdforrester-WMFT300463 Make PHP 8.0 voting on MW master
 ResolvedJdforrester-WMFT313563 Bump lcobucci/jwt & league/uri-components for php8
 ResolvedJdforrester-WMFT313564 Bump onoi/message-reporter in vendor.git to 1.4.2 for php 8 support
 ResolvedJdforrester-WMFT247658 Make Wikimedia Production MediaWiki compatible with PHP 7.4
 Resolved• toanT243590 Fix WikibaseDataModel CI for php 7.4
 ResolvedLucas_Werkmeister_WMDET316923 Restore skipped test in ReferenceListTest.php
 ResolvedJoeT318918 Undeploy patch to use old PHP serialization in PHP 7.4
 ResolvedJdforrester-WMFT264168 Drop PHP 7.2 support from Wikibase master branch, once Wikimedia production is on 7.4
 ResolvedLadsgroupT270740 Drop hacky support for doctrine/dbal class renaming
 InvalidNoneT303505 [S] Remove Deprecated methods "serialize" and "unserialize" after php production upgrade to PHP 7.4
 ResolvedReedyT251043 Cleanup css-sanitizer when it only requires PHP >= 7.4
 OpenNoneT166010 The Great Namespaceization Effort
 ResolvedtstarlingT277618 var_dump() on various objects writes gigabytes of data and takes minutes to run
 ResolvedJdforrester-WMFT261872 Drop PHP 7.2 & 7.3 support from MediaWiki master branch, once Wikimedia production is on 7.4
 StalledNoneT302086 Set scap minimum python version to 3.7
 ResolvedNoneT247045 Migrate all of production metal and VMs to Buster or later
 ResolvedPRODUCTION ERRORLegoktmT293568 PHP Notice: Undefined offset in wikimedia/remex-html when rendering rest.php error page
 ResolvedtstarlingT297667 mysqli/mysqlnd memory leak
 ResolvedJoeT271736 Migrate WMF production from PHP 7.2 to PHP 7.4
 ResolvedKrinkleT248191 Can't reopen table in wikidb-unittest_ (from SpecialPageFatalTest)
 ResolvedhasharT278203 Migrate all CI jobs from stretch to buster or later and drop stretch testing support
 ResolvedJdforrester-WMFT252432 Drop MediaWiki testing in stretch and instead test only in buster
 ResolvedDzahnT245757 Upgrade MediaWiki clusters to Debian Buster (debian 10)
 ResolvedArielGlennT269377 Upgrade snapshot hosts to Buster

Event Timeline

jijiki created this task.Dec 3 2020, 6:22 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 3 2020, 6:22 PM
jijiki added a parent task: T245757: Upgrade MediaWiki clusters to Debian Buster (debian 10).Dec 3 2020, 6:23 PM
• Clarakosi moved this task from Inbox to Tracking/Watching on the Platform Engineering board.Dec 8 2020, 9:13 PM
• Clarakosi added a subscriber: ArielGlenn.
jbond triaged this task as Medium priority.Dec 9 2020, 12:08 PM
ArielGlenn added a project: Dumps-Generation.Dec 9 2020, 3:35 PM

I can do the testbed host first, and then the rest. Do we have a mediawiki server on buster anywhere in the cluster yet?

MoritzMuehlenhoff subscribed.Dec 9 2020, 3:38 PM

Yes, mwdebug1003 is running Buster, you can select it with the latest version of the WikimediaDebug browser extension.

ArielGlenn added a comment.Dec 21 2020, 9:59 AM

Preliminaries:

  • build mwbzutils package for buster and make sure it passes all tests
ArielGlenn moved this task from Backlog to Up Next on the Dumps-Generation board.Dec 21 2020, 5:18 PM
ArielGlenn moved this task from Up Next to Active on the Dumps-Generation board.Jan 27 2021, 5:35 PM

I've built th package and set up a test instance in deployment-prep, but there's issues with mediawiki scripts there; see T273089 for the details.

gerritbot added a comment.Jan 29 2021, 2:17 PM

Change 659886 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] snapshot1007 (testbed host) install with buster

https://gerrit.wikimedia.org/r/659886

gerritbot added a project: Patch-For-Review.Jan 29 2021, 2:17 PM
ArielGlenn added a comment.Jan 29 2021, 2:20 PM

Tests of xml/sql dumps in buster instance in deployment-prep look good. Next step: reimage the snapshot testbed instance in production.

gerritbot added a comment.Jan 29 2021, 2:23 PM

Change 659886 merged by ArielGlenn:
[operations/puppet@production] snapshot1007 (testbed host) install with buster

https://gerrit.wikimedia.org/r/659886

ops-monitoring-bot added a comment.Jan 29 2021, 2:43 PM

Script wmf-auto-reimage was launched by ariel on cumin1001.eqiad.wmnet for hosts:

snapshot1007.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202101291442_ariel_5091_snapshot1007_eqiad_wmnet.log.

Maintenance_bot removed a project: Patch-For-Review.Jan 29 2021, 3:10 PM
ops-monitoring-bot added a comment.Jan 29 2021, 3:16 PM

Completed auto-reimage of hosts:

['snapshot1007.eqiad.wmnet']

and were ALL successful.

RhinosF1 subscribed.Jan 29 2021, 3:24 PM
gerritbot added a comment.Jan 29 2021, 5:05 PM

Change 659957 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] dumps: add a config for xml/sql dumps that writes elsewhere than prod dirs

https://gerrit.wikimedia.org/r/659957

gerritbot added a project: Patch-For-Review.Jan 29 2021, 5:05 PM
gerritbot added a comment.Jan 29 2021, 5:22 PM

Change 659957 merged by ArielGlenn:
[operations/puppet@production] dumps: add a config for xml/sql dumps that writes elsewhere than prod dirs

https://gerrit.wikimedia.org/r/659957

ArielGlenn added a comment.Jan 29 2021, 5:56 PM

Test run of elwikiquote on reimaged testbed server running buster looks good, but I should do a prefetch run tomorrow morning just to be extra sure. Then I'll be able to switch the testbed with an xml/sql dump runner for regular wikis, in time for the Feb 1 run, and see how it goes.

Maintenance_bot removed a project: Patch-For-Review.Jan 29 2021, 6:11 PM
gerritbot added a comment.Jan 31 2021, 3:46 PM

Change 660634 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] make snapshot1007 running buster a dumpsrunner and move testbed to 1005

https://gerrit.wikimedia.org/r/660634

gerritbot added a project: Patch-For-Review.Jan 31 2021, 3:46 PM
gerritbot added a comment.Jan 31 2021, 3:52 PM

Change 660634 merged by ArielGlenn:
[operations/puppet@production] make snapshot1007 running buster a dumpsrunner and move testbed to 1005

https://gerrit.wikimedia.org/r/660634

ArielGlenn added a comment.Jan 31 2021, 4:09 PM

The prefetch runs went well. I ran a small wiki on snapshot1007 (buster) and then on snapshot1005 (stretch) on the same hardware. The times were slightly faster on buster.

Assuming that all goes well with the production run on buster, which we should know in 6 or 7 days, I'll be able to convert snapshot1010 next.

Maintenance_bot removed a project: Patch-For-Review.Jan 31 2021, 4:10 PM
gerritbot added a comment.Feb 1 2021, 10:32 AM

Change 660779 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] add a proper media section to the deployment-prep dumps config file

https://gerrit.wikimedia.org/r/660779

gerritbot added a project: Patch-For-Review.Feb 1 2021, 10:33 AM
gerritbot added a comment.Feb 1 2021, 10:37 AM

Change 660779 merged by ArielGlenn:
[operations/puppet@production] add a proper media section to the deployment-prep dumps config file

https://gerrit.wikimedia.org/r/660779

gerritbot added a comment.Feb 1 2021, 10:53 AM

Change 660781 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] Make media lists dump easily runnable in deployment-prep

https://gerrit.wikimedia.org/r/660781

gerritbot added a comment.Feb 1 2021, 10:55 AM

Change 660781 merged by ArielGlenn:
[operations/puppet@production] Make media lists dump easily runnable in deployment-prep

https://gerrit.wikimedia.org/r/660781

ArielGlenn added a comment.EditedFeb 1 2021, 10:57 AM

I have tested in deployment-prep all of the "other" dumps (not xml/sql) except for the wikidata and adds-changes dumps. Those are next.

Maintenance_bot removed a project: Patch-For-Review.Feb 1 2021, 11:10 AM
gerritbot added a comment.Feb 1 2021, 12:52 PM

Change 660819 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] make adds-changes dumps easier to test in deployment-prep

https://gerrit.wikimedia.org/r/660819

gerritbot added a project: Patch-For-Review.Feb 1 2021, 12:52 PM
gerritbot added a comment.Feb 1 2021, 1:46 PM

Change 660819 merged by ArielGlenn:
[operations/puppet@production] make adds-changes dumps easier to test in deployment-prep

https://gerrit.wikimedia.org/r/660819

gerritbot added a comment.Feb 1 2021, 6:35 PM

Change 660871 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] make wikidata rdf dumps easier to test in deployment-prep

https://gerrit.wikimedia.org/r/660871

gerritbot added a comment.Feb 2 2021, 6:42 PM

Change 661170 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] refactor script for wikidata and commons rdf dumps

https://gerrit.wikimedia.org/r/661170

gerritbot added a comment.Feb 4 2021, 7:47 AM

Change 661642 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] prep for re-install of snapshot1009, 1010 with buster

https://gerrit.wikimedia.org/r/661642

gerritbot added a comment.Feb 4 2021, 7:49 AM

Change 661642 merged by ArielGlenn:
[operations/puppet@production] prep for re-install of snapshot1009, 1010 with buster

https://gerrit.wikimedia.org/r/661642

ops-monitoring-bot added a comment.Feb 4 2021, 7:55 AM

Script wmf-auto-reimage was launched by ariel on cumin1001.eqiad.wmnet for hosts:

snapshot1009.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202102040754_ariel_21056_snapshot1009_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Feb 4 2021, 8:31 AM

Completed auto-reimage of hosts:

['snapshot1009.eqiad.wmnet']

and were ALL successful.

ArielGlenn added a comment.Feb 4 2021, 8:52 AM

snapshot1009 was idle so I converted it. snapshot1010 should become idle in an hour or two, so I'll be able to do that later today. I might not do anything about snapshot1005,6 since they are due to be replaced and the replacements should be here any day now. Thy can simply be installed with buster from the start and the old servers decommissioned.

ops-monitoring-bot added a comment.Feb 4 2021, 12:48 PM

Script wmf-auto-reimage was launched by ariel on cumin1001.eqiad.wmnet for hosts:

snapshot1010.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202102041248_ariel_6167_snapshot1010_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Feb 4 2021, 1:25 PM

Completed auto-reimage of hosts:

['snapshot1010.eqiad.wmnet']

and were ALL successful.

ArielGlenn added a comment.Feb 4 2021, 3:08 PM

snapshot1010 is done. I need to do a bunch more testing before I can reimage snapshot1008.

gerritbot added a comment.Feb 8 2021, 6:51 PM

Change 662756 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[mediawiki/core@master] in deployment-prep some groups don't exist, permit scripts that use them to run

https://gerrit.wikimedia.org/r/662756

gerritbot added a comment.Feb 9 2021, 12:06 PM

Change 662756 merged by jenkins-bot:
[mediawiki/core@master] in deployment-prep some groups don't exist, permit scripts that use them to run

https://gerrit.wikimedia.org/r/662756

ReleaseTaggerBot added a project: MW-1.36-notes (1.36.0-wmf.31; 2021-02-16).Feb 9 2021, 1:00 PM
gerritbot added a comment.Feb 11 2021, 1:28 PM

Change 661170 merged by ArielGlenn:
[operations/puppet@production] refactor script for wikidata and commons rdf dumps

https://gerrit.wikimedia.org/r/661170

ArielGlenn added a comment.Feb 11 2021, 1:41 PM

While it would be nice to continue to make the wikidata entity dumps more easy to run in deployment-prep, it can wait a bit while I move to testing the wikidata json dumps, next needed for the move to buster.

gerritbot added a comment.Feb 11 2021, 7:27 PM

Change 663661 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] refactor wikidata json dumps to be easier to test on deployment-prep

https://gerrit.wikimedia.org/r/663661

gerritbot added a comment.Feb 12 2021, 1:56 PM

Change 663661 merged by ArielGlenn:
[operations/puppet@production] refactor wikidata json dumps to be easier to test on deployment-prep

https://gerrit.wikimedia.org/r/663661

gerritbot added a comment.Feb 15 2021, 7:57 AM

Change 664091 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] now that snapshot1005 is testbed host, make snapshot1007 the enwiki dumps runner

https://gerrit.wikimedia.org/r/664091

gerritbot added a comment.Feb 15 2021, 8:08 AM

Change 664091 merged by ArielGlenn:
[operations/puppet@production] now that snapshot1005 is testbed host, make snapshot1007 the enwiki dumps runner

https://gerrit.wikimedia.org/r/664091

gerritbot added a comment.Feb 15 2021, 8:10 AM

Change 664092 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] prep snapshot1005 and 1006 for reinstall with buster

https://gerrit.wikimedia.org/r/664092

gerritbot added a comment.Feb 15 2021, 8:13 AM

Change 664092 merged by ArielGlenn:
[operations/puppet@production] prep snapshot1005 and 1006 for reinstall with buster

https://gerrit.wikimedia.org/r/664092

ops-monitoring-bot added a comment.Feb 15 2021, 8:17 AM

Script wmf-auto-reimage was launched by ariel on cumin1001.eqiad.wmnet for hosts:

snapshot1005.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202102150817_ariel_8905_snapshot1005_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Feb 15 2021, 8:53 AM

Completed auto-reimage of hosts:

['snapshot1005.eqiad.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Feb 15 2021, 9:12 AM

Script wmf-auto-reimage was launched by ariel on cumin1001.eqiad.wmnet for hosts:

snapshot1006.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202102150912_ariel_1945_snapshot1006_eqiad_wmnet.log.

gerritbot added a comment.Feb 15 2021, 9:24 AM

Change 664225 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] misc dumps: move commons rdf to later on Sunday and media info to earlier

https://gerrit.wikimedia.org/r/664225

ops-monitoring-bot added a comment.Feb 15 2021, 9:50 AM

Completed auto-reimage of hosts:

['snapshot1006.eqiad.wmnet']

and were ALL successful.

ArielGlenn added a comment.Feb 15 2021, 9:57 AM

I was not going to re-image snapshot1005 and 6 because their replacements were due to have come in, but the boxes have not arrived yet and we still do not have an eta. So they are done now.

The last server remaining is snapshot1008. All "misc" dumps have been tested on beta in deployment-prep, and so the re-imaging of this host can happen next Sunday. I am rearranging the Sunday cron jobs a little so that we have a longer maintenance window going forward, see https://gerrit.wikimedia.org/r/c/operations/puppet/ /664225

gerritbot added a comment.Feb 15 2021, 9:59 AM

Change 664225 merged by ArielGlenn:
[operations/puppet@production] misc dumps: move commons rdf to later on Sunday and media info to earlier

https://gerrit.wikimedia.org/r/664225

ops-monitoring-bot added a comment.Feb 21 2021, 9:18 AM

Script wmf-auto-reimage was launched by ariel on cumin1001.eqiad.wmnet for hosts:

snapshot1008.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202102210918_ariel_28889_snapshot1008_eqiad_wmnet.log.

gerritbot added a comment.Feb 21 2021, 9:36 AM

Change 665583 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] snapshot1008 to install from buster image

https://gerrit.wikimedia.org/r/665583

gerritbot added a comment.Feb 21 2021, 9:39 AM

Change 665583 merged by ArielGlenn:
[operations/puppet@production] snapshot1008 to install from buster image

https://gerrit.wikimedia.org/r/665583

ops-monitoring-bot added a comment.Feb 21 2021, 9:48 AM

Completed auto-reimage of hosts:

['snapshot1008.eqiad.wmnet']

and were ALL successful.

ArielGlenn added a comment.Feb 21 2021, 9:52 AM

So the reimage completed but still on stretch. I've updated the install file and here we go again.

ops-monitoring-bot added a comment.Feb 21 2021, 9:52 AM

Script wmf-auto-reimage was launched by ariel on cumin1001.eqiad.wmnet for hosts:

snapshot1008.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202102210952_ariel_3141_snapshot1008_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Feb 21 2021, 10:21 AM

Completed auto-reimage of hosts:

['snapshot1008.eqiad.wmnet']

and were ALL successful.

ArielGlenn closed this task as Resolved.Feb 21 2021, 10:40 AM
ArielGlenn claimed this task.

These are done now. Closing the ticket.

Dzahn awarded a token.Feb 21 2021, 11:51 PM
ArielGlenn moved this task from Active to Done on the Dumps-Generation board.Feb 25 2021, 9:34 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits