This is the meta task for the March 2023 Datacenter switchover (eqiad -> codfw).
Switchover
Schedule
Services: Tuesday, February 28th, 2023 14:00 UTC Traffic: Tuesday, February 28th, 2023 15:00 UTC MediaWiki: Wednesday, March 1st, 2023 14:00 UTC
Repooling
Traffic repooling of eqiad: Wednesday, March 8th, 2023 restbase-async eqiad pooling: Wednesday, March 8th, 2023 Services/mediawiki-RO eqiad pooling: Tuesday, March 14th, 2023
Checklist
- Update statuspage
- Add scap lock
- Log to SAL !log Starting mediawiki datacenter switchover - T327920
- Run sre.switchdc.mediawiki cookbook until 09-restore-ttl.py https://wikitech.wikimedia.org/wiki/Switch_Datacenter#MediaWiki
- Merge https://gerrit.wikimedia.org/r/c/operations/dns/ /891552
- Run authdns-update from any authdns node
- SAL log !log Phase 9.5 Update DNS records for new database masters - T327920
- Remove scap lock
- Merge and deploy https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/ /892428
- Check reading & editing works
- Check https://en.wikipedia.org/wiki/Special:RecentChanges and EventStreams curl -s -H 'Accept: application/json' https://stream.wikimedia.org/v2/stream/recentchange | jq .
- Run exim -bp | exiqsumm on mx1001/mx2001
- Update statuspage
See also:
- https://wikitech.wikimedia.org/wiki/Switch_Datacenter#2023_switches
- https://wikitech.wikimedia.org/wiki/Deployments/Yearly_calendar/2023
Switchback
Services: Tuesday, April 25th, 2023 14:00 UTC Switching back: Wednesday, April 26th, 2023 14:00 UTC
Checklist
- Update statuspage https://manage.statuspage.io/pages/nnqjzz7cd4tj/incidents/02wx5vwvrkpx
- Add scap lock
- Log to SAL !log Starting mediawiki datacenter switchback preparation - T327920
- Run sre.switchdc.mediawiki cookbook until 00-downtime-db-readonly-checks
- Run sre.switchdc.mediawiki 01-stop-maintenance.py
- Run sre.switchdc.mediawiki cookbook until 09-restore-ttl.py https://wikitech.wikimedia.org/wiki/Switch_Datacenter#MediaWiki
- Merge https://gerrit.wikimedia.org/r/c/operations/dns/ /912235
- Run authdns-update from any authdns node
- SAL log !log Phase 9.5 Update DNS records for new database masters - T327920
- Remove scap lock
- Merge and deploy https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/ /911792
- Check reading & editing works
- @Marostegui to change pcX CNAMEs back to eqiad https://gerrit.wikimedia.org/r/912171
- Check https://en.wikipedia.org/wiki/Special:RecentChanges and EventStreams curl -s -H 'Accept: application/json' https://stream.wikimedia.org/v2/stream/recentchange | jq .
- Run exim -bp | exiqsumm on mx1001/mx2001
- Update statuspage