Page MenuHomePhabricator
Create Task
Maniphest T267248

create mwdebug1003 - ganeti VM with buster and appserver role
Closed, ResolvedPublic
Actions

Description

We agreed that we want a new ganeti VM on buster, which uses the MW appserver role, similar to the previous testvm1001 but
named as a proper "mwdebug" server.

  • request VM, define needed CPU/RAM/disk (done in T268044, 4 CPU, 4GB RAM, 50 GB disk, copying existing mwdebug machines)
  • create VM (with insetup role)
  • add to DHCP, install the OS
  • create mcrouter cert (https://wikitech.wikimedia.org/wiki/Memcached_for_MediaWiki/mcrouter#Generate_certs_for_a_new_host)
  • apply the canary_appserver puppet role to it
  • add to conftool data under "testserver" section
  • ensure PHP72 APT component gets installed and PHP packages are installed
  • sync puppet compiler facts and add fake secrets in labs/private, confirm puppet changes can be compiled on mwdebug1003
  • check and list remaining puppet errors and missing packages
  • add in the WikimediaDebug extension (the extension repo itself and matching change in trafficserver LUA for x-debug header routing)

Work on this will start mid-November, from Nov 16 on.

Details

Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedNoneT248925 Make MediaWiki release tarball compatible with PHP 8.0
 ResolvedJdforrester-WMFT300463 Make PHP 8.0 voting on MW master
 ResolvedJdforrester-WMFT313563 Bump lcobucci/jwt & league/uri-components for php8
 ResolvedJdforrester-WMFT313564 Bump onoi/message-reporter in vendor.git to 1.4.2 for php 8 support
 ResolvedJdforrester-WMFT247658 Make Wikimedia Production MediaWiki compatible with PHP 7.4
 Resolved• toanT243590 Fix WikibaseDataModel CI for php 7.4
 ResolvedLucas_Werkmeister_WMDET316923 Restore skipped test in ReferenceListTest.php
 ResolvedJoeT318918 Undeploy patch to use old PHP serialization in PHP 7.4
 ResolvedJdforrester-WMFT264168 Drop PHP 7.2 support from Wikibase master branch, once Wikimedia production is on 7.4
 ResolvedLadsgroupT270740 Drop hacky support for doctrine/dbal class renaming
 InvalidNoneT303505 [S] Remove Deprecated methods "serialize" and "unserialize" after php production upgrade to PHP 7.4
 ResolvedReedyT251043 Cleanup css-sanitizer when it only requires PHP >= 7.4
 OpenNoneT166010 The Great Namespaceization Effort
 ResolvedtstarlingT277618 var_dump() on various objects writes gigabytes of data and takes minutes to run
 ResolvedJdforrester-WMFT261872 Drop PHP 7.2 & 7.3 support from MediaWiki master branch, once Wikimedia production is on 7.4
 StalledNoneT302086 Set scap minimum python version to 3.7
 ResolvedNoneT247045 Migrate all of production metal and VMs to Buster or later
 ResolvedPRODUCTION ERRORLegoktmT293568 PHP Notice: Undefined offset in wikimedia/remex-html when rendering rest.php error page
 ResolvedtstarlingT297667 mysqli/mysqlnd memory leak
 ResolvedJoeT271736 Migrate WMF production from PHP 7.2 to PHP 7.4
 ResolvedKrinkleT248191 Can't reopen table in wikidb-unittest_ (from SpecialPageFatalTest)
 ResolvedhasharT278203 Migrate all CI jobs from stretch to buster or later and drop stretch testing support
 ResolvedJdforrester-WMFT252432 Drop MediaWiki testing in stretch and instead test only in buster
 ResolvedDzahnT245757 Upgrade MediaWiki clusters to Debian Buster (debian 10)
 ResolvedDzahnT267248 create mwdebug1003 - ganeti VM with buster and appserver role
 Resolved• GillesT268188 Release WikimediaDebug for mwdebug1003 addition

Event Timeline

Dzahn created this task.Nov 4 2020, 5:53 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 4 2020, 5:53 PM
Dzahn added a parent task: T245757: Upgrade MediaWiki clusters to Debian Buster (debian 10).Nov 4 2020, 5:54 PM
Dzahn claimed this task.Nov 4 2020, 6:34 PM
Dzahn mentioned this in T268044: eqiad: 1 VM request for mwdebug (mwdebug1003).Nov 17 2020, 5:16 PM
Dzahn triaged this task as High priority.Nov 17 2020, 5:22 PM
Dzahn updated the task description. (Show Details)
Dzahn added a comment.Nov 17 2020, 5:24 PM

Just did a knowledge transfer session with Hugh and we created the VM together as part of that. As we talked about in our meeting 2 weeks ago.

VM now exists in site.pp with "insetup" role. Tomorrow we will have a second session like that where we go from that to the MW appserver role for which we need to
create the mcrouter cert first etc.

Dzahn moved this task from Incoming 🐫 to Doing 😎 on the serviceops board.Nov 17 2020, 5:25 PM
gerritbot added a comment.Nov 18 2020, 4:16 PM

Change 641756 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] site: add canary appserver role on mwdebug1003

https://gerrit.wikimedia.org/r/641756

gerritbot added a project: Patch-For-Review.Nov 18 2020, 4:16 PM

Change 641757 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] add mwdebug1003 to conftool-data

https://gerrit.wikimedia.org/r/641757

gerritbot added a comment.Nov 18 2020, 4:18 PM

Change 641759 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] trafficserver: add mwdebug1003 to x-wikimedia-debug-routing map

https://gerrit.wikimedia.org/r/641759

gerritbot added a comment.Nov 18 2020, 4:20 PM

Change 641760 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] DHCP: add mwdebug1003

https://gerrit.wikimedia.org/r/641760

gerritbot added a comment.Nov 18 2020, 5:09 PM

Change 641760 merged by Dzahn:
[operations/puppet@production] DHCP: add mwdebug1003

https://gerrit.wikimedia.org/r/641760

gerritbot added a comment.Nov 18 2020, 5:29 PM

Change 641802 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[labs/private@master] add fake certificates for mwdebug1003

https://gerrit.wikimedia.org/r/641802

gerritbot added a comment.Nov 18 2020, 5:31 PM

Change 641802 merged by Dzahn:
[labs/private@master] add fake certificates for mwdebug1003

https://gerrit.wikimedia.org/r/641802

Dzahn mentioned this in rLPRI686debe3c534: add fake certificates for mwdebug1003.Nov 18 2020, 5:45 PM
gerritbot added a comment.Nov 18 2020, 5:45 PM

Change 641756 merged by Dzahn:
[operations/puppet@production] site: add canary appserver role on mwdebug1003

https://gerrit.wikimedia.org/r/641756

Dzahn updated the task description. (Show Details)Nov 18 2020, 5:55 PM
jijiki updated the task description. (Show Details)Nov 18 2020, 9:28 PM
gerritbot added a comment.Nov 18 2020, 9:30 PM

Change 641835 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[performance/WikimediaDebug@master] background.js: add mwdebug1003 in the list of servers

https://gerrit.wikimedia.org/r/641835

Dzahn updated the task description. (Show Details)Nov 18 2020, 9:43 PM
gerritbot added a comment.Nov 18 2020, 9:44 PM

Change 641757 merged by Dzahn:
[operations/puppet@production] add mwdebug1003 to conftool-data

https://gerrit.wikimedia.org/r/641757

Dzahn updated the task description. (Show Details)Nov 18 2020, 9:45 PM
Stashbot added a comment.Nov 18 2020, 9:47 PM

Mentioned in SAL (#wikimedia-operations) [2020-11-18T21:47:02Z] <mutante> mwdebug1003 - scap pull - T267248

Dzahn added a comment.Nov 18 2020, 9:49 PM
[cumin1001:~] $ sudo -i confctl select name=mwdebug1003.eqiad.wmnet get
{"mwdebug1003.eqiad.wmnet": {"weight": 0, "pooled": "inactive"}, "tags": "dc=eqiad,cluster=testserver,service=apache2"}
Stashbot added a comment.Nov 18 2020, 9:53 PM

Mentioned in SAL (#wikimedia-operations) [2020-11-18T21:53:52Z] <mutante> mwdebug1003 - restarting ferm because config was generated but service not restarted due to puppet dependency errors, breaking NRPE monitoring T267248

Krinkle added a project: WikimediaDebug.Nov 18 2020, 10:35 PM
Restricted Application added a project: Performance-Team. · View Herald TranscriptNov 18 2020, 10:35 PM
Krinkle mentioned this in T268188: Release WikimediaDebug for mwdebug1003 addition.Nov 18 2020, 10:37 PM
gerritbot added a comment.Nov 19 2020, 9:38 AM

Change 641835 merged by jenkins-bot:
[performance/WikimediaDebug@master] background.js: add mwdebug1003 in the list of servers

https://gerrit.wikimedia.org/r/641835

gerritbot added a comment.Nov 20 2020, 7:06 PM

Change 641759 merged by Dzahn:
[operations/puppet@production] trafficserver: add mwdebug1003 to x-wikimedia-debug-routing map

https://gerrit.wikimedia.org/r/641759

Maintenance_bot removed a project: Patch-For-Review.Nov 20 2020, 7:10 PM
gerritbot added a comment.Nov 20 2020, 10:50 PM

Change 642567 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] mediawiki::php: allow opting-in to use the PHP72 component on buster

https://gerrit.wikimedia.org/r/642567

gerritbot added a project: Patch-For-Review.Nov 20 2020, 10:50 PM
gerritbot added a comment.Nov 20 2020, 11:53 PM

Change 642574 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[labs/private@master] fix the name of fake mcrouter cert for mwdebug1003

https://gerrit.wikimedia.org/r/642574

gerritbot added a comment.Nov 20 2020, 11:54 PM

Change 642574 merged by Dzahn:
[labs/private@master] fix the name of fake mcrouter cert for mwdebug1003

https://gerrit.wikimedia.org/r/642574

gerritbot added a comment.Nov 21 2020, 12:04 AM

Change 642567 merged by Dzahn:
[operations/puppet@production] mediawiki::php: allow opting-in to use the PHP72 component on buster

https://gerrit.wikimedia.org/r/642567

Dzahn mentioned this in rLPRIbab292de52a5: fix the name of fake mcrouter cert for mwdebug1003.Nov 21 2020, 12:10 AM
Dzahn added a subscriber: MoritzMuehlenhoff.Nov 21 2020, 12:14 AM

After the change above the PHP72 APT component has been added on mwdebug1003 and now we have the following PHP 7.2 packages installed by puppet on buster:

[mwdebug1003:~] $ dpkg -l | grep php
ii  php-common                           2:69 0~20190215163918.14 stretch~1.gbpfa617b wmf1 all          Common files for PHP packages
ii  php7.2-bcmath                        7.2.31-1 0~20200514.41 debian9~1.gbpe2a56b wmf1   amd64        Bcmath module for PHP
ii  php7.2-bz2                           7.2.31-1 0~20200514.41 debian9~1.gbpe2a56b wmf1   amd64        bzip2 module for PHP
ii  php7.2-common                        7.2.31-1 0~20200514.41 debian9~1.gbpe2a56b wmf1   amd64        documentation, examples and common module for PHP
ii  php7.2-dba                           7.2.31-1 0~20200514.41 debian9~1.gbpe2a56b wmf1   amd64        DBA module for PHP
ii  php7.2-gd                            7.2.31-1 0~20200514.41 debian9~1.gbpe2a56b wmf1   amd64        GD module for PHP
ii  php7.2-gmp                           7.2.31-1 0~20200514.41 debian9~1.gbpe2a56b wmf1   amd64        GMP module for PHP
ii  php7.2-mbstring                      7.2.31-1 0~20200514.41 debian9~1.gbpe2a56b wmf1   amd64        MBSTRING module for PHP
ii  php7.2-mysql                         7.2.31-1 0~20200514.41 debian9~1.gbpe2a56b wmf1   amd64        MySQL module for PHP
ii  php7.2-opcache                       7.2.31-1 0~20200514.41 debian9~1.gbpe2a56b wmf1   amd64        Zend OpCache module for PHP
ii  php7.2-xml                           7.2.31-1 0~20200514.41 debian9~1.gbpe2a56b wmf1   amd64        DOM, SimpleXML, WDDX, XML, and XSL module for PHP

Thanks to @MoritzMuehlenhoff for building them.

Dzahn updated the task description. (Show Details)Nov 21 2020, 12:15 AM
Dzahn updated the task description. (Show Details)
Dzahn updated the task description. (Show Details)
Dzahn added a comment.Nov 21 2020, 12:45 AM

remaining issues listed at T245757#6637932

gerritbot added a comment.Nov 23 2020, 9:11 PM

Change 643093 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] mediawiki::php: fix hardcoded stretch dist name for PHP 72 packages

https://gerrit.wikimedia.org/r/643093

gerritbot added a comment.Nov 23 2020, 9:31 PM

Change 643093 merged by Dzahn:
[operations/puppet@production] mediawiki::php: fix hardcoded stretch dist name for PHP 72 packages

https://gerrit.wikimedia.org/r/643093

Stashbot added a comment.Nov 23 2020, 9:54 PM

Mentioned in SAL (#wikimedia-operations) [2020-11-23T21:54:07Z] <mutante> mwdebug1003 - removing php packages and letting puppet reinstall them after it has the correct APT config T267248

Dzahn closed this task as Resolved.EditedNov 23 2020, 9:58 PM

VM has been created, added to Debug extension, added to conftool. Puppet role applied, puppet sets up APT component for buster, pulls correct PHP packages now.

Few remaining issues are part of T245757.

They are:

python-pil
python-imaging
Dzahn removed a project: Patch-For-Review.Nov 23 2020, 9:58 PM
Dzahn updated the task description. (Show Details)
Krinkle closed subtask T268188: Release WikimediaDebug for mwdebug1003 addition as Resolved.Nov 27 2020, 7:41 PM
Dzahn mentioned this in T245757: Upgrade MediaWiki clusters to Debian Buster (debian 10).Feb 6 2021, 1:21 AM
Dzahn mentioned this in T274023: Convert mwdebug VMs to debian buster.

T274023 is for mwdebug1004, mwdebug2003, mwdebug2004

Reedy mentioned this in T274026: Add 3 new canaries to WikimediaDebug.Feb 6 2021, 2:02 AM
Stashbot added a comment.Apr 16 2021, 11:47 PM

Mentioned in SAL (#wikimedia-operations) [2021-04-16T23:47:28Z] <mutante> decom'ing mwdebug1003, stretch VM created in T267248

ops-monitoring-bot added a comment.Apr 16 2021, 11:58 PM

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mwdebug1003.eqiad.wmnet

  • mwdebug1003.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqiad.wmnet to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqiad.wmnet to Netbox
gerritbot added a comment.Apr 16 2021, 11:59 PM

Change 680393 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] trafficserver: remove mwdebug1003 from x-wikimedia-debug-routing

https://gerrit.wikimedia.org/r/680393

gerritbot added a project: Patch-For-Review.Apr 16 2021, 11:59 PM
gerritbot added a comment.Apr 19 2021, 5:59 PM

Change 680393 merged by Dzahn:

[operations/puppet@production] trafficserver: remove mwdebug1003 from x-wikimedia-debug-routing

https://gerrit.wikimedia.org/r/680393

gerritbot added a comment.Apr 19 2021, 6:20 PM

Change 681144 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/mediawiki-config@master] remove mwdebug1003 from list of debug servers

https://gerrit.wikimedia.org/r/681144

gerritbot added a comment.Apr 20 2021, 3:19 PM

Change 681144 merged by jenkins-bot:

[operations/mediawiki-config@master] remove mwdebug1003 from list of debug servers

https://gerrit.wikimedia.org/r/681144

Stashbot added a comment.Apr 20 2021, 3:20 PM

Mentioned in SAL (#wikimedia-operations) [2021-04-20T15:20:46Z] <urbanecm@deploy1002> Synchronized debug.json: dc6647b9c674429c0811116e0caca7639b766e77: remove mwdebug1003 from list of debug servers (T267248) (duration: 00m 57s)

Stashbot added a comment.Apr 20 2021, 3:22 PM

Mentioned in SAL (#wikimedia-operations) [2021-04-20T15:21:59Z] <urbanecm@deploy1002> Synchronized docroot/noc/conf/debug.json: dc6647b9c674429c0811116e0caca7639b766e77: remove mwdebug1003 from list of debug servers (T267248) (duration: 00m 58s)

gerritbot added a comment.May 6 2021, 10:17 PM

Change 685907 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] site: update comment on mwdebug servers, remove mwdebug1003

https://gerrit.wikimedia.org/r/685907

gerritbot added a comment.May 6 2021, 10:19 PM

Change 685907 merged by Dzahn:

[operations/puppet@production] site: update comment on mwdebug servers, remove mwdebug1003

https://gerrit.wikimedia.org/r/685907

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits