https://gerrit.wikimedia.org/r/c/operations/puppet/ /1057270

@Jelto After also adding a job to scrape the metrics I am now seeing _some_ data trickle in here:

Change #1014116 merged by Dzahn:

[operations/puppet@production] prometheus/ops: add config for scraping apache metrics on planet servers

https://gerrit.wikimedia.org/r/1014116

Change #1014116 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] prometheus/ops: add config for scraping apache metrics on planet servers

https://gerrit.wikimedia.org/r/1014116

In T359556#9614359, @Jelto wrote:

Probably the argument has to be --scrape_uri in the newer version?

After https://gerrit.wikimedia.org/r/1009775 has been deployed and entirely dropped the argument parameter the package is now installed again on planet* hosts and the service doesn't fail to start any longer.

Change #1009775 merged by Dzahn:

[operations/puppet@production] prometheus/apache_exporter: drop argument parameter

https://gerrit.wikimedia.org/r/1009775

@Jelto I confirmed the change is that on buster and bullseye the commandline args of the exporter start with "-" but in bookworm they now start with "--".

Change 1009775 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] apache_exporter: fix argument syntax in bookworm

https://gerrit.wikimedia.org/r/1009775

Currently the exporter fails with

20:14 < denisse> Hello mutante, if the metrics are already available via HTTP, the next step would be to configure Prometheus to scrape these metrics by adding a new job.

At least the package is installed now on planet* servers.

Change 1009575 merged by Dzahn:

[operations/puppet@production] planet: add prometheus apache exporter to role

https://gerrit.wikimedia.org/r/1009575

Change 1009575 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] planet: add prometheus apache exporter to role

https://gerrit.wikimedia.org/r/1009575

Change 983207 merged by Dzahn:

[operations/puppet@production] planet: remove support for buster

https://gerrit.wikimedia.org/r/983207

This is done! Planet is now hosted on new bookworm VMs and the buster VMs have been deleted.

Change 982157 merged by Dzahn:

[operations/puppet@production] site: remove buster VMs from planet regex

https://gerrit.wikimedia.org/r/982157

cookbooks.sre.hosts.decommission executed by dzahn@cumin2002 for hosts: planet2002.codfw.wmnet

• planet2002.codfw.wmnet (PASS)
  • Downtimed host on Icinga/Alertmanager
  • Found Ganeti VM
  • VM shutdown
  • Started forced sync of VMs in Ganeti cluster codfw to Netbox
  • Removed from DebMonitor
  • Removed from Puppet master and PuppetDB
  • VM removed
  • Started forced sync of VMs in Ganeti cluster codfw to Netbox

Change 983207 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] planet: remove support for buster

https://gerrit.wikimedia.org/r/983207

cookbooks.sre.hosts.decommission executed by dzahn@cumin2002 for hosts: planet1002.eqiad.wmnet

• planet1002.eqiad.wmnet (PASS)
  • Downtimed host on Icinga/Alertmanager
  • Found Ganeti VM
  • VM shutdown
  • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
  • Removed from DebMonitor
  • Removed from Puppet master and PuppetDB
  • VM removed
  • Started forced sync of VMs in Ganeti cluster eqiad to Netbox

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: planet1002.eqiad.wmnet

• planet1002.eqiad.wmnet (FAIL)
  • Downtimed host on Icinga/Alertmanager
  • Host steps raised exception: Error while performing request to RAPI

That's indeed a bit weird, but she also writes that these articles have been published years ago in a banner kind of thing on top.

I can't help but notice that en.planet.wikimedia.org looks a little broken right now (It has posts from sumana from long ago), is that possibly related to this change (Just think timing is coincidental)

We are now serving all lang versions from a bookworm VM in eqiad.

Mentioned in SAL (#wikimedia-operations) [2023-12-12T22:57:09Z] <mutante> planet - switched to eqiad and bookworm backend (T348392 T345617) - https://meta.wikimedia.org/wiki/Planet_Wikimedia

Change 982489 merged by Dzahn:

[operations/dns@master] planet: switch to new eqiad backend

https://gerrit.wikimedia.org/r/982489

Change 982489 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/dns@master] planet: switch to new eqiad backend

https://gerrit.wikimedia.org/r/982489

Mentioned in SAL (#wikimedia-operations) [2023-12-12T22:43:52Z] <mutante> planet2003 -manually upgrade rawdog package to 3.0.2 T348392

Change 982484 merged by Dzahn:

[operations/puppet@production] planet: enable feed updates on both new VMs

https://gerrit.wikimedia.org/r/982484

Change 982156 merged by Dzahn:

[operations/dns@master] Switch planet to bookworm VM backends

https://gerrit.wikimedia.org/r/982156

after some more debugging, fixing privileges on some state files and removing a broken feed from the cs lang version, now our httpbb tests work on both old and new machines:

Change 982484 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] planet: enable feed updates on both new VMs

https://gerrit.wikimedia.org/r/982484

Change 982157 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] site: remove buster VMs from planet regex

https://gerrit.wikimedia.org/r/982157