Page MenuHomePhabricator

Jhancock.wm (Jenn Hancock)
User

Projects

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Saturday

  • Clear sailing ahead.

User Details

User Since
Dec 5 2022, 4:37 PM (102 w, 3 d)
Availability
Available
LDAP User
Jhancock.wm
MediaWiki User
Jhancock.wm [ Global Accounts ]

Recent Activity

Today

Jhancock.wm updated the task description for T378146: Q2:rack/setup/install es204[1-6].
Thu, Nov 21, 6:30 PM · Data-Persistence-SRE, DBA, Patch-For-Review, SRE, Data-Persistence, ops-codfw, DC-Ops
Jhancock.wm closed T371400: Q1:rack/setup/install ms-be208[1-8] as Resolved.
Thu, Nov 21, 6:30 PM · SRE, SRE-swift-storage, Data-Persistence, ops-codfw, DC-Ops
Jhancock.wm closed T378990: Netbox Reporting Triage - Week of 2024-11-04 as Resolved.
Thu, Nov 21, 6:28 PM · DC-Ops
Jhancock.wm closed T380481: PowerSupplyFailure as Resolved.

replaced. errors on bmc cleared.

Thu, Nov 21, 6:27 PM · SRE, DC-Ops, ops-codfw
Jhancock.wm closed T380482: PowerSupplyFailure as Resolved.

reseated power cable and rebooted idrac. all alerts cleared.

Thu, Nov 21, 6:25 PM · SRE, DC-Ops, ops-codfw
Jhancock.wm closed T380480: PowerSupplyFailure as Resolved.

rebooted idrac. up

Thu, Nov 21, 6:24 PM · SRE, DC-Ops, ops-codfw
Jhancock.wm updated subscribers of T378146: Q2:rack/setup/install es204[1-6].

@Papaul this one server fails after the puppet certificate is generated. Can you take a look. I'll check out the other servers this afternoon.
es2041

Thu, Nov 21, 6:15 PM · Data-Persistence-SRE, DBA, Patch-For-Review, SRE, Data-Persistence, ops-codfw, DC-Ops
Jhancock.wm closed T370452: Q1:rack/setup/install thanos-be2005 as Resolved.
Thu, Nov 21, 6:11 PM · Patch-For-Review, SRE, SRE-swift-storage, Data-Persistence, ops-codfw, DC-Ops
Jhancock.wm reassigned T308459: codfw: cp2038 Correctable memory error on DIMM A3 from Jhancock.wm to Papaul.
Thu, Nov 21, 6:07 PM · DC-Ops, SRE, ops-codfw, Traffic
Jhancock.wm added a comment to T308459: codfw: cp2038 Correctable memory error on DIMM A3.

replaced with a new DIMM. coming up now

Thu, Nov 21, 6:05 PM · DC-Ops, SRE, ops-codfw, Traffic
Jhancock.wm closed T380478: PowerSupplyFailure as Resolved.

T380480

Thu, Nov 21, 6:00 PM · SRE, DC-Ops, ops-codfw
Jhancock.wm added a comment to T308459: codfw: cp2038 Correctable memory error on DIMM A3.

We have that on hand. @Vgutierrez (or anyone else in traffic) when is a good time to do this swap?

Thu, Nov 21, 5:57 PM · DC-Ops, SRE, ops-codfw, Traffic
Jhancock.wm moved T369308: Decommission clouddb2002-dev.codfw.wmnet from Backlog to Decommission on the ops-codfw board.
Thu, Nov 21, 5:55 PM · SRE, DC-Ops, ops-codfw, wikitech.wikimedia.org, Data-Persistence, cloud-services-team
Jhancock.wm added a comment to T380480: PowerSupplyFailure.

tried reseating power cable. no change.
reseated psu. no change.
changed power cable. error specifically on psu2 cleared, but misc error has not. might require idrac reboot or psu replacement. will check back after clearing other errors.

Thu, Nov 21, 5:26 PM · SRE, DC-Ops, ops-codfw
Jhancock.wm added a comment to T380478: PowerSupplyFailure.

had to reseat power cable to get alert to clear.

Thu, Nov 21, 5:16 PM · SRE, DC-Ops, ops-codfw
Jhancock.wm updated the task description for T370452: Q1:rack/setup/install thanos-be2005.
Thu, Nov 21, 4:22 PM · Patch-For-Review, SRE, SRE-swift-storage, Data-Persistence, ops-codfw, DC-Ops
Jhancock.wm closed T380341: ManagementSSHDown as Resolved.

might have been caused by fixing a similar issue in C3. reterminated both ends of the patch for C4. Caused C3 to go down again. re-terminated the remaining patch end on C3. both are up now and have remained stable for a bit. All errors have cleared.

Thu, Nov 21, 4:22 PM · collaboration-services, SRE, DC-Ops, ops-codfw
Jhancock.wm moved T380478: PowerSupplyFailure from Backlog to Hardware Failure / Troubleshoot on the ops-codfw board.
Thu, Nov 21, 4:21 PM · SRE, DC-Ops, ops-codfw
Jhancock.wm moved T380480: PowerSupplyFailure from Backlog to Hardware Failure / Troubleshoot on the ops-codfw board.
Thu, Nov 21, 4:21 PM · SRE, DC-Ops, ops-codfw
Jhancock.wm moved T380481: PowerSupplyFailure from Backlog to Hardware Failure / Troubleshoot on the ops-codfw board.
Thu, Nov 21, 4:21 PM · SRE, DC-Ops, ops-codfw
Jhancock.wm moved T380482: PowerSupplyFailure from Backlog to Hardware Failure / Troubleshoot on the ops-codfw board.
Thu, Nov 21, 4:21 PM · SRE, DC-Ops, ops-codfw

Yesterday

Jhancock.wm moved T380341: ManagementSSHDown from Backlog to Hardware Failure / Troubleshoot on the ops-codfw board.
Wed, Nov 20, 2:45 PM · collaboration-services, SRE, DC-Ops, ops-codfw

Tue, Nov 19

Jhancock.wm closed T378034: Q2:rack/setup/install elastic211[0-5] as Resolved.

@bking this is complete!

Tue, Nov 19, 5:41 PM · Data-Platform-SRE (2024.11.09 - 2024.11.29), SRE, Discovery-Search, ops-codfw, DC-Ops
Jhancock.wm updated the task description for T378034: Q2:rack/setup/install elastic211[0-5].
Tue, Nov 19, 5:38 PM · Data-Platform-SRE (2024.11.09 - 2024.11.29), SRE, Discovery-Search, ops-codfw, DC-Ops
Jhancock.wm updated the task description for T378034: Q2:rack/setup/install elastic211[0-5].
Tue, Nov 19, 5:38 PM · Data-Platform-SRE (2024.11.09 - 2024.11.29), SRE, Discovery-Search, ops-codfw, DC-Ops
Jhancock.wm added a comment to T380265: hw troubleshooting: Link down for wikikube-worker2140.codfw.wmnet.

@Papaul you might need to check the switch. I looked in the idrac and the link shows as up. physically up as well.

Tue, Nov 19, 3:54 PM · SRE, serviceops, ops-codfw, DC-Ops
Jhancock.wm closed T380228: PowerSupplyFailure as Resolved.

reseated cable on pdu side

Tue, Nov 19, 3:41 PM · SRE, DC-Ops, ops-codfw
Jhancock.wm moved T380265: hw troubleshooting: Link down for wikikube-worker2140.codfw.wmnet from Backlog to Hardware Failure / Troubleshoot on the ops-codfw board.
Tue, Nov 19, 3:37 PM · SRE, serviceops, ops-codfw, DC-Ops

Mon, Nov 18

Jhancock.wm closed T376965: Q2:rack/setup/install wikikube-worker21[56-70] as Resolved.
Mon, Nov 18, 8:25 PM · SRE, serviceops, ops-codfw, DC-Ops
Jhancock.wm moved T380228: PowerSupplyFailure from Backlog to Hardware Failure / Troubleshoot on the ops-codfw board.
Mon, Nov 18, 8:25 PM · SRE, DC-Ops, ops-codfw
Jhancock.wm closed T377896: Q2:rack/setup/install restbase203[6-8] as Resolved.

@Eevans this is complete!

Mon, Nov 18, 8:25 PM · SRE, Data-Persistence, ops-codfw, DC-Ops
Jhancock.wm updated the task description for T377896: Q2:rack/setup/install restbase203[6-8].
Mon, Nov 18, 8:23 PM · SRE, Data-Persistence, ops-codfw, DC-Ops
Jhancock.wm added a comment to T378034: Q2:rack/setup/install elastic211[0-5].

need to double check the mgmt port on 2110

Mon, Nov 18, 8:20 PM · Data-Platform-SRE (2024.11.09 - 2024.11.29), SRE, Discovery-Search, ops-codfw, DC-Ops
Jhancock.wm updated the task description for T378034: Q2:rack/setup/install elastic211[0-5].
Mon, Nov 18, 8:19 PM · Data-Platform-SRE (2024.11.09 - 2024.11.29), SRE, Discovery-Search, ops-codfw, DC-Ops
Jhancock.wm added a comment to T376965: Q2:rack/setup/install wikikube-worker21[56-70].

@Clement_Goubert last batch done!

Mon, Nov 18, 7:58 PM · SRE, serviceops, ops-codfw, DC-Ops
Jhancock.wm updated the task description for T376965: Q2:rack/setup/install wikikube-worker21[56-70].
Mon, Nov 18, 7:58 PM · SRE, serviceops, ops-codfw, DC-Ops
Jhancock.wm closed T376917: lsw-d[18]-codfw missing console port info in netbox, a subtask of T376915: Netbox Reporting Triage - Week of 2024-10-07, as Resolved.
Mon, Nov 18, 5:37 PM · DC-Ops
Jhancock.wm closed T376917: lsw-d[18]-codfw missing console port info in netbox as Resolved.
Mon, Nov 18, 5:37 PM · SRE, ops-codfw, DC-Ops
Jhancock.wm added a comment to T376965: Q2:rack/setup/install wikikube-worker21[56-70].

2163 is being a pain. gonna take a closer look today. failed during imaging but didn't catch the error.

Mon, Nov 18, 4:58 PM · SRE, serviceops, ops-codfw, DC-Ops
Jhancock.wm updated the task description for T376965: Q2:rack/setup/install wikikube-worker21[56-70].
Mon, Nov 18, 4:57 PM · SRE, serviceops, ops-codfw, DC-Ops
Jhancock.wm updated the task description for T377896: Q2:rack/setup/install restbase203[6-8].
Mon, Nov 18, 2:41 PM · SRE, Data-Persistence, ops-codfw, DC-Ops
Jhancock.wm updated the task description for T378034: Q2:rack/setup/install elastic211[0-5].
Mon, Nov 18, 2:40 PM · Data-Platform-SRE (2024.11.09 - 2024.11.29), SRE, Discovery-Search, ops-codfw, DC-Ops
Jhancock.wm moved T380144: Set up six decommissioned nodes as temporary maps-test cluster from Backlog to Hardware Failure / Troubleshoot on the ops-codfw board.
Mon, Nov 18, 2:38 PM · DC-Ops, ops-codfw, SRE

Fri, Nov 15

Jhancock.wm updated the task description for T377896: Q2:rack/setup/install restbase203[6-8].
Fri, Nov 15, 8:43 PM · SRE, Data-Persistence, ops-codfw, DC-Ops

Thu, Nov 14

Jhancock.wm claimed T378034: Q2:rack/setup/install elastic211[0-5].
Thu, Nov 14, 9:12 PM · Data-Platform-SRE (2024.11.09 - 2024.11.29), SRE, Discovery-Search, ops-codfw, DC-Ops
Jhancock.wm claimed T378146: Q2:rack/setup/install es204[1-6].
Thu, Nov 14, 9:09 PM · Data-Persistence-SRE, DBA, Patch-For-Review, SRE, Data-Persistence, ops-codfw, DC-Ops
Jhancock.wm claimed T377896: Q2:rack/setup/install restbase203[6-8].
Thu, Nov 14, 9:09 PM · SRE, Data-Persistence, ops-codfw, DC-Ops
Jhancock.wm updated the task description for T378146: Q2:rack/setup/install es204[1-6].
Thu, Nov 14, 9:07 PM · Data-Persistence-SRE, DBA, Patch-For-Review, SRE, Data-Persistence, ops-codfw, DC-Ops
Jhancock.wm updated the task description for T377896: Q2:rack/setup/install restbase203[6-8].
Thu, Nov 14, 9:06 PM · SRE, Data-Persistence, ops-codfw, DC-Ops
Jhancock.wm closed T377027: Q2:rack/setup/install wikikube-worker21[36-55] as Resolved.

@Clement_Goubert This one's complete. took me a minute to get that last one to behave.

Thu, Nov 14, 8:55 PM · Patch-For-Review, SRE, serviceops, ops-codfw, DC-Ops
Jhancock.wm updated the task description for T377027: Q2:rack/setup/install wikikube-worker21[36-55].
Thu, Nov 14, 8:51 PM · Patch-For-Review, SRE, serviceops, ops-codfw, DC-Ops
Jhancock.wm added a comment to T379719: wikikube-ctrl2002: Switch network cable from port 2 to port 1 on the 10G NIC.

I could do this today. or we can wait until next week. assuming no one wants to do a maintenance on a friday.

Thu, Nov 14, 3:24 PM · SRE, ops-codfw, Prod-Kubernetes, Kubernetes, DC-Ops, serviceops

Wed, Nov 13

Jhancock.wm updated the task description for T370452: Q1:rack/setup/install thanos-be2005.
Wed, Nov 13, 3:30 PM · Patch-For-Review, SRE, SRE-swift-storage, Data-Persistence, ops-codfw, DC-Ops
Jhancock.wm moved T379719: wikikube-ctrl2002: Switch network cable from port 2 to port 1 on the 10G NIC from Backlog to Hardware Failure / Troubleshoot on the ops-codfw board.
Wed, Nov 13, 3:18 PM · SRE, ops-codfw, Prod-Kubernetes, Kubernetes, DC-Ops, serviceops

Tue, Nov 12

Jhancock.wm added a comment to T378146: Q2:rack/setup/install es204[1-6].

A3 was full. I racked that one in A2 and updated the task description

Tue, Nov 12, 6:22 PM · Data-Persistence-SRE, DBA, Patch-For-Review, SRE, Data-Persistence, ops-codfw, DC-Ops
Jhancock.wm updated the task description for T378146: Q2:rack/setup/install es204[1-6].
Tue, Nov 12, 6:21 PM · Data-Persistence-SRE, DBA, Patch-For-Review, SRE, Data-Persistence, ops-codfw, DC-Ops
Jhancock.wm closed T379349: decommission ganeti2015/ganeti2016 as Resolved.
Tue, Nov 12, 6:21 PM · DC-Ops, ops-codfw, SRE, decommission-hardware
Jhancock.wm added a comment to T378358: ganeti2042 seems to have a broken CPU? (new Supermicro node).

Working on getting the process ironed out. I'll let you know as soon as i have an update.

Tue, Nov 12, 4:59 PM · SRE, ops-codfw, DC-Ops
Jhancock.wm updated the task description for T378990: Netbox Reporting Triage - Week of 2024-11-04.
Tue, Nov 12, 4:55 PM · DC-Ops

Fri, Nov 8

Jhancock.wm updated the task description for T378990: Netbox Reporting Triage - Week of 2024-11-04.
Fri, Nov 8, 6:33 PM · DC-Ops
Jhancock.wm updated the task description for T377027: Q2:rack/setup/install wikikube-worker21[36-55].
Fri, Nov 8, 6:03 PM · Patch-For-Review, SRE, serviceops, ops-codfw, DC-Ops
Jhancock.wm updated the task description for T377027: Q2:rack/setup/install wikikube-worker21[36-55].
Fri, Nov 8, 6:03 PM · Patch-For-Review, SRE, serviceops, ops-codfw, DC-Ops
Jhancock.wm updated the task description for T377027: Q2:rack/setup/install wikikube-worker21[36-55].
Fri, Nov 8, 3:46 PM · Patch-For-Review, SRE, serviceops, ops-codfw, DC-Ops
Jhancock.wm updated the task description for T377027: Q2:rack/setup/install wikikube-worker21[36-55].
Fri, Nov 8, 3:44 PM · Patch-For-Review, SRE, serviceops, ops-codfw, DC-Ops
Jhancock.wm closed T377007: Q2:rack/setup/install wikikube-worker21[28-35] as Resolved.

@Clement_Goubert This set is done!

Fri, Nov 8, 3:42 PM · SRE, serviceops, ops-codfw, DC-Ops
Jhancock.wm updated the task description for T377007: Q2:rack/setup/install wikikube-worker21[28-35].
Fri, Nov 8, 3:41 PM · SRE, serviceops, ops-codfw, DC-Ops
Jhancock.wm moved T379349: decommission ganeti2015/ganeti2016 from Racking Tasks to Decommission on the ops-codfw board.
Fri, Nov 8, 1:28 PM · DC-Ops, ops-codfw, SRE, decommission-hardware
Jhancock.wm moved T379349: decommission ganeti2015/ganeti2016 from Backlog to Racking Tasks on the ops-codfw board.
Fri, Nov 8, 1:28 PM · DC-Ops, ops-codfw, SRE, decommission-hardware
Jhancock.wm added a comment to T377027: Q2:rack/setup/install wikikube-worker21[36-55].

need to check 2149 in the morning. nic port not up.

Fri, Nov 8, 1:31 AM · Patch-For-Review, SRE, serviceops, ops-codfw, DC-Ops
Jhancock.wm updated the task description for T376965: Q2:rack/setup/install wikikube-worker21[56-70].
Fri, Nov 8, 1:29 AM · SRE, serviceops, ops-codfw, DC-Ops
Jhancock.wm updated the task description for T377027: Q2:rack/setup/install wikikube-worker21[36-55].
Fri, Nov 8, 1:28 AM · Patch-For-Review, SRE, serviceops, ops-codfw, DC-Ops
Jhancock.wm updated the task description for T377007: Q2:rack/setup/install wikikube-worker21[28-35].
Fri, Nov 8, 1:27 AM · SRE, serviceops, ops-codfw, DC-Ops

Thu, Nov 7

Jhancock.wm closed T378031: Q2:rack/setup/install wdqs202[67] as Resolved.

@bking all done!

Thu, Nov 7, 10:13 PM · Data-Platform-SRE (2024.10.19 - 2024.11.08), SRE, ops-codfw, Discovery-Search, DC-Ops
Jhancock.wm updated the task description for T378031: Q2:rack/setup/install wdqs202[67].
Thu, Nov 7, 10:13 PM · Data-Platform-SRE (2024.10.19 - 2024.11.08), SRE, ops-codfw, Discovery-Search, DC-Ops
Jhancock.wm updated the task description for T376965: Q2:rack/setup/install wikikube-worker21[56-70].
Thu, Nov 7, 10:12 PM · SRE, serviceops, ops-codfw, DC-Ops
Jhancock.wm updated the task description for T377027: Q2:rack/setup/install wikikube-worker21[36-55].
Thu, Nov 7, 10:11 PM · Patch-For-Review, SRE, serviceops, ops-codfw, DC-Ops
Jhancock.wm updated the task description for T377007: Q2:rack/setup/install wikikube-worker21[28-35].
Thu, Nov 7, 10:10 PM · SRE, serviceops, ops-codfw, DC-Ops
Jhancock.wm updated the task description for T378031: Q2:rack/setup/install wdqs202[67].
Thu, Nov 7, 10:10 PM · Data-Platform-SRE (2024.10.19 - 2024.11.08), SRE, ops-codfw, Discovery-Search, DC-Ops
Jhancock.wm added a comment to T379256: Port with no description on access switch.

this was my fault. alert has cleared.

Thu, Nov 7, 5:09 PM · SRE, ops-codfw, DC-Ops
Jhancock.wm closed T379256: Port with no description on access switch as Resolved.
Thu, Nov 7, 5:09 PM · SRE, ops-codfw, DC-Ops
Jhancock.wm closed T371984: Q1:rack/setup/install backup2012 as Resolved.

@jcrespo We got the BPU installed on this server.It's showing in the BIOS (attached). You can proceed with your tests.

Thu, Nov 7, 4:22 PM · SRE, Data-Persistence, Data-Persistence-Backup, ops-codfw, DC-Ops
Jhancock.wm closed T371984: Q1:rack/setup/install backup2012, a subtask of T376892: Expand media backup storage available space to 960 TB per datacenter, as Resolved.
Thu, Nov 7, 4:19 PM · Patch-For-Review, media-backups, Data-Persistence-Backup, SRE
Jhancock.wm moved T379256: Port with no description on access switch from Backlog to Hardware Failure / Troubleshoot on the ops-codfw board.
Thu, Nov 7, 3:12 PM · SRE, ops-codfw, DC-Ops
Jhancock.wm updated the task description for T378990: Netbox Reporting Triage - Week of 2024-11-04.
Thu, Nov 7, 2:16 PM · DC-Ops

Wed, Nov 6

Jhancock.wm updated the task description for T377027: Q2:rack/setup/install wikikube-worker21[36-55].
Wed, Nov 6, 11:46 PM · Patch-For-Review, SRE, serviceops, ops-codfw, DC-Ops
Jhancock.wm updated the task description for T377027: Q2:rack/setup/install wikikube-worker21[36-55].
Wed, Nov 6, 10:11 PM · Patch-For-Review, SRE, serviceops, ops-codfw, DC-Ops
Jhancock.wm updated the task description for T377027: Q2:rack/setup/install wikikube-worker21[36-55].
Wed, Nov 6, 8:02 PM · Patch-For-Review, SRE, serviceops, ops-codfw, DC-Ops
Jhancock.wm closed T376968: Q2:rack/setup/install mc-gp200[4-6] as Resolved.

@Clement_Goubert this is ready for y'all

Wed, Nov 6, 7:58 PM · SRE, serviceops, ops-codfw, DC-Ops
Jhancock.wm updated the task description for T376968: Q2:rack/setup/install mc-gp200[4-6].
Wed, Nov 6, 7:57 PM · SRE, serviceops, ops-codfw, DC-Ops
Jhancock.wm claimed T378031: Q2:rack/setup/install wdqs202[67].
Wed, Nov 6, 2:10 PM · Data-Platform-SRE (2024.10.19 - 2024.11.08), SRE, ops-codfw, Discovery-Search, DC-Ops
Jhancock.wm updated the task description for T378031: Q2:rack/setup/install wdqs202[67].
Wed, Nov 6, 2:10 PM · Data-Platform-SRE (2024.10.19 - 2024.11.08), SRE, ops-codfw, Discovery-Search, DC-Ops
Jhancock.wm closed T379116: PowerSupplyFailure as Resolved.
Wed, Nov 6, 2:10 PM · SRE, DC-Ops, ops-codfw
Jhancock.wm added a comment to T378031: Q2:rack/setup/install wdqs202[67].

it is! must have overlooked it. thanks!

Wed, Nov 6, 2:10 PM · Data-Platform-SRE (2024.10.19 - 2024.11.08), SRE, ops-codfw, Discovery-Search, DC-Ops
Jhancock.wm moved T379116: PowerSupplyFailure from Backlog to Hardware Failure / Troubleshoot on the ops-codfw board.
Wed, Nov 6, 2:05 PM · SRE, DC-Ops, ops-codfw

Tue, Nov 5

Jhancock.wm updated the task description for T377007: Q2:rack/setup/install wikikube-worker21[28-35].
Tue, Nov 5, 11:59 PM · SRE, serviceops, ops-codfw, DC-Ops
Jhancock.wm updated the task description for T377007: Q2:rack/setup/install wikikube-worker21[28-35].
Tue, Nov 5, 11:58 PM · SRE, serviceops, ops-codfw, DC-Ops
Jhancock.wm updated the task description for T377007: Q2:rack/setup/install wikikube-worker21[28-35].
Tue, Nov 5, 9:51 PM · SRE, serviceops, ops-codfw, DC-Ops
Jhancock.wm added a comment to T378146: Q2:rack/setup/install es204[1-6].

@Marostegui this should make it diverse. lmk if you want something different.
es2043 > C7
es2045 > A7
es2046 > B7

Tue, Nov 5, 5:16 PM · Data-Persistence-SRE, DBA, Patch-For-Review, SRE, Data-Persistence, ops-codfw, DC-Ops
Jhancock.wm added a comment to T378031: Q2:rack/setup/install wdqs202[67].

@bking we got these in. Please update the site.pp file. We should be able to get these to you by End of week. end of next week at the latest.

Tue, Nov 5, 5:15 PM · Data-Platform-SRE (2024.10.19 - 2024.11.08), SRE, ops-codfw, Discovery-Search, DC-Ops
Jhancock.wm updated the task description for T378031: Q2:rack/setup/install wdqs202[67].
Tue, Nov 5, 5:13 PM · Data-Platform-SRE (2024.10.19 - 2024.11.08), SRE, ops-codfw, Discovery-Search, DC-Ops