Page MenuHomePhabricator
Create Task
Maniphest T290899

Q1: eqiad: (32) PDUs for expansion
Closed, ResolvedPublic
Actions

Description

This task will track the racking of the 32 new PDUs into the new eqiad row expansion.

8 cabinets of C2[LS]42CE-YCMFAM00 & 8 cabinets of C2[LS]54CE-YCMFA900 along with (16) PDU mounting brackets for Rittal racks.

The new row will likely be row E [1-8] and F [1-8] or just E [1-16].

The first 8 cabinets will get the C2[LS]42CE-YCMFAM00, the other 8 will get the C2[LS]54CE-YCMFA900 when they arrive in November.

Please note we aren't allowed to energize the PDUs, and must open a smart hands ticket with DC6 staff to do so. Once all 8 of the first racks are staged, a single ticket can be opened to have all 8 energized/plugged in.

Rack E1:

  • - receive in both C2[LS]42CE-YCMFAM00 and the mounting brackets via T286593, enter into Netbox.
  • - align the vertical rails in the cabinet to standard depths to allow full mounting of the PDUs and vertical cable managers.
  • - install C2[LS]42CE-YCMFAM00 into the rack - ensure the master and expansion PDUs are linked by the link cable included in the box. (silver ribbon cable, RJ type connector.)
  • - connect PDU to serial console server
  • - open ticket for PDUs to be plugged in / energized once racks E[1-8] are ready.
  • - confirm PDU energized for setup
  • - setup PDU according to directions on https://wikitech.wikimedia.org/wiki/SRE/Dc-operations/Platform-specific_documentation/ServerTech

Rack E2:

  • - receive in both C2[LS]42CE-YCMFAM00 and the mounting brackets via T286593, enter into Netbox.
  • - align the vertical rails in the cabinet to standard depths to allow full mounting of the PDUs and vertical cable managers.
  • - install C2[LS]42CE-YCMFAM00 into the rack - ensure the master and expansion PDUs are linked by the link cable included in the box. (silver ribbon cable, RJ type connector.)
  • - connect PDU to serial console server
  • - open ticket for PDUs to be plugged in / energized once racks E[1-8] are ready.
  • - confirm PDU energized for setup
  • - setup PDU according to directions on https://wikitech.wikimedia.org/wiki/SRE/Dc-operations/Platform-specific_documentation/ServerTech

Rack E3:

  • - receive in both C2[LS]42CE-YCMFAM00 and the mounting brackets via T286593, enter into Netbox.
  • - align the vertical rails in the cabinet to standard depths to allow full mounting of the PDUs and vertical cable managers.
  • - install C2[LS]42CE-YCMFAM00 into the rack - ensure the master and expansion PDUs are linked by the link cable included in the box. (silver ribbon cable, RJ type connector.)
  • - connect PDU to serial console server
  • - open ticket for PDUs to be plugged in / energized once racks E[1-8] are ready.
  • - confirm PDU energized for setup
  • - setup PDU according to directions on https://wikitech.wikimedia.org/wiki/SRE/Dc-operations/Platform-specific_documentation/ServerTech

Rack E4:

  • - receive in both C2[LS]42CE-YCMFAM00 and the mounting brackets via T286593, enter into Netbox.
  • - align the vertical rails in the cabinet to standard depths to allow full mounting of the PDUs and vertical cable managers.
  • - install C2[LS]42CE-YCMFAM00 into the rack - ensure the master and expansion PDUs are linked by the link cable included in the box. (silver ribbon cable, RJ type connector.)
  • - connect PDU to serial console server
  • - open ticket for PDUs to be plugged in / energized once racks E[1-8] are ready.
  • - confirm PDU energized for setup
  • - setup PDU according to directions on https://wikitech.wikimedia.org/wiki/SRE/Dc-operations/Platform-specific_documentation/ServerTech

Rack E5:

  • - receive in both C2[LS]42CE-YCMFAM00 and the mounting brackets via T286593, enter into Netbox.
  • - align the vertical rails in the cabinet to standard depths to allow full mounting of the PDUs and vertical cable managers.
  • - install C2[LS]42CE-YCMFAM00 into the rack - ensure the master and expansion PDUs are linked by the link cable included in the box. (silver ribbon cable, RJ type connector.)
  • - connect PDU to serial console server
  • - open ticket for PDUs to be plugged in / energized once racks E[1-8] are ready.
  • - confirm PDU energized for setup
  • - setup PDU according to directions on https://wikitech.wikimedia.org/wiki/SRE/Dc-operations/Platform-specific_documentation/ServerTech

Rack E6:

  • - receive in both C2[LS]42CE-YCMFAM00 and the mounting brackets via T286593, enter into Netbox.
  • - align the vertical rails in the cabinet to standard depths to allow full mounting of the PDUs and vertical cable managers.
  • - install C2[LS]42CE-YCMFAM00 into the rack - ensure the master and expansion PDUs are linked by the link cable included in the box. (silver ribbon cable, RJ type connector.)
  • - connect PDU to serial console server
  • - open ticket for PDUs to be plugged in / energized once racks E[1-8] are ready.
  • - confirm PDU energized for setup
  • - setup PDU according to directions on https://wikitech.wikimedia.org/wiki/SRE/Dc-operations/Platform-specific_documentation/ServerTech

Rack E7:

  • - receive in both C2[LS]42CE-YCMFAM00 and the mounting brackets via T286593, enter into Netbox.
  • - align the vertical rails in the cabinet to standard depths to allow full mounting of the PDUs and vertical cable managers.
  • - install C2[LS]42CE-YCMFAM00 into the rack - ensure the master and expansion PDUs are linked by the link cable included in the box. (silver ribbon cable, RJ type connector.)
  • - connect PDU to serial console server
  • - open ticket for PDUs to be plugged in / energized once racks E[1-8] are ready.
  • - confirm PDU energized for setup
  • - setup PDU according to directions on https://wikitech.wikimedia.org/wiki/SRE/Dc-operations/Platform-specific_documentation/ServerTech

Rack E8:

  • - receive in both C2[LS]42CE-YCMFAM00 and the mounting brackets via T286593, enter into Netbox.
  • - align the vertical rails in the cabinet to standard depths to allow full mounting of the PDUs and vertical cable managers.
  • - install C2[LS]42CE-YCMFAM00 into the rack - ensure the master and expansion PDUs are linked by the link cable included in the box. (silver ribbon cable, RJ type connector.)
  • - connect PDU to serial console server
  • - open ticket for PDUs to be plugged in / energized once racks E[1-8] are ready.
  • - confirm PDU energized for setup
  • - setup PDU according to directions on https://wikitech.wikimedia.org/wiki/SRE/Dc-operations/Platform-specific_documentation/ServerTech

Rack F1:

  • - receive in both C2[LS]42CE-YCMFAM00 and the mounting brackets via T286593, enter into Netbox.
  • - align the vertical rails in the cabinet to standard depths to allow full mounting of the PDUs and vertical cable managers.
  • - install C2[LS]42CE-YCMFAM00 into the rack - ensure the master and expansion PDUs are linked by the link cable included in the box. (silver ribbon cable, RJ type connector.)
  • - connect PDU to serial console server
  • - open ticket for PDUs to be plugged in / energized once racks E[9-16]/F[1-8] are ready.
  • - confirm PDU energized for setup
  • - setup PDU according to directions on https://wikitech.wikimedia.org/wiki/SRE/Dc-operations/Platform-specific_documentation/ServerTech

Rack F2:

  • - receive in both C2[LS]42CE-YCMFAM00 and the mounting brackets via T286593, enter into Netbox.
  • - align the vertical rails in the cabinet to standard depths to allow full mounting of the PDUs and vertical cable managers.
  • - install C2[LS]42CE-YCMFAM00 into the rack - ensure the master and expansion PDUs are linked by the link cable included in the box. (silver ribbon cable, RJ type connector.)
  • - connect PDU to serial console server
  • - open ticket for PDUs to be plugged in / energized once racks E[9-16]/F[1-8] are ready.
  • - confirm PDU energized for setup
  • - setup PDU according to directions on https://wikitech.wikimedia.org/wiki/SRE/Dc-operations/Platform-specific_documentation/ServerTech

Rack F3:

  • - receive in both C2[LS]42CE-YCMFAM00 and the mounting brackets via T286593, enter into Netbox.
  • - align the vertical rails in the cabinet to standard depths to allow full mounting of the PDUs and vertical cable managers.
  • - install C2[LS]42CE-YCMFAM00 into the rack - ensure the master and expansion PDUs are linked by the link cable included in the box. (silver ribbon cable, RJ type connector.)
  • - connect PDU to serial console server
  • - open ticket for PDUs to be plugged in / energized once racks E[9-16]/F[1-8] are ready.
  • - confirm PDU energized for setup
  • - setup PDU according to directions on https://wikitech.wikimedia.org/wiki/SRE/Dc-operations/Platform-specific_documentation/ServerTech

Rack F4:

  • - receive in both C2[LS]42CE-YCMFAM00 and the mounting brackets via T286593, enter into Netbox.
  • - align the vertical rails in the cabinet to standard depths to allow full mounting of the PDUs and vertical cable managers.
  • - install C2[LS]42CE-YCMFAM00 into the rack - ensure the master and expansion PDUs are linked by the link cable included in the box. (silver ribbon cable, RJ type connector.)
  • - connect PDU to serial console server
  • - open ticket for PDUs to be plugged in / energized once racks E[9-16]/F[1-8] are ready.
  • - confirm PDU energized for setup
  • - setup PDU according to directions on https://wikitech.wikimedia.org/wiki/SRE/Dc-operations/Platform-specific_documentation/ServerTech

Rack F5:

  • - receive in both C2[LS]42CE-YCMFAM00 and the mounting brackets via T286593, enter into Netbox.
  • - align the vertical rails in the cabinet to standard depths to allow full mounting of the PDUs and vertical cable managers.
  • - install C2[LS]42CE-YCMFAM00 into the rack - ensure the master and expansion PDUs are linked by the link cable included in the box. (silver ribbon cable, RJ type connector.)
  • - connect PDU to serial console server
  • - open ticket for PDUs to be plugged in / energized once racks E[9-16]/F[1-8] are ready.
  • - confirm PDU energized for setup
  • - setup PDU according to directions on https://wikitech.wikimedia.org/wiki/SRE/Dc-operations/Platform-specific_documentation/ServerTech

Rack F6

  • - receive in both C2[LS]42CE-YCMFAM00 and the mounting brackets via T286593, enter into Netbox.
  • - align the vertical rails in the cabinet to standard depths to allow full mounting of the PDUs and vertical cable managers.
  • - install C2[LS]42CE-YCMFAM00 into the rack - ensure the master and expansion PDUs are linked by the link cable included in the box. (silver ribbon cable, RJ type connector.)
  • - connect PDU to serial console server
  • - open ticket for PDUs to be plugged in / energized once racks E[9-16]/F[1-8] are ready.
  • - confirm PDU energized for setup
  • - setup PDU according to directions on https://wikitech.wikimedia.org/wiki/SRE/Dc-operations/Platform-specific_documentation/ServerTech

Rack F7:

  • - receive in both C2[LS]42CE-YCMFAM00 and the mounting brackets via T286593, enter into Netbox.
  • - align the vertical rails in the cabinet to standard depths to allow full mounting of the PDUs and vertical cable managers.
  • - install C2[LS]42CE-YCMFAM00 into the rack - ensure the master and expansion PDUs are linked by the link cable included in the box. (xsilver ribbon cable, RJ type connector.)
  • - connect PDU to serial console server
  • - open ticket for PDUs to be plugged in / energized once racks E[9-16]/F[1-8] are ready.
  • - confirm PDU energized for setup
  • - setup PDU according to directions on https://wikitech.wikimedia.org/wiki/SRE/Dc-operations/Platform-specific_documentation/ServerTech

Rack F8:

  • - receive in both C2[LS]42CE-YCMFAM00 and the mounting brackets via T286593, enter into Netbox.
  • - align the vertical rails in the cabinet to standard depths to allow full mounting of the PDUs and vertical cable managers.
  • - install C2[LS]42CE-YCMFAM00 into the rack - ensure the master and expansion PDUs are linked by the link cable included in the box. (silver ribbon cable, RJ type connector.)
  • - connect PDU to serial console server
  • - open ticket for PDUs to be plugged in / energized once racks E[9-16]/F[1-8] are ready.
  • - confirm PDU energized for setup
  • - setup PDU according to directions on https://wikitech.wikimedia.org/wiki/SRE/Dc-operations/Platform-specific_documentation/ServerTech

Details

SubjectRepoBranchLines /-
Add new PDU's in row E-F[5-8]operations/puppetproduction 48 -0
adding pdus for racks f[1-4]operations/puppetproduction 24 -0
Add new PDU in row E Eqiadoperations/puppetproduction 24 -1
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedRequestwiki_willyT296966 eqiad: Master Tracking Ticket for eqiad expansion cage
 ResolvedJclark-ctrT290899 Q1: eqiad: (32) PDUs for expansion
 Resolved• CmjohnsonT314027 ps1-e4-eqiad alerts

Event Timeline

RobH created this task.Sep 13 2021, 5:29 PM
RobH added a parent task: Unknown Object (Task).
RobH moved this task from Backlog to Racking Tasks on the ops-eqiad board.
Maintenance_bot added a project: SRE.Sep 13 2021, 5:45 PM
RobH updated the task description. (Show Details)Sep 13 2021, 6:27 PM
wiki_willy renamed this task from Q2: eqiad: (32) PDUs for expansion to Q1: eqiad: (32) PDUs for expansion.Sep 13 2021, 10:38 PM
wiki_willy assigned this task to Jclark-ctr.Nov 8 2021, 6:13 PM
wiki_willy added subscribers: • Cmjohnson, wiki_willy.
RobH unsubscribed.Nov 16 2021, 7:32 PM
wiki_willy mentioned this in T296966: eqiad: Master Tracking Ticket for eqiad expansion cage.Dec 2 2021, 9:46 PM
wiki_willy added a parent task: T296966: eqiad: Master Tracking Ticket for eqiad expansion cage.Dec 2 2021, 9:48 PM
RobH added a comment.Jan 21 2022, 6:13 PM

Netbox Updates:

  • added network port to all ps1-[ef]-eqiad
  • added power ports (54 or 42 depending on model) to all ps[12]-[ef]-eqiad
RobH added a comment.Jan 21 2022, 6:38 PM

Added in the IP addresses in the mgmt range that were available, and assigned them to the ps1-[ef]-eqiad with the following:

e1: 10.65.2.45/16
e2: 10.65.2.46/16
e3: 10.65.2.47/16
e4: 10.65.2.48/16
e5: 10.65.2.49/16
e6: 10.65.2.50/16
e7: 10.65.2.51/16
e8: 10.65.2.52/16

f1: 10.65.2.53/16
f2: 10.65.2.54/16
f3: 10.65.2.55/16
f4: 10.65.2.56/16
f5: 10.65.2.57/16
f6: 10.65.2.58/16
f7: 10.65.2.59/16
f8: 10.65.2.60/16

RobH added a comment.EditedJan 24 2022, 7:16 PM

Ok, basic network and our PDU login applied to the admn user (need to setup root via https) all PDUs via SCS connections, next is connection via HTTPS for full configuration of snmp services.

RobH updated the task description. (Show Details)Jan 24 2022, 8:42 PM
RobH added a comment.Jan 24 2022, 8:44 PM

PDUS in E1-E4 are fully setup with syslog, ntp, snmp settings. E5-E8 have their network setup done, but not their root user (admn only currently), snmp, ntp, or other settings done via https interface. Only [EF][1-4] have mgmt switches and the PDUs connected to mgmt network at this time.

Jclark-ctr updated the task description. (Show Details)Feb 10 2022, 8:21 PM
Papaul subscribed.Jul 5 2022, 10:11 PM

ps1-e[1-4] are now in Librenms

gerritbot added a comment.Jul 5 2022, 10:19 PM

Change 811382 had a related patch set uploaded (by Papaul; author: Papaul):

[operations/puppet@production] Add new PDU in row E Eqiad

https://gerrit.wikimedia.org/r/811382

gerritbot added a project: Patch-For-Review.Jul 5 2022, 10:19 PM
gerritbot added a comment.Jul 5 2022, 10:30 PM

Change 811382 merged by Papaul:

[operations/puppet@production] Add new PDU in row E Eqiad

https://gerrit.wikimedia.org/r/811382

Maintenance_bot removed a project: Patch-For-Review.Jul 5 2022, 10:30 PM
Papaul added a comment.Jul 5 2022, 10:41 PM

ps1-e[1-4] are now in Icinga

Papaul added a comment.Jul 5 2022, 10:58 PM

ps1-e[1-4] are now in Grafana
https://grafana.wikimedia.org/d/OBD1jy1Zk/filippo-pdu

wiki_willy added a comment.Jul 5 2022, 11:04 PM

Thanks @Papaul!

In T290899#8053980, @Papaul wrote:

ps1-e[1-4] are now in Grafana
https://grafana.wikimedia.org/d/OBD1jy1Zk/filippo-pdu

Papaul added a comment.Jul 6 2022, 1:52 PM

@wiki_willy you welcome

• Cmjohnson closed subtask T314027: ps1-e4-eqiad alerts as Resolved.Aug 16 2022, 3:57 PM
• ayounsi reopened subtask T314027: ps1-e4-eqiad alerts as Open.Aug 29 2022, 9:41 AM
• ayounsi mentioned this in T314027: ps1-e4-eqiad alerts.
Jclark-ctr closed subtask T314027: ps1-e4-eqiad alerts as Resolved.Aug 31 2022, 8:10 PM
gerritbot added a comment.Dec 22 2022, 8:14 PM

Change 870975 had a related patch set uploaded (by RobH; author: RobH):

[operations/puppet@production] adding pdus for racks f[1-4]

https://gerrit.wikimedia.org/r/870975

gerritbot added a project: Patch-For-Review.Dec 22 2022, 8:14 PM

Change 870975 merged by RobH:

[operations/puppet@production] adding pdus for racks f[1-4]

https://gerrit.wikimedia.org/r/870975

Maintenance_bot removed a project: Patch-For-Review.Dec 22 2022, 8:30 PM
Papaul mentioned this in T332523: Eqiad: Backlog HW failure-racking tasks - Decommision and remote work tasks .Mar 19 2023, 8:38 PM
Jclark-ctr added a comment.Mar 22 2023, 1:22 PM

pdu's have been connected to msw in rack and scs in f8. temp sensors are installed

Papaul added a comment.Mar 22 2023, 1:23 PM

@Jclark-ctr thanks i will start setting them up.

Jclark-ctr updated the task description. (Show Details)Mar 22 2023, 1:24 PM
gerritbot added a comment.Mar 23 2023, 11:52 PM

Change 902519 had a related patch set uploaded (by Papaul; author: Papaul):

[operations/puppet@production] Add new PDU's in row E-F[5-8]

https://gerrit.wikimedia.org/r/902519

gerritbot added a project: Patch-For-Review.Mar 23 2023, 11:52 PM
gerritbot added a comment.Mar 23 2023, 11:56 PM

Change 902519 merged by Papaul:

[operations/puppet@production] Add new PDU's in row E-F[5-8]

https://gerrit.wikimedia.org/r/902519

Papaul updated the task description. (Show Details)Mar 24 2023, 12:03 AM
Papaul added a comment.Mar 24 2023, 12:08 AM

I setup the last 8 PDU's row E [4-8] and row F [4-8] but was not able to setup ps1-e8 ad ps1-f8 because of connectivity issue. I added those to librenms and icinga. so what left here to resolve this task is to fix the connectivity issue wih ps1-e8 and ps1-f8.
@Jclark-ctr can you please check those two PDU's next that you are on site?
Thanks

Maintenance_bot removed a project: Patch-For-Review.Mar 24 2023, 12:10 AM
Papaul closed this task as Resolved.Mar 24 2023, 10:59 PM
Papaul updated the task description. (Show Details)

This is complete

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits