Page MenuHomePhabricator
Create Task
Maniphest T253808

db1138 (s4 master) crashed due to memory issues
Closed, ResolvedPublic
Actions

Description

db1138 crashed mysql due to memory HW issues:

[33134188.608450] mce: [Hardware Error]: Machine check events logged
[33134188.608477] mce: Uncorrected hardware memory error in user-access at 7d3c38f580
[33134188.615864] {1}Hardware error detected on CPU2
[33134188.615874] {1}event severity: recoverable
[33134188.615875] {1} Error 0, type: recoverable
[33134188.615876] {1} fru_text: B4
[33134188.615876] {1}  section_type: memory error
[33134188.615877] {1}  error_status: 0x0000000000000400
[33134188.615878] {1}  physical_address: 0x0000007d3c38f580
[33134188.615880] {1}  node: 3 card: 0 module: 0 rank: 0 bank: 1 row: 55982 column: 1016
[33134188.615882] {1}  DIMM location: not present. DMI handle: 0x0000
[33134188.617181] Memory failure: 0x7d3c38f: Killing mysqld:163407 due to hardware memory corruption
[33134188.626049] Memory failure: 0x7d3c38f: recovery action for dirty LRU page: Recovered
[33134263.297543] MCE: Killing mysqld:163468 due to hardware memory corruption fault at 7feced3dc580
05/27/2020 20:20:26 Critical:  "Multi-bit memory errors detected on a memory device at location(s) DIMM_B4." in SEL on db1138

What I have done for now is:

  • Decreased buffer pool size to 300GB and restarted mysql.

Let's do a master failover on Friday to the candidate master.

@wiki_willy can we get a new DIMM for this host?

Details

SubjectRepoBranchLines /-
db1138: Enable notificationsoperations/puppetproduction 0 -1
db1138: Disable notificationsoperations/puppetproduction 1 -0
wmnet: Update s4-master aliasoperations/dnsmaster 1 -1
mariadb: Promote db1081 to s4 masteroperations/puppetproduction 4 -4
Customize query in gerrit

Related Objects
Search...

Event Timeline

Marostegui created this task.May 27 2020, 8:34 PM
Restricted Application added a subscriber: Liuxinyu970226. · View Herald TranscriptMay 27 2020, 8:34 PM
CDanis triaged this task as High priority.May 27 2020, 8:36 PM
CDanis subscribed.
Marostegui added a comment.May 27 2020, 8:39 PM

I am running a compare from this host to its candidate master (db1081) to make sure we are good for Friday.

Marostegui added a project: Wikimedia-Incident.May 27 2020, 8:41 PM
Marostegui moved this task from Triage to In progress on the DBA board.
• MoritzMuehlenhoff mentioned this in T253810: Alert on ECC warnings in SEL.May 27 2020, 8:53 PM
wiki_willy assigned this task to Jclark-ctr.May 27 2020, 9:12 PM
wiki_willy added a comment.May 27 2020, 9:15 PM

@Marostegui - will do, Papaul and John are working on pulling the TSR right now for the RMA. Thanks, Willy

Jclark-ctr added a comment.May 27 2020, 11:31 PM

Sent TSR report to Dell
Confirmed: Service Request 1025886499 was successfully submitted.

gerritbot added a comment.May 28 2020, 4:56 AM

Change 599155 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Promote db1081 to s4 master

https://gerrit.wikimedia.org/r/599155

gerritbot added a project: Patch-For-Review.May 28 2020, 4:56 AM
gerritbot added a comment.May 28 2020, 4:58 AM

Change 599156 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/dns@master] wmnet: Update s4-master alias

https://gerrit.wikimedia.org/r/599156

Marostegui added a comment.May 28 2020, 5:41 AM

Data check between db1138 and db1081 (candidate master) finished successfully.

Marostegui added a comment.May 28 2020, 6:14 AM

Blocked a maintenance window on the deployment's calendar for tomorrow.

Stashbot added a comment.May 28 2020, 6:30 AM

Mentioned in SAL (#wikimedia-operations) [2020-05-28T06:30:38Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Remove db1081 from API and set its weight to 0 on main traffic - preparation for tomorrow's failover T253808', diff saved to https://phabricator.wikimedia.org/P11329 and previous config saved to /var/cache/conftool/dbconfig/20200528-063037-marostegui.json

Marostegui added a comment.May 28 2020, 7:39 AM

Incident report (draft status) created: https://wikitech.wikimedia.org/wiki/Incident_documentation/20200528-s4_(commonswiki)_on_read-only_for_8_minutes

Marostegui closed subtask T253832: Create a wiki page with information on what to do when a master crashes and comes back as read only as Resolved.May 28 2020, 9:59 AM
Jclark-ctr added a comment.May 28 2020, 2:21 PM

fedex tracking says parts to arrive friday 5/29 @Marostegui would you want to do this tomorrow 3-4pm est. I would prefer not to take a host down on a friday afternoon or can we like to wait till next monday?

Trizek-WMF subscribed.May 28 2020, 2:50 PM
Johan subscribed.May 28 2020, 3:19 PM

@Jclark-ctr @Marostegui Does this mean you're not doing the 05:00 UTC window tomorrow? We've been informing the communities about this, and set banners, so we'd need to know ASAP if this is the case.

• jcrespo subscribed.May 28 2020, 3:21 PM

@Johan plan continues as usual- @Jclark-ctr information is unrelated to the user impacting maintenance.

Johan added a comment.May 28 2020, 3:22 PM

OK, great, thanks.

• jcrespo added a comment.May 28 2020, 3:22 PM

I talked to @Jclark-ctr on IRC, hw replacement will likely happen on Tuesday next week.

Sw emergency maintenance (read only) that is needed *before* hw maintenance is still tomorrow Friday 5am UTC.

Johan unsubscribed.May 28 2020, 3:43 PM
Agusbou2015 subscribed.May 28 2020, 10:24 PM
Stashbot added a comment.May 29 2020, 4:25 AM

Mentioned in SAL (#wikimedia-operations) [2020-05-29T04:25:19Z] <marostegui> Start topology changes in s4 - T253808

gerritbot added a comment.May 29 2020, 4:42 AM

Change 599155 merged by Marostegui:
[operations/puppet@production] mariadb: Promote db1081 to s4 master

https://gerrit.wikimedia.org/r/599155

Stashbot added a comment.May 29 2020, 5:00 AM

Mentioned in SAL (#wikimedia-operations) [2020-05-29T05:00:38Z] <marostegui> Starting s4 failover from db1138 to db1081 -T253808

Stashbot added a comment.May 29 2020, 5:01 AM

Mentioned in SAL (#wikimedia-operations) [2020-05-29T05:01:53Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Set s4 as read-only for maintenance T253808', diff saved to https://phabricator.wikimedia.org/P11333 and previous config saved to /var/cache/conftool/dbconfig/20200529-050153-marostegui.json

Stashbot added a comment.May 29 2020, 5:02 AM

Mentioned in SAL (#wikimedia-operations) [2020-05-29T05:02:25Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Promote db1081 to s4 master and remove read-only from s4 T253808', diff saved to https://phabricator.wikimedia.org/P11334 and previous config saved to /var/cache/conftool/dbconfig/20200529-050224-marostegui.json

gerritbot added a comment.May 29 2020, 5:06 AM

Change 599156 merged by Marostegui:
[operations/dns@master] wmnet: Update s4-master alias

https://gerrit.wikimedia.org/r/599156

Marostegui closed subtask T253825: Emergency read-only time for s4 (commonswiki) as Resolved.May 29 2020, 5:07 AM

The master failover was done successfully.
This was done successfully
RO started at 05:01:54
RO stopped at 05:02:25

Total RO time: 31 seconds

db1138 is no longer s4 master and we can do the on-site maintenance on Tuesday if @Jclark-ctr is available, can you confirm?
I will leave db1138 depooled for the weekend.

Maintenance_bot removed a project: Patch-For-Review.May 29 2020, 5:10 AM
gerritbot added a comment.May 29 2020, 5:15 AM

Change 599583 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1138: Disable notifications

https://gerrit.wikimedia.org/r/599583

gerritbot added a project: Patch-For-Review.May 29 2020, 5:15 AM
gerritbot added a comment.May 29 2020, 5:18 AM

Change 599583 merged by Marostegui:
[operations/puppet@production] db1138: Disable notifications

https://gerrit.wikimedia.org/r/599583

Maintenance_bot removed a project: Patch-For-Review.May 29 2020, 6:10 AM
• Cmjohnson moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.May 29 2020, 12:25 PM
Marostegui added a comment.Jun 1 2020, 12:58 PM

@Jclark-ctr could you confirm if you want to do this maintenance today Monday 1st June or tomorrow Tuesday 2nd June?

Marostegui added a comment.Jun 1 2020, 1:10 PM

John confirmed via IRC that the maintenance will be done on Tuesday - thank you!

Stashbot added a comment.Jun 2 2020, 7:06 AM

Mentioned in SAL (#wikimedia-operations) [2020-06-02T07:06:35Z] <marostegui> Stop MySQL and poweroff on db1138 for on-site maintenance - T253808

Marostegui added a comment.Jun 2 2020, 7:09 AM

@Jclark-ctr db1138 is now off and ready for you to change the memory whenever you get to the DC.
Once you are done, please power the host back on and I will take it from there.

Thank you!

RhinosF1 subscribed.Jun 2 2020, 7:09 AM
Jclark-ctr added a comment.Jun 2 2020, 1:48 PM

@Marostegui Replaced failed DIMM. host is powered back on

Marostegui added a comment.Jun 2 2020, 1:49 PM

Thank you, I will take it from here

gerritbot added a comment.Jun 3 2020, 5:01 AM

Change 601948 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1138: Enable notifications

https://gerrit.wikimedia.org/r/601948

gerritbot added a project: Patch-For-Review.Jun 3 2020, 5:01 AM
gerritbot added a comment.Jun 3 2020, 5:07 AM

Change 601948 merged by Marostegui:
[operations/puppet@production] db1138: Enable notifications

https://gerrit.wikimedia.org/r/601948

Stashbot added a comment.Jun 3 2020, 5:09 AM

Mentioned in SAL (#wikimedia-operations) [2020-06-03T05:09:12Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1138 T253808', diff saved to https://phabricator.wikimedia.org/P11369 and previous config saved to /var/cache/conftool/dbconfig/20200603-050911-marostegui.json

Maintenance_bot removed a project: Patch-For-Review.Jun 3 2020, 5:10 AM
Stashbot added a comment.Jun 3 2020, 5:37 AM

Mentioned in SAL (#wikimedia-operations) [2020-06-03T05:37:48Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1138 T253808', diff saved to https://phabricator.wikimedia.org/P11370 and previous config saved to /var/cache/conftool/dbconfig/20200603-053748-marostegui.json

Stashbot added a comment.Jun 3 2020, 6:01 AM

Mentioned in SAL (#wikimedia-operations) [2020-06-03T06:01:25Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1138 T253808', diff saved to https://phabricator.wikimedia.org/P11373 and previous config saved to /var/cache/conftool/dbconfig/20200603-060124-marostegui.json

Stashbot added a comment.Jun 3 2020, 6:37 AM

Mentioned in SAL (#wikimedia-operations) [2020-06-03T06:37:52Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1138 T253808', diff saved to https://phabricator.wikimedia.org/P11374 and previous config saved to /var/cache/conftool/dbconfig/20200603-063752-marostegui.json

Marostegui closed this task as Resolved.Jun 3 2020, 7:26 AM

Host repooled. All done.
Thanks John for replacing the memory!

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits