Page MenuHomePhabricator
Create Task
Maniphest T119154

Move echo tables from local wiki databases onto extension1 cluster for mediawikiwiki, metawiki, and officewiki
Closed, DeclinedPublic
Actions

Description

InitialiseSettings.php
// Use extension1 db for all wikis
'wmgEchoCluster' => array(
	'default' => 'extension1',
	// The following wikis should be using extension1 db but started with
	// the application main database, stick with the current setting
	'labswiki' => false,
	'mediawikiwiki' => false,
	'metawiki' => false,
	'officewiki' => false,
),
`

labswiki aka wikitech is special and probably needs to stay separate. But mediawikiwiki, metawiki, and officewiki should be moved to the extension1 cluster.

The following tables need to be moved:

  • echo_email_batch
  • echo_event
  • echo_notification
  • echo_target_page

Marking as lowest priority as this is probably difficult and has little gains except for standardization and developer sanity. Just filing this so we don't forget about it.

Details

SubjectRepoBranchLines /-
labtestwikitech: use local database for Echooperations/mediawiki-configmaster 1 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 DeclinedNoneT119154 Move echo tables from local wiki databases onto extension1 cluster for mediawikiwiki, metawiki, and officewiki
 ResolvedMarosteguiT153638 Drop echo tables from local wiki databases
 ResolvedLadsgroupT104459 Detect object, schema and data drifts between mediawiki HEAD, production masters and replicas
 ResolvedNoneT57455 Provide dynamic report of differences between replica databases and production databases
 ResolvedMarosteguiT51199 Add index log_type_action
 ResolvedMarosteguiT217397 logging.log_title_time and logging.log_title_type_time indexes are not on tables.sql but they exist on most of the wikis
 ResolvedMarosteguiT250055 Remove image.img_deleted column from production
 ResolvedMarosteguiT250067 user_newtalk has two indexes not renamed in s4
 ResolvedMarosteguiT250062 ipb_parent_block_id_2 index on ipblocks table on s8 only
 ResolvedMarosteguiT250060 tl_namespace index on templatelinks is unique only in s8
 ResolvedReedyT249972 Wikitech has lots of database drifts with core and rest of the databases
 ResolvedMarosteguiT249973 db1110 has 5 important database drifts that are unique to the host
 ResolvedMarosteguiT250056 lc_lang_key index is lingering in production
 ResolvedMarosteguiT250057 type_acton index in logging table is lingering in production
 DeclinedNoneT250058 searchindex indexes are missing in production
 ResolvedMarosteguiT250063 inverse_timestamp column exists in text table, it shouldn't
 ResolvedMarosteguiT250066 text table still has old_* fields and indexes on some hosts
 ResolvedLadsgroupT250071 Rename ipb_address index on ipb_address to ipb_address_unique
 ResolvedMarosteguiT252126 tl_from index on templatelinks is lingering in production
 ResolvedMarosteguiT256679 text table in db1144 drifts from core considerably
 ResolvedMarosteguiT256680 imagelinks has index mismatch on s8
 ResolvedMarosteguiT256682 page_restrictions indexes have been majestically drifting from code
 ResolvedMarosteguiT256684 pl_from index still lingers in random hosts
 ResolvedMarosteguiT256685 pl_namespace index on pagelinks is unique only in s8
 ResolvedMarosteguiT260111 All sorts of random drifts in wikis in s3
 ResolvedLadsgroupT265321 ipblocks_restrictions.ir_type is tinyint(1) in code but tinyint(4) in production
 ResolvedMarosteguiT265349 querycache qc_type and qc_title have different nullabality on s1 only
 ResolvedMarosteguiT265445 Index uniqueness mismatches in links tables in wikis that were moved from s3 to s5
 ResolvedLadsgroupT269348 wikitech database has almost all of its varbinary fields wrong
 Resolved• KormatT277118 iw_url in interwiki is varbinary(127) in production but blob in code
 ResolvedMarosteguiT277354 "chemical" major mime type was never added to production database
 Resolved• KormatT278619 ipb_timestamp is varbinary(14) in old wikis while being binary(14) in the code since 2007
 ResolvedMarosteguiT278621 job_cmd is varbinary(255) in production while being varbinary(60) in code since 2007
 Resolved• KormatT277116 fa_deleted_timestamp and fa_timestamp are binary(14) in code but varbinary(14) in production
 ResolvedMarosteguiT308206 Slow query on echo_event table

Event Timeline

Legoktm created this task.Nov 20 2015, 12:19 AM
Legoktm raised the priority of this task from to Lowest.
Legoktm updated the task description. (Show Details)
Legoktm added projects: SRE, DBA, Collaboration-Team-Triage.
Legoktm subscribed.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 20 2015, 12:19 AM
Krenair subscribed.Nov 20 2015, 12:22 AM
• jcrespo subscribed.Nov 20 2015, 12:26 AM

I was one of the people that wouldn't understand why this was like this in the first place, but: Are we sure this is the way to go, and that the original reasons, whatever they were, are not relevant anymore?

For example, labswiki is believed to be wanted to be isolated for availability reasons. Is there consensus on this?

Once that is clear, I have no technical reason to not put this on the backlog.

Legoktm added a comment.Nov 20 2015, 12:44 AM
In T119154#1819505, @jcrespo wrote:

I was one of the people that wouldn't understand why this was like this in the first place, but: Are we sure this is the way to go, and that the original reasons, whatever they were, are not relevant anymore?

The original "reasons" is that Echo was deployed to those wikis before it was realized we needed/should use the extension1 cluster (I can dig up the history if you want, but I wasn't involved at that time). Originally just large and medium wikis were on extension1, but then we realized T58222: Wikis get bigger (surprise, surprise!), so *all* wikis were moved to extension1, except for the four remaining exceptions. I don't know what the rationale for not moving those at the time were. I'll do some more research to figure out why...

For example, labswiki is believed to be wanted to be isolated for availability reasons. Is there consensus on this?

The labswiki database should stay where it is, and *not* be moved to extension1. I intentionally did not include it in the bug title.

• jcrespo claimed this task.Nov 20 2015, 12:48 AM

My mistake, I assumed labswiki was part of the migration just because it was part of the configuration.

• jcrespo moved this task from Triage to Backlog on the DBA board.Nov 20 2015, 12:49 AM
• jcrespo mentioned this in T125591: Set up additional filters for Echo tables.Feb 3 2016, 9:45 AM
• jcrespo mentioned this in T125863: Evaluate which Echo tables can be publicly replicated.
Legoktm mentioned this in T54921: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking).Dec 18 2016, 6:26 AM
• MZMcBride subscribed.Dec 19 2016, 4:42 AM
Marostegui added a subtask: T153638: Drop echo tables from local wiki databases.Dec 19 2016, 10:22 AM
• jcrespo mentioned this in T136428: Add primary key to echo_notification table.Feb 9 2017, 9:14 AM
• jcrespo removed • jcrespo as the assignee of this task.Feb 14 2017, 4:03 PM
Marostegui closed subtask T153638: Drop echo tables from local wiki databases as Resolved.Sep 19 2018, 6:13 AM
• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 8:25 PM
Marostegui moved this task from Backlog to Pending comment on the DBA board.Aug 12 2019, 2:07 PM
Aklapper edited projects, added Growth-Team; removed Collaboration-Team-Triage.Oct 12 2019, 4:08 AM
gerritbot added a comment.Oct 31 2019, 3:13 PM

Change 547553 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/mediawiki-config@master] labtestwikitech: use local database for Echo

https://gerrit.wikimedia.org/r/547553

gerritbot added a project: Patch-For-Review.Oct 31 2019, 3:13 PM
Andrew subscribed.Oct 31 2019, 3:13 PM

Attached patch is regarding labtestwikitech -- pretty much any changes made for wikitech need to be made there too.

gerritbot added a comment.Nov 7 2019, 2:29 PM

Change 547553 abandoned by Andrew Bogott:
labtestwikitech: use local database for Echo

Reason:
dropping in favor of the much-more-expansive https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/ /547596/

https://gerrit.wikimedia.org/r/547553

Maintenance_bot removed a project: Patch-For-Review.Nov 7 2019, 3:10 PM
• jcrespo mentioned this in T255174: Extend echo_unread_wikis.euw_wiki.Jul 6 2020, 1:42 PM
LSobanski closed this task as Declined.Oct 13 2020, 11:03 AM
LSobanski subscribed.

We consider this task to be very risky and with the limited gain suggest against going forward with it. Closing, please reach out to me directly if you would like to discuss this.

Marostegui mentioned this in T308206: Slow query on echo_event table.May 12 2022, 7:26 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits