[Epic] Scaling strategy for Wikidata Query Service
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Gehel
	Apr 26 2019, 9:50 AM

Description

As discussed on wiki, we currently don't have a scaling strategy for WDQS, except getting bigger and larger servers. This needs to be addressed.

Scaling a general purpose graph database is a hard problem and won't be solved easily.

Constraints:

spread read and write IO load over multiple servers
spread storage over multiple servers
keep throughput mostly linear with the number of servers
keep the same interface for existing consumers (SPARQL), or as similar as possible to the current interface
(probably a lot more)

Status	Assigned	Task
Resolved	dcausse	T221938 [Epic] Scaling strategy for Wikidata Query Service
Resolved	Gehel	T257045 [Epic] Better understanding of WDQS users and use cases
Declined	None	T258269 Add query result to the current WDQS event logging
Resolved	• Zbyszko	T261841 Tag WDQS query log with the source of the query (UI vs direct access)
Resolved	• Zbyszko	T261937 Add CPU load and query concurrency as context to event logging from WDQS
Resolved	Lydia_Pintscher	T264194 Examine WDQS queries for intent
Resolved	• Zbyszko	T264447 Examine cases where Blazegraph generates results that timeout and don’t make it back to the user
Resolved	JAllemandou	T266022 Programmatically categorize WDQS queries by potential alternative solution
Resolved	None	T280075 Create WDQS UI user survey to collect data on user priorities
Resolved	• MPhamWMF	T280077 Announce WDQS UI user survey
Resolved	None	T257055 WDQS - Planning for disaster (significant failure of the project)
Open	None	T260687 Explore the scaling limits of Blazegraph by generating synthetic data

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Dzahn triaged this task as Medium priority.Apr 30 2019, 9:40 PM

Work is continuing on more specific tickets.