Internal database error while saving translations
Closed, ResolvedPublic
Actions

Description

From the event logging I see ~150 instances of 'internal_api_error_DBQueryError'

To debug, some samples are given below

en->ta
tawiki
timestamp: 20160410071417
Source title: Laminar armour
{"servedby":"mw1115","error":{"code":"internal_api_error_DBQueryError","info":"[1d159c53da9eb0eeb620cb98] Database query error"},"errorCode":"internal_api_error_DBQueryError"}

en->ml
mlwiki
timestamp: 20160502174123
Source: Teacup
{"servedby":"mw1143","error":{"code":"internal_api_error_DBQueryError","info":"[VyeRQgpAEHsAAD5Z@8gAAAAV] Database query error"},"errorCode":"internal_api_error_DBQueryError"}

Details

Subject	Repo	Branch	Lines /-
Avoid deadlock patterns in cx_corpora updates	mediawiki/extensions/ContentTranslation	wmf/1.28.0-wmf.14	62 -27
Avoid deadlock patterns in cx_corpora updates	mediawiki/extensions/ContentTranslation	wmf/1.28.0-wmf.15	62 -27
Avoid deadlock patterns in cx_corpora updates	mediawiki/extensions/ContentTranslation	master	62 -27

Customize query in gerrit

Related Objects

Mentioned Here: T129462: Internal API error for translation save API

Event Timeline

santhosh created this task.May 3 2016, 9:54 AM

Restricted Application added subscribers: Zppix, Aklapper. · View Herald TranscriptMay 3 2016, 9:54 AM

The errors did not prevent translators to complete translation. Both of the above translations were successfully published

I can see two issues with investigating this bug further:

As far as I can see, the trace doesn't say anything substantial except "internal_api_error_DBQueryError". Is the error itself or the line that triggered it logged elsewhere?
What's the right way to measure the frequency of these errors? I currently do this by running https://gerrit.wikimedia.org/r/#/c/282228/7/bash/sorted_save_events.py , which groups by event_session. Should I group by something else?

As far as I can see, the trace doesn't say anything substantial except "internal_api_error_DBQueryError". Is the error itself or the line that triggered it logged elsewhere?

Logstash. Also see T129462: Internal API error for translation save API

What's the right way to measure the frequency of these errors? I currently do this by running https://gerrit.wikimedia.org/r/#/c/282228/7/bash/sorted_save_events.py , which groups by event_session. Should I group by something else?

group by unique translations - a unique translation is identified as combination of source language, target language and source title

Amire80 triaged this task as Medium priority.May 19 2016, 5:27 PM

Amire80 added a project: ContentTranslation-Release9.

Restricted Application added a project: Language-Engineering April-June 2016. · View Herald TranscriptMay 19 2016, 5:27 PM

Amire80 added a project: Essential-Work.May 19 2016, 5:27 PM

Amire80 moved this task from Needs Triage to CX9 on the ContentTranslation board.

Arrbee moved this task from Backlog to CX9 on the Language-Engineering April-June 2016 board.May 25 2016, 8:06 AM

Arrbee raised the priority of this task from Medium to High.Jun 8 2016, 8:09 AM

Arrbee added a project: Language-Q4-2016-Sprint 5.

Arrbee edited projects, added Language-Q1-2016-17 Sprint 1, Language-Engineering July-September 2016; removed Language-Q4-2016-Sprint 5, ContentTranslation-Release9, Language-Engineering April-June 2016.Jul 5 2016, 7:01 AM

Arrbee moved this task from Backlog to In Progress on the Language-Q1-2016-17 Sprint 1 board.Jul 7 2016, 8:08 AM

I located an error in logstash and details are as follows:

ContentTranslation\TranslationStorageManager::{closure} 10.64.16.20 1213 Deadlock found when trying to get lock; try restarting transaction (10.64.16.20) SELECT * FROM cx_corpora WHERE cxc_translation_id = '178280' AND cxc_section_id = 'mwIg' AND cxc_origin = 'user' ORDER BY cxc_timestamp DESC LIMIT 1 FOR UPDATE

backtrace:

{"file":"/srv/mediawiki/php-1.28.0-wmf.8/includes/db/Database.php","line":901,"function":"reportQueryError","class":"DatabaseBase","type":"->","args":["string","integer","string","string","boolean"]}, {"file":"/srv/mediawiki/php-1.28.0-wmf.8/includes/db/Database.php","line":1234,"function":"query","class":"DatabaseBase","type":"->","args":["string","string"]}, {"file":"/srv/mediawiki/php-1.28.0-wmf.8/includes/db/Database.php","line":1293,"function":"select","class":"DatabaseBase","type":"->","args":["string","string","array","string","array","array"]}, {"file":"/srv/mediawiki/php-1.28.0-wmf.8/extensions/ContentTranslation/includes/TranslationStorageManager.php","line":127,"function":"selectRow","class":"DatabaseBase","type":"->","args":["string","string","array","string","array"]}, {"file":"/srv/mediawiki/php-1.28.0-wmf.8/extensions/ContentTranslation/includes/TranslationStorageManager.php","line":94,"function":"doFind","class":"ContentTranslation\\TranslationStorageManager","type":"::","args":["DatabaseMysqli","array","array","string"]}, {"file":"/srv/mediawiki/php-1.28.0-wmf.8/includes/db/Database.php","line":2567,"function":"Closure$ContentTranslation\\TranslationStorageManager::save","args":["DatabaseMysqli","string"]}, {"file":"/srv/mediawiki/php-1.28.0-wmf.8/includes/db/DBConnRef.php","line":39,"function":"doAtomicSection","class":"DatabaseBase","type":"->","args":["string","Closure$ContentTranslation\\TranslationStorageManager::save;363061839"]}, {"file":"/srv/mediawiki/php-1.28.0-wmf.8/includes/db/DBConnRef.php","line":437,"function":"__call","class":"DBConnRef","type":"->","args":["string","array"]}, {"file":"/srv/mediawiki/php-1.28.0-wmf.8/extensions/ContentTranslation/includes/TranslationStorageManager.php","line":101,"function":"doAtomicSection","class":"DBConnRef","type":"->","args":["string","Closure$ContentTranslation\\TranslationStorageManager::save;363061839"]}, {"file":"/srv/mediawiki/php-1.28.0-wmf.8/extensions/ContentTranslation/api/ApiContentTranslationSave.php","line":216,"function":"save","class":"ContentTranslation\\TranslationStorageManager","type":"::","args":["ContentTranslation\\TranslationUnit"]}, {"file":"/srv/mediawiki/php-1.28.0-wmf.8/extensions/ContentTranslation/api/ApiContentTranslationSave.php","line":59,"function":"saveTranslationUnits","class":"ApiContentTranslationSave","type":"->","args":["array"]}, {"file":"/srv/mediawiki/php-1.28.0-wmf.8/includes/api/ApiMain.php","line":1373,"function":"execute","class":"ApiContentTranslationSave","type":"->","args":[]}, {"file":"/srv/mediawiki/php-1.28.0-wmf.8/includes/api/ApiMain.php","line":469,"function":"executeAction","class":"ApiMain","type":"->","args":[]}, {"file":"/srv/mediawiki/php-1.28.0-wmf.8/includes/api/ApiMain.php","line":440,"function":"executeActionWithErrorHandling","class":"ApiMain","type":"->","args":[]}, {"file":"/srv/mediawiki/php-1.28.0-wmf.8/api.php","line":83,"function":"execute","class":"ApiMain","type":"->","args":[]}, {"file":"/srv/mediawiki/w/api.php","line":3,"function":"include","args":["string"]}

Arrbee edited projects, added Language-Q1-2016-17 Sprint 2; removed Language-Q1-2016-17 Sprint 1.Jul 19 2016, 7:01 AM

Arrbee moved this task from Backlog to In Progress on the Language-Q1-2016-17 Sprint 2 board.Jul 19 2016, 7:12 AM

Amire80 moved this task from CX9 to CX10 (July-Sep. 2016) on the ContentTranslation board.Jul 20 2016, 6:15 AM

Amire80 moved this task from Backlog to CX10 on the Language-Engineering July-September 2016 board.Jul 31 2016, 11:29 AM

Arrbee edited projects, added Language-Q1-2016-17 Sprint 3; removed Language-Q1-2016-17 Sprint 2.Aug 2 2016, 7:01 AM

Arrbee moved this task from Backlog to In Progress on the Language-Q1-2016-17 Sprint 3 board.Aug 3 2016, 5:42 AM

Arrbee edited projects, added Language-Q1-2016-17 Sprint 4; removed Language-Q1-2016-17 Sprint 3.Aug 16 2016, 7:18 AM

aaron merged a task: T137275: Corpora table query errors (deadlock found when trying to get a lock).Aug 16 2016, 9:45 AM

aaron added a subscriber: Nikerabbit.

Arrbee moved this task from Backlog to In Progress on the Language-Q1-2016-17 Sprint 4 board.Aug 16 2016, 10:30 AM

Change 305004 had a related patch set uploaded (by Aaron Schulz):
Avoid deadlock patterns in cx_corpora updates

https://gerrit.wikimedia.org/r/305004

gerritbot added a project: Patch-For-Review.Aug 16 2016, 10:58 AM

Change 305004 merged by jenkins-bot:
Avoid deadlock patterns in cx_corpora updates

https://gerrit.wikimedia.org/r/305004

Change 305188 had a related patch set uploaded (by KartikMistry):
Avoid deadlock patterns in cx_corpora updates

https://gerrit.wikimedia.org/r/305188

Change 305190 had a related patch set uploaded (by KartikMistry):
Avoid deadlock patterns in cx_corpora updates

https://gerrit.wikimedia.org/r/305190

Change 305190 merged by jenkins-bot:
Avoid deadlock patterns in cx_corpora updates

https://gerrit.wikimedia.org/r/305190

Change 305188 merged by jenkins-bot:
Avoid deadlock patterns in cx_corpora updates

https://gerrit.wikimedia.org/r/305188

Mentioned in SAL [2016-08-17T15:59:39Z] <thcipriani@tin> Synchronized php-1.28.0-wmf.15/extensions/ContentTranslation: SWAT: [[gerrit:305190|Avoid deadlock patterns in cx_corpora updates (T134245)]] (duration: 00m 52s)

Mentioned in SAL [2016-08-17T16:02:46Z] <thcipriani@tin> Synchronized php-1.28.0-wmf.14/extensions/ContentTranslation: SWAT: [[gerrit:305188|Avoid deadlock patterns in cx_corpora updates (T134245)]] (duration: 00m 50s)