onMwLoadEnd test failing regularly blocking mergings across repositories
Closed, ResolvedPublicBUG REPORT
Actions

Description

This test appears to be flakey. It's become a common occurrence in unrelated changes for example https://gerrit.wikimedia.org/r/c/mediawiki/core/ /753757/ and https://gerrit.wikimedia.org/r/c/mediawiki/skins/Vector/ /753990.

✖ 1 test failed
04:29:04 
04:29:04 FAILED TESTS:
04:29:04   ext.navigationTiming
04:29:04     ✖ onMwLoadEnd - controlled
04:29:04       Chrome Headless 90.0.4430.212 (Linux x86_64)
04:29:04     resolved after fail ok
04:29:04     Expected: [
04:29:04       "call"
04:29:04     ]
04:29:04     Actual: []
04:29:04         at Object.<anonymous> (http://localhost:9876/load.php?modules=mediawiki.interface.helpers.styles|mediawiki.jqueryMsg,language,messagePoster,pulsatingdot,qunit-testrunner,router,storage,template,toc,ui,user,util,viewport,visibleTimeout,widgets|mediawiki.language.months,names,specialCharacters,testdata|mediawiki.libs.guiders,jpegmeta,pluralruleparser|mediawiki.page.gallery.styles|mediawiki.page.ready|mediawiki.page.watch.ajax|mediawiki.rcfilters.filters.ui|mediawiki.skinning.content.parsoid|mediawiki.special.recentchanges|mediawiki.template.handlebars,mustache|mediawiki.ui.button,icon|mediawiki.widgets.AbandonEditDialog,CategoryMultiselectWidget,DateInputWidget,MediaSearch,Table,UserInputWidget,styles|mediawiki.widgets.DateInputWidget.styles|mobile.ooui.icons|mobile.pagelist.styles|mobile.pagesummary.styles|mobile.placeholder.images|mobile.startup|mobile.startup.images|moment,oojs,oojs-router,oojs-ui,oojs-ui-core,oojs-ui-toolbars,oojs-ui-widgets,oojs-ui-windows,papaparse,rangefix,sinonjs,spark-md5|mw.config.values.wbDataTypes,wbRefTabsEnabled,wbRepo,wbSiteDetails|mw.cx.SiteMapper,dm,init,ui,util|mw.cx.dm.Translation|mw.cx.tools.TranslationTool|mw.cx.ui.Infobar|oojs-ui-core.icons,styles|oojs-ui-toolbars.icons|oojs-ui-widgets.icons|oojs-ui-windows.icons|oojs-ui.styles.icons-accessibility,icons-alerts,icons-content,icons-editing-advanced,icons-editing-citation,icons-editing-core,icons-editing-list,icons-editing-styling,icons-interactions,icons-layout,icons-media,icons-moderation,icons-movement,icons-user,icons-wikimedia,indicators|skins.minerva.messageBox.styles|socket.io|test.Cite,ContentTranslation,Echo,EventLogging,Flow,Graph,GrowthExperiments,GuidedTour,MediaWiki,MinervaNeue,NavigationTiming,TemplateData,Thanks,Translate,UniversalLanguageSelector,VisualEditor,WikiEditor,WikibaseClient&version=hywdh:2201:225)
04:29:04         at runTest (node_modules/qunit/qunit/qunit.js:2272:35)
04:29:04         at Test.run (node_modules/qunit/qunit/qunit.js:2255:9)
04:29:04         at node_modules/qunit/qunit/qunit.js:2492:16
04:29:04         at processTaskQueue (node_modules/qunit/qunit/qunit.js:1863:26)
04:29:04         at node_modules/qunit/qunit/qunit.js:1867:13
04:29:04

Details

Subject	Repo	Branch	Lines /-
tests: Replace flaky mock with the real thing in onMwLoadEnd() test	mediawiki/extensions/NavigationTiming	master	10 -17
zuul: Install MobileFrontend when testing Echo	integration/config	master	1 -1
tests: Fix QUnit warnings and resolve complex test_NotificationBadge	mediawiki/extensions/Echo	master	10 -28
tests: Avoid loadCallback() in tests and re-enable onMwLoadEnd test	mediawiki/extensions/NavigationTiming	master	47 -129
tests: Improve stability and visibility of emitCpuBenchmark() test	mediawiki/extensions/NavigationTiming	master	28 -22
ext.navigationTiming: Use early return for isRegularNavigation	mediawiki/extensions/NavigationTiming	master	23 -21
Disable flaky test	mediawiki/extensions/NavigationTiming	master	2 -1
ext.navigationTiming: Don't call main() in test context	mediawiki/extensions/NavigationTiming	master	3 -3

Customize query in gerrit

Related Objects

Mentioned Here: T298732: Flaky test: "ext.navigationTiming: onMwLoadEnd - controlled"

Event Timeline

Jdlrobson created this task.Jan 21 2022, 5:11 PM

Restricted Application added a project: Performance-Team. · View Herald TranscriptJan 21 2022, 5:11 PM

Restricted Application added subscribers: Masumrezarock100, Aklapper. · View Herald Transcript

Umherirrender edited projects, added ci-test-error (WMF-deployed Build Failure); removed ci-test-error.Jan 21 2022, 5:13 PM

I thought we fixed this last week: T298732: Flaky test: "ext.navigationTiming: onMwLoadEnd - controlled". I'll try to reproduce it again to investigate. Please do keep reporting once or twice if you still encounter it.

Change 756099 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/extensions/NavigationTiming@master] ext.navigationTiming: Don't call main() in test context

https://gerrit.wikimedia.org/r/756099

gerritbot added a project: Patch-For-Review.Jan 21 2022, 11:53 PM

Krinkle claimed this task.Jan 21 2022, 11:54 PM

Krinkle triaged this task as High priority.

Change 756099 merged by jenkins-bot:

[mediawiki/extensions/NavigationTiming@master] ext.navigationTiming: Don't call main() in test context

https://gerrit.wikimedia.org/r/756099

ReleaseTaggerBot added a project: MW-1.38-notes (1.38.0-wmf.19; 2022-01-24).Jan 22 2022, 2:00 AM

Maintenance_bot removed a project: Patch-For-Review.Jan 22 2022, 2:10 AM

Hopefully fixed for real this time.

Thanks!

It is still flaky. Just failed here: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/VisualEditor/ /757443

https://integration.wikimedia.org/ci/job/wmf-quibble-vendor-mysql-php72-docker/87847/console

14:55:52 FAILED TESTS:
14:55:52   ext.navigationTiming
14:55:52     ✖ onMwLoadEnd - controlled
14:55:52       Chrome Headless 90.0.4430.212 (Linux x86_64)
14:55:52     resolved after fail ok
14:55:52     Expected: [
14:55:52       "call"
14:55:52     ]
14:55:52     Actual: []
14:55:52         at Object.<anonymous> (http://localhost:9876/load.php?modules=mediawiki.experiments,inspect,jqueryMsg,language,messagePoster,pulsatingdot,qunit-testrunner,router,storage,template,toc,ui,user,util,viewport,visibleTimeout,widgets|mediawiki.interface.helpers.styles|mediawiki.language.months,names,specialCharacters,testdata|mediawiki.libs.guiders,jpegmeta,pluralruleparser|mediawiki.page.gallery.styles|mediawiki.page.ready|mediawiki.page.watch.ajax|mediawiki.rcfilters.filters.ui|mediawiki.skinning.content.parsoid|mediawiki.special.recentchanges|mediawiki.template.handlebars,mustache|mediawiki.ui.button,icon|mediawiki.widgets.AbandonEditDialog,CategoryMultiselectWidget,DateInputWidget,MediaSearch,Table,UserInputWidget,styles|mediawiki.widgets.DateInputWidget.styles|mobile.ooui.icons|mobile.pagelist.styles|mobile.pagesummary.styles|mobile.placeholder.images|mobile.startup|mobile.startup.images|moment,oojs,oojs-router,oojs-ui,oojs-ui-core,oojs-ui-toolbars,oojs-ui-widgets,oojs-ui-windows,papaparse,rangefix,sinonjs,spark-md5|mw.config.values.wbDataTypes,wbRefTabsEnabled,wbRepo,wbSiteDetails|mw.cx.SiteMapper,dm,init,ui,util|mw.cx.dm.Translation|mw.cx.tools.TranslationTool|mw.cx.ui.Infobar|oojs-ui-core.icons,styles|oojs-ui-toolbars.icons|oojs-ui-widgets.icons|oojs-ui-windows.icons|oojs-ui.styles.icons-accessibility,icons-alerts,icons-content,icons-editing-advanced,icons-editing-citation,icons-editing-core,icons-editing-list,icons-editing-styling,icons-interactions,icons-layout,icons-media,icons-moderation,icons-movement,icons-user,icons-wikimedia,indicators|skins.minerva.messageBox.styles|socket.io|test.Cite,ContentTranslation,Echo,EventLogging,Flow,Graph,GrowthExperiments,GuidedTour,Math,MediaWiki,MinervaNeue,NavigationTiming,TemplateData,Thanks,Translate,UniversalLanguageSelector,VisualEditor&version=1x2s6:2206:475)
14:55:52         at runTest (node_modules/qunit/qunit/qunit.js:2272:35)
14:55:52         at Test.run (node_modules/qunit/qunit/qunit.js:2255:9)
14:55:52         at node_modules/qunit/qunit/qunit.js:2492:16
14:55:52         at processTaskQueue (node_modules/qunit/qunit/qunit.js:1863:26)
14:55:52         at node_modules/qunit/qunit/qunit.js:1867:13

Krinkle moved this task from Inbox, needs triage to Doing: Prio Interrupt on the Performance-Team board.Jan 26 2022, 8:15 PM

Change 757538 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/extensions/VisualEditor@master] [DNM] Jenkins demo commit

https://gerrit.wikimedia.org/r/757538

gerritbot added a project: Patch-For-Review.Jan 27 2022, 4:25 AM

I have not been able to reproduce this locally at any point last week or this week.

A few outstanding simplifications and code removals have landed today in NavigationTiming. I am now unable to reproduce it on the above VE patch in CI, nor on the dozens of patches in other repos I just reviewed. Maybe those removals got rid of the racey code.

Next time I hear about this, I will first disable (QUnit.skip) the test in question. Others, feel free to do so without my input if you encounter it first.

Just occurred on https://gerrit.wikimedia.org/r/c/mediawiki/skins/Vector/ /757023

Change 757789 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/extensions/NavigationTiming@master] Disable flaky test

https://gerrit.wikimedia.org/r/757789

Change 757789 merged by jenkins-bot:

[mediawiki/extensions/NavigationTiming@master] Disable flaky test

https://gerrit.wikimedia.org/r/757789

ReleaseTaggerBot edited projects, added MW-1.38-notes (1.38.0-wmf.20; 2022-01-31); removed MW-1.38-notes (1.38.0-wmf.19; 2022-01-24).Jan 28 2022, 2:00 AM

Krinkle lowered the priority of this task from High to Medium.Feb 7 2022, 7:44 PM

Krinkle moved this task from Doing: Prio Interrupt to Backlog: Maintenance, non-prioritized on the Performance-Team board.

Krinkle removed a project: ci-test-error (WMF-deployed Build Failure).

Change 765284 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/extensions/NavigationTiming@master] ext.navigationTiming: Use early return for isRegularNavigation

https://gerrit.wikimedia.org/r/765284

Change 765285 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/extensions/NavigationTiming@master] tests: Improve stability and visibility of emitCpuBenchmark() test

https://gerrit.wikimedia.org/r/765285

Change 765333 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/extensions/NavigationTiming@master] tests: Avoid loadCallback() in tests and re-enable onMwLoadEnd test

https://gerrit.wikimedia.org/r/765333

Change 765284 merged by jenkins-bot:

[mediawiki/extensions/NavigationTiming@master] ext.navigationTiming: Use early return for isRegularNavigation

https://gerrit.wikimedia.org/r/765284

ReleaseTaggerBot edited projects, added MW-1.38-notes (1.38.0-wmf.24; 2022-02-28); removed MW-1.38-notes (1.38.0-wmf.20; 2022-01-31).Feb 24 2022, 7:00 AM

Change 765285 merged by jenkins-bot:

[mediawiki/extensions/NavigationTiming@master] tests: Improve stability and visibility of emitCpuBenchmark() test

https://gerrit.wikimedia.org/r/765285

Change 765333 merged by jenkins-bot:

[mediawiki/extensions/NavigationTiming@master] tests: Avoid loadCallback() in tests and re-enable onMwLoadEnd test

https://gerrit.wikimedia.org/r/765333

Still flaky

https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/ /763755
https://integration.wikimedia.org/ci/job/wmf-quibble-vendor-mysql-php72-docker/90847/console

20:53:27 FAILED TESTS:
20:53:27   ext.navigationTiming
20:53:27     ✖ onMwLoadEnd - controlled
20:53:27       Chrome Headless 90.0.4430.212 (Linux x86_64)
20:53:27     resolved after fail ok
20:53:27     Expected: [
20:53:27       "call"
20:53:27     ]
20:53:27     Actual: []
20:53:27         at Object.<anonymous> (http://localhost:9876/load.php?modules=mediawiki.experiments,inspect,jqueryMsg,language,messagePoster,pulsatingdot,qunit-testrunner,router,storage,template,toc,ui,user,util,viewport,visibleTimeout,widgets|mediawiki.interface.helpers.styles|mediawiki.language.months,names,specialCharacters,testdata|mediawiki.libs.guiders,jpegmeta,pluralruleparser|mediawiki.page.gallery.styles|mediawiki.page.ready|mediawiki.page.watch.ajax|mediawiki.rcfilters.filters.ui|mediawiki.skinning.content.parsoid|mediawiki.special.recentchanges|mediawiki.template.handlebars,mustache|mediawiki.ui.button,icon|mediawiki.widgets.AbandonEditDialog,CategoryMultiselectWidget,DateInputWidget,MediaSearch,Table,UserInputWidget,styles|mediawiki.widgets.DateInputWidget.styles|mobile.ooui.icons|mobile.pagelist.styles|mobile.pagesummary.styles|mobile.placeholder.images|mobile.startup|mobile.startup.images|moment,oojs,oojs-router,oojs-ui,oojs-ui-core,oojs-ui-toolbars,oojs-ui-widgets,oojs-ui-windows,papaparse,rangefix,sinonjs,spark-md5|mw.config.values.wbDataTypes,wbRefTabsEnabled,wbRepo,wbSiteDetails|mw.cx.SiteMapper,dm,init,ui,util|mw.cx.dm.Translation|mw.cx.tools.TranslationTool|mw.cx.ui.Infobar|oojs-ui-core.icons,styles|oojs-ui-toolbars.icons|oojs-ui-widgets.icons|oojs-ui-windows.icons|oojs-ui.styles.icons-accessibility,icons-alerts,icons-content,icons-editing-advanced,icons-editing-citation,icons-editing-core,icons-editing-list,icons-editing-styling,icons-interactions,icons-layout,icons-media,icons-moderation,icons-movement,icons-user,icons-wikimedia,indicators|skins.minerva.messageBox.styles|socket.io|test.Cite,ContentTranslation,Echo,EventLogging,Flow,Graph,GrowthExperiments,GuidedTour,Math,MediaWiki,MinervaNeue,NavigationTiming,TemplateData,Thanks,Translate,UniversalLanguageSelector,VisualEditor&version=9vvsy:2245:70)
20:53:27         at runTest (node_modules/qunit/qunit/qunit.js:2496:35)
20:53:27         at Test.run (node_modules/qunit/qunit/qunit.js:2479:9)
20:53:27         at node_modules/qunit/qunit/qunit.js:2770:16
20:53:27         at processTaskQueue (node_modules/qunit/qunit/qunit.js:2051:26)
20:53:27         at node_modules/qunit/qunit/qunit.js:2055:13

Again for https://gerrit.wikimedia.org/r/c/mediawiki/core/ /767224

✔ 1659 tests completed
19:11:49 ℹ 11 tests skipped
19:11:49 ✖ 1 test failed
19:11:49 
19:11:49 FAILED TESTS:
19:11:49   ext.navigationTiming
19:11:49     ✖ onMwLoadEnd - controlled
19:11:49       Chrome Headless 90.0.4430.212 (Linux x86_64)
19:11:49     resolved after fail ok
19:11:49     Expected: [
19:11:49       "call"
19:11:49     ]
19:11:49     Actual: []
19:11:49         at Object.<anonymous> ( ...

Change 767493 had a related patch set uploaded (by Krinkle; author: Krinkle):

[integration/config@master] zuul: Install MobileFrontend when testing Echo

https://gerrit.wikimedia.org/r/767493

Change 767255 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/extensions/Echo@master] tests: Fix QUnit warnings and resolve complex test_NotificationBadge

https://gerrit.wikimedia.org/r/767255

Change 767493 merged by jenkins-bot:

[integration/config@master] zuul: Install MobileFrontend when testing Echo

https://gerrit.wikimedia.org/r/767493

Ammarpad unsubscribed.Mar 2 2022, 7:13 PM

Change 767255 merged by jenkins-bot:

[mediawiki/extensions/Echo@master] tests: Fix QUnit warnings and resolve complex test_NotificationBadge

https://gerrit.wikimedia.org/r/767255

ReleaseTaggerBot edited projects, added MW-1.38-notes (1.38.0-wmf.25; 2022-03-07); removed MW-1.38-notes (1.38.0-wmf.24; 2022-02-28).Mar 2 2022, 11:00 PM

Ladsgroup subscribed.Mar 4 2022, 1:49 PM

Reedy raised the priority of this task from Medium to High.Mar 5 2022, 4:55 PM

Change 768230 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/extensions/NavigationTiming@master] tests: Replace flaky mock with the real thing in onMwLoadEnd() test

https://gerrit.wikimedia.org/r/768230

Change 768230 merged by jenkins-bot: