Background
We would like to run an A/B test on the page issues feature for two weeks. The test will display the new page issues code to one group and the older version to the remainder (as per the bucketing code built in T193584: Prepare feature flagging gateway for mobile issues work and the instrumentation built in T191532: Mobile page issues - instrument page issues. We will collect data for 4 weeks, plus a run-up time of 1-2 days to limit caching and novelty effects. A separate task has been set up for turning the test off: T200793 . Suggested sampling ratio: 20% (per T200792#4489268)
Questions we are trying to answer
See T200794: Analyze results of page issues A/B test
Acceptance criteria
- Notify the Analytics Engineering team as described here to have both ReadingDepth and PageIssues blacklisted from being stored in MariaDB (see T200792#4489268).
(done, see corresponding AC in T191532)
- Enable the A/B test on a single wiki (Latvian Wikipedia, e.g.https://lv.wikipedia.org/wiki/Ropažu_sporta_centrs) at a sampling ratio of 100%: T204609
- Run for 2-3 days and check the data coming in (TBC by Olga) - checks made on Latvian wiki - https://phabricator.wikimedia.org/T204609#4630216
- Ensure wikis are ready for testing
Japanese: https://phabricator.wikimedia.org/T204090#4626354
Russian: https://phabricator.wikimedia.org/T204090#4626381
Persian: https://phabricator.wikimedia.org/T204090#4626510
- Roll out using several deploy windows in a single day - take care rolling this out in case there are any unexpected spikes in errors or event logging:
- Progressively roll out page issues to the target projects at 5%
- Progressively roll out page issues to the target projects at 10%
- Set A/B test for page issues for select projects, at the A/B test sampling ratio of 20% for enwiki (per T200792#4489268) and 100% for the other three, smaller wikis (per T200792#4632005 ).
- Increase sampling rate on jawiki, ruwiki, fawiki to 100%
Tentative dates:
- Latvian wiki: sept 19
- English, Russian, Japanese, Farsi: oct 1 (Switch from Catalan to Farsi due to some smaller issues on Catalan and so that we can also have an rtl language included in the test)
Sign off steps
- Wait a few days to get confirmation from Tilman that the A/B test instrumentation working as expected.
- If necessary, create follow-up tasks (for bugs, inconsistencies, etc)
--> T204143 in particular
- Setup task to analyse A-B test
--> T200794
- Prepare deploy/remove code task that will be carried out based on A-B test results.
- Add a note to the project page
- Add a note to the release timeline
TODO
- Use @phuedx or @pmiazga's "bucket breaker" script to find a session ID that'll put you in the correct bucket for testing
Run this code to opt into the PageIssues A/B test:
M = mw.mobileFrontend;
AB = M.require( 'skins.minerva.scripts/AB' );
var t = 0, abTest;
function check() {
abTest = new AB( {
testName: 'WME.PageIssuesAB',
// Run AB only on article namespace, otherwise set samplingRate to 0,
// forcing user into control (i.e. ignored/not logged) group.
samplingRate: mw.config.get( 'wgMinervaABSamplingRate', 0 ),
sessionId: t
} );
}
check();
while (!abTest.isB()) {
t =1;
check();
}
mw.storage.session.set('mwuser-sessionId',t);