Calculate revert rate baseline
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	ppelberg
	Jul 29 2020, 9:43 PM

Description

This task is about calculating the desktop revert rate for newcomers (defined below) editing Wikipedia on desktop.

Background

It is possible people who are shown the visual editor on mobile and the people who are shown the source editor on mobile are equally likely to be retained (read: to start and continuing editing Wikipedia).

Should the scenario described above arise, we will look at other metrics – of which revert rate is one – to determine which editing interface should be shown to newcomers by default. Specifically, we will look at whether the revert rate in one test group (default-visual or default-source) is significantly higher than the comparable revert rate on desktop.

This task is about calculating said "comparable revert rate on desktop" which will serve as the baseline to which we will compare the test group revert rates.

Analysis

We are curious to learn the percentage of edits that are reverted on desktop that meeting the following conditions:

Edits made on desktop web
Edits made using the visual editor or the wikitext editor
Edits made by people (read: not bots)
Edits made by people who are logged out or who have made 0 cumulative edits
Edits made to a Wikipedia content namespace
Edits that are reverted within 48 hours of being made

Done

The "Analysis" described above is computed

Related Objects
Search...

Status	Assigned	Task
Open	None	T255327 [Epic] Evaluate which editing interface should be shown by default
Open	None	T227338 Test visual editor as the default mobile editor on select wikis
Duplicate	ppelberg	T221187 VE mobile default: create measurement specifications and experiment plan
Resolved	ppelberg	T235101 Rerun mobile VE as default A/B test
Resolved	ppelberg	T221198 VE mobile default: analyze A/B test results
Resolved	MNeisler	T259196 Calculate revert rate baseline

Event Timeline

ppelberg created this task.Jul 29 2020, 9:43 PM

ppelberg mentioned this in T221198: VE mobile default: analyze A/B test results.

ppelberg updated the task description. (Show Details)Jul 29 2020, 10:19 PM

ppelberg moved this task from Backlog to Analytics on the Editing-team (Tracking) board.

MNeisler triaged this task as Medium priority.Aug 3 2020, 3:00 PM

MNeisler moved this task from Next 2 weeks to Doing on the Product-Analytics (Kanban) board.

Here are the baseline revert rates for anonymous users or first-time editors on Wikipedia desktop on each editing interface. In this analysis, revert rate is defined as the proportion of non-bot edits which were reverted within 48 hours.

Overall revert rate (2019-07 to 2020-07)
VisualEditor: 30.1%
Wikitext: 23.9%

There is a fairly consistent difference in monthly revert rates between the two editing interfaces over the past two years, with the maximum difference of around 8%. The average difference in monthly revert rates between the two editing interfaces for the past two years is 6.2%. Visual Editor has a higher revert rate ranging from 24% to 32% while wikitext's revert rate ranges from about 21% to 26%.

desktop_revert_rate_monthly_ts_plot.png (2×4 px, 263 KB)

desktop_revert_rate_monthly_yoy_plot.png (2×4 px, 282 KB)

Based on these numbers, I think if we see a difference in revert rates above 10% between the two test groups would be significant and should be considered along with the other identified metrics comparing the two groups. If we do find a significant difference in revert rates, I'd recommend comparing the distribution of revert rates and users within each test group to get more insight into the source of the reverts.

Superset Dashboard for Self-Exploratory Data
https://superset.wikimedia.org/r/295

You can use the filter box to review revert rates for each interface, language, if the editor is anonymous, or the editor's cumulative edit count bucket.
Note this data includes edits reverted anytime before the most recent snapshot (not within 48 hours) and can not filter to just editors with 0 cumulative historical edits so the rates will differ slightly from what is described above.

See notebook for further details and results.

What you shared in T259196#6374218 looks good – thank you, @MNeisler.

Per the conversation we had today, I'm going to resolve this task considering we now have the data it was asking for.

If further analysis is needed following the conversation we will be having with the team about the "revert rate significance threshold" [i], we can either re-open this task or file a new one.

i. From T221198's "Open questions" section:

How much higher does the revert rate in either mode need to be for it to be "significant"?

Restricted Application added a project: User-Ryasmeen. · View Herald TranscriptAug 13 2020, 1:36 AM

	F32033347: desktop_revert_rate_monthly_ts_plot.png
	Aug 10 2020, 6:59 PM

	F32033351: desktop_revert_rate_monthly_yoy_plot.png
	Aug 10 2020, 6:59 PM

Calculate revert rate baselineClosed, ResolvedPublicActions