@TyA Yes sorry for the mistake and thanks for letting me know!

I've completed calculating the identified metrics for the usability improvements engagement analysis. This analysis focuses on if the new features are negatively impacting peoples' experiences using Wikipedia talk pages to determine if any changes are needing before running an AB test.

Please see the key results summarized below and the full report for more details on methodology and various other breakdowns.

Methodology

To assess impact, I reviewed changes in the identified metrics two weeks pre and post-deployment at the three wikis where currently deployed as default on desktop (ar.wiki, cs.wiki, and hu.wiki). Reviewed data was limited to any desktop edit on a talk page at the participating Wikipedias.

Note: Since this is a pre and post-deployment analysis (and not an AB test), there may be other factors such as seasonal trends that may have impacted these results. For this analysis, we are primarily looking to identify any significant increases or decreases in the identified metrics. Additional analysis to understand the new feature impacts on peoples' behavior will be addressed in the AB test.

Key Takeaways

Overall, we did not observe any significant changes in the overall or daily revert rate or completion rate of talk page edits on desktop across all three participating Wikipedias post-deployment. Trends vary based on wiki, user type, and the editor interface used. We observed a decrease in revert rate and a increase in edit completion rate specifically for Junior logged-in editors and edits made using visual editor. Additionally, there was no significant increase in user block rate and the opt-out rate by talk page editors was small (Only 15 users of 2122 users (0.71%) that made a talk page edit attempt).

Revert Rate

• We did not observe any changes in the overall talk page edit revert rate pre and post-deployment across the three participating wikis.

• There was a -17% percent decrease in the revert rate of talk page edits made with visual editor and a -4% decrease in the revert rate of edits made with wikitext.

• Similar to results observed for other desktop talk page projects, we observed more of an impact on the quality of Junior Contributors' edits following deployment of the new talk page features. There was a -18% decrease in the revert rate of desktop talk page edits by Junior Contributors (under 100 edits) following the deployment of the feature; compared to 4.5% increase ( 0.1 percentage points) for Senior editors (over 100 edits).

Edit Completion Rate

• Overall across all three participating wikis, we observed a -4.8 percent decrease in edit completion rate but there was no sudden increase or decreases in daily completion rates following deployment of the feature (as shown in the “Daily edit completion rate” chart below)

• Per wiki results vary. We observed an increase (21% increase) at Hungarian Wikipedia while we observed a decrease at Arabic (-19.8% decrease) and Czech Wikipedias (-7.2% decrease).

• We observed slight increases in edit completion rate for talk page edits completed using visual editor and wikitext-2017 and observed a slight decrease for wikitext edits.

• There was a 10.2% increase in talk page edit completion rate for junior editors and a slight decrease (-1.3%) in edit completion rate for senior editors

Block Rate

We did not observe any significant changes in block rate. A total of 4 talk editors across all three wikipedias were issued a sitewide block two weeks post-deployment, compared to 3 talk page editors two weeks pre-deployment.

Opt-Out Rate

There are a total of 48 users across all three wikis that have turned the feature off. 39 of those users from arwiki.
Only 15 users of 2122 users (0.71%) that made a talk page edit attempt following the deployment of usability improvements at the three participating wikis currently have the preference disabled.