Add logging for measuring impact of our work on improving the mobile block messages
Closed, ResolvedPublic3 Estimated Story PointsSep 22 2020
Actions

Assigned To

Authored By

	Niharika
	Aug 12 2020, 4:51 AM

Description

Goal

This ticket relates to T189717: Better handle block reasons on mobile (specifically templates and HTML comments) which has some more context. We are making significant improvements to block messages on mobile. This has been brought up time and again as a problem for our end users who unexpectedly find themselves blocked.
This is important work and we want to make sure we can measure and highlight the impact we're having by doing this work.

Acceptance criteria:

Log the following:
- How many times does the mobile block drawer display for end users?
- How many times does a 'call to action' display (other than 'OK')?
- How many times do users click on the 'call to action' (other than 'OK') (T259775)?
- How many times do users access the special page or the expanded drawer to see the reason (T259755)?
We will use statsd to capture these numbers and we don't log the type of CTA.

Note

This has been done in the past. Related task - T201719: Tracking blocks: Log when the mobile web editor block notice is displayed
Q: Do we have any of the past data still available?

Details

Due Date: Sep 22 2020, 4:00 AM

	Subject	Repo	Branch	Lines /-
	Enable mobile block notice tracking in MobileFrontend	operations/mediawiki-config	master	4 -0
	Reinstate analytics tracking for the block message drawer	mediawiki/extensions/MobileFrontend	master	44 -6

Customize query in gerrit

Related Objects

Mentioned In: T264346: Track improvements made to mobile block messages
T261253: Baseline studies for block messages on mobile
Mentioned Here: T189724: 👩‍👧 Track how often blocked users attempt to edit
T189717: Better handle block reasons on mobile (specifically templates and HTML comments)
T201719: Tracking blocks: Log when the mobile web editor block notice is displayed
T259755: [Spike] Investigate creating a modal or a new Special Page for displaying the block reason on mobile [2HRs]
T259775: Improve call to action in block drawer

Event Timeline

Niharika triaged this task as Medium priority.Aug 12 2020, 4:51 AM

Niharika created this task.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 12 2020, 4:51 AM

@jwang I want to flag this task for you. We can talk about it when we meet next.

Niharika updated the task description. (Show Details)Aug 19 2020, 4:46 PM

ARamirez_WMF set the point value for this task to 3.Aug 19 2020, 4:47 PM

ARamirez_WMF moved this task from Triage/To be Estimated to Cards ready for development on the Anti-Harassment board.

Niharika updated the task description. (Show Details)Aug 19 2020, 4:48 PM

Niharika removed the point value for this task.

Niharika set the point value for this task to 3.Aug 19 2020, 5:06 PM

Niharika moved this task from Cards ready for development to The Letter Song on the Anti-Harassment board.Aug 25 2020, 7:09 PM

Niharika edited projects, added Anti-Harassment (The Letter Song); removed Anti-Harassment.

As discussed with @Niharika and @jwang:

The calls to action are "Log in" and "See random page". There's also "OK", but that dismisses the drawer and is probably less interesting.
We need to know (1) how many times was drawer shown, (2) how many times were the interesting calls to action shown, (3) how many times were the interesting calls to action clicked on

jwang mentioned this in T261253: Baseline studies for block messages on mobile.Aug 25 2020, 9:04 PM

Just found the old data on Grafana before the tracking was removed in this commit as part of T189724: https://grafana.wikimedia.org/explore?orgId=1&left=["1542974555843","1558666631686","graphite",{"datasource":"graphite","target":"sumSeries(MediaWiki.BlockNotices.*.MobileFrontend.shown.count)"},{"ui":[true,true,true,"none"]}]

And here's per-wiki: https://grafana.wikimedia.org/explore?orgId=1&left=["1542974549843","1558666637686","graphite",{"datasource":"graphite","target":"MediaWiki.BlockNotices.*.MobileFrontend.shown.count"},{"ui":[true,true,false,"none"]}]

Thanks @mpopov! This is helpful.

Thank you @mpopov !

Tchanders updated the task description. (Show Details)Aug 31 2020, 12:06 PM

Tchanders claimed this task.Sep 8 2020, 3:08 PM

Tchanders moved this task from Ready 🎬 (ONLY IF YOU HAVE NO MORE CODE TO REVIEW) to In Progress 💪 on the Anti-Harassment (The Letter Song) board.

ARamirez_WMF changed the subtype of this task from "Task" to "Deadline".Sep 9 2020, 8:45 PM

ARamirez_WMF set Due Date to Sep 22 2020, 4:00 AM.

Change 626710 had a related patch set uploaded (by Tchanders; owner: Tchanders):
[mediawiki/extensions/MobileFrontend@master] Reinstate analytics tracking for the block message drawer

https://gerrit.wikimedia.org/r/626710

gerritbot added a project: Patch-For-Review.Sep 11 2020, 3:57 PM

Tchanders moved this task from In Progress 💪 to Code Review 🔍 on the Anti-Harassment (The Letter Song) board.Sep 11 2020, 3:58 PM

Change 626710 merged by jenkins-bot:
[mediawiki/extensions/MobileFrontend@master] Reinstate analytics tracking for the block message drawer

https://gerrit.wikimedia.org/r/626710

Maintenance_bot removed a project: Patch-For-Review.Sep 17 2020, 8:10 PM

ReleaseTaggerBot added a project: MW-1.36-notes (1.36.0-wmf.10; 2020-09-22).Sep 17 2020, 9:00 PM

Tchanders moved this task from Code Review 🔍 to QA/Testing 🐞 on the Anti-Harassment (The Letter Song) board.Sep 18 2020, 5:36 PM

@Tchanders we are not seeing this data in Grafana yet. Jennifer built a dashboard to track it here. The old data is available in the dashboard if you zoom out.
If we are capturing the data the same way, it should show up in the dashboard, correct?

Change 629691 had a related patch set uploaded (by Tchanders; owner: Tchanders):
[operations/mediawiki-config@master] Enable mobile block notice tracking in MobileFrontend

https://gerrit.wikimedia.org/r/629691

gerritbot added a project: Patch-For-Review.Sep 24 2020, 2:24 PM

@Niharika This is only on 1.36.0-wmf.10, so it wouldn't be giving any significant data yet. It also needs to be enabled via a config change, which we'll deploy later today - thanks for the reminder.

Change 629691 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable mobile block notice tracking in MobileFrontend

https://gerrit.wikimedia.org/r/629691

Mentioned in SAL (#wikimedia-operations) [2020-09-24T19:04:05Z] <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: bcf9fcbe3b82ab85b8f97206ceca45b64619c362: Enable mobile block notice tracking in MobileFrontend (T260218) (duration: 01m 04s)

Maintenance_bot removed a project: Patch-For-Review.Sep 24 2020, 7:11 PM

In T260218#6491523, @Tchanders wrote:

@Niharika This is only on 1.36.0-wmf.10, so it wouldn't be giving any significant data yet. It also needs to be enabled via a config change, which we'll deploy later today - thanks for the reminder.

Awesome, thanks! @jwang can you double check if the data is showing up now?

I saw the data after 2020-09-24 19:00:00 showing up. Thanks!

Niharika moved this task from QA/Testing 🐞 to Done: Q1 (2020-21) 💯 on the Anti-Harassment (The Letter Song) board.Sep 28 2020, 6:05 PM

Niharika closed this task as Resolved.Oct 1 2020, 6:19 PM

Niharika mentioned this in T264346: Track improvements made to mobile block messages .

	F32194959: per-wiki.png
	Aug 26 2020, 2:49 AM

	F32194957: all-wikis.png
	Aug 26 2020, 2:49 AM

Add logging for measuring impact of our work on improving the mobile block messagesClosed, ResolvedPublic3 Estimated Story PointsSep 22 2020Actions