Hi @Jdlrobson

As part of the Anti-Harassment Tools team's continuing work on blocking tools we are looking to measure how often blocked users attempt (and fail) to edit pages.

To start this project we'd like to add tracking to the mobile block drawer you helped us build a few months ago (this ticket.) We would appreciate any advice you'd have on where to start — happy to discuss here or have a quick meeting to get started.

Thank you!

It looks like Graphite will listen to EventLogging extension:
https://wikitech.wikimedia.org/wiki/Graphite#EventLogging
Event logging has a method for client-side events:
https://www.mediawiki.org/wiki/Extension:EventLogging/Guide#Underlying_technology
examples:
https://www.mediawiki.org/wiki/Extension:EventLogging/Programming#Client-side_logging

We can also use statsv which is just an HTTP endpoint
https://wikitech.wikimedia.org/wiki/Graphite#statsv

One thing to note about Graphite, especially for Trevor maybe, is that over time the completeness and and correctness of the data it has degrades. Depending on its configuration, Graphite does some roll-up averages (probably not the actual technical phrase) over time. So, if you look at the numbers for this week and want to compare them to a particular week 4 months ago, the scale and accuracy might be hard to reconcile. Since it's a time-series database and not a store like elasticsearch or hadoop or something else, it's focus is on trending over time and not being able to tell you that March of 2017 had 42 blocks while last week had 13.

We ran into this at Rackspace a lot because our POs were trying to use Graphite data to compare time periods across months and the data is just never comparable. We built an ElasticSearch/Kibana setup to track user/application analytics that need solid comparisons over longer timeframes. I'm not suggesting that here but just noting a potential shortfall of the Graphite solution.

That's good to know in advance, Alex. Thank you. We won't need 100% perfect data on this first implementation, just directional. There is room for an acceptable amount of error.

I don't know anything about our logging. :) but it does look like you can use Graphite, or you can also use the schema (db?) logging from the client. So we should be golden either way. It also looks like MobileFrontend is using mw.track (Graphite) for most of it's logging (as far as I can tell).

In T201719#4505087, @aezell wrote:

One thing to note about Graphite [..] is that over time the completeness and and correctness of the data it has degrades. [..] trying to use Graphite data to compare time periods across months and the data is just never comparable. [..]

Graphite can be quite confusing, and for the better part of 5 years, this bothered me as well. However, while I do recommend Prometheus over Graphite any day, I have come to understand Graphite much better and no longer believe the correctness degrades. It comes down to three things:

1. Configuration (For the system admin): Graphite's backend must be configured to not discard low-resolutions of data (it discards <0.1XFF by default), and to appropriately aggregate your metrics (it uses "mean average" otherwise). For example, a metric foo_bar.sum that counts "events occurred" should be aggregated using sum aggregation instead. At WMF, some metrics were misconfigured before 2013, but have been correct since.

2. Query range (For the user): Contrary to popular belief, Graphite does not aggregate over time. Instead, it has (for example) a separate "last 7 days" and "last 3 years" database. Recent data is in both. This means as long as you use the same source, both recent and old will be comparable. Querying "1 day this week" and then "1 day sometime last month" is a problem. Querying the whole month and, within that, looking at a day this week and a day last month is fine.

3. Resolution, or interval (For the user): Shorter-term databases have higher resolutions (shorter interval) than longer-term databases. For example, a metric like foo_bar.sum that is correctly aggregated and consistently receives 1 increment every second, will show "60" for recent data (resolution = 1 minute), but might show "900" for the same time period in the longer-term database (resolution = 15 minutes). The resolution can be observed by comparing the timestamps in Grafana's tooltips. To avoid this problem, our practice is generally to use "rate" instead of "sum", which we aggregate as average and remains a "rate per second" even for older data.

See also https://wikitech.wikimedia.org/wiki/Graphite#Extended_properties where I wrote more about this.

@Krinkle Thanks for the details! I appreciate knowing a bit more about how it actually works. I always had a sense there was a way to configure it to be better in this regard but using other tools for additional reasons is the path I always took before.

Due to my limited familiarity with MobileFrontend I'm looking for some feedback/help on this before I submit the patch for review.

I've created a patch[1] with what I think is the right implementation. All we need is a simple counter every time the BlockMessage drawer is displayed. As per the docs[2], calling mw.track() with a counter prefix should do the trick.

Any help getting this change reporting to statsd locally (I'm using our vagrant setup) will be very much appreciated. I'm currently able to see data coming into statsd from php but not from js using mw.track.

[1] https://gerrit.wikimedia.org/r/c/mediawiki/extensions/MobileFrontend/ /468684
[2] https://wikitech.wikimedia.org/wiki/Graphite#statsv

I think you've got the right idea here.
You can verify this by calling the counter locally
and checking if it comes up in grafana

(see these for example: https://grafana-labs-admin.wikimedia.org/dashboard/db/reading-web-beta-dashboard?orgId=1 )

You might want to consider feature flagging it so you have a kill switch in case of any problems.

Thank you @Jdlrobson, I'll give that a try. I don't think a feature flag is necessary in this case but I defer to your expertise.

@TBolliger, do you care what the key to track this will be like? @dbarratt is suggesting counter.MediaWiki.MobileFrontend.blocked_message_showed. I think that works.

@dmaza — No strong preference from me, as long as it is clear, documented on this ticket, and abides by any naming convention(s) that exist for event logging.

@TBolliger, what exactly are we trying to get out of these numbers?

• Do we want a the sum of all wikis when a user sees a block?
• Do we want the count for each wiki individually (all wikis or a subset)?
• Do we want only a subset of wikis grouped together (E.g. en, it, de as one single count)

Change 468684 had a related patch set uploaded (by Dmaza; owner: Dmaza):
[mediawiki/extensions/MobileFrontend@master] Track when block notices are displayed.

https://gerrit.wikimedia.org/r/468684

In T201719#4686937, @dmaza wrote:

@TBolliger, what exactly are we trying to get out of these numbers?

• Do we want a the sum of all wikis when a user sees a block?
• Do we want the count for each wiki individually (all wikis or a subset)?
• Do we want only a subset of wikis grouped together (E.g. en, it, de as one single count)

individual wikis would be preferable, but sum of all would be second preferable.

Change 468684 had a related patch set uploaded (by Jdlrobson; owner: Dmaza):
[mediawiki/extensions/MobileFrontend@master] Track when block notices are displayed.

https://gerrit.wikimedia.org/r/468684

Change 468684 merged by jenkins-bot:
[mediawiki/extensions/MobileFrontend@master] Track when block notices are displayed.

https://gerrit.wikimedia.org/r/468684

Change 472376 had a related patch set uploaded (by Dmaza; owner: Dmaza):
[mediawiki/extensions/MobileFrontend@master] Fix key for storing metrics when tracking block notices

https://gerrit.wikimedia.org/r/472376

Just pointing out that the main question that the ticket lists is: "to measure how often blocked users attempt (and fail) to edit pages."

As far as I can see you are not sending any data "per user" so will not be able to answer the question, correct? (maybe this is planned for a second instrumentation phase) You are going to get a counter on how many times a blocker notice displays per:

https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/MobileFrontend/ /468684/7/resources/mobile.editor.overlay/BlockMessage.js

This will not tell you (please correct me if i am wrong) wether is the SAME user (or bot) trying to edit over and over or many different users hitting that block message. This is an important distinction cause you could have 1000 users trying to edit receiving the block message or 1 user trying to edit a 1000 times.

Good question and good point, Nuria. Thank you for asking.

This was intentional. We did not want to capture any PII just for simplicity of execution. We wanted to capture some extremely basic information as a starting point — if we find that these messages are rarely/casually/frequently displayed, then we can form some hypotheses and dig in further.

We don't plan to make any important decisions with this data, it's mostly illuminatory. 💡

Review from us is blocked on performance go ahead https://gerrit.wikimedia.org/r/#/c/mediawiki/core/ /472375/

Change 472376 abandoned by Dmaza:
Change key for storing metrics when tracking block notices

Reason:
Not needed anymore

https://gerrit.wikimedia.org/r/472376

Change 476392 had a related patch set uploaded (by Dmaza; owner: Dmaza):
[mediawiki/extensions/MobileFrontend@master] Change config flag for enabling Block Notice stats

https://gerrit.wikimedia.org/r/476392

Change 476392 merged by Jdlrobson:
[mediawiki/extensions/MobileFrontend@master] Change config flag for enabling Block Notice stats

https://gerrit.wikimedia.org/r/476392

Change 476613 had a related patch set uploaded (by Dmaza; owner: Dmaza):
[mediawiki/extensions/MobileFrontend@wmf/1.33.0-wmf.6] Change config flag for enabling Block Notice stats

https://gerrit.wikimedia.org/r/476613

Change 476613 merged by jenkins-bot:
[mediawiki/extensions/MobileFrontend@wmf/1.33.0-wmf.6] Change config flag for enabling Block Notice stats

https://gerrit.wikimedia.org/r/476613

Mentioned in SAL (#wikimedia-operations) [2018-11-30T01:04:19Z] <thcipriani@deploy1001> Synchronized php-1.33.0-wmf.6/extensions/MobileFrontend: SWAT: [[gerrit:476613|Change config flag for enabling Block Notice stats]] T201719 (duration: 00m 48s)