This task is about determining what – if any – new instrumentation is needed for us to measure the impact the notification interventions being implemented (T273920) as part of the Talk pages project.
Background
Work on this task will begin once we know:
- The experiments we will run to evaluate the impact of the notification interventions and
- The metrics we will use in these "experiments"
"1." and "2." will be finalized once the notifications measurement plan (T274215) is drafted.
Instrumentation changes
The instrumentation changes needed to evaluate the impact of the notification interventions we have planned (T273920) are documented in the Topic Notifications/Instrumentation Spec.
Open questions
- 1. How are we – collectively – defining "topics," "comments," and "replies"?
- See T277349#7087010 for written definitions and/or visit a talk page in debug mode (e.g. https://en.wikipedia.beta.wmflabs.org/wiki/Talk:Cats?dtdebug=1) to see these definitions in practice.
- Note: "topics" in the context of the topic subscription feature are limited to ==H2== headings.
- 2. Which of the below do we care to know/measure?
- A. ✅When/if someone posts a comment in response to a new topic 'you've' started?
- We will potentially use this information to evaluate the impact of this feature.
- Emphasis placed on "potentially" above to signal the possibility that T263821 could be de-scoped.
- We will potentially use this information to evaluate the impact of this feature.
- B. ✅ When/if someone comments in a section after you have already commented in it?
- We will not use this information to evaluate the impact of this feature; however, we will use it as a "Curiosity" to understand how topic subscriptions might be impacting the cadence of conversations. This has been added as "Curiosity #3" in T280895's task description.
- ✅ C. When/if someone replies directly to a comment you've written?
- A. ✅When/if someone posts a comment in response to a new topic 'you've' started?
-
3. How will we relate comments to sections?- Note: this assumes our interest is limited to comments and topic NOT replies.
- This question will be answered in T280100.
-
4. How will we relate replies to comments and comments to sections?- Note: this assumes we are interested in topics, comments and replies posted in direct response to comments.
- This question will be answered in T284200
- 5. Is it a priority to be able to track the full "lifecycle" of the notification flow?
- Where "Notification flow" = notifications is sent (start) → response is posted (end).
- No and the rationale can be found in T277349#7087010. Instead, we would like to be able to track/define the notification flow as follows:
- 1) Notification sent
- 2) Notification delivered
- 3) Notification opened/read
- 4) Notification interacted with
- 6. For new notification instrumentation (read: Echo), to what schema will these new events be added? An existing schema? A new schema?
- This is answered in the Topic Notifications/Instrumentation Spec.
Done
- Answers to all ===Open questions are answered and documented in the ticket
- The table in the Instrumentation changes section above is populated with the new instrumentation and/or changes to existing instrumentation that is needed to evaluate the impact of the notification interventions we have planned.