Approved

@KSiebert

not applicable, data scientist will check if tracking works

Instrumentation QA is two-step process. First, the engineer (whether the software engineer or a QTE) needs to QA the instrumentation as much as they can to make sure that it is producing the desired events (see https://wikitech.wikimedia.org/wiki/Metrics_Platform/How_to/Validate_Events, including that there aren't any validation errors).

Will sample consistently if given the same starting value (e.g. if we're sampling on page ID, the same page ID will always return the same assignment).

Questions for @JTannerWMF & @SNowick_WMF:

Hi @ifried! @Iflorez and I discussed your question yesterday and it's going to come down to how the feature actually works. Either:

• Scenario A: User can switch between the tabs seamlessly (powered by JS) without needing to reload the page.
• Scenario B: When page loads it checks for ?tab=Events (default) or ?tab=Communities and switching between them causes the page to reload with different URI query parameter.

@mforns: I just discussed this with @KCVelaga_WMF and confirmed that multiple pieces of information will be "all manifest at the same time, atomically"

Apologies @Tchanders – I created the placeholder ticket based on misunderstanding/misinterpreting the term "rollback" when Niharika's request came in to have a rollback metric (no other details given) so I read "rollback" as rolling back Temp Accounts, and I see now that a much different thing was intended.

@larissagaulia: Can you please check with your team if T52655: Allow adding canonical names for custom namespaces is indeed still a blocker for this or if it has been resolved, just not linked to the ticket?

@cjming @Ottomata: Another negative to document (and think about): event sanitization. We can configure sanitization/retention policies on a per-instrument basis since they are different streams/tables, but with the monostream/monotable we would lose that flexibility. Without changing how the current sanitization pipeline works, we would have a single entry in the allowlist for the monotable. We would have to reconsider how we evaluate risk when it comes to retaining sanitized data longer than 90 days.

@Ottomata @xcollazo: I can't review all the discussion on this ticket but Andrew pointed me here:

How do we measure this in an automated/pipeline-able way without requiring manual data input?

@Niharika: Is this still needed for any decision making with regards to Temp Accounts implementation or can I decline this?

Pausing this work in favor of T371402: [WE4.4] Additional monitoring capabilities for rollout of Temporary Account

Please add the following to Negative Consequences:

• Vast majority of interaction data would be going into one massive table which will create significant limitations for how much data we would be able to query with Presto – potentially only an hour at a time as opposed to multiple days or weeks or even months that is possible now with the smaller, per-instrument tables. Depending on how powerful our Presto cluster is, we may would likely have to switch to working with interaction data exclusively outside of Superset's SQL Lab since Presto and Spark SQL differ substantially and require a high degree to effort to translate queries between those two SQL dialects.
• This will also negatively affect our ability to create Superset dashboards with Presto based on the un-aggregated interaction data, which has become a common practice among Product Analysts. We accept this consequence because the metrics we measure and make available in those dashboards and other reports should be pre-computed with data pipelines (that have access to more powerful and robust Spark SQL) rather than calculated on-the-fly with Presto. We can still use Presto but mainly for working with pre-computed measurements of interaction metrics rather than with raw interaction data.

Rescinding after checking in with @acooper and clarifying that this was created out of a misunderstanding.

(The file attachment – a diagram I suspect? – is missing.)

It also means there's nobody to ask to fix the behavior. I believe this requires engineering help from DPE.

@Mayakp.wiki: Special:GlobalUsage comes from Extension:GlobalUsage (GlobalUsage), which is a volunteer-authored extension.

Thanks for checking in! No objection from me.

@phuedx @VirginiaPoundstone: Okie dokie, I updated the task description with the data model specification and updated the requirements based on a guess of what is involved.

@sbassett: I have folks on my team very eager to use this again for reporting. (I've asked them to pause until it's marked as Approved in ITS' Software Catalog.) Is there an ETA for this review?

Morten completed this as part of SDS 2.5.7. Here is his report: https://docs.google.com/document/d/1HzEuDzEkwBtBexTlL5b4JiRjlhOP-dCBfAbckwMj4vc/edit#

Status update: Product Analytics is going to own a hypothesis under WE 4.2, likely in Q2 (since we are fully booked for Q1), to the effect of:

If we develop a metric for measuring how many event participants are prevented from participating in events due to IP blocks, we will be able to measure baselines across different dimensions (such as geographic regions and languages) to understand the severity and distribution of the problem.

(To be refined by the hypothesis owner (TBD) with @kostajh as KR owner)

Collected various discussions & points into this decision brief (viewable by public). Will follow up here when the decision is made.

I believe mediawiki.web_ui_actions succeeds eventlogging_DesktopWebUIActionsTrackingand eventlogging_MobileWebUIActionsTracking
and mediawiki.web_ui_scroll_migrated succeeds mediawiki.web_ui_scroll

@KSarabia-WMF: The default is in the stream config (pasted above) and it's currently set to 20% of sessions on all projects. And what's happening with the migrated instrumentation is that it's being double-sampled – the instrument initializes on 20% of pageviews (1% on English projects) and when it produces events to the mediawiki.web_ui_actions stream, only 20% of sessions will actually submit an event.

Ensure that the sampling for this instrument is per session not per-pageview.

@phuedx: I reviewed the modeling approaches in PA team sharing (notes w/ link to transcript & recording) and Morten brought up a great point which is the computational cost of using virtual table generation (LATERAL VIEW EXPLODE() in Spark SQL and CROSS JOIN UNNEST() in Presto) for version 1 which reminded him of evaluating custom data approach of monoschema MP.

@phuedx: I see in the Figma designs that instruments/experiments will have user-provided unique names with machine-readable versions auto-generated. Will those instruments/experiments have numeric IDs in the database that also uniquely identify them?

Andrew brought up an excellent point (in the same thread) which is that https://wikitech.wikimedia.org/wiki/Event_Platform/Schemas/Guidelines#Complex_array_element_and_map_value_type_evolution_is_not_well_supported so if we did go the array-of-objects (version 1) route, we MAY want array<map<string,string>> not array<struct<id:int,name:string,group:string>> in case there is anything we would need to add later. Like, if we needed to include a new field for each experiment it'd be easy with the former and impossible with the latter.

That's a great point!

@phuedx @VirginiaPoundstone: T368326: Update Metrics Platform Client Libraries to accept experiment membership and just to confirm: yes I have bandwidth to consult on this

Sounds good, yeah! Thanks for closing the loop on this.

Oh cool! Yeah I'd love to squeeze that into scope. Should I make a separate task?

@Milimetric Once this is finalized can you please document this change at https://wikitech.wikimedia.org/wiki/Data_Platform_Engineering/Decision_log?

As an Experiment Owner, when I select a location(wiki) I need to see the default sample rate for that location(wiki), so I know whether or not I need to modify it according to my experimentation plan.
As Experiment Owner, I need to override the default sample rate at a per wiki level, so I can collect data at different rates based on the various wikis that are part of the experiment.
As an Experiment Owner, I need to set a default sample rate, so that rate will be applied across all locations(wikis) except the ones that I need to override with a different sampling rate.

Should there be standard sampling rates per wiki? Or a low conservative default sampling rate (.1%) for all?

I think that providing defaults for these values is part of the product. I also think that it would be better for our users (and for the platform!) if we provided a conservative default value that the user than can increase (with relatively little friction because MPIC is gonna be _awesome_!).

@MNeisler just caught me up on this. I just want to share some thoughts about

Kind of? Categories are…a giant unstructured, manually-maintained mess and inconsistent across languages. (In general, not just specific to templates.) Categories can also have layers upon layers of sub-categories nested within each other, and there's not an easy way to get them all.

All good! Thanks so much!

First, I want to acknowledge that I appreciate wanting to incorporate data into the design stage of product development cycle. Unfortunately we do not have capacity to take this request (or any request, really) for data analysis for the remainder of Q4.

@JWheeler-WMF Is this for FY24–25 PES 1.2.3?

Thank you for this analysis, KC!

@ngkountas @KCVelaga_WMF: Perhaps this will be useful: https://wikitech.wikimedia.org/wiki/Metrics_Platform/How_to/Create_An_Instrument#In_JavaScript

@CDanis: Can you please paste the Spark code / Spark SQL query you used for reproducibility?

@Krinkle: Thank you for sharing the results of your queries in a manner consistent with the Data Publication Guidelines.