Page MenuHomePhabricator

QA mobile wikitext fallback workflow instrumentation
Closed, ResolvedPublic

Description

This task is about verifying the events we'll have added in T243641 are being logged and stored as we expect them to.

Background

T239676 introduces a way (call it the "fallback workflow") for people seeking to edit in the visual mode on a mobile device to switch to the source editor in situations where the visual editor does not load within a to-be-defined period of time.

T243641 introduces event logging so we can know how people are engaging with this "fallback workflow."

And this task is about ensuring the event logging we'll have added in T243641 is working as we expect it to.

Event Timeline

LGoto triaged this task as Medium priority.Feb 6 2020, 9:08 PM

Confirmed from Peter that T243641 will be completed in Q3. So this QA ticket will be coming up this quarter but not immediately. Moving back to Q3 bucket on the PA-workboard.

A long-overdue comment confirming: the Editing Team considers this ticket to be low priority at this time. Thus, this ticket existing in Product-Analytics's backlog sounds good to us.

MNeisler lowered the priority of this task from Medium to Low.Feb 11 2021, 12:30 AM

Hi Megan, I was cleaning up my backlog and assigned this to you :)

ppelberg edited projects, added Editing-team (Tracking); removed Product-Analytics.
ppelberg moved this task from Backlog to Analytics on the Editing-team (Tracking) board.
MNeisler moved this task from Next 2 weeks to Needs Review on the Product-Analytics (Kanban) board.
MNeisler added a subscriber: DLynch.

I've reviewed the data logged in editattemptstep and visualeditorfeatureuse and confirmed the new events instrumented in T243641 are logging mostly as expected.

@DLynch Just one question/potential issue:
All init_mechanism = 'tooslow' events to date have been logged with init_type = section. Is this expected? Or should this init_mechanism type also occur for editing initiated for the whole page?

All other data appears to be logging as expected. See a summary of checks and some initial findings summarized below.

QA Check Summary
Note: Data reflects events logged from 1 November 2023 through 11 December 2023.

  • Confirmed we are logging both the context-show and fallback-confirm actions associated with the fallback prompt feature. There are more context show events than fallback confirm events as expected.
  • Users in about 5% of the editing sessions where the fallback prompt was shown elect to use the fallback editor.
  • About 1.3% of all edit attempts during the reviewed time period included a fallback prompt event.
  • Confirmed that fallback feature events only appear when the user is editing on mobile (platform = 'phone') and using VisualEditor (editor_interface = visualeditor)
  • The fallback prompt typically only appears once in an editing session. Only about 2% of sessions were shown the fallback prompt more than once.
  • I reviewed the order of events for a sample of editing sessions that included a fallback prompt event. The order of edit events appears as expected.
  • init_mechanism = 'tooslow' in editattemptstep only appears with wikitext and mobile editing sessions.
  • Confirmed that for all init_mechanism = 'tooslow' events, there is an immediately-preceding init event with the same page_token / session_token.
  • As expected, the median load time for sessions shown the fallback prompt (4.3 seconds) is significantly higher than the median load time for sessions not shown the prompt (1 second)

As part of this task, I've also updated the vefu data dictionary with the new events added to track use of this feature.

All init_mechanism = 'tooslow' events to date have been logged with init_type = section. Is this expected? Or should this init_mechanism type also occur for editing initiated for the whole page?

@MNeisler it's not strictly required, but because this is an event we should only be seeing on mobile it's really uncommon to see non-section edits in the first place. It's only been exposed in the UI recently, and you need the advanced editing tools enabled and then to go through the dropdown menu to choose the "edit full page" option. Given which it wouldn't surprise me at all if a complete lack of events is legitimate.

Thanks @DLynch.

I believe we can resolve task this now as events have been verified. Note further analysis of this data will be done in T347437