Methodology

We defined the metrics as the ratio of the number of unique pageview sessions with non-default settings to the total number of unique pageview sessions that visited our site on a given day. We reviewed the minimum, maximum, average daily ratios, and the standard deviation for each individual feature as well as for the overall.

Our analysis uses data starting from October 20, 2023, onward, excluding NULL events in the baseline calculations. We operate on the assumption that the opt-out rate for NULL sessions is similar to that of other sessions. Our Data QA (https://phabricator.wikimedia.org/T346978) confirmed that the presence of NULL values in newly added fields, which had been unexpected, was addressed and resolved on October 18, 2023.(https://phabricator.wikimedia.org/T346106) While we have observed a notable reduction in NULL events on October 18 and 19, 2023, it may take some time for them to completely phase out.

Summary

% of pageview sessions which do not use page previews in the Vector 2022 skin

Metric: Number of unique sessions with preview off (non-default)/ total number of unique initialized sessions
The following statistics are based on the data collected between Oct. 20, 2023 and Oct. 25, 2023

% of pageview sessions which do not use the default width option (and view the site at full width)

Metric: Number of unique sessions with full_width on (non_default)/ total number of unique initialized sessions
The following statistics are based on the data collected between Oct. 20, 2023 and Oct. 25, 2023,

% of pageview sessions which have disabled media viewer

Metric: Number of unique sessions with media viewer disabled (non-default) / total number of unique initialized sessions
The following statistics are based on the data collected between Oct. 20, 2023 and Oct. 25, 2023,

Overall % pageview sessions which enabled non-default settings (disabled preview, or enabled full_width, or disabled media viewer)

Metric: Number of unique sessions with non-default settings (disabled preview, or enabled full_width, or disabled media viewer) / total number of unique initialized sessions
The following statistics are based on the data collected between Oct. 20, 2023 and Oct. 25, 2023,

Next steps

Calculate below baselines once the data for the adjusted instrumentation has been gathered.

• % of pageview sessions which have set a non-default font size in the Minerva skin (On mobile web)
• % of pageview sessions which a viewing a pinned/unpinned version of the menus on the site (main menu, table of contents, tools menu)

Notebook: link

@ovasileva, Please take a look at the summary of the baseline analysis above, let me know if you have any questions.

@ovasileva , here is the baseline collection for font size on mobile web. Let me know if you have any questions.

Summary

% of pageview sessions which have set a non-default font size in the Minerva skin (On mobile web)

Metric: Number of unique sessions with regular font size disabled (non-default) / total number of unique initialized sessions
The following statistics are based on the data collected between Nov.4, 2023 and Nov.8 , 2023 (incomplete date) ,

Preference by font size

• For logged-in users, most users with non-default font sizes opt for small.
• For anonymous users, most users with non-default font sizes opt for large and xlarge.

Notebook: link

@ovasileva ,please see the analysis of pin rate and overall non-default rate below.

Methodology

The event.is_pinned field in the desktopwebuiactionstracking schema is logged as true if the user has pinned either the main menu or tools menu upon initializing the page.

For anonymous users with a viewport size larger than 1000px, both the main menu and tools menu are default to being unpinned. Conversely, for logged-in users with a viewport size larger than 1000px, the main menu and tools menu are pinned by default. If the user's viewport is smaller than 1000px, the main menu and tools menu are automatically unpinned for both anonymous and logged-in users. Given that the main menu and tools menu are automatically unpinned when the user's screen size is smaller than 1000px, regardless of the user's settings, sessions with a viewport smaller than 1000px were excluded from the analysis.

Considering the distinct default values for logged-in and anonymous users, we calculate the pin rate and adjust it to the non-default rate based on their respective default settings.

Summary

% of pageview sessions which are viewing a pinned version of the menus (main menu, tools menu), only considering viewport sizes larger than 1000px

Metric: Number of unique sessions viewing pinned version of the menus on the site (main menu, tools menu) / total number of unique initialized sessions with a viewport size larger than 1000px

The following statistics are based on the data collected between Nov. 22, 2023 and Nov. 28, 2023,

% of pageview sessions which are viewing a non-default version of the menus on the site (main menu, tools menu), only considering viewport size larger than 1000px.

Metric: Number of unique sessions viewing a non-default version of the menus on the site (main menu, tools menu) / total number of unique initialized sessions with a viewport size larger than 1000px. For logged-in users with a viewport size larger than 1000px, the main menu and tools menu are pinned by default. Conversely, for anonymous users with a viewport size larger than 1000px, the main menu & tools menu are unpinned by default.

The following statistics are based on the data collected between Nov. 22, 2023 and Nov. 28, 2023.

Overall % pageview sessions which enabled non-default settings, only viewport > 1000px

Metric: Number of sessions with non-default settings / total number of initialized sessions. (viewport > 1000px). Non-default settings are either disabled preview, or enabled full_width,or disabled media viewer, or unpinned menu for logged-in users, or pinned menu for anonymous users.

The following statistics are based on the data collected between Nov. 22, 2023 and Nov. 28, 2023.

Notebook: link

The first two seem to be the same data, shown two ways.

The last query is interesting, could it be broken down into which of the four defaults (preview, width, media viewer, menu pinning) were changed?

@Sj, please see the broken down of preview, width and media viewer at T346979#9285473.

Thank you @jwang! I missed that.

On my macOS (chrome/ff) the width toggle doesn't appear for windows under 1400px wide — do you have this data on width settings for viewports > 1400px, or an estimate of what fraction of viewports > 1000px are also >1400px?

Hi @ovasileva , when I analyzed the data based on the viewport size buckets, I noticed that the preview disable rate in the 1000px-1199px group was significantly higher than in the adjacent bucket groups for both anonymous users and logged-in users. Is there any specific reason for this?

Below table shows percentage of pageview sessions which do not use previews in the Vector 2022 skin by viewport size buckets.

*The statistics above are based on the data collected between Nov. 22, 2023 and Nov. 28, 2023 from all wikis.

A few additional findings:

• The preview disable rate is also high in the <320px group, which is likely due to small screen users preferring to have more space for content rather than a preview.
• 12% of pageview sessions are in the 1000px-1199px group

Hello @Sj, we only collect data on the viewport size buckets, and these are segmented into six groups. Unfortunately, 1400px is not the threshold to divide groups. I hope the data below still provides insights into how full-width preference varies in each group.

Below table shows the distribution of pageview sessions within each group.

• 80% of pageview sessions have a viewport size larger than 1200px.

The table below shows the full width enable rate based on viewport size buckets.

• In general, the full width enable rate increases as the viewport increases from 1000px to larger sizes.

*The statistics above are based on the data collected between Nov. 22, 2023 and Nov. 28, 2023 from all wikis.

@ovasileva, @Jdlrobson, I have reran the analysis using the recent data as we discussed.

Summery of the results:

• Still see the similar trend: The preview disable rate in the 1000px-1199px group was significantly higher than in the adjacent bucket groups for both anonymous users and logged-in users.
• The preview disable rate is higher than before, reflecting the fix for T352073. For context, previously, due to the bug in T352073, the is_page_preview_on field would be logged as false when it was actually true in some cases.

Data
Below table shows percentage of pageview sessions which do not use previews in the Vector 2022 skin by viewport size buckets.

*The statistics above are based on the data collected between Dec 21, 2023 and Dec, 31 , 2023 from all wikis.

Query

query <- "
SELECT TO_DATE(dt) AS event_date, event.action , event.skin, event.isAnon, event.viewportSizeBucket, event.is_page_preview_on, event.is_full_width, event.is_media_viewer_enabled,event.is_pinned, 
count(Distinct event.token) AS count_session, count(1) AS events
FROM event.desktopwebuiactionstracking
WHERE year=2023 AND  month=12 
AND day BETWEEN 21 AND 31
AND event.is_page_preview_on IS NOT NULL
AND event.is_full_width  IS NOT NULL
AND event.is_media_viewer_enabled  IS NOT NULL
AND event.is_pinned  IS NOT NULL
AND event.action='init' AND event.skin='vector-2022'
AND NOT useragent.is_bot
GROUP BY  TO_DATE(dt), event.action, event.skin, event.isAnon, event.viewportSizeBucket, event.is_page_preview_on, event.is_full_width, event.is_media_viewer_enabled, event.is_pinned
ORDER BY  event.isAnon, event_date, event.viewportSizeBucket, event.is_page_preview_on, event.is_full_width, event.is_media_viewer_enabled, event.is_pinned
LIMIT 10000
"

I have further investigated the preview disable rate in 1000px-1199px viewport bucket, analyzing it by device families and wikis.
In summary, the high preview disable rate in the 1000px-1199px bucket is influenced primarily by devices in the Mac family, specifically those running Mac OS X with the version details: os_major 10 and os_minor 15.

By device family

• 6.5% sessions are from 349 device families, which only have preview off events. They are likely to be touch devices. As touch devices do not have a mouse, page previews is disabled for touch devices.
• 62% of sessions are from Mac device families, which has a high preview disable rate of 93%.
• Among the mac device families in this bucket, the majority run Mac OS X, os_major 10, os_minor 15 version.

Below table shows the preview disable rate by device family based on the data collected on 2024-01-01 8am-8:59am across wikis .

By wikis

• The preview disable rate is high for Mac family across wikis in the 1000px-1199px viewport bucket

In T346979#9457398, @jwang wrote:

I have further investigated the preview disable rate in 1000px-1199px viewport bucket, analyzing it by device families and wikis.
In summary, the high preview disable rate in the 1000px-1199px bucket is influenced primarily by devices in the Mac family, specifically those running Mac OS X with the version details: os_major 10 and os_minor 15.

By device family

• 6.5% sessions are from 349 device families, which only have preview off events. They are likely to be touch devices. As touch devices do not have a mouse, page previews is disabled for touch devices.
• 62% of sessions are from Mac device families, which has a high preview disable rate of 93%.
• Among the mac device families in this bucket, the majority run Mac OS X, os_major 10, os_minor 15 version.

Below table shows the preview disable rate by device family based on the data collected on 2024-01-01 8am-8:59am across wikis .

Device family	Number of sessions with preview off	Number of sessions with preview on	Total sessions	Preview disable rate	Note
Mac	20019	1527	21546	92.91%	Why it's high on Mac? Mainly Mac OS X; os_major: 10; os_minor:15.
Other	1072	10049	11121	9.64%
Other 349 types of device families	2280	0	2280	100%	They are light to be touch devices

By wikis

• The preview disable rate is high for Mac family across wikis in the 1000px-1199px viewport bucket

This seems to be due to iPad's default to the desktop site (but previews still unavailable). @jwang - Perhaps the best thing to do here would be to only consider devices with > 1200px for the desktop milestone. What do you think?

Perhaps the best thing to do here would be to only consider devices with > 1200px for the desktop milestone. What do you think?

Agree. I will re-collect the baseline for devices with viewport > 1200px only.

Here are the baselines for devices with a viewport larger than 1200px. @ovasileva , let me know if you have any questions.

Preview disable rate (viewport > 1200px)

Metric: Number of unique sessions with preview off (non-default)/ total number of unique initialized sessions (viewport > 1200px).
The following statistics are based on the data collected between Dec. 21, 2023 and Dec. 31, 2023

Overall non-default rate (viewport > 1200px)

Metric: Number of sessions with non-default settings / total number of initialized sessions. (viewport > 1200px). Non-default settings are either disabled preview, or enabled full_width,or disabled media viewer, or unpinned menu for logged-in users, or pinned menu for anonymous users.
The following statistics are based on the data collected between Dec. 21, 2023 and Dec. 31, 2023

Notebook: link