Fortunately, all preference changes are already instrumented and that instrumentation is resilient to the addition/deletion of user preferences.

The WikimediaEvents extension is responsible for emitting a PrefUpdate event for every preference that the user changes.

• What is the retention of the feature (i.e. how many users opt-out on test wikis over time, how many users opt-in on test wikis over time)
• What is the average duration of opt-in status (so we can see which features might have more effects than others)

PrefUpdate events include timestamp, user ID, property name, and property value. Therefore, we can construct queries to determine:

• Opt-in/out rate
• Average/median/95 and 99th percentile duration of opt-in status

a) How often do users view the appearance tab in preferences? b) What percentage of these users continue to interact (whether opt-in or opt-out) with the desktop improvements option

a) To answer this question exactly would require adding new instrumentation to the Special:Preferences page.

b) We can determine the number of logged-in users that visited Special:Preferences in a given period by querying the wmf.webrequest Hive table; and we can determine the number of users that changed a preference in a given period by querying the event.PrefUpdate Hive table.

AIUI the answer to b is the lower bound answer to a.

• How do these changes vary based on the number of edits a given user has??

Since PrefUpdate events include user ID, we can determine the number of edits that the user has made during analysis.

However, since the user's edit count increases over time, we might want to update the instrumentation to include the bucketed user edit count. My recommendation would be to implement the edit count bucketing code in MediaWiki Core so that we can remove the independent implementations in the QuickSurveys and Popups codebases. It would be prudent to run the change by Product Analytics before the fact and also to announce the change after the fact.