Page MenuHomePhabricator

Deploy pilot on enwiki for Global Readers Demographic Survey
Closed, ResolvedPublic

Description

Objective: recruit 1000 readers from English Wikipedia to take the Global Readers Demographic Survey. The pilot survey will be identical to the proposed final survey but will include an additional feedback question at the end and ideally only 1000. The results will be used to tweak any questions and sampling strategy.

When: in field September 22 - 26, 2023
Audience: Readers of English Wikipedia (global)
Coverage: 1:300 sampling ratio (but let's start with zero coverage as a test)

Survey Type: External hosted on Limesurvey
Survey Name: Global Readers Demographic Survey 2023 (ID 936187)

QS Message: "Take a short survey and help us improve Wikipedia"

Tasks

  • Pre-deploy survey
  • Create messages
  • Deploy survey
  • Disable survey

Tests

Survey
Messages

Event Timeline

@YLiou_WMF I've pre-deployed this survey with coverage set to 0. I followed the naming convention from T232525 so this is reader-demographics-2. We will be able to trigger the survey via URL parameters after the messages creation . At the planned date I will increase the coverage to the planned value effectively deploying the survey.

DDeSouza changed the task status from Open to In Progress.Sep 11 2023, 1:55 AM
DDeSouza updated the task description. (Show Details)
DDeSouza renamed this task from enwiki pilot quicksurvey deployment to Deploy pilot on enwiki for Global Readers Demographic Survey.Sep 11 2023, 2:00 AM

@DDeSouza I've created the mediawiki pages (but left the description page blank as with T232525)

@YLiou_WMF Ok. I had to create the page otherwise it would show the message key instead of blank.

Screenshot from 2023-09-11 19-06-24.png (194×295 px, 11 KB)

Screenshot from 2023-09-11 19-06-45.png (137×298 px, 11 KB)

DDeSouza updated the task description. (Show Details)

Hi all, a couple questions:

  1. should we be using the external Yes and No buttons
"ext-quicksurveys-external-survey-yes-button": "Visit survey"
"ext-quicksurveys-external-survey-no-button": "No thanks"

instead of:

'yesMsg' => 'ext-quicksurveys-example-internal-survey-answer-positive',
'noMsg' => 'ext-quicksurveys-example-internal-survey-answer-negative',

See current configuration:

'enwiki' => [
		// T344393
		[
			'enabled' => true,
			'type' => 'external',
			'name' => 'reader-demographics-2',
			'question' => 'reader-demographics-2-survey-question',
			'description' => 'reader-demographics-2-survey-description',
			'yesMsg' => 'ext-quicksurveys-example-internal-survey-answer-positive',
			'noMsg' => 'ext-quicksurveys-example-internal-survey-answer-negative',
			'coverage' => 0, // Increase to deploy.
			'platforms' => [
				'desktop' => [ 'stable' ],
				'mobile' => [ 'stable' ]
			],
			'link' => 'reader-demographics-2-survey-link',
			'privacyPolicy' => 'reader-demographics-2-survey-privacy-policy',
		],
	],
  1. @YLiou_WMF do you want to also target logged-in users, or only those who don't have an account? I ask because you'll likely pick up lots of editors if you don't have a specified target (who are of course readers, but possibly a very different type because they tend to spend much more time on-wiki and may have a higher probability of being sampled. One concern is bothering editors if they don't need to be bothered). Likewise, instead of choosing between logged in and not, you could set the max edits instead so it only targets users with less than a certain amount of edits.

@TAndic

  1. If I'm understanding correctly, yes I think so.
  2. No, I don't think we should target logged-in users.

Hi all, a couple questions:

  1. should we be using the external Yes and No buttons
"ext-quicksurveys-external-survey-yes-button": "Visit survey"
"ext-quicksurveys-external-survey-no-button": "No thanks"

instead of:

'yesMsg' => 'ext-quicksurveys-example-internal-survey-answer-positive',
'noMsg' => 'ext-quicksurveys-example-internal-survey-answer-negative',

See current configuration:

'enwiki' => [
		// T344393
		[
			'enabled' => true,
			'type' => 'external',
			'name' => 'reader-demographics-2',
			'question' => 'reader-demographics-2-survey-question',
			'description' => 'reader-demographics-2-survey-description',
			'yesMsg' => 'ext-quicksurveys-example-internal-survey-answer-positive',
			'noMsg' => 'ext-quicksurveys-example-internal-survey-answer-negative',
			'coverage' => 0, // Increase to deploy.
			'platforms' => [
				'desktop' => [ 'stable' ],
				'mobile' => [ 'stable' ]
			],
			'link' => 'reader-demographics-2-survey-link',
			'privacyPolicy' => 'reader-demographics-2-survey-privacy-policy',
		],
	],
  1. @YLiou_WMF do you want to also target logged-in users, or only those who don't have an account? I ask because you'll likely pick up lots of editors if you don't have a specified target (who are of course readers, but possibly a very different type because they tend to spend much more time on-wiki and may have a higher probability of being sampled. One concern is bothering editors if they don't need to be bothered). Likewise, instead of choosing between logged in and not, you could set the max edits instead so it only targets users with less than a certain amount of edits.

@DDeSouza can we ask for these changes? (buttons and audience coverage)

For 1: external buttons, we'll need to change:

			'yesMsg' => 'ext-quicksurveys-external-survey-yes-button',
			'noMsg' => 'ext-quicksurveys-external-survey-no-button',

and for 2: only targeting not-logged-in visitors we'll need (if I'm understanding https://www.mediawiki.org/wiki/Extension:QuickSurveys correctly, please help verify!)

	'audience' => [
		'anons' => true
                 ],

Change 956931 had a related patch set uploaded (by DDesouza; author: DDesouza):

[operations/mediawiki-config@master] Deploy Reader Demographics 2 pilot survey

https://gerrit.wikimedia.org/r/956931

@TAndic sure.

  1. I will just remove those lines as they are optional and "Visit survey" and "No thanks" are the default.
  1. That's right.

@DDeSouza Thank you!

Feels redundant but just to be extra-sure: the "Visit survey" and "No thanks" defaults source from "ext-quicksurveys-external-survey-yes-button" and "ext-quicksurveys-external-survey-no-button", right?
(I found itwiki is in English and want to make sure all of our languages get fixed in the right place)

@TAndic

the "Visit survey" and "No thanks" defaults source from "ext-quicksurveys-external-survey-yes-button" and "ext-quicksurveys-external-survey-no-button", right?

That's right.

I found itwiki is in English and want to make sure all of our languages get fixed in the right place

In this case we can create the missing translation at: https://translatewiki.net/wiki/MediaWiki:Ext-quicksurveys-external-survey-yes-button/it. It takes around 1-2 weeks for a translation to reach production.

FYI the survey uses the user interface language and not the wiki language. This is not a problem for anon users because the interface and wiki languages will be the same. But a user with interface language set to Portuguese, as an example, would see the custom messages in English and the default messages in Portuguese. e.g. https://en.wikipedia.org/wiki/Wikimedia_Foundation?quicksurvey=reader-demographics-2&uselang=pt

@YLiou_WMF Not possible to deploy on 22 Sep due to being a Friday so no deployment windows. Do you want me to deploy it earlier (Thursday) or later (Monday)?

Hi all, in case you're working on this early today please hold off for a few hours, I have a configuration to add and test but I need to consult first -- should be back to you shortly.

Sure @TAndic. I'm planning to deploy on Monday Sep 25.

@DDeSouza thank you so much for your help in figuring out the token linking from QuickSurvey!

I believe this should be the final configuration. Can we deploy with 0 coverage this week so we can make sure the linking for matching and data cleaning works before we increase the coverage next week? (We've never used it on LimeSurvey so it's that side that I'm more concerned about passing the token correctly before we're working with thousands of responses.)

'enwiki' => [
// T344393
	[
	'enabled' => true,
	'type' => 'external',
	'name' => 'reader-demographics-2',
	'question' => 'reader-demographics-2-survey-question',
	'description' => 'reader-demographics-2-survey-description',
	'coverage' => 0, // Increase to deploy.
	'audience' => [
		'anons' => true
	],
	'instanceTokenParameterName' => 'QS',
	'platforms' => [
		'desktop' => [ 'stable' ],
		'mobile' => [ 'stable' ]
	],
	'link' => 'reader-demographics-2-survey-link',
	'privacyPolicy' => 'reader-demographics-2-survey-privacy-policy',
	],
],

@fkaelin any chance you could assist with deploying this today so that we can check before full deployment next week?

@DDeSouza thank you so much for your help in figuring out the token linking from QuickSurvey!

I believe this should be the final configuration. Can we deploy with 0 coverage this week so we can make sure the linking for matching and data cleaning works before we increase the coverage next week? (We've never used it on LimeSurvey so it's that side that I'm more concerned about passing the token correctly before we're working with thousands of responses.)

'enwiki' => [
// T344393
	[
	'enabled' => true,
	'type' => 'external',
	'name' => 'reader-demographics-2',
	'question' => 'reader-demographics-2-survey-question',
	'description' => 'reader-demographics-2-survey-description',
	'coverage' => 0, // Increase to deploy.
	'audience' => [
		'anons' => true
	],
	'instanceTokenParameterName' => 'QS',
	'platforms' => [
		'desktop' => [ 'stable' ],
		'mobile' => [ 'stable' ]
	],
	'link' => 'reader-demographics-2-survey-link',
	'privacyPolicy' => 'reader-demographics-2-survey-privacy-policy',
	],
],

Thank you @DDeSouza! Sorry for the mixup, just wanted to make sure we had time to test tomorrow!

Change 959826 had a related patch set uploaded (by DDesouza; author: DDesouza):

[operations/mediawiki-config@master] Deploy Reader Demographics 2 pilot survey

https://gerrit.wikimedia.org/r/959826

Change 956931 merged by jenkins-bot:

[operations/mediawiki-config@master] Update Reader Demographics 2 pilot survey

https://gerrit.wikimedia.org/r/956931

Mentioned in SAL (#wikimedia-operations) [2023-09-21T20:06:04Z] <brennen@deploy2002> Started scap: Backport for [[gerrit:956931|Update Reader Demographics 2 pilot survey (T345951)]]

Mentioned in SAL (#wikimedia-operations) [2023-09-21T20:17:17Z] <brennen@deploy2002> dani and brennen: Backport for [[gerrit:956931|Update Reader Demographics 2 pilot survey (T345951)]] synced to the testservers mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)

@TAndic Deployment done. Will be live soon.

I tested it and this was the link: https://wikimediafoundation.limesurvey.net/936187?lang=en&QS=85d0b74471846451641d

Mentioned in SAL (#wikimedia-operations) [2023-09-21T20:27:40Z] <brennen@deploy2002> Finished scap: Backport for [[gerrit:956931|Update Reader Demographics 2 pilot survey (T345951)]] (duration: 21m 36s)

Change 959826 merged by jenkins-bot:

[operations/mediawiki-config@master] Deploy Reader Demographics 2 pilot survey

https://gerrit.wikimedia.org/r/959826

Mentioned in SAL (#wikimedia-operations) [2023-09-25T20:03:57Z] <cjming@deploy2002> Started scap: Backport for [[gerrit:959826|Deploy Reader Demographics 2 pilot survey (T345951)]]

Mentioned in SAL (#wikimedia-operations) [2023-09-25T20:15:54Z] <cjming@deploy2002> cjming and dani: Backport for [[gerrit:959826|Deploy Reader Demographics 2 pilot survey (T345951)]] synced to the testservers mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)

Mentioned in SAL (#wikimedia-operations) [2023-09-25T20:25:15Z] <cjming@deploy2002> Finished scap: Backport for [[gerrit:959826|Deploy Reader Demographics 2 pilot survey (T345951)]] (duration: 21m 18s)

@DDeSouza let's plan to undeploy this survey at the earliest opportunity next week (starting October 2) as we're nearly at our target of 1000 completes

Change 962105 had a related patch set uploaded (by DDesouza; author: DDesouza):

[operations/mediawiki-config@master] Undeploy Reader Demographics 2 pilot survey

https://gerrit.wikimedia.org/r/962105

Change 962105 merged by jenkins-bot:

[operations/mediawiki-config@master] Undeploy Reader Demographics 2 pilot survey

https://gerrit.wikimedia.org/r/962105

Mentioned in SAL (#wikimedia-operations) [2023-10-02T20:18:25Z] <kindrobot@deploy2002> Started scap: Backport for [[gerrit:962105|Undeploy Reader Demographics 2 pilot survey (T345951)]], [[gerrit:962629|DiscussionTools: Disable timestamp links in production initially]]

Mentioned in SAL (#wikimedia-operations) [2023-10-02T20:19:46Z] <kindrobot@deploy2002> esanders and dani and kindrobot: Backport for [[gerrit:962105|Undeploy Reader Demographics 2 pilot survey (T345951)]], [[gerrit:962629|DiscussionTools: Disable timestamp links in production initially]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

DDeSouza updated the task description. (Show Details)

Mentioned in SAL (#wikimedia-operations) [2023-10-02T20:27:15Z] <kindrobot@deploy2002> Finished scap: Backport for [[gerrit:962105|Undeploy Reader Demographics 2 pilot survey (T345951)]], [[gerrit:962629|DiscussionTools: Disable timestamp links in production initially]] (duration: 08m 49s)

Change 973784 had a related patch set uploaded (by DDesouza; author: DDesouza):

[operations/mediawiki-config@master] Deploy Reader Demographics 2 survey

https://gerrit.wikimedia.org/r/973784

Change 973784 merged by jenkins-bot:

[operations/mediawiki-config@master] Deploy Reader Demographics 2 survey

https://gerrit.wikimedia.org/r/973784

Mentioned in SAL (#wikimedia-operations) [2023-11-13T14:31:11Z] <urbanecm@deploy2002> Started scap: Backport for [[gerrit:973784|Deploy Reader Demographics 2 survey (T345951)]], [[gerrit:973788|Add mediamoderation_scan table (T350321)]]

Mentioned in SAL (#wikimedia-operations) [2023-11-13T14:32:28Z] <urbanecm@deploy2002> urbanecm and dani: Backport for [[gerrit:973784|Deploy Reader Demographics 2 survey (T345951)]], [[gerrit:973788|Add mediamoderation_scan table (T350321)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2023-11-13T14:40:25Z] <urbanecm@deploy2002> Finished scap: Backport for [[gerrit:973784|Deploy Reader Demographics 2 survey (T345951)]], [[gerrit:973788|Add mediamoderation_scan table (T350321)]] (duration: 09m 13s)

Change 973814 had a related patch set uploaded (by DDesouza; author: DDesouza):

[operations/mediawiki-config@master] Fix Reader Demographics 2 survey

https://gerrit.wikimedia.org/r/973814

Change 973814 merged by jenkins-bot:

[operations/mediawiki-config@master] Fix Reader Demographics 2 survey

https://gerrit.wikimedia.org/r/973814

Mentioned in SAL (#wikimedia-operations) [2023-11-13T21:06:28Z] <urbanecm@deploy2002> Started scap: Backport for [[gerrit:973870|Enable edit check on swwiki (T350921)]], [[gerrit:973814|Fix Reader Demographics 2 survey (T345951)]]

Mentioned in SAL (#wikimedia-operations) [2023-11-13T21:07:46Z] <urbanecm@deploy2002> dani and kemayo and urbanecm: Backport for [[gerrit:973870|Enable edit check on swwiki (T350921)]], [[gerrit:973814|Fix Reader Demographics 2 survey (T345951)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2023-11-13T21:16:43Z] <urbanecm@deploy2002> Finished scap: Backport for [[gerrit:973870|Enable edit check on swwiki (T350921)]], [[gerrit:973814|Fix Reader Demographics 2 survey (T345951)]] (duration: 10m 15s)

@DDeSouza we are now planning to increase coverage ratio for the enwiki survey to 1:50 (0.02), as documented here. This patch can be scheduled for tonight (November 14). We will hold off on increasing the coverage ratio for ALL OTHER projects for now (with the possible exception of hewiki - we will have an update on this tomorrow).