Wikipedia:WikiProject User warnings/Testing

WP:UWTEST

This page is the tracking page for efforts on the English Wikipedia to improve the quality of user talk warnings. Links to this project on other Wikipedias can be found on the cross-wiki hub on Meta.

Scope

The purpose of this page is to measure the efficacy of different user talk templates through randomized testing. This effort does not entail creating new categories of user warnings or to organize them more efficiently, but rather to improve on the quality of our current communication methods.

We're aiming to fine tune the template messages we send to editors, in order to encourage more good faith contributors and discourage outright vandals, spammers, and other bad faith contributors.

Participants

Sign up here if you'd like to stay notified about testing updates.

User:Steven (WMF)
User:Maryana (WMF)
User:Staeiou
User:EpochFail
User:Vitor Mazuco
User:Philippe (WMF)
User:Kudpung
User:Kubigula
User:DGG
User:Fluffernutter
User:Wikipelli
Ebe123
Σ
User:The Blade of the Northern Lights
User:Writ Keeper
Eraserhead1 <talk> 21:53, 25 October 2011 (UTC)[reply]
User:Jtmorgan
User:Chzz
Hurricanefan25
Ljhenshall
XLinkBot (forced by its operators)
Beetstra (as one of the operators of XLinkBot)
Andy Mabbett (Pigsonthewing)
User:Racconish
User:Recon Etc
User:Jayen466
User:gwickwire ^(talk)
User:Thogo
User:Logan
User:Snowolf
User:Johannes Rohr (WMDE)
User:mabdul (AFC related)
Katarighe
Martijn Hoekstra (talk)
User:Gaijin42
User:DMacks
User:Rcsprinter
User:JohnCD
User:Ciphers
Nathan2055^{talk - contribs} 19:49, 27 June 2012 (UTC)[reply]
Rich Farmbrough, 02:49, 15 September 2012 (UTC).[reply]
User:Lexein
User:Rdicerb_(WMF)

Tasks

Any effort to make templates more simple, friendly, and accessible is welcome. Specifically, you can:

Draft new templates or assess the templates that are currently in the draft phase and suggest improvements in their content or structure. We aim to lessen the bitey-ness of warnings, but templates should still comply with the design guidelines and the usage and layout best practices.
Suggest which templates to test next and what changes should be made.
Help us analyze our data. We're using mixed methods, with both quantitative statistics and qualitative coding.
Recruit more participants.

Tests

Templates have been tested in the following places:

Those pages each have lists and dates of all the templates tested including results if available. The following are tracking tables with start and end dates. For tests in projects other than English Wikipedia, please see our hub on Meta.

Test name	Brief description	Date started	Date ended	Duration	Tracking templates	Data (if available)
Huggle 1	Level 1 vandalism warning. 3 test templates (image/no image; teaching, personal) and 1 control	July 19, 2011	August 5, 2011	~2 weeks	{{z49}} {{z50}} {{z51}} {{z52}} {{z53}} {{z54}} {{z55}} {{z56}}
Huggle 2	Level 1 vandalism warning. 2 test templates (personal, personal no directives) and 1 control	September 25, 2011	October 10, 2011	~2 weeks	{{z57}} {{z58}} {{z59}}
Huggle 3	All level 1 warnings (except vandalism), 9 total. 1 test template (personal no directives) and 1 control	October 18, 2011	November 19, 2011	1 month	{{z60}} {{z61}} {{z62}} {{z63}} {{z64}} {{z65}} {{z66}} {{z67}} {{z68}} {{z69}} {{z70}} {{z71}} {{z72}} {{z73}} {{z74}} {{z75}} {{z76}} {{z77}}
Twinkle 1	AfD and PROD notices, 2 total. 1 test template (personal no directives) and 1 control	November 9, 2011	December 9, 2011	1 month	{{z78}} {{z79}} {{z81}} {{z82}}
Huggle short 1	Level 1 vandalism warning. 2 test templates (personal no directives, short) and 1 control	November 8, 2011	December 9, 2011	1 month	{{z84}} {{z85}} {{z86}}
XLinkBot	Level 1-4 spam warnings, 4 total. 1 test template (personal no directives) and 1 control; Welcomeg and welcomeanon, 1 test template (personal no directives) and 1 control	November 17, 2011	December 17, 2011	1 month	spam 1-4: {{z87}} {{z88}} {{z89}} {{z90}} {{z91}} {{z92}} {{z93}} {{z94}}; welcomes: {{z95}} {{z96}} {{z97}} {{z98}}
Huggle short 2	All issue-specific level 1 warnings, 9 total. 1 test template (short) and 1 control	November 22, 2011	December 22, 2011	1 month	{{z99}} {{z100}} {{z101}} {{z102}} {{z103}} {{z104}} {{z105}} {{z106}} {{z107}} {{z108}} {{z109}} {{z110}} {{z111}} {{z112}} {{z113}} {{z114}} {{z115}} {{z116}}
SDPatrolBot	Newbie CSD tag removal warning, 1 test template (personal no directives) and 1 control	November 3, 2011	January 3, 2012	2 months	—	Wikitable
Shared IP archiving	archiving of shared and dynamic IP talk pages regularly to remove irrelevant warnings	December 19, 2011	February 19, 2012	2 months	—	List of all archived talk pages, SharedIPArchiveBot's log
ImageTaggingBot	Image notifications about sourcing/licensing, 4 total. 2 test templates (directives, no directives) and 1 control.	December 21, 2011	February 21, 2012	30 days (minimum)	{{z131}} {{z132}} {{z133}} {{z134}} {{z135}} {{z136}} {{z137}} {{z138}} {{z139}} {{z140}} {{z141}} {{z142}}
CorenSearchBot	Copyvio notifications, 6 total. 1 test template (personal no directives) and 1 control; welcomelaws, 1 test (short) and 1 control.	January 12, 2012	January 31, 2012	1 month	{{z117}} {{z118}} {{z119}} {{z120}} {{z121}} {{z122}} {{z123}} {{z124}} {{z125}} {{z126}} {{z127}} {{z128}}
28bot	Test edit warning, registered and unregistered, 2 total. 1 test template (personal no directives) and 1 control	January 19, 2012	March 27, 2012		{{z143}} {{z144}} {{z145}} {{z146}}

Testing method

The following are the requirements to conduct comparative A/B testing of any user talk template. Doing randomized experiments allows us to get hard data about what kinds of content are most successful at helping us achieve these goals. What you'll need is...

A "randomizer" that delivers all the templates in your test. This is the template that should be included in the configuration of whichever bot or tool you are testing with, and it randomly delivers one of the templates via a parser function.
A control, usually the existing default template. Note that you should replicate the default in a new template rather than use the current template page, in order to avoid including old instances of the default in your experiment.
A new version or versions of the template you want to test. Try to use a canonical name that matches the type, purpose, and level of the warning you're interested in.
A Z number tracking template for all templates being tested. If you do not include a separate Z number in each template, you will lose track of your test cases once they are substituted.

In some cases, such as for bots where all contribs are by one account, this method can be greatly simplified.

Analysis

We have so far used a mixed method of both quantitative measurement and qualitative assessment. If you'd like to help sort and analyze tests, please sign up above.

Testing results from all projects are available here.