Research:New Page Patrol survey/WMF report

This is the authoritative version of the Foundation report — it has also been copied over to the English-language Wikipedia

Editors have, over the last six months, made clear that New Page Patrol is a substantial “problem area” for the community. Patrollers complain of being overworked, while other editors raise issues with the patrollers themselves, stereotyping them as young, poorly educated and immature volunteers who are ignorant of the rules surrounding deletion.

To try and correct the overwork problem, the Foundation is designing a new New Page Patrol interface, “Zoom”. Doing this properly requires a lot of information about patrollers, however; who are they, what do they do, how do they do it, and what do they see as the problems with the status quo? Kudpung proposed running a survey of patrollers, firstly to help with developing Zoom, and also to gather data for his own ideas about community-driven solutions or improvements.

Having gathered this data, we can confirm that the common stereotype of patrollers as young, poorly educated and ignorant are almost entirely without basis. The vast majority of patrollers are over 18 and have undergraduate degrees or above - in some cases, actually exceeding the average for editors overall set in the April 2011 survey. They are largely familiar with relevant policies, and greatly exceed the expectations set by the stereotype. This information provides a useful factual basis to begin developing the Zoom interface, and will be taken into account when designing the software.

Patroller suggestions for improving New Page Patrol have also proven very interesting; many are already being incorporated into Zoom, or will be taken into consideration. Those that are cultural rather than technical in nature should be investigated by the community. Those that are technical will be discussed as part of the development process for the New Page Triage feature. It is worth pointing out that there were a few problems with the survey largely due to the “hybrid” nature of the survey; a community-sourced proposal, with the Foundation relying on the results. We are confident these problems have not impacted on the validity of the data, and fully endorse its reliability.

Background

New Page Patrol

On the English-language version of Wikipedia, New Page Patrol is an editing activity that involves (as the name suggests) patrolling newly created pages. This can be talkpages, userpages or any other kind of standalone entry, but is mostly undertaken in relation to articles, created by both new and experienced editors.

This patrolling is most commonly done through Special:NewPages, a special page built into the MediaWiki software that provides a log of all newly created articles. These articles can be read by editors and checked for problems; if there are no problems, the editor simply marks the article as “patrolled”, which removes it from the log, and moves on. If there are minor problems, maintenance templates are added (or fixes are attempted) before the “patrol” button is clicked, while articles with major (deletion-worthy) problems, such as inherently non-notable pages, pages that attack living subjects and pages that constitute copyright violations, can be marked with a deletion template before being patrolled. In all cases, the “patrol” action removes the page from the standard log.

This log lasts for 30 days; once an article is more than a month old, it falls off the list and cannot be found through Special:NewPages. This creates a constant and pressing need to keep patrolling articles, since if problematic articles are left unpatrolled, they may be automatically removed from the list, making it exceedingly difficult to find them again. As a result, those people who work on Special:NewPages (New Page Patrollers) have justifiably complained about being overworked and under large amounts of stress. These patrollers are, however, the subject of complaints themselves. Critics contend that patrollers tend to deal with new users inappropriately, scaring them of, that they have an unacceptably high error rate when tagging pages for deletion, and that they are more interested in using New Page Patrol as a route to gaining higher userrights than in actually patrolling productively and helping improve new articles.

This criticism has gained some traction, and there are two theories as to where it comes from. The first is that patrollers are uniformly (or overwhelmingly) younger and less experienced than the average editor, and as such are more likely to act immaturely or without foresight when dealing with new editors or new articles. The second is that irascibility and poor judgment is more likely to come from overwork and stress than it is to come from the personal circumstances of the patrollers.

The first theory is something we've heard from quite a few editors, including patrollers themselves. This is that patrollers are very young, poorly educated, largely American editors whose focus is primarily on patrolling, anti-vandalism work and deleting content. Their motivation is to allow for their own advancement, specifically so they can become administrators, and they undertake their tasks without the requisite regard for the new users they are dealing with and largely through automated tools.

Proposed trial restriction of new page creation

In response to the issue of stress, a group of editors proposed curtailing the ability of new users to create articles. Instead of anyone with an account being permitted to create an article (which itself is a restriction specific to the English language Wikipedia), new page creation would be restricted to those with the “autoconfirmed” user right. Users without this permission would be required to go through the "Articles for Creation" process instead.

The initial proposal was for a 6 month-long trial of these restrictions, known as the Autoconfirmed Article Creation Trial, or ACTRIAL. When this proposal was put to the developers, the technical changes needed were denied.

The Wikimedia Foundation argued that "creating a restriction of this type is a strong a statement of exclusion, not inclusion, and that it will confuse and deter good faith editors." Instead, WMF staff proposed working in partnership with the community to explore alternatives to managing the new page patrolling backlog and welcoming constructive good faith editors into the community. This response met significant controversy.

The Zoom interface and other improvements

After ACTRIAL and the voicing of other concerns, the Foundation began actively investigating ways to improve “new page triage”; the way user users submit and experienced editors review new content. One focus was the Special:NewPages interface and underlying software, with the intention of making patrolling easier. The result is the (provisional) Zoom interface, a new design for Special:NewPages that the Foundation hopes will do two things. Firstly, it will reduce the work involved in patrolling new pages by simplifying and smoothing out the process. This will make it easier for existing patrollers, and hopefully means they are less likely to snap at new editors or incorrectly mark pages due to the stress of dealing with the software. Secondly, the hope is to design a system that is self-explanatory for newer editors; someone who hasn't done New Page Patrol before can look at it and gain an instinctive understanding of what they're expected to do and how they should work. This should make it easier to attract more patrollers, reducing the workload and stress by distributing it between more people. The development team at the Foundation will not be able to address all the issues surrounding New Page Patrol, and will need to be disciplined about working on the features that provide the most benefit for the development effort required.

A change to the software under consideration for the Zoom interface (and one users have requested) is a “patroller” user right, which would govern who can and cannot patrol pages. This would mean that users who patrol or tag articles incorrectly, or users who treat article creators inappropriately, can be technically prohibited from patrolling until they demonstrate better judgment or tact. Whether this user right will be implemented has yet to be determined.

Any attempt to tweak the software has to start with looking at the methodology – how editors use the existing interface. Screencasts submitted by editors revealed that a lot of people have wildly different ways of approaching patrolling. Deficiencies in the current software has led to a host of third-party pieces of software, each with different features, focuses and interfaces, making it difficult to pin down any one “method” of patrolling. Additionally, whether the proposed changes will be useful depends largely on which of the two theories about the problems with New Page Patrol is more correct. Are patrollers doing things incorrectly because they are stressed and overworked, or because they, as a group, are (from their background) simply more likely to make rash decisions?

User:Kudpung, a user on the English-language version of Wikipedia, approached the Foundation asking for technical and legal support for a survey, which would be run to answer just these questions. The aims of the survey were to find out:

What patrollers themselves think needs improving and fixing, in software terms;
How patrollers patrol articles;
Why patrollers patrol articles;

And then use this data to make more informed software decisions, both in relation to the Zoom interface and the patroller user right.

The data is also being used to consider the validity of the patroller characterizations, which may impact on software design. Kudpung also intends for the data to be used to (possibly) develop training for new page patrollers; this may link into how the software is designed and used, but is not directly within the Engineering department's remit.

Carrying out the survey

Targeting

Targets for the survey were identified in one of three ways; first, editors who Wikimedia's public logs showed had patrolled more than 10 pages in the last 12 months, found via a script run by Snottywong. Second, editors publicly identified as new page patrollers with a “new page patrol” userbox, and lastly editors who publicly identified with a “this user uses Twinkle to patrol new pages” userbox. The total list came to 3,937 users, excluding blocked editors. Of the non-blocked users:

2,504 came from Snottywong's list of patrollers;
1,300 self-identified with a “new page patroller” userbox;
133 self-identified with a “this user uses Twinkle...” userbox.

Those users who fell into multiple categories (for example, who had both userboxes) were only counted once.

Interpretation

Of the users asked to fill in the survey, 1,255 chose to do so. Of these, 230 were removed as too incomplete to be useful, clear duplications, or obviously inaccurate results (the 10 year old from Africa with a PhD being a classic example) leaving us with 1,022 results. It was then discovered that Snottywong's initial script inadvertantly included both users who had patrolled more than 10 pages in 2011, and also anyone who had created more than 10 new pages in that period (including, say, talkpages) while in possession of the "autopatrolled" userright. Similarly, it would also include those who had patrolled 1 page while creating 9, 2 while creating 8, so on and so forth.

Because of the absence of usernames for most respondents, this error could not be entirely corrected. For those participants who had provided usernames it was possible to correctly identify the number of patrol actions they had undertaken and eliminate those who had done none, or very few. Those users who had undertaken very few patrols numbered 43, leaving us with 979. However, with the inability to check the number of patrols that those participants without usernames had undertaken, we were forced to exclude them entirely - this resulted in the removal of 665 names. A further check, removing those users more than two standard deviations out from the norm and a couple of clearly incorrect entries (again, 10 year olds with PhDs) led to five more removals, resulting in a final pool of 309 survey entries.

While this is noticeably smaller than the initial pool of potential respondents (3,937) it is important to note that, as said, errors in gathering this pool made it noticeably larger than the actual number of patrollers. This means that the 309 entries represent a greater proportion of patrollers overall than is readily apparent. In addition, the data from those 309 respondents was compared to the data from the pool of 1,022, with no great statistical variation, reinforcing the position that these 309 respondents adequately represent new page patrollers as a whole - or at least, adequately represent those patrollers who chose to respond to the survey. The data will be made available to anyone who wants to undertake further research or check our sanitisation.

Analysis was undertaken on both the sanitised group of 309 users, and the top quartile of users by patrols over time as an editor.

Results

Distribution of patrols

A graph of the number of patrols per patroller shows a very prominent “long tail”; the majority of patrollers tend not to patrol many articles, leaving most of the work to a tiny minority of editors. This shows a clear source for the self-reported stress that dedicated patrollers have, with 89 percent of the patrols done by the top 25 percent of editors. We clearly need to make involving more users, and involving patrollers to a greater degree, a priority when designing the new interface.

Demographics

New Page Patrollers are sometimes stereotyped as having certain characteristics: that they are overwhelmingly male, North American teenagers with little experience editing and low educational standards. For example, they are at times portrayed as secondary school students. This study found that while some of these ideas have credence, some do not.

For example, New Page Patrollers are heavily biased towards North America, as well as Europe, with 85% of patrollers coming from these regions. Similarly, they are mostly male – only 8% of patrollers identify directly as female. The geographic bias is of some concern, as it is likely to impact on both familiarity with certain subjects and the times of day at which patrollers are, in their time-zones, available – creating additional stress for those active in periods where there are few patrollers.

However, New Page Patrollers are also overwhelmingly over 18, with only 21% at most below that age, and 90% of them have completed secondary education. Furthermore, the most prolific and hard-working editors are not, in demographic terms, distinct; they maintain a similar male to female ratio, a similar geographic bias, and are actually more likely to be above 18 and to have completed secondary schooling. The median age of a patroller, meanwhile, is 32.

Patrollers are also likely to be highly experienced editors, with the majority joining Wikipedia in 2006 or earlier. While this invalidates a common criticism of patrollers, it is also concerning; patrolling new articles should be a simple and immediately understandable activity which new users can grasp easily. A low number of newer editors choosing to participate may indicate that the interface or the rules are unnecessarily complex, and lines up with the overall trends on editor recruitment.

These results line up closely with the April survey of the editorial community as a whole; there are similar results for education, gender and age. This indicates that patrollers are not, as is generally assumed, particularly distinct from editors overall, and that either the problems associated with new page patrol do not come from anything to do with the patrollers' outside lives, or that the problems are endemic to the entire community.

What gender do you identify as?

Gender, as with many of the other questions relating to demographics, was a mandatory one for respondents to answer – although it did include a “prefer not to say” option. When looking at how to improve the amount of participation in a task, and the quality of that participation, it is useful to see what groups may be significantly less likely to participate – both to help identify some sort of distinct problem with the software or the culture, and because high quality page patrol requires, to some degree, familiarity with the general subject matter of the article. Substantial under-representation of different groups increases systematic bias, and reduces the likelihood of individual patrollers either being familiar with the subject or knowing someone who is.

On Wikipedia, gender has recently been a particular focus of editors and staffers alike trying to combat systematic bias due to the small number of female editors. The April 2011 study showed that only 9% of editors are female, something of some concern. Survey returns showed that of all the respondents, only 8 %identified as female – 1% below the norm – with 89% identifying as male and 3 percent preferring not to say.

Things were slightly different with the 75th percentile and above – only 5 percent identified as women. However, the percentage of men dropped to 88, with the space being filled by those who preferred not to answer – 6 percent.

the gender balance of patrollers

When did you first start editing Wikipedia, either anonymously or with an account?

The amount of time a patroller has spent editing was also a mandatory question. Over 60 percent of patrollers have been editing since 2006 or earlier. With highly active new page patrollers – the 75th percentile and above – this dropped, but only slightly, to 56 percent. The high percentage of tenured editors shows that new page patrollers are not inexperienced. Errors are more likely to come from simple mistakes (caused by, say, stress) than from ignorance or naivete.

Although it is pleasing to see that so many patrollers are experienced editors, it is also concerning. The absence of “new” editors (only 4 percent of patrollers started contributing to Wikipedia in 2011) creates a bias, and ignores the benefits that the participation of relatively recent recruits provides.

What is your year of birth?

The third demographic-related question, and the third mandatory one, was about the ages of new page patrollers. Between 79 and 82 percent of new page patrollers are over the age of 18, in line with the editorial community overall, and this rises to between 83 and 85 percent with the high-workload patrollers. One of the criticisms about the survey was that the date range provided – which allowed for a maximum age of 71 – did not go high enough.

It is worth noting that the survey gathered “year of birth” and not “age”; this means it cannot be used to precisely estimate the ages of editors – hence the range of percentages above – and should not be used in such a fashion.

What is the highest level of education you have completed?

Over 90 percent of participants have completed secondary schooling, and 63 percent have an undergraduate degree or postgraduate qualifications. With high-workload patrollers, the proportion of users who have completed secondary schooling increases, to 93 percent, while 65 percent have an undergraduate degree or above. This lines up, generally speaking, with the editorial community overall. The April 2011 survey found that 61 percent of respondents had undergraduate degrees or above, meaning that patrollers are, on average, actually slightly more educated than the norm, not less.

From what general region do you do most of your new page patrolling?

Geographic location is important for two reasons. Firstly, there is a question of systematic bias and familiarity with subjects; if a patroller is from North America or Europe, they may incorrectly tag an article on a Bollywood film or actor out of their unfamiliarity with the field. This unnecessarily reduces Wikipedia's content, and may also drive off the article's creator – a well-intentioned new editor – something of concern in a time when editor numbers are dropping. A patroller from the Indian subcontinent, on the other hand, may be able to identify and tag the article appropriately, but could have trouble with more western-specific subjects.

Secondly, it also impacts on the availability of patrollers. If the vast majority of patrollers are based in Europe and North America, then during their respective “sleep” period the workload of patrollers is left to a smaller number of users. The fluctuation of patroller capacity could create problems with stress that may lead to errors in judgment and irritability towards new users, as well as disenchantment with new page patrol. It is therefore important to have a wide geographic and cultural spread of patrollers.

We found that new page patrollers are overwhelmingly from North America and Europe; only 9 percent come from the developing world. For the above reasons, this is concerning, and efforts to recruit more patrollers or simplify the interface should involve examining how to get more editors from the developing world to participate.

On what days do you edit Wikipedia?

Editing distribution throughout the week is important for the same reason as geographical distribution; if the workload is unfairly concentrated on a few individuals during a certain time period, it increases the chances of burnout and errors. In general, however, editing activity remained almost uniform throughout the week, including on weekends; this is not simply that editors were present all the time, but that as a group they engaged in the same level of activity, generally speaking, every day.

This doesn't, unfortunately, allow us to work out how patrols are distributed in time, but that data is being gathered separately. In the meantime, it does indicate that patrollers, as a group, tend to be fairly consistent contributors – which ties in with the idea that they're all relatively “tied in” to the movement.

Editing activity

Patrollers spend between 1 and 3 hours a day reading and editing Wikipedia. By edit count, patrollers tend to be in the upper range of editors overall – this can at least partially be explained by the “age” of patrollers' accounts.

Edits tend to be distributed between many areas. Many choose to create new content and improve existing articles, although anti-vandalism work is highly popular. When patrolling, the vast majority of users choose to use automated or semi-automated tools to assist them – something that could indicate deficiencies in the existing patrolling interface.

On average, how many hours per day do you spend reading and editing Wikipedia?

64 percent of new page patrollers spend between 1 and 3 hours a day reading and editing Wikipedia. This doesn't suggest that patrollers are (as is stereotyped) constant “hardcore” editors who are in some way addicted to the process. With the 75th percentile, the number of users who edit for more than 3 hours shifts upwards, but not very significantly; 64 percent, again, choose to spend between 1 and 3 hours a day reading and editing.

What is your approximate lifetime edit count now?

Around 20% of editors overall, according to the April survey, have more than 10,000 edits. For patrollers, however, the figures show that 46% fall into this category; almost half of them are in the top 20 percent, edit-wise, of contributors overall. Again, this supports the idea that patrollers tend to be highly experienced editors (in line with the question on when they joined Wikipedia). For the same reasons, this is both heartening and worrying; it indicates that some of the concerns about patrollers being inexperienced are misplaced, but also that, as an activity, New page Patrol is not attracting new contributors.

It's worth noting that some of the inflated edit count may come from patrolling itself – tagging pages acts as an edit, and even though deleted pages are not counted, adding a deletion tag usually involves notifying the editor who wrote the article – another edit. Particularly hard-working new page patrollers may see substantial parts of their edit count coming from the activity itself.

What user rights do you have on the English Wikipedia?

A majority of users have the ability to roll back edits (crucially, a right entirely used for anti-vandalism activities) – but 43 percent of patrollers lack this ability, suggesting that such work is not as widespread as people might believe. More importantly, 40 percent hold the “autopatrolled” right; a status only given to those users who create a certain number of new articles. This implies that patrollers play a fairly active role in providing content for the encyclopedia, as well as checking content submitted by others. The data from the 75th percentile actually shows that dedicated patrollers are far more likely to have the "autopatrolled" userright than is the norm.

How often do you do the following tasks on Wikipedia?

Data on areas of work backs up this idea – that although a lot of patrollers are, as the stereotype suggests, anti-vandalism volunteers, they also do a lot of work in other areas. 97 percent of patrollers do anti-vandalism work in some form, but 95 percent add content to the project – with 85 percent choosing to do so by creating new articles.

Again, this data changes for the 75th percentile – dedicated patrollers are more likely to do anti-vandalism work, and more likely to consider it something they do more often. However, they are also slightly more likely to add new content and write new pages, with 90 percent of dedicated patrollers choosing to create new articles as part of their editing activity. Most interestingly, a lot of the dedicated patrollers don't actually consider patrolling their primary activity. 25 percent only do it “sometimes”, rather than “frequently” or “very often”.

Have you participated in discussions in any of these venues?

There is a very strong correlation between patrolling and participating in various deletion “venues”, such as Speedy Deletion discussions or Articles for Deletion. This is unsurprising; New Page Patrol will inevitably result in some articles being tagged for deletion, and the taggers going through the various venues to complete the deletion process. With dedicated patrollers, there was also a strong correlation between participation in New Page Patrol and participation in anti-vandalism and “problem” noticeboards, such as AIV and AN/I. This data is not necessarily interesting; it could be that such a correlation is found within the editing population as a whole. Similarly, the question itself could have led respondents to overstate their involvement in these venues. "Have you participated in discussions” does not require regular participation (or even participation more than once) for the answer to be “yes”.

Do you use any of these tools?

The data on tool usage is extremely useful; it shows that the vast majority of patrollers use semi-automated plugins and third-party software to help with their patrolling work. On a stereotyping front, this does show that most patrollers prefer to do things in a semi-automated fashion – although it doesn't necessarily give any hints as to the speed at which they work. Third-party software usage is noticeably more extensive for the more dedicated patrollers, which could be a partial explanation for the higher patrol rates.

For software design, it's important to identify what software is used, how widely it's used, and why. The extent of software use here is indicative of some flaws with the existing Special:NewPages interface; why these particular tools are preferred over others, and what they offer that Special:NewPages does not, will be investigated in our follow-up work (see below) to help improve the design of the Zoom interface.

Patrolling work

Patrollers largely started doing New Page Patrol indirectly; they saw mention of it on a user's talkpage, or followed a link at Special:RecentChanges. A greater effort should be put into advertising the tools and encouraging editors to use them. Patroller motivations for starting and continuing to work are largely positive, with very few patrollers being motivated by ideological concerns or an active desire to see more pages deleted.

Most patrollers do relatively little work per week. This is unsurprising, given the unequal distribution of the workload as documented above. But they take a reasonable amount of time to gauge the value of each page before making a decision. They are also fairly familiar with surrounding policy, but struggle with making judgment calls on articles and dealing with both new users and experienced editors.

How did you first learn about New Page Patrol?

The vast majority of patrollers who remember how they got involved heard about New Page Patrol passively and unintentionally: they saw a userbox on somebody's userpage that advertised New Page Patrol. The results suggest that New Page Patrol is not a highly linked-to task; the only prominent link built into MediaWiki that people use, according to the results, is the one at the top of Special:RecentChanges. Special:NewPages should be more prominently linked as a way of drawing in new contributors.

One interesting side-effect of this passive and unintentional learning is that the majority of new page patrollers go into the task (or are introduced to the task) without anyone to provide oversight or instruction.

Why do you patrol new pages? Tick all that apply

The vast majority of patrollers do their work for positive reasons: they want to keep bad articles out of Wikipedia (83 percent) or because they want to monitor and check the quality of new articles (80 percent). It is important to note that 35 percent of patrollers are motivated because it provides experience and support that may aid their development as an editor.

Despite the notion that patrollers are people who primarily focus on having things deleted, only 16 percent of patrollers identify themselves as being motivated by an urge to see more articles removed. 12 percent, meanwhile, are motivated by a desire to see more articles kept.

It is possible that the wording has biased the results; people may, perhaps, inevitably choose to associate themselves with more altruistic motivations. The low percentage of users motivated by their opinion on keeping/deleting articles may be an indication that patrollers are not, generally, focused on holding a particular position – or it may be that people are simply reticent about admitting it.

On average, how many hours a week do you spend patrolling new pages?

Patrolling work is unevenly distributed, with the top 25 percent doing 89 percent of the work. Almost 40% of patrollers spend 1 hour or less on New Page Patrol a week. While many activities on Wikipedia are unevenly distributed, effort should be put into working out why patrollers don't contribute more, and how we can help improve that. Is it a cultural problem? Is it due to technical deficiencies? Is patrolling simply not attractive to most people?

On average, how long does it take you to patrol one new page?

49 percent of patrollers take a minute or more to patrol a new page. However, this doesn't tell the whole story; the vast majority of patrols (89 percent) are made by the top 25 percent of users. Data on how long it takes the average patroller to patrol an article isn't useful for calculating how long an average patrol takes. For that, you need to look at who makes what patrols. Looking at the top 25 percent, however, it's still fairly similar – 47 percent of patrols take more than a minute to make.

Obviously, a slight majority of patrollers still choose to patrol pages relatively quickly – with under a minute spent on deliberation. Technical changes can't easily force a patroller to spend a certain amount of time working on each a page, and even if they could it would be unfair. Instead, our focus is on ensuring that if patrollers are going to patrol things quickly, they can patrol things quickly and properly. There shouldn't need to be a tradeoff between speed and quality.

What do you think is the most challenging or difficult aspect of New Page Patrol?

When asked what is difficult about patrolling, 28 percent pick some variant on “trying to decide what should be deleted/if something should be deleted”. The proposed interface does include tooltips to explain, quickly and easily, what is required for various deletion criteria. Hopefully this should improve things. In the meantime, the community is welcome to investigate trying to simplify deletion policy, and make it more understandable.

Another interesting point is the 10 percent of patrollers who consider dealing with new editors to be the most difficult aspect of patrol work. This isn't something limited to patrollers in general; in fact, for the 75th percentile, that rises to 12 percent. This shows that at least some patrollers are aware of and appreciate the difficulty of explaining our (sometimes opaque) deletion criteria and policies to new editors.

While the Foundation can't directly help make this easier, the hope is that by making patrolling itself less stressful, and including a larger number of patrollers (and therefore wider perspectives), people will be more amenable to new editors. Obviously, some new editors are genuinely going to be unwilling to compromise, or will be here to cause trouble rather than to contribute productively, but those editing in good faith should be faced with patrollers who have the time to explain things thoroughly.

Which of the following have you read? Check all that apply

An overwhelming majority of patrollers have read the policies surrounding new page patrol. When looking at the 75th percentile, which does the majority of patrolling, every editor has read the speedy deletion criteria and notability guidelines. Interestingly, a lot of users are unfamiliar with related guidelines such as the rules surrounding categorisation. This doesn't necessarily indicate that patrollers don't do article fixing as well as article deletion; fixes could be copyedits or the addition of references rather than categorisation, and people can categorise fairly easily without needing to read the guidelines.

Obviously, simply being familiar with the guidelines does not indicate adherence – while 100 percent of the 75th percentile may have read the speedy deletion criteria, it is inevitable that some bits will be forgotten, either due to the passage of time or in the heat of the moment. In addition, it's possible that the question altered the responses - “I have read this guideline” could mean “I read it in 2006”. Having read something at some point in the past does not mean complete familiarity with it.

When patrolling new pages, where do you usually do this from?

This question looked at where in the “buffer” at Special:NewPages patrollers choose to do their work. The front of the buffer contains the newest articles, while the back contains the oldest – ones that have spent up to 30 days on Wikipedia. Patrolling work can also be done from anywhere between these two points, and also from Category:Speedy Deletions – although that would only be useful to administrators reviewing those pages nominated for deletion, and those patrollers who want to check the validity of nominations made by other editors. A plurality of new page patrollers (45 percent) choose to patrol from the front of the queue, checking recently created articles. This is understandable; for the purpose of keeping attack pages, clearly non-notable articles and copyright violations out of Wikipedia, the newer articles are the ones that should be checked first. It is important to remove copyright violations and defamatory text, for example, as soon as possible.

One important thing to note is that the front of the queue contains not just newer articles but the newest articles. The creators, even if they're not immediately planning on returning to Wikipedia, are far more likely to be editing and responding to (or receiving) talkpage messages, and paying attention to their article. With a plurality of page patrollers choosing to patrol pages created by people like this, it is highly important to ensure that they are patrolling correctly, and that they are patrolling civilly; deleting an article may or may not drive the creator off, but doing so tersely or impolitely is not going to help. Obviously, some articles need to be deleted, but unless they are clearly bad faith creations, they should be evaluated with care and an appreciation of the perspective of the article's creator.

Improving New Page Patrol

When polled on proposed changes to New Page Patrol, including the introduction of a user right for patrollers, users were ambivalent; a bare majority overall opposed the suggestion, while an even narrower majority of the 75th percentile supported it. These results are complicated by the fact that some respondents clearly misunderstood the nature of the question. Patrollers also cannot agree on what requirements would be attached to this right, although did (narrowly) confirm how such a right would be granted.

Asked about their own ideas, however, patrollers suggested a wide range of possible improvements to New page Patrol. Most of these are technical, and in many cases are being worked on or considered; others involved cultural shifts or alterations to policy within Wikipedia, and the community should be encouraged to fully investigate these.

Do you think that the ability to patrol new pages should be tied to a user right? Why?

The creation of a “patroller” user right is something that has been discussed by both the community and the Foundation, and is currently being considered for the Zoom interface. Currently, any editor can patrol new pages; if editors are bad at patrolling, there's no way to stop them short of a block. The introduction of a user right would provide the community with a way of controlling who can and cannot patrol new pages, with the idea being that this would increase the calibre of patrollers Special:NewPages attracts and allow those with poor judgment to be restricted. This question was intended to gauge where patrollers stand on the issue.

A narrow majority of patrollers overall disagreed with the idea, 55 percent to 45 percent. For the 75th percentile, an even narrower majority agreed with it – 51 percent to 49 percent. Those who support the idea primarily argued that existing patrollers were inexperienced or incompetent, and that newer editors (who can currently patrol articles right from the start) cannot be trusted to do it properly.

Those who opposed it pointed to the fact that this would invariably lead to fewer patrollers, in a time when New Page Patrol is already a burden shared by relatively few users. In addition, it was seen as going against the “anyone can edit” philosophy and creating unnecessary additional bureaucracy. New Page Patrol was also seen as a good activity for new users to undertake; it provides an introduction to editing other areas of Wikipedia.

This is ambiguous. Gauging opinions here is made more difficult by the fact that some respondents on both sides clearly thought this was a right that would be required for users to create articles, not to patrol them.

If so, by what process should patrollers be appointed? What should the criteria be?

A narrow majority felt that the patroller user right should be granted to editors automatically, when they reach a certain set of conditions; others believed that they should instead required to go through Requests for Permissions, and have the right granted by an administrator. Again, consensus is not entirely clear.

When asked what criteria should be required for the patroller user right, respondents provided a wide range of opinions. The largest group – just over 20 percent – felt that it should just require a certain number of edits, while an additional 10 percent thought that it should require a certain number of edits, and a certain period as an editor. Other users thought it should be tied to (or granted at the same level as) the “autoconfirmed” right.

It seems clear that some variation on “X edits, and Y months as an editor” is likely to be the most acceptable criteria, but even within the group of users who supported something like this, people disagreed as to how many edits, or how much time on site. Any attempt to get firm consensus on this point, whether made by the community or by the Foundation, is likely to be drawn-out and gruelling.

What would you like to see changed about new page patrol?

The majority of respondents (60 percent) wanted technical changes – this may be because this is the most pressing sort of concern for patrollers, or because the distributor of the survey (the Foundation) is primarily concerned with technical changes, and so respondents wanted to vocalise what they thought we would consider the most relevant problems. Suggestions include:

A better interface;
“pre-sorting” articles by subject area;
More categories than “patrolled” or “not patrolled”;
Better filtering and tagging for suspect articles.

Many of these suggestions are already proposed for the new Zoom interface; one set of suggestions covered a bug that prevented users patrolling pages that have been moved to a new title; this has already been resolved.

Another 20 percent suggested cultural or policy-based changes. While this is not something the Foundation can directly work on, the community may want to investigate the suggestions themselves. Proposed changes included:

Simplifying or widening the deletion criteria;
Providing training or oversight for patrollers;
Encouraging recruitment of more patrollers.

A further 20 percent had no further suggestions, or simply consider the current system workable.

Next steps

Data release

We are releasing the raw data behind this survey, albeit separately and in a slightly "sanitized" form; gender and contact information have all been scrubbed, the former due to European Union law (which some respondents are subject to) and the latter in compliance with the Foundation's own privacy policy. This data, to comply with the terms it was gathered under, is available to anyone who is interested in using it to conduct research and willing to sign the volunteer Non-Disclosure Agreement; if you are interested, email okeyes wikimedia org.

We encourage editors and researchers who find this data interesting to conduct research themselves; this is only one perspective and possible interpretation of the data. Other analyses are most welcome; decisions on future features development will be made with whatever data is available at the time – so the wider the pool of research, the more informed we will be in developing designs.

Developing Zoom

The next step for the Foundation is to use this data to continue developing the Zoom interface. We have already identified a representative sample of patrollers, and contacted them for detailed interviews and to provide “screencasts” of their patrolling work. With these, developers can examine the process of patrolling, get more details on precisely how patrollers do their work, and try to identify unnecessarily difficult areas that can be simplified to make patrolling easier.

The interface that is currently documented on Mediawiki should be taken as preliminary. In its current form, it presents the general design direction for the feature. There are many aspects of the page patrolling workflow (such as editing the article while patrolling) that have not been incorporated into the design. A “patroller” user right is also currently part of the new interface, although that may change (whether the community wishes to pursue this change or other changes, such as the policy or cultural alterations proposed in here, is up to them). The design of this interface will continue to evolve over the next few weeks.