Expert review

Purpose of this page: Elaborate on a lightweight model which will allow us to receive and store reviews by credentialed experts of specific Wikipedia article revisions, and to surface such reviews through the UI and through the API.

Rationale

Assertions by credentialed experts that an article does or does not represent the best available scholarship on a topic can help readers (by making it easier to find high quality content) and editors (by incentivizing high quality work, surfacing problems, etc.), as well as individuals or groups seeking to use subsets of Wikipedia for educational purposes (by allowing extraction and use of reviewed subsets of Wikipedia content). This does not require any changes to Wikipedia's editorial process, and indeed, capturing, storing and making available such reviews does not imply any additional valuation of the credentialed expertise or even its relevance to the article at hand.

Precedents

Magnus Manske's experimental JavaScript widget to display article reviews

Phase II design for the reader feedback tool (does not support expert reviews yet)

Actual practice:

• Experiment with Encyclopedia of Life (species), Rfam (RNA families) and Pfam (protein families)
  • Magnus Manske's user script that surfaces available expert reviews on relevant articles
  • Magnus Manske's proof-of-concept tool to create metadata for printed books, PDF or ODT exports
  • Example API query to return expert review metadata for the "Gulf Snapping Turtle" article in JSON format, based on Magnus Manske's toolserver database storing expert reviews for multiple partners
• Google.org's "Health Speaks" translation project for medicine articles worked with WikiProject Medicine and expert reviewers to post reviews to talk pages and validate the community's assessment of articles
• The Wikipedia:WikiProject United States Public Policy has invited public policy experts to review articles within the scope of the WikiProject, as well as articles specifically improved through the course of the initiatives. The expert ratings are currently stored in a private Google Doc and will be published, together with some accompanying analysis, soon.

Proposals and research:

• Research Committee survey on expert involvement (in development)
• strategy:Proposal:Expert review - more comprehensive proposal that encompasses additional complex and controversial ideas, such as giving experts more influence in the editorial process
• Referees - comprehensive proposal for multiple levels of quality assessment
• Sifter project - historical proposal from the Nupedia days

What you can do right now

If you're a hacker:

• Make Magnus Manske's user script better and help develop it to gadget quality (documentation and known issues)
• Create other proof-of-concept applications that tie into the existing toolserver database / API developed by Magnus
  • Proof-of-concept application for allowing authenticated users to submit reviews
• Help extend the ArticleFeedback MediaWiki extension to support product requirements articulated below

If you're good at analyzing and documenting:

• Help us develop draft product specifications for how expert reviews should be supported by MediaWiki (UI and workflow mock-ups very much welcome)

If you're great at outreach into academia:

• Help identify opportunities for mobilizing large numbers of experts to join the review process

User research

Wikipedians

This section is mainly based on a review of the existing essays and views about expert editors and credentials on the English Wikipedia and the work of Logan et al..

The commonly established consensus among Wikipedians is that "experts" are welcome to edit articles, but should do so just like any other editor. They shouldn't get any preferential treatment because of claimed credentials. They must provide reliable sources for their changes, and can't use their authority as an "expert" to try to win an argument. Like any other Wikipedian, they have to follow the same policies, and must be particularly careful about writing from a neutral point of view, and avoiding self-promotion, conflicts of interest, and original research.

This assessment leads to the conclusion that it is likely to be difficult to recruit editors from targeted "expert" circles like academics and professionals. It is likely that, just like any other population, only a fraction of the "expert" population will demonstrate the qualities and motivations required to be a Wikipedian.

If "experts" are to be specifically involved in the improvement of Wikimedia content, it needs to be through other venues and processes, like independent, externally-hosted reviews.

"Experts"

This section is mainly based on preliminary results from the Research Committee survey on expert involvement.

The main barrier identified by "experts" is the lack of professional benefit from contributing to Wikipedia. It doesn't provide credit, nor does it count for tenure, or advances one's career, so they feel they don't have time for it. Because there is no professional incentive to contribute to Wikipedia, it is not a priority in the busy schedule of academics and professionals.

The other main barrier is social: academics are generally used to be treated with deference. The interaction with a community of egalitarian Wikipedians can rapidly be disconcerting, even frustrating, when their edits are disputed, edited, or removed. "Experts" are also reluctant to engaging in long, difficult, and sometimes hostile discussions with "amateurs" who don't accept the argument of authority.

Product requirements

Very basic notes.

• Should surface expert review information in an unobtrusive manner, ideally integrated with other review information such as reader feedback, FlaggedRevs, etc.
  • Recognizing reviewer organizations with logos and links would be a nice incentive for organizations to join the program
• Should allow for short and extended comments
• Should enable a simple yes/no decision on whether an article should e.g. be included in an offline distribution
• Should recognize that multiple fields of expertise may apply to any given article
• Should support e-mail as a delivery mechanism for new articles requiring review (to simulate peer review process)
• Should capture conflicts of interest by reviewers
• Should initially require only minimal effort towards credentials validation, likely by offloading this requirement to other organizations
  • e.g. in the current Encyclopedia of Life prototype, EOL has its own criteria for curator expertise
• Should have one or multiple authentication mechanisms
  • based on rights-assignment to individual users, possibly granted at sign-up through separately authenticated sign-up form?
  • based on URL keys that can be used for e-mails etc.?
  • based on open authentication/authorization standards such as Shibboleth?
• May need to interface with WikiProjects for purposes such as selecting articles requiring review, outreach, etc.
• Should not cannibalize editing - reviews should be coupled with an invitation to improve the article directly
  • But: reviewers must be independent and not substantially involved in the article's development. Reviewing a revision you haven't touched and improving it going forward would be less problematic than reviewing a revision you've substantially contributed to.

See also

• WikiExperts