Jump to content

Wikipedia:External peer review

From Wikipedia, the free encyclopedia
"WP:EPR" redirects here. For {{edit protected}} requests, see Wikipedia:Edit requests.
Shortcut

External peer review is a Wikipedia project namespace list of reviews of the accuracy of Wikipedia articles and other Wikipedia content posted by newspapers, magazines, and other agencies outside of Wikipedia. Peer reviews are different from general articles about Wikipedia because they give a critique of one or more articles specifically, often noting problems or highlighting their good qualities.

This list aims to be both a record of the findings of these reviews and a way to highlight any problems (or indeed positive aspects) that were encountered. Although Wikipedia has its own article peer review process, reviews from respected publications are more than welcome and can help to improve Wikipedia's articles. In most cases Wikipedia will also see an increase in traffic to the articles mentioned, which means that those with poor reviews should be improved to as high a standard as is possible.

Articles that have undergone an external peer review are tagged with {{external peer review}} on their talk page. Articles that have undergone an external peer review and publication in scholarly journals are tagged with {{academic peer reviewed}}.

Academic

[edit]

Academic peer reviews are those performed by independent experts in a topic, sometimes with resulting corrected articles published in scholarly journals.

Assessments of individual articles

[edit]

Mechanisms for peer review of individual articles are summarised at WP:W2J.

Wikipedia articles published in peer-reviewed literature (10 C, 81 P)
Wikipedia articles with corresponding academic peer reviewed articles (3 C, 124 P)

Assessments of multiple articles

[edit]

Many academic studies have attempted to assess the overall accuracy of topics in Wikipedia (especially medical topics).

Science

[edit]
  • Mandler, Michael D. (2017-01-26). "Glaring Chemical Errors Persist for Years on Wikipedia". Journal of Chemical Education. 94 (3): 271–272. doi:10.1021/acs.jchemed.6b00478. ISSN 0021-9584.

History

[edit]

Medicine

[edit]
See also: Health information on Wikipedia

Formal

[edit]

Reference Services Review (2008)

[edit]

Discussion page

A comparison of Wikipedia, Encyclopaedia Britannica, The Dictionary of American History and American National Biography Online

Findings

[edit]

Summary: "The study did reveal inaccuracies in eight of the nine entries and exposed major flaws in at least two of the nine Wikipedia articles. Overall, Wikipedia's accuracy rate was 80 percent compared with 95-96 percent accuracy within the other sources. This study does support the claim that Wikipedia is less reliable than other reference resources. Furthermore, the research found at least five unattributed direct quotations and verbatim text from other sources with no citations."

Badlands

"All the facts are verifiable and correct; the difference here lay in the facts included." Comparisons were made difficult by the varied focus of the articles, but Britannica Online's article was the most extensive.

Unfortunately, the author failed to notice that Wikipedia did (and still does) have separate articles on Badlands and Badlands National Park; as a result, her comparison was flawed, since she did look at paired articles in two of the other three encyclopedias. -- John Broughton (♫♫) 21:21, 1 March 2009 (UTC)[reply]
Sand Creek

"Wikipedia again has a much higher proportion of unverifiable (three) and factual errors (six) than does either DAH (two incorrect) or Britannica (one incorrect)"

Chautauqua movement

Wikipedia and DAH have more comprehensive articles than Britannica, though the Wikipedia and DAH articles have different focuses. Though the Wikipedia article has more unverifiable facts (eight) and inaccuracies (nine) than does DAH, it still holds a higher ratio of accuracy because it includes more facts than other sources. The Wikipedia entry is "relatively succinct and tightly written".

The last paragraph discussing Zen and the Art of Motorcycle Maintenance contains quotes without attribution. Quotes from Theodore Roosevelt and Sinclair Lewis are also unattributed.

Free Soil party

"Wikipedia has a single fact error – the number of party representatives in Congress after the 1848 election. Additionally, Wikipedia contains a quote apparently from the party’s platform with no attribution."

Niagara Movement

"Although all of the entries are brief, there are still disputable facts; Wikipedia has the most with four unverifiable facts and three containing errors ..."

Battle of Harper's Ferry

"Despite the disparity in length and depth of the two articles, the Wikipedia entry has only one fewer incorrect fact that DAH and has one more unverifiable fact. Still, the Wikipedia article has a rather long direct quote in the last paragraph of the John Brown section without attribution, and the one citation included in that section does not accurately reflect the original source (the Wikipedia entry lists a different date than the original)."

Mexican-American War

"this entry in Wikipedia is written so that small skirmishes have more prominence than major battles (and some major battles were missing entirely), and readers may have difficulty determining the chronology of the war without reference to specific dates because the entry did not sequence events in chronological order. Unlike the other two samples evaluated, this entry obviously is written by multiple authors with little apparent concern for the overall flow and readability of the article."

William Kidd

"Many of these details in Wikipedia are either incorrect (14) or unverifiable (37) as compared to the four unverified facts in DAH and the three partially incorrect facts in the two Britannicas. For example, the Wikipedia article includes several details of Kidd’s childhood that were not confirmed in three principal biographies (Harris, 2002; Ritchie, 1986; Zacks, 2002), and the essay suggests he turned to piracy before most of the biographers indicate. One paragraph of the essay references a children’s book on Kidd, which may oversimplify the man’s career."

Harriet Tubman

"Wikipedia’s entry follows ANB in terms of length and comprehensiveness; it includes 67 facts. All except two are verifiable. Of the remaining 65, ten contain errors. Some of these are minor errors, such as year of birth and number of siblings; others are more significant, such as the number of rescue attempts she made and the number of slaves she helped escaped."

Additional overall comments:

  • "Eight Wikipedia articles contain unattributed quotes and at least five cases of possibly plagiarized content (material found verbatim elsewhere). The researcher found these cases easily without intense scrutiny; more text may have been copied."
  • "One area of further study may be to examine and evaluate Wikipedia’s histories themselves. In the brief review of the more detailed history of the Mexican-American War, for instance, this study found that references were deleted and correct facts were modified incorrectly. Also, it would appear that some inaccuracies may have been typing errors. A more detailed study of individual histories may lead to deeper analysis as to the nature of Wikipedia’s contributors."

Response

[edit]

None that I have seen. Mangostar (talk) 06:26, 10 June 2008 (UTC)[reply]

PC Pro (June 2007)

[edit]

Discussion page

A comparison between Wikipedia, Encarta and Britannica

Findings

[edit]
Otto von Bismarck - Britannica - Encarta

Overview: The Wikipedia article is poorly written in places and not as comprehensive as Britannica. Encarta "lacks the detail offered in the Wiki piece".

Quotes:

  • "Largely sound on the facts"
  • "a marked deterioration towards the end"
  • "leaves out the prominent theme of the Bismarck cult that flowered after his departure from office"
Franco-Prussian War - Britannica (Franco-German War) - Encarta

Overview: Wikipedia article judged best. Encarta is "extremely superficial"; Britannica is "polished and factually correct although ... less richly detailed".

Quotes:

  • "clearer" and "more elegant" [writing than Otto Von Bismarck]
  • a few recent studies are missing from the bibliography
  • Military and technical details are ... well handled
Atherosclerosis - Britannica - Encarta (Circulatory System)

Overview: Wikipedia "performs well", Britannica "fares poorly", and Encarta "earns the Dunce's Cap".

Quotes:

  • of more benefit to the serious student than its Encarta and Britannica equivalents
  • "the layman may find too much information"
Plate tectonics - Britannica - Encarta

Overview: Wikipedia article broadly accurate, but messy. Britannica's entry "contained an error in its opening paragraph", "too long for most people's needs", and "reminded me of one of my worst lectures at university". Encarta "clearly written by an expert" and "the only one to display enthusiasm for the subject".

Quotes:

  • good for the bare facts but didn't read particularly well
  • "slightly messy description of outer Earth layering, which reappears elsewhere in the text, suggesting a lack of decent editing"

Response

[edit]

None yet. violet/riga (t) 15:53, 16 June 2007 (UTC)[reply]

With plate tectonics being a featured article I think we need to at least review the issues raised here. violet/riga (t) 16:07, 16 June 2007 (UTC) It's changed a lot since this version. Over 800 edits.LeadSongDog (talk) 00:00, 27 April 2008 (UTC)[reply]

The Denver Post (May 2007)

[edit]

Discussion page

"The Denver Post asked five Colorado scholars to review the Wikipedia entries on Islam, Bill Clinton, global warming, China and evolution."

Findings

[edit]
Global warming

Colorado State University's Scott Denning, the Monfort Professor of Atmospheric Science.

  • "a great primer on the subject"
  • "Following the links takes the interested reader into greater and greater depth, probably further than any traditional encyclopedia I've seen"
  • pleasantly surprised how the main articles "stick to the science and avoid confusing the reader with political controversy."
  • wishes Wikipedia offered better links to basic weather science.
China

University of Colorado history professor William Wei

  • "simplistic, and in some places, even incoherent."
  • "mishandled the issue of Korean independence from China"
  • "and the context of the Silk Road in China's international relations."
Bill Clinton

Bob Loevy, political science professor at Colorado College and frequent writer on Bill Clinton

  • thorough and unbiased, giving fair weight to both Clinton accomplishments and scandals.
  • The bulk of it appeared to have been written by the Clinton Museum and Library in Little Rock, Ark.
  • "a great place for a student to begin building his or her knowledge" on Clinton
Islam

Retired CU religious studies professor Frederick Denny, 40-year specialist in Islam

  • "quite impressed"
  • "It looks like something that might have been done by a young graduate student, or assistant professor, or two or three"
  • clinical and straightforward, but not boring.
  • where important translations of Arabic language or fine religious distinctions are required, Wikipedia acquits itself well.
Evolution

CU biology professor Jeffrey Mitton

  • "good," even if "stylistic infelicities abound."
  • If a student read through the main entry and the primary links to supporting concepts, he would get a fine introduction
  • first reference cited for the authoritative textbook on evolution by Douglas Futuyma, "so that is excellent, as it should be,"
  • rest of the source list appropriate, and well-rounded

Response

[edit]

Looks like we got lucky, since 3/5 of the articles picked are Wikipedia:Featured articles, we could hope they'd be good. China needs work, not sure what "stylistic infelicities" in Evolution are specifically, otherwise it looks like we just need to bask. :-) --AnonEMouse (squeak) 01:41, 28 July 2007 (UTC)[reply]

An empirical examination of Wikipedia's credibility (November 2006)

[edit]

Findings

[edit]
  • "five reported seeing mistakes and one of those five reported spelling mistakes rather than factual errors"

Response

[edit]

Email sent asking for a list of the articles with errors (or a list of the errors) if possible, so we can create a to-do list similar to Wikipedia:External_peer_review/Nature_December_2005/Errors. -- Jeandré, 2006-11-29t08:10z

Bleh... I had not noticed this yet, and just sent another. Circeus 23:12, 29 November 2006 (UTC)[reply]
Same here. Any replies?-- Piotr Konieczny aka Prokonsul Piotrus | talk  22:20, 16 March 2007 (UTC)[reply]
I received an initial reply without a list of articles or errors but the possibility of some more information later; but nothing since then. I suggest that only 1 person, User:Piotrus, does a follow up. -- Jeandré, 2007-03-17t06:07z
I got a reply, which sais, basically, that "About the errors, the questionnaire was completed anonymously so there is no list of the articles that contained the errors, let alone the errors themselves." With that, I think we can archive the study, as with such design it is not really a usable EPR.-- Piotr Konieczny aka Prokonsul Piotrus | talk  17:04, 24 March 2007 (UTC)[reply]
Hardly a useful study then. How can they be sure that those who found Wikipedia inaccurate were accurate about the inaccuracies in Wikipedia? Ah. That last sentence felt good to write, though I suppose it is not so nice for those who read it. - Ta bu shi da yu 10:39, 28 July 2007 (UTC)[reply]

Journal of American History (June 2006)

[edit]

Roy Rosenzweig (June 2006). "Can History be Open Source? Wikipedia and the Future of the Past". The Journal of American History. 93 (1): 117–146.

Findings

[edit]
  • "History is probably the category encompassing the largest number of articles", but the collaborative editing makes it unlike virtually any other scholarly work in that field.
  • Wikipedia articles have numerous authors, many of whom may be anonymous, and there is no expectation that their names are used when discussing their articles.
  • "This extraordinary freedom and cooperation make Wikipedia the most important application of the principles of the free and open-source software movement to the world of cultural, rather than software, production."
  • The freedom of reuse allowed by open licensing of content is an important advantage for educational uses.

Nature (December 2005)

[edit]
A complete list of errors with their current status can be found at Wikipedia:External peer review/Nature December 2005/Errors

Nature compared Wikipedia and Britannica science articles and sent them to experts in the field. The number of "factual errors, critical omissions and misleading statements" were recorded.

From their blog:

We're trying to see if we can publish the full list of errors found by our reviewers, or least send them to you (and to Britannica if they want). We'll post an update here as soon as we have a firm answer.

I also received a private email from them in response to a request for more information. I hope they don't mind me posting it below:

In light of the amount of interest, we have decided to make the reviews public as far as possible, although obviously we'll have to edit them to remove the names of the reviewers, any libellous statements etc. The reviewers didn't all respond in the same format, and some of them highlighted points that we didn't consider to be significant errors, so we're also writing up an accompanying document to explain which errors we counted, and how we arrived at all the numbers. We're also asking the reviewers if they mind being identified, so we'll name those who give permission. That's all quite a bit of work, especially with Jim being away, but I hope we can send this to you by the end of next week, as well as putting it up (free) on our own website. Thanks for your patience! (15 December 2005)

Update: The reviewer reports are now available on the Nature web site, in Microsoft Word format. See above URL.

This review was covered at Wikipedia:Wikipedia Signpost/2005-12-19/Nature study

Findings

[edit]

Of the 42 articles reviewed, 38 were found to have at least one error – Britannica had 40 articles with at least one error. (NOTE: Nature took some of the excerpts for the study from the version of Britannica for children and youths rather than use the official version for adults. Britannica claims the study is invalid as a result because none of the articles taken from Wikipedia were "for children or youths.") — Preceding unsigned comment added by 64.134.40.134 (talk) 00:02, 25 November 2012 (UTC)[reply]

The following articles had the highest number of errors:

The following articles had no errors highlighted:

Nature's special report also noted the following:

  • "several Nature reviewers" found the Wikipedia article they reviewed to be "poorly structured and confusing" — a criticism that the report notes is common among information scientists;
  • unnamed information scientists also "point to other problems with article quality, such as undue prominence given to controversial scientific theories" (see Wikipedia:Guidelines for controversial articles and Wikipedia:Neutral point of view for guidelines related to this problem);
  • In Wikipedia's defense, Michael Twidale, an information scientist at the University of Illinois at Urbana-Champaign, told Nature that Wikipedia's strongest suit is the speed at which it can be updated, a factor not considered by the journal's reviewers.

Update: the detailed reviewer reports are now available (see above).

Errors per word comparison

[edit]

Please post below a table of errors/word statistics, based upon the Nature article and the word counts in the corresponding articles, so that we can see a more controlled comparison of error rates. —Steven G. Johnson 02:53, 15 December 2005 (UTC)[reply]

If it seems to make sense to calculate and compare the ratio errors/words, it probably makes less sense to compare the ratio omissions/words (1 omission in a 5000 words article might be considered to be more serious than 1 omission in a 1000 words article on the same topic, whereas 1 error in 5000 words is probably better than 1 error in a 1000 words); and it might make even less sense to mix both categories in an attempt to gauge any bias in favour or against Wikipedia (or Britannica) in the Nature experiment. I would therefore be very careful in drawing any conclusions based on the numbers in the table below. --83.180.100.151 20:48, 8 January 2006 (UTC)[reply]
Britannica displays the word count for each article, doesn't it? At least that part shouldn't take long. ‣ᓛᖁ♀ᑐ 03:29, 15 December 2005 (UTC)[reply]
You'd think that not all words are created equal. — Ambush Commander(Talk) 03:43, 15 December 2005 (UTC)[reply]
No doubt, but this seems the quickest reasonable statistic to gather. —Steven G. Johnson
Is this really more controlled? The average WP article has lots of fluff, with no space-pressure to remove same. Errors/omissions-per-article seem a reasonable metric to me. sj 07:37, 15 December 2005 (UTC)[reply]
Your fluff is another person's interesting tidbits. =) Honestly, it's hard to make a quantitative assessment of information content, but if there is a large difference in article length then it is a hint that apples are not being compared to apples. (Nature claims that the article lengths were comparable, but I'm finding this hard to reconcile in some cases, e.g. West Nile virus (see below) where the WP article has apparently been almost 5 times longer than EB's for a year now.) The size differences seem large enough that I'm inclined to think that the 30% difference between EB and WP in the Nature study is washed out by systematic problems, although that of course depends on the type and severity of the errors. —Steven G. Johnson 08:35, 15 December 2005 (UTC)[reply]

Note that, for a fair comparison, we shouldn't include tables of contents, external links, "see also", or references — most Britannica articles do not include these, and the Nature review did not consider referencing quality. Nature refers to "factual errors, omissions or misleading statements", so some of the "errors" listed below may be errors of omission in incomplete articles, rather than factual errors.


Article name Britannica Wikipedia
Word count Errors Errors/word Word count Errors Errors/word
Acheulean industry 500 1 0.002 417 7 0.016787
Agent Orange 252 2 0.00793 1270 2 0.0015748
Aldol reaction 130 4 0.030769 660 3 0.0045455
Archimedes' principle 350 2 0.0057143 607 2 0.0032949
Australopithecus africanus 235 1 0.0042553 496 1 0.0020161
Bethe, Hans 658 1 0.0015198 1823 2 0.0010971
Cambrian explosion 519 10 0.019268 702 (13 Dec.) 11 0.0157
Cavity magnetron 394 2 0.0050761 1121 2 0.0017841
Chandrasekhar, Subrahmanyan 365 4 0.010959 417 0 0
CJD 591 2 0.0033841 1373 5 0.0036417
Cloud 641 3 0.0046802 1689 5 0.0029603
Colloid 561 3 0.0053476 896 6 0.0066964
Dirac, Paul 837 10 0.011947 1044 9 0.0086207
Dolly 1334 1 0.00074963 807 4 0.0049566
Epitaxy 178 5 0.028090 235 2 0.0085106
Ethanol * 315 3 0.0095238 2631 5 0.0019004
Field effect transistor 588 3 0.0051020 933 3 0.00322
Haber process 241 1 0.0041494 531 2 0.0037665
Kinetic isotope effect 210 1 0.0047619 569 2 0.0035149
Kin selection 923 3 0.0032503 404 3 0.0074257
Lipid 349 3 0.0085960 676 0 0
Lomborg, Bjorn 518 1 0.0019305 1501 1 0.00066622
Lymphocyte 479 1 0.0020877 351 2 0.0056980
Mayr, Ernst 357 0 0 753 3 0.0039841
Meliaceae 152 1 0.0065789 281 3 0.010676
Mendeleev, Dmitry 1306 8 0.0061256 1134 19 0.016755
Mutation 728 8 0.010989 1557 6 0.0038536
Neural network 557 2 0.0035907 1233 7 0.0056772
Nobel prize 409 4 0.0097800 2052 5 0.0024366
Pheromone 313 3 0.0095847 461 2 0.0043384
Prion 473 3 0.0063425 1583 7 0.0044220
Punctuated equilibrium 943 1 0.0010604 1265 0 0
Pythagoras' theorem * 688 1 0.0014535 1899 1 0.00052659
Quark 1112 5 0.0044964 2060 0 0
Royal Greenwich Observatory 235 3 0.012766 532 5 0.0093985
Royal Society 416 6 0.014423 869 2 0.0023015
Synchrotron 770 2 0.0025974 1590 2 0.0012579
Thyroid 583 4 0.0068611 1459 7 0.0047978
Vesalius, Andreas 930 2 0.0021505 1174 4 0.0034072
West Nile Virus 245 1 0.0040816 1320 5 0.0037879
Wolfram, Stephen 475 2 0.0042105 559 2 0.0035778
Woodward, Robert Burns 873 0 0 2320 3 0.0012931
Total 22733 123 45254 162
Mean 541.26 2.9286 0.0054106 1077.5 3.8571 0.0035798

* - Articles marked as good articles

  • Word counts were computed by pasting text from the Firefox browser into the Unix wc program. This gives a slight over-estimate because it counts anything surrounded by whitespace as a "word". Britannica counts were taken from Britannica Online. Wikipedia articles were from 14 December 2005 except where otherwise noted.
  • Tables of contents, external links, see also, and reference sections were excluded from word counts.
As a note of caution, since Nature said the lengths of articles they compared were roughly equal, the versions they compared must be different from those whose length we compare here (since our versions are now longer than the EB articles.). This isn't surprising considering the lead time needed to select the articles, send them out for review, gather the reviews, and compile and publish the results. - Nunh-huh 06:43, 15 December 2005 (UTC)[reply]
I find that a bit dubious. By that theory, we should be able to date the review by going back to the last time the articles were comparable to the EB articles (which presumably haven't changed much recently). However, I just checked the Vesalius, Andreas article (which is one of the egregious examples where WP is 10 times the word count of EB), and you have to go back to 2002 to get significantly shorter than it is now, which seems unlikely. I find it more likely that the editors simply tossed out any obvious stubs or near-stubs. —Steven G. Johnson 06:55, 15 December 2005 (UTC)[reply]
So something is amiss. I somehow doubt Nature compared a 62 word EB article on Robert Burns Woodward to his Wikipedia 2300 word article. (And found no significant omissions in EB!) - Nunh-huh 06:59, 15 December 2005 (UTC)[reply]
Oh @#$#@, no it's my fault. I miscounted the EB Vesalius article — it's one of the (very few) EB articles which is spread over multiple pages, and I only counted one page. Similarly with the Woodward article. I'll go back and recheck any others where the imbalance seems to be large. (Update: counts should be corrected now.) —Steven G. Johnson 07:02, 15 December 2005 (UTC)[reply]
If there winds up being a significant difference, then the explanation is most likely error or a difference in versions. Nature specifically states "All entries were chosen to be approximately the same length in both encyclopaedias. In a small number of cases some material, such as reference lists, was removed to make the lengths of the entries more similar." [1]. - Nunh-huh 07:24, 15 December 2005 (UTC)[reply]
Note that the references, external links, and "see also" sections were excluded from all of my counts. I still find discrepancies I can't explain. Our West Nile virus article, even one year ago, was 1108 words or 4.5 times the word count of the EB article. I've searched around EB, and I can't find any huge alternative article on this virus that they could have used instead. —Steven G. Johnson 07:31, 15 December 2005 (UTC)[reply]
Is there an alternative EB website? Are we counting the words in an EB "junior" site? Just a possibility. - Nunh-huh 08:08, 15 December 2005 (UTC)[reply]
According to their website (to which my university gives me access), I am searching the full EB. (They also have "student", "concise", and "elementary" versions of EB, but I'm not using those.) I'm not accusing Nature of dishonesty, but I admit I'm mystified. I wouldn't be surprised if they thought a factor of two was "comparable" length, but a factor of almost 5 seems like a lot. —Steven G. Johnson 08:14, 15 December 2005 (UTC)[reply]
I've emailed the news editor and asked if they could provide the exact article date/time versions that they sent to those 50 experts. That should be useful for figuring out if the errors have already been fixed, and for comparing the versions they sent and the soon-to-be more accurate ones. It's possible that they didn't think to check their versions, but they probably did. I'll pass on the reply when it comes. --Mr. Billion 23:09, 16 December 2005 (UTC)[reply]
No reply. Ah, well. We'll see. --Mr. Billion 19:32, 18 December 2005 (UTC)[reply]

From the editors (see supplementary Nature report above):

Each of the reporters that worked on the survey chose 10 to 15 scientific terms that were roughly in their scientific beat – the sorts of things we ourselves would check in an encyclopaedia. We had not looked at any of these entries in either encyclopaedia when we selected them. Then we weeded out the terms that did not have any entry in Britannica (they all appeared in Wikipedia), and any for which the entries were vastly different in length. Sometimes the lengths were balanced by amalgamating two or three Britannica entries into one coherent piece – for example, 'ethanol' was done this way. We felt this represented 'everything Britannica had to say on the subject' – at least, everything we could find by a quick search of Britannica online, exactly the way a user would approach

So, the criterion was not "vastly different" in length, which would allow e.g. a factor of two difference, and maybe even a factor of 5. —Steven G. Johnson 17:35, 22 December 2005 (UTC)[reply]

Article status

[edit]

None of the articles reviewed have featured article status, and none have undergone our internal peer review process. Two articles, Ethanol and Pythagorean theorem, have good article status with the latter being a former featured article (see here, though there is virtually no discussion there). violet/riga (t) 19:48, 15 December 2005 (UTC)[reply]

Note that Aldol reaction is now tended by a few PhD Chemistry candidates and is FA.--SupperBird 22:19, 3 October 2007 (UTC)[reply]

Tagging every Nature-reviewed articles w/errors an overreaction?

[edit]

Putting a "This article has been identified as possibly containing errors" tag on the front page of each and every articles identified by Nature seemed to some to be a bit of an overreaction. Others were quite comfortable with the practice.

In the long run, it seems that the project survived the tagging of all articles in which Nature's experts found errors. I would suggest that the tagging was even constructive and usefu. I note that we tag tens of thousands of articles as part of the Category:Wikipedia backlog by rank amateurs and teen-agers with éclat. Perhaps our amateurs are simply uncomfortable being reviewed by experts — just as Jimmy Wales finds working with domain experts to be "intimidating".

References:

  1. The Trends Underlying Enterprise 2.0 March 24, 2006. Quote: "Nupedia’s 7-step peer review process was heavily biased toward Ph.D holders and other alleged "true experts in their fields," and was evidently elaborate and daunting.
  2. After failing to make an article contribution to the failed Nupedia proeject, Wales explained years later that even he was intimidated at the thought of submitting an article (in economics: his area of training and experience) to it because was intimidated that experts (rather than amateurs) would review it.Interview: Knowledge to the people 31 Jan 2007.--76.204.176.148 22:15, 3 October 2007 (UTC)[reply]

Correction progress

[edit]

While some progress was made in the days immediately after the report came out, the effort stalled as the Christmas 2005 holiday approached. When one examines the progress made before Christmas, one can see that much of the work was "low hanging fruit" in the form of easy corrections to superficial criticisms or where specific corrections were offered by the expert reviewers.

To be clear: it took several weeks before even 50% of the listed corrections were made. Unfortunately, some Wikipedia commentators like David Weinberger were under the false impression that "almost all" of the corrections were made within the first 24 hours of the effort. See Web of Ideas: The Authority of Wikipedia Around minute 70 of audio track. March 17, 2006. This further propagates the misconception prompted by the fact that Wikipedia has millions of accounts registered. More realistically, Wikipedia has only a few thousand active editors (or ten thousand if you use a generous definition of "active editor"), with varying levels of knowledge and maturity and many of whom contribute little to mainspace articles outside of reverting obvious vandalism. Refer to Wikipedia:List of Wikipedians by featured article nominations which lists appromixate 1000 users for a realistic list of contributors who have ever contributed substantial amounts of original quality content (with perhaps a factor of 5X for substantial sub unlisted collaborators). There are about 1000 administrators, and less than 100 other users with more elevated access. Also, in the new master plan, there will be only about 2000 "trusted" users, but all users will be rated with new "reliability" software to be implemented. Of course, one's personal relationship with Jimmy Wales or one's popularity within the community will still be able to override this number.

If one examines the status at the start of 2006, one sees the pattern: easy fixes or articles with a small number of errors were addressed at the start of the effort. Many of the easy fixes were biographical or narrative in nature and of a "coffee table book" level of understanding typical of an English major. Note that only the Aldol reaction article was a Featured article at the time of the review and that this article is fortunate to have three PhD chemistry candidates as caretakers (having such expertise involved on an extended basis is a rare situation at Wikipedia). The remaining articles required a conceptual understanding of chemistry, math, physics and engineering at around the college undergraduate level. The criticisms of articles such as Archimedes Principle, Dmitri Mendeleev, Colloid, Epitaxy, Field effect transistor, Kinetic isotope effect, Prion, Thyroid were not just syntactical corrections: they required an understanding at the semantic level on these scientifically non-controversial and well-understood topics for which there are, in an absolute and objective sense, right and wrong answers that are not intuitively obvious but require the ability to master technical concepts and facts.

For the academically honest, these later corrections were not subject to public opinion or Wikipedia's style of consensus building akin to voting since most people (and probably most Wikipedians, including the Board of Trustees and past and present members of the Arbitration committee, other WikiMedia foundation advisers and personnel and the employees of Wikia) do not and never will understand these concepts correctly and with depth. One user in particular, Pinktulip, did demonstrate such intellectual mastery and made a steady single-handed effort during January 2006 to ensure that another month did not pass before all the corrections were accomplished. Pinktulip also then informed Jimmy Wales of the completion of the overall effort on 25 January 2006.

Note that Wikipedia:Expert retention is an ongoing problem at Wikipedia.--Simongar 20:14, 26 September 2007 (UTC)[reply]

Britannica's Reply

[edit]

Britannica has replied to Nature's article and claim the nature article was itself innacurate in a number of ways. It would probably be good for Wikipedians to check the sources Britannica gives and note needed changes in our articles.

Nature's Response

[edit]

Nature has now issued comments on Britannica's rebuttal, and stands by its original article. [2] (archive).

Jim Giles made a presentation at Wikimania on 2006-08-04.

Information Quality Discussions in Wikipedia (Fall 2005)

[edit]
  • Source: Besiki Stvilia, Michael B. Twidale, Les Gasser, Linda C. Smith (graduate students at Graduate School of Library and Information Science, University of Illinois)
  • Date: Fall 2005
  • Title: Information Quality Discussions in Wikipedia
  • URLs: http://www.isrl.uiuc.edu/~stvilia/papers/qualWiki.pdf (draft version submitted to CIKM'05)

Findings

[edit]

The focus of the paper was on assessing the information quality of Wikipedia:featured articles — when compared to other samples from the project, including featured article removal candidates, pages marked as NPOV disputes, and a selection of random pages. According to the paper, the study showed how seriously the Wikipedia project views issues of article quality. The authors concluded that as a quality standard, the featured article process "is not ideal, but it does seem relatively rigorous." They also noted that the process is not as resource-intensive as other possibilities, such as blind judging.

Informal

[edit]

Plain Dealer July 2007

[edit]
  • Steve LaTourette: "Non-partisan ratings compiled by National Journal magazine say LaTourette is middle-of-the-road on social issues, not conservative. The National Rifle Association approves of his voting record, but isn't among LaTourette's biggest backers. Ohio's GOP-dominated state legislature eliminated Akron Democrat Tom Sawyer's congressional district after the 2000 Census and gave LaTourette's district its old number, 14."
  • "LaTourette spokeswoman Deborah Setliff said LaTourette's Wikipedia entry contains numerous other mistakes"

Shoutwire

[edit]
  • Source: Shoutwire
  • Date: March 7, 2007
  • Title: http://www.shoutwire.com/comments/56188/The_Accuracy_of_Wikipedia The Accuracy of Wikipedia

Findings

[edit]

This text is copied from the review, which I feel is fair use for improving our encyclopedia:

Sudbury, Ontario is not the most well known city on Earth. I figured that this relatively obscure town would have a paltry Wikipedia entry with very little information, and even less accurate info. I was dead wrong.

The entry for my hometown is absolutely exhaustive. Not a single detail had been overlooked. From history to geography to demographics, the entry could not be more perfect.

Uh oh. Right away I noticed a glaring inaccuracy. Entering “electro-refining” redirected me to “Electrowinning.” Anyone involved in extractive metallurgy can tell you that the two processes are different. Electro-refining should definitely have had it’s own article, since electrowinning pulls the metal out of a solution whereas electro-refining involves a solid anode.

Though they could have gone into more detail, the rest of the article is pretty accurate. That being said, I certainly wouldn’t recommend this article as a source for someone researching either electro-refining or electrowinning.

  • Ore dressing (The author was looking for ore milling, Grade: D-):

I searched for “milling”, and found a very general article about milling. Nothing mining-specific was in the article, although a great amount of information on the process of milling was listed. Since I don’t give up that easily, I tried looking up specific milling-related techniques. Nothing for magnetofloatation, but ore dressing finally turned up a shitty little article. The author could have gone into waaaay more detail. This article is inaccurate because of the amount of detail that has been omitted. My conclusion? Worst. Article. Ever. (Emphasis added is mine. Jesse Viviano 20:23, 8 March 2007 (UTC))

Quite frankly, I expected to be overwhelmed with detail on this one. After all, nickel is something that most people come into contact with every day. However, the article glazed over some of nickel’s unique history, and that kind of irked me because the story of nickel is actually quite entertaining. They could have gone into more detail with the “Old Nick’s Copper” story, for example. Maybe I’m just anal.

The extractive process was reasonably accurate except for what I can only assume was a typo (substitute “sulfide” for “lateridic” in the first sentence) but no information on milling techniques (some specific to nickel) were present.

Other than that, the article was of acceptable quality. My hometown even got a shout-out.

There is really only one astonishing inaccuracy in this one, under the “Comparisons to Digg” section. ShoutWire doesn’t filter the word “Digg” at all. As a test, I submitted an article titled “Digg.com – Best Site On Earth” and sure enough it appeared on the submissions page. The rest of it was OK, though. I found the section on editorials to be quite entertaining.

Responses

[edit]

The Chronicle of Higher Education

[edit]

Findings

[edit]
  • Brave New World (Grade: B-): "The entry provides 'a plethora of links' and more information than the typical literary encyclopedia, says Mr. Firchow, but it is flawed by 'the annoying inaccuracies, the glaring omissions, and the inconsistencies.'" Example: "The article says Huxley wrote Brave New World in Britain and was influenced by Yevgeny Zamyatin's dystopic novel We. Actually, Huxley wrote Brave New World primarily in France and said later that he had not known of We at the time"
  • African-American Civil Rights Movement (1955-1968) (Grade: C): "While there are no glaring inaccuracies, Mr. McAdam says, the entry does not provide any analytical context as to what led to the civil-rights movement or what happened afterward." Example: "Mr. McAdam notes the insufficiency of links to events before and after this particular 13 years of the movement, not to mention the lack of any broader analysis of what brought about the fight for civil rights, and the movement's political and cultural impacts."
  • Flow cytometry (Grade: A): "The main section on flow cytometry was well-done and accurate, ... but as you drop into the subsections, the accuracy drops off. I went in and actually made some corrections to these — I said, 'I can't leave these and let the record stand like this.'" Example: "At one point in the entry, the word "homologous" was misused in talking about blood transfusions."

Responses

[edit]

The Independent

[edit]

Findings

[edit]

The paper reviewed the following articles:

  • Muslim
    • "Yesterday, the entry "Muslim" was changed 12 times. Over the past week, it was changed more than 50 times...these ranged from the puerile...to suggestions which are deliberately offensive...Wikipedia, which constantly checks that changes improve rather than insult, reverted to the previous entry in less than a minute."
  • Russian Revolution of 1917
    • "reads like the work of a second-rate undergraduate student... It is a simplistic account"- Orlando Figes, professor of history at Birkbeck College
  • Kate Moss
    • "Factually, this is dead accurate, though it is cloaked in po-faced language...it does not mention that she never gives interviews and has never been known to purposefully utter a word in public" - Marcel D'Argy Smith, former editor of 'Cosmopolitan' magazine
  • Ann Widdecombe
    • "I think overall that the entry is much better than Dod's parliamentary guide...The references to the 2001 leadership election are categorically wrong...The entry is pretty good though, I would give them 9.5 out of 10." - Ann Widdecombe
  • Tony Blair
    • "That is the problem with Wikipedia - most of it is very good and reliable, but it depends on people interested in a subject being able to pontificate... It is opinionated and written from an anti-war point of view" - John Rentoul, biographer of Tony Blair
  • In vitro fertilisation
    • "This would undoubtedly serve as a useful introduction for those with little idea about the subject; this entry would actually be more useful to the average inquirer with its links than would anything in the Encyclopedia Britannica...its politics are probably closest to those of Liberal Democrats... I am considerably impressed with the quality of information" - Robert Winston, fertility expert and television presenter
  • Philip Larkin
    • "A good and fair account. It sounds approving of Larkin, which is nice, but it is overall a dispassionate account, as one would expect from a dictionary...Though I can see there is an opportunity to whitewash with Wikipedia, the few times I have used it, I have been impressed with it." - Andrew Motion, Poet Laureate
  • BBC Radio 1
    • "Accurate, but with an odd conglomeration of facts without a clear idea of what purpose Radio 1 serves or who listens to it. The odd mixture of facts does not tell you about the wider picture." - Simon Garfield
  • Punt (boat)
    • "I am impressed by the amount of information on punting; the two key books on punting are mentioned, as are the clubs...I am impressed. It works on the presumption that by and large people will correct things, and I changed one small thing on my own biography." - Sandy Nairne, director of the National Portrait Gallery

Responses

[edit]

Most of the articles were tagged with {{High-traffic}}.

  • The criticisms of Tony Blair all relate to assertions added over that weekend to the leading paragraph which were never previously present in the article and were quickly reverted (by me, once I had got back from a weekend break). I think it's a pity because it gives a completely misleading impression of the article. David | Talk 15:34, 13 February 2006 (UTC)[reply]
    • The comment about him "saying he would only serve 2 terms" was only there for 19 minutes before I removed it! -- Arwel (talk) 23:18, 13 February 2006 (UTC)[reply]
      • I think there was another reference which had escaped me which was in the intro and has also been removed (it was a too literal interpretation of the supposed Granita pact). Incidentally the reviewer makes his own howler when he says Lord Liverpool became Prime Minister in 1824. It was actually 1812. David | Talk 09:57, 14 February 2006 (UTC)[reply]
  • The reviewer of the punt (boat) article mentioned two specific minor criticisms: The reference to 1ft is too narrow; the narrowest punt is 1ft 3 inches. The reason given for racing punters to stand in the middle of the punt [...] is slightly bizarre. Thruston has fixed both and added a section on racing. Telsa ((t)(c)) 08:23, 15 February 2006 (UTC)[reply]

Financial Times

[edit]

Findings

[edit]

"To find out how it performs on basic terms in business and economics, we asked Professor Fariba Alamdari, an economist at Cranfield University, to take a look. She says: "I am very impressed. It offers a fairly accurate and comprehensive definition of business terms, with links to other related websites or articles, and sometimes statistics."

Prof Alamdari cites the case of gross domestic product. Wikipedia provides a definition, shows how it is calculated and gives the GDP by country both in nominal terms and based on PPP (purchasing power parity).

She was impressed by the way Wikipedia dealt with such terms as Chapter 11 bankruptcy and return on investment, and those with multiple interpretations such as CRM - which an IT manager would read as Customer Relationship Management but a marketing executive would see as Cause-Related Marketing. Cracks did start to appear, however, with a warning about links to potentially biased commercial sites on the entry for the first interpretation, and no entry at all for the second."

San Antonio Express-News

[edit]

Findings

[edit]

" Well, if you're a sports fan, consider this elsewhere. Among the articles that are substantially correct:

San Antonio Missions: Pretty much on the ball, although describing Henry the Puffy Taco as a "child-friendly mascot themed after local cuisine" seems a little cold and clinical.

San Antonio Texans: The city's one-year adventure into becoming a little bit o' Canada is chronicled in more detail than the Missions' history, which only dates to 1888. It all looks right, too.

San Antonio Spurs: Wow. This reads like NBA.com, which may be why the Baseline Bums aren't included.

Smokey Joe Williams: There's a thorough article about the legendary Negro Leaguer from Seguin, although a link to the San Antonio Black Bronchos, where Williams got his start, is available for the enterprising writer to complete (good luck, by the way).

Ross Youngs: Trust me, this one about our other member of the Baseball Hall of Fame is correct, if too short.

Among those that could use your help:

List of athletes on Wheaties boxes: Even the Wikipedia people admit the list isn't complete, but it is amazing that someone would try to develop one. Among the local notables: Pinkey Whitney, Joel Horlen, Steve Kerr and Dennis Rodman. Among the MIA: David Robinson and Tim Duncan.

And among the missing in action:

San Antonio Tejanos: The one-year experiment in independent baseball, cleverly positioned in San Antonio the year Wolff Stadium opened."

Mail & Guardian

[edit]

Findings

[edit]

Reviewed 8 articles relating to South Africa and gave them scores ranging from 2/10 (Media in South Africa) to 10/10 (South Africa national rugby league team and South Africa national rugby union team).

Responses

[edit]

Many of the articles were quickly improved and the Mail & Guardian published a second article "Wikipedia springs into action after M&G Online article" which said, in part, "...most of the entries have been edited and improved."

The Guardian

[edit]

Findings

[edit]

Experts review 7 articles and rated them on a 0 to 10 scale. Controversially Robert McHenry (former Editor-in-Chief of Britannica) reviewed the encyclopedia article. The scores ranged from 0/10 (Haute couture) to 8/10 (Bob Dylan).

Article name Mark out of 10
Steve Reich 7
Haute couture 0
Basque people 7
TS Eliot 6
Samuel Pepys 6
Bob Dylan 8
Encyclopedia 5

Roanoke Times

[edit]

Findings

[edit]

Three college professors were asked to evaluate articles in their areas of expertise.

  • "Virginia Tech geologist Bob Bodnar looked up the entry on the geologic time scale. 'I found the terminology and ages used to be quite accurate and consistent with the most recent data,' he said."
  • "Roanoke College political science professor Bill Hill looked up 19th century politician John Taylor of Caroline County, political philosopher Alexis De Tocqueville and the Federalist Papers. 'In each case, the program responded with information that was accurate and pertinent, if brief,' he wrote. 'I thought the coverage of De Tocqueville, however, was too superficial.'

    (Note: De Tocqueville's entry weighs in at 1,011 words, while Katie Holmes merits 1,893.) Hill said he would discourage his students from using it as a 'serious source,' but noted that he discourages the use of normal encyclopedias as well.

    A feature of the Wikipedia that Hill likes are the links to original sources. It's pretty handy when reading about De Tocqueville to be able to click a link and read the man's own words. But Hill also noted a link that led to a "very brief and shallow essay" about states' rights."

  • "Dave West, a retired Virginia Tech biology professor, looked up three biology-related sites. One was good, but two were bad.

    To the average person, the article about 19th century naturalist Fritz Muller would look very detailed and well-researched. But to West, who has written a biography of Muller, it was 'fraught with errors.'

    West listed more than half a dozen of these, including the wrong year of birth and a picture of Muller that's not actually Muller." [West helped to correct the errors.]

St. Petersburg Times

[edit]

Findings

[edit]

"The St. Petersburg Times recently asked two University of South Florida professors to read a few Wikipedia articles on topics in their expertise. Chemistry professor Bill Baker said he was surprised at the amount of technical knowledge posted on the site, but said he found several small errors. "The cancer drug Taxol, for example, is not produced by microbial fermentation."

"That bothers me," Baker said of the errors. "I think that even if 99 percent of your facts check out, it is a disservice to promulgate 1 percent inaccuracies."

Professor Philip Levy, an expert on Colonial America, said that "in many respects it's very good," but he, too, had misgivings. He said some articles contained a mishmash of information - the established scholarly knowledge of 15 years ago was mixed in with newer, more controversial theories with little distinction."

Edmund Evans Expert Review

[edit]
  • Expert: Anne Lundin
  • Expert background: Anne Lundin is a retired professor from the School of Library and Information Studies University of Wisconsin, Madison. She is knowledgeable in Victorian picture books in England and the United States and has documented the history of reviewing of children's books in the late 19th century. She is an expert in Walter Crane, Randolph Caldecott, and Kate Greenaway and has published several books/papers on the topic
  • Date: July 28, 2010

Findings

[edit]

The response was:

Excellent article! I corrected the date for Greenaway's UNDER THE WINDOW which is 1879. Congratulations on your exemplary piece, rich and full. - Anne Lundin

Response

[edit]

[Citation Needed]

[edit]

Findings

[edit]

A blog that continuously publishes dubiously correct, informal, or humorous excerpts from Wikipedia articles. Most of these are not yet fixed.

See also

[edit]
Retrieved from "https://en.wikipedia.org/w/index.php?title=Wikipedia:External_peer_review&oldid=1186960867"