The Isuzu Experiment



Update (9/5): Please don’t do this: vandalizing the site is not a good way to test it. If you want to test Wikipedia, please do it non-destructively.

Joi Ito points to an ongoing discussion regarding the authority of wikipedia as a source of information and knowledge. The discussion was prompted by an article in the Syracuse Post-Standard that suggests, in part, that wikipedia “take[s] the idea of open source one step too far” by allowing the user to make corrections.

The article has been correctly ridiculed by many, including Mike at Techdirt. In a later posting, he suggests an experiment: why not go to a certain page, insert something provably incorrect, and see how long it lasts.

No matter which side of the debate you find yourself on, this sounds like an interesting experiment. So, I have made not one, but 13 changes to the wikipedia site. I will leave them there for a bit (probably two weeks) to see how quickly they get cleaned up. I’ll report the results here, and repair any damage I’ve done after the period is complete. My hypothesis is that most of the errors will remain intact.

Does that invalidate Wikipedia? Certainly not! If anything, the general correctness and extent of Wikipedia is a tribute to humankind. It suggests the Kropotkin may be right: that the “survival of the fittest” requires that the fittest cooperate. It means that there are very few Vandals like me who are interfering with its mission.

This entry was posted in Uncategorized and tagged . Bookmark the permalink. Post a comment or leave a trackback: Trackback URL.

27 Comments

  1. Posted 8/29/2004 at 5:16 pm | Permalink

    Interesting experiment indeed. What about experimenting with things that are obviously wrong, some that are factually wrong but are popularly believed and things that are debateable (but not hotly debated because those topics do get a lot of attention).

    P.S. your first link on this post is a little messed up

  2. Posted 8/29/2004 at 5:50 pm | Permalink

    Thanks, David, I relinked.

    All the changes were “factual” in nature, though some fairly obscure. And *all* were identified and removed within a couple of hours. I could have been a bit trickier in how I made changes; nonetheless, I am impressed.

  3. Posted 8/29/2004 at 8:30 pm | Permalink

    Alex,

    Great! I’m really happy that someone took me up on the experiment — and the results (which surprise me a bit as well). That’s great news. I’ll be sure to let Al know next time he emails me (there have been many more this weekend…) to tell me how awful Wikipedia is.

    To be honest, I thought about doing the experiment myself, but couldn’t bring myself to make a change. It just felt… wrong.

    I would be interested, also, in other suggestions on ways to make the experiment even better — it would be nice to have this to show to others who trash Wikipedia as well.

    Finally, I did notice that *someone* (don’t know if it was you) put up Al’s name as a “famous Syracusian” and the page was reverted in 45 minutes (with a note saying “very funny”).

  4. Posted 8/29/2004 at 8:48 pm | Permalink

    I know what you mean about feeling bad about it. It was actually painful for me to do this in some cases, which may be the reason it doesn’t happen too often. I tried to be a bit more subtle on my change to the Syracuse page (saying that Frederick Douglass had lived there for some time–relatively plausible given his time in Rochester).

    If I were to do it again (I won’t), I would have spread my changes out over a few days, and posted from different username/ip addresses. I know that some of the more obscure changes were caught in a chain from some of the less obscure ones. I guess I don’t make a particularly good wiki troll; which isn’t a terrible thing to be bad at.

  5. Posted 8/29/2004 at 9:06 pm | Permalink

    Alex, you may have forgotten that there’s a (fairly large) number of people who read http://en.wikipedia.org/wiki/Wikipedia:Recent_Changes religiously (usually as an RSS feed).

    So when you made your change, it popped up on a lot of screens — and one or another glanced at it, said “that’s wrong,” and fixed it. Indeed, check out the RC Patrol http://en.wikipedia.org/wiki/Wikipedia:RC_patrol for some idea of how this happens.

    One of the results of Wattenberg & Viegas’ “Historyflow” work ( http://researchweb.watson.ibm.com/history/gallery.htm ) is that because of the recent-changes watchers, changes to Wikipedia tend to be catacylsmic and sudden: a great deal of interest gathers around a single page, many changes are made, and a new steady state is negotiated. (Look for pictures in the their gallery labelled as “indexed by time” to see how this works.)

    The short version is: with “The Magic of RSS”, stuff that gets changed needn’t be unearthed–it’s right there to see.

    I look forward to your writeup.

  6. Posted 8/29/2004 at 9:39 pm | Permalink

    Not much to write up, beyond what you see here :). I went, I changed, they conquered.

    I did figure the changes would largely fly under the radar with the several hundred changes made today. Guess I misunderestimated ’em.

  7. Posted 8/30/2004 at 6:17 am | Permalink

    I don’t know how you made your changes, but if you just went about and changed a number of articles, it will be likely that seeing one of your changes being wrong, someone would do a check on your other changes. So unless you took the precaution of making your edits from different IP numbers/user names, the various chances basically improved each other’s chance of being reverted.

  8. Dan Smith
    Posted 8/30/2004 at 8:00 am | Permalink

    Alex, would you go out and spray-paint graffitti on a school building as a way of testing to see whether the school has janitors?

    Don’t your actions feel a little bit irresponsible to you?

  9. Posted 8/30/2004 at 11:09 am | Permalink

    Andre: Yes, you are right, I think that several of these were discovered because people looked to see what other changes were taking place. Nonetheless, three different editor/readers independently found errors within a couple hours, so I suspect the others would have been found fairly quickly.

    Dan: I did consider this before I made the changes. I think your analogy is flawed. A better question would be whether I would chalk the walls or drop dirt on a floor to see whether a school’s janitors were doing their job. This is exactly what Michelin and others do to evaluate hotels.

    As always, you have to weigh the costs of something like this against the benefits. The costs included time spent by editors to review the changes, and the possibility that someone might be misled by the changes until they were fixed. The outcome was to provide a demonstration that the system works. Because I did this, and made it public, it means others do not have to: there is a public demonstration of how well the system works.

    Often, the only way to detect the weaknesses in a system is to attempt to behave like someone willing to exploit those weaknesses. If the changes had not been detected, I (obviously) would have removed them myself. The net impact would have been minimal, but it would have highlighted a flaw in the system that could be exploited, and would have called into question the veracity of the content currently hosted.

    So, I made the determination that the social costs involved in testing the system were worth the social rewards of a demonstration of its strength. I wish I had designed my attack a bit better (see above), but I think that the public knowledge that false statements are quickly dealt with helps to bolster Wikipedia as a more reliable source.

  10. Posted 8/30/2004 at 2:42 pm | Permalink

    Your penance is to fix errors in ten articles, or start three new ones.

  11. Posted 8/30/2004 at 3:26 pm | Permalink

    David G.: Will do.

  12. Posted 8/30/2004 at 7:57 pm | Permalink

    that’s an interesting experiment, but it’s not really comparable to how actual errors would be introduced into wikipedia articles. first, as others have noted, the errors would come from different sources (different IP addresses). also, they wouldn’t be announced on the authors’ websites. i’d like to see how well this would work with these aspects changed. of course, given the criticism you’ve recieved already, i wouldn’t expect you to do it again.

  13. JohnDoe
    Posted 8/31/2004 at 4:23 am | Permalink

    for recent change, there is also #enrc.wikipedia on freenode irc server, but it’s not working as well as it used to work.

  14. Chan Lee Meng
    Posted 8/31/2004 at 6:33 am | Permalink

    Err… how will this prove anything? It’s like announcing you’re going to hijack a plane while you’re at the boarding gate. Do you think anyone will notice?

    For this to be a valid experiment, no announcement should be made until two weeks after.

    “The any fool can go in and insert rubbish” concern is valid. That it took hours to correct the damage is not really cause for celebration. Someone could still have accessed the errorneous data during that time.

  15. Posted 8/31/2004 at 10:59 am | Permalink

    brilliant! I’m still confused by the people who complain about Wikipedia, but don’t go nuts over the factual errors in printed works. Wikipedia can at least be fixed. For my money, the W’pedia is the best reference on the net.

    Even cited it in my master’s paper…

  16. Anonymous
    Posted 8/31/2004 at 11:00 am | Permalink

    Like any good scientist, you must reveal your method. What changes did you make?

  17. Posted 8/31/2004 at 1:43 pm | Permalink

    First, I think your method is flawed, as 13 changes by one user to multiple entries would certainly be detected, and would come under serious scrutiny as soon as one of them turns out to be incorrect.

    Second, your experiment proves that the concept of building a wiki-based encyclopedia is flawed, because any unsuspecting user who read the article(s) that you changed before they were reverted received wrong or misleading information. That’s because encyclopedias are supposed to give information seekers correct information at any given time, not prove that they are self-repairing knowledge-building processes.

    Third, I have been monitoring five articles on the German wikipedia with multiple errors in them for about a month now. In that time, two typos, but none of the factual errors were corrected, and one article was bloated up to twice its original size without adding any additional relevant information.

  18. Posted 8/31/2004 at 2:18 pm | Permalink

    Some people are missing the point, which is that “errors” are weeded out by consensus, not by editorial fiat. This isn’t a weakness, but rather an aspect of the same strength that makes Wikipedia possible in the first place. Namely, the people who use Wikipedia have a vested interest in high-quality content, so it makes more sense to provide them with the means to do so than to impede that process.

    Saying that Alex’s test was “too obvious” is another misunderstanding of the challenge. If Alex’s casual changes were spotted and corrected much more quickly than even he imagined, what does that say about “real” casual vandals and their chances?

  19. Posted 8/31/2004 at 3:10 pm | Permalink

    Alex, if all your changes got reverted, then I think the experiment may in fact have failed. The strength of Wikipedia should be to correct errors, not to remove them. (Of course, this depends on the nature of the errors you introduced.)

    @Horst: according to your own argument, any encyclopaedia that contains at least one error is flawed. Since I find it hard to imagine that there exists one perfect (printed) encyclopaedia, that would mean the whole concept of encyclopaedias is flawed. I think you might want to take a second look at your reasoning.

  20. Posted 8/31/2004 at 6:29 pm | Permalink

    Can you share the changes (or your username/IP address so we can look them up)?

  21. Posted 9/1/2004 at 1:39 am | Permalink

    @Branko: No, that was not my point at all. My point is about knowledge building vs. knowledge retrieval, and how a constant building process (as in a wiki) is not helpful for a user who is looking for information.

    Wikipedia is a great experiment in knowledge sharing and knowledge building, but it’s useless for anyone who is not interested in sharing or building, but instead wants quick, correct answers.

  22. Posted 9/1/2004 at 2:46 am | Permalink

    To Horst and all others who believe there is something like “one absolute correct answer” for topic descriptions which one would find in a ***pedia: Think again. Truth is a fragile tissue people gather around them. The fragility of it enables you to develop. But if you wrap yourself in it too tightly you will loose the broader view on temporal reality or worse get smothered.

  23. Carol Dgill
    Posted 9/1/2004 at 3:00 am | Permalink

    The right way to validate Wikipedia and the strength of this open-editing model is to find some factually incorrect material and use the change log history to determine how long it has been there.

    Intentionally falsifying information as an experiment is no different then pentration testing (old school hacking) but it does disrupt other people’s legitimiate use of the resource.

    If you really want to test: don’t make small one-sentence errors that anyone can verify with google. Make a substituantal contributation with an error in it. Most people go for the easy fixes – they are less likely to question three long paragraphs of well-written text.

    Anyone can identify grafitti done by a vandal, which is what your experiment simulated. It is much harder to identify a well-done Van gough fakery.

  24. Drew
    Posted 9/1/2004 at 10:07 am | Permalink

    Interesting experiment. I’m not sure what it shows, exactly, apart from the active response of the correctors (and whether they found out through their own searches on their particular speciality topics or through recent changes updates remains to be seen.) The other issue here is that announcing that there have been changes made before the experiment is finished immediately sends loads of people running to wikipedia to see if they can find the false errors. If I hadn’t read the comments, I’d have done it myself. So perhaps it doesn’t really say much about the equilibrium of the system if a percentage of the error-seekers aren’t normally users of the wikipedia.

  25. Posted 9/1/2004 at 10:45 am | Permalink

    I’ve seen lots of incorrect entries on Wikipedia but an even more common issue is very politcal statements in many articles. Particular any articles about computers and software.

  26. Stephen Gilbert
    Posted 9/6/2004 at 10:52 am | Permalink

    Yes, the windows *are* made of glass. Now Wikipedia has to deal with random people throwing rocks from random directions. Purposely vandalizing a resource like Wikipedia as a “test” has inspired a slew of copycats with good intentions but poor judgement.

  27. scott preece
    Posted 9/7/2004 at 2:32 pm | Permalink

    It’s worth noting that real (print) encyclopedias often have false entries, as do most commercial maps, phone books, etc. These are inserted by the publishers as IP-protection (if you see them repeated in somebody else’s book, you know they copied yours illegally).

    I would tend to agree with previous comments that the right metric here is the average age of removed errors – that is, scan the change log for changes that fix factual errors, then average the time between introduction of the error and removal of the error.

23 Trackbacks

  1. […] validity and reputation of the Wikipedia, the open source encyclopedia. The experiment by Alex Halavais was to place 13 demonstrably wrong items into the W […]

  2. By Alex Halavais » Return of Joe Isuzu on 9/1/2004 at 2:39 am

    […] through the summer and then the first day of classes, when I have no chance of keeping up, I do something that brings a buttload or two of people to my blog. So, with apologies to the shotgu […]

  3. […] colosa”. L’esperimento è stato allora intrapreso da Alex Halavais, che ha inserito 13 errori in 13 articoli. L’intenzione era di lasciare gli errori un paio di settimane, e in […]

  4. […] around in a library and create an escalating snowball of bad information. There are cases like Alex Halavais’ intentional introduction of false information that might temporarily pois […]

  5. By Irregular Verbiage » Half-way there on 9/5/2004 at 3:00 pm

    […] nsert subtly wrong information and it won’t be detected which is a shame as earlier, less subtle attempts have failed. It’s not going to stop me using wikipedia, and given enoug […]

  6. […] e by Robert Paterson I shortly noticed these two: Adina Levin in describing the recent experiments to test Wikipedia’s resilience spots that the real power of Wikipedia’s def […]

  7. […] This is old news now, but the School Library Journal ran a follow-up article on the piece critical of Wikipedia that appeared in the Syracuse Post-Standard. Tavis Hampton, a media spe […]

  8. By Alex Halavais » Short notice on 6/4/2005 at 11:30 pm

    […] t on Wikipedia’s Heavy Metal Umlaut pageA Wired News story on Wiki NewsMy attempt to break Wikipedia. Sorry again for the short notice; please don’t take it as a model _. […]

  9. By Many-to-Many on 8/29/2004 at 9:57 pm

    Wikipedia Reputation and the Wemedia Project
    The core issue of collaborative editing, that of accuracy and trust, has reached a point in debate where research is needed to advance the practice of content use and development. Hiawatha Bray of the Boston Globle offered a Wikipedia criticism…

  10. By Collin vs. Blog on 8/29/2004 at 10:02 pm

    Better to be ignored?
    Opened up Bloglines to find that “Syracuse” is showing up across many a radar today. Unfortunately, it’s for the wrong reason. Turns out that Al Fasoldt, a staff writer for the Syracuse Post-Standard, has written an article criticizing Wikipedia. Tu…

  11. By j's scratchpad on 8/30/2004 at 12:49 am

    More on Wikipedia’s Reputation

  12. By The Importance of... on 8/30/2004 at 2:31 pm

    The Great Wikipedia Authority Debate
    Last Wednesday, the Syracuse Post-Standard published an article lambasting the authority of Wikipedia because it is user-edited and anyone can make a change to its content (Librarian: Don’t use Wikipedia as source). Techdirt took the article to task f…

  13. By Chromatic Musings on 8/30/2004 at 8:56 pm

    The power of millions of geeks united
    A few days ago, Al Fasoldt wrote an open letter to librarians in the Syracuse Post-Standard to not use Wikipedia on the grounds that it couldn’t be trusted (The Wikipedia is an encyclopedia editted by the users). Challenged by some…

  14. By The Aardvark Speaks on 8/31/2004 at 2:46 pm

    The neverending story
    Alex Halavais proves that Wikipedia is not trustworthy and curiously interprets his experiment as a proof why Wikipedia works. In the meantime, a number of Wikipedia advocates attack a journalist because he quoted the Wikipedia disclaimer and was disap…

  15. By A Bluegrass Blog on 8/31/2004 at 4:18 pm

    On Wikipedia
    I love Wikipedia, an open encyclopedia that is built from the ground up by volunteers. Anyone can create an entry, and anyone can edit any entry (and, in turn, have their revised entry edited by someone else). Wikipedia is an

  16. By Coffeehouse at the End-Of-Days on 8/31/2004 at 5:13 pm

    The Isuzu Experiment
    I love the idea in theory of a Wikipedia-like experiment with collaborative editing by Christians of a biblical ‘commentary-concordance’. Two huge potential problems are the, um, theological differences between us (since Wikis, by intention, allow ed…

  17. By nBLOG on 9/1/2004 at 5:43 am

    Web News
    And another round of web news! Woman Chows 38 Lobsters in Eating Contest Jibjab “This Land Is Your Land” parody wins legal battle sIFR – dynamic browser text replacement MovableType 3.1 released The Giants of Anime are coming Tarantino&#821…

  18. By Made out of people on 9/2/2004 at 9:09 pm

    Life on the Virtual Commons and the Wikipedia
    Alex Halavais tried an experiment in which he slightly defaced 13 Wikipedia pages. To his surprise, all were fixed within hours. On Alex’s blog, I noted that Wikipedia makes a point of looking for damage, and tries hard to fix…

  19. By Preoccupations on 9/3/2004 at 8:58 pm

    Online communities, tags, authority and reputation
    For myself, I found that I was fitted for nothing so well as for the study of Truth; as having a mind nimble and versatile enough to catch the resemblances of things … and at the same time steady enough

  20. By Irregular Verbiage on 9/5/2004 at 10:00 am

    Half-way there
    So I’m busy writing my thesis at the moment which is why this blog is semi-dead, I guess you could say that it’s in a coma as it should recover when I finish (my thesis is due to be finished on the last day of this month!) I think at the moment, I’m…

  21. By Many-to-Many on 9/5/2004 at 1:05 pm

    Stress Testing Wiki Authority
    Slashdot has picked up on the Wikipedia accuracy/authority/reputation meme as Frozen North did the same Techdirt experiment as Alex Halavais. Not so needless to say, testing the rigor of Wikipedia by vandalizing a community resource isn’t the bes…

  22. By Mario tout de go... on 9/5/2004 at 1:23 pm

    Toutes les expériences ne sont pas bonnes à tenter…
    Alex Halavais est une personne expérimentée en matière d’utilisation des nouvelles technologies au service des apprentissages. Dans le contexte où nous venons de débuter l’utilisation d’un wiki, j’ai lu avec beaucoup d’intérêt ce billet du type “…

  23. By Synesthesia on 9/7/2004 at 2:19 pm

    Reed’s law in several places
    It is the nature of mind to filter experience through the most recent or most strongly held concepts.

    No wonder then that after (finally) coming across Reed’s Law thanks to this thought-provoking article by Robert Paterson I shortly spotted these…

Post a Reply to Dimitar Panayotov

Your email is never published nor shared. Required fields are marked *

*
*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>