I had the idea that it would be worthwhile to assemble a panel to peer review sections of Wikipedia, within particular academic disciplines, and “freeze” a chunk of the site and certify it through peer review. That would have a duel purpose. First, it would provide an opportunity to measure the wiki, and second, it would provide some form of external validation — using traditional criteria — for the site.
At least the first part of this seems to have been accomplished. A project undertaken by Nature. Basically, they picked 50 representative articles (this does not appear to be a sample, and I wonder how they arrived at these), and put articles from Wikipedia and Britannica out for review. Britannica won, a fact that is not stressed in the Nature write-up. Instead, they show that Wikipedia is flawed, but so is Britannica. The average Wikipedia article has four innaccuracies, to Britannica’s three.
Of course, such a measurement is fuzzy. After all, if you extrapolate this to the total number of articles in Wikipedia, that’s a whole lot of inaccuracies.
I’ve started working with a doctoral student, Derek Lackaff, to take another approach to identifying the validity of Wikipedia, hopefully one I’ll be able to talk about more in the spring.