Off for Denmark in a few hours. Here are my slides for my Tuesday morning presentation. I didn’t have time to do my now standard video-based presentation style. Much more basic this time around…
Trying Google Presenter, too.
(This is the second in a series of posts on the making of a research article. If you haven’t already, go back and check it out from the start.)
Overlapping sheets, not a point
A common response I get back from undergrads undertaking their first literature review is that they cannot find anything. This is usually an indicator of a systematic problem, since a usual issue is having too much to cover in a lit review, not too little. Generally, this comes in the form of “I can’t find anything on advances in the pizza delivery business in Peru during the 1990s.” If you could find something exactly on point for your work, you might have a problem: after all, what are you adding to the conversation?
While you should definitely look for “near misses” for your own narrow research topic, generally you are going to have stretch out a little broader: any papers or books on pizza delivery are probably worth checking out, whether or not they are in Peru, related to business, or set in the 1990s. Likewise, you are probably interested in the food delivery business in Peru more broadly as well. In other words, by assembling a quilt made up of existing slips of material, you can shape your own literature. Your work should be the point that binds together otherwise relatively disconnected pieces of work. Some of these may not fit together so neatly, others are in much better shape.
Keeping track
Generally, I keep a few documents going,at least when I am organized enough to do so. The main document keeps track of things I’ve found. This includes a full citation, and either relevant quotes (clearly indicated with quotation marks so that I don’t make mistakes later) or summaries of the material. (Many people keep these on notecards, or the digital equivalent, but I have always just kept it all in a document.) A second document records the citations and other information of things I think I should take a look at. A third document includes a list of key search terms and authors that might help in searching for materials.
I generally start with the last list, looking for a set of keywords, works, authors, or phrases that I can use to locate more information. I get these from my own assumptions, or sometimes from places like Wikipedia and other general sources of information. I then use these keywords in a series of places. I generally begin on Google Scholar, these days. I might also make use of ComAbstracts and similar resources. From these, I end up with a set of citations of things I should check out. For articles, I can generally check them from home. These days, I am also likely to check Google Books to see whether I really need a particular book or not before making the trek to the library.
As I go through these, I add to the search bibliography and search terms documents. Often, an article will contain little of interest other than its reference to other literature. It is a very iterative process. Eventually, I feel like I am seeing the same citations and names consistently, and have a handle on that particular segment of the literature.
Finally, for larger projects I try to assemble the bibliographic resources in a citation manager. I used Endnote for many years, with varying degrees of success. I have now switched over to Zotero, and finding it useful. Zotero is a free plug-in for Firefox that serves as a bibliographic manager, and provides a nice way to organize your resources and notes on the things you are citing. No matter your approach, make sure you have complete citations organized in some electronic format. Cleaning up citations and making certain the bibliography is correct seems consistently to take me three or four times as long as I expect it to.
As a practical matter, the above (like the cake) is a lie. I have generally internalized those three documents, and write the lit review as I am collecting information and sources. (Now, if only I could–like Jeremy Hilary Boob–write a positive review of my article as I was composing it!) If, however, you don’t have a lot of experience writing this sort of stuff, a bit of structure can’t hurt, and the “three document” approach has worked for students in the past.
Useful Areas
So what sort of things am I looking for? Obviously, I should at least look at anything that touches on Digg and the culture surrounding it and other collaborative filtering sites. There is surprisingly little literature, I suspect, that deals with the current crop of collaborative filters that involve interaction (Digg, Reddit, StumbleUpon, flock, and several dozen others). Much of what is out there will be technical in nature, and less likely to go into much depth in terms of users or social issues.
I also want to lay the foundation of operant conditioning, particularly as it applies to user interfaces and behaviors within a discourse community. It’s been a long time since I encountered this stuff in intro psych, and I’m not entirely sure where I’m going to find stuff here, or if I will. I suspect that since creating motivation to participate is particularly important in the area of education, there may be some work that looks particularly at reward systems and how they affect behavior in that literature.
Clearly, some of this has to do with the question of how people join groups and become acculturated to them. Although it is rare that such processes have such an obvious marker of success, it may be that there are models that can be applied to the processes observed on Digg.
There may be some work on reputation systems that is applicable. In particular, systems like those found on eBay (where you can be ranked up or down by those with whom you have interactions) might be applicable also to this work.
Finally, I’m hoping I can mine some of my own material. In fact, I’ll probably start there. I presented a paper about Digg and elections at the National Communications Association meeting last year, there is a paper I worked on with Alex Tan that analyzed the structure of eBay ratings, and, as I mentioned, I may be able to draw on a chapter from my dissertation.
Reputation Systems
Since I’m treating this a bit like a cooking show, I won’t go through the entire process, but I’ll walk through one of those pieces that will be woven into a literature review: the literature on reputation systems. As I noted, and to the consternation of traditionalists and my own surprise, Wikipedia has become a useful part of my research process, so I head over there for a survey.
Although the entry itself doesn’t go much beyond a definition, it does link to a few articles, including one I vaguely remember reading in CACM, and more importantly to a treasure cache: this site, which includes a bibliography of relevant research papers. Yay! Always nice when someone does a lot of the work for you. I go about assembling these, looking for information in them that informs my own work, and looking through their bibliographies to find common ancestors and theoretical foundations that are shared.
I also assemble a set of key phrases that seem to be coming up and run them through Google Scholar. Google Scholar is nice because it provides the ability to easily search for citing documents, and move forward to the most recent literature. In all, reading and writing about reputation systems takes only an hour or two, and yields about a paragraph of my literature review.
As with writing generally, if you are stuck, write something. Extract some reference, any reference, and start tunneling into it. Generally being stuck means you are not moving, and the easiest way to start moving is simply to start moving–in any direction at all.
And Everything Else
I generally find that I need to revisit the literature review after the research itself is done, but after working though the areas above and finding out how they connect, I’ve managed to set the foundation of my research, show that it hasn’t already been done (a common issue), and show that the questions I am asking are interesting and important, and will help to build on the existing literature. I ended up organizing the brief review under three headings:
* Filtering People and Content, where I talk about reputation systems and collaborative filtering
* Digg, where I talk more about how the site works
* Responses to Evaluation, where I look a bit at forms of learning and the process of becoming a member of a community.
From here, I have to figure out how it is I am going to do what I am going to do. Generally, this means planning out a method in some detail. The reasons for this are varied, but particularly in research that is collaborative, you will find that you need to propose the research before you can move forward. This may be a proposal to your committee, if you are a graduate student, or to a funding agency. If you are making use of human subjects, you will have to describe to a human subjects board what you plan to do in order to gain their approval.
Here, I know what I need (the public data from a sample of Diggers, organized in such a way that I can make sense of it), and so it is a matter of muddling through. In the next installment, I deal with getting an initial sample.
I’ve been meaning to do this for a while. When I get questions from students and colleagues, it is rarely relating to things I’ve found, but how I found them. How did you get that data and make sense of it? I do a poor job of explaining this, and many are unhappy when there isn’t a simple software tool that can accomplish what they want.
One of the advantages of blogging is the potential to live other people’s lives, and, as a result, learn a bit more about what they do. For an academic, blogs can (and often are) used to work through ideas, but more rarely is the process of doing research and writing threshed out. I’ve decided to do that for a paper I am writing, providing you with a blow-by-blow feel for how I do research, and how I write.
What I describe in the following posts isn’t the only way to do things, and it’s not even the normal way I do things, if such a thing exists. The process will change depending on the object of study, the type of work, and the team (or individual, as in this case) working on it. But hopefully it will provide you with some idea of how I go from nothing to a (hopefully interesting) research article.
There are dangers here. The first and foremost is looking like a complete idiot. But I’m kind of used to that, so I’m not going to worry too much about it. Hopefully, I’ll learn something out of the process. Another is that it messes with blinded peer review, but I’ve found that is already pretty messed up. If you have questions or comments, I’m very happy to hear them!
Ideation
The first step in any piece of research is thinking. I know that sounds obvious, and I only wish that it were. The idea is to come up with a piece of “specified ignorance” as Merton put it: something we should know that we do not know. (Note that this is different from “I can use method X and theory Y, now I’ll apply it to yet another set of easily found data.) It also should be something that you find to be exciting. Others will tell you that there are other requirements: that it be part of a sustained research agenda, that it be fundable, that it be of current interest to the profession, etc. I won’t disagree with any of that, but I don’t personally care very much about those things. I’m generally taken by fairly disparate kinds of issues and just let my curiosity get the best of me. Unsurprisingly (at least to me) these end up clustering together into some form of research agenda on their own.
Usually, I am inspired by “that’s cool” moments. A year or more ago, I found a lot of my time taken up by the Digg site. As it happens, I think that it and similar social filtering sites are really important to how the web works today, and we need to understand them better in general. I guess you could say they are becoming a stream in my research. But at the time, I just thought a tool that let you view your posting history, from Neaveru was pretty cool, and suggested some patterns in people’s behavior. Despite a self-image that claims not to care what other people think, I got pleasure from being widely “dugg,” and was frustrated when my comments were dugg down.
This got me thinking about the new explicit kinds of ratings of people that seem so common on the social web, from Technorati to Compare People to numbers of Twitter followers. All of this starts to feel a bit like distributed whuffie.
Of course, I’ve been thinking about related stuff for a while. And this new interest echos back strongly to a bit of a side-track I engaged in for one of the chapters of my dissertation (pdf). I never did the “make your dissertation a book” thing–I was sick of it at the time and now I wish I had done more with it after graduating. Anyway, I can at least fall back on some of the ideas I engaged there, including an analysis of how experience on Slashdot led people to learn to get more votes up from moderators.
Provisional Research Questions
So, I know I generally want to look at the function of the ratings of comments on Digg, and how they might be related to posting behavior. Although I’ve never been a fan of applying the strict hypothesis model to the social sciences, I do think you need to have some clear ideas of how you want to measure and operationalize your ideas. First, I need to narrow things a bit. Note that I expect that some of these ideas may not make it to the finished product (i.e., they may end up being stupid questions), and I may run across other things in the process that make me revisit my initial ideas.
I suspect, based on the work I did with Slashdot, that people learn to get better “grades” the longer they post on Digg. I’m not sure what “longer” really means, whether it is number of posts or amount of time, but it’s possible to check both. In the case of Slashdot, there appeared to be a “learning period” during which there was an increase in average post ranking. After this learning period, some people continued to post popular comments, while others abandoned popularity for obscurity, or actively posted trolling comments that would get ranked down.
The response of fellow Diggers to a comment also calls to mind something approaching a Skinner box, with small treat rewards for conforming behavior. I wonder if there is a direct measurable effect of positive or negative reinforcement of posting behaviors.
And, in a “mushier” sense, I’m wondering if there are traits that make it more or less likely for a comment on Digg to be voted up or down. I suspect that if I tried posting in French or Japanese, or just posted total nonsense unrelated to the discussion, this would get me voted down. So, being on-topic and writing in a language and style that can be comprehended seem important to a good Digg comment, but can we find certain kinds of things that tend to improve your chances. On Slashdot, humorous remarks were key to high rankings, for example.
So, I’m interested in three things (in a slightly different order than they are addressed above): Is a highly ranked post more likely to encourage a new post sooner? (And perhaps: is dropping out of posting related to consistently being Dugg down?) Do people tend to learn how to post more successfully over time? What content seems to result in the highest ranking?
From here
The truth is that my research and writing process is often pretty muddled, and I cannot promise that this will be any less the case this time. That said, in the following parts, I’ll address the process of assembling a brief literature review, writing up the method, collecting the data, analyzing the data, writing up the research, and going through the conference presentation and publication process.
I haven’t given up on blogging, this has just been one of those semesters. I would like to be keeping everyone up to date, but I just can’t seem to find ten minutes to update my main blog–though I’ve been posting elsewhere. So what is it that is keeping me so busy? Here are the projects I’m working on through the end of the year.
Projects Midway
These are things that are in some stage of completion, though some still require quite a bit of work.
1. National Communication Association conference
I’ll be late for the start of the NCA conference because I didn’t want to miss another set of my grad seminars, but I’m on a flight at an ungodly hour of the morning tomorrow for Chicago to present a paper entitled “Cutting Paths to Political Candidates: Technologies for Social Findability.” I’m not at all happy with the paper (I haven’t been happy with any of my papers this semester), but I’ll decide over the winter break whether to overhaul this one and polish it or roll it into a comparison with the presidential election.
I probably won’t be blogging the conference, since my lappy is still not working properly.
2. Introduction to Interactive Communication Seminar
Tonight was the last “content” day for this class, looking at the effects of search engines. After the Thanksgiving break next week, participants are presenting their briefs on various topics. Like other semesters, the students are blogging. I’ve been pleasantly surprised about two things. First, although there have been blogs that started out good and were consistently good throughout the semester, and blogs that started out struggling and still are, the vast middle has generally gotten better throughout the semester, and I might go so far as to say that the blogs this semester are particularly good as a whole at this point. For a taste, check out I see ‘em, five O won, Brophblog, Waxing, or Graduate Interactive Communication.
The other surprise is a lot of use of video in the blogs. Not just original video like this response to a Smart Mobs reading by JennX:
(kudos for battling through the Japanese!) but also other videos from YouTube and elsewhere that are both relevant and appropriate to their work. This is great, since it suggests a crossover to true multimedia, which is really interesting. Having been taken a bit unawares this semester, I’m going to try to encourage students to do this next time around. Very much looking forward to what the seminar participants accomplish in their final projects and I will post the best of those here for my loyal readers in search of interesting stuffs.
3. Virtual Worlds Seminar
This was intended as a projects-based course, and it’s been a rocky first-time through. I may re-offer it as a special topics course again at some point, drawing on some of the things we’ve learned this time about how best to approach things. Earlier in the semester, groups put together their first machinema from Second Life, like this one:
They are now working on several projects, including a furniture shop, an in-word presence for a school, and a mixed-mode organic T-shirt company. I’ll post links to the projects once we get to the end. After some prompting by one of the participants, I’m doing my own little project: a proof-of-concept for a judo dojo. I’ll try to do some videos of the process, though most of that work is going to have to happen when I get back from Chicago.
4. Internet Research Papers
I presented two papers in Vancouver, neither of which I was super-happy with. I got great feedback on both, though, and so I’ll probably take another run at them. One looked at comparisons of coverage of “localized” search engines, the other with the initial in-world wayfinding experience in SL. I’ll probably scrap the work on the latter for a more traditional usability study of SL for new users. But for now, these are both back-burnered.
5. Book about Search Engines
Working on a book entitled Search Engine Society for Polity Press. Still more to do with this, but once there is something more concrete to say, I’ll let you know. This is a broad overview of the effects of search engines on the information society, and how that relationship is evolving.
6. Association of Internet Research (AoIR) Stuff
I’m trying to make myself redundant here, but there are lots of little things that need to be done, from assisting in the upload of papers to upgrading the wiki, to supporting the creation of a new website. Trying to make incremental progress on those when I have little snatches of time.
7. Reviewing a paper for New Media & Society
Yay.
Early Stage Stuff
1. Hiring a new faculty member
This shouldn’t be so new, since we should be further along on this. I’m chairing a hiring committee for a tenure-track position in interactive communication Quinnipiac University. It’s not too late to contact me if you have mad skillz in the interactive industry and want to teach a group of interesting and bright (see above) grad students. If you want to chat about the position in Chicago, email me ASAP. We officially started reviewing candidates on the first of the month, but we still haven’t made a first cut.
2. Planning two online version of my seminars
We are moving toward an online version of our MS in Interactive Communication program, and to help facilitate this, I’ll be teaching two of my courses online next semester. The introductory seminar and a seminar on the effect of “wikinomics” on the communicating professionals both need to be sewn up shortly.
3. Two research papers
For now, these are under the working titles “Finding Tonto” and “Pavlov’s Blog.” I’ve wanted to do both for a while, but I’m going to try to get the preliminary research done before the end of the year. These may be sent off for the ICWSM conference in Seattle or HICSS. I was talking to a reporter this week about openly blogging my work, and this is that rare thing I won’t blog: ideas that I’m committed to taking on shortly, but I haven’t started the research on. As soon as I get started, I don’t mind writing about it, assuming that I actually have something interesting to say, and that it might be harder to catch up.
4. Walking
I was really doing well on getting to the gym and eating better until I ended up stressed out and overworked starting this summer. Need to recover (somehow) the time to bring this into my life. First step is back to 10,000 steps a day of walking.
5. Next book
As with the papers, I really ought to finish some things before taking on new projects, but I’m at the eary stages of planning for a book on the relationship between creativity and self-government.
Yes, I know it’s been a while. I started this semester out on the wrong foot, and have been scrambling for the last several months. If I haven’t answered your email (and there many hundred that need to be answered), I am very sorry. I promise I will get back to you. I considered email bankruptcy, but won’t go that far. If it’s time critical, email me again, since I’m working through in reverse chronological order, and that will push you to the top of the queue.
I have declared a bit of blogging bankruptcy, and abandoned some ideas for blog posts. Here is a partial list of the things I haven’t blogged, but thought about blogging:
* I was in Vancouver for IR8.0. The Mac Book Pro has developed another problem, and it is not charming–so no conference blogging. Lots of people did, though, including my anagramic colleague. Lots of stuff on Second Life, which was fun.
* Inside Higher Ed has an article that strikes a little close to home, denigrating profs with avatars, blogs, and wearing Hawaiian shirts. I far preferred the Chronicle’s advice on how James Bond can make you a better professor.
* Mutating Pictures allows human viewers to determine fitness and evolves faces from symmetric distributions of triangles. It’s already doing well. I am amazingly curious about the final outcome.
* Speaking of genetics, I am ready for the cow man.
Unlike other semesters, I haven’t been dumping my other writing and activities to this blog, for a variety of reasons, but now that I have a day of breathing space, I’ll try to start doing that again.
Next week is conference time. No, not CMJ–that’s right here in the Big Apple. And not 4S either–are all the conferences in Montreal these days? No, I’m making my regular trip out to the annual conference of the Association of Internet Researchers, appropriately entitled Internet Research out in Vancouver. I had initially intended to make a detour down to my old stomping grounds in Seattle, to visit old friends (like Nasai Teriyaki ^_^ ), but I’ve cut that short since it’s hard enough to be away for a week during the semester.
The natural question is “What’s that conference about?” Luckily, this year I can answer that, since registrants were (mostly) kind enough to tag their interests. And so I present the Internet Research 8.0 official tag cloud! For those of you unfamiliar with these things, the size of the font corresponds to the number of people who tagged their interest that way. Also, in this case, if two terms are close to one another they are more likely to have co-occurred. (This is far more true of the very top of the list than the bottom, where it’s generally isolates.) You can click the image below, or go to an HTML version. Oh, and IR8.0 isn’t a tag–it’s a title.
It’s not too late to become a sponsor of the Internet Research conference in Vancouver this year. On my about page, I note that I will sometimes promote organizations I’m involved in, and this is one of those cases. The IR conference is a great opportunity to make clear your commitment to social research on the internet, and help encourage new ideas and research. At present, a number of organizations are sponsors, including the Annenberg Center, Simon Fraser University, Microsoft Research, my old friends at the University of Washington’s Department of Communication, and the creator of Second Life, Linden Lab.
If you would like to add the name of your own organization to the august examples above, please drop me a note at 06 at halavais.net. There are opportunities to sponsor a range of activities–lunches, drinks, workshops, etc. and I can give you some of the basics or put you in touch with those who can help further.