(This is the second in a series of posts on the making of a research article. If you haven’t already, go back and check it out from the start.)
Overlapping sheets, not a point
A common response I get back from undergrads undertaking their first literature review is that they cannot find anything. This is usually an indicator of a systematic problem, since a usual issue is having too much to cover in a lit review, not too little. Generally, this comes in the form of “I can’t find anything on advances in the pizza delivery business in Peru during the 1990s.” If you could find something exactly on point for your work, you might have a problem: after all, what are you adding to the conversation?
While you should definitely look for “near misses” for your own narrow research topic, generally you are going to have stretch out a little broader: any papers or books on pizza delivery are probably worth checking out, whether or not they are in Peru, related to business, or set in the 1990s. Likewise, you are probably interested in the food delivery business in Peru more broadly as well. In other words, by assembling a quilt made up of existing slips of material, you can shape your own literature. Your work should be the point that binds together otherwise relatively disconnected pieces of work. Some of these may not fit together so neatly, others are in much better shape.
Keeping track
Generally, I keep a few documents going,at least when I am organized enough to do so. The main document keeps track of things I’ve found. This includes a full citation, and either relevant quotes (clearly indicated with quotation marks so that I don’t make mistakes later) or summaries of the material. (Many people keep these on notecards, or the digital equivalent, but I have always just kept it all in a document.) A second document records the citations and other information of things I think I should take a look at. A third document includes a list of key search terms and authors that might help in searching for materials.
I generally start with the last list, looking for a set of keywords, works, authors, or phrases that I can use to locate more information. I get these from my own assumptions, or sometimes from places like Wikipedia and other general sources of information. I then use these keywords in a series of places. I generally begin on Google Scholar, these days. I might also make use of ComAbstracts and similar resources. From these, I end up with a set of citations of things I should check out. For articles, I can generally check them from home. These days, I am also likely to check Google Books to see whether I really need a particular book or not before making the trek to the library.
As I go through these, I add to the search bibliography and search terms documents. Often, an article will contain little of interest other than its reference to other literature. It is a very iterative process. Eventually, I feel like I am seeing the same citations and names consistently, and have a handle on that particular segment of the literature.
Finally, for larger projects I try to assemble the bibliographic resources in a citation manager. I used Endnote for many years, with varying degrees of success. I have now switched over to Zotero, and finding it useful. Zotero is a free plug-in for Firefox that serves as a bibliographic manager, and provides a nice way to organize your resources and notes on the things you are citing. No matter your approach, make sure you have complete citations organized in some electronic format. Cleaning up citations and making certain the bibliography is correct seems consistently to take me three or four times as long as I expect it to.
As a practical matter, the above (like the cake) is a lie. I have generally internalized those three documents, and write the lit review as I am collecting information and sources. (Now, if only I could–like Jeremy Hilary Boob–write a positive review of my article as I was composing it!) If, however, you don’t have a lot of experience writing this sort of stuff, a bit of structure can’t hurt, and the “three document” approach has worked for students in the past.
Useful Areas
So what sort of things am I looking for? Obviously, I should at least look at anything that touches on Digg and the culture surrounding it and other collaborative filtering sites. There is surprisingly little literature, I suspect, that deals with the current crop of collaborative filters that involve interaction (Digg, Reddit, StumbleUpon, flock, and several dozen others). Much of what is out there will be technical in nature, and less likely to go into much depth in terms of users or social issues.
I also want to lay the foundation of operant conditioning, particularly as it applies to user interfaces and behaviors within a discourse community. It’s been a long time since I encountered this stuff in intro psych, and I’m not entirely sure where I’m going to find stuff here, or if I will. I suspect that since creating motivation to participate is particularly important in the area of education, there may be some work that looks particularly at reward systems and how they affect behavior in that literature.
Clearly, some of this has to do with the question of how people join groups and become acculturated to them. Although it is rare that such processes have such an obvious marker of success, it may be that there are models that can be applied to the processes observed on Digg.
There may be some work on reputation systems that is applicable. In particular, systems like those found on eBay (where you can be ranked up or down by those with whom you have interactions) might be applicable also to this work.
Finally, I’m hoping I can mine some of my own material. In fact, I’ll probably start there. I presented a paper about Digg and elections at the National Communications Association meeting last year, there is a paper I worked on with Alex Tan that analyzed the structure of eBay ratings, and, as I mentioned, I may be able to draw on a chapter from my dissertation.
Reputation Systems
Since I’m treating this a bit like a cooking show, I won’t go through the entire process, but I’ll walk through one of those pieces that will be woven into a literature review: the literature on reputation systems. As I noted, and to the consternation of traditionalists and my own surprise, Wikipedia has become a useful part of my research process, so I head over there for a survey.
Although the entry itself doesn’t go much beyond a definition, it does link to a few articles, including one I vaguely remember reading in CACM, and more importantly to a treasure cache: this site, which includes a bibliography of relevant research papers. Yay! Always nice when someone does a lot of the work for you. I go about assembling these, looking for information in them that informs my own work, and looking through their bibliographies to find common ancestors and theoretical foundations that are shared.
I also assemble a set of key phrases that seem to be coming up and run them through Google Scholar. Google Scholar is nice because it provides the ability to easily search for citing documents, and move forward to the most recent literature. In all, reading and writing about reputation systems takes only an hour or two, and yields about a paragraph of my literature review.
As with writing generally, if you are stuck, write something. Extract some reference, any reference, and start tunneling into it. Generally being stuck means you are not moving, and the easiest way to start moving is simply to start moving–in any direction at all.
And Everything Else
I generally find that I need to revisit the literature review after the research itself is done, but after working though the areas above and finding out how they connect, I’ve managed to set the foundation of my research, show that it hasn’t already been done (a common issue), and show that the questions I am asking are interesting and important, and will help to build on the existing literature. I ended up organizing the brief review under three headings:
* Filtering People and Content, where I talk about reputation systems and collaborative filtering
* Digg, where I talk more about how the site works
* Responses to Evaluation, where I look a bit at forms of learning and the process of becoming a member of a community.
From here, I have to figure out how it is I am going to do what I am going to do. Generally, this means planning out a method in some detail. The reasons for this are varied, but particularly in research that is collaborative, you will find that you need to propose the research before you can move forward. This may be a proposal to your committee, if you are a graduate student, or to a funding agency. If you are making use of human subjects, you will have to describe to a human subjects board what you plan to do in order to gain their approval.
Here, I know what I need (the public data from a sample of Diggers, organized in such a way that I can make sense of it), and so it is a matter of muddling through. In the next installment, I deal with getting an initial sample.
[the making of, pt. 2] Assembling the lit review
(This is the second in a series of posts on the making of a research article. If you haven’t already, go back and check it out from the start.)
Overlapping sheets, not a point
A common response I get back from undergrads undertaking their first literature review is that they cannot find anything. This is usually an indicator of a systematic problem, since a usual issue is having too much to cover in a lit review, not too little. Generally, this comes in the form of “I can’t find anything on advances in the pizza delivery business in Peru during the 1990s.” If you could find something exactly on point for your work, you might have a problem: after all, what are you adding to the conversation?
While you should definitely look for “near misses” for your own narrow research topic, generally you are going to have stretch out a little broader: any papers or books on pizza delivery are probably worth checking out, whether or not they are in Peru, related to business, or set in the 1990s. Likewise, you are probably interested in the food delivery business in Peru more broadly as well. In other words, by assembling a quilt made up of existing slips of material, you can shape your own literature. Your work should be the point that binds together otherwise relatively disconnected pieces of work. Some of these may not fit together so neatly, others are in much better shape.
Keeping track
Generally, I keep a few documents going,at least when I am organized enough to do so. The main document keeps track of things I’ve found. This includes a full citation, and either relevant quotes (clearly indicated with quotation marks so that I don’t make mistakes later) or summaries of the material. (Many people keep these on notecards, or the digital equivalent, but I have always just kept it all in a document.) A second document records the citations and other information of things I think I should take a look at. A third document includes a list of key search terms and authors that might help in searching for materials.
I generally start with the last list, looking for a set of keywords, works, authors, or phrases that I can use to locate more information. I get these from my own assumptions, or sometimes from places like Wikipedia and other general sources of information. I then use these keywords in a series of places. I generally begin on Google Scholar, these days. I might also make use of ComAbstracts and similar resources. From these, I end up with a set of citations of things I should check out. For articles, I can generally check them from home. These days, I am also likely to check Google Books to see whether I really need a particular book or not before making the trek to the library.
As I go through these, I add to the search bibliography and search terms documents. Often, an article will contain little of interest other than its reference to other literature. It is a very iterative process. Eventually, I feel like I am seeing the same citations and names consistently, and have a handle on that particular segment of the literature.
Finally, for larger projects I try to assemble the bibliographic resources in a citation manager. I used Endnote for many years, with varying degrees of success. I have now switched over to Zotero, and finding it useful. Zotero is a free plug-in for Firefox that serves as a bibliographic manager, and provides a nice way to organize your resources and notes on the things you are citing. No matter your approach, make sure you have complete citations organized in some electronic format. Cleaning up citations and making certain the bibliography is correct seems consistently to take me three or four times as long as I expect it to.
As a practical matter, the above (like the cake) is a lie. I have generally internalized those three documents, and write the lit review as I am collecting information and sources. (Now, if only I could–like Jeremy Hilary Boob–write a positive review of my article as I was composing it!) If, however, you don’t have a lot of experience writing this sort of stuff, a bit of structure can’t hurt, and the “three document” approach has worked for students in the past.
Useful Areas
So what sort of things am I looking for? Obviously, I should at least look at anything that touches on Digg and the culture surrounding it and other collaborative filtering sites. There is surprisingly little literature, I suspect, that deals with the current crop of collaborative filters that involve interaction (Digg, Reddit, StumbleUpon, flock, and several dozen others). Much of what is out there will be technical in nature, and less likely to go into much depth in terms of users or social issues.
I also want to lay the foundation of operant conditioning, particularly as it applies to user interfaces and behaviors within a discourse community. It’s been a long time since I encountered this stuff in intro psych, and I’m not entirely sure where I’m going to find stuff here, or if I will. I suspect that since creating motivation to participate is particularly important in the area of education, there may be some work that looks particularly at reward systems and how they affect behavior in that literature.
Clearly, some of this has to do with the question of how people join groups and become acculturated to them. Although it is rare that such processes have such an obvious marker of success, it may be that there are models that can be applied to the processes observed on Digg.
There may be some work on reputation systems that is applicable. In particular, systems like those found on eBay (where you can be ranked up or down by those with whom you have interactions) might be applicable also to this work.
Finally, I’m hoping I can mine some of my own material. In fact, I’ll probably start there. I presented a paper about Digg and elections at the National Communications Association meeting last year, there is a paper I worked on with Alex Tan that analyzed the structure of eBay ratings, and, as I mentioned, I may be able to draw on a chapter from my dissertation.
Reputation Systems
Since I’m treating this a bit like a cooking show, I won’t go through the entire process, but I’ll walk through one of those pieces that will be woven into a literature review: the literature on reputation systems. As I noted, and to the consternation of traditionalists and my own surprise, Wikipedia has become a useful part of my research process, so I head over there for a survey.
Although the entry itself doesn’t go much beyond a definition, it does link to a few articles, including one I vaguely remember reading in CACM, and more importantly to a treasure cache: this site, which includes a bibliography of relevant research papers. Yay! Always nice when someone does a lot of the work for you. I go about assembling these, looking for information in them that informs my own work, and looking through their bibliographies to find common ancestors and theoretical foundations that are shared.
I also assemble a set of key phrases that seem to be coming up and run them through Google Scholar. Google Scholar is nice because it provides the ability to easily search for citing documents, and move forward to the most recent literature. In all, reading and writing about reputation systems takes only an hour or two, and yields about a paragraph of my literature review.
As with writing generally, if you are stuck, write something. Extract some reference, any reference, and start tunneling into it. Generally being stuck means you are not moving, and the easiest way to start moving is simply to start moving–in any direction at all.
And Everything Else
I generally find that I need to revisit the literature review after the research itself is done, but after working though the areas above and finding out how they connect, I’ve managed to set the foundation of my research, show that it hasn’t already been done (a common issue), and show that the questions I am asking are interesting and important, and will help to build on the existing literature. I ended up organizing the brief review under three headings:
* Filtering People and Content, where I talk about reputation systems and collaborative filtering
* Digg, where I talk more about how the site works
* Responses to Evaluation, where I look a bit at forms of learning and the process of becoming a member of a community.
From here, I have to figure out how it is I am going to do what I am going to do. Generally, this means planning out a method in some detail. The reasons for this are varied, but particularly in research that is collaborative, you will find that you need to propose the research before you can move forward. This may be a proposal to your committee, if you are a graduate student, or to a funding agency. If you are making use of human subjects, you will have to describe to a human subjects board what you plan to do in order to gain their approval.
Here, I know what I need (the public data from a sample of Diggers, organized in such a way that I can make sense of it), and so it is a matter of muddling through. In the next installment, I deal with getting an initial sample.
[Continue on to part 3…]
Share this: