Open Analytics and Social Fascination Talk

By alex | Published: 12/16/2011

Posted in Research | Tagged , , , , , , , , | 1 Comment

IRBs and Clean Secrets

By alex | Published: 12/8/2011

There’s a comment piece I wrote that appears in today’s issue of the journal Nature that talks a bit about the role of open data and IRBs. But I worry that perhaps in the number of iterations it made before publication the main point got muddied a bit. So here it is:

Funding agencies and journals should require authors to submit and openly publish protocols as submitted to the IRB.

There are a bunch of reasons for this. First, IRB protocols should be public. Right now, it’s treated on the one hand a bit like the Napoleonic code: it doesn’t matter what others have decided, the board decides entirely on the basis of your submitted application. This has some real negative implications.

First, the same protocol may be accepted at one campus and rejected at another. Or research in the same stream (or seeking to replicate) may be rejected at a later date.

Second, the IRBs have to make an original determination each time wasting, in many respects, the efforts of other competent IRBs who have already made a determination. They don’t need to be bound by earlier determinations, but don’t you think it would be worthwhile to be at least aware of them?

Third, when applicants feel as though they are handled unfairly, a transparent system is better for all.

Fourth, the best way to learn to do ethical research is to be able to observe the process from the periphery, and listen to the queries of IRBs and the responses by researchers. Releasing the approved protocols may not get at that deeper conversation completely, but it at least provides a small window.

Fifth, the protocols are an excellent way of “indexing” open data. Open research data often is published with codebooks and other ancillary material, but IRB protocols in many ways are the ideal introduction to an open collection of data, explaining why it was collected, how it was collected, and how it might be used.

For these reasons, among others, there should be open repositories of IRB protocols. Now, we could just try to convince individual IRBs on campuses to open up their process and publish protocols they approve, and I hope that they might. But IRBs are by nature a conservative group, intended to protect, not to disrupt. In many cases, they are made even more conservative by the institutions that they are housed in, and concerns by that institution either that they might be sued by subjects or that they might be investigated by federal regulators. (Those regulators, naturally, have access to the protocols and the decision process of the IRB once an investigation or audit begins, but they might not want to provide any sort of “probable cause.”)

Individuals might be encouraged to submit their own protocols to a repository, and in fact, self-archiving has made an important impact on the way publishing happens. But there are enough open questions surrounding this that it’s a hard place to start the ball rolling.

Funding agencies, and more recently journals, either insist on or facilitate the sharing of research data. In many ways IRB protocols are an important part of those research data. If funders required that IRB protocols be shared just like any other research data, and if journals provided authors the resources to share these protocols, it would revitalize in some ways the role of scholarly publishers and it would make for ethical oversight that was more robust and transparent.

But what if you were not IRB approved? It may be you didn’t need pre-approval of research by the IRB, and as I argue in the article above, I think this should be the case for much of the research that is currently placed under some level of review by IRBs. But if you don’t have to have IRB approval, I think funders should still require you to talk about the ethical considerations of your research, and journals should require you to publish this online when you do not have an IRB-approved protocol to provide.

What this does is creates an environment in which ethical post-review is encouraged. Certainly, when it comes to drug trials–and even to invasive forms of research of vulnerable populations in the social sciences–there should be some sort of oversight before the research occurs. But even after it occurs, peer-reviewers and the reading public should be able to see how the researchers weighed the needs and rights of subjects against the importance of their research questions.

Posted in Research | Tagged , , , , , , , , , | 3 Comments

Retreating on the Grades

By alex | Published: 8/21/2011


Heading into a new semester and assembling the syllabi (well, one–the other, once again this term, is in the hands of the students), I’ve decided to give up on my short-lived “no grades” policy. At least nominally.

What happened? Well, at least pedagogically, I was fine with it. To recap, I was concerned that students were more interested in grades than they were in the actual material. I speculated that replacing the grades with badges would at least move them from focusing on entirely arbitrary markers (letters) to markers that were more explicitly tied to learning objectives.

I still think grades suck, of course. Grades are grand for beef, and actually pretty handy for deciding what restaurants to avoid, but as a tool for learning I think they take away more than they add. Realistically, though, I can’t get away from grading on my own. Unless I can convince all my colleagues to move in that direction, my experiment threatens to be merely a distraction for students, or in the worst case, a good way for them to ignore my course. Purely in terms of learning outcomes, I think being able to get totally away from grades would be great. But that wasn’t the case here.

Not learning, but ranking

The main problem is that more than just the students see the grade. It acts–in fairly limited ways–as a reflection on their skill. Now, to my mind, we simply shouldn’t graduate students we can’t stand behind. I frankly would have no problem “advising out” those students who are not performing at an elite level. I think it would better serve both those who left and those who stayed. But that’s not a realistic option (at least not in that extreme a degree).

Without getting into details, since I can’t simply unilaterally make a course pass/not pass–which is at least closer to ungraded–I ended up saying you get either an A or an F. Really, I intended to give As to everyone, short of really utter non-completion. I think I can say, without naming names, that people got As who really weren’t doing graduate-level work. I was clear in my narrative summaries of their work that this was the case, but they still ended up with As. At least one of my fellow faculty members found this contrast to be wrong, and I can understand why.

Pass and Forget

Many of the courses in our online programs are taken serially, and so I don’t have to compete for attention with other courses. That wasn’t the case this summer, and I suspect that students paid more attention to the courses where they were still “fighting for a grade.”

A lot of the work on pass/not pass grading going back several decades looks at courses in similar contexts. Being the P/NP course in a world of graded courses means that for some students (generally those who are not already high-achievers) the time and effort will be put to the graded courses, as they are afforded a certain degree of prestige as well as attention from students simply because of their grades.

Don’t Picture an Elephant

My intention was to make students think less about grading, but because they needed to keep track of the number of points each badge was worth, and whether they had crossed a certain threshold, they ended up thinking more about it. For some, the idea that the grade would be either an A or an F, and nothing in-between, raised their anxiety level, even after I made clear that an “A” was granted for even minimal completion of work.

In the end, by doing something out of the ordinary, I ended up focusing students more on the grades and grading structures, not less.

The (Non) Solution

So, what’s a person who hates grades to do? I’ve always been considered a “hard grader.” Perhaps that’s why people have been so focused on grades in my classes. Clearly going the other way, and becoming the Oprah of As (You get an A! You get an A!) hasn’t worked out. Two possible alternatives:

Maybe the simple solution is to provide a grading rubric, but simply make it easy to get an A. That doesn’t seem like a good solution. It seems like it contributes to the “menace” of grade inflation. But if I don’t really care about grades, I’m not sure why I should care about their inflation. More importantly, although I don’t get the issue of people dwelling over an odd grading structure, I still have to contend with pulling attention away from courses with a stiffer climb up the grading ladder.

The other alternative is to go old-school. The course is graded on a curve. The top 20% of points-earners get an A, the next 40% get a B, the next 20% get a C, and the next 20% get a D or F. (Come on, you don’t really expect me to curve around a C, do you? I’m not that old.) Of course, this means that students are going head-to-head Paper Chase style, rather than cooperating and collaborating nicely. That sucks, but maybe it is what is needed to get folks to step up their game.

So, for next semester, I’m back to grading meany. A curve (and not a saving one) it is, at least in the course where I’m dictating the policy. That will be more familiar ground for me, and probably also for the students.

Posted in Teaching | Tagged , , , , , , , , , , , | 9 Comments

Google plus what?

By alex | Published: 7/2/2011

Over the last few days, I’ve been exploring Google Plus a bit, in my spare time, along with a quarter million of my closest friends around the globe. There are already a lot of reviews… that’s not what this is. I find it to be an interesting entry point, and I’m curious to see what happens with it. As I said on Twitter, I’m cautiously optimistic about its future.

That said, I’m curious what that future will be. A lot of people worry, rightly so, about the privacy implications of Google properties all combining their connections to users and assimilating us all into a massive, Google-controlled borg. But if anything, the opposite is also a concern.

I work a lot with Google documents with teams, and I find it to be a really useful tool. What works for ad hoc collaboration, however, doesn’t work as well for continuing collaboration or managing groups. If I have a group or organization, keeping track of project teams and the like who want to use Google Docs is a nightmare. Not only does it tend to be fussy with non-Google emails, it’s a pain to create groups in Google and then invite them to a document.

The biggest missing feature in the Googleverse is effective management of groups. Google Groups certainly isn’t it. And I was hopeful that Circles might be.

And it could be if you could share circles. Right now, I can create circles to indicate who is who and who sees what in my update streams. Cool. But no one can see what my circles are. As I noted in an early post on G+, that’s a good thing. I wanted to know this before I created, say, a circle called “people who annoy me.” All of us have people like this in our lives, and for one reason or another are forced to interact with them, and we probably could group them this way in our head. Nonetheless, we wouldn’t want this–or many other groupings–to be public.

I am thinking back to other social networks that attempted to create classes of connections that require a whole range of negotiations. It’s hard enough for me to divide people between “friends” and “associates”–if on top of this I had to then justify that decision to people who are in those two groups, I assume I would have very few of either.

So, as a default, I’m fine with the title and grouping of circles remaining private by default. (Although it’s not as clear that this is true of the latter case, since you can find out the list of people who have access to a particular post, and perhaps surmise some group boundaries that way.) But I want the ability to make my circles public, and perhaps to have people in the circle be able to add new people to the circle.

What sort of a circle would that be? I now have a circle called “QUICM students” that includes past and present students in the grad program where I teach. I took a first stab at dropping people into that group, but I know that there are others I’ve missed. I need something that is much more akin to traditional group management. Others in the group need to be able to add members from the outside, and perhaps with some groups, they need to be able to add themselves.

I don’t see that this is a big push away from Circles, and I really hope that G+ develops in that direction. Without some way of creating and managing groups, it will lose out on a function that is missing not just from their social networking effort, but from a lot of Google applications. One group I am working with has decided to use ManyMoon as a way of managing this function, and others turn to other approaches. But a robust group management feature would be a great addition to G+.

If that group/circle identity management solution could also be cleanly integrated with Docs, with Reader, even with Groups (which might become superfluous in this case), I think we would have a platform for distributed collaboration that would be helpful to businesses, to educators, and to anyone else who wants to get things done in a group.

Posted in Technology | Tagged , , , , , , , , , , , , , | 4 Comments