Crowdsourcing Assessment

I had a brainstorm today as I was driving home today from my former home, which is now known as "the endless list of fixing things." I'd installed a new toilet which refused to take its water gracefully despite my best efforts with crescent wrench and exotic vocabulary. To take my mind off of this vexation I was pondering the mystery of non-computability when I wondered into the mental ditch of educational assessment. Actually in this case, pre-educational assessment might be a better word, since I'm trying to figure out how to better predict student success during the application process. This idea has seen some print here before quite recently.

In my discussion of this idea with a colleague, he suggested using the personal essay that accompanies the application to try to identify traits of successful (future) students. At present the essay is more or less optional, but we could change that easily enough. But in order to be useful, we'd have to generate some kind of rating from the essay. This would then be fodder for a regression analysis on first year grades or retention. Who would do this rating work? A committee is one way, but if you've read my recent article on the beasts, you'd guess it wouldn't be my first choice. Then it occurred to me that we could outsource the readings. Not to ETS--that would cost a fortune. Rather to the Mechanical Turk. If you haven't heard of it, don't be too surprised. It's a way to crowdsource a problem that requires a human brain. Think of many smart ants building a fancy hill. In this case, its many human workers taking in small bits of work for small amounts of money.

Amazon.com runs this exchange. The currency is in HITs produced (human intelligence tasks) at a few cents per. As an example, one James F Lu is currently paying a dime for the following HIT:

Find Academic Calendar Information for Colleges

Please find the relevant academic calendar information. It can be generally done by viewing a school's academic calendar. All fields are required and verified in order to be approved.

Interestingly, someone named Ousama Haffar is offering 18 cents just to find the start and end date of university calendars. One can almost see the invisible hand at work here.

I imagine setting up a system where each (electronically gathered) essay is tagged with an ID number and assigned the following HIT. "Do you think the applicant who wrote this has personality traits necessary to succeed in college?" Note that it doesn't mention preparation--we already get that through high school GPA and (to some extent) SAT scores. I figure 5-8 cents is probably about right for this task, after browsing some of the other HITs on offer.

At the moment there are 57,899 HITs advertised. It doesn't say how many workers are available, but the home page says "Get thousands of HITs completed in minutes." I'd only need a few hundred or so a week, methinks. For 10,000 applicants at a dime each that's $1000. Imagine how much committee time it would take to do the same thing. It freezes the marrow to conjure that thought. The questions is, of course, is it worth it? That's the kind of thing that has to be answered empirically. In my experiences with subjective judgements of general education skills, the inter-rater reliability was surprisingly good. One could do that sort of thing here too--rate each essay three times or something, so as to do reliability checks. There's a certain amount of infrastructure to put in place here, but it's definitely worth a try. Stay tuned!