Assessment in the Wild

A big part of the justification for the Assessing the Elephant project was that when graduates have to actually demonstrate performance in "the wild" that these assessments will not be done with formal instruments. No standardized tests or (necessarily) rubrics, although the HR department may in fact foist some kind of rubric on supervisors. But I mean the informal everyday judgments about job effectiveness that ultimately determine whether or not one is promoted. The same applies to graduate school, doubly so for any complex field (like the humanities), where success is more subjective.

Ronald T. Azuma wrote in 2003 a computer science survival guide intended for graduate students. It's interesting to see the non-cognitive skills he highlights, including initiative:

One of the hallmarks of a senior graduate student is that he or she knows the types of tasks that require permission and those that don't.

Others include tenacity and flexibility and interpersonal skills. Here Dr. Azuma writes:

Computer Science majors are not, in general, known for their interpersonal skills. [...] [Y]our success in graduate school and beyond depends a great deal upon your ability to build and maintain interpersonal relationships with your adviser, your committee, your research and support staff and your fellow students. [...] I did make a serious effort to learn and practice interpersonal skills, and those were crucial to my graduate student career and my current industrial research position.

He then cites "Organizations: The Soft and Gushy Side" by Kerry J. Patterson, published in Fall 1991 issue of The Bent, which contains the nugget I want to highlight:

To determine performance rankings, we would place in front of a senior manager the names of the 10-50 people within his or her organization. Each name would be typed neatly in the middle of a three-by-five card. After asking the manager to rank the employees from top to bottom, the managers would then go through a card sort. Typically the executive would sort the names into three or four piles and then resort each pile again. Whatever the strategy, the exercise usually took only minutes. Just like that, the individual in charge of the professionals in question was able to rank, from top to bottom, as many as 50 people. It rarely took more than three minutes and a couple of head scratches and grunts. Three minutes. Although politics may appear ambiguous to those on the receiving end, those at the top were able to judge performance with crystal clarity.

This is what actual assessment looks like. It happens all the time, formally and informally. Our supervisors, colleagues, neighbors, and friends constantly assess us just like we do them--it's the nature of living in a tribe. Are these impressions valid? If there's enough feedback through inter-rater reliability, then this fact alone will create validity. For example, if a co-worker has few social skills, such that he becomes the butt of jokes at the office, this very fact probably makes it less likely he'll be promoted or enjoy the reciprocation of favors that makes teamwork effective. The implicit agreement of characteristics is powerful.

Besides the ones mentioned above, self-control is another non-cognitive skill that deserves attention. It is mentioned in this New Yorker article simply called "Don't!" The article highlights a study done by a psychology professor named Walter Mischel, who assessed children's ability to wait for a reward in favor of a bigger reward (a measure of self-control). The experiment began in 1981, and he's tracked the children since then to see what happened to them:

Once Mischel began analyzing the results, he noticed that low delayers, the children who [couldn't wait], seemed more likely to have behavioral problems, both in school and at home. They got lower S.A.T. scores. They struggled in stressful situations, often had trouble paying attention, and found it difficult to maintain friendships. The child who could wait fifteen minutes had an S.A.T. score that was, on average, two hundred and ten points higher than that of the kid who could wait only thirty seconds.

I had just finished reading this when my 11-year old daughter came in an begged to go to the bookstore--she'd gotten a gift certificate for her birthday.

"But they're almost closed," I said, "you'll only have 15 minutes by the time we get there."
"I want to go anyway," she said.
"How about this," I proposed, "We go tomorrow instead, and you can have two hours if you want."

She didn't even think about it. We went right away. As it turns out, the store closed later than I thought, and we had an hour to shop, but it was jolting to have this conversation right after reading the New Yorker piece. Why isn't this kind of thing--including assessment--part of the curriculum? Take a look at the AAC&U piece about trends in general education and see if you can spot non-cognitives anywhere. There are experiences that would probably lead to non-cognitive development (internships, for example), but they aren't addressed directly. I think it's time we seriously considered non-academic factors that are important to success and begin to talk about them.