Part Four: Philosophy

Why Assessment is Hard: [Part one] [Part two][Part three]

On the ASSESS listserv, Steven Zerwas wrote yesterday to seek opinions about logical fallacies in assessment. This turned into a good discussion, and a line of thought for me was sparked by Joan Hawthorne's contribution, when she wrote:

[T]here's the intriguing bit of considering the arguments of people resisting assessment from their perspectives -- what are they really concerned about? Some just don't want to do what they see as "more" but I have known many who have serious and valid pedagogical and/or philosophical concerns.

A few posts back I asked where are the error bars? to encourage modesty in our (the assessment community's) claims about what we can and can't legitimately say about outcomes assessment and other statistical magic. The idea I'd like to unpack in this post is that "philosophical concerns" may be indeed serious and valid.

Yesterday I bought a third copy of Bertrand Russell's The History of Western Philosophy. I had given the first two away as gifts, and our own copy is buried in boxes somewhere. In any event, my wife made me give it back after I kept it with me in the car for a year as emergency reading material (when standing in line or eating alone). I had replaced it with Machiavelli's The Prince, but that's far to thin a book for real emergencies.

In his introductory, Russell gives his definition of philosophy, which is of interest to us here.

Philosophy, as I shall understand the word, is something intermediate between theology and science. Like theology, it consists of speculations on matters as to which definite knowledge has, so far, been unascertainable; but like science, it appeals to human reason rather than to authority, whether that of tradition or that of revelation. All definite knowledge--so I should contend--belongs to science; all dogma as to what surpasses definite knowledge belongs to theology. But between theology and science there is a No Man's Land, exposed to attack from both sides; this No Man's Land is philosophy.

The attacks Russell mentioned are launched from both sides at the unfortunate Assessment Coordinator. Claudia Sanny on the ASSESS thread notes that she has encountered the argument that "To do assessment properly, we must have a reliable and valid instrument." This is clearly an attack from the scientific perspective (although the actual motives may just be to delay the exercise, of course). On the other hand, we have committees and directors and other bureaucratic fauna at the highest levels of government applying pressure to produce comparable ratings for institutions on educational attainment of students. From the Spellings Commission Report:

The strategy for the collection and use of data should be designed to recognize the complexity of higher education, have the capacity to accommodate diverse consumer preferences through standard and customizable searches, and make it easy to obtain comparative information including cost, price, admissions data, college completion rates and, eventually, learning outcomes.

The leap of faith here is that comparable information about learning outcomes is achievable in any manner that resembles scientific endeavor. The exemplars given in the report (CLA, and NSSE, for example) are clearly not sufficient for the task at hand. This bold assumption about comparable learning outcomes is a naked belief. We may as well call it theology, to fit Russell's dialectic. In more practical terms, systems of belief are imposed by accrediting bodies often with unintentional irony (as when confusing standards-based and continuous improvement criteria [more]). This institutional theology to covers everything from the Dept of Ed to accreditation associations to our own exhortations to assess.

There is always a problem with passing such memes on to infidels. In religions, some combination of carrot and stick is usually effective. In the case of persuading a faculty member to give up a coffee with colleagues in favor of doing curricular mapping, we often have more sticks than carrots. It's hard to create true believers that way. We may get converts, as in the Spanish Inquisition, but they may not be very heartfelt. Worse, the belief structure may create more problems than it solves--problems like Blue Hat Syndrome.

Stephen Jay Gould liked the idea that theology and science were somehow mutually exclusive as magesteria (so far, so good), and that this obviated the questions that created conflict between the two. I just can't swallow the conclusion. There seems to be good reason why naked belief and predictive validity are natural enemies.

The source of the authority in a theology is We-Say-So (WSS). The source of authority in science is predictive validity. Which of these authorities do we invoke most often when persuading colleagues? This is an uncomfortable question because there is no easy way out here. On the one hand, a WSS convert is not likely to be a long term ally, but it's an effective way to get things done in the short term. On the other hand, hard science is essentially impossible to do in the context of teaching and learning as it's practiced in higher education. Between this Scylla and Charybdis lies Russell's definition of philosophy.

We can appeal to reason without the gold standard of predictive validity and without WSS. This means using politics judicously, telling stories effectively, and not straying far from obvious relevance to faculty. It means creating a language that is as jargon-free as possible, and using WSS only to get the conversation started, if at all. Reason and enlightened discourse are by themselves no panecea nor recipe for dealing with all compliance woes, but rather I suggest that there is a way between WSS and pure science that is likely more productive than either of those. This kind of thing was suggested by David Shinn on the listserv [here], who says it better than I have:

The answer? Listen. Find an ally or two. Cajole. Look for an opening to get your foot in the door. Patience. Small steps. And most important, when your institution finally does conduct assessment - use it! Then more patience and listening.

This seems to me a philosophical approach. I would add a comment that I heard from my conversations yesterday, from an assessment director: you need to be in the classroom practicing what you preach.

As a coda to all this, I'll relate a conversation I had with my dad last night. He retired after teaching many years of college math, high school, and finally middle school. He is humble man, and I take him at his word when he says that the last year of teaching, all of his students passed the state exam in mathematics, and two thirds of them exceeded the standards. He talked about reaching the conclusion that textbooks didn't have enough review built into them, and started a summer program of review for his students: two worksheets per week for ten weeks. The reward for completion was a pizza. Then he discovered the Saxon textbooks, which he convinced the administration to adopt.

But he was baffled by the WSS requirements from the state. Over time, they would create and then drop initiatives. Some were completed, most not, all involving lots of work constructing detailed rubrics and such. To his mind, none of it ever came to anything, and was a waste of time. To me, this is a cautionary tale. Good results don't come from theology. In my dad's case, they didn't come from science either--not really. It's easier in math to measure simple learning outcomes than in some other areas, but it's always hard to do controlled experiments (a point elaborated on by ASSESS listerv contributer Steve Ehrmann in his TLT article here). Reason and the human ability to deal with complex problems, when combined with good intentions, is sufficient to show results. Even when we can't demonstrate predictive validity the way the physicists can.

Next: Part Five