Rongness

Wolfgang Pauli famously had little tolerance for error. The Wiki page on him puts it like this:

Regarding physics, Pauli was famously a perfectionist. This extended not just to his own work, but also to the work of his colleagues. As a result, he became known in the physics community as the "conscience of physics," the critic to whom his colleagues were accountable. He could be scathing in his dismissal of any theory he found lacking, often labelling it ganz falsch, utterly wrong.

However, this was not his most severe criticism, which he reserved for theories or theses so unclearly presented as to be untestable or unevaluatable and, thus, not properly belonging within the realm of science, even though posing as such. They were worse than wrong because they could not be proven wrong. Famously, he once said of such an unclear paper: Das ist nicht nur nicht richtig, es ist nicht einmal falsch! "Not only is it not right, it's not even wrong!"

This "not even wrong" quote shows up all over, and is even the title of a book about string theory:

The publisher plays with the idea by typesetting "wrong" backwards on the cover. I have a different take on this that I used in my (unpublished) novella Canman:

The System is never rong.
--System Documentation v1.0

It looks ironic because it appears to be a typo, but "rong" is deliberately spelled like that. Literally, "rong" is "not even wrong." It's pronounced with a long 'o' sound, so that you can distinguish the two words.

The original idea was that if you use the right approach but make a mistake you can get the wrong answer. But if you use a method that can't possibly work, you might accidentally get the correct answer once in a while, but it's still rong. Astronomy may give you the wrong distance to a remote galaxy, but astrology will lead you to rong conclusions. The idea that the Sun orbits the Earth is wrong, but the idea that Apollo pulls it around in a chariot is rong.

I think this is a useful term because it grounds any subsequent discussion. That is, it identifies the particulars of the argument as the issue (potential wrongness), or the whole methodology as the issue (potential rongness). 

Of course, this opens the door for more meta-levels of error. One could propose "roong" to mean "not even rong," and "rooong" to mean "not even roong," and philosophers could then debate the degree of rongness of a particular idea.

Assessment Philosophy Prezi

I put together a Prezi for an upcoming talk, rather than digging out some tired powerpoint slides. If you don't know Prezi, check it out. There's a very reasonable educational license for the full product, but you get a lot for free.

The idea is to describe the yin and yang character of assessment: the scientific ambition of measuring and demonstrating improvement in learning versus a more modest aim to create a culture that values learning and pays attention to what observation is telling us. In the presentation I term these "Demonstrated Improvement" versus "Intentional Evolution." There are techniques and language that belong to both, and in implementation, some that fall in between. This last category would include things like student portfolios, which can be used for rubricked (if that's a word), sliced and diced number crunching, or messier assessment through observation and discussion.

I intend to do a voice-over for this thing when I get the chance. That's new to me, and our web guy recommended a product from Techsmith called Camtasia that I'll try out. I use the free version of their Jing all the time.

Here's the Prezi:

Assessment on Prezi

Part Five: Creating Do-Gooders

Why Assessment is Hard: [Part one] [Part two] [Part three] [Part four]

Well, this is ironic. I was trying to come up with Plato's quote about men wanting to do good, and only needing to learn what that is. That was supposed to be the foil for some engaging line of thought. What I found was this. It's a service that writes essays for (I presume) college students for $12.95 per page. Their sample happens to be about Plato and Aristotle. Quoting from the essay:

Plato says that once someone understands the good then he or she will do it; he says “...what we desire is always something that is good” (pg.5). We can understand from this that Plato is saying individuals want to do good for themselves; we perform immoral deeds, because we don’t have the understanding of the good.

The existence of the quote where I found it pretty much negates its premise. And here I'd planned to make an argument that resistance to assessment by teachers is caused by them not knowing the Good. Well, I shall forge ahead anyway, and you can play along.

Even when something is clearly Good, it's not obvious that everyone will do it. Otherwise everyone with the means would eat plenty of fruits and veggies every day instead of fried pork rinds for breakfast, or whatever it is that leads to so much heart disease. So there's certainly the issue of how hard it is to do the right thing. But maybe we can take those two parts as necessary conditions to, for example, get teaching faculty to implement all of the beautiful assessment plans that have been cooked up.

First, the Good. If Professor Plum doesn't buy into the idea that this whole outcomes assessment thing is worthwhile, the project can only proceed through wesaysoism, which calls for continual monitoring and tedious administration. Administration is, of course, a hostile word to many faculty, so it's best if the message is delivered from one of their own. If the assessment director isn't in the classroom mixing up his or her own assessment recipes, the project is suspect. But this is only the first step. After all, faculty members have even more crazy ideas than administrators do--it's almost a prerequisite for the job (speaking as one, here).

No, you have to be convincing. There is a certain amount of chicken and proto-chicken here--you really need a program or two that does a good job so that you can prove that the idea can actually be carried out. If you start from zero, then the first priority is to find a spot of fertile ground and begin to cultivate such a program. Plan on this taking years. There are some natural alliances here, in unlikely places perhaps. Art programs already have assessment built in with their crits, as do creative writing programs. Finding a champion or two among the faculty that others respect is key. You can tell who's respected by who gets put on committees.

Unfortunately, the enterprise of assessment is a lot harder in practice that it looks on paper. So having a solid philosophy can help enormously. By this I mean picking your way carefully through the minefield of empirical demands and wesaysoism to find a path the others can follow. If the goals you set are too demanding of science, you'll fail because assessment isn't science. We don't actually measure anything, despite using that language. More on that later. As a result, if "closing the loop" means to you a scientific approach of finding out what's the path to improvement and then acting on it deterministically, it will be like trying to teach a pig to dance: frustrating to you and annoying to the pig.

On the other hand, as we've already noted, relying simply on wesaysoism to get things done means you have to micro-manage every little thing, and the faculty will try to subvert you at every turn. So doing the work of sorting out for yourself how to make a convincing argument, based on cogent principles, is worth it. Read what other people have to say. You might check out Assessing the Elephant for a divergent view. But find something that makes sense.

I find that it helps to separate out thinking about classroom level assessment and what we might call "big picture" assessment. The former should and can be indistinguishable from pedagogy--the integration of assessment directly with assignments and how stakeholders view them. As an example, we weren't happy with students' public speaking skills, so we started videotaping student seminar presentations, and having them self-critique them. It's not rocket science. But it wouldn't have happened if we hadn't explicitly identified effective speaking as a goal, and thought about what that means. And it seemed like a Good thing do to.

Big-picture assessment is extremely easy to do wrong, in my opinion. I think lots of low-cost subjective ratings are a good approach, but opinions will vary. In any event, don't imagine that it's easy to accumulate what happens in the classroom and make it applicable to the university. It's very difficult. Again, have a solid philosophy to back you up. Otherwise you'll be waving your arms around, trying to distract your audience from the logical holes.

In both micro- and macro-scale assessment, try to feed the faculty's own observations back to them in useful summary form (not averages--they're not much use). They don't respect anyone so much as themselves.

Being Good isn't enough. It also has to be easy-peasy. Lower the cost of assessment to the minimum realistic amount of time and energy. Don't meet if it can be done asynchronously. Use Etherpad instead. More generally, use technology in ways that simplify rather than complicate the process. It's all well and good to have a curriculum mapped out in microscopic detail, with boxes and rubrics for every conceivable contingency. But if no one uses it because it's too complicated, it's moot. The barrier to completion may not have to be very high to be effective. Page load times matter. A few dozen milliseconds is enough to turn away customers. Too many clicks, or poorly designed interfaces shed more users. It shouldn't be a death march to enter data into the system, however you do it.

I won't recommend a commercial system here, because I'm not an expert on them, and I also think you can create your own without that much trouble. You just need one decent programmer and a little patience. Again, philosophy is key to building the thing. Whether your system is paper or electronic or smoke signals, think about what the maximum useful information per effort is. It's easier to start small and grow than the other way around.

As a real example, a portfolio system I built for a client started off as the absolute bare minimum--it was really just a drop box, but with students and class sections conveniently set up automatically. Over time we added assessments to the submissions, customized per program. It would have been too much to start there. Remember the only way to solve opaque problems is an evolutionary approach.

Satisfying all the critics isn't easy. Accreditors want different things than the senior administration, which will be yet different from what the faculty find useful. For the former, sad to say, rather meaningless line graphs showing positive slopes of some kind of outcome is usually enough. This is the kind of thing that looks good, but probably doesn't mean much. So there is always the temptation to simply play the game of meeting those (very minimal) expectations. Don't do that, or you'll find yourself wondering why you didn't choose exotic plumbing as a career instead.

Convince the faculty, and everything else Good will follow. It's easy to make pretty graphs. It's much harder to lead an ongoing conversation that your very intelligent colleagues find convincing and insightful. And if you find yourself in trouble, you can always show them the website selling term papers. That should be good for an hour's distraction at least. Meanwhile you can slip out the back and work on your oratory or figure out a way to shave half a second off of page refresh times.

Next: Part Six

Part Four: Philosophy

Why Assessment is Hard: [Part one] [Part two][Part three]

On the ASSESS listserv, Steven Zerwas wrote yesterday to seek opinions about logical fallacies in assessment. This turned into a good discussion, and a line of thought for me was sparked by Joan Hawthorne's contribution, when she wrote:

[T]here's the intriguing bit of considering the arguments of people resisting assessment from their perspectives -- what are they really concerned about? Some just don't want to do what they see as "more" but I have known many who have serious and valid pedagogical and/or philosophical concerns.

A few posts back I asked where are the error bars? to encourage modesty in our (the assessment community's) claims about what we can and can't legitimately say about outcomes assessment and other statistical magic. The idea I'd like to unpack in this post is that "philosophical concerns" may be indeed serious and valid.

Yesterday I bought a third copy of Bertrand Russell's The History of Western Philosophy. I had given the first two away as gifts, and our own copy is buried in boxes somewhere. In any event, my wife made me give it back after I kept it with me in the car for a year as emergency reading material (when standing in line or eating alone). I had replaced it with Machiavelli's The Prince, but that's far to thin a book for real emergencies.

In his introductory, Russell gives his definition of philosophy, which is of interest to us here.

Philosophy, as I shall understand the word, is something intermediate between theology and science. Like theology, it consists of speculations on matters as to which definite knowledge has, so far, been unascertainable; but like science, it appeals to human reason rather than to authority, whether that of tradition or that of revelation. All definite knowledge--so I should contend--belongs to science; all dogma as to what surpasses definite knowledge belongs to theology. But between theology and science there is a No Man's Land, exposed to attack from both sides; this No Man's Land is philosophy.

The attacks Russell mentioned are launched from both sides at the unfortunate Assessment Coordinator. Claudia Sanny on the ASSESS thread notes that she has encountered the argument that "To do assessment properly, we must have a reliable and valid instrument." This is clearly an attack from the scientific perspective (although the actual motives may just be to delay the exercise, of course). On the other hand, we have committees and directors and other bureaucratic fauna at the highest levels of government applying pressure to produce comparable ratings for institutions on educational attainment of students. From the Spellings Commission Report:

The strategy for the collection and use of data should be designed to recognize the complexity of higher education, have the capacity to accommodate diverse consumer preferences through standard and customizable searches, and make it easy to obtain comparative information including cost, price, admissions data, college completion rates and, eventually, learning outcomes.

The leap of faith here is that comparable information about learning outcomes is achievable in any manner that resembles scientific endeavor. The exemplars given in the report (CLA, and NSSE, for example) are clearly not sufficient for the task at hand. This bold assumption about comparable learning outcomes is a naked belief. We may as well call it theology, to fit Russell's dialectic. In more practical terms, systems of belief are imposed by accrediting bodies often with unintentional irony (as when confusing standards-based and continuous improvement criteria [more]). This institutional theology to covers everything from the Dept of Ed to accreditation associations to our own exhortations to assess.

There is always a problem with passing such memes on to infidels. In religions, some combination of carrot and stick is usually effective. In the case of persuading a faculty member to give up a coffee with colleagues in favor of doing curricular mapping, we often have more sticks than carrots. It's hard to create true believers that way. We may get converts, as in the Spanish Inquisition, but they may not be very heartfelt. Worse, the belief structure may create more problems than it solves--problems like Blue Hat Syndrome.

Stephen Jay Gould liked the idea that theology and science were somehow mutually exclusive as magesteria (so far, so good), and that this obviated the questions that created conflict between the two. I just can't swallow the conclusion. There seems to be good reason why naked belief and predictive validity are natural enemies.

The source of the authority in a theology is We-Say-So (WSS). The source of authority in science is predictive validity. Which of these authorities do we invoke most often when persuading colleagues? This is an uncomfortable question because there is no easy way out here. On the one hand, a WSS convert is not likely to be a long term ally, but it's an effective way to get things done in the short term. On the other hand, hard science is essentially impossible to do in the context of teaching and learning as it's practiced in higher education. Between this Scylla and Charybdis lies Russell's definition of philosophy.

We can appeal to reason without the gold standard of predictive validity and without WSS. This means using politics judicously, telling stories effectively, and not straying far from obvious relevance to faculty. It means creating a language that is as jargon-free as possible, and using WSS only to get the conversation started, if at all. Reason and enlightened discourse are by themselves no panecea nor recipe for dealing with all compliance woes, but rather I suggest that there is a way between WSS and pure science that is likely more productive than either of those. This kind of thing was suggested by David Shinn on the listserv [here], who says it better than I have:

The answer? Listen. Find an ally or two. Cajole. Look for an opening to get your foot in the door. Patience. Small steps. And most important, when your institution finally does conduct assessment - use it! Then more patience and listening.

This seems to me a philosophical approach. I would add a comment that I heard from my conversations yesterday, from an assessment director: you need to be in the classroom practicing what you preach.

As a coda to all this, I'll relate a conversation I had with my dad last night. He retired after teaching many years of college math, high school, and finally middle school. He is humble man, and I take him at his word when he says that the last year of teaching, all of his students passed the state exam in mathematics, and two thirds of them exceeded the standards. He talked about reaching the conclusion that textbooks didn't have enough review built into them, and started a summer program of review for his students: two worksheets per week for ten weeks. The reward for completion was a pizza. Then he discovered the Saxon textbooks, which he convinced the administration to adopt.

But he was baffled by the WSS requirements from the state. Over time, they would create and then drop initiatives. Some were completed, most not, all involving lots of work constructing detailed rubrics and such. To his mind, none of it ever came to anything, and was a waste of time. To me, this is a cautionary tale. Good results don't come from theology. In my dad's case, they didn't come from science either--not really. It's easier in math to measure simple learning outcomes than in some other areas, but it's always hard to do controlled experiments (a point elaborated on by ASSESS listerv contributer Steve Ehrmann in his TLT article here). Reason and the human ability to deal with complex problems, when combined with good intentions, is sufficient to show results. Even when we can't demonstrate predictive validity the way the physicists can.

Next: Part Five

The Price of Philosophy

Education is a funny business. Despite all the efforts made in assessing learning outcomes, it may well be that the net effect of educational experiences is not realized until much later in a person's life. Some things that we think are important may actually have little or no effect. This is especially true when considering the more philosophical aspects of higher education. Yesterday I mused about the value of a liberal education. It's an easy target for the practical-minded cost trimmer because the results are presumably not truly felt until years after the experience. And yet the budget executives have to be dealt with, especially in these lean times. They may even be right some of the time.

I've sat in a lot of budget meetings with both administrators and faculty, and an inevitable collision is that between greenbacks and philosophy. At these moments I usually make a mental note: We may not know the worth of philosophical goals, but we can often evaluate the cost. In lean times this skews the decision against philosophy and toward saving money. In fat times, the opposite is true. Some examples will help understand what I mean.

How many books is enough in the library? Is a cataloger in Sanskrit necessary and contributing toward educational outcomes? What about restrictive policies for course transfers, under the theory that courses at other institutions aren't as good as your own? This costs the institution enrollment with nebulous effect on students who do come. Whole departments like foreign languages get their raison d'etre questioned as well.

Like Pythagoras's philosophy of not eating beans, the items in the list above have economic impact (the great philosopher and mathematician is supposed to have died rather than escape his enemies through a bean field). Evaluating the actual worth relative to the cost is not something that can usually be done scientifically--this is where leadership is required. But it helps in these discussions to have the sensitivity to know when the discussion has wandered into philosophical territory.

Occasionally, hard data can come to the rescue, but this is probably an exception. One IR director related to me that the faculty at his institution were unhappy about accepting AP courses instead of their own prerequisite classes. They were on the verge of setting a new policy to ban AP course credit, which would have had a negative impact on admissions. He did some research and found that AP students actually did better in the subsequent classes than home grown ones (it could be that they were better students, of course). This proved to be the case in all instances except one subject area. When they investigated further, they discovered that the AP course content did not match that of the prerequisite course very well. I like this story because it shows what IR can do when the evidence is there. The harder decisions are unfortunately more common: placing a value on Sanskrit cataloging, for example.

More poetically, one might say

अमंत्रमक्षरं नास्ति नास्ति मूलमनौषधम्‌।
अयोग्यः पुरुषो नास्ति योजकस्तत्र दुर्लभः॥

[source and translation]