Closing the Loop CLA Style

I'm not a fan of standardized testing for complex skills. There are many reasons that it seems like a poor tool to use, and among them are:

• Complexity means many kinds of instantiations. A standardized test can only examine a very limited number of these.
• A standardized test is mostly a monologue, whereas a dialogue is exponentially more powerful in elucidating useful responses in a complex situation. (imagine yourself as a 911 operator who is only allowed to give a standardized "problem report" form of yes/no questions to diagnose a caller's problem, instead of asking "what's your emergency?")
• There is an implicit definition of desired outcomes that is created when a test goes into print. These may or may not align with those of the test consumer.
  

This last point brings me to the reason for this post about the CLA. I recently received their newsletter, which mentioned something called CLA in the Classroom. You can read about it on its own official website here. The site describes the program thusly:

CLA in the Classroom is a curricular and pedagogical program that focuses on the higher order skills of critical thinking, analytic reasoning, problem solving and written communication.

I am generally annoyed by lists of skills like this. Critical thinking, in particular, is such a fuzzy concept that I find little use for it as a general learning outcome. It would be fine in particular contexts, like art, where 'crits' are pretty well-defined. The distinctions between critical thinking, analytical reasoning, and problem solving are not at all clear to me either. I assume, however, that the CLA folks have thought a lot about this and have their own definitions and ways of assessing them.

Assessing these skills should be done in context, of course. Therein lies a rub, as the Bard might say. What context do we assume students to be familiar with? A math major might demonstrate analytical thinking by finding the flaw in a proposed proof of a proposition, but that won't work very well for an English major. There has to be some least common denominator--a body of knowledge that all students will be familiar enough with to demonstrate their thinking skills. But is it reasonable to assume that this exists? Count me dubious. We might leap upon the general education curriculum as a natural basis for testing--a common knowledge base from which to launch forays into the critical thinking realms. There are at least two problems with this. First, most general education programs are fragmented. Rarely will one student have the same curricular experience as another. Second, the CLA may be administered before the student has taken the required background course, unless this is controlled for. To my knowledge it is not. So a low score might have as much to do with lack of background as it does with lack of skill.

A lack of clear context contributes to a lack of clarity about the goals of the CLA or any similar instrument, including most standardized tests of general education. Nature abhors a vacuum, as the saying goes, so this creates an opportunity for test-makers. If it's not clear what a test is trying to measure, the only real authority on the matter is the test-maker. Therefore, if you want your students to do well, who better to ask for help than the creator of the instrument? If one is cynical about it, the business model of drug companies is similar to this: advertise some new ailment that "requires" treatment. Who can tell you if you have the condition, and who can help you treat it (with maintenance drugs)? The medical industry of course.

Closing the loop with the CLA is similar. The Department of Higher Education or other authorities pressure institutions to use vaguely-defined measures of achievement, of which the CLA is an example. This grants a lot of power to the test maker. An institution that wants to do well is caught in a trap. How exactly to prepare students for the CLA? What context is necessary, and what particular skills should be taught? The test maker knows best. With CLA in the Classroom the company can not only diagnose your problem, but fix it too. To the naive, this may seem like a good thing, just like a patient demanding the new drug for "Restless Chin Syndrome," or whatever the malaise du jour is. In Assessing the Elephant [pdf] I call this sort of thing a "degenerative loop." That is, it looks like a closed assessment loop, but it has potentially nothing to do with your actual goals. It's natural that in an environment where goals are hard to measure or even define that such things crop up like mushrooms. Think of astrologers and miracle cures and mysticism.

It could be that the CLA does in fact align very nicely with your institution's goals. This is something to investigate carefully. If so, it could probably be as useful as anything you can construct internally, but with the added benefit of having external validation. But if you can't wrap your mind around the outcomes being assessed, or they don't look like the ones you value, you're essentially outsourcing the most important part of assessment. And the pressure to perform will probably have you looking up the phone number for CLA in the Classroom...