The picture of a town sign I used a few days ago illustrated the comedy of mismatched units nicely, summing up population, feet about sea level, and the year founded to get a total.
I was prompted to think about dimensions because of the rubric I came across for Washington State's critical thinking project (CT). The rubric has seven dimensions, with levels of accomplishment ranging from emerging to developing to mastering. The whole thing seems polished, well-thought out, and generally nicely done. If you want at critical thinking rubric, this is probably as good as it gets. Personally, I think the idea of critical thinking itself is too fuzzy to be as useful as it seems, and have written about that previously here and here.
In reading through the CT material, it seems that one of the main effects comes from classes that simply use the rubric in class. Quoting from their findings:
In the four courses where the rubric was used variously for instruction and evaluation, the papers received significantly higher ratings than in the four courses in which the rubric was not used.Dialogue is almost always better than monologue in a teaching environment. See here for more on that idea. A rubric used to spark conversation and generate content is better used than one that simple waits in the drawer until rating time. I guess this should be obvious, but I didn't realize how powerful that technique is until I started using it in the classroom myself. Of course, you can't have a massively complex rubric or you'll just get the 1040 effect (tax form reference--it's April 15, after all!).
Because the WSU example is a good, well developed rubric, we can use it to explore some problems with rubrics in general, particularly as regards dimensionality. For example, the WSU graph of results and accompanying chart makes the usual mistake of averaging the dimensions to get a "CT average score." The resulting numerical goo may actually correlated with something (that is, have some predictive validity), although I didn't see that claim made. But philosophically such operations are always suspect for the same reasons grades are--all dimensionality is surrendered. The implicit assumption is that all dimensions are equally weighted, that the average is somehow related to the actual critical thinking ability of the student. It's similar to saying something like:
I've measured a multidimensional box, of which I only saw seven dimensions. There may be more I don't know about. But the average dimension of the box was 3.4 inches.Actually, the situation is much worse if we assume that the dimensions have different units--that would be like averaging pounds and inches. However, as I said, this is a common practice, and this kind of graph shows up a lot in reports.
Each dimension is assessed subjectively, for example to what extent does a student integrate issues using OTHER (disciplinary) perspectives and positions (dimension 5). Ultimately the rater chooses a numerical score from 1 to 6, representing subjective judgments like for "developing":
Ideas are investigated, if in a limited way, and integrated, if unevenly.This is in the middle of the rubric, for a score of 3 or 4. Not all dimensions apply equally to all problems, however. Taking an actual example, look at the puzzle How many triangles? It shows a geometric figure and asks you to count how many triangles there are. Is this a critical thinking problem? If so, which dimensions apply to it, and which do not? Are there dimensions that are not found in the rubric that are important? For this particular problem, there is a needed insight--a flash of creativity--that is required to notice that small triangles can overlap to make big triangles. I don't see any dimension in the rubric that corresponds directly to creativity. Is creativity part of critical thinking?
Most of the problem is with the idea of critical thinking, not the rubric. But we should be cautioned not to put too much faith in numerical goo that comes out of such grading schemes. The real utility is in the classroom, to spark discussion about techniques of thinking and to focus in on particular ones for particular assignments. But there is another problem that is philosophically worse: the assumption that we can obtain macro-level assessments from micro-level ones by adding up components. This is a mereological fallacy.
What is the real critical thinking score? What if we simply asked instructors, supervisors, or other educated adults to observe the students and assess directly (subjectively) the students' critical thinking ability? This is no more suspect than the subjective assessments of the dimensions given in the rubric, after all. If subjectivity is a problem, then we have to throw out most rubrics altogether. We're already wired with the ability to make such complex judgments. But they are almost certainly not simple linear relationships of components we observe. We understand intuitively that some demonstrations of thought (critical thinking, analytical thinking, creative thinking, effective writing, etc.) will heavily use some techniques and not others. Critical thinking takes so many forms that it's impossible to say from one instance to the next what will be the most important element or mode of analysis.
In summary: don't average rubric scores unless you can relate it to something meaningful through predictive validity, look instead for direct subjective measures, don't take the idea of dimension and measurement too seriously, and realize that the rubrics probably leave out all kinds of important stuff. The best use of the rubric is for pedagogy, not assessment. In fact, it would be a good exercise to have students critique the rubric to see what's missing. That would be a real exercise in critical thinking.
