Numbers and Names

Words have meaning. This is true even if the words are not formally defined; people could talk to each other before dictionaries came around. The facility to speak and understand is so fluid in fully-functioning humans that we underestimate how difficult it is (see Moravec's Paradox). Undoubtedly there was strong evolutionary bias toward creating this ease of communication, as opposed to making our brains facile with long division, for example.

Because words are powerful, they get hijacked. There is economic value attached to the effect of certain words, like "new and improved" and so they get put into use like blue-collar workers marching off to punch in. Sometimes this is manipulative or cynical, as brilliantly illustrated by Orwell in 1984. In the Russian revolution, Bolsheviks were pitted against Mensheviks, names stemming from a narrow vote. The former word means "majority" and the latter "minority." Imagine if your political faction is saddled with with the second name...

In outcomes assessment, or reporting out psychometrics in general, the use of common words is sloppily introduced. I've addressed the big one: "measurement" elsewhere. Another good source for this kind of error propagation is studies that use factor analysis. I came across a good example while reading "A Look across Four Years at the Disposition toward Critical Thinking Among Undergraduate Students" by Giancarlo and Facione while browsing Insight Assessment's research page. This company produces the Critical Thinking Dispositions survey I blogged about here. I don't mean to be critical of the authors, but rather highlight a practice that seems to be endorsed by most who write about such things. The study itself is interesting, giving a before-and-after look at undergraduates as assessed by the survey. They introduce the topic of dispositions thus:

Any conceptualization of critical thinking that focuses exclusively on cognitive skills is incomplete. A more comprehensive view of CT must include the acknowledgement of a characterological component, often referred to as a disposition, to describe a person’s inclination to use critical thinking when faced with problems to solve, ideas to evaluate, or decisions to make. Attitudes, values, and inclinations are dimensions of personality that influence human behavior.

Notice the implication that personality comes in dimensions. Dimensions are by definition independent of one another, and as we shall see, the idea is that we can assemble a linear combination of these pieces to assemble a whole disposition. This is an enthymeme without which the rest of the analysis cannot proceed, but it's a big leap of faith. As such, it ought (in the research community) to be spelled out explicitly. The mindset that attitudes and values and inclinations together create some kind of vector space is so wild that you'd think caution would be advised. If the implication is that these dimensions really are orthogonal (completely independent of one another), it's ridiculous on the face of it. What does it mean to have a very small amount of "attitude" but lots of "inclinations?"

Most things are not linear. If I'm talking softly, you may only hear bits and pieces of what I hear. Increasing the volume will enable you to hear me clearly within a range, but we wouldn't be so bold as to say "talking twice as loud makes you understand me twice as well." We use linearity not because things are linear but because it makes it easy to do the analysis. In small ranges, it often makes sense to approximate non-linear phenomena with linear models, but one has to be careful about reaching conclusions.

In the article, the assumption is that the disposition to think critically is the linear combination of a few component dimensions. These are listed:

Factor analysis of the CCTDI reveals seven distinct elements. In their positive manifestation, these seven bipolar characterological attributes are named truthseeking, open-mindedness, analyticity, systematicity, critical thinking (CT) self-confidence, inquisitiveness, and maturity of judgment.

Notice the passive voice "are named." Are named by whom? Here's the process: The survey is administered and the results recorded in a matrix by student and item. A correlation matrix is computed to see what goes with what. Then a factor analysis (or singular value decomposition, in math terms) is performed, which factors the matrix into orthogonal dimensions. To understand this, it helps to look at an animation of a simple case. If the dimensions have different "sizes" (axes of the ellipse in the animation), then a more-or-less unique factorization results. If the dimensions are close to the same size, it's hard to make that case. Each dimension is defined by survey items and associated coefficients. It supposedly tells us something about the structure of the results. Note that orthogonal means the same thing it did earlier: completely independent. You can have zero of one factor and lots of another, and this needs to make sense in your interpretation.

So: we do some number crunching and find associations between items. These are collected together and named. We could call them vector1, vector2, and so on, but that wouldn't be very impressive. So we call them "openmindedness", "attentiveness," and use words that already have meanings.

It's not even clear what the claim actually is. Is it that we humans perceive critical thinking dispositions as a linear combination of some fundamental types of observation, presumably presented to us in whole form by our perceptive apparatus? Or is it that in reality, our brains are wired in such a way that dispositions are generated in as linear combinations?

It would be relatively easy to test the first case using analysis of language, like the brilliant techniques I wrote about in "High Five." I don't see any evidence that this sort of thing is done routinely. Instead, researchers eyeball the items that are associated with the dimensions that pop out and give them imaginative names. They may or may not be the same names that you and I would give them, and may or may not correspond to actual descriptions that someone on the street would use to describe the test subject.

I hope you can see the sleight of hand by now. In the case of this particular article, the authors go one step further, by describing in detail--in plain English--what the dimensions are (I have bolded what was underlining in the original):

The Truthseeking scale on the CCTDI measures intellectual honesty, the courageous desire for best knowledge in any situation, the inclination to ask challenging questions and to follow the reasons and evidence wherever they lead. Openmindedness measures tolerance for new ideas and divergent views. Analyticity measures alertness to potential difficulties and being alert to the need to intervene by the use of reason and evidence to solve problems. Systematicity measures the inclination to be organized, focused, diligent, and persevering in inquiry. Critical Thinking Self-Confidence measures trust in one’s own reasoning and in one’s ability to guide others to make reasoned decisions. Inquisitiveness measures intellectual curiosity and the intention to learn things even if their immediate application is not apparent. Maturity of Judgment measures judiciousness, which inclines one to see the complexity in problems and to desire prudent and timely decision making, even in uncertain conditions (Facione, et al., 1995).

These descriptions would serve suitably for ordinary definitions of ordinary terms (without the use of "measurement"), but no evidence is presented that the ordinary meanings of all these words corresponds in any way to the factor analysis results, other than that someone decided to give the dimensions these names. The final touch is claiming that we "measure" these elements of personality with precision:

For each of the seven scales a person’s score on the CCTDI may range from a minimum of 10 points to a maximum of 60 points. Scores are interpreted utilizing the following guidelines. A score of 40 points or higher indicates a positive inclination or affirmation of the characteristic; a score of 30 or less indicates opposition, disinclination or hostility toward that same characteristic. A score in the range of 31-39 points indicates ambiguity or ambivalence toward the characteristic.

All of this strikes me as absurd. It's not that surveys can't be useful. To the contrary, they undoubtedly can give us some insights about student habits of mind. But to suppose that we can slice and dice said behaviors with this precision is far over-reaching, particularly in the use of ordinary language to create credibility without proof that these associations are strong enough to withstand challenge.

This practice is unfortunately common. The NSSE reports include dimensions like this, for example.

Dim Dim

It just struck me that I probably now have no way to access the first code I wrote--programs for the Apple II--back in the 1980s. I suppose some of it is on floppy disks in my parents' basement, but I imagine they've leaked more than a few bits by now.

I was led to this thought as I typed the title of this piece because DIM is BASIC syntax for declaring an array. It stands for dimension, and that's what prompted me to put fingers to keyboard. Not the headache inducing 10 dimensions of physical spacetime that string theory (or M-theory) hypothesizes or the span of the metaverse required to banish the cosmological anthropic principle, but ordinary everyday uses and abuses. Well, mostly abuses.

I was amazed to learn over lunch this week that our course numbering scheme is more or less restricted so that the middle digit is the number of credit hours of the course. Given that the first number is used to denote the level of difficulty, this only leaves one digit out of three to actually identify the course content. I was made aware of this when a faculty member asked me if we could increase it to four digits. The problem is obvious: if you have linked course content across levels, like Math 305 Intro to Analysis, then it's nice to number Intermediate Analysis as 405, and so on. But if you only have a total of 10 courses per credit hour size that becomes difficult. Since most courses are three credits, the information in the second digit is mostly wasted. One might say that it's completely wasted since the number of credit hours is always listed next to the course anyway. There are in effect three dimensions crammed into one number:

1. First digit is level of course
2. Second digit is number of credits
3. Third digit identifies the actual course

It would be better to separate out all three into three fields. Tradition dictates that this will never happen with the first one, of course.

Another example is common in any database where code tables are used. Suppose you want to categorize college applicants by day, evening, accepted, rejected, withdrawn, awarded aid, or declined aid. Sometimes you see these all crammed into the same code table with either numbers or letter abbreviations to identify them. It becomes a terrible mess to run reports with such an arrangement. In this case, day/evening is one dimension and should get its very own field. Same with accepted/rejected. Withdrawn needs a field because it could happen before or after being accepted. And financial aid needs a whole bunch of fields, of course.

The examples above are more a nuisance than anything else, but it shows something about how our brains work. We like to be clever and combine things, even when they shouldn't be. Once different qualities are packaged together we sometimes have trouble even imagining that they are separate. In a ACM article by Lance Fornow called "Viewpoint: Time for computer science to grow up" there is evidence that this tendency runs deep indeed.

The author describes how computer science has not grown up around the journal culture, as older disciplines have. Instead, researchers use conferences and conference proceedings as a way to disseminate ideas and document professional competence. The problem he presents is that there are too many conferences, which fragments the community, and that it's too expensive and arbitrary--who gets to present at a conference is not determined as fairly as one might wish. He argues for the growth of a strong journal publication system as a solution.

Maybe you noticed in the second sentence of the paragraph above that there are two goals in two dimensions: disseminate ideas and document competence (e.g. for tenure review). The second goal has been attached to the first for so long--like embedding a difficulty level in a course number--that we don't necessarily notice they're two very different things. Daniel Tunkelang notes this in a comment on the source article, with two points I'll highlight here:

1. [I]t makes no sense for publishers to act as filters in an age of nearly-free digital distribution.
2. The peer-review process (and review processes in general) should serve to endorse content--and ideally even to improve it--rather than to filter it.

In the days of yore, publishing was expensive, and so only so much stuff could be published. (How long did it take them to catch up to everything Euler wrote?) Therefore, filters on what got published were reasonable. Filters imply standards, and turning things around "published => worthwhile" became accepted. It's high time those two dimensions were decoupled. Why on Earth would we want to filter work before it's published if we don't have to? The Internet has lifted all reasonable restrictions on the ease of disseminating ideas. Figuring outhow to decide quality is a whole other problem, and I venture to say the answer that evolves will be far more robust than the slow and arbitrary peer-review that created the journal culture.

The physics preprint site Arxiv.org is a perhaps a prototype for the publication model. There are modest barriers to publication there (you have to get endorsed once, based on a glance at your work to make sure it's not a drawing of a spider or something silly). Do people look at it, or is it a "write-only" repository? Here's Alexa, comparing ACM, IEEE, and Arxiv.
The bottom line is Arxiv.org, showing it to be a third to quarter the traffic of acm.org, measured in page views, reaching (another graph) .0078% of global Internet users. I couldn't figure out Alexa's interface well enough to find the raw number of page views, but Wolfram Alpha came to the rescue (reporting Alexa stats, ironically), putting Arxiv.org at about 100,000 daily visitors, with about two pages views each. That seems pretty healthy to me.

It isn't hard to imagine how ratings or rankings might get layered on top of such an archive, but I hope that doesn't happen. Much more meaningful would be hot-linked citations, keywords, and other richly-connected data, perhaps including comments (NOT anonymous ones, though). Determining the merit of a scholar based on his or her work will always be a messy, subjective, and political process. The tenure review process needs a reboot. The incentive is that those who figure out the best way of doing it get to attract and keep the best researchers. That's how evolution works...

Rubrics and Dimensionality

Maybe I'm just a geek, but I thought it was really cool the first time I ran into the idea of dimension analysis in a physics class. Wikipedia has a nice page on it here, but the basic idea is simple. In common parlance you can't compare apples and oranges. If you derive some cool new equation, but then discover that the left side is a number and the right side is a vector, something went wrong. A related problem, which we'll consider together, is that of mismatched units. You can't legitimately ask "If three apples together weight 2 pounds, and a stick is six inches long, which is more?" My dad claims that in basic training the sergeant used to yell "Ahright yous guys! Line up in alphabetical order according to height!"

The picture of a town sign I used a few days ago illustrated the comedy of mismatched units nicely, summing up population, feet about sea level, and the year founded to get a total.

I was prompted to think about dimensions because of the rubric I came across for Washington State's critical thinking project (CT). The rubric has seven dimensions, with levels of accomplishment ranging from emerging to developing to mastering. The whole thing seems polished, well-thought out, and generally nicely done. If you want at critical thinking rubric, this is probably as good as it gets. Personally, I think the idea of critical thinking itself is too fuzzy to be as useful as it seems, and have written about that previously here and here.

In reading through the CT material, it seems that one of the main effects comes from classes that simply use the rubric in class. Quoting from their findings:

In the four courses where the rubric was used variously for instruction and evaluation, the papers received significantly higher ratings than in the four courses in which the rubric was not used.

Dialogue is almost always better than monologue in a teaching environment. See here for more on that idea. A rubric used to spark conversation and generate content is better used than one that simple waits in the drawer until rating time. I guess this should be obvious, but I didn't realize how powerful that technique is until I started using it in the classroom myself. Of course, you can't have a massively complex rubric or you'll just get the 1040 effect (tax form reference--it's April 15, after all!).

Because the WSU example is a good, well developed rubric, we can use it to explore some problems with rubrics in general, particularly as regards dimensionality. For example, the WSU graph of results and accompanying chart makes the usual mistake of averaging the dimensions to get a "CT average score." The resulting numerical goo may actually correlated with something (that is, have some predictive validity), although I didn't see that claim made. But philosophically such operations are always suspect for the same reasons grades are--all dimensionality is surrendered. The implicit assumption is that all dimensions are equally weighted, that the average is somehow related to the actual critical thinking ability of the student. It's similar to saying something like:

I've measured a multidimensional box, of which I only saw seven dimensions. There may be more I don't know about. But the average dimension of the box was 3.4 inches.

Actually, the situation is much worse if we assume that the dimensions have different units--that would be like averaging pounds and inches. However, as I said, this is a common practice, and this kind of graph shows up a lot in reports.

Each dimension is assessed subjectively, for example to what extent does a student integrate issues using OTHER (disciplinary) perspectives and positions (dimension 5). Ultimately the rater chooses a numerical score from 1 to 6, representing subjective judgments like for "developing":

Ideas are investigated, if in a limited way, and integrated, if unevenly.

This is in the middle of the rubric, for a score of 3 or 4. Not all dimensions apply equally to all problems, however. Taking an actual example, look at the puzzle How many triangles? It shows a geometric figure and asks you to count how many triangles there are. Is this a critical thinking problem? If so, which dimensions apply to it, and which do not? Are there dimensions that are not found in the rubric that are important? For this particular problem, there is a needed insight--a flash of creativity--that is required to notice that small triangles can overlap to make big triangles. I don't see any dimension in the rubric that corresponds directly to creativity. Is creativity part of critical thinking?

Most of the problem is with the idea of critical thinking, not the rubric. But we should be cautioned not to put too much faith in numerical goo that comes out of such grading schemes. The real utility is in the classroom, to spark discussion about techniques of thinking and to focus in on particular ones for particular assignments. But there is another problem that is philosophically worse: the assumption that we can obtain macro-level assessments from micro-level ones by adding up components. This is a mereological fallacy.

What is the real critical thinking score? What if we simply asked instructors, supervisors, or other educated adults to observe the students and assess directly (subjectively) the students' critical thinking ability? This is no more suspect than the subjective assessments of the dimensions given in the rubric, after all. If subjectivity is a problem, then we have to throw out most rubrics altogether. We're already wired with the ability to make such complex judgments. But they are almost certainly not simple linear relationships of components we observe. We understand intuitively that some demonstrations of thought (critical thinking, analytical thinking, creative thinking, effective writing, etc.) will heavily use some techniques and not others. Critical thinking takes so many forms that it's impossible to say from one instance to the next what will be the most important element or mode of analysis.

In summary: don't average rubric scores unless you can relate it to something meaningful through predictive validity, look instead for direct subjective measures, don't take the idea of dimension and measurement too seriously, and realize that the rubrics probably leave out all kinds of important stuff. The best use of the rubric is for pedagogy, not assessment. In fact, it would be a good exercise to have students critique the rubric to see what's missing. That would be a real exercise in critical thinking.