Testing and the Monologue Problem

Assessment of learning is usually done as a monologue. I mentioned this idea a couple of days ago ("Generalized Silliness"), and it's explored more fully in Assessing the Elephant. A monologue in this context is any standard or definition that is rigorously applied to a subject, like holding up a ruler to a banana to see how long it is. This practice is monological in the sense that the banana doesn't get to negotiate or appeal the decision, nor are any factors not contained in the ruler's definition of length that are considered. For example, the banana cannot argue thus:

Dear ruler, you have underestimated me. For if I could just uncurl a bit, you'd see that I am a good two inches longer than you have measured.

That would begin a dialogue, which is not permitted in the example given. More generally, monologues have two basic limitations: they do not permit perspectives outside the original definitions, nor are the adaptive: they cannot take new information into account and change their definition. Examples are all around us.

In sports like basketball, time is carefully meted out in a monological fashion. Time on the scoreboard has some relation to a fan's subjective experience of the temporal flow, but is an artificial 'official' version that the teams had better adhere to if they want to win.

Most tests of education outcomes are like this. Not all, of course--things like eportfolios can be wonderfully complex and are good at producing dialogue (if one wishes to do that). But if you think of the typical testing setting, it consists of students interacting with a pencil and a piece of paper. This is a monologue. Talking back to the test does little good, although I've had adult students write prayers on math finals before. But celestial appeals for the power to solve quadratics aside, a math test is pretty much a one-way conversation.

Note that this is unnatural. In most endeavors with humans, we talk or otherwise correspond to find out things.

* Know any good restaurants around here?
+ Sure--there's an Indian place on Central, about three blocks.
* Anything closer? I'm in a hurry.
+ Oh, well--there's a sub joint just around the corner.

Compare that to the prompt:

Q1) List all eating establishments within three blocks and identify type of food and approximate price for each.

The first example is a dialogue; the second is a Google maps search.

Monologue is good for testing things like 2+2=4. (The proof of which is very complex by the way!) Low complexity tasks and completely algorithm ones are good candidates for assessment by monologue. Solving differential equations, reciting facts about the Civil War (an oxymoron if there ever was one), and so forth may as well be done in a monologue. But for more complex types of learning, it becomes a limitation. Consider the following example.

At the campus of Assess U., seniors are expected to get up early one Saturday in order to go take a standardized test of their thinking skills. This will be a three hour session with pencil and paper. They will never see the results of the test, but they are advertised as being important to judging the effectiveness of the general education curriculum or something. Stanislav's alarm goes off at 7am and he considers his options. Having had a course in Game Theory, he quickly realizes the optimal solution for him is to simply sleep in. What are they going to do--not let him graduate?

In the example, Stanislav does some serious critical thinking while denting his pillow, and decides that taking a test on critical thinking is not in his best interests. His professors, who all know he's great at solving problems, and would recommend him to any employer or graduate school, have no voice on the assessment either--by demonstrating the skill tested by the assessment, Stanislav's 'measurement' completely vanishes from the sample.

The example is tongue-in-cheek, but it does illustrate the point that only 'official' kinds of thinking skills are going to be seen by the test. This list of certified ways of demonstrating skill or knowledge is much more limited in a monlogue than it is in a dialogue. Think of the game of twenty questions, where you try to guess an animal based on yes/no questions. With 20 questions, one can exhaust up to 2^20, or about a billion possibilities (if it's perfectly binary). With a monologue, you can only explore 20. Think about that--with only 20 questions, a dialogue is about fifty million times more discerning. That's because a monologue cannot adapt from one question to the next.

Examples of dialogue-based assessments include interviews, oral exams, conversation, writing with revision, portfolios (potentially), and group work of various types. These are the rich interactions that we use routinely to assess the characteristics of the other humans around us. Formalizing these dialogues (but just barely) is a great way to get authentic assessment data that isn't skewed by the 'official' monologues and that can adapt to new circumstances. In the end, what we assess and how we assess needs to evolve in order to improve. Dialogues are essential to this process, which explains the vapor lock that often occurs when an assessment committee confronts the results of a standardized test and suddenly is faced with having to make something of it.