In the last installment of this assessment melodrama, our heroic assessment director (pictured below) was given two Homeric tasks: convince the denizens of the deep (teaching faculty) to integrate assessment into the classroom AND simultaneously create aggregate "measures" of learning for the high priests of wesaysoism (administration and regulatory bodies)*.
We might characterize these two challenges as the micro and macro, and it's hard enough to accomplish them without confusing the two, so I'll say a few more words about the distinction today, and then start addressing my promise to talk about what we call "measurement" in assessing outcomes. This idea is almost ubiquitous in the field and pertains to both macro and micro scale. In either case it can be wildly misused.Micro Assessment is what I'm calling assessment for learning. The point of the assessment bit is to increase the effectiveness of the classroom, curriculum, and mission of the institution. This may mean coinciding with general education outcomes or something. The point I tried to make in yesterday's harrowing installment was that this effort has to be credible. One of my maxims of life is that people do things for reasons that make sense to them. If it doesn't make sense for the instructor to take the effort to integrate some learning outcomes into assignments, it won't happen. I keep talking about philosophy (in the common, not academic sense) as essential to this project, so it was interesting yesterday to read an article in the Atlantic by a philosopher, Matthew Stewart, who got into the business of management consulting and became quite successful at it. He is nevertheless rather harsh on his adopted profession:
I make no judgments about Mr. Stewart's erstwhile profession (he left eventually), but the account makes a good cautionary tale for the intrepid assessment director: don't approach the task with "papal infallibility," and think more like a coach than a scientist.The thing that makes modern management theory so painful to read isn’t usually the dearth of reliable empirical data. It’s that maddening papal infallibility. [...]
Each new fad calls attention to one virtue or another—first it’s efficiency, then quality, next it’s customer satisfaction, then supplier satisfaction, then self-satisfaction, and finally, at some point, it’s efficiency all over again. If it’s reminiscent of the kind of toothless wisdom offered in self-help literature, that’s because management theory is mostly a subgenre of self-help. Which isn’t to say it’s completely useless. But just as most people are able to lead fulfilling lives without consulting Deepak Chopra, most managers can probably spare themselves an education in management theory.
Macro Assessment is where the real damage can be done. About the worst that can happen by annoying the faculty with administrivia is creeping irrelevance for the director and a lot of unpleasant meetings. But the macro assessments might actually get used in policy. For example, there's the notion of an exit exam. These can be implemented at different times, and for different purposes. One option is to create a graduation requirement of passing the test, so that it becomes the finger of fate, as it were, for many students. Education Week has an April 27 article on the topic, by Debra Viadero called "Scholars Probe Diverse Effects of Exit Exams." Part of a graphic is reproduced below, showing states that employ this technique in public schools.
More than anything, such testing is about faith. The effects of failing are damaging, not just because of the lost time and effort, but because of the apparent loss of faith in one's own abilities by those who fail, as noticed by independent researchers:The cause and effect seems to be plausibly that a (minority or female) student's failure results in a downgraded self-assessment of his or her own abilities, which causes a real degradation in ability to pass the test.[... Researchers] in a study looking just at students who barely passed or barely failed that state’s exit exam in 10th grade, found that being labeled a failure can have a detrimental effect on low-income students in urban schools.
Even though students have plenty of opportunities to retake the exam—and most do—poor, inner-city students who just missed the passing cutoff in 10th grade are 8 percentage points less likely to graduate on time than demographically similar students who just barely passed, even though both groups scored at roughly the same levels on the 10th grade exam. Failing or passing the tests seems to have no statistically significant effect, though, on the probability of graduation for wealthier, suburban students.
What about our own faith in these tests? If we believe in the reliability and validity of the exit exam, we'd have to conclude that simply taking the test and seeing the result was enough to cause students to know less than they did before. There's no real way out of that conclusion if you believe that we are really measuring learning with such processes.
This is just one short example of the subtleties at play, and the unexpected consequences of macro-assessment. There is lots more to discuss on the topic. What I'd like to do over the next few installments is begin to peal away the layers of assumptions about this idea of measurement and see what's there. What is it, really, that we put so much faith in? A quote from Thomas S. Dee, a Swarthmore College economist concludes the article with:
“The cynic in me worries that we’re just going to continue to see these policies proliferate, because it seems like an obvious way to convey the expectations that we should have for students,” Mr. Dee said, “and the negative effects appear to be hidden from public discussion.”As Henry Louis Mencken said, “For every problem there is a solution which is simple, clean and wrong.” This is the second cautionary tale with which to begin to frame the discussion on outcomes measurement. Stay tuned...
Next: Part Seven
*Note that I mean no disrespect for either group, especially since I happen to be in both. But this is how they sometimes speak of each other, so it's useful to highlight this tension.
No comments:
Post a Comment