This is part three of an examination of why it's hard to find compelling outcomes assessment loops in the wild. [Part one] [Part two]
Yesterday I tried to make the point that there can easily be a disconnect between the actual elements of teaching and learning and the goals that we set for the same. This difficulty can be glossed over by assuming that the parts transparently make a whole. What can easily happen is the sort of thing pictured below.
It should not be assumed to be valid to take a list of skills and bits of knowledge in an academic area and "sum them up" to get a learning index. Yes, I know we do it all the time, to create test grades, course grades, overall grade averages, standardized tests, rubric scores, and so forth. But the usefulness of these products is limited by their validity.
Suppose as an advisor you want to help a student improve her overall GPA. How would you do it? Without further information, this is likely to be difficult. You might want to know what courses she's having the most trouble with, what her study skills are like, if she's engaged in a learning community, and so on. The point is that the GPA itself doesn't tell you very much about actual teaching and learning. It's simply a compressed mess of data.
Hence a common confusion. The department of Transcendental Linguistics receives the standardized test scores for Enlightenment, and they are 5% lower than the average from all institutions that use the instrument. The chair of the department is told to do something about it. What to do? Without further information, this index is not of much use. It's still possible to improve the scores by using an evolutionary approach (try a bunch of things randomly and see what works), but this may take millions of years to produce students who are one with everything. On the other hand, if the chair gets a copy of the test and teaches the specific skills found there, he might be accused of 'teaching to the test,' and face a faculty mutiny. There is a real dilemma here, and it derives from the hidden assumption that there is transparency between what happens in a learning environment and how the test scores come out.
Reductionism only works in particular circumstances. It works particularly when a transparency (i.e. cause and effect relationship) can be developed between the micro- and macro-scale. If we want to know why the car won't start, reductionism is a good way to find out because there are clear cause and effect relationships between functions that determine whether or not a car starts. You'll have less success predicting the stock market that way.
The problem isn't that we look at the details (microscopy is not reductionism), and the problem isn't that we want to know about the big picture too. The problem is the link between the two.
A few days ago I wrote about Moravec's Paradox [here], which states that the evolutionary age of our mental wiring is correlated with its complexity. Things that we do effortlessly are deeply optimized, and highly complex. Easy doesn't mean transparent. So, for example, if we read a wonderful new novel, the impressions and emotions that result are undoubtably the result of neurology, but very deep and complex, and not the kind of thing that we could deconstruct with a rubric. If it were that easy, every movie made would be a smash hit. Subjectivity is the most powerful tool of discrenment we have. It's easy to lose sight of that in the quest to regularize and make scientific our educational assessments.
Maybe you can explain jazz music by looking at the way molecules interact, but count me skeptical. Sometimes reductionism is the wrong approach. So what to do? I have a couple of modest suggestions.
Keep track of the details. Do the curriculum maps. Make the rubrics. But don't try to sum up the details into anything meaningful. Don't average results--it just makes numerical mud (a thousand blabbering fools cannot be summed to equal the product of a fine orator). Use non-parametrics instead, like maximum, minimum, and plots of distributions. Keep the data complex, in short. Base changes on the details. If students can't do integration-by-parts, then find another way to teach them. If they can't figure out how to conjugate verbs in the subjective, talk to your colleagues who have better luck with it. For the most part, the big picture will take care of itself.
Assess the big picture. If you've followed this blog for any time, you may be puzzled because I talk mostly about the overall assessment of complex skills. That's the whole assessing the elephant thing. You can follow the link to read the details about how to do it--it works well for small class sizes (20 or less), but won't work for huge lecture halls. Generate subjective impressions about analytical thinking, effective writing, or whatever complex big-picture traits you find important in your graduates.
Create a conversation. If the details are dealt with, there ought to be changes on the large scale. Defining a transparent linkage between the two is probably impossible, however. More important than trying to find one, is generating a dialogue between instructors and administrators and students about what the goals are, and to establish a common language about them. This ongoing conversation will naturally inform the nuts and bolts changes at the detail level. There's absolutely no need to try to construct an analytical ladder between the two by, say, averaging up a bunch of rubric ratings to find an 'overall critical thinking score'. It's vital that the constituents own the process--ideally it is their own subjective impressions of student performance that drive the assessment.
Next: Part FourMore explorations of ideas on this topic in the next installment. I'm taking my daughter to see her grandparents next week and may or may not be able to get back to the internet tubes to blog during that time.
No comments:
Post a Comment