Showing posts with label formative. Show all posts
Showing posts with label formative. Show all posts

Monday, June 01, 2009

Evolving Teaching with Formative Assessments

I found a very thoughtful article by Paul Black and Dylan Wiliam hosted at Phi Delta Kappa called "Inside the Black Box: Raising Standards Through Classroom Assessment." The article summarizes the authors' research in attempting to answer these questions:
  • Is there evidence that improving formative assessment raises standards?
  • Is there evidence that there is room for improvement?
  • Is there evidence about how to improve formative assessment?
The answer, they say, is a clear yes to all three. They make the point that teaching is currently largely about management, and more attention is paid to the bureaucratic duties than teaching effectiveness. The recommendation is to do more than stick a few assessment pieces onto the existing structure like ornamentation:
The research studies [...] show very clearly that effective programs of formative assessment involve far more than the addition of a few observations and tests to an existing program. They require careful scrutiny of all the main components of a teaching plan. Indeed, it is clear that instruction and formative assessment are indivisible.
As an example of the collision between management and teaching, consider the following example, from the paper.

One common problem is that, following a question, teachers do not wait long enough to allow pupils to think out their answers. When a teacher answers his or her own question after only two or three seconds and when a minute of silence is not tolerable, there is no possibility that a pupil can think out what to say.

There are then two consequences. One is that, because the only questions that can produce answers in such a short time are questions of fact, these predominate. The other is that pupils don't even try to think out a response. Because they know that the answer, followed by another question, will come along in a few seconds, there is no point in trying.
So here, the assessment (asking questions) is deemed ineffective. "What emerges very clearly here is the indivisibility of instruction and formative assessment practices."

The article closes with advice on implementation:
What teachers need is a variety of living examples of implementation, as practiced by teachers with whom they can identify and from whom they can derive the confidence that they can do better. They need to see examples of what doing better means in practice.
Assessment directors or coordinators will recognize this as good advice. Training everyone to re-think their teaching in a mass re-education project isn't likely to work. It takes time and focus, diffusing knowledge out from local experts who are trusted. This difficulty is due to the individualized nature of the problem--every teacher has to find ways that work for him or her, for his or her subject area. There is no BandAid fix:
This study suggests that assessment, as it occurs in schools, is far from a merely technical problem. Rather, it is deeply social and personal.
This account rings true to me, and underlines what teachers and many in the assessment profession already know--that it's not enough to mandate "assessment" from on high and expect anything good from it. Approaching the problem as a bureaucratic exercise leads naturally to "assessments" of the same time--reams of statistics dissociated from the practice of teaching and learning. Actual improvement has to start with more personal motivations--"yes, I care about how my students learn." Leaders can move an institution in that direction, but I don't think it can be mandated and then considered solved.

Thursday, April 02, 2009

Why Documenting Learning Outcomes is Hard, Part One of Infinity

Pat Williams recently asked if anyone is closing the proverbial loop on her blog. Quote:

For almost all of us the question, "What are you going to do about it?" has proven the toughest to answer.
If you've ever written assessment reports yourself, or had to judge ones others have written, you'll probably find it hard to disagree with this statement. When I first confronted the task of documenting those for our SACS report, it became obvious that there were some institutional disconnects at the accreditation level. For example, if you look back at the old SACS general education requirement (3.5.1), it was written as a minimum standards policy (you must set standards and show that students meet them), but in practice, every single person I talked to at the SACS conferences used a continuing improvement philosophy to actually judge compliance! This included IR people who'd been on many campus visits as well as senior vice presidents of the Association speaking in formal presentations. I found this bizarre and almost Kafkaesque. I pointed this out in emails when SACS invited comment. I don't know how my suggestions were received, but the standard was changed.

The point here isn't really about SACS. It's about the confusion surrounding standards-based vs. continuing improvement models. Public schools are primarily standards-based in their approach. Results are determined by standardized tests. If the learning objectives are simple enough (in the formal sense of complexity, which I've written about here many times), this approach should work well enough. This is because the items on the test can correspond pretty precisely to the material being taught. For example, if we want to know if third graders know their multiplication tables, this is relatively straightforward.

Things go wrong when the assessment doesn't align with the actual goals. This is a question of validity of the test, but it often goes unnoticed. By believing too much in the wrong assessment, we can create what I call degenerate assessment loops (more here). You can read a nice parable about this in The Well Intentioned Commissar.

Continuous improvement is orthogonal to standards. If one mixes up standards-based assessment and continuous improvement, confusion quickly sets in. Even in the SACS 3.3.1 standard of old, there was ambiguous language (I complained about that here, and it changed too, but it's probably coincidence). The problem is that many people assume that you should be able to show continuous improvement in some particular metric. This is virtually impossible to do once learning outcomes have reached a certain level of complexity.

We think of many assessment situations as falling into one of these categories:
  • Summative and transparent An example of this is low-complexity tasks like learning multiplication tables. It's easy to judge how things are going, and relatively easy to effect change (that's the transparent part). One can apply standards here effectively.
  • Summative and Opaque Here it's easy to see where you are, but hard to know how to get where you want to go. An example of this is the stock market. In learning outcomes assessment, this category applies to standardized tests of complex behaviours, which necessarily reduce the complexity (and hence validity). It's easy to see if the numbers go up or down, but hard to know how to affect them (without, say, directly teaching students how to take the test). Standards can be applied here, but it's very hard to use them for accountability, since there's no guaranteed way to reach new heights of achievement.
  • Formative and transparent In this case, subjectivity is valued more than objectivity, and standardization gives way to things like portfolio review, interviews, and other "fuzzy" types of assessment. This is most appropriate for complex outcomes like analytical thinking or creative writing. Transparency, or the ability to turn results into effective action, depends much on the specificity of the outcome. In other words, reducing the complexity to something manageable. For example, a speech instructor giving advice to a student who has just delivered his first performance. Rubrics can help identify simplifications, but shouldn't be thought of as a panacea. It's not always true that the whole is the sum of the parts. For example, numerically summing up subjective ratings on a rubric to combine to some whole is really not defensible as science or epistemology. Continuous improvement models work in this situation; standards-based are less effective.
  • Formative and opaque These are the hardest nuts to crack. I'd say assessing 'critical thinking' falls into this category. I recommend rethinking the problem, and redefining one's mission if you find yourself unable to come to grips with a problem even subjectively. Neither standards-based nor continuous improvement are going to help you out here. Abandon all hope.

Stay tuned for part two. [update: Part two is here]