I think there is an impulse, especially in the glaring light of an impending accreditation, to create an illusion of science by making nice forms and boxes into which one may pour one's classroom or extra-curricular observations. Each of these gets its own rubric, and the theory is that these micro-observations add up to something meaningful in the end. Maybe. But it's not nearly as easy as it sounds. At a minimum, the assessment organizer has to:
- Get faculty to construct rubrics they believe in. (Imposing them from the outside isn't recommended for obvious reasons.) This usually entails sub-dividing learning objectives into "dimensions" that apply. For example, maybe making eye contact is a component of effective speaking. Again, the faculty who are going to use the rubric ought to be in charge of it. Then, describing the Likert-scale-like rating system so that there is reasonable inter-rater reliability should be done.
- Test the rubrics for inter-rater reliability. No point in going to all this work unless you know the thing performs as expected. It won't, of course, and if you're serious about it, you need to "center" the raters by getting them to agree to agree.
- Establish assignments that are to be used for assessment, since using different ones lowers reliability.
- Build the systems, policies, and information technology to allow this process to be organized.
- Motivate the faculty to actually enter the ratings.
- Produce meaningful reports.
- Motivate faculty to look at the reports and make changes accordingly.
- Quantity. Simply count the number of times an activity is linked to a learning objective.
- Quality. Faculty discuss the student work linked to the learning objective. This is done in a special meeting of the program, department, or course instructors, and has some minimal structure to it based on a standard IE model (what did we want to do, what did we observe, what can we conclude, what shall we do about it).
Frankly, I think it's still to complicated because our information systems are too complex. I'd prefer a simple-as-possible drop box solution, but we have a learning management system that's reasonably well established, and after discussion with the faculty, I think it's best to stick with it rather than creating something new.
I was interested to see a post on the ASSESS listserv that mirrors this line of thought. Linda Suskie (now at Middle States Commission on Higher Ed.) describes three options for rating evidence of learning, which she defines as "tangible, visible, self-explanatory, and compelling evidence of exactly what students have and have not learned." The three approaches she describes could be distilled as:
- Rubrics as I have described them, more or less. She leaves out the details.
- Observers writing impressions qualitatively for learning objectives rather than using a rubric (this is probably more work than a rubric, but might be more natural especially for complex objectives).
- The third is similar to what I've described: "A third option would be to have the discussion, but have all discussions focus consistently on how well the presentation demonstrates achievement of the assignment's key learning outcomes and have notes of the discussion recorded as direct but qualitative evidence of student learning. This would be sort of like completing just one structured observation guide for all observers, rather than have each observer complete one separately."