404: Learning Outcomes

tl;dr Searched SACS reports for learning outcomes. Table of links, general observations, proposal to create a consortium to make public these reports.

In grad school there was a horror story that circulated about a friend of a friend of a cousin, who was a math grad student in algebra. He had created a beautiful theory with wonderful results, and was ready to submit when it was pointed out to him that his axioms were inconsistent--they contradicted one another. The punchline is that you can prove anything of the empty set. This sometimes also happens to degree programs that suddenly have to prove that they've been doing assessment loops, except in reverse: building grand theories from the empty set.

I complained the other day the there weren't many completed learning outcomes reports from universities to be found on the web. So when I noticed Mary Bold's post at Higher Ed Assessment "Reporting Assessment Results (Well): Pairing UNLV and OERL" I thought I'd hit paydirt. The hyperlink took me to a page at University of Nevada, Las Vegas with a link advertising "Student Learning Outcomes by College." Without futher ado, here's the page:

That's just too funny. There are, however, excerpts from actual results listed in the teacher education site, which you can find here. That site is the OERL that Mary refers to in her post.

It did make me think, however. There must be a bunch of SACS compliance certifications out there on the web now, and section 3.3.1 (used to be 3.4.1) covers learning outcomes. Want to see how your peers have handled it? The name of the school in the table below links to the compliance certification home page for that institution. For good measure I'll throw in 3.5.1, general education assessment, too. You're welcome.

Institution	3.3.1	3.5.1
Southeastern Louisiana University	link	link
Western Kentucky University	link	link
Berea College	link	link
The College of William and Mary	link	link
The University of Alabama in Huntsville	link	link
Nashville State Community College	link	link
Mitchell Community College	link	link
The University of Texas Arlington	link	link
University of New Orleans	link	link
Albany State University	link	link
Bevill State Community College	link	link
Louisiana State U. In Shreveport	link	link
Texas Tech University	link	link
Coker College	link	link

I did not try to make a complete list of all available reports. If you find a good one, send me the url and I'll add it. Here's my google search criterion.

Disclosure: I was the liaison, IE chair, webmaster, and editor for the Coker College process (as well as doing the IR and a bunch of other stuff--no wonder I have white hair). The linked documents to that one are turned off for confidentiality, but you can find the complete list of program learning outcomes plans and results here.

Observations:
First, hats off to all the institutions who make these reports public. This is a great resource to anyone else going through the process.

I only scanned through the reports, looking for evidence of learning outcomes. I probably missed a lot, so take my remarks with a grain of salt--go look for yourself and leave a comment if you find something interesting. It should go without saying that in order to be helpful, this has to be a constructive dialogue.

For learning outcomes I didn't find as much evidence-based action as I would have expected from all the emphasis that SACS puts on it. My own experience was that programs were uneven in their application of what is now 3.3.1 (at the time SACS didn't even have section numbers for the requirements--how crazy is that? I invented my own, and then they published official ones just before we had to turn the thing in.). So there was a lot of taking the empty set and trying to build something out of it. That can take various forms, which one notices in scanning certification reports:

• Quick fixes: use a standardized instrument like MAAP, MFAT, CLA, NSSE. Of course, it's not really that quick since it would take at least a year to get results, analyze them, and use them. The conceptual problem is tying results to the curriculum (except for MFAT).
  
  
• Use coursework: passing a certain course certifies students in X (e.g. use of technology), passing a capstone course with grade X ensures broad learning outcomes. This is fairly convincing as gate keeping, but hard to link to changes unless specific learning outcomes are assessed.
  
  
• Rubric-like reporting. Okay, I'm not a big fan of rubrics when employed with religious zeal and utter faith in their validity. But I have to admit that the most convincing report summary I saw on learning outcomes was the one below from Mitchell Community College. Not all the data points are there, but that's realistic. Take a look.
  

Of course, this still has to be tied to some analysis and action to get full points, but the presentation of the learning outcomes is clear and understandable. In general, that was somewhat of a rarity in my cursory review. What there is a LOT of is plans, new plans, minutes describing the construction of new plans and goals, assessment forms, models and processes, and generally new ambitions and enthusiasms. There are standardized test reports like CLA summaries, which solve the data and report problem, but don't touch the hard part: relating it to the practice of teaching in a discipline.

I believe that if our efforts as assessment leaders are to be maximally useful, we have to make the annual, messy, incomplete, inconsistent, but authentic program-level plans and results available to the public. This would encourage us to adopt some kind of uniformity in reporting, and improve the quality of the presentation (maybe I'm a fool for saying that). The only downside is that if we're honest, there will be empty sets here and there--programs that have not been dragged into the 21st century yet. But transparency can help there too, perhaps into shaming some into compliance. Just imagine (really dreaming now) if the quality of the reports were good enough to use for recruiting and paint across the program web page.

The Voluntary System of Accountability tries to do something like that. Unfortunately, that group seems to be enamored of standardized tests for learning outcomes. There's a validity study they just published here that you can consider. This post isn't the place to go into all the reasons I think standardized testing is the wrong approach, so let me just leave it at that.

Thinking more positively: is there any interest out there to form a loose consortium of schools that report out annual learning outcomes for programs? The role of the consortium could be to settle on some standard ways of reporting and defining best practices?