The 2009 Assessment Institute

I'm just back from two full days in Indianapolis at the Assessment Institute. The trip was great. I made contacts and learned some really interesting stuff. Too much to summarize this morning, but here are some bits and pieces to be filled in later.

The opening plenary was shortened by an unfortunate medical incident, but hinted at some interesting fault lines. Trudy Banta, the organizer, does a good job of representing different points of view with her choice of panel, guest, and topic.

The first divergence I noticed centered on the idea of "tuning," a concept borrowed from the Bologna Club in the EU, which I blogged about here and here (the update at the bottom). It's not a lunch meeting, as you might think, but a process of comparing and improving programs across institutions. As a point of trivia, we learned from Jeffery Sybert that concert A is 440 Hz in this country, but 442 Hz in Europe. I'm quite sure I couldn't tell the difference, which is in any case greater than the intra-tuning dissonances of an equi-tempered scale. There must be an interesting story there. The topic was presented by Jamie Merisotis, President/CEO of Lumina Foundation for Education. You can read more about the project here. The panel dissonance in this case was fairly minor, turning on the question of how much authority faculty should have in the process. On the one hand, faculty own curriculum. On the other, they can be protective, self-interested, and unduly academic in their horizons (speaking as one). This is a good case of needing external reviewers to constantly check that goals and progress align. (Such as a stakeholder analysis: see below.)

There was the admonition from the panel that for-profits and their venture capitalist underwriters are very interested in producing a meaningful educational product, and by implication that the usual plodding change of bricks and mortarboard institutions won't be fast enough to compete with cyberdon. As a riff on that, George Kuh suggested that "the credit-hour is a dying concept in higher education, measuring things we no longer value." He elaborated, equating seat time with credit hours under the traditional system.

The other twanging note in the plenary was caused by the use of the term "critical thinking." The consensus position is that this skill is valuable, should be taught and "measured," etc. In opposition to that is the idea that maybe critical thinking is too fuzzy to actually be useful as a learning objective. This turned out to be a minor theme at the conference, which I'll elaborate on tomorrow. In a Q&A at one session, I was given an admonishing mini-lecture that psychology had solved the critical thinking problem and that I only needed to look at the literature. In a session with University of Phoenix that I missed, I'm told that there was rude and contentious debate about assessing critical thinking. Maybe someone who was there can comment.

I met the creator of Waypoint, which I had blogged about here. I watched a demo of the online rubric management and implementation software. More on that later.

At lunch on Monday, Jon and I chatted with an Assistant Director of CIRP at HERI [edit: fixed title] who works on CIRP. I've used the freshman survey, and found it quite useful in finding attrition trends (see this post), and I was interested to learn that CIRP is getting into the constructs business, using item response theory. I went to the session on that on Tuesday, and have some other comments that will have to wait until I have more time.

A session by Tom Zane at Western Governors University gave a fascinating insight into their system of assessing massive amounts of student work. You can read about their innovative system of using assessments to entirely replace grades here. They get between 1500 and 2000 new students each month, and are still growing rapidly. Student work is assessed by human raters using rubrics--this isn't standardized test land. Samples are rated more than once to test for reliability. Although this is still monological, my first impression is that this is as good as it gets for traditional assessment, and that the assessment side of their business could plug into this model if WGU chose do to it--providing a uniform system of credentialing for higher ed. Something like that is conceivably in our future. That might sound scary, but it's infinitely better than standardized tests running our lives.

AAC&U has a project called VALUE, which accumulates rubrics that they have found or created and refined. This seems useful, and complementary to the tuning idea. The next part of the project is to create a repository of student work that has been rated using the rubrics. A problem I noticed more than once is that designers usually don't seem to think much up front about whether their rubric is relative to the curriculum or fixed absolutely. When we built the FACS model, we used an absolute scale, in which raters say things like "student Ecks is working at the freshman/sophomore level." In a relative scale you get stuff like "exceeds expectations." The former is great for tracking longitudinal progress, the latter not so much. A good student will exceed expectations in all classes, showing no progress. By contrast, even a great freshman math major is very unlikely to be doing senior-level work. I'm not sure the VALUE leaders have addressed this.

Jon Shannon and I led a 75-minute session on stakeholder analysis in strategic planning, which had good participation. I blogged about the topic here, and you can find the presentation linked here. This is a great tool for addressing complex planning issues. One of the advantages is that it keeps the conversation on track, focused on goals everyone more or less agrees on.

The backchannel on twitter was pretty thin, or else I just didn't hit the main vein with my search. We set one up at Today's Meet and advertised it on one of our slides, but haven't gotten any activity from participants.

More on some of these topics later.

Prezi for Presentations

Now that PowerPoint presentations have reached their peak in creative expression with the brilliant summit of art embedded below, it's time to move on to other media. In the era of Web Duh Punt Null, new media arrive shiny and gleaming in your in-box about every 10 seconds anyway.


The new kid on the block is Prezi, a Flash-based composition and presentation package that you can use for free, or upgrade to paid packages for more features. In practice, you'd want to upgrade because otherwise everything you create is instantly public. Either way, you can either use an online navigation page or download your stuff along with a desktop presentor to put on a USB drive. If you want to be able to compose offline, however, you have to buy the expensive package. I tried Prezi out last weekend in preparation for a workshop I'm co-presenting at the 2009 Assessment Institute later this month in Indianapolis. It's a great conference for both the presentations and making contacts. It's a very different feel from the SACS meetings (regional accreditor), where you can judge how close someone is to the visit by how much they sweat and the degree to which they jump at sudden movements. By comparison, the Assessment Institute is relaxed and intellectual.

It's easier to show you how Prezi works than to tell you, so I've embedded the draft I've been working on below. You can use the arrows at the bottom to follow the flow, or freely click around, zoom and drag.



The same presentation is here on the Prezi site. The topic of the workshop is strategic planning, and particularly a nice method I learned from Jon Shannon on doing an inventory of stakeholders, their goals, and possible solutions to tactical or strategic challenges. I blogged about that topic in A "Useful Planning Technique." The session is at 10:15 on Tuesday, October 27, and here's the full description:

Stakeholder-Oriented Assessment and Planning
Stakeholder-Oriented Assessment and Planning (SOAP) is an approach to assessment and planning that facilitates efficient discussions and promotes confidence and early alignment of participating groups. By identifying the stakeholder-goal landscape relevant to an assessment or planning target, participants quickly reveal and document the existing and desired state of the system.