Course Evaluation Ranks

This is a follow-up to "Online Evals." In the spring we changed from paper student evaluations (using ETS's SIR II) done in class to electronic online evaluations (also SIR II). Some of these were done in class, with students bringing in laptops. Most were not, although we don't have an exact count.

One of the concerns expressed by faculty beforehand was that the scores would change. The previous post shows that the distribution of course averaged definitely changed from one skewed to the positive end to a more symmetrical one. It occurred to me that perhaps the most important question from a faculty point of view is what happens to the relative rankings of instructors when this happens. To find out, we calculated the average absolute difference |paper rank - online rank| for each faculty who was evaluated both semesters, using the single item on the evaluation that's used for administrative purposes (yeah, I know....). There were 82 faculty in the sample, so the greatest possible change was 81 ranks. The average for paper-paper semesters was 17, and for paper-electronic was 20 or 24, depending on which semester we used. The standard deviations were around 1.8, but the distribution is not normal, as you can see on the histogram below. It shows absolute rank change.

Generally, paper-paper performs better, meaning that the ranks are more stable than paper-online.

Gresham's Law

Well, not exactly, but close enough (from wikipedia):

Gresham's law is commonly stated: "Bad money drives out good", but more accurately stated: "Bad money drives out good if their exchange rate is set by law.".
This law applies specifically when there are two forms of commodity money in circulation which are required by legal-tender laws to be accepted as having similar face values for economic transactions. The artificially overvalued money tends to drive an artificially undervalued money out of circulation ^[1] and is a consequence of price control.

It seems like it needn't be an exchange rate set by law, though, in other contexts: it's the "being accepted at face value" that matters. For example, if a fake degree is accepted just as much as a real one, then who'd bother to get a real one? (Well, lots of people, but you see my point.) Sound ridiculous? Maybe you missed the story about the Pakistan MP scandal, where many were exposed to have faked their credentials.  Here's the quote (from ABC News):

The chief minister of Balochistan Province, Nawab Muhammad Aslam Raisani, told the press: "A degree is a degree, whether it is fake or genuine."

 What else is accepted at face value? Most proxies are, by most people. We generally don't really see what we look at (see Darwin's theory of sexual selection, for example). Here's a quote new to me by Diane Ravitch, now rather infamous for switching on No Child Left Behind. From the Philadelphia Inquirer:

[...] Ravitch notes, we find ourselves quite possibly on the way toward "a paradoxical and terrible outcome: higher test scores and worse education."

In other words, substitute test scores for authentic outcomes, and maximize to absurdity. Here's an opinion piece in Ed Week that describes the result: inflation of scores by any means possible. Here's the "debasing the currency" part:

Over the past several years, efforts to "hold teachers accountable" and "hold schools accountable" have produced perverse consequences. Instead of better education, we are getting cheating scandals, teaching to bad tests, a narrowed curriculum, lowered standards, and gaming of the system. Even if it produces higher test scores (of dubious validity), high-stakes accountability does not produce better education.

It seems like the policy makers have never heard of Murphy's Law either.

Publishing Learning Outcomes

The National Institute for Learning Outcomes Assessment, one of George Kuh's projects (at least, he's the director) has a new report "Exploring the Landscape: What Institutional Websites Reveal About Student Learning Outcomes Assessment Activities" (pdf) that reports on a survey of web sites. This compares what schools reported about what they were doing via a NILOA survey to what is actually found on their website. The findings are not shocking to anyone who's ever gone looking for learning outcomes data (as opposed to plans, which are plentiful). See my post "404: Learning Outcomes," for example.

Judging from the web, the report says, there's a lot going on that's invisible to the public. Also, learning outcomes seems to be driven by regional accreditors. Not surprising. One of the interesting bits is the detail of what institutions are publicizing as their outcomes assessments. The graph below is taken from page 14 of the he report:

This graph is not about who is doing what so much as who is bragging about doing it. So privates are advertising capstones more than publics. Unfortunately, the capstone experience is not an item surveyed on the previous "what are you doing" survey from NILOA cited earlier. The closest comparison I could find is the following graph from page 11 of that report.

The mismatch may be that capstone experiences by themselves aren't really assessments in the usual meaning. It's interesting to note that the for-profits report doing more assessment than other types almost across the board, with the notable exception of national surveys. This may be because national surveys are expensive, or it may be that the for-profits don't want to invite comparisons to national benchmarks, or it may just be that they are deemed inappropriate to what are mostly online programs. This graph does show that the for-profits can play the "assessment game" better than traditional institutions. They have more control and more money. (By "game" I mean satisfying the the external requirements).

Conclusion of the report:

Institutions have more student learning outcomes assessment activities underway than they report on their websites. To meet transparency obligations and responsibilities, institutions should make more information about student and institutional performance accessible via their websites. 

I'm not quite sure what transparency obligations there are. I'm unaware of any from our regional accreditor. Perhaps this just picks up language from the Spellings Report. I do think institutions miss a trick in using learning outcomes in marketing programs, although the privates seem to be talking about capstone courses. Providing comparable and meaningful information that one can put in a spreadsheet and compare colleges is probably a pipe dream, if that's what's meant by transparency.

So much of a college experience is singular, random, and individual. I am on vacation back in Illinois this week, and arranged to have lunch with my math advisor and eventual collaborator on a software project. My daughter was at the lunch too, so we bored her with the story of my first encounter with the guy who would change my life. I mean this literally: without his encouragement I would never have gotten a doctorate in math.

I was seriously into computers at the time, banging out artificial life simulations and space simulators on an Apple II, and was attracted to MAT 475 Numerical Analysis as a junior at SIU. The undergrad advisor I was assigned to told me I'd have to get permission of the instructor, so I went up to the fourth floor with some trepidation.

Me: I'd like to be in the Numerical Analysis class.
He: It'll be bloody hard.
Me: Okay.

He wasn't kidding. I think I had a 50% average going into the final, but was having too much fun to quit. I didn't realize that 50% was a solid B in his class until later. That class led to a long friendship and collaboration that has now lasted decades. But it was random, not the sort of thing that is ever going to be revealed with mass "transparency" reports. This is not to say transparency is a bad thing, but like standardized testing, it's not a substitute for the real thing (the actual experience of an individual student), which is probably not remotely predictable.

Online Evaluations

In the spring we took the plunge and converted our end-of-term student evaluations from paper to electronic. We used the identical form otherwise--that was the only change. Looking at the results, it's obvious that the distribution of scores changed dramatically. The graphs below show the histograms for section averages. The number of sections is around 250 in each.

The bottom two are from paper distribution. The top one is the electronic, and shows a bell-shaped curve with a big spike in the middle. I assume that there are a lot of students who just bubble in 4s all the way down the sheet. That would explain the middle of the graph. But why would the skew in the paper version practically vanish online?