Showing posts with label outcomes assessment. Show all posts
Showing posts with label outcomes assessment. Show all posts

Wednesday, September 02, 2009

Selling Error

In the last few posts I've had fun with the idea that error can have economic value, and that people make money selling it. We probably all do it: we try to edge the odds in our favor when buying a used car (observe its defects to try to negotiate the price down), put new paint on the boat before trying to sell it (to get the price up), wear nice clothes to an interview, and so on. These actions don't affect reality, just its perception.

My main example was SAT coaching, but that horse has been thumped ad nauseum, so I'll give it a rest. Last night I remembered reading a remarkable passage out of Monkey Business by John Rolfe and Peter Troob, subtitled Swinging Through the Wall Street Jungle. John and Peter were new graduates of a fine business school who found out what it's like to be an investment banker. The book is chock-a-block with examples of estimator influence, starting with the gung-ho partner who comes to recruit them painting a picture of (almost) requited greed, luxury, and sophistication. The reality was different, and there's a particular dynamic that I find instructive. First, let them set the stage. The authors describe the competitive environment of investment banking, where banks try to arrange deals (and play middleman, keeping some of the cash torrent that proceeds):
[T]he advisory side of the business has become much more commoditized [than before]. The banker no longer has the lock on relationships. The banker's information is no longer highly proprietary. Information on companies is now so widespread that there's very little company-specific knowledge that bankers can truly call their own. The banker no longer brings enough unique added value to the table to necessarily merit a CEO's granting him a lifelong mandate to provide paid advice on matters of corporate finance.
This describes a shift from a trust relationship based on inside information to a commodity experience. The facts about a company that may be for sale are transparent, so it's up to the bank to find ways to add value to this for the potential buyer. You would think that in a free market what would result is better analysis as the competing investment banks all sharpened their pens to try to outdo each other, right? Wrong.

The real dynamic, according to the narrative, is driven by the loss of trust. In repeated interactions, you are more likely to reciprocate trust. In the "good old days" when a bank was hired to be a long-term advisor, the banker had a vested interest in the relationship continuing. This is not true if the business is one-off, especially when you consider that the people actually making the deal are mostly concerned with their own bonuses and even with the welfare of their own investment bank (my interpolation). Are you more likely to tip a server at a restaurant you visit frequently or one you'll never see again?

This creates a challenge for the banker. How to make my pitch unique? The actual payoff for the buyer is irrelevant, because once the deal is done I have already pocketed my percentage. I think it's obvious that this creates a demand for error: it's in the banker's best interest if the potential clients overestimate the value of the purchase because that increases the probability of the sale. The authors describe how this works when making the pitch. One of the many details described in the text is how to make a bank appear to be the best one for the job. I give excerpts below.
The Expertise section [of the pitch book provided to potential buyers] gives the bankers a chance to perpetrate one of their favorite deceptions--the spinning of the league tables.

The league tables are lists detailing how many deals, representing what total dollar amount, an investment bank has done in a given category. [...] The problem is that only one investment bank can truly be number one for any given type of deal, so the bankers have to get creative.

When spinning the league tables, a banker will steadily whittle down the universe of "appropriate" deals until [his or her] bank comes out number one. There are a million ways to do this. [...] With enough trips to the plastic surgeon, just about any bowlegged trollop can come out looking like a supermodel.

At the end of the day, when the league tables finally make it into the pitch book, the only evidence of subterfuge will e discreet. The heading of the league table will trumpet "Our bank is the top underwriter of IPO's for companies similar to Acme." The caveat, in the form of a footnote in barely legible type at the bottom of the page, will tell the whole story.*

*For IPOs between $50 and $150 million in the telecommunications sector, and excluding foreign issues, real estate investment trusts, and offerings done concurrently with debt issues.
The game is obviously to disguise the bank's true experience with deals of the sort in question. The economic value is perhaps an increased probability of sale. I wonder how effective this game actually is--it seems awfully transparent. The more general principle is that if there is economic value, and you can't affect the actual situation, then try to affect perception of it.

What does this have to do with outcomes assessment? There is a primary lesson to be drawn, I think: avoid competitive relationships in favor of building trust. This is especially true of the relationship between faculty and administration. For example, if outcomes assessments were to be used for assigning merit bonuses to faculty, it would create a competitive environment for the best ratings, giving economic value to the scores. This creates demand for error, which will surely be produced in buckets. This is an argument against high-stakes testing, but especially where the tests or other assessments are complex and easily manipulated.

What would trust-building look like? The obvious things: listen to the bottom-up conversation. Do the faculty actually believe in the assessments, or are they just playing along? Have you used the power of the accreditation bogeyman to ram through changes that almost guarantee the wholesale production of error? When a professor sits down to write an assessment report, what is his or her primary motivation? That's the key question. It would be interesting to do an anonymous survey...

Thursday, April 02, 2009

Why Documenting Learning Outcomes is Hard, Part One of Infinity

Pat Williams recently asked if anyone is closing the proverbial loop on her blog. Quote:

For almost all of us the question, "What are you going to do about it?" has proven the toughest to answer.
If you've ever written assessment reports yourself, or had to judge ones others have written, you'll probably find it hard to disagree with this statement. When I first confronted the task of documenting those for our SACS report, it became obvious that there were some institutional disconnects at the accreditation level. For example, if you look back at the old SACS general education requirement (3.5.1), it was written as a minimum standards policy (you must set standards and show that students meet them), but in practice, every single person I talked to at the SACS conferences used a continuing improvement philosophy to actually judge compliance! This included IR people who'd been on many campus visits as well as senior vice presidents of the Association speaking in formal presentations. I found this bizarre and almost Kafkaesque. I pointed this out in emails when SACS invited comment. I don't know how my suggestions were received, but the standard was changed.

The point here isn't really about SACS. It's about the confusion surrounding standards-based vs. continuing improvement models. Public schools are primarily standards-based in their approach. Results are determined by standardized tests. If the learning objectives are simple enough (in the formal sense of complexity, which I've written about here many times), this approach should work well enough. This is because the items on the test can correspond pretty precisely to the material being taught. For example, if we want to know if third graders know their multiplication tables, this is relatively straightforward.

Things go wrong when the assessment doesn't align with the actual goals. This is a question of validity of the test, but it often goes unnoticed. By believing too much in the wrong assessment, we can create what I call degenerate assessment loops (more here). You can read a nice parable about this in The Well Intentioned Commissar.

Continuous improvement is orthogonal to standards. If one mixes up standards-based assessment and continuous improvement, confusion quickly sets in. Even in the SACS 3.3.1 standard of old, there was ambiguous language (I complained about that here, and it changed too, but it's probably coincidence). The problem is that many people assume that you should be able to show continuous improvement in some particular metric. This is virtually impossible to do once learning outcomes have reached a certain level of complexity.

We think of many assessment situations as falling into one of these categories:
  • Summative and transparent An example of this is low-complexity tasks like learning multiplication tables. It's easy to judge how things are going, and relatively easy to effect change (that's the transparent part). One can apply standards here effectively.
  • Summative and Opaque Here it's easy to see where you are, but hard to know how to get where you want to go. An example of this is the stock market. In learning outcomes assessment, this category applies to standardized tests of complex behaviours, which necessarily reduce the complexity (and hence validity). It's easy to see if the numbers go up or down, but hard to know how to affect them (without, say, directly teaching students how to take the test). Standards can be applied here, but it's very hard to use them for accountability, since there's no guaranteed way to reach new heights of achievement.
  • Formative and transparent In this case, subjectivity is valued more than objectivity, and standardization gives way to things like portfolio review, interviews, and other "fuzzy" types of assessment. This is most appropriate for complex outcomes like analytical thinking or creative writing. Transparency, or the ability to turn results into effective action, depends much on the specificity of the outcome. In other words, reducing the complexity to something manageable. For example, a speech instructor giving advice to a student who has just delivered his first performance. Rubrics can help identify simplifications, but shouldn't be thought of as a panacea. It's not always true that the whole is the sum of the parts. For example, numerically summing up subjective ratings on a rubric to combine to some whole is really not defensible as science or epistemology. Continuous improvement models work in this situation; standards-based are less effective.
  • Formative and opaque These are the hardest nuts to crack. I'd say assessing 'critical thinking' falls into this category. I recommend rethinking the problem, and redefining one's mission if you find yourself unable to come to grips with a problem even subjectively. Neither standards-based nor continuous improvement are going to help you out here. Abandon all hope.

Stay tuned for part two. [update: Part two is here]