Selling Error

In the last few posts I've had fun with the idea that error can have economic value, and that people make money selling it. We probably all do it: we try to edge the odds in our favor when buying a used car (observe its defects to try to negotiate the price down), put new paint on the boat before trying to sell it (to get the price up), wear nice clothes to an interview, and so on. These actions don't affect reality, just its perception.

My main example was SAT coaching, but that horse has been thumped ad nauseum, so I'll give it a rest. Last night I remembered reading a remarkable passage out of Monkey Business by John Rolfe and Peter Troob, subtitled Swinging Through the Wall Street Jungle. John and Peter were new graduates of a fine business school who found out what it's like to be an investment banker. The book is chock-a-block with examples of estimator influence, starting with the gung-ho partner who comes to recruit them painting a picture of (almost) requited greed, luxury, and sophistication. The reality was different, and there's a particular dynamic that I find instructive. First, let them set the stage. The authors describe the competitive environment of investment banking, where banks try to arrange deals (and play middleman, keeping some of the cash torrent that proceeds):

[T]he advisory side of the business has become much more commoditized [than before]. The banker no longer has the lock on relationships. The banker's information is no longer highly proprietary. Information on companies is now so widespread that there's very little company-specific knowledge that bankers can truly call their own. The banker no longer brings enough unique added value to the table to necessarily merit a CEO's granting him a lifelong mandate to provide paid advice on matters of corporate finance.

This describes a shift from a trust relationship based on inside information to a commodity experience. The facts about a company that may be for sale are transparent, so it's up to the bank to find ways to add value to this for the potential buyer. You would think that in a free market what would result is better analysis as the competing investment banks all sharpened their pens to try to outdo each other, right? Wrong.

The real dynamic, according to the narrative, is driven by the loss of trust. In repeated interactions, you are more likely to reciprocate trust. In the "good old days" when a bank was hired to be a long-term advisor, the banker had a vested interest in the relationship continuing. This is not true if the business is one-off, especially when you consider that the people actually making the deal are mostly concerned with their own bonuses and even with the welfare of their own investment bank (my interpolation). Are you more likely to tip a server at a restaurant you visit frequently or one you'll never see again?

This creates a challenge for the banker. How to make my pitch unique? The actual payoff for the buyer is irrelevant, because once the deal is done I have already pocketed my percentage. I think it's obvious that this creates a demand for error: it's in the banker's best interest if the potential clients overestimate the value of the purchase because that increases the probability of the sale. The authors describe how this works when making the pitch. One of the many details described in the text is how to make a bank appear to be the best one for the job. I give excerpts below.

The Expertise section [of the pitch book provided to potential buyers] gives the bankers a chance to perpetrate one of their favorite deceptions--the spinning of the league tables.

The league tables are lists detailing how many deals, representing what total dollar amount, an investment bank has done in a given category. [...] The problem is that only one investment bank can truly be number one for any given type of deal, so the bankers have to get creative.

When spinning the league tables, a banker will steadily whittle down the universe of "appropriate" deals until [his or her] bank comes out number one. There are a million ways to do this. [...] With enough trips to the plastic surgeon, just about any bowlegged trollop can come out looking like a supermodel.

At the end of the day, when the league tables finally make it into the pitch book, the only evidence of subterfuge will e discreet. The heading of the league table will trumpet "Our bank is the top underwriter of IPO's for companies similar to Acme." The caveat, in the form of a footnote in barely legible type at the bottom of the page, will tell the whole story.*

*For IPOs between $50 and $150 million in the telecommunications sector, and excluding foreign issues, real estate investment trusts, and offerings done concurrently with debt issues.

The game is obviously to disguise the bank's true experience with deals of the sort in question. The economic value is perhaps an increased probability of sale. I wonder how effective this game actually is--it seems awfully transparent. The more general principle is that if there is economic value, and you can't affect the actual situation, then try to affect perception of it.

What does this have to do with outcomes assessment? There is a primary lesson to be drawn, I think: avoid competitive relationships in favor of building trust. This is especially true of the relationship between faculty and administration. For example, if outcomes assessments were to be used for assigning merit bonuses to faculty, it would create a competitive environment for the best ratings, giving economic value to the scores. This creates demand for error, which will surely be produced in buckets. This is an argument against high-stakes testing, but especially where the tests or other assessments are complex and easily manipulated.

What would trust-building look like? The obvious things: listen to the bottom-up conversation. Do the faculty actually believe in the assessments, or are they just playing along? Have you used the power of the accreditation bogeyman to ram through changes that almost guarantee the wholesale production of error? When a professor sits down to write an assessment report, what is his or her primary motivation? That's the key question. It would be interesting to do an anonymous survey...