Unintended Consequences

My post a couple of days ago on proxies shows some examples of unintended consequences when you put artificial conditions on performance. Sometimes it involves cheating, like the teachers 'tubing' tests. But sometimes emphasis on a proxy ironically brings about the opposite of what you'd expect. I explored that idea in "The irony of good intentions" a while back. Since the World Cup is on, here's a crazy football (soccer) example. The basic expectation in a football match is that teams try to score in their opponent's goal, NOT their own. Scoring an own-goal is a sure way to make the fans grumpy (or murderous in one tragic case).

The 1994 Shell Caribbean Cup had the odd twist that goals scored in overtime counted double for purposes of the tournament. I'm not sure what the motivation for that was, but it had an unexpected consequence. Here's the wiki description of the Grenada vs Barbados game:

Grenada went into the match with a superior goal difference, meaning that Barbados needed to win by two goals to progress to the finals. The trouble was caused by two things. First, unlike most group stages in football competitions, the organizers had deemed that all games must have a winner. All games drawn over 90 minutes would go to sudden death extra time. Secondly and most importantly, there was an unusual rule which stated that in the event of a game going to sudden death extra time the goal would count double, meaning that the winner would be awarded a two goal victory.

Barbados was leading 2-0 until the 83rd minute, when Grenada scored, making it 2-1. Approaching the dying moments, the Barbadians realized they had no chance of scoring past Grenada's mass defense, so they deliberately scored an own goal to tie the game at 2-2. This would send the game into extra time and give them another half hour to break down the defense. The Grenadians realized what was happening and attempted to score an own goal as well, which would put Barbados back in front by one goal and would eliminate Barbados from the competition.

However, the Barbados players started defending their opposition's goal to prevent them from doing this, and during the game's last five minutes, the fans were treated to the incredible sight of Grenada trying to score in either goal. Barbados also defended both ends of the pitch, and held off Grenada for the final five minutes, sending the game into extra time. In extra time, Barbados notched the game-winner, and, according to the rules, was awarded a 4-2 victory, which put them through to the next round.

 Here's a grainy pre-HD video exerpt.

Testing and Choice and Ruin

Diane Ravitch was an assistant secretary of Education during the latter Bush administration.  She has done an about-face on No Child Left Behind, and written a book about it called The Death and Life of the Great American School System: How Testing and Choice Are Undermining Education.  In a May 25th article in Education Week, Ravitch summarizes what she things is wrong with Race to the Top, the Obama administrations version of NCLB.  Among her points are two game theory arguments:

The NCLB-induced obsession with testing and test-prep activities will intensify under Race to the Top because teachers will know that their future, their reputation, and their livelihood depend on getting the scores higher, by any means necessary.

By raising the stakes for tests even higher, Race to the Top will predictably produce more teaching to bad tests, more narrowing of the curriculum, more cheating, and more gaming the system. If scores rise, it will be the illusion of progress, rather than better education. By ratcheting up the consequences of test scores, education will be corrupted and cheapened. There will be even less time for history, geography, civics, foreign languages, literature, and other important subjects.

I wrote about this sort of effect in "The Irony of Good Intentions," and it's also related to the over-valuing of SAT scores in "Amplification Amplification."  There must be some psychological flaw in our species that leads us to accept easy answers over ones that work, often taking the form of some sort of credential.  For example:

• Bond ratings substitute for real understanding of a company
• Investment firm stock ratings do the same
• Standardized test scores of complex behavior substitute for actual performance
• Degrees and other credentials substitute for demonstrated performance
• Paper currency stands in for something (what?) of value
• Loan applications substitute for actual ability to repay

All of these are undoubtedly useful, but are also subject to bubbles.  We're in a credential bubble now, with online for-profits cranking out degrees, racking up massive loan debts (for students), and milking the government of aid dollars. 

Probably someone has done this already, but this inflation-of-value idea probably has an archeology we can discover in word origins. For example, the word "really" used to mean "actually," according to this etymology site.

really early 15c., originally in reference to the presence of Christ in the Eucharist. Sense of "actually" is from early 15c. Purely emphatic use dates from c.1600; interrogative use (oh, really?) is first recorded 1815.

Now, really means something like "very."  The word "literal" has a precise meaning that describes a certain realness.  I have noticed that's becoming fashionable to use it merely as a false credential: on the radio I heard "and the audience was literally eating out of her hand."  Somehow I doubt that's true.

Similarly, words for "large" seem to have engorged themselves.  Things become Super, and Mega.  It can't be long before the advertising executives discover the untapped vein of other standard scientific prefixes: giga, tera, peta, exa, etc. 

Would you like fries with that?
Sure.
Do you want the Peta-pack or the Exa-plosion?

This arms race of "believe me!" may ultimately inflate so much that we require exponential notation to express how certain we are.  This may be another sign of Ray Kurzweil's Singularity:

Wow, that concert was REALLY10^25 good!
Oh yeah?  Well I saw one last week that was REALLY10^30 good!

And we'll probably be paying those teachers with these:

(image courtesy of Wikipedia)

The Irony of Good Intentions

Okay, I'll admit it: Kung Fu Panda is one of my new favorite movies. We must have rented the thing five times (for a buck each from the Redbox) before I broke down and bought a used copy. The best scene is the "noodles; don't noodles" monologue, but the self-defeating attempt to prevent disaster prefaced by "one often meets his destiny on the road he takes to avoid it" line is pertinent to more than just incarcerated cats.

Viz: Suppose, to present an archetype, one really, really wanted to achieve some thing. I'll call it S, just to show that math types don't always pick X.

In order to achieve S, we set up some kind of machinery or processes, we take actions and lay plans--sometimes even in the other order. At the end of the day we very thoroughly ensure--because of the very attention we are paying to the problem--that the opposite of S is achieved. To be pedantic, we'll call the opposite ~S, and a(*) the "attempt to achieve" function, so:

a(S) => ~S

The fact that we have a word devoted to this idea (irony) shows that it's not uncommon. Maybe it's also because we tend to find it funny if the tax auditor gets audited or the plumber's sink leaks. It's not so funny in other circumstances. Often, unintended consequences are the cause of mischief. A couple of examples:

• Kudzu: (source: Wikipedia) "From 1935 to the early 1950s, the Soil Conservation Service encouraged farmers in the Southeastern United States to plant kudzu to reduce soil erosion as above. The Civilian Conservation Corps planted it widely for many years.

  It was subsequently discovered that the Southeastern US has near-perfect conditions for kudzu to grow out of control — hot, humid summers, frequent rainfall, temperate winters with few hard freezes (kudzu cannot tolerate low freezing temperatures that bring the frost line down through its entire root system, a rare occurrence in this region), and no natural predators. As such, the once-promoted plant was named a pest weed by the United States Department of Agriculture in 1953."

• A programming shop wanted programmers to pay more attention to fixing bugs, so they gave rewards for each bug found. The number of bugs found rose quickly, but the paying out of lots of reward money didn't make the code get any better. (Programmers can create bugs even more easily than fix them). [I read this a looong time ago in a programming magazine.]
  
  
• A state began taxing trucks based on the number of axles it had, as a fair way of generating revenue to fix roads: bigger trucks pay more. This caused shippers to use trucks with fewer axles, putting more pressure on each tire, which caused the roads to degrade faster. [I read this in The Economist, I think, several years ago.]
  

I believe that we can create a simple recipe for creating irony. It goes like this:

1. Find some complex S
2. Work really hard at theorizing about S and simplify it to S'
3. Solve S' using the low-complexity model you built
4. Eventually realize that S' isn't S
5. Further conclude that a(S') => ~S
6. (optional) change your name and move to Alaska

We can create an endless supply of B-grade Hollywood scripts using this formula. To wit:

Dr. Zeus thinks he can solve a flu pandemic by releasing a custom-built virus that inoculates the populate via a common cold vector. But it mutates into a deadly scourge instead.

Okay, that was fairly depressing. Moving on to higher education and assessment...

From the 2008 publication "New Leadership for Student Learning and Accountability" from CHEA/AAC&U:

Each college and university should gather evidence about how well students in various programs are achieving learning goals across the curriculum and about the ability of its graduates to succeed in a challenging and rapidly changing world. The evidence gathered through this process should be used by each institution and its faculty to develop coherent, effective strategies for educational improvement.

This is a desire that has been amplified by the Department of Education as well as regional accreditors. What is described in the quote above is S. The next inevitable stop is S'. Quoting again from the same document:

Accrediting organizations should likewise evaluate institutions by their performance in accord with institutional goals and develop consistent strategies for summarizing and making public their findings.

I'm sure you noticed the switch there from learning outcomes to summarized findings. We humans have a big thing for simplifying stuff that's just too messy to deal with on its own terms. Why have five hundred numbers when one will do? So: simplify and standardize. That's how we create manageable, nominally comparable data, and theorize our way down to S'.

There are high expectations for S. For liberal arts schools, S includes stuff like higher order thinking skills, effective communications, maybe some teamwork and global understanding (AAC&U LEAP) and so forth. This is rather inconvenient, however, for the mass production of sterilized metrics. Complex and standardized are like fish and bicycle, to abuse an expression. The only way to reduce complex data down to a few bytes is to squeeze all the complexity out of it. You can put it in a factor analysis Squish-o-Matic and crank out a principle component, or cook it down to a bland grand average of goop, or apply a fancy economic model and juice it into a parameter with a Greek name. It's all the same. It's the difference between a vine-ripe tomato you take from your garden and one of those pink pasty things from the grocery aisle that look like they came out of a sci-fi movie.

So if S is a well-rounded graduate, S' is a square one. They both have approximately the same area, so we can fool ourselves if we squint. Still, it could turn out right, couldn't it? If we try really hard to apply a(S'), shouldn't we get close to S?

Really, I've been unfair. For most of the things we do in life, we approximate our goal with something simpler and try to reach it. If we spent all our time on the details, we'd never get our taxes done. Simplification is necessary in order to make tasks doable. For lots of cases, a(S') gets you close enough to S. Not everything is ironic.

So the serious question is: can we standardize stuff like critical thinking and creative writing and so on, without losing the essence of the thing? To me, the answer is obviously not. The whole point of complex thinking is the complexity. I've argued ad nauseum on that topic before, and I won't repeat those arguments this morning, but let's say that the standard of proof ought to be very high. Exceptional claims merit exceptional evidence, as the saying goes.

Worst case is that the data-compressed S' is a grotesque caricature of S, and that by amplifying S' through our efforts, we produce only shadows of what it is we're really after. In detail, here it is:

1. We want to create broad-minded critical thinking graduates, our S
   
2. It's hard to define S, let alone assess it, so there's pressure to substitute an assessment of S'--the ability to take a standardized test. It has some face validity, so the top administrators are convinced.
   
3. Having settled on S', we work really hard to train students for the content of the test so we'll look good to the public and the accreditors, a(S')
   
4. At the end, we've drilled our students on the importance of test-taking at the expense of improving S.

That would be ironic and sad.

Here's another possible example. In yesterday's InsideHigherEd "Wannabe U" author Gaye Tuchman is quoted as saying:

Universities are no longer to lead the minds of students to grasp truth; to grapple with intellectual possibilities; to appreciate the best in art, music, and other forms of culture; and to work toward both enlightened politics and public service. Rather they are now to prepare students for jobs. They are not to educate, but to train.

I'm not sure that traditional universities see it that way (certainly not mine), but the for-profits probably do. That's why they launch online nursing degrees and not online philosophy degrees. But this isn't their fault either--they just respond to the market. So the irony derives from the perceptions of the customer:

1. I want at good job when I graduate college. That's my S.
2. I'll never find a job as a dance major, so I better study something vocational. This becomes S'.
3. a(S') leads to the loss of opportunity to have the broad-based education useful at the higher levels of management.
   

I've made some assumptions here, but I have two bits of evidence. First, we had a meeting last week with executives from a large company. We were talking about internships and such, and it became clear from the conversation that all of the half dozen administrators saw the value of liberal arts degrees. Most of the IT hires didn't have IT training in college, for example, they just drifted into it because of interest. The takeaway for me was two-fold: first, general thinking and communications skills really are valuable, and second, attitude and approach to problem solving are just as important as training.

The second bit of evidence comes from research I quoted the other day in "Is General Education Worth It?". It shows that liberal arts degrees really are valuable in a monetary sense. Note that I also argue in that post that if S = {good thinking and communication skills} then our traditional general education may not be the way to achieve it.

If you combine the danger of simplifying a(S) => a(S') with the foibles of human decision-making I wrote about last time, it's a good assumption that administrators of all stripes will continue to provide us with examples of ironic a(S) => ~S. And interestingly, the higher the level of the administrator, the more pressure there is to approximate S with S', and the less assiduously the side effects will be judged, I conjecture. Allow me to name this conjecture after my historian friend Bob.

Robert's Rule of Disorder: The more powerful a decision-making apparatus is, the more likely it is to produce ironic results.

At this point, I could riff on the damaging influence of theory and ideology, which guarantee that S - S' is large, but I'll leave that for another day. I'm anxious to try out my new shortcut to work.