An Addendum and Apology

On Wednesday I wrote about my take-aways from the AALHE meeting in Lexington, and drew on some remarks from Trudy Banta. Some of the responses I got were justly critical of the way I mangled Trudy's message about the wider importance of the SAT example. It's true--I botched it, and I'd like to set the record straight so that Trudy doesn't begin to doubt her communications skills.

In order not to further fold, spindle, or mutilate someone else's message, let me say that what follows is my interpretation, and any faults should be ascribed to the author alone.

In a comment on the still-infantile state of the art of measurement, Trudy observed that even after more than 80 years of development and efforts to improve the SAT for its intended purpose, there is as much disagreement as ever about the validity of the SAT as a predictor of success in college. I intended to emphasize this point and what it portends for other projects that are even more ambitious. In the original article I left this argument dangling rather ineptly. So here it is, starting with a closer look at SAT, generalizing to the characteristics of industrialized tests (meaning massive, usually commercial, standardized instruments), and how we can do better with more authentic assessments.

Even with all the research that has gone into the SAT, in absolute terms it still isn't very good. The 2008 validity study from the College Board gives us (pg. 2):

[T]he weighted average correlation between SAT writing scores and English composition course grades was 0.32, after correcting for range restriction.

This is the most direct link I could find in the study between SAT and learning outcomes. Here we have actual writing in a standardized environment, correlated with course grades in coursework where writing is taught. If the course grade is related to how well students can write, then their potential as demonstrated by the SAT writing component ought to line up with it. And it does, but only to the tune of explaining 10% of the variance in grades (.32 squared). More generally, the variance in first year college grades explained by all the components of the SAT combined is 25% (page 5, squaring the adjusted R).

Any predictor with greater than zero R may be useful in the right context. But if we consider SAT's performance as an upper limit to what industrialized tests can do, it's a warning for current and contemplated high-stakes applications. This is a good place to segue to the mis-step in my prior article.

I gave three advantages of standardized tests. Here they are:

• They can have good reliability, so they seem to measure something.
• Because of this, they can create self-sustaining corporations that provide a standardized service to higher education, so there are fixed points of comparison.
• Even if the validity isn't stellar, some information is better than none, right?

I was still thinking mostly of the SAT when I wrote this, so you might consider this the best case scenario. However, I intended it to be clear (it wasn't) that this is damning with faint praise: although these bullets describe how standardized tests have colonized an ecological niche successfully, these reasons aren't sufficient to warrant their use for high-stakes purposes. Like value-added comparisons of institutions, for example.

The missing "anti-bullets" that describe the corresponding disadvantages are respectively:

• Depending solely on reliability is like searching for your lost keys under the lamp because that's where the light is. The drive for reliability creates artificial conditions like timed multiple-choice tests that have very little mechanical relationship to real-world application of knowledge. So reliability comes at a cost in validity. 
• The size of the testing industry means that it can use its resources to circumvent professional review of its products and sell them directly to politicians. 
• The validity comment is specious: "some" information isn't good enough in many circumstances. Imagine driving on a cliff's edge and only knowing where 10% of the road is. (An unfortunate real case is the publication of ranks of teacher "value-added" scores by the LA Times last year.) Exacerbating the general dearth of validity is is the fact that validity has to be determined locally after application of the test results (that is, to decide if some proposition about the actual results can be supported or not). It's very hard to do. Most of the time we don't know if our applications are valid or not. 

I would like to apologize to Trudy for bungling the transition between my summary of her remarks and the bullets in the original. I detracted from the point and may have misled or confused some readers. Mea culpa.

We can now step beyond the problems and look for solutions, and Trudy gave a list of some of these alternative approaches. Reliability isn't produced in a factory in Iowa City: we can get good reliability using rubrics on authentic student work. Additionally, the convenience of industrialized testing can nowadays be matched with local technological solutions. If we want to use portfolios, we don't have to have a thousand file cabinets, or mail copies to prospective employers; it's all on the web.

The big win for local solutions is validity: with the right application of technique, we can link assessments solidly with pedagogy at many levels of resolution, from individual assignments up to the institutional level.



As noted above, the testing industry has a lot of ability to protect its own interests, including great access to policy makers. With that in mind, I should wrap back around to the  theme of the policy conversations at the conference: the calls for accountability. It's not good enough to just resist the K-12-like solution; we have to find an alternative we find acceptable, and there are some good candidates available. That, I hope, is a clear and unmangled statement of the main message.

I don't mean to demonize "big testing," by the way. Maybe the best solution would be if we could convince them that authentic assessment and electronic portfolios are profitable new markets. I don't know what the chances of that are.

Inputs and Outputs

There's an interesting article in The Atlantic. There's much more to it, but this quote gets to the heart of the experience for many classroom instructors at all levels, I imagine:

Professor X is shrewd about the reasons it’s hard to teach underprepared students how to write. “I have come to think,” he says, “that the two most crucial ingredients in the mysterious mix that makes a good writer may be (1) having read enough throughout a lifetime to have internalized the rhythms of the written word, and (2) refining the ability to mimic those rhythms.” This makes sense. If you read a lot of sentences, then you start to think in sentences, and if you think in sentences, then you can write sentences, because you know what a sentence sounds like. Someone who has reached the age of eighteen or twenty and has never been a reader is not going to become a writer in fifteen weeks.

The source here is the book by "Professor X" called In the Basement of the Ivory Tower.

In Seattle Education there is a three-parter on privatizing education. There's a great game-theory example I didn't know about:

Proponents claim that by encouraging competition, privatization can improve the efficiency of public services. But there can be serious drawbacks. For instance, before fire departments were publicly run, groups of firefighters sometimes set fires just to earn money by putting them out!

 The evidence provided is more suggestive than compelling, but it's a nice business model (from the point of view of investors anyway) to control both supply and demand. In education this would translate as institutions not really caring much about education, but really caring about credentialism, and then seeking to maximize the number of credentials available and perceived as necessary to employment. In other words, if education is a gate to the promised land, put us as many toll booths as possible on the road between here and there.

Policy Questions Raised at AALHE

I flew back yesterday from Lexington and the first AALHE conference. I found it very stimulating. I put faces to names from the ASSESS list server, which was delightful.

In the opening plenary, Trudy Banta gave us a broad perspective on the evolution of the measurement and accountability, pointing out the weaknesses of value-added derivations and standardized tests in particular, and suggesting authentic assessment (e.g. portfolios and their analyses) as useful alternatives.

One point is particularly compelling to me. Trudy mentioned the pedigree of the SAT, and it's not hard to imagine the many hours and dollars that have gone into fine-tuning this test. These are smart people working with a will for a long time toward a well-defined purpose: predicting how well high school students will do in college. In my own experience as an IR person, the SAT does add some predictive strength to linear models, but not much once high school GPA is considered. A handful of percent of R^2 is it. At my present institution, it's virtually worthless as a predictor of first-year grades, which points also to the known biases of the test.

In short, there is some usefulness to the SAT, but it may not warrant all the trouble and expense. And of course some schools are now SAT-optional. I've written before about how, as a market signal, SAT overprices some students and overlooks others, creating the opportunity to use other (e.g. non-cognitive) indicators to find good students.

Trudy's comments did not go this far, but it's not hard to connect the dots. It's an important point: if all this effort yields so little result, maybe we're doing it wrong. The alternative is to admit that maybe this is the best we can do, and that our ways of knowing will just never be much good.

It should be noted that the plenary was planned as a kind of debate with two viewpoints, but that because of a cancellation by Mark Chun, only one side was presented. So in the defense of big standardized tests, here are some advantages:

• They can have good reliability, so they seem to measure something.
• Because of this, they can create self-sustaining corporations that provide a standardized service to higher education, so there are fixed points of comparison. 
• Even if the validity isn't stellar, some information is better than none, right?

The conversation is much more subtle than "down with standardized tests!" But you may have noticed that fine distinctions aren't part of the public debate on the quality of higher education. It would be great if Mark and Trudy and other experts introduced a public discussion like this on a message board or something--somewhere that the complexities of the issues could be fully explored and commented on over time.

[Update, see my "Apology and Addendum" that goes with this section.]

Bookending the conference, the closing plenary was given by David Paris, and also included a historical and political overview of where we are now. He is the executive director of the New Leadership Alliance for Student Learning and Accountability. You can sign up for email updates on their website. Paraphrasing David, the Obama administration talks nicer, but wants the same things as the Bush administration. And what that seems to be is "accountability." One important take-away for me is that we (higher education) have a window of opportunity that will soon close. That is, we can "do it to ourselves" or "have it done to us." The 'it' is amorphous, but centers on the accountability for cost and learning trade-off. In other words, all this public noise about how bad college is will create changes, and we should try to shape those.

It's hard to disagree with that. One approach is the Alliance's Institutional Certification Program, which you can learn about here. I'd like to see some examples of how it works in practice, but on the face of it, it seems like a great idea, similar in some aspects to the Bologna 'centering' process in Europe. I would call it a kind of horizontal professionalization, complementary to the 'vertical' type provided by discipline-based accreditations and professional organizations. Overall, it seems like a serious and well-reasoned approach, and I look forward to finding out more about it.

There was a good Q&A afterwards, and I suggested that higher education needs access to data pertaining to what happens to students after graduation, and that the federal bureaucracy sits on a gold mine of such information. This didn't have a chance to become a real discussion because of the format, but my interpretation of David's response was that the student-level record project that was floated and shot down (with help from higher education lobbyists) would have solved that problem. So we are complicit in causing the problem.

I don't buy this. First of all, what we need most right now is a deep historical understanding of how higher education has affected lives after school (graduation or not), for individual institutions if not individual majors. I understand that the data are imperfect and that there are privacy issues to be solved. But I think 9/11 shows that any privacy concerns can be solved rather quickly if the motivation is there. Specifically, it seems like it should be possible to combine some combination of student loan records, FAFSAs and  PELL grant records, tax records, federal employee records (including military) and so on to link where students took instruction, what their demographics were, and what happened to them afterwards. There are plenty of other places to find data, I'm sure, like state system databases.

The payoff for this is potentially enormous. Suppose we (higher education) can agree with the politicos that employment is a good outcome of a college education, and even talk about the details like what kind, how much pay, where geographically, and so on. Then we could look at what kinds of schools have what kinds of effects on what kinds of students. Is a biology program at my (private) university worth the cost premium over  the public one down the road? What about access? Which schools are economic engines by taking low-income inputs and (eventually) producing high-income outputs?

The payoff is so great that I think that if the government really can't find a way to do this, we should figure out how to do it ourselves. 

What we have now is survey data that shows that college generally pays for itself, but the price is going up. This is in opposition to the cries of critics that say students aren't learning much, if anything.

Here's my analogy. It's imperfect, but has the advantage of being vivid. Students are like movie scripts coming to our production companies. We have our specialties: niche art films or mass market gloss, and so on. Each script has an individual experience, no matter what our organization is. They get cast differently (different instructors), and we do our best to make a good fit usually. Or maybe we just get the cheapest actors we can find and hope for the best. We always have to rewrite the script to some extent, and our mission--the hope that gets buried in the Sisyphean rolling of semesters up and down the calendar--is that the screenplay comes to life and realizes its potential.

It takes a lot of money to make this happen, and it comes from investors who are increasingly grumpy. They say the movies are no good. They're too expensive and nobody likes them. Film critics like Arum and Roksa used data from a large number of scripts in production to claim that a large portion weren't being significantly improved in the rewriting stage.

But we don't have any real information about what the audience thinks. We can analyze the heck out of our own productions, and do six-sigma and professionalize all we want, but until we understand what the audience thinks, we don't really know if we're doing any good. Maybe all those films get shown once, and then end up in a storage shed. 

Maybe some types of films shouldn't be made anymore. Maybe Forensic Dolphin Psychoanalysis shouldn't even be a major because there's no audience for it. 

The recent explosion of for-profits makes the problem more urgent to solve. Most industries have to depend on their products working. We don't have much direct evidence one way or the other in the kind of detail we need to make decisions. It's starting to happen, for example with student load default rates getting attention. But we can go a lot further than that.

Imagine if we could sit down with the investors and agree on the long-term goals, have metrics that more or less tell us how well we're doing, and plan together how to get there. Are the goals strictly economic? Or do we want people to pursue happiness and bear arms? Even if it is just economics, it presents an opportunity to really understand how, say, liberal arts education matters in job mobility, lifetime income, and so on. Institutions could say "yes, you'll get a fine job, but in addition you'll have a fulfilling career." Or whatever their mission is. Instead of talking past each other about standardized assessments and authentic assessments, we could figure out how to work backwards from the real goals to real assessments that matter at the strategic level and then add institutional flavors that matter to us. That would be an exciting and productive conversation.

The alternative is grim. If we only focus on what goes on inside our production studios, the future of the nation is at the mercy of every critic with a theory or an agenda. I'm not sure which is worse: theories or agendas. Some will want to break down every step of movie making into a reductionist flow-chart, and create spreadsheets to show the rate of script-to-casting time or use biometrics to calculate charisma factors of the actors. There's no bottom to this because although movie scripts are only 120 pages, individual brains of our students have perhaps 100 trillion synapses each. If each one is, say eight bits of information, that's about four billion times the complexity of a movie script. Each one. Others will work backwards from agendas to create policies that make no sense in context.

Even if we think we've solved the problem and get universal agreement that our learning outcomes are being achieved at the university, how do we really know what the effect is after graduation unless we measure that? Maybe they're learning the wrong things. Maybe some small college has methods that work twice as good as ours. Maybe we can reduce costs and increase quality. The ingredients are wonderful: a diverse ecology of isolated experiments. We just can't see where the lab results are recorded in order to make conclusions.

Yes, we need individual student records. But we can't wait for that. The problem is too important. Maybe tax records and FASFAs can't be mashed up because of political or technical reasons (but do you really think Mark Zuckerberg couldn't figure this out in an afternoon?). If that's the case, then we have to find another way to measure historical and ongoing long-term outputs in such detail that it can inform institutional decisions. 

What we're doing at present is creating our own dramatic screenplay: an epic version of "No Exit."

UniLeaks

Perhaps this was inevitable...a WikiLeaks clone for higher education. Seems to be UK-centric so far, but don't wait for the rush--dust off those fiery curriculum committee minutes and start some controversy!

Image from UniLeaks.org.

The Long View

In "On Design," I gave three versions of designing for an outcome: soft, forward, and inverse, in increasing degree of difficulty. The question "how difficult is inverse design?" is of utmost importance when we consider complex systems. As a real example, consider how the government of the United States is "designed." By this I mean, the way laws and policies are created and enforced. Because of conflicting goals of different constituents and the inherent difficulty of the project, there is no complete empirical language to describe a state of affairs, let alone do a forward simulation to see the status of the nation in, say, three years. And even if we did have such a language, we would be limited to simulation and prediction of only the "easiest" parameters. And even these would subject to the whims of entropy, a subject I'll take up later.

I submit that individual governments, as well as companies, universities, and militaries use soft design with bits and pieces of forward and inverse design thrown in (for example trying to forecast near term economic conditions to help determine monetary policy).

Government in the general sense has been "designed" by the process of being tossed into the blender of fate, to be tested by real events. See the following video of the history of Europe to see what I mean.

Natural selection would seem to be be at work here, weeding out the worst designs. But it's not Darwinian  because the countries change rapidly over time with the population and minds of leaders. One truly spectacular bad idea (like invading Russia, apparently) can bring down a whole nation. So what we are left with is a very temporary list of "least bad designs." Of course, many other factors are important, such as geography, natural resources, and so on. Even so, the people who live there still have to make use of such advantages. If Switzerland abandoned its natural mountain fortress and invaded Russia, it likely wouldn't end well.

Darwinian evolution is different from this national evolution. In the former, good solutions can be remembered and reused through genes or any other information passed from generation to generation. Diversity is created through recombination, mutation, population isolation, and so on. Darwinian evolution comes with an empirical language that we partly understand. To make a metaphor of it, "programs" are written in phenotypes and these "are computed by" the laws of physics and chemistry using the design and environment as "inputs." The fact that scientists can discover this language and use it to make predictions should be appreciated for the miracle that it is: we are witnesses to a dynamic but understandable problem-solving machine of enormous scope that has worked spectacularly well at producing viable designs with only an empirical language. Evolution does not use predictive techniques (that is anticipating that a critter will need wings and therefore building them--for a dramatic example of this, see this video). But there do exist creatures who do use forward and inverse design to plan their day. If you throw a ball at target, you're predicting. If you go to the fridge to get food, you're using the inverse technique: starting with the outcome (get food) and working back to the solution. But this still isn't good enough.

Here's the rub: forward design isn't enough to guarantee any more than short term outcomes, and our ability to do inverse thinking is very limited. Let me pose a problem:

What present actions will lead to [insert subject]  existing in a healthy state 100 years from now? 1000 years from now? 10,000 years from now?

The question is posed as an inverse design problem, starting with the goal and asking what needs to be done to achieve that goal. If we had a good language with which to describe the states, we could at least imagine an evolutionary approach, shown in the diagram below, with the red dot being the desired goal.

We're asking "where do we need to be NOW in order to end up where we want to be AFTER?" The forward design approach is to simulate lots of "befores" and see where then end up "after," and choose the best solution we can find. If we are lucky, we can use a Darwinian approach, combining partially successful solutions or tweaking "near misses" to home in on the best solution we can find. This depends on the sort of problem we're trying to solve, and specifically whether or not it is continuous in the right way. Sometimes being close isn't any good--those "a miss is as good as a mile" problem. All of this highlights the importance of the empirical language, which must have rules precise and reliable enough to allow this kind of prediction and analysis. Clearly in the case of governments, companies, universities, or even our own selves, this is not possible.

With only forward design techniques and lacking a good empirical language, we can still solve the problem with massive brute force: by actually creating a host of alternatives and seeing what happens in real time. A computer game company could do this, for example. Rather than spending a lot of money to find the bugs in its game, it could just begin selling it with the knowledge that the game will be reproduced on many different systems and display many kinds of problems. With this data in hand, it can begin to debug. This seems to be a real strategy. This sort of solution obviously won't work for a government, although in a democracy we have a non-parallel version: swapping out one set of leadership for another routinely, to see what works best (in theory at least).

Paying it Backward. What would it look like if we had all the tools to solve the inverse problem? We could pose it precisely, work forward simulations like the one pictured above, and we would have a huge advantage, shown in the graphic below.

Again, the red dot is where we've decided we want to be after a while. The inverse solver shows us all of our possible starting places in the now. There are multiple ones if we have not completely specified the eventual outcome, which tells us what opportunities we have now to optimize other things than the one we were thinking about when we posed the problem.

Example: An artillery officer is given the task of attacking a distant building. The locations of his guns and the target are fixed. Using ballistics equations we can work backwards to show all the possible solutions: e.g., a high arcing shell like a mortar, or a flat trajectory. This decision will determine how much powder is used.

The inverse solver illuminates "free" parameters and allows us to customize our solution.

Conclusions. For the real-world messy problems we face in complex human organizations, it's fair to say that we are not very good at long-term planning. In some cases it may be impossible because the forward simulators simply don't exist--there are no reliable cause and effect mechanisms. Trying to predict the stock market might fall into that category. In other areas, short-term goals are given preference over long term goals. This is reasonable for at least two reasons. First, the longer out the prediction is, the more likely it is wrong. Second, as individual humans our ability to affect events is a narrow window (e.g. a term for a legislator), and most of us want positive feedback now, not 1000 years from now.

As a very real example of our collective difficulty with long-term planning, consider the issue of human-caused climate change. With the terminology given in these two posts, it's easy to dissect the arguments:

• Empirical language. The basic facts about temperature change and CO2 levels are accepted by the scientific community, but still debated as a political matter. 
• Forward design: Cause and effects are challenged in the political discourse, computer simulations are therefore called into question. Unaccounted-for causes are conjured to explain away data that is (to some extent) agreed on.
• Inverse design: Prescriptions about what to do now to affect the future climate are attacked as being too detrimental to the present, and/or useless.

The question of what happens to the climate is scientific--this is the arena where we are best at design. If we can't understand and act on such threats intelligently, it's very hard to make the argument that we have any long-term planning ability collectively. Note that I'm not neutral on this particular question. The evidence is overwhelming that the risk to our decedents is very high. But go read the experts at RealClimate.org. 

Higher Education. This isn't a climate change blog; what does this analysis have to do with your day to day job in the academy? Everything. From the oracle at Delphi: "Know Thyself." What functional aspects of your institution are understood? How many are understood well enough to make predictions? How many are understood well enough to make inverse predictions?

We work backwards all the time. Suppose budgetary concerns have pushed up freshman enrollment targets, so the admissions office is tasked with bringing in 1000 new students for the fall. This is posed as an inverse problem, and with the standard empirical language of admissions we can build basic predictors. This is the "admissions funnel" prospects->applicants->accepted->enrolled (this is the simplified version). We usually have historical data on the conversion rates between these stages. If the universe is kind, we can use Darwinian methods--keep what works, assuming what works last time will work again, tweak to see if we can make it better. If there's a significant discontinuity (suppose state grants suddenly dried up, or we have a new competitor) the old solutions may not work anymore.

We can and should work hard to create an empirical language and use it to simulate our futures, trying to end up in a good one. This isn't enough.

Short term optimization leads to long-term optimization sometimes, but not as a rule. Think of it as a maze you're trying to escape. If at every junction you choose a path that takes you closer to the opposite wall of the maze, you may make smart moves in the short term only to discover there's no exit there: a long-term failure.

(original image by Tiger Pixel ) 

Even though we can't solve the inverse problem entirely, we may find that we have enough empirical vocabulary to make some important decisions. What do we want the demographics of the student body to look like in 10 years? Answering this question about the future puts constraints on how we operate now, even if they are fuzzy and inexact. Forget about solving the problem exactly--that's impossible--and think about what constraints are imposed in the big picture. Do we want a strong research program, or a large endowment, a national reputation for X? More idealistically, what do we want to be able to say about our alumni in a decade or two? How much does their success matter, and what sort of success are we talking about? Work backwards. I'll finish with an example.

Example: Student Success. If we focus on a goal for our ultimate accomplishment: the education of students, what does this reveal? How exactly do we want our students to benefit from their education? Some possibilities:

• Happiness with life
• Being good citizens
• Being successful financially
• Being loyal to their alma mater
• Getting a job right out of school

The overwhelming narrative in the public discourse is that we want students to be "globally competitive" and "get well-paying jobs." But that's a means to some end. What is the ultimate aim? On a national scale, the answer might be better national security and a stronger economy. For an institution, the answer might be that we want our products strongly identified with our brand, or that we want them to donate lots of money in the annual campaign. I think it's important to start there and ask "why do we care?" You could follow that up with lots of activities, like telling the students themselves why you care, if that's appropriate.  

Suppose that we care because we want the institution to be able to rely on an international body of alumni who will contribute back in money, connections, expertise, and other intangibles. This will enable the university to grow by presenting global options that may not be apparent now, but also establish an exponentially-growing revenue stream through an expanding endowment as successful alumni give large gifts back to the institution. Working backwards, we might identify several tracks to success:

• The state department track--prepare students for high-level international government positions
• The military track--ditto for the miltary
• The global entrepreneur track--help them achieve independence
• The big corporate track--give students the skills to compete and succeed within vast multi-nationals
• The wildcard track--for those students who don't fit the mold, are intelligent and creative, but don't want to be entrepreneurs or work in a cubical. This could include scientists, philosophers, and artists of all stripes.

If we keep working backwards, even without exact solutions, we can make some good guesses as to the curriculum each track needs, and the type of faculty mentors we need. This neatly sidesteps the drift toward vocational education that the public narrative implies, and gives the institution a raison d'être. There's nothing wrong with telling students "we want you to succeed so that you'll help us succeed." This sort of pseudo-altruism is what keeps the population going, after all. Thanks, mom and dad.

Think Backwards. Short term forward planning is like beer: it's obviously a good idea at the time, but watch out for the hangover.

The Fat Middle

There are those ideas, like Darwinian evolution or the basics of information theory or game theory, that cannot be unthought. They are powerful enough to change one's Weltanschauung, and having done so mark the mind like a growth ring. I suppose that should be the aim of a liberal arts education. We could probably do a better job of selling it that way.

Lately I've been orbiting the gravitational well of an idea that, while not on the order of calculus or market theory, makes a lot of things click into place for me personally. I call it The Fat Middle.

Most things in life are ruled by combinatorics. We experience and try to make sense of many dimensions of sensory data, which if our brains didn't automatically data compress for us, would overwhelm us with complexity. The very molecules we're made of are combinations of more permanent bits--those atoms hypothesized by Democritus about 1500 years ago. Our civilization comprises a vast economy with many, many moving parts. It's easy to get lost in the permutations.

Every day we figure out how to get from where we are to where we want to be, literally or metaphorically. It's that part in the middle where the opportunity lies. Not for you and me usually, but for someone who spends all day thinking about the middle. The middlemen.

Having a way to negotiate the way between trying to sell a house and having sold it, for example, is valuable. Finding someone who knows about applicable laws and contracts, and is hooked into a network designed for moving real estate is a valuable thing.  The creation of such a Middle is like pushing a volatile molecule up the energy scale until it cracks open to spill out far more energy than you put into it.

This Middle provides a public service, whether it's selling houses, buying a good book, or finding a job. Cities sprang up along trading routes (often on rivers) because of the value of the Middle. Paved roads were investments in the Middle where the rivers couldn't go.

Phase One: Creating the Middle

Basically, if you can find a way to be of use between a supply and a demand, you create the Middle, and this is nice because it naturally wants to be standardized. Roads are narrow, standardizing routes. Realtors may compete with one another, but they all cooperate when it comes to making a sale.

Here are some Middles:

• Google: between you and everything on the Internet
• Facebook: between you and your friends
• eBay: between you and the world's flea market
• Music companies: between you and music (pre-Internet)
• Newspapers: between customers and businesses (pre-Internet)
• Higher Education: between high school grads and good jobs
• Medical Establishment: between you and medicines or medical services

The last two of these are not informational, but still play the role of the Middle by providing access for a price to exclusive services and benefits. There didn't used to be a Middle for either education or medicine. You'd simply go to a teacher and pay tuition or take a chicken to the witch doctor, respectively. But standardization and massive organization have created a Middle for accommodating a huge demand.

Phase Two: Getting Fat

Once the Middle is standardized, it becomes a gatekeeper as well as access path. Tolls get higher on the road, until the price is as high as the market will bear.

As an example, consider an entrepreneur who builds a bridge over a river, meeting a demand for transportation. He makes enough money on tolls to maintain the bridge and have enough left over for a nice income. But as the use of the bridge becomes the standardized Middle, and travelers depend on it more and more, he realizes that he controls a natural monopoly. So he increases the price until the total revenue levels out, fattening the Middle.

It's interesting to note what is allowed to get fat and what isn't. Power companies in the US are regulated by the government, presumably as a nod to the fact that providing electricity is a public good. Roads are mostly free to drive on, paid for indirectly through taxes. Access to water provided at very low cost to most of us. On the other hand, access to professional medical care is not a concern of the government in the same way. Education falls in between, given the range of opportunities between public and private schools.

Uncontrolled, the Middle will make itself fat and continue to secure its position. Media outlets are particularly well suited for the latter task, since they are between news and consumers (or were until lately).

We used to pay more than $50/month for a wired telephone line. Then along came the Internet and Skype, and hey, I get that basically for free now. I can only conclude that most of that $50 was Fat Middle for the phone company. Don't cry for the phone company, though; they can still get away with charging a dime for every text message you send--all Fat Middle for them.

Those companies in the Middle can use their profits to tilt legislation their way, like the music industry and copyright laws, or the recent assault on net neutrality. I think net neutrality cannot hang on much longer because of this pressure. Or the cozy relationship between banks and regulators that helped create the overindulgence in consumer debt.

Another technique to fatten the Middle is to over-promise the usefulness of it. Think of it as a "Gold Rush" effect. This is more suitable to some services than others. The creation and marketing of "maintenance" drugs is one example, or Monsanto's attempt to corner the seed market with one-use varieties, so you can't grow your own. Soon, Facebook will be plugged into so many facets of your personal life that you won't know how to live without it, right? A bachelor's degree is absolutely essential to getting good job. Nobody questions that. Why?

Phase Three: Revolution

Once the Fat Middle has squeezed out every bit of extra economic value, it no longer performs a public service. This is an extreme position that few Middles may ever achieve due to the particulars of the betweenness. In the bridge story, if the owner charges so much to pass over the bridge that it would be economically more viable to take the ferry, but there are new laws outlawing such "unsafe" transportation, the fattening has ripened.

Technology has a history of dining on the Fat Middle. Toll roads are mostly gone in the US. Newspapers can't make fat revenue by cornering the advertisement market, and the music producers can't isolate musicians from their audience and charge a rich premium for access. AT&T can't charge me 30 cents a minute to talk to my parents in Illinois anymore (when that bill came, I immediately switched to another phone company. I got a letter from AT&T a couple of months later saying "we're sorry! we want you back!" Fat chance.).

The technology solution is bound to rearrange the Middle, but it will again solidify. Costs and laws around the Internet will slowly evolve under the pressure of the rich rewards dangling there, until you pay per click to Facebook and Youtube, and don't even think about starting a web business without going to the bank first. I hope I'm wrong.

The Fat Middle of Higher Education

In the July 20th Chronicle article "Learning From Socrates and Adam Smith on Financing Universities"
Richard Vedder writes:

If one were to allocate faculty salaries for instruction to account for the non-instructional dimension of university service, faculty compensation for instructional services often is well under 20 percent of revenues raised by institutions, and almost never as much as 50 percent. 

 So what? The point being made here is that the ostensible principal raison d’etre of most universities—the education of our youth—is really a small part of university activities. Put differently, if the faculty salary for instruction to institutional revenue ratio were to rise to, say, 50 percent, by reducing the non-instructional dimension of university spending, the total cost of educating students would fall dramatically—to roughly the levels found in many other industrialized nations in the world. 

This is an argument that universities (public and private) are too fat.

On October 20th, Vedder wrote another article "Why Did 17 Million Students Go to College?", citing two apposite bits of information. Both indicate a "Gold Rush" effect for higher education: that the Middle between high school graduates and good jobs has been oversold by the supplier:

Over 317,000 waiters and waitresses have college degrees (over 8,000 of them have doctoral or professional degrees), along with over 80,000 bartenders, and over 18,000parking lot attendants. All told, some 17,000,000 Americans with college degrees are doing jobs that the BLS says require less than the skill levels associated with a bachelor’s degree.

He goes on to cite a paper about the marginal return on higher education investment:

This week an extraordinarily interesting new study was posted on the Web site of America’s most prestigious economic-research organization, the National Bureau of Economic Research. Three highly regarded economists (one of whom has won the Nobel Prize in Economic Science) have produced “Estimating Marginal Returns to Education,” Working Paper 16474 of the NBER. After very sophisticated and elaborate analysis, the authors conclude “In general, marginal and average returns to college are not the same.” (p. 28)

Unfortunately, the paper itself is behind a paywall (the irony...). A low marginal return means that more investment is hard to justify: we're at the point of "diminished returns."


In a nutshell, the argument is that higher education has become a Fat Middle, and is ripe for revolution. The form of that revolution isn't hard to fathom: low-cost, high quality, online programs. The way Vedder puts it is:

Higher education is on the brink of big change, like it or not.

Expect to see more analyses like the one recently from The Wall Street Journal, where "Putting a Price on Professors" by Stephanie Simon and Stephanie Banchero refers to:

A 265-page spreadsheet, released last month by the chancellor of the Texas A&M University system, amounted to a profit-and-loss statement for each faculty member, weighing annual salary against students taught, tuition generated, and research grants obtained.

This creates a certain kind of business-speak dialog:

"Every conversation we have with these institutions now revolves around productivity," says Jason Bearce, associate commissioner for higher education in Indiana. He tells administrators it's not enough to find efficiencies in their operations; they must seek "academic efficiency" as well, graduating more students more quickly and with more demonstrable skills. The National Governors Association echoes that mantra; it just formed a commission focused on improving productivity in higher education.

A part of this is just the current demonizing of education that seems to be in vogue, and the narrow viewpoint that colleges are like factories with a uniform input that should be able to "six-sigma" end product. But there are valid points to be made about the cost versus return of post-secondary degrees too, particularly focusing on those costs that have little to do with the instructional mission.

It's ironic that the business viewpoint would be used to make this particular criticism of higher education, which (if they are correct) is only doing the same thing that, say, the drug companies do. But for some reason, education is seen as a public good and pharmaceuticals or access to medical care are not.

Opportunity

It's not all broom and doom. You can find a Middle of your own and enlarge your fortune. Here's one idea. One of the most frustrating problems in dealing with corporations is getting problems solved through their customer service departments. Some are good (like our local Time-Warner office), and some aren't. Standardization and transparency would work wonders in this area. So all you have to do is create a "complaint engine" that becomes the standard interface between companies and their customers for resolving disputes, with public ratings showing response time, resolution rate, and individual comments.  At first, companies will hate it. Then the more progressive ones will see the advantages and start asking for plugins so they can feed directly into their PeopleSoft (or whatever) systems. Then you can sit back and watch the Middle fatten.

The Internet (while it lasts) creates enormous potential for Middle solutions. Even within a university, there are opportunities, like creating an institutional document repository.

Viewpoints

Four articles, one for each Horse of the Apocalypse:

NY Times: A Very Interesting Idea  (about getting two years of college while still in high school)

The school, a fascinating collaboration between Bard College and the city’s Department of Education, was founded in 2001 as a way of dealing, at least in part, with the systemic failures of the education system.

How about the systemic failure of American society to value education?  Maybe we should turn the Cartoon Network into an education channel and let Spongebob take over.

Wall Street Journal: A Lament for the Class of 2010

Economists theorize that this may be that very rarest of things—a generation that does not do as well financially as the generation that spawned it.

Are these the same ones who came up with credit swaps?  Is this prediction based on some new doomsday derivative you guys are cooking up and haven't told us about yet?

NY Times: End of the University as We Know it (from 2009)

The division-of-labor model of separate departments is obsolete and must be replaced with a curriculum structured like a web or complex adaptive network. Responsible teaching and scholarship must become cross-disciplinary and cross-cultural. 

Yay! No more department meetings!  We'll have complex adaptive network...uh...seances?

The American Scholar: The Disadvantages of an Elite Education (from 2008)

The first disadvantage of an elite education [...] is that it makes you incapable of talking to people who aren’t like you.

Sorry, what?

---

Update: Three more I found this morning, ruining the whole horse metaphor.


CNN Money: Why Universities Should Hate the iPad

If students embrace textbooks on the iPad, college bookstores may lose their shirts. 

Is income from textbooks really the most important consideration? Is that enough reason to "hate" the technology?

NYTimes: The Value of College

As it happens, though, labor economists have spent years trying to answer this exact question, devising careful studies to see whether students actually benefit from college. The answer, in a nutshell, is that they do. This is a nice example of Occam’s Razor: the simplest explanation is often the correct one.

 These are the real learning outcomes.

Chronicle: Spending on Political Influence by For-Profit Colleges (subscription required)

For-profit college groups are significant players among all educational organizations in their use of campaign contributions to curry favor in Washington, a Chronicle analysis shows.

And where does the money come from?  Oh, yeah, PELL grants and government-sponsored student loans.  It's a nice little feedback loop.

2020 Gogs and Recruiting Costs

Everyone, it seems, is developing "2020" plans, a riff on the perfect vision score. The best quote I saw (on reddit) was that in 10 years we'll look back with 2020 hindsight. In that vein, The Chronicle has published an article called "The College of 2020: Students." You have to pay for the full report, but the executive summary is here (pdf). The abstract:

This is the first Chronicle Research Services report in a three-part series on what higher education will look like in the year 2020. It is based on reviews of research and data on trends in higher education, interviews with experts who are shaping the future of colleges, and the results of a poll of members of a Chronicle Research Services panel of admissions officials. ... Later reports in this series will look at college technology and facilities.

For the IR folks, Noel-Levitz has an interesting Cost of Recruiting report, which you can find in pdf form here. This gives good benchmarking information for assessing the efficiency of your admissions office.

Survey Addresses Drop-Outs

Public Agenda's recent report "With Their Whole Lives Ahead of Them" is subtitled "Myths and Realities About Why So Many Students Fail to Finish College." It's essential reading for anyone interested in student retention. Citing an average 40% six year graduation rate for four-year degrees, the report tries to answer the "why?" question with a survey of 600 young adults who had first hand experience. The complete methodology can be found here. The report is released under the creative commons license.

The demographic characteristics of those surveyed belie the image of a typical college student. Quoting from the article:

• Among students in four-year schools, 45 percent work more than 20 hours a week.
• Among those attending community colleges, 6 in 10 work more than 20 hours a week, and more than a quarter work more than 35 hours a week.
• Just 25 percent of students attend the sort of residential college we often envision.
• Twenty-three percent of college students have dependent children.

The main part of the report is framed around "myths and realities," such as:

MYTH NO. 1: Most students go to college full-time. If they leave without a degree, it’s because they’re bored with their classes and don’t want to work hard.

REALITY NO. 1: Most students leave college because they are working to support themselves and going to school at the same time. At some point, the stress of work and study just becomes too difficult.

According to the survey, "Those who dropped out are almost twice as likely to cite problems juggling work and school as their main problem as they are to blame tuition bills (54 percent to 31 percent)."

Graphs display ranked survey items. A portion is shown below.


The third section addresses an issue that I've discovered independently.

MYTH NO. 3: Most students go through a meticulous process of choosing their college from an array of alternatives.

REALITY NO. 3: Among students who don’t graduate, the college selection process is far more limited and often seems happenstance and uninformed.

My retention study at one institution showed that students whose had reported on the CIRP that they were at their first-choice college had three other interesting characteristics. One was that they tended to be first-generation students. They also were by far at the highest risk for attrition. And in a subsequent survey two months after the CIRP, many of them had changed their minds about the first-choice qualification. In short, they were uninformed consumers and became quickly disaffected--or at least, that's the way I interpreted the data.

The survey asked for proposals to help other students get a degree. The top responses are shown below.

The other two "myths" presented in the report are:

MYTH NO. 2: Most college students are supported by their parents and take advantage of a multitude of available loans, scholarships, and savings plans.

REALITY NO. 2: Young people who fail to finish college are often going it alone financially. They’re essentially putting themselves through school.

and

MYTH NO. 4: Students who don’t graduate understand fully the value of a college degree and the consequences and trade-offs of leaving school without one.

REALITY NO. 4: Students who leave college realize that a diploma is an asset, but they may not fully recognize the impact dropping out of school will have on their future.

Much more information and analysis is available on the report itself. I'm not crazy about post hoc retention surveys because if we want real predictors, we need to find out information before students leave, and afterwards causes and effects may change over time in the minds of the students, as with the first-choice question noted above. On the other hand, this report is thoughtfully done, and does seem to illuminate some interesting issues.

There is a section on what can be done to help. Providing more financial aid for part-time students is one, as well as more flexible options for attending. I presume that online courses would fill the second bill nicely. It seems obvious that more work-study options on campus would help too--a double win for the college, since students will be more engaged, and provide cheap labor. Addressing child-care problems for students ought to be high on the list too.

Increasing Transfers

A Chronicle article "Report Highlights Characteristics of Colleges With High Transfer-Success Rates" cites a new publication from The Pell Institute about successful transfer programs. The new report hasn't shown up on that organization's publications page yet, but there are some interesting tidbits in the review:

The study found that early exposure is critical to ensuring a successful transition to college, especially for students who are from low-income families or are the first in their families to go to college. Such students are likely to be unfamiliar with higher education and what it will take to earn a bachelor's degree.

This echoes what I found in a retention study--first generation students didn't really understand the product they were buying, and quickly became disenchanted once they arrived (here).

The article lists some elements of successful two-year programs that built bridges to four year programs:

• specialized advising
• flexible scheduling of academic and support services
• first-year seminars that include strategies on note taking, test taking, and navigating campus services
• one-stop shops, where services such as registration and financial aid are placed together in one central location
• replacing a tall customer-service counter with desks to make the interaction between students and staff members more accessible and personal
• offering clubs and organizations
• setting aside an hour each day when no classes were scheduled to further encourage participation
• employing faculty and staff members of similar backgrounds to their students

I imagine that many of these elements would work from the other perspective too: implemented for students at four-year institutions who seek to increase transfers from two-year institutions.

Planning Resources

As we get into the season of setting tuition, finalizing aid policies, tweeking strategic plans, and prognosticating about enrollment, it's handy to have a reference library.

Some of these links have been cited before, but I wanted to put them all in one place. I will likely add to them without bothering to add "update" to the post, to make this a reference page of sorts. If you have a good one, please forward it.

Strategic planning:

• Six Essentials—and Six Common Mistakes—in Cabinet-Level Strategic Enrollment Planning
• NACUBO Benchmarking tool
• IPEDS Peer Comparison Tool
• NACUBO Financial Services Benchmarking Survey
  

Retention/Success:

• Noel-Levitz: 2009 Student Retention Practices and Strategies at Four-Year and Two-Year Institutions
• The Pell Institute: Publications on Opportunity and Successful Strategies
  

Recruitment:

• 2009 Student Recruitment Practices and Strategies at Four-Year and Two-Year Institutions
• Lawlor Group: The Year in Review College Admissions Averts a Crisis
• Lawlor Group: When Market Conditions and Public Perception Collide A Looming Crisis
  for Higher Education
• NACUBO:New Survey Predicts Enrollment Will Hold Steady at Independent Institutions
• Applywise.com: New Survey Indicates that College Bound Teens Will be Particularly Hard Hit by Economic Crisis
• IBM and Marist Survey Shows U.S. College Students Want Technology Skills to Compete for Jobs
• Index of whitepapers at Education Dynamics (requires free sign-up to download papers)
• 2009 Report on the Impact of the Economy On College Enrollment
• Index to SEM Works white papers
  

Pricing:

• Admission, Tuition, and Financial Aid Policies in the Market for Higher Education
• Research Tools to Guide Tuition and Financial Aid Decisions
• Overcoming price sensitivity … Means marketing affordability, and it's what every IHE needs to do - On The Money
• College Board: Trends in College Pricing 2009

Financial Aid:

• College Board: Trends in Student Aid
• Noel-Levitz: 2009 Discounting Report
• 2007 NACUBO Tuition Discounting Study Summary Results
  

Web Strategy:

• Scrolling Toward Enrollment Web Site Content and the E-Expectations of College-Bound Seniors
• Web Content Management Systems

Quote, Representation

Reviews of an institution's general education (or liberal arts requirements) might be characterized as nasty, brutish, and interminable, to paraphrase Hobbes. You can get a taste of it in "Progress, But No Votes, For Review" in the Harvard Crimson, about that institution's journey to re-enlightenment. And then there's a fascinating set of minutes from "WCU" here that documents the tortured trip:

The creation of a dynamic general education program that the faculty will "buy into" can perhaps change the University culture in a way that talking cannot. So, how are the faculty enticed to "buy into" a program? It is necessary to ignore the noisy 15-20% who fight any change in the status quo. [3/21/97 Minutes]

Also found in this rich set of minutes is a bibliography of recommended sources for those foolish enough to buy a ticket on this train:

The committee will look first at AAC's New Vitality in General Education, the first half of Weingartner's Undergraduate education: Goals and Means, AAC's Strong Foundations: Twelve Principles for Effective General Education Programs, and Gaff's New Life for the College Curriculum.

This was, of course, before the AAC&U released its LEAP initiative.

There is a virtual thicket of documentation to be found via google search, a salad of despair, Thomas Pynchon might say (see "Meeting Salad" for the quote).

My particular interest this morning was the composition of gen ed review committees, but it's easy to get lost surfing the epics struggles documented online--surely enough material for several posts. Heck, I should invite applications for a gen ed review "sub blog."

The practice I'm accustomed to is to select a representative from each large academic unit (a department or division, for example) that has content usually found in general education. This would include sciences, humanities, social sciences, and PE. Our accreditor, SACS, actually has a list of minimum requirements for a gen ed program.

Here's an example from "MU":

The General Education Review Committee is composed of two faculty representatives from each of the four academic units elected by Faculty Senate for overlapping terms of two years, one representative from non-school faculty elected by Senate for two years, and two student representatives elected by Student Senate for overlapping terms of two years. In addition, a chairperson is elected by Faculty Senate from the Faculty Senate membership to serve a three-year term.

There are also non-voting members with administrative titles.

Both the AAC&U and Academic Commons advertise an interest in the liberal arts. The former has a book on the subject of gen ed review, but you have to buy it to see it. The site whatwilltheylearn.com compares general education curricula to see if they get the basics, as defined by a conservative list of subjects.

The issue that I take up in the title of this post is the question of representation. If we assume that the practice of creating a review team in the way described above is common, then the result is a discriminatory process that almost guarantees political turf battles.

Think about the arrogance in the assumption "we'll pick a math guy to represent all you math people, and a history person to represent all of them, and so forth." This is roughly the same as picking a Cubs fan to represent all Cubs fans on how a stadium should be build or picking someone who likes hot dogs to represent all hot dog eaters on how to choose a new menu.

There are many, many design questions for a general education review team to consider, and only a small part of it concerns the number and type of courses to be chosen. With the usual discriminatory practice, the only basis for "representation" is to lobby for one's own area: let's pack in as many math courses as we can, so we can get more faculty slots. Unless the committee can rise above that (and take the heat from their respective departments), the outcome is almost guaranteed to be bland, and the dynamics encourage unhealthy inter-departmental politics. Of course, there's no way to avoid politics. Why not make that a virtue?

In a representative democracy, we don't go out an pick one Toyota lover to go to congress to represent all Toyota lovers. That kind of blatant discrimination would never be tolerated (okay, gerrymandering aside--but that's ugly too). What if we modeled the selection of a gen ed committee on an actual representative democracy? Nominees for the committee slots would be put up for a vote. They would be expected to set out their views on general education, probably in writing, but perhaps in a debate with other candidates too. It could be a vibrant, rich affair that properly celebrates and illuminates the process as it begins. It would also illustrate the use of liberal ideas that were hard fought in the enlightenment instead of forcing an illogical form onto a titanic exercise in critical thinking. The irony of the usual process is eye-popping.

Just like in a representative democracy, the election of the representatives does not by itself create the design. The hard work of design and compromise, and yes, departmental politics, still has to be done.

Of course, this may be too Utopian. Perhaps what would happen is that the initial gamesmanship over the election of representatives would center solely on discipline loadings for the final product. Maybe the big departments would just vote themselves candidates that would beef up their areas. If so, that would be sad, but at least it would reveal the true motivations of the academy.

Update: I just noticed that InsideHigherEd has an article about the "jeopardy" liberal arts is in, citing a need for proof of relevance. They note that some institutions have presentations that resemble the kind of political debate I described in my Utopia:

The University of Alaska at Anchorage has started a lecture series where professors from different liberal arts disciplines give talks aimed at attracting a popular audience. Utah State University has invited successful alumni back to talk about how their liberal education shaped their careers.

It seems that having more open processes for constructing (via faculty procedures) and validating (with authentic assessment) general education means and results, potential students will have a better chance of seeing the point. In a retention study I did (see "The Value of Retrospection"), we ultimately discovered that many of our students simply didn't understand the product they were buying.

Other Reading

Here are some noteworthy articles that speak for themselves:

Strategic enrollment planning:

From Noel-Levitz (papers and reports):

• Six Essentials—and Six Common Mistakes—in Cabinet-Level Strategic Enrollment Planning
• 2009 Student Retention Practices and Strategies at Four-Year and Two-Year Institutions
• 2009 Student Recruitment Practices and Strategies at Four-Year and Two-Year Institutions

On assessment:

• Do college student surveys have any validity? (not as much as you think) (Update: response from NSSE at InsideHigherEd)
  
• Where is the Student Voice in Assessment? (eportfolios use and misuse)

Educational Technology

• The 2009 21st-Century Campus Report:Defining the Vision (article about the report)

Pricing Higher Ed

My last post included a link to "Admission, Tuition, and Financial Aid Policies in the Market
for Higher Education" by Epple, Romano, and Sieg from 2003. In the paper, they test economic models against actual data and reach some very interesting conclusions about how pricing works. One of the assumptions is "In our model, colleges seek to maximize the quality of the educational
experience provided to their students."

I thought about this for a while. It's not obviously true, is it? I'm trying to remember how many meetings I've sat in where someone talked about the quality of educational experience. Of course, in many small ways programs, individual instructors, chairs, and so on do bits and pieces that impact this quality. And the SACS Quality Enhancement Plan is supposed to turn this into a visible project.

But by and large, I think most of my meeting time has been spent on solving problems, grinding away at the routine bureaucracy, or (once in a while) trying to make the bureaucracy work better. Of course, outcomes assessment is supposed to lead to continual improvements in the quality of education, but it would be a wonderful thing if board meetings were opened with the sentiment: we're here to improve the quality of educational experience.

As it turns out, I'm in the middle of a project to improve the "experience" part of that by helping organize strategic planning action items along those lines, and I'm going to start using that language.

In the article, the authors give some dependencies for quality:

1. peer ability of the student body
2. a measure of peer-student income diversity
3. instructional expenditures per student
   

Quality is relative, and two of the dependencies listed above are intuitive: students don't want to attend classes populated with students who are all less able than themselves. They also perceive the institution's ability to spend money in the classroom. This one is reflected in college rankings too (see "Zza's Best Liberal Arts Schools"), which probably has some affect on decisions. The second dependency, however, is surprising to me.

They see a distinct stratification that bestows economic benefits to the top schools:

Colleges at low and medium quality level have close substitutes in equilibrium and thus a limited amount of market power. Admission policies are largely driven by the “effective marginal costs” of educating students of differing abilities and incomes.

Colleges with high quality have more market power. These colleges do not face competition from higher-quality colleges. Hence, they can set tuitions above effective marginal costs and generate additional revenues that are used to enhance quality.

This suggests a Darwinian struggle for schools at the low and mid-levels of means and quality. In a catch-22, they lack the pricing power to enhance their position much. But once breaking through a ceiling, it becomes easier. At least that's my interpretation.

On the subject of price, the authors illuminate the second dependency (financial diversity):

We also find that colleges at all levels link tuition to student (household) income. Some of this pricing derives from the market power of each college. This allows colleges to extract additional revenues from students that are inframarginal consumers of a college. However, as noted above, our empirical findings suggest that market power of lower and middle ranked colleges is limited. This suggests that pricing by income may be driven by other causes.

I found an explanation of what an "inframarginal consumer" in another source "The inframarginal consumer is willing to pay more for the good than is the marginal consumer." So, if your college has a good market position, you can charge a premium. But the authors argue that that this isn't the whole story:

In this paper, we then also explore the role that income diversity measures play in determining college quality. Our findings here indicate that colleges and students believe that the quality of a student’s educational experience is enhanced by interacting with peers from diverse socioeconomic backgrounds.

Obviously there are many reasons for wanting a diverse student body, but the authors propose to actually use that as a factor that contributes to the price model. This begins to make more sense in Section 6 of the paper, where they verify empirically that college quality increases with income diversity, stating that "To attract students from lower-income backgrounds, colleges give financial aid that is inversely related to income as detailed below." While this is no doubt true for some institutions, others have a more directly self-interested reason for giving need-based aid: to increase enrollment in those students who couldn't otherwise afford to attend. I talked about the revenue-generating effect of this "gap filling" in "The Power of Discriminant Pricing."

Also in Section 6, they make an observation about college size:

Absent scale economies, peer effects and endowments create a force for colleges to reduce size to increase student quality–in the limit maximizing quality by admitting a handful of brilliant students and lavishing the entire endowment on educating those students. The countervailing effect of scale economies is captured in our cost function primarily by the c3 term in the cost function.

This outlines a good strategy for an elite school: keep it small because it's easier to maintain a high level of average student quality, but not so small that the economies of scale drive up costs unreasonably.

A hundred points of SAT is worth between $4688 and $10363 in merit aid (in 2003), according to the model output. The difference depends on what tier of college the applicant applies to.

Conclusions: First, remember I'm not an economist. But the paper is clearly written, and you can skip the mathy bits easily enough. The model presented has errors, as the authors describe, but the approach seems to lead to some insights, like the relationship between size and quality, the effect of financial diversity on institutional quality, and price sensitivity by student ability and income. I have not delved into all of these in my notes above. I don't know how hard it would be to simulate their model numerically to actually use it to build your policies (e.g. by running scenarios), but it's probably worth showing it to your IR office. And if you have an economics department handy, maybe they can shed some light as well.

404: Learning Outcomes

tl;dr Searched SACS reports for learning outcomes. Table of links, general observations, proposal to create a consortium to make public these reports.

In grad school there was a horror story that circulated about a friend of a friend of a cousin, who was a math grad student in algebra. He had created a beautiful theory with wonderful results, and was ready to submit when it was pointed out to him that his axioms were inconsistent--they contradicted one another. The punchline is that you can prove anything of the empty set. This sometimes also happens to degree programs that suddenly have to prove that they've been doing assessment loops, except in reverse: building grand theories from the empty set.

I complained the other day the there weren't many completed learning outcomes reports from universities to be found on the web. So when I noticed Mary Bold's post at Higher Ed Assessment "Reporting Assessment Results (Well): Pairing UNLV and OERL" I thought I'd hit paydirt. The hyperlink took me to a page at University of Nevada, Las Vegas with a link advertising "Student Learning Outcomes by College." Without futher ado, here's the page:

That's just too funny. There are, however, excerpts from actual results listed in the teacher education site, which you can find here. That site is the OERL that Mary refers to in her post.

It did make me think, however. There must be a bunch of SACS compliance certifications out there on the web now, and section 3.3.1 (used to be 3.4.1) covers learning outcomes. Want to see how your peers have handled it? The name of the school in the table below links to the compliance certification home page for that institution. For good measure I'll throw in 3.5.1, general education assessment, too. You're welcome.

Institution	3.3.1	3.5.1
Southeastern Louisiana University	link	link
Western Kentucky University	link	link
Berea College	link	link
The College of William and Mary	link	link
The University of Alabama in Huntsville	link	link
Nashville State Community College	link	link
Mitchell Community College	link	link
The University of Texas Arlington	link	link
University of New Orleans	link	link
Albany State University	link	link
Bevill State Community College	link	link
Louisiana State U. In Shreveport	link	link
Texas Tech University	link	link
Coker College	link	link

I did not try to make a complete list of all available reports. If you find a good one, send me the url and I'll add it. Here's my google search criterion.

Disclosure: I was the liaison, IE chair, webmaster, and editor for the Coker College process (as well as doing the IR and a bunch of other stuff--no wonder I have white hair). The linked documents to that one are turned off for confidentiality, but you can find the complete list of program learning outcomes plans and results here.

Observations:
First, hats off to all the institutions who make these reports public. This is a great resource to anyone else going through the process.

I only scanned through the reports, looking for evidence of learning outcomes. I probably missed a lot, so take my remarks with a grain of salt--go look for yourself and leave a comment if you find something interesting. It should go without saying that in order to be helpful, this has to be a constructive dialogue.

For learning outcomes I didn't find as much evidence-based action as I would have expected from all the emphasis that SACS puts on it. My own experience was that programs were uneven in their application of what is now 3.3.1 (at the time SACS didn't even have section numbers for the requirements--how crazy is that? I invented my own, and then they published official ones just before we had to turn the thing in.). So there was a lot of taking the empty set and trying to build something out of it. That can take various forms, which one notices in scanning certification reports:

• Quick fixes: use a standardized instrument like MAAP, MFAT, CLA, NSSE. Of course, it's not really that quick since it would take at least a year to get results, analyze them, and use them. The conceptual problem is tying results to the curriculum (except for MFAT).
  
  
• Use coursework: passing a certain course certifies students in X (e.g. use of technology), passing a capstone course with grade X ensures broad learning outcomes. This is fairly convincing as gate keeping, but hard to link to changes unless specific learning outcomes are assessed.
  
  
• Rubric-like reporting. Okay, I'm not a big fan of rubrics when employed with religious zeal and utter faith in their validity. But I have to admit that the most convincing report summary I saw on learning outcomes was the one below from Mitchell Community College. Not all the data points are there, but that's realistic. Take a look.
  

Of course, this still has to be tied to some analysis and action to get full points, but the presentation of the learning outcomes is clear and understandable. In general, that was somewhat of a rarity in my cursory review. What there is a LOT of is plans, new plans, minutes describing the construction of new plans and goals, assessment forms, models and processes, and generally new ambitions and enthusiasms. There are standardized test reports like CLA summaries, which solve the data and report problem, but don't touch the hard part: relating it to the practice of teaching in a discipline.

I believe that if our efforts as assessment leaders are to be maximally useful, we have to make the annual, messy, incomplete, inconsistent, but authentic program-level plans and results available to the public. This would encourage us to adopt some kind of uniformity in reporting, and improve the quality of the presentation (maybe I'm a fool for saying that). The only downside is that if we're honest, there will be empty sets here and there--programs that have not been dragged into the 21st century yet. But transparency can help there too, perhaps into shaming some into compliance. Just imagine (really dreaming now) if the quality of the reports were good enough to use for recruiting and paint across the program web page.

The Voluntary System of Accountability tries to do something like that. Unfortunately, that group seems to be enamored of standardized tests for learning outcomes. There's a validity study they just published here that you can consider. This post isn't the place to go into all the reasons I think standardized testing is the wrong approach, so let me just leave it at that.

Thinking more positively: is there any interest out there to form a loose consortium of schools that report out annual learning outcomes for programs? The role of the consortium could be to settle on some standard ways of reporting and defining best practices?