The Most Important Problem in Higher Education

Arrow's Impossibility Theorem is a fascinating application of mathematics to social science. Arrow took up the problem of how to find a voting system that meets certain reasonable criteria, such as taking into account the preferences of more than one voter. He showed that not all of the criteria can be met simultaneously in a single system because it creates a logical contradiction.

I have thought that it would be interesting to try the same trick with higher education. Informally, we might think of a list of best-case qualities we would wish for an educational system, such as universal access, a mechanism for cost deferral (like loans), public subsidization at a given level, and so on. In these lists, which I scribble on napkins at diners, the hardest quality to come to grips with is the idea of certification: a way of knowing that a particular student showed a certain level of accomplishment.

The idea of accomplishment is subtle. Here are at least three ways of looking at it:

1. We might be interested in what a person can do in the future, as a prospective employer would. Can Tatiana synthesize organic compounds? In this sense, accomplishment has to do with the predictive validity of inducing future performance from past performance. Sports statistics embody this idea: a batting average in baseball, for example.
   
2. Another effect of accomplishment is the usefulness of experience in being a consultant. If Tatiana swam the English Channel back in 1981, she might not be able to do it again today, but she could probably tell you some important things to know about it.
   
3. A third effect is social standing. Accomplishment brings its own rewards in terms of access to more connected people, the media, and so on. 

All of these effects usually derive from indicators of direct accomplishment. This isn't always the case, of course. Some people are famous just for being famous. This latter phenomenon should be seen as deleterious because it dilutes and obfuscates the real value of accomplishment (perhaps related to Gresham's Law?).

This last issue is confounded by the fact that accomplishments are often judged subjectively. Someone might think the screenplay I wrote is wonderful, but that would be a minority view. We can disagree about subjective assessments, so 'accomplishment' itself can be very fuzzy. Even in the case where some putative objective fact is presented (e.g. swimming the English Channel), there is room for debate about the significance of that accomplishment, and what it means in terms of the effects listed above. The common denominator is to keep track of evidence of the accomplishment itself (we would call it authentic assessment data in higher ed land), and let the endorsements and comments ebb and flow with the tides. So if Lincoln's Gettysburg Address receives poor reviews the day after, we still have it around 150 years later to review for ourselves. The historiography of accomplishments is almost as interesting as the things themselves.

Existing systems for tracking accomplishment are inadequate to the task. Let's look at some of them.

In higher education, we issue diplomas and certificates, sometimes with decorations like sum laude, and identifying an area of study. The reputation of the issuing institution lends itself to the holder of the degree, and some subject areas are worth more than others. This is rather like a guild system, a gate-keeper approach. Some of the problems are:

• It doesn't allow for autodidacts who learn outside the system, and so demands a significant investment in time and money.
• There isn't much 'partial credit': completing 99% of requirements does not translate to 99% accomplishment since you don't get a diploma.
• It's too coarse-grained: we need more information than "John got an engineering degree from State U," or even transcripts.
• The lending of reputation from a school's name to the individual distorts the actual accomplishment of the individual. 
• It's very expensive and time-consuming to become certified.

At higher levels, the choice of a dissertation advisor can matter greatly, which leads to a second sort of certification: the letter of recommendation and other ways of personally endorsing someone. Problems with this approach include:

• There is no limit to how many letters one can write.
• There is incentive to be overly generous in letters you do write, or at least there is a disincentive to be negative.
• It's very difficult to compare letters from different endorsers since they are not standardized.

Self-certification using a resume to document or summarize one's accomplishments are unsatisfactory because they have to be checked for accuracy, leading to one of the other forms of tracking. There are often significant penalties for getting caught in outright fabrication, but the problem of verifying the truth of claims remains difficult and time-consuming.

If we were interested only in predictive validity of statements like "Tatiana can speak French fluently," then the ideal case would be to have a perfect testing system to ascertain proficiency only in areas of interest. This is the Western Governors University approach, and it also exists in the form of professional board exams. However, these are coarse-grained hurdles that have to be overcome to get credit for a course or enter a profession, and the skills actually needed for a job may have only a passing resemblance to the test. For example, there are many, many types of engineering jobs, and the board exams cannot possibly cover the skill sets in that level of detail. This results in a rather arbitrary, and certainly inefficient, standard of accomplishment.

What are some requirements for an accomplishment tracking system?
Here's a starter list.

1. Identity is valid. We have to be confident that the accomplishments weren't outsourced or bought on eBay and then the credit transferred to someone else.
2. The system has to be open and transparent, so that we don't create a bottleneck that is just another gatekeeper.
3. The system has to be self-correcting with respect to inflation of accomplishments and outright errors.
4. Individuals have the final say over their own accomplishment profiles.

We can probably think of more requirements, but this is enough to begin with. The identity problem can really only be solved with personal contact. That is, a person has to be convinced that some other person did something. No amount of biometrics is ever going to solve that problem, I predict.

The system or systems should be web-based and easily accessed. There shouldn't be fees or artificial walls to viewing profiles. Of course, per number four above, profiles should be able to be restricted by their owners, like current social media sites sort of allow. It also implies that there is no negative information about an individual, only positive. This seems like a drawback, and I might be wrong about it, but I think allowing negative feedback (which necessarily allows others to say things about you publicly that you don't like) is fraught with problems.

So one possibility is a lightweight endorsement system that works something like the science pre-print site arxiv.org mashed up with Facebook. Well, like academia.edu, come to think of it.

1. I do some work that I think shows off my skills and upload a record of it to the system under my profile. This would necessarily be electronic, but could consist of video or any other kind of media--not just papers.
2. A portfolio-like central index keeps track of all my stuff, with pointers and meta-data about the works I have listed in my resume. 
3. The index system also allows for endorsements by other registered users. Everyone's ID is real, verified by credit cards and phone numbers or something (optionally a unique SSL certificate). 
4. Endorsements from experts would claim to authenticate that this is indeed my work, and say something appropriate about it that demonstrates a connection with the work itself.
5. All of this is standardized to the extent possible, and if the user allows it, fed out to a data export.

The result could look something like LinkedIn's recommendation system mixed up with Academia.edu's portfolio/vita.

On the surface, it seems like reciprocal recommendations would be a problem, as it tends to be in journals (citing each other's papers or including as co-authors). But with transparency, I suspect that clever people would create metrics that would attempt to summarize the evidence and connections that exist in the body of work. By tracing links of recommendations to create a network of association, it ought to be possible to see how worthwhile they are. It would be far from perfect, of course. But the possibility exists for a real "market" of reputation, where one's own standing is linked to those whom one has endorsed. It would be like picking stocks. In this way, there is an incentive for high-standing individuals to endorse newcomers who show promise. It probably doesn't prevent fads from distorting the 'reputation market', but if there is a relationship to the real world, eventually it becomes self-correcting. (This doesn't apply to all fields, and that problem may be unsolvable there simply because the discipline is more or less defined by the fads that exist at that time.)

The beauty of a transparent system is that a company or university wanting to evaluate a person's record could either use an off-the-shelf metric provided by one of these hypothetical companies that would spring up, or they could just look at the evidence themselves. Or they could hire their own experts to look at the portfolios. There is a sliding scale between how quick the evaluation is and how customized it is, and the best solution for one circumstance may be different from another. I assume this would be followed by an interview, which could be very substantive and deal directly with the evidence of performance that pertains to the position.

A good test case would be teaching evaluation.  I haven't seen a public teaching portfolio. Maybe I just haven't come across one, but it's never occurred to me to post mine either, until now. I have big binders with student projects I've supervised. Why not post the best of those?  The creation of a teaching-record portal would be a real service to higher education to standardize and validate that important part of academia.

Although I talked about 'a system', it would almost have to be many systems, each associated with some professional endeavor. This still allows for a resume to link all of the pieces together if necessary. In education, learning outcomes assessments would be a necessary component.

The rewards for figuring out how to accurately and efficiently certify accomplishment are worth the trouble. Education systems could be revolutionized to be more flexible and transparent, less bureaucratic and guild-like. Someone who teaches herself would not be valued less than someone who learned at a high-priced university--the proof would be in the sweet dairy desert dish, as the saying goes.