Struggling with the Idea of Critical Thinking

I came across an excellent article (pdf) in Psychological Science by Daniel T. Willingham entitled "Critical Thinking" and subtitled Why Is It So Hard to Teach? The author first presents a case for the teaching of critical thinking as an integral part of the nation's educational product. This skill is defined "in layperson's terms" as

[S]eeing both sides of an issue, being open to new evidence that disconfirms your ideas, reasoning dispassionately, demanding that claims be backed by evidence, deducing and inferring conclusions from available facts, solving problems, and so forth.

It is also noted that there are particular kinds of critical thinking linked to academic pursuits, such as history or science. Here's an example of literary criticism (a review, in fact, so it's meta-criticism), chosen at random from Critical Inquiry (Spring 2003, pg 463):

[The author] demonstrates a thorough historical awareness of the obscene cruelty of the Stalinist and Fascist regimes and, in fact, suggests that the profoundly stupid cruelty of the totalitarian regimes is more theoretically pertinent than theory itself in the dark illumination it casts on the institutions of the West, where the term totalitarian is often used, in a way that to [the author] suggests gross complacency, emplifies the illusion of freedom and agency in contemporary democracy, illusions unavailable in non-democratic societies, where the truth is confronted more directly in the from of a short circuit between authority and brutality.

There are many other examples from a variety of disciplines that we could choose, and they will look different from one another. What is this "critical thinking" stuff that unites them? By the way, I'm not sure "emplifies" is really a word, but that's part of the fun.

Defining the process of critical thinking takes up a good bit of Willingham's piece. He presents the idea that there is a difference between superficial and deep structure, and that humans naturally cling to surface structure. Through training we can see through the surface structure and generalize problem solving. An interesting example is given.

A treasure hunter is going to explore a cave up on a hill near a beach. He suspected there might be many paths inside the cave so he was afraid he might get lost. Obviously, he did not have a map of the cave; all he had with him were some common items such as a flashlight and a bag. What could he do to make sure he did not get lost trying to get back out of the cave later?

The "correct" solution is to put some sand in the bag to use as markers within the cave, to be used to retrace the route back. What's interesting is that when this problem was posed to American and Chinese college students, 75% of the former latched onto the solution, but only 25% of the latter. The hypothesis is that this problem has the same deep structure as that of the story of Hansel and Gretel, which is common to American culture but not Chinese. That is, because of prior exposure, American students could see through the distracting surface structure (beach, cave, etc.) to see the underlying problem (the need to leave a trail behind oneself).

Even if one has been trained in deep structure, like mathematical methods of solving problems that may all look the same, but are really the same logically, one must also be trained to use this knowledge, argues the author--we have to know when to look for deep structure. The reverse is true also--even if one knows to look for deep structure, one must know how to do it. Knowledge of content and methods of inquiry are both prerequisites. But are they sufficient?

Critical thinking is not a skill at all is the hypothesis advanced later in the article, after a summary and discussion of the difficulties and general failure to prove that teaching critical thinking per se has been successful.

Critical thinking does not have certain characteristics normally associated
with skills—in particular, being able to use that skill at any time. If I told you that I learned to read music, for example, you would expect, correctly, that I could use my new skill (i.e., read music) whenever I wanted. But critical thinking is very different. [...] [P]eople can engage in some types of critical thinking without training, but even with extensive training, they will sometimes fail to think critically.

Here I will take my departure from the article and offer my own thoughts. I think the idea presented in the quote above is absolutely on the money. To illuminate, consider the following example. An employee works on a project for his supervisor, and presents the results. The latter reviews the work carefully and pronounces "you did a good job." This is obviously a desired outcome for the employee. Is it something we can teach our students: to do a "good job"? No. The reason is that this is a description of the outcome, and is not directly related to the process of attaining it. Note also that agreeing that the description is accurate does not give us license to assume that we can measure the accomplishment with some metric. This assumption is, unfortunately, blithely made very often in academic assessment circles.

We can understand this problem better by looking at its deep structure. Some abstract problems have very difficult solutions, which are nevertheless easily confirmed once found. Suppose I set up a row of cups numbered 1-1000 and secretely put a pea under one of them. Your job is to figure out where the pea is by turning over cups and looking. Generally speaking, you'll have to turn over 500 cups on average to find it. There are no shortcuts. But once it's found, verification is obvious. You'll know instantly that you did a "good job" if you find the legume on the second try. In computer science this phenomenon is well known, and even famous. An outstanding example is the P=NP problem.

Hard to construct, easy to verify describes many of the outcomes we observe in our professional lives. That's why experience and learned judgment are invaluable in most, if not all, professions. If we accept that critical thinking should not be thought of as a skill that can be taught but rather a judgment about efforts made, then how can we proceed? What can be taught?

I've argued before that critical thinking is a fuzzy concept, so maybe my reading of this article is just confirmation bias on my part, but that won't stop me from presenting my argument anew, framed in the current context.

The idea that knowledge and skills are both required to develop cognitive skills was a point mentioned earlier. But we can be more specific without becoming pedantic or falling into a mereological fallacy. Knowledge and algorithmic skills can be lumped together into a basket called analytical thinking, or if you prefer, deductive reasoning. Knowing facts is the simplest kind of deductive reasoning, but it also includes analyzing chemical processes according to given rules, solving all kinds of math problems, checking grammar, and so on. We spend a lot of time in education trying to convey this kind of knowledge.

Underappreciated, I believe is the complementary cognitive skill of inductive reasoning. During a college curriculum, this type of thinking sneaks up on a student, and he or she may fail simply because of the lack of warning. I think instructors mostly aren't aware themselves when they cross the threshold from analytical/deductive to what I'll call creative/inductive.

Creativity and induction are both generalizing operations: the creation of new knowledge for the learner. When I was in grade school, working with decimals and fractions for the first time, I remember creating the following hypothesis after working enough problems: the reciprocal of an integer n near 10 is about .10 - (n - 10)/100. This looks complicated, but in practice, if I wanted to remember what 1/13 was, I'd take the '3' from 13, turn it into .03 and subtract from one tenth to get .07. The actual reciprocal is .077, so it's not a bad approximation. On the whole, this is a lousy way to get the reciprocals, though, because it only works in a narrow range. My brain is just wired to remember processes rather than facts, so it stuck.

The point of this numerical digression is to illustrate the creative/inductive thought process. It's messy, often inaccurate, and a lot of work to get right. Unlike the step-by-step process of teaching an analytical/deductive skill, there are false starts and inevitable setbacks. There's not even a guarantee that there is a solution. This takes a whole different outlook on the part of the problem-solver. Students who are used to the analytical process, and who suddenly encounter the trial-and-error necessities of a creative process are often stymied. But I think it can be taught, and the first step is to recognize the problem and address it.

Unlike the fuzzy label "critical thinking," which arguably is not a skill at all, but an outcome of skills and knowledge, both analytical/deductive and creative/inductive skills can be taught and assessed. We just have to do so intentionally. Critical thinking as a skill to be taught and assessed is a meme that needs to go away... The sooner the better.

Dilbert Fired

In case you haven't heard, Dilbert (the hapless cubical slave with the pointy-haired boss) was fired for developing his own internet business on company time and is living off his investments, such as they are. The strips are here.

My favorite one:

Admitting the Right Students

All of us who teach like to have students in the classroom who want to learn. In a recent conversation this topic came up. The math teacher I was speaking with said he'd take a hard-working curious student over an indifferent student any day--regardless of academic preparation of the latter. I would too. The real reward of teaching, after all, is to see growth. One of my proudest (shared) accomplishments is that one of my MAT 100 (really a remedial course) students fell in love with the topic and went on to graduate with a BA in mathematics. She now works in a technical profession and is very successful.

All this by way of introduction to my topic: how to judge applicants for admissions purposes? Academic preparation isn't enough if we care about things like motivation and perseverance. I've thought about and written about different angles of this topic for a long time, and now find myself in a position to do something about it.

The conventional wisdom, if there is such a thing, is that grades and SAT are reasonable predictors of achievement. These are called cognitive measures for reasons best known to the psychometricians. As I recall high school, there was a lot more to grades than cognition, but never mind. SAT is a favorite target for those who don't like this narrow thinking, and I'd count myself with the critics. I took the ACT in 1980 in Illinois, and remember liking standardized tests then. I think I got a 28 on the ACT, whatever that means. I also took the ASVAB military battery (in the sense of tests, not artillery), and actually found the results last summer when I was going through stuff in the garage. I remembered taking the test, driving to a National Guard armory in East Saint Louis with my friend Mark, following the lines marked on the floor to the various stations. One was an attitudes and behavior survey, where they discovered that I'd never smoked pot. They made fun of me for that, or else it was some kind of awe. This was 1978, and the stuff was everywhere. One station was the ASVAB, complete with number 2 pencils.

In the Army's eyes, I was being rated for potential by this test, and they took it very seriously. Frankly, I loved taking standardized tests because I always did reasonably well on them, and hence had no stress about the results. What's interesting about the ASVAB is the one domain where I did not do well at all. You can see on the image that there's a 55% bar in the middle. When I pulled this out of the box in the garage last summer I had to squint to read the faded type of the explanation. If you look on the far left, there's a CL designator: that's for "clerical work." The battery decided I'm no good at it. If I recall correctly, this part of the test included such things as counting how many Cs were in a line of Os. Something like this:

OOOOOOOCOOOOOOOOOOCOOOOOOOOCOOOOOOOOOOOOOOOOOOOOOOOOO

That's the sort of thing that drives me crazy. I will tip my hat to the test designers: I am in fact not well suited for mindless clerical work. Did the rest of the battery rate me accurately? I guess that depends on what the expected outcomes were. I wasn't at that time very good leadership potential, was a bit lazy academically from never having had to work hard in school, and didn't have a lot of ambition to go change the world. Yes, I was only 16 or so, but looking at the test results would still have vastly overestimated what kind of officer I would have made at that point. In short, there were a lot of imporant things that the test didn't measure. This anecdote underlines my main thesis about standardized tests: they are good for simple tasks but not complex ones. And there are many complexities to what makes a succesful college graduate. The specifications are likely different from institution to institution as well.

All this by way of introduction to an article I found while researching the link between SAT and race. The article from InsideHigherEd by Scott Jaschik dates from September 2008, and opens with:

[C]ritics of standardized testing — and especially of the SAT — have said that these examinations fail to capture important qualities [...]

What's interesting is that the College Board--creator of the SAT--agrees, and has an active research program for developing new approaches. This is good news and bad.

First it's good, because the old SAT "cognitive test" will lose some of its halo and the market place for talent can become more efficient. This is good for all of us. It's also good because it will put standardized testing--the modern day equivalent of phrenology, in my opinion--under more of a microscope. If policymakers have more sophisticated ways to think about achievement, it's beneficial to everyone.

It's also bad in a very selfish sense. This is because the institutions that are already acting on this market inefficiency will see their lunch being shared around the table. The article mentions a couple of these. Tufts University and Oregon State University are using non-traditional approaches to admissions according to the article. Of course, I'm not really serious about this--I'm very happy that there's competition for the meme that SAT is a real achievement score, and I'm confident that a real research program can keep us on the cutting edge of finding the most suitable students for our institution. Ultimately it's good to have competition, and for reasons outlined below, I think each institution will have to find its own solution anyway.

So what are the new methods looking for besides "cognitive" processes? The Group for Research and Assessment of Student Potential (GRASP) has twelve so-called dimensions they consider:

1. Knowledge, learning, mastery of general principles
2. Continuous learning, intellectual interest and curiosity
3. Artistic cultural appreciation and curiosity
4. Multicultural tolerance and appreciation
5. Leadership
6. Interpersonal skills
7. Social responsibility, citizenship and involvement
8. Physical and psychological health
9. Career orientation
10. Adaptability and life skills
11. Perseverance
12. Ethics and integrity

You can read descriptions of these by following the link to their site. In addition to GRASP, SAT is working on modifications to their instrument, which may result in a whole new test. I'm a bit dubious about this project, however.

There are already ways to game the SAT. How much more will this be true when the 'right' answers are clearer? That is, it's much easier to appear to have attractive behaviors and attitudes than it is to actually possess them. I can imagine a new preparation industry springing up to coach test-takers who can afford it. Ultimately I think the industrialization of this metric is doomed for this reason. That leaves individual admissions policies to find ways to gather information in ways that are less likely to be faked. The article makes the same point.

The results of SAT's trials with their new test items are interesting. By de-emphasizing academics in favor of the College Board's experimental "biodata" and "situational judgment", traditionally under-served minorities showed significant increases in enrollment. Particularly notable is an almost 6% increase in black enrollments at the highly selective institutions that participated in the experiment.

There are problems, like gaming the system, but the payoff is worth it for individual institutions. The article quotes Pamela T. Horne, who has been working with GRASP results. She gives voice to my feelings on the topic:

“This is mission-driven,” she said, noting that colleges don’t define their missions as “enroll students with high SAT scores,” but they do prize leadership, artistic vision and various other qualities that might now be measured.

Unfortunately, if you asked many college presidents and board members, they probably would say that high SAT scores are an institutional priority. Why else would there be all the concern about which scores are reported (first, last, all, average?). Ultimately, more enlightened institutions can take advantage of the limitations of the SAT and other standardized predictors by individualizing their own processes. It's an exciting challenge.

Update: Reading the comments on the article, I found a reference to a textbook on the subject of non-cognitive assessment. Here's the link on Amazon.com. Beyond the Big Test: Noncognitive Assessment in Higher Education (Jossey Bass Higher and Adult Education Series)
by William E. Sedlacek (Feb 26, 2004)

Endowments and Unpronouncable Laws

"Fortunes Falling" is the title of an Insidehighered article chronicling the rapid unwinding of university endowments. As the market value of investments tumbled, so did the payouts for institutional uses like financial aid. Even with a three-year trailing average (the standard), payout amounts will dip appreciably. The common fund is used by many colleges. Their recent study jointly with NACUBO shows that for university endowments generally:

Institutions and their affiliated foundations reported an average rate of return of -3 percent for the fiscal year ending June 30, 2008, and the results of the Follow-up Survey show that endowments fell an additional 22 percent in the first five months of FY09.

In fact, the tables in the report show that across all sizes and types of endowment, the returns were close to -22% from July 1 to November 30, 2008. Of the surveyed institutions, 27% planned to decrease draws, and 1% increase. The rest were status-quo or unknown. Interestingly, the survey was for amounts, not percentages. This must be because policies calculating the annual draw fixed the number before October. Next year the trailing average will come into play. For most, it looks like this would reduce payout by one third of the 22% drop, or about 7%. This obviously means more for institutions that depend heavily on large endowments.

Davidson College is one of those that requires a hefty endowment draw to keep the lights on. And it has the misfortune to be in a state that has not (yet) passed a new law governing endowment draws called UPMIFA, which updates UMIFA (this is the unpronounceable bit--try it). The Uniform Management of Institutional Funds Act, dating from 1972, prevents funds from selling off parts of the corpus of the endowment to raise cash if that would cause the total amount left to dip below its original donated value. The act is quite readable, despite using vocabulary like "eleemosynary." The new act inserts the word "prudent" between "uniform" and "management", and allows the management of the fund to be less prudent (!) but more beneficial to the holder. Specifically, it allows the fund to be drawn down even if it's under water.

With the precipitous drop in market values, the 1972 shift from fixed income endowments to higher return (and higher risk) instruments led to some good years for endowments. But the law only really works when the balloon is going up. Just like the real estate market, bad things happen when values quickly. So North Carolina and the handful of other states that have not yet passed UPMIFA are not good places to have endowments that you depend on in thick and thin. One can imagine that some hard lobbying is going on in NC. According to a notification from NC Center for Nonprofits, it is pushing to have the law passed early in the 2009 session.

Even the new law is no panacea. It requires the institutions spend "prudently" even if it allows dipping into the original corpus of the endowment. What exactly this means will lead to some interesting conversations. One thing is for sure: the conventional wisdom that the stock market is the best place for such investments has surely been shaken. There are risks and there are risks. The risk of staying out of the market is that one has to accept less than optimal returns in the good times. But full investment in even a "balanced" portfolio of equities means taking it on the chin during some part of the business cycle. In some ways this is no different from the dilemma faced by the average investor, but if the university payroll depends on the endowment, the stakes are much higher. If you have to let good people go because you can no longer afford them, that damage takes a long time to repair.

Visualizing Enrollment Change with Class Trajectories

Visualizations like graphs and animations are of obvious value in presenting data. One of the more useful ones I've found for visualizing enrollment is what I call a class trajectory graph, an example of which is shown below (with made-up data).

It shows three classes for four-year programs entering in fall 2000, fall 2001, and fall 2002. The right-most point on each of the trajectories is the first year for those students, and shows enrollment and net revenue per student (tuition minus institutional aid). Multiplying those two numbers gives net revenue for the whole class, which can be compared to the level curves graphed. So one can see that the freshman year for the 2000 entering class netted the institution around 9 million dollars. As time goes on, the classes get smaller because of attrition, so the next point on the blue line is to the left. Also note that net revenue per student fell somewhat. This is normal as students with leaner aid packages attrit at a higher rate. In the second year, the 2000 entering class netted the institution a little less than $7.5M. The entering 2001 class was bigger and produced more revenue per student, so the right-most pink point is up and to the right. It generated about 12 million its first year.

These trajectory graphs quickly convey important information. How bad is attrition, and what does it cost us in net revenue? How big are the classes comparatively? What are overall trends in class sizes--useful for planning everything from budgets to residence halls to class offerings. This is an essential graph for your strategic dashboard.

Bonus. Here are two other two visualizations just for fun. The first one [here] shows you how zip codes are clustered. Could be useful for something, I suppose, but mainly neat to play with.

The second [here] is just plain (or plane in this case) cool. The video shows world-wide flights over 24 hours.

The Committee Driven Life

It occurred to me yesterday while watching presentations throughout a half-day board meeting that one could spend all of one's energy preparing for and attending meetings. I have already written about The Secret Life of Committees and the jinxed number of eight members, but we can go further. We might think of committee work as like a peat bog, where one could sink in, become immobilized, and slowly mummify, perhaps to be found perfectly preserved a thousand years from now. The future archaeologists would marvel at the perfectly bulleted agenda and still-legible inked initials of the committee secretary on the peaty minutes.

I propose the following principles to achieve this aim, this committee-driven life.

• The committee needs you. Never turn down a chance to serve on the most obscure ad hocity that crosses your path. To do so is to counter your purpose.
  
  
• The committee is infallible. Without this tenent comes doubt. Not only must you believe, but you must also proselytize to your colleagues. Burning heretics at the stake should only done if no one is there to put it on Youtube.
  
  
• Minutes are inspired by the committee. There are those who will doubt that the minutes are the true word of the committee, and will seek to modify them, even translate them into other languages. This must not be tolerated.
  
  
• The committee has no purpose other than to be the committee. This may be the hardest tenet to absorb, since committees are often given names that give rise to the implication that they have a purpose external to themselves. For example, the uninformed or unbelieving may think that the Curriculum Committee has a purpose that relates to maintaining the curriculum. This is false. The committee may speak to and make reference to the profane world outside its walls, but its one true purpose is to be. The committee is that it is.

By following these guidelines, you too can have an inspired and purposeful life. That is, if your purpose is to be discovered in a millennium, raised from the strata of bureaucracies past to give glory to the one goal worthy of a life's dedication: sitting around a table with a bunch of bored people over a sheaf of handouts no one read, complaining about the parking situation.

Reading Rise

In this recent post, I converged on a rather glum picture of the future where readers (as opposed to screen viewers) were rare and valued. An article in this week's The Economist takes up that debate, quoting statistics from the National Endowment for the Arts (NEA) that indicate that reading is actually on the rise, with dramatic improvements in some quarters.

The research article from the NEA is "Reading on the Rise" (pdf). Quoting from The Economist:

As the NEA's research director, Sunil Iyengar, points out, almost every demographic and ethnic group seems to be reading more.

The question posed is whether an adult has read a novel, short story, poem, or play in the last 12 months. African-Americans were up 15% from 2002, and Hispanics 20%. Young men (18-24) responded positively 24% more often in 2008 than in 2002.

Still negative is the nation's illiteracy rate; it still stands at 21%. The Economist notes that this is one of the worst percentages among developed nations.

The Price of Philosophy

Education is a funny business. Despite all the efforts made in assessing learning outcomes, it may well be that the net effect of educational experiences is not realized until much later in a person's life. Some things that we think are important may actually have little or no effect. This is especially true when considering the more philosophical aspects of higher education. Yesterday I mused about the value of a liberal education. It's an easy target for the practical-minded cost trimmer because the results are presumably not truly felt until years after the experience. And yet the budget executives have to be dealt with, especially in these lean times. They may even be right some of the time.

I've sat in a lot of budget meetings with both administrators and faculty, and an inevitable collision is that between greenbacks and philosophy. At these moments I usually make a mental note: We may not know the worth of philosophical goals, but we can often evaluate the cost. In lean times this skews the decision against philosophy and toward saving money. In fat times, the opposite is true. Some examples will help understand what I mean.

How many books is enough in the library? Is a cataloger in Sanskrit necessary and contributing toward educational outcomes? What about restrictive policies for course transfers, under the theory that courses at other institutions aren't as good as your own? This costs the institution enrollment with nebulous effect on students who do come. Whole departments like foreign languages get their raison d'etre questioned as well.

Like Pythagoras's philosophy of not eating beans, the items in the list above have economic impact (the great philosopher and mathematician is supposed to have died rather than escape his enemies through a bean field). Evaluating the actual worth relative to the cost is not something that can usually be done scientifically--this is where leadership is required. But it helps in these discussions to have the sensitivity to know when the discussion has wandered into philosophical territory.

Occasionally, hard data can come to the rescue, but this is probably an exception. One IR director related to me that the faculty at his institution were unhappy about accepting AP courses instead of their own prerequisite classes. They were on the verge of setting a new policy to ban AP course credit, which would have had a negative impact on admissions. He did some research and found that AP students actually did better in the subsequent classes than home grown ones (it could be that they were better students, of course). This proved to be the case in all instances except one subject area. When they investigated further, they discovered that the AP course content did not match that of the prerequisite course very well. I like this story because it shows what IR can do when the evidence is there. The harder decisions are unfortunately more common: placing a value on Sanskrit cataloging, for example.

More poetically, one might say

अमंत्रमक्षरं नास्ति नास्ति मूलमनौषधम्‌।
अयोग्यः पुरुषो नास्ति योजकस्तत्र दुर्लभः॥

[source and translation]

The Younging of Higher Education

It was in some book on evolutionary biology--probably Stephen J. Gould--that I came across the interesting idea that the descent of man (in Darwin's sense) was accompanied or even caused by the delayed onset of maturation. In other words, humans are curious and not as aggressive as "wild" animals because we experience an extended childhood, in which we learn the ways of the world. As explained in Human Evolution Through Developmental Change by Nancy Minugh-Purvis and Ken McNamara:

The essential link between human life history and the human way of life has been the suggestion that a prolonged period of infant and childhood dependency upon caretaking adults allows humans a prolonged period of behavioral plasticity and leads to a reliance on learning and learned behavior as hallmarks of human existence. [source]

This idea has been around for a while. It has been hypothesized that dogs are "slowed down" wolves--bred to stay in a puppy-like state and hence friendly and inquisitive. Aldus Huxley used this idea in Crome Yellow as a plot device.

I'd like to call this idea "younging." Over the years, Mickey Mouse has younged. And I would like to present the idea for consideration that our culture and particularly educational system are younging. In this case, it's not a good thing. Children remain malleable and learn new things rapidly. But this is of no use unless someone is instructing them in language, practical skills, and so forth. Growing up slowly is not an evolutionary advantage unless you have parents around to take care of you. The culture as a whole has no parents, if I can push the metaphor. As it youngs, rather than advancing in sophistication, it might just sit on the couch and watch Spongebob. The genesis for this idea was "The Last Professor" by Stanley Fish in the New York Times today--a reflection on recent trends in higher education. Professor Fish is a guy I usually disagree with, but he always has interesting things to say. This piece is about the evaporation of the liberal arts in higher education, particularly the soft humanities (history, literature, ...). He argues that these disciplines are seeing full time tenured professorships being replaced by adjuncts, and that in the competition between traditional and for-profit schools, liberal arts has no chance to survive. From the article:

What is happening in traditional universities where the ethos of the liberal arts is still given lip service is the forthright policy of for-profit universities, which make no pretense of valuing what used to be called the “higher learning.” John Sperling, founder of the group that gave us Phoenix University, is refreshingly blunt: “Coming here is not a rite of passage. We are not trying to develop value systems or go in for that ‘expand their minds’” nonsense.

The for-profit university is the logical end of a shift from a model of education centered in an individual professor who delivers insight and inspiration to a model that begins and ends with the imperative to deliver the information and skills necessary to gain employment.

This is arguably the younging of education in the following sense: the market favors instant gratification over development of long-term advantages. In the long term, philosophy majors will do better than advertising majors (reference: WSJ), but in a younged generation, short term goals are more attractive. This is related to my recent post on viewing vs. reading, I believe--those who read more may become the last bastion of the liberal-arts demographic (and end up ruling the world, one is tempted to add).

Seneca was a successful guy. I can forgive him for writing off mathematics as a mere trade; he didn't know about abstract algebra. His advice in his letter on liberal arts still makes good reading, despite the antiquity.

Thus, whatever phase of things human and divine you have apprehended, you will be wearied by the vast number of things to be answered and things to be learned. And in order that these manifold and mighty subjects may have free entertainment in your soul, you must remove therefrom all superfluous things. Virtue will not surrender herself to these narrow bounds of ours; a great subject needs wide space in which to move.

Notice that "superfluous things" would include skills we would consider the practical parts of learning--that which is immediately useful. Despite his immense wealth, or because of it, he can write "I respect no study, and deem no study good, which results in money-making."

Like a child learning to read, the payoff comes later. A picture book provides immediate gratification, and learning vocabulary and pronunciation is hard work. Contrary to what Mr. Sperling advertises, there is worth in learning broadly and encountering varied modes of thought. Although market forces may young the academy, I predict that it's those who think like Seneca who will be making the rules in the next generation.

Crowdsourcing Assessment

I had a brainstorm today as I was driving home today from my former home, which is now known as "the endless list of fixing things." I'd installed a new toilet which refused to take its water gracefully despite my best efforts with crescent wrench and exotic vocabulary. To take my mind off of this vexation I was pondering the mystery of non-computability when I wondered into the mental ditch of educational assessment. Actually in this case, pre-educational assessment might be a better word, since I'm trying to figure out how to better predict student success during the application process. This idea has seen some print here before quite recently.

In my discussion of this idea with a colleague, he suggested using the personal essay that accompanies the application to try to identify traits of successful (future) students. At present the essay is more or less optional, but we could change that easily enough. But in order to be useful, we'd have to generate some kind of rating from the essay. This would then be fodder for a regression analysis on first year grades or retention. Who would do this rating work? A committee is one way, but if you've read my recent article on the beasts, you'd guess it wouldn't be my first choice. Then it occurred to me that we could outsource the readings. Not to ETS--that would cost a fortune. Rather to the Mechanical Turk. If you haven't heard of it, don't be too surprised. It's a way to crowdsource a problem that requires a human brain. Think of many smart ants building a fancy hill. In this case, its many human workers taking in small bits of work for small amounts of money.

Amazon.com runs this exchange. The currency is in HITs produced (human intelligence tasks) at a few cents per. As an example, one James F Lu is currently paying a dime for the following HIT:

Find Academic Calendar Information for Colleges

Please find the relevant academic calendar information. It can be generally done by viewing a school's academic calendar. All fields are required and verified in order to be approved.

Interestingly, someone named Ousama Haffar is offering 18 cents just to find the start and end date of university calendars. One can almost see the invisible hand at work here.

I imagine setting up a system where each (electronically gathered) essay is tagged with an ID number and assigned the following HIT. "Do you think the applicant who wrote this has personality traits necessary to succeed in college?" Note that it doesn't mention preparation--we already get that through high school GPA and (to some extent) SAT scores. I figure 5-8 cents is probably about right for this task, after browsing some of the other HITs on offer.

At the moment there are 57,899 HITs advertised. It doesn't say how many workers are available, but the home page says "Get thousands of HITs completed in minutes." I'd only need a few hundred or so a week, methinks. For 10,000 applicants at a dime each that's $1000. Imagine how much committee time it would take to do the same thing. It freezes the marrow to conjure that thought. The questions is, of course, is it worth it? That's the kind of thing that has to be answered empirically. In my experiences with subjective judgements of general education skills, the inter-rater reliability was surprisingly good. One could do that sort of thing here too--rate each essay three times or something, so as to do reliability checks. There's a certain amount of infrastructure to put in place here, but it's definitely worth a try. Stay tuned!

Viewing is the New Reading

In the classic science-fiction novels comprising the Foundation Series, I remember one of Asimov's created worlds where people very rarely interacted in person. Robots took care of the manual labor, and people socialized from their remote estates via electronic means. In local parlance, they "viewed" one another on screens. Very rarely did they "see" one another in the flesh--the distinction being between virtual and physical contact. Something like this is happening with books and reading. The situation is complicated by at least two trends.

The availability of entertainment choices naturally puts pressure on books to compete for eyeball time. The (live) google trends chart for books (lower line) and movies (upper) below shows this vividly.

Note the apparent inverse relationship between the two--the sign of competition for time and money between the two media. Following Asimov, I think of this as a tug of war between reading, in the traditional sense, and viewing a screen for one of several (maybe simultaneous) reasons, including reading, watching, listening, and communicating.

Even without competition for time, digital media are replacing print media because of the convenience and cost of access. The word "newspaper" will make about as much sense to the next generation as "dialing a phone." You can look at the trends for "darkroom" and "photoshop" to get an idea of digital replacing wet photography. [Take these graphs with a grain of salt, however. Obviously this is not authoritative data.] Accelerating this obsolescence is the convergence of technologies. Think of all the kinds of things your computer can do. You can play music, talk to your friends on Skype, shop, watch a movie, read the news, get the weather, create your own content in a mind-boggling number of ways, play a game, look at the family photo album, record the baby's first words, and on an on. A book in physical form can only do one thing really well.

I wrote recently about wanting to experiment with ebooks as texts for students, to allay the substantial cost of what can only be called the publishing racket. The Chronicle has a recent article with a case study from Northwest Missouri State University. They tried using Sony's ebook reader to deliver textbooks. It wasn't a smashing success. Students like to highlight passages and make notes, which was not possible. I think perhaps Amazon's Kindle is more capable, but in either case there is a fundamental problem: the convergence mentioned earlier. Eventually a book reader will want to become an Internet browser (the Kindle already is to some extent), and all the rest. In other words a small fully functional computer. We might generically call this "the screen." (At some point it will probably be plugged directly into our visual cortex, and our descendants will wonder what a 'screen' is). Apparently, NMSU has come to the same conclusion, evidenced by the following quote from the article.

This semester the university will continue to experiment with electronic textbooks, but it will deliver them primarily through laptops, rather than dedicated e-book devices. (The institutions requires students to have laptops.) About 500 students will try out electronic textbooks, and an additional 3,000 students will have access to them.

Laptops provide more interactivity than the Sony Readers, Mr. Hubbard said, because they let students participate in interactive quizzes and allow professors to add material to textbooks as needed.

Interestingly, the university also has a cost-effective textbook rental program for paper texts. Their progress will be interesting to track because it seems like a true competition between electronic and traditional print delivery based mostly on convenience and features rather than cost.

"Sophisticated forms of collaborative 'information foraging' will replace solitary deep reading; the connected screen will replace the disconnected book."--according to Christine Rosen in "People of the Screen" in the Fall 2008 New Atlantis. She goes on to ask:

We are increasingly distractible, impatient, and convenience-obsessed—and the paper book just can’t keep up. Shouldn’t we simply acknowledge that we are becoming people of the screen, not people of the book?

The article cites statistics that document how seldom young people read books for pleasure, and add that such reading is correlated with academic success. The salient point for us in higher education may be this:

Despite the attention once paid to the so-called digital divide, the real gap isn’t between households with computers and households without them; it is the one developing between, on the one hand, households where parents teach their children the old-fashioned skill of reading and instill in them a love of books, and, on the other hand, households where parents don’t.

Viewing, in Asimov's sense, is a different thing from reading at the neurological level. The article quotes Jakob Nielsen, who has done some research on the topic.

Rather than reading deliberately, when we scan the screen in search of content our eyes follow an F-shaped pattern, quickly darting across text in search of the central nugget of information we seek. “‛Reading’ is not even the right word” to describe this activity, Nielsen pointedly says.

Given this, is it a lost cause to expect textbooks in their traditional form to make the transition to the electronic realm? I think it is too much to ask. Imagine a best case, where you are curled up with a computer with a big beautifully optimized screen for reading. You open up A Day in the Life of Ivan Denisovitch and begin reading where you left off. Then an icon at the bottom of the screen flickers--you have a new facebook message. Or your calendar pops up with a reminder that tomorrow is Valentine's Day. Or (more likely) Windows wants to reboot itself because it just downloaded a patch. If your mind wanders at all, you may want to google a strange word, or look up Alexander Solzhenitsyn's biography. Now imagine trying to do the same with an organic chemistry textbook instead, where more discipline is required to stay on task.

The printed book forces a kind of digital silence on us. It doesn't have buzzers and bells and blinking lights to distract us, and you can't stream Pandora in the background from it while you read. Perhaps what is needed on digital devices is a "reading mode" where all the blinkenlights go away. You'd probably have to design the operating system from the ground up to accomodate such a thing. Convergence will almost certainly ensure that this doesn't become marketable.

If viewing replaces reading, education will change. I think it already has, actually. Some disciplines will be able to adapt to targeted bite-sized learning chunks, stitched together to make a curriculum. For these classes, textbooks will be the wrong way to deliver information. Technical disciplines like math and engineering will probably see traditional textbooks replaced by electronic widgets that can engage in a limited dialogue with a student, as a coach. In fact, some of these already exist. The point of an algebra class is that students learn how to make certain kinds of symbolic manipulations and be able to visualize this process. It's not naturally given to the narrative form to begin with; books were probably never the best way to deliver that information.

Courses that required reading long passages, like history and literature, will attract students who still know how to read. This audience won't vanish, but will dwindle along with the market for traditional books. At that point we will have the basis for a very interesting sociological natural experiment. What happens to those students after graduation? Which population--the viewers or the readers--is best able to navigate a complex world and succeed? Stay tuned...

The Secret Life of Committees

There really should be a required graduate course like Introduction to Committees. I can remember actually being disappointed (!) the first time the ballots went around and I was not elected to any faculty senate committees. My friend Dave, who came the same year I did, was elected the following cycle to serve on the Earthquake Prevention Committee, and I nursed the envy for months. I'd see him and five other faculty members stepping into private rooms, yellow pads in hand, and mutter cryptically when they thought I couldn't hear. Something about fault lines being created by the athletics program. This state of affairs continued until the following year when I was finally chosen by the collective to serve on the Exotic Plumbing Committee. Then I got my own introduction into the arcane art of committee work.

I see resumes from time to time where the applicant actually lists the academic committees they've been on. I suppose this is something like military officers and their service ribbons. I remember from my National Guard days that the Infantry Combat Badge (picture below) is one of the most prestigious ones. It means you've served under fire.
There should be something like this for academic committees, I think. I don't really mean to draw the comparison between what our servicemen endure in the line of duty and what academic committees do, but what's wrong with a visual representation of the mental scars endured by those who've seen the worst that can be offered up by the political bureaucracy? Here are a few suggestions. These ribbons would adorn faculty gowns on formal occasions.
My own interest began to wane about 2002 after a third term on the Y2K Preparation Committee, which had become a zombie: a wakeful dead committee with no purpose left but to terrorize the living. About that time I read about a species of snail that had accidentally been introduced in New Zealand, to the detriment of the fauna there. The new creatures would wander around until they found the trail of a native snail, and then track it relentlessly until they found and ate it. I decided we needed something like that: a committee to destroy committees. It would have to be innocuously named of course, and the members carefully chosen. Thus was the Transcendental Transition Committee born. It was not a great success, itself spawning at least 12 sub-committees before it thankfully self-destructed in a parlimentary accident. That was the last straw.

I began to avoid committees altogether. I developed a sophisticated system of "committee tag" whereby I had a list of names and a script taped to the wall next to the phone. If a colleague called to invite me to be on a committee, I'd simply read my script and pull a suitable name from the list: "I'm sorry Dr. Chairlice, but my rampant myopia would prevent me from reading the files at good speed. Might I recommend Professor Goodchoice as a superior candidate? She always shows up on time, and will do most of the work." I decided that the optimum committee size was one.

So it was with some interest yesterday when I opened the latest New Scientist to find an article entitled "The Curse of the Committee." It sounds like a sequel to The Mummy franchise, with perhaps Brandon Frazier as the independent chair who defies the president, but it's actually a fascinating history of the theory of committees. The notion--called Parkinson's Law after C. Northcode Parkinson--that work expands to fill alloted time, comes from this body of work. According the research cited, there seems to be a lifecycle of committees or other political bureacracies, whereby they grow and expand in a bubble, until it collapses. This is rather like Ibn Kaldun's notion that

when a society becomes a great civilization (and, presumably, the dominant culture in its region), its high point is followed by a period of decay. This means that the next cohesive group that conquers the diminished civilization is, by comparison, a group of barbarians. Once the barbarians solidify their control over the conquered society, however, they become attracted to its more refined aspects, such as literacy and arts, and either assimilate into or appropriate such cultural practices. Then, eventually, the former barbarians will be conquered by a new set of barbarians, who will repeat the process. [source: wikipedia]

In the case of committees, the transformation is simply one of size and effectiveness. They expand, become less effective, and then collapse into a smaller size again. To investigate this, researchers Peter Klimek and colleagues at the Medical University of Vienna used the following method.

[R]esearchers constructed a simple network model of a committee. They grouped the nodes of the network - the committee members- in tightly knit clusters with a few further links between clusters tying the overall network together, reflecting the clumping tendencies of like-minded people known to exist in human interactions. To start off, each person in the network had one of two opposing opinions, represented as a 0 or a 1. At each time step in the model, each member would adopt the opinion held by the majority of their immediate neighbours. (page 39 from the print edition)

Their conclusions? No committee should number more than twenty, a number reached by Parkinson's original heuristic approach. More surprisingly, odd combinatorical things happen when the committee size is eight, making it more likely to have deadlocks according to the research. Quoting again from the article:

[O]nce again, Parkinson had anticipated it, noting in 1955 that no nation had a cabinet of eight members. Intriguingly, the same is true today, and other committees charged with making momentous decisions tend to fall either side of the bedevilled number: the Bank of England's monetary policy committee, for example, has nine; the US National Security Council has six.

Just to be safe, you might want to consider committee sizes less than two.

[Update: this New York Times article from 1/15/2009 claims that as college costs have risen, disproportionately more money has gone to pay for administration. This is perhaps the same bureaucratic spread that is described above, in a wider context. A contraction is probably not far behind.]

Closing the Loop CLA Style

I'm not a fan of standardized testing for complex skills. There are many reasons that it seems like a poor tool to use, and among them are:

• Complexity means many kinds of instantiations. A standardized test can only examine a very limited number of these.
• A standardized test is mostly a monologue, whereas a dialogue is exponentially more powerful in elucidating useful responses in a complex situation. (imagine yourself as a 911 operator who is only allowed to give a standardized "problem report" form of yes/no questions to diagnose a caller's problem, instead of asking "what's your emergency?")
• There is an implicit definition of desired outcomes that is created when a test goes into print. These may or may not align with those of the test consumer.
  

This last point brings me to the reason for this post about the CLA. I recently received their newsletter, which mentioned something called CLA in the Classroom. You can read about it on its own official website here. The site describes the program thusly:

CLA in the Classroom is a curricular and pedagogical program that focuses on the higher order skills of critical thinking, analytic reasoning, problem solving and written communication.

I am generally annoyed by lists of skills like this. Critical thinking, in particular, is such a fuzzy concept that I find little use for it as a general learning outcome. It would be fine in particular contexts, like art, where 'crits' are pretty well-defined. The distinctions between critical thinking, analytical reasoning, and problem solving are not at all clear to me either. I assume, however, that the CLA folks have thought a lot about this and have their own definitions and ways of assessing them.

Assessing these skills should be done in context, of course. Therein lies a rub, as the Bard might say. What context do we assume students to be familiar with? A math major might demonstrate analytical thinking by finding the flaw in a proposed proof of a proposition, but that won't work very well for an English major. There has to be some least common denominator--a body of knowledge that all students will be familiar enough with to demonstrate their thinking skills. But is it reasonable to assume that this exists? Count me dubious. We might leap upon the general education curriculum as a natural basis for testing--a common knowledge base from which to launch forays into the critical thinking realms. There are at least two problems with this. First, most general education programs are fragmented. Rarely will one student have the same curricular experience as another. Second, the CLA may be administered before the student has taken the required background course, unless this is controlled for. To my knowledge it is not. So a low score might have as much to do with lack of background as it does with lack of skill.

A lack of clear context contributes to a lack of clarity about the goals of the CLA or any similar instrument, including most standardized tests of general education. Nature abhors a vacuum, as the saying goes, so this creates an opportunity for test-makers. If it's not clear what a test is trying to measure, the only real authority on the matter is the test-maker. Therefore, if you want your students to do well, who better to ask for help than the creator of the instrument? If one is cynical about it, the business model of drug companies is similar to this: advertise some new ailment that "requires" treatment. Who can tell you if you have the condition, and who can help you treat it (with maintenance drugs)? The medical industry of course.

Closing the loop with the CLA is similar. The Department of Higher Education or other authorities pressure institutions to use vaguely-defined measures of achievement, of which the CLA is an example. This grants a lot of power to the test maker. An institution that wants to do well is caught in a trap. How exactly to prepare students for the CLA? What context is necessary, and what particular skills should be taught? The test maker knows best. With CLA in the Classroom the company can not only diagnose your problem, but fix it too. To the naive, this may seem like a good thing, just like a patient demanding the new drug for "Restless Chin Syndrome," or whatever the malaise du jour is. In Assessing the Elephant [pdf] I call this sort of thing a "degenerative loop." That is, it looks like a closed assessment loop, but it has potentially nothing to do with your actual goals. It's natural that in an environment where goals are hard to measure or even define that such things crop up like mushrooms. Think of astrologers and miracle cures and mysticism.

It could be that the CLA does in fact align very nicely with your institution's goals. This is something to investigate carefully. If so, it could probably be as useful as anything you can construct internally, but with the added benefit of having external validation. But if you can't wrap your mind around the outcomes being assessed, or they don't look like the ones you value, you're essentially outsourcing the most important part of assessment. And the pressure to perform will probably have you looking up the phone number for CLA in the Classroom...

Ratcheting Learning Outcomes

I've been reading about the Bologna Club [here], which is not related to social snacking as the title might suggest, but is a European agreement on standards in education. There are several parts to it, but I was intrigued by the piece that is described as a ratchet. Ratcheting in this context means that at each higher level of education in a field, particular accomplishments must be demonstrated. Specifically:

• The reference points of “knowledge and understanding”
• The contexts and modes of application of knowledge and understanding;
• Fluency in the use of increasingly complex data and information;
• Breadth and depth of topics communicated, along with the range of audiences for that communication; and
• Degree of autonomy gained for subsequent learning

It's interesting to view these through the lens of what I've come to call the core skills: creative and analytical thinking, effective writing and speaking (see this post for more on creative and analytical thinking). Analytical thinking features prominantly here: knowledge, application, fluency in complex information, breadth of topics. Communication is mentioned once, and I assume it means various forms of communication. The last bullet, autonomy, has something to do with creativity I suppose, but it's very interesting that ratcheting up creativity is not specifically mentioned. I think this is not intentional, but it's unfortunate.

Creativity is where new knowledge is created after all. The bullet items could apply to any technical profession, like plumbing, just fine. I think there's something missing when you consider the research role of many academic disciplines. I'd add another bullet. Something like "Increasing ability to produce new knowledge." This ability probably doesn't grow linearly, but rather as an exponential. If you consider a primary schooler's knowledge of mathematics, there's little ability there to produce novelty (to them, I mean). Really good students will invent zero on their own, and really, really good ones will invent the negatives numbers. Richard Feynman recounts in his autobiography inventing the trigonometric functions. But normally, this ability wouldn't begin to develop for an average undergraduate until their junior or senior year. Those who like that experience may go on to graduate school. Of course, there are exceptions, but the rubrics should be written for the masses primarily.

It's a good exercise to examine a curriculum--including general education--and ask what the expectations are for the production of new knowledge. We spend so much effort trying to inculcate knowledge that we may lose sight of the purpose in doing that. Making this expectation explicit highlights the kinds of classwork that require creativity: creative writing, opinions and analysis with conclusions, or construction of formal systems in the more technical disciplines are examples. Personally, it's been helpful to me to help students by identifying these exercises as such. It eases their frustration somewhat to know that producing knowledge is naturally harder than absorbing it, and much more prone to error.

Cheap is Good, Free is Better

As I mentioned in a previous post, the book Predictably Irrational by Dan Ariely [amazon], people overestimate the value of free. At the same time, students and parents have a keen sense of prices that are too high. For example, textbook prices come to mind. In some cases, fees for college do too--if the student can't see the benefit.

In a staff meeting recently, the issue of textbook costs came up, so I was interested to see this article in Inside Higher Ed, which talks about the experiments at Northwest Missouri State University in reducing costs. They currently rent textbooks for cheap, and are transitioning to an online textbook system. This isn't free, but is at least cheap. And it's high-tech, which may be a selling point.

Can you do free? As a thought experiment, consider how many textbooks your students buy annually. Let's take a nominal population of 1000 undergraduates, each purchasing $1000 in books per year: 10 courses at $100 for each textbook. This is an implicit $1,000,000 investment that the institution makes each year. That is, if that money were not going to textbooks, it could as easily go toward a technology fee or financial aid (cost reduction). At a large institution, this scales up linearly: 30,000 students = about $30M per year. No wonder the textbook companies can afford to send professors free review copies. I used to sell them back to the book weasel (they guy who comes around buying them), and then use the cash as rewards for math problems given to my students. [As an interesting aside, I noticed it was hard to give the money away. If I made the prize too high, most students assumed the problem was too hard for their ability to solve and didn't even try!].

But back to the idea of free. In some evening programs particularly, adult students learn that they can take out more loans than they actually need for college (at least before the recession hit). The word on the street is "go to college at night and they give you free money!". Of course, it isn't really free, nor would textbooks be free, but they could be paid for out of tuition or anonymous fees, so it's a normal operational expense. Free textbooks at Your College.

How many of these texts do you think are actually being used at any one time? If we assumed we could eliminate the need for textbooks in class (using electronic editions projected onto screens or something), what is the maximum number that would be used at any one time? This is a typical IT kind of question. How many simultaneous phone calls need to be made at peak periods? How many wireless connections? Except in this case, we could circulate physical textbooks in the library as a low-tech solution. The most practical version of this is to buy a couple of textbooks for each type in the bookstore (used if you can) and circulate them in the library. Many students would opt to buy their own books, but not all. As demand increases, increase the number of texts. The electronic version is much cleaner, of course. Physically buying and circulating books is not attractive. Much nicer is having a set number of electronic copies that can be simultaneously accessed. Then all that's required is the infrastructure to deliver them to the students. "Free" textbooks plus high tech access is a winner.

We can't afford to miss a trick in this economy. Reducing cost and increasing customer satisfaction are paramount. I predict more institutions starting to use Customer Relations Management systems...

Economic Truth and Consequences

The latest edition of The Economist has an article about the duration of financial crises, which includes the summary table shown below.

The source material is a recent paper (pdf) called "The Aftermath of Financial Crises" by two economists (Carmen M. Reinhart and Kenneth S. Rogoff) who studied other financial dysfunctions and sought to divine what that means for the current situation.

This looks grim. If the past is any indication of the future (we're ironically assured the contrary on every financial prospectus), we can expect several more years of declining confidence, and hence performance. This is the placebo effect in reverse. With trustful relationships in tatters, the masses become conservative with financial transactions, which benefits them individually but to the detriment of the collective--a classic prisoner's dilemma situation. So what to do?

I figure if fear is ubiquitous, the thing to do is sell hope, and this is especially true of higher education. Here are some ideas.

• Emphasize outcomes that applicants care about, like jobs.
  
• Create new kinds of degrees that correspond to new careers
• Defer some costs until graduates are established
• Use work-study to immediately engage students
• Provide transparency about costs, so students see benefits
• Sell the liberal arts as flexibility and adaptability in an uncertain world
• Hire more graduates at the institution: begin to emulate the medieval model of a self-sufficient institution (okay, a monastery)
  
• Plan carefully for the next few years in order to be pro-active rather than reactive to restricted budgets. Use scenario planning.
• Be sure your tuition and financial aid policies put you at the right price point so you can reach the most students (and generate the revenue to educate them).

In Money, John Kenneth Galbraith's history of the stuff, he claims that panics were eventually renamed 'depressions' to seem less intimidating. Then 'recession' was needed to replace depression after the 1930s. At least 'panic' was honest. The message for ourselves and our consituents is the same as Douglas Adams': don't panic. That sounds a lot more reasonable than "don't recess."

First Generation Students

Much of my IR activity lately has centered on first generation students. My former and current institutions both rely on these populations for recruitment to some degree. Philosophically, this is where my heart is too--there's a special joy in seeing a student's success who is disadvantaged by circumstances. In one case, I had a student in MAT 100 (remedial algebra) discover that she really liked the subject, went on to become a graduate of the mathematics program. She has a rewarding professional career now.

The competition in higher education for the (apparently) best and brightest students can be pernicious. There seems to be an economic principle at work, whereby the best of anything gets bid up in the auction of public opinion to irrationally enthusiastic heights. This seems to happen with physical beauty (there has to be a 'most beautiful' man or woman to sell magazines), the financial markets (first tech stocks were beautiful, then real estate, for a moment commodities, now all is dross). I have opined probably too frequently that we over-value some applications and undervalue others. Shortly I'll have the opportunity to put this theory into practice and see to what extent this is true.

Low income, first-generation students are the topic of this article in InsideHighered.com from June, 2006. I came across it in a delicious.com search for "higher_ed". It's a couple of years old, but I can't imagine the situation has improved for this group of potential college students. The statistics given are dismal. The author Doug Lederman speculates about causes:

A broad mix of factors — financial, cultural and academic — may account for the underperformance of low-income first-generation students, the Pell Institute’s data show. The students come into college with many more of the risk factors that researchers have widely embraced as diminishing college success, including delaying entry into postsecondary education after high school, attending college part time, working full-time while enrolled, having dependent children, being a single parent, and having a GED. The average first-generation/low-income student has three such risk factors, while the average student who is neither first generation nor low income has one.

Once they are in college, they are more likely to have unmet financial need than are other students. They also work significantly more than other students, and those who work more are less likely to have earned degrees and to remain enrolled six years after entering [...]

Of these factors, only the financial aid factor sounds familiar to me. But there are cultural factors that show up in our research that are also detrimental to success. New first generation students report less family support, more family problems, and more dependence on families for financial support. It's clear from their responses on the CIRP that they don't really understand what a libearl arts college is, and they're attracted for other reasons, such as proximity and size. The publication Postsecondary Opportunity documents the long decline in need-based aid for such students. The real double-whammy is that they don't have much money to begin with, and their academic preparation does not admit them much talent-based money. The gap between need-based aid and tuition is a big part of the problem.

Closing the article is a set of recommendations to help these students succeed.

• Strengthening academic preparation for college, such as greater access to quality college prep classes and better information about college “gateway” courses while students are still in high school.
• Increasing financial aid for college.
• Improving transfer rates to four-year colleges, by strengthening transfer counseling and developing favorable articulation policies and agreements.
• Easing the transition to college, through better bridge/orientation programs and special programs for at-risk populations.
• Encouraging engagement on the college campus, including by creating better work study policies to let students work on campuses.

The first of these is the hardest. The rest are practical, and line up well with the strategies I'd like us to consider (and in fact have begun discussing in strategy meetings). Unstated in the article is that there are good students in this population--students that would work hard, eagerly consume the liberal arts product, and make any institution proud. Sorting out which is which is part of the solution--there's only so much need-based institutional aid to go around. It will be critical to be able to winnow out the applicants with the best chance for success. To do otherwise is to doom poor students to wasting time and incurring debt, as well as closing the door to a better student.

Finding Interesting Stuff

I recently created a dynamic page to grab RSS feeds from higher ed news sources. There are plenty more out there, and I'll probably create another page or two with different themes when I get the chance. For example, assessment deserves its own page. There are a couple more ways to find interesting stuff. These too can be automated, but it's not as easy.

Delicious.com (or del.icio.us) If you don't have an account here, it's a great site, and very useful for bookmarking with tags. The bookmarks live in the Cloud, and so are are available from anywhere. You can follow other people's tags (here are mine), creating a network of interested parties. The idea is that you can restrict searches to a group of experts in a field, so you get quality hits. This can even be automated to some extent.

For example, this link searches for the tags "assessment" and "higher_ed". At some point I'll see if I can filter the results with pipes.yahoo.com to create an RSS feed out of it. If delicious.com provides that service I can't find it.

Technorati.com is designed to index and search blogs. If you have a blog, you can register it with them, and it will be searchable as soon as you post a story. Searches can be tied to an RSS feed, so you can get the information with little effort. Here's a link for the same keywords as before from this source. Unfortunately, their RSS feed seems to be flakey--I haven't been able to get it to work for weeks.

When I get the chance, I'll try to automate these and a few more sources, and create some targetted higher ed topics pages.

The Talent Bubble

I've argued before that the last decade has seen tuition increases and discount rate increases driven in part by a red-queen's race for the most talented students. The best applications are often seen to be those with high SAT, high high school GPA, and extras like co-curricular activities. Competition for these is fierce, and institutions with the highest endowments, or otherwise can offer the best aid packages, are in a commanding position. The Internet facilitates multiple applications, and the price (from a college's point of view) gets bid up as in an auction.

I was part of a conversation today with an enrollment professional who put the proportion of second-generation African-American applicants at 15%. For an HBCU, this means the pool of "good" applications (in the standard recruiting definition) is tiny. It becomes expensive to create attractive packages for these students. There will be pressure to sacrifice need-based aid in order to buy talent.

This is a lousy business model in the short term. With a decade-long perspective, it's attractive to have a growing pool of successful alumni, but you can bankrupt yourself in the process. It's a zero-sum game--there are only so many really good applications. But is that really true?

In my study of an admission matrix at one institution, the student enrolled at the bottom end (provisionally) succeeded about half the time. That is, there's a 50% chance that an applicant that looks lousy on paper is going to defy expectations on the upside. This isn't really surprising when you consider that predictions of first-year GPA based on grades and SAT aren't very good. This is especially true for low social-capital applications, such as first-generation students.

I've argued that we overprice high SATs at the cost of under-pricing some of the low SATs. If we could tell which low SAT students would succeed, this would be a gold mine for any institution. I actually wrote to ETS years ago and suggested that they develop and alternative instrument, but never heard back.

Here's how it would work. In addition to high school GPA (and SAT if you absolutely have to have it), find other indicators of success. Things like high school attendence records ought to be useful, but there are surely surveys that can be developed that would help. The CIRP is very helpful, for example, in post-facto analysis of attrition. A few attitude and behaviour question slipped into the application form might be enough to get started. The danger is that applicants figure out what combination of responses will help them, and 'game' the responses. There may be some way around that.

If you crack open that puzzle, you find yourself with the 85% of the African-American students (or 65% for caucasian) who are not being bid for aggressively. Of these, perhaps only 20% are of interest to you, but if you can zoom in on that 20% you've done yourself a real favor: found good students who don't cost as much as the high SAT crowd.

This is a project I'll be engaged in soon. I'll start with the app form and add some questions of a the type identified from an analysis of CIRP responses compared to college GPAs. Once we've identified a few indicators of success, we'll focus a few questions on those topics.

The Well Intentioned Commissar

(A parable for academic workers and those who direct their activities)

by
David W. Kammler, Professor
Mathematics Department
Southern Illinois University Carbondale

In the early days of Communist Party rule in Poland, the State Central Committee ordered the Commissar for Building Materials to improve the productivity of the eight factories where iron was turned into nails. The Commissar was a well-trained bureaucrat who decided to implement quantitative management techniques (which he only partially understood).

I must carefully define a measure of performance that will enable me to identify and reward increases in “productivity,” he thought to himself. After some reflection he drafted a memo informing the eight Directors that during the coming year they would be evaluated by using the performance metric
The Commissar was filled with self-satisfaction as he anticipated the presentation he would give to the State Central Committee, precisely quantifying the annual productivity increases.

The ambitious (but unscrupulous) Director of the Wroclaw nail factory, Stanislav Nowak, read the Commissar’s memo and immediately called his counterpart at Poznan. Dear comrade, he said, we are all being pressured to increase productivity and I know that you have been having difficulties with your outdated press for making boxing nails (the 2½" long 8d nails used to attach boards to studs and rafters when building a house). I have a surplus boxing nail press here at Wroclaw that I am willing to swap for your little used wire brad machine since mine is almost beyond repair. And so he traded Wroclaw’s only boxing nail press (the adjective “surplus” being a socialist white lie) for a second machine for producing wire brads (tiny ½" nails occasionally used for small trim work). He made a similar offer to the Director of the nail factory at Lublin, obtaining a third brad press in exchange for his machine for making framing nails (3½” long 16d nails used to fasten studs, joists, rafters, etc. when framing a house).

Throughout the year the Wroclaw factory concentrated on the production of wire brads. Workers who had previously been assigned to the boxing nail and framing nail presses now produced tiny wire brads. Workers who had previously produced roofing nails, finishing nails, etc. during the day were reassigned to the night shift where they made still more wire brads. At the end of the year, the Commissar carefully calculated the figure of merit for each factory. The nail factories at Poznan, Lublin, Gdansk, Szczecin, Katowice each had a noticeable increase in productivity with scores of 1.12, 1.11, 1.08, 1.05, 1.07. But by concentrating on the production of wire brads Stanislav Nowak had achieved a productivity metric of 8.15, in spite of the fact that he had only processed 20% of his iron allotment for the year! The delighted Commissar rewarded the successful Director by assigning him a villa, and he rewarded each of Nowak’s workers with a kilogram of bacon and a large basket of rutabagas.

As one might expect, the storage warehouse at Wroclaw was filled with keg upon keg of unneeded wire brads. Moreover, since the iron that should have been used to make boxing nails and framing nails was also sitting in the Wroclaw warehouse, Polish carpenters began to experience times when the supply depots ran short of the nails they needed most for house construction. The Commissar took note of the situation and reasoned as follows. I will ask the State Central Committee to “solve” the national nail shortage by increasing the iron allotment for each of my factories. (Commissars are always looking for an excuse to ask for additional resources!) Of course, the real reason for this year’s slight shortage of boxing nails and framing nails is that in spite of its outstanding productivity the Wroclaw factory was unable to process all of its annual allotment of iron. I can easily remedy the situation by changing my productivity metric to encourage the conversion of iron into nails. With supreme confidence in his analysis, he informed the Directors of his eight nail factories that for the coming year the productivity metric would be

The Directors of the nail factories at Gdansk and Szczecin invited Nowak to conduct seminars on “The Wroclaw wire brad strategy for productivity enhancement”. They were envious of the stunning success of their colleague at Wroclaw and eager to implement the methods that led to his recognition and reward. They were absolutely delighted when he “reluctantly” agreed to exchange two of his wire brad presses for their seldom used machines for making log spikes (8" long 80d nails used for constructing wood bridges, log cabins, etc.). The Director at Katowice made a similar deal, exchanging his log spike press for the Wroclaw tack machine. After all, he reasoned, tacks are only slightly larger than wire brads so the same management principle must certainly apply.

Throughout the year workers at the Wroclaw plant fed iron into the four log spike presses, keeping them running day and night. Time and again additional iron allotments were obtained from the State Central Committee to keep the factory humming. When the Commissar computed the annual productivity metrics, he was dumbfounded. The nail factories at Gdansk, Szczecin, Katowice that adopted the “tiny is better” strategy had the disappointing production metrics of .22, .21, and .26, respectively. The factories at Czestochowa and Krakow, with Directors who routinely ignored any and all management directives, had lackluster metrics of .98 and 1.01. But the factory at Wroclaw had a production metric of 21.30! This time Nowak was given a Polonez (Polish luxury car) with a driver, and each of his workers received a kilogram of bacon and two large baskets of rutabagas.

The Wroclaw warehouse was now filled with keg upon keg of unneeded wire brads and keg upon keg of unwanted log spikes. Since only four of the eight factories were still producing boxing nails and framing nails, the shortage was now acute, with construction delays of 1-2 months being common during the critical summer building season. The State Central Committee wrote a formal memo to the Commissar, ordering him to investigate the situation and take immediate steps to correct the problem.

It’s merely a matter of choosing the right metric, the Commissar said to himself. Since he had learned a bit of statistics during his school days he was able to identify precisely what had gone wrong during the previous two years. A productive factory must make nails of various sizes to serve the needs of our building trades, he reasoned, so we want the standard deviation of the nail size to be large, not zero as has been the case at Wroclaw during the past two years. A productive factory must also turn a lot of iron into nails. And in view of the current crisis it seems best to focus on annual production rather than some measure of year to year improvement. With this in mind he created the productivity metric


When this management directive was received at Poznan, Lublin, Gdansk, Szczecin, and Katowice, the discouraged (demonstrably unproductive!) Directors had no idea what it meant and no absolutely incentive to find out, so they decided to follow the example of their peers from Czestochowa and Krakow and simply keep on doing exactly what they had done the year before. They could not comprehend why the production of minuscule wire brads or massive log spikes was regarded as meritorious at a time when the Polish building trades were starved for common boxing nails and framing nails.

Meanwhile, the canny Director of the Wroclaw plant once again ordered his workers to maximize their production of log spikes throughout most of the year. Just before the year ended, he asked them produce an identical number of tiny wire brads. (This made the standard deviation of the nail weight approximately half that of a massive log spike.) And for the third year in a row, the Wroclaw factory was judged to be the most productive in all of Poland, with a performance metric 36.2 times larger than that of the next best factory. The Commissar publicly congratulated Comrade Nowak and his workers at the Wroclaw nail plant for a job well done, and bemoaned the fact that in these hard economic times he was unable to offer them even token rewards for their outstanding achievement.

Unfortunately, the national shortage of boxing nails and framing nails was now so severe that thousands of carpenters were unemployed and the whole economy was depressed. The members of the State Central Committee were greatly displeased with the Commissar. After all, they had clearly identified a problem and told him to solve it, but in spite of his best efforts things had gone from bad to worse. He was given one last year to remedy the situation.

Our factories must be made to produce nails that our carpenters can actually use, he reasoned. We cannot encourage them to make nails that sit rusting in their warehouses. With a sudden flash of inspiration he created a corresponding measure of performance

He winced as he realized that this metric reeked with the foul smell of capitalism, but he was desperate and could think of nothing else to try.

The Directors at Poznan, Lublin, Gdansk, Szczecin, Katowice, Czestochowa, and Krakow were shocked to receive a management directive that violated their most deeply held Marxist convictions, so once again they ignored the memo and continued with the same production schedule they had used for the past two years. In contrast, the devious Nowak immediately recognized that the Commissar could not very well chastise him for adopting a capitalist response to an intrinsically capitalist directive. For the third year in a row, he ordered his workers to produce as many log spikes as possible and then left Wroclaw for a six-week holiday at an exclusive Black Sea resort. At the end of the year, he took these log spikes together with all of the log spikes and wire brads from his warehouse (i.e., virtually every nail that his factory had produced during the previous four years!) to a local firm that made tracks for the Polish railroad. He sold the whole lot at half the going rate for scrap iron. And this year, Nowak’s production metric was 9.2 times larger than that of his closest competitor!

The State Central Committee summarily dismissed the Commissar for Building Trades and initiated a search for his replacement. Two weeks later, the Committee met to consider resumes of the applicants. Comrades, said the Chairman as he studied what appeared to be a most promising vita, I believe that I have identified a leader who can set us on a Five Year Plan that will rejuvenate our failing economy. During each of the past four years Stanislav Nowak, the Director of our nail factory at Wroclaw, has led his workers to be the most productive in our nation. I have never seen such consistently high measures of productivity. These numbers do not lie. Even in this severe depression, his workers are 9 times more productive than those at any other nail factory. And so Stanislav Nowak was named Commissar of Building Trades for all of Poland, a most remarkable honor for a man who managed a factory that had produced absolutely nothing but high performance metrics during the previous four years.

MORAL: When we lose sight of our purpose, we inevitably become unproductive, no matter what numbers we generate to prove otherwise.

Permission to copy: You may share copies of this story with anyone who might enjoy thinking about its implications in academia. Such samisdat publication (or to use the equivalent Polish phrase podziemna publikacja) is in keeping with both its setting and with its message.