Transparent Pricing in Higher Ed

If you haven't read it yet, I recommend that you pick up a copy of Paul Tough's book The Years that Matter Most: How College Makes or Breaks Us. In particular, the section starting on page 182 of the hardcover is a revealing picture of how college finances affect recruiting.  You can find the same (or very similar) content in this New York Times article.

In short, most private colleges need to balance academic admissions requirements with what are essentially financial admissions requirements. The latter are needed to ensure sufficient revenue to make a budget. Public institutions are not immune either, since for many of them their revenue also depends heavily on tuition. Academic goals and financial goals for recruitment vary widely from one institution to the next, depending on market position, endowment, and other factors. This leads to a lot of variation in the actual price a given student might pay at different institutions.

Every college now has a net tuition calculator, which--given a prospective student's characteristics--estimates the out-of-pocket cost, which often includes loans. However, this is just an approximation.

Mark Salisbury's idea was to make college pricing transparent by sharing real offers received by accepted applicants: crowdsourcing the problem. The site is about a year old, and you can find it at tuitionfit.org. To get a sense of how it works, look at this article. It looks like his timing (and co-founder Kimberly Dyer) was stellar, given the increased competition colleges are seeing from demographic changes and from the new acceptance of "student poaching" as an enrollment strategy.

2020 Gogs and Recruiting Costs

Everyone, it seems, is developing "2020" plans, a riff on the perfect vision score. The best quote I saw (on reddit) was that in 10 years we'll look back with 2020 hindsight. In that vein, The Chronicle has published an article called "The College of 2020: Students." You have to pay for the full report, but the executive summary is here (pdf). The abstract:

This is the first Chronicle Research Services report in a three-part series on what higher education will look like in the year 2020. It is based on reviews of research and data on trends in higher education, interviews with experts who are shaping the future of colleges, and the results of a poll of members of a Chronicle Research Services panel of admissions officials. ... Later reports in this series will look at college technology and facilities.

For the IR folks, Noel-Levitz has an interesting Cost of Recruiting report, which you can find in pdf form here. This gives good benchmarking information for assessing the efficiency of your admissions office.

Friday Potpourri

This morning my cognitive cogs are engaged in so many projects it seems appropriate to dish up a small buffet of assessment-related items:

1. SES gets you a seat: An article in InsideHigherEd this morning tells that as access to higher ed boomed in the US, so did demand, so that competition naturally favored those best able to negotiate the barriers to admission. Guess who that was? This quote is attributed to the researcher, Sigal Alon:

[T]esting became a more important factor in admissions at more institutions, and that wealthier families are much speedier to adapt to changes in admissions rules.

This sounds like the "economic value of error" stuff I was prattling on about a few weeks back. Although I can't find the source article online, a kind of appendix is here. She reiterates what everyone paying attention knows:

Test scores thus seem largely redundant with high school grades (Crouse and Trusheim 1988). Test scores do have some predictive value for college success, but is this improvement worth the negative impacts of testing?

She goes on to advocate a more aggressive program than simply using noncognitives to better identify talent in lower SES groups. She's probably right--if only because the high SES groups will learn how to negotiate the new system of merit just as well as they did the previous one.

2. Rubics are bad for your brain: Well, maybe I exaggerate, but two articles posted to the ASSESS listserv provide cautionary advice for those constructing rubrics. On another level, there is (I find) a tendency to put too much faith in the things to begin with. Maybe that's because a rubric is tangible evidence of a plan. It looks like science--with its dissecting of some learning outcome into dimensions of quality and component. I won't reiterate my own annoyances with this quasi-religious belief here. You can find better and more constructive comments in the papers themselves:

• What's wrong and what's right with rubrics
• What's still wrong with rubrics?

3. NSSE Dimensions: At a recent IR meeting locally, I saw a very nice analysis of one institution's NSSE data from several years. The question being considered was whether or not the scorecard-type ratings that NSSE reports out are valid. You can see what I'm talking about in the 2008 annual report here. I clipped out a bit from page 8 to show some of the "dimensions" in question:


Here you can see that Level of Academic Challenge, Active and Collaborative Learning, and Student-Faculty Interaction are indices consolidated, summarized, and reported out as being meaningful. There is a great power to words, which gets abused routinely by the testing and assessment profession (like misuse of the word "measurement"). To be fair, it happens anywhere there's room for doubt--opportunity to advertise something more than what's there without being caught. We humans seem to prefer certainty over doubt, even if it guarantees we're wrong. But I digress.

The putative dimensions listed (Enriching Educational Experiences and Supportive Campus Environment are the other two, not shown above) are drawn from the items on the NSSE. In order to validate such things, one would test the construct validity using factor analysis. You can find lots of reports at NSSE referencing validity here. I will have to go through them later, to see what I can find in this vein.

The presentation at the IR meeting used confirmatory factor analysis (CFA), which posits that the dimensions are defined according to given rules, and then tries to verify that using the results at hand. Ideally, the items related to Academic Challenge would all show up as elements of a single vector when the correlation matrix's singular value decomposition is calculated. The actual results were not convincing.

I poked around to see if there were other studies out there, and found this one. Here, researchers at James Madison University and Eastern Mennonite University used similar methods and found similar results. They summarize their findings:

[A] CFA of the benchmarks as specified by NSSE literature produced poor model fit. For the sample of students examined for this study, the five-factor benchmark model supported by NSSE was not upheld, thus a five-factor solution is uninterpretable for the sample. A comparison of benchmark scores from this sample to scores from a sample at another university should not be made, as the benchmark scores from this sample were not empirically supported. Until further studies can provide evidence for an interpretable set of benchmarks for this and similar samples, one should not make policy or programmatic decisions based on the benchmark scores.

The whole article is worth reading. It's well-written, and there are interesting details about particular loadings.

Sock Muffins

Yesterday, as I was racing to get ready for the commute, I slipped on my shoes and jacket, clipped on the blackberry, grabbed the briefcase, and headed downstairs. I noticed about five hours later than I'd put on two different kinds of loafers, one brown and the other kind of reddish. But I digress. When I got downstairs, I made my second cup of coffee and waited for my wife. When she appeared, her eyes immediately went to the counter under the cupboard for the plates and glasses. There, lying in front of the mixer, were the dress socks I'd worn to the gala along with that rather ill-fitting rented tux. What the heck were they doing there? I knew it was no good to blame the sock fairies, because we'd had the house sprayed for them.

"What are these?" she asked? Meaning that she knew what they were--she just wanted to make sure I knew they didn't belong on the counter. I had to think fast.

"Oh, those are for the sock muffins," I said as casually as I could.

"Sock muffins?"

"Yeah. You pack dough in the socks, then squeeze them into little balls like making sausage links. Bake 'em right in the sock, and then cut it off afterwards. You know...sock muffins."

There was a long pause. She's German, and it could just be that this was some crazy American thing, I could almost see her thinking. Then she asked "well, aren't these new socks?"

I'm afraid my ruse failed at that point because I started laughing. I'd make a lousy spy. The moral of the story is, if you do enough crazy things often enough, people will believe you're capable of anything.

At this point, what is called for is a pithy segue. If today's profundities included passing accreditation, this would be relatively easy, but I promised last time to look at this study on expectations vs. reality at elite institutions. I'm not clever enough to figure out how to go from sock muffins to predicting grades, so you'll have to use your imagination.

I missed the point last time that the predicted grades in the study were self-predictions by students before taking classes. The biggest gap in this expectations game was for minorities, particularly Black students.

The article itself is written in economics-speak, which for a math guy is a little like a German reading Dutch. It's not something you can sit down and read casually. It took me a lot of flipping back and forth (during a long meeting) to determine that

just means multiply the number of applicants of a race by the percentage of that population that lie above some cut off (T*) in benefit. This gives the number enrolled in that category. This formula appears throughout the paper. I think I'd have been tempted to condense it to B(Tmin) and W(Tmin) to save a few eyeballs, but never mind.

The comments on the article at insidehighered are interesting and worth browsing if you're interested in this topic. The idea of mismatch is clearly controversial. My interest in the article was more about getting insight into Duke's admissions process. But I will say that it seems obvious that providing more transparency about the likelihood of success is good for everyone. In my attrition studies (see here for example), it's obvious that a mismatch in expectations for new students is very hard to recover from. This has implications from marketing through the first year experience and beyond.

Duke uses variables gathered from the admission process that in this report are called Achievement, Curriculum, Essay, Personal Qualities, Recommendations, Test Scores, high school GPA, and SAT. Note that at least two of these (achievement and personal qualities, and probably recommendations as well) speak to non-cognitive traits. How well does it work? From page 17 of the report :

Notice from Column 3 that controlling for Duke’s rankings increase the R^2 by more than 0.12, again suggesting substantial Duke private information. Note that this still leaves two-thirds of the variation in GPA unexplained, perhaps due to course selection and shocks to how students respond to college life.

So the 'private information,' which would not include things like high school GPA and SAT scores, since those are known to the student, counts for about 12% of the variance in correctly predicting first year college grades averages. The two-thirds unexplained variance in those averages is about typical in my experience. I think this is the real story, and the real opportunity.

In the insidehighered article, one anonymous commenter writes:

Experience at my institution is that students from lousy high schools do not do well in their first year of college, but, if they're bright and willing, eventually catch on to what university-level work is all about. They also find what they want to study, and this adds to their motivation. So focusing solely on the grades in the first year of college vastly overstates the effect being studied.

This point is well taken. A better indicator of success is perhaps first year retention. If a student fails to return, that represents a failure in real terms: investment of time and money that cannot be fully recouped even if courses transfer. Our predictive powers are not stellar in that regard either.

Time to make the commute again. Guess I'd better go look for my socks.

Admissions as Drama

One of the (really excellent) IR staff here forwarded a meaty article on market research for higher education from the Lawlor Group entitled Trends & Tips for 2009. This research firm targeted at independent institutions bills itself as "intelligent marketing solutions for education."

Much of what's in the report is insight into the impact of the worldwide depression on the consumer market. The rise over the last decades in tuition is pegged at 439% since 1982, according to the report, compared to family income median rise of 147%. Even if we assume that discount rates went from zero to 33%, parents are paying far more now than before. And this situation is getting much worse because of the crash in equities and dwindling access to loans of all types.

Parents will do what shoppers always do when money is tight, the report argues: shop around for bargains. Quoting from one of the bullets in the article:

In October 2008, nearly 60% of surveyed high school seniors were considering a less prestigious college for affordability reasons; 14% changed their focus to a two year college; 16% put their college searches on hold. (MeritAid.com)

This is not the year to take your yield rate for granted. This is an important calculation many private schools will be making right now. How much do we shave off of our enroll/admit ratio when predicting next year's entering class?

It is already getting pretty late in the admissions cycle, but there's still time to change the road you're on. One of the recommendations of the Lawlor Group is not to take the value of the institution's product for granted. They recommend supporting it with evidence.

If your applicant pool includes first-generation students, the problem is likely to be worse. This is my own speculation, based on research done during the last several years. Lower-income and/or first generation families are likely to complete their taxes later, which complicates an already complicated FAFSA application. Awards can be put on hold while information that was put on the FAFSA as preliminary is updated and verified against returns when they come in. This is another whammy directed at low-income groups, in addition to low ability to pay, correlated lower entrance test scores, and dwindling need-based aid.

The form is supposed to come under review as one of President Obama's intiatives, but that will be too late for the current cycle. In the meantime, companies have sprung up that will help you with the form, for a fee comparable to H&R Block doing your taxes. One of them is www.fafsa.com, which charges $80-$100 to walk you through the form in under half an hour. That price may not be prohibitive to most middle-class families, but it is almost certain to be a limiting factor for some demographics.

Shopping for scholarships is another trend that is likely to accelerate, methinks. This is the time to be creative with institutional aid. Discount rates are almost certain to grow, unless formal tuition cuts become common. Here's a site that lists institutional awards, both endowed and unfunded: www.meritaid.com. Another barrier to unsophisticated consumers of the higher ed product is negotiating (in all senses of the word) the award hussle. Anything we can do to lower barriers and funnel students from one decision to the next, one bit of the bureacracy to the next, is a good thing.

Think of your application process as a movie script, where the FAFSA is smack in the middle of Act II. Is your film a long complicated thing or a fast-moving action flick? Who can best negotiate the plot twists--have you picked the right audience? Some tips come from the Lawlor Group's final recommendations:

Private colleges must continue their efforts to overcome students’ perceived financial barriers to enrollment by (1) demonstrating a commitment to containing costs, (2) making the net tuition price as transparent as possible as early as possible, (3) structuring institutional aid to increase the enrollment of low- and moderate-income students, and (4) implementing programs that narrow completion gaps across socioeconomic groups.

This is an outline of the story we have to tell in the 120 pages of our three acts (transforming the recommendations above into a dramatic arc of sorts):

1. It's cheaper than you think; you can afford us.
2. See, here are the costs.
3. And here's how we help you afford it.
4. The payoff is emotionally satisfying.

This year and the following ones will be a time of change, no doubt, and maybe it's not that crazy to take a page or two from Hollywood. Hey, maybe they'll start giving out Oscars for Best Admissions Process.

Beyond the Big Test

This weekend I got my copy of William E. Sedlacek's text with that title, which is also subtitled Noncognitive Assessment in Higher Education. Anyone who's read my blog in the last year will know I've scratched my head over the evident value that admissions processes typically put on ACT/SAT, and to a lesser extend high school grades (see this article, for example). Our work with surveys like the CIRP shows that there are behavioral and attitudinal traits that make good predictors of, for example attrition. These are generally ignored in the admissions process. Or it might be better to say they're not looked for.

Well, silly me. Apparently this subject has a long history in the literature and is a quite well developed concept. Some major schools like North Carolina State University have used these methods successfully. So my research program to find variables for use in predicting academic success can accellerate considerably--we just have to customize the work others have done.

I have not finished the book, but can tell already that it's a wonderful resource. The background and history of noncognitive assessment is given, as well as solid research findings, actual survey instruments, and examples of how to coach staff in looking for these traits in interviews, on existing application materials, essays, etc.

The focus of the book is toward evening the playing field for what the author calls non-traditional students. In his usage, this means anyone who isn't a white male. My purpose is more targeted, but the material is no less useful for it.

The specific noncognitive traits identified, with some help from factor analysis in the process of ascertaining construct validity of surveys, is as follows (pg. 7):

1. Positive self-concept
2. Realistic self-appraisal
3. Successfully handling the system
4. Preference for long-term goals
5. Availability of strong support person
6. Leadership experience
7. Community involvement
8. Knowledge acquired in a field

These are described in detail, of course, along with methods of assessing them.

I'll be recommending that our university proceed full speed ahead with this project, to catch what we can of the current cycle. This will undoubtably make Mr. Sedlacek happy, as it will entail buying many more copies of his book for distribution.

Admitting the Right Students

All of us who teach like to have students in the classroom who want to learn. In a recent conversation this topic came up. The math teacher I was speaking with said he'd take a hard-working curious student over an indifferent student any day--regardless of academic preparation of the latter. I would too. The real reward of teaching, after all, is to see growth. One of my proudest (shared) accomplishments is that one of my MAT 100 (really a remedial course) students fell in love with the topic and went on to graduate with a BA in mathematics. She now works in a technical profession and is very successful.

All this by way of introduction to my topic: how to judge applicants for admissions purposes? Academic preparation isn't enough if we care about things like motivation and perseverance. I've thought about and written about different angles of this topic for a long time, and now find myself in a position to do something about it.

The conventional wisdom, if there is such a thing, is that grades and SAT are reasonable predictors of achievement. These are called cognitive measures for reasons best known to the psychometricians. As I recall high school, there was a lot more to grades than cognition, but never mind. SAT is a favorite target for those who don't like this narrow thinking, and I'd count myself with the critics. I took the ACT in 1980 in Illinois, and remember liking standardized tests then. I think I got a 28 on the ACT, whatever that means. I also took the ASVAB military battery (in the sense of tests, not artillery), and actually found the results last summer when I was going through stuff in the garage. I remembered taking the test, driving to a National Guard armory in East Saint Louis with my friend Mark, following the lines marked on the floor to the various stations. One was an attitudes and behavior survey, where they discovered that I'd never smoked pot. They made fun of me for that, or else it was some kind of awe. This was 1978, and the stuff was everywhere. One station was the ASVAB, complete with number 2 pencils.

In the Army's eyes, I was being rated for potential by this test, and they took it very seriously. Frankly, I loved taking standardized tests because I always did reasonably well on them, and hence had no stress about the results. What's interesting about the ASVAB is the one domain where I did not do well at all. You can see on the image that there's a 55% bar in the middle. When I pulled this out of the box in the garage last summer I had to squint to read the faded type of the explanation. If you look on the far left, there's a CL designator: that's for "clerical work." The battery decided I'm no good at it. If I recall correctly, this part of the test included such things as counting how many Cs were in a line of Os. Something like this:

OOOOOOOCOOOOOOOOOOCOOOOOOOOCOOOOOOOOOOOOOOOOOOOOOOOOO

That's the sort of thing that drives me crazy. I will tip my hat to the test designers: I am in fact not well suited for mindless clerical work. Did the rest of the battery rate me accurately? I guess that depends on what the expected outcomes were. I wasn't at that time very good leadership potential, was a bit lazy academically from never having had to work hard in school, and didn't have a lot of ambition to go change the world. Yes, I was only 16 or so, but looking at the test results would still have vastly overestimated what kind of officer I would have made at that point. In short, there were a lot of imporant things that the test didn't measure. This anecdote underlines my main thesis about standardized tests: they are good for simple tasks but not complex ones. And there are many complexities to what makes a succesful college graduate. The specifications are likely different from institution to institution as well.

All this by way of introduction to an article I found while researching the link between SAT and race. The article from InsideHigherEd by Scott Jaschik dates from September 2008, and opens with:

[C]ritics of standardized testing — and especially of the SAT — have said that these examinations fail to capture important qualities [...]

What's interesting is that the College Board--creator of the SAT--agrees, and has an active research program for developing new approaches. This is good news and bad.

First it's good, because the old SAT "cognitive test" will lose some of its halo and the market place for talent can become more efficient. This is good for all of us. It's also good because it will put standardized testing--the modern day equivalent of phrenology, in my opinion--under more of a microscope. If policymakers have more sophisticated ways to think about achievement, it's beneficial to everyone.

It's also bad in a very selfish sense. This is because the institutions that are already acting on this market inefficiency will see their lunch being shared around the table. The article mentions a couple of these. Tufts University and Oregon State University are using non-traditional approaches to admissions according to the article. Of course, I'm not really serious about this--I'm very happy that there's competition for the meme that SAT is a real achievement score, and I'm confident that a real research program can keep us on the cutting edge of finding the most suitable students for our institution. Ultimately it's good to have competition, and for reasons outlined below, I think each institution will have to find its own solution anyway.

So what are the new methods looking for besides "cognitive" processes? The Group for Research and Assessment of Student Potential (GRASP) has twelve so-called dimensions they consider:

1. Knowledge, learning, mastery of general principles
2. Continuous learning, intellectual interest and curiosity
3. Artistic cultural appreciation and curiosity
4. Multicultural tolerance and appreciation
5. Leadership
6. Interpersonal skills
7. Social responsibility, citizenship and involvement
8. Physical and psychological health
9. Career orientation
10. Adaptability and life skills
11. Perseverance
12. Ethics and integrity

You can read descriptions of these by following the link to their site. In addition to GRASP, SAT is working on modifications to their instrument, which may result in a whole new test. I'm a bit dubious about this project, however.

There are already ways to game the SAT. How much more will this be true when the 'right' answers are clearer? That is, it's much easier to appear to have attractive behaviors and attitudes than it is to actually possess them. I can imagine a new preparation industry springing up to coach test-takers who can afford it. Ultimately I think the industrialization of this metric is doomed for this reason. That leaves individual admissions policies to find ways to gather information in ways that are less likely to be faked. The article makes the same point.

The results of SAT's trials with their new test items are interesting. By de-emphasizing academics in favor of the College Board's experimental "biodata" and "situational judgment", traditionally under-served minorities showed significant increases in enrollment. Particularly notable is an almost 6% increase in black enrollments at the highly selective institutions that participated in the experiment.

There are problems, like gaming the system, but the payoff is worth it for individual institutions. The article quotes Pamela T. Horne, who has been working with GRASP results. She gives voice to my feelings on the topic:

“This is mission-driven,” she said, noting that colleges don’t define their missions as “enroll students with high SAT scores,” but they do prize leadership, artistic vision and various other qualities that might now be measured.

Unfortunately, if you asked many college presidents and board members, they probably would say that high SAT scores are an institutional priority. Why else would there be all the concern about which scores are reported (first, last, all, average?). Ultimately, more enlightened institutions can take advantage of the limitations of the SAT and other standardized predictors by individualizing their own processes. It's an exciting challenge.

Update: Reading the comments on the article, I found a reference to a textbook on the subject of non-cognitive assessment. Here's the link on Amazon.com. Beyond the Big Test: Noncognitive Assessment in Higher Education (Jossey Bass Higher and Adult Education Series)
by William E. Sedlacek (Feb 26, 2004)