Creating a Competition Analysis Chart from IPEDS and Clearinghouse Data

The National Student Clearinghouse provides information about applicants to your college that ended up enrolling somewhere else. By combining this with admissions and financial aid data that you have (e.g. high school grades, standardized test score, home state), and with public IPEDS data on institutional characteristics, you can graph the average characteristics of competitor institutions. It's not perfect--the IPEDS data is a couple of years out of date. Nevertheless, it's insightful.

There are many types of reports you can generate using this combination of data, but I'll focus on one that resembles a supply/demand relationship between the quality of student and the quality of the the attended institution. An example is shown below.

Here, the students are split into three categories: those who attended your institution (horizontal black lines), those who chose a private college other than yours (green), and those who went to a public institution. The levels under the circles indicate the academic preparedness as determined by the admissions process (usually a logistic regression model on HSGPA and standardized test scores). I've left the scale off of the graphs so as not to give away all our secrets, but the arrangement of dots in the scatterplot is real.  It shows that students generally sort themselves out pretty well, matching their academic ability (proxied by HSGPA) to the quality of the institution (proxied by graduation rate). Moreover, there is a pretty consistent gap in institutional quality that at least partially justifies the price difference (you can also create a scatterplot like this using graduation rate versus tuition). 

In my actual plot, I have a horizontal line that shows my college in relationship to these. Instead of showing you that, I've drawn three possible ones called case 1 through case 3. If your college's line looks like Case 1, you're not competing on quality because you're losing students to lower quality institutions. In Case 3, the opposite is true, where quality may be an important factor--definitely something to follow up with, e.g. the Admitted Student Questionnaire from the College Board. The middle line (Case 2) leads to a guess that your college is competing on quality with other privates and on something else (like price) for the mid-to-lower tier student, but that for the best qualified students, quality is still an issue. 

None of these conclusions is demonstrated here to be causal, of course. This is a starting point, not an ending point for analysis. But one of the advantages of this type of graph is that it's easily communicated to stakeholder, and generates the right kinds of questions about admissions requirements, marketing, and financial aid.

If you want to try this at home, I've already downloaded and combined some of the data you need. Here's a list:

• [download] Selected IPEDS categories from the most recent years (2010,11), with the OPE institutional ID included, so you can easily connect to your Clearinghouse data
• [download]A rudimentary data dictionary. I renamed columns to be more intuititve than the IPEDS ones.
• [download] A Perl script for joining tables without using a database. You'll need a little experience with Perl (you have to install a module or two), but it's a real time-saver.

Tips:

1. Remove cross-enrolled high school/college credit by filtering on the enroll date, so you don't accidentally pick up a community college a student earned credit in while still in high school.
2. Remove duplicates and generally give the data a careful look as you would for any such project.
3. I recommend Tableau as a reporting tool. It's expensive but worth it.

Feel free to contact me with questions, or leave a note below.

Edit: As a bonus, here's a scatterplot of just the IPEDS data, comparing the 75th percentile of SATR and SATM combined versus graduation rate (click to see a bigger version).

Raising the Standard

My cup hath run over this fall, with only enough pauses in the storm of activity to give me false hope that things are finally getting back to normal. Au contrair...in addition to the normal "whack a mole" quotidian routine, I have an ambitious web site overhaul to manage, a form solution to build into our implementation of OpenIGOR,  two grants to write, one paper to get to a publisher, another to finish (it keeps growing and growing), and several other projects that will have to wait. Oh, and I'm teaching a Calc II section. This last is the highlight of my week. Despite a one hour pre-class review and 75 minute class, I always come out of it with more energy that I went it with (starting at 3pm, mind you!).

But the idea that struck me last evening has nothing to do with these. Rather, it focuses on the problem of how to more efficiently manage a university's enrollment while it tries to raise standards. We are in the middle of such a transformation right now. The usual route is to raise admissions standards. This seems like the obvious thing to do. But HS GPA and SAT/ACT are blunt instruments that explain less than half the variance in first year grades. For less selective institutions this is particularly acute because the high non-completion rate represents a huge waste of time and money for those students who don't finish. At the same time, it represents lost opportunity for the university, because those seats potentially could have been filled by others who tested lower but would have performed better. Finding better predictors is one approach, and we are developing non-cognitive instruments to do that.

However, there is another idea I'd like to share. What if an institution were to accept that predicting success is not all that effective, and adopts the "try and see" approach? This is how it might work:

1. Adopt lower admissions standards, but still try to predict as best as possible
2. Admit double the number of freshman you would actually want to continue
3. Charge half price for the first year
4. Advertise and implement high expectations in the classroom, and foster high quality teaching and support programs: give students every chance, but accept no compromises or excuses for non-performance
5. Accept a 50% attrition rate for Freshman to Sophomore years. For those who are leaving, try to identify them early and have a Plan B outlined (community college perhaps)
6. Maintain high standards throughout the rest of the curriculum.
7. Advertise all of the above with a banner like: we give you the chance, make it affordable, and expect great things.

This would be a radical change, and would require a faculty that endorses the plan, upholds high standards, and constantly works to improve teaching and learning. The advantage of this approach is that you don't have to whittle down enrollment to increase standards--just the opposite.

2020 Gogs and Recruiting Costs

Everyone, it seems, is developing "2020" plans, a riff on the perfect vision score. The best quote I saw (on reddit) was that in 10 years we'll look back with 2020 hindsight. In that vein, The Chronicle has published an article called "The College of 2020: Students." You have to pay for the full report, but the executive summary is here (pdf). The abstract:

This is the first Chronicle Research Services report in a three-part series on what higher education will look like in the year 2020. It is based on reviews of research and data on trends in higher education, interviews with experts who are shaping the future of colleges, and the results of a poll of members of a Chronicle Research Services panel of admissions officials. ... Later reports in this series will look at college technology and facilities.

For the IR folks, Noel-Levitz has an interesting Cost of Recruiting report, which you can find in pdf form here. This gives good benchmarking information for assessing the efficiency of your admissions office.

Planning Resources

As we get into the season of setting tuition, finalizing aid policies, tweeking strategic plans, and prognosticating about enrollment, it's handy to have a reference library.

Some of these links have been cited before, but I wanted to put them all in one place. I will likely add to them without bothering to add "update" to the post, to make this a reference page of sorts. If you have a good one, please forward it.

Strategic planning:

• Six Essentials—and Six Common Mistakes—in Cabinet-Level Strategic Enrollment Planning
• NACUBO Benchmarking tool
• IPEDS Peer Comparison Tool
• NACUBO Financial Services Benchmarking Survey
  

Retention/Success:

• Noel-Levitz: 2009 Student Retention Practices and Strategies at Four-Year and Two-Year Institutions
• The Pell Institute: Publications on Opportunity and Successful Strategies
  

Recruitment:

• 2009 Student Recruitment Practices and Strategies at Four-Year and Two-Year Institutions
• Lawlor Group: The Year in Review College Admissions Averts a Crisis
• Lawlor Group: When Market Conditions and Public Perception Collide A Looming Crisis
  for Higher Education
• NACUBO:New Survey Predicts Enrollment Will Hold Steady at Independent Institutions
• Applywise.com: New Survey Indicates that College Bound Teens Will be Particularly Hard Hit by Economic Crisis
• IBM and Marist Survey Shows U.S. College Students Want Technology Skills to Compete for Jobs
• Index of whitepapers at Education Dynamics (requires free sign-up to download papers)
• 2009 Report on the Impact of the Economy On College Enrollment
• Index to SEM Works white papers
  

Pricing:

• Admission, Tuition, and Financial Aid Policies in the Market for Higher Education
• Research Tools to Guide Tuition and Financial Aid Decisions
• Overcoming price sensitivity … Means marketing affordability, and it's what every IHE needs to do - On The Money
• College Board: Trends in College Pricing 2009

Financial Aid:

• College Board: Trends in Student Aid
• Noel-Levitz: 2009 Discounting Report
• 2007 NACUBO Tuition Discounting Study Summary Results
  

Web Strategy:

• Scrolling Toward Enrollment Web Site Content and the E-Expectations of College-Bound Seniors
• Web Content Management Systems

Admissions as Drama

One of the (really excellent) IR staff here forwarded a meaty article on market research for higher education from the Lawlor Group entitled Trends & Tips for 2009. This research firm targeted at independent institutions bills itself as "intelligent marketing solutions for education."

Much of what's in the report is insight into the impact of the worldwide depression on the consumer market. The rise over the last decades in tuition is pegged at 439% since 1982, according to the report, compared to family income median rise of 147%. Even if we assume that discount rates went from zero to 33%, parents are paying far more now than before. And this situation is getting much worse because of the crash in equities and dwindling access to loans of all types.

Parents will do what shoppers always do when money is tight, the report argues: shop around for bargains. Quoting from one of the bullets in the article:

In October 2008, nearly 60% of surveyed high school seniors were considering a less prestigious college for affordability reasons; 14% changed their focus to a two year college; 16% put their college searches on hold. (MeritAid.com)

This is not the year to take your yield rate for granted. This is an important calculation many private schools will be making right now. How much do we shave off of our enroll/admit ratio when predicting next year's entering class?

It is already getting pretty late in the admissions cycle, but there's still time to change the road you're on. One of the recommendations of the Lawlor Group is not to take the value of the institution's product for granted. They recommend supporting it with evidence.

If your applicant pool includes first-generation students, the problem is likely to be worse. This is my own speculation, based on research done during the last several years. Lower-income and/or first generation families are likely to complete their taxes later, which complicates an already complicated FAFSA application. Awards can be put on hold while information that was put on the FAFSA as preliminary is updated and verified against returns when they come in. This is another whammy directed at low-income groups, in addition to low ability to pay, correlated lower entrance test scores, and dwindling need-based aid.

The form is supposed to come under review as one of President Obama's intiatives, but that will be too late for the current cycle. In the meantime, companies have sprung up that will help you with the form, for a fee comparable to H&R Block doing your taxes. One of them is www.fafsa.com, which charges $80-$100 to walk you through the form in under half an hour. That price may not be prohibitive to most middle-class families, but it is almost certain to be a limiting factor for some demographics.

Shopping for scholarships is another trend that is likely to accelerate, methinks. This is the time to be creative with institutional aid. Discount rates are almost certain to grow, unless formal tuition cuts become common. Here's a site that lists institutional awards, both endowed and unfunded: www.meritaid.com. Another barrier to unsophisticated consumers of the higher ed product is negotiating (in all senses of the word) the award hussle. Anything we can do to lower barriers and funnel students from one decision to the next, one bit of the bureacracy to the next, is a good thing.

Think of your application process as a movie script, where the FAFSA is smack in the middle of Act II. Is your film a long complicated thing or a fast-moving action flick? Who can best negotiate the plot twists--have you picked the right audience? Some tips come from the Lawlor Group's final recommendations:

Private colleges must continue their efforts to overcome students’ perceived financial barriers to enrollment by (1) demonstrating a commitment to containing costs, (2) making the net tuition price as transparent as possible as early as possible, (3) structuring institutional aid to increase the enrollment of low- and moderate-income students, and (4) implementing programs that narrow completion gaps across socioeconomic groups.

This is an outline of the story we have to tell in the 120 pages of our three acts (transforming the recommendations above into a dramatic arc of sorts):

1. It's cheaper than you think; you can afford us.
2. See, here are the costs.
3. And here's how we help you afford it.
4. The payoff is emotionally satisfying.

This year and the following ones will be a time of change, no doubt, and maybe it's not that crazy to take a page or two from Hollywood. Hey, maybe they'll start giving out Oscars for Best Admissions Process.

Admitting the Right Students

All of us who teach like to have students in the classroom who want to learn. In a recent conversation this topic came up. The math teacher I was speaking with said he'd take a hard-working curious student over an indifferent student any day--regardless of academic preparation of the latter. I would too. The real reward of teaching, after all, is to see growth. One of my proudest (shared) accomplishments is that one of my MAT 100 (really a remedial course) students fell in love with the topic and went on to graduate with a BA in mathematics. She now works in a technical profession and is very successful.

All this by way of introduction to my topic: how to judge applicants for admissions purposes? Academic preparation isn't enough if we care about things like motivation and perseverance. I've thought about and written about different angles of this topic for a long time, and now find myself in a position to do something about it.

The conventional wisdom, if there is such a thing, is that grades and SAT are reasonable predictors of achievement. These are called cognitive measures for reasons best known to the psychometricians. As I recall high school, there was a lot more to grades than cognition, but never mind. SAT is a favorite target for those who don't like this narrow thinking, and I'd count myself with the critics. I took the ACT in 1980 in Illinois, and remember liking standardized tests then. I think I got a 28 on the ACT, whatever that means. I also took the ASVAB military battery (in the sense of tests, not artillery), and actually found the results last summer when I was going through stuff in the garage. I remembered taking the test, driving to a National Guard armory in East Saint Louis with my friend Mark, following the lines marked on the floor to the various stations. One was an attitudes and behavior survey, where they discovered that I'd never smoked pot. They made fun of me for that, or else it was some kind of awe. This was 1978, and the stuff was everywhere. One station was the ASVAB, complete with number 2 pencils.

In the Army's eyes, I was being rated for potential by this test, and they took it very seriously. Frankly, I loved taking standardized tests because I always did reasonably well on them, and hence had no stress about the results. What's interesting about the ASVAB is the one domain where I did not do well at all. You can see on the image that there's a 55% bar in the middle. When I pulled this out of the box in the garage last summer I had to squint to read the faded type of the explanation. If you look on the far left, there's a CL designator: that's for "clerical work." The battery decided I'm no good at it. If I recall correctly, this part of the test included such things as counting how many Cs were in a line of Os. Something like this:

OOOOOOOCOOOOOOOOOOCOOOOOOOOCOOOOOOOOOOOOOOOOOOOOOOOOO

That's the sort of thing that drives me crazy. I will tip my hat to the test designers: I am in fact not well suited for mindless clerical work. Did the rest of the battery rate me accurately? I guess that depends on what the expected outcomes were. I wasn't at that time very good leadership potential, was a bit lazy academically from never having had to work hard in school, and didn't have a lot of ambition to go change the world. Yes, I was only 16 or so, but looking at the test results would still have vastly overestimated what kind of officer I would have made at that point. In short, there were a lot of imporant things that the test didn't measure. This anecdote underlines my main thesis about standardized tests: they are good for simple tasks but not complex ones. And there are many complexities to what makes a succesful college graduate. The specifications are likely different from institution to institution as well.

All this by way of introduction to an article I found while researching the link between SAT and race. The article from InsideHigherEd by Scott Jaschik dates from September 2008, and opens with:

[C]ritics of standardized testing — and especially of the SAT — have said that these examinations fail to capture important qualities [...]

What's interesting is that the College Board--creator of the SAT--agrees, and has an active research program for developing new approaches. This is good news and bad.

First it's good, because the old SAT "cognitive test" will lose some of its halo and the market place for talent can become more efficient. This is good for all of us. It's also good because it will put standardized testing--the modern day equivalent of phrenology, in my opinion--under more of a microscope. If policymakers have more sophisticated ways to think about achievement, it's beneficial to everyone.

It's also bad in a very selfish sense. This is because the institutions that are already acting on this market inefficiency will see their lunch being shared around the table. The article mentions a couple of these. Tufts University and Oregon State University are using non-traditional approaches to admissions according to the article. Of course, I'm not really serious about this--I'm very happy that there's competition for the meme that SAT is a real achievement score, and I'm confident that a real research program can keep us on the cutting edge of finding the most suitable students for our institution. Ultimately it's good to have competition, and for reasons outlined below, I think each institution will have to find its own solution anyway.

So what are the new methods looking for besides "cognitive" processes? The Group for Research and Assessment of Student Potential (GRASP) has twelve so-called dimensions they consider:

1. Knowledge, learning, mastery of general principles
2. Continuous learning, intellectual interest and curiosity
3. Artistic cultural appreciation and curiosity
4. Multicultural tolerance and appreciation
5. Leadership
6. Interpersonal skills
7. Social responsibility, citizenship and involvement
8. Physical and psychological health
9. Career orientation
10. Adaptability and life skills
11. Perseverance
12. Ethics and integrity

You can read descriptions of these by following the link to their site. In addition to GRASP, SAT is working on modifications to their instrument, which may result in a whole new test. I'm a bit dubious about this project, however.

There are already ways to game the SAT. How much more will this be true when the 'right' answers are clearer? That is, it's much easier to appear to have attractive behaviors and attitudes than it is to actually possess them. I can imagine a new preparation industry springing up to coach test-takers who can afford it. Ultimately I think the industrialization of this metric is doomed for this reason. That leaves individual admissions policies to find ways to gather information in ways that are less likely to be faked. The article makes the same point.

The results of SAT's trials with their new test items are interesting. By de-emphasizing academics in favor of the College Board's experimental "biodata" and "situational judgment", traditionally under-served minorities showed significant increases in enrollment. Particularly notable is an almost 6% increase in black enrollments at the highly selective institutions that participated in the experiment.

There are problems, like gaming the system, but the payoff is worth it for individual institutions. The article quotes Pamela T. Horne, who has been working with GRASP results. She gives voice to my feelings on the topic:

“This is mission-driven,” she said, noting that colleges don’t define their missions as “enroll students with high SAT scores,” but they do prize leadership, artistic vision and various other qualities that might now be measured.

Unfortunately, if you asked many college presidents and board members, they probably would say that high SAT scores are an institutional priority. Why else would there be all the concern about which scores are reported (first, last, all, average?). Ultimately, more enlightened institutions can take advantage of the limitations of the SAT and other standardized predictors by individualizing their own processes. It's an exciting challenge.

Update: Reading the comments on the article, I found a reference to a textbook on the subject of non-cognitive assessment. Here's the link on Amazon.com. Beyond the Big Test: Noncognitive Assessment in Higher Education (Jossey Bass Higher and Adult Education Series)
by William E. Sedlacek (Feb 26, 2004)

Visualizing Enrollment Change with Class Trajectories

Visualizations like graphs and animations are of obvious value in presenting data. One of the more useful ones I've found for visualizing enrollment is what I call a class trajectory graph, an example of which is shown below (with made-up data).

It shows three classes for four-year programs entering in fall 2000, fall 2001, and fall 2002. The right-most point on each of the trajectories is the first year for those students, and shows enrollment and net revenue per student (tuition minus institutional aid). Multiplying those two numbers gives net revenue for the whole class, which can be compared to the level curves graphed. So one can see that the freshman year for the 2000 entering class netted the institution around 9 million dollars. As time goes on, the classes get smaller because of attrition, so the next point on the blue line is to the left. Also note that net revenue per student fell somewhat. This is normal as students with leaner aid packages attrit at a higher rate. In the second year, the 2000 entering class netted the institution a little less than $7.5M. The entering 2001 class was bigger and produced more revenue per student, so the right-most pink point is up and to the right. It generated about 12 million its first year.

These trajectory graphs quickly convey important information. How bad is attrition, and what does it cost us in net revenue? How big are the classes comparatively? What are overall trends in class sizes--useful for planning everything from budgets to residence halls to class offerings. This is an essential graph for your strategic dashboard.

Bonus. Here are two other two visualizations just for fun. The first one [here] shows you how zip codes are clustered. Could be useful for something, I suppose, but mainly neat to play with.

The second [here] is just plain (or plane in this case) cool. The video shows world-wide flights over 24 hours.

Graduation Rates by Geographical Region

I'm still working on this retention plan, supplying supporting data and statistics. My previous experiments with geo-mapping attrition weren't entirely what I was looking for. I wanted some numbers to assign. So I took a large sample of students who would have had the opportunity to graduate by now, and ran their home cities through the yahoo pipes to geotag them. Then I grabbed that information as JSON formatted data, dumped it into a text file, and used a perl script to extract lon/lat information and their ID (which I'd included). I rounded the lon/lat down to the nearest integer, and found an atlas that showed the lines.

My perl script calculates graduation rates for each of these regions. Then I tediously marked all this up in Paint (yes, I know... I do have The Gimp installed, but it didn't seem necessary) as percentages. A bit of the result is shown below.

It's not shown on this snippet, but it's fairly obvious from the map that our two biggest markets could stand some improvements in the persistence department. Useful to know, and it's more quantitative that the other maps.