Finding and Using Predictors of Student Attrition

A while back, I wrote about finding meaning in data, and this has turned into a productive project. In this article I'll describe some findings on causes of student attrition, a conceptual framework of actions to prevent attrition, and an outline of the methods we use.

In order to find predictors of attrition, we need to find information that was gathered before the student left. A different approach is to ask the student why he or she is leaving during the withdrawal process, but I won't talk about that here. We use HERI's The Freshman Survey each fall and get a high response rate (the new students are all in a room together). Combining this with the kinds of information gathered during the admissions process gives several hundred individual pieces of information. These data rows are 'labeled' per student with the binary variables for attrition (first semester, second semester, and so on). In several years of data, and relying on two different liberal arts colleges, we get the same kinds of predictors of attrition:

• Low social engagement
• High financial need
• Poor academics
• Psychology that leads to attrition: initial intent to transfer, extreme homesickness, and so on.

These can occur in various combinations. A student who is financially stressed and working two jobs off campus may find it hard to keep up her grades. At the AIR Forum (an institutional research conference) this June, we saw a poster from another liberal arts college that identified the same four categories. They used different methods, and we have a meeting set with them to compare notes.

For purposes of forming actions, we assume that these predictive conditions are causal. There's no way to prove that without randomized experiments, which are impossible, and doing nothing is not an option. In order to match up putative causes with actions, we relied on Vincent Tinto's latest book Completing College: Rethinking Institutional Action, taking his categories of action and cross-indexing them with our causes. Then we annotated it with the existing and proposed actions we were considering. The table below shows that conceptual framework for action.

Each letter designates some action. The ones at the bottom are actions related to getting better information. An example of how this approach generates thoughtful action is given next.

Social Engagement may happen through students attending club meetings, having work-study, playing sports, taking a class at the fitness center, and so on. This can be hard to track. This led us to consider adopting a software product that would do two things: (1) help students more easily find social activities to engage with, and (2) help us better track participation. As it turned out, there was a company in town that does exactly this, called Check I'm Here. We had them over for a demo, and then I went to their place to chat with Reuben Pressman, the CEO and founder. I was very impressed with the vision and passion of Reuben and his team. You can click through the link to their web site for a full rundown of features, but here's a quote from Reuben:

The philosophy is based around a continuous process to Manage, Track, Assess, & Engage students & organizations. It flows a lot like the MVP idea behind the book "The Lean Startup" that talks about a process of trying something, seeing how it goes, getting feedback, and making it better, than starting over again. We think of our engagement philosophy the same way:

• Manage -- Organize and structure your organizations, events, and access to the platform.
• Track -- Collect data in real-time and verify students live with mobile devices
• Assess -- Integrate newly collected data and structurally combine it with existing data to give real-time assessment of what works and doesn't and what kinds of students are involved
• Engage -- Use your new information to make educated decisions and use our tools for web and mobile to attract students in new ways
• Rinse, and Repeat for more success!

A blog post talking more about our tracking directly is here. We take a focus on Assessing Involvement, Increasing Engagement, Retaining Students, and Successfully Allocating Funding.

Currently, we can get card-swipe counts for our fitness center, because it's controlled for security reasons. An analysis of the data gives some indication (this is not definitive) that there is an effect present for students who use the fitness center more than those who don't. This manifests itself after about a year, for a bonus of three percentage points in retention. The ability to capture student attendance at club events, academic lectures, and so on with an easy portable card-swipe system like Check I'm Here is very attractive. It also helps these things happen--students can check an app on their phones to see what's coming up, and register their interest in participating.

Methods 

I put this last, because not everyone wants to know about the statistics. At the AIR forum, I gave a talk on this general topic, which was recorded and is available through the organization. I think you might have to pay for access, though.

The problem of finding which variables matter among hundreds of potential ones is sometimes solved with  step-wise linear regression, but in my experience this is problematic. For one thing, it assumes that relationships are linear, when they might well not be. Suppose the students who leave are those with the lowest and the highest grades. That wouldn't show up in a linear model. I suppose you could cross multiply all the variables to get non-linear ones, but now you've got tens of thousands of variables instead of hundreds.

There are more sophisticated methods available now, like lasso, but they aren't attractive for what I want. As far as I can tell, they assume linearity too. Anyway, there's a very simple solution that doesn't assume anything. I began developing the software to quickly implement it two years ago, and you can see an early version here.

I've expanded that software to create a nice work flow that looks like this:

1. Identify what you care about (e.g. retention, grades) and create a binary variable out of it per student
2. Accumulate all the other variables we have at our disposal that might be predictors of the one we care about. I put these in a large CSV file that may have 500 columns
3. Normalize scalar data to (usually) quartiles, and truncate nominal data (e.g. state abbreviations) by keeping only the most frequent ones and calling the rest 'other'. NAs can be included or not.
4. Look at a map of correlates within these variables, to see if there is structure we'd expect (SAT should correlate with grades, for example)
5. Run the univariate predictor algorithm against the one we care about, and rank these best to worst. This usually takes less than a minute to set up and run.
6. Choose a few (1-6 or so) of the best predictors and see how they perform pairwise. This means taking them two at a time to see how much the predictive power improves when both are considered. 
7. Take the best ones that seem independent and combine them in a linear model if that seems appropriate (the variables need to act like linear relationships). Cross-validate the model by generating it on half the data, testing against the other half, do this 100 times an plot the distribution of predictive power.

Once the data set is compiled, all of this takes no more that four or five minutes. A couple of sample ROC curves and AUC histograms are show below.

This predictor is a four-variable model for a largish liberal arts college I worked with. It predicts student retention based on grades, finances, social engagement, and intent to transfer (as asked by the entering freshman CIRP survey).

At the end of this process, we can have some confidence in what predictors are resilient (not merely accidental), and how they work in combination with each other. The idea for me is not to try to predict individual students, but to understand the causes in such a way that we can treat them systematically. The example with social engagement is one such instance. Ideally, the actions taken are natural ones that make sense and have a good chance of improving the campus. 

Survey Addresses Drop-Outs

Public Agenda's recent report "With Their Whole Lives Ahead of Them" is subtitled "Myths and Realities About Why So Many Students Fail to Finish College." It's essential reading for anyone interested in student retention. Citing an average 40% six year graduation rate for four-year degrees, the report tries to answer the "why?" question with a survey of 600 young adults who had first hand experience. The complete methodology can be found here. The report is released under the creative commons license.

The demographic characteristics of those surveyed belie the image of a typical college student. Quoting from the article:

• Among students in four-year schools, 45 percent work more than 20 hours a week.
• Among those attending community colleges, 6 in 10 work more than 20 hours a week, and more than a quarter work more than 35 hours a week.
• Just 25 percent of students attend the sort of residential college we often envision.
• Twenty-three percent of college students have dependent children.

The main part of the report is framed around "myths and realities," such as:

MYTH NO. 1: Most students go to college full-time. If they leave without a degree, it’s because they’re bored with their classes and don’t want to work hard.

REALITY NO. 1: Most students leave college because they are working to support themselves and going to school at the same time. At some point, the stress of work and study just becomes too difficult.

According to the survey, "Those who dropped out are almost twice as likely to cite problems juggling work and school as their main problem as they are to blame tuition bills (54 percent to 31 percent)."

Graphs display ranked survey items. A portion is shown below.


The third section addresses an issue that I've discovered independently.

MYTH NO. 3: Most students go through a meticulous process of choosing their college from an array of alternatives.

REALITY NO. 3: Among students who don’t graduate, the college selection process is far more limited and often seems happenstance and uninformed.

My retention study at one institution showed that students whose had reported on the CIRP that they were at their first-choice college had three other interesting characteristics. One was that they tended to be first-generation students. They also were by far at the highest risk for attrition. And in a subsequent survey two months after the CIRP, many of them had changed their minds about the first-choice qualification. In short, they were uninformed consumers and became quickly disaffected--or at least, that's the way I interpreted the data.

The survey asked for proposals to help other students get a degree. The top responses are shown below.

The other two "myths" presented in the report are:

MYTH NO. 2: Most college students are supported by their parents and take advantage of a multitude of available loans, scholarships, and savings plans.

REALITY NO. 2: Young people who fail to finish college are often going it alone financially. They’re essentially putting themselves through school.

and

MYTH NO. 4: Students who don’t graduate understand fully the value of a college degree and the consequences and trade-offs of leaving school without one.

REALITY NO. 4: Students who leave college realize that a diploma is an asset, but they may not fully recognize the impact dropping out of school will have on their future.

Much more information and analysis is available on the report itself. I'm not crazy about post hoc retention surveys because if we want real predictors, we need to find out information before students leave, and afterwards causes and effects may change over time in the minds of the students, as with the first-choice question noted above. On the other hand, this report is thoughtfully done, and does seem to illuminate some interesting issues.

There is a section on what can be done to help. Providing more financial aid for part-time students is one, as well as more flexible options for attending. I presume that online courses would fill the second bill nicely. It seems obvious that more work-study options on campus would help too--a double win for the college, since students will be more engaged, and provide cheap labor. Addressing child-care problems for students ought to be high on the list too.

Assessing Happiness

In The Chronicle of Higher Ed, an article caught my eye entitled "What's an M.B.A. Worth in Terms of Happiness?" (Subscription required: The Chronicle hasn't caught up to the times yet). The premise:

Any sensible person would rather be happy than rich, although many people often confuse the two—business students among them. Those who choose to attend business school on the assumption that an M.B.A. will help them change jobs, make more money, and therefore be happier are very likely misinformed.

The author claims that there is a game theory problem--a Zog's Lemma instance, if you will--in business schools. I'll let author Robert A. Prentice explain:

Unfortunately, M.B.A. programs are currently ranked—by U.S. News, BusinessWeek, and other unduly influential publications—using criteria that prominently include starting salaries for graduates and salary differentials pre- and post-business school. Rankings have such an important impact on M.B.A. programs in their intense competition for students, faculty members, and resources that it is unsurprising that the schools often try to game them—say, by admitting students not because they are the strongest applicants, but because they are interested in finance and consulting, which have historically been the highest-salaried jobs.

An implication made in the article is that this undue emphasis on money comes at the expense of ethical behavior, which is linked to happiness:

Other studies indicate that people who act ethically tend to be happier than those who do not, suggesting that we are evolutionarily designed to derive pleasure from receiving the approval of others and from doing the "right" thing. Brain scans indicate that when we act consistently with social norms, primary reward centers in the brain are again activated.

Notice the evolutionary psychology argument.

Can you realistically assess happiness? The only book I've read on the topic is Dan Gilbert's "Stumbling on Happiness," which I recommend. It's not a self-helpy book, but a description of the research that has been done in the field of happology. Dr. Gilbert's website has links to other projects he's engaged in, including a link to an application to track your own happiness on your iPhone, which is a brilliant idea. You could do the same thing on Twitter, methings: one day a week send out tweets like #happymeter 5, for 5/10. Want to join me? Here's the link to results.

Maybe I think the idea is brilliant because I created a similar application a few years ago. Our student dropbox (a mini-portal) became wildly popular, and as a little research project I added a star rating system that looks like this:

There were no instructions, just the raters. On the login page, I then had the database calculate averages and display them. Here are the current ones.

From the beginning the trend has been school > life > world. There are about two years of data now, except that I did something stupid during the election and changed the rater to keep a running poll for a while. The graph shows a semester's worth of data, looking at change data week over week.
Self-reported happiness declined more in males than females as the semester wore on for that group of students who continually reported their ratings. This may be connected to higher attrition rates in males, which we also observed. I didn't have much luck in correlating the two, however. Other results showed that older students were significantly happier than younger students. The report is two years old now, but you can look at the whole thing here. After I read Dan Gilbert's book, I emailed him about this research. He replied with something like "Thanks for conditioning your students to take surveys!"

So what professions are the happiest? I found this relatively current (2007) report from the University of Chicago: "Job Satisfaction in the United States."

It's nice to see that education administrators and teachers are both on there. Who'd have guessed? The unhappiest workers of all turned out to be roofers. You can see the whole list and other observations in the article itself. Job satisfaction isn't exactly the same thing as happiness, but perhaps it's close.

So does money buy you happiness? Aside from the inner glow of helping old ladies with their 401ks, isn't there a rewarding feeling that comes from shopping for a new yacht? The (2001) study "Does Money Buy Happiness? A Longitudinal Study Using Data on Windfalls" tries to answer that question by looking at people who won the lottery or received an inheritance. From the abstract:

A windfall of 50,000 pounds (approximately 75,000 US dollars) is associated with a rise in wellbeing of between 0.1 and 0.3 standard deviations. Approximately one million pounds (1.5 million dollars), therefore, would be needed to move someone from close to the bottom of a happiness frequency distribution to close to the top. Whether these happiness gains wear off over time remains an open question.

My guess is that getting the money "for free" isn't as rewarding as say, scraping it off the top of predatory loans, but judge for yourself. I'd love to participate in the next study, regardless. Where do I sign up for the winning lotto numbers?

Update: See Reuben Ternes' comment below about Gross National Happiness. I also fixed a typo.

Targetting Aid

At the 2008 Assessment Institute I spent my time in sessions on first year seminars and retention strategies. I learned some interesting stuff. One was that at one institution where assignments were tracked, it wasn't the quality of student work that predicted retention, but the amount of it. I decided to test that at my institution by looking at our home-grown portfolio system statistics. I compared the average number of paper submissions by students who were retained to those who left for a single semester. There was no significant difference in our case.

Most strategies I saw targeted student engagement in one way or another. Activities like learning communities or work study increased the likelihood of student success. I asked questions about how retention committees worked with financial aid offices to fine tune awards. This was based on my own work here showing that grades and money are the two big predictors of attrition. No one I talked to had done such a thing, citing institutional barriers to efforts. Well, we've done it here, and had some limited success. Here's what we did.

The graph below shows the student population divided into total aid categories in increments of $3000. This is plotted against retention (dark line) for that group and (on the right scale) GPA for that group (magenta).

It's obvious that both grades and retention increase with financial aid, which says some interesting things about they way we recruit students, grant institutional aid, and provide academic support services. Ultimately this has led to a comprehensive retention plan I called Plan 9. But what we did immediately was focus on the group of students that have a decent GPA, but are historically showing low retention. That's the group circled on the graph. We targeted these students with small extra aid awards, and saw retention for that group sore to over 80%. I did a follow-up survey of the students receiving this aid to ask if it made a difference. My response rate was low, and the results lead me to believe there were other factors at work too. Maybe we just got lucky. But the results were so good we're trying it again this year.

Plan 9 includes lots of engagement stuff, and is really a comprehensive look at retention from marketing all the way through a graduate's career. A big part of it will focus on re-engineering aid policies. It always amazes me how budget discussions in the spring focus so much attention on tuition policies, when for private colleges at least, aid policies are much more important.