Wednesday, November 19, 2008

Debating the value of SAT scores

A friend sent me a link to a New York Times article about the connection between student SAT scores and graduation rates in the SUNY system. Peter Salins, the author, poses the following question:
[D]o SATs predict graduation rates more accurately than high school grade-point averages? If we look merely at studies that statistically correlate SAT scores and high school grades with graduation rates, we find that, indeed, the two standards are roughly equivalent, meaning that the better that applicants do on either of these indicators the more likely they are to graduate from college. However, since students with high SAT scores tend to have better high school grade-point averages, this data doesn’t tell us which of the indicators — independent of the other — is a better predictor of college success.
He explains that the admissions standards of the SUNY system in the 1990s created a natural test as some institutions raised SAT requirements while others didn't, and high school GPA requirements remained roughly the same. The former schools saw significant increases in graduation rates. He concludes that those who wish to do away with SAT requirements are ignoring important information.

One question that the article doesn't answer is who exactly is graduating? Are the higher SAT scorers the ones who are responsible for the graduation rate increases? One would assume so, but it doesn't take much experience in institutional research to learn that you shouldn't assume such things. I ran our numbers to see what the situation is here. The graph below shows entering classes from 2000 to 2003 by quantized SAT, showing graduation rates. The students who didn't take the SAT (about half) graduated at the same rate as those who did, by the way.


We're a small school, so the two-standard deviation error bars are pretty intimidating, but I think we can see there's no support for the idea that the higher the SAT, the higher the graduation rate for our institution. On the other hand, actual grades earned are a good indicator. That doesn't help much for predictive purposes, but it shows that classroom accomplishments matter.

So for us, SAT is useful as a predictor in conjunction with high school GPA for predicting first year grade averages, but not much more than that. In fact, for our student population (half are first-generation college students) there's a good chance that the SAT underestimates their potential to graduate, leading them to be 'underpriced' in the admissions market compared to schools that put more emphasis on SAT.

Ultimately, it's more important that a student and the institution be a good match than that the student has a high standardized test score. SAT is a very blunt instrument. I think few would dispute that there are correlations with grades and graduation rates that can be useful predictors, but the real question is: is it worth the cost? Are there better ways to match applicants to institutions where they may have a better chance of finding what they want. I'm convinced that in our case there is. Our recent experience with the CIRP survey has convinced me that there are important variables we're not considering when we just look at grades and test scores. Behavior, attitudes, family support, and "cultural capital" are very important. My attitude toward SAT can be summed up in a newly-minted dictionary entry: meh.

4 comments:

  1. Does your data account for transfers?

    ReplyDelete
  2. It's been a while since I ran this, but I don't think I took out transfers, no.

    ReplyDelete
  3. Karl Munzenberger12:34 PM

    My brother, a college professor, has lived a totally academic life while I have spent an entire career in business. We completely disagree on the notion that any tax-supported institution of higher learning has the right to refuse admittance to any person seeking a college education. He argues that there must be some recognition, or process of identifying, those who will not succeed in advanced institutions to allow room for those who will succeed. Sounds logical but very elitist. I argue simple that American freedom includes the right to fail. Why should anyone be denied their “shot” at their dream of an education by a bunch of university eggheads. What say you??

    ReplyDelete
  4. Karl, If I understand correctly, this is a question of the balance of what's good for an individual versus what's good for the average taxpayer. For the individual, acceptance into any state-supported school is nice, but I think one could argue that the whole system becomes less efficient that way, which affects taxpayers in the aggregate. Also, I think that within any state system there are 'open admit' entry points, e.g. in the two year colleges at least. So the added benefit to the individual is not an entry point into higher ed--that already exists--it's the customized choice he/she wishes. But imagine that everyone in the state decided to go to the best school, and could? It would likely mean that the 'best' school ceases to exist as such. Would you argue that the taxpaying student should be allowed to play on the state college football team if he wants to?

    This is an interesting question. I'd be curious to know what others thought of it.

    ReplyDelete