A Generalization of Value Added

It was probably coincidence, but about the same time I sent an email to the nice folks at CAE telling them about my blog post criticizing the marketing of the CLA, I got a newsletter from them called the collegiate learning pulse, for February 2007.

There’s too much interesting stuff in it for me to write about in one evening, so let me focus on one bit having to do with the relationship between SAT and CLA scores.

[S]ome believe that scores on tests of broad competencies would behave like SAT scores simply because there is a modest correlation between the two.

In footnote 6 of the referenced document, we find such a correlation:

When the school is the unit of analysis, the SAT by itself accounts for about 70% of the variance in performance task scores.

At least, I assume that’s the right one. Supposing that’s in the ballpark, the coefficient would be the square root of .7, which rounds up to .84. I would probably choose a different adjective than ‘modest’ for this level of correlation. But there’s a more salient point here.

[E]mpirical analyses of thousands of students show that the CLA’s measures are sensitive to the effects of instruction; e.g., even after holding SAT scores constant, seniors tend to earn significantly higher CLA scores than freshmen.

Seniors do better on the CLA than freshmen do when the SAT is the same. Two questions occur to me. First, if the freshmen and the seniors are different people (i.e. this isn’t longitudinal study) there are issues with survivorship that are important. You simply can’t compare a group of student who may or may not make it to their senior year to those who have done so demonstrably. At least, you shouldn’t do that without justifying it. Maybe that’s been done, but if it has been put to rest that fact needs to be mentioned with a reference. But I’ve raised this issue before. Enough already about survivorship.

The second thing is this. The freshmen are taking the CLA as freshmen, and the seniors as seniors. But they both probably took the SAT as high school students. So the argument that the CLA results are generally different from SAT scores doesn’t work unless you retest the seniors with the SAT. I suspect that those scores would increase too.

Let me pose this thought experiment. Suppose we administered the SAT to seniors and applied the same methodology that the CLA is using to calculate value-added outcomes. We’d have to find a different predictor, obviously, since SAT is currently be used to predict CLA scores as I understand it. We could probably use high school GPA as a measure of ability. If the results are shown to be roughly similar to the ones CLA is getting, could we conclude that SAT is just as useful as CLA for measuring value-added?

In other words, how general is the CLA method of calculating value-added scores? Can it be used with any standardized test of freshmen and seniors? Could the NSSE, for example, be used to calculate a value-added score for engagement using these methods?