1. SES gets you a seat: An article in InsideHigherEd this morning tells that as access to higher ed boomed in the US, so did demand, so that competition naturally favored those best able to negotiate the barriers to admission. Guess who that was? This quote is attributed to the researcher, Sigal Alon:
[T]esting became a more important factor in admissions at more institutions, and that wealthier families are much speedier to adapt to changes in admissions rules.This sounds like the "economic value of error" stuff I was prattling on about a few weeks back. Although I can't find the source article online, a kind of appendix is here. She reiterates what everyone paying attention knows:
Test scores thus seem largely redundant with high school grades (Crouse and Trusheim 1988). Test scores do have some predictive value for college success, but is this improvement worth the negative impacts of testing?She goes on to advocate a more aggressive program than simply using noncognitives to better identify talent in lower SES groups. She's probably right--if only because the high SES groups will learn how to negotiate the new system of merit just as well as they did the previous one.
2. Rubics are bad for your brain: Well, maybe I exaggerate, but two articles posted to the ASSESS listserv provide cautionary advice for those constructing rubrics. On another level, there is (I find) a tendency to put too much faith in the things to begin with. Maybe that's because a rubric is tangible evidence of a plan. It looks like science--with its dissecting of some learning outcome into dimensions of quality and component. I won't reiterate my own annoyances with this quasi-religious belief here. You can find better and more constructive comments in the papers themselves:
3. NSSE Dimensions: At a recent IR meeting locally, I saw a very nice analysis of one institution's NSSE data from several years. The question being considered was whether or not the scorecard-type ratings that NSSE reports out are valid. You can see what I'm talking about in the 2008 annual report here. I clipped out a bit from page 8 to show some of the "dimensions" in question:
Here you can see that Level of Academic Challenge, Active and Collaborative Learning, and Student-Faculty Interaction are indices consolidated, summarized, and reported out as being meaningful. There is a great power to words, which gets abused routinely by the testing and assessment profession (like misuse of the word "measurement"). To be fair, it happens anywhere there's room for doubt--opportunity to advertise something more than what's there without being caught. We humans seem to prefer certainty over doubt, even if it guarantees we're wrong. But I digress.
The putative dimensions listed (Enriching Educational Experiences and Supportive Campus Environment are the other two, not shown above) are drawn from the items on the NSSE. In order to validate such things, one would test the construct validity using factor analysis. You can find lots of reports at NSSE referencing validity here. I will have to go through them later, to see what I can find in this vein.
The presentation at the IR meeting used confirmatory factor analysis (CFA), which posits that the dimensions are defined according to given rules, and then tries to verify that using the results at hand. Ideally, the items related to Academic Challenge would all show up as elements of a single vector when the correlation matrix's singular value decomposition is calculated. The actual results were not convincing.
I poked around to see if there were other studies out there, and found this one. Here, researchers at James Madison University and Eastern Mennonite University used similar methods and found similar results. They summarize their findings:
[A] CFA of the benchmarks as specified by NSSE literature produced poor model fit. For the sample of students examined for this study, the five-factor benchmark model supported by NSSE was not upheld, thus a five-factor solution is uninterpretable for the sample. A comparison of benchmark scores from this sample to scores from a sample at another university should not be made, as the benchmark scores from this sample were not empirically supported. Until further studies can provide evidence for an interpretable set of benchmarks for this and similar samples, one should not make policy or programmatic decisions based on the benchmark scores.The whole article is worth reading. It's well-written, and there are interesting details about particular loadings.