Grade Transition Maps

There was a question on the ASSESS email list about evaluating grade distributions for courses. It's an interesting topic, and I dug out a data set and tried to construct some interesting non-parametric descriptions of it. Just looking at two years of fall and spring semester college grades, I first wrote a script to map transitions of grades that came in the same or earlier semesters with those that came later. You can download the code here (it's a perl script, but I had to save it as txt because of some weird Apache thing on the server).

The table below shows the fraction of grade transitions from the one on the top to the one down the side. So  of students who got a C one semester (C column) nine percent of them got a D the same or subsequent semester. The columns sum to 100% because only A-F grades are considered here. I could have included Ws, but I didn't.

Note that in this data set, there's no such thing as a "D student." Those who get Ds don't tend to repeat the experience nearly as often as the other grades. Instead they tend to get As and Bs more often than Ds and Fs.

The first table was for all grades and all courses. The next one is just for computer science courses, and rather than showing the actual percentages as I did above, I took the difference with the first table. So positive numbers mean that the transition is higher in computer science (colored green). Red and negative means a lower rate. You can see that B students are less likely to get As in computer science, for example.

The next three tables show the same thing for the disciplines shown.

Psychology gives a LOT of Ds--almost double the 'background' rate for A, B, and C prior grades. These are likely displaced Cs, because they give fewer of those. 

There's a nifty math trick we can play to do an average comparison. Imagine that a student took four classes in a row in the given discipline. The expected matrix of grade transitions would be the one I generated from the data, but raised to the fourth power. I ended up just doing this in Excel, but along the way I found a cool site for doing matrix math here. It will calculate eigenvalues and whatnot for you.

What happens is that the grade distributions converge and you can see the 'asymptotic' effect of the discipline grading scheme emerge, which is like a geometric average. This causes the matrix columns to converge to the same vector. I've taken those and differenced them against the overall transition matrix, and then put these "averages" side by side below in order to compare disciplines.

All of these disciplines push students away from the higher grades to the lower ones, with computer science accumulating Cs, and English and psychology accumulating Ds and Fs. 

Because you don't have the overall matrix to compare, here's a relative percent change over the 'background' rate of transition. The numbers below are no longer straight percentages of grade frequency (as in the table above). They are the percent change over the overall average rate for each cell. The most dramatic one is the psychology course's 68% increase in the production of D's over the 'background' transition rate. 

So if these subjects are all lowering grades, which ones are raising them? It wasn't hard to find one, but I won't name it. Someone might object. The table below shows the transitions on the right and the asymptotic behavior on the left. Everything is compared to the overall background rate.

You can see this discipline pushes grades from the bottom to the top, transitioning 4% more often to Bs over time, which is a 14% increase over the background rate. It's similarly disproportionate with the rate of Cs and Ds, giving 16% and 18% fewer than the background rate.

One could refine this type of analysis to look at majors courses only, or even individual instructors, although large sample sizes are obviously needed.