Evaluation Oddity

This year we changed our course evaluation form from a very long list of management questions ("Did the instructor meet office hours regularly?") to a short one focused on learning outcomes. The report below is from my fall 2010 Calculus II class.

 

That's the whole form, except for free-form written responses. As a first look at the meaningfulness of results, I looked at the correlation between responses for each item, and got a surprise. To wit: while items 9,10, and 11 are tightly correlated, as shown on the matrix below,

and furthermore, items 12 and 13 are correlated at .88,

 these two summative questions (12 and 13) do not correlate well at all with any of the other items, including 9, 10, and 11, which ask if the course was interesting, enjoyable, and did you learn. Do students not associate those things with their summative evaluations, or is there some other register that is engaged (i.e. they perceive that the last two questions are more personal, about the instructor instead of about themselves)? On the other hand, the very low correlation between item 2 (How much effort did you put into the course?) and the overall evaluation seems to be a good sign, implying that students differentiate between their own performance and that of the instructor.

Course Evaluations and Learning Outcomes

I've posted recently about some of our course evaluation statistics, and the effect of going from paper to electronic. A while back I also showed a summary of our Faculty Assessment of Core Skills learning assessment. I'm trying to put the two together by re-engineering the course evaluation to focus squarely on learning. The old version was a standardized one with fifty-five items, only five of which addressed learning at all, and these not very well. Here's my first draft of a new version, with comments afterward. The scale is indicated after each item. The exact wording is still in development.

1. What was the quality of instruction in this course as it contributed to your learning? (try to set aside your feelings about the course content)
   --(ineffective to very effective)
   
2. How much effort did you put into this course
   --(minimal to maximum)
   
3. How much did you know about the course content before taking the course?
   --(nothing to a lot)
   
4. How much do you know about the course content now?
   --(nothing to a lot)
   
5. How much your skills in analytical/deductive thinking (knowing facts, following rules and formulas, learning standard methods) increase in this course?
   --(none to a lot)
   
6. How much did your skills in creative/inductive thinking (trial-and-error, development of ideas, taking chances) increase in this course?
   --(none to a lot)
   
7. How much did your ability to speak effectively increase in this course?
   --(none to a lot)
   
8. How much did your ability to write effectively increase in this course?
   --(none to a lot)
   
9. How much did this course help you understand yourself?
   --(none to a lot)
   
10. How much did this course spark your interest in the content?
    --(none to a lot)
    
11. Was the course enjoyable?
    --(not at all to very much)
    
12. How much course content (the subject area, like chemistry or psychology) do you think you learned in this course?
    --(none to a lot)
    
13. What overall rating would you give this course as a learning experience?
    --(poor to excellent)

Comments.

This is a radical departure from what we do now. The first question is what we use now on the evaluation form, and is the only one used for evaluation. Question 13 is a validity check on it because the answers should be very much the same.

The questions all focus on learning, except numbers 2 and 11. In the old evaluation, almost all of the questions were about the process of teaching, which makes a lot of assumptions about the value of those processes, and doesn’t transfer well to styles like online learning or hybrid courses.

The learning questions are split between the content area and general liberal-arts skills. This gives us a natural complement to the Faculty Assessment of Core Skills (FACS), which we launched very successfully last spring. Taken together, the teacher view and the student view will give us excellent insight into gen ed outcomes across the whole curriculum.

Question 2 is included because it matches the one on the FACS. The noncognitive “effort” is very important to performance. Here’s the graph from the spring FACS, with GPA in red and credits earned in blue numbers. More effort means better grades and better chance of advancing.

Questions 3 and 4 get at how much content was learned by asking in terms of before/after. This is checked for reliability with question 12.

Questions 5-9 are about general learning outcomes. No course would be expected to get max scores in all of these—it’s an environmental scan to help us understanding where students feel what kind of learning is happening where. It complements the NSSE, the QEP, and the FACS, and will be a gold mine of information.

Question 9 is from the temple of Apollo at Delphi: “know thyself.”

Question 11 will raise some hackles, but it’s there as a control. We know from research that students who rate courses as enjoyable also rate everything else higher. This allows us to investigate that phenomenon locally. If we get to the point where we can administer electronically, we can do these studies ourselves by comparing to course grade. With an anonymous paper survey, we’ll have less ability to do that, but can still do intra-response correlations. We could be more direct and just ask “how happy are you right now?” but that would turn off some students.

There are two free-response questions we'll carry over from the old survey. This will let students write on topics they care about most.

The survey is short for two reasons. First, we’ll get better reliability because students won’t get survey fatigue. Second, this leaves room for other surveys customized by a program, department, or college, to be administered in parallel. For example, the Lit folks could ask detailed content-related questions if they wanted, or conversely ask all about processes (office hours, syllabus, etc.).

Course Evaluation Ranks

This is a follow-up to "Online Evals." In the spring we changed from paper student evaluations (using ETS's SIR II) done in class to electronic online evaluations (also SIR II). Some of these were done in class, with students bringing in laptops. Most were not, although we don't have an exact count.

One of the concerns expressed by faculty beforehand was that the scores would change. The previous post shows that the distribution of course averaged definitely changed from one skewed to the positive end to a more symmetrical one. It occurred to me that perhaps the most important question from a faculty point of view is what happens to the relative rankings of instructors when this happens. To find out, we calculated the average absolute difference |paper rank - online rank| for each faculty who was evaluated both semesters, using the single item on the evaluation that's used for administrative purposes (yeah, I know....). There were 82 faculty in the sample, so the greatest possible change was 81 ranks. The average for paper-paper semesters was 17, and for paper-electronic was 20 or 24, depending on which semester we used. The standard deviations were around 1.8, but the distribution is not normal, as you can see on the histogram below. It shows absolute rank change.

Generally, paper-paper performs better, meaning that the ranks are more stable than paper-online.