Higher Ed/

Assessing a QEP

2012-01-28T20:51:00.001-05:00

On Wednesday, Guilford College hosted a NCICU meeting about SACSCOC accreditation. I had volunteered to do a very short introduction to my experience with the Quality Enhancement Plan (QEP) at Coker College, since I had seen the thing from inception to impact report. I got permission from Coker to release the report publicly, so here it is:

Download Coker College's QEP Impact Report

The whole fifth year report passed with no recommendations, and the letter said nice things about the QEP, so it's reasonable to assume that it's an acceptable exemplar to use in guiding your own report.

Assessment of the QEP program is an important part of the impact report, and this is a good place to record how that worked. The QEP at Coker was about improving writing effectiveness in students, and we tried several ways of assessing success. Only one of these really worked, so I will describe them in enough detail so you don't repeat my mistakes. Unless you just feel compelled to.

Portfolio Review.
I hand-built a web-based document repository (see "The Dropbox Idea" for details) to capture student writing. After enough samples were accumulated, I spent a whole day randomly sampling students in four categories: first year/fourth year vs day/evening. There were 30 of each, for 120 students. Then I sampled three writing samples from each to create a student portfolio. There was some back and forth because some students didn't have three samples at that point. I used a box cutter to redact student names, just like I imagine the CIA does. Each portfolio got an ID number that would allow me to look up who it was. The Composition coordinator created a rubric for rating the samples, and one Saturday we brought in faculty, administrators, adjuncts, and a high school English teacher to rate the portfolios. We spent a good part of the day applying the rubric to the papers, and many of the papers were rated three times. All were rated at least twice by different raters.

The results were disappointing. There were some faint indications of trends, but mostly it was noise, and not useful for steering a writing program. In retrospect, there were two conceptual problems. First, the papers we were looking at were not standardized. It's hard to compare a business plan to a short story. Second, the rubrics were not used in the assignments, but conjured later when we wanted to assess. It's essential for rubrics to be effective that they be as integrated as possible into the construction of the assignment.

So this was a lot of work for a dud of a report, most of which is probably my fault.

Pre-post Test
One of the administrators decided we should apply a writing placement test, which we already had data for, as a measure of writing gain by giving it again as a post-test after students took the ENG 101 class. The assignment was to find and correct errors in sample sentences. The English instructors told us it wouldn't work and it didn't. More noise.

Discipline-Specific Rubrics
We did, in fact, learn something from the rubric fiasco. We allowed programs to create their own rubrics, which could be applied to assignments in the repository. So an instructor could look at a work, pull up the custom rubric, and rate it right there and then. Since the prof knew the assignment, this seemed like a way to get more meaningful results. I think this would have worked, but by the time we got all the footwork done, the QEP program was a couple of years under way. I left the college before it was possible to do a large-scale analysis of the results that were in the database. In summary: good idea, executed too late.

Direct Observation by Faculty
Back in 2001, when I got the job of being SACSCOC liason, I got a copy of the brand new Principles and started reading. The more I read, the more I was terrified. And nothing frightened me more than CS 3.5.1, the standard on general education. I didn't know at the time that the standard said one thing, but everyone interpreted in a completely different way (it was written as a minimum standard requirement, but everyone looked for continuous improvement). So I was one of those people you see at the annual meeting who look like they are on potent narcotics, drifting around with a dazed look at the enormousness of the challenge. (Note: I think they should hand out mood rings at the annual meeting so you can see how stressed someone is before you talk to them.)

In an act of desperation, I led an effort to create what we now call the Faculty Assessment of Core Skills (FACS), which is nothing more than subjective faculty ratings of liberal arts skills demonstrated by students in their classes. The skills included writing effectiveness. At the end of the semester, each instructor was supposed to give a subjective rating to each student taught for observed skills on the list. You can read all about this in the Assessing the Elephant manuscript, or on this blog, or in one of the three books I wrote chapters for on the subject.

Because we had started the FACS before the QEP, we had baseline data, plus data for every semester during the project's life. Thousands and thousands of data points about student writing abilities. When we started the FACS I didn't have much hope for it--it was a "Hail Mary" pass at CS 3.5.1. But as it turns out, it was exactly what we needed. We were able to show that FACS scores improved faster for students who had used the writing lab than those students who didn't. Moreover, this effect was sensitive to the overall ability of the student, as judged by high school grades. See "Assessing Writing" for the details.

I have given many talks about the FACS over the years, and get interesting reactions. One pair of psychologists seemed amazed that anything so blatantly subjective could be useful for anything at all, but they were very nice about it. When I post FACS results on the ASSESS-L list serve, you can hear the crickets chirping afterwards. I guess it doesn't seem dignified because it doesn't have a reductionist pedigree.

So I was shocked at the NCICU meeting, when SACSCOC Vice President Steve Sheeley said things like (my notes, probably not his exact words) "Professors' opinions as professionals are more important than standardized tests," and "Professors know what students are good at and what they are not good at."

The reason for my reaction is that when one hears official statements about assessment, it's almost always emphasized that it has to be suitably scientific. "Proven valid and reliable" is a standard formula, and certainly "measurable" figures (see "Measurement Smesurement" for my opinion on that). However it is stated, there isn't much room for something as touchy-feely as subjective opinions of course instructors. I do give good arguments for both validity and reliability in Assessing the Elephant, but FACS is never going to look like a psychometrician's version of assessment. So it was a shock and a very pleasant surprise to hear a note of common sense in the assessment symphony. I think when Steve made that remark, he assumed that this special knowledge professors acquire after working with students was simply inaccessible as assessment data. But it's not, and by now Coker has many thousands of data points over more than a decade to prove it. And it turned out to be the key to showing the QEP actually worked.

I have implemented the FACS at JCSU, and created a cool dashboard for it. I showed this off at the meeting, and you can download a sample of it here if you want. The real one is interactive so you can disaggregate the data down to the level you want to look at, even generating individual student reports for advisors. Setting up and running the FACS is trivial. It costs no money, takes no time, and you get rich data back that can be used for all kinds of things. Everyone should do this as a first, most basic, method of assessment.

Closed and Open Thinking

2012-01-25T06:22:00.000-05:00

Most readers will know William of Occam's principle about not multiplying eventualities unnecessarily. It's commonly thought of as "the simplest explanation is the best explanation." I learned about a countervailing principle in Arora and Barak's Computational Complexity: A Modern Approach. It's even older than the venerable Mr. Occam, dating back to the Epicureans, and it states that we should not abandon any explanation that is consistent with the facts. I have mentioned this before, but I had an interesting thought at lunch today: what if this tension between efficiency and open-mindedness is at the heart of the Dunning-Kruger effect? In case you've missed that bit of news, here's the introduction from the Wikipedia entry:

The Dunning–Kruger effect is a cognitive bias in which unskilled people make poor decisions and reach erroneous conclusions, but their incompetence denies them the metacognitive ability to recognize their mistakes.^[1] The unskilled therefore suffer from illusory superiority, rating their ability as above average, much higher than it actually is, while the highly skilled underrate their own abilities, suffering from illusory inferiority.

Actual competence may weaken self-confidence, as competent individuals may falsely assume that others have an equivalent understanding. As Kruger and Dunning conclude, "the miscalibration of the incompetent stems from an error about the self, whereas the miscalibration of the highly competent stems from an error about others" (p. 1127).^[2] The effect is about paradoxical defects in cognitive ability, both in oneself and as one compares oneself to others.

This just puts some research behind what Bertrand Russell is quoted as having said:

The trouble with the world is that the stupid are cocksure and the intelligent are full of doubt.

So what we have is two epistemologies, and we shouldn't be hasty to choose one as better than the other, despite the obvious bias of the quotes above.

Method 1 (Closed). Obtain a small amount of evidence, and create the most restrictive explanation that fits the facts. Subsequent facts that come to surface do not affect the conclusion.

William of Occam would probably sue me for defamation if he were around to read this. I have intentionally restated his principle in a very narrow sense in order to contrast it with:

Method 2 (Open). Continually gather information and create increasingly complex explanations that account for all the observations. Although the current explanation may be the simplest one that fits the facts, no explanation is ever final--all the others that are consistent with facts are kept in reserve.

I have given the methods intuitive names for convenience (closed vs open), not as prejudgments. The closed method will be the better one in situations where observations can be explained simply. This may be because the underlying cause and effect relationship is of low complexity, or perhaps that the variance in observed characteristics is small. "All dogs have four legs" would be an example of the latter. "Stuff falls when you drop it" applies to the former.

The most basic structure of language is a verb applied to a noun, which is a model for the closed epistemology. "Birds fly," "Fire burns," and so on, are summaries of real world observations that can be arrived at accurately from just a few examples and without much error. It's an easy conjecture that these simple relationships became so integral to understanding that exceptions were met with challenge. Such as: "If an ostrich doesn't fly, then it can't be a bird." This is what school children encounter when they learn that a whale isn't a fish. The language we use rather gracelessly allows these exceptions in the form of conjunctive appendices, but this is clearly a hack. I will suggest below that a formal language is required to overcome that difficulty (for example, expressions of formal logic, which defines a consistent way of using "or" and "and," and allows unlimited nesting of exceptions, so that any true/false relationship can be expressed unambiguously).

Quickly assembling a set of closed rules for a new environment seems like a good idea. It's a fast best-guess approach to finding useful cause and effect relationships.

Of course, the closed method is not suitable to doing science. Khun's The Structure of Scientific Revolutions suggests that closed outlooks solidify at any level of complexity, and require some bashing to break up. An example would be the certainty (due to Aristotle) that celestial bodies move in perfect circles. This is like Gould's idea of "punctuated equilibrium" in biological evolution. I graphed the associated relationship between predictability and complexity recently in "Randomness and Prediction."

The question is when to use the open versus closed approach. Historically, I think the closed approach may have had a blanket "explanation" in the form of mystical associations of cause and effect, which provides a putative low-complexity relationship. "Joe got struck by lightning because he displeased the weather god" has the appearance of an explanation, except that it's not actually predictive. It takes a dedicated effort to discover that fact, however. For that we need an open method.

The disadvantages of the open method make a long list. First, it's energy intensive--you have to continually be making observations, comparing what you see to what you think you should see (e.g. three-legged cat), and updating the every-growing explanation. It also takes more energy to use or communicate the current explanation, and as soon as you do, it's out of date again.

These are not fatal flaws, but ones to be considered. For some phenomena, this is probably how we naturally reason, if in a limited way. For example, our memory and minds do something like Bayesian reasoning (updating the probability of an event based on how frequently we encounter it), although our on-board system has been shown to be deeply flawed (see Daniel Kahneman's recent book, for this and a lot more).

Perhaps the open process needs a kind of empirical 'clean-up' to be really useful. Elegant explanations generally only work with clean data. That is, if you want to discover Newtonian mechanics, it's unlikely that you can do this with just your eyes and ears. When Galileo began measuring the "drop" times on an inclined plane, he was onto something.

In addition to a solid empirical methodology, an open method also needs a way to reduce the size of an explanation while retaining its predictive power. In my graphs in "Randomness and Prediction," I plotted predictability versus complexity, not size. It works like this.

Suppose I have an observed relationship that I have cataloged like this: (1,2), (2,4), (3,8), (4,16), where this might be thought of as a cause and effect. A one 'causes' a two, and so on. Because my empirical methods are sound, I trust that there's not too much error in the observed values. As the list grows by using the open method, I have a better and better 'explanation' of past events and a better and better predictor of future ones (fine print about the inductive hypothesis goes here...). But the list will become too unwieldy to remember, communicate, or use effectively, as the observations accumulate. What I need is a kind of data compression to reduce the list to a manageable size. If I do this correctly, the explanation doesn't change, nor does the complexity, but the size does. I can reduce it to effect = 2^cause if I have the idea of an exponential function. We might call this data reduction the creation of a formal theory.

Conclusions
I started by wondering if people who don't know things, and further don't know that they don't know them, could be attributed to one of the two epistemologies mentioned at the beginning. I think the argument above shows that it's possible that the two barriers of empiricism and abstract thinking needed to effectively use an open method are too formidable for a lot of people. For one thing, it's not hard to get by using closed systems, and it may require formal education in scientific method and meta-cognition to effectively use open systems.

One final note appropriate to the calendar in the US: it's a lot easier to communicate closed explanations than open ones. Even with data compression, "things fall" is less complex than Newton's laws. So in a debate made with sound bites from political candidates, the closed epistemology wins. It's easier, it's comfortable to the listener--the whole construct of English is build to 'hack' a closed way of thinking by adding a few contingencies ("Cats have four legs, but I once saw one with three.")--and the explanations take up less time to say. You have to expand "Drill!" into "Drill, baby drill!" to make it bigger because the basic message can be summed up in one word, and that may seem too short for some audiences as a serious thought.

This is just another reason why we should be deliberate about teaching science and meta-cognition in school, not as alien ways of thinking that only people in white coats use at work, but as the mode of thinking that differentiates us from the other mammals, and might allow us someday to collectively make good decisions.

Assorted Links

2012-01-22T07:50:00.003-05:00

You can file www.sightmap.com under "novel data representation." It's a heat map overlay of Google Maps that shows the most popular spots for taking photos, using the upload site Panoramio as the source.

This could be good fodder for a student research project. The only disappointment for me was not being able to zoom all the way down to street level resolution.

There's a new journal for those interested in the intersection of empiricism and computer science, in the spirit of Wolfram's A New Kind of Science. EPJ.org's new "Data Science" title seeks to address these challenges:

how to extract meaningful data from systems with ever increasing complexity
how to analyse them in a way that allows new insights
how to generate data that is needed but not yet available
how to find new empirical laws, or more fundamental theories, concerning how any natural or artificial (complex) systems work

Now I have one less excuse for not organizing my research notes into actual articles. While we're at it, here's a list of "Best Paper" awards in computer science.

Game theory is a fascinating and powerful set of ideas. Ever notice at the baggage carousel in the airport how everyone crowds up as close as they can, which means no one can see anything? If everyone took three steps back, the whole group would benefit. Paradoxes like these are the subject matter for this subject from mathematics and economics. There's a site that maps out the field in an easily accessible format. It's even easy to remember: GameTheory101.com.

While browsing for study tips for my daughter Epsilon, I found Study Hacks, with this bit of non-cognitive wisdom, originally quoted from a Reddit discussion thread:

The people who fail to graduate from MIT, fail because they come in, encounter problems that are harder than anything they’ve had to do before, and not knowing how to look for help or how to go about wrestling those problems, burn out.

The students who are successful, by contrast, look at that challenge, wrestle with feelings of inadequacy and stupidity, and then begin to take steps hiking that mountain, knowing that bruised pride is a small price to pay for getting to see the view from the top. They ask for help, they acknowledge their inadequacies. They don’t blame their lack of intelligence, they blame their lack of motivation.

Check out this guy's portfolio as a case study.

From University of Portland comes a fascinating case study "Why the Vasa Sunk: 10 Lessons Learned." From the introduction:

Around 4:00 PM on August 10th, 1628 the warship Vasa set sail in Stockholm harbor on its maiden voyage as the newest ship in the Royal Swedish Navy. After sailing about 1300 meters, a light gust of wind caused the Vasa to heel over on its side. Water poured in through the gun portals and the ship sank with a loss of 53 lives.

The rest is a case study in how not to manage a complex project. As Ashleigh Brilliant wrote, "It could be that purpose of your life is only to serve as a warning to others."

Finally, a more positive spin on leadership from The Atlantic: "Humble Leaders are More Liked and More Effective." Take it with a grain of salt (it's a small study), but be proud of your humility.

An Index for Test Accuracy

2012-01-19T20:19:00.000-05:00

This post is an overdue follow-up to "Randomness and Prediction," which takes up the question of how we should judge the quality of a test. There are many kinds of tests, but for the moment I'm only interested in ones that are supposed to predict future performance. Since education is in the preparation business, the measure of success should be "did we prepare the student?" If that question can be answered satisfactorily with a yes or no, this feedback can be used to determine the accuracy of tests that are supposed to predict this outcome.

As an example, I used the College Board's SAT benchmarks (pdf) , in which a test taken during high school years is used to predict first year college grades. The benchmark study is interesting because it is one of the few examples of test-makers who actually check the accuracy of their instruments and report that information publicly. You can find my first thoughts on this in "SAT Error Rates." The source material mainly consists of Table 1a on page 3 of the College Board report:

We can use this to see the power of the SAT to predict first year college grades at any cut-off score on the table. If we picked 1200, for example, we can see that 73% of the students we admit will have a first year grade overage at 2.7 or above. In other words, a 73% true positive rate and a 27% false positive rate. Because we are helpfully given the number of samples in each bin (the N column), we can also calculate the false positive and true negative rates for the test. Just multiply N by the percentage of students with FGPA > 2.7 to find the number of students in that bin who were successful in their first year (by that definition), and subtract that from N to get the number who were not. The graph below shows this visually.

The two graphs look roughly like normal distributions with means about 150 SAT points apart. This is all quite interesting, but for my purposes here I just want to pull one number from this: the total percentage of students with FGPA > 2.7, which we can get by summing up all the heights on the blue line and dividing by the total of all samples. This turns out to be 59%.

The College Board's benchmark has 65% accuracy. In other words:

If a student's SAT score exceeds the benchmark, there is a 65% chance they will have FGPA > 2.7
Of all students, 59% will have FGPA > 2.7

The difference between these numbers is not large: .65 - .59 = 6%. Using the benchmark to select "winners", we can do six percent better than just randomly sampling. If all we care about is the percentage of "good" students we get, that's the end of the story. But there's another dimension: the rate of unfair rejections, or false negatives.

If we randomly sample whom we accept, then 59% of those we reject would have had FGPA > 2.7 (assuming this is the rate for the whole population). Since it's unfair to reject qualified candidates, we might call 1-.59 = 41% the fairness of the method of selection. Another name for fairness is the true negative rate. I plotted it against the accuracy (true positive rate) in the previous article. Here it is again.

The blue line is accuracy, and the red line is fairness. They meet at 65%. So we can see that although using the SAT benchmark is only six percent more accurate than random sampling, it is .65 - .49 = 16% more fair. How do we make sense of how good this is?

One overall measure of test predictive power is the average rate of correct predictions, taking into account both true positives and true negatives. We might call that the "correctness rate" of the cut-off benchmark. Where the lines cross above in the above graph, both the rates for true positives and true negatives is 65%, so the correctness rate is also 65%. In general, the formula for the correctness rate c at a give cut-off benchmark is:

c = (number of actual positives that meet the benchmark + number of actual negatives that do not meet the benchmark) / (total number of all observations)

Below is a graph that adds the correctness rate to the accuracy and fairness plots.

The correctness rate potentially solves the problem of considering accuracy and fairness separately. It does not, however, give us an absolute measure to compare the quality of tests with. This is because the fraction of actual positives in the population can vary, making detection easier or more difficult. If we are interested in comparing different tests over different kinds of detection environments, we need something different. In the next section we will derive an index to try to address this problem.

A Comparative Index

In general, there is not a good way to turn results about predictability into results about complexity. However, using ideas from computational complexity, I stumbled upon a transformation that gives us another way to think about the predictive power of a test.

In order to proceed, imagine an even better version of the test. In this fantasy, a proportion p of the test benchmark results come back marked with an asterisk. Imagine that this notation means that the result is known to be true. The unmarked ones have no guarantee--some will be correct and some not. In this way we imagine separating out the good and useful work of the test in to the p group, whereas the rest is just random guessing.

It's just like a multiple choice test. Some answers you know you know, and others you guess at. By working backwards we can find that "known true" fraction:

Correctness rate = (fraction known correct) + (fraction not known correct)*(rate of correct responses with random sampling)

Using the numbers from the SAT benchmark in the previous section gives us:

.65 = p + (1- p) * .59

p = (.65 - .59)/(1-.59)

The fraction that would have to be "known true" is p = 14.6%. The advantage of this transformation is that we have a single number that is easy to visualize, and takes the context into account. If you wanted to explain it to someone, it would go like this:

The SAT benchmark prediction is like having a perfect understanding of 14.6% of test-takers and guessing at the rest.

The graphs below show the linear relationship between average test accuracy, the larger of the percent of positives or negatives in the population (the "guess rate"), and the index p--the equivalent proportion of "perfect understanding" outcomes.

The "guess rate" is just the bigger of the fraction of negatives or positives in the population. If there are more positives, then without more information, you would guess than any randomly chosen outcome would be positive. If there are more negatives, the best guess (without any other information) is that the outcome would be negative. In formulas, we will call this guess rate "r." For the SAT example, the real positive rate is 59%, so r = .59. If the real positive rate had been 45%, we'd use r = 1 - .45 = 55%.

As an example to illustrate the graph above, if the number of actual positives and negatives are evenly split at r = 50%, then a test that can predict with 80% correctness has the equivalent "perfect understanding" index of 60%. But if the proportion of positives is r = 70% instead of 50%, the index drops to 33%. It's reasonable to say that even though the correctness rate is the same, the first test is almost twice as good as the second one.

Note that if the guess rate equals the test accuracy, the test explains exactly nothing, which is as it should be.

Here's a general formula for computing the index p, which is the proportion of "perfect understanding" test results. The other two variables are c = the test's average correct classification rate, and r = the larger of the proportions of negatives or positive actual outcomes. In the SAT example, 59% were successful according to the FGPA criterion, so r = 59. If it had been 45% successful, then we'd use r = 1-.45 = 55%. Given these inputs, we have a simple formula for the index p:

p = (c - r)/(1 - r)

On the last graph, p is the height of the line, c is the bottom axis, and four values of r (guess rate) are given, one for each curve as noted on the legend.

(Note: edited 1/20/2012 for clarity)

Free Hypothesis-Generating Software

2011-12-16T07:39:00.000-05:00

In the last year there have been announcements of two free software packages that use machine learning techniques to mine data for relationships. The resulting mathematical formulas can be used to form hypotheses about the underlying phenomena (i.e. whatever the data represents).

The first one I have mentioned before. It's Eureqa from Cornell, which uses symbolic regression. There is an example on the Eureqa site that poses this sample problem:

This page describes an illustrative run of genetic programming in which the goal is to automatically create a computer program whose output is equal to the values of the quadratic polynomial x²+x+1 in the range from –1 to +1. That is, the goal is to automatically create a computer program that matches certain numerical data. This process is sometimes called system identification or symbolic regression.

The program proceeds as an evolutionary search. The graph pictured below is a schematic of the way the topology of the evolved "critters" is formed.

A family tree of mathematical functions. (Image Source: geneticprogramming.com)

There is a limitation to genetic programming that is also a threat to any intelligent endeavor: the problem may not be amenable to evolutionary strategies. There are some problems where the only way to solve them is exhaustive search. Only if the solution space is "smooth" in the sense that good solutions are "near" almost-good solutions is the genetic approach going to find solutions faster than exhaustive search. On a philosophical note, modern successes with physical sciences suggest that the universe is kind to us in this regard. The "unreasonable effectiveness" of mathematics (the title of an article by Eugene Wigner) in producing formulas that model real world physics is a hopeful sign that we may be able to decode the external environment so that we can predict it before it kills us. (The internal organization of complex systems is another matter, and there's not much success to look to there.). Note, however, that even here formulas have not really been evolutionary, but revolutionary. The formulation of Newton's laws of motion are derivable from Einstein's relativity, but not vice versa. The "minor tweak" approach doesn't work very often, Einstein's Cosmological Constant notwithstanding.

The second data miner is aptly called MINE, and comes from the Broad Institute of Harvard and MIT. You can read about it on their site broadinstitute.org. The actual program is hosted at exploredata.net, where you can download a java implementation with an R interface. Here's a description from the site:

One way of beginning to explore a many-dimensional dataset is to calculate some measure of dependence for each pair of variables, rank the pairs by their scores, and examine the top-scoring pairs. For this strategy to work, the statistic used to measure dependence should have the following two heuristic properties.

Generality: with sufficient sample size the statistic should capture a wide range of interesting associations, not limited to specific function types (such as linear, exponential, or periodic), or even to all functional relationships.

Equitability: the statistic should give similar scores to equally noisy relationships of different types. For instance, a linear relationship with an R2 of 0.80 should receive approximately the same score as a sinusoidal relationship with an R2 of 0.80.

It's interesting that this is a generalized approach to my correlation mapper software, the difference being that I have only considered linear relationships. For survey data, it's probably not useful to look beyond linear relationships, but I look forward to trying the package out to see what pops up. It looks easy to install and run, and I can plug it into my Perl script to automatically produce output that complements my existing methods. A project for Christmas break, which is coming up fast.

Update: I came across an article at RealClimate.org that illustrates the danger of models without explanations. Providing a correlation between items, or a more sophisticated pattern based on Fourier analysis or the like, isn't a substitute for a credible explanatory mechanism. Take a look at the article and comments for more.

By coincidence, I am reading Emanaul Derman's book Models.Behaving.Badly: Why Confusing Illusion with Reality Can Lead to Disaster, on Wall Street and in Life. It has technical parts, which I find quite interesting, and more philosophical parts that leave me scratching my head. The last chapter, which I haven't read yet, advertises "How to cope with the inadequacies of models, via ethics and pragmatism." Stay tuned...

Update 2: You can read technical information about MINE in this article and supplementary material.

Randomness and Prediction

2011-12-10T10:16:00.001-05:00

I saw a question at stackoverflow.com asking why computer programs can't produce true random numbers. I can't locate the exact page now, but here's a similar one. The question spooked around in my head all day, despite my head-down work to catch up on paperwork after being away at the SACSCOC meeting (see the tweets). After coming home, I finally gave in and wrote some notes down on the topic. It has applications to assessment, believe it or not.

According to complexity theory, "random" means infinitely complex. Complexity is the size of a perfect description of, for example, a list of numbers. If we are given an infinitely long list of numbers like

1, 1, 1, 1, 1, ....

it's easy to see that it has very low complexity. We can describe the sequence as "all ones." Similarly, the powers of two makes a low complexity sequence, or any other simple arithmetic sequence. We could create a more complicated computer program that tries to produce numbers that are as "mixed-up" as possible--this is what pseudo-random number generators do, but if we have access to the program (i.e. the description of the sequence), we could perfectly predict the numbers in the sequence. It's hard to call that random.

Truly random numbers (as far as we know) come from real-world phenomena like radioactive decay. You can have a certain amount of this so-called "entropy" for free from internet sources like Hotbits. I use such services for my artificial life experiments (insert maniacal laugh here). Real randomness is a valuable commodity, and I'm constantly running over my limit for what I can get for free from these sites. Here's a description from their site of where the numbers come from:

HotBits is an Internet resource that brings genuine random numbers, generated by a process fundamentally governed by the inherent uncertainty in the quantum mechanical laws of nature, directly to your computer in a variety of forms. HotBits are generated by timing successive pairs of radioactive decays detected by a Geiger-Müller tube interfaced to a computer.

What would be involved if you wanted to predict a sequence of such numbers (which will come in binary as ones and zeros)? As far as we know, radioactive decay is not predictable from knowing the physical state of the system (see Bell's Theorem for more on such things).

Even in a mechanical system such as a spinning basket of ping-pong balls, like those used for selecting the winning numbers in lotteries, a complete description of the system that is sufficient to allow you to predict which balls will emerge to declare the winner would be a very long set of formulas and data. In other words, even if it's not infinitely complex, it's very, very complex.

But what if we want partial credit? This was the big idea that had half my brain working all day. What if we are content to predict some fraction of the sequence, and not every single output? (Like "lossy" compression of image files versus exact compressors for executables.) For example, if I flip a coin over and over, and I confidently "predict" for each flip that it will come up heads, I will be right about half the time (in fact, any predictor I use is going to be right half the time). So even with the simplest possible predictor, I can get 50% accuracy.

Imagine that we have an infinitely complex binary sequence S-INF that comes from some real source like radioactive decay. We write a program to do the following:

For the first 999,999 bits, we output a 1 each time. For the one-millionth bit we output the first S-INF bit, which is random. Then we repeat this process forever, with very long strings of 1s followed by a random one or zero. Call this sequence S-1

It should be clear that a perfect predictor of S-1 is impossible with a finite program. It's still infinitely complex because of the interjection of bits from S-INF. But on the other hand, we can accurately predict the sequence a large portion of the time. We'll be wrong once in every two million bits, on average, if we just predict that the output will be 1 every time.

There is a big difference between randomness and predictability. If we take this another step, we could imagine making a picture of prediction-difficulty for a given sequence. An example from history may make this clearer.

Before Galileo, many people presumably thought that heavy objects fell faster than lighter objects. This is a simple predictor of a physical system. It works fine with feathers and rocks, but it gives the wrong answer for rocks of different weights. Galileo and then Newton added more description (formulas, units, measurements) that allowed much better predictions. These turned out to be insufficient for very large scale cases, and Einstein added even more description (more math) to create a more complex way of predicting gravitational effects. We know that even relativity is incomplete, however, because it doesn't work on very small scales, so theorists are trying ideas like string theory to find an even more complex predictor that will work in more instances. This process of scientific discovery increases the complexity of the description and increases the accuracy of predictions as it does.

Perfect prediction in the real world can be very, very complex, or even infinitely complex. That means that there isn't enough time or space to do the job perfectly. As someone else has noted (Arthur C. Clark?), some systems are so complex that the only way to "predict" what happens is to watch the system itself. But even very complex systems may be predictable with high probability, as we have seen. What is the relationship between the complexity of our best predictor and the probability of a correct prediction? This will depend on the system we are predicting--there are many possibilities. Below are some graphs to illustrate predictors of a binary sequence. Recall that a constant "predictor" can't do worse that 50%.

The optimistic case is pictured below. This is embodied in the philosophy of progress--as long as we keep working hard, creating more elaborate (and accurate) formulas, the results will come in the form of better and better predictors.

The worst case is shown below. No matter how hard we try, we can't do better than guessing. This is the case with radioactive decay (as far as anyone knows).

The graph below is more like the actual progress of science as a "punctuated equilibrium." There are increasingly large complexity deserts, where no improvement is seen. Compare the relatively few scientists that led to Newton's revolution or the efforts of Einstein and his collaborators to the massive undertaking that is string theory (and its competition, like loop quantum gravity).

Note that merely increasing the complexity of a predictor is easy. The hard part is figuring out how to increase prediction rates. You can always make a formula or description more complex, but by doing so it doesn't guarantee that the predictors are any better. Generally speaking, there is no computable (that is, systematic or deterministic) method for automatically finding optimal predictors for a given complexity level. You might think that you could just try every single program of a given complexity level and proceed by exhausting the possibilities, but you run into the Halting Problem. There are practical ways to tackle the problem though. This is a topic from the part of computer science called machine learning. A new tool that appeared this year from Cornell University is Eureqa, a program for finding formulas to fit patterns in data sets using an evolutionary approach.

Next time I will apply this idea to testing and outcomes assessment. It's very cool.

X-Raying Survey Data

2011-12-09T10:15:00.001-05:00

I continue to develop and use the software I patched together to look at correlates (or covariates) within large scalar or ordinal data sets like surveys. I have gotten requests from several institutions in and out of higher ed to do these. A couple of interesting graphs that resulted are shown below, with permission of the owners of the data, who shall remain anonymous. Both of these are HERI surveys. I have found the HERI surveys the most revealing, partly because they discriminate so well between different dimensions. Some other surveys seem to produce (in the data sets I've seen) big globs of correlated items that are hard to get meaning from.

First the CIRP Freshman Survey at a private college. It neatly divides up the survey respondents into clusters. Rich urban kids negatively correlated to working class or middle class kids, athletes, the religious, and the environmentally-conscious all show up clearly. I've labeled the optional questions with an approximation of the prompt.[Download full-sized graph]

Next is the Your First Year College Survey at a different private college. I find the link between texting in class and recommending the school to others particularly interesting. That's at the bottom. Red lines are negative correlations.[Download full-sized graph]

Higher Ed's 1%

2011-12-09T07:07:00.001-05:00

I found this chart at the Chronicle very interesting. It shows the ratio of presidential pay to average professor pay. For example, at Stevenson University, it's 16 to 1.

My rule of thumb is that the quality of an institution is ranked by Instructional Costs/FTE. Here are some selected institutions from the right side of the chart, along with their "Z-scores", courtesy of CollegeResultsOnline (2009 data).

Rubrics as Dialogue

2011-12-02T07:22:00.001-05:00

In the last few articles beginning with "The End of Preparation", I have contrasted two epistemologies. One proceeds by definition, which I called monological, and the other emerges from dialogue. These are distinct and equally useful ways of understanding the world. We could display the discipline of physics as a very successful monological system. It allows scientists and engineers to model physical systems and design systems that will work as intended. It allows us to understand what the sun is, and where the atoms in our bodies came from, all as part of a broad model that uses consistent monological language. This successful union of reality, language, and model is what makes the physical sciences so powerful. With such a language we can think precisely about complexity, for example, which is the size of the description of some physical system. A jar of marbles being shaken is complex because each marble's position and velocity are different, requiring a long list of physical attributes to be listed. If they are all at rest in a square shape, the description in physical language can be compressed, and the state is less complex. Physicists would refer to this as high entropy versus low entropy.

By comparison, the language and ways of knowing that create popular culture is dialogical. There are no rules set down about what new words (or memes, if you prefer) will arise, and no deterministic rules that could be applied to predict cultural evolution. A stock exchange has some elements of monologism (precise definitions regarding financial transactions, for example), but the evolution of prices is dialogical--unpredictable consensus between buyers and sellers.

One of the characteristics that distinguishes a monological language from a dialogical one is that in the former case, the names can be arbitrary. What matters is their relationship in the model that's used for understanding the world. For example, electricity comes in Volts and Amperes, and its power is measured in Watts. These are names of scientists, as are the Ohm, Henry, and Farad, terms that refer to electrical properties of circuit elements. If they were dialogical names, they would more likely be "Zap," "Spark," and "Shock" or something similarly descriptive. This is because in a dialogue, it's an asset for words to be descriptive--you don't have to waste extra time saying what it is you meant. By contrast, it's enough to know that V=IR when calculating voltage in a circuit. It doesn't matter whether we call it Volts or Zaps.

It's a trope to poke fun at academics who speak in high-falutin' language just to say something ordinary. When Sheldon in Big Bang Theory gets stuck on a climbing wall, he says "I feel somewhat like an inverse tangent function that's approaching an asymptote," which is then reinforced by his desperate follow-up "What part of 'an inverse tangent function approaching an asymptote' did you not understand?" [video clip] Some might argue that some academic disciplines that are inherently more dialogical use language that's unnecessarily opaque. This point was publicly made in the "Sokal Affair," where a scientist submitted a jargon-laden meaningless paper to a humanities journal as a hoax, and it was published.

Using Rubrics for Assessment
In order to connect these ideas to the assessment practice of using rubrics, let me first review what they are.

The term "rubric" in learning outcomes assessment means a matrix that indexes competencies versus accomplishment levels. For example, a rubric for rating student essays might include a "correctness" competency, which is probably one of several on the rubric. There would be a scale attached to correctness, which might be Poor, Average, Good, Excellent (PAGE), or one tied to a development sequence like "Beginning" through "Mastering." In our Faculty Assessment of Core Skills survey, we use Developmental, Fresh/Soph, Jr/Sr, Graduate to relate the scale to the expectations of faculty.

A rubric alone is not enough to do much good. A fully-developed process using rubrics might go something like this, starting with developing your own.

Define a learning objective in ordinary academic language. "Students who graduate from Comp 101 should be able to write a standard essay that uses appropriate voice, is addressed to the target audience, is effective in communicating its content, and is free from errors."
The competencies identified in the outcomes statement are clear: voice, audience, content, and correctness. These define the rows of the rubric matrix.
Decide on a scale and language to go with it, e.g. PAGE.
Describe the levels of each competency in language that is helpful to students. It's better to be positive than negative--that is, define what you want, not what you don't want when possible. There are many resources on constructing rubrics you can consult. The AAC&U's VALUE rubrics are examples to refer to.
The rubric should be used in creating assignments, and distributed with the assignment, so the student is clear about expectations. Use of rubrics in grading varies--it's not necessary to tie an assessment to a grade, but there are some obvious advantages if you do.
Rating the assignment that was designed with the rubric in mind should not be a challenge. If it's essential to have reliable results, then multiple raters can be used, and training sessions can reduce some of the variability in rating. Nevertheless, it's not an exact process.
Over time you create a library of samples to show students (and raters) what constitutes each achievement level.

Note that the way to do this is NOT to take some rubric someone else has created and apply it to assignments that were not created with the rubric in mind. That's how I did it the first time, and wasted everyone's time.

Rubrics as Constructed Language

The learning objectives that rubrics are employed to assess are often complex, so even though an attempt is made to define the levels of accomplishment, these descriptions are in ordinary language. That is, there's no formal deductive structure or accompanying model that deterministically generates output ratings from inputs. Instead, the ratings rely on the judgment of professionals, who are free to disagree with one another. If your attitude is that there is one true rating that all raters must eventually agree on, you're likely to be frustrated. One problem is that although the competencies, like content and correctness in the example, are not independent. If there are too many spelling and grammar mistakes on a paper to gain any sort of comprehension, content, style, voice, and so on are also going to be degraded. One rater of writing samples I remember was adamant that a single spelling mistake implied that all other ratings would be lowered as well.

So using rubrics is dialogical, but by the way of a nice compromise. The power in rubrics comes from restraining the language we use to describe student work, according to a public set of definitions. Even though these are not rigorous, they are still extremely useful in focusing attention on the issues that are deemed important. In addition, rubrics create a common language in the learning domain. It's important for students not to just know content, but how professionals critique content, and rubrics are a way to do that. They can be used for self-reflection or peer review to reinforce the use of that language.

The advantage of generating useful language is one reason I only use a PAGE scale as a last resort. Terms like poor, average, and so on are too generic, and too easily made relative. An excellent freshman paper and an excellent senior paper should not be the same thing, right? Bad choices in these terms early on can have long-term consequences when you want to do a longitudinal analysis of ratings.

There is a tendency among some to view rubric rating as a more monological process, but I can't see how this can be supported for most learning outcomes. In my opinion, they are most useful in creating a common language to employ in teaching, to rein in the vast lexicon that might naturally be used and focus on the elements that we agree are the most important. This has positive benefits for everyone concerned.

Chef's Salad

2011-12-01T07:34:00.001-05:00

Here's another serving of link salad. The articles referenced connect to recent topics of discussion.

HERI just released a report on college graduation rates. They give details on regressions to predict completion, and provide the rates of correct classification rates for same. Here's an example:

Note that SAT scores don't add any information once high school GPA is accounted for. The correct classification rate can be compared to the rate of graduates. For example, if a test correctly predicts a coin flip 50% of the time, on the face of it this isn't very impressive. But it's actually more complicated than that. I have a kind of complexity theory approach to this sketched out on scrap paper, and will write about that later. In this case, the rate of four-year graduation from page 7 of the paper is 38.9%, so a correct classification rate of 68.3 could be compared to the strategy of predicting that no student will graduate, which is correct 61% of the time. Even by this crude comparison, the predictor looks useful.

HERI provides an associated calculator that lets you try out different scenarios related to graduation. Very cool.

The Virginia Assessment Group just published their new edition of Research & Practice in Assessment. I hope to be able to read it on the plane tomorrow, on the way to the annual SACS-COC meeting in Orlando. Last time I was there I got to see a shuttle launch, which was amazing. Not likely this time.

Coursekit is new learning management system that wants to be more like social media. This relates to the topic of connecting professional portfolios to a social network. I learned about Coursekit in this Wired Campus article in The Chronicle. Even more intriguing to me is Commons in a Box, a separate open source project to create professional networks. Quoting from the article in The Chronicle:

Educational groups, scholarly associations, and other nonprofit organizations will be able to leverage the Commons in a Box to give their members a space in which to present themselves as scholars to the public, to share their work, to locate and communicate with peers, and to engage in collaborative scholarship.

The original source is the CUNY Academic Commons.

Link Salad

2011-11-28T06:40:00.001-05:00

A Monday's worth of interesting education-related links:

On non-cognitives, we have two articles from the Boston Globe. The first is "How College Prep is Killing High School":

A number of economists, including Nobel economist James Heckman, have documented the need for noncognitive or so-called soft skills in the labor market, such as motivation, perseverance, risk aversion, self-esteem, and self-control.

The second is "How Willpower Works":

In dozens of studies conducted over the past 25 years, Baumeister has found that taking on specific habits - like brushing your teeth with the opposite hand you’d normally use - can increase levels of self-control. In a phone interview, he likened willpower to a muscle: “If you exercise it, you can make it stronger. There’s nothing magical about it.’’

Then there is the less optimistic offering from the New York Times "The Dwindling Power of a College Degree," which contains a warning for all of us:

A general guideline these days is that people are rewarded when they can do things that take trained judgment and skill — things, in other words, that can’t be done by computers or lower-wage workers in other countries.

The Wall Street Journal has a scorecard of career salaries by degree, in case you're keeping score. The highest 75th percentile salary goes to math and computer science combined. Compare it to math education:

A partial listing of the WSJ salary/major list found here.

The quote in the New York Times article about computers replacing us is especially interesting when juxtaposed to the ambitious research plan described in "Mining the Language of Science," from Phyorg.com:

Scientists are developing a computer that can read vast amounts of scientific literature, make connections between facts and develop hypotheses.

Stanford University is offering a free online course on machine learning if you want to learn how to make a computer smarter than yourself (true story).

To round out that topic, here are two articles on the limits of human understanding. First from Physorg.com again is "People are Biased against Creative Ideas, Studies Find," including these findings:

Creative ideas are by definition novel, and novelty can trigger feelings of uncertainty that make most people uncomfortable.
People dismiss creative ideas in favor of ideas that are purely practical -- tried and true.
Objective evidence shoring up the validity of a creative proposal does not motivate people to accept it.
Anti-creativity bias is so subtle that people are unaware of it, which can interfere with their ability to recognize a creative idea.

The second article, from SciGuru, is "Ignorance is bliss when it comes to challenging social issues."

The less people know about important complex issues such as the economy, energy consumption and the environment, the more they want to avoid becoming well-informed, according to new research published by the American Psychological Association.And the more urgent the issue, the more people want to remain unaware [...]

This illustrates the mechanism I described in "Self-limiting Intelligence." You can test yourself on these last two points. Here's a creative idea from Business Insider, and a challenging social issue from The Economist. Good luck!

Assessments, Signals, and Relevance

2011-11-23T08:18:00.001-05:00

In "Tests and Dialogues" I promised to address the use of rubrics, which I get to. But before I do, I want to extend the ideas presented in the last few articles. By coincidence, my daughter provided an example the same day I wrote the article.

My daughter Epsilon had a math test and a French test yesterday, so naturally I asked how they went. She had spent quite some time reviewing (with my imperfect help in the French class), and said that the tests were easy except that she forgot what the degree of a polynomial is (ugh!). She said she was able to guess at some things she didn't know, which made my eyebrows rise. Guess? Sure, she says, it's almost all multiple choice. Here I began to sputter. What?? Algebra and French...multiple choice? Yes, says she, it's because of the EOCs. That would be the local name for the monological "End of Course" state tests. Since the EOCs are multiple choice, and there is so much weight put on them, it makes economic sense to optimize all testing to resemble "the ones that matter." She's 14, and this is old hat to her by now.

The wrong assessments plus a factory mentality optimizes local relevance at the cost of global irrelevance. David Kammler wrote a marvelous parable in this vein, "The Well Intentioned Commissar." Achieving goals can be inherently very complex. When we try to grasp their workings by simplifying cause and effect (e.g. in order to manage like a factory), we can lose important information. This is detrimental when optimizing the simplified problem is not the same as optimizing the original problem. The impact is not merely academic. I read a story in The Economist years ago that went like this:

A state in the US was spending more on road repair that it thought reasonable, and sought to make the situation more equitable by passing cost onto the owners of the heavy trucks that were doing the most damage. So they instituted an axle fee--the more axles the truck had, the higher the cost to the truck owner to use the roads. This was a simple approximation: the heavier the truck, the more axles. What could go wrong? The outcome was that truckers, not being stupid, started using trucks that carried just as much weight, but on fewer axles. This increased the ground pressure of the trucks (same weight over less area) and damaged the roads even more than before. In this case they didn't merely optimize irrelevance, but actually exacerbated the problem they were trying to fix.

Even being irrelevant has an associated opportunity cost. The time spent learning how to game multiple choice tests could be better spent. We can only imagine what the long term cost is, when students finally figure out that real problems don't come with a built-in 20% chance of guessing the right answer.

It is not a coincidence that all of this ties together with the idea in "Self-Limiting Intelligence," where the problem I tried to illuminate about intelligent systems is that self-change easily turns into self-deception. My last couple of articles have mostly ignored the fact that there are powerful motivations lurking behind official definitions. Here's an example of how motivation subverts definitions from theNewspaper.com, which calls itself "a journal of the politics of driving."

Automated ticketing vendor American Traffic Solutions (ATS) filed suit Tuesday against Knoxville, Tennessee for its failure to issue tickets for turning right on a red light -- and that is costing the company a lot of money. A state law took effect in July banning the controversial turning tickets, but the Arizona-based firm contends the law should not apply to their legal agreement with the city, which anticipated the bulk of the money to come from this type of tickets.

If this seems silly, here's a more disturbing example:

Judge Mark A. Ciavarella and former Senior Judge Michael T. Conahan are accused of taking $2.6 million for sending children to two [correctional] facilities owned by Pittsburgh businessman Greg Zappala.

Privatizing a corrections facility created an economic value for criminal offenders, which increased the supply of such through more application of the monological standard by judges. This is what juries are there to prevent--provide a dialogical check.

In higher education, the definition of educational success adopted by the policymakers is "enrollment," and "graduation," for which the state pays plenty. See my previous article "Flipping Colleges for Profit," for how that turns out in the hands of private investors who seek to maximize dollars/student. We maximize enrollment and (perhaps) graduation at very large monetary expense to the taxpayer in grants and loans to students who default at high rates. This counterproductive effect is evidence of an over-simplified index of success.

It would be understandable if you took away from the discussion so far that monological = bad and dialogical = good, but that's not the case. Systems have to function monologically most of the time. I base this on a simple argument:

Systems of all kinds exist. Some work, some don't. The ones that survive do so in part because they are motivated to survive. The actions a system takes to survive can be ultimately reduced to binary "do this/don't do that" decisions. Motivations drive those decisions based on information from the internal and external environment. This reduction of complex data into a simple binary decision we might call an assessment, and the result is a 'signal.' Pain when you stub your toe is a "don't do that" signal corresponding to the implicit motivation to avoid bodily harm.

If we put together all these signals, they comprise a language. If it works, it models the environment and allows the system to survive (eat this, don't eat that). The diagram that goes with a motivation-driven decision loop is the one I discussed in "Self-Limiting Intelligence."

The point is not that monological motivation-driven signals are bad for us, it is that we have to use the right ones if we want to succeed. Sometimes dialogues get turned into signals, as in a plebiscite or anywhere else where public opinion matters. Marketing is another example, but in reverse--working from a motivation to try to affect dialogue so someone can sell more soap. In those cases, lots of energy is spent in trying to affect conversations. The BBC's recent article "Fake forum comments are 'eroding' trust in the web" is an example.

Part of the decision about what assessment to use to create signals should be driven by the consideration that weighing it down with economic value will probably degrade the quality. This problem is ubiquitous. It includes counterfeiting, cheating, and corruption of all sorts. It even shows up in natural selection, as Darwin figured out, in sexual selection--explaining peacock feathers, for example. It is perhaps embodied in the advice "you have to fake it to make it."

In order to make a decision, a system has to process a potentially infinite amount of data for a few clues as to what will accomplish its goal (fulfilling motivations). This assessment is a massive data compression that, if it's done well, describes in signal-language the important elements of the environment relative to motivation.

An example will illustrate the point. First, let's look at the role of complexity and assessment. I took the photos below at the Cape Fear Serpentarium in Wilmington, North Carolina. (It's a fantastic place to visit, along with Fort Fisher and the nearby aquarium if you're in the area.) This is the Gaboon Viper, and I first read about it in a zoo--I think in Columbia, South Carolina. It's a big, slow snake that prefers to sit and wait for lunch to walk or hop by. If you're a small animal, the picture below shows your perspective:

The Gaboon Viper presents a high-complexity look to prey.

In its natural environment, the snake's colors perfectly fit into the surrounding forest floor. The complex patters of light and dark break up the shape of the form, so that a rodent is unlikely to correctly assess this information and form the signal SNAKE! The snake presents a high complexity visual presentation to the world, and is rewarded for this concealment by a reduced probability that a rodent will correctly assess the situation.

And a low-complexity "here I am!" to large beasts.

However, the snake has another problem. It's got a great lifestyle, sitting around waiting for dinner to walk by, but there are also large hooved beasts that are far too large to eat and impossible to get out of the way of when they come ambling by. It's a good thing to be hidden from small critters that one might consume, but quite another to be hidden from a huge monster that might step on you and break your spine! So the presentation for a viewer looking down on the serpent needs adjustment. Instead of concealment, it wants to create an instant assessment in the bovine brain of SNAKE! As you can see in the photo on the right, this is accomplished (via natural selection, of course) by white stripes that look like the center of a highway. I looked for research that actually demonstrates that cows can see these snakes better than ones without such coloration, but didn't come up with anything. So treat this as informed speculation rather than fact, unless someone can point me to an authoritative source. But the effect is real. In the family of large cats, some females have white dots on the backs of their ears so they can present a low-complexity "follow me" sign to their kittens in low light. Military vehicles do something similar so they don't run into one another in the dark, but also don't make good targets.

To continue the example:

Suppose you are going out for an afternoon hike in a tropical jungle. You check into the matter and see that there are deadly poisonous snakes in the area. Fortunately, there is a guy whose job it is to monitor the jungle nearby for such threats, and post a sign at the trail head with a warning when appropriate. You may rightfully be dubious. There is a very large tract of undeveloped forest out there, and how plausible is it that this guy--who may be the governor's favorite nephew, for all you know--could have checked for every possible snake? So you are not reassured when you see a big green NO SNAKES TODAY THANK YOU sign nailed to a tree. In effect, you've rejected the data reducing assessment and have decided to create your own signals. That is, the final assessment for "is there a snake here?" remains pending. You carefully watch where you step, tying up a large part of your mind to continually test the environment against your snake-matching perception. This is a lot of work and quite stressful, so you give it up and go to lunch instead.

The example shows the trade-off in an early or late assessment into a signal. Early signals make subsequent decisions easier. That's why most businesses like stability.

So the big question for a complex system is when to do the assessment for a given motivation. There are symmetrical arguments for and against early decisions based on limited data:

Early assessment from data to signal:

Pro: If we assess early, we reap the economic benefit similar to mass production. All decisions that depended on the first one can now go about their business. By mandating that 21 is the legal age to drink alcohol, it creates a simple environment for liquor stores, as opposed to say administering an on-the-spot test for "responsible drinking" for each customer.

Con: Creating the signal greatly simplifies the actual state of the world. If subsequent decisions need detailed information, it won't be available. Worse, the signal may be wrong entirely.
"Housing prices never fall" was an early assessment that led to a lot of unfortunate consequences.

Con: Signal manipulation for economic benefit (what we would call corruption in a government) can cause a wide-spread disconnect from reality. Adopting unproven test scores as the measure of educational success creates a false economy and doesn't reflect the actual goal.

Late assessment from data to signal:

Pro: Information isn't lost before it's needed. The example of walking in the jungle illustrates this.

Pro: Local corruptions of signals have only local effects. From a virus's point of view, it's ever-changing protein coat means that it can't be intercepted easily. On the other side, those who have to decide on a vaccine have a difficult decision to make about which variants to target.

Con: It's more costly because decisions are made individually instead of in mass production style. Every state has a different set of paperwork for allowing truckers to use the roads, which impedes commerce. Conversely, internet sales are aided by not having to worry about every locality's rules.

There's no one right answer. The cost for deferring assessment can be very high. The railroad owners finally created time zones to solve the problem of every town running on a different clock. A common currency is obviously good for everyone. Standardized traffic laws are a boon. Formalized ownership of property is a prerequisite for a modern society. All of these involve monological definitions that are somewhat based on early assessment of evidence and are somewhat arbitrary. In some cases, it can be completely arbitrary and still immensely helpful (e.g. which side of the road we drive on). Imagine what a mess it would be if we had to survey the dialogical landscape every morning to see whether most people were driving on the right or left, stopping at green lights or red, and make our adjustments accordingly.

Applying this to Higher Education
Education at all levels has to process so many students that good organization is essential. So it has to be a system, and there are going to be a lot of semi-arbitrary decisions made just to provide a workable system language. There are many, many of these, and those of us who labor within the system take it for granted that these things exist:

Courses
Grades
Credit-hours
Grade levels or course level designations
Set times for instruction (e.g. one hour lecture, which is really fifty minutes)
Set curricula
Degrees and diplomas

None of these have much directly to do with learning. The pressure to define what a credit hour means for online courses shows the rigidity of the system. It's worth a moment to look at one of these in detail, so let's follow that train of thought into the dialogical tunnel. Compare the Carnegie fifty-minute block class with how we naturally learn.

The Kahn Academy comprises a large group of tutorials on YouTube started by Sal Khan (who quit managing a hedge fund to do this), and now operated by an impressive team. The home page claims that over 87 million lessons have been delivered, with more than 2700 videos on offer covering a range of academic subjects and levels. The one shown in the picture is about long division. You can see that it's a little less than 10 minutes long. Why ten minutes? Why not fifty minutes? Some are longer and some shorter, depending on the needs of the subject.

The curriculum in the Khan Academy is not a set of courses with prerequisites. It's much more natural than that, using a "knowledge map" to show connections between the ideas taught in the videos. Here's a sample:

There are suggestions for prerequisites, but they are per topic, not per course. Each of these has associated problems to be solved, challenges, and badges to earn. There are individualized feedback reports, and the ability for coaches to be involved.

I recently signed up for a free course on Machine Learning taught out of Stanford University. The lectures were online with a built-in quiz in each one. The videos are of varying length, but none close to fifty minutes. I skipped stuff I was already familiar with and browsed topics I didn't know as much about. Once I had to go back to the introductory material to clarify a point, had my "aha!" moment, and then forged ahead. To me this is a very natural way to learn. It's not at all systematic. I would never have had the time to sit through a traditional lecture course on the subject, but with the browsing ability I have with the online presentation, I can choose what I want off the menu and maximize the use of my time.

What's the point?
If we create systems that make early decisions for learners so that we can make the logistics work, we save time in administration but suffer opportunity cost for each student. This "early assessment" model leads to preemptive decisions solidified into policy and practice: courses, semesters, grades, and so on.

A "late assessment" version would be to provide just-in-time instruction for each student so that it could be used in some authentic learning situation. By authentic I mean anything that doesn't cause the student to think "when will I ever use this?" For example, individual or group projects that require learning the content, but perhaps only a piece at a time on demand.

The early collapse of complexity into a simple bureaucratic language includes the factory-like quality checks that occur along the way in the form of (increasingly standardized) tests. This early assessment presents problems for consumers of the product: employers and the graduates themselves, and society as a whole. The educational system doesn't give much information about what the students have learned other than these formalized assessments. It's like the NO SNAKES TODAY THANK YOU sign.

Before the Internet, late assessment methods would have been too expensive to use for the whole system. Not anymore. We have the opportunity to adopt a new model whereby we coach students on how to learn for themselves. So much the better if this learning is in the context of some interesting project that the student can show off to peers and (if desired) the world. The creation of a rich portfolio along the way allows employers or anyone else with access to make their own assessments.

I do not think that the massive higher education system will change this radically anytime soon. Nor are we ready to make such a change. Something like the Kahn Academy for every academic discipline would be a massive undertaking. Textbook publishers, if they are forward thinking, may lead the way. Imagine an online "textbook" that was actually a web of interrelated ideas mapped out transparently, with video lectures and automated problem solvers associated. This frees up what I once called the low bandwidth part of the course and allows for more creative official class time. Effectively, it offloads all the monologue to out-of-class time and lets you get to the dialogue directly.

There is an opportunity for programs here and there to begin to experiment with this transition. As the examples in my previous articles show, this is already happening. In the most optimistic case, this prevents further solidification of the factory mentality in higher education by showing valid alternatives.

Tests and Dialogues

2011-11-22T07:13:00.001-05:00

In "The End of Preparation" I argued that standardized tests, as they exist now, are not very suited to the task of correctly classifying quality of the partial products we call students. Certainly the tests give us information beyond mere guessing, but the accuracy (judging from the SAT) is not high enough to support a factory-like production model. I pointed out that test makers do not usually even attempt to ascertain what the accuracy rate is. Instead we get validity reports that use a variety of associations. If we brought that idea back to the factory line, it would look something like this.

Announcing the new Auto Wiper Assessment (AWA). It is designed to test the ability of an auto to wipe water off the windshield. Its validity has been determined by high negative correlations with crash rates of autos on rainy days and low correlation on sunny days.

On a real assembly line, the question would be as simple as Does it work now? and Are the parts reliable enough to keep it working? Both of these can be tested with high precision. And of course, we can throw water on the windshield to directly observe whether the apparatus functions as intended. Direct observation of the functional structure of learning is not possible without brain scanners. Even then, we wouldn't really know what we are looking at--the science isn't there yet. What we do know is fascinating, like the London Taxi Cab study, but we're a long way from understanding brains the way we understand windshield wipers.

Validity becomes a chicken-and-egg problem. Suppose our actual outcome is "critical thinking and complex reasoning," to pick one from Academically Adrift. There are tests that supposedly tell us how capable students are at this, but how do we know how good the tests are? If there were already a really good way to check, we wouldn't need the test! In practice, the test-makers get away with waving their hands and pointing to correlations and factor analyses, like the Auto Wiper Assessment example above. This is obviously not a substitute for actually knowing, and it's impossible to calculate the accuracy rate from the kind of current validity studies that are done. The SAT, as I mentioned is an exception. This is because it does try to predict something measurable: college grades.

This is not a great situation. How do we know if the test makers are selling flim-flam? In practice, I think tests have to "look good enough" to pass casual inspection, and they can amount to neo-phrenology without anyone every knowing. How else can the vast amount of money being spent on standardized tests be explained? I'd be happy to be wrong if someone can point me to validity studies that show test classification error rates similar to the SAT's. A ROC graph would be nice.

The argument might be that since reductionist definitions are not practical, and there really is no way to know whether a test works except through indirect indications like correlations, this is the best we can do. But it isn't. In order to support that claim, let me develop the idea by contrasting two sorts of epistemology. It's essential to the argument and also worth the exposition for its own sake. When I first encountered these ideas, they changed the way I see the world.

Monological Knowing
Sometimes we know something via a simple causal mechanism: an inarguable definition. For example, when the home plate umpire calls a strike in a baseball game, that's what it is. It doesn't matter if the replay on television shows that the pitch was actually out of the strike zone. Any argument about that will be in a different space--perhaps a meta-discussion about the nature of how such calls should be made. But within the game, as veteran umpire Bill Klein is quoted as saying "It ain't nothin till I call it!"

Monological definitions are generally associated with some obvious sign. An umpire jerking his clenched fist after a pitch means it was a strike. Sometimes the definitions come down to chance, as with a jury trial. In the legal system, you are guilty if the jury finds you guilty, which has only indirectly to do with whether or not you committed a crime. The unequivocal sign of your guilt is a verdict from the jury. Other examples include:

Course grades, defining 'A' Student, 'B' Student etc.
Time on the clock at a basketball or football game, which corresponds only roughly to shared perception of time passing (perceived time doesn't stop during a time-out, but monological time can).
Pernicious examples of classifying a person's race, e.g. leading up to the Rwandan genocide. You are what it says you are on your documents.

Sometimes the assignments are random or arbitrary. Sometimes a single person gets to decide the classification, as with course grades. There is sometimes pressure from administrators to create easily understood algorithms for computing grades in order to handle grade appeals, but instructors usually have wide latitude in assigning what amounts to the monological achievement level of the student.

I got bumped from a flight one time, and came away from the gate with the knowledge that I was "confirmed" on the next flight. That didn't mean what I thought it did, however. According to the airline's (monological) definition, "confirmed" means that the airline knows you are in the airport waiting, so you're a sure seat if they have an extra. It does not mean that such a seat is guaranteed for you.

Dialogical Knowing
This might be more properly called polyphonic, but for the sake of parallelism, allow me the indulgence. In contrast to a monological handing down of definitions from some source, dialogical knowledge has these characteristics:

It comes from multiple sources
There isn't universal agreement about it (definitions are not binding if they exist)
It's subjective

Whereas there is a master copy of what a Kilogram is in a controlled chamber in France, there is no such thing for the concept of "heavy." A load you are carrying will feel heavier after an hour than at the beginning of the hour. Furthermore, we can disagree about the heaviness. This is messy and imperfect, but very flexible because no definitions are needed. Anyone can create a dialogical concept, and it gets to compete with all the others in an ecology where the most fit survive. This fact is what prevents loose shared understanding from devolving too far into nonsense as a whole. There's plenty of nonsense (like fortune-telling), but we can communicate in a shared language very effectively even in the absence of formal definitions.

If I tell you that I liked the movie Kung Fu Panda, you know what I mean. There are movies you like too, and you probably assume I feel about this movie the way you feel about those is some vague sense. You may disagree, but that's not a barrier to understanding. We could have a complex conversation about what constitutes a "good" movie, which doesn't have a final, monological answer. In Assessing the Elephant I compared this to the parable of the blind men inspecting an elephant, each sharing their own perspective. I used this as a metaphor for assessing general education outcomes, which are generally broad and hard to define monologically.

Tension between Monologue and Dialogue

Parallel to the tension between accountability and improvement in outcomes assessment, there is a tension between monological and dialogical knowledge in any system. The demand for locked-down monological approaches is the natural consequence of being part of a system, which as I described last time, needs to manage fuzziness and uncertainty in order to function. That's why we have monological definitions for what it means to be an adult, or "legally drunk." It makes systematization possible. Much of the time, this entails replacing a hard dialogical question ("what is an adult?") by a simple monological definition ("anyone 21 years or older"). In ordinary conversation we may switch these meanings without noticing, but sometimes the tension is obvious.

The question "which candidate will do the best job in office?" gets answered by "which candidate got the most votes?" It replaces an intractable question with one that can be answered systematically in a reasonable amount of time Of course it's an approximation of unknown validity. Monologically, the system decides on the "best" candidate, but the dialogical split on the issue can be 49% vs 51%.

Someone put together an page describing the relationship between monological Starbucks definitions of drink sizes and the shared understanding of small, medium, large. The site, which you can find here, is a perfect foil for this discussion. I find it hysterically funny. Here's a bit of it:

The first problem is that Starbucks is right, in a sense. I've established that asking for a "small coffee" gets you the 12-ounce size; "medium" or "medium-sized" gets you 16 ounces; and "large" gets you a 20 ounce cup. However, in absolute rather than relative terms, this is nuts. A "cup" is technically 8 ounces, and in the case of coffee, a nominal "cup" seems to be 6 ounces, as indicated by the calibrations on the water reservoirs of coffee makers, [...]

When a referee makes a bad call in a sports event, the crowd reacts negatively. The dialogical "fact" doesn't agree with the monological one, which is seen as artificial and not reflecting the reality of shared experience.

It may be appalling, but it makes sense that the Oxford English Dictionary now includes the work "nucular" as a synonym for "nuclear." This is the emodiment of a philosophy that the dictionary should reflect the dialogical use of language, not some monological official version.

In assessment, it's quite natural to fall victim to the tension between these two kinds of knowledge. As noted, tests of learning almost never come with warning labels that say This test gives the wrong answer 35% of the time. The test doesn't have any other monological ways of knowing to compete with, other than possibly other similar tests, so by default the test becomes the monological definition of the learning outcome. Because it replaces a hard question ("how well can our students think?") with an easily systematized one ("what was the test score?") it's attractive to anyone who has to watch dial and turn knobs in the system. In the classroom, however, the test may or may not have anything to do with the shared dialogical knowledge--that messy, subjective, imperfect consensus about how well students are really performing.

A Proposal to Bridge the Gap
Until we better understand how brains work, it's not realistic to hope for a physiology-based monological definition of learning to emerge to compete with testing. However, it would be very interesting to see how well tests align with the shared conception of expert observers. This doesn't seem to be a standard part of validity testing in education, and I'm not sure why. It's in everyone's best interests to align the two.

There is a brilliant history of this kind of research in psychology, culminating in the definition of the Big Five personality traits, which you can read about here. From Wikipedia, here is the kernel of the idea:

Sir Francis Galton was the first scientist to recognize what is now known as the Lexical Hypothesis This is the idea that the most salient and socially relevant personality differences in people’s lives will eventually become encoded into language. The hypothesis further suggests that by sampling language, it is possible to derive a comprehensive taxonomy of human personality traits.

Subjective assessments have a bad reputation in education, but lexical hypothesis was shown to be workable in practice. It's not astounding that dialogical language has meaning, but it doesn't seem fashionable to admit it.

Given all this, it's obvious that we should at least try to understand the resemblance between monological tests of "critical thinking and complex reasoning" or "effective writing" and the dialogical equivalent. It's simple and inexpensive to do this if one already has test results. All that's required is to ask people who have had opportunity to observe students what they think. Any way it turns out, the results will be interesting.

Suppose the test results align very well with dialogical perceptions. That's great--we can use either tests or subjective surveys as we prefer.

If the two don't align, then we have to ask who's more likely to be correct. In this case the tests lose out because of a simple fact: test scores don't matter in the real world. What does matter are the subjective impressions of those who employ our graduates or otherwise interact with them professionally. In the world beyond the academy, it's common shared perceptions that are the metric of success, and it won't do any good to point to your test scores. In fact, there is a certain shadenfreude in disproving credentials, as in watching videos of graduates from Ivy U who don't know why the seasons change. It isn't just Missouri: we're a show me society.

You'll notice that either way, the test results are largely unneeded. This illuminates why they are being used. Self-reported dialogical assessments depend on trust. In theory, tests can be administered in an adversarial environment. This restates Peter Ewell's quote in my previous article. This is a recipe for optimizing irrelevance. In Assessing the Elephant, I called this a degenerate assessment loop and gave this example:

A software developer found that there were too many bugs in its products, so it began a new system of rewards. Programmers would be paid a bonus for every software bug they identified and fixed. The number of bugs found skyrocketed. The champagne was quickly put back on ice, however, when the company realized that the new policy had motivated programmers to create more bugs so that they could “find” them.

Similar "degenerate" strategies find their way into educational practices because of the economic value placed on monological simplifications used in low-trust settings. We read about them in the paper sometimes.

Surveying the Dialogical Landscape
I have implemented surveys at two institutions to gather faculty ratings of student learning outcomes. I have many thousands of data points, but no standardized test scores to compare them to, so I can't check the alignment as I described above. The reliability of these ratings is: about a 50% probability of exact match on a four-point scale for the same student, same semester, same learning outcome, with different instructors. I've already written extensively about that, for example here and on this blog, as well as some chapters in assessment books, which you can find on my vita.

Conclusion
In a tight system, monological approaches are useful. The human body is a good example of this, but we should note that at least two important systems are more dialogical than monological: the immune system and the conscious mind. The world beyond graduation resembles a competitive ecology more like what the immune system faces than a systematic by-the-numbers existence like a toenail.

The only reason to use monological tests is if we don't trust faculty. This can't even be done with any intellectual honesty because we can't say that the tests are any good. What I proposed in the "The End of Preparation" is that we move to dialogical methods of assessment throughout and beyond the academy. These can still be summarized for administrators to look at, but only if there is trust at all levels. And really, if there is no trust between faculty and administration, the whole enterprise is doomed.

The mechanism of using public porfolios showing student records of performance can be purely dialogical--a student's work can have different value to different observers inside and outside the academy.

Next time I'll address what all this has to do with rubric-based assessment.

[Next article in this series: "Assessments, Signals, and Relevance"]

Some Frivolous Thoughts
As I said, this dichotomy changed the way I think about the world, and I find interesting tid-bits everywhere. One interesting idea is the hypothesis that as a domain of interest becomes more reliably theoretical (like alchemy becoming chemistry), the nomenclature transitions from descriptive and dialogical to arbitrary and monological. I went poking through several dictionaries looking for evidence of the names of the elements, to find examples. Copper may be an instance, perhaps having been named for Crete, as in "Cretian metal." If the name is too old, the etymology is foggy. Steal is more recent, and it seems to derive from a descriptive Germanic word for stiff. Compare that to Plutonium, which is modern and non-descriptive. Of course, with arbitrary naming, the namer can choose to be descriptive, as Radium arguably is. This thesis needs some work.

In biology, Red-Winged Blackbird is a descriptive name for the monological Agelaius phoeniceus. In a good theory, it doesn't matter what you call something. What matters is the relationships between elements, like the evolutionary links between bird species as laid out in a cladistic family tree.Modern scientists are more or less free to name new species or sub-atomic particles whatever they want. Organic chemistry is an interesting exception, because the names themselves are associated with composition. They are simultaneously descriptive and monological.

Drug names are particularly interesting. Viagra, for example, has a chemical name that describes it, but that obviously wouldn't do for advertising purposes. Here's what one source says about the naming process:

Drug companies use several criteria in selecting a brand name. First and foremost, the name must be easy to remember. Ideally, it should be one physicians will like -- short and with a subliminal connotation of the drug. Some companies associate their drugs with certain letters (e.g., Upjohn with X and Glaxo with Z). If the drug is expected to be used eventually on a nonprescription basis, the name should not sound medicinal. There must be no trademark incompatibilities, and the company must take account of the drug's expected competition.

It sounds like the name is chosen to fit neatly into a dialogical ecology.

The history of the SAT's name is interesting from this perspective, but I will bring this overlong article to a close.

Acknowledgments: The idea for the monological/dialogical dichotomy came out of conversations with Dr. Adelheid Eubanks about her research on Mikhail Bakhtin. I undoubtedly have mangled Bakhtin's original ideas, and neither he nor Adelheid should be held responsible for that.

The End of Preparation

2011-11-20T05:20:00.001-05:00

A few days ago, I wrote "A Perilous Tail" about problems with the underlying distributions of measurements (in the sense of observations turned into numbers) we employ in education. I've shown previously that at least for the SAT, where there is data to check, predictive validity is not very good: we can only classify students correctly 65% of the time. When this much random chance is involved in decision-making, the effect is that we can easily be fooled. Here I cited the example that might lead us to believe that yelling at dice improves their "performance." It's also unfair to hold individuals accountable if their performance is tied to significant factors they can't control, and it invites cheating and finger-pointing.

As I mentioned in the previous article, I have a solution to propose. But I haven't really done the problem justice yet. We encounter randomness every day, so how do we deal with it? Functioning systems have a hard time with too much randomness, so the systematic response is to manage it by reducing or avoiding uncertainties, and when that cannot be done, we might imagine it away (for example, throwing salt over one's shoulder to ward off bad luck). Many of our ancestors undoubtedly faced a great deal of uncertainty about what they would be able to eat, so much of our mental and physical activity and the very way our bodies are constructed, has to do with finding suitable food (which can be of various sorts) and consuming it effectively. Compare that with the homogeneous internal system of energy delivery that feeds and oxygenates the cells in our body via our circulatory system. Complex systems often take messy stuff from the environment and then organize it for internal use. I will use a mass-production line like the one pictured below for such a system.

Source: Wikipedia

The idea is to find materials in the environment that can be used to implement a plan, turning rocks into aluminum and tree sap into rubber, sand into glass, and heating, squashing, and otherwise manipulating raw natural resources until they come to create an airplane. This process is highly systematized so that parts are interchangeable, and the reliability of each step can be very high. Motorola invented the concept of Six Sigma to try to reduce the randomness in a manufacturing process to negligible amounts. This is at least theoretically possible in physical systems that have reliable mechanical properties.

What do we do when randomness can't be eliminated from the assembly line? One approach is to proceed anyway, because assembly lines have great economies of scale, and can perhaps be useful even if there are a large number of faulty items produced. Computer chip makers have to deal with a certain percentage of bad chips at the end of the line, for example. When chemists make organic molecules that randomly choose a left-right symmetry (i.e. chirality), sometimes they have to throw away half of the product, and there's no way around it.

The educational system in the United States has to deal with a great deal of variability in the students it getsI as inputs and the processes individuals experience. It superficially resembles a mass production line. There are stages of completion (i.e. grade levels), and bits of assembly that happen in each one. There are quality checks (grades and promotion), quality assurance checks (often standardized tests), and a final stamp of approval that comes at the end (a diploma).

All this is accomplished while largely ignoring the undeniable fact that students are not standardized along a common design, and their mental machinery cannot be engineered directly the way an airplane can be assembled. In short, the raw material for the process is mind-bendingly more complex than any human-made physical device that exists.

Because of the high variability in outcomes, the means we use for quality assurance is crucially important, and this is where we have real opportunities to improve. This is the assessment problem. Current methods of assessment look a lot like factory floor assessments: the result of analyzing student performance is often a list of numbers that can be aggregated to show the executives how the line is working. Rewards and punishments may be meted out accordingly. In a real factory, the current stage of production must adequately prepare the product for the next stage of production. We must be able to correctly classify parts and pieces as "acceptable" or "not acceptable" according to whether or not they will function as required in the whole assembly. The odd thing about educational testing is that this kind of question doesn't seem to be asked and answered in a way that takes test error into account. Randomness is simply wished away as if it didn't exist. In the case of the SAT (see here), the question might be "is this student going to work out in college?" In practical terms this is defined as earning at least a B- average the first year (as defined by the College Board's benchmark). To their credit, the College Board published the answer, but this transparency is exceptional. Analyzing test quality in this way proceeds like this:

State what the desired observable future effect of the educational component under review.
Compare test scores with actual achievement of the outcome. What percentage succeeded at each score?
Find a suitable compromise between true positives and true negative outcomes to use as your benchmark.
Publish the true positive and true negative predication rate based on that benchmark.

To repeat, the College Board has done this, and the answer is that the SAT benchmark gives the right answer 65% of the time. This would make you rich if we were predicting stock prices, but it seems awfully low for a production line quality check.

Source: Wikipedia

Because we can't make the randomness go away, we imagine it away. So the assessments become de facto the measure of quality, and the quality of the tests themselves remains unexamined. In a real assembly line, an imperfect test would be found out eventually when the planes didn't perform as expected. Someone would notice and eventually track it down to a problem with the quality assurance program. There is so much uncertainty in education that this isn't possible, and the result is truly ironic: the deeper insinuation of tests that are unaccountable for their results. To be clear: any quality test that does not stand in for a clear predictive objective and provide research for its rate of correct classification in actual practice, is being used simply on faith. To be fair, it's virtually impossible to meet this bar. That excuse doesn't make the problem go away, however--it just makes it worse. One result is that test results distort perceptions of reality. If appearance is taken at face value for reality, then appearance has great economic value. There is incentive to do whatever it takes to get higher test scores, with unfortunate and predictable results.

To sum up the problem: variability is too high for good standardized tests to support an educational assembly line, and this fact is generally ignored for convenience.

I don't mean to imply that the major actors in education are incompetent. We are where we are because of historical factors that make sense as a system in evolution, and we have the means to take the next step.

The real world is not an assembly line. There is this expression we use when talking to students "when you get out into the real world...", as if the academy is a walled garden that excludes the vulgar world at large. This too, is a factory mentality. The students have heard it all before. From kindergarten on up, they hear stories about how hard it's going to be "when you get to high school," or "when you get to college." My daughter marveled at this during her first few weeks of high school. She was amazed and somewhat appalled that her middle school teachers had misled her about this. Of course it's not much harder--the system can only work with a high degree of integration, smoothing out the hard bits. If the wing assembly is slowing down the production line, then it needs attention. One could argue that the whole path from Kindergarten through Doctorate is becoming a smooth one for anyone who wants to trod it.

But the real world really is different. The assembly line stops at the hanger door, and the planes are supposed to be ready to fly. The tests don't matter anymore. No one is going to check their validity, nor delve too deeply into what a certification means after graduation. And in the real world, the factory mentality has to be unlearned: one cannot cram all night just before promotions are announced in order to game the system.

One solution is to try to change the real world to be more like the educational system. This is a practical choice for a military career, perhaps, where strict bureaucracy is essential to function. But it certainly is a mismatch for an entrepreneurial career, the engine of the nation's economic growth.

I believe it is now a reasonable and desirable choice to go the other direction, and change the assembly line to look more like the real world. The most important aspect to change is to admit uncertainty and begin to take advantage of it. This means we have to forget about the idea of standardizing and certifying. I will argue that we can do this to our great advantage, and introduce efficiencies into the economic structure of the nation in the process. Currently we pass our uncertainties on to employers. We hide behind test results and certificates, and leave it to employers to actually figure out what all that means. The result is that they have only very crude screening information at their disposal, and have to almost start from scratch to see what a graduate can actually do. The Internet can change all that. To my surprise, I discovered last week that it already is.

I had written a draft of this article last week, but when I ran it through my BS-detector, I couldn't bring myself to publish it. The reason is simple: it's just another untested idea, or so I thought. I hadn't actually employed the solution I will describe below, and so I didn't have anything concrete to show. But by coincidence, I saw exactly what I was looking for at the Virginia Assessment Group conference, at a presentation by Jeffrey Yan, who is the CEO of Digication. I didn't know about the company before last week.Vendors in higher education technology solutions may be excused perhaps for exaggerating the effectiveness of their products, and I generally think they are overpriced, too complicated, and too narrowly focused. I didn't have great expectations for Jeffrey's talk, but after about ten minutes I realized that he was showing off a practical application of what I was hypothesizing.

Digication is an eportfolio product. In what follows, I will not attempt to describe it as a software review would, but as it fits into the flow of ideas in this article.

The main idea is simple: instead of treating students as if we were preparing them for a future beyond the academy, treat them as if they were already there. In the real world, as it's called, our careers are not built on formalized assessments. To be sure, we have to deal with them in performance reviews or board certifications, but these are mostly barriers to success, not guarantees of it. Instead, it's the record of accomplishment we create as we go that matters. In many instances, promotions and accolades are inefficiently distributed, based on personal relationships and tenure, rather than merit, but this is not what we should aspire to. In fact, these imperfections are vulnerable to the sort of transparency that is within our grasp.

Senior Design Project at Stonybrook

In his presentation, Jeffrey showed examples of the sort of thing that's possible. Take a look at this senior design project at Stonybrook University. It's a real world project to design a new sort of sphygmomanometer (blood pressure meter). Quoting from the project page:

We aim to satisfy all the customer needs by designing a [blood pressure measuring] device that translates the vibrations into a visual indication of blood pulses, more specifically the first pulse to force its way through the occluded artery (systolic) and the last pulse detectable before laminar flow is regained (diastolic).

Another showcased student portfolio was from a second year student at the same institution, who created a public portfolio to tell the world about his interests and abilities. He shows how to solve what we call a difference equation (similar to a differential equation) using combinatoric methods here. This shows an interest in versatility in the subject that cannot be communicated with a few numbers in an assembly-line type report.

By concentrating on authentic evidence of accomplishment, rather than artificially standardized means of observation, we create an important opportunity: a public portfolio can be judged on its own merits, rather than via an uncertain intermediary. It's the difference between seeing a movie yourself and knowing only that it got three and a half stars from some critic.

The solution to the factory mentality presents itself. If students see that they are working for themselves and not as part of some unfathomable assembly process, accumulating what will become a public portfolio of their accomplishments, their learning becomes transparent. They can directly compare themselves to peers in class, peers at other institutions, graduates from all over, and professionals in the field. I imagine this leading to a day when it's simply unthinkable for any professional not to have an up-to date professional eportfolio linked to his or her professional social networking presence (see mathoverflow.net, Academia.edu, and LinkedIn.com as examples of such networks). Once started, the competitive edge by those with portfolios will become obvious--you can learn much more from a transparent work history than you can from a resume.

While in school, of course, some work, maybe much of it, needs to be private, to gestate ideas before presenting them to the world. But the goal should be for a forward-looking institution of higher education to begin to create public sites like the Stonybrook showcase and the one at LaGuardia Community College. Ultimately, universities need to hand the portfolios off to the students to develop as their respective careers unfold. I understand that graduates get to keep their portfolios and continue to develop them with Digication's license, as long as it is maintained.

Here's the manifesto version:

We don't need grades. We don't need tests or diplomas or certificates or credit hours. None of that matters except insofar as it is useful to internal processes that may help students produce authentic evidence of achievement. That, and that alone is how they should be judged by third parties.

Some advantages of switching from "assemble and test" to authentic work that is self-evidently valuable:

We change student mentality from "cram and forget" to actual accomplishment. We can make the question "when will I ever really use this stuff?" go away.
The method of assessing a portfolio is deferred to the final observer. You may be interested in someone else's opinion or you may not be. It's simply there to inspect. Once this is established, third parties will undoubtedly create a business out of rating portfolios for suitability for your business if you're too busy to do it yourself.
Instead of just a certificate to carry off at graduation, students could have four years' worth of documentation on their authentic efforts. This idea is second nature to a generation who grew up blogging and posting YouTube videos.
It doesn't matter where you learned what. A student who masters quantum mechanics by watching MIT or Kahn Academy videos might produce as good work as someone sitting in class. It makes a real meritocracy possible.
Intermediate work matters. Even if someone never finishes a degree, they have evidence beyond a list of grades that they learned something. And it's in rich detail.

There's more than this, actually. The very nature of publishing novel work is changing. At present, the remnants of paper bound publication, with its interminable delays, exorbitant costs, virtual inability to correct errors, and tightly bound intellectual property issues, is still around. But it's dying. A journal is nothing more than a news aggregator, and those are now ubiquitous and free. It's hard to say what the final shape of publishing will be, but something like a standardized portfolio will probably be front and center. When I say 'standardized', I mean containing certain key features like metadata and historical archive, so that you can find things, cross-reference, and track changes. As the professional eportfolio develops, it will need help from librarians to keep it all straight, but this can be done at a much lower cost than the publishing business now incurs in lost productivity, restricted access, and cost to libraries.

The focus will, I believe, shift from journals and other information aggregators, to the individuals producing the work. And institutions will share in some of the glory if part of the portfolio was created under their care.

All of this has been around for a while, of course. Eportfolios are almost old news in higher education, and I've blogged about them before. My previous opinion was that there was no need for a university to invest in its own portfolio software because everything you already need is on the web. If you want to put a musical composition on the web, just use Noteflight, and of course there's YouTube for videos, and so on. All that's needed is a way to keep track of hyperlinks to these in a way that can allow instructors to annotate as needed with rubrics and such. The demos convinced me, however, that having a standard platform that can be easily accessible for private, class-wide, collaborative, or public use is worth paying for. I don't know how much it costs in practice, but there is value beyond what one can get for free on the Internet.

Portfolios have been incorporated here and there as just another part of the machinery, amounting to a private repository of student work that can be used for rubric ratings to produce more or less normalized ratings of performance--an advanced sort of grading. This is useful as a formative means of finding all sorts of pedagogical and program strengths and weaknesses. The point of this article is not that portfolios are a better way to produce test-like scores, but that the test scores themselves will become obsolete as external measures of performance. For professors to get feedback on student performance, and for the students themselves to hear directly what the professors and their peers think is invaluable. It's essential for teaching and learning. But it's downright destructive to use this is as a summative measure of performance, for example for holding teachers accountable. The instant you say "accountability," no one trusts anyone else, and there really is no way to run the enterprise but as a factory, with inspectors enforcing every policy. It cannot work in the face of the uncertainties inherent to the inputs and outputs of education.

There is a history of tension in higher education between the desire for authenticity and the simultaneous wish for factory-like operational statistics that show success or failure. The Spellings Commission Report has a nice sidebar about Neumont University and mentions their portfolio approach (their showcase is here), but can't tear itself away from standardized approaches to learning assessment. Three years before, the Council for Higher Education Accreditation beautifully illustrated the tension:

[I]t is imperative for accrediting organizations–as well as the institutions and programs
they accredit–to avoid narrow definitions of student learning or excessively standardized
measures of student achievement. Collegiate learning is complex, and the evidence used
to investigate it must be similarly authentic and contextual. But to pass the test of public
credibility–and thus remain faithful to accreditation’s historic task of quality assurance –
the evidence of student learning outcomes used in the accreditation process must be
rigorous, reliable, and understandable.

This is from CHEA's 2003 paper "Statement Of Mutual Responsibilities for Student Learning Outcomes: Accreditation, Institutions, and Programs." More recently, Peter Ewell wrote "Assessment, Accountability, and Improvement: Revisiting the Tension" as the first Occasional Paper for the National Institute for Learning Outcomes Assessment, in which he illuminates the game-theoretic problem I alluded to above:

Accountability requires the entity held accountable to demonstrate, with evidence, conformity with an established standard of process or outcome. The associated incentive for that entity is to look as good as possible, regardless of the underlying performance. Improvement, in turn, entails an opposite set of incentives. Deficiencies in performance must be faithfully detected and
reported so they can be acted upon. Indeed, discovering deficiencies is one of the major objectives of assessment for improvement.

In a real factory setting, tests of mechanical process can be very precise, eliminating the difference between what the assessment folks call formative (used to ferret out useful improvements) and summative (an overall rating of quality). If a machine is supposed to produce 100 widgets per hour, and it's only producing 80, it's clear what the deficit is, and the mechanic or engineer can be called in to fix it. But when one is held accountable for numbers like standardized test results that have a considerable amount of uncertainty (which itself is probably unknown, as I pointed out before), the game is very different. It is less like a factory and more like going to market with a bag of some good and some counterfeit coins, which I described in "The Economics of Imperfect Tests." One's optimal strategy has less to do with good teaching than with manipulating the test results anyway one can. Unfortunate examples of that have made national news in K-12 education.

My proposal is that we in higher education take a look at what Stonybrook and others are doing, and see if there is not merit to an emphasis on authentic student learning outcomes, showcased when appropriate for their and our benefit. That we don't consider a grade card and a diploma an adequate take-away from four years and a hundred thousand dollars of investment. That instead, we help them begin to use social networking in a professional way. Set them up with a LinkedIn account during the orientation class--why not? Any sea change from teach/test/rinse/repeat to more individual and meaningful experiences will be difficult for most, but I believe there will be a payoff for those who get there first. Showing student portfolios to prospective students as well as prospective employers creates a powerful transparency that will inevitably have valuable side effects. Jeffrey told said that some of the portfolios get millions of Internet views. How many views does a typical traditional assignment get? A handful at most, and maybe only one.

The odd thing is that this idea is already quietly in place and old hat in the fine arts, performing arts, and architecture departments, and there are probably some I'm not aware of. Who would hire a graphic designer without seeing her portfolio, even if she had a wonderful-looking diploma? This means that we probably have experts already on campus. Computer Science is a natural fit for this too, and there's already a professional social network set up at Stackoverflow.com.

A good first step would be to allow portfolio galleries to count for outcomes assessment results in the Voluntary System of Accountability (VSA). Currently, the only way to participate is to agree to use standardized tests. From the agreement's provision 17:

Participate in the VSA pilot project to measure student learning outcomes by selecting one of three tests to measure student learning gains.

a) Collegiate Assessment of Academic Proficiency (CAAP) – two modules: critical thinking and writing essay - http://www.act.org/caap/.

b) Collegiate Learning Assessment (CLA) – including performance task, analytic writing task - http://www.cae.org/content/pro_collegiate.htm.

c) ETS Proficiency Profile (formerly known as MAPP) – two sub scores of the test: critical thinking and written communication - http://www.ets.org/. Either the Standard or the Abbreviated form can be used.

The VSA is a wonderful program, but it is handicapped by this requirement. If you already use one of these tests, that's fine, but it's expensive and a distraction if you don't find them useful. More to the point of this article, there is no option on the list to report authentic outcomes. Adopting another pilot project to see how far the public portfolio idea will sail would be a great addition.

[The next article in this series is "Tests and Dialogues"]

Acknowledgements: Thanks to Jeffrey Yan for letting me chew his ear off after his presentation. And thanks to the coordinators of the Virginia Assessment Group for putting that wonderful event together.

Disclaimer: I have no financial interest in any of the companies mentioned in this article.

A Perilous Tail

2011-11-16T16:43:00.001-05:00

A certain kind of intellectual honesty seems to be critical to systems that want to survive. Even without the subtleties I discussed earlier, it's obvious that a system that ignores reality can only survive as long as the environment is completely benign. By coincidence, I came across Daniel Kahneman's Thinking, Fast and Slow, which catalogs a number of ways in which we humans can fool ourselves. One instance of this is particularly relevant to training, management, and education. It occurs in any rating of performance that involves some element of luck.
Dr. Kahneman describes an episode with military training instructors, where he was talking about studies that show positive reinforcement is the key to better learning. This point of view was flatly contradicted by his audience, who claimed the following (my description):

When a cadet does a bad job at something, I yell at him. Usually he does better the next time. If I praise him for doing a good job, his performance almost always declines the next time!

The author describes this as an ah-ha! moment for him. The solution to this paradox is cleverly described in the book. Here's my version.

As we have seen with SAT scores, the predictive validity of even well-researched tests can be poor (65% correct classification in the case of the SAT benchmark). The remaining variance may as well be chalked up to chance unless we have more information to bear. In addition to measurement error, there can be chance involved in the performance itself. That is, many unpredictable environmental variables may come to bear on the outcome. Baseball games, for example, have a large amount of luck injected into the outcome, so that it's only over a large number of games does relative performance actually reveal itself (see Moneyball by Michael Lewis).

When luck is involved, something called regression to the mean happens--exceptional events are usually followed by unexceptional ones. To make this clear, you can do the following experiment, to mimic the drill instructors' experience. You need some dice.

Roll two dice and add. Consider higher sums as a good performance, and lower sums as poor. Feel free to strongly admonish the dice when they roll low numbers like 2 or 3, and lavish praise on them when they roll 11s and 12s. You'll find that the stern words work wonders--the rolls almost always improve afterwards! On the other hand, the praise is counter-productive since 11s and 12s are usually followed by lower rolls.

We can imagine most events happening in the 'fat part' of a bell curve, and we are generally ill-equipped to encounter events far out on the tails, which by definition are very rare. It's not just the paradox described above. Nassim Taleb wrote a whole book about this called The Black Swan. Other, more speculative thinkers, have imagined thus:

If you assembled all the humans who ever have or ever will live into a distribution according to when they were born, the curve would likely look like some kind of hump with tails on both sides. If you chose one human at random, he or she would likely come from the fat part of the curve. Therefore, that's where we likely are at this moment in time--that is, we would expect to be typical rather than exceptional. If this is true, then it puts probabilistic bounds on our expectations for the duration of human civilization. The math varies, depending on your assumptions, but something like a few thousand years would be a reasonable upper bound using this method.

In business, there's an idea called Six Sigma that is supposed to reduce process errors to an infinitesimal fraction (six sigma means six standard deviations from the mean, which for a Normal distribution is an exceedingly small percentage: about 4 errors per million attempts). Yesterday someone suggested to me that we might use Six Sigma in higher education. I laughed, not because there aren't probably useful ideas there (similar to institutional effectiveness), but because the inherent fuzziness of our core business--changing brains--is so fraught with unknowns. I think one sigma is about as good as we're likely to do. Although we're not well prepared to deal with it, we live in the tail of the distribution.

What percentage of "value-added" indices are due to random chance, do you suppose? This statistical method of computing theoretical filling of the learning vessel has been institutionalized to reward or punish teachers and schools. As a mathematician and assessment professional, it's hair-raising to read the pat descriptions from the link above, like:

Q: How does value-added assessment sort out the teachers' contributions from the students' contributions?

A:Because individual students rather than cohorts are traced over time, each student serves as his or her own "baseline" or control, which removes virtually all of the influence of the unvarying characteristics of the student, such as race or socioeconomic factors.

Test scores are projected for students and then compared to the scores they actually achieve at the end of the school year. Classroom scores that equal or exceed projected values suggest that instruction was highly effective. Conversely, scores that are mostly below projections suggest that the instruction was ineffective.

Taking another page from Dr. Kahneman's book, this is an instance of solving a simple problem that superficially resembles the actual problem, because the original problem is too hard. It's easy to imagine that the distribution is tight and the tails insignificant, that we control and understand all the elements of chance that might contribute to a computed value-added parameter. Unfortunately, the direct link to reality doesn't reveal itself easily, and so there is no immediate feedback that would correct the problem by making it obviously wrong to observers. This is a case where we should be assiduously honest with our reasoning and doubts. Suppose the SAT's accuracy is representative, and the underlying achievement tests classify students correctly no more than 65% of the time. What fraction of the value-added score is simply random?

The general problem is caused by the unknown unknowns that plague complex observations. There is an elegant way out of this mess that doesn't involve advanced math or huge sample sizes. Moreover, it solves the most important problem in higher education. How's that for a cliff-hanger?

Searching IPEDS Data

2011-11-15T06:14:00.001-05:00

I found a nice site for sifting through some of the important bits of IPEDS data at www.collegeresults.org, especially for comparing institutions. My metric of choice is instructional dollars / FTE when trying to roughly compare quality. You can find this under the "Finance and Faculty" tab.

Spark Intelligence

2011-11-06T08:47:00.000-05:00

Yesterday's post about self-limiting intelligence may come off as pessimistic, and indeed, I think very few leaders of organizations (nations, companies, colleges,...) have a standing agenda item labeled "survival." I think they should, and in some ways think that our forebears were more attuned to that, but it's just a notion. The cathedral in Cologne took about 640 years to complete. What projects do we have ongoing now with that sort of horizon? I think if you asked a large corporation's CEO what he or she thought of the company's prospects three or four centuries out, you'd get a strange look. Next quarter is what matters.

Suppose for a moment that it's true that a singular intelligent system (SIS) can only get so smart before it starts working against its own interests. It's doomed, and it would be smart enough to realize its's doomed. What then?

Although we have no evidence of other intelligent life in the universe, since we are here ourselves, it's possible that such life has or will exist somewhere else (the Drake Equation tries to pin that down, but that's not what I'm concerned with). So our hypothetical doomed SIS knows this too. That is, it knows that although all civilizations will eventually collapse, the universe is a fertile ground for new ones to spring up. This is what I call spark intelligence--new SISs cropping up now and then across the galaxy. Therefore, there is a possibility for the universe to maintain a disjointed "stream of consciousness" if these independent SISs could communicate with each other. Time and distance scales make any sort of synchronous communication unlikely, so it has to be asynchronous, like one civilization reading an ancient book left by another long-gone culture. This would enable a sort of meta-intelligence comprising knowledge and culture from a long sequence of dead civilizations: a universal Domesday book. Eventually one of them would have to figure out how to get this package to a new universe before this one suffers heat death, but there are billions of years left to do that.

Imagine these sparks of intelligence going off all around the hundreds of billions of galaxies in our observable patch of space. Many of them reach the same conclusion I just have. Some of them might have the motivation to participate (motivation is essential to survival, recall, and we're talking about survival of knowledge and culture). There are two ways to participate. One is to create a library that can be seen and decoded from a very long way off, and the other is to search for and assimilate the libraries of others.

If this giant inter-library loan program exists, it would depend on the "spark" rate, the probability that a civilization will be motivated to participate, and the window of opportunity it has to do so with existing resources before it collapses. An obvious first step would be to see if there are libraries already out there. Of course, we're already doing that with SETI, but it's not a high priority.

You may be having trouble seeing over the pile of hypotheticals I assembled in the preceding paragraphs, so let me bring all this back to Earth. Much of the analysis above also applies right here. A regulated market economic system, for example, provides fertile ground for "sparks" of a different sort--businesses of all sorts spring into existence and then eventually get eaten or die. A few last hundreds of years. But they too share a common "culture bank," hold conferences and host professional organizations in order to share ideas (while hiding trade secrets, of course) similar to the galactic library I proposed. [Edit: We can also see that there are policy implications for the government that regulates the system: keeping the ground fertile for new 'sparks' and making sure that the eventual end of any enterprise is planned for. That way "too big to fail" wouldn't be the critical issue it is now. If the philosophy is that every enterprise will eventually fail, and that this has to be planned for, it's not a catastrophic surprise when it happens.]

There may be a case made for education being like this too: providing the right environment for novelty to emerge in the form of new research results, new art, and so on. If so, it's probably not intentional from an organizational leadership viewpoint. The general tone of administration in my experience is more about how to keep a bureaucracy running efficiently. The effect of this machinery on learning are a factor, but the delineation between creating an environment for success and simply expecting it is a fuzzy one. As an example, assuming that learning is mostly related to how well students are taught is a bureaucratic simplicity (the inputs have a lot to do with it). Or the idea that if a student passes a writing class she can then write as well as she needs to. This "inoculation" philosophy is purely process-driven, and is almost antithetical to the idea that minds are cultivated, not stamped out in a factory. More on this theme next time.

Self-Limiting Intelligence

2011-11-05T19:16:00.000-05:00

I think intelligence grows only to the point where it begins to interfere with itself. In short, when we get smart enough, we begin to outsmart ourselves and actually undermine our own survival. Here, the inclusive 'we' could apply to individuals, but is more aimed at organizations: corporations, institutions, or governments.

I have been researching survival of these entities in the abstract for several years, and I seem to be all alone in this. This is a real mystery, because survival is the sine qua non for everything else we care about. If you are interested in some of the background, see Survival Strategies [1] or "Surviving Entropy" [2] in this blog.

My interest is in the seemingly pessimistic question "is it likely that an intelligent being or organization can survive for an indefinite period?" This is contrasted with the messy sort of survival exhibited by ecologies that evolve over time, which I refer to in short-hand as a MIC (multiple independent copies). The intelligent systems are shortened to SIS for singular intelligent system. The primary difference is that it's impossible to reproduce and mutate an organization the same way it is a bacterium. All your eggs are in one basket, so to speak. Whereas a bacteria culture can loose 99% of its population and pull through, a singular system can't afford a single lethal mistake.

In [1] I showed a couple of interesting facts about a SIS. First, it has to learn how to predict (or engineer) its environment at a very fast rate, unlike a MIC, which gets this for free via even the most desultory rate of reproduction. In actual fact, we have evidence that the ecology of life on Earth (a MIC) has survived for some billions of years, whereas we have no evidence of any government or other organization (a SIS) surviving for more than a few thousand years (I'm being generous). Put another way, when we look at the vast and enduring features of the universe around us, they are uniformly non-intelligent. This is the source of the so-called Fermi Paradox.

The second interesting fact about a SIS is that although it may be smart enough to change itself, it is impossible for it to predict the ultimate result of those changes. For an organism that is the product of an ecology, this is not an issue. Animals often come prepared for their earthly homes with protective coloration and other adaptations for the environment they will live in. They don't need to change this, or if they do, the provision is built-in but limited (like a chameleon). A frog can't re-engineer itself into a bird if it finds the need to fly. A SIS, on the other hand, may have to adapt to completely foreign environments over time.

The problem a SIS faces is that it generally cannot predict what will happen to it after a self-change, so it doesn't know if this change is good or bad in the long run. It can try to guess by simulating itself, but there's an essential limitation here. There are two types of simulation, detailed below.

Suppose a SIS considers changing its 'constitution' in some way, which will affect the way future decisions are made. It builds a sophisticated computer model of itself making this change to see what will happen. There are two possibilities:

1) The simulation is perfectly good: so good that the SIS cannot change the outcome even if it's a bad one.

2) The simulation is only approximate: the SIS can take a look at the future and change its mind about making the change.

In the first case, a perfect simulation tells us not only what the future holds, but also whether or not the organization will make the change. This is because it incorporates all information about the SIS, including the complete present state. So it will present a result like "you make the change and then X happens," or "you don't make the change." A perfectly true self-simulation has to have this property. So it's like Cassandra's warning--even if it predicts an undesirable future, it still has to live it!

Such perfect simulations are really only possible with completely deterministic machines, like a computer with known inputs. In practice, all sorts of variables might knock it off course. So what about approximations?The essential element of an approximation is to be able to make a decision about the future. The most fundamental one might be "if I make this change, will I eventually self-destruct?" This is the most fundamental question for a SIS. The most dangerous challenge from the environment for a SIS comes from within itself.

The US constitution makes it harder to change the constitution than to pass ordinary laws. This is a prudent approach to self-modification.

Unfortunately, decision problems like this are not reachable by general-purpose processes. This is covered in [1], but you might peek at Rice's Theorem to see the breathtaking limitations of our knowledge of what deterministic systems will do. So we can simulate in the short term, but the long-term effect will be a mystery.

So a SIS can only learn about self-change empirically, by trying things out, or short-term simulations. It can't ask about the general future. Although the external environment may be quite challenging, and survival may be a risk because of factors beyond its control, the internal question of how to manage self-change are just as bad or worse. Hence my hypothesis that the odds will catch up with any SIS eventually, and it will crash. This also jibes with with all the empirical evidence we have.

This is where I left the question in [1], but in the last couple of years I think I've identified a fundamental mechanism for self-destruction that any SIS has to overcome.It has practical implications for institutions of higher learning and other sorts of systems like businesses and governments.

In my last post, I showed a diagram for an institutional effectiveness loop that looks more technical than the usual version. Here it is again, with some decorations from the talk I gave at the Assessment Institute.

The diagram actually comes from my research on systems survival, and it is a schematic for how a SIS operates in its environment. The (R) and (L) notations refer to 'Reality' and 'Language' respectively. Recall that the I in SIS stands for Intelligent, and this is what I mean: the intelligent system has ways of observing the environment, encoding those observations into a language that compresses the data by looking for interesting features, and models the interactions between these. This allows a virtual simulation of reality to be played out in the SIS, enabling it to plan what to do next in order to optimize its goals. This is the same thing as an institutional effectiveness loop in higher education, in theory at least.

Language is much more malleable than reality: we can imagine all sorts of interactions that aren't likely to actually occur. For example, astrology is a language that purports to model reality, but doesn't. It's essential for the SIS to be able to model the real environment increasingly well. The mathematical particulars are given in [1] in terms of increasing survival probabilities.

There's something essential missing from the diagram above. That is the motivation for doing all this. When the SIS plans, it's trying to optimize something. This motivation is not to be taken for granted, because there's no reason to assume that a SIS even wants to survive unless it's specifically designed that way. For example, a modern air-to-air missile has good on-board ways to observe a target aircraft (e.g. radar or heat signature), a model for predicting the physics of its own flight and the target's, and the means to implement a plan to intercept. So by my definition, it's reasonably intelligent. But it doesn't care that it will be blown up along with its target.

Motivation to survive is a decoration on a SIS. Of course it won't likely survive long without it, but it's not to be taken for granted, which makes the question of what happen under self-change very important. It's quite possible to make a change that eliminates the motivation for self-survival. What exactly constitutes survival is a messy topic, so let's just consider this general feature of an SIS, which has applications to personal life as well as governments, corporations, military organizations, and universities:

Motivations can change or be subverted when self-modifications are made.

This doesn't sound very profound; it's the particular mechanism shown below that is the interesting part. Here's how it works. When we observe our environment, we encode this into some kind of language, specialized to help us understand where we are in relation to our goals. For example, if I stub my toe on external reality, I get a finely-tuned message that informs me immediately that my most recent action was inimical to my goals for self-preservation: it hurts! This pain signal is just like any other bit of information encoded into a custom language: it can be intercepted or subverted. There are medicines and anesthetics that can reduce or completely eliminate the pain signal. Because signals are purely informational, they are always vulnerable to such manipulation by any system that can self-change.

Motivations are closely tied to these signals. It may be a simple correspondence, as with pain, or something abstract that comes from modeling the environment, like fear of illness. Sometimes these come into conflict, as the example below illustrates.

Sometimes I get sleepy driving on the interstate. If I find myself beginning to micro-sleep, I pull off the road and nap for 15 minutes. How is it that my brain can be so dumb as to fall asleep while I'm driving? Something very old in there must be saying "it's comfortable here, there's not much going on, so it's a good time to sleep," in opposition to the more abstract model of the car careening off the road at speed. We can try to interfere with the first signal with caffeine or loud music or opening the windows, or we can just admit that it's better to give in to that motivation for a few minutes in a safer place.

The mechanism for limiting intelligence works like this:

A SIS tries to attain goals by acting so as to optimize encoded signals that correspond to motivations. If it can self-modify, the simplest way to do this is to interfere with the signal itself.

I think it is very natural for a SIS to begin to fail because it fools itself to artificially achieve goals by presenting itself with signals that validate that. Even if external reality would disagree.

I just finished reading Michael Lewis' The Big Short, which is rife with examples of signal manipulation. Here are a couple. 1) The ratings agencies (S&P, Moody's) had two motivations in conflict: generating revenue by getting business rating financial instruments (such as CDOs), and generating accurate ratings. These are in conflict because if they rate something poorly (and perhaps unfairly), they may lose business. The information stream got subverted that should have signaled that it was a bad idea repackaging high-risk loans into triple-A rated instruments.

2) According to Lewis, counterparts at Goldman Sachs learned exactly how to tweak the signals in order to get the result they wanted from the bond raters (by manipulating the way risky loans were structured to optimize an average credit rating, for example).

3) In an example of self-deception, risk management offices of the investment banks managed were fooled by the ratings agencies "credit-laundering" and their own trading desks, which allowed vast liabilities to go unnoticed.

4) The whole economic apparatus of the world largely ignored signs that the system was on the verge of collapse.

If we imagine that a SIS is continually trying to increase its survival chances, an observation that probabilities are decreasing instead is obviously bad news. If it can self-modify it has the choice to accept this unwelcome fact about probabilities, or it could interfere with the signal (ignore it, for example).

Alternatively, the internal model of an SIS may associate a potential benefit with a planned act, which is a good thing. Any evidence that this may not work out as intended would decrease the value of the act, and this (also bad) news might be subverted, so that only supporting evidence is accepted. This is usually called confirmation bias in humans.

It's natural to ask, if this is such a problem, why hasn't civilization already collapsed from ignoring bad news and amplifying good news? The answer, I think, is that humans comprise the civilization and all its organized systems, and humans can't completely self-modify. Yet. Imagine if you could.

What if every emotional reaction could be consciously tuned through some mental control panel? What to be happier? Just turn up the dial. Don't like pain? Turn that dial down.

Because humans are actually members of a MIC (that is, an ecology), we are subject to selection pressure from the environment. Viewed as discrete systems, our organizations inherit some of this evolutionary common sense, but it's diluted. Individual humans often have a lot to say about how an organization operates, and can imbue them with denial and confirmation bias. Organizations are easily self-changed, and can't predict how those changes will turn out. I think, however, that certain strategies can ameliorate some of the most self-destructive behaviors. Here they are:

1) Create cultures of intellectual honesty, and actively audit signals, languages, and models to make sure they correspond to what's empirically known, whether it's good news or not. Intellectual honesty should be audited the same way financials are: by an outside agency doing an in-depth review. In the long run this doesn't solve the problem because any such agency will have the same problems (self-deception, inability to predict effects of changes, etc.), but it might increase the quality of decision-making in the short to near term.

2) Be conservative and deliberate about changes to signals, languages, predictive models, and fundamental structure. Audit those continually and transparently. Everyone should know what the motivations are and what signals apply to each. Moreover, 'best practices for survival' should be used. Since much of our learning is from other systems that failed, this wisdom should be carefully archived and used.

These are particularly advisable for organizations that have motivational signals that are difficult or slow to interpret. For enterprises that are very close to objective reality, these measures are less necessary because of the obviousness of the situation. For example, it's hard to argue with the scoreboard in a sporting event. We can close our eyes if we don't like the score, but there's really not much room for misinterpretation. Therefore, one would expect a successful team to be either very lucky or else have good models of reality reflected in their language and signals. The same could be said of military units in active service, traders on a stock exchange, or any other occupation with signals that are hard to interfere with.

Examples in the other direction, where signals are or have been ignored are the financial crisis already mentioned, the looming disaster of global warming, the eventual end of cheap oil, and human overpopulation. On an individual level, unnecessarily bad diets, lack of exercise, smoking, and so on are examples of abstract survival signals ("doctor say so") versus visceral motivations (e.g. tastes good) that show flaws in our motivational calculus.

I intend in the next post or two to show how this is related to the business of higher education.

Language of Assessment: Session Summary

2011-11-02T08:42:00.000-05:00

This post summarizes the conclusions from my presentation yesterday at the Assessment Institute in Indianapolis. Many thanks to Trudy and everyone else who helps make this conference happen!

The topics below are taken from my Conclusions slide. They are related to the relationship between reality (actually affecting/effecting events) versus language (observing, understanding, planning). This is pictured in the diagram below, where R=reality, and L=language.

Language is easy. Reality is hard.
Language is mostly combinations of arbitrary signs (pointing with your finger and 'oink' being two exceptions), and the flexibility is due to the infinite number of ways that we can arrange these signs (words). I used the example of QR codes like the one pictured below to contrast reality (a small square) with language (about $10^{300}$ different codes expressible--probably more atoms than there are in the observable universe).

Source: Wikipedia

The larger point here is to be careful that we don't become disconnected from reality. This can happen in a number of ways. We can, for example, do everything through planning, but then fail to execute. Or we be fixated on some model, some sort of understanding that we preconceive and don't deviate from, and make observations simply conform to this worldview. The language we choose, including the types of assessments and how they convert raw observations into data, necessarily limit us, and it's good to be cognizant of those limits.

Seek meaning: use contextual language
When talking to faculty members and other non-assessment experts, I recommend avoiding jargon like 'measurement', 'validity', and such. It's more productive to use language that they already use. I don't even like to use 'assessment', since the point of the process is really improvement, not just assessment, and the word is now a lightning rod for instant opposition in some quarters. Of course, if you have to get a report out of them for accreditation, it isn't going to matter much what you call it. The Happy Happy Fun-Time Learning Report isn't going to fool them. The deeper philosophy behind the advice under this heading is that the faculty members really are the experts--they know their students, the material, the history of the curriculum, the changes that have been made, know what is on tests, and have all this rich contextual information. They are already familiar with the idea of academic success or failure, and don't really need a new lexicon to discuss it.

Administrators like summative/monological language
From the Department of Education down, 'performance measures', 'accountability', and 'value-added' are words that reflect the top-down mindset. They often want to see dashboards that lead to easy conclusions about the success or failure of some endeavor. For more on monological/dialogical, which I didn't cover, see Assessing the Elephant.

Faculty like formative/dialogical language
Faculty may also want to see summary graphs of indices (e.g. pass rates), but unless there's some model of cause and effect that we can build from the information, it's hard to figure out what to do with the data. In other words, we need to be able to construct a story that makes sense. For this, providing details like percentages (e.g. distributions instead of averages), or other connections like correlates, shown below (see my previous posts on that).

Rubrics are particularly useful, especially if the faculty members construct them using language that makes sense to them (e.g. fitting them into their cause/effect model). The scale used is particularly important. I didn't mention this in the talk, but I will here. Often we use a PAGE rubric (poor, average, good, excellent) for accomplishment levels, but this should only be done for skills or knowledge that are not going to be under long development. For example, you wouldn't want to use this to rate writing because there's really no upper limit on how good you can get. Moreover, a freshman who gets Excellent ratings in the first year is likely to get Excellent in the fourth year as well--demonstrating no improvement! I use a PAGE rubric for rating effort, since the concept of no effort to maximum effort seems a good fit. David Dirlam has a comprehensive way to create very far-reaching rubric language that I won't say more about here. Google him and find out more. I referenced him here.

I mentioned the FACS a few times in the talk. It's a rating scale based on a student's career. You can read all about it in this manuscript or through these links.

Predictive validity of tests is probably not great
I showed examples from the College Board's benchmark on the SAT, which gives a maximum 65% correct classification rate. More on that topic in this article. The SAT is exceptional in that we actually can test its predictive validity. Validity studies for most standardized tests are underwhelming, and don't address this crucial point--the main point, really. If we are content to assume that being able to answer a test question correctly at this moment translates into some future ability without actually testing it, then okay. But I think that's unreasonable given the nature of education. At the national scale, there is also a strange lack of interest in finding out what all this investment in education actually produces in outcomes that can be accurately assessed, like employment and earnings (financial aid data + loan data + IRS data).

Report summarizes, not abstractions
I left this off the conclusions page, but added it here because it relates to the previous topic. Let me illustrate with an example. Suppose we have an English proficiency exam to place incoming students into a first writing course. This is a test we can (and should) check the predictive validity of by looking at success rates. We can make summary statements about test results like this:

The 2011 English proficiency test results showed that 79% met the faculty-established criterion to be placed into ENG 102.

The validity of this statement is unquestionable as long as the calculation was done correctly. Compare that to:

According to the 2011 English proficiency test, 79% of our students can write well.

This substitution of a known fact "met the criterion" with a notion "can write well" takes us from a perfectly valid statement to one that is completely subjective, depending on what the reader's idea of "can write well" is. If the reader is trusting, he/she may just pass over this, but the validity of the statement is essentially unknowable.

This isn't hair-splitting--it's essential. If we allow ourselves to jump from what's know to undefined and untestable abstractions, we may end up creating educational systems that follow this illusion into irrelevance (e.g. a completely artificial test-driven culture like No Child Left Behind). If we stick to what we actually know to be true, we don't have this problem. Of course, this comes into conflict with the summative view demanded by the DoE. We need to make the conversation more nuanced, I think.

Don't aggregate different kinds of stuff
This falls in the nuts and bolts category. You wouldn't normally add different sorts of data together, like age plus shoe size, and expect to get anything useful. Why do we add up rubric dimensions for an 'overall score?' The only purpose it serves is to save space, by reducing everything to one number. But it's really hard to make sense of the number, and comparing one score to another is problematic. For example, suppose student papers are rated on a 1-10 scale on Correctness (e.g. grammar and spelling), Style, and Audience. Is a three point deficit in correctness exactly made up for a by a surfeit of style? It's hard to see how that could be the case.

Average data as a last resort: use richer displays
Learning outcomes data is complex. Reducing a rich dataset into one number is like burning it and poking through the ashes. Use averages (or better: medians) when you really only have room for one number. For Likert scale data, reporting out the percentage of Agree and Strongly Agree is often more meaningful to the reader than an average.

Validity is not a property of a test, but of a statement
The quote below pretty much says it all.

It is a common misconception that validity is a particular phenomenon whose presence in a test may be evaluated concretely and statistically. One often hears exclamations that a given test is “valid” or “not valid.” Such pronouncements are not credible, for they reflect neither the focus nor the complexity of validity.
– College BASE Technical Manual

It is statements that are valid or not, not tests or results from tests. Just like anything we say can be true or not (true = valid). You would think that this would require test makers and test users to only say things they know are true, but this is not the case. The temptation to make the leap to abstraction is irresistible. I wrote a paper on this topic here.

Complexity and reliability don't go together
If something takes more language to describe, it's more complex. When we look at something complex, there are by definition many ways of looking at it. This means that we often have a collision between reality and language when we assess learning. If we insist on a simple bad-to-good scale (one dimensional, in other words), we have to throw out much of the original information in order to data-compress it. It's similar to the complaint about averaging. Complex outcomes will have lower reliability in ratings because there are valid differing opinions. I used the example of rating the taste of green beans. You may like them al dente. I like them mushy. What's the correct response on the answer key?

You can see that the summative approach favored by policy-makers is not a very natural way to look at learning, and the gap widens the more complex the outcome becomes. A simple test for multiplication is probably sufficient because its complexity is so low. But a multiple-choice test for understanding of literature (like the one my daughter took last year) is simply a result of abstraction (test result = ability) and convenience (relatively cheap, easy to score).

Non-cognitives are important
Outcomes other than knowledge and skill include attitude and personal behaviors. These are increasingly recognized as important in higher ed. I've written more about that subject in my blog.

Opportunity to focus on accomplishment
This is the punch line, that provides a hopeful way out. If the critique above seems negative in parts, it's only intended to be an honest look at what we're doing so we can make the next evolutionary improvement. My suggestion is that we stick to facts we know when using assessment results (that is be more modest in our claims), and take two new approaches. One is based on the FACS--gather subjective opinions about our students so we can see if our tests predict those real-world opinions. (I didn't mention this in the talk). The second, and more important suggestion, is that we stop thinking of ourselves as preparing students for something after college. Instead we have the opportunity to lead them to authentic accomplishments that they can use to build their professional portfolio beginning in their first year.

Compare:

We test our students constantly. According to the results, they graduate knowing how to think and communicate. That's what the diploma means.

Our students demonstrate achievement through substantive works that stand on their own. You can look at them in the portfolios that they build during their time with us. You will see their actual accomplishments in their fields of study and be able to judge for yourself their quality.

The first approach is based on unprovable claims, and has students waiting on the 'factory floor' to receive their certificate. This is the summative industrial standarized-in, standardized-out approach. The second approach is more motivating (I suspect) to both students and faculty--it allows students to see real-world outcomes as they go along. Ones that are incremental and achievable, that they can compare and compete with, and retain meaning after graduation as part of their professional history. And it's free--just used LinkedIn for the 'portfolio' system.

I have a lot more to say about this, but it will have to wait.

Thanks to everyone who came to the session--please stay in touch and share your successes and failures with us so we can imitate the former and avoid the latter.

Assessment Workshop Survey Results

2011-10-23T07:25:00.001-05:00

Survey responses for the assessment workshop next week are shown below. Nothing is hyperlinked, since these are just images. I had to do it this way to protect the data, so the individual responses can't be used to identify individuals.

Mapping covariates, Part III

2011-10-18T18:38:00.001-05:00

In my spare time (ha-ha) I refined the software I blogged about in the last two posts in order to automate almost everything about sorting out what's connected to what in a data set. Now I can create a folder with a data file, an index of variables, and an options list, drop that folder on a script on the desktop, and a few seconds later have a graph like the one below. This makes it easy to tweak parameters to find a nice picture. One that tells $2^{10}$ words, give or take.

For the graph below, I dug out the results of a semester of course evaluation using the new form I got implemented a year ago. I wrote previously about the odd fact that the summative evaluation of the course in Q12 and Q13 didn't seem to relate much to the learning outcomes. The closest other item in this topology is how enjoyable the students reported the course being.

This graph shows covariances instead of correlations. The latter have the nice property of being normalized to [-1,1], but suffer from the fact that nearly constant items all correlate highly with each other. The program can run either way. The graph below shows the top 30 correlates. The means are shown too.

Creating Graphs with Perl and GraphViz

2011-10-01T09:20:00.001-05:00

Yesterday I solved one of my data problems, but that just led to another one. I can now filter large correlation tables for (absolute) values above a threshold, but it's still laborious to connect those in a diagram that shows the relationships. So the next step was to look for a program to display graphs. Here, I don't mean bar graphs and pie charts and whatnot, but the mathematical object that was unfortunately also called a graph, which consists of vertices and edges. Or dots and lines connecting them, if you prefer. A map of a subway system is a kind of graph. Here's a very simple one:

With the correlations, I want to see what belongs with what on a graph that is generated automatically from a threshold I assign. In the example above, survey results show a perception that creativity is associated with artistic ability.

Fortunately, other people have already solved this problem, and it's just a matter of putting the machinery in place. I used two pieces of software, GraphViz (thanks to AT&T and the development team) and the perl interface for it (thanks to developer Leon Brocard). Both of these are open source and free to use.

The only other thing I had to do was extract the prompts for each item to correspond to the item codes. Otherwise, you get a nice graph showing that RATE1 connects to RATE2, which doesn't help understand what's going on.

I used the CIRP data with a .5 threshold to get these interesting networks of association. It takes just a few seconds to select the threshold value, run the scripts, and look at the resulting file. The code for generating the graphs from the output of the correlation summaries is here. There are many, many options for displaying the graphs. The gallery of GraphViz images shows off the versatility of the software.

This software does a great job of placing nodes logically so that the edges (the lines) are organized and neat.
Update: Here is a complete output with some cool new modifications.

A Recipe for Finding Correlates in Large Data Sets

2011-09-30T09:02:00.002-05:00

The internet has revolutionized intelligence. I've seen articles about how it's making us dumber, and I don't know if that's true, but it's certainly made me spoiled. In the old days if I had a computer problem I would just use brute trial and error, often giving up before finding a solution. Now, I just assume that someone else has already had the same problem and kindly posted the solution on a message board somewhere. So a few Google searches almost always solves the problem. Not this time.

This problem is a bothersome thing that comes up occasionally, but not often enough that I've taken action on it. It happens when I have a large data set to analyze and I want to see what's related to what. It's easy enough in SPSS to generate a correlation table with everything I want to know, but it's too much information. If there are 100 items on a survey, the correlation matrix is 100x100 = 10,000 cells. Half of them are repeats, but that's still a lot to look at. So I wanted a way to filter out all the results except the ones with a certain significance level.

I poked around at scripting sites for SPSS, but couldn't find what I was looking for. The idea of writing code in a Basic-like language gives me hives too (don't get me wrong--I grew up on AppleSoft Basic, but somehow using it for this sort of thing just seems wrong).

So without further ado, here's the solution I found. I'm sure someone has a more elegant one, but this has the virtue of being simple.

How-to: Finding Significant Correlates

The task: take a set of numerical data (possibly with missing values) with column labels in a comma-separated file and produce a list of what is correlated with what other variables at some given cut-off for the correlation coefficients. Usually we would want to look for ones larger than a certain value.

Note that some names are definable. I was using CIRP data, so I called my data set that. I'll put the names you can define in bold. Everything else is verbatim. The hash # lines denote a comments, which you don't need to enter--it's just to explain what's going on.

Step One
Download R, the free stats package, if you don't have it already. Launch it to get the command prompt and run these commands (cribbed mostly from this site).

# choose a file for input data and name it something
cirp.data=read.csv(file.choose())

# import the columns into R for analysis
attach(cirp.data)

# create a correlation matrix, using pairwise complete observation. other options can be found here
cirp.mat = cor(cirp.data, use ="pairwise.complete.obs")

# output this potentially huge table to a text file. Note that here you use forward slashes even in Windows
write.table(cirp.mat,"c:/cirpcor.txt")

Step Two
Download ActiveState Perl if you don't have it (that's for Windows). Run the following script to filter the table. You can change the file names and the threshold value as you like. [Edit: I had to replace the code below with an image because it wasn't rendering right. You can download the script here.]

Step Three
Go find the output file you just created. It will look like this:

YRSTUDY2 <-> YRSTUDY1 (0.579148687526634)
YRSTUDY3 <-> SATV (0.434618737520563)
YRSTUDY3 <-> SATW (0.491389963307668)
DISAB2 <-> ACTCOMP (-0.513776993632538)
DISAB4 <-> SATV (0.540769639192817)
DISAB4 <-> SATM (0.468981872216475)
DISAB4 <-> DISAB1 (0.493333333333333)

The variable names are linked by the <-> symbol to show a correlation, and the significance level (that is, the coefficient) is show in parenthesis. If you want the p-value, you'll have to do that separately.

Step Four (optional)
Find a nice way to display the results. I am preparing for a board report, and used Prezi to create graphs of connections showing self-reported behaviors, attitudes, and beliefs of a Freshman class. Here's a bit of it. A way to improve this display would be to incorporate the frequency of responses as well as the connections between items, perhaps using font size or color. [Update: see my following post on this topic.]

SAT Error Rates

2011-09-28T19:55:00.001-05:00

In "The Economics of Imperfect Tests" I explored the consequences of errors when making decisions with a test. By coincidence, the College Board came out with something very similar a few days later: their revised College and Career Readiness Benchmark [1]. This article and a related study [2] from 2007 give statistics that can illuminate the ideas in my prior post.

When testing for a given criterion, it's essential to be able to check how well the test is working. This is a nice thing about the SAT: since it predicts success in college, we can actually see how well it works. This new benchmark isn't intended for predicting individual student performance, but groups of them. It looks like a bid for the SAT to become a standardized assessment of how well states, school districts, and the like, are preparing students for college. One caveat is mentioned in [2] on page 24:

One limitation of the proposed SAT benchmark is that students intending to attend collegeare more likely to take the SAT and generally have stronger academic credentials than thosenot taking the exam. This effect is likely to be magnified in states where a low percentage ofthe student population take the exam, since SAT takers in those states are likely to be highachievers and are less representative of the total student population.

The solution there would be to mandate that everyone has to take the test.

As with the test for good/counterfeit coins in my prior post, the benchmark is based on a binary decision:

Logistic regression was used to set the SATbenchmarks, using as a criterion a 65 percent probability of obtaining an FYGPA of a B- orhigher [...]

The idea is to find an SAT score that gives us statistical assurance that students above this threshold have a 65% probability of having a college GPA of 2.67 or better their first year of college. There are some complexities in the analysis, including the odd fact that this 65% figure includes students who do not enroll. Of the students who do enroll, table 4 on page 15 of [1] shows that of those who met the benchmark, 79% of them were 'good' students having FYGPA of B- or better (i.e. 2.67 or more). For the purposes of rating the quality of large groups of students, I suppose including non-enrolled students makes sense, but I will look at the benchmark from the perspective of trying to understand incoming student abilities or engineer the admissions stream, which means only being concerned with enrolled students.

Using the numbers in the two reports, I tried to find all the conditional probabilities needed to populate the tree diagram I used in the prior post to illustrate test quality. For example, I needed to calculate the proportion of "B- or better" students. I did this three ways, using data from tables in the article, and got 62% to within a percentage point all three times. The article [1] notes that this would be less than 50% if we include those who don't enroll, but that must be an estimate because a student obviously doesn't have a college GPA if they don't enroll.

Here are the results of my interpretation of the data given. It's easiest to derive the conditional probabilities this direction:

In the tree diagram above, 44% of students pass the benchmark, which I calculated from table 5 on page 16 of [1]. The conditional probabilities on the branches of the tree come from table 4 on the previous page. Note that there's a bit of rounding in both these displays.Using Bayes Rule, it's easy to transform the tree to the form I used in the first post.

The fraction of 'good' students comes out to 62%, which agrees closely with a calculation from the mean and standard deviation of sampled FYGPA on page 10 of [1], assuming a normal distribution (the tail of "C or worse" grades ends .315 standard deviations left of the mean). It also agrees with the data on page 16 of [1], recalling that high school GPAs are about half a point higher than college GPAs in the aggregate.

Assuming my reading of the data is right, the benchmark is classifying 35% + 28% = 63% of students correctly, doing a much better job with "C or worse" students than with "B- or better" students. Notice that if we don't use any test at all, and assume that everyone is a "B- or better" student, we'll be right 62% of the time, having a perfect record with the good students and zero accuracy with the others. Accepting only students who exceed the benchmark nets us 79% good students, a 17% increase in performance due to the test, but it means rejecting a lot of good students unnecessarily (44% of them).

In the prior post I used the analogy of finding counterfeit coins using an imperfect test. If we use the numbers above, it would be a case where there is a lot of bad currency floating around (38% of it), and on average our transactions, using the test, would leave use with 79 cents on the dollar instead of 62. We have to subtract the cost of the test from this bonus, however. It's probably still worth using, but no one would call it an excellent test of counterfeit coins. Nearly half of all good coins are kicked back because they fail the test, which is pretty inefficient, and half the coins that are rejected are good.

We can create a performance curve using Table 1a from page 3 of [2]. The percentage of B- students is lower here, at about 50% near the benchmark, so I'm not sure how this relates to the numbers in [1] that were used to derive the tree diagrams. But the curves below should be self-consistent at least, coming all from the same data set. They show the ability of the SAT to distinguish between the two types of students.

If we set the bar very high, we can be relatively sure that those who meet the threshold are good (B- or better) students, but this comes at a cost in false negatives as we saw before. The "sweet spot" seems to be at 1100, with a 65% rate of classifying both groups correctly. Using this criterion, it's 15% better than a random coin toss for predicting both good and poor academic performers.

It's clear that although the SAT has some statistically detectable merit as a screening test via this benchmark, it's not really very good at predicting college grades.As others have pointed out, this test has decades of development behind it, and may represent the best there is in standardized testing. Another fact makes the SAT (and ACT) unusual in the catalog of learning outcomes tests: we can check its predictive validity in order to ascertain error rates like the ones above.

Unlike the SAT, most assessments don't have a way to find error rates because there is no measurable outcome beyond the test itself. For example, tests of "complex thinking skills" or "effective writing," or the like. These are not designed to predict outcomes that have their own intrinsic scalar outputs like college GPA. They often use GPAs as correlates to make the case for validity (ironically sometimes simultaneously declaring that grades are not good assessments of anything), but what exactly is being assessed by the test is left to the imagination. This is a great situation for test-makers because there is ultimately no accountability for the test itself. Recall from my previous post that test makers can help their customers show increased performance in two ways: either by helping them improve the product so that the true positives increase (which is impossible if you can't test for a true positive), or by introducing changes that increase the number of positives without regard to whether they are true or not.

It's ironic that standardized tests of learning are somehow seen as leading to accountability when the tests themselves generally have no accountability for their accuracy.

What to Expect When You're Assessing

2011-09-09T12:42:00.000-05:00

Along with Kaye Crook and Terri Flateby, I will be leading a one-day pre-institute workshop at the 2011 Assessment Institute in Indianapolis. This is a large national conference led by Trudy Banta and her team at IUPUI. It runs from October 31- November 1, with the pre-institute workshops on October 30.

The description of our workshop "What to Expect When You're Assessing" is:

This workshop is intended for faculty and administrators whohave responsibility for administering assessment activities at the program,department, or higher level. Through hands-on activities, participants willlearn essential skills for supervision of the whole assessment cycle, includinggood reporting, tips for data analysis, avoiding assessment pitfalls, goodpractices with tools like rubrics and curriculum maps, as well as management approachesto get the best out of your team using calendars, policies, and institutionalreadiness assessment. The workshop is appropriate for those with little assessmentexperience as well as those who would like to further develop their existing practicesto create sustainable and meaningful assessment programs.

The reason for offering the workshop is to help institutions grow their own expertise in leading assessment processes. Because gaining trust of faculty and administrators within the organization is so important, it's a good strategy to find someone who already has that trust and teach them about assessment rather than hiring an assessment expert from outside who then has to win everyone's trust.

I asked the ASSESS-L email list for their "Must-knows for a new assessment coordinator" (thanks to Katy Hill, Sean A McKitrick, and Rhonda A. Waskeiwicz for their responses). The results were interesting for a noticeable dearth of technical items, and an emphasis on political and personal skills, some of which actually de-emphasize technical knowledge, including:

It's okay for things to not be perfect.
One has to 'suspend disbelieve' at times with regard to rigor

The lists are insightful, and have helped me think about the one-day program we're putting together. Roughly, it's about one half technical stuff:

The basic idea of assessment loops
Common terms and what they mean in practice
How to write good reports
Use of rubrics and curriculum maps
Data analysis and presentation

One quarter is strategies for working with people toward a common goal:

Appreciative inquiry
Responding to specific challenges (this was a topic on ASSESS-L too)
Setting expectations

The rest, which is heavily represented in the results of my email are about management of the process, including:

Assessing institutional readiness
Calendars
Working with other groups on campus (e.g. faculty senate, center for teaching and learning)
Administrative buy-in
What software tools can do and what they can't do

The overall objective is for participants to walk out of the workshop with a concrete plan in hand, as well as more resources and contacts that will help them find success.

I welcome comments or suggestions. More materials will be forthcoming.

Edit: In addition to the ASSESS-L archive, there is a wonderful site Internet Resources for Higher Education Outcomes Assessment hosted by University of North Carolina and maintained by Ephraim Schechter, a familiar name in assessment circles. That page is a familiar open window on my browser, and an essential bookmark for anyone interested in assessment.

The Economics of Imperfect Tests

2011-09-05T14:17:00.002-05:00

It's fascinating to me how attracted people are to rankings: colleges, sports teams, best cities to live in, most beautiful people, and so on can seemingly be put in an order. Of course, it's ridiculous if you stop to ask if the quality in question could really be as simply as a one-dimensional scalar that can be measured with such precision. But it doesn't stop new lists from being generated. Rank order statistics (e.g. saying that Denver is number one and Charlotte is number two on the list) come with their own sort of confidence intervals, so that we really should be saying "City C is rank R plus or minus E, with probability P." Computing these confidence bounds is not easy, and I've never seen it done on one of these lists.

Leaving aside the issue of bounding error, the generation of the numbers themselves is highly questionable. Often, as with US News rankings of colleges, a bunch of statistics are microwaved together in Frankenstein's covered dish to create the master ranking number. You can read the FAQ on the US News rankings here. It seems that consumers of these reports are in such a hurry, or have such limited attention spans, that we can only consider one comparative index. That is, we can't simultaneously consider graduation rate in one column and net cost in another to make compromise decisions. Rather, all the variables have assigned weights (by experts, of course), and everything is cooked down into a mush.

A more substantive example is the use of SAT scores for making decisions about admissions to college (in combination with other factors). In conversations in higher ed circles, SATs are sometimes used as a proxy for the academic potential of a student. It's inarguable that although there is some slight predictive validity for, say, first year college grades, tests like these aren't very good as absolute indicators of ability. And so it would seem on the surface of it that the tests are over-valued in the market place. I've argued that this is an opportunity for colleges that want to investigate non-cognitive indicators, or other alternative ways of better valuing student potential.

But the question I've entertained for the last week is what is the economic effect of an imperfect test. I imagine some economist has dealt with this somewhere, but here's a simple approach.

How do we make decisions with imperfect information? We can apply the answer to any number of situations, including college admissions or learning outcomes assessment, but let me take a simpler application as an analogue. Suppose we operate in a market where there is a proportion $g$ of good money and counterfeit, or bad, money $1-g := b$. [You need javascript on to see the formulas properly.] We also have a test that can imperfectly distinguish between the two. I've sketched a diagram to show how well the test works below.

A perfect test would avoid two types of errors--false negatives and false positives. These may happen with different rates. Suppose an agent we call the Source goes to the market to exchange coins for goods or services with another agent, the Receiver. Assume that the Receiver only accepts coins that test "good." It's interesting to see what happens.

The fraction of good coins that the Source apparently gives the receiver is: $gt+b(1-f)$
The fraction of good coins actually received by the Receiver is $gt$

So the Source will obtain more goods and services in return than are warranted, an excess of $b(1-f)$. The inefficiency can be expressed as the ratio $\frac{b(1-f)}{gt+b(1-f)}$, which is also the conditional probability Pr[false positive|test = "true"]. (Pr means probability, and the vertical bar is read "given.")

There are two factors in $b(1-f)$: the fraction of bad coins and the false positive rate. So the Sender has an incentive to increase both. Increasing the fraction of bad coins is easy to understand. Increasing $1-f$ means trying to fool the test. Students who take SAT prep tests are manipulating this fraction, for example.

So we have mathematical proof that it's better to give than to receive!
In a real market, it might work the other direction too. The Receiver might try to fool the Sender with a fraction of worthless goods or services in return. In that case, the best test wins, which should lead to an evolutionary advantage to good tests. In fact, in real markets we see that at least for easily-measured quantities like weight.

In many cases, the test only goes one direction. When you buy a house, for example, there's no issue about counterfeiting money since it all comes from the bank anyway. The only issue is whether your assessment of the value of the house is good. The seller (the Source) has economic incentive to fool your tests by hiding defects and exaggerating the value of upgrades, for example. It's interesting to me how poor the tests for real estate value seem to be.

In terms of college admissions or learning outcomes assessment, we don't hear much about testing inefficiency. The effects are readily apparent, however. For example, the idea of "teaching to the test" that crops up in conversations about standardized testing. If teachers receive economic or other benefit from delivering students that score above certain thresholds on a standardized test, then they are the Source, and the school system (or public) is the Receiver. It's somewhat nebulous what the actual quality the tests are testing for because there isn't any discussion that I can find about the inherent $b(1-f)$ inefficiency that must accompany any less-than-perfect test. For teachers, the supply of "currency" (their students) is fixed, and they don't have any incentive to keep back the "good" currency for themselves. It's a little different from the market scenario described above, but we can easily make the switch. The teachers are motivated to increase the number of tested positives, whether these are true or not. They would also find false negatives galling, which they would see as cheating them out of goods they have delivered. As opposed to the exchange market, they want to increase both $gt$ and $b(1-f)$, not just the latter. They are presumed to have the ability to transmute badly-performing students into academic achievers (shifting the ratio to a higher $g$), and they can also try to fool the test by decreasing $f$. It is generally assumed that the ethical solution is to do the former.

A good case study in this instance is the New York City 2009 Regents exam results, as described in this Wall Street Journal article. The charge is made that the teachers manipulated test results to get higher "true" rates, and the evidence given clearly indicates this possibility. The graphs show that students are somehow getting push over the gap from not acceptable to acceptable, which is analogous to receiving a "good" result on the test. One of these is reproduced below.

Quoting from the article:

Mr. Rockoff, who reviewed the Regents data, said, "It looks like teachers are pushing kids over the edge. They are very reluctant to fail a kid who needs just one or two points to pass."

This could be construed as either teachers trying to fix false negatives or trying to create false positives. The judgments come down as if it were the latter. The argument that this effect is due to fiddling with $1-f$ instead of increasing $g$ is bolstered by this:

Mr. Rockoff points to the eighth-grade math scores in New York City for 2009, which aren't graded by the students' own teachers. There is no similar clustering at the break point for passing the test.

I find this very interesting because it is assumed that this is the normal situation--that if there were no $1-f$ "test spoofing" going on, then we should see a smooth curve with no bump. The implication is that teachers don't have a good understanding of how students will test--that is, despite the incentive to increase the skill levels of students so that they will convert from $b$ to $g$ (and have a better chance of testing "good"), they just don't know how to do it.

Consider an analogous situation on an assembly line for bags of grain. Your job is to make sure that each bag has no less than a kilo of grain, and you have a scale at your disposal to test. Your strategy would probably be to just top up the bags so they have a kilo of grain, and then move on to the next one. Mr. Rockoff (and probably most of us) assumes that this is not possible for educators. It's an admission that teaching and testing are not very well connected. Otherwise, we would expect to see teachers "topping off" the educational level of students to pass the test and then moving on to other students, to maximize the $gt$ part of their payoff. This quote shows that the school system administrators don't even believe this is possible:

After the audit, the state said it took a series of actions and plans to conduct annual "spike/cluster analysis of scores to identify schools with suspicious results."

It's ironic that on the one hand, this latent assumption questions the value of the tests themselves, and at the same time the system is built around their use. Other language in the article includes such expressions of certitude as this:

Michelle Costa, a high-school math teacher in New York City, said she often hears from friends who teach at other schools who [bump up scores on] tests, though she doesn't do it. "They are really doing the student a disservice since the student has so obviously not mastered the material," she said.

Missing the mark by a couple of points is equated to "obviously not mastering the material." There is no discussion about the inherent inefficiencies in the test, although there seems to be a review process that allows for some modification of scores (called scrubbing in the article).

This situation is designed for teachers to try to affect $f$ as the most sensible approach. Teaching students how to take standardized tests is one way of doing that. This critique of one of the tests makes fascinating reading. Here's a short quote:

Do not attempt to write an original essay. You don't have time. Points are awarded and subtracted on the basis of a formula. Write the five-paragraph essay, even though you will never again have a personal or professional occasion to use this format. It requires no comprehension of the text you are citing, and you can feel smart for having wasted no time reflecting on the literature selections.

We don't usually know if the tests are meaningful. If we did, we would know the ratio $g:b$ both before and after the educational process, and we would be able to tell what the efficiency of the test was. This is essentially a question of test validity, which seems to get short shrift. Maybe it seems like a technicality, and maybe the consumers of the tests don't really understand the problem, but it's essential. Imagine buying a test for counterfeit currency without some known standard against which to judge it!

In education, the gold standard is predictive validity: we don't really care whether or not Tatiana can multiply single digit numbers on a test because that's not going to happen in real life. We care about whether she can use multiplication in some real world situation like calculating taxes, and that she be able to actually do that when it's needed, not in some artificial testing environment. If we identified these outcomes, we could ascertain the efficiency of the test. The College Board publishes reports of this nature, relating test scores to first year college grades, for example. From these reports we can see that the efficiency of the test is quite low, and it's a good guess that most academic standardized tests are equally poor.

Yet the default assumption is often that the test is 100% efficient, that is $f=t=1$: we always perfectly distinguish $g$ from $b$.

The perspective from the commercial test makers is enlightening. If the teachers, faced with little way of relating teaching to testing, and no hope of relating that to (probably hypothetical) predicted real outcomes, choose to modify $1-f$ as a strategy, what is the likely motivation of the test makers?

The efficiency of the test is certainly a selling point. Given the usual vagueness about real predictable outcomes, test makers can perhaps sell the idea that their products really are 100% efficient (no false positives or negatives). An informed consumer would have a clear goal as to the real outcomes that are to be predicted, and demand a ROC curve to show how effective the test is. In some situations, like K-12 testing, we have the confused situation of teachers not having a direct relationship to the tests, which have no accountability to predict real outcomes. It's similar to trying to be the "best city to live in," by optimizing the formula that produces the rankings.

Since there is an assumed (but ironically unacknowledged) gap between teaching and testing, even the testing companies have no real incentive to try to improve teaching, improving the $g:b$ ratio. It's far easier for them to sell the means to fool their own tests. By teaching students how to optimize their time, and deal with complexities of the test itself, which very likely has nothing to do with either the material or any real predictable outcomes, test makers can sell to schools the means to increase the number of reported positives. They are selling ways to raise $t$ and $f$ without affecting $g$ at all. Of course, this is an economic advantage and doesn't come for free. Quoting again from the critique of one test:

[S]everal private, for-profit companies have already developed Regents-specific test-preparation courses for students who can afford their fees [...]

It's as if instead of trying to distinguish counterfeit coins from good ones, we all engage in trying to fool the test that imperfectly distinguishes between the two. That way we can pretend that there's more good money in circulation than there actually is.

Understanding Assessment through Language

2011-09-04T07:45:00.004-05:00

Over the summer I wrote a paper that explores the role of language in outcomes assessment: how it can help and how it can get in the way of understanding what's going on. This exercise clarified for me the long-term importance of eportfolios, and the risks inherent to abstract forms of assessment. I will look for a publisher eventually, but in the meantime I welcome your comments.

"Understanding Assessment through Language" (.5MB pdf)

The Most Important Problem in Higher Education

2011-09-03T11:10:00.002-05:00

Arrow's Impossibility Theorem is a fascinating application of mathematics to social science. Arrow took up the problem of how to find a voting system that meets certain reasonable criteria, such as taking into account the preferences of more than one voter. He showed that not all of the criteria can be met simultaneously in a single system because it creates a logical contradiction.

I have thought that it would be interesting to try the same trick with higher education. Informally, we might think of a list of best-case qualities we would wish for an educational system, such as universal access, a mechanism for cost deferral (like loans), public subsidization at a given level, and so on. In these lists, which I scribble on napkins at diners, the hardest quality to come to grips with is the idea of certification: a way of knowing that a particular student showed a certain level of accomplishment.

The idea of accomplishment is subtle. Here are at least three ways of looking at it:

We might be interested in what a person can do in the future, as a prospective employer would. Can Tatiana synthesize organic compounds? In this sense, accomplishment has to do with the predictive validity of inducing future performance from past performance. Sports statistics embody this idea: a batting average in baseball, for example.
Another effect of accomplishment is the usefulness of experience in being a consultant. If Tatiana swam the English Channel back in 1981, she might not be able to do it again today, but she could probably tell you some important things to know about it.
A third effect is social standing. Accomplishment brings its own rewards in terms of access to more connected people, the media, and so on.

All of these effects usually derive from indicators of direct accomplishment. This isn't always the case, of course. Some people are famous just for being famous. This latter phenomenon should be seen as deleterious because it dilutes and obfuscates the real value of accomplishment (perhaps related to Gresham's Law?).

This last issue is confounded by the fact that accomplishments are often judged subjectively. Someone might think the screenplay I wrote is wonderful, but that would be a minority view. We can disagree about subjective assessments, so 'accomplishment' itself can be very fuzzy. Even in the case where some putative objective fact is presented (e.g. swimming the English Channel), there is room for debate about the significance of that accomplishment, and what it means in terms of the effects listed above. The common denominator is to keep track of evidence of the accomplishment itself (we would call it authentic assessment data in higher ed land), and let the endorsements and comments ebb and flow with the tides. So if Lincoln's Gettysburg Address receives poor reviews the day after, we still have it around 150 years later to review for ourselves. The historiography of accomplishments is almost as interesting as the things themselves.

Existing systems for tracking accomplishment are inadequate to the task. Let's look at some of them.

In higher education, we issue diplomas and certificates, sometimes with decorations like sum laude, and identifying an area of study. The reputation of the issuing institution lends itself to the holder of the degree, and some subject areas are worth more than others. This is rather like a guild system, a gate-keeper approach. Some of the problems are:

It doesn't allow for autodidacts who learn outside the system, and so demands a significant investment in time and money.
There isn't much 'partial credit': completing 99% of requirements does not translate to 99% accomplishment since you don't get a diploma.
It's too coarse-grained: we need more information than "John got an engineering degree from State U," or even transcripts.
The lending of reputation from a school's name to the individual distorts the actual accomplishment of the individual.
It's very expensive and time-consuming to become certified.

At higher levels, the choice of a dissertation advisor can matter greatly, which leads to a second sort of certification: the letter of recommendation and other ways of personally endorsing someone. Problems with this approach include:

There is no limit to how many letters one can write.
There is incentive to be overly generous in letters you do write, or at least there is a disincentive to be negative.
It's very difficult to compare letters from different endorsers since they are not standardized.

Self-certification using a resume to document or summarize one's accomplishments are unsatisfactory because they have to be checked for accuracy, leading to one of the other forms of tracking. There are often significant penalties for getting caught in outright fabrication, but the problem of verifying the truth of claims remains difficult and time-consuming.

If we were interested only in predictive validity of statements like "Tatiana can speak French fluently," then the ideal case would be to have a perfect testing system to ascertain proficiency only in areas of interest. This is the Western Governors University approach, and it also exists in the form of professional board exams. However, these are coarse-grained hurdles that have to be overcome to get credit for a course or enter a profession, and the skills actually needed for a job may have only a passing resemblance to the test. For example, there are many, many types of engineering jobs, and the board exams cannot possibly cover the skill sets in that level of detail. This results in a rather arbitrary, and certainly inefficient, standard of accomplishment.

What are some requirements for an accomplishment tracking system?
Here's a starter list.

Identity is valid. We have to be confident that the accomplishments weren't outsourced or bought on eBay and then the credit transferred to someone else.
The system has to be open and transparent, so that we don't create a bottleneck that is just another gatekeeper.
The system has to be self-correcting with respect to inflation of accomplishments and outright errors.
Individuals have the final say over their own accomplishment profiles.

We can probably think of more requirements, but this is enough to begin with. The identity problem can really only be solved with personal contact. That is, a person has to be convinced that some other person did something. No amount of biometrics is ever going to solve that problem, I predict.

The system or systems should be web-based and easily accessed. There shouldn't be fees or artificial walls to viewing profiles. Of course, per number four above, profiles should be able to be restricted by their owners, like current social media sites sort of allow. It also implies that there is no negative information about an individual, only positive. This seems like a drawback, and I might be wrong about it, but I think allowing negative feedback (which necessarily allows others to say things about you publicly that you don't like) is fraught with problems.

So one possibility is a lightweight endorsement system that works something like the science pre-print site arxiv.org mashed up with Facebook. Well, like academia.edu, come to think of it.

I do some work that I think shows off my skills and upload a record of it to the system under my profile. This would necessarily be electronic, but could consist of video or any other kind of media--not just papers.
A portfolio-like central index keeps track of all my stuff, with pointers and meta-data about the works I have listed in my resume.
The index system also allows for endorsements by other registered users. Everyone's ID is real, verified by credit cards and phone numbers or something (optionally a unique SSL certificate).
Endorsements from experts would claim to authenticate that this is indeed my work, and say something appropriate about it that demonstrates a connection with the work itself.
All of this is standardized to the extent possible, and if the user allows it, fed out to a data export.

The result could look something like LinkedIn's recommendation system mixed up with Academia.edu's portfolio/vita.

On the surface, it seems like reciprocal recommendations would be a problem, as it tends to be in journals (citing each other's papers or including as co-authors). But with transparency, I suspect that clever people would create metrics that would attempt to summarize the evidence and connections that exist in the body of work. By tracing links of recommendations to create a network of association, it ought to be possible to see how worthwhile they are. It would be far from perfect, of course. But the possibility exists for a real "market" of reputation, where one's own standing is linked to those whom one has endorsed. It would be like picking stocks. In this way, there is an incentive for high-standing individuals to endorse newcomers who show promise. It probably doesn't prevent fads from distorting the 'reputation market', but if there is a relationship to the real world, eventually it becomes self-correcting. (This doesn't apply to all fields, and that problem may be unsolvable there simply because the discipline is more or less defined by the fads that exist at that time.)

The beauty of a transparent system is that a company or university wanting to evaluate a person's record could either use an off-the-shelf metric provided by one of these hypothetical companies that would spring up, or they could just look at the evidence themselves. Or they could hire their own experts to look at the portfolios. There is a sliding scale between how quick the evaluation is and how customized it is, and the best solution for one circumstance may be different from another. I assume this would be followed by an interview, which could be very substantive and deal directly with the evidence of performance that pertains to the position.

A good test case would be teaching evaluation. I haven't seen a public teaching portfolio. Maybe I just haven't come across one, but it's never occurred to me to post mine either, until now. I have big binders with student projects I've supervised. Why not post the best of those? The creation of a teaching-record portal would be a real service to higher education to standardize and validate that important part of academia.

Although I talked about 'a system', it would almost have to be many systems, each associated with some professional endeavor. This still allows for a resume to link all of the pieces together if necessary. In education, learning outcomes assessments would be a necessary component.

The rewards for figuring out how to accurately and efficiently certify accomplishment are worth the trouble. Education systems could be revolutionized to be more flexible and transparent, less bureaucratic and guild-like. Someone who teaches herself would not be valued less than someone who learned at a high-priced university--the proof would be in the sweet dairy desert dish, as the saying goes.

Building a Web Directory

2011-08-19T09:14:00.000-05:00

The project that consumed a large chunk of my time this summer was being project manager for our new university web site. It launched August 1 (on time and on budget, I'm happy to report), after a very intense summer of teamwork. Knowmad Technologies is the web development company that we partnered with to identify strategies, do design and architecture work, and deploy the site in a WebGUI content management system. On the university side I am fortunate to have a talented and dedicated team: Josh Nypaver and Erin Phipps. I took on small pieces of the project myself; one of those was the site directory.Together with William McKee at Knowmad, we created an end-to-end process for creating and maintaining a web-based directory of staff, faculty, and offices. Here's how it works:

1. I built a light-weight forms solution in openIGOR. The usual way of handling form data is to create a database table to save the entries. In order to maintain flexibility at the cost of query ability, I used plain text files instead, with a name=value syntax, like FirstName=David. Nothing is ever deleted--when entries are edited, it just appends the new version. This allows for complete roll-back and journaling. It also allows me to change the form without destroying a link to database tables. Most of the form is shown below.

2. Users log in to IGOR using their normal authentication through LDAP. This gives me lots of information about them since IGOR keeps track of who belongs to what group. So we know who's a biology professor and who works in the financial aid office, and so on. This is important because the directory is searchable by this criterion. Even better, these memberships are controlled by group administrators, so I don't have to personally modify them (although I spent several days cleaning it up, and we still don't have a great way to purge ex-employees other than manual changes).

3. When the user saves their directory information, the server sends an email to the web editor, who has administrative access to fix spelling, grammar, and standardize building names. Photos have to be hand edited right now to specification.

4. A Perl script is run to download all of this, as well as the names of groups for the directory (there's a flag to say which ones those are--we don't want committees showing up on the public web), and the index that says who is in what group. The three files (directory, groups, and index) are comma delimited text files.

5. The directory data along with edited photos is uploaded to the web server. Two scripts that William wrote put everything in place.Once that's done, it's live.

6. There are two sorts of queries that can be run on the web directory, which you can find here: (click the image)

The first searches on all fields, and the second just on the group name relationships. This allows us to display office directories as well as more general searches. This was William's wonderful idea, and it gives us a simple implementation that is very powerful.

Conclusions
It's hard to keep directories up to date, but by giving staff and faculty members direct control, we can shift responsibility from IT to the users. It seems to make them happy too. So far, the system has worked flawlessly, and the web directory is all I could have hoped it would be.Not all of the information from the IGOR form is yet on the web, and some of the fields really need validation or restricted options (like building names). But as just one element in a huge web project, we're quite happy with how it turned out.

Building a Teaching and Learning Technology lab

2011-07-01T17:19:00.000-05:00

Thanks to our always-helpful government grants office, we were able to find money to develop a technology lab for teaching and learning. We brainstormed for a way to augment our existing installations, which already provide a lot of technology for course instructors. Given our new foray into online learning, lecture capture seemed the best bet--it's focused, manageable under our budget, and can give us an immediate payoff. It's also scalable in application from simple video recording to much more sophisticated productions. It will shine a light on our distribution system for electronic media too, and prompt infrastructure improvements that will have multiplier effects.

"Lecture capture" is of course a terrible name for it. Lecturing means literally reading, which is what our academic forbears did before there were enough books to go around. Like wearing those heavy black robes in the May sun, some ideas are ripe for overhaul.

There's a nice introduction by Educause here (pdf), if you want to see what they have to say.

What if we started from scratch?

It's not just the robes and readings that have seeped into the weave of the academic tapestry. Exposing young minds to the conceptual treasures of our disciplines is constrained by the combinatorics of time and space so that that certain optimizations are so standardized we don't think about them. Like having students sit for fifty minutes at a stretch in classrooms. What if instead we could engage them for ten minutes at a time on focused topics all day long? Another optimization is that all students in the class get the same linear flow of narration, and the discipline itself gets strung out into a one-dimensional string, codified into textbooks and syllabi that have the same linear style.

Imagine if the world wide web were like that: a single linked list of information, so that you had to start at the beginning and work your way forward link by link until you got where you wanted to go. It's a lousy way to organize rich information.

Sometimes concepts do have prerequisites. Math is full of it. But even in math there are optional topics, and opportunities to go into more depth even the simplest subject. If you want, you can think of the long multiplication process you learned in third grade as a convolution product, which can be optimized with Fourier transforms. One of my professors in graduate school (David Kammler) took an approach he compared to "Island hopping" as practiced in the Pacific theater in WWII, which consisted of learning certain important topics in depth and assuming that students could then make logical leaps to nearby topics, which could be skipped or treated as exercises.

Imagine instead of a linear trip through a discipline, we captured key ideas and used those as anchors for a network of related facts, concepts, processes, etc. One immediate advantage would be the ability to easily update the network as the discipline changes. Group evolution is back in vogue? Add a node and stick it in there. No need to fell a forest for the new edition of a book. Another advantage is that focused topics can be crowdsourced, as in Wikipedia, with experts in sub-sub-sub-fields filling in blanks ad infinitum. Other thoughts:

interdisciplinary connections become easier to integrate naturally
because learning happens in detail, it's easier to assess a small topic than a big one, allowing for faster improvement of the captures and related pedagogy
custom methods can be applied more easily to particular topics. Think apps. When you get to the bit about singular value decompositions, you can download the stuff you need to see it in action, and then it can go away. It's modular.
It creates a way for professionals to agree on something. A whole textbook? Never happen. But a five minute tour of Darwinian evolution? Better odds. This standardizes at the right level. You can still customize your curriculum--just match assessments to content.

All of this is probably being done in bits and pieces now, this divide and conquer approach. But it hasn't gelled yet into a common framework supported by real infrastructure and common culture (like say Wikipedia is).

I don't want to go all Utopian here, but let me push this one step further. With a rich network of high quality instruction/assessment/activities modules, the whole idea of a course becomes outdated. I know, I know, that's scary stuff. But if we don't have to have students sitting in desks for fifty minutes at a time, why do we need them for 15 weeks? All that really matters is that they learn and demonstrate mastery of concepts that we think are important. So then our whole bureaucracy can go hard a-port and do something really interesting. All that bricks and mortarboard infrastructure can be focused on motivating students. I've written enough about that already, so I'll stop here.

I should hasten to add that our modest project at my institution has none of these grand ambitions. I will be selling the idea that a "lecture capture" doesn't need to be 50 minutes long, however, and keeping my eye on the horizon...

Teaching 9th graders operations research

2011-07-01T06:39:00.003-05:00

Our two-week project with Upward Bound came to a close yesterday. I've been working with Soumia Ichoua on her NSF-supported project to try to interest high schoolers in the fun side of math.

More to come after we've analyzed the assessment results and put the paper together.

Acknowledgements

Financial support for this work was provided by the National Science Foundation (NSF) through grant number 0927129 (Ichoua). This support is gratefully acknowledged.

An Addendum and Apology

2011-06-10T10:06:00.000-05:00

On Wednesday I wrote about my take-aways from the AALHE meeting in Lexington, and drew on some remarks from Trudy Banta. Some of the responses I got were justly critical of the way I mangled Trudy's message about the wider importance of the SAT example. It's true--I botched it, and I'd like to set the record straight so that Trudy doesn't begin to doubt her communications skills.

In order not to further fold, spindle, or mutilate someone else's message, let me say that what follows is my interpretation, and any faults should be ascribed to the author alone.

In a comment on the still-infantile state of the art of measurement, Trudy observed that even after more than 80 years of development and efforts to improve the SAT for its intended purpose, there is as much disagreement as ever about the validity of the SAT as a predictor of success in college. I intended to emphasize this point and what it portends for other projects that are even more ambitious. In the original article I left this argument dangling rather ineptly. So here it is, starting with a closer look at SAT, generalizing to the characteristics of industrialized tests (meaning massive, usually commercial, standardized instruments), and how we can do better with more authentic assessments.

Even with all the research that has gone into the SAT, in absolute terms it still isn't very good. The 2008 validity study from the College Board gives us (pg. 2):

[T]he weighted average correlation between SAT writing scores and English composition course grades was 0.32, after correcting for range restriction.

This is the most direct link I could find in the study between SAT and learning outcomes. Here we have actual writing in a standardized environment, correlated with course grades in coursework where writing is taught. If the course grade is related to how well students can write, then their potential as demonstrated by the SAT writing component ought to line up with it. And it does, but only to the tune of explaining 10% of the variance in grades (.32 squared). More generally, the variance in first year college grades explained by all the components of the SAT combined is 25% (page 5, squaring the adjusted R).

Any predictor with greater than zero R may be useful in the right context. But if we consider SAT's performance as an upper limit to what industrialized tests can do, it's a warning for current and contemplated high-stakes applications. This is a good place to segue to the mis-step in my prior article.

I gave three advantages of standardized tests. Here they are:

They can have good reliability, so they seem to measure something.
Because of this, they can create self-sustaining corporations that provide a standardized service to higher education, so there are fixed points of comparison.
Even if the validity isn't stellar, some information is better than none, right?

I was still thinking mostly of the SAT when I wrote this, so you might consider this the best case scenario. However, I intended it to be clear (it wasn't) that this is damning with faint praise: although these bullets describe how standardized tests have colonized an ecological niche successfully, these reasons aren't sufficient to warrant their use for high-stakes purposes. Like value-added comparisons of institutions, for example.

The missing "anti-bullets" that describe the corresponding disadvantages are respectively:

Depending solely on reliability is like searching for your lost keys under the lamp because that's where the light is. The drive for reliability creates artificial conditions like timed multiple-choice tests that have very little mechanical relationship to real-world application of knowledge. So reliability comes at a cost in validity.
The size of the testing industry means that it can use its resources to circumvent professional review of its products and sell them directly to politicians.
The validity comment is specious: "some" information isn't good enough in many circumstances. Imagine driving on a cliff's edge and only knowing where 10% of the road is. (An unfortunate real case is the publication of ranks of teacher "value-added" scores by the LA Times last year.) Exacerbating the general dearth of validity is is the fact that validity has to be determined locally after application of the test results (that is, to decide if some proposition about the actual results can be supported or not). It's very hard to do. Most of the time we don't know if our applications are valid or not.

I would like to apologize to Trudy for bungling the transition between my summary of her remarks and the bullets in the original. I detracted from the point and may have misled or confused some readers. Mea culpa.

We can now step beyond the problems and look for solutions, and Trudy gave a list of some of these alternative approaches. Reliability isn't produced in a factory in Iowa City: we can get good reliability using rubrics on authentic student work. Additionally, the convenience of industrialized testing can nowadays be matched with local technological solutions. If we want to use portfolios, we don't have to have a thousand file cabinets, or mail copies to prospective employers; it's all on the web.

The big win for local solutions is validity: with the right application of technique, we can link assessments solidly with pedagogy at many levels of resolution, from individual assignments up to the institutional level.

As noted above, the testing industry has a lot of ability to protect its own interests, including great access to policy makers. With that in mind, I should wrap back around to the theme of the policy conversations at the conference: the calls for accountability. It's not good enough to just resist the K-12-like solution; we have to find an alternative we find acceptable, and there are some good candidates available. That, I hope, is a clear and unmangled statement of the main message.

I don't mean to demonize "big testing," by the way. Maybe the best solution would be if we could convince them that authentic assessment and electronic portfolios are profitable new markets. I don't know what the chances of that are.

Inputs and Outputs

2011-06-09T05:15:00.001-05:00

There's an interesting article in The Atlantic. There's much more to it, but this quote gets to the heart of the experience for many classroom instructors at all levels, I imagine:

Professor X is shrewd about the reasons it’s hard to teach underprepared students how to write. “I have come to think,” he says, “that the two most crucial ingredients in the mysterious mix that makes a good writer may be (1) having read enough throughout a lifetime to have internalized the rhythms of the written word, and (2) refining the ability to mimic those rhythms.” This makes sense. If you read a lot of sentences, then you start to think in sentences, and if you think in sentences, then you can write sentences, because you know what a sentence sounds like. Someone who has reached the age of eighteen or twenty and has never been a reader is not going to become a writer in fifteen weeks.

The source here is the book by "Professor X" called In the Basement of the Ivory Tower.

In Seattle Education there is a three-parter on privatizing education. There's a great game-theory example I didn't know about:

Proponents claim that by encouraging competition, privatization can improve the efficiency of public services. But there can be serious drawbacks. For instance, before fire departments were publicly run, groups of firefighters sometimes set fires just to earn money by putting them out!

The evidence provided is more suggestive than compelling, but it's a nice business model (from the point of view of investors anyway) to control both supply and demand. In education this would translate as institutions not really caring much about education, but really caring about credentialism, and then seeking to maximize the number of credentials available and perceived as necessary to employment. In other words, if education is a gate to the promised land, put us as many toll booths as possible on the road between here and there.

Policy Questions Raised at AALHE

2011-06-08T08:38:00.001-05:00

I flew back yesterday from Lexington and the first AALHE conference. I found it very stimulating. I put faces to names from the ASSESS list server, which was delightful.

In the opening plenary, Trudy Banta gave us a broad perspective on the evolution of the measurement and accountability, pointing out the weaknesses of value-added derivations and standardized tests in particular, and suggesting authentic assessment (e.g. portfolios and their analyses) as useful alternatives.

One point is particularly compelling to me. Trudy mentioned the pedigree of the SAT, and it's not hard to imagine the many hours and dollars that have gone into fine-tuning this test. These are smart people working with a will for a long time toward a well-defined purpose: predicting how well high school students will do in college. In my own experience as an IR person, the SAT does add some predictive strength to linear models, but not much once high school GPA is considered. A handful of percent of R^2 is it. At my present institution, it's virtually worthless as a predictor of first-year grades, which points also to the known biases of the test.

In short, there is some usefulness to the SAT, but it may not warrant all the trouble and expense. And of course some schools are now SAT-optional. I've written before about how, as a market signal, SAT overprices some students and overlooks others, creating the opportunity to use other (e.g. non-cognitive) indicators to find good students.

Trudy's comments did not go this far, but it's not hard to connect the dots. It's an important point: if all this effort yields so little result, maybe we're doing it wrong. The alternative is to admit that maybe this is the best we can do, and that our ways of knowing will just never be much good.

It should be noted that the plenary was planned as a kind of debate with two viewpoints, but that because of a cancellation by Mark Chun, only one side was presented. So in the defense of big standardized tests, here are some advantages:

They can have good reliability, so they seem to measure something.
Because of this, they can create self-sustaining corporations that provide a standardized service to higher education, so there are fixed points of comparison.
Even if the validity isn't stellar, some information is better than none, right?

The conversation is much more subtle than "down with standardized tests!" But you may have noticed that fine distinctions aren't part of the public debate on the quality of higher education. It would be great if Mark and Trudy and other experts introduced a public discussion like this on a message board or something--somewhere that the complexities of the issues could be fully explored and commented on over time.

[Update, see my "Apology and Addendum" that goes with this section.]

Bookending the conference, the closing plenary was given by David Paris, and also included a historical and political overview of where we are now. He is the executive director of the New Leadership Alliance for Student Learning and Accountability. You can sign up for email updates on their website. Paraphrasing David, the Obama administration talks nicer, but wants the same things as the Bush administration. And what that seems to be is "accountability." One important take-away for me is that we (higher education) have a window of opportunity that will soon close. That is, we can "do it to ourselves" or "have it done to us." The 'it' is amorphous, but centers on the accountability for cost and learning trade-off. In other words, all this public noise about how bad college is will create changes, and we should try to shape those.

It's hard to disagree with that. One approach is the Alliance's Institutional Certification Program, which you can learn about here. I'd like to see some examples of how it works in practice, but on the face of it, it seems like a great idea, similar in some aspects to the Bologna 'centering' process in Europe. I would call it a kind of horizontal professionalization, complementary to the 'vertical' type provided by discipline-based accreditations and professional organizations. Overall, it seems like a serious and well-reasoned approach, and I look forward to finding out more about it.

There was a good Q&A afterwards, and I suggested that higher education needs access to data pertaining to what happens to students after graduation, and that the federal bureaucracy sits on a gold mine of such information. This didn't have a chance to become a real discussion because of the format, but my interpretation of David's response was that the student-level record project that was floated and shot down (with help from higher education lobbyists) would have solved that problem. So we are complicit in causing the problem.

I don't buy this. First of all, what we need most right now is a deep historical understanding of how higher education has affected lives after school (graduation or not), for individual institutions if not individual majors. I understand that the data are imperfect and that there are privacy issues to be solved. But I think 9/11 shows that any privacy concerns can be solved rather quickly if the motivation is there. Specifically, it seems like it should be possible to combine some combination of student loan records, FAFSAs and PELL grant records, tax records, federal employee records (including military) and so on to link where students took instruction, what their demographics were, and what happened to them afterwards. There are plenty of other places to find data, I'm sure, like state system databases.

The payoff for this is potentially enormous. Suppose we (higher education) can agree with the politicos that employment is a good outcome of a college education, and even talk about the details like what kind, how much pay, where geographically, and so on. Then we could look at what kinds of schools have what kinds of effects on what kinds of students. Is a biology program at my (private) university worth the cost premium over the public one down the road? What about access? Which schools are economic engines by taking low-income inputs and (eventually) producing high-income outputs?

The payoff is so great that I think that if the government really can't find a way to do this, we should figure out how to do it ourselves.

What we have now is survey data that shows that college generally pays for itself, but the price is going up. This is in opposition to the cries of critics that say students aren't learning much, if anything.

Here's my analogy. It's imperfect, but has the advantage of being vivid. Students are like movie scripts coming to our production companies. We have our specialties: niche art films or mass market gloss, and so on. Each script has an individual experience, no matter what our organization is. They get cast differently (different instructors), and we do our best to make a good fit usually. Or maybe we just get the cheapest actors we can find and hope for the best. We always have to rewrite the script to some extent, and our mission--the hope that gets buried in the Sisyphean rolling of semesters up and down the calendar--is that the screenplay comes to life and realizes its potential.

It takes a lot of money to make this happen, and it comes from investors who are increasingly grumpy. They say the movies are no good. They're too expensive and nobody likes them. Film critics like Arum and Roksa used data from a large number of scripts in production to claim that a large portion weren't being significantly improved in the rewriting stage.

But we don't have any real information about what the audience thinks. We can analyze the heck out of our own productions, and do six-sigma and professionalize all we want, but until we understand what the audience thinks, we don't really know if we're doing any good. Maybe all those films get shown once, and then end up in a storage shed.

Maybe some types of films shouldn't be made anymore. Maybe Forensic Dolphin Psychoanalysis shouldn't even be a major because there's no audience for it.

The recent explosion of for-profits makes the problem more urgent to solve. Most industries have to depend on their products working. We don't have much direct evidence one way or the other in the kind of detail we need to make decisions. It's starting to happen, for example with student load default rates getting attention. But we can go a lot further than that.

Imagine if we could sit down with the investors and agree on the long-term goals, have metrics that more or less tell us how well we're doing, and plan together how to get there. Are the goals strictly economic? Or do we want people to pursue happiness and bear arms? Even if it is just economics, it presents an opportunity to really understand how, say, liberal arts education matters in job mobility, lifetime income, and so on. Institutions could say "yes, you'll get a fine job, but in addition you'll have a fulfilling career." Or whatever their mission is. Instead of talking past each other about standardized assessments and authentic assessments, we could figure out how to work backwards from the real goals to real assessments that matter at the strategic level and then add institutional flavors that matter to us. That would be an exciting and productive conversation.

The alternative is grim. If we only focus on what goes on inside our production studios, the future of the nation is at the mercy of every critic with a theory or an agenda. I'm not sure which is worse: theories or agendas. Some will want to break down every step of movie making into a reductionist flow-chart, and create spreadsheets to show the rate of script-to-casting time or use biometrics to calculate charisma factors of the actors. There's no bottom to this because although movie scripts are only 120 pages, individual brains of our students have perhaps 100 trillion synapses each. If each one is, say eight bits of information, that's about four billion times the complexity of a movie script. Each one. Others will work backwards from agendas to create policies that make no sense in context.

Even if we think we've solved the problem and get universal agreement that our learning outcomes are being achieved at the university, how do we really know what the effect is after graduation unless we measure that? Maybe they're learning the wrong things. Maybe some small college has methods that work twice as good as ours. Maybe we can reduce costs and increase quality. The ingredients are wonderful: a diverse ecology of isolated experiments. We just can't see where the lab results are recorded in order to make conclusions.

Yes, we need individual student records. But we can't wait for that. The problem is too important. Maybe tax records and FASFAs can't be mashed up because of political or technical reasons (but do you really think Mark Zuckerberg couldn't figure this out in an afternoon?). If that's the case, then we have to find another way to measure historical and ongoing long-term outputs in such detail that it can inform institutional decisions.

What we're doing at present is creating our own dramatic screenplay: an epic version of "No Exit."

Evaluation Oddity

2011-05-23T08:52:00.000-05:00

This year we changed our course evaluation form from a very long list of management questions ("Did the instructor meet office hours regularly?") to a short one focused on learning outcomes. The report below is from my fall 2010 Calculus II class.

That's the whole form, except for free-form written responses. As a first look at the meaningfulness of results, I looked at the correlation between responses for each item, and got a surprise. To wit: while items 9,10, and 11 are tightly correlated, as shown on the matrix below,

and furthermore, items 12 and 13 are correlated at .88,

these two summative questions (12 and 13) do not correlate well at all with any of the other items, including 9, 10, and 11, which ask if the course was interesting, enjoyable, and did you learn. Do students not associate those things with their summative evaluations, or is there some other register that is engaged (i.e. they perceive that the last two questions are more personal, about the instructor instead of about themselves)? On the other hand, the very low correlation between item 2 (How much effort did you put into the course?) and the overall evaluation seems to be a good sign, implying that students differentiate between their own performance and that of the instructor.

SACS Changes to the Principles Proposed

2011-05-03T12:46:00.001-05:00

A few weeks ago I posted about recommended changes to our SACS-COC accreditation standards. Today the review committee's recommendations for change were announced. You can find the document here. There is a comment period on this document open until May 18.

Wordle Again

2011-04-29T12:26:00.000-05:00

I mentioned Wordle.net a while back. It's a way to create 'word clouds' from text. It occurred to me this afternoon that it would be neat to put course descriptions in from the catalog for each major. I tried a couple of them, shown below.

Small College Initiative

2011-04-29T08:38:00.001-05:00

I attended the SACS Commission on Colleges Small College Initiative this week. You can find the slides on the SACS web site here. Below are some of my notes and observations.

Mike Johnson talked about CR 2.5 (Institutional Effectiveness) and pointed out the difference between assessment and evaluation. In my interpretation of his remarks, the former is gathering data and the latter is using it to draw conclusions for action. In my experience, this gap is where many IE cycles break down. Signs of this are "Actions for Improvement" that:

Are missing altogether
Are too general or vague to be put into practice
Report that everything is fine, and no improvements are necessary
Suggest only improvements to the assessments

I have a lot more to say about this (there's a surprise), and am preparing a talk for the Assessment Institute and a paper for NILOA on related topics.

Another important point is that IE cannot be outsourced or even in-sourced to a director. The whole point is that is is a collaborative exercise in striving to achieve goals. I think results are proportional to participation. In a similar vein, Mike noted that computer software can help organize reporting, but doesn't magically solve the problem of generating quality IE loops. Garbage in = garbage out.

A wonderful suggestion was to use the creation of "board books"as a way to encapsulate IE reports in a natural way that's already being done. Mike's larger point here is that we already have many real IE processes--all institutions that manage to survive use data one way or another--and there's no need to create an artificial one for reporting. I saw this during a review, where the institution had wonderful processes in place, but didn't include that documentation in the compliance certification, and instead reduced all that rich information into a four-column grid that "looks like it's supposed to." Of course, one problem here (in my opinion) is that there doesn't seem to be a standardized way to look at IE processes. If we were serious about it, we'd do inter-rater reliability studies and create tight rubrics with lots of examples in a library, showing what's acceptable and what's not. I think this would go a long way toward reducing the number of out-of-compliance findings. Way back when--over a decade ago--I heard a SACS VP complaining that even back then, IE had been around a long time and college should know what to do by now. That's true as far as it goes, but it should be acknowledged that: 1) it's very hard to satisfy committees, and 2) it's not entirely clear what is acceptable and what's not. Part of the problem is that while the theory of IE loops is easy to understand, practice is far more difficult. Sort of like Socialism.

There was a clarification that it is acceptable for institutions to sample programs for reporting 3.3.1.1 in lieu of reporting outcomes for every single one. There is supposed to be a policy statement about this on the web site, but I couldn't find it after several minutes of searching the list for 3.3.1, effectiveness, outcomes, sampling, etc. If someone finds it, please let me know. The main thing is that it should be representative and not look like it was cheery-picked (e.g. reporting only programs that have discipline-based accreditation).

It was noted that CS 2.10 implicitly has learning outcomes reporting requirements, making it a pseudo-IE standard. I included this in my recommendations for 'fixes' to the Principles in my letter to SACS, posted here for comment. Not many institutions seem to be flunking it, though, unlike 3.3.1.1 (see below).

The fifth year report was highlighted in a break-out session. You can find additional slides on this topic on the website here. Out of 39 institutions, 28 were cited on 3.3.1.1, and alarmingly, the number of citations on the QEP Impact Report is 33%. Although this says that the review process is no piece of cake (which is good--it should be meaningful), it points to a problem. In fact, the rationale for the Small College Initiative is to help address this problem, which is particularly acute for small schools. As a side note, over lunch I talked to an IR director who speculated that there is a bias against citing large schools, particularly ones with high rankings. It would be really interesting, in conjunction with the inter-rater reliability study I fantasized about above, to have blind reviews of 3.3.1.1. Given the growing emphasis on student learning outcomes (including the new credit-hour rules), a whole separate system for learning outcomes may need to be developed. One of the challenges on the horizon, in my view, is the contradiction of grades. On the one hand they are the basic unit of merit for courses, with a vast bureaucracy behind them. By contrast, grades are not seen as 'real' assessments. This needs to be fixed. I don't know if Western Governors University's model is the answer, but what we have now makes no sense, and it is impossible to explain to the public.

Reasons given for flunking a QEP included:

Bad planning, which leads to a bad report. One kind of bad plan is one that's too broad.
Failure to execute it, e.g. if a new administration comes in and lacks enthusiasm for the old project
Not talking about goals and outcomes in the report. Hard to believe.
Not describing the implementation (just narrating the creation, perhaps)
Not collecting or using data
Bad writing. Ironic, since so many QEPs are about writing.

Tips for writing QEP impact reports:

Follow the directions given in the SACS policy
Address all the elements
Keep narrative to 10 pages. (~~You can apparently link out to other documents, which I hadn't heard before. I thought everything had to be in 10 pages~~.) [Edit: see the update below]
Use data, but include analysis--don't just put in graphs with no explanation.

Networking over lunch, I gleaned a couple of nifty ideas. At one institution, faculty contracts include a 'gotcha' clause, which stipulates that if assessment reports are not done by date X, then the prof has to stick around until date Y to finish them. This provides an incentive to get them done. Also, the reports are broken down into phases across the academic year, so that not everything is done at once. Smart.

Update: Mike Johnson posted a note to the list server saying that the links in the 10 page (max) Impact Report can only be internal to the report itself, which does not allow 'extra room'. In his words:

Links within a disk or flash drive are okay as long as the documents that are part of the link are included in the ten page maximum length. So please do not use hyperlinks to documents as a means to lengthen the report.

Motivation and Intelligence

2011-04-28T06:54:00.001-05:00

Angela Duckworth et al have a new article in the Proceedings of the National Academy of Sciences (PNAS) entitled "Role of test motivation in intelligence testing." The abstract reads, in part:

Intelligence tests are widely assumed to measure maximal intellectual performance, and predictive associations between intelligence quotient (IQ) scores and later-life outcomes are typically interpreted as unbiased estimates of the effect of intellectual ability on academic, professional, and social life outcomes. The current investigation critically examines these assumptions and finds evidence against both. [...] After adjusting for the influence of test motivation, however, the predictive validity of intelligence for life outcomes was significantly diminished, particularly for nonacademic outcomes.

The press on the paper includes Science Daily's "Motivation Plays a Critical Role in Determining IQ Test Scores," and Discover's blog post "IQ scores reflect motivation as well as 'intelligence'."

We include 'Effort' in our faculty-assessed end-of-semester FACS survey, and have found a link between grades and this effort rating. Of course, it could just be that professors who thing students work hard also tend to give them higher grades, so over the summer we will look at multi-year correlations to eliminate that confounding factor.

The graphs below (courtesy of Google charts) shows GPA in red, and hours completed in blue above the distribution bars for rated student effort across all classes. The heights of the bars give the percent of the distribution that received that rating. The left one is our first survey, Spring 2009. The one on the right is from Fall 2010. The sample size has increased as we've gotten better participation.

The drop in credits earned is due to more first year students being included in the sample. The year-by-year story is similar, except that the overall averages have an interesting shape as ecological samples from first year to fourth:

The first year students in the graph are the first class to fall under the new (much higher) admissions standards. The number is the average effort rating on a scale of zero (minimum effort) to three (great effort). This is for N=1403, Fall 2010. Note that there is a survivorship bias, so that we'd expect the averages to grow as the time-in-school increases. I don't yet have true longitudinal data.

Inter-rater reliability was measured by finding the frequency of exact matches for two instructors rating the same student. There were 385 instances of this, with a match rate of 50.7%. It's not hard to find the rate of pure-chance matches (dot-product the distribution with itself), but I haven't done that. In the past, the chance of matching randomly has been around 35%. See this source for more on that.

Getting to Expression

2011-04-13T07:23:00.000-05:00

Barbara Fister's "Why the 'Research Paper' Isn't Working" has some interesting observations about the teaching and assessment of composition, and I made the connection to the deductive/inductive divide I've been going on about lately. Let me reframe the latter as the "language/expression" divide as a preface:

A language is a set of knowledge that usually comprises vocabulary, methods, reference points of common knowledge, and a web of connections between concepts. Understanding a language is always a prerequisite to being able to produce it intelligently.
Expression is the illumination of new ideas, new connections, creation of new parts of the language to contribute to the existing corpus. It is realized with different styles in varying degrees of fluency, and allows the display of insights or brilliance.

This is the analytical/deductive vs. creative/inductive divide that I've blogged about before, for example in the previous post. We see this division everywhere. Learning how to use paint versus expressing yourself in the medium.

A "life-long" learner must either become used to learning new languages all the time or else not plan to live long. My 'stack' of languages to learn currently includes R programming, German, and photography. I think of it in over-generalized terms as a progression from confusion to understanding to expression. I'll come back to that idea.

The "photography language" is one I dabbled in when it meant smelly chemicals and a long time between when you shot a photo and when you got to look at it. Nowadays digital photography obviates many of the skills that one needed, and it does something else very important. It's not any easier to do digital photography--you have to master software instead of stop bath--but it's so much quicker to get from the snap to the view that you can learn from trial and error in real time. This is a huge advantage. Conservatively, the gap between taking a shot on film and holding a print in your hands is at minimum several hours (leaving aside Poloroids or other quickie formats). With digital it's a matter of seconds. So learning the language through sheer trial and error has been accelerated by a factor of, say 2 hrs/2 seconds = 3600.

Photo: David Eubanks, some rights reserved

Different learners approach learning language in different ways. Some people like to read all the manuals first, and others start pushing buttons. My wife (laughing at the end of a long work day, above) learned Italian by working all the exercises in two textbooks and then spending a month in Italy. I struggle along with German because I don't have the patience to memorize vocabulary. I try to bridge the confusion/understanding divide by reading novels translated into German (it makes the language much simpler), and look up words that come up frequently enough. Her way is much more efficient than mine.

So, in this epistemological vivisection of learning, the challenge for faculty is to teach and assess the crossing of two metaphorical bridges:

Land o' confusion -> Understanding -> Expression

In "Complexity as Pedagogy" I showed how it's possible to take a very narrow road straight to Expression. That is, one can encapsulate a small part of the language and use it to get right to the fun part. Because, let's face it, creating is fun! And if anything distinguishes humans from the rest of the biological kingdom, it's our blabbing--we like to talk.

An art professor once told me how to learn to draw. He said, just draw your hand over and over again in different positions. After about 500 times, you should be pretty good at it. I don't know if he was joking or not, but this is an example of simplifying the language to the point where you can quickly become expressive.

The practice of assessment should be very different across this divide. Testing language fluency can take many forms, but it's always about correctness, speed, conformity to convention, and so on. One is not supposed to be creative on a spelling test. Otherwise I would have gotten better grades in grade school. Similarly, we're not suppose to invent better names for state capitals for that test, or help the Germans organize the genders of their nouns better.

Assessing language seems easy because of this necessary emphasis on mastering form. Vocabulary tests, concept inventories, and the like are easily administered, and even testing understanding of subtle connections through the use of the language itself is straightforward.

Example: In teaching logic, it's simple to write down a logical argument and ask students to justify each step with an axiom or theorem, or even let them find errors with the proof. The only way students can be successful is if they have a good understanding of the language.

This ease of assessment is a bane, and a great peril to learning. Let me finally get to the points I liked about the article I cited way back at the beginning of this piece. Starting with the idea of forcing students to master arcane rules of correct citations, the author notes more broadly that

I have long agreed with Richard Larson who wrote way back in 1982 that the research paper as taught in college is an artificial genre, one that works at cross-purposes to actually developing respect for evidence-based reasoning, a measured appreciation for negotiating ideas that are in conflict, or original thought.

An artificial genre that is at cross-purposes with original thought. That's pretty damning. But it's these very mechanics of any language that are easily defined, easy to get agreement on, and easy to assess. It's a quick slide down the slope to standardization of a form that becomes inimical to the actual intent of the enterprise! This happens all over the place. Whole subjects taught in school exist only because of such inertia, like Geometry in high school--there's no reason kids should be learning plane geometry with rulers and protractors in this day and age, but it's been so deeply standardized that it's become part of the culture. But I digress.

Barbara goes on to illustrate the point with a fascinating example of how students react to the low-complexity standard we've set institutionally:

I hate it when students who have hit on a novel and interesting way of looking at an issue tell me they have to change their topic because they can’t find sources that say exactly what they plan to say. I try to persuade them otherwise, but they believe that original ideas are not allowed in “research.” How messed up is that? The other and, sadly, more frequent reference desk winch-making moment involves a student needing help finding sources for a paper he’s already written. Most commonly, students pull together a bunch of sources, many of which they barely understand on a topic they know little about, and do their best to mash the contents up into the required number of pages.

Does this sound like a road map from Confusion to Expression? It doesn't to me--it sounds like a Skinner Box: mash the button to get the food pellet.

It shouldn't be hard to fix this problem. That's the good news. The recipe is simple:

Focus the language to a small useful subset. In terms of composition, it would mean picking a topic that's narrow enough to actually learn something about quickly.
Demonstrate and assess--with feedback!--fluency in this new language. Have conversations about what confusion levels, what you know you know, and what you know you don't know. There are lots of creative ways to organize this with mind maps and such, and it also can be fodder for oral presentations, or other engagement activities. Develop fluency in real time.
Emphasize expression and creativity over form as far as it can be pushed. This isn't always possible, e.g. in logic--you have to be 100% correct--which is why the focus is so important. If you have to absolutely master some topic in order to be creative, make it a small one.

Note that I am not advocating "free form creativity" devoid of any content or ultimate value. This might be fun for the students, but I don't see how it accomplishes any useful learning objectives. But there's a lot of road between many of our current practices and goofing off in the name of creativity.

Barbara's ending paragraph is apposite:

But if you want first year college students to understand what sources are for and why they matter, if you want them to develop curiosity and respect for evidence, your best bet is to start by tossing that generic research paper. As for those who will complain that students should have learned how to paraphrase and cite sources in their first semester – we’ve tried to do that for decades, and it hasn’t worked yet. Isn’t it time to try something else?

Yes.

Creativity as the Pinnacle of Learning

2011-04-07T05:48:00.000-05:00

In a recent post on the topic of memory, I noted that this skill was at the base of the revised Bloom's Taxonomy (or taxidermy?). This morning I woke up thinking about the other end: creativity. I've mused about the role of creativity before, and how to teach and it (here and here).

It's easy to make the unwarranted leap from creativity to aesthetics, because we associate art justifiably with a creative process. But I prefer to think of creativity as the production of new knowledge in any context. Let me give a pedantic example:

All men are mortal.
Socrates is a man.
------------------
Socrates is mortal.

The conclusion follows deductively from the two statements above it, so it is not the production of new knowledge. This is the hammer that fell on Bertrand Russell in his quest for ultimate true by means of logic. Logical, rigorous deductive thinking is an essential skill, but it's not creative. In contrast, Aristotle's encoding of logic into language was creative, but I have a more interesting example.

The other day I saw an interesting problem posted on the math subreddit. The diagram below shows a laser beam coming from the right and striking a perfect mirror (the thick black line at the bottom) at angle b, bouncing off, and then striking another mirror placed at angle a with respect to the first one:

The gray line is imaginary here--it extends the line of the bottom mirror to illustrate angle b, but the light hits the very end of the mirror itself. The light will continue to bounce off the mirrors in some way. Where does the beam end up going? I will put the solution at the end, in case you want to think about it first.

The point is that I suspected there should be an elegant way to think about the problem, where the solution--all solutions--would be obvious. So I cast about, looking for it. This is rather like trying to find the light switch in a dark room, as Andrew Wiles put it (see my other posts on creativity for the link). I gave up before I found it. I found an inelegant solution, which was correct, but wasn't creative enough to be called elegant. It was sort of a plodding, "add up the cumulative effect" solution, where you sort of crush a problem with the weight of logical facts until it leaks out its secrets like a garlic clove exudes oil.

My lack of blazing imagination does, however, illustrate that the creative process itself deserves its own "taxonomy." In other words, there are qualitative differences to creative enterprise. Let's take a look.

Creativity as producing new information can start with sheer randomness. Flipping a coin and writing down the results is creative. This sounds too trivial to be counted, but it's not. In fact, it's the singular most important spark of novelty in history. I have two examples. First, physicists have wondered how galaxies formed. If the big bang started from a single point, for example, why wasn't everything thereafter perfectly uniform? Where did all the novelty come from? One proposal is that tiny differences in the primordial universe were seeded by quantum events, which we know to be deeply random. So the largest structures in the universe may have started with infinitesimal randomness. Cool, right?

The second example is the evolution of life, which explores via an ecology a vast space of possible designs for living things. This exploration proceeds by random mutation of genes, and other ways in which genetic material may get mixed around (like parasitism), or a bacteria's lascivious lifestyle with regard to DNA. This is not the deeply puzzling randomness of quantum mechanics, but the sort that emerges from complex systems that is sometimes called chaos.

Randomness is a great entry point into creative thinking. The casting about for novelty is a skill in itself. It requires courage to be wrong, a good idea of how to recognize your intellectual quarry when you've found it, and determination--because it takes a long time for randomness to hit the right target. Louis Pasteur's "Chance favors the prepared mind." has two parts: chance, and preparation. The latter is a formed in the laborious mastering of some discipline or subject.

The whole idea of serendiptipy is based on these two elements, and our culture has benefited handsomely from it: rubber, penicillin, radioactivity, and many more are on the list. Wikipedia has many examples here.

In the last post, I showed an example of a game designed for high schoolers that is aimed at creative thinking in a mathematical context. The essential skills are being able to understand the problem and do basic math (easy), and cast about for creative solutions (fun, I hope). These solutions will start with guessing.

Guessing is a step up from randomness. Humans aren't very good at true randomness--we have to depend on the world around us for that, like moldy bread crumbs falling accidentally into a Petri dish. I suspect that good guessing is an art unto itself, and that it can be taught and practiced. There's an MIT course on the art of making educated guesses with regard to estimation (how many gas stations are in the US, do you think?). Here's the course description from the Open Courseware site (it's free!).

This course teaches the art of guessing results and solving problems without doing a proof or an exact calculation. Techniques include extreme-cases reasoning, dimensional analysis, successive approximation, discretization, generalization, and pictorial analysis. Applications include mental calculation, solid geometry, musical intervals, logarithms, integration, infinite series, solitaire, and differential equations.

This is targeted at students with a good math foundation (everybody at MIT, I guess), but I find it exciting because it shows how to teach a whole course on guessing in the context of a discipline. There's no reason that this couldn't be done in other subjects just as well. Guess-and-check is a fundamental human skill that reinforced our knowledge of the world. Think about kids and the funny way they conjugate verbs at first because they are guessing based on simple rules (e.g "I eated my peas, daddy"). The guess is close enough to communicate, and as an additional reward, they glean information about new complexities of language, if someone is kind enough to point out the right way of saying it.

Problem-Solving might be the next step in the creative chain of being. This is a natural continuation of randomness and guessing, which results in the production of new knowledge in some applied context. This works in art as well as math, I think. It's the evolution from random doodles to purposeful artistic creation. Problem solving weds the analytical/deductive process, discipline-specific skills and knowledge with the trial-and error process that I've described in prior posts on creativity. This is the nuts and bolts of creative production.

Inspiration may or may not be teachable. If we help students to be good seekers of randomness, good guessers, and good problem-solvers, can we help them elevate themselves to inspired thought? I don't know, but I guess that we can provide a fertile environment for this, and foster it in individuals who might otherwise have not reached their potential. I don't really believe that we can take every math student and produce another Gauss or Euler, but we can ameliorate one of the great hidden human tragedies--the many, many inspired thinkers who never got the intellectual cultivation they needed to allow their talents to flower.

This is all first-draft thinking. An interested group of discipline experts could turn these rough ideas into something applicable to a curriculum or institution. To include ways to assess creativity at each step along the way. Disciplines can learn from each other and share approaches, opening up the possibility of interdisciplinary learning. I often though that the math students could benefit from watching art students critically review each others' work.

Here's the solution to the problem. I have redrawn it, but I saw it first here. The original problem was in terms of a tiny billiard ball, but I changed it to a laser beam. The key insight is that reflections preserve angles, so that instead of imagining the beam bouncing off at the same angle (incidence = reflection), imagine it passing through the mirror as if it were a pane of glass. Then add another pane of glass where the return bounce would have occurred, so that copies of the mirrors look like spokes on a wheel separated by angle a. This illustrates clearly that the beam will swiftly exit the mirrors and go on its way in most arrangements. The whole process is laid bare. I've illustrated it with a=45 degrees below. This is an inspired and elegant solution, unlike my workable but problem-solving brute force approach (not shown).

Complexity as Pedagogy

2011-03-27T10:03:00.002-05:00

Learning an academic subject usually goes like this. First you have to get used to a new language and ideas expressed in that language. At first you're quizzed on meaning, but you're increasingly required to actively use the new concepts (using a different part of your brain to do so). When you get far enough along, you can begin to critique work using the new ideas. Ideally you critique your own work as you produce it. I think of this as the analysis and creativity cycle. The former demands use of the language (terms, ideas, grammar and syntax), whereas the latter depends on insight, trial and error, and imagination.

The problem is that it usually takes a long time to get any good a new language before you can effectively be creative in it. Take speaking in a foreign language, for example. You can't just make up words and sentences that 'sound German' and hope for it to be meaningful. A German friend gave a good example of this in reverse, when his sister tried to order a sandwich in New York:

Using "Ich will einen Hamburger bekommen" (I want to get a hamburger.) to create the English approximation: "I will become a hamburger."

But 'will' and 'become' mean very different things in the two languages, and there's no way to approximate the knowledge.

You have to understand how verbs are consummated and whatnot...then you can express yourself.

All this preparation is a hindrance if you just want to introduce a student to a subject. One of my professors, David Kammler, exposed me to the interesting idea in math that it should be taught like MacArthur's island-hopping campaign in the Pacific in WWII. That is, don't try to do everything, just teach strategic material that an able student can use as basis to explore other parts as needed. This has the effect of lessening the language burden.

There is an advantage to that. Being creative is like play, or it can be. It reinforces the language and gives students the thrill of discovery. This is how games work. If you make the learning curve too steep for beginners, they may not even make it through the rule book. Moreover, it's not necessary to have a lot of language/rules to have an interesting game. Chess, for example, or Backgammon, or Poker. The trick is to create a tight language that is unambiguous, and thereby allow a platform for exploration that can be easily checked by the rules.

Math and computer science are particularly amenable to this approach. My colleague Soumia Ichoua has a grant to do operations research activities, and part of it is an outreach project for middle or high school students. We are working with the university's Upward Bound program to coordinate logistics, and have created several types of logistical games for the kids.

The idea is to create a simple system that has these three properties:

It's easy to understand the rules and check them. (low complexity rules = simple descriptive language)
It's difficult to solve the problem. (high complexity solution = perfect solution method has a long description, and is unlikely to be found.)
There is a wide range of possible solutions. (expressive language, allowing heuristics to be effective)

As it turns out, there is a major class of algorithmic challenges in computer science that has this property. They are in a class called NP. One example is the traveling salesman problem, where you try to find the shortest circuit to visit a list of destinations. The rules are very simple: add up the length of each leg (transit from one location to the next) in the circuit. The total distance is the sum. Finding the shortest path is very hard, however. So it's a perfect environment to employ creativity in a real discipline-based problem without having to learn a bunch of language and rules first.

The image above is taken from Wolfram Alpha, and shows a minimal distance circuit for the given points. The space of possible solutions is vast, but humans can use heuristics to find reasonably good solutions.

One of the game boards was created by my daughter, and is shown below. There's a story behind it about King Lolly, who needs to keep his people fed in winter, and involves moving food markers around from the grainery and the castle to the villages. There are several scenarios to keep it interesting.

Other logistics games include air-dropping supplies into a remote region, delivering hot dogs to hot dog vendors in a city, and planning for natural disasters. The last one is the focus of Dr. Ichoua's research, which has her visiting FEMA to get real data.

A draft of the rules for Lollyland is found below. The plan is to use undergraduate helpers to assist rising 9th and 10th graders in pairs. We are developing assessments for cognitive and non-cognitives.

Rules

Food travels from the castle and keep to the villages over the roads. If you have to move the food through the castle, the hungry people will eat half of whatever you send through (rounded up). Each food marker is enough to satisfy one hungry villager.

Winter in Lollyland

In the small kingdom of Lollyland, King Lollygaster has to plan well to feed his people during the winter. Being wise, he has constructed a grain tower in his castle and another one at a distant keep. Between these two sources, he must distribute emergency food if the winter is harder than expected. Besides the town surrounding the castle, which has its own store and can take care of itself, there are four large villages that have no such protection. It is these that the kind must provide for in time of famine.

Scenario 1.

The ground stayed frozen so long that the crops were put out late. The king must distribute food to the four villages to keep them from starving. He isn’t good with numbers, so he puts you in charge and entrusts you to get his people fed.

Set up: 25 food in the castle, and 11 in the keep. Each village gets 9 hungry people.

Questions:

Can all the villagers be fed?
How many ways are there to distribute the food to feed them?

Scenario 2.

After many happy years King Lollygaster became too frail to rule effectively, and turned the kingdom over to his son Lollyfright, who promptly dumped the old man down a well. Lollyfright neglected to lay up much in the way of food storage, assuming the winter would be mild. You are now the Minister of Happiness, and tasked with the following situation:

Set up: 18 food in the castle and 8 in the keep. Each village gets 9 hungry people. All of the roads are closed due to lack of interest in clearing snow from them. Choose four roads to open and deliver food; the others will remain closed.

Questions:

What is the most number of villagers that can be fed?
What are the best roads to open?

Scenario 3.

After the disaster, the merchant class of Lollyland dethrone Lollyfright and send him into exile. They form a governing council and put you in charge of planning for the next famine.
Set up: Assume there will be 7 hungry villagers in each village, the populating having declined of late. With this decline there is less ability to clear roads of snow. Assume that you can open only three roads in the worst case.

Question: How much food do you need to put at the castle and the keep?

Why Assessment?

2011-03-25T19:10:00.000-05:00

When the word came filtering down through the academic rumor enhancement facility (an endowed building, thankfully) that we'd have to do assessment--this happening in a past demarcated by one (1) millennium post, one (1) century post, and two (2) decades--we academics ran for cover. Some are still down there in their fox holes, waiting for the war to be over, not having access to the headlines proclaiming victory for the other side.

I have to wonder why the emphasis was ever put on assessment in the beginning? It's certainly a bad word to build a PR campaign around, like trying to advertise lamprey on the menu. Anyway, assessment isn't the point at all. This has only recently occurred to me, which may be symptomatic of all those sedimentation layers from said rumor enhancement facility that piled 'assessment' on top of 'measurement' and so on and on. You could make chalk out of the stuff if you had the patience.

When I finally made the synaptic leap, it was a shock. But perhaps I can be excused that because of the continual sustained emphasis on assessment in conferences, publications, and public discourse.

Assessment is one component of a theoretical institutional effectiveness process, but it's not the most important one. The spiral of excellence is to be climbed by:

Setting goals
Finding assessments for same
Gathering data and analyzing it
Making changes that may improve things

The first step and the last are the most important, and if you could only pick one, number four would be it. If the mandate were to set goals and try to achieve them, anyone taking that seriously would pretty quickly figure out some sort of assessment was in order. So why do we start with assessment first? Historical accident?

Compounding the premier position of assessment is the way we sometimes talk about the activity, as if we were doing science in order to do engineering. What I mean here is that I characterize science as pinning down cause and effect, whereas engineering is the use of those principles to accomplish some aim. This is all mixed up in learning outcomes assessment, because it's often expected to do both at the same time. Here again, the heavy emphasis on assessment gets in the way. To find a link between cause and effect, we have to vary starting conditions, and compare ending conditions. Then we have to hope that the universe is reasonable and lets us get away with inductively assuming that because it worked last time it will work again the next time.

Suppose we want to test some variations of fertilizers on corn to see which one works best. We'd try to keep everything constant except our treatment (type of fertilizer), which varies in some suitable way. We'd be careful to randomize over plots of ground to average out soil or topographical variance, and so on. We would carefully document all the conditions (water, acidity, parasites, etc.) during the experiment, and finally assess the yield at the end. Now all that stuff I just reeled off is part of the experiment, but with learning outcomes we typically wish everything away except for the last bit--assessing the yield. I have yet to see an effectiveness model that has a box asking for the experimental conditions during the teaching process. Maybe that's because few would do it.

The effect of this casual approach to experimentation is to make the assessment less valid. This may be remedied to some extent by bringing back into light the context, which is exactly why having teaching faculty involved with broad discussions that include but are not limited to assessment results is so important. Given the typical (and understandable) lack of rigor for most learning assessments, the results are more like a psychic reading than science. And that's fine. It should be told that way, so that faculty members don't get so stressed out about a "scientific" use of the results. Focusing more heavily on goals and improvements makes it easier to engage faculty, and puts the emphasis where it ought to be: taking action.

Memory as a SLO

2011-03-24T07:02:00.000-05:00

Why don't we teach memory skills as learning outcomes? Given the amount of memorizing required for many college courses, wouldn't it make sense to have a first-year course on the general subject: theory and application? Or better yet, integrate techniques within classes. I can remember my uncle, who is a doctor, reciting a long list of bones from some mnemonic device he'd used back in school, decades before. The subject has a long rich history. The Romans practiced memory techniques, and probably considered it essential (see Wiki article "Art of Memory" for sources and details). The book Moonwalking with Einstein: The Art and Science of Remembering Everything is currently the number four best-seller on Amazon.com.

Memory shows up as the base of the pyramid on the revised Bloom's Taxonomy (taken from here).

I'm guessing that there are institutions out there who do this, but I haven't heard about (or ironically forgot about). Does anyone have a "memory across the curriculum" program? It seems like this skill does belong at the base of cognitive skills, and moreover is transferable from one discipline to another. Simple things could easily be done, like teaching students how big flash-card decks should be and how to time the reinforcements, or providing online resources like SuperMemo:

SuperMemo is based on the insight that there is an ideal moment to practice what you've learned. Practice too soon and you waste your time. Practice too late and you've forgotten the material and have to relearn it. The right time to practice is just at the moment you're about to forget.

Here's another tip: move your eyes

If you’re looking for a quick memory fix, move your eyes from side-to-side for 30 seconds, researchers say. Horizontal eye movements are thought to cause the two hemispheres of the brain to interact more with one another, and communication between brain hemispheres is important for retrieving certain types of memories.

Finally, here's a very cool way to remember numbers called pegging:

Basically what pegging does is to turn a number (any number), into a set of phonetic sounds or letters. These sounds are then joined together to form words, and these words may then be linked together to form a series of images. Finally these images may then be committed to memory. This enables an individual to recall numbers of up to (and above) 100 digits, with relative ease.

It seems like helping students improve memory would have broad positive consequences for the rest of their learning. They could spend less time in the process, stop cramming vocabulary just before the test (and forgotten just after), and take away a useful life skill. We could ask students to memorize presentations instead of reading them.

Memory as an outcome is attractive because it's easy to assess, there are straightforward techniques that are known to work, and it can be incorporated into the curriculum. The biggest barrier might be faculty who don't know and don't want to learn the techniques themselves.

Undoubtedly the drama department already does some of this--you can't take the flash cards on stage with you for the performance. Music too, maybe. Can we broaden the scope and take similar advantage institutionally? I'd love to know if someone out there is doing this already.

Compare and Contrast

2011-03-23T07:56:00.000-05:00

These pairs of quotes come from different viewpoints about how we know what we know, and how we should conduct higher education. Write a 3000 word essay in which you reflect how these statements fit into the practice of teaching and learning as you currently understand it. Due Friday.

If we take take in our hand any volume, of divinity or school metaphysics for instance, let us ask, Does it contain any abstract reasoning concerning quantity or number? No. Does it contain any experimental reasoning concerning matter of fact and existence? No. Commit it then to the flames, for it can contain nothing but sophistry and illusion. -- David Hume, An Enquiry Concerning Human Understanding, ed. L. Selby-Bigge, p. 163, taken from here.

Versus:

Epistemological anarchism is an epistemological theory advanced by Austrian philosopher of science Paul Feyerabend which holds that there are no useful and exception-free methodological rules governing the progress of science or the growth of knowledge. It holds that the idea that science can or should operate according to universal and fixed rules is unrealistic, pernicious and detrimental to science itself. --Facebook Group on Epistemological anarchism(!)

And

We say that a sentence is factually significant to any given person, if and only if, [she or] he knows how to verify the proposition which it purports to express—that is, if [she or] he knows what observations would lead [her or him], under certain conditions, to accept the proposition as being true, or reject it as being false. – A. J. Ayer, Language, Truth, and Logic

Versus:

[T]he meaning of a word is its usage in the language. – L. Wittgenstein

And

The CLA measures were designed by nationally recognized experts in psychometrics and assessment, and field tested in order to ensure the highest levels of validity and reliability.-- Advertising flyer for the Collegiate Learning Assessment

Versus:

It is a common misconception that validity is a particular phenomenon whose presence in a test may be evaluated concretely and statistically. One often hears exclamations that a given test is “valid” or “not valid.” Such pronouncements are not credible, for they reflect neither the focus nor the complexity of validity. – College BASE Technical Manual

And finally this pair of quotes from a New York Times article:

In a talk to the nation's governors earlier this month, [Microsoft founder] Mr. Gates emphasized work-related learning, arguing that education investment should be aimed at academic disciplines and departments that are "well-correlated to areas that actually produce jobs."

Versus:

At an event unveiling new Apple products, [Apple CEO] Mr. Jobs said: "It's in Apple's DNA that technology alone is not enough -- it's technology married with liberal arts, married with the humanities, that yields us the result that makes our heart sing and nowhere is that more true than in these post-PC devices."

What positions on the red/blue spectrum are more consistent with your own practices and beliefs? There are not right answers. Well, unless you believe there are I guess.

Improving the Principles of Accreditation

2011-03-15T09:25:00.001-05:00

If you are in the SACS/CoC region, you know all about the Principles of Accreditation, the document that outlines accreditation standards, and which every institution must use to report compliance every ten years and (with fewer sections) every five years in between. Until March 31, the Commission is accepting comments that will inform a review of said document. This is an excellent opportunity to make your voice heard in what is, after all, a peer-review process.

I have some observations I will post here for comment before sending them off to the Commission, in order to see if others agree or can suggest better approaches. I will restrict my comments to the institutional effectiveness (IE) sections. So here goes.

Note: CR = Core Requirement, CS = Comprehensive Standard, and FR = Federal Requirement

CR 2.10 The institution provides student support programs, services, and activities consistent with its mission that promote student learning and enhance the development of its students. (Student Support Services)

Although this isn't in the IE sections formally, it has a requirement that student support services "promote student learning and enhance the development of its students." This is a clear IE requirement, and is exceptional in that no other functional units, including academic programs, are required to pass this level of detailed IE review as a core requirement. Taken literally, an institution can be sanctioned severely for not assessing learning for student support services, which seems out of line with the more strategic level requirements that comprise the CR sections. I would suggest moving the IE language to 3.3.1.1, quoted below in the current version:

3.3.1 The institution identifies expected outcomes, assesses the extent to which it achieves these outcomes, and provides evidence of improvement based on analysis of the results in each of the following areas: (Institutional Effectiveness)
3.3.1.1 educational programs, to include student learning outcomes
3.3.1.2 administrative support services
3.3.1.3 educational support services
3.3.1.4 research within its educational mission, if appropriate
3.3.1.5 community/public service within its educational mission, if appropriate

Note that "student support services" doesn't appear. There are administrative and educational support services, but not student support services. The nomenclature needs to be cleaned up so we know exactly what we're talking about. One approach is to assume that any service that has learning outcomes is an educational support service, and therefore the modification could be as follows:

End the statement of CR 2.10 after "mission," omitting the IE component.
Change 3.3.1.3 to read "educational support programs, to include student learning outcomes."

CS 3.5.1 The institution identifies college-level general education competencies and the extent to which graduates have attained them. (College-level competencies)

This is the general education assessment requirement. Note that it doesn't appear in 3.3.1 unless general education is defined as a program by the institution. On the other hand, 3.5.1 is NOT an IE requirement--there is no statement about use of results to improve, just that you assess the extent to which students meet competencies.This is a knotty puzzle, so let me take it one part at a time.

First, the phrase "the extent to which" is ambiguous. Does it mean relative to an absolute standard or a relative one? This is by no means splitting hairs. If it means the former, then the institution MUST define what an acceptable competency is for each outcome, and presumably report out percentages that meet the standard. If it's a relative "extent to which" then simply reporting raw scores of a standardized test against national norms would work.

Example (absolute): Graduates will score 85% or more on the Comprehensive Brain Test. In 2010, 51% of graduates met this standard.

Example (relative): In 2010, graduates averaged 3.1 on the Comprehensive Brain Test, versus a national average of 2.9.

The standard is silent about the complexities of sampling graduates too. Are ratings from all graduates to be included? I would assume not, since this standard generally doesn't apply to IE processes because of the impracticably of it.

My sense of this is that we should allow institutions to define success in either absolute or relative terms, as best suits them, and include this standard with the other 3.3.1 sections so that it formally becomes part of IE. This will resolve the ambiguity about whether or not general education is program, and require that general education assessments actually be used for improvements. It also would broaden the scope to include students generally, not just graduates, who may have been out of general education courses for two years by then.

The modification could simply be to:

Add: CS 3.3.1.6 general education, to include learning outcomes
Delete: CS 3.5.1

Finally, let's look at 3.3.1.1 itself. The first issue is subtle. It concerns the meaning of the language "provides evidence of improvement based on analysis of the results". This can mean two different things, and I've seen it interpreted both ways, causing confusion.

The first interpretation is that it means that you have to demonstrate that improvement happened. This is a very high standard. It means things like benchmarking before changes are occurred, and then assessing the same way later on to see what impact occurred. When I hear speakers talk about QEP assessment, this is generally assumed, but it leaks over into the other IE areas too. Anytime you take a difference between two measurements it amplifies the relative error--this is a basic fact from numerical analysis. So you have to have very good assessments and they have to be objective (for reliability) and numerous (small standard error) and scalar (so you can subtract and still have meaning). Also, pre-post tests are the only method that can be used with any resemblance to scientific method. That is, you can't survey two different populations to compare unless you think you can explain all the variance between the populations (read Academically Adrift to see how problematic that is even for educational researchers). My objections to this are:

No small program could ever meet this standard; the N will never be big enough.
Many subjective assessments are very valuable, but are useless in this interpretation
We don't yet have the technology to create scientific scalar indices of things like "complex reasoning" or other fuzzy goals, despite the testing companies' sales literature.[1]
Random sampling is often impossible, which introduces biases that are probably not well understood
Pre-post testing is quite limited, and severely restricts the kinds of useful assessments we might employ

The other interpretation is simply that we use analysis of assessment data to take actions that would reasonably be expected to improve things, but we don't have to prove it did. This is the standard I've seen most widely applied, except perhaps for QEP impact. Arguably the higher standard should apply to QEP, but that's beyond the scope of my comments here.

In practical terms, the second interpretation is the most useful. It has to be borne in mind that assessment programs are mostly implemented by teaching faculty, who are probably not educational researchers by training, and in my experience tend to become frozen with a sort of helplessness if they think they are expected to track learning outcomes like a stock ticker tracks equity prices. I blogged about this a while back. Too much emphasis on proofs of improvement is paralyzing and counterproductive. On the other hand, free-ranging discussions that include the meaning of results, subjective impressions from course instructors, and other information that speaks to the learning outcome under consideration is a gold mine of opportunities to make changes for the better.

The most powerful argument against the strict (first) interpretation, however, is that there is simply no way to guarantee improvement on some index unless (1) one cheats somehow, manipulating the index, or (2) the index reflects some goal that is so obviously easy to improve that it's trivial. Either way the meaningfulness of the program vanishes, and we are left with many programs that are out of compliance (not showing improvement) or in compliance in name only (showing fake improvement or showing trivial improvement).

My recommendation is to clarify the meaning of the language to read (bold emphasizes the change):

CS 3.3.1 The institution identifies expected outcomes, assesses the extent to which it achieves these outcomes, and takes actions based on analysis of the results in each of the following areas: (Institutional Effectiveness)

It should be obvious that the actions are intended to effect positive change, since this is rather the whole point of IE.

There is another issue with 3.3.1 that deserves attention. This concerns goals or outcomes that are not about student learning. It seems to me from the interpretations I've seen of 3.3.1 by reviewers and practitioners that the methods that apply to learning outcomes leak over to administrative areas, and that the expectations for compliance are tilted out of plumb. For example, there seems to be an expectation that every unit do surveys and have goals related to those, and that moreover this is necessary and sufficient.

Part of the problem may be the language "the extent to which," which almost insists on a scalar quantity. But in fact, the most important goals for a given unit's effectiveness may have little to do with surveys and not be naturally a scalar.

One example I saw takes issue with a compliance report that presented "action steps" as goals. An action step might be "Approval of architectural drawings of the new library by 5/1/11." This sort of thing shows up all over Gantt charts:

(image courtesy of Wikipedia)

This method of tracking complicated goals to completion is very common and very effective. The bars may or may not represent progress toward completion--they are simply timelines. So, for example, an OK stamp from the city engineer on your electrical plans is not a "percent to completion" item--it's either done or not. In other words it's Boolean.

Reviewers can be allergic to Boolean outcomes like:

Complete the library building on time and on budget.
Implement the new MS-Social Work program by Fall 2012
Gain approval via the substantive change process for a full online program in Dance by 2013 (good luck!)
Maintain a balanced budget every fiscal year.

For some reason, there seems to be a bias against this kind of goal, and I can't figure out why. These are obviously important operational items, key to the effectiveness of respective units. But a unit that includes these and doesn't have a satisfaction survey may be cited, whereas the reverse may be true too.

It may be that I'm making a mountain out of a termite hill here, but I think the language of "extent to which" could be changed to make it clearer that Boolean objectives are sometimes the natural way to express effectiveness goals.

For example, 3.3.1 could read (change bolded):

CS 3.3.1 The institution identifies expected outcomes, assesses success in a manner appropriate to each outcome, and takes actions based on analysis of the results in each of the following areas: (Institutional Effectiveness)

I know the non-parallel structure will offend the grammarians, but someone more expert can consider that conundrum.

One final nit-pick is that outcomes may not really be expected, but simply striven for--aspirational outcomes, in other words. If they are expected, then by nature they are less ambitious than they might otherwise be. I also put the odd dangling " in each of the following areas" at the front where it belongs, so my final version is this:

CS 3.3.1 In each of the following areas, the institution identifies aspirational outcomes, assesses success in a manner appropriate to each outcome, and takes actions based on analysis of the results: (Institutional Effectiveness)

[1] I've written much about assessing complex outcomes before, and this is not the page to rehash that issue. The short version is that learning happens in brains, and unless we understand how brains change when we learn, we are not able to speak about causes and effects as they relate to the physical world. See this article about London taxicab drivers to see a study that links learning to physiological changes to the brain. I don't mean to imply that assessing complex outcomes is useless, but just that we should be modest about our conclusions. It's called complex for a reason.

Academically Adrift

2011-03-14T08:32:00.002-05:00

I've come up for air after posting part three of Life Artificial yesterday evening. Over the last few weeks I've had dozens of browser tabs open to write a post about, but have been too focused on getting particular projects done to write them.

I finished reading most of Academically Adrift last week, meaning I've started but not finished the Appendices. I downloaded the book from Amazon and read it on the iPad's Kindle application. This is very convenient, and the reading experience is fine, but it does have a significant drawback. Despite the ability to bookmark pages and leave yourself notes and highlights, it's not very easy for me to mark up the book in an easily accessible fashion for reference purposes. Normally I'd have sticky notes coming out of the leaves of the volume, and have text circled by scribbled notes in the margin. This by way of apology that I don't have many quotes in this post.

I've never been impressed with the CLA, and the research in Academically Adrift depends heavily on it. I hasten to add that I don't have any problems with the laudable aim of assessing "critical thinking, complex reasoning, and writing." I think the instrument itself can even be useful. It's the exaggerated claims of importance and over-statement of validity that seem unwarranted to me. Because so much of the book relies on CLA scores, let me elaborate.

First, the test is advertised as posing "real world" problems to test subjects. I'm not quite sure what this means, but it sounds good. An example is given on page 22-23:

This seems to be a particularly poor exemplar of a real world problem. On the face of it: would you ask a twenty-something with two years of college to make a decision like this? Undoubtedly the aircraft is an expensive purchase that, more to the point, you'll be trusting your life to. So the way you make the decision is to give a non-expert a limited amount of documentation and ask for a report? I don't think so.

Real life problems are not neatly defined by a few newspaper articles, FAA reports, and such. There's a vast body of knowledge to sort through an analyze if one cares to look for it. It is in the form of web searches, academic and professional articles, interviews with experts, and on and on. I understand that it would make the test much harder to administer to allow open-ended searches, and it would take a long time. Days or weeks. This just highlights how far this exercise is from a real "real life problem."

Real life problems are generally solved by people who know what they are doing. You don't ask the guy at the car wash to take out your gall bladder for a good reason. If you want to know about safety issues with a particular model of airplane, I think it would be really good to talk to a senior mechanic that works on such planes and to pilots that fly them. In fact, putting an expert in the field in charge of your investigation would be the best thing to do, no? This is one of the problems with assessments of "critical thinking" that try to ignore content knowledge. It doesn't work. Content is really important when you want to solve a problem, unless all you care about is sophistry.

Here's a real life problem:

You're the head mechanic who supervises a small fleet of helicopters for a rent-a-bird operation. The chief pilot is planning to fly in tomorrow to test out a new Bell Jet Ranger that you're getting ready for operations. But you and your team haven't been able to get the main rotor balanced, and you suspect that the part that came from Bell to do this is defective. But the pilot has a volatile temper, and is going to be very upset with you if the helicopter isn't ready for him to fly when he gets here. What do you do?

This requires technical knowledge, creativity, and judgment about social dynamics to get through, and there's probably no perfect solution. This was a real world problem I witnessed as a teenager. I saw the mechanics try to balance that rotor blade for hours. They finally gave up, put it on the helicopter anyway, and let the pilot try to take off with it that way! It was so out of balance, that he could only wobble the aircraft around on the ground before giving up. He emerged looking like a ghost, too relieved to be alive to be mad at anyone. Probably not an ideal solution.

A problem with the analysis in the book is the emphasis on differences in scores and the way the information is represented. Others have mentioned (e.g. this article in The Chronicle) that the authors of Academically Adrift put a lot of importance on how many students haven't demonstrated significant learning. The literature I've read on the CLA says that it's not even designed for student-level analysis, but for comparing institutions. But never mind that. The statistical problem that has been pointed out by others is that a hypothesis test that does not find a difference in pre-post tests does not mean that there is no difference. This is from Stats 101. As an analogy, let's say my kid is getting ready for school and can't find her gym clothes. I don't have my glasses on yet and am bumbling around the house before the first cup of coffee has kicked in. But I look here and there for the gym clothes halfheartedly as I walk around. There are two possibilities:

I see the distinctive purple bag. Even without my glasses on, there's a reasonably high probability that I've found the gym clothes.
I look in the room but don't see the bag. I cannot say that the bag isn't in the room, only that I didn't see it. It may, in fact, be just behind the door.

This is the difference between rejecting the null hypothesis (#1) and not rejecting it (#2). Yet the authors lead us to believe through their narrative that not finding a difference means that there is no difference, and go on to claim that not much learning is happening during the first two years of college.

Although the authors wave at the notion of IQ late in the book, they never say anything about the relationship between what they are trying to measure and intelligence (as assessed by standard instruments). Psychologists assure us that IQ is largely static. It's also closely related to verbal skills, which is what the CLA assesses too. If IQ is largely immutable, don't we need to factor it out before making conclusions about improvement? I understand that this is very sensitive issue because you ultimately would have to say that some people are smarter than other people (with all the fine print that goes with that), but to completely ignore the issue seems disingenuous.

We shouldn't necessarily reject all the findings of the book just because the analysis is flawed. The attention on how hard students work and the corresponding rigor of academics is an important discussion. Unfortunately, the way the headline findings of the text overstate the findings on complex learning outcomes only feeds into the current of public discourse advertising that teachers are villains. Despite carefully worded disclaimers that general readers won't decode, the book practically screams that the first two years of college don't produce any meaningful learning. This is a disservice to all the course instructors out there who work hard to teach students calculus, computer programming, world history, composition, and so on.

The fact is that the content of the CLA doesn't correspond well to the course content of a general education curriculum. This could be remedied easily. I am quite sure that a one-semester prep course on items like the ones on the CLA would significantly increase the scores. Would this result in real, meaningful knowledge? Maybe, depending on the depth of the course. A pure test-prep course that teaches how to game the test is probably worthless. But perhaps the current curricula are out of date. Even if that's true, the importance of the CLA and its standardized measurement are not very important because they are too simplified, and rely too much (I assume) on IQ, and not enough on discipline-based knowledge. There are already recommendations (e.g. from the AAC&U) to increase complex problem solving skills, and many institutions are already struggling with how to teach and assess this. But almost by definition, complex problems aren't trivial to solve. If the CLA becomes the measure of these learning outcomes--if we really take analyses like Academically Adrift seriously--they we will shortly have another instance of Campbell's Law at work. See this Wall Street Journal article for a good example. Or this opinion about "impact factors."

Learning outcomes don't matter anyway, in the big picture. That is, test scores and other performance indicators taken during college aren't ultimately the measure of success of a program. It's what happens after graduation that ultimately matters. Are your graduates curious about the world? Do they read newspapers and vote intelligently? Are they engaged in service to their fellow humans? Do they contribute to the overall economy? Depending on the mission of the institution, these goals may vary (but always include "do they donate money?"). At the level of the Education Department, where the strategy should inform the national interests, perhaps the easiest metric of success would be income histories. The federal government has vast data stores on financial aid given. It also has all the IRS data. I asked an official from DoE about mashing these up to answer important questions like "what's the lifetime earnings benefit for attending a private vs public college?". I wasn't encouraged by the answer. No one cares enough or the problem seems too hard. And yet, I'm sure if we gave all the data to the guys at Facebook, they'd have it figured out over the weekend.

UniLeaks

2011-02-25T06:28:00.000-05:00

Perhaps this was inevitable...a WikiLeaks clone for higher education. Seems to be UK-centric so far, but don't wait for the rush--dust off those fiery curriculum committee minutes and start some controversy!

Image from UniLeaks.org.

Individual FACS reports

2011-01-20T07:06:00.000-05:00

I have about two dozen web pages marked to write articles about, but haven't found the time. I'm trying to wrap up part three of my novel (see that blog), and still working on writing up my research. A couple of new things for me:

Our non-cognitive research is going well. Preliminary results based on one semester's grades indicate that a non-cog survey has the potential to add information to the usual enrollment inputs. We'll have much better data in the fall, with a whole year of grades and year-to-year retention data.
We built and launched an early alert system for tracking student who have academic difficulty early in the semester. It's in the testing phase right now.
We're developing a new web site, and one of the most important, and (to all appearances) ignored sections is the description of academic programs. We're spending a large effort there to develop superb pages that will sell programs to students. Stay tuned. Our Google Analytics shows that these are the most frequented pages by outside visitors. Not surprising, since academics is the product a university sells.
I've done a lot of development on reporting FACS scores. There is now a self-serve site to generate reports by term, program, or class. This week I added the ability to drill down to the individual student level, for use by advisors. Here's an example:

The red lines and bars show where this second year student is performing relative to other students, based on faculty assessments from the prior semester. At at glance you can see that this student is performing well below par, and according to professors is also not putting forth effort at the same level as peers. The sample sizes are necessarily small (although they will increase as the assessment becomes institutionalized). Note, however, that all three raters agreed that creative thinking was demonstrated at the pre-college "developmental" level.

Off to the Southern Education Foundation today to attend an assessment meeting in San Antonio.

SACS gets a Back Channel

2010-12-07T14:51:00.001-05:00

Last year's tweets from the SACS/COC December meeting were almost non-existent. That has been remedied this year thanks to contributions from several indefatigable tweeters. You can see them at http://twitter.com/search?q=%23sacs.

I have notes to write up and post here, but will have to wait until the weekend. The big news for me was that Coker's Fifth Year report sailed through with no recommendations. Congrats to Kaye and Pat and Daniel and everyone I don't know about who made that happen.

Here's a gem I found in the resource room, while picking over the 3.3.1 sections (paraphrased):

Objective: 60% of the students taking the test will be in the 100th percentile

I know what they meant, but I got a good laugh out of it. It puts Lake Wobegon to shame!

SACS 2010

2010-12-03T08:42:00.000-05:00

I'm off to Louisville tomorrow morning for the December SACS meeting. Last year's backchannel was about zilch, and it looks like #SACS means something in another context, judging from the Twitter search for #SACS. I'll tweet to #SACS anyway, from my phone or iPad. Send me an email if you want to have a coffee and compare notes.

Collatz Ecologies

2010-11-21T09:03:00.009-05:00

I mentioned the Collatz Conjecture in "On Design" as an example of the qualitative difference between simulation and inverse problem solving. In this article I want to use it for another purpose: to show how structure emerges out of iteration. Specifically, I want to create a very simple model of Darwinian evolution and demonstrate with simulations and mathematical proof that patterns emerge naturally. In a later post I will talk more about what is significant about this, but here's the preview: when stable patterns emerge in some iterated system, it's possible to build new systems on top of the old ones. Moreover, these new systems can be seen as independent of the old ones. The full discussion on that can wait. This is the fun part.

Iteration at the heart of the conjecture is a single branching formula that works on (usually positive) integers:

$N_{new}:=\left\{ {N_{old}/2\mbox{ if even}\atop (3N_{old}+1)/2\mbox{ if odd}}\right.$

I have used the more compact form of the formula that goes ahead and divides by two in the odd case, since 3N+1 will always give an even number when N is odd. As an example, starting with 8, we get the sequence 8 ->; 4 ->; 2 ->;1, since all but the last is even (8 is a power of 2). A more interesting example is 3 -> 5 -> 8 -> 4 -> 2 -> 1. The unproven conjecture is that any starting number eventually ends up at one. If the conjecture is not true, then for some starting N, either the sequence grows without bound, or it forms a repeating loop. The Wikipedia page has a nice summary of what is known.

Artificial Life is the use of computer simulations to understand biological-like behavior. Conway's Game of Life is one of the best known. For more on the general topic see the Wiki on ALife. For our purposes here, I want to use the Collatz iterator to construct a population that is subject to Darwinian evolution. For that we need these components:

An initial state from which to begin the simulation. In practice this will be an individual species, identified with an odd integer 3,5,7...
A simulation of change over time. This will be the Collatz iterator acting on the numerical species.
A fitness function. If a species "evolves to" 1 via the iteration, it is eliminated from the population. The Collatz Conjecture in this context is that all species eventually go extinct.
Reproduction and variation. For any odd numbered species, when it is transformed by the iterator, an imperfect copy is also created. The copy is the species N-2, where N is the new number of the original after iteration. For example, when 11 -> (3*11+1)/2 = 17, the species 17 - 2 = 15 is also added to the population.
A carrying capacity C. When the population comprises C species already, reproduction is paused until a vacancy opens up through some species going extinct.

For my purposes I don't care how many copies of an individual species exist, since they will all share the same deterministic fate. It also helps keep the computation cost down. So if 17 already exists in the population, and another 17 comes along, I don't create two copies of 17.

Initial Results. Alife sims are just plain fun to play with. I include my code at the bottom of this post in case you want to try it yourself. For what follows, the carrying capacity is set to 100. Here are the first seven starting species charted over 20 generations.

We see exponential growth capped at C except for N=3 and N=5 (they overlap), which form the line at the bottom. What's going on there?

A little investigation shows that the population {2,3,4,5,6,8} forms a closed "ecosystem" that is generated from starting with 3 or 5 (or 6 if we include evens). Moreover, this system follows a stable pattern that will go on forever, showing that there exists Collatz ecologies that never go extinct. In fact, if any other N-ecology ever generates 3 or 5, it will create this pattern as well, and so will also live forever. This gives us:

Conjecture 1: All odd N-ecologies with N > 1 survive forever for sufficiently large C

I hedged a bit there, but I don't think C needs to be very large at all. Another question that we can frame as an affirmative in the form of a conjecture is:

Conjecture 2: There is only one bounded ecosystem for odd N > 1.

Here bounded means that the largest species encountered doesn't grow beyond a ceiling. It looks like the other graphs are bounded too, because they run into C, but this is only true for the number of species at any one time. The individual species identifications continue to grow. We'll look at that next.

The graph below shows the population of N=9 after 16 generations. It has been capped by C, and so the number of species will stay about 100, but which ones those are will continue to change. The individual species numbers are shown by the heights of the bars. The x-axis is unimportant--it just arranges the species from smallest to largest. Each bar is a distinct species. The four highest ones are {1893,1895,1896,1898}.

Obviously there's a lot of structure here. If we plot the populations of all 16 generations together, we see a consistent pattern over time. Each generation has its own color:

The quartet we saw as the top plateau for the population seems to be an enduring structure. Moreover, this pattern is visible on the other populations created from N = 9, 13, and 25, for example:

The graph shows the 16th generation of each, when N=9 and 25 are just hitting the carrying capacity. The quartet and plateau patterns are obvious in all three.

A note about the above graphs. I was constantly fiddling with the program, running different numbers of generations and messing around with how reproduction gets handled. The final version of the program kills any number that attempts to grow beyond a billion, in order to prevent integer overflows. So if you run the program at the bottom and compare it to these you'll see minor differences at the upper end. The graphs above were generated without that constraint, but no overflows actually happened in the data represented.

The Quartets are not hard to explain. Whenever N begets another two odd numbers by the action of the iterator and mutator/reproducer, a quartet will be formed. And because the "two odds in a row" means that the starting number is multiplied by essentially (3/2)(3/2) = 2.25, it and its three new kinfolk will leapfrog over the competition. Suppose for some odd N we get another odd number (plus new variation). This will happen when $(3N+1)/2 = 2k+1$ for some $k=1,2,...$. Solving for $N$ gives $N=(4k+1)/3$. This fraction is only an integer when $k$ is of the form $k=3m+2, m=0,1,...$ Plugging all this back in, we find that $N=4m+3, m=0,1,...$, so $N=3,7,11,15,19,...$ all have the property that they are odd numbers that also yield new odd numbers under iteration. When this happens, two pairs of new species are generated. In terms of $m$ they are $9m+8, 9m+6, 9m+5, 9m+3$. This gives exactly the pattern we observe in the simulations.

Plotting the population of N=7 over 100 generations against a log scale shows how the shape of the ecology changes over time. The lines of dots sloping up represent the constant log(3/2) growth of the odds. The dots seem to pretty much cover all the integers (not all at once, but eventually). That gives us a third question:

Conjecture 3: Except for N = 1, 3, or 5, the ecology for odd N eventually reaches any given positive integer.

As supporting evidence (certainly not proof!), if we look at N=7,9,...99 and locate the missing integers after 1000 generations, we see that every one has swept up the integers through 250.

The horizontal axis is N (odd numbers from 7 to 99), and the vertical axis locates the missing value: each dot shows an integer that was missed by the N-ecology after 1000 generations. The distribution of these misses, inclusive of all the populations above shows some structure:

Is there some crazy number out there that isn't reachable by a given N-ecology? Does the carrying capacity change this one way or the other, over straight exponential growth?

Behavior at the point of capacity shows another emergent pattern. We see this in the number of species in an N-ecology after it faces the limitation of C, and new species are created much more slowly. We would expect that larger Ns have lower extinction rates once population size C is reached. This is because in order for a species to die, it has to land on a power of two, after which it zips to 1 and goes extinct. Powers of two are distributed more sparsely (on a linear scale) as N increases. Of course, there is only one "2" in the population at any given time, so the actual extinction rate due to the 2 -> 1 evolution is either zero or one each generation. The graph below shows 1000 generations, averaging the odd Ns from 7 to 99, giving the demand for new population, the actual number created, and the extinction rate. The first of these would be expected to be half the population, since the odds get reproduced, and indeed we see the line bounces around 50. The extinction events come from 2->1 and from the overflow limit, when a number tries to grow beyond a billion.

The red line is more interesting. It shows the actual number of slots that opened up for new population. If we subtract out the ones we can account for due to 2->1 or overflow, we get a constant average of about 5.4. This must be the rate at which two species turn into one species due to the action of the iterator. In other words

$(3N_1 + 1)/2 = N_2 / 2$ or $3N_1 + 1 = N_2$.

How likely is it that $N_2 \mod 3 = 1$? Should be one third, but we have to remember that $N_1$ is odd and $N_2$ is even, so for positive integers $n1$ and $n2$ we have $3(2n_1+1)+1 = 2n_2$ which gives us $3n_1 + 2 = n_2$. That doesn't change the odds. So if all integers were in the population at the same time, we'd expect a third of them to join up after iteration, decreasing the new generation's size by one each. Given that we have to account for an actual average loss of 5.4, this seems to imply that at any given time, the ecology is 16% saturated (5.4%/33%). At 100% saturation, all integers are present, and one third will disappear due to collisions. If this analysis is right, is this saturation level really constant over time? It seems unlikely.

Reference. I discovered by Googling around that Hiroki Sayama had a similar idea to use the Collatz sequence to study artificial life. His version is completely different from mine, and you can find it here.

The code. [download perl script from snipplr]

Edit: I made a couple of minor edits to two sentences after posting.