Waypoint Outcomes

There is a new web-based rubrics + database learning outcomes software called Waypoint Outcomes from a company called Subjective Metrics. At least, it's news to me. It's nice to see "subjective"" used in an assessment context. I think people tend to think subjective = unreliable instead of subjective = complex, and avoid talking about it for fear of execration. I've had psychologists talk to me after my conference presentations on FACS, amazed that anyone would admit to using subjective assessment, as if they expected a mob with torches and pitchforks to appear any time.

There is no demo online, but there are a couple of videos. The CEO shows off the product and reminds us why we need to do assessment (we in higher ed hardly need that!). He uses the word "subjective" often, which warmed my heart, but the content of the videos were a bit hit and miss for what I was looking for. So this isn't really a review, just some impressions from the marketing materials.

Waypoint Outcomes integrates with Moodle and other learning management systems, with the intent of providing a plug-in for assessment rather than provide all the functionality of a LMS--a very sensible approach. Assignments get tagged with outcomes, each with a rubric that you can assign and comment on with observation, advice, and reference. I could not tell from the video how these outcomes are organized--are they in hierarchies by subject matter, or just an alphabetical list? Can you tag learning outcomes to organize arbitrary groups? For example, maybe your Quality Enhancement Plan uses two from general education and two new ones. Can you group them easily? Not sure.

Ratings are assigned directly in the rubric form online, with fields to give students comments on each outcome. You can assign weights to each outcome, so that an overall score is generated (i.e. create customized statistical goo). It has a robust database on the back end, allowing filtering, aggregation, and reporting. One nice feature is the ability to randomly sample student work for after-the-fact assessment, possibly by someone other than the course instructor. Of course, in order for that to be successful, the assignment has to be available too. I don't know if there's a strong link in the system to the assignment, so that it could be referenced as a hyperlink, for example. I can imagine wanting to generate a list of all assignments in a particular discipline that used a certain learning outcome.

My overall impression is that this is a solid design for keeping track of rubrics and ratings. User-interface design is critical, however, and I don't have any sense how easy it is to use to construct new rubrics, attach them to assignments, and pull them up later.

The central problem with the whole method of on-the-spot rubric ratings is the time involved on the instructor's part. This system would work great if every instructor was assigned a rubric rater to do all the assessments. You wouldn't really want to the instructor rating his/her own work anyway would you? If the scores are to be used to judge how much was learned and contributed to performance ratings for an instructor, you wouldn't want to allow self-ratings of performance, which is in effect what these are. Separating teaching from assessment tends to devolve to standardized testing, which can't be a complete solution. Having instructors rate each others' students wouldn't work either--it would be too political.

An approach I like, which would be doable with the Waypoint software, is to tag assignments with the learning outcomes--but not too many of them--and use a small committee of teachers of that subject to periodically scan the student work in the lens of the rubrics and outcomes, and reach some consensus about how well we're doing. That is, rate the work as a team, after the fact, and only for a sample. This creates a nice dialogue that can lead to solving complex issues. The idea of rating every student on every outcome all the time is noble, but you may need to hire more people to pull it off.

Unplanned Obsolescence

On a morning bike ride here in Düren, I returned to the abandoned military post I'd discovered last week, this time with the camera along. The entrance to the installation is shown below, with what I presume to be a guard house.
According to our host Hans, this was originally a WWII era site that was then taken over by the Belgians and not truly abandoned until 10-15 years ago. He has a lot of stories about the Belgians, their two official languages, and the state of their equipment. Whatever the case then, the buildings are falling apart now, aided by local vandals. It's silent and spooky to walk around in these modern ruins.
Part of this large installation has been turned into a nature preserve, and a large chuck is now occupied by commercial interests. AVIS has a car park there, on the other side of the overgrown road I biked in on. It's fenced off, but the cars surround what look like barracks. I stuck the lens between the chain links and snapped the photo below. There are rows of identical buildings, in much better shape than the abandoned ones.
I'm thinking back, but I don't think I've ever visited an abandoned university. They surely exist, and one can easily find photos of them courtesy of your favorite search engine. Here are some links, none of which I've verified, so caveat emptor:

• Some lovely photos on Flickr, but location and name of university unspecified.
• One in Greece
• Institute de Chemie et Metallurgi in Belgium, according to the text and pictures.
• University of Brazil expansion, according to the text.
• Flickr photos of Morristown College, then Knoxville College, now abandoned campus. According to Wikipedia, the college closed in 1994 and has several historical buildings at the site.

The bricks and mortar of established campuses represent substantial costs to operate and maintain, and unless the summer has a good enrollment or other forms of income, 25% of the year many buildings aren't earning their keep. We might compare the established way of delivering education to the now-established ways of running those same institutions using big expensive software packages like Datatel or SCT Banner or Jenzabar products. The installation time is about the same as a building project, and costs of maintenance are probably comparable.

The bricks and mortar approach has been under pressure to change from online-only institutions. Similarly, the big-iron approach to college administrative software is under pressure from open source competition. I'd heard about the Kuali project before reading this article in InsideHigherEd today. The big news (to me at least) is that major universities are converting to the open source software, with Colorado State University and San Joaquin Delta College being live on the system according to the article, and The University of Arizona and Michigan State University soon to follow.

I've often made the point in these blag posts that sometimes the only reasonable approach to a hard problem is an evolutionary one. In fact, we might call this the ultimate creative endeavor. A combination of either random or inspired input, combined with a rational weeding-out process can over time differentiate ideas that work from ones that don't. I argued more technically in "Survival Strategies" that in the long run this is probably the only route to long term survival in a changing environment. In terms of the topic at hand, that means that open source software is better positioned to survive than commercial software because of the evolutionary approach of the first, compared to what I called a monolithic entity in the paper--a one path to success model, that commercial software companies have to follow. It's not practical for Microsoft to create 100 different versions of Windows to see which one is most popular. Open source software, by contrast, thrives with that model. Of course most open source projects fail--that's really the whole point--that's how evolution works, by leaving a lot of bad ideas in the software cemetary.

But I don't want to write about software; the point is a broader one. The very same process is now beginning to play out in the general realm of higher education. Competition for bricks and mortar colleges from online universities is only the beginning. The means to deliver education conveniently in electronic forms creates a whole new environment that is different from the one that universities and colleges have grown up in. Open source software to run a college is small potatoes. Imagine the whole educational product from start to finish as evolutionary instead of monolithical, plodding beasts.

To embark on this excursion, we'll need to consider what the obstacles are currently to success for a university. This is first and foremost: money. Mammon is the oxygen of higher education, and it's the pursuit of it that creates a natural environment for larger and larger beasts. As with any economy of scale problem, there are efficiencies that come from size. The barriers to even functioning as a basic higher education provider are substantial. There are state and federal entities who have a hand in because of the money they dump into financial aid. There are accreditors of all kinds, the costs of finding and processing potential students, and the large bureaucracy that is needed to support the whole edifice. Doing this on a small scale can be done, but there's no room for error.

That is the Jurassic model of large lumbering beasts living in an oxygen-rich enviroment, but what's the open-sourcy alternative? What is it that prospective students really get out of higher education? If the answer is allegiance to a football team, or a brand name diploma, or making great contacts for their future careers, that's one thing. If it's actually learning to think and communicate, that's another. The two are not mutually exclusive, of course, but the point is that the teaching part can be done in an entirely different way that doesn't involve vast amounts of money. It already is. You can download free courseware from MIT and other places, for example. But let's suppose that education requires interaction with teachers and fellow students. This can clearly be done online, as with University of Phoenix Online, for example. But their model serves just to rake in as much financial aid and loan money as possible--essentially porting to the old bricks and mortar model to the digital age without changing much, in the same sort of way that the first cars resembled the horse-drawn carriages they were to replace.

Instead of a big online university, with all the same problems of accreditation and such, imagine something more like eBay for education. In this model, a match-making service pairs up students and teachers online. Probably the course slots are actually bid for in an auction. A ratings system tracks feedback from customers (students), and the prices students would be willing to pay for a particular instructor would serve as another kind of rating. The eBay-like service would provide a minimal administrative structure in return for a percentage. This could, for example, include a minimal standardization of courses, or at least suggestions, for different areas of study. Students ideally would continue to take courses as a life-long process, something the dinosaria can only preach but not really deliver. Why? Because of the money. It would be far, far cheaper to deliver courses in this way. As a first order calculation, imagine that the service provider takes 10% for overhead. This itself would be subject to competition, which would drive the price to some reasonable level, since there's no reason only one such service could exist.

So I as a math professor decide to offer a Digital Logic class, with an enrollment of 20. I'll teach three sections simultaneously, which may be less than a face to face load, but online is more time consuming, so let's call it even. How much do I need to get in tuition to have a reasonable lifestyle? Two semesters worth is a maximum of 120 nominal tuitions T. If I charged $1000 each, that's $120K less the administrative cost. Even at $500 per student, I could afford to live nicely in lots of places, and still have three months a year free. The more specialized the course, the more in demand the professor, the more comfortably he or she could make a living.

Meanwhile, the administrative eBay-like service doesn't have to worry about processing student loans, applications, or accreditation. It doesn't have to build and maintain a lot of buildings, nor hire a football coach. It can run at full speed twelve months per year. And of course, it can do lots of this with open-source software, using Moodle as a course-delivery platform, for example.

I searched for such a service, but haven't found anything like it. If you read this and know of one, please let me know.

The downside is an erosion of standardization, the obsolescence of the 'major' and many other conventions of higher education. That's the nature of evolutionary change, and it's ultimately the marketplace of careers and ideas that decides if it works. Because of the drastically reduced cost of such a model, it's a good bet that something similar already exists or will soon. Whether or not it, or something similar eventually takes root is an open question. But I think it's a good bet that in the long run a lot of old buildings with neo-classical fascades will have weeds growing around them.