Marking up Portfolios

I blagged (here) a while back about a potential transformation in the way we think about eportfolios. The key idea is that the actual portfolios elements reside somewhere (anywhere) on the web. We're long past the point where a single program needs to host and present the portfolio. The multitude of ways to freely create content on the web far exceeds any one portfolio program's capabilities. It's true that these works potentially wouldn't be as tightly linked to a learning management system, with handy rubrics and such already linked up. But the trade-off is potentially worth it.

One essential element is the need to be able to comment on work. Students may peer-review each other's work, and certainly the teacher would like to be able to do the same. I was brainstorming this problem this week because, as it turns out, I have to figure out how to create eportfolios for the fall semester. Our old software is going poof.

By serendipity, I saw a blog at blogged.com called Academic Productivity. Lo and behold, there was an article about a "web highligher and sticky notes" application--exactly the sort of thing I'd been brainstorming about. The service is seen as a competitor for delicious.com, which I use religiously to maintain my bookmarks. It's called diigo.com, and it allows you to bookmark a page, add tags, organize into folders, mark as read or unread, and--best of all--highlight portions of text within a web page and add comments. These can be public, private, or restricted to within a particular group. The image below, snipped from a New York Times article, illustrates such a comment. If you browse to that page with your Diigo account activated, you should be able to see the note I left, since I made it public. (Note: the "Readers' Comments" box on the left has nothing to do with Diigo--it's the usual bloggish comments allowed in NYT.)

Best of all, the creators of the service have a special provision for educators, so that you can easily organize classes into groups. I haven't tried this out yet, but it sounds absolutely perfect.

Diigo comes with a Firefox plug-in that makes it easy to do your markups. The menu bar is shown below.


An "open" portfolio of the type I've described could have a home base in a public blog, which is a nice way to host content, with built-in commenting and RSS feeds. Plus, many kinds of media can be directly embedded, like graphics, sound, video, and anything that runs on Flash or Silverlight.

Clearly, even with markups, the loss of a built-in gradebook may be too much to suffer for some instructors. Perhaps Gary Brown's harvesting gradebook will come to the rescue there. In the meantime, I think it's worth trying out the markups as a pedagogical tool.

The Future of Portfolios

A few weeks back in the public library I picked up a novel by Verner Vinge called A Fire Upon the Deep, which may be the best sci-fi novel I've ever read. Soon after, I found myself reading his novel Rainbows End, about a near future where technology has become truly ubiquitous--one wears his or her computing environment, which overlays the 'real world' with graphics displayed on contact lenses to create new realities. Part of the story takes place in a classroom, and it's fascinating to see how Vinge (who is a mathematician and computer scientist) imagines the new technologies impacting pedagogy. This plays out as students work in collaboration to flex their creative muscles using networking, computing, physical "black box component" machines, and software tools. Except for the sophistication of the wearable computers, all of this is possible in some form today.

Portfolios can be accumulators for this sort of work. There are commercial systems available as well as open source ones like OSP (open source portfolio, literally enough), which integrates with the online learning system Sakai. The OSP home page advertises that portfolio owners have access to:

1. tools to collect items that best represent their accomplishments, their learning, or their work;
2. tools to reflect upon these items and their connections;
3. tools to design a portfolio that showcases the best selections of this work;
4. and tools to publish the portfolio to designated audiences.

The use of eportfolios has been championed by Trudy Banta at IUPUI, which has developed its own custom one in cooperation with OSP. I saw an early version at the IUPUI Assessment Institute some years ago. At that time there was a matrix into which students plugged artifacts that demonstrated their accomplishments in the general education learning outcomes set by the university.

Advantages of eportfolios are obvious. Years ago I was on a committee to review the old-fashioned kind--manila folders bursting with papers that supposedly documented student accomplishments in language and numeracy. The stakes were high--if a student failed to pass this review, his or her graduation could be held up. In practice, the reliability and validity of this review were very doubtful, and the record-keeping terribly time-consuming and imperfect. There was little transparency throughout the process, and it ultimately was abandoned. A few years later, as chair of the Institutional Effectiveness Committee, the accreditation process presented me with the problem of assessing our Quality Enhancement Plan, which addressed student writing. This time, we built an eportfolio. I used the ASAP approach (as simple as possible), so as to minimize support problems and maximize usage. It takes very little in the way of barriers to keep users away. In this, we were very successful--we gathered oodles of writing samples without having to try hard at all. I called the thing {iceBox} because that's what my grandmother always called her refrigerator, and it tickled me to give new meaning to the term.

Assessment was harder than collection. We tried what seemed like the obvious approach, creating a rubric for writing, assembling a sample of portfolios (each was a selection of three writing samples, so it was a "portfolio" only in a general sense), and assembling a committee to rate them. After spending a lot of effort, we concluded that this approach was a good example of an idea that seems obviously correct until you actually try it. It's much better to tie assessment very closely with coursework--that is, leave it centered in the classroom where it can have an immediate impact and has authenticity.

There are two lessons from my experience. First, software that does one thing simply and well is preferred over general tools that try to do everything. In Vinge's novel, this is analogous to the black-box components that fit together like Legos. For two excellent examples of this kind of thing, see my post on collaborative software. There are many, many components on the web that can be used to assemble portfolio-type materials of all kinds, from music to language to geography to graphics to publishing and beyond. Moreover, they are evolving all the time. A hyperlink can take you to an audio file of a speech, a movie, or a location in Second Life, to name three of a virtually unlimited number of species of potential portfolio artifacts.

The early cars looked like carriages, because that was an obvious transition from horse to no horse. Similarly, our first generation of eportfolios tries to be a big electronic manila folder. This too will evolve, I'm sure. Because what's called for is not the design and publish components (3 and 4 in the OSP list), but the means to link together artifacts that can live anywhere in cyberspace. A student's portfolio per se is just the connections between presentations, and could consist of a single hyperlink to a blog as entry point.

The second lesson I learned relates to the second requirement of a portfolio, if it is to be used for educational purposes. Assessment documentation has to get done in a way that makes sense; there are a lot of ways to do it wrong. Assessments need to be pertinent to ongoing educational processes in real-time, need to be authentic (related to class work), and the burden of creating these assessments needs to be minimal if they are to get done at all. That's a tall order.

Yesterday I mused about implicit rules in the academy. The Center for Teaching, Learning, & Technology (TLT) at Washington State University has produced a very interesting spectrum of assessment focus. You can download it here as a pdf, and I've reproduced the first line below as a sample.

Working through this sheet is like taking an inventory of implicit rules and assumptions concerning assessment practices. The third item "Expert consensus from the community of practice validates the assessment instrument," sounds like the primary basis for the general education assessment we had the most success with.

This idea is related to portfolio assessment by WSU's TLT Center through the notion of a "Harvesting Gradebook," which I mentioned briefly yesterday as a kind of disruptive technology--ideas that challenge the implicit rules of how grades and grading work in this case. On their blog, Nils Peterson writes that

As originally articulated by Gary [Brown], the gradebook “harvested” student work, storing copies of the work within itself where it was assessed.

On further discussion, the concept became inverted, what was “harvested” were assessments, from work that remained in-situ.

The key here is that the components of the portfolio can live anywhere on the web. The piece that brings them all together is the assessment. This fits perfectly with the evolution from "horse-drawn" portfolios to an ASAP component model. Do the assessment part really well, in other words, and don't try to recreate a music-editing program or web-based word processor inside your eportfolio software. A sample assignment contains notes about how to write a blog. My understanding is that it doesn't specify where or what software to use, but what the content should look like. This separates the means of creation from the assessment of the product, which simplifies and enriches the portfolio. I recommend taking a look at the sample survey here, which permits feedback from various types of audience with rubrics and comment boxes.

There is a lot of material about this project on the WSU/TLT blog, and I look forward to learning more about it. It's interesting that for all the verbiage in the assessment community about grades being lousy assessments, this is the first time I've actually seen a proposal that would radically transform the practice of grading. I hope the registrar has a defibrillator in the office.

Rules, Damned Rules, and Policy

My daughter has this thing about wanting pets. Because we aren't well situated to play host family to them (the gerbils were a disaster on our first attempt), I try various ruses to change the subject. When she was younger I latched onto the idea of using a laser pointer as a dog substitute (named Spot, inevitably, although Red was a contender). We took it for a lot of walks, always after dark, and watched proudly as it visited all the trees on the block.

Yesterday it was plants. Pet plants need minimal care, I figure. So we went to Home Despot to look. I particularly wanted rosemary, to replace the nice bush we had at the old house. We bought some 'pet' vegetables, but there was no rosemary, so we walked down to Kmort to look. Their outdoor section looked much like a concentration camp, and I'm sure if I spoke plant-ese I'd have been moved to tears by their pleas. My daughter wanted to ask the salespeople if she was allowed to water the plants, but there was no one to ask.

None of this has anything to do with rules, in case you're wondering. That connection came next, when I noticed a gaming store next to Kmort. In graduate school I co-authored a board game of sorts with a friend, so I wanted to peek in and wallow in the ambiance for a bit. My almost-teen daughter was properly horrified, which added to the attraction.

The place was filled with gamers at tables piled with miniatures from a dizzy variety of genres. There were sci-fi tableaux, with someone asking about how far pulse rifles could shoot, fantasy sorts of things I couldn't recognize, and historical battles with box-like formations of hand-painted troops. Around the walls were stocked the complicated rules books I remembered.

You've probably figured out by this point that I played a lot of geeky games as a teen, with rule books that resemble the 1040 tax instruction booklet. The rules got increasingly more complex as time went on, until half the games seemed to consist of searching for the right sub-section with the table on the chance of successfully napkin-folding or whatever. For me, there reached a point where I didn't find it enjoyable anymore. That's when I started coding up rules in Applesoft, to use the computer to keep track of the complicated bits, and truly descended into full-fledged geekdom, from which I never really emerged.

This all by way of introduction to "rules and the academy". (I hope this one worked better than sock muffins did.) There are places where rules are absolutely essential. You can identify them by the lack of thinking that's required by the tasked staffer. Storing backup tapes somewhere safe, following procedure with regard to transcripts, keeping track of financial accounts properly, and so on, are good examples. The lower the complexity, the more suitable for rules-making a process is. At the other end of the spectrum lies general responsibilities like "being president," which is too fuzzy to be described in a President's Operating Manual, or something.

If you like rules, stop by human resources. Here, as in other areas like IT, rules can be used to simply block things that staff don't want to do and generally accumulate power and influence. I read a (possibly apocryphal, but plausible) account of a man interviewing for a job, who was asked by the HR interviewer for the phone number of his previous employer, so as to verify the information on his resume. I can't find the original now, but it went something like this:

"I need to call your previous employer, Mr. Snark."
"Well, I was self-employed, so that would be me."
"Fine. What's the phone number?"
Mr. Snark, bemused, gives the number and watches the numbers being dialed. He pulls out his cell phone and answers on the first ring.
"Mr. Snark?"
"Yes, that's me."
"I need to verify some information about a previous employee."

In this Kafka-esque drama, the interview plays out in full, after which Mr. Snark is given the explanation that rules, after all, must be followed.

It's debated whether or not evolution produces more complexity in living things. I think it's probably true that complexity is more valuable in some situations than others. Like a string in a drawer, systems seem to bow to entropy almost immediately and become more complicated without much effort. Straightening them out is hard. As Machiavelli put it:

It must be considered that there is nothing more difficult to carry out nor more doubtful of success nor more dangerous to handle than to initiate a new order of things; for the reformer has enemies in all those who profit by the old order, and only lukewarm defenders in all those who would profit by the new order; this lukewarmness arising partly from the incredulity of mankind who does not truly believe in anything new until they actually have experience of it.

Whereas complexity naturally emerges in the form of additional rules, in order to properly simplify the resultant mess, the reformer has to overcome the natural resistance of those who benefit from the complications. Think of the US tax code.

So, like biological bodies renewing themselves through reproduction after entropy has corrupted them, it's a healthy process to change administrations and shake things up once in a while. It is perhaps the case that the most insidious rules are not affected by this, however. They may be invisible.

At least too-complex rules can be seen. There are many quasi-rules that are merely implied. Dress codes often are, and are enforced through social conventions, although explicit ones aren't uncommon. More dangerous, I think, to the mission of the academy are the implied rules that pertain to learning. Here are a few. You can add to the list.

• Teaching only takes place in formal sessions
  
• Learning is not as important as ratings like grades
• Education proceeds by check marks on a sheet
• Education is a service one pays for, just like having your car washed
• Learning experiences can be made uniform, like an assembly line
• Student collaboration, unless explicitly allowed, is cheating

The bureaucracy of higher education is necessary to organizing the massive endeavor, no doubt. But leaving unexamined the implications of these practices blinds us to some pernicious effects. Do we really have a right to complain if students see courses as milestones to be passed on a linear journey--points of momentary interest that can be forgotten? Doesn't the very structure of the process from advising through transcripts encourage that point of view?

In order to gain perspective, a complete rethink is in order. I was impressed recently with an idea by Gary Brown at Washington State University about a way to redefine the hoary old idea of a gradebook. No, I don't mean moving it to Excel. You can read more about this "harvesting gradebook" idea on the blog Center for Teaching, Learning & Technology. The authors of this project seem to be questioning what exactly grading is--a review of the associated implicit rules, as it were. It will be interesting to see where it leads. This is an example of the creative disruption that is called for in order to reach more than a superficial review of the stew of formal and informal complexity that comprise the academy.

Update: In the pursuit of sensible database policies this morning, I found myself wandering through the wilds of the FERPA rules, and discovered this gem [pdf].

Under FERPA a school may not disclose a student’s grades to another student without the prior written consent of the parent or eligible student. “Peer-grading” is a common educational practice in which teachers require students to exchange homework assignments, tests, and other papers, grade one another’s work, and then either call out the grade or turn in the work to the teacher for recordation. Even though peer-grading results in students finding out each other’s grades, the U.S. Supreme Court in 2002 issued a narrow holding in Owasso that this practice does not violate FERPA because grades on students’ papers are not “maintained” under the definition of “education records” and, therefore, would not be covered under FERPA at least until the teacher has collected and recorded them in the teacher’s grade book, a decision consistent with the Department’s longstanding position on peer-grading. The Court rejected assertions that students were “parties acting for” an institution when they scored each other’s work and that the student papers were, at that stage, “maintained” within the meaning of FERPA. Among other considerations, the Court expressed doubt that Congress intended to intervene in such a drastic fashion with traditional State functions or that the “federal power would exercise minute control over specific teaching methods and instructional dynamics in classrooms throughout the country.” The final regulations create a new exception to the definition of education records” that excludes grades on peer-graded papers before they are collected and recorded by a teacher. This change clarifies that peer-grading does not violate FERPA.

Exceptions are the hallmark of complexity. If trivial exceptions can't be dealt with by simple common-sense methods, you're stuck with arguing trivialities at the highest, most formal level of adjudication. This is a recipe for entropy-induced "heat death," as it's called when one speaks of the end of time.

Practical Matters

I'll take a break from the heavy philosophy of assessment stuff, and write about two little tricks that make my life a bit easier in the hopes someone else may benefit as well.

The first applies to anyone who still grades the old-fashioned way and ends up subtracting from 100% a lot. It's generally a pain to subtract from 100 because of the 'borrowing' algorithm that we're taught in this country. In Germany and other countries they use a 9's complement approach that I'll show you here. I'll apply it specifically to the 100 problem, but it works for any situation. It's embarassingly simple:

Because 100 = 99 + 1, subtracting from 100 is the same as subtracting from 99 and then adding one.

This makes a big difference because you never have to 'borrow' from 99. Moreover, the subtraction is easy-peasy because each subtraction is symetrical: 9-4 = 5 and so 9-5=4. This makes it all very painless.

Example: If you add up a student's points off and get 27, you can 'flip' 27 around 99, and get 72. Add one and you're done: 73% correct.

Example: 100 - 59 would be 99 - 59 = 40, 40 + 1 = 41.

I know it's not really that difficult to subtract from 100 to begin with, but this makes the process almost fun, and after grading dozens of paper, anything that makes my head hurt a little less is welcome.

The second practical matter is a simple way I've evolved to for organizing projects. I have a lot of projects going on simultaneously in all kinds of domains. There are lots of sophisticated project managers that you can find online, and I've tried out many of them. I finally realized, however, that the amount of information I need to document about most projects is pretty minimal. The most important bit of information is simply recording that the project exists. If management is through some complex system, there will always be things that fall off the table because someone didn't take the time to enter the information. Secondly, a simple green/yellow/red status flag is sufficient for most things. (Use blue instead of green if you have colorblind users.) Finally, a hyperlink to a file or other resource to document details of the project is desirable. I find everything I need with the combination of a mindmeister mind map, and etherpad documents. Both are free for the basics, which is enough for me currently. A section of my project tree looks like this:
The arrows on nodes are hyperlinks to resources. You can make the whole thing editable by others. Etherpad docs, by design, are very open and perfect for collaboration. I don't feel like I'm creating more work for myself using this combination of tools--it's very natural. For bigger projects, a more robust tool will probably be useful, but for keeping track of a large set of small and medium-sized projects, this is perfect. I could wish for calendar features and summary reports and such, but within my current scale of operations, this hasn't become critical.