Thursday, November 15, 2007

An Academic Short Story

Nightmare in Mauve

(With apologies to John D. McDonald)

*** 9:15 am Monday ***


Tatiana stared across the brown conference table at the object of her ire. Exactly three hairs graced the dome of Rocke College’s Assistant Dean for Efficiency, Harry Hapollonian. She thought he resembled the plaster cast of Napoleon’s deathmask, sunken cheeks and all. He rumbled rather than spoke, sounding like James Earl Jones with strep throat.

“It won’t do to have all these distractions from the main point. We have to stay on target here. The simple fact is that we have only around 400 students in the day program. Yet in a year, we teach more than 400 individual classes in the day program, eye, ee, more than one class per student. That simply can’t continue.”

Tatiana half rose from her chair.

“You’re the one missing the point, Harry! How can we call ourselves a college, let alone a liberal arts college, without an Existential Studies program?” None of the other department chairs would meet her eye. Sheep, she thought, but she knew it wasn’t fair.

The scene was far from bucolic. Eight of her colleagues, campus leaders from all disciplines, surrounded the table—itself a symbol of Rocke’s advertised ‘brown table’ educational approach. The soothing darkness of the faux oak was supposed to induce a state of near hypnotic stupor, into which a knowledgeable guide could inject antidotes to ignorance. Assuming, of course, that the guide herself avoided the siren call of the table, a feat not within most of the gathered division chairpeople at the moment. Tatiana thought she saw a bit of drool on the representative from PE, a nearly senile woman named Gertrude Gertrod. There wasn’t much to work with, Tatiana thought, yet she knew she couldn’t afford to alienate these people if she was going to accomplish anything with her argument.

“The next order of business is consideration of reducing class meeting times from fifty minutes to forty,” the assistant dean said, ignoring the pesky Chair of Existential Studies. “This has the effect of increasing efficiency by 20%. Now let’s talk about pedagogy—what goes on in the classroom—how you teach your classes. I’m sure this will be a minor adjustment, but the dean and I want to make sure, ABSOLUTELY sure, that the faculty is on board with this.” He paused and stretched out his arms to include the whole campus in his warm embrace. “So let’s have it. What comments have you solicited from your departments? Your respective departments. Gertrude, let’s hear from PE—eye, ee, physical education.”

“Eh?” Gertrude roused herself and looked around. “Harry, you are a pedantic ass,” she announced to all who hadn’t noticed. Then she opened her mouth and let forth a stream of profanities, general in nature, but revolving around nuns from a certain parochial school. As the volume and anatomical specificity escalated, Harry’s eyes began to bulge and the tendons on his neck strained.

“Thank you for that report!” he attempted, but it was no good to interrupt her. She had to run her course, which she did after a good two minutes, at which point she had to breathe, and promptly fell asleep on the table.

“Ahem. Any more cogent—eye, ee, thoughtful reactions?” he asked finally.

“Well, I do have one tiny observation,” Tatiana said.

Harry glared for an instant, but there was little he could do, so he adopted a cameric smile and nodded for her to continue.

Tatiana began to speak very quickly and fluidly.

“I tried this in class—eye, ee, where I teach my students—tried talking 20% faster than usual and just running sentences together to economize on the empty spaces in conversation I had heretofore left as time for thought but which I realize now is completely unnecessary especially in Existential Studies as time for reflection can only subvert our true motives, and I have found that the result is amazingly good—the only problem is how to maintain this energy level for a whole forty minutes, eye, ee, two thirds of an hour, without collapsing from exhaustion.” She took an ostentatious breath and continued. “So I think it is only fair to prorate this time at a higher level than the comparatively luxurious fifty minute hour allows. Since we are expected to deliver the same content in less time, at a higher energy rate, we should be paid at a commensurately higher rate, eye, ee, we should all get 20% pay raises.”

Harry blanched and fixed her with his most lethal assistant deanship gaze. She did not turn to stone, however, but continued to speak.

“Also, I think this principle should be applied to all committee meetings, foreshortening the Brownian bureaucratic horror we face every Tuesday. Oops, look at the time! By my watch, newly accelerated, we are over time here. I’ve got to go put up my Christmas decorations.” She left the meeting agenda and her sparse notes lying on the table, and exited the room as gracefully as she could. Her head was spinning with the realization of what was happening. She was about two semesters from being out of a job—she and the whole Existential Studies department, except for maybe a stub left to teach the required Zen Cheerleading distribution requirement. She mused about who it might be. As she inserted the brass key in the doorknob to her office, she saw with dread that Milton Morton was shambling up. The large round man swayed with each step, an earnest expression plastered on his face as always.

“I was wondering if I could have a word, Doctor Tolstov,” he said, almost wringing his hands.

“Sure, Mil, step into my lair.”

As Milton settled into a creaking chair, he turned to make sure the door was shut behind him. He deposited a sheaf of wrinkled and besweated notes on the subject of transcendental frenetics on a table, and spoke.

“I heard a rumor, Doctor Tolstov, that we’re all going to be fired. Is it true?”

Amazing, Tatiana thought. The grapevine here was so good that news got out 3.5 milliseconds before it was actually created. She wondered if she should do a paper on endochronic communications for the annual conference. Automatically, the right words came to her lips.

“Well, you know how this place is, Mil. A butterfly landing on the president’s hat causes an earthquake in the locker room thirty minutes later. You can’t put any stock in what you hear from...where did you hear this?”

“Tick-tock’s secretary told me,” he said. Tick-tock was the nickname of the assistant dean for efficiency, eye, ee, Harry Hapollonian.

“Brenda?” Tatiana squelched her next remark as unsuitable for Milton’s ears. Brenda Bender took sadistic pleasure in meting out bad news. A perfect match for her boss.

“Look, Mil, we’ve been through this before. Every five years or so, they suddenly realize again that the traditional program is horribly inefficient and only stays afloat because it’s subsidized by the non-traditional program in the evening. So they do a reductionist bottom-up analysis and find a few arteries to cauterize. Meanwhile the patient needs an organ transplant.”

“So it’s true? I’m too old to go looking for jobs.” Milton’s face began to scrunch up.

“Don’t you dare cry on me again! Look, Mil, you’ve been here longer than any of us. So you only have a Master’s, believe me, that will actually put you ahead of the game. They aren’t going to fire you.”

“You think?” one sniffle escaped, to be absorbed by a generously stained sleeve.

“Sure. Think about it. They still have to cover the distribution requirement. Do you think any self-respecting Ph.D. is going to stick around for that alone, if they eliminate the majors? No offense, naturally. I’ll be out of here in a Shanghai second.”

“I wouldn’t want that,” Milton said sadly. Tatiana had never seen the man happy, but this was too much.

“Cheer up bud, things are darkest just before the execution.” She crinkled her nose to show him she was joking.

“Maybe we could get a petition together. Call a special faculty senate meeting? I’m sure our colleagues won’t stand for this.”

How little you know them, Tatiana thought. But she smiled anyway, at his optimism.

“Sure Mil, I’ll be the first to sign it. Now I’ve got to get some coffee.” The gentle hint was enough to get her guest to leave. She was completely surprised, moments later, when Milton reappeared with what looked exactly like a petition.

*** 10:15 am ***

Yvette Yakkit, Rocke’s one and only economics professor, cursed the ancient copier that had mangled yet another assignment. Red hieroglyphics theoretically showed her how to fix the problem.

“I’m an academic, not a mechanic!” she glared at the machine, as if to will the whole industrial revolution back into the genie’s bottle.

“Doctor Yakkit, would you like to sign my petition?” Milton thrust a clipboard at her. She turned to him, eyes still blazing, and cursed beautifully in Farsi.
“Sad-hezaar La'nat bar Sheytun!”
Then she realized her childishness.

“I’m sorry Milton, it’s this, ...this, machine.”

“It’s okay. I can come back later.”

“No, please, let me sign it. What is it?”

“It’s to call a special senate meeting to discuss the ongoing efficiency study.”

“Efficiency?” her eyes showed a trace of interest.

“You know—Tick-Tock’s committee,” Milton sneaked a peek to each side to make sure no administrator had heard him use the pejorative name for the assistant dean.

“Sure, then. Do you know how to fix this thing? I’ll kiss you if you do.”


*** 10:30 am ***

Bill Baglia was amazed that someone was knocking on the bathroom stall. He lowered the folded Wall Street Journal and forgot mutual fund listings for the moment.

“Yes? It’s occupied. Just a minute.”

“Doctor Baglia, I’m sorry to bother you, but you weren’t in your office, and I thought I might find you in here...”

“Milton? What the hell? What’s so important—is it Kay? Oh, my God, is—”

“—no, no, there’s no emergency. I’m sorry. I can wait.”

“Well, hell, man, out with it! What’s so important?”

Without saying a word, Milton slid the clipboard under the stall, where it was reluctantly retrieved. There was silence from within for a moment.

“Yes, I’ll sign it. About time someone did something about this! Our Biology budget is continually underfunded, while we have to put up with parasite programs like Existential Studies. What the hell is that anyway? Does anyone know? Sure I’ll be there.”

“It’s an important branch of ethics,” Milton attempted, taken aback by the assault on his turf.

“Ethics my arse.” He slid the clipboard back, ending the conversation with a significant rustling of his newspaper.


*** 11:00 am ***

“You got Baglia to sign?” Tatiana couldn’t hide her amazement. “He plays golf with Tick-Tock, you know.”

“Uh, he wasn’t exactly supportive of our agenda. I kind of annoyed him, I think. It was in the bathroom. He was in a stall.”

Tatiana laughed at the image. She shook her head.

“This is wonderful, though, Mil. When people see his signature on here, they’ll think the dean is on board too. Nice work! I think you’re going to get your meeting. How many more names do you need?”

“Seventeen,” Milton said proudly.

“This just might work,” Tatiana said with a grin. “Tell you what—do you have another clipboard?” The normally embarrassing idea of soliciting signatures on a petition had become a Quixotic mission, a last tilt at el molino de viento. She laughed at herself after Milton left, for being sucked into his project by the shear romantic absurdity of it. It felt good to be doing something, even if it was obviously futile. She wondered how many of the women on the Titanic had fixed their hair as the ship was going down.

*** 9:30 am Tuesday ***

Ezra Ebeknedzer, the president of Rocke College, was not a big man. He had a presence, though, and when he spoke, people tended to look around to see where the compelling strains of his voice originated. He met Tatiana at the door and shook her hand with both of hers, lingering just a bit too long for strict political correctness. A large custom carpet in the college’s colors, mauve and yellow, showed a sturdy oak with branches interwoven with a Latin banner proclaiming the academy’s motto: Cogito cogito ergo cogito sum. The president’s paneled oak desk was polished to an impeccable shine, and neatly arranged with piles of papers. There was no ugly computer monitor to destroy the aesthetic. Computers were for minions.

Ezra guided Tatiana to a round oak conference table, and sat her in one of the leather chairs.

“Coffee?”

“Black, thanks.”

“Two coffees, Miss Iona,” he addressed the open door, using the southern affectation he favored.

“Miss Tatiana. Tell me, how are things over in Existential Studies? Is the department pushing us forward or holding us back?”

The blunt question was a standard one, and Tatiana was expecting it.

“Forward, sir. We grease the psychic gears of mind machinery. Expect great things. Hoo-ah.” This last bit was obligatory homage to the CEO’s military career prior to coming to higher education.

“Hoo-ah, indeed. Well then—here’s the hot black stuff. Thank you Miss Iona. We’ll be in conference now.”

Ezra’s most powerful weapon as an executive was his patience. It was he who had called headstrong Chair of ES here after hearing about the bothersome petition, but now that she was here, he simply sat and waited for her to speak. The silence didn’t make him the least uncomfortable, and he had the uncanny ability to infect others with this calm. The silence endured long enough for Tatiana to settle automatically into her committee-trance. Her visual mantra was the image of dozens of memos rolling themselves into tubes and spontaneously assembling a long tunnel into which she was propelled. Blurred To: and From: lines in various fonts escorted her into that otherness of self communication through her internal meta-language (Wittgenstein was full of bratwurst in her opinion). Normally the drone of mindless verbiage from her colleagues and the brown table effect aided her on her journey, but today she had something better: the hypnotic gaze of the president. Time went hyperbolic for a while, and she wasn’t surprised to find herself talking at the end of it. She let the words flow, feeling them burp up from her chest, settle on her tongue, and float off in puffs, but making no sense of them whatever. The president was nodding, taking notes now and then.

“Would you like some more coffee?” he asked finally, breaking the spell. It was rhetorical because she hadn’t touched the stuff.

“Sorry,” she said, somewhat embarrassed. “I babbled, didn’t I?”

“Not too badly. You said what needed to be said.”

Tatiana sat, suddenly drained. The petition seemed like such a silly thing. What the hell had she been thinking? She gave her best sardonic grin.

“So how long do I have to live, doc?”

He threw his head back and laughed like it was the best joke he’d ever heard. A hearty, me-too kind of laugh that turned heads.

He’s so damned good, Tatiana thought, it’s impossible not to like this guy. Even though he’s about to fire me.

“So here’s the way it is, Miss Tatiana, since you’ve asked. Your classes are too small. Not just yours, but practically everybody’s. It’s just that ES is more expendable than, say, English.”

“But small classes are what we sell here...”

“I know, I know. But small and too small are different things. Do you know how much it costs to teach a class?”

“Ummmm.....”

“About eight grand in your department. Do you know how much tuition we get out of a class with four students in it? About $3,600.”

The coldness of the inequality left her defenseless. What good pitting philosophers against accountants? The outcome was clear.


“What if I give you bigger classes?” she asked. She sounded pathetic to herself. I should go buy myself a pocket protector and clip-on calculator, she thought.

The president narrowed his eyes to slits and steepled his fingers, calculating the politics of works in motion. Memos were already in flight to board members, key committee members were primed with the right bias, and a compliant Institutional Planning office had the proper reports prepared.

“I can’t make any promises,” he said. There was no more than a token of warmth left in his voice, and Tatiana knew the interview was over. She nodded, looked stupidly at the surface of the cold coffee, and stood to go.

“Hey,” he said to her back, “Try it. It’s worth a shot.”

“Sure. Thanks for your time.”

“Any time, Miss Tatiana. My door is always open.”
She gave Iona Isis half a smile as she left. She considered giving up, as she walked back to her building. But the thought of telling Milton was too much. Better to play the hand out. At least no one would think her a coward.

*** Lunch ***

Fried Spam quesadillas, folded neatly in half, leaked denatured cheese around the edge of Roland Rhumba’s plate. A half-destroyed textbook, veteran of hundreds of religious wars against ignorance supported the plate like a mythical array of elephants supporting the earth. From this ersatz pedestal he sawed at the tortillas.

“What’s news?” Tatiana asked, sliding in beside him. She was still upset from her meeting with the president, and had taken only salad. Only from the top of the bowl, of course. Underneath the healthy green stuff was the ‘less fresh’ greenery, commonly referred to as ‘compost.’ In the dining hall, any meal you could walk away from was a good meal.

“The news, my dear Tatiana, is that our students, to use an overly optimistic description of the primates I attempt to communicate with twelve hours a week, are insipid, completely materialistic, spoiled dimwits.”

“Uggies,” she replied superciliously, forking a safe-looking carrot.

“Beg your pardon?”

“Undergraduates. Uggies. That’s what I think of them as. It fits. Say it.”

“Uggie. Uggies. I have class on Comparative Religion with twelve uggies in it. Hey! I like it! Much better than ‘student’.”

“Thought you’d like that. It’s more accurate too. ‘Student’ implies learning. ‘Uggie’ just implies that they’re paying us to stay in the dorms and play lacrosse.”

“This you learned in Ukraine?”

“Nope. In my home town of Stanislav, just west of Odessa, we had no uggies. Only students. Anyone who fell by the way ended up smelling like рыба for the rest of her life. I learned about uggies when I came here.”

“I believe it.”

Tatiana sighed.

“Listen Roland, would you like to sign a petition for a called senate meeting? This efficiency thing is getting out of hand. Today it’s Existential Studies, tomorrow Comparative Religion. We have to hang together.”

“Yeah, I heard about this. Quite the rabble-rouser is your colleague Milton. Heard he rousted that insufferable Baglia person from the bathroom.”

“True,” she laughed again.

“I sent Milton packing when he came skulking around with that ridiculous clipboard. He should get a job as a meter reader.”

“His heart’s in the right place.”

“Look, Anna, what’s the point? You know how it works. We’ll have a big meeting, lots of expectations but no real agenda. The usual blowhards will let off steam. We’ll hear about the parking problem, the athletic program, the tuition level, all the usual crap, the complete catalog of emotional tics that parasitize our dear colleagues. Why in God’s name would I want to volunteer to listen to it again?”

Tatiana nodded. Any of her remaining enthusiasm had vanished. He was right, she knew.

“Maybe it doesn’t have to be that way,” she said.

“Right. Maybe there’ll be a burning bush to tell us God’s will.”

Tatiana stared at him. She was suddenly covered with gooseflesh. For the first time in a long, long time, she knew exactly what to do. The amorphous complexity of EVERYDAMNTHING had suddenly rolled over and in an instant shown her a soft underbelly where the rules didn’t apply anymore. She remembered finally to breathe.

“See a ghost?” Roland asked. Something in his eyes told her he was no stranger to the transcendental.

“What did Archimedes say about hammers?” She left the salad uneaten and prophesied as she stood to leave: “Don’t eat all that or it will kill you.”

“Don’t worry about me. And it’s levers, not hammers. Big enough to move the world. You just need a place to stand.”

Tatiana just grinned. She felt like Thor. Hammers indeed.

*** 4:15 pm ***

Milton grinned from Tatiana’s office door. She didn’t think she’d ever seen his teeth before.

“We did it!” he said, holding out the now-infamous clipboard.

“Yeah?” Tatiana stood from behind her desk and gave him a high-five. “When’s the meeting? Have they scheduled it?”

Milton wilted a bit.

“Man, was Penny mad! I mean Dr. Proctum. I thought she’d bite my head off.”

“Ah, yes, I imagine it’s more work for her. Being chair of the faculty senate is not a fun job even ordinarily. What’d she say?”

“She said it was a waste of time. That if even half of the people on the petition even showed up, she’d be shocked out of her underwear.”

“She may have a point about that. What do you suppose we could do to pack them in?”

Milton looked at her blankly.

“A scandal,” Tatiana mused. “We need a good scandal.” She thought for a minute until an idea came to her. There was no good in telling Milton, though. He could only screw things up. “You think of a good one, tell me, okay?”

“Okay,” Milton said, still puzzled. He seemed deflated now that the news had been delivered. “The meeting is on Thursday.”

“This Thursday? Good lord, that’s only two days! Oh, I get it. This is another way to make sure we don’t have a quorum. Very clever. I think dear Penny may have been chatting with the president.


*** 9:00 Wednesday ***


Brenda Bender looked across her cluttered desk at the chair of Existential Studies with the perfect fake smile. Tatiana glanced at the placard on her desk: Assistant to the Assistant Dean for Efficiency.

“Good morning,” Tatiana said. “Is Tick-Tock in?”

Brenda contained her irritation at the use of THAT name.

“No, DOCTOR HAPOLLONIAN is in a meeting.”

“Really? I didn’t see his car in the parking lot. Probably stayed up late playing poker and is sleeping it off. That’d be my guess.”

“What exactly do you want, Professor Tolstov?” Brenda shuffled some papers and turned to her computer.

“Oh, I just need a copy of the appendix to the Program Efficiency Report. My copy had the appendix omitted.” She waved a manila folder stuffed with papers.

Brenda sulked for a moment. “It’s in a file in his office. Just a minute.” She unlocked her boss’s door and vanished inside.

Tatiana quickly stepped to Brenda’s laser printer, slid open the bottom paper tray, and lifted out a thick sliver of the assistant dean’s personal letterhead. She slid the paper into the folder she’d brought. She shut the printer tray just as Brenda reappeared. Tatiana lifted the phone and pretended to dial.

“Sorry,” Tatiana said, “I forgot to call the cleaner’s this morning.”

Brenda just glowered. She ran off a copy of the missing appendix, and handed it to her tormentor.

“Not picking up... Guess I’ll try later. Thanks so much dear. Hope to see you again soon.” Tatiana left. Her heart was beating fast.

Back in her office, she locked her door and started composing.



*** 10:00 ***



Tatiana walked to the science building, remembering a time when she’d made the trip much more frequently. Inside, she stopped outside the office door with the placard for Professor Francisco Fandango, Chemistry. The lights were off. She inspected the course schedule on the wall.

“He’s in lab, down in 205,” a student with white coat and goggles volunteered.

“Thanks.” She knew well where the lab was.

“Cisco?” She called over the lab tables, glass tubes, and mysterious electronic machinery. The smell of the place brought back poignant memories for her.

“Anna?” the familiar voice called from the adjacent storeroom.

Cisco appeared with a beaker of some clear liquid swishing around. There was a crinkle of pain showing around the corners of his eyes.

“What brings you to these parts, stranger?” he attempted.

“Oh Cisco, I’ve missed you. You know that. That year we were on Curriculum together... I was sitting in my office, and it just hit me—it’s been too long since I’ve seen you. I was thinking maybe, if you’re interested, I’d cook something for you, and we could just talk about the ‘good ole’ days.”

“I’d like that,” he said. He squinted in thought. “Why now, Anna? It’s been almost a year since I’ve heard from you. It’s almost as if when our joint committee assignment was over, so was our relationship.”

“You’re so silly, Cisco. It sounds like you’re implying I was using you to get votes favorable to my department or something. What do you think I am?”

“Good lord, no! I wasn’t thinking that, really.” It was pretty clear what he was thinking from his roaming eyes.

“Well I hope not. So Friday night? I was thinking maybe Sea Bass with pomegranate sauce. Do you think Sauvignon Blanc is too light for the sauce?”

“Absolutely. I’d go with a red, in fact. I found a wonderful Chilean wine I’d be happy to bring.”

“It’s a date, then! Say eight o’clock at my place—oh, wait, better make it nine. I’ve got to stop by the mall and exchange something. There was a tear in my new silk pajamas. Can you imagine? A hundred dollars I paid for the things—not that there’s that much silk in them, if you know what I mean. Quite the rip off. Anyway, I’m babbling. See you Friday then?”

“Sure. Absolutely. Nine o’clock. You can count on it.”

Tatiana blew him a kiss and waved over her shoulder as she left. She stopped at the door, though, and turned.

“Oh, I almost forgot. Do you think you could do me a tiny favor, Cisco, my love?”

*** 8:15 Thursday ***

Fred Fromfeld, professor of Applied Ambiguities, paused, key in hand, before his office door. Two sheets of paper were stuck under the door. One was folded, and obviously expensive paper. The other was a plain white sheet with just the corner sticking out. He was a tall man, and his back creaked as he bent over to retrieve the items. He pulled reading glasses from his pocket as he pushed the door open.

The plain sheet was a reminder of the special called senate meeting at ten O’clock. He snorted and wadded it up to toss in the trash bin. He took more time with the thick cream letterhead, absorbing the words.

From: Harry Hapollonian, Assistant Dean for Efficiency
To: Faculty
Subject: Recommendations for improvements

The Efficiency Committee has completed its study, and is about to release the full findings to the Board of Trustees. Here is a brief summary of proposed changes that could impact the faculty, provided to you as a courtesy.

1. Classes will be reduced from 50 lecture minutes to 40 lecture minutes plus 10 minutes of student-to-student communication, which does not require the presence of a professor. The number of credits will remain the same for purposes of counting hours, but professors’ (respective) loads will be proportionally increased to account for this change. I.e. instead of teaching 12 credits at 50 minutes per credit in a semester, you will teach 14 credits at 40 minutes per meeting. There will be, of course, no pay increase, since you are teaching for the same amount of time either way.

2. No classes with fewer than seven students will be offered except as ‘extra’ unpaid offerings volunteered by the faculty. Instead, students will be expected to be lumped together in more generic ‘schoolhouse’ type classes at the upper levels. These may need to be interdisciplinary in nature in order to come up with the requisite number of students.

With the savings in efficiency from these two initiatives, we expect to be able to fully fund both men’s and women’s football teams in the fall. As a side note, all faculty parking spots will be eliminated to make room for the coaches.

Prof. Fromfeld gave a bellow that would have made any bull moose proud and headed straight for his department chair’s office.


*** 9:30 ***

Dining Services had not asked any questions when Tatiana had requested a large bowl of punch to be placed outside the auditorium for this morning’s meeting. All they wanted to know was what department to charge it to. She had gleefully given them the account number for a certain assistant dean.
The crew set it up efficiently, with plenty of ice and paper cups, and left. Tatiana stirred the pink liquid with the ladle and looked into the large ornate mirror on the wall. No one was around. She reached into her bag and retrieved the vial that Cisco had made for her. It had ended up taking a lot more convincing than she had figured on, but she had gotten what she wanted. She quickly emptied the clear contents into the bowl, dropped the vial back into her bag, and then stirred the punch some more. Then she went to the ladies’ room to dispose of the evidence and wait. It didn’t take long for an angry buzz of professor’s voices to be heard. It escalated quickly.





*** 10:10 ***

Penny Proctum banged the senate chair’s gavel for the fourth time.

“Can I have order please? PLEASE!”

The room was packed. Tatiana had never seen a faculty senate meeting so well attended, or so animated. She glanced at her watch, wondering how long it would take the drugs to start having effect for those lucky individuals who had drunk from the punch bowl. Just looking around, she saw that about half those gathered had paper cups in their hands. They would each shortly begin to feel the effects of the combination of a mild hallucinogen, a narcotic, and something like a truth serum, a cocktail produced late during the night by a certain chemist she knew. She was surprised to find that she felt no trace of guilt or fear. Francisco slid into the seat beside her.

“Exciting, huh?” he said, looking down her dress.

“You ain’t seen nothing yet, hon. You ain’t seen nothing yet.”



“HOO-AH!” the president’s voice boomed through the auditorium like an artillery shell. All voices muted. Ezra stood at the front of the room and put his paper cup on the podium, next to Penny’s. He spoke into the microphone.

“I would hope that you would show your elected chairperson a little more respect than this. Madame chair?”

“Thank you Mr. president. I hereby call this special meeting of the faculty senate to order.” She was clearly enjoying her role at the center of this storm. “And I believe that the president has an announcement to make concerning a certain memo that you may have received this morning.” She stepped away to give him the microphone.

“Thank you. Yes. Let me first say that I realize there is a lot of false information circulating out there. I can think of no better forum than this to set a few facts straight.”

He paused to take a sip of punch.

“When I was a boy,” he began, “I had a goat. It was a Billy goat and it would eat everything in sight. Nobody liked the goat but me, and my father in particular would take every opportunity to say what a stupid animal it was, and how it should be made into glue. His name was Goaty—the goat, not my father.” Those assembled laughed just enough to assure the president that they were still here. Almost everybody present had heard some version of the goat story at least once.

“One day,” he continued, “Goaty took a hankering to my father's workshirt. My father worked for the railroad, and they had provided him with a shirt. I remember that it had his name sown into the pocket. Donald, it said. That was my father’s name. My mother took some red thread and put ‘jr.’ after the name, because my grandfather's name was also Donald, and he worked for the railroad too. That shirt was special. Well, you can probably guess what Goaty did.”

Boy, can we, thought Tatiana.

“Goaty ate my father's workshirt. All except the buttons, which he left laying on the ground next to the clothesline in a neat little pile. He even ate the red thread that my mother had sewn into the pocket. I was the one to discover this act of consumption, and I had to decide what to do. On the one hand, I could tell my father what had happened, and suffer the consequences, or I could lie, and say that I didn't know what had happened to the shirt. I knew that if I told the truth, Goaty would be sold or given to Mr. Swanson, the butcher. I didn't say anything at dinner that night when the topic came up. Mother was worried because she couldn’t find the shirt, and allowed as to how it might have gotten caught in a gust and flown away due to the inferior clothespins she had to make do with. I just ate my carrot and potato pie in silence. But I knew I had to tell the truth eventually. And I did so the very next morning. And Goaty left our family for greener pastures.” Ezra looked heavenward for a second, and then started to get around to the point of this recitation of the Goaty saga.

“I believe that when the truth has to be told, the best thing is just to tell it.” He paused for effect.

“We have problem. Last year, despite our best efforts, we have not been able to match revenues with expenses for the academic year.

“The board of trustees has advised me that we will have to make some cuts. This will not be easy. As you know, last year we added several positions in anticipation of higher enrollment. Now those decisions will obviously have to be reviewed.” He took another sip of punch, and Tatiana thought she saw him twitch once.

“Now as to this, uh,..., this memo here. It appears that someone has played a little joke on you. This memo, on the Assistant Dean for Efficiency’s letterhead,” he wagged one of the offending documents over his head, “is a,...,a, uh,...” he swatted at the air in front of him a couple of times. There were scattered chuckles from the faculty.

“I lost my train of thought,” he said. Penny whispered in his ear.

“Ah, yes. My friends...” he leaned forward to be intimate with his audience, “there IS no such thing as women’s football! That was clearly a typo by that frigid excuse for a woman who types Tick-Tock’s memos because he never learned how to do it himself, the boob.”

The senate erupted in laughter. It went on until the president practically screamed into the microphone.

“AT EASE! I will have respect in my own house! Well, now I’ve said my piece. We’re all friends here. Except for a few of you—you know who you are—insufferable academics. Pains in the ass, if you want to know the truth.”

Penny frantically tried to get him to move away from the podium, and ultimately succeed with a powerful bump from her hip that almost sent him sprawling. More laughter greeted this bit of slapstick. No one had ever seen such an entertaining meeting. Professors looked around at each other to see if others were enjoying the carnival nature of the event. About half of them were appreciating it in a preternatural way, enhanced with bits of chemical magic.

“My god,” Cisco whispered to Tatiana, “What’s with these people?” His eyes were as big as bagels. Tatiana smiled a crooked smile and patted his hand.

“So, so, so, I think maybe we should, heh, get on with the completely useless business at hand,” Penny continued. “Milton, would you like to present the reasons for your, uh, petition?”

Milton stood. He swayed a bit, but his voice was strong in a way that Tatiana had never heard. She marveled that there could be a bit of iron in the man after all.

“NEVER in the course of world affairs has a group of people been treated so disrespectfully,” he said, throwing his arms up dramatically. The sight was so out of character and so completely absurd that it shocked some people back to reality. What the hell is going on here, you could almost hear them thinking. Milton continued on with one rhetorical device topping the next until wars had been rumored and declared and the horsemen of the apocalypse were in full gallop. Then he abruptly sat down again. There was dead silence.

“May I make a suggestion?” the mousy professor of Ceramic Meretriculation, Gerald Greer asked. Heads turned to look at him. He cleared his throat nervously.

“We obviously have several serious challenges to the curriculum here, in this memo, and really, in previous attempts to do the same thing. Rather than reacting purely emotionally, and destroying what good will we have with the administration, why don’t we accept that there is some truth to the fact that economic forces have to be considered as well as our philosophical principles.”

“Go on,” someone said.

“Well, I was just thinking. If we broke up into small groups of four or so and workshopped ideas, we might just come up with something that worked.”

Tatiana watched, amazed, as a kind of self-organization happened right before her eyes. Without any bureaucracy at all, professors formed small working groups, whipped out their red pens, and began brainstorming. The combination of drugs had apparently hit some ‘sweet spot’ in the collective consciousness. A buzz of creative energy filled the auditorium. Occasionally arguments broke out, but were quickly muted. The white walls of the auditorium became instant whiteboards, with new mission statements, curriculum flow charts, and cost/revenue ratios competing for space. It went on for hours. Tatiana left to get something to eat, and came back.

When she returned only about half of the faculty were still there. Cisco was gone, and had probably fled the country by now, not being entirely stupid.
There was an impressive diagram on one of the walls, and Carla Carlos, a biologist, was using a laser pointer to explain it to the others.

“This is a brewing vat, used to ferment the grain. See the filters here and here—and coils to maintain a constant temperature. Beer is not distilled like whiskey is, so there’s no need for high pressures or high temperatures. Biology students at all levels will be working closely with Chemistry to make sure that the beer is not only safe to drink, but tastes good too. That concludes my presentation. Who’s next?”

“I think that’s us,” Yvette Yakkit, from Economics, said. “Business majors will be required to have at least twelve credits of internship in the project. This will include financing, accounting, advertising, marketing, and market research. I have just discussed with the Graphic Design folks, and they will be doing logos for the product line.”

Tatiana watched in amazement as each department presented its contribution to the proposal. PE majors would be bouncers for the bar, to be established. English and Drama majors would be bartenders and servers, respectively. At the end of it, there was exhausted silence as the Accounting professors presented the bottom line: given certain conservative assumptions, the income from selling the bottled beer retail and wholesale could generate just as much revenue as the proposed changes from Tick-Tock’s committee would save. The many hours of internships that the students would have to take would be like free tuition dollars because of the low cost of instruction. It was a beautiful scheme. Tatiana left them as they debated what the beer should be called (‘Rocke Rolling Brew’ was the leading contender). She felt completely drained, and started to wonder what she had done.

Later, with a perfect martini at the perfect distance, Tatiana began her nightly self-analysis. She heard the beep of the answering machine from the other room, followed by the voice of a hysterical chemist. She tuned it out, and started down her mental Escher staircase. The question she would not ask and could not answer was the obvious one: Of what use is an Existential Studies major in a brewery?

Monday, August 06, 2007

Learning Outcomes

Our college has a standard template for reporting assessments of learning outcomes, which is used by most departments. At a minimum, each academic major needs learning outcomes for technology literacy, core skills (thinking and communication skills), and specific program outcomes, such as mastery of a body of knowledge or techniques.

Each objective has five parts. The first three comprise the plan, and are written in future tense. This plan typically rolls over from year to year with slight modifications. The other two headings on the report are the results and actions, and are written in past tense. You can find more details in our IE manual. The system for organizing the data has been released as an open-source project called openIGOR. The core skills (liberal arts) evaluation is called FACS, and is described in a paper that will be updated soon with last year's data. All of these links can be found together here.

I have attempted to remove information that would identify individuals, so the report is very slightly edited. There were no negative remarks from SACS about it, although some areas are clearly stronger than others. One external reviewer noted that many areas needed to be more explicit about their particular program outcomes.

The bracketed numbers refer to reference documents that are not included. Many of these are rubrics or matrices of some sort.

You can download the document with the completed learning objectives here. It's in Word. Note that in our haste to prepare the report, I ended up with several copies of this appendix. If you find typos or other nonsense, that means I probably didn't grab the final edited version.

Thursday, April 26, 2007

Strategic Planning On Line

I saw a demo this morning of software designed to help with planning, budgeting, and accreditation. It's sort of like openigor on steroids (but with some different feature sets). It's very pretty, as you can tell from the portal that comes up after log-in.

It's called Strategic Planning On Line, at www.strategicplanningonline.com. The idea is to integrate planning (goals and objectives) to budgeting, so that there is a direct link between the two. Items can be individually approved. As a bonus, accreditation objectives are broken out and can be assigned to individuals. The detail below shows some of the planning interface. It's all Web 2.0 (no click and refresh) running on .NET with SQL Server on the back end.


The screenshot below shows a budgeting sheet. There's a lot of detail here, with the ability to plan multiple years. Items are coded by general ledger types for intelligent reporting.

I have spent under an hour looking at the system, so I'm by no means an expert, but my first impressions are that it's a well-thought out and very cleverly programmed product. I think it could be a great management and institutional effectiveness tool if the administrative staff is really behind it. This kind of detailed planning very likely represents a cultural shift for an administration, with all that implies.

I don't know about pricing yet, but the software can run locally or (soon) remotely as a service.

Friday, April 06, 2007

Space Optimization

Our enrollment has been growing by leaps and bounds, which naturally makes other things want to grow too. Recently we've been discussing the "classroom crunch." Extra sections of service classes create pressures to find new places to teach classes. At the most popular hour, we're simply out of space. Or are we? I ran a query to tally up how many sections are taught in each of the standard time slots. This ignores a few oddball arrangements, but includes the 8am - 2pm MWF 50-min. sections as well as the usual TTh ones (e.g. 8:00am-10:45am TTh). There are 12 in all. There are 164 sections being taught in those slots, or 13.7 on average per slot.

The most popular time turns out to be TTh at 9:30am. (My favorite is 8:00am TTh, which is the least-used, so I can have about any classroom I want). The actual table is shown below.

I think it's fairly obvious that the perfectly optimal arrangement would be to have all slots used at the average level of 13-14 classes per slot. This not only reduces competition for space when scheduling, but reduces demand too. To see this, imagine that you are a student trying to fill out a schedule. If you must take a TTh 9:30am class, then there are 27 other classes that you cannot take because you can't be in two places at once. The graph below shows the actual and optimal distributions of sections across slots.


Can we quantify the difference this makes to students? At first I thought that using information entropy would be a likely metric. This is a cute idea, but doesn't seem to make sense on closer inspection. We're not communicating information here, but talking about course combinations. So the more prosaic solution is in order: the number of arrangements of possible classes is what's important. Ideally, I would like to compute the total number of schedules possible with the actual versus optimal arrangements. This is possible, although it requires some programming. I'll assume that most students would want to choose a five course schedule, which is typical. First, you'd need to generate all of the combinations of 12 objects taken 5 at a time (there are 792 of these) and associate those with the weights in the table, multiply those to get a product that represents the number of class schedules possible for the chosen slots and finally sum those to get the total number of schedules. That sounds like more fun that doing my taxes this morning, but unfortunately April 15 is just around the corner! Let's approximate instead.

Assume that a student's five desired classes are randomly distributed among the twelve slots. Then the expected number of schedules (i.e. average) would be the average number of classes available in a given slot, raised to the fifth power. But here we don't want to use the usual arithmetic average because we're multiplying. So we use the geometric average instead. To do this, we multiply the numbers in the table above and take the 12th root to get 11.9. By comparison, the geometric average of the optimal arrangement is the same as the arithmetic average (because all the numbers are the same--this is the only case where they're equal).

So we can say that after a student has chosen his or her most important class (crossing off one time slot), there are about two more sections available in each of the remaining ones in the optimal arrangement than in the actual one. We can now compute the (average) number of possible schedules that this represents.

Actual: 19,967 ways to fill in the four elective classes, after choosing the favorite.

Optimal: 34,886 ways.

This represents a 75% increase in schedule selections available to the student. I may put a computer science student on the job of actually calculating the exact answer, but this ought to be pretty close. With that kind of positive outcome, I'd argue that it's worth taking up the issue.

UPDATE: After doing the taxes I messed around with this a little more. Suppose a student has a roster of his or her top three picks for classes. All other factors being equal, what is the probability that the student can get all three of these classes? If ALL university classes are taught in one or two time slots the probability is obviously zero. I calculated this for both the given distribution and the optimal one and got 71% and 77%, respectively. So there's a six percent chance increase in the probability that a given student will be able to get his or her top three choices in the optimal distribution. This would increase the number of 'happy' students (using this definition) by about 8%. Not as dramatic as the analysis above, which was based solely on number of permutations available, but still significant.

Tuesday, February 27, 2007

A Generalization of Value Added

It was probably coincidence, but about the same time I sent an email to the nice folks at CAE telling them about my blog post criticizing the marketing of the CLA, I got a newsletter from them called the collegiate learning pulse, for February 2007.

There’s too much interesting stuff in it for me to write about in one evening, so let me focus on one bit having to do with the relationship between SAT and CLA scores.
[S]ome believe that scores on tests of broad competencies would behave like SAT scores simply because there is a modest correlation between the two.
In footnote 6 of the referenced document, we find such a correlation:
When the school is the unit of analysis, the SAT by itself accounts for about 70% of the variance in performance task scores.
At least, I assume that’s the right one. Supposing that’s in the ballpark, the coefficient would be the square root of .7, which rounds up to .84. I would probably choose a different adjective than ‘modest’ for this level of correlation. But there’s a more salient point here.
[E]mpirical analyses of thousands of students show that the CLA’s measures are sensitive to the effects of instruction; e.g., even after holding SAT scores constant, seniors tend to earn significantly higher CLA scores than freshmen.
Seniors do better on the CLA than freshmen do when the SAT is the same. Two questions occur to me. First, if the freshmen and the seniors are different people (i.e. this isn’t longitudinal study) there are issues with survivorship that are important. You simply can’t compare a group of student who may or may not make it to their senior year to those who have done so demonstrably. At least, you shouldn’t do that without justifying it. Maybe that’s been done, but if it has been put to rest that fact needs to be mentioned with a reference. But I’ve raised this issue before. Enough already about survivorship.

The second thing is this. The freshmen are taking the CLA as freshmen, and the seniors as seniors. But they both probably took the SAT as high school students. So the argument that the CLA results are generally different from SAT scores doesn’t work unless you retest the seniors with the SAT. I suspect that those scores would increase too.

Let me pose this thought experiment. Suppose we administered the SAT to seniors and applied the same methodology that the CLA is using to calculate value-added outcomes. We’d have to find a different predictor, obviously, since SAT is currently be used to predict CLA scores as I understand it. We could probably use high school GPA as a measure of ability. If the results are shown to be roughly similar to the ones CLA is getting, could we conclude that SAT is just as useful as CLA for measuring value-added?

In other words, how general is the CLA method of calculating value-added scores? Can it be used with any standardized test of freshmen and seniors? Could the NSSE, for example, be used to calculate a value-added score for engagement using these methods?

Monday, February 26, 2007

Questions of Validity

The debate on the Spellings Commissions’ report continues. For me, the central question is how accreditation will change, and particularly to what extent we will be required to use standardized testing.

I have to confess that it was disheartening to me to see the intermediate reports from the Commission as they seemed to tilt from the beginning toward a standardized testing ‘solution’ to the ‘problem’ of comparing institutions to each other. One should not be surprised that a debate that originates in Washington turns out to be political, I suppose, but the irony is thick. To all appearances, the debate on how to measure critical thinking did not proceed very scientifically.

Personally, I see this rush to judgment about standardized testing as intellectually offensive. Here’s why.
  1. The problem that the Commission wants to solve is not clearly spelled out. They never really say in plain, measurable terms what evidence of learning should be measured.
  2. Having made the fundamental error of not defining the desired outcome, they confuse it with the assessment device that one might use to measure the outcome. Finding a test that makes you happy is not the same as defining the outcomes.
Real learning outcomes are complicated, messy, and hard to measure. Frankly, I think the Commission would have been much better off to focus on the results that a good education should bring, not the education itself. These could include:
  1. Employment
  2. Income earned
  3. Civic engagement
  4. Graduate school degrees
  5. Professional certification
All of these outcomes are measurable with data already on hand (if you’re the US government, that is). Where are the reports showing what investment in a public or private education pays off over a lifetime? Yes, it’s important for graduates to be able to think and communicate effectively, but wouldn’t those abilities show up indirectly as higher employment and income? If not, they must not be that important to begin with.

This is the kind of self-evident homework I would have hoped the Commission would have done. Even their desire to compare institutions could be accommodated using indexes like these, although one would hope that they would take into account the different kinds of starting material.

But the testing steamroller is well underway at this point, so I’d like to explore that topic. It seems that we will be in the situation of having a test define a goal for us. That is, if score comparisons become the basis for performance ratings and institutional rankings, then what the test purports to measure is actually irrelevant.
Example. Joe thinks perhaps Mary doesn’t love him anymore. He reads in a magazine that emotional commitment causes a person’s pupils to dilate. Joe decides to measure Mary’s pupil dilation at certain intervals, and if they fall below a critical value he will leave her.
The example may sound absurd, but it is apt. Joe hasn’t really thought out what he means by ‘love’ and simply latches on to a convenient definition. We might call his functional definition love*, where the asterisk denotes a restricted definition (I borrow this notation from Philosophical Foundations of Neuroscience). Joe is committing a merelogical fallacy—confusing a part with the whole.

There is no danger in defining a restricted type of learning* as long as we’re aware that it’s not the whole picture. Standardized tests can be valuable for comparison to other institutions—it’s hard to do that any other way. But if learning* takes on regulatory and perhaps financial importance, which would be the case if scores are used for ranking institutions, it will create a false economy.

A strong emphasis on learning* will undoubtedly produce it in buckets. As I mentioned earlier, it shouldn’t be hard to improve standardized test scores with a little coaching (an industry devoted to this will certainly spring up). My friend and colleague David Kammler has penned a very funny piece called The Well Intentioned Commissar that describes the effects of such a false economy.

But am I being too cynical here? Perhaps the test is so good that the difference between learning and learning* is acceptable. That is, maybe the test is valid for this purpose. If so, my hypothesis that scores can be quickly improved by a few hours of coaching must be wrong. Otherwise, why do we need to waste four years of college on these students?

If standardized testing is indeed to become mandatory for higher education, the Collegiate Learning Assessment (CLA) is likely to be one of the candidates. This instrument seems to have been a darling of Mr. Charles Miller, chair of the commission. The test and its putative purpose of measuring value-added learning outcomes has been criticized by George Kuh and Trudy Banta, among others, and prompted a response from CLA researchers (see CLA: Facts & Fantasies for the latest salvo).

Let us ask our first question about the validity of this test. How long would it take to coach a freshman so that he or she improved to senior-level scores on the CLA? How many weeks of study do you think would be required to really significantly improve a student’s score? I’d guess about four, but who knows. The point is that it’s likely to be a lot less than four years. Shouldn’t that give us pause? Why are we wasting four years of education on these students if all that matters is four weeks worth of knowledge?

This thought experiment shows that learning* is likely to be significantly different from what students get from a college education. Perhaps it is a small subset, but certainly not enough to compare institutions with. This clearly shows the need to define goals in a comprehensive way before trying to measuring them. Assessing learning will always be incomplete and approximate, and the goal of an instrument precise enough to rate colleges and universities is probably unrealistic. I would argue it’s far better to choose metrics like the ones listed earlier (post-graduation employment, income, etc.) if we really have to have such a rating. What if we find out later that high CLA scores predict unemployment?

Standardized tests like the CLA are attractive because they eliminate complexity by reducing variables, which usually results in reasonably reliable instruments. But one can reliably get wrong answers too, like Joe with his pupil measurements. Richard Feynman commented on this subject.

I think the educational and psychological studies I mentioned are examples of what I would like to call cargo cult science. In the South Seas there is a cargo cult of people. During the war they saw airplanes land with lots of good materials, and they want the same thing to happen now. So they've arranged to imitate things like runways, to put fires along the sides of the runways, to make a wooden hut for a man to sit in, with two wooden pieces on his head like headphones and bars of bamboo sticking out like antennas--he's the controller--and they wait for the airplanes to land. They're doing everything right. The form is perfect. It looks exactly the way it looked before. But it doesn't work. No airplanes land. So I call these things cargo cult science, because they follow all the apparent precepts and forms of scientific investigation, but they're missing something essential, because the planes don't land.
This quote makes me squirm a bit because I spend a fair amount of time trying to measure learning outcomes myself. Am I conducting ‘cargo cult’ science? Feynman goes on to make a pitch for intellectual honesty, which is how I see a way around his criticism.
I would like to add something that's not essential to the science, but something I kind of believe, which is that you should not fool the layman when you're talking as a scientist. I am not trying to tell you what to do about cheating on your wife, or fooling your girlfriend, or something like that, when you're not trying to be a scientist, but just trying to be an ordinary human being. We'll leave those problems up to you and your rabbi. I'm talking about a specific, extra type of integrity that is not lying, but bending over backwards to show how you are maybe wrong, that you ought to have when acting as a scientist. And this is our responsibility as scientists, certainly to other scientists, and I think to laymen.
As long as we’re clear to distinguish learning* from learning, and to expose the weaknesses of our methods, and above all not to try to fool people about these things, I think we have license to talk about measuring learning outcomes.

An excellent example of this kind of professionalism can be found in the College BASE marketing material:
[S]tudies support the conclusion that the scores yielded by College BASE are a reasonable basis for making particular inferences about the achievement of individual examinees, and that the scores neither reflect the assessment of unintended abilities, nor are they beyond reasonable error. While validity studies completed by the Assessment Resource Center provide evidence for using College BASE as an estimation of a student’s mastery of skills for purposes described in its test content, an institution must still document use of the examination for its own specific purposes. Technical assistance for implementing College BASE at an institution is available from the Assessment Resource Center.

Responsibility for ensuring other validity evidence rests with the testing institution. Within its assessment plan the institution should address issues related to test administration conditions, cut score requirements, and appropriate accommodations to students as needed. The needs of each individual institution are unique to its mission and
its assessment program. The Assessment Resource Center can assist administrators in using individual and institutional data in appropriate and meaningful ways to improve student outcomes.
Why all the careful language? Because the test designers realize that equating learning* with learning is problematic. The quote above hints at the pains the designers have taken to try to find evidence of validity for the test. You can also request to be sent a Technical Manual. In the introductory material on validity in this manual they write:
The emphasis [on validity] is not on the instrument itself; rather, it is on the interpretation of the scores yielded by a test.
And later:
It is a common misconception that validity is a particular phenomenon whose presence in a test may be evaluated concretely and statistically. One often hears exclamations that a given test is “valid” or “not valid.” Such pronouncements are not credible, for they reflect neither the focus nor the complexity of validity.
I have heard IR directors use these very words. It’s usually a sentence like “We just use X standardized test because it’s been proven reliable and valid.” Valid for what? That’s how I interpret what the College BASE people are saying: how you interpret the results determines whether or not you’re conducting cargo cult science. I highly recommend taking a look at the whole document—it reads like a manual on how to construct a test. I’m afraid my own attempts at measurement fall far short of this standard.

To sum up where we are at this point: Think carefully about the goals, then try to find ways to assess them, but be skeptical of the ability of these instruments to render complete results. Like Descartes, you should spend a lot of time doubting yourself.

Compare the College BASE materials with a direct mail marketing flyer I got from the CLA. Here’s an excerpt from a box entitled “Validity and Reliability.”
The CLA measures were designed by nationally recognized experts in psychometrics and assessment, and field tested in order to ensure the highest levels of validity and reliability. For more information contact the CLA staff.
This is pure marketing hype, and violates Feynman’s dictum to not fool the public. There’s no indication that deciding validity is a complex process that depends heavily on what the results are to be used for. Perhaps we can charitably assume that the CLA marketing department hasn’t taken direction very well from the researchers. Surely the decision-makers who recommend this test are better informed, right?

Perhaps not. Richard Hersh, one of the main proponents of the CLA, writes in the Atlantic Monthly that:
For our purposes [grades and grade point averages] are nearly useless as indicators of overall educational quality--and not only because grade inflation has rendered GPAs so suspect that some corporate recruiters ask interviewees for their SAT scores instead. Grades are a matter of individual judgment, which varies wildly from class to class and school to school; they tend to reduce learning to what can be scored in short-answer form; and an A on a final exam or a term paper tells us nothing about how well a student will retain the knowledge and tools gained in coursework or apply them in novel situations.
He cannot be unaware that correlation between CLA scores and grade point averages is one of the pieces of evidence for validity of the test. According to a research article on the CLA website, GPA can ‘explain’ around 25% of the variance in CLA scores in the study they conducted. It seems quite odd to reject GPA in such strong language when there is such a strong relationship between the two. As a potential consumer of the product, this kind of disconnect is not reassuring.

My own sense is that the recent success of the CLA is due more to marketing and lobbying than to efficacy of the test. This is not to make any judgment about the test—I’m very interested in the serious research being done on it. From the data we’ve gathered at Coker College, I think you can make an argument that the value added idea works. Our method is the opposite of standardized testing, however, and could not reasonably be used to compare institutions. Incidentally, our research also seems validate that grade point averages are meaningful: higher averages lead to more value added. I blogged about it here.

I do have some questions about the CLA, however. These may be easily answered by the experts.
  1. Are freshman/senior comparisons being done longitudinally (i.e. by waiting four years to retest)? If not, how do you control for survivorship bias? The freshmen who eventually become seniors are likely to have more ability than those who drop out along the way. Even controlling for SAT and GPA are not going to make this question go away.
  2. The heavy reliance on the SAT for a predictor of CLA scores seems like a dubious method to me. The SAT is not very good at predicting success in college (at least at mine), so how do you convince a skeptic that the ‘value added’ is meaningful? Maybe the students are just getting better at taking standardized tests.
  3. I think there will be significant problems in getting students to take a three-hour test for which they are not graded. I can’t speak to whether or not this will affect the results of the test, but I can tell you that it will cause IR directors to pull their hair out.
  4. How much can scores be improved with a weekend training session?
I’m writing this from the perspective of an IR director who has to translate test results into action. In order to do this, I have to be able to explain in simple words what test results mean and how this relates to the curriculum and methods we employ. In short, how do I know I’m not engaging in Feynman’s cargo cult science?

Saturday, February 17, 2007

First, Do No Harm

Our state meeting of independent schools' institutional researchers was yesterday. As usual, I picked up some new ideas. I learned about Remark Office optical mark recognition, which can be used to create your own scannable forms for doing surveys. This eliminates having to buy the expensive forms from Scantron (which work well, however--that's what we currently use without problems). The scanner is apparently more forgiving about what kind of marks are made, also. So no more sharpening 1,000 pencils for student evaluation day.

There was also a good discussion of the role of the IR office in strategic planning. The 'safe' option for an IR office is to produce reports and forward them on to the decision-makers. I've done a fair amount of that myself when I was finding my way as a newly appointed IR director. The problem with that is that decision-makers aren't necessarily in the best position to use the data. Also, the information we work from is often incomplete and open to interpretation. The accreditors like SACS seem to imagine a perfect world where assessment inevitably leads to improvement. But the essence of leadership is courage, not data. In mulling this over, I have come to the conclusion that we in the IR business should help the decision makers as follows:
  1. First, do no harm. If the data are pretty strong against a current or proposed policy, use IR's influence to get the policy changed. Example: One of my colleagues at the meeting mentioned that he'd just finished a project having to do with AP testing. In this case, the faculty were convinced that students who used AP credit to skip the introductory class in a subject were ill-prepared to handle the second course. In looking at the data, however, he found that the opposite was true, except in one case. In that case the curriculum for the AP class wasn't a good match for the 'target' course. So he helped kill a proposal that would have eliminated AP credit at the university--a decision that would have had massive admissions implications.
  2. Most of the time, the data aren't going to point one way or the other. In these cases, it is better to encourage decision-makers to adopt policies that subjectively may have a positive affect. That is, rather than creating 'data paralysis' when there is no clear direction, keep moving and trying out new things. At worst you identify which policies are bad, and at best you may find the right one accidentally. Example. We adopted an initiative to improve student writing. We have some assessment data on writing, but nothing that clearly and unequivocally says DO THIS! So the committee assembled a variety of opinions about remedies and eventually adopted some of these as policy. They are quite reasonable and may be expected to improve the situation. In parallel, our assessments are improving, so we may be able to tell in a year or two whether or now we have succeeded. This is much better than simply giving up because there is no clear way forward.
  3. If the data clearly indicate the need for a new policy, do whatever you can to get it implemented. This is what an IR officer should live for--that one study or report that shows YES--this thing really matters and we should act on it. This is where the courage part of leadership comes into play because said IR director will be out there making the case as the spokesperson. Simply passing it on isn't good enough. Make it happen! Example. We recently did a shotgun-style survey of our students and found stark differences between first-generation students and students of college graduated parents. The differences were in attitudes about money, plans after graduation, and attitudes toward offices and services provided by the college. Given the national statistics on attrition for this group, this data cries out for action. So I've used three committees to sell the idea--the IE committee, our retention committee, and the one on improving writing. The issue of preparing first-gen students for their first year cuts across all these areas.
In summary, I advocate the position that the IR director form opinions about the usefulness of the data and then take the leadership to act on it. If it's equivocal, encourage the debate and look for opportunities to use data or at least improve assessment. But don't just publish a report full of charts and numbers with lists of standard errors and dump it on somebody's desk. Taking action by advocating for or against a policy takes courage because you can find yourself on the 'wrong' side of the administration, AND you can be wrong! But I believe it vastly increases the usefulness of the IR office to the institution. There is a danger, of course, that IR will seem to be more politicized. But if you call them as you see them rather than being a sycophant to the administration, you'll be respected for it.

This relates to another role of IR--to provide artificial certainty. The question "how many students do we have?" probably has a dozen possible answers, depending on what you count as a student (part-time? FTE? degree-seekers?) and when you take the snapshot. Decision-makers don't need to know all the possibilites. So figure out a method of measuring this fuzzy number and then publish it as fact. And it shall be so. As a story about three umpires has it, the third says of called plays: "They ain't nothing until I call them."

Monday, February 05, 2007

Predicting GPA

Every year we get an IR request to find better predictors for student success. This comes at me from three angles. One is the Standards Committee, which periodically reviews the cut-off values for admission and provisional admission. The other two interested parties are Admissions and the Budget Committee. We could improve the student body's entering statistics quite rapidly by eliminating the bottom half of the entering class, but that would cause heartburn in the Budget Committee when the calculations for revenue are done. Balancing these two isn't easy.

A large part of the problem is figuring out how to identify the students who are likely to fail. We can then either not admit them, or give them the help they need to succeed. Traditionally we have used composite SAT and high school GPA for this purpose. A linear regression model is used to predict GPA at the College. As an experiment I tried something new this year. In the Fall we gave 'Assessment Day' surveys to over a third of the traditional students. There were 100 questions, many of which pertain to students who are in attendance. Some of the questions could be answered by applicants, though, if we chose to do that. (For example: has either of your parents graduated from college?). I identified these and did a step-wise regression to see if any of them could add information to SAT and HSGPA in predicting grades. I found two. Both are surprising.

The two items are:
  • I work harder on homework than most students.
  • Family commitments often make studying difficult.
The responses were on a 1-5 Likert scale from Strongly Agree to Strongly Disagree. What's really interesting about these items is that they were significantly correlated (p < .03), and the effect sizes were as great as SAT. Strangely, both of the coefficients were the reverse of what I would have guessed.

The students who reported working harder on homework tended to have lower grades, and those who cited family commitments tended to have higher grades.

This is all experimental at present, but we're exploring the idea of using questions like these on an application form, so that we can better identify matches for the institution's goals. Without such additional information, our models can explain only about 44% of the variance in college grades. This is not bad--we can eliminate 56% of the 'bad' students at a cost of losing only 5% of the 'good' students. There are ways to improve this using demographic characteristics, but that has always seemed unethical to me, due to biases built into the SAT, for example.

Monday, January 22, 2007

Attitudes and Testing

In an article called The Curse of Being Different, in the 13 Jan. 2007 New Scientist, the author describes how assumptions and stereotypes can depress performance on tests. This was demonstrated by overcoming those barriers with a simple device. For example,

Between 2003 and 2006 Ilan Dar-Nimrod and Steven Heine at the University of British Columbia in Vancouver, Canada, tested the mathematical ability of 220 female college students (Science, vol 314, p 435). Before taking a maths test similar to one used by many colleges to screen applicants to graduate programmes, some of the women read passages arguing that there are fixed gender differences in mathematical ability, while others read that differences in ability can be modified by experience. The researchers predicted that viewing ability as changeable would make it easier for women to overcome the negative stereotype that paints maths as a predominantly male pursuit. They were right: the "changeable" intervention raised women's maths scores by an astonishing 50 per cent.
This resonates with something I heard at the AAC&U conference. I think it was in one of George Kuh's talks. He said that some students enter college with the belief that their brains have a fixed capacity beyond which they cannot grow. Others think they can expand their minds to accommodate new material. It's not hard to see that the former attitude could limit one's performance.

Given all the recent articles I've seen (and quoted some on this blog) on the role of attitude and belief, I'm a little surprised that the new liberal arts goals embraced by AAC&U's LEAP initiative do not include attitudes explicitly. There are references to ethics and citizenship, which could be taken to imply attitude changes, but not necessarily. One can certainly learn all about ethics without having any inclination to use that knowledge!