Test Fail

I came across two articles this morning with "test" and "fail" in the title. They're both worth a look.

"Failed tests" at The University of Chicago Magazine. Quote:

Neal, a professor in economics and the Committee on Education, insists it’s a “logical impossibility” that standardized tests, as they’re most often administered, could assess both teachers and students without compromising teacher integrity, student learning, or both. “The idea is that we want faculty held accountable for what students learn, so the tool that we use to measure what students learn is the tool that we should use to hold faculty accountable,” Neal says. “It’s all rhetorically very pleasing, but it has nothing to do with the economics of how you design incentive systems.”

Next is "Standardized Tests That Fail" in Inside HigherEd this morning. Quote:

“We find that placement tests do not yield strong predictions of how students will perform in college,” the researchers wrote. “In contrast, high school GPAs are useful for predicting many aspects of students’ college performance.”  

You may want to contrast that with another article by a test score true believer (my term). This one has stuck in my craw for a long time it's so bad, but I'll say no more about it. Anyone with basic critical thinking skills can figure out what's wrong with it. "'Academically Adrift': The News Gets Worse and Worse" in The Chronicle. I can't disagree with the conclusion of the article, however, so I'll quote that:

For those who are dissatisfied with the methods or findings of Academically Adrift, who chafe at the way it has been absorbed by the politicians and commentariat, there is only one recourse: Get started on research of your own. Higher education needs a much broader examination of how and whether it succeeds in educating students. Some of that research will doubtless become fodder for reckless criticism. But there's no turning back now.

[Update 3/1/2012] Here's one more: "The True Story of Pascale Mauclair" in EdWize. It's a horror story about the abuses of publishing standardized test results that are used to rate public school teachers. The bold in the quote is mine.

On Friday evening, New York Post reporters appeared at the door of the father of Pascale Mauclair, a sixth grade teacher at P.S. 11, the Kathryn Phelan School, which is located in the Woodside section of Queens. They told Mauclair’s father that his daughter was one of the worst teachers in New York City, based solely on the [Teacher Data Results] reports, and that they were looking to interview her.

Links on Learning

"The State of Science Standards 2012" maps out an analysis of pre-college science instruction in the United States.

Quote:

A majority of the states’ standards remain mediocre to awful. In fact, the average grade across all states is—once again—a thoroughly undistinguished C.

There are individual state reports at the site.


Next up is a high school student's ambitious "The education system is broken, and here's how to fix it." One of his complaints relates to the way analytical loading is done (see my previous post):

So, this causes students to go about the "textbook" skipping everything but the formulas, and then memorizing those. Then, when the test comes along, those who had time to memorize their formulas do excellent, and those that had something going on get low grades.

Then, as soon as they're done with the test, they put in all of their efforts into memorizing the next set of formulas and have nothing left from the last set that they memorized except "I love *whatever the topic is*, I got a 96% on that test".

 In a similar vein is a post from mathalicious.com: "Khan Academy: It's Different This Time." The author is critical of the eponymous video-based instruction site and its methods, claiming that:

Khan Academy may be one of the most dangerous phenomenon in education today. Not because of the site itself, but because of what it — or more appropriately, our obsession with it — says about how we as a nation view education, and what we’ve come to expect.

I think the author assumes too much about the implications of this. Video-based instruction is just a tool, which can be used correctly or abused.

A thread that runs through these three pieces is that intrinsic motivation is important. The report in the first link mentions the excitement that science generated during the Space Race, and how that's lacking today. The high school math student in the second article wants to know why, not just how. And the critique of the Khan Academy is alarmed at a potential "view and spew" pedagogy (my term, not the author's).

Learning terms, rules, methods, facts, connections, and so on can be pretty dry. This constitutes what I'm calling the analytical load required to do something more interested. Learning to play chords on a guitar is slow and painful, but then you get to play songs, which is fun.

I see tight, focused on-demand instruction like the Khan Academy as an essential resource for learning and reinforcing an analytical load. This can be augmented by additional material that motivates learners. There are plenty of ways to do that. Anything that looks like a story is good (history of science, for example). Applications that involve creativity are the ultimate objective.

All of the sources linked above are rightly critical of the prepare-and-certify model of education, which in practice turns into drill-and-test, with almost entirely external motivation. Teachers face a battle in winning back student enthusiasm against this machine. There's nothing wrong with being concerned about grades, but if that's all there is to it, students face a rude awakening after graduation.

More on that theme in an article at Common Dreams: "Wild Dreams: Anonymous, Arne Duncan, and High-Stakes Testing."

Finally, a link from Education Week on the subject of testing students who want to be teachers: "Analysis Raises Questions About Rigor of Teacher Tests." This is the meta-problem.

I-ACT: An Alternative to Prepare-and-Certify

In "The End of Preparation" I gave an alternative to the factory-like "prepare and certify" philosophy evident in the current practice of formal education. The purpose of the present article is to develop a modest trial program to test the portfolio approach.

In order to have something specific to talk about, I'm going to outline a course that could fit into many curricula, and could be scaled in sophistication to meet the level of the student. It might be most at home in a general education program, or as a topics course in the sciences. Here's the description:

Rong! Mistakes in Scientific Thought This course explores important conceptual mistakes in the history of scientific understanding. Even the most brilliant thinkers made bad assumptions, over-simplified, and took the wrong path occasionally. Major breakthroughs are as often related to getting rid of errant beliefs as finding better ones. Students will learn about some of these milestones, and perhaps develop some modesty about the certainty of their own beliefs.

On the first day of class, the instructor can explain that the neologism "rong" comes from Wolfgang Pauli's reputed remark that a line of reasoning was "not even wrong." Since rong is literally not even "wrong," it fits. I will use it in a noble sense, not as disparagement. An idea is rong for some fundamental reason that when understood, advances knowledge. The belief that the sun goes around the Earth is not just wrong, it's rong. The rongness may be conceptual (as in the case of geo-centricism) or methological (as with astrology). Both are important, and somewhat humbling to read about. Our forebears weren't stupid after all. As P. L. Seidel  wrote in 1847: "[M]ethodological discoverers are very badly treated. Before their method is accepted it is treated like a cranky theory; after it is treated as a trivial commonplace." Louis Pasteur comes to mind.

A Prepare-and-Certify Approach
The normal way to teach a course is to find a textbook and other source materials, set up a syllabus with major events like reading deadlines and test dates, outline a grading scheme, and list office hours. Students would write papers, get feedback, and ultimately a course grade. Then most of this effort would be forgotten and lost to posterity. In theory, the experience would have incrementally added to the "preparation" of the student for some eventuality that happens after graduation (the event horizon of education). In practice, no one would ever know if this is true because there are no objective measures for it. (See "An Index for Test Accuracy" to see what would be involved.)

Now that I have set the straw man in place, we can proceed to whack the silage out of him. It's clear that the course description cries out for a seminar-type approach, and that these sort of courses already exist. What I'll do below is enlarge the conception of a traditional seminar course. I need a name for this mutation, so let's call it I-ACT, which stands in for Analyze-Create-Publish-Interact (because I-ACT means something, and ACPI sounds like an economic index).

The I-ACT Approach

The role of the instructor is to help students pick good projects, and guide them through the steps Analyze-Create-Publish-Interact, which are described below in turn. But first a schematic, to break up this wall of text with a busy, colorful graphic.

1. Analysis


In order to produce new knowledge (that is, new to the student), one has to start somewhere. Academic disciplines comprise all sorts of knowledge, but here we are interested in whatever raw material can be turned into something new. In History, it might be original sources and a philosophy of history. In chemistry it might be a certain kind of molecule and knowledge of basic chemistry. In common games, like chess, the starting place is understanding of the rules and pieces.

In this bundle is sometimes a list of deductive rules that check for correctness. These would include accepted spelling of words (at a basic level), rules of logic, physics formulas, or any other deterministic method of turning one thing into another, which can be done correctly or incorrectly. A haiku has a particular structure, and a blues song has a certain scale and beat. A knight in chess can only move in a certain way.

The instructor assigns a problem (or the student is tasked to find one) that has an acceptable analytical load. We shouldn't expect kindergartners to solve systems of linear differential equations. The student doesn't need to be a master of this analytical domain, but it must be within reach. Resources include anything on the Internet plus the instructor, peers, and appropriate social networks.

As an example, I will use a real assignment I used for an undergraduate research project in math. The source is Proofs and Refutations by Imre Lakatos. One chapter in the book describes how the great mathematician Augustin-Louis Cauchy was rong about something, and how it got noticed and fixed. Cauchy provided a proof  that when an infinite sum of continuous functions converges to a new function, then that new function would be continuous as well. In this case, the student's analytical load includes basic math analysis techniques (delta-epsilon proofs), and familiarity with infinite series and functions. She should be able to read Cauchy's proof (probably with some difficulty, and needing help), and understand the issue. She should be able to create examples of the ingredients for the proof, such as series of continuous functions, and be able to test them and their sum for continuity.

An outline of Cauchy's errant proof, from page 132 of Proofs and Refutations.

The analysis box is never really complete. In any discipline there's always more to be learned, and part of the  I-ACT learning process is to go back to the well to seek clarification, examples, related concepts, and so forth. Learning this self-help process is an important objective.

2. Creativity

The analytical load need not be huge. Games generally have a small set of rules to make them accessible, and in fact you don't need a lot of rules to be creative. In the creative step, we help the student work with the analytical tools in a trial-and-error exploration. This is only possible if there are wrong answers. Another way to say that is if everything that the student can possibly produce is just fine, there's nothing to be learned from the exercise. A pilot should know a good landing from a bad landing. A doctor should know a live patient from a dead one. A musician should know a major chord from a minor seventh. And so on. The student is likely to make mistakes in this, which is where the instructor, peers, and social network can help.

Even in aesthetic subjects, we don't have to accept total relativism (and hence lost learning opportunities). In a photography class, instead of trying to figure out the exact artistic merits of a photo, one can examine technique. Is the subject in focus or not? Does the rule of thirds apply or not? Are the whites white or not?

To continue the example, the student was asked to find some series of continuous functions that converge to some new function, and then see if that new function was continuous. This took some work, and really exercised what she knew about functions, limits, and continuity. This strengthened her analytical skills, and her confidence increased so that she started to feel like she knew what she was doing. At this point, she was ready to tackle the question of why Cauchy was rong. Once that is discovered, it becomes a question of how to fix it. 

From Proofs and Refutations.

There are plenty of other creative exercises here, such as conjecture and proof of properties of uniformly continuous functions. Why don't Fourier Series work? This one example of rongness can be a point of departure for many analysis topics.

The exercise of checking steps in an argument is purely analytical, but creating a solution to a problem or finding other connections is not. The creative step should produce new knowledge for the student. Internet resources (like wolframalpha.com in the case of my example) can be used to find new connections, examples, and explanations.

3. Publication

Once a student has made some progress, it's time to write it up, or otherwise prepare the material for public display in electronic form on an intranet or world wide. Here, 'public' should at least mean that the instructor can see it, but that's not an advance over traditional delivery. Student peers in the same class or program, other faculty  members at the same or other institutions, social networks, and the whole world wide web are possible audiences.

Publishing interesting questions or intermediate results can be as useful as a finished piece of work.

When I supervised the student project on uniform convergence, social networks didn't exist like they do now. Now I could encourage a student to use reddit.com/r/math or stackoverflow.com to pose questions or try out ideas. This is not certain to succeed--these communities have to be engaged, not just spammed with drive-by questions.

4. Interaction

Interaction is a natural consequence of publishing. Over the summer I came across a delightful paper from Scott Aaronson at MIT entitled "Why Philosophers Should Care About Computational Complexity." I found it through a social network I frequent that scans for interesting stuff like this. After Scott posted the draft of his paper, he received a number of comments and suggestions. This feedback resulted in new drafts that clarified his thinking and fixed problems. Here's a quote from his blog:

Thanks to everyone who offered useful feedback! I uploaded a slightly-revised version, adding a “note of humility” to the introduction, correcting the footnote about Cramer’s Conjecture, incorporating Gil Kalai’s point that an efficient program to pass the Turing Test could exist but be computationally intractable to find, adding some more references, and starting the statement of Valiant’s sample-size theorem with the word “Consider…” instead of “Fix…”

Then there's the meta-commentary from philosophers about the paper on reddit, which adds a perspective and some new references.

Interaction this rich can illuminate everything about the work, including analysis and creativity. It can critique or endorse, dismiss or expand scope.

Desired Outcomes

How is this an improvement over traditional classes or seminars?  For me, there are several answers:

• Learning to rely on a self-help network of resources.
• Public display of one's work can lead to intrinsic motivation that is greater than the extrinsic "I need to get a C in this course."
• The above point is magnified by emphasizing that this work forms part of a life-work portfolio that will be useful for a very long time. In addition to certifications (eventually instead of certifications), students have authentic work to display.
• Engagement in social networks and general audiences that care about the topic is a good long-term investment. It leverages ones own abilities and adds credibility to one's published works.
• Student work competes on merit, not on who they are or what institution they're from, and can help them assess their own skill and knowledge.
• By separating analytical techniques from creativity, we can prepare students for both. Creativity takes self-confidence and practice. This can be nurtured in a controlled environment.
• All the tools and methods used can be applied outside of a college setting. It's a practical real-world skill to develop a skill set, create something with it, publish it online professionally, and generate feedback for improvement. 

Next Steps

I am looking for a handful of science departments to try some of these ideas out. The course description above is one of many that could be used. Once the details are in order, I'll seek some external funding for travel, some implementation costs (setting up a portfolio system maybe), and money to run a small conference. If you're interested, click on my profile or link to my vita and email me.