Showing posts with label critical thinking. Show all posts
Showing posts with label critical thinking. Show all posts

Monday, February 18, 2013

Teaching Critical Thinking

I just came across a 2007 article by Daniel T. Willingham "Critical Thinking: Why is it so hard to teach?" Critical thinking is very commonly found in lists of learning outcomes for general education or even at the institution level. In practice, it's very difficult to even define, let alone teach or assess. The article is a nice survey of the problem.

The approach I've taken in the past (with the FACS assessment) I've simplified 'critical thinking' into two types of reasoning that are easy to identify: deductive and inductive. Interestingly, this shows up in the article too, where the author describes the difference (in his mind) between critical and non-critical thinking:
For example, solving a complex but familiar physics problem by applying a multi-step algorithm isn’t critical thinking because you are really drawing on memory to solve the problem. But devising a new algorithm is critical thinking.
Applying a multi-step algorithm is deductive "follow-the-rules" thinking. He's excluding that from critical thinking per se. To my mind this is splitting hairs: one cannot find a clever chess move unless one knows the rules. We would probably agree that deductive thinking is absolutely prerequisite to critical thinking, and this point is made throughout the article, where it's included in "domain knowledge."

In the quote above, the creation of a new algorithm exemplifies critical thinking--this is precisely inductive thinking, a kind of inference.

Now I don't really believe that even the combination of deductive and inductive reasoning covers all of what people call 'critical thinking,' because it's too amorphous. It's interesting to consider how one might create a curriculum that focuses on 'critical' rather than 'thinking.' It could be a course on all the ways that people are commonly fooled, either by themselves or others. It would be easy enough to come up with a reading list.

Another alternative is to focus on the 'thinking' part first. This seems like a very worthy goal, and in retrospect it's striking that we don't seem to have a model of intelligence that we apply to teaching and learning. We have domain-specific tricks and rules, conventions and received wisdom, but we generally don't try to fit all those into a common framework, which we might call "general intelligence" as easily as "critical thinking." Usually it's the other way around--how do I embed some critical thinking object into  my calculus class? This latter method doesn't work very well because the assessments results (despite our desires) don't transfer easily from one subject to the next. This is the main point of the article linked at the top--domain-specific knowledge is very important to whatever "critical thinking" may be.

A Model for Thinking

I don't presume to have discovered the way thinking works, but it's reasonable to try to organize a framework for the purposes of approaching 'critical thinking' as an educational goal. The following one comes from a series of articles I wrote for the Institute for Ethics and Emerging Technologies (first, second, third), which all began with this article. The theme is how to address threats to the survival of intelligent systems, and it's informed by artificial intelligence research.

A schematic of the model is shown below.


We might think of this as a cycle of awareness, comprising perception, prediction,  motivation, and action. If these correspond to the whims of external reality, then we can reasonably be said to function intelligently.

The part we usually think of as intelligence is the top left box, but it has no usefulness on its own. It's a general purpose predictor that I'll refer to as an informational ontology. It works with language exclusively, just as a computer's CPU does, or the neurons in a our brains do (the "language" of transmitted nerve impulses). Languages have internal organization by some convention (syntax), and associations with the real world (semantics). The latter cannot exist solely as a cognitive element--it has to be hooked up to an input/output system. These are represented by the lower left and right blue boxes. The left one converts reality into language (usually very approximately), and the right one attempts to affect external reality by taking some action described in language.

All of these parts are goal-oriented, as driven by some preset motivation. All of this perfectly models the typical view of institutional effectiveness, by the way, except that the role of the ontology is minimized--which is why IE looks easy until you try to actually do it.

Each of these components is a useful point of analysis for teaching and learning.  Going around the figure from bottom left:

Measurement/Description When we encode physical reality into language, we do so selectively, depending on the bandwidth and motivation, and our ability to use the result in our ontology. At the beach, we could spend the entire day counting grains of sand, so as to get a better idea of how many there are, but we generally don't because we don't care to that level of precision. We do care that there's sand (the point of going to beach), but there are limits to how accurately we want to know.

Some language is precise (as in the sciences), and other sorts not (everyday speech, usually). What makes it usefully precise is not the expression of the language itself (e.g. I drank 13.439594859 oz of coffee this morning), but how reliably that information can be used to make predictions that we care about. This involves the whole cycle of awareness.

Example 1: According to wikipedia, the mass of a proton is 1.672621777×10e-27. This is a very precise bit of language that means something to physicists who work with protons. That is, they have an ontology within which to use this information in ways they care about. Most of us lack this understanding, and so come away with merely "protons weigh a very tiny amount."

Example 1: Your friend says to you "Whatever you do, don't ride in the car with Stanislav driving--he's a maniac!" Assuming you know the person in question, this might be information that you perceive as important enough to act on. The summary and implication in your friend's declaration constitutes the translation from physical reality into language in a way that is instantly usable in the predictive apparatus of the ontology. Assuming you care about life and limb, you may feel disinclined to carpool with Stanislav. On the other hand, if the speaker is someone whom you think exaggerates (this is part of your ontology), then you may discount this observation as not useful information.

The point of these examples is that description is closely tied with the other elements of awareness. This is why our ways of forming information through perception are very approximate. They're good enough for us to get what we want, but no better. (This is called Interface Theory.)

Here are some questions for our nascent critical thinkers:
  1. Where did the information come from? 
  2. Can it be reliably reproduced?
  3. What self-motivations are involved?
  4. What motivations do the information's source have?
  5. What is the ontology that the information is intended to be used in?
  6. How does using the information affect physical reality (as perceived by subsequent observations)?
Notice that these questions are also very applicable to any IE loop.

Question five is a very rich one because it asks us to compare what the provider of the information believes versus what we believe. Every one of us has our own unique ontology, comprising our uses of language, beliefs, and domain-specific language. If I say that "your horoscope predicts bad luck for you tomorrow," then you are being invited to adopt my ontology as your own. You essentially have to if you want to use the information provided. This represents a dilemma that we face constantly as social animals--which bits of ontology do we internalize as our own, and which do we reject? Which brings us to the 'critical' part of 'critical thinking.'

It's interesting that the discussion around critical thinking as an academic object focuses on the cognitive at the expense of the non-cognitive. But in fact, it's purely a question of motivation. I will believe in astrology if I want to, or I will not believe in it because I don't want to. The question is much more complicated than that, of course, because every part of the ontology is linked to every other part. I can't just take my whole system of beliefs and plop astrology down in the middle and then hook up all the pipes so it works again. For me personally, it would require significant rewiring of what I believe about cause and effect, so I'd have to subtract (stop believing some things) part of the ontology. But this, in turn, is only because I like my ontology to be logical. There's no a priori reason why we can't believe two incompatible ideas, other than we may prefer not to. In fact, there are inevitably countless contradictions in what we believe, owing to the fact that we have a jumble of motivations hacked together and presented to us by our evolutionary history.

Intelligence

The usefulness of intelligence lies in being able to predict the future (with or without our active involvement) in order to satisfy motivations. The way we maintain these informational ontologies is a dark mystery. We seem to be able to absorb facts and implications reasonably easily (Moravec's Paradox notwithstanding); we can't deduce nearly as quickly as a computer can, but we manage well enough. It's the inductive/creative process that's the real mystery, and there is a lot of theoretical work on that, trying to reproduce in machines what humans can do. Within this block are several rich topics to teach and assess:
  1. Domain-specific knowledge. This is what a lot of course content is about: facts and deductive rules and conventions of various disciplines, ways of thinking about particular subjects, so that we can predict specific kinds of events. This connects to epistemology when one adds doubt as an element of knowledge, which then leads to...
  2. Inference. How do we get from the specific to the general? At what point do we believe something? This links to philosophy, the scientific method, math and logic, computer science, neuroscience, and so on. Another connection is the role of creativity or random exploration in the process of discovering patterns. We might sum up the situation as "assumptions: you can't live with them, and you can't live without them." Because inference is a fancy word for guessing, it's particularly susceptible to influence from motivation. Superstition,  for example, is an application of inference (if I break a mirror, then I will have bad luck), and one's bias toward or away from this sort of believe comes from a motivational pay-off (e.g. a good feeling that comes from understanding and hence controlling the world).
  3. Meta-cognition. This is the business of improving our ontologies by weeding out things we don't like, or by making things work better by pruning or introducing better methods of (for example) inference. This is what Daniel Kahneman's book Thinking, Fast and Slow, is about. That book alone could be a semester-length course. Any educational treatment of critical thinking is about meta-cognition.
  4. Nominal versus real. Because we live in complex information-laden societies, we deal not just with physical reality but also with system reality. For more on these, refer to my IEET articles. One example will suffice: a system pronouncement of "guilt" in a trial may or  may not correspond to events in physical reality. At the point the verdict is announced, it becomes a system reality (what I call a nominal reality). The ontology of the system becomes a big part of our own personal version, and one could spend a long time sorting out what's real and what's nominal. For more on that topic, see this paper I wrote for a lit conference.
Motivation
Humans and the systems we build are very selective about what we want to know, and what we do with that knowledge. Understanding our own motivations and the that of others (theory of mind), and the ways these influence the cycle of perceive-predict-act, is essential in order to make accurate predictions. That is, intelligence has to take motivation into consideration. This invites a conversation about game theory, for example. The interpretation of critical thinking as the kind of thing that investigative reporters to, for example, must take motivations of sources into consideration as a matter of course.

In economics, motivation looks like a utility function to be optimized. Part of what makes humans so interesting is that we are laden with a hodge-podge of motivations courtesy of our genes and culture, and they are often contradictory (we can be afraid of a car crash, yet fall asleep at the wheel). The search for an 'ultimate' motivation has occupied our race for a long time, with no end in sight.
Here's a critical thinking problem: If motivations are like utility functions, they must be acted on in the context of some particular ontology, which goes out of date as we get smarter. How then are we to update motivations? A specific example would be physical pain--it's an old hardwired system that helped our ancestors survive, but it's a crude instrument, and leads to a lot of senseless suffering. The invention of pain-killers gives us a crude hack to cut the signal, but they have their own drawbacks. Wouldn't it be better to re-engineer the whole system? But we have to be motivated to do that. Now apply that principle generally. Do you see the problem?
Taking Action
This isn't usually thought of in connection with intelligence or critical thinking, but it's integral to the whole project. This is generally not the approach we take in formal education, where we implicitly assume that lectures and tests suffice to increase student abilities. Come to think of it, we don't even have a word for "active use of intelligence." Maybe 'street smarts' comes close, because of its association with 'real-world' rather than academic, but that's an unproductive distinction. I've heard military people call it the X-factor, which I take to mean a seamless connection between perception, prediction, and action (all tied to some underlying motivation, of course).

But of course the point of all this intelligence apparatus is to allow us to act for some purpose. There are great illustrations of this in Michael Lewis's book The Big Short, which show the struggle between hope and fear (motivations) in the analysis of the looming mortgage disaster, and the actions that resulted.

I've argued before (in "The End of Preparation," which is becoming a book) that technological and societal changes allow us to introduce meaningful action as pedagogy. It's the actual proof that someone has learned to think critically--if they act on it.

Being Critical
If some framework like the one described above can be used to examine intelligence in a curriculum, where exactly does the modifier come in? What's critical about critical thinking? Perhaps the simplest interpretation is that critique allows us to distinguish between two important cases (which may vary, but correspond to motivations). For example, in a jury trial, the question is whether or not to convict, based on the perceptions and analysis of the proceedings. It's these sorts of dichotomies--the aggravating fact that we can't take both paths in the wood--that makes intelligence necessary in the first place.

This general task is big business these days, in the form of machine learning, where distinguishing between a movie you will like and one you won't is called a classification problem. Netflix paid a million dollars to the winner of a contest to find a find a better classifier for assigning movie ratings.

It also makes a nice framework for teaching, and it's a common technique to set up a A vs. B problem and ask students to defend a position (and there's a whole library of resources set up to provide support for this kind of thing). In the abstract, these undoubtedly have some value it honing research and debate skills, but it seems to me that they would be more valuable when connect to real actions that a student might take. Is it worth my while to go to Washington to protest against X? Or go door-to-door to raise money for Y? Or invest my efforts in raising awareness about Z with my project? Maybe we need a new name for this: active critical thinking, perhaps.

So as educators, we are then left with the meta-question: is this worth doing?

Next: "Interfaces and Education" continues this line of thought.

Monday, March 14, 2011

Academically Adrift

I've come up for air after posting part three of Life Artificial yesterday evening. Over the last few weeks I've had dozens of browser tabs open to write a post about, but have been too focused on getting particular projects done to write them.

I finished reading most of Academically Adrift last week, meaning I've started but not finished the Appendices. I downloaded the book from Amazon and read it on the iPad's Kindle application. This is very convenient, and the reading experience is fine, but it does have a significant drawback. Despite the ability to bookmark pages and leave yourself notes and highlights, it's not very easy for me to mark up the book in an easily accessible fashion for reference purposes. Normally I'd have sticky notes coming out of the leaves of the volume, and have text circled by scribbled notes in the margin. This by way of apology that I don't have many quotes in this post.

I've never been impressed with the CLA, and the research in Academically Adrift depends heavily on it. I hasten to add that I don't have any problems with the laudable aim of assessing "critical thinking, complex reasoning, and writing." I think the instrument itself can even be useful. It's the exaggerated claims of importance and over-statement of validity that seem unwarranted to me. Because so much of the book relies on CLA scores, let me elaborate.

First, the test is advertised as posing "real world" problems to test subjects. I'm not quite sure what this means, but it sounds good. An example is given on page 22-23:

This seems to be a particularly poor exemplar of a real world problem. On the face of it: would you ask a twenty-something with two years of college to make a decision like this? Undoubtedly the aircraft is an expensive purchase that, more to the point, you'll be trusting your life to. So the way you make the decision is to give a non-expert a limited amount of documentation and ask for a report?  I don't think so.

Real life problems are not neatly defined by a few newspaper articles, FAA reports, and such. There's a vast body of knowledge to sort through an analyze if one cares to look for it. It is in the form of web searches, academic and professional articles, interviews with experts, and on and on. I understand that it would make the test much harder to administer to allow open-ended searches, and it would take a long time. Days or weeks. This just highlights how far this exercise is from a real "real life problem."

Real life problems are generally solved by people who know what they are doing. You don't ask the guy at the car wash to take out your gall bladder for a good reason. If you want to know about safety issues with a particular model of airplane, I think it would be really good to talk to a senior mechanic that works on such planes and to pilots that fly them. In fact, putting an expert in the field in charge of your investigation would be the best thing to do, no? This is one of the problems with assessments of "critical thinking" that try to ignore content knowledge. It doesn't work. Content is really important when you want to solve a problem, unless all you care about is sophistry.

Here's a real life problem:
You're the head mechanic who supervises a small fleet of helicopters for a rent-a-bird operation. The chief pilot is planning to fly in tomorrow to test out a new Bell Jet Ranger that you're getting ready for operations. But you and your team haven't been able to get the main rotor balanced, and you suspect that the part that came from Bell to do this is defective. But the pilot has a volatile temper, and is going to be very upset with you if the helicopter isn't ready for him to fly when he gets here. What do you do?
This requires technical knowledge, creativity, and judgment about social dynamics to get through, and there's probably no perfect solution. This was a real world problem I witnessed as a teenager. I saw the mechanics try to balance that rotor blade for hours. They finally gave up, put it on the helicopter anyway, and let the pilot try to take off with it that way! It was so out of balance, that he could only wobble the aircraft around on the ground before giving up. He emerged looking like a ghost, too relieved to be alive to be mad at anyone. Probably not an ideal solution.

A problem with the analysis in the book is the emphasis on differences in scores and the way the information is represented. Others have mentioned (e.g. this article in The Chronicle) that the authors of Academically Adrift put a lot of importance on how many students haven't demonstrated significant learning. The literature I've read on the CLA says that it's not even designed for student-level analysis, but for comparing institutions. But never mind that. The statistical problem that has been pointed out by others is that a hypothesis test that does not find a difference in pre-post tests does not mean that there is no difference. This is from Stats 101. As an analogy, let's say my kid is getting ready for school and can't find her gym clothes. I don't have my glasses on yet and am bumbling around the house before the first cup of coffee has kicked in. But I look here and there for the gym clothes halfheartedly as I walk around. There are two possibilities:
  1. I see the distinctive purple bag. Even without my glasses on, there's a reasonably high probability that I've found the gym clothes.
  2. I look in the room but don't see the bag. I cannot say that the bag isn't in the room, only that I didn't see it. It may, in fact, be just behind the door.
This is the difference between rejecting the null hypothesis (#1) and not rejecting it (#2). Yet the authors lead us to believe through their narrative that not finding a difference means that there is no difference, and go on to claim that not much learning is happening during the first two years of college. 

Although the authors wave at the notion of IQ late in the book, they never say anything about the relationship between what they are trying to measure and intelligence (as assessed by standard instruments). Psychologists assure us that IQ is largely static. It's also closely related to verbal skills, which is what the CLA assesses too. If IQ is largely immutable, don't we need to factor it out before making conclusions about improvement? I understand that this is very sensitive issue because you ultimately would have to say that some people are smarter than other people (with all the fine print that goes with that), but to completely ignore the issue seems disingenuous.

We shouldn't necessarily reject all the findings of the book just because the analysis is flawed. The attention on how hard students work and the corresponding rigor of academics is an important discussion. Unfortunately, the way the headline findings of the text overstate the findings on complex learning outcomes only feeds into the current of public discourse advertising that teachers are villains. Despite carefully worded disclaimers that general readers won't decode, the book practically screams that the first two years of college don't produce any meaningful learning. This is a disservice to all the course instructors out there who work hard to teach students calculus, computer programming, world history, composition, and so on.

The fact is that the content of the CLA doesn't correspond well to the course content of a general education curriculum. This could be remedied easily. I am quite sure that a one-semester prep course on items like the ones on the CLA would significantly increase the scores. Would this result in real, meaningful knowledge? Maybe, depending on the depth of the course. A pure test-prep course that teaches how to game the test is probably worthless. But perhaps the current curricula are out of date. Even if that's true, the importance of the CLA and its standardized measurement are not very important because they are too simplified, and rely too much (I assume) on IQ, and not enough on discipline-based knowledge. There are already recommendations (e.g. from the AAC&U) to increase complex problem solving skills, and many institutions are already struggling with how to teach and assess this. But almost by definition, complex problems aren't trivial to solve. If the CLA becomes the measure of these learning outcomes--if we really take analyses like Academically Adrift seriously--they we will shortly have another instance of Campbell's Law at work. See this Wall Street Journal article for a good example. Or this opinion about "impact factors."

Learning outcomes don't matter anyway, in the big picture. That is, test scores and other performance indicators taken during college aren't ultimately the measure of success of a program. It's what happens after graduation that ultimately matters. Are your graduates curious about the world? Do they read newspapers and vote intelligently? Are they engaged in service to their fellow humans? Do they contribute to the overall economy? Depending on the mission of the institution, these goals may vary (but always include "do they donate money?"). At the level of the Education Department, where the strategy should inform the national interests, perhaps the easiest metric of success would be income histories. The federal government has vast data stores on financial aid given. It also has all the IRS data. I asked an official from DoE about mashing these up to answer important questions like "what's the lifetime earnings benefit for attending a private vs public college?". I wasn't encouraged by the answer. No one cares enough or the problem seems too hard. And yet, I'm sure if we gave all the data to the guys at Facebook, they'd have it figured out over the weekend.

Saturday, May 08, 2010

Pizza, Dinosaurs, and Critical Thinking

Today at the supermarket with my daughter Epsilon, I sized up the frozen pizzas.  See the photo below.

I asked Epsilon to look at the pizza boxes and analyze them.  She surprised me by observing that the images all had fresh veggies or pictures of plants in the background, suggesting freshness.  Yeah, yeah, I said, but what about that one THERE--Amy's Pizza (Pesto).  That slice they show is disproportionately large, I told her.  See?  They cut an extra big piece to make you think you get more than you really do.

She looked.  Pursed her lips, and held out her fingers to make a crude caliper to measure the distance from the center of the cut to the edge of the pie.  Then she rotated it around carefully, tracing the edge of crust.  No, she said, that's the right size.

She was right.  My conspiracy theory was hot air.

We want our graduates to be able to do that sort of thing, right?  We might call it critical thinking, although the term makes me break out in hives.  Richard Arum and Josipa Roska have a book coming out on that topic called Academically Adrift: Limited Learning on College Campuses, claiming (according to this NYU blog post) that a large number of college students show "no significant gains" in this trait.  The research is based on the College Learning Assessment (CLA), which I've written about before.

I haven't seen the book itself,  only the blog post I cited  and blurb at Amazon.com, which includes this:
Almost everyone strives to go [to college], but almost no one asks the fundamental question posed by Academically Adrift: are undergraduates really learning anything once they get there? For a large proportion of students, Richard Arum and Josipa Roksa’s answer to that question is a definitive no.
Assuming this is an accurate description of the book, that's a pretty bold claim. The description qualifies this a bit by describing the actual skills addressed, including critical thinking, complex reasoning, and writing.  These are left undefined, making one wonder what the difference between critical thinking and complex reasoning is, how complex is "complex" and how do we measure complexity.  But this is a usual practice, repeated here, to use terms that the general readership will associate with something, and then switch to technical results (statistical parameters derived from tests, for example) without drawing attention to the difference between the two.

From the blog review:
A forthcoming book on what students actually learn in college [...] questions whether students at the nation’s colleges and universities are in fact acquiring the skills necessary to compete in the global marketplace. Economists, sociologists, and educators agree that future labor markets will demand sophisticated critical thinking, problem solving, and writing skills from workers, the very skills that a college education is presumed to develop in students.
You can begin to see the shift in meaning in the passage above, establishing some credibility for critical thinking (whatever that is).  In other contexts, this is the set up for a joke.  At a birthday party in Germany once, I saw a humorous skit where one woman wanted to buy a hundred eggs from another, and they talked while the eggs were counted:
One, two..how old is Hans now?  He must be a teenager.
Oh yes, he's thirteen!
Really?  Fourteen, fifteen...  And his sister is at Universität, nicht wahr?
She's already twenty-two, can you believe it?
My, my. Twenty-three, twenty-four
It's funny because the meaning of the numbers shifts between the number of eggs supposedly in the basket and whatever the topic of the conversation is.  The basket feels awfully light at the end.

An analogous ratcheting of meaning escalates the argument in the paragraph quoted from the blog post above. The sleight of hand is hard to see, and it's not revealed until later on.  The heart of the matter is the list "sophisticated critical thinking, problem solving, and writing skills."  These are left undefined, so that they are simply placeholders for what will come later, because CLA scores will be used to pass judgment on these.  In the first instance, general problem solving, for example, means a host of complex activities.  These would include solving differential equations, synthesizing organic molecules, designing logic circuits, inductively finding theories to fit facts, and many, many other tasks that are essential to running a civilization.  Think about how big the domain is for sophisticated critical thinking, problem solving, and writing skills.  It's vast.  By comparison, the CLA is very limited.  The sleight of hand is to substitute the one for the other.

Second quote:
Unlike standardized tests such as the GRE or GMAT, the CLA is not a multiple choice test. Instead, the multi-part test is a holistic assessment based on open-ended prompts. “Performance Task” section prompts students with an imagined “real world” scenario, and provides contextual documents that provide evidence and data. The students are asked to assess and synthesize the data and to construct an argument based on their analysis.
The CLA is a standardized test too, in that it's scored systematically to increase reliability. As such, it's of relatively low complexity, and therefore can be gamed with prep courses. Note that "real world" is in scare quotes.  This may be because the problems can't literally be considered 'real world' under the circumstances: it is a timed, probably low-stakes test with items that are obviously made for the instrument.  Real world scenarios are more like this:
You are an executive at a multinational oil company that has just caused a huge environmental disaster off the coast of the United States.  What should you do?
The post continues with the charge:
The results of Arum and Roska’s research are troubling: 45 percent of the students in their sample demonstrated no significant gains on the CLA between their freshman and sophomore years of college.
After eight months of college classes, the CLA didn't detect an effect for almost half the students.  This sounds serious.  But is the CLA even supposed to be valid on a per-student basis?  My understanding is that different students may not even see the same questions, and that the instrument is for comparing institutions.  The CLA website supports this:
This signaling quality of the CLA allows institutions to benchmark where they stand and how much progress their students have made relative to the progress of students at other colleges.

It's unfair to ask too much technical detail of a blog post, but it's not encouraging to read that:

The average student’s performance on the CLA rose by only 7 percentile points, suggesting that college curricula and instructional strategies are not developing students’ higher order thinking skills.
How can the average performance move at all in terms of percentiles?  The bottom 10% are the bottom 10%.  It may not be the same group before and after, but there's going to be a bottom 10%.  It's not like the range goes from 0-100% into 7-107%--a Lake Wobegon effect.  Does the author mean percentage points?  Percent of what?  How do we put 7% in context?  Furthermore, how do we establish that the beginning level of student "critical thinking" is or is not sufficient to meet the hypothetical global challenge?  As written, this is simply meaningless gobble-dee-goop.  But this doesn't stand in the way of reaching conclusions:
In some ways, Arum says, the results of the study are not surprising. Previous research has shown that the average student spends only 13 hours per week studying, far less time than is spent on social pursuits. Likewise, faculty incentives within higher education (promotion, tenure) are aligned with research pursuits, rather than the quality of undergraduate instruction. According to Arum, this misalignment of goals results in far less attention to teaching and learning than is necessary to cultivate higher order skills in students.
This is all very neat as a presentation, which we can dissect into:
  1. Implied problem: we need to be globally competitive
  2. Elaboration: competitiveness requires critical thinking skills (etc)
  3. CLA measures critical thinking skills (bait and switch)
  4. Change in scores is too low
  5. Colleges are not educating students appropriately
  6. ???
  7. This is caused by two factors: lazy students and unmotivated professors
The sixth step is left as unknown, and I assume more detail is found in the book itself.  Establishing a cause requires work.

The conclusion of the piece is written as a quote attributed to Dr. Arum:
In the future, U.S. higher education will increasingly be held accountable for demonstrating measurable improvement in undergraduate learning.
This isn't entirely logical, since students (in the exposition) bear part of the blame.  Don't we need to hold them accountable too?

It's ironic that a piece about critical thinking has such flimsy foundations.  We can blame the blog format for leaving out information--that's to be expected.  But the basic problem is the disingenuous switch between general and specific about the definition of critical thinking.  That can't be papered over, given the breadth of the argument.

There's another article about assessing critical thinking that came across my desk this week: "Assessing Critical Thinking in STEM and Beyond."  It's about the CAT--an instrument similar to the CLA in that it uses free-form responses.  The big difference, as far as I can tell, is that faculty score the CAT.  It isn't outsourced.  Involving faculty in the assessment process is important.  Especially when felines are involved.

Update:  See "Berry Stein on the CAT" for Dr. Stein's comments that address some of the points I make below.

It's fun to treat the respective articles as source material for a critical thinking exercise.  One point of comparison is the description of what critical thinking actually is.  Recall that this is the hole in the logical exposition of the first article.  From the second:
There is little question that as a result of an increasingly technological and information driven society the ability to think critically has become a cornerstone to both workplace development and effective educational programs.
This is supported with citations that underline how important critical thinking is.  Critical thinking isn't actually defined, but later on a table shows what the test is intended to assess:
  • Separate factual information from inferences that might be used to interpret those facts.
  • Identify inappropriate conclusions.
  • Understand the limitations of correlational data.
  • Identify evidence that might support or contradict a hypothesis.
  • Identify new information that is needed to draw conclusions.
  • Separate relevant from irrelevant information when solving a problem.
  • Learn and understand complex relationships in an unfamiliar domain.
  • Interpret numerical relationships in graphs and separate those relationships from inferences.
  • Use mathematical skills in the context of solving a larger real world problem.
  • Analyze and integrate information from separate sources to solve a complex problem.
  • Recognize how new information might change the solution to a problem.
  • Communicate critical analyses and problem solutions effectively.
This is similar to the CLA (see a description of CLA item scoring here) and these were tested for face validity by asking faculty if they thought the items were representative of critical thinking.  The answer was yes (for the most part).  But even if the list is composed of different kinds of critical thinking, it does not mean that there are not other kinds of critical thinking.  Maybe there are lots and lots of other kinds, so many that generalization from the list is invalid.  In other words, just like with the CLA, good performance on the test might demonstrate ability in some kinds of critical thinking, but that does not mean that the test illuminates general critical thinking skills (it may not even be possible). Both of these papers turn around the logic, going from [My test scores show critical thinking] to [critical thinking always shows up on my test].  This is the logical fallacy called affirming the consequent. In other words, an inappropriate conclusion.
 
So while it is a wonderful idea to provide guidance and resources for integrating into coursework the types of thinking skills assessed in either test, I don't see what value lies in attempting to generalize the findings to some "critical thinking" parameter that is probably so general that it's meaningless.  Why is that even needed?  It's a farce that couldn't be maintained long in any case.  Consider this quote from the CAT paper:
Virtually every business or industry position that involves responsibility and action in the face of uncertainty would benefit if the people filling that position obtained a high level of the ability to think critically.
What skills are most useful if you're dealing with uncertainty?  I would say that a solid understanding of probability and in particular Bayes' Theorem would be essential.  Also, a solid understanding of how computers work, and the kind of logical thinking that is required to debug programs is very valuable in many domains, like figuring out why the stock market fell out of a clear blue sky.  Much of the time uncertainty means considering branching possibilities, and the systematic investigation of that kind of structure is within computer science.  There are hints at these types of thinking in the bullet point list above (the one about correlation, for example), but the notion of conditional probability isn't directly addressed.  In fact, it's interesting to note that in the CAT piece, the face validity survey revealed that:
The question with the lowest overall support (81.2%) involved using a mathematical calculation that was needed to help solve a complex real-world problem.
The point here is that there is already a mismatch between the quote about the need for critical thinking (to address uncertainty) and the skills listed that are supposed to assess critical thinking.  This is a symptom of the over-generalization.  Another example of this is the reference in the earlier quote to a "technological and information driven society," but no obvious correspondence to the skills list.  How about understanding how computers and algorithms work?  Data mining?  Everyone should know how to use a pivot-table to disaggregate data dimensions, read and create graphs (including log and log-log scales), etc. And while we're at it, what about non-cognitives like identifying confirmation bias in your own conclusions? 

The effects of the unwarranted generalization are gross distortions of the value of education.  The CLA takes this to absurd lengths, and sets the stage for a "create a problem, sell the solution" business model, which I've mentioned before.  The CAT seems much more useful, since faculty actually see and evaluate the results, and the scope seems broader (NB: I've not seen either instrument in its entirety).

The odd thing about the CAT is that the main reason for using a standardized instrument is to be able to compare results to other institutions'.  But this probably isn't reliable without outsourcing the scoring.  I would broaden the scope of CAT, and rather than see it as a test instrument, view it as a starting point to customize a general education curriculum through structured inquiry with faculty.  I don't mean deciding all over again what structure and what courses are required (shudder), but rather what kinds of thinking we expect students to demonstrate.  I don't know of a similar program, and the CAT could be a wonderful launch pad for that. 

But thinking, critically or otherwise, is broader than arguments and problems.  Thinking is what shapes who we are as human beings, and I believe that a liberal education should include context as well as process.  The other day Epsilon and I walked to the store.  I pointed down to the divisions of the sidewalk as we stepped over them, and told her each one is a hundred million years.  Here's where the planet formed, about 4 billion years ago.  Point to where you think humans came along.  So we walked and talked.  We walked over 35 divisions, almost around the corner, before getting to the Cambrian Explosion and multi-cellular life.  We had a little celebration there.  We paused to mourn the world's loss during the Permian Event, and finally arrived at the last two concrete rectangles, where everything she was interested in happened: dinosaurs, lasting a whole slab, the K-T gap that ended their career and launched ours, and that last inch or so where humans made an appearance.

This sort of knowledge isn't good for making arguments in board rooms or figuring out how to stop oil spills.  But having a context is no less important.  To know that the carbon atoms we borrow for a while--that then cycle on to be replaced in the combinatorical dance that animates us--that those atoms were coughed out by ancient dying stars and drifted through space for unimaginable stretches of time before playing their current role.  To understand the bizarre nature of the physical world at the micro and macro level, to understand the limits of logic itself.  Those represent important milestones in our evolution as humans, and simultaneously exalt and humble us.  Do they prepare one for the global marketplace?  I can't prove it.

To be more concrete about some big holes in both the CLA and CAT lists, let me give some examples of other thinking skills that are as important in my opinion.

1. Understanding exponential growth


2. Understanding conditional probabilities


3. Understanding The Prisoner's Dilemma and Tragedy of the Commons

4. Understanding the consequences of evolution through natural selection



In summary, the premise of the CLA, the disingenuous generalization of results, and the strong charges alluded to in the blog post and Amazon description of Academically Adrift do not leave a good first impression.  My first look at the CAT through the eyes of the authors of "Assessing Critical Thinking..." still seems over-reaching, but is more interesting to someone who actually needs to get meaningful assessment done.

Tuesday, November 10, 2009

Numbers and Names

Words have meaning. This is true even if the words are not formally defined; people could talk to each other before dictionaries came around. The facility to speak and understand is so fluid in fully-functioning humans that we underestimate how difficult it is (see Moravec's Paradox). Undoubtedly there was strong evolutionary bias toward creating this ease of communication, as opposed to making our brains facile with long division, for example.

Because words are powerful, they get hijacked. There is economic value attached to the effect of certain words, like "new and improved" and so they get put into use like blue-collar workers marching off to punch in. Sometimes this is manipulative or cynical, as brilliantly illustrated by Orwell in 1984. In the Russian revolution, Bolsheviks were pitted against Mensheviks, names stemming from a narrow vote. The former word means "majority" and the latter "minority." Imagine if your political faction is saddled with with the second name...

In outcomes assessment, or reporting out psychometrics in general, the use of common words is sloppily introduced. I've addressed the big one: "measurement" elsewhere. Another good source for this kind of error propagation is studies that use factor analysis. I came across a good example while reading "A Look across Four Years at the Disposition toward Critical Thinking Among Undergraduate Students" by Giancarlo and Facione while browsing Insight Assessment's research page. This company produces the Critical Thinking Dispositions survey I blogged about here. I don't mean to be critical of the authors, but rather highlight a practice that seems to be endorsed by most who write about such things. The study itself is interesting, giving a before-and-after look at undergraduates as assessed by the survey. They introduce the topic of dispositions thus:
Any conceptualization of critical thinking that focuses exclusively on cognitive skills is incomplete. A more comprehensive view of CT must include the acknowledgement of a characterological component, often referred to as a disposition, to describe a person’s inclination to use critical thinking when faced with problems to solve, ideas to evaluate, or decisions to make. Attitudes, values, and inclinations are dimensions of personality that influence human behavior.
Notice the implication that personality comes in dimensions. Dimensions are by definition independent of one another, and as we shall see, the idea is that we can assemble a linear combination of these pieces to assemble a whole disposition. This is an enthymeme without which the rest of the analysis cannot proceed, but it's a big leap of faith. As such, it ought (in the research community) to be spelled out explicitly. The mindset that attitudes and values and inclinations together create some kind of vector space is so wild that you'd think caution would be advised. If the implication is that these dimensions really are orthogonal (completely independent of one another), it's ridiculous on the face of it. What does it mean to have a very small amount of "attitude" but lots of "inclinations?"

Most things are not linear. If I'm talking softly, you may only hear bits and pieces of what I hear. Increasing the volume will enable you to hear me clearly within a range, but we wouldn't be so bold as to say "talking twice as loud makes you understand me twice as well." We use linearity not because things are linear but because it makes it easy to do the analysis. In small ranges, it often makes sense to approximate non-linear phenomena with linear models, but one has to be careful about reaching conclusions.

In the article, the assumption is that the disposition to think critically is the linear combination of a few component dimensions. These are listed:
Factor analysis of the CCTDI reveals seven distinct elements. In their positive manifestation, these seven bipolar characterological attributes are named truthseeking, open-mindedness, analyticity, systematicity, critical thinking (CT) self-confidence, inquisitiveness, and maturity of judgment.
Notice the passive voice "are named." Are named by whom? Here's the process: The survey is administered and the results recorded in a matrix by student and item. A correlation matrix is computed to see what goes with what. Then a factor analysis (or singular value decomposition, in math terms) is performed, which factors the matrix into orthogonal dimensions. To understand this, it helps to look at an animation of a simple case. If the dimensions have different "sizes" (axes of the ellipse in the animation), then a more-or-less unique factorization results. If the dimensions are close to the same size, it's hard to make that case. Each dimension is defined by survey items and associated coefficients. It supposedly tells us something about the structure of the results. Note that orthogonal means the same thing it did earlier: completely independent. You can have zero of one factor and lots of another, and this needs to make sense in your interpretation.

So: we do some number crunching and find associations between items. These are collected together and named. We could call them vector1, vector2, and so on, but that wouldn't be very impressive. So we call them "openmindedness", "attentiveness," and use words that already have meanings.

It's not even clear what the claim actually is. Is it that we humans perceive critical thinking dispositions as a linear combination of some fundamental types of observation, presumably presented to us in whole form by our perceptive apparatus? Or is it that in reality, our brains are wired in such a way that dispositions are generated in as linear combinations?

It would be relatively easy to test the first case using analysis of language, like the brilliant techniques I wrote about in "High Five." I don't see any evidence that this sort of thing is done routinely. Instead, researchers eyeball the items that are associated with the dimensions that pop out and give them imaginative names. They may or may not be the same names that you and I would give them, and may or may not correspond to actual descriptions that someone on the street would use to describe the test subject.

I hope you can see the sleight of hand by now. In the case of this particular article, the authors go one step further, by describing in detail--in plain English--what the dimensions are (I have bolded what was underlining in the original):
The Truthseeking scale on the CCTDI measures intellectual honesty, the courageous desire for best knowledge in any situation, the inclination to ask challenging questions and to follow the reasons and evidence wherever they lead. Openmindedness measures tolerance for new ideas and divergent views. Analyticity measures alertness to potential difficulties and being alert to the need to intervene by the use of reason and evidence to solve problems. Systematicity measures the inclination to be organized, focused, diligent, and persevering in inquiry. Critical Thinking Self-Confidence measures trust in one’s own reasoning and in one’s ability to guide others to make reasoned decisions. Inquisitiveness measures intellectual curiosity and the intention to learn things even if their immediate application is not apparent. Maturity of Judgment measures judiciousness, which inclines one to see the complexity in problems and to desire prudent and timely decision making, even in uncertain conditions (Facione, et al., 1995).
These descriptions would serve suitably for ordinary definitions of ordinary terms (without the use of "measurement"), but no evidence is presented that the ordinary meanings of all these words corresponds in any way to the factor analysis results, other than that someone decided to give the dimensions these names. The final touch is claiming that we "measure" these elements of personality with precision:
For each of the seven scales a person’s score on the CCTDI may range from a minimum of 10 points to a maximum of 60 points. Scores are interpreted utilizing the following guidelines. A score of 40 points or higher indicates a positive inclination or affirmation of the characteristic; a score of 30 or less indicates opposition, disinclination or hostility toward that same characteristic. A score in the range of 31-39 points indicates ambiguity or ambivalence toward the characteristic.
All of this strikes me as absurd. It's not that surveys can't be useful. To the contrary, they undoubtedly can give us some insights about student habits of mind. But to suppose that we can slice and dice said behaviors with this precision is far over-reaching, particularly in the use of ordinary language to create credibility without proof that these associations are strong enough to withstand challenge.

This practice is unfortunately common. The NSSE reports include dimensions like this, for example.

Thursday, November 05, 2009

Habits of Mind

On Tuesday I wrote in "Learning and Intelligence" about the difference between having the ability to think rationally being different from the habit of using said ability. The source was a fascinating NewScientist article.

Yesterday in an article in InsideHigherEd, Peter Facione wrote in a comment that:
The California Critical Thinking Skills Test (CCTST), used by hundreds of colleges and universities throughout the US and worldwide, has quietly become one of the leading measures of critical thinking skills. Its companion tool, the California Critical Thinking Disposition Inventory (CCTDI), assesses the habits of mind which incline one to use critical thinking in real world problem solving and decision making. Research shows that having the skills to think well and having the disposition to use those skills in key judgment situations is not highly correlated and yet few educational settings are taking this into account.
I do not remember reading about the CCTDI before. Their website describes the instrument as:
The California Critical Thinking Disposition Inventory is the premier tool for surveying the dispositional aspects of critical thinking. The CCTDI invites respondents to indicate the extent to which they agree or disagree with 75 statements expressing beliefs, values, attitudes and intentions that relate to the reflective formation of reasoned judgments. The CCTDI measures the "willing" dimension in the expression "willing and able" to think critically. The CCTDI can be administered in 20 minutes.
You can find a list of abstracts for research using the CCTDI here. There are some interesting bits to read there, like:
Significant differences were detected in critical thinking disposition (CCTDI) between the two groups of students, Hong Kong Chinese students failing to show a positive disposition toward critical thinking on the CCTDI total mean score, while the Australian students showed a positive disposition. The study raises questions about the effects of institutional, educational, professional and cultural factors on the disposition to think critically. [Tiwari A, Avery A, Lai P. (2003)]
You may recall that the CIRP is also trying to do this using item response theory to estimate a dimension called habits of mind. It's described on the HERI website as "Interactions with students that foster habits of mind for student learning."

It seems to me that this an opportunity to change the discussion about general education, using these ideas. If disposition is indeed different from ability, then perhaps the marination of a student for two years in a survey courses ought to be focused on developing habits of mind more than trying to assemble a skills list (like critical thinking). Or in addition to it, if you're a glutton for punishment.

A student who completes a solid degree program is going to come out of it with real analytical and creative skills he or she didn't have before. But the way the curriculum generally work, the way our learning outcomes are sketched, I don't think it's common to address this other dimension: intentionally developing open-mindedness, truth-seeking, systematicity, and maturity, as the CCTDI gets factored.

Thursday, October 29, 2009

Critical Thinking Buzz

Yesterday I promised to revisit the different points of view about critical thinking evidenced at the Assessment Institute. This is a big topic, and I've babbled about it before (here). It's important because so many institutions plug it in as a learning objective. It's especially dear and near to the circulatory organs of liberal arts schools.

First, I have some bad news. Just look at the graph of search hits from google trends to unveil the sad tale.
The story is clear: searchers are becoming less interested in thinking and more interested in ignorance, with the latter spiking in about June 2009. In case you think student engagement might be riding to the rescue, the sad little red line at the bottom says otherwise. Interestingly, searchers seem to lose interest in either aspect of cognition around the holidays. After much research, I discovered the reason behind this, shown below.

Okay, maybe that's enough silliness :-)

Here's what I heard about the subject at the conference. Three things. First, as I mentioned yesterday, the opening plenary revealed a divide among the panelists when the topic came up as a part of a wider conversation. I was writing furiously and am not sure who to attribute this statement to:
There is no evidence that there are generalizable skills like critical thinking. You have to master one domain first.
This echoes my own thoughts and classroom experiences on the matter, which I've described previously. In a nutshell: developing significant ability to do deductive reasoning is a prerequisite to doing interesting inductive reasoning. If this is true, major programs are more likely to cultivate complex thinking skills than a broad curriculum like general education.

Other panelists disagreed, but there wasn't time to have a proper debate on the issue.

The second encounter was in a session about integrated assessment, where gen ed-type skills like communicating and thinking are threaded throughout the curriculum. Critical thinking was explicitly one of those. During Q&A I offered my own (heretical, I suppose) thoughts on the topic and was rather sternly admonished that the problem had been solved by psychologists and all I had to do was look in the literature. Heck, that may be true, and I've started looking. I'm a math guy after all, not a psychometrician. But if there was overwhelming theory and empirical evidence for a particular model, why is there still debate? Is it like Darwinian evolution, where some simply reject it because of dogma? I doubt it, but I'll try to find out. In the meantime, color me dubious. I asked the session speaker for some references yesterday by email.

I did find an accessible article by Tom Angelo, who was on the plenary panel, called "Beginning the Dialogue: Thoughts on Promoting Critical Thinking." It was published in 1995, and the opens by saying about critical thinking that "Despite years of debate, no single definition is widely accepted." This was actually confirmed in the session I mentioned, where it was taken for granted that critical thinking in an English course is different from in a Math course. This by itself isn't fatal to the idea; after all, writing is different too, but we can try to teach writing across the curriculum. But the subtleties are important.

For one thing, on a very basic level, I can watch Tatiana write a paragraph and say with confidence "this student did some writing." Evidence of thinking is different. I can look at that same paper and try to imagine what went on inside her head when she wrote it, but I can't really know that she was thinking at all. Maybe she put random words on the paper, or quoted something she had memorized. There's an empiricism gap. I can count words written. Quantifying (even with a binary yes/no) critical thinking is not so straightforward. To push that a bit further, imagine taking a piece of work from an English class and sticking it into a stack of math papers being graded. The math prof squints at it and wonders what the heck this is. Can the prof then pronounce whether or not critical thinking has taken place in the English class by inspection? Or is he/she only competent to judge in the domain of math? Reverse the situation. An English instructor sees a complex page of handwritten formulas and text, purporting to settle the Continuum Hypothesis once and for all. If you don't have technical expertise in an area, it's virtually impossible to judge what level of thinking has occurred. But maybe that's not what we mean. Here's a definition Dr. Angelo likes in the article, quoting Kurfiss:
[Critical thinking is] an investigation whose purpose is to explore a situation, phenomenon, question, or problem to arrive at a hypothesis or conclusion about it that integrates all available information and that can therefore be convincingly justified. In critical thinking, all assumptions are open to question, divergent views are aggressively sought, and the inquiry is not biased in favor of a particular outcome.
Allow me to point out a couple of things here. First, the creation of a hypothesis is (if true) the creation of new knowledge. And because we want it to conform to facts and hold up to new evidence that arrives, this is very similar to inductive reasoning. It sounds like the scientific method. But in order to do any kind of inductive reasoning, you have to have some knowledge of the deductive processes active in that domain. You can't write a math proof without knowing propositional logic; you can't solve a problem the rudder on your 747 unless you know how the thing works; you can't create complex financial leveraging instruments unless you understand the risks. Well, maybe that last one was a bad example.

One curious aspect of critical thinking assessment is that although the language surrounding it mentions all kinds of desirable habits of mind, higher ed kind of dodges the issue. Maybe there's some institution out there that really tackles teaching of self-assessment, open-mindedness, thoroughness, focus, and so forth. I'd love to know. Outcomes like what a student wrote on a piece of paper is far downstream from the actual events that led to its creation. There are some surveys that try to assess habits of mind--the CIRP for one. Who's trying to teach them that outside of orientation class? There is therefore a curious disconnect between our actual desired outcomes and what we teach.

I think it's healthy to ask another level of why. Why do we want students to think critically? We get stuck in our own curricular bubbles, perhaps. Let's step outside for a moment. What is gained if critical thinking--however you define it--is employed?

Is it because we want active citizens? Or we want good problem-solvers? Or we want entrepreneurs? Whatever the answer is, it's likely to be more easily pinned down than the amorphous one of critical thinking. We can actually look at evidence of citizenship or problem-solving. We can create programs and curricula to address them. And don't forget the habits of mind thing--for my money, that's an untapped vein of gold, and also amenable to assessment. At the right point in the report cycle, the assessment director can still aggregate the heck out of all the types of "critical thinking" on record and serve up a glop of statistical goo to the admins. Just put the graphs in color--that means more than any data you put on there. And be sure you put the right logo on the thing.

The third encounter with critical thinking I had was indirect; I heard about it through a conversation. What I was told was that on a Tuesday session U. Phoenix presented served up "Assessment Methods: Creating a Critical Thinking Scoring Instrument as a Tool for Programmatic Assessment." I can't find the slides online, but I'll try to get them, and ask the presenters what happened. Apparently there was essentially heckling of the presenter(s) about the definition of critical thinking, and my companion's assumption was that this was related to the fact that it was U. Phoenix presenting and not, say, Alverno College. If so, this is a sad irony. If a group of professionals gather to talk about critical thinking and don't actually demonstrate the ability to do it, where are we? I will suggest to the organizers that they replace iced tea refreshments with Scotch next time.

Wednesday, October 28, 2009

The 2009 Assessment Institute

I'm just back from two full days in Indianapolis at the Assessment Institute. The trip was great. I made contacts and learned some really interesting stuff. Too much to summarize this morning, but here are some bits and pieces to be filled in later.

The opening plenary was shortened by an unfortunate medical incident, but hinted at some interesting fault lines. Trudy Banta, the organizer, does a good job of representing different points of view with her choice of panel, guest, and topic.

The first divergence I noticed centered on the idea of "tuning," a concept borrowed from the Bologna Club in the EU, which I blogged about here and here (the update at the bottom). It's not a lunch meeting, as you might think, but a process of comparing and improving programs across institutions. As a point of trivia, we learned from Jeffery Sybert that concert A is 440 Hz in this country, but 442 Hz in Europe. I'm quite sure I couldn't tell the difference, which is in any case greater than the intra-tuning dissonances of an equi-tempered scale. There must be an interesting story there. The topic was presented by Jamie Merisotis, President/CEO of Lumina Foundation for Education. You can read more about the project here. The panel dissonance in this case was fairly minor, turning on the question of how much authority faculty should have in the process. On the one hand, faculty own curriculum. On the other, they can be protective, self-interested, and unduly academic in their horizons (speaking as one). This is a good case of needing external reviewers to constantly check that goals and progress align. (Such as a stakeholder analysis: see below.)

There was the admonition from the panel that for-profits and their venture capitalist underwriters are very interested in producing a meaningful educational product, and by implication that the usual plodding change of bricks and mortarboard institutions won't be fast enough to compete with cyberdon. As a riff on that, George Kuh suggested that "the credit-hour is a dying concept in higher education, measuring things we no longer value." He elaborated, equating seat time with credit hours under the traditional system.

The other twanging note in the plenary was caused by the use of the term "critical thinking." The consensus position is that this skill is valuable, should be taught and "measured," etc. In opposition to that is the idea that maybe critical thinking is too fuzzy to actually be useful as a learning objective. This turned out to be a minor theme at the conference, which I'll elaborate on tomorrow. In a Q&A at one session, I was given an admonishing mini-lecture that psychology had solved the critical thinking problem and that I only needed to look at the literature. In a session with University of Phoenix that I missed, I'm told that there was rude and contentious debate about assessing critical thinking. Maybe someone who was there can comment.

I met the creator of Waypoint, which I had blogged about here. I watched a demo of the online rubric management and implementation software. More on that later.

At lunch on Monday, Jon and I chatted with an Assistant Director of CIRP at HERI [edit: fixed title] who works on CIRP. I've used the freshman survey, and found it quite useful in finding attrition trends (see this post), and I was interested to learn that CIRP is getting into the constructs business, using item response theory. I went to the session on that on Tuesday, and have some other comments that will have to wait until I have more time.

A session by Tom Zane at Western Governors University gave a fascinating insight into their system of assessing massive amounts of student work. You can read about their innovative system of using assessments to entirely replace grades here. They get between 1500 and 2000 new students each month, and are still growing rapidly. Student work is assessed by human raters using rubrics--this isn't standardized test land. Samples are rated more than once to test for reliability. Although this is still monological, my first impression is that this is as good as it gets for traditional assessment, and that the assessment side of their business could plug into this model if WGU chose do to it--providing a uniform system of credentialing for higher ed. Something like that is conceivably in our future. That might sound scary, but it's infinitely better than standardized tests running our lives.

AAC&U has a project called VALUE, which accumulates rubrics that they have found or created and refined. This seems useful, and complementary to the tuning idea. The next part of the project is to create a repository of student work that has been rated using the rubrics. A problem I noticed more than once is that designers usually don't seem to think much up front about whether their rubric is relative to the curriculum or fixed absolutely. When we built the FACS model, we used an absolute scale, in which raters say things like "student Ecks is working at the freshman/sophomore level." In a relative scale you get stuff like "exceeds expectations." The former is great for tracking longitudinal progress, the latter not so much. A good student will exceed expectations in all classes, showing no progress. By contrast, even a great freshman math major is very unlikely to be doing senior-level work. I'm not sure the VALUE leaders have addressed this.

Jon Shannon and I led a 75-minute session on stakeholder analysis in strategic planning, which had good participation. I blogged about the topic here, and you can find the presentation linked here. This is a great tool for addressing complex planning issues. One of the advantages is that it keeps the conversation on track, focused on goals everyone more or less agrees on.

The backchannel on twitter was pretty thin, or else I just didn't hit the main vein with my search. We set one up at Today's Meet and advertised it on one of our slides, but haven't gotten any activity from participants.

More on some of these topics later.

Monday, October 26, 2009

Impedance Mismatch

There's an article about NSSE in InsideHigherEd this morning about the "Wabash National Study," which studied changes in freshmen over their first year of liberal arts education. Their summary begins:
In our analysis of data from 3,081 students at 19 institutions in the first round of the study we found that, on the whole, students changed very little on the outcomes that we measured over their first year in college.
The provide an overview of findings (pdf) that give more details. In particular:
[A]lthough students’ improvement on the CAAP Critical Thinking test was statistically significant, the change was so small (less than 1% increase) that it was practically meaningless.
You can find two examples of the sorts of questions that the CAAP employs here (pdf). One asks a question about polling, referring to a hypothetical politician named Favor:
Favor's "unofficial poll" of her constituents at the Johnson County political rally would be more persuasive as evidence for her contentions if the group of people to whom she spoke had:
I. been randomly selected.
II. represented a broad spectrum of the population: young and old, white and non-white, male and female, etc.
III. not included an unusually large number of pharmacists.
The results of the CAAP and other surveys (such as student attitudes and behaviors) were correlated against six "teaching practices and institutional conditions:"
  • Good Teaching and High-Quality Interactions with Faculty
  • Academic Challenge and High Expectations
  • Diversity Experiences
  • Frequency of Interacting with Faculty and Staff
  • Interactions with Peers
  • Cooperative Learning
The summary of results shows that "critical thinking" component was correlated positively and significantly with "good teaching" and "diversity experiences."

I'm not a fan of the status quo in general education, but it does seem only fair that if we're going to judge accomplishment using tests that the tests align with the curriculum. Perhaps at the participating institutions, this is the case, but I have trouble seeing where items like those in the critical thinking part of the CAAP are actually used in first-year liberal arts curricula. Certainly, many schools have critical thinking listed as a goal, but it gets defined in many ways. I don't like the term because of its fuzziness, and this example shows that well.

Take the example given about poll sampling. The answer can be arrived at by a bit of common sense, in which case this resembles perhaps an IQ test, or one might have encountered sampling in a Finite Math course or Intro to Psychology. But it's not exactly the level of material that students come to college for. In finite math, they might learn linear programming, which is a fairly complex analytical tool used to solve constraint problems. Of course, anyone who hasn't had that material would fail miserably.

But isn't that the point? By trying to "measure" (see Measurement Smeasurement for an explanation of the scare quotes) only the least common denominator of freshman learning--a prerequisite to standardization--aren't we really just applying an IQ-like test? It would be better to use targeted tests that correspond to the actual curriculum that students actually take, rather than imagining that can all be treated uniformly. Put another way, we shouldn't be surprised if students don't perform well on tests that don't correspond to what we've actually taught them.

Other posts on critical thinking are listed here.

Tuesday, October 20, 2009

Absurdities

I went to a workshop yesterday on rules I should know and came back with three pages of notes and two pages of ideas about a research project. It was a nice drive too. Mrs Muffet (our GPS, who sits on a tuffet) took me back home by way of a mountainous stretch of I-40, which was more excitement than I needed without coffee or a nap to fortify me. I made some new acquaintances, which is always the highlight of such trips. Getting there was a trip in itself, logically. It went like this, in a series of emails to the organizer, who I'll call Mrs. K:
Me: Hi. Our copiers are being replaced, and I can't fax the form, so my information is enclosed.

Mrs. K: Okay, but I need the fax.

Me: Our machines are still wonky. I've scanned the completed form and attached it as pdf.

Mrs. K: Can you send me the fax?

Me: I was hoping you could just print the pdf...?

Me (day before the meeting): Hi. I haven't got confirmation yet. Am I on the list?

Mrs. K: If you send me the fax we can get you on today.
I sent the damned fax.

While I was driving I amused myself with a riff on a topic from an ASSESS post. Someone had asked about measuring the consequences of values and beliefs. The discussion was very useful I'm sure, but the use of 'measurement' in assessment makes me itch, and I imagined the following dialogue at an imaginary Lenin School of Management:
Stani: How did I do on my test?

Vlad: Ah...not good at all, I'm sorry to say.

Stani: What happened???

Vlad: You don't believe in the precept that the means justifies the ends.

Stani: Why would you think that?

Vlad: We measured your belief. Less than 10%: shameful.

Stani: But I do believe that the ends justifies the means.

Vlad: No, you don't. You believe that you believe it, but you don't actually believe it.

Stani: Your measurement shows that too?

Vlad: Verily.

Stani: And you believe these measurements?

Vlad: Of course. Here is my score--see: 100% belief in the measurements.

Stani: But what if the measurements are wrong?

Vlad: I don't believe the measurements are wrong. I showed you my score.

Stani: But if the scores are wrong, then the measurements don't mean anything,
regardless of what you believe.

Vlad: I don't believe that to be the case.

Stani: So if the scores told you that you believed in the tooth fairy, you would believe them, even if you don't believe in the tooth fairy?

Vlad: I can believe that I believe in the tooth fairy without believing in the tooth fairy. Just like you can believe that you believe that the ends justify the means, even if you don't really believe it.
This seems rather like a Monty Python script. On the other hand, there is a lot of believing in the scores. As evidence, I present iCritical Thinking(tm) test, which aims to solve this problem:
Today's academic and professional environments demand more than current Internet and computer skills. They also require students and employees to navigate, critically evaluate and make sense of the wealth of information available through digital technology.
In college, we call it information literacy. At least, this is pretty close, although it involves using office applications to negotiate "scenarios." This isn't ambitious enough to market, apparently, because the presentation on the website blurs the lines between nuts & bolts online skills and "critical thinking," whatever that is. The marketing material uses quotes like this one attributed to Pres. Obama (you can find the original here):
The solution to low test scores is not lower standards; it's tougher, clearer standards... I'm calling on our nation's governors and state education chiefs to develop standards and assessments [that measure] 21st century skills like problem-solving and critical thinking..."
Another one targets employers, and uses a quote from The Economist (Oct. 5, 2006):
The nature of the economy is changing. It's putting more and more premium upon intellectual skills, analytical skills, creative skills, which are in short supply.
The implication is that this company can certify that you have these valuable and complex skills. There's plenty more, including the usual claims of impeccable validity (for what isn't exactly addressed; see "Questions of Validity"). Saying a test is valid is like saying a tool is useful. Useful for what? Changing the oil filter or pounding nails? Or just weighing down that big stack of meeting minutes on your desk? The construction of the test material is likewise celebrated (here):
The development of the Global Standard 3 took over 9 months of research, data collection from 400+ subject matter experts from over 30 countries and final ratification from the members of the Global Digital Literacy Council. By going through this rigorous process it ensures that Global Standard 3 is not only current and relevant, but a true global digital literacy standard.
So this truly is a one-size fits all test, conveniently administered in an hour, that can be used from high school through career path, through the entire world of 6000+ spoken languages and innumerable cultures, to assess digital literacy (but we want you to think critical thinking).

This is of the same species as the "measuring belief" absurdity above, and represents an insidious problem. The song goes like this (to the tune of "Goodbye Yellow Brick Road"):
  1. There is increasing demand for certification of skills due to the diversity of educational experiences. See "Scaling Higher Ed" for more.

  2. There is also increasingly desperate and more insistent demands from the government(s) for "measurable standards."

  3. Beyond low-complexity skills, it is difficult or impossible to construct valid instruments that are very general in nature (i.e. amenable to standardized testing). For broad-spectrum skills like critical thinking, validity is a slippery concept.

  4. Apply Zog's Lemma--where error with economic value exists, it will be sold. Who better to do that than standardized test makers?

  5. Apply something like Gresham's Law: bad assessments drive out good assessments. This amplifies the largeness of the mistake by industrializing and institutionalizing tests and scores of dubious validity. It creates a monological definition of what "thinking" is, viz., "thinking" means a score on a standardized test. Think of the outsized value placed on SAT scores.
You can see all of these except the last one evidenced in the iCritical Thinking materials. There's little in the way of modesty about the claims. And who's to prove them wrong? Certainly, the business model is a good one: the next thing that comes along is "prep materials" for the test (see "Closing the Loop CLA-style"), and possibly end-to-end control of the curriculum, coaching, and test.

There are already plenty of information literacy programs and tests out there. TILT (now under re-development, apparently) is the one I've heard librarians mention the most. But it doesn't come with slick marketing materials that claim ultimate relevance.

I have to wonder what professional psychometricians think about all this. This industry is theirs, the unfortunate language of "measuring" mental traits is theirs too. There are serious attempts to come to grips with some of the fundamental problems, like Bennett and Hacker's Philosophical Foundations of Neuroscience. And there are plenty of academics in the field who work hard at validity and don't inflate claims. But where is the general hue and cry, the expert advice to lawmakers that, no, we don't really measure minds, and any standardized test is bandwidth-limited simply because it's standardized. I don't see it. Maybe someone can point me in the right direction.

(photo from this site)