Don't Drink the Science

I've been cataloging dangerous things. There's no shortage: airplanes, scorpions, methamphetamines, cliffs, rumors, cigarettes, running up mountains too fast, people to whom you're uncomfortably attracted, George W. Bush, over-analyzing, rancid peanuts, and sometimes, when rabid, squirrels. But nothing is more dangerous than a little bit of science.

Speech-language pathology used to be an art. Then some hapless clinician contracted science and the practice was convulsed. People began heaving up theories and wheezing about evidence-based practice. There were fits of data collection, rashes of empirical impulse. And then there were Measurable Objectives. Bertram will independently answer who, what, and where questions with 80% accuracy during structured activities over two consecutive sessions. Rosalia will, given verbal cuing, use her communication book to sequence three-word, subject-verb-object sentences with 80% accuracy over two consecutive sessions.

In theory, all of this is great. Prior to the Great Scientific Fever, speech-pathologists were in no small sense modern witch doctors, basing their decisions on a combination of Instinct and Ancient Wisdom. Science has done a lot for speech pathology, providing evidence for practices that work and helping to disprove practices that don't.

But a little science, misapplied, is like a handsome, drug-addled squirrel run amok. Case in point: Measurable Objectives. In addition to requiring the SLP to construct clunky, syntactically-overstuffed sentences, Measurable Objectives, at least as they are utilized in the field, arise from a fundamental misunderstanding of the nature of statistics. And that, make no mistake, is dangerous.

I think statistics are fabulous. They are numbers made into rope, tying up the world in neat(-ish) little packages. They are math pulling its head out of its behind. But statistics aren't magic. They require careful, knowledgeable handling. I don't know that much about statistics, but even I can tell that right now the field of clinical speech-language pathology is shipping them to Timbuktu in a milk crate.

Misapprehension #1: If something is true for a population, it's true for each individual in that population. Measurable Objectives assume that whatever intervention prompted statistically significant gains in a population of children or adults will prompt the same gains in each individual client. But statistics are about groups and trends. There's no way of assuring that Bertram or Rosalia will make comparable progress.

Misapprehension #2: Percentages are percentages are percentages. Know how many observations of a target behavior an SLP has time to take in a session? Maybe five. So on Tuesday, Rosalia used her communication book accurately 3/5 trials, or 60% of the time. On Thursday, Rosalia used her book 4/5 trials, or 80% of the time. Aha! the SLP will say, Rosalia has met her goal! But in reality, we're only talking about a difference of one observation. And one observation does not a trend make!

Misapprehension #3: Progress means progress. There is a way to conduct valid single-subject research. It requires multiple baselines, a theoretically-unrelated control behavior, and lots of observations. Asking Bertram 10 why questions on Tuesday and then asking him 10 why questions on Thursday isn't it. Maybe he hit is head on Tuesday. Maybe his brain grew a few more neurons over the course of the week. Maybe he's distracted by the disturbing nerdiness of his name. Something's being measured, but Lord knows if it's actually progress.

What happens in the real world is that speech-language pathologists, required to construct and take data on Measurable Objectives that are effectively impotent in their ability to describe what's actually going on, look for loopholes. I've seen it in action at team meetings. What's Bertram doing now? someone will say. Let's write that in the goal. After all, we have to make sure he can meet his goals.