Transcending Testimonials and Impossible Chickens

There is a Sufi tale about an illiterate peasant who found a very large and flawless pearl in an oyster he was eating. He was delighted and immediately took it to the marketplace and sold it. When asked what price he got, the peasant replied proudly, "One hundred dollars."

"But why didn't you ask for more?" someone asked.

The peasant looked puzzled. "Is there a number larger than 100?" he asked.

For much of its history, chiropractic has used individual patient testimonials rather than controlled scientific research to illustrate its efficacy. I believe that this practice has constituted a self imposed limitation which, like the peasant in our Sufi parable, was partially due to a misconception about what single case reports actually mean.

Because individual testimonials are often vivid, unusual, emotional, or personally relevant to the audience, they are easy for humans to remember; and because they are more available in memory, testimonials create a disproportionately large effect on the informal judgements human beings make, generally much more than single cases ought to. One memorable testimonial can, in the mind of the beholder, outweigh a thousand counter-examples presented only as colorless numbers. Testimonials work, then, partly because of our own mental limitations, because without the benefit of formal (scientific) methods, we are able to perceive, imagine and remember vivid single instances better than large numbers of events.

It is true that reports of very unusual individual cases may also occasionally serve the scientific process by uncovering instances which are not just rare but which are actually impossible or contradictory, relative to a specific hypothetical model. For instance, it would take only one chicken continuing to live after having its head cut off to seriously challenge the hypothesis that chickens require heads to live. An exception which is impossible or contradictory does not "prove the rule," it disproves it. To prove the hypothesis that all chickens need heads would take an infinite amount of time and effort and chickens. To disprove it requires only one impossible chicken, i.e., one which lives without its head.

In the past, individual testimonial reports of chiropractic efficacy may have seemed like "impossible chickens" to many. But this is only true if you play such reports against the implied hypothesis that absolutely nothing but medical and surgical interventions can help people get well. Viewed in that absolute context, a single genuine anecdote or testimonial of chiropractic effectiveness would, indeed, be equivalent to an impossible chicken, and would disprove the hypothesis that only medical or surgical interventions are effective.

But does anyone really believe that? There is a difference between having a political bias toward this position (as may be held by the AMA, for example) and holding it as a genuine scientific hypothesis. Given the continuously changing nature of science in general and health care knowledge in particular, it is unlikely that anyone in recent history has been foolish enough to seriously entertain the scientific hypothesis that only well understood or "approved" medical/surgical interventions can help people. But if it is true that no one really holds that hypothesis, single testimonials of chiropractic care having helped someone will lose their contradictory or impossible quality and become just ordinary individual events, subject to all the vagaries and imperfections of biased, non-representation, and disproportionality already described.

While vivid testimonials may work on human beings with human memories, third party payers make decisions using actuarial statistics and a scientific memory system called money. Money memories respond impartially to all the data, and are not overly impressed by one or two events, no matter how sensational. These are seen for what they are -- individual single cases which, even if we discount the possibility of intentional deception, may well not be representative enough to generalize to any other cases. (As Minnesota Fats tells Fast Eddie in "The Hustler," the way to find out who has the best pool stroke is not to watch flashy individual shots, but rather to count up the money at the end of the game.) Third party payers must use the kind of methods which are as reliable and valid and generalizable as we humans can make them, methods which they can (and do, quite literally) take to the bank. Taking sufficient numbers of relevant instances equally into account, having an unbiased sample large enough to be representative of the target patient population, is not only the essence of scientific method, it is also the bottom line.

The researchers of the chiropractic colleges and institutions in the Consortium for Chiropractic Research understand that individual reports of chiropractic effectiveness are no longer, and probably never were, impossible chickens. We are therefore moving on toward controlling and collecting sufficient numbers of individual events which will allow us to characterize and understand the real variety and complexity of chiropractic effects.

Robert Jansen, Ph.D.
Executive Director,
(Pacific) Consortium for
Chiropractic Research
Sunnyvale, California

Individual memberships are $65 per year and include a subscription to the Consortium newsletter, which keeps doctors updated on consortium activities and projects, e.g., standards of care. The membership dues may be charged to your Mastercard or Visa by calling (800) 327-2289, or you may send a check to: The (Pacific) Consortium for Chiropractic Research, 1095 Dunford Way, Sunnyvale, California 94087.