Under the Radar - Or Is That "Boardwalk"?

Every so often, a neglected item sneaks in to wreak havoc with the order of things. We usually call such a phenomenon as being "under the radar." It is tempting to go a step further and refer to the more hidden, but more immediate beehive of activity. The activities are the more vivid events in their personal lives, which people fondly recall as being "under the boardwalk." This terminology is in shameless deference to Johnny Moore's immortal hit with The Drifters, which eventually landed that doo-wop group in the Rock n' Roll Hall of Fame.

And so it is with a very recent systematic review¹ that seeks to address the shortcomings of randomized controlled trials, and many rating systems of their quality, for coming to grips with the nuances of both CAM and patient clinical encounters in general.^2-4 It views a series of outcome studies through multiple lenses, comprising conventional approaches and one referred to as "whole-systems research" (WSR), which attempts to bring the evaluated research methodology into closer congruence with the paradigm of the investigated system - in this case, chiropractic care as it relates to non-musculoskeletal conditions.

As is true of boardwalks everywhere, regardless of preference (e.g., Coney Island, Atlantic City or the Santa Monica Pier), there is plenty of activity topside at which to witness and marvel. In the case of this systematic review, the new "gentler, kinder" way of ranking things through WSR includes such important and previously underrated elements in clinical research as:

• inclusion of the entire clinical encounter (rather than single procedure) in the clinical encounter;
• assessment of patient preferences and expectations;
• tailoring the intervention to the patient;
• having the intervention representative of usual practice;
• having the comparison group represent "real life;"
• consideration of the importance of outcome assessment measured effects to the patient; and
• assessment of general/systemic/quality-of-life aspects.¹

The study is to be commended for having brought forth all these elements, as well as extending greater recognition to the weight and meaningfulness of observational and experimental studies, a finding that has recurred recently in the medical literature.^5,6 It also has presented an extended and annotated account of non-musculoskeletal conditions, reaching into the osteopathic and physical therapy literature, as well as chiropractic, in documenting intervention. Consequently, it is able to cite upwards of 30 conditions for which responses have been reported, including the well-known conditions of dysmenorrhea, colic, enuresis, otitis media, vertigo, hypertension and asthma. It also includes items such as attention deficit disorder/hyperactivity, seizures, pneumonia, infertility, chronic obstructive pulmonary disease and constipation.¹

The problem arises when one peeks below the surface, just as we realize how much carrying on there is under the boardwalk. The glaring example here is quite dramatic and involves three studies that were rated as "high quality," not only by two conventional scales (Scottish Intercollegiate Guidelines Network⁷ and Jadad Scale⁸), but also by the presumably more sensitive WSR scale. The unsuspecting reader would therefore most likely come away from this paper thinking that these three articles provided some of the strongest material yet for evidence-based medicine. As it turns out, nothing could be further from the truth: The dimensions of the problems in these studies are of such a magnitude that it largely invalidates their results.

Asthma Study: As I have described elsewhere,⁹ the trial used an inappropriate sham procedure (low-force manipulation of the scapular, gluteal and cranial areas, which elicited a positive response) and appears to have been underpowered. Of greater significance is the fact that only the procedure with higher force used in this study failed to produce an additional benefit under their experimental conditions. It should never be interpreted as a negative mark against chiropractic, as both the Cochrane review¹⁰ and the press have done. It also is questionable how generalizable the authors' finding may be, given that there still seemed to be a trend toward improvement in the manipulated group that didn't reach statistical significance.¹¹

Dysmenorrhea Study: A closer look at the eligibility requirements of patients in this full-scale study reveals many departures from the pilot study, most likely in order to achieve sufficient recruitment for the full-scale trial, which was admittedly an arduous process. Patients often admitted having experienced no pain within the past 48 hours and did not have to forego exercises or NSAIDs for a blackout period prior to the trial. This was in contrast to what had been required in the pilot study, which had far stricter admission requirements. In the latter trial, only one practitioner delivered not surprisingly, the experiment delivered positive results.¹² This was opposed to the lack of significant differences between the treatment and sham groups in the full-scale trial.¹³ Again, one could easily have been misled into thinking that manipulation conferred no measurable benefit for patients in this instance - a conjecture that is only reinforced by this study's top ratings in the WSR study.¹

Colic Study: This trial failed to find substantial differences between the treatment and sham groups,¹⁴ unlike the other trials and cohort studies addressing colic.^15-17

No mention was made in this particular study as to which region of the spine was manipulated. This turns out to be a significant piece of information. If the lumbar region, rather than the cervical region, was manipulated (as in the other positive studies^15-17), all bets are off. This would turn the entire trial on its head such that its apparently negative outcome would be egregiously misleading.

The point here is that a comprehensive means for rating a study, and thus experimental evidence, still has a long way to go. The quality scale developed in this systematic review¹ clearly did not encompass major problems in three of the highest-rated trials, which in my opinion, threatens to invalidate their results, relegating them to the level of being, at best, marginally useful. So even though one might suggest that there are privacy issues pertaining to our prying into all that is going on under the boardwalk, this level of "homeland security" surveillance and questioning is necessary if we are to effectively move forward in providing the best data for what we refer to as evidence-based medicine.

References

1. Hawk C, Khorsan R, Lisi AJ, et al. Chiropractic care for nonmusculoskeletal conditions: a systematic review with implications for whole systems research. Journal of Alternative and Complementary Medicine, 2007;13(5):491-512.
2. Walach H, Jonas WB, Lewith GT. The role of outcomes research in evaluating complementary and alternative medicine. Alternative Therapies in Health and Medicine, 2002;8(3):88-95.
3. Tonelli MR. The philosophical limits of evidence-based medicine. Academic Medicine, 1998;73(12):1234-40.
4. Sackett DL. Evidence-based medicine. Seminars in Perinatology, 1997;21:3-5.
5. Benson K, Hartz AJ. A comparison of observational studies and randomized, controlled trials. The New England Journal of Medicine, 2000;342(25):1878-86.
6. Concato J, Shah N, Horwitz RI. Randomized, controlled trials, observational studies and the hierarchy of research designs. The New England Journal of Medicine, 2000;342(25):1887-92.
7. Scottish Intercollegiate Guidelines Network: A Guideline Developers' Handbook. Edinburgh, Scotland: SIGN, 2001.
8. Jadad AR, Moore RA, Caroll D, et al. Assessing the quality of reports of randomized clinical trials: Is blinding necessary? Controlled Clinical Trials, 1996;17(1):1-12.
9. Rosner A. Fables of foibles: Inherent problems with RCTs. Journal of Manipulative and Physiological Therapeutics, 2003;26(7):460-7.
10. Hondras MA, Linde K, Jones AP. Manual therapy for asthma. Cochrane Database Systematic Review, 2001;CD001002.
11. Balon J, Aker PD, Crowther ER, et al. A comparison of active and simulated chiropractic manipulation as adjunctive treatment for childhood asthma. The New England Journal of Medicine, 1998;339(15):1013-20.
12. Kokjohn K, Schmid DM, Triano JJ, Brennan PC. The effect of spinal manipulation on pain and prostaglandin levels in women with primary dysmenorrhea. Journal of Manipulative and Physiological Therapeutics, 1992;15(5):279-85.
13. Hondras MA, Long CR, Brennan PC. Spinal manipulative therapy versus a low force mimic maneuver for women with primary dysmenorrhea: a randomized, observer-blinded, clinical trial. Pain, 1999;81:105-14.
14. Olafsdottir E, Forshei S, Fluge G, Markestad T. Randomised controlled trial of infantile colic treated with chiropractic spinal manipulation. Archives of Diseases of the Child, 2001;84(2):138-41.
15. Wiberg JMM, Nordsteen J, Nilsson N. The short-term effect of spinal manipulation in the treatment of infantile colic: a randomized controlled trial with a blinded observer. Journal of Manipulative and Physiological Therapeutics, 1999;22(8):517-22.
16. Mercer C, Nook BC. The efficacy of chiropractic spinal adjustments as a treatment protocol in the management of infantile colic. Proceedings of the 5^th Biennial Congress, Auckland, New Zealand, May 17-22, 1999:170-1.
17. Klougart N, Nilsson N, Jacobsen J. Infantile colic treated by chiropractors: a prospective study of 316 cases. Journal of Manipulative and Physiological Therapeutics, 1989;12(4):281-8.