Everything’s Bigger in Texas — including Chiropractic memes

Today I came across this in my Facebook newsfeed:

This deserves a much closer look. To my knowledge, Chiropractic is just another snake oil elixir, delivering essentially no benefits beyond the alleviation of lower back pain. Here is the claim:

“A 7 year study showed that patients whose primary care physician was a Chiropractor experienced the following results:

• 60% less hospital admissions
• 59% less days in the hospital
• 62% less outpatient surgeries
• 85% less in pharmaceutical costs”

I am extremely skeptical. My poorly-constructed-study radar is hitting 11 on this one. 85% less in pharmaceutical costs! That should be putting the entire industry out of business! What an incredible claim. But instead of outright dismissing these findings, let’s check out the study.

The fine print cites this:

Journal of Manipulative and Physiological Therapy, May 20017, 30(4). 263-269. Richard L. Sarnat, MD, James Winterstein, DC, Jerrilyn A. Cambron, DC, PhD

Here are some things I’ve found:

1. There is no journal that exists called the “Journal of Manipulative and Physiological Therapy.” There is one with Therapeutics as the last word, so I’ll assume these are supposed to be the same. This is the journal’s website.
2. The journal cited is May 20017. I will give the benefit of the doubt as say they meant 2017, because 20017 is still roughly eighteen thousand years away. Nonetheless, how does this person have access to the journal for six months from now?
3. Since the date is clearly unusable, here is the search results for “Sarnat” in the journal’s database.The meme claims to cite volume 30, issue 4, pages 263-269. So the second result is the target here. Sadly, both are paywalled, and I can’t view them unless pdf copies are floating around elsewhere online.
4. This is everything I can get from the website:Abstract
   Our initial report analyzed clinical and cost utilization data from the years 1999 to 2002 for an integrative medicine independent physician association (IPA) whose primary care physicians (PCPs) were exclusively doctors of chiropractic. This report updates the subsequent utilization data from the IPA for the years 2003 to 2005 and includes first-time comparisons in data points among PCPs of different licensures who were oriented toward complementary and alternative medicine (CAM).

   Methods
   Independent physician association–incurred claims and stratified random patient surveys were descriptively analyzed for clinical utilization, cost offsets, and member satisfaction compared with conventional medical IPA normative values. Comparisons to our original publication’s comparative blinded data, using nonrandom matched comparison groups, were descriptively analyzed for differences in age/sex demographics and disease profiles to examine sample bias.

   Results
   Clinical and cost utilization based on 70274 member-months over a 7-year period demonstrated decreases of 60.2% in-hospital admissions, 59.0% hospital days, 62.0% outpatient surgeries and procedures, and 85% pharmaceutical costs when compared with conventional medicine IPA performance for the same health maintenance organization product in the same geography and time frame.

   Conclusion
   During the past 7 years, and with a larger population than originally reported, the CAM-oriented PCPs using a nonsurgical/nonpharmaceutical approach demonstrated reductions in both clinical and cost utilization when compared with PCPs using conventional medicine alone. Decreased utilization was uniformly achieved by all CAM-oriented PCPs, regardless of their licensure. The validity and generalizability of this observation are guarded given the lack of randomization, lack of statistical analysis possible, and potentially biased data in this population.

5. The references page (link) includes citations to these journals:J Manipulative Physiol Ther. 2004 (their own research)
   Ann Intern Med. 1999
   J Clin Oncol. 2000
   Arch Intern Med. 2002
   JAMA. 1999
   N Engl J Med. 1993
   Am J Manag Care. 2006
   Health Care Financ Rev Annu Suppl. 1991
   CBO; US Government Printing Office, Washington (DC); 1993.
   Harv Bus Rev. 1994
   Johns Hopkins University Press, Baltimore; 1990.
   Health Care Financ Rev. 1992
   Manag Care. 2001
   Altern Ther Health Med 2002
   National Center for Health Statistics. Accessed September 15, 2006. (link included)
   Cancer. 2004

   Two things to note here: most of these are just cost-assessment studies, so it doesn’t matter how legitimate their scientific journal is; also, several of these are other alternative medicine journals, so take that with measured caution.

6. Hooray! I found the study! Here is the pdf link.
7. From the study: “In this article, we are not taking a position on the efficacy of any CAM treatment. Rather, we are interested in the current use of CAM modalities and cost effects of such use, regardless of treatment outcome.”
8. The study is based on 7 doctors of osteopathy and 14 doctors of chiropractic. Osteopathic doctors are basically defined as non-chiropractic doctors of natural medicine, which does exclude pharmaceuticals and surgeries. This is a total of 21 doctors.
9. This should be an enormous red flag: “The HMO’s quality control division, independent of the privately run IPA, distributed an annual survey to more than 45 000 members who were older than 18 years old and who had been enrolled in the HMO and IPA for at least 1 year. Stratified random patient surveys were used to analyze AMI’s lifestyle demographics and member satisfaction. Although the HMO’s quality control division provided these data, the details of the stratified random selection process were not available. Member satisfaction was measured within the survey by asking patients, “are you satisfied overall with your IPA’s performance? ” (emphasis added).
10. Here we find an even larger, more enormous red flag: “Our initial report demonstrated a skewed enrollment population, with fewer children and more adults than the 2 matched control groups. For the years 1999 through 2002 we averaged 12% childhood enrollment vs the 2 control groups, whose childhood enrollment averaged 33% and 19%, respectively. We attributed this population age disparity to a deliberate IPA medical management policy of discouraging childhood enrollment. This management decision was put in place because of the limitations in the scope of practice our DCs and their inability to perform certain requirements, such as immunizations. Our PCPs licensed as medical doctors/DOs have no such limitations in their scope of practice. Accordingly, we have seen our enrollee demographics quickly change and even exceed the childhood enrollment percentages of the 2 matched control populations. In calendar year 2003, the IPA’s childhood enrollment increased to 31%; and by calendar year 2005, it had peaked at 56%. We attribute this demographic shift, above the 2 matched control groups’ childhood enrollment, to the unique group practice of our newly contracted medical doctors /DOs. Before their involvement with AMI’s integrative medicine IPA, they specialized exclusively in the 2 arenas of home birth and ‘natural medical’ childcare” (emphasis added).Later on, the study mentions this: “we were not able to control for differences in baseline characteristics between the integrative medicine group and the conventional IPA. If the baseline demographic or clinical factors differed between the groups, the data may be seriously biased in either direction.”
11. Red flag number three: “The AMI’s enrolled population continues to demonstrate a smaller percentage of ‘well’ members (23.4% in Table 2) vs the 2 matched conventional medical IPA control groups (34.7% and 42%, respectively), as cited in our initial report. This gives continued credence to the premise that patients who go to CAM practitioners are not necessarily the ‘worried well’ and may actually represent an adverse selection of patients who are ‘medical failures’ in the traditional medical system.”
12. Table 4 and its corresponding paragraph (Cost of Utilization) are surprising. Why is the company actuarially predicting 670.0 target units, when by the end of the year only 125 units were used? Either the actuaries are terrible at their jobs, or something is very wrong with the reporting mechanism for units used. This can be viewed as 19.0% of total costs used, which sounds great if you are a budget slasher or are trying to produce a study showing lower costs for Chiropractic. Or, it can be viewed as gross overestimation from the beginning. When all 6 years overpredicted by more than double the ultimate value, the process is broken.
13. Then this appears: “As the necessary data for traditional statistical methods were unavailable to us, we attempted to assess possible population bias via other strategies. We acknowledge that the lack of statistical analysis may have led to a serious bias. However, even without the ability to complete a statistical analysis and with the potential for bias, these preliminary data are important to present within the medical community.” Actually, no. If there is the potential for serious bias, maybe it is better to not report the study’s findings in such plain terms, as if they can be taken at face value.

At this point I’ll put forward some possible explanations for what has happened here:

First, this is a single study. Ask a researcher who complies meta-analysis studies about the legitimacy of individual studies. This is why we reproduce studies several times. 10 researchers will conduct the same experiment, and 3 will find negative correlation, 4 will find no correlation, and 3 will find positive correlation. You would never know this if you looked just at one study. So, these authors are very correct in saying that this study “warrants larger independent third-party funding for multicenter, randomized controlled trials.” All studies do.

Second, the shift in population to a younger group undermines essentially all of the findings in this study. For one, why did the group become younger? The group became younger because in the 2003 iteration of the study, the researches decided to include the extra 7 osteopathy doctors, who had largely specialized in home, family and natural medicine. Now, this shouldn’t have had an impact unless these doctors were carrying over old clients from before the study began. So it looks like either that happened, or that they continued to advertise themselves in a way that would lead to more children than normal being consulted.

Third, and directly related to the previous point, if they inadvertently shifted the demographics of the population in this way, maybe they also shifted the demographics of the population in a less clear, less measurable way. The key variable in my mind is likeliness to reject pharmaceuticals, and that would obvious have a large impact on the total amount and therefore cost of pharmaceuticals distributed. I’d be willing to speculate, and this is very logically founded speculation, that people whose primary care physician is a Chiropractor are probably significantly more likely to also believe in alternative medicine, to use homeopathic cures, to reject established scientific literature and studies, and to be “educated beyond their own good,” meaning that they have done “their own research” which actually just amounts to googling something until confirmation bias is satisfied. The authors admit this with their caveat about the ‘worried well’ and the ‘medical failures.’ Each of these types of people would not register on a simple yes-no questionnaire about patient satisfaction, and I would be willing to bet that this category is much more skewed compared to the general population than the age category.

Fourth, the authors of this study did not have access to enough data to conclude that they have found anything of notable significance. What if, when controlled for any of the variables in the study, the correlation drops to 0.00? They do not have access to this data because of HIPPA laws, but usually accredited researchers have access to patient profiles without identifiable characteristics. So I’m not sure what the hassle was beyond bureaucratic red tape. But in any case, these descriptive statistics like “85% less in pharmaceutical costs” are uncontrolled variables relative to the real general population, to the best of my and the researchers’ knowledge.

Fifth, the solution could also be that Chiropractic doctors are not able to prescribe medicine. The osteopathic doctors can. But those were only 7 of the 21. The 14 Chiropractic doctors would likely have had to refer patients to a state-licensed medical doctor for that doctor to write the prescription. So then, you can see that prescriptions will not be written for things that either have no cure (the giant category of ‘wellness’ being at the top of this list) or things that are not very severe.

Finally, and this is only tangentially related to the actual study, someone on Facebook commented below the original shared article and wrote this:

Chiropractics are constrained to the same free market effects of supply and demand and regular doctors (although perhaps more so because most insurance companies do not cover Chiropractic costs). Here is an example of the owner of a Chiropractic clinic using exactly this study to promote his business interests. It should not come as surprising that the Chiropractic industry, which in the United State is huge and growing, also are “people, and people are motivated by the love of money… not all [chiropractors], but many will follow the [alternative medicine] industry’s talking points by telling their patients things that are designed to promote the [alternative] medical “industry.”” The argument from greed and deception goes both ways.