It’s the science stupid!

Alternative therapies and evidence based medicine don’t mix well

The science based medicine site has an interesting discussion that points out a major problem when researchers evaluate complementary and alternative medicine/veterinary medicine (CAM/CAVM) under the rubric of evidence based medicine (EBM).

The structure of EBM allows clinicians to objectively discern between varying qualities and levels of evidence for therapeutic efficacy regarding whatever modality is being considered. This system provides a solid statistical and epidemiological data base through the use of a variety of techniques. The goal is to create a reliable system of information from where clinicians can develop sound therapeutic decisions.

However, there has been a nagging problem that EBM seems to have difficulty resolving with respect to CAM/ CAVM research. It lies within the basic structure of EBM’s lack of ability to properly assess the science base of many therapies going into studies such as clinical trials.

In other words, EBM (like the Cochrane collaboration) has trouble if you take it beyond its primary mission. Dr Atwood notes “EBM…comprises sets of guidelines for assessing evidence...” These are systems designed to evaluate evidence based on therapeutic and clinical significance-and herein lays the problem.

EBM doesn’t consider that many of the types of testing it utilizes to create statistical information is not geared toward scientifically examining the therapy in question. For example, it can construct a long line of trials for homeopathy –imaginary as this modality is- and yet still allow the ebb and flow of statistical chance to offer up meaningless "weak positive" results. The problem here is that this gives wiggle room from where its practitioners, clinging to false hope, can call for “further studies.”

This is reminiscent of the garbage in/garbage out Achilles heal of many meta-analysis where, if one is not careful, low quality studies can be used to give a “result”- skewed as may be- that may lead to nowhere or worse. In this vein, many CAM/CAVM therapies that seek legitimacy through EBM standards of testing do not even qualify to be tested because they suffer from a far more basic flaw –they lack basic scientific plausibility.

EBM does not take this important detail into account and this often results in an endless loop of back ground noise -like static- of “equivocal” results. There is an accumulation of mostly meaningless interpretations of gibberish because the original input was flawed at a very fundamental level.

Dr Atwood notes “Some claims are so implausible that clinical trials tend to confuse, rather than clarify the issue. Human trials are messy. It is impossible to make them rigorous in ways that are comparable to laboratory experiments. Compared to laboratory investigations, clinical trials are necessarily less powered and more prone to numerous other sources of error: biases, whether conscious or not, causing or resulting from non-comparable experimental and control groups, cuing of subjects, post-hoc analyses, multiple testing artifacts, unrecognized confounding of data due to subjects’ own motivations, non-publication of results, inappropriate statistical analyses, conclusions that don’t follow from the data, inappropriate pooling of non-significant data from several, small studies to produce an aggregate that appears statistically significant, fraud, and more.”

The misinterpretations can be truly mind boggling. One way to avoid this maze of miscues is that, in order for a therapeutic modality to be properly evaluated by EBM, it must first be fundamentally sound from a scientific perspective. It needs to pass under the critical eye of plausibility and not waist the time and effort that EBM requires. In this way, many of the current endless streams of research by the NCCAM for example could finally be put to rest freeing monies for truly worthwhile research.

In discussing this garbage in/garbage out problem in EBM Dr Atwood optimistically adds “Most of those problems are not apparent in primary reports. Several have already been discussed or referenced elsewhere on this site: here, here, here and here, for example. Academics active in the EBM movement are aware of most of them and want to correct them—as a quick scan of the contents of almost any major medical journal will reveal.”

Evidence base medicine, in and of itself, is not the most efficient filter of dubious therapies without the all important underpinning of basic science. For the present then, it behooves readers of the research to study it with a sharp eye, learn from those schooled in its ways, and pay attention to the details- hard but worth the effort.