Evidence-based medicine requires no evidence, it seems

A bunch of scientists were analyzing the behavior of a frog. Cutting off its front legs did not preclude it from jumping when they slapped the table and yelled “Jump frog, jump!” But when they cut off its hind legs, the frog just sat there, despite their banging and yelling. Sequestering themselves for weeks, thinking deep thoughts and stroking their beards, they finally emerged with a pronouncement of their findings. It was irrefutable that the study had proven, conclusively, that cutting off the frog’s hind legs — made the frog deaf.

As demonstrated in a new study, much of medical “knowledge”, especially preventive guidelines, is made of deaf frogs. The analysis was of the quality of data-driving guidelines issued over the past 10 years by the American College of Cardiology, American Heart Association, and European Society of Cardiology. These are the big three when it comes to telling us that saturated fat, eggs, salt, anything else delicious are bad; no, wait, good; no, never mind, bad; oops, the science has evolved, so we reserve the right to change our minds monthly. Less than 15 percent of the thousands of recommendations in their many guidelines were based on the highest quality of research.

{mosads}What this study provides is proof positive that “evidence-based medicine” requires no evidence. While the term “evidence-based practice” is intended to mean “based on really good evidence,” it is generally meant to refer to medical practice based on guidelines in which the evidence was graded. Guidelines still use the terminology as it was originally intended. Level A is a really large amount of high quality and unbiased data. But Level E still qualifies as an evidence-based guideline despite the fact that Level E recommendations are based solely on expert opinion. No data required.

There are many problems with issuing pronouncements based on inappropriate or poor quality data, and the subsequent flip-flops that occur. First, science loses all credibility. Even when we get it right, no one will be interested. Second, without credible science, the void has been filled with an explosion in the growth of quackery and the harm that comes with it. Third, we spend enormous amounts of money and emotional capital trying to live healthfully. So when the adherence to a faulty guideline does not prevent that heart attack, we’re left with either blaming ourselves for not following the guideline to perfection, or blaming science for getting it wrong. But dietary behavioral research clearly shows that we are light-years away from being able to predict an individual’s response to our attempts at getting them to change their diet.

Unfortunately, this leaves us with the conclusion that dietary recommendations, as much as they are in demand by the consumers of medical news, should be shelved until we know that they a) achieve the desired result, b) do so with more benefit than harm, and c) can be accomplished. Interestingly, the quacks don’t seem to suffer from this kind of scrutiny.

Any clinician reading this will be asking, about now, “How does nutrition stay important if it isn’t the lead article on the nightly news?” But this is the wrong question. “Is nutrition important?” is the first question we should be asking. We have identified the questions, through observation and epidemiology. Now we need to ask them the right way: Through large, expensive, randomized trials that may take years to do. But before we can even begin to try to do these, we need to be effective at changing behaviors.

There are the beginnings of recognition of this issue. It is certainly the position of credible professional and scientific organizations, such as the American Society for Nutrition, that randomized studies are needed. Randomized studies are being designed, for example, to test whether sodium restriction is actually beneficial. But because we know that it is nearly impossible for free-living subjects to achieve a restriction of much below 3 grams per day (the American Heart Associate wants everyone at around 2 or even 1.5 grams per day), the researchers are going to have to get creative. They are planning to randomly assign sodium restriction not to individuals but to whole prisons. The investigators argue that this is ethical since they believe strongly that the intervention (sodium restriction) is beneficial. But there is a lot of new data that suggests that restriction of sodium may be harmful. This is data from over 100,000 people in 35 countries observed over 10 years.

In fairness, this new sodium data is only marginally better than yelling at legless frogs. It is still not randomized. But it does study a huge number of subjects for an extended period. It is also perhaps a bit better than pure epidemiology because it pairs and compares different cohorts of subjects. Regardless, cause and effect may only be determined from randomized trials. But this data certainly flies in the face of prior observations, and raises significant concerns, namely that we’ve been recommending practice that might be harmful if it were possible to achieve.

Disease-prevention is something all of us hunger for. It is always a top news story. We claim preventable disease is destroying our economy. However, until we are able to effectively change behavior and we’re certain we will improve our health by doing so, perhaps we should focus our public outreach efforts on educating our science news media, and their consumers, about the dangers of our pronouncements when “we don’t know” is an unacceptable answer. Certainly, it is time for better funding for proper research.

David S. Seres, MD is the director of medical nutrition, associate professor of medicine, and associate clinical ethicist at Columbia University Irving Medical Center, New York, and chair of the Medical Nutrition Council and member of the board of directors of the American Society for Nutrition The views above are his own.