Monday, February 06, 2012

Why that Diet Soda/Stroke Paper is Worthless and a Failure of Peer Review


So without spending too much time on this, here's the thing, that paper that purported daily diet soft drink consumption was associated with several vascular risk factors including strokes? It's worthless, and it's a glaring failure of peer review.


Because the authors didn't even attempt to control for dietary quality, and moreover, the dietary recall data itself was obviously inherently flawed.

First the control issue. As I'm sure you're aware, what we eat has a tremendous impact upon our risk of developing various chronic diseases. Consequently not accounting for the folks who for instance ate 10 meals a week from fast food restaurants, take aways or diners, versus those who actually ate at home and transformed raw ingredients would likely skew the data. But even if you want to try to suggest that such differences would be accounted for by the dietitians conducting the dietary recall effort, if all you actually analyzed at the end of the day was amounts of consumed protein, carbohydrates and fats, what you'd fail miserably in doing would be to actually usefully compare the quality and caliber of the diets you were studying. By not looking at the quality of the macronutrients you'd be comparing quinoa to white rice, salmon to bacon, and olive oil to Crisco. Yet that's exactly what the authors did. And it's certainly not at all implausible that folks who regularly indulge in lower caliber dietary choices assuage some of their dietary guilt and build their own health halos by choosing a diet beverage over a fully loaded one.

But even if the authors accounted for (as they should have at the very least tried) meals out, vs transformed raw ingredients, as well as the actual quality of dietary macronutrients this study would still be useless.


Because according to the dietary recall data presented in their study, and despite an average body mass index of 28 (overweight), the self-reported overall caloric intake of the study population was a measly 1,575 calories, whereas 2000 NHANES data pegs the average for men between the ages of 59-79 to be between 2,123-2,590 and women between 1,596 and 1,828. And that's 12 year old NHANES data.  Given what we've seen with obesity rates, it's certainly quite plausible if not exceedingly likely, that average caloric consumption has risen over the past decade and change.

So basically here we have a study where we know the self-reported dietary recall is inaccurate and where the authors didn't even attempt to account for the actual quality of the participants' diets, that's making conclusions about the impact of diet soft drink consumption on strokes and suggesting diet was a controlled for variable? 

How this got through peer review is completely beyond me, but the worst part is the coverage. This paper got plenty of press making its publication not only an embarrassment to the Journal of General Internal Medicine and its peer review process, but also a powerful source of misinformation in a world that certainly doesn't lack for nutritional confusion.

Gardener, H., Rundek, T., Markert, M., Wright, C., Elkind, M., & Sacco, R. (2012). Diet Soft Drink Consumption is Associated with an Increased Risk of Vascular Events in the Northern Manhattan Study Journal of General Internal Medicine DOI: 10.1007/s11606-011-1968-2

Bookmark and Share


  1. Dr. Freedhoff

    Thank you for writing this explanation. As a RD, I've always struggled with analyzing research papers and methods to assess for quality research. I've even tried to take the Dietitians of Canada online course in Evidence Based Nutrition and struggled with that. I rely on PEN (Practice based Evidence in Nutrition from the Dietitians of Canada) and clinical practice guidelines to help me deliver quality care.

    I am thankful that there are many people in the world, such as yourself, who can analyze research and tell the rest of us 'what it means'.

    Great post!

  2. gee whiz, how are any of us supposed to know what to believe?? thank you for being a constant voice of reason! ps: great appearance on marketplace this week!

  3. Anonymous6:15 pm

    The first thing we learnt in an hour lecture at medical school with dieticians was that patient recall was rubbish. One only has to try and recollect the days diet at night and see how poorly it goes.

  4. Anonymous6:25 pm

    I am not entirely convinced by your arguments against the content of the paper for two reasons. First, there would have to be significant differences in the quality of dietary recall between those who drink diet soft drinks at least daily and everyone else in the population at large. And within the limits of dietary recall, they seemed to have tried to control for dietary quality:

    "Controlling for age, sex, race/ethnicity, education, smoking, physical activity, alcohol consumption, BMI, daily calories, consumption of protein, carbohydrates, total fat, saturated fat, and sodium, ..."

    Second, this study,, seems to implicate artificial sweeteners on biochemical level in the promotion of cardiovascular diseases.

    While there is not conclusive evidence, it seems that there is good reason for continued study of artificial sweeteners as potential promoters of cardiovascular disease. And it would be reasonable to suggest moderation in the consumption of artificial sweeteners pending further research.

  5. Ah Anonymous, the foundation for attacks of the fearful. If you are going to critically analyze Dr. Freedhoff's posts, put a name to it. And you are wrong. Even small changes in dietary recall would make a SUBSTANTIAL difference in the validity of the study. As mentioned by Dr. Freedhoff, if the average BMI was 28, it stands to reason that the average reported daily caloric intake is essentially rubbish. As a very average man, a caloric intake of 1550 calories per day would create a weight loss of roughly 1.5-2 pounds per week, not weight maintenance at an elevated weight. Thus, it is incredibly obvious that the dietary recall modality used here is faulty. Thus, none of the dietary intake measures controlled for in the multivariate analysis can be trusted. And if they cannot be trusted, neither can the results of the multiple regression analysis, meaning that you cannot statistically determine the impact of diet soft drink consumption ALONE on cardiovascular risk factors. We cannot conclusively say anything at all with the poor methodology of this study except exactly what Dr. Freedhoff concludes: the study was rubbish, and unless it was meant to teach others the risk of using recall data in correlational population studies, it also exposes the weaknesses in the peer review system, at least as it exists at this particular journal.

  6. Katherine B11:43 pm

    Great post. While I believe artificial sweeteners should be better studied to determine if there is any risk of long-term consumption, poorly written papers are unhelpful and only serve to promote misinformation and confusion.

    @Anonymous #2: Regardless of dietary quality, the atypically low average caloric intake is an indicator that the diet assessment tool used was flawed. As Dr. Freedhoff mentioned, most people take in between 1800-2500 calories per day (and likely even more than that). Using this tool means that any of the results are highly questionable, since it is extremely unlikely that the tool is accurately measuring what people are eating. And if we don't know what people are eating, we can hardly make sound conclusions about what is and is not healthy.

    Furthermore, as Dr. Freedhoff said in his post, the controls for dietary quality are rather simplistic. We now know the varying health effects of different types of fats beyond saturated and unsaturated, for example. And what about carbohydrates - were they from oatmeal or white bread? Their methods made no mention of these details, which are important confounding factors. I do agree that more research is needed, but hopefully future papers will have more rigourous dietary assessment methods.

  7. I had the very same issues with conclusions drawn by top researchers at Harvard with their much publicized study demonizing all things potato and their impact on weight management. Even reputable names in science can draw faulty conclusions!

  8. I've been dieting for some months - staying pretty religiously at 1500 calories per day with a few "splurges" here and there. And I have to say, it's tough to consistently keep that up. If you tell me these people are actually staying at 1500 calories without really trying, I'm .. objecting strenously in words I won't state here. Tiny things make big differences when you discover that even a packet of catsup has 15 calories. 3 packets of catsup on your fries makes a significant difference to the calorie count, but how many people would recall even that they HAD catsup when figuring dietary intake later, unless they diaried it as they were doing it.

  9. This website facilitates the broad dissemination and use of scientific methods and tools that help to study discrete events.