Monday, March 19, 2012

What Reading that White Rice and Diabetes Study Actually Told Me


The short version of what it told me was that the BMJ is actively and willfully abusing the public's trust, and it also told me that many low-carb and paleo diet guru bloggers and tweeps don't care about bad nutritional epidemiology so long as it vilifies carbs.

Here's the longer version starting with the BMJ.

Last week the BMJ published another observational dietary study out of Harvard. This time the study's focus was on white rice and it involved a meta-analysis of the medical literature on white rice consumption and its impact upon the risk of developing type 2 diabetes.

According to the study's authors, prior studies on white rice and type 2 diabetes had mixed results, and so in order to try to illuminate them, they undertook a literature review. In the end the authors' exclusion criteria left them with only 4 studies detailing 7 separate cohort analyses. Follow ups in these cohorts ranged from 4 to 22 years, and of them, 2 had exceedingly low incidences of type 2 diabetes development and so the authors had to perform some statistical sleight of hand to include them.

Of the 7 cohort analyses, 2 did not control for differences in dietary consumption patterns (probably important if you're looking at the impact of dietary factors on diabetes development), 1 did not control for family history of diabetes, 6 did not control for income/socioeconomic status, and none controlled for consumption of other refined grains or sugars. Let me go over that last one again not one of their 7 cohort analyses controlled for consumption patterns of refined grains or added sugars (though 2 controlled for consumption of "bread" and "noodles" but these weren't broken down any further).

Pooling the data together led researchers to a relative risk of type 2 diabetes development of 1.27 for the highest white rice consumption with a 95% confidence interval possessing significant heterogeneity and starting at 1.04, which for the non-statisticians reading this means the finding is just barely interesting.

When the researchers then stratified the data by ethnicity they found that for Asian populations the pooled relative risk was 1.55 with a confidence interval starting at 1.2, and for Western populations a pooled relative risk of 1.12 with confidence intervals ranging from 0.94-1.33, meaning that according to their findings, the impact of white rice on diabetes risk only seems to exist in Asians.

Translation?

The most generous way I can spin it would be that this study, using pooled cohorts that left out tremendously important controls and considerations, when analyzed, suggested that white rice consumption increases the risk of diabetes development in Asian, but not Western, populations.

The worst way to spin it? The fact the cohorts used to determine this study's conclusions failed to consider incredibly relevant diabetes confounders like family history of diabetes, socioeconomic status, and dietary consumption patterns, including the dietary consumption of other categories of refined grains, makes quantifying the effect on diabetes development due to white rice consumption from this data set impossible.

And yet it was published in the BMJ?

But that's not the truly shocking part. This is. The BMJ published an accompanying editorial that rightly called the paper out on its methodological and statistical inadequacies and in conclusion stated,
"Although the findings of the current study are interesting they have few immediate implications for doctors, patients, or public health services and cannot support large scale action. Further research is needed to develop and substantiate the research hypothesis"
Yet what's the title and first line of the very same BMJ's press release regarding this statistically and methodologically weak paper,
"White Rice Increases Risk of Type 2 Diabetes

The risk of type 2 diabetes is significantly increased if white rice is eaten regularly, claims a study published today on bmj.com.
"
Where the next 6 paragraphs of the press release continue in that same conclusive and important sounding vein only to end with this last line,
"In an accompanying editorial, Dr Bruce Neal from the University of Sydney suggests that more, bigger studies are needed to substantiate the research hypothesis that white rice increases the chances of getting type 2 diabetes."
Guess which 7 of 8 press release paragraphs the media paid attention to?

Here are a sampling of actual media headlines as a consequence of the BMJ's misleading press release,
"Eating White Rice Daily Up Diabetes Risk" - CBS News

"White Rice 'Could Cause Diabetes'" - The Independent

"Think Twice About Rice? New Study's Advice" - ABC News

"White Rice Raises T2 Diabetes Risk, Claim Academics" - The Daily Telegraph

"White Rice Linked to Diabetes Risk" - WebMD

"White Rice May Increase Your Risk of Diabetes" - MSNBC
And of course the Twitterverse went crazy too and rice bashing tweets abounded, and from some rather influential tweeps including Dr. Sanjiv Chopra, Dean of Continuing Education at Harvard Medical School, the Harvard School of Public Health itself, journalist Greta Van Susteren, the American Society of Nephrology, the Drudge Report, and many, many, many, many more.

Now I'm not suggesting white rice is a wise food to consume. On the contrary, I generally recommend people try to minimize its consumption, but to be very clear, this study does not in any formative way, shape or form have the strength to draw any conclusions whatsoever on the specific impact white rice has on the risk of developing type 2 diabetes.

And what the hell is up with the BMJ? Publish a paper so weak that you feel the need to co-publish an editorial questioning the paper's design and conclusions and then simultaneously put out a press release that in turn purposely glosses over and misinforms the media about the paper's weak conclusions?

And lastly, where is the low-carb and paleo bloggers' and tweeps' outrage? We saw it surge like never before when a far more rigorously controlled and designed observational study on red meat came out just a week before, and yet here there's pretty much critical silence (Zoe Harcombe being a notable exception).  There were even examples where prominent low-carb and paleo folks retweeted the press releases to their followers as if they were facts, when just days before those very same tweeps were tweeting their outrage at what they saw as the meat study's undeserved promotion.

The real lessons here?  Confirmation biases and impact factors sure do cloud our judgements and analyses.

Bookmark and Share

11 comments:

  1. I mentioned the thorough and critical NHS Choices analysis of the paper to several people who were promoting the White Rice Increases Risk of Diabetes item.

    What was interesting was that beyond confirmation bias, it was argued that as the personal belief was that white rice is equivalent to candy, then it was effectively acceptable to discount a critical analysis of the paper (I may be doing the people involved a disservice in characterising it this way as it's difficult to convey nuance on Twitter).

    Agreed on the overall thrust that some press releases are substantially misleading.

    ReplyDelete
  2. What I don't understand is if the paper was that flawed, why would a "prominent" journal like BMJ even accept it in the first place? So what if the editorial slams the methodology? We've seen time and again that popular media picks and chooses the most inflammatory parts of papers for "shock factor"...

    I expect more from a so-called high-impact journal.

    ReplyDelete
  3. Anonymous9:07 am

    Personally, after seeing most papers on observational studies, I can't imagine why we keep funding them. Suspect data, poor analysis, and, fundamentally, all they can possibly do, even if done correctly, is generate testable hypothesis. Because correlation doesn't imply causation, all they can possibly do is generate misleading headlines.

    ReplyDelete
  4. As someone who has done science communication (including writing news releases for orgs touting a published paper), this is appalling. It just seems so inappropriate that BMJ would accompany the article with a news release that doesn't adequately capture the nuance in the way the article was published. Frankly, dunno if they should have done a news release at ALL (or even published the study) if the results were so weak, but given the decision to do the release, far more attention should have been paid to the balance between the editorial and the article. Makes me sad. Gives science communications people a deserved bad name.

    ReplyDelete
  5. Well, I wouldn't ding paleo and low-carb bloggers for lack of outrage. If we/they were outraged by every bad study badly reported, it'd be all outrage, all the time.

    But RTing the press release? Yeah, that's a bummer.

    ReplyDelete
    Replies
    1. The only reason the lack of outrage got to me was the degree of publicity of the study was on par with the meat one.....but you're right, the real kicker was the tweeting by folks who definitely ought to know better.

      Delete
  6. Anonymous10:29 am

    When I did my MSc in epidemiology 11 years ago, I reviewed a "study" published in BMJ. I don't remember the details, but was outraged by the authors' (they had no letters following their names) use of bad data AND they used the wrong statistical tests to analyze the data. As for the media - no surprise there.

    ReplyDelete
  7. Anonymous2:25 pm

    I work with a LOT of asian people who seem to eat white rice daily in the lunch room. None of them are over weight, and frankly they all look relatively healthy!
    I myself try to only eat the brown, but hard to find brown rice sushi in Edmonton, so the occassional white rice does get into diet.
    But as I aways say, ALL THINGS IN MODERATION.

    ReplyDelete
  8. Anonymous6:36 pm

    It's at the point where I ignore any "EATING THIS IS REALLY BAD" headline I see. It seems like none of it is relevant if you're already exercising common sense, eating a varied diet, active and managing any risks you may have inherited. I can't think of a single instance of contemporary nutritional fearmongering that's concluded in solid fact beyond what I already know. Anyone zealously tweeting such self-important articles the moment they're published isn't even in the running as a reliable source, IMO. It's always a case of agenda-pushing.

    This of course leads me to wonder how professionals can help the public learn to differentiate between the hype of the week and basic nutritional savvy. When iffy, but nonetheless formal, journal articles are being sent out and digested sans context, it gets really hard to know who to trust.

    ReplyDelete
  9. Anonymous3:25 pm

    Academic lad...

    ReplyDelete
  10. Hopefully these new scientific data will help us better understand the role of carbohydrates in the development of obesity, diabetes and heart disease. It will certainly help us realize that it is not all about cholesterol and fats.
    You can read my blog : "Rice" and shine at
    http://www.docsopinion.com/2012/03/rice-and-shine.html

    ReplyDelete