Monday, December 12, 2016

Guest Post: These Two Raisin Studies Read Like Advertisements

Today's guest post comes from Dr. Dylan MacKay who agreed to writing it after I sent him a breathless press release that claimed that raisins were a great snack choice for people with diabetes. Dylan's perspective is borne out not only of his work as a nutritional biochemist with an interest in functional foods, but as well of the fact that he himself has type 1 diabetes. He also has an open mind, and if need be, a sharp pen.
Last month was National Diabetes Awareness Month (NDAM), the month when everyone cares about people with diabetes. Diabetics (I can use that term because I am one) are especially popular in November, and it seems that it is a great month to dispel diabetes myths. In an email that was forwarded to me, by Yoni, the PR team Ceres was really trying to dispel the diabetes and dried fruit myth.

This myth (apparently) goes “dried fruit, such as raisins, are not suitable for people with diabetes”. It was a new myth to me, but it kind of makes sense as I generally try to avoid more concentrated sources of sugar and try to pick fresh fruit over dried (because water). The email used REAL NUTRITIONAL SCIENCE RESEARCH (which is my favourite by the way) to dispel the myth. To quote the email directly
Now, California Raisins debunks the myth that people with diabetes cannot eat raisins and other dried fruit.
And
In fact, scientific research demonstrates that regular consumption of raisins rather than many popular processed snacks, can actually positively influence both blood glucose levels and systolic blood pressure
Well this sounds amazing, I should eat raisins, all the time.

To be sure, before I ran out and bought a big bag of raisins, I looked up the 2 studies that were cited in the email.

Bays et al. (1) was a 12-week randomized study in 51 participants with type 2 diabetes (n=19 control snacks, n=27 raisins) looking at routine consumption of dark raisins versus “alternative processed snacks” on glucose levels and other cardiovascular risk factors.

Anderson et al. (2) was a 12-week, randomized, controlled study in 46 participants with pre diabetes (n=15 control snacks, n=31 raisins) looking at routine consumption of dark raisins versus “alternative processed snacks” on blood sugar and cardiovascular risk factors. Both studies had a parallel design and were completed by the same research group at the Louisville Metabolic and Atherosclerosis Research Center. In both of these studies raisin consumption was reported as being associated with improved glycemic control and improved blood pressure compared to the “alternative processed snack” consumption.

This is amazing news right? Raisins are great! But before I bought raisin stock I decided to read past the abstract, and well it wasn’t pretty.

I am not a fan of tearing into someone else’s work, glass house and all, I don’t want to throw bricks, I’ve done work with companies and commodity groups, but sometimes bricks are required.

The first thing I wanted to know was what were these “popular processed snacks” or “alternative processed snacks”? Because in nutrition you cannot really add something without replacing something else, the most important part, in my opinion, in designing a nutrition trial is finding the right control. The selection of the control in both of these raisin studies is problematic.

I should have guessed from the California Raisin PR email that “popular processed snacks” were likely to mean ultra-processed garbage (and I would have been right). Participants on the control “alternative processed snacks” had to choose and eat three of the following snacks daily: Fudge Shoppe or Grasshopper cookies; Fudge Shoppe mini fudge stripes cookies; Honey Maid cinnamon roll thin crisps; Keebler Cheez-It crackers; Lorna Doone baked shortbread cookie crisps; Nabisco Chips Ahoy! Baked chocolate chip snacks; Oreo baked chocolate wafer snacks and Pepperidge Farm Goldfish baked snacks. Here is the nutritional values per serving of raisins and “illustrative” or “representative” snacks, interestingly between the studies there are some differences in the snack used.

These “snacks” were consumed three times a day, that is 300 kcal per day of things I would not recommend anyone eat on a daily basis. I know that both papers present their design as a very pragmatic, “real world” example of replacing a person’s current snacks with raisins. However, a better description of the design would be what happens when you replace 300 kcals of ultra-processed junk food with dried fruit, the outcome of this trial would not be very surprising. Also if the study was going to really pragmatic the control should likely have just been the NCEP education vs NCEP plus raisins. With the design used you can’t really say that raisins were “good” for the participants, just not as bad the junk snacks.

The second big problem I have with both of these trials is the overselling of results and the reporting of p values. The overselling of the results is especially disappointing as the Andersen et al. study was a self-stated pilot study. Below is a table which summarizes the findings of both the Bays et al and Andersen et al. studies.

In the statistics sections of both of these papers say “The α for statistical significance was £ 0.05, two-sided.” And neither paper mentions correcting for multiple comparisons (which is another issue). But then right after in the next section they write something like this
compared to snacks, those who consumed raisins had reduced postprandial glucose levels by 36 mg/dL (P = 0.072), reduced fasting glucose levels by 32 mg/dL (P = 0.066), reduced fasting glucose levels by 19% (P = 0.062), and reduced HbA1c by 0.12%, although statistical significance was not achieved.
Additionally both studies used last observation carried forward to deal with missing values, a technique which Ben Goldacre referred to a “dodgy” and biased towards showing a treatment effect in his book Bad Pharma. The reporting off the “trending” p-values makes me think of this tweet I saw recently,

People love to go on about conflicts of interest in nutritional science research and how industry sponsored work can never be trusted etc. (personally I think Kevin C. Klatt has done some of the best writing on this topic).

A quick look at my CV could lead someone to call me a canola oil loving, plant sterol pushing, hemp hugger (I personally think it would be unfair to characterize me like that, but you could). I don’t think industry funding necessarily means poor science (#shill). However, when you produce two papers like the ones used in the California Raisins PR email, and you happen to be receiving money from the California Raisins Marketing Board or the California Marketing Raisin Board you make it too easy for people.

But seriously I could go on for days about what is wrong with these 2 studies and the way they were marketed. They are clearly advertisements and I do not feel they add to the world’s nutritional science knowledge. In fact, they do the exact opposite they just add noise.

So Dylan, what would you have done differently?

For starters a crossover - this type of study screams for a crossover design. It gives you much higher power with the same study population and everyone acts as their own control.

Second, I would have had a NCEP education only treatment and I would have added a positive control instead of the junk food treatment. I personally tend to avoid raisins because, despite having a medium glycemic index (GI =64), they tend to have a high glycemic load (GL=28). I tend to eat too many of them too fast and end up with high blood sugar (I think that might be the idea behind GL). Since raisins are just dried grapes, I think grapes (GI=59, GL =11) would have been good positive control for raisins, and one that has been used by others (3, 4). But I guess it could be argued that grapes are perishable and more expensive than raisins, so maybe apples (GI=39, GL=6) could have been a good control, or maybe pears or peaches that are canned in juice.

And finally I would have just reported the results in a balanced and clear fashion, and I wouldn’t have reported things with p-values above 0.05 as being different. Generally I would have tried to show that I was starting from a position of equipoise. However, I guess equipoise would be pretty hard to maintain when one of your treatments is 300 kcals per day of nutritional dumpster fire.

Ideally, if I was approached by the California Marketing Raisin Board (Big Raisin) to do a trial like this I would have gone to Big Almond or Big Pistachio or Big Banana or Big Avocado etc., to see if they wanted in on the study. Then I’d have competing sponsors to balance the potential outcomes. Unfortunately, this scenario is pretty unrealistic because all those “big” food groups compete against each other, and would likely be worried about “losing” in the study. However, when nutritional scientists do run studies for industries and produce results and papers (like those raisin ones) that are predictable and as subtle as the US president elect, it is ultimately nutritional science and the people in general that “lose”.

If you like/hate this post, let me know on twitter. If you are an author of one of the above mentioned papers I’ll be at EB2017, come find my poster.

P.S. if you are from California Raisin Marketing Board and you just read this give me a call and maybe we can set up a study worth doing.

(Final thoughts on those raisin papers…)

1. Bays H, Weiter K, and Anderson J. A randomized study of raisins versus alternative snacks on glycemic control and other cardiovascular risk factors in patients with type 2 diabetes mellitus. Phys Sportsmed. 2015;43(1):37-43.
2. Anderson JW, Weiter KM, Christian AL, Ritchey MB, and Bays HE. Raisins compared with other snack effects on glycemia and blood pressure: a randomized, controlled trial. Postgrad Med. 2014;126(1):37-43.
3. Patel BP, Bellissimo N, Luhovyy B, Bennett LJ, Hurton E, Painter JE, and Anderson GH. An after-school snack of raisins lowers cumulative food intake in young children. J Food Sci. 2013;78 Suppl 1(A5-A10.
4. Patel BP, Luhovyy B, Mollard R, Painter JE, and Anderson GH. A premeal snack of raisins decreases mealtime food intake more than grapes in young children. Applied physiology, nutrition, and metabolism = Physiologie appliquee, nutrition et metabolisme. 2013;38(4):382-9.

Dylan MacKay PhD is a nutritional biochemist at The Richardson Center for Functional Foods and Nutraceuticals at the University of Manitoba in Winnipeg. He is currently working on The Manitoba Personalized Lifestyle Research (TMPLR) program. Dylan has a special interest in human clinical trials and inter-individual variability. He is originally from St. John’s, Newfoundland where he started his graduate studies at Memorial University, he completed his PhD in human nutritional science at the University of Manitoba.