THE HEADLINES ANNOUNCED a dream come true: Scientists Say Eating Chocolate Can Help You Lose Weight!
They were reporting the results of a study published in the impressive-sounding International Archives of Medicine, an online open-access journal. The study’s authors claimed eating chocolate along with a low-carb diet had helped people lose weight and increase their general well-being, compared to people who just ate a low-carb diet.
The problem is, the whole study was an elaborate hoax.
Journalist John Bohannon and others designed, ran, published, and publicised a diet study of purposely dubious scientific merit to make a point about how claims with little to no scientific backing make headlines and drive diet fads, Bohannon reveals in io9.
The study was designed to be part of a documentary about the junk-science diet industry, and the filmmakers had a doctor (to run the trial) and a statistician (to play with the numbers) all lined up. Bohannon, who has a PhD in the molecular biology of bacteria and a known history of successful scientific pranks, was brought on to help get the study published and publicized.
They recruited 15 participants, who were paid €150, screened for any health problems, and told they were going to be in a documentary film about dieting — which was true.
One set of participants ate a low-carbohydrate diet, another followed the same diet with the addition of a 1.5 oz bar of dark chocolate, and a third set acted as controls. They all weighed themselves daily for three weeks, while the “researchers” measured everything from sleep to BMI.
Lo and behold, the group that ate chocolate plus a low-carb diet lost weight 10% faster than the group on the low-carb diet without chocolate. The difference was statistically significant, and Bohannon had a hook to sell.
It wasn’t hard for Bohannon to get the study published — journals have widely varying standards for peer review, and a lot of them accepted the study. Once the study was accepted, Bohannon wrote a press release, created a fake site for his made-up research center, and — sure enough — reporters started writing about the sham study.
It seemed they’d pulled it off.
In Bohannon’s story on io9, he lists major German daily newspaper “Bild,” the German “Cosmopolitan” website, and the German and Indian “Huffington Post” websites among the news outlets that fell for the ruse. He also includes screenshots of coverage from the “Prevention” website, a few British tabloids, and the June issue of “Shape” magazine.
Bohannon seems pretty pleased that these publications with large audiences picked up his study, “fooling millions,” the io9 headline claimed. But as journalist Daniel Engber pointed out on Twitter, maybe it wasn’t such a feat after all:
Still, it’s worth taking the warning to be a little more skeptical about reports that a certain diet or food has major effects on your health — especially when those reports show up in a tabloid.
This study wasn’t so different from many other diet studies that also use small sample sizes. It’s unfortunately not common for scientists to tweak their data by running an experiment multiple times or excluding outliers to get statistically significant results, mostly innocently.
A good study typically has a hypothesis and then tests it. This study had no real hypothesis and tested 18 different things — not one thing — to see if any statistically significant associations popped up. In poorly designed diet studies, that’s not actually so unusual. If enough things are measured in such a small group, that approach is bound to yield… something. Bohannon explained it this way:
Think of the measurements as lottery tickets. Each one has a small chance of paying off in the form of a “significant” result that we can spin a story around and sell to the media. The more tickets you buy, the more likely you are to win. We didn’t know exactly what would pan out—the headline could have been that chocolate improves sleep or lowers blood pressure—but we knew our chances of getting at least one “statistically significant” result were pretty good.
In other words, while the study did indeed find a statistically significant effect on weight loss, it was designed to do exactly that. They were statistically more likely to find what looked like a significant effect but was actually a fluke.
Read Bohannon’s full account of the hoax here.
To embed this post, copy the code below on your site
have your say