The Joy of Smacking Down Pseudoscience

Support more videos like this at patreon.com/rebecca!

Transcript:

I talk about p-hacking all the time — that’s the thing where a scientist manipulates their data to make it look like they got a statistically significant result when in fact they got boloney — but I don’t remember if I’ve ever actually mentioned one of its most prominent adherents, Brian Wansink of the Cornell Food and Brand Lab. So please allow me to correct this oversight by pointing out that his research is so shady that he was recently put in his place by The Joy of Cooking. Yes, the cookbook. The one that your mother has. Trust me, she has it, or had it if she’s not around anymore. Every time a baby is born, the mother is given a blanket, a diaper, and the Joy of Cooking. And now, the Joy of Cooking has given us some delightful science drama.

I read about all this in the New Yorker, so shout-out to Helen Rosner who wrote about this way back in March, but I only just read it this week. In fact, she wrote about this only a month after a Buzzfeed published an investigation into Wansink’s antics, which I know because, well, Rosner references it in her article and also because it’s been sitting in my “things to make a YouTube video about” folder for ten months.

The tl;dr is that Wansink has for years been grabbing headlines with catchy pop science research, like that one you may have heard about that used a bottomless soup bowl to show that people will just keep eating so long as there is more soup in their bowl, until, presumably, they explode like a goldfish or that guy from the Monty Python sketch who eats a wafer-thin mint.

But for almost as long, statisticians and other researchers have been highly skeptical of his all-too-tidy findings. Good science is messy. I remember in a bad algebra teacher’s class as a kid I would know I found the right number when it worked out to something nice and even. If I did a long-ass formula and wound up with 15.979932 I was probably screwing something up. But by the time I hit calculus, it was the opposite: working out a long equation and finding the answer to be “100” was the sign that something was off, because it’s never that simple and pretty.

Most research, especially tricky social science research involving difficult-to-control humans, is the same. Messy. But Wansink’s findings were always tied up in a neat bow, and the more other researchers examined his practices and the way he processed his data, the more they found that it was a whole lot of bullshit.

Read the Buzzfeed articles for the details on all of that. What I didn’t know back when that came out is that Wansink had used the Joy of Cooking in one of his studies, claiming to show that since it was originally published in 1936, the calories per serving had exploded by the time it got to more recent editions. Wansink claimed that this was proof that it wasn’t just fast food ruining the American waistline, but it was also home cooking.

The current author of the book, John Becker, who is descended from the original author, Irma S. Rombauer, took it at face value at the time the study was published. They hadn’t increased the calories on purpose, but he assumed it must be true because a big-time researcher found that it was so, so who was he to argue?

But then the Buzzfeed article came out, and Becker realized that Wansink was full of shit. So, he decided to conduct his own informal study by digging into the data at his fingertips. What he found was that Wansink chose only 18 recipes to examine, out of 4,500. Wansink claimed that those were the only ones that were the same recipes from the first edition to the most recent, but Becker noted that that was a lie — or an incredibly stupid misunderstanding that no scientist would ever want to admit to committing.

Hundreds of recipes made it through from version to version, though occasionally their names would change. Meanwhile, some recipes in Wansink’s sample had the same name as an older recipe but were entirely different, as with a goulash that was clear soup with vegetables in one early edition but a sausage-filled soup with roux in the 2006 book.

So, huge surprise, another case of Brian Wansink manipulating data to get a good headline. Only this time, he fucked with the wrong family institution and got his ass handed to him.

It may be true that recipes these days tend to have more calories in them then they used to — most people are eating more than they should be, whether they’re at McDonald’s or Grandma’s. But the way to educate people about making healthier choices isn’t to make up data to deride a cookbook that happens to be pretty god damn handy. It just makes people less likely to believe scientists about food, in a time when we are already facing so much ignorance that the majority of people think they’re getting fatter because of “genetics” or “metabolism,” instead of “eating too much.” It’s like if a climate scientist or a vaccine researcher was caught manipulating data: stop giving them ammo, please.