Support more videos like this at patreon.com/rebecca!
Sorta transcript:
The popular radio show and podcast This American Life recently publicized what appeared to be an astounding piece of research that was published in the journal Science: two political scientists named Michael LaCour and Donald Green sent out thousands of surveys to gauge people’s opinions on various issues. They then had the university’s LGBT club knock on the survey respondents’ doors and have 20-minute conversations. The researchers said they found that those people were incredibly likely to change their minds on hot-button issues like gay marriage and abortion, even up to a year later.
It turns out it was too incredible to be true: two other researchers, David Broockman and Joshua Kalla, tried to replicate the study and found that things weren’t adding up. For a start, the data was too perfect. The responses people gave never deviated from the norm in a natural way. That was a red flag, but the game was over when they realized they weren’t getting the thousands of responses that the other researchers claimed to get, so they called the survey company that Brookman and Kalla suspected the other team had used. That firm said they had never heard of the first researchers, but more than that they said the techniques used in the study were not ones they had the capability to do.
Now, the lead author on the first study, LaCour, who is a grad student, is claiming he never actually paid the survey respondents as he promised but the data is real and he’ll reveal it at some future point in time.
The other author, Green, says that LaCour hasn’t given him the raw data so he can’t verify the results. He’s threatening to ask Science to retract the paper.
Outlets are reporting this as being a case of truly next-level fraud on the part of LaCour, which by all accounts it is, considering the detailed excerpts of survey responses that he included in the paper and the hundreds of hours of work that were wasted on behalf of the LGBT canvassers, who did exist and did do the job they were asked.
But this is also the story of a serious problem in scientific research publishing – Green was a co-author on the paper and he needs to take the blame for this fraud as well. LaCour was a grad student, and Green was the prinicipal investigator, or PI. A good PI should have access to all the data already. The fact that Green is telling reporters that he had to ask LaCour for the data proves that he’s not doing his job, and the result is bad science.
So keep this incident in mind whenever we talk about the benefits of science. The system isn’t perfect because humans are involved at every stage, and humans are lazy assholes. For more examples, check out retractionwatch.com and remember that scientific truth isn’t ever about a single un-replicated study. And kudos to This American Life for correcting the record!
This would’ve been extremely nice to believe if had been true; that’s why it got so media coverage. I wanted to use this study in the essay I was just editing before I read this post, actually. It argues for atheists to “come out” to their friends to improve the public’s attitudes about atheists, which is damaged so severely by the New Atheists. That’s the way it ends, anyway.
Alas, it was not to be. I used some solid (and presumable reproducible) poll results by respected pollsters instead. I think coming out has indeed had a great affect on the public’s remarkably rapid acceptance of the LGBT community and marriage equality. This used to be a reliable wedge issue for the Republican party, after all. And I do believe that personal exposure had a lot to do with that.
68% of respondents would vote for a gay president, according to a recent NBC/Wall Street Journal poll. That’s real data, which you can confirm because you can examine their data for yourself.
To be fair, a lot of mainstream media outlets publish what we would call ‘bullshit I saw on Facebook’ as if it were real news.
Ah, but if that bullshit on Facebook is repeated often enough, then it’s a trend story. ;-)
I suppose it’s like the hedge fund managers who fall for Nigerian princes. It’s fun to see them do stupid things.
Cracked has a running list of news stories that were basically BS chain letters.
My pet peeve is articles about studies or finds where there is plenty data available, but the science writer doesn’t present it because we’re not smart enough to evaluate the evidence ourselves. You’re left to scratch your head and wonder how the scientists came to their conclusions.
I think being able to calculate and interpret p should be a requirement to graduate from high school.
The study in question presupposes…
*gay people are inherently visible, as in you can tell who’s gay and who isn’t
*homophobes, having this knowledge, would not simply resort to ad hominem, e.g. “Of course you’re in favor of gay rights, you’re gay.”
“A good PI should have access to all the data already. The fact that Green is telling reporters that he had to ask LaCour for the data proves that he’s not doing his job”
I agreed with you right up until this point.
I am curious why you think a researcher should have all the data from another lab at another institution when the other lab was performing the data analysis. LaCour and Green were from different institutions. This criticism sounds like a shallow statement that comes from someone who has never worked on a multiple lab project that generates huge amounts of data.
Serious questions: how does having a bunch of raw data on a DVD on your shelf make you a better scientist? How would that stop fraud? Are you suggesting the PI should reanalyze everyone’s data every time a surprising result comes up?
My tone may seem snarky, but I don’t see you asking “Why would a PI not have all the raw data?” You are making a blanket statement that if a PI does not have all the raw data from a collaborating lab then he is not doing his job and I do not agree.
As someone who is currently working on a large NIH-funded interdisciplinary research project with people in multiple states in the US, I agree with Rebecca. The PI is responsible for the project, period. This means that they should have access to to all the data. On the project I work on, the PI, the other investigators, and the qualitative and quantitative data managers all have access to the data because it’s shared on a secured server. If anyone else on the team wanted to see it, it would be made available to them. In fact, the PI encourages everyone on the team to be involved in data analysis.
I think your reasoning–that the investigators were at different institutions–is a cop out. That doesn’t seem like a valid reason that data cannot be shared among people who are working on a project together. It literally makes no sense that only one person would have access to the data for a project that both people are working on.
It’s not clear to me from what I’ve read that Green was actually acting as the PI for this project, though. For a variety of reasons, it seems more like LaCour was the PI, and Green was a collaborator. One of the reasons I think this has to do with the reason Green claims he didn’t have access to the data: IRB* approval came from LaCour’s institution, not Green’s, and without getting additional IRB approval from his own institution he couldn’t handle the data from the project. I believe the grant funding the project was also managed by LaCour’s institution, which again points to LaCour being the PI. Usually the more senior collaborator on a project is the PI, but not always, and to me it looks like this case was an exception.
Regardless, I agree that Green isn’t fully off the hook here. Some sort of data sharing plan should have been worked out. I don’t have a whole lot of experience with human subjects research, but it shouldn’t be that great of a barrier. Yes, getting IRB approval is a pain in the ass, and if LaCour were a well-established researcher, or at least a student with whom Green had worked closely enough with to have reason to trust his character and competence, the decision to simplify things by only having LaCour’s IRB approve the protocol might make some sense. But at least from what I read, it sounds like Green didn’t even know LaCour all that well; he was serving on LaCour’s thesis committee, but I didn’t have the sense that they had worked together closely in a typical grad student-PI relationship over the course of several years. And even if logistically it was too difficult for Green to get access to the raw data, it strikes me as odd that Green couldn’t at least get access to de-identified data, which ought to have been enough for him to notice something fishy was going on. Are IRBs really that strict about researchers doing data analysis on de-identified human subject data?
(For non-academics: IRB = institutional review board, a body within a university or research institute which reviews any research protocol involving human subjects to ensure that it meets ethical and legal requirements.)
The person who is listed as the PI on the project is, technically speaking, the PI for the project. If someone else is acting as the PI, they should either be listed as the PI, or they should be listed as a co-PI. We all know he had no problem putting his role as PI on his CV, so he should take all the responsibility of being a PI. This is his fault, if for no other reason than he thought of it as a line on his CV and didn’t much care to pay close attention (which is exactly the problem Rebecca is pointing out–he wasn’t doing his job as a PI).
I don’t think the IRB would have been a barrier or issue in this. Submitting projects for IRB approval is par for the course for any social science researcher, and as much as it can be a pain in the ass, it’s really not that big of a deal in the grand scheme. If it was all approved already through another institution, it would have been even easier to get it approved at his own institution because all the justifications and such were already written, they’d just need to be input into the new institution’s paperwork.
Ah, I didn’t realize that Green had identified himself as the project PI on his CV. In that case, yes, total failure on his part. And also it’s really weird that he had PI status but let LaCour be fully responsible for not only the data collection and analysis but also the IRB and the grant, with no oversight at all. What was there left for Green to contribute?
This definitely feels then like the case of a senior author who’s happy to merely add his name to a student’s work without contributing much himself. Fortunately I haven’t encountered much of that myself, but of course you hear stories. I always think that those kinds of academics are not only behaving unethically, but also kind of stupidly; just like Green, they’re opening their reputations to be forever tarnished by a student whose competence and ethics they haven’t even bothered to vet. If I were one of Green’s colleagues, I would definitely be rethinking my assessment of the quality of his published body of work right now; even if all of his former students have behaved ethically, how many mistakes or sloppiness have slipped through due to their inexperience and Green’s lack of oversight?
Are you implying I wrote the PI was not responsible for the project? I did not. If that is what you believe then you are reading something into my post that is not there.
These are survey data. Survey data are not difficult to falsify. All LaCour would have had to do if the data were kept on servers or otherwise shared is fabricate at a different point in the process, right? All you need to do is make sure the sum of squares come out right on a bunch of numbers rated 1 to 5 (if memory serves).
If what Rebecca proposed would prevent someone as determined as LaCour apparently was from fabricating the results he wanted, explain how.
I’m not implying that at all, I’m straight up saying it. When you say you disagree with Rebecca that he wasn’t doing his job, part of the job of a PI is to monitor the project. He failed to do that. It is his responsibility to ensure the data collection and analysis is ethical and following protocol. He’s not as responsible as the person who falsified the data, but he certainly was not doing his job as a PI.
Anyone can falsify any data in any discipline, I don’t know why you think survey data is somehow less difficult to falsify than any other kind of data. There are obviously ways of getting around this, and one of them is to, you know, actually look at the data (which is what the people who discovered this falsification did). You act as if there is no possible way to notice fabricated data, yet the whole point of this post is that someone noticed fabricated data.
It’s not that simple.
“Given that I did not have IRB approval for the study from my home institution, I took care not to analyze any primary data — the datafiles that I analyzed were the same replication datasets that Michael LaCour posted to his website. Looking back, the failure to verify the original Qualtrics data was a serious mistake.” – Green
The PI for a project almost never looks at all the raw data before a publication. Hell, I’ve been on projects where that would’ve been literally terabytes of sequencing data. You’re demanding far more than what is standard practice. Furthermore, in this case, Green couldn’t look at the raw data because he was following another ethical obligation.
Could Green have done more checking up? Ideally, yes. But it’s not fair or realistic to expect researchers should do a forensic analysis on every bit of data they get from a collaborator. You’re engaging in victim blaming here. Scientific collaboration requires some degree of trust. Green was defrauded. His research collaborator flat out lied to him.
The minute he had doubt, Green asked to retract the paper. A paper in one of the most prestigious journals in the world, and he asked to have it pulled. Ultimately, he did the right thing. We shouldn’t be dragging his name through the mud.
I was very specific about what point I disagreed with. Between my response to you, my original post and my response to Rebecca’s response I have already either answered every question or accusation you are making or have never said or implied what you are criticizing. I feel I have made an honest effort to understand Rebecca’s post and your and Rebecca’s responses while you are either not reading my posts carefully or setting up straw men to attack.
If you want responses, read what I have already written. One misrepresentation of what I wrote is an error. Two is a pattern.
Maybe this will clear things up:
http://nymag.com/scienceofus/2015/05/how-a-grad-student-uncovered-a-huge-fraud.html?mid=twitter-share-scienceofus
Sorry, but this wasn’t just any old kind of publication. It was a study that claimed to be contrary to a large body of literature. Considering this, I don’t think it is expecting too much for Green to want to actually look at the data before putting his name on the publication that was obviously going to make a big splash when published.
I get that the person listed as PI often does not actually function as a PI, but the point is that’s a problem that needs to be fixed. Either let grad students be PIs and take full responsibility for their own work, or do a better job actually being a PI/co-PI on a project you are listed as having that role on. The issue Rebecca brings up is an important one–the fact that listing people as PI is often a mere formality is a problem that needs to be fixed. People need to take that job title more seriously. That criticism goes beyond just this particular case.
I also think it’s important to point out that Green agrees with Rebecca’s assessment. The IRB issue is a cop-out, it’s something he could have remedied. But, again, he was failing in his role as a supervisor. And, again, Green admits and agrees with that assessment (emphasis added):
The nitpick that John Shannonhouse is taking with Rebecca’s criticism is a non-issue because Green agrees with her assessment. We can argue about all the roadblocks that are set up that make things more difficult for researchers, but none of that actually means that Green did not fail in his officially designated role as PI, or in the case as another commenter has claimed that he wasn’t actually the PI, then his role as supervisor of the project, which in the story I linked to does not seem to be in dispute.
Are you serious? Obviously, if Green had access to the data as it was collected (and that access would obviously be in the cloud, not burned onto a DVD and then handed over in person like it’s 1999), he could see that data was actually being collected.
The storage medium doesn’t matter. It can be servers, DVDs, external hard drives, etc. The point being having access to the raw data would not prevent fraud. LaCour would just change the method to produce it. He would write a program to fabricate the raw data he wanted instead of fabricating analyzed data. The end result would be the same. It is surveys, not histological slides or something else that would take real or insurmountable effort to falsify.
I fail to see how everyone having all the raw data would prevent fraud unless everyone independently analyzed the data to look for “too clean” results. You could do that with budgeting it into research grants, of course, but that is not what you are proposing. You seem to think someone who pulled off the fraud LaCour did would be stymied by having to produce falsified survey data.
Ethically, Green couldn’t look at the raw data. We’re talking about human subjects research, and he did not have IRB approval. The separation between Green and the raw polling data was intentional. We should not fault him for being an ethical researcher.
No, ethically he should have sought IRB approval at his own institution. How in the hell is it ethical to be a PI or supervisor on a project and have no access to the data? He was negligent and it led to unethical behavior that he potentially could have prevented.
Specifically, what data should he have had access to? I’m starting to wonder if you’ve actually done any large collaborative research. Not every collaborator has access to all of the raw data. The problem isn’t that he should have had access to every grad student’s hard drive involved in the project. More data is more noise.
He could’ve asked more skeptical questions about his collaborators, but again, you have to have some level of trust. Scientific collaboration is not the courtroom.
What? This wasn’t some huge collaboration. There were two authors, LaCour and Green. Green had exactly one person to check up on and interact with, which in my field we call “mentoring.” But he didn’t.
In science journalism, I often find the phrase “one study shows … .” If you see this phrase, immediately question the results. The point of science isn’t just to have one study and call it a day.
Unfortunately, in today’s news cycle, many studies get reported before any scientist has had a chance to try and replicate the results, even if the published paper has been peer-reviewed in a reputable journal.
Oh, I’ve seen them report studies that hadn’t even been completed, much less reviewed. The authors already knew what the results were going to be. No specter of doubt was raised.
Pedantic point, but Green is more senior but was not the Primary Investigator. LaCour did all the data collection independently (which in itself is unusual for a grad student) and approached Green to write it up together. Not to say this lets him off the hook, or that coauthors don’t also have a responsibility for the data, but he wasn’t in a formal supervisory capacity over LaCour or over the data collection (as far as my reading of the story goes).
Interesting. I haven’t paid too much attention to this story, but looking around for information on the funding, it seems he made up most of the funding listed on his CV (http://nymag.com/scienceofus/2015/05/lacour-made-up-his-biggest-funding-source.html), all of which he had listed himself as PI (grad students generally aren’t allowed to be listed as sole PI, so that should have been a red flag for sure).
I agree Green has some responsibility, even if he wasn’t the PI or the advisor, because his name is on the paper. He should have asked to review the data before agreeing to have his name put on a publication.
You should read the blog article that started all this. Your impression of Green will probably change. He immediately and publicly asked for a retraction of his own Science paper. That is a very strong demonstration of integrity.
http://retractionwatch.com/2015/05/20/author-retracts-study-of-changing-minds-on-same-sex-marriage-after-colleague-admits-data-were-faked/
I did know (and said in the video) that he asked Science to retract the paper. I’m glad he admitted his fuck-up, but that doesn’t make it less of a fuck-up. A huge, embarrassing, costly fuck up.
Alice has a few beers, then decides to drive home. It’s a short route and she knows it well, and has driven it after a few beers several times before, and never had any problems. But tonight, Bob is cycling along her route, and because of her impairment, she fails to react quickly enough when she sees him and hits him, injuring him badly. Alice is horrified and realizes the magnitude of her mistake, and rather than hit-and-run, she immediately calls 911, ensures Bob gets to the hospital, turns herself in to the police, pleads guilty at the trial, and willingly pays all of Bob’s medical bills.
Alice’s actions after hitting Bob demonstrate integrity. But her decision to drive drunk was still a moral lapse.