Researching Honesty Part 2: Can innocent mistakes explain the data?

[Kristin] (0:00 - 0:30)
Must be.

[Regina]
Must be. That's crazy.

[Kristin]
It's totally crazy, right? I mean, it's bonkers. Just because there are two columns does not prove that they have to measure different things.

You can have two columns with the same data in it.

Welcome to Normal Curves. This is a podcast for anyone who wants to learn about scientific studies and the statistics behind them.

I'm Kristin Sainani. I'm a professor at Stanford University.

[Regina] (0:31 - 0:36)
And I'm Regina Nuzzo. I'm a professor at Gallaudet University and part-time lecturer at Stanford.

[Kristin] (0:37 - 0:42)
We are not medical doctors. We are PhDs. So nothing in this podcast should be construed as medical advice.

[Regina] (0:42 - 0:47)
Also, this podcast is separate from our day jobs at Stanford and Gallaudet University.

[Kristin] (0:47 - 1:03)
Regina, today we're going to cover part two on Francesca Gino, the Harvard professor whose tenure was recently revoked for research misconduct. If listeners haven't yet heard part one, they may want to go back and listen to that episode. But Regina, maybe we can just give a quick recap.

[Regina] (1:04 - 1:13)
Yeah. So Gino was a superstar researcher at Harvard Business School, and she studied behavior in the workplace, including honesty and morality.

[Kristin] (1:14 - 1:31)
She was very prominent. She reportedly earned over a million dollars a year at Harvard and commanded enormous speaking fees and wrote a bestselling book called Rebel Talent, Why It Pays to Break the Rules at Work and in Life. And that title, yes, is a bit ironic.

[Regina] (1:32 - 1:51)
It is. And a group of data sleuths called Data Colada found data anomalies in four of her published papers. They alerted Harvard, which then conducted their own investigation, and all four papers have since been retracted.

[Kristin] (1:51 - 2:04)
And Harvard's multi-year investigation concluded that Gino had committed research misconduct. They revoked her tenure in 2025, and that's pretty extraordinary. It hasn't happened in modern times before.

[Regina] (2:05 - 2:29)
So Harvard had dealt with research misconduct before involving tenured faculty. Mark Hauser, for example, had resigned after Harvard found misconduct in his research. And according to Gino, Harvard also encouraged her to resign quietly and offered her financial terms to leave, but she refused.

She said she had not committed research fraud.

[Kristin] (2:30 - 2:33)
Right. She decided not to walk away quietly. She is choosing to fight it.

[Regina] (2:33 - 3:00)
And fighting Harvard is enormously expensive. Billionaire Bill Ackman has publicly acknowledged funding her legal defense, a billionaire, and Harvard law professor Lawrence Lessig has also been advising her pro bono. He devoted nine episodes in his podcast to defending her.

His podcast is called The Law Such As It Is.

[Kristin] (3:00 - 3:35)
Regina, in our last episode, part one of our measly two-part series on this case, we gave the background and chronology of events, and you dug into the data anomalies in one of the four problematic papers, that honesty pledge paper, but you did not address Lawrence Lessig's criticisms. So today I am going to directly address his defense of her and also dig into detail into a second paper, the evil genius paper. And Regina, I have to admit the last episode we did ended up sending me down a rabbit hole.

[Regina] (3:35 - 3:37)
I love your rabbit holes, Kristin.

[Kristin] (3:37 - 3:50)
I originally had a different topic planned for episode 43, but when you told me about Lawrence Lessig's podcast, I ended up listening to the entire series about at least three times now. I couldn't put it down.

[Regina] (3:52 - 3:54)
Because, what, it was so compelling?

[Kristin] (3:55 - 4:13)
More like that feeling when you drive past a car wreck, you know you shouldn't stare, but you just can't look away, that kind of thing. Oh my. Regina, I do appreciate that Lessig says right at the beginning that he is biased.

So he is friends with Gino, he fiercely believes her to be innocent, so he does not claim to be impartial.

[Regina] (4:14 - 4:16)
And we do appreciate that transparency.

[Kristin] (4:17 - 4:40)
Absolutely. But Regina, I should warn you and our listeners, there may be some ranting in this episode and my blood pressure may go up a little. I am approaching this from not a lawyer's perspective, but from a statistician's perspective.

And I have to say, from that perspective, some of the arguments that we are going to discuss, I think they're going to make your blood boil, Regina.

[Regina] (4:41 - 5:09)
But Kristin, you cannot get us into trouble, please. So I am going to be the careful one here. I'm going to try to make sure we don't get sued.

Just as a reminder to listeners, Gino sued the Data Colada folks for $25 million. And now the court dismissed her claims against them. But still, defending a lawsuit is enormously expensive.

So let's not go there.

[Kristin] (5:09 - 5:31)
Let's not go there, Regina. Agreed. You know, Regina, Lessig actually has a guest expert on his podcast who he calls Ava.

But this guest is anonymous. And Lessig has disguised his voice with an AI voice. I'm wondering, should we be doing this podcast anonymously with AI voices, disguising our voices, Regina, to avoid trouble?

[Regina] (5:32 - 5:39)
It might be too late. I think we have given ourselves away already. All right.

[Kristin] (5:40 - 5:58)
So instead of disguising ourselves, we're just going to stay in our lane and talk about the data. We are not lawyers, so we are not going to address the legal aspects of this case. And in fact, a lot of Lessig's defense of Gino has to do with procedural issues rather than data issues.

[Regina] (5:58 - 6:00)
Yeah, procedural issues like what, Kirsten?

[Kristin] (6:01 - 6:29)
So he mentions over and over that Harvard has a rule, with some exceptions, that they don't investigate claims of academic misconduct that are more than six years old. And three of Gino's four disputed papers are more than six years old. Now, maybe her papers meet Harvard's exceptions, and maybe this isn't a good rule to begin with, but this may be a valid legal point.

And Lessig also brings up a lot of other procedural issues, like how much time Gino had to review things.

[Regina] (6:29 - 6:50)
Yeah. So you're saying these procedural issues could be important, right? And some might affect how confident we are in Harvard's conclusions.

But today, we're just going to focus on actual evidence of data alterations and whether Lessig's explanations hold up. Exactly.

[Kristin] (6:50 - 7:26)
We're going to focus on the substance of the allegations rather than the process by which they were adjudicated. Also, Regina, we cannot cover everything that Lessig covered because he had nine episodes, and I'm only getting one here. So I'm only going to dive into detail on one of the four papers.

And Regina, of course, we always have a claim for each episode, and we have to be a little creative when we are discussing the scientific process more than any specific scientific claim. So here's what I came up with for today's episode. Superstar scientists are too important to clean their own data and too smart to commit fraud that's easy to catch.

[Regina] (7:29 - 7:31)
Okay, that is an interesting one.

[Kristin] (7:32 - 7:38)
I had to be creative. And you know what? Creativity is one of the themes of the paper we're going to discuss today.

[Regina] (7:39 - 7:39)
Good point.

[Kristin] (7:39 - 7:53)
All right, Regina, there are some things that everyone, I think, agrees on in this case. So everyone seems to agree, even Gino's defenders, that the four papers in question contain very serious data anomalies.

[Regina] (7:53 - 8:09)
Right. And these are not trivial anomalies. They raise serious questions about integrity of the data, whether we can trust published findings.

And like I said, all four papers have now been retracted.

[Kristin] (8:09 - 8:39)
Exactly. What is in dispute is how those anomalies arose. Were they accidental, deliberate, or both? And who was responsible?

Gino, someone else, or multiple people? So let me start, Regina, by just giving Lessig's broad, overarching defense before we get into the particulars of the Evil Genius paper. Big picture, Lessig argues that all the data anomalies resulted from innocent mistakes and well-intentioned decisions by research assistants rather than deliberate fraud by Gino.

[Regina] (8:39 - 8:44)
That would be a lot of innocent mistakes. Yeah.

[Kristin] (8:44 - 9:09)
To put it bluntly, Lessig and Ava paint a picture of Gino's research practices that maybe avoids deliberate fraud, but if true, is not a flattering picture of her as a scientist. So they describe her as having, quote, an army of research assistants. And it was those assistants who, quote, cleaned her data.

And in this chaos, some of them introduced the anomalies.

[Regina] (9:09 - 9:13)
Okay. So when you're saying cleaning the data, what do they mean?

[Kristin] (9:13 - 9:44)
Right. So they mean everything from entering the data, merging datasets, checking the data, fixing mistakes, processing questions, you know, making derived variables, everything like that. And Lessig and Ava repeatedly say that Harvard Business School professors don't clean their own data, you know, as though data cleaning is some menial, unimportant task that is beneath a Harvard professor.

And my reaction to that as a statistician is, what the heck are you talking about? This picture is disturbing on so many levels.

[Regina] (9:44 - 10:00)
Kristin, let's just stop for a minute and maybe dissect your rage here. Because for listeners who are not as familiar with exactly how statistics is done in practice, this might seem like a reasonable claim on its face.

[Kristin] (10:00 - 10:38)
Well, research assistants absolutely do have legitimate roles in research, like collecting the data, entering the data if it's from a paper form, or coding open-ended responses or responses that need some judgment. But Regina, most data cleaning and processing should not be done manually by humans. We have statistical software for this.

So you take the raw data, you preserve that raw data, and you use code to do things like, you know, calculate the total scores, rename variables, classify the participants. And doing this in code, it saves a ton of time and it greatly reduces mistakes.

[Regina] (10:38 - 11:21)
Right. This is the thing for me. It's really about that documentation, because code can prevent all those manual mistakes, like when you accidentally overwrite a cell, you know, in Excel or duplicate a row or something.

But the thing is, if you keep the script and the original data like you should, you can go back and see exactly what you did. You can check it, you can correct it, you can reproduce it. And, Kristin, very relevant for today, you can, of course, even go back and defend yourself against accusations of fraud, because it's all right there.

The thing is, you should not end up with a final data set that has some mysterious history that no one can actually explain.

[Kristin] (11:22 - 11:35)
Exactly. I mean, Lessig and Ava make it sound as though this chaotic, error-prone, army of research assistants approach that they describe is standard scientific practice, but it is not.

[Regina] (11:35 - 12:06)
I mean, delegating data preparation is not necessarily a problem. The PI does not have to do all the coding herself, right? That is what your statisticians and your trained research staff are there for.

But the thing is that those people doing the data processing should be working closely with the researchers who really understand the study, the point of it, to make sure that the data are actually processed correctly and that all those decisions you're making make sense for the study.

[Kristin] (12:06 - 12:25)
Exactly. And ultimately, the study investigator, especially the principal investigator like Gino was, is responsible for the integrity of the data. So the research assistants ate my homework defense. Not great.

And, Regina, we were at the JMP, the JMP Statistical Software Summit last week.

[Regina] (12:25 - 12:37)
Which was so much fun, so illuminating. Thank you, Anne Milley, one of the fans of the podcast, for inviting us and hosting us as VIPs. Loved it.

[Kristin] (12:37 - 13:26)
It was fabulous. And, Regina, I feel like we've got to put in a plug for the podcast that Anne hosts, which is Stat Speaking with JMP. We've been guests on it.

But this conference brought together data analysts, but also engineers and scientists who deal with data. And, of course, I was telling everyone about this army of research assistants defense, and everyone I talked to was just as appalled as I was. Another major argument Lessig seems to make is he points to some data anomalies that do seem more likely to be innocent sloppiness than fraud.

And he implies that the existence of one innocent mistake must mean that they are all innocent. But obviously, that logic is flawed because a data set can contain both accidental errors from sloppy data handling and deliberate changes intended to improve the results. Those two are not mutually exclusive.

[Regina] (13:27 - 13:43)
They're not mutually exclusive and not binary. Even if some anomalies do turn out to be innocent mistakes, this does not automatically explain away all the other anomalies. You got to look carefully at each one and also at the overall pattern.

[Kristin] (13:44 - 13:59)
Exactly. And with all of this in mind, Regina, let's now dig in to the specific disputed paper that I want to review today. It's called Evil Genius, How Dishonesty Can Lead to Greater Creativity.

It was published in 2014 in Psychological Science.

[Regina] (13:59 - 14:01)
I love that title. Just so creative.

[Kristin] (14:02 - 14:18)
She is creative, right? Creative titles. I want to point out something interesting right off the bat.

There were only two authors on this paper. So it was Francesca Gino from Harvard and Scott Wiltermuth, who was from the University of Southern California. There are no research assistants who are co-authors on this paper.

[Regina] (14:18 - 14:21)
Okay, noted. So what did the paper show?

[Kristin] (14:21 - 14:24)
The paper claims to have found that dishonest people are more creative.

[Regina] (14:27 - 14:32)
The idea being that cheating might somehow get your creative juices flowing.

[Kristin] (14:33 - 14:48)
I think that's it. Yes. And, you know, the paper included five experiments, but we're just going to focus on experiment four because that's the one where the data anomalies occurred.

So in experiment four, they recruited 178 people through Amazon Mechanical Turk.

[Regina] (14:48 - 14:59)
Amazon Mechanical Turk is pretty cool. It's this online thing where people get paid to complete little tasks and surveys. Researchers love it.

[Kristin] (14:59 - 15:48)
Yeah. In this case, participants were paid $1 to participate, but they had an opportunity to earn an additional $1 bonus. So here's how they tried to get the participants to cheat, Regina.

They asked the participants to predict the outcome of a virtual coin toss, heads or tails, and then they had to enter their prediction into the computer. Then they had to click a button to flip a virtual coin, and they were asked whether or not they had guessed correctly. And if they said yes, they got an extra dollar.

But from what we can tell, the computer actually always showed them the opposite of what they guessed. So in other words, nobody predicted correctly. So if they claimed to have guessed correctly and claimed that extra dollar, then they had lied.

They were cheaters. And actually, that is the independent variable of the study, whether or not they cheated on the coin toss task. That's important.

Remember that.

[Regina] (15:48 - 16:05)
Okay. It's actually pretty clever. So they're comparing people who cheated, who lied about being psychic, with people who did not cheat, who did not lie.

And the outcome variable was what?

[Kristin] (16:05 - 16:34)
Right. So the outcome variable here was creativity, which they measured using two different tasks. So one was called the remote associates test, or the RAT.

I'm just going to call it the RAT. And this is a test where you're given three words, and you have to come up with a fourth word that's associated with the other three. So I'm going to give you a test now, Regina.

Here we go. Ready?

[Regina]
Oh, no.

[Kristin]
Okay. Here are your three words, belly, barrel, and root. What is a fourth word associated with all three?

[Regina] (16:34 - 16:39)
Oh, beer. Beer belly, beer barrel, root beer.

[Kristin] (16:40 - 16:42)
Beer. Very good. Excellent. Good job. Okay.

Let me give you one more.

[Regina] (16:42 - 16:46)
I don't even love beer, but I came up with that one right away.

[Kristin] (16:46 - 16:47)
You're fast. Very good.

[Regina] (16:47 - 16:48)
Thank you.

[Kristin] (16:48 - 16:56)
I'll just let all listeners know, I did not give it away to Regina ahead of time. All right. One more for you.

Okay. Measure worm video.

[Regina] (16:56 - 17:15)
Oh, that one's a little harder. It is not, you know, whiskey. Measure worm video, inchworm.

Inch, measure, video, inch. Nope. Earthworm, measure earth.

No. Tape. Tapeworm.

Tape measure, video tape.

[Kristin] (17:16 - 17:53)
Tape. Yes. Excellent.

You're good at this. Very good.

[Regina]
Took me a little while though.

[Kristin]
No, you have time to think about that. These are meant to, you know, you have to think about them. You're given a time limit, but you get some time to think.

So they gave participants 17 of these word triplets and their total RAT score, that's the outcome of the study, was the number that they got right out of 17. So that was one of the creativity tasks. They also gave them another creativity test called the alternative uses task.

And I'm going to make you try this one out too, Regina. So I want you to come up with as many creative uses for a newspaper as you can. Now, participants in this study got one minute to do this.

I'm going to give you 10 seconds. Go.

[Regina] (17:53 - 18:11)
Well, that is not fair. Okay. I need some beer if I'm going to do this.

Newspaper. Bottom of the birdcage. A hat.

You can read it. You can use it when you're sitting in a bar to ward off guys that come up and talk to you and pretend to read it.

[Kristin] (18:12 - 18:14)
I love that one.

[Regina] (18:15 - 18:28)
Halloween mask. You can use it to dry out your hiking boots after you have waded through rivers. You can use it to wipe your dog's butt.

[Kristin] (18:29 - 18:57)
Okay. I've given you more than 10 seconds, but they were so good. I let you keep going.

My favorite use is you stick it in your running shoes, like the hiking shoes to soak up the water. Yep. So they had the participants list as many of these as they could in one minute.

And they actually, this was turned into three different outcome variables. We're going to focus just on how many unique uses they came up with, five or seven or 12. But they also had these other two measures, the originality and flexibility of the uses.

[Regina] (18:58 - 19:07)
Oh, now I want you to rate me on all of those. I think I get extra bonus points for the unwanted bar.

[Kristin] (19:07 - 19:21)
I think you do. Yeah. The problem is they never really say in the paper exactly how they determined originality and flexibility. There's no rubric, but we can count up the number of uses, which you got quite a few.

But yes, I would give you high originality on the bar one. Absolutely.

[Regina] (19:21 - 19:26)
Thank you. Thank you.

Okay. So what were the conclusions, the results that they find?

[Kristin] (19:26 - 20:13)
So they reported that 43 out of the 178 participants cheated on the coin toss tasks. That's about a quarter, pretty high. Wow.

And they reported that the people who cheated, in fact, performed better on all four creativity measures than those who did not cheat. So on that RAT, the remote associates test, the cheaters got an average of 9.5 out of 17, correct, versus 7.8 for the non-cheaters. And that was statistically significant p-value of 0.01. And on the newspaper task, cheaters came up with an average of 8.3 uses compared with only 6.5 in the non-cheater group. P-value there was less than 0.0001. So the authors interpreted this as evidence that dishonesty can actually promote creativity.

[Regina] (20:13 - 20:17)
Oh, that is sexy. That is a sexy finding. Yep.

[Kristin] (20:18 - 20:29)
Right? Creativity and dishonesty go together. That gets great headlines.

It's exactly the kind of result that attracts attention and gets fame and glory for the researcher publishing it.

[Regina] (20:30 - 20:30)
Media love it.

[Kristin] (20:31 - 21:05)
Media love it. All right. So that's the paper.

Now, Regina, let's get into what the Data Colada statistical detectives found. So first of all, interesting. The Data Colada detectives actually got a copy of the dataset from this paper way back in 2014.

[Regina]
Oh, 2014. How did they get it way back then? Yeah.

[Kristin]
So after this paper was published, it appears that Leif Nelson, one of the Data Colada detectives, requested the data for a journal club at Berkeley. And he did receive a CSV file from Gino.

[Regina] (21:05 - 21:12)
Okay. But did Data Colada notice all of these suspicious observations that we're going to talk about? Did they notice them back then?

[Kristin] (21:13 - 21:50)
It's unclear, actually, what they did with the data back then. So according to a 2023 article in The New Yorker, Data Colada had harbored suspicions about Gino's research as early as 2014. But I don't know if those suspicions were related to that specific paper or dataset.

Regina, though, also according to The New Yorker, researchers had been raising concerns about Gino's work as far back as 2012. And in 2015, a Harvard graduate student even raised concerns about possible data manipulation. But it wasn't until 2021 when another researcher brought new concerns to Data Colada that they really launched into a full investigation.

[Regina] (21:51 - 21:58)
Wow. So you are saying there had been questions about some of her research long before the Harvard investigation.

[Kristin] (21:59 - 22:10)
Yeah, exactly. And I mean, these kinds of whispers, of course, don't mean she's guilty of anything. But it does help explain why Data Colada eventually dug pretty deep into her work.

[Regina] (22:10 - 22:15)
Yeah, yeah, it does. Okay, so let's talk about what they found in that evil genius paper.

[Kristin] (22:15 - 22:22)
What they found was that some observations, and we've seen this play before, were out of order in the dataset in a suspicious way.

[Regina] (22:22 - 23:03)
Oh, this is like an oldie but a goodie now at this point. And one way that data detectives can detect possible data fraud or, you know, quote, data anomalies is by spotting these values in the dataset, these rows that appear out of order in an otherwise tidy and neatly sorted dataset. And that suggests that maybe someone sorted the data first and then manually changed some values.

And we saw this in the Honesty Pledge episode, the last episode. But what was out of order in this 2014 dataset?

[Kristin] (23:03 - 23:46)
So the dataset was sorted first by a column called Cheated, which coded a zero for non-cheaters and a one for cheaters, and then by the number of uses that they came up with for the newspaper. So, you know, imagine you open up a spreadsheet, all 135 non-cheaters are together at the top, followed by all 43 cheaters down at the bottom. And then within each of those groups, the number of newspaper uses is sorted from lowest to highest.

For the 135 non-cheaters, those values were perfectly sorted. But for the cheaters, they're almost perfectly sorted. But some of the numbers went something like this.

You're going down the rows and you got 3, 3, 4, 4, 4, 5, 5, 14, 6, 6, 6. You see the problem?

[Regina] (23:47 - 24:06)
I do. 14, out of order. What is it doing?

Did it just wander in there among the fives and the sixes? Right. So if someone had sorted the spreadsheet by the number of newspaper uses, that 14 should be much further down with the other high values and somehow it wandered up there.

[Kristin] (24:07 - 24:22)
Exactly. And Data Colada found 13 observations like this. And here is what is especially suspicious about this. All 13 out-of-order values were in the cheater group and all were higher than you'd expect based on their position in the sorted data.

[Regina] (24:22 - 24:38)
So yeah, that is highly suspicious. All 13 of them make the cheaters look more creative, right? Which is a pretty remarkable pattern to get just from some random mistakes, wouldn't it be?

[Kristin] (24:39 - 25:18)
It is. And the kind of cool part about this though, Regina, I actually have my students do this exercise in class because since the rest of the data are sorted, we can actually make educated guesses about what values those 13 out-of-order observations might originally have had, you know, assuming that they got changed after sorting. So for example, if a value of 14 appears between a five and a six, then we could guess that the original value was either a five or a six.

So what Data Colada did is they reconstructed all the possible original values for those 13 observations and they reran the analysis comparing the cheaters to the non-cheaters and Regina, guess what happened?

[Regina] (25:19 - 25:24)
Hmm, let me guess. The result was not quite as impressive as it was in the published paper.

[Kristin] (25:24 - 25:49)
You got it. It wasn't just less impressive, it was no longer statistically significant. So remember, the original p-value was less than 0.0001. With the reconstructed values, the p-value was somewhere between 0.18 to 0.29, depending on exactly how you assign the values when, you know, sometimes it was between five and six, so you'd have to choose one, all of those not statistically significant.

[Regina] (25:49 - 26:03)
Wow, yeah, that is a big difference in p-values. So under their reconstructed values, that significant difference between cheaters and non-cheaters just disappeared. Yeah, that's pretty suspicious, I'll give you that.

[Kristin] (26:03 - 26:18)
It is, but it turns out this is actually not the biggest smoking gun in the data set because in Harvard's investigation, they were actually able to retrieve earlier versions of this data set from Gino's laptop.

[Regina] (26:18 - 26:26)
Which is super exciting. So what was it, Harvard asked her to turn in her laptop as part of their investigation? Remind me how they got it.

[Kristin] (26:27 - 26:52)
Yeah, and Lessig talks about this on his podcast, actually. He mentions that, you know, one morning basically she woke up and they asked her without warning to appear at some office with all of her devices. So because of that, the investigators were able to recover earlier versions of the data set, which means they were able to examine changes across different versions of the data that Data Colada, of course, could not see.

And what they found was more troubling than just 13 out of order observations.

[Regina] (26:53 - 27:32)
I cannot wait to hear, but first let's take a short break. Welcome back to Normal Curves. Today we're discussing the Francesca Gino case and we were just about to hear about the data sets Harvard investigators recovered from Gino's laptop.

So Kristin, what did they find?

[Kristin] (27:32 - 28:11)
So the Evil Genius study was conducted in November of 2012 and investigators recovered an Excel file dated just hours after the study ended. They also recovered some later versions of the data as we'll talk about. Now, importantly, this 2012 data set, it was not the original raw data downloaded from the survey software Qualtrics, which was used for this survey.

It was an already processed spreadsheet that already had formulas and calculated variables in it. The original raw Qualtrics data were no longer available by the time Harvard conducted its investigation. They were not able to find that.

But this Excel file was saved actually just 11 hours after the study ended.

[Regina] (28:11 - 28:23)
Oh, that is not bad. So this is the closest surviving version we have to the original data and just saved 11 hours after the real thing.

[Kristin] (28:24 - 29:28)
Exactly. Now, I should say Harvard has not made this data set publicly available. So Data Colada doesn't have access to it and neither do we, which means I do not have complete information. But I was able to scrape a lot of details from Harvard's 1300-page report on the Gino case that they released in 2023 and also from a newer 11-page report that they released in 2025.

So what we know is that in the 2012 data set, there were two columns that stored the information about whether the participant was a cheater or not. One column is called Reported Guessed Correctly. So Regina, remember in that coin toss task, anyone who reported to have guessed correctly was lying.

So this column codes those people as one, indicating a cheater, and everyone else is a zero, indicating a non-cheater. Now, toward the end of the data set, there is also another column that is called Cheated. And what Harvard found is that there was a formula in the Excel spreadsheet that simply copied the values from the Reported Guessed Correctly column over to the Cheated column.

So Regina, what would you might have happened here?

[Regina] (29:29 - 29:57)
Okay. So maybe they just copied that entire column, that entire variable, and just gave it a simpler name, right? So Cheated is certainly easier to understand than Reported Guessed Correctly, which kind of falls out of the cell in Excel.

You can't even read the whole thing. Or maybe they wanted the variable somewhere else in the spreadsheet, right? Sometimes I like to move things around to get it closer to the thing I'm analyzing.

[Kristin] (29:58 - 30:26)
Right. Regina, to me, this is not some big mystery, why there would be two columns that hold the same data. Like, we do this all of the time.

But I want you to hold that thought because the existence of the two columns is going to be central to Lessig's argument. Here's the funny thing that Harvard discovered, though. The Excel formula should have made the two columns identical, right?

It's just a copy function. Yet it appears that someone later changed some of the values in the Cheated column from 0 to 1 because there was more 1s in the Cheated column.

[Regina] (30:26 - 30:30)
And do we know when those changes were made?

[Kristin] (30:30 - 31:01)
That's not entirely clear, actually. Harvard found multiple versions of the data set on Gino's computer, including one saved in July of 2013. And Harvard says that the changes occurred somewhere between the earliest recovered version of the data and this July 2013 version.

By then, the original Reported Guessed Correctly coin toss column contained 31 cheaters, whereas the Cheated column contained 43 cheaters. So that's 12 additional people classified as cheaters.

[Regina] (31:01 - 31:12)
Okay. So somewhere between those data sets, 12 people who were originally classified as non cheaters ended up being classified as cheaters.

[Kristin] (31:13 - 31:20)
Exactly. We can definitively say that. Twelve participants who answered honestly on the coin task were nevertheless coded as cheaters in the Cheated column.

[Regina] (31:21 - 31:31)
That is suspicious. So were those 12 related to those 13 out of order participants that Data Colada discovered that we just talked about?

[Kristin] (31:32 - 31:35)
No. These appear to be totally independent anomalies in the data.

[Regina] (31:35 - 31:41)
Could those 12 additional cheating classifications, could they have been mistakes?

[Kristin] (31:41 - 31:57)
It seems unlikely. And as we'll see, even Lessig doesn't try to argue that this was accidental error, because here's the striking thing. All 12 people whose classifications changed from non-cheater to cheater had high scores on the RAT creativity test.

[Regina] (31:58 - 32:10)
Oh, wow. Okay. So all 12 people whose classifications changed had high RAT scores, creativity scores.

Yeah, that's a pretty striking pattern.

[Kristin] (32:10 - 32:26)
And in the 2014 dataset that I am able to access from Data Colada, the Cheated column sits right next to the total RAT score column. So it kind of seems like someone could easily have identified participants with high RAT scores and changed some of them from zero to one.

[Regina] (32:26 - 32:35)
And so reclassifying them as cheaters would increase the average RAT creativity score of the cheaters.

[Kristin] (32:36 - 32:56)
Exactly. And remember, the paper reports a RAT score of 9.5 in the cheaters versus 7.8 in the non-cheaters, P equals 0.01. But if you define cheating based on the original coin task column, the cheated group has a mean of 8.4 versus 8.2 in the non-cheater group, P value of 0.8, not statistically significant.

[Regina] (32:57 - 33:22)
P value of 0.8? Nope. Okay.

So instead of that big difference that they reported in the paper, the two groups now have almost exactly the same average RAT score, same creativity score. There's no longer a statistically significant difference. But I'm curious, so Data Colada did not notice this in their inspection of the data?

What happened?

[Kristin] (33:22 - 33:45)
They did not because in the 2014 dataset that Gino sent to Data Colada, someone had altered the coin task column to match the cheated column. So for those 12 people who had originally had a zero in the reported guess correctly, the coin toss column, but a one in the cheated column, someone changed the zeros in the first column to ones to make the two columns match.

[Regina] (33:45 - 34:01)
Whoa. Okay. So the later changes made those two columns agree again.

I can see why Harvard said, hmm, that looks like an attempt to cover up earlier changes.

[Kristin] (34:01 - 34:28)
It is highly suspicious, yes. And that's not all Harvard found. So remember how the RAT works, participants had to come up with 17 different words, which they got either right or wrong.

So in the dataset, there are 17 columns for the 17 RAT tasks, and they all have zeros or ones to indicate you got it right or not. And there was a formula built into that 2012 Excel spreadsheet that added up those 17 columns to come up with the total RAT score.

[Regina] (34:28 - 34:34)
Oh, a formula. Okay, good. Someone was doing that automatically and not manually. Yay.

[Kristin] (34:35 - 35:01)
Not hand adding, yes. Not doing this in their head. That was good. But Harvard's investigation found that there were four sums that were manually altered by the July 2013 dataset.

So most of the cells had this formula summing up the 17 preceding columns, but for four people, it appears that someone manually overrode the sum. So for example, for the participant with ID 77, the individual items all added up to zero. They had none correct.

But in the total RAT score column, they had a score of 14.

[Regina] (35:02 - 35:07)
Whoa, wait a minute. Okay, zero correct answers, but the total score says 14 out of 17?

[Kristin] (35:08 - 35:17)
Yeah, there were four participants like this, and they were all in the cheater group. And so the other three participants' scores were increased from 2 to 15, 4 to 14, and 3 to 13.

[Regina] (35:17 - 35:25)
Okay, so that also made the cheaters group look much better on the RAT test than the non-cheater group.

[Kristin] (35:25 - 35:35)
Exactly. If you use the original scores for those four from the actual summing of the columns, you still do get a significant difference between cheaters and non-cheaters, but just barely. The p-value goes from 0.01 to 0.04.

[Regina] (35:35 - 35:40)
And Data Colada did not find this either.

[Kristin] (35:41 - 36:00)
No, because in the 2014 dataset, the 17 individual columns had been altered for these four participants. So for example, for ID77, they originally had all zeros in the 17 columns, but the total column said 14. Yet in the 2014 dataset, now they had 14 ones in those 17 columns.

[Regina] (36:02 - 36:11)
So instead of correcting the total to match the individual answers, someone changed the individual answers to match the total.

[Kristin] (36:12 - 37:09)
Yes, that is what it looks like. And remember, at this point, the paper was already published, right? So you don't want to change the totals.

Now, to be fair, we don't know for sure what happened, right? Participants had to type in answers like thief and beer and paper. And maybe if people misspelled things, these could have been marked wrong originally and then corrected later.

But again, it's weird because let's say that one of the Army of Research Assistants, as Lessig will propose, was cleaning the data and they noticed that some people didn't get credit for their answers because of misspellings. You would think that they would fix the individual columns, right? If you misspelled beer as a grizzly bear, then you might change a zero to a one in that column.

It's a lot harder to explain why the sum was changed, but not the individual questions. And then somehow in 2014, suddenly when the data was going to be shared, those individual questions magically are changed to match the sum. This is at least very poor data management.

[Regina] (37:09 - 37:29)
Yeah. And of course, the frustrating thing, the really frustrating thing is we don't have those original raw responses in Qualtrics to check against, right? If we had those, we could see what people actually typed and whether those changes were legitimate corrections, but we don't have it.

[Kristin] (37:30 - 37:52)
Exactly. We really need that and we don't have it here. We ought to have it. Those things ought to be saved, but okay, we don't have it.

Now, Harvard did find some other anomalies. So by November 2013, another 18 participants' answers had been altered from earlier versions of the data. For this one, there's some evidence that data from a different study may have been copied into this spreadsheet.

[Regina] (37:53 - 37:58)
Data from another study, why on earth would someone copy data from another study?

[Kristin] (37:59 - 38:49)
Okay, there may be an innocent explanation here. So if the studies used similar tasks, you can imagine that someone might have copied over the columns with the variable names in the formulas, right? So they didn't have to reconstruct those and maybe they accidentally left some of the old values in place.

I'm not saying that's what happened, but I could build a plausible story around that. But Regina, even if let's say those 18 changes were indeed accidental, just a mistake, sloppy data handling, that would not explain the other anomalies that we've discussed, right? A dataset can contain both innocent errors and intentional manipulation.

So evidence that maybe this one was accidental does not rule out that the other ones were deliberate, right? Harvard found one other anomaly that's relevant here. Seven participants' written responses to the newspaper creativity task had been changed between the earliest dataset and the 2013 version.

[Regina] (38:49 - 38:52)
You mean the actual text that they wrote?

[Kristin] (38:52 - 39:20)
Yes, the text fields, which had lists like clean up after the dog, read, make a paper boat, avoid unwanted attention at a bar. Right. So for six of those seven participants, the responses had been swapped between cheaters and non-cheaters.

So a highly creative answer originally given by a non-cheater was now attributed to a cheater, while the cheater's less creative answer was now attributed to the non-cheater.

[Regina] (39:20 - 39:22)
Oh, that is suspicious.

[Kristin] (39:24 - 39:42)
That is suspicious. Like, why are we swapping rows of data? Yes.

And here's why I find this particularly important. In the 2014 dataset that we have, we don't actually have any of those text fields. So this makes it a lot harder to sleuth.

But thanks to Harvard's findings, we now know that some swaps occurred.

[Regina] (39:42 - 39:57)
Yep, that is important. But Kristin, let's go back to those 13 out-of-order values, the data anomalies that Data Collado originally flagged. Let's go back.

What happened with those?

[Kristin] (39:57 - 41:22)
Right. These are interesting, actually. Harvard did not spend a lot of time on these observations in the end, and Lessig does not address them at all in his podcast, possibly because Harvard doesn't emphasize them. So I am just going to speculate as to what I think happened here, though I can't be sure.

So we know that the dataset was sorted, remember, the sorting was first by cheating status, and that was 135 non-cheaters and 43 cheaters. And then it was sorted by the number of newspaper uses within each of those two groups. To me, this strongly suggests that the 12 people whose cheating status had been changed, that those were changed first, before any changes were made to the newspaper use variable, because otherwise those cheaters would not have been grouped with the other cheaters, right?

So I think the newspaper use scores would have been changed second, right, after you had already sorted by cheaters. So my theory about what happened is someone may have changed those 12 non-cheaters to cheaters because they had high RAT scores. And that may have produced a statistically significant difference between cheaters and non-cheaters on the RAT test, but not on the newspaper use task.

Remember, Data Colada showed that without those out-of-order suspicious newspaper use scores, the difference between the groups was not significant. So I'm speculating that maybe two steps happened here. First, someone changed the cheating classification to get a significant RAT result, but then they also had to increase some of the newspaper use scores in the cheater group to get a significant result there too.

[Regina] (41:22 - 41:29)
Oh, that is very interesting. But why did Harvard not emphasize all of this?

[Kristin] (41:29 - 42:12)
Yeah, because back in those earlier datasets, the text field underlying the newspaper task was still available in the dataset. And of course, that column is no longer present in the 2014 dataset we have, so I don't have access to this, but Harvard was able to compare the text field to the count column. And guess what, Regina?

They actually match. So for example, ID 63, that was one of the out-of-order values, that person had 14 uses of the newspaper, but they were located between rows that had a 7 and an 8 for newspaper uses. But their text field actually had 14 responses.

Kill bugs, start fire, pillow, fan, door stop, insulation, clean, litter, toilet paper, hat, decorate, read, weapon, keep light out. I don't use a newspaper as a weapon, but okay.

[Regina] (42:12 - 42:18)
Weapon, yeah. Well, I don't know. Unwanted attention at the bar is kind of a weapon against that.

[Kristin] (42:19 - 42:21)
That's true, yeah. But that's 14. So those matched.

[Regina] (42:22 - 42:38)
Yeah. Okay. That is an important check, right?

Because if someone had just increased the numbers without changing the written responses, then the numbers would not match the list. So the fact that they do match makes that simple explanation less convincing.

[Kristin] (42:39 - 43:21)
Right. There's no evidence that someone just went through and manually increased the numbers for those 13 without, you know, worrying about the text column. So what could have happened then?

So one innocent explanation is that maybe after sorting the data, someone decided to go back and double check the newspaper use scores, right? Because maybe they had used some kind of algorithm to score these in Excel, but it might depend on, you know, whether people used commas or periods. And so maybe somebody went back and manually recounted and corrected mistakes.

That could have been done after sorting. And a conscientious research assistant, that is exactly what they might have done. That could also explain why all the corrections increased the scores, because maybe the Excel formula like undercounted for everybody.

[Regina] (43:21 - 43:22)
I'm waiting for the but. The however.

[Kristin] (43:23 - 43:44)
There is a however. Yes, you know me well, Regina.

The thing that makes this possibility less likely is that all of the changes after sorting were only made to the cheaters group. So that would mean somehow like a third of the cheaters group had errors, but not a single non-cheater, the bigger group, had errors in this coding.

[Regina] (43:45 - 44:12)
Okay, that could explain the pattern, but we need to know why it was just those particular observations that were corrected and whether the non-cheaters responses were checked the same way. I suppose someone might have checked the cheaters responses more carefully without realizing. That could buy us the comparison.

It wouldn't necessarily be fraud, maybe, but it would definitely be a problem. Right.

[Kristin] (44:12 - 44:55)
We can't rule that out that some research assistant really paid close attention to the cheaters because they wanted to make sure we didn't miss any uses in that group. That would be bad science, but not fraud. But Regina, I think it's more plausible that someone just simply swapped both the text responses and the corresponding use counts between cheaters and non-cheaters.

So it's not as simple as just going through and changing the numbers. You actually have to swap things. But that way, higher scores from the non-cheaters would have ended up in the original text responses moved along with them.

We know from Harvard's investigation that swaps like this occurred. What I don't know, because I don't have enough information, is whether those swaps involved any of the 13 out-of-order observations and whether that can explain them.

[Regina] (44:55 - 45:25)
Wow. Okay, Kristin, there is a lot here. Just to kind of wrap my head around it, Harvard found multiple kinds of data alterations across different versions of the data set. And some of these seem really, really hard to explain as just innocent mistakes.

And Harvard ultimately concluded that Gino had intentionally falsified data. But Lessig thinks Harvard got this wrong.

[Kristin] (45:25 - 45:56)
Yes. He and his expert, Ava, offer alternative explanations for what happened. So, Regina, let's look at some of those arguments. And let's start with those 12 people who were changed to cheaters.

Lessig does not deny that the changes were deliberate. But he suggests that a research assistant out of that army of research assistants might have reclassified those 12 people as cheaters, not to manipulate the results, but because they believed that the participants had cheated in some way beyond the coin-toss task.

[Regina] (45:56 - 46:04)
Beyond the coin-toss task. But I thought the coin-toss task was how they defined cheaters and non-cheaters in the first place, right?

[Kristin] (46:04 - 46:27)
It was. And the published paper clearly defines cheating based on whether participants had lied about the coin-toss. There's no mention of using any other criteria to define a cheater.

But Lessig and Ava, they get really caught up in, remember I mentioned the two columns? They think it's some big mystery that needs to be solved. Why are there two columns in the same spreadsheet that seem to represent the same variable?

[Regina] (46:28 - 46:39)
Right. You talked about those columns before, but again, couldn't someone have just copied the variable over and given it a shorter, more intuitive name? That does happen all the time.

[Kristin] (46:39 - 46:59)
Occam’s razor. That seems like the most reasonable, simple explanation. But listen to this direct quote from Lessig's podcast.

Ava says, when you ask why is there a second column marked cheated, the answer must be that cheated was not simply a function of whether they said they flipped the coin correctly or not.

[Regina] (46:59 - 47:02)
Must be. Must be.

[Kristin] (47:03 - 47:32)
Must be.

[Regina]
That's crazy.

[Kristin]
It's totally crazy. Right?

I mean, it's bonkers. Just because there are two columns does not prove that they have to measure different things. You can have two columns with the same data in it.

And so Regina, they're asking us to kind of choose between either this simple explanation that someone copied a variable and renamed it, or this very incredibly convoluted story that a research assistant independently decided to redefine cheating without consulting Gino or documenting that decision anywhere.

[Regina] (47:33 - 47:44)
So what other information do Lessig and Ava think this research assistant used to classify those 12 people as cheaters if it was not just the coin toss task?

[Kristin] (47:45 - 48:34)
Well, you're going to love this, Regina. And by love this, I mean, it's going to make your blood boil. So remember the RAT, this creativity test where participants had to find the right word?

That was one of the outcomes, the dependent variables. Yes. Well, remember we said that those 12 participants had unusually high RAT scores.

Well, Lessig and Ava point to those unusually high scores and suggest that a research assistant may have suspected that these participants Googled the answers rather than solving the puzzles themselves. And they were, of course, not supposed to Google. And based on that suspicion, this research assistant may have reclassified them as cheaters on the basis that they cheated on the RAT test, even though they had told the truth about the coin toss.

[Regina] (48:34 - 48:34)
Wait. What?

[Kristin] (48:36 - 48:38)
Stop and think about that for a minute. Yes.

[Regina] (48:38 - 49:02)
Okay. Let me make sure I understand. Lessig and Ava are suggesting that a research assistant decided that these people were cheaters because they were unusually good at the creativity test.

Isn't the whole point of this study to see whether cheaters are more creative, so they'd be using creativity to decide who counts as a cheater?

[Kristin] (49:03 - 49:19)
Yes. This is where my head starts to explode, right? Cheating was the predictor.

Creativity was the outcome. But under their convoluted explanation, the research assistant used the outcome, the high creativity scores, to decide who belonged in the cheating group. Well, guess what's going to happen if you do that?

[Regina] (49:20 - 49:35)
That is completely circular, right? So you're using high creativity scores to decide who's a cheater. Well, then, of course, the cheater group is going to look more creative because you just set it up that way.

Oh, my God. No.

[Kristin] (49:35 - 49:38)
Yeah. This is why my head is exploding, Regina.

[Regina] (49:38 - 49:39)
This is horrible.

[Kristin] (49:39 - 50:24)
This is crazy, right? I mean, there's actually zero documentation that anyone actually did this, made this decision. But we are being asked to believe that a research assistant, not even a co-author on the paper, independently redefined the independent variable of the study without consulting anyone, without telling anyone. And not only that, but redefined it based on the outcome of the study.

There's other problems with this explanation beyond just this circular logic. So it wasn't the case that all 12 participants got perfect scores on the RAT, right? Like maybe if they all got 17s, you might think something funny was going on.

They scored higher than average, but that doesn't prove that they looked up the answers. And, even if for some reason you had strong evidence that they cheated on the RAT, what would be the proper thing to do here with those data?

[Regina] (50:24 - 50:38)
Okay. If I suspect they cheated, I might decide to exclude them. You don't go back and move them to another group.

You don't reassign them on their independent variable. No. Yeah, exactly.

[Kristin] (50:38 - 51:09)
And the explanation gets even more convoluted because Lessig and Ava point out that these 12 participants, they scored unusually well on the rat, but not unusually well on the newspaper creativity task. And they argue that this is because a research assistant would have believed that participants could Google the RAT answers, but not Google creative uses for a newspaper. So the assistant would have looked for cheating based on the RAT, but not on the newspaper task.

Regina, I suggest we test this theory. Okay.

[Regina] (51:11 - 51:39)
All right. I'm going to do it. Googling what creative uses for a newspaper right now.

I am getting a whole list, Kristin. Ooh, some of these are good. Why did I not think of this?

Origami. How did I miss that one? Oh, that's a good one.

Weed barrier. What am I going to do? Oh, yeah.

[Kristin]
For planting, yeah.

[Regina]
Gift wrapping. Yeah.

Streak-free glass. You can use it to clean glass.

[Kristin] (51:39 - 51:42)
I've never tried that. I don't know if that would work, but okay.

[Regina] (51:44 - 51:52)
Compost brown material. I'm not sure I want to know what that is. What's the brown part?

Okay. The point is we got a whole list. We got a whole list.

[Kristin] (51:53 - 52:34)
Apparently you can, in fact, Google creative uses for a newspaper.

Imagine that. So this research assistant would have had to believe that these participants cheated on the 17 item RAT, but somehow did not bother to cheat on the newspaper task while they were already on Google. And then the other thing is why would participants bother cheating on these tasks at all?

Remember they could earn an extra dollar by lying about the coin toss, but there was no bonus for performing well on the creativity test. And in fact, if you're on Mechanical Turk, you probably want to just get through as quickly as possible so you can do more of these surveys to earn more money. So why would you spend extra time Googling and making sure you got the right answers?

[Regina] (52:36 - 53:34)
All right. Let me back up again, make sure I have all of this straight. So we know that in the earliest dataset that we have access to, the cheated column was calculated directly from that coin toss task response, right?

And that is how cheating was also defined in the paper. But Lessig is suggesting that a research assistant might have later redefined cheating to include people who were suspiciously good at the creativity test, even though there is no documentation that this was part of the study design at all. Okay.

And let's just say that is what happened. And the researchers would have been using the outcome to define the predictor, yes. Which, so that's not a good story.

That's not helping anything. That is a serious problem.

[Kristin] (53:34 - 53:44)
I know. Is this just the statistician's perspective? Does the lawyer not see this? I mean, I think most people understand that that's not a good way to do science.

But, Regina, the convoluted explanations don't stop there.

[Regina] (53:44 - 54:25)
I cannot wait to hear what else they came up with. More convoluted explanations, but first a break. Welcome back to Normal Curves.

We're talking about the Francesca Gino case and Lawrence Lessig's explanations for the data anomalies in her evil genius paper. And, Tristan, what was the next explanation?

[Kristin] (54:25 - 54:48)
All right. Remember, I told you that, according to Harvard, in the July 2013 version of the data, the reported guess correctly coin toss column had 31 cheaters, but the cheated column had 43. But in the version of the dataset that we have, the one that Data Colada obtained in 2014, those two columns match.

So that suggested that someone had gone back and changed the zeros to ones in the coin toss column.

[Regina] (54:48 - 54:56)
Which Harvard then interpreted as an attempt to cover up the earlier manipulation, right? So how does Lessig explain this?

[Kristin] (54:56 - 55:18)
Well, the army of research assistants, of course, Regina. The army, of course. So he and Ava argue that just as Harvard business professors don't clean their own data, they also don't prepare the data for sharing.

So he claims that when asked to share the data, Gino would have sent it to a research assistant to, quote, prepare.

[Regina] (55:18 - 55:20)
Prepare. What do you mean, prepare the data?

[Kristin] (55:20 - 55:41)
I have no idea. Okay, remember, the paper was already published in 2014 at the point where she was being asked for the data. So the data had already been cleaned and analyzed.

What was left to prepare? I mean, maybe you need to add a data dictionary if there was no documentation, but there would be no reason to go back and alter the data in any way after the paper had already been published.

[Regina] (55:41 - 56:01)
Yeah, definitely you don't alter. I mean, for me, preparing data for sharing, I can imagine maybe they might have to, I don't know, remove identifying information maybe, organize the variables, like you said, add documentation, but you should not be going in and changing actual responses just to make the spreadsheet look tidy. No, not that.

[Kristin] (56:01 - 56:13)
Yes. Maybe, yes, you have to remove some identifying columns or something, absolutely.

But let's say there were identifying columns. Should you even be sharing that with research assistants, right? I mean, prepare the data.

It's weird. Yeah.

[Regina] (56:14 - 56:28)
Yeah, that is super weird. Okay, so Gino says she normally had research assistants prepare data for sharing. That was her normal thing.

So what is the explanation for these particular changes, these suspicious changes we've seen?

[Kristin] (56:28 - 56:38)
Right. So Lessig and Ava suggest that a diligent research assistant preparing the data for sharing might have spotted the inconsistencies in the data and tried to resolve them.

[Regina] (56:39 - 56:51)
So the idea is that an assistant noticed these inconsistencies, decided to resolve them by changing the source data to match the derived variable all on their own.

[Kristin] (56:51 - 57:21)
Exactly. And apparently, without any specific instructions from Gino, this research assistant took it upon themselves to check the data set, identify inconsistencies, and then instead of discussing them with Gino, just change the data to make everything match. And Lessig and Ava go further, and they argue that these changes could not possibly have been made by Gino because changing these zeros back to ones would have been an incredibly stupid way to cover up fraud. Ava says, well, if this was a cover-up, it would have been a cover-up by basically the dumbest means possible.

[Regina] (57:22 - 57:26)
What? Why does Ava think that's a stupid way to cover up earlier manipulation?

[Kristin] (57:26 - 58:08)
I know this is, this might be my favorite of their ridiculous arguments, actually. They say that it would have been smarter to have deleted the original coin toss column. But I'm not sure how that's any better.

Both approaches eliminate the inconsistency. You could delete the column, or you can change the zeros to ones. I think those are kind of equally effective.

The only way either change would be, quote, dumb is if she expected someone to be able to check the 2014 data set against earlier versions of the data. But back in 2014, she was not anticipating that Harvard, years later, was going to access her laptop and get earlier versions of the data, right? I mean, this argument that Gino couldn't have done it because it would have been too stupid, I find that completely unconvincing.

[Regina] (58:08 - 58:18)
Right. We cannot infer innocence just because a proposed cover-up would not have been clever. Right.

That doesn't work.

[Kristin] (58:18 - 58:38)
Right. Also, Regina, remember that there were those four individual RAT scores that didn't come from the underlying data? Lessig and Ava also explained those in the same way.

They said, oh, the diligent research assistant noticed those didn't match. So that research assistant went back and changed the individual RAT score so the individual scores matched the sum.

[Regina] (58:38 - 58:53)
Wait. So the assistant supposedly found that the individual RAT scores did not add up to the total and then changed the individual scores to agree with the total without establishing which numbers were actually correct. What?

[Kristin] (58:53 - 59:12)
Right. I mean, if you are working on a data set and you find a discrepancy between the original responses and a derived variable, you are going to go investigate it. You've got to check the original data, figure out what went wrong, and document any corrections.

You don't make the problem disappear by making the underlying responses match the derived variable.

[Regina] (59:13 - 1:00:03)
No, no, no, no. Okay. Let me get this straight.

Lessig's explanation is that research assistants might have made these changes while preparing the data innocently, not as a deliberate cover-up, but we do not have documentation establishing that that's what happened. And some of these proposed corrections would have involved changing original response variables to match derived variables. Yeah.

That would be a really troubling way to handle data, to say the least. Yes. Yes.

Okay. So I can see why you would want to investigate whether research assistants were involved. That makes sense.

But that does not establish that they actually made these changes or that the changes were legitimate. I don't get it.

[Kristin] (1:00:03 - 1:00:25)
Right. So this whole story that Lessig and Ava present, it's kind of a Rube Goldberg explanation, right? This elaborate chain of unlikely events, each of which has to happen in exactly the right way for this defense to hold water.

Now, is it possible? Sure. But just because you can imagine an alternative explanation does not mean there's evidence to support it.

[Regina] (1:00:27 - 1:00:34)
So exactly. So, Kristin, you think someone deliberately changed the data to make the results look better?

[Kristin] (1:00:35 - 1:00:53)
Yes. We have multiple suspicious changes that strengthened the findings. And I think the evidence points much more strongly toward deliberate manipulation than innocent mistakes, at least for this evil genius paper, which is the one that I've scrutinized carefully.

I'm not saying there weren't also innocent mistakes, but there were definitely mistakes that look like deliberate manipulation.

[Regina] (1:00:54 - 1:01:00)
Mm-hmm. Okay. But even if someone manipulated the data, how do we know it was Gino then?

[Kristin] (1:01:00 - 1:01:17)
Right. We don't know for certain. We didn't have a camera in her office seeing who's typing into her computer.

And I guess you could imagine maybe a research assistant could have made these deliberate changes independently, maybe hoping to please Gino or hoping to help get the paper published.

[Regina] (1:01:18 - 1:01:28)
Okay. If I remember correctly, doesn't Lessig argue that Harvard should have interviewed more of the research assistants? He complained about that.

[Kristin] (1:01:28 - 1:02:00)
Yes. And I think that's a fair criticism. Maybe some of those assistants could have helped clarify what happened.

But, Regina, the research assistants weren't even co-authors on the paper. And also, Gino is the one who appears to have analyzed the data. So she's the one who would have known which changes could strengthen the findings or not.

And also, remember, Harvard found suspicious alterations across four different studies. So if Gino wasn't involved, we'd have to consider the possibility that multiple different research assistants independently manipulated data to strengthen her results.

[Regina] (1:02:01 - 1:02:12)
Hmm. Okay. Just to play devil's advocate here, do we know that there were different research assistants involved in each of the studies?

Could it have been the same people working on several of them?

[Kristin] (1:02:13 - 1:02:29)
Lessig and Ava really describe this as an army of different research assistants. They told us that at least five different RAs worked on the data for the evil genius study and 66 research assistants were working across all four studies. So it seems unlikely that it was the same one in all cases.

[Regina] (1:02:29 - 1:02:48)
Yeah. Okay. That is an army.

We don't know whether it was the same assistants or whatever, but wasn't some of the evidence about who made the changes based on files that were found on Gino's computer, right? And the metadata there?

[Kristin] (1:02:49 - 1:03:31)
Yeah, exactly. Harvard recovered those earlier versions of the datasets from her computer and the Excel metadata identified her as the creator or last person to save the files.

[Regina]
Well, wow. How does Gino explain that then?

[Kristin]
Well, she says that research assistants routinely had access to her computer and her office and her accounts, her passwords, and she allowed them to work on the data directly on her computer. She also says that files were transferred using USB drives and so there's no record because the USB drives have been lost. Now, what she doesn't provide is any evidence showing that she transferred files electronically, such as through email.

[Regina] (1:03:33 - 1:03:45)
I don't know about all of this. Okay. So she said research assistants regularly used her computer and her passwords?

I mean, did she like give out her credit cards too?

[Kristin] (1:03:46 - 1:04:34)
Right. I know. Maybe this is true, right?

But I mean, it's a little strange. I mean, she was too important to clean her own data, but apparently she had a policy of anybody using her laptop, right? Maybe it's true, but it is a little implausible.

And Regina also, in her original defense to Harvard back when they first investigated, Gino actually suggested that someone with access to her computer might have deliberately manipulated the data to sabotage her. This was her main defense originally, and Harvard found that explanation implausible. Even Lessig does not pursue that theory in his podcast at all.

He never mentions it. Instead, of course, he focuses on this army of research assistants making innocent mistakes, which makes me think maybe he also found the sabotage theory implausible.

[Regina] (1:04:35 - 1:04:39)
Okay. So where does that leave us now, Kristin? Right.

[Kristin] (1:04:39 - 1:05:50)
Well, I want to be clear about what I know and what I don't know. So I've reviewed the Harvard reports, Lessig's podcast, the Data Colada blog, the investigative reporting on this, and that 2014 data set from the Evil Genius study. But I have not seen all the evidence that Harvard got to see because I don't have all of the data sets.

I also have not seen... There was an analysis done later by Stanford sociologist Jeremy Freese, and I haven't gotten a copy of that. Lessig talks about that analysis on his podcast, but I don't have a lot of details.

So I cannot independently reconstruct every change or tell you with certainty who made every change. But here's what really bothers me. Even if Lessig's explanation is true, even if this was some extraordinary series of innocent mistakes by research assistants, that would still represent a shocking failure of scientific oversight.

Gina was the principal investigator. She was responsible for understanding and overseeing the data underlying her published research. I don't believe these kinds of practices that Lessig and Ava are describing are standard in her field.

And even if they were, that wouldn't make them acceptable, right? If everybody is doing bad science, it's still bad science.

[Regina] (1:05:50 - 1:05:55)
Yeah. And this was someone top of her field, enormous influence.

[Kristin] (1:05:55 - 1:06:48)
Exactly. Gina became a superstar at Harvard Business School. She published exciting findings.

She was giving high profile talks, building an extraordinary career. It does make you wonder a little, Regina, whether academia maybe sometimes rewards scientists more for telling compelling stories than for making sure those stories are actually true. I think we've talked about that before on this podcast.

Now, I also want to say that I do genuinely feel for Gino. She actually comes onto Lessig's podcast and speaks, tries to defend herself. And I found her sympathetic.

She has four children. This has been an enormous personal and professional downfall. I felt for her and I understand why her friend is making such an impassioned defense of her.

And part of me actually wants him to be right. But if I set the sympathy aside and look just at the evidence, I find Harvard's conclusion much more convincing than any of the alternative explanations that Lessig offers.

[Regina] (1:06:49 - 1:06:56)
Yeah. Okay. Kristin, you think we're ready to wrap up and evaluate your claim?

I think we are. Yes.

[Kristin] (1:06:56 - 1:07:07)
So again, I had to be creative in this claim. My claim was superstar scientists are too important to clean their own data and too smart to commit fraud that's easy to catch.

[Regina] (1:07:08 - 1:07:10)
That's a gimme. That's a gotcha.

[Kristin] (1:07:11 - 1:07:16)
It's a little bit of a convoluted claim, but it does kind of encapsulate the episode for today.

[Regina] (1:07:18 - 1:07:35)
Yeah. Okay. Let's rate the claim with our smooch rating scale where one smooch means little to no evidence to support your claim.

Five means strong evidence to support the claim. So Kristin, kiss it or diss it.

[Kristin] (1:07:35 - 1:07:46)
I'm dissing this one. This is a martini in the face, Regina. Okay.

This is not really a claim that we can evaluate with empirical evidence. This is more opinion, but I'm throwing three martinis in the face on this one.

[Regina] (1:07:47 - 1:08:25)
How about you? Okay. Superstar scientists are too important to clean their own data, too smart to commit fraud that's easy to catch.

You just made that one too easy. Well, of course not. Superstar scientists might think they're too important to clean their own data.

They might think they're too smart to commit fraud that's easy to catch. But clearly not. I don't know.

I might join you on the martini thing. I mean, you made it a little easy. Yeah.

And I'm kind of thirsty, so I don't know if I'm going to throw the martini in the face. I might just drink the martini right now. Three martinis for me.

Three martinis for me.

[Kristin] (1:08:26 - 1:08:32)
I think after listening to his explanations, I might need three martinis even though I don't drink.

[Regina] (1:08:33 - 1:08:45)
I'll take yours. Okay. Maybe we need to come up with some creative uses for the smooches and the martinis.

Go ahead. 10 seconds. Name 10 creative things you could do with a martini.

Clean your windshield.

[Kristin] (1:08:47 - 1:08:54)
Clean a table. I was just cleaning a table from the garage. That would have been great for cleaning.

I used to actually rubbing alcohol for doing that. So same thing.

[Regina] (1:08:55 - 1:08:56)
Perfume.

[Kristin] (1:08:58 - 1:09:11)
Hair tonic.

[Regina]
Pasta. Pasta sauce.

[Kristin]
If you want the boy in the bar to come up to you, hand him the martini rather than put the newspaper over your face. Oh, pasta sauce.

[Regina] (1:09:11 - 1:09:17)
Oh, there you go. Learn how to create a good martini and get the hot men that way.

[Kristin] (1:09:18 - 1:09:18)
Yes.

[Regina] (1:09:18 - 1:09:24)
I'm not going to ask you for creative uses for the smooches because...

[Kristin] (1:09:24 - 1:09:34)
Well, we haven't gotten sex into our episodes in a while. So you want to talk creative uses for smooches. I mean, that might just get the sex in, but I think we just won't go there today.

[Regina] (1:09:36 - 1:09:37)
We'll save that for a future episode.

[Kristin] (1:09:38 - 1:09:45)
All right. Okay, Regina. Methodologic morals.

I feel like this is a very rich episode for those. So what do you have for us?

[Regina] (1:09:45 - 1:09:48)
Oh my gosh. There are just so many in here.

[Kristin] (1:09:48 - 1:09:51)
Regina, I think we're both going to need two for this one.

[Regina] (1:09:51 - 1:10:03)
Oh, I think so. At least two. All right.

How about this? You cannot define your predictor variable using your outcome variable. That circular logic, that does not work.

[Kristin] (1:10:03 - 1:10:07)
I think that one has to get in there, Regina, because it's just so basic, but so important.

[Regina] (1:10:07 - 1:10:18)
How about this? If it takes an army of research assistants to clean your data, you're probably doing something wrong and maybe you need a statistician.

[Kristin] (1:10:19 - 1:10:21)
You need a statistician, yes. We're not just saying that for job security either.

[Regina] (1:10:21 - 1:10:22)
No, we're not.

[Kristin] (1:10:22 - 1:10:43)
That one is true.

[Regina]
What about you?

[Kristin]
How about this, Regina?

Being too smart to commit stupid fraud is not a defense. Smart people do stupid things all the time.

[Regina]
Oh, that's a good one.

[Kristin]
Okay. How about this one? The more hoops your explanation has to jump through, the less convincing it becomes.

[Regina] (1:10:45 - 1:10:54)
Yeah. You can have one coincidence, right? But if you've got a whole string of them and it becomes a whole novel, a fantasy novel, yeah.

[Kristin] (1:10:54 - 1:11:02)
I will give Lessig points for creativity. But creativity might be a bad thing today, right? Because we've said it might be tied to dishonesty.

[Regina] (1:11:03 - 1:11:07)
Yeah. Oh, this is like a meta moral here.

[Kristin] (1:11:07 - 1:11:13)
I know. Yeah. But the paper was retracted and there was probably nothing there. So we're going to say, go ahead and be creative.

It does not mean you're dishonest. Yeah.

[Regina] (1:11:14 - 1:11:26)
Kristin, this has been – I don't even know how to summarize this. Kristin, this has been a wonderful ride on your outrage train.

[Kristin] (1:11:27 - 1:11:32)
I think I was so restrained, actually, Regina. I feel like I was incredibly restrained. Don't you think?

[Regina] (1:11:32 - 1:12:11)
This is true. When you would text me or talk to me about this before the episode, I was expecting like frothing at the mouth, frothing at the mouth.

But you sounded like a normal, sane person here in that.

[Kristin]
Excellent, yes.

[Regina]
I know. You covered it well. You covered it well.

Thank you. Yes. This has been very interesting.

It is a rabbit hole, a deep dive, but with a lot of convoluted turns. Very, very dirty, very messy, some earthworms in there. But I've enjoyed it.

I have a whole new perspective on this entire fraud case.

[Kristin] (1:12:12 - 1:12:20)
It was a totally interesting podcast to listen to, Lessie's podcast. It was like interesting to think about the problem, right? It was interesting from a problem data point of view.

[Regina] (1:12:21 - 1:12:29)
Data detective paradise in here. So, Kristin, thank you so much.

[Kristin]
Thank you, Regina.

And thanks everyone for listening.