Odds Ratios: Do most people get them wrong?

[Regina] (0:00 - 0:14)
Three, and we would interpret that as something like this. Men with puppy pics are three times as likely to get laid on the first date as men without puppy pics. That is definitely going in the abstract, in the headline.

[Kristin] (0:15 - 0:31)
Absolutely. Welcome to Normal Curves. This is a podcast for anyone who wants to learn about scientific studies and the statistics behind them.

I'm Kristin Sainani. I'm a professor at Stanford University.

[Regina] (0:31 - 0:37)
And I'm Regina Nuzzo. I'm a professor at Gallaudet University and part-time lecturer at Stanford.

[Kristin] (0:37 - 0:42)
We are not medical doctors, we are PhDs, so nothing in this podcast should be construed as medical advice.

[Regina] (0:42 - 0:47)
Also, this podcast is separate from our day jobs at Stanford and Gallaudet University.

[Kristin] (0:48 - 0:58)
Regina, today we're going to do another one of our methodologic deep dives, like p-values or diagnostic testing, because those episodes were surprisingly popular.

[Regina] (0:59 - 1:04)
Really surprising. I guess people love learning. Is that what it is?

[Kristin] (1:05 - 1:39)
I think they do.

And today's topic is odds ratios.

[Regina]
Oh my, we're going for the jugular now.

[Kristin]
Well, this was actually suggested by one of our listeners, David Rind.

He's a physician and a chief medical officer at the Institute for Clinical and Economic Review. And he's also a patron. And after listening to our Batman episode, which involved odds ratios, he suggested we do a whole episode on them because he thinks that most people don't understand the subtleties of odds ratios, but that they really should.

[Regina] (1:40 - 2:05)
They really should. He had a whole primary hypothesis and secondary hypothesis about this. And I love, he went to gather evidence.

He did a little informal survey with his medical resident. He reported back to us, he got an N of 5. And the number of them who understood this crucial odds ratio concept was zero.

[Kristin] (2:06 - 2:08)
Zero out of five, I feel like we can all go home now.

[Regina] (2:09 - 2:22)
We can all go home, we're done. That is everything. You know, he tried to boost recruitment.

He bribed them with chocolate truffles. And I don't know. I think that counts as going the extra mile and it was probably effective.

Yeah.

[Kristin] (2:22 - 2:34)
Yeah. It's not coercion at all because chocolate is always appropriate. And Regina, we actually frequently bribe our students with chocolate by making it the prize for these little in-class competitions that we do when we teach.

[Regina] (2:35 - 2:39)
Everyone just gets a little more interested when chocolate is on the line.

[Kristin] (2:39 - 2:50)
Absolutely. So I'm going to make David's hypothesis our claim for today, that many people writing and reading the medical literature don't actually understand odds ratios.

[Regina] (2:50 - 2:53)
Interesting hypothesis. That's all I'll say.

[Kristin] (2:54 - 3:15)
So odds ratios are kind of a strange statistical creature. They have nice mathematical properties, as we're going to see, which is why they come up so much, but they can be tricky to interpret. So today we're talking about what odds ratios actually are, why they are used so much, and we're going to see some examples when they wildly exaggerate effects and cause all sorts of confusion.

[Regina] (3:16 - 3:25)
I actually love odds ratios and this whole topic. I mean, odds, how can you not love something that has the word odd in it? So this is going to be fun.

Yes.

[Kristin] (3:25 - 3:37)
The crucial thing about odds ratios that David wants everyone to understand is how they differ from another statistic called the risk ratio. So let's take a moment to explain risks and risk ratios.

[Regina] (3:38 - 4:01)
Yeah. We've got to understand that. First, the risk ratios really are much more intuitive than odds ratios.

And people sometimes confuse the two, in part because we use both of them in the same kind of situation when our outcome is binary. Binary means the thing that we care about is the answer to a yes-no question, like, did the person develop heart disease or not? Exactly.

[Kristin] (4:02 - 4:13)
But Regina, since heart disease is boring, I thought for our hypothetical example to illustrate this, we should use something more exciting. So I'm going to make the outcome sex on a first date.

[Regina] (4:13 - 4:17)
That is way more exciting than what you're talking about.

[Kristin] (4:17 - 4:21)
Right. I got to make people perk up here because we're about to discuss a lot of numbers.

[Regina] (4:22 - 4:24)
Good job getting sex in there.

[Kristin] (4:24 - 4:59)
Yes. Thank you. Thank you.

So imagine that we do a study with men on dating apps and we want to know what predicts that a first date from a dating app will end in sex. And let's suppose in this study we have a hundred men who included a puppy photo in their dating profile and a hundred men who did not. That is a great predictor.

And let's say, I'm making up numbers, obviously, let's say that 60 of the first dates for the puppy group end in sex versus just 20 of the dates in the non-puppy group.

[Regina] (5:00 - 5:04)
I am definitely on team cute dog photos, a hundred percent.

[Kristin] (5:04 - 5:24)
And clearly I am making these numbers up because they're very high and inflated, but I want to make a mathematical point. So given these numbers, we can say that the probability of the first date ending in sex was 60 percent in the puppy group, 60 out of a hundred, versus just 20 percent in the non-puppy group.

[Regina] (5:24 - 5:29)
Again, big effect size here. You've got some really optimistic views of puppies.

[Kristin] (5:31 - 6:05)
Again, these numbers were chosen to make the math easy, not because they are realistic. All right. These percentages, 60 percent and 20 percent, they actually have a lot of different names.

We can say that there was a 60 percent probability or 60 percent chance or 60 percent likelihood or 60 percent risk of having sex on the first date. Those are all valid names. I'm primarily going to use the term risk here because that's what's commonly used in the medical literature.

Just want to point out that risk doesn't necessarily imply that the outcome is bad. And you know, here people may have different opinions about whether sex on a first date is good or bad.

[Regina] (6:06 - 6:09)
No shame. Either way. No shame here.

[Kristin] (6:09 - 6:31)
Absolutely. Absolutely. Okay.

So, Regina, now we have these two risks in our study and we could just, for our results, present those raw numbers, right? 60 percent versus 20 percent tells us a lot. Or we could divide those risks to make what is called a risk ratio, 60 percent divided by 20 percent, that's three.

So, the risk ratio for having sex on the first date, comparing the puppy to the non-puppy group is three.

[Regina] (6:32 - 6:45)
And we would interpret that as something like this. Men with puppy pics are three times as likely to get laid on the first date as men without puppy pics. That is definitely going in the abstract.

[Kristin] (6:46 - 7:01)
Absolutely. And that number is pretty intuitive. That three is pretty intuitive, I think, to interpret.

But now let's see how this would be reported if we focused on odds rather than risk. Odds are less intuitive and they come up a lot in gambling.

[Regina] (7:01 - 7:05)
But if you don't gamble, then you're probably not thinking in terms of odds.

[Kristin] (7:06 - 7:29)
Exactly. An odds is the risk of something happening divided by the risk of it not happening. So if men in the puppy group had a 60 percent risk of sex on the first date, that means they had a 40 percent risk of no sex.

So the odds are 60 to 40, which we can simplify to 6 to 4 or 3 to 2. Or if you want to think in fractions, 3 to 2 is three halves.

[Regina] (7:29 - 7:45)
I don't think in terms of fractions like that, but you go right ahead. All right. Three to two.

And we interpret that like this. That means that out of the puppy photo dudes, every three who had sex on the first date, there were two who did not.

[Kristin] (7:45 - 7:54)
Right. And for the non-puppy group, the odds are 20 to 80, which simplifies to 2 to 8. Or 1 to 4, which if we want to go fractions again, is one quarter.

[Regina] (7:54 - 8:09)
Right. So in the non-puppy dude group, for every one who has sex on the first date, four do not. So, wow, puppy dudes are faring much better than the non-puppy dudes.

Yes. In our hypothetical example.

[Kristin] (8:10 - 8:22)
Regina, if we divide these two odds, we then get an odds ratio. And can you please do the honors of dividing three halves by one quarter? I'm afraid we're going to give everyone some fraction flashbacks here.

[Regina] (8:22 - 9:00)
It is absolutely a flashback in here. And I always need to work it out explicitly in my head, so I'm going to give you some numbers here. Let's see what we do.

Okay. Three over two divided by one over four and dividing by one over four is the same as multiplying by four over one. So that means we have three over two times four over one, which gives us 12 over two, which is six.

Oh, that's a nice number. That worked out well. So men with puppy pictures have six times the odds of getting laid on the first date compared to men without puppy pictures.

[Kristin] (9:00 - 9:19)
Right. So men with puppy pictures have three times the risk or three times the likelihood of getting laid, but they also have six times the odds. Those numbers are not conflicting.

The results of the study did not change. The numbers did not change. We are just describing the results in two different ways, and those numbers have two different meanings.

[Regina] (9:19 - 10:30)
Exactly. And this is where things can go wrong, right, because we're using English words to interpret or describe what's going on with the numbers and mathematical and statistical terms. And this is one of the crucial points that David wants everyone to understand.

So just to make it explicit, the odds ratio is six, but we cannot interpret that as something like puppy men were six times as likely to get laid. That's wrong because of the word likely in there. It's weird because likely, Kristin, is a word that we use to talk about probabilities, at least that statisticians use to talk about probability, not odds.

The only word that we can use to interpret an odds ratio is odds. That's it. So puppy men had six times the odds of having sex on the first date compared to non-puppy men.

So if we want to talk about the likelihood of having sex or the risk or, you know, chance or probability, we need to use the risk ratio, not the odds ratio, which here was three. So again, we can say puppy men were three times as likely to have sex compared to non-puppy men, not six.

[Kristin] (10:31 - 10:55)
And you can see why we can't say six if you just look at the numbers, because remember, 20 percent of the non-puppy men got laid. And if puppy men were six times as likely than the non-puppy men to get laid, that would mean that 120 percent of puppy men got laid, right? Because 20 percent times six is 120 percent.

That is obviously impossible because a probability cannot be greater than 100 percent.

[Regina] (10:55 - 11:00)
In men's minds, maybe they're getting sex 120 percent of the time.

[Kristin] (11:00 - 11:02)
In their fantasies, Regina.

[Regina] (11:02 - 11:09)
Yeah, fantasies. Right. But in real life, 120 percent probability of sex is impossible.

[Kristin] (11:09 - 11:21)
Exactly.

And, Regina, just so we fully cover the math, I want to look at the situation where the numbers are exactly reversed. So let's look at our predictor of men with fish pictures on their dating profiles.

[Regina] (11:22 - 11:28)
Fish pictures? Is that a euphemism for something? Like men holding up their, I don't know, giant bash?

[Kristin] (11:32 - 12:01)
You know the pictures I'm talking about, yes. I don't know why they put those pictures. I don't know what they're supposed to say.

But let's say that these pictures do not increase your likelihood of getting sex. And let's say only 20 percent of the men with fish pictures get laid on a first date versus 60 percent of men without fish pictures. I've exactly reversed the numbers from the puppy pictures.

So the risk ratio here would be 20 percent divided by 60 percent, which is one-third or 0.33 repeating.

[Regina] (12:01 - 12:33)
Mm-hmm. So, to interpret this, if a man is holding up his giant halibut in his profile, his risk of sex decreased by 67 percent, or two-thirds. And how do we get that?

Because we have to take that risk ratio and compare it to one. So one minus 0.333, or one minus one-third, is 0.67, or two-thirds, so a 67 percent decrease in risk.

[Kristin] (12:34 - 13:03)
Yes, we compare to one because one is the ratio you get when the two groups are exactly equal, right? If you divide two equal things, you get 1.0. So a risk ratio of one means there's no difference between the groups, and that's what we always compare to. All right.

What would the odds ratio be here, Regina? I'm going to skip repeating the fraction math, because since I made the numbers identical, our answer for the odds ratio is just the reciprocal of the odds ratio we calculated before. So it is one over six, or 0.166 repeating.

[Regina] (13:03 - 13:48)
You are so good with digits. So I'm going to round 0.1666 to 0.17 to make it a little easier. That means the man's odds of having sex on the first date drop by 83 percent if he has his giant cod.

So again, let's compare risk ratio and odds ratio. Risk drops by 67 percent, but the odds drop by 83 percent. The drop in odds looks bigger, not unlike his fish.

Oh, is it? It looks bigger. Yeah.

I'm just going to tell them this one.

[Kristin] (13:48 - 14:12)
Okay. Go ahead. All right.

I had no intention of that when I said fish pictures, but okay. Notice that the odds ratio is always farther from one than the risk ratio, either farther above one or farther below one. Remember, in our puppy example, the risk ratio was three, whereas the odds ratio was six farther from one.

And in our fish example, the risk ratio was 0.33, while the odds ratio was 0.17, again, farther from one.

[Regina] (14:12 - 14:47)
So that means the odds ratio always looked more dramatic. Again, like there, I'm running out of fish now. Trout.

Trout. Trout. Tuna.

That's not a dramatic fish. Swordfish. Oh my gosh.

Swordfish.

[Kristin]
Swordfish. Oh, very good, Regina.

Very good.

[Regina]
Oh, you're welcome. Okay.

This is important to remember. Your swordfish odds ratio always looks more dramatic than your trout risk ratio.

[Kristin] (14:48 - 15:34)
That's a good visual. All right. Thank you.

And Regina, it turns out that this distortion of the odds ratio compared with the risk ratio is not a huge big deal when the outcome is rare, because odds and risks happen to be very similar when the outcome is rare. But this matters a lot when the outcome is common, because when the outcome is common, risks and odds are not similar. Let me just put some math to that.

So if, for example, the risk is 1%, 1 out of 100, then the odds would be 1 to 99 or 1 99th, which is very close to 1%. But if the risk is something like 90%, very common, then the odds are 9 to 1, which is 9, which is obviously much larger than 90% or 0.9. Right.

[Regina] (15:34 - 16:00)
So, Kristin, I think it's also important to note here that odds ratios are not inherently bad. We're talking about them, you know, as being exaggerated. The problem is that people often interpret them as if they were risk ratios.

That's the bad part. You can't do that. And when outcomes are common, like you said, misinterpreting them can dramatically exaggerate how large the effect seems.

And that's particularly bad.

[Kristin] (16:01 - 16:36)
That's the problem. Yes. And we talked about one real-life example of this in the Batman effect episode in Season 1.

And I didn't have a lot of time in that episode to go into detail, so we just went by that example kind of fast. So I now want to go into more details about that example. The study was published in the American Journal of Public Health in 2012, and it was about teenagers purchasing sugary beverages.

And actually, I wrote a letter to the editor about this study back in 2012. And Regina, believe it or not, it is not the only letter to the editor that I've ever written in my career about misinterpreted odds ratios.

[Regina] (16:37 - 17:24)
Kristin, do you have a room at home where it's like wallpapered with these letters to the editor? Because you really should. There are enough of them.

[Kristin]
I am not taking it that far, Regina.

[Regina]
Okay. This is a great study.

Let's get to the details after the break. Welcome back to Normal Curves. Today we're talking about odds ratios, and we were about to discuss a real study about sugary beverages and teenagers.

[Kristin] (17:24 - 18:14)
Yeah. This was a 2012 study, and the researchers secretly observed how many teenagers were buying sugary drinks like sodas at some different convenience stores. At baseline, about 93% of the drinks purchased by teenagers were sugary beverages.

The researchers then ran an intervention. They randomly displayed different signs outside of those stores. There were three signs.

The first sign said, did you know that a bottle of soda or fruit juice has about 250 calories? The second sign said, did you know that a bottle of soda or fruit juice has about 10% of your daily calories? And the third said, did you know that working off a bottle of soda or fruit juice takes about 50 minutes of running?

And Regina, which of those signs do you think you would find most compelling?

[Regina] (18:16 - 18:16)
Of course, 50 minutes of running. Clearly.

[Kristin] (18:17 - 18:32)
Yeah, it does seem like that one might have the most effect on behavior, and that is the sign that the researchers were most interested in. They called this kind of framing exercise equivalents, and they are not the only group to try this out as a strategy for getting people to eat less junk food.

[Regina] (18:33 - 18:43)
Yeah, it does seem like it would be pretty effective. No one wants to exercise for an hour just to burn off some soda. Is that really worth it?

Yeah. So they find that it makes a difference?

[Kristin] (18:43 - 19:11)
Well, let me start by giving you the big media headline here, because the big headline was exercise equivalence reduced sugary drink purchases by about 50%. All right, Regina, quick quiz here. If teenagers were buying sugary drinks 93% of the time before the signs were posted and there was a 50% reduction in sugary beverage purchases, how often were they buying sugary beverages when the exercise sign was up?

Hmm. Okay.

[Regina] (19:11 - 19:22)
Well, half or 50% of 93% is, what, 46.5%, so about that, 47%. Not exactly.

[Kristin] (19:22 - 19:27)
When the exercise sign was posted, 86% of teenagers bought sugary beverages.

[Regina] (19:28 - 19:35)
86%? Wait a minute, 86% is nowhere near half of 93%. Sorry.

Someone's off.

[Kristin] (19:35 - 19:46)
Something's off, right? That 50% quote is obviously wrong when you look at the raw numbers. And Regina, just take a wild stab at it here.

Can you guess what happened?

[Regina] (19:46 - 19:57)
Well, this is an episode about odds ratio, so I'm going to go out on a limb and say, did they confuse odds ratios and risk ratios, Kristin?

[Kristin] (19:57 - 20:48)
You're like psychic, Regina, yes. That is exactly what happened. So the researchers in this paper, in addition to the raw percentages, they also published odds ratios, and the odds ratio comparing the exercise sign intervention to baseline was 0.51, and that was a 49% or about a 50% drop in the odds of buying a sugary beverage, but clearly not a 50% drop in the risk. So this headline came from a misinterpretation of odds ratios, indeed.

Now, our listeners may be wondering, why would the authors present an odds ratio at all here, right? There are so many easier numbers, easier statistics that they could have focused on, right? Sugary beverages dropped from 93% at baseline to 86% with the exercise sign.

They could have just said, hey, that's a 7% or 7 out of 100 drop in sugary beverage purchases.

[Regina] (20:49 - 21:05)
But, Kristin, that doesn't sound as impressive. It doesn't get as many headlines. So the conspiracy theorist in me, very pessimistic, says, well, maybe the researchers reported the odds ratio instead of risk ratio because it sounded more dramatic.

[Kristin] (21:06 - 21:58)
Well, Regina, I think it might be a little more subtle than that. I think that might be the reason why they focused on and emphasized the odds ratio in their paper. But the reason that they bothered to calculate it at all, the reason that it's there, is a legitimate reason.

They reported it because they used a statistical technique called logistic regression. So those numbers, the 93% and the 86% that I told you, those were unadjusted. They did not account for confounding.

And the researchers wanted to adjust for some potential confounders here. Things like the store location and the sex of the teenager and the time of the day. And we've talked about on this podcast before that to do that kind of adjustment, you can use something called regression.

And the regression that you use when you have a binary outcome, right, the outcome here is binary, you bought a sugary beverage or not, that regression technique is something called logistic regression.

[Regina] (21:58 - 22:07)
Logistic regression is kind of interesting, but a lot of people don't know about it. So Kristin, let's do a quick statistical detour on logistic regression.

[Kristin] (22:08 - 23:02)
I would love to do that since I teach a whole class in this. All right. So we've talked about regression on this podcast before, but we've mostly focused on linear regression, which is the regression you use when your outcome is a number, a numeric outcome.

When you have a binary outcome, a yes-no outcome, you instead use logistic regression. And Regina, to make this detour a little more fun, can we briefly go back to the sex on the first date as our outcome example? Oh, absolutely.

And also, let's keep on going with our running insider joke on this podcast, your date with the boat man. The boat man. So for listeners who haven't heard those previous episodes, Regina went on a first date with a man who told her on the first date that he owned seven boats.

[Regina]
Seven Boats. Yep.

[Kristin]
I continue to find this hilarious.

[Regina] (23:02 - 23:05)
He did not get sex on the first date, though, Kristin.

[Kristin] (23:05 - 23:13)
No, I know.

[Regina]
That was a zero. That was a zero.

[Kristin]
It was a zero outcome, yes. I think he did get a lifetime of being referenced on this podcast, though.

[Regina] (23:14 - 23:19)
All in good fun, anonymous

[Kristin] (23:20 - 23:40)
Totally anonymous. Yes. Okay. So our outcome in this example is one is for sex on the first date, zero is no sex.

And our predictor in this hypothetical example is going to be the number of boats that a man owns. So imagine plotting those on a scatterplot. One or zero is on the vertical axis and number of boats is on the horizontal axis.

[Regina] (23:41 - 23:54)
Which is nice, but because it's a zero one outcome, you don't get any scatter on a scatterplot in the up and down direction. You just get two stripes, one stripe up top for the ones and one stripe down at the bottom for the zeros.

[Kristin] (23:54 - 24:53)
Yeah. And this is a problem because as statisticians, we love to fit lines and this does not make a good scatterplot to fit a line to. These two stripes don't fit a nice line.

But what we might then recognize is that what we are actually after here is not so much the yes or no, but what we really want to know is what is the probability? What is the probability of sex on the first date if you own seven boats or if you own six or if you own zero boats? So you could imagine that we could change the outcome from one or zero only into probabilities.

Probabilities allow us to have some scatter, right? We can now have some scatter between zero and one, but probabilities are still a little bit awkward because they're trapped between zero and one. And that means that our line would get trapped between zero and one instead of extending on forever in both directions like a regular line should do.

So we don't love probabilities either here as the outcome. And so logistic regression uses a workaround. Instead of fitting a line to the probability, it uses odds.

[Regina] (24:54 - 25:09)
Which is really clever because odds can be any number between zero and infinity. So now instead of being trapped just between zero and one, we can have all of the positive numbers in zero to play with on our graph, which is better.

[Kristin] (25:09 - 25:23)
That is better. But of course, it still leaves out all the negative numbers and we want our line to be able to go anywhere. So we do one more math trick.

We take the log, the natural log of the odds, because when you take logs, that brings back in negative numbers.

[Regina] (25:24 - 25:37)
So for people who don't remember all of their algebra quite as well as we do, decimals are fractions between zero and one. When you take their log, they become a negative number.

[Kristin] (25:38 - 25:57)
Yes. So in other words, we are transforming our outcome into a log odds because log odds is a nice mathematical creature. So here we model not sex or no sex, but the log odds of sex on a first date.

And it turns out that because of this transformation, one of the things that the model ultimately spits back to us is an odds ratio.

[Regina] (25:58 - 26:07)
An odds ratio. And people often are just grabbing that odds ratio from their output and copy and pasting into their paper. Exactly.

[Kristin] (26:07 - 27:06)
But the interesting thing is if the design of the study is right, you actually don't have to report the odds ratio. There are other more useful measures that you can get out of logistic regression. Just often people I think aren't aware of this.

So Regina, let's now get back to the sugary beverage study. That was actually a randomized intervention trial, and that design allows you to pull many other useful things out of logistic regression. For example, you can get adjusted probabilities, such as the adjusted percentage of sugary beverages purchased.

Now, to be fair, the authors did actually present some adjusted probabilities from logistic regression in their paper. That's in figure two. But they presented these adjusted probabilities for eight different beverage types, and they didn't give the overall number.

And they also didn't break it down by type of sign. And in the text, they just refer to the p-values from that analysis and never to the effect sizes. So none of that was in their abstract, and none of that was what they focused on in their results.

[Regina] (27:06 - 27:18)
So it seems like they realized they could put other numbers, other values in the results that they could get from their logistic regression, but they still made odd ratio the whole centerpiece.

[Kristin] (27:19 - 28:11)
Yeah, that's exactly the issue. They focused on the odds ratio rather than these other measures. It turns out, Regina, that the adjustment wasn't even that important because the unadjusted and adjusted probabilities are pretty much the same.

That isn't surprising given that this was a randomized experiment, which means we would not expect much confounding. The other thing besides the adjusted probabilities that they could have and probably should have presented here is that they could have taken those adjusted odds ratios from logistic regression and converted them into an adjusted risk ratio. You can do this very easily, actually, with a mathematical formula.

And of course, I went ahead and did that. And the adjusted risk ratio here comparing the exercise sign to baseline was 0.92, which means there was an 8% relative reduction in the likelihood of buying a sugary beverage when the exercise sign was posted. Clearly not a 50% reduction in the likelihood.

[Regina] (28:12 - 28:21)
Wow. So people were saying 50% reduction in likelihood, huge exaggeration. It was just an 8% reduction.

Yes.

[Speaker 3] (28:21 - 28:21)
Wow.

[Regina] (28:22 - 28:55)
So 8% reduction in relative risk also, besides being accurate, is much more intuitive than 50% reduction in odds. So they could have reported the intuitive framing and did it accurately, but it would have required an extra step, right, this mathematical formula. So I'm guessing that's why they didn't do it?

They didn't take the time? Maybe. Or maybe they just didn't know how to do this, right?

Ah. But we will put details about how to do this in the show notes.

[Kristin] (28:55 - 29:05)
Right, for other people. The skeptical, cynical side of me says maybe also that they did know how to do it, but they just wanted to present the more exaggerated and dramatic-looking numbers, possibly.

[Regina] (29:06 - 29:09)
But with our show notes, they will have no excuse. Exactly.

[Kristin] (29:09 - 29:10)
Yes, exactly. It will be easy to do it.

[Regina] (29:11 - 29:53)
There you go, yes. Okay, so I want to just kind of like bring this back to the bottom line. So interpreting these results as a 50% reduction in the probability of kids buying sugary beverages just wrong and misleading.

It's misleading because we talked about, Kristin, when the outcome is common, the odd ratio is very different, much bigger, much more dramatic than the risk ratio. And the outcome here was super common because you said, what, 93% of teens at baseline— Very common, yes. —were buying it.

Okay, so odd ratio looks so much different than the risk ratio here. And that is why you need to get it accurately. And if you're not accurate, it is very misleading here.

[Kristin] (29:53 - 31:53)
Yeah, I mean, there's nothing technically wrong with presenting an odds ratio here. But I almost want to say, when your outcome is this common, don't present an odds ratio just because it's bound to be misleading, right? It's probably not the best choice.

And there's so many other numbers you could present. Why focus on the odds ratio when there are these alternatives? And Regina, I want to point out another problem caused by the use of odds ratios here.

The other two signs also reduced sugary drink purchases to 87.5% and 86.5%. Remember, it was 86% with the exercise sign. So basically, the signs were all about the same. But once you convert those numbers into odds ratios, those tiny, non-significant differences of 0.5 to 1.5 percentage points, those get exaggerated. The differences look much bigger in odds ratio terms. And that led people to mistakenly conclude that the exercise signs were more effective than the other signs. And Regina, you know, because of all this, I was a little critical in my letter to the editor.

[Regina]
No. You?

[Kristin]
I know, shocking, right?

I'm never mean. Never critical. The authors did get to write a response, because authors get the last word with letters to the editor.

But their response was a little unsatisfying, Regina, because three separate times in their response, they blamed the media. Okay, they said, we cannot be responsible for the media's interpretation of our results. And then they said, we cannot be responsible for how the media presents our results.

And finally, they said, how the media reported our results is outside of our control. Wow. They were really passing the buck.

Here's the hilarious part, though, Regina. The authors had actually released a video for the press, where they themselves interpreted the results wrong. I'm going to put a link to this video in the show notes, but I'm also going to play a clip from the video right now.

This is the first author of the paper speaking.

[Researcher] (31:53 - 32:07)
Of those three signs, the one that was most effective was the physical activity equivalent, or telling the adolescents that a bottle of soda is about 50 minutes of jogging. And we found that when that sign was posted, the likelihood that they would buy a sugary beverage reduced by about 50%.

[Regina] (32:07 - 32:19)
No way. You're kidding. So they're out there blaming the media, oh, it's the media's fault, when they themselves were the ones that were saying 50% reduction.

[Kristin] (32:20 - 32:27)
That's exactly what they fed the media.

So yes, if you say it, you are responsible for what the media picks up from your own interpretation. This is just crazy.

[Regina] (32:27 - 32:35)
Did they think that you would not notice there was evidence out there of them getting it wrong?

[Kristin] (32:35 - 32:58)
Yeah, absolutely. I'm sure they thought that I would probably never go to the university website and check the press kit, or at least nobody who was reading their response would bother. But Regina, I have to say, paperwork, facts, documentation, evidence, kind of my thing.

I am pretty good, actually, at calling BS. So, you know, you probably don't want to pick a fight with me in a venue where facts matter.

[Regina] (32:58 - 33:09)
Oh, absolutely not. You are scary, Kristin. I would never do that.

Yeah, also, you're a marathon runner, so you've got the stamina. No matter what, you're going to keep going.

[Kristin] (33:10 - 33:46)
You cannot scare me with homework or paperwork. Yeah, sorry. And Regina, this is just a larger pet peeve of mine.

Sometimes, of course, the media does get things wrong. I'm not saying that never happens, obviously. But I also see a lot of scientists out there blaming the media for exaggerating or overstating things or getting the stats wrong.

But when you do the detective work and you trace it back to the original paper or the press release, it often turns out that it's the authors themselves who overinterpreted or exaggerated or got the stats wrong. And the reporter is, in fact, just parodying them.

[Regina] (33:47 - 34:00)
You and I give workshops for science journalists. And these science journalists have always been very interested, like passionately interested in getting things right. Science journalists are just, they're great to work with.

[Kristin] (34:00 - 34:23)
Absolutely. But they often trust the scientists a little too much. And again, sometimes just parrot the numbers that they're fed.

That's why we often try to teach them how to interpret numbers on their own so they don't have to trust the scientists. All right, Regina, I have one more example of exaggerated odds ratios that I want to give. This one actually involves a colleague of ours who was not afraid to admit when he got things wrong.

[Regina] (34:23 - 34:48)
Oh, I know the example you are talking about. It's a great one. Let's take a short break first.

Welcome back to Normal Curves. Today we're talking about odds ratios. And we were about to hear about another real-life story when odds ratios go wrong.

[Kristin] (34:48 - 35:00)
Okay, this is a fun story because it involves Kevin Schulman, a professor at Stanford who leads the MCIM program that we teach in. And Regina, remind me, I always forget what that acronym stands for. What does that stand for again?

[Regina] (35:01 - 35:22)
Masters in Clinical Informatics Management, which is really a fantastic program. Very innovative, very interesting. Much more interesting than the name itself might suggest.

Clinical Informatics Management. And yeah, you and I co-teach a stats course in this program, which was actually the origin of this podcast.

[Kristin] (35:23 - 36:09)
Yeah, and I just, I have to give a little plug for the MCIM program right here because we have so much fun teaching in that program. And it is such a well-run program. Credit to Kevin.

And honestly, well-run programs are not always the norm at Stanford.

[Regina]
Or anywhere.

[Kristin]
That's true.

Yeah, not to pick just on Stanford. All right, Kevin though is great because when we were preparing for this class, he pointed out a paper of his that a lot of people now use for teaching odds ratios. And it turned out I had actually used that example in my own teaching years before.

I just hadn't realized that the paper was his paper. And what I appreciated was that he was very open about how the odds ratios in his paper ended up misleading people. And instead of getting defensive about it, he was actually happy to make it into a useful teaching example.

[Regina] (36:09 - 36:21)
I love that Kevin's not hiding things because honestly, making mistakes is human. This is going to happen. But the key thing is transparency and learning from it, which is what he's doing.

[Kristin] (36:21 - 36:26)
Exactly. And this was a study that was published in 1999 in the New England Journal of Medicine.

[Regina] (36:26 - 36:30)
1999, an oldie, Kristin, but a goodie.

[Kristin] (36:30 - 36:31)
We were in grad school then, yes.

[Regina] (36:31 - 36:32)
We were in grad school.

[Kristin] (36:32 - 37:04)
So they were looking at whether race and sex influenced doctors' decisions to refer patients for something called cardiac catheterization. Basically, when a patient is having some specific symptoms, the doctor should be referring them for that procedure. And they wanted to see if doctors were biased against specific groups, like maybe doctors refer women less than men.

The setup was actually pretty clever. They recruited hundreds of physicians at a conference, and they showed them videos of fake patients with chest pain.

[Regina] (37:04 - 37:05)
Fake patients, like actors, not puppets.

[Kristin] (37:06 - 37:33)
Exactly. These were actors on a video. And they were playing out the same scenario. So all the scenarios were identical.

The patients reported identical symptoms, identical test results. And the only thing that changed from video to video was the patient's sex and race. So imagine a doctor watches one version where the patient is a white man.

Another doctor sees basically the same patient scenario, except now the patient is a Black woman. And then the researchers asked, would you recommend cardiac catheterization?

[Regina] (37:33 - 37:39)
Which is really such a clever experiment. I love this. Like, very simple and elegant.

So tell us what they found.

[Kristin] (37:39 - 38:05)
Well, again, I'm going to start with the media headlines, because those were very dramatic. The media headlines were things like, doctor bias may affect heart care. Doctors less likely to refer Blacks than women.

Heart care reflects race and sex, not symptoms. And this actually got picked up by the New York Times. And one of the things that they reported was the following.

They said that doctors were 60% less likely to order the test for Black women compared with white men.

[Regina] (38:06 - 38:12)
Wow. 60% less likely sounds huge. But exaggerated, maybe?

[Kristin] (38:13 - 38:26)
Yes, you can probably guess by now that that is exaggerated. Because if we look at the raw percentages, which were available in the paper, it was about 91% of white men who were referred versus 79% of Black women. Okay.

[Regina] (38:26 - 38:43)
That is less dramatic sounding. So 91% to 79%, that is not a 60% drop. It's not even a 50% drop.

So same thing we talked about before. I'm guessing they were using risk ratio words to interpret odds ratios. Yeah.

[Kristin] (38:43 - 39:23)
Yes, exactly. And, you know, that is some amount of a drop, right? 91% to 79%, it's not nothing.

And it was a statistically significant difference. But the problem is that the magnitude was vastly exaggerated. And you can guess why.

Again, as before, they used logistic regression in their paper to try to adjust for potential confounders. And the odds ratio that they reported for Black women compared with white men, that was 0.4, which would translate to a 60% drop in odds, but not in risk. This totally misled people, including the New York Times.

And actually, the New York Times ended up having to print a correction.

[Regina] (39:24 - 40:08)
Oh, I like that they printed a correction here. Yeah. Saying the record straight.

Because odds are not the same thing as likelihoods. And I think it's worth just emphasizing this because we use those words interchangeably in everyday English. You know, we say that the odds of this or the likelihood of this, but statisticians, we are kind of anal retentive and they are very different things.

And you led us through the math before, but I think sometimes we forget about it when we get to the English part. And especially when the outcome is common, like it was here, you said, what, 91% of white men were referred? Yeah.

That is especially different.

[Kristin] (40:08 - 40:50)
Right. It is really common. And I would argue that in a case like this, again, when you have an outcome that is so common, you probably should not report the odds ratios at all.

That doesn't mean you can't run logistic regression. It just means that you should report instead the adjusted risks, as we talked about, or you should translate that odds ratio into an adjusted risk ratio using that mathematical formula. So Regina, that means a lot of cases where people are reporting odds ratios, I would argue that they shouldn't report them.

But now I want to actually talk about a case when the odds ratios are actually a good statistical choice. They do have a purpose. Particularly, they are useful for a certain study design called the case control study.

[Regina] (40:51 - 40:56)
Ah, yes. Case control studies, we talked about these back in the diagnostic testing episode.

[Kristin] (40:56 - 42:12)
We did. And just to remind everybody, a case control study, that means that researchers go out and they recruit people who have already developed the outcome or disease of interest. And they recruit controls, which is people who don't have the outcome or disease of interest.

And the researchers control the ratio of cases to controls in the sample. Often, there's one control for every case. In other words, the researchers make it so that half of the sample has the disease, but that does not mean that that reflects the prevalence of disease in the real world.

This is important because there are certain statistics that we then cannot calculate directly from a case control study because of this distortion of the sample. And in the diagnostic testing episode, we talked about how it was incorrect to calculate positive predictive value from a case control study. Similarly here, we cannot calculate risks or risk ratios from a case control study.

And I want to illustrate that now with a little hypothetical example. For example, let's say we do a case control study with 50 lung cancer patients and 50 controls. And let's say we find 40 smokers in the case group versus only 10 in the control group.

That's a total of 50 smokers. So Regina, pop quiz, what would you tell me if I ask you, what is the risk of lung cancer in smokers from this study?

[Regina] (42:13 - 42:18)
I would say, trick question, Kristin. You were just checking to see whether I was awake.

[Kristin] (42:18 - 42:19)
Whether you're paying attention, yes.

[Regina] (42:19 - 43:10)
You cannot calculate risks or risk ratios from a case control study. And of course, we're going to be tempted, very tempting to look at these nice numbers and say, oh, look, we had 50 smokers in the sample, 40 in 10, and 40 of them have lung cancer. Therefore, the risk of lung cancer in smokers is 40 out of 50 or 80 percent.

But that would be wrong. So tempting, but wrong. Because our sample is what we call enriched for lung cancer cases.

Like you said, we went out and we found 50 lung cancer patients and 50 controls without lung cancer. So we controlled those numbers. So we cannot say anything about the risk of lung cancer from this case control design.

[Kristin] (43:11 - 43:46)
We can't say anything about risk. We can't say anything about risk ratio. But we are allowed to calculate a very specific odds ratio here.

We can calculate the odds ratio representing the increased odds of being a smoker in cases versus controls. And that turns out to be 16 here because the odds of being a smoker in the case group, that was 4 to 1, while the odds of being a smoker in the control group was 1 to 4, 4 divided by one fourth, that's 16. And that means cases have a 16-fold increase in the odds of being a smoker.

[Regina] (43:47 - 44:08)
But it's backwards, Kristin. I don't really want to know, OK, if someone has cancer, what were their odds of having been a smoker? That's going backwards in time.

What I want to know is, if someone smokes, what are their odds of going on to develop cancer? Exactly.

[Kristin] (44:09 - 44:43)
So it's valid to calculate this odds ratio for being a smoker. But here's the really cool part. It turns out that odds ratios have this amazing mathematical symmetry.

And it turns out that this odds ratio for smoking in cases versus controls, which is valid to calculate from our sample, that is actually mathematically identical to the odds ratio for lung cancer in smokers versus non-smokers. So we can conclude not just that being a case increases the odds of smoking 16-fold, but also that smoking increases the odds of lung cancer 16-fold.

[Regina] (44:44 - 44:54)
Just kind of magical when you think about it. It is. This mathematical symmetry of odds ratio, right?

Like you can go forward and backwards. Yeah. And it's the same thing, but only for odds ratios.

Yes.

[Kristin] (44:55 - 44:57)
This doesn't work for risk ratios, we should point out.

[Regina] (44:57 - 45:26)
No, it does not. The nice thing here is that lung cancer is pretty rare, thankfully. So we actually can use odd ratio as an approximation for the risk ratio of lung cancer.

We can only do it because it's rare. So even if we were to get it wrong and say that there is a 16-fold increase in the risk of lung cancer, we're not going to be far off. So even if we're getting it wrong, we're not getting it too wrong.

[Kristin] (45:26 - 45:56)
Right. It's a pretty good approximation of the risk ratio here. So if we interpret it as a risk ratio, we're not too far off.

And remember, I told you there's a formula that we can use to convert odds ratios to risk ratios. To use that formula, we need to know the risk of lung cancer in the unexposed group, the non-smoker group here. Let's say that was 0.5%, or 1 in 200 non-smokers are going to get lung cancer. If I plug that number into this formula, we get that the risk ratio here is 15. So we're still in misinterpretation.

[Regina] (45:56 - 46:10)
I just want to be clear to say that the odd ratio means there's a 16-fold increase in the risk of lung cancer. Still wrong, but because the outcome is rare, it's close enough. It doesn't exaggerate too much.

Right.

[Kristin] (46:10 - 46:51)
The odds ratio and risk ratio are so close if we use those terms interchangeably, we're pretty much fine for case control studies. And so I want to talk and give one more example, a classic case control study, just to wrap up here. This was published in JAMA in 1982, and this was a case control study on something called Reye syndrome, or sometimes called Reye's syndrome, which is a rare but terrifying disease in children that causes brain swelling and liver failure.

Reye syndrome can occur after a child has a viral illness like chickenpox or flu. It is super rare. At the time, in the early 1980s, it was something like 1 in 100,000 kids would get this per year in the U.S. Because it's so rare then, basically you've got to do a case control study on this.

[Regina] (46:52 - 47:00)
That's the only way you're going to get enough cases. One out of 100,000 kids, you can't just follow a bunch of kids and collect all the ones that have Reye syndrome.

[Kristin] (47:00 - 48:47)
Exactly. You kind of stuck with a case control study, and that's what they did here. So the cases were children who had had Reye syndrome, and the controls were children who had also been sick with viral illnesses, but did not develop Reye syndrome.

And they asked about what medications the children had been given during their illness, because they suspected that Reye syndrome might be related to medication. And they found a big difference. About 93% of the cases had taken aspirin during their illness, versus only about 29% of the controls.

And that translated with some rounding to an odds ratio of 26. A 26-fold increase in odds.

[Regina]
That is gigantic.

[Kristin]
That's huge. Is gigantic. But here's the thing, Regina, the association between aspirin and Reye syndrome is actually gigantic.

Unlike in, say, our sugary beverage example or the cardiac catheterization example, Reye syndrome is incredibly rare. So the odds ratio is actually not an exaggeration. It's a really good approximation of the risk ratio.

So I went back and I plugged into that mathematical formula to translate the odds ratio to a risk ratio, assuming a baseline risk of one per 100,000. And the risk ratio here turns out to be 25.99. So it's basically almost identical to the odds ratio. It just turns out that this is a really strong association, and this is why you are told to never give your children aspirin when they have a viral illness.

You are only supposed to give them ibuprofen or acetaminophen and Tylenol. So bottom line lesson here is that for case control studies, it's often totally appropriate and great to present the odds ratios. And it's even fine to interpret those odds ratios loosely as if they're risk ratios, as long as the disease is rare.

Lots of times for case control studies, the whole point of doing a case control study is that the disease was rare. That's why you're doing a case control study. So in that context, it's okay if you're a little loosey-goosey.

[Regina] (48:48 - 48:59)
Loosey-goosey. Okay. But of course, many people, Kristin, want to know where is the line when it's okay to be loosey-goosey?

What counts as rare enough that I can go ahead and do this little shortcut?

[Kristin] (49:00 - 49:37)
Yes. My students always want to know, well, what's that line? And I want to emphasize that this is a rule of thumb that I'm about to give.

So it is not a hard and fast line. You always have to look at context and a specific situation. But as just kind of a general rule of thumb, we often say if the outcome occurs in less than 10% of the unexposed group, we can consider it kind of rare.

I should point out that how much the odds ratio exaggerates the risk ratio, it depends not only on how common the outcome is, but also how strong the effect is. So if the outcome is both common and the association is big, that's when the exaggeration is the worst. We'll put more details and references about all of this in the show notes, Regina.

[Regina] (49:37 - 49:45)
So, Kristin, I think we're ready to wrap all of this up. To rate the strength of evidence for the claim, repeat the claim for us again.

[Kristin] (49:45 - 49:53)
The claim today was that many people writing and reading the medical literature don't actually understand odds ratios.

[Regina] (49:54 - 50:19)
That is a great claim. Okay, how do we rate the strength of evidence on the podcast for the claim? Our smooch rating scale, highly scientific, almost trademarked.

We're in the process. One to five, one smooch means a little to no evidence for the claim. Five means very strong evidence for the claim.

So, Kristin, kiss it or diss it. What do you say?

[Kristin] (50:20 - 51:00)
So, Regina, I am going to go four smooches here. I do want to put in the caveat that in this episode, we have not really talked about any systematic empirical evidence other than David's zero and five study, showing that, in fact, many people make these kinds of mistakes. There are some empirical studies that have looked at this and, you know, questioned doctors about their understanding or looked at how many times randomized trials report odds ratios when they really shouldn't.

So, there's some evidence that we haven't presented here that I'm basing this on and also just my anecdotal years of experience of seeing people getting this wrong. I'm going four smooches on this one. How about you, Regina?

[Regina] (51:01 - 51:26)
You know, I'm going to go with three smooches on this simply because I know that you cherry-picked what the evidence was. Three out of four to illustrate a point, but also because I'm very hopeful and positive and optimistic about humans and thinking, okay, maybe people understand it more than we think. But totally, totally arbitrary.

Three smooches.

[Kristin] (51:26 - 51:38)
Well, maybe our evaluation of the evidence and putting out this podcast is going to educate everybody. And so, we've then impinged on the outcome. That's called something.

What is that called? Like the Hawthorne effect? Is that Schrödinger's cat?

[Regina] (51:39 - 51:47)
This is true. We have nudged it. We can no longer measure it because we are now a part of it.

Yeah. Self-fulfilling prophecy, maybe.

[Kristin] (51:47 - 51:52)
Yes. Now we've educated the world because everybody's going to listen to this podcast and we're all good. Nobody will ever make this mistake again.

[Regina] (51:53 - 51:56)
I love how optimistic you are sometimes.

[Kristin] (51:56 - 51:56)
Sometimes.

[Regina] (51:57 - 51:58)
Not often.

[Kristin] (52:00 - 52:14)
Methodologic moral. Do you have one? You know, Regina, I'm going to go straight to the heart of this episode, which is a methodologic episode, and say, here's mine.

Just because logistic regression gives you an odds ratio does not mean you have to report it.

[Regina] (52:15 - 52:19)
Oh, I love it. Just because you can doesn't mean that you should.

[Kristin] (52:19 - 52:31)
And in the episode and in the show notes, we're giving people lots of alternatives to reporting that odds ratio from logistic regression so you don't just have to default to the odds ratio. All right, Regina, how about you on methodologic morals?

[Regina] (52:32 - 52:46)
So, Kristin, I'm going to go straight for the media one on this because I was really indignant about people calling me the media. How about this? A lot of bad science communication starts long before the journalist even enters the story.

[Kristin] (52:46 - 52:55)
Oh, I love that. Yes, Regina. These scientists trying to pawn off the blame on the media when it's really what was in their paper.

Yeah, great one. Yep.

[Regina] (52:55 - 53:08)
I feel like all scientists have a certain obligation for communicating their science clearly. Yes, exactly. Kristin, this has been delightful considering that we talked about odds ratios.

[Kristin] (53:09 - 53:13)
Is anybody still listening? There was a lot of numbers in this one, Regina.

[Regina] (53:13 - 53:27)
A lot of numbers. But like David said, it's really crucial to understand both to be a skeptical consumer of the headlines and what we're reading about and also reading the scientific literature yourself.

[Kristin] (53:28 - 53:33)
And I have faith that people like numbers at some level and they will listen to numbers when it's explained well.

[Regina] (53:34 - 53:41)
Oh, very nice. No nightmares.

[Kristin]
No, no, we explained the numbers well.

All right, Regina. Thank you.

[Regina]
Thanks, Kristin.

Thanks, everyone for listening.