10:00
Logistic Regression
Lucy D’Agostino McGowan
Outcome variable
- So far, we’ve only had continuous (numeric, quantitative) outcome variables ( \(y\) )
- We’ve just learned about categorical and binary explanatory variables ( \(x\) )
- What if we have a binary outcome variable?
Outcome variable
What does it mean to be a binary variable?
- So far, we’ve only had continuous (numeric, quantitative) outcome variables ( \(y\) )
- We’ve just learned about categorical and binary explanatory variables ( \(x\) )
- What if we have a binary outcome variable?
Application Exercise
- Get into groups of 3
- Create a “paper football”. Choose one person to be the “player” one to be the “goal” and one to enter the data.
- Pick different distances to start the football and have the player attempt to flick the football into the goal.
- Enter your data at bit.ly/sta-112-s24-football
Binary variable
- Whether the player scored is a binary variable
- Suppose we are interested in answering: Does the distance from the goal is related to whether the player scored?
- What would happen if we try to use linear regression to examine this?
DEMO
Logistic
- Perhaps it would be sensible to find a function that would not extend the predicted probabilities beyond 0 and 1?
- The logistic function has this property
- Instead of modeling the probability that the outcome is 1 (\(P(Y = 1 | x) = \beta_0 + \beta_1x + \varepsilon\)), we can model:
- \(\log\left(\frac{P(Y=1|x)}{1 - P(Y = 1|x)}\right) = \beta_0 + \beta_1 x\)
Why is logistic better?
\[\log\left(\frac{P(Y=1|x)}{1 - P(Y = 1|x)}\right) = \beta_0 + \beta_1 x\]
for simplicity, let’s write \(P(Y=1|x)\) as \(p\)
Why is logistic better?
\[\log\left(\frac{p}{1 - p}\right) = \beta_0 + \beta_1 x\]
Why is logistic better?
\[\exp\left\{\log\left(\frac{p}{1 - p}\right)\right\} = \frac{p}{1 - p} = \exp\{\beta_0 + \beta_1 x\}\]
Why is logistic better?
\[{\require{color}\colorbox{#86a293}{$\exp$}}\left\{\log\left(\frac{p}{1 - p}\right)\right\} = \frac{p}{1 - p} = {\require{color}\colorbox{#86a293}{$\exp$}}\{\beta_0 + \beta_1 x\}\]
Let’s start with ‘undoing’ the log
Why is logistic better?
\[\frac{p}{1 - p} = e^{\beta_0 + \beta_1 x}\\ p = (1-p)e^{\beta_0 + \beta_1x}\]
Why is logistic better?
\[\frac{p}{1 - p} = e^{\beta_0 + \beta_1 x}\\ p = ({\require{color}\colorbox{#86a293}{$1-p$}})e^{\beta_0 + \beta_1x}\]
Move (1-p) to the other side of the equation
Why is logistic better?
\[p = (1-p)e^{\beta_0 + \beta_1x}\\ p (1+e^{\beta_0+\beta_1x}) = e^{\beta_0 + \beta_1x}\]
Why is logistic better?
\[p = (1-p)e^{\beta_0 + \beta_1x}\\ p (1+e^{\beta_0+\beta_1x}) = e^{\beta_0 + \beta_1x}\]
Distribute that \((1-p)\) and then move the part with \(p\) back to the other side of the equation so we can solve with respect to \(p\)
Why is logistic better?
\[p (1+e^{\beta_0+\beta_1x}) = e^{\beta_0 + \beta_1x}\\ p = \frac{e^{\beta_0 + \beta_1x}}{1 + e^{\beta_0 + \beta_1x}}\]
Why is logistic better?
\[p (1+e^{\beta_0+\beta_1x}) = e^{\beta_0 + \beta_1x}\\ p = \frac{e^{\beta_0 + \beta_1x}}{1 + e^{\beta_0 + \beta_1x}}\]
Solve for \(p\)!
Why is logistic better?
We can equivalently write this as
\[p = \frac{1}{1 + e^{-(\beta_0 + \beta_1x)}}\]
- What is this if \(\beta_0 + \beta_1x\) is very large?
- exp of a very negative number is 0: 1 / (1 + 0) = 1
- What is this if \(\beta_0 + \beta_1x\) is very small (i.e. very negative)?
- exp of a very large number is infinity: 1 / (VERY LARGE) \(\approx\) 0
- The probability is bounded by 0 and 1, yay!
Notation
- \(\log\left(\frac{p}{1-p}\right)\): the “log odds”
- \(p\) is the probability that \(y = 1\) - the probability that your outcome is 1.
- \(\frac{p}{1-p}\) is a ratio representing the odds that \(y = 1\)
- \(\log\left(\frac{p}{1-p}\right)\) is the log odds
- The transformation from \(p\) to \(\log\left(\frac{p}{1-p}\right)\) is called the logistic or logit transformation
A bit about “odds”
- The “odds” tell you how likely an event is
- 👛 if I flip a fair coin, what is the probability that I’d get heads?
- \(p = 0.5\)
- What is the probability that I’d get tails?
- \(1 - p = 0.5\)
- The odds are 1:1, 0.5:0.5
- the odds can be written as \(\frac{p}{1-p} =\frac{0.5}{0.5} = 1\)
A bit about “odds”
- The “odds” tell you how likely an event is
- ☂️ Let’s say there is a 60% chance of rain today
- What is the probability that it will rain?
- \(p = 0.6\)
- What is the probability that it won’t rain?
- \(1-p = 0.4\)
- What are the odds that it will rain?
- 3 to 2, 3:2, \(\frac{0.6}{0.4} = 1.5\)
Odds ratios
| __ | Scored | Missed | Total |
|---|---|---|---|
| Distance < 9 | 39 | 22 | 61 |
| Distance \(\ge\) 9 | 61 | 78 | 139 |
| Total | 100 | 100 | 200 |
- What are the odds of scoring for the shots that were taken from < 9 inches away?
- \((39/61)/(22/61) = 39/22 = 1.77\)
- What are the odds of scoring for shots taken from 9 or more inches away?
- \(61/78 = 0.78\)
- Comparing the odds what can we conclude?
- Being closer increases the odds of success
Odds ratios
| __ | Scored | Missed | Total |
|---|---|---|---|
| Distance < 9 | 39 | 22 | 61 |
| Distance \(\ge\) 9 | 61 | 78 | 139 |
| Total | 100 | 100 | 200 |
- We can summarize this relationship with an odds ratio: the ratio of the two odds
\(OR = \frac{39/22}{61/78} = \frac{1.77}{0.78} = 2.27\)
“the odds of scoring was 2.27 times higher when the shots were within 9 inches compared with those further away”
Application Exercise
- Using your group’s data, fill in the table counting how many times your player scored / missed based on their shot distance (categorized as \(<9\) or \(\ge 9\))
- Answer the questions about the corresponding probabilities / odds to calculate the odds ratio quantifying the impact of distance on whether the player scored
- Interpret this result
08:00
DEMO
Transforming logs
- How do you “undo” a \(\log\) base \(e\)?
- Use \(e\)! For example:
- \(e^{\log(10)} = 10\)
- \(e^{\log(1283)} = 1283\)
- \(e^{\log(x)} = x\)
Transforming logs
How would you get the odds from the log(odds)?
- How do you “undo” a \(\log\) base \(e\)?
- Use \(e\)! For example:
- \(e^{\log(10)} = 10\)
- \(e^{\log(1283)} = 1283\)
- \(e^{\log(x)} = x\)
- \(e^{\log(odds)}\) = odds
Transforming odds
- odds = \(\frac{p}{1-p}\)
- Solving for \(p\)
- \(p = \frac{\textrm{odds}}{1+\textrm{odds}}\)
- Plugging in \(e^{\log(odds)}\) = odds
- \(p = \frac{e^{\log(odds)}}{1+e^{\log(odds)}}\)
- Plugging in \(\log(odds) = \beta_0 + \beta_1x\)
- \(p = \frac{e^{\beta_0 + \beta_1x}}{1+e^{\beta_0 + \beta_1x}}\)
The logistic model
- ✌️ forms
| Form | Model |
|---|---|
| Logit form | \(\log\left(\frac{p}{1-p}\right) = \beta_0 + \beta_1x\) |
| Probability form | \(\Large{p} = \frac{e^{\beta_0 + \beta_1x}}{1+e^{\beta_0 + \beta_1x}}\) |
The logistic model
| probability | odds | log(odds) |
|---|---|---|
| \(p\) | \(\frac{p}{1-p}\) | \(\log\left(\frac{p}{1-p}\right)=l\) |
⬅️
| log(odds) | odds | probability |
|---|---|---|
| \(l\) | \(e^l\) | \(\frac{e^l}{1+e^l} = p\) |
The logistic model
- ✌️ forms
- log(odds): \(l = \beta_0 + \beta_1x\)
- P(Outcome = Yes): \(\Large{p} =\frac{e^{\beta_0 + \beta_1x}}{1+e^{\beta_0 + \beta_1x}}\)
