Hypothesis Testing and P-values

Lucy D’Agostino McGowan

Hypothesis testing

Null hypothesis: \(\beta_1 = 0\)
Alternative hypothesis: \(\beta_1 \neq 0\)

Under the null hypothesis

Under the null \((\beta_1 = 0)\) the t-statistic \((\hat\beta_1/se_{\hat\beta_1})\) has a t-distribution with \(n-2\) degrees of freedom.

Code

null <- tibble(
  t = rt(10000, df = 38)
)

ggplot(null, aes(t)) +
  geom_histogram(bins = 30)

Example

What t statistic did we observe?

Code

mod <- lm(frequency_score ~ group, data = sample)
summary(mod)


Call:
lm(formula = frequency_score ~ group, data = sample)

Residuals:
   Min     1Q Median     3Q    Max 
-38.80 -22.55 -11.80  30.20  95.20 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)   39.800      7.757   5.131 8.82e-06 ***
groupsquare   -8.000     10.970  -0.729     0.47    
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 34.69 on 38 degrees of freedom
Multiple R-squared:  0.0138,    Adjusted R-squared:  -0.01215 
F-statistic: 0.5318 on 1 and 38 DF,  p-value: 0.4703

Under the Null Hypothesis

Code

library(geomtextpath)
ggplot(null, aes(t)) +
  geom_histogram(bins = 30) + 
  geom_textvline(xintercept = c(0.729), label = "observed t statistic (flipped)") + 
  geom_textvline(xintercept = c(-0.729), label = "observed t statistic")

How do we compare these to the distribution under the null?

p-value

The probability of observing a statistic as extreme or more extreme than the observed test statistic given the null hypothesis is true

Under the Null Hypothesis

Code

null$color <- ifelse(null$t < 0.729 & null$t > -0.729, "out", "in")
ggplot(null, aes(t, fill = color)) +
  geom_histogram(bins = 30) + 
  geom_textvline(xintercept = c(0.729), label = "observed t statistic (flipped)") + 
  geom_textvline(xintercept = c(-0.729), label = "observed t statistic") + 
  theme(legend.position = "none")

Under the Null Hypothesis

The proportion of area greater than 0.729

Code

null$color <- ifelse(null$t < 0.729 & null$t > -0.729, "out", "in")
ggplot(null, aes(t, fill = color)) +
  geom_histogram(bins = 30) + 
  geom_textvline(xintercept = c(0.729), label = "observed t statistic (flipped)") + 
  geom_textvline(xintercept = c(-0.729), label = "observed t statistic") + 
  theme(legend.position = "none")

pt(0.729, df = 40-2, lower.tail = FALSE)

[1] 0.2352356

Under the Null Hypothesis

The proportion of area less than -0.729

Code

null$color <- ifelse(null$t < 0.729 & null$t > -0.729, "out", "in")
ggplot(null, aes(t, fill = color)) +
  geom_histogram(bins = 30) + 
  geom_textvline(xintercept = c(0.729), label = "observed t statistic (flipped)") + 
  geom_textvline(xintercept = c(-0.729), label = "observed t statistic") + 
  theme(legend.position = "none")

pt(-0.729, df = 40-2)

[1] 0.2352356

Under the Null Hypothesis

The proportion of area greater than 0.729 or less than -0.729

Code

null$color <- ifelse(null$t < 0.729 & null$t > -0.729, "out", "in")
ggplot(null, aes(t, fill = color)) +
  geom_histogram(bins = 30) + 
  geom_textvline(xintercept = c(-0.729), label = "observed t statistic") + 
  geom_textvline(xintercept = c(0.729), label = "observed t statistic (flipped)") + 
  theme(legend.position = "none")

pt(0.729, df = 40-2, lower.tail = FALSE) + pt(-0.729, df = 40-2)

[1] 0.4704711

Under the Null Hypothesis

The proportion of area greater than 0.729 or less than -0.729

Code

null$color <- ifelse(null$t < 0.729 & null$t > -0.729, "out", "in")
ggplot(null, aes(t, fill = color)) +
  geom_histogram(bins = 30) + 
  geom_textvline(xintercept = c(-0.729), label = "observed t statistic") + 
  geom_textvline(xintercept = c(0.729), label = "observed t statistic (flipped)") + 
  theme(legend.position = "none")

pt(0.729, df = 40-2, lower.tail = FALSE) * 2

[1] 0.4704711