Confidence Intervals
Idea: suppose we collected (i.i.d.) measurements X_1, \dotsc, X_n which have some population mean \mu and variance \sigma^2.
We calculate the average:
A_n = \frac{A_1 + \dotsb + A_n}{n}
and we want to estimate the value of \mu from the collected data.
Previously, we've discussed the MLE, the single most likely estimate of the parameter.
Now, we'd like to give a range [\mu_{\ell}, \mu_h] where we can say something like: "we are 95% confident that \mu \in [\mu_{\ell}, \mu_h]."
From the CLT, if n is large enough, we approximate A_n \approx \Norm\left(\mu, \frac{\sigma^2}{n}\right).
We can think of an interval centered at \mu as a collection of numbers within a certain distance d of \mu.
We'd like to choose a distance so that our measured A_n is within d of \mu.
We can shift our point of view here: if A_n is within d of \mu, then \mu is within d of A_n.
TODO: image
To pick the distance d, we want to make our interval large enough so that there's only a small amount of area under the normal curve outside of the interval.
So:
-
Pick some value \alpha \in (0, 1) (representing the area under the curve outside the interval)
-
Let c_{\alpha} = \Phi^{-1}\left(1 - \frac{\alpha}{2}\right) (this is the standardized z-score of the right-hand side of the interval)
-
Then, the interval will be:
\left[A_n - c_{\alpha} \sqrt{\frac{\sigma^2}{n}}, A_n + c_{\alpha} \sqrt{\frac{\sigma^2}{n}}\right].
For \alpha = 0.05, we'll have c_{\alpha} = \Phi^{-1}\left(1 - \frac{0.05}{2}\right) = \Phi^{-1}(0.975) = 1.96.
So:
The 95% confidence limits are given by:
\mu_{\ell} \amp = A_n - 1.96 \sqrt{\frac{\sigma^2}{n}}
\mu_{h} \amp = A_n + 1.96 \sqrt{\frac{\sigma^2}{n}}
The term \sqrt{\frac{\sigma^2}{n}} is called the standard error of the mean.
(It's the standard deviation of A_n.)
We don't know the true value of \sigma^2, just like we don't know the true value of \mu.
So we'll have to use our sample of measurements to estimate \sigma^2, and then use that estimate to give us our range of values for \mu.
Let X_1, \dotsc, X_n be (i.i.d.) measurements.
Then the sample mean is:
A_n = \frac{X_1 + \dotsb + X_n}{n}
and the sample variance is:
s^2 = \frac{\sum (X_i - A_n)^2}{n - 1} = \frac{\left(\sum X_i^2\right) - n(A_n^2)}{n - 1}
Why n - 1 here? It turns out that, as defined above, s^2 is an unbiased estimator for \sigma^2, whereas it wouldn't be if we divided by n instead.
(We'll ommit both the calculation justifying this and a more conceptual explanation for now.)
So, we can adjust the confidence limits from the theorem:
\mu_{\ell} \amp = A_n - 1.96 \sqrt{\frac{s^2}{n}}
\mu_{h} \amp = A_n + 1.96 \sqrt{\frac{s^2}{n}}
Suppose we're given measurements X_1, X_2, X_3, X_4, X_5 below.
| i |
X_i |
| 1 |
19.2 |
| 2 |
20.1 |
| 3 |
21.3 |
| 4 |
20.7 |
| 5 |
19.8 |
We're just practicing the computation here, so we'll pretend that 5 measurements is large enough for the CLT to apply.
A_n \amp = \frac{19.2 + \dotsb + 19.8}{5} = 20.22
\sum X_i^2 \amp = 19.2^2 + \dotsb + 19.8^2 = 2046.87
s^2 \amp = \frac{2046.87 - 5(20.22^2)}{4} = 0.657
\sqrt{\frac{s^2}{n}} \amp = \sqrt{\frac{0.657}{5}} \approx 0.362
\mu_{\ell} \amp = 20.22 - 1.96(0.362) \approx 19.51
\mu_{h} \amp = 20.22 + 1.96(0.362) \approx 20.93
This way of computing confidence limits only applies if we can think of the parameter we're estimating as a mean.
Suppose we flip a coin 100 times and see 40 heads.
Find a 98% confidence interval for the bias p.
Let H_1, \dotsc, H_{100} indicate heads on each flip.
Let S_{100} = H_1 + \dotsb + H_{100} and A_n = \frac{S_n}{n}.
Then:
\E(A_n) \amp = \E\left(\frac{S_n}{n}\right) = \frac{1}{n}\E(S_n) = \frac{1}{n} np = p
So a 98% confidence interval for p is also a 98% confidence interval for p.
In this setting, we can simplify the computation of s^2 and \sqrt{\frac{s^2}{n}}, using our knowledge of the binomial distribution S_n.
\Var(S_n) \amp = np(1 - p)
\Var(A_n) \amp = \Var\left(\frac{S_n}{n}\right) = \frac{1}{n^2} \Var(S_n) = \frac{1}{n^2} np(1-p) = \frac{p(1-p)}{n}
So s^2 = p(1 - p) and \sqrt{\frac{s^2}{n}} = \sqrt{\frac{p(1-p)}{n}}.
We don't know the value of p, so we'll use the MLE \widehat{p} = \frac{40}{100} = 0.4.
So:
\sqrt{\frac{s^2}{n}} = \sqrt{\frac{(0.4)(0.6)}{100}} \approx 0.049.
For a 98% confidence interval, we have \alpha = 1 - 0.98 = 0.02, so:
c_{\alpha} = \Phi^{-1}\left(1 - \frac{\alpha}{2}\right) = \Phi^{-1}(0.99) \approx 2.33.
Then, the confidence limits are:
p_{\ell} \amp = 0.4 - 2.33(0.049) \approx 0.286
p_{h} \amp = 0.4 + 2.33(0.049) \approx 0.514
For comparison, the 95% confidence limits are:
p_{\ell} \amp = 0.4 - 1.96(0.049) \approx 0.304
p_{h} \amp = 0.4 + 1.96(0.049) \approx 0.496