Skip to main content

Section 5.3 Power of a Test

Recall that the power of a test is the probability of rejecting a false null hypothesis. To understand the power, we must know:
  1. What data would lead us to reject the null hypothesis?
  2. What is the probability of seeing such data?

Example 5.3.1.

We find a coin and wonder if it’s fair. Let \(\theta\) be the probability of the coin coming up heads on a flip. Let \(S\) count the number of heads in 100 flips. Under the null hypothesis that \(\theta = 0.5\text{,}\) \(S\) is binomial, with:
\begin{align*} \E(S) \amp = (n)(\theta) = (100)(0.5) = 50 \\ \Var(S) \amp = (n)(\theta)(1 - \theta) = (100)(0.5)(0.5) = 25 \end{align*}
Using a normal approximation, \(S \approx \Norm(50, 25)\text{.}\) In the calculation of the \(p\)-value for a 2-tailed test, we will eventually reach the expression:
\begin{gather*} \Pr(Z \geq z) + \Pr(Z \leq -z) \end{gather*}
for some \(z\)-score. To address the first question above, we can calculate, for a specified significance level \(\alpha\text{,}\) the \(z\)-score that represents data at the cutoff between accepting and rejecting the null hypothesis:
\begin{align*} \alpha \amp = \Pr(Z \geq z) + \Pr(Z \leq -z) \\ \amp = \Phi(-z) + (1 - \Phi(z)) \\ \amp = 2 \Phi(-z) \\ \Rightarrow \Phi(-z) \amp = \frac{\alpha}{2} \\ -z \amp = \Phi^{-1}\left(\frac{\alpha}{2}\right) \end{align*}
For the significance level \(\alpha = 0.05\text{,}\) we’ll have \(z = 1.96\text{.}\) Now, we convert back to values of \(S\text{,}\) keeping in mind that a continuity correction would be involved:
\begin{align*} \frac{S - 0.5 - 50}{\sqrt{25}} \amp = 1.96 \\ S \amp = 50.5 + 1.96 \sqrt{25} = 60.3. \end{align*}
We can’t flip a fractional number of heads, so it would take a result of 61 or more heads (or, equally extreme in the other direction, 39 heads or fewer) to reject the null hypothesis.
To address the second question, we find a obstacle. How can we calculate the probability of seeing data extreme enough data to reject \(H_0\) without knowing the true value of \(\theta\text{?}\) In fact, the power is not a single number, but a function of \(\theta\text{.}\) For each hypothetical value of \(\theta\text{,}\) we’ll calculate the probability of seeing extreme enough data to reject \(H_0\text{.}\)

Example 5.3.2.

Continuing the previous example, we know that, if \(S \geq 61\) or \(S \leq 39\text{,}\) we would reject \(H_0\text{.}\) For a given value of \(\theta\text{:}\)
\begin{align*} \E(S) \amp = n\theta = 100\theta \\ \Var(S) \amp = n\theta(1-\theta) = 100\theta(1-\theta) \\ \Pr(S \geq 61) + \Pr(S \leq 39) \amp \approx \Pr\left(Z \geq \frac{60.5 - 100\theta}{\sqrt{100\theta(1 - \theta)}}\right) + \Pr\left(Z \leq \frac{39.5 - 100\theta}{\sqrt{100\theta(1 - \theta)}}\right) \end{align*}
For example, if the true value of the parameter is \(\theta = 0.5\text{:}\)
\begin{align*} \Pr(S \geq 61) + \Pr(S \leq 39) \amp \approx \Pr(Z \geq 1.11) + \Pr(Z \leq -3.12) \\ \amp \approx (1 - 0.8665) + 0.0009 \\ \amp = 0.1344 \end{align*}
If, instead, \(\theta = 0.6\text{:}\)
\begin{align*} \Pr(S \geq 61) + \Pr(S \leq 39) \amp \approx \Pr(Z \geq 0.10) + \Pr(Z \leq -4.18) \\ \amp \approx (1 - 0.5398) + 0 \\ \amp = 0.4602 \end{align*}
(Note: for the \(z\)-score \(-4.18\text{,}\) which is below the range covered by SectionΒ A.1, we will treat the \(\Phi\) value as close enough to 0. For a \(z\)-score outside the range on the other side, we would treat the \(\Phi\) value as 1. If we needed more precision than that, we would consult an expanded table.)
We can see in the example that the power of the test was greater when \(\theta = 0.6\) than when \(\theta = 0.55\text{.}\) The null hypothesis was \(\theta = 0.5\text{.}\) It is reasonable that, if the coin is more different from the null hypothesis, then it will have a higher probabiilty of producing data extreme enough to reject the null hypothesis.

Exercises Exercises

1.

Suppose we find a coin and wonder whether it’s fair. As a first test, we decide to flip the coin 200 times and count the number of heads, \(S\text{.}\) What values of \(S\) would be extreme enough to reject the null hypothesis of a fair coin? If the coin actually has a 0.6 probability of coming up heads, what is the power of this test?

2.

Suppose we have a coin which we suspect comes up heads more often than a fair coin would. As a first test, we decide to flip the coin 200 times and count the number of heads, \(S\text{.}\) What values of \(S\) would be extreme enough to reject the null hypothesis of a fair coin? If the coin actually has a 0.6 probability of coming up heads, what is the power of this test?

3.

Suppose we find a six-sided die and wonder whether it’s fair. As a first test, we decide to roll the die 100 times and count the number of times it comes up 1. The expected number of 1’s is 50/3, with a variance of 125/9.

(a)

Using a normal approximation, what is the smallest number of 1’s greater than 50/3 that would be extreme enough to reject the null hypothesis of a fair die?

(b)

Using a normal approximation, what is the greatest number of 1’s less than 50/3 that would be extreme enough to reject the null hypothesis of a fair die?

(c)

Suppose that this die is weighted so that it rolls a 1 with probability 0.2. What would be the power of our test?

(d)

Suppose we roll the die 100 times and see 23 1’s. Use the maximum likelihood value for the probability of rolling a 1 to calculate the power of the test.

4.

Suppose a particular plant when grown outdoors has an average height of 39 in with a variance of 20 in\(^2\text{.}\) We suspect that growing this plant in a greenhouse will increase its height. We take the average height of a sample of 50 plants grown in a greenhouse. What is the minimum average height of this sample that would be extreme enough to reject the null hypothesis of equal means at the \(p = 0.05\) significance level? If the plants, when grown in a greenhouse, would truly have an average height of 41 in, what is the power of our test?