Definition 4.1.1.
An estimator is a value of a parameter computed from a sample of data.
Section 4.1 Likelihood
So far, we’ve been concerned with probability theory. Starting with a probability distribution and some parameter values, we’ve tried to answer questions like: What’s the probability of seeing certain experimental results? Statistics is concerned with going in the other direction: Upon seeing the experimental results, can we determine the type of underlying probability distribution? Can we determine its parameters?
Example 4.1.2.
Suppose we find a coin on the street and don’t know whether or not it’s fair. We want to know the probability \(p\) of the coin coming up heads. We might, for example, flip the coin \(n\) times and count the number \(k\) of heads. Then, we’ll estimate \(p = \frac{k}{n}\text{.}\) We’ll refer to this as a common sense estimator. (Other distributions and parameter types will have different notions of "common sense".)
An estimator is, itself, a random variable: it produces a numerical value based on the results of an experiment. We’ll use notation like \(\est{p}\) for a random variable which is an estimator for a parameter \(p\text{.}\) (Similarly, \(\est{\lambda}\) would denote an estimator for a parameter called \(\lambda\text{.}\))
Definition 4.1.3.
An estimator \(\est{p}\) is called unbiased if \(\E(\est{p}) = p\text{.}\)
Example 4.1.4.
Suppose we have a coin with parameter \(p\text{,}\) which we’ll flip \(n\) times and count the number \(k\) of heads. We use the unbiased estimator \(\est{p} = \frac{k}{n}\text{.}\) In this case, notice that \(k \sim \Bin(n, p)\text{,}\) so we know \(\E(k) = np\text{,}\) although we don’t know the value of \(p\text{.}\) (We probably do know the value of \(n\text{;}\) after all, we’re flipping the coin!) Now:
\begin{gather*}
\E(\est{p}) = \E\left(\frac{k}{n}\right) = \frac{1}{n} \cdot \E(k) = \frac{1}{n} \cdot np = p.
\end{gather*}
It’s worth pausing for a moment to be appropriately impressed with ourselves. We still don’t know the true value of \(p\text{.}\) But we managed to show that our common sense method of estimating \(p\) gives, on average, the correct value.
We’d like to be able to collect some data and use that data to estimate the values of whatever parameters our distribution has. Perhaps as a starting point, it would be good to identify the single most likely value of a parameter:
Definition 4.1.5.
Suppose we collect some data to estimate a parameter \(p\text{.}\) The likelihood function is:
\begin{gather*}
\L(p) = \Pr(\text{data} \mid \text{parameter value is } p).
\end{gather*}
The value of \(p\) which maximizes the function \(\L(p)\) is called the maximum likelihood estimation, or MLE, of \(p\text{.}\)
Example 4.1.6.
Suppose we flip a coin 100 times. Let \(N\) be the number of heads. Then \(N\sim \Bin(100, p)\) for some unknown value of \(p\text{:}\)
\begin{align*}
\Pr(N = k) \amp = b(k; 100, p) = {100 \choose k} p^k (1 - p)^{100 - k}
\end{align*}
In this case, \(k\) is the data that we collect, and \(p\) is the value of the parameter. Suppose we see \(52\) heads. Then:
\begin{gather*}
\L(p) = {100 \choose 52} p^{52} (1 - p)^{48}.
\end{gather*}
If we want to know the most likely value of the parameter \(p\text{,}\) then we should maximize \(\L(p)\) over the interval \(0 \leq p \leq 1\text{.}\)
\begin{align*}
\L'(p) \amp = {100 \choose 52} \left[ 52 p^{51}(1 - p)^{48} + p^{52} 48 (1 - p)^{47}(-1)\right] \\
\amp = {100 \choose 52} p^{51} (1 - p)^{47} \left[ 52 (1 - p) - 48p \right] \\
\amp = {100 \choose 52} p^{51} (1 - p)^{47} \left[ 52 - 100 p \right].
\end{align*}
We can see that \(\L'(p) = 0\) when \(p = 0, 1, \frac{52}{100}\text{,}\) and the endpoints of the interval we’re maximizing over are \(p = 0, 1\text{,}\) so we can build a table:
| \(p\) | \(\L(p)\) |
| \(0\) | \(0\) |
| \(52/100\) | \(\gt 0\) |
| \(1\) | \(0\) |
Note that we don’t need to know the exact value of \(\L(52/100)\) in order to see that it’s strictly positive, and therefore the maximum value of \(\L(p)\text{.}\) So, the MLE of \(p\) is \(52/100\text{.}\)
We can see in the previous example that the MLE of \(p\) is also the common sense estimation of \(p\text{.}\) This will be the case in general for the binomial distribution, so we won’t need to redo this work over and over:
Fact 4.1.8. Binomial MLE.
If we see \(k\) heads in \(n\) coin flips, then the MLE of the bias \(p\) is \(\frac{k}{n}\text{.}\)
As we remember from Calculus 1, maximizing a continuous function works slightly differently over a closed interval (like the previous example) or an open interval (like the next example).
Example 4.1.9.
Suppose a radioactive material emits particles as it decays. Let \(N\) count the particles emitted. Then \(N \sim \Poiss(\lambda)\) for some unknown rate parameter \(\lambda\text{:}\)
\begin{gather*}
\Pr(N = k) = p(k; \lambda) = \frac{\lambda^k}{k!} e^{-\lambda}
\end{gather*}
Suppose we observe a sample of material for 1 hour and count 8 particles emitted. Then:
\begin{gather*}
\L(\lambda) = \frac{\lambda^8}{8!} e^{-\lambda}
\end{gather*}
To maximize \(\L(\lambda)\) over the interval \(0 \lt \lambda \lt \infty\text{,}\) we start by finding critical points.
\begin{align*}
\L'(\lambda) \amp = \frac{1}{8!} \left[ 8 \lambda^7 e^{-\lambda} + \lambda^8 e^{-\lambda} (-1)\right] \\
\amp = \frac{1}{8!} \lambda^7 e^{-\lambda} \left[ 8 - e^{-\lambda} \right]
\end{align*}
The only critical point is \(\lambda = 8\text{,}\) but we have not justified that this is the location of a global maximum. A critical point is only a potential location of a local minimum or maximum. But with a bit more justification: observe that \(\L'\) is positive on the interval \((0, 8)\) and negative on the interval \((8, \infty)\text{.}\) Therefore the function \(\L\) increases on \((0, 8)\) and decreases on \((8, \infty)\text{.}\) So it must reach its maximum at \(\lambda = 8\text{,}\) which is therefore the MLE.
As with the binomial distribution, the calculation will be essentially the same regardless of the specific number of particles observed, so:
Fact 4.1.10. Poisson MLE.
If we observe a Poisson process and see \(k\) events occur, then the MLE of the rate parameter \(\lambda\) is \(k\text{.}\)
The situation for continuous random variables is similar, but slightly different. To build the likelihood function, we should use the pdf of the continuous random variable. So the likelihood function will give the probabiliy density given the data collected, rather than the probability itself.
Example 4.1.11.
Suppose we observe a cell and measure the time \(T\) until a toxin molecule leaves the cell. Then \(T \sim \Exp(\lambda)\) for some unknown rate parameter \(\lambda\text{:}\)
\begin{gather*}
f(t) = \lambda e^{-\lambda t}.
\end{gather*}
Suppose we observe a molecule leave the cell at \(t = 0.3\) minutes. Then:
\begin{align*}
L(\lambda) \amp = \lambda e^{-0.3\lambda} \\
L'(\lambda) \amp = e^{-0.3\lambda} + \lambda e^{-0.3t} (-0.3) \\
\amp = e^{-0.3\lambda}\left[ 1 - 0.3\lambda\right]
\end{align*}
The critical point is \(\lambda = \frac{1}{0.3} \approx 3.33\text{.}\) Since \(\L' \gt 0\) on \((0, 3.33)\) and \(\L' \lt 0\) on \((3.33, \infty)\text{,}\) there is a global maximum at \(\lambda \approx 3.33\text{,}\) which is therefore the MLE.
The specific time \(0.3\) minutes doesn’t particularly matter in this calculation. Whatever the time \(t\text{,}\) essentially the same calculation will result in a MLE of \(\lambda = 1/t\text{.}\) But what if we collect multiple pieces of data?
| Molecule | Waiting Time | Rate Estimation |
| 1 | \(0.3\) | \(3.33\) |
| 2 | \(0.8\) | \(1.25\) |
| 3 | \(0.5\) | \(2.00\) |
| 4 | \(0.6\) | \(1.67\) |
| 5 | \(0.9\) | \(1.11\) |
How should take all of this data into account in our maximum likelihood estimation? We might consider taking the average of all of the separate rate estimations, which would give \(1.87\text{.}\) Is this the most likely? We need some mathematical justification.
To account for multiple, independent data points, we should multiply the probability densities for each in the creation of our likelihood function:
\begin{align*}
\L(\lambda) \amp = \left(\lambda e^{-0.3\lambda}\right)\left(\lambda e^{-0.8\lambda}\right)\left(\lambda e^{-0.5\lambda}\right)\left(\lambda e^{-0.6\lambda}\right)\left(\lambda e^{-0.9\lambda}\right) \\
\amp = \lambda^5 e^{-0.3\lambda - 0.8\lambda - 0.5\lambda - 0.6\lambda - 0.9\lambda } \\
\amp = \lambda^5 e^{-3.1\lambda}
\end{align*}
Now we can find the maximum:
\begin{align*}
\L'(\lambda) \amp = 5 \lambda^4 e^{-3.1\lambda} + \lambda^5 e^{-3.1\lambda} (-3.1) \\
\amp = 5 \lambda^4 e^{-3.1\lambda} + \lambda^5 e^{-3.1\lambda} (-3.1) \\
\amp = \lambda^4 e^{-3.1\lambda} \left[5 - 3.1\lambda \right]
\end{align*}
The only critical point is \(5/3.1 \approx 1.61\text{.}\) Since \(\L' \gt 0\) on \((0, 1.61)\) and \(\L' \lt 0\) on \((1.61, \infty)\text{,}\) there is a global maximum at \(\lambda = 1.61\text{,}\) which is therefore the MLE.
It may seem less clear how to generalize this calculation for other tables of data. Observe that the value \(3.1\) is the sum of the five times in the table, so \(3.1/5\) is the average time. The MLE turned out to be the reciprocal of the average time (just as the MLE with only one data point was the reciprocal of that one time). Notice that this does not match the guess we made previously of averaging the individual rate estimations for each data point.
Fact 4.1.13. Exponential MLE.
If we observe a Poisson process and see events occur after waiting times \(t_1, t_2, \dotsc, t_n\text{,}\) then the MLE of the rate parameter \(\lambda\) is \(\frac{n}{t_1 + t_2 + \dotsb + t_n}\text{.}\)
Exercises Exercises
1.
Suppose a coin has an unknown probability of coming up heads. We perform the experiment in five independent trials, during which it takes 4, 5, 4, 3, and 6 flips to see our first heads in each trial. What is the maximum likelihood estimation for the probability of the coin coming up heads on a flip?
2.
Suppose a coin has an unknown probability of coming up heads. We perform the experiment in \(n\) independent trials, during which it takes \(k_1, k_2, \dotsc, k_n\) flips to see our first heads in each trial. Find a "common sense" MLE formula for the geometric distribution.
Hint.
You could set up a calculation analogous to Example 4.1.11. Or, you could consider Fact 4.1.8.
3.
A particular store owner wants to approximate the average hourly rate at which customers come into the store. They observe 80 customers enter during a particular 4-hour shift. What is the maximum likelihood estimation for the hourly customer rate?
4.
A radioactive material emits particles at an unknown probabilistic rate \(\lambda\) particles per minute. We observe particles emitted at times 1.1, 1.7, 1.3, 2.2, 1.9, and 1.8 minutes. Write the likelihood function \(\mathcal{L}(\lambda)\) based on this data. What is the maximum likelihood estimation for \(\lambda\text{?}\)
5.
Suppose a parameter \(\theta\) takes values in \([0, 1]\) with likelihood function \(\mathcal{L}(\theta) = \sqrt{\theta} - \theta^2\text{.}\) Find the maximum likelihood estimation of \(\theta\text{.}\)
