Likelihood
So far, we've been concerned with probability theory.
Starting with a probability distribution and some parameter values, we've tried to answer questions like: What's the probability of seeing certain experimental results? Statistics is concerned with going in the other direction: Upon seeing the experimental results, can we determine the type of underlying probability distribution? Can we determine its parameters?
An estimator is a value of a parameter computed from a sample of data.
Suppose we find a coin on the street and don't know whether or not it's fair.
We want to know the probability p of the coin coming up heads.
We might, for example, flip the coin n times and count the number k of heads.
Then, we'll estimate p = \frac{k}{n}.
We'll refer to this as a common sense estimator.
(Other distributions and parameter types will have different notions of "common sense".)
An estimator is, itself, a random variable: it produces a numerical value based on the results of an experiment.
We'll use notation like \est{p} for a random variable which is an estimator for a parameter p.
(Similarly, \est{\lambda} would denote an estimator for a parameter called \lambda.)
An estimator \est{p} is called unbiased if \E(\est{p}) = p.
Suppose we have a coin with parameter p, which we'll flip n times and count the number k of heads.
We use the unbiased estimator \est{p} = \frac{k}{n}.
In this case, notice that k \sim \Bin(n, p), so we know \E(k) = np, although we don't know the value of p.
(We probably do know the value of n; after all, we're flipping the coin!) Now:
\E(\est{p}) = \E\left(\frac{k}{n}\right) = \frac{1}{n} \cdot \E(k) = \frac{1}{n} \cdot np = p.
It's worth pausing for a moment to be appropriately impressed with ourselves.
We still don't know the true value of p.
But we managed to show that our common sense method of estimating p gives, on average, the correct value.