Expected Value
Discrete Expected Value
A full probability distribution (for a discrete random variable) or a density function (for a continuous random variable) carry all of the probability information.
Often, we seek statistics, i.e., numbers that summarize a distribution in some way.
Let X be a discrete random variable taking the values x_1, x_2, \dotsc, x_n.
The expected value (also called mean, or expectation) of X is the weighted average of the values of X, where the weights are the probabilities of X taking each value:
\E(X) = \sum_{i=1}^n x_i \Pr(X = x_i).
If X takes on the values x_1, x_2, \dotsc, x_n uniformly (i.e., each value having probability \frac{1}{n}), then the expected value is the usual average:
\E(X) \amp = x_1 \cdot \frac{1}{n} + x_2\cdot \frac{1}{n} + \dotsb + x_n \cdot \frac{1}{n}
\amp = \frac{x_1 + x_2 + \dotsb + x_n}{n}
For example, the expected value of a fair 6-sided die roll R is:
\E(R) \amp = 1 \left(\frac{1}{6}\right) + 2 \left(\frac{1}{6}\right) + 3 \left(\frac{1}{6}\right) + 4 \left(\frac{1}{6}\right) + 5 \left(\frac{1}{6}\right) + 6 \left(\frac{1}{6}\right)
\amp = \frac{1 + 2 + \dotsb + 6}{6} = \frac{21}{6} = \frac{7}{2}.
Suppose we roll a fair 6-sided die two times, and let S be the sum of the rolls.
It's straightforward to check the distribution for S shown below.
We've also included a column containing the products k \Pr(S = k), which must be summed up to find \E(S).
| k |
\Pr(S = k) |
k \cdot \Pr(S = k) |
| 2 |
1/36 |
2/36 |
| 3 |
2/36 |
6/36 |
| 4 |
3/36 |
12/36 |
| 5 |
4/36 |
20/36 |
| 6 |
5/36 |
30/36 |
| 7 |
6/36 |
42/36 |
| 8 |
5/36 |
40/36 |
| 9 |
4/36 |
36/36 |
| 10 |
3/36 |
30/36 |
| 11 |
2/36 |
22/36 |
| 12 |
1/36 |
12/36 |
The sum of the third column gives \E(S) = \frac{252}{36} = 7.
Let X indicate an event A which has probability p.
Then:
\E(X) = 0 \cdot (1-p) + 1 \cdot p = p.
That is, the expected value of an indicator random variable is the probability of the event that it indicates.
Having found the expected value for one named distribution, we may be tempted to run through our other discrete distributions (binomial, geometric, and Poisson) and find nice formulas for their expected values.
Unfortunately, both the geometric distribution and the Poisson distribution have infinitely many possible values, and the summation of infinitely many discrete terms is beyond the scope of this course.
(A Calculus 2 course covering series would give the right tools to find expected value formulas with full justification for these distributions.)
The binomial distribution takes only finitely many values, and we have that lets us generate a distribution table reasonably quickly, given the parameters n and p.
Feel free to pick some parameter values (keep n fairly small!) to try this calculation by hand, but we'll use a powerful theorem a bit later to give a very nice general formula depending on n and p.
Continuous Expected Value
Expected value for a discrete random variable is a weighted average of the values taken by the variable with weights provided by the probability distribution.
We'd like to use the same idea for a continuous random variable, but now we're averaging over an interval's worth of values.
Also, individual values don't have probabilities, they have probability densities.
Let X be a continuous random variable taking values in the interval [a, b] with pdf f(x).
The expected value of X is given by:
\E(X) = \int_a^b x f(x)\ dx.
It's worth viewing this definition side-by-side with to understand that they have analogous structure.
The pdf values f(x) are densities, not probabilities.
But the product f(x)\ dx is probability.
So the expression x f(x)\ dx is analogous to x_i \Pr(X = x_i): value times probability.
Finally, the integral symbol \int is meant to look like a stretched out letter "S" because it is a form of infinite summation.
So, both \sum x_i \Pr(X = x_i) and \int_a^b x f(x)\ dx should be understood as taking the products of values with their probabilities and then summing up those products.
Let X be uniform on the interval [a, b].
So X has the pdf f(x) = \frac{1}{b - a}.
We might reasonably expect that the average value of X would be the midpoint of the interval.
Let's check that:
\E(X) \amp = \int_a^b x f(x)\ dx
\amp = \int_a^b \frac{x}{b - a} \ dx
\amp = \frac{1}{b-a} \left(\frac{x^2}{2}\right)\bigg|_a^b
\amp = \frac{1}{b-a} \left( \frac{b^2}{2} - \frac{a^2}{2} \right)
\amp = \frac{1}{b - a}\left( \frac{(b - a)(b + a)}{2}\right)
\amp = \frac{a + b}{2}.
Let X take values in [0, 4] with pdf f(x) = \frac{3\sqrt{x}}{16}.
\E(X) \amp = \int_0^4 x f(x)\ dx
\amp = \int_0^4 \frac{3x^{3/2}}{16}\ dx
\amp = \frac{3}{16}\left(\frac{x^{5/2}}{5/2}\right)\bigg|_0^4
\amp = \frac{3}{16}\left(\frac{2x^{5/2}}{5}\right)\bigg|_0^4
\amp = \frac{3}{40}\left(x^{5/2}\right)\bigg|_0^4
\amp = \frac{3}{40}(32 - 0)
\amp = \frac{12}{5}
\amp = 2.4.
Linearity of Expected Value
In the context of many areas of mathematics, the word "linearity" doesn't refer to some graph being a straight line.
Rather, it refers to a situation in which some mathematical object or operation behaves in the nicest possible way under the operations of addition and scalar multiplication (i.e., multiplication by a constant).
You've already encountered linearity several times in Calculus 1, whether or not it was phrased that way.
-
Limits are linear:
\lim f(x) + g(x) \amp = \lim f(x) + \lim g(x)
\lim c f(x) \amp = c \lim f(x)
-
Derivatives are linear:
\frac{d}{dx}(f(x) + g(x)) \amp = \frac{d}{dx}(f(x)) + \frac{d}{dx}(g(x))
\frac{d}{dx}(c f(x)) \amp = c \frac{d}{dx}(f(x))
-
Integrals (both definite and indefinite) are linear:
\int f(x) + g(x) \amp = \int f(x) + \int g(x)
\int c f(x) \amp = c \int f(x)
Now, we see that expected value is also linear:
Linearity of Expectation
Let X and Y be random variables and \alpha \in \R.
Then:
\E(X + Y) \amp = \E(X) + \E(Y)
\E(\alpha X) \amp = \alpha \E(X)
Some application of the linearity of expectation will seem entirely reasonable:
In , we saw that the expected value of a fair 6-sided die roll is 3.5.
In , we saw that the expected value of the sum of two die rolls is 7, which is precisely 2 times 3.5.
The calculation in would be very tedious to recreate for three die rolls, let alone generalizing for n rolls.
However, suppose we define separate random variables R_1, R_2, \dotsc, R_n for the result of each individual roll.
Let S be the sum of the n rolls.
Then S = R_1 + R_2 + \dotsb + R_n. By linearity:
\E(S) \amp = \E(R_1 + R_2 + \dotsb + R_n)
\amp = \E(R_1) + \E(R_2) + \dotsb + \E(R_n)
\amp = \underbrace{3.5 + 3.5 + \dotsb + 3.5}_{n \text{ times}}
\amp = 3.5n.
For two rolls, we recover our previous result: 3.5(2) = 7. But now we have a general formula that tells the expected sum for any number of rolls.
It also gives a perfectly reasonable answer: the expected sum of n rolls is n times the expected value of a single roll.
Suppose S \sim \Bin(n, p).
describes a binomially distributed random variable as counting the number of occurrences of some event, which either happens or not in each of n independent trials.
This leads us to a very natural idea: identify each individual possible occurrence of the event, and assign it an indicator random variable.
In this case, let H_i indicate that flip i comes up heads.
Then
S = H_1 + H_2 + \dotsb + H_n.
We already know () that the expected value of each H_i is the probability of the indicated eventthat flip i is heads.
This is precisely the parameter p.
Therefore:
\E(S) \amp = \E(H_1 + H_2 + \dotsb + H_n)
\amp = \E(H_1) + \E(H_2) + \dotsb + \E(H_n)
\amp = \underbrace{p + p + \dotsb + p}_{n \text{ times}}
\amp = np.
Remember this formula! We'll make frequent use of it.
The previous examples are so reasonable it may seem unimpressive.
But consider the following use of linearity:
We say a flip sequence has a run of 4 heads starting at k if flips k, k+1, k+2, k+3 are all heads.
We don't care about overlaps: the flip sequence HHHHH has a run of 4 heads starting at 1 and another run starting at 2.
Suppose we flip a fair coin 100 times.
How many runs of 4 heads should we expect?
Can you mimic the structure of ?
Let R_k indicate a run of 4 heads starting at k.
Since this requires 4 precise flips to come up heads, the probability of the indicated event should be
\left(\frac{1}{2}\right)^4 = \frac{1}{16}
The very last starting flip for a run of 4 heads is flip number 97 (so that flips 97, 98, 99, and 100 would all be heads).
Then S = R_1 + R_2 + \dotsb + R_{97} precisely counts the number of runs of 4 heads.
Using linearity:
\E(S) \amp = \E(R_1 + R_2 + \dotsb + R_{97})
\amp = \E(R_1) + \E(R_2) + \dotsb + \E(R_{97})
\amp = \underbrace{\frac{1}{16} + \frac{1}{16} + \dotsb + \frac{1}{16}}_{97 \text{ times}}
\amp = \frac{97}{16}
\amp \approx 6.
Notice that, unlike , the events being indicated here are not independent from each other.
Linearity is unaffected by the fact that that the runs of heads can overlap.
Consider a random variable X with probability distribution below.
Find \E(X).
| x |
\Pr(X = x) |
| 1 |
0.1 |
| 2 |
0.05 |
| 3 |
0.2 |
| 4 |
0.15 |
| 5 |
0.15 |
| 6 |
0.1 |
| 7 |
0.05 |
| 8 |
0.1 |
| 9 |
0.1 |
Let X and Y be random variables each taking the values 1, 2, 3, 4, 5.
Write different distribution tables for X and Y so that they have the same expected value.
Suppose we flip a coin n = 100 times, and let N count the number of heads.
If the coin comes up heads on a flip with probability p = 0.4, what is \E(N)? What if n = 80 and p = 0.6? What if n = 200 and p = 0.5?
If \E(X) = 3, \E(Y) = -2, and \E(Z) = 1, what is \E(4X + 5Y - Z + 3)?
A continuous random variable X taking values in [0, 1] has p.d.f.
f(x) = 2x.
What is \E(X)?
A continuous random variable X taking values in [-1, 1] has p.d.f.
f(x) = \frac{3x^2}{2}.
What is \E(X)?
A continuous random variable X taking values in [1, 4] has p.d.f.
f(x) = \frac{4}{3x^2}.
What is \E(X)?
A continuous random variable X taking values in [1, 4] has p.d.f.
f(x) = k(x - \sqrt{x}) for some constant k.
In a previous problem, you found the value of k.
Now, find \E(X).
A continuous random variable X taking values in [1, 2] has p.d.f.
\displaystyle{f(x) = \frac{1}{2}\left(\frac{1}{x^2} + x\right)}.
Find \E(X).
Variance
The expected value is one of several statistics sometimes referred to as a "measure of central tendency".
These are statistics that try to capture the "typical" value of the random variable.
Our next statistic tries to measure how spread out a distribution is.
Let X be a random variable.
The variance of X is:
\Var(X) = \E\left[ \left(X - \E(X)\right)^2 \right].
This formula tries to capture some measure of how spread out the distribution is.
One way we might hope to measure spread is to calculate how far away X is from its expected value on average.
Squaring the term X - \E(X) ensures that any difference between the value of the random variable and its expected value contributes positively.
("Distance away from" should be positive.) So, the formula in is conceptually clear.
In practice, there's another way to structure the formula that makes for cleaner calculation:
Let X be a random variable.
Then:
\Var(X) = \E\left(X^2\right) - \left(\E(X)\right)^2.
\Var(X) \amp = \E\left[ \left(X - \E(X)\right)^2 \right]
\amp = \E\left[ X^2 - 2 \E(X) X + \left(\E(X)\right)^2\right]
\amp = \E\left(X^2\right) - 2 \E(X) \E(X) + \left(\E(X)\right)^2
\amp = \E\left(X^2\right) - \left(\E(X)\right)^2.
Let R be the value of a fair 6-sided die roll.
We've seen in that \E(R) = \frac{7}{2}.
To find \E\left(R^2\right):
\E\left(R^2\right) \amp = 1^2 \left(\frac{1}{6}\right) + 2^2 \left(\frac{1}{6}\right) + 3^2 \left(\frac{1}{6}\right) + 4^2 \left(\frac{1}{6}\right) + 5^2 \left(\frac{1}{6}\right) + 6^2 \left(\frac{1}{6}\right)
\amp = 1 \left(\frac{1}{6}\right) + 4 \left(\frac{1}{6}\right) + 9 \left(\frac{1}{6}\right) + 16 \left(\frac{1}{6}\right) + 25 \left(\frac{1}{6}\right) + 36 \left(\frac{1}{6}\right)
\amp = \frac{91}{6}.
Then
\Var(R) = \E\left(R^2\right) - \left(\E(R)\right)^2 = \frac{91}{6} - \frac{49}{4} = \frac{35}{12}.
Notice that in the \E\left(R^2\right) calculation, we're just taking the basic structure of the \E(R) calculation and squaring the R values.
The analogous calculation for a continuous random variable looks similar:
Let X be uniform on the interval [a, b].
So X has the pdf f(x) = \frac{1}{b - a}, and we saw in that \E(X) = \frac{a + b}{2}.
Now:
\E\left(X^2\right) \amp = \int_a^b x^2 f(x)\ dx
\amp = \int_a^b \frac{x^2}{b - a} \ dx
\amp = \frac{1}{b-a} \left(\frac{x^3}{3}\right)\bigg|_a^b
\amp = \frac{1}{b-a} \left( \frac{b^3}{3} - \frac{a^3}{3} \right)
\amp = \frac{1}{b - a}\left( \frac{(b - a)(a^2 + ab + b^2)}{3}\right)
\amp = \frac{a^2 + ab + b^2}{3}.
Then:
\Var(X) \amp = \E\left(X^2\right) - \left(\E(X)\right)^2
\amp = \frac{a^2 + ab + b^2}{3} - \frac{(a+b)^2}{4}
\amp = \frac{a^2 + ab + b^2}{3} - \frac{a^2 + 2ab + b^2}{4}
\amp = \frac{4a^2 + 4ab + 4b^2}{12} - \frac{3a^2 + 6ab + 3b^2}{12}
\amp = \frac{a^2 - 2ab + b^2}{12}
\amp = \frac{(a - b)^2}{12}.
Variance is not linear.
In general, you can't split up variance across plus or minus signs, and you can't pull multiplied constants out of the variance like you can with expected value.
However, there are some properties we can use to do similar manipulations to the variance.
Let X and Y be random variables, and let \alpha \in \R.
Then:
-
\Var(X + \alpha) = \Var(X).
-
\Var(\alpha X) = \alpha^2 \Var(X).
-
If X and Y are independent, then \Var(X + Y) = \Var(X) + \Var(Y).
We'll omit the proofs and just casually observe that these are reasonable properties.
X + \alpha is a shift of X, and shifting shouldn't change how spread out the distribution is.
\alpha X scaled X by a constant, so it should scale the spread of the distribution by the same constant.
But recall that the variance calculation involves squaring; it's an indirect measure of spread.
So it's reasonable that scaling the distribution by a constant should scale the variance by its square.
Let's put these properties to use in a pair of related examples:
Let X indicate an event A which has probability p.
In , we saw \E(X) = p.
Notice that X takes the values 0 and 1 (with probabilities 1 - p and p, respectively).
But 0^2 = 0, and 1^1 = 1, so the expected value calculation for X^2 is precisely the same as for X!
\E\left(X^2\right) \amp = 0^2 \cdot (1-p) + 1^2 \cdot p = p.
So:
\Var(X) \amp \E\left(X^2\right) - \left(\E(X)\right)^2 = p - p^2 = p(1 - p).
Recall the setup from : a binomial random variable S is the sum H_1 + H_2 + \dotsb + H_n of n indicator random variables, each with parameter p.
The results of different coin flips are independent from each other, so the random variables H_1, \dotsc, H_n are independent from each other.
From , we know that \Var(H_i) = p(1-p) for each i.
Therefore:
\Var(S) \amp = \Var(H_1 + H_2 + \dotsb + H_n)
\amp = \Var(H_1) + \Var(H_2) + \dotsb + \Var(H_n)
\amp = \underbrace{p(1-p) + p(1-p) + \dotsb + p(1-p)}_{n \text{ times}}
\amp = np(1-p).
Remember this formula! We'll make frequent use of it.
Consider a random variable X with probability distribution below.
Find \Var(X).
| x |
\Pr(X = x) |
| 1 |
0.1 |
| 2 |
0.05 |
| 3 |
0.2 |
| 4 |
0.15 |
| 5 |
0.15 |
| 6 |
0.1 |
| 7 |
0.05 |
| 8 |
0.1 |
| 9 |
0.1 |
Let X be a random variable taking the values 1, 2, 3, 4, 5.
Write a distribution table for X, then use your table to write a distribution for X^2.
Then, find \Var(X).
Suppose we flip a coin n = 100 times, and let N count the number of heads.
If the coin comes up heads on a flip with probability p = 0.4, what is \Var(N)? What if n = 80 and p = 0.6? What if n = 200 and p = 0.5?
If \E(X) = 3, \Var(X) = 2, what is \E(X^2)?
A continuous random variable X taking values in [0, 1] has p.d.f.
f(x) = 2x.
What is \Var(X)?
A continuous random variable X taking values in [-1, 1] has p.d.f.
f(x) = \frac{3x^2}{2}.
What is \Var(X)?
A continuous random variable X taking values in [1, 4] has p.d.f.
f(x) = \frac{4}{3x^2}.
What is \Var(X)?
A continuous random variable X taking values in [1, 4] has p.d.f.
f(x) = k(x - \sqrt{x}) for some constant k.
In a previous problem, you found the value of k.
Now, find \Var(X).
A continuous random variable X taking values in [1, 2] has p.d.f.
\displaystyle{f(x) = \frac{1}{2}\left(\frac{1}{x^2} + x\right)}.
Find \Var(X).
Covariance
Let X and Y be random variables.
The covariance of X and Y is:
\Cov(X, Y) = \E\left[\left(X - \E(X)\right) \left(Y - \E(Y)\right)\right].
Let's write \mu_X = \E(X) and \mu_Y = \E(Y) to make this is a bit cleaner to look at:
\Cov(X, Y) = \E\left[\left(X - \mu_X \right) \left(Y - \mu_Y\right)\right].
This looks like a pretty natural generalization of the variance.
In fact:
\Cov(X, X) \amp = \E\left[ (X - \mu_X)(X - \mu_X)\right] = \E\left[\left(X - \mu_X\right)^2\right] = \Var(X).
One difference worth noting is that \Var(X), as the average of squared values, must always be nonnegative.
The expression (X - \mu_X)(X - \mu_X) can be negative for combinations of X and Y values where one is greater than its average and the other is smaller.
If this is the average behavior, then \Cov(X, Y) can be negative overall.
This is useful information: a negative \Cov(X, Y) would imply that (on average, not necessarily always) when X is larger than typical, Y is smaller than typical, and vice versa.
Also as with variance, the covariance definition formula gives us some useful insight, but it can be unwieldy to work with.
There is an alternative formula, analogous to :
Let X and Y be random variables.
Then:
\Cov(X, Y) = \E\left(XY\right) - \E(X)\E(Y).
To make use of this formula, it's helpful to have both a joint distribution and marginal distributions.
The marginal distributions let you find \E(X) and \E(Y).
The joint distribution lets you find \E(XY).
Let X and Y be indicator random variables with joint and marginal distributions below.
|
X = 0 |
X = 1 |
| Y = 0 |
0.2 |
0.1 |
| Y = 1 |
0.2 |
0.5 |
Since they're indicator random variables, we can immediately see that \E(X) = 0.6 and \E(Y) = 0.7 (see ).
For \E(XY), we'll traverse cell by cell through the joint distribution table, constructing products of the form X-value times Y-value times probability:
\E(XY) =\, \amp (0)(0)(0.2) + (1)(0)(0.1)
\amp (0)(1)(0.2) + (1)(1)(0.5)
=\, \amp 0.5.
Finally:
\Cov(X, Y) = \E(XY) - \E(X)\E(Y) = 0.5 - (0.6)(0.7) = 0.08.
Calculate the covariance of X and Y:
|
X = 1 |
X = 2 |
| Y = 0 |
0.12 |
0.24 |
| Y = 1 |
0.3 |
34 |
Calculate the covariance of X and Y:
|
X = 0 |
X = 1 |
X = 2 |
| Y = 0 |
0.08 |
0.16 |
0.1 |
| Y = 1 |
0.14 |
0.2 |
0.32 |
Suppose we roll a fair, 4-sided die two times.
Let X be the sum of the rolls, and let Y be the product of the rolls.
Find the covariance of X and Y.
[Note: Somewhat tedious.]