Definition 101.
If \(X, Y\) are random variables with expected values of \(\mu_X, \mu_Y\text{,}\) then the covariance of \(X\) and \(Y\) is:
\begin{align*}
\Cov(X, Y) \amp = \E\left[ (X - \mu_X)(Y - \mu_Y)\right].
\end{align*}
Section Thursday, Feb 5
This is an outline of the topics we covered in class. These notes are not a substitute for your own note-taking. I highly recommend that you take your own notes during class. If you ever miss a class for any reason, reach out to another student in class to get a copy of their notes.
Subsection Summary
Here are the expected value and variance formulas for common distributions. Some of these, we’ve shown justification for. Others requires techniques beyond the scope of the class to justify.
| Distribution | Parameters | Expected Value | Variance |
| Indicator | \(p\) | \(p\) | \(p(1-p)\) |
| Binomial | \(n, p\) | \(np\) | \(np(1-p)\) |
| Geometric | \(p\) | \(\frac{1}{p}\) | \(\frac{1-p}{p^2}\) |
| Poisson | \(\lambda\) | \(\lambda\) | \(\lambda\) |
| Exponential | \(\lambda\) | \(\frac{1}{\lambda}\) | \(\frac{1}{\lambda^2}\) |
Subsection Covariance
Observe that:
\begin{align*}
\Cov(X, X) \amp = \E\left[(X - \mu_X)(X - \mu_X)\right] \\
\amp = \E\left[(X - \mu_X)^2\right] \\
\amp = \Var(X),
\end{align*}
so covariance generalizes the variance formula to two variables. As with variance, there’s an alternative formula more suited to doing computations:
\begin{align*}
\Var(X) \amp = \E\left[(X - \mu_X)(X - \mu_X)\right] = \E(X^2) - \mu_X^2 \\
\Cov(X, Y) \amp = \E\left[(X - \mu_X)(Y - \mu_Y)\right] = \E(XY) - \mu_X\mu_Y
\end{align*}
Example 102.
Consider the joint distribution table:
| \(X = 0\) | \(X = 1\) | |
| \(Y = 0\) | 0.2 | 0.1 |
| \(Y = 1\) | 0.05 | 0.65 |
From the table, we can calculate the marginal distributions:
\begin{align*}
\Pr(X = 0) \amp = 0.25 \amp \Pr(Y = 0) \amp = 0.3 \\
\Pr(X = 1) \amp = 0.75 \amp \Pr(Y = 1) \amp = 0.7
\end{align*}
So \(\E(X) = \mu_X = 0.75\) and \(\E(Y) = \mu_Y = 0.7\text{.}\) Then:
\begin{align*}
\E(XY) \amp = (0)(0)(0.2) + (1)(0)(0.1) + (0)(1)(0.05) + (1)(1)(0.65) \\
\amp = 0.65 \\
\Cov(X, Y) \amp = \E(XY) - \mu_X \mu_Y \\
\amp = 0.65 - (0.75)(0.7) \\
\amp = 0.125.
\end{align*}
Question: the formula \(\Var(X) = \E\left[(X - \mu_X)^2\right]\) makes it clear that variance cannot be negative, since squares are nonnegative. What about \(\Cov(X, Y)\text{?}\)
Example 104.
In the previous example, since \(X, Y\) were both indicator random variables, the variances for each were simply equal to the sum of the second row/column. Similarly, \(\E(XY)\) was equal to the \(X = 1, Y = 1\) entry in the table. Using two indicator random variables, significantly simplifies the covariance calculation, so we can vary the table and recalculate covariance quickly. Consider the following joint distribution:
| \(X = 0\) | \(X = 1\) | |
| \(Y = 0\) | 0.1 | 0.4 |
| \(Y = 1\) | 0.3 | 0.2 |
Question: how do we interpret \(\Cov(X, Y)\text{?}\) \(X - \mu_X\) is positive when \(X \gt \mu_X\) and negative when \(X \lt \mu_X\text{.}\) \(Y - \mu_Y\) is positive when \(Y \gt \mu_Y\) and negative when \(Y \lt \mu_Y\text{.}\) So the product \((X - \mu_X)(Y - \mu_Y)\) is positive when \(X, Y\) are both larger or both smaller than their expected values, and negative when one is larger and one is smaller. That is, covariance tries to quantify the tendency of \(X, Y\) to get big/small at the same time.
