andy/Probability-and-Statistics-Notes
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Separating Ch 2 sections

Browse Source
This commit is contained in:
andy
2026-01-22 13:56:11 -05:00
parent c320513339
commit a4f71931a9
4 changed files with 918 additions and 883 deletions
Download Patch File Download Diff File Expand all files Collapse all files
source/ch-Random-Variables.ptx
+3 -883
View File
@@ -9,887 +9,7 @@
</p>
</introduction>
<section xml:id="sec-Discrete-RVs" xmlns:xi="http://www.w3.org/2001/XInclude">
<title>Discrete Random Variables</title>
<subsection xml:id="subsec-Discrete-RVs">
<title>Discrete Random Variables</title>
<definition xml:id="def-RV">
<statement>
<p>
Let <m>\Omega</m> be a sample space.
A <term>random variable</term> is a function <m>X \colon \Omega \to \R</m> which assigns a number to each outcome.
</p>
</statement>
</definition>
<example>
<statement>
<p>
Suppose an experiment consists of flipping a fair coin 4 times.
Let <m>X</m> be the number of heads.
The actual outcomes in the experiment are heads/tails sequences of length 4, such as <m>HTHT</m> and <m>HHHT</m>.
The random variable <m>X</m> assigns a numerical measurement to the outcomes, such as <m>X(HTHT) = 2</m> and <m>X(HHHT) = 3</m>.
</p>
<p>
For any real number value <m>a</m>, we can consider the event consisting of outcomes such that <m>X = a</m>.
For example, the flip sequences <m>HHHT, HHTH, HTHH</m>, and <m>THHH</m> all have 3 heads, so <m>\Pr(X = 3) = \frac{4}{16} = \frac{1}{4}</m>.
</p>
</statement>
</example>
<example xml:id="example-indicator-distribution">
<title>Indicator Random Variable</title>
<statement>
<p>
Let <m>A</m> be an event in the sample space <m>\Omega</m>.
An <term>indicator random variable</term> for <m>A</m> is the random variable <m>X</m> such that
<md>
<mrow> X(\omega) = \begin{cases} 1 \amp \omega \in A \\ 0 \amp \omega \notin A \end{cases} </mrow>
</md>
</p>
</statement>
</example>
<p>
Indicator random variables are also called <term>Bernoulli random variables</term>, although we'll prefer the former term since it more clearly states the purpose of these random variables: to indicate whether or not a particular event has occurred.
We'll use the language "<m>X</m> indicates <m>A</m>" to mean that <m>X</m> is an indicator random variable for the event <m>A</m>.
In this case, <m>\Pr(X = 1) = \Pr(A)</m>, and <m>\Pr(X = 0) = 1 - \Pr(A)</m>.
</p>
<definition xml:id="def-discrete-and-continuous">
<statement>
<p>
Consider the set <m>S</m> of all real numbers which are actually output by a random variable <m>X</m>.
If <m>S</m> does not contain any interval of values, then the random variable <m>X</m> is called <term>discrete</term>.
Otherwise, it's called <term>continuous</term>.
</p>
</statement>
</definition>
<example>
<statement>
<p>
Any random variable <m>X</m> defined on a finite sample space <m>\Omega</m> is discrete<mdash/>the set of outputs of <m>X</m> cannot contain an interval if it only has finitely many values.
</p>
<p>
Suppose an experiment consists of growing a plant in a new fertilizer, and a random variable <m>Y</m> measures the height of the plant after a set growing period.
Now <m>Y</m> could conceivably take on an interval's worth of values (such as, e.g., any real number between 10 inches and 20 inches), so this random variable would be continuous.
</p>
<p>
The distinction between discrete and continuous random variables is not simply the distinction of whether the sample space is finite or infinite.
Consider the experiment in which we flip a coin repeatedly until we first see a coin come up heads.
Let <m>Z</m> be the number of times the coin is flipped.
Then there are infinitely many possible values of <m>Z</m> (1, 2, 3, and so on), but there's no interval of real numbers which are all possible outputs of <m>Z</m>.
The possible outputs of <m>Z</m> are discrete (in the non-technical, English sense of the word: separated).
</p>
</statement>
</example>
<p>
There are several particular types of distributions that appear repeatedly, since they model common behaviors.
Indicator random variables, for example, will appear any time a random process generates a yes/no or success/failure type of answer to a question.
Below are some other common distributions.
</p>
</subsection>
<subsection xml:id="subsec-binomial-distribution">
<title>The Binomial Distribution</title>
<definition xml:id="def-binomial-distribution">
<statement>
<p>
Suppose an event occurs with probability <m>p</m>.
If we perform <m>n</m> independent trials, let <m>S</m> be the random variable which counts the number of trials in which the event occurred.
Then <m>S</m> has the <term>binomial distribution</term> with parameters <m>n</m> and <m>p</m>.
We'll write <m>S \sim \Bin(n, p)</m> to denote this.
For each value <m>0 \leq k \leq n</m>, we'll write <m>b(k)</m> for <m>\Pr(S = k)</m>.
(If we want to keep track of the parameter values, we may write <m>b(k; n, p)</m>.)
</p>
</statement>
</definition>
<example>
<statement>
<p>
Suppose we have a coin which comes up heads with probability <m>p = 0.4</m>.
If we flip the coin three times and let <m>S</m> be the number of heads, then <m>S</m> is binomially distributed with parameters <m>n = 3</m> and <m>p = 0.4</m>.
Then <m>S</m> has the distribution below.
</p>
<table xml:id="table-binomial-example">
<title><m>\Bin(3, 0.4)</m></title>
<tabular halign="center">
<row header="yes" bottom="minor">
<cell><m>k</m></cell>
<cell><m>\Pr(S = k)</m></cell>
</row>
<row>
<cell>0</cell>
<cell>0.216</cell>
</row>
<row>
<cell>1</cell>
<cell>0.432</cell>
</row>
<row>
<cell>2</cell>
<cell>0.288</cell>
</row>
<row>
<cell>3</cell>
<cell>0.064</cell>
</row>
</tabular>
</table>
</statement>
</example>
<p>
To understand how these probabilities are calculated, consider a particular heads/tails sequence, such as <m>HHHT</m>.
If the probability of heads is 0.4, then the probability of tails is 0.6.
Since the results of different coin flips are independent from each other, the probability of seeing precisely the sequence <m>HHT</m> would be the product <m>(0.4)(0.4)(0.6) = 0.096</m>.
The probability of seeing precisely <m>HTH</m> would be <m>(0.4)(0.6)(0.4) = 0.096</m>, the same product of terms in a different order.
So, if we knew exactly the number of flip sequences of length 3 with exactly 2 heads, we could calculate the probability of seeing 2 heads as:
<md>
<mrow> \Pr(S = 2) = (\text{number of sequences})(\text{probability of each sequence}) </mrow>
</md>
The probability for each sequence is straightforward to generalize: if there are <m>n</m> flips and <m>k</m> of them are heads, then <m>n - k</m> of them are tails.
So the product we should calculate will have <m>k</m> copies of the parameter <m>p</m> and <m>n - k</m> copies of the complementary probability <m>1 - p</m>:
<md>
<mrow>\text{probability of each sequence } = p^k(1-p)^k</mrow>
</md>
</p>
<definition xml:id="def-binomial-coefficient">
<statement>
<p>
We'll write the symbol <m>{n \choose k}</m> to mean the number of ways to choose <m>k</m> things out of <m>n</m> things.
The number <m>{n \choose k}</m> is read "<m>n</m> choose <m>k</m>" and called a <term>binomial coefficient</term>.
</p>
</statement>
</definition>
<p>
The term "binomial coefficient" is used here because these numbers form coefficients of a polynomial in an important theorem about binomials called the <url href="https://en.wikipedia.org/wiki/Binomial_theorem">Binomial Theorem</url>.
The binomial coefficients will precisely tell us the number of flip sequences of length <m>n</m> with precisely <m>k</m> heads (to build such a sequence, we must choose <m>k</m> out of <m>n</m> flips in the sequence to be heads; the rest must be tails).
So, now we need a way to find the numbers <m>{n \choose k}</m>.
</p>
<definition xml:id="def-factorial">
<statement>
<p>
Let <m>n</m> be a positive integer.
Then the <term>factorial</term> of <m>n</m>, written <m>n!</m>, is the product of the positive integers up to <m>n</m>, i.e.:
<md>
<mrow> n! = 1 \times 2 \times 3 \times \dotsb \times (n-1) \times n. </mrow>
</md>
</p>
</statement>
</definition>
<fact xml:id="fact-binomial-coefficient-formula">
<title>Binomial Coefficient Formula</title>
<statement>
<p>
For any <m>0 \leq k \leq n</m>,
<md>
<mrow>{n \choose k} = \frac{n!}{k!(n-k)!}</mrow>
</md>
</p>
</statement>
</fact>
<p>
We won't provide a justification of this formula here, but we'll put it to use to calculate probabilities in the binomial distribution.
For example, the number of flip sequences of length 3 with exactly 2 heads is:
<md>
<mrow> {3 \choose 2} = \frac{3!}{2!1!} = \frac{6}{2} = 3. </mrow>
</md>
Since we previously said the probability of such a sequence is 0.096, we can now find the probability of seeing exactly 2 heads in 3 flips of the coin:
<md>
<mrow> \Pr(S = 2) \amp = (\text{number of sequences})(\text{probability of each sequence}) </mrow>
<mrow> \amp = (3)(0.096) </mrow>
<mrow> \amp = 0.288, </mrow>
</md>
which matches the value in <xref ref="table-binomial-example"/>.
More generally:
</p>
<fact xml:id="fact-binomial-probability-formula">
<title>Binomial Probability Formula</title>
<statement>
<p>
<md>
<mrow>b(k; n, p) = {n \choose k} p^k (1-p)^{n-k}</mrow>
</md>
</p>
</statement>
</fact>
</subsection>
<subsection xml:id="subsec-geometric-distribution">
<title>Geometric Distribution</title>
<definition xml:id="def-geometric-distribution">
<statement>
<p>
Suppose an event occurs with probability <m>p</m>.
If we perform independent trials repeatedly, let <m>N</m> be the random variable which counts the number of trials performed until we see the event occur for the first time.
(<m>N</m> counts the final trial in which the event occurs.) Then <m>N</m> has the <term>geometric distribution</term> with parameter <m>p</m>.
We'll write <m>N \sim \Geom(p)</m> to denote this.
For each value <m>k \geq 1</m>, we'll write <m>g(k)</m> for <m>\Pr(N = k)</m>.
(If we want to keep track of the parameter value, we may write <m>g(k; p)</m>.)
</p>
</statement>
</definition>
<p>
Suppose we flip a coin repeatedly until we first see heads.
The number <m>N</m> of flips is geometrically distributed with parameter <m>p</m>, which gives the probability of a flip coming up heads.
Then the probability of tails is <m>1 - p</m>.
For any particular <m>k \geq 1</m>, in order to have <m>N = k</m>, we must see exactly the flip sequence <m>TT\dotsm TH</m>, where the number of tails is exactly <m>k - 1</m>.
The probability of seeing this particular sequence is then:
</p>
<fact xml:id="fact-geometric-probability-formula">
<title>Geometric Probability Formula</title>
<statement>
<p>
<md>
<mrow> g(k; p) = (1-p)^{k-1} p </mrow>
</md>
</p>
</statement>
</fact>
</subsection>
<subsection xml:id="subsec-poisson-distribution">
<title>Poisson Distribution</title>
<definition xml:id="def-poisson-distribution">
<statement>
<p>
A <term>Poisson process</term> is one in which some event occurs randomly at a constant probabilistic rate <m>\lambda</m> over time.
Suppose we observe a Poisson process for a fixed amount of time <m>t</m>, and let <m>N</m> count the number of occurrences of the event.
Then <m>N</m> has the <term>Poisson distribution</term> with parameters <m>\lambda</m> and <m>t</m>.
We'll write <m>N \sim \Poiss(\lambda, t)</m> to denote this.
For each value <m>k \geq 0</m>, we'll write <m>p(k)</m> for <m>\Pr(N = k)</m>.
(If we want to keep track of the parameter values, we may write <m>p(k; \lambda, t)</m>.
See <xref ref="example-poisson-rate"/> for an important detail about units.)
</p>
</statement>
</definition>
<fact xml:id="fact-poisson-probability-formula">
<title>Poisson Probability Formula</title>
<statement>
<p>
<md>
<mrow> p(k; \lambda, t) = \frac{(\lambda t)^k}{k!} e^{-\lambda t}. </mrow>
</md>
</p>
</statement>
</fact>
<p>
The term <m>\lambda t</m> is potentially misleading here depending on the units given for each value.
The idea is that the rate information might be given for a different period of time than the observation period, so the rate should be scaled before calculating a probability.
But the rate and observation time must be using the same time unit before multiplying.
See the following example.
</p>
<example xml:id="example-poisson-rate">
<statement>
<p>
Suppose we observe traffic along a particular stretch of highway which typically has 200 cars pass per hour.
Let <m>N</m> be the number of cars seen in a 2-hour observation period.
Since the parameter <m>\lambda = 200</m> cars per hour is given for a 1-hour period, we should scale it to find the rate for a 2-hour period: <m>\lambda t = (200)(2) = 400</m> cars per 2-hours.
Then, for example:
<md>
<mrow> \Pr(N = 375) = p(375; 200, 2) = \frac{400^{375}}{375!} e^{-400} \approx 0.01. </mrow>
</md>
</p>
<p>
Now consider the random variable <m>M</m> counting the number of cars seen in a 30-minute observation period.
We should not calculate <m>\lambda t = (200)(30) = 6000</m>, since <m>\lambda</m> measures time in hours and <m>t</m> measures time in minutes.
We should first convert <m>t = 0.5</m> hours, then we can find our appropriately scaled rate information: <m>(200)(0.5) = 100</m> cars per half-hour.
Then, for example:
<md>
<mrow> \Pr(M = 110) = p(110; 200, 0.5) = \frac{100^{110}}{110!}e^{-100} \approx 0.02. </mrow>
</md>
Note that when we write the notation <m>p(k; \lambda, t)</m>, we assume that <m>\lambda</m> and <m>t</m> are expressed with the same time units already.
</p>
</statement>
</example>
</subsection>
<exercises xml:id="exercises-Discrete-RVs">
<exercise>
<statement>
<p>
We flip a coin with unknown bias 100 times and observe 43 heads.
What is the maximum likelihood estimation for the probability of the coin coming up heads?
</p>
</statement>
</exercise>
<exercise>
<statement>
<p>
An experiment consists of flipping a biased coin 20 times.
If the coin comes up heads with probability <m>p = 0.3</m>, find the probability of seeing 5 heads.
Find the probability of seeing up to (and including) 3 heads.
</p>
</statement>
</exercise>
<exercise>
<statement>
<p>
An experiment consists of flipping a coin repeatedly until we first see heads.
</p>
</statement>
</exercise>
<exercise>
<statement>
<p>
If the coin comes up heads with probability 0.4, what is the probability we’ll see our first heads within three flips? What about precisely on the third flip?
</p>
</statement>
</exercise>
<exercise>
<statement>
<p>
Suppose the coin has an unknown probability of coming up heads.
We perform the experiment in five independent trials, during which it takes 4, 5, 4, 3, and 6 flips to see our first heads in each trial.
What is the maximum likelihood estimation for the probability of the coin coming up heads on a flip?
</p>
</statement>
</exercise>
<exercise>
<statement>
<p>
A particular store has an average of 20 customers each hour.
During a 4-hour afternoon shift, what is the probability of serving 80 customers.
</p>
</statement>
</exercise>
<exercise>
<statement>
<p>
A particular store owner wants to approximate the average hourly rate at which customers come into the store.
They observe 80 customers enter during a particular 4-hour shift.
What is the maximum likelihood estimation for the hourly customer rate?
</p>
</statement>
</exercise>
</exercises>
</section>
<section xml:id="sec-Continuous-RVs" xmlns:xi="http://www.w3.org/2001/XInclude">
<title>Continuous Random Variables</title>
<subsection xml:id="subsec-pdfs">
<title>Probability Density Functions</title>
<p>
We have to think about probability differently for continuous random variables.
If <m>X</m> is a continuous random variable taking values in an interval <m>[a, b]</m>, it doesn't quite make sense to ask, as we have for discrete random variables, to find the probabilities of each individual outcome.
</p>
<p>
When we roll a fair 6-sided die, there are 6 equally likely outcomes, so they each get assigned the probability <m>1/6</m>.
If we try to select a random number in the interval <m>[0, 1]</m> in a similarly "fair" way (i.e., with each outcome equally likely), then what probability <m>\alpha</m> should we assign to each outcome? If <m>\alpha > 0</m>, then we have a problem: if you add up the probabilities of all the (infinitely many) outcomes, you get an infinite total, whereas the total probability in any sample space should be 1.
If <m>\alpha = 0</m>, then we also have a problem: if you add up the probabilities of all the outcomes, you get total of 0, whereas the total probability in any sample space should be 1.
</p>
<p>
We need a new way to model and calculate probabilities for continuous random variables.
</p>
<definition xml:id="def-pdf">
<statement>
<p>
Let <m>X</m> be a continuous random variable taking values in the interval <m>[a, b]</m>.
The <term>probability density function</term> (or <term>pdf</term>) for <m>X</m> is a function <m>f(x)</m> defined on <m>[a, b]</m> satisfying the properties:
<ol>
<li>
<p>
<m>f(x) \geq 0</m> for all <m>x \in [a, b]</m>, and
</p>
</li>
<li>
<p>
<m>\displaystyle{\int_a^b f(x)\ dx = 1}</m>.
</p>
</li>
</ol>
</p>
</statement>
</definition>
<p>
It's important to keep in mind that the output values of a pdf <m>f(x)</m> are <em>probability densities</em>, not probabilities.
You can (somewhat casually) think of the units of <m>f(x)</m> values as "probability per &lt;units of the sample space&gt;".
You may not have discussed the units of the differential <m>dx</m> very much in your Calculus 1 course, but <m>dx</m> has the same units as <m>x</m> and is being multiplied by the integrand when you take an integral.
So, although <m>f(x)</m> is not a probability, the result of computing <m>\int_c^d f(x)\ dx</m> <em>is</em> a probability.
</p>
<p>
A pdf's purpose in life is to be integrated, which tells us probabilities for a continuous random variable:
<md>
<mrow> \Pr(c \leq X \leq d) = \int_c^d f(x)\ dx. </mrow>
</md>
</p>
<definition xml:id="def-continuous-uniform-distribution">
<statement>
<p>
Let <m>X</m> take values in the interval <m>[a, b]</m>.
We say that <m>X</m> has the <term>uniform distribution</term> if, for any subinterval <m>[c, d] \subset [a, b]</m>, the probability that <m>X</m> lies within <m>[c, d]</m> is equal to the proportion of the total sample space taken by <m>[c, d]</m>.
That is:
<md>
<mrow> \Pr(c \leq X \leq d) = \frac{d - c}{b - a}. </mrow>
</md>
In this case, <m>X</m> has the pdf <m>f(x) = \frac{1}{b - a}</m>.
</p>
</statement>
</definition>
<p>
We'll ignore for the moment how to find this pdf and instead we'll quickly check that it behaves the way we want it to:
<md>
<mrow> \Pr(c \leq X \leq d) \amp = \int_c^d \frac{1}{b-a}\ dx </mrow>
<mrow> \amp = \frac{x}{b - a}\bigg|_c^d </mrow>
<mrow> \amp = \frac{d - c}{b - a}, </mrow>
</md>
as we wanted.
</p>
<example>
<statement>
<p>
Let <m>X</m> be a continuous random variable taking values in the interval <m>[0, 4]</m> with pdf <m>f(x) = k \sqrt{x}</m> for some constant <m>k</m>.
Because of the restriction that the total probability must be 1, we can determine the value of <m>k</m>:
<md>
<mrow> 1 = \Pr(0\leq X \leq 4) \amp = \int_0^4 k\sqrt{x}\ dx </mrow>
<mrow> \amp = \frac{kx^{3/2}}{3/2}\bigg|_0^4 </mrow>
<mrow> \amp = \frac{2kx^{3/2}}{3}\bigg|_0^4 </mrow>
<mrow> \amp = \left(\frac{2k(4^{3/2})}{3}\right) - 0 </mrow>
<mrow> \amp = \frac{16k}{3} </mrow>
</md>
Therefore <m> k = \frac{3}{16}</m>.
Now we can calculate probabilities, e.g.:
<md>
<mrow> \Pr(1 \leq X \leq 2) \amp = \int_1^2 \frac{3\sqrt{x}}{16}\ dx </mrow>
<mrow> \amp = \frac{3}{16}\left(\frac{2x^{3/2}}{3}\right)\bigg|_1^2 </mrow>
<mrow> \amp = \frac{1}{8}\left(x^{3/2}\right)\bigg|_1^2 </mrow>
<mrow> \amp = \frac{1}{8}\left( 2^{3/2} - 1 \right) </mrow>
<mrow> \amp \approx 0.23. </mrow>
</md>
</p>
</statement>
</example>
</subsection>
<subsection xml:id="subsec-cdf">
<title>Cumulative Distribution Functions</title>
<p>
When a random variable takes values in an interval <m>[a, b]</m>, it's often useful to consider how much probability has been accumulated up to some specified value <m>x \in [a, b]</m>.
For each value of <m>x</m>, we can calculate this accumulated probability.
Since there's an association between inputs (the <m>x</m> values) and output (the accumulated probability), we can naturally build a function.
</p>
<definition xml:id="def-cdf">
<statement>
<p>
Let <m>X</m> be a continuous random variable taking values in an interval <m>[a, b]</m>.
The <term>cumulative distribution function</term> (or <term>cdf</term>) is the function <m>F(x)</m> defined on <m>[a, b]</m> such that
<md>
<mrow> F(x) = \Pr(a \leq X \leq x). </mrow>
</md>
</p>
</statement>
</definition>
<p>
How do we actually calculate these cumulative probabilities? Luckily, these are precisely the sort of probabilities we can use the pdf to find:
<md>
<mrow> F(x) = \Pr(a \leq X \leq x) = \int_a^x f(t)\ dt. </mrow>
</md>
(Note that on the far left side of the equation, we used the letter <m>x</m> to refer to some point within <m>[a, b]</m> up to which we're trying to accumulate probability.
So that value <m>x</m> must also appear at the top of the integral sign.
We can't then use the same letter for the integrand and differential inside the integral; we can't use the same letter to refer to two different variables at the same time.
So, we have changed the letter inside the integral to <m>t</m>.)
</p>
<p>
This equation may look familiar from Calculus 1.
If you review <url href="https://en.wikipedia.org/wiki/Fundamental_theorem_of_calculus#First_part">the First Fundamental Theorem of Calculus</url>, you'll see precisely the relationship between the cdf <m>F(x)</m> and the pdf <m>f(x)</m>:
<md>
<mrow> F'(x) = f(x). </mrow>
</md>
Let's see how we can put this relationship to use.
</p>
<example>
<statement>
<p>
Suppose <m>X</m> is uniformly distributed in the interval <m>[a, b]</m>.
We previously said in <xref ref="def-continuous-uniform-distribution"/> that, for any subinterval <m>[c, d] \subset [a, b]</m>, we should have
<md>
<mrow> \Pr(c\leq X \leq d) = \frac{d - c}{b - a}. </mrow>
</md>
In particular, for any <m>x\in [a, b]</m>, we would have:
<md>
<mrow> F(x) = \Pr(a \leq X \leq x) = \frac{x - a}{b - a}. </mrow>
</md>
Therefore, taking the derivative:
<md>
<mrow> f(x) = F'(x) = \frac{1}{b - a}. </mrow>
</md>
</p>
</statement>
</example>
<p>
So, we can see that the idea we wanted to capture in probability led to a formula that gave accumulated probability, which then led to the pdf.
Both the cdf and the pdf contain essentially all of the information we need to answer probability questions about <m>X</m>.
However, keep in mind that these functions output different mathematical objects.
<m>F(x)</m> outputs probabilities already; <m>f(x)</m> must be integrated over a subinterval to find probabilities.
</p>
</subsection>
<subsection xml:id="subsec-exponential-distribution">
<title>The Exponential Distribution</title>
<definition xml:id="def-exponential-distribution">
<statement>
<p>
Suppose we observe a Poisson process in which an event occurs with rate parameter <m>\lambda</m>.
Let <m>T</m> be the time until the event is first observed.
Then <m>T</m> has the <term>exponential distribution</term> with parameter <m>\lambda</m>.
We'll write <m>T \sim \Exp(\lambda)</m> to denote this.
</p>
</statement>
</definition>
<fact xml:id="fact-exponential-pdf">
<title>Exponential Density Function</title>
<statement>
<p>
<md>
<mrow> f(t) = \lambda e^{-\lambda t} </mrow>
</md>
</p>
</statement>
</fact>
<p>
The binomial, geometric, Poisson, and exponential distributions are conceptually grouped together.
Consider the following two dichotomies:
<ol>
<li>
<p>
Is time being measured discretely or continuously?
</p>
</li>
<li>
<p>
Does the random variable count occurrences of an event, or does it measure time until an event?
</p>
</li>
</ol>
There are four combinations of answers to these two questions.
<dl>
<li>
<title>Binomial</title>
<p>
"Time" is discrete; the random variables counts occurrences.
</p>
</li>
<li>
<title>Geometric</title>
<p>
"Time" is discrete; the random variable measures "time" until an event.
</p>
</li>
<li>
<title>Poisson</title>
<p>
Time is continuous; the random variable counts occurrences.
</p>
</li>
<li>
<title>Exponential</title>
<p>
Time is continuous; the random variable measures time until an event.
</p>
</li>
</dl> When we talk about discrete "time", we don't always mean something measured in actual units of time. We might; for example, an experiment may involve a bacterial colony growing in a petri dish over time, but we only check in and measure something about the bacterial colony once each day. In this case, since our measurements of time must occur in whole numbers of days, time is being measured in a discrete way (that is, not involving an interval of real numbers).
</p>
<p>
Consider, instead, an experiment in which we flip a coin repeatedly until we first see heads appear.
The number of flips doesn't have units of time, but counting flips is analogous to measuring time until an event occurs.
The flip count is a discrete stand-in for something analogous to time.
We can consider this experiment to be a discrete analog of a Poisson process.
</p>
<p>
It's also worth including a warning here about the Poisson distribution, specifically.
The dichotomy mentioned above about whether time is being measured discretely or continuously is <em>not</em> the same as the consideration of whether the random variable itself is discrete or continuous.
In a Poisson process, time is being measured continuously.
But the Poisson random variable counts the number of occurrences of the event, and this number must be a whole number.
So a Poisson random variable is a discrete random variable.
</p>
</subsection>
<subsection xml:id="subsec-normal-distribution">
<title>Normal Distribution</title>
<definition xml:id="def-normal-distribution">
<statement>
<p>
Consider the parametrized family of functions:
<md>
<mrow> f(x; \mu, \sigma^2) = \frac{1}{\sqrt{2\pi \sigma^2}} e^{\frac{-(x-\mu)^2}{2\sigma^2}}. </mrow>
</md>
The function <m>f(x; \mu, \sigma^2)</m> is called a <term>normal density function</term> with parameters <m>\mu</m> and <m>\sigma^2</m>.
A continuous random variable <m>X</m> with pdf <m>f(x; \mu, \sigma^2)</m> is said to have a <term>normal distribution</term>.
In the specific case that <m>\mu = 0</m> and <m>\sigma^2 = 1</m>, we call the resulting distribution the <term>standard normal distribution</term>.
</p>
</statement>
</definition>
<p>
It's an unfortunate reality that the normal density functions can't easily be integrated.
(In fact, it is provably impossible to write nice formulas for their antiderivatives).
So we won't worry much about probability computations like <m>\Pr(c \leq X \leq d)</m> for normal distributions.
The normal distributions will play a <em>central</em> role in later chapters (after which you can return here and appreciate this pun).
</p>
</subsection>
<exercises xml:id="exercises-Continuous-RVs">
<exercise>
<statement>
<p>
A continuous random variable <m>X</m> taking values in <m>[1, 4]</m> has p.d.f.
<m>f(x) = k(x - \sqrt{x})</m> for some constant <m>k</m>.
What is the value of <m>k</m>?
</p>
</statement>
</exercise>
<exercise>
<statement>
<p>
A continuous random variable <m>X</m> taking values in <m>[1, 2]</m> has p.d.f.
<m>\displaystyle{f(x) = \frac{1}{2}\left(\frac{1}{x^2} + x\right)}</m>.
Find the c.d.f.
<m>F(x)</m>.
Use your c.d.f.
to find <m>\Pr\left(1 \leq X \leq \frac{3}{2}\right)</m>.
</p>
</statement>
</exercise>
<exercise>
<statement>
<p>
A continuous random variable <m>X</m> taking values in <m>[2, 3]</m> has c.d.f.
<m>F(x) = \frac{x^3}{3} - x^2 + 4</m>.
Find the p.d.f.
<m>f(x)</m>.
</p>
</statement>
</exercise>
<exercise>
<statement>
<p>
A radioactive material emits particles at an unknown probabilistic rate <m>\lambda</m> particles per minute.
We observe particles emitted at times 1.1, 1.7, 1.3, 2.2, 1.9, and 1.8 minutes.
Write the likelihood function <m>\mathcal{L}(\lambda)</m> based on this data.
What is the maximum likelihood estimation for <m>\lambda</m>?
</p>
</statement>
</exercise>
<exercise>
<statement>
<p>
Suppose a parameter <m>\theta</m> takes values in <m>[0, 1]</m> with likelihood function <m>\mathcal{L}(\theta) = \sqrt{\theta} - \theta^2</m>.
Find the maximum likelihood estimation of <m>\theta</m>.
</p>
</statement>
</exercise>
</exercises>
</section>
<section xml:id="sec-Joint-Distributions">
<title>Joint Distributions</title>
<p>
Sometimes multiple measurements are taken simultaneously and data is naturally grouped by combinations of measurement values.
Consider a poll that asks respondents two yes or no questions.
It may be natural to report counts of respondents who answered yes/yes, yes/no, no/yes, no/no.
</p>
<definition xml:id="def-joint-distribution">
<statement>
<p>
Let <m>X</m> be a random variable taking values <m>x_1, x_2, \dotsc, x_m</m>.
Let <m>Y</m> be a random variable taking values <m>y_1, y_2, \dotsc, y_n</m>.
The <term>joint distribution</term> for <m>X</m> and <m>Y</m> is the collection of probabilities <m>\Pr(X = x_i, Y = y_j)</m> for all <m>1 \leq i \leq m, 1 \leq j \leq n</m>.
The individual distributions for <m>X</m> and <m>Y</m> separately are called the <term>marginal distributions</term>.
</p>
</statement>
</definition>
<p>
In the probability <m>\Pr(X = x_i, Y = y_j)</m>, the comma should be read as "and".
Think of the conditions <m>X = x_i</m> and <m>Y = y_j</m> as events, and <m>\Pr(X = x_i, Y = y_j)</m> is the probability of the intersection of these two events.
</p>
<p>
We're restricting our attention here to joint distributions of two random variables, each taking only finitely many values.
It shouldn't be hard to imagine how to generalize this definition for a joint distribution of three or more discrete random variables.
A bit more adjustment would need to be made for a joint distribution of continuous random variables, or a joint distribution between a discrete and continuous random variable.
</p>
<example xml:id="example-joint-indicators">
<statement>
<p>
A poll asks two yes/no questions.
Let <m>X</m> indicate a yes on Question 1 and <m>Y</m> indicate a yes on Question 2.
When the data is collected, the following joint distribution for <m>X</m> and <m>Y</m> is created:
</p>
<table>
<title>Joint Distribution for Indicator Random Variables</title>
<tabular halign="center">
<row bottom="minor">
<cell right="minor"></cell>
<cell right="minor"><m>X = 0</m></cell>
<cell><m>X = 1</m></cell>
</row>
<row bottom="minor">
<cell right="minor"><m>Y = 0</m></cell>
<cell right="minor">0.1</cell>
<cell>0.2</cell>
</row>
<row>
<cell right="minor"><m>Y = 1</m></cell>
<cell right="minor">0.3</cell>
<cell>0.4</cell>
</row>
</tabular>
</table>
<p>
Note that the sum of all values in the table is 1; this is a probability distribution, and total probability must be 1.
This table doesn't show the distributions for <m>X</m> or <m>Y</m> individually.
The data for the two distributions is mixed together.
We can, if we wish, take this information and determine separate distributions for <m>X</m> and <m>Y</m>.
For the distribution for <m>X</m>, we need to know <m>\Pr(X = 0)</m> and <m>\Pr(X = 1)</m> (with no reference to <m>Y</m>).
We can get these probabilities by summing along the columns of the table:
<md>
<mrow> \Pr(X = 0) \amp = 0.1 + 0.3 = 0.4 </mrow>
<mrow> \Pr(X = 1) \amp = 0.2 + 0.4 = 0.6 </mrow>
</md>
To get the distribution for <m>Y</m>, we should add along the rows:
<md>
<mrow> \Pr(Y = 0) \amp = 0.1 + 0.2 = 0.3 </mrow>
<mrow> \Pr(Y = 1) \amp = 0.3 + 0.4 = 0.7 </mrow>
</md>
</p>
</statement>
</example>
<p>
If we start with the joint distribution, we can get the marginal distributions.
If we start with the marginal distributions, we can't necessarily find the joint distribution, because we won't know how the random variables interact.
</p>
<definition xml:id="def-independent-RVs">
<statement>
<p>
Random variables <m>X</m> and <m>Y</m> are called <term>independent</term> if the events <m>X = x</m> and <m>Y = y</m> are independent for every possible combination of <m>x</m> and <m>y</m>.
</p>
</statement>
</definition>
<example xml:id="example-independent-indicators">
<statement>
<p>
Using the joint distribution and marginal distributions in <xref ref="example-joint-indicators"/>, we can check independence of <m>X</m> and <m>Y</m> by going cell by cell through the joint distribution table.
The top-left cell shows <m>\Pr(X = 0, Y = 0) = 0.1</m>.
The marginal distributions have <m>\Pr(X = 0) = 0.4</m> and <m>\Pr(Y = 0) = 0.3</m>.
If <m>X</m> and <m>Y</m> were independent, we would have
<md>
<mrow> \Pr(X = 0, Y = 0) \amp = \Pr(X = 0) \Pr(Y = 0) </mrow>
<mrow> \text{however, } 0.1 \amp \neq (0.4)(0.3) </mrow>
</md>
Therefore, <m>X</m> and <m>Y</m> are not independent.
</p>
<p>
Note that we can stop checking now.
As soon as we find one cell in the joint distribution table where the independence condition fails, the two random variables are not independent.
In order to establish independence of random variables, <em>all</em> cells in the joint distribution table would have to pass our check.
</p>
</statement>
</example>
<exercises xml:id="exercises-Joint-Distributions">
<exercise>
<statement>
<p>
TODO
</p>
</statement>
</exercise>
</exercises>
</section>
<xi:include href="./sec-Discrete-RVs.ptx" />
<xi:include href="./sec-Continuous-RVs.ptx" />
<xi:include href="./sec-Joint-Distributions.ptx" />
</chapter>
source/sec-Continuous-RVs.ptx
+354
View File
@@ -0,0 +1,354 @@
<section xml:id="sec-Continuous-RVs" xmlns:xi="http://www.w3.org/2001/XInclude">
<title>Continuous Random Variables</title>
<subsection xml:id="subsec-pdfs">
<title>Probability Density Functions</title>
<p>
We have to think about probability differently for continuous random variables.
If <m>X</m> is a continuous random variable taking values in an interval <m>[a, b]</m>, it doesn't quite make sense to ask, as we have for discrete random variables, to find the probabilities of each individual outcome.
</p>
<p>
When we roll a fair 6-sided die, there are 6 equally likely outcomes, so they each get assigned the probability <m>1/6</m>.
If we try to select a random number in the interval <m>[0, 1]</m> in a similarly "fair" way (i.e., with each outcome equally likely), then what probability <m>\alpha</m> should we assign to each outcome? If <m>\alpha > 0</m>, then we have a problem: if you add up the probabilities of all the (infinitely many) outcomes, you get an infinite total, whereas the total probability in any sample space should be 1.
If <m>\alpha = 0</m>, then we also have a problem: if you add up the probabilities of all the outcomes, you get total of 0, whereas the total probability in any sample space should be 1.
</p>
<p>
We need a new way to model and calculate probabilities for continuous random variables.
</p>
<definition xml:id="def-pdf">
<statement>
<p>
Let <m>X</m> be a continuous random variable taking values in the interval <m>[a, b]</m>.
The <term>probability density function</term> (or <term>pdf</term>) for <m>X</m> is a function <m>f(x)</m> defined on <m>[a, b]</m> satisfying the properties:
<ol>
<li>
<p>
<m>f(x) \geq 0</m> for all <m>x \in [a, b]</m>, and
</p>
</li>
<li>
<p>
<m>\displaystyle{\int_a^b f(x)\ dx = 1}</m>.
</p>
</li>
</ol>
</p>
</statement>
</definition>
<p>
It's important to keep in mind that the output values of a pdf <m>f(x)</m> are <em>probability densities</em>, not probabilities.
You can (somewhat casually) think of the units of <m>f(x)</m> values as "probability per &lt;units of the sample space&gt;".
You may not have discussed the units of the differential <m>dx</m> very much in your Calculus 1 course, but <m>dx</m> has the same units as <m>x</m> and is being multiplied by the integrand when you take an integral.
So, although <m>f(x)</m> is not a probability, the result of computing <m>\int_c^d f(x)\ dx</m> <em>is</em> a probability.
</p>
<p>
A pdf's purpose in life is to be integrated, which tells us probabilities for a continuous random variable:
<md>
<mrow> \Pr(c \leq X \leq d) = \int_c^d f(x)\ dx. </mrow>
</md>
</p>
<definition xml:id="def-continuous-uniform-distribution">
<statement>
<p>
Let <m>X</m> take values in the interval <m>[a, b]</m>.
We say that <m>X</m> has the <term>uniform distribution</term> if, for any subinterval <m>[c, d] \subset [a, b]</m>, the probability that <m>X</m> lies within <m>[c, d]</m> is equal to the proportion of the total sample space taken by <m>[c, d]</m>.
That is:
<md>
<mrow> \Pr(c \leq X \leq d) = \frac{d - c}{b - a}. </mrow>
</md>
In this case, <m>X</m> has the pdf <m>f(x) = \frac{1}{b - a}</m>.
</p>
</statement>
</definition>
<p>
We'll ignore for the moment how to find this pdf and instead we'll quickly check that it behaves the way we want it to:
<md>
<mrow> \Pr(c \leq X \leq d) \amp = \int_c^d \frac{1}{b-a}\ dx </mrow>
<mrow> \amp = \frac{x}{b - a}\bigg|_c^d </mrow>
<mrow> \amp = \frac{d - c}{b - a}, </mrow>
</md>
as we wanted.
</p>
<example>
<statement>
<p>
Let <m>X</m> be a continuous random variable taking values in the interval <m>[0, 4]</m> with pdf <m>f(x) = k \sqrt{x}</m> for some constant <m>k</m>.
Because of the restriction that the total probability must be 1, we can determine the value of <m>k</m>:
<md>
<mrow> 1 = \Pr(0\leq X \leq 4) \amp = \int_0^4 k\sqrt{x}\ dx </mrow>
<mrow> \amp = \frac{kx^{3/2}}{3/2}\bigg|_0^4 </mrow>
<mrow> \amp = \frac{2kx^{3/2}}{3}\bigg|_0^4 </mrow>
<mrow> \amp = \left(\frac{2k(4^{3/2})}{3}\right) - 0 </mrow>
<mrow> \amp = \frac{16k}{3} </mrow>
</md>
Therefore <m> k = \frac{3}{16}</m>.
Now we can calculate probabilities, e.g.:
<md>
<mrow> \Pr(1 \leq X \leq 2) \amp = \int_1^2 \frac{3\sqrt{x}}{16}\ dx </mrow>
<mrow> \amp = \frac{3}{16}\left(\frac{2x^{3/2}}{3}\right)\bigg|_1^2 </mrow>
<mrow> \amp = \frac{1}{8}\left(x^{3/2}\right)\bigg|_1^2 </mrow>
<mrow> \amp = \frac{1}{8}\left( 2^{3/2} - 1 \right) </mrow>
<mrow> \amp \approx 0.23. </mrow>
</md>
</p>
</statement>
</example>
</subsection>
<subsection xml:id="subsec-cdf">
<title>Cumulative Distribution Functions</title>
<p>
When a random variable takes values in an interval <m>[a, b]</m>, it's often useful to consider how much probability has been accumulated up to some specified value <m>x \in [a, b]</m>.
For each value of <m>x</m>, we can calculate this accumulated probability.
Since there's an association between inputs (the <m>x</m> values) and output (the accumulated probability), we can naturally build a function.
</p>
<definition xml:id="def-cdf">
<statement>
<p>
Let <m>X</m> be a continuous random variable taking values in an interval <m>[a, b]</m>.
The <term>cumulative distribution function</term> (or <term>cdf</term>) is the function <m>F(x)</m> defined on <m>[a, b]</m> such that
<md>
<mrow> F(x) = \Pr(a \leq X \leq x). </mrow>
</md>
</p>
</statement>
</definition>
<p>
How do we actually calculate these cumulative probabilities? Luckily, these are precisely the sort of probabilities we can use the pdf to find:
<md>
<mrow> F(x) = \Pr(a \leq X \leq x) = \int_a^x f(t)\ dt. </mrow>
</md>
(Note that on the far left side of the equation, we used the letter <m>x</m> to refer to some point within <m>[a, b]</m> up to which we're trying to accumulate probability.
So that value <m>x</m> must also appear at the top of the integral sign.
We can't then use the same letter for the integrand and differential inside the integral; we can't use the same letter to refer to two different variables at the same time.
So, we have changed the letter inside the integral to <m>t</m>.)
</p>
<p>
This equation may look familiar from Calculus 1.
If you review <url href="https://en.wikipedia.org/wiki/Fundamental_theorem_of_calculus#First_part">the First Fundamental Theorem of Calculus</url>, you'll see precisely the relationship between the cdf <m>F(x)</m> and the pdf <m>f(x)</m>:
<md>
<mrow> F'(x) = f(x). </mrow>
</md>
Let's see how we can put this relationship to use.
</p>
<example>
<statement>
<p>
Suppose <m>X</m> is uniformly distributed in the interval <m>[a, b]</m>.
We previously said in <xref ref="def-continuous-uniform-distribution"/> that, for any subinterval <m>[c, d] \subset [a, b]</m>, we should have
<md>
<mrow> \Pr(c\leq X \leq d) = \frac{d - c}{b - a}. </mrow>
</md>
In particular, for any <m>x\in [a, b]</m>, we would have:
<md>
<mrow> F(x) = \Pr(a \leq X \leq x) = \frac{x - a}{b - a}. </mrow>
</md>
Therefore, taking the derivative:
<md>
<mrow> f(x) = F'(x) = \frac{1}{b - a}. </mrow>
</md>
</p>
</statement>
</example>
<p>
So, we can see that the idea we wanted to capture in probability led to a formula that gave accumulated probability, which then led to the pdf.
Both the cdf and the pdf contain essentially all of the information we need to answer probability questions about <m>X</m>.
However, keep in mind that these functions output different mathematical objects.
<m>F(x)</m> outputs probabilities already; <m>f(x)</m> must be integrated over a subinterval to find probabilities.
</p>
</subsection>
<subsection xml:id="subsec-exponential-distribution">
<title>The Exponential Distribution</title>
<definition xml:id="def-exponential-distribution">
<statement>
<p>
Suppose we observe a Poisson process in which an event occurs with rate parameter <m>\lambda</m>.
Let <m>T</m> be the time until the event is first observed.
Then <m>T</m> has the <term>exponential distribution</term> with parameter <m>\lambda</m>.
We'll write <m>T \sim \Exp(\lambda)</m> to denote this.
</p>
</statement>
</definition>
<fact xml:id="fact-exponential-pdf">
<title>Exponential Density Function</title>
<statement>
<p>
<md>
<mrow> f(t) = \lambda e^{-\lambda t} </mrow>
</md>
</p>
</statement>
</fact>
<p>
The binomial, geometric, Poisson, and exponential distributions are conceptually grouped together.
Consider the following two dichotomies:
<ol>
<li>
<p>
Is time being measured discretely or continuously?
</p>
</li>
<li>
<p>
Does the random variable count occurrences of an event, or does it measure time until an event?
</p>
</li>
</ol>
There are four combinations of answers to these two questions.
<dl>
<li>
<title>Binomial</title>
<p>
"Time" is discrete; the random variables counts occurrences.
</p>
</li>
<li>
<title>Geometric</title>
<p>
"Time" is discrete; the random variable measures "time" until an event.
</p>
</li>
<li>
<title>Poisson</title>
<p>
Time is continuous; the random variable counts occurrences.
</p>
</li>
<li>
<title>Exponential</title>
<p>
Time is continuous; the random variable measures time until an event.
</p>
</li>
</dl> When we talk about discrete "time", we don't always mean something measured in actual units of time. We might; for example, an experiment may involve a bacterial colony growing in a petri dish over time, but we only check in and measure something about the bacterial colony once each day. In this case, since our measurements of time must occur in whole numbers of days, time is being measured in a discrete way (that is, not involving an interval of real numbers).
</p>
<p>
Consider, instead, an experiment in which we flip a coin repeatedly until we first see heads appear.
The number of flips doesn't have units of time, but counting flips is analogous to measuring time until an event occurs.
The flip count is a discrete stand-in for something analogous to time.
We can consider this experiment to be a discrete analog of a Poisson process.
</p>
<p>
It's also worth including a warning here about the Poisson distribution, specifically.
The dichotomy mentioned above about whether time is being measured discretely or continuously is <em>not</em> the same as the consideration of whether the random variable itself is discrete or continuous.
In a Poisson process, time is being measured continuously.
But the Poisson random variable counts the number of occurrences of the event, and this number must be a whole number.
So a Poisson random variable is a discrete random variable.
</p>
</subsection>
<subsection xml:id="subsec-normal-distribution">
<title>Normal Distribution</title>
<definition xml:id="def-normal-distribution">
<statement>
<p>
Consider the parametrized family of functions:
<md>
<mrow> f(x; \mu, \sigma^2) = \frac{1}{\sqrt{2\pi \sigma^2}} e^{\frac{-(x-\mu)^2}{2\sigma^2}}. </mrow>
</md>
The function <m>f(x; \mu, \sigma^2)</m> is called a <term>normal density function</term> with parameters <m>\mu</m> and <m>\sigma^2</m>.
A continuous random variable <m>X</m> with pdf <m>f(x; \mu, \sigma^2)</m> is said to have a <term>normal distribution</term>.
In the specific case that <m>\mu = 0</m> and <m>\sigma^2 = 1</m>, we call the resulting distribution the <term>standard normal distribution</term>.
</p>
</statement>
</definition>
<p>
It's an unfortunate reality that the normal density functions can't easily be integrated.
(In fact, it is provably impossible to write nice formulas for their antiderivatives).
So we won't worry much about probability computations like <m>\Pr(c \leq X \leq d)</m> for normal distributions.
The normal distributions will play a <em>central</em> role in later chapters (after which you can return here and appreciate this pun).
</p>
</subsection>
<exercises xml:id="exercises-Continuous-RVs">
<exercise>
<statement>
<p>
A continuous random variable <m>X</m> taking values in <m>[1, 4]</m> has p.d.f.
<m>f(x) = k(x - \sqrt{x})</m> for some constant <m>k</m>.
What is the value of <m>k</m>?
</p>
</statement>
</exercise>
<exercise>
<statement>
<p>
A continuous random variable <m>X</m> taking values in <m>[1, 2]</m> has p.d.f.
<m>\displaystyle{f(x) = \frac{1}{2}\left(\frac{1}{x^2} + x\right)}</m>.
Find the c.d.f.
<m>F(x)</m>.
Use your c.d.f.
to find <m>\Pr\left(1 \leq X \leq \frac{3}{2}\right)</m>.
</p>
</statement>
</exercise>
<exercise>
<statement>
<p>
A continuous random variable <m>X</m> taking values in <m>[2, 3]</m> has c.d.f.
<m>F(x) = \frac{x^3}{3} - x^2 + 4</m>.
Find the p.d.f.
<m>f(x)</m>.
</p>
</statement>
</exercise>
<exercise>
<statement>
<p>
A radioactive material emits particles at an unknown probabilistic rate <m>\lambda</m> particles per minute.
We observe particles emitted at times 1.1, 1.7, 1.3, 2.2, 1.9, and 1.8 minutes.
Write the likelihood function <m>\mathcal{L}(\lambda)</m> based on this data.
What is the maximum likelihood estimation for <m>\lambda</m>?
</p>
</statement>
</exercise>
<exercise>
<statement>
<p>
Suppose a parameter <m>\theta</m> takes values in <m>[0, 1]</m> with likelihood function <m>\mathcal{L}(\theta) = \sqrt{\theta} - \theta^2</m>.
Find the maximum likelihood estimation of <m>\theta</m>.
</p>
</statement>
</exercise>
</exercises>
</section>
source/sec-Discrete-RVs.ptx
+433
View File
@@ -0,0 +1,433 @@
<section xml:id="sec-Discrete-RVs" xmlns:xi="http://www.w3.org/2001/XInclude">
<title>Discrete Random Variables</title>
<subsection xml:id="subsec-Discrete-RVs">
<title>Discrete Random Variables</title>
<definition xml:id="def-RV">
<statement>
<p>
Let <m>\Omega</m> be a sample space.
A <term>random variable</term> is a function <m>X \colon \Omega \to \R</m> which assigns a number to each outcome.
</p>
</statement>
</definition>
<example>
<statement>
<p>
Suppose an experiment consists of flipping a fair coin 4 times.
Let <m>X</m> be the number of heads.
The actual outcomes in the experiment are heads/tails sequences of length 4, such as <m>HTHT</m> and <m>HHHT</m>.
The random variable <m>X</m> assigns a numerical measurement to the outcomes, such as <m>X(HTHT) = 2</m> and <m>X(HHHT) = 3</m>.
</p>
<p>
For any real number value <m>a</m>, we can consider the event consisting of outcomes such that <m>X = a</m>.
For example, the flip sequences <m>HHHT, HHTH, HTHH</m>, and <m>THHH</m> all have 3 heads, so <m>\Pr(X = 3) = \frac{4}{16} = \frac{1}{4}</m>.
</p>
</statement>
</example>
<example xml:id="example-indicator-distribution">
<title>Indicator Random Variable</title>
<statement>
<p>
Let <m>A</m> be an event in the sample space <m>\Omega</m>.
An <term>indicator random variable</term> for <m>A</m> is the random variable <m>X</m> such that
<md>
<mrow> X(\omega) = \begin{cases} 1 \amp \omega \in A \\ 0 \amp \omega \notin A \end{cases} </mrow>
</md>
</p>
</statement>
</example>
<p>
Indicator random variables are also called <term>Bernoulli random variables</term>, although we'll prefer the former term since it more clearly states the purpose of these random variables: to indicate whether or not a particular event has occurred.
We'll use the language "<m>X</m> indicates <m>A</m>" to mean that <m>X</m> is an indicator random variable for the event <m>A</m>.
In this case, <m>\Pr(X = 1) = \Pr(A)</m>, and <m>\Pr(X = 0) = 1 - \Pr(A)</m>.
</p>
<definition xml:id="def-discrete-and-continuous">
<statement>
<p>
Consider the set <m>S</m> of all real numbers which are actually output by a random variable <m>X</m>.
If <m>S</m> does not contain any interval of values, then the random variable <m>X</m> is called <term>discrete</term>.
Otherwise, it's called <term>continuous</term>.
</p>
</statement>
</definition>
<example>
<statement>
<p>
Any random variable <m>X</m> defined on a finite sample space <m>\Omega</m> is discrete<mdash/>the set of outputs of <m>X</m> cannot contain an interval if it only has finitely many values.
</p>
<p>
Suppose an experiment consists of growing a plant in a new fertilizer, and a random variable <m>Y</m> measures the height of the plant after a set growing period.
Now <m>Y</m> could conceivably take on an interval's worth of values (such as, e.g., any real number between 10 inches and 20 inches), so this random variable would be continuous.
</p>
<p>
The distinction between discrete and continuous random variables is not simply the distinction of whether the sample space is finite or infinite.
Consider the experiment in which we flip a coin repeatedly until we first see a coin come up heads.
Let <m>Z</m> be the number of times the coin is flipped.
Then there are infinitely many possible values of <m>Z</m> (1, 2, 3, and so on), but there's no interval of real numbers which are all possible outputs of <m>Z</m>.
The possible outputs of <m>Z</m> are discrete (in the non-technical, English sense of the word: separated).
</p>
</statement>
</example>
<p>
There are several particular types of distributions that appear repeatedly, since they model common behaviors.
Indicator random variables, for example, will appear any time a random process generates a yes/no or success/failure type of answer to a question.
Below are some other common distributions.
</p>
</subsection>
<subsection xml:id="subsec-binomial-distribution">
<title>The Binomial Distribution</title>
<definition xml:id="def-binomial-distribution">
<statement>
<p>
Suppose an event occurs with probability <m>p</m>.
If we perform <m>n</m> independent trials, let <m>S</m> be the random variable which counts the number of trials in which the event occurred.
Then <m>S</m> has the <term>binomial distribution</term> with parameters <m>n</m> and <m>p</m>.
We'll write <m>S \sim \Bin(n, p)</m> to denote this.
For each value <m>0 \leq k \leq n</m>, we'll write <m>b(k)</m> for <m>\Pr(S = k)</m>.
(If we want to keep track of the parameter values, we may write <m>b(k; n, p)</m>.)
</p>
</statement>
</definition>
<example>
<statement>
<p>
Suppose we have a coin which comes up heads with probability <m>p = 0.4</m>.
If we flip the coin three times and let <m>S</m> be the number of heads, then <m>S</m> is binomially distributed with parameters <m>n = 3</m> and <m>p = 0.4</m>.
Then <m>S</m> has the distribution below.
</p>
<table xml:id="table-binomial-example">
<title><m>\Bin(3, 0.4)</m></title>
<tabular halign="center">
<row header="yes" bottom="minor">
<cell><m>k</m></cell>
<cell><m>\Pr(S = k)</m></cell>
</row>
<row>
<cell>0</cell>
<cell>0.216</cell>
</row>
<row>
<cell>1</cell>
<cell>0.432</cell>
</row>
<row>
<cell>2</cell>
<cell>0.288</cell>
</row>
<row>
<cell>3</cell>
<cell>0.064</cell>
</row>
</tabular>
</table>
</statement>
</example>
<p>
To understand how these probabilities are calculated, consider a particular heads/tails sequence, such as <m>HHHT</m>.
If the probability of heads is 0.4, then the probability of tails is 0.6.
Since the results of different coin flips are independent from each other, the probability of seeing precisely the sequence <m>HHT</m> would be the product <m>(0.4)(0.4)(0.6) = 0.096</m>.
The probability of seeing precisely <m>HTH</m> would be <m>(0.4)(0.6)(0.4) = 0.096</m>, the same product of terms in a different order.
So, if we knew exactly the number of flip sequences of length 3 with exactly 2 heads, we could calculate the probability of seeing 2 heads as:
<md>
<mrow> \Pr(S = 2) = (\text{number of sequences})(\text{probability of each sequence}) </mrow>
</md>
The probability for each sequence is straightforward to generalize: if there are <m>n</m> flips and <m>k</m> of them are heads, then <m>n - k</m> of them are tails.
So the product we should calculate will have <m>k</m> copies of the parameter <m>p</m> and <m>n - k</m> copies of the complementary probability <m>1 - p</m>:
<md>
<mrow>\text{probability of each sequence } = p^k(1-p)^k</mrow>
</md>
</p>
<definition xml:id="def-binomial-coefficient">
<statement>
<p>
We'll write the symbol <m>{n \choose k}</m> to mean the number of ways to choose <m>k</m> things out of <m>n</m> things.
The number <m>{n \choose k}</m> is read "<m>n</m> choose <m>k</m>" and called a <term>binomial coefficient</term>.
</p>
</statement>
</definition>
<p>
The term "binomial coefficient" is used here because these numbers form coefficients of a polynomial in an important theorem about binomials called the <url href="https://en.wikipedia.org/wiki/Binomial_theorem">Binomial Theorem</url>.
The binomial coefficients will precisely tell us the number of flip sequences of length <m>n</m> with precisely <m>k</m> heads (to build such a sequence, we must choose <m>k</m> out of <m>n</m> flips in the sequence to be heads; the rest must be tails).
So, now we need a way to find the numbers <m>{n \choose k}</m>.
</p>
<definition xml:id="def-factorial">
<statement>
<p>
Let <m>n</m> be a positive integer.
Then the <term>factorial</term> of <m>n</m>, written <m>n!</m>, is the product of the positive integers up to <m>n</m>, i.e.:
<md>
<mrow> n! = 1 \times 2 \times 3 \times \dotsb \times (n-1) \times n. </mrow>
</md>
</p>
</statement>
</definition>
<fact xml:id="fact-binomial-coefficient-formula">
<title>Binomial Coefficient Formula</title>
<statement>
<p>
For any <m>0 \leq k \leq n</m>,
<md>
<mrow>{n \choose k} = \frac{n!}{k!(n-k)!}</mrow>
</md>
</p>
</statement>
</fact>
<p>
We won't provide a justification of this formula here, but we'll put it to use to calculate probabilities in the binomial distribution.
For example, the number of flip sequences of length 3 with exactly 2 heads is:
<md>
<mrow> {3 \choose 2} = \frac{3!}{2!1!} = \frac{6}{2} = 3. </mrow>
</md>
Since we previously said the probability of such a sequence is 0.096, we can now find the probability of seeing exactly 2 heads in 3 flips of the coin:
<md>
<mrow> \Pr(S = 2) \amp = (\text{number of sequences})(\text{probability of each sequence}) </mrow>
<mrow> \amp = (3)(0.096) </mrow>
<mrow> \amp = 0.288, </mrow>
</md>
which matches the value in <xref ref="table-binomial-example"/>.
More generally:
</p>
<fact xml:id="fact-binomial-probability-formula">
<title>Binomial Probability Formula</title>
<statement>
<p>
<md>
<mrow>b(k; n, p) = {n \choose k} p^k (1-p)^{n-k}</mrow>
</md>
</p>
</statement>
</fact>
</subsection>
<subsection xml:id="subsec-geometric-distribution">
<title>Geometric Distribution</title>
<definition xml:id="def-geometric-distribution">
<statement>
<p>
Suppose an event occurs with probability <m>p</m>.
If we perform independent trials repeatedly, let <m>N</m> be the random variable which counts the number of trials performed until we see the event occur for the first time.
(<m>N</m> counts the final trial in which the event occurs.) Then <m>N</m> has the <term>geometric distribution</term> with parameter <m>p</m>.
We'll write <m>N \sim \Geom(p)</m> to denote this.
For each value <m>k \geq 1</m>, we'll write <m>g(k)</m> for <m>\Pr(N = k)</m>.
(If we want to keep track of the parameter value, we may write <m>g(k; p)</m>.)
</p>
</statement>
</definition>
<p>
Suppose we flip a coin repeatedly until we first see heads.
The number <m>N</m> of flips is geometrically distributed with parameter <m>p</m>, which gives the probability of a flip coming up heads.
Then the probability of tails is <m>1 - p</m>.
For any particular <m>k \geq 1</m>, in order to have <m>N = k</m>, we must see exactly the flip sequence <m>TT\dotsm TH</m>, where the number of tails is exactly <m>k - 1</m>.
The probability of seeing this particular sequence is then:
</p>
<fact xml:id="fact-geometric-probability-formula">
<title>Geometric Probability Formula</title>
<statement>
<p>
<md>
<mrow> g(k; p) = (1-p)^{k-1} p </mrow>
</md>
</p>
</statement>
</fact>
</subsection>
<subsection xml:id="subsec-poisson-distribution">
<title>Poisson Distribution</title>
<definition xml:id="def-poisson-distribution">
<statement>
<p>
A <term>Poisson process</term> is one in which some event occurs randomly at a constant probabilistic rate <m>\lambda</m> over time.
Suppose we observe a Poisson process for a fixed amount of time <m>t</m>, and let <m>N</m> count the number of occurrences of the event.
Then <m>N</m> has the <term>Poisson distribution</term> with parameters <m>\lambda</m> and <m>t</m>.
We'll write <m>N \sim \Poiss(\lambda, t)</m> to denote this.
For each value <m>k \geq 0</m>, we'll write <m>p(k)</m> for <m>\Pr(N = k)</m>.
(If we want to keep track of the parameter values, we may write <m>p(k; \lambda, t)</m>.
See <xref ref="example-poisson-rate"/> for an important detail about units.)
</p>
</statement>
</definition>
<fact xml:id="fact-poisson-probability-formula">
<title>Poisson Probability Formula</title>
<statement>
<p>
<md>
<mrow> p(k; \lambda, t) = \frac{(\lambda t)^k}{k!} e^{-\lambda t}. </mrow>
</md>
</p>
</statement>
</fact>
<p>
The term <m>\lambda t</m> is potentially misleading here depending on the units given for each value.
The idea is that the rate information might be given for a different period of time than the observation period, so the rate should be scaled before calculating a probability.
But the rate and observation time must be using the same time unit before multiplying.
See the following example.
</p>
<example xml:id="example-poisson-rate">
<statement>
<p>
Suppose we observe traffic along a particular stretch of highway which typically has 200 cars pass per hour.
Let <m>N</m> be the number of cars seen in a 2-hour observation period.
Since the parameter <m>\lambda = 200</m> cars per hour is given for a 1-hour period, we should scale it to find the rate for a 2-hour period: <m>\lambda t = (200)(2) = 400</m> cars per 2-hours.
Then, for example:
<md>
<mrow> \Pr(N = 375) = p(375; 200, 2) = \frac{400^{375}}{375!} e^{-400} \approx 0.01. </mrow>
</md>
</p>
<p>
Now consider the random variable <m>M</m> counting the number of cars seen in a 30-minute observation period.
We should not calculate <m>\lambda t = (200)(30) = 6000</m>, since <m>\lambda</m> measures time in hours and <m>t</m> measures time in minutes.
We should first convert <m>t = 0.5</m> hours, then we can find our appropriately scaled rate information: <m>(200)(0.5) = 100</m> cars per half-hour.
Then, for example:
<md>
<mrow> \Pr(M = 110) = p(110; 200, 0.5) = \frac{100^{110}}{110!}e^{-100} \approx 0.02. </mrow>
</md>
Note that when we write the notation <m>p(k; \lambda, t)</m>, we assume that <m>\lambda</m> and <m>t</m> are expressed with the same time units already.
</p>
</statement>
</example>
</subsection>
<exercises xml:id="exercises-Discrete-RVs"> <!--
<exercise>
<statement>
<p>
We flip a coin with unknown parameter <m>p</m> 100 times and observe 43 heads.
What is the maximum likelihood estimation for the probability of the coin coming up heads?
</p>
</statement>
</exercise>
-->
<exercise>
<statement>
<p>
An experiment consists of flipping a biased coin 20 times.
If the coin comes up heads with probability <m>p = 0.3</m>, find the probability of seeing 5 heads.
Find the probability of seeing up to (and including) 3 heads.
</p>
</statement>
<solution>
<p>
Let <m>S</m> be the number of heads.
Then <m>S \sim \Bin(20, 0.3)</m>, so:
<md>
<mrow> \Pr(S = 5) \amp = {20 \choose 5} (0.3)^5 (0.7)^{20 - 5} </mrow>
<mrow> \amp = \frac{20!}{(5!)(15!)} (0.3)^5 (0.7)^{15} </mrow>
<mrow> \amp = \frac{20 \times 19 \times 18 \times 17 \times 16}{5 \times 4 \times 3 \times 2 \times 1} (0.3)^5 (0.7)^{15} </mrow>
<mrow> \amp = (19 \times 3 \times 17 \times 16) (0.3)^5 (0.7)^{15} </mrow>
<mrow> \amp \approx 0.179 </mrow>
</md>
</p>
</solution>
</exercise>
<exercise>
<introduction>
<p>
An experiment consists of flipping a coin repeatedly until we first see heads.
</p>
</introduction>
<task>
<statement>
<p>
If the coin comes up heads with probability 0.4, what is the probability we'll see our first heads within three flips? What about precisely on the third flip?
</p>
</statement>
<solution>
<p>
Let <m>T</m> be the number of flips until we see heads.
Then <m>T</m> is geometric with parameter <m>p = 0.4</m>, so:
<md>
<mrow> \Pr(T = k) \amp = (1-0.4)^{k-1}(0.4) = 0.6^{k-1} \cdot 0.4 </mrow>
</md>
</p>
</solution>
</task>
<task>
<statement>
<p>
Which flip has the highest chance of being the first flip to come up heads?
</p>
</statement>
</task>
</exercise>
<!--
<exercise>
<statement>
<p>
Suppose the coin has an unknown probability of coming up heads.
We perform the experiment in five independent trials, during which it takes 4, 5, 4, 3, and 6 flips to see our first heads in each trial.
What is the maximum likelihood estimation for the probability of the coin coming up heads on a flip?
</p>
</statement>
</exercise>
-->
<exercise>
<statement>
<p>
A particular store has an average of 20 customers each hour.
During a 4-hour afternoon shift, what is the probability of serving 80 customers.
</p>
</statement>
</exercise>
<!--
<exercise>
<statement>
<p>
A particular store owner wants to approximate the average hourly rate at which customers come into the store.
They observe 80 customers enter during a particular 4-hour shift.
What is the maximum likelihood estimation for the hourly customer rate?
</p>
</statement>
</exercise>
--> </exercises>
</section>
source/sec-Joint-Distributions.ptx
+128
View File
@@ -0,0 +1,128 @@
<section xml:id="sec-Joint-Distributions">
<title>Joint Distributions</title>
<p>
Sometimes multiple measurements are taken simultaneously and data is naturally grouped by combinations of measurement values.
Consider a poll that asks respondents two yes or no questions.
It may be natural to report counts of respondents who answered yes/yes, yes/no, no/yes, no/no.
</p>
<definition xml:id="def-joint-distribution">
<statement>
<p>
Let <m>X</m> be a random variable taking values <m>x_1, x_2, \dotsc, x_m</m>.
Let <m>Y</m> be a random variable taking values <m>y_1, y_2, \dotsc, y_n</m>.
The <term>joint distribution</term> for <m>X</m> and <m>Y</m> is the collection of probabilities <m>\Pr(X = x_i, Y = y_j)</m> for all <m>1 \leq i \leq m, 1 \leq j \leq n</m>.
The individual distributions for <m>X</m> and <m>Y</m> separately are called the <term>marginal distributions</term>.
</p>
</statement>
</definition>
<p>
In the probability <m>\Pr(X = x_i, Y = y_j)</m>, the comma should be read as "and".
Think of the conditions <m>X = x_i</m> and <m>Y = y_j</m> as events, and <m>\Pr(X = x_i, Y = y_j)</m> is the probability of the intersection of these two events.
</p>
<p>
We're restricting our attention here to joint distributions of two random variables, each taking only finitely many values.
It shouldn't be hard to imagine how to generalize this definition for a joint distribution of three or more discrete random variables.
A bit more adjustment would need to be made for a joint distribution of continuous random variables, or a joint distribution between a discrete and continuous random variable.
</p>
<example xml:id="example-joint-indicators">
<statement>
<p>
A poll asks two yes/no questions.
Let <m>X</m> indicate a yes on Question 1 and <m>Y</m> indicate a yes on Question 2.
When the data is collected, the following joint distribution for <m>X</m> and <m>Y</m> is created:
</p>
<table>
<title>Joint Distribution for Indicator Random Variables</title>
<tabular halign="center">
<row bottom="minor">
<cell right="minor"></cell>
<cell right="minor"><m>X = 0</m></cell>
<cell><m>X = 1</m></cell>
</row>
<row bottom="minor">
<cell right="minor"><m>Y = 0</m></cell>
<cell right="minor">0.1</cell>
<cell>0.2</cell>
</row>
<row>
<cell right="minor"><m>Y = 1</m></cell>
<cell right="minor">0.3</cell>
<cell>0.4</cell>
</row>
</tabular>
</table>
<p>
Note that the sum of all values in the table is 1; this is a probability distribution, and total probability must be 1.
This table doesn't show the distributions for <m>X</m> or <m>Y</m> individually.
The data for the two distributions is mixed together.
We can, if we wish, take this information and determine separate distributions for <m>X</m> and <m>Y</m>.
For the distribution for <m>X</m>, we need to know <m>\Pr(X = 0)</m> and <m>\Pr(X = 1)</m> (with no reference to <m>Y</m>).
We can get these probabilities by summing along the columns of the table:
<md>
<mrow> \Pr(X = 0) \amp = 0.1 + 0.3 = 0.4 </mrow>
<mrow> \Pr(X = 1) \amp = 0.2 + 0.4 = 0.6 </mrow>
</md>
To get the distribution for <m>Y</m>, we should add along the rows:
<md>
<mrow> \Pr(Y = 0) \amp = 0.1 + 0.2 = 0.3 </mrow>
<mrow> \Pr(Y = 1) \amp = 0.3 + 0.4 = 0.7 </mrow>
</md>
</p>
</statement>
</example>
<p>
If we start with the joint distribution, we can get the marginal distributions.
If we start with the marginal distributions, we can't necessarily find the joint distribution, because we won't know how the random variables interact.
</p>
<definition xml:id="def-independent-RVs">
<statement>
<p>
Random variables <m>X</m> and <m>Y</m> are called <term>independent</term> if the events <m>X = x</m> and <m>Y = y</m> are independent for every possible combination of <m>x</m> and <m>y</m>.
</p>
</statement>
</definition>
<example xml:id="example-independent-indicators">
<statement>
<p>
Using the joint distribution and marginal distributions in <xref ref="example-joint-indicators"/>, we can check independence of <m>X</m> and <m>Y</m> by going cell by cell through the joint distribution table.
The top-left cell shows <m>\Pr(X = 0, Y = 0) = 0.1</m>.
The marginal distributions have <m>\Pr(X = 0) = 0.4</m> and <m>\Pr(Y = 0) = 0.3</m>.
If <m>X</m> and <m>Y</m> were independent, we would have
<md>
<mrow> \Pr(X = 0, Y = 0) \amp = \Pr(X = 0) \Pr(Y = 0) </mrow>
<mrow> \text{however, } 0.1 \amp \neq (0.4)(0.3) </mrow>
</md>
Therefore, <m>X</m> and <m>Y</m> are not independent.
</p>
<p>
Note that we can stop checking now.
As soon as we find one cell in the joint distribution table where the independence condition fails, the two random variables are not independent.
In order to establish independence of random variables, <em>all</em> cells in the joint distribution table would have to pass our check.
</p>
</statement>
</example>
<exercises xml:id="exercises-Joint-Distributions">
<exercise>
<statement>
<p>
TODO
</p>
</statement>
</exercise>
</exercises>
</section>