Probability Distributions

Sample Spaces and Events

An experiment is any action or process whose outcome is subject to uncertainty. The sample space of an experiment, denoted $S$, is the set of all possible outcomes of that experiment. An event is any subset of $S$.

Axioms of Probability

Let $S$ be a sample space, $E$ an event and $\{E_1,E_2,...\}$ a countable collection of pairwise disjoint events. Then,

1. $P\left(E\right)\ge 0$
2. $P\left(S\right)=1$
3. $P\left(\bigcup _{i=1}^{\infty }{E}_i\right)=\sum _{i=1}^{\infty }P\left(E_i\right)$

Random Variable

Let $S$ be a sample space and let $X:S\to R$. Then $X$ is a random variable.

Example: For a coin toss we have $S=\{T,H\}$. Let $X:S\to R$ with $X\left(T\right)=0$ and $X\left(H\right)=1$.

A random variable is a discrete random variable if its set of values is countable. A random variable is continuous if its set of values is an interval. (That’s not quite accurate, but it’s good enough.)

Probability Mass Function (pmf)

Let $X:S\to R$ be a discrete random variable. The probability mass function of $X$ (pmf) is the function $p:R\to [0,1]$ such that for all $x\in R$, $p\left(x\right)=P(X=x)$.

Example: The pmf for the random variable in the previous example where the probability of heads is $\theta$ is given by

$$p\left(x\right)=\{\begin{array}{cc}1-\theta & \text{if}x=0\\ \theta & \text{if}x=1\\ 0& \text{otherwise}\end{array}$$

Probability mass functions can be visualized with a line graph. For $\theta =0.2,0.5,$ and $0.8$:

It’s worth noting a couple of properties of a pmf:

1. $p\left(x\right)\ge 0,x\in R$
2. $p\left(x\right)=0,x\notin X\left(S\right)$
3. ${\sum }_{x\in X\left(S\right)}p\left(x\right)=1$

Before moving on to continuous rv’s, a couple more examples.

Example: Consider the experiment of tossing a coin twice and counting the number of heads. Give the sample space, define a sensible random variable and draw a line graph for the pmf.

Solution: $S=\{TT,TH,HT,HH\}$. Define $X:S\to R$ by $X\left(TT\right)=0$, $X\left(TH\right)=X\left(HT\right)=1$ and $X\left(HH\right)=2$. Then the pmf is given by

$$p\left(x\right)=\{\begin{array}{cc}0.25& \text{if}x=0\\ .5& \text{if}x=1\\ 0.25& \text{if}x=2\\ 0& \text{otherwise}\end{array}$$

ex1_df <- tibble(
  x = c(0,1,2),
  p_x = c(0.25, 0.5, 0.25)
)
ex1_df %>% ggplot(aes(x, p_x)) + 
  geom_col(width = 0.01) + 
  geom_point()

Example: Consider the experiment of tossing a biased coin (where the probability of heads is $\theta$) until it lands on heads. Give the sample space, define a sensible random variable and draw a line graph for the pmf.

Solution: $S=\{H,TH,TTH,TTTH,...\}$. Define $X:S\to R$ by the number of flips required. Then the pmf is given by

$$p\left(x\right)=\{\begin{array}{cc}(1-\theta {)}^{x-1}\theta ,& x=1,2,3,...\\ 0& \text{otherwise}\end{array}$$

Probability Density Function (pdf)

Let $X$ be a continuous random variable. The probability density function of $X$ (pdf) is a function $f\left(x\right)$ such that for $a\le b$,

$$P(a\le X\le b)={\int }_a^{b}f\left(x\right)dx.$$

A couple of properties:

1. $f\left(x\right)\ge 0,x\in R$
2. ${\int }_{-\infty }^{\infty }f\left(x\right)dx=1$
3. $P(X=c)={\int }_c^{c}f\left(x\right)dx=0$

Example: A uniform random variable on $[a,b]$ is a model for “pick a random number between a and b.” The pdf is given by

$$f\left(x\right)=\{\begin{array}{cc}\frac{1}{b-a},& x\in [a,b]\\ 0& \text{otherwise}\end{array}.$$

Example: Let $X$ be normal random variable. Then $X$ has the pdf given by

$$f\left(x\right)=\frac{1}{\sqrt{2\pi {\sigma }^2}}{e}^{-\frac{(x-\mu {)}^2}{2{\sigma }^2}}$$

where $\mu$ and ${\sigma }^2$ are the mean and variance of $X$. These are properties of $X$ that we now discuss.

Mean and Variance

Let $X$ be a random variable. The mean or expected value of $X$ is defined by

$$E\left(X\right)=\{\begin{array}{cc}\sum _{x\in X\left(S\right)}x\cdot p\left(x\right),& X\text{discrete}\\ {\int }_{-\infty }^{\infty }x\cdot f\left(x\right)dx,& X\text{continuous}\end{array}.$$

The variance of $X$ is given by

$$V\left(X\right)=E\left((X-E(X){)}^2\right).$$

The expected value is often denoted $\mu$ and the variance, ${\sigma }^2$. The square root of the variance, $\sigma$, is called the standard deviation.

Highest density interval

The mean gives us one way to quantify the central tendency of a distribution. Variance is one way to quantify the spread of a distribution. Another way to quantify the spread of a distribution is to specify the highest density interval (HDI). The 95% HDI for a distribution, for example, is an interval $I$ such that for $x\in I$, $f\left(x\right)>W$ for some fixed $W$ and

$${\int }_{I}f\left(x\right)dx=\mathrm{0.95.}$$

f <- function(x) (1 / sqrt(2 * pi * sig ^ 2)) * exp(-(x - mu) ^ 2 / 2 / sig ^ 2)
dx <- 0.01
x <- seq(-0.6, 0.6, by = dx)
mu <- 0
sig <- 0.2
plot(x, f(x), type = "l")

sum(f(x)[-1]*dx)

## [1] 0.9972947

sum(f(seq(-0.2, 0.2, by = dx))[-1]*dx)

## [1] 0.6825887