SU-CS229 Distribution Sheet
Here’s a bunch of exponential family distributions. Recall:
\begin{equation} p\left(x;\eta\right) = b\left(x\right) \exp \left(\eta^{T}T\left(x\right) - a\left(\eta\right)\right) \end{equation}
normal, berunouli, posisson, binomial, negative binomial, geometric, chi-squared, exponential are all in
normal distribution \mu the mean, \sigma the variance
\begin{equation} p\left(x;\mu, \Sigma\right) = \frac{1}{\left(2\pi\right)^{\frac{|x|}{2}} \text{det}\left(\Sigma\right)^{\frac{1}{2}}} \exp \left(-\frac{1}{2} \left(x-\mu\right)^{T}\Sigma^{-1}\left(x-\mu\right)\right) \end{equation}
\begin{equation} p\left(x; \mu, \sigma\right) = \frac{1}{\sqrt{2\pi\sigma^{2}}} \exp \left({ \frac{-(x-u)^{2}}{2 \sigma^{2}}}\right) \end{equation}
\begin{equation} \mathbb{E}[x] = \mu \end{equation}
\begin{equation} \text{Var}\left[x\right] = \sigma^{2} \end{equation}
This is exponential family distribution. For \sigma^{2} = 1:
\begin{equation} p\left(x;\mu\right) = \frac{1}{\sqrt{2\pi}} \exp \left(-\frac{1}{2} x^{2}\right) \exp \left(\mu x - \frac{1}{2} \mu^{2}\right) \end{equation}
\eta = \mu T\left(x\right) = x a\left(\eta\right) = \frac{\mu^{2}}{2} = \frac{\eta^{2}}{2} b\left(x\right) = \left(\frac{1}{\sqrt{2\pi}}\right) \exp \left(-\frac{x^{2}}{2}\right) Bernoulli distribution Success with probability p, failure with probability 1-p.
\begin{equation} p(x; p) = p^{x} (1-p)^{1-x} \end{equation}
\begin{equation} \mathbb{E}[x] = \mu \end{equation}
\begin{equation} \text{Var}\left[x\right] = p\left(1-p\right) \end{equation}
This is an exponential family distribution.
\begin{equation} p\left(x; p\right) = \exp \left(\left(\log \left(\frac{p}{1-p}\right)\right) x + \log\left(1-p\right)\right) \end{equation}
\eta = \log \left(\frac{p}{1-p}\right) T\left(x\right) = x a\left(\eta\right) = \log \left(1+e^{\eta}\right) b\left(x\right) = 1 Poisson distribution What is the chance of having an event occurring x times in a unit time when on average, this event happens at a rate of \lambda per unit time.
\begin{equation} p\left(x; \lambda\right) = e^{-\lambda} \frac{\lambda^{x}}{x!} \end{equation}
\begin{equation} \mathbb{E}[x] = \lambda \end{equation}
\begin{equation} \text{Var}[x] = \lambda \end{equation}
This is an exponential family distribution.
\begin{equation} p\left(x; \lambda\right) = \frac{1}{x!} \exp \left(\log\left(\lambda\right) x - e^{\log \lambda }\right) \end{equation}
\eta = \log \left(\lambda\right) T\left(x\right) = x a\left(\eta\right) = \exp \left(\eta\right) b\left(x\right) = \frac{1}{x!} Binomial Distribution What’s probability of n coin flips getting exactly x heads given one head’s probability is p
\begin{equation} p\left(x; p,n\right) = \mqty(n \\ x) p^{x} \left(1-p\right)^{n-x} \end{equation}
\begin{equation} \mathbb{E}[x] = np \end{equation}
\begin{equation} \text{Var}[x] = np\left(1-p\right) \end{equation}
This admits an exponential distribution.
\begin{equation} p\left(x; p,n\right) = \mqty(n \\ x) \exp \left(x \log\left(\frac{p}{1-p}\right) - n \log\left(1+ \left(\frac{p}{1-p}\right)\right)\right) \end{equation}
\eta = \log \left(\frac{p}{1-p}\right) T\left(x\right) = x a\left(\eta\right) = n \log \left(1+\left(\frac{p}{1-p}\right)\right) = n \log \left(1+e^{\eta}\right) b\left(x\right) = \mqty(n \\ x) Negative Binomial Distribution What is the probability of it taking x trials until the k th success, given success probability of one trial is p.
\begin{equation} p\left(x; p,k\right) = \mqty(x-1 \\ k-1) \left(1-p\right)^{x-k} p^{k} \end{equation}
\begin{equation} \mathbb{E}[x] = \frac{k\left(1-p\right)}{p} \end{equation}
\begin{equation} \text{Var}[x] = \frac{k\left(1-p\right)}{p^{2}} \end{equation}
This is a exponential distribution as well.
\begin{equation} p\left(x; p, k\right)= \mqty(x-1 \\ k-1) \exp \left(\log\left(1-p\right)x - k \log \left(\frac{1-p}{p}\right)\right) \end{equation}
\eta = \log \left(1-p\right) T\left(x\right) = x a\left(\eta\right) = k \log \left(\frac{1-p}{p}\right) b\left(x\right) = \mqty(x-1 \\ k-1) Geometric Distribution How many times do you have to do the trial to get at least one success The above, but k=1. This admits:
\begin{equation} p\left(x; p\right) = \left(1-p\right)^{x-1} p \end{equation}
Filling k=1 from above.