A Generalized Linear Model is a model of data with the following properties: The model for P(y \mid x; \theta) should come from a exponential family (depending on what your distribution of y is—for real data, we pick Gaussian distribution, for binary data, we pick Bernoulli distribution, for counts, we use poisson distribution, \mathbb{R}^{+} we use gamma distribution or exponential distribution, and for distributions of distributions we use Beta Distribution or Dirichlet Distribution). \eta = \theta^{T}x, where \theta,x \in \mathbb{R}^{d} at test time… we want to output \mathbb{E}\left[y|x; \theta\right] so our predictor is written as h_{\theta}\left(x\right) = \mathbb{E}\left[y|x; \theta\right] at train time, we maximize log likelihood \max_{\theta} \sum_{i=1}^{n} \log P\left(y^{(i)} | \theta^{T}x^{(i)}\right) to update using gradient ascend, \theta_{j} = \theta_{j} + \alpha \sum_{i=1}^{n} \left(y^{(i)} - h_{\theta}\left(x^{(i)}\right)\right)x_{j}^{(i)} We also have two fancy names for things components of exponential distribution canonical response function \begin{equation} \mu = \mathbb{E}[y|\eta] = g\left(\eta\right) \end{equation} canonical link function \begin{equation} \eta = g^{-1}\left(\mu\right) \end{equation} expectation \begin{equation} g\left(\eta\right) = \pdv \eta a\left(\eta\right) \end{equation} example assume P(y|x; \theta) \sim \text{ExpFam}\left(\eta\right) , where \eta = \theta^{T}x (since Bernoulli distribution is in the exponential family). Recall P\left(y=1|x, \theta\right) = \phi = \frac{1}{1+e^{-\eta}} = \frac{1}{1+e^{-\theta^{\top}X}}