Why do Neural Nets Work Suddenly? Regularization see regularization We want to be able to manipulate our parameters so that our models learn better—for instance, we want our weights to be low:
\begin{equation} J_{L2}(\theta) = J_{reg}(\theta) + \lambda \sum_{k}^{} \theta^{2}_{k} \end{equation}
or good ‘ol dropout—“fetaure dependent regularization” Motivation classic view: regularization works to prevent overfitting when we have a lot of features NEW view with big models: regularization produces generalizable models when parameter count is big enough Dropout Dropout: prevents feature co-adaptation => results in good regularization Language Model See Language Model