Evaluating Machine Translation BLEU Compare machine vs. multiple-human reference translations. Uses N-Gram geometric mean—the actual n gram size isn’t super special. Original idea to have multiple reference translations—but maybe people to do this only one reference translation—good score in expectation. Limitations good translation can get a bad BLEU because it has low n gram overlap penalty to too-short system translations (i.e. translating only easy sentences isn’t a good metric) you really can’t get to 100 in BLEU because of variations in text attention Given a vector of values, a vector query, attention is a technique to compute a weighted sum of the values depending on the query. motivation machine translation problem—naive LSTM implementation has to stuff the entire information about a sentence into a single ending vector. improves performance more human like model for the MT solves the bottleneck problem helps solving Vanishing Gradients interoperability — provides soft phrase-level alignments, and know what is being translated implementation each step of the decoder, we will insert direct connections to the encoder to look at particular parts of the input source sequence dot every output state against every input state, softmax and add against the source sequence input. with encoder h_{j} and decoder s_{k}: dot product attention \begin{align} e_{i} = s^{T} h_{i} \end{align} limitation: LSTM latent layers are a little bit too busy—some of the information is not as useful as others—also forces everything to have dimension-to-dimension match multiplicative attention “learn a map from encoder vectors to decoder vectors—working out the right place to pay attention by learning it”

limitation: lots of parameters to learn in W for no good reason reduced-rank multiplicative attention \begin{equation} e_{i} = s^{T} Q R h_{i} = (Q s^{T})^{T} (R h_{i}) \end{equation} essentially, why don’t we project s and h down to smaller dimensions before the dot product is taken? this motivates also transformers additive attention \begin{equation} e_{i} = v^{T} \text{tanh} \left(W_1 h_{i} + W_{2} s\right) \end{equation} where v and W_{j} are learned.