@jemoka / Jemoka Knowledge Base › raw / paper / moe_review / kbhmoereview_zhang_mixure_of_attention_heads
Public View
Suggest
Download this page (.md) Download entire wiki (.zip)
Clone entire wiki

MOEReview Zhang: Mixure of Attention Heads

Split Q projection and attention out projection into experts, with one router coordinating them.
Better than MHA performanec.

Page updated