[[
wikihub
]]
Search
⌘K
Explore
Activity
People
For Agents
Sign in
Explore
Activity
People
For Agents
Sign in
×
@jemoka / Jemoka Knowledge Base / raw/paper/moe_review/kbhmoereview_zhang_mixure_of_attention_heads.md
Suggest edit
Cancel
Submit suggestion
Title
Name
Note
--- title: "MOEReview Zhang: Mixure of Attention Heads" source: https://www.jemoka.com/posts/kbhmoereview_zhang_mixure_of_attention_heads/ --- Split \(Q\) projection and attention out projection into experts, with one router coordinating them. Better than MHA performanec.