Soft Mixture of Experts - An Efficient Sparse Transformer

[@christopherhornle4513]

Thank you! In the paper they also state that one slot per expert seems to be optimal.

[@flipflopsn]

Underrated channel, great video!

[@collin6526]

How does soft MoE reduce computation? If the tokens just get distributed to slots for all the experts, that implies all the experts are running, which prevents the performance gain. Is there some selection at inference time as for which should run?

[@zyzhang1130]

Same question

[@kumargaurav2170]

Great video👍🏻

[@aipapersacademy]

Thank you 😊

[@krishnakosaraju]

Hi, is MoE paradigm beneficial for Decoder only architectures? And are the advantages only for Visual Transformers or the LLM too?

[@postmodernismm]

Thank you 👍 Could you deal with NeRF and something like that ??

[@aipapersacademy]

Thank you 🙏 Noted and will consider this