Mixtral大模型

Mixtral

Mistral AI开源采用SMoE架构的Mixtral 8x7B模型，整体效能超越Llama 2 70B与GPT-3.5

官网： https://mistral.ai/news/mixtral-of-experts/

huggingface 网址： https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1

MoE：Mixture-Of-Experts

https://zhuanlan.zhihu.com/p/335024684

MoE PyTorch实现

https://github.com/lucidrains/mixture-of-experts