Snowflake's Arctic: First Open LLM
Introducing Arctic, the pioneering open Language Model (LLM) from Snowflake, boasting an impressive 480 billion parameters. Engineered with a 10 billion dense transformer model and a 128x3.66 billion Mixture of Experts (MoE) Multi-Layer Perceptron (MLP), Arctic is tailor-made for enterprise AI applications. Key Features of Arctic: Size and Scalability: With 480 billion parameters, Arctic is a formidable presence in the LLM landscape, with 17 billion parameters active during generation. Expertise and Efficiency: Arctic comprises 128 experts, with only two active during generation, ensuring optimized performance and resource utilization. Versions and Focus: Snowflake has released both Instruct and Base versions of Arctic, catering to diverse enterprise tasks such as coding, SQL, reasoning, and following tasks. Open Source and Accessibility: Arctic is released under the Apache 2.0 license, affirming Snowflake's commitment to open collaboration and innovation. It is also available on the Hugging Face platform, enhancing accessibility for developers and researchers. Resource Requirements: Arctic demands substantial computational resources, requiring approximately 900GB of memory in fp16 format and 240GB in int4 format. Training Methodology: Arctic is trained using DeepSpeed-MoE, a training framework optimized for models with a Mixture of Experts architecture. Snowflake's Arctic represents a significant milestone in the realm of open LLMs, offering unparalleled scale, performance, and versatility for enterprise AI applications. With its release, Snowflake empowers developers and researchers to leverage state-of-the-art language models to tackle complex tasks and drive innovation in various domains.