Jamba: Redefining LLM Performance
Discover Jamba, a cutting-edge hybrid SSM-Transformer Large Language Model (LLM) designed to revolutionize natural language processing tasks. With superior throughput gains compared to traditional Transformer-based models, Jamba consistently surpasses or matches leading models within its size class across a wide range of standard benchmarks. Representing the first-ever production-scale implementation of Mamba, Jamba unlocks new avenues for research and application in the field. While initial experimentation has yielded promising results, the potential for performance enhancements through future optimizations and explorations is vast. This model card delineates the specifications of the base version of Jamba, boasting 12 billion active parameters and a grand total of 52 billion parameters across all experts. Equipped with a 256K context length capacity, Jamba can accommodate up to 140K tokens on a single 80GB GPU, offering unparalleled efficiency and scalability.