Llama: GaLore Revolutionizes Fine-Tuning Efficiency
Introducing GaLore, a groundbreaking Memory Efficient Fine-tuning Technique designed specifically for "full-tuning" billion-parameter models such as Llama 2 7B on consumer-grade GPUs. In stark contrast to existing techniques like LoRA, GaLore achieves remarkable memory reduction by projecting optimizer states and gradients into lower dimensions, pushing the boundaries of efficiency in model training. In essence, GaLore enables the training of models with up to 7 billion parameters on consumer GPUs like the NVIDIA RTX 4090, unlocking unprecedented possibilities for leveraging powerful models on accessible hardware. With memory reductions of up to 82.5% for storing optimizer states, GaLore stands out as a game-changer in memory optimization techniques. Moreover, GaLore's compatibility with 8-bit optimizers further enhances memory efficiency, maximizing the utilization of available resources. Its superior performance, demonstrated through outperforming LoRA on benchmark tasks such as GLUE and Pretraining of Llama on C4, underscores its effectiveness in real-world applications. Integrated seamlessly into Hugging Face Transformers, GaLore offers users the flexibility to choose between galore_adamw or galore_adamw_8bit optimizers, facilitating easy adoption and integration into existing workflows. Additionally, layer-wise updates contribute to reducing the memory footprint even further, ensuring optimal performance without compromising efficiency. However, it's worth noting that GaLore currently supports Distributed Data Parallelism (DP) only and does not yet support Model Parallelism with Fully Sharded Data Parallelism (FSDP) or frameworks like Deepspeed. In summary, GaLore represents a significant advancement in memory-efficient fine-tuning techniques, empowering researchers and practitioners to push the boundaries of model training while optimizing resource utilization. As it continues to evolve and integrate with emerging technologies, GaLore promises to redefine the landscape of deep learning and model optimization.