FineWeb: A Treasure Trove of Open Data!
Data remains the cornerstone of AI advancement, and the unveiling of FineWeb reinforces this notion. Boasting a staggering 15 trillion tokens of meticulously cleaned and deduplicated English web data, FineWeb emerges as a pivotal resource for the AI community. Derived from CommonCrawl and crafted with precision at Hugging Face, this open-source dataset promises unparalleled quality and breadth. Key highlights of FineWeb include: A vast repository of 15T tokens of English web data, meticulously curated and deduplicated. Demonstrated superiority over benchmark datasets like C4, Dolma-v1.6, The Pile, and SlimPajama in various tasks. Full dataset released under the permissive ODC-By 1.0 license, promoting accessibility and collaboration. Processing of 95 CommonCrawl dumps spanning from 2013 to 2024, ensuring comprehensive coverage. Shared code and processing setup to facilitate reproducibility and transparency. Commitment to continuous improvement and future releases, promising ongoing enhancements to FineWeb's utility and scope. With FineWeb, the AI community gains access to a wealth of high-quality data, propelling research and innovation in natural language processing and beyond.