- daVinci-LLM is a fully open pretraining research framework from GAIR-NLP that achieved an overall score of 51.72 with a 3-billion-parameter model, matching OLMo-3 7B despite having less than half the parameters.
- The team ran 200+ controlled ablations across 8 trillion training tokens and established that processing depth is a critical scaling dimension alongside data volume.
- The project releases complete data pipelines, training processes, and ablation results under its “Data Darwinism” framework, a systematic L0-L9 taxonomy for data processing.
What Happened
A team of 15 researchers led by Yiwei Qin, Yixiu Liu, and Tiantian Mi published “daVinci-LLM: Towards the Science of Pretraining,” a paper and open framework that systematically investigates how data processing decisions during pretraining affect downstream model capabilities. The paper, submitted to arXiv on March 28, 2026, comes from GAIR-NLP and represents one of the most comprehensive open investigations into pretraining dynamics published to date.
The project addresses what the authors call a “structural paradox”: pretraining is the phase that most determines a model’s capabilities, yet it receives the least rigorous scientific investigation due to its computational cost. Post-training techniques like RLHF and instruction tuning are bounded by whatever the pretraining phase established.
Why It Matters
Most frontier AI labs treat pretraining recipes as proprietary intellectual property. OpenAI, Anthropic, and Google have published limited details about their pretraining data mixtures and processing pipelines. The daVinci-LLM project breaks from this pattern by releasing everything: data pipelines, training configurations, and the results of every ablation experiment. This open approach allows other researchers to build on established findings rather than rediscovering them independently at substantial computational cost.
The practical impact is demonstrated by efficiency: daVinci-LLM-3B achieved an overall benchmark score of 51.72, matching OLMo-3 7B despite having less than half the parameters. On complex reasoning tasks, the gap was even wider. daVinci-LLM-3B scored 62.80 on MATH compared to OLMo-3’s 39.60, a 23-point advantage that suggests careful data processing quality can compensate significantly for raw model size when the right pretraining decisions are made.
Technical Details
The framework introduces what the authors call “Data Darwinism,” a principled L0-L9 taxonomy that categorizes data processing from basic filtering through synthesis. Through 200+ controlled ablations on a 3-billion-parameter model trained on 8 trillion tokens from random initialization, the team established several findings.
L3-level filtering yields modest gains on basic tasks, with a 3.4-point improvement on MBPP. L4 refinement provides substantial gains on complex reasoning, adding 7.0 points on MATH. L5 synthesis shows strong domain alignment but limited transfer to other tasks. General knowledge capabilities plateau at approximately 1 trillion tokens, while reasoning capabilities continue growing past 4 trillion tokens.
The training uses a two-stage adaptive curriculum that progresses from foundational capabilities to reasoning-focused enhancement. Different domains saturate at different rates, requiring tailored strategies that shift from proportion adjustments to format changes as training progresses.
Who’s Affected
The work is most relevant to researchers and organizations training their own foundation models. The finding that processing depth matters as much as data volume challenges the prevailing “more data, bigger model” approach and suggests that smaller, well-trained models can match larger ones on specific capability axes. For companies with limited GPU budgets, the results indicate that investing in data pipeline quality may yield better returns than scaling to larger parameter counts.
The full author list includes Muhang Xie, Zhen Huang, Weiye Si, Pengrui Lu, Siyuan Feng, Xia Wu, Liming Liu, Ye Luo, Jinlong Hou, Qipeng Guo, Yu Qiao, and Pengfei Liu. The project operates under what the authors describe as combining “industrial-scale resources with full research freedom to advance the science of pretraining.”
What’s Next
The team has released the complete exploration process on GitHub under GAIR-NLP/daVinci-LLM, including all data processing code, training scripts, and ablation logs. Future work will expand the scope of parameters investigated, including architectural variations and novel optimization techniques.
The key open question is whether the Data Darwinism findings scale to models with tens or hundreds of billions of parameters, where the computational cost of running 200+ ablations becomes prohibitive. A single ablation at the 70-billion-parameter scale would consume orders of magnitude more compute than the entire daVinci-LLM-3B study, making direct replication at frontier scale a challenge that may require collaborative effort across multiple research groups.
