Context

There are several recent research papers which explore various aspects of R1-Zero-like training on open base models like Qwen2.5-7B and Llama-3.1-8B:

• DAPO: An Open-Source LLM Reinforcement Learning System at Scale
• SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild
• Understanding R1-Zero-Like Training: A Critical Perspective
• Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model

These papers focus on mathematical reasoning (easy to verify) and do not always agree on the key factors needed for R1-Zero-like training. Since TRL now scales to large models, now is the time to train R1-Zero-like models with Open R1!

Main goal: reproduce / improve the performance of the DeepSeek-R1-Zero-Qwen-32B model that DeepSeek trained in the R1 tech report:

Although DeepSeek found that pure RL performed worse than simple SFT distillation, the DAPO paper shows that by tweaking the GRPO training process, one can actually surpass the distilled model (at least on math):

With that in mind, we will explore which subset of ideas in the above papers are sufficient to achieve comparable performance, starting first in math, then code and STEM.

We'll use this post and comments to track progress towards this goal - ideas and suggestions are more than welcome!

Setup

• Models: Qwen2.5-7B for ablations and Qwen2.5-32B for final runs
• Datasets: SynthLabsAI/Big-Math-RL-Verified and BytedTsinghua-SIA/DAPO-Math-17k for math. Code and other domains to be decided.

Links

• Code: I'll be running experiments from this draft PR of open-r1: https://github.com/huggingface/open-r1/pull/569
• Experiment logs: https://api.wandb.ai/links/huggingface/179dbkli
• Models and datasets: https://huggingface.co./collections/open-r1/open-r1-zero-67eba6a037505bbcb5157d07

Experiments to run

1. Train a baseline using "standard" parameters on Big-Math and DAPO-Math-17k to compare relative performance & learning dynamics
2. Measure effect on convergence with μ=2,4 (default is 1 in TRL)
3. Disable KL term with β=0
4. Clip higher with ε_low=0.2 and ε_high=0.28 (DAPO values)
5. Add soft overlong reward function from DAPO paper
6. Add overlong filter (mass loss of truncated completions)
7. DAPO (default) vs Dr. GRPO loss

Features to add to TRL

1. Overlong filter could be exposed as an arg like mask_truncated_completions in GRPOConfig
2. Add logging to measure average stopped length and clip ratio (SimpleRL-Zoo) Done: https://github.com/huggingface/trl/pull/3188

Features to add to Open R1

1. Add logging for pass@k accuracy (SimpleRL-Zero)
2. Add reasoning behaviours callback with LLM APIs to track backtracking and other behaviours during training (SimpleRL-Zero)