Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuning
Paper
•
2503.07572
•
Published
•
25
HF CMU Collab
community
AI & ML interests
None defined yet.
Recent Activity
View all activity
hf-cmu-collab's activity
edbeeching
authored
a
paper
about 15 hours ago
lewtun
authored
a
paper
about 15 hours ago
CohenQu
authored
a
paper
about 21 hours ago
CohenQu
authored
2
papers
about 22 hours ago
Post
1415
Introducing OlympicCoder: a series of open reasoning models that can solve olympiad-level programming problems 🧑💻
- 7B open-r1/OlympicCoder-7B
- 32B open-r1/OlympicCoder-32B
We find that OlympicCoder models outperform Claude 3.7 Sonnet, as well as others over 100x larger 💪
Together with the models, we are releasing:
📊CodeForces-CoTs: new dataset of code problems from the most popular competitive coding platform, with R1 traces in C++ and Python open-r1/codeforces-cots
🏆 IOI'2024: a new benchmark of VERY hard programming problems where even frontier models struggle to match human performance open-r1/ioi
For links to the models and datasets, check out our latest progress report from Open R1: https://huggingface.co./blog/open-r1/update-3
- 7B open-r1/OlympicCoder-7B
- 32B open-r1/OlympicCoder-32B
We find that OlympicCoder models outperform Claude 3.7 Sonnet, as well as others over 100x larger 💪
Together with the models, we are releasing:
📊CodeForces-CoTs: new dataset of code problems from the most popular competitive coding platform, with R1 traces in C++ and Python open-r1/codeforces-cots
🏆 IOI'2024: a new benchmark of VERY hard programming problems where even frontier models struggle to match human performance open-r1/ioi
For links to the models and datasets, check out our latest progress report from Open R1: https://huggingface.co./blog/open-r1/update-3
- 1 reply
aviralku
authored
13
papers
30 days ago
Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models
Paper
•
2310.10639
•
Published
•
3
Vision-Language Models Provide Promptable Representations for Reinforcement Learning
Paper
•
2402.02651
•
Published
ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL
Paper
•
2402.19446
•
Published
Unfamiliar Finetuning Examples Control How Language Models Hallucinate
Paper
•
2403.05612
•
Published
•
3
Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data
Paper
•
2404.14367
•
Published
•
1
RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Fold
Paper
•
2406.14532
•
Published
Recursive Introspection: Teaching Language Model Agents How to Self-Improve
Paper
•
2407.18219
•
Published
•
3
Generative Verifiers: Reward Modeling as Next-Token Prediction
Paper
•
2408.15240
•
Published
•
13
Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning
Paper
•
2410.08146
•
Published
Steering Your Generalists: Improving Robotic Foundation Models via Value Guidance
Paper
•
2410.13816
•
Published
•
2
Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data
Paper
•
2412.07762
•
Published
Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Models
Paper
•
2412.15287
•
Published
Value-Based Deep RL Scales Predictably
Paper
•
2502.04327
•
Published
•
6
Post
4857
Introducing OpenR1-Math-220k!
open-r1/OpenR1-Math-220k
The community has been busy distilling DeepSeek-R1 from inference providers, but we decided to have a go at doing it ourselves from scratch 💪
What’s new compared to existing reasoning datasets?
♾ Based on AI-MO/NuminaMath-1.5: we focus on math reasoning traces and generate answers for problems in NuminaMath 1.5, an improved version of the popular NuminaMath-CoT dataset.
🐳 800k R1 reasoning traces: We generate two answers for 400k problems using DeepSeek R1. The filtered dataset contains 220k problems with correct reasoning traces.
📀 512 H100s running locally: Instead of relying on an API, we leverage vLLM and SGLang to run generations locally on our science cluster, generating 180k reasoning traces per day.
⏳ Automated filtering: We apply Math Verify to only retain problems with at least one correct answer. We also leverage Llama3.3-70B-Instruct as a judge to retrieve more correct examples (e.g for cases with malformed answers that can’t be verified with a rules-based parser)
📊 We match the performance of DeepSeek-Distill-Qwen-7B by finetuning Qwen-7B-Math-Instruct on our dataset.
🔎 Read our blog post for all the nitty gritty details: https://huggingface.co./blog/open-r1/update-2
open-r1/OpenR1-Math-220k
The community has been busy distilling DeepSeek-R1 from inference providers, but we decided to have a go at doing it ourselves from scratch 💪
What’s new compared to existing reasoning datasets?
♾ Based on AI-MO/NuminaMath-1.5: we focus on math reasoning traces and generate answers for problems in NuminaMath 1.5, an improved version of the popular NuminaMath-CoT dataset.
🐳 800k R1 reasoning traces: We generate two answers for 400k problems using DeepSeek R1. The filtered dataset contains 220k problems with correct reasoning traces.
📀 512 H100s running locally: Instead of relying on an API, we leverage vLLM and SGLang to run generations locally on our science cluster, generating 180k reasoning traces per day.
⏳ Automated filtering: We apply Math Verify to only retain problems with at least one correct answer. We also leverage Llama3.3-70B-Instruct as a judge to retrieve more correct examples (e.g for cases with malformed answers that can’t be verified with a rules-based parser)
📊 We match the performance of DeepSeek-Distill-Qwen-7B by finetuning Qwen-7B-Math-Instruct on our dataset.
🔎 Read our blog post for all the nitty gritty details: https://huggingface.co./blog/open-r1/update-2