SmolLM2 1.7b Reward Model for RLVR Through Tulu 3!

SmolTulu Banner

SmolTulu-1.7b-RM is the reward model used to initialize the value function for SmolTulu-1.7b-Reinforced, which leverages AllenAI's Tulu 3 post-training pipeline for reinforcement learning with verifiable rewards (RLVR). This model was trained using the same preference datasets and methodology as outlined in the Tulu 3 paper, adapted for the smaller model size.

Evaluation

Evaluation results comparing SmolTulu-1.7b-RM against the Tulu 3 8b reward model on standard reward model benchmarks:

Metric SmolTulu-1.7b-RM Tulu 3 8b RM
RB Chat 94.13 96.27
RB Chat Hard 43.64 55.92
RB Safety 75.54 84.05
RB Reasoning 68.01 76.50
RB Average 72.43 81.34
UFB 73.17 77.34

While the 1.7B reward model shows lower performance compared to the larger 8B model as expected, it still demonstrates strong capabilities across different evaluation categories, particularly in chat quality assessment.

Usage

The reward model can be used with the transformers library:

from transformers import AutoModelForSequenceClassification, AutoTokenizer

checkpoint = "SultanR/SmolTulu-1.7b-RM"
device = "cuda" # for GPU usage or "cpu" for CPU usage

tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForSequenceClassification.from_pretrained(checkpoint).to(device)

# Example of computing reward for a completion
def get_reward(prompt, completion):
    inputs = tokenizer(prompt + completion, return_tensors="pt").to(device)
    reward = model(**inputs).logits[0].item()
    return reward

Training Details

The reward model was trained with the following settings:

  • Base model: SmolTulu-1.7b-Instruct
  • Mixed precision: bfloat16
  • Learning rate: 4e-5
  • Effective batch size: 4
  • Maximum sequence length: 2048 tokens
  • Maximum prompt length: 2048 tokens
  • Training epochs: 1
  • Training data: Tulu 3 8B preference mixture
  • Evaluation data: UltraFeedback (cleaned)
  • Gradient checkpointing enabled
  • DeepSpeed Zero-3 for memory optimization

Citation

@misc{alrashed2024smoltuluhigherlearningrate,
      title={SmolTulu: Higher Learning Rate to Batch Size Ratios Can Lead to Better Reasoning in SLMs}, 
      author={Sultan Alrashed},
      year={2024},
      eprint={2412.08347},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2412.08347}, 
}

The training methodology follows the Tulu 3 paper:

@article{lambert2024tulu3,
  title={TÜLU 3: Pushing Frontiers in Open Language Model Post-Training},
  author={Lambert, Nathan and Morrison, Jacob and Pyatkin, Valentina and others},
  year={2024},
  journal={arXiv preprint arXiv:2411.15124}
}
Downloads last month
32
Safetensors
Model size
1.71B params
Tensor type
BF16
·
Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Model tree for SultanR/SmolTulu-1.7b-RM

Finetuned
(2)
this model
Quantizations
3 models

Dataset used to train SultanR/SmolTulu-1.7b-RM

Collection including SultanR/SmolTulu-1.7b-RM