sizhkhy/reliance

llm3br256-150steps

This model is a fine-tuned version of meta-llama/Llama-3.2-3B-Instruct on the reliance-oneshot-train dataset. It achieves the following results on the evaluation set:

• Loss: 0.0129

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 32
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• training_steps: 150

Training results

Training Loss	Epoch	Step	Validation Loss
0.0468	0.1786	5	0.0408
0.0274	0.3571	10	0.0312
0.0222	0.5357	15	0.0246
0.0253	0.7143	20	0.0221
0.0197	0.8929	25	0.0204
0.0169	1.0714	30	0.0190
0.0152	1.25	35	0.0177
0.0169	1.4286	40	0.0168
0.0123	1.6071	45	0.0164
0.0139	1.7857	50	0.0159
0.0105	1.9643	55	0.0153
0.0123	2.1429	60	0.0151
0.0105	2.3214	65	0.0149
0.0103	2.5	70	0.0144
0.0097	2.6786	75	0.0139
0.0113	2.8571	80	0.0137
0.0075	3.0357	85	0.0137
0.0084	3.2143	90	0.0137
0.0074	3.3929	95	0.0136
0.0079	3.5714	100	0.0133
0.0078	3.75	105	0.0132
0.0084	3.9286	110	0.0130
0.0081	4.1071	115	0.0130
0.007	4.2857	120	0.0131
0.0068	4.4643	125	0.0131
0.0069	4.6429	130	0.0131
0.0066	4.8214	135	0.0130
0.0057	5.0	140	0.0129
0.0057	5.1786	145	0.0130
0.0057	5.3571	150	0.0129

Framework versions

• PEFT 0.12.0
• Transformers 4.46.1
• Pytorch 2.4.0+cu121
• Datasets 3.1.0
• Tokenizers 0.20.3