lora_final

This model is a fine-tuned version of meta-llama/Meta-Llama-3.1-8B-Instruct on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.2365

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 64
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.02
• num_epochs: 0.3

Training results

Training Loss	Epoch	Step	Validation Loss
1.3759	0.0065	1	1.3988
1.4834	0.0130	2	1.3942
1.255	0.0194	3	1.3840
1.2544	0.0259	4	1.3690
1.4768	0.0324	5	1.3500
1.293	0.0389	6	1.3319
1.2655	0.0453	7	1.3147
1.3413	0.0518	8	1.3033
1.2213	0.0583	9	1.2943
1.2445	0.0648	10	1.2885
1.2504	0.0713	11	1.2847
1.2629	0.0777	12	1.2826
1.3952	0.0842	13	1.2806
1.2751	0.0907	14	1.2773
1.283	0.0972	15	1.2744
1.2798	0.1036	16	1.2706
1.2756	0.1101	17	1.2669
1.1372	0.1166	18	1.2634
1.2445	0.1231	19	1.2600
1.3368	0.1296	20	1.2566
1.3386	0.1360	21	1.2534
1.1837	0.1425	22	1.2503
1.2373	0.1490	23	1.2482
1.1467	0.1555	24	1.2463
1.1782	0.1619	25	1.2447
1.1786	0.1684	26	1.2437
1.2298	0.1749	27	1.2430
1.2577	0.1814	28	1.2424
1.2813	0.1879	29	1.2415
1.2026	0.1943	30	1.2409
1.283	0.2008	31	1.2403
1.1598	0.2073	32	1.2396
1.2361	0.2138	33	1.2389
1.2846	0.2202	34	1.2386
1.3396	0.2267	35	1.2380
1.1525	0.2332	36	1.2377
1.1395	0.2397	37	1.2373
1.1794	0.2462	38	1.2370
1.332	0.2526	39	1.2369
1.2646	0.2591	40	1.2368
1.3159	0.2656	41	1.2367
1.1981	0.2721	42	1.2365
1.3205	0.2785	43	1.2365
1.1876	0.2850	44	1.2366
1.3331	0.2915	45	1.2365
1.1288	0.2980	46	1.2366
1.1745	0.3045	47	1.2366
1.1716	0.3109	48	1.2366
1.2834	0.3174	49	1.2365

Framework versions

• PEFT 0.12.0
• Transformers 4.44.2
• Pytorch 2.3.0+cu121
• Datasets 2.21.0
• Tokenizers 0.19.1