xlm-roberta-peft-biobert-batch-size-32

This model is a fine-tuned version of FacebookAI/xlm-roberta-large on the biobert_json dataset. It achieves the following results on the evaluation set:

• Loss: 0.0776
• Precision: 0.9512
• Recall: 0.9704
• F1: 0.9607
• Accuracy: 0.9817

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0004
• train_batch_size: 32
• eval_batch_size: 32
• seed: 42
• optimizer: Use paged_adamw_8bit with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• training_steps: 2141
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Precision	Recall	F1	Accuracy
2.6809	0.0654	20	1.5258	0.0	0.0	0.0	0.6871
1.5455	0.1307	40	1.4923	0.0	0.0	0.0	0.6871
1.5288	0.1961	60	1.4191	0.0	0.0	0.0	0.6871
1.1733	0.2614	80	0.7552	0.5851	0.5658	0.5753	0.8198
0.6291	0.3268	100	0.3531	0.8134	0.8265	0.8199	0.9073
0.394	0.3922	120	0.2030	0.8208	0.9050	0.8608	0.9414
0.2909	0.4575	140	0.1666	0.8948	0.8982	0.8965	0.9522
0.2146	0.5229	160	0.1519	0.8811	0.9427	0.9108	0.9593
0.207	0.5882	180	0.1350	0.8979	0.9406	0.9187	0.9622
0.1705	0.6536	200	0.1233	0.9056	0.9494	0.9270	0.9658
0.1458	0.7190	220	0.1174	0.9170	0.9369	0.9268	0.9649
0.1474	0.7843	240	0.1227	0.9011	0.9589	0.9291	0.9665
0.1373	0.8497	260	0.1150	0.9220	0.9508	0.9361	0.9689
0.1527	0.9150	280	0.1306	0.8912	0.9496	0.9195	0.9628
0.1537	0.9804	300	0.0986	0.9304	0.9471	0.9387	0.9726
0.1166	1.0458	320	0.0994	0.9295	0.9611	0.9450	0.9742
0.1003	1.1111	340	0.1032	0.9258	0.9577	0.9415	0.9712
0.1167	1.1765	360	0.0997	0.9277	0.9601	0.9436	0.9725
0.1148	1.2418	380	0.0889	0.9327	0.9631	0.9477	0.9758
0.125	1.3072	400	0.0882	0.9324	0.9533	0.9427	0.9740
0.1099	1.3725	420	0.0961	0.9233	0.9414	0.9323	0.9704
0.1025	1.4379	440	0.0841	0.9402	0.9624	0.9512	0.9772
0.0964	1.5033	460	0.0799	0.9447	0.9638	0.9542	0.9773
0.0975	1.5686	480	0.0879	0.9326	0.9667	0.9493	0.9751
0.1015	1.6340	500	0.0865	0.9371	0.9592	0.9480	0.9758
0.0828	1.6993	520	0.0987	0.9194	0.9728	0.9453	0.9731
0.0956	1.7647	540	0.0919	0.9263	0.9584	0.9421	0.9754
0.1052	1.8301	560	0.0925	0.9281	0.9751	0.9510	0.9755
0.0885	1.8954	580	0.0900	0.9306	0.9694	0.9496	0.9749
0.0921	1.9608	600	0.0897	0.9307	0.9663	0.9482	0.9750
0.0788	2.0261	620	0.0854	0.9429	0.9643	0.9534	0.9774
0.0804	2.0915	640	0.0826	0.9385	0.9694	0.9537	0.9788
0.0684	2.1569	660	0.0747	0.9498	0.9631	0.9564	0.9802
0.0786	2.2222	680	0.0872	0.9399	0.9693	0.9544	0.9783
0.0846	2.2876	700	0.0931	0.9269	0.9664	0.9462	0.9747
0.0693	2.3529	720	0.0818	0.9418	0.9721	0.9567	0.9795
0.0749	2.4183	740	0.0906	0.9321	0.9732	0.9522	0.9756
0.0986	2.4837	760	0.0762	0.9440	0.9677	0.9557	0.9783
0.0776	2.5490	780	0.0786	0.9462	0.9712	0.9586	0.9799
0.0719	2.6144	800	0.0778	0.9447	0.9711	0.9577	0.9798
0.0641	2.6797	820	0.0812	0.9490	0.9684	0.9586	0.9792
0.0737	2.7451	840	0.0819	0.9418	0.9695	0.9554	0.9778
0.0731	2.8105	860	0.0767	0.9438	0.9725	0.9579	0.9798
0.0716	2.8758	880	0.0761	0.9441	0.9719	0.9578	0.9798
0.0704	2.9412	900	0.0775	0.9447	0.9670	0.9557	0.9783
0.0706	3.0065	920	0.0738	0.9466	0.9738	0.9600	0.9814
0.0492	3.0719	940	0.0754	0.9448	0.9710	0.9577	0.9804
0.0601	3.1373	960	0.0873	0.9389	0.9729	0.9556	0.9776
0.053	3.2026	980	0.0737	0.9469	0.9684	0.9576	0.9805
0.0582	3.2680	1000	0.0833	0.9406	0.9689	0.9546	0.9784
0.0588	3.3333	1020	0.0728	0.9446	0.9703	0.9573	0.9806
0.0505	3.3987	1040	0.0818	0.9400	0.9765	0.9579	0.9798
0.0637	3.4641	1060	0.0864	0.9327	0.9681	0.9501	0.9766
0.0545	3.5294	1080	0.0792	0.9418	0.9660	0.9537	0.9782
0.0548	3.5948	1100	0.0822	0.9440	0.9730	0.9583	0.9785
0.0467	3.6601	1120	0.0742	0.9491	0.9725	0.9606	0.9816
0.0613	3.7255	1140	0.0731	0.9542	0.9727	0.9634	0.9822
0.0547	3.7908	1160	0.0766	0.9502	0.9725	0.9612	0.9816
0.0668	3.8562	1180	0.0822	0.9424	0.9690	0.9555	0.9786
0.066	3.9216	1200	0.0800	0.9464	0.9706	0.9583	0.9795
0.0662	3.9869	1220	0.0751	0.9460	0.9700	0.9578	0.9809
0.0461	4.0523	1240	0.0754	0.9491	0.9715	0.9602	0.9816
0.0382	4.1176	1260	0.0808	0.9411	0.9704	0.9555	0.9799
0.0447	4.1830	1280	0.0755	0.9521	0.9692	0.9606	0.9813
0.0489	4.2484	1300	0.0751	0.9503	0.9701	0.9601	0.9812
0.0486	4.3137	1320	0.0724	0.9552	0.9667	0.9609	0.9814
0.0448	4.3791	1340	0.0728	0.9509	0.9701	0.9604	0.9811
0.036	4.4444	1360	0.0764	0.9470	0.9708	0.9588	0.9812
0.0349	4.5098	1380	0.0745	0.9503	0.9666	0.9584	0.9812
0.0495	4.5752	1400	0.0743	0.9476	0.9687	0.9580	0.9814
0.048	4.6405	1420	0.0760	0.9452	0.9726	0.9587	0.9804
0.0395	4.7059	1440	0.0776	0.9430	0.9688	0.9558	0.9791
0.0553	4.7712	1460	0.0778	0.9436	0.9714	0.9573	0.9801
0.0389	4.8366	1480	0.0744	0.9527	0.9698	0.9612	0.9814
0.0496	4.9020	1500	0.0816	0.9432	0.9753	0.9590	0.9798
0.0501	4.9673	1520	0.0729	0.9485	0.9698	0.9590	0.9811
0.0394	5.0327	1540	0.0786	0.9444	0.9707	0.9574	0.9801
0.0344	5.0980	1560	0.0740	0.9503	0.9716	0.9608	0.9814
0.0393	5.1634	1580	0.0760	0.9509	0.9737	0.9622	0.9817
0.031	5.2288	1600	0.0829	0.9398	0.9691	0.9542	0.9789
0.0317	5.2941	1620	0.0768	0.9510	0.9664	0.9586	0.9802
0.0327	5.3595	1640	0.0796	0.9480	0.9716	0.9597	0.9809
0.0317	5.4248	1660	0.0746	0.9511	0.9683	0.9596	0.9815
0.0424	5.4902	1680	0.0776	0.9466	0.9701	0.9582	0.9808
0.0383	5.5556	1700	0.0736	0.9533	0.9706	0.9619	0.9823
0.041	5.6209	1720	0.0804	0.9448	0.9669	0.9557	0.9797
0.0361	5.6863	1740	0.0729	0.9524	0.9667	0.9595	0.9811
0.0362	5.7516	1760	0.0765	0.9449	0.9664	0.9555	0.9802
0.0293	5.8170	1780	0.0743	0.9512	0.9681	0.9596	0.9812
0.0391	5.8824	1800	0.0762	0.9491	0.9673	0.9581	0.9806
0.0282	5.9477	1820	0.0767	0.9507	0.9664	0.9585	0.9808
0.0375	6.0131	1840	0.0784	0.9502	0.9683	0.9592	0.9810
0.0307	6.0784	1860	0.0767	0.9517	0.9690	0.9603	0.9813
0.0279	6.1438	1880	0.0793	0.9461	0.9656	0.9557	0.9800
0.0247	6.2092	1900	0.0810	0.9458	0.9664	0.9560	0.9799
0.0346	6.2745	1920	0.0783	0.9496	0.9690	0.9592	0.9812
0.0286	6.3399	1940	0.0781	0.9489	0.9683	0.9585	0.9811
0.0258	6.4052	1960	0.0779	0.9486	0.9686	0.9585	0.9811
0.0268	6.4706	1980	0.0790	0.9482	0.9693	0.9586	0.9811
0.0274	6.5359	2000	0.0785	0.9500	0.9708	0.9603	0.9816
0.0325	6.6013	2020	0.0773	0.9502	0.9686	0.9593	0.9815
0.0243	6.6667	2040	0.0775	0.9506	0.9673	0.9589	0.9813
0.0278	6.7320	2060	0.0780	0.9497	0.9681	0.9588	0.9813
0.0358	6.7974	2080	0.0771	0.9515	0.9692	0.9603	0.9817
0.0241	6.8627	2100	0.0778	0.9514	0.9697	0.9604	0.9816
0.0289	6.9281	2120	0.0779	0.9511	0.9705	0.9607	0.9817
0.0233	6.9935	2140	0.0776	0.9512	0.9704	0.9607	0.9817

Framework versions

• PEFT 0.13.2
• Transformers 4.46.3
• Pytorch 2.5.1+cu121
• Datasets 3.2.0
• Tokenizers 0.20.3