xlm-roberta-peft-biobert-batch-size-16

This model is a fine-tuned version of FacebookAI/xlm-roberta-large on the biobert_json dataset. It achieves the following results on the evaluation set:

• Loss: 0.0864
• Precision: 0.9510
• Recall: 0.9696
• F1: 0.9602
• Accuracy: 0.9812

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0004
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Use paged_adamw_8bit with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• training_steps: 4282
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Precision	Recall	F1	Accuracy
2.3363	0.0327	20	1.1420	0.9730	0.0063	0.0126	0.6885
0.9396	0.0654	40	0.4876	0.7521	0.7189	0.7351	0.8868
0.4691	0.0980	60	0.2740	0.8289	0.8190	0.8239	0.9238
0.3834	0.1307	80	0.2312	0.7956	0.8816	0.8364	0.9349
0.2974	0.1634	100	0.3061	0.7659	0.9015	0.8282	0.9096
0.2836	0.1961	120	0.2028	0.8521	0.9040	0.8773	0.9449
0.2603	0.2288	140	0.1505	0.8951	0.9228	0.9087	0.9582
0.201	0.2614	160	0.1596	0.8936	0.8894	0.8915	0.9502
0.2219	0.2941	180	0.1522	0.8977	0.9239	0.9106	0.9583
0.199	0.3268	200	0.1571	0.8836	0.9115	0.8973	0.9566
0.1878	0.3595	220	0.1263	0.9048	0.9404	0.9223	0.9651
0.2207	0.3922	240	0.1415	0.8829	0.9594	0.9195	0.9613
0.1687	0.4248	260	0.1414	0.8970	0.9246	0.9106	0.9553
0.1934	0.4575	280	0.1131	0.9186	0.9506	0.9343	0.9695
0.1318	0.4902	300	0.1076	0.9269	0.9533	0.9399	0.9720
0.1523	0.5229	320	0.1116	0.9179	0.9365	0.9271	0.9681
0.1648	0.5556	340	0.1445	0.8873	0.9595	0.9220	0.9608
0.1386	0.5882	360	0.1313	0.8627	0.9127	0.8870	0.9557
0.1411	0.6209	380	0.0999	0.9240	0.9550	0.9392	0.9718
0.1266	0.6536	400	0.1008	0.9207	0.9575	0.9387	0.9717
0.1203	0.6863	420	0.1021	0.9231	0.9604	0.9414	0.9716
0.131	0.7190	440	0.0979	0.9312	0.9445	0.9378	0.9715
0.1242	0.7516	460	0.1121	0.9114	0.9673	0.9386	0.9704
0.1343	0.7843	480	0.0989	0.9257	0.9585	0.9418	0.9725
0.1413	0.8170	500	0.0921	0.9390	0.9568	0.9478	0.9756
0.1044	0.8497	520	0.1186	0.9069	0.9611	0.9332	0.9688
0.1239	0.8824	540	0.1302	0.9119	0.9616	0.9361	0.9653
0.1479	0.9150	560	0.1211	0.9094	0.9607	0.9344	0.9662
0.1513	0.9477	580	0.1021	0.9223	0.9579	0.9397	0.9721
0.1329	0.9804	600	0.1032	0.9356	0.9371	0.9363	0.9722
0.1439	1.0131	620	0.1014	0.9266	0.9656	0.9457	0.9742
0.1118	1.0458	640	0.1143	0.9098	0.9630	0.9357	0.9704
0.0979	1.0784	660	0.0920	0.9408	0.9614	0.9510	0.9768
0.0974	1.1111	680	0.0877	0.9401	0.9577	0.9488	0.9756
0.0932	1.1438	700	0.1008	0.9189	0.9740	0.9456	0.9739
0.1027	1.1765	720	0.1066	0.9220	0.9666	0.9437	0.9711
0.0992	1.2092	740	0.1110	0.9412	0.9457	0.9434	0.9744
0.1096	1.2418	760	0.1052	0.9316	0.9526	0.9420	0.9734
0.1492	1.2745	780	0.0952	0.9393	0.9522	0.9457	0.9748
0.0866	1.3072	800	0.0898	0.9445	0.9595	0.9520	0.9771
0.0975	1.3399	820	0.1042	0.9207	0.9588	0.9393	0.9735
0.1265	1.3725	840	0.0882	0.9319	0.9502	0.9410	0.9750
0.1086	1.4052	860	0.0939	0.9315	0.9619	0.9465	0.9747
0.0985	1.4379	880	0.0967	0.9282	0.9664	0.9469	0.9752
0.1223	1.4706	900	0.0924	0.9298	0.9676	0.9483	0.9746
0.0833	1.5033	920	0.0994	0.9406	0.9215	0.9310	0.9674
0.1043	1.5359	940	0.0958	0.9275	0.9680	0.9473	0.9743
0.137	1.5686	960	0.0831	0.9363	0.9644	0.9502	0.9769
0.1156	1.6013	980	0.0825	0.9434	0.9660	0.9546	0.9784
0.0963	1.6340	1000	0.0940	0.9314	0.9684	0.9495	0.9749
0.0669	1.6667	1020	0.1094	0.9291	0.9693	0.9487	0.9731
0.0984	1.6993	1040	0.0921	0.9331	0.9658	0.9492	0.9761
0.0831	1.7320	1060	0.1035	0.9279	0.9686	0.9478	0.9727
0.1059	1.7647	1080	0.0903	0.9372	0.9643	0.9506	0.9775
0.0971	1.7974	1100	0.1004	0.9304	0.9461	0.9382	0.9721
0.106	1.8301	1120	0.1155	0.9090	0.9734	0.9401	0.9712
0.0894	1.8627	1140	0.0910	0.9340	0.9698	0.9516	0.9773
0.0925	1.8954	1160	0.0949	0.9247	0.9577	0.9409	0.9726
0.0989	1.9281	1180	0.0916	0.9339	0.9683	0.9508	0.9769
0.0987	1.9608	1200	0.0918	0.9360	0.9681	0.9518	0.9768
0.0919	1.9935	1220	0.0948	0.9314	0.9727	0.9516	0.9767
0.0773	2.0261	1240	0.0825	0.9377	0.9622	0.9498	0.9775
0.0846	2.0588	1260	0.0900	0.9331	0.9676	0.9500	0.9769
0.0858	2.0915	1280	0.0886	0.9324	0.9706	0.9511	0.9770
0.067	2.1242	1300	0.0965	0.9246	0.9732	0.9483	0.9742
0.0691	2.1569	1320	0.1031	0.9444	0.9478	0.9461	0.9739
0.1099	2.1895	1340	0.0903	0.9339	0.9625	0.9480	0.9753
0.0613	2.2222	1360	0.0964	0.9322	0.9667	0.9491	0.9764
0.0758	2.2549	1380	0.0976	0.9306	0.9727	0.9512	0.9762
0.096	2.2876	1400	0.0982	0.9357	0.9598	0.9476	0.9747
0.0778	2.3203	1420	0.0800	0.9442	0.9685	0.9562	0.9798
0.0546	2.3529	1440	0.0838	0.9456	0.9655	0.9554	0.9796
0.0858	2.3856	1460	0.0979	0.9422	0.9582	0.9502	0.9746
0.0809	2.4183	1480	0.0945	0.9303	0.9675	0.9485	0.9739
0.1069	2.4510	1500	0.1209	0.8963	0.9704	0.9319	0.9669
0.0935	2.4837	1520	0.0900	0.9370	0.9690	0.9528	0.9769
0.0791	2.5163	1540	0.0803	0.9477	0.9687	0.9581	0.9804
0.0759	2.5490	1560	0.0822	0.9443	0.9715	0.9577	0.9799
0.0817	2.5817	1580	0.0801	0.9511	0.9697	0.9603	0.9812
0.0721	2.6144	1600	0.0906	0.9352	0.9756	0.9550	0.9778
0.0661	2.6471	1620	0.0883	0.9394	0.9696	0.9542	0.9774
0.0682	2.6797	1640	0.0855	0.95	0.9568	0.9534	0.9780
0.0779	2.7124	1660	0.0814	0.9463	0.9701	0.9580	0.9804
0.0696	2.7451	1680	0.0792	0.9406	0.9646	0.9525	0.9780
0.0579	2.7778	1700	0.0834	0.9413	0.9704	0.9556	0.9782
0.1837	2.8105	1720	0.0841	0.9393	0.9723	0.9555	0.9794
0.0993	2.8431	1740	0.0816	0.9422	0.9743	0.9580	0.9797
0.0645	2.8758	1760	0.0909	0.9367	0.9633	0.9498	0.9770
0.0771	2.9085	1780	0.0918	0.9277	0.9639	0.9455	0.9754
0.104	2.9412	1800	0.1006	0.9237	0.9636	0.9433	0.9731
0.0883	2.9739	1820	0.0864	0.9412	0.9678	0.9543	0.9779
0.0653	3.0065	1840	0.0939	0.9290	0.9701	0.9491	0.9758
0.1244	3.0392	1860	0.1186	0.9132	0.9614	0.9367	0.9703
0.0524	3.0719	1880	0.0928	0.9355	0.9683	0.9516	0.9771
0.0756	3.1046	1900	0.0897	0.9403	0.9687	0.9543	0.9781
0.0664	3.1373	1920	0.0875	0.9426	0.9676	0.9549	0.9777
0.0653	3.1699	1940	0.1014	0.9265	0.9741	0.9497	0.9749
0.0614	3.2026	1960	0.0892	0.9461	0.9657	0.9558	0.9782
0.0705	3.2353	1980	0.0979	0.9242	0.9626	0.9430	0.9737
0.064	3.2680	2000	0.0845	0.9491	0.9643	0.9567	0.9799
0.059	3.3007	2020	0.0818	0.9442	0.9718	0.9578	0.9803
0.0769	3.3333	2040	0.0788	0.9451	0.9682	0.9565	0.9801
0.0595	3.3660	2060	0.0811	0.9493	0.9698	0.9594	0.9801
0.0542	3.3987	2080	0.0906	0.9367	0.9698	0.9530	0.9777
0.0613	3.4314	2100	0.0819	0.9452	0.9657	0.9553	0.9786
0.0745	3.4641	2120	0.1052	0.9211	0.9691	0.9445	0.9725
0.0654	3.4967	2140	0.0822	0.9469	0.9680	0.9573	0.9794
0.0645	3.5294	2160	0.0798	0.9439	0.9683	0.9559	0.9792
0.0641	3.5621	2180	0.0830	0.9398	0.9725	0.9559	0.9774
0.0517	3.5948	2200	0.0795	0.9503	0.9700	0.9600	0.9801
0.0532	3.6275	2220	0.0877	0.9415	0.9714	0.9562	0.9787
0.0576	3.6601	2240	0.0829	0.9492	0.9670	0.9581	0.9797
0.0609	3.6928	2260	0.0822	0.9517	0.9662	0.9589	0.9796
0.0577	3.7255	2280	0.0850	0.9487	0.9728	0.9606	0.9804
0.0704	3.7582	2300	0.0829	0.9514	0.9724	0.9618	0.9815
0.052	3.7908	2320	0.0870	0.9447	0.9711	0.9577	0.9798
0.0794	3.8235	2340	0.0837	0.9493	0.9709	0.9600	0.9804
0.0693	3.8562	2360	0.0943	0.9428	0.9685	0.9555	0.9769
0.0727	3.8889	2380	0.0928	0.9370	0.9735	0.9549	0.9775
0.0729	3.9216	2400	0.0914	0.9352	0.9698	0.9522	0.9775
0.0805	3.9542	2420	0.0804	0.9452	0.9712	0.9580	0.9804
0.0737	3.9869	2440	0.0807	0.9450	0.9701	0.9574	0.9802
0.0556	4.0196	2460	0.0814	0.9484	0.9723	0.9602	0.9812
0.0533	4.0523	2480	0.0832	0.9424	0.9738	0.9578	0.9801
0.0435	4.0850	2500	0.0824	0.9453	0.9708	0.9579	0.9801
0.0356	4.1176	2520	0.0869	0.9414	0.9688	0.9549	0.9795
0.0565	4.1503	2540	0.0857	0.9536	0.9657	0.9596	0.9806
0.0466	4.1830	2560	0.0884	0.9509	0.9661	0.9584	0.9789
0.0692	4.2157	2580	0.0849	0.9435	0.9742	0.9586	0.9803
0.0388	4.2484	2600	0.0844	0.9412	0.9714	0.9560	0.9795
0.0471	4.2810	2620	0.0780	0.9511	0.9674	0.9592	0.9810
0.0666	4.3137	2640	0.0800	0.9511	0.9663	0.9586	0.9806
0.0574	4.3464	2660	0.0975	0.9253	0.9721	0.9481	0.9751
0.0597	4.3791	2680	0.0821	0.9489	0.9611	0.9550	0.9796
0.045	4.4118	2700	0.0837	0.9457	0.9691	0.9572	0.9799
0.0427	4.4444	2720	0.0871	0.9392	0.9710	0.9548	0.9792
0.0389	4.4771	2740	0.0816	0.9475	0.9712	0.9592	0.9810
0.0481	4.5098	2760	0.0865	0.9465	0.9708	0.9585	0.9805
0.0649	4.5425	2780	0.0827	0.9484	0.9637	0.9560	0.9795
0.0533	4.5752	2800	0.0850	0.9398	0.9698	0.9546	0.9792
0.0431	4.6078	2820	0.0776	0.9478	0.9752	0.9613	0.9815
0.0519	4.6405	2840	0.0813	0.9434	0.9731	0.9580	0.9797
0.0416	4.6732	2860	0.0804	0.9444	0.9673	0.9557	0.9793
0.0405	4.7059	2880	0.0826	0.9504	0.9701	0.9602	0.9799
0.0522	4.7386	2900	0.0833	0.9436	0.9708	0.9570	0.9799
0.0595	4.7712	2920	0.0835	0.9434	0.9733	0.9581	0.9799
0.0362	4.8039	2940	0.0809	0.9484	0.9719	0.9600	0.9805
0.0436	4.8366	2960	0.0835	0.9490	0.9695	0.9592	0.9802
0.059	4.8693	2980	0.0894	0.9363	0.9721	0.9539	0.9776
0.0395	4.9020	3000	0.0793	0.9475	0.9724	0.9598	0.9807
0.0413	4.9346	3020	0.0818	0.9476	0.9690	0.9582	0.9809
0.0666	4.9673	3040	0.0816	0.9449	0.9692	0.9569	0.9794
0.0452	5.0	3060	0.0811	0.9466	0.9739	0.9601	0.9806
0.0367	5.0327	3080	0.0811	0.9453	0.9712	0.9581	0.9807
0.0521	5.0654	3100	0.0749	0.9506	0.9710	0.9607	0.9817
0.0308	5.0980	3120	0.0852	0.9406	0.9728	0.9564	0.9794
0.0434	5.1307	3140	0.0804	0.9501	0.9712	0.9605	0.9808
0.0414	5.1634	3160	0.0799	0.9497	0.9714	0.9604	0.9806
0.0343	5.1961	3180	0.0826	0.9477	0.9712	0.9593	0.9803
0.0355	5.2288	3200	0.0882	0.9436	0.9693	0.9562	0.9792
0.0351	5.2614	3220	0.0869	0.9489	0.9704	0.9595	0.9805
0.0377	5.2941	3240	0.0864	0.9503	0.9669	0.9586	0.9802
0.0331	5.3268	3260	0.0865	0.9508	0.9698	0.9602	0.9809
0.0386	5.3595	3280	0.0850	0.9511	0.9692	0.9601	0.9807
0.0312	5.3922	3300	0.0856	0.9500	0.9665	0.9582	0.9801
0.0382	5.4248	3320	0.0844	0.9551	0.9625	0.9588	0.9807
0.0559	5.4575	3340	0.0841	0.9506	0.9693	0.9598	0.9811
0.0295	5.4902	3360	0.0872	0.9480	0.9671	0.9575	0.9800
0.0448	5.5229	3380	0.0832	0.9537	0.9676	0.9606	0.9814
0.0348	5.5556	3400	0.0874	0.9493	0.9686	0.9588	0.9801
0.0396	5.5882	3420	0.0893	0.9470	0.9677	0.9572	0.9793
0.0477	5.6209	3440	0.0924	0.9455	0.9677	0.9565	0.9782
0.0535	5.6536	3460	0.0824	0.9490	0.9695	0.9592	0.9804
0.0343	5.6863	3480	0.0823	0.9502	0.9690	0.9595	0.9808
0.0409	5.7190	3500	0.0845	0.9480	0.9695	0.9586	0.9805
0.0345	5.7516	3520	0.0847	0.9499	0.9686	0.9592	0.9807
0.0337	5.7843	3540	0.0863	0.9488	0.9707	0.9597	0.9806
0.0379	5.8170	3560	0.0824	0.9534	0.9694	0.9613	0.9814
0.0399	5.8497	3580	0.0820	0.9521	0.9722	0.9620	0.9812
0.038	5.8824	3600	0.0825	0.9497	0.9698	0.9596	0.9803
0.0287	5.9150	3620	0.0812	0.9530	0.9687	0.9608	0.9810
0.0333	5.9477	3640	0.0843	0.9496	0.9705	0.9599	0.9807
0.0448	5.9804	3660	0.0861	0.9485	0.9678	0.9580	0.9793
0.0312	6.0131	3680	0.0863	0.9449	0.9694	0.9570	0.9790
0.037	6.0458	3700	0.0832	0.9519	0.9688	0.9603	0.9811
0.0274	6.0784	3720	0.0841	0.9522	0.9680	0.9600	0.9808
0.0341	6.1111	3740	0.0843	0.9489	0.9704	0.9595	0.9806
0.0247	6.1438	3760	0.0872	0.9448	0.9674	0.9559	0.9799
0.0225	6.1765	3780	0.0883	0.9457	0.9656	0.9556	0.9798
0.0244	6.2092	3800	0.0912	0.9447	0.9679	0.9562	0.9791
0.0357	6.2418	3820	0.0898	0.9492	0.9685	0.9588	0.9805
0.0435	6.2745	3840	0.0895	0.9505	0.9690	0.9596	0.9808
0.0289	6.3072	3860	0.0903	0.9507	0.9684	0.9594	0.9807
0.0299	6.3399	3880	0.0907	0.9487	0.9674	0.9579	0.9803
0.0302	6.3725	3900	0.0898	0.9508	0.9681	0.9594	0.9806
0.0277	6.4052	3920	0.0899	0.9504	0.9680	0.9591	0.9808
0.0296	6.4379	3940	0.0895	0.9497	0.9670	0.9582	0.9808
0.0214	6.4706	3960	0.0909	0.9479	0.9691	0.9584	0.9808
0.0259	6.5033	3980	0.0895	0.9505	0.9702	0.9602	0.9813
0.0316	6.5359	4000	0.0880	0.9517	0.9694	0.9605	0.9812
0.0307	6.5686	4020	0.0873	0.9520	0.9700	0.9609	0.9814
0.0339	6.6013	4040	0.0875	0.9515	0.9699	0.9606	0.9812
0.0202	6.6340	4060	0.0876	0.9511	0.9700	0.9605	0.9812
0.0243	6.6667	4080	0.0877	0.9511	0.9690	0.9600	0.9811
0.0384	6.6993	4100	0.0870	0.9515	0.9690	0.9602	0.9812
0.0276	6.7320	4120	0.0871	0.9508	0.9690	0.9598	0.9811
0.0381	6.7647	4140	0.0867	0.9510	0.9690	0.9599	0.9811
0.035	6.7974	4160	0.0861	0.9521	0.9688	0.9604	0.9813
0.0277	6.8301	4180	0.0863	0.9515	0.9690	0.9602	0.9812
0.0219	6.8627	4200	0.0863	0.9515	0.9693	0.9603	0.9812
0.0232	6.8954	4220	0.0864	0.9509	0.9694	0.9601	0.9811
0.0266	6.9281	4240	0.0866	0.9508	0.9695	0.9601	0.9812
0.0269	6.9608	4260	0.0865	0.9510	0.9695	0.9602	0.9812
0.0237	6.9935	4280	0.0864	0.9510	0.9696	0.9602	0.9812

Framework versions

• PEFT 0.13.2
• Transformers 4.46.3
• Pytorch 2.5.1+cu121
• Datasets 3.2.0
• Tokenizers 0.20.3