YAML Metadata
Error:
"widget[0]" must be of type object
YAML Metadata
Error:
"model-index[0].results[0].metrics" is required
English Verdict Classifier
This model is a fine-tuned version of roberta-base on 2,500 deduplicated verdicts from Google Fact Check Tools API, translated into English with the Google Cloud Translation API. It achieves the following results on the evaluation set, being 1,000 such verdicts translated into English, but here including duplicates to represent the true distribution:
- Loss: 0.1290
- F1 Macro: 0.9171
- F1 Misinformation: 0.9896
- F1 Factual: 0.9890
- F1 Other: 0.7727
- Precision Macro: 0.8940
- Precision Misinformation: 0.9954
- Precision Factual: 0.9783
- Precision Other: 0.7083
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 4
- eval_batch_size: 4
- seed: 42
- gradient_accumulation_steps: 8
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 2500
- num_epochs: 1000
Training results
Training Loss | Epoch | Step | Validation Loss | F1 Macro | F1 Misinformation | F1 Factual | F1 Other | Precision Macro | Precision Misinformation | Precision Factual | Precision Other |
---|---|---|---|---|---|---|---|---|---|---|---|
1.1493 | 0.16 | 50 | 1.1040 | 0.0550 | 0.0 | 0.1650 | 0.0 | 0.0300 | 0.0 | 0.0899 | 0.0 |
1.0899 | 0.32 | 100 | 1.0765 | 0.0619 | 0.0203 | 0.1654 | 0.0 | 0.2301 | 0.6 | 0.0903 | 0.0 |
1.0136 | 0.48 | 150 | 1.0487 | 0.3102 | 0.9306 | 0.0 | 0.0 | 0.2900 | 0.8701 | 0.0 | 0.0 |
0.9868 | 0.64 | 200 | 1.0221 | 0.3102 | 0.9306 | 0.0 | 0.0 | 0.2900 | 0.8701 | 0.0 | 0.0 |
0.9599 | 0.8 | 250 | 0.9801 | 0.3102 | 0.9306 | 0.0 | 0.0 | 0.2900 | 0.8701 | 0.0 | 0.0 |
0.9554 | 0.96 | 300 | 0.9500 | 0.3102 | 0.9306 | 0.0 | 0.0 | 0.2900 | 0.8701 | 0.0 | 0.0 |
0.935 | 1.12 | 350 | 0.9071 | 0.3102 | 0.9306 | 0.0 | 0.0 | 0.2900 | 0.8701 | 0.0 | 0.0 |
0.948 | 1.28 | 400 | 0.8809 | 0.3102 | 0.9306 | 0.0 | 0.0 | 0.2900 | 0.8701 | 0.0 | 0.0 |
0.9344 | 1.44 | 450 | 0.8258 | 0.3102 | 0.9306 | 0.0 | 0.0 | 0.2900 | 0.8701 | 0.0 | 0.0 |
0.9182 | 1.6 | 500 | 0.7687 | 0.3102 | 0.9306 | 0.0 | 0.0 | 0.2900 | 0.8701 | 0.0 | 0.0 |
0.8942 | 1.76 | 550 | 0.5787 | 0.3102 | 0.9306 | 0.0 | 0.0 | 0.2900 | 0.8701 | 0.0 | 0.0 |
0.8932 | 1.92 | 600 | 0.4506 | 0.4043 | 0.9628 | 0.0 | 0.25 | 0.3777 | 0.9753 | 0.0 | 0.1579 |
0.7448 | 2.08 | 650 | 0.2884 | 0.5323 | 0.9650 | 0.3303 | 0.3017 | 0.7075 | 0.9810 | 0.9474 | 0.1942 |
0.6616 | 2.24 | 700 | 0.2162 | 0.8161 | 0.9710 | 0.9724 | 0.5051 | 0.7910 | 0.9824 | 0.9670 | 0.4237 |
0.575 | 2.4 | 750 | 0.1754 | 0.8305 | 0.9714 | 0.9780 | 0.5421 | 0.7961 | 0.9881 | 0.9674 | 0.4328 |
0.5246 | 2.56 | 800 | 0.1641 | 0.8102 | 0.9659 | 0.9175 | 0.5472 | 0.7614 | 0.9892 | 0.8558 | 0.4394 |
0.481 | 2.72 | 850 | 0.1399 | 0.8407 | 0.9756 | 0.9780 | 0.5686 | 0.8082 | 0.9894 | 0.9674 | 0.4677 |
0.4588 | 2.88 | 900 | 0.1212 | 0.8501 | 0.9786 | 0.9783 | 0.5934 | 0.8247 | 0.9871 | 0.9574 | 0.5294 |
0.4512 | 3.04 | 950 | 0.1388 | 0.8270 | 0.9702 | 0.9836 | 0.5273 | 0.7904 | 0.9893 | 0.9677 | 0.4143 |
0.3894 | 3.2 | 1000 | 0.1270 | 0.8411 | 0.9737 | 0.9836 | 0.5660 | 0.8043 | 0.9905 | 0.9677 | 0.4545 |
0.3772 | 3.36 | 1050 | 0.1267 | 0.8336 | 0.9732 | 0.9890 | 0.5385 | 0.8013 | 0.9882 | 0.9783 | 0.4375 |
0.3528 | 3.52 | 1100 | 0.1073 | 0.8546 | 0.9791 | 0.9890 | 0.5957 | 0.8284 | 0.9883 | 0.9783 | 0.5185 |
0.3694 | 3.68 | 1150 | 0.1120 | 0.8431 | 0.9786 | 0.9890 | 0.5618 | 0.8244 | 0.9849 | 0.9783 | 0.5102 |
0.3146 | 3.84 | 1200 | 0.1189 | 0.8325 | 0.9738 | 0.9836 | 0.54 | 0.8016 | 0.9870 | 0.9677 | 0.45 |
0.3038 | 4.01 | 1250 | 0.1041 | 0.8648 | 0.9815 | 0.9836 | 0.6292 | 0.8425 | 0.9884 | 0.9677 | 0.5714 |
0.2482 | 4.17 | 1300 | 0.1245 | 0.8588 | 0.9773 | 0.9836 | 0.6154 | 0.8202 | 0.9929 | 0.9677 | 0.5 |
0.2388 | 4.33 | 1350 | 0.1167 | 0.8701 | 0.9808 | 0.9836 | 0.6458 | 0.8377 | 0.9918 | 0.9677 | 0.5536 |
0.2593 | 4.49 | 1400 | 0.1215 | 0.8654 | 0.9790 | 0.9836 | 0.6337 | 0.8284 | 0.9929 | 0.9677 | 0.5246 |
0.239 | 4.65 | 1450 | 0.1057 | 0.8621 | 0.9803 | 0.9890 | 0.6170 | 0.8349 | 0.9895 | 0.9783 | 0.5370 |
0.2397 | 4.81 | 1500 | 0.1256 | 0.8544 | 0.9761 | 0.9890 | 0.5981 | 0.8162 | 0.9929 | 0.9783 | 0.4776 |
0.2238 | 4.97 | 1550 | 0.1189 | 0.8701 | 0.9802 | 0.9836 | 0.6465 | 0.8343 | 0.9929 | 0.9677 | 0.5424 |
0.1811 | 5.13 | 1600 | 0.1456 | 0.8438 | 0.9737 | 0.9836 | 0.5741 | 0.8051 | 0.9917 | 0.9677 | 0.4559 |
0.1615 | 5.29 | 1650 | 0.1076 | 0.8780 | 0.9838 | 0.9836 | 0.6667 | 0.8581 | 0.9895 | 0.9677 | 0.6170 |
0.1783 | 5.45 | 1700 | 0.1217 | 0.8869 | 0.9831 | 0.9836 | 0.6939 | 0.8497 | 0.9953 | 0.9677 | 0.5862 |
0.1615 | 5.61 | 1750 | 0.1305 | 0.8770 | 0.9808 | 0.9836 | 0.6667 | 0.8371 | 0.9953 | 0.9677 | 0.5484 |
0.155 | 5.77 | 1800 | 0.1218 | 0.8668 | 0.9821 | 0.9890 | 0.6292 | 0.8460 | 0.9884 | 0.9783 | 0.5714 |
0.167 | 5.93 | 1850 | 0.1091 | 0.8991 | 0.9873 | 0.9890 | 0.7209 | 0.8814 | 0.9919 | 0.9783 | 0.6739 |
0.1455 | 6.09 | 1900 | 0.1338 | 0.8535 | 0.9773 | 0.9890 | 0.5941 | 0.8202 | 0.9906 | 0.9783 | 0.4918 |
0.1301 | 6.25 | 1950 | 0.1321 | 0.8792 | 0.9820 | 0.9890 | 0.6667 | 0.8439 | 0.9941 | 0.9783 | 0.5593 |
0.1049 | 6.41 | 2000 | 0.1181 | 0.9031 | 0.9879 | 0.9834 | 0.7381 | 0.8911 | 0.9908 | 0.9780 | 0.7045 |
0.1403 | 6.57 | 2050 | 0.1432 | 0.8608 | 0.9779 | 0.9890 | 0.6154 | 0.8237 | 0.9929 | 0.9783 | 0.5 |
0.1178 | 6.73 | 2100 | 0.1443 | 0.8937 | 0.9844 | 0.9945 | 0.7021 | 0.8644 | 0.9930 | 0.9890 | 0.6111 |
0.1267 | 6.89 | 2150 | 0.1346 | 0.8494 | 0.9786 | 0.9890 | 0.5806 | 0.8249 | 0.9871 | 0.9783 | 0.5094 |
0.1043 | 7.05 | 2200 | 0.1494 | 0.8905 | 0.9832 | 0.9945 | 0.6939 | 0.8564 | 0.9941 | 0.9890 | 0.5862 |
0.0886 | 7.21 | 2250 | 0.1180 | 0.8946 | 0.9873 | 0.9890 | 0.7073 | 0.8861 | 0.9896 | 0.9783 | 0.6905 |
0.1183 | 7.37 | 2300 | 0.1777 | 0.8720 | 0.9790 | 0.9890 | 0.6481 | 0.8298 | 0.9964 | 0.9783 | 0.5147 |
0.0813 | 7.53 | 2350 | 0.1405 | 0.8912 | 0.9856 | 0.9836 | 0.7045 | 0.8685 | 0.9919 | 0.9677 | 0.6458 |
0.111 | 7.69 | 2400 | 0.1379 | 0.8874 | 0.9838 | 0.9836 | 0.6947 | 0.8540 | 0.9941 | 0.9677 | 0.6 |
0.1199 | 7.85 | 2450 | 0.1301 | 0.9080 | 0.9879 | 0.9890 | 0.7473 | 0.8801 | 0.9953 | 0.9783 | 0.6667 |
0.1054 | 8.01 | 2500 | 0.1478 | 0.8845 | 0.9838 | 0.9890 | 0.6809 | 0.8546 | 0.9930 | 0.9783 | 0.5926 |
0.105 | 8.17 | 2550 | 0.1333 | 0.9021 | 0.9879 | 0.9890 | 0.7294 | 0.8863 | 0.9919 | 0.9783 | 0.6889 |
0.09 | 8.33 | 2600 | 0.1555 | 0.8926 | 0.9855 | 0.9890 | 0.7033 | 0.8662 | 0.9930 | 0.9783 | 0.6275 |
0.0947 | 8.49 | 2650 | 0.1572 | 0.8831 | 0.9856 | 0.9890 | 0.6747 | 0.8726 | 0.9885 | 0.9783 | 0.6512 |
0.0784 | 8.65 | 2700 | 0.1477 | 0.8969 | 0.9873 | 0.9890 | 0.7143 | 0.8836 | 0.9908 | 0.9783 | 0.6818 |
0.0814 | 8.81 | 2750 | 0.1700 | 0.8932 | 0.9861 | 0.9890 | 0.7045 | 0.8720 | 0.9919 | 0.9783 | 0.6458 |
0.0962 | 8.97 | 2800 | 0.1290 | 0.9171 | 0.9896 | 0.9890 | 0.7727 | 0.8940 | 0.9954 | 0.9783 | 0.7083 |
0.0802 | 9.13 | 2850 | 0.1721 | 0.8796 | 0.9832 | 0.9890 | 0.6667 | 0.8517 | 0.9918 | 0.9783 | 0.5849 |
0.0844 | 9.29 | 2900 | 0.1516 | 0.9023 | 0.9867 | 0.9890 | 0.7312 | 0.8717 | 0.9953 | 0.9783 | 0.6415 |
0.0511 | 9.45 | 2950 | 0.1544 | 0.9062 | 0.9879 | 0.9890 | 0.7416 | 0.8820 | 0.9942 | 0.9783 | 0.6735 |
0.0751 | 9.61 | 3000 | 0.1748 | 0.8884 | 0.9832 | 0.9945 | 0.6875 | 0.8571 | 0.9930 | 0.9890 | 0.5893 |
0.0707 | 9.77 | 3050 | 0.1743 | 0.8721 | 0.9802 | 0.9890 | 0.6471 | 0.8349 | 0.9941 | 0.9783 | 0.5323 |
0.0951 | 9.93 | 3100 | 0.1660 | 0.8899 | 0.9850 | 0.9890 | 0.6957 | 0.8622 | 0.9930 | 0.9783 | 0.6154 |
0.0576 | 10.1 | 3150 | 0.2029 | 0.8613 | 0.9766 | 0.9890 | 0.6182 | 0.8197 | 0.9952 | 0.9783 | 0.4857 |
0.0727 | 10.26 | 3200 | 0.1709 | 0.8920 | 0.9849 | 0.9890 | 0.7021 | 0.8612 | 0.9942 | 0.9783 | 0.6111 |
0.0654 | 10.42 | 3250 | 0.1599 | 0.8999 | 0.9861 | 0.9945 | 0.7191 | 0.8780 | 0.9919 | 0.9890 | 0.6531 |
0.0553 | 10.58 | 3300 | 0.2091 | 0.8920 | 0.9849 | 0.9890 | 0.7021 | 0.8612 | 0.9942 | 0.9783 | 0.6111 |
Framework versions
- Transformers 4.11.3
- Pytorch 1.9.0+cu102
- Datasets 1.9.0
- Tokenizers 0.10.2
- Downloads last month
- 26
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social
visibility and check back later, or deploy to Inference Endpoints (dedicated)
instead.