--- license: mit model-index: - name: KaLM-Embedding results: - dataset: config: en-ext name: MTEB AmazonCounterfactualClassification (en-ext) revision: e8379541af4e31359cca9fbcf4b00f2671dba205 split: test type: mteb/amazon_counterfactual metrics: - type: accuracy value: 94.68515742128936 - type: ap value: 61.63091951041035 - type: ap_weighted value: 61.63091951041035 - type: f1 value: 87.0707144798333 - type: f1_weighted value: 94.92507672044346 - type: main_score value: 94.68515742128936 task: type: Classification - dataset: config: en name: MTEB AmazonCounterfactualClassification (en) revision: e8379541af4e31359cca9fbcf4b00f2671dba205 split: test type: mteb/amazon_counterfactual metrics: - type: accuracy value: 91.73134328358209 - type: ap value: 68.35026743834399 - type: ap_weighted value: 68.35026743834399 - type: f1 value: 87.90511074439425 - type: f1_weighted value: 92.04737679749206 - type: main_score value: 91.73134328358209 task: type: Classification - dataset: config: default name: MTEB AmazonPolarityClassification revision: e2d317d38cd51312af73b3d32a06d1a08b442046 split: test type: mteb/amazon_polarity metrics: - type: accuracy value: 96.55665 - type: ap value: 95.07549068622335 - type: ap_weighted value: 95.07549068622335 - type: f1 value: 96.55609959481485 - type: f1_weighted value: 96.55609959481488 - type: main_score value: 96.55665 task: type: Classification - dataset: config: en name: MTEB AmazonReviewsClassification (en) revision: 1399c76144fd37290681b995c656ef9b2e06e26d split: test type: mteb/amazon_reviews_multi metrics: - type: accuracy value: 61.42000000000001 - type: f1 value: 59.435009609195 - type: f1_weighted value: 59.435009609195 - type: main_score value: 61.42000000000001 task: type: Classification - dataset: config: default name: MTEB ArguAna revision: c22ab2a51041ffd869aaddef7af8d8215647e41a split: test type: mteb/arguana metrics: - type: main_score value: 58.626999999999995 - type: map_at_1 value: 33.073 - type: map_at_10 value: 49.632 - type: map_at_100 value: 50.33800000000001 - type: map_at_1000 value: 50.341 - type: map_at_20 value: 50.224000000000004 - type: map_at_3 value: 44.832 - type: map_at_5 value: 47.620000000000005 - type: mrr_at_1 value: 33.57041251778094 - type: mrr_at_10 value: 49.811380929802155 - type: mrr_at_100 value: 50.51757318808999 - type: mrr_at_1000 value: 50.52099783252713 - type: mrr_at_20 value: 50.40338101381789 - type: mrr_at_3 value: 45.00948316737797 - type: mrr_at_5 value: 47.77619724988161 - type: nauc_map_at_1000_diff1 value: 13.48102412913682 - type: nauc_map_at_1000_max value: -8.628057456116096 - type: nauc_map_at_1000_std value: -11.093713781480051 - type: nauc_map_at_100_diff1 value: 13.48687495587519 - type: nauc_map_at_100_max value: -8.618385685478875 - type: nauc_map_at_100_std value: -11.08784320605541 - type: nauc_map_at_10_diff1 value: 13.356168065508726 - type: nauc_map_at_10_max value: -8.497284129177803 - type: nauc_map_at_10_std value: -11.179349250075992 - type: nauc_map_at_1_diff1 value: 16.878392009779215 - type: nauc_map_at_1_max value: -10.140489051180705 - type: nauc_map_at_1_std value: -11.39244095065129 - type: nauc_map_at_20_diff1 value: 13.416069003092446 - type: nauc_map_at_20_max value: -8.581890449222536 - type: nauc_map_at_20_std value: -11.121016775402621 - type: nauc_map_at_3_diff1 value: 13.002710999935143 - type: nauc_map_at_3_max value: -9.061445259612642 - type: nauc_map_at_3_std value: -10.95371941334804 - type: nauc_map_at_5_diff1 value: 13.116608774616614 - type: nauc_map_at_5_max value: -9.031302466539435 - type: nauc_map_at_5_std value: -11.482820479074665 - type: nauc_mrr_at_1000_diff1 value: 12.009148181833913 - type: nauc_mrr_at_1000_max value: -9.133318660578208 - type: nauc_mrr_at_1000_std value: -11.090084522113825 - type: nauc_mrr_at_100_diff1 value: 12.015156727781342 - type: nauc_mrr_at_100_max value: -9.123605928993614 - type: nauc_mrr_at_100_std value: -11.084224457426195 - type: nauc_mrr_at_10_diff1 value: 11.928178469104141 - type: nauc_mrr_at_10_max value: -8.980239557009549 - type: nauc_mrr_at_10_std value: -11.174796827155468 - type: nauc_mrr_at_1_diff1 value: 15.42177954114532 - type: nauc_mrr_at_1_max value: -9.874364097911553 - type: nauc_mrr_at_1_std value: -11.176922083255894 - type: nauc_mrr_at_20_diff1 value: 11.950454910420659 - type: nauc_mrr_at_20_max value: -9.084711198849204 - type: nauc_mrr_at_20_std value: -11.117380723428798 - type: nauc_mrr_at_3_diff1 value: 11.669506159876851 - type: nauc_mrr_at_3_max value: -9.513795812047656 - type: nauc_mrr_at_3_std value: -10.945232338570069 - type: nauc_mrr_at_5_diff1 value: 11.587286193937736 - type: nauc_mrr_at_5_max value: -9.692616828656766 - type: nauc_mrr_at_5_std value: -11.52680193639791 - type: nauc_ndcg_at_1000_diff1 value: 13.115734924812534 - type: nauc_ndcg_at_1000_max value: -8.124440036807906 - type: nauc_ndcg_at_1000_std value: -10.823569251091254 - type: nauc_ndcg_at_100_diff1 value: 13.267338628811087 - type: nauc_ndcg_at_100_max value: -7.873039455281453 - type: nauc_ndcg_at_100_std value: -10.68251412725148 - type: nauc_ndcg_at_10_diff1 value: 12.46318225000743 - type: nauc_ndcg_at_10_max value: -7.328112147624243 - type: nauc_ndcg_at_10_std value: -11.267788232147327 - type: nauc_ndcg_at_1_diff1 value: 16.878392009779215 - type: nauc_ndcg_at_1_max value: -10.140489051180705 - type: nauc_ndcg_at_1_std value: -11.39244095065129 - type: nauc_ndcg_at_20_diff1 value: 12.777688953238583 - type: nauc_ndcg_at_20_max value: -7.553266861502126 - type: nauc_ndcg_at_20_std value: -10.926546866836999 - type: nauc_ndcg_at_3_diff1 value: 11.895421239753933 - type: nauc_ndcg_at_3_max value: -8.901090243657334 - type: nauc_ndcg_at_3_std value: -11.116058814242175 - type: nauc_ndcg_at_5_diff1 value: 11.98520372095743 - type: nauc_ndcg_at_5_max value: -8.780043333282341 - type: nauc_ndcg_at_5_std value: -11.985932726758405 - type: nauc_precision_at_1000_diff1 value: -8.254265037555717 - type: nauc_precision_at_1000_max value: 34.439879624882145 - type: nauc_precision_at_1000_std value: 86.59852334082694 - type: nauc_precision_at_100_diff1 value: 35.055902247981955 - type: nauc_precision_at_100_max value: 60.87922050664748 - type: nauc_precision_at_100_std value: 58.03454945999707 - type: nauc_precision_at_10_diff1 value: 6.653457872096913 - type: nauc_precision_at_10_max value: 2.2891313787635483 - type: nauc_precision_at_10_std value: -11.638523230893009 - type: nauc_precision_at_1_diff1 value: 16.878392009779215 - type: nauc_precision_at_1_max value: -10.140489051180705 - type: nauc_precision_at_1_std value: -11.39244095065129 - type: nauc_precision_at_20_diff1 value: 4.779328896089478 - type: nauc_precision_at_20_max value: 14.224776184784957 - type: nauc_precision_at_20_std value: -4.21463291230828 - type: nauc_precision_at_3_diff1 value: 8.514820657150233 - type: nauc_precision_at_3_max value: -8.469060860049108 - type: nauc_precision_at_3_std value: -11.7235889812276 - type: nauc_precision_at_5_diff1 value: 7.710375928363608 - type: nauc_precision_at_5_max value: -7.833484980907636 - type: nauc_precision_at_5_std value: -14.090406741382132 - type: nauc_recall_at_1000_diff1 value: -8.254265037555465 - type: nauc_recall_at_1000_max value: 34.43987962487738 - type: nauc_recall_at_1000_std value: 86.59852334082566 - type: nauc_recall_at_100_diff1 value: 35.05590224798152 - type: nauc_recall_at_100_max value: 60.87922050664376 - type: nauc_recall_at_100_std value: 58.03454945999553 - type: nauc_recall_at_10_diff1 value: 6.653457872096784 - type: nauc_recall_at_10_max value: 2.2891313787632592 - type: nauc_recall_at_10_std value: -11.638523230893094 - type: nauc_recall_at_1_diff1 value: 16.878392009779215 - type: nauc_recall_at_1_max value: -10.140489051180705 - type: nauc_recall_at_1_std value: -11.39244095065129 - type: nauc_recall_at_20_diff1 value: 4.77932889608973 - type: nauc_recall_at_20_max value: 14.224776184784938 - type: nauc_recall_at_20_std value: -4.2146329123081125 - type: nauc_recall_at_3_diff1 value: 8.514820657150297 - type: nauc_recall_at_3_max value: -8.469060860049037 - type: nauc_recall_at_3_std value: -11.723588981227591 - type: nauc_recall_at_5_diff1 value: 7.710375928363697 - type: nauc_recall_at_5_max value: -7.833484980907617 - type: nauc_recall_at_5_std value: -14.090406741382106 - type: ndcg_at_1 value: 33.073 - type: ndcg_at_10 value: 58.626999999999995 - type: ndcg_at_100 value: 61.449 - type: ndcg_at_1000 value: 61.516999999999996 - type: ndcg_at_20 value: 60.712999999999994 - type: ndcg_at_3 value: 48.730000000000004 - type: ndcg_at_5 value: 53.791 - type: precision_at_1 value: 33.073 - type: precision_at_10 value: 8.727 - type: precision_at_100 value: 0.991 - type: precision_at_1000 value: 0.1 - type: precision_at_20 value: 4.769 - type: precision_at_3 value: 20.009 - type: precision_at_5 value: 14.481 - type: recall_at_1 value: 33.073 - type: recall_at_10 value: 87.26899999999999 - type: recall_at_100 value: 99.14699999999999 - type: recall_at_1000 value: 99.644 - type: recall_at_20 value: 95.377 - type: recall_at_3 value: 60.028000000000006 - type: recall_at_5 value: 72.404 task: type: Retrieval - dataset: config: default name: MTEB ArxivClusteringP2P revision: a122ad7f3f0291bf49cc6f4d32aa80929df69d5d split: test type: mteb/arxiv-clustering-p2p metrics: - type: main_score value: 49.682199850672866 - type: v_measure value: 49.682199850672866 - type: v_measure_std value: 14.233482745120106 task: type: Clustering - dataset: config: default name: MTEB ArxivClusteringS2S revision: f910caf1a6075f7329cdf8c1a6135696f37dbd53 split: test type: mteb/arxiv-clustering-s2s metrics: - type: main_score value: 42.209464702274715 - type: v_measure value: 42.209464702274715 - type: v_measure_std value: 14.710188553304171 task: type: Clustering - dataset: config: default name: MTEB AskUbuntuDupQuestions revision: 2000358ca161889fa9c082cb41daa8dcfb161a54 split: test type: mteb/askubuntudupquestions-reranking metrics: - type: main_score value: 60.34670620892353 - type: map value: 60.34670620892353 - type: mrr value: 74.21360418590336 - type: nAUC_map_diff1 value: 11.911350612613562 - type: nAUC_map_max value: 16.783478049340925 - type: nAUC_map_std value: 16.428460426761752 - type: nAUC_mrr_diff1 value: 14.989972807815782 - type: nAUC_mrr_max value: 25.277551101334105 - type: nAUC_mrr_std value: 16.506353229551408 task: type: Reranking - dataset: config: default name: MTEB BIOSSES revision: d3fb88f8f02e40887cd149695127462bbcf29b4a split: test type: mteb/biosses-sts metrics: - type: cosine_pearson value: 88.03646117514994 - type: cosine_spearman value: 86.1423242570783 - type: euclidean_pearson value: 87.25661641211398 - type: euclidean_spearman value: 86.1423242570783 - type: main_score value: 86.1423242570783 - type: manhattan_pearson value: 86.75341029385434 - type: manhattan_spearman value: 85.43447680299268 - type: pearson value: 88.03646117514994 - type: spearman value: 86.1423242570783 task: type: STS - dataset: config: default name: MTEB Banking77Classification revision: 0fd18e25b25c072e09e0d92ab615fda904d66300 split: test type: mteb/banking77 metrics: - type: accuracy value: 84.54220779220778 - type: f1 value: 83.93093828135092 - type: f1_weighted value: 83.9309382813509 - type: main_score value: 84.54220779220778 task: type: Classification - dataset: config: default name: MTEB BiorxivClusteringP2P revision: 65b79d1d13f80053f67aca9498d9402c2d9f1f40 split: test type: mteb/biorxiv-clustering-p2p metrics: - type: main_score value: 43.84044619752363 - type: v_measure value: 43.84044619752363 - type: v_measure_std value: 0.4618669625794188 task: type: Clustering - dataset: config: default name: MTEB BiorxivClusteringS2S revision: 258694dd0231531bc1fd9de6ceb52a0853c6d908 split: test type: mteb/biorxiv-clustering-s2s metrics: - type: main_score value: 37.31116089915042 - type: v_measure value: 37.31116089915042 - type: v_measure_std value: 0.5977139350402765 task: type: Clustering - dataset: config: default name: MTEB CQADupstackAndroidRetrieval revision: f46a197baaae43b4f621051089b82a364682dfeb split: test type: mteb/cqadupstack-android metrics: - type: main_score value: 52.39 - type: map_at_1 value: 34.169 - type: map_at_10 value: 45.942 - type: map_at_100 value: 47.404 - type: map_at_1000 value: 47.514 - type: map_at_20 value: 46.778 - type: map_at_3 value: 42.336 - type: map_at_5 value: 44.415 - type: mrr_at_1 value: 40.772532188841204 - type: mrr_at_10 value: 51.53978472647998 - type: mrr_at_100 value: 52.151143552536816 - type: mrr_at_1000 value: 52.187870164237204 - type: mrr_at_20 value: 51.901680031986686 - type: mrr_at_3 value: 48.90319504053407 - type: mrr_at_5 value: 50.448259418216466 - type: nauc_map_at_1000_diff1 value: 56.78205111804614 - type: nauc_map_at_1000_max value: 36.375389301093755 - type: nauc_map_at_1000_std value: -5.9566477904567625 - type: nauc_map_at_100_diff1 value: 56.78944020915762 - type: nauc_map_at_100_max value: 36.39973376657996 - type: nauc_map_at_100_std value: -5.902276438746319 - type: nauc_map_at_10_diff1 value: 57.01302040488429 - type: nauc_map_at_10_max value: 35.73327053443045 - type: nauc_map_at_10_std value: -6.935563186512875 - type: nauc_map_at_1_diff1 value: 62.1943454407426 - type: nauc_map_at_1_max value: 31.106713330762027 - type: nauc_map_at_1_std value: -9.349274581003204 - type: nauc_map_at_20_diff1 value: 56.96527745781459 - type: nauc_map_at_20_max value: 36.1099391825394 - type: nauc_map_at_20_std value: -6.148724378898889 - type: nauc_map_at_3_diff1 value: 57.97922886237535 - type: nauc_map_at_3_max value: 34.98388578365086 - type: nauc_map_at_3_std value: -7.868364460415703 - type: nauc_map_at_5_diff1 value: 57.19907689819566 - type: nauc_map_at_5_max value: 35.070877663024426 - type: nauc_map_at_5_std value: -7.643819808793027 - type: nauc_mrr_at_1000_diff1 value: 53.9572914102343 - type: nauc_mrr_at_1000_max value: 36.372803215838125 - type: nauc_mrr_at_1000_std value: -6.847011087249551 - type: nauc_mrr_at_100_diff1 value: 53.95057299284283 - type: nauc_mrr_at_100_max value: 36.374095119115765 - type: nauc_mrr_at_100_std value: -6.8350087107843835 - type: nauc_mrr_at_10_diff1 value: 53.77540722618102 - type: nauc_mrr_at_10_max value: 36.34439189912393 - type: nauc_mrr_at_10_std value: -7.189115725803416 - type: nauc_mrr_at_1_diff1 value: 58.516667270098445 - type: nauc_mrr_at_1_max value: 35.99253476386456 - type: nauc_mrr_at_1_std value: -8.950540274950663 - type: nauc_mrr_at_20_diff1 value: 53.92501674276654 - type: nauc_mrr_at_20_max value: 36.31325682241623 - type: nauc_mrr_at_20_std value: -6.842193524470379 - type: nauc_mrr_at_3_diff1 value: 54.30587263243055 - type: nauc_mrr_at_3_max value: 36.67164288224585 - type: nauc_mrr_at_3_std value: -6.720909588714866 - type: nauc_mrr_at_5_diff1 value: 53.80277965232069 - type: nauc_mrr_at_5_max value: 36.026610354561726 - type: nauc_mrr_at_5_std value: -7.405548968254587 - type: nauc_ndcg_at_1000_diff1 value: 54.32192669123723 - type: nauc_ndcg_at_1000_max value: 37.06057531293069 - type: nauc_ndcg_at_1000_std value: -3.858161870894338 - type: nauc_ndcg_at_100_diff1 value: 54.19634435723086 - type: nauc_ndcg_at_100_max value: 37.47554574823893 - type: nauc_ndcg_at_100_std value: -2.782934190456467 - type: nauc_ndcg_at_10_diff1 value: 53.97003966265566 - type: nauc_ndcg_at_10_max value: 36.15287682329226 - type: nauc_ndcg_at_10_std value: -5.669539508175427 - type: nauc_ndcg_at_1_diff1 value: 58.516667270098445 - type: nauc_ndcg_at_1_max value: 35.99253476386456 - type: nauc_ndcg_at_1_std value: -8.950540274950663 - type: nauc_ndcg_at_20_diff1 value: 54.34922000376914 - type: nauc_ndcg_at_20_max value: 36.25023728047867 - type: nauc_ndcg_at_20_std value: -3.810948512343384 - type: nauc_ndcg_at_3_diff1 value: 54.83758663042578 - type: nauc_ndcg_at_3_max value: 36.85494351712768 - type: nauc_ndcg_at_3_std value: -5.525443708811114 - type: nauc_ndcg_at_5_diff1 value: 54.191751120186304 - type: nauc_ndcg_at_5_max value: 35.775634699712214 - type: nauc_ndcg_at_5_std value: -6.139092418343521 - type: nauc_precision_at_1000_diff1 value: -25.057017986568535 - type: nauc_precision_at_1000_max value: -6.253611962247419 - type: nauc_precision_at_1000_std value: -3.576821263647792 - type: nauc_precision_at_100_diff1 value: -15.890618283335048 - type: nauc_precision_at_100_max value: 9.2689967407046 - type: nauc_precision_at_100_std value: 9.941022837167951 - type: nauc_precision_at_10_diff1 value: 6.515251734960673 - type: nauc_precision_at_10_max value: 26.00954176926215 - type: nauc_precision_at_10_std value: 6.18066523804505 - type: nauc_precision_at_1_diff1 value: 58.516667270098445 - type: nauc_precision_at_1_max value: 35.99253476386456 - type: nauc_precision_at_1_std value: -8.950540274950663 - type: nauc_precision_at_20_diff1 value: -2.207402052545381 - type: nauc_precision_at_20_max value: 19.722097586131714 - type: nauc_precision_at_20_std value: 11.781955236419435 - type: nauc_precision_at_3_diff1 value: 28.669702552163834 - type: nauc_precision_at_3_max value: 36.374241470537925 - type: nauc_precision_at_3_std value: 0.5810060004219123 - type: nauc_precision_at_5_diff1 value: 18.454459967860064 - type: nauc_precision_at_5_max value: 30.585219727516012 - type: nauc_precision_at_5_std value: 2.034776181342522 - type: nauc_recall_at_1000_diff1 value: 32.56442385278414 - type: nauc_recall_at_1000_max value: 51.480513202995915 - type: nauc_recall_at_1000_std value: 49.27851711330113 - type: nauc_recall_at_100_diff1 value: 41.029807873528796 - type: nauc_recall_at_100_max value: 42.572875140257565 - type: nauc_recall_at_100_std value: 23.745431356647018 - type: nauc_recall_at_10_diff1 value: 44.87576445883134 - type: nauc_recall_at_10_max value: 31.525509952765297 - type: nauc_recall_at_10_std value: -2.8068541204926394 - type: nauc_recall_at_1_diff1 value: 62.1943454407426 - type: nauc_recall_at_1_max value: 31.106713330762027 - type: nauc_recall_at_1_std value: -9.349274581003204 - type: nauc_recall_at_20_diff1 value: 44.969586307113104 - type: nauc_recall_at_20_max value: 31.29074006190112 - type: nauc_recall_at_20_std value: 5.946087034639837 - type: nauc_recall_at_3_diff1 value: 51.6957387291602 - type: nauc_recall_at_3_max value: 32.97568422788289 - type: nauc_recall_at_3_std value: -4.466963198284326 - type: nauc_recall_at_5_diff1 value: 47.596245200271184 - type: nauc_recall_at_5_max value: 30.791573693357755 - type: nauc_recall_at_5_std value: -4.702986567079468 - type: ndcg_at_1 value: 40.772999999999996 - type: ndcg_at_10 value: 52.39 - type: ndcg_at_100 value: 57.4 - type: ndcg_at_1000 value: 59.12500000000001 - type: ndcg_at_20 value: 54.459 - type: ndcg_at_3 value: 47.339999999999996 - type: ndcg_at_5 value: 49.717 - type: precision_at_1 value: 40.772999999999996 - type: precision_at_10 value: 10.029 - type: precision_at_100 value: 1.552 - type: precision_at_1000 value: 0.199 - type: precision_at_20 value: 5.937 - type: precision_at_3 value: 22.938 - type: precision_at_5 value: 16.423 - type: recall_at_1 value: 34.169 - type: recall_at_10 value: 64.68199999999999 - type: recall_at_100 value: 85.497 - type: recall_at_1000 value: 96.391 - type: recall_at_20 value: 72.053 - type: recall_at_3 value: 50.059 - type: recall_at_5 value: 56.589999999999996 task: type: Retrieval - dataset: config: default name: MTEB CQADupstackEnglishRetrieval revision: ad9991cb51e31e31e430383c75ffb2885547b5f0 split: test type: mteb/cqadupstack-english metrics: - type: main_score value: 47.753 - type: map_at_1 value: 31.211 - type: map_at_10 value: 42.015 - type: map_at_100 value: 43.262 - type: map_at_1000 value: 43.39 - type: map_at_20 value: 42.697 - type: map_at_3 value: 39.034 - type: map_at_5 value: 40.729 - type: mrr_at_1 value: 39.29936305732484 - type: mrr_at_10 value: 47.96489232635733 - type: mrr_at_100 value: 48.64731557532463 - type: mrr_at_1000 value: 48.68894277936992 - type: mrr_at_20 value: 48.39750821161911 - type: mrr_at_3 value: 45.849256900212346 - type: mrr_at_5 value: 47.14225053078562 - type: nauc_map_at_1000_diff1 value: 52.24106468332894 - type: nauc_map_at_1000_max value: 38.375619410420775 - type: nauc_map_at_1000_std value: -4.482915396811548 - type: nauc_map_at_100_diff1 value: 52.24328589744081 - type: nauc_map_at_100_max value: 38.29250166419244 - type: nauc_map_at_100_std value: -4.583741891930366 - type: nauc_map_at_10_diff1 value: 52.454090722359666 - type: nauc_map_at_10_max value: 37.7173958399999 - type: nauc_map_at_10_std value: -5.860564273083544 - type: nauc_map_at_1_diff1 value: 57.37978281090314 - type: nauc_map_at_1_max value: 32.49915109819217 - type: nauc_map_at_1_std value: -11.254055060290668 - type: nauc_map_at_20_diff1 value: 52.37379963537071 - type: nauc_map_at_20_max value: 37.943912770037116 - type: nauc_map_at_20_std value: -5.26512232555932 - type: nauc_map_at_3_diff1 value: 53.315496436307974 - type: nauc_map_at_3_max value: 35.798184763155234 - type: nauc_map_at_3_std value: -8.424700965366764 - type: nauc_map_at_5_diff1 value: 52.65283986998911 - type: nauc_map_at_5_max value: 36.545213941567425 - type: nauc_map_at_5_std value: -7.257895143708675 - type: nauc_mrr_at_1000_diff1 value: 50.16585921071199 - type: nauc_mrr_at_1000_max value: 40.98958361489588 - type: nauc_mrr_at_1000_std value: 0.5225503686128239 - type: nauc_mrr_at_100_diff1 value: 50.16290335172945 - type: nauc_mrr_at_100_max value: 40.983188618358675 - type: nauc_mrr_at_100_std value: 0.5371456026411485 - type: nauc_mrr_at_10_diff1 value: 50.188741789601664 - type: nauc_mrr_at_10_max value: 41.20385325286603 - type: nauc_mrr_at_10_std value: 0.4299969547822729 - type: nauc_mrr_at_1_diff1 value: 52.89957788273348 - type: nauc_mrr_at_1_max value: 40.15889247486093 - type: nauc_mrr_at_1_std value: -2.3643009397477233 - type: nauc_mrr_at_20_diff1 value: 50.18924356147879 - type: nauc_mrr_at_20_max value: 41.03580332941115 - type: nauc_mrr_at_20_std value: 0.5030743116655743 - type: nauc_mrr_at_3_diff1 value: 50.2595645561214 - type: nauc_mrr_at_3_max value: 40.84037333925909 - type: nauc_mrr_at_3_std value: -0.47444597498111607 - type: nauc_mrr_at_5_diff1 value: 50.30028806022225 - type: nauc_mrr_at_5_max value: 40.94735400001521 - type: nauc_mrr_at_5_std value: 0.014502584942351543 - type: nauc_ndcg_at_1000_diff1 value: 50.02853655454531 - type: nauc_ndcg_at_1000_max value: 40.0601698049325 - type: nauc_ndcg_at_1000_std value: 0.5989970984573991 - type: nauc_ndcg_at_100_diff1 value: 49.99180232217219 - type: nauc_ndcg_at_100_max value: 39.828019879089595 - type: nauc_ndcg_at_100_std value: 0.5679819431802621 - type: nauc_ndcg_at_10_diff1 value: 50.49123456226574 - type: nauc_ndcg_at_10_max value: 40.24693971238414 - type: nauc_ndcg_at_10_std value: -1.6801008721493143 - type: nauc_ndcg_at_1_diff1 value: 52.89957788273348 - type: nauc_ndcg_at_1_max value: 40.15889247486093 - type: nauc_ndcg_at_1_std value: -2.3643009397477233 - type: nauc_ndcg_at_20_diff1 value: 50.50251680139964 - type: nauc_ndcg_at_20_max value: 39.72201768032907 - type: nauc_ndcg_at_20_std value: -1.0868971947199884 - type: nauc_ndcg_at_3_diff1 value: 50.58336010302361 - type: nauc_ndcg_at_3_max value: 39.56460688076086 - type: nauc_ndcg_at_3_std value: -3.192413421384452 - type: nauc_ndcg_at_5_diff1 value: 50.670426480094 - type: nauc_ndcg_at_5_max value: 39.30101734473154 - type: nauc_ndcg_at_5_std value: -3.133266437331269 - type: nauc_precision_at_1000_diff1 value: -9.966142918444545 - type: nauc_precision_at_1000_max value: 15.887181550157312 - type: nauc_precision_at_1000_std value: 30.07775267584185 - type: nauc_precision_at_100_diff1 value: -5.39669332549404 - type: nauc_precision_at_100_max value: 26.372684632284045 - type: nauc_precision_at_100_std value: 35.21702840254392 - type: nauc_precision_at_10_diff1 value: 11.258611067705424 - type: nauc_precision_at_10_max value: 38.49722079603548 - type: nauc_precision_at_10_std value: 23.96075630064515 - type: nauc_precision_at_1_diff1 value: 52.89957788273348 - type: nauc_precision_at_1_max value: 40.15889247486093 - type: nauc_precision_at_1_std value: -2.3643009397477233 - type: nauc_precision_at_20_diff1 value: 5.119010028728253 - type: nauc_precision_at_20_max value: 33.86149623123613 - type: nauc_precision_at_20_std value: 28.568877042019192 - type: nauc_precision_at_3_diff1 value: 28.335387516697054 - type: nauc_precision_at_3_max value: 39.550279228497395 - type: nauc_precision_at_3_std value: 10.53626944081313 - type: nauc_precision_at_5_diff1 value: 20.067823876439405 - type: nauc_precision_at_5_max value: 38.012587686917584 - type: nauc_precision_at_5_std value: 15.994020926756388 - type: nauc_recall_at_1000_diff1 value: 32.76791129120816 - type: nauc_recall_at_1000_max value: 37.496559622674766 - type: nauc_recall_at_1000_std value: 25.655864139857222 - type: nauc_recall_at_100_diff1 value: 38.447827716104584 - type: nauc_recall_at_100_max value: 36.474274046070214 - type: nauc_recall_at_100_std value: 16.203946452884637 - type: nauc_recall_at_10_diff1 value: 45.04137348043349 - type: nauc_recall_at_10_max value: 38.413608602825924 - type: nauc_recall_at_10_std value: 0.8596466671993844 - type: nauc_recall_at_1_diff1 value: 57.37978281090314 - type: nauc_recall_at_1_max value: 32.49915109819217 - type: nauc_recall_at_1_std value: -11.254055060290668 - type: nauc_recall_at_20_diff1 value: 43.22936216632159 - type: nauc_recall_at_20_max value: 36.644412179031946 - type: nauc_recall_at_20_std value: 4.295238193474559 - type: nauc_recall_at_3_diff1 value: 48.45830999117096 - type: nauc_recall_at_3_max value: 34.59148426593566 - type: nauc_recall_at_3_std value: -7.2544944905307425 - type: nauc_recall_at_5_diff1 value: 46.818595627596906 - type: nauc_recall_at_5_max value: 35.28346610822803 - type: nauc_recall_at_5_std value: -4.578586774496801 - type: ndcg_at_1 value: 39.299 - type: ndcg_at_10 value: 47.753 - type: ndcg_at_100 value: 52.146 - type: ndcg_at_1000 value: 54.13400000000001 - type: ndcg_at_20 value: 49.573 - type: ndcg_at_3 value: 43.783 - type: ndcg_at_5 value: 45.638 - type: precision_at_1 value: 39.299 - type: precision_at_10 value: 8.955 - type: precision_at_100 value: 1.427 - type: precision_at_1000 value: 0.189 - type: precision_at_20 value: 5.252 - type: precision_at_3 value: 21.295 - type: precision_at_5 value: 14.943000000000001 - type: recall_at_1 value: 31.211 - type: recall_at_10 value: 57.831999999999994 - type: recall_at_100 value: 76.211 - type: recall_at_1000 value: 88.803 - type: recall_at_20 value: 64.548 - type: recall_at_3 value: 45.909 - type: recall_at_5 value: 51.304 task: type: Retrieval - dataset: config: default name: MTEB CQADupstackGamingRetrieval revision: 4885aa143210c98657558c04aaf3dc47cfb54340 split: test type: mteb/cqadupstack-gaming metrics: - type: main_score value: 61.58 - type: map_at_1 value: 42.294 - type: map_at_10 value: 55.625 - type: map_at_100 value: 56.618 - type: map_at_1000 value: 56.672 - type: map_at_20 value: 56.255 - type: map_at_3 value: 52.539 - type: map_at_5 value: 54.118 - type: mrr_at_1 value: 48.08777429467085 - type: mrr_at_10 value: 58.84965417724045 - type: mrr_at_100 value: 59.43662069430218 - type: mrr_at_1000 value: 59.46383163108614 - type: mrr_at_20 value: 59.238920207635395 - type: mrr_at_3 value: 56.645768025078446 - type: mrr_at_5 value: 57.799373040752464 - type: nauc_map_at_1000_diff1 value: 55.574840425792594 - type: nauc_map_at_1000_max value: 34.4701847562437 - type: nauc_map_at_1000_std value: -9.755236496635238 - type: nauc_map_at_100_diff1 value: 55.56633342053875 - type: nauc_map_at_100_max value: 34.463217368520056 - type: nauc_map_at_100_std value: -9.736389955691903 - type: nauc_map_at_10_diff1 value: 55.377779197086085 - type: nauc_map_at_10_max value: 34.119042879932806 - type: nauc_map_at_10_std value: -10.34552914025853 - type: nauc_map_at_1_diff1 value: 59.39306584705475 - type: nauc_map_at_1_max value: 29.113177128377604 - type: nauc_map_at_1_std value: -11.444784267280179 - type: nauc_map_at_20_diff1 value: 55.5279849352151 - type: nauc_map_at_20_max value: 34.40776911487248 - type: nauc_map_at_20_std value: -9.980725430581513 - type: nauc_map_at_3_diff1 value: 56.36831487626787 - type: nauc_map_at_3_max value: 32.50972201654767 - type: nauc_map_at_3_std value: -12.162628011645566 - type: nauc_map_at_5_diff1 value: 55.61245893128041 - type: nauc_map_at_5_max value: 33.29312656827763 - type: nauc_map_at_5_std value: -11.4951400718427 - type: nauc_mrr_at_1000_diff1 value: 54.90317690493004 - type: nauc_mrr_at_1000_max value: 35.56498085453685 - type: nauc_mrr_at_1000_std value: -9.234699847400888 - type: nauc_mrr_at_100_diff1 value: 54.896346456934275 - type: nauc_mrr_at_100_max value: 35.58324293276136 - type: nauc_mrr_at_100_std value: -9.215853996857613 - type: nauc_mrr_at_10_diff1 value: 54.671458608944114 - type: nauc_mrr_at_10_max value: 35.56708736221525 - type: nauc_mrr_at_10_std value: -9.373432397620997 - type: nauc_mrr_at_1_diff1 value: 58.79146952147216 - type: nauc_mrr_at_1_max value: 33.869452831902805 - type: nauc_mrr_at_1_std value: -10.566480052563577 - type: nauc_mrr_at_20_diff1 value: 54.849781607868906 - type: nauc_mrr_at_20_max value: 35.617115851895484 - type: nauc_mrr_at_20_std value: -9.208616079050708 - type: nauc_mrr_at_3_diff1 value: 55.16070050179383 - type: nauc_mrr_at_3_max value: 35.177757122772135 - type: nauc_mrr_at_3_std value: -10.393796685155891 - type: nauc_mrr_at_5_diff1 value: 54.828393662595445 - type: nauc_mrr_at_5_max value: 35.56940994194972 - type: nauc_mrr_at_5_std value: -9.731305997976602 - type: nauc_ndcg_at_1000_diff1 value: 54.50183704484519 - type: nauc_ndcg_at_1000_max value: 36.36712113663723 - type: nauc_ndcg_at_1000_std value: -7.34102789192482 - type: nauc_ndcg_at_100_diff1 value: 54.455028493563226 - type: nauc_ndcg_at_100_max value: 36.78883717977845 - type: nauc_ndcg_at_100_std value: -6.597339807115783 - type: nauc_ndcg_at_10_diff1 value: 53.536209026389535 - type: nauc_ndcg_at_10_max value: 36.18315401424091 - type: nauc_ndcg_at_10_std value: -8.273690022713248 - type: nauc_ndcg_at_1_diff1 value: 58.79146952147216 - type: nauc_ndcg_at_1_max value: 33.869452831902805 - type: nauc_ndcg_at_1_std value: -10.566480052563577 - type: nauc_ndcg_at_20_diff1 value: 54.15670848398847 - type: nauc_ndcg_at_20_max value: 36.74491143155494 - type: nauc_ndcg_at_20_std value: -7.313843626081678 - type: nauc_ndcg_at_3_diff1 value: 55.036932320029564 - type: nauc_ndcg_at_3_max value: 33.91087214672552 - type: nauc_ndcg_at_3_std value: -11.75293941895024 - type: nauc_ndcg_at_5_diff1 value: 54.063110780501546 - type: nauc_ndcg_at_5_max value: 35.052268948433515 - type: nauc_ndcg_at_5_std value: -10.452241125468108 - type: nauc_precision_at_1000_diff1 value: -17.38384602357622 - type: nauc_precision_at_1000_max value: 9.710315321476479 - type: nauc_precision_at_1000_std value: 16.90078109961968 - type: nauc_precision_at_100_diff1 value: -10.339926865487238 - type: nauc_precision_at_100_max value: 15.519587672026788 - type: nauc_precision_at_100_std value: 20.154368516383574 - type: nauc_precision_at_10_diff1 value: 8.295704958690228 - type: nauc_precision_at_10_max value: 27.223683123639592 - type: nauc_precision_at_10_std value: 9.078686113987018 - type: nauc_precision_at_1_diff1 value: 58.79146952147216 - type: nauc_precision_at_1_max value: 33.869452831902805 - type: nauc_precision_at_1_std value: -10.566480052563577 - type: nauc_precision_at_20_diff1 value: 1.206430229577525 - type: nauc_precision_at_20_max value: 23.944135447502873 - type: nauc_precision_at_20_std value: 14.990573969863142 - type: nauc_precision_at_3_diff1 value: 30.534879329433494 - type: nauc_precision_at_3_max value: 31.45998483553432 - type: nauc_precision_at_3_std value: -5.843145834483077 - type: nauc_precision_at_5_diff1 value: 20.082739223007245 - type: nauc_precision_at_5_max value: 29.165093621422066 - type: nauc_precision_at_5_std value: -0.628093664798173 - type: nauc_recall_at_1000_diff1 value: 29.998670252945253 - type: nauc_recall_at_1000_max value: 58.601149100236924 - type: nauc_recall_at_1000_std value: 51.912385311841525 - type: nauc_recall_at_100_diff1 value: 46.15565003366627 - type: nauc_recall_at_100_max value: 49.740072860097115 - type: nauc_recall_at_100_std value: 21.303498072710514 - type: nauc_recall_at_10_diff1 value: 44.934728375231664 - type: nauc_recall_at_10_max value: 39.7256315576848 - type: nauc_recall_at_10_std value: -1.5224958573376588 - type: nauc_recall_at_1_diff1 value: 59.39306584705475 - type: nauc_recall_at_1_max value: 29.113177128377604 - type: nauc_recall_at_1_std value: -11.444784267280179 - type: nauc_recall_at_20_diff1 value: 45.98017763702076 - type: nauc_recall_at_20_max value: 43.26825736903793 - type: nauc_recall_at_20_std value: 5.298466215553723 - type: nauc_recall_at_3_diff1 value: 51.290757427485445 - type: nauc_recall_at_3_max value: 33.21715365617901 - type: nauc_recall_at_3_std value: -12.227694565360014 - type: nauc_recall_at_5_diff1 value: 48.150864238331714 - type: nauc_recall_at_5_max value: 36.177367874135335 - type: nauc_recall_at_5_std value: -9.091237939153817 - type: ndcg_at_1 value: 48.087999999999994 - type: ndcg_at_10 value: 61.58 - type: ndcg_at_100 value: 65.253 - type: ndcg_at_1000 value: 66.267 - type: ndcg_at_20 value: 63.327999999999996 - type: ndcg_at_3 value: 56.496 - type: ndcg_at_5 value: 58.667 - type: precision_at_1 value: 48.087999999999994 - type: precision_at_10 value: 9.824 - type: precision_at_100 value: 1.262 - type: precision_at_1000 value: 0.13799999999999998 - type: precision_at_20 value: 5.4670000000000005 - type: precision_at_3 value: 25.266 - type: precision_at_5 value: 16.915 - type: recall_at_1 value: 42.294 - type: recall_at_10 value: 75.761 - type: recall_at_100 value: 91.208 - type: recall_at_1000 value: 98.37700000000001 - type: recall_at_20 value: 82.113 - type: recall_at_3 value: 61.956999999999994 - type: recall_at_5 value: 67.437 task: type: Retrieval - dataset: config: default name: MTEB CQADupstackGisRetrieval revision: 5003b3064772da1887988e05400cf3806fe491f2 split: test type: mteb/cqadupstack-gis metrics: - type: main_score value: 39.837 - type: map_at_1 value: 25.764 - type: map_at_10 value: 34.829 - type: map_at_100 value: 35.869 - type: map_at_1000 value: 35.953 - type: map_at_20 value: 35.393 - type: map_at_3 value: 32.208 - type: map_at_5 value: 33.641 - type: mrr_at_1 value: 28.0225988700565 - type: mrr_at_10 value: 37.04515290108509 - type: mrr_at_100 value: 37.93924588681648 - type: mrr_at_1000 value: 37.99860736806393 - type: mrr_at_20 value: 37.526581494375456 - type: mrr_at_3 value: 34.59510357815443 - type: mrr_at_5 value: 35.88323917137474 - type: nauc_map_at_1000_diff1 value: 39.89881614209902 - type: nauc_map_at_1000_max value: 26.20512364416086 - type: nauc_map_at_1000_std value: -1.4906980957334761 - type: nauc_map_at_100_diff1 value: 39.856757588861484 - type: nauc_map_at_100_max value: 26.17640926201627 - type: nauc_map_at_100_std value: -1.52726616900199 - type: nauc_map_at_10_diff1 value: 39.928116776753306 - type: nauc_map_at_10_max value: 25.849572817015947 - type: nauc_map_at_10_std value: -1.8553563555900154 - type: nauc_map_at_1_diff1 value: 45.9178313886473 - type: nauc_map_at_1_max value: 24.39859079325555 - type: nauc_map_at_1_std value: -7.08677568969915 - type: nauc_map_at_20_diff1 value: 40.03122889128656 - type: nauc_map_at_20_max value: 26.18324831822517 - type: nauc_map_at_20_std value: -1.4283210926006862 - type: nauc_map_at_3_diff1 value: 41.13705028238216 - type: nauc_map_at_3_max value: 25.227792355607466 - type: nauc_map_at_3_std value: -4.10076689245123 - type: nauc_map_at_5_diff1 value: 40.41343312331809 - type: nauc_map_at_5_max value: 26.019632871475828 - type: nauc_map_at_5_std value: -1.655322674363369 - type: nauc_mrr_at_1000_diff1 value: 38.10335222688471 - type: nauc_mrr_at_1000_max value: 27.76819205009592 - type: nauc_mrr_at_1000_std value: -0.06837137793398154 - type: nauc_mrr_at_100_diff1 value: 38.05479288369399 - type: nauc_mrr_at_100_max value: 27.7490714350333 - type: nauc_mrr_at_100_std value: -0.08908922458485154 - type: nauc_mrr_at_10_diff1 value: 37.96378944519479 - type: nauc_mrr_at_10_max value: 27.41804522591167 - type: nauc_mrr_at_10_std value: -0.19175217620882173 - type: nauc_mrr_at_1_diff1 value: 43.635050719694746 - type: nauc_mrr_at_1_max value: 26.637384998955994 - type: nauc_mrr_at_1_std value: -4.4281228525493574 - type: nauc_mrr_at_20_diff1 value: 38.125362772329744 - type: nauc_mrr_at_20_max value: 27.847086458572683 - type: nauc_mrr_at_20_std value: 0.04840792038341067 - type: nauc_mrr_at_3_diff1 value: 39.284979864146216 - type: nauc_mrr_at_3_max value: 27.57894001717352 - type: nauc_mrr_at_3_std value: -2.0696803670615265 - type: nauc_mrr_at_5_diff1 value: 38.33310714032024 - type: nauc_mrr_at_5_max value: 27.73956844204174 - type: nauc_mrr_at_5_std value: 0.01395219897785327 - type: nauc_ndcg_at_1000_diff1 value: 37.584777068441745 - type: nauc_ndcg_at_1000_max value: 27.742338292986034 - type: nauc_ndcg_at_1000_std value: 1.5919119060984248 - type: nauc_ndcg_at_100_diff1 value: 36.324511379401926 - type: nauc_ndcg_at_100_max value: 27.135186980945264 - type: nauc_ndcg_at_100_std value: 1.012036671263999 - type: nauc_ndcg_at_10_diff1 value: 36.883529139532634 - type: nauc_ndcg_at_10_max value: 26.287452858703826 - type: nauc_ndcg_at_10_std value: 0.5701574244802798 - type: nauc_ndcg_at_1_diff1 value: 43.635050719694746 - type: nauc_ndcg_at_1_max value: 26.637384998955994 - type: nauc_ndcg_at_1_std value: -4.4281228525493574 - type: nauc_ndcg_at_20_diff1 value: 37.26150152174845 - type: nauc_ndcg_at_20_max value: 27.59340643785688 - type: nauc_ndcg_at_20_std value: 1.812718743312418 - type: nauc_ndcg_at_3_diff1 value: 39.291935691856196 - type: nauc_ndcg_at_3_max value: 26.155040259440188 - type: nauc_ndcg_at_3_std value: -3.1636780333695556 - type: nauc_ndcg_at_5_diff1 value: 37.96141298539201 - type: nauc_ndcg_at_5_max value: 26.886149321572532 - type: nauc_ndcg_at_5_std value: 0.9674089088890447 - type: nauc_precision_at_1000_diff1 value: -6.193908740223283 - type: nauc_precision_at_1000_max value: 18.933037425592925 - type: nauc_precision_at_1000_std value: 13.990532282500373 - type: nauc_precision_at_100_diff1 value: 2.0105355129064484 - type: nauc_precision_at_100_max value: 24.049503343204073 - type: nauc_precision_at_100_std value: 10.79645255758947 - type: nauc_precision_at_10_diff1 value: 21.59130230622116 - type: nauc_precision_at_10_max value: 28.24307291885018 - type: nauc_precision_at_10_std value: 9.505053356771134 - type: nauc_precision_at_1_diff1 value: 43.635050719694746 - type: nauc_precision_at_1_max value: 26.637384998955994 - type: nauc_precision_at_1_std value: -4.4281228525493574 - type: nauc_precision_at_20_diff1 value: 19.080704068772512 - type: nauc_precision_at_20_max value: 31.52215796190006 - type: nauc_precision_at_20_std value: 14.119376174372944 - type: nauc_precision_at_3_diff1 value: 31.733564430212695 - type: nauc_precision_at_3_max value: 28.63157366942346 - type: nauc_precision_at_3_std value: 0.7847711198268656 - type: nauc_precision_at_5_diff1 value: 27.2848881133242 - type: nauc_precision_at_5_max value: 30.792243775503593 - type: nauc_precision_at_5_std value: 10.5253825269578 - type: nauc_recall_at_1000_diff1 value: 32.59052385808082 - type: nauc_recall_at_1000_max value: 39.16615094126513 - type: nauc_recall_at_1000_std value: 32.831614951523285 - type: nauc_recall_at_100_diff1 value: 19.499054183956282 - type: nauc_recall_at_100_max value: 25.791611422283385 - type: nauc_recall_at_100_std value: 8.86161971226946 - type: nauc_recall_at_10_diff1 value: 27.966399902667032 - type: nauc_recall_at_10_max value: 24.52174702656756 - type: nauc_recall_at_10_std value: 5.888597161688889 - type: nauc_recall_at_1_diff1 value: 45.9178313886473 - type: nauc_recall_at_1_max value: 24.39859079325555 - type: nauc_recall_at_1_std value: -7.08677568969915 - type: nauc_recall_at_20_diff1 value: 28.92843394705317 - type: nauc_recall_at_20_max value: 29.498567057460452 - type: nauc_recall_at_20_std value: 10.973275179789377 - type: nauc_recall_at_3_diff1 value: 35.866038436338386 - type: nauc_recall_at_3_max value: 25.385826645049935 - type: nauc_recall_at_3_std value: -1.945839877039293 - type: nauc_recall_at_5_diff1 value: 31.781401015413223 - type: nauc_recall_at_5_max value: 26.580275134192338 - type: nauc_recall_at_5_std value: 6.830480292572181 - type: ndcg_at_1 value: 28.022999999999996 - type: ndcg_at_10 value: 39.837 - type: ndcg_at_100 value: 44.995000000000005 - type: ndcg_at_1000 value: 46.928 - type: ndcg_at_20 value: 41.729 - type: ndcg_at_3 value: 34.781 - type: ndcg_at_5 value: 37.136 - type: precision_at_1 value: 28.022999999999996 - type: precision_at_10 value: 6.101999999999999 - type: precision_at_100 value: 0.9159999999999999 - type: precision_at_1000 value: 0.11100000000000002 - type: precision_at_20 value: 3.508 - type: precision_at_3 value: 14.727 - type: precision_at_5 value: 10.237 - type: recall_at_1 value: 25.764 - type: recall_at_10 value: 53.101 - type: recall_at_100 value: 76.955 - type: recall_at_1000 value: 91.175 - type: recall_at_20 value: 60.138999999999996 - type: recall_at_3 value: 39.484 - type: recall_at_5 value: 45.208999999999996 task: type: Retrieval - dataset: config: default name: MTEB CQADupstackMathematicaRetrieval revision: 90fceea13679c63fe563ded68f3b6f06e50061de split: test type: mteb/cqadupstack-mathematica metrics: - type: main_score value: 29.697000000000003 - type: map_at_1 value: 15.958 - type: map_at_10 value: 24.324 - type: map_at_100 value: 25.64 - type: map_at_1000 value: 25.762 - type: map_at_20 value: 25.064999999999998 - type: map_at_3 value: 21.36 - type: map_at_5 value: 23.034 - type: mrr_at_1 value: 19.900497512437813 - type: mrr_at_10 value: 28.495715865118843 - type: mrr_at_100 value: 29.55883123523441 - type: mrr_at_1000 value: 29.623700416228015 - type: mrr_at_20 value: 29.127165464095363 - type: mrr_at_3 value: 25.538971807628535 - type: mrr_at_5 value: 27.392205638474294 - type: nauc_map_at_1000_diff1 value: 34.32209396022877 - type: nauc_map_at_1000_max value: 21.688010888227378 - type: nauc_map_at_1000_std value: 0.9387258101009914 - type: nauc_map_at_100_diff1 value: 34.2873397075491 - type: nauc_map_at_100_max value: 21.69354267495513 - type: nauc_map_at_100_std value: 0.9556799710614994 - type: nauc_map_at_10_diff1 value: 34.84715034635313 - type: nauc_map_at_10_max value: 21.547867843258768 - type: nauc_map_at_10_std value: 0.32632738479135376 - type: nauc_map_at_1_diff1 value: 40.08120759217827 - type: nauc_map_at_1_max value: 22.5404646356335 - type: nauc_map_at_1_std value: 0.46810928901963667 - type: nauc_map_at_20_diff1 value: 34.34837970041168 - type: nauc_map_at_20_max value: 21.60444832596523 - type: nauc_map_at_20_std value: 0.6374326062336801 - type: nauc_map_at_3_diff1 value: 36.058627394912286 - type: nauc_map_at_3_max value: 20.99605214930877 - type: nauc_map_at_3_std value: -0.6363564572418088 - type: nauc_map_at_5_diff1 value: 35.6851873771882 - type: nauc_map_at_5_max value: 21.2333736759757 - type: nauc_map_at_5_std value: -0.11978481640718161 - type: nauc_mrr_at_1000_diff1 value: 33.76519811972675 - type: nauc_mrr_at_1000_max value: 23.08099288276958 - type: nauc_mrr_at_1000_std value: 0.21670889636182983 - type: nauc_mrr_at_100_diff1 value: 33.75104610406262 - type: nauc_mrr_at_100_max value: 23.09993932553629 - type: nauc_mrr_at_100_std value: 0.21684296947206153 - type: nauc_mrr_at_10_diff1 value: 33.890656486559365 - type: nauc_mrr_at_10_max value: 23.004230124232492 - type: nauc_mrr_at_10_std value: -0.17125819000842146 - type: nauc_mrr_at_1_diff1 value: 38.298935378300584 - type: nauc_mrr_at_1_max value: 23.531620590823632 - type: nauc_mrr_at_1_std value: -1.786682602890334 - type: nauc_mrr_at_20_diff1 value: 33.71928046127925 - type: nauc_mrr_at_20_max value: 23.028631286321367 - type: nauc_mrr_at_20_std value: 0.13566082451341543 - type: nauc_mrr_at_3_diff1 value: 34.839766454140396 - type: nauc_mrr_at_3_max value: 23.310126494307852 - type: nauc_mrr_at_3_std value: -1.1749456883533465 - type: nauc_mrr_at_5_diff1 value: 34.33155350261903 - type: nauc_mrr_at_5_max value: 22.722607203637715 - type: nauc_mrr_at_5_std value: -0.4150972255202597 - type: nauc_ndcg_at_1000_diff1 value: 31.8237241348174 - type: nauc_ndcg_at_1000_max value: 23.08752218924686 - type: nauc_ndcg_at_1000_std value: 3.2872043822295653 - type: nauc_ndcg_at_100_diff1 value: 31.190151490173644 - type: nauc_ndcg_at_100_max value: 23.091678618693027 - type: nauc_ndcg_at_100_std value: 3.7042696983779337 - type: nauc_ndcg_at_10_diff1 value: 32.59190071742098 - type: nauc_ndcg_at_10_max value: 21.959767958947943 - type: nauc_ndcg_at_10_std value: 0.9617687304860824 - type: nauc_ndcg_at_1_diff1 value: 38.298935378300584 - type: nauc_ndcg_at_1_max value: 23.531620590823632 - type: nauc_ndcg_at_1_std value: -1.786682602890334 - type: nauc_ndcg_at_20_diff1 value: 31.269080621067157 - type: nauc_ndcg_at_20_max value: 22.118415303912165 - type: nauc_ndcg_at_20_std value: 1.9476492490549662 - type: nauc_ndcg_at_3_diff1 value: 34.75565433874818 - type: nauc_ndcg_at_3_max value: 21.88229440058002 - type: nauc_ndcg_at_3_std value: -1.0791926988814993 - type: nauc_ndcg_at_5_diff1 value: 34.15052980774227 - type: nauc_ndcg_at_5_max value: 21.506747457350297 - type: nauc_ndcg_at_5_std value: 0.16877419719517406 - type: nauc_precision_at_1000_diff1 value: -0.6858280622680386 - type: nauc_precision_at_1000_max value: 3.070259742786672 - type: nauc_precision_at_1000_std value: -0.44683494016016695 - type: nauc_precision_at_100_diff1 value: 5.672431851239273 - type: nauc_precision_at_100_max value: 14.924851867989434 - type: nauc_precision_at_100_std value: 7.98283699598593 - type: nauc_precision_at_10_diff1 value: 17.78733855494319 - type: nauc_precision_at_10_max value: 19.852122273791004 - type: nauc_precision_at_10_std value: 2.212115382310132 - type: nauc_precision_at_1_diff1 value: 38.298935378300584 - type: nauc_precision_at_1_max value: 23.531620590823632 - type: nauc_precision_at_1_std value: -1.786682602890334 - type: nauc_precision_at_20_diff1 value: 12.276912444781432 - type: nauc_precision_at_20_max value: 18.32715011630569 - type: nauc_precision_at_20_std value: 3.9553516927972416 - type: nauc_precision_at_3_diff1 value: 28.513019625970028 - type: nauc_precision_at_3_max value: 20.940923230836134 - type: nauc_precision_at_3_std value: -3.190188512438985 - type: nauc_precision_at_5_diff1 value: 25.002912184047123 - type: nauc_precision_at_5_max value: 20.035111366285058 - type: nauc_precision_at_5_std value: -0.22506988079631962 - type: nauc_recall_at_1000_diff1 value: 13.166705703269512 - type: nauc_recall_at_1000_max value: 35.93453865798138 - type: nauc_recall_at_1000_std value: 26.682834428503483 - type: nauc_recall_at_100_diff1 value: 17.596890918894267 - type: nauc_recall_at_100_max value: 26.383345729307116 - type: nauc_recall_at_100_std value: 16.648053006302884 - type: nauc_recall_at_10_diff1 value: 26.082210572506177 - type: nauc_recall_at_10_max value: 20.491825780854438 - type: nauc_recall_at_10_std value: 2.6495276614534395 - type: nauc_recall_at_1_diff1 value: 40.08120759217827 - type: nauc_recall_at_1_max value: 22.5404646356335 - type: nauc_recall_at_1_std value: 0.46810928901963667 - type: nauc_recall_at_20_diff1 value: 20.786345575454142 - type: nauc_recall_at_20_max value: 20.548104255810635 - type: nauc_recall_at_20_std value: 6.003243727958278 - type: nauc_recall_at_3_diff1 value: 31.619407559646717 - type: nauc_recall_at_3_max value: 19.77889595916651 - type: nauc_recall_at_3_std value: -1.7285416920920254 - type: nauc_recall_at_5_diff1 value: 29.985583590834363 - type: nauc_recall_at_5_max value: 19.291384926277257 - type: nauc_recall_at_5_std value: 0.4751386159254339 - type: ndcg_at_1 value: 19.900000000000002 - type: ndcg_at_10 value: 29.697000000000003 - type: ndcg_at_100 value: 35.873 - type: ndcg_at_1000 value: 38.586 - type: ndcg_at_20 value: 32.167 - type: ndcg_at_3 value: 24.171 - type: ndcg_at_5 value: 26.936 - type: precision_at_1 value: 19.900000000000002 - type: precision_at_10 value: 5.609 - type: precision_at_100 value: 1.004 - type: precision_at_1000 value: 0.13799999999999998 - type: precision_at_20 value: 3.5069999999999997 - type: precision_at_3 value: 11.526 - type: precision_at_5 value: 8.856 - type: recall_at_1 value: 15.958 - type: recall_at_10 value: 42.181999999999995 - type: recall_at_100 value: 68.926 - type: recall_at_1000 value: 87.93299999999999 - type: recall_at_20 value: 51.007999999999996 - type: recall_at_3 value: 27.339999999999996 - type: recall_at_5 value: 34.105000000000004 task: type: Retrieval - dataset: config: default name: MTEB CQADupstackPhysicsRetrieval revision: 79531abbd1fb92d06c6d6315a0cbbbf5bb247ea4 split: test type: mteb/cqadupstack-physics metrics: - type: main_score value: 46.007 - type: map_at_1 value: 29.349999999999998 - type: map_at_10 value: 39.775 - type: map_at_100 value: 41.104 - type: map_at_1000 value: 41.217999999999996 - type: map_at_20 value: 40.561 - type: map_at_3 value: 36.348 - type: map_at_5 value: 38.194 - type: mrr_at_1 value: 35.514918190567855 - type: mrr_at_10 value: 45.159150281864406 - type: mrr_at_100 value: 45.96990278179657 - type: mrr_at_1000 value: 46.01710125216311 - type: mrr_at_20 value: 45.64400545753921 - type: mrr_at_3 value: 42.28424767404553 - type: mrr_at_5 value: 43.87712544112925 - type: nauc_map_at_1000_diff1 value: 48.11895764039965 - type: nauc_map_at_1000_max value: 29.925477824810347 - type: nauc_map_at_1000_std value: 0.11504814310932239 - type: nauc_map_at_100_diff1 value: 48.12098289801071 - type: nauc_map_at_100_max value: 29.86278330417955 - type: nauc_map_at_100_std value: 0.060436831220252285 - type: nauc_map_at_10_diff1 value: 48.39540484393976 - type: nauc_map_at_10_max value: 29.351410400563317 - type: nauc_map_at_10_std value: -0.4621980042134368 - type: nauc_map_at_1_diff1 value: 54.47121878121458 - type: nauc_map_at_1_max value: 28.612001727844355 - type: nauc_map_at_1_std value: -4.093971357183509 - type: nauc_map_at_20_diff1 value: 48.143368911870425 - type: nauc_map_at_20_max value: 29.68026068686464 - type: nauc_map_at_20_std value: -0.163617345028557 - type: nauc_map_at_3_diff1 value: 48.97810056717568 - type: nauc_map_at_3_max value: 29.127235123692852 - type: nauc_map_at_3_std value: -1.567085586310881 - type: nauc_map_at_5_diff1 value: 48.43112906695029 - type: nauc_map_at_5_max value: 29.4251432266822 - type: nauc_map_at_5_std value: -0.8442546630575809 - type: nauc_mrr_at_1000_diff1 value: 47.33498761500292 - type: nauc_mrr_at_1000_max value: 31.61027031045797 - type: nauc_mrr_at_1000_std value: 1.6928194091772466 - type: nauc_mrr_at_100_diff1 value: 47.329981330106705 - type: nauc_mrr_at_100_max value: 31.600851021560985 - type: nauc_mrr_at_100_std value: 1.6838117616288253 - type: nauc_mrr_at_10_diff1 value: 47.30242253222377 - type: nauc_mrr_at_10_max value: 31.541347591823182 - type: nauc_mrr_at_10_std value: 1.6027415899976174 - type: nauc_mrr_at_1_diff1 value: 53.06561519484963 - type: nauc_mrr_at_1_max value: 31.987901603200687 - type: nauc_mrr_at_1_std value: 0.3577806477314526 - type: nauc_mrr_at_20_diff1 value: 47.330716020995595 - type: nauc_mrr_at_20_max value: 31.55066307584678 - type: nauc_mrr_at_20_std value: 1.6320128807973546 - type: nauc_mrr_at_3_diff1 value: 47.20092644242079 - type: nauc_mrr_at_3_max value: 31.297218751277995 - type: nauc_mrr_at_3_std value: 0.888116504301737 - type: nauc_mrr_at_5_diff1 value: 47.13557519365777 - type: nauc_mrr_at_5_max value: 31.612986186592124 - type: nauc_mrr_at_5_std value: 1.453385158208238 - type: nauc_ndcg_at_1000_diff1 value: 46.069421394896096 - type: nauc_ndcg_at_1000_max value: 31.63575075486594 - type: nauc_ndcg_at_1000_std value: 2.980065497846789 - type: nauc_ndcg_at_100_diff1 value: 46.112281840719724 - type: nauc_ndcg_at_100_max value: 30.902568316145064 - type: nauc_ndcg_at_100_std value: 2.4907275571593934 - type: nauc_ndcg_at_10_diff1 value: 46.41376893687042 - type: nauc_ndcg_at_10_max value: 29.23719673315816 - type: nauc_ndcg_at_10_std value: 0.7970981707285222 - type: nauc_ndcg_at_1_diff1 value: 53.06561519484963 - type: nauc_ndcg_at_1_max value: 31.987901603200687 - type: nauc_ndcg_at_1_std value: 0.3577806477314526 - type: nauc_ndcg_at_20_diff1 value: 45.874746478664484 - type: nauc_ndcg_at_20_max value: 29.845280657306272 - type: nauc_ndcg_at_20_std value: 1.3727615463766498 - type: nauc_ndcg_at_3_diff1 value: 46.62956408759762 - type: nauc_ndcg_at_3_max value: 29.76974459191416 - type: nauc_ndcg_at_3_std value: -0.4825766869465331 - type: nauc_ndcg_at_5_diff1 value: 46.379271929390356 - type: nauc_ndcg_at_5_max value: 29.880549956424098 - type: nauc_ndcg_at_5_std value: 0.4800086104773111 - type: nauc_precision_at_1000_diff1 value: -13.96376128722718 - type: nauc_precision_at_1000_max value: 13.458410308076655 - type: nauc_precision_at_1000_std value: 15.301880813229934 - type: nauc_precision_at_100_diff1 value: -4.637278193987726 - type: nauc_precision_at_100_max value: 18.25462789192962 - type: nauc_precision_at_100_std value: 16.309429890343687 - type: nauc_precision_at_10_diff1 value: 15.248083771766987 - type: nauc_precision_at_10_max value: 24.87434955413506 - type: nauc_precision_at_10_std value: 12.167229419982565 - type: nauc_precision_at_1_diff1 value: 53.06561519484963 - type: nauc_precision_at_1_max value: 31.987901603200687 - type: nauc_precision_at_1_std value: 0.3577806477314526 - type: nauc_precision_at_20_diff1 value: 5.696350416504954 - type: nauc_precision_at_20_max value: 22.86200256696701 - type: nauc_precision_at_20_std value: 14.443461022731741 - type: nauc_precision_at_3_diff1 value: 30.97090928535156 - type: nauc_precision_at_3_max value: 29.366143782023656 - type: nauc_precision_at_3_std value: 5.099366151068488 - type: nauc_precision_at_5_diff1 value: 23.276107412097236 - type: nauc_precision_at_5_max value: 28.60311491474139 - type: nauc_precision_at_5_std value: 9.666927035379812 - type: nauc_recall_at_1000_diff1 value: 27.414721915157287 - type: nauc_recall_at_1000_max value: 48.1562394286556 - type: nauc_recall_at_1000_std value: 40.587486120168585 - type: nauc_recall_at_100_diff1 value: 36.31335222272159 - type: nauc_recall_at_100_max value: 29.66932371155468 - type: nauc_recall_at_100_std value: 12.629945573071282 - type: nauc_recall_at_10_diff1 value: 39.37105859307282 - type: nauc_recall_at_10_max value: 23.604377853362408 - type: nauc_recall_at_10_std value: 1.898285967993768 - type: nauc_recall_at_1_diff1 value: 54.47121878121458 - type: nauc_recall_at_1_max value: 28.612001727844355 - type: nauc_recall_at_1_std value: -4.093971357183509 - type: nauc_recall_at_20_diff1 value: 36.648839370171544 - type: nauc_recall_at_20_max value: 24.592852957100266 - type: nauc_recall_at_20_std value: 3.609510577266925 - type: nauc_recall_at_3_diff1 value: 42.52133471177241 - type: nauc_recall_at_3_max value: 26.25576104463365 - type: nauc_recall_at_3_std value: -1.5419672040106491 - type: nauc_recall_at_5_diff1 value: 40.32588576278877 - type: nauc_recall_at_5_max value: 26.295242572486284 - type: nauc_recall_at_5_std value: 0.42561932055282814 - type: ndcg_at_1 value: 35.515 - type: ndcg_at_10 value: 46.007 - type: ndcg_at_100 value: 51.342 - type: ndcg_at_1000 value: 53.359 - type: ndcg_at_20 value: 48.231 - type: ndcg_at_3 value: 40.272999999999996 - type: ndcg_at_5 value: 42.875 - type: precision_at_1 value: 35.515 - type: precision_at_10 value: 8.364 - type: precision_at_100 value: 1.288 - type: precision_at_1000 value: 0.164 - type: precision_at_20 value: 4.962 - type: precision_at_3 value: 18.961 - type: precision_at_5 value: 13.513 - type: recall_at_1 value: 29.349999999999998 - type: recall_at_10 value: 59.267 - type: recall_at_100 value: 81.243 - type: recall_at_1000 value: 94.161 - type: recall_at_20 value: 66.77199999999999 - type: recall_at_3 value: 43.257 - type: recall_at_5 value: 49.933 task: type: Retrieval - dataset: config: default name: MTEB CQADupstackProgrammersRetrieval revision: 6184bc1440d2dbc7612be22b50686b8826d22b32 split: test type: mteb/cqadupstack-programmers metrics: - type: main_score value: 43.104 - type: map_at_1 value: 26.265 - type: map_at_10 value: 37.098 - type: map_at_100 value: 38.582 - type: map_at_1000 value: 38.68 - type: map_at_20 value: 37.911 - type: map_at_3 value: 33.803 - type: map_at_5 value: 35.629 - type: mrr_at_1 value: 32.64840182648402 - type: mrr_at_10 value: 42.31621910560265 - type: mrr_at_100 value: 43.314920371374065 - type: mrr_at_1000 value: 43.35964252174629 - type: mrr_at_20 value: 42.90480897932176 - type: mrr_at_3 value: 39.764079147640786 - type: mrr_at_5 value: 41.25380517503803 - type: nauc_map_at_1000_diff1 value: 43.45029323897613 - type: nauc_map_at_1000_max value: 32.72487667511562 - type: nauc_map_at_1000_std value: 4.635498217182894 - type: nauc_map_at_100_diff1 value: 43.416036838727564 - type: nauc_map_at_100_max value: 32.7136330112318 - type: nauc_map_at_100_std value: 4.691526480787624 - type: nauc_map_at_10_diff1 value: 43.64606477170319 - type: nauc_map_at_10_max value: 32.29286435030195 - type: nauc_map_at_10_std value: 3.8611425120790854 - type: nauc_map_at_1_diff1 value: 48.47333496205567 - type: nauc_map_at_1_max value: 27.743297364140062 - type: nauc_map_at_1_std value: -2.2399535593996704 - type: nauc_map_at_20_diff1 value: 43.32047006337627 - type: nauc_map_at_20_max value: 32.39695913886823 - type: nauc_map_at_20_std value: 4.145069332261304 - type: nauc_map_at_3_diff1 value: 44.17067949379911 - type: nauc_map_at_3_max value: 31.529396644447843 - type: nauc_map_at_3_std value: 1.0278370965255377 - type: nauc_map_at_5_diff1 value: 43.91747436857912 - type: nauc_map_at_5_max value: 32.28375375021212 - type: nauc_map_at_5_std value: 2.4868152401854586 - type: nauc_mrr_at_1000_diff1 value: 43.07485725751252 - type: nauc_mrr_at_1000_max value: 33.834125690650744 - type: nauc_mrr_at_1000_std value: 6.225141076338039 - type: nauc_mrr_at_100_diff1 value: 43.04619032998411 - type: nauc_mrr_at_100_max value: 33.839469490267014 - type: nauc_mrr_at_100_std value: 6.26262791407454 - type: nauc_mrr_at_10_diff1 value: 43.135041859981314 - type: nauc_mrr_at_10_max value: 33.66061195696024 - type: nauc_mrr_at_10_std value: 6.031199962121296 - type: nauc_mrr_at_1_diff1 value: 48.679120171485025 - type: nauc_mrr_at_1_max value: 32.630645939261896 - type: nauc_mrr_at_1_std value: 2.721955383843323 - type: nauc_mrr_at_20_diff1 value: 42.92758496123681 - type: nauc_mrr_at_20_max value: 33.758469763960385 - type: nauc_mrr_at_20_std value: 6.092108139456077 - type: nauc_mrr_at_3_diff1 value: 43.67155702067532 - type: nauc_mrr_at_3_max value: 33.90223139830306 - type: nauc_mrr_at_3_std value: 4.286070486118104 - type: nauc_mrr_at_5_diff1 value: 43.338615104225035 - type: nauc_mrr_at_5_max value: 33.614835693617565 - type: nauc_mrr_at_5_std value: 5.110470665508919 - type: nauc_ndcg_at_1000_diff1 value: 41.49895410329273 - type: nauc_ndcg_at_1000_max value: 34.195234462236854 - type: nauc_ndcg_at_1000_std value: 8.498163780196581 - type: nauc_ndcg_at_100_diff1 value: 40.75234081378347 - type: nauc_ndcg_at_100_max value: 34.23564373044607 - type: nauc_ndcg_at_100_std value: 10.05540839333941 - type: nauc_ndcg_at_10_diff1 value: 41.41477444523898 - type: nauc_ndcg_at_10_max value: 32.833800717913114 - type: nauc_ndcg_at_10_std value: 6.858969811913209 - type: nauc_ndcg_at_1_diff1 value: 48.679120171485025 - type: nauc_ndcg_at_1_max value: 32.630645939261896 - type: nauc_ndcg_at_1_std value: 2.721955383843323 - type: nauc_ndcg_at_20_diff1 value: 40.28434573239677 - type: nauc_ndcg_at_20_max value: 32.87240585064605 - type: nauc_ndcg_at_20_std value: 7.458863737330834 - type: nauc_ndcg_at_3_diff1 value: 42.381887136641296 - type: nauc_ndcg_at_3_max value: 32.99180751163812 - type: nauc_ndcg_at_3_std value: 2.7500063712633787 - type: nauc_ndcg_at_5_diff1 value: 42.12035473910616 - type: nauc_ndcg_at_5_max value: 33.048554190063015 - type: nauc_ndcg_at_5_std value: 4.229238611217661 - type: nauc_precision_at_1000_diff1 value: -7.055058599902845 - type: nauc_precision_at_1000_max value: 5.279974888532298 - type: nauc_precision_at_1000_std value: 6.3972410796255375 - type: nauc_precision_at_100_diff1 value: 0.296177312991241 - type: nauc_precision_at_100_max value: 16.345552573537926 - type: nauc_precision_at_100_std value: 21.61996821825965 - type: nauc_precision_at_10_diff1 value: 18.3634158746968 - type: nauc_precision_at_10_max value: 28.666545042552787 - type: nauc_precision_at_10_std value: 20.253307172595655 - type: nauc_precision_at_1_diff1 value: 48.679120171485025 - type: nauc_precision_at_1_max value: 32.630645939261896 - type: nauc_precision_at_1_std value: 2.721955383843323 - type: nauc_precision_at_20_diff1 value: 9.050777040054417 - type: nauc_precision_at_20_max value: 23.739053156719237 - type: nauc_precision_at_20_std value: 21.201347657987768 - type: nauc_precision_at_3_diff1 value: 31.288764373507032 - type: nauc_precision_at_3_max value: 35.03739616944736 - type: nauc_precision_at_3_std value: 10.687519325124368 - type: nauc_precision_at_5_diff1 value: 26.094294182356514 - type: nauc_precision_at_5_max value: 34.02729032623777 - type: nauc_precision_at_5_std value: 15.155944334598829 - type: nauc_recall_at_1000_diff1 value: 17.067589622044004 - type: nauc_recall_at_1000_max value: 47.0515288862208 - type: nauc_recall_at_1000_std value: 57.35027883483996 - type: nauc_recall_at_100_diff1 value: 21.968219400981674 - type: nauc_recall_at_100_max value: 37.32117389917341 - type: nauc_recall_at_100_std value: 41.057773107985334 - type: nauc_recall_at_10_diff1 value: 33.406908724986046 - type: nauc_recall_at_10_max value: 30.675911712325725 - type: nauc_recall_at_10_std value: 13.85129513699196 - type: nauc_recall_at_1_diff1 value: 48.47333496205567 - type: nauc_recall_at_1_max value: 27.743297364140062 - type: nauc_recall_at_1_std value: -2.2399535593996704 - type: nauc_recall_at_20_diff1 value: 27.265337034919746 - type: nauc_recall_at_20_max value: 29.77179827995252 - type: nauc_recall_at_20_std value: 15.781133217528273 - type: nauc_recall_at_3_diff1 value: 37.251744239338294 - type: nauc_recall_at_3_max value: 30.951729412099954 - type: nauc_recall_at_3_std value: 1.5825770156707786 - type: nauc_recall_at_5_diff1 value: 35.95939054074726 - type: nauc_recall_at_5_max value: 31.08801927030664 - type: nauc_recall_at_5_std value: 5.977374956099989 - type: ndcg_at_1 value: 32.647999999999996 - type: ndcg_at_10 value: 43.104 - type: ndcg_at_100 value: 49.297000000000004 - type: ndcg_at_1000 value: 51.275000000000006 - type: ndcg_at_20 value: 45.587 - type: ndcg_at_3 value: 37.903 - type: ndcg_at_5 value: 40.253 - type: precision_at_1 value: 32.647999999999996 - type: precision_at_10 value: 7.991 - type: precision_at_100 value: 1.298 - type: precision_at_1000 value: 0.163 - type: precision_at_20 value: 4.8 - type: precision_at_3 value: 18.227 - type: precision_at_5 value: 13.014000000000001 - type: recall_at_1 value: 26.265 - type: recall_at_10 value: 55.605000000000004 - type: recall_at_100 value: 81.963 - type: recall_at_1000 value: 95.236 - type: recall_at_20 value: 64.455 - type: recall_at_3 value: 41.105999999999995 - type: recall_at_5 value: 47.260000000000005 task: type: Retrieval - dataset: config: default name: MTEB CQADupstackRetrieval revision: CQADupstackRetrieval_is_a_combined_dataset split: test type: CQADupstackRetrieval_is_a_combined_dataset metrics: - type: main_score value: 41.83275000000001 - type: ndcg_at_10 value: 41.83275000000001 task: type: Retrieval - dataset: config: default name: MTEB CQADupstackStatsRetrieval revision: 65ac3a16b8e91f9cee4c9828cc7c335575432a2a split: test type: mteb/cqadupstack-stats metrics: - type: main_score value: 35.375 - type: map_at_1 value: 23.547 - type: map_at_10 value: 31.142999999999997 - type: map_at_100 value: 32.186 - type: map_at_1000 value: 32.272 - type: map_at_20 value: 31.730999999999998 - type: map_at_3 value: 28.919 - type: map_at_5 value: 30.407 - type: mrr_at_1 value: 26.68711656441718 - type: mrr_at_10 value: 34.079754601227 - type: mrr_at_100 value: 35.01087377397623 - type: mrr_at_1000 value: 35.06789006352093 - type: mrr_at_20 value: 34.633737519268365 - type: mrr_at_3 value: 32.13190184049082 - type: mrr_at_5 value: 33.45092024539879 - type: nauc_map_at_1000_diff1 value: 51.66312260771515 - type: nauc_map_at_1000_max value: 32.57627734882073 - type: nauc_map_at_1000_std value: 6.4793973203324375 - type: nauc_map_at_100_diff1 value: 51.631054229476106 - type: nauc_map_at_100_max value: 32.53946656654599 - type: nauc_map_at_100_std value: 6.471252184684487 - type: nauc_map_at_10_diff1 value: 51.684081771168714 - type: nauc_map_at_10_max value: 32.359366254689355 - type: nauc_map_at_10_std value: 5.952809015194049 - type: nauc_map_at_1_diff1 value: 59.657552965362214 - type: nauc_map_at_1_max value: 30.802326957153465 - type: nauc_map_at_1_std value: 3.7728372494699363 - type: nauc_map_at_20_diff1 value: 51.70440488136916 - type: nauc_map_at_20_max value: 32.49067664088131 - type: nauc_map_at_20_std value: 6.193923730413403 - type: nauc_map_at_3_diff1 value: 53.064689875994844 - type: nauc_map_at_3_max value: 32.02328326484619 - type: nauc_map_at_3_std value: 4.517702546132841 - type: nauc_map_at_5_diff1 value: 51.688093514126024 - type: nauc_map_at_5_max value: 32.140583227534925 - type: nauc_map_at_5_std value: 5.08711820771354 - type: nauc_mrr_at_1000_diff1 value: 52.79217634259703 - type: nauc_mrr_at_1000_max value: 35.61449692331441 - type: nauc_mrr_at_1000_std value: 9.71451506342203 - type: nauc_mrr_at_100_diff1 value: 52.76716243297642 - type: nauc_mrr_at_100_max value: 35.59808789020979 - type: nauc_mrr_at_100_std value: 9.704336095281075 - type: nauc_mrr_at_10_diff1 value: 52.86337314938788 - type: nauc_mrr_at_10_max value: 35.59686259316265 - type: nauc_mrr_at_10_std value: 9.424813284685497 - type: nauc_mrr_at_1_diff1 value: 60.89531921678486 - type: nauc_mrr_at_1_max value: 35.91468314529445 - type: nauc_mrr_at_1_std value: 9.040164187156382 - type: nauc_mrr_at_20_diff1 value: 52.80752951231851 - type: nauc_mrr_at_20_max value: 35.57463035969631 - type: nauc_mrr_at_20_std value: 9.541448915542269 - type: nauc_mrr_at_3_diff1 value: 54.076978586514755 - type: nauc_mrr_at_3_max value: 35.99756880089376 - type: nauc_mrr_at_3_std value: 8.876285954388155 - type: nauc_mrr_at_5_diff1 value: 52.94654507662424 - type: nauc_mrr_at_5_max value: 35.611041695713894 - type: nauc_mrr_at_5_std value: 8.94668482142343 - type: nauc_ndcg_at_1000_diff1 value: 48.706733998206225 - type: nauc_ndcg_at_1000_max value: 34.26000143617516 - type: nauc_ndcg_at_1000_std value: 10.458871338039593 - type: nauc_ndcg_at_100_diff1 value: 48.03288194102992 - type: nauc_ndcg_at_100_max value: 33.617097137272545 - type: nauc_ndcg_at_100_std value: 10.014497313506247 - type: nauc_ndcg_at_10_diff1 value: 48.769660003285175 - type: nauc_ndcg_at_10_max value: 32.851851368841885 - type: nauc_ndcg_at_10_std value: 7.460362891262815 - type: nauc_ndcg_at_1_diff1 value: 60.89531921678486 - type: nauc_ndcg_at_1_max value: 35.91468314529445 - type: nauc_ndcg_at_1_std value: 9.040164187156382 - type: nauc_ndcg_at_20_diff1 value: 48.68879043144177 - type: nauc_ndcg_at_20_max value: 33.06774534919596 - type: nauc_ndcg_at_20_std value: 8.13085582651415 - type: nauc_ndcg_at_3_diff1 value: 50.98985879559205 - type: nauc_ndcg_at_3_max value: 33.44311811948473 - type: nauc_ndcg_at_3_std value: 5.951958511478065 - type: nauc_ndcg_at_5_diff1 value: 48.817189585064455 - type: nauc_ndcg_at_5_max value: 32.77623327236947 - type: nauc_ndcg_at_5_std value: 5.940331677528127 - type: nauc_precision_at_1000_diff1 value: 5.051795756107663 - type: nauc_precision_at_1000_max value: 24.21322563810257 - type: nauc_precision_at_1000_std value: 21.87719442950226 - type: nauc_precision_at_100_diff1 value: 15.777063208167954 - type: nauc_precision_at_100_max value: 30.870104894832583 - type: nauc_precision_at_100_std value: 25.118265012436176 - type: nauc_precision_at_10_diff1 value: 34.065015434826876 - type: nauc_precision_at_10_max value: 36.38759095988058 - type: nauc_precision_at_10_std value: 18.48947634869304 - type: nauc_precision_at_1_diff1 value: 60.89531921678486 - type: nauc_precision_at_1_max value: 35.91468314529445 - type: nauc_precision_at_1_std value: 9.040164187156382 - type: nauc_precision_at_20_diff1 value: 29.80657169255189 - type: nauc_precision_at_20_max value: 36.49470103206119 - type: nauc_precision_at_20_std value: 21.072761177788518 - type: nauc_precision_at_3_diff1 value: 43.34842688286303 - type: nauc_precision_at_3_max value: 38.49299258863947 - type: nauc_precision_at_3_std value: 11.966654977397777 - type: nauc_precision_at_5_diff1 value: 36.75523769003984 - type: nauc_precision_at_5_max value: 37.27006722184756 - type: nauc_precision_at_5_std value: 13.627277452373662 - type: nauc_recall_at_1000_diff1 value: 22.753849072056813 - type: nauc_recall_at_1000_max value: 40.72656308448736 - type: nauc_recall_at_1000_std value: 42.666697726671146 - type: nauc_recall_at_100_diff1 value: 28.6454368683411 - type: nauc_recall_at_100_max value: 32.24000673731748 - type: nauc_recall_at_100_std value: 22.626295555432048 - type: nauc_recall_at_10_diff1 value: 37.83614105516011 - type: nauc_recall_at_10_max value: 29.851664667699644 - type: nauc_recall_at_10_std value: 8.188931396646032 - type: nauc_recall_at_1_diff1 value: 59.657552965362214 - type: nauc_recall_at_1_max value: 30.802326957153465 - type: nauc_recall_at_1_std value: 3.7728372494699363 - type: nauc_recall_at_20_diff1 value: 36.47396166352664 - type: nauc_recall_at_20_max value: 29.641609075618824 - type: nauc_recall_at_20_std value: 10.228815551744836 - type: nauc_recall_at_3_diff1 value: 43.85557915759296 - type: nauc_recall_at_3_max value: 30.512412885830248 - type: nauc_recall_at_3_std value: 3.1042189846227095 - type: nauc_recall_at_5_diff1 value: 38.466143617226166 - type: nauc_recall_at_5_max value: 29.831242696474597 - type: nauc_recall_at_5_std value: 4.352960295433731 - type: ndcg_at_1 value: 26.687 - type: ndcg_at_10 value: 35.375 - type: ndcg_at_100 value: 40.537 - type: ndcg_at_1000 value: 42.821999999999996 - type: ndcg_at_20 value: 37.391000000000005 - type: ndcg_at_3 value: 31.548 - type: ndcg_at_5 value: 33.827 - type: precision_at_1 value: 26.687 - type: precision_at_10 value: 5.552 - type: precision_at_100 value: 0.8880000000000001 - type: precision_at_1000 value: 0.11499999999999999 - type: precision_at_20 value: 3.29 - type: precision_at_3 value: 13.804 - type: precision_at_5 value: 9.724 - type: recall_at_1 value: 23.547 - type: recall_at_10 value: 45.293 - type: recall_at_100 value: 68.75 - type: recall_at_1000 value: 85.944 - type: recall_at_20 value: 52.724000000000004 - type: recall_at_3 value: 35.219 - type: recall_at_5 value: 40.83 task: type: Retrieval - dataset: config: default name: MTEB CQADupstackTexRetrieval revision: 46989137a86843e03a6195de44b09deda022eec7 split: test type: mteb/cqadupstack-tex metrics: - type: main_score value: 29.464000000000002 - type: map_at_1 value: 17.345 - type: map_at_10 value: 24.736 - type: map_at_100 value: 25.877 - type: map_at_1000 value: 26.012999999999998 - type: map_at_20 value: 25.381999999999998 - type: map_at_3 value: 22.407 - type: map_at_5 value: 23.653 - type: mrr_at_1 value: 20.85340674466621 - type: mrr_at_10 value: 28.456928194540072 - type: mrr_at_100 value: 29.432688505330702 - type: mrr_at_1000 value: 29.511944389924988 - type: mrr_at_20 value: 29.035345402730954 - type: mrr_at_3 value: 26.273227804542376 - type: mrr_at_5 value: 27.475911906400626 - type: nauc_map_at_1000_diff1 value: 38.41850506645076 - type: nauc_map_at_1000_max value: 28.159387527502506 - type: nauc_map_at_1000_std value: 0.40464829576261996 - type: nauc_map_at_100_diff1 value: 38.39947010516607 - type: nauc_map_at_100_max value: 28.119389747412498 - type: nauc_map_at_100_std value: 0.3853149037722529 - type: nauc_map_at_10_diff1 value: 38.810894211784564 - type: nauc_map_at_10_max value: 28.1562967790831 - type: nauc_map_at_10_std value: -0.06389798281342944 - type: nauc_map_at_1_diff1 value: 46.03077824417572 - type: nauc_map_at_1_max value: 27.835838674526652 - type: nauc_map_at_1_std value: -2.2312618530096167 - type: nauc_map_at_20_diff1 value: 38.444510158089805 - type: nauc_map_at_20_max value: 28.017692321962418 - type: nauc_map_at_20_std value: 0.142361876797175 - type: nauc_map_at_3_diff1 value: 40.21093903011263 - type: nauc_map_at_3_max value: 27.89662945345146 - type: nauc_map_at_3_std value: -0.8822596536045023 - type: nauc_map_at_5_diff1 value: 39.09699738378812 - type: nauc_map_at_5_max value: 28.036786133706936 - type: nauc_map_at_5_std value: -0.30364471935320875 - type: nauc_mrr_at_1000_diff1 value: 37.920903186739096 - type: nauc_mrr_at_1000_max value: 30.341964918119942 - type: nauc_mrr_at_1000_std value: 0.12133653932025162 - type: nauc_mrr_at_100_diff1 value: 37.893000931019884 - type: nauc_mrr_at_100_max value: 30.32903692824292 - type: nauc_mrr_at_100_std value: 0.1276009986215775 - type: nauc_mrr_at_10_diff1 value: 38.06140499938785 - type: nauc_mrr_at_10_max value: 30.484179453909732 - type: nauc_mrr_at_10_std value: -0.317866444774404 - type: nauc_mrr_at_1_diff1 value: 45.348598364164886 - type: nauc_mrr_at_1_max value: 30.866871548394347 - type: nauc_mrr_at_1_std value: -2.4227529761902358 - type: nauc_mrr_at_20_diff1 value: 37.84945425263586 - type: nauc_mrr_at_20_max value: 30.308513343655353 - type: nauc_mrr_at_20_std value: -0.07137849023495513 - type: nauc_mrr_at_3_diff1 value: 39.35878483294889 - type: nauc_mrr_at_3_max value: 30.665229724037594 - type: nauc_mrr_at_3_std value: -0.9664014843782173 - type: nauc_mrr_at_5_diff1 value: 38.37838860109641 - type: nauc_mrr_at_5_max value: 30.626565531830796 - type: nauc_mrr_at_5_std value: -0.3844103471733726 - type: nauc_ndcg_at_1000_diff1 value: 35.16938356960978 - type: nauc_ndcg_at_1000_max value: 28.822303299475173 - type: nauc_ndcg_at_1000_std value: 3.5818592250918004 - type: nauc_ndcg_at_100_diff1 value: 34.47959205576021 - type: nauc_ndcg_at_100_max value: 28.33493283343504 - type: nauc_ndcg_at_100_std value: 3.5432148259969285 - type: nauc_ndcg_at_10_diff1 value: 35.874095372057184 - type: nauc_ndcg_at_10_max value: 28.66941141362765 - type: nauc_ndcg_at_10_std value: 0.9200279845334768 - type: nauc_ndcg_at_1_diff1 value: 45.348598364164886 - type: nauc_ndcg_at_1_max value: 30.866871548394347 - type: nauc_ndcg_at_1_std value: -2.4227529761902358 - type: nauc_ndcg_at_20_diff1 value: 34.86108989826789 - type: nauc_ndcg_at_20_max value: 27.97517854587127 - type: nauc_ndcg_at_20_std value: 1.7097315945589184 - type: nauc_ndcg_at_3_diff1 value: 38.29252149856027 - type: nauc_ndcg_at_3_max value: 29.126153934582273 - type: nauc_ndcg_at_3_std value: -0.60152975685728 - type: nauc_ndcg_at_5_diff1 value: 36.498054433082125 - type: nauc_ndcg_at_5_max value: 28.906092518483796 - type: nauc_ndcg_at_5_std value: 0.5077251176639481 - type: nauc_precision_at_1000_diff1 value: 1.3184531416184 - type: nauc_precision_at_1000_max value: 19.684606377304007 - type: nauc_precision_at_1000_std value: 5.302901624843209 - type: nauc_precision_at_100_diff1 value: 7.4256150119233 - type: nauc_precision_at_100_max value: 22.722855082465305 - type: nauc_precision_at_100_std value: 10.526852917202822 - type: nauc_precision_at_10_diff1 value: 21.837059591380257 - type: nauc_precision_at_10_max value: 30.143548889601334 - type: nauc_precision_at_10_std value: 3.231246061820218 - type: nauc_precision_at_1_diff1 value: 45.348598364164886 - type: nauc_precision_at_1_max value: 30.866871548394347 - type: nauc_precision_at_1_std value: -2.4227529761902358 - type: nauc_precision_at_20_diff1 value: 16.580204750030568 - type: nauc_precision_at_20_max value: 26.70707505208188 - type: nauc_precision_at_20_std value: 5.506747683457161 - type: nauc_precision_at_3_diff1 value: 31.1299205489302 - type: nauc_precision_at_3_max value: 31.305848044535338 - type: nauc_precision_at_3_std value: 0.034618129555695674 - type: nauc_precision_at_5_diff1 value: 25.654845802781324 - type: nauc_precision_at_5_max value: 31.195207428853237 - type: nauc_precision_at_5_std value: 1.9886594353332991 - type: nauc_recall_at_1000_diff1 value: 16.07368333583504 - type: nauc_recall_at_1000_max value: 24.655100580014963 - type: nauc_recall_at_1000_std value: 29.043087081370988 - type: nauc_recall_at_100_diff1 value: 18.623100894205148 - type: nauc_recall_at_100_max value: 22.852792969198422 - type: nauc_recall_at_100_std value: 16.36400089993136 - type: nauc_recall_at_10_diff1 value: 27.298600372751636 - type: nauc_recall_at_10_max value: 25.527541634019517 - type: nauc_recall_at_10_std value: 3.4331177908132102 - type: nauc_recall_at_1_diff1 value: 46.03077824417572 - type: nauc_recall_at_1_max value: 27.835838674526652 - type: nauc_recall_at_1_std value: -2.2312618530096167 - type: nauc_recall_at_20_diff1 value: 23.32252313064544 - type: nauc_recall_at_20_max value: 22.641199032492505 - type: nauc_recall_at_20_std value: 6.083452565554306 - type: nauc_recall_at_3_diff1 value: 33.39056241547002 - type: nauc_recall_at_3_max value: 26.627371199082884 - type: nauc_recall_at_3_std value: 0.6888774855297141 - type: nauc_recall_at_5_diff1 value: 29.24415512748389 - type: nauc_recall_at_5_max value: 26.090434026985427 - type: nauc_recall_at_5_std value: 2.701355680777494 - type: ndcg_at_1 value: 20.852999999999998 - type: ndcg_at_10 value: 29.464000000000002 - type: ndcg_at_100 value: 34.775 - type: ndcg_at_1000 value: 37.819 - type: ndcg_at_20 value: 31.55 - type: ndcg_at_3 value: 25.289 - type: ndcg_at_5 value: 27.134999999999998 - type: precision_at_1 value: 20.852999999999998 - type: precision_at_10 value: 5.389 - type: precision_at_100 value: 0.936 - type: precision_at_1000 value: 0.13899999999999998 - type: precision_at_20 value: 3.2809999999999997 - type: precision_at_3 value: 12.044 - type: precision_at_5 value: 8.685 - type: recall_at_1 value: 17.345 - type: recall_at_10 value: 39.932 - type: recall_at_100 value: 63.61599999999999 - type: recall_at_1000 value: 85.14 - type: recall_at_20 value: 47.644999999999996 - type: recall_at_3 value: 28.143 - type: recall_at_5 value: 32.958999999999996 task: type: Retrieval - dataset: config: default name: MTEB CQADupstackUnixRetrieval revision: 6c6430d3a6d36f8d2a829195bc5dc94d7e063e53 split: test type: mteb/cqadupstack-unix metrics: - type: main_score value: 42.229 - type: map_at_1 value: 27.296 - type: map_at_10 value: 36.846000000000004 - type: map_at_100 value: 37.875 - type: map_at_1000 value: 37.988 - type: map_at_20 value: 37.38 - type: map_at_3 value: 33.875 - type: map_at_5 value: 35.737 - type: mrr_at_1 value: 31.80970149253731 - type: mrr_at_10 value: 40.725242833451745 - type: mrr_at_100 value: 41.49709600506051 - type: mrr_at_1000 value: 41.56245177490777 - type: mrr_at_20 value: 41.14234345775092 - type: mrr_at_3 value: 38.15298507462684 - type: mrr_at_5 value: 39.785447761193964 - type: nauc_map_at_1000_diff1 value: 43.75538367380107 - type: nauc_map_at_1000_max value: 38.05056227333834 - type: nauc_map_at_1000_std value: 0.18921144568919948 - type: nauc_map_at_100_diff1 value: 43.77667543106843 - type: nauc_map_at_100_max value: 38.050013068730074 - type: nauc_map_at_100_std value: 0.15537099877922939 - type: nauc_map_at_10_diff1 value: 43.659376456555115 - type: nauc_map_at_10_max value: 37.76818280719845 - type: nauc_map_at_10_std value: -0.05751365580793609 - type: nauc_map_at_1_diff1 value: 50.52298458785999 - type: nauc_map_at_1_max value: 36.66943692165301 - type: nauc_map_at_1_std value: -1.0116916260070048 - type: nauc_map_at_20_diff1 value: 43.60251577616886 - type: nauc_map_at_20_max value: 37.85498492792909 - type: nauc_map_at_20_std value: 0.02824789930513129 - type: nauc_map_at_3_diff1 value: 45.05492770932526 - type: nauc_map_at_3_max value: 37.88442451804782 - type: nauc_map_at_3_std value: -1.9385674410094953 - type: nauc_map_at_5_diff1 value: 43.80463652500317 - type: nauc_map_at_5_max value: 37.84812994589745 - type: nauc_map_at_5_std value: -0.4965865567169233 - type: nauc_mrr_at_1000_diff1 value: 43.30154035744434 - type: nauc_mrr_at_1000_max value: 38.630130905058394 - type: nauc_mrr_at_1000_std value: -0.09862664950221302 - type: nauc_mrr_at_100_diff1 value: 43.295124512783886 - type: nauc_mrr_at_100_max value: 38.631685521949564 - type: nauc_mrr_at_100_std value: -0.10930551847396679 - type: nauc_mrr_at_10_diff1 value: 43.21584928155568 - type: nauc_mrr_at_10_max value: 38.47495723593308 - type: nauc_mrr_at_10_std value: -0.19491385817913934 - type: nauc_mrr_at_1_diff1 value: 49.77296386977506 - type: nauc_mrr_at_1_max value: 38.247307218133955 - type: nauc_mrr_at_1_std value: -0.7255129557116325 - type: nauc_mrr_at_20_diff1 value: 43.1173795956077 - type: nauc_mrr_at_20_max value: 38.530148753210206 - type: nauc_mrr_at_20_std value: -0.08733026838418803 - type: nauc_mrr_at_3_diff1 value: 44.3902368759337 - type: nauc_mrr_at_3_max value: 39.14502536820926 - type: nauc_mrr_at_3_std value: -1.9205759363830195 - type: nauc_mrr_at_5_diff1 value: 43.39334344411832 - type: nauc_mrr_at_5_max value: 38.8749655158438 - type: nauc_mrr_at_5_std value: -0.43021416093426457 - type: nauc_ndcg_at_1000_diff1 value: 41.629511704681384 - type: nauc_ndcg_at_1000_max value: 38.98455196695487 - type: nauc_ndcg_at_1000_std value: 2.5283322394192447 - type: nauc_ndcg_at_100_diff1 value: 41.91727965546967 - type: nauc_ndcg_at_100_max value: 39.06244395765021 - type: nauc_ndcg_at_100_std value: 2.2945063835366737 - type: nauc_ndcg_at_10_diff1 value: 40.97178908043215 - type: nauc_ndcg_at_10_max value: 37.79433728861138 - type: nauc_ndcg_at_10_std value: 1.2281320664622282 - type: nauc_ndcg_at_1_diff1 value: 49.77296386977506 - type: nauc_ndcg_at_1_max value: 38.247307218133955 - type: nauc_ndcg_at_1_std value: -0.7255129557116325 - type: nauc_ndcg_at_20_diff1 value: 40.68478954959323 - type: nauc_ndcg_at_20_max value: 37.99002839374275 - type: nauc_ndcg_at_20_std value: 1.6878408766410797 - type: nauc_ndcg_at_3_diff1 value: 43.17600013118951 - type: nauc_ndcg_at_3_max value: 38.36046345841751 - type: nauc_ndcg_at_3_std value: -2.1648956819129026 - type: nauc_ndcg_at_5_diff1 value: 41.300977363206734 - type: nauc_ndcg_at_5_max value: 38.140880536833365 - type: nauc_ndcg_at_5_std value: 0.2184672115286344 - type: nauc_precision_at_1000_diff1 value: -13.09643401766385 - type: nauc_precision_at_1000_max value: 0.9450851398613254 - type: nauc_precision_at_1000_std value: 3.854576973853421 - type: nauc_precision_at_100_diff1 value: 3.4722518789419494 - type: nauc_precision_at_100_max value: 19.486052927401857 - type: nauc_precision_at_100_std value: 7.388218124534871 - type: nauc_precision_at_10_diff1 value: 18.477374805886022 - type: nauc_precision_at_10_max value: 28.3998207738487 - type: nauc_precision_at_10_std value: 2.7979953301766964 - type: nauc_precision_at_1_diff1 value: 49.77296386977506 - type: nauc_precision_at_1_max value: 38.247307218133955 - type: nauc_precision_at_1_std value: -0.7255129557116325 - type: nauc_precision_at_20_diff1 value: 12.253841360319052 - type: nauc_precision_at_20_max value: 24.85110392443306 - type: nauc_precision_at_20_std value: 3.7311212948659613 - type: nauc_precision_at_3_diff1 value: 32.562607666264014 - type: nauc_precision_at_3_max value: 36.40674316469 - type: nauc_precision_at_3_std value: -3.701236546292251 - type: nauc_precision_at_5_diff1 value: 24.80104527226832 - type: nauc_precision_at_5_max value: 33.69917080736034 - type: nauc_precision_at_5_std value: 1.3236747342761015 - type: nauc_recall_at_1000_diff1 value: 24.13767701978169 - type: nauc_recall_at_1000_max value: 50.49269211134329 - type: nauc_recall_at_1000_std value: 47.08476397391495 - type: nauc_recall_at_100_diff1 value: 36.11087195903443 - type: nauc_recall_at_100_max value: 40.79223196983678 - type: nauc_recall_at_100_std value: 14.390360400534908 - type: nauc_recall_at_10_diff1 value: 31.317148127487908 - type: nauc_recall_at_10_max value: 34.436206830017284 - type: nauc_recall_at_10_std value: 6.245127051630293 - type: nauc_recall_at_1_diff1 value: 50.52298458785999 - type: nauc_recall_at_1_max value: 36.66943692165301 - type: nauc_recall_at_1_std value: -1.0116916260070048 - type: nauc_recall_at_20_diff1 value: 30.00266971037392 - type: nauc_recall_at_20_max value: 34.963620751107484 - type: nauc_recall_at_20_std value: 8.42010232962911 - type: nauc_recall_at_3_diff1 value: 38.402162193668296 - type: nauc_recall_at_3_max value: 37.495379694986894 - type: nauc_recall_at_3_std value: -2.7440958505645643 - type: nauc_recall_at_5_diff1 value: 33.08317576475561 - type: nauc_recall_at_5_max value: 36.08067909558661 - type: nauc_recall_at_5_std value: 2.9313830821919717 - type: ndcg_at_1 value: 31.81 - type: ndcg_at_10 value: 42.229 - type: ndcg_at_100 value: 47.016000000000005 - type: ndcg_at_1000 value: 49.544 - type: ndcg_at_20 value: 43.877 - type: ndcg_at_3 value: 36.985 - type: ndcg_at_5 value: 39.794000000000004 - type: precision_at_1 value: 31.81 - type: precision_at_10 value: 7.136000000000001 - type: precision_at_100 value: 1.0710000000000002 - type: precision_at_1000 value: 0.13999999999999999 - type: precision_at_20 value: 4.067 - type: precision_at_3 value: 16.698 - type: precision_at_5 value: 12.034 - type: recall_at_1 value: 27.296 - type: recall_at_10 value: 54.607000000000006 - type: recall_at_100 value: 75.65899999999999 - type: recall_at_1000 value: 93.366 - type: recall_at_20 value: 60.373 - type: recall_at_3 value: 40.394000000000005 - type: recall_at_5 value: 47.448 task: type: Retrieval - dataset: config: default name: MTEB CQADupstackWebmastersRetrieval revision: 160c094312a0e1facb97e55eeddb698c0abe3571 split: test type: mteb/cqadupstack-webmasters metrics: - type: main_score value: 40.617 - type: map_at_1 value: 26.124000000000002 - type: map_at_10 value: 34.857 - type: map_at_100 value: 36.589 - type: map_at_1000 value: 36.832 - type: map_at_20 value: 35.808 - type: map_at_3 value: 31.635 - type: map_at_5 value: 33.225 - type: mrr_at_1 value: 30.8300395256917 - type: mrr_at_10 value: 39.024876090093485 - type: mrr_at_100 value: 40.112831574778625 - type: mrr_at_1000 value: 40.172542355918054 - type: mrr_at_20 value: 39.70461282542164 - type: mrr_at_3 value: 36.429512516469046 - type: mrr_at_5 value: 37.86231884057971 - type: nauc_map_at_1000_diff1 value: 42.17385761787554 - type: nauc_map_at_1000_max value: 29.023497819147003 - type: nauc_map_at_1000_std value: 7.135398677866782 - type: nauc_map_at_100_diff1 value: 42.26672776757606 - type: nauc_map_at_100_max value: 29.232142243363228 - type: nauc_map_at_100_std value: 6.877840055692755 - type: nauc_map_at_10_diff1 value: 42.877953607649324 - type: nauc_map_at_10_max value: 29.46874782942982 - type: nauc_map_at_10_std value: 5.031611824650058 - type: nauc_map_at_1_diff1 value: 50.063501481714425 - type: nauc_map_at_1_max value: 26.941170343192383 - type: nauc_map_at_1_std value: 1.2783766588617693 - type: nauc_map_at_20_diff1 value: 42.44190895193996 - type: nauc_map_at_20_max value: 29.46253769495084 - type: nauc_map_at_20_std value: 6.146541588421648 - type: nauc_map_at_3_diff1 value: 44.21913152729423 - type: nauc_map_at_3_max value: 28.744071788355395 - type: nauc_map_at_3_std value: 3.616132053693665 - type: nauc_map_at_5_diff1 value: 43.187186127431275 - type: nauc_map_at_5_max value: 29.016108070111184 - type: nauc_map_at_5_std value: 4.684957715416093 - type: nauc_mrr_at_1000_diff1 value: 39.22437734736831 - type: nauc_mrr_at_1000_max value: 30.643434055644743 - type: nauc_mrr_at_1000_std value: 8.491481853057229 - type: nauc_mrr_at_100_diff1 value: 39.203338294322144 - type: nauc_mrr_at_100_max value: 30.64454944640062 - type: nauc_mrr_at_100_std value: 8.541568521609001 - type: nauc_mrr_at_10_diff1 value: 39.327335076290325 - type: nauc_mrr_at_10_max value: 30.870046331420216 - type: nauc_mrr_at_10_std value: 8.192608946125315 - type: nauc_mrr_at_1_diff1 value: 43.61118724145502 - type: nauc_mrr_at_1_max value: 28.90069833045206 - type: nauc_mrr_at_1_std value: 4.479542769926193 - type: nauc_mrr_at_20_diff1 value: 39.13464939320264 - type: nauc_mrr_at_20_max value: 30.733559708742213 - type: nauc_mrr_at_20_std value: 8.600789691544723 - type: nauc_mrr_at_3_diff1 value: 39.44504307119285 - type: nauc_mrr_at_3_max value: 30.44308080500852 - type: nauc_mrr_at_3_std value: 6.875960856302759 - type: nauc_mrr_at_5_diff1 value: 39.024350509778614 - type: nauc_mrr_at_5_max value: 30.564593983912243 - type: nauc_mrr_at_5_std value: 8.114542797004818 - type: nauc_ndcg_at_1000_diff1 value: 40.17841832753416 - type: nauc_ndcg_at_1000_max value: 30.41408114991541 - type: nauc_ndcg_at_1000_std value: 10.460324209610734 - type: nauc_ndcg_at_100_diff1 value: 39.290496874327985 - type: nauc_ndcg_at_100_max value: 30.13490396441187 - type: nauc_ndcg_at_100_std value: 11.061850479505644 - type: nauc_ndcg_at_10_diff1 value: 39.92759489069634 - type: nauc_ndcg_at_10_max value: 30.2024288985298 - type: nauc_ndcg_at_10_std value: 8.24558057585065 - type: nauc_ndcg_at_1_diff1 value: 43.61118724145502 - type: nauc_ndcg_at_1_max value: 28.90069833045206 - type: nauc_ndcg_at_1_std value: 4.479542769926193 - type: nauc_ndcg_at_20_diff1 value: 39.29435506522245 - type: nauc_ndcg_at_20_max value: 30.33893728800384 - type: nauc_ndcg_at_20_std value: 10.545750826464893 - type: nauc_ndcg_at_3_diff1 value: 40.35926833516799 - type: nauc_ndcg_at_3_max value: 28.865626442014204 - type: nauc_ndcg_at_3_std value: 6.095820315868252 - type: nauc_ndcg_at_5_diff1 value: 39.624068053041924 - type: nauc_ndcg_at_5_max value: 29.298680802211013 - type: nauc_ndcg_at_5_std value: 7.981261205063178 - type: nauc_precision_at_1000_diff1 value: -15.437017626426188 - type: nauc_precision_at_1000_max value: -11.466653763751436 - type: nauc_precision_at_1000_std value: 27.203862461973344 - type: nauc_precision_at_100_diff1 value: -10.720493705726689 - type: nauc_precision_at_100_max value: -2.2729520096186393 - type: nauc_precision_at_100_std value: 33.378688533496415 - type: nauc_precision_at_10_diff1 value: 10.517312952676674 - type: nauc_precision_at_10_max value: 21.759943967624206 - type: nauc_precision_at_10_std value: 18.479226770461366 - type: nauc_precision_at_1_diff1 value: 43.61118724145502 - type: nauc_precision_at_1_max value: 28.90069833045206 - type: nauc_precision_at_1_std value: 4.479542769926193 - type: nauc_precision_at_20_diff1 value: 0.22630058814929357 - type: nauc_precision_at_20_max value: 14.340458300248256 - type: nauc_precision_at_20_std value: 29.197777051584495 - type: nauc_precision_at_3_diff1 value: 24.684569742393748 - type: nauc_precision_at_3_max value: 26.847701196729407 - type: nauc_precision_at_3_std value: 9.803491488881253 - type: nauc_precision_at_5_diff1 value: 18.218141491873933 - type: nauc_precision_at_5_max value: 25.63292715320136 - type: nauc_precision_at_5_std value: 15.125718294075579 - type: nauc_recall_at_1000_diff1 value: 45.64721815441147 - type: nauc_recall_at_1000_max value: 48.216562415882606 - type: nauc_recall_at_1000_std value: 40.18167967343722 - type: nauc_recall_at_100_diff1 value: 27.550190465714813 - type: nauc_recall_at_100_max value: 29.636326305334048 - type: nauc_recall_at_100_std value: 30.20966431974901 - type: nauc_recall_at_10_diff1 value: 34.30589013251178 - type: nauc_recall_at_10_max value: 30.383485547646043 - type: nauc_recall_at_10_std value: 10.0628614956005 - type: nauc_recall_at_1_diff1 value: 50.063501481714425 - type: nauc_recall_at_1_max value: 26.941170343192383 - type: nauc_recall_at_1_std value: 1.2783766588617693 - type: nauc_recall_at_20_diff1 value: 30.74543679537666 - type: nauc_recall_at_20_max value: 30.138740866468787 - type: nauc_recall_at_20_std value: 19.995537838999724 - type: nauc_recall_at_3_diff1 value: 38.652151861880654 - type: nauc_recall_at_3_max value: 28.806612417729582 - type: nauc_recall_at_3_std value: 5.321890758808099 - type: nauc_recall_at_5_diff1 value: 34.689245376303504 - type: nauc_recall_at_5_max value: 28.89342892932615 - type: nauc_recall_at_5_std value: 9.12730045136017 - type: ndcg_at_1 value: 30.830000000000002 - type: ndcg_at_10 value: 40.617 - type: ndcg_at_100 value: 46.914 - type: ndcg_at_1000 value: 49.319 - type: ndcg_at_20 value: 43.206 - type: ndcg_at_3 value: 35.271 - type: ndcg_at_5 value: 37.436 - type: precision_at_1 value: 30.830000000000002 - type: precision_at_10 value: 7.767 - type: precision_at_100 value: 1.603 - type: precision_at_1000 value: 0.243 - type: precision_at_20 value: 5.059 - type: precision_at_3 value: 16.337 - type: precision_at_5 value: 11.66 - type: recall_at_1 value: 26.124000000000002 - type: recall_at_10 value: 52.015 - type: recall_at_100 value: 79.729 - type: recall_at_1000 value: 94.476 - type: recall_at_20 value: 61.519999999999996 - type: recall_at_3 value: 37.037 - type: recall_at_5 value: 43.034 task: type: Retrieval - dataset: config: default name: MTEB CQADupstackWordpressRetrieval revision: 4ffe81d471b1924886b33c7567bfb200e9eec5c4 split: test type: mteb/cqadupstack-wordpress metrics: - type: main_score value: 33.94 - type: map_at_1 value: 21.728 - type: map_at_10 value: 29.213 - type: map_at_100 value: 30.275000000000002 - type: map_at_1000 value: 30.387999999999998 - type: map_at_20 value: 29.921 - type: map_at_3 value: 26.678 - type: map_at_5 value: 27.97 - type: mrr_at_1 value: 24.029574861367838 - type: mrr_at_10 value: 31.411040694774528 - type: mrr_at_100 value: 32.37670039416044 - type: mrr_at_1000 value: 32.45828597542603 - type: mrr_at_20 value: 32.066710418311054 - type: mrr_at_3 value: 29.174368453481215 - type: mrr_at_5 value: 30.218730745532962 - type: nauc_map_at_1000_diff1 value: 31.91257362997741 - type: nauc_map_at_1000_max value: 30.88489579944952 - type: nauc_map_at_1000_std value: -0.8378972209872841 - type: nauc_map_at_100_diff1 value: 31.91012136001019 - type: nauc_map_at_100_max value: 30.89495455322903 - type: nauc_map_at_100_std value: -0.8363099421850583 - type: nauc_map_at_10_diff1 value: 32.321541752194044 - type: nauc_map_at_10_max value: 30.478856413691663 - type: nauc_map_at_10_std value: -1.7937789797513617 - type: nauc_map_at_1_diff1 value: 39.036529785500946 - type: nauc_map_at_1_max value: 30.15527995919205 - type: nauc_map_at_1_std value: -3.0753665573586875 - type: nauc_map_at_20_diff1 value: 31.995975321229412 - type: nauc_map_at_20_max value: 30.841371991167055 - type: nauc_map_at_20_std value: -1.0599571122276743 - type: nauc_map_at_3_diff1 value: 33.37828040602051 - type: nauc_map_at_3_max value: 30.63773296692759 - type: nauc_map_at_3_std value: -2.6306505462108145 - type: nauc_map_at_5_diff1 value: 32.33262472170732 - type: nauc_map_at_5_max value: 30.968294579332984 - type: nauc_map_at_5_std value: -2.2441870438191134 - type: nauc_mrr_at_1000_diff1 value: 32.924621843184504 - type: nauc_mrr_at_1000_max value: 33.163623964941976 - type: nauc_mrr_at_1000_std value: 0.16588136365013925 - type: nauc_mrr_at_100_diff1 value: 32.906505756942536 - type: nauc_mrr_at_100_max value: 33.17000246238509 - type: nauc_mrr_at_100_std value: 0.17008029070490974 - type: nauc_mrr_at_10_diff1 value: 33.29141185817138 - type: nauc_mrr_at_10_max value: 33.07902410006761 - type: nauc_mrr_at_10_std value: -0.5588437886232123 - type: nauc_mrr_at_1_diff1 value: 39.41491848273575 - type: nauc_mrr_at_1_max value: 33.1045906492973 - type: nauc_mrr_at_1_std value: -1.499033772891368 - type: nauc_mrr_at_20_diff1 value: 33.03454429443921 - type: nauc_mrr_at_20_max value: 33.23117853960766 - type: nauc_mrr_at_20_std value: 0.026450200924183646 - type: nauc_mrr_at_3_diff1 value: 34.273859138345905 - type: nauc_mrr_at_3_max value: 33.709490350419216 - type: nauc_mrr_at_3_std value: -1.0280318099262806 - type: nauc_mrr_at_5_diff1 value: 33.22838031675215 - type: nauc_mrr_at_5_max value: 33.47021788245831 - type: nauc_mrr_at_5_std value: -1.020392885760047 - type: nauc_ndcg_at_1000_diff1 value: 29.228313546730394 - type: nauc_ndcg_at_1000_max value: 31.30181847993907 - type: nauc_ndcg_at_1000_std value: 2.4004671163706286 - type: nauc_ndcg_at_100_diff1 value: 29.007140166456352 - type: nauc_ndcg_at_100_max value: 31.177548656556088 - type: nauc_ndcg_at_100_std value: 2.84104689095153 - type: nauc_ndcg_at_10_diff1 value: 30.333521308398282 - type: nauc_ndcg_at_10_max value: 30.443915052520797 - type: nauc_ndcg_at_10_std value: -0.7776378511941929 - type: nauc_ndcg_at_1_diff1 value: 39.41491848273575 - type: nauc_ndcg_at_1_max value: 33.1045906492973 - type: nauc_ndcg_at_1_std value: -1.499033772891368 - type: nauc_ndcg_at_20_diff1 value: 29.35528623170515 - type: nauc_ndcg_at_20_max value: 31.251908593602046 - type: nauc_ndcg_at_20_std value: 1.5335554093420756 - type: nauc_ndcg_at_3_diff1 value: 32.00742763587945 - type: nauc_ndcg_at_3_max value: 31.43411749352576 - type: nauc_ndcg_at_3_std value: -2.161005612286732 - type: nauc_ndcg_at_5_diff1 value: 30.218028443882382 - type: nauc_ndcg_at_5_max value: 31.436372820890888 - type: nauc_ndcg_at_5_std value: -1.8970324616117942 - type: nauc_precision_at_1000_diff1 value: -15.474984333696856 - type: nauc_precision_at_1000_max value: -0.5988176438045155 - type: nauc_precision_at_1000_std value: 9.649508553406482 - type: nauc_precision_at_100_diff1 value: 6.827122527618587 - type: nauc_precision_at_100_max value: 25.38874258318603 - type: nauc_precision_at_100_std value: 19.74692016049829 - type: nauc_precision_at_10_diff1 value: 21.924392614008934 - type: nauc_precision_at_10_max value: 30.50809713248987 - type: nauc_precision_at_10_std value: 5.182344376975003 - type: nauc_precision_at_1_diff1 value: 39.41491848273575 - type: nauc_precision_at_1_max value: 33.1045906492973 - type: nauc_precision_at_1_std value: -1.499033772891368 - type: nauc_precision_at_20_diff1 value: 17.384644579807794 - type: nauc_precision_at_20_max value: 31.910020552998546 - type: nauc_precision_at_20_std value: 14.10235751314262 - type: nauc_precision_at_3_diff1 value: 27.21660611201887 - type: nauc_precision_at_3_max value: 32.731203308832185 - type: nauc_precision_at_3_std value: -1.2935765023235843 - type: nauc_precision_at_5_diff1 value: 23.103364778326657 - type: nauc_precision_at_5_max value: 33.42544453577581 - type: nauc_precision_at_5_std value: 0.539665839883537 - type: nauc_recall_at_1000_diff1 value: -2.224955557104963 - type: nauc_recall_at_1000_max value: 23.905681787331996 - type: nauc_recall_at_1000_std value: 29.343916482872185 - type: nauc_recall_at_100_diff1 value: 15.113380525840887 - type: nauc_recall_at_100_max value: 25.613295770648165 - type: nauc_recall_at_100_std value: 18.315582271550408 - type: nauc_recall_at_10_diff1 value: 23.40683183845192 - type: nauc_recall_at_10_max value: 26.57810440107745 - type: nauc_recall_at_10_std value: 0.9875785918360439 - type: nauc_recall_at_1_diff1 value: 39.036529785500946 - type: nauc_recall_at_1_max value: 30.15527995919205 - type: nauc_recall_at_1_std value: -3.0753665573586875 - type: nauc_recall_at_20_diff1 value: 18.78081005808975 - type: nauc_recall_at_20_max value: 28.341844976465215 - type: nauc_recall_at_20_std value: 9.15244411341802 - type: nauc_recall_at_3_diff1 value: 28.011427372719595 - type: nauc_recall_at_3_max value: 30.42588662301828 - type: nauc_recall_at_3_std value: -2.2738647119625295 - type: nauc_recall_at_5_diff1 value: 23.97361585809799 - type: nauc_recall_at_5_max value: 30.419498034627367 - type: nauc_recall_at_5_std value: -1.9202931420769696 - type: ndcg_at_1 value: 24.03 - type: ndcg_at_10 value: 33.94 - type: ndcg_at_100 value: 38.911 - type: ndcg_at_1000 value: 41.732 - type: ndcg_at_20 value: 36.327999999999996 - type: ndcg_at_3 value: 29.011 - type: ndcg_at_5 value: 31.020999999999997 - type: precision_at_1 value: 24.03 - type: precision_at_10 value: 5.36 - type: precision_at_100 value: 0.8410000000000001 - type: precision_at_1000 value: 0.11900000000000001 - type: precision_at_20 value: 3.2439999999999998 - type: precision_at_3 value: 12.323 - type: precision_at_5 value: 8.613999999999999 - type: recall_at_1 value: 21.728 - type: recall_at_10 value: 46.562 - type: recall_at_100 value: 69.16799999999999 - type: recall_at_1000 value: 90.325 - type: recall_at_20 value: 55.644000000000005 - type: recall_at_3 value: 32.772 - type: recall_at_5 value: 37.683 task: type: Retrieval - dataset: config: default name: MTEB ClimateFEVER revision: 47f2ac6acb640fc46020b02a5b59fdda04d39380 split: test type: mteb/climate-fever metrics: - type: main_score value: 25.854 - type: map_at_1 value: 10.267 - type: map_at_10 value: 18.04 - type: map_at_100 value: 19.777 - type: map_at_1000 value: 19.958000000000002 - type: map_at_20 value: 19.008 - type: map_at_3 value: 14.879999999999999 - type: map_at_5 value: 16.485 - type: mrr_at_1 value: 22.996742671009773 - type: mrr_at_10 value: 34.10619926580836 - type: mrr_at_100 value: 35.10535379461663 - type: mrr_at_1000 value: 35.15755884184264 - type: mrr_at_20 value: 34.75454375956081 - type: mrr_at_3 value: 30.597176981541768 - type: mrr_at_5 value: 32.6753528773072 - type: nauc_map_at_1000_diff1 value: 16.718537725864763 - type: nauc_map_at_1000_max value: 39.502981434057595 - type: nauc_map_at_1000_std value: 22.12042839190954 - type: nauc_map_at_100_diff1 value: 16.75220577723001 - type: nauc_map_at_100_max value: 39.48722479833852 - type: nauc_map_at_100_std value: 22.055881777406448 - type: nauc_map_at_10_diff1 value: 16.874316455401654 - type: nauc_map_at_10_max value: 39.596829557733344 - type: nauc_map_at_10_std value: 20.473758607475666 - type: nauc_map_at_1_diff1 value: 23.080079172466412 - type: nauc_map_at_1_max value: 36.199027660019325 - type: nauc_map_at_1_std value: 14.511081393076179 - type: nauc_map_at_20_diff1 value: 16.839381398924182 - type: nauc_map_at_20_max value: 39.67988867337892 - type: nauc_map_at_20_std value: 21.35532096892699 - type: nauc_map_at_3_diff1 value: 18.312687041708102 - type: nauc_map_at_3_max value: 38.769416119220736 - type: nauc_map_at_3_std value: 16.175068336485026 - type: nauc_map_at_5_diff1 value: 17.69288278300591 - type: nauc_map_at_5_max value: 39.2535585829467 - type: nauc_map_at_5_std value: 18.460583770602113 - type: nauc_mrr_at_1000_diff1 value: 17.91231859721048 - type: nauc_mrr_at_1000_max value: 35.48664780287595 - type: nauc_mrr_at_1000_std value: 22.024407060245295 - type: nauc_mrr_at_100_diff1 value: 17.90154896033313 - type: nauc_mrr_at_100_max value: 35.48493858099936 - type: nauc_mrr_at_100_std value: 22.031137347443284 - type: nauc_mrr_at_10_diff1 value: 17.764970868121992 - type: nauc_mrr_at_10_max value: 35.647414618204834 - type: nauc_mrr_at_10_std value: 22.177693762457217 - type: nauc_mrr_at_1_diff1 value: 20.72060270445967 - type: nauc_mrr_at_1_max value: 32.05796435073919 - type: nauc_mrr_at_1_std value: 17.801001796830647 - type: nauc_mrr_at_20_diff1 value: 17.81229976929074 - type: nauc_mrr_at_20_max value: 35.508834038837286 - type: nauc_mrr_at_20_std value: 22.03930127358993 - type: nauc_mrr_at_3_diff1 value: 18.184760750984967 - type: nauc_mrr_at_3_max value: 35.06640214914882 - type: nauc_mrr_at_3_std value: 19.85666178099386 - type: nauc_mrr_at_5_diff1 value: 17.94734753821123 - type: nauc_mrr_at_5_max value: 35.5362671414369 - type: nauc_mrr_at_5_std value: 21.65318475967923 - type: nauc_ndcg_at_1000_diff1 value: 14.873923990787 - type: nauc_ndcg_at_1000_max value: 38.880238913938264 - type: nauc_ndcg_at_1000_std value: 28.053954097672957 - type: nauc_ndcg_at_100_diff1 value: 14.878869135136199 - type: nauc_ndcg_at_100_max value: 38.4728834388698 - type: nauc_ndcg_at_100_std value: 27.260605585380198 - type: nauc_ndcg_at_10_diff1 value: 15.052663192838656 - type: nauc_ndcg_at_10_max value: 39.31146423709117 - type: nauc_ndcg_at_10_std value: 23.825559445880963 - type: nauc_ndcg_at_1_diff1 value: 20.72060270445967 - type: nauc_ndcg_at_1_max value: 32.05796435073919 - type: nauc_ndcg_at_1_std value: 17.801001796830647 - type: nauc_ndcg_at_20_diff1 value: 14.949446910080416 - type: nauc_ndcg_at_20_max value: 39.12075299095604 - type: nauc_ndcg_at_20_std value: 25.17649714577982 - type: nauc_ndcg_at_3_diff1 value: 17.117859449815708 - type: nauc_ndcg_at_3_max value: 37.86742070994964 - type: nauc_ndcg_at_3_std value: 17.929095022194748 - type: nauc_ndcg_at_5_diff1 value: 16.552568744225216 - type: nauc_ndcg_at_5_max value: 39.07328909408646 - type: nauc_ndcg_at_5_std value: 21.087837182021957 - type: nauc_precision_at_1000_diff1 value: -2.870078971356406 - type: nauc_precision_at_1000_max value: 7.02871444829994 - type: nauc_precision_at_1000_std value: 23.899933947819544 - type: nauc_precision_at_100_diff1 value: 1.5886111188050933 - type: nauc_precision_at_100_max value: 15.935337497887625 - type: nauc_precision_at_100_std value: 29.556371474338345 - type: nauc_precision_at_10_diff1 value: 6.300177923780286 - type: nauc_precision_at_10_max value: 31.243660622639485 - type: nauc_precision_at_10_std value: 29.554919355292093 - type: nauc_precision_at_1_diff1 value: 20.72060270445967 - type: nauc_precision_at_1_max value: 32.05796435073919 - type: nauc_precision_at_1_std value: 17.801001796830647 - type: nauc_precision_at_20_diff1 value: 4.669392584982594 - type: nauc_precision_at_20_max value: 25.761327413181178 - type: nauc_precision_at_20_std value: 29.741403307857166 - type: nauc_precision_at_3_diff1 value: 13.367853392936027 - type: nauc_precision_at_3_max value: 36.957826896424336 - type: nauc_precision_at_3_std value: 21.740373439631725 - type: nauc_precision_at_5_diff1 value: 10.690045616058065 - type: nauc_precision_at_5_max value: 34.71275252243625 - type: nauc_precision_at_5_std value: 26.90693165393696 - type: nauc_recall_at_1000_diff1 value: 5.712445342001347 - type: nauc_recall_at_1000_max value: 30.531136218449 - type: nauc_recall_at_1000_std value: 39.216291284266546 - type: nauc_recall_at_100_diff1 value: 6.9984637463611925 - type: nauc_recall_at_100_max value: 29.566627607364083 - type: nauc_recall_at_100_std value: 31.079777791452766 - type: nauc_recall_at_10_diff1 value: 8.262518659498706 - type: nauc_recall_at_10_max value: 35.13566657953036 - type: nauc_recall_at_10_std value: 24.290738932523727 - type: nauc_recall_at_1_diff1 value: 23.080079172466412 - type: nauc_recall_at_1_max value: 36.199027660019325 - type: nauc_recall_at_1_std value: 14.511081393076179 - type: nauc_recall_at_20_diff1 value: 7.680549244887501 - type: nauc_recall_at_20_max value: 33.289199278919206 - type: nauc_recall_at_20_std value: 25.97835574263474 - type: nauc_recall_at_3_diff1 value: 13.66331680167438 - type: nauc_recall_at_3_max value: 37.36441061109965 - type: nauc_recall_at_3_std value: 15.4241070336601 - type: nauc_recall_at_5_diff1 value: 12.173354391955048 - type: nauc_recall_at_5_max value: 37.13679975852594 - type: nauc_recall_at_5_std value: 20.394479290376545 - type: ndcg_at_1 value: 22.997 - type: ndcg_at_10 value: 25.854 - type: ndcg_at_100 value: 32.74 - type: ndcg_at_1000 value: 36.187000000000005 - type: ndcg_at_20 value: 28.692 - type: ndcg_at_3 value: 20.575 - type: ndcg_at_5 value: 22.55 - type: precision_at_1 value: 22.997 - type: precision_at_10 value: 8.189 - type: precision_at_100 value: 1.541 - type: precision_at_1000 value: 0.218 - type: precision_at_20 value: 5.296 - type: precision_at_3 value: 15.244 - type: precision_at_5 value: 12.052 - type: recall_at_1 value: 10.267 - type: recall_at_10 value: 32.125 - type: recall_at_100 value: 55.696999999999996 - type: recall_at_1000 value: 75.074 - type: recall_at_20 value: 40.193 - type: recall_at_3 value: 19.509 - type: recall_at_5 value: 24.747 task: type: Retrieval - dataset: config: default name: MTEB DBPedia revision: c0f706b76e590d620bd6618b3ca8efdd34e2d659 split: test type: mteb/dbpedia metrics: - type: main_score value: 38.936 - type: map_at_1 value: 7.969999999999999 - type: map_at_10 value: 18.916 - type: map_at_100 value: 26.464 - type: map_at_1000 value: 28.203 - type: map_at_20 value: 21.671000000000003 - type: map_at_3 value: 13.544 - type: map_at_5 value: 15.964 - type: mrr_at_1 value: 62.25000000000001 - type: mrr_at_10 value: 72.06656746031746 - type: mrr_at_100 value: 72.38229975273143 - type: mrr_at_1000 value: 72.38878819640294 - type: mrr_at_20 value: 72.28867599870888 - type: mrr_at_3 value: 69.87500000000001 - type: mrr_at_5 value: 71.36250000000001 - type: nauc_map_at_1000_diff1 value: 19.321937510366077 - type: nauc_map_at_1000_max value: 13.747024166838576 - type: nauc_map_at_1000_std value: 20.868788180619273 - type: nauc_map_at_100_diff1 value: 19.64393142562319 - type: nauc_map_at_100_max value: 10.251028144343387 - type: nauc_map_at_100_std value: 18.068372645123112 - type: nauc_map_at_10_diff1 value: 23.84916025839186 - type: nauc_map_at_10_max value: 0.09170851785343646 - type: nauc_map_at_10_std value: -0.9386497395956033 - type: nauc_map_at_1_diff1 value: 35.48615861613537 - type: nauc_map_at_1_max value: -4.968376038142286 - type: nauc_map_at_1_std value: -8.722029416996381 - type: nauc_map_at_20_diff1 value: 22.041254266944954 - type: nauc_map_at_20_max value: 3.54673862289231 - type: nauc_map_at_20_std value: 6.076866601495746 - type: nauc_map_at_3_diff1 value: 27.930757708983773 - type: nauc_map_at_3_max value: -4.53116277437896 - type: nauc_map_at_3_std value: -7.364340365588032 - type: nauc_map_at_5_diff1 value: 24.465395628826034 - type: nauc_map_at_5_max value: -3.105817257141636 - type: nauc_map_at_5_std value: -5.855744275070062 - type: nauc_mrr_at_1000_diff1 value: 42.514933484267644 - type: nauc_mrr_at_1000_max value: 40.515988358376646 - type: nauc_mrr_at_1000_std value: 30.22699534554659 - type: nauc_mrr_at_100_diff1 value: 42.51475590269715 - type: nauc_mrr_at_100_max value: 40.49923698224623 - type: nauc_mrr_at_100_std value: 30.235830229223904 - type: nauc_mrr_at_10_diff1 value: 42.45982313098615 - type: nauc_mrr_at_10_max value: 40.23468589619149 - type: nauc_mrr_at_10_std value: 30.217961535871073 - type: nauc_mrr_at_1_diff1 value: 45.27134837755035 - type: nauc_mrr_at_1_max value: 38.4098873035926 - type: nauc_mrr_at_1_std value: 25.42449495242179 - type: nauc_mrr_at_20_diff1 value: 42.50561799373592 - type: nauc_mrr_at_20_max value: 40.46341284202719 - type: nauc_mrr_at_20_std value: 30.176989437307306 - type: nauc_mrr_at_3_diff1 value: 42.015465693213386 - type: nauc_mrr_at_3_max value: 40.88173938795508 - type: nauc_mrr_at_3_std value: 31.535958619028225 - type: nauc_mrr_at_5_diff1 value: 42.13527304829269 - type: nauc_mrr_at_5_max value: 40.612539610295975 - type: nauc_mrr_at_5_std value: 30.670846095147496 - type: nauc_ndcg_at_1000_diff1 value: 20.55967654606238 - type: nauc_ndcg_at_1000_max value: 24.316462778279867 - type: nauc_ndcg_at_1000_std value: 31.496336109850247 - type: nauc_ndcg_at_100_diff1 value: 20.631201886740783 - type: nauc_ndcg_at_100_max value: 13.797070619958301 - type: nauc_ndcg_at_100_std value: 24.805397171967755 - type: nauc_ndcg_at_10_diff1 value: 24.434849609019665 - type: nauc_ndcg_at_10_max value: 18.074612480062648 - type: nauc_ndcg_at_10_std value: 19.17394905361306 - type: nauc_ndcg_at_1_diff1 value: 36.96000425769715 - type: nauc_ndcg_at_1_max value: 29.509211705119004 - type: nauc_ndcg_at_1_std value: 18.50980600867504 - type: nauc_ndcg_at_20_diff1 value: 22.868106188391877 - type: nauc_ndcg_at_20_max value: 12.64309423251832 - type: nauc_ndcg_at_20_std value: 18.94830361891266 - type: nauc_ndcg_at_3_diff1 value: 27.93885229724602 - type: nauc_ndcg_at_3_max value: 24.97470001983418 - type: nauc_ndcg_at_3_std value: 19.53721404824316 - type: nauc_ndcg_at_5_diff1 value: 24.096100442606105 - type: nauc_ndcg_at_5_max value: 21.017433702322183 - type: nauc_ndcg_at_5_std value: 18.03214925832656 - type: nauc_precision_at_1000_diff1 value: -5.070197509753125 - type: nauc_precision_at_1000_max value: 38.94682716070116 - type: nauc_precision_at_1000_std value: 13.85365496749175 - type: nauc_precision_at_100_diff1 value: -6.234968963931368 - type: nauc_precision_at_100_max value: 30.591823702241243 - type: nauc_precision_at_100_std value: 40.95192411908243 - type: nauc_precision_at_10_diff1 value: 3.7146822473142125 - type: nauc_precision_at_10_max value: 35.234593007969735 - type: nauc_precision_at_10_std value: 39.26820665194708 - type: nauc_precision_at_1_diff1 value: 45.27134837755035 - type: nauc_precision_at_1_max value: 38.4098873035926 - type: nauc_precision_at_1_std value: 25.42449495242179 - type: nauc_precision_at_20_diff1 value: -1.1101702192957978 - type: nauc_precision_at_20_max value: 32.05616068273549 - type: nauc_precision_at_20_std value: 44.191729349771 - type: nauc_precision_at_3_diff1 value: 13.654921924572067 - type: nauc_precision_at_3_max value: 34.080562463989054 - type: nauc_precision_at_3_std value: 30.352925731706453 - type: nauc_precision_at_5_diff1 value: 5.007711615213191 - type: nauc_precision_at_5_max value: 33.571297387924865 - type: nauc_precision_at_5_std value: 33.62738186220206 - type: nauc_recall_at_1000_diff1 value: 8.24800794780627 - type: nauc_recall_at_1000_max value: 17.19134351585966 - type: nauc_recall_at_1000_std value: 35.214343810310936 - type: nauc_recall_at_100_diff1 value: 9.207152847999645 - type: nauc_recall_at_100_max value: 2.2099214016635353 - type: nauc_recall_at_100_std value: 21.652430811892163 - type: nauc_recall_at_10_diff1 value: 16.735926854286163 - type: nauc_recall_at_10_max value: -8.153155643353733 - type: nauc_recall_at_10_std value: -8.04241419757528 - type: nauc_recall_at_1_diff1 value: 35.48615861613537 - type: nauc_recall_at_1_max value: -4.968376038142286 - type: nauc_recall_at_1_std value: -8.722029416996381 - type: nauc_recall_at_20_diff1 value: 13.71195242126747 - type: nauc_recall_at_20_max value: -5.54857038214976 - type: nauc_recall_at_20_std value: 0.5747495657296262 - type: nauc_recall_at_3_diff1 value: 23.73157744159544 - type: nauc_recall_at_3_max value: -8.898620891937641 - type: nauc_recall_at_3_std value: -9.357793995872202 - type: nauc_recall_at_5_diff1 value: 16.331831174073077 - type: nauc_recall_at_5_max value: -9.740756642670778 - type: nauc_recall_at_5_std value: -11.361186585459222 - type: ndcg_at_1 value: 49.125 - type: ndcg_at_10 value: 38.936 - type: ndcg_at_100 value: 44.42 - type: ndcg_at_1000 value: 52.080000000000005 - type: ndcg_at_20 value: 38.483000000000004 - type: ndcg_at_3 value: 42.224000000000004 - type: ndcg_at_5 value: 40.263 - type: precision_at_1 value: 62.25000000000001 - type: precision_at_10 value: 31.125000000000004 - type: precision_at_100 value: 10.168000000000001 - type: precision_at_1000 value: 2.2089999999999996 - type: precision_at_20 value: 23.462 - type: precision_at_3 value: 47.583 - type: precision_at_5 value: 40.5 - type: recall_at_1 value: 7.969999999999999 - type: recall_at_10 value: 25.774 - type: recall_at_100 value: 52.913 - type: recall_at_1000 value: 76.281 - type: recall_at_20 value: 32.346000000000004 - type: recall_at_3 value: 15.304 - type: recall_at_5 value: 19.641000000000002 task: type: Retrieval - dataset: config: default name: MTEB EmotionClassification revision: 4f58c6b202a23cf9a4da393831edf4f9183cad37 split: test type: mteb/emotion metrics: - type: accuracy value: 86.9 - type: f1 value: 82.43466297441167 - type: f1_weighted value: 87.21113710044447 - type: main_score value: 86.9 task: type: Classification - dataset: config: default name: MTEB FEVER revision: bea83ef9e8fb933d90a2f1d5515737465d613e12 split: test type: mteb/fever metrics: - type: main_score value: 86.53999999999999 - type: map_at_1 value: 74.97399999999999 - type: map_at_10 value: 82.659 - type: map_at_100 value: 82.836 - type: map_at_1000 value: 82.851 - type: map_at_20 value: 82.765 - type: map_at_3 value: 81.61500000000001 - type: map_at_5 value: 82.313 - type: mrr_at_1 value: 80.87308730873087 - type: mrr_at_10 value: 88.05529957757663 - type: mrr_at_100 value: 88.11601292565871 - type: mrr_at_1000 value: 88.11677924985968 - type: mrr_at_20 value: 88.10230041194373 - type: mrr_at_3 value: 87.29872987298717 - type: mrr_at_5 value: 87.8365336533651 - type: nauc_map_at_1000_diff1 value: 46.70261215324498 - type: nauc_map_at_1000_max value: 7.04045426956567 - type: nauc_map_at_1000_std value: -25.016067857784936 - type: nauc_map_at_100_diff1 value: 46.67118950260566 - type: nauc_map_at_100_max value: 7.024337825785156 - type: nauc_map_at_100_std value: -25.0179014964629 - type: nauc_map_at_10_diff1 value: 46.48727033111116 - type: nauc_map_at_10_max value: 7.030297220664678 - type: nauc_map_at_10_std value: -24.859656571920247 - type: nauc_map_at_1_diff1 value: 52.514894169034974 - type: nauc_map_at_1_max value: 6.4991819409595415 - type: nauc_map_at_1_std value: -26.962624287912067 - type: nauc_map_at_20_diff1 value: 46.58413925547691 - type: nauc_map_at_20_max value: 6.975619778750194 - type: nauc_map_at_20_std value: -24.984054859146365 - type: nauc_map_at_3_diff1 value: 46.35115084281979 - type: nauc_map_at_3_max value: 6.268645069341729 - type: nauc_map_at_3_std value: -25.449448105561935 - type: nauc_map_at_5_diff1 value: 46.25768358653106 - type: nauc_map_at_5_max value: 6.907729829933622 - type: nauc_map_at_5_std value: -24.94987246230526 - type: nauc_mrr_at_1000_diff1 value: 65.35505807237064 - type: nauc_mrr_at_1000_max value: 10.30647297142242 - type: nauc_mrr_at_1000_std value: -37.369411728966185 - type: nauc_mrr_at_100_diff1 value: 65.3558763604295 - type: nauc_mrr_at_100_max value: 10.310911594013163 - type: nauc_mrr_at_100_std value: -37.371872361808315 - type: nauc_mrr_at_10_diff1 value: 65.28234257793653 - type: nauc_mrr_at_10_max value: 10.446154148428406 - type: nauc_mrr_at_10_std value: -37.43279989381229 - type: nauc_mrr_at_1_diff1 value: 67.43046114024251 - type: nauc_mrr_at_1_max value: 9.199791010999588 - type: nauc_mrr_at_1_std value: -35.72647316332683 - type: nauc_mrr_at_20_diff1 value: 65.33718094349057 - type: nauc_mrr_at_20_max value: 10.36042922044663 - type: nauc_mrr_at_20_std value: -37.39896601561335 - type: nauc_mrr_at_3_diff1 value: 64.75542682908058 - type: nauc_mrr_at_3_max value: 9.81685738005414 - type: nauc_mrr_at_3_std value: -37.586465861627715 - type: nauc_mrr_at_5_diff1 value: 64.97675653138579 - type: nauc_mrr_at_5_max value: 10.473645964548574 - type: nauc_mrr_at_5_std value: -37.517720854364725 - type: nauc_ndcg_at_1000_diff1 value: 47.980879951824015 - type: nauc_ndcg_at_1000_max value: 8.294784645952213 - type: nauc_ndcg_at_1000_std value: -25.68035961439311 - type: nauc_ndcg_at_100_diff1 value: 47.17557297242816 - type: nauc_ndcg_at_100_max value: 7.97932909647729 - type: nauc_ndcg_at_100_std value: -25.656209456961797 - type: nauc_ndcg_at_10_diff1 value: 46.44829982304173 - type: nauc_ndcg_at_10_max value: 8.10910045641653 - type: nauc_ndcg_at_10_std value: -25.23311748908822 - type: nauc_ndcg_at_1_diff1 value: 67.43046114024251 - type: nauc_ndcg_at_1_max value: 9.199791010999588 - type: nauc_ndcg_at_1_std value: -35.72647316332683 - type: nauc_ndcg_at_20_diff1 value: 46.71605953667202 - type: nauc_ndcg_at_20_max value: 7.83399009429542 - type: nauc_ndcg_at_20_std value: -25.50914427951171 - type: nauc_ndcg_at_3_diff1 value: 47.09767655901834 - type: nauc_ndcg_at_3_max value: 7.093614795583601 - type: nauc_ndcg_at_3_std value: -27.099378097753064 - type: nauc_ndcg_at_5_diff1 value: 46.230227787105676 - type: nauc_ndcg_at_5_max value: 7.979234136381932 - type: nauc_ndcg_at_5_std value: -25.748004074053082 - type: nauc_precision_at_1000_diff1 value: -5.378736053404685 - type: nauc_precision_at_1000_max value: 10.355124801647387 - type: nauc_precision_at_1000_std value: 2.9095091245038938 - type: nauc_precision_at_100_diff1 value: -6.8190843049279835 - type: nauc_precision_at_100_max value: 10.296510275756647 - type: nauc_precision_at_100_std value: -0.5694377374023152 - type: nauc_precision_at_10_diff1 value: 3.3308965054246853 - type: nauc_precision_at_10_max value: 13.731815856075642 - type: nauc_precision_at_10_std value: -6.390781452502162 - type: nauc_precision_at_1_diff1 value: 67.43046114024251 - type: nauc_precision_at_1_max value: 9.199791010999588 - type: nauc_precision_at_1_std value: -35.72647316332683 - type: nauc_precision_at_20_diff1 value: -2.689991219278092 - type: nauc_precision_at_20_max value: 10.077673782221831 - type: nauc_precision_at_20_std value: -5.163539651519823 - type: nauc_precision_at_3_diff1 value: 32.39846541175425 - type: nauc_precision_at_3_max value: 8.137383096292048 - type: nauc_precision_at_3_std value: -29.232206161111595 - type: nauc_precision_at_5_diff1 value: 15.859211194923645 - type: nauc_precision_at_5_max value: 13.315538292037735 - type: nauc_precision_at_5_std value: -16.424307328219072 - type: nauc_recall_at_1000_diff1 value: -1.4876601880723042 - type: nauc_recall_at_1000_max value: 13.281504681725965 - type: nauc_recall_at_1000_std value: 20.088387638135956 - type: nauc_recall_at_100_diff1 value: 3.510068775817679 - type: nauc_recall_at_100_max value: 7.223126729279164 - type: nauc_recall_at_100_std value: 1.8098697270167292 - type: nauc_recall_at_10_diff1 value: 13.282399274249123 - type: nauc_recall_at_10_max value: 9.5577982617337 - type: nauc_recall_at_10_std value: -5.244097776935379 - type: nauc_recall_at_1_diff1 value: 52.514894169034974 - type: nauc_recall_at_1_max value: 6.4991819409595415 - type: nauc_recall_at_1_std value: -26.962624287912067 - type: nauc_recall_at_20_diff1 value: 9.54558093675037 - type: nauc_recall_at_20_max value: 7.408067716598006 - type: nauc_recall_at_20_std value: -3.497676642036962 - type: nauc_recall_at_3_diff1 value: 27.469730759073112 - type: nauc_recall_at_3_max value: 4.567448843154473 - type: nauc_recall_at_3_std value: -17.977373809321904 - type: nauc_recall_at_5_diff1 value: 19.20255358383672 - type: nauc_recall_at_5_max value: 8.28840114628813 - type: nauc_recall_at_5_std value: -11.478310315804768 - type: ndcg_at_1 value: 80.87299999999999 - type: ndcg_at_10 value: 86.53999999999999 - type: ndcg_at_100 value: 87.167 - type: ndcg_at_1000 value: 87.455 - type: ndcg_at_20 value: 86.834 - type: ndcg_at_3 value: 84.882 - type: ndcg_at_5 value: 85.884 - type: precision_at_1 value: 80.87299999999999 - type: precision_at_10 value: 10.225 - type: precision_at_100 value: 1.0699999999999998 - type: precision_at_1000 value: 0.11199999999999999 - type: precision_at_20 value: 5.1979999999999995 - type: precision_at_3 value: 32.123000000000005 - type: precision_at_5 value: 19.916 - type: recall_at_1 value: 74.97399999999999 - type: recall_at_10 value: 93.301 - type: recall_at_100 value: 95.779 - type: recall_at_1000 value: 97.596 - type: recall_at_20 value: 94.326 - type: recall_at_3 value: 88.73400000000001 - type: recall_at_5 value: 91.35 task: type: Retrieval - dataset: config: default name: MTEB FiQA2018 revision: 27a168819829fe9bcd655c2df245fb19452e8e06 split: test type: mteb/fiqa metrics: - type: main_score value: 44.741 - type: map_at_1 value: 22.178 - type: map_at_10 value: 36.59 - type: map_at_100 value: 38.633 - type: map_at_1000 value: 38.814 - type: map_at_20 value: 37.756 - type: map_at_3 value: 32.036 - type: map_at_5 value: 34.549 - type: mrr_at_1 value: 43.67283950617284 - type: mrr_at_10 value: 53.02285420340973 - type: mrr_at_100 value: 53.80413476223648 - type: mrr_at_1000 value: 53.83329975208768 - type: mrr_at_20 value: 53.45062434754336 - type: mrr_at_3 value: 50.79732510288063 - type: mrr_at_5 value: 52.093621399176925 - type: nauc_map_at_1000_diff1 value: 38.13095193715016 - type: nauc_map_at_1000_max value: 25.140424128002902 - type: nauc_map_at_1000_std value: -2.291055526111446 - type: nauc_map_at_100_diff1 value: 38.062542374616925 - type: nauc_map_at_100_max value: 24.98212831003545 - type: nauc_map_at_100_std value: -2.3288513840196163 - type: nauc_map_at_10_diff1 value: 37.94005408477477 - type: nauc_map_at_10_max value: 23.62450054152409 - type: nauc_map_at_10_std value: -3.029451096370331 - type: nauc_map_at_1_diff1 value: 46.667422498970936 - type: nauc_map_at_1_max value: 14.863760158210667 - type: nauc_map_at_1_std value: -8.783505629854414 - type: nauc_map_at_20_diff1 value: 37.89072113058759 - type: nauc_map_at_20_max value: 24.366706507797332 - type: nauc_map_at_20_std value: -2.6023248317775605 - type: nauc_map_at_3_diff1 value: 39.47298192019393 - type: nauc_map_at_3_max value: 19.685477846714722 - type: nauc_map_at_3_std value: -5.027029063035539 - type: nauc_map_at_5_diff1 value: 39.0514789038465 - type: nauc_map_at_5_max value: 21.400362349268455 - type: nauc_map_at_5_std value: -4.630809029653372 - type: nauc_mrr_at_1000_diff1 value: 45.62207160402544 - type: nauc_mrr_at_1000_max value: 35.21626828049415 - type: nauc_mrr_at_1000_std value: -1.5589979016799316 - type: nauc_mrr_at_100_diff1 value: 45.6049911269738 - type: nauc_mrr_at_100_max value: 35.22606560450809 - type: nauc_mrr_at_100_std value: -1.5267050690413522 - type: nauc_mrr_at_10_diff1 value: 45.4093336826169 - type: nauc_mrr_at_10_max value: 35.13738468801265 - type: nauc_mrr_at_10_std value: -1.8919874430707715 - type: nauc_mrr_at_1_diff1 value: 51.6709580391481 - type: nauc_mrr_at_1_max value: 33.86625581485091 - type: nauc_mrr_at_1_std value: -5.723117301417357 - type: nauc_mrr_at_20_diff1 value: 45.51335104667583 - type: nauc_mrr_at_20_max value: 35.12487418629981 - type: nauc_mrr_at_20_std value: -1.5728658665283186 - type: nauc_mrr_at_3_diff1 value: 45.91684224347824 - type: nauc_mrr_at_3_max value: 34.71207649436817 - type: nauc_mrr_at_3_std value: -2.3652831443919373 - type: nauc_mrr_at_5_diff1 value: 45.24354433953711 - type: nauc_mrr_at_5_max value: 34.766746844223384 - type: nauc_mrr_at_5_std value: -2.414375686428433 - type: nauc_ndcg_at_1000_diff1 value: 38.99167200921022 - type: nauc_ndcg_at_1000_max value: 30.819387325830377 - type: nauc_ndcg_at_1000_std value: 1.0467714066842262 - type: nauc_ndcg_at_100_diff1 value: 38.248888114357484 - type: nauc_ndcg_at_100_max value: 29.58848183633258 - type: nauc_ndcg_at_100_std value: 1.417553970005679 - type: nauc_ndcg_at_10_diff1 value: 37.8683078247863 - type: nauc_ndcg_at_10_max value: 26.751938977033106 - type: nauc_ndcg_at_10_std value: -1.2409485838847412 - type: nauc_ndcg_at_1_diff1 value: 51.6709580391481 - type: nauc_ndcg_at_1_max value: 33.86625581485091 - type: nauc_ndcg_at_1_std value: -5.723117301417357 - type: nauc_ndcg_at_20_diff1 value: 37.8638946400599 - type: nauc_ndcg_at_20_max value: 27.606616644241157 - type: nauc_ndcg_at_20_std value: -0.03526704440544082 - type: nauc_ndcg_at_3_diff1 value: 39.51172857224684 - type: nauc_ndcg_at_3_max value: 29.20532249592107 - type: nauc_ndcg_at_3_std value: -1.8738383813061488 - type: nauc_ndcg_at_5_diff1 value: 38.377746472924514 - type: nauc_ndcg_at_5_max value: 26.329340638530454 - type: nauc_ndcg_at_5_std value: -2.918235844784902 - type: nauc_precision_at_1000_diff1 value: -0.5958931375447998 - type: nauc_precision_at_1000_max value: 36.15147294143512 - type: nauc_precision_at_1000_std value: 11.710171839031613 - type: nauc_precision_at_100_diff1 value: 3.1279602297522304 - type: nauc_precision_at_100_max value: 38.56180438744363 - type: nauc_precision_at_100_std value: 14.063724836230277 - type: nauc_precision_at_10_diff1 value: 12.659130460376236 - type: nauc_precision_at_10_max value: 39.37196817310586 - type: nauc_precision_at_10_std value: 10.085604614097598 - type: nauc_precision_at_1_diff1 value: 51.6709580391481 - type: nauc_precision_at_1_max value: 33.86625581485091 - type: nauc_precision_at_1_std value: -5.723117301417357 - type: nauc_precision_at_20_diff1 value: 9.650139629433262 - type: nauc_precision_at_20_max value: 38.77724238894805 - type: nauc_precision_at_20_std value: 11.958790785998644 - type: nauc_precision_at_3_diff1 value: 24.745989796740357 - type: nauc_precision_at_3_max value: 35.65068216285761 - type: nauc_precision_at_3_std value: 4.072896234633994 - type: nauc_precision_at_5_diff1 value: 18.655049044737424 - type: nauc_precision_at_5_max value: 35.998788688559216 - type: nauc_precision_at_5_std value: 5.391771098181724 - type: nauc_recall_at_1000_diff1 value: 12.949608157518922 - type: nauc_recall_at_1000_max value: 23.391273269349142 - type: nauc_recall_at_1000_std value: 28.42378542198053 - type: nauc_recall_at_100_diff1 value: 20.625416053309404 - type: nauc_recall_at_100_max value: 21.367842929796264 - type: nauc_recall_at_100_std value: 15.399702721384228 - type: nauc_recall_at_10_diff1 value: 27.211773223610237 - type: nauc_recall_at_10_max value: 18.536687796485346 - type: nauc_recall_at_10_std value: 0.8218390873018705 - type: nauc_recall_at_1_diff1 value: 46.667422498970936 - type: nauc_recall_at_1_max value: 14.863760158210667 - type: nauc_recall_at_1_std value: -8.783505629854414 - type: nauc_recall_at_20_diff1 value: 26.189517076492148 - type: nauc_recall_at_20_max value: 19.712210144922732 - type: nauc_recall_at_20_std value: 4.732025980603548 - type: nauc_recall_at_3_diff1 value: 32.03530659219441 - type: nauc_recall_at_3_max value: 15.729199503552953 - type: nauc_recall_at_3_std value: -2.78006696474447 - type: nauc_recall_at_5_diff1 value: 30.303038095198648 - type: nauc_recall_at_5_max value: 16.04989351651554 - type: nauc_recall_at_5_std value: -3.4801349592650674 - type: ndcg_at_1 value: 43.673 - type: ndcg_at_10 value: 44.741 - type: ndcg_at_100 value: 51.815999999999995 - type: ndcg_at_1000 value: 54.55799999999999 - type: ndcg_at_20 value: 47.575 - type: ndcg_at_3 value: 40.973 - type: ndcg_at_5 value: 42.174 - type: precision_at_1 value: 43.673 - type: precision_at_10 value: 12.299 - type: precision_at_100 value: 1.9709999999999999 - type: precision_at_1000 value: 0.246 - type: precision_at_20 value: 7.3999999999999995 - type: precision_at_3 value: 27.315 - type: precision_at_5 value: 20.061999999999998 - type: recall_at_1 value: 22.178 - type: recall_at_10 value: 51.902 - type: recall_at_100 value: 77.802 - type: recall_at_1000 value: 94.12 - type: recall_at_20 value: 60.496 - type: recall_at_3 value: 37.919000000000004 - type: recall_at_5 value: 43.829 task: type: Retrieval - dataset: config: default name: MTEB HotpotQA revision: ab518f4d6fcca38d87c25209f94beba119d02014 split: test type: mteb/hotpotqa metrics: - type: main_score value: 67.585 - type: map_at_1 value: 38.596000000000004 - type: map_at_10 value: 58.378 - type: map_at_100 value: 59.282000000000004 - type: map_at_1000 value: 59.34199999999999 - type: map_at_20 value: 58.921 - type: map_at_3 value: 54.801 - type: map_at_5 value: 57.065999999999995 - type: mrr_at_1 value: 77.19108710330858 - type: mrr_at_10 value: 83.73176103662232 - type: mrr_at_100 value: 83.90019429669105 - type: mrr_at_1000 value: 83.90679291308962 - type: mrr_at_20 value: 83.8435177926089 - type: mrr_at_3 value: 82.81341435966664 - type: mrr_at_5 value: 83.4224623002472 - type: nauc_map_at_1000_diff1 value: 14.307964912952539 - type: nauc_map_at_1000_max value: 12.765468178017315 - type: nauc_map_at_1000_std value: 1.362058994472385 - type: nauc_map_at_100_diff1 value: 14.26768417100101 - type: nauc_map_at_100_max value: 12.750734297430844 - type: nauc_map_at_100_std value: 1.393555710353686 - type: nauc_map_at_10_diff1 value: 14.255931489441185 - type: nauc_map_at_10_max value: 12.523189563101312 - type: nauc_map_at_10_std value: 0.7161967418789583 - type: nauc_map_at_1_diff1 value: 69.87899286905372 - type: nauc_map_at_1_max value: 35.205475784602456 - type: nauc_map_at_1_std value: -10.863394163636846 - type: nauc_map_at_20_diff1 value: 14.212027716409054 - type: nauc_map_at_20_max value: 12.650489544649925 - type: nauc_map_at_20_std value: 1.1271711468045333 - type: nauc_map_at_3_diff1 value: 16.65137376183016 - type: nauc_map_at_3_max value: 13.00998218430069 - type: nauc_map_at_3_std value: -2.2381896578334963 - type: nauc_map_at_5_diff1 value: 14.766891306293303 - type: nauc_map_at_5_max value: 12.45533963465128 - type: nauc_map_at_5_std value: -0.4763780518957518 - type: nauc_mrr_at_1000_diff1 value: 68.55128371980359 - type: nauc_mrr_at_1000_max value: 36.47247958442408 - type: nauc_mrr_at_1000_std value: -9.311243277776393 - type: nauc_mrr_at_100_diff1 value: 68.5495192075432 - type: nauc_mrr_at_100_max value: 36.47800039396858 - type: nauc_mrr_at_100_std value: -9.30009477026889 - type: nauc_mrr_at_10_diff1 value: 68.5213780826157 - type: nauc_mrr_at_10_max value: 36.53470560276756 - type: nauc_mrr_at_10_std value: -9.276757799691362 - type: nauc_mrr_at_1_diff1 value: 69.87899286905372 - type: nauc_mrr_at_1_max value: 35.205475784602456 - type: nauc_mrr_at_1_std value: -10.863394163636846 - type: nauc_mrr_at_20_diff1 value: 68.51230880783687 - type: nauc_mrr_at_20_max value: 36.502918925898356 - type: nauc_mrr_at_20_std value: -9.252393600030487 - type: nauc_mrr_at_3_diff1 value: 68.41383168882547 - type: nauc_mrr_at_3_max value: 36.31820775404627 - type: nauc_mrr_at_3_std value: -9.999151252843035 - type: nauc_mrr_at_5_diff1 value: 68.45324347636361 - type: nauc_mrr_at_5_max value: 36.46139281601083 - type: nauc_mrr_at_5_std value: -9.588349787263777 - type: nauc_ndcg_at_1000_diff1 value: 19.255656584818208 - type: nauc_ndcg_at_1000_max value: 16.119708974764283 - type: nauc_ndcg_at_1000_std value: 3.6997591648265367 - type: nauc_ndcg_at_100_diff1 value: 18.173106007959618 - type: nauc_ndcg_at_100_max value: 15.568458367725462 - type: nauc_ndcg_at_100_std value: 4.536505893658295 - type: nauc_ndcg_at_10_diff1 value: 18.369881058142322 - type: nauc_ndcg_at_10_max value: 14.976758085283695 - type: nauc_ndcg_at_10_std value: 1.9666148808621997 - type: nauc_ndcg_at_1_diff1 value: 69.87899286905372 - type: nauc_ndcg_at_1_max value: 35.205475784602456 - type: nauc_ndcg_at_1_std value: -10.863394163636846 - type: nauc_ndcg_at_20_diff1 value: 17.99231888427834 - type: nauc_ndcg_at_20_max value: 15.195593488428424 - type: nauc_ndcg_at_20_std value: 3.210518329988178 - type: nauc_ndcg_at_3_diff1 value: 22.638085589869977 - type: nauc_ndcg_at_3_max value: 15.98027793904305 - type: nauc_ndcg_at_3_std value: -2.8619254611270484 - type: nauc_ndcg_at_5_diff1 value: 19.628126741608632 - type: nauc_ndcg_at_5_max value: 14.996362948646471 - type: nauc_ndcg_at_5_std value: -0.37278144328821516 - type: nauc_precision_at_1000_diff1 value: -18.870574565264082 - type: nauc_precision_at_1000_max value: 6.003585500284789 - type: nauc_precision_at_1000_std value: 28.22209155996002 - type: nauc_precision_at_100_diff1 value: -14.152417200336195 - type: nauc_precision_at_100_max value: 4.435292073503267 - type: nauc_precision_at_100_std value: 25.999649502498933 - type: nauc_precision_at_10_diff1 value: -4.136070746784591 - type: nauc_precision_at_10_max value: 5.875693813516143 - type: nauc_precision_at_10_std value: 10.909307135721624 - type: nauc_precision_at_1_diff1 value: 69.87899286905372 - type: nauc_precision_at_1_max value: 35.205475784602456 - type: nauc_precision_at_1_std value: -10.863394163636846 - type: nauc_precision_at_20_diff1 value: -7.786302363507167 - type: nauc_precision_at_20_max value: 5.45833862385675 - type: nauc_precision_at_20_std value: 15.783059950464354 - type: nauc_precision_at_3_diff1 value: 7.40890050470361 - type: nauc_precision_at_3_max value: 9.458068780681774 - type: nauc_precision_at_3_std value: 0.0027409382551998934 - type: nauc_precision_at_5_diff1 value: 0.6243863646902605 - type: nauc_precision_at_5_max value: 6.80746180406321 - type: nauc_precision_at_5_std value: 4.826367656810033 - type: nauc_recall_at_1000_diff1 value: -18.870574565263908 - type: nauc_recall_at_1000_max value: 6.003585500284974 - type: nauc_recall_at_1000_std value: 28.222091559960184 - type: nauc_recall_at_100_diff1 value: -14.15241720033625 - type: nauc_recall_at_100_max value: 4.435292073503326 - type: nauc_recall_at_100_std value: 25.999649502498993 - type: nauc_recall_at_10_diff1 value: -4.136070746784523 - type: nauc_recall_at_10_max value: 5.875693813516104 - type: nauc_recall_at_10_std value: 10.909307135721532 - type: nauc_recall_at_1_diff1 value: 69.87899286905372 - type: nauc_recall_at_1_max value: 35.205475784602456 - type: nauc_recall_at_1_std value: -10.863394163636846 - type: nauc_recall_at_20_diff1 value: -7.786302363507246 - type: nauc_recall_at_20_max value: 5.458338623856692 - type: nauc_recall_at_20_std value: 15.783059950464304 - type: nauc_recall_at_3_diff1 value: 7.408900504703652 - type: nauc_recall_at_3_max value: 9.458068780681788 - type: nauc_recall_at_3_std value: 0.002740938255258331 - type: nauc_recall_at_5_diff1 value: 0.6243863646902303 - type: nauc_recall_at_5_max value: 6.807461804063163 - type: nauc_recall_at_5_std value: 4.826367656810076 - type: ndcg_at_1 value: 77.191 - type: ndcg_at_10 value: 67.585 - type: ndcg_at_100 value: 70.652 - type: ndcg_at_1000 value: 71.82600000000001 - type: ndcg_at_20 value: 68.913 - type: ndcg_at_3 value: 62.61600000000001 - type: ndcg_at_5 value: 65.444 - type: precision_at_1 value: 77.191 - type: precision_at_10 value: 14.097000000000001 - type: precision_at_100 value: 1.6480000000000001 - type: precision_at_1000 value: 0.18 - type: precision_at_20 value: 7.475 - type: precision_at_3 value: 39.716 - type: precision_at_5 value: 26.061 - type: recall_at_1 value: 38.596000000000004 - type: recall_at_10 value: 70.486 - type: recall_at_100 value: 82.417 - type: recall_at_1000 value: 90.223 - type: recall_at_20 value: 74.754 - type: recall_at_3 value: 59.575 - type: recall_at_5 value: 65.152 task: type: Retrieval - dataset: config: default name: MTEB ImdbClassification revision: 3d86128a09e091d6018b6d26cad27f2739fc2db7 split: test type: mteb/imdb metrics: - type: accuracy value: 94.93119999999999 - type: ap value: 92.92609880646465 - type: ap_weighted value: 92.92609880646465 - type: f1 value: 94.93020053324759 - type: f1_weighted value: 94.93020053324759 - type: main_score value: 94.93119999999999 task: type: Classification - dataset: config: default name: MTEB MSMARCO revision: c5a29a104738b98a9e76336939199e264163d4a0 split: dev type: mteb/msmarco metrics: - type: main_score value: 34.593 - type: map_at_1 value: 15.776000000000002 - type: map_at_10 value: 27.542 - type: map_at_100 value: 28.893 - type: map_at_1000 value: 28.95 - type: map_at_20 value: 28.381 - type: map_at_3 value: 23.273 - type: map_at_5 value: 25.706 - type: mrr_at_1 value: 16.23209169054441 - type: mrr_at_10 value: 27.99337108291251 - type: mrr_at_100 value: 29.299712175246 - type: mrr_at_1000 value: 29.34980176303808 - type: mrr_at_20 value: 28.815617370683793 - type: mrr_at_3 value: 23.798949379178495 - type: mrr_at_5 value: 26.185052531041066 - type: nauc_map_at_1000_diff1 value: 29.078121539546963 - type: nauc_map_at_1000_max value: -4.660325486096363 - type: nauc_map_at_1000_std value: -16.890470067987078 - type: nauc_map_at_100_diff1 value: 29.076892998159536 - type: nauc_map_at_100_max value: -4.679633015133337 - type: nauc_map_at_100_std value: -16.862781614983454 - type: nauc_map_at_10_diff1 value: 29.13003469335662 - type: nauc_map_at_10_max value: -4.88793866124687 - type: nauc_map_at_10_std value: -17.63030762796993 - type: nauc_map_at_1_diff1 value: 31.692294187492053 - type: nauc_map_at_1_max value: -3.4670833327733073 - type: nauc_map_at_1_std value: -15.691352600942812 - type: nauc_map_at_20_diff1 value: 29.055576137474308 - type: nauc_map_at_20_max value: -4.7776538554194286 - type: nauc_map_at_20_std value: -17.224848225253236 - type: nauc_map_at_3_diff1 value: 28.9101303524098 - type: nauc_map_at_3_max value: -4.652960479072655 - type: nauc_map_at_3_std value: -17.418465326700314 - type: nauc_map_at_5_diff1 value: 28.843601736020492 - type: nauc_map_at_5_max value: -4.674223389697514 - type: nauc_map_at_5_std value: -17.97502181985735 - type: nauc_mrr_at_1000_diff1 value: 28.898068429091573 - type: nauc_mrr_at_1000_max value: -4.399664885905157 - type: nauc_mrr_at_1000_std value: -16.607218168281097 - type: nauc_mrr_at_100_diff1 value: 28.89629415174332 - type: nauc_mrr_at_100_max value: -4.409492230329251 - type: nauc_mrr_at_100_std value: -16.57762186807223 - type: nauc_mrr_at_10_diff1 value: 28.941113994846653 - type: nauc_mrr_at_10_max value: -4.583270921209106 - type: nauc_mrr_at_10_std value: -17.289894093299672 - type: nauc_mrr_at_1_diff1 value: 31.37083827832811 - type: nauc_mrr_at_1_max value: -3.208411493959455 - type: nauc_mrr_at_1_std value: -15.40345578596366 - type: nauc_mrr_at_20_diff1 value: 28.8794180315418 - type: nauc_mrr_at_20_max value: -4.471345918444438 - type: nauc_mrr_at_20_std value: -16.889422348794938 - type: nauc_mrr_at_3_diff1 value: 28.648544038786806 - type: nauc_mrr_at_3_max value: -4.446235699608574 - type: nauc_mrr_at_3_std value: -17.223275526865695 - type: nauc_mrr_at_5_diff1 value: 28.676176998873526 - type: nauc_mrr_at_5_max value: -4.384629457636603 - type: nauc_mrr_at_5_std value: -17.662360967653942 - type: nauc_ndcg_at_1000_diff1 value: 28.512776616817614 - type: nauc_ndcg_at_1000_max value: -4.032666359115543 - type: nauc_ndcg_at_1000_std value: -14.96235335409355 - type: nauc_ndcg_at_100_diff1 value: 28.46827238630747 - type: nauc_ndcg_at_100_max value: -4.297081420480486 - type: nauc_ndcg_at_100_std value: -13.652822791502834 - type: nauc_ndcg_at_10_diff1 value: 28.731734677632335 - type: nauc_ndcg_at_10_max value: -5.307476808688383 - type: nauc_ndcg_at_10_std value: -17.847598875907043 - type: nauc_ndcg_at_1_diff1 value: 31.37083827832811 - type: nauc_ndcg_at_1_max value: -3.208411493959455 - type: nauc_ndcg_at_1_std value: -15.40345578596366 - type: nauc_ndcg_at_20_diff1 value: 28.454343686246368 - type: nauc_ndcg_at_20_max value: -4.98727458049532 - type: nauc_ndcg_at_20_std value: -16.354151805926072 - type: nauc_ndcg_at_3_diff1 value: 28.17849373973975 - type: nauc_ndcg_at_3_max value: -4.8778622367007705 - type: nauc_ndcg_at_3_std value: -17.863750930775883 - type: nauc_ndcg_at_5_diff1 value: 28.150883678367972 - type: nauc_ndcg_at_5_max value: -4.855498175237134 - type: nauc_ndcg_at_5_std value: -18.690049098501387 - type: nauc_precision_at_1000_diff1 value: -2.8337934746656184 - type: nauc_precision_at_1000_max value: 18.96310957410793 - type: nauc_precision_at_1000_std value: 14.930142344173392 - type: nauc_precision_at_100_diff1 value: 14.333530836109675 - type: nauc_precision_at_100_max value: 6.295533804672035 - type: nauc_precision_at_100_std value: 18.019478047265473 - type: nauc_precision_at_10_diff1 value: 26.15274520345387 - type: nauc_precision_at_10_max value: -5.684160746990313 - type: nauc_precision_at_10_std value: -17.184523300085225 - type: nauc_precision_at_1_diff1 value: 31.37083827832811 - type: nauc_precision_at_1_max value: -3.208411493959455 - type: nauc_precision_at_1_std value: -15.40345578596366 - type: nauc_precision_at_20_diff1 value: 23.390317543956684 - type: nauc_precision_at_20_max value: -3.643101669692548 - type: nauc_precision_at_20_std value: -10.138163569909773 - type: nauc_precision_at_3_diff1 value: 26.147860052902338 - type: nauc_precision_at_3_max value: -5.348523907965159 - type: nauc_precision_at_3_std value: -18.984802706187416 - type: nauc_precision_at_5_diff1 value: 25.59347977608168 - type: nauc_precision_at_5_max value: -4.966621480640159 - type: nauc_precision_at_5_std value: -20.197439172622257 - type: nauc_recall_at_1000_diff1 value: 9.353127475581333 - type: nauc_recall_at_1000_max value: 35.43082913054732 - type: nauc_recall_at_1000_std value: 66.50162334521372 - type: nauc_recall_at_100_diff1 value: 24.125268256131005 - type: nauc_recall_at_100_max value: -0.25871115365966996 - type: nauc_recall_at_100_std value: 22.002110879773436 - type: nauc_recall_at_10_diff1 value: 27.79420827880311 - type: nauc_recall_at_10_max value: -6.774583137315747 - type: nauc_recall_at_10_std value: -18.300025227607936 - type: nauc_recall_at_1_diff1 value: 31.692294187492053 - type: nauc_recall_at_1_max value: -3.4670833327733073 - type: nauc_recall_at_1_std value: -15.691352600942812 - type: nauc_recall_at_20_diff1 value: 26.47331981012526 - type: nauc_recall_at_20_max value: -6.053234914161821 - type: nauc_recall_at_20_std value: -12.210795863137527 - type: nauc_recall_at_3_diff1 value: 26.396199364799276 - type: nauc_recall_at_3_max value: -5.5102184367103675 - type: nauc_recall_at_3_std value: -18.893379374309998 - type: nauc_recall_at_5_diff1 value: 26.415672184819467 - type: nauc_recall_at_5_max value: -5.452147280159272 - type: nauc_recall_at_5_std value: -20.488315060196 - type: ndcg_at_1 value: 16.232 - type: ndcg_at_10 value: 34.593 - type: ndcg_at_100 value: 41.193999999999996 - type: ndcg_at_1000 value: 42.594 - type: ndcg_at_20 value: 37.586999999999996 - type: ndcg_at_3 value: 25.921 - type: ndcg_at_5 value: 30.244 - type: precision_at_1 value: 16.232 - type: precision_at_10 value: 5.924 - type: precision_at_100 value: 0.922 - type: precision_at_1000 value: 0.104 - type: precision_at_20 value: 3.5770000000000004 - type: precision_at_3 value: 11.404 - type: precision_at_5 value: 9.052 - type: recall_at_1 value: 15.776000000000002 - type: recall_at_10 value: 56.581 - type: recall_at_100 value: 87.32 - type: recall_at_1000 value: 97.98400000000001 - type: recall_at_20 value: 68.22500000000001 - type: recall_at_3 value: 33.079 - type: recall_at_5 value: 43.418 task: type: Retrieval - dataset: config: en name: MTEB MTOPDomainClassification (en) revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf split: test type: mteb/mtop_domain metrics: - type: accuracy value: 97.53989968080255 - type: f1 value: 97.40324881796998 - type: f1_weighted value: 97.55321112075454 - type: main_score value: 97.53989968080255 task: type: Classification - dataset: config: en name: MTEB MTOPIntentClassification (en) revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba split: test type: mteb/mtop_intent metrics: - type: accuracy value: 85.75923392612859 - type: f1 value: 61.5307155386522 - type: f1_weighted value: 85.67681172442828 - type: main_score value: 85.75923392612859 task: type: Classification - dataset: config: en name: MTEB MassiveIntentClassification (en) revision: 4672e20407010da34463acc759c162ca9734bca6 split: test type: mteb/amazon_massive_intent metrics: - type: accuracy value: 72.51513113651647 - type: f1 value: 69.34853147184798 - type: f1_weighted value: 71.22394439174492 - type: main_score value: 72.51513113651647 task: type: Classification - dataset: config: en name: MTEB MassiveScenarioClassification (en) revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 split: test type: mteb/amazon_massive_scenario metrics: - type: accuracy value: 79.32414256893072 - type: f1 value: 78.7067321614858 - type: f1_weighted value: 79.17371887085407 - type: main_score value: 79.32414256893072 task: type: Classification - dataset: config: default name: MTEB MedrxivClusteringP2P revision: e7a26af6f3ae46b30dde8737f02c07b1505bcc73 split: test type: mteb/medrxiv-clustering-p2p metrics: - type: main_score value: 39.9112733005734 - type: v_measure value: 39.9112733005734 - type: v_measure_std value: 1.3729330580974082 task: type: Clustering - dataset: config: default name: MTEB MedrxivClusteringS2S revision: 35191c8c0dca72d8ff3efcd72aa802307d469663 split: test type: mteb/medrxiv-clustering-s2s metrics: - type: main_score value: 36.793147891812595 - type: v_measure value: 36.793147891812595 - type: v_measure_std value: 1.373356254555773 task: type: Clustering - dataset: config: default name: MTEB MindSmallReranking revision: 59042f120c80e8afa9cdbb224f67076cec0fc9a7 split: test type: mteb/mind_small metrics: - type: main_score value: 31.92214547032066 - type: map value: 31.92214547032066 - type: mrr value: 33.14385477235431 - type: nAUC_map_diff1 value: 11.986123616015425 - type: nAUC_map_max value: -20.27412119718789 - type: nAUC_map_std value: 1.1581550862810461 - type: nAUC_mrr_diff1 value: 11.165093384672165 - type: nAUC_mrr_max value: -14.737175633093665 - type: nAUC_mrr_std value: 2.5082843023127364 task: type: Reranking - dataset: config: default name: MTEB NFCorpus revision: ec0fa4fe99da2ff19ca1214b7966684033a58814 split: test type: mteb/nfcorpus metrics: - type: main_score value: 35.326 - type: map_at_1 value: 5.756 - type: map_at_10 value: 13.23 - type: map_at_100 value: 16.821 - type: map_at_1000 value: 18.285 - type: map_at_20 value: 14.709 - type: map_at_3 value: 9.86 - type: map_at_5 value: 11.41 - type: mrr_at_1 value: 45.20123839009288 - type: mrr_at_10 value: 54.5034154012482 - type: mrr_at_100 value: 55.11212028144428 - type: mrr_at_1000 value: 55.14869280667608 - type: mrr_at_20 value: 54.852116950454885 - type: mrr_at_3 value: 52.32198142414861 - type: mrr_at_5 value: 53.69969040247676 - type: nauc_map_at_1000_diff1 value: 24.103237694692552 - type: nauc_map_at_1000_max value: 28.238145605874905 - type: nauc_map_at_1000_std value: 14.821335297504906 - type: nauc_map_at_100_diff1 value: 25.692696988595603 - type: nauc_map_at_100_max value: 27.282927002710704 - type: nauc_map_at_100_std value: 11.428779187303642 - type: nauc_map_at_10_diff1 value: 30.802161942248162 - type: nauc_map_at_10_max value: 21.533637232032028 - type: nauc_map_at_10_std value: -1.4834314993650302 - type: nauc_map_at_1_diff1 value: 44.692178401496655 - type: nauc_map_at_1_max value: 7.8258037280355826 - type: nauc_map_at_1_std value: -14.548783945519206 - type: nauc_map_at_20_diff1 value: 28.162218272962896 - type: nauc_map_at_20_max value: 24.526508434124295 - type: nauc_map_at_20_std value: 4.5923029452531114 - type: nauc_map_at_3_diff1 value: 36.159346924502394 - type: nauc_map_at_3_max value: 13.037438946030472 - type: nauc_map_at_3_std value: -10.65662898188665 - type: nauc_map_at_5_diff1 value: 34.65868875018595 - type: nauc_map_at_5_max value: 17.60272401270236 - type: nauc_map_at_5_std value: -7.249752394842571 - type: nauc_mrr_at_1000_diff1 value: 30.21772044474309 - type: nauc_mrr_at_1000_max value: 42.78186255622255 - type: nauc_mrr_at_1000_std value: 25.797577782337537 - type: nauc_mrr_at_100_diff1 value: 30.212889955928773 - type: nauc_mrr_at_100_max value: 42.81922498816082 - type: nauc_mrr_at_100_std value: 25.85789713650141 - type: nauc_mrr_at_10_diff1 value: 30.44797936946012 - type: nauc_mrr_at_10_max value: 42.53323068058471 - type: nauc_mrr_at_10_std value: 25.66456895797239 - type: nauc_mrr_at_1_diff1 value: 32.34440752392725 - type: nauc_mrr_at_1_max value: 36.9469056580779 - type: nauc_mrr_at_1_std value: 18.657611594947873 - type: nauc_mrr_at_20_diff1 value: 30.176345122251274 - type: nauc_mrr_at_20_max value: 42.75434764186405 - type: nauc_mrr_at_20_std value: 25.79589315785969 - type: nauc_mrr_at_3_diff1 value: 29.302461225569886 - type: nauc_mrr_at_3_max value: 41.82223481926128 - type: nauc_mrr_at_3_std value: 24.02550946327186 - type: nauc_mrr_at_5_diff1 value: 30.47192933785718 - type: nauc_mrr_at_5_max value: 42.36007501528813 - type: nauc_mrr_at_5_std value: 25.0758442990031 - type: nauc_ndcg_at_1000_diff1 value: 23.68574089704954 - type: nauc_ndcg_at_1000_max value: 44.188191803981134 - type: nauc_ndcg_at_1000_std value: 32.59339851811357 - type: nauc_ndcg_at_100_diff1 value: 22.31584896554311 - type: nauc_ndcg_at_100_max value: 37.059955696641985 - type: nauc_ndcg_at_100_std value: 27.00502292427943 - type: nauc_ndcg_at_10_diff1 value: 20.60773946800146 - type: nauc_ndcg_at_10_max value: 37.61141982157199 - type: nauc_ndcg_at_10_std value: 26.410973914430798 - type: nauc_ndcg_at_1_diff1 value: 34.18370363324457 - type: nauc_ndcg_at_1_max value: 36.36015956329315 - type: nauc_ndcg_at_1_std value: 18.45897389269808 - type: nauc_ndcg_at_20_diff1 value: 20.24173606023381 - type: nauc_ndcg_at_20_max value: 37.415776645800705 - type: nauc_ndcg_at_20_std value: 28.21373791467815 - type: nauc_ndcg_at_3_diff1 value: 23.315676671193806 - type: nauc_ndcg_at_3_max value: 38.7142575726556 - type: nauc_ndcg_at_3_std value: 23.1580283666415 - type: nauc_ndcg_at_5_diff1 value: 22.245075769905913 - type: nauc_ndcg_at_5_max value: 39.635321337970396 - type: nauc_ndcg_at_5_std value: 25.453576926797428 - type: nauc_precision_at_1000_diff1 value: -13.699777209784223 - type: nauc_precision_at_1000_max value: 10.284718353463134 - type: nauc_precision_at_1000_std value: 35.67360319480793 - type: nauc_precision_at_100_diff1 value: -12.08500797758176 - type: nauc_precision_at_100_max value: 20.44925184924704 - type: nauc_precision_at_100_std value: 45.5895403196849 - type: nauc_precision_at_10_diff1 value: 0.06557427029772205 - type: nauc_precision_at_10_max value: 37.55343130127243 - type: nauc_precision_at_10_std value: 39.8774623173031 - type: nauc_precision_at_1_diff1 value: 32.34440752392725 - type: nauc_precision_at_1_max value: 36.9469056580779 - type: nauc_precision_at_1_std value: 18.657611594947873 - type: nauc_precision_at_20_diff1 value: -5.424800947711766 - type: nauc_precision_at_20_max value: 34.400450482662606 - type: nauc_precision_at_20_std value: 46.96466506447469 - type: nauc_precision_at_3_diff1 value: 11.737310947259319 - type: nauc_precision_at_3_max value: 40.38852481780515 - type: nauc_precision_at_3_std value: 29.037703635253433 - type: nauc_precision_at_5_diff1 value: 6.832213390107087 - type: nauc_precision_at_5_max value: 41.3861915622186 - type: nauc_precision_at_5_std value: 34.60777485026518 - type: nauc_recall_at_1000_diff1 value: 8.252648642934155 - type: nauc_recall_at_1000_max value: 21.9593982158523 - type: nauc_recall_at_1000_std value: 19.351920551082532 - type: nauc_recall_at_100_diff1 value: 14.367515824195436 - type: nauc_recall_at_100_max value: 20.91547758690166 - type: nauc_recall_at_100_std value: 15.205925255166164 - type: nauc_recall_at_10_diff1 value: 28.110881426056622 - type: nauc_recall_at_10_max value: 20.127999606996973 - type: nauc_recall_at_10_std value: -1.4778912107941327 - type: nauc_recall_at_1_diff1 value: 44.692178401496655 - type: nauc_recall_at_1_max value: 7.8258037280355826 - type: nauc_recall_at_1_std value: -14.548783945519206 - type: nauc_recall_at_20_diff1 value: 22.894768482358447 - type: nauc_recall_at_20_max value: 23.53906838805954 - type: nauc_recall_at_20_std value: 6.968725573206477 - type: nauc_recall_at_3_diff1 value: 33.912996119453844 - type: nauc_recall_at_3_max value: 12.026372824736068 - type: nauc_recall_at_3_std value: -10.339483350646704 - type: nauc_recall_at_5_diff1 value: 33.302263355698805 - type: nauc_recall_at_5_max value: 17.306357074559127 - type: nauc_recall_at_5_std value: -6.907681462221045 - type: ndcg_at_1 value: 43.498 - type: ndcg_at_10 value: 35.326 - type: ndcg_at_100 value: 32.618 - type: ndcg_at_1000 value: 41.127 - type: ndcg_at_20 value: 33.018 - type: ndcg_at_3 value: 40.858 - type: ndcg_at_5 value: 38.888 - type: precision_at_1 value: 45.201 - type: precision_at_10 value: 26.192 - type: precision_at_100 value: 8.455 - type: precision_at_1000 value: 2.114 - type: precision_at_20 value: 19.582 - type: precision_at_3 value: 38.906 - type: precision_at_5 value: 34.489 - type: recall_at_1 value: 5.756 - type: recall_at_10 value: 17.101 - type: recall_at_100 value: 33.768 - type: recall_at_1000 value: 64.453 - type: recall_at_20 value: 21.001 - type: recall_at_3 value: 11.011 - type: recall_at_5 value: 13.344000000000001 task: type: Retrieval - dataset: config: default name: MTEB NQ revision: b774495ed302d8c44a3a7ea25c90dbce03968f31 split: test type: mteb/nq metrics: - type: main_score value: 47.496 - type: map_at_1 value: 24.751 - type: map_at_10 value: 39.348 - type: map_at_100 value: 40.636 - type: map_at_1000 value: 40.671 - type: map_at_20 value: 40.211999999999996 - type: map_at_3 value: 34.575 - type: map_at_5 value: 37.22 - type: mrr_at_1 value: 27.95480880648899 - type: mrr_at_10 value: 41.9044860122496 - type: mrr_at_100 value: 42.876980645677705 - type: mrr_at_1000 value: 42.900385254956284 - type: mrr_at_20 value: 42.55982972518494 - type: mrr_at_3 value: 37.751062186172234 - type: mrr_at_5 value: 40.077249903437675 - type: nauc_map_at_1000_diff1 value: 32.86038948653992 - type: nauc_map_at_1000_max value: 13.50985063916989 - type: nauc_map_at_1000_std value: -5.820123220528598 - type: nauc_map_at_100_diff1 value: 32.84976636584168 - type: nauc_map_at_100_max value: 13.53587518434608 - type: nauc_map_at_100_std value: -5.785833146376201 - type: nauc_map_at_10_diff1 value: 32.85359459153253 - type: nauc_map_at_10_max value: 13.26285007121336 - type: nauc_map_at_10_std value: -6.394996664397717 - type: nauc_map_at_1_diff1 value: 34.23873886692038 - type: nauc_map_at_1_max value: 9.192506212634111 - type: nauc_map_at_1_std value: -7.111967933900576 - type: nauc_map_at_20_diff1 value: 32.91834546416986 - type: nauc_map_at_20_max value: 13.50516841923876 - type: nauc_map_at_20_std value: -6.014704885883195 - type: nauc_map_at_3_diff1 value: 32.80791993872603 - type: nauc_map_at_3_max value: 11.865588752529158 - type: nauc_map_at_3_std value: -7.46123492208037 - type: nauc_map_at_5_diff1 value: 32.87720923080645 - type: nauc_map_at_5_max value: 12.571139947934668 - type: nauc_map_at_5_std value: -7.1390323037613 - type: nauc_mrr_at_1000_diff1 value: 32.6746011819675 - type: nauc_mrr_at_1000_max value: 13.853989466043798 - type: nauc_mrr_at_1000_std value: -4.306780508073467 - type: nauc_mrr_at_100_diff1 value: 32.66974348220786 - type: nauc_mrr_at_100_max value: 13.87521280963595 - type: nauc_mrr_at_100_std value: -4.281623268758372 - type: nauc_mrr_at_10_diff1 value: 32.61464733720552 - type: nauc_mrr_at_10_max value: 13.832776492439633 - type: nauc_mrr_at_10_std value: -4.542617807886527 - type: nauc_mrr_at_1_diff1 value: 34.72533798771275 - type: nauc_mrr_at_1_max value: 10.861316429794243 - type: nauc_mrr_at_1_std value: -5.254029707816079 - type: nauc_mrr_at_20_diff1 value: 32.677004932710894 - type: nauc_mrr_at_20_max value: 13.89066399124665 - type: nauc_mrr_at_20_std value: -4.394511722991158 - type: nauc_mrr_at_3_diff1 value: 32.55790026673851 - type: nauc_mrr_at_3_max value: 12.873874337856527 - type: nauc_mrr_at_3_std value: -5.3827073588128265 - type: nauc_mrr_at_5_diff1 value: 32.528791050435544 - type: nauc_mrr_at_5_max value: 13.181648760273701 - type: nauc_mrr_at_5_std value: -5.058058338649079 - type: nauc_ndcg_at_1000_diff1 value: 32.402797222670785 - type: nauc_ndcg_at_1000_max value: 15.33827444009085 - type: nauc_ndcg_at_1000_std value: -3.656288878646395 - type: nauc_ndcg_at_100_diff1 value: 32.16580228877547 - type: nauc_ndcg_at_100_max value: 16.110100314396796 - type: nauc_ndcg_at_100_std value: -2.617364679505708 - type: nauc_ndcg_at_10_diff1 value: 32.20757651198573 - type: nauc_ndcg_at_10_max value: 15.196608889204091 - type: nauc_ndcg_at_10_std value: -5.136073100942583 - type: nauc_ndcg_at_1_diff1 value: 34.82910472793821 - type: nauc_ndcg_at_1_max value: 10.929310445823809 - type: nauc_ndcg_at_1_std value: -5.169808829987181 - type: nauc_ndcg_at_20_diff1 value: 32.43129783501163 - type: nauc_ndcg_at_20_max value: 15.93657158518566 - type: nauc_ndcg_at_20_std value: -3.976273055504378 - type: nauc_ndcg_at_3_diff1 value: 32.299740007616464 - type: nauc_ndcg_at_3_max value: 12.584778951510017 - type: nauc_ndcg_at_3_std value: -7.303956420107395 - type: nauc_ndcg_at_5_diff1 value: 32.25306359394614 - type: nauc_ndcg_at_5_max value: 13.568498593362552 - type: nauc_ndcg_at_5_std value: -6.762105103486535 - type: nauc_precision_at_1000_diff1 value: -4.0288327326991515 - type: nauc_precision_at_1000_max value: 10.55924632168659 - type: nauc_precision_at_1000_std value: 14.694651019403377 - type: nauc_precision_at_100_diff1 value: 0.707615597626847 - type: nauc_precision_at_100_max value: 18.847648466075505 - type: nauc_precision_at_100_std value: 21.218326061258423 - type: nauc_precision_at_10_diff1 value: 19.33907100470173 - type: nauc_precision_at_10_max value: 20.278902993078106 - type: nauc_precision_at_10_std value: 4.615875975323101 - type: nauc_precision_at_1_diff1 value: 34.82910472793821 - type: nauc_precision_at_1_max value: 10.929310445823809 - type: nauc_precision_at_1_std value: -5.169808829987181 - type: nauc_precision_at_20_diff1 value: 13.684570079407019 - type: nauc_precision_at_20_max value: 21.004712453781277 - type: nauc_precision_at_20_std value: 10.691183107158766 - type: nauc_precision_at_3_diff1 value: 28.87415756089232 - type: nauc_precision_at_3_max value: 15.9587615402185 - type: nauc_precision_at_3_std value: -4.72662256123726 - type: nauc_precision_at_5_diff1 value: 25.253692703184903 - type: nauc_precision_at_5_max value: 17.20245900234252 - type: nauc_precision_at_5_std value: -2.0902250214667126 - type: nauc_recall_at_1000_diff1 value: 29.045808029763105 - type: nauc_recall_at_1000_max value: 72.0085421054025 - type: nauc_recall_at_1000_std value: 66.674461537018 - type: nauc_recall_at_100_diff1 value: 23.524016343264613 - type: nauc_recall_at_100_max value: 48.87885727542797 - type: nauc_recall_at_100_std value: 41.676931807412714 - type: nauc_recall_at_10_diff1 value: 28.448706131448077 - type: nauc_recall_at_10_max value: 20.60216328000016 - type: nauc_recall_at_10_std value: -2.5499050504498073 - type: nauc_recall_at_1_diff1 value: 34.23873886692038 - type: nauc_recall_at_1_max value: 9.192506212634111 - type: nauc_recall_at_1_std value: -7.111967933900576 - type: nauc_recall_at_20_diff1 value: 29.250434345941436 - type: nauc_recall_at_20_max value: 27.07586767599477 - type: nauc_recall_at_20_std value: 4.674120256113827 - type: nauc_recall_at_3_diff1 value: 29.954539543517395 - type: nauc_recall_at_3_max value: 12.891951328267334 - type: nauc_recall_at_3_std value: -8.58992556784897 - type: nauc_recall_at_5_diff1 value: 29.409761377456885 - type: nauc_recall_at_5_max value: 15.120264304508765 - type: nauc_recall_at_5_std value: -7.629729207075528 - type: ndcg_at_1 value: 27.926000000000002 - type: ndcg_at_10 value: 47.496 - type: ndcg_at_100 value: 52.886 - type: ndcg_at_1000 value: 53.632000000000005 - type: ndcg_at_20 value: 50.285000000000004 - type: ndcg_at_3 value: 38.291 - type: ndcg_at_5 value: 42.764 - type: precision_at_1 value: 27.926000000000002 - type: precision_at_10 value: 8.305 - type: precision_at_100 value: 1.131 - type: precision_at_1000 value: 0.12 - type: precision_at_20 value: 4.8149999999999995 - type: precision_at_3 value: 17.729 - type: precision_at_5 value: 13.209999999999999 - type: recall_at_1 value: 24.751 - type: recall_at_10 value: 69.99600000000001 - type: recall_at_100 value: 93.265 - type: recall_at_1000 value: 98.69200000000001 - type: recall_at_20 value: 80.345 - type: recall_at_3 value: 45.899 - type: recall_at_5 value: 56.233 task: type: Retrieval - dataset: config: default name: MTEB QuoraRetrieval revision: e4e08e0b7dbe3c8700f0daef558ff32256715259 split: test type: mteb/quora metrics: - type: main_score value: 87.469 - type: map_at_1 value: 69.226 - type: map_at_10 value: 83.49499999999999 - type: map_at_100 value: 84.16 - type: map_at_1000 value: 84.175 - type: map_at_20 value: 83.937 - type: map_at_3 value: 80.357 - type: map_at_5 value: 82.354 - type: mrr_at_1 value: 79.66 - type: mrr_at_10 value: 86.41408730158712 - type: mrr_at_100 value: 86.53009949625319 - type: mrr_at_1000 value: 86.53085807986054 - type: mrr_at_20 value: 86.50681664426882 - type: mrr_at_3 value: 85.2949999999998 - type: mrr_at_5 value: 86.09149999999971 - type: nauc_map_at_1000_diff1 value: 76.34480513462263 - type: nauc_map_at_1000_max value: 18.35381451054758 - type: nauc_map_at_1000_std value: -45.66385767188469 - type: nauc_map_at_100_diff1 value: 76.34669058530609 - type: nauc_map_at_100_max value: 18.31665217615866 - type: nauc_map_at_100_std value: -45.71646027130385 - type: nauc_map_at_10_diff1 value: 76.55810725489853 - type: nauc_map_at_10_max value: 17.608690417604674 - type: nauc_map_at_10_std value: -47.80834561676489 - type: nauc_map_at_1_diff1 value: 79.82735407515446 - type: nauc_map_at_1_max value: 12.275645564446238 - type: nauc_map_at_1_std value: -41.53738015617803 - type: nauc_map_at_20_diff1 value: 76.39897445057328 - type: nauc_map_at_20_max value: 18.073183264222344 - type: nauc_map_at_20_std value: -46.53343804460259 - type: nauc_map_at_3_diff1 value: 76.8525339580126 - type: nauc_map_at_3_max value: 15.463017873342727 - type: nauc_map_at_3_std value: -49.30092150838741 - type: nauc_map_at_5_diff1 value: 76.84854418575374 - type: nauc_map_at_5_max value: 16.93909969110639 - type: nauc_map_at_5_std value: -49.26303643788198 - type: nauc_mrr_at_1000_diff1 value: 76.11978747472594 - type: nauc_mrr_at_1000_max value: 20.948054256043136 - type: nauc_mrr_at_1000_std value: -40.538438735309654 - type: nauc_mrr_at_100_diff1 value: 76.1194337607922 - type: nauc_mrr_at_100_max value: 20.947907822336376 - type: nauc_mrr_at_100_std value: -40.53901348313209 - type: nauc_mrr_at_10_diff1 value: 76.10022175253289 - type: nauc_mrr_at_10_max value: 20.93817071185543 - type: nauc_mrr_at_10_std value: -40.69322088799321 - type: nauc_mrr_at_1_diff1 value: 77.05621804280337 - type: nauc_mrr_at_1_max value: 20.602559090153918 - type: nauc_mrr_at_1_std value: -38.09297840209946 - type: nauc_mrr_at_20_diff1 value: 76.11382894317693 - type: nauc_mrr_at_20_max value: 20.960340980416362 - type: nauc_mrr_at_20_std value: -40.538013300953075 - type: nauc_mrr_at_3_diff1 value: 75.71270232724483 - type: nauc_mrr_at_3_max value: 20.823905759297613 - type: nauc_mrr_at_3_std value: -41.0941048562992 - type: nauc_mrr_at_5_diff1 value: 76.03551521362746 - type: nauc_mrr_at_5_max value: 21.061039787483693 - type: nauc_mrr_at_5_std value: -41.00853106709472 - type: nauc_ndcg_at_1000_diff1 value: 76.0262393768304 - type: nauc_ndcg_at_1000_max value: 19.800136317778524 - type: nauc_ndcg_at_1000_std value: -43.33700110014878 - type: nauc_ndcg_at_100_diff1 value: 76.01134308572229 - type: nauc_ndcg_at_100_max value: 19.551124579535546 - type: nauc_ndcg_at_100_std value: -43.552980627915744 - type: nauc_ndcg_at_10_diff1 value: 76.15073912713932 - type: nauc_ndcg_at_10_max value: 18.633106350000777 - type: nauc_ndcg_at_10_std value: -46.874070251398344 - type: nauc_ndcg_at_1_diff1 value: 77.05621804280337 - type: nauc_ndcg_at_1_max value: 20.740334635080874 - type: nauc_ndcg_at_1_std value: -38.14163986670599 - type: nauc_ndcg_at_20_diff1 value: 76.056131198408 - type: nauc_ndcg_at_20_max value: 19.12183899438156 - type: nauc_ndcg_at_20_std value: -45.152222560853474 - type: nauc_ndcg_at_3_diff1 value: 75.32067175696538 - type: nauc_ndcg_at_3_max value: 18.030151234107024 - type: nauc_ndcg_at_3_std value: -46.954391996130255 - type: nauc_ndcg_at_5_diff1 value: 76.07279461161282 - type: nauc_ndcg_at_5_max value: 18.676703248713565 - type: nauc_ndcg_at_5_std value: -48.02252026510146 - type: nauc_precision_at_1000_diff1 value: -43.05385069777404 - type: nauc_precision_at_1000_max value: 5.177830943538669 - type: nauc_precision_at_1000_std value: 41.72849191406853 - type: nauc_precision_at_100_diff1 value: -42.74879635836525 - type: nauc_precision_at_100_max value: 4.331905136884358 - type: nauc_precision_at_100_std value: 40.95092522149582 - type: nauc_precision_at_10_diff1 value: -37.24241604831716 - type: nauc_precision_at_10_max value: 5.858095259543203 - type: nauc_precision_at_10_std value: 26.84790750024595 - type: nauc_precision_at_1_diff1 value: 77.05621804280337 - type: nauc_precision_at_1_max value: 20.740334635080874 - type: nauc_precision_at_1_std value: -38.14163986670599 - type: nauc_precision_at_20_diff1 value: -40.981648074847115 - type: nauc_precision_at_20_max value: 4.952729889757595 - type: nauc_precision_at_20_std value: 34.118899263271544 - type: nauc_precision_at_3_diff1 value: -14.632656898252725 - type: nauc_precision_at_3_max value: 9.219680305543351 - type: nauc_precision_at_3_std value: 2.8267912495305985 - type: nauc_precision_at_5_diff1 value: -28.714307596291906 - type: nauc_precision_at_5_max value: 7.8282629737599905 - type: nauc_precision_at_5_std value: 15.30157769264485 - type: nauc_recall_at_1000_diff1 value: 75.28082712455692 - type: nauc_recall_at_1000_max value: 16.032150210343172 - type: nauc_recall_at_1000_std value: 33.623223824303615 - type: nauc_recall_at_100_diff1 value: 66.48857771758026 - type: nauc_recall_at_100_max value: -0.8298370121173152 - type: nauc_recall_at_100_std value: -47.59134911021335 - type: nauc_recall_at_10_diff1 value: 71.54324183557964 - type: nauc_recall_at_10_max value: 10.71338879553047 - type: nauc_recall_at_10_std value: -66.27316488339173 - type: nauc_recall_at_1_diff1 value: 79.82735407515446 - type: nauc_recall_at_1_max value: 12.275645564446238 - type: nauc_recall_at_1_std value: -41.53738015617803 - type: nauc_recall_at_20_diff1 value: 70.15017942540359 - type: nauc_recall_at_20_max value: 11.699321903394438 - type: nauc_recall_at_20_std value: -61.79366072599887 - type: nauc_recall_at_3_diff1 value: 72.5862364297169 - type: nauc_recall_at_3_max value: 12.145065627274022 - type: nauc_recall_at_3_std value: -56.82607742333548 - type: nauc_recall_at_5_diff1 value: 72.07603033579613 - type: nauc_recall_at_5_max value: 13.121563460576949 - type: nauc_recall_at_5_std value: -63.3551506977008 - type: ndcg_at_1 value: 79.66 - type: ndcg_at_10 value: 87.469 - type: ndcg_at_100 value: 88.742 - type: ndcg_at_1000 value: 88.835 - type: ndcg_at_20 value: 88.19 - type: ndcg_at_3 value: 84.301 - type: ndcg_at_5 value: 86.122 - type: precision_at_1 value: 79.66 - type: precision_at_10 value: 13.366 - type: precision_at_100 value: 1.53 - type: precision_at_1000 value: 0.157 - type: precision_at_20 value: 7.114 - type: precision_at_3 value: 36.953 - type: precision_at_5 value: 24.436 - type: recall_at_1 value: 69.226 - type: recall_at_10 value: 95.31500000000001 - type: recall_at_100 value: 99.586 - type: recall_at_1000 value: 99.978 - type: recall_at_20 value: 97.626 - type: recall_at_3 value: 86.39 - type: recall_at_5 value: 91.429 task: type: Retrieval - dataset: config: default name: MTEB RedditClustering revision: 24640382cdbf8abc73003fb0fa6d111a705499eb split: test type: mteb/reddit-clustering metrics: - type: main_score value: 55.46831401047426 - type: v_measure value: 55.46831401047426 - type: v_measure_std value: 4.874007949262904 task: type: Clustering - dataset: config: default name: MTEB RedditClusteringP2P revision: 385e3cb46b4cfa89021f56c4380204149d0efe33 split: test type: mteb/reddit-clustering-p2p metrics: - type: main_score value: 65.95763514051492 - type: v_measure value: 65.95763514051492 - type: v_measure_std value: 12.11633090692919 task: type: Clustering - dataset: config: default name: MTEB SCIDOCS revision: f8c2fcf00f625baaa80f62ec5bd9e1fff3b8ae88 split: test type: mteb/scidocs metrics: - type: main_score value: 19.967 - type: map_at_1 value: 4.555 - type: map_at_10 value: 11.875 - type: map_at_100 value: 13.828 - type: map_at_1000 value: 14.113999999999999 - type: map_at_20 value: 12.837000000000002 - type: map_at_3 value: 8.368 - type: map_at_5 value: 10.02 - type: mrr_at_1 value: 22.5 - type: mrr_at_10 value: 33.34718253968251 - type: mrr_at_100 value: 34.33837465870857 - type: mrr_at_1000 value: 34.405354703860766 - type: mrr_at_20 value: 33.921715836622894 - type: mrr_at_3 value: 29.666666666666657 - type: mrr_at_5 value: 31.966666666666622 - type: nauc_map_at_1000_diff1 value: 15.246786102454319 - type: nauc_map_at_1000_max value: 29.175127621523284 - type: nauc_map_at_1000_std value: 16.942548728665557 - type: nauc_map_at_100_diff1 value: 15.30276004899313 - type: nauc_map_at_100_max value: 29.17103174716017 - type: nauc_map_at_100_std value: 16.7166806855191 - type: nauc_map_at_10_diff1 value: 14.959570431789576 - type: nauc_map_at_10_max value: 28.741898266169592 - type: nauc_map_at_10_std value: 13.598940269616955 - type: nauc_map_at_1_diff1 value: 25.644593996052723 - type: nauc_map_at_1_max value: 23.219289075841036 - type: nauc_map_at_1_std value: 5.690909359575662 - type: nauc_map_at_20_diff1 value: 15.057405804364974 - type: nauc_map_at_20_max value: 29.136129840808078 - type: nauc_map_at_20_std value: 15.438603298509909 - type: nauc_map_at_3_diff1 value: 17.233527889256564 - type: nauc_map_at_3_max value: 26.441548940276377 - type: nauc_map_at_3_std value: 7.752631479331336 - type: nauc_map_at_5_diff1 value: 16.20805385985111 - type: nauc_map_at_5_max value: 27.923553597579676 - type: nauc_map_at_5_std value: 8.860227830088574 - type: nauc_mrr_at_1000_diff1 value: 22.22324136471676 - type: nauc_mrr_at_1000_max value: 25.803381711627445 - type: nauc_mrr_at_1000_std value: 11.195218592300574 - type: nauc_mrr_at_100_diff1 value: 22.20635139247774 - type: nauc_mrr_at_100_max value: 25.809980468254544 - type: nauc_mrr_at_100_std value: 11.236406379403093 - type: nauc_mrr_at_10_diff1 value: 22.197126629234866 - type: nauc_mrr_at_10_max value: 25.751462289366074 - type: nauc_mrr_at_10_std value: 11.06813122476576 - type: nauc_mrr_at_1_diff1 value: 25.768694915906547 - type: nauc_mrr_at_1_max value: 23.078936227533376 - type: nauc_mrr_at_1_std value: 5.664626871415919 - type: nauc_mrr_at_20_diff1 value: 22.333510048115247 - type: nauc_mrr_at_20_max value: 25.933807629983047 - type: nauc_mrr_at_20_std value: 11.375015615529657 - type: nauc_mrr_at_3_diff1 value: 22.243005847898107 - type: nauc_mrr_at_3_max value: 25.592329404890364 - type: nauc_mrr_at_3_std value: 8.649587914029771 - type: nauc_mrr_at_5_diff1 value: 21.995522212053803 - type: nauc_mrr_at_5_max value: 25.8495186116959 - type: nauc_mrr_at_5_std value: 10.335015509252274 - type: nauc_ndcg_at_1000_diff1 value: 15.720525782244835 - type: nauc_ndcg_at_1000_max value: 29.255225161988598 - type: nauc_ndcg_at_1000_std value: 23.91092705272815 - type: nauc_ndcg_at_100_diff1 value: 16.578411443446196 - type: nauc_ndcg_at_100_max value: 29.914913377694734 - type: nauc_ndcg_at_100_std value: 22.811268453728623 - type: nauc_ndcg_at_10_diff1 value: 16.397948417538824 - type: nauc_ndcg_at_10_max value: 29.251930474900416 - type: nauc_ndcg_at_10_std value: 16.636737668789657 - type: nauc_ndcg_at_1_diff1 value: 25.768694915906547 - type: nauc_ndcg_at_1_max value: 23.078936227533376 - type: nauc_ndcg_at_1_std value: 5.664626871415919 - type: nauc_ndcg_at_20_diff1 value: 16.648110682686614 - type: nauc_ndcg_at_20_max value: 29.892133157795776 - type: nauc_ndcg_at_20_std value: 19.34882283787698 - type: nauc_ndcg_at_3_diff1 value: 17.92949134249488 - type: nauc_ndcg_at_3_max value: 26.689585250329532 - type: nauc_ndcg_at_3_std value: 9.078574087750605 - type: nauc_ndcg_at_5_diff1 value: 16.952122694491774 - type: nauc_ndcg_at_5_max value: 28.146728497211676 - type: nauc_ndcg_at_5_std value: 11.339213600616807 - type: nauc_precision_at_1000_diff1 value: 3.2349869782475853 - type: nauc_precision_at_1000_max value: 18.6171242830797 - type: nauc_precision_at_1000_std value: 34.88770887741456 - type: nauc_precision_at_100_diff1 value: 9.966266554726051 - type: nauc_precision_at_100_max value: 25.428513081164468 - type: nauc_precision_at_100_std value: 31.606347333469454 - type: nauc_precision_at_10_diff1 value: 11.95936021707633 - type: nauc_precision_at_10_max value: 29.308109146502513 - type: nauc_precision_at_10_std value: 22.465491503445836 - type: nauc_precision_at_1_diff1 value: 25.768694915906547 - type: nauc_precision_at_1_max value: 23.078936227533376 - type: nauc_precision_at_1_std value: 5.664626871415919 - type: nauc_precision_at_20_diff1 value: 11.875243219095264 - type: nauc_precision_at_20_max value: 28.83348195510638 - type: nauc_precision_at_20_std value: 26.223512022658024 - type: nauc_precision_at_3_diff1 value: 14.439006009003496 - type: nauc_precision_at_3_max value: 27.64037376883578 - type: nauc_precision_at_3_std value: 10.455585887861348 - type: nauc_precision_at_5_diff1 value: 12.944620616816508 - type: nauc_precision_at_5_max value: 29.09151148910875 - type: nauc_precision_at_5_std value: 14.040461515594629 - type: nauc_recall_at_1000_diff1 value: 3.0792658335602208 - type: nauc_recall_at_1000_max value: 18.252163670467 - type: nauc_recall_at_1000_std value: 35.56094641865878 - type: nauc_recall_at_100_diff1 value: 10.082532567537765 - type: nauc_recall_at_100_max value: 25.178104826511184 - type: nauc_recall_at_100_std value: 31.519235979935452 - type: nauc_recall_at_10_diff1 value: 11.756861941382278 - type: nauc_recall_at_10_max value: 28.92701783055643 - type: nauc_recall_at_10_std value: 22.012165818680277 - type: nauc_recall_at_1_diff1 value: 25.644593996052723 - type: nauc_recall_at_1_max value: 23.219289075841036 - type: nauc_recall_at_1_std value: 5.690909359575662 - type: nauc_recall_at_20_diff1 value: 11.832195217863916 - type: nauc_recall_at_20_max value: 28.501838667089185 - type: nauc_recall_at_20_std value: 25.826929326972863 - type: nauc_recall_at_3_diff1 value: 14.320529120124215 - type: nauc_recall_at_3_max value: 27.674150729188774 - type: nauc_recall_at_3_std value: 10.3005989655898 - type: nauc_recall_at_5_diff1 value: 12.524512695192328 - type: nauc_recall_at_5_max value: 28.617421062881988 - type: nauc_recall_at_5_std value: 13.626386741802321 - type: ndcg_at_1 value: 22.5 - type: ndcg_at_10 value: 19.967 - type: ndcg_at_100 value: 27.489 - type: ndcg_at_1000 value: 32.675 - type: ndcg_at_20 value: 22.508 - type: ndcg_at_3 value: 18.593 - type: ndcg_at_5 value: 16.395 - type: precision_at_1 value: 22.5 - type: precision_at_10 value: 10.48 - type: precision_at_100 value: 2.139 - type: precision_at_1000 value: 0.338 - type: precision_at_20 value: 6.710000000000001 - type: precision_at_3 value: 17.5 - type: precision_at_5 value: 14.540000000000001 - type: recall_at_1 value: 4.555 - type: recall_at_10 value: 21.292 - type: recall_at_100 value: 43.438 - type: recall_at_1000 value: 68.693 - type: recall_at_20 value: 27.235 - type: recall_at_3 value: 10.635 - type: recall_at_5 value: 14.773 task: type: Retrieval - dataset: config: default name: MTEB SICK-R revision: 20a6d6f312dd54037fe07a32d58e5e168867909d split: test type: mteb/sickr-sts metrics: - type: cosine_pearson value: 83.80746802573121 - type: cosine_spearman value: 79.72822038569593 - type: euclidean_pearson value: 81.45002980681224 - type: euclidean_spearman value: 79.7282231441618 - type: main_score value: 79.72822038569593 - type: manhattan_pearson value: 81.45611398621358 - type: manhattan_spearman value: 79.59811193829754 - type: pearson value: 83.80746802573121 - type: spearman value: 79.72822038569593 task: type: STS - dataset: config: default name: MTEB STS12 revision: a0d554a64d88156834ff5ae9920b964011b16384 split: test type: mteb/sts12-sts metrics: - type: cosine_pearson value: 87.3063084337013 - type: cosine_spearman value: 80.16653334500153 - type: euclidean_pearson value: 84.24369437820418 - type: euclidean_spearman value: 80.16592228280822 - type: main_score value: 80.16653334500153 - type: manhattan_pearson value: 84.41598625996673 - type: manhattan_spearman value: 80.37993757607657 - type: pearson value: 87.3063084337013 - type: spearman value: 80.16653334500153 task: type: STS - dataset: config: default name: MTEB STS13 revision: 7e90230a92c190f1bf69ae9002b8cea547a64cca split: test type: mteb/sts13-sts metrics: - type: cosine_pearson value: 82.56917286956067 - type: cosine_spearman value: 83.86019247474584 - type: euclidean_pearson value: 83.50577094515269 - type: euclidean_spearman value: 83.86019240020342 - type: main_score value: 83.86019247474584 - type: manhattan_pearson value: 83.7035831723258 - type: manhattan_spearman value: 84.06951592208223 - type: pearson value: 82.56917286956067 - type: spearman value: 83.86019247474584 task: type: STS - dataset: config: default name: MTEB STS14 revision: 6031580fec1f6af667f0bd2da0a551cf4f0b2375 split: test type: mteb/sts14-sts metrics: - type: cosine_pearson value: 82.57349080713945 - type: cosine_spearman value: 80.5656376543151 - type: euclidean_pearson value: 82.20020711843652 - type: euclidean_spearman value: 80.56562435929797 - type: main_score value: 80.5656376543151 - type: manhattan_pearson value: 82.02137444105155 - type: manhattan_spearman value: 80.49473867409459 - type: pearson value: 82.57349080713945 - type: spearman value: 80.5656376543151 task: type: STS - dataset: config: default name: MTEB STS15 revision: ae752c7c21bf194d8b67fd573edf7ae58183cbe3 split: test type: mteb/sts15-sts metrics: - type: cosine_pearson value: 85.89932616809199 - type: cosine_spearman value: 87.34265089376507 - type: euclidean_pearson value: 86.70361638349567 - type: euclidean_spearman value: 87.34265218193244 - type: main_score value: 87.34265089376507 - type: manhattan_pearson value: 86.51826134790748 - type: manhattan_spearman value: 87.18812519859725 - type: pearson value: 85.89932616809199 - type: spearman value: 87.34265089376507 task: type: STS - dataset: config: default name: MTEB STS16 revision: 4d8694f8f0e0100860b497b999b3dbed754a0513 split: test type: mteb/sts16-sts metrics: - type: cosine_pearson value: 83.57539157713117 - type: cosine_spearman value: 84.8327592322146 - type: euclidean_pearson value: 84.04928896411587 - type: euclidean_spearman value: 84.83275862198806 - type: main_score value: 84.8327592322146 - type: manhattan_pearson value: 83.88919473293718 - type: manhattan_spearman value: 84.64175696279177 - type: pearson value: 83.57539157713117 - type: spearman value: 84.8327592322146 task: type: STS - dataset: config: en-ar name: MTEB STS17 (en-ar) revision: faeb762787bd10488a50c8b5be4a3b82e411949c split: test type: mteb/sts17-crosslingual-sts metrics: - type: cosine_pearson value: 56.77939191972398 - type: cosine_spearman value: 56.44421451336106 - type: euclidean_pearson value: 57.84746999725719 - type: euclidean_spearman value: 56.44421451336106 - type: main_score value: 56.44421451336106 - type: manhattan_pearson value: 56.368375736935185 - type: manhattan_spearman value: 54.443033747967476 - type: pearson value: 56.77939191972398 - type: spearman value: 56.44421451336106 task: type: STS - dataset: config: en-de name: MTEB STS17 (en-de) revision: faeb762787bd10488a50c8b5be4a3b82e411949c split: test type: mteb/sts17-crosslingual-sts metrics: - type: cosine_pearson value: 72.7123285460032 - type: cosine_spearman value: 73.0413412902408 - type: euclidean_pearson value: 73.26576109213129 - type: euclidean_spearman value: 73.0413412902408 - type: main_score value: 73.0413412902408 - type: manhattan_pearson value: 73.18845328161191 - type: manhattan_spearman value: 73.01804685924571 - type: pearson value: 72.7123285460032 - type: spearman value: 73.0413412902408 task: type: STS - dataset: config: en-en name: MTEB STS17 (en-en) revision: faeb762787bd10488a50c8b5be4a3b82e411949c split: test type: mteb/sts17-crosslingual-sts metrics: - type: cosine_pearson value: 85.76937213938014 - type: cosine_spearman value: 86.42806650035523 - type: euclidean_pearson value: 86.6429129637805 - type: euclidean_spearman value: 86.42806650035523 - type: main_score value: 86.42806650035523 - type: manhattan_pearson value: 86.57962956067654 - type: manhattan_spearman value: 86.18220646628836 - type: pearson value: 85.76937213938014 - type: spearman value: 86.42806650035523 task: type: STS - dataset: config: en-tr name: MTEB STS17 (en-tr) revision: faeb762787bd10488a50c8b5be4a3b82e411949c split: test type: mteb/sts17-crosslingual-sts metrics: - type: cosine_pearson value: 53.63838562350928 - type: cosine_spearman value: 51.425433307585536 - type: euclidean_pearson value: 53.89159715974961 - type: euclidean_spearman value: 51.425433307585536 - type: main_score value: 51.425433307585536 - type: manhattan_pearson value: 53.072978089852526 - type: manhattan_spearman value: 50.42207979849882 - type: pearson value: 53.63838562350928 - type: spearman value: 51.425433307585536 task: type: STS - dataset: config: es-en name: MTEB STS17 (es-en) revision: faeb762787bd10488a50c8b5be4a3b82e411949c split: test type: mteb/sts17-crosslingual-sts metrics: - type: cosine_pearson value: 74.86829926679785 - type: cosine_spearman value: 75.97371995137269 - type: euclidean_pearson value: 75.31029737838625 - type: euclidean_spearman value: 75.97371995137269 - type: main_score value: 75.97371995137269 - type: manhattan_pearson value: 75.65839894637686 - type: manhattan_spearman value: 76.1355272163419 - type: pearson value: 74.86829926679785 - type: spearman value: 75.97371995137269 task: type: STS - dataset: config: fr-en name: MTEB STS17 (fr-en) revision: faeb762787bd10488a50c8b5be4a3b82e411949c split: test type: mteb/sts17-crosslingual-sts metrics: - type: cosine_pearson value: 74.71447286299487 - type: cosine_spearman value: 74.06702660747132 - type: euclidean_pearson value: 75.38807553614491 - type: euclidean_spearman value: 74.06702660747132 - type: main_score value: 74.06702660747132 - type: manhattan_pearson value: 75.12369492510138 - type: manhattan_spearman value: 73.91369082324793 - type: pearson value: 74.71447286299487 - type: spearman value: 74.06702660747132 task: type: STS - dataset: config: it-en name: MTEB STS17 (it-en) revision: faeb762787bd10488a50c8b5be4a3b82e411949c split: test type: mteb/sts17-crosslingual-sts metrics: - type: cosine_pearson value: 72.38044871568002 - type: cosine_spearman value: 72.67462698051628 - type: euclidean_pearson value: 72.93088753362804 - type: euclidean_spearman value: 72.67462698051628 - type: main_score value: 72.67462698051628 - type: manhattan_pearson value: 72.36507526375641 - type: manhattan_spearman value: 72.05743987811199 - type: pearson value: 72.38044871568002 - type: spearman value: 72.67462698051628 task: type: STS - dataset: config: nl-en name: MTEB STS17 (nl-en) revision: faeb762787bd10488a50c8b5be4a3b82e411949c split: test type: mteb/sts17-crosslingual-sts metrics: - type: cosine_pearson value: 70.21041932205698 - type: cosine_spearman value: 69.70028178727163 - type: euclidean_pearson value: 70.90563853073118 - type: euclidean_spearman value: 69.70028178727163 - type: main_score value: 69.70028178727163 - type: manhattan_pearson value: 70.57814054683753 - type: manhattan_spearman value: 69.09462658139957 - type: pearson value: 70.21041932205698 - type: spearman value: 69.70028178727163 task: type: STS - dataset: config: en name: MTEB STS22 (en) revision: de9d86b3b84231dc21f76c7b7af1f28e2f57f6e3 split: test type: mteb/sts22-crosslingual-sts metrics: - type: cosine_pearson value: 69.13900601047764 - type: cosine_spearman value: 69.2134140940147 - type: euclidean_pearson value: 70.74438767160798 - type: euclidean_spearman value: 69.2134140940147 - type: main_score value: 69.2134140940147 - type: manhattan_pearson value: 71.02398307661159 - type: manhattan_spearman value: 69.37936110586129 - type: pearson value: 69.13900601047764 - type: spearman value: 69.2134140940147 task: type: STS - dataset: config: de-en name: MTEB STS22 (de-en) revision: de9d86b3b84231dc21f76c7b7af1f28e2f57f6e3 split: test type: mteb/sts22-crosslingual-sts metrics: - type: cosine_pearson value: 64.8805006605935 - type: cosine_spearman value: 63.81067904946749 - type: euclidean_pearson value: 67.42954643546759 - type: euclidean_spearman value: 63.81067904946749 - type: main_score value: 63.81067904946749 - type: manhattan_pearson value: 68.5718639745086 - type: manhattan_spearman value: 64.77027566921338 - type: pearson value: 64.8805006605935 - type: spearman value: 63.81067904946749 task: type: STS - dataset: config: es-en name: MTEB STS22 (es-en) revision: de9d86b3b84231dc21f76c7b7af1f28e2f57f6e3 split: test type: mteb/sts22-crosslingual-sts metrics: - type: cosine_pearson value: 75.50011820404733 - type: cosine_spearman value: 78.22942242197527 - type: euclidean_pearson value: 76.7001402354704 - type: euclidean_spearman value: 78.22942242197527 - type: main_score value: 78.22942242197527 - type: manhattan_pearson value: 78.77879852950737 - type: manhattan_spearman value: 79.58485702006229 - type: pearson value: 75.50011820404733 - type: spearman value: 78.22942242197527 task: type: STS - dataset: config: pl-en name: MTEB STS22 (pl-en) revision: de9d86b3b84231dc21f76c7b7af1f28e2f57f6e3 split: test type: mteb/sts22-crosslingual-sts metrics: - type: cosine_pearson value: 78.98435260035146 - type: cosine_spearman value: 79.0083835281799 - type: euclidean_pearson value: 79.05051294275746 - type: euclidean_spearman value: 79.0083835281799 - type: main_score value: 79.0083835281799 - type: manhattan_pearson value: 79.28546319449809 - type: manhattan_spearman value: 78.79790129500051 - type: pearson value: 78.98435260035146 - type: spearman value: 79.0083835281799 task: type: STS - dataset: config: zh-en name: MTEB STS22 (zh-en) revision: de9d86b3b84231dc21f76c7b7af1f28e2f57f6e3 split: test type: mteb/sts22-crosslingual-sts metrics: - type: cosine_pearson value: 76.5491270627292 - type: cosine_spearman value: 76.34747339191613 - type: euclidean_pearson value: 77.52703884917828 - type: euclidean_spearman value: 76.34747339191613 - type: main_score value: 76.34747339191613 - type: manhattan_pearson value: 77.94417346436772 - type: manhattan_spearman value: 76.15552728253843 - type: pearson value: 76.5491270627292 - type: spearman value: 76.34747339191613 task: type: STS - dataset: config: default name: MTEB STSBenchmark revision: b0fddb56ed78048fa8b90373c8a3cfc37b684831 split: test type: mteb/stsbenchmark-sts metrics: - type: cosine_pearson value: 81.44263012519023 - type: cosine_spearman value: 84.1219543406234 - type: euclidean_pearson value: 83.61379006230743 - type: euclidean_spearman value: 84.1219543406234 - type: main_score value: 84.1219543406234 - type: manhattan_pearson value: 83.40336227949633 - type: manhattan_spearman value: 83.94639826019986 - type: pearson value: 81.44263012519023 - type: spearman value: 84.1219543406234 task: type: STS - dataset: config: default name: MTEB SciDocsRR revision: d3c5e1fc0b855ab6097bf1cda04dd73947d7caab split: test type: mteb/scidocs-reranking metrics: - type: main_score value: 80.99314184170828 - type: map value: 80.99314184170828 - type: mrr value: 94.80081918807409 - type: nAUC_map_diff1 value: 2.519316498883737 - type: nAUC_map_max value: 53.83115362914741 - type: nAUC_map_std value: 71.18884208143534 - type: nAUC_mrr_diff1 value: 49.13813585315955 - type: nAUC_mrr_max value: 82.97821200827201 - type: nAUC_mrr_std value: 80.52851378034694 task: type: Reranking - dataset: config: default name: MTEB SciFact revision: 0228b52cf27578f30900b9e5271d331663a030d7 split: test type: mteb/scifact metrics: - type: main_score value: 72.887 - type: map_at_1 value: 56.983 - type: map_at_10 value: 68.02 - type: map_at_100 value: 68.47800000000001 - type: map_at_1000 value: 68.49300000000001 - type: map_at_20 value: 68.319 - type: map_at_3 value: 64.735 - type: map_at_5 value: 66.797 - type: mrr_at_1 value: 60.0 - type: mrr_at_10 value: 68.80674603174603 - type: mrr_at_100 value: 69.17683357934396 - type: mrr_at_1000 value: 69.19182270659053 - type: mrr_at_20 value: 69.04443591467584 - type: mrr_at_3 value: 66.22222222222221 - type: mrr_at_5 value: 67.88888888888889 - type: nauc_map_at_1000_diff1 value: 65.74751339600985 - type: nauc_map_at_1000_max value: 51.01631394192724 - type: nauc_map_at_1000_std value: -4.749077682272638 - type: nauc_map_at_100_diff1 value: 65.7511025839809 - type: nauc_map_at_100_max value: 51.02179965776254 - type: nauc_map_at_100_std value: -4.750648691834171 - type: nauc_map_at_10_diff1 value: 65.8078854827133 - type: nauc_map_at_10_max value: 51.17250624028733 - type: nauc_map_at_10_std value: -5.145740047184135 - type: nauc_map_at_1_diff1 value: 69.1712101533685 - type: nauc_map_at_1_max value: 46.030719621812224 - type: nauc_map_at_1_std value: -8.492699133614764 - type: nauc_map_at_20_diff1 value: 65.72860985168403 - type: nauc_map_at_20_max value: 51.02243609996885 - type: nauc_map_at_20_std value: -4.897680708725447 - type: nauc_map_at_3_diff1 value: 66.04795347641515 - type: nauc_map_at_3_max value: 49.28417356777598 - type: nauc_map_at_3_std value: -8.144355960324793 - type: nauc_map_at_5_diff1 value: 65.39564471101812 - type: nauc_map_at_5_max value: 49.8786868247306 - type: nauc_map_at_5_std value: -5.361632284686927 - type: nauc_mrr_at_1000_diff1 value: 65.35483966038701 - type: nauc_mrr_at_1000_max value: 52.46038918168655 - type: nauc_mrr_at_1000_std value: -2.9488525087661865 - type: nauc_mrr_at_100_diff1 value: 65.35869633533838 - type: nauc_mrr_at_100_max value: 52.465078488348915 - type: nauc_mrr_at_100_std value: -2.9515006369786185 - type: nauc_mrr_at_10_diff1 value: 65.32937085681456 - type: nauc_mrr_at_10_max value: 52.69576591011457 - type: nauc_mrr_at_10_std value: -3.211748975877107 - type: nauc_mrr_at_1_diff1 value: 68.74146631047786 - type: nauc_mrr_at_1_max value: 50.13505491243696 - type: nauc_mrr_at_1_std value: -1.8066686454932306 - type: nauc_mrr_at_20_diff1 value: 65.30292147401966 - type: nauc_mrr_at_20_max value: 52.506885523141 - type: nauc_mrr_at_20_std value: -3.030133033199617 - type: nauc_mrr_at_3_diff1 value: 64.76693508063609 - type: nauc_mrr_at_3_max value: 52.16422755733336 - type: nauc_mrr_at_3_std value: -4.375097949954107 - type: nauc_mrr_at_5_diff1 value: 64.72226560669719 - type: nauc_mrr_at_5_max value: 52.402248270260976 - type: nauc_mrr_at_5_std value: -2.3470762334639543 - type: nauc_ndcg_at_1000_diff1 value: 65.05291476181804 - type: nauc_ndcg_at_1000_max value: 52.29203656356969 - type: nauc_ndcg_at_1000_std value: -3.2044176855608937 - type: nauc_ndcg_at_100_diff1 value: 65.14234495976609 - type: nauc_ndcg_at_100_max value: 52.648998314377046 - type: nauc_ndcg_at_100_std value: -2.836497638004637 - type: nauc_ndcg_at_10_diff1 value: 65.07401371760211 - type: nauc_ndcg_at_10_max value: 53.576715684408214 - type: nauc_ndcg_at_10_std value: -4.308181380375265 - type: nauc_ndcg_at_1_diff1 value: 68.74146631047786 - type: nauc_ndcg_at_1_max value: 50.13505491243696 - type: nauc_ndcg_at_1_std value: -1.8066686454932306 - type: nauc_ndcg_at_20_diff1 value: 64.84825741925087 - type: nauc_ndcg_at_20_max value: 52.92789159833541 - type: nauc_ndcg_at_20_std value: -3.3072860677499856 - type: nauc_ndcg_at_3_diff1 value: 64.37308313406504 - type: nauc_ndcg_at_3_max value: 51.32442115826017 - type: nauc_ndcg_at_3_std value: -7.029903565507216 - type: nauc_ndcg_at_5_diff1 value: 63.86750880299841 - type: nauc_ndcg_at_5_max value: 51.37420252304249 - type: nauc_ndcg_at_5_std value: -3.6483750498877447 - type: nauc_precision_at_1000_diff1 value: -26.510729177260178 - type: nauc_precision_at_1000_max value: 14.42625173734718 - type: nauc_precision_at_1000_std value: 45.496457507034286 - type: nauc_precision_at_100_diff1 value: -14.374685492703476 - type: nauc_precision_at_100_max value: 21.681911020335644 - type: nauc_precision_at_100_std value: 42.16203415919248 - type: nauc_precision_at_10_diff1 value: 7.815716722545709 - type: nauc_precision_at_10_max value: 36.03562535537121 - type: nauc_precision_at_10_std value: 28.24000548987315 - type: nauc_precision_at_1_diff1 value: 68.74146631047786 - type: nauc_precision_at_1_max value: 50.13505491243696 - type: nauc_precision_at_1_std value: -1.8066686454932306 - type: nauc_precision_at_20_diff1 value: -1.0860405342771857 - type: nauc_precision_at_20_max value: 29.773416761471406 - type: nauc_precision_at_20_std value: 34.781738507563304 - type: nauc_precision_at_3_diff1 value: 36.2217738770724 - type: nauc_precision_at_3_max value: 43.356155845594834 - type: nauc_precision_at_3_std value: 9.506045734283152 - type: nauc_precision_at_5_diff1 value: 20.749541232335027 - type: nauc_precision_at_5_max value: 38.57715809985047 - type: nauc_precision_at_5_std value: 22.581774153881096 - type: nauc_recall_at_1000_diff1 value: 12.278244631182748 - type: nauc_recall_at_1000_max value: -56.34920634920787 - type: nauc_recall_at_1000_std value: 12.278244631182748 - type: nauc_recall_at_100_diff1 value: 64.01833460657008 - type: nauc_recall_at_100_max value: 60.00763941940402 - type: nauc_recall_at_100_std value: 18.572277395806154 - type: nauc_recall_at_10_diff1 value: 62.32242861171786 - type: nauc_recall_at_10_max value: 62.75965074851927 - type: nauc_recall_at_10_std value: -6.085630582444918 - type: nauc_recall_at_1_diff1 value: 69.1712101533685 - type: nauc_recall_at_1_max value: 46.030719621812224 - type: nauc_recall_at_1_std value: -8.492699133614764 - type: nauc_recall_at_20_diff1 value: 59.109050124114724 - type: nauc_recall_at_20_max value: 59.41051217234864 - type: nauc_recall_at_20_std value: 3.246339186081203 - type: nauc_recall_at_3_diff1 value: 60.981933198798856 - type: nauc_recall_at_3_max value: 51.04347606806292 - type: nauc_recall_at_3_std value: -12.852440576759369 - type: nauc_recall_at_5_diff1 value: 58.155380815418766 - type: nauc_recall_at_5_max value: 52.3232345936131 - type: nauc_recall_at_5_std value: -1.264784301127294 - type: ndcg_at_1 value: 60.0 - type: ndcg_at_10 value: 72.887 - type: ndcg_at_100 value: 74.92699999999999 - type: ndcg_at_1000 value: 75.351 - type: ndcg_at_20 value: 73.882 - type: ndcg_at_3 value: 67.289 - type: ndcg_at_5 value: 70.348 - type: precision_at_1 value: 60.0 - type: precision_at_10 value: 9.9 - type: precision_at_100 value: 1.093 - type: precision_at_1000 value: 0.11299999999999999 - type: precision_at_20 value: 5.167 - type: precision_at_3 value: 26.556 - type: precision_at_5 value: 17.933 - type: recall_at_1 value: 56.983 - type: recall_at_10 value: 86.989 - type: recall_at_100 value: 96.333 - type: recall_at_1000 value: 99.667 - type: recall_at_20 value: 90.889 - type: recall_at_3 value: 71.989 - type: recall_at_5 value: 79.578 task: type: Retrieval - dataset: config: default name: MTEB SprintDuplicateQuestions revision: d66bd1f72af766a5cc4b0ca5e00c162f89e8cc46 split: test type: mteb/sprintduplicatequestions-pairclassification metrics: - type: cosine_accuracy value: 99.73861386138614 - type: cosine_accuracy_threshold value: 90.32489657402039 - type: cosine_ap value: 92.64599869853095 - type: cosine_f1 value: 86.43371017471738 - type: cosine_f1_threshold value: 90.2580976486206 - type: cosine_precision value: 88.90063424947145 - type: cosine_recall value: 84.1 - type: dot_accuracy value: 99.73861386138614 - type: dot_accuracy_threshold value: 90.32490253448486 - type: dot_ap value: 92.64595539427177 - type: dot_f1 value: 86.43371017471738 - type: dot_f1_threshold value: 90.25810956954956 - type: dot_precision value: 88.90063424947145 - type: dot_recall value: 84.1 - type: euclidean_accuracy value: 99.73861386138614 - type: euclidean_accuracy_threshold value: 43.988871574401855 - type: euclidean_ap value: 92.64599869853096 - type: euclidean_f1 value: 86.43371017471738 - type: euclidean_f1_threshold value: 44.14045214653015 - type: euclidean_precision value: 88.90063424947145 - type: euclidean_recall value: 84.1 - type: main_score value: 93.05685558073557 - type: manhattan_accuracy value: 99.74752475247524 - type: manhattan_accuracy_threshold value: 1051.1520385742188 - type: manhattan_ap value: 93.05685558073557 - type: manhattan_f1 value: 87.06240487062405 - type: manhattan_f1_threshold value: 1051.5135765075684 - type: manhattan_precision value: 88.36251287332647 - type: manhattan_recall value: 85.8 - type: max_ap value: 93.05685558073557 - type: max_f1 value: 87.06240487062405 - type: max_precision value: 88.90063424947145 - type: max_recall value: 85.8 - type: similarity_accuracy value: 99.73861386138614 - type: similarity_accuracy_threshold value: 90.32489657402039 - type: similarity_ap value: 92.64599869853095 - type: similarity_f1 value: 86.43371017471738 - type: similarity_f1_threshold value: 90.2580976486206 - type: similarity_precision value: 88.90063424947145 - type: similarity_recall value: 84.1 task: type: PairClassification - dataset: config: default name: MTEB StackExchangeClustering revision: 6cbc1f7b2bc0622f2e39d2c77fa502909748c259 split: test type: mteb/stackexchange-clustering metrics: - type: main_score value: 66.37519604053632 - type: v_measure value: 66.37519604053632 - type: v_measure_std value: 3.410007122108013 task: type: Clustering - dataset: config: default name: MTEB StackExchangeClusteringP2P revision: 815ca46b2622cec33ccafc3735d572c266efdb44 split: test type: mteb/stackexchange-clustering-p2p metrics: - type: main_score value: 39.19275669318283 - type: v_measure value: 39.19275669318283 - type: v_measure_std value: 1.433472639700171 task: type: Clustering - dataset: config: default name: MTEB StackOverflowDupQuestions revision: e185fbe320c72810689fc5848eb6114e1ef5ec69 split: test type: mteb/stackoverflowdupquestions-reranking metrics: - type: main_score value: 48.37677316210566 - type: map value: 48.37677316210566 - type: mrr value: 49.172223813767935 - type: nAUC_map_diff1 value: 37.067832495829684 - type: nAUC_map_max value: 12.566172393440276 - type: nAUC_map_std value: 9.594421406244894 - type: nAUC_mrr_diff1 value: 37.721801475851414 - type: nAUC_mrr_max value: 13.229276487693506 - type: nAUC_mrr_std value: 10.265502733168592 task: type: Reranking - dataset: config: default name: MTEB SummEval revision: cda12ad7615edc362dbf25a00fdd61d3b1eaf93c split: test type: mteb/summeval metrics: - type: cosine_pearson value: 24.294425360602 - type: cosine_spearman value: 25.227349196483146 - type: dot_pearson value: 24.294425689673897 - type: dot_spearman value: 25.22595105524031 - type: main_score value: 25.227349196483146 - type: pearson value: 24.294425360602 - type: spearman value: 25.227349196483146 task: type: Summarization - dataset: config: default name: MTEB TRECCOVID revision: bb9466bac8153a0349341eb1b22e06409e78ef4e split: test type: mteb/trec-covid metrics: - type: main_score value: 83.71600000000001 - type: map_at_1 value: 0.22499999999999998 - type: map_at_10 value: 2.149 - type: map_at_100 value: 14.036999999999999 - type: map_at_1000 value: 33.592 - type: map_at_20 value: 4.077999999999999 - type: map_at_3 value: 0.697 - type: map_at_5 value: 1.123 - type: mrr_at_1 value: 88.0 - type: mrr_at_10 value: 93.5 - type: mrr_at_100 value: 93.5 - type: mrr_at_1000 value: 93.5 - type: mrr_at_20 value: 93.5 - type: mrr_at_3 value: 93.0 - type: mrr_at_5 value: 93.5 - type: nauc_map_at_1000_diff1 value: -37.74019405875251 - type: nauc_map_at_1000_max value: 43.36659357895312 - type: nauc_map_at_1000_std value: 78.15639459263728 - type: nauc_map_at_100_diff1 value: -30.01165990846415 - type: nauc_map_at_100_max value: 19.246430352246726 - type: nauc_map_at_100_std value: 48.21182298702515 - type: nauc_map_at_10_diff1 value: -9.760408372906864 - type: nauc_map_at_10_max value: 0.7336574098845274 - type: nauc_map_at_10_std value: 11.384489325092082 - type: nauc_map_at_1_diff1 value: 5.032274247676729 - type: nauc_map_at_1_max value: -4.617648500024458 - type: nauc_map_at_1_std value: 8.469833959848321 - type: nauc_map_at_20_diff1 value: -18.752371978248327 - type: nauc_map_at_20_max value: 2.3046738344836064 - type: nauc_map_at_20_std value: 15.833810469532736 - type: nauc_map_at_3_diff1 value: -2.282653784955422 - type: nauc_map_at_3_max value: -5.94327326591755 - type: nauc_map_at_3_std value: 6.684405011691402 - type: nauc_map_at_5_diff1 value: -9.350967821325415 - type: nauc_map_at_5_max value: -2.1321331142366886 - type: nauc_map_at_5_std value: 9.63344735168679 - type: nauc_mrr_at_1000_diff1 value: 19.244301106794655 - type: nauc_mrr_at_1000_max value: 38.04515135967145 - type: nauc_mrr_at_1000_std value: 67.42651909404088 - type: nauc_mrr_at_100_diff1 value: 19.244301106794655 - type: nauc_mrr_at_100_max value: 38.04515135967145 - type: nauc_mrr_at_100_std value: 67.42651909404088 - type: nauc_mrr_at_10_diff1 value: 19.244301106794655 - type: nauc_mrr_at_10_max value: 38.04515135967145 - type: nauc_mrr_at_10_std value: 67.42651909404088 - type: nauc_mrr_at_1_diff1 value: 30.695195672924008 - type: nauc_mrr_at_1_max value: 40.24021635380201 - type: nauc_mrr_at_1_std value: 64.65160674514797 - type: nauc_mrr_at_20_diff1 value: 19.244301106794655 - type: nauc_mrr_at_20_max value: 38.04515135967145 - type: nauc_mrr_at_20_std value: 67.42651909404088 - type: nauc_mrr_at_3_diff1 value: 9.460377871414925 - type: nauc_mrr_at_3_max value: 36.169634361832266 - type: nauc_mrr_at_3_std value: 69.79747179556875 - type: nauc_mrr_at_5_diff1 value: 19.244301106794655 - type: nauc_mrr_at_5_max value: 38.04515135967145 - type: nauc_mrr_at_5_std value: 67.42651909404088 - type: nauc_ndcg_at_1000_diff1 value: -39.737437841505525 - type: nauc_ndcg_at_1000_max value: 39.588873588822274 - type: nauc_ndcg_at_1000_std value: 78.14902788658362 - type: nauc_ndcg_at_100_diff1 value: -29.178863578506586 - type: nauc_ndcg_at_100_max value: 42.85774941223408 - type: nauc_ndcg_at_100_std value: 79.87357453241586 - type: nauc_ndcg_at_10_diff1 value: -26.484391516589657 - type: nauc_ndcg_at_10_max value: 42.72241600167698 - type: nauc_ndcg_at_10_std value: 68.19151094539976 - type: nauc_ndcg_at_1_diff1 value: 9.915392124959421 - type: nauc_ndcg_at_1_max value: 23.839895867230688 - type: nauc_ndcg_at_1_std value: 66.78490074845433 - type: nauc_ndcg_at_20_diff1 value: -33.579927248156764 - type: nauc_ndcg_at_20_max value: 40.70026877258957 - type: nauc_ndcg_at_20_std value: 68.79058594264934 - type: nauc_ndcg_at_3_diff1 value: -9.53242863670409 - type: nauc_ndcg_at_3_max value: 26.871867846146806 - type: nauc_ndcg_at_3_std value: 64.59635427834456 - type: nauc_ndcg_at_5_diff1 value: -28.159706027117203 - type: nauc_ndcg_at_5_max value: 29.146277434723277 - type: nauc_ndcg_at_5_std value: 62.10571075144466 - type: nauc_precision_at_1000_diff1 value: -12.337137609580335 - type: nauc_precision_at_1000_max value: 37.997646654420365 - type: nauc_precision_at_1000_std value: 44.4108831337026 - type: nauc_precision_at_100_diff1 value: -27.204587318780927 - type: nauc_precision_at_100_max value: 45.85476754361804 - type: nauc_precision_at_100_std value: 79.10162967314747 - type: nauc_precision_at_10_diff1 value: -19.712113145761485 - type: nauc_precision_at_10_max value: 61.12322079696251 - type: nauc_precision_at_10_std value: 69.82612276877653 - type: nauc_precision_at_1_diff1 value: 30.695195672924008 - type: nauc_precision_at_1_max value: 40.24021635380201 - type: nauc_precision_at_1_std value: 64.65160674514797 - type: nauc_precision_at_20_diff1 value: -35.81580541107032 - type: nauc_precision_at_20_max value: 49.85546712252859 - type: nauc_precision_at_20_std value: 68.68339783265114 - type: nauc_precision_at_3_diff1 value: -14.43795260039528 - type: nauc_precision_at_3_max value: 53.1517445687952 - type: nauc_precision_at_3_std value: 78.79361421988142 - type: nauc_precision_at_5_diff1 value: -23.592835847661018 - type: nauc_precision_at_5_max value: 63.607630405057 - type: nauc_precision_at_5_std value: 72.10777610903118 - type: nauc_recall_at_1000_diff1 value: -38.947759396555625 - type: nauc_recall_at_1000_max value: 31.775735742211324 - type: nauc_recall_at_1000_std value: 63.77657941191196 - type: nauc_recall_at_100_diff1 value: -29.033439414452474 - type: nauc_recall_at_100_max value: 6.088477112905269 - type: nauc_recall_at_100_std value: 33.644374882067645 - type: nauc_recall_at_10_diff1 value: -10.68139060078312 - type: nauc_recall_at_10_max value: -7.2806613163056895 - type: nauc_recall_at_10_std value: 3.0678840656131996 - type: nauc_recall_at_1_diff1 value: 5.032274247676729 - type: nauc_recall_at_1_max value: -4.617648500024458 - type: nauc_recall_at_1_std value: 8.469833959848321 - type: nauc_recall_at_20_diff1 value: -18.071212870636995 - type: nauc_recall_at_20_max value: -6.953203874741589 - type: nauc_recall_at_20_std value: 6.118400469549319 - type: nauc_recall_at_3_diff1 value: -7.0258132415628145 - type: nauc_recall_at_3_max value: -11.620061195071214 - type: nauc_recall_at_3_std value: 0.46520505953450686 - type: nauc_recall_at_5_diff1 value: -12.816778576127488 - type: nauc_recall_at_5_max value: -8.490882584106425 - type: nauc_recall_at_5_std value: 2.9444123697147924 - type: ndcg_at_1 value: 85.0 - type: ndcg_at_10 value: 83.71600000000001 - type: ndcg_at_100 value: 66.322 - type: ndcg_at_1000 value: 58.794000000000004 - type: ndcg_at_20 value: 81.339 - type: ndcg_at_3 value: 86.827 - type: ndcg_at_5 value: 85.372 - type: precision_at_1 value: 88.0 - type: precision_at_10 value: 88.4 - type: precision_at_100 value: 68.89999999999999 - type: precision_at_1000 value: 25.913999999999998 - type: precision_at_20 value: 86.0 - type: precision_at_3 value: 92.0 - type: precision_at_5 value: 90.8 - type: recall_at_1 value: 0.22499999999999998 - type: recall_at_10 value: 2.3120000000000003 - type: recall_at_100 value: 16.84 - type: recall_at_1000 value: 55.541 - type: recall_at_20 value: 4.478 - type: recall_at_3 value: 0.735 - type: recall_at_5 value: 1.188 task: type: Retrieval - dataset: config: default name: MTEB Touche2020 revision: a34f9a33db75fa0cbb21bb5cfc3dae8dc8bec93f split: test type: mteb/touche2020 metrics: - type: main_score value: 29.148000000000003 - type: map_at_1 value: 2.9690000000000003 - type: map_at_10 value: 12.183 - type: map_at_100 value: 18.934 - type: map_at_1000 value: 20.657 - type: map_at_20 value: 14.995 - type: map_at_3 value: 6.261 - type: map_at_5 value: 8.463 - type: mrr_at_1 value: 40.816326530612244 - type: mrr_at_10 value: 54.33916423712341 - type: mrr_at_100 value: 55.424228921812166 - type: mrr_at_1000 value: 55.424228921812166 - type: mrr_at_20 value: 55.115252184263994 - type: mrr_at_3 value: 51.360544217687064 - type: mrr_at_5 value: 53.707482993197274 - type: nauc_map_at_1000_diff1 value: 24.39650474897996 - type: nauc_map_at_1000_max value: -24.9951891255025 - type: nauc_map_at_1000_std value: 9.605455246028367 - type: nauc_map_at_100_diff1 value: 25.536859954016666 - type: nauc_map_at_100_max value: -25.804629962474262 - type: nauc_map_at_100_std value: 6.269656637665734 - type: nauc_map_at_10_diff1 value: 24.85518759986692 - type: nauc_map_at_10_max value: -23.813502179044356 - type: nauc_map_at_10_std value: -10.511927624839066 - type: nauc_map_at_1_diff1 value: 38.988840017170084 - type: nauc_map_at_1_max value: -32.91111385240854 - type: nauc_map_at_1_std value: -23.593390331872705 - type: nauc_map_at_20_diff1 value: 21.90593939858651 - type: nauc_map_at_20_max value: -23.88012227409119 - type: nauc_map_at_20_std value: -5.8228272625717965 - type: nauc_map_at_3_diff1 value: 32.30334028131513 - type: nauc_map_at_3_max value: -37.94218032778254 - type: nauc_map_at_3_std value: -20.73094293061375 - type: nauc_map_at_5_diff1 value: 22.700908074741232 - type: nauc_map_at_5_max value: -29.289350038869493 - type: nauc_map_at_5_std value: -15.465462985852394 - type: nauc_mrr_at_1000_diff1 value: 36.35450280091282 - type: nauc_mrr_at_1000_max value: -46.85388076847817 - type: nauc_mrr_at_1000_std value: -11.371621140495272 - type: nauc_mrr_at_100_diff1 value: 36.35450280091282 - type: nauc_mrr_at_100_max value: -46.85388076847817 - type: nauc_mrr_at_100_std value: -11.371621140495272 - type: nauc_mrr_at_10_diff1 value: 36.49601491862039 - type: nauc_mrr_at_10_max value: -48.23156353834522 - type: nauc_mrr_at_10_std value: -10.235106950548067 - type: nauc_mrr_at_1_diff1 value: 30.580995175260696 - type: nauc_mrr_at_1_max value: -41.150802011358095 - type: nauc_mrr_at_1_std value: -18.734158460340613 - type: nauc_mrr_at_20_diff1 value: 36.71852563079837 - type: nauc_mrr_at_20_max value: -47.057569621975745 - type: nauc_mrr_at_20_std value: -10.6425202175546 - type: nauc_mrr_at_3_diff1 value: 37.11798353023653 - type: nauc_mrr_at_3_max value: -41.98131493735321 - type: nauc_mrr_at_3_std value: -14.708927784488285 - type: nauc_mrr_at_5_diff1 value: 36.89165099595256 - type: nauc_mrr_at_5_max value: -46.62294841562043 - type: nauc_mrr_at_5_std value: -12.950360469740291 - type: nauc_ndcg_at_1000_diff1 value: 31.229868567293867 - type: nauc_ndcg_at_1000_max value: -32.96979900127197 - type: nauc_ndcg_at_1000_std value: 28.83389826789891 - type: nauc_ndcg_at_100_diff1 value: 36.52267538437917 - type: nauc_ndcg_at_100_max value: -36.30568539371575 - type: nauc_ndcg_at_100_std value: 21.077780005544017 - type: nauc_ndcg_at_10_diff1 value: 35.624818340564026 - type: nauc_ndcg_at_10_max value: -30.41197929897389 - type: nauc_ndcg_at_10_std value: -2.0879378834665285 - type: nauc_ndcg_at_1_diff1 value: 26.09620989819088 - type: nauc_ndcg_at_1_max value: -36.21961356590305 - type: nauc_ndcg_at_1_std value: -16.336654888564233 - type: nauc_ndcg_at_20_diff1 value: 33.145204376285356 - type: nauc_ndcg_at_20_max value: -31.28329863036936 - type: nauc_ndcg_at_20_std value: -0.4725530619650745 - type: nauc_ndcg_at_3_diff1 value: 26.677773521665433 - type: nauc_ndcg_at_3_max value: -32.429541360857414 - type: nauc_ndcg_at_3_std value: -5.50717933012337 - type: nauc_ndcg_at_5_diff1 value: 27.43507358400893 - type: nauc_ndcg_at_5_max value: -31.930933021675834 - type: nauc_ndcg_at_5_std value: -2.223314270488071 - type: nauc_precision_at_1000_diff1 value: -27.19395370994479 - type: nauc_precision_at_1000_max value: 31.32895786519588 - type: nauc_precision_at_1000_std value: 30.385879238372826 - type: nauc_precision_at_100_diff1 value: 18.037320798123492 - type: nauc_precision_at_100_max value: -12.829019500448858 - type: nauc_precision_at_100_std value: 63.7252345522221 - type: nauc_precision_at_10_diff1 value: 33.03740695134771 - type: nauc_precision_at_10_max value: -28.3418055223018 - type: nauc_precision_at_10_std value: 12.399749606566802 - type: nauc_precision_at_1_diff1 value: 30.580995175260696 - type: nauc_precision_at_1_max value: -41.150802011358095 - type: nauc_precision_at_1_std value: -18.734158460340613 - type: nauc_precision_at_20_diff1 value: 24.61739751282666 - type: nauc_precision_at_20_max value: -26.19505578353661 - type: nauc_precision_at_20_std value: 28.082166318953462 - type: nauc_precision_at_3_diff1 value: 24.262619325154343 - type: nauc_precision_at_3_max value: -35.16939673443783 - type: nauc_precision_at_3_std value: -1.5143701562969083 - type: nauc_precision_at_5_diff1 value: 22.767486223120752 - type: nauc_precision_at_5_max value: -32.39632845953219 - type: nauc_precision_at_5_std value: 4.989881878186032 - type: nauc_recall_at_1000_diff1 value: -3.2461494312684294 - type: nauc_recall_at_1000_max value: -2.112903285457026 - type: nauc_recall_at_1000_std value: 70.89954510826574 - type: nauc_recall_at_100_diff1 value: 29.802898837494297 - type: nauc_recall_at_100_max value: -24.20480944377232 - type: nauc_recall_at_100_std value: 35.89183425780783 - type: nauc_recall_at_10_diff1 value: 27.388757054959328 - type: nauc_recall_at_10_max value: -24.989779334676186 - type: nauc_recall_at_10_std value: -9.164641938439107 - type: nauc_recall_at_1_diff1 value: 38.988840017170084 - type: nauc_recall_at_1_max value: -32.91111385240854 - type: nauc_recall_at_1_std value: -23.593390331872705 - type: nauc_recall_at_20_diff1 value: 24.6080131937692 - type: nauc_recall_at_20_max value: -23.05479953156405 - type: nauc_recall_at_20_std value: 0.059402669402089325 - type: nauc_recall_at_3_diff1 value: 27.848810363284425 - type: nauc_recall_at_3_max value: -36.53123928519583 - type: nauc_recall_at_3_std value: -19.758588460277824 - type: nauc_recall_at_5_diff1 value: 19.64218420357169 - type: nauc_recall_at_5_max value: -29.08402015074908 - type: nauc_recall_at_5_std value: -14.506685298292496 - type: ndcg_at_1 value: 38.775999999999996 - type: ndcg_at_10 value: 29.148000000000003 - type: ndcg_at_100 value: 40.945 - type: ndcg_at_1000 value: 52.303999999999995 - type: ndcg_at_20 value: 29.848000000000003 - type: ndcg_at_3 value: 33.222 - type: ndcg_at_5 value: 31.175000000000004 - type: precision_at_1 value: 40.816 - type: precision_at_10 value: 25.509999999999998 - type: precision_at_100 value: 8.347 - type: precision_at_1000 value: 1.5939999999999999 - type: precision_at_20 value: 19.082 - type: precision_at_3 value: 34.694 - type: precision_at_5 value: 31.019999999999996 - type: recall_at_1 value: 2.9690000000000003 - type: recall_at_10 value: 18.979 - type: recall_at_100 value: 50.932 - type: recall_at_1000 value: 85.402 - type: recall_at_20 value: 27.116 - type: recall_at_3 value: 7.619 - type: recall_at_5 value: 11.448 task: type: Retrieval - dataset: config: default name: MTEB ToxicConversationsClassification revision: edfaf9da55d3dd50d43143d90c1ac476895ae6de split: test type: mteb/toxic_conversations_50k metrics: - type: accuracy value: 89.27734375 - type: ap value: 35.194960239779064 - type: ap_weighted value: 35.194960239779064 - type: f1 value: 74.3697271257718 - type: f1_weighted value: 90.81303890497676 - type: main_score value: 89.27734375 task: type: Classification - dataset: config: default name: MTEB TweetSentimentExtractionClassification revision: d604517c81ca91fe16a244d1248fc021f9ecee7a split: test type: mteb/tweet_sentiment_extraction metrics: - type: accuracy value: 76.3469156762875 - type: f1 value: 76.53667544013393 - type: f1_weighted value: 75.88734027230643 - type: main_score value: 76.3469156762875 task: type: Classification - dataset: config: default name: MTEB TwentyNewsgroupsClustering revision: 6125ec4e24fa026cec8a478383ee943acfbd5449 split: test type: mteb/twentynewsgroups-clustering metrics: - type: main_score value: 49.32687063841142 - type: v_measure value: 49.32687063841142 - type: v_measure_std value: 1.4415471512453628 task: type: Clustering - dataset: config: default name: MTEB TwitterSemEval2015 revision: 70970daeab8776df92f5ea462b6173c0b46fd2d1 split: test type: mteb/twittersemeval2015-pairclassification metrics: - type: cosine_accuracy value: 85.24169994635513 - type: cosine_accuracy_threshold value: 91.52344465255737 - type: cosine_ap value: 71.43912939014582 - type: cosine_f1 value: 66.87586293460525 - type: cosine_f1_threshold value: 90.44243097305298 - type: cosine_precision value: 63.777830979171654 - type: cosine_recall value: 70.29023746701847 - type: dot_accuracy value: 85.24169994635513 - type: dot_accuracy_threshold value: 91.52344465255737 - type: dot_ap value: 71.43910347543935 - type: dot_f1 value: 66.87586293460525 - type: dot_f1_threshold value: 90.44243693351746 - type: dot_precision value: 63.777830979171654 - type: dot_recall value: 70.29023746701847 - type: euclidean_accuracy value: 85.24169994635513 - type: euclidean_accuracy_threshold value: 41.17414355278015 - type: euclidean_ap value: 71.43911775197303 - type: euclidean_f1 value: 66.87586293460525 - type: euclidean_f1_threshold value: 43.72085928916931 - type: euclidean_precision value: 63.777830979171654 - type: euclidean_recall value: 70.29023746701847 - type: main_score value: 71.43912939014582 - type: manhattan_accuracy value: 85.02115992132086 - type: manhattan_accuracy_threshold value: 970.2783584594727 - type: manhattan_ap value: 71.11811615315935 - type: manhattan_f1 value: 66.46334024643161 - type: manhattan_f1_threshold value: 1044.2398071289062 - type: manhattan_precision value: 61.81075561606535 - type: manhattan_recall value: 71.87335092348285 - type: max_ap value: 71.43912939014582 - type: max_f1 value: 66.87586293460525 - type: max_precision value: 63.777830979171654 - type: max_recall value: 71.87335092348285 - type: similarity_accuracy value: 85.24169994635513 - type: similarity_accuracy_threshold value: 91.52344465255737 - type: similarity_ap value: 71.43912939014582 - type: similarity_f1 value: 66.87586293460525 - type: similarity_f1_threshold value: 90.44243097305298 - type: similarity_precision value: 63.777830979171654 - type: similarity_recall value: 70.29023746701847 task: type: PairClassification - dataset: config: default name: MTEB TwitterURLCorpus revision: 8b6510b0b1fa4e4c4f879467980e9be563ec1cdf split: test type: mteb/twitterurlcorpus-pairclassification metrics: - type: cosine_accuracy value: 88.85590095859045 - type: cosine_accuracy_threshold value: 88.8430118560791 - type: cosine_ap value: 85.68756441554032 - type: cosine_f1 value: 77.73691444704485 - type: cosine_f1_threshold value: 87.81020641326904 - type: cosine_precision value: 75.10587897494796 - type: cosine_recall value: 80.55897751770866 - type: dot_accuracy value: 88.85590095859045 - type: dot_accuracy_threshold value: 88.84301781654358 - type: dot_ap value: 85.6875209900991 - type: dot_f1 value: 77.73691444704485 - type: dot_f1_threshold value: 87.810218334198 - type: dot_precision value: 75.10587897494796 - type: dot_recall value: 80.55897751770866 - type: euclidean_accuracy value: 88.85590095859045 - type: euclidean_accuracy_threshold value: 47.23767042160034 - type: euclidean_ap value: 85.68752572543856 - type: euclidean_f1 value: 77.73691444704485 - type: euclidean_f1_threshold value: 49.375683069229126 - type: euclidean_precision value: 75.10587897494796 - type: euclidean_recall value: 80.55897751770866 - type: main_score value: 85.68756441554032 - type: manhattan_accuracy value: 88.86366282454303 - type: manhattan_accuracy_threshold value: 1111.7809295654297 - type: manhattan_ap value: 85.65387985046206 - type: manhattan_f1 value: 77.74802915365164 - type: manhattan_f1_threshold value: 1163.0821228027344 - type: manhattan_precision value: 75.18699654775605 - type: manhattan_recall value: 80.48968278410841 - type: max_ap value: 85.68756441554032 - type: max_f1 value: 77.74802915365164 - type: max_precision value: 75.18699654775605 - type: max_recall value: 80.55897751770866 - type: similarity_accuracy value: 88.85590095859045 - type: similarity_accuracy_threshold value: 88.8430118560791 - type: similarity_ap value: 85.68756441554032 - type: similarity_f1 value: 77.73691444704485 - type: similarity_f1_threshold value: 87.81020641326904 - type: similarity_precision value: 75.10587897494796 - type: similarity_recall value: 80.55897751770866 task: type: PairClassification - dataset: config: default name: MTEB AFQMC revision: b44c3b011063adb25877c13823db83bb193913c4 split: validation type: C-MTEB/AFQMC metrics: - type: cosine_pearson value: 37.12876239452581 - type: cosine_spearman value: 38.067257174211036 - type: euclidean_pearson value: 36.735658383168044 - type: euclidean_spearman value: 38.06725717137013 - type: main_score value: 38.067257174211036 - type: manhattan_pearson value: 36.43829602976309 - type: manhattan_spearman value: 37.73959740816105 - type: pearson value: 37.12876239452581 - type: spearman value: 38.067257174211036 task: type: STS - dataset: config: default name: MTEB ATEC revision: 0f319b1142f28d00e055a6770f3f726ae9b7d865 split: test type: C-MTEB/ATEC metrics: - type: cosine_pearson value: 45.55151167790002 - type: cosine_spearman value: 46.22151486832593 - type: euclidean_pearson value: 48.31921364975583 - type: euclidean_spearman value: 46.22151824485588 - type: main_score value: 46.22151486832593 - type: manhattan_pearson value: 48.12062382773407 - type: manhattan_spearman value: 46.00780031287469 - type: pearson value: 45.55151167790002 - type: spearman value: 46.22151486832593 task: type: STS - dataset: config: zh name: MTEB AmazonReviewsClassification (zh) revision: 1399c76144fd37290681b995c656ef9b2e06e26d split: test type: mteb/amazon_reviews_multi metrics: - type: accuracy value: 50.978 - type: f1 value: 47.04754599162115 - type: f1_weighted value: 47.04754599162114 - type: main_score value: 50.978 task: type: Classification - dataset: config: default name: MTEB BQ revision: e3dda5e115e487b39ec7e618c0c6a29137052a55 split: test type: C-MTEB/BQ metrics: - type: cosine_pearson value: 53.84488943094217 - type: cosine_spearman value: 54.49605523234321 - type: euclidean_pearson value: 53.50943690170382 - type: euclidean_spearman value: 54.49605869546392 - type: main_score value: 54.49605523234321 - type: manhattan_pearson value: 53.24052011197216 - type: manhattan_spearman value: 54.215799016988996 - type: pearson value: 53.84488943094217 - type: spearman value: 54.49605523234321 task: type: STS - dataset: config: default name: MTEB CLSClusteringP2P revision: 4b6227591c6c1a73bc76b1055f3b7f3588e72476 split: test type: C-MTEB/CLSClusteringP2P metrics: - type: main_score value: 46.915444917880286 - type: v_measure value: 46.915444917880286 - type: v_measure_std value: 1.5837973591025165 task: type: Clustering - dataset: config: default name: MTEB CLSClusteringS2S revision: e458b3f5414b62b7f9f83499ac1f5497ae2e869f split: test type: C-MTEB/CLSClusteringS2S metrics: - type: main_score value: 44.89461372862163 - type: v_measure value: 44.89461372862163 - type: v_measure_std value: 1.0128909727353739 task: type: Clustering - dataset: config: default name: MTEB CMedQAv1 revision: 8d7f1e942507dac42dc58017c1a001c3717da7df split: test type: C-MTEB/CMedQAv1-reranking metrics: - type: main_score value: 82.21191216422777 - type: map value: 82.21191216422777 - type: mrr value: 84.98567460317462 - type: nAUC_map_diff1 value: 55.27793122843884 - type: nAUC_map_max value: 61.78021845841622 - type: nAUC_map_std value: 24.32874017213633 - type: nAUC_mrr_diff1 value: 62.56855683834338 - type: nAUC_mrr_max value: 69.83242448232605 - type: nAUC_mrr_std value: 30.753945346898497 task: type: Reranking - dataset: config: default name: MTEB CMedQAv2 revision: 23d186750531a14a0357ca22cd92d712fd512ea0 split: test type: C-MTEB/CMedQAv2-reranking metrics: - type: main_score value: 83.08227189741763 - type: map value: 83.08227189741763 - type: mrr value: 86.19051587301587 - type: nAUC_map_diff1 value: 58.735936259553654 - type: nAUC_map_max value: 60.14841462146858 - type: nAUC_map_std value: 17.761882943403222 - type: nAUC_mrr_diff1 value: 67.43320028439246 - type: nAUC_mrr_max value: 70.96971853942028 - type: nAUC_mrr_std value: 27.963212156051533 task: type: Reranking - dataset: config: default name: MTEB CmedqaRetrieval revision: cd540c506dae1cf9e9a59c3e06f42030d54e7301 split: dev type: C-MTEB/CmedqaRetrieval metrics: - type: main_score value: 42.114000000000004 - type: map_at_1 value: 24.712999999999997 - type: map_at_10 value: 35.977 - type: map_at_100 value: 37.626 - type: map_at_1000 value: 37.768 - type: map_at_20 value: 36.855 - type: map_at_3 value: 32.263 - type: map_at_5 value: 34.458 - type: mrr_at_1 value: 37.80945236309077 - type: mrr_at_10 value: 45.2176833891012 - type: mrr_at_100 value: 46.120647602744036 - type: mrr_at_1000 value: 46.18072259198463 - type: mrr_at_20 value: 45.733062196151344 - type: mrr_at_3 value: 42.95657247645238 - type: mrr_at_5 value: 44.2944069350671 - type: nauc_map_at_1000_diff1 value: 47.73425387510818 - type: nauc_map_at_1000_max value: 35.955688760781 - type: nauc_map_at_1000_std value: -5.337048907024208 - type: nauc_map_at_100_diff1 value: 47.691721439486415 - type: nauc_map_at_100_max value: 35.9120286982508 - type: nauc_map_at_100_std value: -5.388715572820414 - type: nauc_map_at_10_diff1 value: 47.59595769097994 - type: nauc_map_at_10_max value: 34.868575436637464 - type: nauc_map_at_10_std value: -6.38635344513967 - type: nauc_map_at_1_diff1 value: 51.697402901732794 - type: nauc_map_at_1_max value: 27.169028835962006 - type: nauc_map_at_1_std value: -8.36280973549372 - type: nauc_map_at_20_diff1 value: 47.6021479339286 - type: nauc_map_at_20_max value: 35.42358169640706 - type: nauc_map_at_20_std value: -5.995707059550167 - type: nauc_map_at_3_diff1 value: 48.12995337761301 - type: nauc_map_at_3_max value: 32.29594600680629 - type: nauc_map_at_3_std value: -7.995165343021071 - type: nauc_map_at_5_diff1 value: 47.97666743341936 - type: nauc_map_at_5_max value: 33.98865164184045 - type: nauc_map_at_5_std value: -7.202624274535322 - type: nauc_mrr_at_1000_diff1 value: 53.91400893930025 - type: nauc_mrr_at_1000_max value: 42.756637412490015 - type: nauc_mrr_at_1000_std value: -1.7033054319306657 - type: nauc_mrr_at_100_diff1 value: 53.88165315274917 - type: nauc_mrr_at_100_max value: 42.751083094268274 - type: nauc_mrr_at_100_std value: -1.6918871915944649 - type: nauc_mrr_at_10_diff1 value: 53.91581744122965 - type: nauc_mrr_at_10_max value: 42.58709846134356 - type: nauc_mrr_at_10_std value: -1.9103231104709821 - type: nauc_mrr_at_1_diff1 value: 59.37942921389244 - type: nauc_mrr_at_1_max value: 43.889994340792285 - type: nauc_mrr_at_1_std value: -2.9334150073957 - type: nauc_mrr_at_20_diff1 value: 53.82364054543749 - type: nauc_mrr_at_20_max value: 42.657132121329866 - type: nauc_mrr_at_20_std value: -1.814080881374272 - type: nauc_mrr_at_3_diff1 value: 54.82194831261354 - type: nauc_mrr_at_3_max value: 42.867175235515795 - type: nauc_mrr_at_3_std value: -2.3672343827332343 - type: nauc_mrr_at_5_diff1 value: 54.28110236719863 - type: nauc_mrr_at_5_max value: 42.773659727997604 - type: nauc_mrr_at_5_std value: -1.9843150717670213 - type: nauc_ndcg_at_1000_diff1 value: 48.08750394739328 - type: nauc_ndcg_at_1000_max value: 39.354002076766356 - type: nauc_ndcg_at_1000_std value: -1.642209870956442 - type: nauc_ndcg_at_100_diff1 value: 47.02373205823187 - type: nauc_ndcg_at_100_max value: 39.071767544616016 - type: nauc_ndcg_at_100_std value: -1.4318887896374275 - type: nauc_ndcg_at_10_diff1 value: 46.87201101025155 - type: nauc_ndcg_at_10_max value: 36.596107252610715 - type: nauc_ndcg_at_10_std value: -4.491855380124351 - type: nauc_ndcg_at_1_diff1 value: 59.37942921389244 - type: nauc_ndcg_at_1_max value: 43.889994340792285 - type: nauc_ndcg_at_1_std value: -2.9334150073957 - type: nauc_ndcg_at_20_diff1 value: 46.69383079853486 - type: nauc_ndcg_at_20_max value: 37.42651190472187 - type: nauc_ndcg_at_20_std value: -3.8283649672503555 - type: nauc_ndcg_at_3_diff1 value: 48.767394875248215 - type: nauc_ndcg_at_3_max value: 37.896558640830364 - type: nauc_ndcg_at_3_std value: -4.549323419648498 - type: nauc_ndcg_at_5_diff1 value: 47.86427501476368 - type: nauc_ndcg_at_5_max value: 36.911634117836165 - type: nauc_ndcg_at_5_std value: -4.724216711512233 - type: nauc_precision_at_1000_diff1 value: 5.215840982527869 - type: nauc_precision_at_1000_max value: 29.159568859241432 - type: nauc_precision_at_1000_std value: 21.082099239516065 - type: nauc_precision_at_100_diff1 value: 11.484383878054516 - type: nauc_precision_at_100_max value: 36.979612782453096 - type: nauc_precision_at_100_std value: 21.37719349152994 - type: nauc_precision_at_10_diff1 value: 25.43246921524832 - type: nauc_precision_at_10_max value: 42.19035665238435 - type: nauc_precision_at_10_std value: 9.100059364841798 - type: nauc_precision_at_1_diff1 value: 59.37942921389244 - type: nauc_precision_at_1_max value: 43.889994340792285 - type: nauc_precision_at_1_std value: -2.9334150073957 - type: nauc_precision_at_20_diff1 value: 20.71145394651256 - type: nauc_precision_at_20_max value: 40.864109895424676 - type: nauc_precision_at_20_std value: 12.37767593665739 - type: nauc_precision_at_3_diff1 value: 36.796679107157864 - type: nauc_precision_at_3_max value: 43.86084233138964 - type: nauc_precision_at_3_std value: 2.284927565289776 - type: nauc_precision_at_5_diff1 value: 31.736303229558853 - type: nauc_precision_at_5_max value: 43.979769552220695 - type: nauc_precision_at_5_std value: 4.953476938204816 - type: nauc_recall_at_1000_diff1 value: 33.99656164674807 - type: nauc_recall_at_1000_max value: 51.409315146611604 - type: nauc_recall_at_1000_std value: 46.40969047509407 - type: nauc_recall_at_100_diff1 value: 28.008400601965988 - type: nauc_recall_at_100_max value: 35.45902334586444 - type: nauc_recall_at_100_std value: 12.00354716277477 - type: nauc_recall_at_10_diff1 value: 35.11874174482894 - type: nauc_recall_at_10_max value: 29.383597829116265 - type: nauc_recall_at_10_std value: -3.990278366414269 - type: nauc_recall_at_1_diff1 value: 51.697402901732794 - type: nauc_recall_at_1_max value: 27.169028835962006 - type: nauc_recall_at_1_std value: -8.36280973549372 - type: nauc_recall_at_20_diff1 value: 32.85706970308098 - type: nauc_recall_at_20_max value: 30.609762001249297 - type: nauc_recall_at_20_std value: -2.0802705155631305 - type: nauc_recall_at_3_diff1 value: 41.293789470043485 - type: nauc_recall_at_3_max value: 29.091549134985883 - type: nauc_recall_at_3_std value: -7.863274012056401 - type: nauc_recall_at_5_diff1 value: 38.87911660795755 - type: nauc_recall_at_5_max value: 29.81974055210633 - type: nauc_recall_at_5_std value: -5.940913211156581 - type: ndcg_at_1 value: 37.809 - type: ndcg_at_10 value: 42.114000000000004 - type: ndcg_at_100 value: 48.76 - type: ndcg_at_1000 value: 51.41 - type: ndcg_at_20 value: 44.528 - type: ndcg_at_3 value: 37.621 - type: ndcg_at_5 value: 39.627 - type: precision_at_1 value: 37.809 - type: precision_at_10 value: 9.215 - type: precision_at_100 value: 1.468 - type: precision_at_1000 value: 0.18 - type: precision_at_20 value: 5.423 - type: precision_at_3 value: 21.214 - type: precision_at_5 value: 15.323999999999998 - type: recall_at_1 value: 24.712999999999997 - type: recall_at_10 value: 51.08200000000001 - type: recall_at_100 value: 78.855 - type: recall_at_1000 value: 96.897 - type: recall_at_20 value: 59.232 - type: recall_at_3 value: 37.147000000000006 - type: recall_at_5 value: 43.579 task: type: Retrieval - dataset: config: default name: MTEB Cmnli revision: 41bc36f332156f7adc9e38f53777c959b2ae9766 split: validation type: C-MTEB/CMNLI metrics: - type: cosine_accuracy value: 69.70535177390259 - type: cosine_accuracy_threshold value: 85.75683832168579 - type: cosine_ap value: 76.66113306358733 - type: cosine_f1 value: 72.686230248307 - type: cosine_f1_threshold value: 82.12493658065796 - type: cosine_precision value: 62.63531799729364 - type: cosine_recall value: 86.57937806873977 - type: dot_accuracy value: 69.70535177390259 - type: dot_accuracy_threshold value: 85.75684428215027 - type: dot_ap value: 76.68137107496162 - type: dot_f1 value: 72.686230248307 - type: dot_f1_threshold value: 82.12493658065796 - type: dot_precision value: 62.63531799729364 - type: dot_recall value: 86.57937806873977 - type: euclidean_accuracy value: 69.70535177390259 - type: euclidean_accuracy_threshold value: 53.37257385253906 - type: euclidean_ap value: 76.6611302634097 - type: euclidean_f1 value: 72.686230248307 - type: euclidean_f1_threshold value: 59.79139804840088 - type: euclidean_precision value: 62.63531799729364 - type: euclidean_recall value: 86.57937806873977 - type: main_score value: 76.68137107496162 - type: manhattan_accuracy value: 69.27239927841251 - type: manhattan_accuracy_threshold value: 1262.4760627746582 - type: manhattan_ap value: 76.1926237206237 - type: manhattan_f1 value: 72.39749155812832 - type: manhattan_f1_threshold value: 1421.414852142334 - type: manhattan_precision value: 61.62943495400789 - type: manhattan_recall value: 87.72504091653029 - type: max_ap value: 76.68137107496162 - type: max_f1 value: 72.686230248307 - type: max_precision value: 62.63531799729364 - type: max_recall value: 87.72504091653029 - type: similarity_accuracy value: 69.70535177390259 - type: similarity_accuracy_threshold value: 85.75683832168579 - type: similarity_ap value: 76.66113306358733 - type: similarity_f1 value: 72.686230248307 - type: similarity_f1_threshold value: 82.12493658065796 - type: similarity_precision value: 62.63531799729364 - type: similarity_recall value: 86.57937806873977 task: type: PairClassification - dataset: config: default name: MTEB CovidRetrieval revision: 1271c7809071a13532e05f25fb53511ffce77117 split: dev type: C-MTEB/CovidRetrieval metrics: - type: main_score value: 82.311 - type: map_at_1 value: 69.679 - type: map_at_10 value: 78.348 - type: map_at_100 value: 78.596 - type: map_at_1000 value: 78.601 - type: map_at_20 value: 78.545 - type: map_at_3 value: 76.387 - type: map_at_5 value: 77.655 - type: mrr_at_1 value: 69.86301369863014 - type: mrr_at_10 value: 78.34249084249088 - type: mrr_at_100 value: 78.59092165386024 - type: mrr_at_1000 value: 78.59525747388338 - type: mrr_at_20 value: 78.53925752890612 - type: mrr_at_3 value: 76.46645591851072 - type: mrr_at_5 value: 77.68879522304182 - type: nauc_map_at_1000_diff1 value: 79.4462259252306 - type: nauc_map_at_1000_max value: 31.7925049828458 - type: nauc_map_at_1000_std value: -44.987963962793955 - type: nauc_map_at_100_diff1 value: 79.44473843692927 - type: nauc_map_at_100_max value: 31.798193923558003 - type: nauc_map_at_100_std value: -44.98700550161243 - type: nauc_map_at_10_diff1 value: 79.33651124191677 - type: nauc_map_at_10_max value: 31.67512594432897 - type: nauc_map_at_10_std value: -45.317005211451914 - type: nauc_map_at_1_diff1 value: 81.69181674577305 - type: nauc_map_at_1_max value: 30.817373790682357 - type: nauc_map_at_1_std value: -42.72748236505566 - type: nauc_map_at_20_diff1 value: 79.41582541979186 - type: nauc_map_at_20_max value: 31.88246428726322 - type: nauc_map_at_20_std value: -44.99284598367759 - type: nauc_map_at_3_diff1 value: 79.32027545323751 - type: nauc_map_at_3_max value: 31.27336204905954 - type: nauc_map_at_3_std value: -46.25198460252002 - type: nauc_map_at_5_diff1 value: 79.13580536546466 - type: nauc_map_at_5_max value: 31.341889475033714 - type: nauc_map_at_5_std value: -46.04029107468536 - type: nauc_mrr_at_1000_diff1 value: 79.44015632094633 - type: nauc_mrr_at_1000_max value: 32.21963353786539 - type: nauc_mrr_at_1000_std value: -44.52548111878723 - type: nauc_mrr_at_100_diff1 value: 79.43867097380709 - type: nauc_mrr_at_100_max value: 32.22522235955093 - type: nauc_mrr_at_100_std value: -44.52462840708672 - type: nauc_mrr_at_10_diff1 value: 79.33057644986008 - type: nauc_mrr_at_10_max value: 32.09671935370013 - type: nauc_mrr_at_10_std value: -44.860413851737526 - type: nauc_mrr_at_1_diff1 value: 81.44692749809782 - type: nauc_mrr_at_1_max value: 31.849651568624132 - type: nauc_mrr_at_1_std value: -41.3253061502398 - type: nauc_mrr_at_20_diff1 value: 79.40978909435384 - type: nauc_mrr_at_20_max value: 32.3082948412038 - type: nauc_mrr_at_20_std value: -44.5317164768648 - type: nauc_mrr_at_3_diff1 value: 79.1638266419462 - type: nauc_mrr_at_3_max value: 32.0842169384742 - type: nauc_mrr_at_3_std value: -45.46170783914986 - type: nauc_mrr_at_5_diff1 value: 79.05732925048218 - type: nauc_mrr_at_5_max value: 31.866875109079274 - type: nauc_mrr_at_5_std value: -45.47315535458264 - type: nauc_ndcg_at_1000_diff1 value: 79.14506487967446 - type: nauc_ndcg_at_1000_max value: 32.28815962385816 - type: nauc_ndcg_at_1000_std value: -44.84164255319649 - type: nauc_ndcg_at_100_diff1 value: 79.11589619449164 - type: nauc_ndcg_at_100_max value: 32.48358720627568 - type: nauc_ndcg_at_100_std value: -44.77161186390409 - type: nauc_ndcg_at_10_diff1 value: 78.6112419611499 - type: nauc_ndcg_at_10_max value: 32.02532602435404 - type: nauc_ndcg_at_10_std value: -46.22564777302159 - type: nauc_ndcg_at_1_diff1 value: 81.224021579675 - type: nauc_ndcg_at_1_max value: 31.775986317223122 - type: nauc_ndcg_at_1_std value: -41.46817150641969 - type: nauc_ndcg_at_20_diff1 value: 78.91451933721699 - type: nauc_ndcg_at_20_max value: 33.07095994019382 - type: nauc_ndcg_at_20_std value: -44.793080774799556 - type: nauc_ndcg_at_3_diff1 value: 78.45748790579262 - type: nauc_ndcg_at_3_max value: 31.33438980452425 - type: nauc_ndcg_at_3_std value: -47.961287399778165 - type: nauc_ndcg_at_5_diff1 value: 78.00517989020184 - type: nauc_ndcg_at_5_max value: 31.377723112970337 - type: nauc_ndcg_at_5_std value: -47.64749257136376 - type: nauc_precision_at_1000_diff1 value: -36.954287113132764 - type: nauc_precision_at_1000_max value: 33.399149331718306 - type: nauc_precision_at_1000_std value: 64.36619497853745 - type: nauc_precision_at_100_diff1 value: -1.1165501851098696 - type: nauc_precision_at_100_max value: 40.744208492042645 - type: nauc_precision_at_100_std value: 32.59574168316801 - type: nauc_precision_at_10_diff1 value: 41.66995611089304 - type: nauc_precision_at_10_max value: 32.67778815088798 - type: nauc_precision_at_10_std value: -25.51047355817746 - type: nauc_precision_at_1_diff1 value: 81.224021579675 - type: nauc_precision_at_1_max value: 31.775986317223122 - type: nauc_precision_at_1_std value: -41.46817150641969 - type: nauc_precision_at_20_diff1 value: 24.5912939655554 - type: nauc_precision_at_20_max value: 47.02616085142694 - type: nauc_precision_at_20_std value: 4.541053315120063 - type: nauc_precision_at_3_diff1 value: 70.56985549600088 - type: nauc_precision_at_3_max value: 31.221388227405512 - type: nauc_precision_at_3_std value: -50.297375067273066 - type: nauc_precision_at_5_diff1 value: 59.63106705169415 - type: nauc_precision_at_5_max value: 31.785628018433776 - type: nauc_precision_at_5_std value: -43.803356847579565 - type: nauc_recall_at_1000_diff1 value: 86.10840824375175 - type: nauc_recall_at_1000_max value: 86.10840824375175 - type: nauc_recall_at_1000_std value: 67.89043659493016 - type: nauc_recall_at_100_diff1 value: 77.88282713434944 - type: nauc_recall_at_100_max value: 61.93305063016877 - type: nauc_recall_at_100_std value: -16.961439349286298 - type: nauc_recall_at_10_diff1 value: 72.68211095993148 - type: nauc_recall_at_10_max value: 34.46396318256104 - type: nauc_recall_at_10_std value: -55.414854482904104 - type: nauc_recall_at_1_diff1 value: 81.69181674577305 - type: nauc_recall_at_1_max value: 30.817373790682357 - type: nauc_recall_at_1_std value: -42.72748236505566 - type: nauc_recall_at_20_diff1 value: 73.32466963570535 - type: nauc_recall_at_20_max value: 59.70241932930911 - type: nauc_recall_at_20_std value: -33.708044953188335 - type: nauc_recall_at_3_diff1 value: 75.2135837428305 - type: nauc_recall_at_3_max value: 30.365995550867186 - type: nauc_recall_at_3_std value: -55.88540648437351 - type: nauc_recall_at_5_diff1 value: 71.83388219833638 - type: nauc_recall_at_5_max value: 30.57364233205002 - type: nauc_recall_at_5_std value: -57.33381514050736 - type: ndcg_at_1 value: 69.968 - type: ndcg_at_10 value: 82.311 - type: ndcg_at_100 value: 83.325 - type: ndcg_at_1000 value: 83.444 - type: ndcg_at_20 value: 82.99499999999999 - type: ndcg_at_3 value: 78.506 - type: ndcg_at_5 value: 80.735 - type: precision_at_1 value: 69.968 - type: precision_at_10 value: 9.557 - type: precision_at_100 value: 0.9990000000000001 - type: precision_at_1000 value: 0.101 - type: precision_at_20 value: 4.91 - type: precision_at_3 value: 28.310999999999996 - type: precision_at_5 value: 18.102999999999998 - type: recall_at_1 value: 69.679 - type: recall_at_10 value: 94.521 - type: recall_at_100 value: 98.84100000000001 - type: recall_at_1000 value: 99.789 - type: recall_at_20 value: 97.155 - type: recall_at_3 value: 84.48400000000001 - type: recall_at_5 value: 89.831 task: type: Retrieval - dataset: config: default name: MTEB DuRetrieval revision: a1a333e290fe30b10f3f56498e3a0d911a693ced split: dev type: C-MTEB/DuRetrieval metrics: - type: main_score value: 82.19800000000001 - type: map_at_1 value: 23.543 - type: map_at_10 value: 73.175 - type: map_at_100 value: 76.508 - type: map_at_1000 value: 76.556 - type: map_at_20 value: 75.74 - type: map_at_3 value: 49.364999999999995 - type: map_at_5 value: 62.914 - type: mrr_at_1 value: 83.35000000000001 - type: mrr_at_10 value: 88.91954365079361 - type: mrr_at_100 value: 89.00500192836907 - type: mrr_at_1000 value: 89.00778867189919 - type: mrr_at_20 value: 88.97223997708514 - type: mrr_at_3 value: 88.41666666666661 - type: mrr_at_5 value: 88.75916666666663 - type: nauc_map_at_1000_diff1 value: 1.7634218677860924 - type: nauc_map_at_1000_max value: 36.542761970352075 - type: nauc_map_at_1000_std value: 16.743999797731096 - type: nauc_map_at_100_diff1 value: 1.7302879721981035 - type: nauc_map_at_100_max value: 36.51751309286087 - type: nauc_map_at_100_std value: 16.714947992819315 - type: nauc_map_at_10_diff1 value: 4.58612138353187 - type: nauc_map_at_10_max value: 32.412088546560376 - type: nauc_map_at_10_std value: 6.602612093174039 - type: nauc_map_at_1_diff1 value: 40.92916047850798 - type: nauc_map_at_1_max value: -12.132990973344784 - type: nauc_map_at_1_std value: -28.916379676602716 - type: nauc_map_at_20_diff1 value: 2.3048278132160465 - type: nauc_map_at_20_max value: 35.98653993769816 - type: nauc_map_at_20_std value: 14.547155799726797 - type: nauc_map_at_3_diff1 value: 25.02876377684228 - type: nauc_map_at_3_max value: 2.2091937735006324 - type: nauc_map_at_3_std value: -22.118418323376787 - type: nauc_map_at_5_diff1 value: 15.251921354667381 - type: nauc_map_at_5_max value: 15.154235785462586 - type: nauc_map_at_5_std value: -12.229180678598413 - type: nauc_mrr_at_1000_diff1 value: 20.11404624924483 - type: nauc_mrr_at_1000_max value: 50.30723314651651 - type: nauc_mrr_at_1000_std value: 24.38243778856984 - type: nauc_mrr_at_100_diff1 value: 20.096139168665133 - type: nauc_mrr_at_100_max value: 50.30903859418512 - type: nauc_mrr_at_100_std value: 24.390286154308622 - type: nauc_mrr_at_10_diff1 value: 20.181186231780917 - type: nauc_mrr_at_10_max value: 50.58384362961475 - type: nauc_mrr_at_10_std value: 24.64009275793392 - type: nauc_mrr_at_1_diff1 value: 20.447721013067195 - type: nauc_mrr_at_1_max value: 42.25955038562464 - type: nauc_mrr_at_1_std value: 16.11340056741704 - type: nauc_mrr_at_20_diff1 value: 20.131733240741127 - type: nauc_mrr_at_20_max value: 50.43059807829098 - type: nauc_mrr_at_20_std value: 24.515830791430787 - type: nauc_mrr_at_3_diff1 value: 20.142584954251564 - type: nauc_mrr_at_3_max value: 50.982426386940496 - type: nauc_mrr_at_3_std value: 24.855042119442096 - type: nauc_mrr_at_5_diff1 value: 20.55194083518438 - type: nauc_mrr_at_5_max value: 51.01767921199396 - type: nauc_mrr_at_5_std value: 25.06454257772412 - type: nauc_ndcg_at_1000_diff1 value: 3.308799593775925 - type: nauc_ndcg_at_1000_max value: 45.3000272962289 - type: nauc_ndcg_at_1000_std value: 26.858211491434762 - type: nauc_ndcg_at_100_diff1 value: 2.4769775243940577 - type: nauc_ndcg_at_100_max value: 45.06873331760543 - type: nauc_ndcg_at_100_std value: 27.149960146479792 - type: nauc_ndcg_at_10_diff1 value: 3.053577467938038 - type: nauc_ndcg_at_10_max value: 40.86513789655222 - type: nauc_ndcg_at_10_std value: 18.89357502028107 - type: nauc_ndcg_at_1_diff1 value: 20.447721013067195 - type: nauc_ndcg_at_1_max value: 42.25955038562464 - type: nauc_ndcg_at_1_std value: 16.11340056741704 - type: nauc_ndcg_at_20_diff1 value: 2.9037109510330734 - type: nauc_ndcg_at_20_max value: 44.2247958844392 - type: nauc_ndcg_at_20_std value: 23.926393325465583 - type: nauc_ndcg_at_3_diff1 value: 4.6204987018531565 - type: nauc_ndcg_at_3_max value: 38.4238367882706 - type: nauc_ndcg_at_3_std value: 18.41509221702182 - type: nauc_ndcg_at_5_diff1 value: 5.061035908501401 - type: nauc_ndcg_at_5_max value: 35.23535851703071 - type: nauc_ndcg_at_5_std value: 13.77450410228784 - type: nauc_precision_at_1000_diff1 value: -27.983106250756506 - type: nauc_precision_at_1000_max value: 25.86365534534841 - type: nauc_precision_at_1000_std value: 48.74540499812263 - type: nauc_precision_at_100_diff1 value: -29.106950173691263 - type: nauc_precision_at_100_max value: 27.77159360788603 - type: nauc_precision_at_100_std value: 50.73074846170649 - type: nauc_precision_at_10_diff1 value: -31.36561529132309 - type: nauc_precision_at_10_max value: 38.634418900568605 - type: nauc_precision_at_10_std value: 47.64589458589123 - type: nauc_precision_at_1_diff1 value: 20.447721013067195 - type: nauc_precision_at_1_max value: 42.25955038562464 - type: nauc_precision_at_1_std value: 16.11340056741704 - type: nauc_precision_at_20_diff1 value: -29.915732218413915 - type: nauc_precision_at_20_max value: 32.5978721872368 - type: nauc_precision_at_20_std value: 50.40163344959172 - type: nauc_precision_at_3_diff1 value: -21.275233009079773 - type: nauc_precision_at_3_max value: 43.618485232731345 - type: nauc_precision_at_3_std value: 33.897688883129895 - type: nauc_precision_at_5_diff1 value: -27.95940970772729 - type: nauc_precision_at_5_max value: 42.061006167588815 - type: nauc_precision_at_5_std value: 39.63220834267178 - type: nauc_recall_at_1000_diff1 value: -26.449058870321217 - type: nauc_recall_at_1000_max value: 73.0687821403061 - type: nauc_recall_at_1000_std value: 77.88067966296173 - type: nauc_recall_at_100_diff1 value: -24.90707443234635 - type: nauc_recall_at_100_max value: 50.66639373783482 - type: nauc_recall_at_100_std value: 54.350058605469755 - type: nauc_recall_at_10_diff1 value: 0.8584890040490949 - type: nauc_recall_at_10_max value: 33.75876106382897 - type: nauc_recall_at_10_std value: 6.861342565912654 - type: nauc_recall_at_1_diff1 value: 40.92916047850798 - type: nauc_recall_at_1_max value: -12.132990973344784 - type: nauc_recall_at_1_std value: -28.916379676602716 - type: nauc_recall_at_20_diff1 value: -5.517998459738173 - type: nauc_recall_at_20_max value: 45.13561303546834 - type: nauc_recall_at_20_std value: 28.078298519083845 - type: nauc_recall_at_3_diff1 value: 23.80910203367763 - type: nauc_recall_at_3_max value: -1.5249858361677042 - type: nauc_recall_at_3_std value: -24.572258302000353 - type: nauc_recall_at_5_diff1 value: 14.615940357425682 - type: nauc_recall_at_5_max value: 9.484229775296 - type: nauc_recall_at_5_std value: -17.225320352424525 - type: ndcg_at_1 value: 83.35000000000001 - type: ndcg_at_10 value: 82.19800000000001 - type: ndcg_at_100 value: 85.938 - type: ndcg_at_1000 value: 86.367 - type: ndcg_at_20 value: 84.314 - type: ndcg_at_3 value: 79.096 - type: ndcg_at_5 value: 78.881 - type: precision_at_1 value: 83.35000000000001 - type: precision_at_10 value: 40.215 - type: precision_at_100 value: 4.765 - type: precision_at_1000 value: 0.488 - type: precision_at_20 value: 22.29 - type: precision_at_3 value: 71.11699999999999 - type: precision_at_5 value: 61.029999999999994 - type: recall_at_1 value: 23.543 - type: recall_at_10 value: 85.03 - type: recall_at_100 value: 96.769 - type: recall_at_1000 value: 98.904 - type: recall_at_20 value: 91.282 - type: recall_at_3 value: 52.686 - type: recall_at_5 value: 69.625 task: type: Retrieval - dataset: config: default name: MTEB EcomRetrieval revision: 687de13dc7294d6fd9be10c6945f9e8fec8166b9 split: dev type: C-MTEB/EcomRetrieval metrics: - type: main_score value: 62.629000000000005 - type: map_at_1 value: 46.800000000000004 - type: map_at_10 value: 57.311 - type: map_at_100 value: 57.933 - type: map_at_1000 value: 57.949 - type: map_at_20 value: 57.720000000000006 - type: map_at_3 value: 54.617000000000004 - type: map_at_5 value: 56.152 - type: mrr_at_1 value: 46.800000000000004 - type: mrr_at_10 value: 57.31142857142856 - type: mrr_at_100 value: 57.93340200683009 - type: mrr_at_1000 value: 57.94943793755751 - type: mrr_at_20 value: 57.71972088540971 - type: mrr_at_3 value: 54.61666666666664 - type: mrr_at_5 value: 56.151666666666635 - type: nauc_map_at_1000_diff1 value: 62.9251523816579 - type: nauc_map_at_1000_max value: 23.538214459724426 - type: nauc_map_at_1000_std value: -10.382006646813648 - type: nauc_map_at_100_diff1 value: 62.91524869388112 - type: nauc_map_at_100_max value: 23.546842777928948 - type: nauc_map_at_100_std value: -10.377308542639026 - type: nauc_map_at_10_diff1 value: 62.95644445268712 - type: nauc_map_at_10_max value: 23.50898464915184 - type: nauc_map_at_10_std value: -10.680666512130612 - type: nauc_map_at_1_diff1 value: 66.44819716543068 - type: nauc_map_at_1_max value: 19.717201903767986 - type: nauc_map_at_1_std value: -15.336875227297902 - type: nauc_map_at_20_diff1 value: 62.89964860093505 - type: nauc_map_at_20_max value: 23.591388181751878 - type: nauc_map_at_20_std value: -10.439865619968138 - type: nauc_map_at_3_diff1 value: 62.86404655953095 - type: nauc_map_at_3_max value: 22.376787579749553 - type: nauc_map_at_3_std value: -11.663211021980615 - type: nauc_map_at_5_diff1 value: 62.790883650254905 - type: nauc_map_at_5_max value: 22.718439787051057 - type: nauc_map_at_5_std value: -10.782545400900611 - type: nauc_mrr_at_1000_diff1 value: 62.9251523816579 - type: nauc_mrr_at_1000_max value: 23.538214459724426 - type: nauc_mrr_at_1000_std value: -10.382006646813648 - type: nauc_mrr_at_100_diff1 value: 62.91524869388112 - type: nauc_mrr_at_100_max value: 23.546842777928948 - type: nauc_mrr_at_100_std value: -10.377308542639026 - type: nauc_mrr_at_10_diff1 value: 62.95644445268712 - type: nauc_mrr_at_10_max value: 23.50898464915184 - type: nauc_mrr_at_10_std value: -10.680666512130612 - type: nauc_mrr_at_1_diff1 value: 66.44819716543068 - type: nauc_mrr_at_1_max value: 19.717201903767986 - type: nauc_mrr_at_1_std value: -15.336875227297902 - type: nauc_mrr_at_20_diff1 value: 62.89964860093505 - type: nauc_mrr_at_20_max value: 23.591388181751878 - type: nauc_mrr_at_20_std value: -10.439865619968138 - type: nauc_mrr_at_3_diff1 value: 62.86404655953095 - type: nauc_mrr_at_3_max value: 22.376787579749553 - type: nauc_mrr_at_3_std value: -11.663211021980615 - type: nauc_mrr_at_5_diff1 value: 62.790883650254905 - type: nauc_mrr_at_5_max value: 22.718439787051057 - type: nauc_mrr_at_5_std value: -10.782545400900611 - type: nauc_ndcg_at_1000_diff1 value: 62.229954931841206 - type: nauc_ndcg_at_1000_max value: 26.356327484117426 - type: nauc_ndcg_at_1000_std value: -6.826775710969278 - type: nauc_ndcg_at_100_diff1 value: 61.95919238401926 - type: nauc_ndcg_at_100_max value: 26.6734456691839 - type: nauc_ndcg_at_100_std value: -6.57007966783538 - type: nauc_ndcg_at_10_diff1 value: 62.01610935227022 - type: nauc_ndcg_at_10_max value: 26.303092998272376 - type: nauc_ndcg_at_10_std value: -8.283374897295717 - type: nauc_ndcg_at_1_diff1 value: 66.44819716543068 - type: nauc_ndcg_at_1_max value: 19.717201903767986 - type: nauc_ndcg_at_1_std value: -15.336875227297902 - type: nauc_ndcg_at_20_diff1 value: 61.80824854740686 - type: nauc_ndcg_at_20_max value: 26.831041894412593 - type: nauc_ndcg_at_20_std value: -7.188552854116385 - type: nauc_ndcg_at_3_diff1 value: 61.747724042994015 - type: nauc_ndcg_at_3_max value: 23.426174377134767 - type: nauc_ndcg_at_3_std value: -10.41327754115556 - type: nauc_ndcg_at_5_diff1 value: 61.548403438811604 - type: nauc_ndcg_at_5_max value: 24.159239715161462 - type: nauc_ndcg_at_5_std value: -8.651104316213873 - type: nauc_precision_at_1000_diff1 value: 58.1547934489109 - type: nauc_precision_at_1000_max value: 92.86471345294885 - type: nauc_precision_at_1000_std value: 85.92878592878589 - type: nauc_precision_at_100_diff1 value: 52.20107650903497 - type: nauc_precision_at_100_max value: 72.61218212775312 - type: nauc_precision_at_100_std value: 52.523754599879084 - type: nauc_precision_at_10_diff1 value: 58.12459419745063 - type: nauc_precision_at_10_max value: 40.89977104192992 - type: nauc_precision_at_10_std value: 3.492953070918366 - type: nauc_precision_at_1_diff1 value: 66.44819716543068 - type: nauc_precision_at_1_max value: 19.717201903767986 - type: nauc_precision_at_1_std value: -15.336875227297902 - type: nauc_precision_at_20_diff1 value: 55.51306648157717 - type: nauc_precision_at_20_max value: 50.04125223449585 - type: nauc_precision_at_20_std value: 15.62706670420837 - type: nauc_precision_at_3_diff1 value: 58.167034551105914 - type: nauc_precision_at_3_max value: 26.878462993828578 - type: nauc_precision_at_3_std value: -6.362738347287171 - type: nauc_precision_at_5_diff1 value: 57.03963827066697 - type: nauc_precision_at_5_max value: 29.61380946203878 - type: nauc_precision_at_5_std value: -0.5583455330504148 - type: nauc_recall_at_1000_diff1 value: 58.15479344891117 - type: nauc_recall_at_1000_max value: 92.86471345294875 - type: nauc_recall_at_1000_std value: 85.92878592878608 - type: nauc_recall_at_100_diff1 value: 52.201076509035005 - type: nauc_recall_at_100_max value: 72.61218212775306 - type: nauc_recall_at_100_std value: 52.52375459987927 - type: nauc_recall_at_10_diff1 value: 58.12459419745076 - type: nauc_recall_at_10_max value: 40.89977104193005 - type: nauc_recall_at_10_std value: 3.4929530709184156 - type: nauc_recall_at_1_diff1 value: 66.44819716543068 - type: nauc_recall_at_1_max value: 19.717201903767986 - type: nauc_recall_at_1_std value: -15.336875227297902 - type: nauc_recall_at_20_diff1 value: 55.513066481577276 - type: nauc_recall_at_20_max value: 50.041252234495936 - type: nauc_recall_at_20_std value: 15.627066704208364 - type: nauc_recall_at_3_diff1 value: 58.167034551105935 - type: nauc_recall_at_3_max value: 26.878462993828546 - type: nauc_recall_at_3_std value: -6.362738347287183 - type: nauc_recall_at_5_diff1 value: 57.03963827066695 - type: nauc_recall_at_5_max value: 29.613809462038848 - type: nauc_recall_at_5_std value: -0.558345533050415 - type: ndcg_at_1 value: 46.800000000000004 - type: ndcg_at_10 value: 62.629000000000005 - type: ndcg_at_100 value: 65.603 - type: ndcg_at_1000 value: 66.00200000000001 - type: ndcg_at_20 value: 64.086 - type: ndcg_at_3 value: 57.132000000000005 - type: ndcg_at_5 value: 59.874 - type: precision_at_1 value: 46.800000000000004 - type: precision_at_10 value: 7.9399999999999995 - type: precision_at_100 value: 0.932 - type: precision_at_1000 value: 0.096 - type: precision_at_20 value: 4.255 - type: precision_at_3 value: 21.467 - type: precision_at_5 value: 14.2 - type: recall_at_1 value: 46.800000000000004 - type: recall_at_10 value: 79.4 - type: recall_at_100 value: 93.2 - type: recall_at_1000 value: 96.3 - type: recall_at_20 value: 85.1 - type: recall_at_3 value: 64.4 - type: recall_at_5 value: 71.0 task: type: Retrieval - dataset: config: default name: MTEB IFlyTek revision: 421605374b29664c5fc098418fe20ada9bd55f8a split: validation type: C-MTEB/IFlyTek-classification metrics: - type: accuracy value: 48.54944209311274 - type: f1 value: 35.4035877894852 - type: f1_weighted value: 45.154890336987165 - type: main_score value: 48.54944209311274 task: type: Classification - dataset: config: default name: MTEB JDReview revision: b7c64bd89eb87f8ded463478346f76731f07bf8b split: test type: C-MTEB/JDReview-classification metrics: - type: accuracy value: 83.11444652908067 - type: ap value: 50.14519497969601 - type: ap_weighted value: 50.14519497969601 - type: f1 value: 77.87333673269332 - type: f1_weighted value: 84.63457388911604 - type: main_score value: 83.11444652908067 task: type: Classification - dataset: config: default name: MTEB LCQMC revision: 17f9b096f80380fce5ed12a9be8be7784b337daf split: test type: C-MTEB/LCQMC metrics: - type: cosine_pearson value: 66.68236791193614 - type: cosine_spearman value: 70.81518072368738 - type: euclidean_pearson value: 69.14238176341911 - type: euclidean_spearman value: 70.81518071944527 - type: main_score value: 70.81518072368738 - type: manhattan_pearson value: 69.10723297223483 - type: manhattan_spearman value: 70.77165164968557 - type: pearson value: 66.68236791193614 - type: spearman value: 70.81518072368738 task: type: STS - dataset: config: default name: MTEB MMarcoReranking revision: 8e0c766dbe9e16e1d221116a3f36795fbade07f6 split: dev type: C-MTEB/Mmarco-reranking metrics: - type: main_score value: 25.488749669858247 - type: map value: 25.488749669858247 - type: mrr value: 23.813888888888886 - type: nAUC_map_diff1 value: 21.439042551978073 - type: nAUC_map_max value: 4.356646643203048 - type: nAUC_map_std value: -6.4651117756570935 - type: nAUC_mrr_diff1 value: 21.040572282417614 - type: nAUC_mrr_max value: 3.7091513714293187 - type: nAUC_mrr_std value: -7.380296002229526 task: type: Reranking - dataset: config: default name: MTEB MMarcoRetrieval revision: 539bbde593d947e2a124ba72651aafc09eb33fc2 split: dev type: C-MTEB/MMarcoRetrieval metrics: - type: main_score value: 78.975 - type: map_at_1 value: 65.864 - type: map_at_10 value: 75.142 - type: map_at_100 value: 75.472 - type: map_at_1000 value: 75.482 - type: map_at_20 value: 75.369 - type: map_at_3 value: 73.219 - type: map_at_5 value: 74.42 - type: mrr_at_1 value: 68.0515759312321 - type: mrr_at_10 value: 75.65253331514067 - type: mrr_at_100 value: 75.94116743915134 - type: mrr_at_1000 value: 75.9508744592604 - type: mrr_at_20 value: 75.85185969774719 - type: mrr_at_3 value: 73.96131805157593 - type: mrr_at_5 value: 75.01361031518599 - type: nauc_map_at_1000_diff1 value: 77.47587043918091 - type: nauc_map_at_1000_max value: 32.511975057438974 - type: nauc_map_at_1000_std value: -17.666094558943417 - type: nauc_map_at_100_diff1 value: 77.47241550099297 - type: nauc_map_at_100_max value: 32.52837754222784 - type: nauc_map_at_100_std value: -17.63717709810784 - type: nauc_map_at_10_diff1 value: 77.31160893655145 - type: nauc_map_at_10_max value: 32.60670257703009 - type: nauc_map_at_10_std value: -17.771047114996144 - type: nauc_map_at_1_diff1 value: 79.92534779416832 - type: nauc_map_at_1_max value: 24.785321951074515 - type: nauc_map_at_1_std value: -23.45816811226535 - type: nauc_map_at_20_diff1 value: 77.42100083928817 - type: nauc_map_at_20_max value: 32.593057471316435 - type: nauc_map_at_20_std value: -17.608670597024197 - type: nauc_map_at_3_diff1 value: 77.39438847887251 - type: nauc_map_at_3_max value: 31.397054344602548 - type: nauc_map_at_3_std value: -19.4976864523292 - type: nauc_map_at_5_diff1 value: 77.34834480360892 - type: nauc_map_at_5_max value: 32.21397984602674 - type: nauc_map_at_5_std value: -18.45142721452299 - type: nauc_mrr_at_1000_diff1 value: 77.91270423495435 - type: nauc_mrr_at_1000_max value: 33.286202404635816 - type: nauc_mrr_at_1000_std value: -16.988509017415858 - type: nauc_mrr_at_100_diff1 value: 77.90931964992795 - type: nauc_mrr_at_100_max value: 33.302772470570595 - type: nauc_mrr_at_100_std value: -16.95907503705695 - type: nauc_mrr_at_10_diff1 value: 77.74963685519619 - type: nauc_mrr_at_10_max value: 33.402887556509064 - type: nauc_mrr_at_10_std value: -17.007385451413267 - type: nauc_mrr_at_1_diff1 value: 80.44952589549635 - type: nauc_mrr_at_1_max value: 28.725876665931615 - type: nauc_mrr_at_1_std value: -22.18861108564602 - type: nauc_mrr_at_20_diff1 value: 77.86689879110665 - type: nauc_mrr_at_20_max value: 33.37387640439328 - type: nauc_mrr_at_20_std value: -16.908485017918682 - type: nauc_mrr_at_3_diff1 value: 77.85662171057854 - type: nauc_mrr_at_3_max value: 32.7468353039024 - type: nauc_mrr_at_3_std value: -18.339103980341957 - type: nauc_mrr_at_5_diff1 value: 77.81340059404181 - type: nauc_mrr_at_5_max value: 33.15760829968908 - type: nauc_mrr_at_5_std value: -17.53876847769669 - type: nauc_ndcg_at_1000_diff1 value: 77.01438896352477 - type: nauc_ndcg_at_1000_max value: 34.55832672068587 - type: nauc_ndcg_at_1000_std value: -14.559071957898562 - type: nauc_ndcg_at_100_diff1 value: 76.92298480629259 - type: nauc_ndcg_at_100_max value: 35.11598028021352 - type: nauc_ndcg_at_100_std value: -13.554650282912817 - type: nauc_ndcg_at_10_diff1 value: 76.13277495290987 - type: nauc_ndcg_at_10_max value: 35.6330609610323 - type: nauc_ndcg_at_10_std value: -13.944707424342036 - type: nauc_ndcg_at_1_diff1 value: 80.44952589549635 - type: nauc_ndcg_at_1_max value: 28.725876665931615 - type: nauc_ndcg_at_1_std value: -22.18861108564602 - type: nauc_ndcg_at_20_diff1 value: 76.56409694019162 - type: nauc_ndcg_at_20_max value: 35.62960378910709 - type: nauc_ndcg_at_20_std value: -13.255084552264481 - type: nauc_ndcg_at_3_diff1 value: 76.47868292868812 - type: nauc_ndcg_at_3_max value: 33.233395497849614 - type: nauc_ndcg_at_3_std value: -17.748967332305217 - type: nauc_ndcg_at_5_diff1 value: 76.32952180796222 - type: nauc_ndcg_at_5_max value: 34.60345012714885 - type: nauc_ndcg_at_5_std value: -15.8173707360486 - type: nauc_precision_at_1000_diff1 value: -17.52918228066939 - type: nauc_precision_at_1000_max value: 20.626086186594833 - type: nauc_precision_at_1000_std value: 26.231038468619218 - type: nauc_precision_at_100_diff1 value: -4.307304124093403 - type: nauc_precision_at_100_max value: 28.87742760011375 - type: nauc_precision_at_100_std value: 30.66463485009584 - type: nauc_precision_at_10_diff1 value: 21.252248820566297 - type: nauc_precision_at_10_max value: 37.7120251998008 - type: nauc_precision_at_10_std value: 16.537333131912142 - type: nauc_precision_at_1_diff1 value: 80.44952589549635 - type: nauc_precision_at_1_max value: 28.725876665931615 - type: nauc_precision_at_1_std value: -22.18861108564602 - type: nauc_precision_at_20_diff1 value: 10.933686624253406 - type: nauc_precision_at_20_max value: 35.427519568658674 - type: nauc_precision_at_20_std value: 24.992563220729206 - type: nauc_precision_at_3_diff1 value: 46.36733939400364 - type: nauc_precision_at_3_max value: 35.27060526473028 - type: nauc_precision_at_3_std value: -3.8471891263107327 - type: nauc_precision_at_5_diff1 value: 35.16733507102863 - type: nauc_precision_at_5_max value: 36.69354322946151 - type: nauc_precision_at_5_std value: 5.0611235120787725 - type: nauc_recall_at_1000_diff1 value: 64.08540543698155 - type: nauc_recall_at_1000_max value: 77.7366984466171 - type: nauc_recall_at_1000_std value: 73.92508298307646 - type: nauc_recall_at_100_diff1 value: 68.80219044771133 - type: nauc_recall_at_100_max value: 73.37989662095796 - type: nauc_recall_at_100_std value: 59.45952818309578 - type: nauc_recall_at_10_diff1 value: 66.89046359223634 - type: nauc_recall_at_10_max value: 53.497471386132865 - type: nauc_recall_at_10_std value: 10.389938512313515 - type: nauc_recall_at_1_diff1 value: 79.92534779416832 - type: nauc_recall_at_1_max value: 24.785321951074515 - type: nauc_recall_at_1_std value: -23.45816811226535 - type: nauc_recall_at_20_diff1 value: 67.58868553352538 - type: nauc_recall_at_20_max value: 61.29680579230528 - type: nauc_recall_at_20_std value: 27.640944871424384 - type: nauc_recall_at_3_diff1 value: 72.10723775978491 - type: nauc_recall_at_3_max value: 37.98766181226662 - type: nauc_recall_at_3_std value: -12.700827115093238 - type: nauc_recall_at_5_diff1 value: 70.56165414287506 - type: nauc_recall_at_5_max value: 43.87249758193006 - type: nauc_recall_at_5_std value: -4.957329954267046 - type: ndcg_at_1 value: 68.052 - type: ndcg_at_10 value: 78.975 - type: ndcg_at_100 value: 80.415 - type: ndcg_at_1000 value: 80.683 - type: ndcg_at_20 value: 79.752 - type: ndcg_at_3 value: 75.315 - type: ndcg_at_5 value: 77.336 - type: precision_at_1 value: 68.052 - type: precision_at_10 value: 9.602 - type: precision_at_100 value: 1.031 - type: precision_at_1000 value: 0.105 - type: precision_at_20 value: 4.968 - type: precision_at_3 value: 28.405 - type: precision_at_5 value: 18.129 - type: recall_at_1 value: 65.864 - type: recall_at_10 value: 90.29599999999999 - type: recall_at_100 value: 96.712 - type: recall_at_1000 value: 98.817 - type: recall_at_20 value: 93.28999999999999 - type: recall_at_3 value: 80.599 - type: recall_at_5 value: 85.384 task: type: Retrieval - dataset: config: zh-CN name: MTEB MassiveIntentClassification (zh-CN) revision: 4672e20407010da34463acc759c162ca9734bca6 split: test type: mteb/amazon_massive_intent metrics: - type: accuracy value: 68.52723604572965 - type: f1 value: 65.11440259918176 - type: f1_weighted value: 67.36553063592758 - type: main_score value: 68.52723604572965 task: type: Classification - dataset: config: zh-TW name: MTEB MassiveIntentClassification (zh-TW) revision: 4672e20407010da34463acc759c162ca9734bca6 split: test type: mteb/amazon_massive_intent metrics: - type: accuracy value: 60.00672494956289 - type: f1 value: 57.630108926504654 - type: f1_weighted value: 58.013112273357294 - type: main_score value: 60.00672494956289 task: type: Classification - dataset: config: zh-CN name: MTEB MassiveScenarioClassification (zh-CN) revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 split: test type: mteb/amazon_massive_scenario metrics: - type: accuracy value: 75.48755884330868 - type: f1 value: 74.96341150854326 - type: f1_weighted value: 75.11344709750414 - type: main_score value: 75.48755884330868 task: type: Classification - dataset: config: zh-TW name: MTEB MassiveScenarioClassification (zh-TW) revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 split: test type: mteb/amazon_massive_scenario metrics: - type: accuracy value: 69.20645595158035 - type: f1 value: 69.57805291990489 - type: f1_weighted value: 68.41325928547694 - type: main_score value: 69.20645595158035 task: type: Classification - dataset: config: default name: MTEB MedicalRetrieval revision: 2039188fb5800a9803ba5048df7b76e6fb151fc6 split: dev type: C-MTEB/MedicalRetrieval metrics: - type: main_score value: 56.842999999999996 - type: map_at_1 value: 48.4 - type: map_at_10 value: 54.127 - type: map_at_100 value: 54.74099999999999 - type: map_at_1000 value: 54.791999999999994 - type: map_at_20 value: 54.510999999999996 - type: map_at_3 value: 52.900000000000006 - type: map_at_5 value: 53.555 - type: mrr_at_1 value: 48.4 - type: mrr_at_10 value: 54.12742063492064 - type: mrr_at_100 value: 54.74007856972957 - type: mrr_at_1000 value: 54.79120377168234 - type: mrr_at_20 value: 54.50988389233748 - type: mrr_at_3 value: 52.89999999999999 - type: mrr_at_5 value: 53.554999999999986 - type: nauc_map_at_1000_diff1 value: 78.4025670906739 - type: nauc_map_at_1000_max value: 54.375261014986506 - type: nauc_map_at_1000_std value: 20.01404778643528 - type: nauc_map_at_100_diff1 value: 78.38648222989812 - type: nauc_map_at_100_max value: 54.36062306870368 - type: nauc_map_at_100_std value: 20.027286514844604 - type: nauc_map_at_10_diff1 value: 78.66966790097088 - type: nauc_map_at_10_max value: 54.49489942528706 - type: nauc_map_at_10_std value: 19.706732968916686 - type: nauc_map_at_1_diff1 value: 81.3863109971009 - type: nauc_map_at_1_max value: 54.45112546839308 - type: nauc_map_at_1_std value: 16.647233102073926 - type: nauc_map_at_20_diff1 value: 78.35531215313732 - type: nauc_map_at_20_max value: 54.38122778164954 - type: nauc_map_at_20_std value: 19.919068985031778 - type: nauc_map_at_3_diff1 value: 78.76127354528398 - type: nauc_map_at_3_max value: 54.73984834363167 - type: nauc_map_at_3_std value: 18.978057003463505 - type: nauc_map_at_5_diff1 value: 78.86532352860041 - type: nauc_map_at_5_max value: 54.89472733309794 - type: nauc_map_at_5_std value: 19.905059369622226 - type: nauc_mrr_at_1000_diff1 value: 78.40339084857348 - type: nauc_mrr_at_1000_max value: 54.3765713570027 - type: nauc_mrr_at_1000_std value: 20.01339841032316 - type: nauc_mrr_at_100_diff1 value: 78.38730500946644 - type: nauc_mrr_at_100_max value: 54.361931749382975 - type: nauc_mrr_at_100_std value: 20.026637904594022 - type: nauc_mrr_at_10_diff1 value: 78.66966790097088 - type: nauc_mrr_at_10_max value: 54.49489942528706 - type: nauc_mrr_at_10_std value: 19.706732968916686 - type: nauc_mrr_at_1_diff1 value: 81.3863109971009 - type: nauc_mrr_at_1_max value: 54.45112546839308 - type: nauc_mrr_at_1_std value: 16.647233102073926 - type: nauc_mrr_at_20_diff1 value: 78.3561572811205 - type: nauc_mrr_at_20_max value: 54.38258490064744 - type: nauc_mrr_at_20_std value: 19.918521746919023 - type: nauc_mrr_at_3_diff1 value: 78.76127354528398 - type: nauc_mrr_at_3_max value: 54.73984834363167 - type: nauc_mrr_at_3_std value: 18.978057003463505 - type: nauc_mrr_at_5_diff1 value: 78.86532352860041 - type: nauc_mrr_at_5_max value: 54.89472733309794 - type: nauc_mrr_at_5_std value: 19.905059369622226 - type: nauc_ndcg_at_1000_diff1 value: 76.97351511935506 - type: nauc_ndcg_at_1000_max value: 53.77087107676227 - type: nauc_ndcg_at_1000_std value: 22.05965205980897 - type: nauc_ndcg_at_100_diff1 value: 76.53100615219961 - type: nauc_ndcg_at_100_max value: 53.310394382128834 - type: nauc_ndcg_at_100_std value: 22.50789785109818 - type: nauc_ndcg_at_10_diff1 value: 77.54846078594903 - type: nauc_ndcg_at_10_max value: 53.933288762789324 - type: nauc_ndcg_at_10_std value: 20.831152333725047 - type: nauc_ndcg_at_1_diff1 value: 81.3863109971009 - type: nauc_ndcg_at_1_max value: 54.45112546839308 - type: nauc_ndcg_at_1_std value: 16.647233102073926 - type: nauc_ndcg_at_20_diff1 value: 76.34535899505026 - type: nauc_ndcg_at_20_max value: 53.51900448424713 - type: nauc_ndcg_at_20_std value: 21.786462318073514 - type: nauc_ndcg_at_3_diff1 value: 77.9282109219262 - type: nauc_ndcg_at_3_max value: 54.68499432962286 - type: nauc_ndcg_at_3_std value: 19.565421430997855 - type: nauc_ndcg_at_5_diff1 value: 78.08531364224505 - type: nauc_ndcg_at_5_max value: 54.990763159275524 - type: nauc_ndcg_at_5_std value: 21.24642472562554 - type: nauc_precision_at_1000_diff1 value: 60.947712418300526 - type: nauc_precision_at_1000_max value: 48.95614495798286 - type: nauc_precision_at_1000_std value: 52.283934407096034 - type: nauc_precision_at_100_diff1 value: 65.20401832958854 - type: nauc_precision_at_100_max value: 45.622481375866016 - type: nauc_precision_at_100_std value: 39.317645096209475 - type: nauc_precision_at_10_diff1 value: 73.52050673935791 - type: nauc_precision_at_10_max value: 51.501061174147324 - type: nauc_precision_at_10_std value: 24.660571885937692 - type: nauc_precision_at_1_diff1 value: 81.3863109971009 - type: nauc_precision_at_1_max value: 54.45112546839308 - type: nauc_precision_at_1_std value: 16.647233102073926 - type: nauc_precision_at_20_diff1 value: 67.23012065064371 - type: nauc_precision_at_20_max value: 49.276980564706854 - type: nauc_precision_at_20_std value: 30.08840589323484 - type: nauc_precision_at_3_diff1 value: 75.41179105476107 - type: nauc_precision_at_3_max value: 54.46883825911932 - type: nauc_precision_at_3_std value: 21.287190629517518 - type: nauc_precision_at_5_diff1 value: 75.66706449322432 - type: nauc_precision_at_5_max value: 55.30373302312931 - type: nauc_precision_at_5_std value: 25.617663499161512 - type: nauc_recall_at_1000_diff1 value: 60.94771241830054 - type: nauc_recall_at_1000_max value: 48.956144957983234 - type: nauc_recall_at_1000_std value: 52.283934407096 - type: nauc_recall_at_100_diff1 value: 65.20401832958845 - type: nauc_recall_at_100_max value: 45.622481375866066 - type: nauc_recall_at_100_std value: 39.31764509620953 - type: nauc_recall_at_10_diff1 value: 73.52050673935793 - type: nauc_recall_at_10_max value: 51.5010611741474 - type: nauc_recall_at_10_std value: 24.660571885937728 - type: nauc_recall_at_1_diff1 value: 81.3863109971009 - type: nauc_recall_at_1_max value: 54.45112546839308 - type: nauc_recall_at_1_std value: 16.647233102073926 - type: nauc_recall_at_20_diff1 value: 67.2301206506438 - type: nauc_recall_at_20_max value: 49.276980564706946 - type: nauc_recall_at_20_std value: 30.08840589323487 - type: nauc_recall_at_3_diff1 value: 75.411791054761 - type: nauc_recall_at_3_max value: 54.468838259119266 - type: nauc_recall_at_3_std value: 21.287190629517486 - type: nauc_recall_at_5_diff1 value: 75.66706449322434 - type: nauc_recall_at_5_max value: 55.30373302312929 - type: nauc_recall_at_5_std value: 25.61766349916154 - type: ndcg_at_1 value: 48.4 - type: ndcg_at_10 value: 56.842999999999996 - type: ndcg_at_100 value: 60.035000000000004 - type: ndcg_at_1000 value: 61.592999999999996 - type: ndcg_at_20 value: 58.219 - type: ndcg_at_3 value: 54.269 - type: ndcg_at_5 value: 55.435 - type: precision_at_1 value: 48.4 - type: precision_at_10 value: 6.54 - type: precision_at_100 value: 0.8089999999999999 - type: precision_at_1000 value: 0.094 - type: precision_at_20 value: 3.54 - type: precision_at_3 value: 19.400000000000002 - type: precision_at_5 value: 12.2 - type: recall_at_1 value: 48.4 - type: recall_at_10 value: 65.4 - type: recall_at_100 value: 80.9 - type: recall_at_1000 value: 93.60000000000001 - type: recall_at_20 value: 70.8 - type: recall_at_3 value: 58.199999999999996 - type: recall_at_5 value: 61.0 task: type: Retrieval - dataset: config: default name: MTEB MultilingualSentiment revision: 46958b007a63fdbf239b7672c25d0bea67b5ea1a split: test type: C-MTEB/MultilingualSentiment-classification metrics: - type: accuracy value: 78.22 - type: f1 value: 77.72387105131016 - type: f1_weighted value: 77.72387105131016 - type: main_score value: 78.22 task: type: Classification - dataset: config: default name: MTEB Ocnli revision: 66e76a618a34d6d565d5538088562851e6daa7ec split: validation type: C-MTEB/OCNLI metrics: - type: cosine_accuracy value: 66.21548456957228 - type: cosine_accuracy_threshold value: 85.5343759059906 - type: cosine_ap value: 69.21438509764522 - type: cosine_f1 value: 70.44534412955467 - type: cosine_f1_threshold value: 83.85478258132935 - type: cosine_precision value: 61.36363636363637 - type: cosine_recall value: 82.68215417106653 - type: dot_accuracy value: 66.21548456957228 - type: dot_accuracy_threshold value: 85.5343759059906 - type: dot_ap value: 69.21438509764522 - type: dot_f1 value: 70.44534412955467 - type: dot_f1_threshold value: 83.85477662086487 - type: dot_precision value: 61.36363636363637 - type: dot_recall value: 82.68215417106653 - type: euclidean_accuracy value: 66.21548456957228 - type: euclidean_accuracy_threshold value: 53.787779808044434 - type: euclidean_ap value: 69.21438509764522 - type: euclidean_f1 value: 70.44534412955467 - type: euclidean_f1_threshold value: 56.82467818260193 - type: euclidean_precision value: 61.36363636363637 - type: euclidean_recall value: 82.68215417106653 - type: main_score value: 69.21438509764522 - type: manhattan_accuracy value: 65.89063345966431 - type: manhattan_accuracy_threshold value: 1257.1067810058594 - type: manhattan_ap value: 68.93169452602294 - type: manhattan_f1 value: 70.018281535649 - type: manhattan_f1_threshold value: 1321.6852188110352 - type: manhattan_precision value: 61.72441579371475 - type: manhattan_recall value: 80.8870116156283 - type: max_ap value: 69.21438509764522 - type: max_f1 value: 70.44534412955467 - type: max_precision value: 61.72441579371475 - type: max_recall value: 82.68215417106653 - type: similarity_accuracy value: 66.21548456957228 - type: similarity_accuracy_threshold value: 85.5343759059906 - type: similarity_ap value: 69.21438509764522 - type: similarity_f1 value: 70.44534412955467 - type: similarity_f1_threshold value: 83.85478258132935 - type: similarity_precision value: 61.36363636363637 - type: similarity_recall value: 82.68215417106653 task: type: PairClassification - dataset: config: default name: MTEB OnlineShopping revision: e610f2ebd179a8fda30ae534c3878750a96db120 split: test type: C-MTEB/OnlineShopping-classification metrics: - type: accuracy value: 93.09 - type: ap value: 92.02754366798146 - type: ap_weighted value: 92.02754366798146 - type: f1 value: 93.08751542879655 - type: f1_weighted value: 93.092429173112 - type: main_score value: 93.09 task: type: Classification - dataset: config: default name: MTEB PAWSX revision: 9c6a90e430ac22b5779fb019a23e820b11a8b5e1 split: test type: C-MTEB/PAWSX metrics: - type: cosine_pearson value: 14.700393205389778 - type: cosine_spearman value: 16.3328315656736 - type: euclidean_pearson value: 18.37941425496121 - type: euclidean_spearman value: 16.35419723095118 - type: main_score value: 16.3328315656736 - type: manhattan_pearson value: 18.461692452314196 - type: manhattan_spearman value: 16.49877203419426 - type: pearson value: 14.700393205389778 - type: spearman value: 16.3328315656736 task: type: STS - dataset: config: default name: MTEB QBQTC revision: 790b0510dc52b1553e8c49f3d2afb48c0e5c48b7 split: test type: C-MTEB/QBQTC metrics: - type: cosine_pearson value: 31.771341848950048 - type: cosine_spearman value: 35.21248641540282 - type: euclidean_pearson value: 32.86721005963099 - type: euclidean_spearman value: 35.21281584988107 - type: main_score value: 35.21248641540282 - type: manhattan_pearson value: 32.994593737560464 - type: manhattan_spearman value: 35.29332061037625 - type: pearson value: 31.771341848950048 - type: spearman value: 35.21248641540282 task: type: STS - dataset: config: zh name: MTEB STS22 (zh) revision: de9d86b3b84231dc21f76c7b7af1f28e2f57f6e3 split: test type: mteb/sts22-crosslingual-sts metrics: - type: cosine_pearson value: 74.05009462610907 - type: cosine_spearman value: 73.57662463047618 - type: euclidean_pearson value: 71.97192342862905 - type: euclidean_spearman value: 73.57662463047618 - type: main_score value: 73.57662463047618 - type: manhattan_pearson value: 71.66171363912555 - type: manhattan_spearman value: 73.33995365752388 - type: pearson value: 74.05009462610907 - type: spearman value: 73.57662463047618 task: type: STS - dataset: config: zh-en name: MTEB STS22 (zh-en) revision: de9d86b3b84231dc21f76c7b7af1f28e2f57f6e3 split: test type: mteb/sts22-crosslingual-sts metrics: - type: cosine_pearson value: 76.5491270627292 - type: cosine_spearman value: 76.34747339191613 - type: euclidean_pearson value: 77.52703884917828 - type: euclidean_spearman value: 76.34747339191613 - type: main_score value: 76.34747339191613 - type: manhattan_pearson value: 77.94417346436772 - type: manhattan_spearman value: 76.15552728253843 - type: pearson value: 76.5491270627292 - type: spearman value: 76.34747339191613 task: type: STS - dataset: config: default name: MTEB STSB revision: 0cde68302b3541bb8b3c340dc0644b0b745b3dc0 split: test type: C-MTEB/STSB metrics: - type: cosine_pearson value: 76.44164798054604 - type: cosine_spearman value: 77.80818016295537 - type: euclidean_pearson value: 77.23235480541429 - type: euclidean_spearman value: 77.80820115269609 - type: main_score value: 77.80818016295537 - type: manhattan_pearson value: 77.1383518573764 - type: manhattan_spearman value: 77.74335331319868 - type: pearson value: 76.44164798054604 - type: spearman value: 77.80818016295537 task: type: STS - dataset: config: default name: MTEB T2Reranking revision: 76631901a18387f85eaa53e5450019b87ad58ef9 split: dev type: C-MTEB/T2Reranking metrics: - type: main_score value: 66.73229864475817 - type: map value: 66.73229864475817 - type: mrr value: 76.93718364122901 - type: nAUC_map_diff1 value: -10.073077377159336 - type: nAUC_map_max value: 36.06382407917089 - type: nAUC_map_std value: -2.3465936800383975 - type: nAUC_mrr_diff1 value: -7.614514359731517 - type: nAUC_mrr_max value: 29.970126517689895 - type: nAUC_mrr_std value: -4.000393719743965 task: type: Reranking - dataset: config: default name: MTEB T2Retrieval revision: 8731a845f1bf500a4f111cf1070785c793d10e64 split: dev type: C-MTEB/T2Retrieval metrics: - type: main_score value: 84.066 - type: map_at_1 value: 27.311999999999998 - type: map_at_10 value: 76.45100000000001 - type: map_at_100 value: 80.081 - type: map_at_1000 value: 80.148 - type: map_at_20 value: 79.19500000000001 - type: map_at_3 value: 53.785000000000004 - type: map_at_5 value: 66.08800000000001 - type: mrr_at_1 value: 89.5493599859723 - type: mrr_at_10 value: 92.04882420558401 - type: mrr_at_100 value: 92.13851397485055 - type: mrr_at_1000 value: 92.142380377799 - type: mrr_at_20 value: 92.10835206390658 - type: mrr_at_3 value: 91.55634461394584 - type: mrr_at_5 value: 91.86626921503318 - type: nauc_map_at_1000_diff1 value: 13.881593411464955 - type: nauc_map_at_1000_max value: 45.98958134604669 - type: nauc_map_at_1000_std value: 20.749937417202162 - type: nauc_map_at_100_diff1 value: 13.897201544868256 - type: nauc_map_at_100_max value: 45.8975771537659 - type: nauc_map_at_100_std value: 20.702471991537816 - type: nauc_map_at_10_diff1 value: 18.574187481021404 - type: nauc_map_at_10_max value: 33.03798476409084 - type: nauc_map_at_10_std value: 4.923864341255681 - type: nauc_map_at_1_diff1 value: 53.60647320762889 - type: nauc_map_at_1_max value: -25.482104681284262 - type: nauc_map_at_1_std value: -35.242254755025314 - type: nauc_map_at_20_diff1 value: 14.659565827642442 - type: nauc_map_at_20_max value: 43.28938016609046 - type: nauc_map_at_20_std value: 17.356608121513382 - type: nauc_map_at_3_diff1 value: 39.4070801457046 - type: nauc_map_at_3_max value: -12.578348941863455 - type: nauc_map_at_3_std value: -31.8798104413808 - type: nauc_map_at_5_diff1 value: 31.32071840697937 - type: nauc_map_at_5_max value: 4.002877899741811 - type: nauc_map_at_5_std value: -21.303079028844135 - type: nauc_mrr_at_1000_diff1 value: 46.83368353209455 - type: nauc_mrr_at_1000_max value: 75.71133032423971 - type: nauc_mrr_at_1000_std value: 39.48046724592415 - type: nauc_mrr_at_100_diff1 value: 46.83586223804469 - type: nauc_mrr_at_100_max value: 75.72076556031453 - type: nauc_mrr_at_100_std value: 39.499587346326805 - type: nauc_mrr_at_10_diff1 value: 46.844356562367416 - type: nauc_mrr_at_10_max value: 75.82630788248727 - type: nauc_mrr_at_10_std value: 39.57574348115377 - type: nauc_mrr_at_1_diff1 value: 47.31173835991697 - type: nauc_mrr_at_1_max value: 71.62100768371566 - type: nauc_mrr_at_1_std value: 33.215001055441654 - type: nauc_mrr_at_20_diff1 value: 46.81803840032271 - type: nauc_mrr_at_20_max value: 75.77378447923428 - type: nauc_mrr_at_20_std value: 39.56872022078684 - type: nauc_mrr_at_3_diff1 value: 46.860529286384946 - type: nauc_mrr_at_3_max value: 75.76461859710832 - type: nauc_mrr_at_3_std value: 39.2216938919496 - type: nauc_mrr_at_5_diff1 value: 46.789493616730475 - type: nauc_mrr_at_5_max value: 75.90313095614088 - type: nauc_mrr_at_5_std value: 39.65230578423716 - type: nauc_ndcg_at_1000_diff1 value: 17.766598062291752 - type: nauc_ndcg_at_1000_max value: 58.1826623156839 - type: nauc_ndcg_at_1000_std value: 32.709580361957975 - type: nauc_ndcg_at_100_diff1 value: 17.52651950834308 - type: nauc_ndcg_at_100_max value: 57.35240006774048 - type: nauc_ndcg_at_100_std value: 32.8548321688731 - type: nauc_ndcg_at_10_diff1 value: 17.54354007672668 - type: nauc_ndcg_at_10_max value: 47.71924227814137 - type: nauc_ndcg_at_10_std value: 21.21142176035583 - type: nauc_ndcg_at_1_diff1 value: 47.31173835991697 - type: nauc_ndcg_at_1_max value: 71.62100768371566 - type: nauc_ndcg_at_1_std value: 33.215001055441654 - type: nauc_ndcg_at_20_diff1 value: 17.70238226651868 - type: nauc_ndcg_at_20_max value: 51.938599251095376 - type: nauc_ndcg_at_20_std value: 26.4850369109262 - type: nauc_ndcg_at_3_diff1 value: 13.27812172653936 - type: nauc_ndcg_at_3_max value: 62.30760700113597 - type: nauc_ndcg_at_3_std value: 31.816092105184445 - type: nauc_ndcg_at_5_diff1 value: 13.889486796899545 - type: nauc_ndcg_at_5_max value: 55.51019399952754 - type: nauc_ndcg_at_5_std value: 27.369160358894128 - type: nauc_precision_at_1000_diff1 value: -35.999300149931784 - type: nauc_precision_at_1000_max value: 49.84051965547009 - type: nauc_precision_at_1000_std value: 58.58089570689277 - type: nauc_precision_at_100_diff1 value: -35.83612932514131 - type: nauc_precision_at_100_max value: 51.586346962862336 - type: nauc_precision_at_100_std value: 60.23951447272133 - type: nauc_precision_at_10_diff1 value: -35.48923076280097 - type: nauc_precision_at_10_max value: 55.72743866396911 - type: nauc_precision_at_10_std value: 55.44739603788007 - type: nauc_precision_at_1_diff1 value: 47.31173835991697 - type: nauc_precision_at_1_max value: 71.62100768371566 - type: nauc_precision_at_1_std value: 33.215001055441654 - type: nauc_precision_at_20_diff1 value: -35.72916661370677 - type: nauc_precision_at_20_max value: 54.082057684079246 - type: nauc_precision_at_20_std value: 59.4645317147574 - type: nauc_precision_at_3_diff1 value: -30.247794446720594 - type: nauc_precision_at_3_max value: 64.43209282496724 - type: nauc_precision_at_3_std value: 47.32128724076265 - type: nauc_precision_at_5_diff1 value: -35.057736789550646 - type: nauc_precision_at_5_max value: 60.10076766835969 - type: nauc_precision_at_5_std value: 50.98874173280675 - type: nauc_recall_at_1000_diff1 value: 1.863242393322992 - type: nauc_recall_at_1000_max value: 60.729765914144615 - type: nauc_recall_at_1000_std value: 62.59143677558537 - type: nauc_recall_at_100_diff1 value: 8.045141361395181 - type: nauc_recall_at_100_max value: 52.778769966654316 - type: nauc_recall_at_100_std value: 49.10300097655128 - type: nauc_recall_at_10_diff1 value: 16.8879769794751 - type: nauc_recall_at_10_max value: 23.43756027359313 - type: nauc_recall_at_10_std value: -0.017371946235138786 - type: nauc_recall_at_1_diff1 value: 53.60647320762889 - type: nauc_recall_at_1_max value: -25.482104681284262 - type: nauc_recall_at_1_std value: -35.242254755025314 - type: nauc_recall_at_20_diff1 value: 11.099660881919029 - type: nauc_recall_at_20_max value: 39.07089132734822 - type: nauc_recall_at_20_std value: 22.93956030600117 - type: nauc_recall_at_3_diff1 value: 38.03040951694198 - type: nauc_recall_at_3_max value: -16.512296557855894 - type: nauc_recall_at_3_std value: -33.99009868262299 - type: nauc_recall_at_5_diff1 value: 30.476589203240035 - type: nauc_recall_at_5_max value: -4.132009312569869 - type: nauc_recall_at_5_std value: -26.413027108583538 - type: ndcg_at_1 value: 89.549 - type: ndcg_at_10 value: 84.066 - type: ndcg_at_100 value: 87.705 - type: ndcg_at_1000 value: 88.369 - type: ndcg_at_20 value: 85.82300000000001 - type: ndcg_at_3 value: 85.466 - type: ndcg_at_5 value: 84.077 - type: precision_at_1 value: 89.549 - type: precision_at_10 value: 41.778 - type: precision_at_100 value: 4.994 - type: precision_at_1000 value: 0.515 - type: precision_at_20 value: 23.199 - type: precision_at_3 value: 74.772 - type: precision_at_5 value: 62.666999999999994 - type: recall_at_1 value: 27.311999999999998 - type: recall_at_10 value: 82.955 - type: recall_at_100 value: 94.814 - type: recall_at_1000 value: 98.205 - type: recall_at_20 value: 88.864 - type: recall_at_3 value: 55.35600000000001 - type: recall_at_5 value: 69.458 task: type: Retrieval - dataset: config: default name: MTEB TNews revision: 317f262bf1e6126357bbe89e875451e4b0938fe4 split: validation type: C-MTEB/TNews-classification metrics: - type: accuracy value: 51.629999999999995 - type: f1 value: 49.06894119635303 - type: f1_weighted value: 50.97940411274921 - type: main_score value: 51.629999999999995 task: type: Classification - dataset: config: default name: MTEB ThuNewsClusteringP2P revision: 5798586b105c0434e4f0fe5e767abe619442cf93 split: test type: C-MTEB/ThuNewsClusteringP2P metrics: - type: main_score value: 72.29482641581953 - type: v_measure value: 72.29482641581953 - type: v_measure_std value: 1.335959363862232 task: type: Clustering - dataset: config: default name: MTEB ThuNewsClusteringS2S revision: 8a8b2caeda43f39e13c4bc5bea0f8a667896e10d split: test type: C-MTEB/ThuNewsClusteringS2S metrics: - type: main_score value: 65.18517414783898 - type: v_measure value: 65.18517414783898 - type: v_measure_std value: 1.4758720962337082 task: type: Clustering - dataset: config: default name: MTEB VideoRetrieval revision: 58c2597a5943a2ba48f4668c3b90d796283c5639 split: dev type: C-MTEB/VideoRetrieval metrics: - type: main_score value: 71.768 - type: map_at_1 value: 57.8 - type: map_at_10 value: 67.47 - type: map_at_100 value: 67.97800000000001 - type: map_at_1000 value: 67.988 - type: map_at_20 value: 67.827 - type: map_at_3 value: 65.583 - type: map_at_5 value: 66.743 - type: mrr_at_1 value: 57.699999999999996 - type: mrr_at_10 value: 67.4202777777778 - type: mrr_at_100 value: 67.92827763873187 - type: mrr_at_1000 value: 67.93825676811373 - type: mrr_at_20 value: 67.77708320016758 - type: mrr_at_3 value: 65.53333333333336 - type: mrr_at_5 value: 66.69333333333337 - type: nauc_map_at_1000_diff1 value: 69.57667330550309 - type: nauc_map_at_1000_max value: 7.543501399956271 - type: nauc_map_at_1000_std value: -31.243010383568436 - type: nauc_map_at_100_diff1 value: 69.5725467000578 - type: nauc_map_at_100_max value: 7.557154826181643 - type: nauc_map_at_100_std value: -31.244478201857078 - type: nauc_map_at_10_diff1 value: 69.66295657598643 - type: nauc_map_at_10_max value: 7.354933246004566 - type: nauc_map_at_10_std value: -31.85480226841353 - type: nauc_map_at_1_diff1 value: 71.13428332324658 - type: nauc_map_at_1_max value: 5.617556184249036 - type: nauc_map_at_1_std value: -29.319041979284005 - type: nauc_map_at_20_diff1 value: 69.58825754865646 - type: nauc_map_at_20_max value: 7.493499540606082 - type: nauc_map_at_20_std value: -31.421082585601923 - type: nauc_map_at_3_diff1 value: 69.51937616497842 - type: nauc_map_at_3_max value: 6.58989269981783 - type: nauc_map_at_3_std value: -31.99863169578963 - type: nauc_map_at_5_diff1 value: 69.5662738665744 - type: nauc_map_at_5_max value: 7.15847377146298 - type: nauc_map_at_5_std value: -31.915079957015152 - type: nauc_mrr_at_1000_diff1 value: 69.7213828991122 - type: nauc_mrr_at_1000_max value: 7.429696859248949 - type: nauc_mrr_at_1000_std value: -31.443415682632214 - type: nauc_mrr_at_100_diff1 value: 69.71720349997756 - type: nauc_mrr_at_100_max value: 7.44338462432531 - type: nauc_mrr_at_100_std value: -31.444805433177237 - type: nauc_mrr_at_10_diff1 value: 69.80501501763766 - type: nauc_mrr_at_10_max value: 7.243295472335978 - type: nauc_mrr_at_10_std value: -32.05120239232289 - type: nauc_mrr_at_1_diff1 value: 71.36516347743338 - type: nauc_mrr_at_1_max value: 5.4367617772159935 - type: nauc_mrr_at_1_std value: -29.643310130582368 - type: nauc_mrr_at_20_diff1 value: 69.73206575087396 - type: nauc_mrr_at_20_max value: 7.380431929849562 - type: nauc_mrr_at_20_std value: -31.620128440139172 - type: nauc_mrr_at_3_diff1 value: 69.65406365393979 - type: nauc_mrr_at_3_max value: 6.484611159978816 - type: nauc_mrr_at_3_std value: -32.18462142962385 - type: nauc_mrr_at_5_diff1 value: 69.70510268881297 - type: nauc_mrr_at_5_max value: 7.049554381027974 - type: nauc_mrr_at_5_std value: -32.106886637783475 - type: nauc_ndcg_at_1000_diff1 value: 69.13809538133073 - type: nauc_ndcg_at_1000_max value: 9.63283640379659 - type: nauc_ndcg_at_1000_std value: -29.449553581261878 - type: nauc_ndcg_at_100_diff1 value: 68.99395374944251 - type: nauc_ndcg_at_100_max value: 10.176719987248335 - type: nauc_ndcg_at_100_std value: -29.350437600307988 - type: nauc_ndcg_at_10_diff1 value: 69.46707308337217 - type: nauc_ndcg_at_10_max value: 8.763989177891727 - type: nauc_ndcg_at_10_std value: -32.80464097517361 - type: nauc_ndcg_at_1_diff1 value: 71.13428332324658 - type: nauc_ndcg_at_1_max value: 5.617556184249036 - type: nauc_ndcg_at_1_std value: -29.319041979284005 - type: nauc_ndcg_at_20_diff1 value: 69.18406330264568 - type: nauc_ndcg_at_20_max value: 9.638224463038883 - type: nauc_ndcg_at_20_std value: -30.852858198950152 - type: nauc_ndcg_at_3_diff1 value: 69.09204185224701 - type: nauc_ndcg_at_3_max value: 7.001054807139578 - type: nauc_ndcg_at_3_std value: -33.03926415507931 - type: nauc_ndcg_at_5_diff1 value: 69.19746073354963 - type: nauc_ndcg_at_5_max value: 8.129939462557209 - type: nauc_ndcg_at_5_std value: -32.890200669688824 - type: nauc_precision_at_1000_diff1 value: 55.42483660130655 - type: nauc_precision_at_1000_max value: 87.87815126050374 - type: nauc_precision_at_1000_std value: 83.7791783380018 - type: nauc_precision_at_100_diff1 value: 57.369917664035285 - type: nauc_precision_at_100_max value: 67.47729394788239 - type: nauc_precision_at_100_std value: 27.854171971818914 - type: nauc_precision_at_10_diff1 value: 68.85063417125569 - type: nauc_precision_at_10_max value: 17.102325185471244 - type: nauc_precision_at_10_std value: -37.74775895599127 - type: nauc_precision_at_1_diff1 value: 71.13428332324658 - type: nauc_precision_at_1_max value: 5.617556184249036 - type: nauc_precision_at_1_std value: -29.319041979284005 - type: nauc_precision_at_20_diff1 value: 66.21858365339261 - type: nauc_precision_at_20_max value: 30.03311290151776 - type: nauc_precision_at_20_std value: -21.57151520401986 - type: nauc_precision_at_3_diff1 value: 67.63911432841829 - type: nauc_precision_at_3_max value: 8.481916121984119 - type: nauc_precision_at_3_std value: -36.756075465752815 - type: nauc_precision_at_5_diff1 value: 67.81563046786316 - type: nauc_precision_at_5_max value: 12.39051881812358 - type: nauc_precision_at_5_std value: -36.853583944247085 - type: nauc_recall_at_1000_diff1 value: 55.424836601307135 - type: nauc_recall_at_1000_max value: 87.87815126050475 - type: nauc_recall_at_1000_std value: 83.77917833800133 - type: nauc_recall_at_100_diff1 value: 57.36991766403544 - type: nauc_recall_at_100_max value: 67.47729394788247 - type: nauc_recall_at_100_std value: 27.854171971819124 - type: nauc_recall_at_10_diff1 value: 68.8506341712557 - type: nauc_recall_at_10_max value: 17.102325185471305 - type: nauc_recall_at_10_std value: -37.747758955991095 - type: nauc_recall_at_1_diff1 value: 71.13428332324658 - type: nauc_recall_at_1_max value: 5.617556184249036 - type: nauc_recall_at_1_std value: -29.319041979284005 - type: nauc_recall_at_20_diff1 value: 66.21858365339266 - type: nauc_recall_at_20_max value: 30.033112901517956 - type: nauc_recall_at_20_std value: -21.57151520401992 - type: nauc_recall_at_3_diff1 value: 67.63911432841829 - type: nauc_recall_at_3_max value: 8.481916121984066 - type: nauc_recall_at_3_std value: -36.75607546575289 - type: nauc_recall_at_5_diff1 value: 67.81563046786322 - type: nauc_recall_at_5_max value: 12.390518818123626 - type: nauc_recall_at_5_std value: -36.85358394424703 - type: ndcg_at_1 value: 57.8 - type: ndcg_at_10 value: 71.768 - type: ndcg_at_100 value: 74.093 - type: ndcg_at_1000 value: 74.39 - type: ndcg_at_20 value: 73.078 - type: ndcg_at_3 value: 67.93900000000001 - type: ndcg_at_5 value: 70.035 - type: precision_at_1 value: 57.8 - type: precision_at_10 value: 8.51 - type: precision_at_100 value: 0.9560000000000001 - type: precision_at_1000 value: 0.098 - type: precision_at_20 value: 4.515000000000001 - type: precision_at_3 value: 24.9 - type: precision_at_5 value: 15.959999999999999 - type: recall_at_1 value: 57.8 - type: recall_at_10 value: 85.1 - type: recall_at_100 value: 95.6 - type: recall_at_1000 value: 98.0 - type: recall_at_20 value: 90.3 - type: recall_at_3 value: 74.7 - type: recall_at_5 value: 79.80000000000001 task: type: Retrieval - dataset: config: default name: MTEB Waimai revision: 339287def212450dcaa9df8c22bf93e9980c7023 split: test type: C-MTEB/waimai-classification metrics: - type: accuracy value: 88.85 - type: ap value: 74.3186702823446 - type: ap_weighted value: 74.3186702823446 - type: f1 value: 87.23283814404301 - type: f1_weighted value: 88.8207921928931 - type: main_score value: 88.85 task: type: Classification - dataset: config: default name: MTEB AlloProfClusteringP2P revision: 392ba3f5bcc8c51f578786c1fc3dae648662cb9b split: test type: lyon-nlp/alloprof metrics: - type: main_score value: 65.0533654633344 - type: v_measure value: 65.0533654633344 - type: v_measure_std value: 2.2966620551090435 task: type: Clustering - dataset: config: default name: MTEB AlloProfClusteringS2S revision: 392ba3f5bcc8c51f578786c1fc3dae648662cb9b split: test type: lyon-nlp/alloprof metrics: - type: main_score value: 47.602163687995045 - type: v_measure value: 47.602163687995045 - type: v_measure_std value: 2.2689446239597566 task: type: Clustering - dataset: config: default name: MTEB AlloprofReranking revision: 65393d0d7a08a10b4e348135e824f385d420b0fd split: test type: lyon-nlp/mteb-fr-reranking-alloprof-s2p metrics: - type: main_score value: 75.17961983598764 - type: map value: 75.17961983598764 - type: mrr value: 76.42541875702497 - type: nAUC_map_diff1 value: 57.62404220848294 - type: nAUC_map_max value: 23.51522112548143 - type: nAUC_map_std value: 23.388021631358534 - type: nAUC_mrr_diff1 value: 57.37912924766015 - type: nAUC_mrr_max value: 24.08827107711477 - type: nAUC_mrr_std value: 22.59882369784862 task: type: Reranking - dataset: config: default name: MTEB AlloprofRetrieval revision: fcf295ea64c750f41fadbaa37b9b861558e1bfbd split: test type: lyon-nlp/alloprof metrics: - type: main_score value: 51.115 - type: map_at_1 value: 33.29 - type: map_at_10 value: 45.017 - type: map_at_100 value: 45.824 - type: map_at_1000 value: 45.869 - type: map_at_20 value: 45.521 - type: map_at_3 value: 41.947 - type: map_at_5 value: 43.683 - type: mrr_at_1 value: 33.29015544041451 - type: mrr_at_10 value: 45.017099816322656 - type: mrr_at_100 value: 45.82409198237741 - type: mrr_at_1000 value: 45.86948422318224 - type: mrr_at_20 value: 45.52135027955749 - type: mrr_at_3 value: 41.94732297063909 - type: mrr_at_5 value: 43.68307426597593 - type: nauc_map_at_1000_diff1 value: 39.73243793300825 - type: nauc_map_at_1000_max value: 35.182681191334545 - type: nauc_map_at_1000_std value: 0.5807963314175563 - type: nauc_map_at_100_diff1 value: 39.722470553779765 - type: nauc_map_at_100_max value: 35.21852170079372 - type: nauc_map_at_100_std value: 0.6058130117591941 - type: nauc_map_at_10_diff1 value: 39.617580503261316 - type: nauc_map_at_10_max value: 35.07856190730939 - type: nauc_map_at_10_std value: 0.12493876166019499 - type: nauc_map_at_1_diff1 value: 42.93831821879295 - type: nauc_map_at_1_max value: 31.12726818453338 - type: nauc_map_at_1_std value: -1.777716163867983 - type: nauc_map_at_20_diff1 value: 39.71626015027946 - type: nauc_map_at_20_max value: 35.18348804461845 - type: nauc_map_at_20_std value: 0.5105592256773273 - type: nauc_map_at_3_diff1 value: 39.16316684618178 - type: nauc_map_at_3_max value: 33.7216790348974 - type: nauc_map_at_3_std value: -0.3498983530547474 - type: nauc_map_at_5_diff1 value: 39.647098664998204 - type: nauc_map_at_5_max value: 34.7328902446716 - type: nauc_map_at_5_std value: 0.40121258866920295 - type: nauc_mrr_at_1000_diff1 value: 39.73243793300825 - type: nauc_mrr_at_1000_max value: 35.182681191334545 - type: nauc_mrr_at_1000_std value: 0.5807963314175563 - type: nauc_mrr_at_100_diff1 value: 39.722470553779765 - type: nauc_mrr_at_100_max value: 35.21852170079372 - type: nauc_mrr_at_100_std value: 0.6058130117591941 - type: nauc_mrr_at_10_diff1 value: 39.617580503261316 - type: nauc_mrr_at_10_max value: 35.07856190730939 - type: nauc_mrr_at_10_std value: 0.12493876166019499 - type: nauc_mrr_at_1_diff1 value: 42.93831821879295 - type: nauc_mrr_at_1_max value: 31.12726818453338 - type: nauc_mrr_at_1_std value: -1.777716163867983 - type: nauc_mrr_at_20_diff1 value: 39.71626015027946 - type: nauc_mrr_at_20_max value: 35.18348804461845 - type: nauc_mrr_at_20_std value: 0.5105592256773273 - type: nauc_mrr_at_3_diff1 value: 39.16316684618178 - type: nauc_mrr_at_3_max value: 33.7216790348974 - type: nauc_mrr_at_3_std value: -0.3498983530547474 - type: nauc_mrr_at_5_diff1 value: 39.647098664998204 - type: nauc_mrr_at_5_max value: 34.7328902446716 - type: nauc_mrr_at_5_std value: 0.40121258866920295 - type: nauc_ndcg_at_1000_diff1 value: 39.242329273752844 - type: nauc_ndcg_at_1000_max value: 36.778397735569555 - type: nauc_ndcg_at_1000_std value: 1.9890182774165592 - type: nauc_ndcg_at_100_diff1 value: 39.02516880016918 - type: nauc_ndcg_at_100_max value: 37.95005510807288 - type: nauc_ndcg_at_100_std value: 3.0123421168495446 - type: nauc_ndcg_at_10_diff1 value: 38.663112091897084 - type: nauc_ndcg_at_10_max value: 37.30636725701253 - type: nauc_ndcg_at_10_std value: 0.9040375526062125 - type: nauc_ndcg_at_1_diff1 value: 42.93831821879295 - type: nauc_ndcg_at_1_max value: 31.12726818453338 - type: nauc_ndcg_at_1_std value: -1.777716163867983 - type: nauc_ndcg_at_20_diff1 value: 38.929484271597715 - type: nauc_ndcg_at_20_max value: 37.75901008441635 - type: nauc_ndcg_at_20_std value: 2.4125130730180193 - type: nauc_ndcg_at_3_diff1 value: 37.92470631843311 - type: nauc_ndcg_at_3_max value: 34.450507157634625 - type: nauc_ndcg_at_3_std value: 0.11472879002764375 - type: nauc_ndcg_at_5_diff1 value: 38.778362690069535 - type: nauc_ndcg_at_5_max value: 36.356309785771344 - type: nauc_ndcg_at_5_std value: 1.502446979862238 - type: nauc_precision_at_1000_diff1 value: 33.18147748601189 - type: nauc_precision_at_1000_max value: 75.4289210395029 - type: nauc_precision_at_1000_std value: 67.86158379540547 - type: nauc_precision_at_100_diff1 value: 34.33353129136502 - type: nauc_precision_at_100_max value: 66.13041373773511 - type: nauc_precision_at_100_std value: 31.220993361158992 - type: nauc_precision_at_10_diff1 value: 35.191343796022224 - type: nauc_precision_at_10_max value: 46.5030195236983 - type: nauc_precision_at_10_std value: 3.8450864519608947 - type: nauc_precision_at_1_diff1 value: 42.93831821879295 - type: nauc_precision_at_1_max value: 31.12726818453338 - type: nauc_precision_at_1_std value: -1.777716163867983 - type: nauc_precision_at_20_diff1 value: 35.627717677075786 - type: nauc_precision_at_20_max value: 51.28161545904851 - type: nauc_precision_at_20_std value: 13.250115934134326 - type: nauc_precision_at_3_diff1 value: 34.252273608621316 - type: nauc_precision_at_3_max value: 36.573298257048386 - type: nauc_precision_at_3_std value: 1.4913107196473938 - type: nauc_precision_at_5_diff1 value: 36.160987910682564 - type: nauc_precision_at_5_max value: 41.778030453127776 - type: nauc_precision_at_5_std value: 5.2043752514074395 - type: nauc_recall_at_1000_diff1 value: 33.18147748600853 - type: nauc_recall_at_1000_max value: 75.42892103950442 - type: nauc_recall_at_1000_std value: 67.86158379540124 - type: nauc_recall_at_100_diff1 value: 34.33353129136507 - type: nauc_recall_at_100_max value: 66.13041373773494 - type: nauc_recall_at_100_std value: 31.22099336115893 - type: nauc_recall_at_10_diff1 value: 35.191343796022245 - type: nauc_recall_at_10_max value: 46.503019523698185 - type: nauc_recall_at_10_std value: 3.845086451960918 - type: nauc_recall_at_1_diff1 value: 42.93831821879295 - type: nauc_recall_at_1_max value: 31.12726818453338 - type: nauc_recall_at_1_std value: -1.777716163867983 - type: nauc_recall_at_20_diff1 value: 35.62771767707572 - type: nauc_recall_at_20_max value: 51.281615459048524 - type: nauc_recall_at_20_std value: 13.250115934134335 - type: nauc_recall_at_3_diff1 value: 34.2522736086213 - type: nauc_recall_at_3_max value: 36.57329825704846 - type: nauc_recall_at_3_std value: 1.4913107196474313 - type: nauc_recall_at_5_diff1 value: 36.160987910682586 - type: nauc_recall_at_5_max value: 41.778030453127826 - type: nauc_recall_at_5_std value: 5.204375251407459 - type: ndcg_at_1 value: 33.29 - type: ndcg_at_10 value: 51.115 - type: ndcg_at_100 value: 55.169999999999995 - type: ndcg_at_1000 value: 56.37799999999999 - type: ndcg_at_20 value: 52.929 - type: ndcg_at_3 value: 44.800000000000004 - type: ndcg_at_5 value: 47.911 - type: precision_at_1 value: 33.29 - type: precision_at_10 value: 7.042 - type: precision_at_100 value: 0.897 - type: precision_at_1000 value: 0.099 - type: precision_at_20 value: 3.877 - type: precision_at_3 value: 17.689 - type: precision_at_5 value: 12.116 - type: recall_at_1 value: 33.29 - type: recall_at_10 value: 70.423 - type: recall_at_100 value: 89.724 - type: recall_at_1000 value: 99.223 - type: recall_at_20 value: 77.547 - type: recall_at_3 value: 53.066 - type: recall_at_5 value: 60.57900000000001 task: type: Retrieval - dataset: config: fr name: MTEB AmazonReviewsClassification (fr) revision: 1399c76144fd37290681b995c656ef9b2e06e26d split: test type: mteb/amazon_reviews_multi metrics: - type: accuracy value: 52.312000000000005 - type: f1 value: 48.4862498298928 - type: f1_weighted value: 48.4862498298928 - type: main_score value: 52.312000000000005 task: type: Classification - dataset: config: default name: MTEB BSARDRetrieval revision: 5effa1b9b5fa3b0f9e12523e6e43e5f86a6e6d59 split: test type: maastrichtlawtech/bsard metrics: - type: main_score value: 61.260999999999996 - type: map_at_1 value: 9.459 - type: map_at_10 value: 16.678 - type: map_at_100 value: 17.774 - type: map_at_1000 value: 17.858999999999998 - type: map_at_20 value: 17.321 - type: map_at_3 value: 14.414 - type: map_at_5 value: 15.338 - type: mrr_at_1 value: 9.45945945945946 - type: mrr_at_10 value: 16.67792792792793 - type: mrr_at_100 value: 17.774362522691398 - type: mrr_at_1000 value: 17.859514657558048 - type: mrr_at_20 value: 17.321237623095215 - type: mrr_at_3 value: 14.414414414414411 - type: mrr_at_5 value: 15.337837837837837 - type: nauc_map_at_1000_diff1 value: 12.898723868198564 - type: nauc_map_at_1000_max value: 24.74492446599715 - type: nauc_map_at_1000_std value: 1.2568203864373333 - type: nauc_map_at_100_diff1 value: 12.851213507789646 - type: nauc_map_at_100_max value: 24.697430695694827 - type: nauc_map_at_100_std value: 1.2129231125259072 - type: nauc_map_at_10_diff1 value: 13.07707744574533 - type: nauc_map_at_10_max value: 24.72177610914694 - type: nauc_map_at_10_std value: -0.004280467049762924 - type: nauc_map_at_1_diff1 value: 21.11635429465273 - type: nauc_map_at_1_max value: 18.613969823495 - type: nauc_map_at_1_std value: -7.597691141141031 - type: nauc_map_at_20_diff1 value: 13.143191038140104 - type: nauc_map_at_20_max value: 24.904996278522944 - type: nauc_map_at_20_std value: 0.9083792521836654 - type: nauc_map_at_3_diff1 value: 14.048204755527621 - type: nauc_map_at_3_max value: 26.100088213960383 - type: nauc_map_at_3_std value: -0.7368466542848658 - type: nauc_map_at_5_diff1 value: 13.482648315427548 - type: nauc_map_at_5_max value: 24.803356986058585 - type: nauc_map_at_5_std value: -1.2343058157885538 - type: nauc_mrr_at_1000_diff1 value: 12.8977020563612 - type: nauc_mrr_at_1000_max value: 24.742855881415966 - type: nauc_mrr_at_1000_std value: 1.2564589487746798 - type: nauc_mrr_at_100_diff1 value: 12.850193815594121 - type: nauc_mrr_at_100_max value: 24.69536657999111 - type: nauc_mrr_at_100_std value: 1.2125623256841345 - type: nauc_mrr_at_10_diff1 value: 13.07707744574533 - type: nauc_mrr_at_10_max value: 24.72177610914694 - type: nauc_mrr_at_10_std value: -0.004280467049762924 - type: nauc_mrr_at_1_diff1 value: 21.11635429465273 - type: nauc_mrr_at_1_max value: 18.613969823495 - type: nauc_mrr_at_1_std value: -7.597691141141031 - type: nauc_mrr_at_20_diff1 value: 13.143191038140104 - type: nauc_mrr_at_20_max value: 24.904996278522944 - type: nauc_mrr_at_20_std value: 0.9083792521836654 - type: nauc_mrr_at_3_diff1 value: 14.048204755527621 - type: nauc_mrr_at_3_max value: 26.100088213960383 - type: nauc_mrr_at_3_std value: -0.7368466542848658 - type: nauc_mrr_at_5_diff1 value: 13.482648315427548 - type: nauc_mrr_at_5_max value: 24.803356986058585 - type: nauc_mrr_at_5_std value: -1.2343058157885538 - type: nauc_ndcg_at_1000_diff1 value: 11.261286687432275 - type: nauc_ndcg_at_1000_max value: 25.54862203427727 - type: nauc_ndcg_at_1000_std value: 6.9058640640154785 - type: nauc_ndcg_at_100_diff1 value: 10.686081917901197 - type: nauc_ndcg_at_100_max value: 25.79174788871077 - type: nauc_ndcg_at_100_std value: 7.46006445502064 - type: nauc_ndcg_at_10_diff1 value: 11.004401131569718 - type: nauc_ndcg_at_10_max value: 25.55455198645665 - type: nauc_ndcg_at_10_std value: 1.9332719624889871 - type: nauc_ndcg_at_1_diff1 value: 21.11635429465273 - type: nauc_ndcg_at_1_max value: 18.613969823495 - type: nauc_ndcg_at_1_std value: -7.597691141141031 - type: nauc_ndcg_at_20_diff1 value: 11.429743804671329 - type: nauc_ndcg_at_20_max value: 25.76018584848357 - type: nauc_ndcg_at_20_std value: 4.529843418751598 - type: nauc_ndcg_at_3_diff1 value: 12.375824114924743 - type: nauc_ndcg_at_3_max value: 27.899863438524925 - type: nauc_ndcg_at_3_std value: 0.48771263604259235 - type: nauc_ndcg_at_5_diff1 value: 11.669991174251026 - type: nauc_ndcg_at_5_max value: 25.60280203013294 - type: nauc_ndcg_at_5_std value: -0.4851689963092356 - type: nauc_precision_at_1000_diff1 value: 9.217959657671416 - type: nauc_precision_at_1000_max value: 26.785438149619228 - type: nauc_precision_at_1000_std value: 34.586575721918315 - type: nauc_precision_at_100_diff1 value: 6.944903323168987 - type: nauc_precision_at_100_max value: 29.703339699715265 - type: nauc_precision_at_100_std value: 28.76732870912694 - type: nauc_precision_at_10_diff1 value: 6.924015026667702 - type: nauc_precision_at_10_max value: 27.01090600566311 - type: nauc_precision_at_10_std value: 5.987244079632962 - type: nauc_precision_at_1_diff1 value: 21.11635429465273 - type: nauc_precision_at_1_max value: 18.613969823495 - type: nauc_precision_at_1_std value: -7.597691141141031 - type: nauc_precision_at_20_diff1 value: 8.547176188111889 - type: nauc_precision_at_20_max value: 27.008083551891193 - type: nauc_precision_at_20_std value: 12.963207602265662 - type: nauc_precision_at_3_diff1 value: 8.681050417224782 - type: nauc_precision_at_3_max value: 31.88286662067647 - type: nauc_precision_at_3_std value: 3.0982821138307313 - type: nauc_precision_at_5_diff1 value: 7.881738093345716 - type: nauc_precision_at_5_max value: 26.996917506757985 - type: nauc_precision_at_5_std value: 0.8392982481773691 - type: nauc_recall_at_1000_diff1 value: 9.21795965767156 - type: nauc_recall_at_1000_max value: 26.785438149619285 - type: nauc_recall_at_1000_std value: 34.58657572191825 - type: nauc_recall_at_100_diff1 value: 6.944903323168959 - type: nauc_recall_at_100_max value: 29.703339699715258 - type: nauc_recall_at_100_std value: 28.767328709126865 - type: nauc_recall_at_10_diff1 value: 6.924015026667665 - type: nauc_recall_at_10_max value: 27.01090600566308 - type: nauc_recall_at_10_std value: 5.987244079632916 - type: nauc_recall_at_1_diff1 value: 21.11635429465273 - type: nauc_recall_at_1_max value: 18.613969823495 - type: nauc_recall_at_1_std value: -7.597691141141031 - type: nauc_recall_at_20_diff1 value: 8.547176188111841 - type: nauc_recall_at_20_max value: 27.00808355189116 - type: nauc_recall_at_20_std value: 12.963207602265603 - type: nauc_recall_at_3_diff1 value: 8.681050417224794 - type: nauc_recall_at_3_max value: 31.882866620676502 - type: nauc_recall_at_3_std value: 3.098282113830734 - type: nauc_recall_at_5_diff1 value: 7.881738093345706 - type: nauc_recall_at_5_max value: 26.996917506757928 - type: nauc_recall_at_5_std value: 0.8392982481773533 - type: ndcg_at_1 value: 9.459 - type: ndcg_at_10 value: 20.752000000000002 - type: ndcg_at_100 value: 26.454 - type: ndcg_at_1000 value: 29.009 - type: ndcg_at_20 value: 23.128999999999998 - type: ndcg_at_3 value: 16.034000000000002 - type: ndcg_at_5 value: 17.701 - type: precision_at_1 value: 9.459 - type: precision_at_10 value: 3.3779999999999997 - type: precision_at_100 value: 0.613 - type: precision_at_1000 value: 0.082 - type: precision_at_20 value: 2.162 - type: precision_at_3 value: 6.907000000000001 - type: precision_at_5 value: 4.955 - type: recall_at_1 value: 9.459 - type: recall_at_10 value: 33.784 - type: recall_at_100 value: 61.260999999999996 - type: recall_at_1000 value: 81.982 - type: recall_at_20 value: 43.242999999999995 - type: recall_at_3 value: 20.721 - type: recall_at_5 value: 24.775 task: type: Retrieval - dataset: config: default name: MTEB HALClusteringS2S revision: e06ebbbb123f8144bef1a5d18796f3dec9ae2915 split: test type: lyon-nlp/clustering-hal-s2s metrics: - type: main_score value: 27.423835538133385 - type: v_measure value: 27.423835538133385 - type: v_measure_std value: 2.418592769322829 task: type: Clustering - dataset: config: fr name: MTEB MLSUMClusteringP2P (fr) revision: b5d54f8f3b61ae17845046286940f03c6bc79bc7 split: test type: reciTAL/mlsum metrics: - type: main_score value: 47.71996236371622 - type: v_measure value: 47.71996236371622 - type: v_measure_std value: 2.305651972507199 task: type: Clustering - dataset: config: fr name: MTEB MLSUMClusteringS2S (fr) revision: b5d54f8f3b61ae17845046286940f03c6bc79bc7 split: test type: reciTAL/mlsum metrics: - type: main_score value: 47.23617047594239 - type: v_measure value: 47.23617047594239 - type: v_measure_std value: 1.7496378174738418 task: type: Clustering - dataset: config: fr name: MTEB MTOPDomainClassification (fr) revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf split: test type: mteb/mtop_domain metrics: - type: accuracy value: 95.01722518008143 - type: f1 value: 94.84287480029258 - type: f1_weighted value: 95.0452443472173 - type: main_score value: 95.01722518008143 task: type: Classification - dataset: config: fr name: MTEB MTOPIntentClassification (fr) revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba split: test type: mteb/mtop_intent metrics: - type: accuracy value: 79.74945192608833 - type: f1 value: 53.733289773568536 - type: f1_weighted value: 79.64257473311346 - type: main_score value: 79.74945192608833 task: type: Classification - dataset: config: fra name: MTEB MasakhaNEWSClassification (fra) revision: 18193f187b92da67168c655c9973a165ed9593dd split: test type: mteb/masakhanews metrics: - type: accuracy value: 83.3649289099526 - type: f1 value: 79.77180137779536 - type: f1_weighted value: 83.01264683214308 - type: main_score value: 83.3649289099526 task: type: Classification - dataset: config: fra name: MTEB MasakhaNEWSClusteringP2P (fra) revision: 8ccc72e69e65f40c70e117d8b3c08306bb788b60 split: test type: masakhane/masakhanews metrics: - type: main_score value: 56.59401317251221 - type: v_measure value: 56.59401317251221 - type: v_measure_std value: 36.0760983348482 task: type: Clustering - dataset: config: fra name: MTEB MasakhaNEWSClusteringS2S (fra) revision: 8ccc72e69e65f40c70e117d8b3c08306bb788b60 split: test type: masakhane/masakhanews metrics: - type: main_score value: 71.82457224763512 - type: v_measure value: 71.82457224763512 - type: v_measure_std value: 33.00235264955351 task: type: Clustering - dataset: config: fr name: MTEB MassiveIntentClassification (fr) revision: 4672e20407010da34463acc759c162ca9734bca6 split: test type: mteb/amazon_massive_intent metrics: - type: accuracy value: 66.61398789509077 - type: f1 value: 60.956092562089324 - type: f1_weighted value: 64.68321586762302 - type: main_score value: 66.61398789509077 task: type: Classification - dataset: config: fr name: MTEB MassiveScenarioClassification (fr) revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 split: test type: mteb/amazon_massive_scenario metrics: - type: accuracy value: 75.68594485541358 - type: f1 value: 74.71713629733614 - type: f1_weighted value: 75.49907502505853 - type: main_score value: 75.68594485541358 task: type: Classification - dataset: config: fr name: MTEB MintakaRetrieval (fr) revision: efa78cc2f74bbcd21eff2261f9e13aebe40b814e split: test type: jinaai/mintakaqa metrics: - type: main_score value: 28.939999999999998 - type: map_at_1 value: 16.585 - type: map_at_10 value: 24.496000000000002 - type: map_at_100 value: 25.769 - type: map_at_1000 value: 25.857999999999997 - type: map_at_20 value: 25.224000000000004 - type: map_at_3 value: 21.908 - type: map_at_5 value: 23.464 - type: mrr_at_1 value: 16.584766584766587 - type: mrr_at_10 value: 24.49603824603824 - type: mrr_at_100 value: 25.76910538724444 - type: mrr_at_1000 value: 25.857970951595167 - type: mrr_at_20 value: 25.223803880220952 - type: mrr_at_3 value: 21.908271908271885 - type: mrr_at_5 value: 23.46437346437344 - type: nauc_map_at_1000_diff1 value: 24.449324509741782 - type: nauc_map_at_1000_max value: 16.857551392045668 - type: nauc_map_at_1000_std value: 9.029392522714542 - type: nauc_map_at_100_diff1 value: 24.416377398368994 - type: nauc_map_at_100_max value: 16.874285303124278 - type: nauc_map_at_100_std value: 9.042769202417494 - type: nauc_map_at_10_diff1 value: 24.30792102146879 - type: nauc_map_at_10_max value: 16.618278881262178 - type: nauc_map_at_10_std value: 8.925464095934705 - type: nauc_map_at_1_diff1 value: 29.76904844307235 - type: nauc_map_at_1_max value: 14.721060890557535 - type: nauc_map_at_1_std value: 4.1356111912072935 - type: nauc_map_at_20_diff1 value: 24.42346651590614 - type: nauc_map_at_20_max value: 16.780464496035194 - type: nauc_map_at_20_std value: 8.981541510161698 - type: nauc_map_at_3_diff1 value: 25.463420671464554 - type: nauc_map_at_3_max value: 16.4183897697643 - type: nauc_map_at_3_std value: 7.625690829397712 - type: nauc_map_at_5_diff1 value: 24.93868444905917 - type: nauc_map_at_5_max value: 16.091635346098528 - type: nauc_map_at_5_std value: 8.39683861182176 - type: nauc_mrr_at_1000_diff1 value: 24.449324509741782 - type: nauc_mrr_at_1000_max value: 16.857551392045668 - type: nauc_mrr_at_1000_std value: 9.029392522714542 - type: nauc_mrr_at_100_diff1 value: 24.416377398368994 - type: nauc_mrr_at_100_max value: 16.874285303124278 - type: nauc_mrr_at_100_std value: 9.042769202417494 - type: nauc_mrr_at_10_diff1 value: 24.30792102146879 - type: nauc_mrr_at_10_max value: 16.618278881262178 - type: nauc_mrr_at_10_std value: 8.925464095934705 - type: nauc_mrr_at_1_diff1 value: 29.76904844307235 - type: nauc_mrr_at_1_max value: 14.721060890557535 - type: nauc_mrr_at_1_std value: 4.1356111912072935 - type: nauc_mrr_at_20_diff1 value: 24.42346651590614 - type: nauc_mrr_at_20_max value: 16.780464496035194 - type: nauc_mrr_at_20_std value: 8.981541510161698 - type: nauc_mrr_at_3_diff1 value: 25.463420671464554 - type: nauc_mrr_at_3_max value: 16.4183897697643 - type: nauc_mrr_at_3_std value: 7.625690829397712 - type: nauc_mrr_at_5_diff1 value: 24.93868444905917 - type: nauc_mrr_at_5_max value: 16.091635346098528 - type: nauc_mrr_at_5_std value: 8.39683861182176 - type: nauc_ndcg_at_1000_diff1 value: 23.235835294659104 - type: nauc_ndcg_at_1000_max value: 18.130069784749416 - type: nauc_ndcg_at_1000_std value: 11.373761760081479 - type: nauc_ndcg_at_100_diff1 value: 22.573685299807515 - type: nauc_ndcg_at_100_max value: 18.932477112977686 - type: nauc_ndcg_at_100_std value: 12.149429764085687 - type: nauc_ndcg_at_10_diff1 value: 22.056289907300375 - type: nauc_ndcg_at_10_max value: 17.44090382465399 - type: nauc_ndcg_at_10_std value: 11.078636741064129 - type: nauc_ndcg_at_1_diff1 value: 29.76904844307235 - type: nauc_ndcg_at_1_max value: 14.721060890557535 - type: nauc_ndcg_at_1_std value: 4.1356111912072935 - type: nauc_ndcg_at_20_diff1 value: 22.35349136882196 - type: nauc_ndcg_at_20_max value: 18.085452594784922 - type: nauc_ndcg_at_20_std value: 11.380430365055334 - type: nauc_ndcg_at_3_diff1 value: 24.25988476966241 - type: nauc_ndcg_at_3_max value: 16.80378720501912 - type: nauc_ndcg_at_3_std value: 8.666118913493746 - type: nauc_ndcg_at_5_diff1 value: 23.43312793787156 - type: nauc_ndcg_at_5_max value: 16.291356652340085 - type: nauc_ndcg_at_5_std value: 9.985808307664348 - type: nauc_precision_at_1000_diff1 value: 20.59167291436066 - type: nauc_precision_at_1000_max value: 37.11669863650303 - type: nauc_precision_at_1000_std value: 53.79402863337551 - type: nauc_precision_at_100_diff1 value: 16.48046435474148 - type: nauc_precision_at_100_max value: 30.49436900744379 - type: nauc_precision_at_100_std value: 27.593241126791774 - type: nauc_precision_at_10_diff1 value: 16.21264848839721 - type: nauc_precision_at_10_max value: 19.68782276242417 - type: nauc_precision_at_10_std value: 16.66213297771153 - type: nauc_precision_at_1_diff1 value: 29.76904844307235 - type: nauc_precision_at_1_max value: 14.721060890557535 - type: nauc_precision_at_1_std value: 4.1356111912072935 - type: nauc_precision_at_20_diff1 value: 16.782602338251536 - type: nauc_precision_at_20_max value: 22.266838160036038 - type: nauc_precision_at_20_std value: 18.225681654722052 - type: nauc_precision_at_3_diff1 value: 21.258021189556455 - type: nauc_precision_at_3_max value: 17.741568150869735 - type: nauc_precision_at_3_std value: 11.278258942650373 - type: nauc_precision_at_5_diff1 value: 19.74165284616189 - type: nauc_precision_at_5_max value: 16.72366359140884 - type: nauc_precision_at_5_std value: 13.991493395533237 - type: nauc_recall_at_1000_diff1 value: 20.591672914360146 - type: nauc_recall_at_1000_max value: 37.11669863650274 - type: nauc_recall_at_1000_std value: 53.79402863337549 - type: nauc_recall_at_100_diff1 value: 16.480464354741507 - type: nauc_recall_at_100_max value: 30.494369007443765 - type: nauc_recall_at_100_std value: 27.59324112679175 - type: nauc_recall_at_10_diff1 value: 16.2126484883972 - type: nauc_recall_at_10_max value: 19.6878227624242 - type: nauc_recall_at_10_std value: 16.66213297771157 - type: nauc_recall_at_1_diff1 value: 29.76904844307235 - type: nauc_recall_at_1_max value: 14.721060890557535 - type: nauc_recall_at_1_std value: 4.1356111912072935 - type: nauc_recall_at_20_diff1 value: 16.782602338251547 - type: nauc_recall_at_20_max value: 22.266838160036027 - type: nauc_recall_at_20_std value: 18.22568165472206 - type: nauc_recall_at_3_diff1 value: 21.258021189556477 - type: nauc_recall_at_3_max value: 17.741568150869757 - type: nauc_recall_at_3_std value: 11.278258942650384 - type: nauc_recall_at_5_diff1 value: 19.741652846161877 - type: nauc_recall_at_5_max value: 16.723663591408847 - type: nauc_recall_at_5_std value: 13.991493395533256 - type: ndcg_at_1 value: 16.585 - type: ndcg_at_10 value: 28.939999999999998 - type: ndcg_at_100 value: 35.568 - type: ndcg_at_1000 value: 38.201 - type: ndcg_at_20 value: 31.576999999999998 - type: ndcg_at_3 value: 23.634 - type: ndcg_at_5 value: 26.462000000000003 - type: precision_at_1 value: 16.585 - type: precision_at_10 value: 4.316 - type: precision_at_100 value: 0.752 - type: precision_at_1000 value: 0.096 - type: precision_at_20 value: 2.6780000000000004 - type: precision_at_3 value: 9.541 - type: precision_at_5 value: 7.109 - type: recall_at_1 value: 16.585 - type: recall_at_10 value: 43.161 - type: recall_at_100 value: 75.184 - type: recall_at_1000 value: 96.478 - type: recall_at_20 value: 53.563 - type: recall_at_3 value: 28.624 - type: recall_at_5 value: 35.545 task: type: Retrieval - dataset: config: fr name: MTEB OpusparcusPC (fr) revision: 9e9b1f8ef51616073f47f306f7f47dd91663f86a split: test type: GEM/opusparcus metrics: - type: cosine_accuracy value: 82.90190735694823 - type: cosine_accuracy_threshold value: 86.7036521434784 - type: cosine_ap value: 93.15630587159856 - type: cosine_f1 value: 88.28744750349978 - type: cosine_f1_threshold value: 86.63686513900757 - type: cosine_precision value: 83.27464788732394 - type: cosine_recall value: 93.94240317775571 - type: dot_accuracy value: 82.90190735694823 - type: dot_accuracy_threshold value: 86.70365810394287 - type: dot_ap value: 93.15630587159856 - type: dot_f1 value: 88.28744750349978 - type: dot_f1_threshold value: 86.63686513900757 - type: dot_precision value: 83.27464788732394 - type: dot_recall value: 93.94240317775571 - type: euclidean_accuracy value: 82.90190735694823 - type: euclidean_accuracy_threshold value: 51.56809687614441 - type: euclidean_ap value: 93.15630587159856 - type: euclidean_f1 value: 88.28744750349978 - type: euclidean_f1_threshold value: 51.69745683670044 - type: euclidean_precision value: 83.27464788732394 - type: euclidean_recall value: 93.94240317775571 - type: main_score value: 93.15630587159856 - type: manhattan_accuracy value: 82.83378746594006 - type: manhattan_accuracy_threshold value: 1203.4871101379395 - type: manhattan_ap value: 93.13986431895378 - type: manhattan_f1 value: 88.20224719101122 - type: manhattan_f1_threshold value: 1204.0313720703125 - type: manhattan_precision value: 83.43666961913198 - type: manhattan_recall value: 93.545183714002 - type: max_ap value: 93.15630587159856 - type: max_f1 value: 88.28744750349978 - type: max_precision value: 83.43666961913198 - type: max_recall value: 93.94240317775571 - type: similarity_accuracy value: 82.90190735694823 - type: similarity_accuracy_threshold value: 86.7036521434784 - type: similarity_ap value: 93.15630587159856 - type: similarity_f1 value: 88.28744750349978 - type: similarity_f1_threshold value: 86.63686513900757 - type: similarity_precision value: 83.27464788732394 - type: similarity_recall value: 93.94240317775571 task: type: PairClassification - dataset: config: fr name: MTEB PawsXPairClassification (fr) revision: 8a04d940a42cd40658986fdd8e3da561533a3646 split: test type: google-research-datasets/paws-x metrics: - type: cosine_accuracy value: 61.0 - type: cosine_accuracy_threshold value: 99.4693398475647 - type: cosine_ap value: 60.596634996072794 - type: cosine_f1 value: 62.77477477477477 - type: cosine_f1_threshold value: 95.54682970046997 - type: cosine_precision value: 46.52777777777778 - type: cosine_recall value: 96.4562569213732 - type: dot_accuracy value: 61.0 - type: dot_accuracy_threshold value: 99.4693398475647 - type: dot_ap value: 60.60423446409202 - type: dot_f1 value: 62.77477477477477 - type: dot_f1_threshold value: 95.54683566093445 - type: dot_precision value: 46.52777777777778 - type: dot_recall value: 96.4562569213732 - type: euclidean_accuracy value: 61.0 - type: euclidean_accuracy_threshold value: 10.301996022462845 - type: euclidean_ap value: 60.596589949359995 - type: euclidean_f1 value: 62.77477477477477 - type: euclidean_f1_threshold value: 29.843494296073914 - type: euclidean_precision value: 46.52777777777778 - type: euclidean_recall value: 96.4562569213732 - type: main_score value: 60.81315897987771 - type: manhattan_accuracy value: 61.1 - type: manhattan_accuracy_threshold value: 241.11316204071045 - type: manhattan_ap value: 60.81315897987771 - type: manhattan_f1 value: 62.80575539568345 - type: manhattan_f1_threshold value: 701.3707160949707 - type: manhattan_precision value: 46.510388918486946 - type: manhattan_recall value: 96.67774086378738 - type: max_ap value: 60.81315897987771 - type: max_f1 value: 62.80575539568345 - type: max_precision value: 46.52777777777778 - type: max_recall value: 96.67774086378738 - type: similarity_accuracy value: 61.0 - type: similarity_accuracy_threshold value: 99.4693398475647 - type: similarity_ap value: 60.596634996072794 - type: similarity_f1 value: 62.77477477477477 - type: similarity_f1_threshold value: 95.54682970046997 - type: similarity_precision value: 46.52777777777778 - type: similarity_recall value: 96.4562569213732 task: type: PairClassification - dataset: config: default name: MTEB SICKFr revision: e077ab4cf4774a1e36d86d593b150422fafd8e8a split: test type: Lajavaness/SICK-fr metrics: - type: cosine_pearson value: 80.2458052019468 - type: cosine_spearman value: 76.90282547383322 - type: euclidean_pearson value: 76.96025494858225 - type: euclidean_spearman value: 76.90369875962587 - type: main_score value: 76.90282547383322 - type: manhattan_pearson value: 76.96191769993092 - type: manhattan_spearman value: 76.78368027265795 - type: pearson value: 80.2458052019468 - type: spearman value: 76.90282547383322 task: type: STS - dataset: config: fr name: MTEB STS22 (fr) revision: de9d86b3b84231dc21f76c7b7af1f28e2f57f6e3 split: test type: mteb/sts22-crosslingual-sts metrics: - type: cosine_pearson value: 71.8166682422511 - type: cosine_spearman value: 80.753981314684 - type: euclidean_pearson value: 77.26991759068049 - type: euclidean_spearman value: 80.753981314684 - type: main_score value: 80.753981314684 - type: manhattan_pearson value: 79.8934147126028 - type: manhattan_spearman value: 81.7842698600436 - type: pearson value: 71.8166682422511 - type: spearman value: 80.753981314684 task: type: STS - dataset: config: de-fr name: MTEB STS22 (de-fr) revision: de9d86b3b84231dc21f76c7b7af1f28e2f57f6e3 split: test type: mteb/sts22-crosslingual-sts metrics: - type: cosine_pearson value: 59.72069019022611 - type: cosine_spearman value: 66.19562370923092 - type: euclidean_pearson value: 60.38869296832584 - type: euclidean_spearman value: 66.19562370923092 - type: main_score value: 66.19562370923092 - type: manhattan_pearson value: 60.735818878018044 - type: manhattan_spearman value: 66.3310023111313 - type: pearson value: 59.72069019022611 - type: spearman value: 66.19562370923092 task: type: STS - dataset: config: fr-pl name: MTEB STS22 (fr-pl) revision: de9d86b3b84231dc21f76c7b7af1f28e2f57f6e3 split: test type: mteb/sts22-crosslingual-sts metrics: - type: cosine_pearson value: 90.73440998579521 - type: cosine_spearman value: 84.51542547285167 - type: euclidean_pearson value: 90.27154868530998 - type: euclidean_spearman value: 84.51542547285167 - type: main_score value: 84.51542547285167 - type: manhattan_pearson value: 89.71184089391063 - type: manhattan_spearman value: 84.51542547285167 - type: pearson value: 90.73440998579521 - type: spearman value: 84.51542547285167 task: type: STS - dataset: config: fr name: MTEB STSBenchmarkMultilingualSTS (fr) revision: 29afa2569dcedaaa2fe6a3dcfebab33d28b82e8c split: test type: mteb/stsb_multi_mt metrics: - type: cosine_pearson value: 79.50928730728984 - type: cosine_spearman value: 80.0932172585886 - type: euclidean_pearson value: 78.52408843163009 - type: euclidean_spearman value: 80.09288222088635 - type: main_score value: 80.0932172585886 - type: manhattan_pearson value: 78.48826046695687 - type: manhattan_spearman value: 80.02496924199357 - type: pearson value: 79.50928730728984 - type: spearman value: 80.0932172585886 task: type: STS - dataset: config: default name: MTEB SummEvalFr revision: b385812de6a9577b6f4d0f88c6a6e35395a94054 split: test type: lyon-nlp/summarization-summeval-fr-p2p metrics: - type: cosine_pearson value: 31.635895833264797 - type: cosine_spearman value: 29.316197375472907 - type: dot_pearson value: 31.635903578046314 - type: dot_spearman value: 29.320397860238174 - type: main_score value: 29.316197375472907 - type: pearson value: 31.635895833264797 - type: spearman value: 29.316197375472907 task: type: Summarization - dataset: config: default name: MTEB SyntecReranking revision: daf0863838cd9e3ba50544cdce3ac2b338a1b0ad split: test type: lyon-nlp/mteb-fr-reranking-syntec-s2p metrics: - type: main_score value: 85.96785714285713 - type: map value: 85.96785714285713 - type: mrr value: 85.96785714285713 - type: nAUC_map_diff1 value: 62.8939499970397 - type: nAUC_map_max value: 17.78395289837277 - type: nAUC_map_std value: 42.06945307317201 - type: nAUC_mrr_diff1 value: 62.8939499970397 - type: nAUC_mrr_max value: 17.78395289837277 - type: nAUC_mrr_std value: 42.06945307317201 task: type: Reranking - dataset: config: default name: MTEB SyntecRetrieval revision: 19661ccdca4dfc2d15122d776b61685f48c68ca9 split: test type: lyon-nlp/mteb-fr-retrieval-syntec-s2p metrics: - type: main_score value: 81.745 - type: map_at_1 value: 63.0 - type: map_at_10 value: 76.325 - type: map_at_100 value: 76.413 - type: map_at_1000 value: 76.413 - type: map_at_20 value: 76.39200000000001 - type: map_at_3 value: 74.833 - type: map_at_5 value: 76.033 - type: mrr_at_1 value: 63.0 - type: mrr_at_10 value: 76.32500000000002 - type: mrr_at_100 value: 76.41340579710148 - type: mrr_at_1000 value: 76.41340579710148 - type: mrr_at_20 value: 76.3916666666667 - type: mrr_at_3 value: 74.83333333333334 - type: mrr_at_5 value: 76.03333333333335 - type: nauc_map_at_1000_diff1 value: 54.59590007968431 - type: nauc_map_at_1000_max value: 17.62963207909923 - type: nauc_map_at_1000_std value: -20.385142659182947 - type: nauc_map_at_100_diff1 value: 54.59590007968431 - type: nauc_map_at_100_max value: 17.62963207909923 - type: nauc_map_at_100_std value: -20.385142659182947 - type: nauc_map_at_10_diff1 value: 54.65626490051626 - type: nauc_map_at_10_max value: 17.90162538916067 - type: nauc_map_at_10_std value: -19.9397305494493 - type: nauc_map_at_1_diff1 value: 60.494523574484084 - type: nauc_map_at_1_max value: 19.353170685845864 - type: nauc_map_at_1_std value: -26.247669189426365 - type: nauc_map_at_20_diff1 value: 54.59680474780081 - type: nauc_map_at_20_max value: 17.623725255245322 - type: nauc_map_at_20_std value: -20.30147444307871 - type: nauc_map_at_3_diff1 value: 52.61672286351305 - type: nauc_map_at_3_max value: 15.107249942338704 - type: nauc_map_at_3_std value: -19.11937528144186 - type: nauc_map_at_5_diff1 value: 53.612111533008445 - type: nauc_map_at_5_max value: 17.220940492292932 - type: nauc_map_at_5_std value: -19.898915353831192 - type: nauc_mrr_at_1000_diff1 value: 54.59590007968431 - type: nauc_mrr_at_1000_max value: 17.62963207909923 - type: nauc_mrr_at_1000_std value: -20.385142659182947 - type: nauc_mrr_at_100_diff1 value: 54.59590007968431 - type: nauc_mrr_at_100_max value: 17.62963207909923 - type: nauc_mrr_at_100_std value: -20.385142659182947 - type: nauc_mrr_at_10_diff1 value: 54.65626490051626 - type: nauc_mrr_at_10_max value: 17.90162538916067 - type: nauc_mrr_at_10_std value: -19.9397305494493 - type: nauc_mrr_at_1_diff1 value: 60.494523574484084 - type: nauc_mrr_at_1_max value: 19.353170685845864 - type: nauc_mrr_at_1_std value: -26.247669189426365 - type: nauc_mrr_at_20_diff1 value: 54.59680474780081 - type: nauc_mrr_at_20_max value: 17.623725255245322 - type: nauc_mrr_at_20_std value: -20.30147444307871 - type: nauc_mrr_at_3_diff1 value: 52.61672286351305 - type: nauc_mrr_at_3_max value: 15.107249942338704 - type: nauc_mrr_at_3_std value: -19.11937528144186 - type: nauc_mrr_at_5_diff1 value: 53.612111533008445 - type: nauc_mrr_at_5_max value: 17.220940492292932 - type: nauc_mrr_at_5_std value: -19.898915353831192 - type: nauc_ndcg_at_1000_diff1 value: 54.19476697832697 - type: nauc_ndcg_at_1000_max value: 17.552295093432928 - type: nauc_ndcg_at_1000_std value: -19.405594751351334 - type: nauc_ndcg_at_100_diff1 value: 54.19476697832697 - type: nauc_ndcg_at_100_max value: 17.552295093432928 - type: nauc_ndcg_at_100_std value: -19.405594751351334 - type: nauc_ndcg_at_10_diff1 value: 54.50266178555259 - type: nauc_ndcg_at_10_max value: 18.833113705466285 - type: nauc_ndcg_at_10_std value: -16.796489600002328 - type: nauc_ndcg_at_1_diff1 value: 60.494523574484084 - type: nauc_ndcg_at_1_max value: 19.353170685845864 - type: nauc_ndcg_at_1_std value: -26.247669189426365 - type: nauc_ndcg_at_20_diff1 value: 54.209319885404405 - type: nauc_ndcg_at_20_max value: 17.48942074311288 - type: nauc_ndcg_at_20_std value: -18.513618512775565 - type: nauc_ndcg_at_3_diff1 value: 49.81337689077804 - type: nauc_ndcg_at_3_max value: 12.542370633424529 - type: nauc_ndcg_at_3_std value: -15.485399782383524 - type: nauc_ndcg_at_5_diff1 value: 51.49975132120803 - type: nauc_ndcg_at_5_max value: 16.57215179192376 - type: nauc_ndcg_at_5_std value: -16.928852236722147 - type: nauc_precision_at_1000_diff1 value: .nan - type: nauc_precision_at_1000_max value: .nan - type: nauc_precision_at_1000_std value: .nan - type: nauc_precision_at_100_diff1 value: .nan - type: nauc_precision_at_100_max value: .nan - type: nauc_precision_at_100_std value: .nan - type: nauc_precision_at_10_diff1 value: 63.81886087768404 - type: nauc_precision_at_10_max value: 56.13912231559286 - type: nauc_precision_at_10_std value: 71.1718020541554 - type: nauc_precision_at_1_diff1 value: 60.494523574484084 - type: nauc_precision_at_1_max value: 19.353170685845864 - type: nauc_precision_at_1_std value: -26.247669189426365 - type: nauc_precision_at_20_diff1 value: 55.41549953314738 - type: nauc_precision_at_20_max value: 12.278244631185926 - type: nauc_precision_at_20_std value: 55.41549953314738 - type: nauc_precision_at_3_diff1 value: 31.497043261749212 - type: nauc_precision_at_3_max value: -5.192447349310147 - type: nauc_precision_at_3_std value: 9.274821039526845 - type: nauc_precision_at_5_diff1 value: 23.342670401494065 - type: nauc_precision_at_5_max value: 8.228291316527036 - type: nauc_precision_at_5_std value: 24.56816059757235 - type: nauc_recall_at_1000_diff1 value: .nan - type: nauc_recall_at_1000_max value: .nan - type: nauc_recall_at_1000_std value: .nan - type: nauc_recall_at_100_diff1 value: .nan - type: nauc_recall_at_100_max value: .nan - type: nauc_recall_at_100_std value: .nan - type: nauc_recall_at_10_diff1 value: 63.81886087768457 - type: nauc_recall_at_10_max value: 56.13912231559305 - type: nauc_recall_at_10_std value: 71.171802054155 - type: nauc_recall_at_1_diff1 value: 60.494523574484084 - type: nauc_recall_at_1_max value: 19.353170685845864 - type: nauc_recall_at_1_std value: -26.247669189426365 - type: nauc_recall_at_20_diff1 value: 55.415499533146296 - type: nauc_recall_at_20_max value: 12.278244631185359 - type: nauc_recall_at_20_std value: 55.415499533146296 - type: nauc_recall_at_3_diff1 value: 31.497043261749237 - type: nauc_recall_at_3_max value: -5.19244734931015 - type: nauc_recall_at_3_std value: 9.274821039526998 - type: nauc_recall_at_5_diff1 value: 23.34267040149383 - type: nauc_recall_at_5_max value: 8.228291316526445 - type: nauc_recall_at_5_std value: 24.56816059757245 - type: ndcg_at_1 value: 63.0 - type: ndcg_at_10 value: 81.745 - type: ndcg_at_100 value: 82.175 - type: ndcg_at_1000 value: 82.175 - type: ndcg_at_20 value: 81.99499999999999 - type: ndcg_at_3 value: 78.964 - type: ndcg_at_5 value: 81.074 - type: precision_at_1 value: 63.0 - type: precision_at_10 value: 9.8 - type: precision_at_100 value: 1.0 - type: precision_at_1000 value: 0.1 - type: precision_at_20 value: 4.95 - type: precision_at_3 value: 30.333 - type: precision_at_5 value: 19.2 - type: recall_at_1 value: 63.0 - type: recall_at_10 value: 98.0 - type: recall_at_100 value: 100.0 - type: recall_at_1000 value: 100.0 - type: recall_at_20 value: 99.0 - type: recall_at_3 value: 91.0 - type: recall_at_5 value: 96.0 task: type: Retrieval - dataset: config: fra-fra name: MTEB XPQARetrieval (fr) revision: c99d599f0a6ab9b85b065da6f9d94f9cf731679f split: test type: jinaai/xpqa metrics: - type: main_score value: 59.299 - type: map_at_1 value: 33.498 - type: map_at_10 value: 52.705999999999996 - type: map_at_100 value: 54.278999999999996 - type: map_at_1000 value: 54.358 - type: map_at_20 value: 53.599 - type: map_at_3 value: 46.848 - type: map_at_5 value: 50.483999999999995 - type: mrr_at_1 value: 54.33911882510013 - type: mrr_at_10 value: 62.24383834530695 - type: mrr_at_100 value: 62.910121526268924 - type: mrr_at_1000 value: 62.934836010235195 - type: mrr_at_20 value: 62.64855949718011 - type: mrr_at_3 value: 60.23587004895412 - type: mrr_at_5 value: 61.277258566978155 - type: nauc_map_at_1000_diff1 value: 50.40068320668247 - type: nauc_map_at_1000_max value: 45.419834912711345 - type: nauc_map_at_1000_std value: 4.690219602260963 - type: nauc_map_at_100_diff1 value: 50.36635447882883 - type: nauc_map_at_100_max value: 45.4147569946384 - type: nauc_map_at_100_std value: 4.6979944201303825 - type: nauc_map_at_10_diff1 value: 50.2118074250129 - type: nauc_map_at_10_max value: 44.94969058947611 - type: nauc_map_at_10_std value: 4.398792604612038 - type: nauc_map_at_1_diff1 value: 58.603849067897364 - type: nauc_map_at_1_max value: 27.905162533509053 - type: nauc_map_at_1_std value: -0.3803212098210696 - type: nauc_map_at_20_diff1 value: 50.251993141938534 - type: nauc_map_at_20_max value: 45.19947447689957 - type: nauc_map_at_20_std value: 4.513253526305713 - type: nauc_map_at_3_diff1 value: 50.868419013808655 - type: nauc_map_at_3_max value: 38.7440731652971 - type: nauc_map_at_3_std value: 2.1235171982780567 - type: nauc_map_at_5_diff1 value: 50.48944050454372 - type: nauc_map_at_5_max value: 42.85070909512206 - type: nauc_map_at_5_std value: 3.7050886992455614 - type: nauc_mrr_at_1000_diff1 value: 57.229927096025335 - type: nauc_mrr_at_1000_max value: 50.938449952909025 - type: nauc_mrr_at_1000_std value: 6.772124463005406 - type: nauc_mrr_at_100_diff1 value: 57.2303932927905 - type: nauc_mrr_at_100_max value: 50.94674655436572 - type: nauc_mrr_at_100_std value: 6.771170622225867 - type: nauc_mrr_at_10_diff1 value: 57.140428095767994 - type: nauc_mrr_at_10_max value: 51.015650650701325 - type: nauc_mrr_at_10_std value: 6.786343260588584 - type: nauc_mrr_at_1_diff1 value: 59.61237414732749 - type: nauc_mrr_at_1_max value: 50.27947885239657 - type: nauc_mrr_at_1_std value: 6.325448424227606 - type: nauc_mrr_at_20_diff1 value: 57.24043520066768 - type: nauc_mrr_at_20_max value: 51.02257975115745 - type: nauc_mrr_at_20_std value: 6.849295913300065 - type: nauc_mrr_at_3_diff1 value: 57.48754760533025 - type: nauc_mrr_at_3_max value: 50.34268453986772 - type: nauc_mrr_at_3_std value: 5.576028856206244 - type: nauc_mrr_at_5_diff1 value: 57.01340095945496 - type: nauc_mrr_at_5_max value: 50.37124700329385 - type: nauc_mrr_at_5_std value: 6.176803150164063 - type: nauc_ndcg_at_1000_diff1 value: 51.877968132074926 - type: nauc_ndcg_at_1000_max value: 48.36110672671821 - type: nauc_ndcg_at_1000_std value: 6.108794230045721 - type: nauc_ndcg_at_100_diff1 value: 51.404027251165715 - type: nauc_ndcg_at_100_max value: 48.42673179365166 - type: nauc_ndcg_at_100_std value: 6.409635305448258 - type: nauc_ndcg_at_10_diff1 value: 50.555165677188306 - type: nauc_ndcg_at_10_max value: 47.47519310008686 - type: nauc_ndcg_at_10_std value: 5.4093301966396385 - type: nauc_ndcg_at_1_diff1 value: 59.61237414732749 - type: nauc_ndcg_at_1_max value: 50.27947885239657 - type: nauc_ndcg_at_1_std value: 6.325448424227606 - type: nauc_ndcg_at_20_diff1 value: 50.72341403581928 - type: nauc_ndcg_at_20_max value: 47.827204708689806 - type: nauc_ndcg_at_20_std value: 5.697769161434286 - type: nauc_ndcg_at_3_diff1 value: 51.3008421472128 - type: nauc_ndcg_at_3_max value: 45.246290747859376 - type: nauc_ndcg_at_3_std value: 3.0258563412678208 - type: nauc_ndcg_at_5_diff1 value: 50.75750841863859 - type: nauc_ndcg_at_5_max value: 44.6012523853242 - type: nauc_ndcg_at_5_std value: 4.171558602586512 - type: nauc_precision_at_1000_diff1 value: -10.620804642735912 - type: nauc_precision_at_1000_max value: 20.710188752354686 - type: nauc_precision_at_1000_std value: 8.004079823209057 - type: nauc_precision_at_100_diff1 value: -6.003091379108357 - type: nauc_precision_at_100_max value: 26.818229597933264 - type: nauc_precision_at_100_std value: 9.991598973813758 - type: nauc_precision_at_10_diff1 value: 5.876190205156904 - type: nauc_precision_at_10_max value: 38.075509886425046 - type: nauc_precision_at_10_std value: 8.785425849446119 - type: nauc_precision_at_1_diff1 value: 59.61237414732749 - type: nauc_precision_at_1_max value: 50.27947885239657 - type: nauc_precision_at_1_std value: 6.325448424227606 - type: nauc_precision_at_20_diff1 value: 1.9253058099019142 - type: nauc_precision_at_20_max value: 34.8886581006359 - type: nauc_precision_at_20_std value: 9.120051183604845 - type: nauc_precision_at_3_diff1 value: 17.487309868406694 - type: nauc_precision_at_3_max value: 41.703264394864284 - type: nauc_precision_at_3_std value: 6.565690923768522 - type: nauc_precision_at_5_diff1 value: 11.444446099569587 - type: nauc_precision_at_5_max value: 40.40685598234426 - type: nauc_precision_at_5_std value: 8.761851721129617 - type: nauc_recall_at_1000_diff1 value: 30.356073066469964 - type: nauc_recall_at_1000_max value: 59.34211677701755 - type: nauc_recall_at_1000_std value: 45.85472480324881 - type: nauc_recall_at_100_diff1 value: 37.197227716845035 - type: nauc_recall_at_100_max value: 46.43271901177922 - type: nauc_recall_at_100_std value: 14.080669406474067 - type: nauc_recall_at_10_diff1 value: 40.93453402067081 - type: nauc_recall_at_10_max value: 44.03007002975281 - type: nauc_recall_at_10_std value: 5.656063496811249 - type: nauc_recall_at_1_diff1 value: 58.603849067897364 - type: nauc_recall_at_1_max value: 27.905162533509053 - type: nauc_recall_at_1_std value: -0.3803212098210696 - type: nauc_recall_at_20_diff1 value: 38.76049220154408 - type: nauc_recall_at_20_max value: 43.04322846027003 - type: nauc_recall_at_20_std value: 5.986817583151329 - type: nauc_recall_at_3_diff1 value: 45.480809900773586 - type: nauc_recall_at_3_max value: 34.175287910561075 - type: nauc_recall_at_3_std value: -0.9428328336656849 - type: nauc_recall_at_5_diff1 value: 43.134263613996495 - type: nauc_recall_at_5_max value: 38.76946986340381 - type: nauc_recall_at_5_std value: 2.496846951162451 - type: ndcg_at_1 value: 54.339000000000006 - type: ndcg_at_10 value: 59.299 - type: ndcg_at_100 value: 64.895 - type: ndcg_at_1000 value: 66.283 - type: ndcg_at_20 value: 61.594 - type: ndcg_at_3 value: 54.176 - type: ndcg_at_5 value: 55.730999999999995 - type: precision_at_1 value: 54.339000000000006 - type: precision_at_10 value: 14.139 - type: precision_at_100 value: 1.879 - type: precision_at_1000 value: 0.20600000000000002 - type: precision_at_20 value: 7.843999999999999 - type: precision_at_3 value: 33.289 - type: precision_at_5 value: 24.005000000000003 - type: recall_at_1 value: 33.498 - type: recall_at_10 value: 68.389 - type: recall_at_100 value: 90.398 - type: recall_at_1000 value: 99.51 - type: recall_at_20 value: 75.957 - type: recall_at_3 value: 52.452 - type: recall_at_5 value: 59.9 task: type: Retrieval - dataset: config: eng-fra name: MTEB XPQARetrieval (eng-fra) revision: c99d599f0a6ab9b85b065da6f9d94f9cf731679f split: test type: jinaai/xpqa metrics: - type: main_score value: 34.148 - type: map_at_1 value: 13.142000000000001 - type: map_at_10 value: 27.504 - type: map_at_100 value: 30.122 - type: map_at_1000 value: 30.259000000000004 - type: map_at_20 value: 29.012999999999998 - type: map_at_3 value: 22.078 - type: map_at_5 value: 25.16 - type: mrr_at_1 value: 26.034712950600802 - type: mrr_at_10 value: 35.21965795664058 - type: mrr_at_100 value: 36.66580650341633 - type: mrr_at_1000 value: 36.719052382107876 - type: mrr_at_20 value: 36.16989730077047 - type: mrr_at_3 value: 32.51001335113486 - type: mrr_at_5 value: 34.02536715620828 - type: nauc_map_at_1000_diff1 value: 22.045384824751874 - type: nauc_map_at_1000_max value: 37.25875104070248 - type: nauc_map_at_1000_std value: 6.311979763063837 - type: nauc_map_at_100_diff1 value: 21.983455240362048 - type: nauc_map_at_100_max value: 37.2389940633854 - type: nauc_map_at_100_std value: 6.308890135460129 - type: nauc_map_at_10_diff1 value: 22.08654364936706 - type: nauc_map_at_10_max value: 36.83912535726259 - type: nauc_map_at_10_std value: 5.586878201266457 - type: nauc_map_at_1_diff1 value: 27.996481625958005 - type: nauc_map_at_1_max value: 22.170519197242168 - type: nauc_map_at_1_std value: 3.2754228003309094 - type: nauc_map_at_20_diff1 value: 22.084045154169598 - type: nauc_map_at_20_max value: 37.35198536276873 - type: nauc_map_at_20_std value: 6.287528082319668 - type: nauc_map_at_3_diff1 value: 23.518184109219312 - type: nauc_map_at_3_max value: 32.37508088252643 - type: nauc_map_at_3_std value: 2.793040909420183 - type: nauc_map_at_5_diff1 value: 22.617850315991227 - type: nauc_map_at_5_max value: 35.40344848186382 - type: nauc_map_at_5_std value: 4.296806736798404 - type: nauc_mrr_at_1000_diff1 value: 21.45583729151993 - type: nauc_mrr_at_1000_max value: 37.080805671948056 - type: nauc_mrr_at_1000_std value: 8.905986716816184 - type: nauc_mrr_at_100_diff1 value: 21.43908621145099 - type: nauc_mrr_at_100_max value: 37.062698361870055 - type: nauc_mrr_at_100_std value: 8.917647212658137 - type: nauc_mrr_at_10_diff1 value: 21.328903056739907 - type: nauc_mrr_at_10_max value: 36.77272980951551 - type: nauc_mrr_at_10_std value: 8.323076678764995 - type: nauc_mrr_at_1_diff1 value: 24.15510508158492 - type: nauc_mrr_at_1_max value: 38.16775944594129 - type: nauc_mrr_at_1_std value: 9.048540715004725 - type: nauc_mrr_at_20_diff1 value: 21.379354375668658 - type: nauc_mrr_at_20_max value: 37.0178596112037 - type: nauc_mrr_at_20_std value: 8.95011471773535 - type: nauc_mrr_at_3_diff1 value: 21.550470686773544 - type: nauc_mrr_at_3_max value: 37.923958816756326 - type: nauc_mrr_at_3_std value: 8.621492240346312 - type: nauc_mrr_at_5_diff1 value: 21.456166538305947 - type: nauc_mrr_at_5_max value: 37.43457454088489 - type: nauc_mrr_at_5_std value: 8.700414540783243 - type: nauc_ndcg_at_1000_diff1 value: 20.687215881405972 - type: nauc_ndcg_at_1000_max value: 37.07894735512217 - type: nauc_ndcg_at_1000_std value: 7.955407995753924 - type: nauc_ndcg_at_100_diff1 value: 19.899521579418327 - type: nauc_ndcg_at_100_max value: 36.7917656290794 - type: nauc_ndcg_at_100_std value: 8.815031884683332 - type: nauc_ndcg_at_10_diff1 value: 20.59162274283143 - type: nauc_ndcg_at_10_max value: 36.21724206012231 - type: nauc_ndcg_at_10_std value: 6.304038242601423 - type: nauc_ndcg_at_1_diff1 value: 24.15510508158492 - type: nauc_ndcg_at_1_max value: 38.16775944594129 - type: nauc_ndcg_at_1_std value: 9.048540715004725 - type: nauc_ndcg_at_20_diff1 value: 20.559881435708498 - type: nauc_ndcg_at_20_max value: 37.30561288551007 - type: nauc_ndcg_at_20_std value: 8.41336133183866 - type: nauc_ndcg_at_3_diff1 value: 21.41525059572959 - type: nauc_ndcg_at_3_max value: 36.22187375662584 - type: nauc_ndcg_at_3_std value: 4.765376474543177 - type: nauc_ndcg_at_5_diff1 value: 21.423152693078066 - type: nauc_ndcg_at_5_max value: 35.62314257936377 - type: nauc_ndcg_at_5_std value: 5.292965569254064 - type: nauc_precision_at_1000_diff1 value: -1.4052165408318342 - type: nauc_precision_at_1000_max value: 20.517983706015833 - type: nauc_precision_at_1000_std value: 5.528186523222815 - type: nauc_precision_at_100_diff1 value: 1.1467871741865474 - type: nauc_precision_at_100_max value: 27.33879537667429 - type: nauc_precision_at_100_std value: 10.558834828955572 - type: nauc_precision_at_10_diff1 value: 9.212502568904506 - type: nauc_precision_at_10_max value: 41.282116927731224 - type: nauc_precision_at_10_std value: 10.635469423053907 - type: nauc_precision_at_1_diff1 value: 24.15510508158492 - type: nauc_precision_at_1_max value: 38.16775944594129 - type: nauc_precision_at_1_std value: 9.048540715004725 - type: nauc_precision_at_20_diff1 value: 7.720793691116737 - type: nauc_precision_at_20_max value: 38.29599512358724 - type: nauc_precision_at_20_std value: 11.891637673436277 - type: nauc_precision_at_3_diff1 value: 13.834429473708909 - type: nauc_precision_at_3_max value: 43.42560369973884 - type: nauc_precision_at_3_std value: 7.177658330615482 - type: nauc_precision_at_5_diff1 value: 11.832909517425021 - type: nauc_precision_at_5_max value: 43.33773335597342 - type: nauc_precision_at_5_std value: 9.202718262478298 - type: nauc_recall_at_1000_diff1 value: -39.28871527051969 - type: nauc_recall_at_1000_max value: -26.37396882156915 - type: nauc_recall_at_1000_std value: 21.2304966569008 - type: nauc_recall_at_100_diff1 value: 7.038804971203592 - type: nauc_recall_at_100_max value: 27.81013426649376 - type: nauc_recall_at_100_std value: 19.55818251557495 - type: nauc_recall_at_10_diff1 value: 16.372446751980963 - type: nauc_recall_at_10_max value: 29.383411324814325 - type: nauc_recall_at_10_std value: 4.030124533628888 - type: nauc_recall_at_1_diff1 value: 27.996481625958005 - type: nauc_recall_at_1_max value: 22.170519197242168 - type: nauc_recall_at_1_std value: 3.2754228003309094 - type: nauc_recall_at_20_diff1 value: 15.454183279031705 - type: nauc_recall_at_20_max value: 32.477787087697465 - type: nauc_recall_at_20_std value: 11.814027056017764 - type: nauc_recall_at_3_diff1 value: 20.116262186422457 - type: nauc_recall_at_3_max value: 28.490372254786017 - type: nauc_recall_at_3_std value: 0.8537023418579993 - type: nauc_recall_at_5_diff1 value: 18.369329264777864 - type: nauc_recall_at_5_max value: 29.74100621863871 - type: nauc_recall_at_5_std value: 2.713328963890598 - type: ndcg_at_1 value: 26.035000000000004 - type: ndcg_at_10 value: 34.148 - type: ndcg_at_100 value: 44.027 - type: ndcg_at_1000 value: 46.544999999999995 - type: ndcg_at_20 value: 38.395 - type: ndcg_at_3 value: 28.161 - type: ndcg_at_5 value: 30.062 - type: precision_at_1 value: 26.035000000000004 - type: precision_at_10 value: 10.414 - type: precision_at_100 value: 1.8929999999999998 - type: precision_at_1000 value: 0.22300000000000003 - type: precision_at_20 value: 6.722 - type: precision_at_3 value: 20.427 - type: precision_at_5 value: 16.101 - type: recall_at_1 value: 13.142000000000001 - type: recall_at_10 value: 44.751000000000005 - type: recall_at_100 value: 82.86200000000001 - type: recall_at_1000 value: 99.381 - type: recall_at_20 value: 58.29600000000001 - type: recall_at_3 value: 27.262999999999998 - type: recall_at_5 value: 34.795 task: type: Retrieval - dataset: config: fra-eng name: MTEB XPQARetrieval (fra-eng) revision: c99d599f0a6ab9b85b065da6f9d94f9cf731679f split: test type: jinaai/xpqa metrics: - type: main_score value: 51.237 - type: map_at_1 value: 26.733 - type: map_at_10 value: 44.643 - type: map_at_100 value: 46.404 - type: map_at_1000 value: 46.516000000000005 - type: map_at_20 value: 45.659 - type: map_at_3 value: 39.358 - type: map_at_5 value: 42.784 - type: mrr_at_1 value: 43.65821094793058 - type: mrr_at_10 value: 53.10609913747425 - type: mrr_at_100 value: 53.857579329641105 - type: mrr_at_1000 value: 53.89660252126757 - type: mrr_at_20 value: 53.52067135248109 - type: mrr_at_3 value: 50.71206052514461 - type: mrr_at_5 value: 52.167334223408936 - type: nauc_map_at_1000_diff1 value: 45.24362379618829 - type: nauc_map_at_1000_max value: 44.81176515141581 - type: nauc_map_at_1000_std value: -0.8010490746117229 - type: nauc_map_at_100_diff1 value: 45.2119434987754 - type: nauc_map_at_100_max value: 44.80914165276905 - type: nauc_map_at_100_std value: -0.7698807481565286 - type: nauc_map_at_10_diff1 value: 45.15860057920748 - type: nauc_map_at_10_max value: 44.59910057286427 - type: nauc_map_at_10_std value: -1.112850247343371 - type: nauc_map_at_1_diff1 value: 52.51086259511314 - type: nauc_map_at_1_max value: 31.357215864716526 - type: nauc_map_at_1_std value: -3.3316393174260495 - type: nauc_map_at_20_diff1 value: 45.0313164103322 - type: nauc_map_at_20_max value: 44.807072544620915 - type: nauc_map_at_20_std value: -0.9759739768751601 - type: nauc_map_at_3_diff1 value: 46.607279248808844 - type: nauc_map_at_3_max value: 40.93560787464522 - type: nauc_map_at_3_std value: -3.4220516437046884 - type: nauc_map_at_5_diff1 value: 45.483519829247705 - type: nauc_map_at_5_max value: 43.47963235378117 - type: nauc_map_at_5_std value: -2.1213782659613245 - type: nauc_mrr_at_1000_diff1 value: 49.84990195876864 - type: nauc_mrr_at_1000_max value: 47.66319451546816 - type: nauc_mrr_at_1000_std value: -0.43985061175300905 - type: nauc_mrr_at_100_diff1 value: 49.83955797982952 - type: nauc_mrr_at_100_max value: 47.66572696957632 - type: nauc_mrr_at_100_std value: -0.4121021416689912 - type: nauc_mrr_at_10_diff1 value: 49.709689784147315 - type: nauc_mrr_at_10_max value: 47.695007048763955 - type: nauc_mrr_at_10_std value: -0.4323849817636193 - type: nauc_mrr_at_1_diff1 value: 52.630796934345135 - type: nauc_mrr_at_1_max value: 47.857589062201896 - type: nauc_mrr_at_1_std value: -1.8788772946768062 - type: nauc_mrr_at_20_diff1 value: 49.62435690034187 - type: nauc_mrr_at_20_max value: 47.672431418226516 - type: nauc_mrr_at_20_std value: -0.3853547894450724 - type: nauc_mrr_at_3_diff1 value: 50.42320496620572 - type: nauc_mrr_at_3_max value: 47.32934444593811 - type: nauc_mrr_at_3_std value: -1.5796513158364918 - type: nauc_mrr_at_5_diff1 value: 50.156844538277454 - type: nauc_mrr_at_5_max value: 47.500134983097176 - type: nauc_mrr_at_5_std value: -1.2044529262681698 - type: nauc_ndcg_at_1000_diff1 value: 45.83174871455657 - type: nauc_ndcg_at_1000_max value: 46.22325612922571 - type: nauc_ndcg_at_1000_std value: 0.7251836382749244 - type: nauc_ndcg_at_100_diff1 value: 45.26328676530337 - type: nauc_ndcg_at_100_max value: 46.32944733493364 - type: nauc_ndcg_at_100_std value: 1.865625075918929 - type: nauc_ndcg_at_10_diff1 value: 44.537492108596425 - type: nauc_ndcg_at_10_max value: 45.87478673164907 - type: nauc_ndcg_at_10_std value: 0.41953467722933185 - type: nauc_ndcg_at_1_diff1 value: 52.630796934345135 - type: nauc_ndcg_at_1_max value: 47.857589062201896 - type: nauc_ndcg_at_1_std value: -1.8788772946768062 - type: nauc_ndcg_at_20_diff1 value: 44.05058836339881 - type: nauc_ndcg_at_20_max value: 46.428186099703915 - type: nauc_ndcg_at_20_std value: 0.9443804586404068 - type: nauc_ndcg_at_3_diff1 value: 45.53948737812581 - type: nauc_ndcg_at_3_max value: 44.55019513996879 - type: nauc_ndcg_at_3_std value: -3.109851951092217 - type: nauc_ndcg_at_5_diff1 value: 45.41173383961928 - type: nauc_ndcg_at_5_max value: 44.560765413275305 - type: nauc_ndcg_at_5_std value: -1.6824415034098137 - type: nauc_precision_at_1000_diff1 value: -7.887083238422578 - type: nauc_precision_at_1000_max value: 12.124801121563747 - type: nauc_precision_at_1000_std value: 4.673924336255192 - type: nauc_precision_at_100_diff1 value: -2.1326427402923387 - type: nauc_precision_at_100_max value: 19.727068109516082 - type: nauc_precision_at_100_std value: 7.98949733333232 - type: nauc_precision_at_10_diff1 value: 9.723989241310061 - type: nauc_precision_at_10_max value: 36.63938764576557 - type: nauc_precision_at_10_std value: 5.237699304811818 - type: nauc_precision_at_1_diff1 value: 52.630796934345135 - type: nauc_precision_at_1_max value: 47.857589062201896 - type: nauc_precision_at_1_std value: -1.8788772946768062 - type: nauc_precision_at_20_diff1 value: 4.81172201728637 - type: nauc_precision_at_20_max value: 31.803554548820266 - type: nauc_precision_at_20_std value: 5.726820012959347 - type: nauc_precision_at_3_diff1 value: 21.35717778539221 - type: nauc_precision_at_3_max value: 40.745356004203884 - type: nauc_precision_at_3_std value: -1.1539362625545309 - type: nauc_precision_at_5_diff1 value: 15.320566766870824 - type: nauc_precision_at_5_max value: 39.57305927905734 - type: nauc_precision_at_5_std value: 2.0380912770385855 - type: nauc_recall_at_1000_diff1 value: 53.58291277355397 - type: nauc_recall_at_1000_max value: 54.274398762111744 - type: nauc_recall_at_1000_std value: 11.337487733101206 - type: nauc_recall_at_100_diff1 value: 32.3382519427821 - type: nauc_recall_at_100_max value: 43.398292158243606 - type: nauc_recall_at_100_std value: 20.104107748951193 - type: nauc_recall_at_10_diff1 value: 34.89991099982832 - type: nauc_recall_at_10_max value: 42.306920166018436 - type: nauc_recall_at_10_std value: 4.39688933264661 - type: nauc_recall_at_1_diff1 value: 52.51086259511314 - type: nauc_recall_at_1_max value: 31.357215864716526 - type: nauc_recall_at_1_std value: -3.3316393174260495 - type: nauc_recall_at_20_diff1 value: 31.389778218413884 - type: nauc_recall_at_20_max value: 43.91189150158487 - type: nauc_recall_at_20_std value: 6.823493011161174 - type: nauc_recall_at_3_diff1 value: 41.41055013171579 - type: nauc_recall_at_3_max value: 37.636926573569205 - type: nauc_recall_at_3_std value: -4.148275870225593 - type: nauc_recall_at_5_diff1 value: 38.28078114262306 - type: nauc_recall_at_5_max value: 40.03724369846101 - type: nauc_recall_at_5_std value: -1.5995298392319341 - type: ndcg_at_1 value: 43.658 - type: ndcg_at_10 value: 51.237 - type: ndcg_at_100 value: 57.658 - type: ndcg_at_1000 value: 59.628 - type: ndcg_at_20 value: 53.848 - type: ndcg_at_3 value: 46.082 - type: ndcg_at_5 value: 48.055 - type: precision_at_1 value: 43.658 - type: precision_at_10 value: 12.537 - type: precision_at_100 value: 1.806 - type: precision_at_1000 value: 0.20600000000000002 - type: precision_at_20 value: 7.19 - type: precision_at_3 value: 29.372 - type: precision_at_5 value: 21.389 - type: recall_at_1 value: 26.733 - type: recall_at_10 value: 61.248000000000005 - type: recall_at_100 value: 86.37700000000001 - type: recall_at_1000 value: 99.42999999999999 - type: recall_at_20 value: 69.686 - type: recall_at_3 value: 45.343 - type: recall_at_5 value: 53.516 task: type: Retrieval - dataset: config: default name: MTEB AllegroReviews revision: b89853e6de927b0e3bfa8ecc0e56fe4e02ceafc6 split: test type: PL-MTEB/allegro-reviews metrics: - type: accuracy value: 53.359840954274354 - type: f1 value: 41.06302546252184 - type: f1_weighted value: 49.87243147330794 - type: main_score value: 53.359840954274354 task: type: Classification - dataset: config: default name: MTEB ArguAna-PL revision: 63fc86750af76253e8c760fc9e534bbf24d260a2 split: test type: clarin-knext/arguana-pl metrics: - type: main_score value: 48.403 - type: map_at_1 value: 23.471 - type: map_at_10 value: 39.216 - type: map_at_100 value: 40.29 - type: map_at_1000 value: 40.300000000000004 - type: map_at_20 value: 40.042 - type: map_at_3 value: 33.95 - type: map_at_5 value: 36.955 - type: mrr_at_1 value: 23.75533428165007 - type: mrr_at_10 value: 39.335613809297975 - type: mrr_at_100 value: 40.39627181086039 - type: mrr_at_1000 value: 40.40599566225043 - type: mrr_at_20 value: 40.14809098074531 - type: mrr_at_3 value: 34.03271692745371 - type: mrr_at_5 value: 37.11593172119479 - type: nauc_map_at_1000_diff1 value: 5.3772331730748855 - type: nauc_map_at_1000_max value: -2.0329411151791534 - type: nauc_map_at_1000_std value: -3.86511182416739 - type: nauc_map_at_100_diff1 value: 5.3794850605444156 - type: nauc_map_at_100_max value: -2.0168450480018345 - type: nauc_map_at_100_std value: -3.842530148996824 - type: nauc_map_at_10_diff1 value: 4.9964237351426855 - type: nauc_map_at_10_max value: -2.144884912174717 - type: nauc_map_at_10_std value: -4.461427392019594 - type: nauc_map_at_1_diff1 value: 9.516688551341105 - type: nauc_map_at_1_max value: -5.644910020587747 - type: nauc_map_at_1_std value: -3.6198980121175817 - type: nauc_map_at_20_diff1 value: 5.353229630043072 - type: nauc_map_at_20_max value: -1.800952542288357 - type: nauc_map_at_20_std value: -3.906531793826726 - type: nauc_map_at_3_diff1 value: 5.255232382713057 - type: nauc_map_at_3_max value: -3.0281799752800045 - type: nauc_map_at_3_std value: -4.075526006428358 - type: nauc_map_at_5_diff1 value: 4.734362539624116 - type: nauc_map_at_5_max value: -2.503481319824773 - type: nauc_map_at_5_std value: -3.872895860744626 - type: nauc_mrr_at_1000_diff1 value: 4.4146985529959215 - type: nauc_mrr_at_1000_max value: -2.299598690474009 - type: nauc_mrr_at_1000_std value: -3.9402783361244085 - type: nauc_mrr_at_100_diff1 value: 4.417258165352108 - type: nauc_mrr_at_100_max value: -2.283454878974528 - type: nauc_mrr_at_100_std value: -3.9177269448785594 - type: nauc_mrr_at_10_diff1 value: 4.03684960733421 - type: nauc_mrr_at_10_max value: -2.396241516141588 - type: nauc_mrr_at_10_std value: -4.556329903468349 - type: nauc_mrr_at_1_diff1 value: 8.550764063204086 - type: nauc_mrr_at_1_max value: -5.2653218814611655 - type: nauc_mrr_at_1_std value: -3.7478078638245558 - type: nauc_mrr_at_20_diff1 value: 4.399100178540589 - type: nauc_mrr_at_20_max value: -2.065687810815654 - type: nauc_mrr_at_20_std value: -3.980856869079938 - type: nauc_mrr_at_3_diff1 value: 4.356955681808423 - type: nauc_mrr_at_3_max value: -3.305266169391567 - type: nauc_mrr_at_3_std value: -4.187435812922289 - type: nauc_mrr_at_5_diff1 value: 3.797470912759614 - type: nauc_mrr_at_5_max value: -2.7476549709181355 - type: nauc_mrr_at_5_std value: -3.9744581781815724 - type: nauc_ndcg_at_1000_diff1 value: 5.138474520701041 - type: nauc_ndcg_at_1000_max value: -0.8548037167905302 - type: nauc_ndcg_at_1000_std value: -3.4098747932546734 - type: nauc_ndcg_at_100_diff1 value: 5.264678453330276 - type: nauc_ndcg_at_100_max value: -0.39210055156714196 - type: nauc_ndcg_at_100_std value: -2.790588063731217 - type: nauc_ndcg_at_10_diff1 value: 3.7503459140558943 - type: nauc_ndcg_at_10_max value: -0.3384766390247899 - type: nauc_ndcg_at_10_std value: -5.575339046932562 - type: nauc_ndcg_at_1_diff1 value: 9.516688551341105 - type: nauc_ndcg_at_1_max value: -5.644910020587747 - type: nauc_ndcg_at_1_std value: -3.6198980121175817 - type: nauc_ndcg_at_20_diff1 value: 5.270936471430413 - type: nauc_ndcg_at_20_max value: 1.291103316253361 - type: nauc_ndcg_at_20_std value: -3.169503489721584 - type: nauc_ndcg_at_3_diff1 value: 4.271651754613586 - type: nauc_ndcg_at_3_max value: -2.353280506415016 - type: nauc_ndcg_at_3_std value: -4.364225258109044 - type: nauc_ndcg_at_5_diff1 value: 3.3040691006732765 - type: nauc_ndcg_at_5_max value: -1.303365264172454 - type: nauc_ndcg_at_5_std value: -3.955958872876751 - type: nauc_precision_at_1000_diff1 value: 13.241505124398486 - type: nauc_precision_at_1000_max value: 47.97010130202494 - type: nauc_precision_at_1000_std value: 74.8878850565087 - type: nauc_precision_at_100_diff1 value: 19.347701423046445 - type: nauc_precision_at_100_max value: 50.96543811338734 - type: nauc_precision_at_100_std value: 67.94488954909981 - type: nauc_precision_at_10_diff1 value: -1.5877710682220694 - type: nauc_precision_at_10_max value: 8.339455759028867 - type: nauc_precision_at_10_std value: -11.521446364855805 - type: nauc_precision_at_1_diff1 value: 9.516688551341105 - type: nauc_precision_at_1_max value: -5.644910020587747 - type: nauc_precision_at_1_std value: -3.6198980121175817 - type: nauc_precision_at_20_diff1 value: 9.056182158091447 - type: nauc_precision_at_20_max value: 32.304791476693964 - type: nauc_precision_at_20_std value: 7.405945565001408 - type: nauc_precision_at_3_diff1 value: 1.678810658886418 - type: nauc_precision_at_3_max value: -0.5473747492361274 - type: nauc_precision_at_3_std value: -5.194579239515215 - type: nauc_precision_at_5_diff1 value: -1.061704237231085 - type: nauc_precision_at_5_max value: 2.5051492439705387 - type: nauc_precision_at_5_std value: -4.155138551797193 - type: nauc_recall_at_1000_diff1 value: 13.241505124400325 - type: nauc_recall_at_1000_max value: 47.97010130202325 - type: nauc_recall_at_1000_std value: 74.88788505650888 - type: nauc_recall_at_100_diff1 value: 19.34770142304624 - type: nauc_recall_at_100_max value: 50.9654381133864 - type: nauc_recall_at_100_std value: 67.9448895490988 - type: nauc_recall_at_10_diff1 value: -1.587771068221971 - type: nauc_recall_at_10_max value: 8.339455759028958 - type: nauc_recall_at_10_std value: -11.521446364855827 - type: nauc_recall_at_1_diff1 value: 9.516688551341105 - type: nauc_recall_at_1_max value: -5.644910020587747 - type: nauc_recall_at_1_std value: -3.6198980121175817 - type: nauc_recall_at_20_diff1 value: 9.056182158091504 - type: nauc_recall_at_20_max value: 32.30479147669391 - type: nauc_recall_at_20_std value: 7.405945565001497 - type: nauc_recall_at_3_diff1 value: 1.6788106588864458 - type: nauc_recall_at_3_max value: -0.5473747492360812 - type: nauc_recall_at_3_std value: -5.194579239515186 - type: nauc_recall_at_5_diff1 value: -1.0617042372311596 - type: nauc_recall_at_5_max value: 2.505149243970507 - type: nauc_recall_at_5_std value: -4.155138551797231 - type: ndcg_at_1 value: 23.471 - type: ndcg_at_10 value: 48.403 - type: ndcg_at_100 value: 52.979 - type: ndcg_at_1000 value: 53.198 - type: ndcg_at_20 value: 51.349000000000004 - type: ndcg_at_3 value: 37.546 - type: ndcg_at_5 value: 43.009 - type: precision_at_1 value: 23.471 - type: precision_at_10 value: 7.788 - type: precision_at_100 value: 0.979 - type: precision_at_1000 value: 0.1 - type: precision_at_20 value: 4.47 - type: precision_at_3 value: 16.003 - type: precision_at_5 value: 12.276 - type: recall_at_1 value: 23.471 - type: recall_at_10 value: 77.881 - type: recall_at_100 value: 97.866 - type: recall_at_1000 value: 99.502 - type: recall_at_20 value: 89.403 - type: recall_at_3 value: 48.009 - type: recall_at_5 value: 61.38 task: type: Retrieval - dataset: config: default name: MTEB CBD revision: 36ddb419bcffe6a5374c3891957912892916f28d split: test type: PL-MTEB/cbd metrics: - type: accuracy value: 64.36999999999999 - type: ap value: 20.466523167809452 - type: ap_weighted value: 20.466523167809452 - type: f1 value: 54.572432817010075 - type: f1_weighted value: 68.94308188177826 - type: main_score value: 64.36999999999999 task: type: Classification - dataset: config: default name: MTEB CDSC-E revision: 0a3d4aa409b22f80eb22cbf59b492637637b536d split: test type: PL-MTEB/cdsce-pairclassification metrics: - type: cosine_accuracy value: 87.1 - type: cosine_accuracy_threshold value: 98.31513166427612 - type: cosine_ap value: 67.39921410443071 - type: cosine_f1 value: 61.408450704225345 - type: cosine_f1_threshold value: 96.4431643486023 - type: cosine_precision value: 66.06060606060606 - type: cosine_recall value: 57.36842105263158 - type: dot_accuracy value: 87.1 - type: dot_accuracy_threshold value: 98.31513166427612 - type: dot_ap value: 67.39921410443071 - type: dot_f1 value: 61.408450704225345 - type: dot_f1_threshold value: 96.4431643486023 - type: dot_precision value: 66.06060606060606 - type: dot_recall value: 57.36842105263158 - type: euclidean_accuracy value: 87.1 - type: euclidean_accuracy_threshold value: 18.356148898601532 - type: euclidean_ap value: 67.39921410443071 - type: euclidean_f1 value: 61.408450704225345 - type: euclidean_f1_threshold value: 26.67142152786255 - type: euclidean_precision value: 66.06060606060606 - type: euclidean_recall value: 57.36842105263158 - type: main_score value: 67.67003917913883 - type: manhattan_accuracy value: 87.4 - type: manhattan_accuracy_threshold value: 460.4861259460449 - type: manhattan_ap value: 67.67003917913883 - type: manhattan_f1 value: 61.32596685082873 - type: manhattan_f1_threshold value: 640.9262657165527 - type: manhattan_precision value: 64.53488372093024 - type: manhattan_recall value: 58.42105263157895 - type: max_ap value: 67.67003917913883 - type: max_f1 value: 61.408450704225345 - type: max_precision value: 66.06060606060606 - type: max_recall value: 58.42105263157895 - type: similarity_accuracy value: 87.1 - type: similarity_accuracy_threshold value: 98.31513166427612 - type: similarity_ap value: 67.39921410443071 - type: similarity_f1 value: 61.408450704225345 - type: similarity_f1_threshold value: 96.4431643486023 - type: similarity_precision value: 66.06060606060606 - type: similarity_recall value: 57.36842105263158 task: type: PairClassification - dataset: config: default name: MTEB CDSC-R revision: 1cd6abbb00df7d14be3dbd76a7dcc64b3a79a7cd split: test type: PL-MTEB/cdscr-sts metrics: - type: cosine_pearson value: 89.215804614177 - type: cosine_spearman value: 89.54723293152598 - type: euclidean_pearson value: 87.48591094825007 - type: euclidean_spearman value: 89.54723293152598 - type: main_score value: 89.54723293152598 - type: manhattan_pearson value: 87.53292271661934 - type: manhattan_spearman value: 89.59087505602018 - type: pearson value: 89.215804614177 - type: spearman value: 89.54723293152598 task: type: STS - dataset: config: default name: MTEB DBPedia-PL revision: 76afe41d9af165cc40999fcaa92312b8b012064a split: test type: clarin-knext/dbpedia-pl metrics: - type: main_score value: 28.798000000000002 - type: map_at_1 value: 5.805 - type: map_at_10 value: 13.104 - type: map_at_100 value: 17.64 - type: map_at_1000 value: 18.817 - type: map_at_20 value: 14.704999999999998 - type: map_at_3 value: 9.754 - type: map_at_5 value: 11.306 - type: mrr_at_1 value: 48.25 - type: mrr_at_10 value: 59.20525793650796 - type: mrr_at_100 value: 59.68250991989186 - type: mrr_at_1000 value: 59.699200384172535 - type: mrr_at_20 value: 59.47831275084566 - type: mrr_at_3 value: 57.08333333333335 - type: mrr_at_5 value: 58.508333333333354 - type: nauc_map_at_1000_diff1 value: 23.8795526911696 - type: nauc_map_at_1000_max value: 19.033721390433954 - type: nauc_map_at_1000_std value: 31.272197919163798 - type: nauc_map_at_100_diff1 value: 24.30632088878799 - type: nauc_map_at_100_max value: 16.121272162595638 - type: nauc_map_at_100_std value: 27.849470534570877 - type: nauc_map_at_10_diff1 value: 32.383097332850866 - type: nauc_map_at_10_max value: 9.425444212646546 - type: nauc_map_at_10_std value: 12.271778849795199 - type: nauc_map_at_1_diff1 value: 49.09320405513435 - type: nauc_map_at_1_max value: 6.655982988088829 - type: nauc_map_at_1_std value: 3.7535182547407584 - type: nauc_map_at_20_diff1 value: 29.41816510506116 - type: nauc_map_at_20_max value: 11.144536343130254 - type: nauc_map_at_20_std value: 17.682154597592028 - type: nauc_map_at_3_diff1 value: 38.301616834556526 - type: nauc_map_at_3_max value: 2.3072337902833864 - type: nauc_map_at_3_std value: 2.6886051009617775 - type: nauc_map_at_5_diff1 value: 35.2269306057937 - type: nauc_map_at_5_max value: 5.802047613384881 - type: nauc_map_at_5_std value: 6.566180891431155 - type: nauc_mrr_at_1000_diff1 value: 27.342268602546138 - type: nauc_mrr_at_1000_max value: 32.66979042412427 - type: nauc_mrr_at_1000_std value: 31.121503406356233 - type: nauc_mrr_at_100_diff1 value: 27.34852791425925 - type: nauc_mrr_at_100_max value: 32.691852115258435 - type: nauc_mrr_at_100_std value: 31.13220594139922 - type: nauc_mrr_at_10_diff1 value: 27.20195330398617 - type: nauc_mrr_at_10_max value: 32.548431857593854 - type: nauc_mrr_at_10_std value: 31.269333708219015 - type: nauc_mrr_at_1_diff1 value: 29.250995725745877 - type: nauc_mrr_at_1_max value: 30.020096609400724 - type: nauc_mrr_at_1_std value: 25.836889693783288 - type: nauc_mrr_at_20_diff1 value: 27.117511545890967 - type: nauc_mrr_at_20_max value: 32.46327434659793 - type: nauc_mrr_at_20_std value: 31.10670830499771 - type: nauc_mrr_at_3_diff1 value: 27.59208442559707 - type: nauc_mrr_at_3_max value: 31.462102077642008 - type: nauc_mrr_at_3_std value: 29.857593752721684 - type: nauc_mrr_at_5_diff1 value: 27.067779200957986 - type: nauc_mrr_at_5_max value: 31.928750569317742 - type: nauc_mrr_at_5_std value: 31.133046384109658 - type: nauc_ndcg_at_1000_diff1 value: 21.640223689004966 - type: nauc_ndcg_at_1000_max value: 29.383523051307193 - type: nauc_ndcg_at_1000_std value: 43.72948428979381 - type: nauc_ndcg_at_100_diff1 value: 21.818106875955774 - type: nauc_ndcg_at_100_max value: 20.989719330345487 - type: nauc_ndcg_at_100_std value: 36.00564043465215 - type: nauc_ndcg_at_10_diff1 value: 24.873244058417672 - type: nauc_ndcg_at_10_max value: 22.163824659151206 - type: nauc_ndcg_at_10_std value: 29.712807857937996 - type: nauc_ndcg_at_1_diff1 value: 29.053938551788228 - type: nauc_ndcg_at_1_max value: 24.16157362665634 - type: nauc_ndcg_at_1_std value: 22.62596071010886 - type: nauc_ndcg_at_20_diff1 value: 24.81504479286421 - type: nauc_ndcg_at_20_max value: 18.22185007322912 - type: nauc_ndcg_at_20_std value: 29.428332063288348 - type: nauc_ndcg_at_3_diff1 value: 24.599338250525282 - type: nauc_ndcg_at_3_max value: 22.6007814893909 - type: nauc_ndcg_at_3_std value: 23.475949649196757 - type: nauc_ndcg_at_5_diff1 value: 22.905714203451318 - type: nauc_ndcg_at_5_max value: 23.31577015540212 - type: nauc_ndcg_at_5_std value: 26.163489726277543 - type: nauc_precision_at_1000_diff1 value: -8.445727069307745 - type: nauc_precision_at_1000_max value: 44.7890492424445 - type: nauc_precision_at_1000_std value: 32.33459741223448 - type: nauc_precision_at_100_diff1 value: -8.838197938558475 - type: nauc_precision_at_100_max value: 32.580671613119 - type: nauc_precision_at_100_std value: 47.75612079712025 - type: nauc_precision_at_10_diff1 value: -2.220152190516044 - type: nauc_precision_at_10_max value: 34.274996226654686 - type: nauc_precision_at_10_std value: 43.3987185524864 - type: nauc_precision_at_1_diff1 value: 29.250995725745877 - type: nauc_precision_at_1_max value: 30.020096609400724 - type: nauc_precision_at_1_std value: 25.836889693783288 - type: nauc_precision_at_20_diff1 value: -7.0836121714547255 - type: nauc_precision_at_20_max value: 31.270528745365 - type: nauc_precision_at_20_std value: 46.981890317602314 - type: nauc_precision_at_3_diff1 value: 8.668995978640655 - type: nauc_precision_at_3_max value: 28.75982449586511 - type: nauc_precision_at_3_std value: 26.513283670644082 - type: nauc_precision_at_5_diff1 value: 0.3364074432097908 - type: nauc_precision_at_5_max value: 32.17732511859588 - type: nauc_precision_at_5_std value: 33.011424117906 - type: nauc_recall_at_1000_diff1 value: 12.098189411652587 - type: nauc_recall_at_1000_max value: 28.351439601322042 - type: nauc_recall_at_1000_std value: 49.74650470960806 - type: nauc_recall_at_100_diff1 value: 13.0979559604911 - type: nauc_recall_at_100_max value: 18.55326724877563 - type: nauc_recall_at_100_std value: 35.82767686480256 - type: nauc_recall_at_10_diff1 value: 26.172188943201974 - type: nauc_recall_at_10_max value: 7.164754830660332 - type: nauc_recall_at_10_std value: 12.119900158652312 - type: nauc_recall_at_1_diff1 value: 49.09320405513435 - type: nauc_recall_at_1_max value: 6.655982988088829 - type: nauc_recall_at_1_std value: 3.7535182547407584 - type: nauc_recall_at_20_diff1 value: 22.785270973288846 - type: nauc_recall_at_20_max value: 8.971627656710659 - type: nauc_recall_at_20_std value: 17.654532692121595 - type: nauc_recall_at_3_diff1 value: 32.30130185123227 - type: nauc_recall_at_3_max value: -3.703265523226075 - type: nauc_recall_at_3_std value: 1.7639006695170476 - type: nauc_recall_at_5_diff1 value: 27.464879988497863 - type: nauc_recall_at_5_max value: 1.3253246804845857 - type: nauc_recall_at_5_std value: 6.012046846778444 - type: ndcg_at_1 value: 38.375 - type: ndcg_at_10 value: 28.798000000000002 - type: ndcg_at_100 value: 32.452999999999996 - type: ndcg_at_1000 value: 39.461 - type: ndcg_at_20 value: 28.16 - type: ndcg_at_3 value: 32.816 - type: ndcg_at_5 value: 30.653999999999996 - type: precision_at_1 value: 48.25 - type: precision_at_10 value: 22.825 - type: precision_at_100 value: 7.090000000000001 - type: precision_at_1000 value: 1.651 - type: precision_at_20 value: 16.525000000000002 - type: precision_at_3 value: 36.75 - type: precision_at_5 value: 30.25 - type: recall_at_1 value: 5.805 - type: recall_at_10 value: 18.159 - type: recall_at_100 value: 38.269 - type: recall_at_1000 value: 60.692 - type: recall_at_20 value: 22.893 - type: recall_at_3 value: 11.583 - type: recall_at_5 value: 14.429 task: type: Retrieval - dataset: config: default name: MTEB 8TagsClustering revision: 78b962b130c6690659c65abf67bf1c2f030606b6 split: test type: PL-MTEB/8tags-clustering metrics: - type: main_score value: 39.24227294766999 - type: v_measure value: 39.24227294766999 - type: v_measure_std value: 1.3089498460899844 task: type: Clustering - dataset: config: default name: MTEB FiQA-PL revision: 2e535829717f8bf9dc829b7f911cc5bbd4e6608e split: test type: clarin-knext/fiqa-pl metrics: - type: main_score value: 24.288999999999998 - type: map_at_1 value: 11.272 - type: map_at_10 value: 18.348 - type: map_at_100 value: 19.66 - type: map_at_1000 value: 19.857 - type: map_at_20 value: 18.992 - type: map_at_3 value: 15.583 - type: map_at_5 value: 17.044 - type: mrr_at_1 value: 21.604938271604937 - type: mrr_at_10 value: 29.27903929061337 - type: mrr_at_100 value: 30.212764822138023 - type: mrr_at_1000 value: 30.287992850622995 - type: mrr_at_20 value: 29.74703567100871 - type: mrr_at_3 value: 26.74897119341565 - type: mrr_at_5 value: 28.091563786008244 - type: nauc_map_at_1000_diff1 value: 33.18609264749629 - type: nauc_map_at_1000_max value: 16.229216767220418 - type: nauc_map_at_1000_std value: 11.839421176873238 - type: nauc_map_at_100_diff1 value: 33.19728650890772 - type: nauc_map_at_100_max value: 16.079195779561363 - type: nauc_map_at_100_std value: 11.78063322771766 - type: nauc_map_at_10_diff1 value: 33.04853224560657 - type: nauc_map_at_10_max value: 15.37884536409968 - type: nauc_map_at_10_std value: 10.593373167105781 - type: nauc_map_at_1_diff1 value: 41.13226077955785 - type: nauc_map_at_1_max value: 12.34464571217406 - type: nauc_map_at_1_std value: 5.679576212208579 - type: nauc_map_at_20_diff1 value: 33.11097420829782 - type: nauc_map_at_20_max value: 15.682597301121387 - type: nauc_map_at_20_std value: 11.155981746119032 - type: nauc_map_at_3_diff1 value: 35.18892537241778 - type: nauc_map_at_3_max value: 14.365058887923835 - type: nauc_map_at_3_std value: 8.599922294505198 - type: nauc_map_at_5_diff1 value: 33.434953310814244 - type: nauc_map_at_5_max value: 14.97915463074864 - type: nauc_map_at_5_std value: 10.474044410042204 - type: nauc_mrr_at_1000_diff1 value: 35.39726994803205 - type: nauc_mrr_at_1000_max value: 20.90929941337352 - type: nauc_mrr_at_1000_std value: 10.291180202334353 - type: nauc_mrr_at_100_diff1 value: 35.368200822647836 - type: nauc_mrr_at_100_max value: 20.904197225349368 - type: nauc_mrr_at_100_std value: 10.270197351451367 - type: nauc_mrr_at_10_diff1 value: 35.29930377120638 - type: nauc_mrr_at_10_max value: 20.743176056640657 - type: nauc_mrr_at_10_std value: 10.204578137405102 - type: nauc_mrr_at_1_diff1 value: 41.786659564879564 - type: nauc_mrr_at_1_max value: 21.729777030326872 - type: nauc_mrr_at_1_std value: 9.106473308394927 - type: nauc_mrr_at_20_diff1 value: 35.347875836984294 - type: nauc_mrr_at_20_max value: 20.701470411877295 - type: nauc_mrr_at_20_std value: 10.098242346945426 - type: nauc_mrr_at_3_diff1 value: 36.13256745876147 - type: nauc_mrr_at_3_max value: 20.944815901489992 - type: nauc_mrr_at_3_std value: 9.543271423669903 - type: nauc_mrr_at_5_diff1 value: 35.65577306199142 - type: nauc_mrr_at_5_max value: 21.01991709681707 - type: nauc_mrr_at_5_std value: 10.215120686016858 - type: nauc_ndcg_at_1000_diff1 value: 32.03773619887771 - type: nauc_ndcg_at_1000_max value: 20.62329873286057 - type: nauc_ndcg_at_1000_std value: 15.84600327765157 - type: nauc_ndcg_at_100_diff1 value: 31.953212726228347 - type: nauc_ndcg_at_100_max value: 18.594785403398458 - type: nauc_ndcg_at_100_std value: 15.142264504355227 - type: nauc_ndcg_at_10_diff1 value: 31.7344744972378 - type: nauc_ndcg_at_10_max value: 16.61728418738611 - type: nauc_ndcg_at_10_std value: 11.839953128073597 - type: nauc_ndcg_at_1_diff1 value: 41.786659564879564 - type: nauc_ndcg_at_1_max value: 21.729777030326872 - type: nauc_ndcg_at_1_std value: 9.106473308394927 - type: nauc_ndcg_at_20_diff1 value: 31.95134068064315 - type: nauc_ndcg_at_20_max value: 16.909454974818715 - type: nauc_ndcg_at_20_std value: 12.672035929326222 - type: nauc_ndcg_at_3_diff1 value: 34.60931899005471 - type: nauc_ndcg_at_3_max value: 18.294906425122836 - type: nauc_ndcg_at_3_std value: 9.91740552963498 - type: nauc_ndcg_at_5_diff1 value: 32.67109785825235 - type: nauc_ndcg_at_5_max value: 17.56715417050353 - type: nauc_ndcg_at_5_std value: 11.716109414267391 - type: nauc_precision_at_1000_diff1 value: 6.192451799751743 - type: nauc_precision_at_1000_max value: 31.232946403916884 - type: nauc_precision_at_1000_std value: 11.32233619145531 - type: nauc_precision_at_100_diff1 value: 15.110676996819022 - type: nauc_precision_at_100_max value: 27.869316716354476 - type: nauc_precision_at_100_std value: 18.301164363871184 - type: nauc_precision_at_10_diff1 value: 21.070307270094048 - type: nauc_precision_at_10_max value: 24.22361518067922 - type: nauc_precision_at_10_std value: 14.554273388003363 - type: nauc_precision_at_1_diff1 value: 41.786659564879564 - type: nauc_precision_at_1_max value: 21.729777030326872 - type: nauc_precision_at_1_std value: 9.106473308394927 - type: nauc_precision_at_20_diff1 value: 19.816441423202775 - type: nauc_precision_at_20_max value: 24.383538204770737 - type: nauc_precision_at_20_std value: 15.199168636971377 - type: nauc_precision_at_3_diff1 value: 29.262451716999237 - type: nauc_precision_at_3_max value: 22.556259017888365 - type: nauc_precision_at_3_std value: 13.40809071872643 - type: nauc_precision_at_5_diff1 value: 23.41393741017905 - type: nauc_precision_at_5_max value: 22.510618167670824 - type: nauc_precision_at_5_std value: 16.609471253895997 - type: nauc_recall_at_1000_diff1 value: 16.69894332310247 - type: nauc_recall_at_1000_max value: 23.603524439007714 - type: nauc_recall_at_1000_std value: 37.70133731622978 - type: nauc_recall_at_100_diff1 value: 20.68782760606423 - type: nauc_recall_at_100_max value: 14.868760747917944 - type: nauc_recall_at_100_std value: 22.988507759909506 - type: nauc_recall_at_10_diff1 value: 22.897676261389755 - type: nauc_recall_at_10_max value: 11.677499286310164 - type: nauc_recall_at_10_std value: 13.136747166983081 - type: nauc_recall_at_1_diff1 value: 41.13226077955785 - type: nauc_recall_at_1_max value: 12.34464571217406 - type: nauc_recall_at_1_std value: 5.679576212208579 - type: nauc_recall_at_20_diff1 value: 23.52610305455693 - type: nauc_recall_at_20_max value: 12.126334316287 - type: nauc_recall_at_20_std value: 14.890491358767576 - type: nauc_recall_at_3_diff1 value: 29.300377418190145 - type: nauc_recall_at_3_max value: 14.2656548623994 - type: nauc_recall_at_3_std value: 10.365708612559724 - type: nauc_recall_at_5_diff1 value: 25.427037962562448 - type: nauc_recall_at_5_max value: 14.162542226396466 - type: nauc_recall_at_5_std value: 13.685656584050593 - type: ndcg_at_1 value: 21.605 - type: ndcg_at_10 value: 24.288999999999998 - type: ndcg_at_100 value: 30.218 - type: ndcg_at_1000 value: 34.414 - type: ndcg_at_20 value: 26.154 - type: ndcg_at_3 value: 20.707 - type: ndcg_at_5 value: 21.956999999999997 - type: precision_at_1 value: 21.605 - type: precision_at_10 value: 7.022 - type: precision_at_100 value: 1.2930000000000001 - type: precision_at_1000 value: 0.202 - type: precision_at_20 value: 4.267 - type: precision_at_3 value: 13.683 - type: precision_at_5 value: 10.463000000000001 - type: recall_at_1 value: 11.272 - type: recall_at_10 value: 30.721999999999998 - type: recall_at_100 value: 54.039 - type: recall_at_1000 value: 79.935 - type: recall_at_20 value: 36.506 - type: recall_at_3 value: 19.256 - type: recall_at_5 value: 24.008 task: type: Retrieval - dataset: config: default name: MTEB HotpotQA-PL revision: a0bd479ac97b4ccb5bd6ce320c415d0bb4beb907 split: test type: clarin-knext/hotpotqa-pl metrics: - type: main_score value: 59.909 - type: map_at_1 value: 35.631 - type: map_at_10 value: 50.953 - type: map_at_100 value: 51.761 - type: map_at_1000 value: 51.836000000000006 - type: map_at_20 value: 51.403 - type: map_at_3 value: 48.242000000000004 - type: map_at_5 value: 49.89 - type: mrr_at_1 value: 71.2626603646185 - type: mrr_at_10 value: 77.3888942477732 - type: mrr_at_100 value: 77.66937146141727 - type: mrr_at_1000 value: 77.68397288041564 - type: mrr_at_20 value: 77.57074315067958 - type: mrr_at_3 value: 76.25928426738675 - type: mrr_at_5 value: 77.01350438892611 - type: nauc_map_at_1000_diff1 value: 26.96764981061544 - type: nauc_map_at_1000_max value: 22.731836451302726 - type: nauc_map_at_1000_std value: 8.994156423402524 - type: nauc_map_at_100_diff1 value: 26.942497951432788 - type: nauc_map_at_100_max value: 22.71479330722431 - type: nauc_map_at_100_std value: 8.98407440009605 - type: nauc_map_at_10_diff1 value: 27.158114557540152 - type: nauc_map_at_10_max value: 22.58104861100442 - type: nauc_map_at_10_std value: 8.337888297704612 - type: nauc_map_at_1_diff1 value: 72.90723129101286 - type: nauc_map_at_1_max value: 43.326963605865224 - type: nauc_map_at_1_std value: 5.723322726536682 - type: nauc_map_at_20_diff1 value: 26.955640087566074 - type: nauc_map_at_20_max value: 22.627922508820586 - type: nauc_map_at_20_std value: 8.701393241049022 - type: nauc_map_at_3_diff1 value: 29.189900000716555 - type: nauc_map_at_3_max value: 23.347579158922738 - type: nauc_map_at_3_std value: 6.91628130204425 - type: nauc_map_at_5_diff1 value: 27.79243876992334 - type: nauc_map_at_5_max value: 22.740295287033856 - type: nauc_map_at_5_std value: 7.7561319969849665 - type: nauc_mrr_at_1000_diff1 value: 71.28682783448207 - type: nauc_mrr_at_1000_max value: 45.922411180244616 - type: nauc_mrr_at_1000_std value: 9.034961290707159 - type: nauc_mrr_at_100_diff1 value: 71.28283301258217 - type: nauc_mrr_at_100_max value: 45.9275695181984 - type: nauc_mrr_at_100_std value: 9.046986980134697 - type: nauc_mrr_at_10_diff1 value: 71.2493228565909 - type: nauc_mrr_at_10_max value: 45.93632950394277 - type: nauc_mrr_at_10_std value: 8.901287032604865 - type: nauc_mrr_at_1_diff1 value: 72.90723129101286 - type: nauc_mrr_at_1_max value: 43.326963605865224 - type: nauc_mrr_at_1_std value: 5.723322726536682 - type: nauc_mrr_at_20_diff1 value: 71.25254486378807 - type: nauc_mrr_at_20_max value: 45.946582942424556 - type: nauc_mrr_at_20_std value: 9.05239175902674 - type: nauc_mrr_at_3_diff1 value: 71.32833026076825 - type: nauc_mrr_at_3_max value: 45.995038423133785 - type: nauc_mrr_at_3_std value: 8.341868297953612 - type: nauc_mrr_at_5_diff1 value: 71.30890801509685 - type: nauc_mrr_at_5_max value: 45.97654481382387 - type: nauc_mrr_at_5_std value: 8.845360351910351 - type: nauc_ndcg_at_1000_diff1 value: 32.075295168560594 - type: nauc_ndcg_at_1000_max value: 26.856216385104375 - type: nauc_ndcg_at_1000_std value: 12.543512287347856 - type: nauc_ndcg_at_100_diff1 value: 31.464965142947115 - type: nauc_ndcg_at_100_max value: 26.502114112582163 - type: nauc_ndcg_at_100_std value: 12.425894931877595 - type: nauc_ndcg_at_10_diff1 value: 32.367658571844196 - type: nauc_ndcg_at_10_max value: 26.190984389588372 - type: nauc_ndcg_at_10_std value: 9.895477224547669 - type: nauc_ndcg_at_1_diff1 value: 72.90723129101286 - type: nauc_ndcg_at_1_max value: 43.326963605865224 - type: nauc_ndcg_at_1_std value: 5.723322726536682 - type: nauc_ndcg_at_20_diff1 value: 31.66045690140549 - type: nauc_ndcg_at_20_max value: 26.194578795851054 - type: nauc_ndcg_at_20_std value: 10.960870590823447 - type: nauc_ndcg_at_3_diff1 value: 36.05439214093881 - type: nauc_ndcg_at_3_max value: 27.72410316744339 - type: nauc_ndcg_at_3_std value: 7.715317220132202 - type: nauc_ndcg_at_5_diff1 value: 33.81723761665154 - type: nauc_ndcg_at_5_max value: 26.673920056459444 - type: nauc_ndcg_at_5_std value: 8.933439774824867 - type: nauc_precision_at_1000_diff1 value: -0.2221251083991722 - type: nauc_precision_at_1000_max value: 14.280584973386084 - type: nauc_precision_at_1000_std value: 27.7063978378772 - type: nauc_precision_at_100_diff1 value: 4.0475513079562475 - type: nauc_precision_at_100_max value: 15.147221648346376 - type: nauc_precision_at_100_std value: 22.880509039199133 - type: nauc_precision_at_10_diff1 value: 12.84890497551863 - type: nauc_precision_at_10_max value: 17.087762521613513 - type: nauc_precision_at_10_std value: 12.585305065906013 - type: nauc_precision_at_1_diff1 value: 72.90723129101286 - type: nauc_precision_at_1_max value: 43.326963605865224 - type: nauc_precision_at_1_std value: 5.723322726536682 - type: nauc_precision_at_20_diff1 value: 8.933673501828638 - type: nauc_precision_at_20_max value: 15.902286967992069 - type: nauc_precision_at_20_std value: 15.695999265224772 - type: nauc_precision_at_3_diff1 value: 22.71740911034549 - type: nauc_precision_at_3_max value: 21.870782890822923 - type: nauc_precision_at_3_std value: 8.368195274830732 - type: nauc_precision_at_5_diff1 value: 17.23292486248257 - type: nauc_precision_at_5_max value: 18.885190696734025 - type: nauc_precision_at_5_std value: 10.426694032135838 - type: nauc_recall_at_1000_diff1 value: -0.2221251083988545 - type: nauc_recall_at_1000_max value: 14.28058497338619 - type: nauc_recall_at_1000_std value: 27.706397837877343 - type: nauc_recall_at_100_diff1 value: 4.047551307956134 - type: nauc_recall_at_100_max value: 15.14722164834636 - type: nauc_recall_at_100_std value: 22.88050903919902 - type: nauc_recall_at_10_diff1 value: 12.848904975518701 - type: nauc_recall_at_10_max value: 17.087762521613545 - type: nauc_recall_at_10_std value: 12.585305065906043 - type: nauc_recall_at_1_diff1 value: 72.90723129101286 - type: nauc_recall_at_1_max value: 43.326963605865224 - type: nauc_recall_at_1_std value: 5.723322726536682 - type: nauc_recall_at_20_diff1 value: 8.933673501828618 - type: nauc_recall_at_20_max value: 15.902286967991989 - type: nauc_recall_at_20_std value: 15.695999265224723 - type: nauc_recall_at_3_diff1 value: 22.71740911034548 - type: nauc_recall_at_3_max value: 21.870782890822923 - type: nauc_recall_at_3_std value: 8.368195274830695 - type: nauc_recall_at_5_diff1 value: 17.23292486248257 - type: nauc_recall_at_5_max value: 18.88519069673408 - type: nauc_recall_at_5_std value: 10.426694032135869 - type: ndcg_at_1 value: 71.263 - type: ndcg_at_10 value: 59.909 - type: ndcg_at_100 value: 62.93299999999999 - type: ndcg_at_1000 value: 64.447 - type: ndcg_at_20 value: 61.111000000000004 - type: ndcg_at_3 value: 55.835 - type: ndcg_at_5 value: 58.062000000000005 - type: precision_at_1 value: 71.263 - type: precision_at_10 value: 12.262 - type: precision_at_100 value: 1.464 - type: precision_at_1000 value: 0.166 - type: precision_at_20 value: 6.518 - type: precision_at_3 value: 34.855000000000004 - type: precision_at_5 value: 22.668 - type: recall_at_1 value: 35.631 - type: recall_at_10 value: 61.309999999999995 - type: recall_at_100 value: 73.207 - type: recall_at_1000 value: 83.234 - type: recall_at_20 value: 65.179 - type: recall_at_3 value: 52.282 - type: recall_at_5 value: 56.67100000000001 task: type: Retrieval - dataset: config: default name: MTEB MSMARCO-PL revision: 8634c07806d5cce3a6138e260e59b81760a0a640 split: test type: clarin-knext/msmarco-pl metrics: - type: main_score value: 44.882 - type: map_at_1 value: 1.24 - type: map_at_10 value: 7.768999999999999 - type: map_at_100 value: 19.535 - type: map_at_1000 value: 24.29 - type: map_at_20 value: 11.124 - type: map_at_3 value: 3.558 - type: map_at_5 value: 5.112 - type: mrr_at_1 value: 67.44186046511628 - type: mrr_at_10 value: 75.19379844961239 - type: mrr_at_100 value: 75.82587954680979 - type: mrr_at_1000 value: 75.82820745611444 - type: mrr_at_20 value: 75.73643410852712 - type: mrr_at_3 value: 73.64341085271317 - type: mrr_at_5 value: 74.8062015503876 - type: nauc_map_at_1000_diff1 value: -21.989620907518457 - type: nauc_map_at_1000_max value: 74.21088703597528 - type: nauc_map_at_1000_std value: 77.21798713118748 - type: nauc_map_at_100_diff1 value: -15.6773928726037 - type: nauc_map_at_100_max value: 70.22001058099622 - type: nauc_map_at_100_std value: 65.22415371485596 - type: nauc_map_at_10_diff1 value: 21.960342056606727 - type: nauc_map_at_10_max value: 37.1897095260209 - type: nauc_map_at_10_std value: 18.92058870696038 - type: nauc_map_at_1_diff1 value: 15.577838306755954 - type: nauc_map_at_1_max value: 24.863043579374917 - type: nauc_map_at_1_std value: 6.592387370843499 - type: nauc_map_at_20_diff1 value: 9.576438240390903 - type: nauc_map_at_20_max value: 51.30183073674882 - type: nauc_map_at_20_std value: 35.30336831130106 - type: nauc_map_at_3_diff1 value: 15.131873334572553 - type: nauc_map_at_3_max value: 18.797768321440504 - type: nauc_map_at_3_std value: -0.13092645109257714 - type: nauc_map_at_5_diff1 value: 17.04231465807186 - type: nauc_map_at_5_max value: 21.466125385662263 - type: nauc_map_at_5_std value: 1.1575637123352447 - type: nauc_mrr_at_1000_diff1 value: -18.248968234547323 - type: nauc_mrr_at_1000_max value: 64.27602897839343 - type: nauc_mrr_at_1000_std value: 50.131100296201645 - type: nauc_mrr_at_100_diff1 value: -18.238980286255142 - type: nauc_mrr_at_100_max value: 64.27997603786041 - type: nauc_mrr_at_100_std value: 50.13517702252198 - type: nauc_mrr_at_10_diff1 value: -18.512675691928646 - type: nauc_mrr_at_10_max value: 64.09197909286284 - type: nauc_mrr_at_10_std value: 51.031732106064275 - type: nauc_mrr_at_1_diff1 value: -23.142139605578112 - type: nauc_mrr_at_1_max value: 63.19048808731884 - type: nauc_mrr_at_1_std value: 48.774238204316354 - type: nauc_mrr_at_20_diff1 value: -18.640160840586574 - type: nauc_mrr_at_20_max value: 64.430976412527 - type: nauc_mrr_at_20_std value: 50.15695556858976 - type: nauc_mrr_at_3_diff1 value: -18.390465775772828 - type: nauc_mrr_at_3_max value: 63.433292438340814 - type: nauc_mrr_at_3_std value: 47.94619902515134 - type: nauc_mrr_at_5_diff1 value: -18.54623628289419 - type: nauc_mrr_at_5_max value: 63.74074576279657 - type: nauc_mrr_at_5_std value: 48.257682525479225 - type: nauc_ndcg_at_1000_diff1 value: -13.169970965775713 - type: nauc_ndcg_at_1000_max value: 73.73523772051698 - type: nauc_ndcg_at_1000_std value: 71.45235358022637 - type: nauc_ndcg_at_100_diff1 value: -19.8120594180483 - type: nauc_ndcg_at_100_max value: 67.3274027052369 - type: nauc_ndcg_at_100_std value: 64.5956731015882 - type: nauc_ndcg_at_10_diff1 value: -6.277350735880853 - type: nauc_ndcg_at_10_max value: 60.59305682607034 - type: nauc_ndcg_at_10_std value: 49.126971101907 - type: nauc_ndcg_at_1_diff1 value: -8.720324288077949 - type: nauc_ndcg_at_1_max value: 30.501202785637542 - type: nauc_ndcg_at_1_std value: 18.293586568513305 - type: nauc_ndcg_at_20_diff1 value: -18.877995945130856 - type: nauc_ndcg_at_20_max value: 67.39555958720247 - type: nauc_ndcg_at_20_std value: 60.6279774858829 - type: nauc_ndcg_at_3_diff1 value: -5.934464965738841 - type: nauc_ndcg_at_3_max value: 47.27404035349958 - type: nauc_ndcg_at_3_std value: 26.17442248226043 - type: nauc_ndcg_at_5_diff1 value: -3.3782533914725223 - type: nauc_ndcg_at_5_max value: 50.47004778843094 - type: nauc_ndcg_at_5_std value: 28.95390387656202 - type: nauc_precision_at_1000_diff1 value: -35.81194272720232 - type: nauc_precision_at_1000_max value: 44.315527138954394 - type: nauc_precision_at_1000_std value: 63.628095344847246 - type: nauc_precision_at_100_diff1 value: -39.18958442311581 - type: nauc_precision_at_100_max value: 53.986900796770264 - type: nauc_precision_at_100_std value: 71.09932268988669 - type: nauc_precision_at_10_diff1 value: -13.984050096227286 - type: nauc_precision_at_10_max value: 65.42524229112135 - type: nauc_precision_at_10_std value: 66.20226396851875 - type: nauc_precision_at_1_diff1 value: -23.142139605578112 - type: nauc_precision_at_1_max value: 63.19048808731884 - type: nauc_precision_at_1_std value: 48.774238204316354 - type: nauc_precision_at_20_diff1 value: -30.176493123897135 - type: nauc_precision_at_20_max value: 64.06849634487335 - type: nauc_precision_at_20_std value: 69.75485681813515 - type: nauc_precision_at_3_diff1 value: -11.011413097929944 - type: nauc_precision_at_3_max value: 67.46835081745049 - type: nauc_precision_at_3_std value: 48.060582045912426 - type: nauc_precision_at_5_diff1 value: -6.405874699662373 - type: nauc_precision_at_5_max value: 62.60221401593178 - type: nauc_precision_at_5_std value: 46.921707025371575 - type: nauc_recall_at_1000_diff1 value: -8.114402563352586 - type: nauc_recall_at_1000_max value: 60.3001556664617 - type: nauc_recall_at_1000_std value: 63.39890343621265 - type: nauc_recall_at_100_diff1 value: -6.738351023868497 - type: nauc_recall_at_100_max value: 51.32812340863999 - type: nauc_recall_at_100_std value: 49.412743596311934 - type: nauc_recall_at_10_diff1 value: 20.601461386363383 - type: nauc_recall_at_10_max value: 21.60715140677045 - type: nauc_recall_at_10_std value: 6.221071995265527 - type: nauc_recall_at_1_diff1 value: 15.577838306755954 - type: nauc_recall_at_1_max value: 24.863043579374917 - type: nauc_recall_at_1_std value: 6.592387370843499 - type: nauc_recall_at_20_diff1 value: 12.800091088352024 - type: nauc_recall_at_20_max value: 35.04412507104955 - type: nauc_recall_at_20_std value: 21.904897436365363 - type: nauc_recall_at_3_diff1 value: 11.408953973523726 - type: nauc_recall_at_3_max value: 8.260360788102039 - type: nauc_recall_at_3_std value: -7.79122970880463 - type: nauc_recall_at_5_diff1 value: 14.380219584599557 - type: nauc_recall_at_5_max value: 5.975379730591909 - type: nauc_recall_at_5_std value: -12.068230649882123 - type: ndcg_at_1 value: 51.937999999999995 - type: ndcg_at_10 value: 44.882 - type: ndcg_at_100 value: 40.783 - type: ndcg_at_1000 value: 49.486999999999995 - type: ndcg_at_20 value: 42.184 - type: ndcg_at_3 value: 48.329 - type: ndcg_at_5 value: 46.571 - type: precision_at_1 value: 67.44200000000001 - type: precision_at_10 value: 53.256 - type: precision_at_100 value: 24.907 - type: precision_at_1000 value: 5.072 - type: precision_at_20 value: 45.465 - type: precision_at_3 value: 62.016000000000005 - type: precision_at_5 value: 57.67400000000001 - type: recall_at_1 value: 1.24 - type: recall_at_10 value: 9.911 - type: recall_at_100 value: 33.912 - type: recall_at_1000 value: 59.88 - type: recall_at_20 value: 15.456 - type: recall_at_3 value: 4.0329999999999995 - type: recall_at_5 value: 6.265999999999999 task: type: Retrieval - dataset: config: pl name: MTEB MassiveIntentClassification (pl) revision: 4672e20407010da34463acc759c162ca9734bca6 split: test type: mteb/amazon_massive_intent metrics: - type: accuracy value: 58.56086079354406 - type: f1 value: 51.0752914371443 - type: f1_weighted value: 57.25465842170869 - type: main_score value: 58.56086079354406 task: type: Classification - dataset: config: pl name: MTEB MassiveScenarioClassification (pl) revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 split: test type: mteb/amazon_massive_scenario metrics: - type: accuracy value: 68.79959650302622 - type: f1 value: 66.73568572153533 - type: f1_weighted value: 68.69392891616663 - type: main_score value: 68.79959650302622 task: type: Classification - dataset: config: default name: MTEB NFCorpus-PL revision: 9a6f9567fda928260afed2de480d79c98bf0bec0 split: test type: clarin-knext/nfcorpus-pl metrics: - type: main_score value: 26.700000000000003 - type: map_at_1 value: 3.7319999999999998 - type: map_at_10 value: 8.855 - type: map_at_100 value: 10.884 - type: map_at_1000 value: 11.922 - type: map_at_20 value: 9.721 - type: map_at_3 value: 6.178999999999999 - type: map_at_5 value: 7.381 - type: mrr_at_1 value: 34.984520123839005 - type: mrr_at_10 value: 43.8296722197651 - type: mrr_at_100 value: 44.530521314732724 - type: mrr_at_1000 value: 44.58203081100927 - type: mrr_at_20 value: 44.28344960852701 - type: mrr_at_3 value: 41.3312693498452 - type: mrr_at_5 value: 42.81733746130031 - type: nauc_map_at_1000_diff1 value: 22.462363087719815 - type: nauc_map_at_1000_max value: 20.296759775120016 - type: nauc_map_at_1000_std value: 30.209661157733358 - type: nauc_map_at_100_diff1 value: 23.590011889585053 - type: nauc_map_at_100_max value: 18.684278383417727 - type: nauc_map_at_100_std value: 26.82621895459225 - type: nauc_map_at_10_diff1 value: 27.229535241863704 - type: nauc_map_at_10_max value: 12.409798091370948 - type: nauc_map_at_10_std value: 18.28274894942748 - type: nauc_map_at_1_diff1 value: 43.574814034284046 - type: nauc_map_at_1_max value: 4.564434104867417 - type: nauc_map_at_1_std value: 7.935898903116461 - type: nauc_map_at_20_diff1 value: 25.32655921246531 - type: nauc_map_at_20_max value: 15.368931898009667 - type: nauc_map_at_20_std value: 21.95073617629164 - type: nauc_map_at_3_diff1 value: 33.96473904739136 - type: nauc_map_at_3_max value: 6.840960020808816 - type: nauc_map_at_3_std value: 11.193855793167565 - type: nauc_map_at_5_diff1 value: 31.386524819992022 - type: nauc_map_at_5_max value: 7.667289402351551 - type: nauc_map_at_5_std value: 12.835284913547296 - type: nauc_mrr_at_1000_diff1 value: 29.965669376994974 - type: nauc_mrr_at_1000_max value: 31.211237208155946 - type: nauc_mrr_at_1000_std value: 35.73731867290863 - type: nauc_mrr_at_100_diff1 value: 29.947754600374953 - type: nauc_mrr_at_100_max value: 31.266989755619463 - type: nauc_mrr_at_100_std value: 35.79202078225156 - type: nauc_mrr_at_10_diff1 value: 29.945702147569413 - type: nauc_mrr_at_10_max value: 30.886642789129688 - type: nauc_mrr_at_10_std value: 35.63914754058482 - type: nauc_mrr_at_1_diff1 value: 33.713470402720944 - type: nauc_mrr_at_1_max value: 28.751793677785038 - type: nauc_mrr_at_1_std value: 31.803182986705426 - type: nauc_mrr_at_20_diff1 value: 29.91366548301302 - type: nauc_mrr_at_20_max value: 31.131288856999966 - type: nauc_mrr_at_20_std value: 35.882134224012276 - type: nauc_mrr_at_3_diff1 value: 30.860896236484802 - type: nauc_mrr_at_3_max value: 29.519500974357793 - type: nauc_mrr_at_3_std value: 33.72309342817263 - type: nauc_mrr_at_5_diff1 value: 29.49229611633102 - type: nauc_mrr_at_5_max value: 29.356523818764106 - type: nauc_mrr_at_5_std value: 34.403728644234796 - type: nauc_ndcg_at_1000_diff1 value: 20.65065747027027 - type: nauc_ndcg_at_1000_max value: 38.56020828417934 - type: nauc_ndcg_at_1000_std value: 43.11278832738437 - type: nauc_ndcg_at_100_diff1 value: 21.45463562941668 - type: nauc_ndcg_at_100_max value: 30.873631958812254 - type: nauc_ndcg_at_100_std value: 39.4205527116693 - type: nauc_ndcg_at_10_diff1 value: 20.631467977678895 - type: nauc_ndcg_at_10_max value: 23.607571269154562 - type: nauc_ndcg_at_10_std value: 38.079177270694046 - type: nauc_ndcg_at_1_diff1 value: 35.543556832559204 - type: nauc_ndcg_at_1_max value: 26.4137633343316 - type: nauc_ndcg_at_1_std value: 32.09063013779238 - type: nauc_ndcg_at_20_diff1 value: 21.67953499762251 - type: nauc_ndcg_at_20_max value: 25.131819715231092 - type: nauc_ndcg_at_20_std value: 38.67052591163072 - type: nauc_ndcg_at_3_diff1 value: 26.63443060501996 - type: nauc_ndcg_at_3_max value: 22.563735867709223 - type: nauc_ndcg_at_3_std value: 34.29760134985721 - type: nauc_ndcg_at_5_diff1 value: 22.39905347041664 - type: nauc_ndcg_at_5_max value: 21.756018113774005 - type: nauc_ndcg_at_5_std value: 36.157605696574926 - type: nauc_precision_at_1000_diff1 value: -5.432303517532642 - type: nauc_precision_at_1000_max value: 9.32638219741413 - type: nauc_precision_at_1000_std value: 39.7153509814171 - type: nauc_precision_at_100_diff1 value: -2.9578929385950175 - type: nauc_precision_at_100_max value: 23.113346560742485 - type: nauc_precision_at_100_std value: 50.42833972221638 - type: nauc_precision_at_10_diff1 value: 6.4967709091740256 - type: nauc_precision_at_10_max value: 26.245637962599716 - type: nauc_precision_at_10_std value: 45.96670051969855 - type: nauc_precision_at_1_diff1 value: 33.713470402720944 - type: nauc_precision_at_1_max value: 28.751793677785038 - type: nauc_precision_at_1_std value: 31.803182986705426 - type: nauc_precision_at_20_diff1 value: 3.8421416138073177 - type: nauc_precision_at_20_max value: 26.723613496152606 - type: nauc_precision_at_20_std value: 49.38259985536451 - type: nauc_precision_at_3_diff1 value: 19.750562130084283 - type: nauc_precision_at_3_max value: 24.15782294130561 - type: nauc_precision_at_3_std value: 36.02002453018143 - type: nauc_precision_at_5_diff1 value: 12.22400568198802 - type: nauc_precision_at_5_max value: 22.64477994259073 - type: nauc_precision_at_5_std value: 40.56427415099331 - type: nauc_recall_at_1000_diff1 value: 12.448845632560946 - type: nauc_recall_at_1000_max value: 31.622286432469547 - type: nauc_recall_at_1000_std value: 26.309639034445116 - type: nauc_recall_at_100_diff1 value: 10.832581159712385 - type: nauc_recall_at_100_max value: 24.682803257928217 - type: nauc_recall_at_100_std value: 25.86711840077285 - type: nauc_recall_at_10_diff1 value: 20.971221569108366 - type: nauc_recall_at_10_max value: 12.805858961362107 - type: nauc_recall_at_10_std value: 13.633848218463635 - type: nauc_recall_at_1_diff1 value: 43.574814034284046 - type: nauc_recall_at_1_max value: 4.564434104867417 - type: nauc_recall_at_1_std value: 7.935898903116461 - type: nauc_recall_at_20_diff1 value: 17.60417964776668 - type: nauc_recall_at_20_max value: 16.600914108701208 - type: nauc_recall_at_20_std value: 18.462567374206206 - type: nauc_recall_at_3_diff1 value: 30.13359450934449 - type: nauc_recall_at_3_max value: 6.329195402219131 - type: nauc_recall_at_3_std value: 7.804247093066491 - type: nauc_recall_at_5_diff1 value: 24.03163031054558 - type: nauc_recall_at_5_max value: 5.958331833530651 - type: nauc_recall_at_5_std value: 8.356617891524207 - type: ndcg_at_1 value: 33.437 - type: ndcg_at_10 value: 26.700000000000003 - type: ndcg_at_100 value: 23.363999999999997 - type: ndcg_at_1000 value: 31.631999999999998 - type: ndcg_at_20 value: 24.275 - type: ndcg_at_3 value: 30.195 - type: ndcg_at_5 value: 28.992 - type: precision_at_1 value: 34.985 - type: precision_at_10 value: 20.433 - type: precision_at_100 value: 5.913 - type: precision_at_1000 value: 1.746 - type: precision_at_20 value: 14.443 - type: precision_at_3 value: 28.896 - type: precision_at_5 value: 25.635 - type: recall_at_1 value: 3.7319999999999998 - type: recall_at_10 value: 12.943 - type: recall_at_100 value: 24.265 - type: recall_at_1000 value: 53.25 - type: recall_at_20 value: 15.9 - type: recall_at_3 value: 7.205 - type: recall_at_5 value: 9.572 task: type: Retrieval - dataset: config: default name: MTEB NQ-PL revision: f171245712cf85dd4700b06bef18001578d0ca8d split: test type: clarin-knext/nq-pl metrics: - type: main_score value: 29.635 - type: map_at_1 value: 14.895 - type: map_at_10 value: 24.157 - type: map_at_100 value: 25.391000000000002 - type: map_at_1000 value: 25.46 - type: map_at_20 value: 24.932000000000002 - type: map_at_3 value: 20.987000000000002 - type: map_at_5 value: 22.781000000000002 - type: mrr_at_1 value: 17.033603707995365 - type: mrr_at_10 value: 26.199270264305056 - type: mrr_at_100 value: 27.25035497632659 - type: mrr_at_1000 value: 27.307077728677648 - type: mrr_at_20 value: 26.864370993179577 - type: mrr_at_3 value: 23.30050212437239 - type: mrr_at_5 value: 25.002414059482515 - type: nauc_map_at_1000_diff1 value: 26.106475555869068 - type: nauc_map_at_1000_max value: 25.461012594691052 - type: nauc_map_at_1000_std value: 17.376546142427276 - type: nauc_map_at_100_diff1 value: 26.093633746899975 - type: nauc_map_at_100_max value: 25.460594042644274 - type: nauc_map_at_100_std value: 17.362503401189493 - type: nauc_map_at_10_diff1 value: 26.186038172833605 - type: nauc_map_at_10_max value: 24.83017698848067 - type: nauc_map_at_10_std value: 16.189803634117602 - type: nauc_map_at_1_diff1 value: 29.174707786719022 - type: nauc_map_at_1_max value: 18.953264922797644 - type: nauc_map_at_1_std value: 12.17148617810572 - type: nauc_map_at_20_diff1 value: 26.189839001345916 - type: nauc_map_at_20_max value: 25.1873036250334 - type: nauc_map_at_20_std value: 16.82987531231318 - type: nauc_map_at_3_diff1 value: 26.751451062914455 - type: nauc_map_at_3_max value: 22.689487570350828 - type: nauc_map_at_3_std value: 14.472523964376913 - type: nauc_map_at_5_diff1 value: 26.466986102889255 - type: nauc_map_at_5_max value: 23.52302816080052 - type: nauc_map_at_5_std value: 15.038486281773714 - type: nauc_mrr_at_1000_diff1 value: 24.610741329750734 - type: nauc_mrr_at_1000_max value: 25.90460252341745 - type: nauc_mrr_at_1000_std value: 19.229931369441093 - type: nauc_mrr_at_100_diff1 value: 24.602966673307343 - type: nauc_mrr_at_100_max value: 25.909318923953727 - type: nauc_mrr_at_100_std value: 19.23207134491914 - type: nauc_mrr_at_10_diff1 value: 24.54192982829959 - type: nauc_mrr_at_10_max value: 25.514442888519135 - type: nauc_mrr_at_10_std value: 18.455239589874953 - type: nauc_mrr_at_1_diff1 value: 27.50641656592486 - type: nauc_mrr_at_1_max value: 20.77182038095459 - type: nauc_mrr_at_1_std value: 15.260460862822761 - type: nauc_mrr_at_20_diff1 value: 24.62172144081287 - type: nauc_mrr_at_20_max value: 25.762301912792967 - type: nauc_mrr_at_20_std value: 18.881085908672773 - type: nauc_mrr_at_3_diff1 value: 24.894701148248426 - type: nauc_mrr_at_3_max value: 23.661742208973944 - type: nauc_mrr_at_3_std value: 16.95434957782849 - type: nauc_mrr_at_5_diff1 value: 24.658005705956025 - type: nauc_mrr_at_5_max value: 24.467992442785043 - type: nauc_mrr_at_5_std value: 17.532796653975378 - type: nauc_ndcg_at_1000_diff1 value: 24.632830969868674 - type: nauc_ndcg_at_1000_max value: 30.36626493300332 - type: nauc_ndcg_at_1000_std value: 24.14333061184113 - type: nauc_ndcg_at_100_diff1 value: 24.346047050349547 - type: nauc_ndcg_at_100_max value: 30.462122958605747 - type: nauc_ndcg_at_100_std value: 24.123189028744292 - type: nauc_ndcg_at_10_diff1 value: 24.637441274115673 - type: nauc_ndcg_at_10_max value: 27.814466997081162 - type: nauc_ndcg_at_10_std value: 18.979604631236413 - type: nauc_ndcg_at_1_diff1 value: 27.50641656592486 - type: nauc_ndcg_at_1_max value: 20.77182038095459 - type: nauc_ndcg_at_1_std value: 15.260460862822761 - type: nauc_ndcg_at_20_diff1 value: 24.767111944047276 - type: nauc_ndcg_at_20_max value: 28.908654915082167 - type: nauc_ndcg_at_20_std value: 20.884645934796296 - type: nauc_ndcg_at_3_diff1 value: 25.592082856454688 - type: nauc_ndcg_at_3_max value: 23.80753222156324 - type: nauc_ndcg_at_3_std value: 15.71575559898033 - type: nauc_ndcg_at_5_diff1 value: 25.085610054435374 - type: nauc_ndcg_at_5_max value: 25.167359679118857 - type: nauc_ndcg_at_5_std value: 16.687464277549015 - type: nauc_precision_at_1000_diff1 value: 6.176538001587548 - type: nauc_precision_at_1000_max value: 33.99737260769466 - type: nauc_precision_at_1000_std value: 40.4324058481132 - type: nauc_precision_at_100_diff1 value: 10.072303037708917 - type: nauc_precision_at_100_max value: 38.36505815603056 - type: nauc_precision_at_100_std value: 41.970311663044356 - type: nauc_precision_at_10_diff1 value: 17.77691868853893 - type: nauc_precision_at_10_max value: 34.165661141650745 - type: nauc_precision_at_10_std value: 26.088809095572792 - type: nauc_precision_at_1_diff1 value: 27.50641656592486 - type: nauc_precision_at_1_max value: 20.77182038095459 - type: nauc_precision_at_1_std value: 15.260460862822761 - type: nauc_precision_at_20_diff1 value: 16.0122719270748 - type: nauc_precision_at_20_max value: 35.27584767260969 - type: nauc_precision_at_20_std value: 30.804881061720916 - type: nauc_precision_at_3_diff1 value: 21.863534491804486 - type: nauc_precision_at_3_max value: 27.673732654355643 - type: nauc_precision_at_3_std value: 19.92145910413624 - type: nauc_precision_at_5_diff1 value: 20.538193025997955 - type: nauc_precision_at_5_max value: 29.756275018772953 - type: nauc_precision_at_5_std value: 21.41349792358181 - type: nauc_recall_at_1000_diff1 value: 20.367219696261838 - type: nauc_recall_at_1000_max value: 59.62491249007197 - type: nauc_recall_at_1000_std value: 67.37971908143585 - type: nauc_recall_at_100_diff1 value: 19.448555428920216 - type: nauc_recall_at_100_max value: 48.396102452977466 - type: nauc_recall_at_100_std value: 49.77861384791871 - type: nauc_recall_at_10_diff1 value: 21.30947648461549 - type: nauc_recall_at_10_max value: 33.55182519248534 - type: nauc_recall_at_10_std value: 23.209432479779874 - type: nauc_recall_at_1_diff1 value: 29.174707786719022 - type: nauc_recall_at_1_max value: 18.953264922797644 - type: nauc_recall_at_1_std value: 12.17148617810572 - type: nauc_recall_at_20_diff1 value: 21.877824704962652 - type: nauc_recall_at_20_max value: 37.50302867285941 - type: nauc_recall_at_20_std value: 29.539665628885874 - type: nauc_recall_at_3_diff1 value: 23.671582594272646 - type: nauc_recall_at_3_max value: 25.14079018254003 - type: nauc_recall_at_3_std value: 16.40327881618881 - type: nauc_recall_at_5_diff1 value: 22.295165954167263 - type: nauc_recall_at_5_max value: 27.25439949621236 - type: nauc_recall_at_5_std value: 17.929454386934562 - type: ndcg_at_1 value: 17.034 - type: ndcg_at_10 value: 29.635 - type: ndcg_at_100 value: 35.333 - type: ndcg_at_1000 value: 37.217 - type: ndcg_at_20 value: 32.245000000000005 - type: ndcg_at_3 value: 23.443 - type: ndcg_at_5 value: 26.55 - type: precision_at_1 value: 17.034 - type: precision_at_10 value: 5.33 - type: precision_at_100 value: 0.8500000000000001 - type: precision_at_1000 value: 0.10300000000000001 - type: precision_at_20 value: 3.272 - type: precision_at_3 value: 10.989 - type: precision_at_5 value: 8.36 - type: recall_at_1 value: 14.895 - type: recall_at_10 value: 44.426 - type: recall_at_100 value: 70.09 - type: recall_at_1000 value: 84.57900000000001 - type: recall_at_20 value: 54.193000000000005 - type: recall_at_3 value: 28.283 - type: recall_at_5 value: 35.489 task: type: Retrieval - dataset: config: default name: MTEB PAC revision: fc69d1c153a8ccdcf1eef52f4e2a27f88782f543 split: test type: laugustyniak/abusive-clauses-pl metrics: - type: accuracy value: 65.0390964378801 - type: ap value: 74.13204115538645 - type: ap_weighted value: 74.13204115538645 - type: f1 value: 61.679102411303475 - type: f1_weighted value: 65.42516393570598 - type: main_score value: 65.0390964378801 task: type: Classification - dataset: config: default name: MTEB PSC revision: d05a294af9e1d3ff2bfb6b714e08a24a6cabc669 split: test type: PL-MTEB/psc-pairclassification metrics: - type: cosine_accuracy value: 98.33024118738405 - type: cosine_accuracy_threshold value: 84.68939065933228 - type: cosine_ap value: 99.56709058320435 - type: cosine_f1 value: 97.27272727272728 - type: cosine_f1_threshold value: 84.68939065933228 - type: cosine_precision value: 96.6867469879518 - type: cosine_recall value: 97.86585365853658 - type: dot_accuracy value: 98.33024118738405 - type: dot_accuracy_threshold value: 84.68939065933228 - type: dot_ap value: 99.56709058320435 - type: dot_f1 value: 97.27272727272728 - type: dot_f1_threshold value: 84.68939065933228 - type: dot_precision value: 96.6867469879518 - type: dot_recall value: 97.86585365853658 - type: euclidean_accuracy value: 98.33024118738405 - type: euclidean_accuracy_threshold value: 55.33609390258789 - type: euclidean_ap value: 99.56709058320435 - type: euclidean_f1 value: 97.27272727272728 - type: euclidean_f1_threshold value: 55.33609390258789 - type: euclidean_precision value: 96.6867469879518 - type: euclidean_recall value: 97.86585365853658 - type: main_score value: 99.58207452502563 - type: manhattan_accuracy value: 98.33024118738405 - type: manhattan_accuracy_threshold value: 1277.597713470459 - type: manhattan_ap value: 99.58207452502563 - type: manhattan_f1 value: 97.2560975609756 - type: manhattan_f1_threshold value: 1288.2164001464844 - type: manhattan_precision value: 97.2560975609756 - type: manhattan_recall value: 97.2560975609756 - type: max_ap value: 99.58207452502563 - type: max_f1 value: 97.27272727272728 - type: max_precision value: 97.2560975609756 - type: max_recall value: 97.86585365853658 - type: similarity_accuracy value: 98.33024118738405 - type: similarity_accuracy_threshold value: 84.68939065933228 - type: similarity_ap value: 99.56709058320435 - type: similarity_f1 value: 97.27272727272728 - type: similarity_f1_threshold value: 84.68939065933228 - type: similarity_precision value: 96.6867469879518 - type: similarity_recall value: 97.86585365853658 task: type: PairClassification - dataset: config: default name: MTEB PolEmo2.0-IN revision: d90724373c70959f17d2331ad51fb60c71176b03 split: test type: PL-MTEB/polemo2_in metrics: - type: accuracy value: 78.808864265928 - type: f1 value: 71.43202202178325 - type: f1_weighted value: 76.79354444796897 - type: main_score value: 78.808864265928 task: type: Classification - dataset: config: default name: MTEB PolEmo2.0-OUT revision: 6a21ab8716e255ab1867265f8b396105e8aa63d4 split: test type: PL-MTEB/polemo2_out metrics: - type: accuracy value: 62.57085020242916 - type: f1 value: 47.55601056441782 - type: f1_weighted value: 62.511982873074665 - type: main_score value: 62.57085020242916 task: type: Classification - dataset: config: default name: MTEB PPC revision: 2c7d2df57801a591f6b1e3aaf042e7a04ec7d9f2 split: test type: PL-MTEB/ppc-pairclassification metrics: - type: cosine_accuracy value: 74.0 - type: cosine_accuracy_threshold value: 94.31836009025574 - type: cosine_ap value: 86.10558522259642 - type: cosine_f1 value: 79.29936305732484 - type: cosine_f1_threshold value: 94.31836009025574 - type: cosine_precision value: 76.38036809815951 - type: cosine_recall value: 82.45033112582782 - type: dot_accuracy value: 74.0 - type: dot_accuracy_threshold value: 94.31835412979126 - type: dot_ap value: 86.10558522259642 - type: dot_f1 value: 79.29936305732484 - type: dot_f1_threshold value: 94.31835412979126 - type: dot_precision value: 76.38036809815951 - type: dot_recall value: 82.45033112582782 - type: euclidean_accuracy value: 74.0 - type: euclidean_accuracy_threshold value: 33.70946943759918 - type: euclidean_ap value: 86.10558522259642 - type: euclidean_f1 value: 79.29936305732484 - type: euclidean_f1_threshold value: 33.70946943759918 - type: euclidean_precision value: 76.38036809815951 - type: euclidean_recall value: 82.45033112582782 - type: main_score value: 86.10558522259642 - type: manhattan_accuracy value: 73.9 - type: manhattan_accuracy_threshold value: 780.9340476989746 - type: manhattan_ap value: 86.0467795030663 - type: manhattan_f1 value: 79.43485086342228 - type: manhattan_f1_threshold value: 797.8069305419922 - type: manhattan_precision value: 75.5223880597015 - type: manhattan_recall value: 83.77483443708608 - type: max_ap value: 86.10558522259642 - type: max_f1 value: 79.43485086342228 - type: max_precision value: 76.38036809815951 - type: max_recall value: 83.77483443708608 - type: similarity_accuracy value: 74.0 - type: similarity_accuracy_threshold value: 94.31836009025574 - type: similarity_ap value: 86.10558522259642 - type: similarity_f1 value: 79.29936305732484 - type: similarity_f1_threshold value: 94.31836009025574 - type: similarity_precision value: 76.38036809815951 - type: similarity_recall value: 82.45033112582782 task: type: PairClassification - dataset: config: default name: MTEB Quora-PL revision: 0be27e93455051e531182b85e85e425aba12e9d4 split: test type: clarin-knext/quora-pl metrics: - type: main_score value: 72.967 - type: map_at_1 value: 55.144000000000005 - type: map_at_10 value: 67.929 - type: map_at_100 value: 68.86 - type: map_at_1000 value: 68.901 - type: map_at_20 value: 68.516 - type: map_at_3 value: 64.83 - type: map_at_5 value: 66.676 - type: mrr_at_1 value: 63.480000000000004 - type: mrr_at_10 value: 71.84984920634886 - type: mrr_at_100 value: 72.25466310843967 - type: mrr_at_1000 value: 72.26655999913221 - type: mrr_at_20 value: 72.12276439487533 - type: mrr_at_3 value: 70.05499999999962 - type: mrr_at_5 value: 71.20399999999938 - type: nauc_map_at_1000_diff1 value: 68.24286057450547 - type: nauc_map_at_1000_max value: 25.772071887662857 - type: nauc_map_at_1000_std value: -11.038099329313008 - type: nauc_map_at_100_diff1 value: 68.23827609989507 - type: nauc_map_at_100_max value: 25.7528856894633 - type: nauc_map_at_100_std value: -11.068564365152046 - type: nauc_map_at_10_diff1 value: 68.28520996107123 - type: nauc_map_at_10_max value: 25.002003876054346 - type: nauc_map_at_10_std value: -12.335638622901788 - type: nauc_map_at_1_diff1 value: 71.75651974240864 - type: nauc_map_at_1_max value: 18.91563206148769 - type: nauc_map_at_1_std value: -15.766224337269566 - type: nauc_map_at_20_diff1 value: 68.22063759842727 - type: nauc_map_at_20_max value: 25.487015251371098 - type: nauc_map_at_20_std value: -11.562217754911307 - type: nauc_map_at_3_diff1 value: 68.73066371981375 - type: nauc_map_at_3_max value: 23.00894164024716 - type: nauc_map_at_3_std value: -14.416968038747862 - type: nauc_map_at_5_diff1 value: 68.4026048544462 - type: nauc_map_at_5_max value: 24.0635545260521 - type: nauc_map_at_5_std value: -13.317259791673012 - type: nauc_mrr_at_1000_diff1 value: 69.42328633437369 - type: nauc_mrr_at_1000_max value: 29.616854630563104 - type: nauc_mrr_at_1000_std value: -7.407262927873341 - type: nauc_mrr_at_100_diff1 value: 69.41805385692442 - type: nauc_mrr_at_100_max value: 29.622721216167434 - type: nauc_mrr_at_100_std value: -7.391041773641051 - type: nauc_mrr_at_10_diff1 value: 69.3457273244501 - type: nauc_mrr_at_10_max value: 29.65947727398705 - type: nauc_mrr_at_10_std value: -7.53792710769481 - type: nauc_mrr_at_1_diff1 value: 71.8589978314364 - type: nauc_mrr_at_1_max value: 27.26957667791739 - type: nauc_mrr_at_1_std value: -10.100927694941472 - type: nauc_mrr_at_20_diff1 value: 69.36942578766813 - type: nauc_mrr_at_20_max value: 29.660463270488385 - type: nauc_mrr_at_20_std value: -7.376877149626725 - type: nauc_mrr_at_3_diff1 value: 69.29861515799205 - type: nauc_mrr_at_3_max value: 29.33399555998474 - type: nauc_mrr_at_3_std value: -8.240273377090736 - type: nauc_mrr_at_5_diff1 value: 69.35386047017475 - type: nauc_mrr_at_5_max value: 29.584250910701815 - type: nauc_mrr_at_5_std value: -7.667942449440323 - type: nauc_ndcg_at_1000_diff1 value: 67.84970988845713 - type: nauc_ndcg_at_1000_max value: 28.61263305970346 - type: nauc_ndcg_at_1000_std value: -7.206523803073922 - type: nauc_ndcg_at_100_diff1 value: 67.70352902312702 - type: nauc_ndcg_at_100_max value: 28.66451560902271 - type: nauc_ndcg_at_100_std value: -6.851076090960979 - type: nauc_ndcg_at_10_diff1 value: 67.42015737885397 - type: nauc_ndcg_at_10_max value: 27.530444130016168 - type: nauc_ndcg_at_10_std value: -9.427985190987474 - type: nauc_ndcg_at_1_diff1 value: 71.8793742316567 - type: nauc_ndcg_at_1_max value: 27.261298450185308 - type: nauc_ndcg_at_1_std value: -10.04497075691524 - type: nauc_ndcg_at_20_diff1 value: 67.42200058537294 - type: nauc_ndcg_at_20_max value: 28.22986052404655 - type: nauc_ndcg_at_20_std value: -8.030536728985693 - type: nauc_ndcg_at_3_diff1 value: 67.471552537964 - type: nauc_ndcg_at_3_max value: 26.03277670287765 - type: nauc_ndcg_at_3_std value: -11.253279176032436 - type: nauc_ndcg_at_5_diff1 value: 67.42678714699264 - type: nauc_ndcg_at_5_max value: 26.54845879664486 - type: nauc_ndcg_at_5_std value: -10.407611106477633 - type: nauc_precision_at_1000_diff1 value: -30.32934486467821 - type: nauc_precision_at_1000_max value: 11.91523191478659 - type: nauc_precision_at_1000_std value: 31.476552592999436 - type: nauc_precision_at_100_diff1 value: -26.262105638108135 - type: nauc_precision_at_100_max value: 15.193751494600448 - type: nauc_precision_at_100_std value: 32.09374258953343 - type: nauc_precision_at_10_diff1 value: -6.682234712045421 - type: nauc_precision_at_10_max value: 20.59230495370378 - type: nauc_precision_at_10_std value: 19.563221549346537 - type: nauc_precision_at_1_diff1 value: 71.8793742316567 - type: nauc_precision_at_1_max value: 27.261298450185308 - type: nauc_precision_at_1_std value: -10.04497075691524 - type: nauc_precision_at_20_diff1 value: -16.129281421675955 - type: nauc_precision_at_20_max value: 19.013144338631392 - type: nauc_precision_at_20_std value: 26.326173134142287 - type: nauc_precision_at_3_diff1 value: 20.699978586651056 - type: nauc_precision_at_3_max value: 23.0320962309107 - type: nauc_precision_at_3_std value: 4.749967993274664 - type: nauc_precision_at_5_diff1 value: 7.046864204806423 - type: nauc_precision_at_5_max value: 22.16832739332923 - type: nauc_precision_at_5_std value: 11.909222559184276 - type: nauc_recall_at_1000_diff1 value: 52.05206576699943 - type: nauc_recall_at_1000_max value: 51.435199785253246 - type: nauc_recall_at_1000_std value: 51.60940884098615 - type: nauc_recall_at_100_diff1 value: 55.88854763934312 - type: nauc_recall_at_100_max value: 39.21155962587193 - type: nauc_recall_at_100_std value: 27.299887462517518 - type: nauc_recall_at_10_diff1 value: 60.16585088325225 - type: nauc_recall_at_10_max value: 28.663702614424896 - type: nauc_recall_at_10_std value: -6.073039592069796 - type: nauc_recall_at_1_diff1 value: 71.75651974240864 - type: nauc_recall_at_1_max value: 18.91563206148769 - type: nauc_recall_at_1_std value: -15.766224337269566 - type: nauc_recall_at_20_diff1 value: 58.35530372050582 - type: nauc_recall_at_20_max value: 32.60341340698367 - type: nauc_recall_at_20_std value: 3.321152193915959 - type: nauc_recall_at_3_diff1 value: 63.760773357483316 - type: nauc_recall_at_3_max value: 23.313883321559906 - type: nauc_recall_at_3_std value: -13.473871675341206 - type: nauc_recall_at_5_diff1 value: 62.13055752382483 - type: nauc_recall_at_5_max value: 25.401911413697775 - type: nauc_recall_at_5_std value: -10.397116573821439 - type: ndcg_at_1 value: 63.470000000000006 - type: ndcg_at_10 value: 72.967 - type: ndcg_at_100 value: 75.832 - type: ndcg_at_1000 value: 76.41799999999999 - type: ndcg_at_20 value: 74.369 - type: ndcg_at_3 value: 68.731 - type: ndcg_at_5 value: 70.93199999999999 - type: precision_at_1 value: 63.470000000000006 - type: precision_at_10 value: 11.321 - type: precision_at_100 value: 1.413 - type: precision_at_1000 value: 0.152 - type: precision_at_20 value: 6.214 - type: precision_at_3 value: 30.14 - type: precision_at_5 value: 20.182 - type: recall_at_1 value: 55.144000000000005 - type: recall_at_10 value: 83.488 - type: recall_at_100 value: 94.71000000000001 - type: recall_at_1000 value: 98.38900000000001 - type: recall_at_20 value: 88.32799999999999 - type: recall_at_3 value: 71.763 - type: recall_at_5 value: 77.519 task: type: Retrieval - dataset: config: default name: MTEB SCIDOCS-PL revision: 45452b03f05560207ef19149545f168e596c9337 split: test type: clarin-knext/scidocs-pl metrics: - type: main_score value: 13.905000000000001 - type: map_at_1 value: 3.1649999999999996 - type: map_at_10 value: 7.872999999999999 - type: map_at_100 value: 9.437 - type: map_at_1000 value: 9.701 - type: map_at_20 value: 8.685 - type: map_at_3 value: 5.793 - type: map_at_5 value: 6.755999999999999 - type: mrr_at_1 value: 15.6 - type: mrr_at_10 value: 24.506746031746005 - type: mrr_at_100 value: 25.70609394877134 - type: mrr_at_1000 value: 25.792056624320015 - type: mrr_at_20 value: 25.222491915119605 - type: mrr_at_3 value: 21.66666666666669 - type: mrr_at_5 value: 23.191666666666666 - type: nauc_map_at_1000_diff1 value: 23.18232368444192 - type: nauc_map_at_1000_max value: 30.51723263049919 - type: nauc_map_at_1000_std value: 20.20486779374314 - type: nauc_map_at_100_diff1 value: 23.080727823504276 - type: nauc_map_at_100_max value: 30.246387130436638 - type: nauc_map_at_100_std value: 19.68405276888789 - type: nauc_map_at_10_diff1 value: 24.537343409943325 - type: nauc_map_at_10_max value: 28.364069700464334 - type: nauc_map_at_10_std value: 15.13427848315133 - type: nauc_map_at_1_diff1 value: 32.39883848968994 - type: nauc_map_at_1_max value: 23.615723240461623 - type: nauc_map_at_1_std value: 11.148264932586265 - type: nauc_map_at_20_diff1 value: 23.826614277797272 - type: nauc_map_at_20_max value: 29.54306133099886 - type: nauc_map_at_20_std value: 17.408092097298262 - type: nauc_map_at_3_diff1 value: 26.89407078857185 - type: nauc_map_at_3_max value: 26.82448479758581 - type: nauc_map_at_3_std value: 11.203306435103412 - type: nauc_map_at_5_diff1 value: 24.925295727226374 - type: nauc_map_at_5_max value: 27.726964720927285 - type: nauc_map_at_5_std value: 13.165832906881105 - type: nauc_mrr_at_1000_diff1 value: 26.07952885954442 - type: nauc_mrr_at_1000_max value: 26.39620305682346 - type: nauc_mrr_at_1000_std value: 15.4691968563711 - type: nauc_mrr_at_100_diff1 value: 26.068460416892215 - type: nauc_mrr_at_100_max value: 26.380928493559804 - type: nauc_mrr_at_100_std value: 15.489233144751708 - type: nauc_mrr_at_10_diff1 value: 26.03448612594449 - type: nauc_mrr_at_10_max value: 26.364989628019757 - type: nauc_mrr_at_10_std value: 14.991318949655136 - type: nauc_mrr_at_1_diff1 value: 32.723567686923026 - type: nauc_mrr_at_1_max value: 23.607424459530364 - type: nauc_mrr_at_1_std value: 11.214989860076892 - type: nauc_mrr_at_20_diff1 value: 25.914306236878293 - type: nauc_mrr_at_20_max value: 26.254769411936184 - type: nauc_mrr_at_20_std value: 15.167029359994864 - type: nauc_mrr_at_3_diff1 value: 26.641013626531695 - type: nauc_mrr_at_3_max value: 26.588054854826503 - type: nauc_mrr_at_3_std value: 14.237519001411153 - type: nauc_mrr_at_5_diff1 value: 25.99328651624433 - type: nauc_mrr_at_5_max value: 26.13836117183896 - type: nauc_mrr_at_5_std value: 14.420365231524618 - type: nauc_ndcg_at_1000_diff1 value: 21.80668442273199 - type: nauc_ndcg_at_1000_max value: 33.34390999032878 - type: nauc_ndcg_at_1000_std value: 29.662073089048285 - type: nauc_ndcg_at_100_diff1 value: 21.072971411828306 - type: nauc_ndcg_at_100_max value: 31.488496819079593 - type: nauc_ndcg_at_100_std value: 26.73580019278837 - type: nauc_ndcg_at_10_diff1 value: 22.964364053079933 - type: nauc_ndcg_at_10_max value: 28.351440666770383 - type: nauc_ndcg_at_10_std value: 17.383681731219692 - type: nauc_ndcg_at_1_diff1 value: 32.723567686923026 - type: nauc_ndcg_at_1_max value: 23.607424459530364 - type: nauc_ndcg_at_1_std value: 11.214989860076892 - type: nauc_ndcg_at_20_diff1 value: 21.938539210894064 - type: nauc_ndcg_at_20_max value: 29.631859895086222 - type: nauc_ndcg_at_20_std value: 20.386892920365828 - type: nauc_ndcg_at_3_diff1 value: 25.69508679368568 - type: nauc_ndcg_at_3_max value: 27.346589413294353 - type: nauc_ndcg_at_3_std value: 13.27524027859251 - type: nauc_ndcg_at_5_diff1 value: 23.448042036479684 - type: nauc_ndcg_at_5_max value: 27.58741657842951 - type: nauc_ndcg_at_5_std value: 14.768121274771891 - type: nauc_precision_at_1000_diff1 value: 13.325830342851743 - type: nauc_precision_at_1000_max value: 33.12589529001219 - type: nauc_precision_at_1000_std value: 43.19509533550807 - type: nauc_precision_at_100_diff1 value: 13.283631413521633 - type: nauc_precision_at_100_max value: 30.40530489731691 - type: nauc_precision_at_100_std value: 36.67580352196424 - type: nauc_precision_at_10_diff1 value: 18.522857406921162 - type: nauc_precision_at_10_max value: 28.296802001220513 - type: nauc_precision_at_10_std value: 20.645757169009997 - type: nauc_precision_at_1_diff1 value: 32.723567686923026 - type: nauc_precision_at_1_max value: 23.607424459530364 - type: nauc_precision_at_1_std value: 11.214989860076892 - type: nauc_precision_at_20_diff1 value: 16.06141180796256 - type: nauc_precision_at_20_max value: 29.235625664181075 - type: nauc_precision_at_20_std value: 25.145690128045885 - type: nauc_precision_at_3_diff1 value: 23.24883851534181 - type: nauc_precision_at_3_max value: 28.64831854723111 - type: nauc_precision_at_3_std value: 14.372759012060602 - type: nauc_precision_at_5_diff1 value: 19.74922697183535 - type: nauc_precision_at_5_max value: 28.28618112212183 - type: nauc_precision_at_5_std value: 16.76873435538237 - type: nauc_recall_at_1000_diff1 value: 12.47335275188347 - type: nauc_recall_at_1000_max value: 33.00015584848869 - type: nauc_recall_at_1000_std value: 43.70701134051634 - type: nauc_recall_at_100_diff1 value: 12.513041153563837 - type: nauc_recall_at_100_max value: 30.273540501575823 - type: nauc_recall_at_100_std value: 36.71619580684846 - type: nauc_recall_at_10_diff1 value: 18.17624193441234 - type: nauc_recall_at_10_max value: 28.250886102195054 - type: nauc_recall_at_10_std value: 20.261194441451018 - type: nauc_recall_at_1_diff1 value: 32.39883848968994 - type: nauc_recall_at_1_max value: 23.615723240461623 - type: nauc_recall_at_1_std value: 11.148264932586265 - type: nauc_recall_at_20_diff1 value: 15.608763922874088 - type: nauc_recall_at_20_max value: 29.104046594172832 - type: nauc_recall_at_20_std value: 24.75986997225524 - type: nauc_recall_at_3_diff1 value: 22.928537699291248 - type: nauc_recall_at_3_max value: 28.72397719226873 - type: nauc_recall_at_3_std value: 14.044388052473414 - type: nauc_recall_at_5_diff1 value: 19.51285766786134 - type: nauc_recall_at_5_max value: 28.33574712082385 - type: nauc_recall_at_5_std value: 16.400242628056912 - type: ndcg_at_1 value: 15.6 - type: ndcg_at_10 value: 13.905000000000001 - type: ndcg_at_100 value: 20.485999999999997 - type: ndcg_at_1000 value: 25.72 - type: ndcg_at_20 value: 16.297 - type: ndcg_at_3 value: 13.145000000000001 - type: ndcg_at_5 value: 11.413 - type: precision_at_1 value: 15.6 - type: precision_at_10 value: 7.290000000000001 - type: precision_at_100 value: 1.67 - type: precision_at_1000 value: 0.293 - type: precision_at_20 value: 5.025 - type: precision_at_3 value: 12.4 - type: precision_at_5 value: 10.040000000000001 - type: recall_at_1 value: 3.1649999999999996 - type: recall_at_10 value: 14.798 - type: recall_at_100 value: 33.967999999999996 - type: recall_at_1000 value: 59.577999999999996 - type: recall_at_20 value: 20.403 - type: recall_at_3 value: 7.568 - type: recall_at_5 value: 10.218 task: type: Retrieval - dataset: config: default name: MTEB SICK-E-PL revision: 71bba34b0ece6c56dfcf46d9758a27f7a90f17e9 split: test type: PL-MTEB/sicke-pl-pairclassification metrics: - type: cosine_accuracy value: 77.82307378719935 - type: cosine_accuracy_threshold value: 97.00007438659668 - type: cosine_ap value: 62.95127012761914 - type: cosine_f1 value: 60.530590146183 - type: cosine_f1_threshold value: 94.09935474395752 - type: cosine_precision value: 48.82096069868996 - type: cosine_recall value: 79.62962962962963 - type: dot_accuracy value: 77.82307378719935 - type: dot_accuracy_threshold value: 97.00008034706116 - type: dot_ap value: 62.95087763566891 - type: dot_f1 value: 60.530590146183 - type: dot_f1_threshold value: 94.099360704422 - type: dot_precision value: 48.82096069868996 - type: dot_recall value: 79.62962962962963 - type: euclidean_accuracy value: 77.82307378719935 - type: euclidean_accuracy_threshold value: 24.494586884975433 - type: euclidean_ap value: 62.95127012761915 - type: euclidean_f1 value: 60.530590146183 - type: euclidean_f1_threshold value: 34.3529999256134 - type: euclidean_precision value: 48.82096069868996 - type: euclidean_recall value: 79.62962962962963 - type: main_score value: 62.95127012761915 - type: manhattan_accuracy value: 77.6600081532817 - type: manhattan_accuracy_threshold value: 569.0948009490967 - type: manhattan_ap value: 62.59417668418388 - type: manhattan_f1 value: 60.486737763193865 - type: manhattan_f1_threshold value: 803.2670974731445 - type: manhattan_precision value: 49.090102086107414 - type: manhattan_recall value: 78.77492877492878 - type: max_ap value: 62.95127012761915 - type: max_f1 value: 60.530590146183 - type: max_precision value: 49.090102086107414 - type: max_recall value: 79.62962962962963 - type: similarity_accuracy value: 77.82307378719935 - type: similarity_accuracy_threshold value: 97.00007438659668 - type: similarity_ap value: 62.95127012761914 - type: similarity_f1 value: 60.530590146183 - type: similarity_f1_threshold value: 94.09935474395752 - type: similarity_precision value: 48.82096069868996 - type: similarity_recall value: 79.62962962962963 task: type: PairClassification - dataset: config: default name: MTEB SICK-R-PL revision: fd5c2441b7eeff8676768036142af4cfa42c1339 split: test type: PL-MTEB/sickr-pl-sts metrics: - type: cosine_pearson value: 72.0298956351125 - type: cosine_spearman value: 66.17464577596678 - type: euclidean_pearson value: 68.98291370252295 - type: euclidean_spearman value: 66.17463716306638 - type: main_score value: 66.17464577596678 - type: manhattan_pearson value: 68.81382804554632 - type: manhattan_spearman value: 65.97651341583767 - type: pearson value: 72.0298956351125 - type: spearman value: 66.17464577596678 task: type: STS - dataset: config: pl name: MTEB STS22 (pl) revision: de9d86b3b84231dc21f76c7b7af1f28e2f57f6e3 split: test type: mteb/sts22-crosslingual-sts metrics: - type: cosine_pearson value: 41.189800700866094 - type: cosine_spearman value: 45.34202672993038 - type: euclidean_pearson value: 27.19960620267402 - type: euclidean_spearman value: 45.34202672993038 - type: main_score value: 45.34202672993038 - type: manhattan_pearson value: 26.949544922651196 - type: manhattan_spearman value: 45.21613651872827 - type: pearson value: 41.189800700866094 - type: spearman value: 45.34202672993038 task: type: STS - dataset: config: pl-en name: MTEB STS22 (pl-en) revision: de9d86b3b84231dc21f76c7b7af1f28e2f57f6e3 split: test type: mteb/sts22-crosslingual-sts metrics: - type: cosine_pearson value: 78.98435260035146 - type: cosine_spearman value: 79.0083835281799 - type: euclidean_pearson value: 79.05051294275746 - type: euclidean_spearman value: 79.0083835281799 - type: main_score value: 79.0083835281799 - type: manhattan_pearson value: 79.28546319449809 - type: manhattan_spearman value: 78.79790129500051 - type: pearson value: 78.98435260035146 - type: spearman value: 79.0083835281799 task: type: STS - dataset: config: de-pl name: MTEB STS22 (de-pl) revision: de9d86b3b84231dc21f76c7b7af1f28e2f57f6e3 split: test type: mteb/sts22-crosslingual-sts metrics: - type: cosine_pearson value: 41.459968983462765 - type: cosine_spearman value: 53.581740089351996 - type: euclidean_pearson value: 43.45992137124784 - type: euclidean_spearman value: 53.581740089351996 - type: main_score value: 53.581740089351996 - type: manhattan_pearson value: 43.14606772261321 - type: manhattan_spearman value: 53.181416743856836 - type: pearson value: 41.459968983462765 - type: spearman value: 53.581740089351996 task: type: STS - dataset: config: fr-pl name: MTEB STS22 (fr-pl) revision: de9d86b3b84231dc21f76c7b7af1f28e2f57f6e3 split: test type: mteb/sts22-crosslingual-sts metrics: - type: cosine_pearson value: 90.73440998579521 - type: cosine_spearman value: 84.51542547285167 - type: euclidean_pearson value: 90.27154868530998 - type: euclidean_spearman value: 84.51542547285167 - type: main_score value: 84.51542547285167 - type: manhattan_pearson value: 89.71184089391063 - type: manhattan_spearman value: 84.51542547285167 - type: pearson value: 90.73440998579521 - type: spearman value: 84.51542547285167 task: type: STS - dataset: config: default name: MTEB SciFact-PL revision: 47932a35f045ef8ed01ba82bf9ff67f6e109207e split: test type: clarin-knext/scifact-pl metrics: - type: main_score value: 60.526 - type: map_at_1 value: 46.400000000000006 - type: map_at_10 value: 55.991 - type: map_at_100 value: 56.736 - type: map_at_1000 value: 56.8 - type: map_at_20 value: 56.547000000000004 - type: map_at_3 value: 53.603 - type: map_at_5 value: 54.913000000000004 - type: mrr_at_1 value: 49.0 - type: mrr_at_10 value: 57.2739417989418 - type: mrr_at_100 value: 57.853315794381885 - type: mrr_at_1000 value: 57.91488615065038 - type: mrr_at_20 value: 57.701351990516095 - type: mrr_at_3 value: 55.44444444444445 - type: mrr_at_5 value: 56.42777777777778 - type: nauc_map_at_1000_diff1 value: 57.35372319475057 - type: nauc_map_at_1000_max value: 43.875320768732436 - type: nauc_map_at_1000_std value: 13.322022193655078 - type: nauc_map_at_100_diff1 value: 57.313891527698644 - type: nauc_map_at_100_max value: 43.872731261320574 - type: nauc_map_at_100_std value: 13.326500791745696 - type: nauc_map_at_10_diff1 value: 57.39836451111907 - type: nauc_map_at_10_max value: 43.526572478876574 - type: nauc_map_at_10_std value: 12.985890578763163 - type: nauc_map_at_1_diff1 value: 63.12715359944983 - type: nauc_map_at_1_max value: 37.86501536906817 - type: nauc_map_at_1_std value: 6.322678498743812 - type: nauc_map_at_20_diff1 value: 57.34518997447502 - type: nauc_map_at_20_max value: 43.98066371339034 - type: nauc_map_at_20_std value: 13.295989078438582 - type: nauc_map_at_3_diff1 value: 58.349640442692106 - type: nauc_map_at_3_max value: 43.61371135332669 - type: nauc_map_at_3_std value: 12.624087711546398 - type: nauc_map_at_5_diff1 value: 57.60916401452899 - type: nauc_map_at_5_max value: 43.393670573290564 - type: nauc_map_at_5_std value: 12.734291845594658 - type: nauc_mrr_at_1000_diff1 value: 56.86336995088208 - type: nauc_mrr_at_1000_max value: 44.16063126157464 - type: nauc_mrr_at_1000_std value: 14.46809986588675 - type: nauc_mrr_at_100_diff1 value: 56.82713594496301 - type: nauc_mrr_at_100_max value: 44.15947777871654 - type: nauc_mrr_at_100_std value: 14.468630477543204 - type: nauc_mrr_at_10_diff1 value: 56.75309318959002 - type: nauc_mrr_at_10_max value: 44.07955977542136 - type: nauc_mrr_at_10_std value: 14.364429809429623 - type: nauc_mrr_at_1_diff1 value: 61.55378221555377 - type: nauc_mrr_at_1_max value: 40.56404604782708 - type: nauc_mrr_at_1_std value: 11.237989014734051 - type: nauc_mrr_at_20_diff1 value: 56.820233285042356 - type: nauc_mrr_at_20_max value: 44.273215159882724 - type: nauc_mrr_at_20_std value: 14.457019833008061 - type: nauc_mrr_at_3_diff1 value: 57.25776927599811 - type: nauc_mrr_at_3_max value: 44.85862445501357 - type: nauc_mrr_at_3_std value: 15.08481264828146 - type: nauc_mrr_at_5_diff1 value: 56.77734628869654 - type: nauc_mrr_at_5_max value: 44.228014048290945 - type: nauc_mrr_at_5_std value: 14.560343015151084 - type: nauc_ndcg_at_1000_diff1 value: 56.067561048231966 - type: nauc_ndcg_at_1000_max value: 45.16507129881619 - type: nauc_ndcg_at_1000_std value: 15.507347464297675 - type: nauc_ndcg_at_100_diff1 value: 55.14390108893301 - type: nauc_ndcg_at_100_max value: 45.18506685496001 - type: nauc_ndcg_at_100_std value: 15.445626965324468 - type: nauc_ndcg_at_10_diff1 value: 55.178985380720135 - type: nauc_ndcg_at_10_max value: 44.26794162720816 - type: nauc_ndcg_at_10_std value: 13.918171530733483 - type: nauc_ndcg_at_1_diff1 value: 61.55378221555377 - type: nauc_ndcg_at_1_max value: 40.56404604782708 - type: nauc_ndcg_at_1_std value: 11.237989014734051 - type: nauc_ndcg_at_20_diff1 value: 55.27265202726197 - type: nauc_ndcg_at_20_max value: 45.87216107265961 - type: nauc_ndcg_at_20_std value: 14.919203799084935 - type: nauc_ndcg_at_3_diff1 value: 56.27751882008217 - type: nauc_ndcg_at_3_max value: 45.65587414820722 - type: nauc_ndcg_at_3_std value: 14.994891252809934 - type: nauc_ndcg_at_5_diff1 value: 55.53169392843657 - type: nauc_ndcg_at_5_max value: 44.23153109548333 - type: nauc_ndcg_at_5_std value: 13.938058808850082 - type: nauc_precision_at_1000_diff1 value: -17.238089963012836 - type: nauc_precision_at_1000_max value: 23.866846371139047 - type: nauc_precision_at_1000_std value: 36.26953209140314 - type: nauc_precision_at_100_diff1 value: -1.126473234568327 - type: nauc_precision_at_100_max value: 31.108745423284912 - type: nauc_precision_at_100_std value: 30.642388176707968 - type: nauc_precision_at_10_diff1 value: 17.489823438228598 - type: nauc_precision_at_10_max value: 38.62598562468759 - type: nauc_precision_at_10_std value: 26.145677114183353 - type: nauc_precision_at_1_diff1 value: 61.55378221555377 - type: nauc_precision_at_1_max value: 40.56404604782708 - type: nauc_precision_at_1_std value: 11.237989014734051 - type: nauc_precision_at_20_diff1 value: 9.30790121361279 - type: nauc_precision_at_20_max value: 39.74539590936092 - type: nauc_precision_at_20_std value: 28.76347279979629 - type: nauc_precision_at_3_diff1 value: 36.56691202573061 - type: nauc_precision_at_3_max value: 47.33091823598092 - type: nauc_precision_at_3_std value: 25.97270398234439 - type: nauc_precision_at_5_diff1 value: 27.53989814653275 - type: nauc_precision_at_5_max value: 42.906811554469954 - type: nauc_precision_at_5_std value: 25.673969211341785 - type: nauc_recall_at_1000_diff1 value: 64.96265172735714 - type: nauc_recall_at_1000_max value: 70.79831932773095 - type: nauc_recall_at_1000_std value: 85.23576097105469 - type: nauc_recall_at_100_diff1 value: 43.528295808426066 - type: nauc_recall_at_100_max value: 51.37810072663499 - type: nauc_recall_at_100_std value: 28.73429502093998 - type: nauc_recall_at_10_diff1 value: 46.428201019869604 - type: nauc_recall_at_10_max value: 43.55728664156955 - type: nauc_recall_at_10_std value: 13.44915034828845 - type: nauc_recall_at_1_diff1 value: 63.12715359944983 - type: nauc_recall_at_1_max value: 37.86501536906817 - type: nauc_recall_at_1_std value: 6.322678498743812 - type: nauc_recall_at_20_diff1 value: 46.569137618158344 - type: nauc_recall_at_20_max value: 53.92571049318345 - type: nauc_recall_at_20_std value: 19.731744586272395 - type: nauc_recall_at_3_diff1 value: 51.783556556783786 - type: nauc_recall_at_3_max value: 46.66691776547845 - type: nauc_recall_at_3_std value: 16.374424975237936 - type: nauc_recall_at_5_diff1 value: 48.59699452539875 - type: nauc_recall_at_5_max value: 43.94693563083851 - type: nauc_recall_at_5_std value: 14.447872758117455 - type: ndcg_at_1 value: 49.0 - type: ndcg_at_10 value: 60.526 - type: ndcg_at_100 value: 63.583999999999996 - type: ndcg_at_1000 value: 65.017 - type: ndcg_at_20 value: 62.297999999999995 - type: ndcg_at_3 value: 56.345 - type: ndcg_at_5 value: 58.179 - type: precision_at_1 value: 49.0 - type: precision_at_10 value: 8.3 - type: precision_at_100 value: 0.993 - type: precision_at_1000 value: 0.11 - type: precision_at_20 value: 4.567 - type: precision_at_3 value: 22.556 - type: precision_at_5 value: 14.799999999999999 - type: recall_at_1 value: 46.400000000000006 - type: recall_at_10 value: 73.161 - type: recall_at_100 value: 86.5 - type: recall_at_1000 value: 97.333 - type: recall_at_20 value: 79.756 - type: recall_at_3 value: 61.533 - type: recall_at_5 value: 66.22200000000001 task: type: Retrieval - dataset: config: default name: MTEB TRECCOVID-PL revision: 81bcb408f33366c2a20ac54adafad1ae7e877fdd split: test type: clarin-knext/trec-covid-pl metrics: - type: main_score value: 65.563 - type: map_at_1 value: 0.22100000000000003 - type: map_at_10 value: 1.635 - type: map_at_100 value: 8.618 - type: map_at_1000 value: 20.422 - type: map_at_20 value: 2.789 - type: map_at_3 value: 0.609 - type: map_at_5 value: 0.9249999999999999 - type: mrr_at_1 value: 82.0 - type: mrr_at_10 value: 88.50000000000001 - type: mrr_at_100 value: 88.50000000000001 - type: mrr_at_1000 value: 88.50000000000001 - type: mrr_at_20 value: 88.50000000000001 - type: mrr_at_3 value: 87.66666666666667 - type: mrr_at_5 value: 88.16666666666667 - type: nauc_map_at_1000_diff1 value: -25.967067965482478 - type: nauc_map_at_1000_max value: 72.92874918094974 - type: nauc_map_at_1000_std value: 84.9616839707479 - type: nauc_map_at_100_diff1 value: -21.42642167269316 - type: nauc_map_at_100_max value: 62.68226707073076 - type: nauc_map_at_100_std value: 75.56794844224473 - type: nauc_map_at_10_diff1 value: -25.660983963152447 - type: nauc_map_at_10_max value: 41.96962876232467 - type: nauc_map_at_10_std value: 40.45419086736852 - type: nauc_map_at_1_diff1 value: -8.591425903743453 - type: nauc_map_at_1_max value: 17.51978543502864 - type: nauc_map_at_1_std value: 20.076935065027513 - type: nauc_map_at_20_diff1 value: -25.891301436996805 - type: nauc_map_at_20_max value: 47.84153247293719 - type: nauc_map_at_20_std value: 51.816446917682555 - type: nauc_map_at_3_diff1 value: -13.848652828499809 - type: nauc_map_at_3_max value: 26.243795101152205 - type: nauc_map_at_3_std value: 23.036433586747922 - type: nauc_map_at_5_diff1 value: -21.1896066711519 - type: nauc_map_at_5_max value: 31.324922426520786 - type: nauc_map_at_5_std value: 29.15619704280249 - type: nauc_mrr_at_1000_diff1 value: 24.59044392889167 - type: nauc_mrr_at_1000_max value: 62.571343173116645 - type: nauc_mrr_at_1000_std value: 65.96948433652632 - type: nauc_mrr_at_100_diff1 value: 24.59044392889167 - type: nauc_mrr_at_100_max value: 62.571343173116645 - type: nauc_mrr_at_100_std value: 65.96948433652632 - type: nauc_mrr_at_10_diff1 value: 24.59044392889167 - type: nauc_mrr_at_10_max value: 62.571343173116645 - type: nauc_mrr_at_10_std value: 65.96948433652632 - type: nauc_mrr_at_1_diff1 value: 27.541579469104494 - type: nauc_mrr_at_1_max value: 63.63586298050444 - type: nauc_mrr_at_1_std value: 65.66802511289794 - type: nauc_mrr_at_20_diff1 value: 24.59044392889167 - type: nauc_mrr_at_20_max value: 62.571343173116645 - type: nauc_mrr_at_20_std value: 65.96948433652632 - type: nauc_mrr_at_3_diff1 value: 26.89052291903615 - type: nauc_mrr_at_3_max value: 64.43515751051395 - type: nauc_mrr_at_3_std value: 65.28156161557388 - type: nauc_mrr_at_5_diff1 value: 24.226184721205527 - type: nauc_mrr_at_5_max value: 62.856866617864384 - type: nauc_mrr_at_5_std value: 66.58954986809582 - type: nauc_ndcg_at_1000_diff1 value: -28.716159160219824 - type: nauc_ndcg_at_1000_max value: 72.97926866355515 - type: nauc_ndcg_at_1000_std value: 84.63716554628942 - type: nauc_ndcg_at_100_diff1 value: -24.58125590309421 - type: nauc_ndcg_at_100_max value: 64.7106381911207 - type: nauc_ndcg_at_100_std value: 78.21077458187217 - type: nauc_ndcg_at_10_diff1 value: -18.358464881146272 - type: nauc_ndcg_at_10_max value: 67.10104488172087 - type: nauc_ndcg_at_10_std value: 71.68515146865761 - type: nauc_ndcg_at_1_diff1 value: 5.718639098904514 - type: nauc_ndcg_at_1_max value: 46.7462582934732 - type: nauc_ndcg_at_1_std value: 60.704366610091085 - type: nauc_ndcg_at_20_diff1 value: -25.276773792441947 - type: nauc_ndcg_at_20_max value: 65.84214731568228 - type: nauc_ndcg_at_20_std value: 75.85064732973377 - type: nauc_ndcg_at_3_diff1 value: 5.786838453770751 - type: nauc_ndcg_at_3_max value: 56.873202263808054 - type: nauc_ndcg_at_3_std value: 58.29378534996971 - type: nauc_ndcg_at_5_diff1 value: -10.849506785055805 - type: nauc_ndcg_at_5_max value: 60.86780916587185 - type: nauc_ndcg_at_5_std value: 64.67628462822093 - type: nauc_precision_at_1000_diff1 value: -17.658051874711777 - type: nauc_precision_at_1000_max value: 52.243182152280795 - type: nauc_precision_at_1000_std value: 52.221414126675405 - type: nauc_precision_at_100_diff1 value: -21.912367625930017 - type: nauc_precision_at_100_max value: 62.10951205181152 - type: nauc_precision_at_100_std value: 76.90051049763301 - type: nauc_precision_at_10_diff1 value: -24.589938416247513 - type: nauc_precision_at_10_max value: 69.12099854638501 - type: nauc_precision_at_10_std value: 73.49422451702836 - type: nauc_precision_at_1_diff1 value: 27.541579469104494 - type: nauc_precision_at_1_max value: 63.63586298050444 - type: nauc_precision_at_1_std value: 65.66802511289794 - type: nauc_precision_at_20_diff1 value: -29.032893299104007 - type: nauc_precision_at_20_max value: 67.03945899084314 - type: nauc_precision_at_20_std value: 78.94031560468493 - type: nauc_precision_at_3_diff1 value: 5.476377111095327 - type: nauc_precision_at_3_max value: 67.50694790850136 - type: nauc_precision_at_3_std value: 54.68716596593738 - type: nauc_precision_at_5_diff1 value: -20.104469911928415 - type: nauc_precision_at_5_max value: 66.46184176585471 - type: nauc_precision_at_5_std value: 62.40188672292436 - type: nauc_recall_at_1000_diff1 value: -28.649258704821147 - type: nauc_recall_at_1000_max value: 69.16314905082237 - type: nauc_recall_at_1000_std value: 79.32321876901818 - type: nauc_recall_at_100_diff1 value: -19.180264922861635 - type: nauc_recall_at_100_max value: 48.57682348744147 - type: nauc_recall_at_100_std value: 65.9297008167438 - type: nauc_recall_at_10_diff1 value: -23.597918176676906 - type: nauc_recall_at_10_max value: 35.163103297663795 - type: nauc_recall_at_10_std value: 36.537988377094486 - type: nauc_recall_at_1_diff1 value: -8.591425903743453 - type: nauc_recall_at_1_max value: 17.51978543502864 - type: nauc_recall_at_1_std value: 20.076935065027513 - type: nauc_recall_at_20_diff1 value: -25.441741663652127 - type: nauc_recall_at_20_max value: 37.09848404778578 - type: nauc_recall_at_20_std value: 42.94987387680688 - type: nauc_recall_at_3_diff1 value: -14.825486887364592 - type: nauc_recall_at_3_max value: 22.7488507817971 - type: nauc_recall_at_3_std value: 18.642543682159996 - type: nauc_recall_at_5_diff1 value: -21.938394755683525 - type: nauc_recall_at_5_max value: 27.492935982827078 - type: nauc_recall_at_5_std value: 27.013096425647436 - type: ndcg_at_1 value: 71.0 - type: ndcg_at_10 value: 65.563 - type: ndcg_at_100 value: 48.929 - type: ndcg_at_1000 value: 44.061 - type: ndcg_at_20 value: 61.917 - type: ndcg_at_3 value: 71.33500000000001 - type: ndcg_at_5 value: 68.135 - type: precision_at_1 value: 82.0 - type: precision_at_10 value: 69.39999999999999 - type: precision_at_100 value: 50.0 - type: precision_at_1000 value: 19.564 - type: precision_at_20 value: 65.4 - type: precision_at_3 value: 79.333 - type: precision_at_5 value: 73.2 - type: recall_at_1 value: 0.22100000000000003 - type: recall_at_10 value: 1.841 - type: recall_at_100 value: 11.959 - type: recall_at_1000 value: 41.373 - type: recall_at_20 value: 3.376 - type: recall_at_3 value: 0.6459999999999999 - type: recall_at_5 value: 0.9900000000000001 task: type: Retrieval - dataset: config: default name: MTEB CEDRClassification revision: c0ba03d058e3e1b2f3fd20518875a4563dd12db4 split: test type: ai-forever/cedr-classification metrics: - type: accuracy value: 43.76195536663124 - type: f1 value: 42.465506071271584 - type: lrap value: 73.62991498406049 - type: main_score value: 43.76195536663124 task: type: MultilabelClassification - dataset: config: default name: MTEB GeoreviewClassification revision: 3765c0d1de6b7d264bc459433c45e5a75513839c split: test type: ai-forever/georeview-classification metrics: - type: accuracy value: 51.50390625 - type: f1 value: 47.10882921865018 - type: f1_weighted value: 47.10602639734162 - type: main_score value: 51.50390625 task: type: Classification - dataset: config: default name: MTEB GeoreviewClusteringP2P revision: 97a313c8fc85b47f13f33e7e9a95c1ad888c7fec split: test type: ai-forever/georeview-clustering-p2p metrics: - type: main_score value: 60.763743799017725 - type: v_measure value: 60.763743799017725 - type: v_measure_std value: 0.6572810824719637 task: type: Clustering - dataset: config: default name: MTEB HeadlineClassification revision: 2fe05ee6b5832cda29f2ef7aaad7b7fe6a3609eb split: test type: ai-forever/headline-classification metrics: - type: accuracy value: 84.365234375 - type: f1 value: 84.3504758444136 - type: f1_weighted value: 84.34875898591558 - type: main_score value: 84.365234375 task: type: Classification - dataset: config: default name: MTEB InappropriatenessClassification revision: 601651fdc45ef243751676e62dd7a19f491c0285 split: test type: ai-forever/inappropriateness-classification metrics: - type: accuracy value: 65.888671875 - type: ap value: 60.9135452326786 - type: ap_weighted value: 60.9135452326786 - type: f1 value: 65.39737070466047 - type: f1_weighted value: 65.39737070466047 - type: main_score value: 65.888671875 task: type: Classification - dataset: config: default name: MTEB KinopoiskClassification revision: 5911f26666ac11af46cb9c6849d0dc80a378af24 split: test type: ai-forever/kinopoisk-sentiment-classification metrics: - type: accuracy value: 62.28666666666667 - type: f1 value: 58.05000673882692 - type: f1_weighted value: 58.05000673882692 - type: main_score value: 62.28666666666667 task: type: Classification - dataset: config: ru name: MTEB MIRACLReranking (ru) revision: 6d1962c527217f8927fca80f890f14f36b2802af split: dev type: miracl/mmteb-miracl-reranking metrics: - type: MAP@1(MIRACL) value: 30.086000000000002 - type: MAP@10(MIRACL) value: 49.463 - type: MAP@100(MIRACL) value: 51.49 - type: MAP@1000(MIRACL) value: 51.49 - type: MAP@20(MIRACL) value: 50.876999999999995 - type: MAP@3(MIRACL) value: 42.693 - type: MAP@5(MIRACL) value: 46.400999999999996 - type: NDCG@1(MIRACL) value: 49.639 - type: NDCG@10(MIRACL) value: 57.247 - type: NDCG@100(MIRACL) value: 61.378 - type: NDCG@1000(MIRACL) value: 61.378 - type: NDCG@20(MIRACL) value: 59.760000000000005 - type: NDCG@3(MIRACL) value: 51.608 - type: NDCG@5(MIRACL) value: 53.759 - type: P@1(MIRACL) value: 49.639 - type: P@10(MIRACL) value: 15.028 - type: P@100(MIRACL) value: 1.9529999999999998 - type: P@1000(MIRACL) value: 0.19499999999999998 - type: P@20(MIRACL) value: 8.753 - type: P@3(MIRACL) value: 32.05 - type: P@5(MIRACL) value: 24.074 - type: Recall@1(MIRACL) value: 30.086000000000002 - type: Recall@10(MIRACL) value: 67.67 - type: Recall@100(MIRACL) value: 79.952 - type: Recall@1000(MIRACL) value: 79.952 - type: Recall@20(MIRACL) value: 74.409 - type: Recall@3(MIRACL) value: 50.432 - type: Recall@5(MIRACL) value: 58.431999999999995 - type: main_score value: 57.247 - type: nAUC_MAP@1000_diff1(MIRACL) value: 14.49132703139068 - type: nAUC_MAP@1000_max(MIRACL) value: 32.22718193428578 - type: nAUC_MAP@1000_std(MIRACL) value: 13.815910840085923 - type: nAUC_MAP@100_diff1(MIRACL) value: 14.49132703139068 - type: nAUC_MAP@100_max(MIRACL) value: 32.22718193428578 - type: nAUC_MAP@100_std(MIRACL) value: 13.815910840085923 - type: nAUC_MAP@10_diff1(MIRACL) value: 16.504956542658086 - type: nAUC_MAP@10_max(MIRACL) value: 30.050872244941402 - type: nAUC_MAP@10_std(MIRACL) value: 13.136444562982783 - type: nAUC_MAP@1_diff1(MIRACL) value: 34.18909528530817 - type: nAUC_MAP@1_max(MIRACL) value: 15.062828187996063 - type: nAUC_MAP@1_std(MIRACL) value: 3.2097645527643595 - type: nAUC_MAP@20_diff1(MIRACL) value: 15.130045331734273 - type: nAUC_MAP@20_max(MIRACL) value: 31.64805643481855 - type: nAUC_MAP@20_std(MIRACL) value: 13.811628131618614 - type: nAUC_MAP@3_diff1(MIRACL) value: 22.967076674429805 - type: nAUC_MAP@3_max(MIRACL) value: 23.55271431173739 - type: nAUC_MAP@3_std(MIRACL) value: 8.38797791381467 - type: nAUC_MAP@5_diff1(MIRACL) value: 18.864687263601386 - type: nAUC_MAP@5_max(MIRACL) value: 27.137349141940113 - type: nAUC_MAP@5_std(MIRACL) value: 10.729766654007843 - type: nAUC_NDCG@1000_diff1(MIRACL) value: 3.9965439655979824 - type: nAUC_NDCG@1000_max(MIRACL) value: 40.636834454723754 - type: nAUC_NDCG@1000_std(MIRACL) value: 17.562271508485765 - type: nAUC_NDCG@100_diff1(MIRACL) value: 3.9965439655979824 - type: nAUC_NDCG@100_max(MIRACL) value: 40.636834454723754 - type: nAUC_NDCG@100_std(MIRACL) value: 17.562271508485765 - type: nAUC_NDCG@10_diff1(MIRACL) value: 9.20785070475628 - type: nAUC_NDCG@10_max(MIRACL) value: 36.10685520213844 - type: nAUC_NDCG@10_std(MIRACL) value: 16.52302351987603 - type: nAUC_NDCG@1_diff1(MIRACL) value: 21.380050974695997 - type: nAUC_NDCG@1_max(MIRACL) value: 34.082244652607194 - type: nAUC_NDCG@1_std(MIRACL) value: 11.960002005286022 - type: nAUC_NDCG@20_diff1(MIRACL) value: 6.029314693566221 - type: nAUC_NDCG@20_max(MIRACL) value: 38.93324906908806 - type: nAUC_NDCG@20_std(MIRACL) value: 17.800187238709555 - type: nAUC_NDCG@3_diff1(MIRACL) value: 14.75650513824637 - type: nAUC_NDCG@3_max(MIRACL) value: 31.345509680914045 - type: nAUC_NDCG@3_std(MIRACL) value: 12.489558805685855 - type: nAUC_NDCG@5_diff1(MIRACL) value: 12.114746297880274 - type: nAUC_NDCG@5_max(MIRACL) value: 32.80902906671674 - type: nAUC_NDCG@5_std(MIRACL) value: 13.673134481637822 - type: nAUC_P@1000_diff1(MIRACL) value: -28.3393493752439 - type: nAUC_P@1000_max(MIRACL) value: 29.94108745927258 - type: nAUC_P@1000_std(MIRACL) value: 13.00979331888063 - type: nAUC_P@100_diff1(MIRACL) value: -28.33934937524387 - type: nAUC_P@100_max(MIRACL) value: 29.94108745927262 - type: nAUC_P@100_std(MIRACL) value: 13.009793318880677 - type: nAUC_P@10_diff1(MIRACL) value: -24.128263659855403 - type: nAUC_P@10_max(MIRACL) value: 33.69795666200538 - type: nAUC_P@10_std(MIRACL) value: 18.667513470521502 - type: nAUC_P@1_diff1(MIRACL) value: 21.380050974695997 - type: nAUC_P@1_max(MIRACL) value: 34.082244652607194 - type: nAUC_P@1_std(MIRACL) value: 11.960002005286022 - type: nAUC_P@20_diff1(MIRACL) value: -26.851579419952216 - type: nAUC_P@20_max(MIRACL) value: 32.30632064213693 - type: nAUC_P@20_std(MIRACL) value: 16.71537079207824 - type: nAUC_P@3_diff1(MIRACL) value: -11.927325953384901 - type: nAUC_P@3_max(MIRACL) value: 37.1736865658166 - type: nAUC_P@3_std(MIRACL) value: 16.2754083159161 - type: nAUC_P@5_diff1(MIRACL) value: -20.552820285005026 - type: nAUC_P@5_max(MIRACL) value: 36.30994351506595 - type: nAUC_P@5_std(MIRACL) value: 17.40523400441202 - type: nAUC_Recall@1000_diff1(MIRACL) value: -32.466276225017346 - type: nAUC_Recall@1000_max(MIRACL) value: 61.05870930653621 - type: nAUC_Recall@1000_std(MIRACL) value: 27.09559295245869 - type: nAUC_Recall@100_diff1(MIRACL) value: -32.466276225017346 - type: nAUC_Recall@100_max(MIRACL) value: 61.05870930653621 - type: nAUC_Recall@100_std(MIRACL) value: 27.09559295245869 - type: nAUC_Recall@10_diff1(MIRACL) value: -4.979553747407279 - type: nAUC_Recall@10_max(MIRACL) value: 38.39843112329541 - type: nAUC_Recall@10_std(MIRACL) value: 20.600178046207702 - type: nAUC_Recall@1_diff1(MIRACL) value: 34.18909528530817 - type: nAUC_Recall@1_max(MIRACL) value: 15.062828187996063 - type: nAUC_Recall@1_std(MIRACL) value: 3.2097645527643595 - type: nAUC_Recall@20_diff1(MIRACL) value: -18.18036800889588 - type: nAUC_Recall@20_max(MIRACL) value: 48.61058795989584 - type: nAUC_Recall@20_std(MIRACL) value: 26.280248121862066 - type: nAUC_Recall@3_diff1(MIRACL) value: 14.170577180062406 - type: nAUC_Recall@3_max(MIRACL) value: 22.458002088633343 - type: nAUC_Recall@3_std(MIRACL) value: 10.035396517805115 - type: nAUC_Recall@5_diff1(MIRACL) value: 4.550643279455007 - type: nAUC_Recall@5_max(MIRACL) value: 28.457322956745617 - type: nAUC_Recall@5_std(MIRACL) value: 13.335687765558976 task: type: Reranking - dataset: config: ru name: MTEB MIRACLRetrieval (ru) revision: main split: dev type: miracl/mmteb-miracl metrics: - type: main_score value: 54.595000000000006 - type: map_at_1 value: 23.952 - type: map_at_10 value: 43.637 - type: map_at_100 value: 46.6 - type: map_at_1000 value: 46.697 - type: map_at_20 value: 45.378 - type: map_at_3 value: 36.137 - type: map_at_5 value: 39.838 - type: mrr_at_1 value: 48.48242811501598 - type: mrr_at_10 value: 61.81946980069989 - type: mrr_at_100 value: 62.23878510656363 - type: mrr_at_1000 value: 62.24764124248049 - type: mrr_at_20 value: 62.1051846692803 - type: mrr_at_3 value: 58.82587859424925 - type: mrr_at_5 value: 60.60303514377006 - type: nauc_map_at_1000_diff1 value: 28.214871462783854 - type: nauc_map_at_1000_max value: 25.201084766782078 - type: nauc_map_at_1000_std value: 6.765255742651155 - type: nauc_map_at_100_diff1 value: 28.234612621431754 - type: nauc_map_at_100_max value: 25.165912406356338 - type: nauc_map_at_100_std value: 6.7694968272696405 - type: nauc_map_at_10_diff1 value: 28.765507147170922 - type: nauc_map_at_10_max value: 23.216218446772334 - type: nauc_map_at_10_std value: 4.495189160179304 - type: nauc_map_at_1_diff1 value: 36.05018310203833 - type: nauc_map_at_1_max value: 13.651823625223662 - type: nauc_map_at_1_std value: -1.1689324424590004 - type: nauc_map_at_20_diff1 value: 28.471404911269993 - type: nauc_map_at_20_max value: 24.40961097812379 - type: nauc_map_at_20_std value: 5.62151334482884 - type: nauc_map_at_3_diff1 value: 29.40681082033128 - type: nauc_map_at_3_max value: 20.2388315012222 - type: nauc_map_at_3_std value: 1.3400996743731126 - type: nauc_map_at_5_diff1 value: 28.258533407547485 - type: nauc_map_at_5_max value: 21.966173291189982 - type: nauc_map_at_5_std value: 3.105883435302439 - type: nauc_mrr_at_1000_diff1 value: 29.973540045513143 - type: nauc_mrr_at_1000_max value: 32.675046742557335 - type: nauc_mrr_at_1000_std value: 13.045750572535795 - type: nauc_mrr_at_100_diff1 value: 29.974727506122523 - type: nauc_mrr_at_100_max value: 32.68341978762026 - type: nauc_mrr_at_100_std value: 13.06124318794068 - type: nauc_mrr_at_10_diff1 value: 29.811240994564102 - type: nauc_mrr_at_10_max value: 32.63357319986243 - type: nauc_mrr_at_10_std value: 13.063153729215351 - type: nauc_mrr_at_1_diff1 value: 33.4099136989708 - type: nauc_mrr_at_1_max value: 28.034252424559693 - type: nauc_mrr_at_1_std value: 8.771691345021653 - type: nauc_mrr_at_20_diff1 value: 29.906265270643118 - type: nauc_mrr_at_20_max value: 32.752505338776125 - type: nauc_mrr_at_20_std value: 13.092155055233029 - type: nauc_mrr_at_3_diff1 value: 30.159884994169563 - type: nauc_mrr_at_3_max value: 32.93411539112575 - type: nauc_mrr_at_3_std value: 12.479656041605635 - type: nauc_mrr_at_5_diff1 value: 29.747570134961947 - type: nauc_mrr_at_5_max value: 32.92420276622243 - type: nauc_mrr_at_5_std value: 13.228404409126401 - type: nauc_ndcg_at_1000_diff1 value: 26.973170727985845 - type: nauc_ndcg_at_1000_max value: 30.714533698659235 - type: nauc_ndcg_at_1000_std value: 12.646817183789668 - type: nauc_ndcg_at_100_diff1 value: 26.9185712126953 - type: nauc_ndcg_at_100_max value: 30.787272670713378 - type: nauc_ndcg_at_100_std value: 13.347713759949965 - type: nauc_ndcg_at_10_diff1 value: 27.977378333059967 - type: nauc_ndcg_at_10_max value: 26.09323198226889 - type: nauc_ndcg_at_10_std value: 7.669332805590227 - type: nauc_ndcg_at_1_diff1 value: 33.4099136989708 - type: nauc_ndcg_at_1_max value: 28.034252424559693 - type: nauc_ndcg_at_1_std value: 8.771691345021653 - type: nauc_ndcg_at_20_diff1 value: 27.437445515604885 - type: nauc_ndcg_at_20_max value: 28.652644415118107 - type: nauc_ndcg_at_20_std value: 9.979585555866413 - type: nauc_ndcg_at_3_diff1 value: 26.78918281794111 - type: nauc_ndcg_at_3_max value: 26.743605264167563 - type: nauc_ndcg_at_3_std value: 7.636594897931739 - type: nauc_ndcg_at_5_diff1 value: 26.768664779524386 - type: nauc_ndcg_at_5_max value: 25.92430880031359 - type: nauc_ndcg_at_5_std value: 7.567930905471419 - type: nauc_precision_at_1000_diff1 value: -12.46821203258456 - type: nauc_precision_at_1000_max value: 16.582341013228657 - type: nauc_precision_at_1000_std value: 19.378859711054375 - type: nauc_precision_at_100_diff1 value: -11.326145373988833 - type: nauc_precision_at_100_max value: 19.58909158403411 - type: nauc_precision_at_100_std value: 23.328633133712064 - type: nauc_precision_at_10_diff1 value: -0.8745238638551284 - type: nauc_precision_at_10_max value: 24.17820027443609 - type: nauc_precision_at_10_std value: 18.396638849812433 - type: nauc_precision_at_1_diff1 value: 33.4099136989708 - type: nauc_precision_at_1_max value: 28.034252424559693 - type: nauc_precision_at_1_std value: 8.771691345021653 - type: nauc_precision_at_20_diff1 value: -5.306130413062489 - type: nauc_precision_at_20_max value: 23.52655673001502 - type: nauc_precision_at_20_std value: 21.167397186114314 - type: nauc_precision_at_3_diff1 value: 8.580491424277982 - type: nauc_precision_at_3_max value: 31.019714029464208 - type: nauc_precision_at_3_std value: 14.133874593516971 - type: nauc_precision_at_5_diff1 value: 2.554020245713193 - type: nauc_precision_at_5_max value: 29.634541238020105 - type: nauc_precision_at_5_std value: 17.87723197950384 - type: nauc_recall_at_1000_diff1 value: 13.125393258634418 - type: nauc_recall_at_1000_max value: 62.24470202796473 - type: nauc_recall_at_1000_std value: 63.23164754823915 - type: nauc_recall_at_100_diff1 value: 14.423784358841202 - type: nauc_recall_at_100_max value: 38.617863069715675 - type: nauc_recall_at_100_std value: 39.097234031326536 - type: nauc_recall_at_10_diff1 value: 20.936852580771 - type: nauc_recall_at_10_max value: 16.594501361269014 - type: nauc_recall_at_10_std value: 4.532729652479956 - type: nauc_recall_at_1_diff1 value: 36.05018310203833 - type: nauc_recall_at_1_max value: 13.651823625223662 - type: nauc_recall_at_1_std value: -1.1689324424590004 - type: nauc_recall_at_20_diff1 value: 18.26313195777628 - type: nauc_recall_at_20_max value: 22.504290386601888 - type: nauc_recall_at_20_std value: 9.794573138033334 - type: nauc_recall_at_3_diff1 value: 23.738927052170915 - type: nauc_recall_at_3_max value: 17.92200131347856 - type: nauc_recall_at_3_std value: 0.5468756827969604 - type: nauc_recall_at_5_diff1 value: 20.33968031427608 - type: nauc_recall_at_5_max value: 17.602814155288783 - type: nauc_recall_at_5_std value: 2.666858198246704 - type: ndcg_at_1 value: 48.482 - type: ndcg_at_10 value: 54.595000000000006 - type: ndcg_at_100 value: 62.382000000000005 - type: ndcg_at_1000 value: 63.463 - type: ndcg_at_20 value: 58.24399999999999 - type: ndcg_at_3 value: 47.924 - type: ndcg_at_5 value: 49.716 - type: precision_at_1 value: 48.482 - type: precision_at_10 value: 16.901 - type: precision_at_100 value: 2.576 - type: precision_at_1000 value: 0.27799999999999997 - type: precision_at_20 value: 10.188 - type: precision_at_3 value: 33.227000000000004 - type: precision_at_5 value: 25.335 - type: recall_at_1 value: 23.952 - type: recall_at_10 value: 66.591 - type: recall_at_100 value: 91.902 - type: recall_at_1000 value: 97.772 - type: recall_at_20 value: 76.768 - type: recall_at_3 value: 43.954 - type: recall_at_5 value: 52.943 task: type: Retrieval - dataset: config: ru name: MTEB MassiveIntentClassification (ru) revision: 4672e20407010da34463acc759c162ca9734bca6 split: test type: mteb/amazon_massive_intent metrics: - type: accuracy value: 66.0659045057162 - type: f1 value: 61.35064382748825 - type: f1_weighted value: 64.6551910369696 - type: main_score value: 66.0659045057162 task: type: Classification - dataset: config: ru name: MTEB MassiveScenarioClassification (ru) revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 split: test type: mteb/amazon_massive_scenario metrics: - type: accuracy value: 74.80833893745798 - type: f1 value: 73.75893026985123 - type: f1_weighted value: 74.48216382388233 - type: main_score value: 74.80833893745798 task: type: Classification - dataset: config: default name: MTEB RUParaPhraserSTS revision: 43265056790b8f7c59e0139acb4be0a8dad2c8f4 split: test type: merionum/ru_paraphraser metrics: - type: cosine_pearson value: 60.71240323001735 - type: cosine_spearman value: 68.14010632490151 - type: euclidean_pearson value: 65.04715258693518 - type: euclidean_spearman value: 68.14015844762187 - type: main_score value: 68.14010632490151 - type: manhattan_pearson value: 65.00225170739532 - type: manhattan_spearman value: 67.99309271156126 - type: pearson value: 60.71240323001735 - type: spearman value: 68.14010632490151 task: type: STS - dataset: config: default name: MTEB RiaNewsRetrieval revision: 82374b0bbacda6114f39ff9c5b925fa1512ca5d7 split: test type: ai-forever/ria-news-retrieval metrics: - type: main_score value: 78.172 - type: map_at_1 value: 65.97 - type: map_at_10 value: 74.494 - type: map_at_100 value: 74.811 - type: map_at_1000 value: 74.821 - type: map_at_20 value: 74.712 - type: map_at_3 value: 73.072 - type: map_at_5 value: 73.969 - type: mrr_at_1 value: 65.97 - type: mrr_at_10 value: 74.49480158730101 - type: mrr_at_100 value: 74.81143726471089 - type: mrr_at_1000 value: 74.82174846844643 - type: mrr_at_20 value: 74.7122507574977 - type: mrr_at_3 value: 73.07166666666618 - type: mrr_at_5 value: 73.96916666666591 - type: nauc_map_at_1000_diff1 value: 73.08686946777952 - type: nauc_map_at_1000_max value: 24.925205621117783 - type: nauc_map_at_1000_std value: -12.15886477655549 - type: nauc_map_at_100_diff1 value: 73.08244355589704 - type: nauc_map_at_100_max value: 24.93258145762047 - type: nauc_map_at_100_std value: -12.149308732380295 - type: nauc_map_at_10_diff1 value: 72.97848397979895 - type: nauc_map_at_10_max value: 24.887366324397487 - type: nauc_map_at_10_std value: -12.284523808652416 - type: nauc_map_at_1_diff1 value: 76.45682519258669 - type: nauc_map_at_1_max value: 22.373249835264456 - type: nauc_map_at_1_std value: -14.13464876294517 - type: nauc_map_at_20_diff1 value: 73.04810922101197 - type: nauc_map_at_20_max value: 24.92236406693643 - type: nauc_map_at_20_std value: -12.176293947009215 - type: nauc_map_at_3_diff1 value: 72.9755537768131 - type: nauc_map_at_3_max value: 24.620733273355683 - type: nauc_map_at_3_std value: -12.85288384470766 - type: nauc_map_at_5_diff1 value: 73.00265519892334 - type: nauc_map_at_5_max value: 24.786315954855553 - type: nauc_map_at_5_std value: -12.564020496189352 - type: nauc_mrr_at_1000_diff1 value: 73.08804965487802 - type: nauc_mrr_at_1000_max value: 24.924428532055636 - type: nauc_mrr_at_1000_std value: -12.160384139274424 - type: nauc_mrr_at_100_diff1 value: 73.08362309430365 - type: nauc_mrr_at_100_max value: 24.931804884947883 - type: nauc_mrr_at_100_std value: -12.150827182790605 - type: nauc_mrr_at_10_diff1 value: 72.9796448439789 - type: nauc_mrr_at_10_max value: 24.88659980440671 - type: nauc_mrr_at_10_std value: -12.286023526026135 - type: nauc_mrr_at_1_diff1 value: 76.45682519258669 - type: nauc_mrr_at_1_max value: 22.373249835264456 - type: nauc_mrr_at_1_std value: -14.13464876294517 - type: nauc_mrr_at_20_diff1 value: 73.04928271486784 - type: nauc_mrr_at_20_max value: 24.921590820144715 - type: nauc_mrr_at_20_std value: -12.177806033794447 - type: nauc_mrr_at_3_diff1 value: 72.9755537768131 - type: nauc_mrr_at_3_max value: 24.620733273355683 - type: nauc_mrr_at_3_std value: -12.85288384470766 - type: nauc_mrr_at_5_diff1 value: 73.00379294775456 - type: nauc_mrr_at_5_max value: 24.78556305369168 - type: nauc_mrr_at_5_std value: -12.565495148428068 - type: nauc_ndcg_at_1000_diff1 value: 72.17730594095745 - type: nauc_ndcg_at_1000_max value: 26.080483734410535 - type: nauc_ndcg_at_1000_std value: -10.600341719859815 - type: nauc_ndcg_at_100_diff1 value: 72.01297350077037 - type: nauc_ndcg_at_100_max value: 26.370245187512793 - type: nauc_ndcg_at_100_std value: -10.13804122238926 - type: nauc_ndcg_at_10_diff1 value: 71.54838244432331 - type: nauc_ndcg_at_10_max value: 26.132336506589553 - type: nauc_ndcg_at_10_std value: -10.895520797166812 - type: nauc_ndcg_at_1_diff1 value: 76.45682519258669 - type: nauc_ndcg_at_1_max value: 22.373249835264456 - type: nauc_ndcg_at_1_std value: -14.13464876294517 - type: nauc_ndcg_at_20_diff1 value: 71.78002529865489 - type: nauc_ndcg_at_20_max value: 26.34207650975323 - type: nauc_ndcg_at_20_std value: -10.389370339614697 - type: nauc_ndcg_at_3_diff1 value: 71.70864174615559 - type: nauc_ndcg_at_3_max value: 25.445588176267258 - type: nauc_ndcg_at_3_std value: -12.309378837447191 - type: nauc_ndcg_at_5_diff1 value: 71.67055602722155 - type: nauc_ndcg_at_5_max value: 25.818799225442223 - type: nauc_ndcg_at_5_std value: -11.71310643254645 - type: nauc_precision_at_1000_diff1 value: 52.795678598638375 - type: nauc_precision_at_1000_max value: 58.02042698469986 - type: nauc_precision_at_1000_std value: 38.79122201027639 - type: nauc_precision_at_100_diff1 value: 57.635497479970155 - type: nauc_precision_at_100_max value: 50.98528067422545 - type: nauc_precision_at_100_std value: 29.023249788572336 - type: nauc_precision_at_10_diff1 value: 62.88435357133938 - type: nauc_precision_at_10_max value: 33.93536070722828 - type: nauc_precision_at_10_std value: -1.6485440570585936 - type: nauc_precision_at_1_diff1 value: 76.45682519258669 - type: nauc_precision_at_1_max value: 22.373249835264456 - type: nauc_precision_at_1_std value: -14.13464876294517 - type: nauc_precision_at_20_diff1 value: 61.65978869266085 - type: nauc_precision_at_20_max value: 39.11301597287762 - type: nauc_precision_at_20_std value: 6.864118196298169 - type: nauc_precision_at_3_diff1 value: 66.8221462321319 - type: nauc_precision_at_3_max value: 28.6301783798532 - type: nauc_precision_at_3_std value: -10.17852659176101 - type: nauc_precision_at_5_diff1 value: 65.45372225751974 - type: nauc_precision_at_5_max value: 30.71086457162392 - type: nauc_precision_at_5_std value: -7.5734188075962345 - type: nauc_recall_at_1000_diff1 value: 52.79567859863725 - type: nauc_recall_at_1000_max value: 58.020426984700634 - type: nauc_recall_at_1000_std value: 38.79122201027698 - type: nauc_recall_at_100_diff1 value: 57.6354974799695 - type: nauc_recall_at_100_max value: 50.98528067422452 - type: nauc_recall_at_100_std value: 29.023249788571604 - type: nauc_recall_at_10_diff1 value: 62.884353571339155 - type: nauc_recall_at_10_max value: 33.93536070722848 - type: nauc_recall_at_10_std value: -1.6485440570586527 - type: nauc_recall_at_1_diff1 value: 76.45682519258669 - type: nauc_recall_at_1_max value: 22.373249835264456 - type: nauc_recall_at_1_std value: -14.13464876294517 - type: nauc_recall_at_20_diff1 value: 61.65978869266054 - type: nauc_recall_at_20_max value: 39.11301597287748 - type: nauc_recall_at_20_std value: 6.8641181962978965 - type: nauc_recall_at_3_diff1 value: 66.82214623213184 - type: nauc_recall_at_3_max value: 28.630178379853145 - type: nauc_recall_at_3_std value: -10.178526591761056 - type: nauc_recall_at_5_diff1 value: 65.45372225751966 - type: nauc_recall_at_5_max value: 30.710864571624015 - type: nauc_recall_at_5_std value: -7.573418807596015 - type: ndcg_at_1 value: 65.97 - type: ndcg_at_10 value: 78.172 - type: ndcg_at_100 value: 79.63799999999999 - type: ndcg_at_1000 value: 79.915 - type: ndcg_at_20 value: 78.949 - type: ndcg_at_3 value: 75.304 - type: ndcg_at_5 value: 76.924 - type: precision_at_1 value: 65.97 - type: precision_at_10 value: 8.948 - type: precision_at_100 value: 0.962 - type: precision_at_1000 value: 0.098 - type: precision_at_20 value: 4.626 - type: precision_at_3 value: 27.247 - type: precision_at_5 value: 17.136000000000003 - type: recall_at_1 value: 65.97 - type: recall_at_10 value: 89.48 - type: recall_at_100 value: 96.17999999999999 - type: recall_at_1000 value: 98.37 - type: recall_at_20 value: 92.52 - type: recall_at_3 value: 81.74 - type: recall_at_5 value: 85.68 task: type: Retrieval - dataset: config: default name: MTEB RuBQReranking revision: 2e96b8f098fa4b0950fc58eacadeb31c0d0c7fa2 split: test type: ai-forever/rubq-reranking metrics: - type: main_score value: 70.23399911521713 - type: map value: 70.23399911521713 - type: mrr value: 75.79756531884192 - type: nAUC_map_diff1 value: 41.70907433488097 - type: nAUC_map_max value: 19.038198833852398 - type: nAUC_map_std value: 9.364787080049835 - type: nAUC_mrr_diff1 value: 45.427118778337864 - type: nAUC_mrr_max value: 23.422465198632338 - type: nAUC_mrr_std value: 12.352254802172363 task: type: Reranking - dataset: config: default name: MTEB RuBQRetrieval revision: e19b6ffa60b3bc248e0b41f4cc37c26a55c2a67b split: test type: ai-forever/rubq-retrieval metrics: - type: main_score value: 67.97800000000001 - type: map_at_1 value: 38.682 - type: map_at_10 value: 59.602999999999994 - type: map_at_100 value: 60.609 - type: map_at_1000 value: 60.638999999999996 - type: map_at_20 value: 60.295 - type: map_at_3 value: 53.864000000000004 - type: map_at_5 value: 57.357 - type: mrr_at_1 value: 55.43735224586288 - type: mrr_at_10 value: 68.11629892303657 - type: mrr_at_100 value: 68.46735935081414 - type: mrr_at_1000 value: 68.47375950539598 - type: mrr_at_20 value: 68.3555220700062 - type: mrr_at_3 value: 65.64223798266359 - type: mrr_at_5 value: 67.26457840819555 - type: nauc_map_at_1000_diff1 value: 38.947165927647895 - type: nauc_map_at_1000_max value: 21.0400498401016 - type: nauc_map_at_1000_std value: -12.325977772218819 - type: nauc_map_at_100_diff1 value: 38.93708131706777 - type: nauc_map_at_100_max value: 21.05313115659324 - type: nauc_map_at_100_std value: -12.307766320896842 - type: nauc_map_at_10_diff1 value: 38.62500444502093 - type: nauc_map_at_10_max value: 20.841336922136993 - type: nauc_map_at_10_std value: -12.753687299190014 - type: nauc_map_at_1_diff1 value: 42.42534428928902 - type: nauc_map_at_1_max value: 12.831550041111084 - type: nauc_map_at_1_std value: -11.266929686927616 - type: nauc_map_at_20_diff1 value: 38.839757385230854 - type: nauc_map_at_20_max value: 21.155443051360105 - type: nauc_map_at_20_std value: -12.290324296540298 - type: nauc_map_at_3_diff1 value: 38.600886947896804 - type: nauc_map_at_3_max value: 18.67062373078893 - type: nauc_map_at_3_std value: -14.47069134931124 - type: nauc_map_at_5_diff1 value: 38.47519630592911 - type: nauc_map_at_5_max value: 19.424230739972682 - type: nauc_map_at_5_std value: -13.715848064929636 - type: nauc_mrr_at_1000_diff1 value: 45.829087222671134 - type: nauc_mrr_at_1000_max value: 24.0611740840992 - type: nauc_mrr_at_1000_std value: -13.779501374204361 - type: nauc_mrr_at_100_diff1 value: 45.82511154142628 - type: nauc_mrr_at_100_max value: 24.068094029026092 - type: nauc_mrr_at_100_std value: -13.771488599042799 - type: nauc_mrr_at_10_diff1 value: 45.66670841729006 - type: nauc_mrr_at_10_max value: 24.21302734762609 - type: nauc_mrr_at_10_std value: -13.878591942488693 - type: nauc_mrr_at_1_diff1 value: 49.10187561048428 - type: nauc_mrr_at_1_max value: 21.468296250291164 - type: nauc_mrr_at_1_std value: -14.16579631922114 - type: nauc_mrr_at_20_diff1 value: 45.780763156949334 - type: nauc_mrr_at_20_max value: 24.16790762743539 - type: nauc_mrr_at_20_std value: -13.798341816508907 - type: nauc_mrr_at_3_diff1 value: 45.81750015144424 - type: nauc_mrr_at_3_max value: 23.74635317348391 - type: nauc_mrr_at_3_std value: -14.465609504516872 - type: nauc_mrr_at_5_diff1 value: 45.50087795737209 - type: nauc_mrr_at_5_max value: 24.021128950654578 - type: nauc_mrr_at_5_std value: -14.316147652867976 - type: nauc_ndcg_at_1000_diff1 value: 39.98481177488298 - type: nauc_ndcg_at_1000_max value: 23.50673478711865 - type: nauc_ndcg_at_1000_std value: -11.112473477484098 - type: nauc_ndcg_at_100_diff1 value: 39.56224231830938 - type: nauc_ndcg_at_100_max value: 23.886442622042228 - type: nauc_ndcg_at_100_std value: -10.534435913119328 - type: nauc_ndcg_at_10_diff1 value: 38.438066883020696 - type: nauc_ndcg_at_10_max value: 23.871323754524052 - type: nauc_ndcg_at_10_std value: -12.020382617526508 - type: nauc_ndcg_at_1_diff1 value: 49.24897980869228 - type: nauc_ndcg_at_1_max value: 21.476485407508953 - type: nauc_ndcg_at_1_std value: -14.362366603555893 - type: nauc_ndcg_at_20_diff1 value: 39.044984412751454 - type: nauc_ndcg_at_20_max value: 24.627000563476837 - type: nauc_ndcg_at_20_std value: -10.591144903278847 - type: nauc_ndcg_at_3_diff1 value: 38.59706175845992 - type: nauc_ndcg_at_3_max value: 20.660682570744303 - type: nauc_ndcg_at_3_std value: -14.989842570951817 - type: nauc_ndcg_at_5_diff1 value: 37.99262057687399 - type: nauc_ndcg_at_5_max value: 21.323357059546723 - type: nauc_ndcg_at_5_std value: -13.985440057708725 - type: nauc_precision_at_1000_diff1 value: -7.055913327014856 - type: nauc_precision_at_1000_max value: 7.645796200601874 - type: nauc_precision_at_1000_std value: 5.7441793819913425 - type: nauc_precision_at_100_diff1 value: -5.9521730506691695 - type: nauc_precision_at_100_max value: 10.791705567977127 - type: nauc_precision_at_100_std value: 6.971621413042414 - type: nauc_precision_at_10_diff1 value: 1.2124462289393534 - type: nauc_precision_at_10_max value: 17.288772868206426 - type: nauc_precision_at_10_std value: 1.7066485280331372 - type: nauc_precision_at_1_diff1 value: 49.24897980869228 - type: nauc_precision_at_1_max value: 21.476485407508953 - type: nauc_precision_at_1_std value: -14.362366603555893 - type: nauc_precision_at_20_diff1 value: -1.9829512286324857 - type: nauc_precision_at_20_max value: 15.73212540041351 - type: nauc_precision_at_20_std value: 5.881638156347814 - type: nauc_precision_at_3_diff1 value: 15.295884096855977 - type: nauc_precision_at_3_max value: 20.315378698666457 - type: nauc_precision_at_3_std value: -9.87038943277974 - type: nauc_precision_at_5_diff1 value: 7.500406876014883 - type: nauc_precision_at_5_max value: 17.81994110188596 - type: nauc_precision_at_5_std value: -4.56852626899763 - type: nauc_recall_at_1000_diff1 value: 24.05686791288483 - type: nauc_recall_at_1000_max value: 60.34558482395768 - type: nauc_recall_at_1000_std value: 75.24495489488791 - type: nauc_recall_at_100_diff1 value: 14.41757817770598 - type: nauc_recall_at_100_max value: 43.46467354926512 - type: nauc_recall_at_100_std value: 33.594674489620495 - type: nauc_recall_at_10_diff1 value: 22.768211257001038 - type: nauc_recall_at_10_max value: 29.59352351340424 - type: nauc_recall_at_10_std value: -5.643974726375464 - type: nauc_recall_at_1_diff1 value: 42.42534428928902 - type: nauc_recall_at_1_max value: 12.831550041111084 - type: nauc_recall_at_1_std value: -11.266929686927616 - type: nauc_recall_at_20_diff1 value: 22.091666169938616 - type: nauc_recall_at_20_max value: 38.1442003173532 - type: nauc_recall_at_20_std value: 6.8939344221023795 - type: nauc_recall_at_3_diff1 value: 30.141208429704356 - type: nauc_recall_at_3_max value: 19.02721767063729 - type: nauc_recall_at_3_std value: -15.298243677583443 - type: nauc_recall_at_5_diff1 value: 25.13257041212021 - type: nauc_recall_at_5_max value: 19.760375951283518 - type: nauc_recall_at_5_std value: -12.907639506550343 - type: ndcg_at_1 value: 55.37800000000001 - type: ndcg_at_10 value: 67.97800000000001 - type: ndcg_at_100 value: 71.186 - type: ndcg_at_1000 value: 71.666 - type: ndcg_at_20 value: 69.70100000000001 - type: ndcg_at_3 value: 60.045 - type: ndcg_at_5 value: 64.206 - type: precision_at_1 value: 55.37800000000001 - type: precision_at_10 value: 13.694 - type: precision_at_100 value: 1.6 - type: precision_at_1000 value: 0.166 - type: precision_at_20 value: 7.394 - type: precision_at_3 value: 33.353 - type: precision_at_5 value: 23.629 - type: recall_at_1 value: 38.682 - type: recall_at_10 value: 83.624 - type: recall_at_100 value: 95.964 - type: recall_at_1000 value: 99.072 - type: recall_at_20 value: 89.334 - type: recall_at_3 value: 64.039 - type: recall_at_5 value: 73.92999999999999 task: type: Retrieval - dataset: config: default name: MTEB RuReviewsClassification revision: f6d2c31f4dc6b88f468552750bfec05b4b41b05a split: test type: ai-forever/ru-reviews-classification metrics: - type: accuracy value: 67.6953125 - type: f1 value: 65.90453756121588 - type: f1_weighted value: 65.90276417438837 - type: main_score value: 67.6953125 task: type: Classification - dataset: config: default name: MTEB RuSTSBenchmarkSTS revision: 7cf24f325c6da6195df55bef3d86b5e0616f3018 split: test type: ai-forever/ru-stsbenchmark-sts metrics: - type: cosine_pearson value: 76.29471433169448 - type: cosine_spearman value: 76.87365514666403 - type: euclidean_pearson value: 75.61167023011886 - type: euclidean_spearman value: 76.87303787773313 - type: main_score value: 76.87365514666403 - type: manhattan_pearson value: 75.66392608968061 - type: manhattan_spearman value: 76.92734684196252 - type: pearson value: 76.29471433169448 - type: spearman value: 76.87365514666403 task: type: STS - dataset: config: default name: MTEB RuSciBenchGRNTIClassification revision: 673a610d6d3dd91a547a0d57ae1b56f37ebbf6a1 split: test type: ai-forever/ru-scibench-grnti-classification metrics: - type: accuracy value: 62.76367187500001 - type: f1 value: 61.038547511485696 - type: f1_weighted value: 61.05100853971954 - type: main_score value: 62.76367187500001 task: type: Classification - dataset: config: default name: MTEB RuSciBenchGRNTIClusteringP2P revision: 673a610d6d3dd91a547a0d57ae1b56f37ebbf6a1 split: test type: ai-forever/ru-scibench-grnti-classification metrics: - type: main_score value: 58.12317505878799 - type: v_measure value: 58.12317505878799 - type: v_measure_std value: 0.8074216341773242 task: type: Clustering - dataset: config: default name: MTEB RuSciBenchOECDClassification revision: 26c88e99dcaba32bb45d0e1bfc21902337f6d471 split: test type: ai-forever/ru-scibench-oecd-classification metrics: - type: accuracy value: 49.765625 - type: f1 value: 47.21576484201538 - type: f1_weighted value: 47.216631424004845 - type: main_score value: 49.765625 task: type: Classification - dataset: config: default name: MTEB RuSciBenchOECDClusteringP2P revision: 26c88e99dcaba32bb45d0e1bfc21902337f6d471 split: test type: ai-forever/ru-scibench-oecd-classification metrics: - type: main_score value: 48.94279918311604 - type: v_measure value: 48.94279918311604 - type: v_measure_std value: 0.6861637451888378 task: type: Clustering - dataset: config: ru name: MTEB STS22 (ru) revision: de9d86b3b84231dc21f76c7b7af1f28e2f57f6e3 split: test type: mteb/sts22-crosslingual-sts metrics: - type: cosine_pearson value: 64.72567885537859 - type: cosine_spearman value: 67.02766486173321 - type: euclidean_pearson value: 65.20776364711082 - type: euclidean_spearman value: 67.02766486173321 - type: main_score value: 67.02766486173321 - type: manhattan_pearson value: 65.01652728805082 - type: manhattan_spearman value: 67.01526065421112 - type: pearson value: 64.72567885537859 - type: spearman value: 67.02766486173321 task: type: STS - dataset: config: default name: MTEB SensitiveTopicsClassification revision: 416b34a802308eac30e4192afc0ff99bb8dcc7f2 split: test type: ai-forever/sensitive-topics-classification metrics: - type: accuracy value: 31.5966796875 - type: f1 value: 37.51973703475147 - type: lrap value: 47.855767144096504 - type: main_score value: 31.5966796875 task: type: MultilabelClassification - dataset: config: default name: MTEB TERRa revision: 7b58f24536063837d644aab9a023c62199b2a612 split: dev type: ai-forever/terra-pairclassification metrics: - type: cosine_accuracy value: 57.65472312703584 - type: cosine_accuracy_threshold value: 83.7156891822815 - type: cosine_ap value: 55.055463204979915 - type: cosine_f1 value: 67.26057906458797 - type: cosine_f1_threshold value: 76.06719732284546 - type: cosine_precision value: 51.01351351351351 - type: cosine_recall value: 98.69281045751634 - type: dot_accuracy value: 57.65472312703584 - type: dot_accuracy_threshold value: 83.71570110321045 - type: dot_ap value: 55.055463204979915 - type: dot_f1 value: 67.26057906458797 - type: dot_f1_threshold value: 76.06720328330994 - type: dot_precision value: 51.01351351351351 - type: dot_recall value: 98.69281045751634 - type: euclidean_accuracy value: 57.65472312703584 - type: euclidean_accuracy_threshold value: 57.068902254104614 - type: euclidean_ap value: 55.055463204979915 - type: euclidean_f1 value: 67.26057906458797 - type: euclidean_f1_threshold value: 69.18495893478394 - type: euclidean_precision value: 51.01351351351351 - type: euclidean_recall value: 98.69281045751634 - type: main_score value: 55.17437273876084 - type: manhattan_accuracy value: 58.306188925081436 - type: manhattan_accuracy_threshold value: 1299.5019912719727 - type: manhattan_ap value: 55.17437273876084 - type: manhattan_f1 value: 67.25663716814158 - type: manhattan_f1_threshold value: 1602.2712707519531 - type: manhattan_precision value: 50.836120401337794 - type: manhattan_recall value: 99.34640522875817 - type: max_ap value: 55.17437273876084 - type: max_f1 value: 67.26057906458797 - type: max_precision value: 51.01351351351351 - type: max_recall value: 99.34640522875817 - type: similarity_accuracy value: 57.65472312703584 - type: similarity_accuracy_threshold value: 83.7156891822815 - type: similarity_ap value: 55.055463204979915 - type: similarity_f1 value: 67.26057906458797 - type: similarity_f1_threshold value: 76.06719732284546 - type: similarity_precision value: 51.01351351351351 - type: similarity_recall value: 98.69281045751634 task: type: PairClassification pipeline_tag: sentence-similarity tags: - sentence-transformers - feature-extraction - sentence-similarity - mteb --- -

KaLM-Embedding

**KaLM-Embedding** is a series of embedding models adapted from auto-regressive LLMs with superior training data. KaLM-embedding-multilingual-mini is trained from [Qwen/Qwen2-0.5B](https://huggingface.co./Qwen/Qwen2-0.5B) with massive weakly-supervised pre-training and supervised fine-tuning data. ## 📑 Open-source Plan - [x] Model Checkpoint - [x] [KaLM-embedding-multilingual-mini-v1](https://huggingface.co./HIT-TMG/KaLM-embedding-multilingual-mini-v1) - [x] [KaLM-embedding-multilingual-mini-instruct-v1](https://huggingface.co./HIT-TMG/KaLM-embedding-multilingual-mini-instruct-v1) - [x] [KaLM-embedding-multilingual-mini-instruct-v1.5](https://huggingface.co./HIT-TMG/KaLM-embedding-multilingual-mini-instruct-v1.5) - [ ] KaLM-embedding-multilingual-max-v1 - [x] Training and Evaluation Code: [HITsz-TMG/KaLM-Embedding](https://github.com/HITsz-TMG/KaLM-Embedding) - [x] Technical Report: [KaLM-Embedding: Superior Training Data Brings A Stronger Embedding Model](https://arxiv.org/abs/2501.01028) - [ ] Training Data ## Evaluation | Model Name | Model Size | C-MTEB(35) | MTEB(56) | avg |:----:|:---:|:---:|:---:|:---:| | [multilingual-e5-large](https://huggingface.co./intfloat/multilingual-e5-large) | 560M | 58.81 | 61.5 | 60.16 | [bge-m3 (dense)](https://huggingface.co./BAAI/bge-m3) | 560M | 60.80 | 59.84 | 60.32 | [gte-multilingual-base (dense)](https://huggingface.co./Alibaba-NLP/gte-multilingual-base) | **305M** | 62.72 | 61.40 | 62.06 | [KaLM-embedding-multilingual-mini-v1](https://huggingface.co./HIT-TMG/KaLM-embedding-multilingual-mini-v1) | 494M | 62.31 | 61.87 | 62.09 | [KaLM-embedding-multilingual-mini-instruct-v1](https://huggingface.co./HIT-TMG/KaLM-embedding-multilingual-mini-instruct-v1) | 494M | 63.57 | 64.74 | 64.16 | [KaLM-embedding-multilingual-mini-instruct-v1.5](https://huggingface.co./HIT-TMG/KaLM-embedding-multilingual-mini-instruct-v1.5) | 494M | **64.13** | **64.94** | **64.53** ## Requirements Since we have used the Qwen2 model, we advise you to install `transformers>=4.37.0`, or you might encounter the following error: ``` KeyError: 'qwen2' ``` ## Usage Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed: ``` pip install -U sentence-transformers ``` Then you can use the model like this: ```python from sentence_transformers import SentenceTransformer sentences = ["This is an example sentence", "Each sentence is converted"] model = SentenceTransformer('{MODEL_NAME_OR_PATH}') # Do NOT set trust_remote_code model.max_seq_length = 512 embeddings = model.encode( sentences, normalize_embeddings=True, batch_size=256, show_progress_bar=True ) print(embeddings) ``` We add instruction for asymmetric tasks: retrieval, reranking, classification and clustering. If you want to add instruction to the query (no instruction for the corpus), you can use the model like this: ```python from sentence_transformers import SentenceTransformer sentences = ["This is an example sentence", "Each sentence is converted"] model = SentenceTransformer('{MODEL_NAME_OR_PATH}') # Do NOT set trust_remote_code model.max_seq_length = 512 prompt = "Instruct: Classifying the category of french news. \n Query: " embeddings = model.encode( sentences, prompt=prompt, normalize_embeddings=True, batch_size=256, show_progress_bar=True ) print(embeddings) ``` ## Contact If you encounter any issue, feel free to contact us via the email: yanshek.woo@gmail.com