diff --git "a/README.md" "b/README.md" --- "a/README.md" +++ "b/README.md" @@ -1,3 +1,20211 @@ ---- -license: mit ---- +--- +license: mit +model-index: +- name: KaLM-Embedding + results: + - dataset: + config: en-ext + name: MTEB AmazonCounterfactualClassification (en-ext) + revision: e8379541af4e31359cca9fbcf4b00f2671dba205 + split: test + type: mteb/amazon_counterfactual + metrics: + - type: accuracy + value: 94.68515742128936 + - type: ap + value: 61.63091951041035 + - type: ap_weighted + value: 61.63091951041035 + - type: f1 + value: 87.0707144798333 + - type: f1_weighted + value: 94.92507672044346 + - type: main_score + value: 94.68515742128936 + task: + type: Classification + - dataset: + config: en + name: MTEB AmazonCounterfactualClassification (en) + revision: e8379541af4e31359cca9fbcf4b00f2671dba205 + split: test + type: mteb/amazon_counterfactual + metrics: + - type: accuracy + value: 91.73134328358209 + - type: ap + value: 68.35026743834399 + - type: ap_weighted + value: 68.35026743834399 + - type: f1 + value: 87.90511074439425 + - type: f1_weighted + value: 92.04737679749206 + - type: main_score + value: 91.73134328358209 + task: + type: Classification + - dataset: + config: default + name: MTEB AmazonPolarityClassification + revision: e2d317d38cd51312af73b3d32a06d1a08b442046 + split: test + type: mteb/amazon_polarity + metrics: + - type: accuracy + value: 96.55665 + - type: ap + value: 95.07549068622335 + - type: ap_weighted + value: 95.07549068622335 + - type: f1 + value: 96.55609959481485 + - type: f1_weighted + value: 96.55609959481488 + - type: main_score + value: 96.55665 + task: + type: Classification + - dataset: + config: en + name: MTEB AmazonReviewsClassification (en) + revision: 1399c76144fd37290681b995c656ef9b2e06e26d + split: test + type: mteb/amazon_reviews_multi + metrics: + - type: accuracy + value: 61.42000000000001 + - type: f1 + value: 59.435009609195 + - type: f1_weighted + value: 59.435009609195 + - type: main_score + value: 61.42000000000001 + task: + type: Classification + - dataset: + config: default + name: MTEB ArguAna + revision: c22ab2a51041ffd869aaddef7af8d8215647e41a + split: test + type: mteb/arguana + metrics: + - type: main_score + value: 58.626999999999995 + - type: map_at_1 + value: 33.073 + - type: map_at_10 + value: 49.632 + - type: map_at_100 + value: 50.33800000000001 + - type: map_at_1000 + value: 50.341 + - type: map_at_20 + value: 50.224000000000004 + - type: map_at_3 + value: 44.832 + - type: map_at_5 + value: 47.620000000000005 + - type: mrr_at_1 + value: 33.57041251778094 + - type: mrr_at_10 + value: 49.811380929802155 + - type: mrr_at_100 + value: 50.51757318808999 + - type: mrr_at_1000 + value: 50.52099783252713 + - type: mrr_at_20 + value: 50.40338101381789 + - type: mrr_at_3 + value: 45.00948316737797 + - type: mrr_at_5 + value: 47.77619724988161 + - type: nauc_map_at_1000_diff1 + value: 13.48102412913682 + - type: nauc_map_at_1000_max + value: -8.628057456116096 + - type: nauc_map_at_1000_std + value: -11.093713781480051 + - type: nauc_map_at_100_diff1 + value: 13.48687495587519 + - type: nauc_map_at_100_max + value: -8.618385685478875 + - type: nauc_map_at_100_std + value: -11.08784320605541 + - type: nauc_map_at_10_diff1 + value: 13.356168065508726 + - type: nauc_map_at_10_max + value: -8.497284129177803 + - type: nauc_map_at_10_std + value: -11.179349250075992 + - type: nauc_map_at_1_diff1 + value: 16.878392009779215 + - type: nauc_map_at_1_max + value: -10.140489051180705 + - type: nauc_map_at_1_std + value: -11.39244095065129 + - type: nauc_map_at_20_diff1 + value: 13.416069003092446 + - type: nauc_map_at_20_max + value: -8.581890449222536 + - type: nauc_map_at_20_std + value: -11.121016775402621 + - type: nauc_map_at_3_diff1 + value: 13.002710999935143 + - type: nauc_map_at_3_max + value: -9.061445259612642 + - type: nauc_map_at_3_std + value: -10.95371941334804 + - type: nauc_map_at_5_diff1 + value: 13.116608774616614 + - type: nauc_map_at_5_max + value: -9.031302466539435 + - type: nauc_map_at_5_std + value: -11.482820479074665 + - type: nauc_mrr_at_1000_diff1 + value: 12.009148181833913 + - type: nauc_mrr_at_1000_max + value: -9.133318660578208 + - type: nauc_mrr_at_1000_std + value: -11.090084522113825 + - type: nauc_mrr_at_100_diff1 + value: 12.015156727781342 + - type: nauc_mrr_at_100_max + value: -9.123605928993614 + - type: nauc_mrr_at_100_std + value: -11.084224457426195 + - type: nauc_mrr_at_10_diff1 + value: 11.928178469104141 + - type: nauc_mrr_at_10_max + value: -8.980239557009549 + - type: nauc_mrr_at_10_std + value: -11.174796827155468 + - type: nauc_mrr_at_1_diff1 + value: 15.42177954114532 + - type: nauc_mrr_at_1_max + value: -9.874364097911553 + - type: nauc_mrr_at_1_std + value: -11.176922083255894 + - type: nauc_mrr_at_20_diff1 + value: 11.950454910420659 + - type: nauc_mrr_at_20_max + value: -9.084711198849204 + - type: nauc_mrr_at_20_std + value: -11.117380723428798 + - type: nauc_mrr_at_3_diff1 + value: 11.669506159876851 + - type: nauc_mrr_at_3_max + value: -9.513795812047656 + - type: nauc_mrr_at_3_std + value: -10.945232338570069 + - type: nauc_mrr_at_5_diff1 + value: 11.587286193937736 + - type: nauc_mrr_at_5_max + value: -9.692616828656766 + - type: nauc_mrr_at_5_std + value: -11.52680193639791 + - type: nauc_ndcg_at_1000_diff1 + value: 13.115734924812534 + - type: nauc_ndcg_at_1000_max + value: -8.124440036807906 + - type: nauc_ndcg_at_1000_std + value: -10.823569251091254 + - type: nauc_ndcg_at_100_diff1 + value: 13.267338628811087 + - type: nauc_ndcg_at_100_max + value: -7.873039455281453 + - type: nauc_ndcg_at_100_std + value: -10.68251412725148 + - type: nauc_ndcg_at_10_diff1 + value: 12.46318225000743 + - type: nauc_ndcg_at_10_max + value: -7.328112147624243 + - type: nauc_ndcg_at_10_std + value: -11.267788232147327 + - type: nauc_ndcg_at_1_diff1 + value: 16.878392009779215 + - type: nauc_ndcg_at_1_max + value: -10.140489051180705 + - type: nauc_ndcg_at_1_std + value: -11.39244095065129 + - type: nauc_ndcg_at_20_diff1 + value: 12.777688953238583 + - type: nauc_ndcg_at_20_max + value: -7.553266861502126 + - type: nauc_ndcg_at_20_std + value: -10.926546866836999 + - type: nauc_ndcg_at_3_diff1 + value: 11.895421239753933 + - type: nauc_ndcg_at_3_max + value: -8.901090243657334 + - type: nauc_ndcg_at_3_std + value: -11.116058814242175 + - type: nauc_ndcg_at_5_diff1 + value: 11.98520372095743 + - type: nauc_ndcg_at_5_max + value: -8.780043333282341 + - type: nauc_ndcg_at_5_std + value: -11.985932726758405 + - type: nauc_precision_at_1000_diff1 + value: -8.254265037555717 + - type: nauc_precision_at_1000_max + value: 34.439879624882145 + - type: nauc_precision_at_1000_std + value: 86.59852334082694 + - type: nauc_precision_at_100_diff1 + value: 35.055902247981955 + - type: nauc_precision_at_100_max + value: 60.87922050664748 + - type: nauc_precision_at_100_std + value: 58.03454945999707 + - type: nauc_precision_at_10_diff1 + value: 6.653457872096913 + - type: nauc_precision_at_10_max + value: 2.2891313787635483 + - type: nauc_precision_at_10_std + value: -11.638523230893009 + - type: nauc_precision_at_1_diff1 + value: 16.878392009779215 + - type: nauc_precision_at_1_max + value: -10.140489051180705 + - type: nauc_precision_at_1_std + value: -11.39244095065129 + - type: nauc_precision_at_20_diff1 + value: 4.779328896089478 + - type: nauc_precision_at_20_max + value: 14.224776184784957 + - type: nauc_precision_at_20_std + value: -4.21463291230828 + - type: nauc_precision_at_3_diff1 + value: 8.514820657150233 + - type: nauc_precision_at_3_max + value: -8.469060860049108 + - type: nauc_precision_at_3_std + value: -11.7235889812276 + - type: nauc_precision_at_5_diff1 + value: 7.710375928363608 + - type: nauc_precision_at_5_max + value: -7.833484980907636 + - type: nauc_precision_at_5_std + value: -14.090406741382132 + - type: nauc_recall_at_1000_diff1 + value: -8.254265037555465 + - type: nauc_recall_at_1000_max + value: 34.43987962487738 + - type: nauc_recall_at_1000_std + value: 86.59852334082566 + - type: nauc_recall_at_100_diff1 + value: 35.05590224798152 + - type: nauc_recall_at_100_max + value: 60.87922050664376 + - type: nauc_recall_at_100_std + value: 58.03454945999553 + - type: nauc_recall_at_10_diff1 + value: 6.653457872096784 + - type: nauc_recall_at_10_max + value: 2.2891313787632592 + - type: nauc_recall_at_10_std + value: -11.638523230893094 + - type: nauc_recall_at_1_diff1 + value: 16.878392009779215 + - type: nauc_recall_at_1_max + value: -10.140489051180705 + - type: nauc_recall_at_1_std + value: -11.39244095065129 + - type: nauc_recall_at_20_diff1 + value: 4.77932889608973 + - type: nauc_recall_at_20_max + value: 14.224776184784938 + - type: nauc_recall_at_20_std + value: -4.2146329123081125 + - type: nauc_recall_at_3_diff1 + value: 8.514820657150297 + - type: nauc_recall_at_3_max + value: -8.469060860049037 + - type: nauc_recall_at_3_std + value: -11.723588981227591 + - type: nauc_recall_at_5_diff1 + value: 7.710375928363697 + - type: nauc_recall_at_5_max + value: -7.833484980907617 + - type: nauc_recall_at_5_std + value: -14.090406741382106 + - type: ndcg_at_1 + value: 33.073 + - type: ndcg_at_10 + value: 58.626999999999995 + - type: ndcg_at_100 + value: 61.449 + - type: ndcg_at_1000 + value: 61.516999999999996 + - type: ndcg_at_20 + value: 60.712999999999994 + - type: ndcg_at_3 + value: 48.730000000000004 + - type: ndcg_at_5 + value: 53.791 + - type: precision_at_1 + value: 33.073 + - type: precision_at_10 + value: 8.727 + - type: precision_at_100 + value: 0.991 + - type: precision_at_1000 + value: 0.1 + - type: precision_at_20 + value: 4.769 + - type: precision_at_3 + value: 20.009 + - type: precision_at_5 + value: 14.481 + - type: recall_at_1 + value: 33.073 + - type: recall_at_10 + value: 87.26899999999999 + - type: recall_at_100 + value: 99.14699999999999 + - type: recall_at_1000 + value: 99.644 + - type: recall_at_20 + value: 95.377 + - type: recall_at_3 + value: 60.028000000000006 + - type: recall_at_5 + value: 72.404 + task: + type: Retrieval + - dataset: + config: default + name: MTEB ArxivClusteringP2P + revision: a122ad7f3f0291bf49cc6f4d32aa80929df69d5d + split: test + type: mteb/arxiv-clustering-p2p + metrics: + - type: main_score + value: 49.682199850672866 + - type: v_measure + value: 49.682199850672866 + - type: v_measure_std + value: 14.233482745120106 + task: + type: Clustering + - dataset: + config: default + name: MTEB ArxivClusteringS2S + revision: f910caf1a6075f7329cdf8c1a6135696f37dbd53 + split: test + type: mteb/arxiv-clustering-s2s + metrics: + - type: main_score + value: 42.209464702274715 + - type: v_measure + value: 42.209464702274715 + - type: v_measure_std + value: 14.710188553304171 + task: + type: Clustering + - dataset: + config: default + name: MTEB AskUbuntuDupQuestions + revision: 2000358ca161889fa9c082cb41daa8dcfb161a54 + split: test + type: mteb/askubuntudupquestions-reranking + metrics: + - type: main_score + value: 60.34670620892353 + - type: map + value: 60.34670620892353 + - type: mrr + value: 74.21360418590336 + - type: nAUC_map_diff1 + value: 11.911350612613562 + - type: nAUC_map_max + value: 16.783478049340925 + - type: nAUC_map_std + value: 16.428460426761752 + - type: nAUC_mrr_diff1 + value: 14.989972807815782 + - type: nAUC_mrr_max + value: 25.277551101334105 + - type: nAUC_mrr_std + value: 16.506353229551408 + task: + type: Reranking + - dataset: + config: default + name: MTEB BIOSSES + revision: d3fb88f8f02e40887cd149695127462bbcf29b4a + split: test + type: mteb/biosses-sts + metrics: + - type: cosine_pearson + value: 88.03646117514994 + - type: cosine_spearman + value: 86.1423242570783 + - type: euclidean_pearson + value: 87.25661641211398 + - type: euclidean_spearman + value: 86.1423242570783 + - type: main_score + value: 86.1423242570783 + - type: manhattan_pearson + value: 86.75341029385434 + - type: manhattan_spearman + value: 85.43447680299268 + - type: pearson + value: 88.03646117514994 + - type: spearman + value: 86.1423242570783 + task: + type: STS + - dataset: + config: default + name: MTEB Banking77Classification + revision: 0fd18e25b25c072e09e0d92ab615fda904d66300 + split: test + type: mteb/banking77 + metrics: + - type: accuracy + value: 84.54220779220778 + - type: f1 + value: 83.93093828135092 + - type: f1_weighted + value: 83.9309382813509 + - type: main_score + value: 84.54220779220778 + task: + type: Classification + - dataset: + config: default + name: MTEB BiorxivClusteringP2P + revision: 65b79d1d13f80053f67aca9498d9402c2d9f1f40 + split: test + type: mteb/biorxiv-clustering-p2p + metrics: + - type: main_score + value: 43.84044619752363 + - type: v_measure + value: 43.84044619752363 + - type: v_measure_std + value: 0.4618669625794188 + task: + type: Clustering + - dataset: + config: default + name: MTEB BiorxivClusteringS2S + revision: 258694dd0231531bc1fd9de6ceb52a0853c6d908 + split: test + type: mteb/biorxiv-clustering-s2s + metrics: + - type: main_score + value: 37.31116089915042 + - type: v_measure + value: 37.31116089915042 + - type: v_measure_std + value: 0.5977139350402765 + task: + type: Clustering + - dataset: + config: default + name: MTEB CQADupstackAndroidRetrieval + revision: f46a197baaae43b4f621051089b82a364682dfeb + split: test + type: mteb/cqadupstack-android + metrics: + - type: main_score + value: 52.39 + - type: map_at_1 + value: 34.169 + - type: map_at_10 + value: 45.942 + - type: map_at_100 + value: 47.404 + - type: map_at_1000 + value: 47.514 + - type: map_at_20 + value: 46.778 + - type: map_at_3 + value: 42.336 + - type: map_at_5 + value: 44.415 + - type: mrr_at_1 + value: 40.772532188841204 + - type: mrr_at_10 + value: 51.53978472647998 + - type: mrr_at_100 + value: 52.151143552536816 + - type: mrr_at_1000 + value: 52.187870164237204 + - type: mrr_at_20 + value: 51.901680031986686 + - type: mrr_at_3 + value: 48.90319504053407 + - type: mrr_at_5 + value: 50.448259418216466 + - type: nauc_map_at_1000_diff1 + value: 56.78205111804614 + - type: nauc_map_at_1000_max + value: 36.375389301093755 + - type: nauc_map_at_1000_std + value: -5.9566477904567625 + - type: nauc_map_at_100_diff1 + value: 56.78944020915762 + - type: nauc_map_at_100_max + value: 36.39973376657996 + - type: nauc_map_at_100_std + value: -5.902276438746319 + - type: nauc_map_at_10_diff1 + value: 57.01302040488429 + - type: nauc_map_at_10_max + value: 35.73327053443045 + - type: nauc_map_at_10_std + value: -6.935563186512875 + - type: nauc_map_at_1_diff1 + value: 62.1943454407426 + - type: nauc_map_at_1_max + value: 31.106713330762027 + - type: nauc_map_at_1_std + value: -9.349274581003204 + - type: nauc_map_at_20_diff1 + value: 56.96527745781459 + - type: nauc_map_at_20_max + value: 36.1099391825394 + - type: nauc_map_at_20_std + value: -6.148724378898889 + - type: nauc_map_at_3_diff1 + value: 57.97922886237535 + - type: nauc_map_at_3_max + value: 34.98388578365086 + - type: nauc_map_at_3_std + value: -7.868364460415703 + - type: nauc_map_at_5_diff1 + value: 57.19907689819566 + - type: nauc_map_at_5_max + value: 35.070877663024426 + - type: nauc_map_at_5_std + value: -7.643819808793027 + - type: nauc_mrr_at_1000_diff1 + value: 53.9572914102343 + - type: nauc_mrr_at_1000_max + value: 36.372803215838125 + - type: nauc_mrr_at_1000_std + value: -6.847011087249551 + - type: nauc_mrr_at_100_diff1 + value: 53.95057299284283 + - type: nauc_mrr_at_100_max + value: 36.374095119115765 + - type: nauc_mrr_at_100_std + value: -6.8350087107843835 + - type: nauc_mrr_at_10_diff1 + value: 53.77540722618102 + - type: nauc_mrr_at_10_max + value: 36.34439189912393 + - type: nauc_mrr_at_10_std + value: -7.189115725803416 + - type: nauc_mrr_at_1_diff1 + value: 58.516667270098445 + - type: nauc_mrr_at_1_max + value: 35.99253476386456 + - type: nauc_mrr_at_1_std + value: -8.950540274950663 + - type: nauc_mrr_at_20_diff1 + value: 53.92501674276654 + - type: nauc_mrr_at_20_max + value: 36.31325682241623 + - type: nauc_mrr_at_20_std + value: -6.842193524470379 + - type: nauc_mrr_at_3_diff1 + value: 54.30587263243055 + - type: nauc_mrr_at_3_max + value: 36.67164288224585 + - type: nauc_mrr_at_3_std + value: -6.720909588714866 + - type: nauc_mrr_at_5_diff1 + value: 53.80277965232069 + - type: nauc_mrr_at_5_max + value: 36.026610354561726 + - type: nauc_mrr_at_5_std + value: -7.405548968254587 + - type: nauc_ndcg_at_1000_diff1 + value: 54.32192669123723 + - type: nauc_ndcg_at_1000_max + value: 37.06057531293069 + - type: nauc_ndcg_at_1000_std + value: -3.858161870894338 + - type: nauc_ndcg_at_100_diff1 + value: 54.19634435723086 + - type: nauc_ndcg_at_100_max + value: 37.47554574823893 + - type: nauc_ndcg_at_100_std + value: -2.782934190456467 + - type: nauc_ndcg_at_10_diff1 + value: 53.97003966265566 + - type: nauc_ndcg_at_10_max + value: 36.15287682329226 + - type: nauc_ndcg_at_10_std + value: -5.669539508175427 + - type: nauc_ndcg_at_1_diff1 + value: 58.516667270098445 + - type: nauc_ndcg_at_1_max + value: 35.99253476386456 + - type: nauc_ndcg_at_1_std + value: -8.950540274950663 + - type: nauc_ndcg_at_20_diff1 + value: 54.34922000376914 + - type: nauc_ndcg_at_20_max + value: 36.25023728047867 + - type: nauc_ndcg_at_20_std + value: -3.810948512343384 + - type: nauc_ndcg_at_3_diff1 + value: 54.83758663042578 + - type: nauc_ndcg_at_3_max + value: 36.85494351712768 + - type: nauc_ndcg_at_3_std + value: -5.525443708811114 + - type: nauc_ndcg_at_5_diff1 + value: 54.191751120186304 + - type: nauc_ndcg_at_5_max + value: 35.775634699712214 + - type: nauc_ndcg_at_5_std + value: -6.139092418343521 + - type: nauc_precision_at_1000_diff1 + value: -25.057017986568535 + - type: nauc_precision_at_1000_max + value: -6.253611962247419 + - type: nauc_precision_at_1000_std + value: -3.576821263647792 + - type: nauc_precision_at_100_diff1 + value: -15.890618283335048 + - type: nauc_precision_at_100_max + value: 9.2689967407046 + - type: nauc_precision_at_100_std + value: 9.941022837167951 + - type: nauc_precision_at_10_diff1 + value: 6.515251734960673 + - type: nauc_precision_at_10_max + value: 26.00954176926215 + - type: nauc_precision_at_10_std + value: 6.18066523804505 + - type: nauc_precision_at_1_diff1 + value: 58.516667270098445 + - type: nauc_precision_at_1_max + value: 35.99253476386456 + - type: nauc_precision_at_1_std + value: -8.950540274950663 + - type: nauc_precision_at_20_diff1 + value: -2.207402052545381 + - type: nauc_precision_at_20_max + value: 19.722097586131714 + - type: nauc_precision_at_20_std + value: 11.781955236419435 + - type: nauc_precision_at_3_diff1 + value: 28.669702552163834 + - type: nauc_precision_at_3_max + value: 36.374241470537925 + - type: nauc_precision_at_3_std + value: 0.5810060004219123 + - type: nauc_precision_at_5_diff1 + value: 18.454459967860064 + - type: nauc_precision_at_5_max + value: 30.585219727516012 + - type: nauc_precision_at_5_std + value: 2.034776181342522 + - type: nauc_recall_at_1000_diff1 + value: 32.56442385278414 + - type: nauc_recall_at_1000_max + value: 51.480513202995915 + - type: nauc_recall_at_1000_std + value: 49.27851711330113 + - type: nauc_recall_at_100_diff1 + value: 41.029807873528796 + - type: nauc_recall_at_100_max + value: 42.572875140257565 + - type: nauc_recall_at_100_std + value: 23.745431356647018 + - type: nauc_recall_at_10_diff1 + value: 44.87576445883134 + - type: nauc_recall_at_10_max + value: 31.525509952765297 + - type: nauc_recall_at_10_std + value: -2.8068541204926394 + - type: nauc_recall_at_1_diff1 + value: 62.1943454407426 + - type: nauc_recall_at_1_max + value: 31.106713330762027 + - type: nauc_recall_at_1_std + value: -9.349274581003204 + - type: nauc_recall_at_20_diff1 + value: 44.969586307113104 + - type: nauc_recall_at_20_max + value: 31.29074006190112 + - type: nauc_recall_at_20_std + value: 5.946087034639837 + - type: nauc_recall_at_3_diff1 + value: 51.6957387291602 + - type: nauc_recall_at_3_max + value: 32.97568422788289 + - type: nauc_recall_at_3_std + value: -4.466963198284326 + - type: nauc_recall_at_5_diff1 + value: 47.596245200271184 + - type: nauc_recall_at_5_max + value: 30.791573693357755 + - type: nauc_recall_at_5_std + value: -4.702986567079468 + - type: ndcg_at_1 + value: 40.772999999999996 + - type: ndcg_at_10 + value: 52.39 + - type: ndcg_at_100 + value: 57.4 + - type: ndcg_at_1000 + value: 59.12500000000001 + - type: ndcg_at_20 + value: 54.459 + - type: ndcg_at_3 + value: 47.339999999999996 + - type: ndcg_at_5 + value: 49.717 + - type: precision_at_1 + value: 40.772999999999996 + - type: precision_at_10 + value: 10.029 + - type: precision_at_100 + value: 1.552 + - type: precision_at_1000 + value: 0.199 + - type: precision_at_20 + value: 5.937 + - type: precision_at_3 + value: 22.938 + - type: precision_at_5 + value: 16.423 + - type: recall_at_1 + value: 34.169 + - type: recall_at_10 + value: 64.68199999999999 + - type: recall_at_100 + value: 85.497 + - type: recall_at_1000 + value: 96.391 + - type: recall_at_20 + value: 72.053 + - type: recall_at_3 + value: 50.059 + - type: recall_at_5 + value: 56.589999999999996 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackEnglishRetrieval + revision: ad9991cb51e31e31e430383c75ffb2885547b5f0 + split: test + type: mteb/cqadupstack-english + metrics: + - type: main_score + value: 47.753 + - type: map_at_1 + value: 31.211 + - type: map_at_10 + value: 42.015 + - type: map_at_100 + value: 43.262 + - type: map_at_1000 + value: 43.39 + - type: map_at_20 + value: 42.697 + - type: map_at_3 + value: 39.034 + - type: map_at_5 + value: 40.729 + - type: mrr_at_1 + value: 39.29936305732484 + - type: mrr_at_10 + value: 47.96489232635733 + - type: mrr_at_100 + value: 48.64731557532463 + - type: mrr_at_1000 + value: 48.68894277936992 + - type: mrr_at_20 + value: 48.39750821161911 + - type: mrr_at_3 + value: 45.849256900212346 + - type: mrr_at_5 + value: 47.14225053078562 + - type: nauc_map_at_1000_diff1 + value: 52.24106468332894 + - type: nauc_map_at_1000_max + value: 38.375619410420775 + - type: nauc_map_at_1000_std + value: -4.482915396811548 + - type: nauc_map_at_100_diff1 + value: 52.24328589744081 + - type: nauc_map_at_100_max + value: 38.29250166419244 + - type: nauc_map_at_100_std + value: -4.583741891930366 + - type: nauc_map_at_10_diff1 + value: 52.454090722359666 + - type: nauc_map_at_10_max + value: 37.7173958399999 + - type: nauc_map_at_10_std + value: -5.860564273083544 + - type: nauc_map_at_1_diff1 + value: 57.37978281090314 + - type: nauc_map_at_1_max + value: 32.49915109819217 + - type: nauc_map_at_1_std + value: -11.254055060290668 + - type: nauc_map_at_20_diff1 + value: 52.37379963537071 + - type: nauc_map_at_20_max + value: 37.943912770037116 + - type: nauc_map_at_20_std + value: -5.26512232555932 + - type: nauc_map_at_3_diff1 + value: 53.315496436307974 + - type: nauc_map_at_3_max + value: 35.798184763155234 + - type: nauc_map_at_3_std + value: -8.424700965366764 + - type: nauc_map_at_5_diff1 + value: 52.65283986998911 + - type: nauc_map_at_5_max + value: 36.545213941567425 + - type: nauc_map_at_5_std + value: -7.257895143708675 + - type: nauc_mrr_at_1000_diff1 + value: 50.16585921071199 + - type: nauc_mrr_at_1000_max + value: 40.98958361489588 + - type: nauc_mrr_at_1000_std + value: 0.5225503686128239 + - type: nauc_mrr_at_100_diff1 + value: 50.16290335172945 + - type: nauc_mrr_at_100_max + value: 40.983188618358675 + - type: nauc_mrr_at_100_std + value: 0.5371456026411485 + - type: nauc_mrr_at_10_diff1 + value: 50.188741789601664 + - type: nauc_mrr_at_10_max + value: 41.20385325286603 + - type: nauc_mrr_at_10_std + value: 0.4299969547822729 + - type: nauc_mrr_at_1_diff1 + value: 52.89957788273348 + - type: nauc_mrr_at_1_max + value: 40.15889247486093 + - type: nauc_mrr_at_1_std + value: -2.3643009397477233 + - type: nauc_mrr_at_20_diff1 + value: 50.18924356147879 + - type: nauc_mrr_at_20_max + value: 41.03580332941115 + - type: nauc_mrr_at_20_std + value: 0.5030743116655743 + - type: nauc_mrr_at_3_diff1 + value: 50.2595645561214 + - type: nauc_mrr_at_3_max + value: 40.84037333925909 + - type: nauc_mrr_at_3_std + value: -0.47444597498111607 + - type: nauc_mrr_at_5_diff1 + value: 50.30028806022225 + - type: nauc_mrr_at_5_max + value: 40.94735400001521 + - type: nauc_mrr_at_5_std + value: 0.014502584942351543 + - type: nauc_ndcg_at_1000_diff1 + value: 50.02853655454531 + - type: nauc_ndcg_at_1000_max + value: 40.0601698049325 + - type: nauc_ndcg_at_1000_std + value: 0.5989970984573991 + - type: nauc_ndcg_at_100_diff1 + value: 49.99180232217219 + - type: nauc_ndcg_at_100_max + value: 39.828019879089595 + - type: nauc_ndcg_at_100_std + value: 0.5679819431802621 + - type: nauc_ndcg_at_10_diff1 + value: 50.49123456226574 + - type: nauc_ndcg_at_10_max + value: 40.24693971238414 + - type: nauc_ndcg_at_10_std + value: -1.6801008721493143 + - type: nauc_ndcg_at_1_diff1 + value: 52.89957788273348 + - type: nauc_ndcg_at_1_max + value: 40.15889247486093 + - type: nauc_ndcg_at_1_std + value: -2.3643009397477233 + - type: nauc_ndcg_at_20_diff1 + value: 50.50251680139964 + - type: nauc_ndcg_at_20_max + value: 39.72201768032907 + - type: nauc_ndcg_at_20_std + value: -1.0868971947199884 + - type: nauc_ndcg_at_3_diff1 + value: 50.58336010302361 + - type: nauc_ndcg_at_3_max + value: 39.56460688076086 + - type: nauc_ndcg_at_3_std + value: -3.192413421384452 + - type: nauc_ndcg_at_5_diff1 + value: 50.670426480094 + - type: nauc_ndcg_at_5_max + value: 39.30101734473154 + - type: nauc_ndcg_at_5_std + value: -3.133266437331269 + - type: nauc_precision_at_1000_diff1 + value: -9.966142918444545 + - type: nauc_precision_at_1000_max + value: 15.887181550157312 + - type: nauc_precision_at_1000_std + value: 30.07775267584185 + - type: nauc_precision_at_100_diff1 + value: -5.39669332549404 + - type: nauc_precision_at_100_max + value: 26.372684632284045 + - type: nauc_precision_at_100_std + value: 35.21702840254392 + - type: nauc_precision_at_10_diff1 + value: 11.258611067705424 + - type: nauc_precision_at_10_max + value: 38.49722079603548 + - type: nauc_precision_at_10_std + value: 23.96075630064515 + - type: nauc_precision_at_1_diff1 + value: 52.89957788273348 + - type: nauc_precision_at_1_max + value: 40.15889247486093 + - type: nauc_precision_at_1_std + value: -2.3643009397477233 + - type: nauc_precision_at_20_diff1 + value: 5.119010028728253 + - type: nauc_precision_at_20_max + value: 33.86149623123613 + - type: nauc_precision_at_20_std + value: 28.568877042019192 + - type: nauc_precision_at_3_diff1 + value: 28.335387516697054 + - type: nauc_precision_at_3_max + value: 39.550279228497395 + - type: nauc_precision_at_3_std + value: 10.53626944081313 + - type: nauc_precision_at_5_diff1 + value: 20.067823876439405 + - type: nauc_precision_at_5_max + value: 38.012587686917584 + - type: nauc_precision_at_5_std + value: 15.994020926756388 + - type: nauc_recall_at_1000_diff1 + value: 32.76791129120816 + - type: nauc_recall_at_1000_max + value: 37.496559622674766 + - type: nauc_recall_at_1000_std + value: 25.655864139857222 + - type: nauc_recall_at_100_diff1 + value: 38.447827716104584 + - type: nauc_recall_at_100_max + value: 36.474274046070214 + - type: nauc_recall_at_100_std + value: 16.203946452884637 + - type: nauc_recall_at_10_diff1 + value: 45.04137348043349 + - type: nauc_recall_at_10_max + value: 38.413608602825924 + - type: nauc_recall_at_10_std + value: 0.8596466671993844 + - type: nauc_recall_at_1_diff1 + value: 57.37978281090314 + - type: nauc_recall_at_1_max + value: 32.49915109819217 + - type: nauc_recall_at_1_std + value: -11.254055060290668 + - type: nauc_recall_at_20_diff1 + value: 43.22936216632159 + - type: nauc_recall_at_20_max + value: 36.644412179031946 + - type: nauc_recall_at_20_std + value: 4.295238193474559 + - type: nauc_recall_at_3_diff1 + value: 48.45830999117096 + - type: nauc_recall_at_3_max + value: 34.59148426593566 + - type: nauc_recall_at_3_std + value: -7.2544944905307425 + - type: nauc_recall_at_5_diff1 + value: 46.818595627596906 + - type: nauc_recall_at_5_max + value: 35.28346610822803 + - type: nauc_recall_at_5_std + value: -4.578586774496801 + - type: ndcg_at_1 + value: 39.299 + - type: ndcg_at_10 + value: 47.753 + - type: ndcg_at_100 + value: 52.146 + - type: ndcg_at_1000 + value: 54.13400000000001 + - type: ndcg_at_20 + value: 49.573 + - type: ndcg_at_3 + value: 43.783 + - type: ndcg_at_5 + value: 45.638 + - type: precision_at_1 + value: 39.299 + - type: precision_at_10 + value: 8.955 + - type: precision_at_100 + value: 1.427 + - type: precision_at_1000 + value: 0.189 + - type: precision_at_20 + value: 5.252 + - type: precision_at_3 + value: 21.295 + - type: precision_at_5 + value: 14.943000000000001 + - type: recall_at_1 + value: 31.211 + - type: recall_at_10 + value: 57.831999999999994 + - type: recall_at_100 + value: 76.211 + - type: recall_at_1000 + value: 88.803 + - type: recall_at_20 + value: 64.548 + - type: recall_at_3 + value: 45.909 + - type: recall_at_5 + value: 51.304 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackGamingRetrieval + revision: 4885aa143210c98657558c04aaf3dc47cfb54340 + split: test + type: mteb/cqadupstack-gaming + metrics: + - type: main_score + value: 61.58 + - type: map_at_1 + value: 42.294 + - type: map_at_10 + value: 55.625 + - type: map_at_100 + value: 56.618 + - type: map_at_1000 + value: 56.672 + - type: map_at_20 + value: 56.255 + - type: map_at_3 + value: 52.539 + - type: map_at_5 + value: 54.118 + - type: mrr_at_1 + value: 48.08777429467085 + - type: mrr_at_10 + value: 58.84965417724045 + - type: mrr_at_100 + value: 59.43662069430218 + - type: mrr_at_1000 + value: 59.46383163108614 + - type: mrr_at_20 + value: 59.238920207635395 + - type: mrr_at_3 + value: 56.645768025078446 + - type: mrr_at_5 + value: 57.799373040752464 + - type: nauc_map_at_1000_diff1 + value: 55.574840425792594 + - type: nauc_map_at_1000_max + value: 34.4701847562437 + - type: nauc_map_at_1000_std + value: -9.755236496635238 + - type: nauc_map_at_100_diff1 + value: 55.56633342053875 + - type: nauc_map_at_100_max + value: 34.463217368520056 + - type: nauc_map_at_100_std + value: -9.736389955691903 + - type: nauc_map_at_10_diff1 + value: 55.377779197086085 + - type: nauc_map_at_10_max + value: 34.119042879932806 + - type: nauc_map_at_10_std + value: -10.34552914025853 + - type: nauc_map_at_1_diff1 + value: 59.39306584705475 + - type: nauc_map_at_1_max + value: 29.113177128377604 + - type: nauc_map_at_1_std + value: -11.444784267280179 + - type: nauc_map_at_20_diff1 + value: 55.5279849352151 + - type: nauc_map_at_20_max + value: 34.40776911487248 + - type: nauc_map_at_20_std + value: -9.980725430581513 + - type: nauc_map_at_3_diff1 + value: 56.36831487626787 + - type: nauc_map_at_3_max + value: 32.50972201654767 + - type: nauc_map_at_3_std + value: -12.162628011645566 + - type: nauc_map_at_5_diff1 + value: 55.61245893128041 + - type: nauc_map_at_5_max + value: 33.29312656827763 + - type: nauc_map_at_5_std + value: -11.4951400718427 + - type: nauc_mrr_at_1000_diff1 + value: 54.90317690493004 + - type: nauc_mrr_at_1000_max + value: 35.56498085453685 + - type: nauc_mrr_at_1000_std + value: -9.234699847400888 + - type: nauc_mrr_at_100_diff1 + value: 54.896346456934275 + - type: nauc_mrr_at_100_max + value: 35.58324293276136 + - type: nauc_mrr_at_100_std + value: -9.215853996857613 + - type: nauc_mrr_at_10_diff1 + value: 54.671458608944114 + - type: nauc_mrr_at_10_max + value: 35.56708736221525 + - type: nauc_mrr_at_10_std + value: -9.373432397620997 + - type: nauc_mrr_at_1_diff1 + value: 58.79146952147216 + - type: nauc_mrr_at_1_max + value: 33.869452831902805 + - type: nauc_mrr_at_1_std + value: -10.566480052563577 + - type: nauc_mrr_at_20_diff1 + value: 54.849781607868906 + - type: nauc_mrr_at_20_max + value: 35.617115851895484 + - type: nauc_mrr_at_20_std + value: -9.208616079050708 + - type: nauc_mrr_at_3_diff1 + value: 55.16070050179383 + - type: nauc_mrr_at_3_max + value: 35.177757122772135 + - type: nauc_mrr_at_3_std + value: -10.393796685155891 + - type: nauc_mrr_at_5_diff1 + value: 54.828393662595445 + - type: nauc_mrr_at_5_max + value: 35.56940994194972 + - type: nauc_mrr_at_5_std + value: -9.731305997976602 + - type: nauc_ndcg_at_1000_diff1 + value: 54.50183704484519 + - type: nauc_ndcg_at_1000_max + value: 36.36712113663723 + - type: nauc_ndcg_at_1000_std + value: -7.34102789192482 + - type: nauc_ndcg_at_100_diff1 + value: 54.455028493563226 + - type: nauc_ndcg_at_100_max + value: 36.78883717977845 + - type: nauc_ndcg_at_100_std + value: -6.597339807115783 + - type: nauc_ndcg_at_10_diff1 + value: 53.536209026389535 + - type: nauc_ndcg_at_10_max + value: 36.18315401424091 + - type: nauc_ndcg_at_10_std + value: -8.273690022713248 + - type: nauc_ndcg_at_1_diff1 + value: 58.79146952147216 + - type: nauc_ndcg_at_1_max + value: 33.869452831902805 + - type: nauc_ndcg_at_1_std + value: -10.566480052563577 + - type: nauc_ndcg_at_20_diff1 + value: 54.15670848398847 + - type: nauc_ndcg_at_20_max + value: 36.74491143155494 + - type: nauc_ndcg_at_20_std + value: -7.313843626081678 + - type: nauc_ndcg_at_3_diff1 + value: 55.036932320029564 + - type: nauc_ndcg_at_3_max + value: 33.91087214672552 + - type: nauc_ndcg_at_3_std + value: -11.75293941895024 + - type: nauc_ndcg_at_5_diff1 + value: 54.063110780501546 + - type: nauc_ndcg_at_5_max + value: 35.052268948433515 + - type: nauc_ndcg_at_5_std + value: -10.452241125468108 + - type: nauc_precision_at_1000_diff1 + value: -17.38384602357622 + - type: nauc_precision_at_1000_max + value: 9.710315321476479 + - type: nauc_precision_at_1000_std + value: 16.90078109961968 + - type: nauc_precision_at_100_diff1 + value: -10.339926865487238 + - type: nauc_precision_at_100_max + value: 15.519587672026788 + - type: nauc_precision_at_100_std + value: 20.154368516383574 + - type: nauc_precision_at_10_diff1 + value: 8.295704958690228 + - type: nauc_precision_at_10_max + value: 27.223683123639592 + - type: nauc_precision_at_10_std + value: 9.078686113987018 + - type: nauc_precision_at_1_diff1 + value: 58.79146952147216 + - type: nauc_precision_at_1_max + value: 33.869452831902805 + - type: nauc_precision_at_1_std + value: -10.566480052563577 + - type: nauc_precision_at_20_diff1 + value: 1.206430229577525 + - type: nauc_precision_at_20_max + value: 23.944135447502873 + - type: nauc_precision_at_20_std + value: 14.990573969863142 + - type: nauc_precision_at_3_diff1 + value: 30.534879329433494 + - type: nauc_precision_at_3_max + value: 31.45998483553432 + - type: nauc_precision_at_3_std + value: -5.843145834483077 + - type: nauc_precision_at_5_diff1 + value: 20.082739223007245 + - type: nauc_precision_at_5_max + value: 29.165093621422066 + - type: nauc_precision_at_5_std + value: -0.628093664798173 + - type: nauc_recall_at_1000_diff1 + value: 29.998670252945253 + - type: nauc_recall_at_1000_max + value: 58.601149100236924 + - type: nauc_recall_at_1000_std + value: 51.912385311841525 + - type: nauc_recall_at_100_diff1 + value: 46.15565003366627 + - type: nauc_recall_at_100_max + value: 49.740072860097115 + - type: nauc_recall_at_100_std + value: 21.303498072710514 + - type: nauc_recall_at_10_diff1 + value: 44.934728375231664 + - type: nauc_recall_at_10_max + value: 39.7256315576848 + - type: nauc_recall_at_10_std + value: -1.5224958573376588 + - type: nauc_recall_at_1_diff1 + value: 59.39306584705475 + - type: nauc_recall_at_1_max + value: 29.113177128377604 + - type: nauc_recall_at_1_std + value: -11.444784267280179 + - type: nauc_recall_at_20_diff1 + value: 45.98017763702076 + - type: nauc_recall_at_20_max + value: 43.26825736903793 + - type: nauc_recall_at_20_std + value: 5.298466215553723 + - type: nauc_recall_at_3_diff1 + value: 51.290757427485445 + - type: nauc_recall_at_3_max + value: 33.21715365617901 + - type: nauc_recall_at_3_std + value: -12.227694565360014 + - type: nauc_recall_at_5_diff1 + value: 48.150864238331714 + - type: nauc_recall_at_5_max + value: 36.177367874135335 + - type: nauc_recall_at_5_std + value: -9.091237939153817 + - type: ndcg_at_1 + value: 48.087999999999994 + - type: ndcg_at_10 + value: 61.58 + - type: ndcg_at_100 + value: 65.253 + - type: ndcg_at_1000 + value: 66.267 + - type: ndcg_at_20 + value: 63.327999999999996 + - type: ndcg_at_3 + value: 56.496 + - type: ndcg_at_5 + value: 58.667 + - type: precision_at_1 + value: 48.087999999999994 + - type: precision_at_10 + value: 9.824 + - type: precision_at_100 + value: 1.262 + - type: precision_at_1000 + value: 0.13799999999999998 + - type: precision_at_20 + value: 5.4670000000000005 + - type: precision_at_3 + value: 25.266 + - type: precision_at_5 + value: 16.915 + - type: recall_at_1 + value: 42.294 + - type: recall_at_10 + value: 75.761 + - type: recall_at_100 + value: 91.208 + - type: recall_at_1000 + value: 98.37700000000001 + - type: recall_at_20 + value: 82.113 + - type: recall_at_3 + value: 61.956999999999994 + - type: recall_at_5 + value: 67.437 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackGisRetrieval + revision: 5003b3064772da1887988e05400cf3806fe491f2 + split: test + type: mteb/cqadupstack-gis + metrics: + - type: main_score + value: 39.837 + - type: map_at_1 + value: 25.764 + - type: map_at_10 + value: 34.829 + - type: map_at_100 + value: 35.869 + - type: map_at_1000 + value: 35.953 + - type: map_at_20 + value: 35.393 + - type: map_at_3 + value: 32.208 + - type: map_at_5 + value: 33.641 + - type: mrr_at_1 + value: 28.0225988700565 + - type: mrr_at_10 + value: 37.04515290108509 + - type: mrr_at_100 + value: 37.93924588681648 + - type: mrr_at_1000 + value: 37.99860736806393 + - type: mrr_at_20 + value: 37.526581494375456 + - type: mrr_at_3 + value: 34.59510357815443 + - type: mrr_at_5 + value: 35.88323917137474 + - type: nauc_map_at_1000_diff1 + value: 39.89881614209902 + - type: nauc_map_at_1000_max + value: 26.20512364416086 + - type: nauc_map_at_1000_std + value: -1.4906980957334761 + - type: nauc_map_at_100_diff1 + value: 39.856757588861484 + - type: nauc_map_at_100_max + value: 26.17640926201627 + - type: nauc_map_at_100_std + value: -1.52726616900199 + - type: nauc_map_at_10_diff1 + value: 39.928116776753306 + - type: nauc_map_at_10_max + value: 25.849572817015947 + - type: nauc_map_at_10_std + value: -1.8553563555900154 + - type: nauc_map_at_1_diff1 + value: 45.9178313886473 + - type: nauc_map_at_1_max + value: 24.39859079325555 + - type: nauc_map_at_1_std + value: -7.08677568969915 + - type: nauc_map_at_20_diff1 + value: 40.03122889128656 + - type: nauc_map_at_20_max + value: 26.18324831822517 + - type: nauc_map_at_20_std + value: -1.4283210926006862 + - type: nauc_map_at_3_diff1 + value: 41.13705028238216 + - type: nauc_map_at_3_max + value: 25.227792355607466 + - type: nauc_map_at_3_std + value: -4.10076689245123 + - type: nauc_map_at_5_diff1 + value: 40.41343312331809 + - type: nauc_map_at_5_max + value: 26.019632871475828 + - type: nauc_map_at_5_std + value: -1.655322674363369 + - type: nauc_mrr_at_1000_diff1 + value: 38.10335222688471 + - type: nauc_mrr_at_1000_max + value: 27.76819205009592 + - type: nauc_mrr_at_1000_std + value: -0.06837137793398154 + - type: nauc_mrr_at_100_diff1 + value: 38.05479288369399 + - type: nauc_mrr_at_100_max + value: 27.7490714350333 + - type: nauc_mrr_at_100_std + value: -0.08908922458485154 + - type: nauc_mrr_at_10_diff1 + value: 37.96378944519479 + - type: nauc_mrr_at_10_max + value: 27.41804522591167 + - type: nauc_mrr_at_10_std + value: -0.19175217620882173 + - type: nauc_mrr_at_1_diff1 + value: 43.635050719694746 + - type: nauc_mrr_at_1_max + value: 26.637384998955994 + - type: nauc_mrr_at_1_std + value: -4.4281228525493574 + - type: nauc_mrr_at_20_diff1 + value: 38.125362772329744 + - type: nauc_mrr_at_20_max + value: 27.847086458572683 + - type: nauc_mrr_at_20_std + value: 0.04840792038341067 + - type: nauc_mrr_at_3_diff1 + value: 39.284979864146216 + - type: nauc_mrr_at_3_max + value: 27.57894001717352 + - type: nauc_mrr_at_3_std + value: -2.0696803670615265 + - type: nauc_mrr_at_5_diff1 + value: 38.33310714032024 + - type: nauc_mrr_at_5_max + value: 27.73956844204174 + - type: nauc_mrr_at_5_std + value: 0.01395219897785327 + - type: nauc_ndcg_at_1000_diff1 + value: 37.584777068441745 + - type: nauc_ndcg_at_1000_max + value: 27.742338292986034 + - type: nauc_ndcg_at_1000_std + value: 1.5919119060984248 + - type: nauc_ndcg_at_100_diff1 + value: 36.324511379401926 + - type: nauc_ndcg_at_100_max + value: 27.135186980945264 + - type: nauc_ndcg_at_100_std + value: 1.012036671263999 + - type: nauc_ndcg_at_10_diff1 + value: 36.883529139532634 + - type: nauc_ndcg_at_10_max + value: 26.287452858703826 + - type: nauc_ndcg_at_10_std + value: 0.5701574244802798 + - type: nauc_ndcg_at_1_diff1 + value: 43.635050719694746 + - type: nauc_ndcg_at_1_max + value: 26.637384998955994 + - type: nauc_ndcg_at_1_std + value: -4.4281228525493574 + - type: nauc_ndcg_at_20_diff1 + value: 37.26150152174845 + - type: nauc_ndcg_at_20_max + value: 27.59340643785688 + - type: nauc_ndcg_at_20_std + value: 1.812718743312418 + - type: nauc_ndcg_at_3_diff1 + value: 39.291935691856196 + - type: nauc_ndcg_at_3_max + value: 26.155040259440188 + - type: nauc_ndcg_at_3_std + value: -3.1636780333695556 + - type: nauc_ndcg_at_5_diff1 + value: 37.96141298539201 + - type: nauc_ndcg_at_5_max + value: 26.886149321572532 + - type: nauc_ndcg_at_5_std + value: 0.9674089088890447 + - type: nauc_precision_at_1000_diff1 + value: -6.193908740223283 + - type: nauc_precision_at_1000_max + value: 18.933037425592925 + - type: nauc_precision_at_1000_std + value: 13.990532282500373 + - type: nauc_precision_at_100_diff1 + value: 2.0105355129064484 + - type: nauc_precision_at_100_max + value: 24.049503343204073 + - type: nauc_precision_at_100_std + value: 10.79645255758947 + - type: nauc_precision_at_10_diff1 + value: 21.59130230622116 + - type: nauc_precision_at_10_max + value: 28.24307291885018 + - type: nauc_precision_at_10_std + value: 9.505053356771134 + - type: nauc_precision_at_1_diff1 + value: 43.635050719694746 + - type: nauc_precision_at_1_max + value: 26.637384998955994 + - type: nauc_precision_at_1_std + value: -4.4281228525493574 + - type: nauc_precision_at_20_diff1 + value: 19.080704068772512 + - type: nauc_precision_at_20_max + value: 31.52215796190006 + - type: nauc_precision_at_20_std + value: 14.119376174372944 + - type: nauc_precision_at_3_diff1 + value: 31.733564430212695 + - type: nauc_precision_at_3_max + value: 28.63157366942346 + - type: nauc_precision_at_3_std + value: 0.7847711198268656 + - type: nauc_precision_at_5_diff1 + value: 27.2848881133242 + - type: nauc_precision_at_5_max + value: 30.792243775503593 + - type: nauc_precision_at_5_std + value: 10.5253825269578 + - type: nauc_recall_at_1000_diff1 + value: 32.59052385808082 + - type: nauc_recall_at_1000_max + value: 39.16615094126513 + - type: nauc_recall_at_1000_std + value: 32.831614951523285 + - type: nauc_recall_at_100_diff1 + value: 19.499054183956282 + - type: nauc_recall_at_100_max + value: 25.791611422283385 + - type: nauc_recall_at_100_std + value: 8.86161971226946 + - type: nauc_recall_at_10_diff1 + value: 27.966399902667032 + - type: nauc_recall_at_10_max + value: 24.52174702656756 + - type: nauc_recall_at_10_std + value: 5.888597161688889 + - type: nauc_recall_at_1_diff1 + value: 45.9178313886473 + - type: nauc_recall_at_1_max + value: 24.39859079325555 + - type: nauc_recall_at_1_std + value: -7.08677568969915 + - type: nauc_recall_at_20_diff1 + value: 28.92843394705317 + - type: nauc_recall_at_20_max + value: 29.498567057460452 + - type: nauc_recall_at_20_std + value: 10.973275179789377 + - type: nauc_recall_at_3_diff1 + value: 35.866038436338386 + - type: nauc_recall_at_3_max + value: 25.385826645049935 + - type: nauc_recall_at_3_std + value: -1.945839877039293 + - type: nauc_recall_at_5_diff1 + value: 31.781401015413223 + - type: nauc_recall_at_5_max + value: 26.580275134192338 + - type: nauc_recall_at_5_std + value: 6.830480292572181 + - type: ndcg_at_1 + value: 28.022999999999996 + - type: ndcg_at_10 + value: 39.837 + - type: ndcg_at_100 + value: 44.995000000000005 + - type: ndcg_at_1000 + value: 46.928 + - type: ndcg_at_20 + value: 41.729 + - type: ndcg_at_3 + value: 34.781 + - type: ndcg_at_5 + value: 37.136 + - type: precision_at_1 + value: 28.022999999999996 + - type: precision_at_10 + value: 6.101999999999999 + - type: precision_at_100 + value: 0.9159999999999999 + - type: precision_at_1000 + value: 0.11100000000000002 + - type: precision_at_20 + value: 3.508 + - type: precision_at_3 + value: 14.727 + - type: precision_at_5 + value: 10.237 + - type: recall_at_1 + value: 25.764 + - type: recall_at_10 + value: 53.101 + - type: recall_at_100 + value: 76.955 + - type: recall_at_1000 + value: 91.175 + - type: recall_at_20 + value: 60.138999999999996 + - type: recall_at_3 + value: 39.484 + - type: recall_at_5 + value: 45.208999999999996 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackMathematicaRetrieval + revision: 90fceea13679c63fe563ded68f3b6f06e50061de + split: test + type: mteb/cqadupstack-mathematica + metrics: + - type: main_score + value: 29.697000000000003 + - type: map_at_1 + value: 15.958 + - type: map_at_10 + value: 24.324 + - type: map_at_100 + value: 25.64 + - type: map_at_1000 + value: 25.762 + - type: map_at_20 + value: 25.064999999999998 + - type: map_at_3 + value: 21.36 + - type: map_at_5 + value: 23.034 + - type: mrr_at_1 + value: 19.900497512437813 + - type: mrr_at_10 + value: 28.495715865118843 + - type: mrr_at_100 + value: 29.55883123523441 + - type: mrr_at_1000 + value: 29.623700416228015 + - type: mrr_at_20 + value: 29.127165464095363 + - type: mrr_at_3 + value: 25.538971807628535 + - type: mrr_at_5 + value: 27.392205638474294 + - type: nauc_map_at_1000_diff1 + value: 34.32209396022877 + - type: nauc_map_at_1000_max + value: 21.688010888227378 + - type: nauc_map_at_1000_std + value: 0.9387258101009914 + - type: nauc_map_at_100_diff1 + value: 34.2873397075491 + - type: nauc_map_at_100_max + value: 21.69354267495513 + - type: nauc_map_at_100_std + value: 0.9556799710614994 + - type: nauc_map_at_10_diff1 + value: 34.84715034635313 + - type: nauc_map_at_10_max + value: 21.547867843258768 + - type: nauc_map_at_10_std + value: 0.32632738479135376 + - type: nauc_map_at_1_diff1 + value: 40.08120759217827 + - type: nauc_map_at_1_max + value: 22.5404646356335 + - type: nauc_map_at_1_std + value: 0.46810928901963667 + - type: nauc_map_at_20_diff1 + value: 34.34837970041168 + - type: nauc_map_at_20_max + value: 21.60444832596523 + - type: nauc_map_at_20_std + value: 0.6374326062336801 + - type: nauc_map_at_3_diff1 + value: 36.058627394912286 + - type: nauc_map_at_3_max + value: 20.99605214930877 + - type: nauc_map_at_3_std + value: -0.6363564572418088 + - type: nauc_map_at_5_diff1 + value: 35.6851873771882 + - type: nauc_map_at_5_max + value: 21.2333736759757 + - type: nauc_map_at_5_std + value: -0.11978481640718161 + - type: nauc_mrr_at_1000_diff1 + value: 33.76519811972675 + - type: nauc_mrr_at_1000_max + value: 23.08099288276958 + - type: nauc_mrr_at_1000_std + value: 0.21670889636182983 + - type: nauc_mrr_at_100_diff1 + value: 33.75104610406262 + - type: nauc_mrr_at_100_max + value: 23.09993932553629 + - type: nauc_mrr_at_100_std + value: 0.21684296947206153 + - type: nauc_mrr_at_10_diff1 + value: 33.890656486559365 + - type: nauc_mrr_at_10_max + value: 23.004230124232492 + - type: nauc_mrr_at_10_std + value: -0.17125819000842146 + - type: nauc_mrr_at_1_diff1 + value: 38.298935378300584 + - type: nauc_mrr_at_1_max + value: 23.531620590823632 + - type: nauc_mrr_at_1_std + value: -1.786682602890334 + - type: nauc_mrr_at_20_diff1 + value: 33.71928046127925 + - type: nauc_mrr_at_20_max + value: 23.028631286321367 + - type: nauc_mrr_at_20_std + value: 0.13566082451341543 + - type: nauc_mrr_at_3_diff1 + value: 34.839766454140396 + - type: nauc_mrr_at_3_max + value: 23.310126494307852 + - type: nauc_mrr_at_3_std + value: -1.1749456883533465 + - type: nauc_mrr_at_5_diff1 + value: 34.33155350261903 + - type: nauc_mrr_at_5_max + value: 22.722607203637715 + - type: nauc_mrr_at_5_std + value: -0.4150972255202597 + - type: nauc_ndcg_at_1000_diff1 + value: 31.8237241348174 + - type: nauc_ndcg_at_1000_max + value: 23.08752218924686 + - type: nauc_ndcg_at_1000_std + value: 3.2872043822295653 + - type: nauc_ndcg_at_100_diff1 + value: 31.190151490173644 + - type: nauc_ndcg_at_100_max + value: 23.091678618693027 + - type: nauc_ndcg_at_100_std + value: 3.7042696983779337 + - type: nauc_ndcg_at_10_diff1 + value: 32.59190071742098 + - type: nauc_ndcg_at_10_max + value: 21.959767958947943 + - type: nauc_ndcg_at_10_std + value: 0.9617687304860824 + - type: nauc_ndcg_at_1_diff1 + value: 38.298935378300584 + - type: nauc_ndcg_at_1_max + value: 23.531620590823632 + - type: nauc_ndcg_at_1_std + value: -1.786682602890334 + - type: nauc_ndcg_at_20_diff1 + value: 31.269080621067157 + - type: nauc_ndcg_at_20_max + value: 22.118415303912165 + - type: nauc_ndcg_at_20_std + value: 1.9476492490549662 + - type: nauc_ndcg_at_3_diff1 + value: 34.75565433874818 + - type: nauc_ndcg_at_3_max + value: 21.88229440058002 + - type: nauc_ndcg_at_3_std + value: -1.0791926988814993 + - type: nauc_ndcg_at_5_diff1 + value: 34.15052980774227 + - type: nauc_ndcg_at_5_max + value: 21.506747457350297 + - type: nauc_ndcg_at_5_std + value: 0.16877419719517406 + - type: nauc_precision_at_1000_diff1 + value: -0.6858280622680386 + - type: nauc_precision_at_1000_max + value: 3.070259742786672 + - type: nauc_precision_at_1000_std + value: -0.44683494016016695 + - type: nauc_precision_at_100_diff1 + value: 5.672431851239273 + - type: nauc_precision_at_100_max + value: 14.924851867989434 + - type: nauc_precision_at_100_std + value: 7.98283699598593 + - type: nauc_precision_at_10_diff1 + value: 17.78733855494319 + - type: nauc_precision_at_10_max + value: 19.852122273791004 + - type: nauc_precision_at_10_std + value: 2.212115382310132 + - type: nauc_precision_at_1_diff1 + value: 38.298935378300584 + - type: nauc_precision_at_1_max + value: 23.531620590823632 + - type: nauc_precision_at_1_std + value: -1.786682602890334 + - type: nauc_precision_at_20_diff1 + value: 12.276912444781432 + - type: nauc_precision_at_20_max + value: 18.32715011630569 + - type: nauc_precision_at_20_std + value: 3.9553516927972416 + - type: nauc_precision_at_3_diff1 + value: 28.513019625970028 + - type: nauc_precision_at_3_max + value: 20.940923230836134 + - type: nauc_precision_at_3_std + value: -3.190188512438985 + - type: nauc_precision_at_5_diff1 + value: 25.002912184047123 + - type: nauc_precision_at_5_max + value: 20.035111366285058 + - type: nauc_precision_at_5_std + value: -0.22506988079631962 + - type: nauc_recall_at_1000_diff1 + value: 13.166705703269512 + - type: nauc_recall_at_1000_max + value: 35.93453865798138 + - type: nauc_recall_at_1000_std + value: 26.682834428503483 + - type: nauc_recall_at_100_diff1 + value: 17.596890918894267 + - type: nauc_recall_at_100_max + value: 26.383345729307116 + - type: nauc_recall_at_100_std + value: 16.648053006302884 + - type: nauc_recall_at_10_diff1 + value: 26.082210572506177 + - type: nauc_recall_at_10_max + value: 20.491825780854438 + - type: nauc_recall_at_10_std + value: 2.6495276614534395 + - type: nauc_recall_at_1_diff1 + value: 40.08120759217827 + - type: nauc_recall_at_1_max + value: 22.5404646356335 + - type: nauc_recall_at_1_std + value: 0.46810928901963667 + - type: nauc_recall_at_20_diff1 + value: 20.786345575454142 + - type: nauc_recall_at_20_max + value: 20.548104255810635 + - type: nauc_recall_at_20_std + value: 6.003243727958278 + - type: nauc_recall_at_3_diff1 + value: 31.619407559646717 + - type: nauc_recall_at_3_max + value: 19.77889595916651 + - type: nauc_recall_at_3_std + value: -1.7285416920920254 + - type: nauc_recall_at_5_diff1 + value: 29.985583590834363 + - type: nauc_recall_at_5_max + value: 19.291384926277257 + - type: nauc_recall_at_5_std + value: 0.4751386159254339 + - type: ndcg_at_1 + value: 19.900000000000002 + - type: ndcg_at_10 + value: 29.697000000000003 + - type: ndcg_at_100 + value: 35.873 + - type: ndcg_at_1000 + value: 38.586 + - type: ndcg_at_20 + value: 32.167 + - type: ndcg_at_3 + value: 24.171 + - type: ndcg_at_5 + value: 26.936 + - type: precision_at_1 + value: 19.900000000000002 + - type: precision_at_10 + value: 5.609 + - type: precision_at_100 + value: 1.004 + - type: precision_at_1000 + value: 0.13799999999999998 + - type: precision_at_20 + value: 3.5069999999999997 + - type: precision_at_3 + value: 11.526 + - type: precision_at_5 + value: 8.856 + - type: recall_at_1 + value: 15.958 + - type: recall_at_10 + value: 42.181999999999995 + - type: recall_at_100 + value: 68.926 + - type: recall_at_1000 + value: 87.93299999999999 + - type: recall_at_20 + value: 51.007999999999996 + - type: recall_at_3 + value: 27.339999999999996 + - type: recall_at_5 + value: 34.105000000000004 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackPhysicsRetrieval + revision: 79531abbd1fb92d06c6d6315a0cbbbf5bb247ea4 + split: test + type: mteb/cqadupstack-physics + metrics: + - type: main_score + value: 46.007 + - type: map_at_1 + value: 29.349999999999998 + - type: map_at_10 + value: 39.775 + - type: map_at_100 + value: 41.104 + - type: map_at_1000 + value: 41.217999999999996 + - type: map_at_20 + value: 40.561 + - type: map_at_3 + value: 36.348 + - type: map_at_5 + value: 38.194 + - type: mrr_at_1 + value: 35.514918190567855 + - type: mrr_at_10 + value: 45.159150281864406 + - type: mrr_at_100 + value: 45.96990278179657 + - type: mrr_at_1000 + value: 46.01710125216311 + - type: mrr_at_20 + value: 45.64400545753921 + - type: mrr_at_3 + value: 42.28424767404553 + - type: mrr_at_5 + value: 43.87712544112925 + - type: nauc_map_at_1000_diff1 + value: 48.11895764039965 + - type: nauc_map_at_1000_max + value: 29.925477824810347 + - type: nauc_map_at_1000_std + value: 0.11504814310932239 + - type: nauc_map_at_100_diff1 + value: 48.12098289801071 + - type: nauc_map_at_100_max + value: 29.86278330417955 + - type: nauc_map_at_100_std + value: 0.060436831220252285 + - type: nauc_map_at_10_diff1 + value: 48.39540484393976 + - type: nauc_map_at_10_max + value: 29.351410400563317 + - type: nauc_map_at_10_std + value: -0.4621980042134368 + - type: nauc_map_at_1_diff1 + value: 54.47121878121458 + - type: nauc_map_at_1_max + value: 28.612001727844355 + - type: nauc_map_at_1_std + value: -4.093971357183509 + - type: nauc_map_at_20_diff1 + value: 48.143368911870425 + - type: nauc_map_at_20_max + value: 29.68026068686464 + - type: nauc_map_at_20_std + value: -0.163617345028557 + - type: nauc_map_at_3_diff1 + value: 48.97810056717568 + - type: nauc_map_at_3_max + value: 29.127235123692852 + - type: nauc_map_at_3_std + value: -1.567085586310881 + - type: nauc_map_at_5_diff1 + value: 48.43112906695029 + - type: nauc_map_at_5_max + value: 29.4251432266822 + - type: nauc_map_at_5_std + value: -0.8442546630575809 + - type: nauc_mrr_at_1000_diff1 + value: 47.33498761500292 + - type: nauc_mrr_at_1000_max + value: 31.61027031045797 + - type: nauc_mrr_at_1000_std + value: 1.6928194091772466 + - type: nauc_mrr_at_100_diff1 + value: 47.329981330106705 + - type: nauc_mrr_at_100_max + value: 31.600851021560985 + - type: nauc_mrr_at_100_std + value: 1.6838117616288253 + - type: nauc_mrr_at_10_diff1 + value: 47.30242253222377 + - type: nauc_mrr_at_10_max + value: 31.541347591823182 + - type: nauc_mrr_at_10_std + value: 1.6027415899976174 + - type: nauc_mrr_at_1_diff1 + value: 53.06561519484963 + - type: nauc_mrr_at_1_max + value: 31.987901603200687 + - type: nauc_mrr_at_1_std + value: 0.3577806477314526 + - type: nauc_mrr_at_20_diff1 + value: 47.330716020995595 + - type: nauc_mrr_at_20_max + value: 31.55066307584678 + - type: nauc_mrr_at_20_std + value: 1.6320128807973546 + - type: nauc_mrr_at_3_diff1 + value: 47.20092644242079 + - type: nauc_mrr_at_3_max + value: 31.297218751277995 + - type: nauc_mrr_at_3_std + value: 0.888116504301737 + - type: nauc_mrr_at_5_diff1 + value: 47.13557519365777 + - type: nauc_mrr_at_5_max + value: 31.612986186592124 + - type: nauc_mrr_at_5_std + value: 1.453385158208238 + - type: nauc_ndcg_at_1000_diff1 + value: 46.069421394896096 + - type: nauc_ndcg_at_1000_max + value: 31.63575075486594 + - type: nauc_ndcg_at_1000_std + value: 2.980065497846789 + - type: nauc_ndcg_at_100_diff1 + value: 46.112281840719724 + - type: nauc_ndcg_at_100_max + value: 30.902568316145064 + - type: nauc_ndcg_at_100_std + value: 2.4907275571593934 + - type: nauc_ndcg_at_10_diff1 + value: 46.41376893687042 + - type: nauc_ndcg_at_10_max + value: 29.23719673315816 + - type: nauc_ndcg_at_10_std + value: 0.7970981707285222 + - type: nauc_ndcg_at_1_diff1 + value: 53.06561519484963 + - type: nauc_ndcg_at_1_max + value: 31.987901603200687 + - type: nauc_ndcg_at_1_std + value: 0.3577806477314526 + - type: nauc_ndcg_at_20_diff1 + value: 45.874746478664484 + - type: nauc_ndcg_at_20_max + value: 29.845280657306272 + - type: nauc_ndcg_at_20_std + value: 1.3727615463766498 + - type: nauc_ndcg_at_3_diff1 + value: 46.62956408759762 + - type: nauc_ndcg_at_3_max + value: 29.76974459191416 + - type: nauc_ndcg_at_3_std + value: -0.4825766869465331 + - type: nauc_ndcg_at_5_diff1 + value: 46.379271929390356 + - type: nauc_ndcg_at_5_max + value: 29.880549956424098 + - type: nauc_ndcg_at_5_std + value: 0.4800086104773111 + - type: nauc_precision_at_1000_diff1 + value: -13.96376128722718 + - type: nauc_precision_at_1000_max + value: 13.458410308076655 + - type: nauc_precision_at_1000_std + value: 15.301880813229934 + - type: nauc_precision_at_100_diff1 + value: -4.637278193987726 + - type: nauc_precision_at_100_max + value: 18.25462789192962 + - type: nauc_precision_at_100_std + value: 16.309429890343687 + - type: nauc_precision_at_10_diff1 + value: 15.248083771766987 + - type: nauc_precision_at_10_max + value: 24.87434955413506 + - type: nauc_precision_at_10_std + value: 12.167229419982565 + - type: nauc_precision_at_1_diff1 + value: 53.06561519484963 + - type: nauc_precision_at_1_max + value: 31.987901603200687 + - type: nauc_precision_at_1_std + value: 0.3577806477314526 + - type: nauc_precision_at_20_diff1 + value: 5.696350416504954 + - type: nauc_precision_at_20_max + value: 22.86200256696701 + - type: nauc_precision_at_20_std + value: 14.443461022731741 + - type: nauc_precision_at_3_diff1 + value: 30.97090928535156 + - type: nauc_precision_at_3_max + value: 29.366143782023656 + - type: nauc_precision_at_3_std + value: 5.099366151068488 + - type: nauc_precision_at_5_diff1 + value: 23.276107412097236 + - type: nauc_precision_at_5_max + value: 28.60311491474139 + - type: nauc_precision_at_5_std + value: 9.666927035379812 + - type: nauc_recall_at_1000_diff1 + value: 27.414721915157287 + - type: nauc_recall_at_1000_max + value: 48.1562394286556 + - type: nauc_recall_at_1000_std + value: 40.587486120168585 + - type: nauc_recall_at_100_diff1 + value: 36.31335222272159 + - type: nauc_recall_at_100_max + value: 29.66932371155468 + - type: nauc_recall_at_100_std + value: 12.629945573071282 + - type: nauc_recall_at_10_diff1 + value: 39.37105859307282 + - type: nauc_recall_at_10_max + value: 23.604377853362408 + - type: nauc_recall_at_10_std + value: 1.898285967993768 + - type: nauc_recall_at_1_diff1 + value: 54.47121878121458 + - type: nauc_recall_at_1_max + value: 28.612001727844355 + - type: nauc_recall_at_1_std + value: -4.093971357183509 + - type: nauc_recall_at_20_diff1 + value: 36.648839370171544 + - type: nauc_recall_at_20_max + value: 24.592852957100266 + - type: nauc_recall_at_20_std + value: 3.609510577266925 + - type: nauc_recall_at_3_diff1 + value: 42.52133471177241 + - type: nauc_recall_at_3_max + value: 26.25576104463365 + - type: nauc_recall_at_3_std + value: -1.5419672040106491 + - type: nauc_recall_at_5_diff1 + value: 40.32588576278877 + - type: nauc_recall_at_5_max + value: 26.295242572486284 + - type: nauc_recall_at_5_std + value: 0.42561932055282814 + - type: ndcg_at_1 + value: 35.515 + - type: ndcg_at_10 + value: 46.007 + - type: ndcg_at_100 + value: 51.342 + - type: ndcg_at_1000 + value: 53.359 + - type: ndcg_at_20 + value: 48.231 + - type: ndcg_at_3 + value: 40.272999999999996 + - type: ndcg_at_5 + value: 42.875 + - type: precision_at_1 + value: 35.515 + - type: precision_at_10 + value: 8.364 + - type: precision_at_100 + value: 1.288 + - type: precision_at_1000 + value: 0.164 + - type: precision_at_20 + value: 4.962 + - type: precision_at_3 + value: 18.961 + - type: precision_at_5 + value: 13.513 + - type: recall_at_1 + value: 29.349999999999998 + - type: recall_at_10 + value: 59.267 + - type: recall_at_100 + value: 81.243 + - type: recall_at_1000 + value: 94.161 + - type: recall_at_20 + value: 66.77199999999999 + - type: recall_at_3 + value: 43.257 + - type: recall_at_5 + value: 49.933 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackProgrammersRetrieval + revision: 6184bc1440d2dbc7612be22b50686b8826d22b32 + split: test + type: mteb/cqadupstack-programmers + metrics: + - type: main_score + value: 43.104 + - type: map_at_1 + value: 26.265 + - type: map_at_10 + value: 37.098 + - type: map_at_100 + value: 38.582 + - type: map_at_1000 + value: 38.68 + - type: map_at_20 + value: 37.911 + - type: map_at_3 + value: 33.803 + - type: map_at_5 + value: 35.629 + - type: mrr_at_1 + value: 32.64840182648402 + - type: mrr_at_10 + value: 42.31621910560265 + - type: mrr_at_100 + value: 43.314920371374065 + - type: mrr_at_1000 + value: 43.35964252174629 + - type: mrr_at_20 + value: 42.90480897932176 + - type: mrr_at_3 + value: 39.764079147640786 + - type: mrr_at_5 + value: 41.25380517503803 + - type: nauc_map_at_1000_diff1 + value: 43.45029323897613 + - type: nauc_map_at_1000_max + value: 32.72487667511562 + - type: nauc_map_at_1000_std + value: 4.635498217182894 + - type: nauc_map_at_100_diff1 + value: 43.416036838727564 + - type: nauc_map_at_100_max + value: 32.7136330112318 + - type: nauc_map_at_100_std + value: 4.691526480787624 + - type: nauc_map_at_10_diff1 + value: 43.64606477170319 + - type: nauc_map_at_10_max + value: 32.29286435030195 + - type: nauc_map_at_10_std + value: 3.8611425120790854 + - type: nauc_map_at_1_diff1 + value: 48.47333496205567 + - type: nauc_map_at_1_max + value: 27.743297364140062 + - type: nauc_map_at_1_std + value: -2.2399535593996704 + - type: nauc_map_at_20_diff1 + value: 43.32047006337627 + - type: nauc_map_at_20_max + value: 32.39695913886823 + - type: nauc_map_at_20_std + value: 4.145069332261304 + - type: nauc_map_at_3_diff1 + value: 44.17067949379911 + - type: nauc_map_at_3_max + value: 31.529396644447843 + - type: nauc_map_at_3_std + value: 1.0278370965255377 + - type: nauc_map_at_5_diff1 + value: 43.91747436857912 + - type: nauc_map_at_5_max + value: 32.28375375021212 + - type: nauc_map_at_5_std + value: 2.4868152401854586 + - type: nauc_mrr_at_1000_diff1 + value: 43.07485725751252 + - type: nauc_mrr_at_1000_max + value: 33.834125690650744 + - type: nauc_mrr_at_1000_std + value: 6.225141076338039 + - type: nauc_mrr_at_100_diff1 + value: 43.04619032998411 + - type: nauc_mrr_at_100_max + value: 33.839469490267014 + - type: nauc_mrr_at_100_std + value: 6.26262791407454 + - type: nauc_mrr_at_10_diff1 + value: 43.135041859981314 + - type: nauc_mrr_at_10_max + value: 33.66061195696024 + - type: nauc_mrr_at_10_std + value: 6.031199962121296 + - type: nauc_mrr_at_1_diff1 + value: 48.679120171485025 + - type: nauc_mrr_at_1_max + value: 32.630645939261896 + - type: nauc_mrr_at_1_std + value: 2.721955383843323 + - type: nauc_mrr_at_20_diff1 + value: 42.92758496123681 + - type: nauc_mrr_at_20_max + value: 33.758469763960385 + - type: nauc_mrr_at_20_std + value: 6.092108139456077 + - type: nauc_mrr_at_3_diff1 + value: 43.67155702067532 + - type: nauc_mrr_at_3_max + value: 33.90223139830306 + - type: nauc_mrr_at_3_std + value: 4.286070486118104 + - type: nauc_mrr_at_5_diff1 + value: 43.338615104225035 + - type: nauc_mrr_at_5_max + value: 33.614835693617565 + - type: nauc_mrr_at_5_std + value: 5.110470665508919 + - type: nauc_ndcg_at_1000_diff1 + value: 41.49895410329273 + - type: nauc_ndcg_at_1000_max + value: 34.195234462236854 + - type: nauc_ndcg_at_1000_std + value: 8.498163780196581 + - type: nauc_ndcg_at_100_diff1 + value: 40.75234081378347 + - type: nauc_ndcg_at_100_max + value: 34.23564373044607 + - type: nauc_ndcg_at_100_std + value: 10.05540839333941 + - type: nauc_ndcg_at_10_diff1 + value: 41.41477444523898 + - type: nauc_ndcg_at_10_max + value: 32.833800717913114 + - type: nauc_ndcg_at_10_std + value: 6.858969811913209 + - type: nauc_ndcg_at_1_diff1 + value: 48.679120171485025 + - type: nauc_ndcg_at_1_max + value: 32.630645939261896 + - type: nauc_ndcg_at_1_std + value: 2.721955383843323 + - type: nauc_ndcg_at_20_diff1 + value: 40.28434573239677 + - type: nauc_ndcg_at_20_max + value: 32.87240585064605 + - type: nauc_ndcg_at_20_std + value: 7.458863737330834 + - type: nauc_ndcg_at_3_diff1 + value: 42.381887136641296 + - type: nauc_ndcg_at_3_max + value: 32.99180751163812 + - type: nauc_ndcg_at_3_std + value: 2.7500063712633787 + - type: nauc_ndcg_at_5_diff1 + value: 42.12035473910616 + - type: nauc_ndcg_at_5_max + value: 33.048554190063015 + - type: nauc_ndcg_at_5_std + value: 4.229238611217661 + - type: nauc_precision_at_1000_diff1 + value: -7.055058599902845 + - type: nauc_precision_at_1000_max + value: 5.279974888532298 + - type: nauc_precision_at_1000_std + value: 6.3972410796255375 + - type: nauc_precision_at_100_diff1 + value: 0.296177312991241 + - type: nauc_precision_at_100_max + value: 16.345552573537926 + - type: nauc_precision_at_100_std + value: 21.61996821825965 + - type: nauc_precision_at_10_diff1 + value: 18.3634158746968 + - type: nauc_precision_at_10_max + value: 28.666545042552787 + - type: nauc_precision_at_10_std + value: 20.253307172595655 + - type: nauc_precision_at_1_diff1 + value: 48.679120171485025 + - type: nauc_precision_at_1_max + value: 32.630645939261896 + - type: nauc_precision_at_1_std + value: 2.721955383843323 + - type: nauc_precision_at_20_diff1 + value: 9.050777040054417 + - type: nauc_precision_at_20_max + value: 23.739053156719237 + - type: nauc_precision_at_20_std + value: 21.201347657987768 + - type: nauc_precision_at_3_diff1 + value: 31.288764373507032 + - type: nauc_precision_at_3_max + value: 35.03739616944736 + - type: nauc_precision_at_3_std + value: 10.687519325124368 + - type: nauc_precision_at_5_diff1 + value: 26.094294182356514 + - type: nauc_precision_at_5_max + value: 34.02729032623777 + - type: nauc_precision_at_5_std + value: 15.155944334598829 + - type: nauc_recall_at_1000_diff1 + value: 17.067589622044004 + - type: nauc_recall_at_1000_max + value: 47.0515288862208 + - type: nauc_recall_at_1000_std + value: 57.35027883483996 + - type: nauc_recall_at_100_diff1 + value: 21.968219400981674 + - type: nauc_recall_at_100_max + value: 37.32117389917341 + - type: nauc_recall_at_100_std + value: 41.057773107985334 + - type: nauc_recall_at_10_diff1 + value: 33.406908724986046 + - type: nauc_recall_at_10_max + value: 30.675911712325725 + - type: nauc_recall_at_10_std + value: 13.85129513699196 + - type: nauc_recall_at_1_diff1 + value: 48.47333496205567 + - type: nauc_recall_at_1_max + value: 27.743297364140062 + - type: nauc_recall_at_1_std + value: -2.2399535593996704 + - type: nauc_recall_at_20_diff1 + value: 27.265337034919746 + - type: nauc_recall_at_20_max + value: 29.77179827995252 + - type: nauc_recall_at_20_std + value: 15.781133217528273 + - type: nauc_recall_at_3_diff1 + value: 37.251744239338294 + - type: nauc_recall_at_3_max + value: 30.951729412099954 + - type: nauc_recall_at_3_std + value: 1.5825770156707786 + - type: nauc_recall_at_5_diff1 + value: 35.95939054074726 + - type: nauc_recall_at_5_max + value: 31.08801927030664 + - type: nauc_recall_at_5_std + value: 5.977374956099989 + - type: ndcg_at_1 + value: 32.647999999999996 + - type: ndcg_at_10 + value: 43.104 + - type: ndcg_at_100 + value: 49.297000000000004 + - type: ndcg_at_1000 + value: 51.275000000000006 + - type: ndcg_at_20 + value: 45.587 + - type: ndcg_at_3 + value: 37.903 + - type: ndcg_at_5 + value: 40.253 + - type: precision_at_1 + value: 32.647999999999996 + - type: precision_at_10 + value: 7.991 + - type: precision_at_100 + value: 1.298 + - type: precision_at_1000 + value: 0.163 + - type: precision_at_20 + value: 4.8 + - type: precision_at_3 + value: 18.227 + - type: precision_at_5 + value: 13.014000000000001 + - type: recall_at_1 + value: 26.265 + - type: recall_at_10 + value: 55.605000000000004 + - type: recall_at_100 + value: 81.963 + - type: recall_at_1000 + value: 95.236 + - type: recall_at_20 + value: 64.455 + - type: recall_at_3 + value: 41.105999999999995 + - type: recall_at_5 + value: 47.260000000000005 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackRetrieval + revision: CQADupstackRetrieval_is_a_combined_dataset + split: test + type: CQADupstackRetrieval_is_a_combined_dataset + metrics: + - type: main_score + value: 41.83275000000001 + - type: ndcg_at_10 + value: 41.83275000000001 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackStatsRetrieval + revision: 65ac3a16b8e91f9cee4c9828cc7c335575432a2a + split: test + type: mteb/cqadupstack-stats + metrics: + - type: main_score + value: 35.375 + - type: map_at_1 + value: 23.547 + - type: map_at_10 + value: 31.142999999999997 + - type: map_at_100 + value: 32.186 + - type: map_at_1000 + value: 32.272 + - type: map_at_20 + value: 31.730999999999998 + - type: map_at_3 + value: 28.919 + - type: map_at_5 + value: 30.407 + - type: mrr_at_1 + value: 26.68711656441718 + - type: mrr_at_10 + value: 34.079754601227 + - type: mrr_at_100 + value: 35.01087377397623 + - type: mrr_at_1000 + value: 35.06789006352093 + - type: mrr_at_20 + value: 34.633737519268365 + - type: mrr_at_3 + value: 32.13190184049082 + - type: mrr_at_5 + value: 33.45092024539879 + - type: nauc_map_at_1000_diff1 + value: 51.66312260771515 + - type: nauc_map_at_1000_max + value: 32.57627734882073 + - type: nauc_map_at_1000_std + value: 6.4793973203324375 + - type: nauc_map_at_100_diff1 + value: 51.631054229476106 + - type: nauc_map_at_100_max + value: 32.53946656654599 + - type: nauc_map_at_100_std + value: 6.471252184684487 + - type: nauc_map_at_10_diff1 + value: 51.684081771168714 + - type: nauc_map_at_10_max + value: 32.359366254689355 + - type: nauc_map_at_10_std + value: 5.952809015194049 + - type: nauc_map_at_1_diff1 + value: 59.657552965362214 + - type: nauc_map_at_1_max + value: 30.802326957153465 + - type: nauc_map_at_1_std + value: 3.7728372494699363 + - type: nauc_map_at_20_diff1 + value: 51.70440488136916 + - type: nauc_map_at_20_max + value: 32.49067664088131 + - type: nauc_map_at_20_std + value: 6.193923730413403 + - type: nauc_map_at_3_diff1 + value: 53.064689875994844 + - type: nauc_map_at_3_max + value: 32.02328326484619 + - type: nauc_map_at_3_std + value: 4.517702546132841 + - type: nauc_map_at_5_diff1 + value: 51.688093514126024 + - type: nauc_map_at_5_max + value: 32.140583227534925 + - type: nauc_map_at_5_std + value: 5.08711820771354 + - type: nauc_mrr_at_1000_diff1 + value: 52.79217634259703 + - type: nauc_mrr_at_1000_max + value: 35.61449692331441 + - type: nauc_mrr_at_1000_std + value: 9.71451506342203 + - type: nauc_mrr_at_100_diff1 + value: 52.76716243297642 + - type: nauc_mrr_at_100_max + value: 35.59808789020979 + - type: nauc_mrr_at_100_std + value: 9.704336095281075 + - type: nauc_mrr_at_10_diff1 + value: 52.86337314938788 + - type: nauc_mrr_at_10_max + value: 35.59686259316265 + - type: nauc_mrr_at_10_std + value: 9.424813284685497 + - type: nauc_mrr_at_1_diff1 + value: 60.89531921678486 + - type: nauc_mrr_at_1_max + value: 35.91468314529445 + - type: nauc_mrr_at_1_std + value: 9.040164187156382 + - type: nauc_mrr_at_20_diff1 + value: 52.80752951231851 + - type: nauc_mrr_at_20_max + value: 35.57463035969631 + - type: nauc_mrr_at_20_std + value: 9.541448915542269 + - type: nauc_mrr_at_3_diff1 + value: 54.076978586514755 + - type: nauc_mrr_at_3_max + value: 35.99756880089376 + - type: nauc_mrr_at_3_std + value: 8.876285954388155 + - type: nauc_mrr_at_5_diff1 + value: 52.94654507662424 + - type: nauc_mrr_at_5_max + value: 35.611041695713894 + - type: nauc_mrr_at_5_std + value: 8.94668482142343 + - type: nauc_ndcg_at_1000_diff1 + value: 48.706733998206225 + - type: nauc_ndcg_at_1000_max + value: 34.26000143617516 + - type: nauc_ndcg_at_1000_std + value: 10.458871338039593 + - type: nauc_ndcg_at_100_diff1 + value: 48.03288194102992 + - type: nauc_ndcg_at_100_max + value: 33.617097137272545 + - type: nauc_ndcg_at_100_std + value: 10.014497313506247 + - type: nauc_ndcg_at_10_diff1 + value: 48.769660003285175 + - type: nauc_ndcg_at_10_max + value: 32.851851368841885 + - type: nauc_ndcg_at_10_std + value: 7.460362891262815 + - type: nauc_ndcg_at_1_diff1 + value: 60.89531921678486 + - type: nauc_ndcg_at_1_max + value: 35.91468314529445 + - type: nauc_ndcg_at_1_std + value: 9.040164187156382 + - type: nauc_ndcg_at_20_diff1 + value: 48.68879043144177 + - type: nauc_ndcg_at_20_max + value: 33.06774534919596 + - type: nauc_ndcg_at_20_std + value: 8.13085582651415 + - type: nauc_ndcg_at_3_diff1 + value: 50.98985879559205 + - type: nauc_ndcg_at_3_max + value: 33.44311811948473 + - type: nauc_ndcg_at_3_std + value: 5.951958511478065 + - type: nauc_ndcg_at_5_diff1 + value: 48.817189585064455 + - type: nauc_ndcg_at_5_max + value: 32.77623327236947 + - type: nauc_ndcg_at_5_std + value: 5.940331677528127 + - type: nauc_precision_at_1000_diff1 + value: 5.051795756107663 + - type: nauc_precision_at_1000_max + value: 24.21322563810257 + - type: nauc_precision_at_1000_std + value: 21.87719442950226 + - type: nauc_precision_at_100_diff1 + value: 15.777063208167954 + - type: nauc_precision_at_100_max + value: 30.870104894832583 + - type: nauc_precision_at_100_std + value: 25.118265012436176 + - type: nauc_precision_at_10_diff1 + value: 34.065015434826876 + - type: nauc_precision_at_10_max + value: 36.38759095988058 + - type: nauc_precision_at_10_std + value: 18.48947634869304 + - type: nauc_precision_at_1_diff1 + value: 60.89531921678486 + - type: nauc_precision_at_1_max + value: 35.91468314529445 + - type: nauc_precision_at_1_std + value: 9.040164187156382 + - type: nauc_precision_at_20_diff1 + value: 29.80657169255189 + - type: nauc_precision_at_20_max + value: 36.49470103206119 + - type: nauc_precision_at_20_std + value: 21.072761177788518 + - type: nauc_precision_at_3_diff1 + value: 43.34842688286303 + - type: nauc_precision_at_3_max + value: 38.49299258863947 + - type: nauc_precision_at_3_std + value: 11.966654977397777 + - type: nauc_precision_at_5_diff1 + value: 36.75523769003984 + - type: nauc_precision_at_5_max + value: 37.27006722184756 + - type: nauc_precision_at_5_std + value: 13.627277452373662 + - type: nauc_recall_at_1000_diff1 + value: 22.753849072056813 + - type: nauc_recall_at_1000_max + value: 40.72656308448736 + - type: nauc_recall_at_1000_std + value: 42.666697726671146 + - type: nauc_recall_at_100_diff1 + value: 28.6454368683411 + - type: nauc_recall_at_100_max + value: 32.24000673731748 + - type: nauc_recall_at_100_std + value: 22.626295555432048 + - type: nauc_recall_at_10_diff1 + value: 37.83614105516011 + - type: nauc_recall_at_10_max + value: 29.851664667699644 + - type: nauc_recall_at_10_std + value: 8.188931396646032 + - type: nauc_recall_at_1_diff1 + value: 59.657552965362214 + - type: nauc_recall_at_1_max + value: 30.802326957153465 + - type: nauc_recall_at_1_std + value: 3.7728372494699363 + - type: nauc_recall_at_20_diff1 + value: 36.47396166352664 + - type: nauc_recall_at_20_max + value: 29.641609075618824 + - type: nauc_recall_at_20_std + value: 10.228815551744836 + - type: nauc_recall_at_3_diff1 + value: 43.85557915759296 + - type: nauc_recall_at_3_max + value: 30.512412885830248 + - type: nauc_recall_at_3_std + value: 3.1042189846227095 + - type: nauc_recall_at_5_diff1 + value: 38.466143617226166 + - type: nauc_recall_at_5_max + value: 29.831242696474597 + - type: nauc_recall_at_5_std + value: 4.352960295433731 + - type: ndcg_at_1 + value: 26.687 + - type: ndcg_at_10 + value: 35.375 + - type: ndcg_at_100 + value: 40.537 + - type: ndcg_at_1000 + value: 42.821999999999996 + - type: ndcg_at_20 + value: 37.391000000000005 + - type: ndcg_at_3 + value: 31.548 + - type: ndcg_at_5 + value: 33.827 + - type: precision_at_1 + value: 26.687 + - type: precision_at_10 + value: 5.552 + - type: precision_at_100 + value: 0.8880000000000001 + - type: precision_at_1000 + value: 0.11499999999999999 + - type: precision_at_20 + value: 3.29 + - type: precision_at_3 + value: 13.804 + - type: precision_at_5 + value: 9.724 + - type: recall_at_1 + value: 23.547 + - type: recall_at_10 + value: 45.293 + - type: recall_at_100 + value: 68.75 + - type: recall_at_1000 + value: 85.944 + - type: recall_at_20 + value: 52.724000000000004 + - type: recall_at_3 + value: 35.219 + - type: recall_at_5 + value: 40.83 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackTexRetrieval + revision: 46989137a86843e03a6195de44b09deda022eec7 + split: test + type: mteb/cqadupstack-tex + metrics: + - type: main_score + value: 29.464000000000002 + - type: map_at_1 + value: 17.345 + - type: map_at_10 + value: 24.736 + - type: map_at_100 + value: 25.877 + - type: map_at_1000 + value: 26.012999999999998 + - type: map_at_20 + value: 25.381999999999998 + - type: map_at_3 + value: 22.407 + - type: map_at_5 + value: 23.653 + - type: mrr_at_1 + value: 20.85340674466621 + - type: mrr_at_10 + value: 28.456928194540072 + - type: mrr_at_100 + value: 29.432688505330702 + - type: mrr_at_1000 + value: 29.511944389924988 + - type: mrr_at_20 + value: 29.035345402730954 + - type: mrr_at_3 + value: 26.273227804542376 + - type: mrr_at_5 + value: 27.475911906400626 + - type: nauc_map_at_1000_diff1 + value: 38.41850506645076 + - type: nauc_map_at_1000_max + value: 28.159387527502506 + - type: nauc_map_at_1000_std + value: 0.40464829576261996 + - type: nauc_map_at_100_diff1 + value: 38.39947010516607 + - type: nauc_map_at_100_max + value: 28.119389747412498 + - type: nauc_map_at_100_std + value: 0.3853149037722529 + - type: nauc_map_at_10_diff1 + value: 38.810894211784564 + - type: nauc_map_at_10_max + value: 28.1562967790831 + - type: nauc_map_at_10_std + value: -0.06389798281342944 + - type: nauc_map_at_1_diff1 + value: 46.03077824417572 + - type: nauc_map_at_1_max + value: 27.835838674526652 + - type: nauc_map_at_1_std + value: -2.2312618530096167 + - type: nauc_map_at_20_diff1 + value: 38.444510158089805 + - type: nauc_map_at_20_max + value: 28.017692321962418 + - type: nauc_map_at_20_std + value: 0.142361876797175 + - type: nauc_map_at_3_diff1 + value: 40.21093903011263 + - type: nauc_map_at_3_max + value: 27.89662945345146 + - type: nauc_map_at_3_std + value: -0.8822596536045023 + - type: nauc_map_at_5_diff1 + value: 39.09699738378812 + - type: nauc_map_at_5_max + value: 28.036786133706936 + - type: nauc_map_at_5_std + value: -0.30364471935320875 + - type: nauc_mrr_at_1000_diff1 + value: 37.920903186739096 + - type: nauc_mrr_at_1000_max + value: 30.341964918119942 + - type: nauc_mrr_at_1000_std + value: 0.12133653932025162 + - type: nauc_mrr_at_100_diff1 + value: 37.893000931019884 + - type: nauc_mrr_at_100_max + value: 30.32903692824292 + - type: nauc_mrr_at_100_std + value: 0.1276009986215775 + - type: nauc_mrr_at_10_diff1 + value: 38.06140499938785 + - type: nauc_mrr_at_10_max + value: 30.484179453909732 + - type: nauc_mrr_at_10_std + value: -0.317866444774404 + - type: nauc_mrr_at_1_diff1 + value: 45.348598364164886 + - type: nauc_mrr_at_1_max + value: 30.866871548394347 + - type: nauc_mrr_at_1_std + value: -2.4227529761902358 + - type: nauc_mrr_at_20_diff1 + value: 37.84945425263586 + - type: nauc_mrr_at_20_max + value: 30.308513343655353 + - type: nauc_mrr_at_20_std + value: -0.07137849023495513 + - type: nauc_mrr_at_3_diff1 + value: 39.35878483294889 + - type: nauc_mrr_at_3_max + value: 30.665229724037594 + - type: nauc_mrr_at_3_std + value: -0.9664014843782173 + - type: nauc_mrr_at_5_diff1 + value: 38.37838860109641 + - type: nauc_mrr_at_5_max + value: 30.626565531830796 + - type: nauc_mrr_at_5_std + value: -0.3844103471733726 + - type: nauc_ndcg_at_1000_diff1 + value: 35.16938356960978 + - type: nauc_ndcg_at_1000_max + value: 28.822303299475173 + - type: nauc_ndcg_at_1000_std + value: 3.5818592250918004 + - type: nauc_ndcg_at_100_diff1 + value: 34.47959205576021 + - type: nauc_ndcg_at_100_max + value: 28.33493283343504 + - type: nauc_ndcg_at_100_std + value: 3.5432148259969285 + - type: nauc_ndcg_at_10_diff1 + value: 35.874095372057184 + - type: nauc_ndcg_at_10_max + value: 28.66941141362765 + - type: nauc_ndcg_at_10_std + value: 0.9200279845334768 + - type: nauc_ndcg_at_1_diff1 + value: 45.348598364164886 + - type: nauc_ndcg_at_1_max + value: 30.866871548394347 + - type: nauc_ndcg_at_1_std + value: -2.4227529761902358 + - type: nauc_ndcg_at_20_diff1 + value: 34.86108989826789 + - type: nauc_ndcg_at_20_max + value: 27.97517854587127 + - type: nauc_ndcg_at_20_std + value: 1.7097315945589184 + - type: nauc_ndcg_at_3_diff1 + value: 38.29252149856027 + - type: nauc_ndcg_at_3_max + value: 29.126153934582273 + - type: nauc_ndcg_at_3_std + value: -0.60152975685728 + - type: nauc_ndcg_at_5_diff1 + value: 36.498054433082125 + - type: nauc_ndcg_at_5_max + value: 28.906092518483796 + - type: nauc_ndcg_at_5_std + value: 0.5077251176639481 + - type: nauc_precision_at_1000_diff1 + value: 1.3184531416184 + - type: nauc_precision_at_1000_max + value: 19.684606377304007 + - type: nauc_precision_at_1000_std + value: 5.302901624843209 + - type: nauc_precision_at_100_diff1 + value: 7.4256150119233 + - type: nauc_precision_at_100_max + value: 22.722855082465305 + - type: nauc_precision_at_100_std + value: 10.526852917202822 + - type: nauc_precision_at_10_diff1 + value: 21.837059591380257 + - type: nauc_precision_at_10_max + value: 30.143548889601334 + - type: nauc_precision_at_10_std + value: 3.231246061820218 + - type: nauc_precision_at_1_diff1 + value: 45.348598364164886 + - type: nauc_precision_at_1_max + value: 30.866871548394347 + - type: nauc_precision_at_1_std + value: -2.4227529761902358 + - type: nauc_precision_at_20_diff1 + value: 16.580204750030568 + - type: nauc_precision_at_20_max + value: 26.70707505208188 + - type: nauc_precision_at_20_std + value: 5.506747683457161 + - type: nauc_precision_at_3_diff1 + value: 31.1299205489302 + - type: nauc_precision_at_3_max + value: 31.305848044535338 + - type: nauc_precision_at_3_std + value: 0.034618129555695674 + - type: nauc_precision_at_5_diff1 + value: 25.654845802781324 + - type: nauc_precision_at_5_max + value: 31.195207428853237 + - type: nauc_precision_at_5_std + value: 1.9886594353332991 + - type: nauc_recall_at_1000_diff1 + value: 16.07368333583504 + - type: nauc_recall_at_1000_max + value: 24.655100580014963 + - type: nauc_recall_at_1000_std + value: 29.043087081370988 + - type: nauc_recall_at_100_diff1 + value: 18.623100894205148 + - type: nauc_recall_at_100_max + value: 22.852792969198422 + - type: nauc_recall_at_100_std + value: 16.36400089993136 + - type: nauc_recall_at_10_diff1 + value: 27.298600372751636 + - type: nauc_recall_at_10_max + value: 25.527541634019517 + - type: nauc_recall_at_10_std + value: 3.4331177908132102 + - type: nauc_recall_at_1_diff1 + value: 46.03077824417572 + - type: nauc_recall_at_1_max + value: 27.835838674526652 + - type: nauc_recall_at_1_std + value: -2.2312618530096167 + - type: nauc_recall_at_20_diff1 + value: 23.32252313064544 + - type: nauc_recall_at_20_max + value: 22.641199032492505 + - type: nauc_recall_at_20_std + value: 6.083452565554306 + - type: nauc_recall_at_3_diff1 + value: 33.39056241547002 + - type: nauc_recall_at_3_max + value: 26.627371199082884 + - type: nauc_recall_at_3_std + value: 0.6888774855297141 + - type: nauc_recall_at_5_diff1 + value: 29.24415512748389 + - type: nauc_recall_at_5_max + value: 26.090434026985427 + - type: nauc_recall_at_5_std + value: 2.701355680777494 + - type: ndcg_at_1 + value: 20.852999999999998 + - type: ndcg_at_10 + value: 29.464000000000002 + - type: ndcg_at_100 + value: 34.775 + - type: ndcg_at_1000 + value: 37.819 + - type: ndcg_at_20 + value: 31.55 + - type: ndcg_at_3 + value: 25.289 + - type: ndcg_at_5 + value: 27.134999999999998 + - type: precision_at_1 + value: 20.852999999999998 + - type: precision_at_10 + value: 5.389 + - type: precision_at_100 + value: 0.936 + - type: precision_at_1000 + value: 0.13899999999999998 + - type: precision_at_20 + value: 3.2809999999999997 + - type: precision_at_3 + value: 12.044 + - type: precision_at_5 + value: 8.685 + - type: recall_at_1 + value: 17.345 + - type: recall_at_10 + value: 39.932 + - type: recall_at_100 + value: 63.61599999999999 + - type: recall_at_1000 + value: 85.14 + - type: recall_at_20 + value: 47.644999999999996 + - type: recall_at_3 + value: 28.143 + - type: recall_at_5 + value: 32.958999999999996 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackUnixRetrieval + revision: 6c6430d3a6d36f8d2a829195bc5dc94d7e063e53 + split: test + type: mteb/cqadupstack-unix + metrics: + - type: main_score + value: 42.229 + - type: map_at_1 + value: 27.296 + - type: map_at_10 + value: 36.846000000000004 + - type: map_at_100 + value: 37.875 + - type: map_at_1000 + value: 37.988 + - type: map_at_20 + value: 37.38 + - type: map_at_3 + value: 33.875 + - type: map_at_5 + value: 35.737 + - type: mrr_at_1 + value: 31.80970149253731 + - type: mrr_at_10 + value: 40.725242833451745 + - type: mrr_at_100 + value: 41.49709600506051 + - type: mrr_at_1000 + value: 41.56245177490777 + - type: mrr_at_20 + value: 41.14234345775092 + - type: mrr_at_3 + value: 38.15298507462684 + - type: mrr_at_5 + value: 39.785447761193964 + - type: nauc_map_at_1000_diff1 + value: 43.75538367380107 + - type: nauc_map_at_1000_max + value: 38.05056227333834 + - type: nauc_map_at_1000_std + value: 0.18921144568919948 + - type: nauc_map_at_100_diff1 + value: 43.77667543106843 + - type: nauc_map_at_100_max + value: 38.050013068730074 + - type: nauc_map_at_100_std + value: 0.15537099877922939 + - type: nauc_map_at_10_diff1 + value: 43.659376456555115 + - type: nauc_map_at_10_max + value: 37.76818280719845 + - type: nauc_map_at_10_std + value: -0.05751365580793609 + - type: nauc_map_at_1_diff1 + value: 50.52298458785999 + - type: nauc_map_at_1_max + value: 36.66943692165301 + - type: nauc_map_at_1_std + value: -1.0116916260070048 + - type: nauc_map_at_20_diff1 + value: 43.60251577616886 + - type: nauc_map_at_20_max + value: 37.85498492792909 + - type: nauc_map_at_20_std + value: 0.02824789930513129 + - type: nauc_map_at_3_diff1 + value: 45.05492770932526 + - type: nauc_map_at_3_max + value: 37.88442451804782 + - type: nauc_map_at_3_std + value: -1.9385674410094953 + - type: nauc_map_at_5_diff1 + value: 43.80463652500317 + - type: nauc_map_at_5_max + value: 37.84812994589745 + - type: nauc_map_at_5_std + value: -0.4965865567169233 + - type: nauc_mrr_at_1000_diff1 + value: 43.30154035744434 + - type: nauc_mrr_at_1000_max + value: 38.630130905058394 + - type: nauc_mrr_at_1000_std + value: -0.09862664950221302 + - type: nauc_mrr_at_100_diff1 + value: 43.295124512783886 + - type: nauc_mrr_at_100_max + value: 38.631685521949564 + - type: nauc_mrr_at_100_std + value: -0.10930551847396679 + - type: nauc_mrr_at_10_diff1 + value: 43.21584928155568 + - type: nauc_mrr_at_10_max + value: 38.47495723593308 + - type: nauc_mrr_at_10_std + value: -0.19491385817913934 + - type: nauc_mrr_at_1_diff1 + value: 49.77296386977506 + - type: nauc_mrr_at_1_max + value: 38.247307218133955 + - type: nauc_mrr_at_1_std + value: -0.7255129557116325 + - type: nauc_mrr_at_20_diff1 + value: 43.1173795956077 + - type: nauc_mrr_at_20_max + value: 38.530148753210206 + - type: nauc_mrr_at_20_std + value: -0.08733026838418803 + - type: nauc_mrr_at_3_diff1 + value: 44.3902368759337 + - type: nauc_mrr_at_3_max + value: 39.14502536820926 + - type: nauc_mrr_at_3_std + value: -1.9205759363830195 + - type: nauc_mrr_at_5_diff1 + value: 43.39334344411832 + - type: nauc_mrr_at_5_max + value: 38.8749655158438 + - type: nauc_mrr_at_5_std + value: -0.43021416093426457 + - type: nauc_ndcg_at_1000_diff1 + value: 41.629511704681384 + - type: nauc_ndcg_at_1000_max + value: 38.98455196695487 + - type: nauc_ndcg_at_1000_std + value: 2.5283322394192447 + - type: nauc_ndcg_at_100_diff1 + value: 41.91727965546967 + - type: nauc_ndcg_at_100_max + value: 39.06244395765021 + - type: nauc_ndcg_at_100_std + value: 2.2945063835366737 + - type: nauc_ndcg_at_10_diff1 + value: 40.97178908043215 + - type: nauc_ndcg_at_10_max + value: 37.79433728861138 + - type: nauc_ndcg_at_10_std + value: 1.2281320664622282 + - type: nauc_ndcg_at_1_diff1 + value: 49.77296386977506 + - type: nauc_ndcg_at_1_max + value: 38.247307218133955 + - type: nauc_ndcg_at_1_std + value: -0.7255129557116325 + - type: nauc_ndcg_at_20_diff1 + value: 40.68478954959323 + - type: nauc_ndcg_at_20_max + value: 37.99002839374275 + - type: nauc_ndcg_at_20_std + value: 1.6878408766410797 + - type: nauc_ndcg_at_3_diff1 + value: 43.17600013118951 + - type: nauc_ndcg_at_3_max + value: 38.36046345841751 + - type: nauc_ndcg_at_3_std + value: -2.1648956819129026 + - type: nauc_ndcg_at_5_diff1 + value: 41.300977363206734 + - type: nauc_ndcg_at_5_max + value: 38.140880536833365 + - type: nauc_ndcg_at_5_std + value: 0.2184672115286344 + - type: nauc_precision_at_1000_diff1 + value: -13.09643401766385 + - type: nauc_precision_at_1000_max + value: 0.9450851398613254 + - type: nauc_precision_at_1000_std + value: 3.854576973853421 + - type: nauc_precision_at_100_diff1 + value: 3.4722518789419494 + - type: nauc_precision_at_100_max + value: 19.486052927401857 + - type: nauc_precision_at_100_std + value: 7.388218124534871 + - type: nauc_precision_at_10_diff1 + value: 18.477374805886022 + - type: nauc_precision_at_10_max + value: 28.3998207738487 + - type: nauc_precision_at_10_std + value: 2.7979953301766964 + - type: nauc_precision_at_1_diff1 + value: 49.77296386977506 + - type: nauc_precision_at_1_max + value: 38.247307218133955 + - type: nauc_precision_at_1_std + value: -0.7255129557116325 + - type: nauc_precision_at_20_diff1 + value: 12.253841360319052 + - type: nauc_precision_at_20_max + value: 24.85110392443306 + - type: nauc_precision_at_20_std + value: 3.7311212948659613 + - type: nauc_precision_at_3_diff1 + value: 32.562607666264014 + - type: nauc_precision_at_3_max + value: 36.40674316469 + - type: nauc_precision_at_3_std + value: -3.701236546292251 + - type: nauc_precision_at_5_diff1 + value: 24.80104527226832 + - type: nauc_precision_at_5_max + value: 33.69917080736034 + - type: nauc_precision_at_5_std + value: 1.3236747342761015 + - type: nauc_recall_at_1000_diff1 + value: 24.13767701978169 + - type: nauc_recall_at_1000_max + value: 50.49269211134329 + - type: nauc_recall_at_1000_std + value: 47.08476397391495 + - type: nauc_recall_at_100_diff1 + value: 36.11087195903443 + - type: nauc_recall_at_100_max + value: 40.79223196983678 + - type: nauc_recall_at_100_std + value: 14.390360400534908 + - type: nauc_recall_at_10_diff1 + value: 31.317148127487908 + - type: nauc_recall_at_10_max + value: 34.436206830017284 + - type: nauc_recall_at_10_std + value: 6.245127051630293 + - type: nauc_recall_at_1_diff1 + value: 50.52298458785999 + - type: nauc_recall_at_1_max + value: 36.66943692165301 + - type: nauc_recall_at_1_std + value: -1.0116916260070048 + - type: nauc_recall_at_20_diff1 + value: 30.00266971037392 + - type: nauc_recall_at_20_max + value: 34.963620751107484 + - type: nauc_recall_at_20_std + value: 8.42010232962911 + - type: nauc_recall_at_3_diff1 + value: 38.402162193668296 + - type: nauc_recall_at_3_max + value: 37.495379694986894 + - type: nauc_recall_at_3_std + value: -2.7440958505645643 + - type: nauc_recall_at_5_diff1 + value: 33.08317576475561 + - type: nauc_recall_at_5_max + value: 36.08067909558661 + - type: nauc_recall_at_5_std + value: 2.9313830821919717 + - type: ndcg_at_1 + value: 31.81 + - type: ndcg_at_10 + value: 42.229 + - type: ndcg_at_100 + value: 47.016000000000005 + - type: ndcg_at_1000 + value: 49.544 + - type: ndcg_at_20 + value: 43.877 + - type: ndcg_at_3 + value: 36.985 + - type: ndcg_at_5 + value: 39.794000000000004 + - type: precision_at_1 + value: 31.81 + - type: precision_at_10 + value: 7.136000000000001 + - type: precision_at_100 + value: 1.0710000000000002 + - type: precision_at_1000 + value: 0.13999999999999999 + - type: precision_at_20 + value: 4.067 + - type: precision_at_3 + value: 16.698 + - type: precision_at_5 + value: 12.034 + - type: recall_at_1 + value: 27.296 + - type: recall_at_10 + value: 54.607000000000006 + - type: recall_at_100 + value: 75.65899999999999 + - type: recall_at_1000 + value: 93.366 + - type: recall_at_20 + value: 60.373 + - type: recall_at_3 + value: 40.394000000000005 + - type: recall_at_5 + value: 47.448 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackWebmastersRetrieval + revision: 160c094312a0e1facb97e55eeddb698c0abe3571 + split: test + type: mteb/cqadupstack-webmasters + metrics: + - type: main_score + value: 40.617 + - type: map_at_1 + value: 26.124000000000002 + - type: map_at_10 + value: 34.857 + - type: map_at_100 + value: 36.589 + - type: map_at_1000 + value: 36.832 + - type: map_at_20 + value: 35.808 + - type: map_at_3 + value: 31.635 + - type: map_at_5 + value: 33.225 + - type: mrr_at_1 + value: 30.8300395256917 + - type: mrr_at_10 + value: 39.024876090093485 + - type: mrr_at_100 + value: 40.112831574778625 + - type: mrr_at_1000 + value: 40.172542355918054 + - type: mrr_at_20 + value: 39.70461282542164 + - type: mrr_at_3 + value: 36.429512516469046 + - type: mrr_at_5 + value: 37.86231884057971 + - type: nauc_map_at_1000_diff1 + value: 42.17385761787554 + - type: nauc_map_at_1000_max + value: 29.023497819147003 + - type: nauc_map_at_1000_std + value: 7.135398677866782 + - type: nauc_map_at_100_diff1 + value: 42.26672776757606 + - type: nauc_map_at_100_max + value: 29.232142243363228 + - type: nauc_map_at_100_std + value: 6.877840055692755 + - type: nauc_map_at_10_diff1 + value: 42.877953607649324 + - type: nauc_map_at_10_max + value: 29.46874782942982 + - type: nauc_map_at_10_std + value: 5.031611824650058 + - type: nauc_map_at_1_diff1 + value: 50.063501481714425 + - type: nauc_map_at_1_max + value: 26.941170343192383 + - type: nauc_map_at_1_std + value: 1.2783766588617693 + - type: nauc_map_at_20_diff1 + value: 42.44190895193996 + - type: nauc_map_at_20_max + value: 29.46253769495084 + - type: nauc_map_at_20_std + value: 6.146541588421648 + - type: nauc_map_at_3_diff1 + value: 44.21913152729423 + - type: nauc_map_at_3_max + value: 28.744071788355395 + - type: nauc_map_at_3_std + value: 3.616132053693665 + - type: nauc_map_at_5_diff1 + value: 43.187186127431275 + - type: nauc_map_at_5_max + value: 29.016108070111184 + - type: nauc_map_at_5_std + value: 4.684957715416093 + - type: nauc_mrr_at_1000_diff1 + value: 39.22437734736831 + - type: nauc_mrr_at_1000_max + value: 30.643434055644743 + - type: nauc_mrr_at_1000_std + value: 8.491481853057229 + - type: nauc_mrr_at_100_diff1 + value: 39.203338294322144 + - type: nauc_mrr_at_100_max + value: 30.64454944640062 + - type: nauc_mrr_at_100_std + value: 8.541568521609001 + - type: nauc_mrr_at_10_diff1 + value: 39.327335076290325 + - type: nauc_mrr_at_10_max + value: 30.870046331420216 + - type: nauc_mrr_at_10_std + value: 8.192608946125315 + - type: nauc_mrr_at_1_diff1 + value: 43.61118724145502 + - type: nauc_mrr_at_1_max + value: 28.90069833045206 + - type: nauc_mrr_at_1_std + value: 4.479542769926193 + - type: nauc_mrr_at_20_diff1 + value: 39.13464939320264 + - type: nauc_mrr_at_20_max + value: 30.733559708742213 + - type: nauc_mrr_at_20_std + value: 8.600789691544723 + - type: nauc_mrr_at_3_diff1 + value: 39.44504307119285 + - type: nauc_mrr_at_3_max + value: 30.44308080500852 + - type: nauc_mrr_at_3_std + value: 6.875960856302759 + - type: nauc_mrr_at_5_diff1 + value: 39.024350509778614 + - type: nauc_mrr_at_5_max + value: 30.564593983912243 + - type: nauc_mrr_at_5_std + value: 8.114542797004818 + - type: nauc_ndcg_at_1000_diff1 + value: 40.17841832753416 + - type: nauc_ndcg_at_1000_max + value: 30.41408114991541 + - type: nauc_ndcg_at_1000_std + value: 10.460324209610734 + - type: nauc_ndcg_at_100_diff1 + value: 39.290496874327985 + - type: nauc_ndcg_at_100_max + value: 30.13490396441187 + - type: nauc_ndcg_at_100_std + value: 11.061850479505644 + - type: nauc_ndcg_at_10_diff1 + value: 39.92759489069634 + - type: nauc_ndcg_at_10_max + value: 30.2024288985298 + - type: nauc_ndcg_at_10_std + value: 8.24558057585065 + - type: nauc_ndcg_at_1_diff1 + value: 43.61118724145502 + - type: nauc_ndcg_at_1_max + value: 28.90069833045206 + - type: nauc_ndcg_at_1_std + value: 4.479542769926193 + - type: nauc_ndcg_at_20_diff1 + value: 39.29435506522245 + - type: nauc_ndcg_at_20_max + value: 30.33893728800384 + - type: nauc_ndcg_at_20_std + value: 10.545750826464893 + - type: nauc_ndcg_at_3_diff1 + value: 40.35926833516799 + - type: nauc_ndcg_at_3_max + value: 28.865626442014204 + - type: nauc_ndcg_at_3_std + value: 6.095820315868252 + - type: nauc_ndcg_at_5_diff1 + value: 39.624068053041924 + - type: nauc_ndcg_at_5_max + value: 29.298680802211013 + - type: nauc_ndcg_at_5_std + value: 7.981261205063178 + - type: nauc_precision_at_1000_diff1 + value: -15.437017626426188 + - type: nauc_precision_at_1000_max + value: -11.466653763751436 + - type: nauc_precision_at_1000_std + value: 27.203862461973344 + - type: nauc_precision_at_100_diff1 + value: -10.720493705726689 + - type: nauc_precision_at_100_max + value: -2.2729520096186393 + - type: nauc_precision_at_100_std + value: 33.378688533496415 + - type: nauc_precision_at_10_diff1 + value: 10.517312952676674 + - type: nauc_precision_at_10_max + value: 21.759943967624206 + - type: nauc_precision_at_10_std + value: 18.479226770461366 + - type: nauc_precision_at_1_diff1 + value: 43.61118724145502 + - type: nauc_precision_at_1_max + value: 28.90069833045206 + - type: nauc_precision_at_1_std + value: 4.479542769926193 + - type: nauc_precision_at_20_diff1 + value: 0.22630058814929357 + - type: nauc_precision_at_20_max + value: 14.340458300248256 + - type: nauc_precision_at_20_std + value: 29.197777051584495 + - type: nauc_precision_at_3_diff1 + value: 24.684569742393748 + - type: nauc_precision_at_3_max + value: 26.847701196729407 + - type: nauc_precision_at_3_std + value: 9.803491488881253 + - type: nauc_precision_at_5_diff1 + value: 18.218141491873933 + - type: nauc_precision_at_5_max + value: 25.63292715320136 + - type: nauc_precision_at_5_std + value: 15.125718294075579 + - type: nauc_recall_at_1000_diff1 + value: 45.64721815441147 + - type: nauc_recall_at_1000_max + value: 48.216562415882606 + - type: nauc_recall_at_1000_std + value: 40.18167967343722 + - type: nauc_recall_at_100_diff1 + value: 27.550190465714813 + - type: nauc_recall_at_100_max + value: 29.636326305334048 + - type: nauc_recall_at_100_std + value: 30.20966431974901 + - type: nauc_recall_at_10_diff1 + value: 34.30589013251178 + - type: nauc_recall_at_10_max + value: 30.383485547646043 + - type: nauc_recall_at_10_std + value: 10.0628614956005 + - type: nauc_recall_at_1_diff1 + value: 50.063501481714425 + - type: nauc_recall_at_1_max + value: 26.941170343192383 + - type: nauc_recall_at_1_std + value: 1.2783766588617693 + - type: nauc_recall_at_20_diff1 + value: 30.74543679537666 + - type: nauc_recall_at_20_max + value: 30.138740866468787 + - type: nauc_recall_at_20_std + value: 19.995537838999724 + - type: nauc_recall_at_3_diff1 + value: 38.652151861880654 + - type: nauc_recall_at_3_max + value: 28.806612417729582 + - type: nauc_recall_at_3_std + value: 5.321890758808099 + - type: nauc_recall_at_5_diff1 + value: 34.689245376303504 + - type: nauc_recall_at_5_max + value: 28.89342892932615 + - type: nauc_recall_at_5_std + value: 9.12730045136017 + - type: ndcg_at_1 + value: 30.830000000000002 + - type: ndcg_at_10 + value: 40.617 + - type: ndcg_at_100 + value: 46.914 + - type: ndcg_at_1000 + value: 49.319 + - type: ndcg_at_20 + value: 43.206 + - type: ndcg_at_3 + value: 35.271 + - type: ndcg_at_5 + value: 37.436 + - type: precision_at_1 + value: 30.830000000000002 + - type: precision_at_10 + value: 7.767 + - type: precision_at_100 + value: 1.603 + - type: precision_at_1000 + value: 0.243 + - type: precision_at_20 + value: 5.059 + - type: precision_at_3 + value: 16.337 + - type: precision_at_5 + value: 11.66 + - type: recall_at_1 + value: 26.124000000000002 + - type: recall_at_10 + value: 52.015 + - type: recall_at_100 + value: 79.729 + - type: recall_at_1000 + value: 94.476 + - type: recall_at_20 + value: 61.519999999999996 + - type: recall_at_3 + value: 37.037 + - type: recall_at_5 + value: 43.034 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackWordpressRetrieval + revision: 4ffe81d471b1924886b33c7567bfb200e9eec5c4 + split: test + type: mteb/cqadupstack-wordpress + metrics: + - type: main_score + value: 33.94 + - type: map_at_1 + value: 21.728 + - type: map_at_10 + value: 29.213 + - type: map_at_100 + value: 30.275000000000002 + - type: map_at_1000 + value: 30.387999999999998 + - type: map_at_20 + value: 29.921 + - type: map_at_3 + value: 26.678 + - type: map_at_5 + value: 27.97 + - type: mrr_at_1 + value: 24.029574861367838 + - type: mrr_at_10 + value: 31.411040694774528 + - type: mrr_at_100 + value: 32.37670039416044 + - type: mrr_at_1000 + value: 32.45828597542603 + - type: mrr_at_20 + value: 32.066710418311054 + - type: mrr_at_3 + value: 29.174368453481215 + - type: mrr_at_5 + value: 30.218730745532962 + - type: nauc_map_at_1000_diff1 + value: 31.91257362997741 + - type: nauc_map_at_1000_max + value: 30.88489579944952 + - type: nauc_map_at_1000_std + value: -0.8378972209872841 + - type: nauc_map_at_100_diff1 + value: 31.91012136001019 + - type: nauc_map_at_100_max + value: 30.89495455322903 + - type: nauc_map_at_100_std + value: -0.8363099421850583 + - type: nauc_map_at_10_diff1 + value: 32.321541752194044 + - type: nauc_map_at_10_max + value: 30.478856413691663 + - type: nauc_map_at_10_std + value: -1.7937789797513617 + - type: nauc_map_at_1_diff1 + value: 39.036529785500946 + - type: nauc_map_at_1_max + value: 30.15527995919205 + - type: nauc_map_at_1_std + value: -3.0753665573586875 + - type: nauc_map_at_20_diff1 + value: 31.995975321229412 + - type: nauc_map_at_20_max + value: 30.841371991167055 + - type: nauc_map_at_20_std + value: -1.0599571122276743 + - type: nauc_map_at_3_diff1 + value: 33.37828040602051 + - type: nauc_map_at_3_max + value: 30.63773296692759 + - type: nauc_map_at_3_std + value: -2.6306505462108145 + - type: nauc_map_at_5_diff1 + value: 32.33262472170732 + - type: nauc_map_at_5_max + value: 30.968294579332984 + - type: nauc_map_at_5_std + value: -2.2441870438191134 + - type: nauc_mrr_at_1000_diff1 + value: 32.924621843184504 + - type: nauc_mrr_at_1000_max + value: 33.163623964941976 + - type: nauc_mrr_at_1000_std + value: 0.16588136365013925 + - type: nauc_mrr_at_100_diff1 + value: 32.906505756942536 + - type: nauc_mrr_at_100_max + value: 33.17000246238509 + - type: nauc_mrr_at_100_std + value: 0.17008029070490974 + - type: nauc_mrr_at_10_diff1 + value: 33.29141185817138 + - type: nauc_mrr_at_10_max + value: 33.07902410006761 + - type: nauc_mrr_at_10_std + value: -0.5588437886232123 + - type: nauc_mrr_at_1_diff1 + value: 39.41491848273575 + - type: nauc_mrr_at_1_max + value: 33.1045906492973 + - type: nauc_mrr_at_1_std + value: -1.499033772891368 + - type: nauc_mrr_at_20_diff1 + value: 33.03454429443921 + - type: nauc_mrr_at_20_max + value: 33.23117853960766 + - type: nauc_mrr_at_20_std + value: 0.026450200924183646 + - type: nauc_mrr_at_3_diff1 + value: 34.273859138345905 + - type: nauc_mrr_at_3_max + value: 33.709490350419216 + - type: nauc_mrr_at_3_std + value: -1.0280318099262806 + - type: nauc_mrr_at_5_diff1 + value: 33.22838031675215 + - type: nauc_mrr_at_5_max + value: 33.47021788245831 + - type: nauc_mrr_at_5_std + value: -1.020392885760047 + - type: nauc_ndcg_at_1000_diff1 + value: 29.228313546730394 + - type: nauc_ndcg_at_1000_max + value: 31.30181847993907 + - type: nauc_ndcg_at_1000_std + value: 2.4004671163706286 + - type: nauc_ndcg_at_100_diff1 + value: 29.007140166456352 + - type: nauc_ndcg_at_100_max + value: 31.177548656556088 + - type: nauc_ndcg_at_100_std + value: 2.84104689095153 + - type: nauc_ndcg_at_10_diff1 + value: 30.333521308398282 + - type: nauc_ndcg_at_10_max + value: 30.443915052520797 + - type: nauc_ndcg_at_10_std + value: -0.7776378511941929 + - type: nauc_ndcg_at_1_diff1 + value: 39.41491848273575 + - type: nauc_ndcg_at_1_max + value: 33.1045906492973 + - type: nauc_ndcg_at_1_std + value: -1.499033772891368 + - type: nauc_ndcg_at_20_diff1 + value: 29.35528623170515 + - type: nauc_ndcg_at_20_max + value: 31.251908593602046 + - type: nauc_ndcg_at_20_std + value: 1.5335554093420756 + - type: nauc_ndcg_at_3_diff1 + value: 32.00742763587945 + - type: nauc_ndcg_at_3_max + value: 31.43411749352576 + - type: nauc_ndcg_at_3_std + value: -2.161005612286732 + - type: nauc_ndcg_at_5_diff1 + value: 30.218028443882382 + - type: nauc_ndcg_at_5_max + value: 31.436372820890888 + - type: nauc_ndcg_at_5_std + value: -1.8970324616117942 + - type: nauc_precision_at_1000_diff1 + value: -15.474984333696856 + - type: nauc_precision_at_1000_max + value: -0.5988176438045155 + - type: nauc_precision_at_1000_std + value: 9.649508553406482 + - type: nauc_precision_at_100_diff1 + value: 6.827122527618587 + - type: nauc_precision_at_100_max + value: 25.38874258318603 + - type: nauc_precision_at_100_std + value: 19.74692016049829 + - type: nauc_precision_at_10_diff1 + value: 21.924392614008934 + - type: nauc_precision_at_10_max + value: 30.50809713248987 + - type: nauc_precision_at_10_std + value: 5.182344376975003 + - type: nauc_precision_at_1_diff1 + value: 39.41491848273575 + - type: nauc_precision_at_1_max + value: 33.1045906492973 + - type: nauc_precision_at_1_std + value: -1.499033772891368 + - type: nauc_precision_at_20_diff1 + value: 17.384644579807794 + - type: nauc_precision_at_20_max + value: 31.910020552998546 + - type: nauc_precision_at_20_std + value: 14.10235751314262 + - type: nauc_precision_at_3_diff1 + value: 27.21660611201887 + - type: nauc_precision_at_3_max + value: 32.731203308832185 + - type: nauc_precision_at_3_std + value: -1.2935765023235843 + - type: nauc_precision_at_5_diff1 + value: 23.103364778326657 + - type: nauc_precision_at_5_max + value: 33.42544453577581 + - type: nauc_precision_at_5_std + value: 0.539665839883537 + - type: nauc_recall_at_1000_diff1 + value: -2.224955557104963 + - type: nauc_recall_at_1000_max + value: 23.905681787331996 + - type: nauc_recall_at_1000_std + value: 29.343916482872185 + - type: nauc_recall_at_100_diff1 + value: 15.113380525840887 + - type: nauc_recall_at_100_max + value: 25.613295770648165 + - type: nauc_recall_at_100_std + value: 18.315582271550408 + - type: nauc_recall_at_10_diff1 + value: 23.40683183845192 + - type: nauc_recall_at_10_max + value: 26.57810440107745 + - type: nauc_recall_at_10_std + value: 0.9875785918360439 + - type: nauc_recall_at_1_diff1 + value: 39.036529785500946 + - type: nauc_recall_at_1_max + value: 30.15527995919205 + - type: nauc_recall_at_1_std + value: -3.0753665573586875 + - type: nauc_recall_at_20_diff1 + value: 18.78081005808975 + - type: nauc_recall_at_20_max + value: 28.341844976465215 + - type: nauc_recall_at_20_std + value: 9.15244411341802 + - type: nauc_recall_at_3_diff1 + value: 28.011427372719595 + - type: nauc_recall_at_3_max + value: 30.42588662301828 + - type: nauc_recall_at_3_std + value: -2.2738647119625295 + - type: nauc_recall_at_5_diff1 + value: 23.97361585809799 + - type: nauc_recall_at_5_max + value: 30.419498034627367 + - type: nauc_recall_at_5_std + value: -1.9202931420769696 + - type: ndcg_at_1 + value: 24.03 + - type: ndcg_at_10 + value: 33.94 + - type: ndcg_at_100 + value: 38.911 + - type: ndcg_at_1000 + value: 41.732 + - type: ndcg_at_20 + value: 36.327999999999996 + - type: ndcg_at_3 + value: 29.011 + - type: ndcg_at_5 + value: 31.020999999999997 + - type: precision_at_1 + value: 24.03 + - type: precision_at_10 + value: 5.36 + - type: precision_at_100 + value: 0.8410000000000001 + - type: precision_at_1000 + value: 0.11900000000000001 + - type: precision_at_20 + value: 3.2439999999999998 + - type: precision_at_3 + value: 12.323 + - type: precision_at_5 + value: 8.613999999999999 + - type: recall_at_1 + value: 21.728 + - type: recall_at_10 + value: 46.562 + - type: recall_at_100 + value: 69.16799999999999 + - type: recall_at_1000 + value: 90.325 + - type: recall_at_20 + value: 55.644000000000005 + - type: recall_at_3 + value: 32.772 + - type: recall_at_5 + value: 37.683 + task: + type: Retrieval + - dataset: + config: default + name: MTEB ClimateFEVER + revision: 47f2ac6acb640fc46020b02a5b59fdda04d39380 + split: test + type: mteb/climate-fever + metrics: + - type: main_score + value: 25.854 + - type: map_at_1 + value: 10.267 + - type: map_at_10 + value: 18.04 + - type: map_at_100 + value: 19.777 + - type: map_at_1000 + value: 19.958000000000002 + - type: map_at_20 + value: 19.008 + - type: map_at_3 + value: 14.879999999999999 + - type: map_at_5 + value: 16.485 + - type: mrr_at_1 + value: 22.996742671009773 + - type: mrr_at_10 + value: 34.10619926580836 + - type: mrr_at_100 + value: 35.10535379461663 + - type: mrr_at_1000 + value: 35.15755884184264 + - type: mrr_at_20 + value: 34.75454375956081 + - type: mrr_at_3 + value: 30.597176981541768 + - type: mrr_at_5 + value: 32.6753528773072 + - type: nauc_map_at_1000_diff1 + value: 16.718537725864763 + - type: nauc_map_at_1000_max + value: 39.502981434057595 + - type: nauc_map_at_1000_std + value: 22.12042839190954 + - type: nauc_map_at_100_diff1 + value: 16.75220577723001 + - type: nauc_map_at_100_max + value: 39.48722479833852 + - type: nauc_map_at_100_std + value: 22.055881777406448 + - type: nauc_map_at_10_diff1 + value: 16.874316455401654 + - type: nauc_map_at_10_max + value: 39.596829557733344 + - type: nauc_map_at_10_std + value: 20.473758607475666 + - type: nauc_map_at_1_diff1 + value: 23.080079172466412 + - type: nauc_map_at_1_max + value: 36.199027660019325 + - type: nauc_map_at_1_std + value: 14.511081393076179 + - type: nauc_map_at_20_diff1 + value: 16.839381398924182 + - type: nauc_map_at_20_max + value: 39.67988867337892 + - type: nauc_map_at_20_std + value: 21.35532096892699 + - type: nauc_map_at_3_diff1 + value: 18.312687041708102 + - type: nauc_map_at_3_max + value: 38.769416119220736 + - type: nauc_map_at_3_std + value: 16.175068336485026 + - type: nauc_map_at_5_diff1 + value: 17.69288278300591 + - type: nauc_map_at_5_max + value: 39.2535585829467 + - type: nauc_map_at_5_std + value: 18.460583770602113 + - type: nauc_mrr_at_1000_diff1 + value: 17.91231859721048 + - type: nauc_mrr_at_1000_max + value: 35.48664780287595 + - type: nauc_mrr_at_1000_std + value: 22.024407060245295 + - type: nauc_mrr_at_100_diff1 + value: 17.90154896033313 + - type: nauc_mrr_at_100_max + value: 35.48493858099936 + - type: nauc_mrr_at_100_std + value: 22.031137347443284 + - type: nauc_mrr_at_10_diff1 + value: 17.764970868121992 + - type: nauc_mrr_at_10_max + value: 35.647414618204834 + - type: nauc_mrr_at_10_std + value: 22.177693762457217 + - type: nauc_mrr_at_1_diff1 + value: 20.72060270445967 + - type: nauc_mrr_at_1_max + value: 32.05796435073919 + - type: nauc_mrr_at_1_std + value: 17.801001796830647 + - type: nauc_mrr_at_20_diff1 + value: 17.81229976929074 + - type: nauc_mrr_at_20_max + value: 35.508834038837286 + - type: nauc_mrr_at_20_std + value: 22.03930127358993 + - type: nauc_mrr_at_3_diff1 + value: 18.184760750984967 + - type: nauc_mrr_at_3_max + value: 35.06640214914882 + - type: nauc_mrr_at_3_std + value: 19.85666178099386 + - type: nauc_mrr_at_5_diff1 + value: 17.94734753821123 + - type: nauc_mrr_at_5_max + value: 35.5362671414369 + - type: nauc_mrr_at_5_std + value: 21.65318475967923 + - type: nauc_ndcg_at_1000_diff1 + value: 14.873923990787 + - type: nauc_ndcg_at_1000_max + value: 38.880238913938264 + - type: nauc_ndcg_at_1000_std + value: 28.053954097672957 + - type: nauc_ndcg_at_100_diff1 + value: 14.878869135136199 + - type: nauc_ndcg_at_100_max + value: 38.4728834388698 + - type: nauc_ndcg_at_100_std + value: 27.260605585380198 + - type: nauc_ndcg_at_10_diff1 + value: 15.052663192838656 + - type: nauc_ndcg_at_10_max + value: 39.31146423709117 + - type: nauc_ndcg_at_10_std + value: 23.825559445880963 + - type: nauc_ndcg_at_1_diff1 + value: 20.72060270445967 + - type: nauc_ndcg_at_1_max + value: 32.05796435073919 + - type: nauc_ndcg_at_1_std + value: 17.801001796830647 + - type: nauc_ndcg_at_20_diff1 + value: 14.949446910080416 + - type: nauc_ndcg_at_20_max + value: 39.12075299095604 + - type: nauc_ndcg_at_20_std + value: 25.17649714577982 + - type: nauc_ndcg_at_3_diff1 + value: 17.117859449815708 + - type: nauc_ndcg_at_3_max + value: 37.86742070994964 + - type: nauc_ndcg_at_3_std + value: 17.929095022194748 + - type: nauc_ndcg_at_5_diff1 + value: 16.552568744225216 + - type: nauc_ndcg_at_5_max + value: 39.07328909408646 + - type: nauc_ndcg_at_5_std + value: 21.087837182021957 + - type: nauc_precision_at_1000_diff1 + value: -2.870078971356406 + - type: nauc_precision_at_1000_max + value: 7.02871444829994 + - type: nauc_precision_at_1000_std + value: 23.899933947819544 + - type: nauc_precision_at_100_diff1 + value: 1.5886111188050933 + - type: nauc_precision_at_100_max + value: 15.935337497887625 + - type: nauc_precision_at_100_std + value: 29.556371474338345 + - type: nauc_precision_at_10_diff1 + value: 6.300177923780286 + - type: nauc_precision_at_10_max + value: 31.243660622639485 + - type: nauc_precision_at_10_std + value: 29.554919355292093 + - type: nauc_precision_at_1_diff1 + value: 20.72060270445967 + - type: nauc_precision_at_1_max + value: 32.05796435073919 + - type: nauc_precision_at_1_std + value: 17.801001796830647 + - type: nauc_precision_at_20_diff1 + value: 4.669392584982594 + - type: nauc_precision_at_20_max + value: 25.761327413181178 + - type: nauc_precision_at_20_std + value: 29.741403307857166 + - type: nauc_precision_at_3_diff1 + value: 13.367853392936027 + - type: nauc_precision_at_3_max + value: 36.957826896424336 + - type: nauc_precision_at_3_std + value: 21.740373439631725 + - type: nauc_precision_at_5_diff1 + value: 10.690045616058065 + - type: nauc_precision_at_5_max + value: 34.71275252243625 + - type: nauc_precision_at_5_std + value: 26.90693165393696 + - type: nauc_recall_at_1000_diff1 + value: 5.712445342001347 + - type: nauc_recall_at_1000_max + value: 30.531136218449 + - type: nauc_recall_at_1000_std + value: 39.216291284266546 + - type: nauc_recall_at_100_diff1 + value: 6.9984637463611925 + - type: nauc_recall_at_100_max + value: 29.566627607364083 + - type: nauc_recall_at_100_std + value: 31.079777791452766 + - type: nauc_recall_at_10_diff1 + value: 8.262518659498706 + - type: nauc_recall_at_10_max + value: 35.13566657953036 + - type: nauc_recall_at_10_std + value: 24.290738932523727 + - type: nauc_recall_at_1_diff1 + value: 23.080079172466412 + - type: nauc_recall_at_1_max + value: 36.199027660019325 + - type: nauc_recall_at_1_std + value: 14.511081393076179 + - type: nauc_recall_at_20_diff1 + value: 7.680549244887501 + - type: nauc_recall_at_20_max + value: 33.289199278919206 + - type: nauc_recall_at_20_std + value: 25.97835574263474 + - type: nauc_recall_at_3_diff1 + value: 13.66331680167438 + - type: nauc_recall_at_3_max + value: 37.36441061109965 + - type: nauc_recall_at_3_std + value: 15.4241070336601 + - type: nauc_recall_at_5_diff1 + value: 12.173354391955048 + - type: nauc_recall_at_5_max + value: 37.13679975852594 + - type: nauc_recall_at_5_std + value: 20.394479290376545 + - type: ndcg_at_1 + value: 22.997 + - type: ndcg_at_10 + value: 25.854 + - type: ndcg_at_100 + value: 32.74 + - type: ndcg_at_1000 + value: 36.187000000000005 + - type: ndcg_at_20 + value: 28.692 + - type: ndcg_at_3 + value: 20.575 + - type: ndcg_at_5 + value: 22.55 + - type: precision_at_1 + value: 22.997 + - type: precision_at_10 + value: 8.189 + - type: precision_at_100 + value: 1.541 + - type: precision_at_1000 + value: 0.218 + - type: precision_at_20 + value: 5.296 + - type: precision_at_3 + value: 15.244 + - type: precision_at_5 + value: 12.052 + - type: recall_at_1 + value: 10.267 + - type: recall_at_10 + value: 32.125 + - type: recall_at_100 + value: 55.696999999999996 + - type: recall_at_1000 + value: 75.074 + - type: recall_at_20 + value: 40.193 + - type: recall_at_3 + value: 19.509 + - type: recall_at_5 + value: 24.747 + task: + type: Retrieval + - dataset: + config: default + name: MTEB DBPedia + revision: c0f706b76e590d620bd6618b3ca8efdd34e2d659 + split: test + type: mteb/dbpedia + metrics: + - type: main_score + value: 38.936 + - type: map_at_1 + value: 7.969999999999999 + - type: map_at_10 + value: 18.916 + - type: map_at_100 + value: 26.464 + - type: map_at_1000 + value: 28.203 + - type: map_at_20 + value: 21.671000000000003 + - type: map_at_3 + value: 13.544 + - type: map_at_5 + value: 15.964 + - type: mrr_at_1 + value: 62.25000000000001 + - type: mrr_at_10 + value: 72.06656746031746 + - type: mrr_at_100 + value: 72.38229975273143 + - type: mrr_at_1000 + value: 72.38878819640294 + - type: mrr_at_20 + value: 72.28867599870888 + - type: mrr_at_3 + value: 69.87500000000001 + - type: mrr_at_5 + value: 71.36250000000001 + - type: nauc_map_at_1000_diff1 + value: 19.321937510366077 + - type: nauc_map_at_1000_max + value: 13.747024166838576 + - type: nauc_map_at_1000_std + value: 20.868788180619273 + - type: nauc_map_at_100_diff1 + value: 19.64393142562319 + - type: nauc_map_at_100_max + value: 10.251028144343387 + - type: nauc_map_at_100_std + value: 18.068372645123112 + - type: nauc_map_at_10_diff1 + value: 23.84916025839186 + - type: nauc_map_at_10_max + value: 0.09170851785343646 + - type: nauc_map_at_10_std + value: -0.9386497395956033 + - type: nauc_map_at_1_diff1 + value: 35.48615861613537 + - type: nauc_map_at_1_max + value: -4.968376038142286 + - type: nauc_map_at_1_std + value: -8.722029416996381 + - type: nauc_map_at_20_diff1 + value: 22.041254266944954 + - type: nauc_map_at_20_max + value: 3.54673862289231 + - type: nauc_map_at_20_std + value: 6.076866601495746 + - type: nauc_map_at_3_diff1 + value: 27.930757708983773 + - type: nauc_map_at_3_max + value: -4.53116277437896 + - type: nauc_map_at_3_std + value: -7.364340365588032 + - type: nauc_map_at_5_diff1 + value: 24.465395628826034 + - type: nauc_map_at_5_max + value: -3.105817257141636 + - type: nauc_map_at_5_std + value: -5.855744275070062 + - type: nauc_mrr_at_1000_diff1 + value: 42.514933484267644 + - type: nauc_mrr_at_1000_max + value: 40.515988358376646 + - type: nauc_mrr_at_1000_std + value: 30.22699534554659 + - type: nauc_mrr_at_100_diff1 + value: 42.51475590269715 + - type: nauc_mrr_at_100_max + value: 40.49923698224623 + - type: nauc_mrr_at_100_std + value: 30.235830229223904 + - type: nauc_mrr_at_10_diff1 + value: 42.45982313098615 + - type: nauc_mrr_at_10_max + value: 40.23468589619149 + - type: nauc_mrr_at_10_std + value: 30.217961535871073 + - type: nauc_mrr_at_1_diff1 + value: 45.27134837755035 + - type: nauc_mrr_at_1_max + value: 38.4098873035926 + - type: nauc_mrr_at_1_std + value: 25.42449495242179 + - type: nauc_mrr_at_20_diff1 + value: 42.50561799373592 + - type: nauc_mrr_at_20_max + value: 40.46341284202719 + - type: nauc_mrr_at_20_std + value: 30.176989437307306 + - type: nauc_mrr_at_3_diff1 + value: 42.015465693213386 + - type: nauc_mrr_at_3_max + value: 40.88173938795508 + - type: nauc_mrr_at_3_std + value: 31.535958619028225 + - type: nauc_mrr_at_5_diff1 + value: 42.13527304829269 + - type: nauc_mrr_at_5_max + value: 40.612539610295975 + - type: nauc_mrr_at_5_std + value: 30.670846095147496 + - type: nauc_ndcg_at_1000_diff1 + value: 20.55967654606238 + - type: nauc_ndcg_at_1000_max + value: 24.316462778279867 + - type: nauc_ndcg_at_1000_std + value: 31.496336109850247 + - type: nauc_ndcg_at_100_diff1 + value: 20.631201886740783 + - type: nauc_ndcg_at_100_max + value: 13.797070619958301 + - type: nauc_ndcg_at_100_std + value: 24.805397171967755 + - type: nauc_ndcg_at_10_diff1 + value: 24.434849609019665 + - type: nauc_ndcg_at_10_max + value: 18.074612480062648 + - type: nauc_ndcg_at_10_std + value: 19.17394905361306 + - type: nauc_ndcg_at_1_diff1 + value: 36.96000425769715 + - type: nauc_ndcg_at_1_max + value: 29.509211705119004 + - type: nauc_ndcg_at_1_std + value: 18.50980600867504 + - type: nauc_ndcg_at_20_diff1 + value: 22.868106188391877 + - type: nauc_ndcg_at_20_max + value: 12.64309423251832 + - type: nauc_ndcg_at_20_std + value: 18.94830361891266 + - type: nauc_ndcg_at_3_diff1 + value: 27.93885229724602 + - type: nauc_ndcg_at_3_max + value: 24.97470001983418 + - type: nauc_ndcg_at_3_std + value: 19.53721404824316 + - type: nauc_ndcg_at_5_diff1 + value: 24.096100442606105 + - type: nauc_ndcg_at_5_max + value: 21.017433702322183 + - type: nauc_ndcg_at_5_std + value: 18.03214925832656 + - type: nauc_precision_at_1000_diff1 + value: -5.070197509753125 + - type: nauc_precision_at_1000_max + value: 38.94682716070116 + - type: nauc_precision_at_1000_std + value: 13.85365496749175 + - type: nauc_precision_at_100_diff1 + value: -6.234968963931368 + - type: nauc_precision_at_100_max + value: 30.591823702241243 + - type: nauc_precision_at_100_std + value: 40.95192411908243 + - type: nauc_precision_at_10_diff1 + value: 3.7146822473142125 + - type: nauc_precision_at_10_max + value: 35.234593007969735 + - type: nauc_precision_at_10_std + value: 39.26820665194708 + - type: nauc_precision_at_1_diff1 + value: 45.27134837755035 + - type: nauc_precision_at_1_max + value: 38.4098873035926 + - type: nauc_precision_at_1_std + value: 25.42449495242179 + - type: nauc_precision_at_20_diff1 + value: -1.1101702192957978 + - type: nauc_precision_at_20_max + value: 32.05616068273549 + - type: nauc_precision_at_20_std + value: 44.191729349771 + - type: nauc_precision_at_3_diff1 + value: 13.654921924572067 + - type: nauc_precision_at_3_max + value: 34.080562463989054 + - type: nauc_precision_at_3_std + value: 30.352925731706453 + - type: nauc_precision_at_5_diff1 + value: 5.007711615213191 + - type: nauc_precision_at_5_max + value: 33.571297387924865 + - type: nauc_precision_at_5_std + value: 33.62738186220206 + - type: nauc_recall_at_1000_diff1 + value: 8.24800794780627 + - type: nauc_recall_at_1000_max + value: 17.19134351585966 + - type: nauc_recall_at_1000_std + value: 35.214343810310936 + - type: nauc_recall_at_100_diff1 + value: 9.207152847999645 + - type: nauc_recall_at_100_max + value: 2.2099214016635353 + - type: nauc_recall_at_100_std + value: 21.652430811892163 + - type: nauc_recall_at_10_diff1 + value: 16.735926854286163 + - type: nauc_recall_at_10_max + value: -8.153155643353733 + - type: nauc_recall_at_10_std + value: -8.04241419757528 + - type: nauc_recall_at_1_diff1 + value: 35.48615861613537 + - type: nauc_recall_at_1_max + value: -4.968376038142286 + - type: nauc_recall_at_1_std + value: -8.722029416996381 + - type: nauc_recall_at_20_diff1 + value: 13.71195242126747 + - type: nauc_recall_at_20_max + value: -5.54857038214976 + - type: nauc_recall_at_20_std + value: 0.5747495657296262 + - type: nauc_recall_at_3_diff1 + value: 23.73157744159544 + - type: nauc_recall_at_3_max + value: -8.898620891937641 + - type: nauc_recall_at_3_std + value: -9.357793995872202 + - type: nauc_recall_at_5_diff1 + value: 16.331831174073077 + - type: nauc_recall_at_5_max + value: -9.740756642670778 + - type: nauc_recall_at_5_std + value: -11.361186585459222 + - type: ndcg_at_1 + value: 49.125 + - type: ndcg_at_10 + value: 38.936 + - type: ndcg_at_100 + value: 44.42 + - type: ndcg_at_1000 + value: 52.080000000000005 + - type: ndcg_at_20 + value: 38.483000000000004 + - type: ndcg_at_3 + value: 42.224000000000004 + - type: ndcg_at_5 + value: 40.263 + - type: precision_at_1 + value: 62.25000000000001 + - type: precision_at_10 + value: 31.125000000000004 + - type: precision_at_100 + value: 10.168000000000001 + - type: precision_at_1000 + value: 2.2089999999999996 + - type: precision_at_20 + value: 23.462 + - type: precision_at_3 + value: 47.583 + - type: precision_at_5 + value: 40.5 + - type: recall_at_1 + value: 7.969999999999999 + - type: recall_at_10 + value: 25.774 + - type: recall_at_100 + value: 52.913 + - type: recall_at_1000 + value: 76.281 + - type: recall_at_20 + value: 32.346000000000004 + - type: recall_at_3 + value: 15.304 + - type: recall_at_5 + value: 19.641000000000002 + task: + type: Retrieval + - dataset: + config: default + name: MTEB EmotionClassification + revision: 4f58c6b202a23cf9a4da393831edf4f9183cad37 + split: test + type: mteb/emotion + metrics: + - type: accuracy + value: 86.9 + - type: f1 + value: 82.43466297441167 + - type: f1_weighted + value: 87.21113710044447 + - type: main_score + value: 86.9 + task: + type: Classification + - dataset: + config: default + name: MTEB FEVER + revision: bea83ef9e8fb933d90a2f1d5515737465d613e12 + split: test + type: mteb/fever + metrics: + - type: main_score + value: 86.53999999999999 + - type: map_at_1 + value: 74.97399999999999 + - type: map_at_10 + value: 82.659 + - type: map_at_100 + value: 82.836 + - type: map_at_1000 + value: 82.851 + - type: map_at_20 + value: 82.765 + - type: map_at_3 + value: 81.61500000000001 + - type: map_at_5 + value: 82.313 + - type: mrr_at_1 + value: 80.87308730873087 + - type: mrr_at_10 + value: 88.05529957757663 + - type: mrr_at_100 + value: 88.11601292565871 + - type: mrr_at_1000 + value: 88.11677924985968 + - type: mrr_at_20 + value: 88.10230041194373 + - type: mrr_at_3 + value: 87.29872987298717 + - type: mrr_at_5 + value: 87.8365336533651 + - type: nauc_map_at_1000_diff1 + value: 46.70261215324498 + - type: nauc_map_at_1000_max + value: 7.04045426956567 + - type: nauc_map_at_1000_std + value: -25.016067857784936 + - type: nauc_map_at_100_diff1 + value: 46.67118950260566 + - type: nauc_map_at_100_max + value: 7.024337825785156 + - type: nauc_map_at_100_std + value: -25.0179014964629 + - type: nauc_map_at_10_diff1 + value: 46.48727033111116 + - type: nauc_map_at_10_max + value: 7.030297220664678 + - type: nauc_map_at_10_std + value: -24.859656571920247 + - type: nauc_map_at_1_diff1 + value: 52.514894169034974 + - type: nauc_map_at_1_max + value: 6.4991819409595415 + - type: nauc_map_at_1_std + value: -26.962624287912067 + - type: nauc_map_at_20_diff1 + value: 46.58413925547691 + - type: nauc_map_at_20_max + value: 6.975619778750194 + - type: nauc_map_at_20_std + value: -24.984054859146365 + - type: nauc_map_at_3_diff1 + value: 46.35115084281979 + - type: nauc_map_at_3_max + value: 6.268645069341729 + - type: nauc_map_at_3_std + value: -25.449448105561935 + - type: nauc_map_at_5_diff1 + value: 46.25768358653106 + - type: nauc_map_at_5_max + value: 6.907729829933622 + - type: nauc_map_at_5_std + value: -24.94987246230526 + - type: nauc_mrr_at_1000_diff1 + value: 65.35505807237064 + - type: nauc_mrr_at_1000_max + value: 10.30647297142242 + - type: nauc_mrr_at_1000_std + value: -37.369411728966185 + - type: nauc_mrr_at_100_diff1 + value: 65.3558763604295 + - type: nauc_mrr_at_100_max + value: 10.310911594013163 + - type: nauc_mrr_at_100_std + value: -37.371872361808315 + - type: nauc_mrr_at_10_diff1 + value: 65.28234257793653 + - type: nauc_mrr_at_10_max + value: 10.446154148428406 + - type: nauc_mrr_at_10_std + value: -37.43279989381229 + - type: nauc_mrr_at_1_diff1 + value: 67.43046114024251 + - type: nauc_mrr_at_1_max + value: 9.199791010999588 + - type: nauc_mrr_at_1_std + value: -35.72647316332683 + - type: nauc_mrr_at_20_diff1 + value: 65.33718094349057 + - type: nauc_mrr_at_20_max + value: 10.36042922044663 + - type: nauc_mrr_at_20_std + value: -37.39896601561335 + - type: nauc_mrr_at_3_diff1 + value: 64.75542682908058 + - type: nauc_mrr_at_3_max + value: 9.81685738005414 + - type: nauc_mrr_at_3_std + value: -37.586465861627715 + - type: nauc_mrr_at_5_diff1 + value: 64.97675653138579 + - type: nauc_mrr_at_5_max + value: 10.473645964548574 + - type: nauc_mrr_at_5_std + value: -37.517720854364725 + - type: nauc_ndcg_at_1000_diff1 + value: 47.980879951824015 + - type: nauc_ndcg_at_1000_max + value: 8.294784645952213 + - type: nauc_ndcg_at_1000_std + value: -25.68035961439311 + - type: nauc_ndcg_at_100_diff1 + value: 47.17557297242816 + - type: nauc_ndcg_at_100_max + value: 7.97932909647729 + - type: nauc_ndcg_at_100_std + value: -25.656209456961797 + - type: nauc_ndcg_at_10_diff1 + value: 46.44829982304173 + - type: nauc_ndcg_at_10_max + value: 8.10910045641653 + - type: nauc_ndcg_at_10_std + value: -25.23311748908822 + - type: nauc_ndcg_at_1_diff1 + value: 67.43046114024251 + - type: nauc_ndcg_at_1_max + value: 9.199791010999588 + - type: nauc_ndcg_at_1_std + value: -35.72647316332683 + - type: nauc_ndcg_at_20_diff1 + value: 46.71605953667202 + - type: nauc_ndcg_at_20_max + value: 7.83399009429542 + - type: nauc_ndcg_at_20_std + value: -25.50914427951171 + - type: nauc_ndcg_at_3_diff1 + value: 47.09767655901834 + - type: nauc_ndcg_at_3_max + value: 7.093614795583601 + - type: nauc_ndcg_at_3_std + value: -27.099378097753064 + - type: nauc_ndcg_at_5_diff1 + value: 46.230227787105676 + - type: nauc_ndcg_at_5_max + value: 7.979234136381932 + - type: nauc_ndcg_at_5_std + value: -25.748004074053082 + - type: nauc_precision_at_1000_diff1 + value: -5.378736053404685 + - type: nauc_precision_at_1000_max + value: 10.355124801647387 + - type: nauc_precision_at_1000_std + value: 2.9095091245038938 + - type: nauc_precision_at_100_diff1 + value: -6.8190843049279835 + - type: nauc_precision_at_100_max + value: 10.296510275756647 + - type: nauc_precision_at_100_std + value: -0.5694377374023152 + - type: nauc_precision_at_10_diff1 + value: 3.3308965054246853 + - type: nauc_precision_at_10_max + value: 13.731815856075642 + - type: nauc_precision_at_10_std + value: -6.390781452502162 + - type: nauc_precision_at_1_diff1 + value: 67.43046114024251 + - type: nauc_precision_at_1_max + value: 9.199791010999588 + - type: nauc_precision_at_1_std + value: -35.72647316332683 + - type: nauc_precision_at_20_diff1 + value: -2.689991219278092 + - type: nauc_precision_at_20_max + value: 10.077673782221831 + - type: nauc_precision_at_20_std + value: -5.163539651519823 + - type: nauc_precision_at_3_diff1 + value: 32.39846541175425 + - type: nauc_precision_at_3_max + value: 8.137383096292048 + - type: nauc_precision_at_3_std + value: -29.232206161111595 + - type: nauc_precision_at_5_diff1 + value: 15.859211194923645 + - type: nauc_precision_at_5_max + value: 13.315538292037735 + - type: nauc_precision_at_5_std + value: -16.424307328219072 + - type: nauc_recall_at_1000_diff1 + value: -1.4876601880723042 + - type: nauc_recall_at_1000_max + value: 13.281504681725965 + - type: nauc_recall_at_1000_std + value: 20.088387638135956 + - type: nauc_recall_at_100_diff1 + value: 3.510068775817679 + - type: nauc_recall_at_100_max + value: 7.223126729279164 + - type: nauc_recall_at_100_std + value: 1.8098697270167292 + - type: nauc_recall_at_10_diff1 + value: 13.282399274249123 + - type: nauc_recall_at_10_max + value: 9.5577982617337 + - type: nauc_recall_at_10_std + value: -5.244097776935379 + - type: nauc_recall_at_1_diff1 + value: 52.514894169034974 + - type: nauc_recall_at_1_max + value: 6.4991819409595415 + - type: nauc_recall_at_1_std + value: -26.962624287912067 + - type: nauc_recall_at_20_diff1 + value: 9.54558093675037 + - type: nauc_recall_at_20_max + value: 7.408067716598006 + - type: nauc_recall_at_20_std + value: -3.497676642036962 + - type: nauc_recall_at_3_diff1 + value: 27.469730759073112 + - type: nauc_recall_at_3_max + value: 4.567448843154473 + - type: nauc_recall_at_3_std + value: -17.977373809321904 + - type: nauc_recall_at_5_diff1 + value: 19.20255358383672 + - type: nauc_recall_at_5_max + value: 8.28840114628813 + - type: nauc_recall_at_5_std + value: -11.478310315804768 + - type: ndcg_at_1 + value: 80.87299999999999 + - type: ndcg_at_10 + value: 86.53999999999999 + - type: ndcg_at_100 + value: 87.167 + - type: ndcg_at_1000 + value: 87.455 + - type: ndcg_at_20 + value: 86.834 + - type: ndcg_at_3 + value: 84.882 + - type: ndcg_at_5 + value: 85.884 + - type: precision_at_1 + value: 80.87299999999999 + - type: precision_at_10 + value: 10.225 + - type: precision_at_100 + value: 1.0699999999999998 + - type: precision_at_1000 + value: 0.11199999999999999 + - type: precision_at_20 + value: 5.1979999999999995 + - type: precision_at_3 + value: 32.123000000000005 + - type: precision_at_5 + value: 19.916 + - type: recall_at_1 + value: 74.97399999999999 + - type: recall_at_10 + value: 93.301 + - type: recall_at_100 + value: 95.779 + - type: recall_at_1000 + value: 97.596 + - type: recall_at_20 + value: 94.326 + - type: recall_at_3 + value: 88.73400000000001 + - type: recall_at_5 + value: 91.35 + task: + type: Retrieval + - dataset: + config: default + name: MTEB FiQA2018 + revision: 27a168819829fe9bcd655c2df245fb19452e8e06 + split: test + type: mteb/fiqa + metrics: + - type: main_score + value: 44.741 + - type: map_at_1 + value: 22.178 + - type: map_at_10 + value: 36.59 + - type: map_at_100 + value: 38.633 + - type: map_at_1000 + value: 38.814 + - type: map_at_20 + value: 37.756 + - type: map_at_3 + value: 32.036 + - type: map_at_5 + value: 34.549 + - type: mrr_at_1 + value: 43.67283950617284 + - type: mrr_at_10 + value: 53.02285420340973 + - type: mrr_at_100 + value: 53.80413476223648 + - type: mrr_at_1000 + value: 53.83329975208768 + - type: mrr_at_20 + value: 53.45062434754336 + - type: mrr_at_3 + value: 50.79732510288063 + - type: mrr_at_5 + value: 52.093621399176925 + - type: nauc_map_at_1000_diff1 + value: 38.13095193715016 + - type: nauc_map_at_1000_max + value: 25.140424128002902 + - type: nauc_map_at_1000_std + value: -2.291055526111446 + - type: nauc_map_at_100_diff1 + value: 38.062542374616925 + - type: nauc_map_at_100_max + value: 24.98212831003545 + - type: nauc_map_at_100_std + value: -2.3288513840196163 + - type: nauc_map_at_10_diff1 + value: 37.94005408477477 + - type: nauc_map_at_10_max + value: 23.62450054152409 + - type: nauc_map_at_10_std + value: -3.029451096370331 + - type: nauc_map_at_1_diff1 + value: 46.667422498970936 + - type: nauc_map_at_1_max + value: 14.863760158210667 + - type: nauc_map_at_1_std + value: -8.783505629854414 + - type: nauc_map_at_20_diff1 + value: 37.89072113058759 + - type: nauc_map_at_20_max + value: 24.366706507797332 + - type: nauc_map_at_20_std + value: -2.6023248317775605 + - type: nauc_map_at_3_diff1 + value: 39.47298192019393 + - type: nauc_map_at_3_max + value: 19.685477846714722 + - type: nauc_map_at_3_std + value: -5.027029063035539 + - type: nauc_map_at_5_diff1 + value: 39.0514789038465 + - type: nauc_map_at_5_max + value: 21.400362349268455 + - type: nauc_map_at_5_std + value: -4.630809029653372 + - type: nauc_mrr_at_1000_diff1 + value: 45.62207160402544 + - type: nauc_mrr_at_1000_max + value: 35.21626828049415 + - type: nauc_mrr_at_1000_std + value: -1.5589979016799316 + - type: nauc_mrr_at_100_diff1 + value: 45.6049911269738 + - type: nauc_mrr_at_100_max + value: 35.22606560450809 + - type: nauc_mrr_at_100_std + value: -1.5267050690413522 + - type: nauc_mrr_at_10_diff1 + value: 45.4093336826169 + - type: nauc_mrr_at_10_max + value: 35.13738468801265 + - type: nauc_mrr_at_10_std + value: -1.8919874430707715 + - type: nauc_mrr_at_1_diff1 + value: 51.6709580391481 + - type: nauc_mrr_at_1_max + value: 33.86625581485091 + - type: nauc_mrr_at_1_std + value: -5.723117301417357 + - type: nauc_mrr_at_20_diff1 + value: 45.51335104667583 + - type: nauc_mrr_at_20_max + value: 35.12487418629981 + - type: nauc_mrr_at_20_std + value: -1.5728658665283186 + - type: nauc_mrr_at_3_diff1 + value: 45.91684224347824 + - type: nauc_mrr_at_3_max + value: 34.71207649436817 + - type: nauc_mrr_at_3_std + value: -2.3652831443919373 + - type: nauc_mrr_at_5_diff1 + value: 45.24354433953711 + - type: nauc_mrr_at_5_max + value: 34.766746844223384 + - type: nauc_mrr_at_5_std + value: -2.414375686428433 + - type: nauc_ndcg_at_1000_diff1 + value: 38.99167200921022 + - type: nauc_ndcg_at_1000_max + value: 30.819387325830377 + - type: nauc_ndcg_at_1000_std + value: 1.0467714066842262 + - type: nauc_ndcg_at_100_diff1 + value: 38.248888114357484 + - type: nauc_ndcg_at_100_max + value: 29.58848183633258 + - type: nauc_ndcg_at_100_std + value: 1.417553970005679 + - type: nauc_ndcg_at_10_diff1 + value: 37.8683078247863 + - type: nauc_ndcg_at_10_max + value: 26.751938977033106 + - type: nauc_ndcg_at_10_std + value: -1.2409485838847412 + - type: nauc_ndcg_at_1_diff1 + value: 51.6709580391481 + - type: nauc_ndcg_at_1_max + value: 33.86625581485091 + - type: nauc_ndcg_at_1_std + value: -5.723117301417357 + - type: nauc_ndcg_at_20_diff1 + value: 37.8638946400599 + - type: nauc_ndcg_at_20_max + value: 27.606616644241157 + - type: nauc_ndcg_at_20_std + value: -0.03526704440544082 + - type: nauc_ndcg_at_3_diff1 + value: 39.51172857224684 + - type: nauc_ndcg_at_3_max + value: 29.20532249592107 + - type: nauc_ndcg_at_3_std + value: -1.8738383813061488 + - type: nauc_ndcg_at_5_diff1 + value: 38.377746472924514 + - type: nauc_ndcg_at_5_max + value: 26.329340638530454 + - type: nauc_ndcg_at_5_std + value: -2.918235844784902 + - type: nauc_precision_at_1000_diff1 + value: -0.5958931375447998 + - type: nauc_precision_at_1000_max + value: 36.15147294143512 + - type: nauc_precision_at_1000_std + value: 11.710171839031613 + - type: nauc_precision_at_100_diff1 + value: 3.1279602297522304 + - type: nauc_precision_at_100_max + value: 38.56180438744363 + - type: nauc_precision_at_100_std + value: 14.063724836230277 + - type: nauc_precision_at_10_diff1 + value: 12.659130460376236 + - type: nauc_precision_at_10_max + value: 39.37196817310586 + - type: nauc_precision_at_10_std + value: 10.085604614097598 + - type: nauc_precision_at_1_diff1 + value: 51.6709580391481 + - type: nauc_precision_at_1_max + value: 33.86625581485091 + - type: nauc_precision_at_1_std + value: -5.723117301417357 + - type: nauc_precision_at_20_diff1 + value: 9.650139629433262 + - type: nauc_precision_at_20_max + value: 38.77724238894805 + - type: nauc_precision_at_20_std + value: 11.958790785998644 + - type: nauc_precision_at_3_diff1 + value: 24.745989796740357 + - type: nauc_precision_at_3_max + value: 35.65068216285761 + - type: nauc_precision_at_3_std + value: 4.072896234633994 + - type: nauc_precision_at_5_diff1 + value: 18.655049044737424 + - type: nauc_precision_at_5_max + value: 35.998788688559216 + - type: nauc_precision_at_5_std + value: 5.391771098181724 + - type: nauc_recall_at_1000_diff1 + value: 12.949608157518922 + - type: nauc_recall_at_1000_max + value: 23.391273269349142 + - type: nauc_recall_at_1000_std + value: 28.42378542198053 + - type: nauc_recall_at_100_diff1 + value: 20.625416053309404 + - type: nauc_recall_at_100_max + value: 21.367842929796264 + - type: nauc_recall_at_100_std + value: 15.399702721384228 + - type: nauc_recall_at_10_diff1 + value: 27.211773223610237 + - type: nauc_recall_at_10_max + value: 18.536687796485346 + - type: nauc_recall_at_10_std + value: 0.8218390873018705 + - type: nauc_recall_at_1_diff1 + value: 46.667422498970936 + - type: nauc_recall_at_1_max + value: 14.863760158210667 + - type: nauc_recall_at_1_std + value: -8.783505629854414 + - type: nauc_recall_at_20_diff1 + value: 26.189517076492148 + - type: nauc_recall_at_20_max + value: 19.712210144922732 + - type: nauc_recall_at_20_std + value: 4.732025980603548 + - type: nauc_recall_at_3_diff1 + value: 32.03530659219441 + - type: nauc_recall_at_3_max + value: 15.729199503552953 + - type: nauc_recall_at_3_std + value: -2.78006696474447 + - type: nauc_recall_at_5_diff1 + value: 30.303038095198648 + - type: nauc_recall_at_5_max + value: 16.04989351651554 + - type: nauc_recall_at_5_std + value: -3.4801349592650674 + - type: ndcg_at_1 + value: 43.673 + - type: ndcg_at_10 + value: 44.741 + - type: ndcg_at_100 + value: 51.815999999999995 + - type: ndcg_at_1000 + value: 54.55799999999999 + - type: ndcg_at_20 + value: 47.575 + - type: ndcg_at_3 + value: 40.973 + - type: ndcg_at_5 + value: 42.174 + - type: precision_at_1 + value: 43.673 + - type: precision_at_10 + value: 12.299 + - type: precision_at_100 + value: 1.9709999999999999 + - type: precision_at_1000 + value: 0.246 + - type: precision_at_20 + value: 7.3999999999999995 + - type: precision_at_3 + value: 27.315 + - type: precision_at_5 + value: 20.061999999999998 + - type: recall_at_1 + value: 22.178 + - type: recall_at_10 + value: 51.902 + - type: recall_at_100 + value: 77.802 + - type: recall_at_1000 + value: 94.12 + - type: recall_at_20 + value: 60.496 + - type: recall_at_3 + value: 37.919000000000004 + - type: recall_at_5 + value: 43.829 + task: + type: Retrieval + - dataset: + config: default + name: MTEB HotpotQA + revision: ab518f4d6fcca38d87c25209f94beba119d02014 + split: test + type: mteb/hotpotqa + metrics: + - type: main_score + value: 67.585 + - type: map_at_1 + value: 38.596000000000004 + - type: map_at_10 + value: 58.378 + - type: map_at_100 + value: 59.282000000000004 + - type: map_at_1000 + value: 59.34199999999999 + - type: map_at_20 + value: 58.921 + - type: map_at_3 + value: 54.801 + - type: map_at_5 + value: 57.065999999999995 + - type: mrr_at_1 + value: 77.19108710330858 + - type: mrr_at_10 + value: 83.73176103662232 + - type: mrr_at_100 + value: 83.90019429669105 + - type: mrr_at_1000 + value: 83.90679291308962 + - type: mrr_at_20 + value: 83.8435177926089 + - type: mrr_at_3 + value: 82.81341435966664 + - type: mrr_at_5 + value: 83.4224623002472 + - type: nauc_map_at_1000_diff1 + value: 14.307964912952539 + - type: nauc_map_at_1000_max + value: 12.765468178017315 + - type: nauc_map_at_1000_std + value: 1.362058994472385 + - type: nauc_map_at_100_diff1 + value: 14.26768417100101 + - type: nauc_map_at_100_max + value: 12.750734297430844 + - type: nauc_map_at_100_std + value: 1.393555710353686 + - type: nauc_map_at_10_diff1 + value: 14.255931489441185 + - type: nauc_map_at_10_max + value: 12.523189563101312 + - type: nauc_map_at_10_std + value: 0.7161967418789583 + - type: nauc_map_at_1_diff1 + value: 69.87899286905372 + - type: nauc_map_at_1_max + value: 35.205475784602456 + - type: nauc_map_at_1_std + value: -10.863394163636846 + - type: nauc_map_at_20_diff1 + value: 14.212027716409054 + - type: nauc_map_at_20_max + value: 12.650489544649925 + - type: nauc_map_at_20_std + value: 1.1271711468045333 + - type: nauc_map_at_3_diff1 + value: 16.65137376183016 + - type: nauc_map_at_3_max + value: 13.00998218430069 + - type: nauc_map_at_3_std + value: -2.2381896578334963 + - type: nauc_map_at_5_diff1 + value: 14.766891306293303 + - type: nauc_map_at_5_max + value: 12.45533963465128 + - type: nauc_map_at_5_std + value: -0.4763780518957518 + - type: nauc_mrr_at_1000_diff1 + value: 68.55128371980359 + - type: nauc_mrr_at_1000_max + value: 36.47247958442408 + - type: nauc_mrr_at_1000_std + value: -9.311243277776393 + - type: nauc_mrr_at_100_diff1 + value: 68.5495192075432 + - type: nauc_mrr_at_100_max + value: 36.47800039396858 + - type: nauc_mrr_at_100_std + value: -9.30009477026889 + - type: nauc_mrr_at_10_diff1 + value: 68.5213780826157 + - type: nauc_mrr_at_10_max + value: 36.53470560276756 + - type: nauc_mrr_at_10_std + value: -9.276757799691362 + - type: nauc_mrr_at_1_diff1 + value: 69.87899286905372 + - type: nauc_mrr_at_1_max + value: 35.205475784602456 + - type: nauc_mrr_at_1_std + value: -10.863394163636846 + - type: nauc_mrr_at_20_diff1 + value: 68.51230880783687 + - type: nauc_mrr_at_20_max + value: 36.502918925898356 + - type: nauc_mrr_at_20_std + value: -9.252393600030487 + - type: nauc_mrr_at_3_diff1 + value: 68.41383168882547 + - type: nauc_mrr_at_3_max + value: 36.31820775404627 + - type: nauc_mrr_at_3_std + value: -9.999151252843035 + - type: nauc_mrr_at_5_diff1 + value: 68.45324347636361 + - type: nauc_mrr_at_5_max + value: 36.46139281601083 + - type: nauc_mrr_at_5_std + value: -9.588349787263777 + - type: nauc_ndcg_at_1000_diff1 + value: 19.255656584818208 + - type: nauc_ndcg_at_1000_max + value: 16.119708974764283 + - type: nauc_ndcg_at_1000_std + value: 3.6997591648265367 + - type: nauc_ndcg_at_100_diff1 + value: 18.173106007959618 + - type: nauc_ndcg_at_100_max + value: 15.568458367725462 + - type: nauc_ndcg_at_100_std + value: 4.536505893658295 + - type: nauc_ndcg_at_10_diff1 + value: 18.369881058142322 + - type: nauc_ndcg_at_10_max + value: 14.976758085283695 + - type: nauc_ndcg_at_10_std + value: 1.9666148808621997 + - type: nauc_ndcg_at_1_diff1 + value: 69.87899286905372 + - type: nauc_ndcg_at_1_max + value: 35.205475784602456 + - type: nauc_ndcg_at_1_std + value: -10.863394163636846 + - type: nauc_ndcg_at_20_diff1 + value: 17.99231888427834 + - type: nauc_ndcg_at_20_max + value: 15.195593488428424 + - type: nauc_ndcg_at_20_std + value: 3.210518329988178 + - type: nauc_ndcg_at_3_diff1 + value: 22.638085589869977 + - type: nauc_ndcg_at_3_max + value: 15.98027793904305 + - type: nauc_ndcg_at_3_std + value: -2.8619254611270484 + - type: nauc_ndcg_at_5_diff1 + value: 19.628126741608632 + - type: nauc_ndcg_at_5_max + value: 14.996362948646471 + - type: nauc_ndcg_at_5_std + value: -0.37278144328821516 + - type: nauc_precision_at_1000_diff1 + value: -18.870574565264082 + - type: nauc_precision_at_1000_max + value: 6.003585500284789 + - type: nauc_precision_at_1000_std + value: 28.22209155996002 + - type: nauc_precision_at_100_diff1 + value: -14.152417200336195 + - type: nauc_precision_at_100_max + value: 4.435292073503267 + - type: nauc_precision_at_100_std + value: 25.999649502498933 + - type: nauc_precision_at_10_diff1 + value: -4.136070746784591 + - type: nauc_precision_at_10_max + value: 5.875693813516143 + - type: nauc_precision_at_10_std + value: 10.909307135721624 + - type: nauc_precision_at_1_diff1 + value: 69.87899286905372 + - type: nauc_precision_at_1_max + value: 35.205475784602456 + - type: nauc_precision_at_1_std + value: -10.863394163636846 + - type: nauc_precision_at_20_diff1 + value: -7.786302363507167 + - type: nauc_precision_at_20_max + value: 5.45833862385675 + - type: nauc_precision_at_20_std + value: 15.783059950464354 + - type: nauc_precision_at_3_diff1 + value: 7.40890050470361 + - type: nauc_precision_at_3_max + value: 9.458068780681774 + - type: nauc_precision_at_3_std + value: 0.0027409382551998934 + - type: nauc_precision_at_5_diff1 + value: 0.6243863646902605 + - type: nauc_precision_at_5_max + value: 6.80746180406321 + - type: nauc_precision_at_5_std + value: 4.826367656810033 + - type: nauc_recall_at_1000_diff1 + value: -18.870574565263908 + - type: nauc_recall_at_1000_max + value: 6.003585500284974 + - type: nauc_recall_at_1000_std + value: 28.222091559960184 + - type: nauc_recall_at_100_diff1 + value: -14.15241720033625 + - type: nauc_recall_at_100_max + value: 4.435292073503326 + - type: nauc_recall_at_100_std + value: 25.999649502498993 + - type: nauc_recall_at_10_diff1 + value: -4.136070746784523 + - type: nauc_recall_at_10_max + value: 5.875693813516104 + - type: nauc_recall_at_10_std + value: 10.909307135721532 + - type: nauc_recall_at_1_diff1 + value: 69.87899286905372 + - type: nauc_recall_at_1_max + value: 35.205475784602456 + - type: nauc_recall_at_1_std + value: -10.863394163636846 + - type: nauc_recall_at_20_diff1 + value: -7.786302363507246 + - type: nauc_recall_at_20_max + value: 5.458338623856692 + - type: nauc_recall_at_20_std + value: 15.783059950464304 + - type: nauc_recall_at_3_diff1 + value: 7.408900504703652 + - type: nauc_recall_at_3_max + value: 9.458068780681788 + - type: nauc_recall_at_3_std + value: 0.002740938255258331 + - type: nauc_recall_at_5_diff1 + value: 0.6243863646902303 + - type: nauc_recall_at_5_max + value: 6.807461804063163 + - type: nauc_recall_at_5_std + value: 4.826367656810076 + - type: ndcg_at_1 + value: 77.191 + - type: ndcg_at_10 + value: 67.585 + - type: ndcg_at_100 + value: 70.652 + - type: ndcg_at_1000 + value: 71.82600000000001 + - type: ndcg_at_20 + value: 68.913 + - type: ndcg_at_3 + value: 62.61600000000001 + - type: ndcg_at_5 + value: 65.444 + - type: precision_at_1 + value: 77.191 + - type: precision_at_10 + value: 14.097000000000001 + - type: precision_at_100 + value: 1.6480000000000001 + - type: precision_at_1000 + value: 0.18 + - type: precision_at_20 + value: 7.475 + - type: precision_at_3 + value: 39.716 + - type: precision_at_5 + value: 26.061 + - type: recall_at_1 + value: 38.596000000000004 + - type: recall_at_10 + value: 70.486 + - type: recall_at_100 + value: 82.417 + - type: recall_at_1000 + value: 90.223 + - type: recall_at_20 + value: 74.754 + - type: recall_at_3 + value: 59.575 + - type: recall_at_5 + value: 65.152 + task: + type: Retrieval + - dataset: + config: default + name: MTEB ImdbClassification + revision: 3d86128a09e091d6018b6d26cad27f2739fc2db7 + split: test + type: mteb/imdb + metrics: + - type: accuracy + value: 94.93119999999999 + - type: ap + value: 92.92609880646465 + - type: ap_weighted + value: 92.92609880646465 + - type: f1 + value: 94.93020053324759 + - type: f1_weighted + value: 94.93020053324759 + - type: main_score + value: 94.93119999999999 + task: + type: Classification + - dataset: + config: default + name: MTEB MSMARCO + revision: c5a29a104738b98a9e76336939199e264163d4a0 + split: dev + type: mteb/msmarco + metrics: + - type: main_score + value: 34.593 + - type: map_at_1 + value: 15.776000000000002 + - type: map_at_10 + value: 27.542 + - type: map_at_100 + value: 28.893 + - type: map_at_1000 + value: 28.95 + - type: map_at_20 + value: 28.381 + - type: map_at_3 + value: 23.273 + - type: map_at_5 + value: 25.706 + - type: mrr_at_1 + value: 16.23209169054441 + - type: mrr_at_10 + value: 27.99337108291251 + - type: mrr_at_100 + value: 29.299712175246 + - type: mrr_at_1000 + value: 29.34980176303808 + - type: mrr_at_20 + value: 28.815617370683793 + - type: mrr_at_3 + value: 23.798949379178495 + - type: mrr_at_5 + value: 26.185052531041066 + - type: nauc_map_at_1000_diff1 + value: 29.078121539546963 + - type: nauc_map_at_1000_max + value: -4.660325486096363 + - type: nauc_map_at_1000_std + value: -16.890470067987078 + - type: nauc_map_at_100_diff1 + value: 29.076892998159536 + - type: nauc_map_at_100_max + value: -4.679633015133337 + - type: nauc_map_at_100_std + value: -16.862781614983454 + - type: nauc_map_at_10_diff1 + value: 29.13003469335662 + - type: nauc_map_at_10_max + value: -4.88793866124687 + - type: nauc_map_at_10_std + value: -17.63030762796993 + - type: nauc_map_at_1_diff1 + value: 31.692294187492053 + - type: nauc_map_at_1_max + value: -3.4670833327733073 + - type: nauc_map_at_1_std + value: -15.691352600942812 + - type: nauc_map_at_20_diff1 + value: 29.055576137474308 + - type: nauc_map_at_20_max + value: -4.7776538554194286 + - type: nauc_map_at_20_std + value: -17.224848225253236 + - type: nauc_map_at_3_diff1 + value: 28.9101303524098 + - type: nauc_map_at_3_max + value: -4.652960479072655 + - type: nauc_map_at_3_std + value: -17.418465326700314 + - type: nauc_map_at_5_diff1 + value: 28.843601736020492 + - type: nauc_map_at_5_max + value: -4.674223389697514 + - type: nauc_map_at_5_std + value: -17.97502181985735 + - type: nauc_mrr_at_1000_diff1 + value: 28.898068429091573 + - type: nauc_mrr_at_1000_max + value: -4.399664885905157 + - type: nauc_mrr_at_1000_std + value: -16.607218168281097 + - type: nauc_mrr_at_100_diff1 + value: 28.89629415174332 + - type: nauc_mrr_at_100_max + value: -4.409492230329251 + - type: nauc_mrr_at_100_std + value: -16.57762186807223 + - type: nauc_mrr_at_10_diff1 + value: 28.941113994846653 + - type: nauc_mrr_at_10_max + value: -4.583270921209106 + - type: nauc_mrr_at_10_std + value: -17.289894093299672 + - type: nauc_mrr_at_1_diff1 + value: 31.37083827832811 + - type: nauc_mrr_at_1_max + value: -3.208411493959455 + - type: nauc_mrr_at_1_std + value: -15.40345578596366 + - type: nauc_mrr_at_20_diff1 + value: 28.8794180315418 + - type: nauc_mrr_at_20_max + value: -4.471345918444438 + - type: nauc_mrr_at_20_std + value: -16.889422348794938 + - type: nauc_mrr_at_3_diff1 + value: 28.648544038786806 + - type: nauc_mrr_at_3_max + value: -4.446235699608574 + - type: nauc_mrr_at_3_std + value: -17.223275526865695 + - type: nauc_mrr_at_5_diff1 + value: 28.676176998873526 + - type: nauc_mrr_at_5_max + value: -4.384629457636603 + - type: nauc_mrr_at_5_std + value: -17.662360967653942 + - type: nauc_ndcg_at_1000_diff1 + value: 28.512776616817614 + - type: nauc_ndcg_at_1000_max + value: -4.032666359115543 + - type: nauc_ndcg_at_1000_std + value: -14.96235335409355 + - type: nauc_ndcg_at_100_diff1 + value: 28.46827238630747 + - type: nauc_ndcg_at_100_max + value: -4.297081420480486 + - type: nauc_ndcg_at_100_std + value: -13.652822791502834 + - type: nauc_ndcg_at_10_diff1 + value: 28.731734677632335 + - type: nauc_ndcg_at_10_max + value: -5.307476808688383 + - type: nauc_ndcg_at_10_std + value: -17.847598875907043 + - type: nauc_ndcg_at_1_diff1 + value: 31.37083827832811 + - type: nauc_ndcg_at_1_max + value: -3.208411493959455 + - type: nauc_ndcg_at_1_std + value: -15.40345578596366 + - type: nauc_ndcg_at_20_diff1 + value: 28.454343686246368 + - type: nauc_ndcg_at_20_max + value: -4.98727458049532 + - type: nauc_ndcg_at_20_std + value: -16.354151805926072 + - type: nauc_ndcg_at_3_diff1 + value: 28.17849373973975 + - type: nauc_ndcg_at_3_max + value: -4.8778622367007705 + - type: nauc_ndcg_at_3_std + value: -17.863750930775883 + - type: nauc_ndcg_at_5_diff1 + value: 28.150883678367972 + - type: nauc_ndcg_at_5_max + value: -4.855498175237134 + - type: nauc_ndcg_at_5_std + value: -18.690049098501387 + - type: nauc_precision_at_1000_diff1 + value: -2.8337934746656184 + - type: nauc_precision_at_1000_max + value: 18.96310957410793 + - type: nauc_precision_at_1000_std + value: 14.930142344173392 + - type: nauc_precision_at_100_diff1 + value: 14.333530836109675 + - type: nauc_precision_at_100_max + value: 6.295533804672035 + - type: nauc_precision_at_100_std + value: 18.019478047265473 + - type: nauc_precision_at_10_diff1 + value: 26.15274520345387 + - type: nauc_precision_at_10_max + value: -5.684160746990313 + - type: nauc_precision_at_10_std + value: -17.184523300085225 + - type: nauc_precision_at_1_diff1 + value: 31.37083827832811 + - type: nauc_precision_at_1_max + value: -3.208411493959455 + - type: nauc_precision_at_1_std + value: -15.40345578596366 + - type: nauc_precision_at_20_diff1 + value: 23.390317543956684 + - type: nauc_precision_at_20_max + value: -3.643101669692548 + - type: nauc_precision_at_20_std + value: -10.138163569909773 + - type: nauc_precision_at_3_diff1 + value: 26.147860052902338 + - type: nauc_precision_at_3_max + value: -5.348523907965159 + - type: nauc_precision_at_3_std + value: -18.984802706187416 + - type: nauc_precision_at_5_diff1 + value: 25.59347977608168 + - type: nauc_precision_at_5_max + value: -4.966621480640159 + - type: nauc_precision_at_5_std + value: -20.197439172622257 + - type: nauc_recall_at_1000_diff1 + value: 9.353127475581333 + - type: nauc_recall_at_1000_max + value: 35.43082913054732 + - type: nauc_recall_at_1000_std + value: 66.50162334521372 + - type: nauc_recall_at_100_diff1 + value: 24.125268256131005 + - type: nauc_recall_at_100_max + value: -0.25871115365966996 + - type: nauc_recall_at_100_std + value: 22.002110879773436 + - type: nauc_recall_at_10_diff1 + value: 27.79420827880311 + - type: nauc_recall_at_10_max + value: -6.774583137315747 + - type: nauc_recall_at_10_std + value: -18.300025227607936 + - type: nauc_recall_at_1_diff1 + value: 31.692294187492053 + - type: nauc_recall_at_1_max + value: -3.4670833327733073 + - type: nauc_recall_at_1_std + value: -15.691352600942812 + - type: nauc_recall_at_20_diff1 + value: 26.47331981012526 + - type: nauc_recall_at_20_max + value: -6.053234914161821 + - type: nauc_recall_at_20_std + value: -12.210795863137527 + - type: nauc_recall_at_3_diff1 + value: 26.396199364799276 + - type: nauc_recall_at_3_max + value: -5.5102184367103675 + - type: nauc_recall_at_3_std + value: -18.893379374309998 + - type: nauc_recall_at_5_diff1 + value: 26.415672184819467 + - type: nauc_recall_at_5_max + value: -5.452147280159272 + - type: nauc_recall_at_5_std + value: -20.488315060196 + - type: ndcg_at_1 + value: 16.232 + - type: ndcg_at_10 + value: 34.593 + - type: ndcg_at_100 + value: 41.193999999999996 + - type: ndcg_at_1000 + value: 42.594 + - type: ndcg_at_20 + value: 37.586999999999996 + - type: ndcg_at_3 + value: 25.921 + - type: ndcg_at_5 + value: 30.244 + - type: precision_at_1 + value: 16.232 + - type: precision_at_10 + value: 5.924 + - type: precision_at_100 + value: 0.922 + - type: precision_at_1000 + value: 0.104 + - type: precision_at_20 + value: 3.5770000000000004 + - type: precision_at_3 + value: 11.404 + - type: precision_at_5 + value: 9.052 + - type: recall_at_1 + value: 15.776000000000002 + - type: recall_at_10 + value: 56.581 + - type: recall_at_100 + value: 87.32 + - type: recall_at_1000 + value: 97.98400000000001 + - type: recall_at_20 + value: 68.22500000000001 + - type: recall_at_3 + value: 33.079 + - type: recall_at_5 + value: 43.418 + task: + type: Retrieval + - dataset: + config: en + name: MTEB MTOPDomainClassification (en) + revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf + split: test + type: mteb/mtop_domain + metrics: + - type: accuracy + value: 97.53989968080255 + - type: f1 + value: 97.40324881796998 + - type: f1_weighted + value: 97.55321112075454 + - type: main_score + value: 97.53989968080255 + task: + type: Classification + - dataset: + config: en + name: MTEB MTOPIntentClassification (en) + revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba + split: test + type: mteb/mtop_intent + metrics: + - type: accuracy + value: 85.75923392612859 + - type: f1 + value: 61.5307155386522 + - type: f1_weighted + value: 85.67681172442828 + - type: main_score + value: 85.75923392612859 + task: + type: Classification + - dataset: + config: en + name: MTEB MassiveIntentClassification (en) + revision: 4672e20407010da34463acc759c162ca9734bca6 + split: test + type: mteb/amazon_massive_intent + metrics: + - type: accuracy + value: 72.51513113651647 + - type: f1 + value: 69.34853147184798 + - type: f1_weighted + value: 71.22394439174492 + - type: main_score + value: 72.51513113651647 + task: + type: Classification + - dataset: + config: en + name: MTEB MassiveScenarioClassification (en) + revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 + split: test + type: mteb/amazon_massive_scenario + metrics: + - type: accuracy + value: 79.32414256893072 + - type: f1 + value: 78.7067321614858 + - type: f1_weighted + value: 79.17371887085407 + - type: main_score + value: 79.32414256893072 + task: + type: Classification + - dataset: + config: default + name: MTEB MedrxivClusteringP2P + revision: e7a26af6f3ae46b30dde8737f02c07b1505bcc73 + split: test + type: mteb/medrxiv-clustering-p2p + metrics: + - type: main_score + value: 39.9112733005734 + - type: v_measure + value: 39.9112733005734 + - type: v_measure_std + value: 1.3729330580974082 + task: + type: Clustering + - dataset: + config: default + name: MTEB MedrxivClusteringS2S + revision: 35191c8c0dca72d8ff3efcd72aa802307d469663 + split: test + type: mteb/medrxiv-clustering-s2s + metrics: + - type: main_score + value: 36.793147891812595 + - type: v_measure + value: 36.793147891812595 + - type: v_measure_std + value: 1.373356254555773 + task: + type: Clustering + - dataset: + config: default + name: MTEB MindSmallReranking + revision: 59042f120c80e8afa9cdbb224f67076cec0fc9a7 + split: test + type: mteb/mind_small + metrics: + - type: main_score + value: 31.92214547032066 + - type: map + value: 31.92214547032066 + - type: mrr + value: 33.14385477235431 + - type: nAUC_map_diff1 + value: 11.986123616015425 + - type: nAUC_map_max + value: -20.27412119718789 + - type: nAUC_map_std + value: 1.1581550862810461 + - type: nAUC_mrr_diff1 + value: 11.165093384672165 + - type: nAUC_mrr_max + value: -14.737175633093665 + - type: nAUC_mrr_std + value: 2.5082843023127364 + task: + type: Reranking + - dataset: + config: default + name: MTEB NFCorpus + revision: ec0fa4fe99da2ff19ca1214b7966684033a58814 + split: test + type: mteb/nfcorpus + metrics: + - type: main_score + value: 35.326 + - type: map_at_1 + value: 5.756 + - type: map_at_10 + value: 13.23 + - type: map_at_100 + value: 16.821 + - type: map_at_1000 + value: 18.285 + - type: map_at_20 + value: 14.709 + - type: map_at_3 + value: 9.86 + - type: map_at_5 + value: 11.41 + - type: mrr_at_1 + value: 45.20123839009288 + - type: mrr_at_10 + value: 54.5034154012482 + - type: mrr_at_100 + value: 55.11212028144428 + - type: mrr_at_1000 + value: 55.14869280667608 + - type: mrr_at_20 + value: 54.852116950454885 + - type: mrr_at_3 + value: 52.32198142414861 + - type: mrr_at_5 + value: 53.69969040247676 + - type: nauc_map_at_1000_diff1 + value: 24.103237694692552 + - type: nauc_map_at_1000_max + value: 28.238145605874905 + - type: nauc_map_at_1000_std + value: 14.821335297504906 + - type: nauc_map_at_100_diff1 + value: 25.692696988595603 + - type: nauc_map_at_100_max + value: 27.282927002710704 + - type: nauc_map_at_100_std + value: 11.428779187303642 + - type: nauc_map_at_10_diff1 + value: 30.802161942248162 + - type: nauc_map_at_10_max + value: 21.533637232032028 + - type: nauc_map_at_10_std + value: -1.4834314993650302 + - type: nauc_map_at_1_diff1 + value: 44.692178401496655 + - type: nauc_map_at_1_max + value: 7.8258037280355826 + - type: nauc_map_at_1_std + value: -14.548783945519206 + - type: nauc_map_at_20_diff1 + value: 28.162218272962896 + - type: nauc_map_at_20_max + value: 24.526508434124295 + - type: nauc_map_at_20_std + value: 4.5923029452531114 + - type: nauc_map_at_3_diff1 + value: 36.159346924502394 + - type: nauc_map_at_3_max + value: 13.037438946030472 + - type: nauc_map_at_3_std + value: -10.65662898188665 + - type: nauc_map_at_5_diff1 + value: 34.65868875018595 + - type: nauc_map_at_5_max + value: 17.60272401270236 + - type: nauc_map_at_5_std + value: -7.249752394842571 + - type: nauc_mrr_at_1000_diff1 + value: 30.21772044474309 + - type: nauc_mrr_at_1000_max + value: 42.78186255622255 + - type: nauc_mrr_at_1000_std + value: 25.797577782337537 + - type: nauc_mrr_at_100_diff1 + value: 30.212889955928773 + - type: nauc_mrr_at_100_max + value: 42.81922498816082 + - type: nauc_mrr_at_100_std + value: 25.85789713650141 + - type: nauc_mrr_at_10_diff1 + value: 30.44797936946012 + - type: nauc_mrr_at_10_max + value: 42.53323068058471 + - type: nauc_mrr_at_10_std + value: 25.66456895797239 + - type: nauc_mrr_at_1_diff1 + value: 32.34440752392725 + - type: nauc_mrr_at_1_max + value: 36.9469056580779 + - type: nauc_mrr_at_1_std + value: 18.657611594947873 + - type: nauc_mrr_at_20_diff1 + value: 30.176345122251274 + - type: nauc_mrr_at_20_max + value: 42.75434764186405 + - type: nauc_mrr_at_20_std + value: 25.79589315785969 + - type: nauc_mrr_at_3_diff1 + value: 29.302461225569886 + - type: nauc_mrr_at_3_max + value: 41.82223481926128 + - type: nauc_mrr_at_3_std + value: 24.02550946327186 + - type: nauc_mrr_at_5_diff1 + value: 30.47192933785718 + - type: nauc_mrr_at_5_max + value: 42.36007501528813 + - type: nauc_mrr_at_5_std + value: 25.0758442990031 + - type: nauc_ndcg_at_1000_diff1 + value: 23.68574089704954 + - type: nauc_ndcg_at_1000_max + value: 44.188191803981134 + - type: nauc_ndcg_at_1000_std + value: 32.59339851811357 + - type: nauc_ndcg_at_100_diff1 + value: 22.31584896554311 + - type: nauc_ndcg_at_100_max + value: 37.059955696641985 + - type: nauc_ndcg_at_100_std + value: 27.00502292427943 + - type: nauc_ndcg_at_10_diff1 + value: 20.60773946800146 + - type: nauc_ndcg_at_10_max + value: 37.61141982157199 + - type: nauc_ndcg_at_10_std + value: 26.410973914430798 + - type: nauc_ndcg_at_1_diff1 + value: 34.18370363324457 + - type: nauc_ndcg_at_1_max + value: 36.36015956329315 + - type: nauc_ndcg_at_1_std + value: 18.45897389269808 + - type: nauc_ndcg_at_20_diff1 + value: 20.24173606023381 + - type: nauc_ndcg_at_20_max + value: 37.415776645800705 + - type: nauc_ndcg_at_20_std + value: 28.21373791467815 + - type: nauc_ndcg_at_3_diff1 + value: 23.315676671193806 + - type: nauc_ndcg_at_3_max + value: 38.7142575726556 + - type: nauc_ndcg_at_3_std + value: 23.1580283666415 + - type: nauc_ndcg_at_5_diff1 + value: 22.245075769905913 + - type: nauc_ndcg_at_5_max + value: 39.635321337970396 + - type: nauc_ndcg_at_5_std + value: 25.453576926797428 + - type: nauc_precision_at_1000_diff1 + value: -13.699777209784223 + - type: nauc_precision_at_1000_max + value: 10.284718353463134 + - type: nauc_precision_at_1000_std + value: 35.67360319480793 + - type: nauc_precision_at_100_diff1 + value: -12.08500797758176 + - type: nauc_precision_at_100_max + value: 20.44925184924704 + - type: nauc_precision_at_100_std + value: 45.5895403196849 + - type: nauc_precision_at_10_diff1 + value: 0.06557427029772205 + - type: nauc_precision_at_10_max + value: 37.55343130127243 + - type: nauc_precision_at_10_std + value: 39.8774623173031 + - type: nauc_precision_at_1_diff1 + value: 32.34440752392725 + - type: nauc_precision_at_1_max + value: 36.9469056580779 + - type: nauc_precision_at_1_std + value: 18.657611594947873 + - type: nauc_precision_at_20_diff1 + value: -5.424800947711766 + - type: nauc_precision_at_20_max + value: 34.400450482662606 + - type: nauc_precision_at_20_std + value: 46.96466506447469 + - type: nauc_precision_at_3_diff1 + value: 11.737310947259319 + - type: nauc_precision_at_3_max + value: 40.38852481780515 + - type: nauc_precision_at_3_std + value: 29.037703635253433 + - type: nauc_precision_at_5_diff1 + value: 6.832213390107087 + - type: nauc_precision_at_5_max + value: 41.3861915622186 + - type: nauc_precision_at_5_std + value: 34.60777485026518 + - type: nauc_recall_at_1000_diff1 + value: 8.252648642934155 + - type: nauc_recall_at_1000_max + value: 21.9593982158523 + - type: nauc_recall_at_1000_std + value: 19.351920551082532 + - type: nauc_recall_at_100_diff1 + value: 14.367515824195436 + - type: nauc_recall_at_100_max + value: 20.91547758690166 + - type: nauc_recall_at_100_std + value: 15.205925255166164 + - type: nauc_recall_at_10_diff1 + value: 28.110881426056622 + - type: nauc_recall_at_10_max + value: 20.127999606996973 + - type: nauc_recall_at_10_std + value: -1.4778912107941327 + - type: nauc_recall_at_1_diff1 + value: 44.692178401496655 + - type: nauc_recall_at_1_max + value: 7.8258037280355826 + - type: nauc_recall_at_1_std + value: -14.548783945519206 + - type: nauc_recall_at_20_diff1 + value: 22.894768482358447 + - type: nauc_recall_at_20_max + value: 23.53906838805954 + - type: nauc_recall_at_20_std + value: 6.968725573206477 + - type: nauc_recall_at_3_diff1 + value: 33.912996119453844 + - type: nauc_recall_at_3_max + value: 12.026372824736068 + - type: nauc_recall_at_3_std + value: -10.339483350646704 + - type: nauc_recall_at_5_diff1 + value: 33.302263355698805 + - type: nauc_recall_at_5_max + value: 17.306357074559127 + - type: nauc_recall_at_5_std + value: -6.907681462221045 + - type: ndcg_at_1 + value: 43.498 + - type: ndcg_at_10 + value: 35.326 + - type: ndcg_at_100 + value: 32.618 + - type: ndcg_at_1000 + value: 41.127 + - type: ndcg_at_20 + value: 33.018 + - type: ndcg_at_3 + value: 40.858 + - type: ndcg_at_5 + value: 38.888 + - type: precision_at_1 + value: 45.201 + - type: precision_at_10 + value: 26.192 + - type: precision_at_100 + value: 8.455 + - type: precision_at_1000 + value: 2.114 + - type: precision_at_20 + value: 19.582 + - type: precision_at_3 + value: 38.906 + - type: precision_at_5 + value: 34.489 + - type: recall_at_1 + value: 5.756 + - type: recall_at_10 + value: 17.101 + - type: recall_at_100 + value: 33.768 + - type: recall_at_1000 + value: 64.453 + - type: recall_at_20 + value: 21.001 + - type: recall_at_3 + value: 11.011 + - type: recall_at_5 + value: 13.344000000000001 + task: + type: Retrieval + - dataset: + config: default + name: MTEB NQ + revision: b774495ed302d8c44a3a7ea25c90dbce03968f31 + split: test + type: mteb/nq + metrics: + - type: main_score + value: 47.496 + - type: map_at_1 + value: 24.751 + - type: map_at_10 + value: 39.348 + - type: map_at_100 + value: 40.636 + - type: map_at_1000 + value: 40.671 + - type: map_at_20 + value: 40.211999999999996 + - type: map_at_3 + value: 34.575 + - type: map_at_5 + value: 37.22 + - type: mrr_at_1 + value: 27.95480880648899 + - type: mrr_at_10 + value: 41.9044860122496 + - type: mrr_at_100 + value: 42.876980645677705 + - type: mrr_at_1000 + value: 42.900385254956284 + - type: mrr_at_20 + value: 42.55982972518494 + - type: mrr_at_3 + value: 37.751062186172234 + - type: mrr_at_5 + value: 40.077249903437675 + - type: nauc_map_at_1000_diff1 + value: 32.86038948653992 + - type: nauc_map_at_1000_max + value: 13.50985063916989 + - type: nauc_map_at_1000_std + value: -5.820123220528598 + - type: nauc_map_at_100_diff1 + value: 32.84976636584168 + - type: nauc_map_at_100_max + value: 13.53587518434608 + - type: nauc_map_at_100_std + value: -5.785833146376201 + - type: nauc_map_at_10_diff1 + value: 32.85359459153253 + - type: nauc_map_at_10_max + value: 13.26285007121336 + - type: nauc_map_at_10_std + value: -6.394996664397717 + - type: nauc_map_at_1_diff1 + value: 34.23873886692038 + - type: nauc_map_at_1_max + value: 9.192506212634111 + - type: nauc_map_at_1_std + value: -7.111967933900576 + - type: nauc_map_at_20_diff1 + value: 32.91834546416986 + - type: nauc_map_at_20_max + value: 13.50516841923876 + - type: nauc_map_at_20_std + value: -6.014704885883195 + - type: nauc_map_at_3_diff1 + value: 32.80791993872603 + - type: nauc_map_at_3_max + value: 11.865588752529158 + - type: nauc_map_at_3_std + value: -7.46123492208037 + - type: nauc_map_at_5_diff1 + value: 32.87720923080645 + - type: nauc_map_at_5_max + value: 12.571139947934668 + - type: nauc_map_at_5_std + value: -7.1390323037613 + - type: nauc_mrr_at_1000_diff1 + value: 32.6746011819675 + - type: nauc_mrr_at_1000_max + value: 13.853989466043798 + - type: nauc_mrr_at_1000_std + value: -4.306780508073467 + - type: nauc_mrr_at_100_diff1 + value: 32.66974348220786 + - type: nauc_mrr_at_100_max + value: 13.87521280963595 + - type: nauc_mrr_at_100_std + value: -4.281623268758372 + - type: nauc_mrr_at_10_diff1 + value: 32.61464733720552 + - type: nauc_mrr_at_10_max + value: 13.832776492439633 + - type: nauc_mrr_at_10_std + value: -4.542617807886527 + - type: nauc_mrr_at_1_diff1 + value: 34.72533798771275 + - type: nauc_mrr_at_1_max + value: 10.861316429794243 + - type: nauc_mrr_at_1_std + value: -5.254029707816079 + - type: nauc_mrr_at_20_diff1 + value: 32.677004932710894 + - type: nauc_mrr_at_20_max + value: 13.89066399124665 + - type: nauc_mrr_at_20_std + value: -4.394511722991158 + - type: nauc_mrr_at_3_diff1 + value: 32.55790026673851 + - type: nauc_mrr_at_3_max + value: 12.873874337856527 + - type: nauc_mrr_at_3_std + value: -5.3827073588128265 + - type: nauc_mrr_at_5_diff1 + value: 32.528791050435544 + - type: nauc_mrr_at_5_max + value: 13.181648760273701 + - type: nauc_mrr_at_5_std + value: -5.058058338649079 + - type: nauc_ndcg_at_1000_diff1 + value: 32.402797222670785 + - type: nauc_ndcg_at_1000_max + value: 15.33827444009085 + - type: nauc_ndcg_at_1000_std + value: -3.656288878646395 + - type: nauc_ndcg_at_100_diff1 + value: 32.16580228877547 + - type: nauc_ndcg_at_100_max + value: 16.110100314396796 + - type: nauc_ndcg_at_100_std + value: -2.617364679505708 + - type: nauc_ndcg_at_10_diff1 + value: 32.20757651198573 + - type: nauc_ndcg_at_10_max + value: 15.196608889204091 + - type: nauc_ndcg_at_10_std + value: -5.136073100942583 + - type: nauc_ndcg_at_1_diff1 + value: 34.82910472793821 + - type: nauc_ndcg_at_1_max + value: 10.929310445823809 + - type: nauc_ndcg_at_1_std + value: -5.169808829987181 + - type: nauc_ndcg_at_20_diff1 + value: 32.43129783501163 + - type: nauc_ndcg_at_20_max + value: 15.93657158518566 + - type: nauc_ndcg_at_20_std + value: -3.976273055504378 + - type: nauc_ndcg_at_3_diff1 + value: 32.299740007616464 + - type: nauc_ndcg_at_3_max + value: 12.584778951510017 + - type: nauc_ndcg_at_3_std + value: -7.303956420107395 + - type: nauc_ndcg_at_5_diff1 + value: 32.25306359394614 + - type: nauc_ndcg_at_5_max + value: 13.568498593362552 + - type: nauc_ndcg_at_5_std + value: -6.762105103486535 + - type: nauc_precision_at_1000_diff1 + value: -4.0288327326991515 + - type: nauc_precision_at_1000_max + value: 10.55924632168659 + - type: nauc_precision_at_1000_std + value: 14.694651019403377 + - type: nauc_precision_at_100_diff1 + value: 0.707615597626847 + - type: nauc_precision_at_100_max + value: 18.847648466075505 + - type: nauc_precision_at_100_std + value: 21.218326061258423 + - type: nauc_precision_at_10_diff1 + value: 19.33907100470173 + - type: nauc_precision_at_10_max + value: 20.278902993078106 + - type: nauc_precision_at_10_std + value: 4.615875975323101 + - type: nauc_precision_at_1_diff1 + value: 34.82910472793821 + - type: nauc_precision_at_1_max + value: 10.929310445823809 + - type: nauc_precision_at_1_std + value: -5.169808829987181 + - type: nauc_precision_at_20_diff1 + value: 13.684570079407019 + - type: nauc_precision_at_20_max + value: 21.004712453781277 + - type: nauc_precision_at_20_std + value: 10.691183107158766 + - type: nauc_precision_at_3_diff1 + value: 28.87415756089232 + - type: nauc_precision_at_3_max + value: 15.9587615402185 + - type: nauc_precision_at_3_std + value: -4.72662256123726 + - type: nauc_precision_at_5_diff1 + value: 25.253692703184903 + - type: nauc_precision_at_5_max + value: 17.20245900234252 + - type: nauc_precision_at_5_std + value: -2.0902250214667126 + - type: nauc_recall_at_1000_diff1 + value: 29.045808029763105 + - type: nauc_recall_at_1000_max + value: 72.0085421054025 + - type: nauc_recall_at_1000_std + value: 66.674461537018 + - type: nauc_recall_at_100_diff1 + value: 23.524016343264613 + - type: nauc_recall_at_100_max + value: 48.87885727542797 + - type: nauc_recall_at_100_std + value: 41.676931807412714 + - type: nauc_recall_at_10_diff1 + value: 28.448706131448077 + - type: nauc_recall_at_10_max + value: 20.60216328000016 + - type: nauc_recall_at_10_std + value: -2.5499050504498073 + - type: nauc_recall_at_1_diff1 + value: 34.23873886692038 + - type: nauc_recall_at_1_max + value: 9.192506212634111 + - type: nauc_recall_at_1_std + value: -7.111967933900576 + - type: nauc_recall_at_20_diff1 + value: 29.250434345941436 + - type: nauc_recall_at_20_max + value: 27.07586767599477 + - type: nauc_recall_at_20_std + value: 4.674120256113827 + - type: nauc_recall_at_3_diff1 + value: 29.954539543517395 + - type: nauc_recall_at_3_max + value: 12.891951328267334 + - type: nauc_recall_at_3_std + value: -8.58992556784897 + - type: nauc_recall_at_5_diff1 + value: 29.409761377456885 + - type: nauc_recall_at_5_max + value: 15.120264304508765 + - type: nauc_recall_at_5_std + value: -7.629729207075528 + - type: ndcg_at_1 + value: 27.926000000000002 + - type: ndcg_at_10 + value: 47.496 + - type: ndcg_at_100 + value: 52.886 + - type: ndcg_at_1000 + value: 53.632000000000005 + - type: ndcg_at_20 + value: 50.285000000000004 + - type: ndcg_at_3 + value: 38.291 + - type: ndcg_at_5 + value: 42.764 + - type: precision_at_1 + value: 27.926000000000002 + - type: precision_at_10 + value: 8.305 + - type: precision_at_100 + value: 1.131 + - type: precision_at_1000 + value: 0.12 + - type: precision_at_20 + value: 4.8149999999999995 + - type: precision_at_3 + value: 17.729 + - type: precision_at_5 + value: 13.209999999999999 + - type: recall_at_1 + value: 24.751 + - type: recall_at_10 + value: 69.99600000000001 + - type: recall_at_100 + value: 93.265 + - type: recall_at_1000 + value: 98.69200000000001 + - type: recall_at_20 + value: 80.345 + - type: recall_at_3 + value: 45.899 + - type: recall_at_5 + value: 56.233 + task: + type: Retrieval + - dataset: + config: default + name: MTEB QuoraRetrieval + revision: e4e08e0b7dbe3c8700f0daef558ff32256715259 + split: test + type: mteb/quora + metrics: + - type: main_score + value: 87.469 + - type: map_at_1 + value: 69.226 + - type: map_at_10 + value: 83.49499999999999 + - type: map_at_100 + value: 84.16 + - type: map_at_1000 + value: 84.175 + - type: map_at_20 + value: 83.937 + - type: map_at_3 + value: 80.357 + - type: map_at_5 + value: 82.354 + - type: mrr_at_1 + value: 79.66 + - type: mrr_at_10 + value: 86.41408730158712 + - type: mrr_at_100 + value: 86.53009949625319 + - type: mrr_at_1000 + value: 86.53085807986054 + - type: mrr_at_20 + value: 86.50681664426882 + - type: mrr_at_3 + value: 85.2949999999998 + - type: mrr_at_5 + value: 86.09149999999971 + - type: nauc_map_at_1000_diff1 + value: 76.34480513462263 + - type: nauc_map_at_1000_max + value: 18.35381451054758 + - type: nauc_map_at_1000_std + value: -45.66385767188469 + - type: nauc_map_at_100_diff1 + value: 76.34669058530609 + - type: nauc_map_at_100_max + value: 18.31665217615866 + - type: nauc_map_at_100_std + value: -45.71646027130385 + - type: nauc_map_at_10_diff1 + value: 76.55810725489853 + - type: nauc_map_at_10_max + value: 17.608690417604674 + - type: nauc_map_at_10_std + value: -47.80834561676489 + - type: nauc_map_at_1_diff1 + value: 79.82735407515446 + - type: nauc_map_at_1_max + value: 12.275645564446238 + - type: nauc_map_at_1_std + value: -41.53738015617803 + - type: nauc_map_at_20_diff1 + value: 76.39897445057328 + - type: nauc_map_at_20_max + value: 18.073183264222344 + - type: nauc_map_at_20_std + value: -46.53343804460259 + - type: nauc_map_at_3_diff1 + value: 76.8525339580126 + - type: nauc_map_at_3_max + value: 15.463017873342727 + - type: nauc_map_at_3_std + value: -49.30092150838741 + - type: nauc_map_at_5_diff1 + value: 76.84854418575374 + - type: nauc_map_at_5_max + value: 16.93909969110639 + - type: nauc_map_at_5_std + value: -49.26303643788198 + - type: nauc_mrr_at_1000_diff1 + value: 76.11978747472594 + - type: nauc_mrr_at_1000_max + value: 20.948054256043136 + - type: nauc_mrr_at_1000_std + value: -40.538438735309654 + - type: nauc_mrr_at_100_diff1 + value: 76.1194337607922 + - type: nauc_mrr_at_100_max + value: 20.947907822336376 + - type: nauc_mrr_at_100_std + value: -40.53901348313209 + - type: nauc_mrr_at_10_diff1 + value: 76.10022175253289 + - type: nauc_mrr_at_10_max + value: 20.93817071185543 + - type: nauc_mrr_at_10_std + value: -40.69322088799321 + - type: nauc_mrr_at_1_diff1 + value: 77.05621804280337 + - type: nauc_mrr_at_1_max + value: 20.602559090153918 + - type: nauc_mrr_at_1_std + value: -38.09297840209946 + - type: nauc_mrr_at_20_diff1 + value: 76.11382894317693 + - type: nauc_mrr_at_20_max + value: 20.960340980416362 + - type: nauc_mrr_at_20_std + value: -40.538013300953075 + - type: nauc_mrr_at_3_diff1 + value: 75.71270232724483 + - type: nauc_mrr_at_3_max + value: 20.823905759297613 + - type: nauc_mrr_at_3_std + value: -41.0941048562992 + - type: nauc_mrr_at_5_diff1 + value: 76.03551521362746 + - type: nauc_mrr_at_5_max + value: 21.061039787483693 + - type: nauc_mrr_at_5_std + value: -41.00853106709472 + - type: nauc_ndcg_at_1000_diff1 + value: 76.0262393768304 + - type: nauc_ndcg_at_1000_max + value: 19.800136317778524 + - type: nauc_ndcg_at_1000_std + value: -43.33700110014878 + - type: nauc_ndcg_at_100_diff1 + value: 76.01134308572229 + - type: nauc_ndcg_at_100_max + value: 19.551124579535546 + - type: nauc_ndcg_at_100_std + value: -43.552980627915744 + - type: nauc_ndcg_at_10_diff1 + value: 76.15073912713932 + - type: nauc_ndcg_at_10_max + value: 18.633106350000777 + - type: nauc_ndcg_at_10_std + value: -46.874070251398344 + - type: nauc_ndcg_at_1_diff1 + value: 77.05621804280337 + - type: nauc_ndcg_at_1_max + value: 20.740334635080874 + - type: nauc_ndcg_at_1_std + value: -38.14163986670599 + - type: nauc_ndcg_at_20_diff1 + value: 76.056131198408 + - type: nauc_ndcg_at_20_max + value: 19.12183899438156 + - type: nauc_ndcg_at_20_std + value: -45.152222560853474 + - type: nauc_ndcg_at_3_diff1 + value: 75.32067175696538 + - type: nauc_ndcg_at_3_max + value: 18.030151234107024 + - type: nauc_ndcg_at_3_std + value: -46.954391996130255 + - type: nauc_ndcg_at_5_diff1 + value: 76.07279461161282 + - type: nauc_ndcg_at_5_max + value: 18.676703248713565 + - type: nauc_ndcg_at_5_std + value: -48.02252026510146 + - type: nauc_precision_at_1000_diff1 + value: -43.05385069777404 + - type: nauc_precision_at_1000_max + value: 5.177830943538669 + - type: nauc_precision_at_1000_std + value: 41.72849191406853 + - type: nauc_precision_at_100_diff1 + value: -42.74879635836525 + - type: nauc_precision_at_100_max + value: 4.331905136884358 + - type: nauc_precision_at_100_std + value: 40.95092522149582 + - type: nauc_precision_at_10_diff1 + value: -37.24241604831716 + - type: nauc_precision_at_10_max + value: 5.858095259543203 + - type: nauc_precision_at_10_std + value: 26.84790750024595 + - type: nauc_precision_at_1_diff1 + value: 77.05621804280337 + - type: nauc_precision_at_1_max + value: 20.740334635080874 + - type: nauc_precision_at_1_std + value: -38.14163986670599 + - type: nauc_precision_at_20_diff1 + value: -40.981648074847115 + - type: nauc_precision_at_20_max + value: 4.952729889757595 + - type: nauc_precision_at_20_std + value: 34.118899263271544 + - type: nauc_precision_at_3_diff1 + value: -14.632656898252725 + - type: nauc_precision_at_3_max + value: 9.219680305543351 + - type: nauc_precision_at_3_std + value: 2.8267912495305985 + - type: nauc_precision_at_5_diff1 + value: -28.714307596291906 + - type: nauc_precision_at_5_max + value: 7.8282629737599905 + - type: nauc_precision_at_5_std + value: 15.30157769264485 + - type: nauc_recall_at_1000_diff1 + value: 75.28082712455692 + - type: nauc_recall_at_1000_max + value: 16.032150210343172 + - type: nauc_recall_at_1000_std + value: 33.623223824303615 + - type: nauc_recall_at_100_diff1 + value: 66.48857771758026 + - type: nauc_recall_at_100_max + value: -0.8298370121173152 + - type: nauc_recall_at_100_std + value: -47.59134911021335 + - type: nauc_recall_at_10_diff1 + value: 71.54324183557964 + - type: nauc_recall_at_10_max + value: 10.71338879553047 + - type: nauc_recall_at_10_std + value: -66.27316488339173 + - type: nauc_recall_at_1_diff1 + value: 79.82735407515446 + - type: nauc_recall_at_1_max + value: 12.275645564446238 + - type: nauc_recall_at_1_std + value: -41.53738015617803 + - type: nauc_recall_at_20_diff1 + value: 70.15017942540359 + - type: nauc_recall_at_20_max + value: 11.699321903394438 + - type: nauc_recall_at_20_std + value: -61.79366072599887 + - type: nauc_recall_at_3_diff1 + value: 72.5862364297169 + - type: nauc_recall_at_3_max + value: 12.145065627274022 + - type: nauc_recall_at_3_std + value: -56.82607742333548 + - type: nauc_recall_at_5_diff1 + value: 72.07603033579613 + - type: nauc_recall_at_5_max + value: 13.121563460576949 + - type: nauc_recall_at_5_std + value: -63.3551506977008 + - type: ndcg_at_1 + value: 79.66 + - type: ndcg_at_10 + value: 87.469 + - type: ndcg_at_100 + value: 88.742 + - type: ndcg_at_1000 + value: 88.835 + - type: ndcg_at_20 + value: 88.19 + - type: ndcg_at_3 + value: 84.301 + - type: ndcg_at_5 + value: 86.122 + - type: precision_at_1 + value: 79.66 + - type: precision_at_10 + value: 13.366 + - type: precision_at_100 + value: 1.53 + - type: precision_at_1000 + value: 0.157 + - type: precision_at_20 + value: 7.114 + - type: precision_at_3 + value: 36.953 + - type: precision_at_5 + value: 24.436 + - type: recall_at_1 + value: 69.226 + - type: recall_at_10 + value: 95.31500000000001 + - type: recall_at_100 + value: 99.586 + - type: recall_at_1000 + value: 99.978 + - type: recall_at_20 + value: 97.626 + - type: recall_at_3 + value: 86.39 + - type: recall_at_5 + value: 91.429 + task: + type: Retrieval + - dataset: + config: default + name: MTEB RedditClustering + revision: 24640382cdbf8abc73003fb0fa6d111a705499eb + split: test + type: mteb/reddit-clustering + metrics: + - type: main_score + value: 55.46831401047426 + - type: v_measure + value: 55.46831401047426 + - type: v_measure_std + value: 4.874007949262904 + task: + type: Clustering + - dataset: + config: default + name: MTEB RedditClusteringP2P + revision: 385e3cb46b4cfa89021f56c4380204149d0efe33 + split: test + type: mteb/reddit-clustering-p2p + metrics: + - type: main_score + value: 65.95763514051492 + - type: v_measure + value: 65.95763514051492 + - type: v_measure_std + value: 12.11633090692919 + task: + type: Clustering + - dataset: + config: default + name: MTEB SCIDOCS + revision: f8c2fcf00f625baaa80f62ec5bd9e1fff3b8ae88 + split: test + type: mteb/scidocs + metrics: + - type: main_score + value: 19.967 + - type: map_at_1 + value: 4.555 + - type: map_at_10 + value: 11.875 + - type: map_at_100 + value: 13.828 + - type: map_at_1000 + value: 14.113999999999999 + - type: map_at_20 + value: 12.837000000000002 + - type: map_at_3 + value: 8.368 + - type: map_at_5 + value: 10.02 + - type: mrr_at_1 + value: 22.5 + - type: mrr_at_10 + value: 33.34718253968251 + - type: mrr_at_100 + value: 34.33837465870857 + - type: mrr_at_1000 + value: 34.405354703860766 + - type: mrr_at_20 + value: 33.921715836622894 + - type: mrr_at_3 + value: 29.666666666666657 + - type: mrr_at_5 + value: 31.966666666666622 + - type: nauc_map_at_1000_diff1 + value: 15.246786102454319 + - type: nauc_map_at_1000_max + value: 29.175127621523284 + - type: nauc_map_at_1000_std + value: 16.942548728665557 + - type: nauc_map_at_100_diff1 + value: 15.30276004899313 + - type: nauc_map_at_100_max + value: 29.17103174716017 + - type: nauc_map_at_100_std + value: 16.7166806855191 + - type: nauc_map_at_10_diff1 + value: 14.959570431789576 + - type: nauc_map_at_10_max + value: 28.741898266169592 + - type: nauc_map_at_10_std + value: 13.598940269616955 + - type: nauc_map_at_1_diff1 + value: 25.644593996052723 + - type: nauc_map_at_1_max + value: 23.219289075841036 + - type: nauc_map_at_1_std + value: 5.690909359575662 + - type: nauc_map_at_20_diff1 + value: 15.057405804364974 + - type: nauc_map_at_20_max + value: 29.136129840808078 + - type: nauc_map_at_20_std + value: 15.438603298509909 + - type: nauc_map_at_3_diff1 + value: 17.233527889256564 + - type: nauc_map_at_3_max + value: 26.441548940276377 + - type: nauc_map_at_3_std + value: 7.752631479331336 + - type: nauc_map_at_5_diff1 + value: 16.20805385985111 + - type: nauc_map_at_5_max + value: 27.923553597579676 + - type: nauc_map_at_5_std + value: 8.860227830088574 + - type: nauc_mrr_at_1000_diff1 + value: 22.22324136471676 + - type: nauc_mrr_at_1000_max + value: 25.803381711627445 + - type: nauc_mrr_at_1000_std + value: 11.195218592300574 + - type: nauc_mrr_at_100_diff1 + value: 22.20635139247774 + - type: nauc_mrr_at_100_max + value: 25.809980468254544 + - type: nauc_mrr_at_100_std + value: 11.236406379403093 + - type: nauc_mrr_at_10_diff1 + value: 22.197126629234866 + - type: nauc_mrr_at_10_max + value: 25.751462289366074 + - type: nauc_mrr_at_10_std + value: 11.06813122476576 + - type: nauc_mrr_at_1_diff1 + value: 25.768694915906547 + - type: nauc_mrr_at_1_max + value: 23.078936227533376 + - type: nauc_mrr_at_1_std + value: 5.664626871415919 + - type: nauc_mrr_at_20_diff1 + value: 22.333510048115247 + - type: nauc_mrr_at_20_max + value: 25.933807629983047 + - type: nauc_mrr_at_20_std + value: 11.375015615529657 + - type: nauc_mrr_at_3_diff1 + value: 22.243005847898107 + - type: nauc_mrr_at_3_max + value: 25.592329404890364 + - type: nauc_mrr_at_3_std + value: 8.649587914029771 + - type: nauc_mrr_at_5_diff1 + value: 21.995522212053803 + - type: nauc_mrr_at_5_max + value: 25.8495186116959 + - type: nauc_mrr_at_5_std + value: 10.335015509252274 + - type: nauc_ndcg_at_1000_diff1 + value: 15.720525782244835 + - type: nauc_ndcg_at_1000_max + value: 29.255225161988598 + - type: nauc_ndcg_at_1000_std + value: 23.91092705272815 + - type: nauc_ndcg_at_100_diff1 + value: 16.578411443446196 + - type: nauc_ndcg_at_100_max + value: 29.914913377694734 + - type: nauc_ndcg_at_100_std + value: 22.811268453728623 + - type: nauc_ndcg_at_10_diff1 + value: 16.397948417538824 + - type: nauc_ndcg_at_10_max + value: 29.251930474900416 + - type: nauc_ndcg_at_10_std + value: 16.636737668789657 + - type: nauc_ndcg_at_1_diff1 + value: 25.768694915906547 + - type: nauc_ndcg_at_1_max + value: 23.078936227533376 + - type: nauc_ndcg_at_1_std + value: 5.664626871415919 + - type: nauc_ndcg_at_20_diff1 + value: 16.648110682686614 + - type: nauc_ndcg_at_20_max + value: 29.892133157795776 + - type: nauc_ndcg_at_20_std + value: 19.34882283787698 + - type: nauc_ndcg_at_3_diff1 + value: 17.92949134249488 + - type: nauc_ndcg_at_3_max + value: 26.689585250329532 + - type: nauc_ndcg_at_3_std + value: 9.078574087750605 + - type: nauc_ndcg_at_5_diff1 + value: 16.952122694491774 + - type: nauc_ndcg_at_5_max + value: 28.146728497211676 + - type: nauc_ndcg_at_5_std + value: 11.339213600616807 + - type: nauc_precision_at_1000_diff1 + value: 3.2349869782475853 + - type: nauc_precision_at_1000_max + value: 18.6171242830797 + - type: nauc_precision_at_1000_std + value: 34.88770887741456 + - type: nauc_precision_at_100_diff1 + value: 9.966266554726051 + - type: nauc_precision_at_100_max + value: 25.428513081164468 + - type: nauc_precision_at_100_std + value: 31.606347333469454 + - type: nauc_precision_at_10_diff1 + value: 11.95936021707633 + - type: nauc_precision_at_10_max + value: 29.308109146502513 + - type: nauc_precision_at_10_std + value: 22.465491503445836 + - type: nauc_precision_at_1_diff1 + value: 25.768694915906547 + - type: nauc_precision_at_1_max + value: 23.078936227533376 + - type: nauc_precision_at_1_std + value: 5.664626871415919 + - type: nauc_precision_at_20_diff1 + value: 11.875243219095264 + - type: nauc_precision_at_20_max + value: 28.83348195510638 + - type: nauc_precision_at_20_std + value: 26.223512022658024 + - type: nauc_precision_at_3_diff1 + value: 14.439006009003496 + - type: nauc_precision_at_3_max + value: 27.64037376883578 + - type: nauc_precision_at_3_std + value: 10.455585887861348 + - type: nauc_precision_at_5_diff1 + value: 12.944620616816508 + - type: nauc_precision_at_5_max + value: 29.09151148910875 + - type: nauc_precision_at_5_std + value: 14.040461515594629 + - type: nauc_recall_at_1000_diff1 + value: 3.0792658335602208 + - type: nauc_recall_at_1000_max + value: 18.252163670467 + - type: nauc_recall_at_1000_std + value: 35.56094641865878 + - type: nauc_recall_at_100_diff1 + value: 10.082532567537765 + - type: nauc_recall_at_100_max + value: 25.178104826511184 + - type: nauc_recall_at_100_std + value: 31.519235979935452 + - type: nauc_recall_at_10_diff1 + value: 11.756861941382278 + - type: nauc_recall_at_10_max + value: 28.92701783055643 + - type: nauc_recall_at_10_std + value: 22.012165818680277 + - type: nauc_recall_at_1_diff1 + value: 25.644593996052723 + - type: nauc_recall_at_1_max + value: 23.219289075841036 + - type: nauc_recall_at_1_std + value: 5.690909359575662 + - type: nauc_recall_at_20_diff1 + value: 11.832195217863916 + - type: nauc_recall_at_20_max + value: 28.501838667089185 + - type: nauc_recall_at_20_std + value: 25.826929326972863 + - type: nauc_recall_at_3_diff1 + value: 14.320529120124215 + - type: nauc_recall_at_3_max + value: 27.674150729188774 + - type: nauc_recall_at_3_std + value: 10.3005989655898 + - type: nauc_recall_at_5_diff1 + value: 12.524512695192328 + - type: nauc_recall_at_5_max + value: 28.617421062881988 + - type: nauc_recall_at_5_std + value: 13.626386741802321 + - type: ndcg_at_1 + value: 22.5 + - type: ndcg_at_10 + value: 19.967 + - type: ndcg_at_100 + value: 27.489 + - type: ndcg_at_1000 + value: 32.675 + - type: ndcg_at_20 + value: 22.508 + - type: ndcg_at_3 + value: 18.593 + - type: ndcg_at_5 + value: 16.395 + - type: precision_at_1 + value: 22.5 + - type: precision_at_10 + value: 10.48 + - type: precision_at_100 + value: 2.139 + - type: precision_at_1000 + value: 0.338 + - type: precision_at_20 + value: 6.710000000000001 + - type: precision_at_3 + value: 17.5 + - type: precision_at_5 + value: 14.540000000000001 + - type: recall_at_1 + value: 4.555 + - type: recall_at_10 + value: 21.292 + - type: recall_at_100 + value: 43.438 + - type: recall_at_1000 + value: 68.693 + - type: recall_at_20 + value: 27.235 + - type: recall_at_3 + value: 10.635 + - type: recall_at_5 + value: 14.773 + task: + type: Retrieval + - dataset: + config: default + name: MTEB SICK-R + revision: 20a6d6f312dd54037fe07a32d58e5e168867909d + split: test + type: mteb/sickr-sts + metrics: + - type: cosine_pearson + value: 83.80746802573121 + - type: cosine_spearman + value: 79.72822038569593 + - type: euclidean_pearson + value: 81.45002980681224 + - type: euclidean_spearman + value: 79.7282231441618 + - type: main_score + value: 79.72822038569593 + - type: manhattan_pearson + value: 81.45611398621358 + - type: manhattan_spearman + value: 79.59811193829754 + - type: pearson + value: 83.80746802573121 + - type: spearman + value: 79.72822038569593 + task: + type: STS + - dataset: + config: default + name: MTEB STS12 + revision: a0d554a64d88156834ff5ae9920b964011b16384 + split: test + type: mteb/sts12-sts + metrics: + - type: cosine_pearson + value: 87.3063084337013 + - type: cosine_spearman + value: 80.16653334500153 + - type: euclidean_pearson + value: 84.24369437820418 + - type: euclidean_spearman + value: 80.16592228280822 + - type: main_score + value: 80.16653334500153 + - type: manhattan_pearson + value: 84.41598625996673 + - type: manhattan_spearman + value: 80.37993757607657 + - type: pearson + value: 87.3063084337013 + - type: spearman + value: 80.16653334500153 + task: + type: STS + - dataset: + config: default + name: MTEB STS13 + revision: 7e90230a92c190f1bf69ae9002b8cea547a64cca + split: test + type: mteb/sts13-sts + metrics: + - type: cosine_pearson + value: 82.56917286956067 + - type: cosine_spearman + value: 83.86019247474584 + - type: euclidean_pearson + value: 83.50577094515269 + - type: euclidean_spearman + value: 83.86019240020342 + - type: main_score + value: 83.86019247474584 + - type: manhattan_pearson + value: 83.7035831723258 + - type: manhattan_spearman + value: 84.06951592208223 + - type: pearson + value: 82.56917286956067 + - type: spearman + value: 83.86019247474584 + task: + type: STS + - dataset: + config: default + name: MTEB STS14 + revision: 6031580fec1f6af667f0bd2da0a551cf4f0b2375 + split: test + type: mteb/sts14-sts + metrics: + - type: cosine_pearson + value: 82.57349080713945 + - type: cosine_spearman + value: 80.5656376543151 + - type: euclidean_pearson + value: 82.20020711843652 + - type: euclidean_spearman + value: 80.56562435929797 + - type: main_score + value: 80.5656376543151 + - type: manhattan_pearson + value: 82.02137444105155 + - type: manhattan_spearman + value: 80.49473867409459 + - type: pearson + value: 82.57349080713945 + - type: spearman + value: 80.5656376543151 + task: + type: STS + - dataset: + config: default + name: MTEB STS15 + revision: ae752c7c21bf194d8b67fd573edf7ae58183cbe3 + split: test + type: mteb/sts15-sts + metrics: + - type: cosine_pearson + value: 85.89932616809199 + - type: cosine_spearman + value: 87.34265089376507 + - type: euclidean_pearson + value: 86.70361638349567 + - type: euclidean_spearman + value: 87.34265218193244 + - type: main_score + value: 87.34265089376507 + - type: manhattan_pearson + value: 86.51826134790748 + - type: manhattan_spearman + value: 87.18812519859725 + - type: pearson + value: 85.89932616809199 + - type: spearman + value: 87.34265089376507 + task: + type: STS + - dataset: + config: default + name: MTEB STS16 + revision: 4d8694f8f0e0100860b497b999b3dbed754a0513 + split: test + type: mteb/sts16-sts + metrics: + - type: cosine_pearson + value: 83.57539157713117 + - type: cosine_spearman + value: 84.8327592322146 + - type: euclidean_pearson + value: 84.04928896411587 + - type: euclidean_spearman + value: 84.83275862198806 + - type: main_score + value: 84.8327592322146 + - type: manhattan_pearson + value: 83.88919473293718 + - type: manhattan_spearman + value: 84.64175696279177 + - type: pearson + value: 83.57539157713117 + - type: spearman + value: 84.8327592322146 + task: + type: STS + - dataset: + config: en-ar + name: MTEB STS17 (en-ar) + revision: faeb762787bd10488a50c8b5be4a3b82e411949c + split: test + type: mteb/sts17-crosslingual-sts + metrics: + - type: cosine_pearson + value: 56.77939191972398 + - type: cosine_spearman + value: 56.44421451336106 + - type: euclidean_pearson + value: 57.84746999725719 + - type: euclidean_spearman + value: 56.44421451336106 + - type: main_score + value: 56.44421451336106 + - type: manhattan_pearson + value: 56.368375736935185 + - type: manhattan_spearman + value: 54.443033747967476 + - type: pearson + value: 56.77939191972398 + - type: spearman + value: 56.44421451336106 + task: + type: STS + - dataset: + config: en-de + name: MTEB STS17 (en-de) + revision: faeb762787bd10488a50c8b5be4a3b82e411949c + split: test + type: mteb/sts17-crosslingual-sts + metrics: + - type: cosine_pearson + value: 72.7123285460032 + - type: cosine_spearman + value: 73.0413412902408 + - type: euclidean_pearson + value: 73.26576109213129 + - type: euclidean_spearman + value: 73.0413412902408 + - type: main_score + value: 73.0413412902408 + - type: manhattan_pearson + value: 73.18845328161191 + - type: manhattan_spearman + value: 73.01804685924571 + - type: pearson + value: 72.7123285460032 + - type: spearman + value: 73.0413412902408 + task: + type: STS + - dataset: + config: en-en + name: MTEB STS17 (en-en) + revision: faeb762787bd10488a50c8b5be4a3b82e411949c + split: test + type: mteb/sts17-crosslingual-sts + metrics: + - type: cosine_pearson + value: 85.76937213938014 + - type: cosine_spearman + value: 86.42806650035523 + - type: euclidean_pearson + value: 86.6429129637805 + - type: euclidean_spearman + value: 86.42806650035523 + - type: main_score + value: 86.42806650035523 + - type: manhattan_pearson + value: 86.57962956067654 + - type: manhattan_spearman + value: 86.18220646628836 + - type: pearson + value: 85.76937213938014 + - type: spearman + value: 86.42806650035523 + task: + type: STS + - dataset: + config: en-tr + name: MTEB STS17 (en-tr) + revision: faeb762787bd10488a50c8b5be4a3b82e411949c + split: test + type: mteb/sts17-crosslingual-sts + metrics: + - type: cosine_pearson + value: 53.63838562350928 + - type: cosine_spearman + value: 51.425433307585536 + - type: euclidean_pearson + value: 53.89159715974961 + - type: euclidean_spearman + value: 51.425433307585536 + - type: main_score + value: 51.425433307585536 + - type: manhattan_pearson + value: 53.072978089852526 + - type: manhattan_spearman + value: 50.42207979849882 + - type: pearson + value: 53.63838562350928 + - type: spearman + value: 51.425433307585536 + task: + type: STS + - dataset: + config: es-en + name: MTEB STS17 (es-en) + revision: faeb762787bd10488a50c8b5be4a3b82e411949c + split: test + type: mteb/sts17-crosslingual-sts + metrics: + - type: cosine_pearson + value: 74.86829926679785 + - type: cosine_spearman + value: 75.97371995137269 + - type: euclidean_pearson + value: 75.31029737838625 + - type: euclidean_spearman + value: 75.97371995137269 + - type: main_score + value: 75.97371995137269 + - type: manhattan_pearson + value: 75.65839894637686 + - type: manhattan_spearman + value: 76.1355272163419 + - type: pearson + value: 74.86829926679785 + - type: spearman + value: 75.97371995137269 + task: + type: STS + - dataset: + config: fr-en + name: MTEB STS17 (fr-en) + revision: faeb762787bd10488a50c8b5be4a3b82e411949c + split: test + type: mteb/sts17-crosslingual-sts + metrics: + - type: cosine_pearson + value: 74.71447286299487 + - type: cosine_spearman + value: 74.06702660747132 + - type: euclidean_pearson + value: 75.38807553614491 + - type: euclidean_spearman + value: 74.06702660747132 + - type: main_score + value: 74.06702660747132 + - type: manhattan_pearson + value: 75.12369492510138 + - type: manhattan_spearman + value: 73.91369082324793 + - type: pearson + value: 74.71447286299487 + - type: spearman + value: 74.06702660747132 + task: + type: STS + - dataset: + config: it-en + name: MTEB STS17 (it-en) + revision: faeb762787bd10488a50c8b5be4a3b82e411949c + split: test + type: mteb/sts17-crosslingual-sts + metrics: + - type: cosine_pearson + value: 72.38044871568002 + - type: cosine_spearman + value: 72.67462698051628 + - type: euclidean_pearson + value: 72.93088753362804 + - type: euclidean_spearman + value: 72.67462698051628 + - type: main_score + value: 72.67462698051628 + - type: manhattan_pearson + value: 72.36507526375641 + - type: manhattan_spearman + value: 72.05743987811199 + - type: pearson + value: 72.38044871568002 + - type: spearman + value: 72.67462698051628 + task: + type: STS + - dataset: + config: nl-en + name: MTEB STS17 (nl-en) + revision: faeb762787bd10488a50c8b5be4a3b82e411949c + split: test + type: mteb/sts17-crosslingual-sts + metrics: + - type: cosine_pearson + value: 70.21041932205698 + - type: cosine_spearman + value: 69.70028178727163 + - type: euclidean_pearson + value: 70.90563853073118 + - type: euclidean_spearman + value: 69.70028178727163 + - type: main_score + value: 69.70028178727163 + - type: manhattan_pearson + value: 70.57814054683753 + - type: manhattan_spearman + value: 69.09462658139957 + - type: pearson + value: 70.21041932205698 + - type: spearman + value: 69.70028178727163 + task: + type: STS + - dataset: + config: en + name: MTEB STS22 (en) + revision: de9d86b3b84231dc21f76c7b7af1f28e2f57f6e3 + split: test + type: mteb/sts22-crosslingual-sts + metrics: + - type: cosine_pearson + value: 69.13900601047764 + - type: cosine_spearman + value: 69.2134140940147 + - type: euclidean_pearson + value: 70.74438767160798 + - type: euclidean_spearman + value: 69.2134140940147 + - type: main_score + value: 69.2134140940147 + - type: manhattan_pearson + value: 71.02398307661159 + - type: manhattan_spearman + value: 69.37936110586129 + - type: pearson + value: 69.13900601047764 + - type: spearman + value: 69.2134140940147 + task: + type: STS + - dataset: + config: de-en + name: MTEB STS22 (de-en) + revision: de9d86b3b84231dc21f76c7b7af1f28e2f57f6e3 + split: test + type: mteb/sts22-crosslingual-sts + metrics: + - type: cosine_pearson + value: 64.8805006605935 + - type: cosine_spearman + value: 63.81067904946749 + - type: euclidean_pearson + value: 67.42954643546759 + - type: euclidean_spearman + value: 63.81067904946749 + - type: main_score + value: 63.81067904946749 + - type: manhattan_pearson + value: 68.5718639745086 + - type: manhattan_spearman + value: 64.77027566921338 + - type: pearson + value: 64.8805006605935 + - type: spearman + value: 63.81067904946749 + task: + type: STS + - dataset: + config: es-en + name: MTEB STS22 (es-en) + revision: de9d86b3b84231dc21f76c7b7af1f28e2f57f6e3 + split: test + type: mteb/sts22-crosslingual-sts + metrics: + - type: cosine_pearson + value: 75.50011820404733 + - type: cosine_spearman + value: 78.22942242197527 + - type: euclidean_pearson + value: 76.7001402354704 + - type: euclidean_spearman + value: 78.22942242197527 + - type: main_score + value: 78.22942242197527 + - type: manhattan_pearson + value: 78.77879852950737 + - type: manhattan_spearman + value: 79.58485702006229 + - type: pearson + value: 75.50011820404733 + - type: spearman + value: 78.22942242197527 + task: + type: STS + - dataset: + config: pl-en + name: MTEB STS22 (pl-en) + revision: de9d86b3b84231dc21f76c7b7af1f28e2f57f6e3 + split: test + type: mteb/sts22-crosslingual-sts + metrics: + - type: cosine_pearson + value: 78.98435260035146 + - type: cosine_spearman + value: 79.0083835281799 + - type: euclidean_pearson + value: 79.05051294275746 + - type: euclidean_spearman + value: 79.0083835281799 + - type: main_score + value: 79.0083835281799 + - type: manhattan_pearson + value: 79.28546319449809 + - type: manhattan_spearman + value: 78.79790129500051 + - type: pearson + value: 78.98435260035146 + - type: spearman + value: 79.0083835281799 + task: + type: STS + - dataset: + config: zh-en + name: MTEB STS22 (zh-en) + revision: de9d86b3b84231dc21f76c7b7af1f28e2f57f6e3 + split: test + type: mteb/sts22-crosslingual-sts + metrics: + - type: cosine_pearson + value: 76.5491270627292 + - type: cosine_spearman + value: 76.34747339191613 + - type: euclidean_pearson + value: 77.52703884917828 + - type: euclidean_spearman + value: 76.34747339191613 + - type: main_score + value: 76.34747339191613 + - type: manhattan_pearson + value: 77.94417346436772 + - type: manhattan_spearman + value: 76.15552728253843 + - type: pearson + value: 76.5491270627292 + - type: spearman + value: 76.34747339191613 + task: + type: STS + - dataset: + config: default + name: MTEB STSBenchmark + revision: b0fddb56ed78048fa8b90373c8a3cfc37b684831 + split: test + type: mteb/stsbenchmark-sts + metrics: + - type: cosine_pearson + value: 81.44263012519023 + - type: cosine_spearman + value: 84.1219543406234 + - type: euclidean_pearson + value: 83.61379006230743 + - type: euclidean_spearman + value: 84.1219543406234 + - type: main_score + value: 84.1219543406234 + - type: manhattan_pearson + value: 83.40336227949633 + - type: manhattan_spearman + value: 83.94639826019986 + - type: pearson + value: 81.44263012519023 + - type: spearman + value: 84.1219543406234 + task: + type: STS + - dataset: + config: default + name: MTEB SciDocsRR + revision: d3c5e1fc0b855ab6097bf1cda04dd73947d7caab + split: test + type: mteb/scidocs-reranking + metrics: + - type: main_score + value: 80.99314184170828 + - type: map + value: 80.99314184170828 + - type: mrr + value: 94.80081918807409 + - type: nAUC_map_diff1 + value: 2.519316498883737 + - type: nAUC_map_max + value: 53.83115362914741 + - type: nAUC_map_std + value: 71.18884208143534 + - type: nAUC_mrr_diff1 + value: 49.13813585315955 + - type: nAUC_mrr_max + value: 82.97821200827201 + - type: nAUC_mrr_std + value: 80.52851378034694 + task: + type: Reranking + - dataset: + config: default + name: MTEB SciFact + revision: 0228b52cf27578f30900b9e5271d331663a030d7 + split: test + type: mteb/scifact + metrics: + - type: main_score + value: 72.887 + - type: map_at_1 + value: 56.983 + - type: map_at_10 + value: 68.02 + - type: map_at_100 + value: 68.47800000000001 + - type: map_at_1000 + value: 68.49300000000001 + - type: map_at_20 + value: 68.319 + - type: map_at_3 + value: 64.735 + - type: map_at_5 + value: 66.797 + - type: mrr_at_1 + value: 60.0 + - type: mrr_at_10 + value: 68.80674603174603 + - type: mrr_at_100 + value: 69.17683357934396 + - type: mrr_at_1000 + value: 69.19182270659053 + - type: mrr_at_20 + value: 69.04443591467584 + - type: mrr_at_3 + value: 66.22222222222221 + - type: mrr_at_5 + value: 67.88888888888889 + - type: nauc_map_at_1000_diff1 + value: 65.74751339600985 + - type: nauc_map_at_1000_max + value: 51.01631394192724 + - type: nauc_map_at_1000_std + value: -4.749077682272638 + - type: nauc_map_at_100_diff1 + value: 65.7511025839809 + - type: nauc_map_at_100_max + value: 51.02179965776254 + - type: nauc_map_at_100_std + value: -4.750648691834171 + - type: nauc_map_at_10_diff1 + value: 65.8078854827133 + - type: nauc_map_at_10_max + value: 51.17250624028733 + - type: nauc_map_at_10_std + value: -5.145740047184135 + - type: nauc_map_at_1_diff1 + value: 69.1712101533685 + - type: nauc_map_at_1_max + value: 46.030719621812224 + - type: nauc_map_at_1_std + value: -8.492699133614764 + - type: nauc_map_at_20_diff1 + value: 65.72860985168403 + - type: nauc_map_at_20_max + value: 51.02243609996885 + - type: nauc_map_at_20_std + value: -4.897680708725447 + - type: nauc_map_at_3_diff1 + value: 66.04795347641515 + - type: nauc_map_at_3_max + value: 49.28417356777598 + - type: nauc_map_at_3_std + value: -8.144355960324793 + - type: nauc_map_at_5_diff1 + value: 65.39564471101812 + - type: nauc_map_at_5_max + value: 49.8786868247306 + - type: nauc_map_at_5_std + value: -5.361632284686927 + - type: nauc_mrr_at_1000_diff1 + value: 65.35483966038701 + - type: nauc_mrr_at_1000_max + value: 52.46038918168655 + - type: nauc_mrr_at_1000_std + value: -2.9488525087661865 + - type: nauc_mrr_at_100_diff1 + value: 65.35869633533838 + - type: nauc_mrr_at_100_max + value: 52.465078488348915 + - type: nauc_mrr_at_100_std + value: -2.9515006369786185 + - type: nauc_mrr_at_10_diff1 + value: 65.32937085681456 + - type: nauc_mrr_at_10_max + value: 52.69576591011457 + - type: nauc_mrr_at_10_std + value: -3.211748975877107 + - type: nauc_mrr_at_1_diff1 + value: 68.74146631047786 + - type: nauc_mrr_at_1_max + value: 50.13505491243696 + - type: nauc_mrr_at_1_std + value: -1.8066686454932306 + - type: nauc_mrr_at_20_diff1 + value: 65.30292147401966 + - type: nauc_mrr_at_20_max + value: 52.506885523141 + - type: nauc_mrr_at_20_std + value: -3.030133033199617 + - type: nauc_mrr_at_3_diff1 + value: 64.76693508063609 + - type: nauc_mrr_at_3_max + value: 52.16422755733336 + - type: nauc_mrr_at_3_std + value: -4.375097949954107 + - type: nauc_mrr_at_5_diff1 + value: 64.72226560669719 + - type: nauc_mrr_at_5_max + value: 52.402248270260976 + - type: nauc_mrr_at_5_std + value: -2.3470762334639543 + - type: nauc_ndcg_at_1000_diff1 + value: 65.05291476181804 + - type: nauc_ndcg_at_1000_max + value: 52.29203656356969 + - type: nauc_ndcg_at_1000_std + value: -3.2044176855608937 + - type: nauc_ndcg_at_100_diff1 + value: 65.14234495976609 + - type: nauc_ndcg_at_100_max + value: 52.648998314377046 + - type: nauc_ndcg_at_100_std + value: -2.836497638004637 + - type: nauc_ndcg_at_10_diff1 + value: 65.07401371760211 + - type: nauc_ndcg_at_10_max + value: 53.576715684408214 + - type: nauc_ndcg_at_10_std + value: -4.308181380375265 + - type: nauc_ndcg_at_1_diff1 + value: 68.74146631047786 + - type: nauc_ndcg_at_1_max + value: 50.13505491243696 + - type: nauc_ndcg_at_1_std + value: -1.8066686454932306 + - type: nauc_ndcg_at_20_diff1 + value: 64.84825741925087 + - type: nauc_ndcg_at_20_max + value: 52.92789159833541 + - type: nauc_ndcg_at_20_std + value: -3.3072860677499856 + - type: nauc_ndcg_at_3_diff1 + value: 64.37308313406504 + - type: nauc_ndcg_at_3_max + value: 51.32442115826017 + - type: nauc_ndcg_at_3_std + value: -7.029903565507216 + - type: nauc_ndcg_at_5_diff1 + value: 63.86750880299841 + - type: nauc_ndcg_at_5_max + value: 51.37420252304249 + - type: nauc_ndcg_at_5_std + value: -3.6483750498877447 + - type: nauc_precision_at_1000_diff1 + value: -26.510729177260178 + - type: nauc_precision_at_1000_max + value: 14.42625173734718 + - type: nauc_precision_at_1000_std + value: 45.496457507034286 + - type: nauc_precision_at_100_diff1 + value: -14.374685492703476 + - type: nauc_precision_at_100_max + value: 21.681911020335644 + - type: nauc_precision_at_100_std + value: 42.16203415919248 + - type: nauc_precision_at_10_diff1 + value: 7.815716722545709 + - type: nauc_precision_at_10_max + value: 36.03562535537121 + - type: nauc_precision_at_10_std + value: 28.24000548987315 + - type: nauc_precision_at_1_diff1 + value: 68.74146631047786 + - type: nauc_precision_at_1_max + value: 50.13505491243696 + - type: nauc_precision_at_1_std + value: -1.8066686454932306 + - type: nauc_precision_at_20_diff1 + value: -1.0860405342771857 + - type: nauc_precision_at_20_max + value: 29.773416761471406 + - type: nauc_precision_at_20_std + value: 34.781738507563304 + - type: nauc_precision_at_3_diff1 + value: 36.2217738770724 + - type: nauc_precision_at_3_max + value: 43.356155845594834 + - type: nauc_precision_at_3_std + value: 9.506045734283152 + - type: nauc_precision_at_5_diff1 + value: 20.749541232335027 + - type: nauc_precision_at_5_max + value: 38.57715809985047 + - type: nauc_precision_at_5_std + value: 22.581774153881096 + - type: nauc_recall_at_1000_diff1 + value: 12.278244631182748 + - type: nauc_recall_at_1000_max + value: -56.34920634920787 + - type: nauc_recall_at_1000_std + value: 12.278244631182748 + - type: nauc_recall_at_100_diff1 + value: 64.01833460657008 + - type: nauc_recall_at_100_max + value: 60.00763941940402 + - type: nauc_recall_at_100_std + value: 18.572277395806154 + - type: nauc_recall_at_10_diff1 + value: 62.32242861171786 + - type: nauc_recall_at_10_max + value: 62.75965074851927 + - type: nauc_recall_at_10_std + value: -6.085630582444918 + - type: nauc_recall_at_1_diff1 + value: 69.1712101533685 + - type: nauc_recall_at_1_max + value: 46.030719621812224 + - type: nauc_recall_at_1_std + value: -8.492699133614764 + - type: nauc_recall_at_20_diff1 + value: 59.109050124114724 + - type: nauc_recall_at_20_max + value: 59.41051217234864 + - type: nauc_recall_at_20_std + value: 3.246339186081203 + - type: nauc_recall_at_3_diff1 + value: 60.981933198798856 + - type: nauc_recall_at_3_max + value: 51.04347606806292 + - type: nauc_recall_at_3_std + value: -12.852440576759369 + - type: nauc_recall_at_5_diff1 + value: 58.155380815418766 + - type: nauc_recall_at_5_max + value: 52.3232345936131 + - type: nauc_recall_at_5_std + value: -1.264784301127294 + - type: ndcg_at_1 + value: 60.0 + - type: ndcg_at_10 + value: 72.887 + - type: ndcg_at_100 + value: 74.92699999999999 + - type: ndcg_at_1000 + value: 75.351 + - type: ndcg_at_20 + value: 73.882 + - type: ndcg_at_3 + value: 67.289 + - type: ndcg_at_5 + value: 70.348 + - type: precision_at_1 + value: 60.0 + - type: precision_at_10 + value: 9.9 + - type: precision_at_100 + value: 1.093 + - type: precision_at_1000 + value: 0.11299999999999999 + - type: precision_at_20 + value: 5.167 + - type: precision_at_3 + value: 26.556 + - type: precision_at_5 + value: 17.933 + - type: recall_at_1 + value: 56.983 + - type: recall_at_10 + value: 86.989 + - type: recall_at_100 + value: 96.333 + - type: recall_at_1000 + value: 99.667 + - type: recall_at_20 + value: 90.889 + - type: recall_at_3 + value: 71.989 + - type: recall_at_5 + value: 79.578 + task: + type: Retrieval + - dataset: + config: default + name: MTEB SprintDuplicateQuestions + revision: d66bd1f72af766a5cc4b0ca5e00c162f89e8cc46 + split: test + type: mteb/sprintduplicatequestions-pairclassification + metrics: + - type: cosine_accuracy + value: 99.73861386138614 + - type: cosine_accuracy_threshold + value: 90.32489657402039 + - type: cosine_ap + value: 92.64599869853095 + - type: cosine_f1 + value: 86.43371017471738 + - type: cosine_f1_threshold + value: 90.2580976486206 + - type: cosine_precision + value: 88.90063424947145 + - type: cosine_recall + value: 84.1 + - type: dot_accuracy + value: 99.73861386138614 + - type: dot_accuracy_threshold + value: 90.32490253448486 + - type: dot_ap + value: 92.64595539427177 + - type: dot_f1 + value: 86.43371017471738 + - type: dot_f1_threshold + value: 90.25810956954956 + - type: dot_precision + value: 88.90063424947145 + - type: dot_recall + value: 84.1 + - type: euclidean_accuracy + value: 99.73861386138614 + - type: euclidean_accuracy_threshold + value: 43.988871574401855 + - type: euclidean_ap + value: 92.64599869853096 + - type: euclidean_f1 + value: 86.43371017471738 + - type: euclidean_f1_threshold + value: 44.14045214653015 + - type: euclidean_precision + value: 88.90063424947145 + - type: euclidean_recall + value: 84.1 + - type: main_score + value: 93.05685558073557 + - type: manhattan_accuracy + value: 99.74752475247524 + - type: manhattan_accuracy_threshold + value: 1051.1520385742188 + - type: manhattan_ap + value: 93.05685558073557 + - type: manhattan_f1 + value: 87.06240487062405 + - type: manhattan_f1_threshold + value: 1051.5135765075684 + - type: manhattan_precision + value: 88.36251287332647 + - type: manhattan_recall + value: 85.8 + - type: max_ap + value: 93.05685558073557 + - type: max_f1 + value: 87.06240487062405 + - type: max_precision + value: 88.90063424947145 + - type: max_recall + value: 85.8 + - type: similarity_accuracy + value: 99.73861386138614 + - type: similarity_accuracy_threshold + value: 90.32489657402039 + - type: similarity_ap + value: 92.64599869853095 + - type: similarity_f1 + value: 86.43371017471738 + - type: similarity_f1_threshold + value: 90.2580976486206 + - type: similarity_precision + value: 88.90063424947145 + - type: similarity_recall + value: 84.1 + task: + type: PairClassification + - dataset: + config: default + name: MTEB StackExchangeClustering + revision: 6cbc1f7b2bc0622f2e39d2c77fa502909748c259 + split: test + type: mteb/stackexchange-clustering + metrics: + - type: main_score + value: 66.37519604053632 + - type: v_measure + value: 66.37519604053632 + - type: v_measure_std + value: 3.410007122108013 + task: + type: Clustering + - dataset: + config: default + name: MTEB StackExchangeClusteringP2P + revision: 815ca46b2622cec33ccafc3735d572c266efdb44 + split: test + type: mteb/stackexchange-clustering-p2p + metrics: + - type: main_score + value: 39.19275669318283 + - type: v_measure + value: 39.19275669318283 + - type: v_measure_std + value: 1.433472639700171 + task: + type: Clustering + - dataset: + config: default + name: MTEB StackOverflowDupQuestions + revision: e185fbe320c72810689fc5848eb6114e1ef5ec69 + split: test + type: mteb/stackoverflowdupquestions-reranking + metrics: + - type: main_score + value: 48.37677316210566 + - type: map + value: 48.37677316210566 + - type: mrr + value: 49.172223813767935 + - type: nAUC_map_diff1 + value: 37.067832495829684 + - type: nAUC_map_max + value: 12.566172393440276 + - type: nAUC_map_std + value: 9.594421406244894 + - type: nAUC_mrr_diff1 + value: 37.721801475851414 + - type: nAUC_mrr_max + value: 13.229276487693506 + - type: nAUC_mrr_std + value: 10.265502733168592 + task: + type: Reranking + - dataset: + config: default + name: MTEB SummEval + revision: cda12ad7615edc362dbf25a00fdd61d3b1eaf93c + split: test + type: mteb/summeval + metrics: + - type: cosine_pearson + value: 24.294425360602 + - type: cosine_spearman + value: 25.227349196483146 + - type: dot_pearson + value: 24.294425689673897 + - type: dot_spearman + value: 25.22595105524031 + - type: main_score + value: 25.227349196483146 + - type: pearson + value: 24.294425360602 + - type: spearman + value: 25.227349196483146 + task: + type: Summarization + - dataset: + config: default + name: MTEB TRECCOVID + revision: bb9466bac8153a0349341eb1b22e06409e78ef4e + split: test + type: mteb/trec-covid + metrics: + - type: main_score + value: 83.71600000000001 + - type: map_at_1 + value: 0.22499999999999998 + - type: map_at_10 + value: 2.149 + - type: map_at_100 + value: 14.036999999999999 + - type: map_at_1000 + value: 33.592 + - type: map_at_20 + value: 4.077999999999999 + - type: map_at_3 + value: 0.697 + - type: map_at_5 + value: 1.123 + - type: mrr_at_1 + value: 88.0 + - type: mrr_at_10 + value: 93.5 + - type: mrr_at_100 + value: 93.5 + - type: mrr_at_1000 + value: 93.5 + - type: mrr_at_20 + value: 93.5 + - type: mrr_at_3 + value: 93.0 + - type: mrr_at_5 + value: 93.5 + - type: nauc_map_at_1000_diff1 + value: -37.74019405875251 + - type: nauc_map_at_1000_max + value: 43.36659357895312 + - type: nauc_map_at_1000_std + value: 78.15639459263728 + - type: nauc_map_at_100_diff1 + value: -30.01165990846415 + - type: nauc_map_at_100_max + value: 19.246430352246726 + - type: nauc_map_at_100_std + value: 48.21182298702515 + - type: nauc_map_at_10_diff1 + value: -9.760408372906864 + - type: nauc_map_at_10_max + value: 0.7336574098845274 + - type: nauc_map_at_10_std + value: 11.384489325092082 + - type: nauc_map_at_1_diff1 + value: 5.032274247676729 + - type: nauc_map_at_1_max + value: -4.617648500024458 + - type: nauc_map_at_1_std + value: 8.469833959848321 + - type: nauc_map_at_20_diff1 + value: -18.752371978248327 + - type: nauc_map_at_20_max + value: 2.3046738344836064 + - type: nauc_map_at_20_std + value: 15.833810469532736 + - type: nauc_map_at_3_diff1 + value: -2.282653784955422 + - type: nauc_map_at_3_max + value: -5.94327326591755 + - type: nauc_map_at_3_std + value: 6.684405011691402 + - type: nauc_map_at_5_diff1 + value: -9.350967821325415 + - type: nauc_map_at_5_max + value: -2.1321331142366886 + - type: nauc_map_at_5_std + value: 9.63344735168679 + - type: nauc_mrr_at_1000_diff1 + value: 19.244301106794655 + - type: nauc_mrr_at_1000_max + value: 38.04515135967145 + - type: nauc_mrr_at_1000_std + value: 67.42651909404088 + - type: nauc_mrr_at_100_diff1 + value: 19.244301106794655 + - type: nauc_mrr_at_100_max + value: 38.04515135967145 + - type: nauc_mrr_at_100_std + value: 67.42651909404088 + - type: nauc_mrr_at_10_diff1 + value: 19.244301106794655 + - type: nauc_mrr_at_10_max + value: 38.04515135967145 + - type: nauc_mrr_at_10_std + value: 67.42651909404088 + - type: nauc_mrr_at_1_diff1 + value: 30.695195672924008 + - type: nauc_mrr_at_1_max + value: 40.24021635380201 + - type: nauc_mrr_at_1_std + value: 64.65160674514797 + - type: nauc_mrr_at_20_diff1 + value: 19.244301106794655 + - type: nauc_mrr_at_20_max + value: 38.04515135967145 + - type: nauc_mrr_at_20_std + value: 67.42651909404088 + - type: nauc_mrr_at_3_diff1 + value: 9.460377871414925 + - type: nauc_mrr_at_3_max + value: 36.169634361832266 + - type: nauc_mrr_at_3_std + value: 69.79747179556875 + - type: nauc_mrr_at_5_diff1 + value: 19.244301106794655 + - type: nauc_mrr_at_5_max + value: 38.04515135967145 + - type: nauc_mrr_at_5_std + value: 67.42651909404088 + - type: nauc_ndcg_at_1000_diff1 + value: -39.737437841505525 + - type: nauc_ndcg_at_1000_max + value: 39.588873588822274 + - type: nauc_ndcg_at_1000_std + value: 78.14902788658362 + - type: nauc_ndcg_at_100_diff1 + value: -29.178863578506586 + - type: nauc_ndcg_at_100_max + value: 42.85774941223408 + - type: nauc_ndcg_at_100_std + value: 79.87357453241586 + - type: nauc_ndcg_at_10_diff1 + value: -26.484391516589657 + - type: nauc_ndcg_at_10_max + value: 42.72241600167698 + - type: nauc_ndcg_at_10_std + value: 68.19151094539976 + - type: nauc_ndcg_at_1_diff1 + value: 9.915392124959421 + - type: nauc_ndcg_at_1_max + value: 23.839895867230688 + - type: nauc_ndcg_at_1_std + value: 66.78490074845433 + - type: nauc_ndcg_at_20_diff1 + value: -33.579927248156764 + - type: nauc_ndcg_at_20_max + value: 40.70026877258957 + - type: nauc_ndcg_at_20_std + value: 68.79058594264934 + - type: nauc_ndcg_at_3_diff1 + value: -9.53242863670409 + - type: nauc_ndcg_at_3_max + value: 26.871867846146806 + - type: nauc_ndcg_at_3_std + value: 64.59635427834456 + - type: nauc_ndcg_at_5_diff1 + value: -28.159706027117203 + - type: nauc_ndcg_at_5_max + value: 29.146277434723277 + - type: nauc_ndcg_at_5_std + value: 62.10571075144466 + - type: nauc_precision_at_1000_diff1 + value: -12.337137609580335 + - type: nauc_precision_at_1000_max + value: 37.997646654420365 + - type: nauc_precision_at_1000_std + value: 44.4108831337026 + - type: nauc_precision_at_100_diff1 + value: -27.204587318780927 + - type: nauc_precision_at_100_max + value: 45.85476754361804 + - type: nauc_precision_at_100_std + value: 79.10162967314747 + - type: nauc_precision_at_10_diff1 + value: -19.712113145761485 + - type: nauc_precision_at_10_max + value: 61.12322079696251 + - type: nauc_precision_at_10_std + value: 69.82612276877653 + - type: nauc_precision_at_1_diff1 + value: 30.695195672924008 + - type: nauc_precision_at_1_max + value: 40.24021635380201 + - type: nauc_precision_at_1_std + value: 64.65160674514797 + - type: nauc_precision_at_20_diff1 + value: -35.81580541107032 + - type: nauc_precision_at_20_max + value: 49.85546712252859 + - type: nauc_precision_at_20_std + value: 68.68339783265114 + - type: nauc_precision_at_3_diff1 + value: -14.43795260039528 + - type: nauc_precision_at_3_max + value: 53.1517445687952 + - type: nauc_precision_at_3_std + value: 78.79361421988142 + - type: nauc_precision_at_5_diff1 + value: -23.592835847661018 + - type: nauc_precision_at_5_max + value: 63.607630405057 + - type: nauc_precision_at_5_std + value: 72.10777610903118 + - type: nauc_recall_at_1000_diff1 + value: -38.947759396555625 + - type: nauc_recall_at_1000_max + value: 31.775735742211324 + - type: nauc_recall_at_1000_std + value: 63.77657941191196 + - type: nauc_recall_at_100_diff1 + value: -29.033439414452474 + - type: nauc_recall_at_100_max + value: 6.088477112905269 + - type: nauc_recall_at_100_std + value: 33.644374882067645 + - type: nauc_recall_at_10_diff1 + value: -10.68139060078312 + - type: nauc_recall_at_10_max + value: -7.2806613163056895 + - type: nauc_recall_at_10_std + value: 3.0678840656131996 + - type: nauc_recall_at_1_diff1 + value: 5.032274247676729 + - type: nauc_recall_at_1_max + value: -4.617648500024458 + - type: nauc_recall_at_1_std + value: 8.469833959848321 + - type: nauc_recall_at_20_diff1 + value: -18.071212870636995 + - type: nauc_recall_at_20_max + value: -6.953203874741589 + - type: nauc_recall_at_20_std + value: 6.118400469549319 + - type: nauc_recall_at_3_diff1 + value: -7.0258132415628145 + - type: nauc_recall_at_3_max + value: -11.620061195071214 + - type: nauc_recall_at_3_std + value: 0.46520505953450686 + - type: nauc_recall_at_5_diff1 + value: -12.816778576127488 + - type: nauc_recall_at_5_max + value: -8.490882584106425 + - type: nauc_recall_at_5_std + value: 2.9444123697147924 + - type: ndcg_at_1 + value: 85.0 + - type: ndcg_at_10 + value: 83.71600000000001 + - type: ndcg_at_100 + value: 66.322 + - type: ndcg_at_1000 + value: 58.794000000000004 + - type: ndcg_at_20 + value: 81.339 + - type: ndcg_at_3 + value: 86.827 + - type: ndcg_at_5 + value: 85.372 + - type: precision_at_1 + value: 88.0 + - type: precision_at_10 + value: 88.4 + - type: precision_at_100 + value: 68.89999999999999 + - type: precision_at_1000 + value: 25.913999999999998 + - type: precision_at_20 + value: 86.0 + - type: precision_at_3 + value: 92.0 + - type: precision_at_5 + value: 90.8 + - type: recall_at_1 + value: 0.22499999999999998 + - type: recall_at_10 + value: 2.3120000000000003 + - type: recall_at_100 + value: 16.84 + - type: recall_at_1000 + value: 55.541 + - type: recall_at_20 + value: 4.478 + - type: recall_at_3 + value: 0.735 + - type: recall_at_5 + value: 1.188 + task: + type: Retrieval + - dataset: + config: default + name: MTEB Touche2020 + revision: a34f9a33db75fa0cbb21bb5cfc3dae8dc8bec93f + split: test + type: mteb/touche2020 + metrics: + - type: main_score + value: 29.148000000000003 + - type: map_at_1 + value: 2.9690000000000003 + - type: map_at_10 + value: 12.183 + - type: map_at_100 + value: 18.934 + - type: map_at_1000 + value: 20.657 + - type: map_at_20 + value: 14.995 + - type: map_at_3 + value: 6.261 + - type: map_at_5 + value: 8.463 + - type: mrr_at_1 + value: 40.816326530612244 + - type: mrr_at_10 + value: 54.33916423712341 + - type: mrr_at_100 + value: 55.424228921812166 + - type: mrr_at_1000 + value: 55.424228921812166 + - type: mrr_at_20 + value: 55.115252184263994 + - type: mrr_at_3 + value: 51.360544217687064 + - type: mrr_at_5 + value: 53.707482993197274 + - type: nauc_map_at_1000_diff1 + value: 24.39650474897996 + - type: nauc_map_at_1000_max + value: -24.9951891255025 + - type: nauc_map_at_1000_std + value: 9.605455246028367 + - type: nauc_map_at_100_diff1 + value: 25.536859954016666 + - type: nauc_map_at_100_max + value: -25.804629962474262 + - type: nauc_map_at_100_std + value: 6.269656637665734 + - type: nauc_map_at_10_diff1 + value: 24.85518759986692 + - type: nauc_map_at_10_max + value: -23.813502179044356 + - type: nauc_map_at_10_std + value: -10.511927624839066 + - type: nauc_map_at_1_diff1 + value: 38.988840017170084 + - type: nauc_map_at_1_max + value: -32.91111385240854 + - type: nauc_map_at_1_std + value: -23.593390331872705 + - type: nauc_map_at_20_diff1 + value: 21.90593939858651 + - type: nauc_map_at_20_max + value: -23.88012227409119 + - type: nauc_map_at_20_std + value: -5.8228272625717965 + - type: nauc_map_at_3_diff1 + value: 32.30334028131513 + - type: nauc_map_at_3_max + value: -37.94218032778254 + - type: nauc_map_at_3_std + value: -20.73094293061375 + - type: nauc_map_at_5_diff1 + value: 22.700908074741232 + - type: nauc_map_at_5_max + value: -29.289350038869493 + - type: nauc_map_at_5_std + value: -15.465462985852394 + - type: nauc_mrr_at_1000_diff1 + value: 36.35450280091282 + - type: nauc_mrr_at_1000_max + value: -46.85388076847817 + - type: nauc_mrr_at_1000_std + value: -11.371621140495272 + - type: nauc_mrr_at_100_diff1 + value: 36.35450280091282 + - type: nauc_mrr_at_100_max + value: -46.85388076847817 + - type: nauc_mrr_at_100_std + value: -11.371621140495272 + - type: nauc_mrr_at_10_diff1 + value: 36.49601491862039 + - type: nauc_mrr_at_10_max + value: -48.23156353834522 + - type: nauc_mrr_at_10_std + value: -10.235106950548067 + - type: nauc_mrr_at_1_diff1 + value: 30.580995175260696 + - type: nauc_mrr_at_1_max + value: -41.150802011358095 + - type: nauc_mrr_at_1_std + value: -18.734158460340613 + - type: nauc_mrr_at_20_diff1 + value: 36.71852563079837 + - type: nauc_mrr_at_20_max + value: -47.057569621975745 + - type: nauc_mrr_at_20_std + value: -10.6425202175546 + - type: nauc_mrr_at_3_diff1 + value: 37.11798353023653 + - type: nauc_mrr_at_3_max + value: -41.98131493735321 + - type: nauc_mrr_at_3_std + value: -14.708927784488285 + - type: nauc_mrr_at_5_diff1 + value: 36.89165099595256 + - type: nauc_mrr_at_5_max + value: -46.62294841562043 + - type: nauc_mrr_at_5_std + value: -12.950360469740291 + - type: nauc_ndcg_at_1000_diff1 + value: 31.229868567293867 + - type: nauc_ndcg_at_1000_max + value: -32.96979900127197 + - type: nauc_ndcg_at_1000_std + value: 28.83389826789891 + - type: nauc_ndcg_at_100_diff1 + value: 36.52267538437917 + - type: nauc_ndcg_at_100_max + value: -36.30568539371575 + - type: nauc_ndcg_at_100_std + value: 21.077780005544017 + - type: nauc_ndcg_at_10_diff1 + value: 35.624818340564026 + - type: nauc_ndcg_at_10_max + value: -30.41197929897389 + - type: nauc_ndcg_at_10_std + value: -2.0879378834665285 + - type: nauc_ndcg_at_1_diff1 + value: 26.09620989819088 + - type: nauc_ndcg_at_1_max + value: -36.21961356590305 + - type: nauc_ndcg_at_1_std + value: -16.336654888564233 + - type: nauc_ndcg_at_20_diff1 + value: 33.145204376285356 + - type: nauc_ndcg_at_20_max + value: -31.28329863036936 + - type: nauc_ndcg_at_20_std + value: -0.4725530619650745 + - type: nauc_ndcg_at_3_diff1 + value: 26.677773521665433 + - type: nauc_ndcg_at_3_max + value: -32.429541360857414 + - type: nauc_ndcg_at_3_std + value: -5.50717933012337 + - type: nauc_ndcg_at_5_diff1 + value: 27.43507358400893 + - type: nauc_ndcg_at_5_max + value: -31.930933021675834 + - type: nauc_ndcg_at_5_std + value: -2.223314270488071 + - type: nauc_precision_at_1000_diff1 + value: -27.19395370994479 + - type: nauc_precision_at_1000_max + value: 31.32895786519588 + - type: nauc_precision_at_1000_std + value: 30.385879238372826 + - type: nauc_precision_at_100_diff1 + value: 18.037320798123492 + - type: nauc_precision_at_100_max + value: -12.829019500448858 + - type: nauc_precision_at_100_std + value: 63.7252345522221 + - type: nauc_precision_at_10_diff1 + value: 33.03740695134771 + - type: nauc_precision_at_10_max + value: -28.3418055223018 + - type: nauc_precision_at_10_std + value: 12.399749606566802 + - type: nauc_precision_at_1_diff1 + value: 30.580995175260696 + - type: nauc_precision_at_1_max + value: -41.150802011358095 + - type: nauc_precision_at_1_std + value: -18.734158460340613 + - type: nauc_precision_at_20_diff1 + value: 24.61739751282666 + - type: nauc_precision_at_20_max + value: -26.19505578353661 + - type: nauc_precision_at_20_std + value: 28.082166318953462 + - type: nauc_precision_at_3_diff1 + value: 24.262619325154343 + - type: nauc_precision_at_3_max + value: -35.16939673443783 + - type: nauc_precision_at_3_std + value: -1.5143701562969083 + - type: nauc_precision_at_5_diff1 + value: 22.767486223120752 + - type: nauc_precision_at_5_max + value: -32.39632845953219 + - type: nauc_precision_at_5_std + value: 4.989881878186032 + - type: nauc_recall_at_1000_diff1 + value: -3.2461494312684294 + - type: nauc_recall_at_1000_max + value: -2.112903285457026 + - type: nauc_recall_at_1000_std + value: 70.89954510826574 + - type: nauc_recall_at_100_diff1 + value: 29.802898837494297 + - type: nauc_recall_at_100_max + value: -24.20480944377232 + - type: nauc_recall_at_100_std + value: 35.89183425780783 + - type: nauc_recall_at_10_diff1 + value: 27.388757054959328 + - type: nauc_recall_at_10_max + value: -24.989779334676186 + - type: nauc_recall_at_10_std + value: -9.164641938439107 + - type: nauc_recall_at_1_diff1 + value: 38.988840017170084 + - type: nauc_recall_at_1_max + value: -32.91111385240854 + - type: nauc_recall_at_1_std + value: -23.593390331872705 + - type: nauc_recall_at_20_diff1 + value: 24.6080131937692 + - type: nauc_recall_at_20_max + value: -23.05479953156405 + - type: nauc_recall_at_20_std + value: 0.059402669402089325 + - type: nauc_recall_at_3_diff1 + value: 27.848810363284425 + - type: nauc_recall_at_3_max + value: -36.53123928519583 + - type: nauc_recall_at_3_std + value: -19.758588460277824 + - type: nauc_recall_at_5_diff1 + value: 19.64218420357169 + - type: nauc_recall_at_5_max + value: -29.08402015074908 + - type: nauc_recall_at_5_std + value: -14.506685298292496 + - type: ndcg_at_1 + value: 38.775999999999996 + - type: ndcg_at_10 + value: 29.148000000000003 + - type: ndcg_at_100 + value: 40.945 + - type: ndcg_at_1000 + value: 52.303999999999995 + - type: ndcg_at_20 + value: 29.848000000000003 + - type: ndcg_at_3 + value: 33.222 + - type: ndcg_at_5 + value: 31.175000000000004 + - type: precision_at_1 + value: 40.816 + - type: precision_at_10 + value: 25.509999999999998 + - type: precision_at_100 + value: 8.347 + - type: precision_at_1000 + value: 1.5939999999999999 + - type: precision_at_20 + value: 19.082 + - type: precision_at_3 + value: 34.694 + - type: precision_at_5 + value: 31.019999999999996 + - type: recall_at_1 + value: 2.9690000000000003 + - type: recall_at_10 + value: 18.979 + - type: recall_at_100 + value: 50.932 + - type: recall_at_1000 + value: 85.402 + - type: recall_at_20 + value: 27.116 + - type: recall_at_3 + value: 7.619 + - type: recall_at_5 + value: 11.448 + task: + type: Retrieval + - dataset: + config: default + name: MTEB ToxicConversationsClassification + revision: edfaf9da55d3dd50d43143d90c1ac476895ae6de + split: test + type: mteb/toxic_conversations_50k + metrics: + - type: accuracy + value: 89.27734375 + - type: ap + value: 35.194960239779064 + - type: ap_weighted + value: 35.194960239779064 + - type: f1 + value: 74.3697271257718 + - type: f1_weighted + value: 90.81303890497676 + - type: main_score + value: 89.27734375 + task: + type: Classification + - dataset: + config: default + name: MTEB TweetSentimentExtractionClassification + revision: d604517c81ca91fe16a244d1248fc021f9ecee7a + split: test + type: mteb/tweet_sentiment_extraction + metrics: + - type: accuracy + value: 76.3469156762875 + - type: f1 + value: 76.53667544013393 + - type: f1_weighted + value: 75.88734027230643 + - type: main_score + value: 76.3469156762875 + task: + type: Classification + - dataset: + config: default + name: MTEB TwentyNewsgroupsClustering + revision: 6125ec4e24fa026cec8a478383ee943acfbd5449 + split: test + type: mteb/twentynewsgroups-clustering + metrics: + - type: main_score + value: 49.32687063841142 + - type: v_measure + value: 49.32687063841142 + - type: v_measure_std + value: 1.4415471512453628 + task: + type: Clustering + - dataset: + config: default + name: MTEB TwitterSemEval2015 + revision: 70970daeab8776df92f5ea462b6173c0b46fd2d1 + split: test + type: mteb/twittersemeval2015-pairclassification + metrics: + - type: cosine_accuracy + value: 85.24169994635513 + - type: cosine_accuracy_threshold + value: 91.52344465255737 + - type: cosine_ap + value: 71.43912939014582 + - type: cosine_f1 + value: 66.87586293460525 + - type: cosine_f1_threshold + value: 90.44243097305298 + - type: cosine_precision + value: 63.777830979171654 + - type: cosine_recall + value: 70.29023746701847 + - type: dot_accuracy + value: 85.24169994635513 + - type: dot_accuracy_threshold + value: 91.52344465255737 + - type: dot_ap + value: 71.43910347543935 + - type: dot_f1 + value: 66.87586293460525 + - type: dot_f1_threshold + value: 90.44243693351746 + - type: dot_precision + value: 63.777830979171654 + - type: dot_recall + value: 70.29023746701847 + - type: euclidean_accuracy + value: 85.24169994635513 + - type: euclidean_accuracy_threshold + value: 41.17414355278015 + - type: euclidean_ap + value: 71.43911775197303 + - type: euclidean_f1 + value: 66.87586293460525 + - type: euclidean_f1_threshold + value: 43.72085928916931 + - type: euclidean_precision + value: 63.777830979171654 + - type: euclidean_recall + value: 70.29023746701847 + - type: main_score + value: 71.43912939014582 + - type: manhattan_accuracy + value: 85.02115992132086 + - type: manhattan_accuracy_threshold + value: 970.2783584594727 + - type: manhattan_ap + value: 71.11811615315935 + - type: manhattan_f1 + value: 66.46334024643161 + - type: manhattan_f1_threshold + value: 1044.2398071289062 + - type: manhattan_precision + value: 61.81075561606535 + - type: manhattan_recall + value: 71.87335092348285 + - type: max_ap + value: 71.43912939014582 + - type: max_f1 + value: 66.87586293460525 + - type: max_precision + value: 63.777830979171654 + - type: max_recall + value: 71.87335092348285 + - type: similarity_accuracy + value: 85.24169994635513 + - type: similarity_accuracy_threshold + value: 91.52344465255737 + - type: similarity_ap + value: 71.43912939014582 + - type: similarity_f1 + value: 66.87586293460525 + - type: similarity_f1_threshold + value: 90.44243097305298 + - type: similarity_precision + value: 63.777830979171654 + - type: similarity_recall + value: 70.29023746701847 + task: + type: PairClassification + - dataset: + config: default + name: MTEB TwitterURLCorpus + revision: 8b6510b0b1fa4e4c4f879467980e9be563ec1cdf + split: test + type: mteb/twitterurlcorpus-pairclassification + metrics: + - type: cosine_accuracy + value: 88.85590095859045 + - type: cosine_accuracy_threshold + value: 88.8430118560791 + - type: cosine_ap + value: 85.68756441554032 + - type: cosine_f1 + value: 77.73691444704485 + - type: cosine_f1_threshold + value: 87.81020641326904 + - type: cosine_precision + value: 75.10587897494796 + - type: cosine_recall + value: 80.55897751770866 + - type: dot_accuracy + value: 88.85590095859045 + - type: dot_accuracy_threshold + value: 88.84301781654358 + - type: dot_ap + value: 85.6875209900991 + - type: dot_f1 + value: 77.73691444704485 + - type: dot_f1_threshold + value: 87.810218334198 + - type: dot_precision + value: 75.10587897494796 + - type: dot_recall + value: 80.55897751770866 + - type: euclidean_accuracy + value: 88.85590095859045 + - type: euclidean_accuracy_threshold + value: 47.23767042160034 + - type: euclidean_ap + value: 85.68752572543856 + - type: euclidean_f1 + value: 77.73691444704485 + - type: euclidean_f1_threshold + value: 49.375683069229126 + - type: euclidean_precision + value: 75.10587897494796 + - type: euclidean_recall + value: 80.55897751770866 + - type: main_score + value: 85.68756441554032 + - type: manhattan_accuracy + value: 88.86366282454303 + - type: manhattan_accuracy_threshold + value: 1111.7809295654297 + - type: manhattan_ap + value: 85.65387985046206 + - type: manhattan_f1 + value: 77.74802915365164 + - type: manhattan_f1_threshold + value: 1163.0821228027344 + - type: manhattan_precision + value: 75.18699654775605 + - type: manhattan_recall + value: 80.48968278410841 + - type: max_ap + value: 85.68756441554032 + - type: max_f1 + value: 77.74802915365164 + - type: max_precision + value: 75.18699654775605 + - type: max_recall + value: 80.55897751770866 + - type: similarity_accuracy + value: 88.85590095859045 + - type: similarity_accuracy_threshold + value: 88.8430118560791 + - type: similarity_ap + value: 85.68756441554032 + - type: similarity_f1 + value: 77.73691444704485 + - type: similarity_f1_threshold + value: 87.81020641326904 + - type: similarity_precision + value: 75.10587897494796 + - type: similarity_recall + value: 80.55897751770866 + task: + type: PairClassification + - dataset: + config: default + name: MTEB AFQMC + revision: b44c3b011063adb25877c13823db83bb193913c4 + split: validation + type: C-MTEB/AFQMC + metrics: + - type: cosine_pearson + value: 37.12876239452581 + - type: cosine_spearman + value: 38.067257174211036 + - type: euclidean_pearson + value: 36.735658383168044 + - type: euclidean_spearman + value: 38.06725717137013 + - type: main_score + value: 38.067257174211036 + - type: manhattan_pearson + value: 36.43829602976309 + - type: manhattan_spearman + value: 37.73959740816105 + - type: pearson + value: 37.12876239452581 + - type: spearman + value: 38.067257174211036 + task: + type: STS + - dataset: + config: default + name: MTEB ATEC + revision: 0f319b1142f28d00e055a6770f3f726ae9b7d865 + split: test + type: C-MTEB/ATEC + metrics: + - type: cosine_pearson + value: 45.55151167790002 + - type: cosine_spearman + value: 46.22151486832593 + - type: euclidean_pearson + value: 48.31921364975583 + - type: euclidean_spearman + value: 46.22151824485588 + - type: main_score + value: 46.22151486832593 + - type: manhattan_pearson + value: 48.12062382773407 + - type: manhattan_spearman + value: 46.00780031287469 + - type: pearson + value: 45.55151167790002 + - type: spearman + value: 46.22151486832593 + task: + type: STS + - dataset: + config: zh + name: MTEB AmazonReviewsClassification (zh) + revision: 1399c76144fd37290681b995c656ef9b2e06e26d + split: test + type: mteb/amazon_reviews_multi + metrics: + - type: accuracy + value: 50.978 + - type: f1 + value: 47.04754599162115 + - type: f1_weighted + value: 47.04754599162114 + - type: main_score + value: 50.978 + task: + type: Classification + - dataset: + config: default + name: MTEB BQ + revision: e3dda5e115e487b39ec7e618c0c6a29137052a55 + split: test + type: C-MTEB/BQ + metrics: + - type: cosine_pearson + value: 53.84488943094217 + - type: cosine_spearman + value: 54.49605523234321 + - type: euclidean_pearson + value: 53.50943690170382 + - type: euclidean_spearman + value: 54.49605869546392 + - type: main_score + value: 54.49605523234321 + - type: manhattan_pearson + value: 53.24052011197216 + - type: manhattan_spearman + value: 54.215799016988996 + - type: pearson + value: 53.84488943094217 + - type: spearman + value: 54.49605523234321 + task: + type: STS + - dataset: + config: default + name: MTEB CLSClusteringP2P + revision: 4b6227591c6c1a73bc76b1055f3b7f3588e72476 + split: test + type: C-MTEB/CLSClusteringP2P + metrics: + - type: main_score + value: 46.915444917880286 + - type: v_measure + value: 46.915444917880286 + - type: v_measure_std + value: 1.5837973591025165 + task: + type: Clustering + - dataset: + config: default + name: MTEB CLSClusteringS2S + revision: e458b3f5414b62b7f9f83499ac1f5497ae2e869f + split: test + type: C-MTEB/CLSClusteringS2S + metrics: + - type: main_score + value: 44.89461372862163 + - type: v_measure + value: 44.89461372862163 + - type: v_measure_std + value: 1.0128909727353739 + task: + type: Clustering + - dataset: + config: default + name: MTEB CMedQAv1 + revision: 8d7f1e942507dac42dc58017c1a001c3717da7df + split: test + type: C-MTEB/CMedQAv1-reranking + metrics: + - type: main_score + value: 82.21191216422777 + - type: map + value: 82.21191216422777 + - type: mrr + value: 84.98567460317462 + - type: nAUC_map_diff1 + value: 55.27793122843884 + - type: nAUC_map_max + value: 61.78021845841622 + - type: nAUC_map_std + value: 24.32874017213633 + - type: nAUC_mrr_diff1 + value: 62.56855683834338 + - type: nAUC_mrr_max + value: 69.83242448232605 + - type: nAUC_mrr_std + value: 30.753945346898497 + task: + type: Reranking + - dataset: + config: default + name: MTEB CMedQAv2 + revision: 23d186750531a14a0357ca22cd92d712fd512ea0 + split: test + type: C-MTEB/CMedQAv2-reranking + metrics: + - type: main_score + value: 83.08227189741763 + - type: map + value: 83.08227189741763 + - type: mrr + value: 86.19051587301587 + - type: nAUC_map_diff1 + value: 58.735936259553654 + - type: nAUC_map_max + value: 60.14841462146858 + - type: nAUC_map_std + value: 17.761882943403222 + - type: nAUC_mrr_diff1 + value: 67.43320028439246 + - type: nAUC_mrr_max + value: 70.96971853942028 + - type: nAUC_mrr_std + value: 27.963212156051533 + task: + type: Reranking + - dataset: + config: default + name: MTEB CmedqaRetrieval + revision: cd540c506dae1cf9e9a59c3e06f42030d54e7301 + split: dev + type: C-MTEB/CmedqaRetrieval + metrics: + - type: main_score + value: 42.114000000000004 + - type: map_at_1 + value: 24.712999999999997 + - type: map_at_10 + value: 35.977 + - type: map_at_100 + value: 37.626 + - type: map_at_1000 + value: 37.768 + - type: map_at_20 + value: 36.855 + - type: map_at_3 + value: 32.263 + - type: map_at_5 + value: 34.458 + - type: mrr_at_1 + value: 37.80945236309077 + - type: mrr_at_10 + value: 45.2176833891012 + - type: mrr_at_100 + value: 46.120647602744036 + - type: mrr_at_1000 + value: 46.18072259198463 + - type: mrr_at_20 + value: 45.733062196151344 + - type: mrr_at_3 + value: 42.95657247645238 + - type: mrr_at_5 + value: 44.2944069350671 + - type: nauc_map_at_1000_diff1 + value: 47.73425387510818 + - type: nauc_map_at_1000_max + value: 35.955688760781 + - type: nauc_map_at_1000_std + value: -5.337048907024208 + - type: nauc_map_at_100_diff1 + value: 47.691721439486415 + - type: nauc_map_at_100_max + value: 35.9120286982508 + - type: nauc_map_at_100_std + value: -5.388715572820414 + - type: nauc_map_at_10_diff1 + value: 47.59595769097994 + - type: nauc_map_at_10_max + value: 34.868575436637464 + - type: nauc_map_at_10_std + value: -6.38635344513967 + - type: nauc_map_at_1_diff1 + value: 51.697402901732794 + - type: nauc_map_at_1_max + value: 27.169028835962006 + - type: nauc_map_at_1_std + value: -8.36280973549372 + - type: nauc_map_at_20_diff1 + value: 47.6021479339286 + - type: nauc_map_at_20_max + value: 35.42358169640706 + - type: nauc_map_at_20_std + value: -5.995707059550167 + - type: nauc_map_at_3_diff1 + value: 48.12995337761301 + - type: nauc_map_at_3_max + value: 32.29594600680629 + - type: nauc_map_at_3_std + value: -7.995165343021071 + - type: nauc_map_at_5_diff1 + value: 47.97666743341936 + - type: nauc_map_at_5_max + value: 33.98865164184045 + - type: nauc_map_at_5_std + value: -7.202624274535322 + - type: nauc_mrr_at_1000_diff1 + value: 53.91400893930025 + - type: nauc_mrr_at_1000_max + value: 42.756637412490015 + - type: nauc_mrr_at_1000_std + value: -1.7033054319306657 + - type: nauc_mrr_at_100_diff1 + value: 53.88165315274917 + - type: nauc_mrr_at_100_max + value: 42.751083094268274 + - type: nauc_mrr_at_100_std + value: -1.6918871915944649 + - type: nauc_mrr_at_10_diff1 + value: 53.91581744122965 + - type: nauc_mrr_at_10_max + value: 42.58709846134356 + - type: nauc_mrr_at_10_std + value: -1.9103231104709821 + - type: nauc_mrr_at_1_diff1 + value: 59.37942921389244 + - type: nauc_mrr_at_1_max + value: 43.889994340792285 + - type: nauc_mrr_at_1_std + value: -2.9334150073957 + - type: nauc_mrr_at_20_diff1 + value: 53.82364054543749 + - type: nauc_mrr_at_20_max + value: 42.657132121329866 + - type: nauc_mrr_at_20_std + value: -1.814080881374272 + - type: nauc_mrr_at_3_diff1 + value: 54.82194831261354 + - type: nauc_mrr_at_3_max + value: 42.867175235515795 + - type: nauc_mrr_at_3_std + value: -2.3672343827332343 + - type: nauc_mrr_at_5_diff1 + value: 54.28110236719863 + - type: nauc_mrr_at_5_max + value: 42.773659727997604 + - type: nauc_mrr_at_5_std + value: -1.9843150717670213 + - type: nauc_ndcg_at_1000_diff1 + value: 48.08750394739328 + - type: nauc_ndcg_at_1000_max + value: 39.354002076766356 + - type: nauc_ndcg_at_1000_std + value: -1.642209870956442 + - type: nauc_ndcg_at_100_diff1 + value: 47.02373205823187 + - type: nauc_ndcg_at_100_max + value: 39.071767544616016 + - type: nauc_ndcg_at_100_std + value: -1.4318887896374275 + - type: nauc_ndcg_at_10_diff1 + value: 46.87201101025155 + - type: nauc_ndcg_at_10_max + value: 36.596107252610715 + - type: nauc_ndcg_at_10_std + value: -4.491855380124351 + - type: nauc_ndcg_at_1_diff1 + value: 59.37942921389244 + - type: nauc_ndcg_at_1_max + value: 43.889994340792285 + - type: nauc_ndcg_at_1_std + value: -2.9334150073957 + - type: nauc_ndcg_at_20_diff1 + value: 46.69383079853486 + - type: nauc_ndcg_at_20_max + value: 37.42651190472187 + - type: nauc_ndcg_at_20_std + value: -3.8283649672503555 + - type: nauc_ndcg_at_3_diff1 + value: 48.767394875248215 + - type: nauc_ndcg_at_3_max + value: 37.896558640830364 + - type: nauc_ndcg_at_3_std + value: -4.549323419648498 + - type: nauc_ndcg_at_5_diff1 + value: 47.86427501476368 + - type: nauc_ndcg_at_5_max + value: 36.911634117836165 + - type: nauc_ndcg_at_5_std + value: -4.724216711512233 + - type: nauc_precision_at_1000_diff1 + value: 5.215840982527869 + - type: nauc_precision_at_1000_max + value: 29.159568859241432 + - type: nauc_precision_at_1000_std + value: 21.082099239516065 + - type: nauc_precision_at_100_diff1 + value: 11.484383878054516 + - type: nauc_precision_at_100_max + value: 36.979612782453096 + - type: nauc_precision_at_100_std + value: 21.37719349152994 + - type: nauc_precision_at_10_diff1 + value: 25.43246921524832 + - type: nauc_precision_at_10_max + value: 42.19035665238435 + - type: nauc_precision_at_10_std + value: 9.100059364841798 + - type: nauc_precision_at_1_diff1 + value: 59.37942921389244 + - type: nauc_precision_at_1_max + value: 43.889994340792285 + - type: nauc_precision_at_1_std + value: -2.9334150073957 + - type: nauc_precision_at_20_diff1 + value: 20.71145394651256 + - type: nauc_precision_at_20_max + value: 40.864109895424676 + - type: nauc_precision_at_20_std + value: 12.37767593665739 + - type: nauc_precision_at_3_diff1 + value: 36.796679107157864 + - type: nauc_precision_at_3_max + value: 43.86084233138964 + - type: nauc_precision_at_3_std + value: 2.284927565289776 + - type: nauc_precision_at_5_diff1 + value: 31.736303229558853 + - type: nauc_precision_at_5_max + value: 43.979769552220695 + - type: nauc_precision_at_5_std + value: 4.953476938204816 + - type: nauc_recall_at_1000_diff1 + value: 33.99656164674807 + - type: nauc_recall_at_1000_max + value: 51.409315146611604 + - type: nauc_recall_at_1000_std + value: 46.40969047509407 + - type: nauc_recall_at_100_diff1 + value: 28.008400601965988 + - type: nauc_recall_at_100_max + value: 35.45902334586444 + - type: nauc_recall_at_100_std + value: 12.00354716277477 + - type: nauc_recall_at_10_diff1 + value: 35.11874174482894 + - type: nauc_recall_at_10_max + value: 29.383597829116265 + - type: nauc_recall_at_10_std + value: -3.990278366414269 + - type: nauc_recall_at_1_diff1 + value: 51.697402901732794 + - type: nauc_recall_at_1_max + value: 27.169028835962006 + - type: nauc_recall_at_1_std + value: -8.36280973549372 + - type: nauc_recall_at_20_diff1 + value: 32.85706970308098 + - type: nauc_recall_at_20_max + value: 30.609762001249297 + - type: nauc_recall_at_20_std + value: -2.0802705155631305 + - type: nauc_recall_at_3_diff1 + value: 41.293789470043485 + - type: nauc_recall_at_3_max + value: 29.091549134985883 + - type: nauc_recall_at_3_std + value: -7.863274012056401 + - type: nauc_recall_at_5_diff1 + value: 38.87911660795755 + - type: nauc_recall_at_5_max + value: 29.81974055210633 + - type: nauc_recall_at_5_std + value: -5.940913211156581 + - type: ndcg_at_1 + value: 37.809 + - type: ndcg_at_10 + value: 42.114000000000004 + - type: ndcg_at_100 + value: 48.76 + - type: ndcg_at_1000 + value: 51.41 + - type: ndcg_at_20 + value: 44.528 + - type: ndcg_at_3 + value: 37.621 + - type: ndcg_at_5 + value: 39.627 + - type: precision_at_1 + value: 37.809 + - type: precision_at_10 + value: 9.215 + - type: precision_at_100 + value: 1.468 + - type: precision_at_1000 + value: 0.18 + - type: precision_at_20 + value: 5.423 + - type: precision_at_3 + value: 21.214 + - type: precision_at_5 + value: 15.323999999999998 + - type: recall_at_1 + value: 24.712999999999997 + - type: recall_at_10 + value: 51.08200000000001 + - type: recall_at_100 + value: 78.855 + - type: recall_at_1000 + value: 96.897 + - type: recall_at_20 + value: 59.232 + - type: recall_at_3 + value: 37.147000000000006 + - type: recall_at_5 + value: 43.579 + task: + type: Retrieval + - dataset: + config: default + name: MTEB Cmnli + revision: 41bc36f332156f7adc9e38f53777c959b2ae9766 + split: validation + type: C-MTEB/CMNLI + metrics: + - type: cosine_accuracy + value: 69.70535177390259 + - type: cosine_accuracy_threshold + value: 85.75683832168579 + - type: cosine_ap + value: 76.66113306358733 + - type: cosine_f1 + value: 72.686230248307 + - type: cosine_f1_threshold + value: 82.12493658065796 + - type: cosine_precision + value: 62.63531799729364 + - type: cosine_recall + value: 86.57937806873977 + - type: dot_accuracy + value: 69.70535177390259 + - type: dot_accuracy_threshold + value: 85.75684428215027 + - type: dot_ap + value: 76.68137107496162 + - type: dot_f1 + value: 72.686230248307 + - type: dot_f1_threshold + value: 82.12493658065796 + - type: dot_precision + value: 62.63531799729364 + - type: dot_recall + value: 86.57937806873977 + - type: euclidean_accuracy + value: 69.70535177390259 + - type: euclidean_accuracy_threshold + value: 53.37257385253906 + - type: euclidean_ap + value: 76.6611302634097 + - type: euclidean_f1 + value: 72.686230248307 + - type: euclidean_f1_threshold + value: 59.79139804840088 + - type: euclidean_precision + value: 62.63531799729364 + - type: euclidean_recall + value: 86.57937806873977 + - type: main_score + value: 76.68137107496162 + - type: manhattan_accuracy + value: 69.27239927841251 + - type: manhattan_accuracy_threshold + value: 1262.4760627746582 + - type: manhattan_ap + value: 76.1926237206237 + - type: manhattan_f1 + value: 72.39749155812832 + - type: manhattan_f1_threshold + value: 1421.414852142334 + - type: manhattan_precision + value: 61.62943495400789 + - type: manhattan_recall + value: 87.72504091653029 + - type: max_ap + value: 76.68137107496162 + - type: max_f1 + value: 72.686230248307 + - type: max_precision + value: 62.63531799729364 + - type: max_recall + value: 87.72504091653029 + - type: similarity_accuracy + value: 69.70535177390259 + - type: similarity_accuracy_threshold + value: 85.75683832168579 + - type: similarity_ap + value: 76.66113306358733 + - type: similarity_f1 + value: 72.686230248307 + - type: similarity_f1_threshold + value: 82.12493658065796 + - type: similarity_precision + value: 62.63531799729364 + - type: similarity_recall + value: 86.57937806873977 + task: + type: PairClassification + - dataset: + config: default + name: MTEB CovidRetrieval + revision: 1271c7809071a13532e05f25fb53511ffce77117 + split: dev + type: C-MTEB/CovidRetrieval + metrics: + - type: main_score + value: 82.311 + - type: map_at_1 + value: 69.679 + - type: map_at_10 + value: 78.348 + - type: map_at_100 + value: 78.596 + - type: map_at_1000 + value: 78.601 + - type: map_at_20 + value: 78.545 + - type: map_at_3 + value: 76.387 + - type: map_at_5 + value: 77.655 + - type: mrr_at_1 + value: 69.86301369863014 + - type: mrr_at_10 + value: 78.34249084249088 + - type: mrr_at_100 + value: 78.59092165386024 + - type: mrr_at_1000 + value: 78.59525747388338 + - type: mrr_at_20 + value: 78.53925752890612 + - type: mrr_at_3 + value: 76.46645591851072 + - type: mrr_at_5 + value: 77.68879522304182 + - type: nauc_map_at_1000_diff1 + value: 79.4462259252306 + - type: nauc_map_at_1000_max + value: 31.7925049828458 + - type: nauc_map_at_1000_std + value: -44.987963962793955 + - type: nauc_map_at_100_diff1 + value: 79.44473843692927 + - type: nauc_map_at_100_max + value: 31.798193923558003 + - type: nauc_map_at_100_std + value: -44.98700550161243 + - type: nauc_map_at_10_diff1 + value: 79.33651124191677 + - type: nauc_map_at_10_max + value: 31.67512594432897 + - type: nauc_map_at_10_std + value: -45.317005211451914 + - type: nauc_map_at_1_diff1 + value: 81.69181674577305 + - type: nauc_map_at_1_max + value: 30.817373790682357 + - type: nauc_map_at_1_std + value: -42.72748236505566 + - type: nauc_map_at_20_diff1 + value: 79.41582541979186 + - type: nauc_map_at_20_max + value: 31.88246428726322 + - type: nauc_map_at_20_std + value: -44.99284598367759 + - type: nauc_map_at_3_diff1 + value: 79.32027545323751 + - type: nauc_map_at_3_max + value: 31.27336204905954 + - type: nauc_map_at_3_std + value: -46.25198460252002 + - type: nauc_map_at_5_diff1 + value: 79.13580536546466 + - type: nauc_map_at_5_max + value: 31.341889475033714 + - type: nauc_map_at_5_std + value: -46.04029107468536 + - type: nauc_mrr_at_1000_diff1 + value: 79.44015632094633 + - type: nauc_mrr_at_1000_max + value: 32.21963353786539 + - type: nauc_mrr_at_1000_std + value: -44.52548111878723 + - type: nauc_mrr_at_100_diff1 + value: 79.43867097380709 + - type: nauc_mrr_at_100_max + value: 32.22522235955093 + - type: nauc_mrr_at_100_std + value: -44.52462840708672 + - type: nauc_mrr_at_10_diff1 + value: 79.33057644986008 + - type: nauc_mrr_at_10_max + value: 32.09671935370013 + - type: nauc_mrr_at_10_std + value: -44.860413851737526 + - type: nauc_mrr_at_1_diff1 + value: 81.44692749809782 + - type: nauc_mrr_at_1_max + value: 31.849651568624132 + - type: nauc_mrr_at_1_std + value: -41.3253061502398 + - type: nauc_mrr_at_20_diff1 + value: 79.40978909435384 + - type: nauc_mrr_at_20_max + value: 32.3082948412038 + - type: nauc_mrr_at_20_std + value: -44.5317164768648 + - type: nauc_mrr_at_3_diff1 + value: 79.1638266419462 + - type: nauc_mrr_at_3_max + value: 32.0842169384742 + - type: nauc_mrr_at_3_std + value: -45.46170783914986 + - type: nauc_mrr_at_5_diff1 + value: 79.05732925048218 + - type: nauc_mrr_at_5_max + value: 31.866875109079274 + - type: nauc_mrr_at_5_std + value: -45.47315535458264 + - type: nauc_ndcg_at_1000_diff1 + value: 79.14506487967446 + - type: nauc_ndcg_at_1000_max + value: 32.28815962385816 + - type: nauc_ndcg_at_1000_std + value: -44.84164255319649 + - type: nauc_ndcg_at_100_diff1 + value: 79.11589619449164 + - type: nauc_ndcg_at_100_max + value: 32.48358720627568 + - type: nauc_ndcg_at_100_std + value: -44.77161186390409 + - type: nauc_ndcg_at_10_diff1 + value: 78.6112419611499 + - type: nauc_ndcg_at_10_max + value: 32.02532602435404 + - type: nauc_ndcg_at_10_std + value: -46.22564777302159 + - type: nauc_ndcg_at_1_diff1 + value: 81.224021579675 + - type: nauc_ndcg_at_1_max + value: 31.775986317223122 + - type: nauc_ndcg_at_1_std + value: -41.46817150641969 + - type: nauc_ndcg_at_20_diff1 + value: 78.91451933721699 + - type: nauc_ndcg_at_20_max + value: 33.07095994019382 + - type: nauc_ndcg_at_20_std + value: -44.793080774799556 + - type: nauc_ndcg_at_3_diff1 + value: 78.45748790579262 + - type: nauc_ndcg_at_3_max + value: 31.33438980452425 + - type: nauc_ndcg_at_3_std + value: -47.961287399778165 + - type: nauc_ndcg_at_5_diff1 + value: 78.00517989020184 + - type: nauc_ndcg_at_5_max + value: 31.377723112970337 + - type: nauc_ndcg_at_5_std + value: -47.64749257136376 + - type: nauc_precision_at_1000_diff1 + value: -36.954287113132764 + - type: nauc_precision_at_1000_max + value: 33.399149331718306 + - type: nauc_precision_at_1000_std + value: 64.36619497853745 + - type: nauc_precision_at_100_diff1 + value: -1.1165501851098696 + - type: nauc_precision_at_100_max + value: 40.744208492042645 + - type: nauc_precision_at_100_std + value: 32.59574168316801 + - type: nauc_precision_at_10_diff1 + value: 41.66995611089304 + - type: nauc_precision_at_10_max + value: 32.67778815088798 + - type: nauc_precision_at_10_std + value: -25.51047355817746 + - type: nauc_precision_at_1_diff1 + value: 81.224021579675 + - type: nauc_precision_at_1_max + value: 31.775986317223122 + - type: nauc_precision_at_1_std + value: -41.46817150641969 + - type: nauc_precision_at_20_diff1 + value: 24.5912939655554 + - type: nauc_precision_at_20_max + value: 47.02616085142694 + - type: nauc_precision_at_20_std + value: 4.541053315120063 + - type: nauc_precision_at_3_diff1 + value: 70.56985549600088 + - type: nauc_precision_at_3_max + value: 31.221388227405512 + - type: nauc_precision_at_3_std + value: -50.297375067273066 + - type: nauc_precision_at_5_diff1 + value: 59.63106705169415 + - type: nauc_precision_at_5_max + value: 31.785628018433776 + - type: nauc_precision_at_5_std + value: -43.803356847579565 + - type: nauc_recall_at_1000_diff1 + value: 86.10840824375175 + - type: nauc_recall_at_1000_max + value: 86.10840824375175 + - type: nauc_recall_at_1000_std + value: 67.89043659493016 + - type: nauc_recall_at_100_diff1 + value: 77.88282713434944 + - type: nauc_recall_at_100_max + value: 61.93305063016877 + - type: nauc_recall_at_100_std + value: -16.961439349286298 + - type: nauc_recall_at_10_diff1 + value: 72.68211095993148 + - type: nauc_recall_at_10_max + value: 34.46396318256104 + - type: nauc_recall_at_10_std + value: -55.414854482904104 + - type: nauc_recall_at_1_diff1 + value: 81.69181674577305 + - type: nauc_recall_at_1_max + value: 30.817373790682357 + - type: nauc_recall_at_1_std + value: -42.72748236505566 + - type: nauc_recall_at_20_diff1 + value: 73.32466963570535 + - type: nauc_recall_at_20_max + value: 59.70241932930911 + - type: nauc_recall_at_20_std + value: -33.708044953188335 + - type: nauc_recall_at_3_diff1 + value: 75.2135837428305 + - type: nauc_recall_at_3_max + value: 30.365995550867186 + - type: nauc_recall_at_3_std + value: -55.88540648437351 + - type: nauc_recall_at_5_diff1 + value: 71.83388219833638 + - type: nauc_recall_at_5_max + value: 30.57364233205002 + - type: nauc_recall_at_5_std + value: -57.33381514050736 + - type: ndcg_at_1 + value: 69.968 + - type: ndcg_at_10 + value: 82.311 + - type: ndcg_at_100 + value: 83.325 + - type: ndcg_at_1000 + value: 83.444 + - type: ndcg_at_20 + value: 82.99499999999999 + - type: ndcg_at_3 + value: 78.506 + - type: ndcg_at_5 + value: 80.735 + - type: precision_at_1 + value: 69.968 + - type: precision_at_10 + value: 9.557 + - type: precision_at_100 + value: 0.9990000000000001 + - type: precision_at_1000 + value: 0.101 + - type: precision_at_20 + value: 4.91 + - type: precision_at_3 + value: 28.310999999999996 + - type: precision_at_5 + value: 18.102999999999998 + - type: recall_at_1 + value: 69.679 + - type: recall_at_10 + value: 94.521 + - type: recall_at_100 + value: 98.84100000000001 + - type: recall_at_1000 + value: 99.789 + - type: recall_at_20 + value: 97.155 + - type: recall_at_3 + value: 84.48400000000001 + - type: recall_at_5 + value: 89.831 + task: + type: Retrieval + - dataset: + config: default + name: MTEB DuRetrieval + revision: a1a333e290fe30b10f3f56498e3a0d911a693ced + split: dev + type: C-MTEB/DuRetrieval + metrics: + - type: main_score + value: 82.19800000000001 + - type: map_at_1 + value: 23.543 + - type: map_at_10 + value: 73.175 + - type: map_at_100 + value: 76.508 + - type: map_at_1000 + value: 76.556 + - type: map_at_20 + value: 75.74 + - type: map_at_3 + value: 49.364999999999995 + - type: map_at_5 + value: 62.914 + - type: mrr_at_1 + value: 83.35000000000001 + - type: mrr_at_10 + value: 88.91954365079361 + - type: mrr_at_100 + value: 89.00500192836907 + - type: mrr_at_1000 + value: 89.00778867189919 + - type: mrr_at_20 + value: 88.97223997708514 + - type: mrr_at_3 + value: 88.41666666666661 + - type: mrr_at_5 + value: 88.75916666666663 + - type: nauc_map_at_1000_diff1 + value: 1.7634218677860924 + - type: nauc_map_at_1000_max + value: 36.542761970352075 + - type: nauc_map_at_1000_std + value: 16.743999797731096 + - type: nauc_map_at_100_diff1 + value: 1.7302879721981035 + - type: nauc_map_at_100_max + value: 36.51751309286087 + - type: nauc_map_at_100_std + value: 16.714947992819315 + - type: nauc_map_at_10_diff1 + value: 4.58612138353187 + - type: nauc_map_at_10_max + value: 32.412088546560376 + - type: nauc_map_at_10_std + value: 6.602612093174039 + - type: nauc_map_at_1_diff1 + value: 40.92916047850798 + - type: nauc_map_at_1_max + value: -12.132990973344784 + - type: nauc_map_at_1_std + value: -28.916379676602716 + - type: nauc_map_at_20_diff1 + value: 2.3048278132160465 + - type: nauc_map_at_20_max + value: 35.98653993769816 + - type: nauc_map_at_20_std + value: 14.547155799726797 + - type: nauc_map_at_3_diff1 + value: 25.02876377684228 + - type: nauc_map_at_3_max + value: 2.2091937735006324 + - type: nauc_map_at_3_std + value: -22.118418323376787 + - type: nauc_map_at_5_diff1 + value: 15.251921354667381 + - type: nauc_map_at_5_max + value: 15.154235785462586 + - type: nauc_map_at_5_std + value: -12.229180678598413 + - type: nauc_mrr_at_1000_diff1 + value: 20.11404624924483 + - type: nauc_mrr_at_1000_max + value: 50.30723314651651 + - type: nauc_mrr_at_1000_std + value: 24.38243778856984 + - type: nauc_mrr_at_100_diff1 + value: 20.096139168665133 + - type: nauc_mrr_at_100_max + value: 50.30903859418512 + - type: nauc_mrr_at_100_std + value: 24.390286154308622 + - type: nauc_mrr_at_10_diff1 + value: 20.181186231780917 + - type: nauc_mrr_at_10_max + value: 50.58384362961475 + - type: nauc_mrr_at_10_std + value: 24.64009275793392 + - type: nauc_mrr_at_1_diff1 + value: 20.447721013067195 + - type: nauc_mrr_at_1_max + value: 42.25955038562464 + - type: nauc_mrr_at_1_std + value: 16.11340056741704 + - type: nauc_mrr_at_20_diff1 + value: 20.131733240741127 + - type: nauc_mrr_at_20_max + value: 50.43059807829098 + - type: nauc_mrr_at_20_std + value: 24.515830791430787 + - type: nauc_mrr_at_3_diff1 + value: 20.142584954251564 + - type: nauc_mrr_at_3_max + value: 50.982426386940496 + - type: nauc_mrr_at_3_std + value: 24.855042119442096 + - type: nauc_mrr_at_5_diff1 + value: 20.55194083518438 + - type: nauc_mrr_at_5_max + value: 51.01767921199396 + - type: nauc_mrr_at_5_std + value: 25.06454257772412 + - type: nauc_ndcg_at_1000_diff1 + value: 3.308799593775925 + - type: nauc_ndcg_at_1000_max + value: 45.3000272962289 + - type: nauc_ndcg_at_1000_std + value: 26.858211491434762 + - type: nauc_ndcg_at_100_diff1 + value: 2.4769775243940577 + - type: nauc_ndcg_at_100_max + value: 45.06873331760543 + - type: nauc_ndcg_at_100_std + value: 27.149960146479792 + - type: nauc_ndcg_at_10_diff1 + value: 3.053577467938038 + - type: nauc_ndcg_at_10_max + value: 40.86513789655222 + - type: nauc_ndcg_at_10_std + value: 18.89357502028107 + - type: nauc_ndcg_at_1_diff1 + value: 20.447721013067195 + - type: nauc_ndcg_at_1_max + value: 42.25955038562464 + - type: nauc_ndcg_at_1_std + value: 16.11340056741704 + - type: nauc_ndcg_at_20_diff1 + value: 2.9037109510330734 + - type: nauc_ndcg_at_20_max + value: 44.2247958844392 + - type: nauc_ndcg_at_20_std + value: 23.926393325465583 + - type: nauc_ndcg_at_3_diff1 + value: 4.6204987018531565 + - type: nauc_ndcg_at_3_max + value: 38.4238367882706 + - type: nauc_ndcg_at_3_std + value: 18.41509221702182 + - type: nauc_ndcg_at_5_diff1 + value: 5.061035908501401 + - type: nauc_ndcg_at_5_max + value: 35.23535851703071 + - type: nauc_ndcg_at_5_std + value: 13.77450410228784 + - type: nauc_precision_at_1000_diff1 + value: -27.983106250756506 + - type: nauc_precision_at_1000_max + value: 25.86365534534841 + - type: nauc_precision_at_1000_std + value: 48.74540499812263 + - type: nauc_precision_at_100_diff1 + value: -29.106950173691263 + - type: nauc_precision_at_100_max + value: 27.77159360788603 + - type: nauc_precision_at_100_std + value: 50.73074846170649 + - type: nauc_precision_at_10_diff1 + value: -31.36561529132309 + - type: nauc_precision_at_10_max + value: 38.634418900568605 + - type: nauc_precision_at_10_std + value: 47.64589458589123 + - type: nauc_precision_at_1_diff1 + value: 20.447721013067195 + - type: nauc_precision_at_1_max + value: 42.25955038562464 + - type: nauc_precision_at_1_std + value: 16.11340056741704 + - type: nauc_precision_at_20_diff1 + value: -29.915732218413915 + - type: nauc_precision_at_20_max + value: 32.5978721872368 + - type: nauc_precision_at_20_std + value: 50.40163344959172 + - type: nauc_precision_at_3_diff1 + value: -21.275233009079773 + - type: nauc_precision_at_3_max + value: 43.618485232731345 + - type: nauc_precision_at_3_std + value: 33.897688883129895 + - type: nauc_precision_at_5_diff1 + value: -27.95940970772729 + - type: nauc_precision_at_5_max + value: 42.061006167588815 + - type: nauc_precision_at_5_std + value: 39.63220834267178 + - type: nauc_recall_at_1000_diff1 + value: -26.449058870321217 + - type: nauc_recall_at_1000_max + value: 73.0687821403061 + - type: nauc_recall_at_1000_std + value: 77.88067966296173 + - type: nauc_recall_at_100_diff1 + value: -24.90707443234635 + - type: nauc_recall_at_100_max + value: 50.66639373783482 + - type: nauc_recall_at_100_std + value: 54.350058605469755 + - type: nauc_recall_at_10_diff1 + value: 0.8584890040490949 + - type: nauc_recall_at_10_max + value: 33.75876106382897 + - type: nauc_recall_at_10_std + value: 6.861342565912654 + - type: nauc_recall_at_1_diff1 + value: 40.92916047850798 + - type: nauc_recall_at_1_max + value: -12.132990973344784 + - type: nauc_recall_at_1_std + value: -28.916379676602716 + - type: nauc_recall_at_20_diff1 + value: -5.517998459738173 + - type: nauc_recall_at_20_max + value: 45.13561303546834 + - type: nauc_recall_at_20_std + value: 28.078298519083845 + - type: nauc_recall_at_3_diff1 + value: 23.80910203367763 + - type: nauc_recall_at_3_max + value: -1.5249858361677042 + - type: nauc_recall_at_3_std + value: -24.572258302000353 + - type: nauc_recall_at_5_diff1 + value: 14.615940357425682 + - type: nauc_recall_at_5_max + value: 9.484229775296 + - type: nauc_recall_at_5_std + value: -17.225320352424525 + - type: ndcg_at_1 + value: 83.35000000000001 + - type: ndcg_at_10 + value: 82.19800000000001 + - type: ndcg_at_100 + value: 85.938 + - type: ndcg_at_1000 + value: 86.367 + - type: ndcg_at_20 + value: 84.314 + - type: ndcg_at_3 + value: 79.096 + - type: ndcg_at_5 + value: 78.881 + - type: precision_at_1 + value: 83.35000000000001 + - type: precision_at_10 + value: 40.215 + - type: precision_at_100 + value: 4.765 + - type: precision_at_1000 + value: 0.488 + - type: precision_at_20 + value: 22.29 + - type: precision_at_3 + value: 71.11699999999999 + - type: precision_at_5 + value: 61.029999999999994 + - type: recall_at_1 + value: 23.543 + - type: recall_at_10 + value: 85.03 + - type: recall_at_100 + value: 96.769 + - type: recall_at_1000 + value: 98.904 + - type: recall_at_20 + value: 91.282 + - type: recall_at_3 + value: 52.686 + - type: recall_at_5 + value: 69.625 + task: + type: Retrieval + - dataset: + config: default + name: MTEB EcomRetrieval + revision: 687de13dc7294d6fd9be10c6945f9e8fec8166b9 + split: dev + type: C-MTEB/EcomRetrieval + metrics: + - type: main_score + value: 62.629000000000005 + - type: map_at_1 + value: 46.800000000000004 + - type: map_at_10 + value: 57.311 + - type: map_at_100 + value: 57.933 + - type: map_at_1000 + value: 57.949 + - type: map_at_20 + value: 57.720000000000006 + - type: map_at_3 + value: 54.617000000000004 + - type: map_at_5 + value: 56.152 + - type: mrr_at_1 + value: 46.800000000000004 + - type: mrr_at_10 + value: 57.31142857142856 + - type: mrr_at_100 + value: 57.93340200683009 + - type: mrr_at_1000 + value: 57.94943793755751 + - type: mrr_at_20 + value: 57.71972088540971 + - type: mrr_at_3 + value: 54.61666666666664 + - type: mrr_at_5 + value: 56.151666666666635 + - type: nauc_map_at_1000_diff1 + value: 62.9251523816579 + - type: nauc_map_at_1000_max + value: 23.538214459724426 + - type: nauc_map_at_1000_std + value: -10.382006646813648 + - type: nauc_map_at_100_diff1 + value: 62.91524869388112 + - type: nauc_map_at_100_max + value: 23.546842777928948 + - type: nauc_map_at_100_std + value: -10.377308542639026 + - type: nauc_map_at_10_diff1 + value: 62.95644445268712 + - type: nauc_map_at_10_max + value: 23.50898464915184 + - type: nauc_map_at_10_std + value: -10.680666512130612 + - type: nauc_map_at_1_diff1 + value: 66.44819716543068 + - type: nauc_map_at_1_max + value: 19.717201903767986 + - type: nauc_map_at_1_std + value: -15.336875227297902 + - type: nauc_map_at_20_diff1 + value: 62.89964860093505 + - type: nauc_map_at_20_max + value: 23.591388181751878 + - type: nauc_map_at_20_std + value: -10.439865619968138 + - type: nauc_map_at_3_diff1 + value: 62.86404655953095 + - type: nauc_map_at_3_max + value: 22.376787579749553 + - type: nauc_map_at_3_std + value: -11.663211021980615 + - type: nauc_map_at_5_diff1 + value: 62.790883650254905 + - type: nauc_map_at_5_max + value: 22.718439787051057 + - type: nauc_map_at_5_std + value: -10.782545400900611 + - type: nauc_mrr_at_1000_diff1 + value: 62.9251523816579 + - type: nauc_mrr_at_1000_max + value: 23.538214459724426 + - type: nauc_mrr_at_1000_std + value: -10.382006646813648 + - type: nauc_mrr_at_100_diff1 + value: 62.91524869388112 + - type: nauc_mrr_at_100_max + value: 23.546842777928948 + - type: nauc_mrr_at_100_std + value: -10.377308542639026 + - type: nauc_mrr_at_10_diff1 + value: 62.95644445268712 + - type: nauc_mrr_at_10_max + value: 23.50898464915184 + - type: nauc_mrr_at_10_std + value: -10.680666512130612 + - type: nauc_mrr_at_1_diff1 + value: 66.44819716543068 + - type: nauc_mrr_at_1_max + value: 19.717201903767986 + - type: nauc_mrr_at_1_std + value: -15.336875227297902 + - type: nauc_mrr_at_20_diff1 + value: 62.89964860093505 + - type: nauc_mrr_at_20_max + value: 23.591388181751878 + - type: nauc_mrr_at_20_std + value: -10.439865619968138 + - type: nauc_mrr_at_3_diff1 + value: 62.86404655953095 + - type: nauc_mrr_at_3_max + value: 22.376787579749553 + - type: nauc_mrr_at_3_std + value: -11.663211021980615 + - type: nauc_mrr_at_5_diff1 + value: 62.790883650254905 + - type: nauc_mrr_at_5_max + value: 22.718439787051057 + - type: nauc_mrr_at_5_std + value: -10.782545400900611 + - type: nauc_ndcg_at_1000_diff1 + value: 62.229954931841206 + - type: nauc_ndcg_at_1000_max + value: 26.356327484117426 + - type: nauc_ndcg_at_1000_std + value: -6.826775710969278 + - type: nauc_ndcg_at_100_diff1 + value: 61.95919238401926 + - type: nauc_ndcg_at_100_max + value: 26.6734456691839 + - type: nauc_ndcg_at_100_std + value: -6.57007966783538 + - type: nauc_ndcg_at_10_diff1 + value: 62.01610935227022 + - type: nauc_ndcg_at_10_max + value: 26.303092998272376 + - type: nauc_ndcg_at_10_std + value: -8.283374897295717 + - type: nauc_ndcg_at_1_diff1 + value: 66.44819716543068 + - type: nauc_ndcg_at_1_max + value: 19.717201903767986 + - type: nauc_ndcg_at_1_std + value: -15.336875227297902 + - type: nauc_ndcg_at_20_diff1 + value: 61.80824854740686 + - type: nauc_ndcg_at_20_max + value: 26.831041894412593 + - type: nauc_ndcg_at_20_std + value: -7.188552854116385 + - type: nauc_ndcg_at_3_diff1 + value: 61.747724042994015 + - type: nauc_ndcg_at_3_max + value: 23.426174377134767 + - type: nauc_ndcg_at_3_std + value: -10.41327754115556 + - type: nauc_ndcg_at_5_diff1 + value: 61.548403438811604 + - type: nauc_ndcg_at_5_max + value: 24.159239715161462 + - type: nauc_ndcg_at_5_std + value: -8.651104316213873 + - type: nauc_precision_at_1000_diff1 + value: 58.1547934489109 + - type: nauc_precision_at_1000_max + value: 92.86471345294885 + - type: nauc_precision_at_1000_std + value: 85.92878592878589 + - type: nauc_precision_at_100_diff1 + value: 52.20107650903497 + - type: nauc_precision_at_100_max + value: 72.61218212775312 + - type: nauc_precision_at_100_std + value: 52.523754599879084 + - type: nauc_precision_at_10_diff1 + value: 58.12459419745063 + - type: nauc_precision_at_10_max + value: 40.89977104192992 + - type: nauc_precision_at_10_std + value: 3.492953070918366 + - type: nauc_precision_at_1_diff1 + value: 66.44819716543068 + - type: nauc_precision_at_1_max + value: 19.717201903767986 + - type: nauc_precision_at_1_std + value: -15.336875227297902 + - type: nauc_precision_at_20_diff1 + value: 55.51306648157717 + - type: nauc_precision_at_20_max + value: 50.04125223449585 + - type: nauc_precision_at_20_std + value: 15.62706670420837 + - type: nauc_precision_at_3_diff1 + value: 58.167034551105914 + - type: nauc_precision_at_3_max + value: 26.878462993828578 + - type: nauc_precision_at_3_std + value: -6.362738347287171 + - type: nauc_precision_at_5_diff1 + value: 57.03963827066697 + - type: nauc_precision_at_5_max + value: 29.61380946203878 + - type: nauc_precision_at_5_std + value: -0.5583455330504148 + - type: nauc_recall_at_1000_diff1 + value: 58.15479344891117 + - type: nauc_recall_at_1000_max + value: 92.86471345294875 + - type: nauc_recall_at_1000_std + value: 85.92878592878608 + - type: nauc_recall_at_100_diff1 + value: 52.201076509035005 + - type: nauc_recall_at_100_max + value: 72.61218212775306 + - type: nauc_recall_at_100_std + value: 52.52375459987927 + - type: nauc_recall_at_10_diff1 + value: 58.12459419745076 + - type: nauc_recall_at_10_max + value: 40.89977104193005 + - type: nauc_recall_at_10_std + value: 3.4929530709184156 + - type: nauc_recall_at_1_diff1 + value: 66.44819716543068 + - type: nauc_recall_at_1_max + value: 19.717201903767986 + - type: nauc_recall_at_1_std + value: -15.336875227297902 + - type: nauc_recall_at_20_diff1 + value: 55.513066481577276 + - type: nauc_recall_at_20_max + value: 50.041252234495936 + - type: nauc_recall_at_20_std + value: 15.627066704208364 + - type: nauc_recall_at_3_diff1 + value: 58.167034551105935 + - type: nauc_recall_at_3_max + value: 26.878462993828546 + - type: nauc_recall_at_3_std + value: -6.362738347287183 + - type: nauc_recall_at_5_diff1 + value: 57.03963827066695 + - type: nauc_recall_at_5_max + value: 29.613809462038848 + - type: nauc_recall_at_5_std + value: -0.558345533050415 + - type: ndcg_at_1 + value: 46.800000000000004 + - type: ndcg_at_10 + value: 62.629000000000005 + - type: ndcg_at_100 + value: 65.603 + - type: ndcg_at_1000 + value: 66.00200000000001 + - type: ndcg_at_20 + value: 64.086 + - type: ndcg_at_3 + value: 57.132000000000005 + - type: ndcg_at_5 + value: 59.874 + - type: precision_at_1 + value: 46.800000000000004 + - type: precision_at_10 + value: 7.9399999999999995 + - type: precision_at_100 + value: 0.932 + - type: precision_at_1000 + value: 0.096 + - type: precision_at_20 + value: 4.255 + - type: precision_at_3 + value: 21.467 + - type: precision_at_5 + value: 14.2 + - type: recall_at_1 + value: 46.800000000000004 + - type: recall_at_10 + value: 79.4 + - type: recall_at_100 + value: 93.2 + - type: recall_at_1000 + value: 96.3 + - type: recall_at_20 + value: 85.1 + - type: recall_at_3 + value: 64.4 + - type: recall_at_5 + value: 71.0 + task: + type: Retrieval + - dataset: + config: default + name: MTEB IFlyTek + revision: 421605374b29664c5fc098418fe20ada9bd55f8a + split: validation + type: C-MTEB/IFlyTek-classification + metrics: + - type: accuracy + value: 48.54944209311274 + - type: f1 + value: 35.4035877894852 + - type: f1_weighted + value: 45.154890336987165 + - type: main_score + value: 48.54944209311274 + task: + type: Classification + - dataset: + config: default + name: MTEB JDReview + revision: b7c64bd89eb87f8ded463478346f76731f07bf8b + split: test + type: C-MTEB/JDReview-classification + metrics: + - type: accuracy + value: 83.11444652908067 + - type: ap + value: 50.14519497969601 + - type: ap_weighted + value: 50.14519497969601 + - type: f1 + value: 77.87333673269332 + - type: f1_weighted + value: 84.63457388911604 + - type: main_score + value: 83.11444652908067 + task: + type: Classification + - dataset: + config: default + name: MTEB LCQMC + revision: 17f9b096f80380fce5ed12a9be8be7784b337daf + split: test + type: C-MTEB/LCQMC + metrics: + - type: cosine_pearson + value: 66.68236791193614 + - type: cosine_spearman + value: 70.81518072368738 + - type: euclidean_pearson + value: 69.14238176341911 + - type: euclidean_spearman + value: 70.81518071944527 + - type: main_score + value: 70.81518072368738 + - type: manhattan_pearson + value: 69.10723297223483 + - type: manhattan_spearman + value: 70.77165164968557 + - type: pearson + value: 66.68236791193614 + - type: spearman + value: 70.81518072368738 + task: + type: STS + - dataset: + config: default + name: MTEB MMarcoReranking + revision: 8e0c766dbe9e16e1d221116a3f36795fbade07f6 + split: dev + type: C-MTEB/Mmarco-reranking + metrics: + - type: main_score + value: 25.488749669858247 + - type: map + value: 25.488749669858247 + - type: mrr + value: 23.813888888888886 + - type: nAUC_map_diff1 + value: 21.439042551978073 + - type: nAUC_map_max + value: 4.356646643203048 + - type: nAUC_map_std + value: -6.4651117756570935 + - type: nAUC_mrr_diff1 + value: 21.040572282417614 + - type: nAUC_mrr_max + value: 3.7091513714293187 + - type: nAUC_mrr_std + value: -7.380296002229526 + task: + type: Reranking + - dataset: + config: default + name: MTEB MMarcoRetrieval + revision: 539bbde593d947e2a124ba72651aafc09eb33fc2 + split: dev + type: C-MTEB/MMarcoRetrieval + metrics: + - type: main_score + value: 78.975 + - type: map_at_1 + value: 65.864 + - type: map_at_10 + value: 75.142 + - type: map_at_100 + value: 75.472 + - type: map_at_1000 + value: 75.482 + - type: map_at_20 + value: 75.369 + - type: map_at_3 + value: 73.219 + - type: map_at_5 + value: 74.42 + - type: mrr_at_1 + value: 68.0515759312321 + - type: mrr_at_10 + value: 75.65253331514067 + - type: mrr_at_100 + value: 75.94116743915134 + - type: mrr_at_1000 + value: 75.9508744592604 + - type: mrr_at_20 + value: 75.85185969774719 + - type: mrr_at_3 + value: 73.96131805157593 + - type: mrr_at_5 + value: 75.01361031518599 + - type: nauc_map_at_1000_diff1 + value: 77.47587043918091 + - type: nauc_map_at_1000_max + value: 32.511975057438974 + - type: nauc_map_at_1000_std + value: -17.666094558943417 + - type: nauc_map_at_100_diff1 + value: 77.47241550099297 + - type: nauc_map_at_100_max + value: 32.52837754222784 + - type: nauc_map_at_100_std + value: -17.63717709810784 + - type: nauc_map_at_10_diff1 + value: 77.31160893655145 + - type: nauc_map_at_10_max + value: 32.60670257703009 + - type: nauc_map_at_10_std + value: -17.771047114996144 + - type: nauc_map_at_1_diff1 + value: 79.92534779416832 + - type: nauc_map_at_1_max + value: 24.785321951074515 + - type: nauc_map_at_1_std + value: -23.45816811226535 + - type: nauc_map_at_20_diff1 + value: 77.42100083928817 + - type: nauc_map_at_20_max + value: 32.593057471316435 + - type: nauc_map_at_20_std + value: -17.608670597024197 + - type: nauc_map_at_3_diff1 + value: 77.39438847887251 + - type: nauc_map_at_3_max + value: 31.397054344602548 + - type: nauc_map_at_3_std + value: -19.4976864523292 + - type: nauc_map_at_5_diff1 + value: 77.34834480360892 + - type: nauc_map_at_5_max + value: 32.21397984602674 + - type: nauc_map_at_5_std + value: -18.45142721452299 + - type: nauc_mrr_at_1000_diff1 + value: 77.91270423495435 + - type: nauc_mrr_at_1000_max + value: 33.286202404635816 + - type: nauc_mrr_at_1000_std + value: -16.988509017415858 + - type: nauc_mrr_at_100_diff1 + value: 77.90931964992795 + - type: nauc_mrr_at_100_max + value: 33.302772470570595 + - type: nauc_mrr_at_100_std + value: -16.95907503705695 + - type: nauc_mrr_at_10_diff1 + value: 77.74963685519619 + - type: nauc_mrr_at_10_max + value: 33.402887556509064 + - type: nauc_mrr_at_10_std + value: -17.007385451413267 + - type: nauc_mrr_at_1_diff1 + value: 80.44952589549635 + - type: nauc_mrr_at_1_max + value: 28.725876665931615 + - type: nauc_mrr_at_1_std + value: -22.18861108564602 + - type: nauc_mrr_at_20_diff1 + value: 77.86689879110665 + - type: nauc_mrr_at_20_max + value: 33.37387640439328 + - type: nauc_mrr_at_20_std + value: -16.908485017918682 + - type: nauc_mrr_at_3_diff1 + value: 77.85662171057854 + - type: nauc_mrr_at_3_max + value: 32.7468353039024 + - type: nauc_mrr_at_3_std + value: -18.339103980341957 + - type: nauc_mrr_at_5_diff1 + value: 77.81340059404181 + - type: nauc_mrr_at_5_max + value: 33.15760829968908 + - type: nauc_mrr_at_5_std + value: -17.53876847769669 + - type: nauc_ndcg_at_1000_diff1 + value: 77.01438896352477 + - type: nauc_ndcg_at_1000_max + value: 34.55832672068587 + - type: nauc_ndcg_at_1000_std + value: -14.559071957898562 + - type: nauc_ndcg_at_100_diff1 + value: 76.92298480629259 + - type: nauc_ndcg_at_100_max + value: 35.11598028021352 + - type: nauc_ndcg_at_100_std + value: -13.554650282912817 + - type: nauc_ndcg_at_10_diff1 + value: 76.13277495290987 + - type: nauc_ndcg_at_10_max + value: 35.6330609610323 + - type: nauc_ndcg_at_10_std + value: -13.944707424342036 + - type: nauc_ndcg_at_1_diff1 + value: 80.44952589549635 + - type: nauc_ndcg_at_1_max + value: 28.725876665931615 + - type: nauc_ndcg_at_1_std + value: -22.18861108564602 + - type: nauc_ndcg_at_20_diff1 + value: 76.56409694019162 + - type: nauc_ndcg_at_20_max + value: 35.62960378910709 + - type: nauc_ndcg_at_20_std + value: -13.255084552264481 + - type: nauc_ndcg_at_3_diff1 + value: 76.47868292868812 + - type: nauc_ndcg_at_3_max + value: 33.233395497849614 + - type: nauc_ndcg_at_3_std + value: -17.748967332305217 + - type: nauc_ndcg_at_5_diff1 + value: 76.32952180796222 + - type: nauc_ndcg_at_5_max + value: 34.60345012714885 + - type: nauc_ndcg_at_5_std + value: -15.8173707360486 + - type: nauc_precision_at_1000_diff1 + value: -17.52918228066939 + - type: nauc_precision_at_1000_max + value: 20.626086186594833 + - type: nauc_precision_at_1000_std + value: 26.231038468619218 + - type: nauc_precision_at_100_diff1 + value: -4.307304124093403 + - type: nauc_precision_at_100_max + value: 28.87742760011375 + - type: nauc_precision_at_100_std + value: 30.66463485009584 + - type: nauc_precision_at_10_diff1 + value: 21.252248820566297 + - type: nauc_precision_at_10_max + value: 37.7120251998008 + - type: nauc_precision_at_10_std + value: 16.537333131912142 + - type: nauc_precision_at_1_diff1 + value: 80.44952589549635 + - type: nauc_precision_at_1_max + value: 28.725876665931615 + - type: nauc_precision_at_1_std + value: -22.18861108564602 + - type: nauc_precision_at_20_diff1 + value: 10.933686624253406 + - type: nauc_precision_at_20_max + value: 35.427519568658674 + - type: nauc_precision_at_20_std + value: 24.992563220729206 + - type: nauc_precision_at_3_diff1 + value: 46.36733939400364 + - type: nauc_precision_at_3_max + value: 35.27060526473028 + - type: nauc_precision_at_3_std + value: -3.8471891263107327 + - type: nauc_precision_at_5_diff1 + value: 35.16733507102863 + - type: nauc_precision_at_5_max + value: 36.69354322946151 + - type: nauc_precision_at_5_std + value: 5.0611235120787725 + - type: nauc_recall_at_1000_diff1 + value: 64.08540543698155 + - type: nauc_recall_at_1000_max + value: 77.7366984466171 + - type: nauc_recall_at_1000_std + value: 73.92508298307646 + - type: nauc_recall_at_100_diff1 + value: 68.80219044771133 + - type: nauc_recall_at_100_max + value: 73.37989662095796 + - type: nauc_recall_at_100_std + value: 59.45952818309578 + - type: nauc_recall_at_10_diff1 + value: 66.89046359223634 + - type: nauc_recall_at_10_max + value: 53.497471386132865 + - type: nauc_recall_at_10_std + value: 10.389938512313515 + - type: nauc_recall_at_1_diff1 + value: 79.92534779416832 + - type: nauc_recall_at_1_max + value: 24.785321951074515 + - type: nauc_recall_at_1_std + value: -23.45816811226535 + - type: nauc_recall_at_20_diff1 + value: 67.58868553352538 + - type: nauc_recall_at_20_max + value: 61.29680579230528 + - type: nauc_recall_at_20_std + value: 27.640944871424384 + - type: nauc_recall_at_3_diff1 + value: 72.10723775978491 + - type: nauc_recall_at_3_max + value: 37.98766181226662 + - type: nauc_recall_at_3_std + value: -12.700827115093238 + - type: nauc_recall_at_5_diff1 + value: 70.56165414287506 + - type: nauc_recall_at_5_max + value: 43.87249758193006 + - type: nauc_recall_at_5_std + value: -4.957329954267046 + - type: ndcg_at_1 + value: 68.052 + - type: ndcg_at_10 + value: 78.975 + - type: ndcg_at_100 + value: 80.415 + - type: ndcg_at_1000 + value: 80.683 + - type: ndcg_at_20 + value: 79.752 + - type: ndcg_at_3 + value: 75.315 + - type: ndcg_at_5 + value: 77.336 + - type: precision_at_1 + value: 68.052 + - type: precision_at_10 + value: 9.602 + - type: precision_at_100 + value: 1.031 + - type: precision_at_1000 + value: 0.105 + - type: precision_at_20 + value: 4.968 + - type: precision_at_3 + value: 28.405 + - type: precision_at_5 + value: 18.129 + - type: recall_at_1 + value: 65.864 + - type: recall_at_10 + value: 90.29599999999999 + - type: recall_at_100 + value: 96.712 + - type: recall_at_1000 + value: 98.817 + - type: recall_at_20 + value: 93.28999999999999 + - type: recall_at_3 + value: 80.599 + - type: recall_at_5 + value: 85.384 + task: + type: Retrieval + - dataset: + config: zh-CN + name: MTEB MassiveIntentClassification (zh-CN) + revision: 4672e20407010da34463acc759c162ca9734bca6 + split: test + type: mteb/amazon_massive_intent + metrics: + - type: accuracy + value: 68.52723604572965 + - type: f1 + value: 65.11440259918176 + - type: f1_weighted + value: 67.36553063592758 + - type: main_score + value: 68.52723604572965 + task: + type: Classification + - dataset: + config: zh-TW + name: MTEB MassiveIntentClassification (zh-TW) + revision: 4672e20407010da34463acc759c162ca9734bca6 + split: test + type: mteb/amazon_massive_intent + metrics: + - type: accuracy + value: 60.00672494956289 + - type: f1 + value: 57.630108926504654 + - type: f1_weighted + value: 58.013112273357294 + - type: main_score + value: 60.00672494956289 + task: + type: Classification + - dataset: + config: zh-CN + name: MTEB MassiveScenarioClassification (zh-CN) + revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 + split: test + type: mteb/amazon_massive_scenario + metrics: + - type: accuracy + value: 75.48755884330868 + - type: f1 + value: 74.96341150854326 + - type: f1_weighted + value: 75.11344709750414 + - type: main_score + value: 75.48755884330868 + task: + type: Classification + - dataset: + config: zh-TW + name: MTEB MassiveScenarioClassification (zh-TW) + revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 + split: test + type: mteb/amazon_massive_scenario + metrics: + - type: accuracy + value: 69.20645595158035 + - type: f1 + value: 69.57805291990489 + - type: f1_weighted + value: 68.41325928547694 + - type: main_score + value: 69.20645595158035 + task: + type: Classification + - dataset: + config: default + name: MTEB MedicalRetrieval + revision: 2039188fb5800a9803ba5048df7b76e6fb151fc6 + split: dev + type: C-MTEB/MedicalRetrieval + metrics: + - type: main_score + value: 56.842999999999996 + - type: map_at_1 + value: 48.4 + - type: map_at_10 + value: 54.127 + - type: map_at_100 + value: 54.74099999999999 + - type: map_at_1000 + value: 54.791999999999994 + - type: map_at_20 + value: 54.510999999999996 + - type: map_at_3 + value: 52.900000000000006 + - type: map_at_5 + value: 53.555 + - type: mrr_at_1 + value: 48.4 + - type: mrr_at_10 + value: 54.12742063492064 + - type: mrr_at_100 + value: 54.74007856972957 + - type: mrr_at_1000 + value: 54.79120377168234 + - type: mrr_at_20 + value: 54.50988389233748 + - type: mrr_at_3 + value: 52.89999999999999 + - type: mrr_at_5 + value: 53.554999999999986 + - type: nauc_map_at_1000_diff1 + value: 78.4025670906739 + - type: nauc_map_at_1000_max + value: 54.375261014986506 + - type: nauc_map_at_1000_std + value: 20.01404778643528 + - type: nauc_map_at_100_diff1 + value: 78.38648222989812 + - type: nauc_map_at_100_max + value: 54.36062306870368 + - type: nauc_map_at_100_std + value: 20.027286514844604 + - type: nauc_map_at_10_diff1 + value: 78.66966790097088 + - type: nauc_map_at_10_max + value: 54.49489942528706 + - type: nauc_map_at_10_std + value: 19.706732968916686 + - type: nauc_map_at_1_diff1 + value: 81.3863109971009 + - type: nauc_map_at_1_max + value: 54.45112546839308 + - type: nauc_map_at_1_std + value: 16.647233102073926 + - type: nauc_map_at_20_diff1 + value: 78.35531215313732 + - type: nauc_map_at_20_max + value: 54.38122778164954 + - type: nauc_map_at_20_std + value: 19.919068985031778 + - type: nauc_map_at_3_diff1 + value: 78.76127354528398 + - type: nauc_map_at_3_max + value: 54.73984834363167 + - type: nauc_map_at_3_std + value: 18.978057003463505 + - type: nauc_map_at_5_diff1 + value: 78.86532352860041 + - type: nauc_map_at_5_max + value: 54.89472733309794 + - type: nauc_map_at_5_std + value: 19.905059369622226 + - type: nauc_mrr_at_1000_diff1 + value: 78.40339084857348 + - type: nauc_mrr_at_1000_max + value: 54.3765713570027 + - type: nauc_mrr_at_1000_std + value: 20.01339841032316 + - type: nauc_mrr_at_100_diff1 + value: 78.38730500946644 + - type: nauc_mrr_at_100_max + value: 54.361931749382975 + - type: nauc_mrr_at_100_std + value: 20.026637904594022 + - type: nauc_mrr_at_10_diff1 + value: 78.66966790097088 + - type: nauc_mrr_at_10_max + value: 54.49489942528706 + - type: nauc_mrr_at_10_std + value: 19.706732968916686 + - type: nauc_mrr_at_1_diff1 + value: 81.3863109971009 + - type: nauc_mrr_at_1_max + value: 54.45112546839308 + - type: nauc_mrr_at_1_std + value: 16.647233102073926 + - type: nauc_mrr_at_20_diff1 + value: 78.3561572811205 + - type: nauc_mrr_at_20_max + value: 54.38258490064744 + - type: nauc_mrr_at_20_std + value: 19.918521746919023 + - type: nauc_mrr_at_3_diff1 + value: 78.76127354528398 + - type: nauc_mrr_at_3_max + value: 54.73984834363167 + - type: nauc_mrr_at_3_std + value: 18.978057003463505 + - type: nauc_mrr_at_5_diff1 + value: 78.86532352860041 + - type: nauc_mrr_at_5_max + value: 54.89472733309794 + - type: nauc_mrr_at_5_std + value: 19.905059369622226 + - type: nauc_ndcg_at_1000_diff1 + value: 76.97351511935506 + - type: nauc_ndcg_at_1000_max + value: 53.77087107676227 + - type: nauc_ndcg_at_1000_std + value: 22.05965205980897 + - type: nauc_ndcg_at_100_diff1 + value: 76.53100615219961 + - type: nauc_ndcg_at_100_max + value: 53.310394382128834 + - type: nauc_ndcg_at_100_std + value: 22.50789785109818 + - type: nauc_ndcg_at_10_diff1 + value: 77.54846078594903 + - type: nauc_ndcg_at_10_max + value: 53.933288762789324 + - type: nauc_ndcg_at_10_std + value: 20.831152333725047 + - type: nauc_ndcg_at_1_diff1 + value: 81.3863109971009 + - type: nauc_ndcg_at_1_max + value: 54.45112546839308 + - type: nauc_ndcg_at_1_std + value: 16.647233102073926 + - type: nauc_ndcg_at_20_diff1 + value: 76.34535899505026 + - type: nauc_ndcg_at_20_max + value: 53.51900448424713 + - type: nauc_ndcg_at_20_std + value: 21.786462318073514 + - type: nauc_ndcg_at_3_diff1 + value: 77.9282109219262 + - type: nauc_ndcg_at_3_max + value: 54.68499432962286 + - type: nauc_ndcg_at_3_std + value: 19.565421430997855 + - type: nauc_ndcg_at_5_diff1 + value: 78.08531364224505 + - type: nauc_ndcg_at_5_max + value: 54.990763159275524 + - type: nauc_ndcg_at_5_std + value: 21.24642472562554 + - type: nauc_precision_at_1000_diff1 + value: 60.947712418300526 + - type: nauc_precision_at_1000_max + value: 48.95614495798286 + - type: nauc_precision_at_1000_std + value: 52.283934407096034 + - type: nauc_precision_at_100_diff1 + value: 65.20401832958854 + - type: nauc_precision_at_100_max + value: 45.622481375866016 + - type: nauc_precision_at_100_std + value: 39.317645096209475 + - type: nauc_precision_at_10_diff1 + value: 73.52050673935791 + - type: nauc_precision_at_10_max + value: 51.501061174147324 + - type: nauc_precision_at_10_std + value: 24.660571885937692 + - type: nauc_precision_at_1_diff1 + value: 81.3863109971009 + - type: nauc_precision_at_1_max + value: 54.45112546839308 + - type: nauc_precision_at_1_std + value: 16.647233102073926 + - type: nauc_precision_at_20_diff1 + value: 67.23012065064371 + - type: nauc_precision_at_20_max + value: 49.276980564706854 + - type: nauc_precision_at_20_std + value: 30.08840589323484 + - type: nauc_precision_at_3_diff1 + value: 75.41179105476107 + - type: nauc_precision_at_3_max + value: 54.46883825911932 + - type: nauc_precision_at_3_std + value: 21.287190629517518 + - type: nauc_precision_at_5_diff1 + value: 75.66706449322432 + - type: nauc_precision_at_5_max + value: 55.30373302312931 + - type: nauc_precision_at_5_std + value: 25.617663499161512 + - type: nauc_recall_at_1000_diff1 + value: 60.94771241830054 + - type: nauc_recall_at_1000_max + value: 48.956144957983234 + - type: nauc_recall_at_1000_std + value: 52.283934407096 + - type: nauc_recall_at_100_diff1 + value: 65.20401832958845 + - type: nauc_recall_at_100_max + value: 45.622481375866066 + - type: nauc_recall_at_100_std + value: 39.31764509620953 + - type: nauc_recall_at_10_diff1 + value: 73.52050673935793 + - type: nauc_recall_at_10_max + value: 51.5010611741474 + - type: nauc_recall_at_10_std + value: 24.660571885937728 + - type: nauc_recall_at_1_diff1 + value: 81.3863109971009 + - type: nauc_recall_at_1_max + value: 54.45112546839308 + - type: nauc_recall_at_1_std + value: 16.647233102073926 + - type: nauc_recall_at_20_diff1 + value: 67.2301206506438 + - type: nauc_recall_at_20_max + value: 49.276980564706946 + - type: nauc_recall_at_20_std + value: 30.08840589323487 + - type: nauc_recall_at_3_diff1 + value: 75.411791054761 + - type: nauc_recall_at_3_max + value: 54.468838259119266 + - type: nauc_recall_at_3_std + value: 21.287190629517486 + - type: nauc_recall_at_5_diff1 + value: 75.66706449322434 + - type: nauc_recall_at_5_max + value: 55.30373302312929 + - type: nauc_recall_at_5_std + value: 25.61766349916154 + - type: ndcg_at_1 + value: 48.4 + - type: ndcg_at_10 + value: 56.842999999999996 + - type: ndcg_at_100 + value: 60.035000000000004 + - type: ndcg_at_1000 + value: 61.592999999999996 + - type: ndcg_at_20 + value: 58.219 + - type: ndcg_at_3 + value: 54.269 + - type: ndcg_at_5 + value: 55.435 + - type: precision_at_1 + value: 48.4 + - type: precision_at_10 + value: 6.54 + - type: precision_at_100 + value: 0.8089999999999999 + - type: precision_at_1000 + value: 0.094 + - type: precision_at_20 + value: 3.54 + - type: precision_at_3 + value: 19.400000000000002 + - type: precision_at_5 + value: 12.2 + - type: recall_at_1 + value: 48.4 + - type: recall_at_10 + value: 65.4 + - type: recall_at_100 + value: 80.9 + - type: recall_at_1000 + value: 93.60000000000001 + - type: recall_at_20 + value: 70.8 + - type: recall_at_3 + value: 58.199999999999996 + - type: recall_at_5 + value: 61.0 + task: + type: Retrieval + - dataset: + config: default + name: MTEB MultilingualSentiment + revision: 46958b007a63fdbf239b7672c25d0bea67b5ea1a + split: test + type: C-MTEB/MultilingualSentiment-classification + metrics: + - type: accuracy + value: 78.22 + - type: f1 + value: 77.72387105131016 + - type: f1_weighted + value: 77.72387105131016 + - type: main_score + value: 78.22 + task: + type: Classification + - dataset: + config: default + name: MTEB Ocnli + revision: 66e76a618a34d6d565d5538088562851e6daa7ec + split: validation + type: C-MTEB/OCNLI + metrics: + - type: cosine_accuracy + value: 66.21548456957228 + - type: cosine_accuracy_threshold + value: 85.5343759059906 + - type: cosine_ap + value: 69.21438509764522 + - type: cosine_f1 + value: 70.44534412955467 + - type: cosine_f1_threshold + value: 83.85478258132935 + - type: cosine_precision + value: 61.36363636363637 + - type: cosine_recall + value: 82.68215417106653 + - type: dot_accuracy + value: 66.21548456957228 + - type: dot_accuracy_threshold + value: 85.5343759059906 + - type: dot_ap + value: 69.21438509764522 + - type: dot_f1 + value: 70.44534412955467 + - type: dot_f1_threshold + value: 83.85477662086487 + - type: dot_precision + value: 61.36363636363637 + - type: dot_recall + value: 82.68215417106653 + - type: euclidean_accuracy + value: 66.21548456957228 + - type: euclidean_accuracy_threshold + value: 53.787779808044434 + - type: euclidean_ap + value: 69.21438509764522 + - type: euclidean_f1 + value: 70.44534412955467 + - type: euclidean_f1_threshold + value: 56.82467818260193 + - type: euclidean_precision + value: 61.36363636363637 + - type: euclidean_recall + value: 82.68215417106653 + - type: main_score + value: 69.21438509764522 + - type: manhattan_accuracy + value: 65.89063345966431 + - type: manhattan_accuracy_threshold + value: 1257.1067810058594 + - type: manhattan_ap + value: 68.93169452602294 + - type: manhattan_f1 + value: 70.018281535649 + - type: manhattan_f1_threshold + value: 1321.6852188110352 + - type: manhattan_precision + value: 61.72441579371475 + - type: manhattan_recall + value: 80.8870116156283 + - type: max_ap + value: 69.21438509764522 + - type: max_f1 + value: 70.44534412955467 + - type: max_precision + value: 61.72441579371475 + - type: max_recall + value: 82.68215417106653 + - type: similarity_accuracy + value: 66.21548456957228 + - type: similarity_accuracy_threshold + value: 85.5343759059906 + - type: similarity_ap + value: 69.21438509764522 + - type: similarity_f1 + value: 70.44534412955467 + - type: similarity_f1_threshold + value: 83.85478258132935 + - type: similarity_precision + value: 61.36363636363637 + - type: similarity_recall + value: 82.68215417106653 + task: + type: PairClassification + - dataset: + config: default + name: MTEB OnlineShopping + revision: e610f2ebd179a8fda30ae534c3878750a96db120 + split: test + type: C-MTEB/OnlineShopping-classification + metrics: + - type: accuracy + value: 93.09 + - type: ap + value: 92.02754366798146 + - type: ap_weighted + value: 92.02754366798146 + - type: f1 + value: 93.08751542879655 + - type: f1_weighted + value: 93.092429173112 + - type: main_score + value: 93.09 + task: + type: Classification + - dataset: + config: default + name: MTEB PAWSX + revision: 9c6a90e430ac22b5779fb019a23e820b11a8b5e1 + split: test + type: C-MTEB/PAWSX + metrics: + - type: cosine_pearson + value: 14.700393205389778 + - type: cosine_spearman + value: 16.3328315656736 + - type: euclidean_pearson + value: 18.37941425496121 + - type: euclidean_spearman + value: 16.35419723095118 + - type: main_score + value: 16.3328315656736 + - type: manhattan_pearson + value: 18.461692452314196 + - type: manhattan_spearman + value: 16.49877203419426 + - type: pearson + value: 14.700393205389778 + - type: spearman + value: 16.3328315656736 + task: + type: STS + - dataset: + config: default + name: MTEB QBQTC + revision: 790b0510dc52b1553e8c49f3d2afb48c0e5c48b7 + split: test + type: C-MTEB/QBQTC + metrics: + - type: cosine_pearson + value: 31.771341848950048 + - type: cosine_spearman + value: 35.21248641540282 + - type: euclidean_pearson + value: 32.86721005963099 + - type: euclidean_spearman + value: 35.21281584988107 + - type: main_score + value: 35.21248641540282 + - type: manhattan_pearson + value: 32.994593737560464 + - type: manhattan_spearman + value: 35.29332061037625 + - type: pearson + value: 31.771341848950048 + - type: spearman + value: 35.21248641540282 + task: + type: STS + - dataset: + config: zh + name: MTEB STS22 (zh) + revision: de9d86b3b84231dc21f76c7b7af1f28e2f57f6e3 + split: test + type: mteb/sts22-crosslingual-sts + metrics: + - type: cosine_pearson + value: 74.05009462610907 + - type: cosine_spearman + value: 73.57662463047618 + - type: euclidean_pearson + value: 71.97192342862905 + - type: euclidean_spearman + value: 73.57662463047618 + - type: main_score + value: 73.57662463047618 + - type: manhattan_pearson + value: 71.66171363912555 + - type: manhattan_spearman + value: 73.33995365752388 + - type: pearson + value: 74.05009462610907 + - type: spearman + value: 73.57662463047618 + task: + type: STS + - dataset: + config: zh-en + name: MTEB STS22 (zh-en) + revision: de9d86b3b84231dc21f76c7b7af1f28e2f57f6e3 + split: test + type: mteb/sts22-crosslingual-sts + metrics: + - type: cosine_pearson + value: 76.5491270627292 + - type: cosine_spearman + value: 76.34747339191613 + - type: euclidean_pearson + value: 77.52703884917828 + - type: euclidean_spearman + value: 76.34747339191613 + - type: main_score + value: 76.34747339191613 + - type: manhattan_pearson + value: 77.94417346436772 + - type: manhattan_spearman + value: 76.15552728253843 + - type: pearson + value: 76.5491270627292 + - type: spearman + value: 76.34747339191613 + task: + type: STS + - dataset: + config: default + name: MTEB STSB + revision: 0cde68302b3541bb8b3c340dc0644b0b745b3dc0 + split: test + type: C-MTEB/STSB + metrics: + - type: cosine_pearson + value: 76.44164798054604 + - type: cosine_spearman + value: 77.80818016295537 + - type: euclidean_pearson + value: 77.23235480541429 + - type: euclidean_spearman + value: 77.80820115269609 + - type: main_score + value: 77.80818016295537 + - type: manhattan_pearson + value: 77.1383518573764 + - type: manhattan_spearman + value: 77.74335331319868 + - type: pearson + value: 76.44164798054604 + - type: spearman + value: 77.80818016295537 + task: + type: STS + - dataset: + config: default + name: MTEB T2Reranking + revision: 76631901a18387f85eaa53e5450019b87ad58ef9 + split: dev + type: C-MTEB/T2Reranking + metrics: + - type: main_score + value: 66.73229864475817 + - type: map + value: 66.73229864475817 + - type: mrr + value: 76.93718364122901 + - type: nAUC_map_diff1 + value: -10.073077377159336 + - type: nAUC_map_max + value: 36.06382407917089 + - type: nAUC_map_std + value: -2.3465936800383975 + - type: nAUC_mrr_diff1 + value: -7.614514359731517 + - type: nAUC_mrr_max + value: 29.970126517689895 + - type: nAUC_mrr_std + value: -4.000393719743965 + task: + type: Reranking + - dataset: + config: default + name: MTEB T2Retrieval + revision: 8731a845f1bf500a4f111cf1070785c793d10e64 + split: dev + type: C-MTEB/T2Retrieval + metrics: + - type: main_score + value: 84.066 + - type: map_at_1 + value: 27.311999999999998 + - type: map_at_10 + value: 76.45100000000001 + - type: map_at_100 + value: 80.081 + - type: map_at_1000 + value: 80.148 + - type: map_at_20 + value: 79.19500000000001 + - type: map_at_3 + value: 53.785000000000004 + - type: map_at_5 + value: 66.08800000000001 + - type: mrr_at_1 + value: 89.5493599859723 + - type: mrr_at_10 + value: 92.04882420558401 + - type: mrr_at_100 + value: 92.13851397485055 + - type: mrr_at_1000 + value: 92.142380377799 + - type: mrr_at_20 + value: 92.10835206390658 + - type: mrr_at_3 + value: 91.55634461394584 + - type: mrr_at_5 + value: 91.86626921503318 + - type: nauc_map_at_1000_diff1 + value: 13.881593411464955 + - type: nauc_map_at_1000_max + value: 45.98958134604669 + - type: nauc_map_at_1000_std + value: 20.749937417202162 + - type: nauc_map_at_100_diff1 + value: 13.897201544868256 + - type: nauc_map_at_100_max + value: 45.8975771537659 + - type: nauc_map_at_100_std + value: 20.702471991537816 + - type: nauc_map_at_10_diff1 + value: 18.574187481021404 + - type: nauc_map_at_10_max + value: 33.03798476409084 + - type: nauc_map_at_10_std + value: 4.923864341255681 + - type: nauc_map_at_1_diff1 + value: 53.60647320762889 + - type: nauc_map_at_1_max + value: -25.482104681284262 + - type: nauc_map_at_1_std + value: -35.242254755025314 + - type: nauc_map_at_20_diff1 + value: 14.659565827642442 + - type: nauc_map_at_20_max + value: 43.28938016609046 + - type: nauc_map_at_20_std + value: 17.356608121513382 + - type: nauc_map_at_3_diff1 + value: 39.4070801457046 + - type: nauc_map_at_3_max + value: -12.578348941863455 + - type: nauc_map_at_3_std + value: -31.8798104413808 + - type: nauc_map_at_5_diff1 + value: 31.32071840697937 + - type: nauc_map_at_5_max + value: 4.002877899741811 + - type: nauc_map_at_5_std + value: -21.303079028844135 + - type: nauc_mrr_at_1000_diff1 + value: 46.83368353209455 + - type: nauc_mrr_at_1000_max + value: 75.71133032423971 + - type: nauc_mrr_at_1000_std + value: 39.48046724592415 + - type: nauc_mrr_at_100_diff1 + value: 46.83586223804469 + - type: nauc_mrr_at_100_max + value: 75.72076556031453 + - type: nauc_mrr_at_100_std + value: 39.499587346326805 + - type: nauc_mrr_at_10_diff1 + value: 46.844356562367416 + - type: nauc_mrr_at_10_max + value: 75.82630788248727 + - type: nauc_mrr_at_10_std + value: 39.57574348115377 + - type: nauc_mrr_at_1_diff1 + value: 47.31173835991697 + - type: nauc_mrr_at_1_max + value: 71.62100768371566 + - type: nauc_mrr_at_1_std + value: 33.215001055441654 + - type: nauc_mrr_at_20_diff1 + value: 46.81803840032271 + - type: nauc_mrr_at_20_max + value: 75.77378447923428 + - type: nauc_mrr_at_20_std + value: 39.56872022078684 + - type: nauc_mrr_at_3_diff1 + value: 46.860529286384946 + - type: nauc_mrr_at_3_max + value: 75.76461859710832 + - type: nauc_mrr_at_3_std + value: 39.2216938919496 + - type: nauc_mrr_at_5_diff1 + value: 46.789493616730475 + - type: nauc_mrr_at_5_max + value: 75.90313095614088 + - type: nauc_mrr_at_5_std + value: 39.65230578423716 + - type: nauc_ndcg_at_1000_diff1 + value: 17.766598062291752 + - type: nauc_ndcg_at_1000_max + value: 58.1826623156839 + - type: nauc_ndcg_at_1000_std + value: 32.709580361957975 + - type: nauc_ndcg_at_100_diff1 + value: 17.52651950834308 + - type: nauc_ndcg_at_100_max + value: 57.35240006774048 + - type: nauc_ndcg_at_100_std + value: 32.8548321688731 + - type: nauc_ndcg_at_10_diff1 + value: 17.54354007672668 + - type: nauc_ndcg_at_10_max + value: 47.71924227814137 + - type: nauc_ndcg_at_10_std + value: 21.21142176035583 + - type: nauc_ndcg_at_1_diff1 + value: 47.31173835991697 + - type: nauc_ndcg_at_1_max + value: 71.62100768371566 + - type: nauc_ndcg_at_1_std + value: 33.215001055441654 + - type: nauc_ndcg_at_20_diff1 + value: 17.70238226651868 + - type: nauc_ndcg_at_20_max + value: 51.938599251095376 + - type: nauc_ndcg_at_20_std + value: 26.4850369109262 + - type: nauc_ndcg_at_3_diff1 + value: 13.27812172653936 + - type: nauc_ndcg_at_3_max + value: 62.30760700113597 + - type: nauc_ndcg_at_3_std + value: 31.816092105184445 + - type: nauc_ndcg_at_5_diff1 + value: 13.889486796899545 + - type: nauc_ndcg_at_5_max + value: 55.51019399952754 + - type: nauc_ndcg_at_5_std + value: 27.369160358894128 + - type: nauc_precision_at_1000_diff1 + value: -35.999300149931784 + - type: nauc_precision_at_1000_max + value: 49.84051965547009 + - type: nauc_precision_at_1000_std + value: 58.58089570689277 + - type: nauc_precision_at_100_diff1 + value: -35.83612932514131 + - type: nauc_precision_at_100_max + value: 51.586346962862336 + - type: nauc_precision_at_100_std + value: 60.23951447272133 + - type: nauc_precision_at_10_diff1 + value: -35.48923076280097 + - type: nauc_precision_at_10_max + value: 55.72743866396911 + - type: nauc_precision_at_10_std + value: 55.44739603788007 + - type: nauc_precision_at_1_diff1 + value: 47.31173835991697 + - type: nauc_precision_at_1_max + value: 71.62100768371566 + - type: nauc_precision_at_1_std + value: 33.215001055441654 + - type: nauc_precision_at_20_diff1 + value: -35.72916661370677 + - type: nauc_precision_at_20_max + value: 54.082057684079246 + - type: nauc_precision_at_20_std + value: 59.4645317147574 + - type: nauc_precision_at_3_diff1 + value: -30.247794446720594 + - type: nauc_precision_at_3_max + value: 64.43209282496724 + - type: nauc_precision_at_3_std + value: 47.32128724076265 + - type: nauc_precision_at_5_diff1 + value: -35.057736789550646 + - type: nauc_precision_at_5_max + value: 60.10076766835969 + - type: nauc_precision_at_5_std + value: 50.98874173280675 + - type: nauc_recall_at_1000_diff1 + value: 1.863242393322992 + - type: nauc_recall_at_1000_max + value: 60.729765914144615 + - type: nauc_recall_at_1000_std + value: 62.59143677558537 + - type: nauc_recall_at_100_diff1 + value: 8.045141361395181 + - type: nauc_recall_at_100_max + value: 52.778769966654316 + - type: nauc_recall_at_100_std + value: 49.10300097655128 + - type: nauc_recall_at_10_diff1 + value: 16.8879769794751 + - type: nauc_recall_at_10_max + value: 23.43756027359313 + - type: nauc_recall_at_10_std + value: -0.017371946235138786 + - type: nauc_recall_at_1_diff1 + value: 53.60647320762889 + - type: nauc_recall_at_1_max + value: -25.482104681284262 + - type: nauc_recall_at_1_std + value: -35.242254755025314 + - type: nauc_recall_at_20_diff1 + value: 11.099660881919029 + - type: nauc_recall_at_20_max + value: 39.07089132734822 + - type: nauc_recall_at_20_std + value: 22.93956030600117 + - type: nauc_recall_at_3_diff1 + value: 38.03040951694198 + - type: nauc_recall_at_3_max + value: -16.512296557855894 + - type: nauc_recall_at_3_std + value: -33.99009868262299 + - type: nauc_recall_at_5_diff1 + value: 30.476589203240035 + - type: nauc_recall_at_5_max + value: -4.132009312569869 + - type: nauc_recall_at_5_std + value: -26.413027108583538 + - type: ndcg_at_1 + value: 89.549 + - type: ndcg_at_10 + value: 84.066 + - type: ndcg_at_100 + value: 87.705 + - type: ndcg_at_1000 + value: 88.369 + - type: ndcg_at_20 + value: 85.82300000000001 + - type: ndcg_at_3 + value: 85.466 + - type: ndcg_at_5 + value: 84.077 + - type: precision_at_1 + value: 89.549 + - type: precision_at_10 + value: 41.778 + - type: precision_at_100 + value: 4.994 + - type: precision_at_1000 + value: 0.515 + - type: precision_at_20 + value: 23.199 + - type: precision_at_3 + value: 74.772 + - type: precision_at_5 + value: 62.666999999999994 + - type: recall_at_1 + value: 27.311999999999998 + - type: recall_at_10 + value: 82.955 + - type: recall_at_100 + value: 94.814 + - type: recall_at_1000 + value: 98.205 + - type: recall_at_20 + value: 88.864 + - type: recall_at_3 + value: 55.35600000000001 + - type: recall_at_5 + value: 69.458 + task: + type: Retrieval + - dataset: + config: default + name: MTEB TNews + revision: 317f262bf1e6126357bbe89e875451e4b0938fe4 + split: validation + type: C-MTEB/TNews-classification + metrics: + - type: accuracy + value: 51.629999999999995 + - type: f1 + value: 49.06894119635303 + - type: f1_weighted + value: 50.97940411274921 + - type: main_score + value: 51.629999999999995 + task: + type: Classification + - dataset: + config: default + name: MTEB ThuNewsClusteringP2P + revision: 5798586b105c0434e4f0fe5e767abe619442cf93 + split: test + type: C-MTEB/ThuNewsClusteringP2P + metrics: + - type: main_score + value: 72.29482641581953 + - type: v_measure + value: 72.29482641581953 + - type: v_measure_std + value: 1.335959363862232 + task: + type: Clustering + - dataset: + config: default + name: MTEB ThuNewsClusteringS2S + revision: 8a8b2caeda43f39e13c4bc5bea0f8a667896e10d + split: test + type: C-MTEB/ThuNewsClusteringS2S + metrics: + - type: main_score + value: 65.18517414783898 + - type: v_measure + value: 65.18517414783898 + - type: v_measure_std + value: 1.4758720962337082 + task: + type: Clustering + - dataset: + config: default + name: MTEB VideoRetrieval + revision: 58c2597a5943a2ba48f4668c3b90d796283c5639 + split: dev + type: C-MTEB/VideoRetrieval + metrics: + - type: main_score + value: 71.768 + - type: map_at_1 + value: 57.8 + - type: map_at_10 + value: 67.47 + - type: map_at_100 + value: 67.97800000000001 + - type: map_at_1000 + value: 67.988 + - type: map_at_20 + value: 67.827 + - type: map_at_3 + value: 65.583 + - type: map_at_5 + value: 66.743 + - type: mrr_at_1 + value: 57.699999999999996 + - type: mrr_at_10 + value: 67.4202777777778 + - type: mrr_at_100 + value: 67.92827763873187 + - type: mrr_at_1000 + value: 67.93825676811373 + - type: mrr_at_20 + value: 67.77708320016758 + - type: mrr_at_3 + value: 65.53333333333336 + - type: mrr_at_5 + value: 66.69333333333337 + - type: nauc_map_at_1000_diff1 + value: 69.57667330550309 + - type: nauc_map_at_1000_max + value: 7.543501399956271 + - type: nauc_map_at_1000_std + value: -31.243010383568436 + - type: nauc_map_at_100_diff1 + value: 69.5725467000578 + - type: nauc_map_at_100_max + value: 7.557154826181643 + - type: nauc_map_at_100_std + value: -31.244478201857078 + - type: nauc_map_at_10_diff1 + value: 69.66295657598643 + - type: nauc_map_at_10_max + value: 7.354933246004566 + - type: nauc_map_at_10_std + value: -31.85480226841353 + - type: nauc_map_at_1_diff1 + value: 71.13428332324658 + - type: nauc_map_at_1_max + value: 5.617556184249036 + - type: nauc_map_at_1_std + value: -29.319041979284005 + - type: nauc_map_at_20_diff1 + value: 69.58825754865646 + - type: nauc_map_at_20_max + value: 7.493499540606082 + - type: nauc_map_at_20_std + value: -31.421082585601923 + - type: nauc_map_at_3_diff1 + value: 69.51937616497842 + - type: nauc_map_at_3_max + value: 6.58989269981783 + - type: nauc_map_at_3_std + value: -31.99863169578963 + - type: nauc_map_at_5_diff1 + value: 69.5662738665744 + - type: nauc_map_at_5_max + value: 7.15847377146298 + - type: nauc_map_at_5_std + value: -31.915079957015152 + - type: nauc_mrr_at_1000_diff1 + value: 69.7213828991122 + - type: nauc_mrr_at_1000_max + value: 7.429696859248949 + - type: nauc_mrr_at_1000_std + value: -31.443415682632214 + - type: nauc_mrr_at_100_diff1 + value: 69.71720349997756 + - type: nauc_mrr_at_100_max + value: 7.44338462432531 + - type: nauc_mrr_at_100_std + value: -31.444805433177237 + - type: nauc_mrr_at_10_diff1 + value: 69.80501501763766 + - type: nauc_mrr_at_10_max + value: 7.243295472335978 + - type: nauc_mrr_at_10_std + value: -32.05120239232289 + - type: nauc_mrr_at_1_diff1 + value: 71.36516347743338 + - type: nauc_mrr_at_1_max + value: 5.4367617772159935 + - type: nauc_mrr_at_1_std + value: -29.643310130582368 + - type: nauc_mrr_at_20_diff1 + value: 69.73206575087396 + - type: nauc_mrr_at_20_max + value: 7.380431929849562 + - type: nauc_mrr_at_20_std + value: -31.620128440139172 + - type: nauc_mrr_at_3_diff1 + value: 69.65406365393979 + - type: nauc_mrr_at_3_max + value: 6.484611159978816 + - type: nauc_mrr_at_3_std + value: -32.18462142962385 + - type: nauc_mrr_at_5_diff1 + value: 69.70510268881297 + - type: nauc_mrr_at_5_max + value: 7.049554381027974 + - type: nauc_mrr_at_5_std + value: -32.106886637783475 + - type: nauc_ndcg_at_1000_diff1 + value: 69.13809538133073 + - type: nauc_ndcg_at_1000_max + value: 9.63283640379659 + - type: nauc_ndcg_at_1000_std + value: -29.449553581261878 + - type: nauc_ndcg_at_100_diff1 + value: 68.99395374944251 + - type: nauc_ndcg_at_100_max + value: 10.176719987248335 + - type: nauc_ndcg_at_100_std + value: -29.350437600307988 + - type: nauc_ndcg_at_10_diff1 + value: 69.46707308337217 + - type: nauc_ndcg_at_10_max + value: 8.763989177891727 + - type: nauc_ndcg_at_10_std + value: -32.80464097517361 + - type: nauc_ndcg_at_1_diff1 + value: 71.13428332324658 + - type: nauc_ndcg_at_1_max + value: 5.617556184249036 + - type: nauc_ndcg_at_1_std + value: -29.319041979284005 + - type: nauc_ndcg_at_20_diff1 + value: 69.18406330264568 + - type: nauc_ndcg_at_20_max + value: 9.638224463038883 + - type: nauc_ndcg_at_20_std + value: -30.852858198950152 + - type: nauc_ndcg_at_3_diff1 + value: 69.09204185224701 + - type: nauc_ndcg_at_3_max + value: 7.001054807139578 + - type: nauc_ndcg_at_3_std + value: -33.03926415507931 + - type: nauc_ndcg_at_5_diff1 + value: 69.19746073354963 + - type: nauc_ndcg_at_5_max + value: 8.129939462557209 + - type: nauc_ndcg_at_5_std + value: -32.890200669688824 + - type: nauc_precision_at_1000_diff1 + value: 55.42483660130655 + - type: nauc_precision_at_1000_max + value: 87.87815126050374 + - type: nauc_precision_at_1000_std + value: 83.7791783380018 + - type: nauc_precision_at_100_diff1 + value: 57.369917664035285 + - type: nauc_precision_at_100_max + value: 67.47729394788239 + - type: nauc_precision_at_100_std + value: 27.854171971818914 + - type: nauc_precision_at_10_diff1 + value: 68.85063417125569 + - type: nauc_precision_at_10_max + value: 17.102325185471244 + - type: nauc_precision_at_10_std + value: -37.74775895599127 + - type: nauc_precision_at_1_diff1 + value: 71.13428332324658 + - type: nauc_precision_at_1_max + value: 5.617556184249036 + - type: nauc_precision_at_1_std + value: -29.319041979284005 + - type: nauc_precision_at_20_diff1 + value: 66.21858365339261 + - type: nauc_precision_at_20_max + value: 30.03311290151776 + - type: nauc_precision_at_20_std + value: -21.57151520401986 + - type: nauc_precision_at_3_diff1 + value: 67.63911432841829 + - type: nauc_precision_at_3_max + value: 8.481916121984119 + - type: nauc_precision_at_3_std + value: -36.756075465752815 + - type: nauc_precision_at_5_diff1 + value: 67.81563046786316 + - type: nauc_precision_at_5_max + value: 12.39051881812358 + - type: nauc_precision_at_5_std + value: -36.853583944247085 + - type: nauc_recall_at_1000_diff1 + value: 55.424836601307135 + - type: nauc_recall_at_1000_max + value: 87.87815126050475 + - type: nauc_recall_at_1000_std + value: 83.77917833800133 + - type: nauc_recall_at_100_diff1 + value: 57.36991766403544 + - type: nauc_recall_at_100_max + value: 67.47729394788247 + - type: nauc_recall_at_100_std + value: 27.854171971819124 + - type: nauc_recall_at_10_diff1 + value: 68.8506341712557 + - type: nauc_recall_at_10_max + value: 17.102325185471305 + - type: nauc_recall_at_10_std + value: -37.747758955991095 + - type: nauc_recall_at_1_diff1 + value: 71.13428332324658 + - type: nauc_recall_at_1_max + value: 5.617556184249036 + - type: nauc_recall_at_1_std + value: -29.319041979284005 + - type: nauc_recall_at_20_diff1 + value: 66.21858365339266 + - type: nauc_recall_at_20_max + value: 30.033112901517956 + - type: nauc_recall_at_20_std + value: -21.57151520401992 + - type: nauc_recall_at_3_diff1 + value: 67.63911432841829 + - type: nauc_recall_at_3_max + value: 8.481916121984066 + - type: nauc_recall_at_3_std + value: -36.75607546575289 + - type: nauc_recall_at_5_diff1 + value: 67.81563046786322 + - type: nauc_recall_at_5_max + value: 12.390518818123626 + - type: nauc_recall_at_5_std + value: -36.85358394424703 + - type: ndcg_at_1 + value: 57.8 + - type: ndcg_at_10 + value: 71.768 + - type: ndcg_at_100 + value: 74.093 + - type: ndcg_at_1000 + value: 74.39 + - type: ndcg_at_20 + value: 73.078 + - type: ndcg_at_3 + value: 67.93900000000001 + - type: ndcg_at_5 + value: 70.035 + - type: precision_at_1 + value: 57.8 + - type: precision_at_10 + value: 8.51 + - type: precision_at_100 + value: 0.9560000000000001 + - type: precision_at_1000 + value: 0.098 + - type: precision_at_20 + value: 4.515000000000001 + - type: precision_at_3 + value: 24.9 + - type: precision_at_5 + value: 15.959999999999999 + - type: recall_at_1 + value: 57.8 + - type: recall_at_10 + value: 85.1 + - type: recall_at_100 + value: 95.6 + - type: recall_at_1000 + value: 98.0 + - type: recall_at_20 + value: 90.3 + - type: recall_at_3 + value: 74.7 + - type: recall_at_5 + value: 79.80000000000001 + task: + type: Retrieval + - dataset: + config: default + name: MTEB Waimai + revision: 339287def212450dcaa9df8c22bf93e9980c7023 + split: test + type: C-MTEB/waimai-classification + metrics: + - type: accuracy + value: 88.85 + - type: ap + value: 74.3186702823446 + - type: ap_weighted + value: 74.3186702823446 + - type: f1 + value: 87.23283814404301 + - type: f1_weighted + value: 88.8207921928931 + - type: main_score + value: 88.85 + task: + type: Classification + - dataset: + config: default + name: MTEB AlloProfClusteringP2P + revision: 392ba3f5bcc8c51f578786c1fc3dae648662cb9b + split: test + type: lyon-nlp/alloprof + metrics: + - type: main_score + value: 65.0533654633344 + - type: v_measure + value: 65.0533654633344 + - type: v_measure_std + value: 2.2966620551090435 + task: + type: Clustering + - dataset: + config: default + name: MTEB AlloProfClusteringS2S + revision: 392ba3f5bcc8c51f578786c1fc3dae648662cb9b + split: test + type: lyon-nlp/alloprof + metrics: + - type: main_score + value: 47.602163687995045 + - type: v_measure + value: 47.602163687995045 + - type: v_measure_std + value: 2.2689446239597566 + task: + type: Clustering + - dataset: + config: default + name: MTEB AlloprofReranking + revision: 65393d0d7a08a10b4e348135e824f385d420b0fd + split: test + type: lyon-nlp/mteb-fr-reranking-alloprof-s2p + metrics: + - type: main_score + value: 75.17961983598764 + - type: map + value: 75.17961983598764 + - type: mrr + value: 76.42541875702497 + - type: nAUC_map_diff1 + value: 57.62404220848294 + - type: nAUC_map_max + value: 23.51522112548143 + - type: nAUC_map_std + value: 23.388021631358534 + - type: nAUC_mrr_diff1 + value: 57.37912924766015 + - type: nAUC_mrr_max + value: 24.08827107711477 + - type: nAUC_mrr_std + value: 22.59882369784862 + task: + type: Reranking + - dataset: + config: default + name: MTEB AlloprofRetrieval + revision: fcf295ea64c750f41fadbaa37b9b861558e1bfbd + split: test + type: lyon-nlp/alloprof + metrics: + - type: main_score + value: 51.115 + - type: map_at_1 + value: 33.29 + - type: map_at_10 + value: 45.017 + - type: map_at_100 + value: 45.824 + - type: map_at_1000 + value: 45.869 + - type: map_at_20 + value: 45.521 + - type: map_at_3 + value: 41.947 + - type: map_at_5 + value: 43.683 + - type: mrr_at_1 + value: 33.29015544041451 + - type: mrr_at_10 + value: 45.017099816322656 + - type: mrr_at_100 + value: 45.82409198237741 + - type: mrr_at_1000 + value: 45.86948422318224 + - type: mrr_at_20 + value: 45.52135027955749 + - type: mrr_at_3 + value: 41.94732297063909 + - type: mrr_at_5 + value: 43.68307426597593 + - type: nauc_map_at_1000_diff1 + value: 39.73243793300825 + - type: nauc_map_at_1000_max + value: 35.182681191334545 + - type: nauc_map_at_1000_std + value: 0.5807963314175563 + - type: nauc_map_at_100_diff1 + value: 39.722470553779765 + - type: nauc_map_at_100_max + value: 35.21852170079372 + - type: nauc_map_at_100_std + value: 0.6058130117591941 + - type: nauc_map_at_10_diff1 + value: 39.617580503261316 + - type: nauc_map_at_10_max + value: 35.07856190730939 + - type: nauc_map_at_10_std + value: 0.12493876166019499 + - type: nauc_map_at_1_diff1 + value: 42.93831821879295 + - type: nauc_map_at_1_max + value: 31.12726818453338 + - type: nauc_map_at_1_std + value: -1.777716163867983 + - type: nauc_map_at_20_diff1 + value: 39.71626015027946 + - type: nauc_map_at_20_max + value: 35.18348804461845 + - type: nauc_map_at_20_std + value: 0.5105592256773273 + - type: nauc_map_at_3_diff1 + value: 39.16316684618178 + - type: nauc_map_at_3_max + value: 33.7216790348974 + - type: nauc_map_at_3_std + value: -0.3498983530547474 + - type: nauc_map_at_5_diff1 + value: 39.647098664998204 + - type: nauc_map_at_5_max + value: 34.7328902446716 + - type: nauc_map_at_5_std + value: 0.40121258866920295 + - type: nauc_mrr_at_1000_diff1 + value: 39.73243793300825 + - type: nauc_mrr_at_1000_max + value: 35.182681191334545 + - type: nauc_mrr_at_1000_std + value: 0.5807963314175563 + - type: nauc_mrr_at_100_diff1 + value: 39.722470553779765 + - type: nauc_mrr_at_100_max + value: 35.21852170079372 + - type: nauc_mrr_at_100_std + value: 0.6058130117591941 + - type: nauc_mrr_at_10_diff1 + value: 39.617580503261316 + - type: nauc_mrr_at_10_max + value: 35.07856190730939 + - type: nauc_mrr_at_10_std + value: 0.12493876166019499 + - type: nauc_mrr_at_1_diff1 + value: 42.93831821879295 + - type: nauc_mrr_at_1_max + value: 31.12726818453338 + - type: nauc_mrr_at_1_std + value: -1.777716163867983 + - type: nauc_mrr_at_20_diff1 + value: 39.71626015027946 + - type: nauc_mrr_at_20_max + value: 35.18348804461845 + - type: nauc_mrr_at_20_std + value: 0.5105592256773273 + - type: nauc_mrr_at_3_diff1 + value: 39.16316684618178 + - type: nauc_mrr_at_3_max + value: 33.7216790348974 + - type: nauc_mrr_at_3_std + value: -0.3498983530547474 + - type: nauc_mrr_at_5_diff1 + value: 39.647098664998204 + - type: nauc_mrr_at_5_max + value: 34.7328902446716 + - type: nauc_mrr_at_5_std + value: 0.40121258866920295 + - type: nauc_ndcg_at_1000_diff1 + value: 39.242329273752844 + - type: nauc_ndcg_at_1000_max + value: 36.778397735569555 + - type: nauc_ndcg_at_1000_std + value: 1.9890182774165592 + - type: nauc_ndcg_at_100_diff1 + value: 39.02516880016918 + - type: nauc_ndcg_at_100_max + value: 37.95005510807288 + - type: nauc_ndcg_at_100_std + value: 3.0123421168495446 + - type: nauc_ndcg_at_10_diff1 + value: 38.663112091897084 + - type: nauc_ndcg_at_10_max + value: 37.30636725701253 + - type: nauc_ndcg_at_10_std + value: 0.9040375526062125 + - type: nauc_ndcg_at_1_diff1 + value: 42.93831821879295 + - type: nauc_ndcg_at_1_max + value: 31.12726818453338 + - type: nauc_ndcg_at_1_std + value: -1.777716163867983 + - type: nauc_ndcg_at_20_diff1 + value: 38.929484271597715 + - type: nauc_ndcg_at_20_max + value: 37.75901008441635 + - type: nauc_ndcg_at_20_std + value: 2.4125130730180193 + - type: nauc_ndcg_at_3_diff1 + value: 37.92470631843311 + - type: nauc_ndcg_at_3_max + value: 34.450507157634625 + - type: nauc_ndcg_at_3_std + value: 0.11472879002764375 + - type: nauc_ndcg_at_5_diff1 + value: 38.778362690069535 + - type: nauc_ndcg_at_5_max + value: 36.356309785771344 + - type: nauc_ndcg_at_5_std + value: 1.502446979862238 + - type: nauc_precision_at_1000_diff1 + value: 33.18147748601189 + - type: nauc_precision_at_1000_max + value: 75.4289210395029 + - type: nauc_precision_at_1000_std + value: 67.86158379540547 + - type: nauc_precision_at_100_diff1 + value: 34.33353129136502 + - type: nauc_precision_at_100_max + value: 66.13041373773511 + - type: nauc_precision_at_100_std + value: 31.220993361158992 + - type: nauc_precision_at_10_diff1 + value: 35.191343796022224 + - type: nauc_precision_at_10_max + value: 46.5030195236983 + - type: nauc_precision_at_10_std + value: 3.8450864519608947 + - type: nauc_precision_at_1_diff1 + value: 42.93831821879295 + - type: nauc_precision_at_1_max + value: 31.12726818453338 + - type: nauc_precision_at_1_std + value: -1.777716163867983 + - type: nauc_precision_at_20_diff1 + value: 35.627717677075786 + - type: nauc_precision_at_20_max + value: 51.28161545904851 + - type: nauc_precision_at_20_std + value: 13.250115934134326 + - type: nauc_precision_at_3_diff1 + value: 34.252273608621316 + - type: nauc_precision_at_3_max + value: 36.573298257048386 + - type: nauc_precision_at_3_std + value: 1.4913107196473938 + - type: nauc_precision_at_5_diff1 + value: 36.160987910682564 + - type: nauc_precision_at_5_max + value: 41.778030453127776 + - type: nauc_precision_at_5_std + value: 5.2043752514074395 + - type: nauc_recall_at_1000_diff1 + value: 33.18147748600853 + - type: nauc_recall_at_1000_max + value: 75.42892103950442 + - type: nauc_recall_at_1000_std + value: 67.86158379540124 + - type: nauc_recall_at_100_diff1 + value: 34.33353129136507 + - type: nauc_recall_at_100_max + value: 66.13041373773494 + - type: nauc_recall_at_100_std + value: 31.22099336115893 + - type: nauc_recall_at_10_diff1 + value: 35.191343796022245 + - type: nauc_recall_at_10_max + value: 46.503019523698185 + - type: nauc_recall_at_10_std + value: 3.845086451960918 + - type: nauc_recall_at_1_diff1 + value: 42.93831821879295 + - type: nauc_recall_at_1_max + value: 31.12726818453338 + - type: nauc_recall_at_1_std + value: -1.777716163867983 + - type: nauc_recall_at_20_diff1 + value: 35.62771767707572 + - type: nauc_recall_at_20_max + value: 51.281615459048524 + - type: nauc_recall_at_20_std + value: 13.250115934134335 + - type: nauc_recall_at_3_diff1 + value: 34.2522736086213 + - type: nauc_recall_at_3_max + value: 36.57329825704846 + - type: nauc_recall_at_3_std + value: 1.4913107196474313 + - type: nauc_recall_at_5_diff1 + value: 36.160987910682586 + - type: nauc_recall_at_5_max + value: 41.778030453127826 + - type: nauc_recall_at_5_std + value: 5.204375251407459 + - type: ndcg_at_1 + value: 33.29 + - type: ndcg_at_10 + value: 51.115 + - type: ndcg_at_100 + value: 55.169999999999995 + - type: ndcg_at_1000 + value: 56.37799999999999 + - type: ndcg_at_20 + value: 52.929 + - type: ndcg_at_3 + value: 44.800000000000004 + - type: ndcg_at_5 + value: 47.911 + - type: precision_at_1 + value: 33.29 + - type: precision_at_10 + value: 7.042 + - type: precision_at_100 + value: 0.897 + - type: precision_at_1000 + value: 0.099 + - type: precision_at_20 + value: 3.877 + - type: precision_at_3 + value: 17.689 + - type: precision_at_5 + value: 12.116 + - type: recall_at_1 + value: 33.29 + - type: recall_at_10 + value: 70.423 + - type: recall_at_100 + value: 89.724 + - type: recall_at_1000 + value: 99.223 + - type: recall_at_20 + value: 77.547 + - type: recall_at_3 + value: 53.066 + - type: recall_at_5 + value: 60.57900000000001 + task: + type: Retrieval + - dataset: + config: fr + name: MTEB AmazonReviewsClassification (fr) + revision: 1399c76144fd37290681b995c656ef9b2e06e26d + split: test + type: mteb/amazon_reviews_multi + metrics: + - type: accuracy + value: 52.312000000000005 + - type: f1 + value: 48.4862498298928 + - type: f1_weighted + value: 48.4862498298928 + - type: main_score + value: 52.312000000000005 + task: + type: Classification + - dataset: + config: default + name: MTEB BSARDRetrieval + revision: 5effa1b9b5fa3b0f9e12523e6e43e5f86a6e6d59 + split: test + type: maastrichtlawtech/bsard + metrics: + - type: main_score + value: 61.260999999999996 + - type: map_at_1 + value: 9.459 + - type: map_at_10 + value: 16.678 + - type: map_at_100 + value: 17.774 + - type: map_at_1000 + value: 17.858999999999998 + - type: map_at_20 + value: 17.321 + - type: map_at_3 + value: 14.414 + - type: map_at_5 + value: 15.338 + - type: mrr_at_1 + value: 9.45945945945946 + - type: mrr_at_10 + value: 16.67792792792793 + - type: mrr_at_100 + value: 17.774362522691398 + - type: mrr_at_1000 + value: 17.859514657558048 + - type: mrr_at_20 + value: 17.321237623095215 + - type: mrr_at_3 + value: 14.414414414414411 + - type: mrr_at_5 + value: 15.337837837837837 + - type: nauc_map_at_1000_diff1 + value: 12.898723868198564 + - type: nauc_map_at_1000_max + value: 24.74492446599715 + - type: nauc_map_at_1000_std + value: 1.2568203864373333 + - type: nauc_map_at_100_diff1 + value: 12.851213507789646 + - type: nauc_map_at_100_max + value: 24.697430695694827 + - type: nauc_map_at_100_std + value: 1.2129231125259072 + - type: nauc_map_at_10_diff1 + value: 13.07707744574533 + - type: nauc_map_at_10_max + value: 24.72177610914694 + - type: nauc_map_at_10_std + value: -0.004280467049762924 + - type: nauc_map_at_1_diff1 + value: 21.11635429465273 + - type: nauc_map_at_1_max + value: 18.613969823495 + - type: nauc_map_at_1_std + value: -7.597691141141031 + - type: nauc_map_at_20_diff1 + value: 13.143191038140104 + - type: nauc_map_at_20_max + value: 24.904996278522944 + - type: nauc_map_at_20_std + value: 0.9083792521836654 + - type: nauc_map_at_3_diff1 + value: 14.048204755527621 + - type: nauc_map_at_3_max + value: 26.100088213960383 + - type: nauc_map_at_3_std + value: -0.7368466542848658 + - type: nauc_map_at_5_diff1 + value: 13.482648315427548 + - type: nauc_map_at_5_max + value: 24.803356986058585 + - type: nauc_map_at_5_std + value: -1.2343058157885538 + - type: nauc_mrr_at_1000_diff1 + value: 12.8977020563612 + - type: nauc_mrr_at_1000_max + value: 24.742855881415966 + - type: nauc_mrr_at_1000_std + value: 1.2564589487746798 + - type: nauc_mrr_at_100_diff1 + value: 12.850193815594121 + - type: nauc_mrr_at_100_max + value: 24.69536657999111 + - type: nauc_mrr_at_100_std + value: 1.2125623256841345 + - type: nauc_mrr_at_10_diff1 + value: 13.07707744574533 + - type: nauc_mrr_at_10_max + value: 24.72177610914694 + - type: nauc_mrr_at_10_std + value: -0.004280467049762924 + - type: nauc_mrr_at_1_diff1 + value: 21.11635429465273 + - type: nauc_mrr_at_1_max + value: 18.613969823495 + - type: nauc_mrr_at_1_std + value: -7.597691141141031 + - type: nauc_mrr_at_20_diff1 + value: 13.143191038140104 + - type: nauc_mrr_at_20_max + value: 24.904996278522944 + - type: nauc_mrr_at_20_std + value: 0.9083792521836654 + - type: nauc_mrr_at_3_diff1 + value: 14.048204755527621 + - type: nauc_mrr_at_3_max + value: 26.100088213960383 + - type: nauc_mrr_at_3_std + value: -0.7368466542848658 + - type: nauc_mrr_at_5_diff1 + value: 13.482648315427548 + - type: nauc_mrr_at_5_max + value: 24.803356986058585 + - type: nauc_mrr_at_5_std + value: -1.2343058157885538 + - type: nauc_ndcg_at_1000_diff1 + value: 11.261286687432275 + - type: nauc_ndcg_at_1000_max + value: 25.54862203427727 + - type: nauc_ndcg_at_1000_std + value: 6.9058640640154785 + - type: nauc_ndcg_at_100_diff1 + value: 10.686081917901197 + - type: nauc_ndcg_at_100_max + value: 25.79174788871077 + - type: nauc_ndcg_at_100_std + value: 7.46006445502064 + - type: nauc_ndcg_at_10_diff1 + value: 11.004401131569718 + - type: nauc_ndcg_at_10_max + value: 25.55455198645665 + - type: nauc_ndcg_at_10_std + value: 1.9332719624889871 + - type: nauc_ndcg_at_1_diff1 + value: 21.11635429465273 + - type: nauc_ndcg_at_1_max + value: 18.613969823495 + - type: nauc_ndcg_at_1_std + value: -7.597691141141031 + - type: nauc_ndcg_at_20_diff1 + value: 11.429743804671329 + - type: nauc_ndcg_at_20_max + value: 25.76018584848357 + - type: nauc_ndcg_at_20_std + value: 4.529843418751598 + - type: nauc_ndcg_at_3_diff1 + value: 12.375824114924743 + - type: nauc_ndcg_at_3_max + value: 27.899863438524925 + - type: nauc_ndcg_at_3_std + value: 0.48771263604259235 + - type: nauc_ndcg_at_5_diff1 + value: 11.669991174251026 + - type: nauc_ndcg_at_5_max + value: 25.60280203013294 + - type: nauc_ndcg_at_5_std + value: -0.4851689963092356 + - type: nauc_precision_at_1000_diff1 + value: 9.217959657671416 + - type: nauc_precision_at_1000_max + value: 26.785438149619228 + - type: nauc_precision_at_1000_std + value: 34.586575721918315 + - type: nauc_precision_at_100_diff1 + value: 6.944903323168987 + - type: nauc_precision_at_100_max + value: 29.703339699715265 + - type: nauc_precision_at_100_std + value: 28.76732870912694 + - type: nauc_precision_at_10_diff1 + value: 6.924015026667702 + - type: nauc_precision_at_10_max + value: 27.01090600566311 + - type: nauc_precision_at_10_std + value: 5.987244079632962 + - type: nauc_precision_at_1_diff1 + value: 21.11635429465273 + - type: nauc_precision_at_1_max + value: 18.613969823495 + - type: nauc_precision_at_1_std + value: -7.597691141141031 + - type: nauc_precision_at_20_diff1 + value: 8.547176188111889 + - type: nauc_precision_at_20_max + value: 27.008083551891193 + - type: nauc_precision_at_20_std + value: 12.963207602265662 + - type: nauc_precision_at_3_diff1 + value: 8.681050417224782 + - type: nauc_precision_at_3_max + value: 31.88286662067647 + - type: nauc_precision_at_3_std + value: 3.0982821138307313 + - type: nauc_precision_at_5_diff1 + value: 7.881738093345716 + - type: nauc_precision_at_5_max + value: 26.996917506757985 + - type: nauc_precision_at_5_std + value: 0.8392982481773691 + - type: nauc_recall_at_1000_diff1 + value: 9.21795965767156 + - type: nauc_recall_at_1000_max + value: 26.785438149619285 + - type: nauc_recall_at_1000_std + value: 34.58657572191825 + - type: nauc_recall_at_100_diff1 + value: 6.944903323168959 + - type: nauc_recall_at_100_max + value: 29.703339699715258 + - type: nauc_recall_at_100_std + value: 28.767328709126865 + - type: nauc_recall_at_10_diff1 + value: 6.924015026667665 + - type: nauc_recall_at_10_max + value: 27.01090600566308 + - type: nauc_recall_at_10_std + value: 5.987244079632916 + - type: nauc_recall_at_1_diff1 + value: 21.11635429465273 + - type: nauc_recall_at_1_max + value: 18.613969823495 + - type: nauc_recall_at_1_std + value: -7.597691141141031 + - type: nauc_recall_at_20_diff1 + value: 8.547176188111841 + - type: nauc_recall_at_20_max + value: 27.00808355189116 + - type: nauc_recall_at_20_std + value: 12.963207602265603 + - type: nauc_recall_at_3_diff1 + value: 8.681050417224794 + - type: nauc_recall_at_3_max + value: 31.882866620676502 + - type: nauc_recall_at_3_std + value: 3.098282113830734 + - type: nauc_recall_at_5_diff1 + value: 7.881738093345706 + - type: nauc_recall_at_5_max + value: 26.996917506757928 + - type: nauc_recall_at_5_std + value: 0.8392982481773533 + - type: ndcg_at_1 + value: 9.459 + - type: ndcg_at_10 + value: 20.752000000000002 + - type: ndcg_at_100 + value: 26.454 + - type: ndcg_at_1000 + value: 29.009 + - type: ndcg_at_20 + value: 23.128999999999998 + - type: ndcg_at_3 + value: 16.034000000000002 + - type: ndcg_at_5 + value: 17.701 + - type: precision_at_1 + value: 9.459 + - type: precision_at_10 + value: 3.3779999999999997 + - type: precision_at_100 + value: 0.613 + - type: precision_at_1000 + value: 0.082 + - type: precision_at_20 + value: 2.162 + - type: precision_at_3 + value: 6.907000000000001 + - type: precision_at_5 + value: 4.955 + - type: recall_at_1 + value: 9.459 + - type: recall_at_10 + value: 33.784 + - type: recall_at_100 + value: 61.260999999999996 + - type: recall_at_1000 + value: 81.982 + - type: recall_at_20 + value: 43.242999999999995 + - type: recall_at_3 + value: 20.721 + - type: recall_at_5 + value: 24.775 + task: + type: Retrieval + - dataset: + config: default + name: MTEB HALClusteringS2S + revision: e06ebbbb123f8144bef1a5d18796f3dec9ae2915 + split: test + type: lyon-nlp/clustering-hal-s2s + metrics: + - type: main_score + value: 27.423835538133385 + - type: v_measure + value: 27.423835538133385 + - type: v_measure_std + value: 2.418592769322829 + task: + type: Clustering + - dataset: + config: fr + name: MTEB MLSUMClusteringP2P (fr) + revision: b5d54f8f3b61ae17845046286940f03c6bc79bc7 + split: test + type: reciTAL/mlsum + metrics: + - type: main_score + value: 47.71996236371622 + - type: v_measure + value: 47.71996236371622 + - type: v_measure_std + value: 2.305651972507199 + task: + type: Clustering + - dataset: + config: fr + name: MTEB MLSUMClusteringS2S (fr) + revision: b5d54f8f3b61ae17845046286940f03c6bc79bc7 + split: test + type: reciTAL/mlsum + metrics: + - type: main_score + value: 47.23617047594239 + - type: v_measure + value: 47.23617047594239 + - type: v_measure_std + value: 1.7496378174738418 + task: + type: Clustering + - dataset: + config: fr + name: MTEB MTOPDomainClassification (fr) + revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf + split: test + type: mteb/mtop_domain + metrics: + - type: accuracy + value: 95.01722518008143 + - type: f1 + value: 94.84287480029258 + - type: f1_weighted + value: 95.0452443472173 + - type: main_score + value: 95.01722518008143 + task: + type: Classification + - dataset: + config: fr + name: MTEB MTOPIntentClassification (fr) + revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba + split: test + type: mteb/mtop_intent + metrics: + - type: accuracy + value: 79.74945192608833 + - type: f1 + value: 53.733289773568536 + - type: f1_weighted + value: 79.64257473311346 + - type: main_score + value: 79.74945192608833 + task: + type: Classification + - dataset: + config: fra + name: MTEB MasakhaNEWSClassification (fra) + revision: 18193f187b92da67168c655c9973a165ed9593dd + split: test + type: mteb/masakhanews + metrics: + - type: accuracy + value: 83.3649289099526 + - type: f1 + value: 79.77180137779536 + - type: f1_weighted + value: 83.01264683214308 + - type: main_score + value: 83.3649289099526 + task: + type: Classification + - dataset: + config: fra + name: MTEB MasakhaNEWSClusteringP2P (fra) + revision: 8ccc72e69e65f40c70e117d8b3c08306bb788b60 + split: test + type: masakhane/masakhanews + metrics: + - type: main_score + value: 56.59401317251221 + - type: v_measure + value: 56.59401317251221 + - type: v_measure_std + value: 36.0760983348482 + task: + type: Clustering + - dataset: + config: fra + name: MTEB MasakhaNEWSClusteringS2S (fra) + revision: 8ccc72e69e65f40c70e117d8b3c08306bb788b60 + split: test + type: masakhane/masakhanews + metrics: + - type: main_score + value: 71.82457224763512 + - type: v_measure + value: 71.82457224763512 + - type: v_measure_std + value: 33.00235264955351 + task: + type: Clustering + - dataset: + config: fr + name: MTEB MassiveIntentClassification (fr) + revision: 4672e20407010da34463acc759c162ca9734bca6 + split: test + type: mteb/amazon_massive_intent + metrics: + - type: accuracy + value: 66.61398789509077 + - type: f1 + value: 60.956092562089324 + - type: f1_weighted + value: 64.68321586762302 + - type: main_score + value: 66.61398789509077 + task: + type: Classification + - dataset: + config: fr + name: MTEB MassiveScenarioClassification (fr) + revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 + split: test + type: mteb/amazon_massive_scenario + metrics: + - type: accuracy + value: 75.68594485541358 + - type: f1 + value: 74.71713629733614 + - type: f1_weighted + value: 75.49907502505853 + - type: main_score + value: 75.68594485541358 + task: + type: Classification + - dataset: + config: fr + name: MTEB MintakaRetrieval (fr) + revision: efa78cc2f74bbcd21eff2261f9e13aebe40b814e + split: test + type: jinaai/mintakaqa + metrics: + - type: main_score + value: 28.939999999999998 + - type: map_at_1 + value: 16.585 + - type: map_at_10 + value: 24.496000000000002 + - type: map_at_100 + value: 25.769 + - type: map_at_1000 + value: 25.857999999999997 + - type: map_at_20 + value: 25.224000000000004 + - type: map_at_3 + value: 21.908 + - type: map_at_5 + value: 23.464 + - type: mrr_at_1 + value: 16.584766584766587 + - type: mrr_at_10 + value: 24.49603824603824 + - type: mrr_at_100 + value: 25.76910538724444 + - type: mrr_at_1000 + value: 25.857970951595167 + - type: mrr_at_20 + value: 25.223803880220952 + - type: mrr_at_3 + value: 21.908271908271885 + - type: mrr_at_5 + value: 23.46437346437344 + - type: nauc_map_at_1000_diff1 + value: 24.449324509741782 + - type: nauc_map_at_1000_max + value: 16.857551392045668 + - type: nauc_map_at_1000_std + value: 9.029392522714542 + - type: nauc_map_at_100_diff1 + value: 24.416377398368994 + - type: nauc_map_at_100_max + value: 16.874285303124278 + - type: nauc_map_at_100_std + value: 9.042769202417494 + - type: nauc_map_at_10_diff1 + value: 24.30792102146879 + - type: nauc_map_at_10_max + value: 16.618278881262178 + - type: nauc_map_at_10_std + value: 8.925464095934705 + - type: nauc_map_at_1_diff1 + value: 29.76904844307235 + - type: nauc_map_at_1_max + value: 14.721060890557535 + - type: nauc_map_at_1_std + value: 4.1356111912072935 + - type: nauc_map_at_20_diff1 + value: 24.42346651590614 + - type: nauc_map_at_20_max + value: 16.780464496035194 + - type: nauc_map_at_20_std + value: 8.981541510161698 + - type: nauc_map_at_3_diff1 + value: 25.463420671464554 + - type: nauc_map_at_3_max + value: 16.4183897697643 + - type: nauc_map_at_3_std + value: 7.625690829397712 + - type: nauc_map_at_5_diff1 + value: 24.93868444905917 + - type: nauc_map_at_5_max + value: 16.091635346098528 + - type: nauc_map_at_5_std + value: 8.39683861182176 + - type: nauc_mrr_at_1000_diff1 + value: 24.449324509741782 + - type: nauc_mrr_at_1000_max + value: 16.857551392045668 + - type: nauc_mrr_at_1000_std + value: 9.029392522714542 + - type: nauc_mrr_at_100_diff1 + value: 24.416377398368994 + - type: nauc_mrr_at_100_max + value: 16.874285303124278 + - type: nauc_mrr_at_100_std + value: 9.042769202417494 + - type: nauc_mrr_at_10_diff1 + value: 24.30792102146879 + - type: nauc_mrr_at_10_max + value: 16.618278881262178 + - type: nauc_mrr_at_10_std + value: 8.925464095934705 + - type: nauc_mrr_at_1_diff1 + value: 29.76904844307235 + - type: nauc_mrr_at_1_max + value: 14.721060890557535 + - type: nauc_mrr_at_1_std + value: 4.1356111912072935 + - type: nauc_mrr_at_20_diff1 + value: 24.42346651590614 + - type: nauc_mrr_at_20_max + value: 16.780464496035194 + - type: nauc_mrr_at_20_std + value: 8.981541510161698 + - type: nauc_mrr_at_3_diff1 + value: 25.463420671464554 + - type: nauc_mrr_at_3_max + value: 16.4183897697643 + - type: nauc_mrr_at_3_std + value: 7.625690829397712 + - type: nauc_mrr_at_5_diff1 + value: 24.93868444905917 + - type: nauc_mrr_at_5_max + value: 16.091635346098528 + - type: nauc_mrr_at_5_std + value: 8.39683861182176 + - type: nauc_ndcg_at_1000_diff1 + value: 23.235835294659104 + - type: nauc_ndcg_at_1000_max + value: 18.130069784749416 + - type: nauc_ndcg_at_1000_std + value: 11.373761760081479 + - type: nauc_ndcg_at_100_diff1 + value: 22.573685299807515 + - type: nauc_ndcg_at_100_max + value: 18.932477112977686 + - type: nauc_ndcg_at_100_std + value: 12.149429764085687 + - type: nauc_ndcg_at_10_diff1 + value: 22.056289907300375 + - type: nauc_ndcg_at_10_max + value: 17.44090382465399 + - type: nauc_ndcg_at_10_std + value: 11.078636741064129 + - type: nauc_ndcg_at_1_diff1 + value: 29.76904844307235 + - type: nauc_ndcg_at_1_max + value: 14.721060890557535 + - type: nauc_ndcg_at_1_std + value: 4.1356111912072935 + - type: nauc_ndcg_at_20_diff1 + value: 22.35349136882196 + - type: nauc_ndcg_at_20_max + value: 18.085452594784922 + - type: nauc_ndcg_at_20_std + value: 11.380430365055334 + - type: nauc_ndcg_at_3_diff1 + value: 24.25988476966241 + - type: nauc_ndcg_at_3_max + value: 16.80378720501912 + - type: nauc_ndcg_at_3_std + value: 8.666118913493746 + - type: nauc_ndcg_at_5_diff1 + value: 23.43312793787156 + - type: nauc_ndcg_at_5_max + value: 16.291356652340085 + - type: nauc_ndcg_at_5_std + value: 9.985808307664348 + - type: nauc_precision_at_1000_diff1 + value: 20.59167291436066 + - type: nauc_precision_at_1000_max + value: 37.11669863650303 + - type: nauc_precision_at_1000_std + value: 53.79402863337551 + - type: nauc_precision_at_100_diff1 + value: 16.48046435474148 + - type: nauc_precision_at_100_max + value: 30.49436900744379 + - type: nauc_precision_at_100_std + value: 27.593241126791774 + - type: nauc_precision_at_10_diff1 + value: 16.21264848839721 + - type: nauc_precision_at_10_max + value: 19.68782276242417 + - type: nauc_precision_at_10_std + value: 16.66213297771153 + - type: nauc_precision_at_1_diff1 + value: 29.76904844307235 + - type: nauc_precision_at_1_max + value: 14.721060890557535 + - type: nauc_precision_at_1_std + value: 4.1356111912072935 + - type: nauc_precision_at_20_diff1 + value: 16.782602338251536 + - type: nauc_precision_at_20_max + value: 22.266838160036038 + - type: nauc_precision_at_20_std + value: 18.225681654722052 + - type: nauc_precision_at_3_diff1 + value: 21.258021189556455 + - type: nauc_precision_at_3_max + value: 17.741568150869735 + - type: nauc_precision_at_3_std + value: 11.278258942650373 + - type: nauc_precision_at_5_diff1 + value: 19.74165284616189 + - type: nauc_precision_at_5_max + value: 16.72366359140884 + - type: nauc_precision_at_5_std + value: 13.991493395533237 + - type: nauc_recall_at_1000_diff1 + value: 20.591672914360146 + - type: nauc_recall_at_1000_max + value: 37.11669863650274 + - type: nauc_recall_at_1000_std + value: 53.79402863337549 + - type: nauc_recall_at_100_diff1 + value: 16.480464354741507 + - type: nauc_recall_at_100_max + value: 30.494369007443765 + - type: nauc_recall_at_100_std + value: 27.59324112679175 + - type: nauc_recall_at_10_diff1 + value: 16.2126484883972 + - type: nauc_recall_at_10_max + value: 19.6878227624242 + - type: nauc_recall_at_10_std + value: 16.66213297771157 + - type: nauc_recall_at_1_diff1 + value: 29.76904844307235 + - type: nauc_recall_at_1_max + value: 14.721060890557535 + - type: nauc_recall_at_1_std + value: 4.1356111912072935 + - type: nauc_recall_at_20_diff1 + value: 16.782602338251547 + - type: nauc_recall_at_20_max + value: 22.266838160036027 + - type: nauc_recall_at_20_std + value: 18.22568165472206 + - type: nauc_recall_at_3_diff1 + value: 21.258021189556477 + - type: nauc_recall_at_3_max + value: 17.741568150869757 + - type: nauc_recall_at_3_std + value: 11.278258942650384 + - type: nauc_recall_at_5_diff1 + value: 19.741652846161877 + - type: nauc_recall_at_5_max + value: 16.723663591408847 + - type: nauc_recall_at_5_std + value: 13.991493395533256 + - type: ndcg_at_1 + value: 16.585 + - type: ndcg_at_10 + value: 28.939999999999998 + - type: ndcg_at_100 + value: 35.568 + - type: ndcg_at_1000 + value: 38.201 + - type: ndcg_at_20 + value: 31.576999999999998 + - type: ndcg_at_3 + value: 23.634 + - type: ndcg_at_5 + value: 26.462000000000003 + - type: precision_at_1 + value: 16.585 + - type: precision_at_10 + value: 4.316 + - type: precision_at_100 + value: 0.752 + - type: precision_at_1000 + value: 0.096 + - type: precision_at_20 + value: 2.6780000000000004 + - type: precision_at_3 + value: 9.541 + - type: precision_at_5 + value: 7.109 + - type: recall_at_1 + value: 16.585 + - type: recall_at_10 + value: 43.161 + - type: recall_at_100 + value: 75.184 + - type: recall_at_1000 + value: 96.478 + - type: recall_at_20 + value: 53.563 + - type: recall_at_3 + value: 28.624 + - type: recall_at_5 + value: 35.545 + task: + type: Retrieval + - dataset: + config: fr + name: MTEB OpusparcusPC (fr) + revision: 9e9b1f8ef51616073f47f306f7f47dd91663f86a + split: test + type: GEM/opusparcus + metrics: + - type: cosine_accuracy + value: 82.90190735694823 + - type: cosine_accuracy_threshold + value: 86.7036521434784 + - type: cosine_ap + value: 93.15630587159856 + - type: cosine_f1 + value: 88.28744750349978 + - type: cosine_f1_threshold + value: 86.63686513900757 + - type: cosine_precision + value: 83.27464788732394 + - type: cosine_recall + value: 93.94240317775571 + - type: dot_accuracy + value: 82.90190735694823 + - type: dot_accuracy_threshold + value: 86.70365810394287 + - type: dot_ap + value: 93.15630587159856 + - type: dot_f1 + value: 88.28744750349978 + - type: dot_f1_threshold + value: 86.63686513900757 + - type: dot_precision + value: 83.27464788732394 + - type: dot_recall + value: 93.94240317775571 + - type: euclidean_accuracy + value: 82.90190735694823 + - type: euclidean_accuracy_threshold + value: 51.56809687614441 + - type: euclidean_ap + value: 93.15630587159856 + - type: euclidean_f1 + value: 88.28744750349978 + - type: euclidean_f1_threshold + value: 51.69745683670044 + - type: euclidean_precision + value: 83.27464788732394 + - type: euclidean_recall + value: 93.94240317775571 + - type: main_score + value: 93.15630587159856 + - type: manhattan_accuracy + value: 82.83378746594006 + - type: manhattan_accuracy_threshold + value: 1203.4871101379395 + - type: manhattan_ap + value: 93.13986431895378 + - type: manhattan_f1 + value: 88.20224719101122 + - type: manhattan_f1_threshold + value: 1204.0313720703125 + - type: manhattan_precision + value: 83.43666961913198 + - type: manhattan_recall + value: 93.545183714002 + - type: max_ap + value: 93.15630587159856 + - type: max_f1 + value: 88.28744750349978 + - type: max_precision + value: 83.43666961913198 + - type: max_recall + value: 93.94240317775571 + - type: similarity_accuracy + value: 82.90190735694823 + - type: similarity_accuracy_threshold + value: 86.7036521434784 + - type: similarity_ap + value: 93.15630587159856 + - type: similarity_f1 + value: 88.28744750349978 + - type: similarity_f1_threshold + value: 86.63686513900757 + - type: similarity_precision + value: 83.27464788732394 + - type: similarity_recall + value: 93.94240317775571 + task: + type: PairClassification + - dataset: + config: fr + name: MTEB PawsXPairClassification (fr) + revision: 8a04d940a42cd40658986fdd8e3da561533a3646 + split: test + type: google-research-datasets/paws-x + metrics: + - type: cosine_accuracy + value: 61.0 + - type: cosine_accuracy_threshold + value: 99.4693398475647 + - type: cosine_ap + value: 60.596634996072794 + - type: cosine_f1 + value: 62.77477477477477 + - type: cosine_f1_threshold + value: 95.54682970046997 + - type: cosine_precision + value: 46.52777777777778 + - type: cosine_recall + value: 96.4562569213732 + - type: dot_accuracy + value: 61.0 + - type: dot_accuracy_threshold + value: 99.4693398475647 + - type: dot_ap + value: 60.60423446409202 + - type: dot_f1 + value: 62.77477477477477 + - type: dot_f1_threshold + value: 95.54683566093445 + - type: dot_precision + value: 46.52777777777778 + - type: dot_recall + value: 96.4562569213732 + - type: euclidean_accuracy + value: 61.0 + - type: euclidean_accuracy_threshold + value: 10.301996022462845 + - type: euclidean_ap + value: 60.596589949359995 + - type: euclidean_f1 + value: 62.77477477477477 + - type: euclidean_f1_threshold + value: 29.843494296073914 + - type: euclidean_precision + value: 46.52777777777778 + - type: euclidean_recall + value: 96.4562569213732 + - type: main_score + value: 60.81315897987771 + - type: manhattan_accuracy + value: 61.1 + - type: manhattan_accuracy_threshold + value: 241.11316204071045 + - type: manhattan_ap + value: 60.81315897987771 + - type: manhattan_f1 + value: 62.80575539568345 + - type: manhattan_f1_threshold + value: 701.3707160949707 + - type: manhattan_precision + value: 46.510388918486946 + - type: manhattan_recall + value: 96.67774086378738 + - type: max_ap + value: 60.81315897987771 + - type: max_f1 + value: 62.80575539568345 + - type: max_precision + value: 46.52777777777778 + - type: max_recall + value: 96.67774086378738 + - type: similarity_accuracy + value: 61.0 + - type: similarity_accuracy_threshold + value: 99.4693398475647 + - type: similarity_ap + value: 60.596634996072794 + - type: similarity_f1 + value: 62.77477477477477 + - type: similarity_f1_threshold + value: 95.54682970046997 + - type: similarity_precision + value: 46.52777777777778 + - type: similarity_recall + value: 96.4562569213732 + task: + type: PairClassification + - dataset: + config: default + name: MTEB SICKFr + revision: e077ab4cf4774a1e36d86d593b150422fafd8e8a + split: test + type: Lajavaness/SICK-fr + metrics: + - type: cosine_pearson + value: 80.2458052019468 + - type: cosine_spearman + value: 76.90282547383322 + - type: euclidean_pearson + value: 76.96025494858225 + - type: euclidean_spearman + value: 76.90369875962587 + - type: main_score + value: 76.90282547383322 + - type: manhattan_pearson + value: 76.96191769993092 + - type: manhattan_spearman + value: 76.78368027265795 + - type: pearson + value: 80.2458052019468 + - type: spearman + value: 76.90282547383322 + task: + type: STS + - dataset: + config: fr + name: MTEB STS22 (fr) + revision: de9d86b3b84231dc21f76c7b7af1f28e2f57f6e3 + split: test + type: mteb/sts22-crosslingual-sts + metrics: + - type: cosine_pearson + value: 71.8166682422511 + - type: cosine_spearman + value: 80.753981314684 + - type: euclidean_pearson + value: 77.26991759068049 + - type: euclidean_spearman + value: 80.753981314684 + - type: main_score + value: 80.753981314684 + - type: manhattan_pearson + value: 79.8934147126028 + - type: manhattan_spearman + value: 81.7842698600436 + - type: pearson + value: 71.8166682422511 + - type: spearman + value: 80.753981314684 + task: + type: STS + - dataset: + config: de-fr + name: MTEB STS22 (de-fr) + revision: de9d86b3b84231dc21f76c7b7af1f28e2f57f6e3 + split: test + type: mteb/sts22-crosslingual-sts + metrics: + - type: cosine_pearson + value: 59.72069019022611 + - type: cosine_spearman + value: 66.19562370923092 + - type: euclidean_pearson + value: 60.38869296832584 + - type: euclidean_spearman + value: 66.19562370923092 + - type: main_score + value: 66.19562370923092 + - type: manhattan_pearson + value: 60.735818878018044 + - type: manhattan_spearman + value: 66.3310023111313 + - type: pearson + value: 59.72069019022611 + - type: spearman + value: 66.19562370923092 + task: + type: STS + - dataset: + config: fr-pl + name: MTEB STS22 (fr-pl) + revision: de9d86b3b84231dc21f76c7b7af1f28e2f57f6e3 + split: test + type: mteb/sts22-crosslingual-sts + metrics: + - type: cosine_pearson + value: 90.73440998579521 + - type: cosine_spearman + value: 84.51542547285167 + - type: euclidean_pearson + value: 90.27154868530998 + - type: euclidean_spearman + value: 84.51542547285167 + - type: main_score + value: 84.51542547285167 + - type: manhattan_pearson + value: 89.71184089391063 + - type: manhattan_spearman + value: 84.51542547285167 + - type: pearson + value: 90.73440998579521 + - type: spearman + value: 84.51542547285167 + task: + type: STS + - dataset: + config: fr + name: MTEB STSBenchmarkMultilingualSTS (fr) + revision: 29afa2569dcedaaa2fe6a3dcfebab33d28b82e8c + split: test + type: mteb/stsb_multi_mt + metrics: + - type: cosine_pearson + value: 79.50928730728984 + - type: cosine_spearman + value: 80.0932172585886 + - type: euclidean_pearson + value: 78.52408843163009 + - type: euclidean_spearman + value: 80.09288222088635 + - type: main_score + value: 80.0932172585886 + - type: manhattan_pearson + value: 78.48826046695687 + - type: manhattan_spearman + value: 80.02496924199357 + - type: pearson + value: 79.50928730728984 + - type: spearman + value: 80.0932172585886 + task: + type: STS + - dataset: + config: default + name: MTEB SummEvalFr + revision: b385812de6a9577b6f4d0f88c6a6e35395a94054 + split: test + type: lyon-nlp/summarization-summeval-fr-p2p + metrics: + - type: cosine_pearson + value: 31.635895833264797 + - type: cosine_spearman + value: 29.316197375472907 + - type: dot_pearson + value: 31.635903578046314 + - type: dot_spearman + value: 29.320397860238174 + - type: main_score + value: 29.316197375472907 + - type: pearson + value: 31.635895833264797 + - type: spearman + value: 29.316197375472907 + task: + type: Summarization + - dataset: + config: default + name: MTEB SyntecReranking + revision: daf0863838cd9e3ba50544cdce3ac2b338a1b0ad + split: test + type: lyon-nlp/mteb-fr-reranking-syntec-s2p + metrics: + - type: main_score + value: 85.96785714285713 + - type: map + value: 85.96785714285713 + - type: mrr + value: 85.96785714285713 + - type: nAUC_map_diff1 + value: 62.8939499970397 + - type: nAUC_map_max + value: 17.78395289837277 + - type: nAUC_map_std + value: 42.06945307317201 + - type: nAUC_mrr_diff1 + value: 62.8939499970397 + - type: nAUC_mrr_max + value: 17.78395289837277 + - type: nAUC_mrr_std + value: 42.06945307317201 + task: + type: Reranking + - dataset: + config: default + name: MTEB SyntecRetrieval + revision: 19661ccdca4dfc2d15122d776b61685f48c68ca9 + split: test + type: lyon-nlp/mteb-fr-retrieval-syntec-s2p + metrics: + - type: main_score + value: 81.745 + - type: map_at_1 + value: 63.0 + - type: map_at_10 + value: 76.325 + - type: map_at_100 + value: 76.413 + - type: map_at_1000 + value: 76.413 + - type: map_at_20 + value: 76.39200000000001 + - type: map_at_3 + value: 74.833 + - type: map_at_5 + value: 76.033 + - type: mrr_at_1 + value: 63.0 + - type: mrr_at_10 + value: 76.32500000000002 + - type: mrr_at_100 + value: 76.41340579710148 + - type: mrr_at_1000 + value: 76.41340579710148 + - type: mrr_at_20 + value: 76.3916666666667 + - type: mrr_at_3 + value: 74.83333333333334 + - type: mrr_at_5 + value: 76.03333333333335 + - type: nauc_map_at_1000_diff1 + value: 54.59590007968431 + - type: nauc_map_at_1000_max + value: 17.62963207909923 + - type: nauc_map_at_1000_std + value: -20.385142659182947 + - type: nauc_map_at_100_diff1 + value: 54.59590007968431 + - type: nauc_map_at_100_max + value: 17.62963207909923 + - type: nauc_map_at_100_std + value: -20.385142659182947 + - type: nauc_map_at_10_diff1 + value: 54.65626490051626 + - type: nauc_map_at_10_max + value: 17.90162538916067 + - type: nauc_map_at_10_std + value: -19.9397305494493 + - type: nauc_map_at_1_diff1 + value: 60.494523574484084 + - type: nauc_map_at_1_max + value: 19.353170685845864 + - type: nauc_map_at_1_std + value: -26.247669189426365 + - type: nauc_map_at_20_diff1 + value: 54.59680474780081 + - type: nauc_map_at_20_max + value: 17.623725255245322 + - type: nauc_map_at_20_std + value: -20.30147444307871 + - type: nauc_map_at_3_diff1 + value: 52.61672286351305 + - type: nauc_map_at_3_max + value: 15.107249942338704 + - type: nauc_map_at_3_std + value: -19.11937528144186 + - type: nauc_map_at_5_diff1 + value: 53.612111533008445 + - type: nauc_map_at_5_max + value: 17.220940492292932 + - type: nauc_map_at_5_std + value: -19.898915353831192 + - type: nauc_mrr_at_1000_diff1 + value: 54.59590007968431 + - type: nauc_mrr_at_1000_max + value: 17.62963207909923 + - type: nauc_mrr_at_1000_std + value: -20.385142659182947 + - type: nauc_mrr_at_100_diff1 + value: 54.59590007968431 + - type: nauc_mrr_at_100_max + value: 17.62963207909923 + - type: nauc_mrr_at_100_std + value: -20.385142659182947 + - type: nauc_mrr_at_10_diff1 + value: 54.65626490051626 + - type: nauc_mrr_at_10_max + value: 17.90162538916067 + - type: nauc_mrr_at_10_std + value: -19.9397305494493 + - type: nauc_mrr_at_1_diff1 + value: 60.494523574484084 + - type: nauc_mrr_at_1_max + value: 19.353170685845864 + - type: nauc_mrr_at_1_std + value: -26.247669189426365 + - type: nauc_mrr_at_20_diff1 + value: 54.59680474780081 + - type: nauc_mrr_at_20_max + value: 17.623725255245322 + - type: nauc_mrr_at_20_std + value: -20.30147444307871 + - type: nauc_mrr_at_3_diff1 + value: 52.61672286351305 + - type: nauc_mrr_at_3_max + value: 15.107249942338704 + - type: nauc_mrr_at_3_std + value: -19.11937528144186 + - type: nauc_mrr_at_5_diff1 + value: 53.612111533008445 + - type: nauc_mrr_at_5_max + value: 17.220940492292932 + - type: nauc_mrr_at_5_std + value: -19.898915353831192 + - type: nauc_ndcg_at_1000_diff1 + value: 54.19476697832697 + - type: nauc_ndcg_at_1000_max + value: 17.552295093432928 + - type: nauc_ndcg_at_1000_std + value: -19.405594751351334 + - type: nauc_ndcg_at_100_diff1 + value: 54.19476697832697 + - type: nauc_ndcg_at_100_max + value: 17.552295093432928 + - type: nauc_ndcg_at_100_std + value: -19.405594751351334 + - type: nauc_ndcg_at_10_diff1 + value: 54.50266178555259 + - type: nauc_ndcg_at_10_max + value: 18.833113705466285 + - type: nauc_ndcg_at_10_std + value: -16.796489600002328 + - type: nauc_ndcg_at_1_diff1 + value: 60.494523574484084 + - type: nauc_ndcg_at_1_max + value: 19.353170685845864 + - type: nauc_ndcg_at_1_std + value: -26.247669189426365 + - type: nauc_ndcg_at_20_diff1 + value: 54.209319885404405 + - type: nauc_ndcg_at_20_max + value: 17.48942074311288 + - type: nauc_ndcg_at_20_std + value: -18.513618512775565 + - type: nauc_ndcg_at_3_diff1 + value: 49.81337689077804 + - type: nauc_ndcg_at_3_max + value: 12.542370633424529 + - type: nauc_ndcg_at_3_std + value: -15.485399782383524 + - type: nauc_ndcg_at_5_diff1 + value: 51.49975132120803 + - type: nauc_ndcg_at_5_max + value: 16.57215179192376 + - type: nauc_ndcg_at_5_std + value: -16.928852236722147 + - type: nauc_precision_at_1000_diff1 + value: .nan + - type: nauc_precision_at_1000_max + value: .nan + - type: nauc_precision_at_1000_std + value: .nan + - type: nauc_precision_at_100_diff1 + value: .nan + - type: nauc_precision_at_100_max + value: .nan + - type: nauc_precision_at_100_std + value: .nan + - type: nauc_precision_at_10_diff1 + value: 63.81886087768404 + - type: nauc_precision_at_10_max + value: 56.13912231559286 + - type: nauc_precision_at_10_std + value: 71.1718020541554 + - type: nauc_precision_at_1_diff1 + value: 60.494523574484084 + - type: nauc_precision_at_1_max + value: 19.353170685845864 + - type: nauc_precision_at_1_std + value: -26.247669189426365 + - type: nauc_precision_at_20_diff1 + value: 55.41549953314738 + - type: nauc_precision_at_20_max + value: 12.278244631185926 + - type: nauc_precision_at_20_std + value: 55.41549953314738 + - type: nauc_precision_at_3_diff1 + value: 31.497043261749212 + - type: nauc_precision_at_3_max + value: -5.192447349310147 + - type: nauc_precision_at_3_std + value: 9.274821039526845 + - type: nauc_precision_at_5_diff1 + value: 23.342670401494065 + - type: nauc_precision_at_5_max + value: 8.228291316527036 + - type: nauc_precision_at_5_std + value: 24.56816059757235 + - type: nauc_recall_at_1000_diff1 + value: .nan + - type: nauc_recall_at_1000_max + value: .nan + - type: nauc_recall_at_1000_std + value: .nan + - type: nauc_recall_at_100_diff1 + value: .nan + - type: nauc_recall_at_100_max + value: .nan + - type: nauc_recall_at_100_std + value: .nan + - type: nauc_recall_at_10_diff1 + value: 63.81886087768457 + - type: nauc_recall_at_10_max + value: 56.13912231559305 + - type: nauc_recall_at_10_std + value: 71.171802054155 + - type: nauc_recall_at_1_diff1 + value: 60.494523574484084 + - type: nauc_recall_at_1_max + value: 19.353170685845864 + - type: nauc_recall_at_1_std + value: -26.247669189426365 + - type: nauc_recall_at_20_diff1 + value: 55.415499533146296 + - type: nauc_recall_at_20_max + value: 12.278244631185359 + - type: nauc_recall_at_20_std + value: 55.415499533146296 + - type: nauc_recall_at_3_diff1 + value: 31.497043261749237 + - type: nauc_recall_at_3_max + value: -5.19244734931015 + - type: nauc_recall_at_3_std + value: 9.274821039526998 + - type: nauc_recall_at_5_diff1 + value: 23.34267040149383 + - type: nauc_recall_at_5_max + value: 8.228291316526445 + - type: nauc_recall_at_5_std + value: 24.56816059757245 + - type: ndcg_at_1 + value: 63.0 + - type: ndcg_at_10 + value: 81.745 + - type: ndcg_at_100 + value: 82.175 + - type: ndcg_at_1000 + value: 82.175 + - type: ndcg_at_20 + value: 81.99499999999999 + - type: ndcg_at_3 + value: 78.964 + - type: ndcg_at_5 + value: 81.074 + - type: precision_at_1 + value: 63.0 + - type: precision_at_10 + value: 9.8 + - type: precision_at_100 + value: 1.0 + - type: precision_at_1000 + value: 0.1 + - type: precision_at_20 + value: 4.95 + - type: precision_at_3 + value: 30.333 + - type: precision_at_5 + value: 19.2 + - type: recall_at_1 + value: 63.0 + - type: recall_at_10 + value: 98.0 + - type: recall_at_100 + value: 100.0 + - type: recall_at_1000 + value: 100.0 + - type: recall_at_20 + value: 99.0 + - type: recall_at_3 + value: 91.0 + - type: recall_at_5 + value: 96.0 + task: + type: Retrieval + - dataset: + config: fra-fra + name: MTEB XPQARetrieval (fr) + revision: c99d599f0a6ab9b85b065da6f9d94f9cf731679f + split: test + type: jinaai/xpqa + metrics: + - type: main_score + value: 59.299 + - type: map_at_1 + value: 33.498 + - type: map_at_10 + value: 52.705999999999996 + - type: map_at_100 + value: 54.278999999999996 + - type: map_at_1000 + value: 54.358 + - type: map_at_20 + value: 53.599 + - type: map_at_3 + value: 46.848 + - type: map_at_5 + value: 50.483999999999995 + - type: mrr_at_1 + value: 54.33911882510013 + - type: mrr_at_10 + value: 62.24383834530695 + - type: mrr_at_100 + value: 62.910121526268924 + - type: mrr_at_1000 + value: 62.934836010235195 + - type: mrr_at_20 + value: 62.64855949718011 + - type: mrr_at_3 + value: 60.23587004895412 + - type: mrr_at_5 + value: 61.277258566978155 + - type: nauc_map_at_1000_diff1 + value: 50.40068320668247 + - type: nauc_map_at_1000_max + value: 45.419834912711345 + - type: nauc_map_at_1000_std + value: 4.690219602260963 + - type: nauc_map_at_100_diff1 + value: 50.36635447882883 + - type: nauc_map_at_100_max + value: 45.4147569946384 + - type: nauc_map_at_100_std + value: 4.6979944201303825 + - type: nauc_map_at_10_diff1 + value: 50.2118074250129 + - type: nauc_map_at_10_max + value: 44.94969058947611 + - type: nauc_map_at_10_std + value: 4.398792604612038 + - type: nauc_map_at_1_diff1 + value: 58.603849067897364 + - type: nauc_map_at_1_max + value: 27.905162533509053 + - type: nauc_map_at_1_std + value: -0.3803212098210696 + - type: nauc_map_at_20_diff1 + value: 50.251993141938534 + - type: nauc_map_at_20_max + value: 45.19947447689957 + - type: nauc_map_at_20_std + value: 4.513253526305713 + - type: nauc_map_at_3_diff1 + value: 50.868419013808655 + - type: nauc_map_at_3_max + value: 38.7440731652971 + - type: nauc_map_at_3_std + value: 2.1235171982780567 + - type: nauc_map_at_5_diff1 + value: 50.48944050454372 + - type: nauc_map_at_5_max + value: 42.85070909512206 + - type: nauc_map_at_5_std + value: 3.7050886992455614 + - type: nauc_mrr_at_1000_diff1 + value: 57.229927096025335 + - type: nauc_mrr_at_1000_max + value: 50.938449952909025 + - type: nauc_mrr_at_1000_std + value: 6.772124463005406 + - type: nauc_mrr_at_100_diff1 + value: 57.2303932927905 + - type: nauc_mrr_at_100_max + value: 50.94674655436572 + - type: nauc_mrr_at_100_std + value: 6.771170622225867 + - type: nauc_mrr_at_10_diff1 + value: 57.140428095767994 + - type: nauc_mrr_at_10_max + value: 51.015650650701325 + - type: nauc_mrr_at_10_std + value: 6.786343260588584 + - type: nauc_mrr_at_1_diff1 + value: 59.61237414732749 + - type: nauc_mrr_at_1_max + value: 50.27947885239657 + - type: nauc_mrr_at_1_std + value: 6.325448424227606 + - type: nauc_mrr_at_20_diff1 + value: 57.24043520066768 + - type: nauc_mrr_at_20_max + value: 51.02257975115745 + - type: nauc_mrr_at_20_std + value: 6.849295913300065 + - type: nauc_mrr_at_3_diff1 + value: 57.48754760533025 + - type: nauc_mrr_at_3_max + value: 50.34268453986772 + - type: nauc_mrr_at_3_std + value: 5.576028856206244 + - type: nauc_mrr_at_5_diff1 + value: 57.01340095945496 + - type: nauc_mrr_at_5_max + value: 50.37124700329385 + - type: nauc_mrr_at_5_std + value: 6.176803150164063 + - type: nauc_ndcg_at_1000_diff1 + value: 51.877968132074926 + - type: nauc_ndcg_at_1000_max + value: 48.36110672671821 + - type: nauc_ndcg_at_1000_std + value: 6.108794230045721 + - type: nauc_ndcg_at_100_diff1 + value: 51.404027251165715 + - type: nauc_ndcg_at_100_max + value: 48.42673179365166 + - type: nauc_ndcg_at_100_std + value: 6.409635305448258 + - type: nauc_ndcg_at_10_diff1 + value: 50.555165677188306 + - type: nauc_ndcg_at_10_max + value: 47.47519310008686 + - type: nauc_ndcg_at_10_std + value: 5.4093301966396385 + - type: nauc_ndcg_at_1_diff1 + value: 59.61237414732749 + - type: nauc_ndcg_at_1_max + value: 50.27947885239657 + - type: nauc_ndcg_at_1_std + value: 6.325448424227606 + - type: nauc_ndcg_at_20_diff1 + value: 50.72341403581928 + - type: nauc_ndcg_at_20_max + value: 47.827204708689806 + - type: nauc_ndcg_at_20_std + value: 5.697769161434286 + - type: nauc_ndcg_at_3_diff1 + value: 51.3008421472128 + - type: nauc_ndcg_at_3_max + value: 45.246290747859376 + - type: nauc_ndcg_at_3_std + value: 3.0258563412678208 + - type: nauc_ndcg_at_5_diff1 + value: 50.75750841863859 + - type: nauc_ndcg_at_5_max + value: 44.6012523853242 + - type: nauc_ndcg_at_5_std + value: 4.171558602586512 + - type: nauc_precision_at_1000_diff1 + value: -10.620804642735912 + - type: nauc_precision_at_1000_max + value: 20.710188752354686 + - type: nauc_precision_at_1000_std + value: 8.004079823209057 + - type: nauc_precision_at_100_diff1 + value: -6.003091379108357 + - type: nauc_precision_at_100_max + value: 26.818229597933264 + - type: nauc_precision_at_100_std + value: 9.991598973813758 + - type: nauc_precision_at_10_diff1 + value: 5.876190205156904 + - type: nauc_precision_at_10_max + value: 38.075509886425046 + - type: nauc_precision_at_10_std + value: 8.785425849446119 + - type: nauc_precision_at_1_diff1 + value: 59.61237414732749 + - type: nauc_precision_at_1_max + value: 50.27947885239657 + - type: nauc_precision_at_1_std + value: 6.325448424227606 + - type: nauc_precision_at_20_diff1 + value: 1.9253058099019142 + - type: nauc_precision_at_20_max + value: 34.8886581006359 + - type: nauc_precision_at_20_std + value: 9.120051183604845 + - type: nauc_precision_at_3_diff1 + value: 17.487309868406694 + - type: nauc_precision_at_3_max + value: 41.703264394864284 + - type: nauc_precision_at_3_std + value: 6.565690923768522 + - type: nauc_precision_at_5_diff1 + value: 11.444446099569587 + - type: nauc_precision_at_5_max + value: 40.40685598234426 + - type: nauc_precision_at_5_std + value: 8.761851721129617 + - type: nauc_recall_at_1000_diff1 + value: 30.356073066469964 + - type: nauc_recall_at_1000_max + value: 59.34211677701755 + - type: nauc_recall_at_1000_std + value: 45.85472480324881 + - type: nauc_recall_at_100_diff1 + value: 37.197227716845035 + - type: nauc_recall_at_100_max + value: 46.43271901177922 + - type: nauc_recall_at_100_std + value: 14.080669406474067 + - type: nauc_recall_at_10_diff1 + value: 40.93453402067081 + - type: nauc_recall_at_10_max + value: 44.03007002975281 + - type: nauc_recall_at_10_std + value: 5.656063496811249 + - type: nauc_recall_at_1_diff1 + value: 58.603849067897364 + - type: nauc_recall_at_1_max + value: 27.905162533509053 + - type: nauc_recall_at_1_std + value: -0.3803212098210696 + - type: nauc_recall_at_20_diff1 + value: 38.76049220154408 + - type: nauc_recall_at_20_max + value: 43.04322846027003 + - type: nauc_recall_at_20_std + value: 5.986817583151329 + - type: nauc_recall_at_3_diff1 + value: 45.480809900773586 + - type: nauc_recall_at_3_max + value: 34.175287910561075 + - type: nauc_recall_at_3_std + value: -0.9428328336656849 + - type: nauc_recall_at_5_diff1 + value: 43.134263613996495 + - type: nauc_recall_at_5_max + value: 38.76946986340381 + - type: nauc_recall_at_5_std + value: 2.496846951162451 + - type: ndcg_at_1 + value: 54.339000000000006 + - type: ndcg_at_10 + value: 59.299 + - type: ndcg_at_100 + value: 64.895 + - type: ndcg_at_1000 + value: 66.283 + - type: ndcg_at_20 + value: 61.594 + - type: ndcg_at_3 + value: 54.176 + - type: ndcg_at_5 + value: 55.730999999999995 + - type: precision_at_1 + value: 54.339000000000006 + - type: precision_at_10 + value: 14.139 + - type: precision_at_100 + value: 1.879 + - type: precision_at_1000 + value: 0.20600000000000002 + - type: precision_at_20 + value: 7.843999999999999 + - type: precision_at_3 + value: 33.289 + - type: precision_at_5 + value: 24.005000000000003 + - type: recall_at_1 + value: 33.498 + - type: recall_at_10 + value: 68.389 + - type: recall_at_100 + value: 90.398 + - type: recall_at_1000 + value: 99.51 + - type: recall_at_20 + value: 75.957 + - type: recall_at_3 + value: 52.452 + - type: recall_at_5 + value: 59.9 + task: + type: Retrieval + - dataset: + config: eng-fra + name: MTEB XPQARetrieval (eng-fra) + revision: c99d599f0a6ab9b85b065da6f9d94f9cf731679f + split: test + type: jinaai/xpqa + metrics: + - type: main_score + value: 34.148 + - type: map_at_1 + value: 13.142000000000001 + - type: map_at_10 + value: 27.504 + - type: map_at_100 + value: 30.122 + - type: map_at_1000 + value: 30.259000000000004 + - type: map_at_20 + value: 29.012999999999998 + - type: map_at_3 + value: 22.078 + - type: map_at_5 + value: 25.16 + - type: mrr_at_1 + value: 26.034712950600802 + - type: mrr_at_10 + value: 35.21965795664058 + - type: mrr_at_100 + value: 36.66580650341633 + - type: mrr_at_1000 + value: 36.719052382107876 + - type: mrr_at_20 + value: 36.16989730077047 + - type: mrr_at_3 + value: 32.51001335113486 + - type: mrr_at_5 + value: 34.02536715620828 + - type: nauc_map_at_1000_diff1 + value: 22.045384824751874 + - type: nauc_map_at_1000_max + value: 37.25875104070248 + - type: nauc_map_at_1000_std + value: 6.311979763063837 + - type: nauc_map_at_100_diff1 + value: 21.983455240362048 + - type: nauc_map_at_100_max + value: 37.2389940633854 + - type: nauc_map_at_100_std + value: 6.308890135460129 + - type: nauc_map_at_10_diff1 + value: 22.08654364936706 + - type: nauc_map_at_10_max + value: 36.83912535726259 + - type: nauc_map_at_10_std + value: 5.586878201266457 + - type: nauc_map_at_1_diff1 + value: 27.996481625958005 + - type: nauc_map_at_1_max + value: 22.170519197242168 + - type: nauc_map_at_1_std + value: 3.2754228003309094 + - type: nauc_map_at_20_diff1 + value: 22.084045154169598 + - type: nauc_map_at_20_max + value: 37.35198536276873 + - type: nauc_map_at_20_std + value: 6.287528082319668 + - type: nauc_map_at_3_diff1 + value: 23.518184109219312 + - type: nauc_map_at_3_max + value: 32.37508088252643 + - type: nauc_map_at_3_std + value: 2.793040909420183 + - type: nauc_map_at_5_diff1 + value: 22.617850315991227 + - type: nauc_map_at_5_max + value: 35.40344848186382 + - type: nauc_map_at_5_std + value: 4.296806736798404 + - type: nauc_mrr_at_1000_diff1 + value: 21.45583729151993 + - type: nauc_mrr_at_1000_max + value: 37.080805671948056 + - type: nauc_mrr_at_1000_std + value: 8.905986716816184 + - type: nauc_mrr_at_100_diff1 + value: 21.43908621145099 + - type: nauc_mrr_at_100_max + value: 37.062698361870055 + - type: nauc_mrr_at_100_std + value: 8.917647212658137 + - type: nauc_mrr_at_10_diff1 + value: 21.328903056739907 + - type: nauc_mrr_at_10_max + value: 36.77272980951551 + - type: nauc_mrr_at_10_std + value: 8.323076678764995 + - type: nauc_mrr_at_1_diff1 + value: 24.15510508158492 + - type: nauc_mrr_at_1_max + value: 38.16775944594129 + - type: nauc_mrr_at_1_std + value: 9.048540715004725 + - type: nauc_mrr_at_20_diff1 + value: 21.379354375668658 + - type: nauc_mrr_at_20_max + value: 37.0178596112037 + - type: nauc_mrr_at_20_std + value: 8.95011471773535 + - type: nauc_mrr_at_3_diff1 + value: 21.550470686773544 + - type: nauc_mrr_at_3_max + value: 37.923958816756326 + - type: nauc_mrr_at_3_std + value: 8.621492240346312 + - type: nauc_mrr_at_5_diff1 + value: 21.456166538305947 + - type: nauc_mrr_at_5_max + value: 37.43457454088489 + - type: nauc_mrr_at_5_std + value: 8.700414540783243 + - type: nauc_ndcg_at_1000_diff1 + value: 20.687215881405972 + - type: nauc_ndcg_at_1000_max + value: 37.07894735512217 + - type: nauc_ndcg_at_1000_std + value: 7.955407995753924 + - type: nauc_ndcg_at_100_diff1 + value: 19.899521579418327 + - type: nauc_ndcg_at_100_max + value: 36.7917656290794 + - type: nauc_ndcg_at_100_std + value: 8.815031884683332 + - type: nauc_ndcg_at_10_diff1 + value: 20.59162274283143 + - type: nauc_ndcg_at_10_max + value: 36.21724206012231 + - type: nauc_ndcg_at_10_std + value: 6.304038242601423 + - type: nauc_ndcg_at_1_diff1 + value: 24.15510508158492 + - type: nauc_ndcg_at_1_max + value: 38.16775944594129 + - type: nauc_ndcg_at_1_std + value: 9.048540715004725 + - type: nauc_ndcg_at_20_diff1 + value: 20.559881435708498 + - type: nauc_ndcg_at_20_max + value: 37.30561288551007 + - type: nauc_ndcg_at_20_std + value: 8.41336133183866 + - type: nauc_ndcg_at_3_diff1 + value: 21.41525059572959 + - type: nauc_ndcg_at_3_max + value: 36.22187375662584 + - type: nauc_ndcg_at_3_std + value: 4.765376474543177 + - type: nauc_ndcg_at_5_diff1 + value: 21.423152693078066 + - type: nauc_ndcg_at_5_max + value: 35.62314257936377 + - type: nauc_ndcg_at_5_std + value: 5.292965569254064 + - type: nauc_precision_at_1000_diff1 + value: -1.4052165408318342 + - type: nauc_precision_at_1000_max + value: 20.517983706015833 + - type: nauc_precision_at_1000_std + value: 5.528186523222815 + - type: nauc_precision_at_100_diff1 + value: 1.1467871741865474 + - type: nauc_precision_at_100_max + value: 27.33879537667429 + - type: nauc_precision_at_100_std + value: 10.558834828955572 + - type: nauc_precision_at_10_diff1 + value: 9.212502568904506 + - type: nauc_precision_at_10_max + value: 41.282116927731224 + - type: nauc_precision_at_10_std + value: 10.635469423053907 + - type: nauc_precision_at_1_diff1 + value: 24.15510508158492 + - type: nauc_precision_at_1_max + value: 38.16775944594129 + - type: nauc_precision_at_1_std + value: 9.048540715004725 + - type: nauc_precision_at_20_diff1 + value: 7.720793691116737 + - type: nauc_precision_at_20_max + value: 38.29599512358724 + - type: nauc_precision_at_20_std + value: 11.891637673436277 + - type: nauc_precision_at_3_diff1 + value: 13.834429473708909 + - type: nauc_precision_at_3_max + value: 43.42560369973884 + - type: nauc_precision_at_3_std + value: 7.177658330615482 + - type: nauc_precision_at_5_diff1 + value: 11.832909517425021 + - type: nauc_precision_at_5_max + value: 43.33773335597342 + - type: nauc_precision_at_5_std + value: 9.202718262478298 + - type: nauc_recall_at_1000_diff1 + value: -39.28871527051969 + - type: nauc_recall_at_1000_max + value: -26.37396882156915 + - type: nauc_recall_at_1000_std + value: 21.2304966569008 + - type: nauc_recall_at_100_diff1 + value: 7.038804971203592 + - type: nauc_recall_at_100_max + value: 27.81013426649376 + - type: nauc_recall_at_100_std + value: 19.55818251557495 + - type: nauc_recall_at_10_diff1 + value: 16.372446751980963 + - type: nauc_recall_at_10_max + value: 29.383411324814325 + - type: nauc_recall_at_10_std + value: 4.030124533628888 + - type: nauc_recall_at_1_diff1 + value: 27.996481625958005 + - type: nauc_recall_at_1_max + value: 22.170519197242168 + - type: nauc_recall_at_1_std + value: 3.2754228003309094 + - type: nauc_recall_at_20_diff1 + value: 15.454183279031705 + - type: nauc_recall_at_20_max + value: 32.477787087697465 + - type: nauc_recall_at_20_std + value: 11.814027056017764 + - type: nauc_recall_at_3_diff1 + value: 20.116262186422457 + - type: nauc_recall_at_3_max + value: 28.490372254786017 + - type: nauc_recall_at_3_std + value: 0.8537023418579993 + - type: nauc_recall_at_5_diff1 + value: 18.369329264777864 + - type: nauc_recall_at_5_max + value: 29.74100621863871 + - type: nauc_recall_at_5_std + value: 2.713328963890598 + - type: ndcg_at_1 + value: 26.035000000000004 + - type: ndcg_at_10 + value: 34.148 + - type: ndcg_at_100 + value: 44.027 + - type: ndcg_at_1000 + value: 46.544999999999995 + - type: ndcg_at_20 + value: 38.395 + - type: ndcg_at_3 + value: 28.161 + - type: ndcg_at_5 + value: 30.062 + - type: precision_at_1 + value: 26.035000000000004 + - type: precision_at_10 + value: 10.414 + - type: precision_at_100 + value: 1.8929999999999998 + - type: precision_at_1000 + value: 0.22300000000000003 + - type: precision_at_20 + value: 6.722 + - type: precision_at_3 + value: 20.427 + - type: precision_at_5 + value: 16.101 + - type: recall_at_1 + value: 13.142000000000001 + - type: recall_at_10 + value: 44.751000000000005 + - type: recall_at_100 + value: 82.86200000000001 + - type: recall_at_1000 + value: 99.381 + - type: recall_at_20 + value: 58.29600000000001 + - type: recall_at_3 + value: 27.262999999999998 + - type: recall_at_5 + value: 34.795 + task: + type: Retrieval + - dataset: + config: fra-eng + name: MTEB XPQARetrieval (fra-eng) + revision: c99d599f0a6ab9b85b065da6f9d94f9cf731679f + split: test + type: jinaai/xpqa + metrics: + - type: main_score + value: 51.237 + - type: map_at_1 + value: 26.733 + - type: map_at_10 + value: 44.643 + - type: map_at_100 + value: 46.404 + - type: map_at_1000 + value: 46.516000000000005 + - type: map_at_20 + value: 45.659 + - type: map_at_3 + value: 39.358 + - type: map_at_5 + value: 42.784 + - type: mrr_at_1 + value: 43.65821094793058 + - type: mrr_at_10 + value: 53.10609913747425 + - type: mrr_at_100 + value: 53.857579329641105 + - type: mrr_at_1000 + value: 53.89660252126757 + - type: mrr_at_20 + value: 53.52067135248109 + - type: mrr_at_3 + value: 50.71206052514461 + - type: mrr_at_5 + value: 52.167334223408936 + - type: nauc_map_at_1000_diff1 + value: 45.24362379618829 + - type: nauc_map_at_1000_max + value: 44.81176515141581 + - type: nauc_map_at_1000_std + value: -0.8010490746117229 + - type: nauc_map_at_100_diff1 + value: 45.2119434987754 + - type: nauc_map_at_100_max + value: 44.80914165276905 + - type: nauc_map_at_100_std + value: -0.7698807481565286 + - type: nauc_map_at_10_diff1 + value: 45.15860057920748 + - type: nauc_map_at_10_max + value: 44.59910057286427 + - type: nauc_map_at_10_std + value: -1.112850247343371 + - type: nauc_map_at_1_diff1 + value: 52.51086259511314 + - type: nauc_map_at_1_max + value: 31.357215864716526 + - type: nauc_map_at_1_std + value: -3.3316393174260495 + - type: nauc_map_at_20_diff1 + value: 45.0313164103322 + - type: nauc_map_at_20_max + value: 44.807072544620915 + - type: nauc_map_at_20_std + value: -0.9759739768751601 + - type: nauc_map_at_3_diff1 + value: 46.607279248808844 + - type: nauc_map_at_3_max + value: 40.93560787464522 + - type: nauc_map_at_3_std + value: -3.4220516437046884 + - type: nauc_map_at_5_diff1 + value: 45.483519829247705 + - type: nauc_map_at_5_max + value: 43.47963235378117 + - type: nauc_map_at_5_std + value: -2.1213782659613245 + - type: nauc_mrr_at_1000_diff1 + value: 49.84990195876864 + - type: nauc_mrr_at_1000_max + value: 47.66319451546816 + - type: nauc_mrr_at_1000_std + value: -0.43985061175300905 + - type: nauc_mrr_at_100_diff1 + value: 49.83955797982952 + - type: nauc_mrr_at_100_max + value: 47.66572696957632 + - type: nauc_mrr_at_100_std + value: -0.4121021416689912 + - type: nauc_mrr_at_10_diff1 + value: 49.709689784147315 + - type: nauc_mrr_at_10_max + value: 47.695007048763955 + - type: nauc_mrr_at_10_std + value: -0.4323849817636193 + - type: nauc_mrr_at_1_diff1 + value: 52.630796934345135 + - type: nauc_mrr_at_1_max + value: 47.857589062201896 + - type: nauc_mrr_at_1_std + value: -1.8788772946768062 + - type: nauc_mrr_at_20_diff1 + value: 49.62435690034187 + - type: nauc_mrr_at_20_max + value: 47.672431418226516 + - type: nauc_mrr_at_20_std + value: -0.3853547894450724 + - type: nauc_mrr_at_3_diff1 + value: 50.42320496620572 + - type: nauc_mrr_at_3_max + value: 47.32934444593811 + - type: nauc_mrr_at_3_std + value: -1.5796513158364918 + - type: nauc_mrr_at_5_diff1 + value: 50.156844538277454 + - type: nauc_mrr_at_5_max + value: 47.500134983097176 + - type: nauc_mrr_at_5_std + value: -1.2044529262681698 + - type: nauc_ndcg_at_1000_diff1 + value: 45.83174871455657 + - type: nauc_ndcg_at_1000_max + value: 46.22325612922571 + - type: nauc_ndcg_at_1000_std + value: 0.7251836382749244 + - type: nauc_ndcg_at_100_diff1 + value: 45.26328676530337 + - type: nauc_ndcg_at_100_max + value: 46.32944733493364 + - type: nauc_ndcg_at_100_std + value: 1.865625075918929 + - type: nauc_ndcg_at_10_diff1 + value: 44.537492108596425 + - type: nauc_ndcg_at_10_max + value: 45.87478673164907 + - type: nauc_ndcg_at_10_std + value: 0.41953467722933185 + - type: nauc_ndcg_at_1_diff1 + value: 52.630796934345135 + - type: nauc_ndcg_at_1_max + value: 47.857589062201896 + - type: nauc_ndcg_at_1_std + value: -1.8788772946768062 + - type: nauc_ndcg_at_20_diff1 + value: 44.05058836339881 + - type: nauc_ndcg_at_20_max + value: 46.428186099703915 + - type: nauc_ndcg_at_20_std + value: 0.9443804586404068 + - type: nauc_ndcg_at_3_diff1 + value: 45.53948737812581 + - type: nauc_ndcg_at_3_max + value: 44.55019513996879 + - type: nauc_ndcg_at_3_std + value: -3.109851951092217 + - type: nauc_ndcg_at_5_diff1 + value: 45.41173383961928 + - type: nauc_ndcg_at_5_max + value: 44.560765413275305 + - type: nauc_ndcg_at_5_std + value: -1.6824415034098137 + - type: nauc_precision_at_1000_diff1 + value: -7.887083238422578 + - type: nauc_precision_at_1000_max + value: 12.124801121563747 + - type: nauc_precision_at_1000_std + value: 4.673924336255192 + - type: nauc_precision_at_100_diff1 + value: -2.1326427402923387 + - type: nauc_precision_at_100_max + value: 19.727068109516082 + - type: nauc_precision_at_100_std + value: 7.98949733333232 + - type: nauc_precision_at_10_diff1 + value: 9.723989241310061 + - type: nauc_precision_at_10_max + value: 36.63938764576557 + - type: nauc_precision_at_10_std + value: 5.237699304811818 + - type: nauc_precision_at_1_diff1 + value: 52.630796934345135 + - type: nauc_precision_at_1_max + value: 47.857589062201896 + - type: nauc_precision_at_1_std + value: -1.8788772946768062 + - type: nauc_precision_at_20_diff1 + value: 4.81172201728637 + - type: nauc_precision_at_20_max + value: 31.803554548820266 + - type: nauc_precision_at_20_std + value: 5.726820012959347 + - type: nauc_precision_at_3_diff1 + value: 21.35717778539221 + - type: nauc_precision_at_3_max + value: 40.745356004203884 + - type: nauc_precision_at_3_std + value: -1.1539362625545309 + - type: nauc_precision_at_5_diff1 + value: 15.320566766870824 + - type: nauc_precision_at_5_max + value: 39.57305927905734 + - type: nauc_precision_at_5_std + value: 2.0380912770385855 + - type: nauc_recall_at_1000_diff1 + value: 53.58291277355397 + - type: nauc_recall_at_1000_max + value: 54.274398762111744 + - type: nauc_recall_at_1000_std + value: 11.337487733101206 + - type: nauc_recall_at_100_diff1 + value: 32.3382519427821 + - type: nauc_recall_at_100_max + value: 43.398292158243606 + - type: nauc_recall_at_100_std + value: 20.104107748951193 + - type: nauc_recall_at_10_diff1 + value: 34.89991099982832 + - type: nauc_recall_at_10_max + value: 42.306920166018436 + - type: nauc_recall_at_10_std + value: 4.39688933264661 + - type: nauc_recall_at_1_diff1 + value: 52.51086259511314 + - type: nauc_recall_at_1_max + value: 31.357215864716526 + - type: nauc_recall_at_1_std + value: -3.3316393174260495 + - type: nauc_recall_at_20_diff1 + value: 31.389778218413884 + - type: nauc_recall_at_20_max + value: 43.91189150158487 + - type: nauc_recall_at_20_std + value: 6.823493011161174 + - type: nauc_recall_at_3_diff1 + value: 41.41055013171579 + - type: nauc_recall_at_3_max + value: 37.636926573569205 + - type: nauc_recall_at_3_std + value: -4.148275870225593 + - type: nauc_recall_at_5_diff1 + value: 38.28078114262306 + - type: nauc_recall_at_5_max + value: 40.03724369846101 + - type: nauc_recall_at_5_std + value: -1.5995298392319341 + - type: ndcg_at_1 + value: 43.658 + - type: ndcg_at_10 + value: 51.237 + - type: ndcg_at_100 + value: 57.658 + - type: ndcg_at_1000 + value: 59.628 + - type: ndcg_at_20 + value: 53.848 + - type: ndcg_at_3 + value: 46.082 + - type: ndcg_at_5 + value: 48.055 + - type: precision_at_1 + value: 43.658 + - type: precision_at_10 + value: 12.537 + - type: precision_at_100 + value: 1.806 + - type: precision_at_1000 + value: 0.20600000000000002 + - type: precision_at_20 + value: 7.19 + - type: precision_at_3 + value: 29.372 + - type: precision_at_5 + value: 21.389 + - type: recall_at_1 + value: 26.733 + - type: recall_at_10 + value: 61.248000000000005 + - type: recall_at_100 + value: 86.37700000000001 + - type: recall_at_1000 + value: 99.42999999999999 + - type: recall_at_20 + value: 69.686 + - type: recall_at_3 + value: 45.343 + - type: recall_at_5 + value: 53.516 + task: + type: Retrieval + - dataset: + config: default + name: MTEB AllegroReviews + revision: b89853e6de927b0e3bfa8ecc0e56fe4e02ceafc6 + split: test + type: PL-MTEB/allegro-reviews + metrics: + - type: accuracy + value: 53.359840954274354 + - type: f1 + value: 41.06302546252184 + - type: f1_weighted + value: 49.87243147330794 + - type: main_score + value: 53.359840954274354 + task: + type: Classification + - dataset: + config: default + name: MTEB ArguAna-PL + revision: 63fc86750af76253e8c760fc9e534bbf24d260a2 + split: test + type: clarin-knext/arguana-pl + metrics: + - type: main_score + value: 48.403 + - type: map_at_1 + value: 23.471 + - type: map_at_10 + value: 39.216 + - type: map_at_100 + value: 40.29 + - type: map_at_1000 + value: 40.300000000000004 + - type: map_at_20 + value: 40.042 + - type: map_at_3 + value: 33.95 + - type: map_at_5 + value: 36.955 + - type: mrr_at_1 + value: 23.75533428165007 + - type: mrr_at_10 + value: 39.335613809297975 + - type: mrr_at_100 + value: 40.39627181086039 + - type: mrr_at_1000 + value: 40.40599566225043 + - type: mrr_at_20 + value: 40.14809098074531 + - type: mrr_at_3 + value: 34.03271692745371 + - type: mrr_at_5 + value: 37.11593172119479 + - type: nauc_map_at_1000_diff1 + value: 5.3772331730748855 + - type: nauc_map_at_1000_max + value: -2.0329411151791534 + - type: nauc_map_at_1000_std + value: -3.86511182416739 + - type: nauc_map_at_100_diff1 + value: 5.3794850605444156 + - type: nauc_map_at_100_max + value: -2.0168450480018345 + - type: nauc_map_at_100_std + value: -3.842530148996824 + - type: nauc_map_at_10_diff1 + value: 4.9964237351426855 + - type: nauc_map_at_10_max + value: -2.144884912174717 + - type: nauc_map_at_10_std + value: -4.461427392019594 + - type: nauc_map_at_1_diff1 + value: 9.516688551341105 + - type: nauc_map_at_1_max + value: -5.644910020587747 + - type: nauc_map_at_1_std + value: -3.6198980121175817 + - type: nauc_map_at_20_diff1 + value: 5.353229630043072 + - type: nauc_map_at_20_max + value: -1.800952542288357 + - type: nauc_map_at_20_std + value: -3.906531793826726 + - type: nauc_map_at_3_diff1 + value: 5.255232382713057 + - type: nauc_map_at_3_max + value: -3.0281799752800045 + - type: nauc_map_at_3_std + value: -4.075526006428358 + - type: nauc_map_at_5_diff1 + value: 4.734362539624116 + - type: nauc_map_at_5_max + value: -2.503481319824773 + - type: nauc_map_at_5_std + value: -3.872895860744626 + - type: nauc_mrr_at_1000_diff1 + value: 4.4146985529959215 + - type: nauc_mrr_at_1000_max + value: -2.299598690474009 + - type: nauc_mrr_at_1000_std + value: -3.9402783361244085 + - type: nauc_mrr_at_100_diff1 + value: 4.417258165352108 + - type: nauc_mrr_at_100_max + value: -2.283454878974528 + - type: nauc_mrr_at_100_std + value: -3.9177269448785594 + - type: nauc_mrr_at_10_diff1 + value: 4.03684960733421 + - type: nauc_mrr_at_10_max + value: -2.396241516141588 + - type: nauc_mrr_at_10_std + value: -4.556329903468349 + - type: nauc_mrr_at_1_diff1 + value: 8.550764063204086 + - type: nauc_mrr_at_1_max + value: -5.2653218814611655 + - type: nauc_mrr_at_1_std + value: -3.7478078638245558 + - type: nauc_mrr_at_20_diff1 + value: 4.399100178540589 + - type: nauc_mrr_at_20_max + value: -2.065687810815654 + - type: nauc_mrr_at_20_std + value: -3.980856869079938 + - type: nauc_mrr_at_3_diff1 + value: 4.356955681808423 + - type: nauc_mrr_at_3_max + value: -3.305266169391567 + - type: nauc_mrr_at_3_std + value: -4.187435812922289 + - type: nauc_mrr_at_5_diff1 + value: 3.797470912759614 + - type: nauc_mrr_at_5_max + value: -2.7476549709181355 + - type: nauc_mrr_at_5_std + value: -3.9744581781815724 + - type: nauc_ndcg_at_1000_diff1 + value: 5.138474520701041 + - type: nauc_ndcg_at_1000_max + value: -0.8548037167905302 + - type: nauc_ndcg_at_1000_std + value: -3.4098747932546734 + - type: nauc_ndcg_at_100_diff1 + value: 5.264678453330276 + - type: nauc_ndcg_at_100_max + value: -0.39210055156714196 + - type: nauc_ndcg_at_100_std + value: -2.790588063731217 + - type: nauc_ndcg_at_10_diff1 + value: 3.7503459140558943 + - type: nauc_ndcg_at_10_max + value: -0.3384766390247899 + - type: nauc_ndcg_at_10_std + value: -5.575339046932562 + - type: nauc_ndcg_at_1_diff1 + value: 9.516688551341105 + - type: nauc_ndcg_at_1_max + value: -5.644910020587747 + - type: nauc_ndcg_at_1_std + value: -3.6198980121175817 + - type: nauc_ndcg_at_20_diff1 + value: 5.270936471430413 + - type: nauc_ndcg_at_20_max + value: 1.291103316253361 + - type: nauc_ndcg_at_20_std + value: -3.169503489721584 + - type: nauc_ndcg_at_3_diff1 + value: 4.271651754613586 + - type: nauc_ndcg_at_3_max + value: -2.353280506415016 + - type: nauc_ndcg_at_3_std + value: -4.364225258109044 + - type: nauc_ndcg_at_5_diff1 + value: 3.3040691006732765 + - type: nauc_ndcg_at_5_max + value: -1.303365264172454 + - type: nauc_ndcg_at_5_std + value: -3.955958872876751 + - type: nauc_precision_at_1000_diff1 + value: 13.241505124398486 + - type: nauc_precision_at_1000_max + value: 47.97010130202494 + - type: nauc_precision_at_1000_std + value: 74.8878850565087 + - type: nauc_precision_at_100_diff1 + value: 19.347701423046445 + - type: nauc_precision_at_100_max + value: 50.96543811338734 + - type: nauc_precision_at_100_std + value: 67.94488954909981 + - type: nauc_precision_at_10_diff1 + value: -1.5877710682220694 + - type: nauc_precision_at_10_max + value: 8.339455759028867 + - type: nauc_precision_at_10_std + value: -11.521446364855805 + - type: nauc_precision_at_1_diff1 + value: 9.516688551341105 + - type: nauc_precision_at_1_max + value: -5.644910020587747 + - type: nauc_precision_at_1_std + value: -3.6198980121175817 + - type: nauc_precision_at_20_diff1 + value: 9.056182158091447 + - type: nauc_precision_at_20_max + value: 32.304791476693964 + - type: nauc_precision_at_20_std + value: 7.405945565001408 + - type: nauc_precision_at_3_diff1 + value: 1.678810658886418 + - type: nauc_precision_at_3_max + value: -0.5473747492361274 + - type: nauc_precision_at_3_std + value: -5.194579239515215 + - type: nauc_precision_at_5_diff1 + value: -1.061704237231085 + - type: nauc_precision_at_5_max + value: 2.5051492439705387 + - type: nauc_precision_at_5_std + value: -4.155138551797193 + - type: nauc_recall_at_1000_diff1 + value: 13.241505124400325 + - type: nauc_recall_at_1000_max + value: 47.97010130202325 + - type: nauc_recall_at_1000_std + value: 74.88788505650888 + - type: nauc_recall_at_100_diff1 + value: 19.34770142304624 + - type: nauc_recall_at_100_max + value: 50.9654381133864 + - type: nauc_recall_at_100_std + value: 67.9448895490988 + - type: nauc_recall_at_10_diff1 + value: -1.587771068221971 + - type: nauc_recall_at_10_max + value: 8.339455759028958 + - type: nauc_recall_at_10_std + value: -11.521446364855827 + - type: nauc_recall_at_1_diff1 + value: 9.516688551341105 + - type: nauc_recall_at_1_max + value: -5.644910020587747 + - type: nauc_recall_at_1_std + value: -3.6198980121175817 + - type: nauc_recall_at_20_diff1 + value: 9.056182158091504 + - type: nauc_recall_at_20_max + value: 32.30479147669391 + - type: nauc_recall_at_20_std + value: 7.405945565001497 + - type: nauc_recall_at_3_diff1 + value: 1.6788106588864458 + - type: nauc_recall_at_3_max + value: -0.5473747492360812 + - type: nauc_recall_at_3_std + value: -5.194579239515186 + - type: nauc_recall_at_5_diff1 + value: -1.0617042372311596 + - type: nauc_recall_at_5_max + value: 2.505149243970507 + - type: nauc_recall_at_5_std + value: -4.155138551797231 + - type: ndcg_at_1 + value: 23.471 + - type: ndcg_at_10 + value: 48.403 + - type: ndcg_at_100 + value: 52.979 + - type: ndcg_at_1000 + value: 53.198 + - type: ndcg_at_20 + value: 51.349000000000004 + - type: ndcg_at_3 + value: 37.546 + - type: ndcg_at_5 + value: 43.009 + - type: precision_at_1 + value: 23.471 + - type: precision_at_10 + value: 7.788 + - type: precision_at_100 + value: 0.979 + - type: precision_at_1000 + value: 0.1 + - type: precision_at_20 + value: 4.47 + - type: precision_at_3 + value: 16.003 + - type: precision_at_5 + value: 12.276 + - type: recall_at_1 + value: 23.471 + - type: recall_at_10 + value: 77.881 + - type: recall_at_100 + value: 97.866 + - type: recall_at_1000 + value: 99.502 + - type: recall_at_20 + value: 89.403 + - type: recall_at_3 + value: 48.009 + - type: recall_at_5 + value: 61.38 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CBD + revision: 36ddb419bcffe6a5374c3891957912892916f28d + split: test + type: PL-MTEB/cbd + metrics: + - type: accuracy + value: 64.36999999999999 + - type: ap + value: 20.466523167809452 + - type: ap_weighted + value: 20.466523167809452 + - type: f1 + value: 54.572432817010075 + - type: f1_weighted + value: 68.94308188177826 + - type: main_score + value: 64.36999999999999 + task: + type: Classification + - dataset: + config: default + name: MTEB CDSC-E + revision: 0a3d4aa409b22f80eb22cbf59b492637637b536d + split: test + type: PL-MTEB/cdsce-pairclassification + metrics: + - type: cosine_accuracy + value: 87.1 + - type: cosine_accuracy_threshold + value: 98.31513166427612 + - type: cosine_ap + value: 67.39921410443071 + - type: cosine_f1 + value: 61.408450704225345 + - type: cosine_f1_threshold + value: 96.4431643486023 + - type: cosine_precision + value: 66.06060606060606 + - type: cosine_recall + value: 57.36842105263158 + - type: dot_accuracy + value: 87.1 + - type: dot_accuracy_threshold + value: 98.31513166427612 + - type: dot_ap + value: 67.39921410443071 + - type: dot_f1 + value: 61.408450704225345 + - type: dot_f1_threshold + value: 96.4431643486023 + - type: dot_precision + value: 66.06060606060606 + - type: dot_recall + value: 57.36842105263158 + - type: euclidean_accuracy + value: 87.1 + - type: euclidean_accuracy_threshold + value: 18.356148898601532 + - type: euclidean_ap + value: 67.39921410443071 + - type: euclidean_f1 + value: 61.408450704225345 + - type: euclidean_f1_threshold + value: 26.67142152786255 + - type: euclidean_precision + value: 66.06060606060606 + - type: euclidean_recall + value: 57.36842105263158 + - type: main_score + value: 67.67003917913883 + - type: manhattan_accuracy + value: 87.4 + - type: manhattan_accuracy_threshold + value: 460.4861259460449 + - type: manhattan_ap + value: 67.67003917913883 + - type: manhattan_f1 + value: 61.32596685082873 + - type: manhattan_f1_threshold + value: 640.9262657165527 + - type: manhattan_precision + value: 64.53488372093024 + - type: manhattan_recall + value: 58.42105263157895 + - type: max_ap + value: 67.67003917913883 + - type: max_f1 + value: 61.408450704225345 + - type: max_precision + value: 66.06060606060606 + - type: max_recall + value: 58.42105263157895 + - type: similarity_accuracy + value: 87.1 + - type: similarity_accuracy_threshold + value: 98.31513166427612 + - type: similarity_ap + value: 67.39921410443071 + - type: similarity_f1 + value: 61.408450704225345 + - type: similarity_f1_threshold + value: 96.4431643486023 + - type: similarity_precision + value: 66.06060606060606 + - type: similarity_recall + value: 57.36842105263158 + task: + type: PairClassification + - dataset: + config: default + name: MTEB CDSC-R + revision: 1cd6abbb00df7d14be3dbd76a7dcc64b3a79a7cd + split: test + type: PL-MTEB/cdscr-sts + metrics: + - type: cosine_pearson + value: 89.215804614177 + - type: cosine_spearman + value: 89.54723293152598 + - type: euclidean_pearson + value: 87.48591094825007 + - type: euclidean_spearman + value: 89.54723293152598 + - type: main_score + value: 89.54723293152598 + - type: manhattan_pearson + value: 87.53292271661934 + - type: manhattan_spearman + value: 89.59087505602018 + - type: pearson + value: 89.215804614177 + - type: spearman + value: 89.54723293152598 + task: + type: STS + - dataset: + config: default + name: MTEB DBPedia-PL + revision: 76afe41d9af165cc40999fcaa92312b8b012064a + split: test + type: clarin-knext/dbpedia-pl + metrics: + - type: main_score + value: 28.798000000000002 + - type: map_at_1 + value: 5.805 + - type: map_at_10 + value: 13.104 + - type: map_at_100 + value: 17.64 + - type: map_at_1000 + value: 18.817 + - type: map_at_20 + value: 14.704999999999998 + - type: map_at_3 + value: 9.754 + - type: map_at_5 + value: 11.306 + - type: mrr_at_1 + value: 48.25 + - type: mrr_at_10 + value: 59.20525793650796 + - type: mrr_at_100 + value: 59.68250991989186 + - type: mrr_at_1000 + value: 59.699200384172535 + - type: mrr_at_20 + value: 59.47831275084566 + - type: mrr_at_3 + value: 57.08333333333335 + - type: mrr_at_5 + value: 58.508333333333354 + - type: nauc_map_at_1000_diff1 + value: 23.8795526911696 + - type: nauc_map_at_1000_max + value: 19.033721390433954 + - type: nauc_map_at_1000_std + value: 31.272197919163798 + - type: nauc_map_at_100_diff1 + value: 24.30632088878799 + - type: nauc_map_at_100_max + value: 16.121272162595638 + - type: nauc_map_at_100_std + value: 27.849470534570877 + - type: nauc_map_at_10_diff1 + value: 32.383097332850866 + - type: nauc_map_at_10_max + value: 9.425444212646546 + - type: nauc_map_at_10_std + value: 12.271778849795199 + - type: nauc_map_at_1_diff1 + value: 49.09320405513435 + - type: nauc_map_at_1_max + value: 6.655982988088829 + - type: nauc_map_at_1_std + value: 3.7535182547407584 + - type: nauc_map_at_20_diff1 + value: 29.41816510506116 + - type: nauc_map_at_20_max + value: 11.144536343130254 + - type: nauc_map_at_20_std + value: 17.682154597592028 + - type: nauc_map_at_3_diff1 + value: 38.301616834556526 + - type: nauc_map_at_3_max + value: 2.3072337902833864 + - type: nauc_map_at_3_std + value: 2.6886051009617775 + - type: nauc_map_at_5_diff1 + value: 35.2269306057937 + - type: nauc_map_at_5_max + value: 5.802047613384881 + - type: nauc_map_at_5_std + value: 6.566180891431155 + - type: nauc_mrr_at_1000_diff1 + value: 27.342268602546138 + - type: nauc_mrr_at_1000_max + value: 32.66979042412427 + - type: nauc_mrr_at_1000_std + value: 31.121503406356233 + - type: nauc_mrr_at_100_diff1 + value: 27.34852791425925 + - type: nauc_mrr_at_100_max + value: 32.691852115258435 + - type: nauc_mrr_at_100_std + value: 31.13220594139922 + - type: nauc_mrr_at_10_diff1 + value: 27.20195330398617 + - type: nauc_mrr_at_10_max + value: 32.548431857593854 + - type: nauc_mrr_at_10_std + value: 31.269333708219015 + - type: nauc_mrr_at_1_diff1 + value: 29.250995725745877 + - type: nauc_mrr_at_1_max + value: 30.020096609400724 + - type: nauc_mrr_at_1_std + value: 25.836889693783288 + - type: nauc_mrr_at_20_diff1 + value: 27.117511545890967 + - type: nauc_mrr_at_20_max + value: 32.46327434659793 + - type: nauc_mrr_at_20_std + value: 31.10670830499771 + - type: nauc_mrr_at_3_diff1 + value: 27.59208442559707 + - type: nauc_mrr_at_3_max + value: 31.462102077642008 + - type: nauc_mrr_at_3_std + value: 29.857593752721684 + - type: nauc_mrr_at_5_diff1 + value: 27.067779200957986 + - type: nauc_mrr_at_5_max + value: 31.928750569317742 + - type: nauc_mrr_at_5_std + value: 31.133046384109658 + - type: nauc_ndcg_at_1000_diff1 + value: 21.640223689004966 + - type: nauc_ndcg_at_1000_max + value: 29.383523051307193 + - type: nauc_ndcg_at_1000_std + value: 43.72948428979381 + - type: nauc_ndcg_at_100_diff1 + value: 21.818106875955774 + - type: nauc_ndcg_at_100_max + value: 20.989719330345487 + - type: nauc_ndcg_at_100_std + value: 36.00564043465215 + - type: nauc_ndcg_at_10_diff1 + value: 24.873244058417672 + - type: nauc_ndcg_at_10_max + value: 22.163824659151206 + - type: nauc_ndcg_at_10_std + value: 29.712807857937996 + - type: nauc_ndcg_at_1_diff1 + value: 29.053938551788228 + - type: nauc_ndcg_at_1_max + value: 24.16157362665634 + - type: nauc_ndcg_at_1_std + value: 22.62596071010886 + - type: nauc_ndcg_at_20_diff1 + value: 24.81504479286421 + - type: nauc_ndcg_at_20_max + value: 18.22185007322912 + - type: nauc_ndcg_at_20_std + value: 29.428332063288348 + - type: nauc_ndcg_at_3_diff1 + value: 24.599338250525282 + - type: nauc_ndcg_at_3_max + value: 22.6007814893909 + - type: nauc_ndcg_at_3_std + value: 23.475949649196757 + - type: nauc_ndcg_at_5_diff1 + value: 22.905714203451318 + - type: nauc_ndcg_at_5_max + value: 23.31577015540212 + - type: nauc_ndcg_at_5_std + value: 26.163489726277543 + - type: nauc_precision_at_1000_diff1 + value: -8.445727069307745 + - type: nauc_precision_at_1000_max + value: 44.7890492424445 + - type: nauc_precision_at_1000_std + value: 32.33459741223448 + - type: nauc_precision_at_100_diff1 + value: -8.838197938558475 + - type: nauc_precision_at_100_max + value: 32.580671613119 + - type: nauc_precision_at_100_std + value: 47.75612079712025 + - type: nauc_precision_at_10_diff1 + value: -2.220152190516044 + - type: nauc_precision_at_10_max + value: 34.274996226654686 + - type: nauc_precision_at_10_std + value: 43.3987185524864 + - type: nauc_precision_at_1_diff1 + value: 29.250995725745877 + - type: nauc_precision_at_1_max + value: 30.020096609400724 + - type: nauc_precision_at_1_std + value: 25.836889693783288 + - type: nauc_precision_at_20_diff1 + value: -7.0836121714547255 + - type: nauc_precision_at_20_max + value: 31.270528745365 + - type: nauc_precision_at_20_std + value: 46.981890317602314 + - type: nauc_precision_at_3_diff1 + value: 8.668995978640655 + - type: nauc_precision_at_3_max + value: 28.75982449586511 + - type: nauc_precision_at_3_std + value: 26.513283670644082 + - type: nauc_precision_at_5_diff1 + value: 0.3364074432097908 + - type: nauc_precision_at_5_max + value: 32.17732511859588 + - type: nauc_precision_at_5_std + value: 33.011424117906 + - type: nauc_recall_at_1000_diff1 + value: 12.098189411652587 + - type: nauc_recall_at_1000_max + value: 28.351439601322042 + - type: nauc_recall_at_1000_std + value: 49.74650470960806 + - type: nauc_recall_at_100_diff1 + value: 13.0979559604911 + - type: nauc_recall_at_100_max + value: 18.55326724877563 + - type: nauc_recall_at_100_std + value: 35.82767686480256 + - type: nauc_recall_at_10_diff1 + value: 26.172188943201974 + - type: nauc_recall_at_10_max + value: 7.164754830660332 + - type: nauc_recall_at_10_std + value: 12.119900158652312 + - type: nauc_recall_at_1_diff1 + value: 49.09320405513435 + - type: nauc_recall_at_1_max + value: 6.655982988088829 + - type: nauc_recall_at_1_std + value: 3.7535182547407584 + - type: nauc_recall_at_20_diff1 + value: 22.785270973288846 + - type: nauc_recall_at_20_max + value: 8.971627656710659 + - type: nauc_recall_at_20_std + value: 17.654532692121595 + - type: nauc_recall_at_3_diff1 + value: 32.30130185123227 + - type: nauc_recall_at_3_max + value: -3.703265523226075 + - type: nauc_recall_at_3_std + value: 1.7639006695170476 + - type: nauc_recall_at_5_diff1 + value: 27.464879988497863 + - type: nauc_recall_at_5_max + value: 1.3253246804845857 + - type: nauc_recall_at_5_std + value: 6.012046846778444 + - type: ndcg_at_1 + value: 38.375 + - type: ndcg_at_10 + value: 28.798000000000002 + - type: ndcg_at_100 + value: 32.452999999999996 + - type: ndcg_at_1000 + value: 39.461 + - type: ndcg_at_20 + value: 28.16 + - type: ndcg_at_3 + value: 32.816 + - type: ndcg_at_5 + value: 30.653999999999996 + - type: precision_at_1 + value: 48.25 + - type: precision_at_10 + value: 22.825 + - type: precision_at_100 + value: 7.090000000000001 + - type: precision_at_1000 + value: 1.651 + - type: precision_at_20 + value: 16.525000000000002 + - type: precision_at_3 + value: 36.75 + - type: precision_at_5 + value: 30.25 + - type: recall_at_1 + value: 5.805 + - type: recall_at_10 + value: 18.159 + - type: recall_at_100 + value: 38.269 + - type: recall_at_1000 + value: 60.692 + - type: recall_at_20 + value: 22.893 + - type: recall_at_3 + value: 11.583 + - type: recall_at_5 + value: 14.429 + task: + type: Retrieval + - dataset: + config: default + name: MTEB 8TagsClustering + revision: 78b962b130c6690659c65abf67bf1c2f030606b6 + split: test + type: PL-MTEB/8tags-clustering + metrics: + - type: main_score + value: 39.24227294766999 + - type: v_measure + value: 39.24227294766999 + - type: v_measure_std + value: 1.3089498460899844 + task: + type: Clustering + - dataset: + config: default + name: MTEB FiQA-PL + revision: 2e535829717f8bf9dc829b7f911cc5bbd4e6608e + split: test + type: clarin-knext/fiqa-pl + metrics: + - type: main_score + value: 24.288999999999998 + - type: map_at_1 + value: 11.272 + - type: map_at_10 + value: 18.348 + - type: map_at_100 + value: 19.66 + - type: map_at_1000 + value: 19.857 + - type: map_at_20 + value: 18.992 + - type: map_at_3 + value: 15.583 + - type: map_at_5 + value: 17.044 + - type: mrr_at_1 + value: 21.604938271604937 + - type: mrr_at_10 + value: 29.27903929061337 + - type: mrr_at_100 + value: 30.212764822138023 + - type: mrr_at_1000 + value: 30.287992850622995 + - type: mrr_at_20 + value: 29.74703567100871 + - type: mrr_at_3 + value: 26.74897119341565 + - type: mrr_at_5 + value: 28.091563786008244 + - type: nauc_map_at_1000_diff1 + value: 33.18609264749629 + - type: nauc_map_at_1000_max + value: 16.229216767220418 + - type: nauc_map_at_1000_std + value: 11.839421176873238 + - type: nauc_map_at_100_diff1 + value: 33.19728650890772 + - type: nauc_map_at_100_max + value: 16.079195779561363 + - type: nauc_map_at_100_std + value: 11.78063322771766 + - type: nauc_map_at_10_diff1 + value: 33.04853224560657 + - type: nauc_map_at_10_max + value: 15.37884536409968 + - type: nauc_map_at_10_std + value: 10.593373167105781 + - type: nauc_map_at_1_diff1 + value: 41.13226077955785 + - type: nauc_map_at_1_max + value: 12.34464571217406 + - type: nauc_map_at_1_std + value: 5.679576212208579 + - type: nauc_map_at_20_diff1 + value: 33.11097420829782 + - type: nauc_map_at_20_max + value: 15.682597301121387 + - type: nauc_map_at_20_std + value: 11.155981746119032 + - type: nauc_map_at_3_diff1 + value: 35.18892537241778 + - type: nauc_map_at_3_max + value: 14.365058887923835 + - type: nauc_map_at_3_std + value: 8.599922294505198 + - type: nauc_map_at_5_diff1 + value: 33.434953310814244 + - type: nauc_map_at_5_max + value: 14.97915463074864 + - type: nauc_map_at_5_std + value: 10.474044410042204 + - type: nauc_mrr_at_1000_diff1 + value: 35.39726994803205 + - type: nauc_mrr_at_1000_max + value: 20.90929941337352 + - type: nauc_mrr_at_1000_std + value: 10.291180202334353 + - type: nauc_mrr_at_100_diff1 + value: 35.368200822647836 + - type: nauc_mrr_at_100_max + value: 20.904197225349368 + - type: nauc_mrr_at_100_std + value: 10.270197351451367 + - type: nauc_mrr_at_10_diff1 + value: 35.29930377120638 + - type: nauc_mrr_at_10_max + value: 20.743176056640657 + - type: nauc_mrr_at_10_std + value: 10.204578137405102 + - type: nauc_mrr_at_1_diff1 + value: 41.786659564879564 + - type: nauc_mrr_at_1_max + value: 21.729777030326872 + - type: nauc_mrr_at_1_std + value: 9.106473308394927 + - type: nauc_mrr_at_20_diff1 + value: 35.347875836984294 + - type: nauc_mrr_at_20_max + value: 20.701470411877295 + - type: nauc_mrr_at_20_std + value: 10.098242346945426 + - type: nauc_mrr_at_3_diff1 + value: 36.13256745876147 + - type: nauc_mrr_at_3_max + value: 20.944815901489992 + - type: nauc_mrr_at_3_std + value: 9.543271423669903 + - type: nauc_mrr_at_5_diff1 + value: 35.65577306199142 + - type: nauc_mrr_at_5_max + value: 21.01991709681707 + - type: nauc_mrr_at_5_std + value: 10.215120686016858 + - type: nauc_ndcg_at_1000_diff1 + value: 32.03773619887771 + - type: nauc_ndcg_at_1000_max + value: 20.62329873286057 + - type: nauc_ndcg_at_1000_std + value: 15.84600327765157 + - type: nauc_ndcg_at_100_diff1 + value: 31.953212726228347 + - type: nauc_ndcg_at_100_max + value: 18.594785403398458 + - type: nauc_ndcg_at_100_std + value: 15.142264504355227 + - type: nauc_ndcg_at_10_diff1 + value: 31.7344744972378 + - type: nauc_ndcg_at_10_max + value: 16.61728418738611 + - type: nauc_ndcg_at_10_std + value: 11.839953128073597 + - type: nauc_ndcg_at_1_diff1 + value: 41.786659564879564 + - type: nauc_ndcg_at_1_max + value: 21.729777030326872 + - type: nauc_ndcg_at_1_std + value: 9.106473308394927 + - type: nauc_ndcg_at_20_diff1 + value: 31.95134068064315 + - type: nauc_ndcg_at_20_max + value: 16.909454974818715 + - type: nauc_ndcg_at_20_std + value: 12.672035929326222 + - type: nauc_ndcg_at_3_diff1 + value: 34.60931899005471 + - type: nauc_ndcg_at_3_max + value: 18.294906425122836 + - type: nauc_ndcg_at_3_std + value: 9.91740552963498 + - type: nauc_ndcg_at_5_diff1 + value: 32.67109785825235 + - type: nauc_ndcg_at_5_max + value: 17.56715417050353 + - type: nauc_ndcg_at_5_std + value: 11.716109414267391 + - type: nauc_precision_at_1000_diff1 + value: 6.192451799751743 + - type: nauc_precision_at_1000_max + value: 31.232946403916884 + - type: nauc_precision_at_1000_std + value: 11.32233619145531 + - type: nauc_precision_at_100_diff1 + value: 15.110676996819022 + - type: nauc_precision_at_100_max + value: 27.869316716354476 + - type: nauc_precision_at_100_std + value: 18.301164363871184 + - type: nauc_precision_at_10_diff1 + value: 21.070307270094048 + - type: nauc_precision_at_10_max + value: 24.22361518067922 + - type: nauc_precision_at_10_std + value: 14.554273388003363 + - type: nauc_precision_at_1_diff1 + value: 41.786659564879564 + - type: nauc_precision_at_1_max + value: 21.729777030326872 + - type: nauc_precision_at_1_std + value: 9.106473308394927 + - type: nauc_precision_at_20_diff1 + value: 19.816441423202775 + - type: nauc_precision_at_20_max + value: 24.383538204770737 + - type: nauc_precision_at_20_std + value: 15.199168636971377 + - type: nauc_precision_at_3_diff1 + value: 29.262451716999237 + - type: nauc_precision_at_3_max + value: 22.556259017888365 + - type: nauc_precision_at_3_std + value: 13.40809071872643 + - type: nauc_precision_at_5_diff1 + value: 23.41393741017905 + - type: nauc_precision_at_5_max + value: 22.510618167670824 + - type: nauc_precision_at_5_std + value: 16.609471253895997 + - type: nauc_recall_at_1000_diff1 + value: 16.69894332310247 + - type: nauc_recall_at_1000_max + value: 23.603524439007714 + - type: nauc_recall_at_1000_std + value: 37.70133731622978 + - type: nauc_recall_at_100_diff1 + value: 20.68782760606423 + - type: nauc_recall_at_100_max + value: 14.868760747917944 + - type: nauc_recall_at_100_std + value: 22.988507759909506 + - type: nauc_recall_at_10_diff1 + value: 22.897676261389755 + - type: nauc_recall_at_10_max + value: 11.677499286310164 + - type: nauc_recall_at_10_std + value: 13.136747166983081 + - type: nauc_recall_at_1_diff1 + value: 41.13226077955785 + - type: nauc_recall_at_1_max + value: 12.34464571217406 + - type: nauc_recall_at_1_std + value: 5.679576212208579 + - type: nauc_recall_at_20_diff1 + value: 23.52610305455693 + - type: nauc_recall_at_20_max + value: 12.126334316287 + - type: nauc_recall_at_20_std + value: 14.890491358767576 + - type: nauc_recall_at_3_diff1 + value: 29.300377418190145 + - type: nauc_recall_at_3_max + value: 14.2656548623994 + - type: nauc_recall_at_3_std + value: 10.365708612559724 + - type: nauc_recall_at_5_diff1 + value: 25.427037962562448 + - type: nauc_recall_at_5_max + value: 14.162542226396466 + - type: nauc_recall_at_5_std + value: 13.685656584050593 + - type: ndcg_at_1 + value: 21.605 + - type: ndcg_at_10 + value: 24.288999999999998 + - type: ndcg_at_100 + value: 30.218 + - type: ndcg_at_1000 + value: 34.414 + - type: ndcg_at_20 + value: 26.154 + - type: ndcg_at_3 + value: 20.707 + - type: ndcg_at_5 + value: 21.956999999999997 + - type: precision_at_1 + value: 21.605 + - type: precision_at_10 + value: 7.022 + - type: precision_at_100 + value: 1.2930000000000001 + - type: precision_at_1000 + value: 0.202 + - type: precision_at_20 + value: 4.267 + - type: precision_at_3 + value: 13.683 + - type: precision_at_5 + value: 10.463000000000001 + - type: recall_at_1 + value: 11.272 + - type: recall_at_10 + value: 30.721999999999998 + - type: recall_at_100 + value: 54.039 + - type: recall_at_1000 + value: 79.935 + - type: recall_at_20 + value: 36.506 + - type: recall_at_3 + value: 19.256 + - type: recall_at_5 + value: 24.008 + task: + type: Retrieval + - dataset: + config: default + name: MTEB HotpotQA-PL + revision: a0bd479ac97b4ccb5bd6ce320c415d0bb4beb907 + split: test + type: clarin-knext/hotpotqa-pl + metrics: + - type: main_score + value: 59.909 + - type: map_at_1 + value: 35.631 + - type: map_at_10 + value: 50.953 + - type: map_at_100 + value: 51.761 + - type: map_at_1000 + value: 51.836000000000006 + - type: map_at_20 + value: 51.403 + - type: map_at_3 + value: 48.242000000000004 + - type: map_at_5 + value: 49.89 + - type: mrr_at_1 + value: 71.2626603646185 + - type: mrr_at_10 + value: 77.3888942477732 + - type: mrr_at_100 + value: 77.66937146141727 + - type: mrr_at_1000 + value: 77.68397288041564 + - type: mrr_at_20 + value: 77.57074315067958 + - type: mrr_at_3 + value: 76.25928426738675 + - type: mrr_at_5 + value: 77.01350438892611 + - type: nauc_map_at_1000_diff1 + value: 26.96764981061544 + - type: nauc_map_at_1000_max + value: 22.731836451302726 + - type: nauc_map_at_1000_std + value: 8.994156423402524 + - type: nauc_map_at_100_diff1 + value: 26.942497951432788 + - type: nauc_map_at_100_max + value: 22.71479330722431 + - type: nauc_map_at_100_std + value: 8.98407440009605 + - type: nauc_map_at_10_diff1 + value: 27.158114557540152 + - type: nauc_map_at_10_max + value: 22.58104861100442 + - type: nauc_map_at_10_std + value: 8.337888297704612 + - type: nauc_map_at_1_diff1 + value: 72.90723129101286 + - type: nauc_map_at_1_max + value: 43.326963605865224 + - type: nauc_map_at_1_std + value: 5.723322726536682 + - type: nauc_map_at_20_diff1 + value: 26.955640087566074 + - type: nauc_map_at_20_max + value: 22.627922508820586 + - type: nauc_map_at_20_std + value: 8.701393241049022 + - type: nauc_map_at_3_diff1 + value: 29.189900000716555 + - type: nauc_map_at_3_max + value: 23.347579158922738 + - type: nauc_map_at_3_std + value: 6.91628130204425 + - type: nauc_map_at_5_diff1 + value: 27.79243876992334 + - type: nauc_map_at_5_max + value: 22.740295287033856 + - type: nauc_map_at_5_std + value: 7.7561319969849665 + - type: nauc_mrr_at_1000_diff1 + value: 71.28682783448207 + - type: nauc_mrr_at_1000_max + value: 45.922411180244616 + - type: nauc_mrr_at_1000_std + value: 9.034961290707159 + - type: nauc_mrr_at_100_diff1 + value: 71.28283301258217 + - type: nauc_mrr_at_100_max + value: 45.9275695181984 + - type: nauc_mrr_at_100_std + value: 9.046986980134697 + - type: nauc_mrr_at_10_diff1 + value: 71.2493228565909 + - type: nauc_mrr_at_10_max + value: 45.93632950394277 + - type: nauc_mrr_at_10_std + value: 8.901287032604865 + - type: nauc_mrr_at_1_diff1 + value: 72.90723129101286 + - type: nauc_mrr_at_1_max + value: 43.326963605865224 + - type: nauc_mrr_at_1_std + value: 5.723322726536682 + - type: nauc_mrr_at_20_diff1 + value: 71.25254486378807 + - type: nauc_mrr_at_20_max + value: 45.946582942424556 + - type: nauc_mrr_at_20_std + value: 9.05239175902674 + - type: nauc_mrr_at_3_diff1 + value: 71.32833026076825 + - type: nauc_mrr_at_3_max + value: 45.995038423133785 + - type: nauc_mrr_at_3_std + value: 8.341868297953612 + - type: nauc_mrr_at_5_diff1 + value: 71.30890801509685 + - type: nauc_mrr_at_5_max + value: 45.97654481382387 + - type: nauc_mrr_at_5_std + value: 8.845360351910351 + - type: nauc_ndcg_at_1000_diff1 + value: 32.075295168560594 + - type: nauc_ndcg_at_1000_max + value: 26.856216385104375 + - type: nauc_ndcg_at_1000_std + value: 12.543512287347856 + - type: nauc_ndcg_at_100_diff1 + value: 31.464965142947115 + - type: nauc_ndcg_at_100_max + value: 26.502114112582163 + - type: nauc_ndcg_at_100_std + value: 12.425894931877595 + - type: nauc_ndcg_at_10_diff1 + value: 32.367658571844196 + - type: nauc_ndcg_at_10_max + value: 26.190984389588372 + - type: nauc_ndcg_at_10_std + value: 9.895477224547669 + - type: nauc_ndcg_at_1_diff1 + value: 72.90723129101286 + - type: nauc_ndcg_at_1_max + value: 43.326963605865224 + - type: nauc_ndcg_at_1_std + value: 5.723322726536682 + - type: nauc_ndcg_at_20_diff1 + value: 31.66045690140549 + - type: nauc_ndcg_at_20_max + value: 26.194578795851054 + - type: nauc_ndcg_at_20_std + value: 10.960870590823447 + - type: nauc_ndcg_at_3_diff1 + value: 36.05439214093881 + - type: nauc_ndcg_at_3_max + value: 27.72410316744339 + - type: nauc_ndcg_at_3_std + value: 7.715317220132202 + - type: nauc_ndcg_at_5_diff1 + value: 33.81723761665154 + - type: nauc_ndcg_at_5_max + value: 26.673920056459444 + - type: nauc_ndcg_at_5_std + value: 8.933439774824867 + - type: nauc_precision_at_1000_diff1 + value: -0.2221251083991722 + - type: nauc_precision_at_1000_max + value: 14.280584973386084 + - type: nauc_precision_at_1000_std + value: 27.7063978378772 + - type: nauc_precision_at_100_diff1 + value: 4.0475513079562475 + - type: nauc_precision_at_100_max + value: 15.147221648346376 + - type: nauc_precision_at_100_std + value: 22.880509039199133 + - type: nauc_precision_at_10_diff1 + value: 12.84890497551863 + - type: nauc_precision_at_10_max + value: 17.087762521613513 + - type: nauc_precision_at_10_std + value: 12.585305065906013 + - type: nauc_precision_at_1_diff1 + value: 72.90723129101286 + - type: nauc_precision_at_1_max + value: 43.326963605865224 + - type: nauc_precision_at_1_std + value: 5.723322726536682 + - type: nauc_precision_at_20_diff1 + value: 8.933673501828638 + - type: nauc_precision_at_20_max + value: 15.902286967992069 + - type: nauc_precision_at_20_std + value: 15.695999265224772 + - type: nauc_precision_at_3_diff1 + value: 22.71740911034549 + - type: nauc_precision_at_3_max + value: 21.870782890822923 + - type: nauc_precision_at_3_std + value: 8.368195274830732 + - type: nauc_precision_at_5_diff1 + value: 17.23292486248257 + - type: nauc_precision_at_5_max + value: 18.885190696734025 + - type: nauc_precision_at_5_std + value: 10.426694032135838 + - type: nauc_recall_at_1000_diff1 + value: -0.2221251083988545 + - type: nauc_recall_at_1000_max + value: 14.28058497338619 + - type: nauc_recall_at_1000_std + value: 27.706397837877343 + - type: nauc_recall_at_100_diff1 + value: 4.047551307956134 + - type: nauc_recall_at_100_max + value: 15.14722164834636 + - type: nauc_recall_at_100_std + value: 22.88050903919902 + - type: nauc_recall_at_10_diff1 + value: 12.848904975518701 + - type: nauc_recall_at_10_max + value: 17.087762521613545 + - type: nauc_recall_at_10_std + value: 12.585305065906043 + - type: nauc_recall_at_1_diff1 + value: 72.90723129101286 + - type: nauc_recall_at_1_max + value: 43.326963605865224 + - type: nauc_recall_at_1_std + value: 5.723322726536682 + - type: nauc_recall_at_20_diff1 + value: 8.933673501828618 + - type: nauc_recall_at_20_max + value: 15.902286967991989 + - type: nauc_recall_at_20_std + value: 15.695999265224723 + - type: nauc_recall_at_3_diff1 + value: 22.71740911034548 + - type: nauc_recall_at_3_max + value: 21.870782890822923 + - type: nauc_recall_at_3_std + value: 8.368195274830695 + - type: nauc_recall_at_5_diff1 + value: 17.23292486248257 + - type: nauc_recall_at_5_max + value: 18.88519069673408 + - type: nauc_recall_at_5_std + value: 10.426694032135869 + - type: ndcg_at_1 + value: 71.263 + - type: ndcg_at_10 + value: 59.909 + - type: ndcg_at_100 + value: 62.93299999999999 + - type: ndcg_at_1000 + value: 64.447 + - type: ndcg_at_20 + value: 61.111000000000004 + - type: ndcg_at_3 + value: 55.835 + - type: ndcg_at_5 + value: 58.062000000000005 + - type: precision_at_1 + value: 71.263 + - type: precision_at_10 + value: 12.262 + - type: precision_at_100 + value: 1.464 + - type: precision_at_1000 + value: 0.166 + - type: precision_at_20 + value: 6.518 + - type: precision_at_3 + value: 34.855000000000004 + - type: precision_at_5 + value: 22.668 + - type: recall_at_1 + value: 35.631 + - type: recall_at_10 + value: 61.309999999999995 + - type: recall_at_100 + value: 73.207 + - type: recall_at_1000 + value: 83.234 + - type: recall_at_20 + value: 65.179 + - type: recall_at_3 + value: 52.282 + - type: recall_at_5 + value: 56.67100000000001 + task: + type: Retrieval + - dataset: + config: default + name: MTEB MSMARCO-PL + revision: 8634c07806d5cce3a6138e260e59b81760a0a640 + split: test + type: clarin-knext/msmarco-pl + metrics: + - type: main_score + value: 44.882 + - type: map_at_1 + value: 1.24 + - type: map_at_10 + value: 7.768999999999999 + - type: map_at_100 + value: 19.535 + - type: map_at_1000 + value: 24.29 + - type: map_at_20 + value: 11.124 + - type: map_at_3 + value: 3.558 + - type: map_at_5 + value: 5.112 + - type: mrr_at_1 + value: 67.44186046511628 + - type: mrr_at_10 + value: 75.19379844961239 + - type: mrr_at_100 + value: 75.82587954680979 + - type: mrr_at_1000 + value: 75.82820745611444 + - type: mrr_at_20 + value: 75.73643410852712 + - type: mrr_at_3 + value: 73.64341085271317 + - type: mrr_at_5 + value: 74.8062015503876 + - type: nauc_map_at_1000_diff1 + value: -21.989620907518457 + - type: nauc_map_at_1000_max + value: 74.21088703597528 + - type: nauc_map_at_1000_std + value: 77.21798713118748 + - type: nauc_map_at_100_diff1 + value: -15.6773928726037 + - type: nauc_map_at_100_max + value: 70.22001058099622 + - type: nauc_map_at_100_std + value: 65.22415371485596 + - type: nauc_map_at_10_diff1 + value: 21.960342056606727 + - type: nauc_map_at_10_max + value: 37.1897095260209 + - type: nauc_map_at_10_std + value: 18.92058870696038 + - type: nauc_map_at_1_diff1 + value: 15.577838306755954 + - type: nauc_map_at_1_max + value: 24.863043579374917 + - type: nauc_map_at_1_std + value: 6.592387370843499 + - type: nauc_map_at_20_diff1 + value: 9.576438240390903 + - type: nauc_map_at_20_max + value: 51.30183073674882 + - type: nauc_map_at_20_std + value: 35.30336831130106 + - type: nauc_map_at_3_diff1 + value: 15.131873334572553 + - type: nauc_map_at_3_max + value: 18.797768321440504 + - type: nauc_map_at_3_std + value: -0.13092645109257714 + - type: nauc_map_at_5_diff1 + value: 17.04231465807186 + - type: nauc_map_at_5_max + value: 21.466125385662263 + - type: nauc_map_at_5_std + value: 1.1575637123352447 + - type: nauc_mrr_at_1000_diff1 + value: -18.248968234547323 + - type: nauc_mrr_at_1000_max + value: 64.27602897839343 + - type: nauc_mrr_at_1000_std + value: 50.131100296201645 + - type: nauc_mrr_at_100_diff1 + value: -18.238980286255142 + - type: nauc_mrr_at_100_max + value: 64.27997603786041 + - type: nauc_mrr_at_100_std + value: 50.13517702252198 + - type: nauc_mrr_at_10_diff1 + value: -18.512675691928646 + - type: nauc_mrr_at_10_max + value: 64.09197909286284 + - type: nauc_mrr_at_10_std + value: 51.031732106064275 + - type: nauc_mrr_at_1_diff1 + value: -23.142139605578112 + - type: nauc_mrr_at_1_max + value: 63.19048808731884 + - type: nauc_mrr_at_1_std + value: 48.774238204316354 + - type: nauc_mrr_at_20_diff1 + value: -18.640160840586574 + - type: nauc_mrr_at_20_max + value: 64.430976412527 + - type: nauc_mrr_at_20_std + value: 50.15695556858976 + - type: nauc_mrr_at_3_diff1 + value: -18.390465775772828 + - type: nauc_mrr_at_3_max + value: 63.433292438340814 + - type: nauc_mrr_at_3_std + value: 47.94619902515134 + - type: nauc_mrr_at_5_diff1 + value: -18.54623628289419 + - type: nauc_mrr_at_5_max + value: 63.74074576279657 + - type: nauc_mrr_at_5_std + value: 48.257682525479225 + - type: nauc_ndcg_at_1000_diff1 + value: -13.169970965775713 + - type: nauc_ndcg_at_1000_max + value: 73.73523772051698 + - type: nauc_ndcg_at_1000_std + value: 71.45235358022637 + - type: nauc_ndcg_at_100_diff1 + value: -19.8120594180483 + - type: nauc_ndcg_at_100_max + value: 67.3274027052369 + - type: nauc_ndcg_at_100_std + value: 64.5956731015882 + - type: nauc_ndcg_at_10_diff1 + value: -6.277350735880853 + - type: nauc_ndcg_at_10_max + value: 60.59305682607034 + - type: nauc_ndcg_at_10_std + value: 49.126971101907 + - type: nauc_ndcg_at_1_diff1 + value: -8.720324288077949 + - type: nauc_ndcg_at_1_max + value: 30.501202785637542 + - type: nauc_ndcg_at_1_std + value: 18.293586568513305 + - type: nauc_ndcg_at_20_diff1 + value: -18.877995945130856 + - type: nauc_ndcg_at_20_max + value: 67.39555958720247 + - type: nauc_ndcg_at_20_std + value: 60.6279774858829 + - type: nauc_ndcg_at_3_diff1 + value: -5.934464965738841 + - type: nauc_ndcg_at_3_max + value: 47.27404035349958 + - type: nauc_ndcg_at_3_std + value: 26.17442248226043 + - type: nauc_ndcg_at_5_diff1 + value: -3.3782533914725223 + - type: nauc_ndcg_at_5_max + value: 50.47004778843094 + - type: nauc_ndcg_at_5_std + value: 28.95390387656202 + - type: nauc_precision_at_1000_diff1 + value: -35.81194272720232 + - type: nauc_precision_at_1000_max + value: 44.315527138954394 + - type: nauc_precision_at_1000_std + value: 63.628095344847246 + - type: nauc_precision_at_100_diff1 + value: -39.18958442311581 + - type: nauc_precision_at_100_max + value: 53.986900796770264 + - type: nauc_precision_at_100_std + value: 71.09932268988669 + - type: nauc_precision_at_10_diff1 + value: -13.984050096227286 + - type: nauc_precision_at_10_max + value: 65.42524229112135 + - type: nauc_precision_at_10_std + value: 66.20226396851875 + - type: nauc_precision_at_1_diff1 + value: -23.142139605578112 + - type: nauc_precision_at_1_max + value: 63.19048808731884 + - type: nauc_precision_at_1_std + value: 48.774238204316354 + - type: nauc_precision_at_20_diff1 + value: -30.176493123897135 + - type: nauc_precision_at_20_max + value: 64.06849634487335 + - type: nauc_precision_at_20_std + value: 69.75485681813515 + - type: nauc_precision_at_3_diff1 + value: -11.011413097929944 + - type: nauc_precision_at_3_max + value: 67.46835081745049 + - type: nauc_precision_at_3_std + value: 48.060582045912426 + - type: nauc_precision_at_5_diff1 + value: -6.405874699662373 + - type: nauc_precision_at_5_max + value: 62.60221401593178 + - type: nauc_precision_at_5_std + value: 46.921707025371575 + - type: nauc_recall_at_1000_diff1 + value: -8.114402563352586 + - type: nauc_recall_at_1000_max + value: 60.3001556664617 + - type: nauc_recall_at_1000_std + value: 63.39890343621265 + - type: nauc_recall_at_100_diff1 + value: -6.738351023868497 + - type: nauc_recall_at_100_max + value: 51.32812340863999 + - type: nauc_recall_at_100_std + value: 49.412743596311934 + - type: nauc_recall_at_10_diff1 + value: 20.601461386363383 + - type: nauc_recall_at_10_max + value: 21.60715140677045 + - type: nauc_recall_at_10_std + value: 6.221071995265527 + - type: nauc_recall_at_1_diff1 + value: 15.577838306755954 + - type: nauc_recall_at_1_max + value: 24.863043579374917 + - type: nauc_recall_at_1_std + value: 6.592387370843499 + - type: nauc_recall_at_20_diff1 + value: 12.800091088352024 + - type: nauc_recall_at_20_max + value: 35.04412507104955 + - type: nauc_recall_at_20_std + value: 21.904897436365363 + - type: nauc_recall_at_3_diff1 + value: 11.408953973523726 + - type: nauc_recall_at_3_max + value: 8.260360788102039 + - type: nauc_recall_at_3_std + value: -7.79122970880463 + - type: nauc_recall_at_5_diff1 + value: 14.380219584599557 + - type: nauc_recall_at_5_max + value: 5.975379730591909 + - type: nauc_recall_at_5_std + value: -12.068230649882123 + - type: ndcg_at_1 + value: 51.937999999999995 + - type: ndcg_at_10 + value: 44.882 + - type: ndcg_at_100 + value: 40.783 + - type: ndcg_at_1000 + value: 49.486999999999995 + - type: ndcg_at_20 + value: 42.184 + - type: ndcg_at_3 + value: 48.329 + - type: ndcg_at_5 + value: 46.571 + - type: precision_at_1 + value: 67.44200000000001 + - type: precision_at_10 + value: 53.256 + - type: precision_at_100 + value: 24.907 + - type: precision_at_1000 + value: 5.072 + - type: precision_at_20 + value: 45.465 + - type: precision_at_3 + value: 62.016000000000005 + - type: precision_at_5 + value: 57.67400000000001 + - type: recall_at_1 + value: 1.24 + - type: recall_at_10 + value: 9.911 + - type: recall_at_100 + value: 33.912 + - type: recall_at_1000 + value: 59.88 + - type: recall_at_20 + value: 15.456 + - type: recall_at_3 + value: 4.0329999999999995 + - type: recall_at_5 + value: 6.265999999999999 + task: + type: Retrieval + - dataset: + config: pl + name: MTEB MassiveIntentClassification (pl) + revision: 4672e20407010da34463acc759c162ca9734bca6 + split: test + type: mteb/amazon_massive_intent + metrics: + - type: accuracy + value: 58.56086079354406 + - type: f1 + value: 51.0752914371443 + - type: f1_weighted + value: 57.25465842170869 + - type: main_score + value: 58.56086079354406 + task: + type: Classification + - dataset: + config: pl + name: MTEB MassiveScenarioClassification (pl) + revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 + split: test + type: mteb/amazon_massive_scenario + metrics: + - type: accuracy + value: 68.79959650302622 + - type: f1 + value: 66.73568572153533 + - type: f1_weighted + value: 68.69392891616663 + - type: main_score + value: 68.79959650302622 + task: + type: Classification + - dataset: + config: default + name: MTEB NFCorpus-PL + revision: 9a6f9567fda928260afed2de480d79c98bf0bec0 + split: test + type: clarin-knext/nfcorpus-pl + metrics: + - type: main_score + value: 26.700000000000003 + - type: map_at_1 + value: 3.7319999999999998 + - type: map_at_10 + value: 8.855 + - type: map_at_100 + value: 10.884 + - type: map_at_1000 + value: 11.922 + - type: map_at_20 + value: 9.721 + - type: map_at_3 + value: 6.178999999999999 + - type: map_at_5 + value: 7.381 + - type: mrr_at_1 + value: 34.984520123839005 + - type: mrr_at_10 + value: 43.8296722197651 + - type: mrr_at_100 + value: 44.530521314732724 + - type: mrr_at_1000 + value: 44.58203081100927 + - type: mrr_at_20 + value: 44.28344960852701 + - type: mrr_at_3 + value: 41.3312693498452 + - type: mrr_at_5 + value: 42.81733746130031 + - type: nauc_map_at_1000_diff1 + value: 22.462363087719815 + - type: nauc_map_at_1000_max + value: 20.296759775120016 + - type: nauc_map_at_1000_std + value: 30.209661157733358 + - type: nauc_map_at_100_diff1 + value: 23.590011889585053 + - type: nauc_map_at_100_max + value: 18.684278383417727 + - type: nauc_map_at_100_std + value: 26.82621895459225 + - type: nauc_map_at_10_diff1 + value: 27.229535241863704 + - type: nauc_map_at_10_max + value: 12.409798091370948 + - type: nauc_map_at_10_std + value: 18.28274894942748 + - type: nauc_map_at_1_diff1 + value: 43.574814034284046 + - type: nauc_map_at_1_max + value: 4.564434104867417 + - type: nauc_map_at_1_std + value: 7.935898903116461 + - type: nauc_map_at_20_diff1 + value: 25.32655921246531 + - type: nauc_map_at_20_max + value: 15.368931898009667 + - type: nauc_map_at_20_std + value: 21.95073617629164 + - type: nauc_map_at_3_diff1 + value: 33.96473904739136 + - type: nauc_map_at_3_max + value: 6.840960020808816 + - type: nauc_map_at_3_std + value: 11.193855793167565 + - type: nauc_map_at_5_diff1 + value: 31.386524819992022 + - type: nauc_map_at_5_max + value: 7.667289402351551 + - type: nauc_map_at_5_std + value: 12.835284913547296 + - type: nauc_mrr_at_1000_diff1 + value: 29.965669376994974 + - type: nauc_mrr_at_1000_max + value: 31.211237208155946 + - type: nauc_mrr_at_1000_std + value: 35.73731867290863 + - type: nauc_mrr_at_100_diff1 + value: 29.947754600374953 + - type: nauc_mrr_at_100_max + value: 31.266989755619463 + - type: nauc_mrr_at_100_std + value: 35.79202078225156 + - type: nauc_mrr_at_10_diff1 + value: 29.945702147569413 + - type: nauc_mrr_at_10_max + value: 30.886642789129688 + - type: nauc_mrr_at_10_std + value: 35.63914754058482 + - type: nauc_mrr_at_1_diff1 + value: 33.713470402720944 + - type: nauc_mrr_at_1_max + value: 28.751793677785038 + - type: nauc_mrr_at_1_std + value: 31.803182986705426 + - type: nauc_mrr_at_20_diff1 + value: 29.91366548301302 + - type: nauc_mrr_at_20_max + value: 31.131288856999966 + - type: nauc_mrr_at_20_std + value: 35.882134224012276 + - type: nauc_mrr_at_3_diff1 + value: 30.860896236484802 + - type: nauc_mrr_at_3_max + value: 29.519500974357793 + - type: nauc_mrr_at_3_std + value: 33.72309342817263 + - type: nauc_mrr_at_5_diff1 + value: 29.49229611633102 + - type: nauc_mrr_at_5_max + value: 29.356523818764106 + - type: nauc_mrr_at_5_std + value: 34.403728644234796 + - type: nauc_ndcg_at_1000_diff1 + value: 20.65065747027027 + - type: nauc_ndcg_at_1000_max + value: 38.56020828417934 + - type: nauc_ndcg_at_1000_std + value: 43.11278832738437 + - type: nauc_ndcg_at_100_diff1 + value: 21.45463562941668 + - type: nauc_ndcg_at_100_max + value: 30.873631958812254 + - type: nauc_ndcg_at_100_std + value: 39.4205527116693 + - type: nauc_ndcg_at_10_diff1 + value: 20.631467977678895 + - type: nauc_ndcg_at_10_max + value: 23.607571269154562 + - type: nauc_ndcg_at_10_std + value: 38.079177270694046 + - type: nauc_ndcg_at_1_diff1 + value: 35.543556832559204 + - type: nauc_ndcg_at_1_max + value: 26.4137633343316 + - type: nauc_ndcg_at_1_std + value: 32.09063013779238 + - type: nauc_ndcg_at_20_diff1 + value: 21.67953499762251 + - type: nauc_ndcg_at_20_max + value: 25.131819715231092 + - type: nauc_ndcg_at_20_std + value: 38.67052591163072 + - type: nauc_ndcg_at_3_diff1 + value: 26.63443060501996 + - type: nauc_ndcg_at_3_max + value: 22.563735867709223 + - type: nauc_ndcg_at_3_std + value: 34.29760134985721 + - type: nauc_ndcg_at_5_diff1 + value: 22.39905347041664 + - type: nauc_ndcg_at_5_max + value: 21.756018113774005 + - type: nauc_ndcg_at_5_std + value: 36.157605696574926 + - type: nauc_precision_at_1000_diff1 + value: -5.432303517532642 + - type: nauc_precision_at_1000_max + value: 9.32638219741413 + - type: nauc_precision_at_1000_std + value: 39.7153509814171 + - type: nauc_precision_at_100_diff1 + value: -2.9578929385950175 + - type: nauc_precision_at_100_max + value: 23.113346560742485 + - type: nauc_precision_at_100_std + value: 50.42833972221638 + - type: nauc_precision_at_10_diff1 + value: 6.4967709091740256 + - type: nauc_precision_at_10_max + value: 26.245637962599716 + - type: nauc_precision_at_10_std + value: 45.96670051969855 + - type: nauc_precision_at_1_diff1 + value: 33.713470402720944 + - type: nauc_precision_at_1_max + value: 28.751793677785038 + - type: nauc_precision_at_1_std + value: 31.803182986705426 + - type: nauc_precision_at_20_diff1 + value: 3.8421416138073177 + - type: nauc_precision_at_20_max + value: 26.723613496152606 + - type: nauc_precision_at_20_std + value: 49.38259985536451 + - type: nauc_precision_at_3_diff1 + value: 19.750562130084283 + - type: nauc_precision_at_3_max + value: 24.15782294130561 + - type: nauc_precision_at_3_std + value: 36.02002453018143 + - type: nauc_precision_at_5_diff1 + value: 12.22400568198802 + - type: nauc_precision_at_5_max + value: 22.64477994259073 + - type: nauc_precision_at_5_std + value: 40.56427415099331 + - type: nauc_recall_at_1000_diff1 + value: 12.448845632560946 + - type: nauc_recall_at_1000_max + value: 31.622286432469547 + - type: nauc_recall_at_1000_std + value: 26.309639034445116 + - type: nauc_recall_at_100_diff1 + value: 10.832581159712385 + - type: nauc_recall_at_100_max + value: 24.682803257928217 + - type: nauc_recall_at_100_std + value: 25.86711840077285 + - type: nauc_recall_at_10_diff1 + value: 20.971221569108366 + - type: nauc_recall_at_10_max + value: 12.805858961362107 + - type: nauc_recall_at_10_std + value: 13.633848218463635 + - type: nauc_recall_at_1_diff1 + value: 43.574814034284046 + - type: nauc_recall_at_1_max + value: 4.564434104867417 + - type: nauc_recall_at_1_std + value: 7.935898903116461 + - type: nauc_recall_at_20_diff1 + value: 17.60417964776668 + - type: nauc_recall_at_20_max + value: 16.600914108701208 + - type: nauc_recall_at_20_std + value: 18.462567374206206 + - type: nauc_recall_at_3_diff1 + value: 30.13359450934449 + - type: nauc_recall_at_3_max + value: 6.329195402219131 + - type: nauc_recall_at_3_std + value: 7.804247093066491 + - type: nauc_recall_at_5_diff1 + value: 24.03163031054558 + - type: nauc_recall_at_5_max + value: 5.958331833530651 + - type: nauc_recall_at_5_std + value: 8.356617891524207 + - type: ndcg_at_1 + value: 33.437 + - type: ndcg_at_10 + value: 26.700000000000003 + - type: ndcg_at_100 + value: 23.363999999999997 + - type: ndcg_at_1000 + value: 31.631999999999998 + - type: ndcg_at_20 + value: 24.275 + - type: ndcg_at_3 + value: 30.195 + - type: ndcg_at_5 + value: 28.992 + - type: precision_at_1 + value: 34.985 + - type: precision_at_10 + value: 20.433 + - type: precision_at_100 + value: 5.913 + - type: precision_at_1000 + value: 1.746 + - type: precision_at_20 + value: 14.443 + - type: precision_at_3 + value: 28.896 + - type: precision_at_5 + value: 25.635 + - type: recall_at_1 + value: 3.7319999999999998 + - type: recall_at_10 + value: 12.943 + - type: recall_at_100 + value: 24.265 + - type: recall_at_1000 + value: 53.25 + - type: recall_at_20 + value: 15.9 + - type: recall_at_3 + value: 7.205 + - type: recall_at_5 + value: 9.572 + task: + type: Retrieval + - dataset: + config: default + name: MTEB NQ-PL + revision: f171245712cf85dd4700b06bef18001578d0ca8d + split: test + type: clarin-knext/nq-pl + metrics: + - type: main_score + value: 29.635 + - type: map_at_1 + value: 14.895 + - type: map_at_10 + value: 24.157 + - type: map_at_100 + value: 25.391000000000002 + - type: map_at_1000 + value: 25.46 + - type: map_at_20 + value: 24.932000000000002 + - type: map_at_3 + value: 20.987000000000002 + - type: map_at_5 + value: 22.781000000000002 + - type: mrr_at_1 + value: 17.033603707995365 + - type: mrr_at_10 + value: 26.199270264305056 + - type: mrr_at_100 + value: 27.25035497632659 + - type: mrr_at_1000 + value: 27.307077728677648 + - type: mrr_at_20 + value: 26.864370993179577 + - type: mrr_at_3 + value: 23.30050212437239 + - type: mrr_at_5 + value: 25.002414059482515 + - type: nauc_map_at_1000_diff1 + value: 26.106475555869068 + - type: nauc_map_at_1000_max + value: 25.461012594691052 + - type: nauc_map_at_1000_std + value: 17.376546142427276 + - type: nauc_map_at_100_diff1 + value: 26.093633746899975 + - type: nauc_map_at_100_max + value: 25.460594042644274 + - type: nauc_map_at_100_std + value: 17.362503401189493 + - type: nauc_map_at_10_diff1 + value: 26.186038172833605 + - type: nauc_map_at_10_max + value: 24.83017698848067 + - type: nauc_map_at_10_std + value: 16.189803634117602 + - type: nauc_map_at_1_diff1 + value: 29.174707786719022 + - type: nauc_map_at_1_max + value: 18.953264922797644 + - type: nauc_map_at_1_std + value: 12.17148617810572 + - type: nauc_map_at_20_diff1 + value: 26.189839001345916 + - type: nauc_map_at_20_max + value: 25.1873036250334 + - type: nauc_map_at_20_std + value: 16.82987531231318 + - type: nauc_map_at_3_diff1 + value: 26.751451062914455 + - type: nauc_map_at_3_max + value: 22.689487570350828 + - type: nauc_map_at_3_std + value: 14.472523964376913 + - type: nauc_map_at_5_diff1 + value: 26.466986102889255 + - type: nauc_map_at_5_max + value: 23.52302816080052 + - type: nauc_map_at_5_std + value: 15.038486281773714 + - type: nauc_mrr_at_1000_diff1 + value: 24.610741329750734 + - type: nauc_mrr_at_1000_max + value: 25.90460252341745 + - type: nauc_mrr_at_1000_std + value: 19.229931369441093 + - type: nauc_mrr_at_100_diff1 + value: 24.602966673307343 + - type: nauc_mrr_at_100_max + value: 25.909318923953727 + - type: nauc_mrr_at_100_std + value: 19.23207134491914 + - type: nauc_mrr_at_10_diff1 + value: 24.54192982829959 + - type: nauc_mrr_at_10_max + value: 25.514442888519135 + - type: nauc_mrr_at_10_std + value: 18.455239589874953 + - type: nauc_mrr_at_1_diff1 + value: 27.50641656592486 + - type: nauc_mrr_at_1_max + value: 20.77182038095459 + - type: nauc_mrr_at_1_std + value: 15.260460862822761 + - type: nauc_mrr_at_20_diff1 + value: 24.62172144081287 + - type: nauc_mrr_at_20_max + value: 25.762301912792967 + - type: nauc_mrr_at_20_std + value: 18.881085908672773 + - type: nauc_mrr_at_3_diff1 + value: 24.894701148248426 + - type: nauc_mrr_at_3_max + value: 23.661742208973944 + - type: nauc_mrr_at_3_std + value: 16.95434957782849 + - type: nauc_mrr_at_5_diff1 + value: 24.658005705956025 + - type: nauc_mrr_at_5_max + value: 24.467992442785043 + - type: nauc_mrr_at_5_std + value: 17.532796653975378 + - type: nauc_ndcg_at_1000_diff1 + value: 24.632830969868674 + - type: nauc_ndcg_at_1000_max + value: 30.36626493300332 + - type: nauc_ndcg_at_1000_std + value: 24.14333061184113 + - type: nauc_ndcg_at_100_diff1 + value: 24.346047050349547 + - type: nauc_ndcg_at_100_max + value: 30.462122958605747 + - type: nauc_ndcg_at_100_std + value: 24.123189028744292 + - type: nauc_ndcg_at_10_diff1 + value: 24.637441274115673 + - type: nauc_ndcg_at_10_max + value: 27.814466997081162 + - type: nauc_ndcg_at_10_std + value: 18.979604631236413 + - type: nauc_ndcg_at_1_diff1 + value: 27.50641656592486 + - type: nauc_ndcg_at_1_max + value: 20.77182038095459 + - type: nauc_ndcg_at_1_std + value: 15.260460862822761 + - type: nauc_ndcg_at_20_diff1 + value: 24.767111944047276 + - type: nauc_ndcg_at_20_max + value: 28.908654915082167 + - type: nauc_ndcg_at_20_std + value: 20.884645934796296 + - type: nauc_ndcg_at_3_diff1 + value: 25.592082856454688 + - type: nauc_ndcg_at_3_max + value: 23.80753222156324 + - type: nauc_ndcg_at_3_std + value: 15.71575559898033 + - type: nauc_ndcg_at_5_diff1 + value: 25.085610054435374 + - type: nauc_ndcg_at_5_max + value: 25.167359679118857 + - type: nauc_ndcg_at_5_std + value: 16.687464277549015 + - type: nauc_precision_at_1000_diff1 + value: 6.176538001587548 + - type: nauc_precision_at_1000_max + value: 33.99737260769466 + - type: nauc_precision_at_1000_std + value: 40.4324058481132 + - type: nauc_precision_at_100_diff1 + value: 10.072303037708917 + - type: nauc_precision_at_100_max + value: 38.36505815603056 + - type: nauc_precision_at_100_std + value: 41.970311663044356 + - type: nauc_precision_at_10_diff1 + value: 17.77691868853893 + - type: nauc_precision_at_10_max + value: 34.165661141650745 + - type: nauc_precision_at_10_std + value: 26.088809095572792 + - type: nauc_precision_at_1_diff1 + value: 27.50641656592486 + - type: nauc_precision_at_1_max + value: 20.77182038095459 + - type: nauc_precision_at_1_std + value: 15.260460862822761 + - type: nauc_precision_at_20_diff1 + value: 16.0122719270748 + - type: nauc_precision_at_20_max + value: 35.27584767260969 + - type: nauc_precision_at_20_std + value: 30.804881061720916 + - type: nauc_precision_at_3_diff1 + value: 21.863534491804486 + - type: nauc_precision_at_3_max + value: 27.673732654355643 + - type: nauc_precision_at_3_std + value: 19.92145910413624 + - type: nauc_precision_at_5_diff1 + value: 20.538193025997955 + - type: nauc_precision_at_5_max + value: 29.756275018772953 + - type: nauc_precision_at_5_std + value: 21.41349792358181 + - type: nauc_recall_at_1000_diff1 + value: 20.367219696261838 + - type: nauc_recall_at_1000_max + value: 59.62491249007197 + - type: nauc_recall_at_1000_std + value: 67.37971908143585 + - type: nauc_recall_at_100_diff1 + value: 19.448555428920216 + - type: nauc_recall_at_100_max + value: 48.396102452977466 + - type: nauc_recall_at_100_std + value: 49.77861384791871 + - type: nauc_recall_at_10_diff1 + value: 21.30947648461549 + - type: nauc_recall_at_10_max + value: 33.55182519248534 + - type: nauc_recall_at_10_std + value: 23.209432479779874 + - type: nauc_recall_at_1_diff1 + value: 29.174707786719022 + - type: nauc_recall_at_1_max + value: 18.953264922797644 + - type: nauc_recall_at_1_std + value: 12.17148617810572 + - type: nauc_recall_at_20_diff1 + value: 21.877824704962652 + - type: nauc_recall_at_20_max + value: 37.50302867285941 + - type: nauc_recall_at_20_std + value: 29.539665628885874 + - type: nauc_recall_at_3_diff1 + value: 23.671582594272646 + - type: nauc_recall_at_3_max + value: 25.14079018254003 + - type: nauc_recall_at_3_std + value: 16.40327881618881 + - type: nauc_recall_at_5_diff1 + value: 22.295165954167263 + - type: nauc_recall_at_5_max + value: 27.25439949621236 + - type: nauc_recall_at_5_std + value: 17.929454386934562 + - type: ndcg_at_1 + value: 17.034 + - type: ndcg_at_10 + value: 29.635 + - type: ndcg_at_100 + value: 35.333 + - type: ndcg_at_1000 + value: 37.217 + - type: ndcg_at_20 + value: 32.245000000000005 + - type: ndcg_at_3 + value: 23.443 + - type: ndcg_at_5 + value: 26.55 + - type: precision_at_1 + value: 17.034 + - type: precision_at_10 + value: 5.33 + - type: precision_at_100 + value: 0.8500000000000001 + - type: precision_at_1000 + value: 0.10300000000000001 + - type: precision_at_20 + value: 3.272 + - type: precision_at_3 + value: 10.989 + - type: precision_at_5 + value: 8.36 + - type: recall_at_1 + value: 14.895 + - type: recall_at_10 + value: 44.426 + - type: recall_at_100 + value: 70.09 + - type: recall_at_1000 + value: 84.57900000000001 + - type: recall_at_20 + value: 54.193000000000005 + - type: recall_at_3 + value: 28.283 + - type: recall_at_5 + value: 35.489 + task: + type: Retrieval + - dataset: + config: default + name: MTEB PAC + revision: fc69d1c153a8ccdcf1eef52f4e2a27f88782f543 + split: test + type: laugustyniak/abusive-clauses-pl + metrics: + - type: accuracy + value: 65.0390964378801 + - type: ap + value: 74.13204115538645 + - type: ap_weighted + value: 74.13204115538645 + - type: f1 + value: 61.679102411303475 + - type: f1_weighted + value: 65.42516393570598 + - type: main_score + value: 65.0390964378801 + task: + type: Classification + - dataset: + config: default + name: MTEB PSC + revision: d05a294af9e1d3ff2bfb6b714e08a24a6cabc669 + split: test + type: PL-MTEB/psc-pairclassification + metrics: + - type: cosine_accuracy + value: 98.33024118738405 + - type: cosine_accuracy_threshold + value: 84.68939065933228 + - type: cosine_ap + value: 99.56709058320435 + - type: cosine_f1 + value: 97.27272727272728 + - type: cosine_f1_threshold + value: 84.68939065933228 + - type: cosine_precision + value: 96.6867469879518 + - type: cosine_recall + value: 97.86585365853658 + - type: dot_accuracy + value: 98.33024118738405 + - type: dot_accuracy_threshold + value: 84.68939065933228 + - type: dot_ap + value: 99.56709058320435 + - type: dot_f1 + value: 97.27272727272728 + - type: dot_f1_threshold + value: 84.68939065933228 + - type: dot_precision + value: 96.6867469879518 + - type: dot_recall + value: 97.86585365853658 + - type: euclidean_accuracy + value: 98.33024118738405 + - type: euclidean_accuracy_threshold + value: 55.33609390258789 + - type: euclidean_ap + value: 99.56709058320435 + - type: euclidean_f1 + value: 97.27272727272728 + - type: euclidean_f1_threshold + value: 55.33609390258789 + - type: euclidean_precision + value: 96.6867469879518 + - type: euclidean_recall + value: 97.86585365853658 + - type: main_score + value: 99.58207452502563 + - type: manhattan_accuracy + value: 98.33024118738405 + - type: manhattan_accuracy_threshold + value: 1277.597713470459 + - type: manhattan_ap + value: 99.58207452502563 + - type: manhattan_f1 + value: 97.2560975609756 + - type: manhattan_f1_threshold + value: 1288.2164001464844 + - type: manhattan_precision + value: 97.2560975609756 + - type: manhattan_recall + value: 97.2560975609756 + - type: max_ap + value: 99.58207452502563 + - type: max_f1 + value: 97.27272727272728 + - type: max_precision + value: 97.2560975609756 + - type: max_recall + value: 97.86585365853658 + - type: similarity_accuracy + value: 98.33024118738405 + - type: similarity_accuracy_threshold + value: 84.68939065933228 + - type: similarity_ap + value: 99.56709058320435 + - type: similarity_f1 + value: 97.27272727272728 + - type: similarity_f1_threshold + value: 84.68939065933228 + - type: similarity_precision + value: 96.6867469879518 + - type: similarity_recall + value: 97.86585365853658 + task: + type: PairClassification + - dataset: + config: default + name: MTEB PolEmo2.0-IN + revision: d90724373c70959f17d2331ad51fb60c71176b03 + split: test + type: PL-MTEB/polemo2_in + metrics: + - type: accuracy + value: 78.808864265928 + - type: f1 + value: 71.43202202178325 + - type: f1_weighted + value: 76.79354444796897 + - type: main_score + value: 78.808864265928 + task: + type: Classification + - dataset: + config: default + name: MTEB PolEmo2.0-OUT + revision: 6a21ab8716e255ab1867265f8b396105e8aa63d4 + split: test + type: PL-MTEB/polemo2_out + metrics: + - type: accuracy + value: 62.57085020242916 + - type: f1 + value: 47.55601056441782 + - type: f1_weighted + value: 62.511982873074665 + - type: main_score + value: 62.57085020242916 + task: + type: Classification + - dataset: + config: default + name: MTEB PPC + revision: 2c7d2df57801a591f6b1e3aaf042e7a04ec7d9f2 + split: test + type: PL-MTEB/ppc-pairclassification + metrics: + - type: cosine_accuracy + value: 74.0 + - type: cosine_accuracy_threshold + value: 94.31836009025574 + - type: cosine_ap + value: 86.10558522259642 + - type: cosine_f1 + value: 79.29936305732484 + - type: cosine_f1_threshold + value: 94.31836009025574 + - type: cosine_precision + value: 76.38036809815951 + - type: cosine_recall + value: 82.45033112582782 + - type: dot_accuracy + value: 74.0 + - type: dot_accuracy_threshold + value: 94.31835412979126 + - type: dot_ap + value: 86.10558522259642 + - type: dot_f1 + value: 79.29936305732484 + - type: dot_f1_threshold + value: 94.31835412979126 + - type: dot_precision + value: 76.38036809815951 + - type: dot_recall + value: 82.45033112582782 + - type: euclidean_accuracy + value: 74.0 + - type: euclidean_accuracy_threshold + value: 33.70946943759918 + - type: euclidean_ap + value: 86.10558522259642 + - type: euclidean_f1 + value: 79.29936305732484 + - type: euclidean_f1_threshold + value: 33.70946943759918 + - type: euclidean_precision + value: 76.38036809815951 + - type: euclidean_recall + value: 82.45033112582782 + - type: main_score + value: 86.10558522259642 + - type: manhattan_accuracy + value: 73.9 + - type: manhattan_accuracy_threshold + value: 780.9340476989746 + - type: manhattan_ap + value: 86.0467795030663 + - type: manhattan_f1 + value: 79.43485086342228 + - type: manhattan_f1_threshold + value: 797.8069305419922 + - type: manhattan_precision + value: 75.5223880597015 + - type: manhattan_recall + value: 83.77483443708608 + - type: max_ap + value: 86.10558522259642 + - type: max_f1 + value: 79.43485086342228 + - type: max_precision + value: 76.38036809815951 + - type: max_recall + value: 83.77483443708608 + - type: similarity_accuracy + value: 74.0 + - type: similarity_accuracy_threshold + value: 94.31836009025574 + - type: similarity_ap + value: 86.10558522259642 + - type: similarity_f1 + value: 79.29936305732484 + - type: similarity_f1_threshold + value: 94.31836009025574 + - type: similarity_precision + value: 76.38036809815951 + - type: similarity_recall + value: 82.45033112582782 + task: + type: PairClassification + - dataset: + config: default + name: MTEB Quora-PL + revision: 0be27e93455051e531182b85e85e425aba12e9d4 + split: test + type: clarin-knext/quora-pl + metrics: + - type: main_score + value: 72.967 + - type: map_at_1 + value: 55.144000000000005 + - type: map_at_10 + value: 67.929 + - type: map_at_100 + value: 68.86 + - type: map_at_1000 + value: 68.901 + - type: map_at_20 + value: 68.516 + - type: map_at_3 + value: 64.83 + - type: map_at_5 + value: 66.676 + - type: mrr_at_1 + value: 63.480000000000004 + - type: mrr_at_10 + value: 71.84984920634886 + - type: mrr_at_100 + value: 72.25466310843967 + - type: mrr_at_1000 + value: 72.26655999913221 + - type: mrr_at_20 + value: 72.12276439487533 + - type: mrr_at_3 + value: 70.05499999999962 + - type: mrr_at_5 + value: 71.20399999999938 + - type: nauc_map_at_1000_diff1 + value: 68.24286057450547 + - type: nauc_map_at_1000_max + value: 25.772071887662857 + - type: nauc_map_at_1000_std + value: -11.038099329313008 + - type: nauc_map_at_100_diff1 + value: 68.23827609989507 + - type: nauc_map_at_100_max + value: 25.7528856894633 + - type: nauc_map_at_100_std + value: -11.068564365152046 + - type: nauc_map_at_10_diff1 + value: 68.28520996107123 + - type: nauc_map_at_10_max + value: 25.002003876054346 + - type: nauc_map_at_10_std + value: -12.335638622901788 + - type: nauc_map_at_1_diff1 + value: 71.75651974240864 + - type: nauc_map_at_1_max + value: 18.91563206148769 + - type: nauc_map_at_1_std + value: -15.766224337269566 + - type: nauc_map_at_20_diff1 + value: 68.22063759842727 + - type: nauc_map_at_20_max + value: 25.487015251371098 + - type: nauc_map_at_20_std + value: -11.562217754911307 + - type: nauc_map_at_3_diff1 + value: 68.73066371981375 + - type: nauc_map_at_3_max + value: 23.00894164024716 + - type: nauc_map_at_3_std + value: -14.416968038747862 + - type: nauc_map_at_5_diff1 + value: 68.4026048544462 + - type: nauc_map_at_5_max + value: 24.0635545260521 + - type: nauc_map_at_5_std + value: -13.317259791673012 + - type: nauc_mrr_at_1000_diff1 + value: 69.42328633437369 + - type: nauc_mrr_at_1000_max + value: 29.616854630563104 + - type: nauc_mrr_at_1000_std + value: -7.407262927873341 + - type: nauc_mrr_at_100_diff1 + value: 69.41805385692442 + - type: nauc_mrr_at_100_max + value: 29.622721216167434 + - type: nauc_mrr_at_100_std + value: -7.391041773641051 + - type: nauc_mrr_at_10_diff1 + value: 69.3457273244501 + - type: nauc_mrr_at_10_max + value: 29.65947727398705 + - type: nauc_mrr_at_10_std + value: -7.53792710769481 + - type: nauc_mrr_at_1_diff1 + value: 71.8589978314364 + - type: nauc_mrr_at_1_max + value: 27.26957667791739 + - type: nauc_mrr_at_1_std + value: -10.100927694941472 + - type: nauc_mrr_at_20_diff1 + value: 69.36942578766813 + - type: nauc_mrr_at_20_max + value: 29.660463270488385 + - type: nauc_mrr_at_20_std + value: -7.376877149626725 + - type: nauc_mrr_at_3_diff1 + value: 69.29861515799205 + - type: nauc_mrr_at_3_max + value: 29.33399555998474 + - type: nauc_mrr_at_3_std + value: -8.240273377090736 + - type: nauc_mrr_at_5_diff1 + value: 69.35386047017475 + - type: nauc_mrr_at_5_max + value: 29.584250910701815 + - type: nauc_mrr_at_5_std + value: -7.667942449440323 + - type: nauc_ndcg_at_1000_diff1 + value: 67.84970988845713 + - type: nauc_ndcg_at_1000_max + value: 28.61263305970346 + - type: nauc_ndcg_at_1000_std + value: -7.206523803073922 + - type: nauc_ndcg_at_100_diff1 + value: 67.70352902312702 + - type: nauc_ndcg_at_100_max + value: 28.66451560902271 + - type: nauc_ndcg_at_100_std + value: -6.851076090960979 + - type: nauc_ndcg_at_10_diff1 + value: 67.42015737885397 + - type: nauc_ndcg_at_10_max + value: 27.530444130016168 + - type: nauc_ndcg_at_10_std + value: -9.427985190987474 + - type: nauc_ndcg_at_1_diff1 + value: 71.8793742316567 + - type: nauc_ndcg_at_1_max + value: 27.261298450185308 + - type: nauc_ndcg_at_1_std + value: -10.04497075691524 + - type: nauc_ndcg_at_20_diff1 + value: 67.42200058537294 + - type: nauc_ndcg_at_20_max + value: 28.22986052404655 + - type: nauc_ndcg_at_20_std + value: -8.030536728985693 + - type: nauc_ndcg_at_3_diff1 + value: 67.471552537964 + - type: nauc_ndcg_at_3_max + value: 26.03277670287765 + - type: nauc_ndcg_at_3_std + value: -11.253279176032436 + - type: nauc_ndcg_at_5_diff1 + value: 67.42678714699264 + - type: nauc_ndcg_at_5_max + value: 26.54845879664486 + - type: nauc_ndcg_at_5_std + value: -10.407611106477633 + - type: nauc_precision_at_1000_diff1 + value: -30.32934486467821 + - type: nauc_precision_at_1000_max + value: 11.91523191478659 + - type: nauc_precision_at_1000_std + value: 31.476552592999436 + - type: nauc_precision_at_100_diff1 + value: -26.262105638108135 + - type: nauc_precision_at_100_max + value: 15.193751494600448 + - type: nauc_precision_at_100_std + value: 32.09374258953343 + - type: nauc_precision_at_10_diff1 + value: -6.682234712045421 + - type: nauc_precision_at_10_max + value: 20.59230495370378 + - type: nauc_precision_at_10_std + value: 19.563221549346537 + - type: nauc_precision_at_1_diff1 + value: 71.8793742316567 + - type: nauc_precision_at_1_max + value: 27.261298450185308 + - type: nauc_precision_at_1_std + value: -10.04497075691524 + - type: nauc_precision_at_20_diff1 + value: -16.129281421675955 + - type: nauc_precision_at_20_max + value: 19.013144338631392 + - type: nauc_precision_at_20_std + value: 26.326173134142287 + - type: nauc_precision_at_3_diff1 + value: 20.699978586651056 + - type: nauc_precision_at_3_max + value: 23.0320962309107 + - type: nauc_precision_at_3_std + value: 4.749967993274664 + - type: nauc_precision_at_5_diff1 + value: 7.046864204806423 + - type: nauc_precision_at_5_max + value: 22.16832739332923 + - type: nauc_precision_at_5_std + value: 11.909222559184276 + - type: nauc_recall_at_1000_diff1 + value: 52.05206576699943 + - type: nauc_recall_at_1000_max + value: 51.435199785253246 + - type: nauc_recall_at_1000_std + value: 51.60940884098615 + - type: nauc_recall_at_100_diff1 + value: 55.88854763934312 + - type: nauc_recall_at_100_max + value: 39.21155962587193 + - type: nauc_recall_at_100_std + value: 27.299887462517518 + - type: nauc_recall_at_10_diff1 + value: 60.16585088325225 + - type: nauc_recall_at_10_max + value: 28.663702614424896 + - type: nauc_recall_at_10_std + value: -6.073039592069796 + - type: nauc_recall_at_1_diff1 + value: 71.75651974240864 + - type: nauc_recall_at_1_max + value: 18.91563206148769 + - type: nauc_recall_at_1_std + value: -15.766224337269566 + - type: nauc_recall_at_20_diff1 + value: 58.35530372050582 + - type: nauc_recall_at_20_max + value: 32.60341340698367 + - type: nauc_recall_at_20_std + value: 3.321152193915959 + - type: nauc_recall_at_3_diff1 + value: 63.760773357483316 + - type: nauc_recall_at_3_max + value: 23.313883321559906 + - type: nauc_recall_at_3_std + value: -13.473871675341206 + - type: nauc_recall_at_5_diff1 + value: 62.13055752382483 + - type: nauc_recall_at_5_max + value: 25.401911413697775 + - type: nauc_recall_at_5_std + value: -10.397116573821439 + - type: ndcg_at_1 + value: 63.470000000000006 + - type: ndcg_at_10 + value: 72.967 + - type: ndcg_at_100 + value: 75.832 + - type: ndcg_at_1000 + value: 76.41799999999999 + - type: ndcg_at_20 + value: 74.369 + - type: ndcg_at_3 + value: 68.731 + - type: ndcg_at_5 + value: 70.93199999999999 + - type: precision_at_1 + value: 63.470000000000006 + - type: precision_at_10 + value: 11.321 + - type: precision_at_100 + value: 1.413 + - type: precision_at_1000 + value: 0.152 + - type: precision_at_20 + value: 6.214 + - type: precision_at_3 + value: 30.14 + - type: precision_at_5 + value: 20.182 + - type: recall_at_1 + value: 55.144000000000005 + - type: recall_at_10 + value: 83.488 + - type: recall_at_100 + value: 94.71000000000001 + - type: recall_at_1000 + value: 98.38900000000001 + - type: recall_at_20 + value: 88.32799999999999 + - type: recall_at_3 + value: 71.763 + - type: recall_at_5 + value: 77.519 + task: + type: Retrieval + - dataset: + config: default + name: MTEB SCIDOCS-PL + revision: 45452b03f05560207ef19149545f168e596c9337 + split: test + type: clarin-knext/scidocs-pl + metrics: + - type: main_score + value: 13.905000000000001 + - type: map_at_1 + value: 3.1649999999999996 + - type: map_at_10 + value: 7.872999999999999 + - type: map_at_100 + value: 9.437 + - type: map_at_1000 + value: 9.701 + - type: map_at_20 + value: 8.685 + - type: map_at_3 + value: 5.793 + - type: map_at_5 + value: 6.755999999999999 + - type: mrr_at_1 + value: 15.6 + - type: mrr_at_10 + value: 24.506746031746005 + - type: mrr_at_100 + value: 25.70609394877134 + - type: mrr_at_1000 + value: 25.792056624320015 + - type: mrr_at_20 + value: 25.222491915119605 + - type: mrr_at_3 + value: 21.66666666666669 + - type: mrr_at_5 + value: 23.191666666666666 + - type: nauc_map_at_1000_diff1 + value: 23.18232368444192 + - type: nauc_map_at_1000_max + value: 30.51723263049919 + - type: nauc_map_at_1000_std + value: 20.20486779374314 + - type: nauc_map_at_100_diff1 + value: 23.080727823504276 + - type: nauc_map_at_100_max + value: 30.246387130436638 + - type: nauc_map_at_100_std + value: 19.68405276888789 + - type: nauc_map_at_10_diff1 + value: 24.537343409943325 + - type: nauc_map_at_10_max + value: 28.364069700464334 + - type: nauc_map_at_10_std + value: 15.13427848315133 + - type: nauc_map_at_1_diff1 + value: 32.39883848968994 + - type: nauc_map_at_1_max + value: 23.615723240461623 + - type: nauc_map_at_1_std + value: 11.148264932586265 + - type: nauc_map_at_20_diff1 + value: 23.826614277797272 + - type: nauc_map_at_20_max + value: 29.54306133099886 + - type: nauc_map_at_20_std + value: 17.408092097298262 + - type: nauc_map_at_3_diff1 + value: 26.89407078857185 + - type: nauc_map_at_3_max + value: 26.82448479758581 + - type: nauc_map_at_3_std + value: 11.203306435103412 + - type: nauc_map_at_5_diff1 + value: 24.925295727226374 + - type: nauc_map_at_5_max + value: 27.726964720927285 + - type: nauc_map_at_5_std + value: 13.165832906881105 + - type: nauc_mrr_at_1000_diff1 + value: 26.07952885954442 + - type: nauc_mrr_at_1000_max + value: 26.39620305682346 + - type: nauc_mrr_at_1000_std + value: 15.4691968563711 + - type: nauc_mrr_at_100_diff1 + value: 26.068460416892215 + - type: nauc_mrr_at_100_max + value: 26.380928493559804 + - type: nauc_mrr_at_100_std + value: 15.489233144751708 + - type: nauc_mrr_at_10_diff1 + value: 26.03448612594449 + - type: nauc_mrr_at_10_max + value: 26.364989628019757 + - type: nauc_mrr_at_10_std + value: 14.991318949655136 + - type: nauc_mrr_at_1_diff1 + value: 32.723567686923026 + - type: nauc_mrr_at_1_max + value: 23.607424459530364 + - type: nauc_mrr_at_1_std + value: 11.214989860076892 + - type: nauc_mrr_at_20_diff1 + value: 25.914306236878293 + - type: nauc_mrr_at_20_max + value: 26.254769411936184 + - type: nauc_mrr_at_20_std + value: 15.167029359994864 + - type: nauc_mrr_at_3_diff1 + value: 26.641013626531695 + - type: nauc_mrr_at_3_max + value: 26.588054854826503 + - type: nauc_mrr_at_3_std + value: 14.237519001411153 + - type: nauc_mrr_at_5_diff1 + value: 25.99328651624433 + - type: nauc_mrr_at_5_max + value: 26.13836117183896 + - type: nauc_mrr_at_5_std + value: 14.420365231524618 + - type: nauc_ndcg_at_1000_diff1 + value: 21.80668442273199 + - type: nauc_ndcg_at_1000_max + value: 33.34390999032878 + - type: nauc_ndcg_at_1000_std + value: 29.662073089048285 + - type: nauc_ndcg_at_100_diff1 + value: 21.072971411828306 + - type: nauc_ndcg_at_100_max + value: 31.488496819079593 + - type: nauc_ndcg_at_100_std + value: 26.73580019278837 + - type: nauc_ndcg_at_10_diff1 + value: 22.964364053079933 + - type: nauc_ndcg_at_10_max + value: 28.351440666770383 + - type: nauc_ndcg_at_10_std + value: 17.383681731219692 + - type: nauc_ndcg_at_1_diff1 + value: 32.723567686923026 + - type: nauc_ndcg_at_1_max + value: 23.607424459530364 + - type: nauc_ndcg_at_1_std + value: 11.214989860076892 + - type: nauc_ndcg_at_20_diff1 + value: 21.938539210894064 + - type: nauc_ndcg_at_20_max + value: 29.631859895086222 + - type: nauc_ndcg_at_20_std + value: 20.386892920365828 + - type: nauc_ndcg_at_3_diff1 + value: 25.69508679368568 + - type: nauc_ndcg_at_3_max + value: 27.346589413294353 + - type: nauc_ndcg_at_3_std + value: 13.27524027859251 + - type: nauc_ndcg_at_5_diff1 + value: 23.448042036479684 + - type: nauc_ndcg_at_5_max + value: 27.58741657842951 + - type: nauc_ndcg_at_5_std + value: 14.768121274771891 + - type: nauc_precision_at_1000_diff1 + value: 13.325830342851743 + - type: nauc_precision_at_1000_max + value: 33.12589529001219 + - type: nauc_precision_at_1000_std + value: 43.19509533550807 + - type: nauc_precision_at_100_diff1 + value: 13.283631413521633 + - type: nauc_precision_at_100_max + value: 30.40530489731691 + - type: nauc_precision_at_100_std + value: 36.67580352196424 + - type: nauc_precision_at_10_diff1 + value: 18.522857406921162 + - type: nauc_precision_at_10_max + value: 28.296802001220513 + - type: nauc_precision_at_10_std + value: 20.645757169009997 + - type: nauc_precision_at_1_diff1 + value: 32.723567686923026 + - type: nauc_precision_at_1_max + value: 23.607424459530364 + - type: nauc_precision_at_1_std + value: 11.214989860076892 + - type: nauc_precision_at_20_diff1 + value: 16.06141180796256 + - type: nauc_precision_at_20_max + value: 29.235625664181075 + - type: nauc_precision_at_20_std + value: 25.145690128045885 + - type: nauc_precision_at_3_diff1 + value: 23.24883851534181 + - type: nauc_precision_at_3_max + value: 28.64831854723111 + - type: nauc_precision_at_3_std + value: 14.372759012060602 + - type: nauc_precision_at_5_diff1 + value: 19.74922697183535 + - type: nauc_precision_at_5_max + value: 28.28618112212183 + - type: nauc_precision_at_5_std + value: 16.76873435538237 + - type: nauc_recall_at_1000_diff1 + value: 12.47335275188347 + - type: nauc_recall_at_1000_max + value: 33.00015584848869 + - type: nauc_recall_at_1000_std + value: 43.70701134051634 + - type: nauc_recall_at_100_diff1 + value: 12.513041153563837 + - type: nauc_recall_at_100_max + value: 30.273540501575823 + - type: nauc_recall_at_100_std + value: 36.71619580684846 + - type: nauc_recall_at_10_diff1 + value: 18.17624193441234 + - type: nauc_recall_at_10_max + value: 28.250886102195054 + - type: nauc_recall_at_10_std + value: 20.261194441451018 + - type: nauc_recall_at_1_diff1 + value: 32.39883848968994 + - type: nauc_recall_at_1_max + value: 23.615723240461623 + - type: nauc_recall_at_1_std + value: 11.148264932586265 + - type: nauc_recall_at_20_diff1 + value: 15.608763922874088 + - type: nauc_recall_at_20_max + value: 29.104046594172832 + - type: nauc_recall_at_20_std + value: 24.75986997225524 + - type: nauc_recall_at_3_diff1 + value: 22.928537699291248 + - type: nauc_recall_at_3_max + value: 28.72397719226873 + - type: nauc_recall_at_3_std + value: 14.044388052473414 + - type: nauc_recall_at_5_diff1 + value: 19.51285766786134 + - type: nauc_recall_at_5_max + value: 28.33574712082385 + - type: nauc_recall_at_5_std + value: 16.400242628056912 + - type: ndcg_at_1 + value: 15.6 + - type: ndcg_at_10 + value: 13.905000000000001 + - type: ndcg_at_100 + value: 20.485999999999997 + - type: ndcg_at_1000 + value: 25.72 + - type: ndcg_at_20 + value: 16.297 + - type: ndcg_at_3 + value: 13.145000000000001 + - type: ndcg_at_5 + value: 11.413 + - type: precision_at_1 + value: 15.6 + - type: precision_at_10 + value: 7.290000000000001 + - type: precision_at_100 + value: 1.67 + - type: precision_at_1000 + value: 0.293 + - type: precision_at_20 + value: 5.025 + - type: precision_at_3 + value: 12.4 + - type: precision_at_5 + value: 10.040000000000001 + - type: recall_at_1 + value: 3.1649999999999996 + - type: recall_at_10 + value: 14.798 + - type: recall_at_100 + value: 33.967999999999996 + - type: recall_at_1000 + value: 59.577999999999996 + - type: recall_at_20 + value: 20.403 + - type: recall_at_3 + value: 7.568 + - type: recall_at_5 + value: 10.218 + task: + type: Retrieval + - dataset: + config: default + name: MTEB SICK-E-PL + revision: 71bba34b0ece6c56dfcf46d9758a27f7a90f17e9 + split: test + type: PL-MTEB/sicke-pl-pairclassification + metrics: + - type: cosine_accuracy + value: 77.82307378719935 + - type: cosine_accuracy_threshold + value: 97.00007438659668 + - type: cosine_ap + value: 62.95127012761914 + - type: cosine_f1 + value: 60.530590146183 + - type: cosine_f1_threshold + value: 94.09935474395752 + - type: cosine_precision + value: 48.82096069868996 + - type: cosine_recall + value: 79.62962962962963 + - type: dot_accuracy + value: 77.82307378719935 + - type: dot_accuracy_threshold + value: 97.00008034706116 + - type: dot_ap + value: 62.95087763566891 + - type: dot_f1 + value: 60.530590146183 + - type: dot_f1_threshold + value: 94.099360704422 + - type: dot_precision + value: 48.82096069868996 + - type: dot_recall + value: 79.62962962962963 + - type: euclidean_accuracy + value: 77.82307378719935 + - type: euclidean_accuracy_threshold + value: 24.494586884975433 + - type: euclidean_ap + value: 62.95127012761915 + - type: euclidean_f1 + value: 60.530590146183 + - type: euclidean_f1_threshold + value: 34.3529999256134 + - type: euclidean_precision + value: 48.82096069868996 + - type: euclidean_recall + value: 79.62962962962963 + - type: main_score + value: 62.95127012761915 + - type: manhattan_accuracy + value: 77.6600081532817 + - type: manhattan_accuracy_threshold + value: 569.0948009490967 + - type: manhattan_ap + value: 62.59417668418388 + - type: manhattan_f1 + value: 60.486737763193865 + - type: manhattan_f1_threshold + value: 803.2670974731445 + - type: manhattan_precision + value: 49.090102086107414 + - type: manhattan_recall + value: 78.77492877492878 + - type: max_ap + value: 62.95127012761915 + - type: max_f1 + value: 60.530590146183 + - type: max_precision + value: 49.090102086107414 + - type: max_recall + value: 79.62962962962963 + - type: similarity_accuracy + value: 77.82307378719935 + - type: similarity_accuracy_threshold + value: 97.00007438659668 + - type: similarity_ap + value: 62.95127012761914 + - type: similarity_f1 + value: 60.530590146183 + - type: similarity_f1_threshold + value: 94.09935474395752 + - type: similarity_precision + value: 48.82096069868996 + - type: similarity_recall + value: 79.62962962962963 + task: + type: PairClassification + - dataset: + config: default + name: MTEB SICK-R-PL + revision: fd5c2441b7eeff8676768036142af4cfa42c1339 + split: test + type: PL-MTEB/sickr-pl-sts + metrics: + - type: cosine_pearson + value: 72.0298956351125 + - type: cosine_spearman + value: 66.17464577596678 + - type: euclidean_pearson + value: 68.98291370252295 + - type: euclidean_spearman + value: 66.17463716306638 + - type: main_score + value: 66.17464577596678 + - type: manhattan_pearson + value: 68.81382804554632 + - type: manhattan_spearman + value: 65.97651341583767 + - type: pearson + value: 72.0298956351125 + - type: spearman + value: 66.17464577596678 + task: + type: STS + - dataset: + config: pl + name: MTEB STS22 (pl) + revision: de9d86b3b84231dc21f76c7b7af1f28e2f57f6e3 + split: test + type: mteb/sts22-crosslingual-sts + metrics: + - type: cosine_pearson + value: 41.189800700866094 + - type: cosine_spearman + value: 45.34202672993038 + - type: euclidean_pearson + value: 27.19960620267402 + - type: euclidean_spearman + value: 45.34202672993038 + - type: main_score + value: 45.34202672993038 + - type: manhattan_pearson + value: 26.949544922651196 + - type: manhattan_spearman + value: 45.21613651872827 + - type: pearson + value: 41.189800700866094 + - type: spearman + value: 45.34202672993038 + task: + type: STS + - dataset: + config: pl-en + name: MTEB STS22 (pl-en) + revision: de9d86b3b84231dc21f76c7b7af1f28e2f57f6e3 + split: test + type: mteb/sts22-crosslingual-sts + metrics: + - type: cosine_pearson + value: 78.98435260035146 + - type: cosine_spearman + value: 79.0083835281799 + - type: euclidean_pearson + value: 79.05051294275746 + - type: euclidean_spearman + value: 79.0083835281799 + - type: main_score + value: 79.0083835281799 + - type: manhattan_pearson + value: 79.28546319449809 + - type: manhattan_spearman + value: 78.79790129500051 + - type: pearson + value: 78.98435260035146 + - type: spearman + value: 79.0083835281799 + task: + type: STS + - dataset: + config: de-pl + name: MTEB STS22 (de-pl) + revision: de9d86b3b84231dc21f76c7b7af1f28e2f57f6e3 + split: test + type: mteb/sts22-crosslingual-sts + metrics: + - type: cosine_pearson + value: 41.459968983462765 + - type: cosine_spearman + value: 53.581740089351996 + - type: euclidean_pearson + value: 43.45992137124784 + - type: euclidean_spearman + value: 53.581740089351996 + - type: main_score + value: 53.581740089351996 + - type: manhattan_pearson + value: 43.14606772261321 + - type: manhattan_spearman + value: 53.181416743856836 + - type: pearson + value: 41.459968983462765 + - type: spearman + value: 53.581740089351996 + task: + type: STS + - dataset: + config: fr-pl + name: MTEB STS22 (fr-pl) + revision: de9d86b3b84231dc21f76c7b7af1f28e2f57f6e3 + split: test + type: mteb/sts22-crosslingual-sts + metrics: + - type: cosine_pearson + value: 90.73440998579521 + - type: cosine_spearman + value: 84.51542547285167 + - type: euclidean_pearson + value: 90.27154868530998 + - type: euclidean_spearman + value: 84.51542547285167 + - type: main_score + value: 84.51542547285167 + - type: manhattan_pearson + value: 89.71184089391063 + - type: manhattan_spearman + value: 84.51542547285167 + - type: pearson + value: 90.73440998579521 + - type: spearman + value: 84.51542547285167 + task: + type: STS + - dataset: + config: default + name: MTEB SciFact-PL + revision: 47932a35f045ef8ed01ba82bf9ff67f6e109207e + split: test + type: clarin-knext/scifact-pl + metrics: + - type: main_score + value: 60.526 + - type: map_at_1 + value: 46.400000000000006 + - type: map_at_10 + value: 55.991 + - type: map_at_100 + value: 56.736 + - type: map_at_1000 + value: 56.8 + - type: map_at_20 + value: 56.547000000000004 + - type: map_at_3 + value: 53.603 + - type: map_at_5 + value: 54.913000000000004 + - type: mrr_at_1 + value: 49.0 + - type: mrr_at_10 + value: 57.2739417989418 + - type: mrr_at_100 + value: 57.853315794381885 + - type: mrr_at_1000 + value: 57.91488615065038 + - type: mrr_at_20 + value: 57.701351990516095 + - type: mrr_at_3 + value: 55.44444444444445 + - type: mrr_at_5 + value: 56.42777777777778 + - type: nauc_map_at_1000_diff1 + value: 57.35372319475057 + - type: nauc_map_at_1000_max + value: 43.875320768732436 + - type: nauc_map_at_1000_std + value: 13.322022193655078 + - type: nauc_map_at_100_diff1 + value: 57.313891527698644 + - type: nauc_map_at_100_max + value: 43.872731261320574 + - type: nauc_map_at_100_std + value: 13.326500791745696 + - type: nauc_map_at_10_diff1 + value: 57.39836451111907 + - type: nauc_map_at_10_max + value: 43.526572478876574 + - type: nauc_map_at_10_std + value: 12.985890578763163 + - type: nauc_map_at_1_diff1 + value: 63.12715359944983 + - type: nauc_map_at_1_max + value: 37.86501536906817 + - type: nauc_map_at_1_std + value: 6.322678498743812 + - type: nauc_map_at_20_diff1 + value: 57.34518997447502 + - type: nauc_map_at_20_max + value: 43.98066371339034 + - type: nauc_map_at_20_std + value: 13.295989078438582 + - type: nauc_map_at_3_diff1 + value: 58.349640442692106 + - type: nauc_map_at_3_max + value: 43.61371135332669 + - type: nauc_map_at_3_std + value: 12.624087711546398 + - type: nauc_map_at_5_diff1 + value: 57.60916401452899 + - type: nauc_map_at_5_max + value: 43.393670573290564 + - type: nauc_map_at_5_std + value: 12.734291845594658 + - type: nauc_mrr_at_1000_diff1 + value: 56.86336995088208 + - type: nauc_mrr_at_1000_max + value: 44.16063126157464 + - type: nauc_mrr_at_1000_std + value: 14.46809986588675 + - type: nauc_mrr_at_100_diff1 + value: 56.82713594496301 + - type: nauc_mrr_at_100_max + value: 44.15947777871654 + - type: nauc_mrr_at_100_std + value: 14.468630477543204 + - type: nauc_mrr_at_10_diff1 + value: 56.75309318959002 + - type: nauc_mrr_at_10_max + value: 44.07955977542136 + - type: nauc_mrr_at_10_std + value: 14.364429809429623 + - type: nauc_mrr_at_1_diff1 + value: 61.55378221555377 + - type: nauc_mrr_at_1_max + value: 40.56404604782708 + - type: nauc_mrr_at_1_std + value: 11.237989014734051 + - type: nauc_mrr_at_20_diff1 + value: 56.820233285042356 + - type: nauc_mrr_at_20_max + value: 44.273215159882724 + - type: nauc_mrr_at_20_std + value: 14.457019833008061 + - type: nauc_mrr_at_3_diff1 + value: 57.25776927599811 + - type: nauc_mrr_at_3_max + value: 44.85862445501357 + - type: nauc_mrr_at_3_std + value: 15.08481264828146 + - type: nauc_mrr_at_5_diff1 + value: 56.77734628869654 + - type: nauc_mrr_at_5_max + value: 44.228014048290945 + - type: nauc_mrr_at_5_std + value: 14.560343015151084 + - type: nauc_ndcg_at_1000_diff1 + value: 56.067561048231966 + - type: nauc_ndcg_at_1000_max + value: 45.16507129881619 + - type: nauc_ndcg_at_1000_std + value: 15.507347464297675 + - type: nauc_ndcg_at_100_diff1 + value: 55.14390108893301 + - type: nauc_ndcg_at_100_max + value: 45.18506685496001 + - type: nauc_ndcg_at_100_std + value: 15.445626965324468 + - type: nauc_ndcg_at_10_diff1 + value: 55.178985380720135 + - type: nauc_ndcg_at_10_max + value: 44.26794162720816 + - type: nauc_ndcg_at_10_std + value: 13.918171530733483 + - type: nauc_ndcg_at_1_diff1 + value: 61.55378221555377 + - type: nauc_ndcg_at_1_max + value: 40.56404604782708 + - type: nauc_ndcg_at_1_std + value: 11.237989014734051 + - type: nauc_ndcg_at_20_diff1 + value: 55.27265202726197 + - type: nauc_ndcg_at_20_max + value: 45.87216107265961 + - type: nauc_ndcg_at_20_std + value: 14.919203799084935 + - type: nauc_ndcg_at_3_diff1 + value: 56.27751882008217 + - type: nauc_ndcg_at_3_max + value: 45.65587414820722 + - type: nauc_ndcg_at_3_std + value: 14.994891252809934 + - type: nauc_ndcg_at_5_diff1 + value: 55.53169392843657 + - type: nauc_ndcg_at_5_max + value: 44.23153109548333 + - type: nauc_ndcg_at_5_std + value: 13.938058808850082 + - type: nauc_precision_at_1000_diff1 + value: -17.238089963012836 + - type: nauc_precision_at_1000_max + value: 23.866846371139047 + - type: nauc_precision_at_1000_std + value: 36.26953209140314 + - type: nauc_precision_at_100_diff1 + value: -1.126473234568327 + - type: nauc_precision_at_100_max + value: 31.108745423284912 + - type: nauc_precision_at_100_std + value: 30.642388176707968 + - type: nauc_precision_at_10_diff1 + value: 17.489823438228598 + - type: nauc_precision_at_10_max + value: 38.62598562468759 + - type: nauc_precision_at_10_std + value: 26.145677114183353 + - type: nauc_precision_at_1_diff1 + value: 61.55378221555377 + - type: nauc_precision_at_1_max + value: 40.56404604782708 + - type: nauc_precision_at_1_std + value: 11.237989014734051 + - type: nauc_precision_at_20_diff1 + value: 9.30790121361279 + - type: nauc_precision_at_20_max + value: 39.74539590936092 + - type: nauc_precision_at_20_std + value: 28.76347279979629 + - type: nauc_precision_at_3_diff1 + value: 36.56691202573061 + - type: nauc_precision_at_3_max + value: 47.33091823598092 + - type: nauc_precision_at_3_std + value: 25.97270398234439 + - type: nauc_precision_at_5_diff1 + value: 27.53989814653275 + - type: nauc_precision_at_5_max + value: 42.906811554469954 + - type: nauc_precision_at_5_std + value: 25.673969211341785 + - type: nauc_recall_at_1000_diff1 + value: 64.96265172735714 + - type: nauc_recall_at_1000_max + value: 70.79831932773095 + - type: nauc_recall_at_1000_std + value: 85.23576097105469 + - type: nauc_recall_at_100_diff1 + value: 43.528295808426066 + - type: nauc_recall_at_100_max + value: 51.37810072663499 + - type: nauc_recall_at_100_std + value: 28.73429502093998 + - type: nauc_recall_at_10_diff1 + value: 46.428201019869604 + - type: nauc_recall_at_10_max + value: 43.55728664156955 + - type: nauc_recall_at_10_std + value: 13.44915034828845 + - type: nauc_recall_at_1_diff1 + value: 63.12715359944983 + - type: nauc_recall_at_1_max + value: 37.86501536906817 + - type: nauc_recall_at_1_std + value: 6.322678498743812 + - type: nauc_recall_at_20_diff1 + value: 46.569137618158344 + - type: nauc_recall_at_20_max + value: 53.92571049318345 + - type: nauc_recall_at_20_std + value: 19.731744586272395 + - type: nauc_recall_at_3_diff1 + value: 51.783556556783786 + - type: nauc_recall_at_3_max + value: 46.66691776547845 + - type: nauc_recall_at_3_std + value: 16.374424975237936 + - type: nauc_recall_at_5_diff1 + value: 48.59699452539875 + - type: nauc_recall_at_5_max + value: 43.94693563083851 + - type: nauc_recall_at_5_std + value: 14.447872758117455 + - type: ndcg_at_1 + value: 49.0 + - type: ndcg_at_10 + value: 60.526 + - type: ndcg_at_100 + value: 63.583999999999996 + - type: ndcg_at_1000 + value: 65.017 + - type: ndcg_at_20 + value: 62.297999999999995 + - type: ndcg_at_3 + value: 56.345 + - type: ndcg_at_5 + value: 58.179 + - type: precision_at_1 + value: 49.0 + - type: precision_at_10 + value: 8.3 + - type: precision_at_100 + value: 0.993 + - type: precision_at_1000 + value: 0.11 + - type: precision_at_20 + value: 4.567 + - type: precision_at_3 + value: 22.556 + - type: precision_at_5 + value: 14.799999999999999 + - type: recall_at_1 + value: 46.400000000000006 + - type: recall_at_10 + value: 73.161 + - type: recall_at_100 + value: 86.5 + - type: recall_at_1000 + value: 97.333 + - type: recall_at_20 + value: 79.756 + - type: recall_at_3 + value: 61.533 + - type: recall_at_5 + value: 66.22200000000001 + task: + type: Retrieval + - dataset: + config: default + name: MTEB TRECCOVID-PL + revision: 81bcb408f33366c2a20ac54adafad1ae7e877fdd + split: test + type: clarin-knext/trec-covid-pl + metrics: + - type: main_score + value: 65.563 + - type: map_at_1 + value: 0.22100000000000003 + - type: map_at_10 + value: 1.635 + - type: map_at_100 + value: 8.618 + - type: map_at_1000 + value: 20.422 + - type: map_at_20 + value: 2.789 + - type: map_at_3 + value: 0.609 + - type: map_at_5 + value: 0.9249999999999999 + - type: mrr_at_1 + value: 82.0 + - type: mrr_at_10 + value: 88.50000000000001 + - type: mrr_at_100 + value: 88.50000000000001 + - type: mrr_at_1000 + value: 88.50000000000001 + - type: mrr_at_20 + value: 88.50000000000001 + - type: mrr_at_3 + value: 87.66666666666667 + - type: mrr_at_5 + value: 88.16666666666667 + - type: nauc_map_at_1000_diff1 + value: -25.967067965482478 + - type: nauc_map_at_1000_max + value: 72.92874918094974 + - type: nauc_map_at_1000_std + value: 84.9616839707479 + - type: nauc_map_at_100_diff1 + value: -21.42642167269316 + - type: nauc_map_at_100_max + value: 62.68226707073076 + - type: nauc_map_at_100_std + value: 75.56794844224473 + - type: nauc_map_at_10_diff1 + value: -25.660983963152447 + - type: nauc_map_at_10_max + value: 41.96962876232467 + - type: nauc_map_at_10_std + value: 40.45419086736852 + - type: nauc_map_at_1_diff1 + value: -8.591425903743453 + - type: nauc_map_at_1_max + value: 17.51978543502864 + - type: nauc_map_at_1_std + value: 20.076935065027513 + - type: nauc_map_at_20_diff1 + value: -25.891301436996805 + - type: nauc_map_at_20_max + value: 47.84153247293719 + - type: nauc_map_at_20_std + value: 51.816446917682555 + - type: nauc_map_at_3_diff1 + value: -13.848652828499809 + - type: nauc_map_at_3_max + value: 26.243795101152205 + - type: nauc_map_at_3_std + value: 23.036433586747922 + - type: nauc_map_at_5_diff1 + value: -21.1896066711519 + - type: nauc_map_at_5_max + value: 31.324922426520786 + - type: nauc_map_at_5_std + value: 29.15619704280249 + - type: nauc_mrr_at_1000_diff1 + value: 24.59044392889167 + - type: nauc_mrr_at_1000_max + value: 62.571343173116645 + - type: nauc_mrr_at_1000_std + value: 65.96948433652632 + - type: nauc_mrr_at_100_diff1 + value: 24.59044392889167 + - type: nauc_mrr_at_100_max + value: 62.571343173116645 + - type: nauc_mrr_at_100_std + value: 65.96948433652632 + - type: nauc_mrr_at_10_diff1 + value: 24.59044392889167 + - type: nauc_mrr_at_10_max + value: 62.571343173116645 + - type: nauc_mrr_at_10_std + value: 65.96948433652632 + - type: nauc_mrr_at_1_diff1 + value: 27.541579469104494 + - type: nauc_mrr_at_1_max + value: 63.63586298050444 + - type: nauc_mrr_at_1_std + value: 65.66802511289794 + - type: nauc_mrr_at_20_diff1 + value: 24.59044392889167 + - type: nauc_mrr_at_20_max + value: 62.571343173116645 + - type: nauc_mrr_at_20_std + value: 65.96948433652632 + - type: nauc_mrr_at_3_diff1 + value: 26.89052291903615 + - type: nauc_mrr_at_3_max + value: 64.43515751051395 + - type: nauc_mrr_at_3_std + value: 65.28156161557388 + - type: nauc_mrr_at_5_diff1 + value: 24.226184721205527 + - type: nauc_mrr_at_5_max + value: 62.856866617864384 + - type: nauc_mrr_at_5_std + value: 66.58954986809582 + - type: nauc_ndcg_at_1000_diff1 + value: -28.716159160219824 + - type: nauc_ndcg_at_1000_max + value: 72.97926866355515 + - type: nauc_ndcg_at_1000_std + value: 84.63716554628942 + - type: nauc_ndcg_at_100_diff1 + value: -24.58125590309421 + - type: nauc_ndcg_at_100_max + value: 64.7106381911207 + - type: nauc_ndcg_at_100_std + value: 78.21077458187217 + - type: nauc_ndcg_at_10_diff1 + value: -18.358464881146272 + - type: nauc_ndcg_at_10_max + value: 67.10104488172087 + - type: nauc_ndcg_at_10_std + value: 71.68515146865761 + - type: nauc_ndcg_at_1_diff1 + value: 5.718639098904514 + - type: nauc_ndcg_at_1_max + value: 46.7462582934732 + - type: nauc_ndcg_at_1_std + value: 60.704366610091085 + - type: nauc_ndcg_at_20_diff1 + value: -25.276773792441947 + - type: nauc_ndcg_at_20_max + value: 65.84214731568228 + - type: nauc_ndcg_at_20_std + value: 75.85064732973377 + - type: nauc_ndcg_at_3_diff1 + value: 5.786838453770751 + - type: nauc_ndcg_at_3_max + value: 56.873202263808054 + - type: nauc_ndcg_at_3_std + value: 58.29378534996971 + - type: nauc_ndcg_at_5_diff1 + value: -10.849506785055805 + - type: nauc_ndcg_at_5_max + value: 60.86780916587185 + - type: nauc_ndcg_at_5_std + value: 64.67628462822093 + - type: nauc_precision_at_1000_diff1 + value: -17.658051874711777 + - type: nauc_precision_at_1000_max + value: 52.243182152280795 + - type: nauc_precision_at_1000_std + value: 52.221414126675405 + - type: nauc_precision_at_100_diff1 + value: -21.912367625930017 + - type: nauc_precision_at_100_max + value: 62.10951205181152 + - type: nauc_precision_at_100_std + value: 76.90051049763301 + - type: nauc_precision_at_10_diff1 + value: -24.589938416247513 + - type: nauc_precision_at_10_max + value: 69.12099854638501 + - type: nauc_precision_at_10_std + value: 73.49422451702836 + - type: nauc_precision_at_1_diff1 + value: 27.541579469104494 + - type: nauc_precision_at_1_max + value: 63.63586298050444 + - type: nauc_precision_at_1_std + value: 65.66802511289794 + - type: nauc_precision_at_20_diff1 + value: -29.032893299104007 + - type: nauc_precision_at_20_max + value: 67.03945899084314 + - type: nauc_precision_at_20_std + value: 78.94031560468493 + - type: nauc_precision_at_3_diff1 + value: 5.476377111095327 + - type: nauc_precision_at_3_max + value: 67.50694790850136 + - type: nauc_precision_at_3_std + value: 54.68716596593738 + - type: nauc_precision_at_5_diff1 + value: -20.104469911928415 + - type: nauc_precision_at_5_max + value: 66.46184176585471 + - type: nauc_precision_at_5_std + value: 62.40188672292436 + - type: nauc_recall_at_1000_diff1 + value: -28.649258704821147 + - type: nauc_recall_at_1000_max + value: 69.16314905082237 + - type: nauc_recall_at_1000_std + value: 79.32321876901818 + - type: nauc_recall_at_100_diff1 + value: -19.180264922861635 + - type: nauc_recall_at_100_max + value: 48.57682348744147 + - type: nauc_recall_at_100_std + value: 65.9297008167438 + - type: nauc_recall_at_10_diff1 + value: -23.597918176676906 + - type: nauc_recall_at_10_max + value: 35.163103297663795 + - type: nauc_recall_at_10_std + value: 36.537988377094486 + - type: nauc_recall_at_1_diff1 + value: -8.591425903743453 + - type: nauc_recall_at_1_max + value: 17.51978543502864 + - type: nauc_recall_at_1_std + value: 20.076935065027513 + - type: nauc_recall_at_20_diff1 + value: -25.441741663652127 + - type: nauc_recall_at_20_max + value: 37.09848404778578 + - type: nauc_recall_at_20_std + value: 42.94987387680688 + - type: nauc_recall_at_3_diff1 + value: -14.825486887364592 + - type: nauc_recall_at_3_max + value: 22.7488507817971 + - type: nauc_recall_at_3_std + value: 18.642543682159996 + - type: nauc_recall_at_5_diff1 + value: -21.938394755683525 + - type: nauc_recall_at_5_max + value: 27.492935982827078 + - type: nauc_recall_at_5_std + value: 27.013096425647436 + - type: ndcg_at_1 + value: 71.0 + - type: ndcg_at_10 + value: 65.563 + - type: ndcg_at_100 + value: 48.929 + - type: ndcg_at_1000 + value: 44.061 + - type: ndcg_at_20 + value: 61.917 + - type: ndcg_at_3 + value: 71.33500000000001 + - type: ndcg_at_5 + value: 68.135 + - type: precision_at_1 + value: 82.0 + - type: precision_at_10 + value: 69.39999999999999 + - type: precision_at_100 + value: 50.0 + - type: precision_at_1000 + value: 19.564 + - type: precision_at_20 + value: 65.4 + - type: precision_at_3 + value: 79.333 + - type: precision_at_5 + value: 73.2 + - type: recall_at_1 + value: 0.22100000000000003 + - type: recall_at_10 + value: 1.841 + - type: recall_at_100 + value: 11.959 + - type: recall_at_1000 + value: 41.373 + - type: recall_at_20 + value: 3.376 + - type: recall_at_3 + value: 0.6459999999999999 + - type: recall_at_5 + value: 0.9900000000000001 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CEDRClassification + revision: c0ba03d058e3e1b2f3fd20518875a4563dd12db4 + split: test + type: ai-forever/cedr-classification + metrics: + - type: accuracy + value: 43.76195536663124 + - type: f1 + value: 42.465506071271584 + - type: lrap + value: 73.62991498406049 + - type: main_score + value: 43.76195536663124 + task: + type: MultilabelClassification + - dataset: + config: default + name: MTEB GeoreviewClassification + revision: 3765c0d1de6b7d264bc459433c45e5a75513839c + split: test + type: ai-forever/georeview-classification + metrics: + - type: accuracy + value: 51.50390625 + - type: f1 + value: 47.10882921865018 + - type: f1_weighted + value: 47.10602639734162 + - type: main_score + value: 51.50390625 + task: + type: Classification + - dataset: + config: default + name: MTEB GeoreviewClusteringP2P + revision: 97a313c8fc85b47f13f33e7e9a95c1ad888c7fec + split: test + type: ai-forever/georeview-clustering-p2p + metrics: + - type: main_score + value: 60.763743799017725 + - type: v_measure + value: 60.763743799017725 + - type: v_measure_std + value: 0.6572810824719637 + task: + type: Clustering + - dataset: + config: default + name: MTEB HeadlineClassification + revision: 2fe05ee6b5832cda29f2ef7aaad7b7fe6a3609eb + split: test + type: ai-forever/headline-classification + metrics: + - type: accuracy + value: 84.365234375 + - type: f1 + value: 84.3504758444136 + - type: f1_weighted + value: 84.34875898591558 + - type: main_score + value: 84.365234375 + task: + type: Classification + - dataset: + config: default + name: MTEB InappropriatenessClassification + revision: 601651fdc45ef243751676e62dd7a19f491c0285 + split: test + type: ai-forever/inappropriateness-classification + metrics: + - type: accuracy + value: 65.888671875 + - type: ap + value: 60.9135452326786 + - type: ap_weighted + value: 60.9135452326786 + - type: f1 + value: 65.39737070466047 + - type: f1_weighted + value: 65.39737070466047 + - type: main_score + value: 65.888671875 + task: + type: Classification + - dataset: + config: default + name: MTEB KinopoiskClassification + revision: 5911f26666ac11af46cb9c6849d0dc80a378af24 + split: test + type: ai-forever/kinopoisk-sentiment-classification + metrics: + - type: accuracy + value: 62.28666666666667 + - type: f1 + value: 58.05000673882692 + - type: f1_weighted + value: 58.05000673882692 + - type: main_score + value: 62.28666666666667 + task: + type: Classification + - dataset: + config: ru + name: MTEB MIRACLReranking (ru) + revision: 6d1962c527217f8927fca80f890f14f36b2802af + split: dev + type: miracl/mmteb-miracl-reranking + metrics: + - type: MAP@1(MIRACL) + value: 30.086000000000002 + - type: MAP@10(MIRACL) + value: 49.463 + - type: MAP@100(MIRACL) + value: 51.49 + - type: MAP@1000(MIRACL) + value: 51.49 + - type: MAP@20(MIRACL) + value: 50.876999999999995 + - type: MAP@3(MIRACL) + value: 42.693 + - type: MAP@5(MIRACL) + value: 46.400999999999996 + - type: NDCG@1(MIRACL) + value: 49.639 + - type: NDCG@10(MIRACL) + value: 57.247 + - type: NDCG@100(MIRACL) + value: 61.378 + - type: NDCG@1000(MIRACL) + value: 61.378 + - type: NDCG@20(MIRACL) + value: 59.760000000000005 + - type: NDCG@3(MIRACL) + value: 51.608 + - type: NDCG@5(MIRACL) + value: 53.759 + - type: P@1(MIRACL) + value: 49.639 + - type: P@10(MIRACL) + value: 15.028 + - type: P@100(MIRACL) + value: 1.9529999999999998 + - type: P@1000(MIRACL) + value: 0.19499999999999998 + - type: P@20(MIRACL) + value: 8.753 + - type: P@3(MIRACL) + value: 32.05 + - type: P@5(MIRACL) + value: 24.074 + - type: Recall@1(MIRACL) + value: 30.086000000000002 + - type: Recall@10(MIRACL) + value: 67.67 + - type: Recall@100(MIRACL) + value: 79.952 + - type: Recall@1000(MIRACL) + value: 79.952 + - type: Recall@20(MIRACL) + value: 74.409 + - type: Recall@3(MIRACL) + value: 50.432 + - type: Recall@5(MIRACL) + value: 58.431999999999995 + - type: main_score + value: 57.247 + - type: nAUC_MAP@1000_diff1(MIRACL) + value: 14.49132703139068 + - type: nAUC_MAP@1000_max(MIRACL) + value: 32.22718193428578 + - type: nAUC_MAP@1000_std(MIRACL) + value: 13.815910840085923 + - type: nAUC_MAP@100_diff1(MIRACL) + value: 14.49132703139068 + - type: nAUC_MAP@100_max(MIRACL) + value: 32.22718193428578 + - type: nAUC_MAP@100_std(MIRACL) + value: 13.815910840085923 + - type: nAUC_MAP@10_diff1(MIRACL) + value: 16.504956542658086 + - type: nAUC_MAP@10_max(MIRACL) + value: 30.050872244941402 + - type: nAUC_MAP@10_std(MIRACL) + value: 13.136444562982783 + - type: nAUC_MAP@1_diff1(MIRACL) + value: 34.18909528530817 + - type: nAUC_MAP@1_max(MIRACL) + value: 15.062828187996063 + - type: nAUC_MAP@1_std(MIRACL) + value: 3.2097645527643595 + - type: nAUC_MAP@20_diff1(MIRACL) + value: 15.130045331734273 + - type: nAUC_MAP@20_max(MIRACL) + value: 31.64805643481855 + - type: nAUC_MAP@20_std(MIRACL) + value: 13.811628131618614 + - type: nAUC_MAP@3_diff1(MIRACL) + value: 22.967076674429805 + - type: nAUC_MAP@3_max(MIRACL) + value: 23.55271431173739 + - type: nAUC_MAP@3_std(MIRACL) + value: 8.38797791381467 + - type: nAUC_MAP@5_diff1(MIRACL) + value: 18.864687263601386 + - type: nAUC_MAP@5_max(MIRACL) + value: 27.137349141940113 + - type: nAUC_MAP@5_std(MIRACL) + value: 10.729766654007843 + - type: nAUC_NDCG@1000_diff1(MIRACL) + value: 3.9965439655979824 + - type: nAUC_NDCG@1000_max(MIRACL) + value: 40.636834454723754 + - type: nAUC_NDCG@1000_std(MIRACL) + value: 17.562271508485765 + - type: nAUC_NDCG@100_diff1(MIRACL) + value: 3.9965439655979824 + - type: nAUC_NDCG@100_max(MIRACL) + value: 40.636834454723754 + - type: nAUC_NDCG@100_std(MIRACL) + value: 17.562271508485765 + - type: nAUC_NDCG@10_diff1(MIRACL) + value: 9.20785070475628 + - type: nAUC_NDCG@10_max(MIRACL) + value: 36.10685520213844 + - type: nAUC_NDCG@10_std(MIRACL) + value: 16.52302351987603 + - type: nAUC_NDCG@1_diff1(MIRACL) + value: 21.380050974695997 + - type: nAUC_NDCG@1_max(MIRACL) + value: 34.082244652607194 + - type: nAUC_NDCG@1_std(MIRACL) + value: 11.960002005286022 + - type: nAUC_NDCG@20_diff1(MIRACL) + value: 6.029314693566221 + - type: nAUC_NDCG@20_max(MIRACL) + value: 38.93324906908806 + - type: nAUC_NDCG@20_std(MIRACL) + value: 17.800187238709555 + - type: nAUC_NDCG@3_diff1(MIRACL) + value: 14.75650513824637 + - type: nAUC_NDCG@3_max(MIRACL) + value: 31.345509680914045 + - type: nAUC_NDCG@3_std(MIRACL) + value: 12.489558805685855 + - type: nAUC_NDCG@5_diff1(MIRACL) + value: 12.114746297880274 + - type: nAUC_NDCG@5_max(MIRACL) + value: 32.80902906671674 + - type: nAUC_NDCG@5_std(MIRACL) + value: 13.673134481637822 + - type: nAUC_P@1000_diff1(MIRACL) + value: -28.3393493752439 + - type: nAUC_P@1000_max(MIRACL) + value: 29.94108745927258 + - type: nAUC_P@1000_std(MIRACL) + value: 13.00979331888063 + - type: nAUC_P@100_diff1(MIRACL) + value: -28.33934937524387 + - type: nAUC_P@100_max(MIRACL) + value: 29.94108745927262 + - type: nAUC_P@100_std(MIRACL) + value: 13.009793318880677 + - type: nAUC_P@10_diff1(MIRACL) + value: -24.128263659855403 + - type: nAUC_P@10_max(MIRACL) + value: 33.69795666200538 + - type: nAUC_P@10_std(MIRACL) + value: 18.667513470521502 + - type: nAUC_P@1_diff1(MIRACL) + value: 21.380050974695997 + - type: nAUC_P@1_max(MIRACL) + value: 34.082244652607194 + - type: nAUC_P@1_std(MIRACL) + value: 11.960002005286022 + - type: nAUC_P@20_diff1(MIRACL) + value: -26.851579419952216 + - type: nAUC_P@20_max(MIRACL) + value: 32.30632064213693 + - type: nAUC_P@20_std(MIRACL) + value: 16.71537079207824 + - type: nAUC_P@3_diff1(MIRACL) + value: -11.927325953384901 + - type: nAUC_P@3_max(MIRACL) + value: 37.1736865658166 + - type: nAUC_P@3_std(MIRACL) + value: 16.2754083159161 + - type: nAUC_P@5_diff1(MIRACL) + value: -20.552820285005026 + - type: nAUC_P@5_max(MIRACL) + value: 36.30994351506595 + - type: nAUC_P@5_std(MIRACL) + value: 17.40523400441202 + - type: nAUC_Recall@1000_diff1(MIRACL) + value: -32.466276225017346 + - type: nAUC_Recall@1000_max(MIRACL) + value: 61.05870930653621 + - type: nAUC_Recall@1000_std(MIRACL) + value: 27.09559295245869 + - type: nAUC_Recall@100_diff1(MIRACL) + value: -32.466276225017346 + - type: nAUC_Recall@100_max(MIRACL) + value: 61.05870930653621 + - type: nAUC_Recall@100_std(MIRACL) + value: 27.09559295245869 + - type: nAUC_Recall@10_diff1(MIRACL) + value: -4.979553747407279 + - type: nAUC_Recall@10_max(MIRACL) + value: 38.39843112329541 + - type: nAUC_Recall@10_std(MIRACL) + value: 20.600178046207702 + - type: nAUC_Recall@1_diff1(MIRACL) + value: 34.18909528530817 + - type: nAUC_Recall@1_max(MIRACL) + value: 15.062828187996063 + - type: nAUC_Recall@1_std(MIRACL) + value: 3.2097645527643595 + - type: nAUC_Recall@20_diff1(MIRACL) + value: -18.18036800889588 + - type: nAUC_Recall@20_max(MIRACL) + value: 48.61058795989584 + - type: nAUC_Recall@20_std(MIRACL) + value: 26.280248121862066 + - type: nAUC_Recall@3_diff1(MIRACL) + value: 14.170577180062406 + - type: nAUC_Recall@3_max(MIRACL) + value: 22.458002088633343 + - type: nAUC_Recall@3_std(MIRACL) + value: 10.035396517805115 + - type: nAUC_Recall@5_diff1(MIRACL) + value: 4.550643279455007 + - type: nAUC_Recall@5_max(MIRACL) + value: 28.457322956745617 + - type: nAUC_Recall@5_std(MIRACL) + value: 13.335687765558976 + task: + type: Reranking + - dataset: + config: ru + name: MTEB MIRACLRetrieval (ru) + revision: main + split: dev + type: miracl/mmteb-miracl + metrics: + - type: main_score + value: 54.595000000000006 + - type: map_at_1 + value: 23.952 + - type: map_at_10 + value: 43.637 + - type: map_at_100 + value: 46.6 + - type: map_at_1000 + value: 46.697 + - type: map_at_20 + value: 45.378 + - type: map_at_3 + value: 36.137 + - type: map_at_5 + value: 39.838 + - type: mrr_at_1 + value: 48.48242811501598 + - type: mrr_at_10 + value: 61.81946980069989 + - type: mrr_at_100 + value: 62.23878510656363 + - type: mrr_at_1000 + value: 62.24764124248049 + - type: mrr_at_20 + value: 62.1051846692803 + - type: mrr_at_3 + value: 58.82587859424925 + - type: mrr_at_5 + value: 60.60303514377006 + - type: nauc_map_at_1000_diff1 + value: 28.214871462783854 + - type: nauc_map_at_1000_max + value: 25.201084766782078 + - type: nauc_map_at_1000_std + value: 6.765255742651155 + - type: nauc_map_at_100_diff1 + value: 28.234612621431754 + - type: nauc_map_at_100_max + value: 25.165912406356338 + - type: nauc_map_at_100_std + value: 6.7694968272696405 + - type: nauc_map_at_10_diff1 + value: 28.765507147170922 + - type: nauc_map_at_10_max + value: 23.216218446772334 + - type: nauc_map_at_10_std + value: 4.495189160179304 + - type: nauc_map_at_1_diff1 + value: 36.05018310203833 + - type: nauc_map_at_1_max + value: 13.651823625223662 + - type: nauc_map_at_1_std + value: -1.1689324424590004 + - type: nauc_map_at_20_diff1 + value: 28.471404911269993 + - type: nauc_map_at_20_max + value: 24.40961097812379 + - type: nauc_map_at_20_std + value: 5.62151334482884 + - type: nauc_map_at_3_diff1 + value: 29.40681082033128 + - type: nauc_map_at_3_max + value: 20.2388315012222 + - type: nauc_map_at_3_std + value: 1.3400996743731126 + - type: nauc_map_at_5_diff1 + value: 28.258533407547485 + - type: nauc_map_at_5_max + value: 21.966173291189982 + - type: nauc_map_at_5_std + value: 3.105883435302439 + - type: nauc_mrr_at_1000_diff1 + value: 29.973540045513143 + - type: nauc_mrr_at_1000_max + value: 32.675046742557335 + - type: nauc_mrr_at_1000_std + value: 13.045750572535795 + - type: nauc_mrr_at_100_diff1 + value: 29.974727506122523 + - type: nauc_mrr_at_100_max + value: 32.68341978762026 + - type: nauc_mrr_at_100_std + value: 13.06124318794068 + - type: nauc_mrr_at_10_diff1 + value: 29.811240994564102 + - type: nauc_mrr_at_10_max + value: 32.63357319986243 + - type: nauc_mrr_at_10_std + value: 13.063153729215351 + - type: nauc_mrr_at_1_diff1 + value: 33.4099136989708 + - type: nauc_mrr_at_1_max + value: 28.034252424559693 + - type: nauc_mrr_at_1_std + value: 8.771691345021653 + - type: nauc_mrr_at_20_diff1 + value: 29.906265270643118 + - type: nauc_mrr_at_20_max + value: 32.752505338776125 + - type: nauc_mrr_at_20_std + value: 13.092155055233029 + - type: nauc_mrr_at_3_diff1 + value: 30.159884994169563 + - type: nauc_mrr_at_3_max + value: 32.93411539112575 + - type: nauc_mrr_at_3_std + value: 12.479656041605635 + - type: nauc_mrr_at_5_diff1 + value: 29.747570134961947 + - type: nauc_mrr_at_5_max + value: 32.92420276622243 + - type: nauc_mrr_at_5_std + value: 13.228404409126401 + - type: nauc_ndcg_at_1000_diff1 + value: 26.973170727985845 + - type: nauc_ndcg_at_1000_max + value: 30.714533698659235 + - type: nauc_ndcg_at_1000_std + value: 12.646817183789668 + - type: nauc_ndcg_at_100_diff1 + value: 26.9185712126953 + - type: nauc_ndcg_at_100_max + value: 30.787272670713378 + - type: nauc_ndcg_at_100_std + value: 13.347713759949965 + - type: nauc_ndcg_at_10_diff1 + value: 27.977378333059967 + - type: nauc_ndcg_at_10_max + value: 26.09323198226889 + - type: nauc_ndcg_at_10_std + value: 7.669332805590227 + - type: nauc_ndcg_at_1_diff1 + value: 33.4099136989708 + - type: nauc_ndcg_at_1_max + value: 28.034252424559693 + - type: nauc_ndcg_at_1_std + value: 8.771691345021653 + - type: nauc_ndcg_at_20_diff1 + value: 27.437445515604885 + - type: nauc_ndcg_at_20_max + value: 28.652644415118107 + - type: nauc_ndcg_at_20_std + value: 9.979585555866413 + - type: nauc_ndcg_at_3_diff1 + value: 26.78918281794111 + - type: nauc_ndcg_at_3_max + value: 26.743605264167563 + - type: nauc_ndcg_at_3_std + value: 7.636594897931739 + - type: nauc_ndcg_at_5_diff1 + value: 26.768664779524386 + - type: nauc_ndcg_at_5_max + value: 25.92430880031359 + - type: nauc_ndcg_at_5_std + value: 7.567930905471419 + - type: nauc_precision_at_1000_diff1 + value: -12.46821203258456 + - type: nauc_precision_at_1000_max + value: 16.582341013228657 + - type: nauc_precision_at_1000_std + value: 19.378859711054375 + - type: nauc_precision_at_100_diff1 + value: -11.326145373988833 + - type: nauc_precision_at_100_max + value: 19.58909158403411 + - type: nauc_precision_at_100_std + value: 23.328633133712064 + - type: nauc_precision_at_10_diff1 + value: -0.8745238638551284 + - type: nauc_precision_at_10_max + value: 24.17820027443609 + - type: nauc_precision_at_10_std + value: 18.396638849812433 + - type: nauc_precision_at_1_diff1 + value: 33.4099136989708 + - type: nauc_precision_at_1_max + value: 28.034252424559693 + - type: nauc_precision_at_1_std + value: 8.771691345021653 + - type: nauc_precision_at_20_diff1 + value: -5.306130413062489 + - type: nauc_precision_at_20_max + value: 23.52655673001502 + - type: nauc_precision_at_20_std + value: 21.167397186114314 + - type: nauc_precision_at_3_diff1 + value: 8.580491424277982 + - type: nauc_precision_at_3_max + value: 31.019714029464208 + - type: nauc_precision_at_3_std + value: 14.133874593516971 + - type: nauc_precision_at_5_diff1 + value: 2.554020245713193 + - type: nauc_precision_at_5_max + value: 29.634541238020105 + - type: nauc_precision_at_5_std + value: 17.87723197950384 + - type: nauc_recall_at_1000_diff1 + value: 13.125393258634418 + - type: nauc_recall_at_1000_max + value: 62.24470202796473 + - type: nauc_recall_at_1000_std + value: 63.23164754823915 + - type: nauc_recall_at_100_diff1 + value: 14.423784358841202 + - type: nauc_recall_at_100_max + value: 38.617863069715675 + - type: nauc_recall_at_100_std + value: 39.097234031326536 + - type: nauc_recall_at_10_diff1 + value: 20.936852580771 + - type: nauc_recall_at_10_max + value: 16.594501361269014 + - type: nauc_recall_at_10_std + value: 4.532729652479956 + - type: nauc_recall_at_1_diff1 + value: 36.05018310203833 + - type: nauc_recall_at_1_max + value: 13.651823625223662 + - type: nauc_recall_at_1_std + value: -1.1689324424590004 + - type: nauc_recall_at_20_diff1 + value: 18.26313195777628 + - type: nauc_recall_at_20_max + value: 22.504290386601888 + - type: nauc_recall_at_20_std + value: 9.794573138033334 + - type: nauc_recall_at_3_diff1 + value: 23.738927052170915 + - type: nauc_recall_at_3_max + value: 17.92200131347856 + - type: nauc_recall_at_3_std + value: 0.5468756827969604 + - type: nauc_recall_at_5_diff1 + value: 20.33968031427608 + - type: nauc_recall_at_5_max + value: 17.602814155288783 + - type: nauc_recall_at_5_std + value: 2.666858198246704 + - type: ndcg_at_1 + value: 48.482 + - type: ndcg_at_10 + value: 54.595000000000006 + - type: ndcg_at_100 + value: 62.382000000000005 + - type: ndcg_at_1000 + value: 63.463 + - type: ndcg_at_20 + value: 58.24399999999999 + - type: ndcg_at_3 + value: 47.924 + - type: ndcg_at_5 + value: 49.716 + - type: precision_at_1 + value: 48.482 + - type: precision_at_10 + value: 16.901 + - type: precision_at_100 + value: 2.576 + - type: precision_at_1000 + value: 0.27799999999999997 + - type: precision_at_20 + value: 10.188 + - type: precision_at_3 + value: 33.227000000000004 + - type: precision_at_5 + value: 25.335 + - type: recall_at_1 + value: 23.952 + - type: recall_at_10 + value: 66.591 + - type: recall_at_100 + value: 91.902 + - type: recall_at_1000 + value: 97.772 + - type: recall_at_20 + value: 76.768 + - type: recall_at_3 + value: 43.954 + - type: recall_at_5 + value: 52.943 + task: + type: Retrieval + - dataset: + config: ru + name: MTEB MassiveIntentClassification (ru) + revision: 4672e20407010da34463acc759c162ca9734bca6 + split: test + type: mteb/amazon_massive_intent + metrics: + - type: accuracy + value: 66.0659045057162 + - type: f1 + value: 61.35064382748825 + - type: f1_weighted + value: 64.6551910369696 + - type: main_score + value: 66.0659045057162 + task: + type: Classification + - dataset: + config: ru + name: MTEB MassiveScenarioClassification (ru) + revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 + split: test + type: mteb/amazon_massive_scenario + metrics: + - type: accuracy + value: 74.80833893745798 + - type: f1 + value: 73.75893026985123 + - type: f1_weighted + value: 74.48216382388233 + - type: main_score + value: 74.80833893745798 + task: + type: Classification + - dataset: + config: default + name: MTEB RUParaPhraserSTS + revision: 43265056790b8f7c59e0139acb4be0a8dad2c8f4 + split: test + type: merionum/ru_paraphraser + metrics: + - type: cosine_pearson + value: 60.71240323001735 + - type: cosine_spearman + value: 68.14010632490151 + - type: euclidean_pearson + value: 65.04715258693518 + - type: euclidean_spearman + value: 68.14015844762187 + - type: main_score + value: 68.14010632490151 + - type: manhattan_pearson + value: 65.00225170739532 + - type: manhattan_spearman + value: 67.99309271156126 + - type: pearson + value: 60.71240323001735 + - type: spearman + value: 68.14010632490151 + task: + type: STS + - dataset: + config: default + name: MTEB RiaNewsRetrieval + revision: 82374b0bbacda6114f39ff9c5b925fa1512ca5d7 + split: test + type: ai-forever/ria-news-retrieval + metrics: + - type: main_score + value: 78.172 + - type: map_at_1 + value: 65.97 + - type: map_at_10 + value: 74.494 + - type: map_at_100 + value: 74.811 + - type: map_at_1000 + value: 74.821 + - type: map_at_20 + value: 74.712 + - type: map_at_3 + value: 73.072 + - type: map_at_5 + value: 73.969 + - type: mrr_at_1 + value: 65.97 + - type: mrr_at_10 + value: 74.49480158730101 + - type: mrr_at_100 + value: 74.81143726471089 + - type: mrr_at_1000 + value: 74.82174846844643 + - type: mrr_at_20 + value: 74.7122507574977 + - type: mrr_at_3 + value: 73.07166666666618 + - type: mrr_at_5 + value: 73.96916666666591 + - type: nauc_map_at_1000_diff1 + value: 73.08686946777952 + - type: nauc_map_at_1000_max + value: 24.925205621117783 + - type: nauc_map_at_1000_std + value: -12.15886477655549 + - type: nauc_map_at_100_diff1 + value: 73.08244355589704 + - type: nauc_map_at_100_max + value: 24.93258145762047 + - type: nauc_map_at_100_std + value: -12.149308732380295 + - type: nauc_map_at_10_diff1 + value: 72.97848397979895 + - type: nauc_map_at_10_max + value: 24.887366324397487 + - type: nauc_map_at_10_std + value: -12.284523808652416 + - type: nauc_map_at_1_diff1 + value: 76.45682519258669 + - type: nauc_map_at_1_max + value: 22.373249835264456 + - type: nauc_map_at_1_std + value: -14.13464876294517 + - type: nauc_map_at_20_diff1 + value: 73.04810922101197 + - type: nauc_map_at_20_max + value: 24.92236406693643 + - type: nauc_map_at_20_std + value: -12.176293947009215 + - type: nauc_map_at_3_diff1 + value: 72.9755537768131 + - type: nauc_map_at_3_max + value: 24.620733273355683 + - type: nauc_map_at_3_std + value: -12.85288384470766 + - type: nauc_map_at_5_diff1 + value: 73.00265519892334 + - type: nauc_map_at_5_max + value: 24.786315954855553 + - type: nauc_map_at_5_std + value: -12.564020496189352 + - type: nauc_mrr_at_1000_diff1 + value: 73.08804965487802 + - type: nauc_mrr_at_1000_max + value: 24.924428532055636 + - type: nauc_mrr_at_1000_std + value: -12.160384139274424 + - type: nauc_mrr_at_100_diff1 + value: 73.08362309430365 + - type: nauc_mrr_at_100_max + value: 24.931804884947883 + - type: nauc_mrr_at_100_std + value: -12.150827182790605 + - type: nauc_mrr_at_10_diff1 + value: 72.9796448439789 + - type: nauc_mrr_at_10_max + value: 24.88659980440671 + - type: nauc_mrr_at_10_std + value: -12.286023526026135 + - type: nauc_mrr_at_1_diff1 + value: 76.45682519258669 + - type: nauc_mrr_at_1_max + value: 22.373249835264456 + - type: nauc_mrr_at_1_std + value: -14.13464876294517 + - type: nauc_mrr_at_20_diff1 + value: 73.04928271486784 + - type: nauc_mrr_at_20_max + value: 24.921590820144715 + - type: nauc_mrr_at_20_std + value: -12.177806033794447 + - type: nauc_mrr_at_3_diff1 + value: 72.9755537768131 + - type: nauc_mrr_at_3_max + value: 24.620733273355683 + - type: nauc_mrr_at_3_std + value: -12.85288384470766 + - type: nauc_mrr_at_5_diff1 + value: 73.00379294775456 + - type: nauc_mrr_at_5_max + value: 24.78556305369168 + - type: nauc_mrr_at_5_std + value: -12.565495148428068 + - type: nauc_ndcg_at_1000_diff1 + value: 72.17730594095745 + - type: nauc_ndcg_at_1000_max + value: 26.080483734410535 + - type: nauc_ndcg_at_1000_std + value: -10.600341719859815 + - type: nauc_ndcg_at_100_diff1 + value: 72.01297350077037 + - type: nauc_ndcg_at_100_max + value: 26.370245187512793 + - type: nauc_ndcg_at_100_std + value: -10.13804122238926 + - type: nauc_ndcg_at_10_diff1 + value: 71.54838244432331 + - type: nauc_ndcg_at_10_max + value: 26.132336506589553 + - type: nauc_ndcg_at_10_std + value: -10.895520797166812 + - type: nauc_ndcg_at_1_diff1 + value: 76.45682519258669 + - type: nauc_ndcg_at_1_max + value: 22.373249835264456 + - type: nauc_ndcg_at_1_std + value: -14.13464876294517 + - type: nauc_ndcg_at_20_diff1 + value: 71.78002529865489 + - type: nauc_ndcg_at_20_max + value: 26.34207650975323 + - type: nauc_ndcg_at_20_std + value: -10.389370339614697 + - type: nauc_ndcg_at_3_diff1 + value: 71.70864174615559 + - type: nauc_ndcg_at_3_max + value: 25.445588176267258 + - type: nauc_ndcg_at_3_std + value: -12.309378837447191 + - type: nauc_ndcg_at_5_diff1 + value: 71.67055602722155 + - type: nauc_ndcg_at_5_max + value: 25.818799225442223 + - type: nauc_ndcg_at_5_std + value: -11.71310643254645 + - type: nauc_precision_at_1000_diff1 + value: 52.795678598638375 + - type: nauc_precision_at_1000_max + value: 58.02042698469986 + - type: nauc_precision_at_1000_std + value: 38.79122201027639 + - type: nauc_precision_at_100_diff1 + value: 57.635497479970155 + - type: nauc_precision_at_100_max + value: 50.98528067422545 + - type: nauc_precision_at_100_std + value: 29.023249788572336 + - type: nauc_precision_at_10_diff1 + value: 62.88435357133938 + - type: nauc_precision_at_10_max + value: 33.93536070722828 + - type: nauc_precision_at_10_std + value: -1.6485440570585936 + - type: nauc_precision_at_1_diff1 + value: 76.45682519258669 + - type: nauc_precision_at_1_max + value: 22.373249835264456 + - type: nauc_precision_at_1_std + value: -14.13464876294517 + - type: nauc_precision_at_20_diff1 + value: 61.65978869266085 + - type: nauc_precision_at_20_max + value: 39.11301597287762 + - type: nauc_precision_at_20_std + value: 6.864118196298169 + - type: nauc_precision_at_3_diff1 + value: 66.8221462321319 + - type: nauc_precision_at_3_max + value: 28.6301783798532 + - type: nauc_precision_at_3_std + value: -10.17852659176101 + - type: nauc_precision_at_5_diff1 + value: 65.45372225751974 + - type: nauc_precision_at_5_max + value: 30.71086457162392 + - type: nauc_precision_at_5_std + value: -7.5734188075962345 + - type: nauc_recall_at_1000_diff1 + value: 52.79567859863725 + - type: nauc_recall_at_1000_max + value: 58.020426984700634 + - type: nauc_recall_at_1000_std + value: 38.79122201027698 + - type: nauc_recall_at_100_diff1 + value: 57.6354974799695 + - type: nauc_recall_at_100_max + value: 50.98528067422452 + - type: nauc_recall_at_100_std + value: 29.023249788571604 + - type: nauc_recall_at_10_diff1 + value: 62.884353571339155 + - type: nauc_recall_at_10_max + value: 33.93536070722848 + - type: nauc_recall_at_10_std + value: -1.6485440570586527 + - type: nauc_recall_at_1_diff1 + value: 76.45682519258669 + - type: nauc_recall_at_1_max + value: 22.373249835264456 + - type: nauc_recall_at_1_std + value: -14.13464876294517 + - type: nauc_recall_at_20_diff1 + value: 61.65978869266054 + - type: nauc_recall_at_20_max + value: 39.11301597287748 + - type: nauc_recall_at_20_std + value: 6.8641181962978965 + - type: nauc_recall_at_3_diff1 + value: 66.82214623213184 + - type: nauc_recall_at_3_max + value: 28.630178379853145 + - type: nauc_recall_at_3_std + value: -10.178526591761056 + - type: nauc_recall_at_5_diff1 + value: 65.45372225751966 + - type: nauc_recall_at_5_max + value: 30.710864571624015 + - type: nauc_recall_at_5_std + value: -7.573418807596015 + - type: ndcg_at_1 + value: 65.97 + - type: ndcg_at_10 + value: 78.172 + - type: ndcg_at_100 + value: 79.63799999999999 + - type: ndcg_at_1000 + value: 79.915 + - type: ndcg_at_20 + value: 78.949 + - type: ndcg_at_3 + value: 75.304 + - type: ndcg_at_5 + value: 76.924 + - type: precision_at_1 + value: 65.97 + - type: precision_at_10 + value: 8.948 + - type: precision_at_100 + value: 0.962 + - type: precision_at_1000 + value: 0.098 + - type: precision_at_20 + value: 4.626 + - type: precision_at_3 + value: 27.247 + - type: precision_at_5 + value: 17.136000000000003 + - type: recall_at_1 + value: 65.97 + - type: recall_at_10 + value: 89.48 + - type: recall_at_100 + value: 96.17999999999999 + - type: recall_at_1000 + value: 98.37 + - type: recall_at_20 + value: 92.52 + - type: recall_at_3 + value: 81.74 + - type: recall_at_5 + value: 85.68 + task: + type: Retrieval + - dataset: + config: default + name: MTEB RuBQReranking + revision: 2e96b8f098fa4b0950fc58eacadeb31c0d0c7fa2 + split: test + type: ai-forever/rubq-reranking + metrics: + - type: main_score + value: 70.23399911521713 + - type: map + value: 70.23399911521713 + - type: mrr + value: 75.79756531884192 + - type: nAUC_map_diff1 + value: 41.70907433488097 + - type: nAUC_map_max + value: 19.038198833852398 + - type: nAUC_map_std + value: 9.364787080049835 + - type: nAUC_mrr_diff1 + value: 45.427118778337864 + - type: nAUC_mrr_max + value: 23.422465198632338 + - type: nAUC_mrr_std + value: 12.352254802172363 + task: + type: Reranking + - dataset: + config: default + name: MTEB RuBQRetrieval + revision: e19b6ffa60b3bc248e0b41f4cc37c26a55c2a67b + split: test + type: ai-forever/rubq-retrieval + metrics: + - type: main_score + value: 67.97800000000001 + - type: map_at_1 + value: 38.682 + - type: map_at_10 + value: 59.602999999999994 + - type: map_at_100 + value: 60.609 + - type: map_at_1000 + value: 60.638999999999996 + - type: map_at_20 + value: 60.295 + - type: map_at_3 + value: 53.864000000000004 + - type: map_at_5 + value: 57.357 + - type: mrr_at_1 + value: 55.43735224586288 + - type: mrr_at_10 + value: 68.11629892303657 + - type: mrr_at_100 + value: 68.46735935081414 + - type: mrr_at_1000 + value: 68.47375950539598 + - type: mrr_at_20 + value: 68.3555220700062 + - type: mrr_at_3 + value: 65.64223798266359 + - type: mrr_at_5 + value: 67.26457840819555 + - type: nauc_map_at_1000_diff1 + value: 38.947165927647895 + - type: nauc_map_at_1000_max + value: 21.0400498401016 + - type: nauc_map_at_1000_std + value: -12.325977772218819 + - type: nauc_map_at_100_diff1 + value: 38.93708131706777 + - type: nauc_map_at_100_max + value: 21.05313115659324 + - type: nauc_map_at_100_std + value: -12.307766320896842 + - type: nauc_map_at_10_diff1 + value: 38.62500444502093 + - type: nauc_map_at_10_max + value: 20.841336922136993 + - type: nauc_map_at_10_std + value: -12.753687299190014 + - type: nauc_map_at_1_diff1 + value: 42.42534428928902 + - type: nauc_map_at_1_max + value: 12.831550041111084 + - type: nauc_map_at_1_std + value: -11.266929686927616 + - type: nauc_map_at_20_diff1 + value: 38.839757385230854 + - type: nauc_map_at_20_max + value: 21.155443051360105 + - type: nauc_map_at_20_std + value: -12.290324296540298 + - type: nauc_map_at_3_diff1 + value: 38.600886947896804 + - type: nauc_map_at_3_max + value: 18.67062373078893 + - type: nauc_map_at_3_std + value: -14.47069134931124 + - type: nauc_map_at_5_diff1 + value: 38.47519630592911 + - type: nauc_map_at_5_max + value: 19.424230739972682 + - type: nauc_map_at_5_std + value: -13.715848064929636 + - type: nauc_mrr_at_1000_diff1 + value: 45.829087222671134 + - type: nauc_mrr_at_1000_max + value: 24.0611740840992 + - type: nauc_mrr_at_1000_std + value: -13.779501374204361 + - type: nauc_mrr_at_100_diff1 + value: 45.82511154142628 + - type: nauc_mrr_at_100_max + value: 24.068094029026092 + - type: nauc_mrr_at_100_std + value: -13.771488599042799 + - type: nauc_mrr_at_10_diff1 + value: 45.66670841729006 + - type: nauc_mrr_at_10_max + value: 24.21302734762609 + - type: nauc_mrr_at_10_std + value: -13.878591942488693 + - type: nauc_mrr_at_1_diff1 + value: 49.10187561048428 + - type: nauc_mrr_at_1_max + value: 21.468296250291164 + - type: nauc_mrr_at_1_std + value: -14.16579631922114 + - type: nauc_mrr_at_20_diff1 + value: 45.780763156949334 + - type: nauc_mrr_at_20_max + value: 24.16790762743539 + - type: nauc_mrr_at_20_std + value: -13.798341816508907 + - type: nauc_mrr_at_3_diff1 + value: 45.81750015144424 + - type: nauc_mrr_at_3_max + value: 23.74635317348391 + - type: nauc_mrr_at_3_std + value: -14.465609504516872 + - type: nauc_mrr_at_5_diff1 + value: 45.50087795737209 + - type: nauc_mrr_at_5_max + value: 24.021128950654578 + - type: nauc_mrr_at_5_std + value: -14.316147652867976 + - type: nauc_ndcg_at_1000_diff1 + value: 39.98481177488298 + - type: nauc_ndcg_at_1000_max + value: 23.50673478711865 + - type: nauc_ndcg_at_1000_std + value: -11.112473477484098 + - type: nauc_ndcg_at_100_diff1 + value: 39.56224231830938 + - type: nauc_ndcg_at_100_max + value: 23.886442622042228 + - type: nauc_ndcg_at_100_std + value: -10.534435913119328 + - type: nauc_ndcg_at_10_diff1 + value: 38.438066883020696 + - type: nauc_ndcg_at_10_max + value: 23.871323754524052 + - type: nauc_ndcg_at_10_std + value: -12.020382617526508 + - type: nauc_ndcg_at_1_diff1 + value: 49.24897980869228 + - type: nauc_ndcg_at_1_max + value: 21.476485407508953 + - type: nauc_ndcg_at_1_std + value: -14.362366603555893 + - type: nauc_ndcg_at_20_diff1 + value: 39.044984412751454 + - type: nauc_ndcg_at_20_max + value: 24.627000563476837 + - type: nauc_ndcg_at_20_std + value: -10.591144903278847 + - type: nauc_ndcg_at_3_diff1 + value: 38.59706175845992 + - type: nauc_ndcg_at_3_max + value: 20.660682570744303 + - type: nauc_ndcg_at_3_std + value: -14.989842570951817 + - type: nauc_ndcg_at_5_diff1 + value: 37.99262057687399 + - type: nauc_ndcg_at_5_max + value: 21.323357059546723 + - type: nauc_ndcg_at_5_std + value: -13.985440057708725 + - type: nauc_precision_at_1000_diff1 + value: -7.055913327014856 + - type: nauc_precision_at_1000_max + value: 7.645796200601874 + - type: nauc_precision_at_1000_std + value: 5.7441793819913425 + - type: nauc_precision_at_100_diff1 + value: -5.9521730506691695 + - type: nauc_precision_at_100_max + value: 10.791705567977127 + - type: nauc_precision_at_100_std + value: 6.971621413042414 + - type: nauc_precision_at_10_diff1 + value: 1.2124462289393534 + - type: nauc_precision_at_10_max + value: 17.288772868206426 + - type: nauc_precision_at_10_std + value: 1.7066485280331372 + - type: nauc_precision_at_1_diff1 + value: 49.24897980869228 + - type: nauc_precision_at_1_max + value: 21.476485407508953 + - type: nauc_precision_at_1_std + value: -14.362366603555893 + - type: nauc_precision_at_20_diff1 + value: -1.9829512286324857 + - type: nauc_precision_at_20_max + value: 15.73212540041351 + - type: nauc_precision_at_20_std + value: 5.881638156347814 + - type: nauc_precision_at_3_diff1 + value: 15.295884096855977 + - type: nauc_precision_at_3_max + value: 20.315378698666457 + - type: nauc_precision_at_3_std + value: -9.87038943277974 + - type: nauc_precision_at_5_diff1 + value: 7.500406876014883 + - type: nauc_precision_at_5_max + value: 17.81994110188596 + - type: nauc_precision_at_5_std + value: -4.56852626899763 + - type: nauc_recall_at_1000_diff1 + value: 24.05686791288483 + - type: nauc_recall_at_1000_max + value: 60.34558482395768 + - type: nauc_recall_at_1000_std + value: 75.24495489488791 + - type: nauc_recall_at_100_diff1 + value: 14.41757817770598 + - type: nauc_recall_at_100_max + value: 43.46467354926512 + - type: nauc_recall_at_100_std + value: 33.594674489620495 + - type: nauc_recall_at_10_diff1 + value: 22.768211257001038 + - type: nauc_recall_at_10_max + value: 29.59352351340424 + - type: nauc_recall_at_10_std + value: -5.643974726375464 + - type: nauc_recall_at_1_diff1 + value: 42.42534428928902 + - type: nauc_recall_at_1_max + value: 12.831550041111084 + - type: nauc_recall_at_1_std + value: -11.266929686927616 + - type: nauc_recall_at_20_diff1 + value: 22.091666169938616 + - type: nauc_recall_at_20_max + value: 38.1442003173532 + - type: nauc_recall_at_20_std + value: 6.8939344221023795 + - type: nauc_recall_at_3_diff1 + value: 30.141208429704356 + - type: nauc_recall_at_3_max + value: 19.02721767063729 + - type: nauc_recall_at_3_std + value: -15.298243677583443 + - type: nauc_recall_at_5_diff1 + value: 25.13257041212021 + - type: nauc_recall_at_5_max + value: 19.760375951283518 + - type: nauc_recall_at_5_std + value: -12.907639506550343 + - type: ndcg_at_1 + value: 55.37800000000001 + - type: ndcg_at_10 + value: 67.97800000000001 + - type: ndcg_at_100 + value: 71.186 + - type: ndcg_at_1000 + value: 71.666 + - type: ndcg_at_20 + value: 69.70100000000001 + - type: ndcg_at_3 + value: 60.045 + - type: ndcg_at_5 + value: 64.206 + - type: precision_at_1 + value: 55.37800000000001 + - type: precision_at_10 + value: 13.694 + - type: precision_at_100 + value: 1.6 + - type: precision_at_1000 + value: 0.166 + - type: precision_at_20 + value: 7.394 + - type: precision_at_3 + value: 33.353 + - type: precision_at_5 + value: 23.629 + - type: recall_at_1 + value: 38.682 + - type: recall_at_10 + value: 83.624 + - type: recall_at_100 + value: 95.964 + - type: recall_at_1000 + value: 99.072 + - type: recall_at_20 + value: 89.334 + - type: recall_at_3 + value: 64.039 + - type: recall_at_5 + value: 73.92999999999999 + task: + type: Retrieval + - dataset: + config: default + name: MTEB RuReviewsClassification + revision: f6d2c31f4dc6b88f468552750bfec05b4b41b05a + split: test + type: ai-forever/ru-reviews-classification + metrics: + - type: accuracy + value: 67.6953125 + - type: f1 + value: 65.90453756121588 + - type: f1_weighted + value: 65.90276417438837 + - type: main_score + value: 67.6953125 + task: + type: Classification + - dataset: + config: default + name: MTEB RuSTSBenchmarkSTS + revision: 7cf24f325c6da6195df55bef3d86b5e0616f3018 + split: test + type: ai-forever/ru-stsbenchmark-sts + metrics: + - type: cosine_pearson + value: 76.29471433169448 + - type: cosine_spearman + value: 76.87365514666403 + - type: euclidean_pearson + value: 75.61167023011886 + - type: euclidean_spearman + value: 76.87303787773313 + - type: main_score + value: 76.87365514666403 + - type: manhattan_pearson + value: 75.66392608968061 + - type: manhattan_spearman + value: 76.92734684196252 + - type: pearson + value: 76.29471433169448 + - type: spearman + value: 76.87365514666403 + task: + type: STS + - dataset: + config: default + name: MTEB RuSciBenchGRNTIClassification + revision: 673a610d6d3dd91a547a0d57ae1b56f37ebbf6a1 + split: test + type: ai-forever/ru-scibench-grnti-classification + metrics: + - type: accuracy + value: 62.76367187500001 + - type: f1 + value: 61.038547511485696 + - type: f1_weighted + value: 61.05100853971954 + - type: main_score + value: 62.76367187500001 + task: + type: Classification + - dataset: + config: default + name: MTEB RuSciBenchGRNTIClusteringP2P + revision: 673a610d6d3dd91a547a0d57ae1b56f37ebbf6a1 + split: test + type: ai-forever/ru-scibench-grnti-classification + metrics: + - type: main_score + value: 58.12317505878799 + - type: v_measure + value: 58.12317505878799 + - type: v_measure_std + value: 0.8074216341773242 + task: + type: Clustering + - dataset: + config: default + name: MTEB RuSciBenchOECDClassification + revision: 26c88e99dcaba32bb45d0e1bfc21902337f6d471 + split: test + type: ai-forever/ru-scibench-oecd-classification + metrics: + - type: accuracy + value: 49.765625 + - type: f1 + value: 47.21576484201538 + - type: f1_weighted + value: 47.216631424004845 + - type: main_score + value: 49.765625 + task: + type: Classification + - dataset: + config: default + name: MTEB RuSciBenchOECDClusteringP2P + revision: 26c88e99dcaba32bb45d0e1bfc21902337f6d471 + split: test + type: ai-forever/ru-scibench-oecd-classification + metrics: + - type: main_score + value: 48.94279918311604 + - type: v_measure + value: 48.94279918311604 + - type: v_measure_std + value: 0.6861637451888378 + task: + type: Clustering + - dataset: + config: ru + name: MTEB STS22 (ru) + revision: de9d86b3b84231dc21f76c7b7af1f28e2f57f6e3 + split: test + type: mteb/sts22-crosslingual-sts + metrics: + - type: cosine_pearson + value: 64.72567885537859 + - type: cosine_spearman + value: 67.02766486173321 + - type: euclidean_pearson + value: 65.20776364711082 + - type: euclidean_spearman + value: 67.02766486173321 + - type: main_score + value: 67.02766486173321 + - type: manhattan_pearson + value: 65.01652728805082 + - type: manhattan_spearman + value: 67.01526065421112 + - type: pearson + value: 64.72567885537859 + - type: spearman + value: 67.02766486173321 + task: + type: STS + - dataset: + config: default + name: MTEB SensitiveTopicsClassification + revision: 416b34a802308eac30e4192afc0ff99bb8dcc7f2 + split: test + type: ai-forever/sensitive-topics-classification + metrics: + - type: accuracy + value: 31.5966796875 + - type: f1 + value: 37.51973703475147 + - type: lrap + value: 47.855767144096504 + - type: main_score + value: 31.5966796875 + task: + type: MultilabelClassification + - dataset: + config: default + name: MTEB TERRa + revision: 7b58f24536063837d644aab9a023c62199b2a612 + split: dev + type: ai-forever/terra-pairclassification + metrics: + - type: cosine_accuracy + value: 57.65472312703584 + - type: cosine_accuracy_threshold + value: 83.7156891822815 + - type: cosine_ap + value: 55.055463204979915 + - type: cosine_f1 + value: 67.26057906458797 + - type: cosine_f1_threshold + value: 76.06719732284546 + - type: cosine_precision + value: 51.01351351351351 + - type: cosine_recall + value: 98.69281045751634 + - type: dot_accuracy + value: 57.65472312703584 + - type: dot_accuracy_threshold + value: 83.71570110321045 + - type: dot_ap + value: 55.055463204979915 + - type: dot_f1 + value: 67.26057906458797 + - type: dot_f1_threshold + value: 76.06720328330994 + - type: dot_precision + value: 51.01351351351351 + - type: dot_recall + value: 98.69281045751634 + - type: euclidean_accuracy + value: 57.65472312703584 + - type: euclidean_accuracy_threshold + value: 57.068902254104614 + - type: euclidean_ap + value: 55.055463204979915 + - type: euclidean_f1 + value: 67.26057906458797 + - type: euclidean_f1_threshold + value: 69.18495893478394 + - type: euclidean_precision + value: 51.01351351351351 + - type: euclidean_recall + value: 98.69281045751634 + - type: main_score + value: 55.17437273876084 + - type: manhattan_accuracy + value: 58.306188925081436 + - type: manhattan_accuracy_threshold + value: 1299.5019912719727 + - type: manhattan_ap + value: 55.17437273876084 + - type: manhattan_f1 + value: 67.25663716814158 + - type: manhattan_f1_threshold + value: 1602.2712707519531 + - type: manhattan_precision + value: 50.836120401337794 + - type: manhattan_recall + value: 99.34640522875817 + - type: max_ap + value: 55.17437273876084 + - type: max_f1 + value: 67.26057906458797 + - type: max_precision + value: 51.01351351351351 + - type: max_recall + value: 99.34640522875817 + - type: similarity_accuracy + value: 57.65472312703584 + - type: similarity_accuracy_threshold + value: 83.7156891822815 + - type: similarity_ap + value: 55.055463204979915 + - type: similarity_f1 + value: 67.26057906458797 + - type: similarity_f1_threshold + value: 76.06719732284546 + - type: similarity_precision + value: 51.01351351351351 + - type: similarity_recall + value: 98.69281045751634 + task: + type: PairClassification +pipeline_tag: sentence-similarity +tags: +- sentence-transformers +- feature-extraction +- sentence-similarity +- mteb +--- +- + +

KaLM-Embedding

+ + +**KaLM-Embedding** is a series of embedding models adapted from auto-regressive LLMs with superior training data. + +KaLM-embedding-multilingual-mini is trained from [Qwen/Qwen2-0.5B](https://huggingface.co./Qwen/Qwen2-0.5B) with massive weakly-supervised pre-training and supervised fine-tuning data. + + +## 📑 Open-source Plan + +- [x] Model Checkpoint + - [x] [KaLM-embedding-multilingual-mini-v1](https://huggingface.co./HIT-TMG/KaLM-embedding-multilingual-mini-v1) + - [x] [KaLM-embedding-multilingual-mini-instruct-v1](https://huggingface.co./HIT-TMG/KaLM-embedding-multilingual-mini-instruct-v1) + - [x] [KaLM-embedding-multilingual-mini-instruct-v1.5](https://huggingface.co./HIT-TMG/KaLM-embedding-multilingual-mini-instruct-v1.5) + - [ ] KaLM-embedding-multilingual-max-v1 +- [x] Training and Evaluation Code: [HITsz-TMG/KaLM-Embedding](https://github.com/HITsz-TMG/KaLM-Embedding) +- [ ] Technical Report (in 2024) +- [ ] Training Data + + +## Evaluation + + | Model Name | Model Size | C-MTEB(35) | MTEB(56) | avg +|:----:|:---:|:---:|:---:|:---:| +| [multilingual-e5-large](https://huggingface.co./intfloat/multilingual-e5-large) | 560M | 58.81 | 61.5 | 60.16 +| [bge-m3 (dense)](https://huggingface.co./BAAI/bge-m3) | 560M | 60.80 | 59.84 | 60.32 +| [gte-multilingual-base (dense)](https://huggingface.co./Alibaba-NLP/gte-multilingual-base) | **305M** | 62.72 | 61.40 | 62.06 +| [KaLM-embedding-multilingual-mini-v1](https://huggingface.co./HIT-TMG/KaLM-embedding-multilingual-mini-v1) | 494M | 62.31 | 61.87 | 62.09 +| [KaLM-embedding-multilingual-mini-instruct-v1](https://huggingface.co./HIT-TMG/KaLM-embedding-multilingual-mini-instruct-v1) | 494M | 63.57 | 64.74 | 64.16 +| [KaLM-embedding-multilingual-mini-instruct-v1.5](https://huggingface.co./HIT-TMG/KaLM-embedding-multilingual-mini-instruct-v1.5) | 494M | **64.13** | **64.94** | **64.53** + + + +## Requirements +Since we have used the Qwen2 model, we advise you to install `transformers>=4.37.0`, or you might encounter the following error: +``` +KeyError: 'qwen2' +``` + + +## Usage + +Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed: + +``` +pip install -U sentence-transformers +``` + +Then you can use the model like this: + +```python +from sentence_transformers import SentenceTransformer + + +sentences = ["This is an example sentence", "Each sentence is converted"] + +model = SentenceTransformer('{MODEL_NAME_OR_PATH}') # Do NOT set trust_remote_code +model.max_seq_length = 512 + +embeddings = model.encode( + sentences, + normalize_embeddings=True, + batch_size=256, + show_progress_bar=True + ) +print(embeddings) +``` + +We add instruction for asymmetric tasks: retrieval, reranking, classification and clustering. + +If you want to add instruction to the query (no instruction for the corpus), you can use the model like this: + +```python +from sentence_transformers import SentenceTransformer + + +sentences = ["This is an example sentence", "Each sentence is converted"] + +model = SentenceTransformer('{MODEL_NAME_OR_PATH}') # Do NOT set trust_remote_code +model.max_seq_length = 512 + +prompt = "Instruct: Classifying the category of french news. \n Query: " +embeddings = model.encode( + sentences, + prompt=prompt, + normalize_embeddings=True, + batch_size=256, + show_progress_bar=True + ) +print(embeddings) +``` + + +## Contact +If you encounter any issue, feel free to contact us via the email: yanshek.woo@gmail.com