Text Generation
Transformers
Safetensors
llama
text-generation-inference
Inference Endpoints
mfromm commited on
Commit
f8ed8af
·
verified ·
1 Parent(s): 6836a9a

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +14 -7
README.md CHANGED
@@ -202,15 +202,22 @@ More information regarding the pre-training are available in our model preprint
202
 
203
  <!-- This section describes the evaluation protocols and provides the results. -->
204
 
205
- More information regarding our translated benchmarks are available in our preprint ["Towards Multilingual LLM Evaluation for European Languages"](https://arxiv.org/abs/2410.08928).
206
-
207
- ### Testing Data, Factors & Metrics
 
 
 
 
 
 
 
 
 
208
 
209
- #### Testing Data
210
-
211
- <!-- This should link to a Dataset Card if possible. -->
212
 
213
- The model was evaluated in 21 languages on ARC, GSM8K, HellaSwag, TruthfulQA, Translation and MMLU. Results can be seen in the [European LLM Leaderboard](https://huggingface.co/spaces/openGPT-X/european-llm-leaderboard).
214
 
215
  ## Technical Specifications
216
 
 
202
 
203
  <!-- This section describes the evaluation protocols and provides the results. -->
204
 
205
+ Results on multilingual benchmarks for 21 European languages with instruction-tuned models
206
+ | Model | Avg. | EU21-ARC | EU21-HeSw | EU21-TQA | EU21-MMLU |
207
+ |--------------------------------|--------|----------|-----------|----------|-----------|
208
+ | Meta-Llama-3.1-8B-Instruct | **.563** | .563 | .579 | .532 | **.576** |
209
+ | Mistral-7B-Instruct-v0.3 | .527 | .530 | .538 | **.548** | _ .491 _ |
210
+ | Salamandra-7B-Instruct | _ .543 _ | **.595** | **.637** | .482 | .459 |
211
+ | Aya-23-8B | .485 | .475 | .535 | .476 | .455 |
212
+ | Occiglot-7B-eu5-Instruct | .475 | .484 | .519 | .471 | .428 |
213
+ | Pharia-1-LLM-7B-C-A | .417 | .396 | .438 | .469 | .366 |
214
+ | Bloomz-7B1 | .358 | .316 | .354 | .461 | .302 |
215
+ | **Ours (Base)** | .496 | .550 | .615 | .469 | .349 |
216
+ | **Ours (Instruct)** | _ .543 _ | _ .581 _ | _ .624 _ | _ .543 _ | .425 |
217
 
218
+ More information regarding our translated benchmarks are available in our preprint ["Towards Multilingual LLM Evaluation for European Languages"](https://arxiv.org/abs/2410.08928).
 
 
219
 
220
+ The model was evaluated in 21 languages on ARC, GSM8K, HellaSwag, TruthfulQA, Translation and MMLU. Results can also be seen in the [European LLM Leaderboard](https://huggingface.co/spaces/openGPT-X/european-llm-leaderboard).
221
 
222
  ## Technical Specifications
223