Update README.md
Browse files
README.md
CHANGED
@@ -202,15 +202,22 @@ More information regarding the pre-training are available in our model preprint
|
|
202 |
|
203 |
<!-- This section describes the evaluation protocols and provides the results. -->
|
204 |
|
205 |
-
|
206 |
-
|
207 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
208 |
|
209 |
-
|
210 |
-
|
211 |
-
<!-- This should link to a Dataset Card if possible. -->
|
212 |
|
213 |
-
The model was evaluated in 21 languages on ARC, GSM8K, HellaSwag, TruthfulQA, Translation and MMLU. Results can be seen in the [European LLM Leaderboard](https://huggingface.co/spaces/openGPT-X/european-llm-leaderboard).
|
214 |
|
215 |
## Technical Specifications
|
216 |
|
|
|
202 |
|
203 |
<!-- This section describes the evaluation protocols and provides the results. -->
|
204 |
|
205 |
+
Results on multilingual benchmarks for 21 European languages with instruction-tuned models
|
206 |
+
| Model | Avg. | EU21-ARC | EU21-HeSw | EU21-TQA | EU21-MMLU |
|
207 |
+
|--------------------------------|--------|----------|-----------|----------|-----------|
|
208 |
+
| Meta-Llama-3.1-8B-Instruct | **.563** | .563 | .579 | .532 | **.576** |
|
209 |
+
| Mistral-7B-Instruct-v0.3 | .527 | .530 | .538 | **.548** | _ .491 _ |
|
210 |
+
| Salamandra-7B-Instruct | _ .543 _ | **.595** | **.637** | .482 | .459 |
|
211 |
+
| Aya-23-8B | .485 | .475 | .535 | .476 | .455 |
|
212 |
+
| Occiglot-7B-eu5-Instruct | .475 | .484 | .519 | .471 | .428 |
|
213 |
+
| Pharia-1-LLM-7B-C-A | .417 | .396 | .438 | .469 | .366 |
|
214 |
+
| Bloomz-7B1 | .358 | .316 | .354 | .461 | .302 |
|
215 |
+
| **Ours (Base)** | .496 | .550 | .615 | .469 | .349 |
|
216 |
+
| **Ours (Instruct)** | _ .543 _ | _ .581 _ | _ .624 _ | _ .543 _ | .425 |
|
217 |
|
218 |
+
More information regarding our translated benchmarks are available in our preprint ["Towards Multilingual LLM Evaluation for European Languages"](https://arxiv.org/abs/2410.08928).
|
|
|
|
|
219 |
|
220 |
+
The model was evaluated in 21 languages on ARC, GSM8K, HellaSwag, TruthfulQA, Translation and MMLU. Results can also be seen in the [European LLM Leaderboard](https://huggingface.co/spaces/openGPT-X/european-llm-leaderboard).
|
221 |
|
222 |
## Technical Specifications
|
223 |
|