Magic-Dolphin-7b
The follow-up to this model has been released, check out the updated benchmarks here for Excalibur-7b
A full suite of GGUF quantizations can be found here, courtesy of RichardErkhov
A linear merge of:
- cognitivecomputations/dolphin-2.6-mistral-7b-dpo-laser
- Locutusque/Hyperion-1.5-Mistral-7B
- ibm/merlinite-7b
These three models showed excellent acumen in technical topics so I wanted to see how they would behave together in a merge. Several different ratios were tested before this release, in the end a higher weighting for merlinite-7b helped smooth out some edges. This model is a test of how LAB tuning is impacted by merges with models leveraging DPO.
Benchmark Performance
Name | Avg. | ARC | HellaSwag | MMLU | TruthfulQA | Winogrande | GSM8K |
---|---|---|---|---|---|---|---|
Magic-Dolphin-7b | 67.48 | 65.78 | 85.61 | 64.64 | 58.01 | 79.64 | 51.18 |
dolphin-2.6-mistral-7b-dpo-laser | 67.28 | 66.3 | 85.73 | 63.16 | 61.71 | 79.16 | 47.61 |
merlinite-7b | 64 | 63.65 | 84.52 | 64.91 | 50.15 | 79.72 | 41.09 |
Hyperion-1.5-Mistral-7B | 61.43 | 60.49 | 83.64 | 63.57 | 41.78 | 78.61 | 40.49 |
This was my first experiment with merging models so any feedback is greatly appreciated.
Uses Alpaca template.
Sample Question
Merge Details
Merge Method
This model was merged using the linear merge method.
Models Merged
The following models were included in the merge:
- cognitivecomputations/dolphin-2.6-mistral-7b-dpo-laser
- Locutusque/Hyperion-1.5-Mistral-7B
- ibm/merlinite-7b
Configuration
The following YAML configuration was used to produce this model:
models:
- model: models/dolphin-2.6-mistral-7b-dpo-laser
parameters:
weight: 1.0
- model: models/Hyperion-1.5-Mistral-7B
parameters:
weight: 0.3
- model: models/merlinite-7b
parameters:
weight: 0.5
merge_method: linear
dtype: float16
Open LLM Leaderboard Evaluation Results
Detailed results can be found here
Metric | Value |
---|---|
Avg. | 67.48 |
AI2 Reasoning Challenge (25-Shot) | 65.78 |
HellaSwag (10-Shot) | 85.61 |
MMLU (5-Shot) | 64.64 |
TruthfulQA (0-shot) | 58.01 |
Winogrande (5-shot) | 79.64 |
GSM8k (5-shot) | 51.18 |
- Downloads last month
- 413
Model tree for InferenceIllusionist/Magic-Dolphin-7b
Collection including InferenceIllusionist/Magic-Dolphin-7b
Evaluation results
- normalized accuracy on AI2 Reasoning Challenge (25-Shot)test set Open LLM Leaderboard65.780
- normalized accuracy on HellaSwag (10-Shot)validation set Open LLM Leaderboard85.610
- accuracy on MMLU (5-Shot)test set Open LLM Leaderboard64.640
- mc2 on TruthfulQA (0-shot)validation set Open LLM Leaderboard58.010
- accuracy on Winogrande (5-shot)validation set Open LLM Leaderboard79.640
- accuracy on GSM8k (5-shot)test set Open LLM Leaderboard51.180