File size: 1,845 Bytes
6285e4f |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 |
---
license: apache-2.0
language:
- en
inference: false
---
# Model Card for TinyMixtral-x8-Clonebase-7b
This model is based on [TinyLlama-1.1B](https://huggingface.co./TinyLlama/TinyLlama-1.1B-intermediate-step-1195k-token-2.5T), converted to a mistral model, and then placed the clone in mixtral.
**This model was created experimentally for training a small mixtral.**
# How it was made
First, since tinyllama is an llama model, I converted it to a mistral model.
After that, I cloned the FFN part and made it experts.
Since they are all the same tensor, the performance does not change.
All gates have the same value.
# How To Convert
use colab cpu-high-memory.
This model was created with experts=8, but since it is a clone, you can create as many experts as you like.
[tinyllama_to_mixtral_clonebase.ipynb](https://huggingface.co./mmnga/TinyMixtral-x8-Clonebase-7b)
# Usage
~~~python
pip install transformers --upgrade
pip install flash_attn
~~~
~~~python
from transformers import AutoTokenizer, AutoModelForCausalLM, MixtralForCausalLM
import torch
model_name_or_path = "mmnga/TinyMixtral-x8-Clonebase-7b"
tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)
model = MixtralForCausalLM.from_pretrained(model_name_or_path, device_map="auto")
# set num_experts_per_tok 1 or 2 ?
model.config.num_experts_per_tok = 2
# message
messages = [
{"role": "user", "content": "Tell me what's for dinner tonight."},
]
with torch.no_grad():
token_ids = tokenizer.apply_chat_template(messages, return_tensors="pt")
output_ids = model.generate(
token_ids.to(model.device),
temperature=0.5,
do_sample=True,
top_p=0.95,
top_k=40,
max_new_tokens=128,
repetition_penalty=1.5
)
output = tokenizer.decode(output_ids[0][token_ids.size(1) :])
print(output)
~~~ |