moxin-chat-7b / README.md
piuzha's picture
Update README.md
64e4874 verified
metadata
license: apache-2.0

Moxin Chat 7B

Home Page    |    Technical Report    |    Base Model    |    Chat Model

Model

You can download our base 7B model from this link and our chat 7B model from this link.

Inference

You can use the following code to run inference with the model. The model is saved under './model/' directory. Change the model directory accordingly or use the Huggingface link.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline

torch.backends.cuda.enable_mem_efficient_sdp(False)
torch.backends.cuda.enable_flash_sdp(False)

model_name = 'moxin-org/moxin-chat-7b'
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
        model_name,
        torch_dtype=torch.bfloat16,
        device_map="auto",
        trust_remote_code=True,
    )

pipe = pipeline(
    "text-generation",
    model=model,
    tokenizer = tokenizer,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

prompt = "Can you explain the concept of regularization in machine learning?"

sequences = pipe(
    prompt,
    do_sample=True,
    max_new_tokens=1000,
    temperature=0.7,
    top_k=50,
    top_p=0.95,
    num_return_sequences=1,
)
print(sequences[0]['generated_text'])

Chat template

The chat template is available via the apply_chat_template() method:

from transformers import AutoModelForCausalLM, AutoTokenizer

device = "cuda" 

model = AutoModelForCausalLM.from_pretrained("moxin-org/moxin-chat-7b")
tokenizer = AutoTokenizer.from_pretrained("moxin-org/moxin-chat-7b")

messages = [
    {"role": "user", "content": "What is your favourite condiment?"},
    {"role": "assistant", "content": "Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!"},
    {"role": "user", "content": "Do you have mayonnaise recipes?"}
]

encodeds = tokenizer.apply_chat_template(messages, return_tensors="pt")

model_inputs = encodeds.to(device)
model.to(device)

generated_ids = model.generate(model_inputs, max_new_tokens=1000, do_sample=True)
decoded = tokenizer.batch_decode(generated_ids)
print(decoded[0])

Evaluation

We test the performance of our model with lm-evaluation-harness. The evaluation results on common datasets are shown below. We test on AI2 Reasoning Challenge (25-shot), HellaSwag (10-shot), MMLU (5-shot), and Winogrande (5-shot). We release the Moxin-7B-finetuned as our base model. We further finetune our base model on Tulu v2 to obtain our chat model.

Models ARC-C Hellaswag MMLU WinoGrade Ave
Mistral-7B 57.59 83.25 62.42 78.77 70.51
LLaMA 3.1-8B 54.61 81.95 65.16 77.35 69.77
LLaMA 3-8B 55.46 82.09 65.29 77.82 70.17
LLaMA 2-7B 49.74 78.94 45.89 74.27 62.21
Qwen 2-7B 57.68 80.76 70.42 77.43 71.57
gemma-7b 56.48 82.31 63.02 78.3 70.03
internlm2.5-7b 54.78 79.7 68.17 80.9 70.89
Baichuan2-7B 47.87 73.89 54.13 70.8 61.67
Yi-1.5-9B 58.36 80.36 69.54 77.53 71.48
Moxin-7B-original 53.75 75.46 59.43 70.32 64.74
Moxin-7B-finetuned 59.47 83.08 60.97 78.69 70.55

We also test the zero shot performance on AI2 Reasoning Challenge (0-shot), AI2 Reasoning Easy (0-shot), HellaSwag (0-shot), PIQA (0-shot) and Winogrande (0-shot). The results are shown below.

Models HellaSwag WinoGrade PIQA ARC-E ARC-C Ave
Mistral-7B 80.39 73.4 82.15 78.28 52.22 73.29
LLaMA 2-7B 75.99 69.06 79.11 74.54 46.42 69.02
LLaMA 2-13B 79.37 72.22 80.52 77.4 49.06 71.71
LLaMA 3.1-8B 78.92 74.19 81.12 81.06 53.67 73.79
gemma-7b 80.45 73.72 80.9 79.97 54.1 73.83
Qwen v2-7B 78.9 72.38 79.98 74.71 50.09 71.21
internlm2.5-7b 79.14 77.9 80.52 76.16 51.37 73.02
Baichuan2-7B 72.25 67.17 77.26 72.98 42.15 66.36
Yi-1.5-9B 77.86 73.01 80.74 79.04 55.03 73.14
deepseek-7b 76.13 69.77 79.76 71.04 44.8 68.3
Moxin-7B-original 72.06 66.31 78.07 71.47 48.15 67.21
Moxin-7B-finetune 80.03 75.17 82.24 81.12 58.64 75.44

Citation

@article{zhao2024fully,
  title={Fully Open Source Moxin-7B Technical Report},
  author={Zhao, Pu and Shen, Xuan and Kong, Zhenglun and Shen, Yixin and Chang, Sung-En and Rupprecht, Timothy and Lu, Lei and Nan, Enfu and Yang, Changdi and He, Yumei and others},
  journal={arXiv preprint arXiv:2412.06845},
  year={2024}
}