neuralmagic/OpenHermes-2.5-Mistral-7B-marlin

openhermes-2.5-mistral-7b

This repo contains model files for OpenHermes-2.5-Mistral-7b optimized for nm-vllm, a high-throughput serving engine for compressed LLMs.

This model was quantized with GPTQ and saved in the Marlin format for efficient 4-bit inference. Marlin is a highly optimized inference kernel for 4 bit models.

Inference

Install nm-vllm for fast inference and low memory-usage:

pip install nm-vllm[sparse]

Run in a Python pipeline for local inference:

from transformers import AutoTokenizer
from vllm import LLM, SamplingParams

model_id = "neuralmagic/OpenHermes-2.5-Mistral-7B-marlin"
model = LLM(model_id)

tokenizer = AutoTokenizer.from_pretrained(model_id)
messages = [
    {"role": "user", "content": "What is synthetic data in machine learning?"},
]
formatted_prompt =  tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
sampling_params = SamplingParams(max_tokens=200)
outputs = model.generate(formatted_prompt, sampling_params=sampling_params)
print(outputs[0].outputs[0].text)

"""
Synthetic data is data that has been artificially created or modified to serve the needs of machine learning and data analysis tasks. It can be generated either through title methods like stochastic simulations or through processes of data augmentation that take original data and modify/manipulate it to create new samples. Synthetic data is often used in machine learning when the available amount of real-world data is insufficient or in cases where the creation of real-world data can be dangerous, costly, or time-consuming.
"""

Quantization

For details on how this model was quantized and converted to marlin format, run the quantization/apply_gptq_save_marlin.py script:

pip install -r quantization/requirements.txt
python3 quantization/apply_gptq_save_marlin.py --model-id teknium/OpenHermes-2.5-Mistral-7B --save-dir ./openhermes-marlin

Slack

For further support, and discussions on these models and AI in general, join Neural Magic's Slack Community