Hugging Face's logo

tensorblock
/
TinyMistral-6x248M-Instruct-GGUF

GGUF
English
Mixture of Experts
TensorBlock
GGUF
Inference Endpoints
Model card Files Community
TinyMistral-6x248M-Instruct-GGUF / README.md
morriszms's picture
morriszms
Upload folder using huggingface_hub
029c5fb verified
preview code
|
raw
history blame contribute delete
5.78 kB
metadata
license: apache-2.0
datasets:
  - Locutusque/hercules-v1.0
language:
  - en
base_model: M4-ai/TinyMistral-6x248M-Instruct
inference:
  parameters:
    do_sample: true
    temperature: 0.2
    top_p: 0.14
    top_k: 12
    max_new_tokens: 250
    repetition_penalty: 1.1
widget:
  - text: |
      <|im_start|>user
      Write me a Python program that calculates the factorial of n. <|im_end|>
      <|im_start|>assistant
  - text: >-
      An emerging clinical approach to treat substance abuse disorders involves
      a form of cognitive-behavioral therapy whereby addicts learn to reduce
      their reactivity to drug-paired stimuli through cue-exposure or extinction
      training. It is, however,
  - text: |
      <|im_start|>user
      How do I say hello in Spanish? <|im_end|>
      <|im_start|>assistant
tags:
  - moe
  - TensorBlock
  - GGUF
TensorBlock

Feedback and support: TensorBlock's Twitter/X, Telegram Group and Discord server

M4-ai/TinyMistral-6x248M-Instruct - GGUF

This repo contains GGUF format model files for M4-ai/TinyMistral-6x248M-Instruct.

The files were quantized using machines provided by TensorBlock, and they are compatible with llama.cpp as of commit b4242.

Prompt template

Model file specification

Filename Quant type File Size Description
TinyMistral-6x248M-Instruct-Q2_K.gguf Q2_K 0.379 GB smallest, significant quality loss - not recommended for most purposes
TinyMistral-6x248M-Instruct-Q3_K_S.gguf Q3_K_S 0.445 GB very small, high quality loss
TinyMistral-6x248M-Instruct-Q3_K_M.gguf Q3_K_M 0.487 GB very small, high quality loss
TinyMistral-6x248M-Instruct-Q3_K_L.gguf Q3_K_L 0.527 GB small, substantial quality loss
TinyMistral-6x248M-Instruct-Q4_0.gguf Q4_0 0.574 GB legacy; small, very high quality loss - prefer using Q3_K_M
TinyMistral-6x248M-Instruct-Q4_K_S.gguf Q4_K_S 0.577 GB small, greater quality loss
TinyMistral-6x248M-Instruct-Q4_K_M.gguf Q4_K_M 0.613 GB medium, balanced quality - recommended
TinyMistral-6x248M-Instruct-Q5_0.gguf Q5_0 0.695 GB legacy; medium, balanced quality - prefer using Q4_K_M
TinyMistral-6x248M-Instruct-Q5_K_S.gguf Q5_K_S 0.695 GB large, low quality loss - recommended
TinyMistral-6x248M-Instruct-Q5_K_M.gguf Q5_K_M 0.715 GB large, very low quality loss - recommended
TinyMistral-6x248M-Instruct-Q6_K.gguf Q6_K 0.824 GB very large, extremely low quality loss
TinyMistral-6x248M-Instruct-Q8_0.gguf Q8_0 1.067 GB very large, extremely low quality loss - not recommended

Downloading instruction

Command line

Firstly, install Huggingface Client

pip install -U "huggingface_hub[cli]"

Then, downoad the individual model file the a local directory

huggingface-cli download tensorblock/TinyMistral-6x248M-Instruct-GGUF --include "TinyMistral-6x248M-Instruct-Q2_K.gguf" --local-dir MY_LOCAL_DIR

If you wanna download multiple model files with a pattern (e.g., *Q4_K*gguf), you can try:

huggingface-cli download tensorblock/TinyMistral-6x248M-Instruct-GGUF --local-dir MY_LOCAL_DIR --local-dir-use-symlinks False --include='*Q4_K*gguf'