meta-llama/Llama-3.2-11B-Vision-Instruct, UQFF quantization

Run with mistral.rs. Documentation: UQFF docs.

  1. Flexible ๐ŸŒ€: Multiple quantization formats in one file format with one framework to run them all.
  2. Reliable ๐Ÿ”’: Compatibility ensured with embedded and checked semantic versioning information from day 1.
  3. Easy ๐Ÿค—: Download UQFF models easily and quickly from Hugging Face, or use a local file.
  4. Customizable ๐Ÿ› ๏ธ: Make and publish your own UQFF files in minutes.

Examples

Quantization type(s) Example
FP8 ./mistralrs-server -i vision-plain -m EricB/Llama-3.2-11B-Vision-Instruct-UQFF -a vllama --from-uqff llama3.2-vision-instruct-f8e4m3.uqff
HQQ4 ./mistralrs-server -i vision-plain -m EricB/Llama-3.2-11B-Vision-Instruct-UQFF -a vllama --from-uqff llama3.2-vision-instruct-hqq4.uqff
HQQ8 ./mistralrs-server -i vision-plain -m EricB/Llama-3.2-11B-Vision-Instruct-UQFF -a vllama --from-uqff llama3.2-vision-instruct-hqq8.uqff
Q3K ./mistralrs-server -i vision-plain -m EricB/Llama-3.2-11B-Vision-Instruct-UQFF -a vllama --from-uqff llama3.2-vision-instruct-q3k.uqff
Q4K ./mistralrs-server -i vision-plain -m EricB/Llama-3.2-11B-Vision-Instruct-UQFF -a vllama --from-uqff llama3.2-vision-instruct-q4k.uqff
Q5K ./mistralrs-server -i vision-plain -m EricB/Llama-3.2-11B-Vision-Instruct-UQFF -a vllama --from-uqff llama3.2-vision-instruct-q5k.uqff
Q8_0 ./mistralrs-server -i vision-plain -m EricB/Llama-3.2-11B-Vision-Instruct-UQFF -a vllama --from-uqff llama3.2-vision-instruct-q8_0.uqff
Downloads last month
50
Inference API
Unable to determine this model's library. Check the docs .

Model tree for EricB/Llama-3.2-11B-Vision-Instruct-UQFF

Quantized
(10)
this model

Collection including EricB/Llama-3.2-11B-Vision-Instruct-UQFF