Hugging Face's logo

duyntnet
/
SmallThinker-3B-Preview-imatrix-GGUF

Text Generation
Transformers
GGUF
English
imatrix
SmallThinker-3B-Preview
conversational
Model card Files Community
SmallThinker-3B-Preview-imatrix-GGUF / README.md
duyntnet's picture
duyntnet
Upload README.md
d1f7f23 verified
preview code
|
raw
history blame contribute delete
2.03 kB
metadata
license: other
language:
  - en
pipeline_tag: text-generation
inference: false
tags:
  - transformers
  - gguf
  - imatrix
  - SmallThinker-3B-Preview

Quantizations of https://huggingface.co./PowerInfer/SmallThinker-3B-Preview

Inference Clients/UIs

From original readme

We introduce SmallThinker-3B-preview, a new model fine-tuned from the Qwen2.5-3b-Instruct model.

Now you can directly deploy SmallThinker On your phones with PowerServe.

Benchmark Performance

Model AIME24 AMC23 GAOKAO2024_I GAOKAO2024_II MMLU_STEM AMPS_Hard math_comp
Qwen2.5-3B-Instruct 6.67 45 50 35.8 59.8 - -
SmallThinker 16.667 57.5 64.2 57.1 68.2 70 46.8
GPT-4o 9.3 - - - 64.2 57 50

Limitation: Due to SmallThinker's current limitations in instruction following, for math_comp we adopt a more lenient evaluation method where only correct answers are required, without constraining responses to follow the specified AAAAA format.

Colab Link: Colab

Intended Use Cases

SmallThinker is designed for the following use cases:

  1. Edge Deployment: Its small size makes it ideal for deployment on resource-constrained devices.
  2. Draft Model for QwQ-32B-Preview: SmallThinker can serve as a fast and efficient draft model for the larger QwQ-32B-Preview model. From my test, in llama.cpp we can get 70% speedup (from 40 tokens/s to 70 tokens/s).