---
language:
- en
- fr
- es
- pt
tags:
- falcon3
license: other
license_name: falcon-llm-license
license_link: https://falconllm.tii.ae/falcon-terms-and-conditions.html
library_name: transformers
---
# Falcon3-3B-Base
**Falcon3** family of Open Foundation Models is a set of pretrained and instruct LLMs ranging from 1B to 10B parameters.
This repository contains the **Falcon3-3B-Base**. It achieves strong results on reasoning, language understanding, instruction following, code and mathematics tasks.
Falcon3-3B-Base supports 4 languages (English, French, Spanish, Portuguese) and a context length of up to 8K.
It was pruned in terms of depth and width from Falcon3-7B-Base and was efficiently trained on only 100 GT using a knowledge distillation objective.
⚠️ **This is a raw, pretrained model, which should be further finetuned using SFT, RLHF, continued pretraining, etc. for most use cases.**
## Model Details
- Architecture
- Transformer-based causal decoder-only architecture
- 22 decoder blocks
- Grouped Query Attention (GQA) for faster inference: 12 query heads and 4 key-value heads
- Wider head dimension: 256
- High RoPE value to support long context understanding: 1000042
- Uses SwiGLU and RMSNorm
- 8K context length
- 131K vocab size
- Pruned and healed from Falcon3-7B-Base on only 100 Gigatokens of datasets comprising of web, code, STEM, high quality and mutlilingual data using 1024 H100 GPU chips
- Supports EN, FR, ES, PT
- Developed by [Technology Innovation Institute](https://www.tii.ae)
- License: TII Falcon-LLM License 2.0
- Model Release Date: December 2024
## Getting started
Click to expand
```python
import torch
from transformers import pipeline
pipe = pipeline(
"text-generation",
model="tiiuae/Falcon3-3B-Base",
torch_dtype=torch.bfloat16,
device_map="auto"
)
response = pipe("Question: How many hours in one day? Answer: ")
print(response[0]['generated_text'])
```
## Benchmarks
We report in the following table our internal pipeline benchmarks.
- We use [lm-evaluation harness](https://github.com/EleutherAI/lm-evaluation-harness).
- We report **raw scores**.
- We use same batch-size across all models.
Category |
Benchmark |
Llama3.2-3B |
Qwen2.5-3B |
Minitron-4B |
Falcon3-3B-Base |
General |
MMLU (5-shot) |
56.1 |
65.6 |
58.7 |
55.5 |
MMLU-PRO (5-shot) |
24.9 |
32 |
26.2 |
28.8 |
IFEval |
12.8 |
27 |
22.8 |
27.7 |
Math |
GSM8K (5-shot) |
26.7 |
69 |
25.7 |
63.9 |
MATH Lvl-5 (4-shot) |
1.4 |
8.4 |
1.7 |
9.4 |
Reasoning |
Arc Challenge (25-shot) |
50.8 |
55.5 |
50.3 |
54.9 |
GPQA (0-shot) |
27.5 |
27.5 |
28.6 |
31.2 |
MUSR (0-shot) |
35.2 |
43 |
42.1 |
37.5 |
BBH (3-shot) |
38.6 |
46.1 |
40.9 |
44.2 |
CommonSense Understanding |
PIQA (0-shot) |
77.4 |
78.9 |
78.3 |
75.6 |
SciQ (0-shot) |
92.7 |
95.6 |
96.1 |
93.1 |
Winogrande (0-shot) |
69.7 |
68.8 |
68.4 |
64.6 |
OpenbookQA (0-shot) |
43.2 |
42.2 |
43 |
39.4 |
## Useful links
- View our [release blogpost](https://huggingface.co./blog/falcon3).
- Feel free to join [our discord server](https://discord.gg/fwXpMyGc) if you have any questions or to interact with our researchers and developers.
## Technical Report
Coming soon....
## Citation
If the Falcon3 family of models were helpful to your work, feel free to give us a cite.
```
@misc{Falcon3,
title = {The Falcon 3 Family of Open Models},
url = {https://huggingface.co./blog/falcon3},
author = {Falcon-LLM Team},
month = {December},
year = {2024}
}
```