Falcon3-3B-Base / README.md
Dhia-GB's picture
Fix typo in Minitron GPQA score.
092c29e verified
metadata
language:
  - en
  - fr
  - es
  - pt
tags:
  - falcon3
license: other
license_name: falcon-llm-license
license_link: https://falconllm.tii.ae/falcon-terms-and-conditions.html
library_name: transformers
drawing

Falcon3-3B-Base

Falcon3 family of Open Foundation Models is a set of pretrained and instruct LLMs ranging from 1B to 10B parameters.

This repository contains the Falcon3-3B-Base. It achieves strong results on reasoning, language understanding, instruction following, code and mathematics tasks. Falcon3-3B-Base supports 4 languages (English, French, Spanish, Portuguese) and a context length of up to 8K. It was pruned in terms of depth and width from Falcon3-7B-Base and was efficiently trained on only 100 GT using a knowledge distillation objective.

⚠️ This is a raw, pretrained model, which should be further finetuned using SFT, RLHF, continued pretraining, etc. for most use cases.

Model Details

  • Architecture
    • Transformer-based causal decoder-only architecture
    • 22 decoder blocks
    • Grouped Query Attention (GQA) for faster inference: 12 query heads and 4 key-value heads
    • Wider head dimension: 256
    • High RoPE value to support long context understanding: 1000042
    • Uses SwiGLU and RMSNorm
    • 8K context length
    • 131K vocab size
  • Pruned and healed from Falcon3-7B-Base on only 100 Gigatokens of datasets comprising of web, code, STEM, high quality and mutlilingual data using 1024 H100 GPU chips
  • Supports EN, FR, ES, PT
  • Developed by Technology Innovation Institute
  • License: TII Falcon-LLM License 2.0
  • Model Release Date: December 2024

Getting started

Click to expand
import torch
from transformers import pipeline

pipe = pipeline(
    "text-generation", 
    model="tiiuae/Falcon3-3B-Base", 
    torch_dtype=torch.bfloat16, 
    device_map="auto"
)
response = pipe("Question: How many hours in one day? Answer: ")
print(response[0]['generated_text'])

Benchmarks

We report in the following table our internal pipeline benchmarks.

Category Benchmark Llama3.2-3B Qwen2.5-3B Minitron-4B Falcon3-3B-Base
General MMLU (5-shot) 56.1 65.6 58.7 55.5
MMLU-PRO (5-shot) 24.9 32 26.2 28.8
IFEval 12.8 27 22.8 27.7
Math GSM8K (5-shot) 26.7 69 25.7 63.9
MATH Lvl-5 (4-shot) 1.4 8.4 1.7 9.4
Reasoning Arc Challenge (25-shot) 50.8 55.5 50.3 54.9
GPQA (0-shot) 27.5 27.5 28.6 31.2
MUSR (0-shot) 35.2 43 42.1 37.5
BBH (3-shot) 38.6 46.1 40.9 44.2
CommonSense Understanding PIQA (0-shot) 77.4 78.9 78.3 75.6
SciQ (0-shot) 92.7 95.6 96.1 93.1
Winogrande (0-shot) 69.7 68.8 68.4 64.6
OpenbookQA (0-shot) 43.2 42.2 43 39.4

Useful links

Technical Report

Coming soon....

Citation

If the Falcon3 family of models were helpful to your work, feel free to give us a cite.

@misc{Falcon3,
    title = {The Falcon 3 Family of Open Models},
    url = {https://huggingface.co./blog/falcon3},
    author = {Falcon-LLM Team},
    month = {December},
    year = {2024}
}