Hugging Face's logo
Garreth Lee's picture
13 6 13

Garreth Lee PRO

garrethlee
aswinsson's profile picture Pisello's profile picture reach-vb's profile picture
·

AI & ML interests

None yet

Recent Activity

upvoted a paper 5 days ago
Qwen2.5 Technical Report
updated a Space 11 days ago
huggingface/number-tokenization-blog
liked a dataset 15 days ago
HuggingFaceFW/fineweb-2
View all activity

Organizations

Hugging Face's profile picture Hugging Face TB Research's profile picture HuggingFaceFW's profile picture HuggingFaceFW-Dev's profile picture Hugging Face Science's profile picture AI Starter Pack's profile picture

Posts 2

view post
Post
1886
The latest o1 model from OpenAI is still unable to answer 9.11 > 9.9 correctly 🤔

A possible explanation? Tokenization - and our latest work investigates how it affects a model's ability to do math!

In this blog post, we discuss:
🔢 The different ways numbers are tokenized in modern LLMs
🧪 Our detailed approach in comparing these various methods
🥪 How we got a free boost in arithmetic performance by adding a few lines of code to the base Llama 3 tokenizer
👑 and a definitive, best tokenization method for math in LLMs!

Check out our work here: huggingface/number-tokenization-blog
view post
Post
360
Does tokenizing numbers into single digits outperform three-digit or BPE tokenization for arithmetic tasks? We explore various tokenization methods in our upcoming blog (releasing next week 👀)!

🔹 Bringing objectivity to comparisons

Existing comparisons of number tokenization methods often ignore the difference in models’ compute budgets: larger tokenizer vocabularies naturally lead to more parameters, which produces less objective comparisons of model performances due to more “learning” being done by these bigger models.

We addressed this by keeping architectures consistent but adjusting the number of hidden layers to produce roughly equal parameter counts.

🔹 Key results

We trained models on the same data mix and evaluated their performance on various arithmetic tasks (digits, operations, floats vs. ints):

- When splitting evals based on operators, single-digit tokenization consistently outperformed other methods.
- Right-to-left tokenization (which I covered in a previous post) matched or exceeded left-to-right approaches in all tasks.

All in all, single-digit tokenization is best compared to other methods, and similar to our previous post’s finding, R2L works better than L2R tokenization, although not as significant as the gap between single-digit and the rest!

The wait is almost over 🤗, the full report is coming next week - stay tuned!

models

None public yet