Hugging Face's logo
gilfan baihaq's picture
1

gilfan baihaq

debuxe
·

AI & ML interests

None yet

Recent Activity

upvoted a collection about 1 month ago
Cephalo
reacted to joylarkin's post with 🔥 6 months ago
Introducing Fineweb-Edu-Fortified: An enhanced Fineweb-Edu dataset. 📚 This dataset is tailored for NLP tasks and helps streamline model training by offering a more refined, unique dataset. Perfect for startups and researchers looking for high-quality educational content to train, evaluate, or fine-tune AI models. The dataset is based on the Fineweb-Edu subset of the large Fineweb dataset and includes: - Exact-match deduplication across all crawls - Embeddings for each row using the TaylorAI/bge-micro model - Count column indicating duplication frequency - Includes data from 95 Common Crawl crawls (2013-2024) - Rows have been reduced from 1.279B to 0.324B after deduplication - It is comprised of ~375B tokens (down from 1,320B in Fineweb-Edu) Access the entire Fineweb-Edu-Fortified dataset on Hugging Face → https://huggingface.co./datasets/airtrain-ai/fineweb-edu-fortified Try a semantic search demo via this Hugging Face Space → https://huggingface.co./spaces/airtrain-ai/fineweb-edu-fortified-search-demo Many thanks to the amazing @josh-sematic for his work on this project, the Fineweb/Fineweb-Edu team at Hugging Face for producing the original datasets and for their support during our work on Fineweb-Edu-Fortified, and also thanks to @underspirit for pointing out the reduction in dataset size that could be achieved via deduplication. 🤗
updated a collection 6 months ago
justbetter
View all activity

Organizations

None yet

debuxe's activity

upvoted a collection about 1 month ago