LLM training - a pszemraj Collection

pszemraj 's Collections

BookSum-based Summarizers

Grammar Synthesis

OCR Quality Classifiers

tFINE

LLM training

updated Oct 27

small-scale pretraining experiments of mine

BEE-spoke-data/smol_llama-101M-GQA

Text Generation • Updated Dec 25, 2023 • 2.73k • 28
BEE-spoke-data/smol_llama-220M-GQA

Text Generation • Updated Jun 28 • 2.59k • 12
BEE-spoke-data/smol_llama-220M-GQA-fineweb_edu

Text Generation • Updated Jul 18 • 55 • 1

Note smol_llama-220M-GQA CPT on fineweb-edu for 10 billion tokens
BEE-spoke-data/smol_llama-81M-tied

Text Generation • Updated Nov 20, 2023 • 1.21k • 6
BEE-spoke-data/mega-ar-126m-4k

Text Generation • Updated Jan 28 • 3.28k • 4
BEE-spoke-data/verysmol_llama-v11-KIx2

Text Generation • Updated Jan 10 • 1.21k • 4
pszemraj/pythia-31m-KI_v1-2048-scratch

Text Generation • Updated Nov 18, 2023 • 1.18k
BEE-spoke-data/bert-plus-L8-4096-v1.0

Fill-Mask • Updated Feb 14 • 21
BEE-spoke-data/mega-encoder-small-16k-v1

Fill-Mask • Updated Mar 17 • 11 • 4
BEE-spoke-data/NanoLlama-GQA-L10-A32_KV8-v13-KI

Text Generation • Updated Mar 4 • 52 • 1

Note this is a mid-training checkpoint of what is now smol_llama-220M
pszemraj/jamba-900M-v0.13-KIx2

Text Generation • Updated May 18 • 35 • 4