rwkv-x-dev

community

AI & ML interests

hugging face space, for rwkv-x related developments, including build assets, etc Nothing in here are considered "official releases" AKA - we treat this as a giant file dump

Recent Activity

Hazzzardous authored a paper about 1 month ago

Lina-Speech: Gated Linear Attention is a Fast and Parameter-Efficient Learner for text-to-speech synthesis

picocreator updated a dataset 5 months ago

rwkv-x-dev/lm-eval-data

picocreator authored a paper 5 months ago

GoldFinch: High Performance RWKV/Transformer Hybrid with Linear Pre-Fill and Extreme KV-Cache Compression

View all activity

rwkv-x-dev's activity

BlinkDL

posted an update 8 days ago

Post

1385

RWKV-7 "Goose" 0.4B trained w/ ctx4k automatically extrapolates to ctx32k+, and perfectly solves NIAH ctx16k 🤯 100% RNN and attention-free. Only trained on the Pile. No finetuning. Replicable training runs. tested by our community: https://github.com/Jellyfish042/LongMamba

Hazzzardous

authored a paper about 1 month ago

Lina-Speech: Gated Linear Attention is a Fast and Parameter-Efficient Learner for text-to-speech synthesis

Paper • 2410.23320 • Published Oct 30 • 7

BlinkDL

posted an update about 1 month ago

Post

3973

RWKV-6-world-v3 (+3.1T tokens) is our best multilingual 7B model as of now: BlinkDL/rwkv-6-world

It's 100% RNN and attention-free. MMLU 54.2% (previous world-v2.1 = 47.9%. note: without eval-boosting tricks such as annealing).

RWKV-7-world-v4 soon :)

BlinkDL

posted an update 3 months ago

Post

5438

RWKV-7 "Goose" preview rc2 => Peak RNN architecture?😃Will try to squeeze more performance for the final release. Preview code & model: https://github.com/BlinkDL/RWKV-LM/tree/main/RWKV-v7