nieshen/SMDM

Pretrained models for the paper Scaling up Masked Diffusion Models on Text

Scaling law experiments: We provided all pre-trained models in the ar_safetensors and mdm_safetensors folders. For instance, the checkpoint mdm-1028M-1600e18.safetensors represents an MDM model with 1,028 million non-embedding parameters and 1,600e18 training FLOPs. Similarly, the checkpoint mdm-170M-100e18-rsl-0.01.safetensors indicates an MDM model with 170 million non-embedding parameters, 100e18 training FLOPs, and 1% of the dataset subjected to random sequence lengths during pretraining.

Math reasoning: please see the gsm8k_safetensors folder.

Conditional generation: please see the sharegpt_safetensors folder.

Reverse curse: please see the reverse_safetensors folder

For all models, we provide models in .pth and .safetensors formats.