what part of the code is diffusion?
Im not entirely sure which part of the modeling.py is "diffusion".
It seems to lacks the defining features of diffusion models—noise addition, denoising, time-step conditioning, or a reverse process. Instead, it processes input tokens autoregressively to predict the next token, consistent with standard language models of transformers.
Which part is exactly "diffusion" maybe you can point the lines in the code?
Thank you for your interest!
For the code related to the forward and reverse processes, please refer to https://github.com/ML-GSAI/LLaDA. To learn how the models in this repository are created, please see the https://github.com/ML-GSAI/LLaDA/blob/main/GUIDELINES.md.
Please note that our model is not a Gaussian continuous diffusion model, but a mask-based discrete diffusion model. Prior work (https://arxiv.org/abs/2406.03736) has theoretically shown that mask-based discrete diffusion models do not require time conditioning.