| |
| Alina Gorbunova | |
| Georgia Institute of Technology |
Diffusion models such as OpenAI’s DALL·E and Stability AI’s Stable Diffusion have dazzled users with their ability to output everything from silly cat images to sophisticated diagrams of mechanical parts. Pulling back the curtain on these models reveals an elegant and sophisticated statistical framework that at its core is a combination of stochastic processes, variational inference, and score-based estimation: the same language that we as researchers in operations research speak.
A diffusion model learns in two phases: a forward process that slowly adds bits of Gaussian noise over T time steps to a data sample, such as an image, and a reverse process that undoes this added noise to recover the original data sample. The goal of this forward and backward process is to learn how to convert a Gaussian distribution to a different target (data) distribution.
Going Forwards and Backwards
The mathematical foundations for diffusion models were first presented by Sohl-Dickstein et al. (2015). During the forward process a small Gaussian perturbation is added to the sample from the previous time step so that xt depends only on xt−1. Does this sound like a familiar property? If you thought "Markov Property", you are right on the money. A diffusion model’s forward process q(x1:T |x ) can be defined as a Markov chain that gradually adds Gaussian noise:
where
The term \(\{\beta_t\}_{t=1}^T\) is the diffusion rate, also called the variance scheduler. It determines how much noise is added at each time step to the previous sample to get the next sample, and it increases as \(t\) increases. There are several ways to determine how \(\beta_t\) increases from a linear to an exponential schedule, with each schedule having a different impact on the performance and precision of the model. Examples of different variance schedulers and their impacts can be found in Table 1. Guo et al. (2025) provides a more comprehensive list of schedulers.
| Schedule | \(\bar{\alpha}_t\) Behavior | Practical Effect |
|---|---|---|
| Linear | Quick drop | Wastes capacity at high \(t\) |
| Cosine | Slow, smooth decline | More informative time steps, better quality than linear |
| Learnable | Data-adaptive, optimized jointly | Best flexibility, higher training complexity |
A key analytical convenience is that at any time step \(t\), the joint distribution can be computed in closed form. This is called reparameterization and it defines \(\alpha_t = 1 - \beta_t\) and \(\bar{\alpha}_t = \prod_{s=1}^t \alpha_s\) to write
When \(t = T\), the sample is pure Gaussian noise such that \(\mathbf{x}_T = \mathcal{N}(\mathbf{0}, \mathbf{I})\). Then the reverse process occurs. During this process, a small amount of noise is removed from the sample with the goal of returning to the original data sample \(\mathbf{x}_0\) using a Markov chain defined as
where
The parameters \(\theta\) are typically a neural network.
Training the Model
To successfully train a diffusion model, the reverse Markov chain needs to match the actual reversal of the forward Markov chain. In other words, the parameters \(\theta\) need to be learned such that the joint distribution of the reverse process closely approximates the forward process. And what metric can be used to measure the similarity between two distributions? If Kullback-Leibler (KL) divergence popped into your mind, you are on the right track!
The training loss combines minimizing the KL divergence with maximizing the log-likelihood of predicting the correct data \(p_\theta(\mathbf{x}_0)\), which can also be written as minimizing the negative log-likelihood, as
The \(\log(p_\theta(\mathbf{x}_T))\) is difficult to calculate, so the variational lower bound (as known as the Evidence Lower Bound, or ELBO) is used to approximate it. Ho et al. (2020) showed that the ELBO can be written as a sum of KL divergences between each Gaussian distribution. This simplification means the loss function can be rewritten as a denoising loss function such that given noisy sample \(\mathbf{x}_t\), predict the noise \(\epsilon\) that needs to be added:
This is a mean squared error loss, a simple equation at the heart of diffusion model's training. Using this simple loss function, helps to prioritize more challenging tasks throughout the training, such as removing large amounts of noise at higher values of \(t\).
A diffusion model's entire training objective reduces to a single mean squared error loss: given a noisy sample \(\mathbf{x}_t\), predict the noise \(\epsilon\) that was added. This deceptively simple equation is the statistical engine at the heart of modern generative AI and it is one that any researcher fluent in probability and optimization is already well-equipped to understand, extend, and apply.
Sampling and Inference
Once a diffusion model is trained, it can be used to generate new samples by running the reverse process from scratch. In other words, the model starts from pure Gaussian noise \(\mathbf{x}_T \sim \mathcal{N}(\mathbf{0}, \mathbf{I})\) and iteratively denoises across \(T\) time steps using the learned reverse transition \(p_\theta(\mathbf{x}_{t-1} \mid \mathbf{x}_t)\). This is called ancestral sampling and mirrors the reverse Markov chain defined in Equations 4 and 5.
At each time step, the model predicts the noise \(\epsilon_\theta(\mathbf{x}_t, t)\) present in the current sample and uses it to estimate the mean \(\mu_\theta(\mathbf{x}_t, t)\) in order to draw the next sample \(\mathbf{x}_{t-1}\). The mean can be written as
Using this equation, the next sample \(\mathbf{x}_{t-1}\) can be written as the mean plus some noise level \(\sigma_t\) injected at time step \(t\)
where \(\mathbf{z} \sim \mathcal{N}(\mathbf{0}, \mathbf{I})\). It's this \(\mathbf{z}\) noise injection that makes the diffusion model stochastic. Two new samples from the same \(\mathbf{x}_T\) will produce different results. Repeating this noise injection over \(T\) time steps will produce a new sample \(\mathbf{x}_0\). This process is illustrated in Figure 1.
Figure 1: An example of a new sample x0 generated by a diffusion model Ho et al. (2020)
A critical drawback is the cost. If \(T = 1000\), generating one new sample requires 1000 sequential neural network computations, creating a significant bottleneck. This is addressed by Song et al. (2020) with the introduction of denoising diffusion implicit models (DDIM). This model defines a subsequence of time steps \(\tau_1 < \tau_2 < \dots < \tau_S\) with \(S \ll T\) to construct a deterministic rule to skip the intermediate steps between each \(\tau_s\). DDIM
also does not require retraining the model, it uses a different solver applied to the same trained network. This reduces the number of neural network computations from \(T = 1000\) to as few as 10-50 with a minimal loss in quality, but with a 20-100x speedup.
Conclusion
Diffusion models are one of the most statistically principled frameworks in modern generative AI, and their foundations read like a syllabus from an OR/IE graduate program: Markov chains, variational inference, KL divergence, MSE estimation, and stochastic processes. From the elegant reparameterization trick that makes training tractable, to the ELBO decomposition that reduces a complex likelihood problem to a simple noise prediction task, to the schedule design problem that is at its heart a constrained optimization, every component of a diffusion model has a mathematical story worth understanding.
As these models find their way into applications from molecular design to demand forecasting to combinatorial optimization, researchers equipped with this statistical vocabulary will be well-positioned — not just to use these tools, but to improve them.
"Diffusion models are an open research frontier with problems that speak directly to the operations research toolkit. Schedule design is a constrained optimization problem. Sampling efficiency is a resource allocation problem. Constrained generation during the reverse process remains largely unsolved. For graduate students looking for a foothold in machine learning research, diffusion models offer something rare: a state-of-the-art method where the most important open questions are not about network architecture, but about probability, optimization, and mathematical rigor.
"
References
Guo, Z., Lang, J., Huang, S., Gao, Y., Ding, X., 2025. A comprehensive review on noise control of diffusion model. arXiv preprint arXiv:2502.04669.
Ho, J., Jain, A., Abbeel, P., 2020. Denoising diffusion probabilistic models. Advances in neural information processing systems 33, 6840–6851.
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., Ganguli, S., 2015. Deep unsupervised learning using nonequilibrium thermodynamics, in: International conference on machine learning, pmlr. pp. 2256–2265.
Song, J., Meng, C., Ermon, S., 2020. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502.
Acknowledgements: We would like to thank Ronak Tiwari for taking time to review this article. Photo credit goes to TheDigitalArtist for the header photo and Emile Perron for the footer photo.
