A Hitchhiker's Guide To Diffusion RL

0How to read the diagrams

The diagrams below all use the same format. Columns are environment decisions: one condition \(c\) for image generation, or three robot decisions \(t,t+1,t+2\). Rows go from initial noise \(x_K\), through denoising \(x_k\to x_{k-1}\), to the output \(x_0\) / action \(a_t\), an optional residual, and finally the environment or reward model. I only draw five denoising rows, even for a 50-step sampler. The colors, stars, arrows, and weights mean the same thing across diagrams.

• Gray: frozen computation, or a timestep excluded from the loss.
• Solid accent: the output is an RL action; evaluate \(\log\pi\) and a likelihood ratio here.
• Accent outline: trainable computation without a local RL action, such as regression or a pathwise gradient.
• White: data or the environment: samples, actions, simulator, reward model.
• Dashed: reuse an earlier result instead of recomputing this column, as in chunked replanning.
• Star: where reward enters.
• Cell text: the assigned advantage weight. \(\hat A\) means the same broadcast value; \(\gamma^k\hat A\) means step weighting. Lighter fill indicates lower weight.
• Red reverse arrow: pathwise backpropagation from a differentiable reward or \(\nabla_aQ\).
• Stacked dashed boxes ×G: a group of sibling samples compared for relative advantages.
• Teal pill: the critic location, such as \(V(s_t)\) or \(Q(s,a)\).
• Teal arrows below the environment: backward recursion of GAE / discounted return across environment time.
• Right bracket: an operation spanning rows, such as transition KL, a moving window, or a joint-chain ratio.

1. 1Initial noise \(x_K\sim\mathcal N(0,I)\) is a white data cell here. A solid cell at this location (Noise PPO, DSRL) makes noise the RL action.
2. 2Gray: an early frozen denoising prefix, as in DPPO finetuning only the last \(K^{\prime}\) steps.
3. 3Outline: trainable parameters without a local log-probability term, updated through regression or pathwise gradients.
4. 4Solid: RL action with ratio \(\rho\). Labels \(\gamma\hat A\) and \(\hat A\) give its advantage weight; lighter fill means a smaller weight.
5. 5The final action \(a_t\) or sample \(x_0\).
6. 6An optional residual row. A solid cell makes the residual the RL action.
7. 7Environment / reward model. A star marks reward injection.
8. 8Critic attachment point.
9. 9Red arrow: how far the pathwise gradient is propagated backward.
10. 10Stacked ×G boxes: sibling samples under one condition.

Below each grid are the three parts of a training iteration: rollout, reward calculation, and the update.

1Method genealogy

I group the methods by where RL acts: noise space, inside the chain with likelihood ratios, likelihood-free / regression, and after the output. Each edge describes what changes between two methods. It does not necessarily mean one paper builds on the other. Click a method to jump to its section.

2Two-level MDP derivation: two horizons

Environment time is \(t=0,1,\dots\); denoising time is \(k=K,\dots,1,0\), from pure noise to clean sample/action. Terminal-reward image generation has one denoising horizon conditioned on prompt \(c\). In robotics, the condition is observation \(s_t\), and each environment decision generates an entire chain to obtain \(a_t^0\).

2.1A single denoising MDP: DDPO / DPOK

DDPO (Black et al., 2023) and DPOK (Fan et al., 2023) view one generation as a \(K\)-step episode:

The MDP transition is deterministic: the next state contains the \(x_{k-1}\) we just sampled. The randomness comes from the policy. Since each policy is diagonal Gaussian, we can compute its log probability:

Intermediate rewards are zero, and the inner discount is one. Every transition therefore has return \(r(x_0,c)\). DDPO has two estimators:

For \(\hat A\), we can normalize rewards using a running mean and standard deviation for each prompt. This gives us a baseline without training a critic. The idea is related to GRPO, although GRPO computes its statistics from samples in the same batch. DPOK adds a KL term to the same MDP. If the initial-noise distribution is unchanged, the chain-rule bound lets us replace the terminal KL with a sum of expected Gaussian transition KLs:

DPOK also considers a learned baseline \(V(x_k,c)\approx\mathbb E[r(x_0,c)\mid x_k]\) in its appendix: a critic over denoising states. Compare this with the environment-only critic in DPPO.

2.2The DPPO two-level MDP

DPPO (Ren et al., 2024) considers robots. At decision \(t\), start at \(a_t^K\sim\mathcal N(0,I)\), denoise to \(a_t^0\), execute the action (often a chunk), and transition to \(s_{t+1}\). Flatten the two levels using \(\bar t(t,k)=tK+(K-k-1)\). In this subsection, \(k=K-1,\dots,0\) indexes the destination of \(a^{k+1}\to a^k\):

The policy is still Gaussian: \(\pi_\theta(a_t^k\mid s_t,a_t^{k+1})=\mathcal N(a_t^k;\mu_\theta(a_t^{k+1},s_t,k),\sigma_{k+1}^2I)\). We can compute the log probability. The remaining question is how to turn the environment reward at \(k=0\) into an advantage for each \((t,k)\).

Computing and distributing advantage. DPPO uses two levels:

• The critic depends on \(s_t\), not the intermediate noisy action \(a_t^{k+1}\). The paper reports instability in an ablation that also conditions the value function on the noisy action.
• We compute advantage once per environment decision. All \(k\) use the same \(A_t\), with a weight \(\gamma_{\text{denoise}}^k\) (0.99 in the configurations discussed here). In the code, this is advantages *= gamma_denoising ** (ft_denoising_steps - i - 1). The index \(i\) increases through the trainable steps, so the last step has weight one.
• My understanding is the following. If the critic only sees \(s_t\) and the inner discount is one, the inner TD error is \(0+V(s_t)-V(s_t)=0\). We can then pass the environment advantage to the denoising steps. The extra \(\gamma_{\text{denoise}}\) downweights early, noisy steps; it does not discount physical time. This is an intuition, though. It does not prove that running GAE on an arbitrarily flattened chain gives the same estimator as the environment-level implementation.

PPO ratios and clipping. Each \((t,k)\) contributes a separate PPO sample:

• We save \(a_t^{k+1}\) and \(a_t^k\) during rollout and treat them as constants when updating. The gradient \(\nabla_\theta\log\pi_\theta\) only goes through the current \(\mu_\theta(a_t^{k+1},s_t,k)\). It does not backpropagate through the sampled denoising chain. If we batch the updates by step, we do not need to keep the whole chain’s activations in memory. We still have to store the rollout, and both storage and total computation grow with \(K\).
• The implementation supports timestep-dependent clipping: \(\epsilon_k=\epsilon_{\text{base}}+(\epsilon-\epsilon_{\text{base}})\frac{e^{\beta\tau_k}-1}{e^{\beta}-1}\), with \(\tau_k\in[0,1]\) increasing along denoising. Later transitions can have a wider clipping interval. The cited Robomimic configuration goes from clip_ploss_coef_base=0.001 to clip_ploss_coef=0.01; Gym uses 0.01 for both, giving no schedule.

Exploration and numerical stability. Small posterior \(\sigma_k\) near the clean end gives weak exploration and makes log probability highly sensitive to small mean changes. DPPO distinguishes a sampling floor \(\sigma^{\text{exp}}_{\min}\) (0.01–0.1 across cited tasks) from a log-probability floor \(\sigma^{\text{prob}}_{\min}=0.1\). Both are 0.1 in the cited Gym / Robomimic configurations.

Only finetune the last few steps. Make a copy \(\theta_{\text{ft}}\). Use frozen \(\theta\) for the first \(K-K^{\prime}\) transitions, then the trainable copy for the last \(K^{\prime}\), which enter the PPO loss. The configurations discussed here use \(K=20,K^{\prime}=10\). Another option is roughly five stochastic DDIM steps with \(\eta=1\) during training and \(\eta=0\) for evaluation. Execute \(T_a\) primitive actions from a chunk, then replan.

2.3Other factorizations

(a) Treat the whole chain as one action. Add the transition log probabilities to get the joint log probability; multiply the ratios to get the joint ratio:

The clean-action marginal ratio \(\pi_\theta(a^0)/\pi_{\mathrm{old}}(a^0)\) is generally intractable, but \(\rho^{\mathrm{chain}}\) is a valid weight on the extended trajectory space. At the on-policy point, the score gradients have the same form \(\mathbb E[A\sum_k\nabla\log\pi_k]\). Clipping their product and clipping individual ratios are different surrogates: many small local changes can accumulate as roughly \((1\pm\epsilon)^K\), while joint clipping can activate frequently on long chains. PPO clipping is not a hard constraint on either ratio or KL. OGPO uses the joint product.

(b) Outcome-only group comparison. Sample \(G\) chains for condition \(c\), compute \(\hat A_i=(r_i-\bar r)/(\operatorname{std}(r)+\varepsilon)\), and broadcast it through each chain before per-step ratio clipping (Flow-GRPO, DanceGRPO). No critic is needed, but the signal remains terminal rather than denoising-step-specific. SDE construction, sample grouping, fewer steps, and shared initial noise make practical differences.

(c) RL only in noise / latent space. Freeze \(f_\theta\), absorb it into the environment, and learn \(\pi_\psi(w\mid s)\) with output \(a=f_\theta(s,w)\), often using deterministic DDIM. The density belongs to the noise policy, not the denoising chain. DSRL uses off-policy SAC; Noise PPO uses a one-decision PPO formulation.

(d) Avoid chain likelihoods altogether. One family builds surrogate ratios from ELBO / flow-matching losses (FPO). Another learns through regression, including DiffusionNFT’s positive and negative implicit policies. A simpler related baseline is advantage-weighted regression:

These losses do not require the reverse chain’s likelihood, so we can use ODE or higher-order samplers. The training objectives are still different: NFT uses positive/negative implicit policies, while FPO uses a surrogate ratio. We also need to check whether each method works with classifier-free guidance.

2.4ODE → SDE: why flow policies use it

Flow matching / rectified flow samples with the ODE \(\mathrm dx_\tau=v_\theta(x_\tau,\tau)\,\mathrm d\tau\). Given the current state, the next step is deterministic. Its Dirac transition cannot give us the ordinary stochastic density needed for local PPO. This says nothing about whether the final output has a density. Let \(\tau=1\) denote noise and \(\tau=0\) denote data, so generation runs with \(\mathrm d\tau<0\). With an ideal velocity field / score in continuous time, we can construct a reverse SDE with the same marginals:

For rectified flow, \(x_t=(1-t)x_0+t\varepsilon\), and the score is \(\nabla\log p_t(x)=-[x+(1-t)v_\theta(x,t)]/t\). Substitution and Euler–Maruyama discretization give the Flow-GRPO transition below. Here \(t\) is the flow interpolation variable, not environment time:

Each discrete transition is isotropic Gaussian. With the same diffusion coefficient, its KL is \(\mathrm{KL}=\frac{|\Delta t|}{2}\big(\frac{\sigma_t(1-t)}{2t}+\frac1{\sigma_t}\big)^2\lVert v_\theta-v_{\mathrm{ref}}\rVert^2\). The hyperparameter \(a\) controls exploration. Generation has \(\Delta t<0\); covariance and KL use \(|\Delta t|\). Finite-step solvers need not exactly preserve continuous-time marginals, and endpoints require appropriate treatment.

ReinFlow instead adds a learned noise network directly to each flow step:

This gives us a discrete Markov chain, whose joint likelihood is a product of Gaussian transitions. RL learns the noise scale within upper and lower bounds, and the models discussed here use only 1–4 steps. The price is that we no longer have a guarantee of preserving the pretrained marginals. πRL tries both approaches: Flow-Noise uses learned noise and a single-level formulation; Flow-SDE uses an SDE and a DPPO-like two-level MDP.

2.5Where does the gradient go?

2.6General on-policy estimators

Practical difficulties for diffusion policies

1. Which likelihood is tractable? Deterministic ODE / DDIM steps do not supply ordinary local stochastic densities; Gaussian stochastic transitions do. The marginal \(\pi(a^0\mid s)\) still generally requires integrating out the chain. Flow policies can use SDEs, injected noise, or alternative objectives (FPO / NFT). GenPO uses a specialized invertible construction with augmented dummy actions to obtain tractable likelihoods.
2. Credit within denoising. Reward arrives only at \(x_0\). Common choices share that signal (DPPO adds \(\gamma_{\text{denoise}}^k\); GRPO broadcasts), or learn a denoising-state critic, such as DPOK’s optional \(V(x_k,c)\). Step-aware preference/reward methods require additional models or sampling.
3. Variance. There are \(K\) terms in the score sum. Its variance depends on each term and on their covariances, so it does not simply grow with \(K\). Still, long chains can be hard to optimize. DPPO only trains the last few steps, DanceGRPO samples a subset (about 60% in the setup discussed here), and MixGRPO trains a moving window.
4. Compute. Each environment decision requires \(K\) model evaluations, and every trained transition adds another forward/backward evaluation. Examples include 10 training steps versus 40 inference steps in Flow-GRPO, five-step DDIM, and 1–4-step shortcut / ReFlow policies in ReinFlow.
5. Exploration noise. Tiny late-step variance weakens exploration and can make likelihood ratios numerically unstable. Solutions include DPPO variance floors, Flow-GRPO’s noise multiplier \(a\), and ReinFlow’s learned noise. Larger perturbations can also reduce sample quality and move samples away from familiar behavior.
6. KL to the pretrained model. When the initial-noise distribution is the same, the sum of expected transition KLs upper-bounds the terminal KL. For Gaussians with equal covariance, we can calculate it in closed form. DanceGRPO reports little benefit from KL in its experiments. I would not take that as a reason to drop regularization everywhere: learned rewards can be exploited, so KL, BC, and independent evaluation may still be needed.

3Inside the chain: likelihood ratios on denoising transitions

These methods treat some denoising transitions as RL actions. Those are the solid cells, where we compute log probabilities and ratios. The differences are which steps get updated, how we compute their advantages, and where we put the critic. We use condition \(c\) or \(s_t\), latent \(x_k\), old policy \(\theta_{\mathrm{old}}\), reference \(\theta_{\mathrm{ref}}\), advantage \(\hat A\), and ratio \(\rho\) throughout.

4Noise space: only \(x_K\) is the action

The generator is frozen: all denoising cells are gray and only the initial-noise cell is solid. Any required density belongs to the noise policy itself, independently of the denoising chain.

5Likelihood-free / regression: no chain \(\log\pi\)

Here most denoising cells are outlined. We still train the parameters, but use regression, flow-matching losses, or pathwise gradients instead of local likelihood ratios.

6Outside the chain: residuals after \(x_0\)

The generator column stays frozen and only the residual row is solid. The relevant density is the residual actor’s own distribution.

7Image / video versus robotics: what actually corresponds?

After going through the methods, I want to compare image/video generation and robotics more directly. The questions are the same: what do we generate, where do we get reward, which random variable do we compute the ratio on, and how do we assign credit across time? Here I compare the formulations, rather than rank scores from different benchmarks.

7.1 One score-function identity, two return problems

Let us first write down the objective for generating and scoring one image or video. If the initial-noise distribution is fixed, only the conditional transitions contribute a gradient. For now, leave out the baseline, PPO clipping, and KL terms:

For a robot, write the final action distribution as \(\pi_\theta(A_j\mid s_j)\). The policy gradient uses the advantage of this decision for future environment return. We can expand the score into denoising-level terms in the implementation. DPPO uses a particular estimate and weighting, \(\gamma_{\mathrm{denoise}}^k A_j\). Simply replacing an image score \(R\) with the immediate environment reward \(r_j\) does not give us the same update.

The state expectation here uses the visitation distribution for the discounted objective. For generation, we often give all steps in a chain the final sample’s score. For control, we also need to estimate what this action does to future states. This is what the critic helps us with; it is not required just because the task involves a robot.

7.2 Action chunking: define the duration of an environment step

Suppose that at replanning step \(j\), we predict \(H\) actions but only execute the first \(h_j\le H\), without calling the generator again. Let \(n_j\) be the starting primitive control step. Set \(b_j=0\) at true termination and 1 otherwise. If \(\gamma\) is the discount per primitive control step, we get the following chunk reward and bootstrap:

Here \(\lambda\) is defined per replanning decision. We could instead define the discount per chunk, but that would change the time scale. We also need to distinguish a time-limit truncation from true termination when deciding whether to bootstrap. The remaining \(H-h_j\) actions were never executed, so they contributed no direct environment reward to this chunk. We cannot count predicted actions as executed ones. These equations use one chunk-level convention; an implementation may choose another.

7.3 Why “same-prompt groups” do not become “same-batch groups”

Flow-GRPO / DanceGRPO compare samples generated under the same condition \(c\). In a robot batch, however, \(s_1,s_2,\ldots\) may come from different task stages and difficulty levels. If we normalize their returns together, we mix up the quality of the action with how good the starting state already was.

One option is to restore the same state and sample several future trajectories. Another is to generate several candidate actions at that state and use a critic to estimate their returns. The first needs resets or environment branching. The second depends on the critic, including its error on out-of-distribution actions; OGPO’s same-state chain comparison takes this approach. Also, subtracting a mean and dividing by a standard deviation are different choices. They give different update scales in the two equations above.

7.4 Which log probability is tractable?

We can calculate each Gaussian transition’s density, but this does not automatically give us the final-action marginal. With a saved latent trajectory, we can calculate a joint ratio. For the final-action density, we still have to integrate out the intermediate variables. Clipping each step, clipping the product over the chain, and doing PPO on the final-action marginal give three different objectives.

This is something to check in the code when reading DPPO, ReinFlow, OGPO, or GenPO. A fixed initial-noise prior cancels in the ratio. If we train the noise policy, as in Noise PPO / DSRL, we are optimizing a different object. FPO’s CFM-loss ratio is a surrogate; it is different from both exact density ratios above.

7.5 What should a comparison hold constant?

When a method is “more efficient,” I want to know what it saves: training FLOPs, memory, reward calls, or environment interactions. Running an image reward model once and resetting a physical robot after a failed trial have very different costs. These are the comparisons I would make using the table above; each method’s actual setup is given in its cited paper.

8Pairwise comparison

Select two methods to see their grids and details side by side. Differences are highlighted. The default is DDPO versus DPPO, showing what changes when we add the environment MDP.

Highlighted rows show a difference between the two methods. The definitions are the same as in their sections above.

9Overview table

“Likelihood” means an exact policy log probability on a denoising transition, latent noise, or residual action. “On/off” describes the data used for policy updates.

References

Reference numbers, titles, and authors follow the original notes dated October 7, 2026; years refer to first arXiv submission. Numerical results and configurations belong to the cited papers or implementations and are not universal defaults.

1. Black et al. Training Diffusion Models with Reinforcement Learning (DDPO). 2023. arXiv:2305.13301
2. Fan et al. DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models. 2023. arXiv:2305.16381
3. Zhang et al. Large-scale Reinforcement Learning for Diffusion Models. 2024. arXiv:2401.12244
4. Wallace et al. Diffusion Model Alignment Using Direct Preference Optimization. 2023. arXiv:2311.12908
5. Yang et al. Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model (D3PO). 2023. arXiv:2311.13231
6. Liang et al. Aesthetic Post-Training Diffusion Models from Generic Preferences with Step-by-step Preference Optimization (SPO). 2024. arXiv:2406.04314
7. Xu et al. ImageReward (ReFL). 2023. arXiv:2304.05977
8. Clark et al. Directly Fine-Tuning Diffusion Models on Differentiable Rewards (DRaFT). 2023. arXiv:2309.17400
9. Prabhudesai et al. Aligning Text-to-Image Diffusion Models with Reward Backpropagation (AlignProp). 2023. arXiv:2310.03739
10. Liu et al. Flow-GRPO: Training Flow Matching Models via Online RL. 2025. arXiv:2505.05470
11. Xue et al. DanceGRPO: Unleashing GRPO on Visual Generation. 2025. arXiv:2505.07818
12. Li et al. MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDE. 2025. arXiv:2507.21802
13. Zheng et al. DiffusionNFT: Online Diffusion Reinforcement with Forward Process. 2025. arXiv:2509.16117
14. Miao et al. A Minimalist Method for Fine-tuning Text-to-Image Diffusion Models (Noise PPO). 2025. arXiv:2506.12036
15. Eyring et al. ReNO: Reward-based Noise Optimization. 2024. arXiv:2406.04312
16. Ren et al. Diffusion Policy Policy Optimization (DPPO). 2024. arXiv:2409.00588, code
17. Yang et al. Fine-tuning Diffusion Policies with Backpropagation Through Diffusion Timesteps (NCDPO). 2025. arXiv:2505.10482
18. Zhang et al. ReinFlow: Fine-tuning Flow Matching Policy with Online Reinforcement Learning. 2025. arXiv:2505.22094
19. McAllister et al. Flow Matching Policy Gradients (FPO). 2025. arXiv:2507.21053
20. Ding et al. GenPO: Generative Diffusion Models Meet On-Policy Reinforcement Learning. 2025. arXiv:2505.18763
21. Chen et al. πRL: Online RL Fine-tuning for Flow-based Vision-Language-Action Models. 2025. arXiv:2510.25889
22. Patil et al. OGPO: Sample Efficient Full-Finetuning of Generative Control Policies. 2026. arXiv:2605.03065
23. Wagenmaker et al. Steering Your Diffusion Policy with Latent Space Reinforcement Learning (DSRL). 2025. arXiv:2506.15799
24. Ankile et al. From Imitation to Refinement: Residual RL for Precise Assembly (ResiP). 2024. arXiv:2407.16677
25. Yuan et al. Policy Decorator: Model-Agnostic Online Refinement for Large Policy Model. 2024. arXiv:2412.13630
26. Wang et al. Diffusion-QL. 2022. arXiv:2208.06193；Hansen-Estruch et al. IDQL. 2023. arXiv:2304.10573；Yang et al. DIPO. 2023. arXiv:2305.13122；Psenka et al. QSM. 2023. arXiv:2312.11752；Wang et al. DACER. 2024. arXiv:2405.15177；Mark et al. PA-RL. 2024. arXiv:2412.06685
27. Schulman et al. PPO. 2017. arXiv:1707.06347；Schulman et al. GAE. 2015. arXiv:1506.02438
28. Shao et al. DeepSeekMath (GRPO). 2024. arXiv:2402.03300；Ahmadian et al. RLOO. 2024. arXiv:2402.14740；Hu et al. REINFORCE++. 2025. arXiv:2501.03262
29. Han et al. ReinDiffuse. 2024. arXiv:2410.07296；Yue et al. RLPF. 2025. arXiv:2506.12769；PhysMoDPO. 2026. arXiv:2603.13228