Policy Representation via Diffusion Probability Model for Reinforcement Learning
[* L.Yang and Z.Huang share equal contributions.]

Long Yang$^{1,*}$, Zhixiong Huang $^{2,*}$, Fenghao Lei$^2$, Yucun Zhong$^3$, Yiming Yang$^4$,

Cong Fang$^1$, Shiting Wen$^5$, Binbin Zhou$^6$, Zhouchen Lin$^1$

$^1$School of Artificial Intelligence, Peking University, Beijing, China

$^2$College of Computer Science and Technology, Zhejiang University, China

$^3$MOE Frontiers Science Center for Brain and Brain-Machine Integration & College of
Computer Science, Zhejiang University, China.

$^4$Institute of Automation Chinese Academy of Sciences Beijing, China

$^5$School of Computer and Data Engineering, NingboTech University, China

$^6$College of Computer Science and Technology, Hangzhou City University, China

August 27, 2024

Popular reinforcement learning (RL) algorithms tend to produce a unimodal policy distribution, which weakens the expressiveness of complicated policy and decays the ability of exploration. The diffusion probability model is powerful to learn complicated multimodal distributions, which has shown promising and potential applications to RL.

In this paper, we formally build a theoretical foundation of policy representation via the diffusion probability model and provide practical implementations of diffusion policy for online model-free RL. Concretely, we character diffusion policy as a stochastic process induced by stochastic differential equations, which is a new approach to representing a policy.
Then we present a convergence guarantee for diffusion policy, which provides a theory to understand the multimodality of diffusion policy.
Furthermore, we propose the DIPO, which implements model-free online RL with DIffusion POlicy.
To the best of our knowledge, DIPO is the first algorithm to solve model-free online RL problems with the diffusion model.
Finally, extensive empirical results show the effectiveness and superiority of DIPO on the standard continuous control Mujoco benchmark.

§ INTRODUCTION

Existing policy representations (e.g., Gaussian distribution) for reinforcement learning (RL) tend to output a unimodal distribution over the action space,
which may be trapped in a locally optimal solution due to its limited expressiveness of complex distribution and may result in poor performance.

Diffusion probability model [Sohl-Dickstein et al., 2015, Ho et al., 2020, Song et al., 2021] is powerful to learn complicated multimodal distributions, which has been applied to RL tasks (e.g., [Ajay et al., 2023, Reuss et al., 2023, Chi et al., 2023]).

Although the diffusion model (or diffusion policy) shows its promising and potential applications to RL tasks, previous works are all empirical or only consider offline RL settings. This raises some fundamental questions:
How to character diffusion policy? How to show the expressiveness of diffusion policy? How to design a diffusion policy for online model-free RL?
Those are the focuses of this paper.

every picture/.style=line width=0.75pt

[color=rgb, 255:red, 0; green, 0; blue, 0   ,draw opacity=1 ][line width=1.5]  (209.16,159.55) – (288.76,159.55)(248.52,102.85) – (248.52,172.53) (281.76,154.55) – (288.76,159.55) – (281.76,164.55) (243.52,109.85) – (248.52,102.85) – (253.52,109.85)  ;

[color=rgb, 255:red, 0; green, 0; blue, 0   ,draw opacity=1 ][line width=1.5]  (370.28,159.41) – (449.88,159.41)(409.63,102.71) – (409.63,172.4) (442.88,154.41) – (449.88,159.41) – (442.88,164.41) (404.63,109.71) – (409.63,102.71) – (414.63,109.71)  ;

[color=rgb, 255:red, 0; green, 0; blue, 0   ,draw opacity=1 ][line width=1.5]  (48.86,159.55) – (128.46,159.55)(88.21,102.85) – (88.21,172.53) (121.46,154.55) – (128.46,159.55) – (121.46,164.55) (83.21,109.85) – (88.21,102.85) – (93.21,109.85)  ;

[color=rgb, 255:red, 255; green, 0; blue, 0   ,draw opacity=1 ][line width=1.5]    (51.67,153.61) .. controls (78.83,153.61) and (75.85,115.88) .. (89.13,115.77) .. controls (93.53,115.73) and (95.94,119.77) .. (98.15,125.23) .. controls (102.61,136.23) and (106.25,152.99) .. (123.78,153.61) ;

[color=rgb, 255:red, 0; green, 0; blue, 0   ,draw opacity=1 ][line width=1.5]  (491.4,159.41) – (571,159.41)(530.75,102.71) – (530.75,172.4) (564,154.41) – (571,159.41) – (564,164.41) (525.75,109.71) – (530.75,102.71) – (535.75,109.71)  ;

[color=rgb, 255:red, 255; green, 0; blue, 0   ,draw opacity=1 ][line width=1.5]    (493.27,153.94) .. controls (505.22,154.45) and (509.1,139.33) .. (511.47,130.51) .. controls (513.36,123.42) and (514.29,120.38) .. (517.62,132.71) .. controls (525.11,160.4) and (531.65,114.09) .. (538.22,115.17) .. controls (544.8,116.25) and (542.91,164.09) .. (550.4,141.02) .. controls (554.48,128.43) and (556.4,133.55) .. (558.64,140.89) .. controls (560.5,147) and (562.58,154.65) .. (566.32,154.86) ;

[color=rgb, 255:red, 255; green, 0; blue, 0   ,draw opacity=1 ][line width=1.5]    (376.72,154.32) .. controls (389.27,154.32) and (393.05,146.33) .. (396.43,140.33) .. controls (399.81,134.34) and (402.15,113.43) .. (409.29,113.37) .. controls (416.43,113.31) and (409.29,156.2) .. (420.01,136.11) .. controls (430.73,116.01) and (426.81,163.45) .. (432.87,148.79) .. controls (438.93,134.14) and (446.68,147.51) .. (448.82,154.32) ;

[color=rgb, 255:red, 255; green, 0; blue, 0   ,draw opacity=1 ][line width=1.5]    (213.45,153.39) .. controls (232.18,153.39) and (228.7,112.98) .. (235.93,113.71) .. controls (243.15,114.43) and (242.07,127.19) .. (245.29,126.63) .. controls (248.51,126.06) and (263.18,106.58) .. (264.96,124.78) .. controls (266.73,142.98) and (267.8,153.55) .. (289.31,154.32) ;

[line width=0.75]    (321.93,141.33) – (297.6,141.41) ;
[shift=(294.6,141.42), rotate = 360] [fill=rgb, 255:red, 0; green, 0; blue, 0   ][line width=0.08]  [draw opacity=0] (10.72,-5.15) – (0,0) – (10.72,5.15) – (7.12,0) – cycle    ;

[line width=0.75]    (372.31,141.33) – (347.98,141.41) ;
[shift=(344.98,141.42), rotate = 359.82] [fill=rgb, 255:red, 0; green, 0; blue, 0   ][line width=0.08]  [draw opacity=0] (10.72,-5.15) – (0,0) – (10.72,5.15) – (7.12,0) – cycle    ;

[line width=0.75]    (491.41,141.61) – (456.23,141.92) ;
[shift=(453.23,141.95), rotate = 359.5] [fill=rgb, 255:red, 0; green, 0; blue, 0   ][line width=0.08]  [draw opacity=0] (10.72,-5.15) – (0,0) – (10.72,5.15) – (7.12,0) – cycle    ;

[line width=0.75]    (152.58,141.33) – (128.25,141.41) ;
[shift=(125.25,141.42), rotate = 360] [fill=rgb, 255:red, 0; green, 0; blue, 0   ][line width=0.08]  [draw opacity=0] (10.72,-5.15) – (0,0) – (10.72,5.15) – (7.12,0) – cycle    ;

[line width=0.75]    (201.89,141.33) – (177.55,141.41) ;
[shift=(174.55,141.42), rotate = 359.82] [fill=rgb, 255:red, 0; green, 0; blue, 0   ][line width=0.08]  [draw opacity=0] (10.72,-5.15) – (0,0) – (10.72,5.15) – (7.12,0) – cycle    ;

[color=rgb, 255:red, 0; green, 0; blue, 0   ,draw opacity=1 ][line width=1.5]  (209.16,276.92) – (288.76,276.92)(248.52,220.22) – (248.52,289.91) (281.76,271.92) – (288.76,276.92) – (281.76,281.92) (243.52,227.22) – (248.52,220.22) – (253.52,227.22)  ;

[color=rgb, 255:red, 0; green, 0; blue, 0   ,draw opacity=1 ][line width=1.5]  (370.28,276.79) – (449.88,276.79)(409.63,220.09) – (409.63,289.77) (442.88,271.79) – (449.88,276.79) – (442.88,281.79) (404.63,227.09) – (409.63,220.09) – (414.63,227.09)  ;

[color=rgb, 255:red, 0; green, 0; blue, 0   ,draw opacity=1 ][line width=1.5]  (48.86,276.92) – (128.46,276.92)(88.21,220.22) – (88.21,289.91) (121.46,271.92) – (128.46,276.92) – (121.46,281.92) (83.21,227.22) – (88.21,220.22) – (93.21,227.22)  ;

[color=rgb, 255:red, 245; green, 166; blue, 35   ,draw opacity=1 ][line width=1.5]    (51.67,270.99) .. controls (78.83,270.99) and (75.85,233.26) .. (89.13,233.15) .. controls (93.53,233.11) and (95.94,237.15) .. (98.15,242.6) .. controls (102.61,253.61) and (106.25,270.37) .. (123.78,270.99) ;

[color=rgb, 255:red, 0; green, 0; blue, 0   ,draw opacity=1 ][line width=1.5]  (491.4,276.79) – (571,276.79)(530.75,220.09) – (530.75,289.77) (564,271.79) – (571,276.79) – (564,281.79) (525.75,227.09) – (530.75,220.09) – (535.75,227.09)  ;

[color=rgb, 255:red, 245; green, 166; blue, 35   ,draw opacity=1 ][line width=1.5]    (493.27,271.31) .. controls (505.22,271.83) and (509.1,256.71) .. (511.47,247.88) .. controls (513.36,240.8) and (514.29,237.76) .. (517.62,250.09) .. controls (525.11,277.77) and (531.65,231.47) .. (538.22,232.55) .. controls (544.8,233.63) and (542.91,281.47) .. (550.4,258.39) .. controls (554.48,245.81) and (556.4,250.93) .. (558.64,258.26) .. controls (560.5,264.38) and (562.58,272.03) .. (566.32,272.24) ;

[color=rgb, 255:red, 245; green, 166; blue, 35   ,draw opacity=1 ][line width=1.5]    (376.72,271.69) .. controls (389.27,271.69) and (393.05,263.71) .. (396.43,257.71) .. controls (399.81,251.72) and (402.15,230.81) .. (409.29,230.75) .. controls (416.43,230.69) and (409.29,273.57) .. (420.01,253.48) .. controls (430.73,233.39) and (426.81,280.83) .. (432.87,266.17) .. controls (438.93,251.52) and (446.68,264.88) .. (448.82,271.69) ;

[color=rgb, 255:red, 245; green, 166; blue, 35   ,draw opacity=1 ][line width=1.5]    (213.45,270.77) .. controls (232.18,270.77) and (228.7,230.36) .. (235.93,231.08) .. controls (243.15,231.8) and (242.07,244.57) .. (245.29,244) .. controls (248.51,243.44) and (263.18,223.96) .. (264.96,242.16) .. controls (266.73,260.36) and (267.8,270.93) .. (289.31,271.69) ;

[line width=0.75]    (455.38,259.32) – (490.97,259.32) ;
[shift=(493.97,259.32), rotate = 180] [fill=rgb, 255:red, 0; green, 0; blue, 0   ][line width=0.08]  [draw opacity=0] (10.72,-5.15) – (0,0) – (10.72,5.15) – (7.12,0) – cycle    ;

[color=rgb, 255:red, 245; green, 166; blue, 35   ,draw opacity=1 ][dash pattern=on 2pt off 3pt][line width=0.75]  (48,202) – (452,202) – (452,291.06) – (48,291.06) – cycle ;

[color=rgb, 255:red, 255; green, 0; blue, 0   ,draw opacity=1 ][dash pattern=on 2pt off 3pt][line width=0.75]  (48,97) – (452,97) – (452,189.52) – (48,189.52) – cycle ;

[line width=0.75]    (124.17,258.27) – (150.11,258.74) ;
[shift=(153.11,258.79), rotate = 181.05] [fill=rgb, 255:red, 0; green, 0; blue, 0   ][line width=0.08]  [draw opacity=0] (10.72,-5.15) – (0,0) – (10.72,5.15) – (7.12,0) – cycle    ;

[line width=0.75]    (175.62,258.27) – (200.49,258.74) ;
[shift=(203.49,258.79), rotate = 181.09] [fill=rgb, 255:red, 0; green, 0; blue, 0   ][line width=0.08]  [draw opacity=0] (10.72,-5.15) – (0,0) – (10.72,5.15) – (7.12,0) – cycle    ;

[line width=0.75]    (293.53,258.27) – (319.47,258.74) ;
[shift=(322.47,258.79), rotate = 181.05] [fill=rgb, 255:red, 0; green, 0; blue, 0   ][line width=0.08]  [draw opacity=0] (10.72,-5.15) – (0,0) – (10.72,5.15) – (7.12,0) – cycle    ;

[line width=0.75]    (344.98,258.27) – (369.85,258.74) ;
[shift=(372.85,258.79), rotate = 181.09] [fill=rgb, 255:red, 0; green, 0; blue, 0   ][line width=0.08]  [draw opacity=0] (10.72,-5.15) – (0,0) – (10.72,5.15) – (7.12,0) – cycle    ;

(75.94,89.43) .. controls (75.94,82.71) and (81.46,77.27) .. (88.27,77.27) .. controls (95.07,77.27) and (100.59,82.71) .. (100.59,89.43) .. controls (100.59,96.14) and (95.07,101.59) .. (88.27,101.59) .. controls (81.46,101.59) and (75.94,96.14) .. (75.94,89.43) – cycle ;

(518.62,297.75) .. controls (518.62,291.03) and (524.14,285.59) .. (530.94,285.59) .. controls (537.75,285.59) and (543.27,291.03) .. (543.27,297.75) .. controls (543.27,304.46) and (537.75,309.91) .. (530.94,309.91) .. controls (524.14,309.91) and (518.62,304.46) .. (518.62,297.75) – cycle ;

(75.94,297.75) .. controls (75.94,291.03) and (81.46,285.59) .. (88.27,285.59) .. controls (95.07,285.59) and (100.59,291.03) .. (100.59,297.75) .. controls (100.59,304.46) and (95.07,309.91) .. (88.27,309.91) .. controls (81.46,309.91) and (75.94,304.46) .. (75.94,297.75) – cycle ;

(518.62,89.43) .. controls (518.62,82.71) and (524.14,77.27) .. (530.94,77.27) .. controls (537.75,77.27) and (543.27,82.71) .. (543.27,89.43) .. controls (543.27,96.14) and (537.75,101.59) .. (530.94,101.59) .. controls (524.14,101.59) and (518.62,96.14) .. (518.62,89.43) – cycle ;

[line width=1.5]    (110,89)–  (230,89) ;
[shift=(102,89), rotate = 360] [fill=rgb, 255:red, 0; green, 0; blue, 0   ][line width=0.08]  [draw opacity=0] (13.4,-6.43) – (0,0) – (13.4,6.44) – (8.9,0) – cycle    ;

[line width=1.5]    (410,89) – (518,89) ;

[line width=1.5]    (102,300) – (120,300) ;

[line width=1.5]    (485,300) – (512,300) ;
[shift=(516.47,300), rotate = 180] [fill=rgb, 255:red, 0; green, 0; blue, 0   ][line width=0.08]  [draw opacity=0] (13.4,-6.43) – (0,0) – (13.4,6.44) – (8.9,0) – cycle    ;

(522,85) node [anchor=north west][inner sep=0.75pt]    $\bara_{0}$;
(80,85) node [anchor=north west][inner sep=0.75pt]    $\bara_{T}$;
(83,186) node [anchor=north west][inner sep=0.75pt]    $\big\|$;
(80,293) node [anchor=north west][inner sep=0.75pt]    $\tildea_{0}$;
(522,293) node [anchor=north west][inner sep=0.75pt]    $\tildea_{T}$;

(383,170) node [anchor=north west][inner sep=0.75pt]    $\bara_{1}\sim\barpi_{1}(\cdot|\bs)$;

(220.56,170) node [anchor=north west][inner sep=0.75pt]    $\bara_{t}\sim\barpi_{t}(\cdot|\bs)$;

(48,170) node [anchor=north west][inner sep=0.75pt]    $\bara_{T}\sim\barpi_{T}(\cdot|\bs)\approx\calN(\bm{0},\bI)$;

(504,170) node [anchor=north west][inner sep=0.75pt]    $\bara_{0}\sim\pi(\cdot|\bs)$;
(460,125) node [anchor=north west][inner sep=0.75pt]    input;

(460,245) node [anchor=north west][inner sep=0.75pt]    ouput;

(323,137.5) node [anchor=north west][inner sep=0.75pt]    $\cdots $;

(153,137.5) node [anchor=north west][inner sep=0.75pt]    $\cdots$;

(153,254) node [anchor=north west][inner sep=0.75pt]    $\cdots $;

(48,203) node [anchor=north west][inner sep=0.75pt]    $\tildea_{0} \sim \tildepi_{0}(\cdot|\bs)=\barpi_{T}(\cdot|\bs)$;

(208,203) node [anchor=north west][inner sep=0.75pt]    $\tildea_{T-t} \sim \tildepi_{T-t}(\cdot|\bs)$;

(350,203) node [anchor=north west][inner sep=0.75pt]    $\tildea_{T-1} \sim \tildepi_{T-1}(\cdot|\bs)$;

(500,203) node [anchor=north west][inner sep=0.75pt]    $\tildea_{T} \sim \tildepi_{T}(\cdot|\bs)$;

(323,254) node [anchor=north west][inner sep=0.75pt]    $\cdots $;

(230,78) node [anchor=north west][inner sep=0.75pt]    $\dd \bara_t=-\frac{1}{2}g(t)\bara_t\dd t+g(t)\dd \bw_{t}$;
(210,52) node [anchor=north west][inner sep=0.75pt]    Forward SDE: $\ba\sim\pi(\cdot|\bs)\rightarrow \calN(\bm{0},\bI)$;

(123,291) node [anchor=north west][inner sep=0.75pt]    $\dd \tildea_{t}=\frac{1}{2}g^2(T-t)\left[\tildea_{t}+2\bnabla \log p_{T-t}( \tildea_{t})\right]\dd t+g(T-t)\dd \bar{\bw}_{t}$;

(210,320) node [anchor=north west][inner sep=0.75pt]    Reverse SDE: $\calN(\bm{0},\bI)\rightarrow\pi(\cdot|\bs)$;

Diffusion Policy: Policy Representation via Stochastic Process. For a given state $\bs$, the forward stochastic process $\{\bara_{t}|\bs\}$ maps the input $\bara_{0}=:\ba\sim\pi(\cdot|\bs)$ to be a noise; then we recover the input by the stochastic process $\{\tildea_{t}|\bs\}$ that reverses the reversed SDE if we know the score function $\bnabla \log p_{t}(\cdot)$, where $p_{t}(\cdot)$ is the probability distribution of the forward process, i.e., $p_{t}(\cdot)=\barpi_{t}(\cdot|\bs)$.

§.§ Our Main Work

In this paper, we mainly consider diffusion policy from the next three aspects.

Charactering Diffusion Policy as Stochastic Process.  We formulate diffusion policy as a stochastic process that involves two processes induced by stochastic differential equations (SDE), see Figure <ref>, where the forward process disturbs the input policy $\pi$ to noise, then the reverse process infers the policy $\pi$ according to a corresponding reverse SDE. Although this view is inspired by the score-based generative model [Song et al., 2021], we provide a brand new approach to represent a policy: via a stochastic process induced by SDE, neither via value function nor parametric function.
Under this framework, the diffusion policy is flexible to generate actions according to numerical SDE solvers.

Convergence Analysis of Diffusion Policy.
Under mild conditions, Theorem <ref> presents a theoretical convergence guarantee for diffusion policy.
The result shows that if the score estimator is sufficiently accurate, then diffusion policy efficiently infers the actions from any realistic policy that generates the training data.
It is noteworthy that Theorem <ref> also shows that diffusion policy is powerful to represent a multimodal distribution, which leads to sufficient exploration and better reward performance, Section <ref> and Appendix <ref> provide more discussions with numerical verifications for this view.

Diffusion Policy for Model-free Online RL. Recall the standard model-free online RL framework, see Figure <ref>, where the policy improvement produces a new policy $\pi^{'}\succeq\pi$ according to the data $\calD$.
However, Theorem <ref> illustrates that the diffusion policy only fits the distribution of the policy $\pi$ but does not improve the policy $\pi$.
We can not embed the diffusion policy into the standard RL training framework, i.e., the policy improvement in Figure <ref> can not be naively replaced by diffusion policy. To apply diffusion policy to model-free online RL task, we propose the DIPO algorithm, see Figure <ref>.
The proposed DIPO considers a novel way for policy improvement, we call it action gradient that updates each $\ba_t\in\calD$ along the gradient field (over the action space) of state-action value:

\[
\ba_{t}\leftarrow\ba_{t}+\eta \nabla_{\ba} Q_{\pi}( \bs_t,\ba_t),
\]

where for a given state $\bs$, $Q_{\pi}(\bs,\ba)$ measures the reward performance over the action space $\calA$.
Thus, DIPO improves the policy according to the actions toward to better reward performance. To the best of our knowledge, this paper first presents the idea of action gradient, which provides an efficient way to make it possible to design a diffusion policy for online RL.

§.§ Paper Organization

Section <ref> presents the background of reinforcement learning.
Section <ref> presents our motivation from the view of policy representation.
Section <ref> presents the theory of diffusion policy.
Section <ref> presents the practical implementation of diffusion policy for model-free online reinforcement learning.
Section <ref> presents the experiment results.

every picture/.style=line width=0.75pt

(70.5,504.01) .. controls (70.5,498.22) and (75.36,493.53) .. (81.35,493.53) .. controls (87.35,493.53) and (92.21,498.22) .. (92.21,504.01) .. controls (92.21,509.81) and (87.35,514.5) .. (81.35,514.5) .. controls (75.36,514.5) and (70.5,509.81) .. (70.5,504.01) – cycle ;

(96.21,505) – (125.5,505) ;
[shift=(128.5,505), rotate = 180] [fill=rgb, 255:red, 0; green, 0; blue, 0   ][line width=0.08]  [draw opacity=0] (8.93,-4.29) – (0,0) – (8.93,4.29) – cycle    ;

(274,505) – (428,505) ;
[shift=(432,505), rotate = 180] [fill=rgb, 255:red, 0; green, 0; blue, 0   ][line width=0.08]  [draw opacity=0] (8.93,-4.29) – (0,0) – (8.93,4.29) – cycle    ;

(82.5,515) – (82.5,530) ;
[shift=(82.5,514), rotate = 90] [fill=rgb, 255:red, 0; green, 0; blue, 0   ][line width=0.08]  [draw opacity=0] (8.93,-4.29) – (0,0) – (8.93,4.29) – cycle    ;

(445,515) – (445,530) ;

(82,530) – (445.5,530) ;

(433.5,504.44) .. controls (433.5,498.65) and (438.36,493.95) .. (444.35,493.95) .. controls (450.35,493.95) and (455.21,498.65) .. (455.21,504.44) .. controls (455.21,510.23) and (450.35,514.92) .. (444.35,514.92) .. controls (438.36,514.92) and (433.5,510.23) .. (433.5,504.44) – cycle ;

(75.36,500) node [anchor=north west][inner sep=0.75pt]    $\pi $;

(129,496) node [anchor=north west][inner sep=0.75pt]    $\calD=\{\bs_{t},\ba_{t} ,\bs_{t+1} ,r_{t+1}\}$;

(95,488) node [anchor=north west][inner sep=0.75pt]   [align=left] data;

(290,485) node [anchor=north west][inner sep=0.75pt]   [align=left] policy improvement;
(261.66,510) node [anchor=north west][inner sep=0.75pt]   [align=left] e.g.,Q-learning/SAC/PPO;

(215,513) node [anchor=north west][inner sep=0.75pt]    $\pi \gets \pi ^{'}$;

(436,495) node [anchor=north west][inner sep=0.75pt]    $\pi^{'}$;

Standard Training Framework for Model-free Online RL.

every picture/.style=line width=0.75pt

(70.5,504.01) .. controls (70.5,498.22) and (75.36,493.53) .. (81.35,493.53) .. controls (87.35,493.53) and (92.21,498.22) .. (92.21,504.01) .. controls (92.21,509.81) and (87.35,514.5) .. (81.35,514.5) .. controls (75.36,514.5) and (70.5,509.81) .. (70.5,504.01) – cycle ;

(96.21,505) – (125.5,505) ;
[shift=(128.5,505), rotate = 180] [fill=rgb, 255:red, 0; green, 0; blue, 0   ][line width=0.08]  [draw opacity=0] (8.93,-4.29) – (0,0) – (8.93,4.29) – cycle    ;

(263,505) – (345,505) ;
[shift=(350,505), rotate = 180] [fill=rgb, 255:red, 0; green, 0; blue, 0   ][line width=0.08]  [draw opacity=0] (8.93,-4.29) – (0,0) – (8.93,4.29) – cycle    ;

(82.5,515) – (82.5,530) ;
[shift=(82.5,514), rotate = 90] [fill=rgb, 255:red, 0; green, 0; blue, 0   ][line width=0.08]  [draw opacity=0] (8.93,-4.29) – (0,0) – (8.93,4.29) – cycle    ;

(527.5,505.01) .. controls (527.5,499.22) and (532.36,494.53) .. (538.35,494.53) .. controls (544.35,494.53) and (549.21,499.22) .. (549.21,505.01) .. controls (549.21,510.81) and (544.35,515.5) .. (538.35,515.5) .. controls (532.36,515.5) and (527.5,510.81) .. (527.5,505.01) – cycle ;

(538,515) – (538,530) ;

(425,505) –(520,505) ;
[shift=(525,505), rotate = 180] [fill=rgb, 255:red, 0; green, 0; blue, 0   ][line width=0.08]  [draw opacity=0] (8.93,-4.29) – (0,0) – (8.93,4.29) – cycle    ;

[fill=rgb, 255:red, 184; green, 233; blue, 134   ,fill opacity=1 ][dash pattern=on 2pt off 3pt] (180,496) – (195,496) – (195,515) – (180,515) – cycle ;

(82,530) – (538.5,530) ;

[fill=rgb, 255:red, 184; green, 233; blue, 134   ,fill opacity=1 ][dash pattern=on 2pt off 3pt] (406,496) – (421,496) – (421,515) – (406,515) – cycle ;

[color=rgb, 255:red, 65; green, 117; blue, 5   ,draw opacity=1 ]   (187.5,478) – (187.5,495) ;

[color=rgb, 255:red, 65; green, 117; blue, 5   ,draw opacity=1 ]   (187,478)– (414,478)  ;

[color=rgb, 255:red, 65; green, 117; blue, 5   ,draw opacity=1 ]   (413.5,478) – (413.5,495) ;
[shift=(413.5,500), rotate = 270] [fill=rgb, 255:red, 65; green, 117; blue, 5   ,fill opacity=1 ][line width=0.08]  [draw opacity=0] (8.93,-4.29) – (0,0) – (8.93,4.29) – cycle    ;

(75.36,500) node [anchor=north west][inner sep=0.75pt]    $\pi $;

(124,496) node [anchor=north west][inner sep=0.75pt]    $\calD=\{\bs_{t},\ba_{t} ,\bs_{t+1} ,r_{t+1}\}$;

(95,483) node [anchor=north west][inner sep=0.75pt]   [align=left] data;

(261.66,483) node [anchor=north west][inner sep=0.75pt]   [align=left] action gradient;

(300,513) node [anchor=north west][inner sep=0.75pt]    $\pi \gets \pi ^{'}$;

(530,495) node [anchor=north west][inner sep=0.75pt]    $\pi^{'}$;

(426,483) node [anchor=north west][inner sep=0.75pt]   [align=left] diffusion policy;

(346,495) node [anchor=north west][inner sep=0.75pt]    $\calD^{'}=\{\bs_{t},\ba_{t}\}$;

(240,460) node [anchor=north west][inner sep=0.75pt]    $ \ba_{t}+\eta \nabla_{\ba} Q_{\pi}( \bs_t,\ba_t)\rightarrow\ba_{t}$;

Framework of DIPO: Implementation for Model-free Online RL with DIffusion POlicy.

§ REINFORCEMENT LEARNING

Reinforcement learning (RL)[Sutton and Barto, 2018] is formulated as Markov decision process $\mathcal{M}=(\mathcal{S},\mathcal{A},\Pro(\cdot),{r},\gamma,d_{0})$,
where $\mathcal{S}$ is the state space; $\mathcal{A}\subset\R^{p}$ is the continuous action space;
$\Pro(\bs^{'}|\bs,\ba)$ is the probability of state transition from $\bs$ to $\bs^{'}$ after playing $\ba$;
$r(\bs'|\bs,\ba)$ denotes the reward that the agent observes when the state transition from $\bs$ to $\bs^{'}$ after playing $\ba$;
$\gamma\in(0,1)$ is the discounted factor, and $d_{0}(\cdot)$ is the initial state distribution.
A policy $\pi$ is a probability distribution defined on $\mathcal{S}\times\mathcal{A}$, and $\pi(\ba|\bs)$ denotes the probability of playing $\ba$ in state $\bs$.
Let $\{\bs_{t},\ba_{t},\bs_{t+1},r(\bs_{t+1}|\bs_{t},\ba_{t})\}_{t\ge0}\sim\pi$ be the trajectory sampled by the policy $\pi$, where $\bs_{0}\sim d_{0}(\cdot)$, $\ba_{t}\sim\pi(\cdot|\bs_{t})$, $\bs_{t+1}\sim \Pro(\cdot|\bs_t,\ba_t)$.
The goal of RL is to find a policy $\pi$ such that
\pi_{\star}=:\arg\max_{\pi} \E_{\pi}\left[\sum_{t=0}^{\infty}\gamma^t r(\bs_{t+1}|\bs_t,\ba_t)\right].

§ MOTIVATION: A VIEW FROM POLICY REPRESENTATION

In this section, we clarify our motivation from the view of policy representation: diffusion model is powerful to policy representation, which leads to sufficient exploration and better reward performance.

§.§ Policy Representation for Reinforcement Learning

Value function and parametric function based are the main two approaches to represent policies, while diffusion policy expresses a policy via a stochastic process (shown in Figure <ref>) that is essentially difficult to the previous representation.
In this section, we will clarify this view.
Additionally, we will provide an empirical verification with a numerical experiment.

§.§.§ Policy Representation via Value Function
A typical way to represent policy is $\epsilon$-greedy policy [Sutton and Barto, 1998] or energy-based policy [Sallans and Hinton, 2004, Peters et al., 2010],

max_^'∈ Q_π(,^')     w.p.  1-ϵ;

randomly play ∈    w.p.  ϵ;

\[Q_{\pi}(\bs,\ba)=:\E_{\pi}\left[\sum_{t=0}^{\infty}\gamma^t r(\bs_{t+1}|\bs_t,\ba_t)|\bs_{0}=\bs,\ba_{0}=\ba\right],\]

the normalization term $Z_{\pi}(\bs)=\int_{\R^{p}}\exp\left\{Q_{\pi}(\bs,\ba)\right\}\dd \ba$, and “w.p.” is short for “with probability”.
The representation (<ref>) illustrates a connection between policy and value function, which is widely used in value-based methods
(e.g., SASRA [Rummery and Niranjan, 1994],
Q-Learning [Watkins, 1989],
DQN [Mnih et al., 2015]) and energy-based methods (e.g.,
SQL [Schulman et al., 2017, Haarnoja et al., 2017, Haarnoja et al., 2018],
SAC [Haarnoja et al., 2018]).

§.§.§ Policy Representation via Parametric Function
Instead of consulting a value function, the parametric policy is to represent a policy by a parametric function (e.g., neural networks), denoted as $\pi_{\btheta}$, where $\btheta$ is the parameter. Policy gradient theorem [Sutton et al., 1999, Silver et al., 2014] plays a center role to learn $\btheta$, which is fundamental in modern RL (e.g., TRPO [Schulman et al., 2015], DDPG [Lillicrap et al., 2016], PPO [Schulman et al., 2017], IMPALA [Espeholt et al., 2018], et al).

§.§.§ Policy Representation via Stochastic Process
It is different from both value-based and parametric policy representation; the diffusion policy (see Figure <ref>) generates an action via a stochastic process, which is a fresh view for the RL community.
The diffusion model with RL first appears in [Janner et al., 2022], where it proposes the diffuser that plans by iteratively refining trajectories, which is an essential offline RL method. [Ajay et al., 2023, Reuss et al., 2023] model a policy as a return conditional diffusion model, [Chen et al., 2023, Wang et al., 2023, Chi et al., 2023] consider to generate actions via diffusion model. The above methods are all to solve offline RL problems. To the best of our knowledge, our proposed method is the first diffusion approach to online model-free reinforcement learning.

every picture/.style=line width=0.75pt

[color=rgb, 255:red, 0; green, 0; blue, 255  ,draw opacity=1 ][line width=0.75]  (381.25,105.49) .. controls (381.25,104.41) and (380.71,103.87) .. (379.63,103.87) – (379.63,103.87) .. controls (378.08,103.87) and (377.31,103.33) .. (377.31,102.25) .. controls (377.31,103.33) and (376.54,103.87) .. (375,103.87)(375.69,103.87) – (375,103.87) .. controls (373.92,103.87) and (373.38,104.41) .. (373.38,105.49) ;

[line width=1.5]  (325,106.38) – (496,106.38)(411.09,6.22) – (411.09,116.11) (489,101.38) – (496,106.38) – (489,111.38) (406.09,13.22) – (411.09,6.22) – (416.09,13.22)  ;

[color=rgb, 255:red, 255; green, 0; blue, 0   ,draw opacity=1 ][line width=1.5]    (327.9,104.36) .. controls (389.49,104.94) and (390.21,24.12) .. (411.95,23.83) .. controls (418.9,23.73) and (423.56,32.06) .. (428.67,43.43) .. controls (439.53,67.6) and (452.43,105.52) .. (493.83,105.52) ;

[color=rgb, 255:red, 255; green, 0; blue, 0   ,draw opacity=1 ][line width=0.4]  [dash pattern=on 0.84pt off 2.51pt]  (429.03,106.04) – (428.67,43.43) ;

[color=rgb, 255:red, 255; green, 0; blue, 0   ,draw opacity=1 ] [line width=0.4]  [dash pattern=on 0.84pt off 2.51pt]  (394.09,105.31) – (394.09,42.92) ;

[color=rgb, 255:red, 255; green, 0; blue, 0   ,draw opacity=1 ] (394.09,106.41) .. controls (394.09,111.08) and (396.42,113.41) .. (401.09,113.41) – (401.41,113.41) .. controls (408.08,113.41) and (411.41,115.74) .. (411.41,120.41) .. controls (411.41,115.74) and (414.74,113.41) .. (421.41,113.41)(418.41,113.41) – (422.03,113.41) .. controls (426.7,113.41) and (429.03,111.08) .. (429.03,106.41) ;

[color=rgb, 255:red, 0; green, 0; blue, 255  ,draw opacity=1 ][line width=1.5]    (332.25,105.23) .. controls (378.62,105.23) and (366.3,44.61) .. (377.17,44.61) .. controls (388.04,44.61) and (383.69,68.86) .. (392.39,68.28) .. controls (401.08,67.7) and (401.08,31.33) .. (411.95,31.33) .. controls (422.82,31.33) and (416.77,74.96) .. (427.17,74.63) .. controls (437.56,74.3) and (437.31,59.04) .. (449.63,58.47) .. controls (461.94,57.89) and (461.94,103.5) .. (489.48,104.65) ;

[color=rgb, 255:red, 0; green, 0; blue, 255  ,draw opacity=1 ][line width=0.4]  [dash pattern=on 0.84pt off 2.51pt]  (444.56,105.23) – (444.56,59.62) ;

[color=rgb, 255:red, 0; green, 0; blue, 255  ,draw opacity=1 ][line width=0.4]  [dash pattern=on 0.84pt off 2.51pt]  (455.42,105.23) – (454.13,67.12) – (455,61.22) ;

[color=rgb, 255:red, 0; green, 0; blue, 255  ,draw opacity=1 ][line width=0.4]  [dash pattern=on 0.84pt off 2.51pt]  (403.98,105.8) – (403.25,43.46) ;

[color=rgb, 255:red, 0; green, 0; blue, 255  ,draw opacity=1 ][line width=0.4]  [dash pattern=on 0.84pt off 2.51pt]  (417.75,105.8) – (417.75,44.61) ;

[color=rgb, 255:red, 0; green, 0; blue, 255  ,draw opacity=1 ][line width=0.4]  [dash pattern=on 0.84pt off 2.51pt]  (372.82,105.8) – (372.82,51.25) ;

[color=rgb, 255:red, 0; green, 0; blue, 255  ,draw opacity=1 ][line width=0.4]  [dash pattern=on 0.84pt off 2.51pt]  (381.52,105.8) – (381.25,46.97) ;

[color=rgb, 255:red, 0; green, 0; blue, 255  ,draw opacity=1 ][line width=0.4]  (417.81,105.74) .. controls (417.81,103.89) and (416.88,102.96) .. (415.03,102.96) – (415.03,102.96) .. controls (412.38,102.96) and (411.06,102.03) .. (411.06,100.18) .. controls (411.06,102.03) and (409.74,102.96) .. (407.09,102.96)(408.28,102.96) – (407.09,102.96) .. controls (405.24,102.96) and (404.31,103.89) .. (404.31,105.74) ;

[color=rgb, 255:red, 0; green, 0; blue, 255  ,draw opacity=1 ][line width=0.75]  (454.94,105.99) .. controls (454.94,104.52) and (454.21,103.79) .. (452.74,103.79) – (452.74,103.79) .. controls (450.64,103.79) and (449.59,103.06) .. (449.59,101.59) .. controls (449.59,103.06) and (448.54,103.79) .. (446.45,103.79)(447.39,103.79) – (446.45,103.79) .. controls (444.98,103.79) and (444.25,104.52) .. (444.25,105.99) ;

(438.34,87) node [anchor=north west][inner sep=0.75pt]      $A_3$;

(364.09,87) node [anchor=north west][inner sep=0.75pt]      $A_1$;

(398.41,87) node [anchor=north west][inner sep=0.75pt]     $A_2$;

(422,108) node [anchor=north west][inner sep=0.75pt]    $A$;

(365,5) node [anchor=north west][inner sep=0.75pt]    $\pi ( \cdot |\bs)$;

(466.63,108) node [anchor=north west][inner sep=0.75pt]    $\calA$;

(447,31) node [anchor=north west][inner sep=0.75pt]   [align=left] unimodal;

[color=rgb, 255:red, 0; green,0; blue, 255  ,draw opacity=1 ]   (347,48.72) – (365.25,58.75) ;
[shift=(367,59.72), rotate = 208.81] [color=rgb, 255:red, 0; green, 0; blue, 255   ,draw opacity=1 ][line width=0.75]    (10.93,-3.29) .. controls (6.95,-1.4) and (3.31,-0.3) .. (0,0) .. controls (3.31,0.3) and (6.95,1.4) .. (10.93,3.29)   ;

[color=rgb, 255:red, 255; green, 0; blue, 0   ,draw opacity=1 ]   (450,45.72) – (437.22,56.17) ;
[shift=(435.67,57.43), rotate = 320.75] [color=rgb, 255:red, 255; green, 0; blue, 0  ,draw opacity=1 ][line width=0.75]    (10.93,-3.29) .. controls (6.95,-1.4) and (3.31,-0.3) .. (0,0) .. controls (3.31,0.3) and (6.95,1.4) .. (10.93,3.29)   ;

(324,30) node [anchor=north west][inner sep=0.75pt]   [align=left] multimodal;

Unimodal Distribution vs Multimodal Distribution.

[Diffusion Policy]

Policy representation comparison of different policies on multimodal environment.

§.§ Diffusion Model is Powerful to Policy Representation

This section presents the diffusion model is powerful to represent a complex policy distribution.
by two following aspects:
1) fitting a multimodal policy distribution is efficient for exploration;
2) empirical verification with a numerical experiment.

The Gaussian policy is widely used in RL, which is a unimodal distribution, and it plays actions around the region of its mean center with a higher probability, i.e., the red region $A$ in Figure <ref>.
The unimodal policy weakens the expressiveness of complicated policy and decays the agent's ability to explore the environment.

While for a multimodal policy, it plays actions among the different regions: $A_1$$\cup$$A_2$$\cup$$A_3$.
Compared to the unimodal policy, the multimodal policy is powerful to explore the unknown world, making the agent understand the environment efficiently and make a more reasonable decision.

We compare the ability of policy representation among SAC, TD3 [Fujimoto et al., 2018], PPO and diffusion policy on the “multi-goal” environment [Haarnoja et al., 2017] (see Figure <ref>),

where the $x$-axis and $y$-axis are 2D states, the four red dots denote the states of the goal at $(0,5)$, $(0,-5)$, $(5,0)$ and $(-5,0)$ symmetrically.

A reasonable policy should be able to take actions uniformly to those four goal positions with the same probability,
which characters the capacity of exploration of a policy to understand the environment.

In Figure <ref>, the red arrowheads represent the directions of actions, and the length of the red arrowheads represents the size of the actions.
Results show that diffusion policy accurately captures a multimodal distribution landscape, while both SAC, TD3, and PPO are not well suited to capture such a multimodality.
From the distribution of action direction and length, we also know the diffusion policy keeps a more gradual and steady action size than the SAC, TD3, and PPO to fit the multimodal distribution. For more details about 2D/3D plots, environment, comparisons, and discussions, please refer to Appendix <ref>.

§ DIFFUSION POLICY

In this section, we present the details of diffusion policy from the following three aspects: its stochastic dynamic equation (shown in Figure <ref>), discretization implementation, and finite-time analysis of its performance for the policy representation.

§.§ Stochastic Dynamics of Diffusion Policy

Recall Figure <ref>, we know diffusion policy contains two processes: forward process and reverse process. We present its dynamic in this section.

Forward Process. To simplify the expression, we only consider $g(t)=\sqrt{2}$, which is parallel to the general setting in Figure <ref>.
For any given state $\bs$, the forward process produces a sequence $\{{(\bara_t|\bs)}\}_{t=0:T}$ that starting with $\bara_{0}\sim\pi(\cdot|\bs)$,
and it follows the Ornstein-Uhlenbeck process (also known as Ornstein-Uhlenbeck SDE),

Let $\bara_t\sim \barpi_t(\cdot|\bs)$ be the evolution distribution along the Ornstein-Uhlenbeck flow (<ref>).
According to Proposition <ref> (see Appendix <ref>), we know the conditional distribution of $\bara_t|\bara_0$ is Gaussian,

That implies the forward process (<ref>) transforms policy $\pi(\cdot|\bs)$ to the Gaussian noise $\calN(\bm{0},\bI)$.

Reverse Process. For any given state $\bs$, if we reverse the stochastic process $\{{(\bara_{t}|\bs)}\}_{t=0:T}$,

then we obtain a process that transforms noise into the policy $\pi(\cdot|\bs)$. Concretely, we model the policy as the process $\{(\tildea_t|\bs)\}_{t=0:T}$ according to the next Ornstein-Uhlenbeck process (running forward in time),

where $p_{t}(\cdot)$ is the probability density function of $\barpi_t(\cdot|\bs)$.
Furthermore, according to
[Anderson, 1982], with an initial action $\tildea_{0}\sim\barpi_{T}(\cdot|\bs)$,
the reverse process $\{\tildea_{t}\}_{t=0:T}$ shares the same distribution as the time-reversed version of the forward process $\{\bara_{T-t}\}_{t=0:T}$. That also implies for all $t=0,1,\cdots,T$,

_t(·|)=_T-t(·|),   if _0∼_T(·|).

Score Matching.
The score function $\bnabla \log p_{T-t}(\cdot)$ defined in (<ref>) is not explicit, we consider an estimator ${\bm{\hat \mathbf{S}}}(\cdot,\bs,T-t)$ to approximate the score function at a given state $\bs$. We consider the next problem,

S(·,,T-t)=:   min_ŝ(·)∈_∼_t(·|)[

(<ref>)=   min_ŝ(·)∈_∼_t(·|)[

where $\calF$ is the collection of function approximators (e.g., neural networks).
We will provide the detailed implementations with a parametric function approximation later; please refer to Section <ref> or Appendix <ref>.

§.§ Exponential Integrator Discretization for Diffusion Policy

In this section, we consider the implementation of the reverse process (<ref>) with exponential integrator discretization [Zhang, 2022, Lee et al., 2023].

Let $h>0$ be the step-size, assume reverse length $K=\frac{T}{h}\in\N$, and $t_{k}=:hk$, $k=0,1,\cdots,K$.
Then we give a partition on the interval $[0,T]$ as follows, $0=t_{0}<t_{1}<\cdots<t_{k}<t_{k+1}<\cdots<t_{K}=T$.
Furthermore, we take the discretization to the reverse process (<ref>) according to the following equation,

_t=(_t+2 S(_t_k,,T-t_k))t+√(2)_t,  t∈[t_k,t_k+1],

where it runs initially from $\hata_0\sim\calN(\bm{0},\bI)$.

By Itô integration to the two sizes of (<ref>) on the $k$-th interval $[t_{k},t_{k+1}]$, we obtain the exact solution of the SDE (<ref>), for each $k=0,1,2,\cdots,K-1$,

_t_k+1=   e^h_t_k+(e^h-1)(_t_k+2 S(_t_k,,T-t_k))+√(e^2h-1)_k, _k∼(0,).

For the derivation from the SDE (<ref>) to the iteraion (<ref>), please refer to Appendix <ref>, and
we have shown the implementation in Algorithm <ref>.

Diffusion Policy with Exponential Integrator Discretization to Approximate $\pi(\cdot|\bs)$

[1]
input: state $\bs$, horizon $T$, reverse length $K$, step-size $h=\frac{T}{K}$, score estimators $\hat{\mathbf{S}}(\cdot,\bs,T-t)$;
initialization: a random action $\hata_{0}\sim\calN(\bm{0},\bI)$;
for $k=0,1,\cdots,K-1$ do

a random $\bz_{k}\sim\calN(\bm{0},\bI)$, set $t_k=hk$;
$\hata_{t_{k+1}}=\mathrm{e}^{h}\hata_{t_{k}}+\left(\mathrm{e}^{h}-1\right)\left(\hata_{t_{k}}+2 \bm{\hat \mathbf{S}}(\hata_{t_k},\bs,T-t_{k})\right)+\sqrt{\mathrm{e}^{2h}-1}\bz_{k}$;

output: $\hata_{t_{K}}$;

§.§ Convergence Analysis of Diffusion Policy

In this section, we present the convergence analysis of diffusion policy, we need the following notations and assumptions before we further analyze.
Let $\rho(\bx)$ and $\mu(\bx)$ be two smooth probability density functions on the space $\R^{p}$,
the Kullback–Leibler (KL) divergence and relative Fisher information (FI) from $\mu(\bx)$ to $\rho(\bx)$ are defined as follows,

\[
\KL(\rho\|\mu)=\bigintsss_{\R^{p}}\rho(\bx)\log\dfrac{\rho(\bx)}{\mu(\bx)}\dd \bx, ~\FI(\rho\|\mu)=\bigintsss_{\R^{p}}\rho(\bx)\left\|\bnabla\log\left(\dfrac{\rho(\bx)}{\mu(\bx)}\right)\right\|_{2}^{2}\dd \bx.
\]

[Lipschitz Score Estimator and Policy]

The score estimator is $L_s$-Lipschitz over action space $\calA$, and the policy $\pi(\cdot|\bs)$ is $L_{p}$-Lipschitz over action space $\calA$, i.e., for any $\ba$, $\ba^{'}\in\calA$, the following holds,

\[
\|\bm{\hat \mathbf{S}}(\ba,\bs,t)-\bm{\hat \mathbf{S}}(\ba^{'},\bs,t)\|\leq L_{s}\|\ba-\ba^{'}\|, ~\|\bnabla\log\pi(\ba|\bs)-\bnabla\log\pi(\ba^{'}|\bs)\|\leq L_{p}\|\ba-\ba^{'}\|.
\]

[Policy with $\nu$-LSI Setting]

The policy $\pi(\cdot|\bs)$ satisfies $\nu$-Log-Sobolev inequality (LSI) that defined as follows,
there exists constant $\nu>0$, for any probability distribution $\mu(\bx)$ such that

\[
\KL(\mu\|\pi)\leq\frac{1}{2\nu}\FI(\mu\|\pi).
\]

Assumption <ref> is a standard setting for Langevin-based algorithms (e.g., [Wibisono and Yang, 2022, Vempala and Wibisono, 2019]), and we extend it with RL notations.
Assumption <ref> presents the policy distribution class that we are concerned,
which contains many complex distributions that are not restricted to be log-concave, e.g. any slightly smoothed bound distribution admits the condition (see <cit.>).

For a given state $\bs$, let $\{\barpi_{t}(\cdot|\bs)\}_{t=0:T}$ and $\{\tildepi_{t}(\cdot|\bs)\}_{t=0:T}$ be the distributions along the flow (<ref>) and (<ref>) correspondingly, where $\{\barpi_{t}(\cdot|\bs)\}_{t=0:T}$ starts at $\barpi_{0}(\cdot|\bs)=\pi(\cdot|\bs)$ and
$\{\tildepi_{t}(\cdot|\bs)\}_{t=0:T}$ starts at $\tildepi_{0}(\cdot|\bs)=\barpi_{T}(\cdot|\bs)$.
Let $\hatpi_{k}(\cdot|\bs)$ be the distribution of the iteration (<ref>) at the $k$-th time $t_{k}=hk$, i.e.,
$\hata_{t_k}\sim \hatpi_{k}(\cdot|\bs)$ denotes the diffusion policy (see Algorithm <ref>) at the time $t_{k}=hk$.
Let $\{\hatpi_{k}(\cdot|\bs)\}_{k=0:K}$ be starting at $\hatpi_{0}(\cdot|\bs)=\calN(\bm{0},\bI)$, under Assumption <ref> and <ref>, let the reverse length
K\ge T\cdot\max\left\{\tau^{-1}_{0},\mathtt{T}^{-1}_{0},12L_{s},\nu\right\}$, where constants $\tau_{0}$ and $\mathtt{T}_{0}$ will be special later.
Then the KL-divergence between diffusion policy $\hatpi_{K}(\cdot|\bs)$ and input policy $\pi(\cdot|\bs)$ is upper-bounded as follows,

(_K(·|)π(·|))≤e^-9/4νhK((0,)π(·|))_convergence of forward process (<ref>)+(64pL_s√(5/ν))h_errors from discretization (<ref>)+203ϵ_score,

where ϵ_score=sup_(k,t)∈[K]×[t_k,t_k+1]{log_∼_t(·|)[expS(,,T-hk)-log_t(|)_2^2]}_errors from score matching (<ref>).

Theorem <ref> illustrates that the errors involve the following three terms.
The first error involves $\KL \big(\calN(\bm{0},\bI)\|\pi(\cdot|\bs)\big)$ that represents how close the distribution of the input policy $\pi$ is to the standard Gaussian noise, which is bounded by the exponential convergence rate of
Ornstein-Uhlenbeck flow (<ref>) [Bakry et al., 2014, Wibisono and Jog, 2018, Chen et al., 2023].
The second error is sourced from exponential integrator discretization implementation (<ref>), which scales with the discretization step-size $h$. The discretization error term implies a first-order convergence rate with respect to the discretization step-size $h$ and scales polynomially on other parameters.
The third error is sourced from score matching (<ref>), which represents how close the score estimator $\hat{\mathbf{S}}$ is to the score function $\bnabla\log p_{T-t}(\cdot)$ defined in (<ref>).
That implies for the practical implementation, the error from score matching could be sufficiently small if we find a good score estimator $\hat{\mathbf{S}}$.

Furthermore, for any $\epsilon>0$, if we find a good score estimator that makes the score matching error satisfy $\epsilon_{\mathrm{score}}<\frac{1}{20}\epsilon$, the step-size $h=\calO\left(\frac{\epsilon\sqrt{\nu}}{pL_{s}}\right)$, and reverse length $K=\frac{9}{4\nu h}\log\frac{3\KL (\calN(\bm{0},\bI)\|\pi(\cdot|\bs))}{\epsilon}$, then Theorem <ref> implies the output of diffusion policy $(\hatpi_{K}(\cdot|\bs)$ makes a sufficient close to the input policy $\pi(\cdot|\bs)$ with the measurement by $\KL(\hatpi_{K}(\cdot|\bs)\|\pi(\cdot|\bs))\leq\epsilon$.

§ DIPO: IMPLEMENTATION OF DIFFUSION POLICY FOR MODEL-FREE ONLINE RL

In this section, we present the details of DIPO, which is an implementation of  DIffusion POlicy for model-free reinforcement learning. According to Theorem <ref>, diffusion policy only fits the current policy $\pi$ that generates the training data (denoted as $\calD$), but it does not improve the policy $\pi$. It is different from traditional policy-based RL algorithms, we can not improve a policy according to the policy gradient theorem since diffusion policy is not a parametric function but learns a policy via a stochastic process. Thus, we need a new way to implement policy improvement, which is nontrivial.
We have presented the framework of DIPO in Figure <ref>, and shown the key steps of DIPO in Algorithm <ref>. For the detailed implementation, please refer to Algorithm <ref> (see Appendix <ref>).

§.§ Training Loss of DIPO

It is intractable to directly apply the formulation (<ref>) to estimate the score function since $\bnabla \log p_{t}(\cdot)=\bnabla \log \barpi_{t}(\cdot|\bs)$ is unknown, which is sourced from the initial distribution $\bara_{0}\sim\pi(\cdot|\bs)$ is unknown. According to denoising score matching [Vincent, 2011, Hyvärinen, 2005], a practical way is to solve the next optimization problem (<ref>). For any given $\bs\in\calS$,

where $\omega(t):[0,T]\rightarrow \R_{+}$ is a positive weighting function; $\varphi_{t}(\bara_t|\bara_0)=\calN\left(\mathrm{e}^{-t}\bara_0,\left(1-\mathrm{e}^{-2t}\right)\bI\right)$ denotes the transition kernel of the forward process (<ref>); $\E_{\bara_t|\bara_0}[\cdot]$ denotes the expectation with respect to $\varphi_{t}(\bara_t|\bara_0)$; and $\bphi$ is the parameter needed to be learned.
Then, according to Theorem <ref> (see Appendix <ref>), we rewrite the objective (<ref>) as follows,

()   =_k∼({1,2,⋯,K}),∼(0,),(,)∼[-_(√(α̅_k)+√(1-α̅_k),,k)_2^2],

where $\calU(\cdot)$ denotes uniform distribution,

\[\bm{\epsilon}_{\bphi}\left(\cdot,\cdot,k\right)=-\sqrt{1-\bar{\alpha}_{k}}\hat{\bm{s}}_{\bphi}\left(\cdot,\cdot,T-t_k\right),\]
and $\bar{\alpha}_k$ will be special.
The objective (<ref>) provides a way to learn $\bphi$ from samples; see line 14-16 in Algorithm <ref>.

§.§ Playing Action of DIPO

Replacing the score estimator $\hat{\mathbf{S}}$ (defined in Algorithm <ref>) according to $\hat{\bm{\epsilon}}_{\bphi}$, after some algebras (see Appendix <ref>), we rewrite diffusion policy (i.e., Algorithm <ref>) as follows,

where $k=0,1,\cdots,K-1$ runs forward in time, the noise $\bz_{k}\sim\calN(\bm{0},\bI)$. The agent plays the last (output) action $\hata_{K}$.

(DIPO) Model-Free Reinforcement Learning with DIffusion POlicy

[1]
initialize ${\bphi}$, critic network $Q_{\bpsi}$; $\{\alpha_i\}_{i=0}^{K}$; $\bar{\alpha}_{k}=\prod_{i=1}^{k}\alpha_{i}$; step-size $\eta$;

dataset $\calD\gets\emptyset$; initialize $\bs_{0}\sim d_{0}(\cdot)$;

for $t=0,1,\cdots,T$ do
play $\ba_{t}$ follows (<ref>);
$\bs_{t+1}\sim \Pro(\cdot|\bs_{t},\ba_{t})$;

repeat $N$ times
sample $(\bs_{t},\ba_{t},\bs_{t+1},r(\bs_{t+1}|\bs_{t},\ba_{t}))\sim\calD$ i.i.d;
take gradient descent on $\ell_{\mathrm{Q}}(\bpsi)$ (<ref>);

for $t=0,1,\cdots,T$ do
replace each action $\ba_{t}\in\calD$ follows

repeat $N$ times
sample $(\bs,\ba)$ from $\calD$ i.i.d, sample index $k\sim \calU(\{1,\cdots,K\})$, $\bz\sim \calN(\bm{0},\bI)$;
take gradient decent on the loss $\ell_{\mathrm{d}}(\bphi)=\|\bz-\bepsilon_{\bphi}(\sqrt{\bar\alpha_k}\ba+\sqrt{1-\bar{\alpha}_k}\bz,\bs,k)\|_{2}^{2}$;
the policy performs well in the environment.

§.§ Policy Improvement of DIPO

According to (<ref>), we know that only the state-action pairs $(\bs,\ba)\in\calD$ are used to learn a policy.
That inspires us that if we design a method that transforms a given pair $(\bs,\ba)\in\calD$ to be a  “better” pair, then we use the “better” pair to learn a new diffusion policy $\pi^{'}$,  then $\pi^{'}\succeq\pi$.
About  “better” state-action pair should maintain a higher reward performance than the originally given pair $(\bs,\ba)\in\calD$.
We break our key idea into two steps:
1) first, we regard the reward performance as a function with respect to actions, $J_{\pi}(\ba)=\E_{\bs\sim d_{0}(\cdot)}[Q_{\pi}(\bs,\ba)]$, which quantifies how the action $\ba$ affects the performance;
2) then, we update all the actions $\ba\in\calD$ through the direction $\bnabla_{\ba}J_{\pi}(\ba)$ by gradient ascent method:

where $\eta>0$ is step-size, and we call $\nabla_{\ba}J_{\pi}(\ba)$ as action gradient.

To implement (<ref>) from samples, we need a neural network $Q_{\bpsi}$ to estimate $Q_{\pi}$.
Recall $\{\bs_{t},\ba_{t},\bs_{t+1},r(\bs_{t+1}|\bs_{t},\ba_{t})\}_{t\ge0}\sim\pi$, we train the parameter $\bpsi$ by minimizing the following Bellman residual error,

Finally, we consider each pair $(\bs_{t},\ba_{t})\in\calD$, and replace the action $\ba_{t}\in\calD$ as follows,

§ RELATED WORK

Due to the diffusion model being a fast-growing field, this section only presents the work that relates to reinforcement learning, a recent work [Yang et al., 2022] provides a comprehensive survey on the diffusion model.
In this section, first, we review recent advances in diffusion models with reinforcement learning.
Then, we review the generative models for reinforcement learning.

§.§ Diffusion Models for Reinforcement Learning

The diffusion model with RL first appears in [Janner et al., 2022], where it proposes the diffuser that plans by iteratively refining trajectories, which is an essential offline RL method. Later [Ajay et al., 2023] model a policy as a return conditional diffusion model, [Chen et al., 2023, Wang et al., 2023, Chi et al., 2023] consider to generate actions via diffusion model. SE(3)-diffusion fields [Urain et al., 2023] consider learning data-driven SE(3) cost functions as diffusion models.
[Pearce et al., 2023] model the imitating human behavior with diffusion models.
[Reuss et al., 2023] propose score-based diffusion policies for the goal-conditioned imitation learning problems.
ReorientDiff [Mishra and Chen, 2023] presents a reorientation planning method that utilizes a diffusion model-based approach.
StructDiffusion [Liu et al., 2022] is an object-centric transformer with a diffusion model,  based on high-level language goals, which constructs structures out of a single
RGB-D image.
[Brehmer et al., 2023] propose an equivariant diffuser for generating interactions (EDGI), which trains a diffusion model on an offline trajectory dataset, where EDGI learns a world model and planning in it as a conditional generative modeling problem follows the diffuser [Janner et al., 2022].
DALL-E-Bot [Kapelyukh et al., 2022] explores the web-scale image diffusion models for robotics.
AdaptDiffuser [Liang et al., 2023] is an evolutionary planning algorithm with diffusion, which is adapted to unseen tasks.

The above methods are all to solve offline RL problems, to the best of our knowledge, the proposed DIPO is the first diffusion approach to solve online model-free RL problems.
The action gradient plays a critical way to implement DIPO, which never appears in existing RL literature.
In fact, the proposed DIPO shown in Figure <ref> is a general training framework for RL, where we can replace the diffusion policy with any function fitter (e.g., MLP or VAE).

§.§ Generative Models for Policy Learning

In this section, we mainly review the generative models, including VAE [Kingma and Welling, 2013], GAN [Goodfellow et al., 2020], Flow  [Rezende and Mohamed, 2015],
and GFlowNet [Bengio et al., 2021, Bengio et al., 2021] for policy learning.
Generative models are mainly used in cloning diverse behaviors [Pomerleau, 1988], imitation learning [Osa et al., 2018], goal-conditioned imitation learning [Argall et al., 2009], or offline RL [Levine et al., 2020], a recent work [Yang et al., 2023] provides a foundation presentation for the generative models for policy learning.

VAE for Policy Learning. [Lynch et al., 2020, Ajay et al., 2021] have directly applied  auto-encoding variational Bayes (VAE) [Kingma and Welling, 2013] and VQ-VAE [Van Den Oord et al., 2017]
model behavioral priors.
[Mandlekar et al., 2020] design the low-level policy that is conditioned on latent from the CVAE.
[Pertsch et al., 2021] joint the representation of skill embedding and skill prior via a deep latent variable model.
[Mees et al., 2022, Rosete-Beas et al., 2023] consider seq2seq CVAE [Lynch et al., 2020, Wang et al., 2022] to model of conditioning the action decoder on the latent plan allows the policy to use the entirety of its capacity for learning unimodal behavior.

GAN for Imitation Learning. GAIL [Ho and Ermon, 2016] considers the Generative Adversarial Networks (GANs)  [Goodfellow et al., 2020] to imitation learning.
These methods consist of a generator and a discriminator, where the generator policy learns to imitate the experts'  behaviors, and the discriminator distinguishes between real and fake trajectories, which models the imitation learning as a distribution matching problem between the expert policy's state-action distribution and the agent’s policy [Fu et al., 2018, Wang et al., 2021]. For several advanced results and applications, please refer to [Chen et al., 2023, Deka et al., 2023, 66, Taranovic et al., 2023].

Flow and GFlowNet Model for Policy Learning. [Singh et al., 2020] consider normalizing flows [Rezende and Mohamed, 2015] for the multi-task RL tasks.
[Li et al., 2023] propose diverse policy optimization, which consider the GFlowNet [Bengio et al., 2021, Bengio et al., 2021] for the structured action spaces.
[Li et al., 2023] propose CFlowNets that combines GFlowNet with continuous control.
Stochastic GFlowNet [Pan et al., 2023] learns a model of the environment to capture the stochasticity of state transitions. [Malkin et al., 2022] consider training a GFlowNet with trajectory balance.

Other Methods. Decision Transformer (DT) [Chen et al., 2021] model the offline RL tasks as a conditional sequence problem, which does not learn a policy follows the traditional methods (e.g., [Sutton, 1988, Sutton and Barto, 1998]).
Those methods with DT belong to the task-agnostic behavior learning methods, which is an active direction in policy learning (e,g.,  [Cui et al., 2023, Brohan et al., 2022, Zheng et al., 2022, Konan et al., 2023, Kim et al., 2023]).
Energy-based models [LeCun et al., 2006] are also modeled as conditional policies  [Florence et al., 2022] or applied to inverse RL [Liu et al., 2021].
Autoregressive model [Vaswani et al., 2017, Brown et al., 2020] represents the policy as the distribution of action, where it considers the distribution of the whole trajectory [Reed et al., 2022, Shafiullah et al., 2022].

§ EXPERIMENTS

In this section, we aim to cover the following three issues: How does DIPO compare to the widely used RL algorithms (SAC, PPO, and TD3) on the standard continuous control benchmark?
How to show and illustrate the empirical results?
How does the diffusion model compare to VAE [Kingma and Welling, 2013] and multilayer perceptron (MLP) for learning distribution?
How to choose the reverse length $K$ of DIPO for the reverse inference?

§.§ Comparative Evaluation and Illustration

We provide an evaluation on MuJoCo tasks [Todorov et al., 2012].
Figure <ref> shows the reward curves for SAC, PPO, TD3, and DIPO on MuJoCo tasks.
To demonstrate the robustness of the proposed DIPO, we train DIPO with the same hyperparameters for all those 5 tasks, where we provide the hyperparameters in Table <ref>, see Appendix <ref>. For each algorithm, we plot the average return of 5 independent trials as the solid curve and plot the standard deviation across 5 same seeds as the transparent shaded region. We evaluate all the methods with $10^{6}$ iterations. Results show that the proposed DIPO achieves the best score across all those 5 tasks, and DIPO learns much faster than SAC, PPO, and TD3 on the tasks of Ant-3v and Walker2d-3v.
Although the asymptotic reward performance of DIPO is similar to baseline algorithms on other 3 tasks, the proposed DIPO achieves better performance at the initial iterations,
we will try to illustrate some insights for such empirical results of HalfCheetah-v3 in Figure <ref>, for more discussions, see Appendix <ref>.

Average performances on MuJoCo Gym environments with $\pm$ std shaded, where the horizontal axis of coordinate denotes the iterations $(\times 10^6)$, the plots smoothed with a window of 10.

State-visiting visualization by each algorithm on the Ant-v3 task, where states get dimension reduction by t-SNE. The points with different colors represent the states visited by the policy with the style. The distance between points represents the difference between states.

State-visiting visualization for comparison between DIPO and SAC on HalfCheetah-v3.

§.§ State-Visiting Visualization

From Figure <ref>, we also know that DIPO achieves the best initial reward performance among all the 5 tasks, a more intuitive illustration has been shown in Figure <ref> and <ref>, where we only consider Ant-v3 and HalfCheetah-v3; for more discussions and observations, see Appendix <ref>.
We show the state-visiting region to compare both the exploration and final reward performance, where we use the same t-SNE [Van der Maaten and Hinton, 2008] to transfer the high-dimensional states visited by all the methods for 2D visualization.
Results of Figure <ref> show that the DIPO explores a wider range of state-visiting, covering TD3, SAC, and PPO.
Furthermore, from  Figure <ref>, we also know DIPO achieves a more dense state-visiting at the final period, which is a reasonable result since after sufficient training, the agent identifies and avoids the "bad" states, and plays actions transfer to "good" states.
On the contrary, PPO shows an aimless exploration in the Ant-v3 task, which partially explains why PPO is not so good in the Ant-v3 task.

From Figure <ref> we know, at the initial time, DIPO covers more regions than SAC in the HalfCheetah-v3, which results in DIPO obtaining a better reward performance than SAC. This result coincides with the results of Figure <ref>, which demonstrates that DIPO is efficient for exploration, which leads DIPO to better reward performance. While we also know that SAC starts with a narrow state visit that is similar to the final state visit, and SAC performs with the same reward performance with DIPO at the final, which implies SAC runs around the "good" region at the beginning although SAC performs a relatively worse initial reward performance than DIPO.
Thus, the result of Figure <ref> partially explains why DIPO performs better than SAC at the initial iterations but performs with same performance with SAC at the final for the HalfCheetah-v3 task.

Reward Performance Comparison to VAE and MLP with DIPO, SAC, PPO and TD3.

Learning curves with different reverse lengths $K=20,50,100$.

Reverse length             Ant-v3     HalfCheetah-v3     Hopper-v3     Humanoid-v3    Walker2d-v3

$K=100$               $\textbf{5622.30}\pm\textbf{487.09}$     $\textbf{10472.31}\pm\textbf{654.96}$     $3123.14\pm636.23$     $\textbf{4878.41}\pm\textbf{822.03}$     $\textbf{4409.18}\pm\textbf{469.06}$

$K=50$             $4877.41\pm1010.35$         $9198.20\pm1738.25$     $\textbf{3214.83}\pm\textbf{491.15}$    $4513.39\pm1075.94$     $4199.34\pm1062.31$

$K=20$              $5288.77\pm970.35$     $9343.69\pm 986.82$     $2511.63\pm 837.03$    $4294.79\pm1583.48$     $\textbf{4467.20}\pm\textbf{368.13}$

Average return over final 6E5 iterations with different reverse lengths $K=20,50,100$, and maximum value is bolded for each task.

§.§ Ablation Study

In this section, we consider the ablation study to compare the diffusion model with VAE and MLP for policy learning, and show a trade-off on the reverse length $K$ for reverse inference.

§.§.§ Comparison to VAE and MLP

Both VAE and MLP are widely used to learn distribution in machine learning, a fundamental question is:
why must we consider the diffusion model to learn a policy distribution? what the reward performance is if we use VAE and MLP to model a policy distribution?
We show the answer in Figure <ref>, where the VAE (or MLP) is the result we replace the diffusion policy of DIPO (see Figure <ref>) with VAE (or MLP), i.e.,
we consider VAE (or MLP)+action gradient for the tasks.
Results show that the diffusion model is more powerful than VAE and MLP for learning a distribution.
This implies the diffusion model is an expressive and flexible family to model a distribution, which is also consistent with the field of the generative model.

§.§.§ Comparison with Different Reverse Lengths
Reverse length $K$ is an important parameter for the diffusion model, which not only affects the reward performance but also affects the training time, we show the results in Figure <ref> and Table <ref>.
The results show that the reverse time $K=100$ returns a better reward performance than other cases (except Hopper-v3 task).
Longer reverse length consumes more reverse time for inference, we hope to use less time for reverse time for action inference.
However, a short reverse length $K=20$ decays the reward performance among (except Walker2d-v3 task), which implies a trade-off between reward performance and reverse length $K$. In practice, we set $K=100$ throughout this paper.

§ CONLUSION

We have formally built a theoretical foundation of diffusion policy, which shows a policy representation via the diffusion probability model and which is a new way to represent a policy via a stochastic process. Then, we have shown a convergence analysis for diffusion policy, which provides a theory to understand diffusion policy.
Furthermore, we have proposed an implementation for model-free online RL with a diffusion policy, named DIPO.
Finally, extensive empirical results show the effectiveness of DIPO among the Mujoco tasks.

[Ajay et al., 2021]
Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum.
Opal: Offline primitive discovery for accelerating offline
reinforcement learning.
In International Conference on Learning Representations
(ICLR), 2021.

[Ajay et al., 2023]
Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit
Is conditional generative modeling all you need for decision-making?
In International Conference on Learning Representations
(ICLR), 2023.

[Anderson, 1982]
Brian DO Anderson.
Reverse-time diffusion equation models.
Stochastic Processes and their Applications, 120
(3):0 313–326, 1982.

[Argall et al., 2009]
Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning.
A survey of robot learning from demonstration.
Robotics and autonomous systems, 570 (5):0
469–483, 2009.

[Bakry et al., 2014]
Dominique Bakry, Ivan Gentil, Michel Ledoux, et al.
Analysis and geometry of Markov diffusion operators, volume
Springer, 2014.

[Bengio et al., 2021]
Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio.
Flow network based generative models for non-iterative diverse
candidate generation.
Advances in Neural Information Processing Systems (NeurIPS),
34:0 27381–27394, 2021a.

[Bengio et al., 2021]
Yoshua Bengio, Salem Lahlou, Tristan Deleu, Edward J Hu, Mo Tiwari, and
Emmanuel Bengio.
Gflownet foundations.
arXiv preprint arXiv:2111.09266, 2021b.

[Brehmer et al., 2023]
Johann Brehmer, Joey Bose, Pim De Haan, and Taco Cohen.
Edgi: Equivariant diffusion for planning with embodied agents.
In Workshop on Reincarnating Reinforcement Learning at ICLR,

[Brohan et al., 2022]
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis,
Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine
Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth,
Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal,
Kuang-Huei Lee, Sergey Levine, Yao Lu, Utsav Malla, Deeksha Manjunath, Igor
Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl
Pertsch, Jornell Quiambao, Kanishka Rao, Michael Ryoo, Grecia Salazar, Pannag
Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Sontakke, Austin Stone, Clayton
Tan, Huong Tran, Vincent Vanhoucke, Steve Vega, Quan Vuong, Fei Xia, Ted
Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich.
Rt-1: Robotics transformer for real-world control at scale.
In arXiv preprint arXiv:2212.06817, 2022.

[Brown et al., 2020]
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla
Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell,
et al.
Language models are few-shot learners.
Advances in neural information processing systems (NeurIPS),
33:0 1877–1901, 2020.

[Chen et al., 2022]
Hongrui Chen, Holden Lee, and Jianfeng Lu.
Improved analysis of score-based generative modeling: User-friendly
bounds under minimal smoothness assumptions.
arXiv preprint arXiv:2211.01916, 2022.

[Chen et al., 2023]
Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu.
Offline reinforcement learning via high-fidelity generative behavior
In International Conference on Learning Representations
(ICLR), 2023a.

[Chen et al., 2023]
Jinyin Chen, Shulong Hu, Haibin Zheng, Changyou Xing, and Guomin Zhang.
Gail-pt: An intelligent penetration testing framework with generative
adversarial imitation learning.
Computers & Security, 126:0 103055,

[Chen et al., 2021]
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha
Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch.
Decision transformer: Reinforcement learning via sequence modeling.
Advances in neural information processing systems(NeurIPS),
34:0 15084–15097, 2021.

[Chen et al., 2023]
Sitan Chen, Sinho Chewi, Jerry Li, Yuanzhi Li, Adil Salim, and Anru R Zhang.
Sampling is as easy as learning the score: theory for diffusion
models with minimal data assumptions.
In International Conference on Learning Representations
(ICLR), 2023c.

[Chi et al., 2023]
Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin
Burchfiel, and Shuran Song.
Diffusion policy: Visuomotor policy learning via action diffusion.
arXiv preprint arXiv:2303.04137, 2023.

[Chung and Williams, 1990]
Kai Lai Chung and Ruth J Williams.
Introduction to stochastic integration, volume 2.
Springer, 1990.

[Cui et al., 2023]
Zichen Jeff Cui, Yibin Wang, Nur Muhammad, Lerrel Pinto, et al.
From play to policy: Conditional behavior generation from uncurated
robot data.
International Conference on Learning Representations (ICLR),

[Deka et al., 2023]
Ankur Deka, Changliu Liu, and Katia P Sycara.
Arc-actor residual critic for adversarial imitation learning.
In Conference on Robot Learning (CORL), pages 1446–1456.
PMLR, 2023.

[Donsker and Varadhan, 1983]
Monroe D Donsker and SR Srinivasa Varadhan.
Asymptotic evaluation of certain markov process expectations for
large time. iv.
Communications on pure and applied mathematics, 360
(2):0 183–212, 1983.

[Espeholt et al., 2018]
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward,
Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al.
Impala: Scalable distributed deep-rl with importance weighted
actor-learner architectures.
In International conference on machine learning (ICML), pages
1407–1416. PMLR, 2018.

[Florence et al., 2022]
Pete Florence, Corey Lynch, Andy Zeng, Oscar A Ramirez, Ayzaan Wahid, Laura
Downs, Adrian Wong, Johnny Lee, Igor Mordatch, and Jonathan Tompson.
Implicit behavioral cloning.
In Conference on Robot Learning (CORL), pages 158–168. PMLR,

[Fokker, 1914]
Adriaan Daniël Fokker.
Die mittlere energie rotierender elektrischer dipole im
Annalen der Physik, 3480 (5):0 810–820, 1914.

[Fu et al., 2018]
Justin Fu, Katie Luo, and Sergey Levine.
Learning robust rewards with adverserial inverse reinforcement
In International Conference on Learning Representations
(ICLR), 2018.

[Fujimoto et al., 2018]
Scott Fujimoto, Herke Hoof, and David Meger.
Addressing function approximation error in actor-critic methods.
In International conference on machine learning (ICML), pages
1587–1596. PMLR, 2018.

[Goodfellow et al., 2020]
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley,
Sherjil Ozair, Aaron Courville, and Yoshua Bengio.
Generative adversarial networks.
Communications of the ACM, 630 (11):0
139–144, 2020.

[Haarnoja et al., 2017]
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine.
Reinforcement learning with deep energy-based policies.
In International conference on machine learning (ICML), pages
1352–1361. PMLR, 2017.

[Haarnoja et al., 2018]
Tuomas Haarnoja, Vitchyr Pong, Aurick Zhou, Murtaza Dalal, Pieter Abbeel, and
Sergey Levine.
Composable deep reinforcement learning for robotic manipulation.
In 2018 IEEE international conference on robotics and
automation (ICRA), pages 6244–6251. IEEE, 2018a.

[Haarnoja et al., 2018]
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine.
Soft actor-critic: Off-policy maximum entropy deep reinforcement
learning with a stochastic actor.
In International conference on machine learning (ICML), pages
1861–1870. PMLR, 2018b.

[Hasselt, 2010]
Hado Hasselt.
Double q-learning.
Advances in Neural Information Processing Systems (NeurIPS),
23, 2010.

[Haussmann and Pardoux, 1986]
Ulrich G Haussmann and Etienne Pardoux.
Time reversal of diffusions.
The Annals of Probability, pages 1188–1205, 1986.

[Ho and Ermon, 2016]
Jonathan Ho and Stefano Ermon.
Generative adversarial imitation learning.
Advances in neural information processing systems (NeurIPS),
29, 2016.

[Ho et al., 2020]
Jonathan Ho, Ajay Jain, and Pieter Abbeel.
Denoising diffusion probabilistic models.
Advances in Neural Information Processing Systems (NeurIPS),
33:0 6840–6851, 2020.

[Hyvärinen, 2005]
Aapo Hyvärinen.
Estimation of non-normalized statistical models by score matching.
Journal of Machine Learning Research (JMLR), 60 (4),

[Janner et al., 2022]
Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine.
Planning with diffusion for flexible behavior synthesis.
In International Conference on Machine Learning (ICML), 2022.

[Kapelyukh et al., 2022]
Ivan Kapelyukh, Vitalis Vosylius, and Edward Johns.
Dall-e-bot: Introducing web-scale diffusion models to robotics.
In CoRL 2022 Workshop on Pre-training Robot Learning, 2022.

[Kim et al., 2023]
Changyeon Kim, Jongjin Park, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin
Preference transformer: Modeling human preferences using transformers
for rl.
In The Eleventh International Conference on Learning
Representations (ICLR), 2023.

[Kim et al., 2022]
Dongjun Kim, Seungjae Shin, Kyungwoo Song, Wanmo Kang, and Il-Chul Moon.
Soft truncation: A universal training technique of score-based
diffusion model for high precision score estimation.
In Proceedings of the 39th International Conference on Machine
Learning (ICML), volume 162, pages 11201–11228, 2022.

[Kingma and Welling, 2013]
Diederik P Kingma and Max Welling.
Auto-encoding variational bayes.
arXiv preprint arXiv:1312.6114, 2013.

[Kolmogoroff, 1931]
Andrei Kolmogoroff.
Über die analytischen methoden in der
Mathematische Annalen, 104:0 415–458, 1931.

[Konan et al., 2023]
Sachin G Konan, Esmaeil Seraj, and Matthew Gombolay.
Contrastive decision transformers.
In Conference on Robot Learning (CORL), pages 2159–2169.
PMLR, 2023.

[LeCun et al., 2006]
Yann LeCun, Sumit Chopra, Raia Hadsell, Marc’Aurelio Ranzato, and Fu Jie
A tutorial on energy-based learning.
Predicting Structured Data, 1:0 0, 2006.

[Lee et al., 2023]
Holden Lee, Jianfeng Lu, and Yixin Tan.
Convergence of score-based generative modeling for general data
In International Conference on Algorithmic Learning Theory
(ALT), pages 946–985. PMLR, 2023.

[Levine et al., 2020]
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu.
Offline reinforcement learning: Tutorial, review, and perspectives on
open problems.
arXiv preprint arXiv:2005.01643, 2020.

[Li et al., 2023]
Wenhao Li, Baoxiang Wang, Shanchao Yang, and Hongyuan Zha.
Diverse policy optimization for structured action space.
In Proceedings of the 22th International Conference on
Autonomous Agents and MultiAgent Systems (AAMAS), 2023a.

[Li et al., 2023]
Yinchuan Li, Shuang Luo, Haozhi Wang, and Hao Jianye.
Cflownets: Continuous control with generative flow networks.
In The Eleventh International Conference on Learning
Representations (ICLR), 2023b.

[Liang et al., 2023]
Zhixuan Liang, Yao Mu, Mingyu Ding, Fei Ni, Masayoshi Tomizuka, and Ping Luo.
Adaptdiffuser: Diffusion models as adaptive self-evolving planners.
arXiv preprint arXiv:2302.01877, 2023.

[Lillicrap et al., 2016]
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom
Erez, Yuval Tassa, David Silver, and Daan Wierstra.
Continuous control with deep reinforcement learning.
In International Conference on Learning Representations
(ICLR), 2016.

[Liu et al., 2021]
Minghuan Liu, Tairan He, Minkai Xu, and Weinan Zhang.
Energy-based imitation learning.
In Proceedings of the 20th International Conference on
Autonomous Agents and MultiAgent Systems (AAMAS), pages 809–817, 2021.

[Liu et al., 2022]
Weiyu Liu, Tucker Hermans, Sonia Chernova, and Chris Paxton.
Structdiffusion: Object-centric diffusion for semantic rearrangement
of novel objects.
In Workshop on Language and Robotics at CoRL, 2022.

[Lynch et al., 2020]
Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey
Levine, and Pierre Sermanet.
Learning latent plans from play.
In Conference on robot learning (CORL), pages 1113–1132.
PMLR, 2020.

[Ma et al., 2019]
Yi-An Ma, Yuansi Chen, Chi Jin, Nicolas Flammarion, and Michael I Jordan.
Sampling can be faster than optimization.
Proceedings of the National Academy of Sciences, 1160
(42):0 20881–20885, 2019.

[Malkin et al., 2022]
Nikolay Malkin, Moksh Jain, Emmanuel Bengio, Chen Sun, and Yoshua Bengio.
Trajectory balance: Improved credit assignment in gflownets.
In Advances in Neural Information Processing Systems, 2022.

[Mandlekar et al., 2020]
Ajay Mandlekar, Danfei Xu, Roberto Martín-Martín, Silvio Savarese,
and Li Fei-Fei.
Learning to generalize across long-horizon tasks from human
In Robotics: Science and Systems (RSS), 2020.

[Mees et al., 2022]
Oier Mees, Lukas Hermann, and Wolfram Burgard.
What matters in language conditioned robotic imitation learning over
unstructured data.
IEEE Robotics and Automation Letters, 70 (4):0
11205–11212, 2022.

[Mishra and Chen, 2023]
Utkarsh A Mishra and Yongxin Chen.
Reorientdiff: Diffusion model based reorientation for object
arXiv preprint arXiv:2303.12700, 2023.

[Misra, 2019]
Diganta Misra.
Mish: A self regularized non-monotonic activation function.
arXiv preprint arXiv:1908.08681, 2019.

[Mnih et al., 2015]
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness,
Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg
Ostrovski, et al.
Human-level control through deep reinforcement learning.
nature, 5180 (7540):0 529–533, 2015.

[Osa et al., 2018]
Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J Andrew Bagnell, Pieter Abbeel,
Jan Peters, et al.
An algorithmic perspective on imitation learning.
Foundations and Trends® in Robotics,
70 (1-2):0 1–179, 2018.

[Pan et al., 2023]
Ling Pan, Dinghuai Zhang, Moksh Jain, Longbo Huang, and Yoshua Bengio.
Stochastic generative flow networks.
arXiv preprint arXiv:2302.09465, 2023.

[Pearce et al., 2023]
Tim Pearce, Tabish Rashid, Anssi Kanervisto, Dave Bignell, Mingfei Sun, Raluca
Georgescu, Sergio Valcarcel Macua, Shan Zheng Tan, Ida Momennejad, Katja
Hofmann, et al.
Imitating human behaviour with diffusion models.
In International Conference on Learning Representations
(ICLR), 2023.

[Pertsch et al., 2021]
Karl Pertsch, Youngwoon Lee, and Joseph Lim.
Accelerating reinforcement learning with learned skill priors.
In Conference on robot learning (CORL), pages 188–204, 2021.

[Peters et al., 2010]
Jan Peters, Katharina Mulling, and Yasemin Altun.
Relative entropy policy search.
In Proceedings of the AAAI Conference on Artificial
Intelligence, volume 24, pages 1607–1612, 2010.

[Planck, 1917]
VM Planck.
Über einen satz der statistischen dynamik und seine erweiterung
in der quantentheorie.
Sitzungsberichte der Preussischen Akademie der Wissenschaften
zu Berlin, 24:0 324–341, 1917.

[Pomerleau, 1988]
Dean A Pomerleau.
Alvinn: An autonomous land vehicle in a neural network.
Advances in neural information processing systems (NeurIPS),
1, 1988.

[66]
Rafael Rafailov, Victor Kolev, Kyle Beltran Hatch, John D Martin, Mariano
Phielipp, Jiajun Wu, and Chelsea Finn.
Model-based adversarial imitation learning as online fine-tuning.
In Workshop on Reincarnating Reinforcement Learning at ICLR

[Reed et al., 2022]
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gómez Colmenarejo,
Alexander Novikov, Gabriel Barth-maron, Mai Giménez, Yury Sulsky, Jackie
Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley
Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar
Bordbar, and Nando de Freitas.
A generalist agent.
Transactions on Machine Learning Research (TMLR), 2022.
ISSN 2835-8856.

[Reuss et al., 2023]
Moritz Reuss, Maximilian Li, Xiaogang Jia, and Rudolf Lioutikov.
Goal-conditioned imitation learning using score-based diffusion
arXiv preprint arXiv:2304.02532, 2023.

[Rezende and Mohamed, 2015]
Danilo Rezende and Shakir Mohamed.
Variational inference with normalizing flows.
In International conference on machine learning (ICML), pages
1530–1538, 2015.

[Risken and Risken, 1996]
Hannes Risken and Hannes Risken.
Fokker-planck equation.
Springer, 1996.

[Rosete-Beas et al., 2023]
Erick Rosete-Beas, Oier Mees, Gabriel Kalweit, Joschka Boedecker, and Wolfram
Latent plans for task-agnostic offline reinforcement learning.
In Conference on Robot Learning (CORL), pages 1838–1849,

[Rummery and Niranjan, 1994]
Gavin A Rummery and Mahesan Niranjan.
On-line Q-learning using connectionist systems, volume 37.
University of Cambridge, Department of Engineering, 1994.

[Sallans and Hinton, 2004]
Brian Sallans and Geoffrey E Hinton.
Reinforcement learning with factored states and actions.
The Journal of Machine Learning Research (JMLR), 5:0
1063–1088, 2004.

[Särkkä and Solin, 2019]
Simo Särkkä and Arno Solin.
Applied stochastic differential equations, volume 10.
Cambridge University Press, 2019.

[Schulman et al., 2015]
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp
Trust region policy optimization.
In International conference on machine learningn (ICML), pages
1889–1897. PMLR, 2015.

[Schulman et al., 2017]
John Schulman, Xi Chen, and Pieter Abbeel.
Equivalence between policy gradients and soft q-learning.
arXiv preprint arXiv:1704.06440, 2017a.

[Schulman et al., 2017]
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.
Proximal policy optimization algorithms.
arXiv preprint arXiv:1707.06347, 2017b.

[Shafiullah et al., 2022]
Nur Muhammad Shafiullah, Zichen Cui, Ariuntuya Arty Altanzaya, and Lerrel
Behavior transformers: Cloning $ k $ modes with one stone.
Advances in neural information processing systems (NeurIPS),
35:0 22955–22968, 2022.

[Silver et al., 2014]
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and
Martin Riedmiller.
Deterministic policy gradient algorithms.
In International conference on machine learning (ICML), pages
387–395. Pmlr, 2014.

[Singh et al., 2020]
Avi Singh, Huihan Liu, Gaoyue Zhou, Albert Yu, Nicholas Rhinehart, and Sergey
Parrot: Data-driven behavioral priors for reinforcement learning.
In International Conference on Learning Representations
(ICLR), 2020.

[Sohl-Dickstein et al., 2015]
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.
Deep unsupervised learning using nonequilibrium thermodynamics.
In International Conference on Machine Learning (ICML), pages
2256–2265. PMLR, 2015.

[Song et al., 2021]
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano
Ermon, and Ben Poole.
Score-based generative modeling through stochastic differential
In International Conference on Learning Representations
(ICLR), 2021.

[Sutton, 1988]
Richard S Sutton.
Learning to predict by the methods of temporal differences.
Machine learning, 3:0 9–44, 1988.

[Sutton and Barto, 1998]
Richard S Sutton and Andrew G Barto.
Reinforcement learning: An introduction.
MIT press, 1998.

[Sutton and Barto, 2018]
Richard S Sutton and Andrew G Barto.
Reinforcement learning: An introduction.
MIT press, 2018.

[Sutton et al., 1999]
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour.
Policy gradient methods for reinforcement learning with function
Advances in neural information processing systems (NeurIPS),
12, 1999.

[Taranovic et al., 2023]
Aleksandar Taranovic, Andras Gabor Kupcsik, Niklas Freymuth, and Gerhard
Adversarial imitation learning with preferences.
In The Eleventh International Conference on Learning
Representations (ICLR), 2023.

[Todorov et al., 2012]
Emanuel Todorov, Tom Erez, and Yuval Tassa.
Mujoco: A physics engine for model-based control.
In 2012 IEEE/RSJ International Conference on Intelligent Robots
and Systems, pages 5026–5033. IEEE, 2012.

[Urain et al., 2023]
Julen Urain, Niklas Funk, Georgia Chalvatzaki, and Jan Peters.
Se(3)-diffusionfields: Learning cost functions for joint grasp and
motion optimization through diffusion.
In 2018 IEEE international conference on robotics and
automation (ICRA), 2023.

[Van Den Oord et al., 2017]
Aaron Van Den Oord, Oriol Vinyals, et al.
Neural discrete representation learning.
Advances in neural information processing systems (NeurIPS),
30, 2017.

[Van der Maaten and Hinton, 2008]
Laurens Van der Maaten and Geoffrey Hinton.
Visualizing data using t-sne.
Journal of machine learning research (JMLR), 90 (11),

[Vaswani et al., 2017]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,
Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.
Attention is all you need.
Advances in neural information processing systems (NeurIPS),
30, 2017.

[Vempala and Wibisono, 2019]
Santosh Vempala and Andre Wibisono.
Rapid convergence of the unadjusted langevin algorithm: Isoperimetry
In Advances in neural information processing systems
(NeurIPS), volume 32, 2019.

[Vincent, 2011]
Pascal Vincent.
A connection between score matching and denoising autoencoders.
Neural computation, 230 (7):0 1661–1674,

[Wang et al., 2022]
Lirui Wang, Xiangyun Meng, Yu Xiang, and Dieter Fox.
Hierarchical policies for cluttered-scene grasping with latent plans.
IEEE Robotics and Automation Letters, 70 (2):0
2883–2890, 2022.

[Wang et al., 2021]
Yunke Wang, Chang Xu, Bo Du, and Honglak Lee.
Learning to weight imperfect demonstrations.
In International Conference on Machine Learning (ICML), pages
10961–10970. PMLR, 2021.

[Wang et al., 2023]
Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou.
Diffusion policies as an expressive policy class for offline
reinforcement learning.
In International Conference on Learning Representations
(ICLR), 2023.

[Watkins, 1989]
Christopher John Cornish Hellaby Watkins.
Learning from delayed rewards.
PhD thesis, King's College, Cambridge, 1989.

[Wibisono and Jog, 2018]
Andre Wibisono and Varun Jog.
Convexity of mutual information along the ornstein-uhlenbeck flow.
In 2018 International Symposium on Information Theory and Its
Applications (ISITA), pages 55–59. IEEE, 2018.

[Wibisono and Yang, 2022]
Andre Wibisono and Kaylee Yingxi Yang.
Convergence in kl divergence of the inexact langevin algorithm with
application to score-based generative models.
arXiv preprint arXiv:2211.01512, 2022.

[Yang et al., 2022]
Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao,
Yingxia Shao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang.
Diffusion models: A comprehensive survey of methods and applications.
arXiv preprint arXiv:2209.00796, 2022.

[Yang et al., 2023]
Sherry Yang, Ofir Nachum, Yilun Du, Jason Wei, Pieter Abbeel, and Dale
Foundation models for decision making: Problems, methods, and
arXiv preprint arXiv:2303.04129, 2023.

[Zhang, 2022]
Chen Yongxin Zhang, Qinsheng.
Fast sampling of diffusion models with exponential integrator.
In International Conference on Learning Representations
(ICLR), 2022.

[Zheng et al., 2022]
Qinqing Zheng, Amy Zhang, and Aditya Grover.
Online decision transformer.
In International Conference on Machine Learning (ICML), pages
27042–27059. PMLR, 2022.

§ REVIEW ON NOTATIONS

This section reviews some notations and integration by parts formula, using the notations consistent with [Vempala and Wibisono, 2019].

Given a smooth function $f:\R^{n} \rightarrow\R$, its gradient $\bnabla f:\R^{n} \rightarrow\R^{n}$ is the vector of partial derivatives:

\[
\bnabla f(\bx) = \left(\part{f(\bx)}{x_1}, \dots, \part{f(\bx)}{x_n} \right)^{\top}.
\]

The Hessian $\bnabla^2 f \colon \R^n \to \R^{n \times n}$ is the matrix of second partial derivatives:

\[
\bnabla^2 f(\bx) = \left(\part{^2f(\bx)}{x_i x_j} \right)_{1 \le i,j \le n}.
\]

The Laplacian $\Delta f \colon \R^n \to \R$ is the trace of its Hessian:

\[
\Delta f(\bx) = \mathrm{Tr}\left(\bnabla^2 f(\bx)\right) = \sum_{i=1}^n \part{^2 f(\bx)}{x_i^2}.
\]

Given a smooth vector field $\bp = (p_1,\dots,p_n) \colon \R^n \to \R^n$, its divergence is $\DIV \cdot \bp \colon \R^n \to \R$:

(·)() = ∑_i=1^n

When the variable of a function is clear and without causing ambiguity, we also denote the above notation as follows,

·(()) =:
(·)() = ∑_i=1^n

In particular, the divergence of the gradient is the Laplacian:

(·f)() = ∑_i=1^n

^2 f()x_i^2 = Δf().

Let $\rho(\bx)$ and $\mu(\bx)$ be two smooth probability density functions on the space $\R^{p}$,
the Kullback–Leibler (KL) divergence and relative Fisher information (FI) from $\mu(\bx)$ to $\rho(\bx)$ are defined as follows,

(ρμ)=_^pρ()logρ()μ(),  (ρμ)=_^pρ()logρ()μ()_2^2.

Before we further analyze, we need the integration by parts formula.
For any function $f \colon \R^p \to \R$ and vector field $\bm{v} \colon \R^p \to \R^p$ with sufficiently fast decay at infinity,
we have the following integration by parts formula:

_^p ⟨v(), ∇f() ⟩= -_^p f() (·v)() .

§ AUXILIARY RESULTS

§.§ Diffusion Probability Model (DPM).

This section reviews some basic background about the diffusion probability model (DPM).
For a given (but unknown) $p$-dimensional data distribution $q(\bx_0)$,
DPM [Sohl-Dickstein et al., 2015, Ho et al., 2020, Song et al., 2021] is a latent variable generative model that learns a parametric model to approximate the distribution $q(\bx_0)$.
To simplify the presentation in this section, we only focus on the continuous-time diffusion [Song et al., 2021].
The mechanism of DPM contains two processes, forward process and reverse process; we present them as follows.

Forward Process. The forward process produces a sequence $\{\bx_{t}\}_{t=0:T}$ that perturbs the initial $\bx_{0}\sim q(\cdot)$ into a Gaussian noise, which follows the next stochastic differential equation (SDE),

where $\bbf(\cdot,\cdot)$ is the drift term, $g(\cdot)$ is the diffusion term, and $\bw_{t}$ is the standard Wiener process.

Reverse Process. According to [Anderson, 1982, Haussmann and Pardoux, 1986], there exists a corresponding reverse SDE that exactly coincides with the solution of the forward SDE (<ref>):

where $\dd\bar{t}$ is the backward time differential, $\dd\bar{\bw}_{t}$ is a standard Wiener process flowing backward in time, and $p_{t}(\bx_{t})$is the marginal probability distribution of the random variable $\bx_{t}$ at time $t$.

Once the score function $\bnabla \log p_{t}(\bx_{t})$ is known for each time $t$, we can derive the reverse diffusion process from SDE (<ref>) and simulate it to sample from $q(\bx_{0})$ [Song et al., 2021].

§.§ Transition Probability for Ornstein-Uhlenbeck Process

Consider the next SDEs,

\[
\dd \bx_{t}=-\dfrac{1}{2}\beta(t)\bx_{t}\dd t+g(t)\dd \bw_{t},
\]

where $\beta(\cdot)$ and $g(\cdot)$ are real-valued functions. Then, for a given $\bx_{0}$, the conditional distribution of $\bx_{t}|\bx_{0}$ is a Gaussian distribution, i.e.,

\[
\bx_{t}|\bx_{0}\sim\calN\left(\bx_{t}|\bm{\mu}_t,\bm{\Sigma}_{t}\right),
\]

μ_t=   exp{-12∫_0^tβ(s)s}_0,

Σ_t=   exp{-∫_0^tβ(s)s}(∫_0^texp{∫_0^τβ(s)s}g^2(τ)τ).

See <cit.> or <cit.>.

§.§ Exponential Integrator Discretization

The next Proposition <ref> provides a fundamental way for us to derivate the exponential integrator discretization (<ref>).

For a given state $\bs$, we consider the following continuous time process, for $t\in[t_{k},t_{k+1}]$,

_t=(_t+2  S(_t_k,,T-t_k))t+√(2)_t.

Then with Itô integration [Chung and Williams, 1990],

\[
\bx_{t}-\bx_{t_{k}}=\left(\mathrm{e}^{t-t_{k}}-1\right)\left(\bx_{t_{k}}+2  \bm{\hat \mathbf{S}}(\bx_{t_k},\bs,T-t_{k})\right)+\sqrt{2}\int_{t_k}^{t}\mathrm{e}^{t^{'}-t_{k}}\dd \bw_{t^{'}},
\]

where $t\in[t_{k},t_{k+1}]$, and $t_{k}=hk$.

See <cit.>.

Recall SDE (<ref>), according to Proposition <ref>, we know the next SDE

_t=(_t+2 S(_t_k,,T-t_k))t+√(2)_t,  t∈[t_k,t_k+1]

formulates the exponential integrator discretization as follows,

_t_k+1-_t_k=   (e^t_k+1-t_k-1)(_t_k+2 S(_t_k,,T-t_k))+√(2)∫_t_k^t_k+1e^t^'-t_k_t^'

=   (e^h-1)(_t_k+2 S(_t_k,,T-t_k))+√(e^2h-1),

where last equation holds due to Wiener process is a stationary process with independent increments, i.e., the following holds,

=   √(2)∫_0^t_k+1e^t^'-t_k_t^'-√(2)∫_0^t_ke^t^'-t_k_t^'

=   √(e^2h-1),

where $\bz\sim\calN(\bm{0},\bI)$.

Then, we rewrite (<ref>) as follows,

_t_k+1=   e^h_t_k+(e^h-1)(_t_k+2 S(_t_k,,T-t_k))+√(e^2h-1),

which concludes the iteration defined in (<ref>).

§.§ Fokker–Planck Equation

The Fokker–Planck equation is named after Adriaan Fokker and Max Planck, who described it in 1914 and 1917 [Fokker, 1914, Planck, 1917].
It is also known as the Kolmogorov forward equation, after Andrey Kolmogorov, who independently discovered it in 1931 [Kolmogoroff, 1931].
For more history and background about Fokker–Planck equation, please refer to [Risken and Risken, 1996] or <https://en.wikipedia.org/wiki/Fokker%E2%80%93Planck_equation#cite_note-1>.

For an Itô process driven by the standard Wiener process
\bw_{t}$ and described by the stochastic differential equation

𝐱 _t=μ(𝐱 _t,t)t+Σ(𝐱 _t,t)𝐰 _t,

where $\mathbf {x} _{t}$ and ${\boldsymbol {\mu }}(\mathbf {x} _{t},t)$ are $N$-dimensional random vectors,
${\boldsymbol {\Sigma }}(\mathbf {x} _{t},t)$ is an ${\displaystyle n\times m}$ matrix and
$\mathbf {w} _{t}$ is an $m$-dimensional standard Wiener process, the probability density
$p(\mathbf {x} ,t)$ for $\mathbf {x} _{t}$ satisfies the Fokker–Planck equation

∂p(𝐱 ,t)/∂t=-∑_i=1^N∂/∂x_i[μ_i(𝐱 ,t)p(𝐱 ,t)]+∑_i=1^N∑_j=1^N∂^2/∂x_i ∂x_j[D_ij(𝐱 ,t)p(𝐱 ,t)],

with drift vector
{\boldsymbol {\mu }}=(\mu _{1},\ldots ,\mu _{N})
and diffusion tensor
{\textstyle \mathbf {D} (\bx,t)={\dfrac {1}{2}}{\boldsymbol {\Sigma \Sigma }}^{\top}}$, i.e.

\[
{\displaystyle D_{ij}(\mathbf {x} ,t)={\dfrac {1}{2}}\sum _{k=1}^{M}\sigma _{ik}(\mathbf {x} ,t)\sigma _{jk}(\mathbf {x} ,t),}
\]

and $\sigma_{ij}$ denotes the $(i,j)$-th element of the matrix $\bm\Sigma$.

§.§ Donsker-Varadhan Representation for KL-divergence

Let $\rho,\mu$ be  two probability distributions on the measure space $(\calX,\calF)$, where $\calX\in\R^{p}$. Then

\[
\KL(\rho\|\mu)=\sup_{f:\calX\rightarrow\R}\left\{\int_{\R^{p}}\rho(\bx)f(\bx)\dd\bx-\log\int_{\R^{p}}\mu(\bx)\exp(f(\bx))\dd\bx\right\}.
\]

The Donsker-Varadhan representation for KL-divergence implies for any $f(\cdot)$,

which is useful later.

§.§ Some Basic Results for Diffusion Policy

Let $\pi(\cdot|\bs)$ satisfy $\nu$-Log-Sobolev inequality (LSI) (see Assumption <ref>),
the initial random action ${\color{red}{\bar{\ba}_{0}}}\sim\pi(\cdot|\bs)$, and ${\color{red}{\bar{\ba}_{t}}}$ evolves according to
the following Ornstein-Uhlenbeck process,

Let ${\color{red}{\bar{\ba}_{t}}}\sim {\color{red}{\bar{\pi}_{t}}}(\cdot|\bs)$ be the evolution along the Ornstein-Uhlenbeck flow (<ref>), then
${\color{red}{\bar{\pi}_{t}}}(\cdot|\bs)$ is $\nu_t$-LSI, where
\[\nu_t=\dfrac{\nu}{\nu+(1-\nu)\mathrm{e}^{-2t}}.\]

See <cit.>.

Under Assumption <ref>, then $\bnabla\log {\color{orange}{\tilde{\pi}_{t}}}(\cdot|\bs)$ is $L_{p}\mathrm{e}^{t}$-Lipschitz on the time interval $[0,\mathtt{T}_{0}]$, where the policy ${\color{orange}{\tilde{\pi}_{t}}}(\cdot|\bs)$ is the evolution along the flow (<ref>),
and the time $\mathtt{T}_{0}$ is defined as follows,

\[
\mathtt{T}_{0}=:\sup_{t\ge0}\left\{t:1-\mathrm{e}^{-2t}\leq \frac{\mathrm{e}^{-t}}{L_p}\right\}.
\]

The positive scalar $\mathtt{T}_{0}$ is well-defined, i.e., $\mathtt{T}_{0}$ always exists. In fact, let $1-\mathrm{e}^{-2t}\leq \frac{\mathrm{e}^{-t}}{L_p}\ge0$, then the following holds,

\[
\mathrm{e}^{-t}\ge\sqrt{\dfrac{1}{L^{2}_{p}}+4
\]

\[
\mathtt{T}_{0}=\log\left(\dfrac{1}{4}\left(
\sqrt{\dfrac{1}{L^{2}_{p}}+4
\right)\right).
\]

Let $\rho(\bx)$ be a probability distribution function on $\R^{p}$, and let $f(\bx)=-\log \rho(\bx)$ be a $L$-smooth, i.e., there exists a positive constant $L$ such that $-L\bI\preceq\bnabla^{2} f(\bx)\preceq L\bI$ for all $\bx\in\R^{p}$.
Furthermore, let $\rho(\bx)$ satisfy the LSI condition with constant $\nu>0$, i.e., for any probability distribution $\mu(\bx)$,
\KL(\mu\|\rho)\leq\dfrac{1}{2\nu}\FI(\mu\|\rho).
Then for any distribution $\mu(\bx)$, the following equation holds,

\[
\E_{\bx\sim\mu(\cdot)}\left[\|\bnabla\log \rho(\bx)\|_2^{2}\right]=\int_{\R^{p}}\mu(\bx)\left\|\bnabla\log \rho(\bx)\right\|_2^{2}\dd\bx\leq\dfrac{4L^2}{\nu}\KL(\mu\|\rho)+2pL.
\]

§ IMPLEMENTATION DETAILS OF DIPO

In this section, we provide all the details of our implementation for DIPO.

(DIPO): Model-Free Learning with Diffusion Policy

[1]
Initialize parameter ${\bphi}$, critic networks $Q_{\bpsi}$, target networks $Q_{\bpsi^{'}}$, length $K$;
Initialize $\{\beta_{i}\}_{i=1}^{K}$; $\alpha_i=:1-\beta_i,\bar{\alpha}_k=:\prod_{i=1}^{k}\alpha_i,~\sigma_{k}=:\sqrt{\dfrac{1-\bar{\alpha}_{k-1}}{1-\bar{\alpha}_{k}}\beta_k}$;
Initialize $\bpsi^{'}\gets\bpsi$, $\bphi^{'}\gets\bphi$;

dataset $\calD_{\mathrm{env}}\gets\emptyset$; initial state $\bs_{0}\sim d_{0}(\cdot)$;
initial $\hata_{K}\sim\calN(\bm{0},\bI)$;
$\bz_{k}\sim\calN(\bm{0},\bI)$, if $k>1$; else $\bz_{k}=0$;
$\ba_{t}\gets\hata_{0}$; $\bs_{t+1}\sim \Pro(\cdot|\bs_{t},\ba_{t})$; $\calD_{\mathrm{env}}\gets\calD_{\mathrm{env}}\cup\{\bs_{t},\ba_{t},\bs_{t+1},r(\bs_{t+1}|\bs_{t},\ba_{t})\}$;

each mini-batch data
sample mini-batch $\calD$ from $\calD_{\mathrm{env}}$ with size $N$, $\calD=\{\bs_{j},\ba_{j},\bs_{j+1},r(\bs_{j+1}|\bs_{j},\ba_{j})\}_{j=1}^{N}$;
take gradient descent as follows
\[\bpsi\gets\bpsi-\eta_{\psi}\bnabla_{\bpsi}\dfrac{1}{N}\sum_{j=1}^{N}\Big(r(\bs_{j+1}|\bs_j,\ba_j)+\gamma Q_{\bpsi^{'}}(\bs_{j+1},\ba_{j+1})-Q_{\bpsi}(\bs_j,\ba_j)\Big)^2;\]

replace the action $\ba_{t}\in\calD_{\mathrm{env}}$ as follows

\[\ba_{t}\gets\ba_{t}+\eta_{a}\bnabla_{\ba}Q_{\bpsi}(\bs_{t},\ba)\big|_{\ba=\ba_t};\]

each pair
sample a pair $(\bs,\ba)\sim\calD_{\mathrm{env}}$ uniformly; $k\sim \mathrm{Uniform}(\{1,\cdots,K\})$; $\bz\sim \calN(\bm{0},\bI)$;
take gradient descent as follows
\[\bphi\gets\bphi-\eta_{\phi}\bnabla_{\bphi}\left\|\bz-\bepsilon_{\bphi}\left(\sqrt{\bar\alpha_k}\ba+\sqrt{1-\bar{\alpha}_k}\bz,\bs,k\right)\right\|_{2}^{2};\]

soft update $\bpsi^{'}\gets\rho\bpsi^{'}+(1-\rho)\bpsi$;
soft update $\bphi^{'}\gets\rho\bphi^{'}+(1-\rho)\bphi$;
the policy performs well in the real environment.

§.§ DIPO: Model-Free Learning with Diffusion Policy

Our source code follows the Algorithm <ref>.

§.§ Loss Function of DIPO

In this section, we provide the details of  presented in Algorithm <ref>.
We present the derivation of the loss of score matching (<ref>) and present the details of updating the diffusion from samples.
First, the next Theorem <ref> shows an equivalent version of the loss defined in (<ref>), then we present the learning details from samples.

§.§.§ Conditional Sampling Version of Score Matching

For give a partition on the interval $[0,T]$, $0=t_{0}<t_{1}<\cdots<t_{k}<t_{k+1}<\cdots<t_{K}=T$, let $\alpha_{0}=\mathrm{e}^{-2T}$,
$\alpha_{k}=\mathrm{e}^{2(-t_{k+1}+t_{k})},$ and $\bar{\alpha}_{k-1}=\prod_{k^{'}=0}^{k}\alpha_{k^{'}}$.
Setting $\omega(t)$ according to the next (<ref>), then the objective (<ref>) follows the next expectation version,

where $[K]=:\{1,2,\cdots,K\}$, $\calU(\cdot)$ denotes uniform distribution, the parametic funciton
\[\bm{\epsilon}_{\bphi}(\cdot,\cdot,\cdot):\calA\times\calS\times [K]\rightarrow\R^{p}\]
shares the parameter $\bphi$ according to:
\[\bm{\epsilon}_{\bphi}\left(\cdot,\cdot,k\right)

According to (<ref>), and Proposition <ref>, we know $ \varphi_{t}(\bara_t|\bara_0)=\calN\left(\mathrm{e}^{-t}\bara_0,\left(1-\mathrm{e}^{-2t}\right)\bI\right)$, then

where $\bz_{t}\sim\calN(\bm{0},\bI)$,

Let $\sigma_{t}=\sqrt{1-\mathrm{e}^{-2t}}$, according to (<ref>), we know

_t =e^-t_0+(√(1-e^-2t))_t=

where $\bz_{t}\sim\calN(\bm{0},\bI)$.

Recall (<ref>), we obtain

()=   _0^T

(<ref>)=   _0^Tω(t)σ_t^2

t←T-t=   _0^Tω(T-t)σ_T-t^2

Furthermore, we define an indicator function $\I_{t^{'}}(t)$ as follows,

\[
\I_{t^{'}}(t)=:\left\{
\begin{array}{rcl}
1 ,   & {\text{if}~t^{'}= t;}\\
0   ,     &  {\text{if}~t^{'}\ne t.}
\end{array} \right.
\]

Let the weighting function be defined as follows, for any $t\in[0,T]$,

where we give a partition on the interval $[0,T]$ as follows,

\[0=t_{0}<t_{1}<\cdots<t_{k}<t_{k+1}<\cdots<t_{K}=T.\]

Then, we rewrite (<ref>) as follows,

We consider the next term contained in (<ref>)

ŝ_(_T-t_k,,T-t_k)(<ref>)=   ŝ_(e^-(T-t_k)_0+σ_T-t_k_T-t_k,,T-t_k)

=   ŝ_(e^-(T-t_k)_0+√(1-e^-2(T- t_k))_T-t_k,,T-t_k),

where $\bz_{T-t_{k}}\sim\calN(\bm{0},\bI)$,
then obtain

__T- t_k|_0[ŝ_(_T-t_k,,T-t_k)+_T-t_k_2^2]

=   __T-t_k∼(0,)[σ_T-t_kŝ_(e^-(T-t_k)_0+√(1-e^-2(T- t_k))_T-t_k,,T-t_k)].

Now, we rewrite (<ref>) as the next expectation version,

()=_k∼([K]),_t_k∼(0,),_0∼π(·|)[σ_T-t_kŝ_(e^-(T-t_k)_0+√(1-e^-2(T- t_k))_T-t_k,,T-t_k)]

For $k=0,1,\cdots,K$, and $\alpha_{0}=\mathrm{e}^{-2T}$ and

Then we obtain

\[
\bar{\alpha}_{k}=\prod_{k^{'}=0}^{k-1}\alpha_{k^{'}}=\mathrm{e}^{-2(T- t_{k})}.
\]

With those notations, we rewrite (<ref>) as follows,

Finally, we define a function $\bm{\epsilon}_{\bphi}(\cdot,\cdot,\cdot):\calS\times\calA\times [K]\rightarrow\R^{p}$, and

i,e. we estimate the score function via an estimator $\epsilon_{\bphi}$ as follows,

Then we rewrite (<ref>) as follows,

\[
\calL(\bphi)=\E_{k\sim\calU([K]),\bz_{k}\sim\calN(\bm{0},\bI),\bara_{0}\sim\pi(\cdot|\bs)}\left[\bz_{k}-\bm{\epsilon}_{\bphi}\left(\sqrt{\bar{\alpha}_{k}}\bara_0+\sqrt{1-\bar{\alpha}_{k}}\bz_{{k}},\bs,k\right)\right].
\]

This concludes the proof.

§.§.§ Learning from Samples

According to the expectation version of loss (<ref>), we know, for each pair $(\bs,\ba)$ sampled from experience memory, let $k\sim \mathrm{Uniform}(\{1,\cdots,K\})$ and $\bz\sim \calN(\bm{0},\bI)$, the following empirical loss

\[\ell_{\mathrm{d}}(\bphi)=\|\bz-\bepsilon_{\bphi}\left(\sqrt{\bar\alpha_k}\ba+\sqrt{1-\bar{\alpha}_k}\bz,\bs,k\right)\|_{2}^{2}\]

is a unbiased estimator of $\calL(\bphi)$ defined in (<ref>).

Finally, we learn the parameter $\bphi$ by minimizing the empirical loss $\ell_{\mathrm{d}}(\bphi)$ according to gradient decent method:

\[\bphi\gets\bphi-\eta_{\phi}\bnabla_{\bphi}\left\|\bz-\bepsilon_{\bphi}\left(\sqrt{\bar\alpha_k}\ba+\sqrt{1-\bar{\alpha}_k}\bz,\bs,k\right)\right\|_{2}^{2},\]

where $\bepsilon_{\bphi}$ is the step-size.
For the implementation, see lines 25-28 in Algorithm <ref>.

§.§ Playing Actions of DIPO

Diffusion Policy (A Backward Version [Ho et al., 2020])

[1]
input state $\bs$; parameter $\bphi$; reverse length $K$;
initialize $\{\beta_{i}\}_{i=1}^{K}$; $\alpha_i=:1-\beta_i,\bar{\alpha}_k=:\prod_{i=1}^{k}\alpha_i,~\sigma_{k}=:\sqrt{\dfrac{1-\bar{\alpha}_{k-1}}{1-\bar{\alpha}_{k}}\beta_k}$;
initial $\hata_{K}\sim\calN(\bm{0},\bI)$;
$\bz_{k}\sim\calN(\bm{0},\bI)$, if $k>1$; else $\bz_{k}=0$;
return $\hata_{0}$

In this section, we present all the details of  presented in Algorithm <ref>.

Let $\beta_{k}=1-\alpha_{k}$, then according to Taylar formualtion, we know

Recall the exponential integrator discretization (<ref>), we know

_t_k+1-_t_k=   (e^t_k+1-t_k-1)(_t_k+2 s_(_t_k,,T-t_k))+√(2)∫_t_k^t_k+1e^t^'-t_k_t^',

which implies

_t_k+1=   _t_k+(e^t_k+1-t_k-1)(_t_k+2 s_(_t_k,,T-t_k))+√(e^2(t_k+1-t_k)-1)_t_k

(<ref>)=   _t_k+(1√(α_k)-1)(_t_k
+2 s_(_t_k,,T-t_k))+√(1-α_kα_k)_t_k

(<ref>)=   1√(α_k)_t_k-2(1√(α_k)-1) 1√(1-α̅_k) _(_t_k,,k)+√(1-α_kα_k)_t_k

=   1√(α_k)_t_k-β_k√(α_k)·1√(1-α̅_k) _(_t_k,,k)

where $\bz_{t_k}\sim\calN(\bm{0},\bI)$, Eq.(<ref>) holds since we use the fact (<ref>), which implies

\[
\]

To simplify the expression, we rewrite (<ref>) as follows,

_k+1=   1√(α_k)_k-β_k√(α_k)·1√(1-α̅_k)_(_k,,k)

=   1√(α_k)(_k-β_k√(1-α̅_k)_(_k,,k))

=   1√(α_k)(_k-1-α_k√(1-α̅_k)_(_k,,k))

where $k=0,1,\cdots,K-1$ runs forward in time, $\bz_{k}\sim\calN(\bm{0},\bI)$. The agent plays the last action $\hata_{K}$.

Since we consider the SDE of the reverse process (<ref>) that runs forward in time, while most diffusion probability model literature (e.g., [Ho et al., 2020, Song et al., 2021]) consider the backward version for sampling.
To coordinate the relationship between the two versions, we also present the backward version in Algorithm <ref>, which is essentially identical to the iteration (<ref>) but rewritten in the running in backward time version.

§ TIME DERIVATIVE OF KL DIVERGENCE BETWEEN DIFUFFUSION POLICY AND TRUE REVERSE PROCESS

In this section, we provide the time derivative of KL divergence between diffusion policy (Algorithm <ref>) and true reverse process (defined in (<ref>)).

§.§ Time Derivative of KL Divergence at Reverse Time $k=0$

In this section, we consider the case $k=0$ of diffusion policy (see Algorithm <ref> or the iteration (<ref>)).
If $k=0$, then for $0\leq t\leq h$, the SDE (<ref>) is reduced as follows,

_t=(_t+2 S(_0,,T))t+√(2)_t, t∈[0,h],

where $\bw_{t}$ is the standard Wiener process starting at $\bw_{0}=\bm{0}$.

Let the action $\hata_{t}\sim\hatpi_{t}(\cdot|\bs)$ follows the process (<ref>).
The next Proposition <ref> considers the distribution difference between the diffusion policy $\hatpi_{t}(\cdot|\bs)$ and the true distribution of backward process (<ref>) $\tildepi_{t}(\cdot|\bs)$ on the time interval $t\in[0,h]$.

Under Assumption  <ref> and <ref>, let $\tildepi_{t}(\cdot|\bs)$ be the distribution at time $t$ with the process (<ref>), and let $\hatpi_{t}(\cdot|\bs)$ be the distribution at time $t$ with the process (<ref>).

τ_0=:sup{t:te^t≤√(5ν)96L_sL_p}, τ=:min{ τ_0,112L_s},

and $0\leq t\leq h\leq \tau$, then the following equation holds,

Before we show the details of the proof, we need to define some notations, which is useful later.

denote the distribution of the action ${\hata_{t}}=\hata$ be played at time $t$ along the process (<ref>), where $t\in[0,h]$.
For each $t>0$, let $\rho_{0,t}({\hata_{0}},{\hata_{t}}|\bs)$ denote the joint distribution of $({\hata_{0}},{\hata_{t}})$ conditional on the state $\bs$,  which can be written in terms of the conditionals and marginals as follows,

\[\rho_{0|t}({\hata_{0}}|{\hata_{t}},\bs)=\dfrac{\rho_{0,t}({\hata_{0}},{\hata_{t}}|\bs)}{p({\hata_{t}}={\color{blue}{\hat{\ba}}}|\bs,t)}
\]

§.§ Auxiliary Results For Reverse Time $k=0$

Let $\hatpi_{t}({\color{blue}{\hat{\ba}}}|\bs)$ be the distribution at time $t$ along interpolation SDE (<ref>), where $\hatpi_{t}({\color{blue}{\hat{\ba}}}|\bs)$ is short for $p({\hata_{t}}={\color{blue}{\hat{\ba}}}|\bs,t)$, which is  the distribution of the action ${\hata_{t}}=\hata$ be played at time $t$ alongs the process (<ref>) among the time $t\in[0,h]$.
Then its derivation with respect to time satisfies

=-_t(|)·(+2__0∼ρ_0|t(·|,) [S(_0,,T)|_t=]

Before we show the details of the proof, we need to clear the divergence term $\DIV$. In this section, all the notation is defined according to (<ref>), and its value is at the point $\hata$.

For example, in Eq.(<ref>), the divergence term $\DIV$ is defined as follows,

·(+2__0∼ρ_0|t(·|,) [S(_0,,T)|_t=])=(·𝐩)(),

𝐩()=+2__0∼ρ_0|t(·|,) [S(_0,,T)|_t=].

For example, in Eq.(<ref>), the divergence term $\DIV$ is defined as follows,

·(p(|_0,,t)(+2 S(_0,,T)))=(·𝐩)(),

()=p(|_0,,t)(+2 S(_0,,T)).

Similar definitions are parallel in Eq.(<ref>), from Eq.(<ref>) to Eq.(<ref>).

First, for a given state $\bs$, conditioning on the initial action ${\hata_{0}}$, we introduce a notation

p(·|_0,,t): ^p→[0,1],

and each
\[p({\color{blue}{\hat{\ba}}}|{\hata_{0}},\bs,t)=:p({\hata_{t}}={\color{blue}{\hat{\ba}}}|{\hata_{0}},\bs,t)\]

that denotes the conditional probability distribution starting from ${\hata_{0}}$ to the action ${\hata_{t}}={\color{blue}{\hat{\ba}}}$ at time $t$ under the state $\bs$.
Besides, we also know,

=∫_^p ρ_0(_0)p(·|_0,,t)_0,

where $\rho_{0}(\cdot)=\calN(\bm{0},\bI)$ is the initial action distribution for reverse process.

For each $t>0$, let $\rho_{0,t}({\hata_{0}},{\hata_{t}}|\bs)$ denote the joint distribution of $({\hata_{0}},{\hata_{t}})$ conditional on the state $\bs$,  which can be written in terms of the conditionals and marginals as follows,

\[\rho_{0,t}({\hata_{0}},{\hata_{t}}|\bs)
\]

Then we obtain the Fokker–Planck equation for the distribution $p(\cdot|{\hata_{0}},\bs,t)$ as follows,

∂∂tp(|_0,,t)=-·(p(|_0,,t)(+2 S(_0,,T)))+Δp(|_0,,t),

where the $\DIV$ term is defined according to (<ref>) and (<ref>) if $\bp(\ba)=p(\ba|{\hata_{0}},\bs,t)\left(\ba+2 \bm{\hat \mathbf{S}}({\hata_{0}},\bs,T)\right)$.

Furthermore, according to (<ref>), we know

∂∂t_t(|)=   ∂∂t∫_^p ρ_0(_0)p(|_0,,t)_0
=∫_^p ρ_0(_0)∂∂t p(|_0,,t)_0

=   ∫_^p ρ_0(_0)
-·(p(|_0,,t)(+2 S(_0,,T)))+Δp(|_0,,t)

=   -_t(|)·
_t(|)__0∼ρ_0|t(·|,) [S(_0,,T)|_t=]

=   -_t(|)·(+2__0∼ρ_0|t(·|,) [S(_0,,T)|_t=]

where Eq.(<ref>) holds since: with the definition of $\hatpi_{t}({\color{blue}{\hat{\ba}}}|\bs)=:p({\color{blue}{\hat{\ba}}}|\bs,t)$, we obtain

∫_^p ρ_0(_0)

we know

ρ_0(_0) p(|_0,,t)=p(,_0|,t), *$\blacktriangleright$ Bayes' theorem

p(,_0|,t)=   p(|,t)
_t(|)p(_0|_t=,,t) ,

then we obtain

-∫_^p ρ_0(_0)
·(p(|_0,,t) S(_0,,T))

=   -∫_^p ·(p(,_0|,t) S(_0,,T))

=   -∫_^p ·(_t(|)p(_0|_t=,,t)  S(_0,,T))
_0  *$\blacktriangleright$ see Eq.(<ref>)

=   -·(
_t(|)∫_^p  p(_0|_t=,,t)  S(_0,,T)

=   -·(
_t(|)__0∼ρ_0|t(·|,) [S(_0,,T)|_t=]

where the last equation holds since

∫_^p  p(_0|_t=,,t)  S(_0,,T)
=__0∼ρ_0|t(·|,) [S(_0,,T)|_t=].

Finally, consider (<ref>) with (<ref>) and (<ref>), we conclude the Lemma <ref>.

We consider the time derivative of KL-divergence between the distribution $\hatpi_t(\cdot|\bs)$ and $\tildepi_{t}(\cdot|\bs)$, and decompose it as follows.

The time derivative of KL-divergence between the distribution $\hatpi_t(\cdot|\bs)$ and $\tildepi_{t}(\cdot|\bs)$ can be decomposed  as follows,

We consider the time derivative of KL-divergence between the distribution $\hatpi_t(\cdot|\bs)$ and $\tildepi_{t}(\cdot|\bs)$, and we know

t(_t(·|)_t(·|))=   t_^p_t(|)log_t(|)_t(|)

=   _^p∂_t(|)∂tlog_t(|)_t(|)+_^p_t(|)∂∂t(_t(|)_t(|))

=   _^p∂_t(|)∂tlog_t(|)_t(|)+_^p(∂_t(|)∂t- _t(|)_t(|)∂_t(|)∂t)

=   _^p∂_t(|)∂tlog_t(|)_t(|)-_^p_t(|)_t(|)∂_t(|)∂t,

where the last equation holds since

\[
\bigintsss_{\R^{p}}\dfrac{\partial\hatpi_t(\ba|\bs)}{\partial t}\dd\ba=\dfrac{\dd}{\dd t}\underbrace{\bigintsss_{\R^{p}}\hatpi_t(\ba|\bs)\dd\ba}_{=1}=0.
\]

That concludes the proof.

The relative entropy and relative Fisher information $\FI \big(\hatpi_t(\cdot|\bs)\|\tildepi_t(\cdot|\bs)\big)$ can be rewritten as follows.

The relative entropy and relative Fisher information $\FI \big(\hatpi_t(\cdot|\bs)\|\tildepi_t(\cdot|\bs)\big)$ can be rewritten as the following identity,

We consider the following identity,

=   _^p(⟨_t(|)_t(|)-_t(|)_t(|)_t(|),log_t(|)⟩-_t(|)⟨log_t(|),log_t(|)_t(|)⟩)

=   _^p_t(|)⟨log_t(|)_t(|),log_t(|)⟩-_^p_t(|)⟨log_t(|),log_t(|)_t(|)⟩

=   -_^p_t(|)⟨log_t(|)_t(|),log_t(|)_t(|)⟩

=   -_^p_t(|)log_t(|)_t(|)_2^2=:-(_t(·|)_t(·|)),

which concludes the proof.

Lemma <ref> implies the following identity, which is useful later,

=   _^p(⟨_t(|)_t(|),_t(|)⟩-⟨_t(|),log_t(|)_t(|)⟩-2⟨_t(|)_t(|),_t(|)⟩)

=   -(_t(·|)_t(·|))-2_^p_t(|)⟨log_t(|)_t(|),log_t(|)⟩.

The time derivative of KL-divergence between the distribution $\hatpi_t(\cdot|\bs)$ and $\tildepi_{t}(\cdot|\bs)$ can be further decomposed  as follows,

According to Lemma <ref>, we need to consider the two terms in (<ref>) correspondingly.

First term in (<ref>).
Recall Lemma <ref>, we know

∂∂t_t(|)=   -_t(|)·(+2__0∼ρ_0|t(·|,) [S(_0,,T)|_t=]

(<ref>)=   ·(-
(+2__0∼ρ_0|t(·|,) [S(_0,,T)|_t=]

To short the expression, we define a notation $\bg_{t}(\cdot,\cdot):\calS\times\calA\rightarrow\R^{p}$ as follows,

\[\bg_{t}(\bs,\ba)=:\ba+2\E_{{\hata_{0}}\sim\rho_{0|t}(\cdot|\ba,\bs) }\big[\bm{\hat \mathbf{S}}({\hata_{0}},\bs,T)\big|{\hata_{t}}={\ba}\big],
\]

then we rewrite the distribution at time $t$ along interpolation SDE (<ref>) as follows,

We consider the first term in (<ref>), according to integration by parts formula (<ref>), we know

_^p∂_t(|)∂tlog_t(|)_t(|)=   _^p·(-_t(,)_t(|)+_t(|))log_t(|)_t(|)

(<ref>)=   _^p⟨_t(,)_t(|)-_t(|),log_t(|)_t(|)⟩

=   _^p_t(|)⟨_t(,),log_t(|)_t(|)⟩

Second term in (<ref>). According to the Kolmogorov backward equation, we know

then we obtain

_^p_t(|)_t(|)∂_t(|)∂t=   -_^p_t(|)_t(|)·(_t(|)+_t(|))

(<ref>)=   _^p⟨_t(|)_t(|),_t(|)+_t(|)⟩

=   _^p_t(|)⟨_t(|)_t(|),⟩+

Time derivative of KL-divergence. We consider the next identity,

=   _^p(_t(|)⟨_t(,),log_t(|)_t(|)⟩-_t(|)_t(|)_t(|)⟨_t(|)_t(|),⟩)

=   2_^p_t(|)⟨__0∼ρ_0|t(·|,) [S(_0,,T)|_t=],log_t(|)_t(|)⟩,

then according to (<ref>), and with the results (<ref>), (<ref>), we obtain

+2_^p_t(|)⟨log_t(|)_t(|),__0∼ρ_0|t(·|,) [S(_0,,T)|_t=]-log_t(|)⟩.

Furthermore, we consider

_^p_t(|)⟨log_t(|)_t(|),__0∼ρ_0|t(·|,) [S(_0,,T)|_t=]-log_t(|)⟩

=   _^p_^pρ_0,t(_0,|)⟨log_t(|)_t(|),S(_0,,T)-log_t(|)⟩_0,

where Eq.(<ref>) holds due to $\rho_{0,t}({\hata_{0}},{\hata_{t}}|\bs)$ denotes the joint distribution of $({\hata_{0}},{\hata_{t}})$ conditional on the state $\bs$,  which can be written in terms of the conditionals and marginals as follows,

and in Eq.(<ref>), we denote ${\hata_{t}}={\ba}$.

Finally, combining (<ref>) and (<ref>), we obtain the following equation,

which concludes the proof.

The time derivative of KL-divergence between the distribution $\hatpi_t(\cdot|\bs)$ and $\tildepi_{t}(\cdot|\bs)$ is bounded as follows,

≤   -34(_t(·|)_t(·|))+4_^p_^pρ_0,t(_0,|)S(_0,,T)-log_t(|)_2^2

First, we consider

≤   _^p_^pρ_0,t(_0,|)(2S(_0,,T)-log_t(|)_2^2+18log_t(|)_t(|)_2^2

=   2_^p_^pρ_0,t(_0,|)S(_0,,T)-log_t(|)_2^2

where Eq.(<ref>) holds since we consider $\langle \ba,\bb\rangle\leq 2\|\ba\|^2+\frac{1}{8}\|\bb\|^2$.

Then, according to Lemma <ref>, we obtain

≤   -34(_t(·|)_t(·|))+4_^p_^pρ_0,t(_0,|)S(_0,,T)-log_t(|)_2^2

which concludes the proof.

Before we provide further analysis to show the boundedness of (<ref>)., we need to consider SDE (<ref>).
Let $h>0$ be the step-size, assume $K=\frac{T}{h}\in\N$, and $t_{k}=:hk$, $k=0,1,\cdots,K$.
SDE (<ref>) considers as follows, for $t\in[hk,h(k+1)]$,

_t=(_t+2 S(_t_k,,T-t_k))t+√(2)_t,

Recall the SDE (<ref>), in this section, we only consider $k=0$, and we obtain the following SDE,

_t=(_t+2 S(_0,,T))t+√(2)_t,

where $\bw_{t}$ is the standard Wiener process starting at $\bw_{0}=\bm{0}$, and $t$ is from $0$ to $h$.

Integration with (<ref>), we obtain

_t-_0=(e^t-1)(_0+2 S(_0,,T))+√(2)∫^t_0e^t_t,

which implies

_t=e^t_0+2(e^t-1) S(_0,,T)+√(e^t-1), ∼(0,).

Under Assumption <ref>, for all $0\leq t\leq\frac{1}{12L_{s}}$, then the following holds,

≤   36pt(1+t)L^2_s+

where $\hata_t$ updated according to (<ref>).

See Section <ref>.

§.§ Proof for Result at Reverse Time $k = 0$

According to the definition of diffusion policy, we know $\tildepi_{t}(\cdot|\bs)=\barpi_{T-t}(\cdot|\bs)$. Then according to Proposition  <ref>, we know $\tildepi_{t}(\cdot|\bs)$ is $\nu_{T-t}$-LSI, where

\[
\nu_{T-t}=\dfrac{\nu}{\nu+(1-\nu)\mathrm{e}^{-2(T-t)}}.
\]

Since we consider the time-step $0\leq t\leq T$, then

ν_T-t=νν+(1-ν)e^-2(T-t)≥1, ∀t∈[0,T].

According to Proposition <ref>, we know under Assumption <ref>, $\bnabla\log {\color{orange}{\tilde{\pi}_{t}}}(\cdot|\bs)$ is $L_{p}\mathrm{e}^{t}$-Lipschitz on the time interval $[0,\mathtt{T}_{0}]$, where

\[
\mathtt{T}_{0}=:\sup_{t\ge0}\left\{t:1-\mathrm{e}^{-2t}\leq \frac{\mathrm{e}^{t}}{L_p}\right\}.
\]

Then according to Proposition <ref>, we obtain

Furthermore, according to Donsker-Varadhan representation (see Section <ref>), let
\[ f(\ba)=:{\beta_t}\Big\|\bm{\hat \mathbf{S}}(\ba,\bs,T)-\bnabla\log\tildepi_t(\ba|\bs)\Big\|_{2}^{2}

the positive constant ${\beta_t}$ will be special later, see Eq.(<ref>).
With the result (<ref>), we know

which implies

≤   1β_t  (_t(·|)_t(·|))+1β_t log_^p_t(|)exp(S(,,T)-log_t(|)_2^2)

=   1β_t  (_t(·|)_t(·|))+1β_t log_∼_t(·|)[expS(,,T)-log_t(|)_2^2].

Finally, according to Lemma <ref>-<ref>, Eq.(<ref>)-(<ref>), we obtain

(<ref>)≤   -34(_t(·|)_t(·|))+4_^p_^pρ_0,t(_0,|)S(_0,,T)-log_t(|)_2^2

Lemma <ref>≤   -34(_t(·|)_t(·|))+576t^2L^2_s_^p_t(|)log_t(|)_2^2

≤   -34(_t(·|)_t(·|))+576t^2L^2_s(4L_p^2e^2tν_T-t(_t(·|)_t(·|))+2pL_pe^t) *$\blacktriangleright$ due to Eq.(<ref>)

+576t^2L^2_sβ_t ( (_t(·|)_t(·|))+ log_∼_t(·|)[expS(,,T)-log_t(|)_2^2])*$\blacktriangleright$ due to Eq.(<ref>)

=   -34(_t(·|)_t(·|))+576t^2L^2_s(4L_p^2e^2tν_T-t+1β_t)(_t(·|)_t(·|))

≤   (576t^2L^2_s(4L_p^2e^2tν_T-t+1β_t)-32ν)(_t(·|)_t(·|))
*$\blacktriangleright$ due to Assumption <ref>

=   (576t^2L^2_s(4c_t+1β_t)-32ν)(_t(·|)_t(·|))
*$\blacktriangleright$ due to $\frac{L_{p}^2\mathrm{e}^{2t}}{\nu_{T-t}}=:c_{t}$

(<ref>)=   -ν4(_t(·|)_t(·|))

(<ref>)≤   -ν4(_t(·|)_t(·|))

Eq.(<ref>) holds since we set ${\beta_t}$ as follows,
we set

where Eq.(<ref>) holds since

Now, we consider the time-step $t$ keeps the constant $\beta_t$ positive, it is sufficient to consider the next condition due to the property (<ref>),

which implies $ t\mathrm{e}^{t}\leq\dfrac{\sqrt{5\nu}}{96L_{s}L_{p}}.$

Formally, we define a notation

τ_0   =:sup{t:te^t≤√(5ν)96L_sL_p},

τ   =:min{ τ_0,𝚃_0,112L_s}.

Then with result of Eq.(<ref>), if $0\leq t\leq h\leq \tau$, we rewrite Eq.(<ref>) as follows,

=   -ν4(_t(·|)_t(·|))+12pL_s√(5ν)t

+54νsup_(k,t)∈[K]×[t_k,t_k+1]   {log_∼_t(·|)[expS(,,T-hk)-log_t(|)_2^2]},

which concludes the proof.

The result of Proposition <ref> only depends on Assumption <ref>, thus the result (<ref>) does not depend on additional assumption of the uniform $L$-smooth of $\log\tildepi_{t}$ on the time interval $[0,T]$, e.g., [Wibisono and Yang, 2022]. Instead of  the uniform $L$-smooth of $\log\tildepi_{t}$, we consider the $L_{p}\mathrm{e}^{t}$-Lipschitz on the time interval $[0,\mathtt{T}_{0}]$, which is one of the difference between our proof and [Wibisono and Yang, 2022]. Although we obtain a similar convergence rate from the view of Langevin-based algorithms, we need a weak condition.

§.§ Proof for Result at Arbitrary Reverse Time $k$

Under Assumption <ref> and <ref>.
Let $\tildepi_{k}(\cdot|\bs)$ be the distribution at the time $t=hk$ along the process (<ref>) that starts from $\tildepi_{0}(\cdot|\bs)=\barpi_{T}(\cdot|\bs)$, then $\tildepi_{k}(\cdot|\bs)=\barpi_{T-hk}(\cdot|\bs)$.
Let $\hatpi_{k}(\cdot|\bs)$ be the distribution of the iteration (<ref>) at the $k$-the time $t_{k}=hk$, starting from $\hatpi_{0}(\cdot|\bs)=\calN(\bm{0},\bI)$.
Let $0<h\leq \tau$, then for all $k=0,1,\cdots,K-1$,

≤   e^-1/4νh(_k(·|)_k(·|))+54νϵ_score h+12pL_s√(5ν)h^2,

where $\tau$ is defined in (<ref>).

Recall Proposition <ref>, we know for any $0\leq t\leq h\leq \tau$, the following holds

where comparing to (<ref>), we use the condition $t\leq h$.

We rewrite (<ref>) as follows,

Then, on the interval $[0,h]$, we obtain

≤_0^h e^1/4νt

which implies

Furthermore, we obtain

≤   e^-1/4νh(_0(·|)_0(·|))+4ν(1-e^-1/4νh)(54νϵ_score+12pL_s√(5ν)h)

≤   e^-1/4νh(_0(·|)_0(·|))+54νϵ_score h+12pL_s√(5ν)h^2,

where last equation holds since we use $1-\mathrm{e}^{-x}\leq x$, if $x\ge0$.

Recall $\tildepi_{k}(\cdot|\bs)$ is the distribution at the time $t=hk$ along the process (<ref>) that starts from $\tildepi_{0}(\cdot|\bs)=\barpi_{T}(\cdot|\bs)$, then $\tildepi_{k}(\cdot|\bs)=\barpi_{T-hk}(\cdot|\bs)$.

Recall $\hatpi_{k}(\cdot|\bs)$ is the distribution of the iteration (<ref>) at the $k$-the time $t_{k}=hk$, starting from $\hatpi_{0}(\cdot|\bs)=\calN(\bm{0},\bI)$.

According to (<ref>), we rename the $\tildepi_{0}(\cdot|\bs)$ with $\tildepi_{k}(\cdot|\bs)$, $\tildepi_{h}(\cdot|\bs)$ with $\tildepi_{k+1}(\cdot|\bs)$,
$\hatpi_{0}(\cdot|\bs)$ with $\hatpi_{k}(\cdot|\bs)$ and $\hatpi_{h}(\cdot|\bs)$ with $\hatpi_{k+1}(\cdot|\bs)$, then we obtain

≤   e^-1/4νh(_k(·|)_k(·|))+54νϵ_score h+12pL_s√(5ν)h^2,

which concludes the result.

§ PROOF OF THEOREM <REF>

Theorem <ref> (Finite-time Analysis of Diffusion Policy).

For a given state $\bs$, let $\{\barpi_{t}(\cdot|\bs)\}_{t=0:T}$ and $\{\tildepi_{t}(\cdot|\bs)\}_{t=0:T}$ be the distributions along the Ornstein-Uhlenbeck flow (<ref>) and (<ref>) correspondingly, where $\{\barpi_{t}(\cdot|\bs)\}_{t=0:T}$ starts at $\barpi_{0}(\cdot|\bs)=\pi(\cdot|\bs)$ and
$\{\tildepi_{t}(\cdot|\bs)\}_{t=0:T}$ starts at $\tildepi_{0}(\cdot|\bs)=\barpi_{T}(\cdot|\bs)$.
Let $\hatpi_{k}(\cdot|\bs)$ be the distribution of the exponential integrator discretization iteration (<ref>) at the $k$-the time $t_{k}=hk$, i.e.,
$\hata_{t_k}\sim \hatpi_{k}(\cdot|\bs)$ denotes the distribution of the diffusion policy (see Algorithms <ref>) at the time $t_{k}=hk$.
Let $\{\hatpi_{k}(\cdot|\bs)\}_{k=0:K}$ be starting at $\hatpi_{0}(\cdot|\bs)=\calN(\bm{0},\bI)$, under Assumption <ref> and <ref>, let the reverse length $K$ satisfy

\[K\ge
\]

\[\tau_{0}=:\sup_{t\ge0}\left\{t:t\mathrm{e}^{t}\leq\dfrac{\sqrt{5\nu}}{96L_{s}L_{p}}\right\},\mathtt{T}_{0}=:\sup_{t\ge0}\left\{t:1-\mathrm{e}^{-2t}\leq \frac{\mathrm{e}^{t}}{L_p}\right\}.\]

Then the KL-divergence between the diffusion policy $\hata_{K}\sim\hatpi_{K}(\cdot|\bs)$ and input policy $\pi(\cdot|\bs)$ is upper-bounded as follows,

(_K(·|)π(·|))≤e^-9/4νhK((0,)π(·|))_convergence of forward process+64pL_s√(5ν)·TK_errors from discretization

+203sup_(k,t)∈[K]×[t_k,t_k+1]{log_∼_t(·|)[expS(,,T-hk)-log_t(|)_2^2]}_errors from score matching.

Recall $\tildepi_{k}(\cdot|\bs)=\barpi_{T-hk}(\cdot|\bs)$, then we know

then according to Proposition <ref>, we know

(_K(·|)π(·|))(<ref>)=   (_K(·|)_K(·|))

≤   e^-1/4νK(_0(·|)_0(·|))+∑_j=0^K-1e^-1/4νh j(54νϵ_score h+12pL_s√(5ν)h^2)

≤   e^-1/4νK(_0(·|)_0(·|))+11-e^-1/4νh(54νϵ_score h+12pL_s√(5ν)h^2)

≤   e^-1/4νK(_0(·|)_0(·|))+163νh(54νϵ_score h+12pL_s√(5ν)h^2)

=   e^-1/4νK(_0(·|)_0(·|))+203ϵ_score+64√(5ν)pL_sh,

where Eq.(<ref>)  holds since we consider the

1-e^-x≥34x,  if 0<x≤14,

and we set the step-size $h$ satisfies the next condition:

\[h\nu\leq 1,~\text{i.e.,}~h\leq\dfrac{1}{\nu}.\]

Let $\xi(\cdot)$ be standard Gaussian distribution on $\R^{p}$, i.e., $\xi(\cdot)\sim\calN(\bm{0},\bI)$, then we obtain the following result: for a given state $\bs$,

t(ξ(·)_t(·|))   =t_^pξ()logξ()_t(|)

)*$\blacktriangleright$ Fokker–Planck Equation

=_^p⟨ξ()_t(|),_t(|)log_t(|)ξ()⟩*$\blacktriangleright$ Integration by Parts

≤-2ν_t(ξ(·)_t(·|))*$\blacktriangleright$ Assumption <ref> and Proposition <ref>

where the last equation holds since $\mathrm{e}^{-t}\leq 1$ with $t\ge0$.

Eq.(<ref>) implies

\[
\dfrac{\dd}{\dd t}\log\KL \left(\xi(\cdot)\|\barpi_{t}(\cdot|\bs)\right)\leq-2\nu,
\]

integrating both sides of above equation on the interval $[0,T]$, we obtain

According to definition of diffusion policy, since: $\hata_{0}\sim\calN(\bm{0},\bI)$, and $\tildepi_{0}(\cdot|\bs)\overset{(\ref{relation-policy-revers-forward})}=\barpi_{T}(\cdot|\bs)$, then we know

which implies

Combining (<ref>) and (<ref>), we obtain

(_K(·|)π(·|))≤   e^-1/4νhK-T(ξ(·)_0(·|))+203ϵ_score+64√(5ν)pL_sh

(<ref>)=   e^-9/4νhK((0,)π(·|))+203ϵ_score+64√(5ν)pL_sh.

Recall the following conditions (<ref>), (<ref>), and (<ref>) on the step-size $h$,

\[
h\leq \min\left\{ \tau_{0},\mathtt{T}_{0},\dfrac{1}{12L_{s}},\dfrac{1}{\nu}\right\},
\]

which implies the reverse length $K$ satisfy the following condition

\[
K=\dfrac{T}{h}\ge T\cdot\max\left\{\dfrac{1}{\tau_{0}},\dfrac{1}{\mathtt{T}_{0}},12L_{s},\nu\right\}.
\]

Finally, recall the definition of $\epsilon$ (<ref>), we rewrite (<ref>) as follows

which concludes the proof.

§ ADDITIONAL DETAILS

§.§ Proof of Lemma <ref>

Under Assumption <ref>, for all $0\leq t^{'}\leq T$, if $t\leq\frac{1}{12L_{s}}$, then for any given state $\bs$

where $\hata_t$ updated according to (<ref>).

(of Lemma <ref>).
First, we consider

≤    L_s_t-_0

=   (e^t-1)_0+2(e^t-1) S(_0,,t^')+√(e^t-1)

≤   2L_st_0_2^2+4L_stS(_0,,t^')+2L_s√(t),

where the last equation holds due to $\mathrm{e}^{t}-1\leq 2t$.

Furthermore, we consider the case with $t\leq\frac{1}{12L_s}$, then we obtain the boundedness of the term

S(_0,,t^')≤    S(_t,,t^')+L_s_t-_0

≤    S(_t,,t^')+2L_st_0+4L_stS(_0,,t^')+2L_s√(t)

≤    S(_t,,t^')+2L_st_0+13S(_0,,t^')+2L_s√(t),

which implies

Taking Eq.(<ref>) into Eq.(<ref>), and with $t\leq\frac{1}{12L_s}$, we obtain

Finally, we know

which concludes the proof of Lemma <ref>.

§.§ Proof of Lemma <ref>

(Lemma <ref>)
Recall the update rule of $\hata_t$ (<ref>),

\[
\hata_t=\mathrm{e}^{t}\hata_0+2(\mathrm{e}^{t}-1) \bm{\hat \mathbf{S}}(\hata_0,\bs,T)+\sqrt{\mathrm{e}^{t}-1}\bz,~\bz\sim\calN(\bm{0},\bI).
\]

To simplify the expression, in this section, we introduce the following notation

∼ρ_z(·), where ρ_z(·)=(0,).

According to the definition of $\rho_{0,t}({\hata_{0}},{\hata_{t}}|\bs)$ (<ref>), we denote
$\hata_t=\ba$, then we know,

≤   2_^p_^pρ_0,t(_0,|)(S(_0,,T)-S(_t,,T)_2^2

=   2_^p_^pρ_0,t(_0,|)(S(_0,,T)-S(,,T)_2^2

Recall Lemma <ref>, we know

(<ref>)≤   _^p_^p_^pρ_0,t(_0,|)ρ_z()(36L^2_st^2_0_2^2+72L^2_st^2S(, ,T)_2^2+36L^2_st^2_2)_0

=   _^p_^p_^pρ_0,t(_0,|)ρ_z()(36L^2_st^2_0_2^2+72L^2_st^2S(,,T)_2^2)_0

=   36L^2_st^2_^p_0(_0|)_0_2^2_0

=   36L^2_spt^2+

≤   36pt(1+t)L^2_s+

where the first term in Eq.(<ref>) holds since:

\[
\int_{\R^{p}}\int_{\R^{p}}\rho_{0,t}({\hata_{0}},\ba|\bs)\rho_{z}(\bz)\dd\ba\dd\bz=\hatpi_{0}({\hata_{0}}|\bs);
\]

the second term in Eq.(<ref>) holds since:

\[
\int_{\R^{p}}\int_{\R^{p}}\int_{\R^{p}}\rho_{0,t}({\hata_{0}},\ba|\bs)\rho_{z}(\bz)\dd {\hata_{0}}\dd\bz=\hatpi_{t}(\ba|\bs);
\]

the third term in Eq.(<ref>) holds since: $\bz\sim\calN(\bm{0},\bI)$, then $\|\bz\|_{2}^{2}\sim\chi ^{2}(p)$-distribution with $p$ degrees of freedom, then

Eq.(<ref>) holds with the same analysis of (<ref>), since ${\hata_{0}}\sim\calN(\bm{0},\bI)$, then $\|{\hata_{0}}\|_{2}^{2}\sim\chi ^{2}(p)$, which implies

\[
\int_{\R^{p}}\hatpi_{0}({\hata_{0}}|\bs)\left\|\hata_{0}\right\|_{2}^{2}\dd {\hata_{0}}=p;
\]

Eq.(<ref>) holds since we use the fact: $\|\langle \bm{\alpha}+\bm{\beta}\rangle\|_{2}^{2}\leq 2\|\bm{\alpha}\|_{2}^{2}+2\|\bm{\beta}\|_{2}^{2}$.

§ DETAILS AND DISCUSSIONS FOR MULTIMODAL EXPERIMENTS

In this section, we present all the implementation details and the plots of both 2D and 3D Visualization.
Then we provide additional discussions for empirical results of the task of the multimodal environment in Section <ref>.

[1E$3$ iterations]

[2E$3$ iterations]

[3E$3$ iterations]

[4E$3$ iterations]

[5E$3$ iterations]

[6E$3$ iterations]

[7E$3$ iterations]

[8E$3$ iterations]

[9E$3$ iterations]

[10E$3$ iterations]

Policy representation comparison of diffusion policy with different iterations.

§.§ Multimodal Environment

In this section, we clarify the task and reward of the multimodal environment.

§.§.§ Task

We design a simple “multi-goal” environment according to the Didactic Example [Haarnoja et al., 2017], in which the agent is a 2D point mass on the $7\times7$ plane, and the agent tries to reach one of four points $(0,5)$, $(0,-5)$, $(5,0)$ and $(-5,0)$ symmetrically placed goals.

[1E$3$ iterations]

[2E$3$ iterations]

[3E$3$ iterations]

[4E$3$ iterations]

[5E$3$ iterations]

[6E$3$ iterations]

[7E$3$ iterations]

[8E$3$ iterations]

[9E$3$ iterations]

[10E$3$ iterations]

Policy representation comparison of SAC with different iterations.

§.§.§ Reward
The reward is defined according to the following three parts:

\[
\]

* $r_{1}\propto -\|\ba\|_{2}^{2}$ if agent plays the action $\ba$;

* $r_{2}=-\min\{\|(x,y)-\mathbf{target}\|^{2}_{2}\}$, $\mathbf{target}$ denotes one of the target points $(0,5)$, $(0,-5)$, $(5,0)$, and $(-5,0)$;

* if the agent reaches one of the targets among $\{(0,5),~(0,-5),~(5,0),~(-5,0)\}$, then it receives a reward $r_{3}=10$.

Since the goal positions are symmetrically distributed at the four points $(0,5)$, $(0,-5)$, $(5,0)$ and $(-5,0)$,
a reasonable policy should be able to take actions uniformly to those four goal positions with the same probability, which characters the capacity of exploration of a policy to understand the environment. Furthermore, we know that the shape of the reward curve should be symmetrical with four equal peaks.

§.§ Plots Details of Visualization

[1E$3$ iterations]

[2E$3$ iterations]

[3E$3$ iterations]

[4E$3$ iterations]

[5E$3$ iterations]

[6E$3$ iterations]

[7E$3$ iterations]

[8E$3$ iterations]

[9E$3$ iterations]

[10E$3$ iterations]

Policy representation comparison of TD3 with different iterations.

This section presents all the details of the 2D and 3D visualization for the multi-goal task. At the end of this section, we present the shape of the reward curve.

§.§.§ 2D Visualization
For the 2D visualization, the red arrowheads denote actions learned by the corresponding RL algorithms, where each action starts at one of the totals of $7\times7=49$ points (corresponding to all the states) with horizontal and vertical coordinates ranges among $\{-3, -2, -1,0,1,2,3\}\times \{-3, -2, -1,0,1,2,3\}$. The length of the red arrowheads denotes the length of the action vector, and the direction of the red arrowheads denotes the direction of actions.
This is to say; for each figure, we plot all the actions starting from the same coordinate points.

§.§.§ 3D Visualization

For the 3D visualization, we provide a decomposition of the the region $[-7,7]\times[-7,7]$ into $100\times100=10000$ points, each point $(x,y)\in [-7,7]\times[-7,7]$ denotes a state.
For each state $(x,y)$, a corresponding action is learned by its corresponding RL algorithms, denoted as $\ba$.
Then according to the critic neural network, we obtain the state-action value function $Q$ value of the corresponding point $((x,y),\ba)$. The 3D visualization shows the state-action $Q$ (for PPO, is value function $V$) with respect to the states.

§.§.§ Shape of Reward Curve

Since the shape of the reward curve is symmetrical with four equal peaks, the 2D visualization presents the distribution of actions toward those four equal peaks.
A good algorithm should take actions with a uniform distribution toward those four points $(0,5)$, $(0,-5)$, $(5,0)$, and $(-5,0)$ on the  2D visualization.
The 3D visualization presents the learned shape according to the algorithm during the learning process. A good algorithm should fit the symmetrical reward shape with four equal peaks.
A multimodal policy distribution is efficient for exploration, which may lead an agent to learn a good policy and perform better.
Thus, both 2D and 3D visualizations character the algorithm's capacity to represent the multimodal policy distribution.

[1E$3$ iterations]

[2E$3$ iterations]

[3E$3$ iterations]

[4E$3$ iterations]

[5E$3$ iterations]

[6E$3$ iterations]

[7E$3$ iterations]

[8E$3$ iterations]

[9E$3$ iterations]

[10E$3$ iterations]

Policy representation comparison of PPO with different iterations.

§.§ Results Report

We have shown all the results in Figure <ref> (for diffusion policy), <ref> (for SAC), <ref> (for TD3) and <ref> (for PPO), where we train the policy with a total $10000$ iterations, and show the 2D and 3D visualization every $1000$ iteration.