# Fast dispersion tailoring of multi-mode photonic crystal resonators

Francesco Rinaldo Talenti1,2,∗, Stefan Wabnitz1,3, Inès Ghorbel2, Sylvain
Combrié2, Luca Aimone-Giggio2, Alfredo De Rossi2 1Dipartimento di Ingegneria
dell’Informazione, Elettronica e Telecomunicazioni, Sapienza University of
Rome, 00184 Rome, Italy.
2 Thales Research and Technology, Campus Polytechnique, 1 Avenue Augustin
Fresnel, 91767 Palaiseau, France.
3 CNR-INO, Istituto Nazionale di Ottica, Via Campi Flegrei 34, 80078 Pozzuoli
(NA), Italy. ∗Corresponding author<EMAIL_ADDRESS>
###### Abstract

We introduce a numerical procedure which permits to drastically accelerate the
design of multimode photonic crystal resonators. Specifically, we demonstrate
that the optical response of an important class of such nanoscale structures
is reproduced accurately by a simple, one-dimensional model, within the entire
spectral range of interest. This model can describe a variety of tapered
photonic crystal structures. Orders of magnitude faster to solve, our approach
can be used to optimize certain properties of the nanoscale cavity. Here we
consider the case of a nanobeam cavity, where the confinement results from the
modulation of its width. The profile of the width is optimized, in order to
flatten the resonator dispersion profile (so that all modes are equally spaced
in frequency). This result is particularly relevant for miniaturizing
parametric generators of non-classical light, optical nano-combs and mode-
locked laser sources. Our method can be easily extended to complex geometries,
described by multiple parameters.

##  I Introduction

The nonlinear interaction among several resonant fields in an optical
resonator leads to efficient Raman and Brillouin scattering, three and four-
wave-mixing, optical parametric oscillation[1], laser mode locking and
frequency comb generation[2]. Scaling down the size of optical resonators
implies that the optical power level for triggering nonlinear effects
decreases as $V^{-1}$ or $V^{-2}$, where $V$ is an effective volume of the
spatial distribution of the interacting fields. In the context of photonic
integration, the decrease of the power budget is of paramount importance.
Nanoscale optical resonators such as photonic crystals are able to confine
light within $V\approx\lambda^{3}$, i.e., a wavelength-sized volume, with a
photon decay time, or interaction time, well above 1 ns (i.e., the cavity
quality factor $Q\gg 10^{6}$). Owing to these properties, it has been possible
to demonstrate nanoscale lasers[3, 4, 5, 6], Raman sources[7] and, more
recently, optical parametric oscillators[8], all operating with a power supply
(optical or electrical) in the $\mu$W range. Yet, a major challenge remains in
achieving the nonlinear interaction of multiple longitudinal modes, as it
occurs in mode-locked lasers or in micro-combs. While ring or microdisk
resonators naturally provide the necessary, nearly frequency equispaced set of
cavity resonances, achieving the same condition in nanoscale resonators is
notoriously a nontrivial task. On the other hand, nanoscale resonators could,
in principle, be designed in a way that a specified number of modes, starting
from the fundamental, and only these, are allowed to take part to a nonlinear
interaction. This unique property implies not only that a much higher degree
of control on power transfer among modes (which is crucial in quantum and
signal processing applications[9]) can be achieved, but also leads to
maximizing the interaction efficiency. This is because, in the typical
configuration of a nanoscale resonator, the lowest order modes are also the
most tightly confined. Moreover, in a mode-locked nanolaser, the control of
the interacting modes would enable a favorable scaling of repetition rate vs.
the size of the device[10].
It has been shown that some specific designs of a photonic crystal cavity
lead, for some set of parameters, to frequency equispaced eigenmodes;
moreover, their mode envelopes are described by Hermite-Gauss functions. This
suggests that, within a certain spectral range, the complex photonic crystal
structure can be well approximated by a quantum-mechanical harmonic oscillator
model[11, 12]. It has also been shown that post-fabrication trimming is
effective in correcting for fabrication tolerances, thereby demonstrating an
almost perfect alignment of the cavity resonances[13]. Yet, a systematic
design approach for generating a given number of equispaced modes, or, more
generally, with a prescribed dispersion profile, while at the same time
maximizing the radiation-limited Q-factor, is still missing, while brute-force
methods are extremely inefficient.
Finding a cavity geometry, or more generally, a physical system whose response
to an input excitation corresponds to a well-defined target function, e.g., a
spatial distribution of the dielectric permittivity such that the
electromagnetic field has prescribed resonances, belongs to the class of
inverse problems, which are notoriously difficult to solve. Yet, the progress
of nanofabrication techniques has motivated the development of powerful
methods such as topological optimization (TO) [14] and inverse design (ID)
[15]. The common feature of these two approaches is that their result is a
spatial distribution of $\varepsilon(\mathbf{x})$, rather than an optimized
set of parameters for a pre-defined geometry. These methods are therefore able
to create novel geometries, hence the reference to design. Moreover, automatic
differentiation[16] and the adjoint method[17] enable a very efficient
computation of the gradient, which is required in the iterative search of the
optimum distribution, even in the presence of nonlinearity.

Figure 1: Common design of high-Q resonators based on gentle confinement: (a)
tapered Distributed Feedback Grating[18], (b) 1-D nanobeam with parabolic
width[19], (c) 1-D nanobeam bichromatic[20], and (d) 2-D bichromatic
resonator[11], and corresponding tapering parameter $\Delta$.

Here we follow a radically different approach, which is arguably more suited
for the class of problems under consideration. This is motivated by the fact
that the geometries of nanoscale resonators with the largest experimentally
reported Q factors[21, 22, 23, 18, 24] are still based on the principle of
gentle confinement[25]. In other words, these nanostructures are essentially
periodic, with an adiabatic tapering of some parameters, i.e., a gentle change
of the radius of the holes, the period or the magnitude of a ”dislocation”
defect, etc.. We note that more aggressive design strategies, including TO or
ID, have instead been considered for different tasks, e.g., for maximizing
light-matter interactions in single-mode resonators[16, 26].
Let us restrict our search to a family of structures which can be described by
means of a periodic pattern $\varepsilon(\mathbf{x},\Delta)$ that depends on a
control parameter $\Delta$, which is supposed to adiabatically vary in space
(i.e., gently). Some examples of such geometries are given in Fig. 1. The
crucial point is that it is possible to map the three-dimensional (3D) Maxwell
equations (ME) into an equivalent system of one-dimensional (1D) equations,
which will be referred to as the reduced model (RM). Remarkably, the relative
precision of the resonances predicted by the RM turns out to be at least as
good as the precision of the direct numerical solution of the 3D ME. The
search of the desired optimal spatial dependence of $\Delta$ will be performed
by using any suitable optimization method, leveraging on the extremely faster
solution of the RM, when compared the direct solution of the 3D ME. The RM in
itself only requires a single direct solution of the 3D ME, for building an
initial approximation of the structure. Subsequent applications of the RM are
used, in order to refine the first approximation. As we shall see, in total
only three 3D solves are sufficient for obtaining a design that matches our
target, with an accuracy that is equivalent to that of directly solving the 3D
ME, but with a comparatively much larger number of iterations.
Hereafter we will first discuss the derivation of the RM, then we will
formulate a design target followed by the introduction of the optimization
procedure, including model calibration. Finally, we will discuss possible
applications and generalizations of the model.

##  II Reduced model for a periodic photonic crystal

Figure 2: (a) Dispersion diagram of a periodic structure (see inset) with
$w=450$nm and period $a=465$nm, centered in the $K$ point of the reduced
Brillouin zone ($k_{0}=\pi/a$). The reduced model considers coupled forward
and backward waves (dashed lines), generating the valence and conduction bands
(solid lines). The filled circles represent the valence band calculated by
periodic 3D ME. (b) The corresponding residuals $\sigma/2\pi$ of the fit.

The reduced model is inspired by the so-called $\mathbf{k}\cdot\mathbf{p}$
method [27], which is used in solid state physics to model the electronic band
structure of crystals. The main idea of the method is to describe the
dispersion relation of the electronic bands (i.e., electron energy vs.
wavevector $\mathbf{k}$) through a suitable algebraic equation, which is built
upon the eigenfunctions of the exact Hamiltonian at the bands extrema (at
points of high symmetry, e.g., $\mathbf{k}=0$, or gamma point). Within a range
of energies of interest, the dispersion relation is extrapolated from the
gamma point by treating the $\mathbf{k}\cdot\mathbf{p}$ term as a
perturbation. In this way, the complexity of solving the full Schrödinger
equation for the crystal is reduced by using a much simpler model, where only
a few parameters need to be suitably adjusted. As a consequence of this
approach, a local modulation of a semiconductor, e.g., of a heterostructure,
can be well described in terms of a change of these parameters within the
energy range of interest. As a result, it is possible to introduce a much
simpler Schrödinger equation, which only depends on these parameters.
In optics, the simplest model describing the propagation of waves in a
periodic dielectric is provided by the case of a distributed Bragg reflector.
Here, a modulation with period $\Lambda$ of the dielectric permittivity
couples forward and backward waves. A simple algebraic equation approximates
the dispersion in the spectral range that is centered at the Bragg angular
frequency $\omega_{0}=\pi cn^{-1}\Lambda^{-1}$. Here $c$ is the speed of light
and $n$ is an effective refractive index, which describes the optical field
distribution as a result of the dielectric inhomogeneity [28]. In the presence
of an intensity dependent contribution to the refractive index, one obtains
coupled wave propagation equations which generalize the Massive Thirring model
of field theory; their solitary wave solutions (gap or Bragg solitons)
describe the localisation of wavepackets in periodic media[29]. The simplicity
of the gap soliton model has facilitated the study of soliton stability by
using analytical criteria[30]. Soliton dynamics has been experimentally
demonstrated in nanoscale photonics[31], and it has been shown that nonlinear
coupled waves models are able to fully capture the underlying physics[32].
Hereafter, we will only consider a generalized linear version of the gap
soliton model, and demonstrate that the model accurately describes wave
propagation in adiabatically modulated photonic crystal structures.
Let us consider two counter-propagating waves
$E^{\pm}(x,t)=A_{k}^{\pm}\exp(\imath\omega t\pm\imath kx)$, with group
velocity $v_{g}$, and coupled by a periodic modulation of the dielectric
permittivity with scaled magnitude $\mathcal{K}$. In the presence of this
linear coupling, the dispersion relation of the waves is described by the
coupled equations:

$\displaystyle(v_{g}k+\mathcal{D}_{2}k^{2}+\omega_{0})A_{k}^{+}+\mathcal{K}A_{k}^{-}=\omega
A_{k}^{+}$
$\displaystyle(-v_{g}k+\mathcal{D}_{2}k^{2}+\omega_{0})A_{k}^{-}+\mathcal{K}A_{k}^{+}=\omega
A_{k}^{-}$ (1)

Let us note that we introduced the Bragg angular frequency $\omega_{0}$, and
added the second-order dispersion term $\mathcal{D}_{2}$. The set
$\\{\omega_{0},\mathcal{K},v_{g},\mathcal{D}_{2}\\}$ describes the dispersion
of the coupled waves, and we will refer to it as the structure parameters. We
now derive these parameters for periodic structures with different widths $w$,
by focusing on a specific portion of the dispersion relation, namely one or
more bands, as shown in Fig. 2(a).

Here we consider a so-called nanobeam photonic crystal, where the width $w$ of
the beam takes the role of the control parameter (Fig. 1(b)). The nanobeam is
supposed to be made out of a III-V group semiconductor alloy In0.5Ga0.5P, with
refractive index $n=3.17$. The nanobeam is $h=180$ nm thick, with a $w=450$ nm
width, and a $a=465$ nm period; the holes radius is $0.27a$. The valence band
$\omega_{v}(k)$ is obtained by solving the 3D ME with Bloch periodic boundary
conditions along $x$, i.e.,
$\mathbf{E}_{k}(\mathbf{r}+a\hat{x})=E_{k}(\mathbf{r})\exp{(\imath ka)}$. This
is performed by means of the periodic FDTD (finite difference in time domain)
algorithm with a perfectly matched layer placed at the $z$ and $y$ boundaries.
The parameters are adjusted in order to minimize the error
$N^{-1}\sum_{i=0}^{N-1}{\sigma^{2}(k_{i})}$ over the $N$ points in the
reciprocal space, with
$\sigma^{2}(k_{i})=[\omega_{v}(k_{i})-\omega_{v}^{(RM)}(k_{i})]^{2}$ obtained
from the reduced model $\omega_{v}^{(RM)}$, i.e. the characteristic equation
solutions of the linear system eq. 1. This generates the set of parameters
$\\{\omega_{0},\mathcal{K},v_{g},\mathcal{D}_{2}\\}(w)$, which depends on the
control parameter $w$. The average of the residual error in the reciprocal
space region of interest $\sigma_{fit}=\overline{\sigma(k)}$ is about $100$
GHz, cfr. Fig. 2(b). Let us note that this error is about the same as the
estimated discretization error of the FDTD method[33]. This point is further
discussed in the Appendix.
Fig. 3 describes the dependence of the structure parameters on the control
parameter $w$. Panel (a) shows that the fit error decreases when $w$ grows
from $0.45\mu$m to $0.52\mu$m, meaning that the dispersion relation is
increasingly closer to that of the RM. In panels (c-f), the blue dashed line
represents the polynomial fit of the extracted parameters with respect to $w$.
Here, we make a crucial assumption, namely that the dependence of the
structure parameters on $w$ is smooth. The figure shows that a low-order
($3^{rd}$) polynomial is a good approximation, moreover the residual of
$\omega_{0}$ (blue solid line in panel b) is about 100 GHz or below. The
result here is a set of polynomial coefficients
$\mathcal{C}_{i}^{(\mathcal{P})}$ for each parameter
$\mathcal{P}=\\{\omega_{0},\mathcal{K},v_{g},\mathcal{D}_{2}\\}$. This two-
step interpolation of the dispersion of the periodic structure removes the
minute deviations which might be related to the discretization error.

Figure 3: Calculated structural parameters for the RM, vs. the control
parameter ${w}$. (a) averaged residual of the fit; (c-f) extracted parameters
(black squares), polynomial fit (blue dashed line) and polynomial fit on the
updated RM (solid red). In (e) the group velocity is reported in units of
$c_{0}$, i.e. the speed of light in vacuum. (b) residual of the fit of
$\omega_{0}$ (dashed blue), and after updating the RM (red solid).

##  III Reduced model of a tapered nanobeam cavity

Let us now consider an optical resonator where the confinement is due to the
tapering of the width $w(x)=\rho_{0}+\rho x^{2}$ ($\rho>0$) of the nanobeam,
as described in ref. [19]. When considering the $w$ dependence of $\omega_{0}$
and $\mathcal{K}$ in Fig. 3(d), it is immediate to realize that the edge of
the valence band $\omega_{vb}=\omega_{0}-\mathcal{K}$ decreases as $w$
increases. This leads to localisation of Bloch waves in the valence band of
the nanobeam, if $w$ is smaller in the center of the nanobeam. Let us consider
the case with $\rho=500$m-1. Fig. 4(b) shows the corresponding spatial
distribution of the modes, as it is obtained from the solution of the 3D ME
using the Finite Element Method (details are discussed in the Appendix).
Let us now build a RM for the cavity, and define the linear operator
$\mathcal{L}_{RM}$ acting on complex-valued functions of space $x$
($\mathcal{R}\rightarrow\mathcal{R}^{2}$):

$\mathcal{L}_{RM}=\begin{bmatrix}-\mathcal{D}_{2}\partial_{x}^{2}+\imath
v_{g}\partial_{x}+\omega_{0}&\mathcal{K}\\\
\mathcal{K}&-\mathcal{D}_{2}\partial_{x}^{2}-\imath
v_{g}\partial_{x}+\omega_{0}\end{bmatrix},$ (2)

Here, the structure parameters
$\\{\omega_{0},\mathcal{K},v_{g},\mathcal{D}_{2}\\}$ are all functions of $x$
via the profile $w(x)$ and the polynomials $\mathcal{C}_{i}^{(\mathcal{P})}$.
Namely, for each parameter $\mathcal{P}$ the corresponding function of $x$
reads as

$\mathcal{P}(x)=\sum_{l}\mathcal{C}_{l}^{(\mathcal{P})}w(x)^{l}$ (3)

The polynomial expansion is replaced by constant values for $|x|>x_{max}$,
namely $\mathcal{P}|_{|x|>x_{max}}=\mathcal{P}(x_{max})$. The eigenfunctions
$\psi=[\mathcal{A}^{+},\mathcal{A}^{-}]$ of the equation:

$(\mathcal{L}_{RM}-\omega)\psi=0$ (4)

correspond to the envelopes of the cavity modes, as predicted by the RM. The
equation is solved by finite difference discretization (see Appendix). It is
apparent that the field envelopes and the eigenfrequencies are close to the
corresponding results from a direct solution of the 3D ME. A very important
figure to describe the dispersion in multimode resonators is the integrated
dispersion[2], which measures the deviation of the cavity resonances
$\omega_{m}$ from a constant free spectral range (FSR):
$D_{int,m}=\omega_{m}-\omega_{0}-\mathcal{D}_{1}m$. This quantity is shown in
Fig. 4(c). The FSR is fixed to $\mathcal{D}_{1}=\omega_{1}-\omega_{0}$ from
the solution of the 3D ME, where $\omega_{0}$ is always the first eigenvalue
(the fundamental mode); this implies that we only consider the relative error
on the eigenvalues between the 3D ME results and the RM predictions. Let us
also note that the modes are ordered with decreasing frequencies, because the
valence band has an upper bound. The deviation
$\sigma_{m}=|\omega_{m}(3DME)-\omega_{m}(RM)|$ is shown in Fig. 4(e): as can
be seen, its value is of about 100 GHz.

Figure 4: Tapered nanobeam optical cavity. (a) simplified layout; (b) spatial
dependence of the valence band edge at $k=k_{0}$ (dashed black line),
normalized distribution of the squared electric field corresponding to the
calculated Bloch modes (3D ME) of the cavity along the axis $y=0$, $z(d)=0$
(solid grey), envelopes calculated from the reduced model of the cavity (blue
dashed) and after model update (red solid). The vertical offset corresponds to
the frequency at resonance. (c) Integrated dispersion $D_{int}$ calculated by
solving the 3D ME (squares), with the reduced cavity model (blue dashed line),
and after update (red solid). (d) Spatial dependence of $w$. (e) Frequency
deviation for each mode $\sigma_{RM}/2\pi$ of the RM (blue dashed) and the
updated RM (red solid), relative to the 3D ME calculations)

Let us now allow the polynomial coefficients $\mathcal{C}_{i}$ to be adjusted,
in order to minimize the mismatch between the eigenfrequencies obtained from
the solution of the 3D ME and from the RM, namely:

$\epsilon=\dfrac{1}{N}\sum^{N}_{m}\left|\dfrac{\sigma_{m}}{\omega_{0}}\right|^{2}$
(5)

The results correspond to the red lines in Fig. 4(b), which are now much
closer to the Bloch modes obtained from the solution of the 3D ME. This is
even more visible when inspecting $D_{int}$, Fig. 4(c), and the corresponding
residual, Fig. 4(e), which is now below 10 GHz. Let us now analyze the change
of the polynomial coefficients, by inspecting the change of the dependence of
$\mathcal{P}$ on $w$ in Fig. 3(b-f). The relative change of the parameters is
very small, and merely appears as an offset. The relative change of
$\mathcal{D}_{2}$ is larger, but this parameter represents a higher-order
correction to the coupled wave model. Thus, a slight adjustment of the
parameters is sufficient to let our reduced model to converge to the solution
of the 3D ME. We will refer hereafter to it as the ”updated” RM. The fact that
a correction of the parameters is needed is justified by the fact that the
adiabatic condition for the tapering is only partially satisfied. Yet, it is
noteworthy that the RM already generates a very good approximation of the
numerically exact result, and that a slight change of the polynomial
coefficients is enough to match the exact result within the discretization
error in the solution of the 3D ME. The updated RM solution is in this sense
fully equivalent to the 3D ME solution.

##  IV Design of a multimode resonator with a flat dispersion

Figure 5: Design of a nanobeam multimode resonator with flat dispersion. (a)
Bloch modes (from the 3D ME) and envelopes from the RM after optimisation (as
in Fig. 4); (b) Integral dispersion $D_{int}$ for the resonator with parabolic
tapering $w_{0}(x)$ (gray), after the first $w_{1}(x)$ (red) and the second
$w_{2}(x)$ (blue) optimization, respectively. Markers represent the solution
of the 3D ME, and lines correspond to the RM solutions. Note the logarithmic
vertical scale. (c) Corresponding tapering profiles $w_{i}(x)$; (d) residuals
(difference between RM and 3D ME) computed after updating the RM with
$w_{0}(x)$ (red dashed), after second update and $w_{1}(x)$ (red solid) and
with $w_{2}(x)$ (blue).

Let us now consider the updated RM, which consists of the eigenvalue equation
4 with the operator 2, and parameters defined by the updated coefficients
$\mathcal{C}_{i}$. The profile $w(x)$ is now allowed to change, so that the
integrated dispersion $D_{int}$ converges towards a prescribed target. As a
notable example, we consider a flat dispersion profile for the first 7 modes
as a target, i.e. $D_{int,m}=0$ for $m=1,\dots,7$. The tapering profile is
defined by a polynomial with even orders up to $2M=6$:
$w(x)=\rho_{0}+\sum_{l=1}^{M}\rho_{l}x^{2l}$. By this choice we have three
degrees of freedom in the optimization process ($\rho_{l}$, with $l=2,4,6$),
ensuring both convergence and high computational efficiency. The cost function
$\sum_{m}|D_{int,m}|$ is minimized with respect to the parameters $\rho_{l}$.
This results into a new profile $w_{1}(x)$, Fig. 5(c), for which the RM
predicts $D_{int}/2\pi$ decreasing by almost 2 orders of magnitude to about
2-4 GHz, Fig. 5(b). The 3D ME are solved again with $w_{1}(x)$ and the
resulting $D_{int}/2\pi$ is reduced to about 10 GHz, i.e. not as much as the
prediction of the RM. The coefficients $\mathcal{C}$ are updated such that RM
approaches the 3D ME, as in the previous section. This is necessary since
$w_{1}(x)$ has considerably departed from a parabola. Indeed, panel (d) shows
that the residuals (red dashed line) are larger than the estimated accuracy of
the 3D ME (green dashed line), yet they decrease below it after the second
update (solid red line). A new optimized profile is then generated $w_{2}(x)$,
yet no appreciable change is achieved (panels b,c,d), indicating that the
procedure has reached convergence, which is essentially set by the accuracy of
the 3D ME solver. In summary, the method has required one solve of the
periodic 3D ME, and two additional 3D ME solves for the cavity, the third one
only confirms convergence.
Finally, we analyze how the ID procedure affects the Q factors. Since our
procedure does not consider Q as a target for optimisation, there is no
guarantee that high-Q values are preserved. This is examined in Fig. 6. The Q
factors have been calculated either deterministically (circles) or by modeling
the fabrication imperfections by introducing disorder, i.e. by randomly
varying (r.m.s 0.5 nm) the position and the diameter of all the holes. The
error bars in the figure represent statistics (mean value and the standard
deviation of a log-normal distribution) over an ensemble of 20 simulations.
Thus, circles correspond to the radiation-limited Q, which decreases Q to less
than $10^{6}$ after the optimisation. Importantly, this is no longer true when
disorder is taken into account, as Q factor are basically unchanged.

Figure 6:  Calculated Q-factors for the reference cavity with parabolic
tapering $w_{0}(x)$ (orange) and the dispersion-flattened cavity with an
optimized profile (blue). Circles correspond to the radiation-limited Q, error
bars are an estimate of Q accounting for fabrication disorder. On the x-axis
is reported the frequency of the resonances.

##  V Conclusions

We have introduced a procedure for the inverse design of the dispersion of a
multimode nanoscale resonator. The main idea behind our approach harnesses the
fact that nanoscale cavities with large quality factors are in general
designed according to the principle of the ”gentle confinement”. This implies
that they can be described as almost adiabatically tapered periodic
structures. Inspired by well-known methods of solid state physics, we have
introduced a reduced model which is able to capture very well the dispersion
of the nanoscale structure in the spectral domain of interest, where the
cavity modes exist. The reduced model consists of a linear operator acting on
complex functions of a single variable, whose parameters are slowly varying.
These parameters are initially determined by fitting the dispersion of a
reference periodic structure via a function of the ”control” parameter. Here
we consider the case of a ”nanobeam” photonic crystal cavity. The width of the
nanobeam is decreased in the middle, which creates a confining potential in
the valence band of the photonic crystal. Next, the model is updated by
adapting the nanobeam parameters to the cavity. This two-step procedure avoids
issues related with the possible presence of sub-optimal minima in the fitting
procedure. We show that the reduced model is equivalent to the solution 3D
Maxwell equations within the discretization accuracy of the numerical solver,
but it is three orders of magnitude faster. For this reason, any optimisation
algorithm can be used. As an example, we consider the problem of flattening
the dispersion of a nanoscale resonator. The integrated dispersion $D_{int}$
is reduced below $\sim$10 GHz, essentially limited by the numerical accuracy
of the Maxwell solver.
Our method can be applied to any cavity geometry which can be described via
the one or two-dimensional tapering of a periodic structure. The model can be
extended to use more than one control parameters, and could also leverage on
the presence of multiple waves. In contrast to topological optimisation or
inverse design, which are intended to solve a very general class of problems,
our procedure is particularly suited to a specific but important class of
optical resonators, and could considerably help with the development of
nanoscale optical combs, mode-locked lasers, and special purpose parametric
generators of non-classical light.

## Appendix

###  V.1 Numerical Implementation

The reduced model requires the solution of the eigenvalue problem 1, which is
a system of linear partial differential equations. This is solved by finite-
difference discretization of the operator $\hat{\mathcal{L}}_{RM}$

$\hat{\mathcal{L}}_{RM}=\begin{bmatrix}\hat{\omega}_{0}-\hat{\mathcal{D}_{2}}\hat{D}_{x}^{2}+\imath\hat{v}_{g}\hat{D}_{x}^{FW}&\hat{\mathcal{K}}\\\
\hat{\mathcal{K}}&\hat{\omega}_{0}-\hat{\mathcal{D}_{2}}\hat{D}_{x}^{2}-\imath\hat{v}_{g}\hat{D}_{x}^{BW}\end{bmatrix}\
\ \ \ $ (6)

where the hat symbol $\hat{D}_{2}$, $\hat{D}_{x}$ means the finite difference
approximation of the differential operators, i.e., a $2N\times 2N$ matrix,
where N is the number of points used to approximate the spatial domain.
Therefore $\hat{\mathcal{L}}_{RM}$ is 4 $N\times N$ matrix. The operator
$\mathcal{L}_{RM}$ is generally non-Hermitian, thus its eigensolutions are not
real. Localized eigenfunctions correspond to nearly real eigenvalues
($Re\\{\tilde{\omega}\\}\gg Im\\{\tilde{\omega}\\}$). The difference operators
$\hat{D}^{2}_{x}$ and $\hat{D}^{FW,BW}_{x}$ are implemented on a regular grid
$x_{j}=j\Delta x$ using a second-order central difference scheme, and third-
order forward and backward upwind schemes[34, 35],

$\displaystyle\hat{D}_{x}^{FW}f_{j}$ $\displaystyle=$
$\displaystyle\dfrac{-f_{j+2}+6f_{j+1}-3f_{j}-2f_{j-1}}{6\Delta x}$
$\displaystyle\hat{D}_{x}^{BW}f_{j}$ $\displaystyle=$
$\displaystyle\dfrac{f_{j-2}-6f_{j-1}+3f_{j}+2f_{j+1}}{6\Delta x},$ (7)

and, for the second order derivative:

$D_{x}^{2}f_{j}=\dfrac{f_{j+1}-2f_{j}+f_{j-1}}{h^{2}}\ \ \ .$ (8)

As the terms of the difference scheme outside the domain are implicitly set to
zero, these imply Dirichelet boundary conditions, which are not appropriate to
represent either evanescent field decay, or dispersive waves. Therefore, the
considered computation domain is much larger than the size of the cavity. The
other operators are diagonal $\hat{v}_{g}=\delta_{i,j}v_{g}(x_{j})$,
$\hat{\mathcal{\kappa}}=\delta_{i,j}\mathcal{\kappa}(x_{j})$,
$\hat{\omega}_{0}=\delta_{i,j}\omega_{0}(x_{j})$, with $\delta_{i,j}$ the
Kronecker delta. All of the code written for optimization and evaluation of
the reduced model is written in Julia[36].

###  V.2 Numerical Accuracy

A critical issue when calculating the dispersion of multimode nanoscale
cavities is that the relative error in the calculation of frequencies can
hardly decrease below $10^{-4}$, which translates to inaccuracies of the order
of tens of GHz. In ref. [33] a variety of methods for solving the ME are
compared, for computing the resonances of a nanoscale cavity. It was observed
that finite element methods (FEM) converge better than finite differences in
time domain (FDTD) methods. Yet, it was concluded that the FEM error is likely
to be underestimated, since different implementations of the FEM converge to
slightly different results. This underlines how critical the numerical
accuracy is with these methods. For this reason, both approaches have been
used here. The FDTD algorithm is an in-house code, graphically accelerated
with sub-pixel smoothing [37]. The FEM method is implemented within the COMSOL
commercial code.

Figure 7:  Numerical solution of 3D ME with the nanobeam cavity: convergence
of FEM (red) and FDTD (blue) methods against number of elements and size of
the grid respectively; the error bar stands for the stochastic simulation on
an ensemble of 20 realisations. Fundamental mode frequency (a), the integral
dispersion up to the $7^{th}$ order mode (b) and the CPU time in units of
seconds (c).

Fig. 7 compares the two methods by considering the convergence of the
frequency of the fundamental order mode $\nu_{0}$ (a) and the integrated
dispersion up to the $7^{th}$ order mode (b) and the time required for the
computation (c). Let us note that the computation time scales with $(\Delta
x/a)^{4}$ for the FDTD and moderately superlinear with the number of elements
used in the FEM, thus the two horizontal scales cannot be compared directly.
Moreover, a more reliable computation of the frequencies through FDTD is
obtained by adding random fluctuations in the geometry (as discussed earlier),
performing the calculation on 20 to 60 realizations (depending on the
resolution) of the structure and considering averages and standard deviations
of the histograms of calculated frequencies. The standard deviation depends on
the disorder introduced ($rms=1nm$) and it is not an estimate of a numerical
accuracy. Thus, the time required by FDTD is much longer indeed, if this
method is used. Panel (a) shows that the inaccuracy for the frequency is about
10 GHz when FEM is used. More precisely, the accuracy assessed by comparing
the resonances computed by using either a low ($67252$ elements) or a high
($364550$ elements) resolution on the tetrahedral mesh, namely
$\sigma_{FEM}^{2}=\dfrac{1}{N}\sum_{m}^{N}|\omega_{FEM,H,m}-\omega_{FEM,L,m}|^{2}$,
hence $\sigma_{FEM}/2\pi=13$ GHz. As shown in Fig. 4, this is comparable with
the average residual between the frequencies computed with our RM and with
FEM. The FDTD converges to the same value with $a/\Delta x=40$. The inaccuracy
on the integrated dispersion, panel (b), is similar, and it is matched by FDTD
for $a/\Delta x=30$. The two methods give almost identical results for
$a/\Delta x=40$. In terms of computing time (for a single FDTD realization)
the resolution $a/\Delta x=40$ corresponds to the high resolution FEM mesh,
while the low resolution mesh corresponds to $a/\Delta x=25$. With this
resolution, we deduce from panels (a,b) that the error of the FDTD is about
100 GHz.
We conjecture that the accuracy of the RM may be better than that of the
numerical solution of 3D ME. As a matter of fact, the discretization of space
through finite differences or finite elements results into uncorrelated
deviations of the resonances. This source of randomness should vanish in the
exact solution, and it may be much reduced in the RM, since the method
inherently averages out random deviations. However, the proof of this is
problematic, because of the accuracy limitation of numerical solutions. Still,
our method guarantees that dispersion flatness remains within the order of 10
GHz in terms of the integrated dispersion, over the first seven confined
modes, which is clearly better than what is achievable with the strictly
bichromatic design that was reported in [8, 11], or with the parabolic
tapering design of ref [12]. Moreover, here we have shown how, by means of the
RM, it is possible to tailor the dispersion of a wide class of resonators,
regardless of their initial dispersion curves. Specifically, we could flatten
the integrated dispersion of the highest-order modes by almost three orders of
magnitude, i.e., from $\geq 1$THz down to $\sim$10 GHz, by drastically
changing the $D_{int}$ curve.

###  V.3 The optimization algorithm: workflow and performances

Figure 8:  Flowchart of the design algorithm: the band diagram calculation of
a perfectly periodic structure (I) gives a first estimate of the structure
parameters. A reference $w(x)$ PhC profile is taken into account (II) and
numerically solved (III). After the calibration of the RM (IV), we proceed
with the actual optimization (V). A loop across steps (IV-V) might be
necessary, if the accuracy is not sufficient upon optimization (red arrows).

|  $w_{0}$ |  $w_{0}$ upd. |  $w_{1}$ |  $w_{1}$ upd. |  $w_{2}$
---|---|---|---|---|---
$\sigma_{RM}$(GHz) |  200 |  8 |  30 |  12 |  13
$\Delta^{(2)}$(%FSR) |  27 |  31 |  1.7 |  0.17 |  0.12
3D ME |  period. |  $w_{0}$ | $w_{1}$ |  $w_{2}$
CPU time (s) |  80314 |  2948 | 2776 |  2816
fun. calls |  - |  723 |  220 |  557 |  254
CPU time (s) |  - |  356 |  107 |  261 |  125

Table 1: Workflow and overall performances of the method. The reduced model is
built from the periodized structure ($w_{0}$); RM is updated by comparison
with the 3D ME calculation (FEM) of the reference cavity ($w_{0}$ upd.);
profile is modified to one with flat dispersion as target ($w_{1}$); RM is
updated again against 3D ME calculation of the optimized cavity ($w_{1}$
upd.); the geometry is optimized again with the updated model ($w_{2}$).
$\sigma_{RM}$ is the averaged residual between RM and 3D ME,
$\Delta^{(2)}=(N-2)^{-1}\sum_{j}\Delta^{(2)}_{j}$, with
$\Delta^{(2)}_{j}=\omega_{j+1}+\omega_{j-1}-2\omega_{j}$ is the averaged
second order dispersion, normalized to the FSR. Note that both the update and
the optimization steps result in a minimization of two different figures of
merit, i.e. $\sigma_{RM}$ (blue arrows) and $\Delta^{(2)}$ (red arrows),
respectively. The computing time for the 3D FDTD periodic and the FEM
calculations for $w_{0}$, $w_{1}$ and $w_{2}$. The time needed to optimize the
profile using the RM and the number of calls needed to converge.

The performances of our method are summarized in Table 1. Computation is
performed using a 32 cores CPU, AMD EPYC 7351, with clock frequency equal to
2.4 GHz, and 64 GB RAM. The first step consists in establishing the RM, which
requires the calculation of the bands as a function of the control parameter
(here $w$). This takes $\sim 3000$ seconds for each of the $26$ values of $w$.
The second step is the calculation of the frequencies for the reference
cavity, as well as of the first and second optimized geometries, which takes
about 3000 s each by using FEM. An error minimization iterative procedure
based on the steepest gradient is performed twice for updating the RM, and
twice for the optimization of the profile $w(x)$. Convergence requires about
few hundreds function calls, and the average time for evaluating the RM is 0.6
s. This gives a sense of the acceleration provided by replacing the solution
of the 3D ME with that of the RM.
In FIG.8 we sketch a flowchart, showing each step of our design technique: the
initial step (1) consists in the band diagram calculation of a perfectly
periodic structure for different values of the control parameter. A first
estimate of the structure parameters and of their dependence on $w$ can be
extrapolated by means of a polynomial fit, as sketched in FIG.3. Consequently,
a reference profile of $w=\rho_{0}+\rho x^{2}$ is taken into account (II), and
the structure is solved by means of both RM and 3D ME solvers (III). At this
point, the RM is calibrated, and the set
$\\{\omega_{0},\mathcal{K},v_{g},\mathcal{D}\\}(w)$ updated to recover the
best accuracy(IV). The following step is the actual optimization of the
$\rho_{i}|_{i>0}$ coefficients of an even M-th order polynomial expansion
$w(x)=\rho_{0}+\sum_{l=1}^{M}\rho_{l}x^{2l}$ (V). After the optimization
cycle, it is essential to verify if the RM accuracy was degraded (red arrows):
if so, an other loop of the step (IV-V) will be needed in order to recover the
prescribed accuracy.

### Acknowledgments

The project leading to this application has received funding from the European
Union’s Horizon 2020 research and innovation programme under the Marie
Skłodowska-Curie EID project ”MOCCA” (G.A. No 814147), the ITN project
”OPHELLIA” (G.A. No. 101017136), and the ERC project ”STEMS” (G.A. No.
740355). The authors thank Loredana Maria Massaro and Fabrice Raineri for
stimulating discussions.

## References

  * Boyd [2020] R. W. Boyd,  _Nonlinear optics_ (Academic press, 2020).
  * Kippenberg  _et al._ [2018] T. J. Kippenberg, A. L. Gaeta, M. Lipson, and M. L. Gorodetsky, Dissipative kerr solitons in optical microresonators, Science 361, eaan8083 (2018).
  * Matsuo  _et al._ [2010] S. Matsuo, A. Shinya, T. Kakitsuka, K. Nozaki, T. Segawa, T. Sato, Y. Kawaguchi, and M. Notomi, High-speed ultracompact buried heterostructure photonic-crystal laser with 13 fj of energy consumed per bit transmitted, Nature Photonics 4, 648 (2010).
  * Crosnier  _et al._ [2017] G. Crosnier, D. Sanchez, S. Bouchoule, P. Monnier, G. Beaudoin, I. Sagnes, R. Raj, and F. Raineri, Hybrid Indium Phosphide-on-Silicon nanolaser diode, Nature Photonics 11, 297 (2017).
  * Yu  _et al._ [2017] Y. Yu, W. Xue, E. Semenova, K. Yvind, and J. Mork, Demonstration of a self-pulsing photonic crystal fano laser, Nature Photonics 11, 81 (2017).
  * Nozaki  _et al._ [2019] K. Nozaki, S. Matsuo, T. Fujii, K. Takeda, A. Shinya, E. Kuramochi, and M. Notomi, Femtofarad optoelectronic integration demonstrating energy-saving signal conversion and nonlinear functions, Nature Photonics , 1 (2019).
  * Takahashi  _et al._ [2013] Y. Takahashi, Y. Inui, M. Chihara, T. Asano, R. Terawaki, and S. Noda, A micrometre-scale raman silicon laser with a microwatt threshold, Nature 498, 470 (2013).
  * Marty  _et al._ [2021] G. Marty, S. Combrié, F. Raineri, and A. De Rossi, Photonic crystal optical parametric oscillator, Nature photonics 15, 53 (2021).
  * Stone  _et al._ [2022] J. R. Stone, G. Moille, X. Lu, and K. Srinivasan, Conversion efficiency in kerr-microresonator optical parametric oscillators: From three modes to many modes, Physical Review Applied 17, 024038 (2022).
  * Sun  _et al._ [2019] Y. Sun, S. Combrié, F. Bretenaker, and A. De Rossi, Mode locking of the hermite-gaussian modes of a nanolaser, Physical Review Letters 123, 233901 (2019).
  * Combrié  _et al._ [2017] S. Combrié, G. Lehoucq, G. Moille, A. Martin, and A. De Rossi, Comb of high-Q resonances in a compact photonic cavity, Laser & Photonics Reviews 11, 1700099 (2017).
  * Marty  _et al._ [2019] G. Marty, S. Combrié, A. De Rossi, and F. Raineri, Hybrid InGaP nanobeam on silicon photonics for efficient four wave mixing, APL Photonics 4, 120801 (2019).
  * Clementi  _et al._ [2019] M. Clementi, A. Barone, T. Fromherz, D. Gerace, and M. Galli, Selective tuning of optical modes in a silicon comb-like photonic crystal cavity, Nanophotonics 1, 205–210 (2019).
  * Jensen and Sigmund [2011] J. S. Jensen and O. Sigmund, Topology optimization for nano-photonics, Laser & Photonics Reviews 5, 308 (2011).
  * Molesky  _et al._ [2018] S. Molesky, Z. Lin, A. Y. Piggott, W. Jin, J. Vucković, and A. W. Rodriguez, Inverse design in nanophotonics, Nature Photonics 12, 659 (2018).
  * Minkov  _et al._ [2020] M. Minkov, I. A. Williamson, L. C. Andreani, D. Gerace, B. Lou, A. Y. Song, T. W. Hughes, and S. Fan, Inverse design of photonic crystals through automatic differentiation, Acs Photonics 7, 1729 (2020).
  * Hughes  _et al._ [2018] T. W. Hughes, M. Minkov, I. A. Williamson, and S. Fan, Adjoint method and inverse design for nonlinear nanophotonic devices, ACS Photonics 5, 4781 (2018).
  * Santis  _et al._ [2014] C. T. Santis, S. T. Steger, Y. Vilenchik, A. Vasilyev, and A. Yariv, High-coherence semiconductor lasers based on integral high-q resonators in hybrid si/iii-v platforms, Proceedings of the National Academy of Sciences 111, 2879 (2014).
  * Ahn  _et al._ [2010] B.-H. Ahn, J.-H. Kang, M.-K. Kim, J.-H. Song, B. Min, K.-S. Kim, and Y.-H. Lee, One-dimensional parabolic-beam photonic crystal laser, Optics Express 18, 5654 (2010).
  * Ghorbel  _et al._ [2019] I. Ghorbel, F. Swiadek, R. Zhu, D. Dolfi, G. Lehoucq, A. Martin, G. Moille, L. Morvan, R. Braive, S. Combrié, and A. De Rossi, Optomechanical gigahertz oscillator made of a two photon absorption free piezoelectric III/V semiconductor, APL Photonics 4, 116103 (2019).
  * Asano  _et al._ [2017] T. Asano, Y. Ochi, Y. Takahashi, K. Kishimoto, and S. Noda, Photonic crystal nanocavity with a Q factor exceeding eleven million, Optics Express 25, 1769 (2017).
  * Notomi  _et al._ [2007] M. Notomi, T. Tanabe, A. Shinya, E. Kuramochi, H. Taniyama, S. Mitsugi, and M. Morita, Nonlinear and adiabatic control of high-q photonic crystal nanocavities, Optics Express 15, 17458 (2007).
  * Quan and Loncar [2011] Q. Quan and M. Loncar, Deterministic design of wavelength scale, ultra-high q photonic crystal nanobeam cavities, Optics express 19, 18529 (2011).
  * Bazin  _et al._ [2014] A. Bazin, R. Raj, and F. Raineri, Design of silica encapsulated high-q photonic crystal nanobeam cavity, Journal of lightwave technology 32, 952 (2014).
  * Akahane  _et al._ [2003] Y. Akahane, T. Asano, B.-S. Song, and S. Noda, High-Q photonic nanocavity in a two-dimensional photonic crystal, Nature 425, 944 (2003).
  * Wang  _et al._ [2018] F. Wang, R. E. Christiansen, Y. Yu, J. Mørk, and O. Sigmund, Maximizing the quality factor to mode volume ratio for ultra-small photonic crystal cavities, Applied Physics Letters 113, 241101 (2018).
  * Yu and Cardona [2005] P. Yu and M. Cardona,  _Fundamentals of Semiconductors: Physics and Materials_ , Advanced texts in physics No. Bd. 3 (Springer Berlin Heidelberg, 2005).
  * Yariv [1989] A. Yariv,  _Quantum electronics_ (John Wiley & Sons, 1989).
  * Aceves and Wabnitz [1989] A. Aceves and S. Wabnitz, Self-induced transparency solitons in nonlinear refractive periodic media, Physics Letters A 141, 37 (1989).
  * De Rossi  _et al._ [1998] A. De Rossi, C. Conti, and S. Trillo, Stability, multistability, and wobbling of optical gap solitons, Physical review letters 81, 85 (1998).
  * Colman  _et al._ [2010] P. Colman, C. Husko, S. Combrié, I. Sagnes, C. W. Wong, and A. De Rossi, Temporal solitons and pulse compression in photonic crystal waveguides, Nature Photonics 4, 862 (2010).
  * Malaguti  _et al._ [2012] S. Malaguti, G. Bellanca, S. Combrie, A. de Rossi, and S. Trillo, Temporal gap solitons and all-optical control of group delay in line-defect waveguides, Physical Review Letters 109, 163902 (2012).
  * de Lasson  _et al._ [2018] J. R. de Lasson, L. H. Frandsen, P. Gutsche, S. Burger, O. S. Kim, O. Breinbjerg, A. Ivinskaya, F. Wang, O. Sigmund, T. Häyrynen, A. V. Lavrinenko, J. Mørk, and N. Gregersen, Benchmarking five numerical simulation techniques for computing resonance wavelengths and quality factors in photonic crystal membrane line defect cavities, Opt. Express 26, 11366 (2018).
  * Taylor [2016] C. R. Taylor, Finite difference coefficients calculator, https://web.media.mit.edu/~crtaylor/calculator.html (2016).
  * [35] Wikipedia, Upwind scheme, https://en.wikipedia.org/wiki/Upwind_scheme.
  * Bezanson  _et al._ [2017] J. Bezanson, A. Edelman, S. Karpinski, and V. B. Shah, Julia: A fresh approach to numerical computing, SIAM review 59, 65 (2017).
  * Oskooi  _et al._ [2009] A. F. Oskooi, C. Kottke, and S. G. Johnson, Accurate finite-difference time-domain simulation of anisotropic media by subpixel smoothing, Opt. Lett. 34, 2778 (2009).


# Action-GPT: Leveraging Large-scale Language Models for Improved and
Generalized Action Generation

Sai Shashank Kalakonda
<EMAIL_ADDRESS>
Shubh Maheshwari
<EMAIL_ADDRESS>
Ravi Kiran Sarvadevabhatla
<EMAIL_ADDRESS>
Centre for Visual Information Technology
IIIT Hyderabad, Hyderabad, INDIA 500032

###### Abstract

We introduce Action-GPT, a plug-and-play framework for incorporating Large
Language Models (LLMs) into text-based action generation models. Action
phrases in current motion capture datasets contain minimal and to-the-point
information. By carefully crafting prompts for LLMs, we generate richer and
fine-grained descriptions of the action. We show that utilizing these detailed
descriptions instead of the original action phrases leads to better alignment
of text and motion spaces. We introduce a generic approach compatible with
stochastic (e.g. VAE-based) and deterministic (e.g. MotionCLIP) text-to-motion
models. In addition, the approach enables multiple text descriptions to be
utilized. Our experiments show (i) noticeable qualitative and quantitative
improvement in the quality of synthesized motions, (ii) benefits of utilizing
multiple LLM-generated descriptions, (iii) suitability of the prompt function,
and (iv) zero-shot generation capabilities of the proposed approach. Code,
pretrained models and sample videos will be made available at
https://actiongpt.github.io.

Figure 2: Action-GPT Overview: Given an action phrase ($x$), we first create a
suitable prompt using an engineered prompt function $f_{\text{prompt}}(x)$.
The result ($x_{\text{prompt}}$) is passed to a large-scale language model
(GPT-3) to obtain multiple action descriptions ($D_{i}$) containing fine-
grained body movement details. The corresponding deep text representations
$v_{i}$ are obtained using Description Embedder. The aggregated version of
these embeddings $v_{aggr}$ is processed by the Text Encoder. During training,
the action pose sequence $H_{1},...,H_{N}$ is processed by a Motion Encoder.
The encoders are associated with a deterministic sampler (autoencoder) [3] or
a VAE style generative model [4, 1]. During training (shown with black), the
latent text embedding $Z_{T}$ and the latent motion embedding $Z_{M}$ are
aligned. During inference (shown in green), the sampled text embedding is
provided to the Motion Decoder, which outputs the generated action sequence
${\widehat{H}}$.

Figure 1: Sample text conditioned action generations from a state-of-the-art
model TEACH [1] (top row) and our large language model-based approach (Action-
GPT-TEACH - bottom row). By incorporating large language models, our approach
results in noticeably improved generation quality for seen and unseen
categories. The conditioning action phrases for seen categories are taken from
BABEL [2] whereas unseen action phrases were provided by a user.

##  1 Introduction

Human motion generation finds a vast set of applications spanning from
entertainment (e.g. game and film industry) to virtual reality and robotics.
There have been significant contributions on category-conditioned human motion
generations [5], with some works capable of generation at scale [6, 7].
However, the generated samples are restricted to a finite set of action
categories. More recent approaches focus on text-conditioned motion generation
by constraining motion and language representations via a jointly optimized
latent space [4, 3, 1].

The recent development of large-scale language models (LLMs) [8, 9] has
triggered a paradigm shift in the field of Natural Language Processing (NLP).
These models, pre-trained on enormous amounts of text [10], have demonstrated
impressive generalization capabilities for challenging zero-shot setting tasks
such as text generation [11]. This exciting advance has also driven progress
for various applications in computer vision [12, 13], including the related
task of pose-based human action recognition [12].

The appeal of LLM models lies in their ability to generate task-relevant text
when provided a so-called prompt \- a small piece of text - as input.
Motivated by this observation and the advances mentioned above, we introduce
Action-GPT, an approach that utilizes the generative power of LLMs to improve
the quality and generalization capabilities of action generative models. In
particular, we demonstrate that our plug-and-play approach can be used to
advance existing state-of-the-art motion generation architectures in a
practical manner.

Our contributions are summarized below:

  * •

To the best of our knowledge, we are the first to incorporate Large Language
Models (LLMs) for text-conditioned motion generation.

  * •

We introduce a carefully crafted prompt function that enables the generation
of meaningful descriptions for a given action phrase.

  * •

We introduce Action-GPT, a generic plug-and-play framework which is compatible
with stochastic (e.g. VAE-based [1, 4]) and deterministic (e.g. MotionCLIP
[3]) text-to-motion models. In addition, our framework enables multiple
generated text descriptions to be utilized for action generation.

  * •

Via qualitative and quantitative experiments, we demonstrate (i) noticeable
improvement in the quality of synthesized motions, (ii) benefits of utilizing
multiple LLM-generated descriptions, (iii) suitability of the prompt function,
and (iv) zero-shot generation capabilities of the proposed approach.

Code, pretrained models and sample videos will be made available at
https://actiongpt.github.io.

Figure 3: Visual comparison of generated motion sequences across models
trained on Action-GPT framework on BABEL [2] dataset. Note that the
generations using Action-GPT are well-aligned with the semantic information of
action phrases. The example in the bottom right row shows latent space
editing. Action-GPT is better able to transfer the drink from mug style from
standing to sitting pose.

##  2 Action-GPT

Our objective is to generate an actor performing the motion conditioned on the
given action phrase. The input action phrase is a natural language text that
gives a high-level description of the action. It is denoted as a sequence of
words $x=[w_{1},w_{2},...,w_{M}]$. The action is represented as a sequence of
human poses ${H}=\\{H_{1},\dots,H_{n},\dots,H_{N}\\}$ where $N$ represents the
number of timesteps. The human pose $H_{n}\in\mathcal{R}^{|J|\times 6}$, where
J is the number of joints, is the parametric SMPL [14] representation wh ich
encodes the global trajectory and the parent relative joint rotation using the
6D [15] rotation representation.

Our proposed framework Action-GPT can be incorporated in an autoencoder [3] or
a Variational Auto Encoder [4, 1] based text-conditioned motion generation
model. These motion generation models aim to generate a motion sequence
conditioned on the text input by learning a joint latent space between the
text and motion modalities. The key components in these models are Text
Encoder $\mathcal{T}_{enc}$, Motion Encoder $\mathcal{M}_{enc}$ and Motion
Decoder $\mathcal{M}_{dec}$. The two text and motion encoders encode the text
sequence and motion sequence to text $Z_{T}$ and motion $Z_{M}$ latent
embeddings of the same dimension, respectively. In the case of autoencoders,
the latent embeddings are obtained in a deterministic fashion, whereas in
Variational Auto Encoders, the latent embeddings are sampled from the Gaussian
distribution $\mathcal{N}(\mu,\Sigma)$, where $(\mu,\Sigma)$ are the outputs
of the encoder. The motion decoder, on the other hand, uses the latent
embedding $Z$ as input to generate a sequence of motion poses
${\widehat{H}}=\\{\widehat{H}_{1},\dots,\widehat{H}_{n},\dots,\widehat{H}_{N}\\}$

Fig. 2 provides an overview of our approach to incorporate LLM (GPT-3 in our
case) into the text-conditioned motion generation models. In contrast to
training directly using the action phrase $x$ from the dataset, our framework
uses carefully crafted GPT-3 generated text descriptions $D_{i}$, which
provide low-level details about the movement of individual body parts. The
proposed framework consists of three steps (1) Constructing a prompt function
$f_{prompt}$, (2) Aggregating multiple GPT-3 generated text descriptions
$D_{i}$, and finally, (3) utilizing the GPT-3 generated text descriptions
$D_{i}$ in T2M models.

###  2.1 Prompt strategy

For a given action phrase $x$, we generate low-level body movement details
using GPT-3 [9]. GPT-3 is an autoregressive transformer model which generates
human-like textual descriptions relevant to the small amount of input text
provided. However, directly providing the action phrase as input to GPT-3
fails to output text containing the desired detail body movement information
and leads to unrealistic motion generations (see Fig. 4). This necessitates
the need for a suitable prompt function [10]. After multiple empirical trials,
we determine the following prompting function $f_{prompt}$: Describe a
person’s body movements who is performing the action [x] in detail.
Specifically, adding Describe a person’s to the prompt restricts the
description from generic information to character movement. The phrase body
movements forces GPT-3 to explain the motion of individual body parts. Lastly,
in detail forces the descriptions to provide low-level details. Fig. 5
showcases the importance of each component of our prompt function. We provide
GPT-3 generated text description $(D)$ and corresponding generated action
sequence ${\widehat{H}}$ for the action phrase act like a dog along with the
observations in the rightmost column.

Figure 4: This figure highlights the importance of the prompt function.
Observe that directly feeding the action phrase text ($x$) to GPT-3 results in
poor-quality generations. In contrast, the fine-grained body movement details
in the prompt-based text enable higher fidelity generations (last column).
Note that the coloured text descriptions correspond to different body movement
details. Figure 5: The table showcases the descriptions generated by GPT-3
(D), generated action sequences (${\widehat{H}}$) for the action phrase (x =)
act like a dog using different prompt strategies along with the observations
(right most column). Notice that our prompt function (bottom row) generates
the highest amount of required body movement descriptions, generating the most
realistic action sequence. Note that the coloured text descriptions correspond
to the body movement details.

###  2.2 Aggregating multiple descriptions

Given an action prompt $x_{prompt}$, GPT-3 is capable of generating multiple
textual descriptions $D_{1},\dots,D_{k}$ describing the action-specific
information. The randomly generated $k$ descriptions contain common and
description-specific text segments, which enhance the overall richness of
action description (see Fig. 6). Therefore, we utilize multiple descriptions
as part of the text-processing pipeline. The GPT-3 generated $k$ text
descriptions $D_{1},\dots,D_{k}$ are passed through a Description Embedder
$D_{emb}$ to obtain corresponding description embeddings $v_{1},\dots,v_{k}$.
These $k$ description embeddings are aggregated into a single embedding
$v_{aggr}$ using an Embedding Aggregator $E_{aggr}$. We consider average
operation as our Embedding Aggregator unless stated otherwise.

###  2.3 Utilizing GPT-3 generated text descriptions in T2M models

The text encoder $\mathcal{T}_{enc}$ inputs the aggregated embedding
$v_{aggr}$, and the outputs are sampled to generate text latent embeddings
$Z_{T}$. In a similar fashion, motion encoder $\mathcal{M}_{enc}$ inputs the
sequence of motion poses ${H}$ = {${H}_{1},\dots,{H}_{N}$}, where
${H}_{n}\in\mathcal{R}^{|J|\times 6}$ and samples motion latent embeddings
$Z_{M}$. In a deterministic approach [3], the latent embeddings are generated
directly as outputs of the encoder, whereas in a VAE-based approach [4, 1],
the encoders generate distribution parameters $\mu$ and $\Sigma$. The latent
embeddings are then sampled from the Gaussian distributions $N(\mu$,
$\Sigma)$. On the other hand, the generated text and motion embeddings are
provided to the motion decoder, which generates the human motion sequence
${\widehat{H}}$ = {${\widehat{H}}_{1},\dots,{\widehat{H}}_{N}$}, where
${\widehat{H}}_{i}\in\mathcal{R}^{|J|\times 6}$ which is passed through
forward kinematics to generate corresponding 3D mesh sequence.

Figure 6: This figure highlights the importance of using multiple GPT-3
generated descriptions ($D_{1},...,D_{k}$), $k=4$ for each action phrase in
Action-GPT framework for TEACH [1]. Notice the visibly improved generation
quality when multiple prompted descriptions are used (right column). Body
movement text common across descriptions is highlighted in blue. Movements
unique to each description are highlighted in pink.

##  3 Experiments

BABEL [2] is a large dataset with language labels describing the actions being
performed in motion capture sequences. It contains about 43 hours of mocap
sequences, comprising over 65k textual labels which belong to over 250 unique
action categories. We primarily focus our results on BABEL, considering its
vast and diverse set of motion sequences assigned to short text sequences,
which contain an average of 3-4 words. The action phrases of the BABEL dataset
are to the point and precise about the action information without any
additional details about the actor.

###  3.1 Models

We demonstrate our framework on state-of-the-art text conditioned motion
generation models – TEMOS [2], MotionCLIP [3] and TEACH [1]. Since TEACH is an
extension of TEMOS and uses only pairs of motion data of BABEL, we also
demonstrate our results on TEMOS by retraining it on all single action data
segments of BABEL. We train these three models as per our framework using
their publicly available codes and will call them Action-GPT-[model] further.

Action-GPT-MotionCLIP: Similar to MotionCLIP [3], we use CLIP’s pretrained
text encoder [16] as our description embedder, but we inputs all the $k$ text
descriptions and generate the aggregated vector representation $v_{aggr}$
using the embedding aggregator. Note that similar to MotionCLIP, we use CLIP-
ViT-B/32 frozen model. We follow the same motion auto-encoder setup as that of
MotionCLIP. There is no additional text encoder and sampling process as the
constructed $v_{aggr}$ itself is used as the text embedding $Z_{T}$, and the
output of the motion encoder is used as the motion embedding $Z_{M}$.

Action-GPT-TEMOS: Instead of providing action phrase text, we pass multiple
textual descriptions extracted from GPT-3 to DistilBERT [17] to obtain
description embedding $v_{i}\in R^{n_{i}\times e}$, where $n_{i}$ is the
number of words in description $D_{i}$ and $e$ is the DistilBERT embedding
dimension, thus $v_{aggr}\in R^{G\times e}$ where $G=max(n_{i})$. Note that
similar to TEMOS, we use pre-trained DistilBERT and freeze its weights during
training. The text encoder, motion encoder, and motion decoder used is the
same as that of TEMOS. The sampled text embedding $Z_{T}$ and motion embedding
$Z_{M}$ are both $\in R^{d}$ where $d$ is the dimension of latent space.

Action-GPT-TEACH: Since TEACH [1] is an extension of TEMOS, the process of
generating description embeddings $v_{i}$ is the same as that of in Action-
GPT-TEMOS. The text encoder, motion encoder, and motion decoder are used the
same as that of TEACH. As TEACH is trained on the pairs of action data, the
training iteration consists of two forward passes where an action phrase and
its corresponding motion sequence are provided as input in each pass. In
addition, a set of the last few frames of generated motion in the first pass
are also provided as input in the second pass. In both passes, our framework
uses the generated description embeddings corresponding to the input action
phrases.

Detailed diagrams illustrating the differences between original architectures
and their LLM-based variants along with the additional details regarding
training and testing can be found in the appendix 5.

###  3.2 Implementation details

We access GPT-3 via OpenAI API Beta Access program. Unless stated otherwise,
we use the largest GPT-3 model available, davinci-002. The Action-GPT prompt
strategy consumes a maximum of 140 tokens together for prompt and generation.
We use the completions API endpoint with the parameters temperature and top-p
set to 0.5 and 1, ensuring we have well-defined diverse descriptions. All the
other parameters are set to default. We conduct all our experiments on cluster
machines with Intel Xenon E5 2640 v4 and Nvidia GeForce GTX Ti 12GB GPUs with
Ubuntu 16.04 OS.

###  3.3 Quantitative analysis

We follow the metrics employed in TEACH [1] for quantitative evaluation,
namely Average Positional Error (APE) and Average Variational Error (AVE),
measured on the root joint and the rest of the body joints separately. Mean
local correspond to the joint position in the local coordinate system (with
respect to the root), whereas mean global corresponds to the joint position in
the global coordinate system. The APE and AVE for a joint are the averages of
the L2 distances between the generated and ground truth joint positions and
variances, respectively – refer to appendix 8 for details. Tab. 1 summarizes
the results of using our framework in comparison with the default setup for
each model. Incorporating detailed descriptions using GPT-3 shows an
improvement over all the APE (except for MotionCLIP) and AVE metrics. The
metrics of root joints for MotionCLIP are empty since it generates only local
pose without any locomotion.

###  3.4 Qualitative analysis

In Fig. 3, we provide qualitative comparisons of the model generations. We
observe that the generations from our framework are more realistic and well-
aligned with the semantic information of the action phrases compared to the
default approach. The generations are able to capture the low-level fine-
grained details of the action suggested by the original text phrase input.
Please refer to appendix for video examples of zero-shot generations,
comparisons with baselines, and examples showing diversity in generated
sequences.

|  | Average Positional Error ↓ | Average Variance Error ↓
---|---|---|---
Model | Method | root joint | global traj | mean local | mean global | root joint | global traj | mean local | mean global
MotionCLIP | Default | - | - | 0.556 | 0.541 | - | - | 0.056 | 0.02
Action-GPT | - | - | 0.590 | 0.571 | - | - | 0.042 | 0.019
TEMOS | Default | 0.597 | 0.574 | 0.162 | 0.644 | 0.113 | 0.112 | 0.010 | 0.122
Action-GPT | 0.561 | 0.540 | 0.151 | 0.605 | 0.101 | 0.100 | 0.010 | 0.109
TEACH | Default | 0.674 | 0.654 | 0.159 | 0.717 | 0.222 | 0.220 | 0.014 | 0.234
Action-GPT | 0.606 | 0.586 | 0.159 | 0.650 | 0.204 | 0.202 | 0.014 | 0.216

Table 1: Quantitative evaluation on the BABEL test set

|  | Average Positional Error ↓ | Average Variance Error ↓
---|---|---|---
Architectural Component | Ablation Details | root joint | global traj | mean local | mean global | root joint | global traj | mean local | mean global
number of generated descriptions ($K$) | $k=1$ | 0.655 | 0.635 | 0.159 | 0.698 | 0.216 | 0.214 | 0.015 | 0.228
$k=2$ | 0.637 | 0.617 | 0.158 | 0.680 | 0.211 | 0.209 | 0.015 | 0.223
| $k=8$ | 0.632 | 0.613 | 0.157 | 0.674 | 0.212 | 0.210 | 0.015 | 0.224
GPT-3 Capacity | curie | 0.642 | 0.622 | 0.159 | 0.680 | 0.216 | 0.214 | 0.015 | 0.228
Ours ($\mathbf{k=4}$) | davinci | 0.606 | 0.586 | 0.158 | 0.650 | 0.204 | 0.202 | 0.014 | 0.216

Table 2: Performance scores for ablative variants.

###  3.5 Ablations

We perform an ablation study to understand the underlying effects of the
Action-GPT framework. All of the ablation experiments are carried out on the
Action-GPT-TEACH model unless stated otherwise, as it is capable of handling a
series of action phrases as input.

Number of GPT-3 Text Sequences: We analyzed the influence of number of
generated descriptions in Action-GPT-TEACH framework by varying $k$ in $1,2,4$
and $8$. We observed that for all the values of $k$, Action-GPT-TEACH performs
better than the default TEACH and the best results are obtained for $k=4$ (see
Tab. 2). Increasing the value of $k$ up to a certain value improves
performance. However, aggregating too many descriptions can lead to the
injection of excessive noise, which dilutes the presence of text related to
body movement details.

Language Model Capacity: Open AI provides GPT-3 in different model capacities,
davinci being the largest. We analyzed the influence of curie, the second
largest GPT-3 model, on the motion sequence generations of the Action-GPT-
TEACH $(k=4)$ framework. Results show that having a larger model capacity
helps in generating more realistic motion sequences, as the generated text
descriptions provide much relevant and detailed information as required.

##  4 Discussion and Conclusion

The key to good quality and generalizable text-conditioned action generation
models lies in improving the alignment between the text and motion
representations. Through our Action-GPT framework, we show that such alignment
can be achieved efficiently by employing Large Language Models whose operation
is guided by a judiciously crafted prompt function. Sentences in GPT-generated
descriptions contain procedural text which corresponds to sub-actions. During
training, the regularity and frequency of such procedural text likely enable
better alignment with corresponding motion sequence counterparts. We also
hypothesize that the diversity of procedural sentences in the descriptions
enables better compositionality for unseen (zero-shot) generation settings.

The plug-and-play nature of our approach is practical for adoption within
state-of-the-art text-conditioned action generation models. Our experimental
results demonstrate the generalization capabilities and action fidelity
improvement for multiple adopted models, qualitatively and quantitatively. In
addition, we also highlight the role of various prompt function components and
the benefit of utilizing multiple prompts for improved generation quality.

## References

  * [1] Nikos Athanasiou, Mathis Petrovich, Michael J. Black, and Güll Varol,  “TEACH: Temporal Action Compositions for 3D Humans,”  in International Conference on 3D Vision (3DV), 2022.
  * [2] Abhinanda R. Punnakkal, Arjun Chandrasekaran, Nikos Athanasiou, Alejandra Quiros-Ramirez, and Michael J. Black,  “BABEL: Bodies, action and behavior with english labels,”  in Proceedings IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), June 2021, pp. 722–731.
  * [3] Guy Tevet, Brian Gordon, Amir Hertz, Amit H Bermano, and Daniel Cohen-Or,  “Motionclip: Exposing human motion generation to clip space,”  arXiv preprint arXiv:2203.08063, 2022.
  * [4] Mathis Petrovich, Michael J. Black, and Gül Varol,  “TEMOS: Generating diverse human motions from textual descriptions,”  in European Conference on Computer Vision (ECCV), 2022.
  * [5] Mathis Petrovich, Michael J. Black, and Gül Varol,  “Action-conditioned 3D human motion synthesis with transformer VAE,”  in International Conference on Computer Vision (ICCV), 2021.
  * [6] Shubh Maheshwari, Debtanu Gupta, and Ravi Kiran Sarvadevabhatla,  “Mugl: Large scale multi person conditional action generation with locomotion,”  in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), January 2022, pp. 257–265.
  * [7] Debtanu Gupta, Shubh Maheshwari, Sai Shashank Kalakonda, Manasvi, and Ravi Kiran Sarvadevabhatla,  “Dsag: A scalable deep framework for action-conditioned multi-actor full body motion synthesis,”  in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), January 2023.
  * [8] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro,  “Megatron-lm: Training multi-billion parameter language models using model parallelism,” 2019,  cite arxiv:1909.08053.
  * [9] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.,  “Language models are few-shot learners,”  arXiv preprint arXiv:2005.14165, 2020.
  * [10] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig,  “Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing,” 2021.
  * [11] Kang Min Yoo, Dongju Park, Jaewook Kang, Sang-Woo Lee, and Woomyeong Park,  “Gpt3mix: Leveraging large-scale language models for text augmentation,” 2021.
  * [12] Wangmeng Xiang, Chao Li, Yuxuan Zhou, Biao Wang, and Lei Zhang,  “Language supervised training for skeleton-based action recognition,” 2022.
  * [13] Sarah Pratt, Rosanne Liu, and Ali Farhadi,  “What does a platypus look like? generating customized prompts for zero-shot image classification,” 2022.
  * [14] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black,  “SMPL: A skinned multi-person linear model,”  ACM Trans. Graphics (Proc. SIGGRAPH Asia), vol. 34, no. 6, pp. 248:1–248:16, Oct. 2015.
  * [15] Yi Zhou, Connelly Barnes, Lu Jingwan, Yang Jimei, and Li Hao,  “On the continuity of rotation representations in neural networks,”  in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [16] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever,  “Learning transferable visual models from natural language supervision,”  in Proceedings of the 38th International Conference on Machine Learning, Marina Meila and Tong Zhang, Eds. 18–24 Jul 2021, vol. 139 of Proceedings of Machine Learning Research, pp. 8748–8763, PMLR.
  * [17] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf,  “Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter,” 2019.

Appendix

##  5 Models

We demonstrate our framework on state-of-the-art text conditioned motion
generation models – TEMOS [2], MotionCLIP [3] and TEACH [1], all trained on
BABEL [2]. We name their LLM (i.e. GPT here) based variants as Action-
GPT-[model].

Action-GPT-MotionCLIP: In MotionCLIP [3], for a given action phrase $x$, the
CLIP text embedding of the phrase $x$ is considered as its corresponding
latent text embedding $Z_{T}$, whereas in our Action-GPT framework the
aggregated vector embedding $v_{aggr}\in R^{c}$, where $c$ is the CLIP text
embedding dimesnion, constructed for the action phrase $x$ is considered as
its latent text embedding $Z_{T}$. In detail, we first construct $x_{prompt}$
using the action phrase $x$ and prompt function $f_{prompt}$. The $x_{prompt}$
is then input to LLM (i.e. GPT-3) to generate $k$ textual descriptions
$D_{i}$. Using the CLIP text encoder, we then construct $k$ corresponding CLIP
text embeddings $v_{i}$. These $k$ CLIP text embeddings $v_{i}$ are then
aggregated into a single embedding $v_{aggr}$ using an Embedding aggregator
(average operation here). The constructed $v_{aggr}$ is the corresponding
latent text embedding $Z_{T}$ for the action phrase $x$. (see Fig. 7)

We trained this model using the same training configurations as mentioned in
MotionCLIP [3] and provided the results on the test split. We generated the
metrics for the baseline MotionCLIP [3] using the pre-trained model provided.

Figure 7: Action-GPT-MotionCLIP Overview: We extend MotionCLIP [3] by
incorporating LLM (i.e. GPT-3). The box highlighted in green showcases the
generation of $k$ text descriptions $D_{i}$ as the output of LLM on input
$x_{prompt}$, which is constructed using the prompt function $f_{prompt}$ and
action phrase $x$. The box highlighted in blue showcases the aggregation of
description embeddings $v_{i}$, outputs of the CLIP text encoder. The
aggregated embedding $v_{aggr}$ is considered as the latent text embedding
$Z_{T}$, on which the text loss $\mathcal{L}_{text}$ is computed. All the
other components apart from the highlighted boxes represent the original
architecture of MotionCLIP [3].

Action-GPT-TEMOS: Instead of providing the action phrase $x$, directly to
DistilBERT [17] as that in TEMOS [4], we first construct $x_{prompt}$ and
generate $k$ textual descriptions $D_{i}$ using LLM (i.e. GPT-3). These $k$
textual descriptions are then input to DistilBERT [17] to obtain $k$
corresponding description embeddings $v_{i}\in R^{n_{i}\times e}$, where
$n_{i}$ is the number of words in description $D_{i}$ and $e$ is the
DistilBERT embedding dimension. The $k$ description embeddings are aggregated
to a single embedding $v_{aggr}\in R^{G\times e}$, where $G=max(n_{i})$ using
the average operation. This aggregated embedding $v_{aggr}$ is then used as
input to Text Encoder along with the text distribution tokens
$\mu_{token}^{T}$ and $\sum_{token}^{T}$. The later training and inference
process is carried out the same as that of in TEMOS [4]. (see Fig. 8)

We trained this model on a 4 GPU setup with a batch size of 4 on each GPU,
keeping rest all the parameters same as provided in TEMOS [4]. We trained the
baseline model of TEMOS [4] and its LLM-based variant on BABEL [2] using the
single action data segments provided by TEACH [1] and generated the metrics on
the corresponding test set.

Figure 8: Action-GPT-TEMOS Overview: We extend TEMOS [4] by incorporating LLM
(i.e. GPT-3). The box highlighted in green showcases the generation of $k$
text descriptions $D_{i}$ using LLM on input $x_{prompt}$, which is
constructed using the prompt function $f_{prompt}$ and action phrase $x$. The
$k$ text descriptions $D_{i}$ are input to DistilBERT to generate
corresponding description embeddings $v_{i}$. The box highlighted in blue
showcases the aggregation of description embeddings $v_{i}$ to
$v_{aggr}^{1:G}$ using an embedding aggregator. All the other components apart
from the highlighted boxes represent the original architecture of TEMOS [4].
Figure 9: Action-GPT-TEACH Overview: We extend TEACH [1] by incorporating LLM
(i.e. GPT-3). As TEACH can generate an action sequence for a series of action
phrases, we use $x^{i}$ to denote the $i^{th}$ phrase. The box highlighted in
green showcases the $k$ generated text descriptions $D_{j}^{i}$ using LLM on
input $x_{prompt}^{i}$, which is constructed using the prompt function
$f_{prompt}$ and action phrase $x^{i}$. The $k$ text descriptions $D_{j}^{i}$
are input to DistilBERT to generate corresponding sentence embeddings
$v_{j}^{i}$, which are aggregated into a single embedding $v_{aggr}^{i,1:G}$
using an embedding aggregator. All the other components apart from the
highlighted boxes represent the original architecture of TEACH [1].

Action-GPT-TEACH: Since TEACH [1] is an extension of TEMOS [4] the process of
generating aggregated description embedding $v_{aggr}$ is same as that of
Action-GPT-TEMOS. In addition, TEACH can generate an action sequence for a
series of action phrases as input $x=\\{x^{1},\dots,x^{i},\dots,x^{s}\\}$. So,
we compute $v_{aggr}$ $s$ number of times, once for each phrase $x^{i}$. In
detail, for an action phrase $x^{i}$ we generate $k$ text descriptions
$D_{1}^{i},..,D_{j}^{i},..,D_{k}^{i}$. The $k$ text descriptions are input to
DistilBERT to generate corresponding description embeddings
$v_{1}^{i},..,v_{j}^{i},..,v_{k}^{i}$, where $v_{j}^{i}\in R^{n_{j}^{i}\times
e}$, where $n_{j}^{i}$ is the number of words in description $D_{j}^{i}$ and
$e$ is the DistilBERT embedding dimension. The $k$ description embeddings
$v_{j}^{i}$ are aggregated to a single embedding $v_{aggr}^{i}\in R^{G\times
e}$, where $G=max(n_{j}^{i})$ using the average operation. The aggregated
embedding $v_{aggr}^{i}$ is input to Past-conditioned Text Encoder $T_{enc}$
along with the learnable tokens $\mu_{token},\sum_{token},$ SEP token and
${{I}}^{i-1}_{N_{i-1}-P:N_{i-1}}$, the motion features generated using Past
Encoder, corresponding to the last $P$ frames of previous generated action
sequence ${\widehat{H}}^{i-1}_{N_{i-1}-P:N_{i-1}}$. The later training and
inference process followed is the same as that of TEACH.(see Fig. 9)

Similar to Action-GPT-TEMOS, we trained this model on a 4 GPU setup with a
batch size of 4 on each GPU, keeping rest all the parameters same as provided
in TEACH [1]. We generated the metrics for the baseline TEACH [1] using the
pre-trained model provided.

##  6 Diverse Generations

Our Action-GPT framework can generate diverse action sequences for a given
action phrase $x$, utilizing the capability of LLMs to generate diverse text
descriptions for a given prompt. We generate multiple text descriptions
$D_{i}$ for an action phrase $x$, and using them as input, multiple action
sequences ${\widehat{H}}_{i}$ are generated.

##  7 Zero-Shot Generations

Our approach enables the generation of action sequences ${\widehat{H}}$ for
unseen action phrases (zero-shot). The intuition behind this is that our
Action-GPT framework uses low-level detailed body movements textual
information to align text and motion spaces instead of using the action phrase
directly. Hence the action phrase might be unseen, but the low-level body
movement details in the generated corresponding text descriptions $D_{i}$ will
not be completely unseen.

Results corresponding to comparison of the baseline models to their LLM-based
variants, diverse and zero-shot generations can be found at
https://actiongpt.github.io.

Figure 10: Actions with locomotion generated using Action-GPT-TEACH. The color
of the text (column 2) represents the detailed text generated by GPT-3 and the
same color of the mesh represents the pose sequence generated for the sub-
action. The green curve shows the trajectory. The red points show the starting
and end points of the motion. Action-GPT is able to generate diverse examples
involving locomotion such as walk in a circle, run, hop over obstacle and
kick.

##  8 Metrics

We show results using the generative quality metrics followed by [4, 1],
namely Average Positional Error (APE) and Average Variational Error (AVE). For
all the metrics, the smaller the score, the better the generative quality.

Average Positional Error (APE) : For a joint j, APE is calculated as the
average of the L2 distances between the generated and ground truth joint
positions over the timesteps (N) and the number of test samples (S).

APE$[j]=\dfrac{1}{SN}\sum_{s\in S}\sum_{n\in
N}\|\widehat{H}_{s,n}[j]-H_{s,n}[j]\|_{2}$

Average Variational Error (AVE) : For a joint j, AVE is calculated as the
average of the L2 distances between the generated and ground truth variances.

AVE$[j]=\dfrac{1}{S}\sum_{s\in
S}\|\widehat{\sigma}_{s}[j]-\sigma_{s}[j]\|_{2}$

where $\sigma[j]$ denotes the variance of the joint $j$,

$\sigma[j]=\dfrac{1}{N-1}\sum_{n\in N}(\widetilde{H}[j]-H_{n}[j])^{2}$

$\widetilde{H}[j]$ is calculated as the mean of the joint $j$ over $N$
timesteps.

We calculate four variants of errors for both APE and AVE,

  * •

root joint error is calculated on the root joint using all the 3 coordinates
$X,Y$ and $Z$.

  * •

global traj error is calculated on the root joint using only the 2 coordinates
$X$ and $Y$.

  * •

mean local error is calculated as the average of all the joint errors in the
local coordinate system with respect to the root joint.

  * •

mean global error is calculated as the average of all the joint errors in the
global coordinate system.

##  9 Locomotion and root movement

Fig. 10 shows the diverse actions involving locomotion generated using Action-
GPT. Our prompt strategy provides detailed descriptions which include the
direction of the locomotion along with the coordination among relevant body
parts. We further verify this by quantitatively evaluating using Average
Positional Error and Average Variance Error of the global trajectory metric –
see Table. 1. We observe that the Action-GPT variants are better aligned with
root trajectory than the baseline.

##  10 Current limitations

  * •

Finger motion: Current frameworks use SMPL to represent the pose. SMPL does
not contain detailed finger joints. Therefore, GPT-generated descriptions for
actions requiring detailed finger motion such as rock-paper-scissors, pointing
fingers cannot be generated satisfactorily by the current framework.

  * •

Complex Actions: Actions containing complex and vigorous body movements such
as yoga and dance poses cannot be generated by our current framework.

  * •

Long duration action sequences: Due to the limited duration of training data
action sequences ($<10$ secs), our method cannot generate long sequences.

Model | Method | Avg. Training Time (secs) | Avg. Inference Time (secs)
---|---|---|---
MotionCLIP | Default | 585 - 598 | 0.32 - 0.34
Action-GPT | 700 - 712 | 0.53 - 0.62
TEMOS | Default | 225 - 230 | 0.8 - 0.96
Action-GPT | 255 - 260 | 1.44 - 1.76
TEACH | Default | 234 - 240 | 1.6 - 2.1
Action-GPT | 315 - 320 | 3.4 - 3.8

Table 3: Computation costs of baselines and their Action-GPT ($k=4$) variants.

##  11 Computation cost analysis

There will be no change in the number of parameters of the Action-GPT variants
when compared to the baseline models as we are using the frozen pre-trained
text embedding models. Although the computation time of the models both during
training and inference is higher for the Action-GPT ($k=4$) variants when
compared with the original baselines as shown in Table. 3. For training time,
we take the mean of the time taken for hundred epochs. For inference time, we
take a batch-size of $16$ and average over $10$ repetitions. The increase in
the computation time of the Action-GPT variants is because of the usage of $k$
GPT-3 text descriptions, each containing around $128$ words, whereas, in the
baseline models, a single action phrase is used, which contains around $5-8$
words.