# Periodic DMP formulation for Quaternion Trajectories

Fares J. Abu-Dakka1, Matteo Saveriano2, and Luka Peternel3 1Intelligent
Robotics Group, Dept of Electrical Engineering and Automation, Aalto
University, Finland (e-mail: fares.abu-dakka@aalto.fi).2Dept of Computer
Science and Digital Science Center, University of Innsbruck, Austria (e-mail:
matteo.saveriano@uibk.ac.at).3Delft Haptics Lab, Dept of Cognitive Robotics,
Delft University of Technology, Delft, Netherlands (e-mail:
l.peternel@tudelft.nl).This work has been partially supported by CHIST-ERA
project IPALM (Academy of Finland decision 326304), and by the Austrian
Research Foundation (Euregio IPN 86-N30, OLIVER).

###### Abstract

Imitation learning techniques have been used as a way to transfer skills to
robots. Among them, dynamic movement primitives (DMPs) have been widely
exploited as an effective and an efficient technique to learn and reproduce
complex discrete and periodic skills. While DMPs have been properly formulated
for learning point-to-point movements for both translation and orientation,
periodic ones are missing a formulation to learn the orientation. To address
this gap, we propose a novel DMP formulation that enables encoding of periodic
orientation trajectories. Within this formulation we develop two approaches:
Riemannian metric-based projection approach and unit quaternion based periodic
DMP. Both formulations exploit unit quaternions to represent the orientation.
However, the first exploits the properties of Riemannian manifolds to work in
the tangent space of the unit sphere. The second encodes directly the unit
quaternion trajectory while guaranteeing the unitary norm of the generated
quaternions. We validated the technical aspects of the proposed methods in
simulation. Then we performed experiments on a real robot to execute daily
tasks that involve periodic orientation changes (i.e., surface
polishing/wiping and liquid mixing by shaking).

##  I Introduction

The ability to control movements and interaction behaviour is one of the
fundamental aspects of robots performing their intended tasks. The underlying
properties of the task can be encoded and represented by reference
trajectories of motion impedance and/or force. Therefore, it is imperative to
have a reliable and powerful trajectory encoding method.

One of the most widely used methods is Dynamic Movement Primitives (DMPs) [1,
2]. DMPs are capable of encoding discrete and periodic trajectories. Discrete
trajectories are point-to-point motions with distinct starting and ending
point (goal), which are suitable for many daily tasks (e.g., pick and place).
Nevertheless, many other daily tasks are of periodic nature and periodic
trajectories have their own specifics. For example, they have the same
starting or ending point and all the neighbouring points have to be smoothly
connected to each-other to form a repetitive pattern. To consider these
specifics, Ijspeert et al. [1] introduced a different formulation for periodic
(also called rhythmic) DMPs. Since then, periodic DMPs and their upgrades have
been successfully applied to various realistic tasks of periodic nature, such
as surface wiping [3], sawing [4], bolt screwing [5], humanoid locomotion [6]
and exoskeleton assistance [7].

Figure 1: Franka Emika Panda performs polishing of a curved surface.

DMPs were originally designed to encode one Degree of Freedom (DoF)
trajectories and are well suited for representing independent signals like
joint or Cartesian positions. Synchronization between different DoF is usually
achieved by exploiting a common phase variable. However, in common orientation
representations like rotation matrix or unit quaternions the DoFs are
interrelated by geometric constraints (e.g., unitary norm). To properly
consider this constraint both learning and integration need to be carried out
by considering the structure of the orientation and perform the integration of
rotation elements while obeying its constraint manifold. To account for this
interdependence among DoFs, Pastor et al. [8] reformulated the original
discrete DMP to encode unit quaternion profiles. However, this formulation
does not take into account the geometry of the unit sphere, since it only used
the vector part of the quaternion product. To address this issue DMPs were
reformulated for direct unit quaternion encoding in [9, 10]. The quaternion-
based DMPs were also extended to include real-time goal switching mechanism
[11]. As an alternative to the quaternion representation, a rotation matrix
can be used as in [11]. The existing methods are suitable for discrete (point-
to-point) orientation trajectories. However, there is no DMP approach that can
effectively encode periodic orientation trajectories.

To address this gap, we propose a novel DMP formulation that enables encoding
of periodic orientation trajectories. Within this formulation we develop two
approaches:  Riemannian Metric-based Periodic  (RMP-DMP) and  Unit Quaternion-
based Periodic  (QP-DMP). The first exploits the fact that the space of unit
quaternions is a Riemannian manifold that locally behaves as an Euclidean
space (tangent space). Therefore, we can project unit quaternion trajectories
onto the tangent space, fit a periodic DMP in the tangent space, and project
back the output of this DMP onto the unit quaternion manifold. The second
encodes directly the unit quaternion trajectory and uses quaternion operations
to ensure the unitary norm of the integrated quaternions. The proposed
approaches are tested on synthetic data and then used to perform surface
polishing/wiping and mixing liquids by shaking with a robotic manipulator (see
Fig. 1).

##  II Related Works

Learning from Demonstration (LfD) has been widely used as a convenient way to
transfer human skills to robots. It aims at extracting relevant motion
patterns from human demonstrations and subsequently applying them to different
situations. In the past decades, several LfD based approaches have been
developed such as: DMPs [12, 9], Probabilistic Movement Primitives (ProMP)
[13], Stable Dynamical Systems (SDS) [14, 15], GMM and Task-Parameterized
(TP-GMM) [16], and Kernelized Movement Primitive (KMP) [17]. In many previous
works, quaternion trajectories are learned and adapted without considering the
unit norm constraint (e.g., orientation DMP [18] and TP-GMM [19]), leading to
improper quaternions and hence requiring an additional re-normalization.

The problem of learning proper orientations has been investigated in several
works. In [8] was the first attempt to modify the DMP formulation to deal with
unit quaternions. Their DMP considers the vector part of the quaternion
product as a measure of the error between the current and the goal
orientation. However, this formulation has slow convergence to an attractor
point as it is not considering the full geometry constraints of the unit
quaternions. Later, the authors in [9, 11] considered a geometrically
consistent orientation error, i.e., the angular velocity needed to rotate the
current quaternion into the goal one in a unitary time. The stability of both
DMP formulations is shown in [20] using Lyapunov arguments.

In [21], GMM is used to model the distribution of the quaternion
displacements. This probabilistic encoding, together with the exploitation of
Riemannian metrics, has been employed in [22] to learn orientation
trajectories with TP-GMM.

All previously mentioned approaches exploited LfD techniques in order to learn
discrete (point-to-point) orientation trajectories and not periodic ones. At
the best of the authors knowledge, this is the first work that focuses on
learning periodic DMP for orientation trajectories.

The idea of learning a KMP in the tangent space of the orientation manifold is
presented in [17]. While the authors also successfully tested orientation-KMP
against periodic unit quaternion trajectories, the orientation-KMP is an
alternative approach and does not resolve the issue within the periodic-DMP
formulation itself. Due to popularity of DMPs, there is still a significant
functionality gap for many robotic systems that are already based on DMPs and
for future applications for those who would prefer to use DMP approach.

##  III Background

In this section, we provide a brief introduction to the classical periodic DMP
formulation (sometimes called rhythmic DMP). Periodic DMPs are used when the
encoded motion follows a rhythmic pattern. Moreover, we provide basic
quaternion notations and operations used in the paper.

###  III-A Classical Periodic DMP formulation

The basic idea of periodic DMPs is to model movements by a system of
differential equations that ensure some desired behavior, e.g. convergence to
a specified movement cycle in order to encode motion of rhythmic patterns [1].
A DMP for a single DoF periodic trajectory $y$ is defined by the following set
of nonlinear differential equations

$\displaystyle\dot{z}$
$\displaystyle=\Omega\left({\alpha_{z}\left({\beta_{z}\left(g-y\right)-z}\right)+f(\phi)}\right),$
(1) $\displaystyle\dot{y}$ $\displaystyle=\Omega z,$ (2)
$\displaystyle\tau\dot{\phi}$ $\displaystyle=1,$ (3)

where $g$ is the goal and its value can be set to zero or alternatively to the
average of the demonstrated trajectory cycle. $\Omega$ is the frequency and
$y$ is the desired periodic trajectory that we want to encode with a DMP.

The main difference between periodic DMPs and point-to-point DMPs is that the
time constant related to trajectory duration is replaced by the frequency of
trajectory execution (refer to [12, 1] for details). In addition, the periodic
DMPs must ensure that the values of the initial phase ($\phi=0$) and the final
phase ($\phi=2\pi$) coincide in order to achieve smooth transition during the
repetitions. $f(\phi)$ is defined with $N$ Gaussian kernels according to the
following equation

$\displaystyle f(\phi)$
$\displaystyle=\frac{{\sum_{i=1}^{N}{\Psi_{i}(\phi)w_{i}}}}{{\sum_{i=1}^{N}{\Psi_{i}(\phi)}}}\,r,$
(4) $\displaystyle\Psi_{i}(\phi)$
$\displaystyle=\exp\left(h\left({\cos\left({\phi-
c_{i}}\right)-1}\right)\right),$ (5)

where the weights are uniformly distributed along the phase space, and $r$ is
used to modulate the amplitude of the periodic signal [1, 23] (if not used, it
can be set to $r=1$ [7]).

The classical periodic DMP described by (1)–(3) does not encode the transit
motion needed to start the periodic one. Transients are important in several
applications, e.g., in humanoid robot walking, where the first step from a
resting position is usually a transient and is needed to start the periodic
motion. To overcome this limitation, [24] modified the classical formulation
of periodic DMPs to explicitly consider transients as motion trajectory that
converge towards the limit cycle (i.e., periodic) one.

###  III-B Periodic state estimation and control

The phase and frequency of periodic DMPs can be controlled by an adaptive
oscillator, which estimates them as [23]

$\displaystyle\dot{\phi}$ $\displaystyle=\Omega-K\cdot e\cdot\sin(\phi),$ (6)
$\displaystyle\dot{\Omega}$ $\displaystyle=-K\cdot e\cdot\sin(\phi),$ (7)

where $K$ is a positive-value coupling constant and $e=U-\hat{U}$ is a
difference between some external signal $U$ (e.g., human muscle activity in
exoskeleton control [7]) and its internal estimation $\hat{U}$ constructed by
a Fourier series [25]

$\displaystyle\hat{U}$
$\displaystyle=\sum_{c=0}^{M}(\alpha_{c}\cos(c\phi)+\beta_{c}\sin(c\phi)),$
(8)

where $M$ is the size of the Fourier series. Fourier series parameters are
learnt in the following manner

$\displaystyle\dot{\alpha}_{c}$ $\displaystyle=\eta\cos(c\phi)\cdot e,$ (9)
$\displaystyle\dot{\beta}_{c}$ $\displaystyle=\eta\sin(c\phi)\cdot e,$ (10)

where parameter $\eta$ is a learning rate. The open parameters can be set to
$K=10$, $M=10$ and $\eta=2$ [25]. Adaptive oscillators are most useful during
online learning to infer periodic state (phase and frequency) in real-time.
Nevertheless, they are also useful for offline learning when the recorded
signal has variable frequency.

###  III-C Quaternion operations

A hyper-complex number is the basis of the quaternion mathematical object. Let
us define a quaternion as
$\boldsymbol{\mathbf{q}}=\nu+\boldsymbol{\mathbf{u}}\,:\,\nu\in\mathbb{R},\,\boldsymbol{\mathbf{u}}=[u_{x},u_{y},u_{z}]{{}^{\top}}\in\mathbb{R}^{3},$
$\boldsymbol{\mathbf{q}}\in\mathbb{S}^{3}$ and $\mathbb{S}^{3}$ is a unit
sphere in $\mathbb{R}^{4}$. A quaternion with a unit norm is called a unit
quaternion and is used to represent an orientation in 3–D space. In this
representation $\boldsymbol{\mathbf{q}}$ and $-\boldsymbol{\mathbf{q}}$
represent the same orientation.
$||\boldsymbol{\mathbf{q}}||=\sqrt{\nu^{2}+u_{x}^{2}+u_{y}^{2}+u_{z}^{2}}$ is
the quaternion norm where $||\cdot||$ denotes $\ell_{2}$ norm. The quaternion
conjugation of $\boldsymbol{\mathbf{q}}$ is defined as
$\bar{\boldsymbol{\mathbf{q}}}=\nu+(-\boldsymbol{\mathbf{u}})$, while the
multiplication of
$\boldsymbol{\mathbf{q}}_{1},\boldsymbol{\mathbf{q}}_{2}\in\mathbb{S}^{3}$ is
defined as

$\boldsymbol{\mathbf{q}}_{1}*\boldsymbol{\mathbf{q}}_{2}=(\nu_{1}\nu_{2}-\boldsymbol{\mathbf{u}}_{1}{{}^{\top}}\boldsymbol{\mathbf{u}}_{2})+(\nu_{1}\boldsymbol{\mathbf{u}}_{2}+\nu_{2}\boldsymbol{\mathbf{u}}_{1}+\boldsymbol{\mathbf{u}}_{1}\times\boldsymbol{\mathbf{u}}_{2})$
(11)

In order to project unit quaternions back and forth between the unit sphere
manifold $\mathbb{S}^{3}$ and the tangent space $\mathbb{R}^{3}$ we use
logarithmic and exponential mapping operators.
$\boldsymbol{\mathbf{\zeta}}={\text{Log}^{q}_{\boldsymbol{\mathbf{q}}_{2}}(\boldsymbol{\mathbf{q}}_{1}):\mathbb{S}^{3}\mapsto\mathbb{R}^{3}}$
maps $\boldsymbol{\mathbf{q}}_{1}\in\mathbb{S}^{3}$ to
$\boldsymbol{\mathbf{\zeta}}\in\mathbb{R}^{3}$ w.r.t. to
$\boldsymbol{\mathbf{q}}_{2}$. Consider
$\boldsymbol{\mathbf{q}}=\boldsymbol{\mathbf{q}}_{1}*\bar{\boldsymbol{\mathbf{q}}}_{2}$,

$\begin{split}\boldsymbol{\mathbf{\zeta}}=\text{Log}^{q}_{\boldsymbol{\mathbf{q}}_{2}}(\boldsymbol{\mathbf{q}}_{1})&=\text{Log}^{q}(\boldsymbol{\mathbf{q}}_{1}*\bar{\boldsymbol{\mathbf{q}}}_{2})=\text{Log}^{q}(\boldsymbol{\mathbf{q}})\\\
&=\begin{cases}\arccos(\nu)\frac{\boldsymbol{\mathbf{u}}}{||\boldsymbol{\mathbf{u||}}},&||\boldsymbol{\mathbf{\zeta||}}\neq
0\\\ [0\,\,0\,\,0]{{}^{\top}},&\text{otherwise}\end{cases}\end{split}$ (12)

Inversely,
$\boldsymbol{\mathbf{q}}={\text{Exp}^{q}_{\boldsymbol{\mathbf{q}}_{2}}(\boldsymbol{\mathbf{\zeta}}):\mathbb{R}^{3}\mapsto\mathbb{S}^{3}}$
maps $\boldsymbol{\mathbf{\zeta}}\in\mathbb{R}^{3}$ to
$\boldsymbol{\mathbf{q}}\in\mathbb{S}^{3}$ so that it lies on the geodesic
starting point from $\boldsymbol{\mathbf{q}}_{2}$ in the direction of
$\boldsymbol{\mathbf{\zeta}}$.

$\displaystyle\text{Exp}^{q}(\boldsymbol{\mathbf{\zeta}})$ $\displaystyle=$
$\displaystyle\begin{cases}\cos(||\boldsymbol{\mathbf{\zeta||}})+\sin(||\boldsymbol{\mathbf{\zeta||}})\frac{\boldsymbol{\mathbf{\zeta}}}{||\boldsymbol{\mathbf{\zeta||}}},&||\boldsymbol{\mathbf{\zeta||}}\neq
0\\\ 1+[0\,\,0\,\,0]{{}^{\top}},&\text{otherwise}\end{cases}$ (13)
$\displaystyle\boldsymbol{\mathbf{q}}_{1}$ $\displaystyle=$
$\displaystyle\text{Exp}^{q}(\boldsymbol{\mathbf{\zeta}})*\boldsymbol{\mathbf{q}}_{2}.$
(14)

In order to ensure discontinuity-free demonstration of an orientation profile
and avoid singularity problems, the following two assumptions should be
satisfied:

#### Assumption 1

The dot product between each adjacent unit quaternion is greater than zero,
such that $\boldsymbol{\mathbf{q}}_{t}\cdot\boldsymbol{\mathbf{q}}_{t+1}>0$,
which guarantees that $\boldsymbol{\mathbf{q}}_{t}$ and
$\boldsymbol{\mathbf{q}}_{t+1}$ are in the same hemisphere. Otherwise, we flip
$\boldsymbol{\mathbf{q}}_{t+1}$ such as
$\boldsymbol{\mathbf{q}}_{t+1}=\bar{\boldsymbol{\mathbf{q}}}_{t+1}$.

#### Assumption 2

As discussed in [9, 11], the domain of $\text{Log}^{q}(\cdot)$ extends to all
$\mathbb{S}^{3}$ except $-1+[0\,\,0\,\,0]{{}^{\top}}$, while the domain of
$\text{Exp}^{q}(\cdot)$ is constrained by
$\|\boldsymbol{\mathbf{\zeta}}\|<\pi$. Restricting the domain to
$\|\boldsymbol{\mathbf{\zeta}}\|<\pi$ makes (12) and (13) bijective.

##  IV Proposed Approach

Multidimensional periodic variables are encoded using one DMP for each DoF and
synchronized by a common phase. This works for variables like joint or
Cartesian positions, forces, torques, etc, where every DoF of each variable
can be encoded and integrated independently form the rest and still reproduce
the desired combined behaviour of the robot. However, this is not enough to
successfully encode orientations, whose elements are interdependent variables
and subject to additional constraints (i.e., the orthogonality, the unit
norm), without pre- and/or post-processing the data.

In order to overcome this limitation, we propose two approaches to learn
periodic orientation movements, namely the  Riemannian Metric-based Periodic
(RMP-DMP) and the  Unit Quaternion-based Periodic  (QP-DMP). In what follows,
we consider a periodic demonstration of length $T$ as the trajectory
$\boldsymbol{\mathbf{\mathcal{Q}}}_{q}=\\{\boldsymbol{\mathbf{q}}_{t}\\}_{t=1}^{T}$,
where $\boldsymbol{\mathbf{q}}_{t}$ are unit quaternions. We assume that the
unit quaternions are collected from a single demonstration of a rhythmic
pattern and then used to train a periodic DMP.

###  IV-A Riemannian Metric-based Periodic  (RMP-DMP)

Compute
---

$\mathcal{T}_{\boldsymbol{\mathbf{\mu}}_{q}}\mathcal{M}$
---

$\mathcal{T}_{\boldsymbol{\mathbf{\mu}}_{q}}\mathcal{M}$
---

Periodic
---
DMP
---
demonstration
---
$\boldsymbol{\mathbf{\mathcal{Q}}}_{q}$
reproduction
---
$\boldsymbol{\mathbf{\mathcal{Q}}}_{q}^{new}$
$\boldsymbol{\mathbf{\zeta}}$
---
$\boldsymbol{\mathbf{\mu}}_{q}$
---
$\boldsymbol{\mathbf{\mu}}_{q}$
---
$=\text{Log}^{q}_{\boldsymbol{\mathbf{\mu}}_{q}}($
---
$\boldsymbol{\mathbf{\zeta}}$
---
$(\boldsymbol{\mathbf{\mathcal{Q}}}_{q}$
---
$)$
---
$\boldsymbol{\mathbf{\mathcal{Q}}}_{q}^{new}$
---
$=\text{Exp}^{q}_{\boldsymbol{\mathbf{\mu}}_{q}}($
---
$\boldsymbol{\mathbf{\zeta}}$
---
$)*$
---
$\boldsymbol{\mathbf{\mu}}_{q}$
---
Figure 2: Diagram for the proposed Riemannian metric-based projection for
learning and adapting unit quaternion trajectories. The blue data represent
unit quaternion demonstration trajectory
$\boldsymbol{\mathbf{\mathcal{Q}}}_{q}$. The red data represent the projection
of $\boldsymbol{\mathbf{\mathcal{Q}}}_{q}$ onto the tangent space
$\mathcal{T}_{\boldsymbol{\mathbf{\mu}}_{q}}\mathcal{M}$ w.r.t. the mean
$\boldsymbol{\mathbf{\mu}}_{q}$. The brown data represent the reproduced unit
quaternion trajectory after projecting the reproduced periodic DMP trajectory
from $\mathcal{T}_{\boldsymbol{\mathbf{\mu}}_{q}}\mathcal{M}$.

A manifold is a topological space that resembles the Euclidean space around
each point, i.e., it has locally the properties of the Euclidean space. A
Riemannian manifold $\mathcal{M}$ is a smooth manifold that is equipped with a
Riemannian metric. For every point $\boldsymbol{\mathbf{m}}$ of a manifold
$\mathcal{M}$, i.e., $\boldsymbol{\mathbf{m}}\in\mathcal{M}$, it is possible
to compute a tangent space $\mathcal{T}_{\boldsymbol{\mathbf{m}}}\mathcal{M}$,
its metric is flat, which allows the use of classical arithmetic tools. In
other words, one can perform typical Euclidean operations in the tangent space
and project back the result to the manifold. The overall idea is sketched in
Fig. 2.

Inspired by [17], we exploit Riemannian operators to project a unit quaternion
demonstration $\boldsymbol{\mathbf{\mathcal{Q}}}_{q}$ from
$\mathcal{M}=\mathbb{S}^{3}$ onto the tangent space
$\mathcal{T}_{\boldsymbol{\mathbf{\mu}}_{q}}\mathcal{M}$. The tangent space is
attached to an auxiliary unit quaternion $\boldsymbol{\mathbf{\mu}}_{q}$ and
each unit quaternion in $\boldsymbol{\mathbf{\mathcal{Q}}}_{q}$ is moved to
the tangent space by means of
$\text{Log}^{q}_{\boldsymbol{\mathbf{\mu}}_{q}}(\cdot)$ defined in (12). Thus,
(12) is used to project $\boldsymbol{\mathbf{\mathcal{Q}}}_{q}$ onto
$\mathcal{T}_{\boldsymbol{\mathbf{\mu}}_{q}}\mathcal{M}$ creating
$\boldsymbol{\mathbf{\mathcal{Q}}}_{\zeta}=\\{\boldsymbol{\mathbf{\zeta}}_{t}\\}_{t=1}^{T}$.
Each point in $\boldsymbol{\mathbf{\mathcal{Q}}}_{\zeta}$ belongs to the
tangent space and has the same property of $3$D vectors, i.e.,
$\boldsymbol{\mathbf{\zeta}}_{t}\in\mathbb{R}^{3}$. The trajectory
$\boldsymbol{\mathbf{\mathcal{Q}}}_{\zeta}$ is (numerically) differentiated to
compute the 1st and 2nd time-derivatives
$\boldsymbol{\mathbf{\dot{\zeta}}}_{t},\boldsymbol{\mathbf{\ddot{\zeta}}}_{t}\in\mathbb{R}^{3}$.
The training data
$\boldsymbol{\mathbf{\zeta}}_{t},\boldsymbol{\mathbf{\dot{\zeta}}}_{t},\boldsymbol{\mathbf{\ddot{\zeta}}}_{t}\in\mathbb{R}^{3}$
are subsequently encoded using the periodic dynamic system in (1)–(2). During
the DMP execution, at each time-step we use (13) and (14) to project back each
of the generated $\boldsymbol{\mathbf{\zeta}}_{t}^{*}$ from
$\mathcal{T}_{\boldsymbol{\mathbf{\mu}}_{q}}\mathcal{M}$ onto
$\mathbb{S}^{3}$, creating a new unit quaternion
$\boldsymbol{\mathbf{q}}_{t}^{*}$.

The center of the tangent space $\boldsymbol{\mathbf{\mu}}_{q}$ is an open
parameter to determine. In [17], $\boldsymbol{\mathbf{\mu}}_{q}$ was selected
as the initial point of the demonstration. However, this choice cause problems
if the extreme orientations in the demonstrated trajectory are close to $0$ or
$\pi\,$rad. Using such extreme quaternion as $\boldsymbol{\mathbf{\mu}}_{q}$
will produce different $\boldsymbol{\mathbf{\zeta}}$ due to finite precision
algebra. This problem may be avoided by selecting
$\boldsymbol{\mathbf{\mu}}_{q}$ as the mean point of demonstrations. This is
because the central tendency of the mean—that is the farthest point from the
extreme—allows to avoid restricting the input domain, which may be necessary
if the initial point is used. In the case of unit quaternions, this mean can
be estimated using Maximum Likelihood Estimation (MLE) [22].

###  IV-B Unit Quaternion-based Periodic  (QP-DMP)

The original formulation of periodic DMPs were successfully applied to
multidimensional independent variables for their individual DoF
$\in\mathbb{R}$. These variables can be joint or Cartesian positions, forces,
torques, etc, where every DoF of each variable can be encoded and integrated
independently form the rest and still reproduce the desired combined behaviour
of the robot. However, such formulation is not enough to successfully encode
orientations, whose elements are interdependent variables and subject to
additional constraints (i.e., unitary norm), without pre- and/or post-
processing the data.

To properly encode unit quaternions trajectory, inspired by the work on
discrete quaternion DMPs [9, 11], we reformulate the dynamic system in (1) and
(2) as

$\displaystyle\boldsymbol{\mathbf{\dot{\eta}}}$
$\displaystyle=\boldsymbol{\mathbf{\Omega}}\left(\alpha_{z}(\beta_{z}2\,\text{Log}^{q}(\boldsymbol{\mathbf{g}}*\overline{\boldsymbol{\mathbf{q}}})-\boldsymbol{\mathbf{\eta}})+\boldsymbol{\mathbf{f}}(\boldsymbol{\mathbf{\phi}})\right),$
(15) $\displaystyle\boldsymbol{\mathbf{\dot{q}}}$
$\displaystyle=\boldsymbol{\mathbf{\Omega}}\frac{1}{2}\boldsymbol{\mathbf{\eta}}*\boldsymbol{\mathbf{q}}.$
(16)

The unit quaternion
$\boldsymbol{\mathbf{g}}\in\boldsymbol{\mathbf{\mathbb{S}}}^{3}$ in (15)
denotes the goal orientation and it can be the identity orientation
$1+[0\,\,0\,\,0]{{}^{\top}}$ or the average of the demonstration quaternion
profile, which can be estimated using MLE proposed in [22].
$2\,\text{Log}^{q}(\boldsymbol{\mathbf{g}}*\overline{\boldsymbol{\mathbf{q}}})$
defines the angular velocity $\boldsymbol{\mathbf{\omega}}$ that rotates a
unit quaternion $\boldsymbol{\mathbf{q}}$ into $\boldsymbol{\mathbf{g}}$
within a unit sampling time. $\boldsymbol{\mathbf{\Omega}}$ is the $3\times 3$
diagonal matrix of frequencies and the nonlinear forcing term is

$\boldsymbol{\mathbf{f}}(\boldsymbol{\mathbf{\phi}})=\boldsymbol{\mathbf{A}}_{r}\frac{{\sum_{i=1}^{N}{\boldsymbol{\mathbf{w}}_{i}\Psi_{i}(\phi)}}}{{\sum_{i=1}^{N}{\Psi_{i}(\phi)}}},$
(17)

where $\boldsymbol{\mathbf{w}}_{i}$ are the weights needed to follow any given
rotation profile. We estimate the weights by

$\begin{split}\small\frac{{\sum_{i=1}^{N}{\boldsymbol{\mathbf{w}}_{i}\Psi_{i}(\phi)}}}{{\sum_{i=1}^{N}{\Psi_{i}(\phi)}}}=\boldsymbol{\mathbf{A}}_{r}^{-1}\left(\boldsymbol{\mathbf{\Omega}}^{-1}\boldsymbol{\mathbf{\dot{\omega}}}-(\alpha_{z}(\beta_{z}2\,\text{Log}^{q}(\boldsymbol{\mathbf{g}}*\overline{\boldsymbol{\mathbf{q}}})-\boldsymbol{\mathbf{\omega}}))\right),\end{split}$
(18)

where $\boldsymbol{\mathbf{A}}_{r}$ is $3\times 3$ diagonal matrix of
amplitude modulators. The integration of (16) is done similarly as in (14),
i.e.,

$\boldsymbol{\mathbf{q}}(t+\delta t)=\text{Exp}^{q}\left(\frac{\delta
t}{2}\boldsymbol{\mathbf{\Omega}}\,\boldsymbol{\mathbf{\eta}}(t)\right).$ (19)

##  V Validation

In order to illustrate the performance of both proposed approaches, we have
conducted several examples in simulation as well as in real setup as follows:

  * •

In simulation: (_i_) comparison between both approaches while learning
periodic orientation trajectories, and (_ii_) coupling the adaptive oscillator
to the first approach (Section IV-A) to control periodic state variables.

  * •

Real experiments: (_i_) wiping of uneven surface, and (_ii_) mixing of liquids
by shaking a bottle.

###  V-A Simulations

$\nu$
---

$\boldsymbol{\mathbf{\dot{\omega}}}$ [rad$\backslash$s2]
---

$\boldsymbol{\mathbf{\omega}}$ [rad$\backslash$s]
---

$u_{z}$
---

$u_{y}$
---

$u_{x}$
---

-0.5
---
0.5
---
0
---
-0.6
---
-0.2
---
0.2
---
-0.2
---
0.2
---
0
---
-0.4
---
0.5
---
1
---
Unit quaternion $\mathbf{q}=\nu+\mathbf{u}$
---
2
---
0
---
-2
---
2
---
0
---
-2
---
$\omega_{x}$
---
$\omega_{y}$
---
$\omega_{z}$
---
$\dot{\omega}_{x}$
---
$\dot{\omega}_{y}$
---
$\dot{\omega}_{z}$
---
0
---
10
---
20
---
30
---
40
---
50
---
Time [s]
---
0
---
10
---
20
---
30
---
40
---
50
---
Time [s]
---
Figure 3: Reproduction of a rhythmic motion using QP-DMP (left column graphs)
and RMP-DMP (right column graphs). Dashed black lines correspond to the
original motion.

Figure 3 shows the simulation results of using QP-DMP and RMP-DMP. We
generated a synthetic periodic orientation trajectory (dashed lines) and use
both DMP approaches to encode it. Both approaches were able to successfully
encode the periodic trajectory with unit quaternions (solid lines). The first
four graphs in Fig. 3 show the individual elements of quaternions, while the
last two show angular velocity and acceleration, respectively. There is a
negligible error between the demonstrated orientation motion and the encoded
motion. Also, we did not observe any significant difference between the two
DMP formulations. In both cases, we initialize the DMP state such that the
initial orientation differs from the demonstrated one (first four graphs in
Fig. 3). This is to show that the DMP effectively converges to a limit cycle
that reproduces the demonstration. In this test, we set $\alpha_{z}=48$,
$\lambda=0.994$, and
$\boldsymbol{\mathbf{A}}_{r}=\text{diag}\left([1\,1\,1]\right)$.

In order to correctly represent the orientation, it is of importance that the
norm of the quaternions is unitary. To estimate the ability of the proposed
two approaches in keeping this condition, we performed a simulation that
tested the norm during the adaptation.

$u_{z}$
---

$u_{y}$
---

$u_{x}$
---

$\nu$
---

0.5
---
-0.5
---
0
---
0.5
---
-0.5
---
0
---
0.5
---
-0.5
---
0
---
0.5
---
-0.5
---
0
---
0
---
5
---
10
---
15
---
7
---
7.5
---
8
---
zoom in
---
Time [s]
---

$\Omega$ [Hz]
---

0
---
5
---
10
---
15
---
20
---
0.7
---
0.8
---
0.9
---
1
---
Time [s]
---
Figure 4: Results of coupling the proposed periodic orientation DMPs with an
adaptive oscillator. The first four graphs show the elements of quaternions as
demonstrated (dashed line) and learned (solid line). The frequency of the
system is shown in the bottom graph.

The phase and frequency of the demonstrated periodic motion has to be known in
order to be able to learn and encode the DMP. However, in practical scenario
where the learning is done online, it is difficult to maintain a constant pre-
planned frequency in real-time. Therefore, adaptive oscillators can be used to
dynamically estimate the phase and frequency of the demonstrated motion in
real-time. We performed a simulation to show that the periodic state variables
(phase and frequency), as estimated by the adaptive oscillator, can be
successfully coupled to the periodic quaternion DMPs that is operating in the
tangent space. The result of the coupling is shown in Fig. 4. We can see that
the system successfully converged to the given periodic frequency, as
estimated from the demonstrated input signal.

###  V-B Experiments

We chose surface polishing/wiping and mixing liquids by shaking tasks because
they both involve periodic changes of orientation. Many objects are not flat
and to polish or wipe them the robot should adapt the orientation periodically
in way that the contact with the polishing/wiping tool is perpendicular to the
object surface. In case of mixing liquids, we typically change the orientation
to shake the container. Therefore, the robot should periodically change the
orientation back and forth to stir the liquids inside the container. Both
tasks are executed by a $7$ DoF Franka Emika Panda robot using Cartesian
impedance control. In the mixing task, the position is kept fixed at its
initial value.

The series of photos in top-row of Fig. 5 shows the robot performing an object
shaking task, which is important for mixing various liquids. To perform this
task the robot had to periodically change the orientation of the endpoint
around a certain point, while holding the container with liquids, in order to
produce rapid changes in acceleration (i.e., jerk). Like in the previous
tasks, we used the proposed DMP method to encode the appropriate orientation
modulating trajectories in order to successfully execute the object shaking
task. The demonstrated orientation trajectory and the encoded one are shown in
Fig. 6. We can see that the desired trajectory was reproduced well by the
robot during the experiment.

In a different scenario, bottom-row of Fig. 5 shows the robot performing
polishing/wiping of a highly curved surface. Such task requires the robot to
periodically change its end-effector orientation and position in order to keep
the contact with the surface of the object. We used the proposed DMP method to
encode the appropriate orientation modulating trajectories in order to
successfully execute the task. The demonstrated and encoded periodic
orientation trajectories are shown in Fig. 7.

Figure 5: Franka Emika Panda executes two tasks that require periodic
orientation motion with/without periodic position motion: shaking a bottle of
liquids (top) and polishing task of curved objects (bottom). 0
---
10
---
20
---
30
---
40
---
50
---
60
---
Time [s]
---
-1
---
-0.5
---
0
---
0.5
---

$\boldsymbol{\mathbf{q}}$
---

$\boldsymbol{\mathbf{q}}^{rob}$
---
$\boldsymbol{\mathbf{q}}^{dmp}$
---
$\boldsymbol{\mathbf{q}}^{demo}$
---
Figure 6: The response of the proposed DMP for periodic orientation during the
execution of the shaking task. $\boldsymbol{\mathbf{q}}^{rob}$ is actual robot
orientation trajectory, $\boldsymbol{\mathbf{q}}^{dmp}$ is the reference
orientation trajectory reproduced by our approach, while
$\boldsymbol{\mathbf{q}}^{demo}$ is the orientation trajectory used to
demonstrate the task.

##  VI Conclusions

In this paper, we proposed two new approaches to learn periodic orientation
motion, represented by unit quaternions, using DMP. In the first approach, we
exploited Riemannian metrics and operators to perform the learning online on
the tangent space of $\mathbb{S}^{3}$. In the second one, we reformulate the
periodic DMP equations to directly learn and integrate the unit quaternions
online. The performance of both approaches has been validated in simulation as
well as in real setups.

In our experiments, we did not observe a significant outperforming of one of
the approaches, as they both behaved similarly. However, the RMP-DMP has a
simpler implementation as it only requires to map the training data onto the
tangent space and learn a classical DMP there. In other words, one does not
have to change the underlying DMP formulation and can reuse existing
implementations. Moreover, working on the tangent space is a general approach
that can be potentially applied to other Riemannian manifolds. Extending the
periodic DMP formulation to different Riemannian manifolds is the focus of our
future research.

0
---
20
---
40
---
60
---
80
---
Time [s]
---
-1
---
-0.5
---
0
---

$\boldsymbol{\mathbf{q}}$
---

0
---
0.2
---
0.4
---

$\boldsymbol{\mathbf{p}}$
---

$\boldsymbol{\mathbf{q}}^{rob}$
---
$\boldsymbol{\mathbf{q}}^{dmp}$
---
$\boldsymbol{\mathbf{q}}^{demo}$
---
$\boldsymbol{\mathbf{p}}^{rob}$
---
$\boldsymbol{\mathbf{p}}^{dmp}$
---
$\boldsymbol{\mathbf{p}}^{demo}$
---
Figure 7: The response of the proposed DMP for periodic orientation while
executing the polishing/wiping task. $\boldsymbol{\mathbf{q}}^{rob}$,
$\boldsymbol{\mathbf{q}}^{dmp}$, and $\boldsymbol{\mathbf{q}}^{demo}$ are
defined as in Fig. 6.
$\boldsymbol{\mathbf{p}}=[p_{x}\,p_{y}\,p_{z}]{{}^{\top}}$ is the position
trajectory of the end-effector.

## References

  * [1] A. J. Ijspeert, J. Nakanishi, and S. Schaal, “Learning rhythmic movements by demonstration using nonlinear oscillators,” in _IROS_ , vol. 1, Lausanne, Switzerland, 2002, pp. 958–963.
  * [2] M. Saveriano, F. J. Abu-Dakka, A. Kramberger, and L. Peternel, “Dynamic movement primitives in robotics: A tutorial survey,” _arXiv preprint arXiv:2102.03861_ , 2021.
  * [3] A. Gams, T. Petrič, M. Do, B. Nemec, J. Morimoto, T. Asfour, and A. Ude, “Adaptation and coaching of periodic motion primitives through physical and visual interaction,” _Rob. Auton. Syst._ , vol. 75, pp. 340–351, 2016.
  * [4] L. Peternel, N. Tsagarakis, D. Caldwell, and A. Ajoudani, “Robot adaptation to human physical fatigue in human–robot co-manipulation,” _Auton. Robots_ , vol. 42, no. 5, pp. 1011–1021, 2018.
  * [5] L. Peternel, T. Petrič, and J. Babič, “Robotic assembly solution by human-in-the-loop teaching method based on real-time stiffness modulation,” _Auton. Robots_ , vol. 42, no. 1, pp. 1–17, 2018.
  * [6] E. Rückert and A. d’Avella, “Learned parametrized dynamic movement primitives with shared synergies for controlling robotic and musculoskeletal systems,” _Front. Comput. Neurosci._ , 2013.
  * [7] L. Peternel, T. Noda, T. Petrič, A. Ude, J. Morimoto, and J. Babič, “Adaptive control of exoskeleton robots for periodic assistive behaviours based on emg feedback minimisation,” _PLOS ONE_ , 2016.
  * [8] P. Pastor, L. Righetti, M. Kalakrishnan, and S. Schaal, “Online movement adaptation based on previous sensor experiences,” in _IROS_ , San Francisco, CA, USA, 2011, pp. 365–371.
  * [9] F. J. Abu-Dakka, B. Nemec, J. A. Jørgensen, T. R. Savarimuthu, N. Krüger, and A. Ude, “Adaptation of manipulation skills in physical contact with the environment to reference force profiles,” _Auton. Robots_ , vol. 39, no. 2, pp. 199–217, 2015.
  * [10] L. Koutras and Z. Doulgeri, “A correct formulation for the orientation dynamic movement primitives for robot control in the cartesian space,” in _Conference on robot learning_ , 2020, pp. 293–302.
  * [11] A. Ude, B. Nemec, T. Petric, and J. Morimoto, “Orientation in cartesian space dynamic movement primitives,” in _ICRA_ , 2014, pp. 2997–3004.
  * [12] A. J. Ijspeert, J. Nakanishi, H. Hoffmann, P. Pastor, and S. Schaal, “Dynamical Movement Primitives: Learning Attractor Models for Motor Behaviors,” _Neural Comput._ , vol. 25, no. 2, pp. 328–373, 2013.
  * [13] A. Paraschos, C. Daniel, J. Peters, and G. Neumann, “Probabilistic movement primitives,” in _NeurIPS_ , 2013, pp. 2616–2624.
  * [14] S. M. Khansari-Zadeh and A. Billard, “Learning stable non-linear dynamical systems with gaussian mixture models,” _IEEE Trans. Robot._ , vol. 27, no. 5, pp. 943–957, 2011.
  * [15] M. Saveriano, “An energy-based approach to ensure the stability of learned dynamical systems,” in _ICRA_ , 2020, pp. 4407–4413.
  * [16] S. Calinon, D. Bruno, and D. G. Caldwell, “A task-parameterized probabilistic model with minimal intervention control,” in _ICRA_ , 2014, pp. 3339–3344.
  * [17] Y. Huang, F. J. Abu-Dakka, J. Silvério, and D. G. Caldwell, “Toward orientation learning and adaptation in cartesian space,” _IEEE Trans. Robot._ , vol. 37, no. 1, pp. 82–98, 2021.
  * [18] P. Pastor, H. Hoffmann, T. Asfour, and S. Schaal, “Learning and generalization of motor skills by learning from demonstration,” in _ICRA_ , 2009, pp. 763–768.
  * [19] J. Silvério, L. Rozo, S. Calinon, and D. G. Caldwell, “Learning bimanual end-effector poses from demonstrations using task-parameterized dynamical systems,” in _IROS_ , 2015, pp. 464–470.
  * [20] M. Saveriano, F. Franzel, and D. Lee, “Merging position and orientation motion primitives,” in _ICRA_ , 2019, pp. 7041–7047.
  * [21] S. Kim, R. Haschke, and H. Ritter, “Gaussian mixture model for 3-dof orientations,” _Rob. Auton. Syst._ , vol. 87, pp. 28–37, 2017.
  * [22] M. J. Zeestraten, I. Havoutis, J. Silvério, S. Calinon, and D. G. Caldwell, “An approach for imitation learning on riemannian manifolds,” _IEEE Robot. Autom. Lett._ , vol. 2, no. 3, pp. 1240–1247, 2017.
  * [23] A. Gams, A. Ijspeert, S. Schaal, and J. Lenarčič, “On-line learning and modulation of periodic movements with nonlinear dynamical systems,” _Auton. Robots_ , vol. 27, no. 1, pp. 3–23, 2009.
  * [24] J. Ernesti, L. Righetti, M. Do, T. Asfour, and S. Schaal, “Encoding of periodic and their transient motions by a single dynamic movement primitive,” in _IEEE-RAS Humanoids_ , 2012, pp. 57–64.
  * [25] T. Petrič, A. Gams, A. J. Ijspeert, and L. Žlajpah, “On-line frequency adaptation and movement imitation for rhythmic robotic tasks,” _Int. J. Robot. Res._ , vol. 30, no. 14, pp. 1775–1788, 2011.

  *[DMPs]: Dynamic Movement Primitive
  *[DoF]: Degree of Freedom
  *[DoFs]: Degree of Freedom
  *[DMP]: Dynamic Movement Primitive
  *[]:
  *[RMP-DMP]: Riemannian Metric-based Periodic
  *[QP-DMP]: Unit Quaternion-based Periodic
  *[LfD]: Learning from Demonstration
  *[ProMP]: Probabilistic Movement Primitives
  *[SDS]: Stable Dynamical Systems
  *[GMM]: Gaussian Mixture Model
  *[TP-GMM]: Task-Parameterized   ()
  *[KMP]: Kernelized Movement Primitive
  *[MLE]: Maximum Likelihood Estimation


# HDRfeat: A Feature-Rich Network
for High Dynamic Range Image Reconstruction

Lingkai Zhu, Fei Zhou, Bozhi Liu, Orcun Goksel L. Zhu and O. Goksel are with
the Department of Information Technology, Uppsala University, Sweden. F. Zhou
and B. Liu are with the College of Electronic and Information Engineering,
Shenzhen University, China. This work was performed during B. Liu’s visit at
Uppsala University, Sweden.

###### Abstract

A major challenge for high dynamic range (HDR) image reconstruction from
multi-exposed low dynamic range (LDR) images, especially with dynamic scenes,
is the extraction and merging of relevant contextual features in order to
suppress any ghosting and blurring artifacts from moving objects. To tackle
this, in this work we propose a novel network for HDR reconstruction with deep
and rich feature extraction layers, including residual attention blocks with
sequential channel and spatial attention. For the compression of the rich-
features to the HDR domain, a residual feature distillation block (RFDB) based
architecture is adopted. In contrast to earlier deep-learning methods for HDR,
the above contributions shift focus from merging/compression to feature
extraction, the added value of which we demonstrate with ablation experiments.
We present qualitative and quantitative comparisons on a public benchmark
dataset, showing that our proposed method outperforms the state-of-the-art.

###### Index Terms:

HDR, hierarchical features, attention

##  I Introduction

Dynamic range is the luminance ratio between the brightest and darkest areas
in a scene. Natural scenes have a much higher luminance range than digital
cameras can represent, resulting in captured low dynamic range (LDR) images to
often have over- or under-exposed regions [1]. High dynamic range (HDR) images
can represent a wide range of illuminations.

The typical way to generate an HDR image is to merge a series of LDR images
with different exposures of the same scene captured by a positionally-fixed
camera. Given the acquired LDR images, some methods focus on recovering the
camera response function [2, 3, 4, 1, 5, 6], which is the relation between the
irradiance map and the HDR-image pixel values. Others adopt a multiple
exposure fusion approach [7, 8, 9, 10], which combines the multi-exposed LDR
images to estimate the irradiance value for each pixel. However, if the input
LDR images contain large foreground motions, the merged HDR image will suffer
from ghosting and blurring artifacts due to the misalignment among the LDR
images.

Figure 1: Visual results on the test dataset [11]. The top row shows three
input LDR images, the tone-mapped HDR image, and sample LDR patches at the
same image location. The bottom row compares the same patch from HDR images
reconstructed by state-of-the-art methods and our proposed method HDRfeat. The
arrows highlight some artifacts, discussed later in our Results.

Several deep learning approaches have been proposed to address the
aforementioned problems [11, 12, 13, 14, 15, 16, 17]. For example, Kalantari
et al. [11] used optical flow to align input images and then merged the
aligned images with a four-layer fully convolutional neural network (CNN) to
generate a restored HDR image. However, such a simple neural network structure
cannot handle the artifacts and distortion caused by errors from an unreliable
optical flow. Wu et al. [12] proposed the first end-to-end deep neural network
without optical flow alignment for HDR reconstruction, which includes three
encoder networks, a merger network, and a decoder network. However, due to the
lack of an explicit alignment mechanism among LDR images, this approach
suffers from occlusion and ghosting. In their seminal work, Yan et al. [13]
were the first to use attention networks (AHDRNet) aiming to highlight or
align large motions in the features prior to the merger network, which can
then suppress any ghosting artifacts. Besides the attention network
dramatically increasing the computational cost, there still existed blurring
or artifacts in large motion scenarios as shown in Fig. 1. Nevertheless, this
work inspired several following works, which adapted their proposed
architecture; for example, see several solutions in the recent NTIRE 2021 and
2022 challenges on HDR [18, 19]. In [15], Yan et al. proposed DAHDRNet with
recursive channel and spatial attention for more effective de-ghosting. Most
deep-learning solutions to HDR (e.g. AHDRNet and its derivatives) involve two
consecutive stages overall: extraction of features from LDR images, and based
on these merging/compression into an HDR image. The latter stage in essence is
a function of LDR images, e.g. how to scale and merge them locally. It is the
features from the former stage that parametrizes such merging function, e.g.
that should highlight/suppress misaligned regions, occlusions, etc. In other
words, one can expect the former task to be far more complex than the latter
task. Based on this motivation, we propose a network architecture (HDRfeat)
which is rich in feature extraction with attention and channel-wise
hierarchical layers, while lighter on the latter reconstruction side, compared
to most state-of-the-art solutions. Accordingly, our main contributions herein
include: $\bullet$ a novel channel-wise hierarchical feature extraction
network to efficiently extract rich contextual information from multi-exposure
images, with a channel bottleneck structure to compress these LDR feature
representations purposed for subsequent HDR reconstruction; $\bullet$ a
residual attention block with sequential channel and spatial attention to
allow the features to focus where needed, e.g. large motions; $\bullet$ the
state-of-the-art on HDR reconstruction, demonstrated via extensive evaluations
on a public benchmark dataset.

##  II Methods

Figure 2: Proposed network architecture, involving sub-networks for
hierarchical rich feature extraction with sequential attention blocks and HDR
reconstruction via residual feature distillation. The former sub-network
applies channel-wise hierarchical processing (in contrast to spatial
hierarchy/level-of-detail, e.g., as in [17]) together with two sequential
(channel-spatial) attention blocks with residual connections and shared
weights to learn merging information from non-reference exposures. The
reconstruction sub-network inspired by [13, 15] uses the merged features and
the first-level feature of the medium-exposure image to generate an HDR image
via three consecutive residual feature distillation blocks (RFDBs) inspired by
[20] in addition to a long skip-connection that helps preserve and incorporate
early features closer to the original input image.

Let $\\{I_{1},I_{2},I_{3}\\}$ be a set of 3 LDR images denoting short, medium,
and long exposures, respectively. Following [11], we preprocess $\mathcal{I}$
by gamma correction ($\gamma$ = 2.2 herein) to generate a set of mapped images
$\\{H_{1},H_{2},H_{3}\\}$, which concatenate with the LDR images, i.e.
$X_{i}=\text{concat}(I_{i},H_{i})$ as input to our network. Accordingly, an
HDR image $H$ is restored by the proposed HDRfeat network $h(\cdot)$ as

$\displaystyle H=h(X_{1},X_{2},X_{3};\theta)$ (1)

where $\theta$ are the network parameters. We train this network for visual
acuity using a loss on tone-mapped images [11] as

$\displaystyle\mathcal{L}=\|\mathcal{T}(H)-\mathcal{T}(\hat{H})\|_{1}$ (2)

where $H$ and $\hat{H}$ are the predicted and groundtruth HDR images, and
$\mathcal{T}(H)=\frac{\log(1+\mu H)}{\log(1+\mu)}$ is the $\mu$-law tone
mapping with $H$ scaled within $[0,1]$ and $\mu$ = 5000 in this work.

###  II-A Channel-wise Hierarchical Feature Extraction Network

As the input LDR images contain rich contextual information, e.g. about
foreground motions and the scene background, simple layers such as in [13] may
not be sufficient to effectively extract such information. In the GAN-based
solution of [17], a spatially multi-scale deep encoder was proposed to extract
visual feature from LDR images. Nevertheless, such spatial down- and
upsampling may lose the fine alignment information required in the subsequent
reconstruction stage. Furthermore, this operation is limited in enriching
features in the channel dimension, which is potentially more relevant.
Therefore, inspired by the spatially hierarchical feature extraction in the
generator of [17], we herein introduce a channel-wise hierarchical rich
feature extractor (Fig. 2) and incorporate this in a CNN solution, which is
more stable to train compared to GANS. For channel-wise hierarchy, we extract
features at the same scale with expanding channel sizes, which are then merged
at bottlenecks for channel dimensions to create summary features for the
reconstruction.

Architecturally, each depth of channel-wise feature extractor is composed of
three shared convolution operators $E$ (3$\times$3 Conv with zero-padding of 1
pixel), extracting different levels of information denoted here as
$F_{i,j}(i,j\in\\{1,2,3\\})$ with $i$ the hierarchical depth and $j$ the LDR
index, i.e.:

$\displaystyle F_{1,j}$ $\displaystyle=E_{1,j}(X_{j})$ (3) $\displaystyle
F_{2,j}$ $\displaystyle=E_{2,j}(F_{1,j})$ (4) $\displaystyle F_{3,j}$
$\displaystyle=E_{3,j}(F_{2,j})$ (5)

with respective dimensions of 64$\times$128$\times$128,
128$\times$128$\times$128, and 196$\times$128$\times$128.

In the spirit of several works that demonstrated the value of attention in
HDR, we also adopt this strategy. To that end, we introduce convolutional
block attention modules (CBAM) with residual attention for the first time in
HDR reconstruction, as described in detail in the following subsection.
Ultimately, our attention residual blocks $A_{1}$ and $A_{3}$ help introduce
attention from the medium-exposure features $F_{1,2}$ as reference, into the
short- and long-exposure features $F_{1,1}$ and $F_{1,3}$, generating the
feature maps

$\displaystyle a_{i}=A_{i}(\,F_{1,i}\,,\,F_{1,2}\,),\qquad i\in\\{1,3\\}\,.$
(6)

The rich contextual information (denoted by the red lines in Fig. 2) are next
compressed via sequential concatenation and (3$\times$3) convolutions, to
obtain summary feature representations $F_{j}$ at each hierarchical depth $j$
as follows:

$\displaystyle F_{3}$
$\displaystyle=\text{Conv}\big{(}\text{Concat}(F_{3,1},F_{3,2},F_{3,3})\big{)}$
(7) $\displaystyle F_{2}$
$\displaystyle=\text{Conv}\big{(}\text{Concat}(F_{2,1},F_{2,2},F_{2,3},F_{3})\big{)}$
(8) $\displaystyle F_{1}$
$\displaystyle=\text{Conv}\big{(}\text{Concat}(a_{1},F_{1,2},a_{3},F_{2})\big{)}$
(9)

all with dimensions 64$\times$128$\times$128.

###  II-B Residual Attention Block

Attention mechanisms have been shown to be beneficial in several deep learning
tasks, such as object detection [21] and super-resolution [22], as well as HDR
reconstruction [13, 15]. Although spatial and channel attention can both be
instrumental and can help reduce ghosting artifacts [13, 15], their
introduction using traditional parallel layer branching, as in [13, 15],
requires high computational resources [23]. Furthermore,the multiplication of
such multiple attention branches makes the gradients in general unstable to
effectively back-propagate during training. To address these, we adopt a
sequential attention mechanism, inspired by Convolutional Block Attention
Module (CBAM) [21], where first the global information is put in focus via
channel attention and then the local information is explored via spatial
attention. Thus, such attention mechanism operates in a global-to-local
manner. In contrast to [21], which involves a single image input (for object
detection purposes), we wish to inject attention with the help of one image
(reference) into another (target). Accordingly, we first concatenate both
image features as our block input, and after the addition of attention, we
bring back the target image as a residual input, such that the attention will
only need to robustly learn the difference from the target (see Fig. 3).

Figure 3: Our attention module brings in attention from the medium-exposed
(reference) image features $F_{\mathrm{R}}$ into the short- or long-exposed
image (target) feature $F_{\mathrm{T}}$, by first fusing them (concatenation +
1$\times$1 convolution) into $F_{\mathrm{F}}$, then finding channel and
spatial attention sequentially, then applying this on the target features, and
finally adding target features as a residual input.

For a formal definition, let the medium-exposed image (reference) features
$F_{1,2}$ be denoted as $F_{\mathrm{R}}$, and the short- or long-exposed image
(target) features, $F_{1,1}$ or $F_{1,3}$, as $F_{\mathrm{T}}$. We then find
the final attention applied features $M\in[0,1]$, with the same dimension as
$F_{R}$ and $F_{T}$, as follows:

$\displaystyle M$ $\displaystyle=a(F_{\mathrm{R}},F_{\mathrm{T}})$ (10)
$\displaystyle M$ $\displaystyle=F_{\mathrm{T}}+F_{\mathrm{T}}\otimes
F_{\mathrm{S}}$ (11) $\displaystyle F_{\mathrm{S}}$
$\displaystyle=F_{\mathrm{C}}\otimes m_{\mathrm{S}}(F_{\mathrm{C}})$ (12)
$\displaystyle F_{\mathrm{C}}$ $\displaystyle=F_{\mathrm{S}}\otimes
m_{\mathrm{C}}(F_{\mathrm{I}})$ (13) $\displaystyle F_{\mathrm{F}}$
$\displaystyle=\text{Conv}(\text{Concat}(F_{\mathrm{R}},F_{\mathrm{T}}))$ (14)

where the operator $\otimes$ denotes the element-wise multiplication;
$F_{\mathrm{F}}$, $F_{\mathrm{C}}$, and $F_{\mathrm{S}}$ denote features for
fused features, spatial attention, and channel attention; Conv is a 1$\times$1
convolution; and $m_{\mathrm{C}}$ and $m_{\mathrm{S}}$ are core attention
modules with small light-weight CNNs for low computation cost. Channel
attention $m_{\mathrm{C}}$ of dimension 64 first calculates the average-pooled
feature and max-pooled feature separately, then forwards them to a multilayer
perceptron (MLP) with one hidden layer. Next, the two output features of MLP
are merged via element-wise summation and gated by a sigmoid function. Spatial
attention $m_{\mathrm{S}}$ of dimension 128$\times$128 first calculates the
average-pooled feature and max-pooled feature separately, which then will be
concatenated, forwarded to a 7$\times$7 Conv layer, and gated by the sigmoid
function.

Intuitively, max-pooling extracts the most salient features that are the
features with large foreground motions while the average-pooling extract the
global features. By combining both features, our attention module is able to
concentrate on features with large object movement in the global scenery.

###  II-C HDR Reconstruction Network for HDR Image Restoration

We adapt the HDR reconstruction architecture from [13, 15]. Since, given
suitable features as input, this latter reconstruction processing stage should
be intuitively far less complex compared to the former feature extraction, we
propose the replacement of the original cumbersome dilated residual dense
blocks (DRDBs) with a more light-weight structure. To that end, we adopt
residual feature distillation blocks (RFDBs) proposed in [20] for image super-
resolution. We extend RFDB by incorporating dilated convolutions in order to
capture larger context for our purposes of HDR reconstruction. Accordingly,
our RFDB adaptation consists of 2-dilated convolution layers, shallow residual
blocks and enhanced spatial attention (ESA) blocks. In our preliminary
experiments, we observed this reconstruction architecture to have performance
comparable to DRDB, but with much smaller number of parameters, which we then
use to invest in our rich-feature extraction network. Since training such
networks with reasonable batch sizes takes a large part of typical GPU memory,
designing and investing parameters (degrees-of-freedom) within an overall
network architecture in line with the particular problem structure is
therefore fundamental in preventing overfitting and making best use of
available resources. Accordingly, our proposed shift of network complexity
from reconstruction to feature extraction is a major contribution of this
work, and it is a main component that makes HDRfeat competitive and superior
to state-of-the-art.

##  III Experiments and Results

###  III-A Experimental Setting and Details

We train our proposed model on Kalantari’s HDR dataset [11], which contains 74
images for training and 15 images for evaluation, all with corresponding LDR
and HDR images (of resolution 1500$\times$1000) as groundtruth. Our proposed
method was implemented in Pytorch and trained on a RTX 3090 GPU with 16 000
epochs. For training, instead of feeding entire images into the network, which
would require unattainable amount of memory, we randomly crop images into 256
$\times$ 256 patches. We apply data augmentation (flips and rotations) during
training. We use the Adam optimizer [24], with a learning rate initialized to
$10^{-4}$ for all layers and decreased to $10^{-5}$ after 12 000 epochs. All
convolution layer weights are initialized with the Kaiming method [25]. For
evaluation, we compute the PSNR and SSIM scores between the restored HDR image
and the groundtruth HDR image in the linear domain (PSNR-L, SSIM-L) as well as
after tone mapping using $\mu$-law (PSNR-T, SSIM-T). We also compute the HDR-
VDP-2 score [26].

###  III-B Comparison with state-of-the-art

We compare our proposed method with state-of-the-art approaches, including a
patch-based technique [8], an optical-flow based method with a CNN merger
[11], as well as four other well-known deep-learning based methods [12, 15,
13, 17]. All methods are reproduced using the codes provided by the authors,
except for the results of [15] reported directly from their paper. The
comparison is presented in Table I.

TABLE I: Quantitative comparison of our method on the test set [11].  | PSNR-T | PSNR-L | SSIM-T | SSIM-L | HDR-VDP-2
---|---|---|---|---|---
Sen et al.[8] | 40.95 | 38.27 | 0.9858 | 0.9762 | 61.72
Kalantari et al.[11] | 42.67 | 41.21 | 0.9889 | 0.9829 | 65.01
Wu et al.[12] | 42.70 | 41.13 | 0.9910 | 0.9889 | 66.20
Yan et al.[13] | 43.62 | 41.03 | 0.9919 | 0.9887 | 65.79
Yan et al.[15] | 43.84 | 41.31 | - | - | -
Niu et al.[17] | 43.92 | 41.57 | 0.9925 | 0.9898 | 65.81
HDRfeat (ours) | 44.11 | 41.79 | 0.9931 | 0.9912 | 66.74

The result shows that our method outperforms the state-of-the-art techniques
on all metrics. In general, our method is seen to produce high quality output
with less artifacts in saturated and motion-involved (occluded) image regions.
Fig. 1 shows a visual comparison of the tone mapped HDR images, where a
saturated image region is shown as zoomed-in: As shown with the arrows in the
images, [8] and [11] show superfluous contrast, and [12], [13], and [17] show
quantization artifacts, while our results are superior visually. Note that no
visual comparison could be included from [15] since the code was not available
at the time of this work. Fig. 4 shows an example, where the zoomed in
location shows the car door that is occluded by the moving arm in one
exposure: As seen, our proposed method has minimal to no artifacts, whereas
[8] and [11] show irrelevant content; [12] and [13] show artifactual
boundaries from occluding arm; and [17] shows some leakage.

Figure 4: Visual comparison on [11]. The arrows highlight some artifacts in
the generated HDR images. Figure 5: Architectures in our ablation study of
feature extraction network, with varying depths and with/without attention
modules.

###  III-C Ablation study

We investigate the components of our proposed hierarchical rich feature
extraction, by ablation the attention architecture as well as the hierarchical
depth (richness) of the feature extraction. We accordingly compare our
proposed HDRfeat with its ablated variants having different feature extraction
depths and enabled/disabled attention modules, as shown in Fig. 5. In
addition, we compared our sequential arrangement of spatial and channel
attention, to its parallel counter-part [21], within the same HDRfeat
structure as in Fig. 5(f). Results are tabulated in Table II.

TABLE II: Ablation study, where Depth indicates the hierarchical feature extraction depth as an indicator for the richness of features, Att indicates whether attention is applied (+) or not (–), where +∗ indicates the use of parallel attention. Depth | Att | PSNR-T | PSNR-L | SSIM-T | SSIM-L | HDR-VDP-2
---|---|---|---|---|---|---
1 | – | 43.58 | 41.26 | 0.9924 | 0.9903 | 65.54
+ | 43.44 | 40.94 | 0.9924 | 0.9902 | 65.08
2 | – | 43.38 | 41.24 | 0.9926 | 0.9905 | 65.54
+ | 43.78 | 41.61 | 0.9926 | 0.9902 | 66.38
3 | – | 43.52 | 41.62 | 0.9928 | 0.9906 | 66.79
+ | 44.11 | 41.79 | 0.9931 | 0.9912 | 66.74
3 |  +∗ | 43.86 | 41.33 | 0.9928 | 0.9903 | 65.72

As seen, both the attention modules and the hierarchical rich-feature
extraction contribute to the results, with their combination as proposed
achieving the best results for most (4 out of 5) metrics. We also show that
our adapted sequential channel and spatial attention structure is superior to
its conventional parallel implementation.

##  IV conclusion

In this work, we propose a novel feature-rich network (HDRfeat) for HDR
reconstruction including a channel-wise feature extraction network to extract
and bottleneck rich contextual information from multi-exposure images to fit
in the HDR domain, a residual attention block with sequential channel and
spatial attention to concentrate on features with large motions and an HDR
reconstruction network with dilated residual feature distillation block (RFDB)
as the backbone. We perform qualitative and quantitative comparisons on the
public benchmark dataset, showing that the proposed method outperforms the
state-of-the-art methods.

## References

  * [1] E. Reinhard, W. Heidrich, P. Debevec, S. Pattanaik, G. Ward, and K. Myszkowski, _High dynamic range imaging: acquisition, display, and image-based lighting_. Morgan Kaufmann, 2010.
  * [2] P. E. Debevec and J. Malik, “Recovering high dynamic range radiance maps from photographs,” in _ACM SIGGRAPH Classes_ , 2008, pp. 1–10.
  * [3] S. Mann and R. Picard, “Being ‘undigital’; with digital cameras: Extending dynamic range by combining differently exposed pictures mit media lab perceptual computing section, boston,” MA, Tech. Rep. 323, Tech. Rep., 1994.
  * [4] M. Granados, B. Ajdin, M. Wand, C. Theobalt, H.-P. Seidel, and H. P. Lensch, “Optimal hdr reconstruction with linear digital cameras,” in _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2010, pp. 215–222.
  * [5] Q. Yan, J. Sun, H. Li, Y. Zhu, and Y. Zhang, “High dynamic range imaging by sparse representation,” _Neurocomputing_ , vol. 269, pp. 160–169, 2017.
  * [6] Y. Salih, A. S. Malik, N. Saad _et al._ , “Tone mapping of hdr images: A review,” in _IEEE International Conference on Intelligent and Advanced Systems (ICIAS)_ , vol. 1, 2012, pp. 368–373.
  * [7] T. Jinno and M. Okuda, “Multiple exposure fusion for high dynamic range image acquisition,” _IEEE Trans on Image Processing_ , vol. 21, no. 1, pp. 358–365, 2012.
  * [8] P. Sen, N. K. Kalantari, M. Yaesoubi, S. Darabi, D. B. Goldman, and E. Shechtman, “Robust Patch-Based HDR Reconstruction of Dynamic Scenes,” _ACM Trans on Graphics (TOG) (Procs of SIGGRAPH Asia)_ , vol. 31, no. 6, pp. 203:1–11, 2012.
  * [9] F. Xu, J. Liu, Y. Song, H. Sun, and X. Wang, “Multi-exposure image fusion techniques: A comprehensive review,” _Remote Sensing_ , vol. 14, no. 3, p. 771, 2022.
  * [10] X. Zhang, “Benchmarking and comparing multi-exposure image fusion algorithms,” _Information Fusion_ , vol. 74, pp. 111–131, 2021.
  * [11] N. K. Kalantari, R. Ramamoorthi _et al._ , “Deep high dynamic range imaging of dynamic scenes.” _ACM Trans on Graphics (TOG)_ , vol. 36, no. 4, pp. 144–1, 2017.
  * [12] S. Wu, J. Xu, Y.-W. Tai, and C.-K. Tang, “Deep high dynamic range imaging with large foreground motions,” in _European Conference on Computer Vision (ECCV)_ , 2018, pp. 117–132.
  * [13] Q. Yan, D. Gong, Q. Shi, A. v. d. Hengel, C. Shen, I. Reid, and Y. Zhang, “Attention-guided network for ghost-free high dynamic range imaging,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 1751–1760.
  * [14] Q. Yan, D. Gong, P. Zhang, Q. Shi, J. Sun, I. Reid, and Y. Zhang, “Multi-scale dense networks for deep high dynamic range imaging,” in _2019 IEEE Winter Conference on Applications of Computer Vision (WACV)_ , 2019, pp. 41–50.
  * [15] Q. Yan, D. Gong, J. Q. Shi, A. van den Hengel, C. Shen, I. Reid, and Y. Zhang, “Dual-attention-guided network for ghost-free high dynamic range imaging,” _International Journal of Computer Vision_ , vol. 130, no. 1, pp. 76–94, 2022\.
  * [16] J. Wang, X. Li, and H. Liu, “Exposure fusion using a relative generative adversarial network,” _IEICE Trans on Information and Systems_ , vol. 104, no. 7, pp. 1017–1027, 2021.
  * [17] Y. Niu, J. Wu, W. Liu, W. Guo, and R. W. H. Lau, “Hdr-gan: Hdr image reconstruction from multi-exposed ldr images with large motions,” _IEEE Trans on Image Processing_ , vol. 30, pp. 3885–3896, 2021.
  * [18] E. Pérez-Pellitero, S. Catley-Chandar, A. Leonardis, and R. Timofte, “NTIRE 2021 challenge on high dynamic range imaging: Dataset, methods and results,” _CoRR arXiv_ , no. 2106.01439, 2021.
  * [19] E. Pérez-Pellitero, S. Catley-Chandar, R. Shaw, A. Leonardis, R. Timofte, Z. Zhang, C. Liu, Y. Peng, Y. Lin, G. Yu _et al._ , “Ntire 2022 challenge on high dynamic range imaging: Methods and results,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 1009–1023.
  * [20] J. Liu, J. Tang, and G. Wu, “Residual feature distillation network for lightweight image super-resolution,” in _ECCV Workshops_ , 2020, pp. 41–55.
  * [21] S. Woo, J. Park, J.-Y. Lee, and I. S. Kweon, “Cbam: Convolutional block attention module,” in _European Conference on Computer Vision (ECCV)_ , 2018\.
  * [22] Y. Zhang, Y. Tian, Y. Kong, B. Zhong, and Y. Fu, “Residual dense network for image super-resolution,” in _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 2472–2481.
  * [23] L. Wang and K.-J. Yoon, “Deep learning for hdr imaging: State-of-the-art and future trends,” _IEEE Trans on Pattern Analysis and Machine Intelligence_ , 2021.
  * [24] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _CoRR arXiv_ , no. 1412.6980, 2015.
  * [25] K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,” in _IEEE Int Conference on Computer Vision (ICCV)_ , 2015, pp. 1026–1034.
  * [26] R. Mantiuk, K. J. Kim, A. G. Rempel, and W. Heidrich, “Hdr-vdp-2: A calibrated visual metric for visibility and quality predictions in all luminance conditions,” _ACM Trans on Graphics (TOG)_ , vol. 30, no. 4, pp. 1–14, 2011\.


# Open-source Frame Semantic Parsing

David Chanin
Department of Computer Science
University College London
<EMAIL_ADDRESS>

###### Abstract

While the state-of-the-art for frame semantic parsing has progressed
dramatically in recent years, it is still difficult for end-users to apply
state-of-the-art models in practice. To address this, we present Frame
Semantic Transformer, an open-source Python library which achieves near state-
of-the-art performance on FrameNet 1.7, while focusing on ease-of-use. We use
a T5 model fine-tuned on Propbank and FrameNet exemplars as a base, and
improve performance by using FrameNet lexical units to provide hints to T5 at
inference time. We enhance robustness to real-world data by using textual data
augmentations during training.

##  1 Introduction

Frame semantic parsing (Gildea and Jurafsky, 2002) is a natural language
understanding (NLU) task involving finding structured semantic frames and
their arguments from natural language text as formalized by the FrameNet
project (Baker et al., 1998). Frame semantics has proved useful in
understanding user intent from text, finding use in modern voice assistants
(Chen et al., 2019), dialog systems (Chen et al., 2013), and even text
analysis (Zhao et al., 2023).

A semantic frame in FrameNet describes an event, relation, or situation and
its participants. When a frame occurs in a sentence, there is typically a
"trigger" word in the sentence which is said to evoke the frame. In addition,
a frame contains a list of arguments known as frame elements which describe
the semantic roles that pertain to the frame. A sample sentence parsed for
frame and frame elements is shown in Figure 1.

FrameNet provides a list of lexical units (LUs) for each frame, which are word
senses with may evoke the frame when they occur in a sentence. For instance,
the frame "Attack" has lexical units "ambush.n", "ambush.v", "assault.v",
"attack.v", "attack.n", "bomb.v", and many others. These lexical units are not
exhaustive, however - a frame trigger may not necessarily be one of the
lexical units listed in the frame, but the lexical units provide a strong hint
that the frame may be present.

Figure 1: The sentence "Jaclyn gave the box to Mark" annotated with frame
trigger and frame elements for the "Giving" frame.

In this paper we treat frame semantic parsing as sequence-to-sequence text
generation task, and fine-tune a T5 transformer model (Raffel et al., 2020) as
the base model. We increase performance by pretraining on related datasets,
providing in-context prompt hints to T5 based on FrameNet data, and using
textual data augmentations (Ma, 2019) to increase training data. More details
on our implementation are given in Section 2

We evaluate the performance of Frame Semantic Transformer using the same
dataset splits as Open Sesame (Swayamdipta et al., 2017), the previous state-
of-the-art open-source parser. Frame Semantic Transformer exceeds the
performance of Open Sesame, and achieves near state-of-the-art performance
compared with modern frame semantic parsers that do not publish models
(Kalyanpur et al., 2020, Zheng et al., 2022).

The performance of Frame Semantic Transformer does not, however, come at the
cost of usability. The library can be installed via PyPI (pyp, ) with the
following command:

    pip install frame-semantic-transformer

Performing frame semantic parsing on a sentence can be achieved with a few
lines of code. We leverage the Huggingface (Wolf et al., 2020) model hub and
NLTK (Bird et al., 2009) corpora so that all required models and datasets are
automatically downloaded when frame semantic transformer is first run,
requiring no further action by the user aside from installing the library from
PyPI. Basic usage is shown in Figure 2. Pretrained models are provided based
on T5-base and T5-small, with T5-base being the default model used. The code
for Frame Semantic Transformer is available on Github
111https://github.com/chanind/frame-semantic-transformer.

            from frame_semantic_transformer import FrameSemanticTransformer
            frame_transformer = FrameSemanticTransformer()
            sentence = "The hallway smelt of boiled cabbage and old rag mats."
            results = frame_transformer.detect_frames(sentence)

Figure 2: Performing frame semantic parsing requires only a few lines of code
using Frame Semantic Tranformer. All needed pretrained models and datasets are
downloaded automatically.

##  2 Method

Typically, frame semantic parsing approaches treat the task as a set of 3
subtasks which happen in serial (Kalyanpur et al., 2020). First, in the
trigger identification subtask, all trigger locations are identified in the
text where a frame occurs. Second, in the frame classification subtask, each
identified trigger location is classified with a FrameNet frame. Finally, in
the arguments extraction subtask, frame elements and their arguments are
identified in the text.

We treat each of the 3 subtasks as sequence-to-sequence tasks performed in
series by a fine-tuned T5 model. Each of these tasks follows the format "<task
name> <task-specific hints> : <text>".

###  2.1 Trigger identification

Given a sentence, the trigger identification task identifies locations in the
sentence text which could be frame triggers. This task is conceptually the
simplest of the three - it has no task-specific hints, and the goal of the
task is to insert markers in the text to indicate frame triggers. For this
task, we us the asterisk character * to indicate a frame trigger. This is
shown in Figure 3.

input: | "TRIGGER: It was no use trying the lift."
---|---
output: | "It was no use *trying the *lift."
Figure 3: Trigger identification input and expected output for the text "It
was no use trying the lift.". Trigger locations are indicated by * in the
output

###  2.2 Frame classification

For each trigger identified in the trigger identification step, a frame
classification task is created to classify the frame that the trigger refers
to. To make this task easier for the model, we use the LUs from each frame to
build a list of possible frames this trigger could refer to.

We normalize trigger words and frame LUs using a similar process. First, we
lowercase the word and stem and lemmatize it using multiple stemmers and
lemmatizers. Each stemmer and lemmatizer may treat different English words
slightly differently, so multiple are used to increase the chance the
normalized trigger word will match a normalized LU from framenet.
Specifically, we use an English Snowball stemmer (Porter, 1980), a Lancaster
stemmer (Paice, 1990), a Porter stemmer (Porter, 1980), and a lemmatizer based
on WordNet (Miller, 1995), all from NLTK to generate a set of up to 4 possible
normalized versions of the trigger word.

For LUs, we also remove the part of speech (POS) tag. T5 is a powerful
transformer model and likely does not need to be provided with POS info,
although this is something that could be explored in future work. In addition,
for the trigger word, we also generate bigrams for the trigger and the words
on either side of the trigger, and normalize the bigrams in the same way. Some
LUs contain multiple words, so generating bigrams increases the chance that
after this normalization process the matching frame is found and can be added
as a hint.

For instance, for the trigger word "trying" from Figure 3, this word has the
bigrams "use_trying" and "trying_to", and the monogram "trying". After
normalization, these become the lookup set:

{ us_tri, us_try, us_trying, use_tri, use_try, use_trying, tri, try, trying,
tri_to, try_to, trying_to }

This lookup set overlaps with the normalized LUs for the following frames:

{ Attempt, Attempt_means, Operational_testing, Tasting, Trial, Try_defendant,
Trying_out, }

finally, these overlapping frames are provided as part of the prompt for the
frame classification task as shown below:

input: | ‘‘FRAME Attempt Attempt_means Operational_testing Tasting Trial
---|---
|  Try_defendant Trying_out: It was no use *trying the lift.’’
output: | ‘‘Attempt_means’’

Likewise, a frame classification task is generated for the trigger “lift" as
well, as shown below:

input: | ‘‘FRAME Body_movement Building_subparts Cause_motion Cause_to_end
---|---
|  Connecting_architecture Theft: It was no use trying the *lift.’’
output: | ‘‘Connecting_architecture’’

###  2.3 Argument extraction

After a frame is identified, the next task is to identify the frame elements and arguments for that frame in the text. An argument extraction task is generated for every frame classified. We include all available frame element names from FrameNet for the frame in question as part of the prompt input to make the argument extraction task easier for T5. The output is of the form ‘‘<element 1>="<arguments 1>" | <element 2>="<arguments 2>" | …’’. For instance, the arguments extraction task for the Attempt_means frame from above is shown below:

input: | ‘‘ARGS Attempt_means | Agent Means Goal Circumstances Degree Depictive Domain
---|---
|  Duration Frequency Manner Outcome Particular_iteration Place Purpose
|  Time: It was no use *trying the lift.’’
output: | ‘‘Means="the lift"’’

Likewise, the arguments extraction task for the Connecting_architecture frame
is shown below:

input: | ‘‘ARGS Connecting_architecture | Part Connected_locations Creator
---|---
|  Descriptor Direction Goal Material Orientation Source Whole:
|  It was no use trying the *lift.’’
output: | ‘‘Part="the lift"’’

###  2.4 Pretraining

The training data for FrameNet 1.7 is relatively small with under 6,000 fully
annotated sentences total, so it is common to leverage FrameNet exemplar data
as well to increase the amount of training data available. This exemplar data
includes around 100,000 sentences. Exemplar sentence annotates only a single
frame per sentence, so it is not suitable for generating trigger
identification tasks, but it is still a rich source of data to improve
performance on frame classification and argument extraction tasks. The
distribution of exemplar sentences is different from the distribution of
training data for FrameNet 1.7 (Kshirsagar et al., 2015), so rather than train
on exemplar data directly we instead use it for pretraining.

Another rich source of additional training data is PropBank (Kingsbury and
Palmer, 2002). PropBank is a similar frame parsing dataset to FrameNet,
although PropBank tends to focus more on verbs than FrameNet and has simpler
arguments. Still, the tasks are similar enough that pretraining on PropBank
can help the model score higher on FrameNet 1.7. Specifically, we use the
PropBank training data from OntoNotes (Weischedel et al., 2013) and the
English Web Treebank (Bies et al., 2012).

During training, we begin with a pretrained T5 model from Huggingface (Wolf et
al., 2020), and then go through two additional iterations of pretraining.
First we pretrain on PropBank data, and then on FrameNet 1.7 exemplars, before
finally training on the FrameNet 1.7 training set.

###  2.5 Data augmentation

The FrameNet 1.7 training data is well formatted and grammatically correct,
but in reality a lot of text that needs to be semantically parsed is not well
formatted and may have errors and typos. To help make our model more robust to
real world data, we also use data augmentation to expose the model to
misspellings, synonyms, and other differently formatted sentences.

The textual augmentations used are the following, leveraging the nlpaug Python
library (Ma, 2019):

  * •

Synonyms: swaps out a word for a synonym from WordNet (Miller, 1995).

  * •

Quotations: replaces latex-style quotes with standard double quotes and vice-
versa.

  * •

Random misspelling: replaces characters in words with different characters at
random.

  * •

Keyboard misspelling: replace characters with typos likely based on key
locations on keyboards.

  * •

Uppercase and lowercase: fully uppercase or lowercase the sentence.

  * •

Delete punctuation: randomly deletes punctuation characters in the sentence.

When augmenting text during training, we make sure to adjust the indices of
triggers and frame elements to match the new locations after the augmentation
is applied.

###  2.6 Task balancing

There is a mismatch between the 3 subtasks in terms of how much training data
each task has. Frame classification and argument extraction have multiple
examples per training sentence, since a task is generated for every frame in a
sentence. Furthermore, these tasks also benefit from pretraining with FrameNet
exemplar data. Trigger identification, however, has only 1 training example
per sentence, and cannot learn from exemplar data, so there is a large
mismatch between the amount of trigger identification samples available and
the amount of frame classification and argument extraction samples.

To help address this, we sample trigger identification tasks at a 3x higher
rate than frame classification and argument extraction tasks during training
to help ensure that the trigger identification performance does not trail
behind that of the other tasks per training epoch. We also increase the data
augmentation rate for trigger identification tasks to help increase the number
of training samples available.

##  3 Evaluation

FrameNet 1.7 does not include an official train / test / dev split, so we
follow the split and evaluation used by Open Sesame (Swayamdipta et al., 2017)
as this is the most popular open-source frame semantic parser on Github, and
was also the previous state-of-the-art. Other parsers also use the same split
for this reason (Kalyanpur et al., 2020, Zheng et al., 2022).

We calculate f1 score for each of the subtasks against the dev and test sets
from Open Sesame. For trigger identification, each trigger location that is
identified correctly is considered a true positive, each location that is
missed is a false negative, and each location that is incorrectly marked is a
false positive. For frame classification, an incorrectly classified frame is
considered both a false positive and a false negative. For the argument
extraction task, each frame element that is correctly identified and labeled
is a true positive. If a frame element is missed it is a false negative. If a
frame element is marked incorrectly it is a false positive. If a frame element
is labeled, but the element is incorrectly classified or the arguments are not
labeled entirely correctly, this is considered both a false positive and a
false negative.

Model |  FN 1.7 dev set |  FN 1.7 test set
---|---|---
| Trigger ID | Frame ID | Args ID | Trigger ID | Frame ID | Args ID
(Peng et al., 2018) |  - |  - |  - |  - |  0.891 |  -
(Zheng et al., 2022) |  - |  - |  - |  - |  - |  0.756
(Kalyanpur et al., 2020) |  - |  - |  0.77 |  - |  - |  0.76
Open Sesame (Swayamdipta et al., 2017) |  0.80 |  0.90 |  0.61 |  0.73 |  0.87 |  0.61
Frame Semantic Transformer (T5-small) |  0.75 |  0.87 |  0.76 |  0.71 |  0.86 |  0.73
Frame Semantic Transformer (T5-base) |  0.78 |  0.91 |  0.78 |  0.74 |  0.89 |  0.75
Figure 4: Evaluation results comparing Frame Semantic Transformer with other
frame semantic parsers on FrameNet 1.7, where comparable data can be found.
The top section of the table contains frame semantic parsers without available
pretrained models, while the bottom section contains open-source parsers with
pretrained models. Bold indicates the best performance in each group.

We also include an ablation study showing the effects of pretraining and data
augmentation on model performance in Figure 5. Data augmentation actually
slighltly hurts performance in argument extraction, but we still think it is
worth it to give the model more robustness to messy examples that may appear
in real-world data. Lack of pretraining appears to slightly harm performance
on all tasks. We did not do a statistical significance test on this data.

Model |  Trigger ID |  Frame ID |  Args ID
---|---|---|---
Frame semantic transformer (T5-base) |  0.74 |  0.89 |  0.75
No data augmentation |  0.74 |  0.89 |  0.76
No pretrain |  0.72 |  0.88 |  0.74
Figure 5: Ablation study comparing performance without data augmentation and
without pretraining on the FrameNet 1.7 test set.

##  4 Related work

Recent work on frame semantic parsing has focused on incorporating more
information from FrameNet into the parsing process. (Zheng et al., 2022)
encode the frame relation graph into an embedding space during inference to
improve performance. (Su et al., 2021) encodes the full text of frame and
element descriptions to aid in classification. Our work also follows in this
vein by using lexical unit data to provide hints to our model during frame
classification. However, both (Zheng et al., 2022) and (Su et al., 2021) do
not provide pretrained, open-source models.

Most similar to our work, and largely a point of inspiration, is (Kalyanpur et
al., 2020). In this work, a T5 model (Raffel et al., 2020) is fine-tuned on
the frame classification and argument extraction tasks. In a variant of their
work, the T5 decoder is replaced with a classification head for frame
classification. However, this work does not deal with trigger identification,
and does not use lexical unit hints during frame classification. Furthermore,
no code or models are open-sourced as part of this work, making it difficult
for end-users to easily make use of the model.

Previous open-source frame parsers include Open Sesame (Swayamdipta et al.,
2017) and SEMAFOR (Das et al., 2010). However, both of these projects predate
the rise of the transformer architecture, and their performance lags behind
transformer-based solutions, especially in argument extraction.

##  5 Conclusion

Frame Semantic Transformer approaches or matches state-of-the-art performance
on frame semantic parsing tasks while also being easy to use as an end-user.
We improve performance by pretraining both on FrameNet exemplars and PropBank
data. We also incorporate frame knowledge from FrameNet via lexical units and
available frame elements and pass that knowledge to T5 in-context as part of
task prompts. In addition, we add NLP data augmentations to help the model
generalize to real-world data which will likely be formatted differently than
the FrameNet 1.7 training set. At present, Frame Semantic Transformer only
provides pretrained models for English FrameNet, but we hope to support other
languages and PropBank in the future as well.

## References

  * (1) Python package index - pypi.  URL https://pypi.org/.
  * Baker et al. (1998) C. F. Baker, C. J. Fillmore, and J. B. Lowe.  The berkeley framenet project.  In _COLING 1998 Volume 1: The 17th International Conference on Computational Linguistics_ , 1998.
  * Bies et al. (2012) A. Bies, J. Mott, C. Warner, and S. Kulick.  English web treebank dataset ldc2012t13.  _Linguistic Data Consortium, Philadelphia, PA_ , 2012.
  * Bird et al. (2009) S. Bird, E. Klein, and E. Loper.  _Natural language processing with Python: analyzing text with the natural language toolkit_.  "O’Reilly Media, Inc.", 2009.
  * Chen et al. (2019) Q. Chen, Z. Zhuo, and W. Wang.  Bert for joint intent classification and slot filling.  _arXiv preprint arXiv:1902.10909_ , 2019.
  * Chen et al. (2013) Y.-N. Chen, W. Y. Wang, and A. I. Rudnicky.  Unsupervised induction and filling of semantic slots for spoken dialogue systems using frame-semantic parsing.  In _2013 IEEE Workshop on Automatic Speech Recognition and Understanding_ , pages 120–125. IEEE, 2013.
  * Das et al. (2010) D. Das, N. Schneider, D. Chen, and N. A. Smith.  Probabilistic frame-semantic parsing.  In _Human language technologies: The 2010 annual conference of the North American chapter of the association for computational linguistics_ , pages 948–956, 2010.
  * Gildea and Jurafsky (2002) D. Gildea and D. Jurafsky.  Automatic labeling of semantic roles.  _Computational linguistics_ , 28(3):245–288, 2002\.
  * Kalyanpur et al. (2020) A. Kalyanpur, O. Biran, T. Breloff, J. Chu-Carroll, A. Diertani, O. Rambow, and M. Sammons.  Open-domain frame semantic parsing using transformers.  _arXiv preprint arXiv:2010.10998_ , 2020.
  * Kingsbury and Palmer (2002) P. R. Kingsbury and M. Palmer.  From treebank to propbank.  In _LREC_ , pages 1989–1993, 2002.
  * Kshirsagar et al. (2015) M. Kshirsagar, S. Thomson, N. Schneider, J. G. Carbonell, N. A. Smith, and C. Dyer.  Frame-semantic role labeling with heterogeneous annotations.  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 218–224, 2015.
  * Ma (2019) E. Ma.  Nlp augmentation.  https://github.com/makcedward/nlpaug, 2019.
  * Miller (1995) G. A. Miller.  Wordnet: a lexical database for english.  _Communications of the ACM_ , 38(11):39–41, 1995\.
  * Paice (1990) C. D. Paice.  Another stemmer.  In _ACM Sigir Forum_ , volume 24, pages 56–61. ACM New York, NY, USA, 1990.
  * Peng et al. (2018) H. Peng, S. Thomson, S. Swayamdipta, and N. A. Smith.  Learning joint semantic parsers from disjoint data.  _arXiv preprint arXiv:1804.05990_ , 2018.
  * Porter (1980) M. F. Porter.  An algorithm for suffix stripping.  _Program_ , 14(3):130–137, 1980.
  * Raffel et al. (2020) C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ , 21(1):5485–5551, 2020.
  * Su et al. (2021) X. Su, R. Li, X. Li, J. Z. Pan, H. Zhang, Q. Chai, and X. Han.  A knowledge-guided framework for frame identification.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 5230–5240, 2021.
  * Swayamdipta et al. (2017) S. Swayamdipta, S. Thomson, C. Dyer, and N. A. Smith.  Frame-semantic parsing with softmax-margin segmental rnns and a syntactic scaffold.  _arXiv preprint arXiv:1706.09528_ , 2017.
  * Weischedel et al. (2013) R. Weischedel, M. Palmer, M. Marcus, E. Hovy, S. Pradhan, L. Ramshaw, N. Xue, A. Taylor, J. Kaufman, M. Franchini, et al.  Ontonotes release 5.0 ldc2013t19.  _Linguistic Data Consortium, Philadelphia, PA_ , 23, 2013.
  * Wolf et al. (2020) T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, and A. M. Rush.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online, Oct. 2020. Association for Computational Linguistics.  URL https://www.aclweb.org/anthology/2020.emnlp-demos.6.
  * Zhao et al. (2023) X. Zhao, X. Walton, S. Shrestha, and A. Rios.  Bike frames: Understanding the implicit portrayal of cyclists in the news.  _arXiv preprint arXiv:2301.06178_ , 2023.
  * Zheng et al. (2022) C. Zheng, X. Chen, R. Xu, and B. Chang.  A double-graph based framework for frame semantic parsing.  _arXiv preprint arXiv:2206.09158_ , 2022.