# Does canonical quantization lead to GKSL dynamics?

T. Koide  tomoikoide<EMAIL_ADDRESS>Instituto de Física,
Universidade Federal do Rio de Janeiro, C.P. 68528, 21941-972, Rio de Janeiro,
RJ, Brazil  F. Nicacio<EMAIL_ADDRESS>Instituto de Física, Universidade
Federal do Rio de Janeiro, C.P. 68528, 21941-972, Rio de Janeiro, RJ, Brazil

###### Abstract

We introduce a generalized classical model of Brownian motion for describing
thermal relaxation processes which is thermodynamically consistent. Applying
the canonical quantization to this model, a quantum equation for the density
operator is obtained. This equation has a thermal equilibrium state as its
stationary solution, but the time evolution is not necessarily a Completely
Positive and Trace-Preserving (CPTP) map. In the application to the harmonic
oscillator potential, however, the requirement of the CPTP map is shown to be
satisfied by choosing parameters appropriately and then our equation
reproduces a Gorini-Kossakowski-Sudarshan-Lindblad (GKSL) equation satisfying
the detailed balance condition. This result suggests a quantum-classical
correspondence in thermal relaxation processes and will provide a new insight
to the study of decoherence.

##  I Introduction

The accelerating development of experimental methods enables us to access
individual thermal random processes in a small fluctuating system. Standard
thermodynamics is not directly applicable to this system because the
macroscopic and microscopic time scales are not necessarily clearly separated
and the effect of fluctuations is important. There is no established method
for describing such systems, but it is known that Brownian motion can be used
to describe them consistently with thermodynamics. Such a framework is called
stochastic energetics (stochastic thermodynamics) [1, 2, 3].

It should be understood that classical stochastic models do not consider
quantum fluctuation and hence are not applicable to extremely microscopic
systems. Regarding this limitation, there are various studies to derive
quantum dissipative equations from either the Heisenberg equations of motion
or the Liouville-von Neumann equations, introducing systematic coarse-graining
with, for example, projection operator techniques [4, 5]. However, the derived
equations often do not satisfy the (complete) positivity of probability and
its conservation (preservation of trace of a density operator) at the same
time. The linear time evolutions satisfying these requisites are called
Completely Positive and Trace-Preserving (CPTP) maps [5]. Such a map, when the
Markovianity is assumed, leads to the well-known Gorini-Kossakowski-Sudarshan-
Lindblad (GKSL) equation, which is used to describe quantum thermal relaxation
processes [5]. Indeed, by applying this to the harmonic potential and imposing
the detailed balance conditions to the parameters, the behavior of the GKSL
equation are compatible with the laws of thermodynamics [6, 7, 8, 9, 10].

In standard applications, quantum dynamics is obtained from the corresponding
classical equation through the canonical quantization [11]. Therefore, it is
natural to ask whether a GKSL equation can be derived from Brownian motion
provided the canonical quantization recipe. This question is investigated by
Oliveira in Refs. [12, 13], where he considers the Kramers (Fokker-Planck)
equation for a classical particle interacting with a heat bath. Regarding it
as the classical correspondence of the density operator equation (master
equation) in quantum physics, his quantum dissipative equation is obtained by
the canonical quantization, which however does not reproduce a GKSL equation.
Prior to Oliveira, Burzlaff studies conditions for the Liouville equation to
be mapped into a classical counterpart of the GKSL equation [14]. Such a
mapping is not always found and this approach is not applicable to thermal
relaxation processes which are not described by the Liouville equation. The
applicability of the canonical quantization is thus not known in the
derivation of the GKSL equation.

In this paper, we show that the GKSL equation is obtained from a generalized
Kramers equation, at least, for the harmonic oscillator Hamiltonian. There are
two differences comparing to the approach by Oliveira. First, the interaction
with heat baths affects not only the equation of momentum but also that of
position of Brownian motion. Although this seems to be fanciful in standard
stochastic processes, we should recall that position and momentum are not
distinguished by the symplectic structure of the phase space. Moreover, in
quantum physics, it is known that the equation of position is modified by
random noises induced by quantum fluctuation in the derivation of the
Schrödinger equation based on Brownian motion which is studied by Nelson and
Yasue [15, 16, 17, 18], for instance. Indeed, we show that this generalized
model of Brownian motion is consistent with the thermodynamical laws. Second,
supported by the non-uniqueness of the symmetrization among products of
position and momentum operators in the quantization procedure, we adopt a
different symmetrization procedure, while preserving the standard mapping
between the commutator and the Poisson bracket of canonically conjugated
variables of position and momentum. This symmetrization is inspired by the
relation found in the Fokker-Planck operator considered in Refs. [19, 20, 21,
22].

This paper is organized as follows. The generalized model of Brownian motion
is developed in Sec. II. The canonical quantization of this classical model is
studied in Sec. III. When we apply this quantum model to the harmonic
oscillator potential, the GKSL equation is reproduced. Section IV is devoted
to the concluding remarks.

##  II Generalized thermal relaxation model

Let us consider a $D+1$ dimensional system composed of $N$-particles. In the
absence of the interaction with environment, the Hamiltonian of the closed
system is

$\displaystyle H(\\{{\bf q}_{t},{\bf p}_{t}\\},\vec{\lambda}_{t})\,,$ (1)

where $\\{{\bf q}_{t},{\bf p}_{t}\\}=({\bf q}_{(1)t},{\bf
p}_{(1)t}),\cdots,({\bf q}_{(N)t},{\bf p}_{(N)t})$. The Hamiltonian depends on
a set of external parameters
$\vec{\lambda_{t}}=(\lambda^{a}_{t},\lambda^{b}_{t},\cdots)$ through of which
we control, for example, the form of an external confinement potential of the
$N$-particles. The Hamiltonian is arbitrary: a relativistic thermal relaxation
observed in the rest frame of heat baths is described by choosing a
relativistic Hamiltonian and the result is consistent with relativistic
stochastic energetics [23, 19, 24].

We introduce $N$-independent heat baths of temperature $T_{i}$ and consider
that the $i$-th particle interacts only with the heat bath of $T_{i}$. Suppose
that the dynamics of this system is described by the following stochastic
differential equations for $i=1,\dots,N$,

$\displaystyle d\widetilde{\bf q}_{(i)t}$ $\displaystyle=$
$\displaystyle\left(\frac{\partial H(\\{\widetilde{\bf q}_{t},\widetilde{\bf
p}_{t}\\},\vec{\lambda}_{t})}{\partial\widetilde{\bf
p}_{(i)t}}-\gamma_{q_{i}}\frac{\partial H(\\{\widetilde{\bf
q}_{t},\widetilde{\bf p}_{t}\\},\vec{\lambda}_{t})}{\partial\widetilde{\bf
q}_{(i)t}}\right)dt+\sqrt{\frac{2\gamma_{q_{i}}}{\beta_{i}}}d\widetilde{\bf
B}_{q(i)t}\,,$ (2) $\displaystyle d\widetilde{\bf p}_{(i)t}$ $\displaystyle=$
$\displaystyle\left(-\frac{\partial H(\\{\widetilde{\bf q}_{t},\widetilde{\bf
p}_{t}\\},\vec{\lambda}_{t})}{\partial\widetilde{\bf
q}_{(i)t}}-\gamma_{p_{i}}\frac{\partial H(\\{\widetilde{\bf
q}_{t},\widetilde{\bf p}_{t}\\},\vec{\lambda}_{t})}{\partial\widetilde{\bf
p}_{(i)t}}\right)dt+\sqrt{\frac{2\gamma_{p_{i}}}{\beta_{i}}}d\widetilde{\bf
B}_{p(i)t}\,,$ (3)

where $dt>0$, $\beta_{i}=1/(k_{B}T_{i})$ for the Boltzmann constant $k_{B}$,
and $\gamma_{q_{i}}$ and $\gamma_{p_{i}}$ are real constants. The symbol
“$\,\widetilde{\bullet}\,$” is conveniently introduced to denote stochastic
variables. Unlike the standard Brownian motion [1], we consider the
interaction of heat baths with the stochastic positions $d\widetilde{\bf
q}_{(i)t}$. That is, the second and third terms on the right-hand side of Eq.
(2) do not exist in the standard description which is recovered by setting
$\gamma_{q(i)}=0$. The Wiener processes satisfy the following correlation
properties

$\displaystyle\mathrm{E}\left[d\tilde{B}_{q(i)t}\right]=\mathrm{E}\left[d\tilde{B}_{p(i)t}\right]=0\,,$
(4)
$\displaystyle\mathrm{E}\left[d\tilde{B}_{q(i)t}d\tilde{B}_{q(j)t^{\prime}}\right]=\mathrm{E}\left[d\tilde{B}_{p(i)t}d\tilde{B}_{p(j)t^{\prime}}\right]=dt\,\delta_{ij}\delta_{t,t^{\prime}}\,,$
(5)
$\displaystyle\mathrm{E}\left[d\tilde{B}_{q(i)t}d\tilde{B}_{p(j)t^{\prime}}\right]=0\,,$
(6)

where $\mathrm{E}[\,\bullet\,]$ denotes the ensemble average for the Wiener
process. It should be emphasized that $({\bf q}_{(i)t},{\bf p}_{(i)t})$ form
canonical pairs for $H(\\{{\bf q}_{t},{\bf p}_{t}\\},\vec{\lambda}_{t})$, but
are not necessarily canonical variables under the influence of the heat baths,
because we do not define the Lagrangian for such a system.

Introducing a normalized phase space distribution $f(\\{{\bf q},{\bf
p}\\},t)$, its temporal evolution is obtained by using Ito’s lemma [25] and
the stochastic differential equations (2) and (3):

$\displaystyle\partial_{t}f(\\{{\bf q},{\bf p}\\},t)$ $\displaystyle=$
$\displaystyle-\\{f(\\{{\bf q},{\bf p}\\},t),H\\}_{\text{PB}}$ (7)
$\displaystyle+\sum_{i=1}^{N}\sum_{\alpha=1}^{D}\frac{\gamma_{p_{i}}}{\beta_{i}}\\{e^{-\beta_{i}H}\\{e^{\beta_{i}H}f(\\{{\bf
q},{\bf
p}\\},t),\,q^{\alpha}_{(i)}\\}_{\text{PB}},\,q^{\alpha}_{(i)}\\}_{\text{PB}}$
$\displaystyle+\sum_{i=1}^{N}\sum_{\alpha=1}^{D}\frac{\gamma_{q_{i}}}{\beta_{i}}\\{e^{-\beta_{i}H}\\{e^{\beta_{i}H}f(\\{{\bf
q},{\bf
p}\\},t),\,p^{\alpha}_{(i)}\\}_{\text{PB}},\,p^{\alpha}_{(i)}\\}_{\text{PB}}\,,$

where, for arbitrary functions $g$ and $h$, the Poisson bracket is defined by

$\displaystyle\\{g,h\\}_{\text{PB}}=\sum_{i=1}^{N}\sum_{\alpha=1}^{D}\left(\frac{\partial
g}{\partial q^{\alpha}_{(i)}}\frac{\partial h}{\partial
p^{\alpha}_{(i)}}-\frac{\partial g}{\partial p^{\alpha}_{(i)}}\frac{\partial
h}{\partial q^{\alpha}_{(i)}}\right)\,.$ (8)

The bath-position couplings considered in the stochastic equation (2) appears
in the third term on the right hand side of Eq. (7), thus this is called here
the generalized Kramers equation. Like the standard version, this
generalization also drives any initial distribution asymptotically towards the
thermal equilibrium state, as shown in Appendix A.

To unveil the thermodynamical structure, we define the absorbed heat from the
heat baths as the works done by these baths [1],

$\displaystyle d\widetilde{Q}^{c}_{t}=\sum_{i=1}^{N}d\tilde{Q}^{c}_{(i)t}\,,$
(9)

where

$\displaystyle d\tilde{Q}^{c}_{(i)t}$ $\displaystyle=$
$\displaystyle\sum_{\alpha=1}^{D}\left(-\gamma_{p_{i}}(t)\frac{\partial
H(\\{\widetilde{\bf q}_{t},\widetilde{\bf
p}_{t}\\},\vec{\lambda}_{t})}{\partial\widetilde{p}^{\alpha}_{(i)t}}dt+\sqrt{\frac{2\gamma_{p_{i}}(t)}{\beta_{i}}}d\widetilde{B}^{\alpha}_{p(i)t}\right)\circ
d\widetilde{q}^{\alpha}_{(i)t}$ (10)
$\displaystyle-\sum_{\alpha=1}^{D}\left(-\gamma_{q_{i}}(t)\frac{\partial
H(\\{\widetilde{\bf q}_{t},\widetilde{\bf
p}_{t}\\},\vec{\lambda}_{t})}{\partial\widetilde{q}^{\alpha}_{(i)t}}dt+\sqrt{\frac{2\gamma_{q_{i}}(t)}{\beta_{i}}}d\widehat{B}^{\alpha}_{q(i)t}\right)\circ
d\widehat{p}^{\alpha}_{(i)t}\,.$

Here the symbol “$\circ$” indicates the Stratonovich definition for the
product of stochastic quantities [25]. The second term on the right-hand side
does not exist in stochastic energetics and is induced in our description by
interactions with heat baths in Eq. (2). See Appendix B for more details.

The expectation value of the Hamiltonian defines the energy of the system
through

$\displaystyle E^{c}_{t}$ $\displaystyle=$ $\displaystyle\int
d\Gamma_{0}f_{0}(\\{{\bf q}_{0},{\bf p}_{0}\\}){\rm E}[H(\\{\widetilde{\bf
q}_{t},\widetilde{\bf p}_{t}\\},\vec{\lambda}_{t})]\,,$ (11)

where $\\{{\bf q}_{0},{\bf p}_{0}\\}$ are the position and momentum at an
initial time, $d\Gamma_{0}$ is the corresponding phase space volume, and
$f_{0}(\\{{\bf q}_{0},{\bf p}_{0}\\})$ is the initial probability
distribution. The change of energy induced by the control parameters
$\vec{\lambda}_{t}$ is interpreted as the work done to the system,

$\displaystyle dW^{c}_{t}$ $\displaystyle=$ $\displaystyle\int
d\Gamma_{0}\,f_{0}(\\{{\bf q}_{0},{\bf p}_{0}\\}){\rm
E}\\!\left[\sum_{j}d\lambda^{j}_{t}\frac{\partial H(\\{\widetilde{\bf
q}_{t},\widetilde{\bf
p}_{t}\\},\vec{\lambda}_{t})}{\partial\lambda^{j}_{t}}\right]\,,$ (12)

while, from Eq.(9),

$\displaystyle dQ^{c}_{t}$ $\displaystyle=$ $\displaystyle\int
d\Gamma_{0}f_{0}(\\{{\bf q}_{0},{\bf p}_{0}\\}){\rm
E}[d\widetilde{Q}^{c}_{t}]=\int d\Gamma\frac{\partial f(\\{{\bf q},{\bf
p}\\},t)}{\partial t}H(\\{{\bf q},{\bf p}\\},\vec{\lambda}_{t})\,,$ (13)

represents the mean absorbed heat. Here $d\Gamma$ denotes the phase space
volume element for $\\{{\bf q},{\bf p}\\}$. These definitions lead to an
equation analogous to the first law of thermodynamics,

$\displaystyle E^{c}_{t+dt}-E^{c}_{t}=dQ^{c}_{t}+dW^{c}_{t}\,.$ (14)

Moreover, the Shannon information entropy [26]

$\displaystyle S^{c}(t)=-k_{B}\int d\Gamma\,f(\\{{\bf q},{\bf p}\\},t)\ln
f(\\{{\bf q},{\bf p}\\},t)$ (15)

enables us to deduce the following inequality:

$\displaystyle\frac{dS^{c}(t)}{dt}-\sum_{i=1}^{N}k_{B}\beta_{i}\frac{dQ^{c}_{(i)t}}{dt}$
(16) $\displaystyle=$ $\displaystyle\sum_{i=1}^{N}k_{B}\int
d\Gamma\frac{\beta_{i}}{f(\\{{\bf q},{\bf
p}\\},t)}\left[\gamma_{p_{i}}\sum_{\alpha=1}^{D}\left(\frac{1}{\beta_{i}}\frac{\partial
f(\\{{\bf q},{\bf p}\\},t)}{\partial p^{\alpha}_{i}}+\frac{\partial H(\\{{\bf
q}_{t},{\bf p}_{t}\\},\vec{\lambda}_{t})}{\partial p^{\alpha}_{i}}f(\\{{\bf
q},{\bf p}\\},t)\right)^{2}\right.$
$\displaystyle\left.+\gamma_{q_{i}}\sum_{\alpha=1}^{D}\left(\frac{1}{\beta_{i}}\frac{\partial
f(\\{{\bf q},{\bf p}\\},t)}{\partial q^{\alpha}_{i}}+\frac{\partial H(\\{{\bf
q}_{t},{\bf p}_{t}\\},\vec{\lambda}_{t})}{\partial q^{\alpha}_{i}}f(\\{{\bf
q},{\bf p}\\},t)\right)^{2}\right]\geq 0\,.$

This is analogous to the Clausius inequality in a thermodynamical cycle if
$\gamma_{q(i)}$ and $\gamma_{p(i)}$ are taken as time-dependent functions such
that the system interacts only with heat baths with the same temperature at a
time. When all temperatures of the heat baths are the same,
$\beta_{1}=\cdots=\beta_{N}=\beta$, the above inequality reduces to a
continuous time-version of the second law:

$\displaystyle\frac{dS^{c}(t)}{dt}-k_{B}\beta\frac{dQ^{c}_{t}}{dt}\geq 0\,.$
(17)

##  III Canonical quantization

Following the standard canonical quantization procedure [11], we consider the
self-adjoint operators associated with the classical variables $({\bf
q}_{(i)},{\bf p}_{(i)})$ satisfying

$\displaystyle[\hat{q}^{\alpha}_{(i)},\hat{p}^{\beta}_{(j)}]$ $\displaystyle=$
$\displaystyle\mathsf{i}\hbar\delta_{\alpha\beta}\delta_{ij}\,,$ (18)
$\displaystyle{[\hat{q}^{\alpha}_{(i)},\hat{q}^{\beta}_{(j)}]}$
$\displaystyle=$ $\displaystyle 0\,,$ (19)
$\displaystyle{[\hat{p}^{\alpha}_{(i)},\hat{p}^{\beta}_{(j)}]}$
$\displaystyle=$ $\displaystyle 0\,,$ (20)

where “$\,\hat{\bullet}\,$”is introduced to denote operators. We further admit
the following quantization rules

$\displaystyle f(\\{{\bf q},{\bf p}\\},t)$ $\displaystyle\longrightarrow$
$\displaystyle\hat{\rho}(t)\,,$ (21)
$\displaystyle\left\\{g,h\right\\}_{\text{PB}}$ $\displaystyle\longrightarrow$
$\displaystyle-\frac{\mathsf{i}}{\hbar}[\hat{g},\hat{h}]\,,$ (22)
$\displaystyle e^{\pm\beta_{i}H}g$ $\displaystyle\longrightarrow$
$\displaystyle e^{\pm\beta_{i}\hat{H}/2}\hat{g}e^{\pm\beta_{i}\hat{H}/2}\,,$
(23)

where $\hat{\rho}(t)$ is the density operator describing the system state. The
last rule is a key assumption in our approach, which is inspired by the
relation satisfied between the time generators in Hamiltonian and stochastic
dynamics considered in Refs. [19]. See also the discussion in Refs. [20, 21].
A similar relation is, further, associated with detailed balance conditions in
Ref. [22].

Applying the above quantization rules to the generalized Kramers equation (7),
a master equation for the density operator is obtained,

$\displaystyle\frac{d}{dt}\hat{\rho}(t)$ $\displaystyle=$
$\displaystyle\frac{\mathsf{i}}{\hbar}\ [\hat{\rho}(t),\hat{H}]$ (24)
$\displaystyle-\sum_{i=1}^{N}\sum_{\alpha=1}^{D}\frac{\gamma_{p_{i}}}{\beta_{i}\hbar^{2}}\left[e^{-\beta_{i}\hat{H}/2}\left[e^{\beta_{i}\hat{H}/2}\hat{\rho}(t)e^{\beta_{i}\hat{H}/2},\hat{q}^{\alpha}_{(i)}\right]e^{-\beta_{i}\hat{H}/2},\hat{q}^{\alpha}_{(i)}\right]$
$\displaystyle-\sum_{i=1}^{N}\sum_{\alpha=1}^{D}\frac{\gamma_{q_{i}}}{\beta_{i}\hbar^{2}}\left[e^{-\beta_{i}\hat{H}/2}\left[e^{\beta_{i}\hat{H}/2}\hat{\rho}(t)e^{\beta_{i}\hat{H}/2},\hat{p}^{\alpha}_{(i)}\right]e^{-\beta\hat{H}/2},\hat{p}^{\alpha}_{(i)}\right]\,,$

where $\hat{H}$ is a Hamiltonian operator. When all temperatures are the same,
it is easily seen that the stationary solution is given by the thermal
equilibrium state, $de^{-\beta\hat{H}}/dt=0$. The above equation can be mapped
into the generalized form of the quantum Fokker-Planck equation considered in
Ref. [27].

The evolution described by the master equation, however, does not necessarily
satisfy the requirement of the CPTP map and thus we wonder under which
circumstances this will happen. In the following, we will consider a set of
harmonic oscillators interacting with thermal baths in Eq. (24) and show that
this evolution will be reduced to the GKSL structure, satisfying automatically
the detailed balance condition.

Writing the Hamiltonian of the system as

$\displaystyle\hat{H}=\sum_{i=1}^{N}\hat{H}_{(i)}=\sum_{i=1}^{N}\sum_{\alpha=1}^{D}\frac{1}{2m_{i}}\hat{p}^{\alpha\,2}_{(i)}+\frac{m_{i}\omega_{i}^{2}}{2}\hat{q}^{\alpha\,2}_{(i)}\,,$
(25)

where $m_{i}$ and $\omega_{i}$ are masses and angular frequencies,
respectively. In general, $\omega_{i}$ is regarded as external parameters
which control the width of the harmonic oscillator potentials and thus is
time-dependent, as is considered in Refs. [20, 21]. This time dependence
however does not affect the following discussions and will be omitted for
simplicity. In this case, the master equation is reexpressed as a time-
convolutionless form [28]

$\displaystyle\frac{d}{dt}\hat{\rho}(t)=\frac{\mathsf{i}}{\hbar}\
[\hat{\rho}(t),\hat{H}^{\prime}]+\sum_{i=1}^{N}\sum_{\alpha=1}^{D}{\mathcal{L}}_{(i)}^{\alpha}[\hat{\rho}(t)],$
(26)

where the effective Hamiltonian is

$\displaystyle\hat{H}^{\prime}$ $\displaystyle=$
$\displaystyle\hat{H}-\sum_{i=1}^{N}\sum_{\alpha=1}^{D}\frac{\delta_{i}\gamma_{p_{i}}}{2\beta_{i}\hbar
m_{i}\omega_{i}}\sinh(\beta_{i}\hbar\omega_{i}/2)[\hat{q}^{\alpha}_{(i)},\hat{p}^{\alpha}_{(i)}]_{+}\,,$
(27) $\displaystyle\delta_{i}$ $\displaystyle=$
$\displaystyle\frac{\gamma_{q_{i}}}{\gamma_{p_{i}}}(m_{i}\omega_{i})^{2}-1\,,$
(28)

and the dissipative term is

$\displaystyle{\mathcal{L}}_{(i)}^{\alpha}[\hat{\rho}(t)]$ $\displaystyle=$
$\displaystyle-\frac{1}{2\hbar}\sum_{\mu,\nu=1}^{4}g_{\mu\nu}\left[[\hat{L}^{\alpha\,\dagger}_{(i),\mu}\hat{L}^{\alpha}_{(i),\nu},\hat{\rho}(t)]_{+}-2\hat{L}^{\alpha}_{(i),\mu}\hat{\rho}(t)\hat{L}^{\alpha\,\dagger}_{(i),\nu}\right]\,,$
(29) $\displaystyle g_{\mu\nu}$ $\displaystyle=$ $\displaystyle{\rm
Diag}(1,1,1,-1)\,.$ (30)

The Lindblad operators are defined by

$\displaystyle\hat{L}^{\alpha}_{(i),1}=\Gamma_{(i),1}\,\hat{a}_{(i)}^{\alpha}\,,\,\,\,\,\,\,\,\hat{L}^{\alpha}_{(i),2}=\Gamma_{(i),2}\,\hat{a}_{(i)}^{\alpha\,{\dagger}}\,,$
(31)
$\displaystyle\hat{L}^{\alpha}_{(i),3}=\sqrt{\delta_{i}}\,\hat{q}^{\alpha}_{(i)}\,,\,\,\,\,\,\,\,\hat{L}^{\alpha}_{(i),4}=\frac{\sqrt{\delta_{i}}}{m_{i}\omega_{i}}\,\hat{p}^{\alpha}_{(i)}\,,$
(32)

with

$\displaystyle\Gamma_{(i),j}=\sqrt{\frac{(\delta_{i}+2)\gamma_{p_{i}}}{2\beta_{i}m_{i}\omega_{i}}\exp\left(\frac{(-1)^{j+1}\beta_{i}\hbar\omega_{i}}{2}\right)}\,,$
(33)

and the lowering operator
$\hat{a}_{(i)}^{\alpha}:=[\hat{q}^{\alpha}_{(i)}+i\hat{p}^{\alpha}_{(i)}/(m_{i}\omega_{i})]\sqrt{m_{i}\omega_{i}/2\hbar}$.

Due to the negative matrix element $g_{44}=-1$, the master equation is reduced
to the GKSL equation by choosing $\delta_{i}=0,\forall i$, which implies
$\hat{L}^{\alpha}_{(i),3}=\hat{L}^{\alpha}_{(i),4}=0,\forall i$ and
$\hat{H}^{\prime}=\hat{H}$. It should be emphasized that the ratio of the
dissipative coefficients satisfies the detailed balance condition:
$\Gamma_{(i),1}/\Gamma_{(i),2}=e^{\beta_{i}\hbar\omega_{i}/2}$. Normally this
condition is introduced by hand to describe thermal relaxation processes, but
is induced automatically in the present approach.

As is well-known, in this case, the inequality analogous to the second law of
thermodynamics can be introduced [6, 7, 8, 9, 10]. The von Neumann entropy,
$S^{q}(t)=-k_{B}{\rm Tr}\left[\hat{\rho}\ln\hat{\rho}\right]$, satisfies

$\displaystyle\frac{dS^{q}(t)}{dt}-\sum_{i=1}^{N}k_{B}\beta_{i}\frac{dQ^{q(i)}_{t}}{dt}\geq
0\,,$ (34)

where the heat-currents associated with the bath of temperature $\beta_{i}$
are defined by

$\displaystyle\frac{dQ^{q(i)}_{t}}{dt}={\rm
Tr}\left[\frac{d\hat{\rho}(t)}{dt}\hat{H}_{(i)}\right]\,.$ (35)

Comparing this with Eq. (13), one can easily see that the representations of
classical and quantum heats have a clear correspondence. The introduction of
the first law is straightforward by the identification $E^{q}_{t}={\rm
Tr}[\hat{\rho}(t)\hat{H}]$.

It is noteworthy that, in the limit of vanishing $\gamma_{q_{i}}$, or
$\delta_{i}=-1$, the generalized Kramers equation (7) is reduced to its
standard version, as explained bellow Eq. (3). In this situation, it is easy
to see that $\hat{L}^{\alpha}_{(i),4}$ does not disappear and the master
equation is still not a GKSL equation.

The employed definitions of thermodynamical quantities are still applicable to
the generic time-convolutionless mater equation (26), but the positivity of
the entropy production expressed in Eq. (34) is not necessarily satisfied.
Indeed this positivity is associated with Markovian properties of the master
equation [29]. We will return to this point in the concluding remarks.

A remarkable consequence of the algebraic structure of the mater equation (24)
is that our formulation is applicable not only to Bosons but also to Fermions.
The fermionic excitation of an harmonic oscillator is described by the anti-
commutation relations $[\widehat{p},\widehat{q}]_{+}=0$,
$[\hat{p},\hat{p}]_{+}/(m\omega)=m\omega[\hat{q},\hat{q}]_{+}=\hbar/2$ and the
Hamiltonian operator $\widehat{H}=\mathsf{i}\omega\widehat{q}\widehat{p}$. The
raising and lowering operators are defined in the same fashion as the bosonic
harmonic oscillator and then satisfy fermionic anti-commutation relations.
Applying these to the master equation (24) and choosing parameters
appropriately, the GKSL equation for fermions is obtained. The heat and the
entropy of this fermionic system are shown to satisfy the inequality analogous
the second law again.

##  IV concluding remarks

We introduced a generalized classical model for describing thermal relaxation
processes by considering the interaction with heat baths even through the
equation of velocity. In this model, we can still define laws analogous to the
thermodynamical ones. Applying the canonical quantization to this, a quantum
master equation for the density operator is obtained. This equation has a
thermal equilibrium state as the stationary solution, but the time evolution
is not necessarily the CPTP map. In the application to the harmonic oscillator
potential, however, the requirement for the CPTP map is shown to be satisfied
by choosing parameters appropriately and then our equation reproduces the GKSL
equation with the detailed balance condition.

L. Onsager assumed that the average regression of thermal fluctuations behaves
like the corresponding macroscopic irreversible process [30]. This hypothesis
is, however, considered not to be applicable to quantum systems [31, 32, 33,
34, 35]. It is worth investigating this problem using the generalized
classical model proposed here.

There is a complementary relation between our quantum master equation and the
GKSL equation. In the GKSL equation, the time evolution is always a CPTP map
but the stationary solution is not necessarily given by the thermal
equilibrium state. This is satisfied by introducing the detailed balance
condition [36]. Contrary to this, our equation always has the equilibrium
state as the stationary solution but the evolution is not necessarily given by
a CPTP map. The compatibility of our master equation with the CPTP map for
general Hamiltonians has not yet been known. The time evolution of the GKSL
equation is a CPTP map, but not vice versa. Indeed there are non-Markovian
master equations which satisfy the requirement of the CPTP map [28]. Thus the
applicability of our strategy to more general cases is an open question.

The interaction with environments (heat baths) plays an important role in
understanding the quantum-classical transition in terms of decoherence [37].
That is, such an interaction has been considered crucial to distinguish
classical and quantum worlds. Meanwhile, our result suggests that the
interactions with environment in classical and quantum worlds have a
correspondence, which is a new perspective and will provide a new insight to
the study of the measurement theory.

The authors acknowledge P. Talkner for valuable comment. T. K. acknowledges
the financial supports by CNPq (No. 305654/2021-7). A part of this work has
been done under the project INCT-Nuclear Physics and Applications (No.
464898/2014-5); F. N. is a member of the Brazilian National Institute of
Science and Technology for Quantum Information [CNPq INCT-IQ (465469/2014-0)].

##  Appendix A Asymptotic behavior

When all temperatures of the heat baths are the same
$\beta_{1}=\cdots=\beta_{n}=\beta$ and the external parameters are constants
$\vec{\lambda}_{t}=\vec{\lambda}$, the stationary solution of the generalized
Kramers equation is given by the thermal equilibrium distribution,

$\displaystyle f^{*}(\\{{\bf q},{\bf p}\\})=\frac{1}{Z}e^{-\beta H(\\{{\bf
q}_{t},{\bf p}_{t}\\},\vec{\lambda})}\,,$ (36)

where $Z$ is a normalization constant,

$\displaystyle Z=\int d\Gamma\,f^{*}(\\{{\bf q},{\bf p}\\})\,.$ (37)

Indeed, we can define the Kullback-Leibler divergence by

$\displaystyle J(f|f^{*})=\int d\Gamma f\ln\frac{f}{f^{*}}\,.$ (38)

This is a monotonically decreasing function and asymptotically converges to
$f^{*}$,

$\displaystyle\frac{d}{dt}J(f|f^{*})\leq 0\,.$ (39)

##  Appendix B Generalization of concept of work

We consider a single particle of mass $m$ described by

$\displaystyle\frac{d{\bf q}}{dt}$ $\displaystyle=$ $\displaystyle\frac{\bf
p}{m}+{\bf F}^{(q)}_{ex}\,$ (40) $\displaystyle\frac{d{\bf p}}{dt}$
$\displaystyle=$ $\displaystyle-\nabla V+{\bf F}^{(p)}_{ex}\,$ (41)

Here ${\bf F}^{(p)}_{ex}$ is an external force. We further assume that the
relation between the velocity and momentum in the first equation is modified
by an external perturbation ${\bf F}^{(q)}_{ex}$. As pointed out in the
introduction, such a modification is observed in the stochastic formulation of
quantum mechanics [15, 16, 18]. Then the change of the energy of the particle,
$E={\bf p}^{2}/(2m)+V$, is represented by

$\displaystyle\frac{d}{dt}E=\frac{{\bf p}}{m}\cdot\frac{d{\bf p}}{dt}+\nabla
V\cdot\frac{d{\bf q}}{dt}={\bf F}^{(p)}_{ex}\cdot\frac{d{\bf
q}}{dt}-\frac{d{\bf p}}{dt}\cdot{\bf F}^{(q)}_{ex}\,.$ (42)

The first term on the right-hand side is the standard work and the second term
is a new contribution. This result is used to define the heat in the present
model.

## References

  * [1] K. Sekimoto, Stochastic Energetics (Springer, Berlin, 2010).
  * [2] U. Seifert, “Stochastic thermodynamics, fluctuation theorems and molecular machines”, Rep. Prog. Phys. 75, 126001 (2012).
  * [3] L. Peliti and S. Pigolotti, Stochastic Thermodynamics: An introduction, (Princeton University Press , 2021)
  * [4] R. Zwanzig, Nonequilibrium Statistical Mechanics (Oxford University Press, New York, 2004).
  * [5] H.-P. Breuer and F. Petruccione, The Theory of Open Quantum Systems (Oxford, 2002).
  * [6] H. Spohn, “Entropy production for quantum dynamical semigroups” J. Math. Phys. 19, 1227 (1978).
  * [7] R. Alicki, “The quantum open system as a model of the heat engine”, J. Phys. A: Mathematical and General 12, L103 (1979).
  * [8] E. B. Davies and H. Spohn, “Open quantum systems with time-dependent hamiltonians and their linear response”, J. of Stat. Phys. 19, 511 (1978).
  * [9] H. Spohn and J. L. Lebowitz, “Irreversible thermodynamics for quantum systems weakly coupled to thermal reservoirs”, in S.A. Rice (Ed), _Advances in Chemical Physics_ (John Wiley & Sons, Ltd, 1978).
  * [10] W. Pusz and S. L. Woronowicz, “Passive states and KMS states for general quantum systems”, Comm. Math. Phys. 58, 273 (1978).
  * [11] J. J. Sakurai and J. Napolitano, Modern Quantum Mechanics (Addison-Wesley, Boston, 2nd Ed. 2011).
  * [12] M. J. de Oliveira, “Quantum Fokker-Planck-Kramers equation and entropy production”, Phys. Rev. E 94, 012128 (2016).
  * [13] M. J. de Oliveira, “Stochastic quantum thermodynamics, entropy production, and transport properties of a bosonic system”, Phys. Rev. E 97, 012105 (2018).
  * [14] J. Burzlaf, “Canonical Quantization of Dissipative Systems”, Rep. Math. Phys. 16, 101 (1979).
  * [15] E. Nelson, “Derivation of the Schrödinger equation from Newtonian mechanics”, Phys. Rev. 150, 1079 (1966).
  * [16] K. Yasue, “Stochastic calculus of variation”, J. Funct. Anal. 41, 327 (1981).
  * [17] J. C. Zambrini, “Stochastic Dynamics: A Review of Stochastic Calculus of Variations”, Int. J. Theor. Phys. 24, 277 (1985).
  * [18] G. Gonçalves de Matos, T. Kodama and T. Koide, “Uncertainty relations in Hydrodynamics”, Water 12, 3263 (2020).
  * [19] T. Koide, “Nonequilibrium work relation from Schrödinger’s unrecognized probability theory”, J. Phys. Commun. 2, 021001 (2018).
  * [20] T. Koide, “Perturbative expansion of irreversible work in Fokker-Planck equation à la quantum mechanics”, J. Phys. A : Math. Theor. 50, 325001 (2017),
  * [21] T. Koide, “Perturbative expansion of irreversible work in symmetric and asymmetric processes”, Phys. Rev. E 106, 014145 (2022),
  * [22] J. Kurchan, “Non-equilibrium work relations”, J. Stat. Phys. 2007, P07005 (2007).
  * [23] T. Koide and T. Kodama, “Thermodynamic laws and equipartition theorem in relativistic Brownian motion”, Phys. Rev.  E83, 061111 (2011).
  * [24] P. S. Pal and S. Deffner, “Stochastic thermodynamics of relativistic Brownian motion”,‘ New J. Phys. 22, 073054 (2020).
  * [25] C. W. Gardiner, Handbook of Stochastic Method: for Physics, Chemistry and Natural Sciences (Springer, New York, 2004).
  * [26] E. T. Jaynes, “Information Theory and Statistical Mechanics”, Phys. Rev. 106, 620 (1957).
  * [27] M. J. de Oliveira, “Quantum Fokker-Planck structure of the Lindblad equation”, arXiv: 2305.05805.
  * [28] H-P. Breuer, E-M. Laine, J. Piilo, and B. Vacchini, “Colloquium: non-Markovian dynamics in open quantum systems”, Rev. Mod. Phys. 88, 021002 (2016).
  * [29] A. Colla and H-P. Breuer, “Exact Open System Approach to Strong Coupling Quantum Thermodynamics”, Phys. Rev. A 105, 052216 (2022).
  * [30] L. Onsager, “Reciprocal Relations in Irreversible Processes II.”, Phys. Rev. 38, 2265 (1931).
  * [31] H. Grabert, “Nonlinear relaxation and fluctuations of damped quantum systems”, Z. Phys. B 49, 161 (1982).
  * [32] P. Talkner, “The Failure of the Quantum Regression Hypothesis”, Ann. Phys. (N.Y.) 167, 390 (1986).
  * [33] G. W. Ford and R. F. O’Connell, “There is No Quantum Regression Theorem”, Phys. Rev. Lett. 77, 798 (1996).
  * [34] G. Guarnieri, A. Smirne and B. Vacchini, “Quantum regression theorem and non-Markovianity of quantum dynamics”, Phys. Rev. A 90, 022110 (2014).
  * [35] M. Cosacchi, T. Seidelmann, M. Cygorek, A. Vagov, D. E. Reiter and V. M. Axt, “Accuracy of the Quantum Regression Theorem for Photon Emission from a Quantum Dot”, Phys. Rev. Lett. 127, 100402 (2021); Erratum Phys. Rev. Lett. 128, 079901 (2022).
  * [36] F. Toscano and F. Nicacio, “Thermal equilibrium in Gaussian dynamical semigroups”, Phys. Rev. A 106, 062207 (2022).
  * [37] M. Schlosshauer, Decoherence and the Quantum-To-Classical Transition (Springer, Berlin, 2008).


11institutetext: Aflac NI, Belfast, Northern Ireland
11email<EMAIL_ADDRESS>
# Unimodal and Multimodal Representation Training for Relation Extraction

Ciaran Cooney  Rachel Heyburn  Liam Madigan  Mairead O’Cuinn  Chloe Thompson
Joana Cavadas

###### Abstract

Multimodal integration of text, layout and visual information has achieved
SOTA results in visually rich document understanding (VrDU) tasks, including
relation extraction (RE). However, despite its importance, evaluation of the
relative predictive capacity of these modalities is less prevalent. Here, we
demonstrate the value of shared representations for RE tasks by conducting
experiments in which each data type is iteratively excluded during training.
In addition, text and layout data are evaluated in isolation. While a bimodal
text and layout approach performs best (F1=0.684), we show that text is the
most important single predictor of entity relations. Additionally, layout
geometry is highly predictive and may even be a feasible unimodal approach.
Despite being less effective, we highlight circumstances where visual
information can bolster performance. In total, our results demonstrate the
efficacy of training joint representations for RE.

###### Keywords:

relation extraction multimodal deep learning joint representation training
information retrieval.

##  1 Introduction

With many sectors such as healthcare, insurance and e-commerce now relying on
digitization and artificial intelligence to exploit document information,
Visually-rich Document Understanding (VrDU) has become a highly active
research domain [24, 14, 21, 11]. VrDU is the task of analyzing scanned or
digital business documents to allow structured information to be extracted for
downstream business applications [21]. Sub-fields including Named-Entity
Recognition (NER) [2], layout understanding [7] and document classification
[22] all seek to extract meaningful information from documents. Another sub-
field of VrDU, relation extraction (RE) offers the possibility of linking
named entities in documents so that a paired relationship can be identified
[11, 6, 5, 3, 23]. Typically, relations are defined in a question-answer
(_Q/A_) format and the RE task is to define a function which predicts if a
pair of entities in a document are related or not [11, 23].

Concurrent with recent developments in VrDU, advances in multimodal deep
learning have seen novel methods applied across fields as diverse as medical
imaging [16], neurotechnology [4] and early prediction of Alzheimer’s disease
[17]. Commercial and opensource optical character recognition engines such as
AWS Textract111https://aws.amazon.com/textract/, Microsoft Read
API222https://docs.microsoft.com/en-us/azure/cognitive-services/computer-
vision/overview-ocr and PyTesserect333https://pypi.org/project/pytesseract/
enable extraction of detailed text and geometric information from visually-
rich documents, which along with visual information has led to a plethora of
multimodal architectures being applied to VrDU tasks [14, 12, 19, 24, 22, 21].
These approaches enable learning of joint representations in a single end-to-
end training procedure with the aim of maximising the total information in a
document. Although transformer-based architectures are prominent in this field
[23, 13], other methods for optimizing RE tasks, such as graph neural networks
[3, 6], have been reported.

Datasets for RE facilitate the use of multimodal representations for training
co-adaptive networks [11, 23]. However, despite the growth of multimodal
approaches in VrDU tasks, the extent to which learning joint representations
is an enhancement remains unclear, as does the relative capacity of each data
type. Although text is likely more predictive than either geometric layout or
visual information, the extent to which this is the case and the interaction
between modes is not known. Even in studies with ablation tests, consideration
of the effects of training without text representations has not been applied
[9].

We aim to address uncertainty regarding the predictive capacity of different
data by performing a series of experiments with different multimodal and
unimodal configurations. We apply our analysis to the RE task, due to it being
an unresolved information extraction challenge relevant to several industry
applications, and one that could benefit from appropriately trained joint
representations. Our contributions are summarised as follows: (1) We prove the
efficacy of using joint representations for VrDU RE. Specifically we
demonstrate that a text/layout configuration yields the best performance. (2)
We analyse the asymmetric predictive capacity of text, layout and visual data,
exhibiting the anticipated relative importance of text over the other data
while highlighting where layout and visual information can be effectual. (3)
We present a simplified classifier for RE based on the LayoutXLM
classification head [23].

Section 2 describes previous works related to multimodal approaches to
document understanding tasks and questions current understanding of the impact
of different modalities. Section 3 reports our methodology, including the
dataset used, model architecture, and experimental procedures. In Section 4,
we present the results of our experiments. Section 5 contains limitations and
suggestions for future work. In Section 6 we forward concluding remarks.

##  2 Related Work

Two datasets exist for the VrDU RE task. These are FUNSD [11] and XFUND [23].
Both contain annotations which include an indication of linked entities
comprising of two entity IDs which are linked, or an empty array indicating no
relationship.

The provision of text, geometry and document images in these datasets enables
the use of multimodal methods for document understanding. The LayoutLM family
of models [22, 21, 23] utilise combined text and position embeddings to
leverage the layout of the document, with LayoutLMv2 extending this approach
to fully incorporate visual information. The approach of [18] uses a similar
trifecta of inputs to perform tasks on the FUNSD and MedForm datasets. [1] use
a multimodal approach for text and image-based document classification, while
others focus on text and layout representations [15, 12, 13].

The use of multimodal approaches poses an important question: what are the
relative effects of the different data types? It is not always clear from
reported results, even in studies that do present ablation findings, what the
impact of different modalities is. Particularly since text since is usually
retained in conducted experiments [15, 9]. For industry applications, the
additional training and inference costs associated with large-scale multimodal
approaches must be mitigated by performance benefits. The original XFUND paper
does not report on the relative impact of the different components (text,
layout and visual information) [23], informing the approach taken here.

##  3 Methodology

We use the XFUND dataset444https://github.com/doc-analysis/XFUND to experiment
with different modalities for the RE task [23]. The dataset consists of
document images for form understanding in seven languages. Annotations
corresponding to each of the documents contain a unique identifier, class
label, bounding box coordinates (${x_{left},y_{top},x_{right},y_{bottom}}$),
text and a linking indicator. This linking indicator facilitates the use of
XFUND in VrDU RE. Entities for RE are designated key-value pairs corresponding
to questions and answers in the forms. For further information on dataset
collection and curation, see [23]. Dataset statistics differ from those
reported in [23] and are therefore presented in Table 1. _ZH_ , _JA_ , _ES_ ,
_FR_ , _IT_ , _DE_ and _PT_ correspond to Chinese, Japanese, Spanish, French,
Italian, German and Portuguese, respectively.

Table 1: Train/Test split for XFUND data. | `ZH` | `JA` | `ES` | `FR` | `IT` | `DE` | `PT`
---|---|---|---|---|---|---|---
Train | 187 | 194 | 243 | 202 | 265 | 189 | 233
Test | 65 | 71 | 74 | 71 | 92 | 63 | 85
Figure 1: Multimodal transformer with data exlusions color-coded. Pink denotes
exclusion of viusal components, blue exclusion of layout, and green exclusion
of text representations.

###  3.1 LayoutXLM for Relation Extraction

The multimodal deep learning architecture we use to perform our experiments is
LayoutXLM, a pretrained transformer for document understanding [23], based on
the LayoutLMv2 architecture [21]. The model ingests text, layout (bounding
boxes) and visual information which are encoded in embedding layers (Figure
1). It utilises traditional position embeddings to model word position in a
sequence, and 2D-position embeddings to consider relative spatial position. A
visual backbone encodes image representations using ResNeXt 101-FPN [20].

For the RE task, a bespoke classification layer is attached to the pretrained
model for further fine-tuning. In the original LayoutXLM, a bi-affine
classifier receives representations of _Q/A_ entities which are a processed
version of the first token vector and an entity type embedding for each. A
feed-forward neural network is applied to these representations before they
are fed to the bi-affine classifer. Here, we have slightly modified the
classification layer to simplify the feed-forward neural network and therefore
reduce the number of parameters. In this work the feed-forward neural network
consists of a single fully-connected layer, leaky relu activation and dropout
(_p_ =0.2).

###  3.2 Experimental Procedures

We conduct experiments for six different multimodal and unimodal
configurations for each of the seven sets in the multilingual XFUND data.
Experiments consist of fine-tuning the pretrained LayoutXLM model on various
configurations of the available data. The six model configurations are: (1)
Multimodal text, layout and visual (MM), (2) bimodal text and layout
(text/layout), (3) bimodal text and visual (text/visual), (4) bimodal layout
and visual (layout/visual), (5) unimodal layout (layout) and (6) unimodal text
(text). We initially planned to include a unimodal visual experiment but early
results indicated this was not feasible for the RE task.

For experiment 1, there are no further modifications to the network beyond
those specified in Section 3.1. For experiment 2, all visual components of the
architecture are removed. This includes the visual backbone and all related
embeddings, including 2D and 1D visual position embeddings (Figure 1; pink).
The model is therefore only trained on text and layout information. In
experiment 3, layout information is removed from the network in the form of 2D
position embeddings. Included in this step is the removal of 2D position
embeddings from the visual component (Figure 1; blue). Experiment 4 excludes
all text information, including tokenized text and the associated 1D and 2D
position embeddings (Figure 1; green). Tasks 5 and 6 combine the relevant
exclusions applied in experiments 2, 3 and 4. Studies often neglect to
experiment with the exclusion of text [15, 9], despite its importance when
analysing the predictive capacity of multimodal approaches.

Due to the nature of the ablation experiments we hypothesised that a degree of
variation in optimal learning rates across the different data types and
perhaps between the individual datasets was likely. For this reason, learning
rate is the only hyperparameter optimized. Three learning rates were optimized
with simple gridsearch: 5e-5, 1e-5 and 5e-6. All other learning parameters are
identical across experiments. For fine-tuning, batch size is 2, and all models
are allowed to train for 50 epochs.

##  4 Results

For all experiments, F1 score is the primary evaluation metric, with precision
and recall also reported. Although multimodal results have previously been
reported for the XFUND RE task [23], we chose not to include them in our
reporting due differences in network configuration, dataset statistics and
training procedures.

###  4.1 Bimodal training outperforms trimodal

F1 scores obtained from models trained with each of the six different network
configurations are reported in Table 2 and Figure 2. Additional recall and
precision scores are in Tables 3 and 4. Overall results validate the utility
of training on joint representations for the VrDU RE task, while also
exhibiting asymmetric predictive capacity across the different modalities
(Table 2). Here, the bimodal text/layout configuration outperforms the three-
pronged multimodal approach with mean F1 scores of 0.684 and 0.673,
respectively. This is consistent with previous work demonstrating the utility
of text and layout information without the requirement of visual information
[12]. These two training configurations significantly outperform all other
approaches with scores 8.08% and 6.93% greater than the next best approach
(Figure 2). The other bimodal approaches, text/visual and layout/visual,
result in reasonably strong F1 scores of 0.604 and 0.558 and further suggest
that joint representations can be effective for this task. However, the
overall picture indicates that in this particular application of multimodal
deep learning there is a hierarchy of predictive capacity with text on top and
visual information at the bottom.

Pairwise comparison of results from full multimodal training and the text/
layout configuration indicate that the impact of visual information is
negligible or that its inclusion is even counterproductive (Table 2). However,
comparison of unimodal and bimodal results suggest visual information can be
effective in the correct context. Most obvious is the positive impact of
visual information when compared to the unimodal layout results. In this case
the inclusion of visual data improves upon layout only results by improving
the F1 score from 0.471 to 0.558. In fact, the combination of these two
modalities results in similar performance to the unimodal text method. It is
possible that this approach is useful in scenarios where text quality is
degraded but visual and layout information are sufficient to classify the
document. However, results do not suggest that visual information is necessary
for the RE task.

Figure 2: Bar chart reporting the validation F1 scores for each language
dataset with each network configuration.

Significant variation in performance corresponding to the different document
languages may have been expected, particularly as there is a clear dichotomy
between those using Kanji characters and those using the Latin alphabet.
Despite some of the latin languages exhibiting less effective impact of layout
information, Figure 2 indicates similar levels of overall performances across
datasets, despite differences in the number of document samples per set (Table
1).

Table 2: XFUND F1 scores for different training configurations. | MM | Txt/Lay | Txt/Im | Lay/Im | Layout | Text
---|---|---|---|---|---|---
`ZH` | 0.6935 | 0.7212 | 0.6192 | 0.6334 | 0.5417 | 0.5636
`JA` | 0.6987 | 0.7181 | 0.6406 | 0.6061 | 0.5674 | 0.6321
`ES` | 0.7198 | 0.7159 | 0.6069 | 0.5657 | 0.4483 | 0.5885
`FR` | 0.6573 | 0.6747 | 0.5888 | 0.5820 | 0.5021 | 0.5285
`IT` | 0.6841 | 0.7090 | 0.6281 | 0.4906 | 0.4825 | 0.5724
`DE` | 0.6782 | 0.6701 | 0.6041 | 0.5397 | 0.4035 | 0.5821
`PT` | 0.5779 | 0.5812 | 0.5367 | 0.4909 | 0.3511 | 0.5053
Mean | 0.6728 | 0.6843 | 0.6035 | 0.5583 | 0.4709 | 0.5675
Table 3: Recall scores for XFUND data for different training configurations. | MM | Txt/Lay | Txt/Im | Lay/Im | Layout | Text
---|---|---|---|---|---|---
`ZH` | 0.6109 | 0.7607 | 0.6754 | 0.7011 | 0.6639 | 0.6440
`JA` | 0.5638 | 0.7540 | 0.6601 | 0.6619 | 0.6932 | 0.6762
`ES` | 0.6475 | 0.7210 | 0.6817 | 0.6807 | 0.4487 | 0.6136
`FR` | 0.6231 | 0.7278 | 0.6365 | 0.7278 | 0.6956 | 0.5774
`IT` | 0.6314 | 0.7090 | 0.6584 | 0.5649 | 0.5966 | 0.7009
`DE` | 0.6990 | 0.6518 | 0.6267 | 0.6158 | 0.4279 | 0.5931
`PT` | 0.4851 | 0.6640 | 0.5308 | 0.5891 | 0.4082 | 0.5089
Mean | 0.6087 | 0.7126 | 0.6385 | 0.6488 | 0.5620 | 0.6163
Table 4: Precision scores for XFUND data for different training configurations. | MM | Txt/Lay | Txt/Im | Lay/Im | Layout | Text
---|---|---|---|---|---|---
`ZH` | 0.6109 | 0.6855 | 0.5717 | 0.5777 | 0.4576 | 0.5010
`JA` | 0.5638 | 0.6854 | 0.6223 | 0.5589 | 0.4802 | 0.5934
`ES` | 0.6475 | 0.7108 | 0.5469 | 0.4768 | 0.4478 | 0.5654
`FR` | 0.6231 | 0.6288 | 0.5478 | 0.4848 | 0.3928 | 0.4872
`IT` | 0.6314 | 0.6862 | 0.6004 | 0.4336 | 0.4050 | 0.4837
`DE` | 0.6990 | 0.6894 | 0.5831 | 0.5804 | 0.3817 | 0.5715
`PT` | 0.4851 | 0.5167 | 0.5427 | 0.4207 | 0.3080 | 0.5016
Mean | 0.6087 | 0.6575 | 0.5736 | 0.5046 | 0.4104 | 0.5291

###  4.2 Text is the anchor for relation extraction

As expected, results clearly indicate the most significant drop-off in F1
score occurs when text is excluded. Of the six approaches, the two for which
text is excluded are the poorest performing. The fact that the unimodal text
method exhibits better classification performance than the bimodal
layout/visual approach is a strong indicator of the dominance of text for the
RE task.

Notwithstanding, the inclusion of supplementary data to enhance performance is
extremely valuable. Results show that each of the three configurations that
included the addition of other modalities alongside text produced improved
performance. In this case, text is necessary but not sufficient to achieve the
best possible performance. Clearly, layout information provides very important
supplementary information for the RE task. Not only is it used along with text
in the highest performing configuration but it also exhibits reasonable
performance in the absence of text, in both unimodal (0.471) and bimodal
(0.558) networks.

Despite clear dominance, there is notable variation in the strength of
predictions across languages in the absence of text representations. For ZH
and FR there is minimal difference or even improvement when text is replaced
with layout information, whereas for IT and DE this difference is substantial
(Table 2). This suggests that document diversity across regions or business
sectors may modulate the effectiveness of different approaches to training
joint representations. It may not always be obvious _a priori_ whether or not
supplementing text data with other modalities will actually enhance model
performance. Given trade-offs associated with speed and complexity,
particularly with the inclusion of visual information, efforts should be made
to evaluate the value proposition when applying multimodal techniques in
information retrieval tasks.

Figure 3: Training loss over 50 epochs for each network configuration. Shadows
indicate the range of loss across each language dataset for a given epoch.

###  4.3 Training variability is data dependent

The model’s receptivity to learning the RE task from text is further
illuminated by training loss trends for each configuration (Figure 3). A much
sharper decline towards convergence is present in those model configurations
considering text than those excluding it. Each of the four approaches using
text converge on a similar loss after 50 epochs. The other two methods (layout
and layout/image) exhibit shallower learning trajectories, greater variance
between datasets and less ability to converge within the prescribed training
time. Again this validates the extent to which text is the most important
anchor in training even when other data modalities are also included. The
higher variance in loss exhibited by the non-text approaches may also indicate
that the effectiveness of layout and image data may be more dependent on
specific datasets than text. Another factor influencing these results is the
optimal learning rate selected for each experiment. While this varied somewhat
between the different language sets within experiments, there is more obvious
variation between experiments. The full multimodal approach and the
text/layout approach are both trained with an optimal learning rate of 5e-5.
Experiments with text excluded used learning rates of either 1e-5 or 5e-6.

##  5 Limitations and Future Work

Limitations associated with this work include the relative sparsity of data
currently available for the VrDU RE task and the constraint of applying the
experiments to LayoutXLM only. To the best of our knowledge FUNSD [11] is the
only other dataset currently available for the RE task. Increasing the
diversity of datasets and also the volume of samples within datasets could
allow us to validate these results further. Additionally, other multimodal
architectures exist with different approaches to encoding and combining
modalities [8, 10]. Extending this work to include analysis of these different
methods would provide a stronger basis on which to judge the relative
contributions of text, layout and visual information to the RE task.

As well as addressing these limitations, future work may involve a large-scale
analysis of multimodal approaches to a variety of VrDU tasks. This could
include document classification, semantic entity recognition, and key
information extraction tasks and ablation analysis could be applied to a
variety of model architectures in order to fully understand how their
performance differs according to modality. In addition, time and complexity
analysis is required to understand the relative utility of different
approaches within business environments that may process documents at high
daily volumes. This would facilitate understanding of the efficiency and cost
value of the different methods.

##  6 Conclusions

Multimodal methods can have trade-offs with respect to complexity, performance
and speed when applied to industry applications. We trained a multimodal
transformer using several data configurations to understand the impact of
training joint representations for the VrDU RE task. The bimodal text and
layout approach resulted in the best performance, even beating the full
multimodal configuration. Individually, text accounts for a greater portion of
the predictive capacity than either layout or visual data. Unimodal text
achieved higher mean F1 score than bimodal layout/visual data and all
configurations with text included outperformed those with text excluded.
Nevertheless, both layout and visual information was proven to be effective in
specific conditions. Although, visual information is currently exhibiting
value as a supplementary data source to boost overall performance, our results
show that layout information is extremely important to the RE task. We also
showed that training various depending on the inclusion/exclusion of different
data types. Future work is required to examine methods for optimizing joint
representation training for document understanding, including how to best
combine different data in multimodal approaches.

## References

  * [1] Audebert, N., Herold, C., Slimani, K., Vidal, C.: Multimodal deep networks for text and image-based document classification. arXiv preprint arXiv:1907.06370 (2019)
  * [2] Carbonell, M., Fornés, A., Villegas, M., Lladós, J.: A neural model for text localization, transcription and named entity recognition in full pages. Pattern Recognition Letters 136, 219–227 (2020)
  * [3] Carbonell, M., Riba, P., Villegas, M., Fornés, A., Lladós, J.: Named entity recognition and relation extraction with graph neural networks in semi structured documents. In: 2020 25th International Conference on Pattern Recognition (ICPR). pp. 9622–9627. IEEE (2021)
  * [4] Cooney, C., Folli, R., Coyle, D.: A bimodal deep learning architecture for eeg-fnirs decoding of overt and imagined speech. IEEE Transactions on Biomedical Engineering (2021)
  * [5] Dang, T.A.N., Hoang, D.T., Tran, Q.B., Pan, C.W., Nguyen, T.D.: End-to-end hierarchical relation extraction for generic form understanding. In: 2020 25th International Conference on Pattern Recognition (ICPR). pp. 5238–5245. IEEE (2021)
  * [6] Davis, B., Morse, B., Price, B., Tensmeyer, C., Wiginton, C.: Visual fudge: Form understanding via dynamic graph editing. arXiv preprint arXiv:2105.08194 (2021)
  * [7] Gralinski, F., Stanislawek, T., Wróblewska, A., Lipinski, D., Kaliska, A., Rosalska, P., Topolski, B., Biecek, P.: Kleister: A novel task for information extraction involving long documents with complex layout. CoRR abs/2003.02356 (2020), https://arxiv.org/abs/2003.02356
  * [8] Gu, Z., Meng, C., Wang, K., Lan, J., Wang, W., Gu, M., Zhang, L.: Xylayoutlm: Towards layout-aware multimodal networks for visually-rich document understanding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4583–4592 (2022)
  * [9] Hong, T., Kim, D., Ji, M., Hwang, W., Nam, D., Park, S.: Bros: A pre-trained language model focusing on text and layout for better key information extraction from documents. arXiv preprint arXiv:2108.04539 (2021)
  * [10] Huang, Y., Lv, T., Cui, L., Lu, Y., Wei, F.: Layoutlmv3: Pre-training for document ai with unified text and image masking. arXiv preprint arXiv:2204.08387 (2022)
  * [11] Jaume, G., Ekenel, H.K., Thiran, J.P.: Funsd: A dataset for form understanding in noisy scanned documents. In: 2019 International Conference on Document Analysis and Recognition Workshops (ICDARW). vol. 2, pp. 1–6. IEEE (2019)
  * [12] Li, C., Bi, B., Yan, M., Wang, W., Huang, S., Huang, F., Si, L.: Structurallm: Structural pre-training for form understanding. arXiv preprint arXiv:2105.11210 (2021)
  * [13] Li, Y., Qian, Y., Yu, Y., Qin, X., Zhang, C., Liu, Y., Yao, K., Han, J., Liu, J., Ding, E.: Structext: Structured text understanding with multi-modal transformers. In: Proceedings of the 29th ACM International Conference on Multimedia. pp. 1912–1920 (2021)
  * [14] Liu, X., Gao, F., Zhang, Q., Zhao, H.: Graph convolution for multimodal information extraction from visually rich documents. arXiv preprint arXiv:1903.11279 (2019)
  * [15] Pramanik, S., Mujumdar, S., Patel, H.: Towards a multi-modal, multi-task learning based pre-training framework for document representation learning. arXiv preprint arXiv:2009.14457 (2020)
  * [16] Sharif, M.I., Khan, M.A., Alhussein, M., Aurangzeb, K., Raza, M.: A decision support system for multimodal brain tumor classification using deep learning. Complex & Intelligent Systems pp. 1–14 (2021)
  * [17] Venugopalan, J., Tong, L., Hassanzadeh, H.R., Wang, M.D.: Multimodal deep learning models for early detection of alzheimer’s disease stage. Scientific reports 11(1), 1–13 (2021)
  * [18] Wang, Z., Zhan, M., Liu, X., Liang, D.: Docstruct: A multimodal method to extract hierarchy structure in document for general form understanding. arXiv preprint arXiv:2010.11685 (2020)
  * [19] Wei, M., He, Y., Zhang, Q.: Robust layout-aware ie for visually rich documents with pre-trained language models. In: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. pp. 2367–2376 (2020)
  * [20] Xie, S., Girshick, R., Dollár, P., Tu, Z., He, K.: Aggregated residual transformations for deep neural networks. arXiv preprint arXiv:1611.05431 (2016)
  * [21] Xu, Y., Xu, Y., Lv, T., Cui, L., Wei, F., Wang, G., Lu, Y., Florencio, D., Zhang, C., Che, W., et al.: Layoutlmv2: Multi-modal pre-training for visually-rich document understanding. arXiv preprint arXiv:2012.14740 (2020)
  * [22] Xu, Y., Li, M., Cui, L., Huang, S., Wei, F., Zhou, M.: Layoutlm: Pre-training of text and layout for document image understanding. In: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. pp. 1192–1200 (2020)
  * [23] Xu, Y., Lv, T., Cui, L., Wang, G., Lu, Y., Florencio, D., Zhang, C., Wei, F.: Layoutxlm: Multimodal pre-training for multilingual visually-rich document understanding. arXiv preprint arXiv:2104.08836 (2021)
  * [24] Zhang, P., Xu, Y., Cheng, Z., Pu, S., Lu, J., Qiao, L., Niu, Y., Wu, F.: Trie: End-to-end text reading and information extraction for document understanding. In: Proceedings of the 28th ACM International Conference on Multimedia. pp. 1413–1422 (2020)