# Interaction enhanced quantum heat engine

Mohamed Boubakour<EMAIL_ADDRESS>Thomás Fogarty  Thomas Busch
Quantum Systems Unit, Okinawa Institute of Science and Technology Graduate
University, Onna, Okinawa 904-0495, Japan

(August 28, 2024)

###### Abstract

We study a minimal quantum Otto heat engine, where the working medium consists
of an interacting few-body system in a harmonic trap. This allows us to
consider the interaction strength as an additional tunable parameter during
the work strokes. We calculate the figures of merit of this engine as a
function of the temperature and show clearly in which parameter regimes the
interactions assist in engine performance. We also study the finite time
dynamics and the subsequent trade-off between the efficiency and the power,
comparing the interaction enhanced cycle with the case where the system
remains scale-invariant.

##  I Introduction

The study of quantum heat engines (QHEs) is a central part of the field of
quantum thermodynamics [1, 2, 3, 4]. They can be used to understand the role
that quantum effects play when comparing to classical settings, while at the
same time they have implications for the development of quantum technologies.
Usually a QHE will consist of a quantum system as the working medium (WM) to
which a conventional thermodynamic cycle (Carnot, Otto etc.) is applied in
order to extract work from the heat exchanged with a cold and a hot bath.
Historically, most works have addressed single particle systems [5, 6, 7, 8,
9, 10, 11, 3, 12, 13, 14], however more recently QHE that use interacting
systems have attracted more attention [15, 16, 17, 18]. In particular, it is
interesting to understand the effect of the interaction on the performance,
and to identify parameter regimes in which cooperative effects due to these
interactions allow to outperform single particle QHEs [19, 20, 21, 22, 23,
24]. However, care must be taken as interactions have also been shown to
reduce engine performance [25], and the dynamical control of these systems can
be more complex leading to the creation of irreversible excitations [19, 26,
27, 28, 29, 30, 31, 32].

In this work we show that a suitable tuning of the interactions can be used to
improve the performance of QHEs when compared to systems with non-interacting
working media. For this, we consider interacting bosons confined in a harmonic
trapping potential and realize the adiabatic compression and expansion strokes
of the Otto cycle through increasing and decreasing the trap frequency.
However, we also drive the interactions between two distinct values during
these strokes and show that optimal interaction strengths exist that increase
the work output and the efficiency when compared to a non-interacting engine.

We also show that the effect of the interaction strongly depends on if one
considers distinguishable or indistinguishable particles, and we calculate the
efficiency at maximum work output (EMW) showing that the interaction
significantly improves this quantity in the low temperature regime. Finally we
explore the finite time dynamics of the cycle, finding the optimal operation
times of the QHE with interacting working media.

The manuscript is organized as follows: in Sec. II we present the model that
describes the WM and the thermodynamic cycle of our QHE. In Sec. III, we study
the performance in the adiabatic limit by first looking at the case of non-
interacting particles and comparing the performance of distinguishable
particles and indistinguishable bosons in Sec. III.1. We then consider the
case of two interacting particles in Sec. III.2, and discuss the performance
of an engine with two indistinguishable bosons in Sec. III.3 and with two
distinguishable particles in Sec. III.4. We extend our results to the case of
three interacting particles in Sec. III.5. In Sec. IV we study the engine
performance with two particles for cycles run in finite time, and finally we
conclude in Sec. V.

##  II Quantum Otto heat engine with driven interaction

We consider a QHE cycle where the WM is an interacting quantum gas confined to
one dimension and trapped in a harmonic potential. The Hamiltonian is given by

$H(\omega,g)=\sum_{n=1}^{N}-\frac{\hbar^{2}}{2m}\frac{\partial^{2}}{\partial
x_{n}^{2}}+\frac{1}{2}m\omega^{2}x_{n}^{2}+g\sum_{n<p}\delta(x_{n}-x_{p}),$
(1)

where $m$ is the mass of the particles and $\omega$ is the trap frequency.
Since we only consider low temperatures, we can approximate the interaction by
a point-like potential where $g$ is the 1D interaction strength between the
particles. For $N=2$ this Hamiltonian can be analytically solved [33], however
for $N\geq 3$ numerical methods are required to find the eigenstates [34, 35].
The engine cycle we explore is similar to a standard Otto cycle except that
the adiabatic strokes occur by changing two parameters: the trap frequency
$\omega$ and the interaction strength $g$. A schematic is shown in Fig. 1 and
the individual strokes are given by

  1. Adiabatic compression $(1\rightarrow 2)$: the WM is initially trapped in a harmonic potential with frequency $\omega_{i}$ and at equilibrium with inverse cold temperature $\beta_{c}$. The interaction strength is given by $g_{i}$. From there a compression stroke is carried out that performs work on the system by increasing the trap frequency to $\omega_{f}$ and changing the interaction strength to $g_{f}$. The work is given by $W_{c}=\left<H(\omega_{f},g_{f})\right>_{2}-\left<H(\omega_{i},g_{i})\right>_{1}$.

  2. Hot isochore $(2\rightarrow 3)$: the next stroke increases the temperature of the WM by coupling it to an external hot bath at the inverse temperature $\beta_{h}$ with the control parameters $g_{f}$ and $\omega_{f}$ fixed. In equilibrium the heat exchanged during this stroke is given by $Q_{h}=\left<H(\omega_{f},g_{f})\right>_{3}-\left<H(\omega_{f},g_{f})\right>_{2}$.

  3. Adiabatic expansion $(3\rightarrow 4)$: the system is then decoupled from the hot bath and work is extracted from the WM by adiabatically driving the trap frequency and interaction strength back to $\omega_{i}$ and $g_{i}$. The work is given by $W_{e}=\left<H(\omega_{i},g_{i})\right>_{4}-\left<H(\omega_{f},g_{f})\right>_{3}$.

  4. Cold isochore $(4\rightarrow 1)$: in the last stroke the WM is cooled down by exchanging heat with a cold bath at the inverse temperature $\beta_{c}$. It returns to the initial state and the heat exchanged during this stroke is given by $Q_{c}=\left<H(\omega_{i},g_{i})\right>_{1}-\left<H(\omega_{i},g_{i})\right>_{4}$.

It is worth noting the difference between adiabatic strokes in the quantum and
in the classical regime. While carrying out an adiabatic stroke in a classical
setting means that no heat exchange occurs during the process, for quantum
systems it refers to the condition that the occupation populations of the
eigenstates remain constant. This difference in the definition implies a
difference in the time scale of the strokes. In classical heat engines, the WM
will be driven quickly in order to prevent the system from relaxing and
therefore exchanging heat with the environment, while for QHEs one needs to
drive it quasistatically based on the adiabatic theorem. The performance of
the engine is characterized by the work output $W=W_{e}+W_{c}$ and the
efficiency $\eta=\frac{\absolutevalue{W}}{Q_{h}}=1+\frac{Q_{c}}{Q_{h}}$. By
convention, we chose the variation of energy to be negative when the WM loses
energy, which means that the engine produces extractable work when $W<0$. Like
in a conventional quantum heat engine, we choose the trap frequency at the end
of the compression to be larger than the initial frequency,
$\omega_{f}>\omega_{i}$, however $g_{f}$ can be larger or smaller than
$g_{i}$.

Figure 1: Schematic of the heat engine cycle. The y-axis represents the
entropy of the WM and the x-axis represent the trap frequency and the
interaction.

##  III Engine performance in the adiabatic limit

###  III.1 Non-interacting limit and statistical influence on the performance

Before examining the effects of the interactions in the working medium, let us
first consider the non-interacting limit ($g_{i}=g_{f}=0$) in order to outline
the influence of the statistical properties on the engine performance. Below
we consider distinguishable particles and indistinguishable bosons, where
their respective statistics leads to a difference in the degeneracy of the
energy levels given by

$d(E_{n})=\sum_{n_{1}}...\sum_{n_{N}}\delta_{E_{n},\hbar\omega\left(\sum_{j=1}^{N}n_{j}+\frac{1}{2}\right)},$
(2)

where $\delta_{a,b}$ is the Kronecker symbol. We illustrate this in Fig.
2(a)-(b) for two and three particles systems and unsurprisingly the number of
states for a given energy is higher for distinguishable particles. In fact,
the gap between these two distributions increases exponentially with the
number of particles. The probability for $N$ indistinguishable bosons to be at
the same energy is therefore higher than for $N$ distinguishable particles and
this increases with the number of particles. In particular, indistinguishable
bosons will most likely stay in the ground state and the probability for a
boson to transition to an excited state will be small for low temperatures. As
a consequence, the performance of an engine realized with non-interacting
bosons will be limited in terms of work output in the temperature regimes of
our interest (which corresponds, as we will see later, to the temperature
regime where interactions lead to interesting behaviors). The respective work
output of the Otto-cycle of non-interacting bosons and distinguishable
particles as a function of the number of particles is shown in Fig. 2(c). As
expected, the work output for distinguishable particles increases linearly and
from physical arguments one can expect the work output for bosons to be sub-
linear. However in Fig. 2(c), one can see that the behaviour is more than sub-
linear and it, in fact, reaches a plateau for $N\geq 3$. This means that the
mean occupations of the energies for bosons at the hot and cold temperatures
become so similar that adding particles only contributes negligibly to the
work output. Given this drastically different behavior in the non-interacting
limit, let us next study distinguishable particles and indistinguishable
bosons in the presence of interactions.

Figure 2: Degeneracy of the energy levels for a system of (a) two
distinguishable particles (green dots) and two indistinguishable bosons (brown
dots) and (b) three distinguishable particles (green dots) and three
indistinguishable bosons (brown dots). (c) Work output of the Otto cycle of
non-interacting distinguishable particles (green dots) and non-interacting
indistinguishable bosons (brown dots) as a function of the number of
particles. The compression ratio is
$\kappa=\frac{\omega_{i}}{\omega_{f}}=\frac{1}{3}$, the cold inverse
temperature is $\beta_{c}=\frac{10}{\hbar\omega_{i}}$ and the hot inverse
temperature is $\beta_{h}=\frac{1}{\hbar\omega_{i}}$.

###  III.2 Two particles working medium

We now take account of the interaction by considering first the two particle
case ($N=2$). The Hamiltonian can be solved by introducing the center of mass
coordinate $X=\frac{x_{1}+x_{2}}{\sqrt{2}}$ and the relative coordinate
$x=\frac{x_{1}-x_{2}}{\sqrt{2}}$, which allows one to split the Hamiltonian
(1) into two decoupled single particle Hamiltonians
$H(\omega,g)=H_{CM}(\omega)+H_{r}(\omega,g)$ with

$\displaystyle H_{CM}(\omega)$
$\displaystyle=-\frac{\hbar^{2}}{2m}\frac{\partial^{2}}{\partial
X^{2}}+\frac{1}{2}m\omega^{2}X^{2},$ (3) $\displaystyle H_{r}(\omega,g)$
$\displaystyle=-\frac{\hbar^{2}}{2m}\frac{\partial^{2}}{\partial
x^{2}}+\frac{1}{2}m\omega^{2}x^{2}+\frac{g}{\sqrt{2}}\delta(x).$ (4)

The eigenstates are thus given by the two particle states $\ket{n,\nu}$, where
$\ket{n}$ is the eigenstate of the center of mass, and $\ket{\nu}$ the
eigenstate of the relative coordinate. Since the center of mass is not
affected by the interaction, the eigenstates of $H_{CM}$ are simply the
standard harmonic oscillator eigenfunctions
$\bra{X}\ket{n}=\frac{1}{\sqrt{2^{n}n!}}\left(\frac{m\omega}{\pi\hbar}\right)^{\frac{1}{4}}e^{-\frac{X^{2}}{2a^{2}}}H_{n}(\frac{X}{a})$,
where $a=\sqrt{\frac{\hbar}{m\omega}}$ and $H_{n}$ are the Hermite
polynomials, and the eigenenergies are given by
$E_{CM}^{n}=\hbar\omega(n+\frac{1}{2})$. For the relative coordinate, only the
even states are affected by the interaction and are given by
$\bra{x}\ket{2\nu}=N_{2\nu}e^{-\frac{x^{2}}{2a^{2}}}U\left(\frac{1}{4}-\frac{E_{r}^{2\nu}}{2\hbar\omega},\frac{1}{2},\frac{x^{2}}{a^{2}}\right)$
where $N_{2\nu}$ is a normalization factor, $E_{r}^{2\nu}$ the eigenenergy and
$U$ is the Kummer function [33]. The eigenenergies are determined by the
solutions of the transcendental equation

$-\tilde{g}=2\frac{\Gamma\left(-\frac{E^{2\nu}_{r}}{2\hbar\omega}+\frac{3}{4}\right)}{\Gamma\left(-\frac{E^{2\nu}_{r}}{2\hbar\omega}+\frac{1}{4}\right)},$
(5)

where $\tilde{g}=\frac{g}{\sqrt{2}\hbar\omega a}$ and $\Gamma(x)$ is the gamma
function. The odd eigenstates are again just harmonic oscillator states with
the eigenenergies $E^{2\nu+1}_{r}=\hbar\omega(2\nu+\frac{3}{2})$.

To understand how the interaction affects the efficiency, one can note that
the eigenenergies of the relative coordinate can be effectively written as

$E_{r}^{\nu}=\hbar\omega(\nu+1/2+\epsilon(\nu,\tilde{g})),$ (6)

where $\epsilon(\nu,\tilde{g})$ is an extra energy term due to the
interaction, which depends on the quantum number $\nu$ and the rescaled
interaction $\tilde{g}$. Since the interaction only affects the even states,
we have $\epsilon(2\nu+1,\tilde{g})=0$ $\forall$ $\nu$. In the limit of
repulsive infinite interactions, fermionization occurs [36] and the even
eigenenergies asymptotically approach the next higher lying odd eigenenergies,
which leads to $\epsilon(2\nu,+\infty)=1$ $\forall$ $\nu$. It is also worth
noting that the contact interaction has the strongest effect on the ground
state energy $\epsilon(0,\tilde{g})\geq\epsilon(2\nu,\tilde{g})$ $\forall$
$(\nu,\tilde{g})$.

###  III.3 Two indistinguishable bosons

Let us focus on the situation where the minimal QHE has a working medium
consisting of two indistinguishable bosons, in which case only the states that
preserve even parity can be occupied in the relative coordinate. To calculate
the efficiency of the engine, we first express the heat exchanged during the
hot and cold isochores as

$\displaystyle
Q_{h}=\sum_{n,\nu}E^{f}_{n,2\nu}\left(p_{n,2\nu}^{f}-p_{n,2\nu}^{i}\right),$
(7) $\displaystyle
Q_{c}=\sum_{n,\nu}E^{i}_{n,2\nu}\left(p_{n,2\nu}^{i}-p_{n,2\nu}^{f}\right),$
(8)

where
$E_{n,2\nu}^{s}=\bra{n,2\nu}H(\omega_{s},g_{s})\ket{n,2\nu}=\hbar\omega_{s}(n+2\nu+1+\epsilon(2\nu,\tilde{g}_{s}))$
(with $s\in\\{i,f\\}$) and the occupation populations are given by
$p_{n,2\nu}^{i}=\bra{n,2\nu}\frac{\exp\left(-\beta_{c}H(\omega_{i},g_{i})\right)}{Z(\omega_{i},g_{i},\beta_{c})}\ket{n,2\nu}$
and
$p_{n,2\nu}^{f}=\bra{n,2\nu}\frac{\exp\left(-\beta_{h}H(\omega_{f},g_{f})\right)}{Z(\omega_{f},g_{f},\beta_{h})}\ket{n,2\nu}$
(where $Z(\omega,g,\beta)=\mbox{Tr}\left[\exp\left(-\beta
H(\omega,g)\right)\right]$ is the partition function). From this, the
efficiency can be expressed as

$\eta=1+\frac{Q_{c}}{Q_{h}}=1-\frac{\sum_{n,\nu}\lambda_{n,2\nu}E_{n,2\nu}^{f}\left(p_{n,2\nu}^{f}-p_{n,2\nu}^{i}\right)}{\sum_{n,\nu}E_{n,2\nu}^{f}\left(p_{n,2\nu}^{f}-p_{n,2\nu}^{i}\right)},$
(9)

where we have introduced $\lambda_{n,2\nu}$ as the ratio between the
eigenenergies before and after the compression [25]

$\lambda_{n,2\nu}=\frac{E_{n,2\nu}^{i}}{E_{n,2\nu}^{f}}=\kappa\frac{n+2\nu+1+\epsilon(2\nu,\tilde{g}_{i})}{n+2\nu+1+\epsilon(2\nu,\tilde{g}_{f})},$
(10)

with $\kappa=\frac{\omega_{i}}{\omega_{f}}$ being the compression ratio. From
Eq. (9) one can see that the efficiency is influenced by the interaction
through the ratio $\lambda_{n,2\nu}$, and the change of population occupation
$p_{n,2\nu}^{f}-p_{n,2\nu}^{i}$. Let us recall that the eigenstates of the
harmonic oscillator for two different frequencies $\omega_{i}$ and
$\omega_{f}$ are related by the scaling transformation
$\bra{x}\ket{n(\omega_{f})}=\kappa^{-\frac{1}{4}}\bra{x\kappa^{-\frac{1}{2}}}\ket{n(\omega_{i})}$.
Also the contact interaction described by a delta function obeys the scaling
law $g\delta(\lambda x)=\frac{g}{\lambda}\delta(x)$. As a consequence, if one
chooses the final interaction to be $g_{f}=g_{i}\kappa^{-\frac{1}{2}}$ (and so
$\tilde{g}_{f}=\tilde{g}_{i}$), then the system will remain scale-invariant
i.e. it is self-similar in space and all the eigenenergies change by the same
ratio given by $\kappa$, i. e.

$E^{i}_{n,2\nu}=\kappa E^{f}_{n,2\nu}\;\forall(n,\nu).$ (11)

In that case the efficiency is given by the Otto efficiency
$\eta_{O}=1-\kappa$, which also corresponds to the efficiency of the Otto
cycle of non-interacting particles. This is illustrated in Fig. 3(a), where
the ratio between $\eta$ and $\eta_{O}$ is plotted as a function of
$\epsilon(0,\tilde{g}_{i})$ and $\epsilon(0,\tilde{g}_{f})$ and the diagonal
corresponds to the case where the WM is scale-invariant. To obtain an
efficiency that differs from $\eta_{O}$, one therefore needs to consider
systems where the eigenenergies do not change uniformly during the adiabatic
strokes [37]. To do that, we tune the initial and final interactions such that
$g_{f}\neq g_{i}\kappa^{-\frac{1}{2}}$, which allows one to distinguish two
possible cases.

The first case is when the interaction weakens during the compression stroke
($\tilde{g}_{i}>\tilde{g}_{f}$), which leads to
$\epsilon(2\nu,\tilde{g}_{f})<\epsilon(2\nu,\tilde{g_{i}})$ and
$\lambda_{n,2\nu}>\kappa$ (the region above the diagonal in Fig. 3(a)). One
then needs to be careful with the sign of the change of the occupation
population $p_{n,2\nu}^{f}-p_{n,2\nu}^{i}$. For the excited states, the sign
will be positive since at higher temperatures, the occupation population in
the excited states increases. However the change of population for the ground
state will be negative since it decreases for higher temperatures. Thus,
depending on which terms have the largest contribution, the efficiency can be
higher or lower than $\eta_{O}$. When the interactions affect the ground state
more than the excited states, the change of the occupation population of the
ground state is thus more important and we get $\eta>\eta_{O}$ (red area above
the diagonal in Fig. 3(a)). However, when the interactions are such that the
extra energy $\epsilon(2\nu,\tilde{g})$ affects significantly the excited
states, the change of the occupation population for the excited states can be
large enough that $\eta<\eta_{O}$ (blue area above the diagonal in Fig. 3(a)).
We also indicate the situation where the interaction is fixed $g_{i}=g_{f}$
(black dashed line in Fig. 3(a)), which is similar to the situation studied in
[25]. In their case the WM is an interacting gas trapped in a box and the
efficiency only decreases in the presence of the interaction. For the harmonic
oscillator, however, we observe that the interaction can enhance or hinder the
performance of the engine.

Figure 3: (a) Efficiency and (b) work output normalised to their respective
Otto-cycle values for an engine with a WM made of two interacting bosons as a
function of $\epsilon(0,\tilde{g}_{i})$ and $\epsilon(0,\tilde{g}_{f})$. The
black dash line shows the situation where the interaction is fixed
$\left(g_{i}=g_{f}\right)$. Note that the efficiency converges to $\eta_{O}$
in the limit of strong interactions due to the fermionization in the system.
In both plots the compression ratio is $\kappa=\frac{1}{3}$
($\eta_{O}=\frac{2}{3}$), the cold inverse temperature is
$\beta_{c}=\frac{10}{\hbar\omega_{i}}$ and the hot inverse temperature is
$\beta_{h}=\frac{1}{\hbar\omega_{i}}$.

The second case is when the interaction strength increases during the
compression stroke ($\tilde{g}_{f}>\tilde{g}_{i}$). This implies that
$\epsilon(2\nu,\tilde{g}_{f})>\epsilon(2\nu,\tilde{g}_{i})$ and
$\lambda_{n,2\nu}<\kappa$. By doing the same analysis as above, we reach the
opposite conclusion to the first case: if the interactions affect the ground
state more we get $\eta<\eta_{O}$ (blue area below the diagonal in Fig. 3(a)),
and if the interactions affect the excited states sufficiently we find
$\eta>\eta_{O}$ (red area below the diagonal in Fig. 3(a)). We also highlight
the area near the anti-diagonal in Fig. 3(a), where the efficiency is equal to
$\eta_{O}$. This area is not exactly the anti-diagonal and corresponds to a
crossover between the regime where $\eta>\eta_{O}$ and $\eta<\eta_{O}$ in
which the contribution from the ground state and the excited states are such
that they compensate each over and one recovers the Otto efficiency.

The maximum efficiency shown in Fig. 3(a) is $\eta\approx 1.003\eta_{0}$ and
achieved for $\tilde{g}_{i}=1.6$ and $\tilde{g}_{f}=50$
($\epsilon(0,\tilde{g}_{i})\approx 0.52$ and $\epsilon(0,\tilde{g}_{f})\approx
0.98$). One can see from the work output shown in panel (b) that the engine
outperforms the Otto cycle of two non-interacting bosons when the final
interaction takes intermediate values, while the initial interaction does not
seem to influence the work output significantly. This plot also shows that the
work output always exceeds $W_{O}$ and becomes equal to it when the initial
and final interaction are zero or go to infinity (the four corners in Fig.
3(b)). In the infinite interaction regime this is due to the system behaving
like two non-interacting fermions. The maximum work output is $W\approx
1.039W_{O}$ for $\tilde{g}_{i}=\tilde{g}_{f}=1.6$.

###  III.4 Two distinguishable particles

While driving the interaction during the cycle can clearly modify the
performance of the engine, the changes observed above for a working medium
made from two identical bosons are not very significant and the performance of
the engine stays relatively close to its non-interacting counterpart. Let us
therefore next consider the situation where the working medium consists of two
distinguishable particles, for which two major differences come into play:
first, as shown above, the degeneracy of the states is different in the non-
interacting limit, and, second, for such a system the odd states of the energy
spectrum of the relative coordinate need to be taken into account.

The efficiency and work output for this engine are shown as a function of the
interaction energies in Figs. 4(a)-(b). Like in case for indistinguishable
particles, the diagonal represents the scale-invariant cycle and therefore
retains the Otto efficiency, while this can be exceeded for
$\epsilon(0,\tilde{g}_{i})>\epsilon(0,\tilde{g}_{f})$. Indeed, we note that
interactions can noticeably improve the performance of the distinguishable
cycle (note the difference in the colour scale) with the maximum efficiency
$\eta\approx 1.124\eta_{O}\approx 0.75$ for $\tilde{g}_{i}=3$ and
$\tilde{g}_{f}=0.8$ ($\epsilon(0,\tilde{g}_{i})\approx 0.69$ and
$\epsilon(0,\tilde{g}_{f})\approx 0.34$), and maximum work $W\approx
1.43W_{O}$ for $\tilde{g}_{i}=1.95$ and $\tilde{g}_{f}=1.4$
($\epsilon(0,\tilde{g}_{i})\approx 0.58$ and $\epsilon(0,\tilde{g}_{f})\approx
0.48$). In contrast to indistinguishable bosons the efficiency is always
reduced when the final interaction is larger than the initial one,
$\tilde{g}_{f}>\tilde{g}_{i}$. Furthermore, the work output can be
significantly lower than for a cycle with a non-interacting particles, and we
note that the WM can act as a dissipator ($W>0$) for combinations of strong
and weak interactions, $(\tilde{g}_{i}\approx
0,\tilde{g}_{f}\rightarrow\infty)$ and
$(\tilde{g}_{i}\rightarrow\infty,\tilde{g}_{f}\approx 0)$, indicated by the
grey regions in Figs.4(a)-(b). We have also calculated the efficiency for
higher temperatures and have observed the same general behavior, however, the
variations of the efficiency and the work output become less pronounced.

To illustrate and better understand the behaviour of the QHE with
distinguishable particles, we calculate the efficiency as a function of the
initial interaction $\tilde{g}_{i}$ while tuning the final interaction such
that $\tilde{g}_{f}=\alpha\tilde{g}_{i}$, with $\alpha$ fixed. Fig. 4(c) shows
the efficiency for three different values of $\alpha$ ($\alpha=1$, $3$ and
$\frac{1}{3}$) and one can clearly see that the changes in the efficiency are
more significant and also very different from the setting using
indistinguishable particles. The efficiency can be enhanced when
$\tilde{g}_{f}<\tilde{g}_{i}$, while it is significantly reduced when
$\tilde{g}_{f}>\tilde{g}_{i}$. To understand this one can consider the amount
of heat exchanged with the hot and cold baths as shown in Fig. 4(d,e). In Fig.
4(d) we see that the presence of the interaction increases the amount of heat
received by the hot bath in all three cases, for weak and intermediate values
of $\tilde{g}_{i}$. However it decreases for large $\tilde{g}_{i}$ and reaches
a limit that is approximately half that of the non-interacting working medium.
In this limit the even eigenstates in the relative coordinate approach the
next higher-lying odd eigenstates and thus the spectrum becomes doubly
degenerated for distinguishable particles which implies less heat is required
for the WM to thermalize.

We observe similar behavior for $Q_{c}$ in the large $\tilde{g}_{i}$ limit
(Fig. 4(e)), however for weak and intermediate values of $\tilde{g}_{i}$ the
amount of heat dissipated in the cold bath is significantly larger when
$\tilde{g}_{f}=3\tilde{g}_{i}$, which is the reason for the decreasing
efficiency. For the same initial interaction $\tilde{g}_{i}$ the best strategy
to reduce energy loss in the cold bath is therefore to choose a weaker final
interaction $\tilde{g}_{f}$. This allows the statistics at the inverse
temperature $\beta_{h}$ to be closer to the initial statistics of the WM and
thus the change of the occupation population $p_{n,\nu}^{i}-p_{n,\nu}^{f}$
becomes smaller in magnitude such that less heat is released during the cold
isochore. The changes in the performance of the QHE are more extreme when the
baths are at low temperatures because the WM is more affected by finite
interactions.

Figure 4: (a) Efficiency and (b) work output normalised to their respective
Otto-cycle values for an engine with a WM made of two interacting
distinguishable particles as a function of $\epsilon(0,\tilde{g}_{i})$ and
$\epsilon(0,\tilde{g}_{f})$. The gray areas correspond to interaction regimes
where the system does not work as a heat engine but rather like a dissipator
with $W>0$. The black dashed line shows the case where the interaction is
fixed ($g_{i}=g_{f}$). (c) Efficiency normalized to the Otto efficiency and
heat exchanged with the (d) hot bath and (e) cold bath as a function of the
initial interaction $\tilde{g}_{i}$, with the final interaction given by
$\tilde{g}_{f}=\tilde{g}_{i}$ (black line),
$\tilde{g}_{f}=\frac{\tilde{g}_{i}}{3}$ (green line) and
$\tilde{g}_{f}=3\tilde{g}_{i}$ (orange line). The compression ratio for all
plots is $\kappa=\frac{1}{3}$ ($\eta_{O}=\frac{2}{3}$), the cold inverse
temperature is $\beta_{c}=\frac{10}{\hbar\omega_{i}}$ and the hot inverse
temperature is $\beta_{h}=\frac{1}{\hbar\omega_{i}}$.

It is also insightful to look at the efficiency at maximum work output (EMW)
which allows to quantify the engine performance for different temperature
scales and helps understanding finite time dynamics. We compare the EMW to the
Curzon-Ahlborn bound [38, 39]

$\eta_{CA}=1-\sqrt{\frac{\beta_{h}}{\beta_{c}}}\,,$ (12)

which originally corresponded to the efficiency at maximum power of the
endoreversible Carnot cycle. However it has been shown that it also holds for
the quantum Otto cycle at high temperature and in the adiabatic limit [40]. We
calculate the efficiency by maximizing the work output over $\kappa$,
$\tilde{g}_{i}$ and $\tilde{g_{f}}$ for two fixed cold bath temperatures
$\beta_{c}\hbar\omega_{i}=1$ and $\beta_{c}\hbar\omega_{i}=10$ (see Fig. 5).
We also compare the EMW of both engines, with distinguishable and
indistinguishable interacting working media, with their non-interacting
counterparts in the low temperature regime $\beta_{c}\hbar\omega_{i}=10$ in
Fig. 5(a). One can see that for the non-interacting engines, this quantity is
far below the CA bound, which is due to the fact that at low temperature the
WM dissipates a large amount of energy into the cold bath in order to close
the cycle. At the same time the efficiency for non-interacting bosons is the
lowest as its statistics makes low energy states more favorable than in the
distinguishable case. The EMW of the interacting bosons is extremely close to
that of non-interacting bosons, which is not surprising since the influence of
the interaction on the engine performance is very small (as seen in Fig. 3).
However for the two distinguishable particles, it is vastly improved by the
presence of the interaction and even coincides with the CA bound for
$\beta_{h}/\beta_{c}\gtrsim 0.5$. One can see that the gap between the
efficiency of bosons and distinguishable particles decreases when the
temperature of the hot bath is large, $\beta_{h}/\beta_{c}\rightarrow 0$, as
their statistics become identical and are given by the classical Maxwell-
Boltzmann distribution. Also the energy scales of the hot bath dwarf that of
the cold bath and therefore the influence of the initial interaction
$\tilde{g}_{i}$ becomes negligible.

This is highlighted in the inset of Fig. 5(a) where we show the corresponding
extra energy $\epsilon(0,\tilde{g}_{i})$ and $\epsilon(0,\tilde{g}_{f})$ for
the case of two distinguishable particles. As expected from our previous
analysis, the final interaction $\tilde{g}_{f}$ has to be smaller than
$\tilde{g}_{i}$ in order to improve the performance of the engine. The optimal
final interaction $\tilde{g}_{f}$ decreases when the temperature of the hot
bath increases and becomes zero for $\beta_{h}/\beta_{c}\lesssim 0.08$. As we
have already mentioned, the work output in the high temperature limit is
mostly dictated by the energy of the WM at the inverse temperature
$\beta_{h}$. Therefore the influence of the interaction becomes negligible
since the particles behave like non-interacting classical particles with an
energy approximately given by $\beta_{h}^{-1}$. Moreover, from the preceding
analysis, we know that the work output is significantly improved for non-zero
and finite interaction strengths (Fig. 4(b)) and we can thus conclude that the
interactions start to influence the performance of the engine for
$\beta_{h}/\beta_{c}\gtrsim 0.08$. We can also observe this in the
efficiencies of the different cycles which start to deviate from each other at
around this temperature (see Fig. 5(a)).

When $\beta_{h}/\beta_{c}\rightarrow 1$ both $\tilde{g}_{i}$ and
$\tilde{g}_{f}$ tend to the same limit in which the cycle is scale invariant.
We note the slight irregularities can be seen for the behavior of
$\epsilon(0,g_{f})$ in the inset of Fig. 5(a) when $\beta_{h}/\beta_{c}\gtrsim
0.8$. This is due to numerical issues, as the optimization algorithm has
difficulties in finding the maximum work output when the temperatures of both
baths are close, and therefore the work output starts to vanish. Regardless,
in this regime the efficiencies of each cycle converge to the Curzon-Ahlborn
bound as expected. Finally, we show that for a larger temperature of the cold
bath $\beta_{c}\hbar\omega_{i}=1$, the EMW for both working media are exactly
equal to $\eta_{CA}$ (see Fig. 5(b)). In this temperature regime, the effect
of the short-range interaction becomes negligible and the particles behave
like a non-interacting ideal and classical gas.

Figure 5: Efficiency at maximum work output (EMW) for the engine using two
interacting bosons (brown dots) and two interacting distinguishable particles
(green dots) for two different temperature regimes. Panel (a) shows the
efficiency calculated in the low temperature regime with
$\beta_{c}\hbar\omega_{i}=10$ and panel (b) in the intermediate temperature
regime with $\beta_{c}\hbar\omega_{i}=1$. In both panels the back line
corresponds to the Curzon-Ahlborn bound $\eta_{CA}$. The inset in panel (a)
shows the corresponding extra energies $\epsilon(0,\tilde{g}_{i})$ (orange
dots) and $\epsilon(0,\tilde{g}_{f})$ (green dots) for the case of two
distinguishable particles. The values of $\epsilon(0,g_{f})$ we obtained for
$\beta_{h}/\beta_{c}\gtrsim 0.8$ become less accurate because the work output
starts to vanish in this regime. We also show the efficiency at maximum work
output of two non-interacting distinguishable particles (green line) and two
non-interacting bosons (brown line) at the low temperature regime (a). Note
the EMW for two interacting bosons is extremely close to that of the two non-
interacting bosons.

###  III.5 Three particles working medium

In order to show how the influence of the interaction on the engine
performance scales with the number of particles, we next extend the analysis
to a three particle system. In that case the Hamiltonian can no longer be
analytically solved and a numerical method such as exact diagonalization is
required to calculate the quantities of interest. To compare with the two
particle engines we consider equivalent temperatures and compression ratios,
and also define the interaction energy $\epsilon_{3P}(n,\tilde{g})$ in a
similar way. With $n$ being the quantum index that characterizes the $n$-th
three particle eigenstate, the ground state interaction energy term is given
by

$\epsilon_{3P}(0,\tilde{g})=\frac{\bra{0}H(\omega,g)\ket{0}}{\hbar\omega}-\frac{3}{2}.$
(13)

This excess energy is such that $\epsilon_{3P}(0,0)=0$ and
$\epsilon_{3P}(0,+\infty)=3$ and again allows us to quantify the interaction
strength in the system. In Fig. 6 we show the efficiency and work output for
both indistinguishable and distinguishable particles. We note that the optimal
interactions needed for maximizing performance does not significantly change
compared to the two particle case, however, the degree of enhancement is
marginally increased, with a gain of $0.1\%$ for the maximum efficiency
($\eta\approx 1.004\eta_{O}\approx 0.668$ for
$\epsilon_{3P}(0,\tilde{g}_{i})\approx 1.64$ and
$\epsilon_{3P}(0,\tilde{g}_{f})\approx 2.96$) and $1.1\%$ for the work output
($W\approx 1.05W_{O}$ for
$\epsilon_{3P}(0,\tilde{g}_{i})=\epsilon_{3P}(0,\tilde{g}_{f})\approx 1.64$).
However, the engine using a WM of distinguishable particles shows a more
significant enhancement of the performance as the number of states that are
not affected by the interaction is much larger than in the two particle case
(see Fig. 2(a)-(b)) allowing for a more efficient work extraction process. The
resulting maximum efficiency and work output, $\eta\approx 1.21\eta_{O}\approx
0.807$ for $\epsilon_{3P}(0,\tilde{g}_{i})\approx 2.48$ and
$\epsilon_{3P}(0,\tilde{g}_{f}=0.8)\approx 1.49$ and $W\approx 1.59W_{O}$ for
$\epsilon_{3P}(0,\tilde{g}_{i})\approx 1.73$ and
$\epsilon_{3P}(0,\tilde{g}_{f})\approx 1.49$, allowing for gains of $8.6\%$
and $16\%$ respectively over the distinguishable two particle engine. This
highlights the important role the density of states plays in the performance
and how this can be modified by the statistics.

Figure 6: Efficiency and work output normalised to their respective Otto
values as a function of $\epsilon_{3P}(0,\tilde{g}_{i})$ and
$\epsilon_{3P}(0,\tilde{g}_{f})$ for a WM consisting of (a)-(b) three
indistinguishable bosons and (c)-(d) three distinguishable particles.

Finally, we show in Fig. 7 the EMW and compare the cases for two and three
distinguishable particles. While the EMWs for the engines with
indistinguishable particles are very similar, one can note a slight
enhancement of the efficiency for the WM made from indistinguishable particles
in the intermediate temperature regime.

Figure 7: Efficiency at maximum work output for three interacting bosons
(brown dots) and three distinguishable particles (green dots) in the low
temperature regime $\beta_{c}\hbar\omega_{i}=10$. The black line corresponds
to the Curzon-Ahlborn bound $\eta_{CA}$ and the purple dashed line shows the
two distinguishable interacting particles case.

##  IV Finite time dynamics

So far we studied the performance of the engine in the adiabatic limit, which
results in a vanishing power output due to the long time-scale of the strokes.
While reducing the duration of the strokes increases the power, it will,
however, inevitably lead to the decrease of the efficiency due to the
generation of irreversibility and inner friction [40, 41, 42, 43, 44, 45, 46,
47, 48, 49, 50]. In order to understand the trade-off between the power and
the efficiency of the engine, we next study the compression and expansion
strokes at finite-time for an engine made from two distinguishable particles.
For this we consider three cases. First, we change the interaction between two
values that are known to give a boost to the efficiency and work output in the
adiabatic limit (the optimal case), and second we drive the interaction in
such a way that the WM remains scale-invariant, for which the efficiency in
the adiabatic limit is given by $\eta_{O}$. As a third case we consider two
non-interacting particles in order to benchmark our results. As our focus is
on the non-adiabatic excitation during the compression and expansion strokes,
we do not consider the dynamics of the isochoric strokes. The duration $\tau$
of the compression and expansion stokes are taken to be same and for the
optimal case the ramps for the time-dependent protocols for the trap and
interaction strengths are given by

$f(t)=f(0)+10\Delta f\left(\frac{t}{\tau}\right)^{3}-15\Delta
f\left(\frac{t}{\tau}\right)^{4}+6\Delta f\left(\frac{t}{\tau}\right)^{5},$
(14)

where $\Delta f=f(\tau)-f(0)$ for $f=\\{g,\omega\\}$. In the case of scale-
invariant dynamics the interaction strength is connected to the trap frequency
through $g(t)=g(0)\sqrt{\frac{\omega(t)}{\omega(0)}}$ and we choose
$\omega(t)$ to be given by Eq. (14). To quantify the performance of the engine
at finite-time, we calculate the efficiency $\eta$ and also the effective
power (EP) of the engine defined as

$P(\tau)=-\frac{W(\tau)}{2\tau}.$ (15)

While, the latter does not strictly correspond to the power of the engine
since we only consider the total duration of the compression and expansion
strokes $2\tau$, we always assume that the WM fully thermalizes during the
isochoric stokes for a short fixed time. The EP then tells us how the power of
the engine is affected by non-adiabatic excitations created during the work
strokes, and if the duration of the isochoric stokes is short enough, it
corresponds to the first approximation of the engine power. Finally we also
quantify the irreversibility of the cycle by calculating the irreversible work

$W_{irr}(\tau)=W(\tau)-W_{ad},$ (16)

where $W_{ad}$ is the work output of the engine in the adiabatic limit.

Figure 8: (a) Effective power output (EP), (b) irreversible work and (c)
efficiency as a function of $\tau$. The green line corresponds to the optimal
case, the red dashed line corresponds to the scale-invariant case and the
black line to the non-interacting case. The compression ratio is
$\kappa=\frac{1}{3}$, the cold inverse temperature is
$\beta_{c}=\frac{10}{\hbar\omega_{i}}$ and the hot inverse temperature is
$\beta_{h}=\frac{1}{\hbar\omega_{f}}$. For the optimal case the interactions
are $\tilde{g}_{i}=1.95$ and $\tilde{g}_{f}=1.4$ such that the efficiency in
the adiabatic limit is $\eta\approx 0.7$, and for the scale-invariant case the
interactions are $\tilde{g}_{i}=\tilde{g}_{f}=1.95$.

The EP, the irreversible work and the efficiency as functions of $\tau$ are
shown in Fig. 8. Compared to the non-interacting case, using an interacting
system in a cycle of finite duration provides a significant boost to the power
in both the optimal case and the scale-invariant case. Moreover, one can see
that the EP in the optimal case is larger than in the scale-invariant case for
longer stroke durations, which is due to the work output being larger in the
adiabatic limit. However for fast strokes the EP for the scale-invariant case
becomes larger and the maximum value is reached at a shorter time than in the
optimal case. While this is a small effect, it is consistent with the greater
amount of irreversible work being generated in the optimal case (see Fig.
8(b)). It can also be seen from the fact that even if the efficiency in the
adiabatic limit in the optimal case is greater, it decreases faster for
shorter times than in the scale-invariant case (see Fig. 8(c)). Finally, one
can note that the irreversible work generated by driving the interaction in
the scale-invariant case is not very significant and therefore the efficiency
in this case stays relatively close to the efficiency of the non-interacting
case, even for short times.

##  V Conclusion

In summary, we have investigated a quantum Otto heat engine, where the working
medium is an interacting quantum system and the interaction is driven at the
same time as the trap frequency of the system during the adiabatic strokes. In
our case the interaction is a short-ranged and repulsive, and we have shown
that it allows to change the performance of the engine and outperform a
quantum Otto heat engine realized with non-interacting particles.

The performance of the engine is modified for two reasons: the first is that
the interaction does not have the same effect on all eigenstates and therefore
the eigenenergies do not change uniformly during the adiabatic strokes. This
is in contrast to models where the energy shift due to the interactions is the
same for all states, such as the Calogero-Sutherland model, and which do not
diverge from the Otto efficiency in the adiabatic limit [18, 15]. This means
that our engine shows a different behavior for finite interaction values. The
second reason is that the interaction affects the energy distribution which
allows, for example, to lose less energy during the cold isochore. The
interaction has however only a very small effect in the case of
indistinguishable bosons, while it can significantly modify the engine
performance for distinguishable particles due to the presence of odd
eigenstates states that are not affected by the interaction. While the tiny
influence of the interaction in the indistinguishable case would be extremely
difficult to experimentally observe, the significant improvement obtained in
the distinguishable case can be expected to be experimentally measurable.
Furthermore, we have found that the interactions mostly matter at low
temperatures, and that increasing the number of particles does not seem to
modify the influence of the interaction on the engine performance. In fact, it
continues to enhance the work output and the efficiency in the case of
distinguishable particles.

We have also studied non-adiabatic engine cycles of finite duration and
quantified the trade-off between the power and efficiency. We have shown that
driving the trap and the interaction by keeping the WM scale-invariant
generates less irreversible work which allows better performances in terms of
power but also efficiency at short time scales. More generally, using an
interacting system in a finite time cycle is significantly advantageous in
terms of power. Our work therefore shows the potential for developing QHEs
that possess multiple control parameters which can be changed during the work
strokes.

A number of interesting questions immediately emerge from our work. While we
have only considered repulsive interactions, attractive interactions could
potentially also lead to higher efficiencies. Furthermore, since the external
potential has a strong influence on the spectrum, optimising the trapping
potential would be a insightful study relating to the quantum nature of the
engine [25]. Finally, considering long range interactions within the working
medium [51] could lead to engines that show different behaviour at higher
temperatures.

## Acknowledgements

This work was supported by the Okinawa Institute of Science and Technology
Graduate University, and used the computing resources of the Scientific
Computing and Data Analysis section at OIST. TF acknowledges support from JSPS
under the grant KAKENHI-21K13856. TF and TB are also supported by JST Grant
Number JPMJPF2221.

## References

  * Vinjanampathy and Anders [2016] S. Vinjanampathy and J. Anders, Quantum thermodynamics, Contemporary Physics 57, 545 (2016).
  * Alicki and Kosloff [2018] R. Alicki and R. Kosloff, Introduction to Quantum Thermodynamics: History and Prospects, in  _Thermodynamics in the Quantum Regime: Fundamental Aspects and New Directions_, edited by F. Binder, L. A. Correa, C. Gogolin, J. Anders, and G. Adesso (Springer International Publishing, Cham, 2018) pp. 1–33.
  * Kosloff and Levy [2014] R. Kosloff and A. Levy, Quantum Heat Engines and Refrigerators: Continuous Devices, Annual Review of Physical Chemistry 65, 365 (2014).
  * Binder  _et al._ [2018] F. Binder, L. A. Correa, C. Gogolin, J. Anders, and G. Adesso, eds.,  _Thermodynamics in the Quantum Regime: Fundamental Aspects and New Directions_ (Springer International Publishing, Cham, 2018).
  * Abah  _et al._ [2012] O. Abah, J. Roßnagel, G. Jacob, S. Deffner, F. Schmidt-Kaler, K. Singer, and E. Lutz, Single-Ion Heat Engine at Maximum Power, Phys. Rev. Lett. 109, 203006 (2012).
  * Scovil and Schulz-DuBois [1959] H. E. D. Scovil and E. O. Schulz-DuBois, Three-Level Masers as Heat Engines, Phys. Rev. Lett. 2, 262 (1959).
  * Geva and Kosloff [1992] E. Geva and R. Kosloff, A quantum‐mechanical heat engine operating in finite time. A model consisting of spin‐1/2 systems as the working fluid, The Journal of Chemical Physics 96, 3054 (1992).
  * Humphrey  _et al._ [2002] T. E. Humphrey, R. Newbury, R. P. Taylor, and H. Linke, Reversible Quantum Brownian Heat Engines for Electrons, Phys. Rev. Lett. 89, 116801 (2002).
  * Hugel  _et al._ [2002] T. Hugel, N. B. Holland, A. Cattani, L. Moroder, M. Seitz, and H. E. Gaub, Single-Molecule Optomechanical Cycle, Science 296, 1103 (2002).
  * Roßnagel  _et al._ [2016] J. Roßnagel, S. T. Dawkins, K. N. Tolazzi, O. Abah, E. Lutz, F. Schmidt-Kaler, and K. Singer, A single-atom heat engine, Science 352, 325 (2016).
  * Kosloff and Rezek [2017] R. Kosloff and Y. Rezek, The Quantum Harmonic Otto Cycle, Entropy 19, 136 (2017).
  * Goswami and Harbola [2013] H. P. Goswami and U. Harbola, Thermodynamics of quantum heat engines, Phys. Rev. A 88, 013842 (2013).
  * Newman  _et al._ [2017] D. Newman, F. Mintert, and A. Nazir, Performance of a quantum heat engine at strong reservoir coupling, Phys. Rev. E 95, 032139 (2017).
  * Ono  _et al._ [2020] K. Ono, S. N. Shevchenko, T. Mori, S. Moriyama, and F. Nori, Analog of a Quantum Heat Engine Using a Single-Spin Qubit, Phys. Rev. Lett. 125, 166802 (2020).
  * Jaramillo  _et al._ [2016] J. Jaramillo, M. Beau, and A. del Campo, Quantum supremacy of many-particle thermal machines, New Journal of Physics 18, 075019 (2016).
  * Chen  _et al._ [2019] Y.-Y. Chen, G. Watanabe, Y.-C. Yu, X.-W. Guan, and A. del Campo, An interaction-driven many-particle quantum heat engine and its universal behavior, npj Quantum Inf 5, 88 (2019).
  * Carollo  _et al._ [2020] F. Carollo, F. M. Gambetta, K. Brandner, J. P. Garrahan, and I. Lesanovsky, Nonequilibrium Quantum Many-Body Rydberg Atom Engine, Phys. Rev. Lett. 124, 170602 (2020).
  * Myers  _et al._ [2021] N. M. Myers, J. McCready, and S. Deffner, Quantum Heat Engines with Singular Interactions, Symmetry 13, 10.3390/sym13060978 (2021).
  * Fogarty and Busch [2020] T. Fogarty and T. Busch, A many-body heat engine at criticality, Quantum Science and Technology 6, 015003 (2020).
  * Yunger Halpern  _et al._ [2019] N. Yunger Halpern, C. D. White, S. Gopalakrishnan, and G. Refael, Quantum engine based on many-body localization, Phys. Rev. B 99, 024203 (2019).
  * Uzdin [2016] R. Uzdin, Coherence-Induced Reversibility and Collective Operation of Quantum Heat Machines via Coherence Recycling, Phys. Rev. Applied 6, 024004 (2016).
  * Vroylandt  _et al._ [2017] H. Vroylandt, M. Esposito, and G. Verley, Collective effects enhancing power and efficiency, EPL (Europhysics Letters) 120, 30009 (2017).
  * Niedenzu and Kurizki [2018] W. Niedenzu and G. Kurizki, Cooperative many-body enhancement of quantum thermal machine power, New Journal of Physics 20, 113038 (2018).
  * Myers  _et al._ [2022] N. M. Myers, F. J. Peña, O. Negrete, P. Vargas, G. D. Chiara, and S. Deffner, Boosting engine performance with bose–einstein condensation, New Journal of Physics 24, 025001 (2022).
  * Chen  _et al._ [2018] J. Chen, H. Dong, and C.-P. Sun, Bose-Fermi duality in a quantum Otto heat engine with trapped repulsive bosons, Phys. Rev. E 98, 062119 (2018).
  * Keller  _et al._ [2020] T. Keller, T. Fogarty, J. Li, and T. Busch, Feshbach engine in the Thomas-Fermi regime, Phys. Rev. Research 2, 033335 (2020).
  * Hartmann  _et al._ [2020] A. Hartmann, V. Mukherjee, W. Niedenzu, and W. Lechner, Many-body quantum heat engines with shortcuts to adiabaticity, Phys. Rev. Research 2, 023145 (2020).
  * Beau  _et al._ [2016] M. Beau, J. Jaramillo, and A. Del Campo, Scaling-Up Quantum Heat Engines Efficiently via Shortcuts to Adiabaticity, Entropy 18, 168 (2016).
  * Deng  _et al._ [2018] S. Deng, A. Chenu, P. Diao, F. Li, S. Yu, I. Coulamy, A. del Campo, and H. Wu, Superadiabatic quantum friction suppression in finite-time thermodynamics, Science Advances 4, 5909 (2018).
  * Li  _et al._ [2018] J. Li, T. Fogarty, S. Campbell, X. Chen, and T. Busch, An efficient nonlinear Feshbach engine, New Journal of Physics 20, 015005 (2018).
  * Fogarty  _et al._ [2019] T. Fogarty, L. Ruks, J. Li, and T. Busch, Fast control of interactions in an ultracold two atom system: Managing correlations and irreversibility, SciPost Phys. 6, 21 (2019).
  * Mikkelsen  _et al._ [2022] M. Mikkelsen, T. Fogarty, and T. Busch, Connecting scrambling and work statistics for short-range interactions in the harmonic oscillator, Phys. Rev. Lett. 128, 070605 (2022).
  * Busch  _et al._ [1998] T. Busch, B.-G. Englert, K. Rzażewski, and M. Wilkens, Two Cold Atoms in a Harmonic Trap, Foundations of Physics 28, 549 (1998).
  * Weiße and Fehske [2008] A. Weiße and H. Fehske, Exact Diagonalization Techniques, in  _Computational Many-Particle Physics_, edited by H. Fehske, R. Schneider, and A. Weiße (Springer Berlin Heidelberg, Berlin, Heidelberg, 2008) pp. 529–544.
  * Na and Marsiglio [2017] M. Na and F. Marsiglio, Two and three particles interacting in a one-dimensional trap, American Journal of Physics 85, 769 (2017).
  * Girardeau [1960] M. Girardeau, Relationship between Systems of Impenetrable Bosons and Fermions in One Dimension, Journal of Mathematical Physics 1, 516 (1960).
  * de Oliveira and Jonathan [2021] T. R. de Oliveira and D. Jonathan, Efficiency gain and bidirectional operation of quantum engines with decoupled internal levels, Phys. Rev. E 104, 044133 (2021).
  * Curzon and Ahlborn [1975] F. L. Curzon and B. Ahlborn, Efficiency of a Carnot engine at maximum power output, American Journal of Physics 43, 22 (1975).
  * Leff [1987] H. S. Leff, Thermal efficiency at maximum work output: New results for old heat engines, American Journal of Physics 55, 602 (1987).
  * Rezek and Kosloff [2006] Y. Rezek and R. Kosloff, Irreversible performance of a quantum harmonic heat engine, New Journal of Physics 8, 83 (2006).
  * Alecce  _et al._ [2015] A. Alecce, F. Galve, N. L. Gullo, L. Dell’Anna, F. Plastina, and R. Zambrini, Quantum Otto cycle with inner friction: finite-time and disorder effects, New Journal of Physics 17, 075007 (2015).
  * Feldmann and Kosloff [2000] T. Feldmann and R. Kosloff, Performance of discrete heat engines and heat pumps in finite time, Phys. Rev. E 61, 4774 (2000).
  * Feldmann and Kosloff [2003] T. Feldmann and R. Kosloff, Quantum four-stroke heat engine: Thermodynamic observables in a model with intrinsic friction, Phys. Rev. E 68, 016101 (2003).
  * Çakmak  _et al._ [2017] S. Çakmak, F. Altintas, A. Gençten, , and O. E. Müstecaplıoğlu, Irreversible work and internal friction in a quantum Otto cycle of a single arbitrary spin, The European Physical Journal D 71, 016101 (2017).
  * Esposito  _et al._ [2010] M. Esposito, R. Kawai, K. Lindenberg, and C. Van den Broeck, Efficiency at Maximum Power of Low-Dissipation Carnot Engines, Phys. Rev. Lett. 105, 150603 (2010).
  * Andresen [2011] B. Andresen, Current Trends in Finite-Time Thermodynamics, Angewandte Chemie International Edition 50, 2690 (2011).
  * Whitney [2014] R. S. Whitney, Most Efficient Quantum Thermoelectric at Finite Power Output, Phys. Rev. Lett. 112, 130601 (2014).
  * Shiraishi  _et al._ [2016] N. Shiraishi, K. Saito, and H. Tasaki, Universal Trade-Off Relation between Power and Efficiency for Heat Engines, Phys. Rev. Lett. 117, 190601 (2016).
  * Lin and Chen [2003] B. Lin and J. Chen, Performance analysis of an irreversible quantum heat engine working with harmonic oscillators, Phys. Rev. E 67, 046105 (2003).
  * Abah and Lutz [2016] O. Abah and E. Lutz, Optimal performance of a quantum Otto refrigerator, EPL (Europhysics Letters) 113, 60002 (2016).
  * Beau  _et al._ [2020] M. Beau, S. M. Pittman, G. E. Astrakharchik, and A. del Campo, Exactly solvable system of one-dimensional trapped bosons with short- and long-range interactions, Phys. Rev. Lett. 125, 220602 (2020).


# Lucid Explanations Help: Using a Human-AI Image-Guessing Game to Evaluate
Machine Explanation Helpfulness

Arijit Ray Giedrius Burachas Yi Yao Ajay Divakaran
SRI International, Princeton, NJ, USA
<EMAIL_ADDRESS>
###### Abstract

While there have been many proposals on how to make AI algorithms more
transparent, few have attempted to evaluate the impact of AI explanations on
human performance on a task using AI. We propose a Twenty-Questions style
collaborative image guessing game, Explanation-assisted Guess Which (ExAG) as
a method of evaluating the efficacy of explanations in the context of Visual
Question Answering (VQA) - the task of answering natural language questions on
images. We study the effect of VQA agent explanations on the game performance
as a function of explanation type and quality. We observe that “helpful”
explanations are conducive to game performance (by almost 22% for “excellent”
rated explanation games), and having at least one “correct” explanation is
significantly helpful when VQA system answers are mostly noisy (by almost 30%
compared to no explanation games). We also see that players develop a
preference for explanations even when penalized and that the explanations are
mostly rated as “helpful”.

Figure 1: In the ExAG game, a human player is tasked to guess the secret image
among the pool of images shown based on the answers given by the VQA , which
may produce noisy answers. Good explanations shown along with the answer help
the human judge whether the answer is trustworthy or not, hence helping
him/her succeed more often than when there is no explanation or when there is
a bad explanation. In the example shown, the player asks “is there a clock in
the image?” to figure out if the clock image (third image) is the secret
image. The machine answers “no”. Without explanations, a user has no way of
knowing if the answer is correct or not. However, when explanations are shown,
the user sees that the machine did pay attention to the clock in the third
image and yet answered “no”. Hence, it is likely that he/she should trust the
answer and that the third image is not the secret image.

##  1 Introduction

Deep networks are often deemed as un-interpretable and black-box models. This
raises issues of trust and reliability for an end user. In the context of
Visual Question Answering (VQA) [2], the task of answering questions asked to
images, various methods for shedding light on the inner workings of these
networks have been proposed — interpretable attention models [17, 13, 28],
visual-semantic explanations [17, 19, 28, 22] and human-readable text-based
justifying explanations [19] to mention a few. However, the empirical evidence
that such a-priori devised explanations can actually be meaningful and useful
for a human-machine collaborative task is lacking.

We propose a Twenty-Questions [15] style human-machine collaborative game,
Explanation-Assisted GuessWhich (ExAG) using VQA [2] as the underlying task
for evaluating the efficacy of explanations. An explanation, in this context,
is defined as information that sheds light on the reasoning of the VQA agent
for predicting a certain answer. In this game, the human and a VQA machine
must collaborate to retrieve a secret image. The machine has access to the
secret image while the human doesn’t. The job of the machine is to help the
human identify the image by answering questions. Since the VQA machine is
noisy in its answer predictions, our hypothesis is that humans will succeed
more often in finding the correct image when the machine also explains its
reasoning, compared to just answering. Practical applications of such a game
are image retrieval using free-form queries. Finding the correct image in this
game means a VQA system could successfully convey the right image in the
player’s mind by simply answering questions about it. This can have a wide
variety of applications. For example, assisting disaster personnel, where a
rescuer may have to rely on audio answers from a VQA machine because he/she is
too busy to look at a video/image feed. It can also help medical
professionals, where a doctor may use visual explanations to judge the
confidence of a certain diagnosis among others.

We ran two versions of the ExAG game internally and on Amazon Mechanical Turk
(AMT) and collected user performance as a function of their use of the
explanations. The first version allowed users to choose explanations at will.
This study provided evidence that the win rate is increased significantly when
explanations (even when noisy) are used and humans increasingly prefer
explanations even when their use is penalized. The second version was
controlled to show certain modes of explanations (or lack thereof) to certain
workers and  was designed as a tighter metric for evaluating explanation
quality. We show evidence that “helpful” explanations, as self-reported by the
players while asking questions (before knowing the outcome of the game), are
conducive to game performance by almost 29% when explanations are rated
“excellent”. Moreover, having at least one “Correct” explanation (rated
independently) help performance when answers are noisy in a game (by almost
22% compared to no explanations). On the other hand, we also show that
incorrect explanations significantly hurt game performance. We also show that
good explanations improve game performance when image sets are difficult and
when VQA answers are wrong. Since correctness and helpfulness of the
explanations correlate with the game performance (for a certain image set
difficulty level and VQA accuracy), we believe this game to be an effective
evaluation framework for explanations.

Our ExAG game demo is available for public playing
(https://gwapp.nautilus.optiputer.net/)

##  2 Related Work

Explainable AI  Early work on explainable models involves template-based
systems that spanned from medical systems [23] to educational settings [14,
27]. Recent interest in explaining the inferences of a deep networks for
computer vision applications includes introspective explanations that show the
intermediate features of importance in making a decision [17, 19, 28, 10, 9,
22, 30], as well as post-hoc rationalization techniques such as justifying
textual explanations [19] and generating visual explanations [12]. We focus on
using attention-based visualization of important image regions [17, 28, 13,
26], object/scene detection [25, 11] and related question answers [21] for our
game. [6] shows that humans and machines look differently at images when
answering questions. Hence, it is not clear whether such explanations are
indeed helpful to humans. In this paper, we try to quantify how much these
explanation modes help in human-machine collaboration performance.

Visual Question Answering  We use Visual Question Answering (VQA) [2] as the
core underlying technology for our human-AI collaborative game. VQA was
introduced by [2] as an AI complete task for image and text understanding.
Most of the effective approaches to VQA consist of works with attention on
image features [29, 17, 26, 28, 13] guided by the question in order to answer
it. We implement a custom model that attends to both objects and free-form
spatial regions in the image to answer the question similar to [29].

20 Questions Game  Our choice of the image-guessing game is the visual version
of the popular 20-questions game, which is more formally, a specific version
of the classic Lewis Signaling Game [15]. There have also been efforts at
training AI agents to play such an image-guessing game with humans/AI’s [8]
using reinforcement learning- [7]. [5] used this game to evaluate visual
conversational agent performance. However, to our knowledge, we are the first
ones to use such a game to evaluate the effect of VQA explanations on human-
machine team performance.

Mental Model of an AI System  Along the lines of quantifying explanation
efficacy, [4, 3] quantifies whether attention-based explanation improves human
prediction of VQA system answer correctness and accuracy. While they [4, 3]
show no significant trend of being able to predict model outcome using
attention-based explanations, we show that only good combined attention +
related question-answer explanations are helpful in a game setting where
multiple rounds of question-answering are involved. We also show that
erroneous explanations are severely hurtful to game performance. [19] show
that showing both attention-based and textual explanations is helpful for
predicting model performance. We use related question-answers as a form of
textual explanation and also see similar trends for such a collaborative
question-answering task.

##  3 Game Outline

In our game setting, there are two agents: a state-of-the-art custom VQA deep
learning model trained to answer questions about images (the “VQA machine”),
and a human volunteer (the “player”) that has to guess a secret image picked
by the machine while losing as few points as possible. A secret image is
picked from a pool of 1500 images at random. We select $N-1$ more images from
the pool using a difficulty measure based on the VGG16 [24] FC7 distance that
is adjusted so that the final $N$ image set is challenging enough to require
multiple questions and answers. The player starts with $P_{o}$ points and is
allowed to ask free-form questions to the VQA machine in order to guess the
secret image and each question costs one point. The final score is $P=P_{o}-Q$
if the correct image is guessed, where $Q$ is the number of questions asked.
Wrong image guess gets a score of $P=0$. A success is defined as the player
correctly selecting the secret image while keeping $P>0$. Players are
encouraged to keep $P$ as high as possible.

###  3.1 The VQA Model

Figure 2: ResNet-based VQA model (Model A). This network attends to ResNet 161
features. This model is based on [13]. Figure 3: Mask-RCNN-based VQA Model
(Model B). This network has attention on both global ResNet features (i.e.,
spatial attention) and region proposal network (RPN) features (i.e., object
attention). This model is influenced by [13] and [26]

We used two SOTA VQA models, one with a ResNet-based image encoder, referred
to as Model A (see Figure 2) and the other uses both ResNet [25] and Mask-
RCNN-based [11] image encoders, referred to as Model B (see Figure 3). Both
models use an attention mechanism to select visual features generated by an
image encoder and an answer classifier that predicts an answer from 3000
candidates. Our VQA models are very similar to [13, 26, 29]. More details
about architecture and training are included in supplementary.

###  3.2 Modes of Explanations

We define explanations as information that provides insight into why the VQA
predicted a certain answer. The insight can be visualizations of the evidence
used to infer the answer, such as weights applied to visual encoder features
(i.e., attention). It can also be rationalizations, such as stating the
semantic beliefs about a fact that led to the answer. Below, we outline three
modes of explanations and illustrate how they can be used in our ExAG game to
aid players in succeeding in the game.

#### Attention:

We use spatial and object attention masks computed by the attention modules to
highlight spatial locations/objects in the image that are weighted more
heavily and thus contribute more as evidence in the inference process of
answer prediction. A player can check if the attention masks correspond to the
relevant part of the image given the question to determine if the machine
generated answer is trustworthy.

Model A (Figure 2) employs spatial attention only on ResNet161 [25] features.
Model B (Figure 3) employs two types of attentions - spatial attention that
gates visual features in the image space and object attention derived from
region proposals and centering at objects detected in the image. Accordingly,
we have two types of attention-based explanations: 1) heatmaps based on
spatial attention overlaid on the input images and 2) object segmentation
masks weighted by object attention and overlaid on input images. In both
cases, the attention is question and image guided.

#### Object and Scene Predictions:

We display a list of the most relevant object and scene predictions observed
in the image. The relevance of an object/scene $O$ is measured via an
importance score $S(O)$ by $S(O)=\frac{word2vec(O,A)}{p(O|I)}$ where $I$ and
$A$ denote the image and machine generated answer, respectively, and
$word2vec(O,A)$ denotes the Word2Vec [18] similarity of the object and answer
words. When using object-based attention, this importance score calculation is
skipped and attention weights are used instead. The player needs to judge if
the listed relevant objects make sense with the image and machine generated
answer. This can help judging if the answer is correct or not.

#### Answers to related questions (RelQAS or Related QAS):

We list five questions from the VQA2.0 Validation Dataset that are most
relevant to the asked question and answer them using the VQA. The relevance of
a question is measured via a semantic similarity based on the averaged
word2vec distance [18] over all the words in the question and answer for the
image. The five answers predicted to these closely related questions are used
as another mode of explanation [21]. Similar to explanation based on relevant
object/scene, the player should judge the trustworthiness of machine decision
based on the correctness as well as coherence of these answers. An example is
shown in Figure 4.

Figure 4: Related question answer explanations shown in response to the
question “what is the image?” for the given image. The answer predicted by the
VQA for the image is “surfer”

###  3.3 Game Settings

#### Setting A:

In this setting, we show $N=20$ images and the player has the option of
receiving explanations for the answer or not. Each question asked costs one
point and explanation, if requested, costs an additional two points. All
explanation modes are shown once the player chooses to receive explanations.
This includes heatmaps of spatial attention for all the images, a list of
relevant objects/scenes for the secret image, and related question-answer
pairs for the secret image. Note that the detected objects/scenes and the
answers to the related questions are from actual model predictions and hence,
noisy. Thus, they have to be correct/coherent with the original answer and not
be misleading (listing objects of the similar distractor images) in order to
be helpful.

#### Setting B:

In Setting B, we show RelQAS and attention for all the images. This ensures
that extra information is given for all images and only the coherence of the
extra information can be used to aid in the game. In order to reduce the
cognitive load on the players, we reduce image set size from $N=20$ to $N=5$
and make the images more similar to each other to maintain the same level of
difficulty. Since we use object-based attention, we drop the “object/scene”
explanations in this setting. We also drop the penalty for using explanations
since we randomly assign each explanation type to a group of AMT workers. As
the players play the game, they are also asked to rate how helpful the
explanations are after each round of question answering. Note that at the time
of rating, players do not yet know the outcome of the game or the secret
image. So, their rating reflects how helpful the explanations are in narrowing
down the secret image and identifying the proper question to ask next.

Figure 5: Attention visualization Version 1. The question is “What is in the
image?”. The answer predicted is “banana”. Figure 6: Attention visualization
Version 2. The question is “What is in the image?”. The answer predicted is
“sheep”.

In order to examine the effect of different attention visualization, we have
tested two variations:

  * •

Version 1: Transparent spatial attention, transparent object attention with
semantic labels (see Figure 5). The object labels come from a pool of 88
MSCOCO [16] classes detected by the Mask-RCNN [11].

  * •

Version 2: Spatial attention visualized via the jet color map, transparent
object attention without object labels (see Figure 6). Initial studies on AMT
show this version to be more helpful in terms of game performance and ease of
understanding (object labels are often confusing since the 88 class labels are
different from VQA answer vocabulary. eg, “dress” vs “shirt”). Hence, all
future analyses are done with attention version 2.

##  4 Results

Table 1: Pilot data game setting A performance. | # Plays | # Wins | Win Rate
---|---|---|---
Total Plays | 206 | 89 | 43.20%
| Expl. Used Atleast once
---
157 | 75 | 47.77%
No explanations used | 49 | 14 | 28.57%
Table 2: Each explanation type was randomly assigned to a group of AMT workers. “With Expl” are plays with explanations, “No Expl” are plays by the same worker group with no explanations. “Group Baseline” are the the first 5 plays of that worker group without explanations to set an initial baseline for that group. “Both” explanation type refers to plays where Attention and RelQAS were shown together. “Overall” is the average across all three groups. “Overall Improv” refers to the improvement compared to the overall baseline. Expl Type | With Expl | No Expl | Group Baseline | Overall Improv
---|---|---|---|---
Score | Win Rate | Score | Win Rate | Score | Win Rate | Score | Win Rate
Attention | 6.23 | 66.67 | 6 | 64.92 | 5.66 | 62.1 | 0.42 | 2.82
Rel QAS | 6.8 | 71.48 | 6.03 | 64.54 | 6.02 | 65.45 | 0.99 | 7.63
Both | 6.44 | 69.03 | 5.83 | 63.25 | 5.68 | 63.75 | 0.63 | 5.18
Overall | 6.52 | 69.29 | 5.97 | 64.3 | 5.81 | 63.85 | 0.71 | 5.44

###  4.1 Game Setting A

Figure 7: Explanation Adoption Over Time

As a pilot study, the ExAG game setting A was played in a competitive setting
(with cash rewards for group that won the most) by about 60 people grouped
into 6 teams. The players were free to choose explanations or forgo them.

Of 206 total games played, the average win rate was 43%. We divided the game
plays into games where explanations were never used (49) and those where
explanations were used at least once (157; see Table 1). The win rate with
explanations was 47% (N=75) and 28% when explanations were never used (N=14).
The z-test for proportions indicates that this is a statistically significant
difference at 95% confidence level (p=0.019).

Moreover, as the players proceeded to play the ExAG game, they tended to opt
for using explanations even though that resulted in additional 2 point
penalty. Figure 7 shows this spontaneous adoption of explanation with
increasing number of plays. A z-test comparing the proportion of games using
the explanations during the first half of plays (61.2%, N=103) vs. during the
second half (91.3%, N=103) indicates highly statistically significant increase
in explanation use ($p<10^{-5}$).

Figure 8: Histogram of ratings of how “helpful” explanations seemed while
playing the game. These ratings were collected after each question was asked.
Workers didn’t know the GT image or the outcome of the game while rating them.
Figure 9: Histogram of ratings of how “correct” explanations were for the GT
image and question asked. These ratings were collected separately by three
independent AMT workers. “Correct” is defined by whether the attention looks
at the relevant parts of the image for the given question and whether the
related question answers are actually related and correct.

###  4.2 Game Setting B

Figure 10: Game win percentage as a function of self-reported helpfulness
rating. Baseline Group Performance is the performance of the first 5 runs
without explanations for the same group of workers to set an initial baseline
for that group. Helpfulness was self rated by workers before they knew the GT
image or the outcome of the game. The ratings were averaged for a game and
binned into the original 5 likert scale choices. Figure 11: Game win
percentage as a function of independently rated explanation accuracy. Baseline
Group Performance is the performance of the first 5 runs without explanations
for the same group of workers to set an initial baseline for that group. Note
that the baseline performance here may differ from Figure 10 because the
correctness ratings were collected for a subset of the games and baseline was
adjusted to reflect the baseline of only that subset of workers.

Game Setting B shows related question-answer pairs and segmented objects (in
attention) for all images and hence, players have to solely rely on the
quality of explanations to the answer to aid their game. Hence, extra
information about GT image cannot be misused. All analyses henceforth are
performed on Game Setting B.

The games were played by 69 individual AMT workers. The instructions were to
try to win the game by guessing the secret image correctly and were warned
that an obvious lack of effort would lead to rejection. For AMT worker
selection, worker qualification threshold was set to above 98% (number of HITs
$>1000$) and location set to US only for IRB regulations. We will anonymize
and release all game play data publicly. The workers played 1469 games in
total covering the three explanation modes - attention, related question-
answer pairs and both, and also without any explanations.

Each individual worker always saw only one type of explanations while playing.
Each worker was instructed to play at least 4 sessions with each session
consisting of five games. Sessions with and without explanation alternate. For
instance, the first session dose not provide explanations, followed by a
session with assigned choice of explanations. We also have a group that sees
no explanations ever. The first block is used as the baseline no-explanation
performance for the assigned explanation worker group. Workers who played with
one explanation never play any other explanation type. Moreover, once each
worker finished their HIT, they were not allowed to play another HIT so as to
not mess up the first baseline block performance. We also kept track of worker
ID to ensure game play progress was saved for each worker, and didn’t reset if
they returned to the interface at a later time.

#### Overall impact of explanations

The overall impact of explanations on the performance of the ExAG game is
summarized in Table 2. Both the average game score and the game win rate are
modestly improved by explanations. Consistently with prior reports [4, 3], we
don’t see a tangible effect of attention-based explanations when presented in
isolation. When compared to the baseline no-explanation performance for the
group (Baseline Group), related question-answers (relQAS) and showing combined
(Attention + RelQAS) explanations improves game performance the most.

We observe that the helpfulness of explanations is not necessarily how correct
they are, but a more elusive function of how telling they are to VQA
success/failure. Hence, we collect two separate ratings for the explanations.
While playing the game, workers are asked to self rate their “perceived
helpfulness” of the explanation for zeroing down on an image after each
question asked (Figure 8). Note that the workers do not know the GT image or
the outcome of the game while rating. We also separately collect the
correctness of the explanation from 3 independent AMT workers by showing them
the explanation for the asked question and the GT image (Figure 9). We use
these ratings to analyze how “correctness” and “perceived helpfulness” affect
game performance.

#### Impact of explanations as a function of self-reported helpfulness ratings

We collect self-reported helpfulness ratings of explanations while workers are
playing the game (Figure 8), while the play outcome is not yet known in order
to avoid circularity in the ratings. We analyze game performance as a function
of these ratings to see if explanations perceived as “helpful” do help game
performance. Since workers didn’t know the correct image or the game outcome
while rating, their decision was not affected by the (future) play outcome
(win/lose) and likely reflected the helpfulness of explanations in narrowing
down their image choices. The workers were asked to rate the explanations
according to the following options: “Helping a lot” (Excellent), “Mostly
helping” (Mostly Good), “Somewhat Helpful” (Somewhat), “Not helping much” (Not
Much) and “Completely Confusing” (Misleading). For all plots, the ratings were
averaged for a game and were binned into the original choices.

We see that overall (Figure 10a), explanations perceived as Excellent
significantly increase game performance. When explanations are less helpful,
performance is similar to playing without explanations supporting the notion
that workers seem to ignore the explanations, or somewhat degraded as the
workers were confused by them.

Next we break down explanation helpfulness-dependent performance by
explanation type. Panel b of Figure 10 shows that combining attention and
related QAs improves performance significantly for explanations rated as
Excellent, but hurts performance slightly when rated below “somewhat”. Panel c
of Figure 10 indicates that attention-based explanations don’t help much on
their own, however, when they seem very helpful for making a choice, they do
help game performance slightly. Excellent Related QAs were the most helpful
for game performance when presented in isolation as in Figure 10d. We reason
that this is due to the consistency of related question answers being a better
indicator of VQA answer accuracy. We calculate the correlation coefficient of
the explanation correctness to answer correctness and observe that it is
$0.37$ for related question-answers as compared to $0.33$ for attention. This
combined with the ease of parsing textual attention than heatmaps make related
question answers more effective.

We also check if the winning games had easier to distinguish images (since
there is some randomness in the image set difficulty due to availability of
certain types of images in the 1500 pool of images we select from) than non-
winning games. We analyze the average VGG16 FC7 [24] distance of images in the
pool from GT image for winning and losing games for games with explanations
and without explanations. We observe that for games with explanations, winning
game image sets were just as difficult (or slightly more difficult) than non-
winning games. For games without explanations, winning games have much lesser
difficult image sets than non-winning games.

#### Impact of explanations as a function of independent explanation accuracy
ratings

Since it is not obvious what makes an explanation helpful, we ran a separate
AMT task to collect correctness ratings of the explanations given the image,
the question that was asked and the answer given by the VQA model (Figure 9).
The {image, question, answer} triplets were taken from the set of those used
prior in the ExAG games. The workers were asked to rate the explanations
according to the following options: “Exactly on-point” (Excellent), “Mostly
on-point” (Mostly on-point), “Indifferent” (Indifferent), “Somewhat off”
(Somewhat off) and “Completely Wrong” (Wrong). For all plots, the ratings were
averaged for a game and were binned into the original choices.

Figure 11 shows game performance as a function of explanation accuracy
ratings. We see that overall (panel 11a), combined (11b) and attention-based
(11c) correct explanations are not sufficient for improving game performance
(and hence being helpful), however, incorrect explanations can severely hurt
performance. Related QAS answers, as long as they are mostly on-point, help
game performance substantially (see Figure 11d).

#### Impact of explanations as a function of VQA answer accuracy

We also collect the answer accuracy ratings for the ExAG games using an
independent AMT study. The workers were required to rate the answer for the
question asked to the GT image as either “correct”, “somewhat correct” or
“completely wrong”. We analyzed the game performance as a function of average
VQA answer correctness in a game for game plays with explanations rated as
mostly “correct” (average rating above indifferent) and without explanations.

Interestingly, we see that having at least one correct explanation
significantly helps in games where the VQA answers are mostly incorrect as
shown in Figure 12. This effect was observed only when the attention and
related QA explanations were used together, and was not observed for
attention-alone and related QA-alone explanations. We also didn’t see any
significant improvement in performance when explanations were mostly
“incorrect” (average rating below indifferent) for mostly incorrect answer
games. Moreover, we observed a significant lack of single question incorrect
answer games when played with explanations as compared to no explanation
games. This seems to suggest that explanations helped workers realize the
answer may be wrong and encouraged them to ask more questions. However, it was
the correct explanation that helped them fish out the correct answer among the
mostly incorrect answers in the game. As expected, Without explanations (red
line of Figure 12), game performance degraded as VQA answers got less
accurate, suggesting that a player had no way of telling if an answer is wrong
or correct without having the benefit of inspecting both types of
explanations.

Figure 12: Explanations help when VQA answers are wrong. Without explanations
(red line), if the answer from the VQA is wrong, user performance drops
dramatically. However, at least a few good explanations (black line) help
reveal VQA answer correctness so that it can be taken into account. Hence,
game performance without explanations is much lower when answers are wrong
than with explanations. The black line is defined as games where there was at
least one explanation above indifferent.

##  5 Conclusion

We propose the ExAG game as an evaluation framework for explanations and show
that game performance correlates to the explanation helpfulness and
correctness. Using ExaG, we analyze the effectiveness of various explanations.

Our experiments provide empirical evidence that effective (rated very helpful
and above) explanations help improve performance on a human-machine
collaborative image guessing task whereas ineffective explanations hurt
performance (Figure 8). Moreover, since the self-rated explanation helpfulness
is not influenced by the (subsequent) outcome of the game, users can use their
insight into explanation helpfulness to decide when to trust them and include
them in decision making process vs. when not.

This is also supported by our observation that explanations help significantly
when machine predictions are noisy (Figure 12. Without explanations, users
blindly trust incorrect machine predictions, which hurts game performance.
However, with explanations, users ask follow-up questions and are able to fish
out the correct answer based on at least one good explanation.

Interestingly, we also observe that when image sets are difficult, players
tend to win more when playing with explanations rather than without.

We believe our ExaG framework can serve towards effectively evaluating
explanations. This will help designing more accurate and helpful explanations
that improve human-AI collaboration outcomes in terms of performance, trust
and satisfaction.

## Acknowledgements

This research was developed with funding from the Defense Advanced Research
Projects Agency (DARPA) under the Explainable AI (XAI) program. The views,
opinions and/or findings expressed are those of the author and should not be
interpreted as representing the official views or policies of the Department
of Defense or the U.S. Government.

## References

  * [1] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng.  TensorFlow: Large-scale machine learning on heterogeneous systems, 2015\.  Software available from tensorflow.org.
  * [2] S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. Lawrence Zitnick, and D. Parikh.  Vqa: Visual question answering.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2425–2433, 2015.
  * [3] A. Chandrasekaran, V. Prabhu, D. Yadav, P. Chattopadhyay, and D. Parikh.  Do explanations make vqa models more predictable to a human?  arXiv preprint arXiv:1810.12366, 2018.
  * [4] A. Chandrasekaran, D. Yadav, P. Chattopadhyay, V. Prabhu, and D. Parikh.  It takes two to tango: Towards theory of ai’s mind.  arXiv preprint arXiv:1704.00717, 2017.
  * [5] P. Chattopadhyay, D. Yadav, V. Prabhu, A. Chandrasekaran, A. Das, S. Lee, D. Batra, and D. Parikh.  Evaluating visual conversational agents via cooperative human-ai games.  In Proceedings of the Fifth AAAI Conference on Human Computation and Crowdsourcing (HCOMP), 2017.
  * [6] A. Das, H. Agrawal, L. Zitnick, D. Parikh, and D. Batra.  Human attention in visual question answering: Do humans and deep networks look at the same regions?  Computer Vision and Image Understanding, 163:90–100, 2017.
  * [7] A. Das, S. Kottur, J. M. Moura, S. Lee, and D. Batra.  Learning cooperative visual dialog agents with deep reinforcement learning.  arXiv preprint arXiv:1703.06585, 2017.
  * [8] H. de Vries, F. Strub, S. Chandar, O. Pietquin, H. Larochelle, and A. C. Courville.  Guesswhat?! visual object discovery through multi-modal dialogue.  CoRR, abs/1611.08481, 2016.
  * [9] R. Fong and A. Vedaldi.  Interpretable explanations of black boxes by meaningful perturbation.  In Proceedings of the International Conference on Computer Vision (ICCV), 2017.
  * [10] R. Fong and A. Vedaldi.  Net2Vec: Quantifying and explaining how concepts are encoded by filters in deep neural networks.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  * [11] K. He, G. Gkioxari, P. Dollár, and R. Girshick.  Mask r-cnn.  In Computer Vision (ICCV), 2017 IEEE International Conference on, pages 2980–2988. IEEE, 2017.
  * [12] L. A. Hendricks, Z. Akata, M. Rohrbach, J. Donahue, B. Schiele, and T. Darrell.  Generating visual explanations.  In European Conference on Computer Vision, pages 3–19. Springer, 2016.
  * [13] V. Kazemi and A. Elqursh.  Show, ask, attend, and answer: A strong baseline for visual question answering.  arXiv preprint arXiv:1704.03162, 2017.
  * [14] H. C. Lane, M. G. Core, M. Van Lent, S. Solomon, and D. Gomboc.  Explainable artificial intelligence for training and tutoring.  Technical report, UNIVERSITY OF SOUTHERN CALIFORNIA MARINA DEL REY CA INST FOR CREATIVE TECHNOLOGIES, 2005.
  * [15] D. Lewis.  Convention: A philosophical study.  John Wiley & Sons, 2008.
  * [16] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick.  Microsoft coco: Common objects in context.  In European conference on computer vision, pages 740–755. Springer, 2014.
  * [17] J. Lu, J. Yang, D. Batra, and D. Parikh.  Hierarchical question-image co-attention for visual question answering.  In Advances In Neural Information Processing Systems, pages 289–297, 2016.
  * [18] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean.  Distributed representations of words and phrases and their compositionality.  In Advances in neural information processing systems, pages 3111–3119, 2013.
  * [19] D. H. Park, L. A. Hendricks, Z. Akata, A. Rohrbach, B. Schiele, T. Darrell, and M. Rohrbach.  Multimodal explanations: Justifying decisions and pointing to the evidence.  arXiv preprint arXiv:1802.08129, 2018.
  * [20] J. Pennington, R. Socher, and C. Manning.  Glove: Global vectors for word representation.  In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543, 2014.
  * [21] A. Ray, G. T. Burachas, K. Sikka, A. Roy, A. Ziskind, Y. Yao, and A. Divakaran.  Make up your mind: Towards consistent answer predictions in vqa models.
  * [22] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  See https://arxiv. org/abs/1610.02391 v3, 7(8), 2016.
  * [23] E. H. Shortliffe and B. G. Buchanan.  A model of inexact reasoning in medicine.  Rule-based expert systems, pages 233–262, 1984.
  * [24] K. Simonyan and A. Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [25] C. Szegedy, S. Ioffe, V. Vanhoucke, and A. A. Alemi.  Inception-v4, inception-resnet and the impact of residual connections on learning.  In AAAI, volume 4, page 12, 2017.
  * [26] D. Teney, P. Anderson, X. He, and A. v. d. Hengel.  Tips and tricks for visual question answering: Learnings from the 2017 challenge.  arXiv preprint arXiv:1708.02711, 2017.
  * [27] M. Van Lent, W. Fisher, and M. Mancuso.  An explainable artificial intelligence system for small-unit tactical behavior.  In Proceedings of the National Conference on Artificial Intelligence, pages 900–907. Menlo Park, CA; Cambridge, MA; London; AAAI Press; MIT Press; 1999, 2004.
  * [28] H. Xu and K. Saenko.  Ask, attend and answer: Exploring question-guided spatial attention for visual question answering.  In European Conference on Computer Vision, pages 451–466. Springer, 2016.
  * [29] Yu Jiang*, Vivek Natarajan*, Xinlei Chen*, M. Rohrbach, D. Batra, and D. Parikh.  Pythia v0.1: the winning entry to the vqa challenge 2018.  arXiv preprint arXiv:1807.09956, 2018.
  * [30] M. D. Zeiler and R. Fergus.  Visualizing and understanding convolutional networks.  In European conference on computer vision, pages 818–833. Springer, 2014.

## Supplementary

### VQA Architecture Details

Figure 13: This image shows two game plays without explanations (each row is a
game-play example). When the VQA is fairly accurate, a user is easily able to
pick out the correct image as shown in row 1. However, in the second row, the
VQA answer leads to a reasonably wrong selection by the user since the fifth
image is the image with the most prominent umbrella. Figure 14: This image
shows a game with both explanations. Even though the most prominent toilet is
in the fourth image, note how the explanations make it clear that the fifth
image could also be the GT image since it suggests that the machine saw a
toilet there as well and that it probably mis-detected the bathtub or sink for
a toilet. This hints the user to ask follow-up questions like “is there a
sink?” to finally select the correct image.

We use Tensorflow [1] for all our implementations. The network we use for Game
Setting 1 is shown in Figure 2. Outlined below are the network details:

  * •

Input Image - size $448\times 448$. We center crop all images to the mentioned
size during training and reshape during evaluation. We encode the image using
a ResNet 161 [25] network.

  * •

Question Input - Each question word is encoded using the Glove [20] 300
dimensional embeddings before feeding into an LSTM word by word. We take the
final 512 dimensional LSTM features as the question feature. Embeddings are
fine-tuned.

  * •

Attention- For Resnet feature attention, We tile the question features (512)
into $14\times 14\times 512$ and concatenate with $14\times 14\times 2048$
image features. Attention predicts a set of weights in the shape of $14\times
14$ using a 2-D convolutional layer.

  * •

Answer classifier- We concatenate weighted flattened ResNet features and
question features and pass it through a fully connected layer to get 3000
answer logits. We compute a softmax to get probabilities.

For Game Setting 2, we use Network 2 as shown in Figure 3. Details are as
follows:

  * •

Input Image - For ResNet161 [25] image features, we center crop images to
$448\times 448$ similar to Network 1 to get $14\times 14\times 2048$
dimensional features. We also use the Region Proposal Network from Mask RCNN
[11] to generate 100 object proposals per image. The input image size to Mask
RCNN is $1024\times 1024$ and the images are re-sized without cropping. We
pool 1024 dimensional features from each of the 100 proposal boxes.

  * •

Question Input - Each question word is encoded using the Glove [20] 300
dimensional embeddings before feeding into an LSTM similar to network 1. The
embeddings are fine-tuned.

  * •

Attention- We have two attention modules - one for attending to the ResNet
features (same as Network 1) and one for attending to the 100 object proposals
in the image (object attention). Resnet feature attention is same as setting
1. For object proposal attention, we concatenate the question features (512)
to each 1024 image feature for the 100 proposals. Attention predicts a weight
of shape $100\times 1024$ using a 1-D convolutional layer.

  * •

Answer classifier- We concatenate weighted flattened ResNet features, averaged
weighted object features and question features and pass it through a fully
connected layer to get 3000 answer logits. We compute a softmax to get
probabilities.

### Qualitative Results

Figure 13 and Figure 14 show qualitative examples of plays without and with
explanations and how explanations may help in choosing the correct image more
often when VQA answers are noisy.