# Optimal time-entropy bounds and speed limits for Brownian thermal shortcuts

Luís Barbosa Pires  University of Strasbourg and CNRS, CESQ and ISIS, UMR
7006, F-67000 Strasbourg, France  Rémi Goerlich  University of Strasbourg and
CNRS, Institut de Physique et Chimie des Matériaux de Strasbourg, UMR 7504,
F-67000 Strasbourg, France  University of Strasbourg and CNRS, CESQ and ISIS,
UMR 7006, F-67000 Strasbourg, France  Arthur Luna da Fonseca  Instituto de
Fisica, Universidade Federal do Rio de Janeiro, Caixa Postal 68528, Rio de
Janeiro, RJ, 21941-972, Brazil  University of Strasbourg and CNRS, CESQ and
ISIS, UMR 7006, F-67000 Strasbourg, France  Maxime Debiossac  Vienna Center
for Quantum Science and Technology, Faculty of Physics, University of Vienna,
A-1090 Vienna, Austria  Paul-Antoine Hervieux  University of Strasbourg and
CNRS, Institut de Physique et Chimie des Matériaux de Strasbourg, UMR 7504,
F-67000 Strasbourg, France  Giovanni Manfredi
<EMAIL_ADDRESS>University of Strasbourg and CNRS, Institut
de Physique et Chimie des Matériaux de Strasbourg, UMR 7504, F-67000
Strasbourg, France  Cyriaque Genet<EMAIL_ADDRESS>University of Strasbourg
and CNRS, CESQ and ISIS, UMR 7006, F-67000 Strasbourg, France

###### Abstract

By controlling in real-time the variance of the radiation pressure exerted on
an optically trapped microsphere, we engineer temperature protocols that
shortcut thermal relaxation when transferring the microsphere from one thermal
equilibrium state to an other. We identify the entropic footprint of such
accelerated transfers and derive optimal temperature protocols that either
minimize the production of entropy for a given transfer duration or accelerate
as much as possible the transfer for a given entropic cost. Optimizing the
trade-off yields time-entropy bounds that put speed limits on thermalization
schemes. We further show how optimization expands the possibilities for
accelerating Brownian thermalization down to its fundamental limits. Our
approach paves the way for the design of optimized, finite-time thermodynamic
cycles at the mesoscale. It also offers a platform for investigating
fundamental connections between information geometry and finite-time
processes.

The time needed for a body to thermalize with its environment is a natural
constraint for operating many physical systems and devices. Controlling
thermalization has emerged as one salient challenge at meso and nanoscales
scales [1, 2, 3, 4, 5]. At such scales, the methods of stochastic
thermodynamics have proven their efficiency, capable of extending the concepts
of work, heat and entropy to single, fluctuating systems [6, 7].
Experimentally, new strategies have recently been implemented on optically
trapped Brownian particles to emulate effective, and thereby controllable,
thermal baths [8, 9, 10, 11, 12]. The fine control of the time-dependence of
effective temperatures has led to the definition of thermal protocols and
optimized cycles [13, 14, 15]. Exploiting finite-time thermodynamics, these
strategies have also provided the means to circumvent natural thermalization
by proposing accelerated paths that a Brownian system can be forced to follow
[10, 16, 17, 18, 19, 20, 21]. Such means form a major topic of current
research in the realm of shortcuts to adiabaticity [22, 23].

Obviously, speeding-up transitions from one equilibrium state to another
demands to follow non-equilibrium paths that have a thermodynamic cost. Once
such cost evaluated, the design of protocols that optimize the mutually
exclusive relation between the rate of acceleration and the energetic expense
should be possible. There is a variety of approaches proposed for evaluating
that energetic expense [24, 25, 26, 27, 28, 29, 30], but the challenge remains
to identify the proper one which makes it possible to treat duration and cost
on an equal footing, the prerequisite for this optimization [31, 32, 33].

In this Letter, we set up a bath engineering strategy involving radiation
pressure to directly control the kinetic temperature of an optically trapped,
overdamped, Brownian microsphere [34]. This control allows us to impose abrupt
transfers from one to another equilibrium states, either increasing or
decreasing the temperature down to a minimum set by room temperature $T_{R}$.
Such step-like transfers are followed by thermal relaxations measured
precisely through the diffusive dynamics of the microsphere inside the
harmonic optical trap. Our strategy gives the possibility to accelerate such
thermal relaxation processes by imposing an overshoot in temperature during
the transfer. This leads us to extend to isochoric transitions the Engineered
Swift Equilibration (ESE) processes developed so far for isothermal
transitions [35]. We show how thermal ESE protocols – hereafter named ThESE
protocols – do accelerate thermalization, and demonstrate experimentally the
shortening of the duration of initial-to-final thermal equilibrium transfers.
We further quantify the thermodynamic cost of this acceleration in terms of
entropy production.

The identification of the entropic cost of a thermal shortcut brings us, in
the context of harmonic trapping, to a class of optimal thermal protocols,
defined as those protocols that speed-up thermalization while minimizing the
associated production of entropy. The trade-off involved in our optimization
procedure sets time-entropy bounds that put speed limits on physically
realizable thermal shortcuts, in a remarkable asymmetry between heating and
cooling protocols. We also demonstrate that optimal cooling gives access to
higher acceleration rates that are unreachable using standard overshooting,
ThESE-like protocols. These results are finally discussed from an energetic
viewpoint for the three families (step-like, ThESE and optimized) of state-to-
state transitions, which clarifies the different contributions to the global
in-take of heat by the trapped microsphere under the action of the fluctuating
radiation pressure. We also stress that our optimization-under-cost constraint
leads to results that are different from the thermal brachistochrones recently
proposed in [26].

Our experiment consists of a single microsphere trapped in an optical tweezer
and evolving in a harmonic potential [36, 37]. The microsphere diffuses in
water with a Stokes drag $\gamma=2.695\times 10^{-8}$ kg/s at room temperature
$T_{R}=293K$. The trap is characterized by a stiffness $\kappa=13.1\pm 0.2$
fN/nm and the overdamped diffusion dynamics by a relaxation time
$\tau=\gamma/\kappa=2.06\pm 0.04$ ms. As described in detail in Appendices A
and B, an additional radiation pressure is exerted on the sphere by a pushing
laser whose intensity $I(t)=I_{0}+\delta I(t)$ is digitally controlled over
time by an acousto-optic modulator [34]. When $\delta I(t)$ is random (white
noise spectrum), this radiation pressure increases the motional variance of
the center-of-mass motion of the sphere along the optical axis of the trap. By
building a statistical ensemble $\\{j\\}$ of $1.7\times 10^{4}$ diffusing
trajectories $x_{j}(t)$, we extract an ensemble average variance $s(t)$ in
direct relation with the pushing laser intensity variance $\langle\delta
I^{2}(t)\rangle$. This random forcing of the microsphere can be interpreted as
emulating an effective thermal bath whose temperature $T(t)$ can be set
instantaneously in strict relation with the intensity variance with
$T(t)\propto\langle\delta I^{2}(t)\rangle$. The mechanical response of the
microsphere is measured through the time-evolution of $s(t)$ according to –see
Appendix C:

$\frac{ds(t)}{dt}=\frac{2}{\tau}\left(\frac{k_{\rm
B}T(t)}{\kappa}-s(t)\right).$ (1)

The effective nature of $T(t)$ implies that thermal changes impact the
diffusion coefficient simply like $D(t)=k_{\rm B}T(t)/\gamma$ and the system
relaxation time $\tau$ remains constant. By thus transforming temperature into
an external control parameter $T(t)$, the crucial asset of our bath
engineering strategy is the possibility to perform specific temperature
protocols that can be arbitrarily fast from one initial $T_{R}+T_{i}$ to
another final target temperature $T_{R}+T_{f}$. As discussed below, this opens
rich analogies with recent works that have demonstrated how time-dependent
optical trap stiffness protocols $\kappa(t)$ can lead to shortcut, engineer
and even optimize state-to-state isothermal processes [35, 38, 33].

Figure 1: Upper panel: Time evolution $T(t)$ plotted for three cooling
protocols connecting two thermal equilibrium states, one initial at
$T_{R}+T_{i}$ and one final at $T_{R}+T_{f}$ where $T_{R}=300$ K corresponds
to room temperature and $T_{R}+T_{i}=2200$ K, $T_{R}+T_{f}=1200$ K to the
target temperatures fixed by the laser fluctuation spectrum. The orange curve
corresponds to a step-like protocol (STEP) with a simple temperature
discontinuous quench, the blue curve to the ThESE protocol through which
$T_{i}$ and $T_{f}$ are connected by a third degree polynomial, the green
curve to the optimal protocol (Opt) with its two discontinuities ensuring the
connection between the initially hot and finally cold equilibrium states.
Lower panel: Measured time evolutions (data points with same color coding as
in the upper panel) of the motional variances $s(t)$ induced by the different
thermal protocols starting at $t_{i}/\tau=0$. The variances for the ThESE and
Opt shortcuts reach equilibrium at a rate $\Delta t/\tau=0.6$, as indicated by
the vertical line. The shaded areas give the experimental errors at a $95\%$
confidence interval for $s(t)$. The analytical solutions for $s(t)$ for the
three protocols, calculated in Appendix E from Eq. (1), correspond to the
continuous lines drawn on the experimental data points. Inset: Levels of
produced entropy $\Sigma_{\text{gen}}$ associated to each of the three
protocols and evaluated using Eq. (2). Error bars for $\Sigma_{\text{gen}}$
correspond to the combination of the uncertainty in $s(t)$, stiffness and
temperature (see more details in Appendix H).

Let us start by implementing a sudden, step-like change $\Delta T=T_{f}-T_{i}$
between an initial $T_{R}+T_{i}$ to a final $T_{R}+T_{f}$ temperature. This
STEP protocol is described in the upper panel of Fig. 1. It produces a
transient response of the variance $s(t)$ that we measure and plot in the
lower panel of Fig. 1. As observed in excellent agreement with Eq. (1), the
variance relaxes towards the new thermal equilibrium state with a relaxation
time $2\tau$. Such a relaxation corresponds to the definition of natural
thermalization, in which the system is left free to evolve towards a new
equilibrium state. We now show that it is possible to impose a temperature
protocol that displays a much shorter thermalization time for the same $\Delta
T$ change. To do so, we extend to temperature the class of ESE isothermal
protocols presented in [35] with a polynomial temperature overshoot evaluated
in Appendix E.2, imposing stationary thermal equilibrium $\kappa
s(t_{i,f})=k_{\rm B}(T_{R}+T_{i,f})$ and $\dot{s}(t_{i,f})=0$ at the initial
and final steps of the process. The corresponding $T(t)$ protocol is plotted
in the upper panel of Fig. 1 for a chosen transfer duration $\Delta
t=t_{f}-t_{i}=1.23$ ms imposed to be shorter than $2\tau$ with a ratio $\Delta
t/\tau=0.6$. It is implemented experimentally and we measure in the lower
panel of Fig. 1 the time evolution of the system’s variance $s(t)$ in
excellent agreement with the theory.

The central question for such accelerated thermalization protocols remains
their possible optimization with respect to a well-identified footprint. In
the case of isothermal stiffness protocols $\kappa(t)$, the thermodynamic cost
of acceleration was evaluated through the associated work expense and the
minimization procedure designed accordingly [31, 32, 33]. Temperature
protocols, in contrast, are entropic by nature and have been recently
characterized using the concept of thermal (entropic) work [5]. This entropic
nature clearly appears when interpreting Eq. (1) thermodynamically: whenever
the temperature changes faster than $\tau$, the system will evolve along an
irreversible, non-equilibrium process in which the instantaneous variance
$s(t)$ will be different from the one expected by equipartition. The
difference between $s(t)$ and $k_{\rm B}T(t)/\kappa$ thus measures the
deviation from a reversible process and, as such, is associated to a given
production of entropy that we now evaluate.

For our experiments, we define the system’s stochastic entropy $\sigma_{\rm
sys}(x_{j}(t),T(t))=-k_{\rm B}\ln p(x_{j}(t),T(t))$ [7] from an extension of
the Boltzmann probability density $p(x_{j}(t),T(t))=\sqrt{\kappa/2\pi k_{\rm
B}T(t)}\exp(-\kappa x^{2}_{j}(t)/(2k_{\rm B}T(t)))$ to non-equilibrium
processes that connect two equilibrium states. Using this definition, the
infinitesimal variation of the system’s entropy is evaluated as $d\sigma_{\rm
sys}(x_{j}(t),T(t))=\kappa x_{i}dx_{j}/T(t)+(k_{\rm B}T(t)-\kappa
x_{j}^{2})dT/(2T^{2}(t))$. The first term involves the quantity of heat
$dq=-\kappa x_{i}dx_{j}$ associated with the change of internal of energy of
the system. It corresponds to (the opposite of) the variation of the entropy
of the medium ${\mkern 3.0mu\mathchar 22\relax\mkern-12.0mud}\sigma_{\rm
med}=dq/T(t)$ where we use the ${\mkern 3.0mu\mathchar
22\relax\mkern-12.0mud}$ notation for non-exact differentials. The second
term, written as $d\sigma_{\rm sys}+{\mkern 3.0mu\mathchar
22\relax\mkern-12.0mud}\sigma_{\rm med}$, thus gives the infinitesimal amount
of total entropy generated along the elementary path $dT$ –see Appendix D.

The generated total entropy, once ensemble averaged and cumulated from the
initial $t_{i}$ to a given time $t$ of the isochoric transformation

$\Sigma_{\rm
gen}(t)=\frac{1}{2}\int_{t_{i}}^{t}\frac{\dot{T}(\zeta)}{T^{2}(\zeta)}\left(k_{\rm
B}T(\zeta)-\kappa s(\zeta)\right)d\zeta,$ (2)

directly involves the non-equilibrium nature of the transformation imprinted
in the difference $k_{\rm B}T(t)-\kappa s(t)$ between the measured variance
and equipartition. It therefore corresponds to the cumulated entropy produced
along the irreversible transition and constitutes the entropic footprint of a
finite-time isochoric process. For the entire STEP and ThESE protocols, in
which $t\rightarrow\infty$ and $t=t_{f}$ respectively, the total entropy
production can be easily calculated using Eq. (2), and the results are plotted
in the inset of Fig. 1. When compared, these values reveal in a striking
manner the entropic cost of thermal acceleration with $\Sigma_{\rm gen}^{\rm
ThESE}>\Sigma_{\rm gen}^{\rm STEP}$.

Our analysis now gives the possibility to derive the actual temporal profile
of an optimal protocol that minimizes this entropic cost for a given choice of
transfer duration $\Delta t=t_{f}-t_{i}$ from one equilibrium state at
$T_{R}+T_{i}$ to another at $T_{R}+T_{f}$. In close relation with our previous
work [33], we express the transfer duration as a functional of the variance
according to Eq. (1) and integrate by parts the generated entropy (2) –see
Appendix E.3– to build a functional

$J[T(s)]=\int_{s_{i}}^{s_{f}}\left(\frac{\gamma}{k_{\rm
B}T(s)-s\kappa}-\lambda\frac{\kappa}{k_{\rm B}T(s)}\right)\textrm{d}s.$ (3)

This combines on an equal footing the transfer duration and the corresponding
generation of entropy with a Lagrange multiplier $\lambda$ to regulate the
trade-off between the two quantities. The optimization procedure consists in
searching for the paths in the $[s,T(s)]$ space that minimize $J[T(s)]$ while
keeping the same initial and final equilibrium conditions imposed by the
equipartition theorem, just like for the ThESE protocol. As explained in
Appendix E, the procedure yields two families of optimized thermal protocols
$T_{\rm heat/cool}(s)$ associated respectively to heating $T_{i}<T_{f}$ and
cooling $T_{i}>T_{f}$. We emphasize that the $T_{\rm heat/cool}(s)$ protocol
does not satisfy thermal equilibrium at both initial and final times and must
be therefore supplemented by two discontinuous transitions, just like in the
case of optimal isothermal processes [31, 33]. During the interval $\Delta t$,
those two solutions correspond to an exponential evolution of the variance
with $s_{\rm opt}(t)=s_{i}((T_{R}+T_{f})/(T_{R}+T_{i}))^{(t-t_{i})/\Delta t}$
. It is remarkable that both $T_{\rm heat/cool}(s)$ protocols can be described
with a single expression

$T_{\rm opt}(t)=\frac{\kappa s(t)}{k_{\rm B}}\left(1+\frac{\tau}{k_{\rm
B}}\frac{\Delta\Sigma_{\rm sys}}{\Delta t}\right)$ (4)

with $\Delta\Sigma_{\rm sys}=k_{\rm B}\ln[(T_{R}+T_{f})/(T_{R}+T_{i})]/2$ the
protocol-independent total variation in the system’s entropy. This expression
is plotted in the upper panel of Fig. 1 for an optimal cooling protocol and
with the same shortening rate $\Delta t/\tau=0.6$ as the ThESE protocol
discussed above. As one expected important result of our work, $\Sigma_{\rm
gen}^{\rm ThESE}>\Sigma_{\rm gen}^{\rm opt}$.

The optimal protocol given by Eq. (4) together with $s_{\rm opt}(t)$ injected
into Eq. (2) lead to evaluate the minimal entropy produced through an
isochoric transformation of duration $\Delta t$ as

$\Sigma_{\rm min}=\frac{\Delta\Sigma_{\rm sys}}{1+\frac{k_{\rm
B}}{\tau}\frac{\Delta t}{\Delta\Sigma_{\rm sys}}},$ (5)

an expression valid both for cooling and heating protocols but with different
consequences, as discussed below.

In Fig. 2, we first plot Eq. (5) for cooling (upper panel) and heating (lower
panel) optimal protocols (solid black lines). The curves draw exclusion
regions for entropy production that correspond to the minimal amount of
entropy that can be generated in an isochore for a given $\Delta t$: they thus
correspond to optimal time-entropy bounds. Our experimental results obtained
for different optimal cooling and heating protocols $T_{\rm opt}(t)$ injected
within our optical trap (same set of temperatures but different transfer
durations) all precisely fall on the expected bounds.

For a cooling process with $\Delta\Sigma_{\rm sys}<0$, Eq. (5) also puts an
asymptotic limit to the transfer rate with a minimal transfer duration of
$\Delta t_{\rm min}/\tau=-\Delta\Sigma_{\rm sys}/k_{\rm B}$. These bounds on
the dynamical evolution of our system, extracted from the trade-off involved
in the optimization procedure between the transfer duration and the production
of entropy, must be considered as true speed limits on the state-to-state
connection [39]. They are directly associated with a divergence in the
entropic cost as clearly seen experimentally in Fig. 2 for the shortest
transfer rate that we probed (vertical dashed line). This limit in the cooling
acceleration is directly related to the fact that the lowest temperature
$T_{\rm min}=T_{\rm opt}(t_{f}^{-})=T_{f}(1+\frac{\tau}{k_{\rm
B}}\frac{\Delta\Sigma_{\rm sys}}{\Delta t})$ which the optimal protocol passes
through, cannot be smaller than $0$ K, a temperature limit reached when
$\Delta t_{\rm min}/\tau=-\frac{\Delta\Sigma_{\rm sys}}{k_{\rm B}}=0.3$ for
our $(T_{i},T_{f})$ choice. However, experimentally we necessarily have
$T_{\rm min}\geq T_{R}$ and for the case presented in Fig. 2, this implies
that the shortest transfer rate reachable is $\Delta t_{\rm
min}/\tau=\frac{\Delta\Sigma_{\rm sys}}{k_{\rm B}(1-T_{R}/T_{f})}\simeq 0.4$.

Room temperature obviously bounds from below all overshoot temperatures that
can be physically hit. This leads to an interesting consequence when comparing
optimal and ThESE cooling protocols for identical shortening rates and target
temperatures $T_{i}>T_{f}$. Because the overshoot temperature for the ThESE
protocol is necessarily lower than $T_{\rm min}$ for the optimal protocol for
a given $\Delta t$ –see Fig. 1 (upper panel)– the room temperature bound is
reached by the ThESE protocol before the optimal one. More precisely, the
ThESE protocol cannot accelerate cooling beyond $\Delta t/\tau=0.6$, while
remarkably and as perfectly measured, the optimal protocols can still have
access to stronger acceleration rates with ratios between $\Delta t/\tau=0.6$
and $0.4$ that remain available experimentally. This important result reveals
another, yet unexpected, thermodynamic advantage of optimization giving access
to time-entropy regions that are simply forbidden to non-optimized protocols.

Figure 2: Minimal time-entropy bound (solid black lines) corresponding to an
optimal $(T_{R}+T_{i}=2200{\rm K},T_{R}+T_{f}=1200{\rm K})$ cooling process
(upper panel) and heating protocols performed between $T_{R}+T_{i}=350$ K to
$T_{R}+T_{f}=1100$ K (lower panel). The gray hatched region are forbidden to
any acceleration method. Experimental measurements for optimal protocols are
shown with green open circles and with blue open squares for ThESE protocols.
Error bars correspond to experimental errors propagated through Eq. (2) at a
$95\%$ confidence level. The fundamental limit put on ThESE cooling protocols
set at $\Delta t/\tau=0.6$ for the chosen experimental parameters is depicted
as a second exclusion region (blue hatched region) for such overshooting
temperature protocols.

Contrasting with cooling, optimal heating protocols are not constrained by any
fundamental limit (Fig. 2, lower panel). With $\Delta\Sigma_{\rm sys}>0$ in
Eq. (5), the production of entropy does not diverge and the system can be
forced to thermalize arbitrarily fast. The optimal time-entropy bound for
heating protocols is plotted in Fig. 2 together with the experimental
measurements obtained when implementing heating ThESE and optimal protocols.

We finally measure the instantaneous, ensemble average, heat generated by the
action of the radiation pressure ${\mkern 3.0mu\mathchar
22\relax\mkern-12.0mud}Q_{gen}=-T(t)d\Sigma_{\rm gen}$ and transferred to the
microsphere while evolving from the initial state $s(t_{i})$ to one non-
equilibrium state $s(t)$ 111The minus sign in this convention means that for a
generation of entropy $d\Sigma_{\rm gen}>0$, the energy flows from medium to
the system and ${\mkern 3.0mu\mathchar 22\relax\mkern-12.0mud}Q_{gen}<0$,
consistent with the standard convention of stochastic thermodynamics [44]. The
time-dependent cumulative in-take of heat for isochores can be evaluated
directly from Eq. (2) as $Q_{\rm
gen}(t)=-\int_{t_{i}}^{t}dt^{\prime}\dot{T}(t^{\prime})[k_{\rm
B}T(t^{\prime})-\kappa s(t^{\prime})]/(2T(t^{\prime}))$. This quantity of heat
is plotted in Fig. 3 for the three families of protocols studied here –STEP,
ThESE, and optimal– fixing a shortening ratio of $0.6$ for the ThESE and the
optimal protocols. The time evolution and the final amount of $Q_{\rm gen}(t)$
strongly depend on the type of protocol. The energetic cost of the STEP
protocol is relatively low, but it requires a long thermalization time. When
comparing the other two protocols that have the same $\Delta t$, it is clear
that the optimal solution mitigates the energetic cost of the transfer when
compared to the ThESE protocol.

Figure 3: Ensemble average, time-dependent cumulative generated heats $Q_{\rm
gen}$ for a STEP (orange), a ThESE (blue) and an optimal protocol (green), all
set at $t_{i}=0$ ms. Same color coding as in Fig. 1 and same shaded areas
associated with experimental errors at a $95\%$ confidence level, including
calibration and temperature uncertainties evaluated by the expression of
$Q_{\rm gen}(t)$.

In conclusion, we have used a fluctuating, white-noise, radiation pressure to
emulate temperature protocols applied to an optically trapped microsphere and
to extend the concept of engineered swift equilibration to thermal protocols.
A central result was to identify the entropic cost of such non-equilibrium
protocols. The trade-off between the state-to-state transfer duration and the
entropic cost led us to the design of optimal cooling and heating protocols.
We identified minimal time-entropy bounds for all possible shortcut strategies
in harmonic potentials and derived speed limits on the transfer rates. An
energetic analysis showed in addition how optimization yields the best
thermodynamic compromise between acceleration and cost. This optimization is
important in the context of thermodynamic cycles and Brownian heat engines,
and bears a fundamental appeal considering that the entropic cost can be
described as a thermodynamic length [39]. From this perspective, our optimal
cooling and heating shortcuts $T_{\rm opt}(t)$ correspond to geodesics within
an information geometry viewpoint that draws fascinating connections yet to be
further explored [41, 42].

## Acknowledgments

This work is part of the Interdisciplinary Thematic Institute QMat of the
University of Strasbourg, CNRS, and Inserm. It was supported by the following
pro- grams: IdEx Unistra (ANR-10-IDEX-0002), SFRI STRATUS project
(ANR-20-SFRI-0012), and USIAS (ANR-10-IDEX- 0002-02), under the framework of
the French Investments for the Future Program.

##  Appendix A Experimental setup

Figure 4: Experimental set-up. The trapping laser beam –laser $1$– is focused
inside the fluidic cell by objective $1$ ($60\times$, NA = $1.2$, water
immersion). The instantaneous position of the trapped sphere is probed using
laser $2$ whose scattered intensity is recorded using a P.I.N. photodiode.
Laser $3$ exerts radiation pressure on the trapped sphere. Both lasers $2$ and
$3$ are sent to the cell from the opposite side with respect to laser $1$
using objective $2$ ($60\times$, NA = $0.7$). It is important that neither
laser $2$ nor laser $3$ induces any spurious gradient force inside the trap.
For that, they are sent into objective $2$ in underfilling conditions. Laser
$3$ passes through an acousto-optic modulator (AOM) –using the front lens of
the telescope stage to adjust laser waist and divergence to the AOM– that
enables driving in real-time its intensity (mean and fluctuations).

Our experimental setup consists in optically trapping a single polystyrene
microsphere (Duke Scientific Corp. $3\ \mu$\mathrm{m}$$ diameter) in ultra
pure water by a Gaussian beam –laser $1$ on Fig. 4 OBIS Coherent, CW $785\
$\mathrm{n}\mathrm{m}$$, $110\ $\mathrm{m}$W$. The laser beam is expanded to
overfill a high numerical aperture (NA) water immersion objective –objective
$1$ Nikon Plan Apochomat $60\times$, NA = $1.2$– that focuses the beam on a
$18\ $\mu\mathrm{L}$$ fluidic cell (glass slide and coverslip separeted by a
spacer -Grace Bio-Labs) of thickness $120\ $\mu\mathrm{m}$$. The microsphere
solution is diluted to $0.5\times 10^{-4}$ % to ensure that only single
spheres are trapped for all experiments. This is verified by looking to a
bright field image produced by an additional laser – not shown – focused on
the back focal plane of a low NA objective –objective $2$ Nikon Plan-fluo
extra large working distance $60\times$, NA=$0.7$.

The instantaneous position of the trapped sphere is recorded from the signal
scattered off the trapped sphere of a diode laser beam –laser $2$ Thorlabs
HL6323MG CW $639\ $\mathrm{n}\mathrm{m}$$, $30\ $\mathrm{m}\mathrm{W}$$–
focused and injected into the trap by Objective $2$. The scattered light is
coolected by objective $1$ and directed toward a P.I.N. photodiode (Thorlabs,
model Det100A2). The signal, recorded in volts, is sent to a low noise
amplifier (Stanford Research, SR560) that removes through a $0.3\
$\mathrm{H}\mathrm{z}$$ high-pass filter the DC component of the signal. A
$100\ $\mathrm{k}\mathrm{H}\mathrm{z}$$ low-pas filter is used in addition, to
prevent aliasing. Both filters are set at $6\
$\mathrm{d}\mathrm{B}\mathrm{/}\mathrm{o}\mathrm{c}\mathrm{t}$$. The signal
acquisition is finally done using an analog-to-digital card (National
Instrument, PCI-6251) with an acquisition rate of $32768\
$\mathrm{H}\mathrm{z}$$.

A third laser beam –laser $3$ Ti:Sapphire Spectra Physics 3900S– adjusted to
$800\ $\mathrm{n}\mathrm{m}$$ is used to exert fluctuating radiation pressure
on the trapped sphere and emulate a secondary thermal bath. To do so, the
laser is sent through an acousto-optic modulator (AOM - $3200$ S Gooch &
Housego) that modulates in real-time the intensity of the first-order
diffracted beam. This beam is transmitted through Objective $2$ in
underfilling conditions to couple to the trapped microsphere. The AOM output
is measured by another P.I.N. photodiode – not shown – (Thorlabs, model
Det100A/M), with the same configuration of the previous one. The control of
the AOM is done using a digital-to-analog card (National Instruments
PXRe-6738) with a generation rate of $20\ $\mathrm{k}\mathrm{H}\mathrm{z}$$.

The divergence of laser $3$ is important to control to have an efficient
radiation pressure coupling between this beam and the trapped microsphere.
Since the focal plane of objective $2$ is first positioned to have a high
signal-to-noise ratio for the detection of the instantaneous motion of the
trapped sphere using laser $2$, a telescope is used to fine-tune laser $3$’s
divergence so that both couplings are efficiently maintained through the
common Objective $2$ configuration.

##  Appendix B Temperature calibration

The intensity of laser $3$ is transformed, using the AOM, into a fluctuating
signal with a white-noise spectrum. Exerting a white-noise fluctuating
radiation pressure on the trapped microsphere, laser $3$ thus modifies the
position variance $\langle x_{j}^{2}\rangle=s(t)$ of the microsphere, as
measured over multiple trajectories $j$ ($1,7\times 10^{4}$ trajectories
forming the experimental ensemble). More precisely, the laser instantaneous
intensity is defined as $I_{j}=I_{0}+\delta I_{j}$ for one trajectory.
Ensemble averaging over the set ${j}$ of trajectories, we write $\langle
I_{j}\rangle=I_{0}$ and $\langle(I_{j}-I_{0})^{2}\rangle=\langle\delta
I_{j}^{2}\rangle\propto\Delta T$. This radiation pressure emulates an
effective thermal bath whose temperature can be changed instantaneously. The
microsphere then thermalize within this effective, secondary bath, with
variance values that depart from the equipartition set by room temperature, as
schematized in Fig. 5.

Figure 5: A pushing laser, in orange on the left-hand side of the schematics
(laser $3$ in Fig. 4), exerts unidirectional radiation pressure on the
harmonically trapped microsphere and adds noise to the diffusing motion. As
explained in the main text, one generates a step-like change in effective
temperatures (here from hot –red– to cold –blue) by modifying abruptly the
variance of the laser white-noise intensity fluctuations at $t_{i}$. In
contrast, the microsphere responds mechanically to this abrupt change by a
transient evolution with a natural relaxation time $2\tau$.

We make sure that the power spectrum of laser $3$ remains white over
frequencies much larger than the roll-off frequency of the trap
$f_{c}=2\pi/\tau$, and that the additional fluctuations imposed on the sphere
do not affect the trap stiffness. This condition is met for sufficiently low
intensities –$\sim 2$ mW– so that, transmitted through objective $2$, laser
$3$ does not impact the roll-off frequency of the Lorentzian position power
spectrum (PSD) of the sphere set by the optical trap induced by Laser $1$ –see
Fig. 6. In the absence of fluctuations in the radiation pressure
($\langle\delta I^{2}_{j}\rangle=0$), the natural PSD inside an optical trap
of stiffness $\kappa$ is defined around the roll-off frequency
$f_{c}=\kappa/(2\pi\gamma)$ as

$S(f)=\frac{1}{2\pi^{2}\gamma}\frac{k_{B}T_{\rm R}}{f^{2}+f^{2}_{c}},$ (6)

where the viscosity of the water is taken at room temperature $T_{\rm R}=293$
K with $\eta(T_{R})=9.532\times 10^{-4}\ \textrm{Pa}\times\textrm{s}$, the
Stokes drag $\gamma=6\pi\eta r$ evaluated for a sphere radius $r=1.5\
\mu\textrm{m}$. We calibrate the recorded voltage by fitting Eq. (6) in the
case of $\langle\delta I^{2}_{j}\rangle=0$, following the standard procedure
given in [43, 33].

The effective temperature associated with laser $3$ white-noise intensity
fluctuations is measured from the evolution of the PSD –see Fig. 6– accounting
for the limited operational bandwidth of the AOM. The temperature calibration
exploits the linearity between the intensity variance of laser $3$ and the
position variance of the trapped microsphere $s(t)=k_{\rm B}T_{\rm
R}/\kappa+\alpha k_{\rm B}\langle\delta I^{2}_{j}(t)\rangle/\kappa$, for
stationary conditions (or when the time dependence of $\langle\delta
I^{2}_{j}(t)\rangle$ is slow enough to consider that $s(t)$ evolve as a
succession of equilibrium states through which, at each time, $s(t)=k_{\rm
B}T(t)/\kappa$ with $T(t)=T_{\rm R}+\alpha\langle\delta I^{2}_{j}(t)\rangle$).

Figure 6: (a) Position PSD for three non-fluctuating radiation pressure sets
–lowest overlapping spectra– and for three different white-noise intensity
variances corresponding respectively to $T_{\rm R}+T=500,\ 800,\ 1200\
$\mathrm{K}$$. The dashed lines corresponds to Lorentzian fits of Eq. (6) for
finite bandwidths extending over the dashed fits. (b) Comparison between the
two temperature calibration methods. The blue dots are the calibration data
obtained by these Lorentzian fits and the red ones correspond to the
temperatures obtained by assuming that $s$ obeys equipartition. The lines
correspond to linear fits $T(\langle\delta
I_{j}^{2}\rangle)=\alpha\langle\delta I_{j}^{2}\rangle+T_{\rm R}$.

The temperature calibration is performed using two different methods. The
first one involves equipartition $\kappa s=k_{\rm B}(T_{\rm R}+T)$ between the
steady-state, thermalized measured position variances $s$ and the effective
temperatures $T$ set by laser $3$. This method assumes that high frequencies,
those above the threshold imposed by the limited bandwidth of the AOM, do not
contribute significantly to $s$. A second method that does not rely on this
approximation is also implemented over a finite bandwidth analysis. For
$\langle\delta I^{2}_{j}\rangle\neq 0$, we use Eq. (6) to fit a Lorentzian on
the position PSD, but now using the volt-to-meter conversion factor and the
roll-off frequency value $f_{c}$ obtained in the case of a non fluctuating
Laser $3$ intensity $I=I_{0}$. In that case, the only fitting parameter for
the PSD is the effective temperature $T_{\rm R}+T$. The finite bandwidth
extends over the dashed lines on Fig. 6 (a) in which the PSD for three
different $\langle\delta I^{2}_{j}\rangle$ that corresponds to $T_{\rm
R}+T=500,\ 800,\ 1200\ $\mathrm{K}$$. The comparison between the two
calibration methods is displayed in Fig. 6 (b) as a function of the input
white-noise (squared amplitude) of laser $3$ fluctuations, $\langle\delta
I_{j}^{2}\rangle$. The calibration factor $\alpha$ involved in the temperature
changes $\Delta T=\alpha\langle\delta I_{j}^{2}\rangle$ is then used in a
python code [34] to build a generic $T(t)$ protocol applying a time dependent
variance envelope $\langle\delta I_{j}^{2}(t)\rangle$ in the noise signal sent
by the AOM.

The uncertainties in stiffness and temperature are directly given by the
calibration procedures. For the stiffness, uncertainties stem from the
dispersion of the three measurements performed with their Lorentzian fits for
$I=I_{0}$ and corresponds to $0.2$ fN/nm. For the temperature, uncertainty
come from the errors in the fitting parameter involved in the second
calibration method. They correspond to $40\ $\mathrm{K}$$.

##  Appendix C Variance $s(t)$ equation of motion

At the one trajectory $j$ level, the motion $x_{j}(t)$ of the trapped
microsphere along the optical axis $\hat{x}$ of the optical trap follows the
Ornstein-Uhlenbeck process

$\gamma\dot{x}_{j}(t)=-\kappa(x_{j}(t)-x_{0}(t))+F_{\rm
ther}(\xi_{j}(t))+F_{\rm ext}(t),$ (7)

where, $F_{\rm ther}(\xi_{j}(t))=\sqrt{2k_{\rm B}T_{\rm R}\gamma}\xi_{j}(t)$
is Gaussian distributed with zero average $\langle\xi_{j}(t)\rangle=0$ and
delta correlated
$\langle\xi_{j}(t)\xi_{j^{\prime}}(t^{\prime})\rangle=\delta_{jj^{\prime}}\delta(t-t^{\prime})$.

The external radiation pressure produces a fluctuating force $F_{\rm
ext}(t)=F_{0}+\delta F_{j}(t)$ with two contributions: the stochastic force
contribution that acts on the Brownian particle and increases its variance,
and the mean, constant contribution $F_{0}$ that produces a displacement in
the equilibrium position. This spatial shift of the equilibrium position
inside the trap can be taken into account by the following change of variable
$x_{j}(t)\rightarrow x_{j}(t)+F_{0}/\kappa$, with the new variable obeying Eq.
(7) with $x_{0}(t)\equiv 0$.

The other contribution $\delta F_{j}$ can be combined with $F_{\rm ther}$.
When the fluctuation spectrum of the external radiation pressure is set to a
white noise, $F_{\rm ext}(t)$ acts as a secondary, thermal bath according to
$F_{\rm ther}(\xi_{j}(t))+\delta
F_{j}(t)\rightarrow\sqrt{2k_{B}T(t)\gamma}\xi_{j}(t)$. This gives the
possibility to perform kinetic temperature changes by simply adjusting the
amplitude of such white-noise fluctuations.

Multiplying Eq. (7) by $x_{j}(t)$ leads to the stochastic equation

$\dot{x}_{j}^{2}(t)=-\frac{2}{\tau}x_{j}^{2}(t)+\sqrt{\frac{8k_{\rm
B}T(t)}{\gamma}}\xi_{j}(t)x_{j}(t),$ (8)

where $x^{2}_{j}$ is experimentally accessible together with its ensemble
average done over all stochastic trajectories $j$ at identical reference times
$\langle x^{2}_{j}\rangle=s(t)$. Rather than given by an instantaneous
derivative of the stochastic trajectory, $\dot{x}^{2}_{j}$ is given by the
instantaneous difference $2x_{j}(t)(F_{\rm ther}/\gamma-x_{j}(t)/\tau)$.

In order to transform Eq. (8) in the deterministic variance equation
evolution, we used the stochastic solution for $x_{j}(t)$ considering a
generic $T(t)$ protocol

$x_{j}(t)=x_{j}(0^{-})\textrm{e}^{-t/\tau}+\int_{0}^{t}\sqrt{\frac{2k_{B}T(\zeta)}{\gamma}}\xi_{j}(\zeta)\exp{(\zeta-t)/\tau}\textrm{d}\zeta.$
(9)

From it, we evaluate the ensemble average correlation function between the
stochastic trajectory and the effective thermal force $\langle x_{j}(t)F_{\rm
ther}(\xi_{j}(t))\rangle=\sqrt{k_{\rm B}T(t)/2\gamma}$ and get Eq. ($1$), main
text

$\frac{{\rm d}s}{{\rm d}t}=-\frac{2}{\tau}s(t)+2D(t),$ (10)

with the time dependent diffusion coefficient $D(t)=k_{\rm B}T(t)/\gamma$ and
reminding that $\gamma=\tau\kappa$.

##  Appendix D Generation of entropy $\Sigma_{\rm gen}$

Assuming that our Brownian sphere driven by a thermal protocol $T(t)$ evolves
through a succession of equilibirum states, the probability density associated
with the dynamical evolution can be written as:

$p(x_{j},T(t))=\frac{1}{\sqrt{2\pi k_{\rm B}T(t)/\kappa}}\exp{-\frac{\kappa
x_{j}^{2}}{2k_{\rm B}T(t)}}.$ (11)

Looking at the stochastic entropy $\sigma_{\rm sys}=-k_{\rm B}\ln
p(x_{j},T(t))$ as a state function, the total variation of this quantity
through a transition between two equilibrium states
$(\kappa,T_{i})\rightarrow(\kappa,T(t))$ is protocol independent. This gives
the possibility to extend Eq. (11) to the case of irreversible transformations
for which the infinitesimal variation of $\sigma_{\rm sys}$ can be evaluated
as:

${\rm d}\sigma_{\rm
sys}(x_{j},T(t))=\frac{\kappa}{T(t)}x_{j}\textrm{d}x_{j}+\frac{k_{\rm
B}}{2T(t)}\left(1-\frac{\kappa x_{j}^{2}}{k_{\rm B}T(t)}\right)\textrm{d}T.$
(12)

From the thermodynamic interpretation of Eq. (7) presented in [44], the
infinitesimal stochastic heat is defined as ${\rm d}q=-\kappa x_{j}{\rm
d}x_{j}$. We can thus identify the first term on the right-hand side of Eq.
(12) as (the opposite of) the infinitesimal variation of the medium entropy
${\mkern 3.0mu\mathchar 22\relax\mkern-12.0mud}\sigma_{\rm med}={\rm d}q_{\rm
qs}/T(t)$. The infinitesimal variation in the total entropy corresponds to the
entropy generated through the irreversible transformation ${\mkern
3.0mu\mathchar 22\relax\mkern-12.0mud}\sigma_{\rm gen}={\mkern 3.0mu\mathchar
22\relax\mkern-12.0mud}\sigma_{\rm med}+{\rm d}\sigma_{\rm sys}$. After an
ensemble average among non equilibrium trajectories, it writes as
$\langle{\mkern 3.0mu\mathchar 22\relax\mkern-12.0mud}\sigma_{\rm
gen}\rangle={\mkern 3.0mu\mathchar 22\relax\mkern-12.0mud}\Sigma_{\rm gen}$,
so that the cumulative, ensemble average, generated entropy is:

$\Sigma_{\rm
gen}(t)=\frac{1}{2}\int_{t_{i}}^{t}\frac{\dot{T}(\zeta)}{T^{2}(\zeta)}\left(k_{\rm
B}T(\zeta)-\kappa s(\zeta)\right)\ {\rm d}\zeta,$ (13)

in which $\dot{T}={\rm d}T/{\rm d}t$.

##  Appendix E  Temperature protocols

In this section, the time dependent expressions for the three temperature
protocols discussed in the main text (STEP, ThESE and optimal) are derived,
together with the corresponding system’s motional responses through the time-
evolution of the variance $s(t)$. The expressions obtained here are plotted in
Fig. $2$, main text.

###  E.1 STEP protocols

The STEP protocol corresponds to an abrupt temperature change. Using the step
function for which $\Theta(t-t_{i})=0$ if $t<t_{i}$ and $\Theta(t-t_{i})=1$ if
$t\geq t_{i}$, the protocol that connects an initial temperature
$T(t_{i}^{-})=T_{\rm R}+T_{i}$ to a final one $T(t_{i}^{+})=T_{\rm R}+T_{f}$
in which $T_{f}-T_{i}=\Delta T$ writes as

$T(t)=(T_{\rm R}+T_{i})+\Theta(t-t_{i})\Delta T,$ (14)

Using this protocol into Eq. (10) –Eq. (1) in the main text– and imposing an
initial equilibrium condition $s(t_{i})=k_{\rm B}(T_{\rm R}+T_{i})/\kappa$
corresponding to equipartition, the evolution of $s(t)$ for $t>t_{i}$ is given
by

$s(t)=\frac{k_{B}}{\kappa}\left(T_{\rm R}+T_{f}-\Delta
T\exp{-\frac{2}{\tau}(t-t_{i})}\right).$ (15)

###  E.2 ThESE protocols

Adapting [35] (in which trap stiffness protocols $\kappa(t)$ are developed) to
temperature protocols $T(t)$, we use the same third degree polynomial ansatz
for the variance $s(t)=At^{3}+Bt^{2}+Ct+D$, imposing initial
$s(t_{i})=s_{i}=k_{\rm B}(T_{i}+T_{\rm R})/\kappa$ and final
$s(t_{f})=s_{f}=k_{\rm B}(T_{f}+T_{\rm R})/\kappa$ equilibrium conditions
together with $\dot{s}(t_{i})=\dot{s}(t_{f})=0$. We calculate the time-
dependent variance $s(t)$ along the transition duration time
$t_{f}-t_{i}=\Delta t$ as

$s(t)=\frac{k_{\rm B}}{\kappa}\left(-2\Delta T\frac{t^{3}}{\Delta
t^{3}}+3\Delta T\frac{t^{2}}{\Delta t^{2}}+T_{\rm R}+T_{i}\right).$ (16)

Substituting Eq. (16) and its derivative $\dot{s}(t)$ into Eq. (10) yields the
explicit time dependent protocol

$T(t)=-2\Delta T\frac{t^{3}}{\Delta t^{3}}+3\Delta T\left(-\frac{\tau}{\Delta
t}+1\right)\frac{t^{2}}{\Delta t^{2}}+3\Delta T\frac{\tau t}{\Delta
t^{2}}+T_{\rm R}+T_{i}.$ (17)

###  E.3 Optimal protocols

Optimal protocols are derived following the optimization method that we
developed in [33] for trap stiffness protocols $\kappa(t)$. This method relies
on treating the equilibrium state-to-state transfer duration and the energetic
cost on an equal footing, with trade-off regulated by a Lagrange multiplier to
built a functional that can then be minimized using standard Euler-Lagrange
equations. In the case of trap stiffness protocols $\kappa(t)$, the energetic
cost was identified as the dissipative work [33]. As discussed in the main
text, for the case of thermal protocol that correspond to isochoric transition
(i.e. work-free), the thermodynamic footprint of the protocol is of entropic
nature. The trade-off is then regulated between the transfer duration $\Delta
t=t_{f}-t_{i}$ and the production of entropy.

The expression of the generated entropy $\Sigma_{\rm gen}$ given by Eq. (13)
–Eq. ($2$) in the main text– can be integrated by parts:

$\Sigma_{\rm gen}[s_{i},s_{f};T(s)]=\frac{k_{\rm B}}{2}\ln\frac{T_{\rm
R}+T_{f}}{T_{\rm R}+T_{i}}+\frac{k_{\rm B}}{2}\left(\frac{\kappa s_{f}}{k_{\rm
B}(T_{\rm R}+T_{f})}-\frac{\kappa s_{i}}{k_{\rm B}(T_{\rm
R}+T_{i})}\right)-\frac{\kappa}{2}\int_{s_{i}}^{s_{f}}\frac{\textrm{d}{s}}{T(s)}.$
(18)

The first term is identified as the protocol independent total variation of
the system’s entropy $\Delta\Sigma_{\rm sys}$. The second one is zero when
considering that the initial and final states are fixed to be at thermal
equilibrium, obeying equipartition, just like for ThESE protocols. In
contrast, the last term depends on the profile of the protocol and thus
carries the entropic contribution of the non-equilibrium process.

The duration $\Delta t=t_{f}-t_{i}$ for the $T_{\rm R}+T_{i}\rightarrow T_{\rm
R}+T_{f}$ transfer can be written as a functional of the variance $s(t)$ using
Eq. (10):

$\Delta
t=\int_{t_{i}}^{t_{f}}\textrm{d}t=\frac{1}{2}\int_{s_{i}}^{s_{f}}\frac{\gamma{\rm
d}s}{k_{\rm B}T(s)-s\kappa}.$ (19)

The trade-off between entropy production and state-to-state transfer duration
is then given by:

$\int_{s_{i}}^{s_{f}}L[s,T(s)]{\rm
d}s=\int_{s_{i}}^{s_{f}}\left(\frac{\gamma}{k_{\rm
B}T(s)-s\kappa}-\frac{\lambda\kappa}{k_{\rm B}T(s)}\right)\textrm{d}s,$ (20)

where $\lambda/k_{\rm B}$ is a Lagrange multiplier. The Euler-Lagrange
equation $\textrm{d}/\textrm{d}s(\partial L/\partial T^{\prime})-\partial
L/\partial T=0$, with $T^{\prime}\equiv\textrm{d}T/\textrm{d}s$, will lead to
the second order polynomial equation

$\left(1-\frac{\lambda}{\tau}\right)T^{2}+2\frac{\lambda}{\tau}\frac{\kappa
s}{k_{\rm B}}T-\frac{\lambda}{\tau}\left(\frac{\kappa s}{k_{\rm
B}}\right)^{2}=0.$ (21)

with two solutions that form the temperature protocols related either to a
heating protocol or to a cooling one, refereed with the sub-index “h” and “c”
respectively. The protocols can be written in terms of the variance according
to

$T_{\rm h/c}(s)=\frac{\kappa s}{k_{\rm B}\left(1\mp\sqrt{\tau/\lambda_{\rm
h/c}}\right)},$ (22)

with associated heating/cooling Lagrange multipliers. The quasi-static limit
of a reversible transition corresponds to $\lambda_{\rm
h/c}\rightarrow\infty$.

To be implemented, explicit time dependent solutions $T(t)$ for the protocol
are necessary. For the variance, time-dependent solutions are obtained by
substituting Eq. (22) into Eq. (10) to give:

$s_{\rm
h/c}(t)=s_{i}\exp[-2\frac{(t-t_{i})}{\tau}\left(1-\frac{1}{1\mp\sqrt{{\tau}/{\lambda_{\rm
h/c}}}}\right)].$ (23)

Another way to express those optimal solutions is through the transfer time
$\Delta t=t_{f}-t_{i}$, considering that at the final time of the transfer,
the system is at thermal equilibrium with $s_{\rm h/c}(t_{f})=k_{\rm B}(T_{\rm
R}+T_{f})/\kappa$. The relation between $\lambda_{\rm h/c}$ and $\Delta t$ is
thus given by

$1\mp\sqrt{{\tau}/{\lambda_{\rm h/c}}}=\left(1+\frac{\tau}{2\Delta
t}\ln\frac{T_{\rm R}+T_{f}}{T_{\rm R}+T_{i}}\right)^{-1},$ (24)

The asset of this parametrization is to lead to a similar expression for a
heating or a cooling protocol for $T(t)$ and $s(t)$. By substituting Eq. (24)
into Eqs. (22) and (23), we end up respectively with:

$T_{\rm opt}(s,\Delta t,T_{i},T_{f})=\frac{\kappa s}{k_{\rm
B}}\left(1+\frac{\tau}{2\Delta t}\ln\frac{T_{\rm R}+T_{f}}{T_{\rm
R}+T_{i}}\right),$ (25) $s_{\rm opt}(t)=s_{i}\left(\frac{T_{\rm
R}+T_{f}}{T_{\rm R}+T_{i}}\right)^{(t-t_{i})/\Delta t},$ (26)

##  Appendix F Minimal entropy production

Eq. ($5$) from the main text is derived by substituting the optimal protocol,
Eq. (26) and Eq. (25), into Eq. (18). This leads to the entropy produced
throughout the isochoric transition of duration $\Delta t$ associated with an
optimal protocol, that is the minimal amount of produced entropy.

As discussed in the main text, the optimal protocol goes through three stages:
two discontinuities at the beginning and at the end of the protocol, and an
exponential, non-equilibrium, evolution in-between for $t_{i}^{+}\leq t\geq
t_{f}^{-}$. The first discontinuity with $T(t_{i}^{-})=T_{\rm R}+T_{i}$ and
$T(t_{i}^{+})=(T_{\rm R}+T_{i})(1+\frac{\tau}{2\Delta t}\ln\frac{T_{\rm
R}+T_{f}}{T_{\rm R}+T_{i}})$, with $s(t_{i}^{-})=s_{i}$ and
$s(t_{i}^{+})=s_{i}$, has an entropy production of

$\Sigma_{\rm min}^{(1)}=\frac{k_{\rm B}}{2}\ln\left[1+\frac{\tau}{2\Delta
t}\ln\frac{T_{\rm R}+T_{f}}{T_{\rm R}+T_{i}}\right]+\frac{k_{\rm
B}}{2}\left(\frac{1}{1+\frac{\tau}{2\Delta t}\ln\frac{T_{\rm R}+T_{f}}{T_{\rm
R}+T_{i}}}-1\right).$ (27)

Through the second intermediate stage $T(t_{i}^{+})=(T_{\rm
R}+T_{i})(1+\frac{\tau}{2\Delta t}\ln\frac{T_{\rm R}+T_{f}}{T_{\rm
R}+T_{i}})$, $T(t_{f}^{-})=(T_{\rm R}+T_{f})(1+\frac{\tau}{2\Delta
t}\ln\frac{T_{\rm R}+T_{f}}{T_{\rm R}+T_{i}})$, $s(t_{i}^{+})=s_{i}$ and
$s(t_{f}^{-})=s_{f}$, the entropy production is

$\Sigma_{\rm min}^{(2)}=\frac{k_{\rm B}}{2}\ln\frac{T_{\rm R}+T_{f}}{T_{\rm
R}+T_{i}}-\frac{k_{\rm B}\ln\frac{T_{\rm R}+T_{f}}{T_{\rm
R}+T_{i}}}{2\left(1+\frac{\tau}{2\Delta t}\ln\frac{T_{\rm R}+T_{f}}{T_{\rm
R}+T_{i}}\right)}.$ (28)

The last stage corresponds to the second discontinuity with
$T(t_{f}^{-})=(T_{\rm R}+T_{f})(1+\frac{\tau}{2\Delta t}\ln\frac{T_{\rm
R}+T_{f}}{T_{\rm R}+T_{i}})$, $T(t_{f}^{+})=T_{\rm R}+T_{f}$, while
$s(t_{f}^{-})=s_{f}$ and $s(t_{f}^{+})=s_{f}$. It leads to the entropy
production

$\Sigma_{\rm min}^{(3)}=-\frac{k_{\rm B}}{2}\ln\left[1+\frac{\tau}{2\Delta
t}\ln\frac{T_{\rm R}+T_{f}}{T_{\rm R}+T_{i}}\right]+\frac{k_{\rm
B}}{2}\left(1-\frac{1}{1+\frac{\tau}{2\Delta t}\ln\frac{T_{\rm
R}+T_{f}}{T_{\rm R}+T_{i}}}\right),$ (29)

with $\Sigma_{\rm min}^{(3)}=-\Sigma_{\rm min}^{(1)}$. The minimal amount of
entropy production for an optimized thermal protocol through a transfer
duration $\Delta t$ is $\Sigma_{\rm min}=\Sigma_{\rm min}^{(1)}+\Sigma_{\rm
min}^{(2)}+\Sigma_{\rm min}^{(3)}=\Sigma_{\rm min}^{(2)}$. Using the
definition of the total variation of the system’s entropy $\Delta\Sigma_{\rm
sys}=(k_{\rm B}/2)\ln(T_{\rm R}+T_{f})/(T_{\rm R}+T_{i})$, Eq. (28) leads to
Eq. ($5$) in the main text.

##  Appendix G Energetics

As introduced above Sec. D, the heat identified on Eq. (7) corresponds to a
total differential. After averaging over the ensemble of trajectories, ${\rm
d}Q=-(1/2)\kappa\left<{\rm d}x^{2}_{j}\right>$, the cumulative heat
corresponds to:

$Q(t)=-\frac{\kappa}{2}\int_{s(t_{i})}^{s(t)}{\rm d}s.$ (30)

Through an isochoric transformation (work-free), this cumulative heat is equal
to the system’s internal energy change, $\Delta U_{\rm sys}=-Q(t)$. The first
law however does not account for the heat, given from the bath to the system,
generated through the irreversible isochoric transformation. This energetic
footprint can be evaluated based on $\Sigma_{\rm gen}$, using its differential
form from Eq. (13)

${\mkern 3.0mu\mathchar 22\relax\mkern-12.0mud}\Sigma_{\rm gen}=\frac{k_{\rm
B}T-\kappa s}{T^{2}}{\rm d}T.$ (31)

This expression leads to define the in-take heat $Q_{\rm gen}(t)$ evaluated in
the main text. In the reversible, quasi-static limit, the internal energies of
the system and the medium coincide at all times, leading to $Q_{\rm
gen}(t)\rightarrow 0$ with therefore $k_{\rm B}\Delta T(t)/2\rightarrow\Delta
U_{\rm sys}(t)$ as expected.

##  Appendix H Data analysis and error bar

Once the temperature and trap calibrations are performed, as discussed above
–Sec. B– the different temperature protocols are defined by setting the
initial $T_{i}$ and final $T_{f}$ target temperatures and fixing the state-to-
state transfer duration $\Delta t$. Each protocol is repeated to form an
ensemble of $N_{cycles}\sim 17000$ trajectories recorded over $6$ minutes.
These trajectories are combined, using for a time reference the intensity
variance envelope sent by the AOM (see Sec. A). The uncertainty on the
experimental variances are computed using a $\chi^{2}$ law with $N_{cycles}-1$
degrees of freedom with a confidence interval of $95\ \%$. The uncertainties
on the entropy and heat measurements are obtained by standard methods for the
propagation of errors.

## References

  * Chang  _et al._ [2010] D. E. Chang, C. A. Regal, S. B. Papp, D. J. Wilson, J. Ye, O. Painter, H. J. Kimble, and P. Zoller, Cavity opto-mechanics using an optically levitated nanosphere, Proc. Nat. Acad. Sci. 107, 1005 (2010).
  * Martínez  _et al._ [2017] I. A. Martínez, É. Roldán, L. Dinis, and R. A. Rica, Colloidal heat engines: A review, Soft Matter 13, 22 (2017).
  * Albay  _et al._ [2021] J. A. Albay, Z.-Y. Zhou, C.-H. Chang, and Y. Jun, Shift a laser beam back and forth to exchange heat and work in thermodynamics, Sci. Rep. 11, 1 (2021).
  * Gonzalez-Ballestero  _et al._ [2021] C. Gonzalez-Ballestero, M. Aspelmeyer, L. Novotny, R. Quidant, and O. Romero-Isart, Levitodynamics: Levitation and control of microscopic objects in vacuum, Science 374, eabg3027 (2021).
  * Rademacher  _et al._ [2022] M. Rademacher, M. Konopik, M. Debiossac, D. Grass, E. Lutz, and N. Kiesel, Nonequilibrium control of thermal and mechanical changes in a levitated system, Phys. Rev. Lett. 128, 070601 (2022).
  * Sekimoto [2010] K. Sekimoto,  _Stochastic energetics_ , Vol. 799 (Springer, 2010).
  * Seifert [2012] U. Seifert, Stochastic thermodynamics, fluctuation theorems and molecular machines, Rep. Prog. Phys. 75, 126001 (2012).
  * Raizen  _et al._ [2012] M. G. Raizen, S. Kheifets, and T. Li, Optical trapping and cooling of glass microspheres, in  _Optical Trapping and Optical Micromanipulation IX_ , Vol. 8458 (SPIE, 2012) pp. 66–72.
  * Martinez  _et al._ [2013] I. A. Martinez, E. Roldán, J. M. Parrondo, and D. Petrov, Effective heating to several thousand kelvins of an optically trapped sphere in a liquid, Phys. Rev. E 87, 032159 (2013).
  * Chupeau  _et al._ [2018] M. Chupeau, B. Besga, D. Guéry-Odelin, E. Trizac, A. Petrosyan, and S. Ciliberto, Thermal bath engineering for swift equilibration, Phys. Rev. E 98, 010104 (2018).
  * Delić  _et al._ [2020] U. Delić, M. Reisenbauer, K. Dare, D. Grass, V. Vuletić, N. Kiesel, and M. Aspelmeyer, Cooling of a levitated nanoparticle to the motional quantum ground state, Science 367, 892 (2020).
  * van der Laan  _et al._ [2021] F. van der Laan, F. Tebbenjohanns, R. Reimann, J. Vijayan, L. Novotny, and M. Frimmer, Sub-Kelvin feedback cooling and heating dynamics of an optically levitated librator, Phys. Rev. Lett. 127, 123605 (2021).
  * Martínez  _et al._ [2016a] I. A. Martínez, É. Roldán, L. Dinis, D. Petrov, J. M. Parrondo, and R. A. Rica, Brownian Carnot engine, Nature Phys. 12, 67 (2016a).
  * Plata  _et al._ [2020a] C. A. Plata, D. Guéry-Odelin, E. Trizac, and A. Prados, Building an irreversible Carnot-like heat engine with an overdamped harmonic oscillator, J. Stat. Mech. 2020, 093207 (2020a).
  * Watanabe and Minami [2022] G. Watanabe and Y. Minami, Finite-time thermodynamics of fluctuations in microscopic heat engines, Phys. Rev. Res. 4, L012008 (2022).
  * Kumar and Bechhoefer [2020] A. Kumar and J. Bechhoefer, Exponentially faster cooling in a colloidal system, Nature 584, 64 (2020).
  * Plata  _et al._ [2020b] C. A. Plata, D. Guéry-Odelin, E. Trizac, and A. Prados, Finite-time adiabatic processes: Derivation and speed limit, Phys. Rev. E 101, 032129 (2020b).
  * Nakamura  _et al._ [2020] K. Nakamura, J. Matrasulov, and Y. Izumida, Fast-forward approach to stochastic heat engine, Phys. Rev. E 102, 012129 (2020).
  * Jun and Lai [2021] Y. Jun and P.-Y. Lai, Instantaneous equilibrium transition for Brownian systems under time-dependent temperature and potential variations: Reversibility, heat and work relations, and fast isentropic process, Phys. Rev. Res. 3, 033130 (2021).
  * Chen [2022] J.-F. Chen, Optimizing Brownian heat engine with shortcut strategy, Phys. Rev. E 106, 054108 (2022).
  * Patrón  _et al._ [2022] A. Patrón, A. Prados, and C. A. Plata, Thermal brachistochrone for harmonically confined Brownian particles, Eur. Phys. J. Plus 137, 1 (2022).
  * Guéry-Odelin  _et al._ [2019] D. Guéry-Odelin, A. Ruschhaupt, A. Kiely, E. Torrontegui, S. Martínez-Garaot, and J. G. Muga, Shortcuts to adiabaticity: Concepts, methods, and applications, Rev. Mod. Phys. 91, 045001 (2019).
  * Guéry-Odelin  _et al._ [2023] D. Guéry-Odelin, C. Jarzynski, C. A. Plata, A. Prados, and E. Trizac, Driving rapidly while remaining in control: Classical shortcuts from Hamiltonian to stochastic dynamics, Rep. Prog. Phys. 86, 035902 (2023).
  * Albay  _et al._ [2019] J. A. Albay, S. R. Wulaningrum, C. Kwon, P.-Y. Lai, and Y. Jun, Thermodynamic cost of a shortcuts-to-isothermal transport of a Brownian particle, Phys. Rev. Res. 1, 033122 (2019).
  * Debiossac  _et al._ [2020] M. Debiossac, D. Grass, J. J. Alonso, E. Lutz, and N. Kiesel, Thermodynamics of continuous non-Markovian feedback control, Nature Comm. 11, 1 (2020).
  * Prados [2021] A. Prados, Optimizing the relaxation route with optimal control, Phys. Rev. Res. 3, 023128 (2021).
  * Frim and DeWeese [2022] A. G. Frim and M. R. DeWeese, Optimal finite-time Brownian Carnot engine, Phys. Rev. E 105, L052103 (2022).
  * Ye  _et al._ [2022] Z. Ye, F. Cerisola, P. Abiuso, J. Anders, M. Perarnau-Llobet, and V. Holubec, Optimal finite-time heat engines under constrained control, Phys. Rev. Res. 4, 043130 (2022).
  * Jun and Lai [2022] Y. Jun and P.-Y. Lai, Minimal dissipation protocols of an instantaneous equilibrium Brownian particle under time-dependent temperature and potential variations, Phys. Rev. Res. 4, 023157 (2022).
  * Paraguassú  _et al._ [2022] P. V. Paraguassú, R. Aquino, L. Defaveri, and W. A. M. Morgado, Effects of the kinetic energy in heat for overdamped systems, Phys. Rev. E 106, 044106 (2022).
  * Schmiedl and Seifert [2007] T. Schmiedl and U. Seifert, Optimal finite-time processes in stochastic thermodynamics, Phys. Rev. Lett. 98, 108301 (2007).
  * Bonança and Deffner [2018] M. V. S. Bonança and S. Deffner, Minimal dissipation in processes far from equilibrium, Phys. Rev. E 98, 042103 (2018).
  * Rosales-Cabara  _et al._ [2020] Y. Rosales-Cabara, G. Manfredi, G. Schnoering, P.-A. Hervieux, L. Mertz, and C. Genet, Optimal protocols and universal time-energy bound in Brownian thermodynamics, Phys. Rev. Res. 2, 012012 (2020).
  * Goerlich  _et al._ [2022] R. Goerlich, L. B. Pires, G. Manfredi, P.-A. Hervieux, and C. Genet, Harvesting information to control nonequilibrium states of active matter, Phys. Rev. E 106, 054617 (2022).
  * Martínez  _et al._ [2016b] I. A. Martínez, A. Petrosyan, D. Guéry-Odelin, E. Trizac, and S. Ciliberto, Engineered swift equilibration of a Brownian particle, Nature Phys. 12, 843 (2016b).
  * Padgett  _et al._ [2010] M. J. Padgett, J. Molloy, and D. McGloin,  _Optical Tweezers: methods and applications_ (CRC press, 2010).
  * Gennerich [2017] A. Gennerich,  _Optical tweezers_ (Springer, 2017).
  * Le Cunuder  _et al._ [2016] A. Le Cunuder, I. A. Martínez, A. Petrosyan, D. Guéry-Odelin, E. Trizac, and S. Ciliberto, Fast equilibrium switch of a micro mechanical oscillator, Appl. Phys. Lett. 109, 113502 (2016).
  * Shiraishi  _et al._ [2018] N. Shiraishi, K. Funo, and K. Saito, Speed limit for classical stochastic processes, Phys. Rev. Lett. 121, 070601 (2018).
  * Note [1] The minus sign in this convention means that for a generation of entropy $d\Sigma_{\rm gen}>0$, the energy flows from medium to the system and ${\mkern 3.0mu\mathchar 22\relax\mkern-12.0mud}Q_{gen}<0$, consistent with the standard convention of stochastic thermodynamics [44].
  * Ito [2018] S. Ito, Stochastic thermodynamic interpretation of information geometry, Phys. Rev. Lett. 121, 030605 (2018).
  * Ito [2022] S. Ito, Geometric thermodynamics for the Fokker-Planck equation: Stochastic thermodynamic links between information geometry and optimal transport, arXiv:2209.00527 10.48550/arXiv.2209.00527 (2022).
  * Berg-Sørensen and Flyvbjerg [2004] K. Berg-Sørensen and H. Flyvbjerg, Power spectrum analysis for optical tweezers, Rev. Sci. Instr. 75, 594 (2004).
  * Sekimoto [1998] K. Sekimoto, Langevin equation and thermodynamics, Prog.Theor. Phys. Suppl. 130, 17 (1998).


# Image Superresolution using Scale-Recurrent Dense Network

Kuldeep Purohit  Srimanta Mandal  A. N. Rajagopalan

###### Abstract

Recent advances in the design of convolutional neural network (CNN) have
yielded significant improvements in the performance of image super-resolution
(SR). The boost in performance can be attributed to the presence of residual
or dense connections within the intermediate layers of these networks. The
efficient combination of such connections can reduce the number of parameters
drastically while maintaining the restoration quality. In this paper, we
propose a scale recurrent SR architecture built upon units containing series
of dense connections within a residual block (Residual Dense Blocks (RDBs))
that allow extraction of abundant local features from the image. Our scale
recurrent design delivers competitive performance for higher scale factors
while being parametrically more efficient as compared to current state-of-the-
art approaches. To further improve the performance of our network, we employ
multiple residual connections in intermediate layers (referred to as Multi-
Residual Dense Blocks), which improves gradient propagation in existing
layers. Recent works have discovered that conventional loss functions can
guide a network to produce results which have high PSNRs but are perceptually
inferior. We mitigate this issue by utilizing a Generative Adversarial Network
(GAN) based framework and deep feature (VGG) losses to train our network. We
experimentally demonstrate that different weighted combinations of the VGG
loss and the adversarial loss enable our network outputs to traverse along the
perception-distortion curve. The proposed networks perform favorably against
existing methods, both perceptually and objectively (PSNR-based) with fewer
parameters.

##  1 Introduction

Super-resolution (SR) techniques are devised to cope up with the issue of
limited-resolution while imaging by generating a high resolution (HR) image
from a low resolution (LR) image. However, the possibility of multiple HR
images leading to the same LR image makes the problem ill-posed. This can be
addressed by regularized mapping of LR image patches to HR counterparts, which
are generally extracted from some example images. However, a constrained
linear mapping may not be able to represent complex textures of natural
images. Deep learning based techniques can behave better in this case by
learning a non-linear mapping function.

SR for different scale factors requires separate training of the network.
Joint training for different scale factors can address the issue, as has been
attempted by VDSR [18], which needs a bicubic interpolated LR image as input.
However, this strategy can come in the way of exploiting hierarchical features
from the original LR image, and crucial details may be lost. Further,
processing such a high dimensional image for a large number of layers demands
higher computational resources. Another way to deal with the situation is to
learn the model for lower scale factor such as 2 and use it to initialize the
learning for higher factors such as 3, 4, etc [24]. However, this strategy is
parametrically inefficient and does not work well for higher scale factors
(e.g., $8$).

In order to accommodate different up-sampling factors while keeping a check on
the number of parameters, we propose a scale-recurrent strategy that helps in
transferring learned filters from lower scale factors to higher ones. We use
our scale-recurrent strategy in conjunction with a smaller version of Residual
Dense Network (RDN) [50], where we use fewer Residual Dense Blocks (RDBs) to
reduce the number of parameters as compared to the original RDN. We choose
RDBs as building blocks since the combination of residual and dense
connections can help in overcoming their individual limitations. This
combination allows for efficient flow of information throughout the layers
while eliminating the vanishing gradient issue. We refer to this scale-
recurrent residual dense network as SRRDN.

Motivated by the recent developments in network designs based on dense
connections, we introduce multiple residual connections within an RDB using
$1\times 1$ convolutions that results in superior performance with marginal
parametric cost. The proposed units are termed as Multi-Residual Dense Blocks
(MRDB). Our proposed scale-recurrent network with MRDBs is termed as multi-
residual dense network (MRDN).

##  2 Related Works

Super-resolving a single image generally requires some example HR images to
import relevant information for generating the HR image. Two streams of
approaches make use of the HR example images in their frameworks: i)
Conventional, and ii) deep learning based. The functioning of conventional SR
approaches depends on finding patches, similar to the target patch in the
database of patches. Since there could be many similarities, one needs to
regularize the problem. Thus, most of the conventional approaches focus on
discovering regularization techniques in SR such as Tikhinov [49], total-
variation [28], Markov random field [17], non-local-mean [26, 11, 27],
sparsity-based prior [46, 47, 10], and so on [9, 27, 36, 37, 39, 3, 2, 34, 33,
43, 35].

Although, the sparsity-based prior works quite efficiently, the linear mapping
of information may fail to represent complex structures of an image. Here,
deep-learning based approaches have an upper hand as they can learn a non-
linear mapping between LR and corresponding HR image [6, 8, 45, 18, 40, 21,
22, 38, 41, 24, 48, 15]. Deep learning stepped into the field of SR via SRCNN
[7] by extending the notion of sparse representation using CNN. The non-
linearity involved in CNN is able to better represent complex structures than
conventional approaches to yield superior results. However, going to deeper
architectures increases the difficulty in training such networks. Employing a
residual network into the frame along with skip connections and recursive
convolution can mitigate this issue [18, 19]. Following such an approach, VDSR
[18] and DRCN [19] methods have demonstrated performance improvement. The
power of recursive blocks involving residual units to create a deeper network
was explored in [40]. Recursive unit in conjunction with a gate unit can act
as a memory unit that adaptively combines the previous states with the current
state to produce a super-resolved image [41]. However, these approaches
interpolate the LR image to the HR grid and feed it to the network. But this
increases the computational requirement due to the higher dimension.

To circumvent the dimension issue, networks exists that are tailored to
extract features from the LR image which are then processed in subsequent
layers. At the end layer, up-sampling is performed to match with the HR
dimension [8, 23]. This process can be made faster by reducing the dimension
of the features going to the layers that map from LR to HR and is known as
FSRCNN [8]. ResNet [13] based deeper network with generative adversarial
network (GAN) [12] can produce photo-realistic HR results by including
perceptual loss [16] in the network, as devised in SRResNet [23]. The
perceptual loss is further used with a texture synthesis mechanism in GAN
based model to improve SR performance [38]. Though these approaches are able
to add textures in the image, sometimes the results contain artifacts. The
model architecture of SRResNet [23] has been simplified and optimized to
achieve further improvements in EDSR [24]. This was later modified in MDSR
[24], which performs joint training for different scale factors by introducing
scale-specific feature extraction and pixel-shuffle layers.

##  3 Architecture Design

The success of recent approaches has emphasized the importance of network
design. Specifically, most recent image and video SR approaches are built upon
two popular image classification networks: residual networks [13] and densely
connected networks [14]. These network designs have also enjoyed success and
achieved state-of-the-art performance in other image restoration tasks such as
image denoising, dehazing, and deblurring. Motivated by the generalization
capability of such advances in network designs, the recent work of RDN [50]
proposed a super-resolution network which involves a mixture of both residual
and dense connections and yields state-of-the-art results. The fundamental
block of this network is RDB, which we too adopt in our work.

While DenseNet was proposed for high-level computer vision tasks (e.g., object
recognition), RDN adopted and improved upon this design to address image SR.
Specifically, batch-normalization (BN) layers were removed as they hinder the
performance of the network by increasing computational complexity and memory
requirements. The pooling layers are removed too since they could discard
important pixel-level information. To enable a higher growth rate, each dense
block is terminated with a $1\times 1$ conv layer (Local Feature Fusion) and
its output is added to the input of the block using Local Residual Learning.
This strategy has been demonstrated to be very effective for SR [50].

###  3.1 Scale-Recurrent Design

Most of the existing SR approaches handle different scale factors
independently by neglecting inter-scale relationships. They need to be trained
independently for different scale factors. However, VDSR [18] can address the
issue by jointly training a network for multiple scales. This kind of training
requires LR images of different resolutions to be up-sampled by bi-cubic
interpolation prior to feeding to the network. Interpolation by a large factor
causes loss of information. Further, it requires higher computational time and
memory as compared to scale-specific networks.

Our network’s global design is a multi-scale pyramid which recursively uses
the same convolutional filters across scales. This is motivated by the fact
that a network capable of super-resolving an image by a factor of 2 can be
recursively used to super-resolve the image by a factor $2s,s=1,2,3\dots$.
Even with the same training data, the recurrent exploitation of shared weights
works in a way similar to using data multiple times to learn parameters, which
actually amounts to data augmentation with respect to scales. We design the
network to reconstruct HR images in intermediate steps by progressively
performing a $2\times$ upsampling of the input from the previous level.
Specifically, we first train a network to perform SR by a factor of $2$ and
then re-utilize the same weights to take the output of $2\times$ as input and
result into an output at resolution $4\times$. This architecture is then fine-
tuned to perform $4\times$ SR. We experimentally found that such
initialization (training for the task of $2\times$ SR) leads to better
convergence for larger scale factors. Ours is one of the first approaches to
re-utilize the parameters across scales, which significantly reduces the
number of trainable parameters while yielding performance gains for higher
scale factors. We term our network SRRDN, whose $4\times$ SR version is shown
in Fig. 1.

Figure 1: Network architecture of the proposed Scale-Recurrent Multi-Residual
Dense Network for $4\times$ SR.

###  3.2 Multi-Residual Dense Blocks

We also propose improvements in the structure of RDB for efficient extraction
of high-resolution features from low-resolution images. The effectiveness of
residual and dense connections has been proved in various vision tasks; yet,
they cannot be considered as optimum topology. For example, too many additions
on the same feature space may impede information flow in ResNet [14]. The
possibility of same type of raw features from different layers can lead to
redundancies in DenseNet [5]. Some of these issues are addressed in recent
image classification networks [5, 44]. However, these designs are optimized
for image classification tasks and their applicability to image restoration
has not been explored yet.

An RDB of SRRDN already contains multiple paths connecting the current layer
to previous network layers. One connection is present in the form of a
concatenation of features, which is similar to the connections in DenseNet.
Although growth rates affect the performance positively, it is harder to train
a large number of dense blocks which possess a higher growth rate, as has been
experimentally demonstrated in [50]. This can be addressed by Local Feature
Fusion (see Fig. 2), by including a second connection that stabilizes the
training of wide network. This brings down the number of output feature-maps
to the number of input feature-maps and enables introduction of a single
residual connection between the input and the output of the block (Local
Residual Learning).

In order to further improve the gradient flow during training, we introduce a
third connection: Multi-Residual connections. Essentially, at each
intermediate layer of the block, we convolve the input features using a
$1\times 1$ conv layer and add them to the output obtained after the
concatenation operation.

Figure 2: Structure of our Multi-Residual Dense Block. Within each module,
concatenation operation is performed using the features estimated by the conv
$3\times 3$ layer, and addition operation is performed on the features
estimated by $1\times 1$ conv layer (which continuously increase the number of
feature maps to match the size of concatenated output). At the end of the
block, a $1\times 1$ conv layer performs local feature fusion to adaptively
control the output information.

This type of connection has two properties: Firstly, existing feature channels
get modified, which helps in deeper and hierarchical feature extraction.
Secondly, it enables learning of equally meaningful features even with a lower
growth-rate during feature concatenation. This strategy promotes new feature
exploration with a moderate growth rate and avoids learning of redundant
features. These two features enable improved error gradient propagation during
training. Our scale-recurrent framework built using MRDBs as basic blocks is
termed as MRDN.

##  4 Experimental Results

###  4.1 Experimental Setup

Here, we specify the details of training setup, test data and evaluation
metrics.

Datasets and degradation models. Following [42, 24, 50, 48], we use 800
training images from DIV2K dataset [42] as training set. For testing, we use
five standard benchmark datasets: Set5 [1], Set14 [47], B100 [29], Urban100
[15], Manga109 [30], and PIRM-self [4]. We consider bicubic(BI) down-sampling
to generate the LR images.

Evaluation metrics. The SR results are evaluated with two metrics: PSNR and a
perceptual metric. For a given image $I$, the perceptual metric is defined as

$P(I)=\frac{1}{2}((10-M(I)+N(I))$ (1)

where $M(I)$ and $N(I)$ are estimated using [25] and [31], respectively. These
metrics have been used to evaluate different approaches in the PIRM SR
Challenge.

Training settings. Data augmentation is performed on the 800 training images,
which are randomly rotated by 90∘, 180∘, 270∘ and flipped horizontally. Our
model is trained by ADAM optimizer [20] with $\beta_{1}=0.9$,
$\beta_{2}=0.999$, and $\epsilon=10^{-8}$. The initial leaning rate is set to
$10^{-4}$ and is then decreased by half every $2\times 10^{5}$ iterations of
back-propagation.

Implementation Details The network is implemented using Pytorch library. For
training the first network, we used a weighted sum of VGG54 loss and L1 Loss.
For the second network, we used a weighted sum of VGG54 loss and conditional-
GAN loss. The experiments have been conducted on a machine with i7-4790K CPU,
64GB RAM and 1 NVIDIA Titan X GPU using PyTorch [32]. During training, we
considered a batch of randomly extracted 16 LR RGB patches of size $32\times
32$ pixels. Training the first network (MRDN) took approximately 40 hours. The
second network (MRDN-GAN) was then trained for 26 hours.

###  4.2 Perceptually Motivated Results

This work has been used for the purpose of participating in the PIRM 2018 SR
Challenge, which focuses on photo-realistic results (measured using
perceptually motivated metric) while maintaining certain levels of tolerance
in terms of root mean squared error (RMSE). In this challenge, there exist
three tracks corresponding to different ranges of RMSE for scale factor of
$\times 4$. Track 1 corresponds to RMSE $\leq 11.5$. Track two: RMSE between
$11.5$ and $12.5$, while Track 3 included results with RMSE $\geq 12.5$.
Perceptually attractive images are generally rich in various high-frequency
(HF) image details. Thus, the objective is to bring out HF details while
super-resolving the given LR images such that the resultant images yield
better perceptual score. We employed our networks to generate results with
scores suitable for each track and proved that our technique can elegantly
facilitate quality control during test time. Our team REC-SR secured the
$7^{th}$, $7^{th}$ and $10^{th}$ ranks in Tracks 1, 2 and 3, respectively.

Set14 ($4\times$): BSD100 ($3\times$): Set14 ($4\times$): ppt3 78004 Barbara
---
HR Bicubic HR Bicubic HR Bicubic SRResNet [23] SRGAN [23] SRResNet [23] SRGAN
[23] SRResNet [23] SRGAN [23] ENet-E [38] ENet-PAT [38] ENet-E [38] ENet-PAT
[38] ENet-E [38] ENet-PAT [38] MRDN (Ours) MRDN-GAN (Ours) MRDN (Ours) MRDN-
GAN (Ours) MRDN (Ours) MRDN-GAN (Ours)
Figure 3:  Visual comparison for $4\times$ SR on images from Set14 and BSD100
datasets.  Urban100 ($4\times$): img_092 HR Bicubic SRCNN [6] FSRCNN [8] VDSR
[18] LapSRN [21] MemNet [41] SRMDNF [48] EDSR [24] MRDN-GAN (Ours)
---
Urban100 ($4\times$): img_093 HR Bicubic SRCNN [6] FSRCNN [8] VDSR [18] LapSRN
[21] MemNet [41] SRMDNF [48] EDSR [24] MRDN-GAN (Ours)
Figure 4: Visual comparisons for $4\times$ SR on Urban100 dataset.

Visual comparisons of the results of our networks MRDN and MRDN-GAN with these
techniques on images from standard SR benchmarks are given in Fig. 3. In all
the images, it can be seen that the results of SRResNet and ENetE suffer from
blurring artifacts. This demonstrates the insufficiency of only pixel-
reconstruction losses. However, the efficient design of our MRDN leads to
improved recovery of scene texture in challenging regions. For example, in
image “ppt3”, all the compared methods fail to recover the letters ‘i’ and
‘t’. However, our proposed MRDN recovers them. On the other hand, GAN-based
methods of SRGAN, and ENetPAT produce distorted scene textures. The results of
ENetPAT are sharper than SRGAN but it generates unwanted artifacts and
arbitrary edges (e.g., the result for the image “78004”). In contrast, our
proposed MRDN-GAN leads to textures which are closer to that of the ground-
truth HR image too. Similar observations can be found in other images. These
comparisons show that the design of SR network plays an important role in both
objective and perceptual quality of SR.

In Fig. 4, we compare the results of our model on Urban100 dataset with state-
of-the-art SR approaches which are not perceptually motivated for a scale
factor of 4. For such texture-rich scenes, a major challenge is to bring out
high frequency image details. One can observe that most of the existing
approaches fail in this aspect and their results are blurred (see Fig. 4).
However, our MRDN-GAN is capable of generating sufficiently detailed textures.

##  5 Conclusions

We proposed a scale-recurrent deep architecture, which enables transfer of
weights from lower scale factors to the higher ones, in order to reduce the
number of parameters as compared to state-of-the-art approaches. We
experimentally demonstrated that our scale-recurrent design is well-suited for
higher up-sampling factors. The error gradient flow was improved by elegantly
including multiple residual units (MRDN) within the Residual Dense Blocks. To
produce perceptually better results, VGG-based loss functions were utilized
along with a GAN framework. Different weights were assigned to the loss
functions to obtain networks focused on improving either perceptual quality or
objective quality during super-resolution. The perception-distortion trade-off
was addressed by a soft-thresholding technique during test time. We
demonstrated the effectiveness of our parametrically efficient model on
various datasets.

Refined and complete version of this work appeared in ECCVW 2018.

## References

  * [1] Bevilacqua, M., Roumy, A., Guillemot, C., line Alberi Morel, M.: Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In: Proceedings of the British Machine Vision Conference. pp. 135.1–135.10. BMVA Press (2012). https://doi.org/http://dx.doi.org/10.5244/C.26.135
  * [2] Bhavsar, A.V., Rajagopalan, A.N.: Range map superresolution-inpainting, and reconstruction from sparse data. Computer Vision and Image Understanding 116(4), 572–591 (2012)
  * [3] Bhavsar, A.V., Rajagopalan, A.: Resolution enhancement in multi-image stereo. IEEE transactions on pattern analysis and machine intelligence 32(9), 1721–1728 (2010)
  * [4] Blau, Y., Mechrez, R., Timofte, R., Michaeli, T., Zelnik-Manor, L.: 2018 PIRM Challenge on Perceptual Image Super-resolution. ArXiv e-prints pp. 1–22 (Sep 2018)
  * [5] Chen, Y., Li, J., Xiao, H., Jin, X., Yan, S., Feng, J.: Dual path networks. In: Advances in Neural Information Processing Systems. pp. 4467–4475 (2017)
  * [6] Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 38(2), 295–307 (Feb 2016). https://doi.org/10.1109/TPAMI.2015.2439281
  * [7] Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) Computer Vision – ECCV 2014. pp. 184–199. Springer International Publishing, Cham (2014)
  * [8] Dong, C., Loy, C.C., Tang, X.: Accelerating the super-resolution convolutional neural network. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) Computer Vision – ECCV 2016. pp. 391–407. Springer International Publishing, Cham (2016)
  * [9] Dong, W., Zhang, L., Shi, G., Li, X.: Nonlocally centralized sparse representation for image restoration. IEEE Transactions on Image Processing 22(4), 1620–1630 (April 2013). https://doi.org/10.1109/TIP.2012.2235847
  * [10] Dong, W., Zhang, L., Shi, G., Wu, X.: Image deblurring and super-resolution by adaptive sparse domain selection and adaptive regularization. IEEE Transactions on Image Processing 20(7), 1838 –1857 (Jul 2011). https://doi.org/10.1109/TIP.2011.2108306
  * [11] Glasner, D., Bagon, S., Irani, M.: Super-resolution from a single image. In: IEEE International Conference on Computer Vision (ICCV). pp. 349–356 (Sept 2009). https://doi.org/10.1109/ICCV.2009.5459271
  * [12] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N.D., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems 27, pp. 2672–2680. Curran Associates, Inc. (2014)
  * [13] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
  * [14] Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: CVPR. vol. 1, p. 3 (2017)
  * [15] Huang, J., Singh, A., Ahuja, N.: Single image super-resolution from transformed self-exemplars. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 5197–5206 (June 2015). https://doi.org/10.1109/CVPR.2015.7299156
  * [16] Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) Computer Vision – ECCV 2016. pp. 694–711. Springer International Publishing, Cham (2016)
  * [17] Kanemura, A., ichi Maeda, S., Ishii, S.: Superresolution with compound markov random fields via the variational {EM} algorithm. Neural Networks 22(7), 1025 – 1034 (2009). https://doi.org/10.1016/j.neunet.2008.12.005
  * [18] Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 1646–1654 (June 2016). https://doi.org/10.1109/CVPR.2016.182
  * [19] Kim, J., Lee, J.K., Lee, K.M.: Deeply-recursive convolutional network for image super-resolution. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 1637–1645 (June 2016). https://doi.org/10.1109/CVPR.2016.181
  * [20] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. CoRR abs/1412.6980 (2014), http://arxiv.org/abs/1412.6980
  * [21] Lai, W., Huang, J., Ahuja, N., Yang, M.: Deep laplacian pyramid networks for fast and accurate super-resolution. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 5835–5843 (July 2017). https://doi.org/10.1109/CVPR.2017.618
  * [22] Lai, W., Huang, J., Ahuja, N., Yang, M.: Fast and accurate image super-resolution with deep laplacian pyramid networks. CoRR abs/1710.01992 (2017), http://arxiv.org/abs/1710.01992
  * [23] Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., Shi, W.: Photo-realistic single image super-resolution using a generative adversarial network. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 105–114 (July 2017). https://doi.org/10.1109/CVPR.2017.19
  * [24] Lim, B., Son, S., Kim, H., Nah, S., Lee, K.M.: Enhanced deep residual networks for single image super-resolution. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). pp. 1132–1140 (July 2017). https://doi.org/10.1109/CVPRW.2017.151
  * [25] Ma, C., Yang, C.Y., Yang, X., Yang, M.H.: Learning a no-reference quality metric for single-image super-resolution. Computer Vision and Image Understanding 158, 1–16 (2017)
  * [26] Mairal, J., Bach, F., Ponce, J., Sapiro, G., Zisserman, A.: Non-local sparse models for image restoration. In: IEEE 12th International Conference on Computer Vision. pp. 2272 –2279 (29 2009-oct 2 2009). https://doi.org/10.1109/ICCV.2009.5459452
  * [27] Mandal, S., Bhavsar, A., Sao, A.K.: Noise adaptive super-resolution from single image via non-local mean and sparse representation. Signal Processing 132, 134 – 149 (2017). https://doi.org/http://dx.doi.org/10.1016/j.sigpro.2016.09.017
  * [28] Marquina, A., Osher, S.J.: Image super-resolution by TV-regularization and bregman iteration. Journal of Scientific Computing 37, 367–382 (2008). https://doi.org/10.1007/s10915-008-9214-8
  * [29] Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001. vol. 2, pp. 416–423 vol.2 (July 2001). https://doi.org/10.1109/ICCV.2001.937655
  * [30] Matsui, Y., Ito, K., Aramaki, Y., Fujimoto, A., Ogawa, T., Yamasaki, T., Aizawa, K.: Sketch-based manga retrieval using manga109 dataset. Multimedia Tools and Applications 76(20), 21811–21838 (Oct 2017). https://doi.org/10.1007/s11042-016-4020-z, https://doi.org/10.1007/s11042-016-4020-z
  * [31] Mittal, A., Soundararajan, R., Bovik, A.C.: Making a” completely blind” image quality analyzer. IEEE Signal Process. Lett. 20(3), 209–212 (2013)
  * [32] Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., Lerer, A.: Automatic differentiation in pytorch (2017)
  * [33] Punnappurath, A., Nimisha, T.M., Rajagopalan, A.N.: Multi-image blind super-resolution of 3d scenes. IEEE Transactions on Image Processing 26(11), 5337–5352 (2017)
  * [34] Punnappurath, A., Rengarajan, V., Rajagopalan, A.: Rolling shutter super-resolution. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 558–566 (2015)
  * [35] Purohit, K., Mandal, S., Rajagopalan, A.: Mixed-dense connection networks for image and video super-resolution. Neurocomputing 398, 360–376 (2020)
  * [36] Rajagopalan, A.N., Kiran, V.P.: Motion-free superresolution and the role of relative blur. JOSA A 20(11), 2022–2032 (2003)
  * [37] Rajagopalan, A., Chellappa, R., Koterba, N.T.: Background learning for robust face recognition with pca in the presence of clutter. IEEE Transactions on Image Processing 14(6), 832–843 (2005)
  * [38] Sajjadi, M.S.M., Schölkopf, B., Hirsch, M.: Enhancenet: Single image super-resolution through automated texture synthesis. In: 2017 IEEE International Conference on Computer Vision (ICCV). pp. 4501–4510 (Oct 2017). https://doi.org/10.1109/ICCV.2017.481
  * [39] Suresh, K.V., Rajagopalan, A.N.: Robust and computationally efficient superresolution algorithm. JOSA A 24(4), 984–992 (2007)
  * [40] Tai, Y., Yang, J., Liu, X.: Image super-resolution via deep recursive residual network. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 2790–2798 (July 2017). https://doi.org/10.1109/CVPR.2017.298
  * [41] Tai, Y., Yang, J., Liu, X., Xu, C.: Memnet: A persistent memory network for image restoration. In: 2017 IEEE International Conference on Computer Vision (ICCV). pp. 4549–4557 (Oct 2017). https://doi.org/10.1109/ICCV.2017.486
  * [42] Timofte, R., Agustsson, E., Gool, L.V., Yang, M., Zhang, L., et al.: Ntire 2017 challenge on single image super-resolution: Methods and results. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). pp. 1110–1121 (July 2017). https://doi.org/10.1109/CVPRW.2017.149
  * [43] Vasu, S., Thekke Madam, N., Rajagopalan, A.: Analyzing perception-distortion tradeoff using enhanced perceptual super-resolution network. In: Proceedings of the European Conference on Computer Vision (ECCV) Workshops. pp. 0–0 (2018)
  * [44] Wang, W., Li, X., Yang, J., Lu, T.: Mixed link networks. arXiv preprint arXiv:1802.01808 (2018)
  * [45] Wang, Z., Liu, D., Yang, J., Han, W., Huang, T.: Deep networks for image super-resolution with sparse prior. In: 2015 IEEE International Conference on Computer Vision (ICCV). pp. 370–378 (Dec 2015). https://doi.org/10.1109/ICCV.2015.50
  * [46] Yang, J., Wright, J., Huang, T., Ma, Y.: Image super-resolution via sparse representation. IEEE Transactions on Image Processing 19(11), 2861–2873 (Nov 2010). https://doi.org/10.1109/TIP.2010.2050625
  * [47] Zeyde, R., Elad, M., Protter, M.: On single image scale-up using sparse-representations. In: Boissonnat, J.D., Chenin, P., Cohen, A., Gout, C., Lyche, T., Mazure, M.L., Schumaker, L. (eds.) Curves and Surfaces. pp. 711–730. Springer Berlin Heidelberg, Berlin, Heidelberg (2012)
  * [48] Zhang, K., Zuo, W., Zhang, L.: Learning a single convolutional super-resolution network for multiple degradations. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 3262–3271 (June 2018)
  * [49] Zhang, X., Lam, E., Wu, E., Wong, K.: Application of Tikhonov regularization to super-resolution reconstruction of brain MRI images. In: Gao, X., Müller, H., Loomes, M., Comley, R., Luo, S. (eds.) Medical Imaging and Informatics, Lecture Notes in Computer Science, vol. 4987, pp. 51–56. Springer Berlin Heidelberg (2008). https://doi.org/10.1007/978-3-540-79490-5_8
  * [50] Zhang, Y., Tian, Y., Kong, Y., Zhong, B., Fu, Y.: Residual dense network for image super-resolution. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)