# Electron-induced non-monotonic pressure dependence of the lattice thermal
conductivity of $\theta$-TaN

Ashis Kundu2,3 Yani Chen2,1 Xiaolong Yang4 Fanchen Meng5,6 Jesús Carrete7,8
Mukul Kabir3 Georg K. H. Madsen8 Wu Li1,2<EMAIL_ADDRESS>1Eastern
Institute for Advanced Study, Eastern Institute of Technology, Ningbo 315200,
China  2Institute for Advanced Study, Shenzhen University, Shenzhen 518060,
China  3Department of Physics, Indian Institute of Science Education and
Research (IISER) Pune, P.O. 411008, India  4College of Physics and Center of
Quantum Materials and Devices, Chongqing University, Chongqing 401331, China
5Research Computing and Data, Clemson University, Clemson, SC, 29634, USA
6Center for Functional Nanomaterials, Brookhaven National Laboratory, Upton,
New York 11973, USA  7Instituto de Nanociencia y Materiales de Aragón (INMA),
CSIC-Universidad de Zaragoza, E-50009 Zaragoza, Spain  8Institute of Materials
Chemistry, TU Wien, A-1060 Vienna, Austria

###### Abstract

Recent theoretical and experimental research suggests that $\theta$-TaN is a
semimetal with high thermal conductivity ($\kappa$), primarily due to the
contribution of phonons ($\kappa_{\texttt{ph}}$). By using first-principles
calculations, we show a non-monotonic pressure dependence of the $\kappa$ of
$\theta$-TaN. $\kappa_{\texttt{ph}}$ first increases until it reaches a
maximum at around 60 GPa, and then decreases. This anomalous behaviour is a
consequence of the competing pressure responses of phonon-phonon and phonon-
electron interactions, in contrast to the known materials BAs and BP, where
the non-monotonic pressure dependence is caused by the interplay between
different phonon-phonon scattering channels. Although TaN has phonon
dispersion features similar to BAs at ambient pressure, its response to
pressure is different and an overall stiffening of the phonon branches takes
place. Consequently, the relevant phonon-phonon scattering weakens as pressure
increases. However, the increased electronic density of states near the Fermi
level, and specifically the emergence of additional pockets of the Fermi
surface at the high-symmetry L point in the Brillouin zone, leads to a
substantial increase in phonon-electron scattering at high pressures, driving
a decrease in $\kappa_{\mathrm{ph}}$. At intermediate pressures ($\sim$
20$-$70 GPa), the $\kappa$ of TaN surpasses that of BAs. Our work provides
deeper insight into phonon transport in semimetals and metals where phonon-
electron scattering is relevant.

The pressure response of materials is critical to understanding their
properties and performance in many areas, including condensed-matter physics,
materials science, and geophysics [1, 2, 3, 4, 5, 6, 7]. Among the many
properties affected by pressure, the thermal conductivity ($\kappa$) of a
material stands out as a fundamental transport parameter for heat conduction.
$\kappa$ is limited by resistive scattering between intrinsic carriers such as
phonons and electrons [8]. Modulation of $\kappa$ with pressure and
temperature over a wide range can help in understanding heat conduction in the
Earth’s interior [5, 6] and developing mechanisms to control thermal
management in microelectronic devices [9, 10, 11].

Under hydrostatic pressure, a crystal’s volume decreases, generally resulting
in a hardening of the phonon modes. This typically leads to an increase in
lattice $\kappa$ unless a structural phase transition occurs [12, 8]. This
intuition is supported by various model calculations, although those have also
shown that a large mass ratio can reverse the trend [13]. In fact, a recent
ab-initio study has revealed a non-monotonic pressure dependence of $\kappa$
for BAs, BSb and BP, with BAs and BSb share a similar microscopic origin
underlying this behavior, which differs from that of BP [14, 15]. Experimental
observations for BAs also support this finding [16].

In our previous work, we predicted that the metallic compound TaN has an
ultrahigh thermal conductivity, mainly contributed by phonons and higher than
that of the best electrical conductor, silver. This was found to be due to its
unique combination of phononic, electronic, and nuclear properties [17]. Among
the transition-metal carbides and nitrides, TaN was found to be the only
compound with such high $\kappa$. Our research also revealed that the
mechanism responsible for TaN’s ultrahigh $\kappa$ is similar to that of BAs
[18], with an additional factor of weak phonon-electron scattering. Recent
experiments have indicated that this predicted high ambient-pressure value of
$\kappa$ for TaN can be achieved by increasing the grain size of samples and
decreasing their defect concentrations [19].

In this study, we investigate the impact of pressure on $\kappa$ in TaN using
first-principles calculations. Our findings reveal a non-monotonic pressure
dependence of $\kappa$ due to competing responses of phonon-phonon and phonon-
electron interactions. However, the hardening of all phonon modes and the
decrease in phonon scattering strength with increased pressure indicate that
the non-monotonic behavior arises from strong phonon-electron scattering at
higher pressure, which is influenced by the increased electronic density of
states around the Fermi energy. Thus our work provides a new perspective on
the non-monotonic pressure dependence of $\kappa$ in TaN.

In the framework of the linearized Boltzmann transport equation (BTE), the
lattice thermal conductivity tensor ($\kappa_{\texttt{ph}}^{\alpha\beta}$) is
expressed as [20]

$\kappa_{\texttt{ph}}^{\alpha\beta}=\frac{1}{8\pi^{3}}\int\limits_{\mathrm{BZ}}\sum\limits_{p}C_{V}\left(p\mathbf{q}\right)v^{\alpha}_{p\mathbf{q}}F^{\beta}_{p\mathbf{q}}d^{3}\mathbf{q},$
(1)

where $p$ runs over all phonon branches, BZ denotes the Brillouin zone of the
crystal, $\alpha$ and $\beta$ are Cartesian axes, and $\omega_{p\mathbf{q}}$,
$C_{V}\left(p\mathbf{q}\right)$, $v^{\alpha}_{p\mathbf{q}}$, and
$F^{\beta}_{p\mathbf{q}}$ are the angular frequency, mode heat capacity at
constant volume, group velocity and mean free displacement, respectively, of
phonons from branch $p$ with wave vector $\mathbf{q}$.

Phonon scattering is the main ingredient for determining
$\kappa_{\mathrm{ph}}$. It enters the expression above through
$F^{\beta}_{p\mathbf{q}}$. $F^{\beta}_{p\mathbf{q}}$ is determined by [20]

$F^{\beta}_{p\mathbf{q}}=\tau_{p\mathbf{q}}\left(v^{\beta}_{p\mathbf{q}}+\Delta^{\beta}_{p\mathbf{q}}\right).$
(2)

Here, $\tau_{p\mathbf{q}}$ is the phonon lifetime and
$\Delta^{\beta}_{p\mathbf{q}}$ depends linearly on $F^{\beta}_{p\mathbf{q}}$
and describes the deviation from the relaxation-time approximation (RTA). We
obtain $F^{\beta}_{p\mathbf{q}}$ iteratively starting from the RTA [20, 21].

The inverse of $\tau_{p\mathbf{q}}$ is the scattering rate and is expressed as
a sum of the contributions from the different scattering mechanisms. We
include all the essential scattering mechanisms to calculate the thermal
conductivity. Specifically, we take three-phonon (3ph), four-phonon (4ph),
phonon-isotope (ph-iso), and phonon-electron (ph-el) scattering into account
in our calculations.

$\frac{1}{\tau_{p\mathbf{q}}}=\frac{1}{\tau^{\mathrm{3ph}}_{p\mathbf{q}}}+\frac{1}{\tau^{\mathrm{4ph}}_{p\mathbf{q}}}+\frac{1}{\tau^{\mathrm{ph\mbox{-}iso}}_{p\mathbf{q}}}+\frac{1}{\tau^{\mathrm{ph\mbox{-}el}}_{p\mathbf{q}}}.$
(3)

Phonon-electron scattering involves only one phonon and does not affect
$\Delta^{\beta}_{p\mathbf{q}}$ [22]. Furthermore, four-phonon scattering is
dominated by Umklapp processes[18, 23] and their contribution to
$\Delta^{\beta}_{p\mathbf{q}}$ can be neglected. The detailed expressions for
$\Delta_{p\mathbf{q}}$, $1/\tau^{\mathrm{ph\mbox{-}iso}}_{p\mathbf{q}}$,
$1/\tau^{\mathrm{3ph}}_{p\mathbf{q}}$, $1/\tau^{\mathrm{4ph}}_{p\mathbf{q}}$,
$1/\tau^{\mathrm{ph\mbox{-}el}}_{p\mathbf{q}}$ can be found in Refs. 24, 20,
21, 25, 22, 26, 18.

To solve the phonon BTE, we need harmonic (second-order) and anharmonic
(third- and fourth-order) interatomic force constants (IFCs), which are
extracted from density-functional-theory calculations carried out using the
projector-augmented plane wave method [27] as implemented in VASP [28, 29].
For the calculation of phonon-electron scattering, we used the EPW software
[30], based on maximally localized Wannier functions, along with the quantum
espresso package [31]. We used the FourPhonon package [32] to compute the
four-phonon scattering rates. Finally, we used a modified ShengBTE package
[20] for calculating $\kappa_{\mathrm{ph}}$ after including phonon-electron
and four-phonon scattering rates. The temperature induced phonon
renormalization effect is considered using self-consistent phonon (SCPH)
theory [33, 34], and it is found to have a minimal impact on
$\kappa_{\mathrm{ph}}$ (see Supplemental Material [35]). The coherence effect
of the phonons [36, 37, 38, 39] is not considered in our calculations as we
estimate that at ambient pressure the average phonon interband spacing ($\sim
1.5-2$ THz) in TaN is orders of magnitude larger than the phonon linewidths
($\sim 0.001-0.1$ THz) and can be considered negligible. Further computational
details, including our approach to calculating the electronic contribution to
thermal transport, are given in the Supplemental Material [35].

Figure 1: Calculated room-temperature total thermal conductivity
($\kappa=\kappa_{\texttt{ph}}+\kappa_{\texttt{e}}$) as a function of pressure
for TaN along the a and c axes in comparison with available results of
$\kappa$ under pressure. Theoretical results of $\kappa$ for diamond [40], Al
[41], BP [15], c-BN [14], and BAs [14] and experimental results for BAs
(right-pointing triangles [16]) under pressure are also shown.

Figure 1 shows the calculated pressure dependence of the total thermal
conductivity ($\kappa$) of TaN compared to diamond [40], BAs [14, 16], c-BN
[14], BP [15], and Al [41], which have been reported in the literature. At
ambient pressure, the room-temperature total $\kappa$ of TaN is 1037 and 824
$\mathrm{W\,m^{-1}\,K^{-1}}$ along the a and c axes, respectively, consistent
with the values reported earlier [17, 19]. The electronic contributions to the
thermal conductivity ($\kappa_{\texttt{el}}$) at ambient pressure are 27 and
15 $\mathrm{W\,m^{-1}\,K^{-1}}$, respectively. They remain negligible at high
pressures (see Supplemental Material [35]). Interestingly, the non-monotonic
behavior revealed originally in BAs [14, 16] and BP [15] also occurs in TaN.
However, that non-monotonic behavior diminishes at high temperatures. Above 20
GPa, TaN overtakes BAs as the bulk material with the second largest $\kappa$.
In contrast, $\kappa$ increases monotonically in diamond [40], c-BN [14], and
Al [41]. $\kappa$ is dominated almost exclusively by the electronic
contribution in metals. The increase of $\kappa$ in Al is mainly attributable
to the decrease of phonon-electron coupling. In usual cases like diamond and
c-BN, the phonon branches stiffen, and the three-phonon scattering weakens
under pressure. A monotonic increase in $\kappa$ with pressure follows. In BP,
a gap between acoustic and optical modes ($a$-$o$ gap) is present in the
phonon dispersion at ambient pressure. The widening of the $a$-$o$ gap with
rising pressure decreases the scattering involving two acoustic and one
optical phonon ($aao$), driving $\kappa$ to increase at low pressures. In BAs,
the $a$-$o$ gap is so large that $aoo$ scattering is irrelevant. However,
four-phonon scattering plays an important role in $\kappa$. The thermal
conductivity increases with increasing pressures in the low-pressure regime
because of the progressive weakening of four-phonon scattering. In both BP and
BAs, the reduced bunching (the proximity of acoustic phonon branches) in the
phonon dispersion with increasing pressure, leading to enhanced scattering
among three acoustic phonons ($aaa$ processes), is the origin of the
decreasing $\kappa$ at high pressure.

Figure 2: (a) Calculated lattice thermal conductivity ($\kappa_{\texttt{ph}}$)
and (b) scaled $\kappa_{\texttt{ph}}$ for TaN along the a axis for different
pressures with different combinations of scattering mechanisms at 300 and 800
K.

TaN has a phonon dispersion similar to BAs as regards the presence of a large
$a$-$o$ gap and the bunching of acoustic phonons, a key characteristic leading
to high $\kappa$ in TaN and BAs [17]. To see whether the mechanism underlying
the non-monotonic change of $\kappa$ in TaN is the same as in BAs, we show the
pressure-dependent $\kappa_{\texttt{ph}}$ calculated by considering only
specific combinations of phonon scattering mechanisms for two different
temperatures, 300 and 800 K, in Fig. 2 (a). Surprisingly, when only
considering three phonon scattering, $\kappa_{\texttt{ph}}$ constantly
increases with pressure in TaN, in contrast to BAs. Similar to BAs, the four-
phonon scattering plays a vital role in determining $\kappa_{\texttt{ph}}$ in
TaN. At ambient pressure, $\kappa_{\texttt{ph}}$ is reduced by 48% after
including four-phonon scattering on top of three-phonon scattering. Although
four-phonon scattering has a reduced effect on $\kappa_{\texttt{ph}}$ at
higher pressures, it can still lead to a 27% reduction in
$\kappa_{\texttt{ph}}$ at 80 GPa as compared to the case with three-phonon
scattering only. By comparison, in BAs the effect of four-phonon scattering is
weak at higher pressure (e.g. 17% at 51 GPa). When further including phonon-
electron scattering, the influence on $\kappa_{\texttt{ph}}$ begins to
manifest at 40 GPa. Above 60 GPa, $\kappa_{\texttt{ph}}$ can be significantly
reduced by phonon-electron scattering, which is so strong as to cause
$\kappa_{\texttt{ph}}$ to decrease with pressure. Therefore, the non-monotonic
pressure dependence of $\kappa_{\texttt{ph}}$ is a consequence of the
interplay between the phonon-phonon and phonon-electron scattering, rather
than between three-phonon and four-phonon scattering channels as in BAs [14],
and the one between $aaa$ and $aao$ scattering channels as in BP [15]. Next,
we will elucidate why $\kappa_{\texttt{ph}}^{(3)}$ in TaN behaves differently
from BAs and why phonon-electron scattering gives rise to decreasing
$\kappa_{\texttt{ph}}$ at high pressures.

Figure 3: (a) Calculated phonon dispersion along high-symmetry directions for
TaN for different pressures.The size of circles superimposed onto the line
corresponds to phonon linewidth due to phonon-electron interaction. The
linewidths of acoustic and optical phonons are scaled by factors of 200 and
20, respectively. (b) Calculated three-phonon (3ph) and four-phonon (4ph)
scattering rates for TaN for different pressures at 300 K.

Due to a large $a$-$o$ gap combined with the small bandwidth of optical phonon
frequencies, the three-phonon scattering of the heat-carrying acoustic phonons
is limited by $aaa$ processes in BAs [14]. Further, due to bunching, the $aaa$
scattering rates are extremely weak, with a characteristic dip in the
intermediate frequency range. With increasing pressure, the longitudinal
acoustic phonons stiffen evidently while the transverse acoustic phonons
change marginally. Therefore, the bunching effect is significantly reduced,
increasing the scattering phase space and causing the $aaa$ scattering rates
to increase fast.

In the case of TaN, the phonon dispersion and three-phonon scattering rates
for different pressures are plotted in Fig. 3 (a) and (b), respectively. The
$a$-$o$ gap is not large enough to completely suppress the $aao$ scattering at
high acoustic frequencies, and the $aao$ scattering is reduced partially due
to a larger $a$-$o$ gap under pressure (see Supplemental Material [35]). The
optical bandwidth even allows $aoo$ processes to dominate the scattering rates
at low frequencies, and the $aoo$ scattering is strengthened partially because
of a larger optical band width under pressure. However, the primary
contribution to $\kappa_{\texttt{ph}}^{(3)}$ arises from acoustic phonons in
an intermediate frequency range, for which scattering is almost exclusively
limited by $aaa$ processes. Under applied pressure, all acoustic phonon modes
experience hardening at the same pace, as manifested by the fact that the
acoustic phonon dispersions can overlap if scaled properly [Fig. 3 (a)](see
Supplemental Material [35]). This indicates that the bunching of acoustic
modes remains nearly unchanged with increasing pressure in TaN. The difference
in the response of the phonon dispersion, particularly the acoustic part to
pressure, gives rise to different responses of the scattering rates to
pressure. The $aaa$ scattering rate for a given q point decreases with
pressure in TaN (see Supplemental Material [35]) compared to the increasing
behavior in BAs. As a consequence, $\kappa_{\texttt{ph}}^{(3)}$ monotonically
increases with pressure in TaN.

As shown in Fig. 2 (a), the four-phonon scattering plays an important role in
determining $\kappa_{\texttt{ph}}$ in TaN throughout the whole pressure range
considered, and the four-phonon effect becomes weaker at higher pressure. To
understand this, the four-phonon scattering rates are plotted in Fig. 3 (b).
Similarly to the three-phonon scattering, especially for the $aaa$ processes,
the four-phonon scattering rates decrease with increasing pressure but more
rapidly. This stronger pressure dependence of four-phonon scattering leads to
a weaker effect at higher pressure. Consequently, the actual
$\kappa_{\texttt{ph}}$ has a stronger pressure dependence than it would have
if only three-phonon scattering was considered. This is more apparent when
plotting the pressure-dependent $\kappa_{\texttt{ph}}$ scaled by the
corresponding zero-pressure value with specific combinations of phonon
scattering mechanisms considered [Fig. 2 (b)]. As a matter of fact,
$\mathrm{d}\kappa/\mathrm{d}P$ for TaN is 17.3 and 14.2
$\mathrm{W\,m^{-1}\,K^{-1}\,GPa^{-1}}$ along the a and c axes, respectively,
which are similar to the values of approximately 19.6, 15.4, and 12.6
$\mathrm{W\,m^{-1}\,K^{-1}\,GPa^{-1}}$ for diamond [40], BP [15], and BAs
[14], respectively. In addition to the stronger pressure dependence, the four-
phonon scattering also has a stronger $T$ dependence than the three-phonon
scattering. Therefore, the actual $\kappa_{\texttt{ph}}$ has a stronger
pressure dependence at higher temperatures for BAs. In contrast, the pressure
dependence of $\kappa_{\texttt{ph}}^{(3)}$ for TaN is almost $T$ independent
[Fig. 2 (b)].

Figure 4: (a) Calculated electronic DOS for TaN for different pressures. Ef
denotes the Fermi energy. (b) First Brillouin zone of TaN with high-symmetry
points labeled. (c) Calculated Fermi surfaces for TaN for different pressures.

The fact that four-phonon scattering has a stronger pressure dependence than
three-phonon scattering can be understood from the weighted phase space[20,
42, 43], which is extremely sensitive to changes in the phonon spectrum. For
example, if the entire spectrum is scaled by a factor of $c$, then the three-
phonon weighted phase space is scaled by a factor of $c^{-5}$, while the four-
phonon weighted phase space can be scaled by $c^{-6}$. At 80 GPa, the acoustic
phonon frequency increased by a factor of $1.215$ compared to the frequency at
0 GPa. The ratios $1.215^{5}=2.70$ and $1.215^{6}=3.30$ closely correspond to
the obtained weighted phase space ratios of $2.5$ and $3.6$ for three-phonon
and four-phonon, respectively, between 0 and 80 GPa. Hence, the higher
weighted phase-space ratio suggests that four-phonon scattering weakens faster
than three-phonon scattering with increasing pressure (see Supplemental
Material [35]).

Finally, we study the phonon-electron scattering, leading to decreasing
$\kappa_{\texttt{ph}}$ at high pressures. At ambient pressure, phonon-electron
scattering is negligible compared to anharmonic phonon-phonon scattering. This
weak phonon-electron scattering is actually a key factor resulting in the high
value of $\kappa$ for TaN compared to WC with the same type of crystalline
structure, as established in our previous work [44, 17]. Phonon-electron
scattering is positively correlated with the electronic density of states
(DOS) at the Fermi level [45, 46, 47, 44]. The weak phonon-electron scattering
is attributed to the direct $d-d$ bonding between Ta atoms, resulting in a
semimetallic band structure and a low DOS around the Fermi energy [48].
However, the phonon-electron scattering rates can be increased as pressure
increases (see Supplemental Material [35]). Fig. 4 (a) illustrates that there
is a simultaneous increase in the DOS at the Fermi level with increasing
pressure. Consequently, the DOS around the Fermi energy gradually increases
and starts to become significant at 40 GPa. Therefore, the increased DOS and
subsequently higher phonon-electron scattering rates are at the origin of the
decrease of $\kappa_{\texttt{ph}}$ at high pressures in TaN. Note that the
phonon-electron scattering rates are generally insignificant even in metals
with large DOS at the Fermi level [25, 46, 49]. Only a few expectations have
been reported recently in systems with weak anharmonic scattering, including
transition metal carbides [50, 44] and tungsten [22].

To understand the increase in DOS around the Fermi energy and its effect on
phonon-electron scattering, we examine the changes in Fermi surfaces and the
mode projected phonon linewidth [51] due to phonon-electron interaction with
increasing pressure, shown in Fig. 4 (c) and Fig. 3 (a), respectively. The
phonon-electron scattering rates are proportional to the phonon linewidth
through a factor of $2/\hbar$ [25]. As pressure rises, additional pockets of
the Fermi surface appear at L point, in line with band structure’s evolution
where the band around the L point intersects the Fermi level due to increased
Ta-$d$ ($d_{xy}+d_{x2-y2}$) and N-$p$ ($p_{z}$) hybridization (see
Supplemental Material [35]). Consequently, we observe a marked enhancement in
linewidths of acoustic and, much more evidently, the optical phonons at
$\Gamma$ and M points beginning at 40 GPa [Fig. 3 (a)]. This is a consequence
of intravalley and intervalley electron scatterings which are associated with
phonons around $\Gamma$ and M, K, and H, respectively. For instance,
considering the symmetry, twelve equivalent L points exist in the Brillouin
zone [Fig. 4 (b)]. Each electron state at L can be linked to its nearest
counterpart at L through a phonon at the M point (see Supplemental Material
[35]). A similar scenario unfolds around $\Gamma$, K and H phonon states. The
increase in phonon linewidth at $\Gamma$ is associated with intravalley
scattering between identical electron states located at different high
symmetry points. Conversely, a slight reduction in phonon linewidth around K
and H is linked to intervalley scattering between two electron states around A
and K points, corresponding to the contraction of the Fermi surface area.
However, the subsequent decrease in phonon linewidth due to phonon-electron
interaction at the K and H points has less influence than the emergent phonon
linewidth at M and $\Gamma$ points. Consequently, we observe a marked
reduction in the slope of $\kappa_{\texttt{ph}}$ from 40 GPa when considering
phonon-electron scattering rates [Fig. 2 (b)]. Note that only effects on the
acoustic phonons are relevant to the thermal conduction, although the optical
phonons have larger linewidths due to phonon-electron interaction than
acoustic phonons do.

In summary, starting from first-principles calculations, we reveal a non-
monotonic pressure dependence of $\kappa$ of TaN similar to that observed in
BAs and BP but due to a different mechanism. This anomalous behaviour results
from the competing responses of phonon-phonon and phonon-electron scattering
to pressure, in contrast to the interplay between different phonon-phonon
scattering channels in BAs and BP. The overall phonon dispersion stiffens
almost at the same pace, leading to continuously weakened three-acoustic-
phonon and four-phonon scatterings, giving rise to an initially increasing
$\kappa$. At higher pressures, the increased electronic DOS, explicitly the
emergence of additional pockets of the Fermi surface at the high-symmetry L
point in the Brillouin zone, leading to an increase in phonon-electron
scattering and consequently driving a decrease in $\kappa_{\mathrm{ph}}$. The
$\kappa$ of TaN can surpass that of BAs in the range $\sim$ 20$-$70 GPa.
Finally, it is worth noting that TaN has been synthesized and remains stable
at high pressures [52, 53, 19]. Our reported anomalous pressure dependence of
$\kappa$ occurs at around 60 GPa, and thus within the experimentally
accessible range. The non-monotonic pressure dependence may also be observed
in some other metals, where the electronic DOS changes suddenly with pressure.

The supporting data for this paper are available from Zenodo [59] under open
licenses. The data package includes the structure file, second-, third- and
fourth-order IFCs and phonon-electron scattering rates for each pressure.

We acknowledge support from the National Key R&D Program of China (Grant No.
2023YFA1407001), the Natural Science Foundation of China (NSFC) (Grants No.
12174261 and No.12104312), and the Guangdong Basic and Applied Basic Research
Foundation (Grants No. 2021A1515010042, No. 2023A1515010365, and No.
2022A1515011877). A. K. and M. K gratefully acknowledge the support and
resources provided by the PARAM Brahma Facility at the IISER, Pune, under the
National Supercomputing Mission of the Government of India. A. K. acknowledges
CHANAKYA Post-Doctoral fellowship from the National Mission on
Interdisciplinary Cyber-Physical Systems (NM-ICPS) of the Department of
Science and Technology, Government of India, through the I-HUB Quantum
Technology Foundation, Pune, India. X.Y. acknowledges support from the NSFC
(Grants No. 12374038 and No. 12147102).

## References

  * Zhang  _et al._ [2017] L. Zhang, Y. Wang, J. Lv, and Y. Ma, Nat. Rev. Mater. 2, 17005 (2017).
  * Miao  _et al._ [2020] M. Miao, Y. Sun, E. Zurek, and H. Lin, Nat. Rev. Chem. 4, 508 (2020).
  * Knudson  _et al._ [2015] M. D. Knudson, M. P. Desjarlais, A. Becker, R. W. Lemke, K. R. Cochrane, M. E. Savage, D. E. Bliss, T. R. Mattsson, and R. Redmer, Science 348, 1455 (2015).
  * Fratanduono  _et al._ [2021] D. E. Fratanduono, M. Millot, D. G. Braun, S. J. Ali, A. Fernandez-Pañella, C. T. Seagle, J.-P. Davis, J. L. Brown, Y. Akahama, R. G. Kraus, M. C. Marshall, R. F. Smith, E. F. O’Bannon, J. M. McNaney, and J. H. Eggert, Science 372, 1063 (2021).
  * Pozzo  _et al._ [2012] M. Pozzo, C. Davies, D. Gubbins, and D. Alfè, Nature 485, 355 (2012).
  * Konôpková  _et al._ [2016] Z. Konôpková, R. S. McWilliams, N. Gómez-Pérez, and A. F. Goncharov, Nature 534, 99 (2016).
  * Zhou  _et al._ [2022] Y. Zhou, Z.-Y. Dong, W.-P. Hsieh, A. F. Goncharov, and X.-J. Chen, Nat. Rev. Phys. 4, 319 (2022).
  * Ziman [1960] J. M. Ziman,  _Electrons and phonons: the theory of transport phenomena in solids_ (Oxford university press, 1960).
  * Ball [2012] P. Ball, Nature News 492, 174 (2012).
  * Garimella  _et al._ [2012] S. V. Garimella, L.-T. Yeh, and T. Persoons, IEEE Transactions on Components, Packaging and Manufacturing Technology 2, 1307 (2012).
  * Waldrop [2016] M. M. Waldrop, Nature News 530, 144 (2016).
  * Bridgman [1935] P. W. Bridgman, Rev. Mod. Phys. 7, 1 (1935).
  * Lindsay  _et al._ [2015] L. Lindsay, D. A. Broido, J. Carrete, N. Mingo, and T. L. Reinecke, Phys. Rev. B 91, 121202 (2015).
  * Ravichandran and Broido [2019] N. K. Ravichandran and D. Broido, Nat. Commun. 10, 827 (2019).
  * Ravichandran and Broido [2021] N. K. Ravichandran and D. Broido, Nat. Commun. 12, 3473 (2021).
  * Li  _et al._ [2022] S. Li, Z. Qin, H. Wu, M. Li, M. Kunz, A. Alatas, A. Kavner, and Y. Hu, Nature 612, 459 (2022).
  * Kundu  _et al._ [2021] A. Kundu, X. Yang, J. Ma, T. Feng, J. Carrete, X. Ruan, G. K. H. Madsen, and W. Li, Phys. Rev. Lett. 126, 115901 (2021).
  * Feng  _et al._ [2017] T. Feng, L. Lindsay, and X. Ruan, Phys. Rev. B 96, 161201 (2017).
  * Lee  _et al._ [2023] H. Lee, Y. Zhou, S. Jung, H. Li, Z. Cheng, J. He, J. Chen, P. Sokalski, A. Dolocan, R. Gearba-Dolocan, K. C. Matthews, F. Giustino, J. Zhou, and L. Shi, Adv. Funct. Mater. 33, 2212957 (2023).
  * Li  _et al._ [2014] W. Li, J. Carrete, N. A. Katcho, and N. Mingo, Comput. Phys. Commun. 185, 1747 (2014).
  * Li [2015] W. Li, Phys. Rev. B 92, 075405 (2015).
  * Chen  _et al._ [2019a] Y. Chen, J. Ma, and W. Li, Phys. Rev. B 99, 020305 (2019a).
  * Yang  _et al._ [2019] X. Yang, T. Feng, J. Li, and X. Ruan, Phys. Rev. B 100, 245203 (2019).
  * Ward  _et al._ [2009] A. Ward, D. A. Broido, D. A. Stewart, and G. Deinzer, Phys. Rev. B 80, 125203 (2009).
  * Liao  _et al._ [2015] B. Liao, B. Qiu, J. Zhou, S. Huberman, K. Esfarjani, and G. Chen, Phys. Rev. Lett. 114, 115901 (2015).
  * Feng and Ruan [2016] T. Feng and X. Ruan, Phys. Rev. B 93, 045202 (2016).
  * Blöchl [1994] P. E. Blöchl, Phys. Rev. B 50, 17953 (1994).
  * Kresse and Furthmüller [1996] G. Kresse and J. Furthmüller, Phys. Rev. B 54, 11169 (1996).
  * Kresse and Joubert [1999] G. Kresse and D. Joubert, Phys. Rev. B 59, 1758 (1999).
  * Poncé  _et al._ [2016] S. Poncé, E. Margine, C. Verdi, and F. Giustino, Comput. Phys. Commun. 209, 116 (2016).
  * Giannozzi  _et al._ [2009] P. Giannozzi, S. Baroni, N. Bonini, M. Calandra, R. Car, C. Cavazzoni, D. Ceresoli, G. L. Chiarotti, M. Cococcioni, I. Dabo, A. D. Corso, S. de Gironcoli, S. Fabris, G. Fratesi, R. Gebauer, U. Gerstmann, C. Gougoussis, A. Kokalj, M. Lazzeri, L. Martin-Samos, N. Marzari, F. Mauri, R. Mazzarello, S. Paolini, A. Pasquarello, L. Paulatto, C. Sbraccia, S. Scandolo, G. Sclauzero, A. P. Seitsonen, A. Smogunov, P. Umari, and R. M. Wentzcovitch, J. Phys. Condens. Matter 21, 395502 (2009).
  * Han  _et al._ [2022] Z. Han, X. Yang, W. Li, T. Feng, and X. Ruan, Comput. Phys. Commun. 270, 108179 (2022).
  * Errea  _et al._ [2014] I. Errea, M. Calandra, and F. Mauri, Phys. Rev. B 89, 064302 (2014).
  * Eriksson  _et al._ [2019] F. Eriksson, E. Fransson, and P. Erhart, Adv. Theor. Simul. 2, 1800184 (2019).
  * [35] See Supplemental Material at URL for detailed information, including computational details, convergence testing of $\kappa_{\texttt{ph}}$ relative to the cutoff for third-order IFCs, fine grid size used for calculating phonon-electron scattering rates, lattice parameters, temperature-dependent contributions to thermal conductivity, scaled phonon dispersion, phonon renormalization, process-specific scattering rates and spectral contributions, weighted phase space, phonon-electron scattering rates, electronic band structures, and Fermi surfaces for different pressures, which includes Refs. [28, 29, 54, 20, 32, 26, 30, 31, 55, 56, 22, 21, 57, 58, 52, 33, 34, 42, 43].
  * Simoncelli  _et al._ [2019] M. Simoncelli, N. Marzari, and F. Mauri, Nat. Phys. 15, 809 (2019).
  * Isaeva  _et al._ [2019] L. Isaeva, G. Barbalinardo, D. Donadio, and S. Baroni, Nat. Commun. 10, 3853 (2019).
  * Fiorentino and Baroni [2023] A. Fiorentino and S. Baroni, Phys. Rev. B 107, 054311 (2023).
  * Simoncelli  _et al._ [2023] M. Simoncelli, F. Mauri, and N. Marzari, npj Comput. Mater. 9, 106 (2023).
  * Broido  _et al._ [2012] D. A. Broido, L. Lindsay, and A. Ward, Phys. Rev. B 86, 115203 (2012).
  * Giri  _et al._ [2022] A. Giri, P. Karna, and P. E. Hopkins, J. Phys. Chem. Lett. 13, 10918 (2022), pMID: 36394985.
  * Li and Mingo [2014a] W. Li and N. Mingo, Phys. Rev. B 89, 184304 (2014a).
  * Li and Mingo [2014b] W. Li and N. Mingo, Phys. Rev. B 90, 094302 (2014b).
  * Kundu  _et al._ [2020] A. Kundu, J. Ma, J. Carrete, G. K. H. Madsen, and W. Li, Mater. Today Phys. 13, 100214 (2020).
  * Xu and Verstraete [2014] B. Xu and M. J. Verstraete, Phys. Rev. Lett. 112, 196603 (2014).
  * Wang  _et al._ [2016] Y. Wang, Z. Lu, and X. Ruan, J. Appl. Phys. 119, 225109 (2016).
  * Dongre  _et al._ [2020] B. Dongre, J. Carrete, S. Wen, J. Ma, W. Li, N. Mingo, and G. K. H. Madsen, J. Mater. Chem. A 8, 1273 (2020).
  * Wijeyesekera and Hoffmann [1984] S. D. Wijeyesekera and R. Hoffmann, Organometallics 3, 949 (1984).
  * Jain and McGaughey [2016] A. Jain and A. J. H. McGaughey, Phys. Rev. B 93, 081206 (2016).
  * Li  _et al._ [2018] C. Li, N. K. Ravichandran, L. Lindsay, and D. Broido, Phys. Rev. Lett. 121, 175901 (2018).
  * Chen  _et al._ [2019b] Y. Chen, J. Ma, S. Wen, and W. Li, npj Comput. Mater. 5, 1 (2019b).
  * hao Feng  _et al._ [2018] L.-H. Feng, Q.-W. Hu, L. Lei, L.-M. Fang, L. Qi, L.-L. Zhang, M.-F. Pu, Z.-L. Kou, F. Peng, X.-P. Chen, Y.-H. Xia, Y. Kojima, H. Ohfuji, D.-W He, B. Chen, and T. Irifune, Chin. Phys. B 27, 026201 (2018).
  * Jia  _et al._ [2019] Y.-T. Jia, J.-F. Zhao, S.-J. Zhang, S. Yu, G.-Y. Dai, W.-M. Li, L. Duan, G.-Q. Zhao, X.-C. Wang, X. Zheng, Q.-Q. Liu, Y.-W. Long, Z. Li, X.-D. Li, H.-M. Weng, R.-Z. Yu, R.-C. Yu, and C.-Q. Jin, Chin. Phys. Lett. 36, 087401 (2019).
  * Togo and Tanaka [2015] A. Togo and I. Tanaka, Scr. Mater. 108, 1 (2015).
  * Baroni  _et al._ [2001] S. Baroni, S. de Gironcoli, A. Dal Corso, and P. Giannozzi, Rev. Mod. Phys. 73, 515 (2001).
  * van Setten  _et al._ [2018] M. van Setten, M. Giantomassi, E. Bousquet, M. Verstraete, D. Hamann, X. Gonze, and G.-M. Rignanese, Comput. Phys. Commun. 226, 39 (2018).
  * Li  _et al._ [2012a] W. Li, N. Mingo, L. Lindsay, D. A. Broido, D. A. Stewart, and N. A. Katcho, Phys. Rev. B 85, 195436 (2012a).
  * Li  _et al._ [2012b] W. Li, L. Lindsay, D. A. Broido, D. A. Stewart, and N. Mingo, Phys. Rev. B 86, 174307 (2012b).
  * Kundu [2024] A. Kundu, Zenodo (2024).


# Transformer-based Selective Super-Resolution for Efficient Image Refinement

Tianyi Zhang1, Kishore Kasichainula2, Yaoxin Zhuo2, Baoxin Li2, Jae-Sun Seo3,
Yu Cao1

###### Abstract

Conventional super-resolution methods suffer from two drawbacks: substantial
computational cost in upscaling an entire large image, and the introduction of
extraneous or potentially detrimental information for downstream computer
vision tasks during the refinement of the background. To solve these issues,
we propose a novel transformer-based algorithm, Selective Super-Resolution
(SSR), which partitions images into non-overlapping tiles, selects tiles of
interest at various scales with a pyramid architecture, and exclusively
reconstructs these selected tiles with deep features. Experimental results on
three datasets demonstrate the efficiency and robust performance of our
approach for super-resolution. Compared to the state-of-the-art methods, the
FID score is reduced from 26.78 to 10.41 with 40% reduction in computation
cost for the BDD100K dataset. The source code is available at
https://github.com/destiny301/SSR.

## Introduction

Super-resolution (SR) is a fundamental task aimed at enhancing image
resolution by producing intricate details from low-resolution (LR) images. It
supplies high-resolution (HR) images that are pivotal for downstream computer
vision tasks, such as object detection and image classification, with wide-
ranging applications in the real world. For instance, in the context of
autonomous driving, higher-resolution images facilitate more precise and early
object detection, particularly for diminutive objects. Although various super-
resolution methods based on convolutional neural networks (CNNs) have been
proposed, which enhance high-frequency information through low-resolution
image reconstruction, their efficacy is impeded by a lack of long-range
dependency integration.

Figure 1: The reconstruction of high-frequency background information in
conventional SR methods (e.g., HAT) often results in discrepancies in features
compared to ground truth high-resolution (HR) images. SSR effectively resolves
this issue by exclusively enhancing foreground pixels.

Recently, leveraging transformer-based architectures to capture the extended
contextual information, pioneering efforts like SwinIR (Liang et al. 2021) and
HAT (Chen et al. 2023), have achieved notable advancements in super-
resolution. Nevertheless, two key issues persist with these algorithms.
Firstly, due to the substantial scale of transformer-based networks, the
computational demand becomes exceedingly high when reconstructing entire
images, particularly when input low-resolution image sizes are not small, such
as 256$\times$256\. Secondly, even in state-of-the-art super-resolution
approaches, the refined images fail to match the performance of ground truth
HR images for typical downstream tasks. To delve into the cause of this
degradation, we conduct a comparison between features generated by the
Inception model from refined images and original HR images. This analysis
unveils that features derived from background pixels exhibit markedly
different details compared to the ground truth feature map, as depicted in
Figure 1. This divergence suggests that overemphasizing background pixel
details can introduce erroneous information and impede feature generation for
downstream tasks.

This paper introduces a novel algorithm, Selective Super-Resolution (SSR),
designed to address these challenges. Specifically, leveraging object location
information, we partition images into non-overlapping tiles and employ a cost-
efficient transformer-based network for tile selection. To ensure
comprehensive coverage of objects, a pyramid structure is devised for tile
selection across multiple scales. In the final layer of this selection
network, we integrate a Gumbel-Softmax layer (Jang, Gu, and Poole 2016) to
make hard decisions, subsequently directing positive tiles to an ensuing
super-resolution (SR) module. This module comprises a convolution layer for
shallow feature extraction, transformer blocks for deep feature extraction and
an image reconstruction block. In contrast, negative tiles are reconstructed
directly from shallow features. This framework not only reduces computation by
extracting deep features solely for positive tiles but also enhances image
generation by avoiding excessive background detail addition. To validate the
robustness of SSR, alongside common evaluation metrics, such as Structural
Similarity Index Measure (SSIM), Fréchet Inception Distance (FID), and Kernel
Inception Distance (KID), we introduce a novel metric, inspired by KID and
tailored to evaluate features from an object detection (OD) model, YOLO,
called Kernel YOLO Distance (KYD). Our approach is experimented on three
distinct image datasets, BDD100K, MSRA10K, and COCO2017, demonstrating both
substantial computational reduction and image generation improvement.

To summarize, our key contributions are as follows:

  * •

We design a low-cost Tile Selection module, employing transformer blocks, to
effectively extract desired object-containing tiles from images. The
introduction of a pyramid structure ensures accurate selection of positive
tiles.

  * •

By seamlessly integrating two transformer-based modules, Tile Selection (TS)
and Tile Refinement (TR), our SSR efficiently reconstructs the input images by
exclusively adding high-frequency information for object-containing tiles,
effectively mitigating computational costs and enhancing visual quality.

  * •

Through comprehensive experiments on three diverse image datasets and various
evaluation metrics, we showcase SSR’s robust performance and specifically
lower FID from 26.78 to 10.41 for BDD100K, accompanied by a 40% reduction in
computation cost.

## Related Work

### Super-resolution

In this context, our primary focus revolves around single image super-
resolution (SISR) techniques, without delving into methodologies that
reconstruct high-resolution images using multiple input images. Previously,
the SRCNN model (Dong et al. 2014) based on convolutional neural networks
(CNNs) inspires many works in SR (Tai, Yang, and Liu 2017; Niu et al. 2020;
Mei, Fan, and Zhou 2021). This seminal approach employed bicubic interpolation
and trained a three-layer CNN for SR, achieving remarkable success. A diverse
range of CNN-based methodologies has emerged to map low-resolution images to
their high-resolution counterparts, leveraging distinct block designs, such as
the incorporation of residual blocks (Lim et al. 2017). Moreover, harnessing
the unprecedented image generation capabilities of generative adversarial
networks (GANs) (Goodfellow et al. 2020). Certain studies have notably
advanced the quality of generated images, such as SRGAN (Ledig et al. 2017;
Wang et al. 2021, 2018; Zhang et al. 2019), which introduce adversarial
learning to improve perceptual quality. Recent strides have been witnessed in
the adoption of the attention mechanism, which excels in capturing long-range
dependencies. This mechanism has lately been adopted to further enhance SISR
methodologies(Liang et al. 2021; Chen et al. 2023).

### Transformer in Computer Vision

Given the remarkable success of transformers in natural language processing
(NLP) (Vaswani et al. 2017), this architectural paradigm is progressively
permeating diverse computer vision tasks (Chu et al. 2021; Huang et al. 2021;
Dong et al. 2022; He et al. 2022; Zhang et al. 2023a, b). For instance, Vision
Transformer (ViT) divides input images into $16\times 16$ patches, which are
subsequently treated as tokens for the application of the attention mechanism
(Dosovitskiy et al. 2020). For object detection (OD), DETR conceptualizes it
as a direct set prediction issue and crafts a transformer-based network
(Carion et al. 2020). DINO introduces self-supervised learning to propose a
novel network rooted in the ViT architecture (Caron et al. 2021). Swin
Transformer integrates window-based self-attention and shifted window-based
self-attention mechanisms, to reduce the computation cost by limiting the
computation inside small windows (Liu et al. 2021, 2022). By capturing long-
range dependencies and facilitating enhanced visual representation,
transformer-based networks have exhibited superiority in various domains,
including super-resolution (Liang et al. 2021; Chen et al. 2023).

## Methodology

The overall architecture of our approach is illustrated in Figure 2. Selective
Super-Resolution (SSR) network comprises two fundamental modules: Tile
Selection (TS) and Tile Refinement (TR). Both modules are transformer-based
networks, with TS being notably smaller in scale. Upon receiving an input
image, the TS module initiates the process by partitioning it into non-
overlapping tiles and then selects pertinent tiles based on object location
cues. The tiles containing objects are directed through a computationally
intensive block for intricate refinement, while the remaining tiles traverse a
cost-efficient block for straightforward upscaling. This section is dedicated
to a comprehensive discourse on all modules, elucidating their specifics. We
outline the precise algorithm in Algorithm 1.

Figure 2: Overall architecture of Selective Super-Resolution (SSR).Input low-
resolution (LR) images are split into non-overlapping tiles and classified to
two classes by Tile Selection (TS) module. And only positive tiles containing
objects are intricately reconstructed by Positive Tile Refinement (PTR) path
with deep features. The specific Residual Transformer Block (RTB) in the
feature extraction module will be introduced in detail later.

### Tile Selection Module

By splitting the input images into $4\times 4$ non-overlapping tiles and
utilizing the location information of objects in these images, this module
classifies each tile by whether it contains objects or not. Specifically, as
shown in Figure 2, this module consists of an image encoder and several tile
classifiers. First, the embedding layer and the first transformer layer of the
encoder embeds the low-resolution (LR) input image
$\boldsymbol{I}_{LR}\in\mathbb{R}^{H_{LR}\times W_{LR}\times C_{LR}}$
($H_{LR}$, $W_{LR}$, and $C_{LR}$ are the input LR image height, width and the
number of channels) into representations
$r_{0}\in\mathbb{R}^{\frac{H_{LR}}{p}\times\frac{W_{LR}}{p}\times C}$, where
$p$ is the patch size of each token and $C$ is the number of channels for each
token. After that, three transformer layers, $TL_{1}$, $TL_{2}$, and $TL_{3}$,
generate representations of these tokens at three different scales. The
transformer layer adopts the structure of Swin Transformer (Liu et al. 2021,
2022). Basically, with the incorporation of window-based self-attention and
shifted window-based self-attention mechanisms, this layer can capture global
contextual information with attention computation within small windows, which
is much cheaper than the traditional attention mechanism. Besides, with the
feature merging layer, we can obtain features at various scales to enable the
implementation of pyramid structure. The whole process is as follows,

$\boldsymbol{r}_{1}=TL_{1}(\boldsymbol{r}_{0}),\;\;\;\boldsymbol{r}_{2}=TL_{2}(\boldsymbol{r}_{1}),\;\;\;\boldsymbol{r}_{3}=TL_{3}(\boldsymbol{r}_{2})$
(1)

where
$\boldsymbol{r}_{1}\in\mathbb{R}^{\frac{H_{LR}}{2p}\times\frac{W_{LR}}{2p}\times
2C}$,
$\boldsymbol{r}_{2}\in\mathbb{R}^{\frac{H_{LR}}{4p}\times\frac{W_{LR}}{4p}\times
4C}$,
$\boldsymbol{r}_{3}\in\mathbb{R}^{\frac{H_{LR}}{8p}\times\frac{W_{LR}}{8p}\times
8C}$ are three generated representations for classification. And, each token
of these representations corresponds to tiles of size $2p\times 2p$, $4p\times
4p$, and $8p\times 8p$, respectively.

To classify these tiles, we adopt the cross-attention mechanism by introducing
a learnable classification token, denoted as $c$. We obtain the query matrix
$Q$ by applying one linear layer to image features $r_{1}$, $r_{2}$ or $r_{3}$
as $Q_{i}=r_{i}W_{i}^{q},i\in 1,2,3$ while computing the key matrix and value
matrix with the classification token as equations
$K_{i}=cW_{i}^{k},V_{i}=cW_{i}^{v},i\in 1,2,3$, then the attention computation
is expressed as follows,

$\boldsymbol{A}_{i}=softmax(\frac{\boldsymbol{Q}_{i}\boldsymbol{K}_{i}^{T}}{\sqrt{d}})\boldsymbol{V}_{i}\;\;\;\forall
i\in{1,2,3}$ (2)

Next, each of the three features undergoes individual processing through a
multi-layer perceptron (MLP) and a Gumbel-Softmax layer. This step facilitates
making definitive classifications for the tile classes. This is crucial for
the subsequent refinement module to apply the corresponding network. The
process are as follows:

$\boldsymbol{s}_{i}=GumbelSoftmax(MLP_{i}(\boldsymbol{A}_{i}))\;\;\;\forall
i\in{1,2,3}$ (3)

where $MLP_{i}$ denotes the output layer for the $i$th feature embeddings.

Accordingly to the network structure, after pooling the instance segmentation
label of the input images at three different scales, we introduce a pyramid
label that contains,
$\boldsymbol{y}_{1}\in\mathbb{R}^{\frac{H_{LR}}{2p}\times\frac{W_{LR}}{2p}\times
1}$,
$\boldsymbol{y}_{2}\in\mathbb{R}^{\frac{H_{LR}}{4p}\times\frac{W_{LR}}{4p}\times
1}$, and
$\boldsymbol{y}_{3}\in\mathbb{R}^{\frac{H_{LR}}{8p}\times\frac{W_{LR}}{8p}\times
1}$, to supervise the training of TS module by allocating positive labels to
the tiles that contain objects. This hierarchical structure ensures the
preservation of a larger number of tiles and minimizes the loss of positive
instances. Finally, all tiles are divided into two groups to be processed by
the subsequent refinement module.

### Tile Refinement Module

As depicted in Figure 2, there are two Tile Refinement (TR) paths: Positive
Tile Refinement (PTR) targeting object-containing tiles, and Negative Tile
Refinement (NTR) for tiles with solely background pixels.

Algorithm 1 Selective Super-Resolution (SSR)

Input: Low-resolution (LR) Image data $\boldsymbol{I}_{LR}$, tile class labels
$\boldsymbol{y}_{1}$, $\boldsymbol{y}_{2}$, $\boldsymbol{y}_{3}$, and ground
truth high-resolution (HR) image $\boldsymbol{I}_{GT}$
Parameter: loss weight $\alpha$, learning rate $\eta$

1:Initialize model parameters $\theta$

2:for each epoch t = 1, 2, … do

3:
$\boldsymbol{r}_{1},\boldsymbol{r}_{2},\boldsymbol{r}_{3}=f_{\theta_{IE}}(\boldsymbol{I}_{LR})$
$\triangleright$ Generate representations with Image Encoder (IE)

4:
$\boldsymbol{s}_{i}=GumbelSoftmax(f_{\theta_{CL}}(\boldsymbol{r}_{i}))\;\;\;\forall
i\in{1,2,3}$ $\triangleright$ Classify tiles with Classification Layer (CL)

5:
$\mathcal{L}_{TS}(\theta)=\sum_{i=1}^{3}(-\boldsymbol{y}_{i}log(\boldsymbol{s}_{i})-(1-\boldsymbol{y}_{i})log(1-\boldsymbol{s}_{i}))$

6: for each tile $T_{LR}^{1},T_{LR}^{2},...,T_{LR}^{N}$ do

7: if $s_{3}^{n}=1$ then $\triangleright$ Positive tiles

8: $F_{S}^{n}=f_{\theta_{Conv}}(T_{LR}^{n})$ $\triangleright$ Shallow feature
extraction with convolution layer

9: $F_{D}^{n}=f_{\theta_{FE}}(F_{S}^{n}))$ $\triangleright$ Deep Feature
Extraction (FE)

10: $T_{HR}^{n}=f_{\theta_{IR}}(F_{S}^{n}+F_{D}^{n})$ $\triangleright$ HR
Image Reconstruction (IR)

11: else$\triangleright$ Negative tiles

12: $F_{S}^{n}=f_{\theta_{Conv}}(T_{LR}^{n})$

13: $T_{HR}^{n}=f_{\theta_{IR}}(F_{S}^{n})$

14: end if

15: end for

16: Group all output tiles $T_{HR}$ to entire images $I_{HR}$

17: $\mathcal{L}_{TR}(\theta)=||I_{HR}-I_{GT}||_{1}$

18:
$\mathcal{L}_{SSR}(\theta)=\mathcal{L}_{TS}(\theta)+\alpha\mathcal{L}_{TR}(\theta)$

19: $\theta\leftarrow\theta-\eta\nabla_{\theta}\mathcal{L}_{SSR}(\theta)$
$\triangleright$ Update model

20:end for

Positive Tile Refinement. For positive tiles which contain objects, our
refinement involves a transformer-based process for deep feature extraction
and image reconstruction. To be specific, for a given tile
$T_{LR}\in\mathbb{R}^{8p\times 8p\times C_{LR}}$, the convolution layer first
extract the shallow feature $F_{S}\in\mathbb{R}^{8p\times 8p\times C_{f}}$,
where $C_{f}$ is the number of channels for features. Subsequently, a series
of $K$ residual transformer blocks (RTBs), based on the Swin transformer
architecture, are employed to derive deep features
$F_{D}\in\mathbb{R}^{8p\times 8p\times C_{f}}$ as the following equations:

$F_{i}=RTB_{i}(F_{i-1}),i=1,2,...,K,$ (4) $F_{D}=Conv(F_{K})$ (5)

where $F_{0}$ is the input shallow feature $F_{S}$, $RTB_{i}$ denotes the
$i$-th RTB, and $Conv$ is the final convolution layer.

The keypoint is the design of RTB. Figure 3 presents the specific structure of
RTB. Basically, it consists of a series of transformer layers and one
convolution layer. Primarily, the inclusion of an additional convolutional
layer at the end serves to optimize the transformer more effectively, yielding
enhanced representations. This is due to the fact that direct similarity
comparisons across all tokens often introduce redundancy, as evidenced in
various works(Zhang et al. 2018; Liang et al. 2021; Li et al. 2023; Wu et al.
2021; Xiao et al. 2021). Secondly, the skip connection within the RTB
establishes a link between features at different levels and the image
reconstruction block. This facilitates the aggregation of features from
diverse levels, promoting the integration of multi-level information.
Additionally, we adopt distinct attention blocks proposed in (Chen et al.
2023) to activate more pixels for high-resolution image reconstruction.

After obtaining both the shallow feature $F_{S}$ and deep feature $F_{D}$, we
merge them to reconstruct HR tiles using the following equation,

$T_{HR}=IR(F_{S}+F_{D})$ (6)

where $IR$ is the image reconstruction block. By transmitting the shallow
feature containing low-frequency information and the deep feature which
highlights high-frequency details via a long skip connection, this module
effectively concentrates on reconstructing high-frequency information. And in
this block, the sub-pixel convolution layer (Shi et al. 2016) is employed to
upsample the feature.

Negative Tile Refinement. Since the intricate refinement for the background
introduce some irrelevant or even detrimental features which degrade the image
quality for downstream tasks, we remove the transformer-based feature
extraction block and directly reconstruct the negative tiles with shallow
features to obtain HR tiles with the same resolution as refined positive
tiles.

After obtaining HR tiles via both PTR and NTR, we consolidate them to produce
the refined HR image.

Figure 3: Structure of RTB. The incorporation of a skip connection and
convolution layer in RTB contributes to improved feature learning
capabilities.

### Loss function

For the TS module, the loss is computed using cross-entropy. With the
introduction of a pyramid structure aimed at reducing false negative
predictions (FN) and ensuring the selection of all positive patches, the
losses from the three scales are combined to formulate the final loss as
follows:

$\mathcal{L}_{TS}=\sum_{i=1}^{3}(-\boldsymbol{y}_{i}log(\boldsymbol{s}_{i})-(1-\boldsymbol{y}_{i})log(1-\boldsymbol{s}_{i}))$
(7)

For the TR module, we leverage $L_{1}$ loss to quantify the discrepancy
between the generated high-resolution image $I_{HR}$ and the ground truth
high-resolution image $I_{GT}$ as follows:

$\mathcal{L}_{TR}=||I_{HR}-I_{GT}||_{1}$ (8)

Finally, the loss function of SSR is the weighted sum of these two modules and
the formulation is expressed as,

$\mathcal{L}_{SSR}=\mathcal{L}_{TS}+\alpha\mathcal{L}_{TR}$ (9)

where $\alpha$ is a hyper-parameter to adjust the weight of TS module in
training.

## Experimental Results

### Datasets and Experimental Settings

We evaluate the effectiveness of our SSR model on three datasets, BDD100K (Yu
et al. 2020), COCO 2017 (Lin et al. 2014) and MSRA10K (Cheng et al. 2015).
These datasets provide segmentation or object detection labels, enabling us to
generate tile classification labels for tile selection. BDD100K is the most
complex dataset focused on autonomous driving, containing approximately
100,000 high-quality images with an original resolution of $1280\times 720$.
Each image encompasses various objects, including vehicles, pedestrians, and
eight other object categories. COCO2017 comprises around 100,000 training
images and 5,000 validation images. Each image encompasses multiple objects
spanning 80 different classes. MSRA10K provides 10000 images and their
segmentation labels.

For all datasets, we resize them into low-resolution (LR) images with
resolution $256\times 256$, and high-resolution (HR) images with resolution
$1024\times 1024$ by bicubic interpolation. The embedding dimension of the
Tile Selection (TS) module is 96 while it’s 180 for Tile Refinement (TR)
module. We set the learning rates to 0.00001. Each TL utilizes a depth of 2, a
window size of 7, and 3 attention heads. We employ a patch size of 2 for
embedding, which corresponds to tile sizes of $16\times 16$, $32\times 32$,
and $64\times 64$, yielding tile labels of $4\times 4$, $8\times 8$, and
$16\times 16$ respectively. The weight parameter $\alpha$ for the loss
function is set to 1. The number of RTB is 6. All experiments are conducted
for 50 epochs on two Linux servers, each equipped with two NVIDIA A6000 GPUs.

#### Evaluation Metrics.

Besides the common metrics for image quality, like Peak Signal-to-Noise Ratio
(PSNR), Structural Similarity Index Measure (SSIM), Fréchet Inception Distance
(FID), and Kernel Inception Distance (KID), we introduce an additional metric
named Kernel YOLO Distance (KYD) to demonstrate the robustness of our
approach. To evaluate FID, we employ a pre-trained image classification model,
Inceptionv3, to generate features and compare the distributions of two image
sets. For KYD, we focus on the comparison of features for object detection
(OD) by utilizing a pre-trained YOLOv8 model to generate features and compute
the kernel distance similarly to KID.

Furthermore, we assess the performance of the TS module using True Positive
Rate (TPR) and the maximum F1 score (maxF) for accuracy, along with the
average number of selected tiles for computation efficiency.

### Tile Selection Results

For Tile Selection (TS), we explore various configurations to identify the
optimal setup for our objectives. We adjusted the patch size of the first
embedding layer, which consequently influenced the number of transformer
layers (TL) in the network. Specifically, by setting the patch size to 2, 4,
or 8, we achieved 5, 4, or 3 TLs respectively. We also experimented with
different loss functions, including pyramid labels at various scales and a
single label. Additionally, we introduced a max block at the end of the TS
module to consolidate three predictions by selecting the maximum. The results,
presented in Table 1, highlighted several key insights. Firstly, increasing
the number of transformer layers enhanced performance, albeit at the cost of
increased computation. Secondly, the pyramid structure consistently yielded
better outcomes. Lastly, while the max block improved selection results, it
also introduced additional computational overhead for the refinement module.

Model | TPR | maxF | #Tiles | #MACs
---|---|---|---|---
B (w/o pyramid) | 0.9137 | 0.8807 | 62% | 563.21M
S (w/o pyramid) | 0.9086 | 0.8522 | 65% | 155.30M
T (w/o pyramid) | 0.8973 | 0.8225 | 66% | 53.11M
B (w/ pyramid) | 0.9341 | 0.8967 | 62% | 563.21M
S (w/ pyramid) | 0.9048 | 0.8327 | 66% | 155.30M
T (w/ pyramid) | 0.8855 | 0.8384 | 62% | 53.11M
B (w/ max) | 0.9692 | 0.8893 | 77% | 563.21M
S (w/ max) | 0.9681 | 0.8572 | 79% | 155.30M
T (w/ max) | 0.9666 | 0.8400 | 81% | 53.11M
Table 1: Results of TS module. Encode images with 5, 4, or 3 transformer layers, corresponding to TS-Base (B), TS-Small (S) and TS-Tiny (T) in this table. More layers provide better performance with increasing computation. The adoption of the pyramid structure for tile selection across different scales improves the network’s effectiveness. The integration of the max block further enhances the selection. Method | BDD100K | COCO 2017 | MSRA10K
---|---|---|---
PSNR$\uparrow$ | SSIM$\uparrow$ | FID$\downarrow$ | KID$\downarrow$ | KYD$\downarrow$ | SSIM$\uparrow$ | FID$\downarrow$ | KYD$\downarrow$ | SSIM$\uparrow$ | FID$\downarrow$ | KYD$\downarrow$
SwinIR | 30.08 | 0.8757 | 24.87 | 0.0104 | 0.0092 | 0.8420 | 8.483 | 0.0036 | 0.9441 | 9.502 | 0.0035
HAT | 30.09 | 0.8750 | 26.78 | 0.0115 | 0.0096 | 0.8391 | 10.03 | 0.0037 | 0.9473 | 9.577 | 0.0030
SSR (ours) | 29.91 | 0.8833 | 10.41 | 0.0017 | 0.0041 | 0.8512 | 2.844 | 0.0006 | 0.9653 | 1.905 | 0.0002
Table 2: Comparison of SR results. The table presents the outcomes for a
4$\times$ upscaling factor. For SwinIR and HAT, we utilize the official tile-
based implementation of HAT for equitable assessment. Both SwinIR and HAT
demonstrate results inferior to our SSR in terms of image quality across all
three datasets. Generated images by SSR can perform better for downstream
tasks with lower FID and KYD.

### Super-Resolution (SR) results

As shown in figure 1, by comparing the feature visualization results of
reconstructed images by SR methods with ground truth HR images, we found the
high-frequency refinement of conventional methods for background adversely
impacts the image quality. To substantiate this observation, we devised two
new datasets: one by replacing foreground pixels and the other by substituting
background pixels with upscaled images via bicubic interpolation. This
simulation aimed to explore the impact of high-frequency removal. Upon
evaluating the image quality of these datasets, we discerned an intriguing
finding. Even the mere replacement of pixels lacking high-frequency
information for the background led to enhanced image quality. Figure 4
graphically illustrates this comparison. The top example juxtaposes generated
foreground via SR with coarse background upscaled by bicubic interpolation,
while the bottom image maintains high-frequency details exclusively in the
background. The top dataset exhibited superior image quality, as indicated by
lower FID and higher SSIM. This quantitative assessment provides further
support for our SSR design.

Figure 4: By replacing either the background or foreground pixels with
bicubic-interpolated upscaled images, we effectively simulate the removal of
high-frequency details from these regions. The results illustrate that the
high-frequency information of background introduced by SR module impairs the
generated image quality. Figure 5: Visual comparison with the state-of-the-art
methods for $4\times$ upscaling. We generate features with Inceptionv3 model
and visualize them. The features extracted from SSR images exhibit a higher
degree of similarity to the GT features.

#### Quantitative results.

Table 2 shows the quantitative comparison of our approach with the state-of-
the-art methods (Liang et al. 2021; Chen et al. 2023) on three different
datasets, considering a 4$\times$ upscale of the input images. The results
clearly demonstrate the consistent superiority of our proposed method, SSR, in
terms of various evaluation metrics. This robust performance underlines SSR’s
potential to generate images with improved features for downstream tasks such
as image classification and object detection, as indicated by its lower FID
and KYD.

In Table 3, we extend the comparison to 2$\times$ and 3$\times$ magnification
factors on BDD100K. Given SwinIR’s challenges on this dataset, we focus on
contrasting HAT with our method. The results showcased in this table
demonstrate SSR’s proficiency across different scales.

Method | Scale | SSIM | FID | FID | KYD
---|---|---|---|---|---
HAT | $\times 2$ | 0.8764 | 27.58 | 0.0123 | 0.0092
SSR (ours) | $\times 2$ | 0.8890 | 10.28 | 0.0016 | 0.0037
HAT | $\times 3$ | 0.8768 | 30.88 | 0.0150 | 0.0096
SSR (ours) | $\times 3$ | 0.8918 | 10.80 | 0.0017 | 0.0027
Table 3: Quantitative comparison with HAT for 2$\times$ and 3$\times$ up-
sampling. SSR consistently outperforms HAT across all evaluation metrics,
demonstrating its superiority in refining images at various scales.

#### Visual comparison.

Figure 5 illustrates the visual comparison across three datasets. Besides the
ground truth (GT) images and the images generated by HAT and our SSR approach,
we also present the extracted features from these images. The features derived
from our SSR-generated images exhibit a closer resemblance to GT images,
emphasizing the fidelity of our approach.

#### Efficiency.

Actually, with more tiles selected by the max block, the generated image
quality remains quite similar, as shown in Table 4. So, we opt not to include
this block in our final algorithm. For our SSR, only positive tiles are
directed through the expensive SR module. In Table 5, we summarize the total
number of parameters and computation cost for three different datasets. It
underscores the efficiency of SSR with about 40% computation reduction.

Method | #Tiles | SSIM | FID | KID | KYD
---|---|---|---|---|---
w/ max | 62% | 0.8833 | 10.41 | 0.0017 | 0.0041
w/o max | 77% | 0.8835 | 10.62 | 0.0017 | 0.0041
Table 4: With the max block, the visual quality of images generated by SSR can be slightly improved with additional 15% computation overhead. Dataset | Method | #Tiles |  #Params (M) |  #MACs (G)
---|---|---|---|---
BDD100K | HAT | - | 1675.99 | 191.06
SSR | 62% | 1233.13 | 119.40
COCO 2017 | HAT | - | 1675.99 | 191.06
SSR | 65% | 1233.13 | 125.15
MSRA10K | HAT | - | 1675.99 | 191.06
SSR | 57% | 1233.13 | 109.82
Table 5: Computation cost comparison. Compared to HAT, SSR achieves about 40%
computation reduction for all three datasets. All metrics are evaluated for a
$256\times 256$ input.

### Ablation Study

#### Pre-training with ImageNet.

Similar to other computer vision work, SSR can substantially benefit from a
pre-training strategy employing a large dataset, such as ImageNet. A
comparison of results obtained with and without pre-training highlights the
enhancement in image quality, with PSNR scores improving from 29.56 to 29.91.
Figure 6 presents one example for visual comparison.

Figure 6: With pre-training, SSR can generate better images compared to the
outcomes without pre-training.

#### Negative Tile Refinement.

For background pixels, high-frequency information recovery is unnecessary. We
experimented with two different designs: the first employs a single Upsample
layer, while the second retains the convolution layers from the Positive Tile
Refinement (PTR) path and omits the transformer blocks. The results are shown
in Table 6. Adding convolution layers in NTR marginally enhances image
quality. This enhancement can be attributed to the compensation for a few
positive tiles that might be missed by the selection module. However, this
improvement comes at the cost of increased 19.2% computational load.
Therefore, it is a trade-off between the computation and image generation
performance.

Method | PSNR | SSIM | FID | KID | KYD
---|---|---|---|---|---
w/o conv | 29.91 | 0.8833 | 10.41 | 0.0017 | 0.0041
w/ conv | 30.03 | 0.8863 | 9.56 | 0.0013 | 0.0042
Table 6: With convolution layers in the NTR path, SSR showcases an improvement
in image generation.

## Conclusion

In this paper, we delve into the cause behind the image quality gap between
generated images through conventional super-resolution (SR) techniques and
high-resolution (HR) ground truth images. Our investigation reveals that the
high-frequency refinement of background pixels undermines the overall image
feature generation for downstream tasks. To address this challenge and
concurrently reduce computational overhead, we introduce a novel algorithm
termed Selective Super-Resolution (SSR). By leveraging a cost-efficient
transformer-based network, we partition the input low-resolution (LR) image
into non-overlapping tiles, assessing the presence of objects within each
tile. By exclusively refining the high-frequency characteristics of object-
containing tiles, we improve visual quality with a lower computation cost. Our
approach’s superior performance is substantiated across three distinct
datasets employing diverse evaluation metrics. Specifically, experiments on
BDD100K exhibit an improvement of FID from 26.78 to 10.41 with 40% computation
reduction.

## Acknowledgments

This work is supported by the Center for the Co-Design of Cognitive Systems
(CoCoSys), one of seven centers in Joint University Microelectronics Program
2.0 (JUMP 2.0), a Semiconductor Research Corporation (SRC) program sponsored
by the Defense Advanced Research Projects Agency (DARPA).

## References

  * Carion et al. (2020) Carion, N.; Massa, F.; Synnaeve, G.; Usunier, N.; Kirillov, A.; and Zagoruyko, S. 2020.  End-to-end object detection with transformers.  In _European conference on computer vision_ , 213–229. Springer International Publishing Cham.
  * Caron et al. (2021) Caron, M.; Touvron, H.; Misra, I.; Jégou, H.; Mairal, J.; Bojanowski, P.; and Joulin, A. 2021.  Emerging properties in self-supervised vision transformers.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , 9650–9660.
  * Chen et al. (2023) Chen, X.; Wang, X.; Zhou, J.; Qiao, Y.; and Dong, C. 2023.  Activating more pixels in image super-resolution transformer.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 22367–22377.
  * Cheng et al. (2015) Cheng, M.-M.; Mitra, N. J.; Huang, X.; Torr, P. H. S.; and Hu, S.-M. 2015.  Global Contrast based Salient Region Detection.  _IEEE TPAMI_ , 37(3): 569–582.
  * Chu et al. (2021) Chu, X.; Tian, Z.; Wang, Y.; Zhang, B.; Ren, H.; Wei, X.; Xia, H.; and Shen, C. 2021.  Twins: Revisiting the design of spatial attention in vision transformers.  _Advances in Neural Information Processing Systems_ , 34: 9355–9366.
  * Dong et al. (2014) Dong, C.; Loy, C. C.; He, K.; and Tang, X. 2014.  Learning a deep convolutional network for image super-resolution.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part IV 13_ , 184–199. Springer.
  * Dong et al. (2022) Dong, X.; Bao, J.; Chen, D.; Zhang, W.; Yu, N.; Yuan, L.; Chen, D.; and Guo, B. 2022.  Cswin transformer: A general vision transformer backbone with cross-shaped windows.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 12124–12134.
  * Dosovitskiy et al. (2020) Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. 2020.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_.
  * Goodfellow et al. (2020) Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; and Bengio, Y. 2020.  Generative adversarial networks.  _Communications of the ACM_ , 63(11): 139–144.
  * He et al. (2022) He, K.; Chen, X.; Xie, S.; Li, Y.; Dollár, P.; and Girshick, R. 2022.  Masked autoencoders are scalable vision learners.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 16000–16009.
  * Huang et al. (2021) Huang, Z.; Ben, Y.; Luo, G.; Cheng, P.; Yu, G.; and Fu, B. 2021.  Shuffle transformer: Rethinking spatial shuffle for vision transformer.  _arXiv preprint arXiv:2106.03650_.
  * Jang, Gu, and Poole (2016) Jang, E.; Gu, S.; and Poole, B. 2016.  Categorical reparameterization with gumbel-softmax.  _arXiv preprint arXiv:1611.01144_.
  * Ledig et al. (2017) Ledig, C.; Theis, L.; Huszár, F.; Caballero, J.; Cunningham, A.; Acosta, A.; Aitken, A.; Tejani, A.; Totz, J.; Wang, Z.; et al. 2017.  Photo-realistic single image super-resolution using a generative adversarial network.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 4681–4690.
  * Li et al. (2023) Li, K.; Wang, Y.; Zhang, J.; Gao, P.; Song, G.; Liu, Y.; Li, H.; and Qiao, Y. 2023.  Uniformer: Unifying convolution and self-attention for visual recognition.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_.
  * Liang et al. (2021) Liang, J.; Cao, J.; Sun, G.; Zhang, K.; Van Gool, L.; and Timofte, R. 2021.  Swinir: Image restoration using swin transformer.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , 1833–1844.
  * Lim et al. (2017) Lim, B.; Son, S.; Kim, H.; Nah, S.; and Mu Lee, K. 2017.  Enhanced deep residual networks for single image super-resolution.  In _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_ , 136–144.
  * Lin et al. (2014) Lin, T.-Y.; Maire, M.; Belongie, S.; Hays, J.; Perona, P.; Ramanan, D.; Dollár, P.; and Zitnick, C. L. 2014.  Microsoft coco: Common objects in context.  In _European conference on computer vision_ , 740–755. Springer.
  * Liu et al. (2022) Liu, Z.; Hu, H.; Lin, Y.; Yao, Z.; Xie, Z.; Wei, Y.; Ning, J.; Cao, Y.; Zhang, Z.; Dong, L.; et al. 2022.  Swin transformer v2: Scaling up capacity and resolution.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 12009–12019.
  * Liu et al. (2021) Liu, Z.; Lin, Y.; Cao, Y.; Hu, H.; Wei, Y.; Zhang, Z.; Lin, S.; and Guo, B. 2021.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , 10012–10022.
  * Mei, Fan, and Zhou (2021) Mei, Y.; Fan, Y.; and Zhou, Y. 2021.  Image super-resolution with non-local sparse attention.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 3517–3526.
  * Niu et al. (2020) Niu, B.; Wen, W.; Ren, W.; Zhang, X.; Yang, L.; Wang, S.; Zhang, K.; Cao, X.; and Shen, H. 2020.  Single image super-resolution via a holistic attention network.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XII 16_ , 191–207. Springer.
  * Shi et al. (2016) Shi, W.; Caballero, J.; Huszár, F.; Totz, J.; Aitken, A. P.; Bishop, R.; Rueckert, D.; and Wang, Z. 2016.  Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 1874–1883.
  * Tai, Yang, and Liu (2017) Tai, Y.; Yang, J.; and Liu, X. 2017.  Image super-resolution via deep recursive residual network.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 3147–3155.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Wang et al. (2021) Wang, X.; Xie, L.; Dong, C.; and Shan, Y. 2021.  Real-esrgan: Training real-world blind super-resolution with pure synthetic data.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , 1905–1914.
  * Wang et al. (2018) Wang, X.; Yu, K.; Wu, S.; Gu, J.; Liu, Y.; Dong, C.; Qiao, Y.; and Change Loy, C. 2018.  Esrgan: Enhanced super-resolution generative adversarial networks.  In _Proceedings of the European conference on computer vision (ECCV) workshops_ , 0–0.
  * Wu et al. (2021) Wu, H.; Xiao, B.; Codella, N.; Liu, M.; Dai, X.; Yuan, L.; and Zhang, L. 2021.  Cvt: Introducing convolutions to vision transformers.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , 22–31.
  * Xiao et al. (2021) Xiao, T.; Singh, M.; Mintun, E.; Darrell, T.; Dollár, P.; and Girshick, R. 2021.  Early convolutions help transformers see better.  _Advances in neural information processing systems_ , 34: 30392–30400.
  * Yu et al. (2020) Yu, F.; Chen, H.; Wang, X.; Xian, W.; Chen, Y.; Liu, F.; Madhavan, V.; and Darrell, T. 2020.  BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning.  In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Zhang et al. (2023a) Zhang, T.; Kasichainula, K.; Jee, D.-W.; Yeo, I.; Zhuo, Y.; Li, B.; Seo, J.-s.; and Cao, Y. 2023a.  Improving the Efficiency of CMOS Image Sensors through In-Sensor Selective Attention.  In _2023 IEEE International Symposium on Circuits and Systems (ISCAS)_ , 1–4. IEEE.
  * Zhang et al. (2023b) Zhang, T.; Kasichainula, K.; Zhuo, Y.; Li, B.; Seo, J.-S.; and Cao, Y. 2023b.  Patch-based Selection and Refinement for Early Object Detection.  _arXiv preprint arXiv:2311.02274_.
  * Zhang et al. (2019) Zhang, W.; Liu, Y.; Dong, C.; and Qiao, Y. 2019.  Ranksrgan: Generative adversarial networks with ranker for image super-resolution.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 3096–3105.
  * Zhang et al. (2018) Zhang, Y.; Li, K.; Li, K.; Wang, L.; Zhong, B.; and Fu, Y. 2018.  Image super-resolution using very deep residual channel attention networks.  In _Proceedings of the European conference on computer vision (ECCV)_ , 286–301.