# Controlling Quantum Cascade Laser Optical Frequency Combs through Microwave
Injection

###### Abstract

In this work, control over the precise state emitted by quantum cascade laser
frequency combs through strong radio-frequency current modulation close to
their repetition frequency is demonstrated. In particular, broadening of the
spectrum from about 20 cm-1 to 60cm-1 can be achieved throughout most of the
current dynamical range while preserving the coherence, as measured by shifted
wave interference Fourier transform spectroscopy (SWIFTS). The required
modulation frequency to achieve this broadening is red-shifted compared to the
free-running beatnote frequency at increasing modulation powers starting from
25 dBm, whereas the range where it occurs narrows. Outside of this maximum-
bandwidth range, the spectral bandwidth of the laser output is gradually
reduced and the new center frequency is red- or blue-shifted, directly
dependent on the detuning of the modulation frequency. By switching between
two modulation frequencies detuned symmetrically with respect to the free-
running beatnote, we can generate two multiplexed spectral regions with
negligible overlap from the same device at rates of at least 20 kHz. In the
time-domain we show with both SWIFTS and interferometric autocorrelation (IAC)
measurements a transition from quasi-continuous output to pulsed
($\tau_{p}\approx 55$ ps) output by ramping up the injection power to 35 dBm.

###### keywords:

quantum cascade lasers, frequency comb, injection locking, mid-infrared region

Barbara Schneider* Filippos Kapsalidis Mathieu Bertrand Matthew Singleton
Johannes Hillbrand Mattias Beck Jérôme Faist

B. Schneider, F. Kapsalidis, Dr. M. Bertrand, M. Singleton, Dr. J. Hillbrand,
Dr. M. Beck, Prof. J. Faist
Institute for Quantum Electronics
ETH-Zurich, CH-8093 Zurich, Switzerland
Email Address<EMAIL_ADDRESS>
##  1 Introduction

Mid-infrared (mid-IR) spectroscopy is a powerful tool in sensing applications
and environmental monitoring. Since it covers the molecular fingerprint
regions of light gas molecules and functional groups of larger organic
compounds, it is relevant in industrial and environmental sensing as well as
biomedicine.[1, 2, 3, 4] Quantum cascade lasers[5] (QCLs) are ideally suited
for mid-IR spectroscopy, as they are compact and electrically pumped sources,
which can operate at room-temperature while spanning the range of 3 to 20 µm
with Watt-level output powers.[6, 7] They possess the innate ability to form
optical frequency combs via four-wave mixing (FWM) caused by the strong
nonlinearity of their gain-medium and waveguides[8, 9] in Fabry-Pérot and ring
QCLs.[10, 11, 12, 13] Optical frequency combs are constituted of evenly spaced
phase-locked modes.[14] The resulting high mode intensities and resolution, as
compared to incoherent broadband sources, make them ideally suited for high-
resolution spectroscopy, especially dual-comb spectroscopy.[15, 16, 17, 18] A
peculiarity of QCL frequency combs is that due to their ultrashort carrier-
lifetimes on the order of picoseconds[19] their free-running output is not
pulsed, as in most visible and near-infrared frequency combs. Rather, it is
characterized by a linear frequency chirp spanning the entire round-trip
period.[20, 21] Therefore it is not possible to use the same characterization
techniques tailored for ultrashort pulses[22] for fully explaining the
underlying dynamics of QCL frequency combs.

For optimal spectroscopic results, tunability of the spectroscopic light
sources is of utmost importance. In general, QCLs allow for spectral tuning
via operational temperature and electrical DC bias. In QCL frequency combs,
the fast gain dynamics allow for a modulation of the carrier-populations
within the cavity of at least up to tens of GHz.[23] This property enables
electronic read-out of the optical intermode-beating from their bias line.
Likewise, external radio-frequency (RF) injection can be used as a means for
injection-locking and thus manipulation of these internal oscillations.[24,
25, 26] In earlier experiments, stabilization[27, 28] active mode-locking[29,
30, 31] and tuning of the free spectral range of the device[32] have been
demonstrated. The trade-off between RF-compatibility and thermal efficiency
has been recently addressed by Kapsalidis et al.. Microstrip-waveguide
geometry based designs with a double-metal waveguide, while providing superior
RF-modulation capabilities, can typically only be operated at cryogenic
temperatures.[33, 34, 32] This is due to the difficulty of combining a buried
heterostructure and wafer bonding process. By instead employing a highly doped
substrate, the optical losses could sufficiently be reduced to allow for
operation close to room-temperature.[35] We use this novel QCL design to
demonstrate the potential of RF-modulation as a means for fast and precise
tuning of the spectral and temporal properties of QCL frequency combs.
Furthermore, RF-injection offers us a novel tool for probing the lasing
dynamics of QCL frequency combs and thus allows us to gain further insight
into the complex comb formation dynamics underlying these devices.

##  2 Results and Discussion

The device used for all measurements in this paper is a 4 mm long mid-IR QCL
optical frequency comb centered at 8.2 µm.[35] For a systematic study of the
modulation-dependent behavior, the modulation frequency was swept across the
free-running beatnote frequency while keeping the modulation power constant.
These sweeps were repeated for different modulation powers. A detailed
description of the experimental setup can be found in the experimental section
below.

Figure 1: Experimental results for modulation-frequency sweeps while injecting
at constant power on the device operated with a constant DC-bias of 950 mA at
-25∘C. (a) Logarithmic spectral map of the device under constant RF-injection
power of 30 dBm and stepped modulation frequency between 11.085-11.095 GHz.
(b) SWIFTS spectra and the corresponding spectrum products (black dots) at
selected points of the spectral map in (a). (c) Ratio between the integrated
spectral power from the SWIFTS spectrum and the integrated spectral power of
the spectrum product indicating the average coherence for injected RF-power
from -5 to 35 dBm as a function of modulation frequency. (d) Modulation
frequency and bandwidth of broadest spectrum in modulation frequency spectral
maps as a function of injected power.

In Figure 1 a) the resulting spectral map is shown for such a modulation
frequency sweep at 30 dBm injection power. There is a clear broadening in the
spectral bandwidth close to the center of the sweep with ”sweet spot” marking
the largest bandwidth. Notably, this point occurs at $~{}11.89$ GHz, about 1
MHz offset from the free-running beatnote frequency located around $11.090$
GHz. The symmetry of the QCL-spectrum is highest at the ”sweet spot”, as is
also visible in Figure 1 b), in which the corresponding SWIFTS spectra,
$|X_{n}+iY_{n}|$, where $X_{n}$ and $Y_{n}$ are the mode amplitudes of mode
$n$ of the quadrature interferograms, are plotted. At injection frequencies
above this point the optical power is concentrated towards lower wavenumbers
inside the spectrum and vice versa. The closest comparable behavior has been
observed and modeled by Hillbrand et al.[31] for a bifunctional active region.
In their work, the spectral shape evolution has been attributed to a negative
Kerr non-linearity in the system. However, they start at single-mode operation
and they assume an antisymmetric shape around the maximally broadened
spectrum. In our measurements, there is a clear asymmetry around the ”sweet
spot”, which grows with increasing injection power, marked by a much faster
decay in spectral bandwidth towards lower than towards higher modulation
frequencies. This points to the emergence of additional factors involved in
the spectral response of the QCL. As a measure of coherence, the corresponding
spectrum products, $|A_{n}||A_{n+1}|$, where $|A_{n}|$ is the mode amplitude
of mode $n$ of the comb measured on the MCT, are displayed as black dots for
comparison with the SWIFTS spectra in Figure 1 b). Since the SWIFTS spectra
emerge from the intermode-beating of the comb, the agreement between the two
datasets directly indicates the degree of mutual coherence among adjacent
modes. Around the sweet spot frequency at 11.089 GHz the shapes of the SWIFTS
spectrum and the spectrum product are very close throughout the entire
spectral region, whereas further away there are clear gaps between the two
traces when going from the spectral maximum to the center frequency. As a
measure for the average coherence of the spectrum, the ratio between the
integrated spectral power of the SWIFTS spectrum and the spectrum product is
plotted in Figure 1 c), as a function of modulation frequency at different
injection levels. For -5 dBm injection power there is only a narrow injection
range which results in high coherence. Going to 5 and 15 dBm injection, a
locking range is still apparent in a step in the coherence on both ends of the
sweep, however, there seems to be partial locking even outside the locking-
range as indicated by the slightly increased coherence toward the edges of the
sweep. Surprisingly, at these intermediate injection powers, the stable
locking range stays nearly identical. Going to 25 and 35 dBm, the coherence at
the edges of the sweep steadily increases, but still shows a drop compared to
the coherence close to the free-running beatnote, which agrees with the
results in Figure 1 b). The extracted broadening and position of the ”sweet
spot” in dependence of the injection power are displayed in Figure 1 d) for 1
dBm steps in modulation power. Interestingly, the maximum spectral bandwidth
occurs around the free-running beatnote frequency at RF-powers up to 25 dBm,
accompanied by a steadily growing spectral bandwidth. Above 25 dBm the maximal
spectral bandwidth is clamped to around 60 cm-1, whereas the injection
frequency at which this broadening occurs starts decreasing up to an offset of
2 MHz. This indicates a fixed upper bandwidth limit of the device. The shift
in frequency could be caused by an increase in the effective temperature of
the active region at high-power modulation, by a Kerr-effect induced change in
the effective index due to the increasing modulation depth and thus
instantaneous optical power inside the cavity or a combination thereof.

As a further step, to study the effect of DC bias on the spectral response to
RF-injection, we conducted a series of frequency sweeps at a fixed injection
power of 35 dBm while stepping the DC-bias throughout the dynamical range of
the QCL. For the selected bias points 700, 950 and 1200 mA, Figure 2 a) shows
the free-running spectra and b) the corresponding maximum bandwidth SWIFTS
spectra. The free-running spectrum is single-mode at 700 mA, broadening to
about 30 cm-1 bandwidth in the range of 900 to 1200 mA. Conversely, the
injected QCL spectra all cover approximately the same spectral range with
bandwidths around 60 cm-1. The coherence is close to uniform across all three
characterized bias points. In Figure 2 c), the injection frequencies at the
respective ”sweet spots” are plotted as white dots against the free-running
beatnote map of the laser. Above 900 mA, the broadest spectra are generated 2
MHz below the free-running beatnote frequency for all bias points. The maximal
spectral bandwidths at 35 dBm injection are shown in Figure 2 d) together with
the corresponding free-running bandwidths as a function of DC-bias. The
injected bandwidth stays at 60 cm-1, independent of the DC driving current.

Figure 2: (a) Free-running spectra of the QCl at 700, 950, 1200 mA and in (b)
the corresponding SWIFTS spectra under 35 dBm injection at the ”sweet spot”
(maximum bandwidth) together with the spectrum product (black dots). (c)
Electronic beatnote-map of the device without modulation and the injection
frequencies corresponding to the largest spectral broadening at 35 dBm
injection (white dots) and (d) the maximum spectral bandwidth in free-running
(red) vs. under injection (black). The dashed lines represent the average
bandwidth of each dataset. In the free-running case only the data starting at
900 mA bias are used. Figure 3: (a) Logarithmic spectral map as a function of
modulation frequency at 35 dBm injection level. The white vertical lines
indicate the modulation frequencies $f_{1}$ and $f_{2}$ which generate spectra
with minimal overlap. (b) Schematic depiction of the time-dependent switching
between modulation frequencies $f_{1}$ (orange) and $f_{2}$ (blue) for
demonstrating the acquisition of multiplexed interferograms. The dots
represent the samples acquired by the oscilloscope. (c) Zoomed-in section of
the experimental interferogram. The data points corresponding to the different
modulation frequencies are colored according to (b). (d) Comparison of the
spectra on the white lines in (a) and (e) spectra of two interferograms
extracted by demultiplexing from the data displayed in (c).

The possibility of fast, modulation frequency controlled spectral tuning opens
up the opportunity for time-resolved multi-species gas monitoring by
multiplexing at least two separate spectral regions, as was previously
demonstrated on two-section dual-wavelength devices.[36, 37] In our case the
device is single section and offers superior fine-tuning capabilities as we
can freely switch between any spectral distribution found in the spectral map
in Figure 3 a). While each individual spectrum covers a relatively small
bandwidth, multiplexing several well-known spectral regions allows for
monitoring multiple species based on relative intensity without the need for a
spectrometer which increases the overall detection speed. At 35 dBm injection,
the position of the ”sweet spot” in our device is well defined with a sharp
transition between a blue-shifted center-frequency of the QCL-spectrum at
lower injection frequencies and a red-shifted center-frequency at higher
injection frequencies, as depicted in the spectral map in Figure 3 a). The
respective shifts are sufficiently large that there are spectra on both sides
of the ”sweet spot” with small overlap with respect to each other, such as the
ones marked by the white lines. To experimentally verify the viability of fast
spectral tuning of our QCL comb, we acquired an interferogram of the injected
QCL while switching between the frequencies $f_{1}$ (orange) and $f_{2}$
(blue) (Figure 3 b)). A zoomed-in version of the experimentally acquired data
is shown in Figure 3 c), where the data points are colored according to the
corresponding injected frequencies $f_{1}$ (orange) and $f_{2}$ (blue). For
the spectral analysis, the acquired data was demultiplexed to yield several
independent interferograms with only one sample point per switching period,
each. Our switching-rate and thus the achievable time-resolution was limited
by the frequency-modulation capabilities of our RF-generator output, which
went up to 20 kHz in square-wave operation. Comparison between the
continuously modulated case and switching modulation at 20 kHz is shown in
Figures 3 d) and e). Here, the spectrum generated under injection at $f_{1}$
is plotted in orange together with the spectrum at $f_{2}$ in blue for each
case. While there is some residual overlap between the two spectral ranges,
the relative intensity differs by at least 20 dB. This should be sufficient to
allow for independent monitoring of intensity changes of both ranges based on
amplitude. The spectra show a high level of agreement between the continuously
injected and the switched case, indicating the potential of achieving similar
results with even higher switching rates between the injected frequencies and
thereby improving the time-resolution even further. Moreover, this approach
offers a high level of in-situ tunability, as the covered spectral ranges
directly depend on the modulation frequency. By fine-tuning the modulation
parameters, different types of gases as well as more than two species might be
accommodated simultaneously without change in device, or DC operational point.

Figure 4: (a) Reconstructed time-traces from the SWIFTS measurements in free-
running and with 25 and 35 dBm RF-injection. The orange lines in the 35 dBm
trace mark the start and end of the pulse during each round-trip. (b)
Instantaneous frequency of the QCL output, as retrieved from the same SWIFTS
measurements. The orange lines coincide with the ones in (a). (c) The
corresponding SWIFTS spectra, the Spectrum Products are added as black dots
for comparison.

The effect of high-power RF-injection on the temporal shape of the QCL output
can be evaluated by looking at the reconstructed time-traces from the SWIFTS
measurements. In Figure 4 a) the averaged reconstructed time-traces from
SWIFTS measurements are shown at free-running operation as well as under 25
and 35 dBm injection at 11.090 GHz and 11.088 GHz, respectively. As observed
in the work of Kapsalidis et al.,[35] the reconstructed time-trace from the
SWIFTS data changes from a quasi-continuous waveform typical for free-running
QCLs to having a significant amplitude modulation at 25 dBm injection. Upon
further increasing the modulation power to 35 dBm the modulation makes way for
pulsed operation. The pulse-length is roughly on the order of half the round-
trip time of the device. This is also apparent in the frequency traces in
Figure 4 b) where the uppermost trace ceases to have a well-defined frequency
throughout the round-trip due to the lack of intensity in this region.
Notably, there still is a linear chirp which spans the whole spectral
bandwidth, where there is signal. This is analog to what has been demonstrated
in ICLs[38] and therefore indicates that the output could be further
compressed in a suitable compressor in order to obtain shorter pulses.[39] The
comparison between SWIFTS and spectrum product in Figure 4 shows a gap close
to the center of the spectrum injected with 35 dBm power for this measurement
series. This is a direct consequence from the sharp spectral transition at the
maximum bandwidth at very high injection power. As seen in Figure 1 c), there
is a slight drop in average coherence of the spectra directly below injection
of 11.088 GHz, which coincides with the frequency where the largest bandwidth
occurs. Therefore, the coherence is critically dependent on the injection
frequency at this point. However, the gap is at rather lower spectral power
and other SWIFTS measurements with higher coherence have resulted in the same
temporal shape. Therefore the impact on the disparity between reconstructed
and actual output signal should be small enough to account for a weak
incoherent background at most.

We have corroborated these results by measuring the interferometric auto-
correlation (IAC) of the device under injection. Additionally, to confirm the
validity of our results we simulated the IAC traces from the reconstructed
SWIFTS data. The results for modulation at 25 and 35 dBm are shown in Figure 5
a). In both, simulation and experiment there is a clear difference between the
strongly modulated (25 dBm) and the pulsed signal (35 dBm). The shapes of
experiment and simulation agree well for both modulation powers, which further
reinforces the evidence for the pulsed waveform retrieved by SWIFTS. The
linear chirp during the pulses is manifest in the modulation depth of the
baseline of the IAC at 35 dBm injection. It needs to be noted that in order to
achieve the high incident power required for two-photon detection, any
attenuating optics were removed from the beam-path, thus increasing the amount
of optical feedback on the QCL. Thus, a discrepancy between experimental and
simulated data from previous measurements is expected. Therefore it is worth
noting that the IAC data at 35 dBm contains significantly less noise than the
one at 25 dBm, indicating an increased robustness to feedback at 35 dBm. In
b), the injection-power dependent maximum-to-baseline ratio is plotted as a
function of injected power. The kinks in the data indicate a non-trivial
relationship between injected power and degree of pulsing. For injection
frequencies further from the ”sweet spot” the maximum-to-baseline ratio is
comparatively lower at high injection powers, which indicates that the pulse-
characteristics of the output are weaker.

Figure 5: (a) Experimental (blue) and simulated (orange) two-photon absorption
data for injection levels 25 and 35 dBm. The ratio between maximum and
baseline is annotated in green. (b) Maximum-baseline ratios as a function of
injected RF-power at modulation frequencies 11.089 GHz (blue) and 11.090 GHz
(orange).

##  3 Conclusion

In this work we could demonstrate control over the spectral and temporal
properties of a QCL frequency comb using electronic RF-injection. We could
achieve consistent coherent spectral broadening throughout the device’s
dynamical range, which amounts to a frequency comb with an almost constant
spectral distribution of the modes but a repetition rate that is tunable by up
to 80 MHz. Moreover we could show rapid switching between two distinct
spectral ranges by switching between injection frequencies at a switching rate
limited by the capabilities of our RF-generator. In the time-domain we could
demonstrate the ability to transition from quasi-CW to pulsed output, verified
by IAC measurements, by ramping up the injection power.

These results show promise for improving spectroscopic applications based on
QCLs, especially dual-comb spectroscopy or monolithic time-resolved monitoring
of multiple gas species. Improved RF-coupling for a stronger response as well
as a broadened gain bandwidth are steps that could be taken to further improve
these devices. Moreover, the potential application in pulse generation after
recompression opens up the pathway to QCL-based non-linear optics, such as
supercontinuum generation.

##  4 Experimental Section

To assess the spectral and temporal response of the device, the optical output
was fed through a Fourier-transform spectrometer (FTS) and onto an MCT
detector. In addition, either a fast QWIP was used when performing shifted
wave interference Fourier transform spectroscopy (SWIFTS), or an InSb detector
with a cut-off frequency at 5 µm as a two-photon detector[40] in the case of
interferometric autocorrelation (IAC) measurements. The measurement setup is
shown in Figure 6. All RF-powers indicated in this paper refer to the nominal
output level of our RF-synthesizer after a 35 dBm amplifier.

Figure 6: Schematic of the experimental setup. The QCL is electrically driven
through a high power DC and RF-probe (inset). An RF-circulator is connected to
the circuit via a bias-tee, which allows for simultaneous injection and read-
out of current modulation in the laser bias. The optical output coming from
the QCL is fed into an FTS. The DC spectrum is recorded using a MCT detector.
A high-speed QWIP is used to acquire the SWIFTS quadrature spectra. Further,
an InSb detector with 5 µm cut-off wavelength, used as a two-photon detector,
allows us to record the second-order autocorrelation of the QCL.

Acknowledgements

The authors would like to thank the BRIDGE program by the Swiss National
Science Foundation and Innosuisse under the Project CombTrace
(No.20B2-1_176584/1) and the Qombs Project by the European Union’s Horizon
2020 research and innovation program (No. 820419) for their financial support.
They would also like to thank Philipp Täschler for proofreading of the paper
and fruitful discussions.

Conflict of Interest

The authors declare no conﬂict of interest.

## References

  * [1] M. Phillips,  _Scientific American_ 1992, _267_ , 1 74.
  * [2] D. J. Bamford, D. J. Cook, S. J. Sharpe, A. D. Van Pelt,  _Applied Optics_ 2007, _46_ , 19 3958.
  * [3] M. Sigrist, R. Bartlome, D. Marinov, J. Rey, D. Vogler, H. Wächter,  _Applied Physics B_ 2008, _90_ , 2 289.
  * [4] B. H. Lee, E. C. Wood, M. S. Zahniser, J. B. McManus, D. D. Nelson, S. C. Herndon, G. W. Santoni, S. C. Wofsy, J. W. Munger,  _Applied Physics B_ 2011, _102_ , 2 417.
  * [5] J. Faist, F. Capasso, D. L. Sivco, C. Sirtori, A. L. Hutchinson, A. Y. Cho,  _Science_ 1994, _264_ , 5158 553.
  * [6] P. Jouy, J. M. Wolf, Y. Bidaux, P. Allmendinger, M. Mangold, M. Beck, J. Faist,  _Applied Physics Letters_ 2017, _111_ , 14 141102.
  * [7] B. Schwarz, C. A. Wang, L. Missaggia, T. S. Mansuripur, P. Chevalier, M. K. Connors, D. McNulty, J. Cederberg, G. Strasser, F. Capasso,  _ACS Photonics_ 2017, _4_ , 5 1225.
  * [8] P. Friedli, H. Sigg, B. Hinkov, A. Hugi, S. Riedi, M. Beck, J. Faist,  _Applied Physics Letters_ 2013, _102_ , 22 222104.
  * [9] J. B. Khurgin, Y. Dikmelik, A. Hugi, J. Faist,  _Applied Physics Letters_ 2014, _104_ , 8 081118.
  * [10] A. Hugi, G. Villares, S. Blaser, H. C. Liu, J. Faist,  _Nature_ 2012, _492_ , 7428 229.
  * [11] D. Burghoff, T.-Y. Kao, N. Han, C. W. I. Chan, X. Cai, Y. Yang, D. J. Hayton, J.-R. Gao, J. L. Reno, Q. Hu,  _Nature Photonics_ 2014, _8_ , 6 462.
  * [12] M. Piccardo, B. Schwarz, D. Kazakov, M. Beiser, N. Opačak, Y. Wang, S. Jha, J. Hillbrand, M. Tamagnone, W. T. Chen, A. Y. Zhu, L. L. Columbo, A. Belyanin, F. Capasso,  _Nature_ 2020, _582_ , 7812 360.
  * [13] B. Meng, M. Singleton, M. Shahmohammadi, F. Kapsalidis, R. Wang, M. Beck, J. Faist,  _Optica_ 2020, _7_ , 2 162.
  * [14] T. Udem, R. Holzwarth, T. W. Hänsch,  _Nature_ 2002, _416_ , 6877 233.
  * [15] I. Coddington, N. Newbury, W. Swann,  _Optica_ 2016, _3_ , 4 414.
  * [16] Z. Chen, T. W. Hänsch, N. Picqué,  _Proceedings of the National Academy of Sciences_ 2019, _116_ , 9 3454.
  * [17] L. A. Sterczewski, J. Westberg, M. Bagheri, C. Frez, I. Vurgaftman, C. L. Canedy, W. W. Bewley, C. D. Merritt, C. S. Kim, M. Kim, J. R. Meyer, G. Wysocki,  _Optics Letters_ 2019, _44_ , 8 2113.
  * [18] M. Gianella, A. Nataraj, B. Tuzson, P. Jouy, F. Kapsalidis, M. Beck, M. Mangold, A. Hugi, J. Faist, L. Emmenegger,  _Optics Express_ 2020, _28_ , 5.
  * [19] R. Paiella, R. Martini, F. Capasso, C. Gmachl, H. Y. Hwang, D. L. Sivco, J. N. Baillargeon, A. Y. Cho, E. A. Whittaker, H. C. Liu,  _Applied Physics Letters_ 2001, _79_ , 16 2526.
  * [20] M. Singleton, P. Jouy, M. Beck, J. Faist,  _Optica_ 2018, _5_ , 8 948.
  * [21] J. Hillbrand, D. Auth, M. Piccardo, N. Opačak, E. Gornik, G. Strasser, F. Capasso, S. Breuer, B. Schwarz,  _Physical Review Letters_ 2020, _124_ , 2 023901.
  * [22] D. J. Kane, R. Trebino,  _IEEE Journal of Quantum Electronics_ 1993, _29_ , 2 571.
  * [23] R. Martini, C. Gmachl, J. Falciglia, F. Curti, C. Bethea, F. Capasso, E. Whittaker, R. Paiella, A. Tredicucci, A. Hutchinson, D. Sivco, A. Cho,  _Electronics Letters_ 2001, _37_ , 3 191.
  * [24] R. Paiella, F. Capasso, C. Gmachl, H. Y. Hwang, D. L. Sivco, A. L. Hutchinson, A. Y. Cho, H. C. Liu,  _Applied Physics Letters_ 2000, _77_ , 2 169.
  * [25] P. Gellie, S. Barbieri, J.-F. Lampin, P. Filloux, C. Manquest, C. Sirtori, I. Sagnes, S. P. Khanna, E. H. Linfield, A. G. Davies, H. Beere, D. Ritchie,  _Optics Express_ 2010, _18_ , 20 20799.
  * [26] M. R. St‐Jean, M. I. Amanti, A. Bernard, A. Calvar, A. Bismuto, E. Gini, M. Beck, J. Faist, H. C. Liu, C. Sirtori,  _Laser & Photonics Reviews_ 2014, _8_ , 3 443.
  * [27] J. Hillbrand, A. M. Andrews, H. Detz, G. Strasser, B. Schwarz,  _Nature Photonics_ 2019, _13_ , 2 101.
  * [28] A. Forrer, L. Bosco, M. Beck, J. Faist, G. Scalari,  _Photonics_ 2020, _7_ , 1 9.
  * [29] A. Mottaghizadeh, D. Gacemi, P. Laffaille, H. Li, M. Amanti, C. Sirtori, G. Santarelli, W. Hänsel, R. Holzwart, L. H. Li, E. H. Linfield, S. Barbieri,  _Optica_ 2017, _4_ , 1 168.
  * [30] S. Barbieri, M. Ravaro, P. Gellie, G. Santarelli, C. Manquest, C. Sirtori, S. P. Khanna, E. H. Linfield, A. G. Davies,  _Nature Photonics_ 2011, _5_ , 5 306.
  * [31] J. Hillbrand, N. Opačak, M. Piccardo, H. Schneider, G. Strasser, F. Capasso, B. Schwarz,  _Nature Communications_ 2020, _11_ , 1 5788.
  * [32] E. Rodriguez, A. Mottaghizadeh, D. Gacemi, M. Jeannin, Z. Asghari, A. Vasanelli, Y. Todorov, Q. J. Wang, C. Sirtori,  _Laser & Photonics Reviews_ 2020, _14_ , 5 1900389\.
  * [33] K. Unterrainer, R. Colombelli, C. Gmachl, F. Capasso, H. Y. Hwang, A. M. Sergent, D. L. Sivco, A. Y. Cho,  _Applied Physics Letters_ 2002, _80_ , 17 3060.
  * [34] A. Calvar, M. I. Amanti, M. Renaudat St-Jean, S. Barbieri, A. Bismuto, E. Gini, M. Beck, J. Faist, C. Sirtori,  _Applied Physics Letters_ 2013, _102_ , 18 181114.
  * [35] F. Kapsalidis, B. Schneider, M. Singleton, M. Bertrand, E. Gini, M. Beck, J. Faist,  _Applied Physics Letters_ 2021, _118_ , 7 071101.
  * [36] J. Jágerská, P. Jouy, B. Tuzson, H. Looser, M. Mangold, P. Soltic, A. Hugi, R. Brönnimann, J. Faist, L. Emmenegger,  _Optics Express_ 2015, _23_ , 2 1512.
  * [37] F. Kapsalidis, M. Shahmohammadi, M. J. Süess, J. M. Wolf, E. Gini, M. Beck, M. Hundt, B. Tuzson, L. Emmenegger, J. Faist,  _Applied Physics B: Lasers and Optics_ 2018, _124_ , 6 107.
  * [38] J. Hillbrand, M. Beiser, A. M. Andrews, H. Detz, R. Weih, A. Schade, S. Höfling, G. Strasser, B. Schwarz,  _Optica_ 2019, _6_ , 10 1334.
  * [39] M. Singleton, M. Beck, J. Faist,  _Journal of the Optical Society of America B_ 2019, _36_ , 6 1676.
  * [40] D. L. Boiko, A. V. Antonov, D. I. Kuritsyn, A. N. Yablonskiy, S. M. Sergeev, E. E. Orlova, V. V. Vaks,  _Applied Physics Letters_ 2017, _111_ , 17 171102.


#  LRP-QViT: Mixed-Precision Vision Transformer Quantization via Layer-wise
Relevance Propagation

Navin Ranjan Andreas Savakis
Rochester Institute of Technology
Rochester, New York 14623, USA
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Vision transformers (ViTs) have demonstrated remarkable performance across
various visual tasks. However, ViT models suffer from substantial
computational and memory requirements, making it challenging to deploy them on
resource-constrained platforms. Quantization is a popular approach for
reducing model size, but most studies mainly focus on equal bit-width
quantization for the entire network, resulting in sub-optimal solutions. While
there are few works on mixed precision quantization (MPQ) for ViTs, they
typically rely on search space-based methods or employ mixed precision
arbitrarily. In this paper, we introduce LRP-QViT, an explainability-based
method for assigning mixed-precision bit allocations to different layers based
on their importance during classification. Specifically, to measure the
contribution score of each layer in predicting the target class, we employ the
Layer-wise Relevance Propagation (LRP) method. LRP assigns local relevance at
the output layer and propagates it through all layers, distributing the
relevance until it reaches the input layers. These relevance scores serve as
indicators for computing the layer contribution score. Additionally, we have
introduced a clipped channel-wise quantization aimed at eliminating outliers
from post-LayerNorm activations to alleviate severe inter-channel variations.
To validate and assess our approach, we employ LRP-QViT across ViT, DeiT, and
Swin transformer models on various datasets. Our experimental findings
demonstrate that both our fixed-bit and mixed-bit post-training quantization
methods surpass existing models in the context of 4-bit and 6-bit
quantization.

##  1 Introduction

Vision Transformers (ViTs) have achieved state-of-the-art (SOTA) performance
across various vision tasks, including image classification [13, 28, 42],
object detection [32, 4] and segmentation [50, 38]. However, their
computational demands, high memory footprint, and significant energy
consumption make them impractical for deployment on resource constrained
platforms. Compression and acceleration techniques have been investigated for
ViTs, aiming to reduce the original network size while maintaining
performance. Various approaches for model reduction include network pruning
[15, 47], low-rank decomposition [9], quantization [26, 23, 19], knowledge
distillation [7, 24], and dynamic token reduction [34, 46].

Model quantization is an effective and popular approach to reduce the model
size by converting floating-point parameters into lower-bit representations,
but its drawback is an associated drop in performance. Thus, research focused
on quantization methods that mitigate the performance drop is of significant
importance. Quantization-Aware Training (QAT) involves quantizing model
parameters to lower bit precision and retraining on the entire dataset to
optimize the quantized model and recover from the performance drop. However,
this approach results in significant training costs in time and resources. In
contrast, Post Training Quantization (PTQ) is considered an efficient and
practical compression technique. It directly quantizes the model without the
need for retraining. Instead, it simply uses a small sample of images to
calibrate the quantized parameters.

Various studies have been conducted using PTQ for ViT quantization [26, 10,
48, 23]. These works have identified several bottleneck components that limit
the performance of the quantized model, such as LayerNorm, Softmax and GELU,
and proposed PTQ schemes with improvements. However, most of the existing work
is based on the premise that all the layers in ViTs contribute equally towards
the model output and therefore use equal bit precision for the entire network
leading to sub-optimal solutions. Fixed bit quantization forces important
layers to be quantized with the same bit precision as the unimportant ones,
missing an opportunity to further reduce the model size and enhance
performance. Mixed-precision quantization (MPQ) addresses this limitation by
allowing different bit precision for different layers. This enables the
crucial layer to use higher bit precision than others. Most of the prior works
for MPQ focus on convolutional neural networks (CNNs) and utilize policy
search based methods [30, 31, 44], or criterion based methods [11, 12] to
determine the optimal bit precision.

In this paper, we propose an explainability approach for mixed-bit
quantization of vision transformer models [5, 6], based on the contribution of
each layer to the model performance. We utilize layer relevance propagation to
obtain contribution scores for all layers that inform a mixed-precision bit
allocation strategy for the quantization of different layers.

In our framework, we adopt and improve the RepQ-ViT [23] framework for model
quantization. Specifically, for the post-LayerNorm activation, we introduce
clipped channel-wise quantization to remove outliers and mitigate the effects
of excessive inter-channel variation at the inference stages. This clipping is
achieved by adjusting LayerNorm’s affine factors and next layer weights. For
post-Softmax activations, we fully adopt the modification made in [23], which
is initially quantized with $log\sqrt{2}$ quantizer to achieve higher
representation for accuracy and rescale to $log2$ quantizer during the
inference process for friendly quantization. With the improved quantization
model and explainable mixed precision bit allocation, our main contributions
are summarized as follows.

  * •

We propose LRP-QViT, a mixed precision framework for the quantization of
vision transformer models.

  * •

LPR-QViT uses layer relevance propagation to assign contribution scores to
each layer and uses them to guide the bit allocation.

  * •

We additionally introduce clipped channel-wise quantization for post-LayerNorm
activation, that removes outliers and improves performance for both fixed-bit
and mixed-precision quantization.

  * •

Our results on ViT, DeiT and Swin models demonstrate the superiority of LRP-
QViT over fixed-bit quantization methods on classification, detection and
segmentation benchmarks.

##  2 Related Work

###  2.1 Vision Transformer

The success of transformers in natural language processing (NLP) has inspired
the application of vision transformer models on computer vision tasks. The
seminal ViT architecture [13] takes image patches as tokens and applies a
transformer-based model for image classification, achieving remarkable
success, which is largely attributed to the global receptive fields captured
by the powerful self-attention (SA) mechanism. After the success of ViT,
several works such as Swin [28], DeiT [39] and PVT [42], have further improved
performance. DeiT [39] introduced a knowledge distillation strategy that
relies on distillation tokens and employed various data augmentation
techniques, significantly enhancing the effectiveness and efficiency of ViTs.
Swin [28] adopts a hierarchical architecture with shifted windows, capturing
both local and global contextual information for improved performance. Beyond
image classification, vision transformers have been successfully applied to
other tasks, such as object detection [3, 8], segmentation [17], and pose
estimation [45].

To address the high computational demand of ViTs, various efforts have been
directed towards developing smaller and faster variants, such as DynamicViT
[34], EfficientFormer [20], TinyViT [43] and MiniViT [49]. However, these
models retain full-precision parameters. In this paper, we focus on model
quantization to address the challenge of high computational resources.

###  2.2 Model Quantization

Quantization reduces memory usage and computational demand by representing
network parameters with lower precision than the standard full-precision
model. Previous research has mainly focused on employing fixed-bit precision
across all the layers in both QAT [19, 21] and PTQ [48, 26, 10, 27, 23]. PTQ
stands out as an efficient and practical compression approach, as it directly
quantizes the model without retraining. PTQ4ViT [48] used twin uniform
quantization method to reduce quantization error in softmax and GELU
activations and proposed a Hessian guided metric to search for the
quantization scale. APQ-ViT [10] introduced a block-wise calibration scheme to
address the optimization challenges in quantized networks and proposed
Matthew-effect preserving quantization to maintain the power-law
redistribution of the softmax layer. FQ-ViT [26] proposed the Power-of-Two
Factor to handle inter-channel variation in LayerNorm and Log-Int-Softmax to
quantize softmax layer. RepQ-ViT [23] separated the quantization and inference
stages to achieve accurate quantization and efficient inference. In the
quantization stage, it used channel-wise quantization to address inter-channel
variation in LayerNorm and log$\sqrt{2}$ quantization to enhance the
representation capabilities of softmax layer. During the inference stage, it
reparameterized channel-wise to layer-wise quantization and log$\sqrt{2}$ to
log2 quantization for hardware-friendly inference.

Most existing works on mixed precision are primarily focused on CNNs using
criterion based methods [11, 12], or search-based methods [41, 31]. There are
few works on mixed-precision for ViTs [29, 44]. In [29], mixed precision for
the multi-head self-attention (MHSA) and multi-layer perceptron (MLP) module
is based on sensitivity, which is calculated using the nuclear norm of the
attention map for MHSA and the output feature for MLP. PMQ [44] estimated the
sensitivity of the layer by measuring the error induced by the network when
the layer is removed and used a Pareto frontier approach to allocate optimal
bit widths. In this paper, we leverage transformer explainability to estimate
the ViT layer sensitivity for allocating mixed precision for quantization.
Furthermore, we adopt and improve the RepQ [23] framework for efficient mixed
precision PTQ.

###  2.3 Explainability for Transformers

The majority of prior research on explainability has concentrated on CNNs,
with a main focus on gradient methods [37, 36] or attribution methods [2, 14].
However, as vision transformer architectures have advanced, there has been a
notable upswing in research on explainability for transformers. In the
Attention Rollout method [1], attention scores from different layers are
linearly combined, yet this approach struggles to distinguish between positive
and negative contributions. The Layer-wise Relevance Propagation method
propagates relevance from the predicted class backward to the input image.
Several works have applied LRP to Transformers [40, 2]. However, many of these
studies overlook the propagation of attention across all layers and neglect
parts of the network that perform mixing of two activation maps, such as skip
connections and matrix multiplication. Additionally, most of the work does
distinguish between the positive and negative contribution provided by the
layers toward the model decision. Without such distinction, both positive and
negative contributions are mixed, resulting in higher than necessary relevance
scores.

In [6], relevance and gradients information are used in a way that iteratively
removes the negative contribution and calculate the accurate relevance score
for each attention head in each layers of a transformer model. In this work,
we adopt the relevance score calculation mentioned in [6] and extend the
approach to calculate the relevancy score for other layers, such as qkv layer,
linear projection layer, matrix multiplication layers, and fully connected
layers.

##  3 Methodology

###  3.1 Vision Transformer Architecture

The vision transformer takes an image and reshapes into a sequence of $N$
flattened 2D patches. Each patch is mapped into a vector of $D$ dimensions via
a linear projection layer, denoted by $X\in{R}^{N\times D}$. The core
structure of the standard ViT contains several blocks, each consisting of a
MHSA module and a MLP module. The MHSA module is designed to understand the
relationship between tokens, extracting features with a global perspective. In
the $l^{th}$ transformer block, the MHSA module process the input sequence
$X^{l}$, which undergoes initial linear projections to obtain the query
$Q_{l}=X^{l}W^{q}_{i}$, key $K_{l}=X^{l}W^{k}_{i}$, and value
$V_{l}=X^{l}W^{v}_{i}$, abbreviated as qkv. Subsequently, attention scores are
computed between queries and keys, followed by a softmax layer. The final
output of MHSA is obtained by concatenating the outputs from multiple heads
within the MHSA, as follows:

$\displaystyle\text{Attn}_{i}=\text{Softmax}\left(\frac{Q_{i}\cdot
K_{i}^{T}}{\sqrt{D_{h}}}\right)V_{i},$ (1) $\displaystyle\text{MHSA}(X^{l})=[\
\text{Attn}_{1},\text{Attn}_{2},...,\text{Attn}_{i}]\ W^{o},$ (2)

where $h$ is the number of attention heads and $D_{h}$ is the feature size of
each head, and $i=1,2,...,h$. The MLP module employs two fully connected
layers separated by a GELU activation to project the features into a high-
dimensional space and learn representations, as follows:

$\displaystyle\text{MLP}(Y^{l})=\text{GELU}\left(Y^{l}W^{1}+b^{1}\right)W^{2}+b^{2}.$
(3)

where $Y^{l}$ denotes input for MLP, $W^{1}\in\mathbb{R}^{D\times D}$,
$b^{1}\in\mathbb{R}^{D_{f}}$, $W^{2}\in\mathbb{R}^{D_{f}\times D}$, and
$b^{2}\in\mathbb{R}^{D}$. The LayerNorm $(LN)$ are applied before each modules
and residuals are added after each module, the transformer block is formulated
as:

$\displaystyle Y^{l}=X^{l}+\text{MHSA}\left(LN\left(X^{l}\right)\right),$ (4)
$\displaystyle X^{l+1}=Y^{l}+\text{MLP}\left(LN\left(Y^{l}\right)\right).$ (5)

The large matrix multiplications in MHSA and MLP contribute significantly to
computational costs. Following [29, 23], we quantize all the weights and
inputs involved in matrix multiplication, linear embedding, and softmax while
keeping layer normalization at full precision.

###  3.2 Model Quantization

In this paper, we quantize the weights and activations of linear layers,
convolutional layers, and matrix multiplication using uniform quantizer
function and quantized softmax activation using a $log2$ quantizer function.
The uniform quantization splits the data range equally and is defined as:

$\displaystyle\text{Quant: }x^{q}$
$\displaystyle=\text{clip}\left(\left\lfloor\frac{x}{s}\right\rceil+z,0,2^{b}-1\right)$
(6) $\displaystyle\text{DeQuant: }\bar{x}$
$\displaystyle=s\cdot(x^{q}-z)\approx x$ (7) $\displaystyle s$
$\displaystyle=\frac{\max(x)-\min(x)}{2^{b}-1},\quad\text{and}$ $\displaystyle
z$ $\displaystyle=\left\lfloor-\frac{\min(x)}{s}\right\rceil$ (8)

Here, $x$ is the original floating-point weights or inputs, $x^{q}$ represents
quantized values, b is the quantization bit-precision, $\lfloor\cdot\rceil$
denotes the round operator, and clip denotes the elements in the tensor that
exceed the ranges of the quantization domain are clipped. $s$ is the
quantization scale and $z$ is the zero-point, both of which is determined by
the lower and upper bound of $x$. In de-quantization process, the de-quantized
value $\bar{x}$ approximately recovers $x$.

The $log2$ quantization function converts the quantization process from linear
to exponential and is defined as:

$\displaystyle\text{Quant: }x^{q}$
$\displaystyle=\text{clip}\left(\left\lfloor-\log_{2}\frac{x}{s}\right\rceil,0,2^{b}-1\right)$
(9) $\displaystyle\text{DeQuant: }\bar{x}$ $\displaystyle=s\cdot
2^{-x^{q}}\approx x$ (10)

####  3.2.1 Clipped Reparameterization for LayerNorm Activations

In ViTs, during inference, LayerNorm computes the statistics $\mu_{x}$, and
$\sigma_{x}$ in each forward step and normalizes the input
$X\in\mathbb{R}^{N\times D}$. Then, affine parameters
$\gamma\in\mathbb{R}^{D}$ and $\beta\in\mathbb{R}^{D}$ re-scale the normalized
input to another learned distribution. The LayerNorm process is defined as:

$\displaystyle\text{LayerNorm(X)}=\frac{X-\mu_{x}}{\sqrt{{\sigma_{x}^{2}}+\epsilon}}\odot\gamma+\beta,$
(11)

where $\odot$ denotes Hadamard product.

Looking into the Post-LayerNorm activations, we observe an extreme inter-
channel variation, a critical factor that reduces post training quantization
performance. Several studies have investigated this issue and proposed
solutions. In [35], group-wise quantization was implemented, treating
individual matrices associated with a head in MHSA as one group and assigning
different quantization parameters to each group. In [26], channel-wise
quantization was applied, providing different channels with different
parameters based on a power-of-two-factor approach. In [23], quantization-
inference decoupling techniques were used, performing quantization using a
channel-wise approach. Then, during inference, channel-wise quantized
parameters were re-parameterized to layer-wise quantization by taking an
average.

Taking inspiration from [26, 23], we adopt the quantization-inference
decoupling paradigm and propose a simple but effective strategy using Clipped
Reparameterization for LayerNorm activations (CRL). This involves restricting
inter-channel variations by clipping outliers within a set number of standard
deviations around the channel mean value. Specifically, for $l^{th}$
transformer block, given the input $X^{l}_{LN}$, we perform channel-wise
quantization to obtain the scale $s\in\mathbb{R}^{D}$ and zero-point
$z\in\mathbb{R}^{D}$. Next, we calculate a clipped channel-wise quantized
scale $\hat{s}$ and zero-point $\hat{z}$, as follows:

$\displaystyle\hat{s}=\text{clip}(s,\mu_{s}-2\sigma,\mu_{s}+2\sigma)$
$\displaystyle\hat{z}=\text{clip}(z,\mu_{z}-2\sigma,\mu_{z}+2\sigma)$ (12)

The variation factors between the original and clipped parameters are denoted
as $v_{1}=s/\hat{s}$ and $v_{2}=z-\hat{z}$. Eq. (8) can then be expressed as:

$\displaystyle\hat{s}=\frac{s}{v_{1}}=\frac{\left[\max(x)-\min(x)\right]/v_{1}}{2^{b}-1}$
(13) $\displaystyle\hat{z}=z-v_{2}=\left\lfloor-\frac{\min(x)+s\odot
v_{2}}{s}\right\rceil$ (14)

In Eq. (13), dividing each channel of $X^{l}_{LN}$ by $v_{1}$ results in
$\hat{s}$. Similarly, in Eq. (14), adding $s\odot v_{2}$ to each channel of
$X^{l}_{LN}$ gives $\hat{z}$. These operations can be achieved by adjusting
the LayerNorm’s affine factors as follows:

$\displaystyle\hat{\beta}=\frac{\beta+s\odot v_{2}}{v_{1}},\text{\hskip
7.22743pt}\hat{\gamma}=\frac{\gamma}{v_{1}}$ (15)

This reparameterization induces a change in the activation distribution,
specifically expressed as $\hat{X}^{l}_{LN}=\left(X^{l}+s\odot
v_{2}\right)/v_{1}$. In the MHSA module, the layer following LayerNorm is a
linear projector layer for qkv. The reparameterized shift in the qkv layer is
expressed as

$\displaystyle X^{l}_{LN}\cdot W^{qkv}=\frac{X^{l}_{LN}+s\odot
v_{2}}{v_{1}}\left(v_{1}\odot W^{qkv}\right)$
$\displaystyle+\left(b^{qkv}-\left(s\odot v_{2}\right)W^{qkv}\right)$ (16)

Here, $W^{qkv}\in\mathbb{R}^{D\times D_{h}}$ and
$b^{qkv}\in\mathbb{R}^{3D_{h}}$ represent the weight and bias of the qkv
layer. To offset this distribution shift, the weight of the subsequent layer
can be adjusted, as outlined below:

$\displaystyle\hat{W}^{qkv}=v_{1}\odot W^{qkv}\text{\hskip 36.135pt}$
$\displaystyle\hat{b}^{qkv}=b^{qkv}-\left(s\odot v_{2}\right)W^{qkv}$ (17)

A similar adjustment strategy is applied to the input $Y^{l}$ in the MLP
module. Consequently, through the modification of LayerNorm affine factors and
the weights and bias of the next layer, we reparameterize channel-wise
quantization to achieve clipped channel-wise quantization for LayerNorm
activation, effectively addressing the issue of inter-channel variation.

####  3.2.2 Nonlinear Quantization for Softmax Activations

In ViTs, the softmax operation transforms the attention scores of the MHSA
module into probabilities, exhibiting a power-law distribution that is highly
unbalanced and unsuitable for quantization. We observe that the majority of
the distribution comprises very small values, and only a few values have
larger magnitudes. Previous approaches [26] directly applied a $log2$
quantizer, while in [10], a Matthew-effect preserving quantization was
proposed. Although these methods outperform uniform quantization, they do not
consistently achieve satisfactory performance.

In [23], a quantization-inference decoupling approach is employed. A
$log\sqrt{2}$ quantizer is used to quantize the softmax activation, instead of
$log2$, as it offers higher quantization resolution and accurately describes
the power-law distribution. During the inference stage, the $log\sqrt{2}$
quantizer parameter is reparameterized to a hardware-friendly $log2$
quantizer, achieving both accuracy through $log\sqrt{2}$ and efficiency
through $log2$. In this we adopt RepQ-ViT.

###  3.3 Layer-Wise Relevance Propagation

Layer-wise Relevance Propagation is an explainable artificial intelligence
approach that provides insight into the contribution of each feature to the
model output. It works by assigning a relevance score to the output layer and
then back-propagating it through all layers of the network to the input
features. These relevance scores serve as indicators for computing a
contribution score for each layer. We use these contribution scores to assign
mixed-precision bit allocation for PTQ.

####  3.3.1 Relevance and Gradients

Let $C$ be the number of classes of the classification head, and $c\in
1\dots|C|$ the class to be visualized. We propagate relevance $\left(R\right)$
and gradients $\left(\nabla\right)$ with respect to class $c$. Let
$x^{\left(n\right)}$ be the input of $L^{n}$ layer of of the network, where
$n\in\left[1\dots N\right]$, $x^{N}$ is the input and $x^{1}$ is the output of
network. The gradients with respect to the classifier’s output $y$, at class
$t$, namely $y_{t}$, is given by

$\displaystyle\nabla{x}^{n}_{j}=\frac{\partial{y}_{t}}{\partial{x}^{n}_{j}}=\sum_{i}{\frac{\partial{y}_{t}}{\partial{x}^{n-1}_{i}}\frac{\partial{x}^{n-1}_{i}}{\partial{x}^{n}_{j}}}$
(18)

where the index $j$ corresponds to elements in $x^{n}$ and $i$ corresponds to
elements in $x^{n-1}$. Let $L^{n}_{i}\left(X,W\right)$ be layer operation on
two tensor $X$ and $W$. Relevance propagation follows the generic Deep Taylor
Decomposition [33] as follows:

$\displaystyle R^{n}_{j}=\mathcal{G}\left(X,W,R^{n-1}\right){\textbf{\hskip
86.72377pt}}$ $\displaystyle=\sum_{i}X_{j}\frac{\partial
L^{n}_{j}\left(X,W\right)}{\partial
X_{j}}\frac{R^{n-1}_{i}}{\sum_{j^{\prime}}L^{n}_{j^{\prime}}\left(X,W\right)}$
(19)

The transformer block consist of GELU [16], which outputs both positive and
negative values. To preserve the conservation rule, i.e., sum of relevance is
always same for all the layers, we remove all the elements with negative
relevance. We construct a subset of indices
$p=\\{\left(i,j\right)|x_{j}w_{ji}\geq)\\}$, resulting in the following
propagation:

$\displaystyle
R^{n}_{j}=\mathcal{G}_{p}\left(x,w,p,R^{n-1}\right){\textbf{\hskip
72.26999pt}}$ $\displaystyle=\sum_{\left(i,j\right)\in
p}\frac{x_{j}w_{ji}}{\sum_{\\{j^{\prime}|\left(j^{\prime},i\right)\in
p\\}}x_{j^{\prime}}w_{j^{\prime}i}}R^{n-1}_{i}$ (20)

To initiate the relevance propagation, we set $R^{0}=1_{c}$, where $1_{c}$ is
a one-hot indicating the target class $c$.

####  3.3.2 Layer Importance Score

In LRP, both the relevance and gradients propagate backward from the
classification head through all layers within each block to the input patch
embedding layers. During this process, each layer in the network learns a
relevance score map. Specifically, for image sample $t$, for the attention
layer $A$ in transformer block $L$, with the layer gradients denoted as
$\nabla A^{L}_{t}$ and relevance as $R^{L}_{A,t}$, the relevance score map of
the attention layer $\left(A^{L}_{s,t}\right)$ is defined as:

$\displaystyle{A}^{L}_{s,t}=\mathbb{E}_{h}\left(\nabla A^{L}_{t}\odot
R^{L}_{A,t}\right)^{+}$ (21)

where $\mathbb{E}_{h}$ is the mean across the multi-head, and our analysis
focuses exclusively on the positive value of the gradients-relevance
multiplication. To quantitatively measure the contribution score of the
attention layer of $L^{th}$ toward the output for the sample image, we simply
take the mean of the relevance score map. Since the relevance score map
produced by the LRP method is class-specific, i.e., different maps are
generated for various image samples. The overall contribution score of the
attention layer of the $L^{th}$ block is calculated by taking average over
$T$=50,000 randomly selected images from the ImageNet1k training dataset,
given as:

$\displaystyle{C}^{L}_{A}=\frac{1}{T}\sum_{t=1}^{T}\mathbb{E}\left({A}^{L}_{s,t}\right)$
(22)

The identical method is utilized for all other quantizing layers, including
qkv layers, matrix multiplication layers (matmul1 and matmul2), the projection
layer, and the fully connected layers (fc1 and fc2) to calculate their
contribution score toward output classification. The relative importance score
for a any layer in a any block is the average normalized value of the
contribution score of that layer across all contribution scores. The
expression for the relative layer importance score of attention layer in the
$L^{th}$ block is as follows:

$\displaystyle{I}^{L}_{A}=\frac{C^{L}_{A}}{\sum_{l=1}^{L}\sum_{u\in
U_{l}}C^{l}_{u}}$ (23)

Here, $U_{l}$ represents all the quantized layers in the $L^{th}$ transformer
block. We use the relative importance score $I$ to determine the bit-width
during model quantization, allocating a higher bit count to layers with higher
relative importance scores.

####  3.3.3 Mixed-Precision Bit Allocation

The early blocks of the transformer architecture are typically responsible for
capturing low-level features and details in the input data. These features are
crucial for the network to learn representations effectively. In addition,
early blocks are sensitive to small variations in the input data. If these
blocks are quantized too aggressively, the network might lose its
representation capacity, potentially leading to a decrease in its overall
performance. Therefore, to avoid loss of information early in the processing
stages, we allocate higher precision to all the layers of the first two blocks
during post training quantization. To maintain the model size, we use the
layer importance score to identify the layers that are important and reduce
the bit allocation of those layers.

##  4 Experiments

###  4.1 Experimental Setup

For post training quantization, all the pretrained weights and backbone
architecture are adopted from Timms library. We randomly sample 32 images from
the ImageNet1K training set for image classification and 1 sample from the
COCO dataset for object detection and instance segmentation to calibrate the
quantization parameters. We apply percentile method for the calibration
process. Clipped scale reparameterization is applied to the post-LayerNorm
activations in all the blocks and the scale reparameterization is applied to
all the MHSA modules. For layer importance analysis, all the pretrained
weights are obtained from Timms library and the backbone architecture is
adopted from [6].

###  4.2 Results on ImageNet1K

To demonstrate the superiority of LRP-QVIT in image classification, we
conducted extensive experiments on the ImageNet1K [18] dataset by employing
various vision transformer architectures, including ViT [13], DeiT [39], and
Swin [28]. The comparison of quantization results with PTQ methods is
presented in Table 1. Our fixed-bit clipped Reparameterized Layer method (CRL-
QViT) and our mixed-bit method (LRP-QViT) outperform SOTA methods by
significant margins. Specifically, for 4-bit weight and activation
quantization, our fixed-bit CRL-QViT method achieves approximately (2-3)%
higher accuracy compared to the current state-of-the-art RepQ-ViT [23]. With
the addition of the mixed-precision strategy, LRP-QViT significantly improves
accuracy, averaging up to 4% compared to RepQ-ViT.

For specific models like ViT-S and ViT-B, our mixed-precision LRP-QViT
outperforms RepQ-ViT by 5.76% and 6.89%, respectively. Similarly, for 6-bit
quantization, fixed-bit CRL-QViT and mixed-bit LRP-QViT outperform all other
methods. Our methods can achieve an accuracy comparable to that of the full-
precision baseline. In DeiT-B and Swin-S quantization, mixed-precision LRP-
QViT achieves 81.44% and 82.86% accuracy, respectively, with only a 0.36% and
0.37% accuracy drop over the full precision models. We also observe FQViT
crashing when quantizing the model to 4 bits, while our methods achieve top
performance. Furthermore, our methods do not depend on hyperparameter tuning
or a reconstruction process, as seen in FQViT, PTQ4ViT, and APQ-ViT.

Method | No HP | No REC. | Prec.(W/A) | ViT-S | ViT-B | DeiT-T | DeiT-S | DeiT-B | Swin-S | Swin-B
---|---|---|---|---|---|---|---|---|---|---
Full-Precision | - | - | 32/32 | 81.39 | 84.54 | 72.21 | 79.85 | 81.80 | 83.23 | 85.27
FQ-ViT [26] | ✕ | ✓ | 4/4 | 0.10 | 0.10 | 0.10 | 0.10 | 0.10 | 0.10 | 0.10
PTQ4ViT-ViT [48] | ✕ | ✕ | 4/4 | 42.57 | 30.69 | 36.96 | 34.08 | 64.39 | 76.09 | 74.02
APQ-ViT [10] | ✕ | ✕ | 4/4 | 47.95 | 41.41 | 47.94 | 43.55 | 67.48 | 77.15 | 76.48
RepQ-ViT [23] | ✓ | ✓ | 4/4 | 65.05 | 68.48 | 57.43 | 69.03 | 75.61 | 79.45 | 78.32
CRL-QViT(ours) | ✓ | ✓ | 4/4 | 68.25 | 73.54 | 59.06 | 70.78 | 77.40 | 80.55 | 80.04
LRP-QViT(ours) | ✓ | ✓ | MP4/MP4 | 70.81 | 75.37 | 61.24 | 72.43 | 78.13 | 81.37 | 80.77
FQ-ViT [26] | ✕ | ✓ | 6/6 | 4.26 | 0.10 | 58.66 | 45.51 | 64.63 | 66.50 | 52.09
PSAQ-ViT [22] | ✕ | ✓ | 6/6 | 37.19 | 41.52 | 57.58 | 63.61 | 67.95 | 72.86 | 76.44
Ranking [29] | ✕ | ✕ | 6/6 | - | 75.26 | - | 74.58 | 77.02 | - | -
PTQ4ViT [48] | ✕ | ✕ | 6/6 | 78.63 | 81.65 | 69.68 | 76.28 | 80.25 | 82.38 | 84.01
APQ-ViT [10] | ✕ | ✕ | 6/6 | 79.10 | 82.21 | 70.49 | 77.76 | 80.42 | 82.67 | 84.18
RepQ-ViT [23] | ✓ | ✓ | 6/6 | 80.43 | 83.62 | 70.76 | 78.90 | 81.27 | 82.79 | 84.57
CRL-QViT (ours) | ✓ | ✓ | 6/6 | 80.52 | 83.83 | 70.96 | 79.00 | 81.39 | 82.77 | 84.63
LRP-QViT (ours) | ✓ | ✓ | MP6/MP6 | 80.59 | 83.87 | 71.03 | 79.03 | 81.44 | 82.86 | 84.72
Table 1: Quantization results of image classification on ImageNet1K dataset,
each value presents the Top-1 accuracy (%) obtained by quantizing each model.
CRL-QViT utilizes Clipped Reparameterization for LayerNorm, while LRP-QViT
incorporates both CRL and Layer-wise Relevance Propagation. Here, ”No Hyper-
parameters” is abbreviated as ”No HP”, ”No Reconstruction” as ”No REC”, and
”Prec. (W/A)” indicates the quantization bit-precision for weights and
activations as ”W” and ”A” bits, respectively. ’MP’ represents mixed
precision. Bold data indicate best performance

###  4.3 Results on COCO

To further assess the effectiveness of LRP-QViT, we conduct evaluations on
object detection and instance segmentation tasks using the COCO [25] dataset.
Employing Mask R-CNN and Cascade Mask R-CNN detectors with Swin transformers
as the backbone, we present the results in Table 2. PTQ4ViT exhibits severe
performance degradation, and APQ-ViT demonstrates improved results but still
performs suboptimally with Swin-T. Moreover, both methods require
hyperparameter tuning and a reconstruction process. In comparison to the
state-of-the-art RepQ-ViT, our LRP-QViT method achieves superior performance.
For 4-bit quantization using the Swin backbone in the Mask R-CNN framework,
our mixed-precision LRP-QViT outperforms RepQ-ViT by an average of 2 box AP
and 1.55 mask AP. In the Cascade Mask R-CNN framework, our mixed-precision
LRP-QViT approach exhibits slightly better performance compared to RepQ-ViT.
Similarly, for 6-bit quantization, LRP-QViT shows slightly better performance
compared to RepQViT. When quantizing the Mask R-CNN framework with Swin-T, our
approach achieves 45.6 box AP and 41.3 mask AP, which is only 0.4 box AP and
0.3 mask AP lower than the full-precision baseline. Similarly, comparable
results can be obtained with the Swin-S backbone, achieving 48.1 box AP and
43.3 mask AP, which is just 0.4 box AP and 0.3 mask AP lower compared to full
precision.

Method | No HP | No REC | Prec. (W/A) | Mask R-CNN | Cascade Mask R-CNN
---|---|---|---|---|---
w.Swin-T | w.Swin-S | w.Swin-T | w.Swin-S
APbox |  APmask |  APbox |  APmask |  APbox |  APmask |  APbox |  APmask
Full-Precision | - | - | 32/32 | 46.0 | 41.6 | 48.5 | 43.3 | 50.4 | 43.7 | 51.9 | 45.0
PTQ4ViT [48] | ✕ | ✕ | 4/4 | 6.9 | 7.0 | 26.7 | 26.6 | 14.7 | 13.5 | 0.5 | 0.5
APQ-ViT [10] | ✕ | ✕ | 4/4 | 23.7 | 22.6 | 44.7 | 40.1 | 27.2 | 24.4 | 47.7 | 41.1
RepQ-ViT [23] | ✓ | ✓ | 4/4 | 36.1 | 36.0 | 44.2 | 40.2 | 47.0 | 41.4 | 49.3 | 43.1
CRL-QViT (ours) | ✓ | ✓ | 4/4 | 37.2 | 37.4 | 45.8 | 40.9 | 47.4 | 41.5 | 49.5 | 43.4
LRP-QViT (ours) | ✓ | ✓ | MP4/MP4 | 37.9 | 38.2 | 46.4 | 41.4 | 47.7 | 41.6 | 50.1 | 43.6
PTQ4ViT [48] | ✕ | ✕ | 6/6 | 5.8 | 6.8 | 6.5 | 6.6 | 14.7 | 13.6 | 12.5 | 10.8
APQ-ViT [10] | ✕ | ✕ | 6/6 | 45.4 | 41.2 | 47.9 | 42.9 | 48.6 | 42.5 | 50.5 | 43.9
RepQ-ViT [23] | ✓ | ✓ | 6/6 | 45.1 | 41.2 | 47.8 | 43.0 | 50.0 | 43.5 | 51.4 | 44.6
CRL-QViT (ours) | ✓ | ✓ | 6/6 | 45.3 | 41.2 | 47.9 | 42.9 | 50.0 | 43.5 | 51.5 | 44.6
LRP-QViT (ours) | ✓ | ✓ | MP6/MP6 | 45.6 | 41.3 | 48.1 | 43.0 | 49.9 | 43.5 | 51.4 | 44.6

Table 2: Quantization results on object detection and instance segmentation on
COCO dataset. Here, $AP^{box}$ is the box average precision for object
detection, and $AP^{mask}$ is the mask average precision for instance
segmentation. CRL-QViT utilizes Clipped Reparameterization for LayerNorm,
while LRP-QViT incorporates both CRL and Layer-wise Relevance Propagation. ”No
Hyper-parameters” is abbreviated as ”No HP”, ”No Reconstruction” as ”No REC”,
and ”Prec. (W/A)” indicates the quantization bit-precision for weights and
activations as ”W” and ”A” bits, respectively. ’MP’ represents mixed
precision. Bold data indicate best performance.

Model | Method | Precision(W/A) | Top-1%
---|---|---|---
| Full-Precision | 32/32 | 79.85
|  Layer-wise Quant [23] | 4/4 | 33.17
|  CW Quant [23] | 4/4 | 70.28
DeiT-S |  Scale Reparm [23] | 4/4 | 69.03
| Clipped CW Quant (ours) | 4/4 | 70.78
| Clipped CW Quant (ours) | MP4/MP4 | 72.43
| Full-Precision | 32/32 | 79.85
|  Layer-wise Quant [23] | 4/4 | 57.63
|  CW Quant [23] | 4/4 | 80.52
Swin-S |  Scale Reparm [23] | 4/4 | 79.45
| Clipped CW Quant (ours) | 4/4 | 80.55
| Clipped CW Quant (ours) | MP4/MP4 | 81.37

Table 3: Ablation studies of different quantizers for post-LayerNorm activation. Here, CW denotes channel-wise quantization and W/A represents weights and activation bit allocation, and MP denotes mixed precision bit allocation. Model | Prec. (W/A) | Top-1% |
---|---|---|---
| 32/32 | 79.85 |
| 4/4 | 70.78 |
DeiT-S |  $B_{1}$: 5/5, $B_{2-12}$: 4/4 | 74.35 |
|  $B_{1-2}$: 5/5, $B_{3-12}$: 4/4 | 75.66 |
|  $B_{1}$: 5/5, $B_{2-12}$: LRP | 72.43 |
|  $B_{1-2}$: 5/5, $B_{3-12}$: LRP | 72.82 |
| 32/32 | 83.23 |
| 4/4 | 80.55 |
|  $B_{1}$: 5/5, $B_{2-12}$: 4/4 | 82.03 |
Swin-S |  $B_{1-2}$: 5/5, $B_{3-12}$: 4/4 | 82.37 |
|  $B_{1}$: 5/5, $B_{2-12}$: LRP | 81.13 |
|  $B_{1-2}$: 5/5, $B_{3-12}$: LRP | 81.37 |
Table 4: Ablation study on mixed-precision bit allocation scheme. Here $B_{1-2}$ represents the Transformer blocks 1 and 2. Model | Method | Top-1 | Calib Data | Min.
---|---|---|---|---
| Full-Precision | 79.85 | - | -
|  FQ-ViT [26] | 0.10 | 1000 | 0.5
|  PTQ4ViT [48] | 34.08 | 32 | 3.2
DeiT-S |  RepQ-ViT [23] | 69.03 | 32 | 1.3
| CRL-QViT(ours) | 70.78 | 32 | 1.4
| LRP-QViT(ours) | 72.43 | 32 | 1.4
| Full-Precision | 79.85 | - | -
|  FQ-ViT [26] | 0.10 | 1000 | 1.1
|  PTQ4ViT [48] | 76.09 | 32 | 7.7
Swin-S |  RepQ-ViT [23] | 79.45 | 32 | 2.9
| CRL-QViT(ours) | 80.55 | 32 | 3.0
| LRP-QViT(ours) | 81.37 | 32 | 3.0
Table 5: Comparison of the data quantity and time consumption (in minutes)
during the quantization calibration.

###  4.4 Ablation Study

To verify the effectiveness of our proposed framework, we conducted two
ablation studies: clipped channel-wise quantization for post-LayerNorm,
presented in Table 3, and mixed-precision bit allocation for post-training
quantization, detailed in Table 4.

Table 3 presents the effects in accuracy of DeiT-S and Swin-S models with
various quantizer schemes for post-LayerNorm activations. For both models,
direct layer-wise quantization resulted in severe performance degradation,
achieving 33.17% and 57.63%, due to its inability to represent data
distribution adequately. Applying channel-wise quantization resolved these
issues, resulting in accuracy improvements to 70.28% and 80.52%. The scale
reparameterization method, using channel-wise quantization during the
quantization process and converting the learned parameters to layer-wise
quantization for efficient inference, came with a drop in accuracy of about
1.25% and 1.07%. The clipped channel-wise method, maintained and improved the
accuracy of the model by removing outliers from the post-LayerNorm parameters.
This approach improved performance by 0.5% in DeiT-S with respect to channel-
wise quantization. Furthermore, adding the mixed-precision strategy further
enhanced the performance of both networks by 1.7% and 0.8%.

Table 4 showcases the outcomes of the classification task under various bit
configurations for the quantization process. When applying 4-bit quantization
to DeiT-S, enhancing the bit allocation of all layers in Block 1 ($B_{1}$) to
5 bits while maintaining other layers in Block 2 to 12 ($B_{2-12}$) at 4 bits
raises performance from 70.78% to 74.35%. Similarly, allocating 5 bits to all
layers in Blocks 1 and 2 ($B_{1-2}$), while keeping other blocks at 4-bits,
further improves performance from 70.78% to 75.66%.

The crucial importance of the first two layers in the transformer block is
evident, as their output influences all subsequent layers. Therefore, in this
study, we assign higher bit precision to these layers, necessitating a
reduction in bit allocation from other blocks to maintain the model size at 4
bits average quantization. We achieve this by reducing the bits of the same
layer but from another block, guided by the layer importance scores. Two mixed
precision cases are analyzed. First, we allocate 5 bits for all layers in
Block 1 and provide 3 bits to one layer from Blocks 2 to 12, resulting in a
classification accuracy of 72.43%. Secondly, for further enhancement, we
assign 5 bits to the first two blocks and reduce bit allocation from 2 layers
from Blocks 3 to 12, resulting in 72.82% accuracy. Our observations suggest
that simply providing higher bits to the first two blocks of the transformer
significantly improves performance, and the LRP method effectively identifies
unimportant layers, allowing us to reduce bits and maintain the model size.
The same approach is applied in the Swin-S ablation study. All results
mentioned in Table 1 follow the second case for mixed precision, where the
first two layers are given high bit precision, and the LRP method is employed
to trim down the bit allocation for other less important layers

###  4.5 Efficiency Analysis

We assess the effectiveness of various methods in terms of data requirements
and time consumption for quantization calibration, as detailed in Table 5. The
calibration time is measured using a single 3090 GPU. FQViT, being
reconstruction-free, exhibits rapid performance. However, even with 1000
calibration images, its 4-bit quantization performance is only 0.10%. On the
other hand, PTQ4ViT involves a reconstruction process, resulting in
significantly higher calibration time requirements. Our LRP-QViT, requiring
only 32 samples (same as RepQ-ViT), achieves a comparable calibration time,
yet surpasses the performance by a substantial margin.

##  5 Conclusions

In this paper, we present a novel approach, LRP-QViT, for post-training mixed-
bit quantization of vision transformers. LRP-QViT learns the layer importance
score by back-propagating relevance and gradient, and based on the importance
score, optimal bits are selected for mixed-precision quantization.
Additionally, we introduce clipped channel-wise quantization for post-
LayerNorm activations, which removes outliers and prevents inter-channel
variation, thereby improving the model’s performance. Comprehensive
experiments demonstrate that our LRP-QViT outperforms existing methods in low-
bit post-training quantization.

## References

  * Abnar and Zuidema [2020] Samira Abnar and Willem Zuidema.  Quantifying attention flow in transformers.  _arXiv preprint arXiv:2005.00928_ , 2020.
  * Binder et al. [2016] Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, Klaus-Robert Müller, and Wojciech Samek.  Layer-wise relevance propagation for neural networks with local renormalization layers.  In _Artificial Neural Networks and Machine Learning–ICANN 2016: 25th International Conference on Artificial Neural Networks, Barcelona, Spain, September 6-9, 2016, Proceedings, Part II 25_ , pages 63–71. Springer, 2016.
  * Carion et al. [2020a] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In _European conference on computer vision_ , pages 213–229. Springer, 2020a.
  * Carion et al. [2020b] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16_ , pages 213–229. Springer, 2020b.
  * Chefer et al. [2021a] Hila Chefer, Shir Gur, and Lior Wolf.  Transformer interpretability beyond attention visualization.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 782–791, 2021a.
  * Chefer et al. [2021b] Hila Chefer, Shir Gur, and Lior Wolf.  Transformer interpretability beyond attention visualization.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 782–791, 2021b.
  * Chen et al. [2022] Xianing Chen, Qiong Cao, Yujie Zhong, Jing Zhang, Shenghua Gao, and Dacheng Tao.  Dearkd: data-efficient early knowledge distillation for vision transformers.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12052–12062, 2022.
  * Dai et al. [2021] Zhigang Dai, Bolun Cai, Yugeng Lin, and Junying Chen.  Up-detr: Unsupervised pre-training for object detection with transformers.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 1601–1610, 2021.
  * Denil et al. [2013] Misha Denil, Babak Shakibi, Laurent Dinh, Marc’Aurelio Ranzato, and Nando De Freitas.  Predicting parameters in deep learning.  _Advances in neural information processing systems_ , 26, 2013.
  * Ding et al. [2022] Yifu Ding, Haotong Qin, Qinghua Yan, Zhenhua Chai, Junjie Liu, Xiaolin Wei, and Xianglong Liu.  Towards accurate post-training quantization for vision transformer.  In _Proceedings of the 30th ACM International Conference on Multimedia_ , pages 5380–5388, 2022.
  * Dong et al. [2019] Zhen Dong, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer.  Hawq: Hessian aware quantization of neural networks with mixed-precision.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 293–302, 2019.
  * Dong et al. [2020] Zhen Dong, Zhewei Yao, Daiyaan Arfeen, Amir Gholami, Michael W Mahoney, and Kurt Keutzer.  Hawq-v2: Hessian aware trace-weighted quantization of neural networks.  _Advances in neural information processing systems_ , 33:18518–18529, 2020.
  * Dosovitskiy et al. [2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _International Conference on Learning Representations_ , 2021.
  * Gu et al. [2019] Jindong Gu, Yinchong Yang, and Volker Tresp.  Understanding individual decisions of cnns via contrastive backpropagation.  In _Computer Vision–ACCV 2018: 14th Asian Conference on Computer Vision, Perth, Australia, December 2–6, 2018, Revised Selected Papers, Part III 14_ , pages 119–134. Springer, 2019.
  * He et al. [2018] Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang.  Soft filter pruning for accelerating deep convolutional neural networks.  In _Proceedings of the 27th International Joint Conference on Artificial Intelligence_ , pages 2234–2240, 2018.
  * Hendrycks and Gimpel [2016] Dan Hendrycks and Kevin Gimpel.  Gaussian error linear units (gelus).  _arXiv preprint arXiv:1606.08415_ , 2016.
  * Jain et al. [2023] Jitesh Jain, Jiachen Li, Mang Tik Chiu, Ali Hassani, Nikita Orlov, and Humphrey Shi.  Oneformer: One transformer to rule universal image segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2989–2998, 2023.
  * Krizhevsky et al. [2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  _Advances in neural information processing systems_ , 25, 2012.
  * Li et al. [2022a] Yanjing Li, Sheng Xu, Baochang Zhang, Xianbin Cao, Peng Gao, and Guodong Guo.  Q-vit: Accurate and fully quantized low-bit vision transformer.  _Advances in Neural Information Processing Systems_ , 35:34451–34463, 2022a.
  * Li et al. [2022b] Yanyu Li, Geng Yuan, Yang Wen, Ju Hu, Georgios Evangelidis, Sergey Tulyakov, Yanzhi Wang, and Jian Ren.  Efficientformer: Vision transformers at mobilenet speed.  _Advances in Neural Information Processing Systems_ , 35:12934–12949, 2022b.
  * Li and Gu [2023] Zhikai Li and Qingyi Gu.  I-vit: Integer-only quantization for efficient vision transformer inference.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 17065–17075, 2023.
  * Li et al. [2022c] Zhikai Li, Liping Ma, Mengjuan Chen, Junrui Xiao, and Qingyi Gu.  Patch similarity aware data-free quantization for vision transformers.  In _European Conference on Computer Vision_ , pages 154–170. Springer, 2022c.
  * Li et al. [2023] Zhikai Li, Junrui Xiao, Lianwei Yang, and Qingyi Gu.  Repq-vit: Scale reparameterization for post-training quantization of vision transformers.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 17227–17236, 2023.
  * Lin et al. [2022a] Sihao Lin, Hongwei Xie, Bing Wang, Kaicheng Yu, Xiaojun Chang, Xiaodan Liang, and Gang Wang.  Knowledge distillation via the target-aware transformer.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10915–10924, 2022a.
  * Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_ , pages 740–755. Springer, 2014.
  * Lin et al. [2022b] Yang Lin, Tianyu Zhang, Peiqin Sun, Zheng Li, and Shuchang Zhou.  Fq-vit: Post-training quantization for fully quantized vision transformer.  In _Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22_ , pages 1173–1179, 2022b.
  * Liu et al. [2023] Yijiang Liu, Huanrui Yang, Zhen Dong, Kurt Keutzer, Li Du, and Shanghang Zhang.  Noisyquant: Noisy bias-enhanced post-training activation quantization for vision transformers.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 20321–20330, 2023.
  * Liu et al. [2021a] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 10012–10022, 2021a.
  * Liu et al. [2021b] Zhenhua Liu, Yunhe Wang, Kai Han, Wei Zhang, Siwei Ma, and Wen Gao.  Post-training quantization for vision transformer.  _Advances in Neural Information Processing Systems_ , 34:28092–28103, 2021b.
  * Liu et al. [2021c] Zhenhua Liu, Yunhe Wang, Kai Han, Wei Zhang, Siwei Ma, and Wen Gao.  Post-training quantization for vision transformer.  _Advances in Neural Information Processing Systems_ , 34:28092–28103, 2021c.
  * Lou et al. [2020] Qian Lou, Feng Guo, Minje Kim, Lantao Liu, and Lei Jiang.  Autoq: Automated kernel-wise neural network quantization.  In _International Conference on Learning Representations_ , 2020.
  * Mao et al. [2022] Yanghao Li Hanzi Mao, Ross Girshick, and Kaiming He.  Exploring plain vision transformer backbones for object detection.  ECCV, 2022.
  * Montavon et al. [2017] Grégoire Montavon, Sebastian Lapuschkin, Alexander Binder, Wojciech Samek, and Klaus-Robert Müller.  Explaining nonlinear classification decisions with deep taylor decomposition.  _Pattern recognition_ , 65:211–222, 2017.
  * Rao et al. [2021] Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh.  Dynamicvit: Efficient vision transformers with dynamic token sparsification.  _Advances in neural information processing systems_ , 34:13937–13949, 2021.
  * Shen et al. [2020] Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer.  Q-bert: Hessian based ultra low precision quantization of bert.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 8815–8821, 2020.
  * Smilkov et al. [2017] Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg.  Smoothgrad: removing noise by adding noise.  _arXiv preprint arXiv:1706.03825_ , 2017.
  * Srinivas and Fleuret [2019] Suraj Srinivas and François Fleuret.  Full-gradient representation for neural network visualization.  _Advances in neural information processing systems_ , 32, 2019.
  * Strudel et al. [2021] Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid.  Segmenter: Transformer for semantic segmentation.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 7262–7272, 2021.
  * Touvron et al. [2021] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  In _International conference on machine learning_ , pages 10347–10357. PMLR, 2021.
  * Voita et al. [2019] Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov.  Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5797–5808, 2019.
  * Wang et al. [2019] Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han.  Haq: Hardware-aware automated quantization with mixed precision.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 8612–8620, 2019.
  * Wang et al. [2021] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao.  Pyramid vision transformer: A versatile backbone for dense prediction without convolutions.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 568–578, 2021.
  * Wu et al. [2022] Kan Wu, Jinnian Zhang, Houwen Peng, Mengchen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan.  Tinyvit: Fast pretraining distillation for small vision transformers.  In _European Conference on Computer Vision_ , pages 68–85. Springer, 2022.
  * Xiao et al. [2023] Junrui Xiao, Zhikai Li, Lianwei Yang, and Qingyi Gu.  Patch-wise mixed-precision quantization of vision transformer.  _arXiv preprint arXiv:2305.06559_ , 2023.
  * Xu et al. [2022a] Yufei Xu, Jing Zhang, Qiming Zhang, and Dacheng Tao.  Vitpose: Simple vision transformer baselines for human pose estimation.  _Advances in Neural Information Processing Systems_ , 35:38571–38584, 2022a.
  * Xu et al. [2022b] Yifan Xu, Zhijie Zhang, Mengdan Zhang, Kekai Sheng, Ke Li, Weiming Dong, Liqing Zhang, Changsheng Xu, and Xing Sun.  Evo-vit: Slow-fast token evolution for dynamic vision transformer.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 2964–2972, 2022b.
  * Yu et al. [2022] Fang Yu, Kun Huang, Meng Wang, Yuan Cheng, Wei Chu, and Li Cui.  Width & depth pruning for vision transformers.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 3143–3151, 2022.
  * Yuan et al. [2022] Zhihang Yuan, Chenhao Xue, Yiqi Chen, Qiang Wu, and Guangyu Sun.  Ptq4vit: Post-training quantization for vision transformers with twin uniform quantization.  In _European Conference on Computer Vision_ , pages 191–207. Springer, 2022.
  * Zhang et al. [2022] Jinnian Zhang, Houwen Peng, Kan Wu, Mengchen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan.  Minivit: Compressing vision transformers with weight multiplexing.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12145–12154, 2022.
  * Zheng et al. [2021] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al.  Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 6881–6890, 2021.