# Towards Robust Velocity and Position Estimation of Opponents for Autonomous
Racing Using Low-Power Radar

Andrea Ronco, Nicolas Baumann, Marco Giordano, Michele Magno  Center for
Project Based Learning - D-ITET
ETH Zurich
Zurich, Switzerland
<EMAIL_ADDRESS>
###### Abstract

This paper presents the design and development of an intelligent subsystem
that includes a novel low-power radar sensor integrated into an autonomous
racing perception pipeline to robustly estimate the position and velocity of
dynamic obstacles. The proposed system, based on the Infineon BGT60TR13D
radar, is evaluated in a real-world scenario with scaled race cars. The paper
explores the benefits and limitations of using such a sensor subsystem and
draws conclusions based on field-collected data. The results demonstrate a
tracking error up to 0.21 $\boldsymbol{\pm}$ 0.29 $\mathrm{m}$ in distance
estimation and 0.39 $\boldsymbol{\pm}$ 0.19 $\mathrm{m}\mathrm{/}\mathrm{s}$
in velocity estimation, despite the power consumption in the range of 10s of
milliwatts. The presented system provides complementary information to other
sensors such as LiDAR and camera, and can be used in a wide range of
applications beyond autonomous racing.

###### Index Terms:

sensors, embedded systems, radar, autonomous driving

##  I Introduction

The field of autonomous racing has gained significant attention in recent
years, with the development of self-driving cars and the increasing popularity
of motorsports, which opens the opportunity to enable knowledge transfer from
academia to industry [1, 2, 3, 4]. A fundamental component of autonomous
systems is perception, namely the part of the system that enables the vehicle
to observe and acquire information on the surrounding environment, which is
necessary to generate appropriate responses [5, 6, 7, 8].

LiDARs, cameras, and radars are the main exteroceptive sensors used for
perception purposes on autonomous vehicles [9]. Cameras use visible light to
capture images and record video. They can provide high-resolution images and
color information, but they rely on good lighting conditions and can be
affected by shadows and reflections.

LiDARs use infrared lasers to create three-dimensional point clouds of the
surrounding environment, measuring the time of flight of the laser beam from
the sensor to the reflecting objects. Due to the rotating parts, LiDARs have a
fairly high cost and large size. They only provide ranging information and
they are severely affected by environmental conditions (rain, snowflakes, and
fog), as well as by the reflectivity properties of the targets. For example,
opaque black objects are often difficult to detect with this technology [10].
Radars use radio waves with various frequencies and modulations to detect
objects in their field of view. Thanks to the larger radio wavelengths, radars
are less affected by adverse weather conditions, and they can measure reliably
through raindrops, snowflakes, and dust [11, 12]. Radars on the other hand are
very robust against the aforementioned object reflectivity and further robust
to adverse weather conditions, such as rain and snowflakes [13]. Depending on
the technology they can also provide valuable relative velocity information
and spatial position of objects, which is highly beneficial in racing contexts
[2, 14]. Each one of these technologies has its own strengths and weaknesses,
and they are often used in combination to provide a more comprehensive view of
the environment.

Emerging novel low-power radars are capable of high-resolution measurements
with a peak power consumption in the order of 100s of milliwatts, and an
average consumption down to less than $10\text{\,}\mathrm{mW}$ [15]. Moreover,
their low cost and reduced form factor, also thanks to the Antenna in Package
(AiP), make them an attractive option for achieving robust velocity and
position estimation of opponents in the context of autonomous driving [16] and
racing, especially on small-scale vehicles. LiDAR and radar sensor fusion is a
popular technique in autonomous driving, robotics, and other applications
where accurate and reliable sensing is critical [13, 12, 11]. By combining the
strengths of both technologies, it is possible to create a more comprehensive
view of the environment, allowing for safer, more efficient, and more robust
operations [12, 11].

Autonomous racing is gaining popularity in the research community as an
application for novel perception and control algorithms [2, 7, 6, 17].
Especially autonomous driving and racing on small-scale vehicles have been
used to develop and test algorithms for autonomous vehicles in a safe and
efficient way [18, 6]. One example of an autonomous vehicle project is the
F1TENTH project [6, 19], which is a global engineering competition for
university students that challenges teams to design, build, and race a single-
seat racing car. The F1TENTH vehicle consists of various sensor modalities
such as LiDAR, camera, radar, and optical flow to achieve a high level of
autonomy in racing scenarios. The perception system of the F1TENTH vehicle is
crucial for achieving a good racing performance [2] by accurately detecting
opponents, estimating their positions and velocities, and avoiding collisions.

In this context, the use of radar solutions for opponent detection and
velocity estimation can be a valuable addition to the existing sensor suite of
the F1TENTH vehicle. The adoption of low-power, small radar options enables
the integration of the system in the vehicle without affecting the overall
weight and aerodynamics and paves the road for a wider adoption on small-scale
vehicles by assessing its effectiveness in a real racing scenario.

This paper presents a preliminary evaluation of a small-size, low-power radar
sensor that operates in the $60\text{\,}\mathrm{GHz}$ frequency band to asses
its performance in autonomous vehicles and perception applications. While more
established radar systems already exist for these tasks, we argue that the
advantages of this small-scale, integrated solution could enable new classes
of vehicles to take advantage of radar technology. We evaluate the sensor for
range and velocity estimation with different targets and set up the grounds
for future work involving sensor fusion of LiDAR and radar data on our racing
platform.

The rest of the paper is organized as follows: In Section II we present
existing work on the topic of autonomous driving and radar technology for
perception and autonomous vehicles, and we summarize the contributions of the
paper. In Section III we introduce a taxonomy of the most common radar
technologies and provide the required theoretical background on radars. The
evaluation setup is described in Section IV, while the results of our analysis
are reported in Section V. We draw conclusions and present future work in
Section VI.

##  II Related Work

Radars are commonly used in autonomous driving applications to provide
information about the surrounding environment, including the position and
velocity of other vehicles on the road. However, accurately estimating the
position and velocity of opponents in autonomous racing can be challenging due
to factors such as noisy data, complex dynamics of the opponents, and high
speeds.

Radars showed promising results when deployed on cars, both for driving
assistance devices and autonomous driving. Previous work showed how radar,
especially when fused with other kinds of sensors, can reliably detect road
boundaries and other vehicles [20], especially with adverse weather conditions
[21]. Previous work already showed how mmWave radars can be used to estimate
the velocity and position of other vehicles in autonomous driving
scenarios[22]. However, the radars used were automotive-grade radars, which
are orders of magnitude more expensive and power-hungry than the novel mmWave
radars used in our work.

Radars have also been used in autonomous racing, as reported in [23, 24],
where the authors argue their use in the Indy Autonomous Challenge (IAC), a
competition of fully autonomous race cars at the Indianapolis Motor Speedway,
promoting innovation and technological advancements in autonomous vehicle
technology [7].

On the notes of the IAC, the F1TENTH association is a student competition of
1:10 scaled race cars, with the similar goal of competing on miniature
racetracks for time-trials and head-to-head races. Due to the compact
dimension of the car, it is not possible to use automotive radars on this
platform, and therefore opponent estimation has traditionally been carried out
through LiDAR and/or camera-based sensing modalities.

This paper wants to investigate novel low-power radars in a tiny form factor
in autonomous vehicles’ perception, in particular regarding other vehicles’
detection, ranging, and exteroceptive velocity estimates.

The contributions of this paper are as follows:

  * •

A novel low-power radar sensor is introduced and integrated into an autonomous
racing perception pipeline.

  * •

The sensor is evaluated in a real-world scenario with scaled race cars.

  * •

Benefits and limitations of such a sensor are explored and conclusions are
drawn on the base of field-collected data.

##  III Radar Background

Radar systems emit an electromagnetic wave signal (known as the illumination
signal), which eventually hits and is reflected by the target. The reflected
(echo) signal contains information about the target, that can be extracted.
The properties of the illumination signal can differ significantly in
frequency, modulation, and other characteristics, defining different radar
technologies, each suitable for different applications.

  * •

Pulse radars transmit high-frequency signals in small bursts and exploit the
propagation delay and the antenna placement to extract information about the
position of the target.

  * •

Continuous Wave (CW) radars illuminate the target with continuous power and
exploit the Doppler effect to estimate its velocity.

  * •

Frequency Modulated Continuous Wave (FMCW) radars add frequency modulation to
the illumination signal. This property allows estimating simultaneously the
velocity and the distance of the targets.

The capability of estimating both the distance and the velocity of the target
makes the FMCW type a common choice for automotive and industrial
applications. A simplified block diagram of a typical FMCW radar system is
shown in Fig. 1. The modulator is responsible for generating the correct
waveform for the illumination signal. The signal is amplified to the desired
power level and transmitted by one or more antennas. At the same time, the
echo signal is picked up by the receiving antenna, amplified, and mixed with
the transmitted signal. The mixing process generates a signal with a new
frequency and phase equal to the difference between the frequency and phase of
the two input signals. For example, given two inputs signals

$\displaystyle y_{1}(t)$ $\displaystyle=A\exp(2\pi jtf_{1}+\phi_{1})$
$\displaystyle y_{2}(t)$ $\displaystyle=A\exp(2\pi jtf_{2}+\phi_{2})$ (1)

the output of the mixer will be

$x(t)=A\exp(2\pi jt(f_{2}-f_{1})+\phi_{2}-\phi_{1})$ (2)

This signal is called Intermediate Frequency (IF) signal or beat signal. It is
typically much lower in frequency with respect to the transmitted signal and
can be sampled with a traditional Analog to Digital Converter (ADC).

Figure 1: Simplified block diagram of a typical FMCW radar system. Figure 2:
Transmitted and Received signals in FMCW radars. Typically
$T_{c}~{}\ll~{}T_{s}$.

A common modulation for FMCW radars is the linear one, where the frequency is
linearly increased with time. This signal is also often referred to as chirp
and can be expressed as

$s(t)=A_{tx}\exp{(2\pi jt(f_{low}+St))}$ (3)

where $A_{t}x$ is the signal amplitude, $f_{low}$ is the starting frequency
and $S$ is the chirp slope, equivalent to $\frac{f_{high}-f_{low}}{T_{c}}$
with $T_{c}$ being the chirp duration. $f_{high}-f_{low}$ is also defined as
modulation bandwidth $B$.

###  III-A Distance Measurement

Given a reflective object at distance $d$, the echo signal is received with a
round-trip time delay $\tau=2d/c$ where $c$ is the speed of light. This effect
is depicted in Fig. 2. This time delay is taken into account in the mixer,
which will generate an IF signal equal to

$\displaystyle y(t)$ $\displaystyle=A\exp{(2\pi jt(S\tau)+\phi)}$ with
$\displaystyle\tau$ $\displaystyle=\frac{2d}{c}$ (4)

Equation 4 shows that the distance of the targets is proportional to the
frequency of the IF signal, whose spectrum shows peaks corresponding to the
target range. The spectrum can be evaluated with a Fast Fourier Transform
(FFT), which is often called Range FFT.

From Fourier transform theory, we know that in a window of duration $T_{c}$ we
can only resolve frequencies larger than $\frac{1}{T_{c}}$. From the round-
trip time, we know that the minimum frequency $f_{min}=2d_{min}/S$. From these
to equations we can derive that the distance resolution $d_{min}$ only depends
on the bandwidth $B$, as shown in Eq. 5.

$\displaystyle f_{min}$ $\displaystyle>\frac{1}{T_{c}}$ $\displaystyle
f_{min}$ $\displaystyle=\frac{2d_{min}}{c}S$ $\displaystyle d_{min}$
$\displaystyle>\frac{c}{2ST_{c}}=\frac{c}{2B}$ (5)

###  III-B Velocity

The radial velocity of the target is estimated by observing the phase of two
consecutive chirps with a small time spacing $T_{s}$. Given a sufficiently
small $T_{s}$, the distance of the target in the range FFT will be unchanged
across the chirps. However, the phase difference depends on the variation of
the round trip time $\Delta\tau$

$\displaystyle\Delta\phi_{IF}$ $\displaystyle=2\pi f_{low}\Delta\tau$
$\displaystyle\Delta\phi_{IF}$ $\displaystyle=\frac{4\pi\Delta d}{\lambda}$
(6)

From Eq. 6 we can derive the angular velocity caused by the moving target,
which is equal to the phase difference across the two chirps.

$\displaystyle\omega:=\Delta\phi$ $\displaystyle=\frac{4\pi vT_{s}}{\lambda}$
$\displaystyle v$ $\displaystyle=\frac{\lambda\Delta\phi}{4\pi T_{s}}$ (7)

The velocity of the target can be resolved without ambiguity when
$\Delta\phi<\pi$, which sets the limit for the maximum velocity as
$v<\frac{\lambda}{4T_{s}}$

In the case of multiple targets with different speeds at the same range, the
velocity for both targets can be estimated by increasing the number of equi-
spaced chirps $N$. A sequence of $N$ chirps is often referred to as a radar
frame. The velocities can be derived with a complex FFT, called Doppler FFT,
phasors corresponding to each range.

From the properties of the FFT, two frequencies $\omega_{1}$ and $\omega_{2}$
can be separated if $|\omega_{1}-\omega_{2}|>2\pi/N$. Taking Eq. 7 in
consideration we can calculate the velocity resolution $v_{min}$

$\displaystyle\Delta\omega$ $\displaystyle=\frac{4\pi
vT_{s}}{\lambda}>\frac{2\pi}{N}$ $\displaystyle
v_{min}:=v>\frac{\lambda}{2T_{s}N}$ (8)

###  III-C Angle of Arrival

Radar systems can also estimate the Angle of Arrival (AoA) of the signal
thanks to an array of receiving antennas. Large antenna arrays (with many
receivers) can differentiate multiple targets with equal speed and distance,
by evaluating a third FFT on the antenna dimension. Since the radar evaluated
in this paper has a small antenna array (of only 2 receivers on azimuth and
elevation), we omit the details of AoA evaluation. However, it is still worth
mentioning that the angular resolution $\Theta_{min}$ increases with the
length of the array, following the relation

$\displaystyle\Theta_{min}$ $\displaystyle=\frac{2}{N}$ (9)

where $N$ is the number of antennas and d is the antenna spacing, and the
antenna spacing is optimal with $d_{spacing}=\lambda/2$.

##  IV Evaluation Setup

This paper evaluates the capability of a novel low-power radar sensor from
Infineon Technologies for autonomous racing on our F1TENTH platform. In
particular, we focus on the evaluation of the capabilities for estimating the
velocity and position of the opponents placed in front of the car, with the
aim of improving the overtaking maneuvers.

The selected sensor device operates in the $60\text{\,}\mathrm{GHz}$ band and
embeds one transmitting and three receiving antennas directly in the package
(AiP), which simplifies the integration in existing systems by removing the
need for high-frequency antenna design expertise. The 60GHz frequency band and
the associated bandwidth provide a range resolution of about
$3\text{\,}\mathrm{cm}$, which is suitable for our racing scenario. The peak
transmission power is $5\text{\,}\mathrm{dBm}$, which results in a limited
maximum range below $10\text{\,}\mathrm{m}$. However, the reduced form factor
and the low-power nature of the device make it suitable for small-scale,
battery-operated applications, such as nano drones and small vehicle models,
which could exploit the same subsystem to detect other moving or static
objects.

To have an accurate evaluation we concluded two different experiments with a
similar setup, which we describe shortly. Two vehicles were used for the
evaluation. The radar subsystem was attached to the front of the first car,
facing towards the front of the vehicle, and connected to the Intel NUC via
USB. A custom Robot Operating System (ROS) driver was developed to interface
the USB device, configure it, and retrieve the data. The second vehicle was
placed in front, in the field of view of the radar, and acted as a target for
the radar, as seen in Fig. 3.

Figure 3: Initial state of the evaluation setup with the initial distance d
marked in red.

###  IV-A Experimental evaluation

This first experiment was designed to study the accuracy of the radar in
tracking the distance and the velocity of an opponent car in the scenario with
the lowest interference. For this setup, the car equipped with the radar did
not drive. The second car was placed in front of the first one and manually
controlled to drive away at different speeds. The radar data was logged
alongside the odometry data of the second car, in order to have a partial
ground truth of the velocity readings. It must be noted that the odometry
estimation for velocity is less accurate since it depends on indirect
measurements.

In the second experiment, both cars were driving at approximately the same
speed, both manually controlled. This experiment served the purpose of
observing the effect of the ego-velocity of the car on the radar data, which
will be discussed in Section V.

##  V Experimental Results

For this evaluation, the signal processing on the radar signal was kept as
simple as possible, only using standard FFT processing. This evaluation
allowed us to properly identify the benefits that the additional radar data
could bring, as well as the challenges of extracting the important information
depending on the context.

The radar is configured to produce radar frames at $20\text{\,}\mathrm{Hz}$.
Each frame is composed by $64$ chirps, and each chirp is sampled $64$ times at
$2\text{\,}\mathrm{MHz}$. The chirp timing is set in order to allow a maximum
range of $3.7\text{\,}\mathrm{m}$, a maximum velocity of
$8\text{\,}\mathrm{m}\text{\,}{\mathrm{s}}^{-1}$, and a velocity resolution of
$0.25\text{\,}\mathrm{m}\text{\,}{\mathrm{s}}^{-1}$. The current hardware is
limited to a bandwidth of about
$5\text{\,}\mathrm{Mbit}\text{\,}{\mathrm{s}}^{-1}$ for radar data, which sets
an upper bound on the resolution. DC removal and a Hanning window are used to
improve the quality of the resulting map and reduce noise. The data is zero-
padded to $256$ samples before evaluating the spectra, in order to increase
the resolution. A sample range-doppler map is shown in Fig. 4.

Figure 4: Sample range-doppler map from the static experiments. The moving
target can be seen as a high-energy point.

###  V-A Static Measurements

The static measurements are evaluated to characterize the sensor in the best
case. They show that distance and velocity can be easily extracted from the
range-doppler maps by tracking the max-energy point. This result shows that,
despite the low power output, the radar is capable of detecting moving targets
in its range reliably in optimal conditions. Fig. 5 shows a quantitative view
of the range and velocity estimations from the range-doppler map with respect
to the values estimated by the opponent car. The grey area marks the time when
the opponent car was outside the maximum range of the radar, and therefore not
visible.

We estimated the range and velocity discrepancy for two different top
velocities, each with five repeated experiments to reduce the effect of
variability. The errors are evaluated only within the maximum range of the
radar, and can be seen in Table I.

|  $V_{max}$ [$\text{\,}\mathrm{m}\text{\,}{\mathrm{s}}^{-1}$] |  $R$ RMSE [$\text{\,}\mathrm{m}$] |  $V$ RMSE [$\text{\,}\mathrm{m}\text{\,}{\mathrm{s}}^{-1}$]
---|---|---|---
Exp. 1 | 3.0 | $0.21\pm 0.29$ | $0.39\pm 0.19$
Exp. 2 | 0.5 | $0.05\pm 0.03$ | $0.03\pm 0.01$
TABLE I: Evaluation of range and velocity error with respect to the odometry
data from the front car. R indicates range and V indicates velocity. Figure 5:
In orange, the speed (top) and distance (bottom) estimated with the radar, in
blue the odometry data. Radar data in the grey area is out of range.

###  V-B Dynamic Measurements

In the case of a moving radar, the relative velocity of the environment with
respect to the radar is not zero. This results in multiple targets in the
range-doppler map, whose energy depends on properties such as the reflectivity
of the material, the angle, and the distance to the radar. In our tests, most
of the reflections are caused by the track boundaries. In Fig. 6 such targets
appear on the side of negative velocity (2), forming the characteristic curved
pattern. This is due to the environment being projected into the map. Objects
in the far field appear at all distances, with a velocity approximately equal
to the ego velocity of the car. However, the radial velocity of the track
boundaries lowers as we consider points closer to the car since the relative
angle also increases. This can be observed from the curved pattern. Some
reflections from the ground can also be observed at close range (1).

In the frame shown, the opponent vehicle is still visible in the range-doppler
map (3), as it is on the positive velocity side. However, proper tracking in
the range-doppler space requires further processing, since the opponent could
be masked by the environment reflections at times. This demonstrates the
necessity of a more sophisticated filtering technique that will be addressed
in future work. Specifically, in the context of autonomous racing, we believe
we can exploit the knowledge of the environment and the odometry information
of the car to isolate the opponent in the range-doppler space. At the same
time, we expect that our method will increase the robustness of the LiDAR
point cloud by validating LiDAR points against their expected relative
velocity, allowing for LiDAR filtering within highly dynamic environments.
Finally, the fusion method will also improve the ability to correctly classify
static and dynamic obstacles in racing conditions.

Figure 6: Range-doppler map acquired with both vehicles in motion. Both the
opponent (3) and the environment (1,2) are visible in the range-doppler space.

##  VI Conclusions and Future Work

A novel low-power FMCW radar sensor was evaluated in the context of autonomous
racing. We evaluated the accuracy of distance and velocity tracking in a
radar-static scenario, showing that despite the radars low transmission power,
in the range of 10s of milliwatts, the sensor is capable of tracking distance
and velocity of the target with relatively low tracking error of
$0.21\text{\,}\mathrm{m}$ and
$0.39\text{\,}\mathrm{m}\text{\,}{\mathrm{s}}^{-1}$ respectively.

Dynamic experiments with the radar on a moving car were also conducted, in
order to simulate a more realistic racing context. The subsequent results shed
light on the challenges that a dynamic scenario will pose for the velocity
estimation of the target, posing a starting point for incremental research on
a novel LiDAR-in-radar sensor fusion algorithms.

## Acknowledgment

The authors would like to thank Steven Peter, who developed and tested the ROS
driver to interface the radar sensor into the racing stack.

## References

  * [1] S. Jarvenpaa and W. Standaert, “Emergent ecosystem for radical innovation: Entrepreneurial probing at formula e,” in _Proceedings of the 50th HICSS_ , 2017.
  * [2] J. Betz, H. Zheng, A. Liniger, U. Rosolia, P. Karle, M. Behl, V. Krovi, and R. Mangharam, “Autonomous vehicles on the edge: A survey on autonomous vehicle racing.” [Online]. Available: https://arxiv.org/abs/2202.07008
  * [3] E. Ghignone, N. Baumann, and M. Magno, “Tc-driver: A trajectory conditioned reinforcement learning approach to zero-shot autonomous racing,” _Journal of Field Robotics (JFR)_ , 2023, accepted for presentation at 2023 Journal of Field Robotics.
  * [4] J. Becker, N. Imholz, L. Schwarzenbach, E. Ghignone, N. Baumann, and M. Magno, “Model- and acceleration-based pursuit controller for high-performance autonomous racing,” 2023, accepted for presentation at 2023 International Conference on Robotics and Automation (ICRA), July 2023.
  * [5] S. Jung, S. Cho, D. Lee, H. Lee, and D. H. Shim, “A direct visual servoing-based framework for the 2016 IROS autonomous drone racing challenge,” _J. field robot._ , vol. 35, no. 1, pp. 146–166, Jan. 2018.
  * [6] M. O’Kelly, H. Zheng, D. Karthik, and R. Mangharam, “F1tenth: An open-source evaluation environment for continuous control and reinforcement learning,” in _NeurIPS 2019 Competition and Demonstration Track_. PMLR, 2020, pp. 77–89.
  * [7] A. Wischnewski, M. Geisslinger, J. Betz, T. Betz, F. Fent, A. Heilmeier, L. Hermansdorfer, T. Herrmann, S. Huch, P. Karle, F. Nobis, L. Ögretmen, M. Rowold, F. Sauerbeck, T. Stahl, R. Trauth, M. Lienkamp, and B. Lohmann, “Indy autonomous challenge – autonomous race cars at the handling limits,” 2022\. [Online]. Available: https://arxiv.org/abs/2202.03807
  * [8] J. Betz, A. Wischnewski, A. Heilmeier, F. Nobis, T. Stahl, L. Hermansdorfer, and M. Lienkamp, “A software architecture for an autonomous racecar,” in _2019 IEEE 89th Vehicular Technology Conference (VTC2019-Spring)_ , 2019, pp. 1–6.
  * [9] S. Campbell, N. O’Mahony, L. Krpalcova, D. Riordan, J. Walsh, A. Murphy, and C. Ryan, “Sensor technology in autonomous vehicles : A review,” in _2018 29th Irish Signals and Systems Conference (ISSC)_ , 2018, pp. 1–4.
  * [10] M. A. Cooper, J. F. Raquet, and R. Patton, “Range information characterization of the hokuyo ust-20lx lidar sensor,” _Photonics_ , vol. 5, no. 2, 2018. [Online]. Available: https://www.mdpi.com/2304-6732/5/2/12
  * [11] A. V. Malawade, T. Mortlock, and M. A. A. Faruque, “Ecofusion: Energy-aware adaptive sensor fusion for efficient autonomous vehicle perception,” in _Proceedings of the 59th ACM/IEEE Design Automation Conference_ , ser. DAC ’22. New York, NY, USA: Association for Computing Machinery, 2022, p. 481–486. [Online]. Available: https://doi.org/10.1145/3489517.3530489
  * [12] T. Broedermann, C. Sakaridis, D. Dai, and L. V. Gool, “Hrfuser: A multi-resolution sensor fusion architecture for 2d object detection,” 2022.
  * [13] Y. Zhang, A. Carballo, H. Yang, and K. Takeda, “Perception and sensing for autonomous vehicles under adverse weather conditions: A survey,” _ISPRS Journal of Photogrammetry and Remote Sensing_ , vol. 196, pp. 146–177, 2023.
  * [14] A. Raji, A. Liniger, A. Giove, A. Toschi, N. Musiu, D. Morra, M. Verucchi, D. Caporale, and M. Bertogna, “Motion planning and control for multi vehicle autonomous racing at high speeds,” in _2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC)_ , 2022, pp. 2775–2782.
  * [15] M. Scherer, M. Magno, J. Erb, P. Mayer, M. Eggimann, and L. Benini, “Tinyradarnn: Combining spatial and temporal convolutional neural networks for embedded gesture recognition with short range radars,” _IEEE Internet of Things Journal_ , vol. 8, no. 13, pp. 10 336–10 346, 2021.
  * [16] S. Saponara, M. S. Greco, and F. Gini, “Radar-on-chip/in-package in autonomous driving vehicles and intelligent transport systems: Opportunities and challenges,” _IEEE Signal Processing Magazine_ , vol. 36, no. 5, pp. 71–84, 2019.
  * [17] J. Kabzan, M. I. Valls, V. J. F. Reijgwart, H. F. C. Hendrikx, C. Ehmke, M. Prajapat, A. Bühler, N. Gosala, M. Gupta, R. Sivanesan, A. Dhall, E. Chisari, N. Karnchanachari, S. Brits, M. Dangel, I. Sa, R. Dubé, A. Gawel, M. Pfeiffer, A. Liniger, J. Lygeros, and R. Siegwart, “AMZ driverless: The full autonomous racing system,” _J. field robot._ , no. rob.21977, Aug. 2020.
  * [18] A. Liniger, A. Domahidi, and M. Morari, “Optimization-based autonomous racing of 1:43 scale rc cars,” _Optimal Control Applications and Methods_ , vol. 36, no. 5, p. 628–647, Jul 2014. [Online]. Available: http://dx.doi.org/10.1002/oca.2123
  * [19] F1TENTH Foundation, “Building the F1TENTH Car.” [Online]. Available: https://f1tenth.org/build.html
  * [20] R. Nabati and H. Qi, “Radar-camera sensor fusion for joint object detection and distance estimation in autonomous vehicles,” _arXiv preprint arXiv:2009.08428_ , 2020.
  * [21] J.-G. Wang, S. J. Chen, L.-B. Zhou, K.-W. Wan, and W.-Y. Yau, “Vehicle detection and width estimation in rain by fusing radar and vision,” in _2018 15th International Conference on Control, Automation, Robotics and Vision (ICARCV)_. IEEE, 2018, pp. 1063–1068.
  * [22] T. Zhou, M. Yang, K. Jiang, H. Wong, and D. Yang, “Mmw radar-based technologies in autonomous driving: A review,” _Sensors_ , vol. 20, no. 24, p. 7283, 2020.
  * [23] A. Wischnewski, M. Geisslinger, J. Betz, T. Betz, F. Fent, A. Heilmeier, L. Hermansdorfer, T. Herrmann, S. Huch, P. Karle _et al._ , “Indy autonomous challenge-autonomous race cars at the handling limits,” in _12th International Munich Chassis Symposium 2021: chassis. tech plus_. Springer, 2022, pp. 163–182.
  * [24] J. Betz, T. Betz, F. Fent, M. Geisslinger, A. Heilmeier, L. Hermansdorfer, T. Herrmann, S. Huch, P. Karle, M. Lienkamp _et al._ , “Tum autonomous motorsport: An autonomous racing software for the indy autonomous challenge,” _Journal of Field Robotics_ , 2022.


# Attentive Contractive Flow: Improved Contractive Flows with Lipschitz-
constrained Self-Attention

Avideep Mukherjee1, Badri N. Patro 2, Sahil Sidheekh 3, Maneesh Singh 3, Vinay
P. Namboodiri 4 This research was done in IITK

###### Abstract

Normalizing flows provide an elegant method for obtaining tractable density
estimates from distributions by using invertible transformations. The main
challenge is to improve the expressivity of the models while keeping the
invertibility constraints intact. We propose to do so via the incorporation of
localized self-attention. However, conventional self-attention mechanisms
don’t satisfy the requirements to obtain invertible flows and can’t be naively
incorporated into normalizing flows. To address this, we introduce a novel
approach called Attentive Contractive Flow (ACF) which utilizes a special
category of flow-based generative models - contractive flows. We demonstrate
that ACF can be introduced into a variety of state of the art flow models in a
plug-and-play manner. This is demonstrated to not only improve the
representation power of these models (improving on the bits per dim metric),
but also to results in significantly faster convergence in training them.
Qualitative results, including interpolations between test images, demonstrate
that samples are more realistic and capture local correlations in the data
well. We evaluate the results further by performing perturbation analysis
using AWGN demonstrating that ACF models (especially the dot-product variant)
show better and more consistent resilience to additive noise.111The paper is
under consideration at Computer Vision and Image Understanding

##  1 Introduction

While deep generative models based on generative adversarial networks (GANs)
and variational autoencoders (VAEs) produce state of the art results showing
impressive results on mega-pixel images, they do not have the ability to
obtain exact likelihood estimates. To address this need, flow based approaches
such as real NVP (Dinh, Sohl-Dickstein, and Bengio 2016) and invertible
residual networks (Behrmann et al. 2019) have been proposed. However, flow-
based models are still limited in their modeling capabilities as compared to
GANs and VAEs. This paper focuses on improving the modeling capability of
these models through incorporation of self-attention in them. For this
purpose, a sub-category of flow-based generative models called contractive
flows (Rezende and Mohamed 2015) is leveraged.

The challenge in obtaining effective mappings from a simple latent space
representation to a complex target distribution lies in maintaining the
constraints for a valid normalizing flow: (a) invertibility of the
transformation function, and, (b) the tractable computation of the log-
determinant of the Jacobian of the transformation function. Unfortunately,
attention in normalizing flows can’t be naively incorporated while satisfying
the above constraints. In this paper, we show that a category of flow-based
models, called contractive flows, can be effectively leveraged for
incorporating attention while addressing the above needs. Such models, called
Attentive Contractive Flows (ACF) are demonstrated to have faster convergence
during training, better modeling capability as well as better resilience to
noise in the input data.

Learning data distributions has witnessed remarkable success through
generative adversarial networks (GANs) such as StyleGAN2 (Karras et al. 2020).
Similarly, remarkable progress has been made using hierarchical variational
autoencoders and normalizing flows: NVAE (Vahdat and Kautz 2020) and Flow++
(Ho et al. 2019) provide impressive results for generative modeling based on
variational autoencoders and normalizing flows, respectively. Flow++
introduces a step where self-attention is applied to a slice of the image in
the coupling layer to improve the expressivity of the transformation. However,
so far, incorporating attention for the whole image space at a given step in
normalizing flow-based models has not been proposed to the best of our
knowledge.

Localized importance in the image space and latent space can be achieved by
applying the concept of self-attention (Vaswani et al. 2017; Cheng, Dong, and
Lapata 2016). Self-attention offers a balance between the ability to model
inter-dependent features and the computational and statistical efficiency. The
self-attention module calculates response at a position as a weighted sum of
the features at all positions, where the weights – or attention vectors – are
calculated with only a small computational cost. Hence, our main motivation is
derived from the fact that during density estimation of complex and high
dimensional data, it is important to have information about the key positions
in the image that is representative of a particular sample, which in turn is
obtained through attention. While convolutional neural network does the job of
extracting local features from an image and its subsequent feature
transformations, having self-attention as an additional building block
definitely improves the expressivity of the model. This information not only
helps in tasks like image recognition but also proves to be valuable in the
area of generative modelling. Self-attention has been incorporated in GANs
(Zhang et al. 2019) and VAEs (Lin, Liu, and Liang 2019). However, as
Normalizing Flow models have different transformation functions, it is
challenging to incorporate self-attention in Normalizing Flows in a generic
sense.

A main challenge we need to solve in order to achieve this task is to examine
the invertitbility of the self-attention module. In this paper we show that
while the general self-attention module is not uniquely invertible, a variant
of self-attention is possible that satisfies this property by being a
contraction. This class of self-attention is incorporated in three different
contractive flows: iResNet (Behrmann et al. 2019), Residual Flows (Chen et al.
2019) and iDenseNets (Perugachi-Diaz, Tomczak, and Bhulai 2021) each one being
an improvement over the previous one. We show that the performance of all the
three contractive flows gets better respectively with Self Attention. A
contractive flow uses Banach’s Fixed Point Theorem to guarantee exact
iterative inverses of arbitrarily complex neural networks, as long as the
neural network function remains a contraction. We use the variant of self-
attention mechanism inside a contractive neural network to provide a perfectly
law-abiding flow-based generative model. Through this model we are also able
to obtain improved expressive capability for obtaining flows and show
significantly improved performance with fewer steps as compared to other
state-of-the-art NF models.

##  2 Background

###  2.1 Flow-Based Generative Models

Normalizing Flows are a class of generative models where an ‘simple’
paramaterizable base distribution is transformed into a more complex
approximation for the posterior distribution (Rezende and Mohamed 2015). This
transformation is achieved by passing the base distribution through a series
of invertible and bijective mappings. Let $\mathbf{z}\in\mathbb{R}^{D}$ and
$\mathbf{y=f(z)}$ Let $\mathbf{z}\sim q(\mathbf{z})$, be an simple base
distribution. The change of variables theorem expresses a relation between the
probability density functions $p_{Y}(\mathbf{y})$ and $q(\mathbf{z})$:

$p_{Y}(\mathbf{y})=q(\mathbf{z})\left|\mathrm{det}\frac{\partial
f^{-1}}{\partial\mathbf{y}\
}\right|=q(\mathbf{z})\left|\mathrm{det}\frac{\partial f}{\partial\mathbf{z}\
}\right|^{-1}$ (1)

If we apply a series of such mappings $f_{k}$, $k\in{1,\dots,K}$ with
$K\in\mathbb{N}_{+}$, we obtain a normalizing flow. The log probability of the
final distribution can thus be obtained by:

$\log p_{Y}(\mathbf{y})=\log p_{Y}(\mathbf{z}_{K})=\log
q(\mathbf{z}_{0})-\sum_{k=1}^{K}\log\left|\mathrm{det}\frac{\partial{f_{k}}}{\partial{\mathbf{z}_{k-1}}}\right|$
(2)

From Equation 1 & 2, it is clearly observed that every normalizing flow
architecture must satisfy two conditions. First, the transformation function
should be invertible. Secondly, the log-determinant of the Jacobian should be
tractable.

###  2.2 Contractive Flows

#### iResNet & Residual Flows

Residual Flows (Papamakarios et al. 2019) are a class of invertible functions
of the form:

$\mathbf{z^{\prime}=f(z)=z+g_{\phi}(z)}$ (3)

where $g_{\phi}:\mathbb{R}^{D}\mapsto\mathbb{R}^{D}$ is a neural network with
parameters $\phi$. (Behrmann et al. 2019) proposed the transformation of
Equation 3 as an invertible residual network or iResNet. Residual
transformations like this can be made invertible with certain constraints on
$g_{\phi}$. (Behrmann et al. 2019) show that a residual transformation is
guaranteed to be invertible if the function $g_{\phi}$ is a contraction. A
contraction is a special case of a Lipschitz continuous function. A function
$F:\mathbb{R}^{D}\mapsto\mathbb{R}^{D}$ is said to be $K$-Lipschitz continuous
when for a given distance measure $\delta$, there exists a constant $K$ such
that for two inputs $\mathbf{x_{1}}$ and $\mathbf{x_{2}}$ we have:

$\delta(F(\mathbf{x_{1}}),F(\mathbf{x_{2}}))\leq
K\delta(\mathbf{x_{1}},\mathbf{x_{2}})$ (4)

The smallest such $K$ is called the Lipschitz constant of $F$ or
$\text{Lip}(F)$. If $\text{Lip}(F)\leq 1$, then $F$ is said to be a
contraction. Let us consider the following equation involving the contraction
in Equation 3.

$\mathbf{F(\hat{z})=z^{\prime}-g_{\phi}(\hat{z})}$ (5)

Since $\mathbf{g_{\phi}}$ is contractive, $\mathbf{F}$ is also contractive
with the same Lipschitz constant. Therefore, from Banach’s Fixed Point Theorem
(Rudin 2006, 1976), it is ensured that there exists a unique $\mathbf{z_{*}}$
such that $\mathbf{z*=z^{\prime}-g_{\phi}(z_{*})}$ which can be rearranged to
$\mathbf{z^{\prime}=f(z_{*})}$. Hence it follows that $\mathbf{f}$ is
invertible (Papamakarios et al. 2019). In fact, the inversion algorithm is
iteratively designed from Equation 5 as follows:

$z_{k+1}=z^{\prime}-g_{\phi}(z_{k})\quad\text{for}\quad k\geq 0$ (6)

The Banach’s fixed-point theorem guarantees the convergence of the recursive
algorithm at an exponential rate to $\mathbf{z_{*}=f^{-1}(z^{\prime})}$ for
any arbitrary initialization of $z_{0}$ (usually it is preferred to have
$z_{0}=z^{\prime}$). It is clearly observed that composition of $K$ such
residual transformations also preserve the contractive properties with
Lipschitz constant being $\prod_{i=1}^{K}L_{K}$, where $L_{K}$ is the
respective Lipschitz constant of $F_{K}$. However, there are two major
challenges to build residual flows. First off, the design of the neural
network function is restricted to being Lipschitz continuous, that too
contractive which limits the flexibility of the network. Secondly, the
calculation of log-determinant of the Jacobian of such a transformation cannot
be efficiently computed except for automatic differentiation which takes
$\mathcal{O}(\mathcal{D}^{3})$ time. However, the log-determinant can be
approximated using the results of (Hall 2015) and (Withers and Nadarajah 2010)
and re-written as a power series of the trace of the Lipschitz network
$\mathbf{g_{\phi}}$:

$\begin{split}\log|\det
J_{f_{\phi}}(\mathbf{z})|&=\log|\det\left(\mathbf{I}+J_{g_{\phi}}(\mathbf{z})\right)|\\\
&=\sum_{k=1}^{\infty}\frac{(-1)^{k+1}}{k}\text{Tr}\left\\{J^{k}_{g_{\phi}}(\mathbf{z})\right\\}\end{split}$
(7)

where $J^{k}_{g_{\phi}}(\mathbf{z})$ is the $k$-th power of the Jacobian of
$g_{\phi}$ at $\mathbf{z}$.The trace can be estimated using Hutchinson trace
estimator (Hutchinson 1989). Residual Flows (Chen et al. 2019) improved this
method where the power series can be finitely approximated using the unbiased
Russian-roulette estimator. This results in lower requirement of computation
of the power series than in case of iResNet. Also, they introduce LipSwish
activation function to avoid derivative saturation.

#### Invertible DenseNets

Invertible DenseNets (Perugachi-Diaz, Tomczak, and Bhulai 2021) use a
DenseBlock as a residual layer. A DenseBlock in iDenseNet is slightly
different than a standard DenseBlock and is defined as
$F:\mathbb{R}^{d}\mapsto\mathbb{R}^{d}$ with $F(x)=x+g(x)$ where $g$ is
comprised of dense layers $\\{h_{i}\\}_{i=i}^{n+1}$. $h_{n+1}$ is a $1\times
1$ convolution to match the dimension of the output size $\mathbb{R}^{d}$.
Each $h_{i}$ has two concatenated parts, the input and the transformed input:

$\displaystyle h_{i}(x)$ $\displaystyle=\begin{bmatrix}x\\\ \phi(W_{i}(x))\\\
\end{bmatrix}$ (8)

where $W_{i}$ is convolutional matrix and $\phi$ is a non-linearity with
Lip($\phi$) $\leq$ 1 such as ReLU, ELU, LipSwish (Chen et al. 2019) or
CLipSwish (Perugachi-Diaz, Tomczak, and Bhulai 2021).

Figure 1: Model diagram comparing one step of the transformation function of a generic contractive flow (CF) (left) and a contractive flow with Self Attention (ACF)(right). SA stands for Self Attention. |  |
---|---|---
Figure 2: Convergence plots on (a) iResNet on MNIST, (b) Residual Flows on
CIFAR10 and (c) Invertible DenseNet on ImageNet32, in terms of train bits/dim
across iterations. We observe that contractive flows with Self Attention
converge faster (even with fewer steps of the flow) than their non-attention
counterparts.

##  3 Method: Contractive Flows with Self-Attention

Before defining the transformation function with self-attention, we need to
make sure that the two conditions of normalizing flow function are satisfied
i.e, the function is invertible and the log-determinant of the Jacobian is
tractable. In order to have attention incorporated in contractive flows, the
attention module needs to be inserted in between the convolutional layers in
the function $\mathbf{g}_{\phi}$ of Equation 3. This would require the
attention function to be a Lipschitz continuous function. Since the attention
function involves computing a dot-product between the query and key matrices,
it will be referred to as dot-product self-attention. Even if the dot-product
self-attention module can be inverted, the log-determinant of the Jacobian is
hard to compute. A trick to compute non-tractable log-determinant of the
Jacobian is to use the results of (Withers and Nadarajah 2010) that reduces
computing the determinant to computing the trace. The result shows that for
any non-singular matrix $A\in\mathbb{R}^{d\times d}$

$\ln(\det A)=\text{tr}(\ln A)$

where $ln$ is a matrix logarithm, $tr$ is the trace of a matrix. However, this
result requires the determinant be a positive quantity. It is proven that the
Lipschitz constrained perturbations of the form $x+g(x)$ yield positive
Jacobian determinants (Behrmann et al. 2019). Therefore, for a function
$F(x)=x+g(x)$, we have,

$|\det J_{F}(x)|=\det J_{F}(x)$ (9)

Here comes the role of contractive flows which defines transformation
functions as $F(x)=x+g(x)$. But, as explained in Section 2.2, in order to make
the transformation function invertible, $g$ should not only be Lipschitz
continuous but also needs to be a contraction, i.e., $\text{Lip}(g)<1$. The
traditional self-attention function that is popular in different architectures
like Transformers (Vaswani et al. 2017) or Self-Attention GANs (Zhang et al.
2019) is not Lipschitz continuous (Kim, Papamakarios, and Mnih 2020).
Fortunately, there’s an alternative formulation of the self-attention function
given by Kim et al. (Kim, Papamakarios, and Mnih 2020) that is proven to be
Lipschitz continuous. The traditional Self Attention was referred as dot-
product Self Attention and the alternative was called $L_{2}$ Self Attention,
since the expression replaces computation of the dot product between the key
and value convolutions with an $L_{2}$ Norm. The expressions of the $L_{2}$
Self Attention are provided keeping in mind a multi-headed Self Attention
module that is most commonly prevalent in several transformer architectures.
In this work, since the attention mechanism is to be applied over images and
not text, we have considered the number of heads to be one and have modified
the equations of (Kim, Papamakarios, and Mnih 2020) in the following way. The
$L_{2}$ self-attention function on an image $X\in\mathbb{R}^{N\times D}$
(where $N$ is the product of the height and width of the image and $D$ is the
number of channels) replaces the dot-product operation performed in dot-
product self-attention module by:

$P_{i,j}\propto\text{exp}(L_{i,j})=\text{exp}\left(-\frac{||x_{i}^{T}W^{Q}-x_{j}^{T}W^{K}||^{2}_{2}}{\sqrt{D}}\right)$
(10)

Here $W^{Q}=W^{K}\in\mathbb{R}^{D\times\bar{D}}$. This is necessary for the
function to be Lipschitz as proven in (Kim, Papamakarios, and Mnih 2020). So,
the full attention-mechanism function is:

$F(X)=PXAW^{V}W^{O}$ (11)

where,$W^{O},W^{V}\in\mathbb{R}^{D\times D}$,
$A=W^{Q}(W^{Q})^{T}/\sqrt{D}\in\mathbb{R}^{D\times D}$ and
$P\in\mathbb{R}^{N\times N}$ is given by with the normalization constant that
makes sure that $\sum_{j}P_{i,j}=1$. Since there is no non-linear activation
function involved between $W^{V}$ and $W^{O}$ in the expression of Equation
11, the product of the two $1\times 1$ convolution matrices can be replaced by
a single one. Hence, the equation can be further simplified to:

$F(X)=PXAW^{L}$ (12)

where $W^{L}$ is a $1\times 1$ convolution matrix which acts as a replacement
parameter of $W^{V}W^{O}$. Substituting $W^{V}W^{O}$ by $W^{L}$ reduces the
number of parameters significantly and in turn makes the training process time
and memory efficient. Also, since the second dimension of $W^{Q}$, i.e.
$\bar{D}$ gets dissolved in multiplication, we can replace it by $\frac{D}{k}$
for $k=1,2,4,8,\cdot$. We have used $\bar{D}=\frac{D}{8}$ for memory
efficiency.

Input: network $f$, residual block $g$, number of power series terms $n$,
$W^{Q}$: the query convolution, $W^{V},W^{O}$ : the value and out convolution
respectively

Data: $X\in\mathbb{R}^{N\times D}$ (where $N$ is the product of the height and
width of the image and $D$ is the number of channels)

1 for  _each residual block_ do

2 Lip constraint: $\hat{W_{j}}$ := $SN(W_{j},X)$ for Layer $W_{j}$

3
$P:=\texttt{S}\left(-\frac{||XW^{Q}||^{2}_{r}\mathds{1}^{T}-2XW^{Q}(XW^{Q})^{T}+\mathds{1}||XW^{Q}||^{2T}_{r}}{\sqrt{D}}\right)$

4 Replacement Matrix: $W^{L}:=W^{V}\times W^{O}$ as in eq: 12

5 $A:=W^{Q}(W^{Q})^{T}/\sqrt{D}$

6 $F:=P\times X\times A\times W^{L}$ as in eq: 11

7
$\text{Lip}_{2}(F):=\frac{\sqrt{N}}{\sqrt{D}}(4\phi^{-1}(N-1)+1)||W^{Q}||_{2}||W^{V}||_{2}W^{O}||_{2}$

8 $\hat{W_{j+1}}:=\gamma\frac{F}{\text{Lip}_{2}(F)}+X$: the final attention
output as mentioned in eq: 15

9 Draw $v$ from $\mathcal{N}(0,\mathbf{I})$

10 $w^{T}=v^{T}$

11 ln det := 0

12 for  _k = 1 to n_ do

13 $w^{T}:=w^{T}J_{g}$ (vector-Jacobian product)

14 ln det := ln det $+(-1)^{k+1}w^{T}v/k$

15

16

Algorithm 1 Pseudo Code for a forward pass of an ACF. SN stands for Spectral
Normalization as in (Behrmann et al. 2019)

Equation 10 can be rewritten using matrix operations for efficient computation
as:

$P=\textbf{S}\left(-\frac{||XW^{Q}||^{2}_{r}\mathds{1}^{T}-2XW^{Q}(XW^{K})^{T}+\mathds{1}||XW^{K}||^{2T}_{r}}{\sqrt{D}}\right)$
(13)

where S is the softmax function and $||A||^{2}_{r}$ indicates the squared
$L_{2}$ norm to each row of $A$, so if $A\in\mathbb{R}^{m\times n}$, then
$||A||^{2}_{r}\in\mathbb{R}^{m}$. This formulation of self-attention, also
known as the $L_{2}$ Self-Attention is proven to be Lipschitz continuous with
the following bound on $\text{Lip}_{2}(F)$:

$\text{Lip}_{2}(F)\leq\frac{\sqrt{N}}{\sqrt{D}}(4\phi^{-1}(N-1)+1)||W^{Q}||_{2}||W^{V}||_{2}W^{O}||_{2}$
(14)

where $\phi(x)=x\exp(x+1)$ is an invertible univariate function on $x>0$ and
$N$ is the input size. Also, $\phi^{-1}(N-1)=W_{0}(\frac{N}{e})$ where $W_{0}$
is the Lambert W-function (Kim, Papamakarios, and Mnih 2020). Hence, in order
to make $F$ a contraction, we divide $F$ by the upper bound of
$\text{Lip}_{2}(F)$ to obtain contractive-$L_{2}$ Self-Attention. This
function satisfies every property of being a part of the transformation
function of the normalizing flow. Therefore, final attention output is given
by:

$\text{out}=\gamma\frac{F}{\text{Lip}_{2}(F)}+X$ (15)

where $\gamma$ is a learnable scalar initialized to 0. $\gamma$ helps the
network to first attend to the local features in the neighborhood and then
gradually learn to assign more weight to the non-local evidence (Zhang et al.
2019). A detailed procedure of the forward pass of one step of the flow along
with the computation of the $L_{2}$ Self Attention is provided in Algorithm 1.

##  4 Experiments

We evaluated the inclusion of Self Attention in three different contractive
flows: invertible ResNets, Residual Flows, and invertible DenseNets.
Quantitative analysis includes comparison with other state-of-the-art methods
using the popular bits/dim metric (Table 1) on various datasets as discussed
in section 4.1. We analyze the rate of convergence of the flows both with and
without attention and discuss them in Figure 2. The qualitative analysis
includes visualizations for the reconstruction of real images as well as
generated samples (Figure 3). Further, we also interpolate between latent
variables to demonstrate that ACF helps achieve smooth transitions between
samples in the image space (Figure 4). Finally, we conduct ablation studies to
validate the robustness and efficacy of ACF.

Model | MNIST |  CIFAR 10 |  IMAGE NET32 |  IMAGE NET64
---|---|---|---|---
Real NVP | 1.06 | 3.49 | 4.28 | 3.98
Glow | 1.05 | 3.35 | 4.09 | 3.81
FFJORD | 0.99 | 3.40 | - | -
Flow++ | - | 3.29 | $3.86^{*}$ | $3.69^{*}$
iResNet (iR) | 1.05 | 3.45 |  | -
iR + SA (ACF) | 0.87 | 3.40 | - | -
ResFlow (RF) | 0.97 | 3.28 | 4.01 | 3.76
RF + SA (ACF) | 0.92 | 3.34 | 3.86 | 3.70
iDenseNet (iD) | - | 3.25 | 3.98 | -
iD + SA (ACF) | - | 3.14 | 3.75 | -
Table 1:  Results [bits/dim] on standard benchmark datasets for density
estimation. * are the results obtained through variational dequantization (Ho
et al. 2019) which we do not compare against (following Residual Flow).

###  4.1 Datasets

To experimentally validate the efficacy of ACF, we consider the datasets -
MNIST (LeCun and Cortes 2010), CIFAR10 (Krizhevsky, Nair, and Hinton 2014),
ImageNet32 and ImageNet64 (Chrabaszcz, Loshchilov, and Hutter 2017). Due to
constrained resources we could not conduct experiments on higher-dimensional
datasets like CelebA HQ 256. However, we conduct experiments on a down-sampled
version of the CelebA-HQ dataset ( 5bit 64 $\times$ 64) and provide
qualitative results. We follow the standard train-test split for each of these
datasets, as further elaborated below. The CIFAR10 dataset consists of 60,000
$32\times 32$ color images belonging to 10 classes, with 6000 images per
class. We train the models on 50,000 images and keep the rest 10,000 images to
generate the test results. The MNIST dataset contains 70,000 $28\times 28$
black and white images belonging to 10 classes, each class signifying a digit.
We use 60,000 for the training of the models and the remaining 10,000 images
for testing. The ImageNet32 and ImageNet64 datasets each comprise 1,281,167
training images from 1000 classes and 50,000 test images(50 images per class).
The 5bit CelebA-HQ64 dataset contains 202,599 face images and is pre-processed
as defined in (Papamakarios et al. 2019).

###  4.2 Density Estimation and Generative Modeling

We train the attentive contractive flow models (iResNet+SA, Residual Flow+SA,
iDenseNet+SA) on the datasets reported in their non-attentive counterparts
(section 5.1). We also follow the same architecture for each of these models,
except for the addition of the self-attention layer within each transformation
block. Specifically, the models follow a multi-scale architecture with
multiple blocks of invertible transformation and Self Attention incorporated
into them. A diagram of how each attentive block compares with the non-
attentive counterpart is provided in Figure 1. Though the block-wise
architectures are similar, the added expressivity provided by self-attention
enables us to use fewer no. of transformation blocks in ACF, all the while
attaining superior performance. We summarize the density estimation results
for ACF quantitatively using the test bits/dim metric in Table 1, comparing it
with the non-attentive contractive flows as well as other state-of-the-art
models. We observe that ACF not only helps improve contractive flows but also
outperforms the other state-of-the-art results. Further, ACF also complements
the lift in performance with a faster convergence rate (Figure 2). Despite
using fewer transformation blocks, ACF trained for a smaller no. of epochs is
able to outperform (on average) its non-attentive counterparts. For example,
the ACF results shown (Table 1) are generated from models that have been
trained for fewer than 100 epochs, whereas the numbers reported in iResNet
(Behrmann et al. 2019), Residual Flows (Chen et al. 2019), and Invertible
DenseNets (Perugachi-Diaz, Tomczak, and Bhulai 2021) are from models that
underwent 200-300 epochs of training for all the datasets. This empirically
validates the added expressivity of ACF. In all our experiments, we use the
log determinant approximation that is usually followed in various contractive
flows and also briefly described in Section 2.2. Further model-specific
architectures and implementation details are provided in the supplementary
material.

(a) CIFAR10

(b) ImageNet32

Figure 3: The images in the top, bottom and middle row are respectively the
real, reconstructed and generated images. (a) CIFAR10 results from ACF
(Residual Flows). (b) ImageNet32 results from ACF (iDenseNet).

Figure 4:  Interpolation between CelebA images, from one face to another using ACF(Residual Flow) and Eq.- 16. |  |
---|---|---
Figure 5: Variation of (a)bits/dim, (b)log likelihood and (c) volume correction values in ACF with dot-product and ACF with $L_{2}$ Self Attention with the change in intensity of Gaussian noises applied to perturb the input MNIST images. While the variations in the the values with $L_{2}$ Self Attention are always consistent, the dot-product plots again fail to produce such consistent variations. (a) Input | (b) $\epsilon(\sigma=0.1)$ Dot Attention $L_{2}$ Attention | (c) $\epsilon(\sigma=0.001)$ Dot Attention $L_{2}$ Attention
---|---|---
|  |
Figure 6: Reconstructions of perturbed input images by a Contractive Flow with
Dot and $L_{2}$ Self Attention. The perturbation is done by adding Gaussian
noises ($\epsilon$) of different variances ($\sigma$) as shown in (b) and (c).
ACF with $L_{2}$ Self Attention produces better reconstructions with decrease
in the intensity of noise. ACF with dot-product Self Attention fails to do so.

###  4.3  Qualitative Results

Figure 3 depicts qualitative samples and reconstructed images for ACF(Residual
Flow) and ACF(iDenseNet) on CIFAR10 and ImageNet32. We observe that ACF is
able to obtain exact reconstructions as well as generate realistic samples.
Further qualitative samples reconstructions and samples on different datasets
are provided in the supplementary material.

We also train ACF(Residual Flow) on the CelebA dataset and perform
interpolation between test images to provide a better visual intuition. We
apply the transformation function (trained ACF) on the test images and obtain
the corresponding samples in the latent space, i.e., the simple base
distribution. We then perform linear interpolation between random pairs of
images in the latent space, obtaining $N$ equally spaced interpolated latent
samples per pair. We then apply the inverse of the transformation function on
the interpolated latent samples to obtain their corresponding images. More
formally, the input images, say $x_{1}$ and $x_{2}$ are passed through the
forward transformation function of the model to generate the random variables
from the simple distribution, say $z_{1}$ and $z_{2}$. The intermediate random
variables are generated by the following incremental operation:

$\delta=z_{1}+\frac{i}{N}\times(z_{2}-z_{1})\quad\forall
i\in\\{0,1,2,\cdots,N+1\\}$ (16)

We follow the increment rule provided in the computation of integrated
gradients (Sundararajan, Taly, and Yan 2017). The procedure to generate the
interpolated reconstructions is described in Algorithm 2. The results are
depicted in Figure 4. We observe that ACF is able to interpolate effectively,
providing smooth transitions between pairs of diverse and unseen faces.

Input: $f$: model, $nSteps$: interpolation steps)

Data: $x_{C_{1}}$ and $x_{C_{2}}$

1 $z_{C_{1}}$ = $f(x_{C_{1}})$ and $z_{C_{2}}$ = $f(x_{C_{2}})$

2 for  _i $\in$ $\\{0,1,2,\cdots,nSteps+1\\}$_ do

3 $\delta$ = $z_{C_{1}}+\frac{i}{nSteps}\times(z_{C_{2}}-z_{C_{1}})$
reconstructed_image = $f^{-1}(\delta)$

Algorithm 2 Interpolation Between Two Images

Further interpolation results are provided in the supplementary material.

Noise ($\sigma$) | ACF(dot-product) | ACF($L_{2}$)
---|---|---
0.1 | -726.63 | 42889.64
0.01 | 35772.76 | 4959.88
0.001 | -9298.64 | 512.88
0.0001 | -15126.88 | 47.50
Table 2: Comparison of bits/dim of reconstructions of the perturbed input
images with Gaussian noises with the corresponding variances on ACF with dot-
product Self Attention and ACF with $L_{2}$ Self Attention.

###  4.4 Perturbation Analysis with Gaussian Noise

We perform ablations to study the robustness of Attentive Contractive Flows
and validate the need for $L_{2}$ Self Attention over dot-product Self
Attention by qualitatively and quantitatively evaluating the reconstructions
of perturbed input images. For this experiment, we consider ACF(iResNet) over
the MNIST dataset. The input images are perturbed by adding Gaussian noises of
different variances($\sigma$) - $0.00001,0.0001,0.001,0.01,0.1,1$. We
quantitatively evaluate the model by reporting the bits/dim values of the
reconstructions for the different levels of noises. We compare the performance
of $L_{2}$ Self Attention based ACF with dot product Self Attention based ACF
in Table 2. The fact that dot-product self attention is not Lipschitz
constrained leads to wrongful computation of the log-probability using the
change of variable formula which sometime results in negative bits/dim as we
can see in Table 2.On the other hand, we observe that the model with $L_{2}$
Self Attention performs better and more consistently. The qualitative results
in Figure 6 also demonstrate that the reconstructions for ACF($L_{2}$) improve
with decreasing values of $\sigma$. However, such is not the case with dot-
product Self Attention. The robustness of using $L_{2}$ Self Attention with
ACF is also graphically demonstrated in Figure 5(a),(b) and (c). While ACF
with $L_{2}$ Self Attention follows the natural and consistent increase in
bits/dim, dot product Self Attention fails to perform as expected and produces
random undesirable values. We notice similar behavioral consistencies w.r.t
the change in the trace values and also the log probability values when
$L_{2}$ SA is used instead of dot-product SA.

##  5 Related Works

We provide an overview of the main related works in the following areas.

###  5.1 Attention in generative models

Deep generative modeling has drawn significant research interest in recent
years, and many state-of-the-art techniques have emerged. The objective is to
estimate the density or the distribution of the data and generate new samples
from the estimated distribution. This estimation can either be done implicitly
or explicitly, giving rise to varied approaches. Perhaps the most famous among
models that do implicit density estimation is GANs (Goodfellow et al. 2014),
which revolutionized the field of image synthesis and slowly found
applications in different modes of generative modeling. A GAN effectively
poses the unsupervised task of data generation as a supervised learning
problem through an adversarial min-max game between two agents; A generator
that maps a latent space to the data space and a discriminator that learns to
distinguish the real data samples from the fake samples that the generator
produces. Self Attention GAN (Zhang et al. 2019) incorporates the self-
attention mechanism on both the generator and discriminator of a GAN,
establishing competitive results.

In the domain of explicit density estimation, the density is either
approximated or tractably estimated. Variational autoencoders (VAEs)
constitute the most famous example among approximate density estimation
models, apart from Boltzmann Machines (Smolensky 1986). VAEs explicitly
optimize the log-likelihood of the data by maximizing the evidence lower bound
(ELBO) (Kingma and Welling 2013). Assuming the data distribution to be
Gaussian, the objective function tries to find the best parameters that fit
the data. A recent work proposes to incorporate Self Attention on the encoder
feature space of VAEs, to enable a better approximation of the posterior (Lin,
Liu, and Liang 2019).

###  5.2 Normalizing Flows

In our work, we focus on normalizing flow models. They are a class of
generative models that aims to estimate a tractable density of the data. Other
popular approaches in this domain include fully visible belief networks like
NADE (Uria et al. 2016), MADE (Kumar et al. 2016), PixelCNN, PixelRNN (Oord,
Kalchbrenner, and Kavukcuoglu 2016; Van den Oord et al. 2016) and WaveNet
(Oord et al. 2016). Normalizing Flows (Rezende and Mohamed 2015) fall under
the category broadly known as change of variable models, which employ the
theorem to transform a simple parameterizable base distribution into some
complex approximation of the posterior distribution. In contrast to GANs and
VAEs, normalizing flows are an attractive class due to their ability to obtain
tractable density estimates. Prominent normalizing flow models include Non-
Linear Independent Components Estimation (NICE) (Dinh, Krueger, and Bengio
2014), real non-volume preserving (real NVP) flow (Dinh, Sohl-Dickstein, and
Bengio 2016), inverse autoregressive flow (IAF) (Kingma et al. 2016), masked
autoregressive flow (MAF) (Papamakarios, Pavlakou, and Murray 2017), and Glow
(Kingma and Dhariwal 2018).

Flow++ (Ho et al. 2019) suggests technical and implementation tweaks for
designing powerful normalizing flow models. One of the suggestions is to use
Self Attention in the affine coupling layers of models such as real NVP (Dinh,
Sohl-Dickstein, and Bengio 2016). However, the model only provides attention
to a slice of the image or feature space at any given step instead of the
whole image. Hence, while using Self Attention in a part of the input improves
the performance marginally, we provide a way to incorporate Self Attention
into normalizing flows in such a way that the attention is applied to the
whole image and feature spaces at once instead of just a slice, thereby
improving the performance of the normalizing flow models by a greater margin.

##  6 Conclusion

Through this work, we present a principled method for incorporating self-
attention into contractive flows for modeling complex image distributions. The
addition of global self-attention, i.e., self-attention on the whole
image/transformed feature space, helps improve the expressivity of normalizing
flows. We establish, through exhaustive experimentation, that $L_{2}$ self-
attention helps attain state-of-the-art results while driving faster model
convergence. Interesting future directions to explore include understanding
how/why self-attention helps improve specific model architectures better than
others. We believe that our contributions open new avenues for research to
strengthen the representative power of normalizing flows, bringing principled,
efficient, and tractable density estimation a step closer.

## References

  * Behrmann et al. (2019) Behrmann, J.; Grathwohl, W.; Chen, R. T.; Duvenaud, D.; and Jacobsen, J.-H. 2019\.  Invertible residual networks.  In _International Conference on Machine Learning_ , 573–582.
  * Chen et al. (2019) Chen, R. T.; Behrmann, J.; Duvenaud, D. K.; and Jacobsen, J.-H. 2019.  Residual flows for invertible generative modeling.  In _Advances in Neural Information Processing Systems_ , 9916–9926.
  * Cheng, Dong, and Lapata (2016) Cheng, J.; Dong, L.; and Lapata, M. 2016.  Long short-term memory-networks for machine reading.  _arXiv preprint arXiv:1601.06733_.
  * Chrabaszcz, Loshchilov, and Hutter (2017) Chrabaszcz, P.; Loshchilov, I.; and Hutter, F. 2017.  A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets.  arXiv:1707.08819.
  * Dinh, Krueger, and Bengio (2014) Dinh, L.; Krueger, D.; and Bengio, Y. 2014.  Nice: Non-linear independent components estimation.  _arXiv preprint arXiv:1410.8516_.
  * Dinh, Sohl-Dickstein, and Bengio (2016) Dinh, L.; Sohl-Dickstein, J.; and Bengio, S. 2016.  Density estimation using real nvp.  _arXiv preprint arXiv:1605.08803_.
  * Goodfellow et al. (2014) Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; and Bengio, Y. 2014.  Generative adversarial nets.  In _Advances in neural information processing systems_ , 2672–2680.
  * Hall (2015) Hall, B. 2015.  _Lie groups, Lie algebras, and representations: an elementary introduction_ , volume 222.  Springer.
  * Ho et al. (2019) Ho, J.; Chen, X.; Srinivas, A.; Duan, Y.; and Abbeel, P. 2019.  Flow++: Improving flow-based generative models with variational dequantization and architecture design.  _arXiv preprint arXiv:1902.00275_.
  * Hutchinson (1989) Hutchinson, M. F. 1989.  A stochastic estimator of the trace of the influence matrix for Laplacian smoothing splines.  _Communications in Statistics-Simulation and Computation_ , 18(3): 1059–1076.
  * Karras et al. (2020) Karras, T.; Laine, S.; Aittala, M.; Hellsten, J.; Lehtinen, J.; and Aila, T. 2020\.  Analyzing and Improving the Image Quality of StyleGAN.  _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Kim, Papamakarios, and Mnih (2020) Kim, H.; Papamakarios, G.; and Mnih, A. 2020.  The Lipschitz Constant of Self-Attention.  _arXiv preprint arXiv:2006.04710_.
  * Kingma and Dhariwal (2018) Kingma, D. P.; and Dhariwal, P. 2018.  Glow: Generative flow with invertible 1x1 convolutions.  In _Advances in neural information processing systems_ , 10215–10224.
  * Kingma et al. (2016) Kingma, D. P.; Salimans, T.; Jozefowicz, R.; Chen, X.; Sutskever, I.; and Welling, M. 2016.  Improved variational inference with inverse autoregressive flow.  In _Advances in neural information processing systems_ , 4743–4751.
  * Kingma and Welling (2013) Kingma, D. P.; and Welling, M. 2013.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_.
  * Krizhevsky, Nair, and Hinton (2014) Krizhevsky, A.; Nair, V.; and Hinton, G. 2014.  The cifar-10 dataset.  _online: http://www. cs. toronto. edu/kriz/cifar. html_ , 55.
  * Kumar et al. (2016) Kumar, A.; Irsoy, O.; Ondruska, P.; Iyyer, M.; Bradbury, J.; Gulrajani, I.; Zhong, V.; Paulus, R.; and Socher, R. 2016.  Ask me anything: Dynamic memory networks for natural language processing.  In _International conference on machine learning_ , 1378–1387.
  * LeCun and Cortes (2010) LeCun, Y.; and Cortes, C. 2010.  MNIST handwritten digit database.
  * Lin, Liu, and Liang (2019) Lin, L.; Liu, X.; and Liang, W. 2019.  Improving Variational Auto-Encoder with Self-Attention and Mutual Information for Image Generation.  In _Proceedings of the 3rd International Conference on Video and Image Processing_ , ICVIP 2019, 162–167. New York, NY, USA: Association for Computing Machinery.  ISBN 9781450376822.
  * Oord et al. (2016) Oord, A. v. d.; Dieleman, S.; Zen, H.; Simonyan, K.; Vinyals, O.; Graves, A.; Kalchbrenner, N.; Senior, A.; and Kavukcuoglu, K. 2016.  Wavenet: A generative model for raw audio.  _arXiv preprint arXiv:1609.03499_.
  * Oord, Kalchbrenner, and Kavukcuoglu (2016) Oord, A. v. d.; Kalchbrenner, N.; and Kavukcuoglu, K. 2016.  Pixel recurrent neural networks.  _arXiv preprint arXiv:1601.06759_.
  * Papamakarios et al. (2019) Papamakarios, G.; Nalisnick, E.; Rezende, D. J.; Mohamed, S.; and Lakshminarayanan, B. 2019.  Normalizing flows for probabilistic modeling and inference.  _arXiv preprint arXiv:1912.02762_.
  * Papamakarios, Pavlakou, and Murray (2017) Papamakarios, G.; Pavlakou, T.; and Murray, I. 2017.  Masked autoregressive flow for density estimation.  In _Advances in Neural Information Processing Systems_ , 2338–2347.
  * Perugachi-Diaz, Tomczak, and Bhulai (2021) Perugachi-Diaz, Y.; Tomczak, J. M.; and Bhulai, S. 2021.  Invertible densenets with concatenated lipswish.  _arXiv preprint arXiv:2102.02694_.
  * Rezende and Mohamed (2015) Rezende, D. J.; and Mohamed, S. 2015.  Variational inference with normalizing flows.  _arXiv preprint arXiv:1505.05770_.
  * Rudin (1976) Rudin, W. 1976.  International series in pure and applied mathematics.  _Principles of mathematical analysis_.
  * Rudin (2006) Rudin, W. 2006.  _Real and complex analysis_.  Tata McGraw-hill education.
  * Smolensky (1986) Smolensky, P. 1986.  Information processing in dynamical systems: Foundations of harmony theory.  Technical report, Colorado Univ at Boulder Dept of Computer Science.
  * Sundararajan, Taly, and Yan (2017) Sundararajan, M.; Taly, A.; and Yan, Q. 2017.  Axiomatic attribution for deep networks.  In _International Conference on Machine Learning_ , 3319–3328. PMLR.
  * Uria et al. (2016) Uria, B.; Côté, M.-A.; Gregor, K.; Murray, I.; and Larochelle, H. 2016.  Neural autoregressive distribution estimation.  _The Journal of Machine Learning Research_ , 17(1): 7184–7220.
  * Vahdat and Kautz (2020) Vahdat, A.; and Kautz, J. 2020.  NVAE: A Deep Hierarchical Variational Autoencoder.  _arXiv preprint arXiv:2007.03898_.
  * Van den Oord et al. (2016) Van den Oord, A.; Kalchbrenner, N.; Espeholt, L.; Vinyals, O.; Graves, A.; et al. 2016.  Conditional image generation with pixelcnn decoders.  In _Advances in neural information processing systems_ , 4790–4798.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017.  Attention is all you need.  In _Advances in neural information processing systems_ , 5998–6008.
  * Withers and Nadarajah (2010) Withers, C. S.; and Nadarajah, S. 2010.  log det A= tr log A.  _International Journal of Mathematical Education in Science and Technology_ , 41(8): 1121–1124.
  * Zhang et al. (2019) Zhang, H.; Goodfellow, I.; Metaxas, D.; and Odena, A. 2019.  Self-attention generative adversarial networks.  In _International Conference on Machine Learning_ , 7354–7363. PMLR.


$\displaystyle+2C^{2}DA-4BD^{2}A+2D^{3}A-14BA^{2}-2BCA^{2}+4DA^{2}$ (326)
$\displaystyle-4CDA^{2}-2BA^{3}+2DA^{3}$ (327) $\displaystyle c_{0}=$
$\displaystyle~{}B^{2}-2B^{4}-4C+4B^{2}C-10C^{2}-6B^{2}C^{2}-8C^{3}-2C^{4}+18BD$
(328)
$\displaystyle+4B^{3}D-24BCD-37D^{2}-8B^{2}D^{2}-16CD^{2}-6C^{2}D^{2}+4BD^{3}$
(329) $\displaystyle-2D^{4}+4A-16B^{2}A+40CA+16C^{2}A+4C^{3}A+32BDA$ (330)
$\displaystyle-8BCDA-8D^{2}A+8CD^{2}A-28A^{2}-6B^{2}A^{2}-16CA^{2}$ (331)
$\displaystyle-8C^{2}A^{2}+8BDA^{2}-8D^{2}A^{2}-4A^{3}+4CA^{3}-2A^{4}\ .$
(332)

Dropping the factor $(w-1)^{6}(w+1)^{6}/w^{6}$, rewriting the remaining
polynomial in terms of $x\equiv w+w^{-1}$ and computing its discriminant in
terms of $x$, we obtain the following double discriminant:

$\displaystyle\Delta=\Delta_{\text{unphys}}\Delta_{\text{phys}}$ (333)
$\displaystyle\Delta_{\text{unphys}}=65536A^{3}D^{4}\left(\sum_{m+n\leq
9}c_{m,n}B{}^{m}C{}^{n}\right)^{3}$ (334)
$\displaystyle\Delta_{\text{phys}}=\sum_{m+n\leq 9}d_{m,n}B{}^{m}C{}^{n}$
(335)

where

$\displaystyle c_{00}=$ $\displaystyle
D^{6}+84D^{8}-159D^{10}+D^{12}+39D^{6}A+186D^{8}A-96D^{10}A+123D^{6}A^{2}$
(336)
$\displaystyle+414D^{8}A^{2}-6D^{10}A^{2}-12D^{4}A^{3}+73D^{6}A^{3}+156D^{8}A^{3}-12D^{4}A^{4}-336D^{6}A^{4}$
(337)
$\displaystyle+15D^{8}A^{4}-39D^{4}A^{5}+93D^{6}A^{5}+48D^{2}A^{6}+114D^{4}A^{6}-20D^{6}A^{6}-156D^{2}A^{7}$
(338)
$\displaystyle-255D^{4}A^{7}-81D^{2}A^{8}+15D^{4}A^{8}-64A^{9}+87D^{2}A^{9}+48A^{10}-6D^{2}A^{10}+15A^{11}+A^{12},$
(339) $\displaystyle c_{01}=$
$\displaystyle-3(45D^{8}+18D^{10}-8D^{6}A+209D^{8}A+D^{10}A-4D^{4}A^{2}-144D^{6}A^{2}+94D^{8}A^{2}$
(340)
$\displaystyle-78D^{4}A^{3}-364D^{6}A^{3}-5D^{8}A^{3}-46D^{4}A^{4}-151D^{6}A^{4}+32D^{2}A^{5}+189D^{4}A^{5}$
(341)
$\displaystyle+10D^{6}A^{5}-34D^{2}A^{6}-48D^{4}A^{6}-122D^{2}A^{7}-{10}D^{4}A^{7}-64A^{8}+83D^{2}A^{8}+88A^{9}$
(342) $\displaystyle+5D^{2}A^{9}+4A^{10}-A^{11}),$ (343) $\displaystyle
c_{02}=$
$\displaystyle-3(D^{6}-7D^{8}+D^{10}+D^{4}A+{10}0D^{6}A+54D^{8}A+27D^{4}A^{2}+372D^{6}A^{2}-4D^{8}A^{2}$
(344)
$\displaystyle-34D^{4}A^{3}+146D^{6}A^{3}-24D^{2}A^{4}-447D^{4}A^{4}+6D^{6}A^{4}-92D^{2}A^{5}-195D^{4}A^{5}$
(345)
$\displaystyle+238D^{2}A^{6}-4D^{4}A^{6}+80A^{7}-21D^{2}A^{7}-156A^{8}+D^{2}A^{8}+16A^{9}),$
(346) $\displaystyle c_{03}=$ $\displaystyle
27D^{8}-3D^{4}A+135D^{6}A+3D^{8}A-150D^{4}A^{2}-207D^{6}A^{2}-24D^{2}A^{3}-1104D^{4}A^{3}$
(347)
$\displaystyle-D^{6}A^{3}-2{10}D^{2}A^{4}-447D^{4}A^{4}+846D^{2}A^{5}-15D^{4}A^{5}+160A^{6}+354D^{2}A^{6}$
(348) $\displaystyle-390A^{7}+21D^{2}A^{7}+30A^{8}-8A^{9},$ (349)
$\displaystyle c_{04}=$ $\displaystyle
3(D^{6}+D^{8}+2D^{4}A+33D^{6}A+D^{2}A^{2}+76D^{4}A^{2}+D^{6}A^{2}+7D^{2}A^{3}-63D^{4}A^{3}$
(350)
$\displaystyle-178D^{2}A^{4}-7D^{4}A^{4}-20A^{5}-67D^{2}A^{5}+64A^{6}+7D^{2}A^{6}+16A^{7}-2A^{8}),$
(351) $\displaystyle c_{05}=$ $\displaystyle
3(2D^{4}A+D^{6}A+2D^{2}A^{2}+39D^{4}A^{2}+44D^{2}A^{3}-4D^{4}A^{3}+4A^{4}-33D^{2}A^{4}$
(352) $\displaystyle-24A^{5}+D^{2}A^{5}-8A^{6}+2A^{7}),$ (353) $\displaystyle
c_{06}=$
$\displaystyle-D^{6}-3D^{4}A-6D^{4}A^{2}-A^{3}+51D^{2}A^{3}+21A^{4}-9D^{2}A^{4}-12A^{5}+8A^{6},$
(354) $\displaystyle c_{07}=$
$\displaystyle-3(D^{4}A+2D^{2}A^{2}+A^{3}+3D^{2}A^{3}-2A^{4}),$ (355)
$\displaystyle c_{08}=$ $\displaystyle-3(D^{2}A^{2}+A^{3}+A^{4}),$ (356)
$\displaystyle c_{09}=$ $\displaystyle-A^{3},$ (357) $\displaystyle c_{10}=$
$\displaystyle
3(-44D^{7}+173D^{9}+D^{11}-4D^{5}A-117D^{7}A+31D^{9}A-15D^{5}A^{2}-301D^{7}A^{2}$
(358)
$\displaystyle-5D^{9}A^{2}+35D^{5}A^{3}+92D^{7}A^{3}+32D^{3}A^{4}+135D^{5}A^{4}+{10}D^{7}A^{4}-97D^{3}A^{5}$
(359)
$\displaystyle-219D^{5}A^{5}-59D^{3}A^{6}-{10}D^{5}A^{6}-64DA^{7}+38D^{3}A^{7}+52DA^{8}+5D^{3}A^{8}$
(360) $\displaystyle+58DA^{9}-DA^{10}),$ (361) $\displaystyle c_{11}=$
$\displaystyle
3(D^{5}+146D^{7}+28D^{9}+21D^{5}A+500D^{7}A+2D^{9}A-246D^{5}A^{2}+99D^{7}A^{2}$
(362)
$\displaystyle-40D^{3}A^{3}-570D^{5}A^{3}-8D^{7}A^{3}+64D^{3}A^{4}+102D^{5}A^{4}+342D^{3}A^{5}+12D^{5}A^{5}$
(363)
$\displaystyle+144DA^{6}-127D^{3}A^{6}-272DA^{7}-8D^{3}A^{7}-102DA^{8}+2DA^{9}),$
(364) $\displaystyle c_{12}=$
$\displaystyle-3(-D^{5}+81D^{7}+D^{9}-165D^{5}A-49D^{7}A-16D^{3}A^{2}-765D^{5}A^{2}+7D^{7}A^{2}$
(365)
$\displaystyle-D^{3}A^{3}-138D^{5}A^{3}+735D^{3}A^{4}-27D^{5}A^{4}+128DA^{5}-63D^{3}A^{5}$
(366) $\displaystyle-483DA^{6}+29D^{3}A^{6}+7DA^{7}-{10}DA^{8}),$ (367)
$\displaystyle c_{13}=$
$\displaystyle-3(2D^{5}+20D^{7}+2D^{3}A+229D^{5}A+10D^{7}A+2D^{3}A^{2}-57D^{5}A^{2}-607D^{3}A^{3}$
(368)
$\displaystyle-22D^{5}A^{3}-56DA^{4}-7D^{3}A^{4}+392DA^{5}+14D^{3}A^{5}-91DA^{6}-2DA^{7}),$
(369) $\displaystyle c_{14}=$
$\displaystyle-3(2D^{5}+D^{7}+4D^{3}A+39D^{5}A+174D^{3}A^{2}-6D^{5}A^{2}+12DA^{3}-55D^{3}A^{3}$
(370) $\displaystyle-148DA^{4}-11D^{3}A^{4}+60DA^{5}+16DA^{6}),$ (371)
$\displaystyle c_{15}=$ $\displaystyle
3(D^{5}+6D^{5}A+DA^{2}-8D^{3}A^{2}-27DA^{3}+20D^{3}A^{3}+8DA^{4}-{10}DA^{5}),$
(372) $\displaystyle c_{16}=$ $\displaystyle
3(D^{5}+4D^{3}A+3DA^{2}+13D^{3}A^{2}+9DA^{3}+6DA^{4}),$ (373) $\displaystyle
c_{17}=$ $\displaystyle 3(2D^{3}A+3DA^{2}+6DA^{3}),$ (374) $\displaystyle
c_{18}=$ $\displaystyle 3DA^{2},$ (375) $\displaystyle c_{20}=$
$\displaystyle-3(-17D^{6}+199D^{8}-{10}1D^{6}A-69D^{8}A-15D^{4}A^{2}-175D^{6}A^{2}+D^{8}A^{2}$
(376)
$\displaystyle+99D^{4}A^{3}+215D^{6}A^{3}+39D^{4}A^{4}-4D^{6}A^{4}+56D^{2}A^{5}+12D^{4}A^{5}$
(377)
$\displaystyle-79D^{2}A^{6}+6D^{4}A^{6}-150D^{2}A^{7}+16A^{8}-4D^{2}A^{8}-8A^{9}+A^{10}),$
(378) $\displaystyle c_{21}=$ $\displaystyle
3(-142D^{6}+29D^{8}-7D^{4}A-393D^{6}A+10D^{8}A+159D^{4}A^{2}+5D^{6}A^{2}$
(379)
$\displaystyle+417D^{4}A^{3}-29D^{6}A^{3}+80D^{2}A^{4}-42D^{4}A^{4}-423D^{2}A^{5}+27D^{4}A^{5}$
(380) $\displaystyle-209D^{2}A^{6}+48A^{7}-7D^{2}A^{7}-26A^{8}-A^{9}),$ (381)
$\displaystyle c_{22}=$ $\displaystyle
3(D^{4}+176D^{6}+3D^{8}-48D^{4}A+32D^{6}A-636D^{4}A^{2}-42D^{2}A^{3}+78D^{4}A^{3}$
(382)
$\displaystyle+645D^{2}A^{4}-6D^{4}A^{4}-60D^{2}A^{5}-56A^{6}+31A^{7}+3A^{8}),$
(383) $\displaystyle c_{23}=$
$\displaystyle-3(-2D^{4}+9D^{6}-264D^{4}A+5D^{6}A-10D^{2}A^{2}+36D^{4}A^{2}+350D^{2}A^{3}$
(384)
$\displaystyle+27D^{4}A^{3}-153D^{2}A^{4}-32A^{5}-29D^{2}A^{5}+2A^{6}-3A^{7}),$
(385) $\displaystyle c_{24}=$
$\displaystyle-3(2D^{6}+D^{2}A+58D^{4}A-66D^{2}A^{2}+39D^{4}A^{2}-13D^{2}A^{3}+9A^{4}$
(386) $\displaystyle-4D^{2}A^{4}+32A^{5}+3A^{6}),$ (387) $\displaystyle
c_{25}=$
$\displaystyle-3(2D^{4}+3D^{2}A+12D^{4}A+60D^{2}A^{2}-A^{3}+21D^{2}A^{3}-28A^{4}+3A^{5}),$
(388) $\displaystyle c_{26}=$
$\displaystyle-3(D^{4}+3D^{2}A+8D^{2}A^{2}+7A^{3}-A^{4}),$ (389)
$\displaystyle c_{27}=$ $\displaystyle-3(D^{2}A-A^{3}),$ (390) $\displaystyle
c_{30}=$
$\displaystyle-3D^{5}+231D^{7}-8D^{9}-189D^{5}A-198D^{7}A-54D^{5}A^{2}+21D^{7}A^{2}-16D^{3}A^{3}$
(391)
$\displaystyle-201D^{5}A^{3}+237D^{3}A^{4}-15D^{5}A^{4}+591D^{3}A^{5}-144DA^{6}-D^{3}A^{6}+51DA^{7}$
(392) $\displaystyle+3DA^{8},$ (393) $\displaystyle c_{31}=$ $\displaystyle
3(43D^{5}-55D^{7}+151D^{5}A+2D^{7}A-2D^{3}A^{2}+37D^{5}A^{2}-323D^{3}A^{3}-14D^{5}A^{3}$
(394) $\displaystyle-181D^{3}A^{4}+104DA^{5}+22D^{3}A^{5}-44DA^{6}-10DA^{7}),$
(395) $\displaystyle c_{32}=$ $\displaystyle
3(-134D^{5}+3D^{7}+45D^{5}A+297D^{3}A^{2}+29D^{5}A^{2}-118D^{3}A^{3}-80DA^{4}-27D^{3}A^{4}$
(396) $\displaystyle+39DA^{5}-5DA^{6}),$ (397) $\displaystyle c_{33}=$
$\displaystyle
D^{3}+147D^{5}-237D^{3}A+60D^{5}A+120D^{3}A^{2}+78DA^{3}+40D^{3}A^{3}+69DA^{4}+60DA^{5},$
(398) $\displaystyle c_{34}=$ $\displaystyle
3(D^{3}+2D^{5}+85D^{3}A-3DA^{2}+11D^{3}A^{2}-56DA^{3}+7DA^{4}),$ (399)
$\displaystyle c_{35}=$ $\displaystyle 3(D^{3}+2D^{3}A+21DA^{2}-{10}DA^{3}),$
(400) $\displaystyle c_{36}=$ $\displaystyle D^{3}-9DA^{2},$ (401)
$\displaystyle c_{40}=$
$\displaystyle-3(-8D^{6}+2D^{8}-8D^{4}A+42D^{6}A-29D^{4}A^{2}-7D^{6}A^{2}-139D^{4}A^{3}+47D^{2}A^{4}$
(402) $\displaystyle+7D^{4}A^{4}-15D^{2}A^{5}-D^{2}A^{6}+4A^{7}-A^{8}),$ (403)
$\displaystyle c_{41}=$
$\displaystyle-3(2D^{4}+4D^{6}+64D^{4}A+16D^{6}A+33D^{4}A^{2}-62D^{2}A^{3}-11D^{4}A^{3}+39D^{2}A^{4}$
(404) $\displaystyle-6D^{2}A^{5}-12A^{6}+A^{7}),$ (405) $\displaystyle
c_{42}=$
$\displaystyle-3(-33D^{4}+3D^{6}+83D^{4}A+24D^{2}A^{2}-4D^{4}A^{2}-32D^{2}A^{3}+39D^{2}A^{4}+13A^{5}+2A^{6}),$
(406) $\displaystyle c_{43}=$ $\displaystyle
3(-36D^{4}+3D^{2}A+7D^{4}A+21D^{2}A^{2}+11D^{2}A^{3}+6A^{4}+2A^{5}),$ (407)
$\displaystyle c_{44}=$ $\displaystyle
3(D^{4}-21D^{2}A+18D^{2}A^{2}-A^{3}+A^{4}),$ (408) $\displaystyle c_{45}=$
$\displaystyle 3(3D^{2}A-A^{3}),$ (409) $\displaystyle c_{50}=$ $\displaystyle
3(-7D^{5}+2D^{7}+39D^{5}A-14D^{3}A^{2}+D^{5}A^{2}+11D^{3}A^{3}-4D^{3}A^{4}-12DA^{5}+DA^{6}),$
(410) $\displaystyle c_{51}=$
$\displaystyle-3(-26D^{5}-6D^{3}A+{10}D^{5}A+28D^{3}A^{2}-20D^{3}A^{3}-25DA^{4}-6DA^{5}),$
(411) $\displaystyle c_{52}=$
$\displaystyle-3(D^{3}+3D^{5}-14D^{3}A+21D^{3}A^{2}+16DA^{3}+12DA^{4}),$ (412)
$\displaystyle c_{53}=$ $\displaystyle-3(-7D^{3}+{10}D^{3}A-3DA^{2}-2DA^{3}),$
(413) $\displaystyle c_{54}=$ $\displaystyle-3(D^{3}-3DA^{2}),$ (414)
$\displaystyle c_{60}=$ $\displaystyle
3WD^{4}+8D^{6}+15D^{4}A-9D^{4}A^{2}-36D^{2}A^{3}-6D^{2}A^{4}-A^{6},$ (415)
$\displaystyle c_{61}=$ $\displaystyle
3(-7D^{4}+6D^{4}A+14D^{2}A^{2}+13D^{2}A^{3}+A^{5}),$ (416) $\displaystyle
c_{62}=$ $\displaystyle 3(D^{4}-3D^{2}A-8D^{2}A^{2}-A^{4}),$ (417)
$\displaystyle c_{63}=$ $\displaystyle-9D^{2}A+A^{3},$ (418) $\displaystyle
c_{70}=$ $\displaystyle-3(4D^{3}A+3D^{3}A^{2}+DA^{4}),$ (419) $\displaystyle
c_{71}=$ $\displaystyle 3(D^{3}+6D^{3}A+2DA^{3}),$ (420) $\displaystyle
c_{72}=$ $\displaystyle 3(D^{3}-DA^{2}),$ (421) $\displaystyle c_{80}=$
$\displaystyle-3(D^{4}+D^{2}A^{2}),$ (422) $\displaystyle c_{81}=$
$\displaystyle 3D^{2}A,$ (423) $\displaystyle c_{90}=$ $\displaystyle-D^{3}$
(424) $\displaystyle d_{0,0}=$ $\displaystyle
729D^{6}A+432D^{4}A^{2}+2187D^{6}A^{2}+864D^{4}A^{3}+2187D^{6}A^{3}$ (426)
$\displaystyle-1512D^{4}A^{4}+729D^{6}A^{4}-1024D^{2}A^{5}-1944D^{4}A^{5}-729D^{4}A^{6},$
(427) $\displaystyle d_{0,1}=$ $\displaystyle
3(216D^{4}A^{2}+1404D^{4}A^{3}+512D^{2}A^{4}+702D^{4}A^{4}-768D^{2}A^{5}+243D^{4}A^{5}),~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}$
(428) $\displaystyle d_{0,2}=$ $\displaystyle
3(-243D^{4}A^{2}-160D^{2}A^{3}+486D^{4}A^{3}+1120D^{2}A^{4}-504D^{2}A^{5}),$
(429) $\displaystyle d_{0,3}=$
$\displaystyle-27D^{4}A-16D^{2}A^{2}-540D^{4}A^{2}-984D^{2}A^{3}+216D^{4}A^{3}+2088D^{2}A^{4}$
(430) $\displaystyle-216D^{2}A^{5},$ (431) $\displaystyle d_{0,4}=$
$\displaystyle 8(-3D^{2}A^{2}-69D^{2}A^{3}-2A^{4}+27D^{2}A^{4}),$ (432)
$\displaystyle d_{0,5}=$ $\displaystyle 8(3D^{2}A^{2}+2A^{3}-4A^{4}),$ (433)
$\displaystyle d_{0,6}=$ $\displaystyle 16(D^{2}A^{2}+2A^{3}-A^{4}),$ (434)
$\displaystyle d_{0,7}=$ $\displaystyle 16A^{3},$ (435) $\displaystyle
d_{0,8}=$ $\displaystyle 0,$ (436) $\displaystyle d_{0,9}=$ $\displaystyle 0,$
(437) $\displaystyle d_{1,0}=$
$\displaystyle-9(108D^{5}A+540D^{5}A^{2}+128D^{3}A^{3}+513D^{5}A^{3}-256D^{3}A^{4}+81D^{5}A^{4}$
(438) $\displaystyle-288D^{3}A^{5}),$ (439) $\displaystyle d_{1,1}=$
$\displaystyle-9(-135D^{5}A-64D^{3}A^{2}-27D^{5}A^{2}+656D^{3}A^{3}+108D^{5}A^{3}+336D^{3}A^{4}$
(440) $\displaystyle-108D^{3}A^{5}),$ (441) $\displaystyle d_{1,2}=$
$\displaystyle-4(-513D^{3}A^{2}+297D^{3}A^{3}-32DA^{4}+243D^{3}A^{4}),$ (442)
$\displaystyle d_{1,3}=$
$\displaystyle-4(-9D^{3}A-90D^{3}A^{2}+32DA^{3}+54D^{3}A^{3}-60DA^{4}),$ (443)
$\displaystyle d_{1,4}=$
$\displaystyle-4(9D^{3}A+4DA^{2}+36D^{3}A^{2}+52DA^{3}-36DA^{4}),$ (444)
$\displaystyle d_{1,5}=$ $\displaystyle-16(4DA^{2}+9DA^{3}),$ (445)
$\displaystyle d_{1,6}=$ $\displaystyle-16DA^{2},$ (446) $\displaystyle
d_{1,7}=$ $\displaystyle 0$ (447) $\displaystyle d_{1,8}=$ $\displaystyle 0,$
(448) $\displaystyle d_{2,0}=$ $\displaystyle
6(36D^{4}A+540D^{4}A^{2}+423D^{4}A^{3}-128D^{2}A^{4}),$ (449) $\displaystyle
d_{2,1}=$
$\displaystyle-1566D^{4}A-351D^{4}A^{2}+1024D^{2}A^{3}+972D^{4}A^{3},$ (450)
$\displaystyle d_{2,2}=$ $\displaystyle
513D^{4}A-112D^{2}A^{2}+270D^{4}A^{2}-504D^{2}A^{3}-270D^{2}A^{4},$ (451)
$\displaystyle d_{2,3}=$ $\displaystyle
2(-4D^{2}A+365D^{2}A^{2}+135D^{2}A^{3}),$ (452) $\displaystyle d_{2,4}=$
$\displaystyle 2(23D^{2}A+72D^{2}A^{2}-4A^{3}),$ (453) $\displaystyle
d_{2,5}=$ $\displaystyle-8(D^{2}A-A^{2}-A^{3}),$ (454) $\displaystyle
d_{2,6}=$ $\displaystyle-8A^{2},$ (455) $\displaystyle d_{2,7}=$
$\displaystyle 0,$ (456) $\displaystyle d_{3,0}=$
$\displaystyle-27D^{5}-540D^{5}A-832D^{3}A^{2}+216D^{5}A^{2}+576D^{3}A^{3}-216D^{3}A^{4},$
(457) $\displaystyle d_{3,1}=$ $\displaystyle
12(24D^{3}A-43D^{3}A^{2}+18D^{3}A^{3}),$ (458) $\displaystyle d_{3,2}=$
$\displaystyle-591D^{3}A-270D^{3}A^{2}+64DA^{3},$ (459) $\displaystyle
d_{3,3}=$ $\displaystyle D^{3}+68D^{3}A-64DA^{2}-68DA^{3},$ (460)
$\displaystyle d_{3,4}=$ $\displaystyle 4(-2DA+17DA^{2}),$ (461)
$\displaystyle d_{3,5}=$ $\displaystyle 8DA,$ (462) $\displaystyle d_{3,6}=$
$\displaystyle 0,$ (463) $\displaystyle d_{4,0}=$
$\displaystyle-3(-9D^{4}-252D^{4}A+72D^{4}A^{2}+64D^{2}A^{3}),$ (464)
$\displaystyle d_{4,1}=$
$\displaystyle-4(9D^{4}+36D^{4}A-56D^{2}A^{2}-36D^{2}A^{3}),$ (465)
$\displaystyle d_{4,2}=$ $\displaystyle-2(-17D^{2}A+72D^{2}A^{2}),$ (466)
$\displaystyle d_{4,3}=$ $\displaystyle-D^{2}-68D^{2}A,$ (467) $\displaystyle
d_{4,4}=$ $\displaystyle D^{2}-A^{2},$ (468) $\displaystyle d_{4,5}=$
$\displaystyle A,$ (469) $\displaystyle d_{5,0}=$ $\displaystyle-200D^{3}A,$
(470) $\displaystyle d_{5,1}=$ $\displaystyle 36(D^{3}+4D^{3}A),$ (471)
$\displaystyle d_{5,2}=$ $\displaystyle-8(D^{3}-DA^{2}),$ (472) $\displaystyle
d_{5,3}=$ $\displaystyle-8DA,$ (473) $\displaystyle d_{5,4}=$
$\displaystyle-D,$ (474) $\displaystyle d_{6,0}=$ $\displaystyle
16(D^{4}-D^{2}A^{2}),$ (475) $\displaystyle d_{6,1}=$ $\displaystyle
16D^{2}A,$ (476) $\displaystyle d_{6,2}=$ $\displaystyle 8D^{2},$ (477)
$\displaystyle d_{6,3}=$ $\displaystyle 0,$ (478) $\displaystyle d_{7,0}=$
$\displaystyle-16D^{3},$ (479) $\displaystyle d_{7,1}=$ $\displaystyle
d_{7,2}=d_{8,0}=d_{8,1}=d_{9,0}=0.$ (480)

If we identify the coefficients in (303) as

$\displaystyle A=M^{-\frac{1}{2}}q^{-1},\quad B=-Uq^{-1},\quad
C=VM^{\frac{1}{2}}q^{-1},\quad D=-Mq^{-1},$ (481)

then the physical part of the discriminant $\Delta_{\text{phys}}$ agrees with
the double discriminant of the Seiberg-Witten curve for
SU(3)${}_{-\frac{1}{2}}+1\mathbf{F}$ given in (215) up to an overall factor
independent of $U$ and $V$.

## References

  * (1) O. Aharony, A. Hanany, and B. Kol, Webs of (p,q) five-branes, five-dimensional field theories and grid diagrams, JHEP 9801 (1998) 002, [hep-th/9710116].
  * (2) D. R. Morrison and N. Seiberg, Extremal transitions and five-dimensional supersymmetric field theories, Nucl.Phys. B483 (1997) 229–247, [hep-th/9609070].
  * (3) M. R. Douglas, S. H. Katz, and C. Vafa, Small instantons, Del Pezzo surfaces and type I-prime theory, Nucl.Phys. B497 (1997) 155–172, [hep-th/9609071].
  * (4) K. A. Intriligator, D. R. Morrison, and N. Seiberg, Five-dimensional supersymmetric gauge theories and degenerations of Calabi-Yau spaces, Nucl.Phys. B497 (1997) 56–100, [hep-th/9702198].
  * (5) N. A. Nekrasov, Seiberg-Witten prepotential from instanton counting, Adv.Theor.Math.Phys. 7 (2004) 831–864, [hep-th/0206161].
  * (6) N. Nekrasov and A. Okounkov, Seiberg-Witten theory and random partitions, Prog. Math. 244 (2006) 525–596, [hep-th/0306238].
  * (7) M. Aganagic, A. Klemm, M. Marino, and C. Vafa, The Topological vertex, Commun.Math.Phys. 254 (2005) 425–478, [hep-th/0305132].
  * (8) A. Iqbal, C. Kozcaz, and C. Vafa, The Refined topological vertex, JHEP 0910 (2009) 069, [hep-th/0701156].
  * (9) H. Awata and H. Kanno, Refined BPS state counting from Nekrasov’s formula and Macdonald functions, Int.J.Mod.Phys. A24 (2009) 2253–2306, [arXiv:0805.0191].
  * (10) H. Nakajima and K. Yoshioka, Instanton counting on blowup. 1., Invent. Math. 162 (2005) 313–355, [math/0306198].
  * (11) H. Nakajima and K. Yoshioka, Instanton counting on blowup. II. K-theoretic partition function, math/0505553.
  * (12) L. Gottsche, H. Nakajima, and K. Yoshioka, K-theoretic Donaldson invariants via instanton counting, Pure Appl. Math. Quart. 5 (2009) 1029–1111, [math/0611945].
  * (13) C. A. Keller and J. Song, Counting Exceptional Instantons, JHEP 07 (2012) 085, [arXiv:1205.4722].
  * (14) M.-x. Huang, K. Sun, and X. Wang, Blowup Equations for Refined Topological Strings, JHEP 10 (2018) 196, [arXiv:1711.09884].
  * (15) J. Kim, S.-S. Kim, K.-H. Lee, K. Lee, and J. Song, Instantons from Blow-up, JHEP 11 (2019) 092, [arXiv:1908.11276]. [Erratum: JHEP 06, 124 (2020)].
  * (16) H.-C. Kim, M. Kim, S.-S. Kim, and K.-H. Lee, Bootstrapping BPS spectra of 5d/6d field theories, JHEP 04 (2021) 161, [arXiv:2101.00023].
  * (17) N. Seiberg and E. Witten, Electric - magnetic duality, monopole condensation, and confinement in N=2 supersymmetric Yang-Mills theory, Nucl. Phys. B 426 (1994) 19–52, [hep-th/9407087]. [Erratum: Nucl.Phys.B 430, 485–486 (1994)].
  * (18) N. Seiberg and E. Witten, Monopoles, duality and chiral symmetry breaking in N=2 supersymmetric QCD, Nucl. Phys. B 431 (1994) 484–550, [hep-th/9408099].
  * (19) K. Sakai, $E_{n}$ Jacobi forms and Seiberg–Witten curves, Commun. Num. Theor. Phys. 13 (2019) 53–80, [arXiv:1706.04619].
  * (20) C. Closset, S. Giacomelli, S. Schafer-Nameki, and Y.-N. Wang, 5d and 4d SCFTs: Canonical Singularities, Trinions and S-Dualities, JHEP 05 (2021) 274, [arXiv:2012.12827].
  * (21) N. Haouzi and J. Oh, On the Quantization of Seiberg-Witten Geometry, JHEP 01 (2021) 184, [arXiv:2004.00654].
  * (22) C. Closset and H. Magureanu, The $U$-plane of rank-one 4d $\mathcal{N}=2$ KK theories, SciPost Phys. 12 (2022), no. 2 065, [arXiv:2107.03509].
  * (23) Q. Jia and P. Yi, Aspects of 5d Seiberg-Witten theories on $\mathbb{S}^{1}$, JHEP 02 (2022) 125, [arXiv:2111.09448].
  * (24) A. Brini and K. Osuga, Five-dimensional gauge theories and the local B-model, Lett. Math. Phys. 112 (2022), no. 3 44, [arXiv:2110.11638].
  * (25) F. Del Monte and P. Longhi, The threefold way to quantum periods: WKB, TBA equations and q-Painlevé, arXiv:2207.07135.
  * (26) K. Sakai, E-strings, $F_{4}$, and $D_{4}$ triality, arXiv:2304.04878.
  * (27) B. Haghighat and W. Yan, M-strings in thermodynamic limit: Seiberg-Witten geometry, arXiv:1607.07873.
  * (28) B. Haghighat, J. Kim, W. Yan, and S.-T. Yau, D-type fiber-base duality, JHEP 09 (2018) 060, [arXiv:1806.10335].
  * (29) X. Li and F. Yagi, Thermodynamic limit of Nekrasov partition function for 5-brane web with O5-plane, JHEP 06 (2021) 004, [arXiv:2102.09482].
  * (30) J. Chen, B. Haghighat, H.-C. Kim, and M. Sperling, Elliptic quantum curves of class ${\mathcal{S}}_{k}$, JHEP 03 (2021) 028, [arXiv:2008.05155].
  * (31) J. Chen, B. Haghighat, H.-C. Kim, M. Sperling, and X. Wang, E-string quantum curve, Nucl. Phys. B 973 (2021) 115602, [arXiv:2103.16996].
  * (32) J. Chen, B. Haghighat, H.-C. Kim, K. Lee, M. Sperling, and X. Wang, Elliptic quantum curves of 6d SO(N) theories, JHEP 03 (2022) 154, [arXiv:2110.13487].
  * (33) J. Chen, Y. Lü, and X. Wang, D-type Minimal Conformal Matter: Quantum Curves, Elliptic Garnier Systems, and the 5d Descendants, arXiv:2304.04383.
  * (34) L. Bao, V. Mitev, E. Pomoni, M. Taki, and F. Yagi, Non-Lagrangian Theories from Brane Junctions, JHEP 1401 (2014) 175, [arXiv:1310.3841].
  * (35) S.-S. Kim and F. Yagi, 5d En Seiberg-Witten curve via toric-like diagram, JHEP 06 (2015) 082, [arXiv:1411.7903].
  * (36) H. Hayashi, S.-S. Kim, K. Lee, and F. Yagi, Discrete theta angle from an O5-plane, JHEP 11 (2017) 041, [arXiv:1707.07181].
  * (37) H. Hayashi, S.-S. Kim, Y. Sugimoto, X.-Y. Wei, and F. Yagi, Seiberg-Witten curves for 6d SCFTs and Little string theories from 5-brane webs, (in progress) (2023).
  * (38) F. Benini, S. Benvenuti, and Y. Tachikawa, Webs of five-branes and N=2 superconformal field theories, JHEP 0909 (2009) 052, [arXiv:0906.0359].
  * (39) A. Hanany and B. Kol, On orientifolds, discrete torsion, branes and M theory, JHEP 06 (2000) 013, [hep-th/0003025].
  * (40) G. Zafrir, Brane webs and $O5$-planes, JHEP 03 (2016) 109, [arXiv:1512.08114].
  * (41) A. Sen, F theory and orientifolds, Nucl.Phys. B475 (1996) 562–578, [hep-th/9605150].
  * (42) O. Bergman and G. Zafrir, 5d fixed points from brane webs and O7-planes, JHEP 12 (2015) 163, [arXiv:1507.03860].
  * (43) H. Hayashi, S.-S. Kim, K. Lee, M. Taki, and F. Yagi, A new 5d description of 6d D-type minimal conformal matter, JHEP 08 (2015) 097, [arXiv:1505.04439].
  * (44) H. Hayashi, S.-S. Kim, K. Lee, and F. Yagi, 5-brane webs for 5d $\mathcal{N}$ = 1 G2 gauge theories, JHEP 03 (2018) 125, [arXiv:1801.03916].
  * (45) A. Sen, Stable nonBPS states in string theory, JHEP 06 (1998) 007, [hep-th/9803194].
  * (46) A. Sen, Stable nonBPS bound states of BPS D-branes, JHEP 08 (1998) 010, [hep-th/9805019].
  * (47) A. Kapustin, D(n) quivers from branes, JHEP 12 (1998) 015, [hep-th/9806238].
  * (48) A. Hanany and A. Zaffaroni, Issues on orientifolds: On the brane construction of gauge theories with SO(2n) global symmetry, JHEP 07 (1999) 009, [hep-th/9903242].
  * (49) N. Seiberg, Five-dimensional SUSY field theories, nontrivial fixed points and string dynamics, Phys.Lett. B388 (1996) 753–760, [hep-th/9608111].
  * (50) P. Jefferson, S. Katz, H.-C. Kim, and C. Vafa, On Geometric Classification of 5d SCFTs, JHEP 04 (2018) 103, [arXiv:1801.04036].
  * (51) H. Hayashi, S.-S. Kim, K. Lee, and F. Yagi, Dualities and 5-brane webs for 5d rank 2 SCFTs, JHEP 12 (2018) 016, [arXiv:1806.10569].
  * (52) P. Jefferson, H.-C. Kim, C. Vafa, and G. Zafrir, Towards Classification of 5d SCFTs: Single Gauge Node, arXiv:1705.05836.
  * (53) K. Landsteiner and E. Lopez, New curves from branes, Nucl. Phys. B 516 (1998) 273–296, [hep-th/9708118].
  * (54) H. Hayashi, S.-S. Kim, K. Lee, M. Taki, and F. Yagi, More on 5d descriptions of 6d SCFTs, JHEP 10 (2016) 126, [arXiv:1512.08239].
  * (55) L. Bao, E. Pomoni, M. Taki, and F. Yagi, M5-Branes, Toric Diagrams and Gauge Theory Duality, JHEP 1204 (2012) 105, [arXiv:1112.5228].
  * (56) L. Bhardwaj, On the classification of 5d SCFTs, JHEP 09 (2020) 007, [arXiv:1909.09635].
  * (57) M.-X. Huang, A. Klemm, and M. Poretschkin, Refined stable pair invariants for E-, M- and $[p,q]$-strings, JHEP 1311 (2013) 112, [arXiv:1308.0619].
  * (58) S.-S. Kim, X. Li, S. Nawata, and F. Yagi, in progress, .
  * (59) H.-C. Kim, M. Kim, S.-S. Kim, and G. Zafrir, in progress, .
  * (60) E. Witten, Solutions of four-dimensional field theories via M theory, Nucl. Phys. B500 (1997) 3–42, [hep-th/9703166].
  * (61) K. Landsteiner, E. Lopez, and D. A. Lowe, N=2 supersymmetric gauge theories, branes and orientifolds, Nucl. Phys. B507 (1997) 197–226, [hep-th/9705199].
  * (62) A. Brandhuber, J. Sonnenschein, S. Theisen, and S. Yankielowicz, M theory and Seiberg-Witten curves: Orthogonal and symplectic groups, Nucl. Phys. B504 (1997) 175–188, [hep-th/9705232].