# Integrated Dataset of Brazilian Flights

Claudio Teixeira
CEFET/RJ
<EMAIL_ADDRESS>
Lucas Giusti
CEFET/RJ
<EMAIL_ADDRESS>
Jorge Soares
CEFET/RJ
<EMAIL_ADDRESS>
Joel dos Santos
CEFET/RJ
<EMAIL_ADDRESS>
Glauco Amorim
CEFET/RJ
<EMAIL_ADDRESS>
Eduardo Ogasawara
CEFET/RJ
<EMAIL_ADDRESS>

###### Abstract

The Brazilian commercial aviation system achieved the first position among
Latin American countries and the fifteenth place worldwide on the Revenue
Passenger-Kilometer (RPK) ranking. The availability of data regarding flight,
including flight information and meteorological conditions, enables studies
about the Brazilian flight system, such as flight delays and timetabling.
Therefore, this paper contributes to such studies by offering an integrated
dataset containing data on departure and arrival for flights departing and
arriving on Brazilian airports comprising the period from $2000$ to $2019$.
This paper presents a dataset composed of $15,505,922$ records of flight data,
each containing $45$ attributes. The attributes include data regarding the
airline, flight, airports, meteorological conditions, scheduled and elapsed
times for departure and arrival.

_K_ eywords Flight delays $\cdot$ Commercial aviation $\cdot$ Brazilian system

##  1 Introduction

The Brazilian commercial aviation system contains more than one hundred
airports. It transported 95.9 million revenue passengers during $2014$. It
achieved the first position among Latin American countries and the fifteenth
place worldwide on the Revenue Passenger-Kilometer (RPK) ranking 3. The
commercial aviation network in Brazil is organized towards regional hubs in
contrast to airline hubs. The main reason is the Brazilian territorial
extension and that few Brazilian states have more than one major airport. One
exception to this rule is Campinas (in the state of São Paulo), where airline
company Azul holds $77\%$ of its commercial flights. Besides, due to market
deregulation instituted in $2005$, the Brazilian commercial aviation system
experienced significant changes in its players, leading to market share
changes and flight availability.

The National Civil Aviation Agency (ANAC) is responsible for regulating and
supervising the Brazilian civil aviation activities. Since $2000$, ANAC keeps
track of departure and arrival data for Brazilian flights in its Active
Regular Flight (VRA) dataset 1. The data available in VRA are registered by
the airlines and consolidated by ANAC. It contains data about each flight
stage, _i.e._ , the aircraft’s necessary steps from its takeoff to the next
landing. These steps are established regardless of where the object of
transport has been loaded or unloaded. For each flight step, VRA provides data
such as airline, flight number, type (such as international, domestic, and
cargo), class (such as regular, extra, charter, and instruction), airports,
and scheduled and elapsed times for departure and arrival. ANAC monthly
provides VRA data on its webpage.

VRA enables studying the Brazilian commercial aviation system. Examples of
studies are flight delay patterns 7 and their prediction 4, 5. Although
meteorological conditions play an essential role in analyzing flight
information, such data is not present in VRA. Thus, this paper presents a
dataset that integrates Brazilian flight data. It fuses all monthly data
available in VRA. It enriches it with meteorological data from the ASOS
(Automated Surface Observing Systems) dataset 2 provided by the IOWA
University in the USA. ASOS contains weather sensor data from airports around
the world. During the entire data integration, data cleaning and data
preprocessing techniques were also applied to improve its quality.

##  2 Data Acquisition

According to the flight regulation of ANAC, commercial airline companies must
register flight metadata indicating changes in flight time, either delay,
anticipation, or canceling. They have to log the time a flight happened and a
justification for the alteration. Table 1 indicates the flight metadata
together with their semantics.

Table 1: Flight metadata registered by airline companies available in VRA Attribute |  Description
---|---
Airline |  ICAO code representing the airline company
Flight |  Flight number
Authorization code |  Identifies the authorization type for each flight step
Flight type |  Identifies the type of operation performed
Origin |  ICAO code of origin airport
Destination |  ICAO code of destination airport
Expected Departure |  Date and time of scheduled departure
Real departure |  Date and time of departure performed informed by the airline
Estimated Arrival |  Date and time of estimated arrival
Real Arrival |  Date and time of arrival, informed by the airline
Flight status |  Informs if the flight was performed or canceled
Justification Code |  Identifies the delay, cancellation, and other changes concerning the planned flight

According to the regulation of ANAC, the metadata indicated in Table 2 must be
registered in a paper form, either typed or handwritten. ANAC then consolidate
the data sent by the airline companies into the VRA dataset. VRA is published
monthly, comprising all flight steps expected to depart in a given month.

The primary goal of ANAC is to use the recorded metadata to compute the
punctuality rate of airlines. Thus, sector regulation obliges airline
companies to provide the data presented in Table 1. Therefore, it comprises
all flight steps that took place in a given period. However, around 20% of the
records may be considered inconsistent due to errors while filling the report
form. As will be presented in Section 3.1, the causes of errors include
arrival time before departure or flight duration inconsistent with the
regulation of ANAC.

Meteorological conditions play an important role in aviation operations. The
Automated Surface Observing Systems (ASOS) is a program that involves several
American government agencies. It was created to become an official network of
meteorological information to support primarily aviation entities. It includes
meteorological, climatological, and hydrological components. ASOS data come
from weather sensors in locations all over the planet. In Brazil, ASOS covers
all $154$ airports available in VRA, as seen in Figure 1.

Figure 1: Brazilian airports included in the ASOS dataset 2

The Department of Agronomy at Iowa State University, in the United States,
compiles daily information from the US ASOS system. It creates an hourly
report of meteorological observations in all of its sites. Table 2 indicates
the meteorological data together with their semantics.

Table 2: ASOS meteorological data Attribute |  Description
---|---
Sky condition |  Cloud height and amount (clear, scattered, broken, overcast) up to 12,000 feet
Visibility |  To at least ten statute miles
Weather |  Type and intensity for rain, snow, and freezing rain.
Obstructions to vision |  fog, haze
Pressure |  Sea-level pressure, altimeter setting
Temperature |  Ambient and dew point temperature
Wind |  Direction, speed, and character (gusts, squalls)
Precipitation accumulation |

##  3 Integrated Dataset

The integrated _Brazilian Flight Dataset_ (BFD) presented in this paper
includes both the flight data present in VRA and meteorological information
present in ASOS. It is intended to enable studies regarding the Brazilian
commercial aviation system. BFD is composed of $15,505,922$ records of flight
data, each containing $45$ attributes. The dataset, together with its
integration process description and R scripts, is available on IEEE
DataPort111Dataset is available at http://dx.doi.org/10.21227/k10b-qn21.
Additional information can be found at 8..

Figure 2: The data model for the BFD

Figure 2 presents the data model of BFD. It is detailed in the following
sections. As can be seen, BFD aggregates data from VRA and ASOS for flight
information and meteorological information, respectively. It also includes
data currently unavailable in VRA, such as describing the justification codes
of ANAC, airline and airport names, and ISO codes for country names.

BFD focus on flight data regarding flights that departed or arrived in Brazil.
When both origin and destination airports are located in Brazil, those flights
are considered domestic flights. Conversely, when either the origin or the
destination airport is located outside of Brazil, it is considered
international. The data integration process for creating BFD was organized
into three main activities: (i) data preprocessing, (ii) data enrichment, and
(iii) data fusion. Those activities resemble the traditional Extraction,
Transformation, and Load (ETL) process 10.

###  3.1 Data Preprocessing

The preprocessing stage was performed in three parts. First, VRA attribute
names were translated from Brazilian Portuguese to English. It was unnecessary
to translate the acronyms used in each variable since they were already
following the International Civil Aviation Organization (ICAO) standards. It
was necessary to convert temperature and dew point data to the International
System of Units regarding the ASOS data. Data from ASOS was filtered to
consider the $154$ airports available in VRA.

The second part consisted of data cleaning for both VRA and ASOS datasets.
Given that flight information is usually recorded by hand, VRA data was
cleaned to remove inconsistent data. During cleaning, records with missing
variables were removed. Also, records with departure time (either elapsed or
expected) greater or equal to arrival time were removed. They corresponded to
approximately $0.02\%$ of the records. Approximately $3.77\%$ of VRA records
were removed for being out of BFD scope, _i.e._ , with origin and destination
out of Brazil. Finally, the regulation of ANAC prohibits delays higher than
$24$ hours. Thus, during cleaning records with departure or arrival delays
exceeding this norm were removed. The complete data cleaning removed $21.07\%$
of VRA records.

The third part of the preprocessing stage consisted of removing outliers. For
each pair of airports $\langle o,d\rangle$ in VRA, it was considered both the
expected and elapsed duration of a flight from origin $o$ and destination $d$.
Flights whose duration (either elapsed or expected) were not in the interval
$[Q_{1}-3\cdot IQR,Q_{3}+3\cdot IQR]$ were considered as outliers. They
corresponded to 2.76% of VRA records. The preprocessing step resulted in
$15,505,922$ flight records from VRA to be used in the fusion stage.

###  3.2 Data Enrichment

After preprocessing, the dataset is enriched as follows. The dataset schema is
changed by separating departure and arrival data attributes (see Table 1 into
an hour and date attributes. Besides, it included attributes related to flight
duration, departure and arrival delays.

Additionally, two discrete attributes were included for the time of the day
for departures and arrivals. It divides the time attribute into seven ranges,
as presented in Table 3.

Table 3: Time attribute discretization Period | Start Time | End Time
---|---|---
Night | 23:00 | 04:00
Early Morning | 05:00 | 08:00
Mid Morning | 09:00 | 10:00
Late Morning | 11:00 | 12:00
Afternoon | 13:00 | 16:00
Early Evening | 17:00 | 19:00
Late Evening | 20:00 | 22:00

Two discrete attributes are included in ASOS while enriching the dataset. The
use the wind velocity in knots to include the wind intensity using a Beaufort
Scale. The second uses the wind direction in degrees to include the wind
direction using Wind Rose with 16 cardinal directions (N, NNE, NE, ENE, E,
ESE, SE, SSE, S, SSW, SW, WSW, W, WNW, NW and NNW)222Wind Rose Data - US
Department of Agriculture - Natural Resources Conservation Service (NRCS)
available at https://www.wcc.nrcs.usda.gov/climate/windrose.html.

###  3.3 Data Fusion

Data fusion was applied over VRA data from 2000 to 2019, except for June, July
2014, and March 2018, when ANAC did not collect the data. It is worth
mentioning that ASOS provides hourly meteorological data.

During the fusion process for the meteorological and flight data, it was
necessary to group all flight data in a given hour. The grouping was performed
for each elapsed departure and arrival of the flight to determine its
meteorological information.

Furthermore, the fusion stage resolved airport and airline names from VRA
data. It also included an ISO code for country names whenever the flight
departs or arrives at a non-Brazilian airport. Finally, the justification
codes for flight delay were also expanded to their descriptions.

##  4 Dataset Usage

BFD allows for studies regarding the Brazilian commercial aviation system. In
this section, we present previous and ongoing work conducted on top of BFD
together with an exploratory analysis of BFD data. To present the importance
of using the database, we conduct an exploratory analysis and mention studies
that used the data in their research.

As discussed before, the Brazilian flight system is oriented towards regional
hubs instead of company hubs. Figure 3 presents the number of flights per
airport, considering just the 25 biggest airports on flights. It also divides
flights into domestic (D), international (I), and cargo (C) flights.

As can be seen in Figure 3, in the top five busiest airports, the first two
are in São Paulo (SBSP and SBGR), the third in Brasília (SBBR), and the last
two in Rio de Janeiro (SBGL and SBRJ). Rio and São Paulo are the two higher
Gross Domestic Products (GDPs) in Brazil. They are two major gateways for
flights coming and exiting Brazil. Approximately one-third of the flight in
Guarulhos Airport (SBGR) and Galeão Airport (SBGL) are international flights.

Figure 3: Number of flights per airport, for the top-25 most active airports

Brasilia is the capital of the country and is located in the middle of Brazil.
It acts as a hub for flights from and to cities in the north and northeast
regions. It can be seen, however, that it has few international flights.

Brazil and Argentina have strong touristic relations. Thus we can see the
Buenos Aires international airport (SAEZ) in the top-25 busiest airports.
Since BFD has only flights from and to Brazil, SAEZ has only international and
cargo flights.

Figure 4 presents the takeoff and arrival delay per airport for the top-25
busiest airports. It indicates whether an airport has recover capabilities for
arrival delays. The radius of the airport also indicates the level of
punctuality. The higher the radius, the airports are more punctual.

Figure 4: Mean takeoff delay and punctuality rate per mean arrival delay for
the top-25 busiest airports

Figure 5 presents the distribution of flights according to the period of the
day. As shown, most of the flight departures (Figure 5.a) occur in the
afternoon and early evening. Most arrivals (Figure 5.b) occur in the afternoon
and early morning. During the mid and late morning, the number of flights
decreases significantly for both departure and arrival.

Figure 5: Number of flights per period of the day: (a) departure; (b) arrival

According to ANAC regulation, a flight is considered to be delayed when its
departure or arrival time surpasses, respectively, the expected departure or
arrival by more than $30$ minutes. Figure 6 presents the punctuality rate
considering all the Brazilian flight systems per year, from $2000$ to $2019$.
It is possible to observe that the Brazilian flight crises that occurred in
$2007$ interfered with both punctuality rates and mean delay 9.

Figure 6: Punctuality rate and mean delay per year. The charts present the
mean delay together with its confidence interval of $95\%$

Figure 7 analysis of the Brazilian systems monthly. Historically, months of
school break (December, January, and July) have the lowest punctuality rates
and the highest mean delay. August is the month with the highest level of
punctuality and lowest mean delay.

Figure 7: Punctuality rate and mean delay per month of the year

Finally, Figure 8 presents the punctuality rate (circle size) and the average
delay in minutes per number of flights for the top-25 companies. According to
Figure 8, two airlines present the most significant number of flights, TAM and
Gol (GLO). It is also possible to observe that airlines with lower punctuality
rates tend to have a higher mean delay.

Figure 8: Mean delay and Punctuality rate per number of flights for the top-25
airline companies

Given the various inconveniences for airlines, airports, and passengers caused
by flight delays, it is fundamental to mitigate their occurrence and optimize
an air transport system’s decision-making process. Mainly, airlines, airports,
and users may be more interested in when delays are likely to occur than the
accurate prediction of the absence of delays. In that context, Moreira et al.
4 use BFD to analyze Flight delays in the period between $2009$ and $2015$.
The authors present a classification model capable of predicting delays,
getting about $60\%$ of hits.

Flight delays fall into two main categories: root delay and delay propagation.
Root delays are related to events that are intrinsic to a particular flight.
In delay propagation, it is presumed that a delay has already occurred at some
point in the network, _i.e._ , new delays occur due to previous delays. The
understanding of delay propagation patterns among airports is essential for
decision-making processes.

That study may devise patterns in flight delays and the way the system recover
from it. Focusing on unveiling those patterns, Sternberg et al. 6 apply data
indexing techniques combined with BFD data association rules. The authors
observed that the Brazilian flight system has difficulties recovering from
previous delay when operating under adverse meteorological conditions, when
delays occurrences may increase up to $216\%$.

##  5 Conclusion

This work aimed to create a reliable and enriched database on national and
international flights that arrived and departed from Brazilian airports. With
the data offered by this database, it is possible to carry out several studies
to aid the decision-making process. For example, it is possible to answer the
following questions: (i) “Which airport suffers the most delays?”; (ii) “What
month of the year is an airport most likely to be delayed?”; or (iii) “What
part of the day is a particular airport most likely to experience a delay in
departure?”

The answers to these questions can help companies and governments review their
protocols and optimize their services. Additionally, we intend to update this
dataset yearly, conducting the entire data integration.

## Acknowledgments

The authors thank CNPq, CAPES (finance code 001), FAPERJ, and CEFET/RJ for
partially funding this research.

## Conflict of interest

On behalf of all authors, the corresponding author states that there is no
conflict of interest.

## Author’s contributions

All authors contributed equally to the study. EO conceptualized the study
design. CT acquired the data. LT and JS conducted data analysis and
interpretation. Furthermore JAS and GA revised it critically for intellectual
content. All authors have approval of the final version.

## References

  * ANAC (2015) ANAC.  Agência Nacional de Aviação Civil.  Technical report, https://www.gov.br/anac/pt-br, 2015.
  * ASOS (2000) ASOS.  Automated Surface Observing System.  Technical report, https://mesonet.agron.iastate.edu/ASOS/, 2000\.
  * ICAO (2015) ICAO.  Annual Report of the Council 2014.  Technical report, http://www.icao.int/annual-report-2014/Pages/default.aspx, 2015.
  * Moreira et al. (2018) L. Moreira, C. Dantas, L. Oliveira, J. Soares, and E. Ogasawara.  On Evaluating Data Preprocessing Methods for Machine Learning Models for Flight Delays.  In _Proceedings of the International Joint Conference on Neural Networks_ , volume 2018-July, 2018.
  * Scarpel and Pelicioni (2018) R. A. Scarpel and L. Pelicioni.  A data analytics approach for anticipating congested days at the São Paulo International Airport.  _Journal of Air Transport Management_ , 72:1–10, 2018.
  * Sternberg et al. (2016a) A. Sternberg, D. Carvalho, L. Murta, J. Soares, and E. Ogasawara.  An analysis of Brazilian flight delays based on frequent patterns.  _Transportation Research Part E: Logistics and Transportation Review_ , 95:282–298, 2016a.
  * Sternberg et al. (2016b) A. Sternberg, D. Carvalho, L. Murta, J. Soares, and E. Ogasawara.  Experimental Evaluation.  Technical report, https://eic.cefet-rj.br/~dal/an-analysis-of-brazilian-flight-delays-based-on-frequent-patterns/, 2016b.
  * Teixeira et al. (2020) C. Teixeira, L. Teixeira, J. dos Santos, G. Amorim, J. Soares, and E. Ogasawara.  Integrated Brazilian Flight Datasets Description.  Technical report, https://eic.cefet-rj.br/~dal/brazilian-flight-dataset-description, 2020\.
  * Times (2007) N. Y. Times.  Brazil Demands Solution to Aviation Crisis.  Technical report, https://www.nytimes.com/2007/07/19/world/americas/19brazil.html, 2007.
  * Vassiliadis (2009) P. Vassiliadis.  A survey of extract-transform-load technology.  _International Journal of Data Warehousing and Mining_ , 5(3):1–27, 2009.


# Quasiparticle metamorphosis in the random $t$-$J$ model

Aman Kumar  Department of Theoretical Physics, Tata Institute of Fundamental
Research, Homi Bhabha Road, Navy Nagar, Mumbai 400005, India  Subir Sachdev
Department of Physics, Harvard University, Cambridge MA-02138, USA  School of
Natural Sciences, Institute for Advanced Study, Princeton, NJ-08540, USA
Vikram Tripathi  Department of Theoretical Physics, Tata Institute of
Fundamental Research, Homi Bhabha Road, Navy Nagar, Mumbai 400005, India

(August 27, 2024)

###### Abstract

Motivated by the pseudogap-Fermi liquid transition in doped Mott insulators,
we examine the excitations of a $t$-$J$ model with random and all-to-all
hopping and exchange. The stability of quasiparticles such as spin-$1/2$
fermions, spin-$1$ magnons, and emergent Jordan-Wigner (JW) spinless fermions
is cast as a problem of localization in the many-body Hilbert space, which is
studied by the FEAST eigensolver algorithm. At low dopings, magnons and JW
fermions are stable and better defined than spin-$1/2$ fermions, which are
unstable. Upon crossing a critical value of doping around $p_{c}=1/3$, their
stabilities are interchanged. Near the critical doping, these quasiparticles
are all found to be ill-defined. The critical point is thus associated with a
localization transition in the many-body Hilbert space.

Doped Mott insulators are central to understanding the unusual normal state of
high-temperature superconductors, particularly the cuprates. The low hole
doping pseudogap phase of the cuprates is characterised by collinear
antiferromagnetic order, low carrier concentration with poor metallic
conduction, a small Fermi surface Frachet et al. (2020), and an anomalously
large thermal Hall conductivity even in the absence of doping Grissonnanche et
al. (2019). At large hole doping, the system is a paramagnetic Landau Fermi
liquid with a large Fermi surface and spinful fermionic quasiparticles. In the
presence of magnetic order, spin-$1$ magnons are natural excitations Arrachea
and Rozenberg (2002), but the collinear magnetic order and the low hole
concentration in the underdoped phase make it unlikely that magnons or holes
are the mechanism for the observed large thermal Hall conductivity
Grissonnanche et al. (2019, 2020); Samajdar et al. (2019); Dalla Piazza et al.
(2015). Instead, in analogy with the Kitaev model Kitaev (2006) where emergent
free (Jordan-Wigner) Majorana fermions have been predicted to result in a
quantized thermal Hall effect Yokoi et al. (2021); Kasahara et al. (2018), and
reportedly confirmed in a Mott insulating Kitaev material
$\alpha$-RuCl${}_{3},$ it has been suggested that the large thermal Hall
effect in the cuprates Grissonnanche et al. (2020) could similarly arise from
phonons coupling to some emergent fermionic excitations in the pseudogap
phase, including at zero doping. At a certain critical value of doping,
magnons are not well-defined since magnetic order vanishes before this doping
is reached Frachet et al. (2020), and neither are Landau quasiparticles since
transport properties Varma (2020); Cha et al. (2020); Guo et al. (2020) are
inconsistent with a Landau Fermi liquid. These observations motivate us to
pose the question of stability of different quasiparticles across doping
regimes of a Mott insulator. We identify quasiparticle metamorphosis upon
varying the doping with many-body localization transitions in the Hilbert
space.

We shall study the fully-connected and random $t$-$J$ model, which has all-to-
all hopping $t_{ij}$ and exchange $J_{ij}$ realized as independent random
variables with zero mean. At zero doping, the model with SU($M$) symmetry, and
$M$ large, realizes a Sachdev-Ye-Kitaev model of fractionalized fermionic
spinons Sachdev and Ye (1993); Kitaev (2015). We are interested here in the
case with SU(2) symmetry and non-zero doping: this has emerged a useful
theoretical paradigm for studying the normal state properties of doped Mott
insulators Chowdhury et al. (2021). Recent analytical treatments of this model
Joshi et al. (2020); Tarnopolsky et al. (2020) present a picture of a spin
glass phase in low doping regime (with bosonic spinon and fermionic holon
quasiparticles), surviving up to a critical value of doping $p_{c}$, separated
from a disordered Fermi liquid at large doping. However whether any of these
quasiparticles or something different is responsible for the anomalously large
thermal Hall effect in the underdoped regime remains an open question. In this
paper, we study the stability of three different quasiparticles across doping
regimes in the random $t$-$J$ model of a doped Mott insulator by casting the
problem as one of localization in the many-body Hilbert space Altshuler et al.
(1997); Kumar and Tripathi (2020). We use a numerical exact diagonalization
method based on the recently developed FEAST algorithm Polizzi (2009) that
allows calculation of large numbers of eigenstates in user-specified energy
windows required for this approach.

The Hamiltonian of the random $t$-$J$ model is

${}H=\sum_{i\neq
j;\alpha=\uparrow,\downarrow}^{N}{\frac{t_{ij}}{\sqrt{N}}Pc^{\dagger}_{i\alpha}c_{j\alpha}P}+\sum_{i<j}^{N}{\frac{J_{ij}}{\sqrt{N}}\mathbf{S}_{i}\cdot\mathbf{S}_{j}},$
(1)

where $t_{ij}$ and $J_{ij}$ are both randomly chosen from a Gaussian
distribution with mean zero and unit variance. $P$ is the projector
prohibiting double occupancy at any site, and $N$ is the total number of
sites. A fraction $p$ of the sites is empty, corresponding to hole doping. The
spin operator at each site is
$\mathbf{S}_{i}=\frac{1}{2}\sum_{\alpha\beta}c^{\dagger}_{i\alpha}\boldsymbol{\sigma}_{\alpha\beta}c_{i\beta}.$

At low hole doping, the system has magnetic order and we show that spin-$1$
magnon excitations are good quasiparticles - a fact confirmed in numerous
numerical Arrachea and Rozenberg (2002); Shackleton et al. (2021); Dumitrescu
et al. (2021) and theoretical studies Joshi et al. (2020); Tarnopolsky et al.
(2020). Remarkably, in this doping regime, we also find that emergent Jordan-
Wigner (JW) fermionic excitations are good quasiparticles reminiscent of the
Mott insulating Kitaev materials Banerjee et al. (2016). In contrast, spinful
fermions are poor quasiparticles. In the opposite limit of large hole doping,
we confirm that spinful fermionic quasiparticles are stable while the magnons
and JW fermions are not.

Our study indicates, similar to the cuprates, a critical point separating
these regimes, which we identify as a many-body localization transition. We
find that all the above quasiparticles are bad at criticality.

We formulate the problem of quasiparticle stability as one of many-body
localization (MBL) Altshuler et al. (1997); Kumar and Tripathi (2020) in the
Hilbert space. Quasiparticles are approximations to the exact many-body
eigenstates. Their lifetime and decay can be understood as follows. Any
quasiparticle state $|\phi_{i}(t)\rangle$ can always be expanded as a linear
superposition of the exact many-body stationary states $|\psi_{j}\rangle$ of
the Hamiltonian, $|\phi_{i}(0)\rangle=\sum_{j}b_{ij}|\psi_{j}\rangle.$ At a
later time $t,$ the quasiparticle wavefunction is
$|\phi_{i}(t)\rangle=\sum_{j}\exp(-iE_{j}t)b_{ij}|\psi_{j}\rangle,$ where
$E_{j}$ are the energy eigenvalues. If the weights $b_{ij}$ are significant
only in a finite window of energies, $\Delta E,$ then the wavepacket decays in
a time $\Delta t=1/\Delta E.$ Since the number of energy eigenstates of the
interacting system generally grows exponentially with the system size while
the bandwidth grows much more slowly, the energy spread of the quasiparticle
will be vanishingly small in the thermodynamic limit if the number of
significant $b_{ij}$ (in other words, the support size of the quasiparticle in
the many-body Hilbert space) does not scale in proportion to the dimension of
the Hilbert space. On the other hand, if the support size scales in proportion
to the Hilbert space dimension, the quasiparticle has a finite lifetime in the
thermodynamic limit. These ideas are put on a quantitative footing below.

The quasiparticle stability is determined as follows. We first define our
magnon and fermionic quasiparticles. We construct the set of $N$ magnon states
by applying the spin raising operator at each site on the ground state
$|\chi\rangle$ of only the exchange part of the Hamiltonian,
$\\{c^{\dagger}_{\uparrow,1}c_{\downarrow,1}|\chi\rangle,....,c^{\dagger}_{\uparrow,N}c_{\downarrow,N}|\chi\rangle\\}.$
We then use the Gram-Schmidt orthonormalization to construct a set of $N$
orthonormal states. We project the Hamiltonian in Eq. 1 to this magnon basis,
and diagonalization yields the magnon quasiparticle states. We represent this
set of $N$ orthonormal magnon states as $\\{|\chi\prime_{i}\rangle\\}$. To
measure the resemblance of a given magnon state $|\chi\prime_{i}\rangle$ with
the exact many-body states $|\psi_{j}\rangle$ of the full Hamiltonian, we
express it as a linear superposition,

$|\chi\prime_{i}\rangle=\sum_{j=1}^{D}{a_{ij}|\psi_{j}\rangle},$ (2)

where $D$ is the dimension of the Fock space of physically relevant states
(i.e. do not include doubly occupied sites). The JW fermion creation
(annihilation) operators are constructed by taking the product of the spin
raising (lowering) operator at some site $i$ and a (nonlocal) 1D Jorgan-Wigner
string $\prod_{j<i}[p_{j}I+n_{j}\sigma^{z}_{j}],$ where $n_{j}=1-p_{j}.$ Note
our Jordan Wigner strings differ from the string operator introduced in ref
Grusdt et al. (2019) for the study of spinon chargon correlations in the
antiferromagnetic $t$-$J$ model. We repeat a similar procedure for the
fermionic quasiparticles. In particular, the hole states are obtained by
applying annihilation operators ($c_{i\alpha}$) on the ground state of only
the hopping part of the Hamiltonian. There are $2N$ number of hole
excitations. After the aforementioned Gram-Schmidt orthonormalization,
projection of the full Hamiltonian to this basis, and subsequent
diagonalization, we obtain a set of $2N$ fermionic hole quasiparticle states.
The hole quasiparticles are then expressed as a linear superposition of the
exact many-body states. Note that the magnon or hole quasiparticles are not
eigenstates of the random $t$-$J$ model.

Figure 1: Plot showing (disorder averaged) squares of the overlap of a magnon
(a), a JW fermion (b), and a hole (c) quasiparticle state (labeled $i$) with
exact many-body states (excitation energy $E_{j}$ measured from disorder
averaged ground state energy of many body state) of the random $t-J$ model for
different values of doping density $p,$ and system size $N.$ For low doping
density $p<0.33,$ (the spin glass phase) magnons (a) and JW fermions have
small support sizes (sharply peaked curves in (a), (b)) indicating a many-body
localized state in the Fock space (a stable quasiparticle), while spinful
fermions have large support sizes (broadly peaked curves in (c)), indicating a
fully delocalized state in the Fock space (a decaying quasiparticle). At high
doping density $p>0.33,$ spinful fermions have a small support size (sharply
peaked curve in (c)) while magnons and JW fermions have large support sizes
(broadly peaked curves in (a), (b)). At the critical value of doping,
$p=0.33,$ all three excitations have large widths indicating that they are bad
quasiparticles. Localization and delocalization are further confirmed by a
finite size scaling analysis (d-f) for the support sizes of (d) magnon and (e)
JW fermion and (f) spinful fermionic hole quasiparticle states for different
doping densities, as a function of the many-body Hilbert space dimension $D.$
The fitted solid lines are to the form $\xi=fD^{c}.$ The abscissae show values
of $1/\log_{2}D,$ while the ordinates show $\log_{2}\xi/\log_{2}D.$ An
intercept $c\approx 1$ signifies a fully many-body delocalized state (bad
quasiparticle) while a small value of the intercept $c\approx 0$ corresponds
to a many-body localized or fractal (stable quasiparticle) state. Note that
there is an abrupt jump in exponent $c$ in all three cases upon crossing the
localization threshold. For all values of doping except $p=0.33$, where none
of the quasiparticles are stable, at all other doping one or more
quasiparticles are found to be good.

The support size $\xi_{i}$ of the quasiparticle state $|\chi_{i}\prime\rangle$
in the many-body Hilbert space of exact eigenstates is given by
$\xi_{i}=1/P_{i},$ where $P_{i}$ is the inverse participation ratio (IPR),

$P_{i}=\sum_{j=1}^{D}|a_{ij}|^{4}.$ (3)

The stability of the magnon and the two fermionic quasiparticles is
quantitatively determined by performing a finite size scaling analysis of
their respective support sizes. This requires estimation of large number of
excited states for which the recently developed FEAST Polizzi (2009)
eigensolver is used.

Fig. 1(a) shows plots of the disorder averaged 111For every disorder
realization $\alpha$, the overlaps $|a^{(\alpha)}_{ij}|^{2}$ are computed, and
the combined data for all realizations is binned in equally spaced energy
intervals. The plots show the disorder averaged values,
$|a_{ij}|^{2}=(1/N_{\text{dis}})\sum_{\alpha}|a^{(\alpha)}_{ij}|^{2}$ vs
average many-body energy in the bin. squares of the overlap, $|a_{ij}|^{2}$,
for the second lowest magnon state $|\chi_{i}\prime\rangle$ with exact many-
body states $|\psi_{j}\rangle$ of the random $t-J$ model as a function of the
many-body excitation energy measured from the disorder averaged ground state
energy. Curves for different values of hole doping corresponding to under
(spin glass phase), over (Fermi liquid phase) and the putative critical doping
$p=p_{c}=1/3$ are shown. Figs. 1(b) and 1(c) respectively show similar plots
for respectively, the second lowest Jordan-Wigner (JW) fermion and a spinful
fermionic (hole) quasiparticle. At low doping densities $p<p_{c},$ the magnon
and JW quasiparticles have relatively small energy widths when compared to
spinful quasiparticles. At high doping densities $p>p_{c},$ the magnon and JW
quasiparticles have large widths compared to the spinful quasiparticles. We
also note that JW quasiparticles are generally less stable than magnons but
still good quasiparticles in the underdoped phase. The plots shown are
disorder averaged over $\sim 10^{3}$ configurations. Low-lying states have
been chosen for stability analysis here as they are more stable than the
higher energy ones and also easier to compute. The above overlap plots include
the effects of both disorder and interaction related broadening, with the
former not associated with decay.

In the SI, we present plots for lifetime broadening $\Delta\epsilon$ as a
function of quasiparticle energy $\epsilon,$ which is analogous to
$\text{Im}\Sigma$ studied elsewhere Deng et al. (2013); Cha et al. (2020). The
expected superlinear energy dependence of $\Delta\epsilon$ for low-lying
Landau quasiparticles in the overdoped regime is clearly seen. In the
underdoped regime, we see a flat and large $\Delta\epsilon$ expected for bad
Landau quasiparticles. For magnons, the overlaps of many-body states with the
quasiparticle wavefunction essentially determine the dynamical susceptibility
Shackleton et al. (2021). Starting from the underdoped side, we note the
broadening of magnon quasiparticles with doping as well as energy,
qualitatively consistent with Ref Shackleton et al. (2021). However our decay
width analysis does not give a clear signature of the quasiparticle stability
and transition, for which we find the IPR method to be better.

To place our above observations on a more quantitative footing, and also
determine the stabilities near critical doping, we perform a finite size
scaling analysis (for details see Kumar and Tripathi (2020); Altshuler et al.
(1997)) of the quasiparticle support sizes in the many-body Hilbert space.

If the support size $\xi$ of a quasiparticle state scales with the
dimensionality $D$ of the Fock space as $\xi\sim D,$ the quasiparticle is said
to be fully many-body delocalized state, and has a finite lifetime that is
inversely proportional to its energy width extracted from the support size. If
on the other hand, the support size scales as $\xi\sim D^{0},$ the
quasiparticle state is many-body localized in the Fock space - such a state
has an infinite lifetime. A third possibility may also occurs where the
scaling of the support size is fractal, $\xi\sim D^{c}$, with $c<1$ \- this
represents a nonergodic delocalized state, which is also a good quasiparticle
for our purposes. A perturbative treatment in Ref. Rivas et al. (2002) shows
that quasiparticle broadening calculated from imaginary part of self-energy
($\text{Im}\Sigma$) is equivalent to an IPR-based analysis.

Figure 1 (d-f) shows a plot of $(\log_{2}(\xi))/(\log_{2}(D))$ versus
$1/\log_{2}(D)$ for (d) magnon, (e) JW fermions and (f) spinful fermionic hole
quasiparticles. The fits are to the law $\xi\sim fD^{c},$ where $f\equiv f(D)$
has an arbitrary but weaker than power-law dependence on $D.$ Since the
interaction term generally does not connect all fermions or magnon
quasiparticle states to every many body state even in the fully delocalized
phase, we expect $f<1$ in the delocalized regime. In the many-body localized
phase, although $c=0,$ the support size $\xi\geq 1,$ and in general also an
increasing function of $D;$ consequently, $f(D)\geq 1$ is expected to
monotonously increase with $D.$ Fitted lines with positive (negative) slope
$(-\log_{2}(1/f))$ thus correspond to localized (delocalized) phases. For
small doping densities ($p<p_{c}$), the magnon state (Fig. 1(d)) has a small
support size, a positive slope $(-\log_{2}(1/f))$ and $c\approx 0$
corresponding to magnon localization and stability, while beyond the critical
doping $p_{c}=0.33,$ support size of the magnon scales exponentially with
$c\approx 1,$ and has negative slope corresponding to full delocalization and
instability. Elsewhere in the underdoped phase $p<p_{c},$ we observe for the
magnon a negative slope with intercept $c<1$ corresponding to a nonergodic
(fractal) delocalized state, still corresponding to well-defined
quasiparticles. The same behavior is shown by the JW fermions (Fig. 1(e))
although they are systematically more unstable than the magnons in all doping
regimes. The quasiparticle stability gets reversed for the spinful fermions
(Fig. 1(f)), which are stable for $p>p_{c}$ and unstable in the underdoped
regime. These scaling trends are consistent with the behavior seen in the
plots (a-c) of the overlap squares.

Figure 2: Plot shows the MBL scaling exponent $c$ versus doping density for
all three quasiparticles. A sharp jump in $c$ (to values $c\approx 1$) is seen
for all quasiparticles upon crossing to the doping regime where they are
unstable.

We now discuss the behavior in the vicinity of $p_{c}.$ Note that the
dimensionality $D$ of the Fock space takes its maximum value at $p_{c}=1/3;$
in the thermodynamic limit it is easily checked that at this critical doping,
$D\sim 3^{N}.$ This indicates three degrees of freedom at each site,
signifying some emergent symmetry at this point. In comparison, at zero
doping, there are only two degrees of freedom at each site ($D=2^{N}$), and at
very high values of the doping, $D$ increases slower than exponential. From
the scaling analysis in Fig. 1 (d-f), it is clear that the exponent $c\approx
1$ for all three types of excitations when $p\approx p_{c}=0.33,$ consistent
with the understanding from earlier field-theoretical studiesJoshi et al.
(2020) that there are no good quasiparticles at critical doping in this model.

Figure 2 shows the plot of the intercept $c$ versus doping density $p.$ A
sharp step in $c$ is observed near the vicinity of $p_{c}$ for all
quasiparticles. The change in $c$ near $p_{c}$ for JW fermions is evident
(showing they are good quasiparticles that decay in the overdoped side), but
less sharp than the jump for magnons, which happen to be better
quasiparticles. Interestingly, similar jumps in the scaling exponent have been
seen in MBL transitions in disordered XXZ Heisenberg chains Rivas et al.
(2002).

In conclusion, using a many-body localization treatment, we confirm the
existence of two phases as a function of doping density - a spin glass phase
at low doping and a Fermi liquid phase at high values of doping. At large
doping density $p>0.33,$ spinful fermionic excitations are well-defined while
at small doping $p<0.33,$ magnons and emergent Jordan-Wigner fermions are
well-defined. Magnons and spinful fermions of comparable energy are not
observed to be simultaneously good quasiparticles for any value of the doping.
Near the critical point $p=0.33,$ we find that all these three excitations are
bad quasiparticles. A central observation is that the deconfined critical
point is associated with a localization transition in the many-body Hilbert
space. The existence of stable Jordan-Wigner fermions in the spin glass phase
could be of experimental significance. In the ground state of the underdoped
(spin glass) phase, the JW quasiparticle density is fairly high, at $(1-p)/2.$
Our analysis lends support to the recent view that these quasiparticles are
likely responsible via their coupling to phonons for the anomalously large
thermal Hall conductivity seen in the underdoped cuprates (concomitant with
poor electrical conductivity).

###### Acknowledgements.

AK and VT acknowledge support of the Department of Atomic Energy, Government
of India, under Project Identification No. RTI 4002, and the Department of
Theoretical Physics, TIFR, for computational resources. SS was supported by
the U.S. National Science Foundation grant No. DMR-2002850 by the Simons
Collaboration on Ultra-Quantum Matter which is a grant from the Simons
Foundation (651440, S.S.)

##  Appendix A Case of a good quasiparticle: comparison of single disorder
realization and disorder averaging

Figure 3:  Distribution of the overlap square of the second lowest magnon
quasiparticle as function of many-body excitation energy $E_{j}$ at zero
doping, $p=0,$ where the magnon is stable. The plot is for one disorder
realization, and the system size is $N=15.$ The quasiparticle has significant
overlap with around five ($O(1)$) many-body states.

Figure 3 shows the plot of overlap squares $|a_{ij}|^{2}$ for the second
lowest magnon quasiparticle at zero doping, and in one disorder realization.
The magnon quasiparticle state significantly overlaps with only around five
(i.e. $O(1)$) exact many body states, even though the Hilbert space has
$2^{15}$ states. This proves the quasiparticle is very good. We generate such
data for the second lowest magnon for different disorder realizations, and
then perform the disorder average. For each disorder realization, the second
lowest magnon is similarly found to have significant overlaps with a very
small number of many-body states. This is shown in Fig. 5 for the distribution
of support sizes for 1000 disorder realizations, which is seen to be dominated
at support size of around five.

Even though Fig. 3 and Fig. 5 show that this magnon is a very good
quasiparticle, we also see from Fig. 4 (see also Fig. 1 (left panel) in the
manuscript) that the energy spread of this state is not very small, and is
comparable to the disorder strength. This spread is not a sign of
quasiparticle decay since it’s a noninteracting effect arising from different
disorder realizations giving different energies for the many-body states.

Figure 4:  Plot showing a histogram of the mean excitation energy $\epsilon$
(after quantum averaging ) for a stable (second lowest) magnon at zero doping,
15-site system. The $\epsilon$ are measured from the disorder averaged ground
state energy for 1000 disorder configurations. The distribution peaks at a
small nonzero positive energy although there is a large spread on account of
the disorder. Note that in a small fraction of samples, the excitation energy
is lower than the disorder averaged ground state energy - this is on account
of rare disorder configurations.  Figure 5:  Plot showing histogram of support
sizes ($1/\text{IPR}$) for a stable (second lowest) magnon at zero doping for
a 15-site system. The distribution is sharply peaked at a small value $\approx
5$ although the number of states is $2^{15}.$

It is clear from the above example that even when the overlap squares of
individual disorder realizations indicate a good quasiparticle, it is not
manifest in the disorder averaged plots of the same quantity. Instead, we
found that the support size distribution conveys this information more
accurately, even after disorder averaging. Physically, this is because a good
quasiparticle always has a small support size over a wide rangle of disorder
configurations.

##  Appendix B Energy width of quasiparticle

Figure 6:  Plot shows the evolution of the disorder averaged quasiparticle
energy spread $\Delta\epsilon$ versus its mean excitation energy. See Eq. (5)
for an explanation. Good quasiparticles are characterized by relatively small
quasiparticle widths, and bad ones have large and flat $\Delta\epsilon.$ On
the overdoped side, low-energy spinful quasiparticles have a super-linear
increase of energy spread, consistent with landau Fermi liquid theory.

In the literature, the lifetime broadening (for, say, Landau quasiparticles)
in the doped Mott insulators is usually extracted from the imaginary part of
the single particle self energy, $\text{Im}\Sigma$ Deng et al. (2013); Cha et
al. (2020). Although our IPR-based method does not use Green functions, it is
possible to obtain a similar measure of the quasiparticle width. For this
purpose, we first calculate the energy spread of the quasiparticle for a
single disorder realization, and then we perform the average over disorder
configurations. For a good quasiparticle, the spread will be small for typical
realizations of disorder. For a single disorder realization, $\alpha,$ the
energy spread $\Delta\epsilon_{i}^{(\alpha)}$ of the $i^{\text{th}}$
quasiparticle is given by

$\displaystyle(\Delta\epsilon_{i}^{(\alpha)})^{2}$
$\displaystyle=\sum_{j}|a_{ij}^{(\alpha)}|^{2}(E_{j}^{\alpha})^{2}-\left[\sum_{j}|a_{ij}^{(\alpha)}|^{2}E_{j}^{(\alpha)}\right]^{2},$
(4)

where $E_{j}^{(\alpha)}$ are the many-body energies for this realization. The
disorder averaged spread is given by

$\displaystyle(\Delta\epsilon_{i})^{2}$
$\displaystyle=\frac{1}{N_{\text{dis}}}\sum_{\alpha}(\Delta\epsilon_{i}^{(\alpha)})^{2},$
(5)

where $N_{\text{dis}}$ is the number of disorder realizations (taken to be
1000 in our case).

Fig. 6 shows the disorder averaged energy width $\Delta\epsilon$ versus mean
energy $\epsilon$ of the quasiparticle. For the spinful fermions (Landau
quasiparticles), far on the overdoped side (here $p=0.80$), $\Delta\epsilon$
starts at a small value and curves upwards, reminiscent of the
$\text{Im}\Sigma\sim\epsilon^{2}$ for a Landau quasiparticle, and at higher
energies it flattens out at a certain large value, which is comparable to the
quasiparticle bandwidth. At critical doping ($p=0.33$), $\Delta\epsilon$
starts at a large value and varies little with $\epsilon.$ It is also gapped
(even though the many-body spectrum is not), indicating it’s not a good
approximation for a low energy excitation. Closer to the critical doping, but
still on the overdoped side (here $p=0.5$), $\Delta\epsilon$ starts at larger
values, still appears to increase super-linearly with $\epsilon,$ and soon
reaches the saturation value of the order of the quasiparticle bandwidth. A
similar trend is seen for magnons and JW fermions. For the magnons at zero
doping, $\Delta\epsilon$ starts at small values for low-lying $\epsilon,$ but
soon saturates at a relatively small value. The JW fermions on the underdoped
side show a sharper increase of $\Delta\epsilon$ similar to that shown by
Landau quasiparticles on the overdoped sice. with At critical doping, both
magnons and JW fermions have large gaps, and $\Delta\epsilon$ is flat, at a
relatively high value comparable to the quasiparticle bandwidth. The trends
(e.g. stability and energy dependence of $\Delta\epsilon$ Landau
quasiparticles on the overdoped side) are in qualitative agreement with
analyses based on other techniques such as DMFT Cha et al. (2020); Deng et al.
(2013), although quantitative comparison is not possible because our
$\Delta\epsilon$ is not exactly $\text{Im}\Sigma$ of a single electron Green
function, as the latter is extracted from a part of the spectral function that
corresponds to the quasiparticle pole. Note that the transition from stable to
unstable quasipaticles does not reveal sharply in our calculations of
$\Delta\epsilon$ but is readily detected in the finite size scaling of the
IPR. Note that

## References

  * Frachet et al. (2020) M. Frachet, I. Vinograd, R. Zhou, S. Benhabib, S. Wu, H. Mayaffre, S. Krämer, S. K. Ramakrishna, A. P. Reyes, J. Debray, et al., Nature Physics 16, 1064 (2020), eprint 1909.10258.
  * Grissonnanche et al. (2019) G. Grissonnanche, A. Legros, S. Badoux, E. Lefrançois, V. Zatko, M. Lizaire, F. Laliberté, A. Gourgout, J.-S. Zhou, S. Pyon, et al., Nature 571, 376 (2019).
  * Arrachea and Rozenberg (2002) L. Arrachea and M. J. Rozenberg, Phys. Rev. B 65, 224430 (2002), eprint cond-mat/0203537.
  * Grissonnanche et al. (2020) G. Grissonnanche, S. Thériault, A. Gourgout, M.-E. Boulanger, E. Lefrançois, A. Ataei, F. Laliberté, M. Dion, J.-S. Zhou, S. Pyon, et al., Nature Physics 16, 1108 (2020).
  * Samajdar et al. (2019) R. Samajdar, M. S. Scheurer, S. Chatterjee, H. Guo, C. Xu, and S. Sachdev, Nature Phys. 15, 1290 (2019), eprint 1903.01992.
  * Dalla Piazza et al. (2015) B. Dalla Piazza, M. Mourigal, N. B. Christensen, G. J. Nilsen, P. Tregenna-Piggott, T. G. Perring, M. Enderle, D. F. McMorrow, D. A. Ivanov, and H. M. Rønnow, Nature Physics 11, 62 (2015), eprint 1501.01767.
  * Kitaev (2006) A. Kitaev, Annals of Physics 321, 2 (2006).
  * Yokoi et al. (2021) T. Yokoi, S. Ma, Y. Kasahara, S. Kasahara, T. Shibauchi, N. Kurita, H. Tanaka, J. Nasu, Y. Motome, C. Hickey, et al., Science 373, 568 (2021).
  * Kasahara et al. (2018) Y. Kasahara, T. Ohnishi, Y. Mizukami, O. Tanaka, S. Ma, K. Sugii, N. Kurita, H. Tanaka, J. Nasu, Y. Motome, et al., Nature 559, 227 (2018).
  * Varma (2020) C. M. Varma, Rev. Mod. Phys. 92, 031001 (2020), URL https://link.aps.org/doi/10.1103/RevModPhys.92.031001.
  * Cha et al. (2020) P. Cha, N. Wentzell, O. Parcollet, A. Georges, and E.-A. Kim, Proceedings of the National Academy of Sciences 117, 18341 (2020), ISSN 1091-6490, eprint 2002.07181, URL http://dx.doi.org/10.1073/pnas.2003179117.
  * Guo et al. (2020) H. Guo, Y. Gu, and S. Sachdev, Annals Phys. 418, 168202 (2020), eprint 2004.05182.
  * Sachdev and Ye (1993) S. Sachdev and J. Ye, Phys. Rev. Lett. 70, 3339 (1993), eprint cond-mat/9212030.
  * Kitaev (2015) A. Kitaev, USA April 2015 (2015).
  * Chowdhury et al. (2021) D. Chowdhury, A. Georges, O. Parcollet, and S. Sachdev (2021), eprint 2109.05037.
  * Joshi et al. (2020) D. G. Joshi, C. Li, G. Tarnopolsky, A. Georges, and S. Sachdev, Phys. Rev. X 10, 021033 (2020), eprint 1912.08822.
  * Tarnopolsky et al. (2020) G. Tarnopolsky, C. Li, D. G. Joshi, and S. Sachdev, Phys. Rev. B 101, 205106 (2020), eprint 2002.12381.
  * Altshuler et al. (1997) B. L. Altshuler, Y. Gefen, A. Kamenev, and L. S. Levitov, Phys. Rev. Lett. 78, 2803 (1997), URL https://link.aps.org/doi/10.1103/PhysRevLett.78.2803.
  * Kumar and Tripathi (2020) A. Kumar and V. Tripathi, Phys. Rev. B 102, 100401 (2020), URL https://link.aps.org/doi/10.1103/PhysRevB.102.100401.
  * Polizzi (2009) E. Polizzi, Phys. Rev. B 79, 115112 (2009), URL https://link.aps.org/doi/10.1103/PhysRevB.79.115112.
  * Shackleton et al. (2021) H. Shackleton, A. Wietek, A. Georges, and S. Sachdev, Phys. Rev. Lett. 126, 136602 (2021), eprint 2012.06589.
  * Dumitrescu et al. (2021) P. T. Dumitrescu, N. Wentzell, A. Georges, and O. Parcollet (2021), eprint 2103.08607.
  * Banerjee et al. (2016) A. Banerjee, C. A. Bridges, J. Q. Yan, A. A. Aczel, L. Li, M. B. Stone, G. E. Granroth, M. D. Lumsden, Y. Yiu, J. Knolle, et al., Nature Materials 15, 733 (2016), eprint 1504.08037.
  * Grusdt et al. (2019) F. Grusdt, A. Bohrdt, and E. Demler, Phys. Rev. B 99, 224422 (2019), eprint 1901.01113.
  * Deng et al. (2013) X. Deng, J. Mravlje, M. Ferrero, G. Kotliar, A. Georges, et al., Physical review letters 110, 086401 (2013).
  * Rivas et al. (2002) A. M. Rivas, E. R. Mucciolo, and A. Kamenev, Physical Review B 65, 155309 (2002).


# DeepClean: Integrated Distortion Identification and Algorithm Selection for
Rectifying Image Corruptions

Aditya Kapoor Corresponding Author Email<EMAIL_ADDRESS>Harshad
Khadilkar  Jayvardhana Gubbi  Tata Consultancy Services

###### Abstract

Distortion identification and rectification in images and videos is vital for
achieving good performance in downstream vision applications. Instead of
relying on fixed trial-and-error based image processing pipelines, we propose
a two-level sequential planning approach for automated image distortion
classification and rectification. At the higher level it detects the class of
corruptions present in the input image, if any. The lower level selects a
specific algorithm to be applied, from a set of externally provided candidate
algorithms. The entire two-level setup runs in the form of a single forward
pass during inference and it is to be queried iteratively until the retrieval
of the original image. We demonstrate improvements compared to three baselines
on the object detection task on COCO image dataset with rich set of
distortions. The advantage of our approach is its dynamic reconfiguration,
conditioned on the input image and generalisability to unseen candidate
algorithms at inference time, since it relies only on the comparison of their
output of the image embeddings.

1794

##  1 Introduction

Due to recent advances in Machine Learning, images and videos have been
extensively used in the applications of robotics, healthcare, satellite
imagery, social media, and security to name a few. In each of these
applications, the quality of image and video plays a vital role in achieving
good performance in downstream vision tasks like image classification, object
detection and segmentation. Even before the images are used for a particular
task, they need to be acquired, processed, compressed, transmitted, stored and
then retrieved. The quality of the image may deteriorate in either of the
aforementioned stages and hence can introduce different kinds of distortions
that can affect the downstream vision task’s performance. For example, during
acquisition the image sensor or circuitry of a digital camera or scanner can
inject noise in the image. If an image is being taken with a hand-camera,
motion blur can be introduced in the image. Some image compression algorithms
are lossy and may erase information that maybe crucial. While transmission,
data packets can be lost which might degrade the image quality. As a result,
identification and rectification of the distortions in the image is very
useful to ensure decent performance of the end vision goal.

Currently, to the best of our knowledge, in the community most experts still
configure the image processing pipeline by visually inspecting the image for
every specific use case. If we were to break down the expert methodology into
a procedure, at first the expert identifies the possible distortions that are
present in the image and then later uses existing algorithms in their toolkit
to rectify the distortions in the image with trial and error. Once the experts
arrive at a pipeline, it remains fixed for the lifetime of the task. Since
this process is manual and heavily relies on intuition, we believe there is a
need to automate this process since the image quality heavily impacts the
downstream task. If the image processing pipeline is not carefully configured,
it can cause more harm than good. In Sec 6 we show empirically how a wrong
image processing pipeline can heavily impact the metrics of object detection.
It is also not straightforward to determine the choice of algorithms to be
used to recover the image. For example, to restore an image if it needs to
undergo exposure correction and then denoising, the choice of algorithms also
plays a vital role. Two individually best performing algorithms might not
yield the best result when used in conjunction with each other. As a result
algorithm choice also becomes crucial alongside distortion identification.
Finally, most image processing pipelines are fixed for the entire dataset
which is not ideal since different images in the same dataset might require
different sequence and choice of correction algorithms for rectification.

As a result, this paper attempts to deal with distortion identification and
correction algorithm selection for every input image. We believe that such a
framework can be very useful to the computer vision community.

Our proposed framework has the following main contributions:

  * •

We propose DeepClean, a plug-and-play data-driven simple multi-task framework
that automatically identifies distortions and dynamically selects appropriate
algorithms in the vision pipeline without any manual engineering efforts.

  * •

Our framework does not assume any prior knowledge about the pool of candidate
algorithms to choose from and hence has generalization capabilities.

  * •

We also experiment with new forms of distortions that the framework has not
seen during training time and show that it adapts well to such situations.

  * •

We demonstrate the advantage of DeepClean compared to baselines on the COCO
dataset on object detection as the downstream vision task.

##  2 Related Work

In this section, we review some prior works that attempt to identify image
distortions. Some of these works consider distortion identification as the
primary task while some consider it to be a secondary task of an image quality
assessment (IQA) process. Further, we also explore other algorithm selection
frameworks that have been used to identify the best single-agent and multi-
agent path finding algorithm given a particular problem instance described
using an image. Lastly, we also look into AI-planning and Reinforcement
Learning (RL) methods that have been previously used to address a similar
problem setting.

###  2.1 Distortion Classification

Namhyuk Ahn et al. [1] used CNN architectures pretrained on ImageNet dataset
[9] to classify distortion type present in the input image without a clean
reference image. Mateusz et al. [4] trained two CNN architectures that were
similar in terms of number of layers and layout but had different size and
number of parameters to predict the distortion in an image. Bianco et al. in
[3] analyzed the features extracted from different layers of different deep
neural network (DNN) architectures and then evaluate their relevance using
clustering. The best features were then used for the recognition of the type
of distortion. Unsupervised Deep visual representations show cases the ability
to recognize the various types of image distortion.

Distortion identification is useful for IQA since it can either help to select
the most appropriate metric for evaluation or provide vital information about
the distortions from the metric. Falk et al. [10] proposed to linearly combine
a set of features from a pool of full-reference IQMs that were relevant for
distortion identification. Moorthy et al. [21] describes a classifier that
relies on the statistics of the distorted image to classify the distortion
type. Peng et al. [24] proposed a two step procedure for quality assessment
where in the first step, a SVM was used to identify the distortion type and in
the second step using the knowledge of the distortion type, three existing
IQMs were fused using K-nearest neighbors. Chetouani et al. [5] proposed a
framework built on the assumption that there is no single universal IQM that
can effectively estimate the image quality across all different types of
distortions. They used linear discriminant analysis (LDA) to classify the
distortions before image quality estimation. During classification, they
relied on relevant quality scores derived from different IQMs that were
applied on both the clean and the distorted version of the clean image. Kang
et al. [16] proposed IQA-CNN, a model that estimates image quality and
identifies distortion types. IQA-CNN+ [16] and IQA-CNN++ [16] are the two
advanced versions of IQA-CNN. A CNN model was also proposed by Wang et al.
[31] for distortion identification and quality assessment without the clean
reference image. Kede et al. [20] proposed a CNN-based IQM with two sub-
networks, distortion identification network and quality prediction network.
Both these networks shared the early layers. At first, the distortion
identification network is trained followed by quality prediction network
starting from the pre-trained first network. Huang et al. [14] proposed a mask
gated convolutional network (MCGN) that predicts distortions as well as image
quality score simultaneously. MCGN uses an encoder which is designed to
capture the transformation between clean and distorted version of the clean
image as low level features. Later, the high level features are extracted and
utilized to predict the image quality score and distortion type. Ameur et al.
[2] proposes a standalone multitask learner to identify multiple distortions
present in the image. They deploy a DenseNet-169 backbone feature extractor
pretrained on ImageNet dataset and subsequently learn one prediction head to
predict the absence or presence of each possible distortion in the image.

The above works achieve good results on distortion identification. However,
most of the approaches rely on domain knowledge to manually design feature
descriptors, almost all of them are not scalable solutions and none of them
focus on selecting appropriate correction algorithms to rectify the distortion
present in the image.

###  2.2 Algorithm Selection in Path Finding (PF)

Algorithm selection has been an active area of research in Single-Agent and
Multi-Agent PF since different PF algorithms work well for different problem
instances. Sigurdson et al. [28] selects heuristic based search algorithms for
single-agent path finding using a generalizable approach that makes use of
image classifiers. They train AlexNet to learn a mapping between the different
types of problem instances and the corresponding best path finding algorithm.
Sigurdson et al. [29] extends his previous work and trains AlexNet to map
different multi-agent path finding instances represented using RGB images to
different multi-agent path finding algorithms depending on a few metrics used
to measure the overall performance of the different algorithms. The authors
ensure that the portfolio of the algorithms in consideration are diverse and
well suited to address the different problem instances in the dataset.
Similarly, Ren et al. [25] also deploy a CNN with Inception module and a
different problem instance definition and algorithm portfolios to address the
multi-agent path finding setting.

The above approaches try to alleviate the need for manual feature extraction
and switch to CNN based approaches since they discover that CNNs are capable
to learn to extract relevant features that are necessary to identify the
algorithms for a given problem instance. However, the above approaches require
the dataset to be re-prepared and also need to be retrained if newer
algorithms are to be introduced which can be computationally expensive. On the
other hand, Ren et al. [25] also requires additional information alongside RGB
images of the map, that is, single-agent optimal paths to improve the
classification accuracy. As a result, the performance of the algorithm is
conditioned on the choice of the single-agent path finding algorithm that can
lead to sub-optimal predictions.

###  2.3 Algorithm selection using AI planning

The early attempts to select appropriate algorithms for automating the
generation of image processing workflows using AI planning only focus on
certain specific applications [6, 18]. Nadarajan et al. [22] proposed a system
based on decomposition-based planning and ontologies for the automatic
construction of video processing solutions and demonstrated it for detection
and tracking in underwater videos. In this paper, the pre-processing steps
only includes video capture and frame image grabbing. Similarly, Clouard et
al. [8] proposed BORG a knowledge-rich problem-solving system that is used for
dynamically constructing image processing procedures through the selection,
parameter tuning and scheduling of existing algorithms. Such rule-based
knowledge-driven systems can only succeed in solving problems corresponding to
well-identified tasks. Sengar et al. [27] developed a framework that can adapt
the solutions based on changing requirements by integrating planning with RL
and learning from goal-directed reward. However, the cost of each algorithm is
learned based on certain image characteristics and its corresponding state
abstractions rather than the entire image features directly. Similarly, Kapoor
et al. [17], introduced a Transformer Architecture combined with Deep
Reinforcement Learning to recommend algorithms that can be incorporated at
different stages of the vision workflow. However, this approach is limited
since it searches for the optimal algorithm for each preprocessing step by
fixing the image processing pipeline. A recent attempt was made by Choudhury
et al. [7] to couple planning and learning based approaches to construct
dynamic vision pipelines for downstream tasks.

##  3 Problem Definition

Figure 1: Description of three levels of hierarchy to generate a Vision
Pipeline.

We believe that there are three steps involved to compose the pipeline for a
goal vision task. In Fig. 1 the first step is to define the high level
pipeline which we term as L1-hierarchy. The L1-hierarchy is a skeleton which
broadly identifies the different set of sub-goals that lead to achieving the
downstream vision task. In most cases, for a given goal vision task, the
L1-hierarchy is task dependent and hence can be known apriori. The second step
in the hierarchy, L2-hierarchy, is to identify the arrangement of a sequence
of sub-tasks that are necessary to achieve the sub-goals of the L1-hierarchy.
For example, the problem of identifying the right sequence of image processing
steps belongs to L2-hierarchy. Finally, we define L3-hierarchy as the problem
of identifying the algorithm to be used for each of the sub-tasks.

Let’s take an example of object detection, refer Fig. 1. Given the
L1-hierarchy skeleton is fixed, the problem of L2-hierarchy is to identify the
sequence of sub-tasks that need to be taken within each subgoal of the
L1-hierarchy. For example, within image processing, the identification of
various image processing steps like denoising, exposure-correction, blur-
correction etc and their sequence. Finally, L3-hierarchy deals with the choice
of the candidate algorithms that need to be employed for execution the sub-
tasks like SCUNet for denoising, gamma correction for exposure rectification,
etc.

The orchestration of the composition of each of the hierarchy levels has the
capability to generate a vision pipeline to achieve a certain goal task. One
should note that the L1-hierarchy remains fixed (in most cases) irrespective
of the nature of the input images since it’s only dependent on the downstream
vision task. However, the L2 and L3 hierarchy may undergo changes depending on
the input images. For example, if $image_{1}$ has undergone exposure
distortion and noise addition and $image_{2}$ has undergone noise addition
followed by exposure distortion; the L2 and L3 hierarchy in each of the case
will be different, that is denoising followed by exposure correction, and
exposure correction followed by denoising respectively. Each image might also
require different algorithms at each stage depending on the nature of
distortion.

In this paper, we deal with L2 and L3 hierarchy and treat them as a
hierarchical sequential decision making problem. We propose a framework that
can simultaneously identify the next processing step and algorithm in the
sequence.

We show the effectiveness of our planning approach in Image Processing
pipeline generation. However, our approach is not limited to Image Processing.
Based on our definitions of the L-2 and L-3 hierarchies, the problem of
identification of the correct sequence of image processing steps fall in the
L2-hierarchy, and that of algorithm selection belongs to the domain of
L3-hierarchy.

Mathematically, we define this problem by a tuple $\langle I,S,A,Q\rangle$,
where I $=\\{i_{1},i_{2},..\\}$ are the set of input images, S
$=\\{s_{1},s_{2},..\\}$ is the set of image processing steps and A
$=\\{a_{1},a_{2},..\\}$ is the set of image processing algorithms. Further,
for each image processing step in S there is an image processing algorithm in
A. Q $:$ S $\times$ A $\to\mathbb{R}$ is a function that returns the quality
of the solution obtained when a image processing step $s_{i}$ $\in$ S and a
corresponding image processing algorithm $a_{i}$ $\in$ A is chosen to process
the input image $i_{i}$ $\in$ I.

We learn a mapping function that $\pi:$ I $\to$ $\langle S,A\rangle$ that maps
each image $i_{i}\,\in$ I to a image processing step $s_{i}\,\in$ S in the
first stage and an image processing algorithm $a_{i}\,\in$ A in the second
stage.

Our set of images I consists of a combination of no distortions, single-level
distortion or combination of two distortions. The distortions comprise of
exposure changes and noise addition. Within exposure change, we either
underexpose the image or overexpose the image and in case of noise addition,
we add low or high levels of Gaussian noise in the images. We discuss about
this in more detail in the subsequent section that describes the dataset
preparation for training and testing our framework 4. We fix the downstream
vision task to be Object Detection and thus fix the L1-hierarchy to be Capture
RGB Image followed by Image Processing followed by Object Detection. The set S
comprises of Denoising, Underexposure Correction, Overexposure Correction and
No Correction as the image processing steps and the corresponding correction
algorithms are included in the set A. Each algorithm $a_{i}$ is specialized to
address a distortion of a particular type in image $i_{i}$. Thus for each
image processing step $s_{i}$ $\in$ S there exists atleast two algorithms
$a_{i}$ $\in$ A.

##  4 Dataset Preparation

To prepare our dataset, we extend the COCO validation dataset [19]. The images
in our dataset fall in either of the categories; clean, distorted exposure
(underexposed/overexposed), noise addition (low/high) or a combination of both
distorted exposure and noise addition in different sequences. We saw that if
the image were corrupted using three or more distortions, it would become
really difficult to restore the image back to its original format and thus we
restricted ourselves to a maximum of two levels of distortions in this study.
To distort the exposure of the image, we use the Gamma correction algorithm
present in Pytorch [23] which is based on the power law:

$I_{out}=255\times gain\times(I_{in}/255)^{\gamma}$ (1)

We set the gain value to be 1 at all times and use $\gamma$ $\in$
$\\{0.2,0.8,2.0,3.0\\}$. The former two values of gamma are used to
underexpose the images and the latter two are used to overexpose the images.

To introduce noise in the images, we used Gaussian noise. The following
equation was used to do the same:

$I_{out}=((I_{in}/255)+\mathcal{N}(\mu,\,\sigma^{2}))\times 255$ (2)

We fix $\mu$ to be 0.0 throughout and sample $\sigma$ from
$\\{0.04,0.08,0.15,0.2\\}$. Again the former two values add low-level of noise
in the image and the latter two values add high-level noise in the image.

Overall we generate a dataset with 202,622 images out of which 4942 are clean
images, 9,884 images comprise of low and high level noise, and overexposed and
underexposed images and the rest are a combination of each of the distortions
in different sequences.

In case of generating the test dataset with different distortions, we adopt
the following $\gamma$ values: $\\{0.3,0.9,2.2,3.2\\}$ and $\sigma$ values:
$\\{0.06,0.1,0.17,0.25\\}$

##  5 Proposed Method

The aim of our proposed framework is to develop a robust and reliable method
for distortion identification and algorithm selection for image rectification
without depending on the corresponding clean images. In order to achieve this,
we extend the previous work [2] which only deals with distortion
identification and also make it compatible to select corresponding algorithms
that can rectify the identified distortion. We first empirically show that why
a deep learning approach is suitable for this method and later describe our
framework in more detail.

###  5.1 Motivation

The features obtained from a deep learning feature extractor (such as ResNet
[12] or DenseNet [15]) capture various attributes of the images in the latent
space. These feature vectors can capture information about the different
distortions present in the image. We empirically validate this assumption by
training a ResNet-50 classifier (pretrained on ImageNet dataset) to identify
the different distortion sequences present in the image. For our use-case, the
classes of distortion sequence were clean image, underexposed image,
overexposed image, low noise-level, high noise-level, underexposed followed by
low noise-level image, low noise-level followed by underexposing the image and
so on. Our classifier was able to tell apart the different distortions with
96% accuracy. We trained both the ResNet backbone and the latter
classification layer to identify the latest distortion that was added to the
input image. Thus one can rely on the features extracted by the ResNet-50
feature extractor to identify different distortions present in the image.

We take inspiration from the methodology used by human experts to identify
distortions in the image and select subsequent algorithms to rectify them. For
the former, human experts solely rely on the visual aspect of the input image.
In case of the latter, they also rely on the relative capability of the
correction algorithm; that can be abstracted from metrics like PSNR [11] (Peak
Signal-to-Noise Ratio). However, to make an algorithmic choice, human experts
make a relative comparison between the different correction algorithms that
are capable to rectify the existing distortion either via visually inspecting
the output image of the algorithm or use some metrics well suited for the
task. We use this systematic approach as the basis to construct our framework
which we discuss in detail in the next subsection.

Figure 2: Framework to identify image distortion. Figure 3: An example of the
framework to select the correction algorithm for distortion rectification.
Assuming that in the first stage if the identified distortion is an
“underexposed image”.

###  5.2 Architecture Details

Our algorithm deploys a two step procedure to first identify the distortion in
the image and second to select an algorithm for rectifying the distortion Fig.
2 & Fig. 3. In the first step, the algorithm is responsible for identifying
the distortion present in the image. It is an extension of the previous work
[2] that makes use of the Multi-Task Learning (MTL) framework. Instead of
using a DenseNet-169 as the backbone, we used ResNet-50 as the backbone
network which is pretrained on ImageNet dataset. The features extracted from
the distorted images are further sent to each head which are Fully Connected
Networks. Each head is responsible for the binary classification of a single
distortion type present in the image. As discussed earlier, such an
architectural setting is capable to identify distortion types since the
feature vector extracted from the distorted image captures information about
the distortions present in the image.

Our model takes an input image and identifies the latest distortion that
corrupted the image. For example, if an image has undergone two distortions
wherein its exposure was compromised followed by noise being added, our
algorithm would identify noise as the distortion type in the first pass and
suggests an algorithm to denoise the image. Images with multiple distortions
have to be passed into our model until it predicts the image to be clean. This
problem closely resembles the process of sequential decision making.

In the second step, the framework is used to identify the subsequent
correction algorithm that is necessary to rectify the distortion in the image.
We perform a guided algorithm search based on the identified distortion thus
restricting the search space to a limited number of algorithms.

Once the distortion type is identified, the search is performed only on those
correction algorithms that address the identified distortion. The distorted
image is passed through each of the relevant correction algorithms, the output
of which is an image (Fig. 3). Now each of these generated images are further
passed through the ResNet-50 feature extractor and the corresponding head of
the identified distortion, trained in first step, to generate corresponding
algorithm feature vectors. These algorithm specific feature vectors have
knowledge about the the presence or absence of the identified distortion and
can be used as a good proxy to identify the correct candidate. For example, in
Fig.3, if the identified distortion is an “underexposed image”, then the
feature vectors generated from the output of “underexposure correction” in the
second stage can be processed to judge which algorithm has rectified the
distortion (since each algorithm can have different degrees of correction). As
a result, the distorted image feature vector has information about the
distortions present in the original distorted image and the feature vector of
the image generated from the correction algorithm has information about just
the uncorrected distortions present in the original distorted image. In order
to identify the correction algorithm, we perform a cosine similarity measure
between each of the algorithm generated image feature vectors and the
distorted image feature vector. The feature vector of the correction
algorithm, that can correct the distorted image the most, should have a
smaller cosine similarity with the feature vector of the distorted image since
the angle between them should be relatively larger. Thus, we can essentially
rank the correction algorithms according to their performance on the current
input image using any similarity metric.

This entire procedure is repeated until our framework predicts the image to be
clean.

In the next subsection, we describe the different algorithms used in our
framework and discuss the training procedure in detail.

###  5.3 Denoising Algorithms

The aim of denoising algorithms is to filter out the additive noise from the
distorted image and restore back the clean image. We use 3 variants of SCUNet
as denoising algorithms. In recent years, the field of image restoration has
seen a rise in the use of Transformer-based techniques. For combining the
local modelling capability of residual convolutions with the non-local
modelling capability of effective shifted window attention, SwinConv-
Unet(SCUNet) [32]. This allows each pixel to contain global correlation while
compensating for the drawbacks of conventional CNNs that only consider local
receptive fields and ignore global features. Each variant of SCUNet is better
suited for a particular noise range. We used SCUNet-15, SCUNet-25 and
SCUNet-50 that were specialized to correct noise levels of 15 pixels, 25
pixels and 50 pixels in the image.

###  5.4 Exposure Correction Algorithms

We have used gamma correction algorithm with multiple gamma values as
candidate exposure correction algorithms. Gamma correction is a non linear
pixelwise transformation of input image. The intensity of the image pixels
must first be scaled from [0, 255] to [0, 1]. After normalization, the power
law equation as given by Eq. 1 is applied to get the output gamma corrected
image, where $I_{in}$ will be input distorted image and $I_{out}$ will be
output image. Afterwards, the output image is scaled back to the [0, 255]
range. Gamma values less than 1 will make the image appear lighter
(overexposed), while the image becomes darker when gamma values are greater
than 1 (underexposed). The input image will not be affected by a gamma value
of 1. In our experiments, we have used gamma values as inverse of the gamma
values that are used to create the distortion. In theory, the inverse of the
gamma values should restore the image back to its original format but due to
clipping (in case when pixel values go beyond the true ranges) and other
restraints we tend to see a drop in performance. We used the values of 0.33,
0.5, 1.25 and 5.0 for exposure correction.

###  5.5 Training

We trained this entire approach in an end-to-end fashion. The ResNet-50
feature extractor was initialized with pretrained ImageNet weights and the
Fully Connected Network (FCN) heads corresponding to different distortion
types were randomly initialized (see Fig. 2). We used the proposed dataset to
train this setup to identify the distortion types present in the image. This
enabled the ResNet-50 feature extractor and the FCN heads to learn to capture
relevant information about the distortions present in the image. The input to
the network was the distorted image and the output was the identification of
the latest distortion that the image had undergone. We used Binary Cross
Entropy Loss (BCE) for each of the FCN heads with label 1 for the latest
distortion and 0 for the rest. We add the BCE losses for each head with equal
weightage due to reasons stated in [2]. The FCN hidden units for each of the
different distortion task was (2048, 512, 512, 1) with Leaky ReLU activation
function. The learning rate was fixed to be 0.0001 for the entire training
period using Adam optimizer with a weight decay of 0.0005.

To summarise the training process with an example, if an image had undergone
two levels of distortion namely, underexposure and addition of low-level
noise, the network would at first predict low-level noise in the image. As a
result the head corresponding to low-level noise would be regressed with the
label=1 and the rest of the heads with label=0 and the sum of the losses would
be used while backpropogating.

##  6 Experiments

In this section, we first introduce four baselines and compare their
performance with our approach on object detection as the downstream vision
task. We use Pytorch’s implementation of FasterRCNN [26] as the object
detector which reports a map score of 0.3959 on the clean images of our
training dataset and 0.3955 on the clean images of our test dataset. Finally,
we conduct a series of ablation experiments to analyse the behavior of the
proposed framework with new distorted image datasets that the network had not
been trained on. While comparing our approach with the four baselines, we make
sure that the trainable parameters (including seed) and the training
conditions remain similar amongst all.

###  6.1 Hard-coded Distortion Classifier Model (HC-C)

This baselines is inspired from the MAPF-Algorithms that train a image
classification neural network to directly predict the choice of the algorithm.
In this baseline we train a vanilla ResNet-50 feature extractor with a fully
connected prediction network to classify the latest distortions present in the
image. The ResNet-50 backbone was pretrained on ImageNet dataset and the last
classification layer was initialized randomly. We use Cross Entropy Loss with
a learning rate of 0.0001 alongside Adam optimizer with a weight decay of
0.0005. The hidden units of the prediction network comprise of (2048, 1024,
256, 64, 5) neurons with Leaky ReLU activation. The last layer comprises of 5
neurons because we consider 5 distortion types namely underexposed image,
overexposed image, low-level noise image, high-level noise image and clean
image. In case of algorithm selection, we mapped each of the distortions to
the best correction algorithm for the identified distortion in the image. In a
way, this is a rule based selection so we call this baseline Hard-Coded
Distortion Classifier Model.

###  6.2 Random Multi-Task Learning Distortion Model (R-MTL)

In this baseline, we extend the prior work [2] to train a vanilla ResNet-50
backbone pretrained on ImageNet dataset as a feature extractor. Subsequently,
we train each head to identify a distortion type present in the image.
Further, we map each of these heads to a set of candidate algorithms that can
correct the distorted image. While choosing the correction algorithm, we
uniformly sample one of the candidate correction algorithms (capable to
correct the identified distortion type) to rectify the distorted image. The
network architecture is the same as Fig.2. We trained 5 heads with hidden
units comprising of (2048, 512, 512, 1) neurons with Leaky ReLU activation.

###  6.3 Hard-Coded Multi-Task Learning Algorithm Selection Model (HC-MTL)

This baseline’s architecture is similar to the above Random Multi-Task
Learning Distortion Model, however, based on the identified distortion type,
we use the right algorithm for correcting the distortion. As a result, we
treat this baseline as the oracle algorithm. Since this is a heuristic based
algorithm selector, we name this baseline Hard-Coded Multi-Task Learning
Algorithm Selection Model.

###  6.4 Fixed Pipeline

We introduce another baseline for comparison wherein we fix the image
processing sequence irrespective of the image type to replicate the most
commonly used methodoly to do image processing for images in the dataset. The
Fixed Pipeline 1 in our case is Exposure Correction (Gamma Correction with
$\gamma=2.0$) followed by Denoising (SCUNet-15) and Fixed Pipeline 2 is the
opposite arrangement of Fixed Pipeline 1. As discussed earlier, certain
algorithm arrangements might not be suitable for a given goal task and can
have deteriorating effects on performance as highlighted in case of Fixed
Pipeline 2 (refer Table 1).

###  6.5 Distortion Identification Accuracy

The distortion identification accuracy of each of HC-C and MTL based methods
is 94% and 99.6% on the train dataset respectively, and 84.99% and 99.5% on
the test dataset respectively.

Table 1: Comparison between mAP score and normalized mAP score of the baseline
algorithms and our approach when Faster-RCNN is used as the object detection
model. The normalized mAP score denotes the placement of the given approach on
a normalised scale between Fixed Pipeline 1 (a lower bound) and the Oracle
(the upper bound with prior knowledge of the distortion).

Algorithm | mAP score (Train) | mAP score (Test) | Normalised mAP (Train) | Normalised mAP (Test)
---|---|---|---|---
| HC-MTL (Oracle)
---
0.2444 | 0.2432 | 100% | 100%
DeepClean | 0.2402 | 0.2395 | 95.1% | 96.2%
| R-MTL
---
0.2238 | 0.2132 | 75.8% | 69.2%
| HC-C
---
0.2234 | 0.2001 | 75.3% | 55.8%
| Fixed Pipeline 1
---
0.1593 | 0.1457 | 0% | 0%
| Fixed Pipeline 2
---
0.0 | 0.0 | - | -

###  6.6 DeepClean and Baselines Discussion

From Table 1, it is apparent that our proposed algorithm, DeepClean
outperforms other baselines by a significant margin on the train and test
dataset. HC-MTL, R-MTL and DeepClean are fairly resilient to newer forms of
distortions of the same kind (see discussion above). It is also worth noting
that DeepClean framework is not trained on algorithm generated images however
it is still able to identify a good correction algorithm to restore the image
back. Since, DeepClean does not impose any restrictions or assumptions on the
set of candidate algorithms it becomes possible to introduce new algorithms
directly into the framework which makes it a plug-and-play module.

##  7 Conclusion & Future Work

In this paper, we have proposed a framework for distortion identification and
selecting a corresponding correction algorithm to rectify the identified
distortion. The proposed framework comprises of a feature extractor that is
shared across the distortion identification network. The feature extractor
learns to extract important information that is relevant to the different
distortions present in the image. Each head in the identification network
makes use of these captured features to identify a unique type of distortion
in the input image. The experimental results showed that our proposed
framework provides better performance on the object detection task. The choice
of framework not only allows it to be scalable with respect to the number of
distortion types but also with their corresponding candidate algorithms. As
future work, firstly we are investigating ways to learn to capture better
feature representations learnt by the network that can further aid the
framework. Secondly, we intend to extend it to other kinds of distortions and
correction algorithms. Finally, we want to investigate the use of
autoregressive architectures like LSTMs [13] and Transformers [30] for
distortion identification and algorithm selection

## References

  * Ahn et al. [2018] N. Ahn, B. Kang, and K.-A. Sohn.  Image distortion detection using convolutional neural network, 2018.  URL https://arxiv.org/abs/1805.10881.
  * Ameur et al. [2021] Z. Ameur, S. A. Fezza, and W. Hamidouche.  Deep multi-task learning for image/video distortions identification.  _Neural Computing and Applications_ , 34, 10 2021.  10.1007/s00521-021-06576-5.
  * Bianco et al. [2021] S. Bianco, L. Celona, and P. Napoletano.  Disentangling image distortions in deep feature space.  _Pattern Recognition Letters_ , 148:128–135, aug 2021.  10.1016/j.patrec.2021.05.008.  URL https://doi.org/10.1016%2Fj.patrec.2021.05.008.
  * Buczkowski and Stasiński [2019] M. Buczkowski and R. Stasiński.  Convolutional neural network-based image distortion classification.  In _2019 International Conference on Systems, Signals and Image Processing (IWSSIP)_ , pages 275–279, 2019.  10.1109/IWSSIP.2019.8787212.
  * Chetouani et al. [2012] A. Chetouani, A. Beghdadi, and M. Deriche.  A hybrid system for distortion classification and image quality evaluation.  _Signal Processing: Image Communication_ , 27(9):948–960, 2012.  ISSN 0923-5965.  https://doi.org/10.1016/j.image.2012.06.001.  URL https://www.sciencedirect.com/science/article/pii/S0923596512001105.
  * Chien [1994] S. Chien.  Using ai planning techniques to automatically generate image processing procedures: A preliminary report.  In _Proceedings of the Second International Conference on AI Planning Systems, Chicago, IL_ , pages 219–224, 1994.
  * CHOUDHURY et al. [2023] A. R. CHOUDHURY, V. Vatsal, M. Rangarajan, N. K. B. Anitha, A. Kapoor, J. R. G. LAKSHMINARASIMHA, A. Saravanan, V. Sengar, B. Purushothaman, A. Pal, et al.  Methods and systems for autonomous task composition of vision pipelines using an algorithm selection framework, Nov. 23 2023.  US Patent App. 18/199,708.
  * Clouard et al. [1999] R. Clouard, A. Elmoataz, C. Porquet, and M. Revenu.  Borg: A knowledge-based system for automatic generation of image processing programs.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 21(2):128–144, 1999.
  * Deng et al. [2009] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE Conference on Computer Vision and Pattern Recognition_ , pages 248–255, 2009.  10.1109/CVPR.2009.5206848.
  * Falk et al. [2007] T. H. Falk, Y. Guo, and W.-Y. G. Chan.  Improving robustness of image quality measurement with degradation classification and machine learning.  _2007 Conference Record of the Forty-First Asilomar Conference on Signals, Systems and Computers_ , pages 503–507, 2007.
  * Fardo et al. [2016] F. A. Fardo, V. H. Conforto, F. C. de Oliveira, and P. S. Rodrigues.  A formal evaluation of psnr as quality measurement parameter for image segmentation algorithms, 2016.  URL https://arxiv.org/abs/1605.07116.
  * He et al. [2015] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition, 2015.  URL https://arxiv.org/abs/1512.03385.
  * Hochreiter and Schmidhuber [1997] S. Hochreiter and J. Schmidhuber.  Long short-term memory.  _Neural computation_ , 9(8):1735–1780, 1997.
  * Huang et al. [2019] C. Huang, T. Jiang, and M. Jiang.  Encoding distortions for multi-task full-reference image quality assessment.  In _2019 IEEE International Conference on Multimedia and Expo (ICME)_ , pages 1864–1869, 2019.  10.1109/ICME.2019.00320.
  * Huang et al. [2016] G. Huang, Z. Liu, L. van der Maaten, and K. Q. Weinberger.  Densely connected convolutional networks, 2016.  URL https://arxiv.org/abs/1608.06993.
  * Kang et al. [2015] L. Kang, P. Ye, Y. Li, and D. Doermann.  Simultaneous estimation of image quality and distortion via multi-task convolutional neural networks.  In _2015 IEEE International Conference on Image Processing (ICIP)_ , pages 2791–2795, 2015.  10.1109/ICIP.2015.7351311.
  * Kapoor et al. [2022] A. Kapoor, N. George, V. Sengar, V. Vatsal, and J. Gubbi.  Auto-transrl: Autonomous composition of vision pipelines for robotic perception.  _arXiv preprint arXiv:2209.02991_ , 2022.
  * Lansky et al. [1995] A. Lansky, M. Friedman, L. Getoor, S. Schmidler, and N. Short Jr.  The collage/khoros link: Planning for image processing tasks.  _Integrated Planning Applications_ , pages 67–76, 1995.
  * Lin et al. [2014] T. Lin, M. Maire, S. J. Belongie, L. D. Bourdev, R. B. Girshick, J. Hays, P. Perona, D. Ramanan, P. Doll’a r, and C. L. Zitnick.  Microsoft COCO: common objects in context.  _CoRR_ , abs/1405.0312, 2014.  URL http://arxiv.org/abs/1405.0312.
  * Ma et al. [2018] K. Ma, W. Liu, K. Zhang, Z. Duanmu, Z. Wang, and W. Zuo.  End-to-end blind image quality assessment using deep neural networks.  _IEEE Transactions on Image Processing_ , 27(3):1202–1213, 2018.  10.1109/TIP.2017.2774045.
  * Moorthy and Bovik [2010] A. K. Moorthy and A. C. Bovik.  Statistics of natural image distortions.  In _2010 IEEE International Conference on Acoustics, Speech and Signal Processing_ , pages 962–965, 2010.  10.1109/ICASSP.2010.5495298.
  * Nadarajan et al. [2011] G. Nadarajan, Y.-H. Chen-Burger, R. B. Fisher, and C. Spampinato.  A flexible system for automated composition of intelligent video analysis.  In _2011 7th International Symposium on Image and Signal Processing and Analysis (ISPA)_ , pages 259–264. IEEE, 2011.
  * Paszke et al. [2019] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In _Advances in Neural Information Processing Systems 32_ , pages 8024–8035. Curran Associates, Inc., 2019.  URL http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf.
  * Peng and Li [2012] P. Peng and Z. Li.  Image quality assessment based on distortion-aware decision fusion.  In Y. Zhang, Z.-H. Zhou, C. Zhang, and Y. Li, editors, _Intelligent Science and Intelligent Data Engineering_ , pages 644–651, Berlin, Heidelberg, 2012. Springer Berlin Heidelberg.  ISBN 978-3-642-31919-8.
  * Ren et al. [2021] J. Ren, V. Sathiyanarayanan, E. Ewing, B. Senbaslar, and N. Ayanian.  Mapfast: A deep algorithm selector for multi agent path finding using shortest path embeddings, 2021.  URL https://arxiv.org/abs/2102.12461.
  * Ren et al. [2017] S. Ren, K. He, R. Girshick, and J. Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 39(6):1137–1149, 2017.  10.1109/TPAMI.2016.2577031.
  * Sengar et al. [2021] V. Sengar, J. Gubbi, A. Rajbhoj, and P. Balamuralidhar.  Automatic synthesis of computer vision workflow using ai planning.  In _2021 IEEE International Conference on Imaging Systems and Techniques (IST)_ , pages 1–6. IEEE, 2021.
  * Sigurdson and Bulitko [2017] D. Sigurdson and V. Bulitko.  Deep learning for real-time heuristic search algorithm selection.  In _Artificial Intelligence and Interactive Digital Entertainment Conference_ , 2017.
  * Sigurdson et al. [2019] D. Sigurdson, V. Bulitko, S. Koenig, C. Hernandez, and W. Yeoh.  Automatic algorithm selection in multi-agent pathfinding, 2019.  URL https://arxiv.org/abs/1906.03992.
  * Vaswani et al. [2017] A. Vaswani, N. M. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin.  Attention is all you need.  In _Neural Information Processing Systems_ , 2017.  URL https://api.semanticscholar.org/CorpusID:13756489.
  * Wang et al. [2016] H. Wang, L. Zuo, and J. Fu.  Distortion recognition for image quality assessment with convolutional neural network.  In _2016 IEEE International Conference on Multimedia and Expo (ICME)_ , pages 1–6, 2016.  10.1109/ICME.2016.7552936.
  * Zhang et al. [2022] K. Zhang, Y. Li, J. Liang, J. Cao, Y. Zhang, H. Tang, R. Timofte, and L. Van Gool.  Practical blind denoising via swin-conv-unet and data synthesis.  _arXiv preprint arXiv:2203.13278_ , 2022.