rightsretained

# Grounding Explainability Within the Context of Global South in XAI

Deepa Singh

Michal Slupczynski

Ajit G. Pillai

Vinoth Pandian Sermuga Pandian

Department of Philosophy  University of Delhi, India<EMAIL_ADDRESS>RWTH Aachen University, Germany<EMAIL_ADDRESS>Affetive
Interactions Lab  The University of Sydney, Australia
<EMAIL_ADDRESS>RWTH Aachen University  Aachen, Germany
<EMAIL_ADDRESS>
(2020)

###### Abstract

In this position paper, we propose building a broader and deeper understanding
around Explainability in AI by ‘grounding’ it in social contexts, the socio-
technical systems operate in. We situate our understanding of grounded
explainability in the ‘Global South’ in general and India in particular and
express the need for more research within the global south context when it
comes to explainability and AI.

###### doi:

https://hcxai.jimdosite.com/

###### keywords:

XAI; Global South; HCID; HCXAI; Human AI Interaction.

††conference: CHI’22, Workshop on Human-Centered Explainable AI (HCXAI):
Beyond Opening the Black-Box of AI May 12–13, 2022, New Orleans, LA, USA

<ccs2012> <concept> <concept_id>10003120.10003121</concept_id>
<concept_desc>Human-centered computing Human computer interaction
(HCI)</concept_desc> <concept_significance>500</concept_significance>
</concept> <concept>
<concept_id>10003120.10003121.10003125.10011752</concept_id>
<concept_desc>Human-centered computing Haptic devices</concept_desc>
<concept_significance>300</concept_significance> </concept> <concept>
<concept_id>10003120.10003121.10003122.10003334</concept_id>
<concept_desc>Human-centered computing User studies</concept_desc>
<concept_significance>100</concept_significance> </concept> </ccs2012>

[500]Human-centered computing Human computer interaction (HCI) [300]Human-
centered computing Haptic devices [100]Human-centered computing User studies

##  1 Introduction

Explainable AI (XAI) is a form of artificial intelligence (AI) that provides
additional information to illustrate the decision process of a machine
learning algorithm. An AI system can provide the end-users with explanations
to increase interactivity by allowing them to re-enact and retrace AI/ML
outcomes, for example, to verify results for plausibility [10]. An extension
to this definition is the concept of Responsible Artificial Intelligence
(RAI), which describes an AI methodology with fairness, model explainability,
and accountability at its core [4]. Though there are clear definitions
available in XAI, there are different perspectives on XAI depending on
community context [7] as a result of which there are various ways to
operationalize and evaluate XAI technology.

Explainability, interpretability, intelligibility, and transparency have been
used interchangeably [4, 3, 16, 7]. To comprehend the formative and
substantive human components of XAI systems, as well as the context of the
many stakeholders who use the system, a contextual sociotechnical approach is
required [6]. In this paper, we propose building a broader and deeper
understanding of Explainability by ‘grounding’ it in the social contexts in
which these socio-technical systems operate. We situate our understanding of
grounded explainability in the ‘Global South’ in general and India in
particular. The vast majority of the articles written on XAI were concentrated
on the western counterparts [11]. One of the major existing issues with XAI
systems within contexts of the global south, especially in rural areas, is
that most of the current systems are built with small amounts of
explainability or with non-explainability, clearly showcasing the power
structures between the developers of the system and marginalized communities
using these systems. Ehsan and Reidal [6], paint a picture of consequential
technological systems used in society in different use cases are nested in
social relationships. There is a pattern of neglecting this socially situated
aspect within many AI and XAI contexts, especially when engineers are
disconnected from the users, resulting in a half baked, partial and failed
image of the involved information [6]. Rather than focusing on how people
interact with technologies, XAI should focus on what explainability, autonomy,
and control mean to individuals with diverse backgrounds. Research and
analysis of what these concepts imply are necessary to acquire a better grasp
of how explainability might be fostered more successfully within these groups.
In this ongoing investigation, we focus conducting research with a
transformative worldview and mixed methods research with quantitative and
qualitative studies for ‘Grounding of explainability’ within the context of
global south by centering the local communities is the approach we take in
this paper to further understand what does such a grounding demand of us. How
can we build the concept of Explainability or XAI in such a way that it leads
to a tangible empowerment of local communities; as citizens, political
participants, users and stakeholders? These are the questions we look to work
on in the future. This workshop acts as a spring board for us, to have a
discourse around this topic with fellow researches, gain feedback on our
direction, resulting in a more informed research design for us to take
forward. Our aim for the research to be conducted would be to look at what
role XAI plays in enabling of trust and democratic participation by enabling
communities in seeking accountability.

##  2 Background

There is a massive body of research that builds the case that though AI has
benefits, it also has a panoply of potential harms associated with it.
Research has also highlighted, that AI systems disproportionately harm the
vulnerable, poor and the marginalized sections of society [17, 5, 8]. In
addition, the development, research, and design of AI are mostly
“West”-centric / “Global North”-centric [18, 15]. In this paper, we not only
highlight the need to go beyond the established, Euro-America dominated the
understanding of AI Explainability but we need to ground it in the real-time
socio-political contexts such that it not only steers growth, development, and
trust but also promotes democratic values while centering the worst-off of the
society.

AI is currently under deployment in various sectors across India. According to
NITI Aayog’s (public policy think-tank of the Government of India) Approach
Document for India, “[…] it is expected that AI usage will become ingrained
and integrated with society. In India, large-scale applications of AI are
being trialed every day across sectors.” [2]. There is evidence of AI being
increasingly used in policing [13], development of the highly aspirational
‘smart city’ projects, and even in welfare administration [9].

There is a much-publicized government flagship project which aims to deploy AI
and a host of other emerging technologies in India’s agricultural sector [12].
In such a scenario, where incredible aspiration driving large scale AI
deployments throughout the country, we wish to contextualize the pressing need
for practical models / understanding / definitions of Explainability by
‘grounding’ its meaning and relevance in the well-being of the worst-off in
the Indian society who are predominantly situated in rural India. The idea
behind XAI is to render it transparent, open to scrutiny, and foster public
trust. In this manner, XAI thus is also an effort in the direction of
promoting and nurturing democratic values in societies, we argue. The
fundamentals of XAI should therefore be grounded in the socio-political
contexts of deployment to be enabling and have a tangible ground-level impact.
The question that arises is what does Explainability mean in the context of
the under-served, marginalized, and unlettered populations situated in the
Global South, specifically, rural India? India, at present, does not even have
a basic data protection framework. The ‘pacing gap’; between innovation and
regulation is vast. Pre-existing institutional mechanisms for enabling
transparency and accountability in sociotechnical systems are either weak or
wholly absent. In want of adequate institutional mechanisms which can enable
explainability and recourse post-hoc, we as the research community have to
actively look into developing notions of explainability which could be grasped
and exercised by the rural populace in India / the Global South. This is the
central question that our research aims to tackles by proposing how and to
what extent can we achieve such ‘grounding’.

##  3 Research Direction

In this research, we focus on looking at XAI in a transformative worldview. To
tackle social injustice at its various levels, a transformative worldview
emphasizes that academic inquiry must be interwoven with politics and a
political transformation agenda, which also implies that the inquirer will
ensure that the participants are not more marginalized as a result of the
investigation [14].

We aim to conduct a mixed-method study with marginalized communities currently
using AI systems with a participatory approach to understanding their problems
with explainable AI and co-designing what explainability means to them when it
comes to interacting with consequential AI systems. We plan to scope the XAI
based on the problem domain where the tool disturbs our participant’s
livelihood — for example, AI for loan calculation by banks or AI for seed
distribution to farmers. From this formative research, we then plan to modify
the XAI system and create a framework to provide an impactful solution to the
target participants. Further, we will evaluate this system with the same
community to assess its impact on the problem domain.

##  4 About the authors

Deepa Singh is a PhD researcher at the Department of Philosophy, University of
Delhi, where she is an Indian Council of Philosophical Research (ICPR) Fellow.
Her research focuses on contextualising and (re)interpreting AI Ethics
principles within the Indian context, and is situated at the intersections of
Philosophy, HCI and AI.

Michal Slupczynski is a researcher and PhD student at the Chair for Computer
Science 5 at RWTH Aachen University. He conducts research in infrastructures
for multimodal learning environments.

Ajit G. Pillai is a researcher and PhD student at Design Lab in The University
of Sydney. He regularly conducts research on the ethics of emerging
technologies in HCI in industries such as healthcare. He regularly conducts
his research with older adults and children within the health and well-being
context. His PhD looks into the ethics of technology design.

Vinoth Pandian is a HCI researcher and full stack web developer at MeetAnyway
GmbH. He has a PhD in human centered artificial intelligence from RWTH Aachen
university. His research focus is on the intersection of HCI and AI.

## References

  * [1]
  * [2] 2021\.  Responsible AI #AIFor All: Approach document for India.  (Feb 2021).  https://webdev24x7.com/niti/sites/default/files/2021-02/Responsible-AI-22022021.pdf
  * [3] Ashraf Abdul, Jo Vermeulen, Danding Wang, Brian Y Lim, and Mohan Kankanhalli. 2018.  Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In Proceedings of the 2018 CHI conference on human factors in computing systems. 1–18.
  * [4] Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador García, Sergio Gil-López, Daniel Molina, Richard Benjamins, and others. 2020.  Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI.  Information fusion 58 (2020), 82–115.
  * [5] Meredith Broussard. 2018.  Artificial unintelligence: How computers misunderstand the world.  mit Press.
  * [6] Upol Ehsan and Mark O Riedl. 2020.  Human-centered explainable ai: towards a reflective sociotechnical approach. In International Conference on Human-Computer Interaction. Springer, 449–466.
  * [7] Upol Ehsan, Philipp Wintersberger, Q Vera Liao, Martina Mara, Marc Streit, Sandra Wachter, Andreas Riener, and Mark O Riedl. 2021.  Operationalizing human-centered perspectives in explainable AI. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. 1–6.
  * [8] Virginia Eubanks. 2018.  Automating inequality: How high-tech tools profile, police, and punish the poor.  St. Martin’s Press.
  * [9] Rohin Garg. 2022.  #privacyofthepeople - Agristack and farmers’ issues.  (Feb 2022).  https://internetfreedom.in/privacy-of-the-people-agristack-and-farmer-issues/
  * [10] Andreas Holzinger, André Carrington, and Heimo Müller. 2020.  Measuring the quality of explanations: the system causability scale (SCS).  KI-Künstliche Intelligenz 34, 2 (2020), 193–198.
  * [11] Mir Riyanul Islam, Mobyen Uddin Ahmed, Shaibal Barua, and Shahina Begum. 2022.  A systematic review of explainable artificial intelligence in terms of different application domains and tasks.  Applied Sciences 12, 3 (2022), 1353.
  * [12] Divij Joshi. 2022.  The legal, institutional, & Technical architecture of ADMS in India.  (Jan 2022).  https://ai-observatory.in/Section1.pdf
  * [13] Vidushi Marda and Shivangi Narayan. 2020.  Data in New Delhi’s predictive policing system. In Proceedings of the 2020 conference on fairness, accountability, and transparency. 317–324.
  * [14] Donna M Mertens. 2010.  Philosophy in mixed methods teaching: The transformative paradigm as illustration.  International Journal of Multiple Research Approaches 4, 1 (2010), 9–18.
  * [15] Sabelo Mhlambi. 2020.  From rationality to relationality: ubuntu as an ethical and human rights framework for artificial intelligence governance.  Carr Center for Human Rights Policy Discussion Paper Series 9 (2020).
  * [16] Sina Mohseni, Niloofar Zarei, and Eric D Ragan. 2021.  A multidisciplinary survey and framework for design and evaluation of explainable AI systems.  ACM Transactions on Interactive Intelligent Systems (TiiS) 11, 3-4 (2021), 1–45.
  * [17] Cathy O’neil. 2016.  Weapons of math destruction: How big data increases inequality and threatens democracy.  Broadway Books.
  * [18] Nithya Sambasivan, Erin Arnesen, Ben Hutchinson, Tulsee Doshi, and Vinodkumar Prabhakaran. 2021.  Re-imagining algorithmic fairness in india and beyond. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. 315–328.


Population synthesis of Galactic pulsars with machine learning Prof. Nanda Rea
Dr. Vanessa Graber Doctor of Philosophy in Physics Physics and astrophysics
UNIVERSITAT AUTÒNOMA DE BARCELONA
DEPARTAMENT DE FÍSICA Institute of Space Sciences (ICE-CSIC/IEEC)

Doctoral Thesis

_Author:_

_Advisors:_

_Tutor:_
Prof. Lluís Font Guiteras

A thesis submitted for the degree of

Barcelona, December 2023

###### Abstract

Abstract

The recent advancements in data availability and computational power have
allowed the development of new machine-learning algorithms and data-analysis
techniques that have opened up new avenues to study the Galactic neutron-star
population.

This thesis work represents the first efforts to combine population synthesis
studies of the Galactic isolated neutron stars with deep-learning techniques
with the aim of better understanding neutron-star birth properties and
evolution.

In particular, we develop a flexible population-synthesis framework to model
the dynamical and magneto-rotational evolution of neutron stars, their
emission in radio and their detection with radio telescopes. We first study
the feasibility of using deep neural networks to infer the dynamical
properties at birth of a simulated population of neutron stars from density
maps storing the information on their sky position and proper motion. We then
explore a simulation-based inference approach to constrain our physical models
from the observed radio pulsar population. We employ a neural density
estimator to predict the posterior distribution of the parameters describing
the birth magnetic-field and spin-period distributions and the late-time
magnetic-field decay from density maps containing information on the spin
period and spin-period derivative. Our results for the initial magneto-
rotational properties agree with the findings of previous works while we
constrain the late-time evolution of the magnetic field in neutron stars for
the first time.

Besides the development of the population-synthesis framework, this thesis
also studies possible scenarios to explain the puzzling nature of recently
discovered periodic radio sources with very long periods of the order of
thousands of seconds.

In particular, by assuming a neutron-star origin, we study the spin-period
evolution of a newborn neutron star interacting with a supernova fallback disk
and find that the combination of strong, magnetar-like magnetic fields and
moderate accretion rates can lead to very large spin periods on timescales of
ten thousands of years. Moreover, we perform population synthesis studies to
assess the possibility for these sources to be either neutron stars or
magnetic white dwarfs emitting coherently through magnetic dipolar losses.
These discoveries have opened up a new perspective on the neutron-star
population and have started to question our current understanding of how
coherent radio emission is produced in pulsar magnetospheres.

Overall this thesis work represents the first step towards developing a
simulation framework able to model the different neutron star classes in a
unified scenario and constraining the properties of the neutron-star
population as a whole from multi-wavelength observations through machine
learning.

## This thesis is based on the following publications:

  * •

[Ronchi2021]
“Analyzing the Galactic Pulsar Distribution with Machine Learning”
Published in: ApJ 916.2, 100, p. 100

  * •

[Ronchi2022]
“Long-period Pulsars as Possible Outcomes of Supernova Fallback Accretion”
Published in: ApJ 934.2, 184, p. 184

  * •

[Rea2022]
“Constraining the Nature of the 18 min Periodic Radio Transient GLEAM-X
J162759.5-523504.3 via Multiwavelength Observations and Magneto-thermal
Simulations”
Published in: ApJ 940.1, 72, p. 72

  * •

[Rea2023]
“A long-period radio transient active for three decades: population study in
the neutron star and white dwarf rotating dipole scenarios”
Accepted for publication in ApJ

  * •

[Graber2023]
“Simulation-based inference for pulsar population synthesis”
Submitted to ApJ

The research reported in this thesis was carried out at the Institute of Space
Sciences (ICE – CSIC), within the framework of the doctoral program in Physics
at the Autonomous University of Barcelona (UAB). It was supported by the ERC
via the Consolidator Grant “MAGNESIA” (No. 817661, PI: Nanda Rea), by the
program Unidad de Excelencia María de Maeztu CEX2020-001058-M and partially by
the grant SGR2021-01269 (PI: Vanessa Graber).

Cover image © by Olena Shmahalo

###### Acknowledgements.

These past four years, besides being challenging, have been very enjoyable,
stimulating and rewarding in a lot of aspects and I am very grateful to all
the people that I met and who shared this part of my life with me. First of
all I would like to deeply thank my supervisors Nanda and Vanessa. Your
guidance, exceptional expertise, enthusiasm, support and understanding have
been fundamental during these years. I learnt a lot from you both from a
scientific and a human point of view. Thanks to you I became more aware of
both the positive aspects and drawbacks of academia and your advices are going
to be incredibly valuable for choosing my future path. People who have
previously been and are currently part of the MAGNESIA group are all
fantastic. The atmosphere has always been very friendly and supportive. Thanks
to the entire Italian crew (Alice, Alessio, Francesco, Stefano, Daniele,
Arianna and the new-comers Davide, Thomas and Anna) who always made me feel
like at home and to all the people coming from Spain and other parts of the
world (Albert I and Albert II, Celsa, Clara, Abu, Emilie, Kostas, Christine
and Raj) who opened up my horizons. I feel enriched of having met so many
interesting people with their diverse perspectives, vast knowledge and
experience. I learned a lot from all of you. Special thanks to my PhD mates,
Abu for your kindness and wisdom in every situation, Clara for your incredible
determination and Celsa, also team-mate (and Spanish teacher), for your
contagious enthusiasm and authenticity in everything you do. Together with
Vanessa, the harmony and synergy in our population-synthesis group was
fantastic and this thesis is only possible thanks to such an amazing team-
work. I extend my gratitude to all the people I met at ICE, a very
international and stimulating environment with a lot of PhD students sharing
the same journey and people at PIC, for all the helpful conversations on the
computing side. I would further like to thank my examiners, Ben Stappers,
Alessandro Patruno and Helena Dominguez Sanchez, who took their time to read
this thesis and give constructive feedback that helped me to improve this
work. Lastly, I would like to thank my family, my mum, dad, brother and my
grandmother and all my friends back home for their support, encouragement and
understanding in all these years.

lr@${}={}$l

Speed of light in a vacuum & $c$ $2.99792458\times
10^{10}\,\mathrm{cm\,s^{-1}}$
Gravitational constant $G$ $6.67259\times
10^{-8}\,\mathrm{cm^{3}\,g^{-1}\,s^{-2}}$
Planck constant $h$ $6.6260755\times 10^{-27}\,\mathrm{erg\,s}$
Electron charge $e$ $4.8032068\times 10^{-10}\,\mathrm{esu}$
Electron mass $m_{e}$ $9.1093897\times 10^{-28}\,\mathrm{g}$
Proton mass $m_{p}$ $1.6726231\times 10^{-24}\,\mathrm{g}$
Neutron mass $m_{n}$ $1.6749286\times 10^{-24}\,\mathrm{g}$
Boltzmann constant $k_{\rm B}$ $1.380658\times 10^{-16}\,\mathrm{erg\,K^{-1}}$
Thomson cross-section $\sigma_{\rm T}$ $6.65245\times
10^{-25}\,\mathrm{cm^{-2}}$

Solar mass $M_{\odot}$ $1.989\times 10^{33}\,\mathrm{g}$
Parsec pc $3.0856\times 10^{18}\,\mathrm{cm}$
Jansky Jy $1.0\times 10^{-23}\,\mathrm{erg\,cm^{-2}\,s^{-1}\,Hz^{-1}}$
Year yr $3.1557\times 10^{7}\,\mathrm{s}$


# Evolution of coherent waves driving a single artificial atom

A. V. Vasenin<EMAIL_ADDRESS>Skolkovo Institute of Science and
Technology, Nobel Street 3, 143026 Moscow, Russia  Laboratory of Artificial
Quantum Systems, Moscow Institute of Physics and Technology, 141701
Dolgoprudny, Russia  Sh. V. Kadyrmetov  Laboratory of Artificial Quantum
Systems, Moscow Institute of Physics and Technology, 141701 Dolgoprudny,
Russia  A. N. Bolgar  Laboratory of Artificial Quantum Systems, Moscow
Institute of Physics and Technology, 141701 Dolgoprudny, Russia  A. Yu.
Dmitriev  Laboratory of Artificial Quantum Systems, Moscow Institute of
Physics and Technology, 141701 Dolgoprudny, Russia  O. V. Astafiev  Skolkovo
Institute of Science and Technology, Nobel Street 3, 143026 Moscow, Russia
Laboratory of Artificial Quantum Systems, Moscow Institute of Physics and
Technology, 141701 Dolgoprudny, Russia

###### Abstract

An electromagnetic wave propagating through a waveguide with a strongly
coupled superconducting artificial two-level atom exhibits an evolving
superposition with the atom. The Rabi oscillations in the atom result from a
single excitation-relaxation, corresponding to photon absorption and
stimulated emission from/to the field. In this study, we investigate the time-
dependent behavior of the transmitted field and extract its spectra. The
scattered fields are described using input-output theory. We demonstrate that
the time evolution of the propagating fields, due to interaction, encapsulates
all information about the atom. Additionally, we deduce the dynamics of the
incoherent radiation component from the measured first-order correlation
function of the field.

††preprint: APS/123-QED

Introduction. —  Waveguide Quantum Electrodynamics (w-QED) involves artificial
atoms that couple strongly to the propagating field [1, 2, 3, 4]. The first
successful experiments with superconducting artificial atoms established the
foundation for w-QED [5, 6]. Subsequently, the focus shifted to more complex
systems, including quantum optical effects with three-level atoms [7, 8, 9]
and scattering by many qubits coupled to a waveguide [10, 11, 12]. Researchers
demonstrated the use of coherent and incoherent emission for inferring a qubit
state [13]. Other practical applications of qubit-waveguide systems include
the creation of efficient single-photon sources [14, 15, 16], and coupling two
distant qubits by a waveguide to observe their entanglement [10, 17]. Despite
reported prototypes [18, 19, 20], robust detection of gigahertz single photons
remains challenging. Nevertheless, tomographic methods utilizing linear field
detectors reliably characterize microwave fields emitted by superconducting
quantum systems [21, 22]. These methods enable the resolution of peculiar
properties of resonance fluorescence [23, 24], as well as the characterization
of numerous non-Gaussian states of light [25] and non-classical multi-photon
states with time-bin entanglement [26].

Despite the aforementioned achievements, gaps still exist in experimental
characterization of non-stationary resonance fluorescence of a two-level atom
[27, 28]. Researchers primarily study driven two-level systems in the
stationary regime [29], focusing on stationary fluorescence obtained through
measurements of time-averaged spectral properties [8] or stationary radiation
fields [30]. Previous attempts to observe the dynamics of the driving field
were limited to spontaneous emission [31, 32] or very short pulses [33, 34,
35]. However, none of these works offers a complete picture of the phenomenon,
specifically a thorough time-domain characterization of a propagating pulse
that interacts with the atom and then flies away. Nonetheless, such a
technique is necessary for observing the properties of non-stationary
resonance fluorescence.

Figure 1: Stimulated absorption and emission observed in a modulated pulse
shape scattered by a two-level atom. (a) Evolution of the pulse shape as it
travels through a waveguide coupled to a two-level atom. (b) Voltage traces of
the modulated pulse transmitted through a coplanar waveguide with a strongly-
coupled transmon qubit. (c) Modulation caused by stimulated absorption and
emission processes compared to an unmodulated pulse shape.

In this work we study the quantum dynamics of propagating electromagnetic
waves interacting with an artificial atom strongly coupled to the field. Due
to the atom-field superposition produced from this interaction, the scattered
wave carries information about the atomic evolution. Figure 1(a) illustrates a
gedankenexperiment, where a rectangular pulse of a coherent field
$|\alpha\rangle$ with a carrier frequency matching the qubit resonance
traverses an infinite waveguide. After a certain time, it causes photon
absorption and excites the qubit. As a result, the pulse amplitude and power
decrease. However, the continuing pulse stimulates the qubit to emit its
excitation, thus increasing the pulse amplitude and power higher than their
initial magnitudes. These processes repeat until the pulse ends. Decoherence
causes the modulating oscillations to decay. After the pulse passes, the qubit
excitation decays into the waveguide vacuum, producing an exponential tail, a
phenomenon discussed in Ref. [5, 13, 31].

Unlike prior methods [13] that utilized repeated pulse sequences to infer
qubit states from post-pulse spontaneous emission, we directly observe
stimulated photon dynamics between the qubit and the field during pulse drive,
extracting Rabi oscillations, qubit population, and the first-order
correlation function. The first-order correlation function and instantaneous
power density spectra reveal the dynamics of incoherent radiation and confirm
the quantum nature of the observed phenomenon. Notably, as the averaged field
oscillations diminish, the field correlation function retains its
oscillations, highlighting the interplay between coherent and incoherent
fluorescence in a non-stationary regime.

Theory. — The model Hamiltonian for our atom-waveguide system features a two-
level atom symmetrically coupled to the left- and right-moving free field
modes of a one-dimensional waveguide:

$\hat{H}=-\frac{\hbar\omega_{q}}{2}\hat{\sigma}_{z}+\sum_{k=l,r}\Biggl{[}\int
d\omega\hbar\omega\hat{a}^{\dagger}_{k,\omega}\hat{a}_{k,\omega}\\\ +\hbar
g\int
d\omega\left(\hat{a}_{k,\omega}\hat{\sigma}_{+}+\hat{a}^{\dagger}_{k,\omega}\hat{\sigma}_{-}\right)\biggr{]},$
(1)

where $\hbar\omega_{q}$ denotes the qubit energy splitting, $l$ and $r$
indices represent left and right modes, and $g$ is the atomic coupling
constant to the waveguide field modes.

Experimentally, the right-moving mode field is measured. Per the input-output
theory for a tightly coupled atom-waveguide system [36, 37, 38, 39, 40] in the
interaction picture, the field at the right waveguide output $\hat{a}_{o}(t)$
corresponds to the left input field $\hat{a}_{i}(t)$ as:

$\hat{a}_{o}(t)=\hat{a}_{i}(t)+i\sqrt{\frac{\Gamma_{1}}{2}}\hat{\sigma}_{-}(t).$
(2)

We use IQ-mixers which downconvert the signal and produce two field
quadratures: $\hat{V}_{I}(t)=V_{0}(\hat{a}_{o}(t)+\hat{a}^{\dagger}_{o}(t))/2$
and $\hat{V}_{Q}(t)=-iV_{0}(\hat{a}_{o}(t)-\hat{a}^{\dagger}_{o}(t))/2$. Here,
$V_{0}=\sqrt{G\hbar\omega Z_{0}}$, where $G$ represents the amplification
coefficient and $Z_{0}$ is the waveguide impedance. Together, they define the
full output field $\hat{V}_{o}(t)=V_{I}(t)+iV_{Q}(t)=V_{0}\hat{a}_{o}(t)$.

For a coherent rectangular pulse driving an atom, the atom state displays Rabi
oscillations at frequency $\Omega_{R}(t)=\sqrt{2\Gamma_{1}}\alpha(t)$. These
oscillations, notable during the initial pumping $0\leq t\lesssim
1/\Gamma_{1}$, manifest in the averaged output voltage trace:

$\left\langle{\hat{V}_{o}(t)}\right\rangle=V_{0}\left(\alpha(t)+i\sqrt{\Gamma_{1}/2}\left\langle{\hat{\sigma}_{-}(t)}\right\rangle\right).$
(3)

From equation (2), it can be derived that the energy absorbed from the
transmitted pulse in the first half of Rabi period nearly matches the single-
photon energy. The same applies for the energy emitted in the subsequent half
cycle. Conversely, energy reflected in the left direction is minimal.
Analytical calculations are detailed in [41].

While $\left\langle{\hat{V}_{o}(t)}\right\rangle$ reveals the coherent part of
radiation, the non-stationary first-order correlation function exposes the
incoherent component of the field:

$G^{(1)}\left(t_{1},t_{2}\right)=V_{0}^{2}\big{(}\left\langle{\hat{a}^{\dagger}_{o}(t_{1})\hat{a}_{o}(t_{2})}\right\rangle\\\
-\left\langle{\hat{a}^{\dagger}_{o}(t_{1})}\right\rangle\left\langle{\hat{a}_{o}(t_{2})}\right\rangle\big{)}.$
(4)

For a classical coherent wave in a transmission line, without qubit
interaction, the average field is
$\left\langle{\hat{V}_{o}(t)}\right\rangle=V_{0}\alpha(t)$ and
$G^{(1)}\left(t_{1},t_{2}\right)=0$, with $|\alpha(t)|^{2}$ being the photon
rate. With atom interaction, $G^{(1)}\left(t_{1},t_{2}\right)$ becomes:

$G^{(1)}\left(t_{1},t_{2}\right)=\frac{V_{0}^{2}\Gamma_{1}}{2}\big{(}\left\langle{\hat{\sigma}_{+}(t_{1})\hat{\sigma}_{-}(t_{2})}\right\rangle\\\
-\left\langle{\hat{\sigma}_{+}(t_{1})}\right\rangle\left\langle{\hat{\sigma}_{-}(t_{2})}\right\rangle\big{)}.$
(5)

The first-order correlation function highlights the $\hat{a}$ operator
property to maintain a coherent state
$\hat{a}\left|{\alpha}\right\rangle=\alpha\left|{\alpha}\right\rangle$ and
$\hat{a}^{\dagger}$ to disrupt coherence [42]. When evolution operator
$\hat{U}(\delta t)\approx 1-ig\delta
t(\hat{a}(t)\hat{\sigma}_{+}+\hat{a}^{\dagger}(t)\hat{\sigma}_{-})$, assuming
$\alpha(t)g\delta t\ll 1$, acts on the atom-field wavefunction
$\left|{\Psi_{q}(t),\alpha(t)}\right\rangle$, it preserves the coherent state
of the field if

$\hat{U}(\delta
t)\left|{g}\right\rangle\left|{\alpha(t)}\right\rangle\approx(\left|{g}\right\rangle-
ig\alpha(t)\delta t\left|{e}\right\rangle)\left|{\alpha(t)}\right\rangle$ (6)

and spoils it if

$\hat{U}(\delta
t)\left|{e}\right\rangle\left|{\alpha(t)}\right\rangle\approx(\left|{e}\right\rangle\left|{\alpha(t)}\right\rangle-
ig\delta
t\left|{g}\right\rangle\hat{a}^{\dagger}(t)\left|{\alpha(t)}\right\rangle).$
(7)

Therefore, it is expected that $G^{(1)}\left(t,t+\tau\right)\approx 0$ at
$t=\frac{2\pi n}{\Omega_{R}}$ and $G^{(1)}\left(t,t+\tau\right)\neq 0$ at
$t=\frac{\pi(2n-1)}{\Omega_{R}}$, as confirmed by our experiments.

Results and discussion. —  We first measure the averaged voltage trace of a
pulse on resonance with the qubit. Fig. 1(b) displays oscillations near the
front of the pulse, which increase in frequency with pulse amplitude while
maintaining their amplitude consistent. In Fig. 1(c), comparing the pulse
shape when qubit is on-resonance and far-detuned reveals that modulation
begins with a decrease in pulse amplitude, likely signifying photon
absorption. To further investigate this modulation, we subtract the two
measurements.

Figure 2: Radiation of a qubit as a function of detuning of a pulse carrier
frequency ($\omega_{d}$) from the qubit transition frequency ($\omega_{q}$).
The radiation is defined as the difference between a modulated pulse and an
unperturbed pulse. The pulse is rectangular with 120 ns length. (a) Single
traces of radiation for resonance and slightly off-resonance (-30 MHz) cases.
(b) Experimental traces plotted in color as a function of detuning. (c)
Results of a simulation for radiation traces. (d) Magnitude spectra of single
traces taken from (a). Arrows indicate peaks corresponding to Rabi, qubit and
drive frequencies. (e) Experimental spectra plotted in color as a function of
detuning. (f) Simulated spectra plotted in color.

Fig. 2(a) shows such differences for a resonant case (I) and a slightly off-
resonant case (II) when the carrier frequency deviated from the qubit
frequency. We measure and digitally rotate two quadratures of the field so
that the real quadrature is in phase with the pulse, and the imaginary
quadrature has an orthogonal phase. Resonant traces display oscillations
during the pulse drive and an exponential decay post-pulse. Their magnitude
spectra are in Fig. 2(d), with the horizontal axis centered at the pulse
carrier frequency. A distinct peak in the spectra corresponds to post-pulse
oscillations, flanked by two broader peaks. Simulation of
$\left\langle{\hat{\sigma}_{-}(t)}\right\rangle$ identifies these traces as
Rabi oscillations. The sharp peak matches the qubit frequency, while the wide
peaks represent positive and negative Rabi frequencies in the resonant trace.
For the off-resonant trace, the right peak aligns with the positive Rabi
frequency, and the left peak is at 0 Hz, indicating a constant voltage offset
during pulse drive.

Traces were scanned against the detuning $\delta=\omega_{d}-\omega_{q}$ from
the qubit resonance. Experimental results and simulations are in Fig. 2(b-c),
with magnitude spectra in Fig. 2(e-f). Fitting data yielded qubit parameters:
qubit frequency $f_{q}=4.835~{}\text{GHz}$, Rabi frequency
$\Omega_{R}/(2\pi)=19.8~{}\text{MHz}$, decay rate
$\Gamma_{1}/(2\pi)=0.9~{}\text{MHz}$, and dephasing rate
$\gamma/(2\pi)=0.6~{}\text{MHz}$. Measurements were performed not at the qubit
sweet-spot because of a spurious TLS, hence large dephasing. Fig. 2(b-c)
reveals two regions: the first is the qubit radiation during the microwave
drive yielding three spectral peaks at $\omega_{d}-\Omega$, $\omega_{d}$, and
$\omega_{d}+\Omega$ where $\Omega=\sqrt{\Omega_{R}^{2}+\delta^{2}}$. Their
width is mostly defined by pulse length. The second is the spontaneous
emission [31] post-pulse, which yields a peak limited by $\Gamma_{2}$,
consistent at the qubit frequency as the carrier shifts. With known Rabi
frequency and decay rate, voltages were recalibrated to atom-radiated values
on the chip [43].

Solving the evolution of $\left\langle{\hat{\sigma}_{-}(t)}\right\rangle$ in
the dressed basis [44] explains asymmetry of side-band amplitudes at
$\omega_{d}\pm\Omega$ when $\delta\neq 0$. According to this approach, when a
coherent state populates one of the waveguide modes, levels
$\left|{g,n+1}\right\rangle$ and $\left|{e,n}\right\rangle$ hybridize to new
basis states $\left|{+n}\right\rangle$ and $\left|{-n}\right\rangle$ with
energy splitting
$E_{\left|{+n}\right\rangle}-E_{\left|{-n}\right\rangle}=\hbar\Omega$.

$\displaystyle\left|{+n}\right\rangle=\sqrt{\frac{\Omega+\delta}{2\Omega}}\left|{g,n+1}\right\rangle+\sqrt{\frac{\Omega-\delta}{2\Omega}}\left|{e,n}\right\rangle$
(8)
$\displaystyle\left|{-n}\right\rangle=\sqrt{\frac{\Omega-\delta}{2\Omega}}\left|{g,n+1}\right\rangle-\sqrt{\frac{\Omega+\delta}{2\Omega}}\left|{e,n}\right\rangle$
(9)

Amplitudes of the peaks at $\omega_{d}-\Omega$ depend on detuning as

$\left\langle{g,n}|{+(n-1)}\right\rangle\left\langle{+(n-1)}\right|\hat{\sigma}_{-}\left|{-n}\right\rangle\cdot\\\
\cdot\left\langle{-n}|{g,n+1}\right\rangle=-\frac{\Omega_{R}(\Omega+\delta)}{4\Omega^{2}}$
(10)

and at $\omega_{d}+\Omega$ as

$\left\langle{g,n}|{-(n-1)}\right\rangle\left\langle{-(n-1)}\right|\hat{\sigma}_{-}\left|{+n}\right\rangle\cdot\\\
\cdot\left\langle{+n}|{g,n+1}\right\rangle=\frac{\Omega_{R}(\Omega-\delta)}{4\Omega^{2}}.$
(11)

The first-order correlation function $G^{(1)}\left(t_{1},t_{2}\right)$ for a
finite-length pulse was measured using equation (4) and methods from [45, 15].
Fig. 3(c) shows the real part of function (5) for the resonant case, while
Fig. 3(d) displays 2D correlation function slices, indicating the imaginary
part is near zero. The central square in Fig. 3(c) highlights correlations
during the pulse drive, with primarily real positive values in both resonant
and non-resonant scenarios. Another square along the diagonal indicates post-
pulse correlations, real and positive only in resonance. The rectangles
flanking the diagonal show voltage correlations between pulse and post-pulse
times. The function’s real, positive nature during the pulse underscores the
quantum nature of the effect, since classical coherent oscillating fields
cannot exhibit the same property. Additionally, the averaged field, combined
with the diagonal trace of the correlation function, provides a complete
tomography of the qubit state evolution.

To analyze the measured $G^{(1)}\left(t_{1},t_{2}\right)$, a Fourier transform
is performed as

$IPSD=\left|\int_{0}^{+\infty}G^{(1)}\left(t,t+\tau\right)e^{-i\omega\tau}d\tau\right|,$
(12)

with $\tau=t_{2}-t_{1}>0$ representing the time difference between two points.
The IPSD (instant power spectral density) reveals three distinct peaks (Fig.
3(e)) during the pulse drive, with sidebands at Rabi frequencies. Post-pulse,
only the central peak remains, related to spontaneous emission.

Figure 3: Measurement of the second-order momenta of the field in the resonant
case. (a) Traces of field and qubit population as power. (b) Qubit state
evolution extracted from traces in (a). (c) Real part of the first-order
correlation function. Its diagonal trace is the population trace in (a).
Vertical slices are shown in (d). (e) Fourier spectra taken along
$\tau=t_{2}=t_{1}$ for the first-order correlation function. Vertical slices
from (e) are shown in (f). Figure 4: Properties of qubit radiation during a
long pulse with a large amplitude. (a)-(b) Dynamics of the coherent field and
its spectra as a function of the frequency difference between the qubit
resonance and the carrier frequency. (c)-(d) Non-stationary two-time first-
order correlation function and instant P.S.D. for the pulse on resonance. (e)
Comparison of the squared spectrum of the average field with the power
spectral density in the steady state, which exhibits a Mollow triplet. To
smooth the P.S.D. we averaged it over the stationary period between the two
green lines in the subfigure (d).

A long pulse measurement (Fig. 4) enhances spectral resolution. Stimulated
emission peaks in the field spectra become narrower, constrained by
$(\Gamma_{1}+\Gamma_{2})/2$. At $\omega_{d}\neq\omega_{q}$, a sharp peak
appears above a broader one with $\Gamma_{2}/2$ width. The free decay spectral
line is absent in Fig. 4(b) because we have cut voltage traces at the end of
the pulse. Fig. 4(c) depicts the real part of the first-order correlation
function for the in-resonance case, covering the initial non-stationary period
and the steady period $t>0.5$ $\mu$s, where qubit state oscillations have
faded. In this steady region, the correlation function oscillates with the
time difference $\tau=t_{2}-t_{1}$. Fourier transforming along $\tau$ and
averaging within the steady region (between green lines in Fig. 4(d)) produces
a power spectral density (PSD) resembling a Mollow triplet [46, 5]. Comparing
this with the squared spectrum of the averaged field in resonance shows that
the sidebands align while the central peak is absent due to its incoherent
nature.

As discussed in the theory section, our results (Fig. 3(c-f)) demonstrate that
$G^{(1)}\left(t,t+\tau\right)$ approaches zero for any $\tau$ at times $t$
between two adjacent Rabi periods, while it is non-zero at other time points.
For instance, all peaks in IPSD (Fig. 3(e) and 4(d)) exhibit oscillations with
the Rabi frequency along the $t_{1}$ axis. In Fig. 4(c, d), it is also
observed that decoherence gradually diminishes the initially coherent Rabi
oscillations for a very long pulse, as time progresses beyond $t>1/\Gamma_{1}$
where $G^{(1)}\left(t,t+\tau\right)$ no longer turns to zero.

Conclusion. — In our experiments, we examined a coherent rectangular pulse
evolution as it traversed a 1D waveguide with a strongly coupled
superconducting artificial atom. We demonstrated that the artificial atom
modulates the pulse field with Rabi oscillations. Our findings align closely
with simulations based on the input-output theory expression (2). The field
spectra were interpreted in terms of transitions between dressed qubit levels.
Measuring the non-stationary first-order correlation function
$G^{(1)}\left(t_{1},t_{2}\right)$ provided insights into emitted incoherent
radiation dynamics and highlighted the $\hat{a}^{\dagger}$-operator’s inherent
property to disrupt a coherent state $\left|{\alpha}\right\rangle$.

Earlier quantum optical experiments mainly observed the exponential tail post-
pulse, which encapsulates the energy of a photon over an extended decay time.
Yet, within the pulse, the qubit typically absorbs and emits a photon. Given
the swift Rabi oscillations, this photon energy is confined to a briefer span,
resulting in power oscillations of greater amplitude than the exponential
tail. Thus, beyond quantum optics, qubit state measurements also can benefit
from measuring modulation of a driving pulse after interaction with the atom.

###### Acknowledgements.

We wish to acknowledge the support of Russian Science Foundation (grant N
21-42-00025). This work was performed using equipment of MIPT Shared
Facilities Center.

## References

  * Hoi  _et al._ [2013] I.-C. Hoi, C. M. Wilson, G. Johansson, J. Lindkvist, B. Peropadre, T. Palomaki, and P. Delsing, New Journal of Physics 15, 025011 (2013).
  * Gu  _et al._ [2017] X. Gu, A. F. Kockum, A. Miranowicz, Y. xi Liu, and F. Nori, Physics Reports 718-719, 1 (2017).
  * Roy  _et al._ [2017] D. Roy, C. Wilson, and O. Firstenberg, Reviews of Modern Physics 89, 021001 (2017).
  * Kockum [2020] A. F. Kockum, in  _International Symposium on Mathematics, Quantum Theory, and Cryptography_ (Springer Singapore, 2020) pp. 125–146.
  * Astafiev  _et al._ [2010a] O. Astafiev, A. M. Zagoskin, A. A. Abdumalikov, Y. A. Pashkin, T. Yamamoto, K. Inomata, Y. Nakamura, and J. S. Tsai, Science 327, 840 (2010a).
  * Hoi  _et al._ [2011] I.-C. Hoi, C. M. Wilson, G. Johansson, T. A. Palomaki, B. Peropadre, and P. Delsing, Physical review letters 107 7, 073601 (2011).
  * Astafiev  _et al._ [2010b] O. V. Astafiev, A. A. Abdumalikov, A. M. Zagoskin, Y. A. Pashkin, Y. Nakamura, and J. S. Tsai, Physical Review Letters 104, 183603 (2010b).
  * Abdumalikov  _et al._ [2010] A. A. Abdumalikov, O. Astafiev, A. M. Zagoskin, Y. A. Pashkin, Y. Nakamura, and J. S. Tsai, Physical Review Letters 104, 193601 (2010).
  * Hönigl-Decrinis  _et al._ [2018] T. Hönigl-Decrinis, I. V. Antonov, R. Shaikhaidarov, V. N. Antonov, A. Y. Dmitriev, and O. V. Astafiev, Physical Review A 98, 041801 (2018).
  * van Loo  _et al._ [2013] A. F. van Loo, A. Fedorov, K. Lalumière, B. C. Sanders, A. Blais, and A. Wallraff, Science 342, 1494 (2013).
  * Kannan  _et al._ [2020a] B. Kannan, M. J. Ruckriegel, D. L. Campbell, A. F. Kockum, J. Braumüller, D. K. Kim, M. Kjaergaard, P. Krantz, A. Melville, B. M. Niedzielski, A. Vepsäläinen, R. Winik, J. L. Yoder, F. Nori, T. P. Orlando, S. Gustavsson, and W. D. Oliver, Nature 583, 775 (2020a).
  * Kannan  _et al._ [2020b] B. Kannan, D. L. Campbell, F. Vasconcelos, R. Winik, D. K. Kim, M. Kjaergaard, P. Krantz, A. Melville, B. M. Niedzielski, J. L. Yoder, T. P. Orlando, S. Gustavsson, and W. D. Oliver, Science Advances 6, 10.1126/sciadv.abb8780 (2020b).
  * Abdumalikov  _et al._ [2011] A. A. Abdumalikov, O. V. Astafiev, Y. A. Pashkin, Y. Nakamura, and J. S. Tsai, Physical Review Letters 107, 043604 (2011).
  * Peng  _et al._ [2016] Z. H. Peng, S. E. de Graaf, J. S. Tsai, and O. V. Astafiev, Nature Communications 7, 10.1038/ncomms12588 (2016).
  * Zhou  _et al._ [2020] Y. Zhou, Z. Peng, Y. Horiuchi, O. Astafiev, and J. Tsai, Physical Review Applied 13, 034007 (2020).
  * Lu  _et al._ [2021a] Y. Lu, A. Bengtsson, J. J. Burnett, B. Suri, S. R. Sathyamoorthy, H. R. Nilsson, M. Scigliuzzo, J. Bylander, G. Johansson, and P. Delsing, npj Quantum Information 7, 140 (2021a).
  * Storz  _et al._ [2023] S. Storz, J. Schär, A. Kulikov, P. Magnard, P. Kurpiers, J. Lütolf, T. Walter, A. Copetudo, K. Reuer, A. Akin, J.-C. Besse, M. Gabureac, G. J. Norris, A. Rosario, F. Martin, J. Martinez, W. Amaya, M. W. Mitchell, C. Abellan, J.-D. Bancal, N. Sangouard, B. Royer, A. Blais, and A. Wallraff, Nature 617, 265 (2023).
  * Inomata  _et al._ [2016] K. Inomata, Z. Lin, K. Koshino, W. D. Oliver, J.-S. Tsai, T. Yamamoto, and Y. Nakamura, Nature Communications 7, 12303 (2016).
  * Kono  _et al._ [2018] S. Kono, K. Koshino, Y. Tabuchi, A. Noguchi, and Y. Nakamura, Nature Physics 14, 546 (2018).
  * Besse  _et al._ [2018] J.-C. Besse, S. Gasparinetti, M. C. Collodo, T. Walter, P. Kurpiers, M. Pechal, C. Eichler, and A. Wallraff, Physical Review X 8, 021003 (2018).
  * Mallet  _et al._ [2011] F. Mallet, M. A. Castellanos-Beltran, H. S. Ku, S. Glancy, E. Knill, K. D. Irwin, G. C. Hilton, L. R. Vale, and K. W. Lehnert, Phys. Rev. Lett. 106, 220502 (2011).
  * Eichler  _et al._ [2011] C. Eichler, D. Bozyigit, C. Lang, L. Steffen, J. Fink, and A. Wallraff, Physical Review Letters 106, 220503 (2011).
  * Campagne-Ibarcq  _et al._ [2014] P. Campagne-Ibarcq, L. Bretheau, E. Flurin, A. Auffèves, F. Mallet, and B. Huard, Physical review letters 112, 180402 (2014).
  * Toyli  _et al._ [2016] D. Toyli, A. Eddins, S. Boutin, S. Puri, D. Hover, V. Bolkhovsky, W. Oliver, A. Blais, and I. Siddiqi, Physical Review X 6, 031004 (2016).
  * Kudra  _et al._ [2022] M. Kudra, M. Kervinen, I. Strandberg, S. Ahmed, M. Scigliuzzo, A. Osman, D. P. Lozano, M. O. Tholén, R. Borgani, D. B. Haviland, G. Ferrini, J. Bylander, A. F. Kockum, F. Quijandría, P. Delsing, and S. Gasparinetti, PRX Quantum 3, 030301 (2022).
  * Kurpiers  _et al._ [2019] P. Kurpiers, M. Pechal, B. Royer, P. Magnard, T. Walter, J. Heinsoo, Y. Salathé, A. Akin, S. Storz, J.-C. Besse, _et al._ , Physical Review Applied 12, 044067 (2019).
  * Renaud  _et al._ [1977] B. Renaud, R. Whitley, and C. Stroud Jr, Journal of Physics B: Atomic and Molecular Physics 10, 19 (1977).
  * Eberly  _et al._ [1980] J. H. Eberly, C. V. Kunasz, and K. Wodkiewicz, Journal of Physics B: Atomic and Molecular Physics 13, 217 (1980).
  * Lu  _et al._ [2021b] Y. Lu, A. Bengtsson, J. J. Burnett, E. Wiegand, B. Suri, P. Krantz, A. F. Roudsari, A. F. Kockum, S. Gasparinetti, G. Johansson, _et al._ , npj Quantum Information 7, 35 (2021b).
  * Lu  _et al._ [2021c] Y. Lu, I. Strandberg, F. Quijandría, G. Johansson, S. Gasparinetti, and P. Delsing, Phys. Rev. Lett. 126, 253602 (2021c).
  * Sharafiev  _et al._ [2021] A. Sharafiev, M. L. Juan, O. Gargiulo, M. Zanner, S. Wögerer, J. J. García-Ripoll, and G. Kirchmair, Quantum 5, 474 (2021).
  * Gunin  _et al._ [2023] S. Gunin, A. Vasenin, A. Dmitriev, K. Tikhonov, G. Fedorov, and O. Astafiev, The quantum and classical field scattered on a single two-level system (2023), arXiv:2302.13762 [quant-ph] .
  * Dmitriev  _et al._ [2017] A. Y. Dmitriev, R. Shaikhaidarov, V. N. Antonov, T. Hönigl-Decrinis, and O. V. Astafiev, Nature Communications 8, 10.1038/s41467-017-01471-x (2017).
  * Dmitriev  _et al._ [2019] A. Y. Dmitriev, R. Shaikhaidarov, T. Hönigl-Decrinis, S. E. de Graaf, V. N. Antonov, and O. V. Astafiev, Physical Review A 100, 013808 (2019).
  * Vasenin  _et al._ [2022] A. V. Vasenin, A. Y. Dmitriev, S. V. Kadyrmetov, A. N. Bolgar, and O. V. Astafiev, Physical Review A 106, l041701 (2022).
  * Gardiner and Collett [1985] C. W. Gardiner and M. J. Collett, Physical Review A 31, 3761 (1985).
  * Fan  _et al._ [2010] S. Fan, Ş. E. Kocabaş, and J.-T. Shen, Physical Review A 82, 063821 (2010).
  * Peropadre  _et al._ [2013] B. Peropadre, J. Lindkvist, I.-C. Hoi, C. M. Wilson, J. J. Garcia-Ripoll, P. Delsing, and G. Johansson, New Journal of Physics 15, 035009 (2013).
  * Lalumière  _et al._ [2013] K. Lalumière, B. C. Sanders, A. F. van Loo, A. Fedorov, A. Wallraff, and A. Blais, Physical Review A 88, 043806 (2013).
  * Lindkvist and Johansson [2014] J. Lindkvist and G. Johansson, New Journal of Physics 16, 055018 (2014).
  * [41] See Supplemental Material at [URL will be inserted by publisher].
  * Agarwal and Tara [1991] G. S. Agarwal and K. Tara, Physical Review A 43, 492 (1991).
  * Hönigl-Decrinis  _et al._ [2020] T. Hönigl-Decrinis, R. Shaikhaidarov, S. de Graaf, V. Antonov, and O. Astafiev, Physical Review Applied 13, 024066 (2020).
  * Cohen-Tannoudji  _et al._ [1998] C. Cohen-Tannoudji, J. Dupont-Roc, and G. Grynberg,  _Atom—Photon Interactions_ (Wiley, 1998).
  * da Silva  _et al._ [2010] M. P. da Silva, D. Bozyigit, A. Wallraff, and A. Blais, Physical Review A 82, 043804 (2010).
  * Mollow [1969] B. R. Mollow, Physical Review 188, 1969 (1969).


# Supervision Levels Scale (SLS)

Dima Damen and Michael Wray
University of Bristol, UK
{Dima.Damen<EMAIL_ADDRESS>
###### Abstract

We propose a three-dimensional discrete and incremental scale to encode a
method’s level of supervision — i.e. the data and labels used when training a
model to achieve a given performance. We capture three aspects of supervision,
that are known to give methods an advantage while requiring additional costs:
pre-training, training labels and training data. The proposed three-
dimensional scale can be included in result tables or leaderboards to handily
compare methods not only by their performance, but also by the level of data
supervision utilised by each method. The Supervision Levels Scale (SLS) is
first presented generally for any task/dataset/challenge. It is then applied
to the EPIC-KITCHENS-100 dataset [3], to be used for the various leaderboards
and challenges associated with this dataset.

##  1 Motivation

Performance gains for supervised learning methods have been achieved through a
variety of means. Of these, additional loss/regularisation functions,
architectural changes, and data augmentation techniques have all demonstrated
high performance gains. Acknowledging models are typically data-hungry, a
primary way to improve performance is to increase the amount of labelled
training data. However, this is restricted to problems where additional data
is readily available and labelling costs are not an issue. Alternatively,
using a large-scale dataset for pre-training, before fine-tuning on the target
data, has been shown to provide a significant advantage. Particularly of note,
there has been a recent surge of methods using self-supervised pre-training
which offers a competitive alternative to increasing the amount of labelled
pre-training data. Additionally, during fine-tuning, some methods use
additional training labels, such as segmentation or object graphs, which have
also provided performance boosts.

All of the above makes method comparisons far from trivial. One method can
outperform another purely based on the amount and type of data used for pre-
training (i.e. initialisation) or because additional labels have been utilised
during training. We propose a discrete and incremental scale that can be
incorporated into leaderboards for handily capturing the levels of supervision
between methods, compared on the same test set. Thus allowing direct
comparison between methods that use the same level of supervision.
Additionally, this scale also enables assessing the impact of various
dimensions of supervision on the performance.

Figure 1: The three-dimensional SLS captures incrementally levels of
supervision: pre-training, labels and amount of training data.

It is important to note that the proposed scale does not include any knowledge
of the method’s implementation details (e.g. augmentation, optimisation
choice, loss functions, …). Instead, it focuses on the data-relevant knowledge
available to the researchers before they make any method-relevant choices.
This makes the scale suitable for anonymous submissions to leaderboards, not
disclosing methods’ novelties.

We thus restrict our scale to include three supervision dimensions (Fig 1).
These are:

  1. 1.

Pre-training: The amount of data used in pre-training gives an advantage,
particularly when the type of data used in pre-training is relevant to the
task.

  2. 2.

Training labels: The nature of labels associated with the training data, is
another dimension to distinguish between methods’ supervision. For example, a
method that takes advantage of not only image-level labels but also
localisation or segmentation labels has an advantage for image captioning.

  3. 3.

Training data: The amount of training data offers a clear advantage. When less
labelled data is utilised, while achieving comparable performance, such
methods deserve to be highlighted.

In the next sections, we provide a general framework for this discrete three-
dimensional scale, which we refer to as the Supervision Levels Scale (SLS). We
believe SLS achieves the following objectives:

  * •

SLS can be used to readily compare methods in results tables, so as to
highlight any data-relevant advantages when comparing state-of-the-art
performances.

  * •

SLS can be used in anonymous leaderboards in order to encourage and highlight
methods that utilise less labels to demonstrate their competitive performance.

  * •

SLS enables one leaderboard to compare a variety of self-supervised, weakly-
supervised, few-shot and other distinct levels of supervision, as well as
their combination. This would encourage the community to converge to a single
evaluation leaderboard for the task, while acknowledging these methods differ
in data usage.

  * •

SLS can be used to analyse the impact of data-relevant advantages (pre-
training, training data and training labels) for one task on the same test
set.

We additionally demonstrate how SLS can be used in the leaderboards for the
EPIC-KITCHENS dataset, showing how it can compare the supervision levels of
previous years’ competition winners. This is achieved through self-
declaration, i.e. each submission is requested to declare the supervision
level across each dimension.

##  2 Related Efforts

The proposed Supervision Levels Scale (SLS) can be applied to provide
additional supervision tags as methods are compared on the same test set.
Previous efforts to find discrete scales or categories have focused on
identifying easy or hard examples within one datasets, in order to provide
insights into methods failure cases (e.g. [9, 10, 1]). Up to our knowledge,
there is no previous attempt to provide a discrete scale of the levels of
supervision on the same dataset. We hope that this first attempt can be
further utilised and elaborated for a variety of computer vision tasks.

##  3 Supervision Levels Scale

The proposed SLS has three dimensions: Pre-training (PT), Training labels (TL)
and (amount of) Training data (TD), explained below. These dimensions are
orthogonal and a method is likely to consider a different level in each
dimension. The discrete levels are incremental, with the lowest level implying
the least amount of supervision on the corresponding scale.

For each dimension, we define five supervision levels (1-5), in addition to 0
where no supervision on that dimension is expected. A more fine-grained or
wider range could be adopted by future tasks, but in this proposal we believe
this level of discretisation to be sufficient.

### SLS-PT: Pre-Training

Pre-training is the data utilised to provide model initialisation, and is
independent from data used in training the model itself. When the model is
made of several parts or stages (e.g. feature extraction followed by temporal
modelling), the PT supervision level is calculated for each stage, and the
maximum PT value is considered as the supervision level of the overall method.
For example, if a method uses features (pre-trained and then fine-tuned) while
the classifier is trained from scratch, then the feature’s pre-training is
used to define the level of PT supervision of the full method.

PT |  Description
---|---
0 |  No pre-training was used. Models are randomly initialised, including features — i.e. features are trained from randomly initialised models.
1 |  Standard pre-training on data of limited relevance to the task. For example, in a task on medical data, ImageNet [7] is used for pre-training.
2 |  Relevant pre-training to the task. The data for pre-training is chosen to best fit the problem, and is believed to have a significant impact on learning low-level as well as medium-level features.
3 |  Self-supervision on large-scale unlabelled public data. Importantly, as public data is been used, the pre-training can be replicated.
4 |  Self-supervision using task-specific data. That is, a model has used training and/or test data on which performance will be trained/evaluated, with/without other large-scale public data, thus offering stronger pre-training supervision.
5 |  In addition to any or all of the above, pre-training on private data. This level of pre-training supervision is restricted to approaches that pre-train on data not accessible to other researchers and thus cannot be replicated. Even when these models are made available, other researchers are unable to replicate or improve this pre-training thus should be considered as an advantage.

### SLS-TL: Training Labels

The second dimension of the SLS focuses on the training labels utilised by the
method. This includes labels already available with the dataset, or additional
labels acquired by the authors for the model specifically.

TL |  Description
---|---
0 |  No labelled supervision was used. Training data was used in an unsupervised or self-supervised manner without employing any labels.
1 |  Weak supervision - L1. Weak labels are provided for multiple instances. One-to-one mapping between labels and instances is not available.
2 |  Weak supervision - L2. Weak labels (noisy or incomplete) are associated with each instance.
3 |  Strong supervision - an instance-level label is given.
4 |  Strong supervision - in addition to instance-level labels, additional labels are provided (e.g. segmentation in images or spatio-temporal bounding boxes in video).
5 |  Strong supervision with additional labels, not available with the dataset by default.

### SLS-TD: Training Data

Even with the same pre-training and labels supervision, methods can vary by
the amount of training data used. The last dimension of the SLS captures these
differences, unifying few-shot and many-shot training paradigms, when
evaluated on the same test set. Note that smart approaches to utilise less
data (e.g. avoiding noisy labels) can result in improved performance. However,
in this case the data _has been fully taken advantage of_ and thus these
methods have access to the same amount of training data.

TD |  Description
---|---
0 |  Zero-shot learning - the training set has no label class/category overlap with the test set.
1 |  Few-shot learning - In line with [18], (up to) 5-shot training data is used (per label class/category).
2 |  Efficient learning - Randomly selected fraction (commonly 25%) of the data was used. The remainder of the training data were not used and the choice of sample is not optimised.
3 |  Train set - training set used in full.
4 |  Train+Val sets - after optimising any hyperparameters on the validation set, the combination of training and validation sets are used in training the model. Note that we here assume an official Train/Val split.
5 |  Train+Val++ sets - additional data is used during training (note that this is different from pre-training. This could be used with or without labels, but utilised during training the model itself.

#### Putting it together

The three dimensions are then put together for each method, in the order:

SLS-PT-TL-TD

To give an example, a method trained from scratch (i.e. with no pre-training),
uses instance-level strong supervision as well as being trained with the full
training set would be referred to as: SLS-0-3-3. Conversely, a method that
uses private data for pre-training, weak-supervision from incomplete labels
and few-shot learning would be referred to as: SLS-5-2-1. The two methods
would be evaluated on the same test set, and thus the performance (given
chosen evaluation metrics) are directly comparable, though with significantly
different levels of supervision.

Next, we give an example of SLS on our dataset EPIC-KITCHENS [3] and provide
some comparative analysis from the public leaderboard of the 2019 and 2020
leaderboards of the Action Recognition challenge.

Figure 2: SLS for the EPIC-KITCHENS Action Recognition challenge

|  |  | Submissions | SLS | Top-1 Accuracy | Top-5 Accuracy
---|---|---|---|---|---|---
| Rank | Team | Entries | Date | PT | TL | TD | VERB | NOUN | ACTION $\blacktriangle$ | VERB | NOUN | ACTION
S1 | 1 | UTS-Baidu [21] | 14 | 05/28/20 | 5 | 4 | 4 | $70.41$ | $52.85$ | $42.57$ | $90.78$ | $76.62$ | $63.55$
2 | NUS-CVML [16] | 18 | 05/29/20 | 2 | 4 | 4 | $63.23$ | $46.45$ | $41.59$ | $87.50$ | $70.49$ | $64.11$
| UTS-Baidu [20] | 16 | 05/30/19 | 2 | 4 | 4 | $69.80$ | $52.27$ | $41.37$ | $90.95$ | $76.71$ | $63.59$
3 | SAIC-Cambridge [15] | 34 | 05/27/20 | 1 | 3 | 4 | $69.43$ | $49.71$ | $40.00$ | $91.23$ | $73.18$ | $60.53$
3 | FBK-HuPBA [17] | 50 | 05/29/20 | 5 | 3 | 4 | $68.68$ | $49.35$ | $40.00$ | $90.97$ | $72.45$ | $60.23$
4 | GT-WISC-MPI [12] | 12 | 01/30/20 | 2 | 3 | 4 | $68.51$ | $49.96$ | $38.75$ | $89.33$ | $72.30$ | $58.99$
5 | G-Blend [19] | 14 | 05/28/20 | 5 | 3 | 4 | $66.67$ | $48.48$ | $37.12$ | $88.90$ | $71.36$ | $56.21$
6 | TBN [11] | 2 | 05/30/19 | 2 | 3 | 4 | $66.10$ | $47.89$ | $36.66$ | $91.28$ | $72.80$ | $58.62$
| FAIR [8] | 9 | 10/30/19 | 5 | 3 | 4 | $64.14$ | $47.65$ | $35.75$ | $87.64$ | $70.66$ | $54.65$

Table 1: Top Teams in the 2019 (gray) and 2020 Action Recognition Challenges
in EPIC-KITCHENS-55. Coloured columns include the added SLS demonstrating the
various levels of supervision employed by these methods. Figure 3: Recently
released EPIC-KITCHENS-100 Action Recognition Leaderboard, with self-reporting
submission of SLS on the various baselines. Note 6 only uses weak supervision
hence the SLS-TL of 2.

##  4 SLS for EPIC-KITCHENS

EPIC-KITCHENS is the largest egocentric dataset, captured in a non-scripted
manner, and recorded in participants’ kitchens. First released in 2018 [4],
and later extended [3] to 20M frames, 90K action segments and 100 hours of
densely annotated actions, the dataset offers an ideal testbed for assessing
video understanding benchmarks in untrimmed videos. Several benchmarks are
associated with the dataset - currently 5 leaderboards are available for
comparative evaluation, see challenges at:

https://epic-kitchens.github.io/

We showcase how SLS can apply to the EPIC-KITCHENS Action Recognition (AR)
challenge (Sec 4.1 in [3]), then demonstrate how it can be used to compare the
levels of supervision of methods submitted to this leaderboard.

We specialise the SLS for this task as follows:

  * •

SLS-PT for AR in EPIC-KITCHENS:

    * –

SLS-1-X-X: standard pre-training, in this case would be any image-based pre-
training.

    * –

SLS-2-X-X: Relevant pre-training would be video pretraining (e.g. large scale
video datasets such as Kinetics [2] or HowTo100M [13]).

  * •

SLS-TL for AR in EPIC-KITCHENS:

    * –

SLS-X-1-X: Would be utilising no temporal information about action segments or
instances per video, i.e. video-level labels of actions are only known,
without their rough or exact locations.

    * –

SLS-X-2-X: Would utilise weak temporal labels, known as single timestamps in
EPIC-KITCHENS [14]. These are rough single time points within or close to the
action of interest.

    * –

SLS-X-3-X: Full temporal labels (i.e. start-end times) without any spatial
labels have been used.

    * –

SLS-X-4-X: Methods have taken advantage of spatio-temporal labels, i.e. start-
end times plus bounding boxes, hand detections and/or masks.

  * •

SLS-TD for AR in EPIC-KITCHENS: No adjustments needed to adapt this dimension.
Train and Val splits correspond to the splits in Table 1 from [3].

Fig. 2 visualises SLS for AR in EPIC-KITCHENS, across the three dimensions. We
next demonstrate how this can be applied to the leaderboard, using examples
from the previous years’ challenge leaderboards. Note that these were
evaluated on a previous version (subset) of the dataset known as EPIC-
KITCHENS-55.

In Table 1 we list the winners from the AR challenge for years 2019 and 2020
ranked by their performance. More details of these methods are available in
the technical reports [6, 5]. We denote SLS for each method based on the
technical reports submitted. The table shows that: methods vary in pre-
training, with some utilising the weights from the private dataset IG-Kinetics
[8]. Methods that employed spatio-temporal labels (TL=4) as opposed to
temporal labels solely, consistently outperform others, highlighting the
nature of this dataset: actions take place in the messy environment of
participants’ kitchens and identifying where the action takes place is
consistently helpful. Finally, no work has attempted few-show or efficient
learning on the dataset to-date. SLS-TD in this table is thus consistent for
all methods.

We have also integrated SLS into the new leaderboard for Action Recognition in
EPIC-KITCHENS-100, open for the 2021 round of the challenges. Fig 3 shows the
leaderboard recently opened with the baselines from [3]. Importantly, this
leaderboard shows both full temporal supervised (TL=3) as well as weak
temporal supervision (TL=2) in the same leaderboard. This enables direct
comparison of the drop in performance between methods that utilise single
timestamps [14] and full temporal supervision. In the given baseline, start-
end times improve Top-1 Action Accuracy by 11-16%.

##  5 Further discussion and limitations of SLS

There are a few known limitations of the proposed SLS, due to decisions that
were made with the aim to reduce complexity. First, the discrete nature of SLS
only indicates part of the picture, and details to replicate any method are
only available by reading about the methods themselves. We believe this
compromise to be acceptable. Second, when a method is formed of multiple
stages (e.g. feature extraction then temporal modelling for example), SLS
encodes only one scale per dimension; the maximum level of supervision across
all the stages. Due to some models using a large number of stages and sources
for pre-training, we also consider this simplicity to be acceptable. Third,
SLS relies on self-reporting of each method’s supervision scale, and it is
left to authors to justify their declared supervision levels. Misunderstanding
any dimension can result in incorrect reporting. We hope that this technical
report can assist in alleviating any ambiguity, and will be reviewing its
content as we receive inquiries or concerns.

The remaining (fourth and fifth) limitations require future work to be
addressed. Fourth, SLS assumes all training data uses the same training
labels. Semi-supervised approaches cannot be correctly represented by SLS.
Fifth, multi-modal data is not captured. For example, in Table 1 the method
TBN [11] utilises audio information in addition to video, giving it an
advantage. Similarly, when performing tasks like retrieval or captioning, two
modalities (vision + langauage) will consider different SLS for each modality.
We leave this consideration of multiple modalities and their relationship to
SLS for future work and focus only on the visual modality.

##  6 Conclusion

This report introduces a three-dimensional, incremental and discrete scale to
encode the level of supervision of methods, compared on the same test set. It
aims to unify and directly compare methods that attempt self-supervised pre-
training, few-shot learning and weak supervision, when compared for the same
task and on one test set. We believe SLS can provide new insights into how
methods learn from more/less labelled data. We will be analysing the SLS of
various methods submitted to the five leaderboards of EPIC-KITCHENS-100 in the
next round of challenge submissions.

#### Acknowledgement.

We would like to thank Hazel Doughty and Will Price for early discussions on
this idea.

## References

  * [1] D. Bolya, S. Foley, J. Hays, and J. Hoffman.  Tide: A general toolbox for identifying object detection errors.  In ECCV, 2020.
  * [2] J. Carreira and A. Zisserman.  Quo vadis, action recognition? a new model and the kinetics dataset.  In Proc. CVPR, 2017.
  * [3] D. Damen, H. Doughty, G. M. Farinella, , A. Furnari, J. Ma, E. Kazakos, D. Moltisanti, J. Munro, T. Perrett, W. Price, and M. Wray.  Rescaling egocentric vision.  CoRR, abs/2006.13256, 2020.
  * [4] D. Damen, H. Doughty, G. M. Farinella, S. Fidler, A. Furnari, E. Kazakos, D. Moltisanti, J. Munro, T. Perrett, W. Price, and M. Wray.  Scaling egocentric vision: The epic-kitchens dataset.  In ECCV, 2018.
  * [5] D. Damen, E. Kazakos, W. Price, J. Ma, H. Doughty, A. Furnari, and G. M. Farinella.  Epic-kitchens-55 - 2020 challenges report.  Technical report, 2020.
  * [6] D. Damen, W. Price, E. Kazakos, A. Furnari, and G. M. Farinella.  Epic-kitchens - 2019 challenges report.  Technical report, 2019.
  * [7] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.  ImageNet: A Large-Scale Hierarchical Image Database.  In CVPR, 2009.
  * [8] D. Ghadiyaram, D. Tran, and D. Mahajan.  Large-scale weakly-supervised pre-training for video action recognition.  In CVPR, 2019.
  * [9] D. Hoiem, Y. Chodpathumwan, and Q. Dai.  Diagnosing error in object detectors.  In ECCV, 2012.
  * [10] J. Hosang, R. Benenson, P. Dollár, and B. Schiele.  What makes for effective detection proposals?  IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(4):814–830, 2016.
  * [11] E. Kazakos, A. Nagrani, A. Zisserman, and D. Damen.  Epic-fusion: Audio-visual temporal binding for egocentric action recognition.  In ICCV, 2019.
  * [12] M. Liu, S. Tang, Y. Li, and J. Rehg.  Forecasting human object interaction: Joint prediction of motor attention and actions in first person video.  In ECCV, 2020.
  * [13] A. Miech, D. Zhukov, J.-B. Alayrac, M. Tapaswi, I. Laptev, and J. Sivic.  HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips.  In ICCV, 2019.
  * [14] D. Moltisanti, S. Fidler, and D. Damen.  Action Recognition from Single Timestamp Supervision in Untrimmed Videos.  In CVPR, 2019.
  * [15] J.-M. Perez-Rua, B. Martinez, X. Zhu, A. Toisoul, V. Escorcia, and T. Xiang.  Knowing what, where and when to look: Efficient video action modeling with attention.  Technical report, 2020.
  * [16] F. Sener, D. Singhania, and A. Yao.  Temporal aggregate representations for long-range video understanding.  In ECCV, 2020.
  * [17] S. Sudhakaran, S. Escalera, and O. Lanz.  Fbk-hupba submission to the epic-kitchens action recognition 2020 challenge.  Technical report, 2020.
  * [18] O. Vinyals, C. Blundell, T. Lillicrap, k. kavukcuoglu, and D. Wierstra.  Matching networks for one shot learning.  In Advances in Neural Information Processing Systems, pages 3630–3638, 2016.
  * [19] W. Wang, D. Tran, and M. Feiszli.  What makes training multi-modal classification networks hard?  Technical report, 2020.
  * [20] W. Xiaohan, Y. Wu, L. Zhu, and Y. Yang.  Baidu-uts submission to the epic-kitchens action recognition challenge 2019.  Technical report, 2019.
  * [21] W. Xiaohan, Y. Wu, L. Zhu, and Y. Yang.  Symbiotic attention with privileged information for egocentric action recognition.  In AAAI, 2020.