# Collisionless Pattern Discovery in Robot Swarms Using Deep Reinforcement
Learning

Nelson Sharma1, Aswini Ghosh1, Rajiv Misra1, Supratik Mukhopadhyay2, and
Gokarna Sharma3
3Department of Computer Science, Kent State University, Kent, Ohio, USA
E-mail: {nelson_2121cs07, aswini_2121cs02<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>2Department of Environmental Sciences,
Louisiana State University, Baton Rouge, Louisiana, USA  1Department of
Computer Science and Engineering, Indian Institute of Technology, Patna, India

###### Abstract

We present a deep reinforcement learning-based framework for automatically
discovering patterns available in any given initial configuration of fat robot
swarms. In particular, we model the problem of collision-less gathering and
mutual visibility in fat robot swarms and discover patterns for solving them
using our framework. We show that by shaping reward signals based on certain
constraints like mutual visibility and safe proximity, the robots can discover
collision-less trajectories leading to “well-formed” gathering and visibility
patterns.

###### Index Terms:

Swarm robots, Pattern formation, Gathering, Collision avoidance, Deep
reinforcement learning

##  I Introduction

In recent years, swarm robotics [1] has gained a lot of attention from the
research community. In a swarm, a number of robots work together collectively
for performing a certain task, such as disaster recovery, foraging, mapping,
etc [2]. In many cases, it is assumed that there is no explicit means of
communication between the robots (e.g., in a bandwidth-limited disaster zone);
a robot can only see others through a camera that provides a $360^{\circ}$
view. The robots are anonymous (no unique identifiers), disoriented (no
agreement on coordinate systems or units of distance), autonomous (no external
control), and indistinguishable (no external identifiers). In the classical
oblivious swarm model [4, 52], the position of every robot is considered as a
point on 2D plane and has a local co-ordinate system. Each robot senses other
robots using sensory equipment installed on it, such as a camera. However, the
robots are considered silent since it is assumed that there is no direct
communication among them [4, 52]. Each robot executes the same algorithm.

Additionally, each robot executes Look-Compute-Move (LCM) cycle – when a robot
is activated, it takes a snapshot of its surroundings from its local
perspective (Look), then it uses the snapshot and the algorithm it is
executing to determine an action to take (Compute), and finally it performs
the action (Move). A swarm of robots can operate asynchronously, semi-
synchronously or fully synchronously. In the fully synchronous operating
model, every robot in the swarm becomes active in every LCM cycle and cycles
of every robot are synchronized (i.e., there is a global clock). In the semi-
synchronous operating model, not all swarm robots necessarily become active in
every LCM cycle, but LCM cycles are synchronized (i.e., there is still a
global clock). In the asynchronous operating model, not every robot is
necessarily active all the time, the duration of each LCM cycle is arbitrary,
and the LCM cycles of different robots are not synchronized (i.e., no global
clock). In the classical model, robots are oblivious because they do not keep
memory of what they did in any past LCM cycle.

There has been a lot of research recently in forming patterns with maximum
visibility and gathering without collisions for swarm robots in the classical
model [7, 8, 9, 10, 11, 12, 13, 14]. Pattern formation concerns arranging $n$
robots in a given pattern on the plane. A configuration of robots matches the
pattern if rotation, translation, and scaling of the configuration can produce
the pattern explicitly [7, 8, 9, 10, 11, 12, 13, 14]. An algorithm solves a
pattern formation problem, if given any configuration of $n$ robots located
initially at distinct points on a plane, it enables them to reach a
configuration that matches the given pattern and remain stationary thereafter
[39]. Specific patterns like circle formation have been considered in [23, 24,
25, 26, 27]. Convex hull and cubic spline formations as solutions to the
mutual visibility problem – i.e., each robot sees all others in the swarm,
have also been studied in the classical model [17, 41]. Pattern formation has
been studied in [42] for UAVs. Pattern Formation via Deep Reinforcement
Learning (DRL) has been studied in [43]. The objective in the gathering
problem is to arrange the robots as close together as possible around a
gathering point (that may or may not be known a priori) [38].

Existing pattern formation algorithms belong to two broad classes:
combinatorial algorithms and machine learning-based approaches. In the
combinatorial paradigm [3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17,
18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36,
37], it is assumed that the target pattern to be formed is given as input to
the robots and one designs a sophisticated algorithm to arrange robots to be
positioned on the target pattern positions, starting from a given initial
configuration. Designing such algorithms is labor-intensive, error-prone, and
time consuming. Additionally, the designed combinatorial algorithm may not
work for any initial configuration, for example, many combinatorial algorithms
on the literature assume that the initial configurations are asymmetric [3, 4,
5, 6]. The same kind of assumption of asymmetric initial configuration extends
also to the combinatorial algorithms for gathering in the literature. Existing
machine learning based approaches [38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48,
49, 50, 51] attempt to solve pattern formation for a given target pattern,
i.e., similarly as in the combinatorial algorithms, the target pattern needs
to be given to the machine learning algorithm.

It would be ideal, given any initial configuration (symmetric/asymmetric), if
the valid target patterns could be identified and discovered, even when the
target pattern is not given as input a priori. In this paper, we precisely aim
to do that for the first time in the context of swarm robotics. Particularly,
we seek a _single algorithm_ that can identify and _discover_ patterns that
provide solutions to problems such as mutual visibility or gathering, rather
than giving as input the solution pattern beforehand and the goal of the
algorithm is to reach to the given target from the given initial
configuration.

We present a DRL-based framework for automatically discovering patterns
available in any given initial configuration of robot swarms. We depart from
the literature and consider robot swarms which are not points but fat – and
opaque robots with an extent and hence they occupy certain area. In
particular, we model the problem of collision-less gathering and mutual
visibility in fat opaque robot swarms and discover patterns for solving them
using our framework. Since our proposed approach is DRL-based, it is based on
designing good reward functions. We show that designing reward signals based
on constraints like mutual visibility and safe proximity, the robots can
discover collision-less trajectories leading to “well-formed” gathering and
visibility patterns.

In summary, we make the following two major contributions.

  * •

We model the problem of automatic pattern discovery for collision-less
gathering in robot swarms and formulate it in terms of DRL framework.

  * •

We propose DRL based approach for the collision-less gathering and pattern
discovery using multiple policies to discover “well-formed” gathering
patterns.

##  II RELATED WORK

Specific target pattern formation for swarm robots have been extensively
studied in [40, 41, 27, 28, 29, 30, 31]. Optimal convex hull pattern formation
assuming obstructed visibility has been studied in [40]. In [41], the authors
studied proposed self organised cubic spline based pattern formation method.
Several researchers have studied circular pattern formation [27] problem. In
[29], authors have investigated the plane formation problem that requires a
swarm of robots moving in the three-dimensional Euclidean space to land on a
common plane and considered the fully synchronous setting and where robots are
endowed with only visual perception.Most works that studied pattern formation
problem considered the classic oblivious point swarm model [30, 39]. In [39],
the authors presented a combinatorial framework for the pattern formation
problem and used a randomized procedure for leader election in their proposed
algorithm. They measured the time complexity of their algorithms in “epochs”
(i.e., time interval in which every robot performs at least one look-compute-
move step).

For the fundamental problem of gathering, the authors in [16] considered
gathering of $N$ autonomous robots on a plane, which requires all robots to
meet at a single point under limited visibility. Here the point of gathering
is not known beforehand. The authors in [38] significantly improved the time
complexity of gathering under limited visibility model with the assumption
that the point of gathering is known beforehand.

The authors in [37] considered the state machine representation to formulate
the gathering problem and developed a distributed algorithm that solves the
problem of gathering for any number of fat robots. In [43], the authors
proposed a Deep Reinforcement Learning (DRL)-based method that generates
general-purpose pattern formation strategies for any target pattern. A bio-
inspired algorithm was proposed in [47] to control robot swarms for pattern
formation. The authors in [49] have formulated reward functions to ensure
drones arrive at their destinations following predefined routes avoiding
collisions. The authors in [50] presented the idea of guided policy learning
and investigated how to learn to control a group of cooperative agents with
limited sensing capabilities such as robot swarms. The authors in [51]
presented a framework for collision-less flying of drones over a directed
circulant digraph.

However, none of the above discussed approaches have attempted discovering
collision-less patterns where mutual visibility is maximized with or without
predefined gathering points. In this paper, we consider a DRL-based framework
for discovering “well-formed” patterns for mutual visibility and gathering
where gathering point may not be known beforehand.

##  III System Model and Preliminaries

Robot Swarm. A robot swarm is a collection on $n$ robots moving autonomously
inside a rectangular region of size $(2X_{w},2Y_{w})$ on a 2D Cartesian plane
with origin being at the center of the rectangle. The robots are _fat_ and
_opaque_ , meaning that a robot $(n\in N)$ is represented as a circular opaque
disk of radius $R_{bot}$. As shown in Fig. 1 (left), a robot has a center and
a circular body of fixed size. A robot’s position vector $(P_{n})$ and
velocity vector $(V_{n})$ are assumed to be known at all times, recorded using
sensors internal to robots. It is assumed that all robots are fitted with
sensors that can detect the positions of neighboring robots within a fixed
distance from their centers, called the _scan-radius_ ($R_{scan}$). The
sensors’ readings for a robot provide: (i) the number of neighbors
($G^{all}_{n}$), (ii) the number of fully visible neighbors ($G^{vis}_{n}$),
and (iii) the number of occluded neighbors ($G^{occ}_{n}$). Additionally, a
_safe distance_ ($\delta_{s}$) is defined, which can be used to calculate a
measure of how unsafe is the current positioning of the robots.

Figure 1: (left) Fat robot model (right) Visibility model

Since robots are fat and opaque, a robot $B_{i}$ is said to be occluded by
robot $B_{j}$ from the view of robot $B_{k}$ if any part of $B_{i}$ lies in
the shadow formed by $B_{j}$ assuming that a point source of light placed at
the center of $B_{k}$, as shown in Fig. 1 (right). Two robots are _mutually
visible_ to each other if and only if none of them are occluded by any robot
from the other’s view. The robots are allowed to move in any direction by
changing their velocity vector $(V_{n}=[V_{n}^{x},V_{n}^{y}])$ within a fixed
range ($V^{min}\leq V_{n}^{x},V_{n}^{y}\leq V^{max}$ ) at each time cycle.

Problem Formulation. The goal of robot swarm is to discover appropriate
patterns for gathering such that the robots do not collide with each other and
maintain maximum visibility among one another during pattern formation. We
consider two gathering problem variations as follows:

Gathering at a Predefined Point. In this case, the robots are initially
scattered far apart, i.e., mostly outside the scan-radius of each other. It is
assumed that the gathering point is at the origin of the environment.

Gathering without a Predefined Point. We assume a very large scan-radius and
all robots are within the scan-radius of each other. Gathering point is not
known a priori and the robot swarm has to decide the appropriate gathering
place.

Markov Decision Process. We formulate the above variations of the gathering
problem as an appropriate _Markov Decision Process (MDP)_ and solve it using a
DRL-based approach that tries to maximize its underlying reward function. The
MDP is formulated as follows:

  * •

State Space ($\mathcal{S}$): The State Space contains the position coordinates
of all the robots.

$\mathcal{S}=\\{P_{n}\\}:\forall n\in\\{1...N\\}$ (1)

  * •

Action Space ($\mathcal{A}$): The Action Space contains the velocity
components that can be set at a given time-step, for all the robots.

$\mathcal{A}=\\{[V_{n}^{x},V_{n}^{y}]\\}:\forall n\in\\{1...N\\}$ (2)

  * •

Reward Function ($\mathcal{R}$): The reward for each time-step is based on the
_Reward Signal_ ($\mathcal{C}$) from the current state ($S_{t}$) and the next
state ($S_{t+1}$).

$\mathcal{R}(S_{t+1}|S_{t},A_{t})=\mathcal{C}(S_{t+1})-\mathcal{C}(S_{t})$ (3)

where, the Reward Signal for a given state is a weighted sum of multiple
signals. We define two reward signals, $\mathcal{C}_{1}$ and $\mathcal{C}_{2}$
as follows:

$\mathcal{C}_{1}(S_{t})=\sum_{C\in C_{1}}C(S_{t})$ (4)
$\mathcal{C}_{2}(S_{t})=\sum_{C\in C_{2}}C(S_{t})$ (5)

where, $C_{1}$ and $C_{2}$ are sets of signals defined as:

$C_{1}=\\{C_{close},C_{safety},C_{neighbors},C_{visible}\\}$ (6)
$C_{2}=\\{C_{nclose},C_{safety},C_{neighbors},C_{visible}\\}$ (7)

where, the members are defined as follows:

    * –

$C_{close}$ : measure of closeness of robots from the origin. This signal
causes robots to move closer to the gathering point, i.e., the origin.

$C_{close}(S_{t})=1-W_{close}\sum_{n\in N}||P_{n}(t)||$ (8)

    * –

$C_{safety}$ : is the sum of safety measure ($F_{safe}$) for all robots, which
helps robots to avoid collisions maintaining appropriate distance between each
other.

$C_{safety}(S_{t})=W_{safety}\sum_{i,j\in N,i\neq j}F_{safe}(i,j,t)$ (9)

where, $F_{safe}$ is defined between robots $i$ and $j$ at time-step $t$,
based on a known constant _safe-distance_ ($\delta_{s}$), as follows:

$F_{d}=||P_{i}(t)-P_{j}(t)||-\delta_{s}$
$F_{safe}(i,j,t)=\begin{cases}\sqrt{\delta_{s}+R_{bot}-F_{d}},&F_{d}\geq 0,\\\
\text{0},&F_{d}<0,\\\ \end{cases}$

    * –

$C_{neighbors}$ : denotes the number of neighbors and encourages the robots to
have more robots in their neighborhood.

$C_{neighbors}(S_{t})=W_{neighbors}\sum_{n\in N}G_{n}^{all}(t)$ (10)

    * –

$C_{visible}$ : denotes total number of neighbors that are currently fully
visible for each robot and encourages maximum visibility within the robot
swarm.

$C_{visible}(S_{t})=W_{visible}\sum_{n\in N}G_{n}^{vis}(t)$ (11)

    * –

$C_{nclose}$ : measures the average distances from all neighbors for each
robot. In the absence of a priori gathering point, this signal encourages
robots to move closer to each other for gathering.

$C_{nclose}(S_{t})=1-W_{nclose}\sum_{i,j\in N,i\neq j}||P_{i}(t)-P_{j}(t)||$
(12)

The weights assigned to each signal ($W_{i}$) are chosen so as to normalize
the values to the range $[0,1]$.

  * •

Initial States ($\mathcal{S}_{0}$): We consider three types of initial state
configurations:

    * –

_Scattered_ : robots are uniformly scattered along a roughly circular shape
with radius $\min(X_{w},Y_{w})$.

    * –

_Distributed_ : robots are equally scattered near any two of the corners of
the environment.

    * –

_Packed_ : robots are closely placed in a line (or multiple lines).

An initial state configuration $(\mathcal{S}_{0},\epsilon)$ is chosen at the
start of each episode where $\mathcal{S}_{0}$ is a set of $n$ coordinates, one
for each robot, and $\epsilon$ is the noise radius. A robot $(B_{n})$ is
placed at a location ($P_{n}(0)$) given by:

$P_{n}(0)=P_{n}^{0}+\mathcal{U}(-\epsilon,\epsilon)$ (13)

where $P_{n}^{0}$ is an predefined point generated based on the initial state
$(\mathcal{S}_{0})$. Initial position of a robot is determined by adding
uniformly sampled noise ($\epsilon$) to each dimension of $P_{n}^{0}$.

For gathering at a predefined point, we use the reward signal
$\mathcal{C}_{1}$, which tries to maximize the $C_{close}$ signal causing
robots to move closer to a pre-defined gathering point. In contrast, for
gathering without a predefined point, we shall use the reward signal
$\mathcal{C}_{2}$ in our MDP which tries to maximize the $C_{nclose}$ signal
causing robots to move closer to each other. For both the problems, the common
reward signal is $\mathcal{C}_{safe}$ which encourages collision-less movement
at a safe distance; and $\mathcal{C}_{visible}$ which encourages maximum
mutual visibility.

Proximal Policy Optimization (PPO). PPO [53] is a policy optimization
algorithm where the parameterized policy is updated using one of two objective
functions. The first is _clipped objective_ function ($L_{C}$) defined as
follows:

$L_{C}=\mathbb{E}_{t}\left[\min(r_{t}\hat{A}_{t},clip(r_{t},1-\epsilon_{c},1+\epsilon_{c})\hat{A}_{t})\right]$
(14)

where, $\hat{A}_{t}$ represents the advantage estimate, $\epsilon_{c}$
represents the _clipping hyper-parameter_ and $r_{t}$ is the ratio of
probabilities given by the current policy ($\pi_{i}$) and an older version of
it ($\pi_{i-1}$), defined as:

$r_{t}=\frac{\pi_{i}(A_{t}|S_{t})}{\pi_{i-1}(A_{t}|S_{t})}$ (15)

The clipped objective can be maximized using a mini-batch ($\mathcal{D}$) of
trajectories, in which case the objective becomes:

$L_{C}=\frac{1}{|\mathcal{D}|T}\sum_{\mathcal{D}}\sum_{t=0}^{T}\min(r_{t}\hat{A}_{t},clip(r_{t},1-\epsilon_{c},1+\epsilon_{c})\hat{A}_{t})$
(16)

where, $T$ represents the trajectory length. The advantage is estimated using
a truncated version of Generalized Advantage Estimation (GAE)[54] defined as:

$\delta_{t}=\mathcal{R}_{t}+\gamma V(S_{t+1})-V(S_{t})$ (17)
$\hat{A}_{i}(\gamma,\lambda)=\sum_{l=0}^{T-t-1}(\gamma\lambda)^{l}\delta_{l+t}$
(18)

where, $\lambda$ is a hyper-parameter, $\gamma$ is the discount factor, and
$V$ represents the value function.

Notice that PPO is an on-policy algorithm, i.e., it uses the latest policy to
sample actions during the learning process. The exploration is dependent on
the randomness in the stochastic policy. As the policy is tuned over time, the
randomness in sampled action decreases which often causes the agent to explore
less, leading to the policy settling in a local optima. We shall take
advantage of this fact and use multiple policies to move around local optima
until we find a satisfactory one. This can be achieved, in one way, by
altering the value of discount factor during multiple runs of the learning
algorithm.

##  IV Algorithm

To discover a gathering pattern, we use Algorithm 2 to generate two policies
sequentially using an augmented version of the Clipped Proximal Policy
Optimization (Clipped-PPO) algorithm developed in [53]. The first policy
($\Pi_{0}$) is called the _Base_ policy and the second one ($\Pi_{1}$) is
called the _Auxiliary_ policy. The base policy finds near-optimal patterns in
most cases but may sometimes get struck on sub-optimal patterns (i.e., local
optima). The Auxiliary policy is used to break out of local optima and improve
the reward further. The base policy uses a larger value of the discount factor
($\gamma$) and trains on longer horizons. On the other hand, the auxiliary
policy uses a smaller value of discount factor and trains on short horizons
using only the sub-optimal patterns formed by the base policy as the initial
states. The pattern can be sub-optimal, in the sense that the robots do
satisfy the gathering criteria only to a certain degree. In other words, it
might be possible that the reward can be further maximized to obtain a more
optimal pattern. In such a case, the auxiliary policy is trained starting at
various sub-optimal patterns formed by the base policy and is expected to
improve the pattern further using a few more steps. The algorithm additionally
uses hyper-parameters which are described in Table I.

Table I: Hyper-parameters used in proposed algorithms Hyper-Parameter | Description
---|---
$\gamma$ | Discount Factor
$\lambda$ | GAE Lambda
$\epsilon_{c}$ | Clipping Range for Objective
$\alpha$ | Learning Rate for policy updates
$\beta$ | Learning Rate for value updates
$Z$ | No. of Learning Epochs
Algorithm 1 Gathering using Clipped PPO

0: Set of Initial States $({\sigma})$

0: Hyper-Parameters $(\gamma,\lambda,\epsilon_{c},\alpha,\beta,Z)$

1: Initialize Policy-Network $(\pi_{\theta})$ and Value-Network $(V_{\phi})$
with parameters $\theta_{0}$ and $\phi_{0}$ respectively

2: Initialize Learning Environment with a uniform Initial State Distribution
on set $\sigma$.

3: Initialize an empty set $\sigma^{*}$

4: for $i=1$ to $Z$ do

5: Collect batch of trajectories
$\mathcal{D}_{i}=\\{S_{t},A_{t},R_{t},S_{t+1}\\}$ from the environment using
current policy $\pi_{\theta_{i}}$ and compute _Rewards-to-go_ ($\hat{R}_{t}$)
at each time-step $t$
$\hat{R}_{t}=\sum_{l=t}^{T}\mathcal{R}(S_{l+1}|(S_{l},A_{l}))$

6: Compute Clipped Objective ($L_{C}$) for batch $\mathcal{D}_{i}$ using
equation (16)

7: Update Policy Network using:
$\theta_{i+1}=\theta_{i}+\alpha\cdot\nabla_{\theta}L_{C}$

8: Update Value Network using:
$E(\phi_{i})=\frac{1}{|\mathcal{D}_{i}|T}\sum_{\mathcal{D}_{i}}\sum_{t=0}^{T}(V_{\phi_{i}}(S_{t})-\hat{R}_{t})^{2}$
$\phi_{i+1}=\phi_{i}-\beta\cdot\nabla_{\phi}E(\phi_{i})$

9: end for

10: Evaluate Policy $\pi_{\theta}$ in the Environment using initial states
from $\sigma$. For each $S_{i}\in\sigma$, add the final state $S^{*}_{i}$ to
set $\sigma^{*}$ only if $\mathcal{S}^{*}_{i}$ is not a terminal state.

11: Output policy $\pi_{\theta}$ and set of states $\sigma^{*}$

The clipping parameter ($\epsilon_{c}$) is gradually increased with each epoch
of Algorithm 1. The Discount Factor Schedule ($F_{\gamma}$) is a function
which provides appropriate value of discount factor on each successive run of
Algorithm 2. We train only two successive policies where the value of Discount
Factor is high for the first run and reduced during the second run. The
auxiliary policy is responsible for improving patterns discovered by the base
policy and can be trained on customized reward signals that give rise to
various patterns while satisfying given gathering criteria.

Algorithm 2 Pattern Discovery for robot swarm

0: Initial State Configuration $(\mathcal{S}_{0},\epsilon)$

0: Hyper-Parameters $(\lambda,\epsilon_{c},\alpha,\beta,Z)$

0: Discount Factor Schedule $(F_{\gamma})$

1: Initialize set of initial states
$\sigma\leftarrow(\mathcal{S}_{0},\epsilon)$

2: Initialize $t\leftarrow 0$

3: Initialize empty list of policies ($\Pi$)

4: while $|\sigma|>0$ do

5: Run Algorithm[1] with Set of Initial States ($\sigma$) and hyper-parameters
$(F_{\gamma}(t),\lambda,\epsilon_{c},\alpha,\beta,Z)$ to obtain $\pi^{t}$ and
$\sigma^{t}$

6: Append $\pi^{t}$ to list of policies $\Pi$

7: Update set of initial states $\sigma\leftarrow\sigma^{t}$

8: $t\leftarrow t+1$

9: Evaluate policy list ($\Pi$) with given Initial State Configuration
$(\mathcal{S}_{0},\epsilon)$ to obtain desired trajectory set $\mathcal{T}$

10: If $\mathcal{T}$ forms valid collision-less patterns then Break

11: end while

12: if $|\sigma|=0$ then

13: No valid pattern formation trajectory was discovered

14: else

15: Output Policy List $\Pi$

16: end if

##  V Experiments and Results

To test the pattern discovery algorithm (Algorithm 2), we conducted a total of
six experiments under three different swarm compositions with two different
gathering criteria (reward signals) starting in one of the three initial state
configurations (Scattered, Distributed, Packed). The experimental settings are
described in Table II.

Table II: Experiment Settings Experiment | Swarm Size | Gathering Target | Initial States
---|---|---|---
A | $6$ | Origin | Packed
B | $8$ | Origin | Scattered
C | $10$ | Origin | Distributed
D | 6 | Undefined | Packed
E | 8 | Undefined | Scattered
F | 10 | Undefined | Distributed

The training results for swarm size of 10 and 8 (in experiments C and E) are
shown in Figs. 2 and 3, respectively. Note that, in both cases, the auxiliary
policy improves the patterns formed by the base policy which is evident from
the improvement seen in the reward signal during auxiliary training.

Figure 2: [Exp C] Training result for Base Policy and Auxiliary Policy for
swarm of size 10 with pre-defined gathering point  Figure 3: [Exp E] Training
result for Base Policy and Auxiliary Policy for swarm of size 8 with undefined
gathering point

Fig. 4 shows the reward obtained by robot during a single episode of pattern
formation. When the _step-reward_ received by the agent becomes zero, it
indicates that the base policy cannot improve the pattern any further, and
hence, has converged to a stable pattern. It can be seen that the base policy
leads to a sub-optimal pattern which is then improved by the auxiliary policy.

Figure 4: [Exp C] Cumulative Reward obtained during single episode: (a) Reward
Obtained by Base Policy, (b) Reward Obtained by Auxiliary Policy.

The patterns discovered by the trained policies are shown in Figs. 5–8, which
show the (a) initial states, (b) the final pattern formed by base policy, and
(c) the improved pattern formed by auxiliary policy. It should be noted that
the patterns discovered are quite different in both the cases and the proposed
algorithm is able to gather robots even when gathering without a predefined
point. Similar results can be seen for swarm size of 8 (in experiments B and
E) in Figs. 6 and 8 which also show that the auxiliary policy is able to
improve the base pattern even though the improved pattern is only slightly
different in case of experiment E as seen in Fig. 8.

Figure 5: [Exp C] Pattern formation for swarm of size 10 in distributed initial state with pre-defined gathering point  Figure 6: [Exp F] Pattern formation for swarm of size 10 in distributed initial state with undefined gathering point  Figure 7: [Exp B] Pattern formation for swarm of size 8 in scattered initial state with pre-defined gathering point  Figure 8: [Exp E] Pattern formation for swarm of size 8 in scattered initial state with undefined gathering point  Figure 9: Swarm size vs average pattern formation steps for distributed and scattered initial state configurations  Table III: Average Steps taken for pattern formation Experiment | Swarm Size | Base Steps | Auxiliary Steps | Total Steps
---|---|---|---|---
A | 6 | 113.63 | 4.01 | 117.64
B | 8 | 728.37 | 18.22 | 746.59
C | 10 | 1730.20 | 162.38 | 1892.58
D | 6 | 269.88 | 2.89 | 272.77
E | 8 | 521.71 | 223.56 | 1168.83
F | 10 | 1550.26 | 486.97 | 2037.23

Table III shows the average number of time-steps taken for robots to form a
collision-less gathering pattern under different experimental settings and
standard initial state configurations with uniform noise
$\sim\mathcal{U}(-2R_{bot},2R_{bot})$. A total of $200$ random initial states
were generated to test pattern formation steps under the corresponding initial
state configuration. To study the relationship between swarm size and number
of steps needed for pattern formation, we conducted similar experiments with
robot swarms of sizes ranging from $4$ to $10$ under distributed and scattered
initial state configurations; see Fig. 9. Note that the number of steps varies
almost linearly with the swarm size.

##  VI Conclusions and Future Work

In this paper, we proposed a DRL-based framework for automatic pattern
discovery and gathering for swarms of oblivious fat opaque robots. Using
state-of-the-art DRL techniques for policy optimization and proper reward
system design, the agents trained using the proposed methods were shown to be
able to discover collision-less trajectories for gathering in desirable
patterns. With the use of multiple policies at different stages of gathering,
it was possible for the robots to improve their patterns successively instead
of being struck at local optima.

In the future, we shall work upon improving the robustness and scale of our
models by making them fully distributed and autonomous while also being able
to handle environmental uncertainty like unseen obstacles, actuator failures,
and recovery mechanisms. We look forward to explore trajectory planing and
environmental mapping as well.

## References

  * [1] M. Dorigo, M. Birattari, and M. Brambilla, “Swarm robotics,” _Scholarpedia_ , vol. 9, no. 1, p. 1463, 2014.

  * [2] C. Blum and D. Merkle, _Swarm intelligence: introduction and applications_. Springer Science & Business Media, 2008.
  * [3] R. Vaidyanathan, G. Sharma, J.L. Trahan, On fast pattern formation by autonomous robots, in: SSS, 2018, pp.203–220.
  * [4] P. Flocchini, G. Prencipe, N. Santoro, Distributed Computing by Oblivious Mobile Robots, Synthesis Lectures on Distributed Computing Theory, Morgan and Claypool Publishers, 2012.
  * [5] S. Das, P. Flocchini, G. Prencipe, N. Santoro, M. Yamashita, Autonomous mobile robots with lights, Theor. Comput. Sci. 609(P1) (2016) 171–184.
  * [6] D. Peleg, Distributed coordination algorithms for mobile robot swarms: new directions and challenges, in: IWDC, 2005, pp.1–12.
  * [7] Q. Bramas, S. Tixeuil, Probabilistic asynchronous arbitrary pattern formation (short paper), in: SSS, 2016, pp.88–93.
  * [8] Y. Dieudonné, F. Petit, V. Villain, Leader election problem versus pattern formation problem, in: DISC, 2010, pp.267–281.
  * [9] P. Flocchini, G. Prencipe, N. Santoro, P. Widmayer, Arbitrary pattern formation by asynchronous, anonymous, oblivious robots, Theor. Comput. Sci. 407(1–3) (2008) 412–447.
  * [10] N. Fujinaga, Y. Yamauchi, H. Ono, S. Kijima, M. Yamashita, Pattern formation by oblivious asynchronous mobile robots, SIAM J. Comput. 44(3) (2015) 740–785.
  * [11] I. Suzuki, M. Yamashita, Distributed anonymous mobile robots: formation of geometric patterns, SIAM J. Comput. 28(4) (1999) 1347–1363.
  * [12] M. Yamashita, I. Suzuki, Characterizing geometric patterns formable by oblivious anonymous mobile robots, Theor. Comput. Sci. 411(26–28) (2010) 2433–2453.
  * [13] Y. Yamauchi, M. Yamashita, Pattern formation by mobile robots with limited visibility, in: SIROCCO, 2013, pp.201–212.
  * [14] Y. Yamauchi, M. Yamashita, Randomized pattern formation algorithm for asynchronous oblivious mobile robots, in: DISC, 2014, pp.137–151.
  * [15] R. Vaidyanathan, C. Busch, J.L. Trahan, G. Sharma, S. Rai, Logarithmic-time complete visibility for robots with lights, in: IPDPS, 2015, pp.375–384.
  * [16] P. Poudel, G. Sharma, Universally optimal gathering under limited visibility, in: SSS, 2017, pp.323–340.
  * [17] G. Sharma, C. Busch, S. Mukhopadhyay, Brief announcement: complete visibility for oblivious robots in linear time, in: SPAA, 2017, pp.325–327.
  * [18] G. Sharma, R. Vaidyanathan, J.L. Trahan, Constant-time complete visibility for asynchronous robots with lights, in: SSS, 2017, pp.265–281.
  * [19] G. Sharma, R. Vaidyanathan, J.L. Trahan, C. Busch, S. Rai, Logarithmic-time complete visibility for asynchronous robots with lights, in: IPDPS, 2017, pp.513–522.
  * [20] S. Cicerone, G.D. Stefano, A. Navarra, Asynchronous arbitrary pattern formation: the effects of a rigorous approach, Distrib. Comput. 32(2) (2019) 91–132.
  * [21] S. Cicerone, G.D. Stefano, A. Navarra, Asynchronous embedded pattern formation without orientation, in: DISC, 2016, pp.85–98.
  * [22] S. Das, P. Flocchini, N. Santoro, M. Yamashita, Forming sequences of geometric patterns with oblivious mobile robots, Distrib. Comput. 28(2) (2015) 131–145.
  * [23] X. Défago, A. Konagaya, Circle formation for oblivious anonymous mobile robots with no common sense of orientation, in: FOMC, 2002, pp.97–104.
  * [24] X. Défago, S. Souissi, Non-uniform circle formation algorithm for oblivious mobile robots with convergence toward uniformity, Theor. Comput. Sci. 396(1–3) (2008) 97–112.
  * [25] Y. Dieudonné, O. Labbani-Igbida, F. Petit, Circle formation of weak mobile robots, in: SSS, 2006, pp.262–275.
  * [26] A. Dutta, S.G. Chaudhuri, S. Datta, K. Mukhopadhyaya, Circle formation by asynchronous fat robots with limited visibility, in: ICDCIT, 2012, pp.83–93.
  * [27] B. Katreniak, Biangular circle formation by asynchronous mobile robots, in: SIROCCO, 2005, pp.185–199.
  * [28] P. Flocchini, G. Prencipe, N. Santoro, G. Viglietta, Distributed computing by mobile robots: uniform circle formation, Distrib. Comput. 30(6) (2017) 413–457.
  * [29] Y. Yamauchi, T. Uehara, S. Kijima, M. Yamashita, Plane formation by synchronous mobile robots in the three-dimensional Euclidean space, J. ACM 64(3) (2017) 16.
  * [30] S. Das, P. Flocchini, G. Prencipe, N. Santoro, Synchronized dancing of oblivious chameleons, in: FUN, 2014, pp.113–124.
  * [31] Z. Derakhshandeh, R. Gmyr, A.W. Richa, C. Scheideler, T. Strothmann, Universal shape formation for programmable matter, in: SPAA, 2016, pp.289–299.
  * [32] [30]G.A. Di Luna, P. Flocchini, N. Santoro, G. Viglietta, Y. Yamauchi, Brief announcement: shape formation by programmable particles, in: DISC, 2017, pp.48:1–48:3.
  * [33] A. Cord-Landwehr, B. Degener, M. Fischer, M. Hüllmann, B. Kempkes, A. Klaas, P. Kling, S. Kurras, M. Märtens, F. Meyer auf der Heide, C. Raupach, K. Swierkot, D. Warner, C. Weddemann, D. Wonisch, A new approach for analyzing convergence algorithms for mobile robots, in: ICALP, 2011, pp.650–661.
  * [34] J.D. Lawrence, A Catalog of Special Plane Curves, Dover Books on Mathematics, Dover, New York, NY, 2014.
  * [35] H. Attiya, J. Welch, Distributed Computing: Fundamentals, Simulations and Advanced Topics, John Wiley & Sons, 2004.
  * [36] H.M. El-Boghdadi, R. Vaidyanathan, J.L. Trahan, S. Rai, On the communication capability of the self-reconfigurable gate array architecture, in: IPDPS, 2002, pp.1–8.
  * [37] C. Agathangelou, C. Georgiou, M. Mavronicolas, A distributed algorithm for gathering many fat mobile robots in the plane, in: PODC, 2013, pp.250–259.
  * [38] GOKARNA SHARMA,COSTAS BUSCH, SUPRATIK MUKHOPADHYAY, and CHARLES MALVEAUX, Tight Analysis of a Collisionless Robot Gathering Algorithm ACM Transactions on Autonomous and Adaptive SystemsVolume 12 Issue 1 March 2017 Article No.: 3pp 1–20https://doi.org/10.1145/3056460
  * [39] Ramachandran ,Vaidyanathana,GokarnaSharma ,bJerryTrahana, On fast pattern formation by autonomous robots,https://doi.org/10.1016/j.ic.2021.104699
  * [40] Rory Hector; Ramachandran Vaidyanathan; Gokarna Sharma; Jerry L. Trahan ,Optimal Convex Hull Formation on a Grid by Asynchronous Robots With Lights,IEEE Transactions on Parallel and Distributed Systems ( Volume: 33, Issue: 12, 01 December 2022)
  * [41] Belkacem Khaldi , Fouzi Harrou , Foudil Cherif , and Ying Sun,Toward Emerging Cubic-Spline Patterns With a Mobile Robotics Swarm System, IEEE TRANSACTIONS ON COGNITIVE AND DEVELOPMENTAL SYSTEMS, VOL. 14, NO. 2, JUNE 2022
  * [42] Gunasekaran Raja, Saran V S, Sudha Anbalagan, Ali Kashif Bashir,Collisionless Fast Pattern Formation Mechanism for Dynamic Number of UAVs ,GLOBECOM 2020 - 2020 IEEE Global Communications Conference.
  * [43] Jia Wang; Jiannong Cao; Milos Stojmenovic; Miao Zhao; Jinlin Chen; Shan Jiang,Pattern-RL: Multi-robot Cooperative Pattern Formation via Deep Reinforcement Learning.
  * [44] Abhik Singla, Sindhu Padakandla , and Shalabh Bhatnagar, Memory-Based Deep Reinforcement Learning for Obstacle Avoidance in UAV With Limited Environment Knowledge, IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS, VOL. 22, NO. 1, JANUARY 2021.
  * [45] Linbo Luo , Xinyu Wang, Jianfeng Ma , Member, IEEE, and Yew-Soon Ong , GrpAvoid: Multigroup Collision-Avoidance Control and Optimization for UAV Swarm, IEEE TRANSACTIONS ON CYBERNETICS
  * [46] Haitao Zhao ,Hai Liu, Yiu-Wing Leung,and Xiaowen Chu, Self-Adaptive Collective Motion of Swarm Robots
  * [47] Seongin Na , Hanlin Niu , Barry Lennox ,and Farshad Arvin ,Bio-Inspired Collision Avoidance in Swarm Systems via Deep Reinforcement Learning IEEE TRANSACTIONS ON VEHICULAR TECHNOLOGY, VOL. 71, NO. 3, MARCH 2022.
  * [48] E. Sahin; T.H. Labella; V. Trianni; J.-L. Deneubourg; P. Rasse; D. Floreano; L. Gambardella; F. Mondada; S. Nolfi; M. Dorigo SWARM-BOT: pattern formation in a swarm of self-assembling mobile robots ,IEEE International Conference on Systems, Man and Cybernetics (SMC)
  * [49] Sirui Song, Yuanhang Zhang, Xi Qin, Kirk Saunders, Jundong Liu , Vision-guided Collision Avoidance Through Deep Reinforcement Learning, NAECON 2021 - IEEE National Aerospace and Electronics Conference
  * [50] Maximilian Huttenrauch, Adrian sosic,and Gerhard Neumann, Guided Deep Reinforcement Learning for Swarm Systems, https://doi.org/10.48550/arXiv.1709.06011
  * [51] ZBIGNIEW R. BOGDANOWICZ , Flying Swarm of Drones Over Circulant Digraph,Armament Research, Development and Engineering Center, Picatinny Arsenal, NJ USA
  * [52] Paola Flocchini, Giuseppe Prencipe, and Nicola Santoro, Distributed Computing by Mobile Entities, Current Research in Moving and Computing, Lecture Notes in Computer Science 11340, Springer, 2019
  * [53] John Schumann, Fillip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov ,Proximal Policy Optimization Algorithms,arXiv:1707.06347v2 [cs.LG] 28 Aug 2017
  * [54] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, Pieter Abbeel,High-Dimensional Continuous Control Using Generalized Advantage Estimation,arXiv:1506.02438v6 [cs.LG] 20 Oct 2018


# The multiplexed light storage of Orbital Angular Momentum based on atomic
ensembles

Xin Yang  Ministry of Education Key Laboratory for Nonequilibrium Synthesis
and Modulation of Condensed Matter, Shaanxi Province Key Laboratory of Quantum
Information and Quantum Optoelectronic Devices, School of Physics, Xi’an
Jiaotong University, Xi’an, 710049, China  Hong Chang  Key Laboratory of Time
and Frequency Primary Standards, National Time Service Center, Chinese Academy
of Sciences, Xi’an 710600, China  Jinwen Wang  Ministry of Education Key
Laboratory for Nonequilibrium Synthesis and Modulation of Condensed Matter,
Shaanxi Province Key Laboratory of Quantum Information and Quantum
Optoelectronic Devices, School of Physics, Xi’an Jiaotong University, Xi’an,
710049, China  Yan Ma  Key Laboratory of Time and Frequency Primary Standards,
National Time Service Center, Chinese Academy of Sciences, Xi’an 710600, China
Yun Chen  Ministry of Education Key Laboratory for Nonequilibrium Synthesis
and Modulation of Condensed Matter, Shaanxi Province Key Laboratory of Quantum
Information and Quantum Optoelectronic Devices, School of Physics, Xi’an
Jiaotong University, Xi’an, 710049, China  Shuwei Qiu  Ministry of Education
Key Laboratory for Nonequilibrium Synthesis and Modulation of Condensed
Matter, Shaanxi Province Key Laboratory of Quantum Information and Quantum
Optoelectronic Devices, School of Physics, Xi’an Jiaotong University, Xi’an,
710049, China  Zehao Shen  Ministry of Education Key Laboratory for
Nonequilibrium Synthesis and Modulation of Condensed Matter, Shaanxi Province
Key Laboratory of Quantum Information and Quantum Optoelectronic Devices,
School of Physics, Xi’an Jiaotong University, Xi’an, 710049, China  Chengyuan
Wang  Ministry of Education Key Laboratory for Nonequilibrium Synthesis and
Modulation of Condensed Matter, Shaanxi Province Key Laboratory of Quantum
Information and Quantum Optoelectronic Devices, School of Physics, Xi’an
Jiaotong University, Xi’an, 710049, China  Quan Quan  Ministry of Education
Key Laboratory for Nonequilibrium Synthesis and Modulation of Condensed
Matter, Shaanxi Province Key Laboratory of Quantum Information and Quantum
Optoelectronic Devices, School of Physics, Xi’an Jiaotong University, Xi’an,
710049, China  Dong Wei  Ministry of Education Key Laboratory for
Nonequilibrium Synthesis and Modulation of Condensed Matter, Shaanxi Province
Key Laboratory of Quantum Information and Quantum Optoelectronic Devices,
School of Physics, Xi’an Jiaotong University, Xi’an, 710049, China  Haixia
Chen  Ministry of Education Key Laboratory for Nonequilibrium Synthesis and
Modulation of Condensed Matter, Shaanxi Province Key Laboratory of Quantum
Information and Quantum Optoelectronic Devices, School of Physics, Xi’an
Jiaotong University, Xi’an, 710049, China  Mingtao Cao  Key Laboratory of Time
and Frequency Primary Standards, National Time Service Center, Chinese Academy
of Sciences, Xi’an 710600, China  Hong Gao  Ministry of Education Key
Laboratory for Nonequilibrium Synthesis and Modulation of Condensed Matter,
Shaanxi Province Key Laboratory of Quantum Information and Quantum
Optoelectronic Devices, School of Physics, Xi’an Jiaotong University, Xi’an,
710049, China  Corresponding<EMAIL_ADDRESS>Fuli Li  Ministry of
Education Key Laboratory for Nonequilibrium Synthesis and Modulation of
Condensed Matter, Shaanxi Province Key Laboratory of Quantum Information and
Quantum Optoelectronic Devices, School of Physics, Xi’an Jiaotong University,
Xi’an, 710049, China

###### Abstract

The improvement of the multi-mode capability of quantum memory can further
improve the utilization efficiency of the quantum memory and reduce the
requirement of quantum communication for storage units. In this letter, we
experimentally investigate the multi-mode light multiplexing storage of
orbital angular momentum (OAM) mode based on rubidium vapor, and
demultiplexing by a photonic OAM mode splitter which combines a Sagnac loop
with two dove prisms. Our results show a mode extinction ratio higher than
80$\%$ at 1 $\mu$s of storage time. Meanwhile, two OAM modes have been
multiplexing stored and demultiplexed in our experimental configuration. We
believe the experimental scheme may provide a possibility for high channel
capacity and multi-mode quantum multiplexed quantum storage based on atomic
ensembles.

††journal: journal

Qauntum memories, enabling the storage of an input photonic qubit and
retrieval on controllable time, which can beat any classical device and
constitute essential components in quantum repeaters and optical quantum
information processing[1, 2]. Over the past years, various memory protocols
have been proposed, such as electromagnetically induced transparency (EIT)[3],
gradient echo storage[4], optical frequency comb [5], Raman scheme [6] and the
Duan–Lukin–Cirac–Zoller protocol [7]. The EIT storage protocol is widely used
among these storage protocols because of its simple configuration and easy
implementation.

Improving the multi-mode storage capability in the spatial domain can
effectively improve the working efficiency of quantum repeaters and reduce the
requirement for quantum communication on the storage unit. Orbital angular
momentum (OAM) beam is an attractive light field mode, and its topological
charge can be taken as an arbitrary integer[8]. In principle, an infinite-
dimensional Hilbert space can be constructed to realize higher dimensional
spatial mode encoding of photon OAM. Therefore, using the OAM mode for storage
is an effective way to realize efficient quantum repeaters. In recent years,
research on the storage of photon OAM light modes has been carried out [9, 10,
11]. These studies mainly focus on storing low-order vector beams and finite-
dimensional photon OAM quantum states [12, 13, 11], but the high-dimensional
properties of OAM have not been fully utilized in any quantum memory
experiment and multi-OAM mode storage simultaneously based on atomic ensembles
has not been studied.

It is well known that effectively identifying the OAM mode is significant in
multi-mode storage. To date, there are many traditional measurement methods
for the photon OAM quantum state, including the round-hole diffraction
measurement [14], optical transformation method [15] and interference method
[16, 17]. However, the schemes of state identification of OAM quantum states
after storage are all based on the projection measurement scheme[18]; The
projection measurement scheme can only be used for the identification of the
OAM quantum states but cannot achieve the separation of the OAM quantum
states.An applicable quantum memory with multimode capacity should be achieved
multiplexed storage and demultiplexed retrieval simultaneously. Therefore, if
photons encoded with high-dimensional OAM in a quantum memory, not only the
multimode storage shall be realized but also the stored OAM quantum states
need to be efficiently separated after storage.

Figure 1: (a)The experimental setup of Multiplexing quantum memory. PBS,
polarization beam splitter; BS, beam splitter; HWP, half-wave plate; QWP,
quarter-wave plate; SLM, spatial light modulator. (b) The setup of
demultiplexing; DP, Dove prism; ICCD, intensified charge coupled device (c)
Schematic energy diagram of 87Rb and light coupling.

In this letter, we experimentally investigate multi-mode quantum multiplexing
storage and demultiplexing based on rubidium vapor. Specifically, OAM beams
with different topological charges have been stored in atomic ensembles
through the use of electromagnetic induced transparency storage protocol and
demultiplexed into different output ports by using the photon OAM mode
splitter which combines a Sagnac loop with two dove prisms. Our results show a
mode extinction ratio higher than 80 $\%$ at 1 $\mu$s of storage time.
Meanwhile, two OAM modes have been multiplexed storage and demultiplexed
output in different ports by using an OAM mode splitter. The results show that
our experimental system can absolutely separate these two OAM modes after
storage. We believe that the experimental scheme has applications in high
channel capacity multi-mode quantum multiplexing storage and demultiplexing in
atomic ensembles.

The experimental setup is shown in Fig.1. The output of a 795 nm external
cavity diode laser is divided into two parts. One part is applied for
frequency locking, and the other one is sent through a single-mode fiber (SMF)
to improve the spatial mode. After the fiber, the beam passes through a half-
wave plate and polarization beam splitter to control the intensity of the
control and probe beams. The transmitted part is chosen as the probe beam with
the waist diameter of 2 mm (shown in Fig.1(a)), while the reflected part is
selected as the control beam, whose size is expanded three times larger than
the probe beam by a telescope. The acousto-optic modulations(AOM) are inserted
into these two parts for pulse modulation. The spatial light
modulator(HAMAMATSU-X10468-07) is used for loading grating to generate the OAM
light in the probe beam part. For effective imaging, the probe beam with OAM
passed through the 4f imaging system (not shown in the Fig.1) and propagates
collinearly with the control beam. Finally, Both beams incident into the
Rubidium cell and with the corresponding Zeeman sublevels of 87Rb. The energy-
level configuration is shown in Fig.1(c). In our experiment, the laser
frequency is locked to the $5S_{1/2},F=2\to 5P_{1/2},F^{\prime}=1$ transition
of the 87Rb D1 line. The Rb cell has a length of 50 mm. A three-layer $\mu$
-metal magnetic shield isolates the cell from ambient magnetic fields. In our
experiment, the power of the control and probe beam are The temperature of the
cell is set to 65°C with a controller.

The control and the probe beams are separated into the reflected and
transmitted parts using a QWP and a PBS after the Rb cell, and the reflected
part is the probe beam carrying OAM. Then a Sagnac loop with two Dove prisms
is applied to demultiplex OAM beams after storage. As shown in Fig.1(b), the
OAM probe beam is injected into the Sagnac loop and divided into two parts.
These two parts go through the same path (clockwise and counterclockwise
paths) and inject into the Dove prism with different rotation angles. The Dove
prisms in the two paths produce different phases, and finally, these two parts
interfere with each other in the beam splitter passes through the Sagnac loop.
The spatial intensity distribution of the retrieval signal is recorded by an
intensified charge-coupled device camera with a shutter time of 2 nanoseconds
(ICCD,ANDOR TECHNOLOGY), as shown in Fig.1(b).

Now let us focus on the experimental procedure shown in Fig.1(d). First, the
probe beam pulse with 5$\mu$s is input to the spatial light modulator(SLM)
before the rubidium cell, and an OAM light pulse is generated to load a fork-
shaped grating. In one-third of the back edge of the probe light pulse, the
control light is turned off and then switched on again by using an AOM after a
controllable time. The retrieved probe beam can be demultiplexed after the
sagnac loop and detected by ICCD1 or ICCD2, depending on the topological
charge of the probe light pulse. Secondly, our system realizes the
multiplexing memory and demultiplexing of OAM states using the OAM mode
splitter. To clearly understand the principle of demultiplexing based on the
Sagnac loop, we assume that the quantum state of the probe beam is
$\left|l\right\rangle$($l$ can be any integer). The transmitted and reflected
states become $\frac{1}{2}\left|l\right\rangle$ and
$\frac{i}{2}\left|l\right\rangle$. The outgoing light at the ICCD1 and the
ICCD2 ports are expressed as

$\displaystyle
T_{ICCD1}=\frac{i}{2}\exp(i(2l\alpha+\pi))\left|l\right\rangle+\frac{i}{2}\exp(i\pi)\left|l\right\rangle$
(1a) $\displaystyle
T_{ICCD2}=-\frac{1}{2}\exp(i(2l\alpha+\pi))\left|l\right\rangle+\frac{1}{2}\exp(i\pi)\left|l\right\rangle$
(1b)

The first term of Eq.1 is the quantum state that undergoes a counterclockwise
path, and the second term describes the quantum state that undergoes a
clockwise path. $\alpha$ is the rotation angle of dove prism. Actually, we set
the angle of Dove Prism1 (DP1) to 90 degrees experimentally. Thus, the
expression above becomes

$\displaystyle
T_{ICCD1}=-\frac{i}{2}\exp(il\pi)\left|l\right\rangle-\frac{i}{2}\left|l\right\rangle$
(2a) $\displaystyle
T_{ICCD2}=\frac{1}{2}\exp(il\pi))\left|l\right\rangle-\frac{1}{2}\left|l\right\rangle$
(2b)

According to the Eq.2, when the topological charge of the incident probe beam
is an odd number, we can obtain the $T_{ICCD1}=0$ and
$T_{ICCD2}=\left|l\right\rangle$, which means that the probe beam with quantum
state $\left|l\right\rangle$ constructive interference in the ICCD2 port and
destructive interference at the ICCD1 port. The output is reversed when the
topological charge is an even number.

##  1 Results and analysis

Figure 2: (a) The demultiplexing of the probe beam after 0.25 $\mu$s of
memory. (b) The demultiplexing of the probe beam after 1 $\mu$s of memory.

After checking the system reliability again, mainly including light leakage
and ICCD normal trigger. We shown Our experimental results in Fig. 2(a).
Clearly, the retrieval signal when carrying topological charge of 0, 2, 4 is
captured by ICCD1, while, the signal carrying topological charge of 1 and 3
are captured by ICCD2 after 0.25$\mu$s storage time, respectively. The
demultiplexing system still has good characteristics after 1 $\mu$s storage
time as shown in Fig. 2(b). It can be seen that our demultiplexing system can
separate the OAM mode very well and has a good mode contrast. In addition, we
use oblique lens to realize self-interference of retrieval signal in
experiment. The interference pattern is shown in the upper right corner of
Fig. 2. As the results show that the OAM states remain unchanged after
storage, and different OAM states are separated in different output channels.

In order to study the quantum multiplexing memory and demultiplexing
characteristics of OAM probe beams with different topological charges at
different storage times. For effectively compare the retrieval mode quality
under specific storage time, the mode extinction ratio is defined as

$V_{T_{ICCD1}/T_{ICCD2}}=|\frac{T_{ICCD1}-T_{ICCD2}}{T_{ICCD1}+T_{ICCD2}}|$
(3)

We experimentally changed the clock switch of the control beam and the
grating, and all results are shown in Fig. 3.

Figure 3: The separation contrast of different OAM probe beams at different
storage times. The black, red, blue, pink and green lines express the OAM
probe beam with $l=0$, $l=1$, $l=2$, $l=3$ and $l=4$, respectively.

We can obtain several differences by comparing the mode extinction ratio
obtained from our demultiplexing system. It is observed that the Gauss pulse
($l=0$) maintains high mode extinction ratio about 80 $\%$ after 2.5 $\mu$s of
quantum memory, as the black line shows in Fig. 3. The OAM pulse with $l=1$
remains an extinction ratio higher than 80 $\%$ after 1.5 $\mu$s of storage,
as shown in the red line. However, the optical mode extinction ratio drops
dramatically when the storage time exceeds 1.5 $\mu$s. The reason is due to
the motion of the atomic spin wave that intensifies the diffusion of the
readout optical mode when the optical mode carrying OAM is recorded into the
atomic spin wave, which leads to a decrease in the extinction ratio after
demultiplexing in our system. This influence becomes more pronounced when the
optical mode carries higher-order topological charges, as shown in the blue
line, pink and green lines of Fig. 3.

In order to verify the multi-mode capabilities of multiplexing storage and
demultiplexing in our experimental setup. Two OAM modes expressed as
$\left|1\right\rangle+\left|2\right\rangle$,
$\left|2\right\rangle+\left|3\right\rangle$ and
$\left|3\right\rangle+\left|4\right\rangle$ have been generated for multi-mode
storage, and the intensity distribution as shown in the left column of Fig. 4.
The figure shows that, two OAM modes injected into the memory system interfere
to form a moon-like intensity distribution. After 0.25 $\mu$s stored, this two
OAM modes with different topological charges are separated into different
paths, as shown in the two right columns of Fig. 4. All odd OAM modes have
been recorded by ICCD1, while ICCD2 has recorded the even OAM modes, and the
separated OAM modes have a good intensity distribution. An effective OAM mode
separator can not only effectively separate the OAM modes in different
channels, but also maintain the OAM modes characteristic. Therefore, oblique
lenses are applied again to achieve OAM mode self-interference. The experiment
results are shown in the upper right corner of Fig. 4, the two OAM states
remain unchanged after demultiplexing. At this point, our experimental system
can not only realize the storage of the single OAM state and separated it in
different ports by using an OAM modes splitter but can also store and separate
two OAM modes simultaneously.

Figure 4: Multiplexing memory and demultiplexing output of two OAM modes at
0.25 $\mu$s.

In conclusion, we have demonstrated the experimental realization of multi-mode
multiplexed storage and demultiplexed of OAM beams at room temperature based
on atomic ensembles. With the Sagnac interferometer embedding two dove prisms,
we obtained a retrieval signal of the OAM mode with an odder or even
topological charge in different interfering ports and a mode extinction ratio
higher than 80 $\%$ at the specific storage time. Meanwhile, two OAM
superimposed modes have been stored and demultiplexed successfully, and the
results show that our experimental scheme can effectively separate the two OAM
modes after quantum memory. The OAM demultiplexing devices based on the Sagnac
loop structure not only have higher separation contrast but can also be
extended to a higher dimensional multimode multiplexing storage and
demultiplexing of OAM modes. It is worth noting that quantum memory with
multimode capacity not only needs to maintain the storage modes, but also
needs to maintain the high fidelity. Our results may have applications in high
channel capacity multi-mode quantum multiplexing storage and demultiplexing
based on atomic ensembles.

Funding National Natural Science Foundation of China (NSFC) (11374238,
11534008, 11574247, 11604257, 11604258, 11774286); National Science Foundation
(NSF) (1602755);

Disclosures

Disclosures The authors declare no conflicts of interest.

Data Availability Statement Data underlying the results presented in this
paper are not publicly available at this time but may be obtained from the
authors upon reasonable request.

## References

  * [1] A. I. Lvovsky, B. C. Sanders, and W. Tittel, Nature photonics 3, 706 (2009).
  * [2] R. Zhao, Y. Dudin, S. Jenkins, C. Campbell, D. Matsukevich, T. Kennedy, and A. Kuzmich, Nature Physics 5, 100 (2009).
  * [3] J. P. Marangos, Journal of modern optics 45, 471 (1998).
  * [4] D. B. Higginbottom, B. M. Sparkes, M. Rancic, O. Pinel, M. Hosseini, P. K. Lam, and B. C. Buchler, Physical Review A 86, 023801 (2012).
  * [5] T. Fortier and E. Baumann, Communications Physics 2, 1 (2019).
  * [6] D. Saunders, J. Munns, T. Champion, C. Qiu, K. Kaczmarek, E. Poem, P. Ledingham, I. Walmsley, and J. Nunn, Physical review letters 116, 090501 (2016).
  * [7] L.-M. Duan, M. D. Lukin, J. I. Cirac, and P. Zoller, Nature 414, 413 (2001).
  * [8] Y. Shen, X. Wang, Z. Xie, C. Min, X. Fu, Q. Liu, M. Gong, and X. Yuan, Light: Science & Applications 8, 1 (2019).
  * [9] A. Nicolas, L. Veissier, L. Giner, E. Giacobino, D. Maxein, and J. Laurat, Nature Photonics 8, 234 (2014).
  * [10] T.-S. Yang, Z.-Q. Zhou, Y.-L. Hua, X. Liu, Z.-F. Li, P.-Y. Li, Y. Ma, C. Liu, P.-J. Liang, X. Li _et al._ , Nature communications 9, 1 (2018).
  * [11] C. Wang, Y. Yu, Y. Chen, M. Cao, J. Wang, X. Yang, S. Qiu, D. Wei, H. Gao, and F. Li, Quantum Science and Technology 6, 045008 (2021).
  * [12] V. Parigi, V. D’Ambrosio, C. Arnold, L. Marrucci, F. Sciarrino, and J. Laurat, Nature communications 6, 1 (2015).
  * [13] Y.-H. Ye, M.-X. Dong, Y.-C. Yu, D.-S. Ding, and B.-S. Shi, Optics letters 44, 1528 (2019).
  * [14] G. C. Berkhout and M. W. Beijersbergen, Physical review letters 101, 100801 (2008).
  * [15] G. C. Berkhout, M. P. Lavery, M. J. Padgett, and M. W. Beijersbergen, Optics letters 36, 1863 (2011).
  * [16] H. Huang, Y. Ren, Y. Yan, N. Ahmed, Y. Yue, A. Bozovich, B. I. Erkmen, K. Birnbaum, S. Dolinar, M. Tur _et al._ , Optics letters 38, 2348 (2013).
  * [17] M. Padgett, J. Arlt, N. Simpson, and L. Allen, American Journal of Physics 64, 77 (1996).
  * [18] A. Mair, A. Vaziri, G. Weihs, and A. Zeilinger, Nature 412, 313 (2001).

sample


# VL-InterpreT: An Interactive Visualization Tool for Interpreting Vision-
Language Transformers

Estelle Aflalo111Equal Contributions
Intel Labs
<EMAIL_ADDRESS>Meng Du111Equal Contributions
Intel Labs, UCLA
<EMAIL_ADDRESS>Shao-Yen Tseng
Intel Labs
<EMAIL_ADDRESS>Yongfei Liu
Microsoft Research
<EMAIL_ADDRESS>Chenfei Wu
Microsoft Research
<EMAIL_ADDRESS>Nan Duan
Microsoft Research
<EMAIL_ADDRESS>Vasudev Lal
Intel Labs
<EMAIL_ADDRESS>
###### Abstract

Breakthroughs in transformer-based models have revolutionized not only the NLP
field, but also vision and multimodal systems. However, although visualization
and interpretability tools have become available for NLP models, internal
mechanisms of vision and multimodal transformers remain largely opaque. With
the success of these transformers, it is increasingly critical to understand
their inner workings, as unraveling these black-boxes will lead to more
capable and trustworthy models. To contribute to this quest, we propose VL-
InterpreT, which provides novel interactive visualizations for interpreting
the attentions and hidden representations in multimodal transformers. VL-
InterpreT is a task agnostic and integrated tool that (1) tracks a variety of
statistics in attention heads throughout all layers for both vision and
language components, (2) visualizes cross-modal and intra-modal attentions
through easily readable heatmaps, and (3) plots the hidden representations of
vision and language tokens as they pass through the transformer layers. In
this paper, we demonstrate the functionalities of VL-InterpreT through the
analysis of KD-VLP, an end-to-end pretraining vision-language multimodal
transformer-based model, in the tasks of Visual Commonsense Reasoning (VCR)
and WebQA, two visual question answering benchmarks. Furthermore, we also
present a few interesting findings about multimodal transformer behaviors that
were learned through our tool.

##  1 Introduction

Since transformers were introduced in Vaswani _et al_. [30], not only have
they seen massive success in NLP applications, their impact on computer vision
and multimodal problems has also become increasingly disruptive. However, the
internal mechanisms of transformers that lead to such successes are not well
understood. Although efforts have been made to interpret the attentions [8]
and hidden states [22] of transformers for NLP, such as BERT [12],
investigations in the mechanisms of vision and multimodal transformers are
relatively scarce, and tools for probing such transformers are also limited.
Given the fast-growing number of successful vision and multimodal transformers
(_e.g_., ViT [13] and CLIP [25]), enhanced interpretability of these models is
needed to guide better designs in the future.

Past research has shown the importance of interpreting the inner mechanisms of
transformers. For example, Clark _et al_. [8] found certain BERT attention
heads specialized in handling certain syntactic relations, as well as
interesting ways in which BERT attention utilizes special tokens and
punctuation marks. Additionally, Lin _et al_. [21] showed that the linguistic
information encoded in BERT becomes increasingly abstract and hierarchical in
later layers. These studies provide valuable insights into the functions of
various elements in transformer architecture for NLP, and shed light on their
limitations.

This paper presents VL-InterpreT111A screencast of our application is
available at https://www.youtube.com/watch?v=4Rj15Hi_Pdo. Source code and a
link to a live demo: https://github.com/IntelLabs/VL-InterpreT, which is an
interactive visualization tool for interpreting the attentions and hidden
representations of vision-language (VL) transformers. Importantly, it is a
single system that analyzes and visualizes several aspects of multimodal
transformers: first, it tracks behaviors of both vision and language attention
components in attention heads throughout all layers, as well as the
interactions across the two modalities. Second, it also visualizes the hidden
representations of vision and language tokens as they pass through transformer
layers.

The main contributions of our work are:

  * •

Our tool allows interactive visualization for probing hidden representations
of tokens in VL transformers.

  * •

Our tool allows systematic analysis, interpretation, and interactive
visualization of cross- and intra-modal components of attention in VL
transformers.

  * •

As an application of VL-InterpreT, we demonstrate multimodal coreference in
two analyses: 1) how contextualized tokens in different modalities referring
to the same concept are mapped to proximate representations, and 2) how
attention components capture the conceptual alignment within and across
modalities.

##  2 Related Work

As deep learning models flourish, many tools and methods have been proposed to
offer insight into their inner workings. Some methods are general-purpose [24,
32], while others are nuanced for specific models such as CNNs [35, 36] or
RNNs [16, 27, 17]. In transformers, the introduction of attention not only
helped improve performance, but also served as an additional component towards
interpretability.

Interpretability of NLP transformers was initially approached through the
analysis of attention to capture its alignments with syntactic or semantic
relationships [31, 8]. Following this, subsequent works introduced additional
functionalities including visualizations of hidden representations, task
matching of attention heads, aggregate metrics, and interactive datapoint
analysis [28, 15, 18, 20]. While common in allowing a user to understand the
inner workings of transformers, each tool introduces novel applications. For
example, LIT [28] enables probing for bias through examination of coreferences
in counterfactuals. InterpreT [18] allows tracking of token representations
through the layers and offers users the ability to define new metrics to
identify coreference relationships in attention heads. Additionally, T3-Vis
[20] focused on allowing users to improve transformer training by integrating
the training dynamics in their visualization tool.

Interpreting vision transformers, such as those for object detection [4, 13]
or image captioning [10, 19], has also become increasingly popular. Cordonnier
_et al_. [9] showed that the first few layers in transformers can learn to
behave similarly to convolutional layers, and demonstrated the filter patterns
through visualization of image-to-image attention. As illustrated in this
paper, the attention mechanism in transformers is a natural gateway to
understanding vision models, as the heatmaps of attention can be used to
highlight salient image regions. Furthermore, Chefer _et al_. [7] proposed a
method for visualizing self-attention models by calculating a LRP [1]-based
relevancy score for each attention head in each layer, and propagating
relevancies through the network. The end result is a class-specific
visualization of image regions that led to the classification outcome.

Multimodal interpretability has, up until now, mainly entailed using probing
tasks to study the impact of each modality on the responses generated by the
model. These probing tasks aim to quantify the information captured in the
hidden representations by training classifiers, or applying metrics to
embeddings at different points in a model. For instance, Cao _et al_. [3]
proposed various probing tasks to analyze VL transformers, where the authors
observed modality importance during inference, and identified attention heads
tailored for cross-modal interactions as well as alignments between image and
text representations. Additionally, other works have proposed probing tasks to
interpret VL transformers for aspects such as visual-semantics [11], verb
understanding [14], and other concepts such as shape and size [26]. However, a
disadvantage of probing tasks is the amount of work: additional training of
the classifiers is often required, and specific task objectives must be
defined to capture different embedded concepts. Finally, most aforementioned
works require image-caption pairs as input, and are therefore not best suited
for interpreting multimodal transformers in tasks such as visual question
answering.

Recently, a first attempt at explaining predictions by a VL transformer was
proposed in [6]. There the authors constructed a relevancy map using the
model’s attention layers to track the interactions between modalities. The
relevancy map is updated by a set of update rules that back-propagates
relevancies of the prediction result back to the input. This map is very
useful in understanding how model decisions are formed, but a more
comprehensive interpretation for other aspects of VL transformers is still
needed.

Our proposed tool, VL-InterpreT, differs from previous works in that it
interprets various aspects of multimodal transformers in a single interface.
This interactive interface allows users to explore interactions between tokens
in each modality from a bottom-up perspective, without tying to task-specific
inputs and outcomes. To the best of our knowledge, this is the first
interactive tool for interpreting multimodal transformers.

##  3 System Design

###  3.1 Workflow

Figure 1: VL-InterpreT workflow. Data shown in this figure are for
illustration purposes only.

VL-InterpreT is designed as a two-stage workflow: First, the attentions and
hidden states of a given multimodal model are generated and saved for a set of
examples (in this case, 100 examples). Next, the saved data, along with the
metadata of the corresponding examples, are loaded into our tool to enable
visualizations of the inner workings of the model. The workflow of VL-
InterpreT is shown in Figure 1, and the user interface is shown in Figure 3.
Different from interpretability tools for NLP transformers, VL-InterpreT
addresses the analysis and interpretation of the following properties of
multimodal transformers:

  * •

Input: In general, multimodal transformers are able to process inputs
originating from different modalities, _e.g_., video, audio, or language.
Here, we only consider VL transformers where the input is composed of visual
and textual tokens. These tokens are mapped into a shared space, allowing for
concept-level alignment between the two modalities.

  * •

Attention: Because of the bi-modal nature of the input, the resulting
attention can be split into four components: language-to-language, vision-to-
vision, vision-to-language, and language-to-vision. An illustration of these
components is shown in Figure 1.

  * •

Hidden states: In each layer, the transformer produces as many hidden states
as the number of input tokens. Each input token is embedded as a _d_
-dimension vector after processed by each layer.

Figure 2: The VL-InterpreT user interface (rearranged for print).

Figure 3: The Attention Head Summary plot colored by the metrics selected from
the dropdown menu

###  3.2 Visualizations

####  3.2.1 Attention heads components

In this section, we describe each attention component and how VL-InterpreT
visualizes them interactively. The attention matrix in a VL transformer, as
they are loaded in VL-InterpreT, is of size
$(N_{layers},N_{heads},L_{v}+L_{l},L_{v}+L_{l})$, where $N_{layers}$ and
$N_{heads}$ correspond to the number of layers and heads, respectively,
$L_{v}$ to the number of visual tokens, and $L_{l}$ to the number of text
tokens.

  * •

The Language-to-Vision attention component (L2V) of size
$(N_{layers},N_{heads},L_{l},L_{v})$ reflects the text tokens’ dependency on
the visual tokens. This partition of the attention contains the attention
scores calculated from the dot product of the query vector based on the
selected image patch and the key vectors from text tokens. These attention
scores are the weights given to value vectors of text tokens when summing for
the updated representation of a specific image patch in the next layer. A user
can select any attention head and image patch in the interface of our tool,
and the corresponding L2V attention weights are displayed as a heatmap
overlaid on the input text, to visualize how much each text token contributes
to the updated image patch representation (see Figure 7).

  * •

The Vision-to-Language attention component (V2L) of size
$(N_{layers},N_{heads},L_{v},L_{l})$ reflects the visual tokens’ dependency on
the text tokens. This component of attention in a VL transformer arises
through the query-key dot product where the query vectors are computed from
text token embeddings, and the key vectors are computed from the image token
embeddings. A user can select a specific attention head and a text token, and
the corresponding attention scores will be overlaid onto the image as a
heatmap (see Figure 6). This visualization helps users understand the relative
contributions of various parts of the image to the updated representations of
the text tokens. The interactive application also allows users to play an
animation, in which the heatmap over the image is automatically displayed in a
sequence for each word.

  * •

The Language-to-Language attention component (L2L) of size
$(N_{layers},N_{heads},L_{l},L_{l})$ corresponds to the attention mapping in
NLP Transformers. This component visualizes how all text tokens attends to
each individual token of the input sentence (see Figure 6).

  * •

The Vision-to-Vision attention component (V2V) of size
$(N_{layers},N_{heads},L_{v},L_{v})$ is analogous to language-to-language
attention, but in the visual space. It represents the attention between one
visual tokens and all visual tokens, including itself. Similar to the V2L
component, an attention vector (of size $(L_{v},1)$) here in a given head can
also be translated into a heatmap and overlaid onto the image. This
visualization is useful for identifying the contributions of different image
patches to the updated representation of the image patch selected by a user.

####  3.2.2 Attention head summary

This functionality allows users to visualize a head summary plot containing
statistics of the attentions calculated for all heads and layers. For an
attention matrix of size $(N_{layers},N_{heads},L_{v}+L_{l},L_{v}+L_{l})$, the
head summary computes statistical metrics over the last two dimensions,
resulting in a plot of size $(N_{layers},N_{heads})$.

  * •

The mean attention in an attention head is generated by calculating the
average of the corresponding attention matrix, while some tokens can be
excluded from this calculation. A user can select a summary metric from the
list of options. Each metric can be restricted to different components of the
attention. For example, instead of computing the mean of all input tokens
regardless of modality, the calculation can be limited to the vision-to-
language part or the vision-to-vision part of the attention. Additionally,
users may also focus on both cross-modal components (i.e., V2L and L2V
averaged) or both intra-modal ones (i.e., V2V and V2L averaged). See the drop
down menu in Figure 3.

  * •

Custom metrics: Based on users’ interests, custom metrics can be integrated in
this plot to show relevant attention heads. For example, we created a custom
metric to look for the attention heads responsible for aligning the same
person between vision and language modalities, based on the V2L component.
This metric, labeled Spearman correlation b/w person attention and segment
(see Figure 3), is the Spearman correlation between the panoptic segmentation
mask for a person in the image (generated by Maskformer model [2]), and the
attention heatmap to the corresponding person token in the Vision-to-Language
attention component.

This metric allows a user to identify attention heads that perform a function
similar to panoptic segmentation for people (see Figure 4).

Figure 4: Custom metric: Average correlation between the V2L attention to the
[PERSON] tokens (bottom right) and the person’s panoptic segmentation mask
(top right).

Finally, for each metric, a mean is automatically computed for each layer and
shown in the rightmost column of the attention head summary (see Figures 3 and
4). This column visualizes the general behavior in each layer for the selected
metrics, and shows its evolution throughout the layers of the transformer.

####  3.2.3 Hidden state representation

For each input token, a _d-_ dimensional hidden representation is produced by
the transformer after each layer (in our setup, $d=768$). This pool of hidden
representations is then filtered by two criteria: (1) if the related text is a
stop word and (2) if the related image patch comes from a part of the
background (e.g., wall, ground, etc.). In order to visualize the remaining
hidden representations in a readable form, t-distributed Stochastic Neighbor
Embedding (t-SNE) [29] was applied to reduce dimensions and create disjoint
t-SNE spaces for different layers. This way, given a selected example, VL-
InterpreT tracks the hidden representations both before the first layer and
after each subsequent layer, and plots them in two-dimensional spaces.

Figure 8(a) shows the data points representing the visual (in blue) and
textual (in red) tokens from a given example. When hovering on a data point
from language, the corresponding text is displayed. When hovering on a data
point representing visual tokens, the image is shown with a highlighted blue
patch corresponding to the visual token. Further observations on the hidden
states often reveal the concept-level vision-text alignments that are learnt
in this multimodal setup (see Section 4.2.3). To help further understand this
alignment, VL-InterpreT allows a user to select a token (text or image patch)
from a given example, and shows the nearest token in the other modality from
the whole subset of examples, marked with a green star.

##  4 Case Studies

To demonstrate the functionalities of VL-InterpreT, we analyze an end-to-end
VL transformer model, KD-VLP [23], on two benchmarks: Visual Commonsense
Reasoning (VCR) [34] and WebQA [5]. Nonetheless, our tool is generally
applicable to a variety of multimodal transformer configurations and types of
VL datasets.

###  4.1 Model

The KD-VLP model used in our case study is a transformer for end-to-end
vision-language processing. This model utilizes a ResNet backbone for visual
inputs, and is pretrained using text-oriented, image-oriented, and cross-modal
tasks in the form of masked language modeling, object-guided masked vision
modeling, and phrase-region alignment, respectively. Depending on the
application, the KD-VLP model can be fine-tuned for classification or
generation tasks given bi-modal input of image and text.

###  4.2 Analysis on VCR

The VCR benchmark consists of 290K multiple choice QA problems derived from
110K movie scenes. This dataset is uniquely valuable in that it requires
higher-order cognition and commonsense reasoning about the world. Given an
image and a question, the objective is to select an appropriate answer from
four possible choices, and then provide the rationale. To predict the correct
answer and rationale, the KD-VLP model is fed with the image, the question,
and each answer or rationale individually. The predicted answer $a_{p}$ is the
answer/rationale choice that receives the highest probability score, _i.e_.,

$a_{p}=\underset{a_{j}}{\arg\\!\max}~{}f(v,q,a_{j})$ (1)

where $f$ is the KD-VLP model, $v$ is the image, $q$ the question, and
$\\{a_{j}|~{}j\in[1,2,3,4]\\}$ are the possible answer choices.

The functionalities of VL-InterpreT are demonstrated using example val-445
from the VCR validation set, or example ID 89 in the VL-InterpreT live demo.
This data sample, shown in Figure 6, comprises an image showing a little girl
running to a man and a woman in a garden. The question is:

Where is [PERSON_0] running to?

where [PERSON_0] corresponds to the little girl on the right side of the image
(the locations of persons are provided in the VCR dataset and also passed to
the model). We also analyze the answer predicted by the model (which in this
case is correct):

[PERSON_0] is running to help [PERSON_1] and [PERSON_2] with the plants.

where [PERSON_1] and [PERSON_2] refer to the couple.

The following analyses on this example will highlight the visualization
capabilities that our application provides.

####  4.2.1 Attention head summary

This functionality allows a user to identify interesting heads based on
various metrics. By selecting Mean cross-modal attention from the dropdown
menu (see Figure 3), a user can identify attention heads specialized in cross-
modal attention. For instance, in Figure 5 the eighth attention head in layer
11 (denoted as (11, 8)) has, on average, the highest attention across
modalities. Thus, we focus on this specific head and plot its cross-modal (V2L
and L2V) attention. Apart from cross-modal attention, specific heads could
also been identified through Mean image-to-text attention for analyzing V2L
attentions, or Mean text-to-image attention for L2V attentions. Analogous
procedure also applies to L2L and V2V attention components – for instance, the
metric Mean image-to-image attention (without self) shows the heads’
attentions from every image patch to the every other image patches, excluding
each patch itself and its neighbors. As described in section 3.2.2, a custom
metric was used to identify attention heads with V2L components for [PERSON]
tokens highly correlated with their panoptic segmentations. As shown in Figure
4, head (8, 9) scores particularly high in this metric. With head (8, 9)
selected, the V2L attention attention heatmap for the person token (Figure 4,
bottom right) indeed aligns well with the panoptic segmentation of the
corresponding woman (Figure 4, top right). Furthermore, this correlation
metric exhibits a trend where middle and later layers tend to have higher
correlation coefficients on average (above 0.5) than early layers (less than
0.1), showing the evolution of attention patterns as the transformer layers
grow deeper.

Figure 5: Attention Head Summary

####  4.2.2 Attention components

As described in Section 3.2.1, Figure 6 shows the heatmaps over the image and
the text generated by the Language-to-Language and Vision-to-Language
attention components of head (11, 8). This figure shows how attentions differ
for two selected tokens: [PERSON_0] and plants. The V2L components are
represented as heatmaps over the images at the bottom right of Figures 6(a)
and 6(b). It can be observed that the attention is concentrated on regions
corresponding to the text, namely the little girl for the attention to
[PERSON_0] (Figure 6(a)) and the plants for the attention to the plants token
(Figure 6(b)). The L2L components are on the top right of these figures. For
both selected text tokens, related text tokens (including themselves) are
highlighted in the heatmaps. For the example in Figure 6(a), the attention to
[PERSON_0] is mostly from [PERSON_0], running to, and running to help
[PERSON_1]. In the other example in Figure 6(b), the attention to plants is
mostly from the plants token itself.

Figure 7 visualizes the attentions to vision tokens, i.e., the Language-to-
Vision and the Vision-to-Vision attentions. In this example, we select an
image patch that is a part of the plants on the left image for analysis.
Accordingly, the L2V component (top right of Figure 7) shows that the
attention to this image patch is mainly from the [CLS] token as well as the
plants token, which aligns with the concept behind the selected region. As for
the V2V component (bottom right of the figure), it is also interesting that
most of the regions containing plants in the image attend to this specific
patch of plants, which again shows a conceptual alignment. In summary, this
example shows evidence of a unified concept of “plants”, where the attention
has a consistent pattern between intra- and cross-modal components.

(a) Attention to the text token [PERSON_0]

(b) Attention to the text token plants

Figure 6: Two selected text tokens and the corresponding L2L (top right) and
V2L (bottom right) attention to them. Figure 7: Attentions to vision for a
selected image patch in the left image. The heatmaps show the attention from
the text (top right) as well as other patches in the image (bottom right).

####  4.2.3 Hidden states

Figure 8 demonstrates the capabilities of the VL-InterpreT for visualizing
hidden states. When selecting the text token plants (marked in orange) from
the previous example val-445 (ex 89), Figure 8(b) shows that in layer 11, the
closest image patch from the whole pool of examples is the [IMG_52] (marked
with a green star) from val-495 (ex 99). It is interesting that even when it
comes from a very different example, this token also refers to the plants in
the image. Other than layer 11, users can select different layers on the right
to see other closest image tokens to plants throughout layers.

(a) Hovering over a data point representing a visual token displays the
corresponding image patch.

(b) Clicking on the text token plants marks it orange, and displays the
closest image token from the whole dataset, marked as a green star.

Figure 8: t-SNE plot from the hidden representations of the selected example.

(a) Predicted: A cross appears at the top of the dome of Saint Peter’s
Basilica.

(b) Predicted: There are 6 pillars in front of the Façade of the Kurhaus.

(c) Predicted: The center of both the Aster amellus and the Wood Anemone is
yellow.

(d) Predicted: The ring around the eye of Trogon surrucura is red.

(e) Predicted: The Traditional wedding attire of the Yoruba culture in Nigeria
is white.
Correct color: maroon.

(f) Predicted: The olive that goes in Moroccan Tagine is yellow.
Correct color: green.

Figure 9: V2L attention heatmaps for WebQA examples and generated answers.
Figures have been rearranged for print. (a)-(d) are correct predictions and
(e) and (f) are incorrect.

Sections 4.2.2 and 4.2.3 show evidence of alignment between visual and textual
concepts of plants. We see that the concept of ”plants” in both modalities and
across examples is captured by proximate representations. As such, VL-
InterpreT allows studying how such sense of objectness emerges by probing the
attentions and hidden states across all layers of the transformer.

###  4.3 Analysis on WebQA

The WebQA benchmark focuses on multimodal, multihop reasoning for open-domain
question answering. This benchmark emulates a knowledge-seeking query to a
search engine for information which may be contained in either text-based
articles or images. Given a query, the goal is to identify which information
is relevant across modalities, and to generate a full natural language answer
based on the selected sources. The dataset contains 50k QA pairs, half of
which are text-based and the other half are image-based.

We use the KD-VLP model to first select relevant sources using a
classification head. Then, by adding a decoder in a Fusion-in-Decoder manner
as in [33], a predicted answer is generated based on the retrieved sources.
Similar behaviors can be studied for WebQA as in the previous section. The
following analyses will focus on the L2V attention in the KD-VLP encoder, as
these visualizations helps in understanding how the model generates answers.

For this analysis, attention head (11, 5), identified in the same way as
previous analyses, was selected. Figure 9 shows the V2L attention from image
to the highlighted word above the pictures. These heatmaps show that the model
attends more to the regions that help answer the question. For instance, when
the question is about pillars, Figure 9(b) shows that the model attention
comes particularly from the columns in the picture. In order to determine the
color of the center of the flower in Figure 9(c), the model exhibits attention
from the flower center in the image and generates the correct color (yellow).
Furthermore, these visualizations can also be generated for incorrectly
answered questions, providing insights into the reason why incorrect answers
were generated. For example, one may imagine why the model answered
incorrectly in Figure 9(e) and 9(f): In 9(e), the attire that the question
asks about was misidentified. That is, instead of getting attention from
patches of the maroon dress, the model focuses on the white feathers. A
similar behavior is also seen in 9(f), where the model fails to locate the
olives but focuses on the yellow vegetable in the tagine. In both cases, the
model answers according to the identified object color (i.e., ”white” feathers
and ”yellow” olive). These interpretations of attention helps users identify
why a model fails in certain cases, and provides guidance for future efforts
in improving model accuracy.

##  5 Conclusions and Future Directions

In this paper we presented VL-InterpreT, an interactive visualization tool for
interpreting vision-language transformers. This tool allows for interactive
analysis of attention and hidden representations in each layer of any VL
transformer. VL-InterpreT can be used to freely explore the interactions
between and within different modalities to better understand the inner
mechanisms of a transformer model, and to obtain insight into why certain
predictions are made. Through case studies, we demonstrated how VL-InterpreT
can be used to validate the learning of cross-modal concepts, as well as to
“explain” cases of failure.

In the latest version, VL-Interpret is able to run a live model to process
user-generated examples in real time. This allows interactive manipulations of
inputs, including both text and image, to study their effects on the attention
and hidden representations.

For future work, we would like to include aggregated metrics and
visualizations over multiple samples to obtain a more comprehensive
understanding of model operation. In addition, we hope to experiment with any
additional functionalities that will assist users in interpreting multimodal
transformers, and continue to enhance this interpretability tool.

## References

  * [1] Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek.  On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation.  PloS one, 10(7):e0130140, 2015.
  * [2] Alexander Kirillov Bowen Cheng, Alexander G. Schwing.  Per-pixel classification is not all you need for semantic segmentation.  2021\.
  * [3] Jize Cao, Zhe Gan, Yu Cheng, Licheng Yu, Yen-Chun Chen, and Jingjing Liu.  Behind the scene: Revealing the secrets of pre-trained vision-and-language models.  In European Conference on Computer Vision, pages 565–580. Springer, 2020.
  * [4] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In European Conference on Computer Vision, pages 213–229. Springer, 2020.
  * [5] Yingshan Chang, Mridu Narang, Hisami Suzuki, Guihong Cao, Jianfeng Gao, and Yonatan Bisk.  Webqa: Multihop and multimodal qa.  arXiv preprint arXiv:2109.00590, 2021.
  * [6] Hila Chefer, Shir Gur, and Lior Wolf.  Generic attention-model explainability for interpreting bi-modal and encoder-decoder transformers.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 397–406, October 2021.
  * [7] Hila Chefer, Shir Gur, and Lior Wolf.  Transformer interpretability beyond attention visualization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 782–791, 2021.
  * [8] Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning.  What does BERT look at? An analysis of BERT’s attention.  In BlackboxNLP@ACL, 2019.
  * [9] Jean-Baptiste Cordonnier, Andreas Loukas, and Martin Jaggi.  On the relationship between self-attention and convolutional layers.  In International Conference on Learning Representations, 2020.
  * [10] Marcella Cornia, Matteo Stefanini, Lorenzo Baraldi, and Rita Cucchiara.  Meshed-memory transformer for image captioning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10578–10587, 2020.
  * [11] Adam Dahlgren Lindström, Johanna Björklund, Suna Bensch, and Frank Drewes.  Probing multimodal embeddings for linguistic properties: the visual-semantic case.  In Proceedings of the 28th International Conference on Computational Linguistics, pages 730–744, Barcelona, Spain (Online), Dec. 2020\. International Committee on Computational Linguistics.
  * [12] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
  * [13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In International Conference on Learning Representations, 2021.
  * [14] Lisa Anne Hendricks and Aida Nematzadeh.  Probing image-language transformers for verb understanding.  In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3635–3644, Online, Aug. 2021. Association for Computational Linguistics.
  * [15] Benjamin Hoover, Hendrik Strobelt, and Sebastian Gehrmann.  exBERT: A Visual Analysis Tool to Explore Learned Representations in Transformer Models.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 187–196, Online, July 2020. Association for Computational Linguistics.
  * [16] Bo-Jian Hou and Zhi-Hua Zhou.  Learning with interpretable structure from gated rnn.  IEEE transactions on neural networks and learning systems, 31(7):2267–2279, 2020.
  * [17] Andrej Karpathy, Justin Johnson, and Li Fei-Fei.  Visualizing and understanding recurrent networks.  arXiv preprint arXiv:1506.02078, 2015.
  * [18] Vasudev Lal, Arden Ma, Estelle Aflalo, Phillip Howard, Ana Simoes, Daniel Korat, Oren Pereg, Gadi Singer, and Moshe Wasserblat.  InterpreT: An interactive visualization tool for interpreting transformers.  In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 135–142, Online, Apr. 2021. Association for Computational Linguistics.
  * [19] Guang Li, Linchao Zhu, Ping Liu, and Yi Yang.  Entangled transformer for image captioning.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8928–8937, 2019.
  * [20] Raymond Li, Wen Xiao, Lanjun Wang, Hyeju Jang, and Giuseppe Carenini.  T3-vis: visual analytic for training and fine-tuning transformers in NLP.  In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 220–230, Online and Punta Cana, Dominican Republic, Nov. 2021. Association for Computational Linguistics.
  * [21] Yongjie Lin, Yi Chern Tan, and Robert Frank.  Open sesame: Getting inside BERT’s linguistic knowledge.  In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 241–253, Florence, Italy, Aug. 2019. Association for Computational Linguistics.
  * [22] Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith.  Linguistic knowledge and transferability of contextual representations.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1073–1094, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
  * [23] Yongfei Liu, Chenfei Wu, Shao-Yen Tseng, Vasudev Lal, Xuming He, and Nan Duan.  KD-VLP: Improving end-to-end vision-and-language pretraining with object knowledge distillation, 2021.
  * [24] Harsha Nori, Samuel Jenkins, Paul Koch, and Rich Caruana.  Interpretml: A unified framework for machine learning interpretability.  arXiv preprint arXiv:1909.09223, 2019.
  * [25] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In ICML, 2021.
  * [26] Emmanuelle Salin, Badreddine Farah, Stéphane Ayache, and Benoit Favre.  Are vision-language transformers learning multimodal representations? A probing perspective.  In AAAI 2022, 2022.
  * [27] Hendrik Strobelt, Sebastian Gehrmann, Hanspeter Pfister, and Alexander M Rush.  Lstmvis: A tool for visual analysis of hidden state dynamics in recurrent neural networks.  IEEE transactions on visualization and computer graphics, 24(1):667–676, 2017.
  * [28] Ian Tenney, James Wexler, Jasmijn Bastings, Tolga Bolukbasi, Andy Coenen, Sebastian Gehrmann, Ellen Jiang, Mahima Pushkarna, Carey Radebaugh, Emily Reif, and Ann Yuan.  The language interpretability tool: Extensible, interactive visualizations and analysis for NLP models.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 107–118, Online, Oct. 2020. Association for Computational Linguistics.
  * [29] Laurens van der Maaten.  Learning a parametric embedding by preserving local structure.  In Proceeding of AISTATS 2009., 2009.
  * [30] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
  * [31] Jesse Vig and Yonatan Belinkov.  Analyzing the structure of attention in a transformer language model.  In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 63–76, 2019.
  * [32] James Wexler, Mahima Pushkarna, Tolga Bolukbasi, Martin Wattenberg, Fernanda Viégas, and Jimbo Wilson.  The what-if tool: Interactive probing of machine learning models.  IEEE transactions on visualization and computer graphics, 26(1):56–65, 2019.
  * [33] Donghan Yu, Chenguang Zhu, Yuwei Fang, Wenhao Yu, Shuohang Wang, Yichong Xu, Xiang Ren, Yiming Yang, and Michael Zeng.  KG-fid: Infusing knowledge graph in fusion-in-decoder for open-domain question answering, 2022.
  * [34] Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi.  From recognition to cognition: Visual commonsense reasoning.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6720–6731, 2019.
  * [35] Quanshi Zhang, Ruiming Cao, Feng Shi, Ying Nian Wu, and Song-Chun Zhu.  Interpreting cnn knowledge via an explanatory graph.  In Thirty-Second AAAI Conference on Artificial Intelligence, 2018\.
  * [36] Quanshi Zhang, Yu Yang, Haotian Ma, and Ying Nian Wu.  Interpreting cnns via decision trees.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6261–6270, 2019.