# Robust Multi-Agent Reinforcement Learning with Social Empowerment for
Coordination and Communication

Tessa van der Heiden,1 Herke van Hoof, 2 Efstratios Gavves, 2 Christoph Salge
3

###### Abstract

We consider the problem of robust multi-agent reinforcement learning (MARL)
for cooperative communication and coordination tasks. MARL agents, mainly
those trained in a centralized way, can be brittle because they can overfit to
their training partners. This overfitting can produce agents that adopt
policies that act under the expectation that other agents will act a certain
way rather than react to their actions. Our objective is to bias the learning
process towards finding strategies that remain reactive towards others’
behavior. Social empowerment measures the potential influence between agents’
actions. We propose it as an additional reward term, so agents better adapt to
other agents’ actions. We show that the proposed method results in obtaining
higher rewards faster and a higher success rate in three cooperative
communication and coordination tasks.

##  1 Introduction

Multi-agent reinforcement learning (MARL) holds considerable promise to help
address a variety of cooperative multi-agent problems. However, a problem with
MARL is that agents can derive strong policies that are overfitted to their
partners’ behavior. Specifically, with centralized training agents can adopt
strategies that expect other agents to act in a certain way rather than react
to their actions. Such systems are undesirable as they may fail when their
partners alter their strategies or have to collaborate with novel partners,
either during the learning or deployment phase.

Robust Reinforcement learning (RL) was introduced by Morimoto and Doya (2005)
considering the generalization ability of the learned policy in the single-
agent setting. We consider robustness to be a training agent’s ability to deal
with changes in the other agents’ behavior.

Our objective is to obtain multi-agent policies that avoid this specific lack
of robustness and stay reactive to changes in other agents’ policy. Agents
acting without any sensor input and attention towards their partners cannot
have this ability. We want to introduce an additional reward to bias learning
towards socially reactive strategies - which should fulfil the following
constraints: 1) it should, with minimal adaptation, apply to a wide range of
problems with different sensor-actuator configurations, to preserve the
universality of the RL framework, 2) it should not negatively affect the
performance, i.e., once good policies are found, it should not harm
exploitation. Optimizing for the mutual information between agents’ actions
(Jaques et al. 2019), requires the agents’ policies to have a certain degree
of entropy and this might interfere with an exploitation strategy. Fulfilling
the above criteria would provide a general-purpose multi-agent learning
algorithm for various cooperative tasks.

##  2 Related work

###  2.1 Robust Multi-Agent Reinforcement Learning

There is a large body of research on constructing agents that are robust to
their partners. In self-play, for example, agents train against themselves
rather than a fixed opponent strategy to prevent developing exploitable
strategies (Tesauro 1994). Population based-training goes one step further by
training agents to play against a population of other agents rather than only
a copy of itself. For instance, Jaderberg et al. (2018) and Lowe et al. (2017)
train an ensemble of policies with a variety of collaborators and competitors.
By using a whole population rather than only a copy of itself, the agent is
forced to be robust to a wide variety of potential perturbations instead of a
single perturbation. However, it requires a great deal of engineering because
the policy parameters suitable for the previous environment are not
necessarily the next stage’s best initialization.

Some works combine the minimax framework and MARL to find policies that are
robust to opponents with different strategies. Minimax is a concept in game
theory that can be applied to find an approach that minimizes the possible
loss in a worst-case scenario (Osborne et al. 2004). Li et al. (2019) use it
during training to optimize the reward for each agent under the assumption
that all other agents act adversarial. We are interested in methods that can
deal with perturbations in the training partners’ behavior, which differs from
dealing with partners with various strategies.

In zero-shot coordination, agents need to cooperate with novel partners that
are not seen during training. Other-play (Hu et al. 2020) constructs zero-shot
coordination strategies by making their agents robust to how their partners
break symmetries in the underlying system. We instead frame our problem as
one-shot adaptation, i.e. our policy should be able to quickly adapt if the
other agents alter their policies.

###  2.2 Intrinsic Social Motivation

Due to centralized training in MARL, agents might adopt non-reactive
strategies that may struggle with other agents’ changing behaviors. Social
intrinsic motivation can give an additional incentive to find reactive
policies towards other agents.

Intrinsic motivation (IM) in RL refers to reward functions that allow agents
to learn interesting behavior, sometimes in the absence of an environmental
reward (Chentanez, Barto, and Singh 2005). Computational models of IM are
generally separated into two categories (Santucci, Baldassarre, and Mirolli
2012), (Baldassarre and Mirolli 2013), those that focus on curiosity (Burda et
al. 2018), (Pathak et al. 2017) and exploration (Gregor, Rezende, and Wierstra
2016), (Eysenbach et al. 2018), and those that focus on competence and control
(Oudeyer and Kaplan 2009), (Karl et al. 2017). The information-theoretic
Empowerment formalism is in the latter category, trying to capture how much an
agent is in control of the world it can perceive. Empowerment has produced
robust behaviour linked to controllability, operationality and self-
preservation - in both robots (van der Heiden et al. 2020), (Karl et al.
2017), (Leu et al. 2013) and simulations (Guckelsberger, Salge, and Colton
2016), with (de Abril and Kanai 2018) and without (Guckelsberger, Salge, and
Togelius 2018) reinforcement learning and neural network approximations (Karl
et al. 2017).

Empowerment has also been applied to multi-agent simulations, under the term
of coupled empowerment maximization (Guckelsberger, Salge, and Colton 2016),
in which it was used to produce supportive and antagonistic behavior. Of
particular interest is the idea of transfer empowerment - a measure that
quantifies behaviors such as collaboration, coordination, and lead-taking
(Salge and Polani 2017). Empowerment has also been tested in collaborative
scenarios involving humans and robots. In all scenarios, transfer empowerment
provides an additional incentive for agents to interact and coordinate.

There are similar techniques that quantify the interaction between agents for
improving coordination between agents. Barton et al. (2018) analyze the degree
of dependence between two agents’ policies to measure coordination,
specifically by using Convergence Cross Mapping (CCM). Strouse et al. (2018)
show how agents can share (or hide) intentions by maximising the mutual
information between actions and a categorical goal. One notably relevant work
is by Jaques et al. (2019) that design social influence which is the influence
of one agent on the policies of other agents, measured by the mutual
information between action pairs of distinct agents. In contrast to social
influence, transfer empowerment considers the potential mutual information or
channel capacity. When optimizing for actual mutual information, then its
value is bounded from above by the lowest entropy of both agent’s action
variables. This might easily interfere with an exploitation strategy and may
need to be regularized once a good strategy is found. On the other hand,
empowerment does not have this limitation and the action sets could have very
narrow distributions.

##  3 Preliminaries

Let us consider a Dec-POMDP, an extension of the MDP for multi-agent systems,
being both decentralized and partially observable (Nair et al., 2003). This
means that each of the $n$ agents conditions the choice of its action on its
partial observation of the world. It is defined by the following tuple:
$(\textbf{S},\boldsymbol{A},T,\boldsymbol{O},O,R,n)$. $\boldsymbol{S}$ is the
set of true global states, while
$\boldsymbol{O}=\otimes_{i\in[1,..,n]}\boldsymbol{O}_{i}$ is the set of joint
local observations and
$\boldsymbol{A}=\otimes_{i\in[1,..,n]}\boldsymbol{A}_{i}$ the set of possible
joint actions. At each time step, the state transition function $T$ maps the
joint action and state to a new state, with probability distribution
$P(s_{t+1}|s_{t},\boldsymbol{a}_{t})$.

We consider fully cooperative tasks, so agents share a reward $r_{t}$ which is
conditioned on the joint action and state. $R$ is the reward function that
maps the state and joint actions into a single scalar reward.

##  4 Methodology

In the introduction, we have argued that agents that are trained in a
centralized manner can adopt brittle policies and perform poorly, even in
simple coordination tasks. The reason is that they adopt strategies that
expect other agents will act in a certain way rather than observe and
anticipate their actions.

This section aims to describe an additional heuristic that biases the learning
process in obtaining policies towards those that are reactive towards other
agents’ actions.

First we introduce our specific version of transfer empowerment, that rewards
the idea of an agent being responsive to adaptations in the other’s policy.
Transfer empowerment is computed for a each pair of agents, which scales
quadratically with the number of agents.

By adopting the framework of centralized training, we can compute empowerment
over the joint action and states. This notion captures the causal influence of
all agents’ actions on the overall resulting state. Since it is calculated
over the joint actions and states and not per pair, we arrive at a more
scalable solution.

###  4.1 Transfer Empowerment

Suppose there are two agents, $j$ and $k$, both taking actions and changing
the overall state. Each time agent $k$ takes an action, the state of agent $j$
is modified and therefore agent $j$’s policy conditions on it. The objective
of coordination is that by modifying the actions of agent $k$ , agent $j$ also
_reliably_ adapts its actions. Here we look at transfer empowerment, namely
the _potential_ causal influence that one agent has on another. It is computed
for pairs of agents by maximizing the mutual information $\mathcal{I}$ between
one agent’s action $a^{k}_{t}$ and another agent’s action $a^{j}_{t+1}$ at a
later point in time, conditioned on the current and next state, $s_{t}$ and
$s_{t+1}$ respectively:

$\mathcal{E}^{T,k\rightarrow
j}(s_{t})=\underset{\omega^{k}}{\max~{}}\mathcal{I}\left[a^{j}_{t+1},a^{k}_{t}|s_{t}\right]$
(1)

Here, $\omega^{k}(a^{k}_{t+1}|s_{t+1})$ is the _hypothetical_ policy of agent
$k$, that takes an action $a^{k}_{t+1}$ after observing state $s_{t+1}$ and
influencing $a^{j}_{t+2}$ at a later time step.

Our version of transfer empowerment differs slightly from the one introduced
by (Salge and Polani 2017), as we consider the potential information flow, or
channel capacity, between one agent’s action and another agent’s action at a
later point in time. Salge and Polani (2017) consider the empowerment between
one agent’s action and another agent’s _sensor_ state. Transfer empowerment to
another agent’s sensor captures the direct influence to change the other
agents environment, and also influencing the agent to take certain actions,
which then influences their world. Using transfer empowerment to another
agent’s _action_ focuses on just the influence that affects the other agent’s
decision. Since an agent is usually in control of their actions, influence on
their action has to flow through their sensor and be mediated by their policy.
In other words, they have to react to the first agent’s actions. This is more
in line with our goal of biasing a policy towards more reactive strategies.

Transfer empowerment has ties with but is different from social influence
(Jaques et al. 2019). Social influence is the mutual information between
agents’ actions. It is high when both action variables have a certain entropy,
i.e., that different actions are performed. Once an acceptable policy is
found, e.g. towards end of training, a high entropy policy distribution might
actually be suboptimal. Empowerment on the other hand, considers the potential
and not actual information flow, so the agent will only calculate how it could
have affected the other agent, rather than actually carry out its potential.
As such, action sets can have very narrow distributions, as long as the system
would still be reactive if those actions change. Therefore it does not
interfere with the constraints stated earlier.

Here we described transfer empowerment computed from agent $k$’s actions to
agent’s $j$’ action, denoted by $\mathcal{E}^{T,k\rightarrow j}$.
$\mathcal{E}^{T,1:n\rightarrow j}=\sum_{\begin{subarray}{c}i=1\\\ i\neq
j\end{subarray}}^{n}\mathcal{E}^{T,i\rightarrow j}$ is the extension for
multiple agents and results in agent $j$ being reactive to agents’ actions.
The main goal is to train a _behavioral_ policy $\pi^{j}$ to maximize the
transfer empowerment, which is done by estimating $\mathcal{E}^{T}$ with
$\omega^{k}$ and by changing $\pi^{j}$.

###  4.2 Joint Empowerment as Scalable Proxy

Ideally, we would want to compute the transfer empowerment from all agents to
all other agents. However, this grows quadratically and might become quickly
infeasible. Therefore we introduce joint empowerment as a better computable
heuristic proxy. We consider multiple agent with
$\boldsymbol{\omega}=[\omega^{1},\omega^{2},...,\omega^{n}]$ denoting the
joint policy. Since we can adopt the framework of centralized training with
decentralized execution, policies are allowed to use all agents’ local
observations during training. Joint empowerment
$\mathcal{E}^{J}(\boldsymbol{a}_{t},s_{t})$ is computed over the joint action
$\boldsymbol{a}_{t}$ and next state $s_{t+1}$ for all agents:

$\mathcal{E}^{J}(s_{t})=\underset{\boldsymbol{\omega}}{\max~{}}\mathcal{I}\left[s_{t+1},\boldsymbol{a}_{t}\middle|s_{t}\right]$
(2)

This is the empowerment from all agent’s actions to the state at a later point
in time. It captures the causal influence of all agents actions on the overall
resulting state. It also provides an upper bound on each single agent self-
empowerment, a measure associated with robust single-agent behavior. It also
provides a measure of the agents’ collective empowerment if seen as a single
organism. As such, it might also favor strategies that keep the overall
population operational and increase their controllability over the world,
which may positively affect the learning rate and performance.

This reward is preferred in systems with many agents as it scales linearly
with the number of agents, rather than quadratic.

###  4.3 Training with Social Empowerment

In summary, training with joint and transfer empowerment results in joint
policies that are reactive, because for empowerment to be high it requires
considering the decisions of others. As such, social empowerment rewards a
very generalized idea of coordination that requires paying attention to each
other and reliably reacting to a variation in their actions. While empowerment
does not care how this reaction looks, or even if it is good, combined with
the actual reward should lead to the selection of a strategy that both solves
the problem, while also avoiding the brittleness that comes from not being
reactive to information from other agents. Specifically, we will modify an
agent’s reward function $R_{i}$ so that it becomes:

$R_{i,t}(s_{t+1},\boldsymbol{a_{t}},s_{t})=r(\boldsymbol{a_{t}},s_{t})+\mathcal{E}(s_{t+1})$
(3)

where $r$ is the environmental reward and $\mathcal{E}$ can be either transfer
$\mathcal{E}^{T}$ or joint $\mathcal{E}^{J}$ empowerment. Policies
$\boldsymbol{\pi}$ that are trained with $\mathcal{E}^{J}$ jointly select
actions that result in highly empowered states (like in classical empowerment
algorithms). A policy $\pi^{j}$ trained with $\mathcal{E}^{T,n\rightarrow j}$
selects more reactive actions, therefore also increases the empowerment of
states.

###  4.4 Lower bound on $\mathcal{E}$

Many works on empowerment operate in discrete action-state space because the
computation requires integrating over all possible actions and states, which
in many cases is intractable. We choose to operate in continuous action-state
space to extend the applicability of our approach. Karl et al. (2017)
introduced an efficient implementation to estimate empowerment that maximises
a lower bound on mutual information (Barber and Agakov 2003). The lower bound
on empowerment contains a source distribution, $\omega(a_{t},s_{t})$ and a
variational approximation of the planning distribution,
$q(a_{t}|s_{t+1},s_{t})$. By representing both distributions by a neural
network, maximizing $\hat{\mathcal{I}}$ requires backpropagation using
gradient _ascent_. Taking inspiration from, Karl et al. (2017), transfer
empowerment can be estimated by:

$\mathcal{E}^{T}_{i}(\theta,\phi)=\ln
q_{\xi}(\boldsymbol{a}^{\omega}_{i+1}|\pi^{j}_{\phi}(s_{i+2}),s_{i+2},s_{i+1})-\ln\omega_{\chi}^{k}(\boldsymbol{a}^{\omega}_{i+1}|s_{i+1})$
(4)

where the combination of the parameters $\xi$ and $\chi$ are denoted by
$\theta$. Similarly, joint empowerment can be estimated by:

$\mathcal{E}^{J}_{i}(\theta)=\ln
q_{\xi}(\boldsymbol{a}^{\omega}_{i+1}|s_{i+2},s_{i+1})-\ln\omega_{\chi}(\boldsymbol{a}^{\omega}_{i+1}|s_{i+1})$
(5)

The training procedures are summarized in algorithm 1 and 2 for transfer and
joint empowerment respectively. First, samples from the system dynamics with
the behavioral policy are collected. The transition network $p$ is trained to
minimize the squared error between a future state and its predicted future
state. In algorithm 2, the estimation of empowerment is done by $\omega$, $q$
and $p$. Next, it is used as an additional reward term for the joint policy
$\boldsymbol{\pi}$ to select actions that lead to high-empowered states. In
algorithm 1, empowerment is estimated with the networks $\omega$, $q$, $p$ and
$\pi^{j}$. So here, $\pi^{j}$ is trained to select actions that drive to
highly empowered states and to increase the states’ empowerment values.

Set random parameters for $\omega_{\chi}$ and $q_{\xi}$ with
$\theta=\\{\chi,\xi\\}$ and $\boldsymbol{\pi}_{\phi}$ and $Q_{\beta}$.
Initialize experience replay memory with mini batches $\mathcal{B}$ with $N$
transitions.

for each
$\\{s_{i},\boldsymbol{a}^{\pi}_{i},s_{i+1},r_{i}(\boldsymbol{a}^{\pi},s_{i})\\}\in\mathcal{B}$
do

$\epsilon_{i}=(s_{i+1}-p_{\psi}(\boldsymbol{a}^{\pi}_{i},s_{i}))^{2}$

$\boldsymbol{a}^{\omega}_{i+1}\sim\omega_{\chi}(s_{i+1})$, $s_{i+2}\sim
p_{\psi}(\boldsymbol{a}^{\omega}_{i+1},s_{i+1})$

Estimation of $\mathcal{E}^{T}$ as in eq. (4)

$y_{i}=\mathcal{E}^{T}_{i}(\theta,\phi)+r_{i}+\gamma
Q^{\prime}_{\beta^{\prime}}(s_{i+1},\boldsymbol{\pi^{\prime}}_{\phi^{\prime}}(s_{i+1}))$

end for

$\psi\leftarrow\psi-\lambda\nabla_{\psi}\frac{1}{N}\sum_{i}^{N}\epsilon_{i}$

$\theta\leftarrow\theta+\lambda\nabla_{\theta}\frac{1}{N}\sum_{i}^{N}\mathcal{E}^{T}_{i}(\theta,\phi)$

$\beta\leftarrow\beta+\lambda\nabla_{\beta}\frac{1}{N}\sum_{i}^{N}(Q_{\beta}(s_{i},\boldsymbol{a}_{i})-y_{i})^{2}$

$\phi\leftarrow\phi+\lambda\nabla_{\phi}\frac{1}{N}\sum_{i}^{N}(Q_{\beta}(s_{i},\boldsymbol{\pi}_{\phi}(s_{i}))+\mathcal{E}^{T}_{i}(\theta,\phi))$

Algorithm 1 $\pi^{j}_{\phi}$ trained with $\mathcal{E}^{T}$

Set random parameters for $\omega_{\chi}$ and $q_{\xi}$ with
$\theta=\\{\chi,\xi\\}$ and $\boldsymbol{\pi}_{\phi}$ and $Q_{\beta}$,
$p_{\psi}$ trained as in algorithm 1. Initialize experience replay memory with
mini batches $\mathcal{B}$ with $N$ transitions.

for each
$\\{s_{i},\boldsymbol{a}^{\pi}_{i},s_{i+1},r_{i}(\boldsymbol{a}^{\pi},s_{i})\\}\in\mathcal{B}$
do

$\boldsymbol{a}^{\omega}_{i+1}\sim\omega_{\chi}(s_{i+1})$,

$s_{i+2}\sim p_{\psi}(\boldsymbol{a}^{\omega}_{i+1},s_{i+1})$

Estimation of $\mathcal{E}^{J}$ as in eq. (5)

$y_{i}=\mathcal{E}^{J}_{i}(\theta)+r_{i}+\gamma
Q^{\prime}_{\beta^{\prime}}(s_{i+1},\boldsymbol{\pi^{\prime}_{\phi^{\prime}}}(s_{i+1}))$

end for

$\theta\leftarrow\theta+\lambda\nabla_{\theta}\frac{1}{N}\sum_{i}^{N}\mathcal{E}^{J}_{i}(\theta)$

$\beta\leftarrow\beta+\lambda\nabla_{\beta}\frac{1}{N}\sum_{i}^{N}(Q_{\beta}(s_{i},\boldsymbol{a}_{i})-y_{i})^{2}$

$\phi\leftarrow\phi+\lambda\nabla_{\phi}\frac{1}{N}\sum_{i}^{N}(Q_{\beta}(s_{i},\boldsymbol{\pi}_{\phi}(s_{i}))$

Algorithm 2 $\boldsymbol{\pi}_{\phi}$ trained with $\mathcal{E}^{J}$

##  5 Experimental Results

###  5.1 Setup

We construct three environments that test various capabilities of our
approach, transfer $\mathcal{E}^{T}$, joint $\mathcal{E}^{J}$ and social
influence SI (Jaques et al. 2019) and baselines DDPG (Lillicrap et al. 2015)
and MADDPG (Lowe et al. 2017). The MADDPG is a centralized version of the
DDPG, meaning the value function of each agent conditions on the observations
of all agents.

We first analyze the robustness of the methods by looking at the training
curves. During training, all agents change their policies and a robust
learning method should deal with these perturbations. Second, we assess each
method’s ability to obtain reactive agents. We analyze this in a simulator
with two agents, one sending messages and the other receiving them. The task
can only be fulfilled if messages are correctly decoded, and random action
taking is avoided. Third, we evaluate each method’s ability to construct
agents that remain responsive to its partners and not rely upon expectations.
We hypothesize that centralized training does not incentivize this behavior.
To this end, we utilize a simulator containing multiple particles that need to
cover _distinct_ landmarks. If agents collide or go to the same landmark, we
consider them not be reactive to each other. Last, we compare methods in a
more complicated multi-agent car simulator and analyze their performance in
more real-world scenarios.

The first two scenarios are constructed by OpenAI’s Multi-Agent Particle
Environment 111https://github.com/openai/multiagent-particle-envs. We choose
this simulator, since we are primarily interested in addressing agent
interaction and are satisfied with relatively simple physics, i.e. it only
contains inertia and friction. Transfer empowerment is evaluated in the first
experiment and joint empowerment in the second. The last experiment is done in
a simulator that is a combination of the multi-agent and single agent OpenAI
Gym 222https://github.com/openai/gym. It allows to evaluate more complicated
interactions, e.g. merging and overtaking behaviors.The code for the simulator
can be found on https://github.com/tessavdheiden/social˙empowerment

Table 1: Percentage of episodes in which the listener reached the target landmark and average distance from the landmark. The right columns (3rd-6th) show results of the task in which the environment contains less channels than landmarks (L=6, C=5) and obstacles (O=6). Alternating frequency is the rate at which the speaker alters its message per episode. A low number indicates that the speaker efficiently uses its communication channel.  | $L=3,C=3,O=0$ | $L=6,C=5,O=6$
---|---|---
| Reward | Average | Target | Reward | Average | Target | Obstacle
|  | distance | reach % |  | distance | reach % | hit %
DDPG | -0.439 | 0.456 | 32.0 | -0.851 | 0.653 | 30.1 | 60.8
MADDPG | -0.301 | 0.133 | 92.0 | -0.596 | 0.260 | 40.5 | 49.4
SI | -0.569 | 0.290 | 88.0 | -0.666 | 0.360 | 31.5 | 51.2
$\mathcal{E}^{T}$ | -0.292 | 0.120 | 91.1 | -0.422 | 0.072 | 61.1 | 31.1
Table 2: Average percentage of collisions and successes per episode consisting of 25 time steps. Distance from landmark are averaged over all time steps. An episode is successful if all agents are within .1 distance to a distinct landmarks at the last 5 time steps. Shaded areas indicate best (dark grey) and second best (light grey) numbers.  | Reward | Average distance | Collisions % | Success %
---|---|---|---|---
DDPG | -2.302 | 1.324 | 50.0 | 69.3
MADDPG | -2.198 | 1.108 | 53.3 | 80.5
SI | -2.461 | 1.459 | 70.3 | 61.9
$\mathcal{E}^{T}$ | -2.098 | 0.857 | 43.3 | 89.9
$\mathcal{E}^{J}$ | -2.063 | 0.957 | 13.3 | 95.9

###  5.2 Training with Empowerment

During training, all agents change their policies, and a robust learning
method should deal with these perturbations. We hypothesize that training with
empowerment results in robust methods because it incentivizes being responsive
to changes in others’ behaviors.

Figures 1(i) and 1(ii) show the training curves for the baselines and our
method in the first two tasks. When an agent has empowerment as an additional
utility function the value of the average return is higher.

Training Curves

(i) Cover Landmarks

(ii) Speaker Listener

Figure 1: Training curves for Cover Landmarks and Speaker Listener tasks. —
DDPG — MADDPG — SI — $\mathcal{E}$

###  5.3 Experiment 1: Cooperative Communication

The first task involves one silent moving agent ($j$) and the other one ($k$)
sending messages that help navigate the listening agent. The objective for the
moving agent is to reach a target landmark out of $L$ landmarks by decoding
the speaker’s message (a symbol chosen from $C$ distinct symbols). The team
reward is the Euclidean distance from the target landmark. The task requires
both agents to reliably act and pay attention to each other’s actions. The
agents can fail if the speaker outputs random symbols or if the listener
incorrectly decodes or ignores the messages.

If $L=C$, the speaker simply needs to output a unique symbol identifying the
target landmark. By having more landmarks than distinct symbols, i.e. $L>C$,
the speaker is forced to send more indicative signals, e.g. indicating
movement direction. Last, by adding $O$ obstacles that are _only visible to
the speaker_ , the listener is challenged more to decode the messages
correctly and navigate precisely.

Table 1 shows the performance of the baselines and our method on the two
versions of this task. We present $\mathcal{E}^{T,k\rightarrow j}$ which
should motivate the listener ($j$) to adapt its policy to be more responsive
to the messages. $\mathcal{E}^{T,j\rightarrow k}$ should have the speaker
adapt its policy. However, as the messages are not blocked, there is little in
term of policy change for the speaker that is helpful to increase the
empowerment.

The lowest number of obstacle hits was obtained by training with
$\mathcal{E}^{T,k\rightarrow j}$. This can be explained by the listener’s
policy adaptation to decode the messages and navigate better.

Fig. 2 shows an episode. DDPG does not reach the target and hits obstacles,
see 2(i) . While MADDPG avoids the obstacles (see, 2(ii)), it remains at a
larger distance from the target than our method, in 2(iii).

Speaker Listener Task

(i) DDPG

(ii) MADDPG

(iii) $\mathcal{E}^{T,k\rightarrow j}$

Figure 2: The listener (big blue circle) has to move to a target landmark (blue small circle) out of $L=6$ landmarks (black small circles) while avoiding the obstacles (red crosses). The sender can only send messages of length $C=5<L$.  Table 3: Three cars need to avoid collisions and obstacles on the road. All agents are rewarded for staying on the road and penalized if they hit an obstacle or another car.  | Junctions | Obstacles
---|---|---
| Off road % | Collisions % | Obstacle hit % | Collisions %
MADDPG | 11.8 | 12.7 | 29.3 | 31.9
$\mathcal{E}^{J}$ | 6.6 | 5.6 | 15.9 | 12.4

###  5.4 Experiment 2: Cooperative Navigation

In this task three agents have to move to three distinct landmarks. They are
not told to go to a specific one, but rather need to attend and react to their
partners and choose distinct ones. They observe the relative coordinates to
their partners and landmarks. The team reward is the Euclidean distance
between a landmark and the closed agent, summed over all landmarks and agents
obtain a penalty for colliding with each other. If agents collide or go to the
same landmark, we consider them not to be reactive to each other. We
hypothesize that joint empowerment has agents moving to distinct landmarks
faster without colliding and thus obtain a higher reward.

Table 2 presents results. Our method achieves the lowest average distance and
the lowest number of collisions. Near collisions are avoided by using joint
empowerment as additional reward, because it is low in states in which the
agents have low maneuverability.

Cover Landmarks Task

(i) DDPG

(ii) MADDPG

(iii) $\mathcal{E}^{J}$

Figure 3: The task of the agents (big circles) is to reach distinct landmarks
(small circles). Each 3rd time step is shown (from transparent to opaque).

Fig. 3 shows a full episode. The three agents are the big colored circles and
the landmarks are the smaller black ones. MADDPG and MADDPG+$\mathcal{E}$
reach distinct landmarks. Empowerment captures the causal influence of all
agents’ actions on the overall resulting state and agents avoid states with
limited maneuverability. This can be seen by the blue agent that does not move
to the closest landmark, but rather one that is further away from its green
partner 3(iii). MADDPG has two colliding agents. See Fig. 3(ii).

###  5.5 Experiment 3: Cooperative Driving

In the last task we consider three autonomous vehicles that have to interact
in novel traffic situations. The simulator contains physical properties such
as road friction and inertia. Agents observe a stack of 4 top view grey scale
images, centered around their center of mass from which they need to infer the
relative location of their partners and obstacles, and their own speed. They
are rewarded by the number of pixels in their observation corresponding to the
color of the road, motivating them to stay on the road, and are penalized for
colliding with each other.

Intersections and obstacles are added to assess the agents’ abilities to react
in time to others. For example, if multiple agents enter a point where two
roads meet or a location with many obstacles, agents can collide if neither
one gives way. We therefore count the number of collisions and obstacle hits.

Table 3 shows the two different scenarios, the road with junctions and
obstacles. Agents trained with empowerment results in agents outperforming the
MADDPG on all metrics.

In Fig 4 we show two snapshots in the two scenarios. The agents that are
trained with the MADDPG come closer to each other and respond very late. In
contrast, when using $\mathcal{E}^{J}$ the vehicles respond quicker and give
others space to merge in or to avoid obstacles. Agents (blue car) give way,
see 4(ii) and 4(iv), to let the other cars (green, red) pass.

Cooperative Driving

(i) MADDPG

(ii) $\mathcal{E}^{J}$

(iii) MADDPG

(iv) $\mathcal{E}^{J}$

Figure 4: Comparison in the dynamic car simulator in which 3 vehicles need to
merge 4(i), 4(ii) and avoid obstacles 4(iii), 4(iv) while staying on the road
and keeping safe distance between each other.

### Technical details

All methods are trained for 25k episodes, each episode consisting of 25 time-
steps in the first two task and 50 time-steps in the last. The hyper-
parameters as well as the optimization algorithm are equal to (Lowe et al.
2017). In the last experiment agents obtained a stack of 4 images and all
networks had a 5-layered CNN to convert the stack into a single-dimension
vector. The code for the implementation can be found on
https://github.com/tessavdheiden/social˙empowerment

##  6 Conclusions and Future Work

In this paper, we consider the problem of agents that overfit to their
training partners and that adapt strategies that act under certain
expectations, rather than react to their actions. We propose to use social
empowerment that biases the learning process to construct agents that remain
reactive to their partners’ behaviors.

In three experiments we show how empowerment motivates the construction of
agents that are responsive to each other’s behavior. We have explained
different variants, transfer empowerment that can be applied between a pair of
agents, and a faster computational variant, joint empowerment, which can be
applied for a group of agents with any size.

Our method, uses centralized training with decentralized execution, thus can
be deployed with local partial observations of the policies. We show that not
only agents coordinate better by taking into account the dynamics of the whole
group, but also more precisely, avoiding dangerous situations.

An exciting direction of research, which we have not explored is cooperation
with partners that agents have not seen during training, e.g. humans.
Furthermore, many real-life scenarios have competitive aspects, and agents can
also be trained by minimizing the empowerment of their opponents. Focusing on
the before mentioned problems allows tackling a large number of scenarios,
such as robots developed by different brands that need to interact with
humans.

## References

  * Baldassarre and Mirolli (2013) Baldassarre, G.; and Mirolli, M. 2013.  _Intrinsically motivated learning in natural and artificial systems_.  Springer.
  * Barber and Agakov (2003) Barber, D.; and Agakov, F. V. 2003.  The IM algorithm: a variational approach to information maximization.  In _Advances in neural information processing systems_.
  * Barton et al. (2018) Barton, S. L.; Waytowich, N. R.; Zaroukian, E.; and Asher, D. E. 2018.  Measuring collaborative emergent behavior in multi-agent reinforcement learning.  In _International Conference on Human Systems Engineering and Design: Future Trends and Applications_ , 422–427. Springer.
  * Burda et al. (2018) Burda, Y.; Edwards, H.; Pathak, D.; Storkey, A.; Darrell, T.; and Efros, A. A. 2018\.  Large-scale study of curiosity-driven learning.  _arXiv preprint arXiv:1808.04355_ .
  * Chentanez, Barto, and Singh (2005) Chentanez, N.; Barto, A. G.; and Singh, S. P. 2005.  Intrinsically motivated reinforcement learning.  In _Advances in neural information processing systems_ , 1281–1288.
  * de Abril and Kanai (2018) de Abril, I. M.; and Kanai, R. 2018.  A unified strategy for implementing curiosity and empowerment driven reinforcement learning.  _arXiv preprint arXiv:1806.06505_ .
  * Eysenbach et al. (2018) Eysenbach, B.; Gupta, A.; Ibarz, J.; and Levine, S. 2018.  Diversity is all you need: Learning skills without a reward function.  _arXiv preprint arXiv:1802.06070_ .
  * Gregor, Rezende, and Wierstra (2016) Gregor, K.; Rezende, D. J.; and Wierstra, D. 2016.  Variational intrinsic control.  _arXiv preprint arXiv:1611.07507_ .
  * Guckelsberger, Salge, and Colton (2016) Guckelsberger, C.; Salge, C.; and Colton, S. 2016.  Intrinsically motivated general companion npcs via coupled empowerment maximisation.  In _2016 IEEE Conference on Computational Intelligence and Games (CIG)_ , 1–8. IEEE.
  * Guckelsberger, Salge, and Togelius (2018) Guckelsberger, C.; Salge, C.; and Togelius, J. 2018.  New and surprising ways to be mean. adversarial npcs with coupled empowerment minimisation.  _arXiv preprint arXiv:1806.01387_ .
  * Hu et al. (2020) Hu, H.; Lerer, A.; Peysakhovich, A.; and Foerster, J. 2020.  ” Other-Play” for Zero-Shot Coordination.  _arXiv preprint arXiv:2003.02979_ .
  * Jaderberg et al. (2018) Jaderberg, M.; Czarnecki, W.; Dunning, I.; Marris, L.; Lever, G.; Castaneda, A.; et al. 2018.  Human-level performance in first-person multiplayer games with population-based deep reinforcement learning. arXiv.  _arXiv preprint arXiv:1807.01281_ .
  * Jaques et al. (2019) Jaques, N.; Lazaridou, A.; Hughes, E.; Gulcehre, C.; Ortega, P.; Strouse, D.; Leibo, J. Z.; and De Freitas, N. 2019.  Social influence as intrinsic motivation for multi-agent deep reinforcement learning.  In _International Conference on Machine Learning_ , 3040–3049. PMLR.
  * Karl et al. (2017) Karl, M.; Soelch, M.; Becker-Ehmck, P.; Benbouzid, D.; van der Smagt, P.; and Bayer, J. 2017.  Unsupervised real-time control through variational empowerment.  _arXiv preprint arXiv:1710.05101_ .
  * Leu et al. (2013) Leu, A.; Ristić-Durrant, D.; Slavnić, S.; Glackin, C.; Salge, C.; Polani, D.; Badii, A.; Khan, A.; and Raval, R. 2013.  CORBYS cognitive control architecture for robotic follower.  In _Proceedings of the 2013 IEEE/SICE International Symposium on System Integration_ , 394–399. IEEE.
  * Li et al. (2019) Li, S.; Wu, Y.; Cui, X.; Dong, H.; Fang, F.; and Russell, S. 2019.  Robust multi-agent reinforcement learning via minimax deep deterministic policy gradient.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, 4213–4220.
  * Lillicrap et al. (2015) Lillicrap, T. P.; Hunt, J. J.; Pritzel, A.; Heess, N.; Erez, T.; Tassa, Y.; Silver, D.; and Wierstra, D. 2015.  Continuous control with deep reinforcement learning.  _arXiv preprint arXiv:1509.02971_ .
  * Lowe et al. (2017) Lowe, R.; Wu, Y. I.; Tamar, A.; Harb, J.; Abbeel, O. P.; and Mordatch, I. 2017.  Multi-agent actor-critic for mixed cooperative-competitive environments.  In _Advances in neural information processing systems_ , 6379–6390.
  * Morimoto and Doya (2005) Morimoto, J.; and Doya, K. 2005.  Robust reinforcement learning.  _Neural computation_ 17(2): 335–359.
  * Osborne et al. (2004) Osborne, M. J.; et al. 2004.  _An introduction to game theory_ , volume 3.  Oxford university press New York.
  * Oudeyer and Kaplan (2009) Oudeyer, P.-Y.; and Kaplan, F. 2009.  What is intrinsic motivation? A typology of computational approaches.  _Frontiers in neurorobotics_ 1: 6.
  * Pathak et al. (2017) Pathak, D.; Agrawal, P.; Efros, A. A.; and Darrell, T. 2017.  Curiosity-driven exploration by self-supervised prediction.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , 16–17.
  * Salge and Polani (2017) Salge, C.; and Polani, D. 2017.  Empowerment as replacement for the three laws of robotics.  _Frontiers in Robotics and AI_ 4: 25.
  * Santucci, Baldassarre, and Mirolli (2012) Santucci, V. G.; Baldassarre, G.; and Mirolli, M. 2012.  Intrinsic motivation mechanisms for competence acquisition.  In _2012 IEEE International Conference on Development and Learning and Epigenetic Robotics (ICDL)_ , 1–6. IEEE.
  * Strouse et al. (2018) Strouse, D.; Kleiman-Weiner, M.; Tenenbaum, J.; Botvinick, M.; and Schwab, D. J. 2018.  Learning to share and hide intentions using information regularization.  In _Advances in Neural Information Processing Systems_ , 10249–10259.
  * Tesauro (1994) Tesauro, G. 1994.  TD-Gammon, a self-teaching backgammon program, achieves master-level play.  _Neural computation_ 6(2): 215–219.
  * van der Heiden et al. (2020) van der Heiden, T.; Weiss, C.; Shankar, N. N.; Gavves, E.; and van Hoof, H. 2020\.  Social navigation with human empowerment driven reinforcement learning.  _arXiv preprint arXiv:2003.08158_ .


Tri-Bi-Maximal Mixing in Asymmetric Textures

Pierre Ramond

University of Florida

In the Honor of A. Balachandran’s eighty-fifth Birthday

##  1 Syracuse to Trieste

In honor of Professor Ayalam Balachandran who had the patience to teach me
theoretical physics.

When accepted to graduate school at Syracuse University, I wanted to study
General Relativity with Peter Bergmann. After an arduous first year in
transition from engineering to physics, I took a wonderful course from
Professor MacFarlane’s on Advanced Quantum Mechanics which turned me into a
budding Particle Physicist. George Sudarshan, my first adviser, asked me this
simple question: “Does the Spin-Statistic Theorem apply to infinite component
wave equations?”, and left shortly after for a sabbatical in India. This is
how I got acquainted with Majorana’s equation and Streater and Whightman’s
“PCT, Spin and Statistics, and All That”. Professor Sudarshan came back to
India and told me that Ivan Todorov had built a counter example. So much for
that! Left adrift, I was rescued by Professor Balachandran who, I found out,
lived in momentum space. He proposed a problem: given a four-point S-matrix
for massive particles, is it possible to continue into the Mandelstam Triangle
and emerge to a different channel. This was a concrete task, where I learned
of the Appel-Kampé de Feriet’s orthogonal polynomials in the triangle, but the
physics problem was much too difficult because of the cuts, and I was left
with an incomplete answer, much to my dismay. During this process, Bal
continued his encouragements, and when away, with helpful, lengthy and
detailed letters. With Bal, my fellow graduate student Bill Meggs, and Jean
Nuyts, I participated in three published papers. This was almost not enough:
after four years, I was on the job market, but was not picked until much later
when Bob Wilson decided at the last moment to initiate a theory group at the
newly approved National Accelerator Laboratory (NAL), later known as FermiLab.
I will be forever grateful to Bal for securing me a summer appointment at
Trieste’s International Center for Theoretical Physics that Professor Salam
had founded. There, I had the good fortune to meet my collaborator Jean Nuyts
and Hirotaka Sugawara, who introduced me to the amazing mathematical structure
behind the Veneziano model. Hirotaka and I worked hard to extract the triple
Reggeon vertex using tensor methods, only to find Sciuto’s elegant version
using ladder operators - thankfully we never published our version, but I was
hooked on Veneziano’s Dual Resonance model.

I am honored to contribute to this volume in Bal’s honor. Without him, I would
have gone back to engineering, and perhaps been wealthier, but surely not as
fulfilled nor as happy!

##  2 Asymmetric Texture

The presence of two large angles in the neutrino mixings is, I believe, the
most important yet most neglected puzzle of particle physics. Whenever Nature
displays a breakdown in previously accepted patterns, there lies the path to
further progress. Quarks and leptons, with similar gauge couplings, show
totally different mixings. Quarks follow small mixing angles while neutrinos
display only one small mixing angle and two large mixing angle whose values
approximate “geometric” values. This led to an inspired approximation[1] of
the lepton mixing matrix,

$\mathcal{U}_{PMNS}=\mathcal{U}^{(-1)}\mathcal{U}_{TBM},\qquad\mathcal{U}_{\rm
TBM}=\begin{pmatrix}\sqrt{\dfrac{2}{3}}&\dfrac{1}{\sqrt{3}}&0\\\\[8.00003pt]
-\dfrac{1}{\sqrt{6}}&\dfrac{1}{\sqrt{3}}&\dfrac{1}{\sqrt{2}}\\\\[10.00002pt]
\dfrac{1}{\sqrt{6}}&-\dfrac{1}{\sqrt{3}}&\dfrac{1}{\sqrt{2}}\end{pmatrix},$

where $\mathcal{U}^{(-1)}$ comes from the diagonalization of the charged
lepton Yukawa matrix. In the $SU_{5}$ Grand-Unified extension of the Standard
Model, $\mathcal{U}^{(-1)}$ should generate a “Cabibbo Haze” that fully
account for the third (reactor) angle. In $SU_{5}$, is summarized by the
elegant Georgi-Jarlskog[2] construction,

$\mathcal{U}^{(-1)}=\mathcal{U}_{\mathrm{CKM}}(c~{}\rightarrow~{}-3c),$ (1)

where $c$ is the (22) element of the down quarks Yukawa matrix, and the GUT-
Scale relations

$~{}m_{\tau}=m_{b},\,\,m_{\mu}=3m_{s},\,\,m_{e}=\frac{1}{3}m_{d},~{}~{}\longrightarrow~{}~{}\det
Y^{(-1/3)}=\det Y^{(-1)}$

Before the reactor angle had been measured, a plethora of models[3] satisfying
these relations appeared in the literature, with predictions for the reactor
angle around one-third the Cabibbo angle. When measured[4] to be twice this
value, the TBM approximation was (too) quickly forgotten.

As a SuperString person, I always look for mathematical beauty in physics,
following my hero Paul Dirac[5]; the TBM matrix is beautiful since it makes
one think of angles between crystal faces, finite group theory, etc … .

Closer examination reveals that these models assume that all Yukawa matrices
are symmetric, an assumption based not on theory but on convenience. The
Georgi-Jarlskog relation of Eq(1) no longer applies if the charged lepton and
down quarks Yukawa matrices are asymmetric.

The search for asymmetric Yukawa matrices that reproduce the same G-J
relations while violating Eq(1) was conducted[6] in the “Damn The Torpedoes”
mode, an expression often used by Feynman (after Farragut). It meant do the
calculation and do not worry - if difficulties arise, fix them later if
possible. The Yukawa matrices turn out to be fine-tuned and models will be
required to explain their fine-tunings.

Such matrices exist[6] but in a highly constrained form,

$Y^{(2/3)}=\begin{pmatrix}\lambda^{8}&0&0\cr 0&\lambda^{4}&0\cr
0&0&1\end{pmatrix},\,\,Y^{(-1/3)}=\begin{pmatrix}{bd\lambda^{4}}&a\lambda^{3}&{b\lambda^{3}}\cr
a\lambda^{3}&{c\lambda^{2}}&g\lambda^{2}\cr{d\lambda}&g\lambda^{2}&1\end{pmatrix},\,\,Y^{(-1)}=\begin{pmatrix}{bd\lambda^{4}}&a\lambda^{3}&{d\lambda}\cr
a\lambda^{3}&{-3c\lambda^{2}}&g\lambda^{2}\cr{b\lambda^{3}}&g\lambda^{2}&1\end{pmatrix}$

where $a=c=1/3,b=.306,g=0.811,d=2a/g$ are Wolfenstein-like parameters.

Our “Asymmetric Texture”, with TBM hypothesis yields,

$\theta_{12}=39.81^{\circ},\quad 6.16^{\circ}\text{ above
PDG};\qquad\theta_{23}=42.67^{\circ},\quad 2.90^{\circ}\text{below PDG},$

and $\theta_{13}$ overshoots its PDG value:

$\sin\theta_{13}=\sin\theta_{\rm reactor}=0.184,\quad\text{ compared
to}=\sin\theta^{PDG}_{\rm reactor}=0.145$

It is fortunate that an overshoot can be corrected by adding to TBM a CP-
violating phase

$\mathcal{U}_{\mathrm{TBM}}~{}\longrightarrow~{}U_{\mathrm{TBM}}(\delta)={\rm
Diag}(1,1,e^{i\delta})\,\mathcal{U}_{\mathrm{TBM}},$

with

$\sin|\theta_{\mathrm{reactor}}|~{}\rightarrow~{}\sin|\theta_{\mathrm{reactor}}(\delta)|=\dfrac{\lambda}{3\sqrt{2}}\left|1+\dfrac{2e^{i\delta}}{g}\right|<\sin|\theta_{\mathrm{reactor}}|.$

The phase is thus determined by setting $\theta_{\rm reactor}$ at its PDG
value, up to a sign,

$\delta_{CP}=\pm 1.32\pi,~{}~{}~{}{\rm
compared~{}with~{}global~{}fits:}~{}~{}~{}~{}\delta^{PDG}_{CP}=1.36\begin{matrix}\scriptstyle+0.20\cr\scriptstyle-0.16\end{matrix}\,\pi.$
It is satisfying that this one phase corrects both solar and atmospheric
mixing angles to their experimentally-consistent values,

$\theta_{\rm Atm}=44.9^{\circ}~{}(0.66^{\circ}~{}\text{below
PDG}),\quad\theta_{\rm Solar}=34.16^{\circ}~{}(0.51^{\circ}\text{above PDG}).$

##  3 Family Symmetry

As mentioned above the asymmetric texture was designed to satisfy the G-J
$SU_{5}$ relations, and the CKM matrix for quark mixings and the Gatto
relation. With a phased TBM it predicts the magnitude of the phase. It also
contains many fine-tuned patterns and values. To alleviate these problems, a
discrete family symmetry is required. The general features of this model[7]
are:

\- Standard Model Chiral Fields transform as $SU_{5}~{}{\rm
irreps}:~{}{\bf\bar{5}},\,{\bf 10},\,{\bf 1}$

\- No Tree Level Electroweak Yukawa Couplings. \- Vector-Like Fermion
Messengers coupling to SM fields and “Familons”, \- Familons: scalar SM
singlets with only family charges \- Non-Abelian Family Symmetry with CG
coefficients to single out asymmetry.

The Family Symmetry of choice is the Non-Abelian “Frobenius” discrete group,
the semi-direct product $\mathcal{Z}_{3}$, $T_{13}=(Z_{13}\rtimes Z_{3})$,
with two inequivalent triplet representations $Z_{13}\rtimes Z_{3}~{}~{}{\rm
Irreps}:~{}~{}{\bf 3_{1}},\,{\bf 3_{2}},\,{\bf
1^{\prime}},\,{\bf\bar{3}_{1}},\,{\bf\bar{3}_{2}},\,{\bf\bar{1}^{\prime}},\,{\bf
1}$ It arises from continuous $G_{2}$ via the simple group $PSL(2,13)$:
$Z_{13}\rtimes Z_{3}=T_{13}\subset PSL(2,13)\subset G_{2}$ \- The chiral
matter is assigned to different family triplets: $({\bf\bar{5}},{\bf
3_{1}})+({\bf 10},{\bf 3_{2}})$ \- Electroweak-breaking Higgs fields:
$H\sim({\bf\bar{5}},{\bf 1}),\,\,H^{\prime}\sim({\bf\overline{45}},{\bf 1})$\-
Vectorlike Messengers: $\Delta\sim({\bf 5},{\bf 3_{2}});\,\text{ explains
charged lepton (down-quarks) Yukawa with H}$ $\Sigma\sim({\bf 10},{\bf
3_{1}})~{}\text{ generates the G-J term with H'}$
$\Gamma\sim({\bf\overline{10}},{\bf
3_{2}}),\,\Theta\sim({\bf\overline{10}},{\bf\bar{3}_{1}})~{}\text{ generate
up-quarks Yukawas}$

\- $\text{ Familons: four}~{}\sim({\bf 1},{\bf 3_{2}}),\quad{\rm
two}~{}\sim({\bf 1},{\bf 3_{2}})$

\- $\text{ Familon vacuum values lie along simple directions}:$
$(1,0,0),(0,1,0),(0,0,1),(0,1,1,),(1,0,1)$

The effect of the model is to explain the fine-tunings in terms of simple
direction in the familons’ vacuum manifold.

The next step is to generate the Seesaw sector. To wit, as in the original
Seesaw, one upgrades from $SU_{5}$ to $SO_{10}$ where the right-handed
(sterile) neutrinos appear:

$SO_{10}\supset SU_{5}\times U(1):~{}~{}~{}~{}~{}{\bf 16}={\bf\bar{5}}+{\bf
10}+{\bf 1};~{}~{}{\bf 10}_{v}={\bf\bar{5}}+{\bf 5}.$ The chiral assignment
upgrades naturally to $SU_{5}\times T_{13}:~{}~{}({\bf 10},{\bf
3_{2}})+({\bf\bar{5}},{\bf 3_{1}})~{}~{}\longrightarrow~{}~{}SO_{10}\times
T_{13}:~{}~{}~{}({\bf 16},{\bf 3_{2}})+({\bf 10}_{v},{\bf 3_{1}})$ It
reproduces the original chiral fields once the conjugate quintet in the spinor
${\bf 16}$ couples to the quintet in the vector ${\bf 10}_{v}$ generate a
vector-like mass through a familon interaction, and three sterile neutrinos
$\overline{N}$ which transform as $({\bf 1},{\bf 3_{2}})$ under $SU_{5}\times
T_{13}$. Assume a dimension-four coupling of the sterile neutrinos to one
familon: $\overline{N}\,\overline{N}\varphi_{\mathcal{M}}$. $T_{13}$
invariance allows $\varphi_{\mathcal{M}}\sim({\bf 1},{\bf 3_{2}}).$ In the
vacuum direction, $<\varphi_{\mathcal{M}}>_{0}=M(1,-1,1)$, the Majorana mass
of the three sterile neutrinos is diagonalized by the TBM matrix:

$\mathcal{U}_{TBM}^{t}\frac{1}{\mathcal{M}}\mathcal{U}_{TBM}=\frac{1}{M}{\rm
Diag}(1,-\frac{1}{2},1)$

With $T_{13}$ a simple vacuum structure yields TBM diagonalization.

The Seesaw Mechanism[8] requires the diagonalization of

$\mathcal{S}=\mathcal{D}\,\frac{1}{\mathcal{M}}\,\mathcal{D}^{t},$

where $\mathcal{D}$ is the Dirac mass stemming from the coupling of
$\overline{F}\sim({\bf\bar{5}},{3_{1}})$ with $\overline{N}$:
$\overline{F}\,\overline{N}\varphi_{\mathcal{D}}$, with
$\varphi_{\mathcal{D}}\sim{\bf\bar{3}_{1}}.$

The simple familon vacuum: $<\varphi_{\mathcal{D}}>=a(1,-1,1)$, $\mathcal{S}$
is diagonalized by TBM,

${\mathcal{S}}=\frac{a^{2}}{M}{\mathcal{U}_{\rm TBM}}{\rm
Diag}(1,-\frac{1}{2},1){\mathcal{U}_{\rm TBM}^{t}},$

from which we extract the light neutrino masses: $m_{\nu_{3}}=2m_{\nu_{2}}$
implies normal ordering, but the degeneracy $m_{\nu_{1}}=m_{\nu_{3}}$ implies
no $(\nu_{1}-\nu_{3})$ oscillations, in obvious contradiction with experiment!

Could there be more sterile neutrinos? The sequence from
$SU_{5}~{}\rightarrow~{}SO_{10}$ continues to the exceptional group $E_{6}$ as
we showed long ago.

Fourth Sterile Neutrino: $SO_{10}\rightarrow E_{6}$ and the group theory
follows,

\- $E_{6}\supset SO_{10}\times U_{1}:~{}~{}~{}{\bf 27}={\bf 16}+{\bf 10}+{\bf
1}$ \- $T_{!3}$’s mother group $PSL(2,13)$ contains a complex septet: ${\bf
7}={\bf 3_{1}}+{\bf 3_{2}}+{\bf 1^{\prime}}$ A natural marriage of $E_{6}$ and
$PSL(2,13)$ ensues with chiral assignments

$SO_{10}\times T_{13}:~{}~{}({\bf 16},{\bf 3_{2}})+({\bf 10},{\bf
3_{1}})~{}\longrightarrow~{}E_{6}\times T_{13}:~{}~{}({\bf 16},{\bf
3_{2}})+({\bf 10},{\bf 3_{1}})+({\bf 1},{\bf 1^{\prime}})$

With a fourth sterile neutrino, the Seesaw matrix contains a new addition

$\mathcal{S}~{}\rightarrow~{}\mathcal{S}^{\prime}=\mathcal{S}+\mathcal{W}\mathcal{W}^{t}$,
where $\mathcal{W}\mathcal{W}^{t}$ has two zero eigenvalues. Of the three
choices, only one agrees with oscillations, with required vacuum value
$<\mathcal{W}^{t}>_{0}\propto(2,\,e^{i\pi},\,e^{i\delta}).$ Using
oscillations, this model predicts normal ordering of the light neutrino
masses:

$m_{\nu_{1}}=27.6~{}{\rm meV},~{}~{}m_{\nu_{2}}=28.9~{}{\rm
meV},~{}~{}m_{\nu_{3}}=57.8~{}{\rm meV}$

We found recently [9] a version of the asymmetric texture, with the smaller
family symmetry $T_{7}=\mathcal{Z}_{7}\rtimes\mathcal{Z}_{3}$. Its ancestry
originates in continuous $G_{2}\supset PSL(2,7)\supset T_{7}$. $PSL(2,7)$
contains a real septet irrep ${\bf 7}={\bf 3}+{\bf\bar{3}}+{\bf 1}$, which
suggests a similar mixture based on $E_{6}$ and $PSL(2,7)$: $E_{6}\times
T_{7}:~{}~{}~{}~{}~{}~{}({\bf 16},{\bf 3})+({\bf 10},{\bf\bar{3}})+({\bf
1},{\bf 1})$ These curious chiral assignments whose modular symmetries point
to $G_{2}$ may construct a path to coset manifolds in eleven dimensions.

##  4 Acknowledgements

I wish to thank Bal again for allowing me to spend my life chasing the fabrics
of our Universe. I thank Professors Pérez and Stuart, as well as Drs Rahat and
Xu for helpful discussions. This research was supported in part by the
Department of Energy under Grant No. DE-SC0010296.

## References

  * [1] P. F. Harrison, D. H. Perkins and W. G. Scott, Phys. Lett. B 530, 167 (2002) [hep-ph/0202074]; P. F. Harrison and W. G. Scott, Phys. Lett. B 535, 163 (2002) [hep-ph/0203209].
  * [2] Howard Georgi and C. Jarlskog, Phys.Lett.B 86 (1979) 297-300
  * [3] For a recent review, see F. Feruglio and A. Romanino, Rev.Mod.Phys. 93 (2021) 1, 015007
  * [4] Y. Abe et al. [DOUBLE-CHOOZ Collaboration], Phys. Rev. Lett. 108, 131801 (2012); F. P. An et al. [DAYA-BAY Collaboration], Phys. Rev. Lett. 108, 171803 (2012) J. K. Ahn et al. [RENO Collaboration], Phys. Rev. Lett. 108, 191802 (2012)
  * [5] P. A. M. Dirac, “The Relation between Mathematics and Physics”, lecture delivered on Presentation of the JAMES SCOTT prize, February 6, 1939.
  * [6] M.H. Rahat, P. R., B. Xu, Phys. Rev. D 98 055030 (2018)
  * [7] M.J. Pérez, M.H. Rahat, P. R., A.J. Stuart, B. Xu, Phys. Rev. D 100 075008 (2019) ; Phys. Rev. D 101 075018(2020) 075018
  * [8] P. Minkowski, Phys. Lett. B 67, 421 (1977); M. Gell-Mann, P. Ramond, and R. Slansky, in _Sanibel talk_ , retroprinted as hep-ph/9809459, and in _Supergravity_ , North-Holland, Amsterdam (1979), PRINT-80-0576, retroprinted as [arXiv:1306.4669 [hep-th]]; T. Yanagida, in _Proceedings of the Workshop on Unified Theory and Baryon Number of the Universe_ , KEK, Japan (1979)
  * [9] M.J. Pérez, M.H. Rahat, P. R., A.J. Stuart, B. Xu, in Preparation