# Learning Visual-Audio Representations for Voice-Controlled Robots

Peixin Chang, Shuijing Liu, and Katherine Driggs-Campbell P. Chang, S. Liu,
and K. Driggs-Campbell are with the Department of Electrical and Computer
Engineering at the University of Illinois at Urbana-Champaign. emails:
<EMAIL_ADDRESS>work is supported by Agriculture and
Food Research Initiative (AFRI) grant no. 2020-67021-32799/project accession
no.1024178 from the USDA National Institute of Food and Agriculture.

###### Abstract

Inspired by sensorimotor theory, we propose a novel pipeline for task-oriented
voice-controlled robots. Previous method relies on a large amount of labels as
well as task-specific reward functions. Not only can such an approach hardly
be improved after the deployment, but also has limited generalization across
robotic platforms and tasks. To address these problems, we learn a visual-
audio representation (VAR) that associates images and sound commands with
minimal supervision. Using this representation, we generate an intrinsic
reward function to learn robot policies with reinforcement learning, which
eliminates the laborious reward engineering process. We demonstrate our
approach on various robotic platforms, where the robots hear an audio command,
identify the associated target object, and perform precise control to fulfill
the sound command. We show that our method outperforms previous work across
various sound types and robotic tasks even with fewer amount of labels. We
successfully deploy the policy learned in a simulator to a real Kinova Gen3.
We also demonstrate that our VAR and the intrinsic reward function allows the
robot to improve itself using only a small amount of labeled data collected in
the real world.

##  I Introduction

As robots begin to enter people’s daily lives, there is an increasing need for
intuitive ways that allow non-experts to interact and communicate with robots.
Spoken language is a natural choice for communication, leading to a wealth of
research in task-oriented voice-controlled robots [1]. Conventional pipelines
for task-oriented voice-controlled robots consist of independent modules for
automatic speech recognition (ASR), natural language understanding (NLU),
symbol grounding, and motion planning, as shown in Fig. 2a. These pipelines
suffer from intermediate and cascading errors between different modules and
the difficulties in relating language to the physical entities it refers to
[2]. To address these challenges, several approaches attempt to jointly
optimize some of the independent modules [3, 4]. However, these approaches
only consider part of the pipeline and cannot solve voice-controlled tasks
alone.

The first end-to-end model for task-oriented voice-controlled robots and the
concept of Robot Sound Interpretation (RSI) is proposed in [5]. RSI considers
typical tasks for voice-controlled robots, where an audio command serves as a
goal, as illustrated in Fig. 1. After hearing a sound command, the robot must
identify the goal of the commander (intent), draw the correspondence between
the raw visual and audio inputs, and develop a policy to approach the
corresponding target to finish the task given limited sensing. The end-to-end
model in [5] directly maps sound commands and image observations to a sequence
of robot actions that fulfill the commands, as illustrated in Fig. 2b. The
model is first trained in a simulated environment using reinforcement learning
(RL) and auxiliary losses, and then the model is deployed to a real robot
without further training.

However, this model has its own weaknesses. First, the reward functions need
to be hand-tuned and are task-specific, which limits the generalization of the
method. Second, it is costly to improve the system after its deployment in a
new domain. Ideally, when an intelligent voice-controlled robot encounters
unseen scenarios, it should be able to continually improve its interpretation
of language and skills from non-experts in daily life. However, improving the
system requires prohibitive amount of labels to optimize auxiliary losses and
accurate measurements to construct reward functions, both of which are
difficult to obtain or not available from non-experts in the real world.

Figure 1: Illustration of our robotic tasks. The robots must approach a target
mentioned by a sound command to finish a task. Windowed image shows the robot
camera view. The green and red lines are for illustration purposes and is
invisible to the robot. Top left: The TurtleBot searches for then navigates to
the target object among four objects. Top right: The iTHOR agent navigates and
finishes household tasks. Lower left: The Kuka moves its gripper tip to the
target block among four identical blocks. Lower right: The deployment of a
learned policy to a physical Kinova Gen3 and the fine-tuning of the policy in
the real world.

Based on recent advancements in representation learning, we address these
weaknesses in [5] by proposing a two-stage pipeline for training RSI agents as
shown in Fig. 2c. At the first stage, we train a visual-audio representation
(VAR) whose training data can be collected easily and intuitively by non-
experts. At the second stage, we use the VAR to produce an intrinsic reward
function for RL training. The intrinsic reward function is general across
tasks or robotic platforms, which avoids the laborious reward engineering.
After the robot is deployed in a new domain where an accurate reward is hard
to obtain or not available, non-experts can easily fine-tune the VAR by
providing a reasonable amount of training data, which updates intrinsic reward
function and allows the robot to perform self-supervised skill improvement.

Figure 2: Illustration of different models for voice-controlled robots. (a) A
conventional pipeline for voice-controlled robots. Independent modules process
the inputs. End-to-end SLU combines the ASR and the NLU, while end-to-end
grounding combines the NLU and the grounding module. (b) The first RSI model
[5]. Auxiliary losses such as $\mathcal{L}_{I}$ and $\mathcal{L}_{S}$ are used
for classifying images and the intent in the speech, respectively. The model
is trained end-to-end with an RL loss, $\mathcal{L}_{RL}$. (c) The proposed
model.

Our method is inspired by sensorimotor contingency which is the relations
between an agent’s actions and the resulting sensory changes [6]. Imagine that
an infant touches a bell by accident and hears a bell ring. Motivated by her
“intrinsic curiosity,” she attempts to replicate the sound by repeating and
refining the touching action [7]. When she hears a bell ring later, she should
identify what actions are needed to ring the bell. We believe that a robot can
associate the motor actions with the corresponding audio command through a
similar learning process.

Our main contributions are as follows. (1) We present a novel pipeline for
task-oriented voice-controlled robots inspired by sensorimotor theory. (2) We
build a novel VAR that provides reward signals to the downstream RL training.
The VAR can be treated as a unification of the ASR, NLU, and the grounding
module. (3) Our system exhibits promising results in multiple navigation and
manipulation tasks. We demonstrate that our new pipeline enables the robot to
self-improve its policies with limited amount of data from a new domain using
both simulated and real-world experiments. (4) We create and solve the first
visually high-fidelity AI environment that uses real multi-word speech
commands for instruction following tasks.

##  II Related Works

###  II-A Conventional voice-controlled robots

As shown in Fig. 2a, a conventional pipeline of voice-controlled robots
usually consists of four independent modules: an ASR system to transcribe
speech to text [8], an NLU system to map text transcripts to speaker intent
[9, 2], a grounding module to associate the intent with physical entities [10,
11], and a planner to generate feasible trajectories for task execution [12,
13, 1]. However, the conventional pipeline is prone to intermediate errors.
Off-the-shelf ASR systems usually operate in a general-purpose setting
irrelevant to specific robotic tasks. Typical NLU systems, however, are
trained on clean text [3]. As a result, the text outputs could be inevitably
out-of-context or erroneous [14, 2], which causes the overall task performance
to degrade. Rule-based systems based on symbolic AI are used for grounding
[15]. However, developing such system requires significant labor and these
systems do not scale beyond their programmed domains [2, 4].

###  II-B End-to-end language understanding and grounding

Different from the conventional ASR plus NLU pipeline, the end-to-end spoken
language understanding (SLU) system maps the speech directly to the speaker’s
intent without translating the speech to text [3, 16, 17]. The end-to-end
learning avoids the intermediate errors and allows the models to fully exploit
subtle information such as speaker emotion that are lost during transcription
[16, 17]. Similar to a human, the end-to-end SLU system extracts high-level
concepts in speech instead of understanding the speech by recognizing word by
word [3]. However, end-to-end SLU systems are mainly used for virtual digital
assistants and do not consider the complex grounding process in robotics.

End-to-end language grounding agents have been developed to perform tasks such
as navigation [18, 19], object finding [4, 20], and household tasks [21]
according to text-based natural language instructions and visual observations.
Our work is different from them in several aspects. First, we focus on voice-
controlled agents, while the above works consider text-based input which
limits the potential of natural human-robot communication. Second, training
these agents require either experts’ actions at every timestep or step-by-step
instructions for imitation learning [18, 21] or a carefully designed extrinsic
reward function for reinforcement learning [19, 4, 20]. In contrast, our
method needs neither the expert demonstrations nor extrinsic reward functions,
and thus requires less efforts to improve.

Studying how an embodied agent learns and builds its own interpretation of
sound commands, RSI is the first work that combines the ideas from end-to-end
SLU and language grounding agents for voice-controlled robots, where the ASR,
the NLU, the grounding module, and the robot control are unified as an end-to-
end model [5]. However, the method in [5] has limited ability to improve in
different domains and generalize across different tasks and robots, as
mentioned in Sec. I. In contrast, our work can be improved in a new domain and
is general across tasks.

###  II-C Representation learning for robotics

Representation learning has shown great potential in learning useful
embeddings for downstream control tasks. Deep autoencoders are used to encode
high-dimensional observations such as images into lower-dimensional vector
embeddings. These vectors are then used as states or intrinsic reward
functions for reinforcement learning [22, 23]. At test time, however, the
methods in [22, 23] require users to provide goal images for task execution
while we use voice control, which is an more natural and convenient way of
human-robot communication. Additionally, reconstruction of the input images
makes the autoencoders computationally expensive. Another line of work uses
contrastive loss to learn representations for downstream tasks such as
grasping and water pouring [24, 25, 26]. One advantage of using contrastive
loss is that the loss avoids the reconstruction required by autoencoders.
Different from all of these works that focus mainly on the visual or the text
modality, we address the interplay between sight and sound.

Figure 3: The network architectures. (a) The VAR is a three-branch Siamese
network optimized with triplet loss. (b) The latent space of the VAR is a unit
hypersphere such that vector embeddings of images and audios of the same
object are closer than those of different objects. (c) The policy network for
RL training. The portion in blue is VAR and the weights are frozen in RL
training. We use $\bigoplus$ to denote element-wise addition, $FC$ to denote
fully connected layers, and $[\bullet\bullet]$ to denote concatenation.

##  III Methodology

In this section, we describe our two-stage training pipeline and the fine-
tuning procedure in a new domain. In training, we assume the availability of
sufficiently large datasets and robotic simulators that can be used by
professionals to develop and train the model. In fine-tuning, we assume that
datasets, simulators, and professionals are not available.

###  III-A Visual-audio representation learning

In the first stage of our pipeline, we learn a joint representation of images
and audio, named VAR, that associates the image with the corresponding sound
command. To mimic the sensorimotor learning experiences, the simulated
environments for training the VAR act like children’s parents who tell the
robot what it sees and does not see. For example, the environment provides
triplet data to the robot, which is similar to a mother saying, “This is a
cube, not a cone,” when her child is looking at a cube-shaped object. The
agent is thus told whether an image is semantically similar or dissimilar to
an audio, while the exact class of the image and the audio remains unknown to
the agent.

####  III-A1 Simulation environment and data collection

At each timestep, the environment produces a positive and a negative sound. If
an object is in the view of the robot camera in a navigation task, or when the
robot’s gripper tip aligns with an object in a manipulation task, the robot
receives a positive sound command corresponding to the object and a negative
sound command that does not match the object. We let the robot randomly move
in the environment and automatically collect visual-audio triplets
$(\mathbf{I},\mathbf{S^{+}},\mathbf{S^{-}})$, where
$\mathbf{I}\in\mathbb{R}^{n\times n}$ is the current RGB image from the
robot’s camera and serves as the anchor, $\mathbf{S^{+}}\in\mathbb{R}^{l\times
m}$ is the Mel Frequency Cepstral Coefficients (MFCC) [27] of the positive
sound commands, and $\mathbf{S^{-}}\in\mathbb{R}^{l\times m}$ is the MFCC of
the negative sound commands. We use $l$ to denote the number of frames in time
axis and $m$ to denote the number of features in frequency axis. If the agent
does not trigger a sound, the robot will hear no positive sound,
$\mathbf{S^{+}}=\mathbf{0}_{l\times m}$. More specifically, in a manipulation
task, the empty sound happens when its gripper tip is above none of the
objects; in a navigation task, the empty sound happens when the robot does not
see or is far away from all objects.

####  III-A2 Training the VAR

Our goal is to encode both auditory and visual modalities into a joint latent
space, where the embeddings of images and audios of the same intent are close
together, and those of different intents are far apart. We formulate the
problem as metric learning. As shown in Fig. 3a, the VAR is a Siamese network
that has three branches for encoding the anchor image $\mathbf{I}$, the
positive sound $\mathbf{S^{+}}$, and the negative sound $\mathbf{S^{-}}$. We
use $f_{S}:\mathbb{R}^{l\times m}\rightarrow\mathbb{R}^{d}$ to encode both
$\mathbf{S^{+}}$ and $\mathbf{S^{-}}$, where $d$ is the dimension of the
latent space, and we use $f_{I}:\mathbb{R}^{n\times
n}\rightarrow\mathbb{R}^{d}$ to encode $\mathbf{I}$. Then,
$\mathbf{v^{i}}=f_{I}(\mathbf{I})$, $\mathbf{v^{+}}=f_{S}(\mathbf{S^{+}})$ ,
and $\mathbf{v^{-}}=f_{S}(\mathbf{S^{-}})$ are the latent representation of
$\mathbf{I}$, $\mathbf{S^{+}}$, and $\mathbf{S^{-}}$, respectively. We enforce
the norm of $\mathbf{v^{i}}$, $\mathbf{v^{+}}$, and $\mathbf{v^{-}}$ to be $1$
by applying a $L2$-normalization, such that the latent space lives on a unit
hypersphere as shown in Fig. 3b. We use convolutional neural networks (CNN)
and bidirectional recurrent neural network (BRNN) to process the sound
signals, and we use CNN to process the images [28, 29, 30]. Practically, any
deep models for sound and image processing can be used for $f_{S}$ and
$f_{I}$. We use triplet loss as the objective, which encourages the distance
of vector embeddings between the anchor and the negative to be greater than
that between the anchor and the positive by a margin of $\alpha$ [31]. The
objective for optimizing $f_{I}$ and $f_{S}$ with a visual-audio triplet
$(\mathbf{I},\mathbf{S^{+}},\mathbf{S^{-}})$ is

$\mathcal{L}_{\textrm{VAR}}=\max(0,\lVert\mathbf{v^{i}}-\mathbf{v^{+}}\rVert^{2}_{2}-\lVert\mathbf{v^{i}}-\mathbf{v^{-}}\rVert^{2}_{2}+\alpha)$
(1)

###  III-B RL with visual-audio representation

The second stage of our pipeline is to use RL to train the robot to reach
goals using intrinsic reward functions generated by the trained VAR. We model
this interaction as a Markov Decision Process (MDP), defined by the tuple
$\langle\mathcal{X},\mathcal{A},P,R,\gamma\rangle$. At each time step $t$, the
agent receives a current observation $x_{t}\in\mathcal{X}$ and takes an action
$a_{t}\in\mathcal{A}$ based on its policy $\pi_{\theta}(a_{t}|x_{t})$
parameterized by $\theta$. In return, the agent receives an intrinsic reward
$r_{t}$ from VAR and transitions to the next state $x_{t+1}$ according to an
unknown state transition $P(\cdot|x_{t},a_{t})$. The process continues until
$t$ exceeds the maximum episode length $T$ and the next episode starts. The
goal of the agent is to maximize the expected return,
$R_{t}=\mathbb{E}[\sum^{T}_{i=t}\gamma^{i-t}r_{i}]$, where $\gamma$ is a
discount factor. The value function $V^{\pi}(x)$ is defined as the expected
return starting from $x$, and successively following policy $\pi$.

####  III-B1 VAR-aided RL

At each time step $t$, the agent receives an image $\mathbf{I_{t}}$ from its
camera, the MFCC of the current sound $\mathbf{S_{t}}$, and robot states
$\mathbf{M_{t}}$ such as end-effector location or odometry. The current sound
$\mathbf{S_{t}}$ is triggered in the same way as $\mathbf{S^{+}}$ in Sec.
III-A1. At $t=0$, the agent receives an additional one-time sound command
$\mathbf{S^{g}}$ corresponding to an intent. We define the state
$x_{t}\in\mathcal{X}$ for the MDP to be
$x_{t}=[\mathbf{I_{t}},f_{I}(\mathbf{I_{t}}),f_{S}(\mathbf{S^{g}}),\mathbf{M_{t}}]$,
where $f_{I}$ and $f_{S}$ come from the VAR which is fixed in this stage of
the pipeline. The VAR compactly represents the high-dimensional observations
and the intent contained in $\mathbf{S^{g}}$.

The VAR also generates intrinsic reward function which motivates the agent to
reach the target. Suppose the embeddings of an image and a sound command of
the same intent are close to each other in the VAR, the intrinsic reward
function is the sum of two dot products

$r_{t}=f_{I}(\mathbf{I_{t}})\cdot
f_{S}(\mathbf{S^{g}})+f_{S}(\mathbf{S_{t}})\cdot f_{S}(\mathbf{S^{g}})$ (2)

The first dot product $f_{I}(\mathbf{I_{t}})\cdot f_{S}(\mathbf{S^{g}})$
measures the similarity between the current image and the sound command, while
the second dot product $f_{S}(\mathbf{S_{t}})\cdot f_{S}(\mathbf{S^{g}})$ is
the similarity between the current sound and the sound command. Intuitively,
the agent receives high reward when both the scene it sees and the current
sound it hears match the sound command. Note that providing current sound
signals to the system is optional and is feasible in the simulation. The
current sound $\mathbf{S_{t}}$ is not part of the state $x_{t}$ and is only
used to calculate the reward. Thus, the robot policy is not conditioned on
$\mathbf{S_{t}}$ and does not require sound feedback at test time. Chang et al
use extrinsic reward functions that rely on task-specific or robot-specific
ground-truth information from the simulator [5]. In contrast, our intrinsic
reward function depends on the similarity of the embeddings and is thus
agnostic to tasks and robots. Combining our intrinsic reward function with an
extrinsic task-specific reward function is a design choice. Nevertheless, we
show in Sec. IV that our model outperforms the baselines even with only the
intrinsic reward.

####  III-B2 Network architecture

Our policy network architecture is in Fig. 3c. The network takes $x_{t}$ as
input and output the value $V(x_{t})$ and the policy $\pi(a_{t}|x_{t})$. The
reason that we need another CNN in the policy network is that $f_{I}$ and the
CNN may need to extract different features. For example, in a navigation task,
the CNN needs to encode information about obstacles for collision avoidance,
while $f_{I}$ does not need to. The embeddings of image $\mathbf{I_{t}}$ and
robot state $\mathbf{M_{t}}$ are fed into an LSTM for long-term decision
making. But the goal sound $\mathbf{S^{g}}$ is not passed into an LSTM because
it is an one-time signal. We use Proximal Policy Optimization (PPO) for policy
and value function learning [32].

###  III-C Fine-tuning

We assume that accurate speech transcriptions, one-hot labels, and tuned
reward functions are not available during fine-tuning because they cannot be
provided by non-experts. Since performing accurate measurements and data
collection in the real world is expensive and hard, only scarce and noisy data
from the non-experts are available. Our pipeline allows non-experts to
intuitively fine-tune the model whose performance degrades due to domain shift
after the deployment. Non-experts only need to provide triplets to the VAR as
in Sec. III-A based on their common knowledge using their own voices. The non-
experts do not need to know the total number of objects or the class index of
each object. The collected triplets are used to fine-tune the VAR. We use
$r_{t}=f_{I}(\mathbf{I_{t}})\cdot f_{S}(\mathbf{S^{g}})$ as reward function if
current sounds are not available. The agent can then perform self-supervised
skill improvement with the intrinsic reward function from VAR. No additional
labels or measurements other than the collected triplets are needed to support
its RL training. If a user is able to provide current sounds, we can use Eq. 2
to perform real-time voice-based supervision to the robot.

##  IV Simulation experiments

In this section, we first briefly describe the three robotic environments and
sound data for the experiments. Then, we compare the performance and data
efficiency of our pipeline with several baselines and ablation models.

###  IV-A Robotic Environments

We use three challenging simulation environments implemented using PyBullet
and AI2-THOR in Fig. 1 to show that our model is general in terms of robotic
platforms and tasks. The TurtleBot and the Kuka environments pose difficulties
in fine motor control and have moderate difficulty for perception. The iTHOR
environment is challenging in perception, while the control is discretized and
simplified. In all environments, the monocular uncalibrated cameras only
provide RGB images. No tags, depth, and models of the objects are given to the
agent. The illustrations of the environments can be found in the Supplementary
Video.

####  IV-A1 TurtleBot

The TurtleBot environment contains a $4\text{m}^{2}$ arena and four objects: a
cube, a sphere, a cone, and a cylinder. Each object is associated with an
intent. The robot’s goal is to navigate to the object mentioned in a command
based on RGB images. The agent needs to learn exploration skills to find the
target object as quickly as possible.

####  IV-A2 Kuka

The Kuka environment contains a table and four identical blocks. Each block is
associated with an intent. The robot’s goal is to move its gripper tip on the
top of the block mentioned in a command based on images from a location-fixed
camera. The agent needs to develop spatial reasoning skills to differentiate
the target block from other identical blocks using the observed relative
positional information.

####  IV-A3 iTHOR

Our iTHOR environment is the first AI environment with real multi-word speech
commands. The environment consists of 30 different floor plans of living rooms
and simulates real-world applications of household robots. The intents include
toggling on and toggling off the floor lamp and the television in a scene. The
goal of the robot is to navigate to and manipulate the object mentioned in a
sound command based on images and a noisy local occupancy map. The agent in
the iTHOR environment needs to associate complex speech commands with rich
visual observations and perform mapless navigation to finish the task.

###  IV-B Sound data processing

We consider various types of sounds including speech signals, environmental
sounds, single-tone signals, and a mixture among them. For single-word speech
signals, we choose “zero,” “one,” “two,” and “three” as Wordset1 from the
Speech Commands Dataset [33]. For multi-word speech signals, we choose the
speech with duration at most 6 seconds from the Fluent Speech Commands Dataset
(FSC) [16]. FSC dataset is collected for contemporary end-to-end SLU system
and contains commands used for a smart home or virtual assistant. The four
intents we choose are “activate light,” “deactivate light,” “activate music,”
“deactivate music.” For each intent, there are multiple possible wordings. For
example, the intent of “activate light” can be expressed as “turn on the
lights” and “lamp on.” For single-tone signals, we choose four musical notes,
$C_{4}$ (Middle C), $D_{4}$, $E_{4}$, and $F_{4}$, performed by various
musical instruments from NSynth datasets [34]. We mix the sound data from two
datasets to show that our model can map different types of sounds to one
object or concept. In the Mix dataset, single-word speech signals are mixed
with environmental sound coming from the UrbanSound8K Dataset [35]. For each
intent, 1,000 samples are used for training and 50 samples are used for
testing.

###  IV-C Experiment setup

We now introduce the training and evaluation of our pipeline in simulation. In
our experiments, the overall architectures are kept the same for solving all
the environments. Designing a specific network architecture for each
individual task is not the focus of our paper. The sound commands for the
iTHOR environment come from the FSC dataset, while the commands for the Kuka
and the TurtleBot environments come from the other datasets.

####  IV-C1 Training

For all tasks, we first let the robot randomly explore the environment and
collect 50,000 audio-visual triplets. The triplet margin $\alpha\in[1.0,1.2]$
and the latent space dimension $d$ is $3$. During RL training, the VAR is
fixed and $8$ instances of the environment run in parallel.

####  IV-C2 Evaluation criteria

We test the trained policy for $50$ episodes for each intent ($200$ episodes
in total) with unheard sound data. For the Kuka environment, an episode is
considered a success if the robot’s gripper stays close to the target block
for more than $50$ timesteps. For the TurtleBot environment, we define an
episode as a success if the robot stays close to the target object for more
than $5$ timesteps. For the iTHOR environment, we define an episode as a
success if the robot correctly toggles the floor lamp or television in a
scene.

We evaluate the model with two metrics. The first metric is success rate,
defined as the percentage of successful episodes in all test episodes. The
second metric is label usage which includes one-hot labels and queries to the
simulator. For example, measuring the distance between the robot and the goal
is a query. One triplet requires two labels to indicate the positive and the
negative.

####  IV-C3 Baselines and Ablations

We compare our method with three baselines and one ablation. The first
baseline, denoted as “RSI 1,” is the method in [5] which has the access to
hand-tuned task-specific reward functions and ground-truth class labels for
the images and sounds. Every RSI 1 training step requires 3 labels, including
2 simulator queries to calculate the extrinsic reward and 1 one-hot label for
auxiliary losses111The queries include target object state checking and
distance measuring. For convenience purposes, we count the one-hot labels for
all the auxiliary losses as one for RSI 1, which is lower than the real label
counts..

The second baseline, denoted as “NoAux+SparseR,” is a variant of RSI 1. We
collect 50,000 one-hot labels to pretrain the network. During the RL training,
we remove all the auxiliary losses, and we give the agent a reward of $1$ if a
task is completed and $0$ otherwise at each timestep. This baseline shows the
performance of “RSI 1” without ground-truth class labels and hand-tuned reward
functions.

The third baseline, denoted as “ASR+NLU+RSI 1,” is a conventional
pipeline222This pipeline is not used in the Kuka and the TurtleBot
environment, because the sound commands in these environments involve non-
speech content or only single words.. The speech is first transcribed by an
off-the-shelf deep learning-based ASR named Mozilla DeepSpeech [30]. The
output of the ASR is noisy. For example, “Play the music” is sometimes
transcribed as “by the music.” We therefore train a learning-based NLU so that
the correct intent can still be inferred given noisy inputs. The intent is
then transformed into an one-hot encoding, which is fed to an RL agent whose
architecture is the same as the RSI 1 except that sound processing network is
removed.

The ablation model, denoted as “No current sound,” removes
$f_{S}(\mathbf{S_{t}})\cdot f_{S}(\mathbf{S^{g}})$ from the intrinsic reward
function, Eq. 2. The purpose of the ablation model is to quantify the effect
of the current sound signal.

###  IV-D Results for control policies

####  IV-D1 Unheard sounds

In this experiment, all models are first trained with the same amount of RL
steps without fine-tuning in each environment. Then, we test the models with
sound commands that never appear during training. We use this experiment to
compare the performance of the models when the amount of data and the training
steps are sufficient. For the iTHOR environment, the total training steps is 6
million (M), and the agent is tested within the training floor plans (Floor
Plan 201 - 220). The total training steps is 3M for both the Kuka and the
TurtleBot environments.

From Table I and Table II, we find that our method generalizes well in terms
of different types of sound commands and robotic tasks. Our method achieves
$1\%\sim 5\%$ higher success rates compared with RSI 1 in all three
environments, even though it has no access to ground-truth class labels or
extrinsic reward functions during the whole RL training. We believe the reason
is that RSI 1 optimizes multiple losses jointly, which may cause biased
gradients. In contrast, our two-stage pipeline learns the two tasks
separately, which benefits the training of both tasks. A common failure case
for the iTHOR agent is that the agent is blocked by obstacles during its move
to the correct target.

From Table I and Table II, the success rates for “NoAux+SparseR” baseline for
all the environments are much lower than RSI 1 and our method. The result
suggests that the performance of RSI 1 degrades significantly without
unlimited ground-truth labels for its auxiliary losses and heavily hand-
engineered reward function. In contrast, our method can perform better even
with weaker supervision and fewer engineering efforts and is thus more
general.

From Table II, The success rate for “ASR+NLU+RSI 1” baseline is lower than
both the RSI 1 and our method. As mentioned in Sec. II-A, the pipeline is
prone to intermediate errors and thus has lower success rate. This result
justifies the usage of end-to-end SLU in RSI 1 and our work.

The success rate for “No current sound” ablation is only $9.5\%$ lower than
our method, but the label usage of this ablation is significantly lower than
all other models. The result indicates that our method works reasonably well
without the current sound signals that can be hard to obtain in the real world
and providing current sound signals further increases the performance.

TABLE I: Testing results in the Kuka and the TurtleBot environments with different types of sounds. Environment | Dataset | Success rate
---|---|---
NoAux+SparseR | RSI 1 | Ours
Kuka | Wordset1 | 46.0 | 95.5 | 97.0
NSynth | 28.5 | 92.5 | 98.0
Mix | 23.0 | 94.0 | 95.5
TurtleBot | Wordset1 | 22.0 | 92.0 | 94.0
NSynth | 9.5 | 95.0 | 96.0
Mix | 21.0 | 87.0 | 91.0
TABLE II: Testing results in the iTHOR 201-220 Dataset | Models | Label usage | Success rate
---|---|---|---
FSC | NoAux+SparseR | $6.0\times 10^{6}$ | 26.0
No current sound | $\mathbf{0.1\times 10^{6}}$ | 61.5
ASR+NLU+RSI 1 | $18.0\times 10^{6}$ | 66.0
RSI 1 | $18.0\times 10^{6}$ | 68.0
Ours | $6.1\times 10^{6}$ | 71.0

####  IV-D2 Unseen scenes

We test the performance of our method and the RSI 1 with unheard sound
commands in unseen floor plans in the iTHOR environment. Different from the
experiment in Sec. IV-D1, where the models are trained without constraints on
label usages, this experiment simulates the fine-tuning in the real world and
quantifies the label efficiency of the models. Each model is given the same
amount of labels, and a high-efficiency model should achieve higher success
rate compared to a low-efficiency model. We use “0” to denote the direct
deployment of the models without further training in any unseen floor plans.
We use “5000” to denote the settings where a total amount of 5000 labels are
collected and are used for the fine-tuning. We follow the fine-tuning
procedure mentioned in Sec. III-C without providing the current sound signals.
We train another RSI 1 model for 1M steps, which uses about 3M labels as an
oracle model to approximate the highest possible success rates. The results
are shown in Table III.

TABLE III: Success rates of different methods after fine-tuning in unseen Floor Plan 226 - 230 with FSC dataset. Labels | 0 | 5000 | 3M
---|---|---|---
Methods | RSI 1 | Ours | RSI 1 | Ours | Ours | Ours | RSI 1
0.08M | 0.4M | 1M
Avg. | 30.7 | 31.5 | 37.2 | 67.5 | 73.7 | 84.7 | 90.8

From Table III, we fine that the RSI 1 baseline can only be improved by
$6.5\%$ using 5000 labels, while our method improves itself by $36\%$, $42\%$,
and $53\%$ after $0.08$M, $0.4$M, and $1$M RL training steps, respectively,
using the same amount of labels. We also see that the average gap between Ours
1M and the oracle is only $6\%$, suggesting that our model achieves high
performance using limited amount of labels. In the fine-tuning stage, RSI 1
uses the labels for policy network fine-tuning, which leads to fast depletion
of the label quotas and less RL experience, which suggests that fine-tuning
the RSI 1 is unrealistic. The richer RL experience in our method is due to its
higher efficiency because labels are used to predict the intrinsic reward and
there is no label consumption during its self-supervised RL exploration. We
believe that this result is satisfying given that the current sound is not
provided and that the amount of label usage is 600 times less than that of the
oracle model.

TABLE IV: Model performance and the number of triplets collected for the fine tuning Floor Plan | RL Training Steps |  | Number of Triplets
---|---|---|---
|  $<$500 | 1000 | 2500
226 | 0.08M |  | 59.0 | 79.0 | 83.5
0.4M |  | 75.0 | 84.5 | 88.0
1M |  | 86.5 | 91.0 | 92.5
230 | 0.08M |  | 49.0 | 45.0 | 45.0
0.4M |  | 62.0 | 63.0 | 57.0
1M |  | 77.0 | 80.5 | 86.5

####  IV-D3 Performance and the number of triplets

We explore the relation between the model performance during the fine-tuning
and the number of triplets in two randomly picked floor plans. From Table IV,
we see that the more triplets are provided, the better the final performance
becomes. This observation is reasonable since more training data leads to
better VAR and reward functions. We find that the performance is reasonably
well even if the amount of triplets provided are less than 500, which suggests
that the VAR has high data efficiency.

Figure 4: Qualitative Result in the simulation and the real world. The target
objects are indicated by the red arrows. Green and red lines are invisible to
the robots. Top: Simulated TurtleBot searches and approaches its target, the
purple cylinder. Middle: iTHOR agent searches for the television and turn it
off. Bottom: The real Kinova Gen3 moves its gripper to the target block and
performs a successful grasp. Figure 5: Visualizations of the VAR in the Kuka
environments with Wordset1. The colors indicate the ground truth labels of
sound and image data. In the sphere, the embeddings of images are marked by
circles and sounds are marked by triangles. The black stars are the vector
embeddings of the 8 images. Block 1 is the leftmost block. The “Empty” class
consists of empty sounds and images of the gripper tip above none of the
blocks.

####  IV-D4 Qualitative results

Fig. 4 and the attached video show examples of the agent successfully reaching
their goals.

###  IV-E Qualitative results of the VAR

To better understand the VAR and how it produces an effective intrinsic
reward, we plot the vector embeddings of some positive pairs from the audio-
visual triplets with the trained VAR in Fig. 5. In the VARs of all three
environments, the embeddings of images and sounds of the same concept form a
cluster and all clusters are separated from each other. Thus, the agents
successfully associate images and sounds with the same meanings and
distinguish those with different meanings, even though we do not explicitly
classify the objects or inform the agent of the number of classes.

We also find that the VAR is able to map the images with ambiguous labels to
meaningful locations on the spheres. For example, in Fig. 5, the gripper tip
in #2 is not directly above but is very close to Block 1. In the VAR sphere,
the vector embedding is indeed between the Block 1 (red) and the Empty (cyan)
cluster. With the ability to represent ambiguous images, the VAR generates a
relatively dense reward function that guides the robot exploration in RL. In
the same example, suppose Block 1 is the target block, the reward in Eq. 2 are
higher in #2 than those in #4 or #5. Thus, the reward is higher when the
gripper is around the target block and lower when the gripper is far away from
the target.

##  V Real-world experiments

To show the effectiveness of our method on real physical robots and its
potential to improve itself in the real world, we use a Kinova Gen3 arm for
the Kuka choosing task, train the robot in the simulator and fine-tune the
model in the real world. We assign the four blocks from left to right with the
words “zero,” “one,” “two,” and “three.” At every timestep, the trained agent
receives an image from an uncalibrated location-fixed RGB camera and its end-
effector location in the workspace. All sound commands for testing are unheard
during training. The trained model controls the robot gripper to move towards
the target block and adjusts its pre-grasp pose. At the end of an episode, the
robot lowers its gripper vertically and perform a grasp on the chosen block.

We use domain randomization to learn robust features in both stages of our
pipeline. We randomize the color and texture of the blocks and the background
during training. We also randomize the camera viewpoint and the relative
positions among the blocks.

Results with domain randomization: We first deploy the model trained with
domain randomization to a real Kinova Gen3 without real-world fine-tuning and
count the number of successful grasps. In the first row of Table V, the
results indicate that the simulation to real world transfer is successful in
some cases. We believe the failures are caused by the difference in camera
pose between the simulation and the real world. The result suggests that the
domain randomization only helps bridge the sim2real gap to some extent, and
thus this technique alone can be insufficient for real-world learning-based
robotic applications.

TABLE V: Success counts for each intent and average success rate with a total of 48 tests (12 for each intent) Words | “zero” | “one” | “two” | “three” | avg.
---|---|---|---|---|---
Domain randomization | 8 | 11 | 9 | 7 | 72.9
Fine-tuning | 11 | 9 | 11 | 12 | 89.6

Results with fine-tuning: We show that our method can self-improve in the real
world by fine-tuning the model with real world data. Specifically, in the real
world environment, we spend about 1 hour in manually collecting 993 triplets,
which are used to fine-tune the VAR and the RL policy. No additional labels or
supervision are needed from the humans since the intrinsic rewards are
calculated by VAR. We train the model with randomized block positions for
10000 steps, which costs about 1 hour. In Table V, the average performance
after fine-tuning increases $16.7\%$ compared with domain randomization, which
is significant given the relatively small number of real-world data and
training time. In the failure cases, the agent is able to move towards the
correct block, but only fails at grasping because the block is not completely
within its jaw. A typical successful episode is shown in Fig. 4.

##  VI Conclusion and Future work

We propose a general and self-improvable pipeline for vision-based voice-
controlled robots. In our pipeline, we first learn a VAR, which associates
images and sounds of the same concept. Then, we use the representation to
generate an intrinsic reward function and embeddings for the downstream
control tasks. Our method requires fewer labels yet outperforms baselines that
has access to extrinsic reward functions across various types of sounds and
robotic tasks in simulation. We also successfully transfer the policy to a
real robot and improve the policy with small amounts of data in the real
world. Possible directions for the future work include: (1) generalizing the
same VAR to different downstream tasks, and (2) learning the representation in
a self-supervised way without labeled triplets.

## References

  * [1] R. Liu and X. Zhang, “A review of methodologies for natural-language-facilitated human–robot cooperation,” _International Journal of Advanced Robotic Systems_ , vol. 16, no. 3, 2019.
  * [2] Y. Tada, Y. Hagiwara, H. Tanaka, and T. Taniguchi, “Robust understanding of robot-directed speech commands using sequence to sequence with noise injection,” _Frontiers in Robotics and AI_ , vol. 6, p. 144, 2020.
  * [3] D. Serdyuk, Y. Wang, C. Fuegen, A. Kumar, B. Liu, and Y. Bengio, “Towards end-to-end spoken language understanding,” in _International Conference on Acoustics, Speech, and Signal Processing (ICASSP)_. IEEE, 2018, pp. 5754–5758.
  * [4] K. M. Hermann, F. Hill, S. Green, F. Wang, R. Faulkner, H. Soyer, D. Szepesvari, W. M. Czarnecki, M. Jaderberg, D. Teplyashin, _et al._ , “Grounded language learning in a simulated 3d world,” _arXiv preprint arXiv:1706.06551_ , 2017.
  * [5] P. Chang, S. Liu, H. Chen, and K. Driggs-Campbell, “Robot sound interpretation: Combining sight and sound in learning-based control,” in _IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2020, pp. 5580–5587.
  * [6] J. K. O’Regan, _Why red doesn’t sound like a bell: Understanding the feel of consciousness_. Oxford University Press, 2011.
  * [7] L. Jacquey, G. Baldassarre, V. G. Santucci, and J. K. O’regan, “Sensorimotor contingencies as a key drive of development: from babies to robots,” _Frontiers in neurorobotics_ , vol. 13, p. 98, 2019.
  * [8] R. A. S. Fernandez, J. L. Sanchez-Lopez, C. Sampedro, H. Bavle, M. Molina, and P. Campoy, “Natural user interfaces for human-drone multi-modal interaction,” in _IEEE International Conference on Unmanned Aircraft Systems (ICUAS)_ , 2016, pp. 1013–1022.
  * [9] E. Bastianelli, D. Croce, A. Vanzo, R. Basili, and D. Nardi, “A discriminative approach to grounded spoken language understanding in interactive robotics.” in _International Joint Conference on Artificial Intelligence (IJCAI)_ , 2016, pp. 2747–2753.
  * [10] R. Paul, J. Arkin, D. Aksaray, N. Roy, and T. M. Howard, “Efficient grounding of abstract spatial concepts for natural language interaction with robot platforms,” _The International Journal of Robotics Research_ , vol. 37, no. 10, pp. 1269–1299, 2018.
  * [11] A. Magassouba, K. Sugiura, A. T. Quoc, and H. Kawai, “Understanding natural language instructions for fetching daily objects using gan-based multimodal target–source classification,” _IEEE Robotics and Automation Letters_ , vol. 4, no. 4, pp. 3884–3891, 2019.
  * [12] F. Stramandinoli, V. Tikhanoff, U. Pattacini, and F. Nori, “Grounding speech utterances in robotics affordances: An embodied statistical language model,” in _Joint IEEE International Conference on Development and Learning and Epigenetic Robotics (ICDL-EpiRob)_ , 2016, pp. 79–86.
  * [13] H. Chen, H. Tan, A. Kuntz, M. Bansal, and R. Alterovitz, “Enabling robots to understand incomplete natural language instructions using commonsense reasoning,” in _IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2020, pp. 1963–1969.
  * [14] A. Vanzo, D. Croce, E. Bastianelli, R. Basili, and D. Nardi, “Robust spoken language understanding for house service robots,” _Polibits_ , no. 54, pp. 11–16, 2016.
  * [15] J. Savage, D. A. Rosenblueth, M. Matamoros, M. Negrete, L. Contreras, J. Cruz, R. Martell, H. Estrada, and H. Okada, “Semantic reasoning in service robots using expert systems,” _Robotics and Autonomous Systems_ , vol. 114, pp. 77–92, 2019.
  * [16] L. Lugosch, M. Ravanelli, P. Ignoto, V. S. Tomar, and Y. Bengio, “Speech model pre-training for end-to-end spoken language understanding,” in _Annual Conference of the International Speech Communication Association (INTERSPEECH)_ , 2019.
  * [17] M. Kim, G. Kim, S.-W. Lee, and J.-W. Ha, “St-bert: Cross-modal language model pre-training for end-to-end spoken language understanding,” in _International Conference on Acoustics, Speech, and Signal Processing (ICASSP)_. IEEE, 2021, pp. 7478–7482.
  * [18] P. Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. Sünderhauf, I. Reid, S. Gould, and A. van den Hengel, “Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments,” in _IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018, pp. 3674–3683.
  * [19] H. Yu, H. Zhang, and W. Xu, “Interactive grounded language acquisition and generalization in a 2d world,” in _International Conference on Learning Representations (ICLR)_ , 2018.
  * [20] D. S. Chaplot, K. M. Sathyendra, R. K. Pasumarthi, D. Rajagopal, and R. Salakhutdinov, “Gated-attention architectures for task-oriented language grounding,” in _Conference on Artificial Intelligence (AAAI)_ , 2018, pp. 2819–2826.
  * [21] M. Shridhar, J. Thomason, D. Gordon, Y. Bisk, W. Han, R. Mottaghi, L. Zettlemoyer, and D. Fox, “Alfred: A benchmark for interpreting grounded instructions for everyday tasks,” in _IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2020.
  * [22] A. Nair, V. Pong, M. Dalal, S. Bahl, S. Lin, and S. Levine, “Visual reinforcement learning with imagined goals,” in _Advances in Neural Information Processing Systems (NeurIPS)_ , vol. 31, 2018.
  * [23] Y. Wang, G. N. Narasimhan, X. Lin, B. Okorn, and D. Held, “Roll: Visual self-supervised reinforcement learning with object reasoning,” in _Conference on Robot Learning (CoRL)_ , 2020.
  * [24] P. Sermanet, C. Lynch, Y. Chebotar, J. Hsu, E. Jang, S. Schaal, S. Levine, and G. Brain, “Time-contrastive networks: Self-supervised learning from video,” in _IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2018, pp. 1134–1141.
  * [25] E. Jang, C. Devin, V. Vanhoucke, and S. Levine, “Grasp2vec: Learning object representations from self-supervised grasping,” in _Conference on Robot Learning (CoRL)_ , 2018.
  * [26] T. Nguyen, N. Gopalan, R. Patel, M. Corsaro, E. Pavlick, and S. Tellex, “Robot Object Retrieval with Contextual Natural Language Queries,” in _Proceedings of Robotics: Science and Systems_ , 2020.
  * [27] S. Davis and P. Mermelstein, “Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences,” _IEEE transactions on acoustics, speech, and signal processing_ , vol. 28, no. 4, pp. 357–366, 1980.
  * [28] D. Harwath, A. Torralba, and J. R. Glass, “Unsupervised learning of spoken language with visual context,” in _Advances in Neural Information Processing Systems (NeurIPS)_ , 2017.
  * [29] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in _International Conference on Learning Representations (ICLR)_ , 2015.
  * [30] A. Hannun, C. Case, J. Casper, B. Catanzaro, G. Diamos, E. Elsen, R. Prenger, S. Satheesh, S. Sengupta, A. Coates, _et al._ , “Deep speech: Scaling up end-to-end speech recognition,” _arXiv preprint arXiv:1412.5567_ , 2014.
  * [31] F. Schroff, D. Kalenichenko, and J. Philbin, “Facenet: A unified embedding for face recognition and clustering,” in _IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2015, pp. 815–823.
  * [32] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” _arXiv preprint arXiv:1707.06347_ , 2017\.
  * [33] P. Warden, “Speech commands: A dataset for limited-vocabulary speech recognition,” _arXiv preprint arXiv:1804.03209_ , 2018.
  * [34] J. Engel, C. Resnick, A. Roberts, S. Dieleman, D. Eck, K. Simonyan, and M. Norouzi, “Neural audio synthesis of musical notes with wavenet autoencoders,” in _International Conference on Machine Learning (ICML)_ , 2017, pp. 1068–1077.
  * [35] J. Salamon, C. Jacoby, and J. P. Bello, “A dataset and taxonomy for urban sound research,” in _International Conference on Multimedia (ACM-MM)_ , Orlando, FL, USA, Nov. 2014, pp. 1041–1044.


[CPT] [CPT] CPT]Center for Theoretical Physics, Polish Academy of Sciences,
Al. Lotnikow 32/46, 02-668 Warsaw, Poland

# ACCRETION OF THE MAGNETIZED NEUTRINO-COOLED TORUS

Fatemeh Hossein Nouri  Agnieszka Janiuk  [

###### Abstract

Neutrino-cooled accretion flow around a black hole, produced by a compact
binary merger, is a promising scenario for jet formation and magnetic-driven
winds to explain short duration gamma ray bursts (GRBs) central engine and
kilonovae based on GW170817 gravitational wave observation. Magnetorotational
instability (MRI) turbulence and Blandford-Znajek (BZ) mechanism are expected
to play key roles in the thermal equilibrium of the disk (balancing neutrino
cooling) and in driving accretion and creating jets. Using the open-source
GRMHD HARM-COOL code, we study the magnetically-driven evolution of an
accretion disk with realistic equation of state in the fixed curved space-time
background. We identify the effects of the neutrino cooling and the magnetic
field, paying particular attention to the dynamical, thermal and composition
evolution of the disk and outflows.

##  1 Introduction

Given the hot, dense accretion flow in a post-merger black hole-accretion disk
remnants, energy can be channeled into ultra-relativistic outflow needed to
explain GRB properties, Nakar (2007); Berger (2014). In such systems, the
accretion gas cools by neutrino emission continuously, Popham et al. (1999);
Di Matteo et al. (2002); Janiuk et al. (2004). On the other hand, magnetic
fields can also extract energy from the disk and black hole spin, by
generating the magnetic driven winds and Poynting flux-dominated jets,
Blandford & Znajek (1977). Moreover, it can support the heating process of the
plasma through the viscous effects of the MRI, Balbus & Hawley (1991). The
dynamical ejecta outflows during the merger, and the post-merger outflows are
considered as possible sources explaining the observation of kilonova and
other electromagnetic (EM) counterparts following the gravitational wave
detections, Metzger & Berger (2012); Dietrich & Ujevic (2017). In this series
of studies, we utilize the numerical simulations to investigate the evolution
and different properties of the disk and the outflows in the presence of
magnetic fields and neutrino cooling.

##  2 Numerical Methods and Initial Setups

We evolve two analytical Fishbone-Moncrief disk models, Fishbone & Moncrief
(1976), one with magnetic field and the other without magnetic field using the
GRMHD HARM-COOL code, Sapountzis & Janiuk (2019). The metric is Kerr-Schild
and frozen in time, and the hydro equations are evolved in conservative
formalism using HLL shock capturing scheme. In this setup the black hole mass
is set to $3M_{\odot}$, with spin $a=0.98$, and the disk size is set to have
the total mass of $M_{disk}=0.1M_{\odot}$. For the magnetized case, the
initial magnetic field is confined within the torus by poloidal loops with the
ratio of the maximum gas pressure to the maximum magnetic pressure
$\beta=P_{gas,max}/P_{B,max}=50$.

The equation of state is a realistic nuclear matter composed of free protons,
neutrons, electron–positron pairs, and helium nuclei, Reddy et al. (1998),
considering weak interactions to determine neutrino opacities in hot dense
matter. For neutrino treatment we use the formalism from Janiuk et al. (2013);
Janiuk (2019), where the neutrino cooling rate is given by

$Q_{\nu}^{-}=\frac{\frac{7}{8}\sigma
T^{4}}{\frac{3}{4}}\sum_{i=e,\mu}\frac{1}{\frac{\tau_{a,\nu_{i}}+\tau_{s}}{2}+\frac{1}{\sqrt{3}}+\frac{1}{3\tau_{a,\nu_{i}}}}\times\frac{1}{H}~{}[erg~{}s^{-1}~{}cm^{-3}],$
(1)

while the neutrino optical depth for different species are approximated as, Di
Matteo et al. (2002):

$\tau_{a,\nu_{i}}=\frac{H}{4\frac{7}{8}\sigma T^{4}}q_{a,\nu_{i}},$ (2)

where $q_{a,\nu_{i}}$ is the absorption rate for different neutrino species.

These simulations were performed on the Prometheus cluster with 24 processors
within a 2D grid with a resolution of 264*256 points on $r$ and $\theta$ in
spherical coordinates. These models were evolved for $t=90$ms.

##  3 Results

The results of the magnetized simulation are presented in Figs. 2-4 at
$t=0.03$s time snapshot. The 2D profiles show that disk becomes turbulent and
expands from the outer boundary. There is a significant accretion happening as
a result of MRI, and the disk is quite transparent for neutrino emissions. The
funnel shape jet structure with high magnetic field is formed at the early
time of the evolution. Measuring the volume density-averaged electron fraction
shows that the disk becomes less neutron-rich and less degenerate at the end
of the simulation.

The comparison between the magnetized and non-magnetized cases are given in
Figs. 6 and 6. As one expects, MRI provides the required mechanism for the
angular momentum transport and matter accreting into the black hole. On the
other hand, we observe that the neutrino luminosity increases by factor of
four for the magnetized case during the evolution. This can be explained as a
thermal competition between magnetic-driven viscous heating effect and the
neutrino and advection cooling effects, which shifts the equilibrium condition
for the weak interactions and generates more neutrinos.

For the magnetized case, we mark the fluid with positive energy $-hu_{t}>1$ as
unbound matter (here $h$ is the enthalpy and $u_{t}$ is the $t$ component of
the 4-velocity). Taking summation over density of the unbound matter for the
entire grid, we plot the histograms to measure the total mass over the
electron fraction $Y_{e}$, temperature and velocity bins. These analysis are
done at two time intervals, by taking the average over several snapshots from
the first half of the evolution, and the same for the second half. These
histograms and the mass of the unbound matter verses time are illustrated in
Figs. 8-10. Overall, the outflows become colder with lower kinetic energy at
the end of the simulation. This observation can be explained by the fact that
the magnetic field dissipates over time in a two dimensional evolution as a
result of the anti-dynamo effect. Hence, the generation of the magnetic-driven
winds slows down by further evolution.

##  4 Summary and Conclusions

We evolved two analytical disk-black hole models with and without magnetic
field. We observe the creation of the jets and turbulent plasma as a result of
BZ mechanism and MRI. The neutrino cooling has a significant effect on the
energy budget of the disk and its thermal and composition evolution. Measuring
the unbounded mass, and its features such as temperature, velocity and
composition, shows that our model does not produce enough energetic outflows
(yet) to compete with the dynamical ejecta to explain the source of the EM
counterparts (see Dietrich & Ujevic (2017) for the outflow estimates for black
hole-neutron star and binary neutron star merger simulations). We plan to
perform more simulations with different setup such as different spin and
different mass ratios for disks and black holes to investigate their effects
on the outflows. 3D simulations and more accurate approximation for neutrino
treatment such as neutrino leakage scheme are required for future studies to
estimate the outflows’ properties more precisely.

###### Acknowledgements.

This research was supported in part by grant No. 2019/35/B/ST9/04000 from the
Polish National Science Center.

Figure 1: The 2D profile of density and the contours of the optical depth of
the magnetized disk with neutrino cooling at $t=0.03$s snapshot.

Figure 2: The 2D profile of the ratio of gas pressure to magnetic pressure
($\beta$) and magnetic field contour lines of the magnetized disk with
neutrino cooling at $t=0.03$s snapshot.

Figure 3: The 2D profile of the electron fraction ($Y_{e}$) at $t=0.03$s
snapshot.

Figure 4: The 2D profile of the temperature at $t=0.03$s snapshot.

Figure 5: The accretion rate comparison between magnetized and non-magnetized
cases.

Figure 6: The neutrino luminosity comparison between magnetized and non-
magnetized cases.

Figure 7: Histograms of unbounded mass as a function of electron fraction. The
yellow is the average taken over several time snapshots from the first half of
the evolution time, while the cyan is the average from the second half.

Figure 8: The unbound mass generated during the evolution. The rate of the
unbound mass decreases by time, because it is not possible to maintain the
magnetic field in 2D for a long evolution in order to support the angular
momentum transport via MRI.

Figure 9: Histograms of unbounded mass as a function of temperature. The
averages are taken same as Fig. 8.

Figure 10: Histograms of unbounded mass as a function of velocity. The
averages are taken same as Fig. 8.

## References

  * Balbus & Hawley (1991) Balbus, S. A., Hawley, J. F., _ApJ_ 376, 214 (1991)
  * Berger (2014) Berger, E., _Ann. Rev. Astron. Astrophys._ 52, 43 (2014)
  * Blandford & Znajek (1977) Blandford, R. D., Znajek, R. L. 179, 433 (1977)
  * Di Matteo et al. (2002) Di Matteo, T., Perna, R., Narayan, R., _ApJ_ 579, 2, 706 (2002)
  * Dietrich & Ujevic (2017) Dietrich, T., Ujevic, M., _Classical and Quantum Gravity_ 34, 10, 105014 (2017)
  * Fishbone & Moncrief (1976) Fishbone, L. G., Moncrief, V., _ApJ_ 207, 962 (1976)
  * Janiuk (2019) Janiuk, A., _ApJ_ 882, 2, 163 (2019)
  * Janiuk et al. (2013) Janiuk, A., Mioduszewski, P., Moscibrodzka, M., _ApJ_ 776, 2, 105 (2013)
  * Janiuk et al. (2004) Janiuk, A., Perna, R., Di Matteo, T., Czerny, B., _Mon. Not. Roy. Astron. Soc._ 355, 950 (2004)
  * Metzger & Berger (2012) Metzger, B. D., Berger, E. 746, 48 (2012)
  * Nakar (2007) Nakar, E., _Phys. Rep._ 442, 166 (2007)
  * Popham et al. (1999) Popham, R., Woosley, S. E., Fryer, C., _Astrophys. J._ 518, 356 (1999)
  * Reddy et al. (1998) Reddy, S., Prakash, M., Lattimer, J. M., _Phys. Rev. D_ 58, 013009 (1998), URL https://link.aps.org/doi/10.1103/PhysRevD.58.013009
  * Sapountzis & Janiuk (2019) Sapountzis, K., Janiuk, A., _ApJ_ 873, 1, 12 (2019)