# SNeRL: Semantic-aware Neural Radiance Fields for Reinforcement Learning

Dongseok Shim  Seungjae Lee  H. Jin Kim

###### Abstract

As previous representations for reinforcement learning cannot effectively
incorporate a human-intuitive understanding of the 3D environment, they
usually suffer from sub-optimal performances. In this paper, we present
Semantic-aware Neural Radiance Fields for Reinforcement Learning (SNeRL),
which jointly optimizes semantic-aware neural radiance fields (NeRF) with a
convolutional encoder to learn 3D-aware neural implicit representation from
multi-view images. We introduce 3D semantic and distilled feature fields in
parallel to the RGB radiance fields in NeRF to learn semantic and object-
centric representation for reinforcement learning. SNeRL outperforms not only
previous pixel-based representations but also recent 3D-aware representations
both in model-free and model-based reinforcement learning.

Machine Learning, ICML

##  1 Introduction

Developing agents that can achieve complex control tasks directly from image
inputs has been a long-standing problem in reinforcement learning (RL).
Previous works over the past few years have made notable progress in the data
efficiency of learning visual control problems. The most challenging part of
solving visual control tasks is obtaining the low-dimensional latent
representations from high-dimensional observations. To this end, they pre-
train the encoder in various ways such as unsupervised representation learning
via image reconstruction using offline datasets (Finn et al., 2016; Kulkarni
et al., 2019; Islam et al., 2022), contrastive learning (Zhan et al., 2022),
reconstructing task information (Yang & Nachum, 2021; Yamada et al., 2022),
and training multi-view consistency (Dwibedi et al., 2018). Other approaches
utilize joint learning of auxiliary unsupervised tasks (Laskin et al., 2020b;
Schwarzer et al., 2020), and data-augmented reinforcement learning (Laskin et
al., 2020a; Yarats et al., 2021).

Figure 1:  Semantic-aware NeRF for reinforcement learning. We present SNeRL, a
reinforcement learning framework that learns 3D-aware representation with a
convolutional encoder and semantic-aware NeRF decoder. The latent vectors from
the encoder are propagated to the policy network to generate an action for RL
agents.

While a number of works have been proposed to improve the data efficiency in
visual control problems, the majority of the encoders trained from those
methods have limited capability in obtaining 3D structural information and
lack equivariance to 3D transformations. Such limitations come from ignoring
3D structural information and learning visual representation from a single-
view observation.

Recently, there have been attempts to consider 3D information of the
environment in robot control and manipulation (Li et al., 2022; Driess et al.,
2022) by learning implicit spatial representation via neural radiance fields
(NeRF) (Mildenhall et al., 2020). They map pixel-level multi-view observations
of a scene to a latent vector through an autoencoder structure, where the NeRF
decoder provides 3D structured neural scene representation via RGB self-
supervision for each view.

Even though the aforementioned pioneers achieved better performance compared
to the previous RL algorithms with a single-view observation, they still did
not take full advantage of 3D-aware representation learning. It is because
those methods only exploit RGB supervision to train NeRF, which makes it
difficult for the encoder to learn object-centric or semantic representation
for RL downstream tasks. Although NeRF-RL (Driess et al., 2022) proposes
compositional NeRF to mitigate such limitations, the RL agents in NeRF-RL
require object-individual masks during training and deployment to utilize
semantic representations, which is quite unrealistic.

In this work, we propose Semantic-aware Neural Radiance Fields for
Reinforcement Learning (SNeRL) which learns both 3D-aware semantic and
geometric representation for RL agents in a 3-dimensional environment. First
of all, our proposed method learns 3D-aware semantic representation by
predicting 3D semantic fields with ground truth labels. As a result, SNeRL
enables downstream visual control tasks without object-individual masks and
addresses the limitation of the prior work (Driess et al., 2022). Also, to
capture further fine-grained features that could not be fully expressed in
semantic fields and to take advantage of data-driven approaches, we employ an
off-the-shelf feature descriptor (Caron et al., 2021) as a teacher network and
learn to predict feature fields via a distillation method such as Kobayashi et
al. (2022).

We also introduce a multi-view adaptation of recent self-predictive
representation learning (Chen & He, 2021) as an auxiliary task which further
improves the performance of SNeRL. In the proposed auxiliary task, SNeRL
computes the target representation by utilizing the observations from
different camera views in the same timestep to learn spatially consistent
representation.

Our proposed SNeRL outperforms not only the previous single-view
representation learning algorithms for RL (Laskin et al., 2020b; Yarats et
al., 2021) but also the state-of-the-art method with multi-view observations
(Driess et al., 2022) in four different visual control tasks.

To sum up, our contribution can be summarized as follows:

  * •

We present SNeRL, a framework that utilizes NeRF with semantic and distilled
feature fields to learn 3D-aware semantic representation for reinforcement
learning.

  * •

We validate the effectiveness of SNeRL both with model-free and model-based
methods. To the best of our knowledge, SNeRL is the first work that leverages
semantic-aware representations without object masks in RL downstream tasks.
Also, this is the first study to utilize 3D-aware representations to model-
based RL.

  * •

The proposed SNeRL outperforms the previous single and multi-view image-based
RL algorithms in four different 3D environments from Meta-world. In addition,
auxiliary self-predictive representation learning with multi-view observations
proposed for spatially consistent representation can enable further
improvements.

##  2 Related Work

###  2.1 3D Scene Representation Learning

To learn 3D-aware representation from a single view image, the previous
methods exploit standard convolutional autoencoder architecture conditioned by
the camera poses, which generates scenes from arbitrary views with either
deterministic (Tatarchenko et al., 2016; Worrall et al., 2017) or stochastic
(Eslami et al., 2018) latent vectors. Recently, neural radiance fields (NeRF)
have achieved an exceptional progress in understanding 3D scenes and
synthesizing novel views. Following, some approaches propose latent-
conditioned NeRF (Martin-Brualla et al., 2021; Yu et al., 2021; Wang et al.,
2021), but the major objective of the aforementioned methods is improving the
quality of synthesized images rather than extracting time-variant latent
vectors with 3D dynamic scene understanding from multi-view inputs. In this
paper, we leverage the autoencoder with convolutional encoder and NeRF-style
decoder (Li et al., 2022; Driess et al., 2022) so that the encoder can extract
3D-aware representation from multi-view inputs for RL downstream tasks.

###  2.2 Representation Learning for RL

The RL frameworks with image inputs typically have an encoder, which maps
high-dimensional observations to a low-dimensional latent vector. RL agent is
trained over the latent state space to maximize its objective functions, e.g.,
the total discounted reward for each episode. While a number of works have
made significant advancements, it still remains a challenging open problem.

To address the sample inefficiency of image-based RL, prior works adopt
various data-augmentation techniques (Laskin et al., 2020a; Yarats et al.,
2021), contrastive learning with data augmentation (Laskin et al., 2020b;
Schwarzer et al., 2020; Stooke et al., 2021; Liu & Abbeel, 2021; Zhan et al.,
2022), representation learning from image reconstruction (Islam et al., 2022;
Kulkarni et al., 2019), or task information reconstruction (Yang & Nachum,
2021; Yamada et al., 2022). Other approaches propose to capture the relations
between multi-view data (Dwibedi et al., 2018; Kinose et al., 2022; Sermanet
et al., 2018) or keypoints (Manuelli et al., 2020). There are also some
approaches leveraging transition sequence data (Hansen et al., 2020; You et
al., 2022), or pre-training with offline image-based RL (Wang et al., 2022).
Unfortunately, these works have limited capability in learning 3D-structural
information and could not obtain an intuitive understanding of the 3D
environments that humans have because of the 2D bias that 2D convolutional
neural networks have.

In recent, there have been attempts to learn the 3D structure of the real
world (Li et al., 2022; Driess et al., 2022). Li et al. (2022) firstly
proposes autoencoder with convolutional encoder and NeRF (Mildenhall et al.,
2020) decoder to control the visuomotor with learned dynamics model and model-
predictive control (MPC). Following, NeRF-RL (Driess et al., 2022) extends the
prior study and firstly introduces NeRF-based architecture to the general
model-free RL framework. However, they could not learn semantic features due
to the limited RGB supervision with naïve NeRF. To learn object-centric
representation only with RGB supervision, NeRF-RL presents compositional NeRF
with object-individual masks, but requiring masks during the deployment of RL
agents seems to be a strong assumption.

In this paper, we propose SNeRL which learns both geometric and semantic
information with RGB, semantic, and distilled feature supervision for RL
downstream tasks without any object masks during the inference phase.

##  3 Preliminaries

###  3.1 Neural Radiance Fields

The concept of neural radiance fields (NeRF) (Mildenhall et al., 2020) is to
represent the 3D scene with learnable and continuous volumetric fields
$f_{\theta}$. Specifically, at any 3D world coordinate
$\mathbf{x}\in\mathbb{R}^{3}$ and unit viewing direction
$\mathbf{d}\in\mathbb{R}^{3}$, $f_{\theta}$ estimates the differntiable volume
density $\sigma$ and RGB color $\mathbf{c}$:
$f_{\theta}(\mathbf{x},\mathbf{d})=(\sigma,\mathbf{c})$. Let the camera ray of
the pixel in the camera coordinate be $\mathbf{r}=\mathbf{o}+t\mathbf{d}$,
where $\mathbf{o}$ indicates the camera origin. The corresponding pixel value
from an arbitrary view can be rendered through volumetric radiance fields as:

$C(\mathbf{r})=\int^{t_{f}}_{t_{n}}T(t)\sigma(t)\mathbf{c}(t)dt$ (1)

where $T(t)=\mathrm{exp}(-\int^{t}_{t_{n}}\sigma(s)ds)$ and $t_{n}$ and
$t_{f}$ indicate pre-defined lower and upper bound of the depth respectively.

Then, $f_{\theta}$, which is usually formulated with MLP, is optimized through
pixel-wise RGB supervision from multiple views as:

$\mathcal{L}=\sum_{i,j}||\hat{C}(\mathbf{r}_{i,j})-C(\mathbf{r}_{i,j})||_{2}^{2},$
(2)

where $\mathbf{r}_{i,j}$ indicates ray $j$ from images of $i^{th}$ view.
$\hat{C}$ and $C$ represents the rendered volumetric fields into 2D image and
ground truth pixel value respectively.

###  3.2 Reinforcement Learning

We consider a finite-horizon Markov Decision Process (MDP)
$\mathcal{M}=(\mathcal{O},\mathcal{A},\mathcal{T},\mathcal{R},\gamma)$, where
$\mathcal{O}$ denotes the high-dimensional observation space (image pixels),
$\mathcal{A}$ the action space, $\mathcal{T}(o^{\prime}|o,a)$ the transition
dynamics $(o,o^{\prime}\in\mathcal{O},a\in\mathcal{A})$,
$\mathcal{R}:\mathcal{O}\times\mathcal{A}\rightarrow\mathbb{R}$ the reward
function, and $\gamma\in[0,1)$ the discount factor. Following the general idea
of learning RL downstream tasks with pre-trained scene representations, we
consider an encoder $\Omega:\mathcal{O}\rightarrow\mathcal{Z}$ that maps and
high-dimensional observation $o\in\mathcal{O}$ to a low-dimensional latent
state $z\in\mathcal{Z}$ on which an RL agent operates. To learn how to succeed
in downstream tasks, the RL policy $\pi_{\theta}(a\in\mathcal{A}|z=\Omega(o))$
maximizes the total discounted reward
$\sum_{t=0}^{H-1}=\gamma^{t}\mathcal{R}(o_{t},a_{t})$ of trajectories
$\tau_{i}=(z_{0},o_{0},...,z_{H},o_{H})_{i}$.

##  4 Method

Figure 2:  SNeRL Overview. SNeRL consists of two stages, which are pre-
training NeRF-based autoencoder and fine-tuning to the downstream RL tasks,
respectively. With observations from three different camera views, an encoder
produces a single latent vector $z$, and a decoder with neural rendering
function $f_{\theta}$ takes the position $\mathbf{x}$, viewing direction
$\mathbf{d}$ in the 3D coordinates and $z$ as inputs to synthesize three
different fields in the arbitrary views. An auxiliary multi-view self-
prediction loss is applied to enable view-invariant representation. Then, the
encoder and the decoder are jointly optimized in a supervised manner with an
offline dataset. The pre-trained encoder is utilized as a feature extractor to
train the policy with off-the-shelf RL algorithms.

In this section, we demonstrate the details of SNeRL which consists of a
multi-view convolutional image encoder and a latent-conditioned NeRF decoder
to learn the 3D-aware representation. Compared to the previous method (Driess
et al., 2022) which also proposes NeRF supervision for RL, SNeRL is capable of
extracting object-centric or semantic representation without any object-
individual masks during deployment. The pre-trained image encoder is exploited
as a feature extractor for downstream RL tasks, and the overview of SNeRL
framework is depicted in Figure 2.

###  4.1 Multi-view Encoder

Similar to Li et al. (2022), we adopt the multi-view encoder $\Omega$ which
fuses the observations from multiple camera views together to learn a single
latent vector $z$ for RL tasks. The encoder takes the pixel-level observations
$o^{i}\in\mathbb{R}^{H\times W\times 3}$, and the corresponding camera
projection matrices $K^{i}\in\mathbb{R}^{3\times 4}$ captured from $V$
different camera view as inputs, i.e., $i=1\cdots V$. To generate
$z\in\mathcal{Z}$ from the inputs, a convolutional network $E_{\mathrm{CNN}}$
first extracts viewpoint-invariant features from each image. The features from
different camera views are channel-wise concatenated with their corresponding
(flattened) camera projection matrices to reflect the viewpoint information to
the following feature vectors. Then, the concatenated vectors are passed
through MLP layers, $g_{\mathrm{MLP}}$, to produce mid-level viewpoint-aware
encodings. Lastly, the feature encodings from different camera views are
averaged to generate a single encoding, and the averaged feature encoding is
projected to the latent space $\mathcal{Z}$ with the latent encoder
$h_{\mathrm{MLP}}$ as follows:

$\begin{split}z=&\,\Omega(o^{1:V},K^{1:V})\\\
=&\,h_{\mathrm{MLP}}(\frac{1}{V}\sum_{i=1}^{V}g_{\mathrm{MLP}}(E_{\mathrm{CNN}}(o^{i}),K^{i}))\end{split}$
(3)

###  4.2 Semantic-aware NeRF Decoder

To inject 3D structural information into the latent vector $z$, we leverage a
latent-conditioned NeRF architecture (Yu et al., 2021; Martin-Brualla et al.,
2021; Wang et al., 2021) for the decoder. The difference between previous
latent-conditioned NeRF and our proposed SNeRL is that the neural rendering
function $f_{\theta}$ from SNeRL not only synthesizes novel views with RGB
pixel value $\mathbf{c}$ but also with the semantic label $\mathbf{s}$ (Zhi et
al., 2021; Fu et al., 2022; Kundu et al., 2022) and high-dimensional distilled
features $\mathbf{f}$ from the large-scale teacher network (Kobayashi et al.,
2022) as follows:

$\mathbf{c}=f_{\theta}(z,\mathbf{x},\mathbf{d}),\,\,\mathbf{s},\,\mathbf{f}=f_{\theta}(z,\mathbf{x})$
(4)

By estimating three different radiance fields (semantic, feature, and RGB),
the latent vector $z$ is jointly optimized to learn the geometric and semantic
representations of the 3D environment. Unlike RGB value $\mathbf{c}$ which is
dependent on both the position $\mathbf{x}$ and the viewing direction
$\mathbf{d}$, we formulate the semantic label and distilled feature to be
invariant to the viewing direction $\mathbf{d}$ because the inherent
properties of the scene or the object do not change according to the direction
of the camera ray.

As SNeRL predicts three different fields, RGB, semantic, and distilled
feature, by adding field-wise branches, they share the neural rendering
function $f_{\theta}$ until estimating the density $\sigma$. It indicates that
three radiance fields have the same accumulated transmittance $T(t)$ at depth
$t\in[t_{n},t_{f}]$ along the ray $\mathbf{r}=\mathbf{o}+t\mathbf{d}$ as

$T(t)=\mathrm{exp}(-\int^{t}_{t_{n}}\sigma(\mathbf{r}(s))ds).$ (5)

For rendering the RGB field, we follow the same training framework as general
latent-conditioned NeRF (Yu et al., 2021; Martin-Brualla et al., 2021; Wang et
al., 2021), which optimizes the neural rendering function $f_{\theta}$ via
pixel-wise RGB supervision. RGB supervision enables the encoder to extract
geometric features from the observed environment by learning the RGB and
density distribution in the 3-dimensional space. The rendered pixel value
$\hat{C}(\mathbf{r})$ can be calculated as

$\hat{C}(\mathbf{r})=\int^{t_{f}}_{t_{n}}T(t)\sigma(\mathbf{r})\mathbf{c}(\mathbf{r},\mathbf{d})dt,$
(6)

and the loss function for RGB field, $\mathcal{L}_{\mathrm{RGB}}$, can be
formulated with simple L2 loss between the rendered $\hat{C}(\mathbf{r})$ and
the ground truth pixel colors $C(\mathbf{r})$,

$\mathcal{L}_{\mathrm{RGB}}=\sum_{i,j}||\hat{C}(\mathbf{r}_{i,j})-C(\mathbf{r}_{i,j})||_{2}^{2},$
(7)

where $\mathbf{r}_{i,j}$ indicates the camera ray $j$ from the observation
$i$, $o^{i}$.

Unfortunately, optimizing the encoder only with only an RGB reconstruction is
difficult to capture the semantic or object-centric properties of the 3D
scene, which are crucial for downstream RL tasks. Therefore, we extend NeRF-
based decoder by appending additional branches before injecting the viewing
direction $\mathbf{d}$ into the rendering function, $f_{\theta}$ for semantic
segmentation. The rendered semantic labels $\hat{S}(\mathbf{r})$ can be
calculated as

$\hat{S}(\mathbf{r})=\int^{t_{f}}_{t_{n}}T(t)\sigma(\mathbf{r})\mathbf{s}(\mathbf{r})dt$
(8)

and the loss function for semantic field, $\mathcal{L}_{seg}$, can be
formulated with the standard cross entropy loss,

$\mathcal{L}_{sem}=-\sum_{i,j}\sum_{l=1}^{L}S^{l}(\mathbf{r}_{i,j})\mathrm{log}\,\hat{S}^{l}(\mathbf{r}_{i,j}),$
(9)

where $\hat{S}^{l}$ and $S^{l}$ denote the probability of the ray $j$ in
observation $i$ belonging to the class $l$ and its corresponding ground-truth
semantic labels, respectively.

To capture further fine-grained features that could not be fully expressed in
semantic fields, SNeRL also synthesizes distilled feature fields (Kobayashi et
al. (2022)) that predict the output of a pre-trained feature descriptor in a
knowledge-distillation manner (Hinton et al., ). It is well known from prior
literature (Caron et al., 2021) that Vision Transformer (ViT) (Dosovitskiy et
al., 2020) trained in a self-supervised manner, e.g. DINO (Caron et al.,
2021), can work as an excellent feature descriptor which explicitly represents
the scene layouts such as object boundaries. Since the output of the ViT
feature descriptor contains high-dimensional information with different values
in all pixels depending on the geometric relationship or semantic meaning, the
pre-trained ViT becomes a good feature descriptor with another advantage from
the semantic label.

Therefore, we take advantage of such benefits to the NeRF-based decoder so
that the latent vector $z$ learns high-level information by distilling the
knowledge from ViT teacher network which cannot be learned via ground-truth
semantic supervision. The distilled feature fields can be rendered as

$\hat{F}(\mathbf{r})=\int^{t_{f}}_{t_{n}}T(t)\sigma(\mathbf{r})\mathbf{f}(\mathbf{r})dt.$
(10)

The loss function for distilled feature field, $\mathcal{L}_{feat}$, is
formulated by penalizing the difference between the rendered features
$\hat{F}(\mathbf{r})$ and the outputs of ViT feature descriptor
$F(o,\mathbf{r})$ as

$\mathcal{L}_{feat}=\sum_{i,j}||\hat{F}(\mathbf{r}_{i,j})-F(o^{i},\mathbf{r}_{i,j})||_{1}.$
(11)

Finally, the total loss function $\mathcal{L}$ for jointly optimizing the
multi-view encoder and NeRF-based decoder can be formulated as the linear
combination of aforementioned losses as:

$\mathcal{L}=\mathcal{L}_{\mathrm{RGB}}+\lambda_{sem}\mathcal{L}_{sem}+\lambda_{feat}\mathcal{L}_{feat}$
(12)

where $\lambda_{sem}$ and $\lambda_{feat}$ are set to 0.004 and 0.04,
respectively, to balance the losses (Zhi et al., 2021; Kobayashi et al.,
2022). After training, the multi-view encoder $\Omega$ is exploited as a 3D
structural and semantic feature extractor for any off-the-shelf downstream RL
algorithms.

###  4.3 Multi-view Self Predictive Representation

We additionally enforce the multi-view self-predictive loss to the latent
vector $z$ to ensure that the encoder learns the viewpoint-invariant
representation with observations from the same scene. The randomly sampled
observations from two different camera pose, $o^{1}$ and $o^{2}$, are
processed by the convolutional feature extractor, $E_{\mathrm{CNN}}$, and the
weights of the feature extractor are shared between two inputs. A feature from
one view, $z_{1}$, is mapped with a prediction network, $h_{pred}$, to match
it to the feature from the other view, $z_{2}$. We formulate the self-
predictive loss function $D$ with negative cosine similarity as follows:

$\mathcal{D}(p_{1},z_{2})=-\frac{p_{1}}{{||p_{1}||_{2}}}\cdot\frac{z_{2}}{{||z_{2}||_{2}}},$
(13)

where $p_{1}$ and $z_{2}$ indicate two output vectors, $p_{1}\triangleq
h_{pred}(E_{\mathrm{CNN}}(o^{1}))$ and $z_{2}\triangleq
E_{\mathrm{CNN}}(o^{2})$, respectively. We assume that $z_{2}$ is constant and
the encoder $E_{\mathrm{CNN}}$ only receives the gradient from $p_{1}$
following Chen & He (2021).

The symmetrized auxiliary representation loss function can be formulated as
follows:

$\mathcal{L}_{\mathrm{aux}}=\frac{1}{2}\mathcal{D}(p_{1},z_{2})+\frac{1}{2}\mathcal{D}(p_{2},z_{1}).$
(14)

##  5 Experiments

Figure 3:  Episode returns of the evaluation results. Shading indicates a
standard deviation across 4 seeds. The curves are not visible in the Hammer-v2
environment as they overlap each other. Note that SNeRL in this figure is
obtained without auxiliary loss in section 4.3 (multi-view self-predictive
presentation), which could enable further improvements in some environments.

In this section, we demonstrate several experiments on the 3-dimensional
environments to explore the effectiveness of SNeRL compared to existing state-
of-the-art RL algorithms both in model-free and model-based settings. We fix
the downstream RL algorithms and adopt Soft Actor-Critic (Haarnoja et al.,
2018) in the model-free setting and Dreamer (Hafner et al., 2019) in the
model-based setting for SNeRL and all the baselines for a fair evaluation.

#### Environments.

We evaluate SNeRL on four visual control environments based on the MuJoCo
(Todorov et al., 2012), including some complex control tasks that require
clever use of interactions between the objects to obtain high rewards. All the
tasks are performed by a simulated Sawyer robot which has a single arm and
gripper in hand (4-DoF). The action space of the Sawyer robot consists of the
position (x,y,z) of the end-effector and gripper control (open/close). The
agent takes 128x128 images from three different camera views as pixel-level
inputs and receives dense rewards from the environment provided by Meta-world
(Yu et al., 2020).

  * •

Window-open-v2 : This environment involves the Sawyer robot opening a sliding
window with a handle. The initial state of the robotic hand is [0, 0.4, 0.2]
and the robot receives rewards for pushing the handle and opening a window
located in [-0.1, 0.785, 0.16].

  * •

Hammer-v2 : The Sawyer robot is supposed to grasp the handle of the hammer,
which is generated in a random position, and hit the head of the nail to drive
it. The initial state of the robotic hand is generated randomly in
$\\{(x,y,z)|-0.5\leq x\leq 0.5,0.4\leq y\leq 1,0.05\leq z\leq 0.5\\}$. The
robot receives rewards for picking up the hammer and inserting the nail into a
piece of wood.

  * •

Drawer-open-v2 : The Sawyer robot is supposed to open a drawer by holding the
handle of the drawer and pulling it. The initial state of the robotic hand is
the same as Hammer-v2. The robot receives rewards for opening a drawer.

  * •

Soccer-v2 : In this task, the Sawyer robot tries to score by pushing a soccer
ball that is generated in a random position. The initial state of the robotic
hand is the same as Hammer-v2. The robot receives rewards for touching the
soccer ball and putting it into the net.

We refer to Meta-world (Yu et al., 2020) for more details including the reward
function and the range of the random positions.

Figure 4:  Ablation study. (a): SNeRL with both semantic and feature
supervision shows higher performance than the case where only one of the two
is applied. Also, in relatively simple environments, using only one of the two
could be enough to improve the performance of the prior work. (b): Additional
multi-view self-predictive loss can further improve SNeRL in some
environments. (c): Learned representations via SNeRL can also be adopted in
model-based RL.

#### Baselines.

We compare SNeRL to several state-of-the-art visual RL methods and a 3D-aware
RL method, which are briefly described below. DrQ-v2 (Yarats et al., 2021) is
an improved version of DrQ (Yarats et al., 2020), which solves visual control
tasks with data augmentation and scheduled exploration noise. CURL (Laskin et
al., 2020b) trains RL agents with an auxiliary contrastive loss which ensures
that the embeddings for data-augmented versions of observations match. CURL-
multiview is a multi-view adaptation of CURL, which utilize 3 different camera
views and has a CNN encoder with the same structure as that of SNeRL. CNN-AE
uses a standard CNN autoencoder (instead of NeRF decoder) to pre-train an
encoder using the reconstruction loss proposed in Finn et al. (2016). NeRF-RL
(Driess et al., 2022) pre-trains an autoencoder with convolutional encoder and
naïve NeRF-style decoder, without semantic and feature supervision.

We note that learning downstream RL tasks in CNN-AE and NeRF-RL shares the
identical method as SNeRL, and they use the same offline dataset collected by
random actions and the policies provided by Meta-world (half-and-half mixed).
We refer the reader to Appendix B.2 for the experiments on other datasets.
Also, all the multi-view methods (CURL-multiview, CNN-AE, NeRF-RL, SNeRL)
receive the same observations and do not receive per-object masks from the
environment. For the rest of the baselines which operate on a single view, we
choose a single camera position from which the states of each object can be
observed clearly.

###  5.1 Experiment Result

Figure 3 shows the episode returns of SNeRL and baselines in 4 different
visual control tasks. Thanks to the learned object-centric representation via
semantic and distilled feature supervision, SNeRL consistently outperforms
state-of-the-art visual RL methods and the prior 3D-aware RL method (NeRF-RL)
in terms of data efficiency and performance.

Specifically, the contrastive baselines (CURL, CURL-multiview) and DrQ-v2
could not achieve high returns in the difficult environments (soccer and
hammer) even though some of them succeed in the relatively easy environment
(window). The results also show that pre-training CNN via naïve reconstruction
loss (CNN-AE) with offline data could not succeed in the environments at all.
These imply that extracting not only the 3D-aware geometric but also the
object-centric and semantic information from multi-view observations is
critical for RL performances.

Interestingly, we observe that pre-training a NeRF-based autoencoder only with
RGB supervision (NeRF-RL) is not sufficient to learn the features for RL
downstream tasks, and it can not outperform multi-view adaptation of the
visual RL method with contrastive loss (CURL-multiview). These are contrary to
the results reported in the prior work (Driess et al., 2022), which we analyze
as follows: the environments we adopted in this work are more challenging
compared to those of Driess et al. (2022), which consist of simple-shaped
objects with primary colors. Therefore, it is relatively difficult to obtain
semantic information simply using RGB supervision. Thus, leveraging a
semantic-aware NeRF decoder is required to extract the features for better
performances in RL downstream tasks in the practical use of 3D-aware RL, which
is consistent with our analysis.

Figure 5: Qualitative results on the image reconstruction in 3 different
camera views via neural rendering. The synthesized images from SNeRL achieve
better fidelity compared to NeRF-RL in several environments.

###  5.2 Ablation Study

#### Semantic and Distilled Feature Fields.

To validate how each semantic-aware radiance field leveraged in SNeRL
(semantic and distilled feature fields) contributes to the downstream RL
performances, we evaluate its two ablated variants without semantic and
feature supervision, respectively. As shown in Figure 4(a), SNeRL, which takes
advantage of both semantic and feature supervision from ground-truth labels
and a ViT-based feature descriptor, achieves the best performance compared to
all the ablated models. We observe that the performance gap between SNeRL and
the ablated models depends on the environment, as semantic labels are
sufficient to learn semantic information in a relatively simple environment
(window), but it requires both supervision in a complex environment (soccer).

#### Multi-view Representation Learning.

We also introduce auxiliary representation learning which is suitable for
multi-view observations with self-predictive loss. To demonstrate its
effectiveness on downstream RL tasks, we evaluate two different models, which
are SNeRL with and without multi-view self-predictive representation learning.
By enforcing the latent vector to be invariant to the viewpoint of the
observation, we report that the proposed representation learning improves the
RL agent’s performance in some environments as shown in Figure 4(b).

###  5.3 Image Reconstruction via Neural Rendering

Even though we only leverage the convolutional encoder for downstream RL
tasks, we compare the image rendering performance of NeRF-based decoders from
SNeRL and NeRF-RL to explore the relationship between the synthesized image
quality and RL performances. As NeRF originally aims to synthesize images of
arbitrary camera views from a static scene, NeRF-RL, which also trains the
volumetric field with a sole RGB supervision, cannot reconstruct dynamic
objects, e.g. the robot arm, in input images without semantic information. On
the other hand, SNeRL, which utilizes semantic labels and feature outputs from
the ViT teacher network as additional supervision signals, not only achieves
better RL performance but also well represents the dynamic scene and produces
high-fidelity rendering outputs as shown in Figure 5.

###  5.4 SNeRL for Model-based RL

In this section, we evaluate whether the learned representation via SNeRL can
also be adopted in off-the-shelf model-based reinforcement learning
algorithms, which train a world model to characterize the environment and
conduct planning over the learned model. We adopt Dreamer (Hafner et al.,
2019) as a downstream model-based RL agent and replace the encoder of the
representation model with our pre-trained encoder. Refer to Appendix A for
additional implementation details and an architectural overview.

Our results are shown in Figure 4(c). We observe that learning model-based RL
with the pre-trained encoder of SNeRL outperforms the pre-trained weights of
the prior 3D-aware RL method (NeRF-RL) and naïve CNN autoencoder. This
empirical evidence is consistent with the case of model-free RL in section
5.1, indicating that the proposed method allows the encoder to learn
representations that are important for general off-the-shelf RL agents.

##  6 Conclusion

In this paper, we present SNeRL, a semantic-aware radiance field for RL, that
outperforms existing representation learning methods for RL algorithms across
four different 3-dimensional environments. SNeRL leverages semantic and
distilled feature supervision with latent condition NeRF autoencoders as well
as RGB supervision to enable image encoders to express 3D-aware geometric and
semantic representation on downstream RL tasks. We also propose a multi-view
self-predictive loss as an auxiliary representation learning to force latent
vectors to be viewpoint invariant. Finally, we verify that SNeRL is effective
in both model-free and model-based RL algorithms.

#### Limitations.

Despite these improvements, SNeRL inherits the limitations of the prior
3D-aware RL method. First of all, SNeRL requires multi-view offline data, and
collecting an offline dataset covering the state space in some complex control
tasks might be challenging. Also, our method uses a NeRF decoder that consumes
more computational budget than CNN, so there might be limitations in extending
our method to an online setup which trains the encoder concurrently with RL
agents.

##  7 Acknowledgement

This research was supported by Institute of Information & communications
Technology Planning & Evaluation (IITP) grant funded by the Korea
government(MSIT) [NO.2021-0-01343, Artificial Intelligence Graduate School
Program (Seoul National University)]. Also, this research was supported by
Unmanned Vehicles Core Technology Research and Development Program through the
National Research Foundation of Korea(NRF) and Unmanned Vehicle Advanced
Research Center(UVARC) funded by the Ministry of Science and ICT, the Republic
of Korea(NRF-2020M3C1C1A01086411). Seungjae Lee would like to acknowledge
financial support from Hyundai Motor Chung Mong-Koo Foundation.

## References

  * Caron et al. (2021) Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A.  Emerging properties in self-supervised vision transformers.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 9650–9660, 2021.
  * Chen & He (2021) Chen, X. and He, K.  Exploring simple siamese representation learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 15750–15758, 2021.
  * Dosovitskiy et al. (2020) Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _International Conference on Learning Representations_ , 2020.
  * Driess et al. (2022) Driess, D., Schubert, I., Florence, P., Li, Y., and Toussaint, M.  Reinforcement learning with neural radiance fields.  _arXiv preprint arXiv:2206.01634_ , 2022.
  * Dwibedi et al. (2018) Dwibedi, D., Tompson, J., Lynch, C., and Sermanet, P.  Learning actionable representations from visual observations.  In _2018 IEEE/RSJ international conference on intelligent robots and systems (IROS)_ , pp. 1577–1584. IEEE, 2018.
  * Eslami et al. (2018) Eslami, S. A., Jimenez Rezende, D., Besse, F., Viola, F., Morcos, A. S., Garnelo, M., Ruderman, A., Rusu, A. A., Danihelka, I., Gregor, K., et al.  Neural scene representation and rendering.  _Science_ , 360(6394):1204–1210, 2018.
  * Finn et al. (2016) Finn, C., Tan, X. Y., Duan, Y., Darrell, T., Levine, S., and Abbeel, P.  Deep spatial autoencoders for visuomotor learning.  In _2016 IEEE International Conference on Robotics and Automation (ICRA)_ , pp. 512–519. IEEE, 2016.
  * Fu et al. (2022) Fu, X., Zhang, S., Chen, T., Lu, Y., Zhu, L., Zhou, X., Geiger, A., and Liao, Y.  Panoptic nerf: 3d-to-2d label transfer for panoptic urban scene segmentation.  _arXiv preprint arXiv:2203.15224_ , 2022.
  * Haarnoja et al. (2018) Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., et al.  Soft actor-critic algorithms and applications.  _arXiv preprint arXiv:1812.05905_ , 2018.
  * Hafner et al. (2019) Hafner, D., Lillicrap, T., Ba, J., and Norouzi, M.  Dream to control: Learning behaviors by latent imagination.  _arXiv preprint arXiv:1912.01603_ , 2019.
  * Hansen et al. (2020) Hansen, N., Jangir, R., Sun, Y., Alenyà, G., Abbeel, P., Efros, A. A., Pinto, L., and Wang, X.  Self-supervised policy adaptation during deployment.  _arXiv preprint arXiv:2007.04309_ , 2020.
  * (12) Hinton, G., Vinyals, O., Dean, J., et al.  Distilling the knowledge in a neural network.
  * Islam et al. (2022) Islam, R., Zang, H., Goyal, A., Lamb, A., Kawaguchi, K., Li, X., Laroche, R., Bengio, Y., and Combes, R. T. D.  Discrete factorial representations as an abstraction for goal conditioned reinforcement learning.  _arXiv preprint arXiv:2211.00247_ , 2022.
  * Kinose et al. (2022) Kinose, A., Okada, M., Okumura, R., and Taniguchi, T.  Multi-view dreaming: Multi-view world model with contrastive learning.  _arXiv preprint arXiv:2203.11024_ , 2022.
  * Kobayashi et al. (2022) Kobayashi, S., Matsumoto, E., and Sitzmann, V.  Decomposing nerf for editing via feature field distillation.  _arXiv preprint arXiv:2205.15585_ , 2022.
  * Kulkarni et al. (2019) Kulkarni, T. D., Gupta, A., Ionescu, C., Borgeaud, S., Reynolds, M., Zisserman, A., and Mnih, V.  Unsupervised learning of object keypoints for perception and control.  _Advances in neural information processing systems_ , 32, 2019.
  * Kundu et al. (2022) Kundu, A., Genova, K., Yin, X., Fathi, A., Pantofaru, C., Guibas, L. J., Tagliasacchi, A., Dellaert, F., and Funkhouser, T.  Panoptic neural fields: A semantic object-aware neural scene representation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 12871–12881, 2022.
  * Laskin et al. (2020a) Laskin, M., Lee, K., Stooke, A., Pinto, L., Abbeel, P., and Srinivas, A.  Reinforcement learning with augmented data.  _Advances in neural information processing systems_ , 33:19884–19895, 2020a.
  * Laskin et al. (2020b) Laskin, M., Srinivas, A., and Abbeel, P.  Curl: Contrastive unsupervised representations for reinforcement learning.  In _International Conference on Machine Learning_ , pp. 5639–5650. PMLR, 2020b.
  * Li et al. (2022) Li, Y., Li, S., Sitzmann, V., Agrawal, P., and Torralba, A.  3d neural scene representations for visuomotor control.  In _Conference on Robot Learning_ , pp. 112–123. PMLR, 2022.
  * Liu & Abbeel (2021) Liu, H. and Abbeel, P.  Behavior from the void: Unsupervised active pre-training.  _Advances in Neural Information Processing Systems_ , 34:18459–18473, 2021.
  * Manuelli et al. (2020) Manuelli, L., Li, Y., Florence, P., and Tedrake, R.  Keypoints into the future: Self-supervised correspondence in model-based reinforcement learning.  _arXiv preprint arXiv:2009.05085_ , 2020.
  * Martin-Brualla et al. (2021) Martin-Brualla, R., Radwan, N., Sajjadi, M. S., Barron, J. T., Dosovitskiy, A., and Duckworth, D.  Nerf in the wild: Neural radiance fields for unconstrained photo collections.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 7210–7219, 2021.
  * Mildenhall et al. (2020) Mildenhall, B., Srinivasan, P., Tancik, M., Barron, J., Ramamoorthi, R., and Ng, R.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In _European conference on computer vision_ , 2020.
  * Schwarzer et al. (2020) Schwarzer, M., Anand, A., Goel, R., Hjelm, R. D., Courville, A., and Bachman, P.  Data-efficient reinforcement learning with self-predictive representations.  _arXiv preprint arXiv:2007.05929_ , 2020.
  * Sermanet et al. (2018) Sermanet, P., Lynch, C., Chebotar, Y., Hsu, J., Jang, E., Schaal, S., Levine, S., and Brain, G.  Time-contrastive networks: Self-supervised learning from video.  In _2018 IEEE international conference on robotics and automation (ICRA)_ , pp. 1134–1141. IEEE, 2018.
  * Stooke et al. (2021) Stooke, A., Lee, K., Abbeel, P., and Laskin, M.  Decoupling representation learning from reinforcement learning.  In _International Conference on Machine Learning_ , pp. 9870–9879. PMLR, 2021.
  * Tatarchenko et al. (2016) Tatarchenko, M., Dosovitskiy, A., and Brox, T.  Multi-view 3d models from single images with a convolutional network.  In _European Conference on Computer Vision_ , pp. 322–337. Springer, 2016.
  * Todorov et al. (2012) Todorov, E., Erez, T., and Tassa, Y.  Mujoco: A physics engine for model-based control.  In _2012 IEEE/RSJ international conference on intelligent robots and systems_ , pp. 5026–5033. IEEE, 2012.
  * Wang et al. (2022) Wang, C., Luo, X., Ross, K., and Li, D.  Vrl3: A data-driven framework for visual deep reinforcement learning.  _arXiv preprint arXiv:2202.10324_ , 2022.
  * Wang et al. (2021) Wang, Q., Wang, Z., Genova, K., Srinivasan, P. P., Zhou, H., Barron, J. T., Martin-Brualla, R., Snavely, N., and Funkhouser, T.  Ibrnet: Learning multi-view image-based rendering.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 4690–4699, 2021.
  * Worrall et al. (2017) Worrall, D. E., Garbin, S. J., Turmukhambetov, D., and Brostow, G. J.  Interpretable transformations with encoder-decoder networks.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pp. 5726–5735, 2017.
  * Yamada et al. (2022) Yamada, J., Pertsch, K., Gunjal, A., and Lim, J. J.  Task-induced representation learning.  _arXiv preprint arXiv:2204.11827_ , 2022.
  * Yang & Nachum (2021) Yang, M. and Nachum, O.  Representation matters: offline pretraining for sequential decision making.  In _International Conference on Machine Learning_ , pp. 11784–11794. PMLR, 2021.
  * Yarats et al. (2020) Yarats, D., Kostrikov, I., and Fergus, R.  Image augmentation is all you need: Regularizing deep reinforcement learning from pixels.  In _International Conference on Learning Representations_ , 2020.
  * Yarats et al. (2021) Yarats, D., Fergus, R., Lazaric, A., and Pinto, L.  Mastering visual continuous control: Improved data-augmented reinforcement learning.  _arXiv preprint arXiv:2107.09645_ , 2021.
  * You et al. (2022) You, B., Arenz, O., Chen, Y., and Peters, J.  Integrating contrastive learning with dynamic models for reinforcement learning from images.  _Neurocomputing_ , 476:102–114, 2022.
  * Yu et al. (2021) Yu, A., Ye, V., Tancik, M., and Kanazawa, A.  pixelnerf: Neural radiance fields from one or few images.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 4578–4587, 2021.
  * Yu et al. (2020) Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., and Levine, S.  Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning.  In _Conference on robot learning_ , pp. 1094–1100. PMLR, 2020.
  * Zhan et al. (2022) Zhan, A., Zhao, R., Pinto, L., Abbeel, P., and Laskin, M.  Learning visual robotic control efficiently with contrastive pre-training and data augmentation.  In _2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pp. 4040–4047. IEEE, 2022.
  * Zhi et al. (2021) Zhi, S., Laidlow, T., Leutenegger, S., and Davison, A. J.  In-place scene labelling and understanding with implicit scene representation.  2021\.

##  Appendix A Algorithms

###  A.1 Model-free RL : Soft Actor-Critic

In this project, we adopt Soft Actor-Critic algorithm (Haarnoja et al., 2018)
(SAC) in all the experiments of the model-free downstream RL. SAC optimizes
stochastic policies to maximize both the expected trajectory returns and the
expected entropy of the actions. Although SAC shows promising performances on
a range of control tasks over continuous action spaces including many
benchmark tasks, it fails or suffers from data inefficiency in some visual
control tasks. To train actor network $\pi_{\phi}$ and critic networks
$Q_{\theta_{1}},Q_{\theta_{2}}$ SAC algorithm minimizes the following
objective functions

$J_{\pi}(\phi)=\mathbb{E}_{{s_{t}}\sim\mathcal{D}}\big{[}\mathbb{E}_{a_{t}\sim\pi_{\phi}}[\alpha\log(\pi_{\phi}(a_{t}|s_{t}))-\min_{i=1,2}Q_{\theta_{i}}(s_{t},a_{t})]\big{]}$
(15)

$J_{Q}(\theta_{i})=\mathbb{E}_{{s_{t},a_{t},s_{t+1},r}\sim\mathcal{D},a_{t+1}\sim\pi_{\phi}(s_{t+1})}\bigg{[}\frac{1}{2}(Q_{\theta_{i}}(s_{t},a_{t})-(r+\gamma\mathcal{T}))^{2}\bigg{]},$
(16)

where $\mathcal{D}$ denotes the replay buffer, $\alpha$ the temperature
hyperparameter, and $\gamma$ the discount factor. The target value
$\mathcal{T}$ in Eq. 16 is

$\mathcal{T}=\min_{i=1,2}Q_{\hat{\theta}_{i}}(s_{t+1},a_{t+1})-\alpha\log\pi_{\phi}(a_{t+1}|s_{t+1}).$
(17)

SAC also utilizes target networks $Q_{\hat{\theta}_{1}},Q_{\hat{\theta}_{2}}$
which are obtained as an Exponentially Moving Average (EMA) of the Q networks
($Q_{\theta_{1}},Q_{\theta_{2}}$) for better learning stability, and gradient-
based temperature tuning to determine the relative importance of the entropy,

$J(\alpha)=\mathbb{E}_{a_{t}\sim\pi_{\phi}(s_{t})}[-\alpha\log\pi_{\phi}(a_{t}|s_{t})-\alpha\bar{\mathcal{H}}].$
(18)

###  A.2 Model-based RL : Dreamer

To evaluate whether SNeRL can also be adopted in model-based RL algorithms, we
adopt Dreamer (Hafner et al., 2019). Dreamer learns the world model which
consists of the following components:

$\begin{split}&p_{\theta}(s_{t}|s_{t-1},a_{t-1},o_{t})\\\
&q_{\theta}(o_{t}|s_{t})\\\ &q_{\theta}(r_{t}|s_{t})\\\
&q_{\theta}(s_{t}|s_{t-1},a_{t-1}).\end{split}$ (19)

These components are jointly optimized to increase the variational lower
bound, which includes the following terms:

$\begin{split}\mathcal{J}_{O}^{t}&=\ln q(o_{t}|s_{t})\\\
\mathcal{J}_{R}^{t}&=\ln q(r_{t}|s_{t})\\\ \mathcal{J}_{D}^{t}&=-\beta
KL\big{(}p(s_{t}|s_{t-1},a_{t-1},o_{t})||q(s_{t}|s_{t-1},a_{t-1})\big{)}.\end{split}$
(20)

We replace the convolutional encoder of Dreamer with our feature extractor
$\Omega$ (shared encoder of SNeRL) to design an RL with a dynamics model over
the latent space with pre-trained mapping. To learn the action and value
models, Dreamer optimizes the value model $v_{\psi}$ and the action model
$q_{\phi}$ using the objectives

$\begin{split}&\max_{\phi}\mathbb{E}\bigg{(}\sum_{\tau=t}^{t+H}V_{\lambda}(s_{\tau})\bigg{)}\\\
&\max_{\psi}\mathbb{E}_{q_{\theta},q_{\phi}}\bigg{(}\sum_{\tau=t}^{t+H}\frac{1}{2}\Big{|}\Big{|}v_{\psi}(s_{\tau})-V_{\lambda}(s_{\tau})\Big{|}\Big{|}^{2}\bigg{)},\end{split}$
(21)

where H denotes the horizon, and $V_{\lambda}$ the exponentially-weight value
estimation. Refer to Dreamer (Hafner et al., 2019) for details.

###  A.3 Pseudo-code

Algorithm 1 Stage 1. pre-train multi-view encoder with SNeRL

1: Input: encoder $\Omega$, off-the-shelf feature descriptor $\hat{F}$,
offline dataset $\mathcal{D}$

2: for iteration=1,2,…,N do

3: for sample minibatch $d$ from $\mathcal{D}$ do

4: $z\leftarrow\Omega(o^{1:V},K^{1:V})$

5:
$\mathcal{L}\leftarrow\mathcal{L}_{\mathrm{RGB}}+\lambda_{sem}\mathcal{L}_{sem}+\lambda_{feat}\mathcal{L}_{feat}$

6: update the parameters of $\Omega$ to minimize $\mathcal{L}$

7: end for

8: end for

Algorithm 2 Stage 2. Downstream Model-free RL (SAC)

1: Input: total training episodes $N$, Env, environment horizon $H$, actor
network $\pi_{\phi}$, critic networks $Q_{\theta_{i=1,2}}$, target critic
networks $Q_{\hat{\theta}_{i=1,2}}$, temperature $\alpha$, replay buffer
$\mathcal{B}$, pre-trained encoder $\Omega$.

2: for iteration=1,2,…,N do

3: $o_{t=0}\leftarrow\mathtt{Env.reset()}$

4: for $t$=0,1,…,$H$-1 do

5: $a_{t}\leftarrow\pi_{\phi}(\cdot|\Omega(o_{t}))$

6: $r_{t},o_{t+1}\leftarrow\mathtt{Env.step}(a_{t})$

7: $\mathcal{B}\leftarrow\mathcal{B}\cup\\{(o_{t},a_{t},r_{t},o_{t+1})\\}$

8: end for

9: for each gradient step do

10:
$\theta_{i}\leftarrow{\theta_{i}}-\lambda_{Q}\nabla_{\theta_{i}}J_{Q}(\theta_{i})$

11: $\phi\leftarrow{\phi}-\lambda_{\pi}\nabla_{\phi}J_{\pi}(\phi)$

12: $\alpha\leftarrow{\alpha}-\lambda\nabla_{\alpha}J(\alpha)$

13: if update target critic networks then

14: $\hat{\theta}_{i}\leftarrow{\tau{\theta}_{i}}+(1-\tau)\hat{\theta}_{i}$

15: end if

16: end for

17: end for

Algorithm 3 Stage 2. Downstream Model-based RL (Dreamer)

1: Input: total training episodes $N$, update step $C$, Env, environment
horizon $T$, imagination horizon $H$, Neural network parameters
$\theta,\phi,\psi$, replay buffer $\mathcal{B}$, pre-trained encoder $\Omega$.

2: for iteration=1,2,…,N do

3: for $c$=1,…,$C$ do

4: sample data sequence
$\\{(a_{t},o_{t},r_{t})\\}_{t=0,...,H-1}\sim\mathcal{B}$

5: compute model states $s_{t}\sim p_{\theta}(s_{t}|\Omega(o_{t-1}),a_{t-1})$
and reward $q_{\theta}(r_{t}|\Omega(o_{t}))$

6: update $\theta$ using representation learning

7: imaging trajectories and compute value estimates $V_{\lambda}(s_{\tau})$

8:
$\phi\leftarrow\phi+\alpha\nabla_{\phi}\sum_{\tau=t}^{t+H}V_{\lambda}(s_{\tau})$

9:
$\psi\leftarrow\psi+\alpha\nabla_{\psi}\sum_{\tau=t}^{t+H}\frac{1}{2}||v_{\psi}(s_{\tau})-V_{\lambda}(s_{\tau})||^{2}$

10: end for

11: $o_{t=0}\leftarrow\mathtt{Env.reset()}$

12: for $t$=0,1,…,$T-1$ do

13: $a_{t}\leftarrow q_{\phi}(\cdot|\Omega(o_{t}))$

14: $r_{t},o_{t+1}\leftarrow\mathtt{Env.step}(a_{t})$

15: $\mathcal{B}\leftarrow\mathcal{B}\cup\\{(o_{t},a_{t},r_{t},o_{t+1})\\}$

16: end for

17: end for

##  Appendix B Training & Experiments Details

###  B.1 Encoder Architecture

We design the encoder architecture similar to Laskin et al. (2020b), which
consists of multiple convolutional layers and ReLU activation, but modify it
to be applicable in the multi-view observation inputs. The same encoder is
also adopted in the actor and critic to embed the pixel-level (multi-view)
observations. We demonstrate the details of the convolutional encoder with
PyTorch-like pseudo-code as below.

Algorithm 4 Multi-view Encoder Pseudocode, PyTorch-like

⬇

def encoder(x1, x2, x3, K1, K2, K3, z_dim):

"""

␣␣␣␣Multi-view␣ConvNet␣encoder

␣␣␣␣args:

␣␣␣␣␣␣␣␣B␣=␣batch_size,␣C␣=␣channels,

␣␣␣␣␣␣␣␣H,␣W␣=spatial_dims

␣␣␣␣␣␣␣␣x1,␣x2,␣x3:␣images␣from␣3␣different␣camera␣views

␣␣␣␣␣␣␣␣x1,␣x2,␣x3␣shape:␣[B,␣C,␣H,␣W]

␣␣␣␣␣␣␣␣K1,␣K2,␣K3:␣camera␣poses␣from␣3␣different␣camera␣views

␣␣␣␣␣␣␣␣K1,␣K2,␣K3␣shape:␣[B,␣4,␣4]

␣␣␣␣␣␣␣␣z_dim␣=␣latent␣dimension

␣␣␣␣"""

x = x / 255.

# c: channels, f: filters

# k: kernel, s: stride

z1 = Conv2d(c=x1.shape[1], f=32, k=3, s=2)(x1)

z1 = ReLU(z1)

z2 = Conv2d(c=x2.shape[1], f=32, k=3, s=2)(x2)

z2 = ReLU(z2)

z3 = Conv2d(c=x3.shape[1], f=32, k=3, s=2)(x3)

z3 = ReLU(z3)

for _ in range(num_layers-1):

z1 = Conv2d(c=32, f=32, k=3, s=1)(z1)

z1 = ReLU(z1)

z2 = Conv2d(c=32, f=32, k=3, s=1)(z2)

z2 = ReLU(z2)

z3 = Conv2d(c=32, f=32, k=3, s=1)(z3)

z3 = ReLU(z3)

z1 = flatten(z1)

z2 = flatten(z2)

z3 = flatten(z3)

z1 = concat([z1, K1.view(B,16)], dim=1)

z2 = concat([z2, K1.view(B,16)], dim=1)

z3 = concat([z3, K1.view(B,16)], dim=1)

z1 = Linear(z1.shape[1], z_dim)(z1)

z2 = Linear(z2.shape[1], z_dim)(z2)

z3 = Linear(z3.shape[1], z_dim)(z3)

z = concat([z1, z2, z3], dim=1).mean(dim=1)

z = Linear(z.shape[1], z_dim)(z)

z = LayerNorm(z)

z = tanh(z)

return z

###  B.2 Datasets

The offline datasets for SNeRL and baselines consist of 14400 scenes. Each
scene consists of three image observations taken from different camera views.
The observations from each camera view are represented in Figure 1 (Window-
open-v2) and Figure 5 (Soccer-v2, Hammer-v2, Drawer-open-v2). To collect the
dataset, we utilized random actions and the policies provided by Meta-world
(half-and-half mixed).

To observe how the performance of the proposed method varies with the quality
of the dataset, we further trained the SNeRL encoder with a dataset collected
by a single expert demo and random actions. Only 120 scenes of the total
scenes (14400, 120/14400$\simeq$1%) were obtained from the path of the expert
demo, and the remaining 14280 scenes were obtained by taking random actions
from one moment of the path of the expert demo. As shown in Figure 6, we
observe that the quality of the dataset slightly affects the learning
stability, but there is no dramatic performance degradation. The results show
that there would be no significant degradation in the performance of the SNeRL
if the dataset adequately covers the state space, even if the policy that
collects the offline dataset is suboptimal.

Figure 6: The performance of SNeRL with different quality of offline datasets.

###  B.3 Computational Resources

Stage 1 (pre-training encoder) in our experiments has been performed using a
single NVIDIA RTX A6000 and AMD Ryzen 2950X, and stage 2 (RL downstream tasks)
has been performed using an NVIDIA RTX A5000 and AMD Ryzen 2950X. Training the
SNeRL encoder takes 4-5 days and learning model-based RL and model-free RL
takes 1-2 days.

###  B.4 Hyperparameters

Table 1: Hyperparameters for pre-training multi-view encoder | SNeRL
---|---
Convolution layers | 4
Number of filters | 32
Non-linearity | ReLU
MLP layers for NeRF | 8
Hidden units (MLP) | 256
Number of different views | 3
NeRF learning rate | 5e-4
Number of rays per gradient step | 1024
Number of samples per ray | 64
Table 2: Hyperparameters for SAC (for SNeRL and baselines) | SAC
---|---
hidden layer | (1024, 1024)
frame stack | 2
replay buffer size | 100000
initial random steps | 1000
batch size | 128
actor learning rate | 1e-3
critic learning rate | 1e-3
$\alpha$ learning rate | 1e-4
$\beta$ for Adam optimizer (actor, critic) | 0.9
eps for Adam optimizer ($\alpha$) | 1e-08
$\beta$ for Adam optimizer ($\alpha$) | 0.5
eps for Adam optimizer ($\alpha$) | 1e-08
critic target update interval | 2
actor network update interval | 2
actor log std min, max | -10, 2
init temperature | 0.1
$\tau$ for EMA | 0.01
discount factor $\gamma$ | 0.99
Table 3: Hyperparameters for Dreamer (for SNeRL and baselines) | Dreamer
---|---
embedding size | 63
hidden / belief size | 128
state size | 30
action noise | 0.3
batch size | 32
world model learning rate | 1e-3
actor learning rate | 5e-5
value network learning rate | 5e-5
discount factor $\gamma$ | 0.99
replay buffer size | 100000
planning horizon | 15
eps for Adam optimizer | 1e-07


††thanks: The extended article has been published in Review of Scientific
Instruments. It is available online at https://dx.doi.org/10.1063/1.4979611.

# A Reconfigurable Cryogenic Platform for the Classical Control of Scalable
Quantum Computers

Harald Homulle  Stefan Visser  Bishnu Patra  QuTech, Delft University of
Technology, 2628CD Delft, The Netherlands  Giorgio Ferrari  Politecnico di
Milano, 20133 Milan, Italy  Enrico Prati  Istituto di Fotonica e
Nanotecnologie, 20133 Milan, Italy  Fabio Sebastiano  Edoardo Charbon
<EMAIL_ADDRESS>QuTech, Delft University of Technology, 2628CD Delft, The
Netherlands

(2 February 2016)

###### Abstract

Recent advances in solid-state qubit technology are paving the way to fault-
tolerant quantum computing systems. However, qubit technology is limited by
qubit coherence time and by the complexity of coupling the quantum system with
a classical electronic infrastructure.

We propose an infrastructure, enabling to read and control qubits, that is
implemented on a field-programmable gate array (FPGA). The FPGA platform
supports functionality required by several qubit technologies and can operate
physically close to the qubits over a temperature range from 4K to 300K.
Extensive characterization of the platform over this temperature range
revealed all major components (such as LUTs, MMCM, PLL, BRAM, IDELAY2) operate
correctly and the logic speed is very stable. The stability is finally
concretized by operating an integrated ADC with relatively stable performance
over temperature.

††preprint: AIP/2016

##  I Introduction

In a fault-tolerant quantum computing system based on either electron / hole
spin in semiconductors or superconductors, each qubit generally operates at
sub-1K temperatures and requires a classical loop capable of reading the state
of the qubit and controlling it based on a (localized) decision. To be
effective, such error-correcting loops need to perform a complete correction
cycle faster than the qubit decoherence time, which, depending on the qubit
technology, can vary from ms to $\mu$s Braakman  _et al._ (2013); Kawakami
_et al._ (2014); Kim  _et al._ (2014); Colless  _et al._ (2013); Colless and
Reilly (2012); Kalra  _et al._ (2014); Muhonen  _et al._ (2014, 2015); DiCarlo
_et al._ (2009); Ristè  _et al._ (2015); Barends  _et al._ (2014); Jeffrey
_et al._ (2014). Reading a qubit requires a certain signal-to-noise ratio
(SNR) and it should not interfere with the qubit state itself, whereas, to
support large systems, error-correcting loops should be physically small and
compatible with massively parallel operation.

Future error-correcting loops will likely be implemented in micro-
architectures supported by a hardware infrastructure that operates at
cryogenic temperatures, so as to ensure proximity to qubits and compactness.
Moreover, to avoid time- and energy-consuming cooling cycles, micro-
architectures supporting error correction should be programmable and/or
reconfigurable. Several technologies exist that could support logic circuits
at deep cryogenic temperatures: GaAs high electron mobility transistors
(HEMTs) Chow  _et al._ (2014), rapid single flux quantum (RSFQ) devices
Castellano  _et al._ (2006); Mukhanov (2011), and custom semiconductors Ward
_et al._ (2013); Al-Taie  _et al._ (2013); Puddy  _et al._ (2015); Prager  _et
al._ (2011), to name a few. These technologies are currently expensive and
generally not scalable, while CMOS processes can leverage a mature
infrastructure that is likely to continue improving for several more decades
Prati and Shinada (2014). To meet the reconfigurability requirement, the
obvious choice is an FPGA fabricated in a deep-submicron (DSM) CMOS process,
due to its density, versatility, and full compatibility with standard CMOS
circuits.

DSM CMOS circuits have been known to operate at deep cryogenic temperatures,
down to several hundreds of mK Ward  _et al._ (2013); Al-Taie  _et al._
(2013); Puddy  _et al._ (2015); Prager  _et al._ (2011); Prati  _et al._
(2012); Turchetti  _et al._ (2016). Cryogenic FPGAs can operate at 4K as a
quantum controller Hornibrook  _et al._ (2015); Conway Lamb  _et al._ (2016).
The Spartan 3, Spartan 6, and Artix 7 FPGAs by Xilinx were shown to operate at
4K, however the limits of operability at 4K is still largely unknown.

We explore the regime of operability of a commercial FPGA (Artix 7, Xilinx) to
enable a full error-correcting loop for fault-tolerant operation of solid-
state qubits. The resulting infrastructure is scalable and versatile,
efficiently integrated in an FPGA and operating at low power. In order to
implement such a platform, a subset of the components required in error-
correction loops were designed and tested. Moreover extensive characterisation
of the FPGA performance in cryogenic conditions was carried out. While the
target of this study is a specific spin qubit Kawakami  _et al._ (2014);
Braakman  _et al._ (2013), the platform can in principle also be used for
superconducting qubits DiCarlo  _et al._ (2009); Ristè  _et al._ (2015) and
for other qubits.

##  II An FPGA based control platform

The architecture of the control electronics of quantum devices is shown in
Figure 1; it comprises multiplexers and demultiplexers, in close proximity to
or integrated with the qubits, amplifiers, analog-to-digital converters
(ADCs), digital-to-analog converters (DACs), oscillator(s), down- and up-
converting mixers, and general-purpose digital logic.

Operating semiconductor devices at deep cryogenic temperatures is a challenge,
due to freeze-out, non-idealities in transistor I-V characteristics, and
increased mismatch Das, Lehmann, and Dzurak (2014). While freeze-out effects
at 4K are less problematic in DSM CMOS technologies due to the high levels of
doping, MOS transistors I-V characteristics exhibit a so-called ‘kink’, which
causes elevated current levels at high V${}_{\text{DS}}$ (due to a reduction
in the transistors threshold voltage V${}_{\text{t}}$). Furthermore,
hysteresis in the drain current when sweeping the drain voltage upwards or
downwards is increased at cryogenic temperatures.

To overcome these limitations, we opted for a fully digital approach by way of
FPGAs. The redesign of analog circuits adjusted to cryogenic conditions can
(at least partially) be avoided, since the large majority of the required
components can be implemented as a digital block, as shown in Figure 1.

Analog signals are converted to digital codes at the interface with the FPGA.
This data is processed digitally to measure the qubit state and to synthesize
the appropriate correction. To avoid further unnecessary signal conversions,
and thus maximizing SNR and signal integrity, control signals are generated
directly by the FPGA, whenever possible, or by DACs. Although DACs can be
integrated in the FPGA, they will not be considered in this work.

Figure 1:  Cryogenic control platform implementing the infrastructure
supporting error-correcting micro-architectures. An all-digital implementation
of the main components and integration inside the FPGA is shown. Although in
theory both red and green components can be integrated, in the following the
interest is focussed on the green parts.

Operating an FPGA almost 250K below the standard operating range, is not
trivial and it comes with several challenges.

For what concerns power dissipation, when reducing the temperature, power
dissipation budgets become increasingly stricter, down to a few watts at 4K,
due to limitations in refrigeration technologies. It imposes care in designing
custom integrated circuits, and even more so in FPGAs. To fully characterize
FPGA performance, the platform is first tested in liquid helium directly, thus
maximizing cooling power if compared to closed-loop refrigeration.

Next, we consider the physical implementation of the platform, needing ad-hoc
printed circuit boards (PCBs) hosting robust passive components and connectors
that can reliably survive several cooling cycles without performance
degradation. In our design, we used a minimal number of discrete components
certified for operation over a temperature range, wider than industrial
standards.

The most critical challenge is the operating condition and the corresponding
behaviour of the FPGA. Especially for high performance circuits, such as time-
to-digital converters (TDCs) and ADCs, the FPGA behaviour has to be well
understood.

##  III A cryogenic FPGA

A small PCB with a Xilinx Artix 7 (XC7A100T-2FTG256I) FPGA as the sole active
component was designed for cryogenic testing. The PCB is shown in Figure 2.
Besides the FPGA, various passive components, most noticeably the decoupling
capacitors, are present on board. More details on the PCB are presented in
appendix A.

\begin{overpic}[width=212.47617pt]{CryoFPGA_PCB_image}
\put(0.0,0.0){\includegraphics[width=212.47617pt]{CryoFPGA_PCB_text}}
\end{overpic} Figure 2: Cryogenic FPGA PCB. More details in appendix A.

Modern FPGAs, such as the Artix 7 from Xilinx, integrate a variety of
different components, from simple look-up tables to advanced clock managers
Xilinx (2015). In Table 1 an overview of the main FPGA components is given
together with the performance change when operating at 4K. All those
components are functional and their performance is comparable to room
temperature.

Table 1: Overview of the working elements inside the CryoFPGA operating around 4K. Component details in Xilinx (2015). The test procedure is detailed in Table 5. Module | Functional |  Performance w.r.t. RT
---|---|---
IOs | ✓ |
LVDS | ✓ |
LUTs | ✓ |  Propagation delay changes $<$ 5%, jitter increases $<$ 15% (11.8 to 13.5 ps)
CARRY4 | ✓ |  Propagation delay changes $<$ 2%, jitter increases $<$ 3% (11.5 to 11.8 ps)
BRAM | ✓ |  No corruption in 10 test sets of 80 kB
MMCM | ✓ |  Jitter reduction of roughly 20% (11.4 to 8.9 ps)
PLL | ✓ |  Jitter reduction of roughly 20% (12.2 to 9.7 ps)
IDELAYE2 | ✓ |  Delay change of up to 30%, jitter increase up to 50%
Temperature diode | ✓ |  Operating range 4K $-$ 300K

The delay change of look-up tables and carrychains is less than 5%. While this
is negligible in most applications, the change in logic speed is significant
for the performance of both ADC and TDC.

Therefore the main structure of those two circuits, i.e. the carrychain (used
as delayline) is characterized more extensively over both FPGA internal
voltage and temperature. Figure 3 shows the average delay per carryblock
versus the FPGA voltage for 300K and 4K. The delay change is significantly
larger over the voltage range then over temperature, again signifying the
performance stability over temperature. However, it also shows that the
voltage must be very stable in order to achieve TDCs and ADCs that can be
properly calibrated.

Figure 3: Delay of the carry elements in the carrychain versus FPGA voltage
(VCC_INT) over multiple temperatures.

To study self-heating effects and change in power consumption over
temperature, a circuit was designed to be able to sweep the power consumption
of the FPGA. The sweep was repeated at different temperature levels as shown
in Figure 4LABEL:sub@fig:oscillator_sweep. A significant difference is
observed in the response at the different temperatures. First, the idle power
consumption increases from 83 mW at 300K to 228 mW at 4K. Since leakage is
expected to decrease at lower temperatures, the increase of idle power is
attributed to malfunctions of the support bias circuitry.

Secondly, a change in logic efficiency occurs. Namely, the average power per
transition decreases and the power consumption starts to scale non-linearly,
while lowering temperature. The average energy used per LUT decreases from 32
to 21 $\mu$W, i.e. the energy consumed per transition in a LUT decreases from
23 to 15 pJ. This implies a decrease in dynamic power consumption of over 30%.

(a)

(b)

Figure 4: LABEL:sub@fig:oscillator_sweep Power consumption of the FPGA while
increasing the number of running oscillators in the FPGA fabric.
LABEL:sub@fig:current_sweep FPGA power consumption versus the resulting die
temperature. The coloured dots indicate (left to right): powered off FPGA,
idle FPGA, 0 active oscillators, 4500 active oscillators.

As shown in Figure 4LABEL:sub@fig:current_sweep, the FPGAs junction
temperature increases with power consumption from 4.3K (powered off FPGA) to
17.3K (at 1.25 W power consumption). While the die temperature is elevated due
to self-heating, the environment remained stable at a temperature of 4.2K
($\pm$0.1K).

From the slope we can estimate a thermal resistance of the FPGA package of
8.5K/W, which is lower than the thermal resistance at room temperature in air
(31K/W).

One of the most important elements in the quantum error-correcting control
loop is the analog-to-digital converter. Various attempts have been undertaken
to optimize ASIC ADCs for cryogenic conditions, but reaching high performance
is still challenging, as discussed before. Therefore, we opted to implement
the ADC as a completely digital building block inside the FPGA. The operating
principle is shown in Figure 5Homulle, Regazzoni, and Charbon (2015). A high
speed clock signal is output on an FPGA pin connected, through a resistor, to
an $LVDS$ capable input pin. Thanks to the resistor and the parasitic
capacitance of the $LVDS$ buffer, an $RC$-ramp is created. The conversion is
based on measuring the time required for the $RC$-ramp to cross the input
signal, as done in a single-slope converter. The $LVDS$, used as comparator,
switches as soon as the ramp voltage exceeds the input voltage and the $LVDS$
switching time is measured using a TDC. The performance of the TDC at
cryogenic temperatures is given in appendix B.

With the use of multi clock-phase interleaving and calibration features,
sampling rates, up to 1.2 GSa/s, above the clock frequency (400 MHz) are
achieved. Furthermore, thanks to calibration, the ADC performance was more
stable over temperature, process and voltage variations Visser (2015).

Figure 5: Basic principle of analog-to-digital conversion in the FPGA using an
$RC$-filter to create a ramp from the clock output signal. The crossings of
the input and the ramp are timestamped in a TDC.

The performance of a 1.2 GSa/s ADC, created from interleaving 6 different
phases of a 200 MHz clock, is summarized in Table 2. The FPGA is operating at
a slightly elevated junction temperature of 15K, while the decoupling could
not sustain the high power demand (of almost 0.9 W) at 4K to reach optimal
performance. As the ADC is calibrated at both operating temperatures, the non
linearities can be seen to be only slightly larger at cryogenic temperatures.

In Figure 6 the performance of the ADC is plotted in terms of ENOB over
different signal input frequencies for both 300K and 15K. The ENOB is lowered
with roughly one bit at cryogenic temperatures due to decreased decoupling
stability and increased IDELAYE2 jitter. Although one bit has to be sacrificed
while going down towards 15K, the performance is still in line with the
requirements for the qubit control loop, which demands a high sampling rate
rather than a high voltage resolution.

Table 2: Summary of calibrated ADC performance at 300K and 15K. The results are achieved after merging the 6 phase interleaved channels.  Temperature | [K] | 300 | 15
---|---|---|---
Sampling rate | [MHz] | 1.2 GSa/s | 1.2 GSa/s
Input range | [V] | [0.9-1.6] | [0.9-1.6]
ENOB | [bits] | 6.0 | 5.0
DNL | [LSB] | [-0.75 1.04] | [-0.85 1.04]
INL | [LSB] | [-0.36 0.52] | [-0.68 0.77]
Figure 6: Effective number of bits versus input signal frequency at room
temperature and 15K for the FPGA ADC sampling with 1.2 GSa/s. The length of
the error bars is 2$\sigma$, the variation over 10 measurements. The solid
lines are merely a guide for the eye.

##  IV Conclusions

To conclude, a cryogenic all-digital FPGA based control platform for error-
correcting loops has been implemented. The FPGA operates over a large
temperature range from 4K to room temperature, without significant change in
performance. Logic speeds showed very high stability by changing less than 5%
over the whole temperature range. All major components such as MMCM, PLL,
BRAM, IDELAY2 are adequately stable over the complete range.

Digital logic, analog-to-digital converters, temperature sensors and
oscillators have been shown to operate correctly also at base temperature,
while being integrated in the FPGA. Such set of control electronics, to be
completed as a future perspective with digital-to-analog converters, paves the
way towards practical control at cryogenic temperature of fault-tolerant solid
state quantum information processing.

## References

## References

  * Braakman  _et al._ (2013) F. R. Braakman, P. Barthelemy, C. Reichl, W. Wegscheider, and L. M. K. Vandersypen, Nat. Nanotechnol. 8, 432 (2013).
  * Kawakami  _et al._ (2014) E. Kawakami, P. Scarlino, D. R. Ward, F. R. Braakman, D. E. Savage, M. G. Lagally, M. Friesen, S. N. Coppersmith, M. A. Eriksson, and L. M. K. Vandersypen, Nat. Nanotechnol. 9, 666 (2014).
  * Kim  _et al._ (2014) D. Kim, Z. Shi, C. B. Simmons, D. R. Ward, J. R. Prance, T. S. Koh, J. K. Gamble, D. E. Savage, M. G. Lagally, M. Friesen, S. N. Coppersmith, and M. A. Eriksson, Nature 511, 70 (2014).
  * Colless  _et al._ (2013) J. I. Colless, A. C. Mahoney, J. M. Hornibrook, A. C. Doherty, H. Lu, A. C. Gossard, and D. J. Reilly, Phys. Rev. Lett. 110, 046805 (2013).
  * Colless and Reilly (2012) J. I. Colless and D. J. Reilly, Rev. Sci. Instrum. 83, 023902 (2012).
  * Kalra  _et al._ (2014) R. Kalra, A. Laucht, C. D. Hill, and A. Morello, Phys. Rev. X 4, 021044 (2014).
  * Muhonen  _et al._ (2014) J. T. Muhonen, J. P. Dehollain, A. Laucht, F. E. Hudson, R. Kalra, T. Sekiguchi, K. M. Itoh, D. N. Jamieson, J. C. McCallum, A. S. Dzurak, and A. Morello, Nat. Nanotechnol. 9, 986 (2014).
  * Muhonen  _et al._ (2015) J. T. Muhonen, A. Laucht, S. Simmons, J. P. Dehollain, R. Kalra, F. E. Hudson, S. Freer, K. M. Itoh, D. N. Jamieson, J. C. McCallum, A. S. Dzurak, and A. Morello, J. Phys. Condens. Matter 27, 154205 (2015).
  * DiCarlo  _et al._ (2009) L. DiCarlo, J. M. Chow, J. M. Gambetta, L. S. Bishop, B. R. Johnson, D. I. Schuster, J. Majer, A. Blais, L. Frunzio, S. M. Girvin, and R. J. Schoelkopf, Nature 460, 240 (2009).
  * Ristè  _et al._ (2015) D. Ristè, S. Poletto, M.-Z. Huang, A. Bruno, V. Vesterinen, O.-P. Saira, and L. DiCarlo, Nat. Commun. 6, 6983 (2015).
  * Barends  _et al._ (2014) R. Barends, J. Kelly, A. Megrant, A. Veitia, D. Sank, E. Jeffrey, T. C. White, J. Mutus, A. G. Fowler, B. Campbell, Y. Chen, Z. Chen, B. Chiaro, A. Dunsworth, C. Neill, P. O’Malley, P. Roushan, A. Vainsencher, J. Wenner, A. N. Korotkov, A. N. Cleland, and J. M. Martinis, Nature 508, 500 (2014).
  * Jeffrey  _et al._ (2014) E. Jeffrey, D. Sank, J. Y. Mutus, T. C. White, J. Kelly, R. Barends, Y. Chen, Z. Chen, B. Chiaro, A. Dunsworth, A. Megrant, P. J. J. O’Malley, C. Neill, P. Roushan, A. Vainsencher, J. Wenner, A. N. Cleland, and J. M. Martinis, Phys. Rev. Lett. 112, 190504 (2014).
  * Chow  _et al._ (2014) J. M. Chow, J. M. Gambetta, E. Magesan, D. W. Abraham, A. W. Cross, B. R. Johnson, N. A. Masluk, C. A. Ryan, J. A. Smolin, S. J. Srinivasan, and M. Steffen, Nat. Commun. 5, 4015 (2014).
  * Castellano  _et al._ (2006) M. G. Castellano, L. Grönberg, P. Carelli, F. Chiarello, C. Cosmelli, R. Leoni, S. Poletto, G. Torrioli, J. Hassel, and P. Helistö, Supercond. Sci. Technol. 19, 860 (2006).
  * Mukhanov (2011) O. A. Mukhanov, IEEE Trans. Appl. Supercond. 21, 760 (2011).
  * Ward  _et al._ (2013) D. R. Ward, D. E. Savage, M. G. Lagally, S. N. Coppersmith, and M. A. Eriksson, Appl. Phys. Lett. 102, 213107 (2013).
  * Al-Taie  _et al._ (2013) H. Al-Taie, L. W. Smith, B. Xu, P. See, J. P. Griffiths, H. E. Beere, G. A. C. Jones, D. A. Ritchie, M. J. Kelly, and C. G. Smith, Appl. Phys. Lett. 102, 243102 (2013).
  * Puddy  _et al._ (2015) R. K. Puddy, L. W. Smith, H. Al-Taie, C. H. Chong, I. Farrer, J. P. Griffiths, D. A. Ritchie, M. J. Kelly, M. Pepper, and C. G. Smith, Appl. Phys. Lett. 107, 143501 (2015).
  * Prager  _et al._ (2011) A. A. Prager, H. C. George, A. O. Orlov, and G. L. Snider, J. Vac. Sci. Technol. B Microelectron. Nanom. Struct. 29, 041004 (2011).
  * Prati and Shinada (2014) E. Prati and T. Shinada, in  _2014 IEEE Int. Electron Devices Meet._ (IEEE, 2014) pp. 1.2.1–1.2.4.
  * Prati  _et al._ (2012) E. Prati, M. De Michielis, M. Belli, S. Cocco, M. Fanciulli, D. Kotekar-Patil, M. Ruoff, D. P. Kern, D. A. Wharam, J. Verduijn, G. C. Tettamanzi, S. Rogge, B. Roche, R. Wacquez, X. Jehl, M. Vinet, and M. Sanquer, Nanotechnology 23, 215204 (2012).
  * Turchetti  _et al._ (2016) M. Turchetti, H. Homulle, F. Sebastiano, G. Ferrari, E. Charbon, and E. Prati, Appl. Phys. Express 9, 014001 (2016).
  * Hornibrook  _et al._ (2015) J. M. Hornibrook, J. I. Colless, I. D. Conway Lamb, S. J. Pauka, H. Lu, A. C. Gossard, J. D. Watson, G. C. Gardner, S. Fallahi, M. J. Manfra, and D. J. Reilly, Phys. Rev. Appl. 3, 024010 (2015).
  * Conway Lamb  _et al._ (2016) I. D. Conway Lamb, J. I. Colless, J. M. Hornibrook, S. J. Pauka, S. J. Waddy, M. K. Frechtling, and D. J. Reilly, Rev. Sci. Instrum. 87, 014701 (2016).
  * Das, Lehmann, and Dzurak (2014) K. Das, T. Lehmann, and A. S. Dzurak, IEEE Trans. Circuits Syst. I Regul. Pap. 61, 2816 (2014).
  * Xilinx (2015) Xilinx, “7 series FPGAs overview,”  (2015).
  * Homulle, Regazzoni, and Charbon (2015) H. Homulle, F. Regazzoni, and E. Charbon, in  _FPGA ’15_ (ACM Press, New York, New York, USA, 2015) pp. 228–235.
  * Visser (2015) S. Visser,  _A 1 GSa/s Deep Cryogenic, Reconfigurable Soft-core FPGA ADC for Quantum Computing Applications_ , Master thesis, TU Delft (2015).
  * Teyssandier and Prêle (2010) F. Teyssandier and D. Prêle, “Commercially Available Capacitors at Cryogenic Temperatures,”  (2010).
  * Kirschman (2014) R. Kirschman, WOLTE11 (2014).
  * Favi and Charbon (2009) C. Favi and E. Charbon, in  _FPGA ’09_ (2009) p. 113.
  * Homulle (2014) H. Homulle,  _Development of a Multichannel TCSPC System in a Spartan 6 FPGA_ , Master thesis, TU Delft (2014).

##  Appendix A Cryogenic FPGA design & test

The cryogenic test set-up was designed to be both functional and simple. We
thus excluded all those components that might fail (and negatively influence
the FPGA) and made the FPGA the sole active component under test.

###  A.1 PCB design

The design of the cryogenic FPGA PCB focusses on simplicity while allowing the
possibility of making all relevant tests with the PCB. The PCB was limited in
size to fit in the test set-up (see Subsection A.2) and is shown in Figure 2.

The PCB was fabricated with standard FR4 dielectric and 4 layers (one ground
plane and three signal / power planes). The FPGA (Xilinx Artix 7 series), the
high-frequency connectors and some resistors were placed on the front side.
The back-side hosts only decoupling capacitors. A detailed list of the
components on the PCB is presented in Table 3.

Table 3: Bill of Materials (simplified) of the FPGA PCB. Component | Manufacturer | Part
---|---|---
FPGA | Xilinx | XC7A100T-2FTG256I
Capacitors | Kemet | C-series (Ceramic (NP0/COG, X8L))
Capacitors | Kemet | T-series (Tantalum)
Capacitors | Panasonic | TQC-series (Polymer)
Resistors | Panasonic | ERA-sries (Metal film)
Connectors | Amphenol | MMCX
Connectors | Samtec |  8$\times$2 female connector

In particular, we chose the FPGA XC7A100T-2FTG256I due to its small size, low
power consumption (28 nm process node) and industrial grade (higher standard
temperature range down to -40∘C). It features over $100k$ logic cells (in
$16k$ slices), up to 170 user IOs and measures 17$\times$17 mm. Capacitors
were chosen after selecting the best materials for cryogenic conditions and
their availability in various capacitor values. Low permittivity materials,
such as ceramic NP0/COG materials, are generally well behaved at 4K
Teyssandier and Prêle (2010); Kirschman (2014). However, since they are only
available in 0.47 $\mu$F format, tantalum polymer was used for the bigger
capacitors. As the FPGA is powered over long cables (over 4 meters), special
care was taken in minimizing the total capacitors effective series resistance
and inductance (ESR and ESL). ESR effects the capacitors ability to quickly
source and sink current, especially in fast current switching applications.
According to Teyssandier and Prêle (2010) not only the capacitance can drop
significantly at cryogenic temperatures, but also the ESR can increase up to
1.000 times, stressing the importance of selecting the appropriate capacitor
types for operation in a cryogenic environment. For high frequency signals
MMCX connectors were selected for their small size.

A detailed summary of the used connections to the FPGA board is given in Table
4. For communication both JTAG (programming) and UART were used, occupying
most of the user IO pins available on the board. For the power supplies,
additional cables were added to avoid a large $IR$ drop in the shielded
cables. Special care has been taken to ensure a proper ground of the complete
set-up, by shorting the cable shield terminals both at the instrument side
(room temperature) and at the PCB side (deep cryogenic temperature).

Table 4: Connections towards the FPGA PCB.

Connector | Occupied pins | Purpose
---|---|---
Shields |  20∗ | Ground
DC | 3 | Voltages (VCC_INT, VCC_O and VCC_AUX)
DC | 4 | External temperature sensor (DT-670)
8$\times$2 | 4 | JTAG (TDI, TDO, TMS, TCK)
8$\times$2 | 3 | UART (RX, TX, CTS)
8$\times$2 | 5 | Voltages + ground
8$\times$2 | 2 | Internal temperature diode
8$\times$2 | 2 | Debug
MMCX | 2 | Clock (differential)
MMCX | 1 | ADC input signal
MMCX | 1 | High frequency debug

∗ 16+4 coaxial shields

The measurement set-up is depicted in Figure 7. For the power, two power
supplies from Aim-TTI Instruments (EL302RD) were used, generating 1, 1.8 and
2.5 V. Programming and communication with the FPGA was done with a Digilent
JTAG controller (XUP USB-JTAG) and an UART-USB dongle (FTDI UMFT234XD),
respectively. The clock signal was provided differentially from an SP605
(Xilinx Spartan 6 evaluation kit) and was set to 100 MHz. Additionally a Rohde
& Schwarz HMF2550 signal generator has been used to provide the ADC test
input. For accurate temperature measurements, a Keithley source meter unit
(2636B), was added to the system. It provides a precision current bias for the
FPGAs internal temperature-measuring diode and for an external reference
sensor (Lake Shore Cryotronics, DT-670 silicon diode) mounted in thermal
contact to the FPGA. Both diodes were biased with a current of 10 $\mu$A and
the diode voltage is read as a function of temperature.

Figure 7: Overview of the equipment used outside the cryogenic environment to
control and operate the FPGA. The ground is not indicated, but is common for
all equipment and the PCB.

###  A.2 Test set-up

In order to test the FPGA in deep cryogenic conditions, a set-up was built
which uses a dip-stick to immerse the PCB in liquid helium. A schematic cross-
section of the set-up is shown in Figure 8. The dip-stick consists of a steel
pipe (2 m), a break-out box for cables placed on the top end and a sample
holder at the low end of the pipe. The pipe limits the size of the PCB
(diameter of 1.5 inch) and the number of cables. In total, 16 shielded cables
(standard RG174/U coaxial wires with a 0.5 mm diameter core) are used for low
frequency connections, 4 high frequency cables are connected to the MMCX and
up to 9 additional cables for the temperature sensor and power supplies.

Figure 8: Schematic depiction of our cryogenic test set-up. The sample is
mounted on a long steel pipe which is shifted into a helium dewar to reach the
liquid helium. The connections to the sample are made with long (standard)
cables that are attached to a break-out box on the top. There are 16 shielded
connections to a 8$\times$2 pin header, 4 shielded connections to high
frequency MMCX connectors and up to 9 additional DC wires for a temperature
sensor and the various supply voltages.

As the pipe is able to move up and down through a vacuum flange, the
temperature can be shifted from 4K in the liquid phase to around 200K in
helium vapours at the top of the dewar. As the sample is immersed in either
the gas or the liquid helium, the cooling process is relatively fast and the
cooling power relatively high compared to a vacuum refrigerator.

##  Appendix B Cryogenic FPGA results

Table 5: Overview of the working elements inside the CryoFPGA operating around 4K. Module | Functional |  Test |  Performance w.r.t. RT
---|---|---|---
IOs | ✓ |  |
LVDS | ✓ |  |
LUTs | ✓ |  LUTs connected to form oscillator of approximately 100 MHz at RT |  Oscillation frequency changes $<$ 5%, jitter increases $<$ 15% (11.8 to 13.5 ps)
CARRY4 | ✓ |  Carrychains connected to form oscillator of approximately 100 MHz at RT |  Oscillation frequency changes $<$ 2%, jitter increases $<$ 3% (11.5 to 11.8 ps)
BRAM | ✓ |  Transfers of 8 kB (write & read) |  No corruption in 10 test sets of 80 kB
MMCM | ✓ |  100 MHz differential input clock multiplied by 10 and divided by 20 to 50 MHz single ended output |  Jitter reduction of roughly 20% (11.4 to 8.9 ps)
PLL | ✓ |  100 MHz differential input clock multiplied by 10 and divided by 20 to 50 MHz single ended output |  Jitter reduction of roughly 20% (12.2 to 9.7 ps)
IDELAYE2 | ✓ |  IDELAYE2 elements connected to form a tunable oscillator (output frequency variable 13-70 MHz) |  Delay change of up to 30%, jitter increase up to 50%

###  B.1 Extended evaluation of FPGA functionality & performance

A test procedure was defined to sequentially test the most vital FPGA
components. As each component requires a different test procedure, different
VHDL programs were designed and programmed on the FPGA through the JTAG
interface. It should be noted that programming was working over the complete
temperature range, allowing us to test the different components separately in
the cryogenic environment. However the communication speed had to be set below
1 MHz, for both JTAG and UART, due to not proper termination of the coaxial
lines causing too strong reflections at higher frequencies.

In Table 5 an overview of the main FPGA components is given together with the
test procedure and the performance change when cooling to 4K.

Figure 9 shows that the change in oscillation frequency of two different
oscillators (both tuned to be around 100 MHz) implemented with either look-up
tables and carrychains is below 5%.

Figure 9: Change in oscillation frequency of oscillators build from look-up
tables and carrychains respectively over a temperature sweep from 4K up to
300K.

The operating voltage range of the FPGA changes significantly over
temperature. The range is wider at room temperature (0.85 V - 1.1 V) than the
specified range (0.95 V - 1.05 V). At low temperature, it reduces
significantly on both ends (0.92 V - 1.02 V).

The FPGAs internal temperature diode was characterized against a precise
external reference (Lake Shore Cryotronics, DT-670 silicon diode), which is
calibrated down to 1.4K. During the test, the FPGA was switched off
completely, to minimize self-heating effects.

Figure 10: FPGA diode calibration against external reference diode, both
diodes biased with 10 $\mu$A. Over 3.000 measurement points were used in order
to recreate the diode curve. As some hysteresis was present in the measurement
data, the curve is averaged out between temperature decrease and increase.

The FPGA diode was calibrated with a 10 $\mu$A current and the voltage is
stored as a function of the reference temperature (acquired by the reference
sensor) as shown in Figure 10.

Idle power consumption for the various supplies at different temperatures is
given in Table 6.

Table 6: Idle FPGA power consumption at different temperatures. Bias voltages for VCC_INT, VCC_AUX and VCC_O are 1, 1.8 and 2.5 V respectively.  Temperature | VCC_INT | VCC_AUX | VCC_O | Power consumption
---|---|---|---|---
[K] |  [mA] |  [mA] |  [mA] |  [mW]
300 | 24 | 26 | 5 | 83
40 | 19 | 69 | 4 | 153
4 | 20 | 110 | 4 | 228

###  B.2 Time-to-digital converters

As the FPGA is fully functional (even at 4K) and performance is comparable to
that of room temperature, complex and high-speed circuits were implemented and
tested. As we do not expect any (negative) change in performance for normal
digital logic operations, some more demanding circuits were implemented. A TDC
was implemented using the carrychain with 200 delay stages (50 Carry4 blocks)
and running at 400 MHz. The sampling clock was generated by a mixed mode clock
manager (MMCM) module on the FPGA together with a 100 and 200 MHz clock for
the remainder of the logic. Following the TDC, a histogrammer was implemented
to store the generated data in a compressed format (16 bits of data for each
of the 200 bins). Each histogram bin is incremented when the corresponding
time stamp is measured in the TDC. The histogram was consecutively read out
through the UART module and transferred to the host. The basic structure of
the TDC is described by Favi and Charbon (2009); Visser (2015). The
histogrammer units are detailed in Homulle (2014).

This circuit employs all basic components: LUTs, carrychains, block RAM, clock
managers and IO delays. It not only validates the conclusions from Section
III, but shows all components can work together reliably at low temperatures
with TDC performance comparable to room temperature.

Table 7: Summary of (uncalibrated) TDC performance at 300K and 4K. The FPGA was operating at the base voltage, VCC_INT of 1 V.  Temperature | [K] | 300 | 4
---|---|---|---
Sampling rate | [MHz] | 400 | 400
Resolution | [ps] | 20.1 | 20
DNL | [LSB] | [-1 4.6] | [-1 2.8]
INL | [LSB] | [-2.6 3.3] | [-3.9 2.3]
Jitter (average) | [LSB] | 0.9 | 1.6

The most important parameters for a given TDC are its non linearity, jitter
performance, resolution and sampling rate. A summary of the uncalibrated TDC
performance at 300K and 4K is presented in Table 7. The differential non
linearity (DNL) and integral non linearity (INL) were obtained through a
density test; over 30 million samples were used (from 400 integration runs).
The extracted DNL and INL are shown in Figure 11LABEL:sub@fig:cryo_tdc_a and
LABEL:sub@fig:cryo_tdc_b respectively. Both room temperature and cryogenic non
linearities are comparable. On contrary, the average jitter increases to 0.7
LSB (1$\sigma$)at cryogenic temperatures, thus w.r.t. room temperature. The
jitter increase can be attributed mainly to the increase in jitter of the
IDELAYE2 component as noted previously in Table 5.

(a)

(b)

(c)

Figure 11:  Performance comparison of a time-to-digital converter operating in
300K and 4K. Both LABEL:sub@fig:cryo_tdc_a differential non linearity and
LABEL:sub@fig:cryo_tdc_b integral non linearity are obtained from a density
test with over 30 million counts in the histogram. LABEL:sub@fig:cryo_tdc_c
jitter performance (1$\sigma$) of the TDC, captured with a synchronised clock
shifted through the carrychain. All results obtained without calibration of
the TDC.