# An open-source voice type classifier for child-centered daylong recordings

###### Abstract

Spontaneous conversations in real-world settings such as those found in child-
centered recordings have been shown to be amongst the most challenging audio
files to process. Nevertheless, building speech processing models handling
such a wide variety of conditions would be particularly useful for language
acquisition studies in which researchers are interested in the quantity and
quality of the speech that children hear and produce, as well as for early
diagnosis and measuring effects of remediation. In this paper, we present our
approach to designing an open-source neural network to classify audio segments
into vocalizations produced by the child wearing the recording device,
vocalizations produced by other children, adult male speech, and adult female
speech. To this end, we gathered diverse child-centered corpora which sums up
to a total of 260 hours of recordings and covers 10 languages. Our model can
be used as input for downstream tasks such as estimating the number of words
produced by adult speakers, or the number of linguistic units produced by
children. Our architecture combines SincNet filters with a stack of recurrent
layers and outperforms by a large margin the state-of-the-art system, the
Language ENvironment Analysis (LENA) that has been used in numerous child
language studies.

Index Terms: Child-Centered Recordings, Voice Type Classification, SincNet,
Long Short-Term Memory, Speech Processing, LENA

##  1 Introduction and related work

In the past, language acquisition researchers’ main material was short
recordings [1] or times of in-person observations [2]. However, investigating
the language phenomenon in this manner can lead to biased observations,
potentially resulting in divergent conclusions [3]. More recently, technology
has allowed researchers to efficiently collect and analyze recordings over a
whole day. By the combined use of a small wearable device and speech
processing algorithms, one can get meaningful insights of children’s daily
language experiences. While daylong recordings are becoming a central tool for
studying how children learn language, a relatively small effort has been made
to propose robust and bias-free speech processing models to analyze such data.
It may however be noticed that some collaborative works that benefit both the
speech processing and the child language acquisition communities have been
done. In particular, we may cite Homebank, an online repository of daylong
child-centered audio recordings [4] that allow researchers to share data more
easily. Some efforts have also been made to gather state-of-the-art pretrained
speech processing models in DiViMe [5], a user-friendly and open-source
virtual machine. Challenges and workshops using child-centered recordings [6,
7], also attracted the attention of the speech processing community.
Additionally, the task of classifying audio events has often been addressed in
the speech technology literature. In particular, the speech activity detection
task [8] or the acoustic event detection problem [9, 10] are similar to the
voice type classification task we address in this paper.

Given the lack of open-source and easy-to-use speech processing models for
treating child-centered recordings, researchers have been relying, for the
most part, on the Language ENvironment Analysis (LENA) software [11] to
extract meaningful information about children’s language environment. This
system will be introduced in more detail in the next section.

KCHIOCHMALFEMSPEECHSincNetLSTMLSTMFFFFFF

Figure 1: Proposed architecture. The network takes the raw waveform of a 2s
audio chunk as input and passes on to SincNet [12]. The low-level
representations learnt by SincNet are then fed to a stack of two bi-
directional LSTMs, followed by three feed-forward layers. The output layer is
activated by a sigmoid function that returns a score ranging between 0 and 1
for each of the classes.

###  1.1 The LENA system

The LENA system consists of a small wearable device combined with an automated
vocal analysis pipeline that can be used to study child language acquisition.
The audio recorder has been designed to be worn by young children as they go
through a typical day. In the current LENA system, after a full day of audio
has been been captured by the recorder, the audio files are transferred to a
cloud and analyzed by signal processing models. These latter have been trained
on 150 hours of proprietary audio collected from recorders worn by American
English-speaking children. The speech processing pipeline consists of the
following steps [11, 13, 14]:

  1. 1

First, the audio is segmented into mutually exclusive categories that include:
key child vocalizations (i.e., vocalizations produced by the child wearing the
recording device), adult male speech, adult female speech, other child speech,
overlapping sounds, noise, and electronic sounds.

  2. 2

The key child vocalization segments are further categorized into speech and
non-speech sounds. Speech encompasses not only words, but also babbling and
pre-speech communicative sounds (such as squeals and growls). Child non-speech
sounds include emotional reactions such as cries, screams, laughs and
vegetative sounds such as breathing and burping.

  3. 3

A model based on a phone decoder estimates the number of words in each adult
speech segment.

  4. 4

Further analyses are performed to detect conversational turns, or back and
forth alternations between the key child and an adult.

The LENA system has been used in multiple studies covering a wide range of
expertise including a vocal analysis of children suffering from hearing loss
[15], the assessment of a parent coaching intervention [16], and a study of
autism spectrum disorders [17]. An extensive effort has been made to assess
the performance of the LENA speech processing pipeline [18, 19, 20].

Despite its wide use in the child language community, LENA imposes several
limiting factors to scientific progress. First, as their software is closed
source, there is no way to build upon their models to improve performance, and
we cannot be certain about all design choices and their potential impact on
performance. Moreover, since their models have been trained only on American
English-speaking children recorded with one specific piece of hardware in
urban settings, the model might potentially be overfit to these settings, with
a loss of generalization to other languages, cultures, and recording devices.

###  1.2 The present work

Our work aims at proposing a viable open-source alternative to LENA for
classifying audio frames into segments of key child vocalizations, adult male
speech, adult female speech, other child vocalizations, and silence. The
general architecture is presented in 2.1. Additionally, we gathered multiple
child-centered corpora covering a wide range of conditions to train our model
and compare it against LENA. This data set is described in further details in
2.2.

##  2 Experiments

###  2.1 End-to-end voice type classification

The voice type classification problem can be described as the task of
identifying voice signal sources in a given audio stream. It can be tackled as
a multi-label classification problem where the input is the audio stream
divided into $N$ frames $\bm{S}=\\{s_{1},s_{2},\ldots,s_{N}\\}$ and the
expected output is the corresponding sequence of labels
$\bm{y}=\\{\bm{y_{1}},\bm{y_{2}},\ldots,\bm{y_{N}}\\}$ where each $\bm{y_{i}}$
is of dimension $K$ (the number of labels) with $y_{i,j}=1$ if the $j^{th}$
class is activated, $y_{i,j}=0$ otherwise. Note that, in the multi-label
setup, multiple classes can be activated at the same time.

At training time, fixed-length sub-sequences made of multiple successive
frames, are drawn randomly from the training set to form mini-batches of size
$M$.

As illustrated in Figure 1, these fixed-length sub-sequences are processed by
a SincNet [12] that aims at learning meaningful filter banks specifically
customized to solve the voice type classification task. These low-level signal
representations are then fed into a stack of bi-directional long short-term
memory (LSTM) layers followed by a stack of feed-forward (FF) layers. Finally,
the sigmoid activation function is applied to the final output layer of
dimension $K$ so that each predicted score $\hat{y}_{i,j}$ consists of a
number ranging between 0 and 1. The network is trained to minimize the binary
cross-entropy loss:

$\mathcal{L}=-\dfrac{1}{KM}\sum_{i=1}^{M}\sum_{j=1}^{K}y_{i,j}\log\hat{y}_{i,j}+\left(1-y_{i,j}\right)\log\left(1-\hat{y}_{i,j}\right)$
(1)

At test time, audio files are processed using overlapping sliding sub-
sequences of the same length as the one used in training. For each time step
$t$, and each class $j$, this results in several overlapping sequences of
prediction scores, which are averaged to obtain the final score for class $j$.
Finally, time steps with prediction scores greater than a tunable threshold
$\sigma_{j}$ are marked as being activated for the class $j$.

Our use case considers $K=5$ different classes or sources which are:

  * •

KCHI, for key-child vocalizations, i.e., vocalizations produced by the child
wearing the recording device

  * •

OCH, for all the vocalizations produced by other children in the environment

  * •

FEM, for adult female speech

  * •

MAL, for adult male speech

  * •

SPEECH, for when there is speech

As the LENA voice type classification model is often used to sample audio in
order to extract segments containing the most speech, it appeared to us that
it was useful to consider a class for speech segments produced by any type of
speaker. Moreover, in our data set, some of the segments have been annotated
as UNK (for unknown) when the annotator was not certain of which type of
speaker was speaking (See Table 1). Considering the SPEECH class allows our
model to handle these cases.

One major design difference with the LENA model is that we chose to treat the
problem as a multi-label classification task, hence multiple classes can be
activated at the same time (e.g., in case of overlapping speech). In contrast,
LENA treats the problem as a multi-class classification task where only one
class can be activated at a given time step. In the case of overlapping
speech, LENA model returns the OVL class (which is also used for overlap
between speech and noise). More details about the performance obtained by LENA
on this class can be found in [18].

###  2.2 Datasets

Table 1: Description of the BabyTrain data set. Child-centered corpora
included cover a wide range of conditions (including different languages and
recording devices). ACLEW-Random is kept as a hold-out data set on which LENA
and our model are compared. DB correpond to datasets that can be found on
Databrary, HB the ones that can be found on Homebank.

|  |  |  |  | Cumulated utterance duration
---|---|---|---|---|---
Corpus | Access | LENA-recorded? | Language | Tot. Dur. | KCHI | OCH | MAL | FEM | UNK
BabyTrain |
ACLEW-Starter | mixture (DB) | mostly | Mixture | 1h30m | 10m | 5m | 6m | 20m | 0m
Lena Lyon | private (HB) | yes | French | 26h51m | 4h33m | 1h14m | 1h9m | 5h02m | 1h0m
Namibia | upon agreement | no |  Ju$|$’hoan | 23h44m | 1h56m | 1h32m | 41m | 2h22m | 1h01m
Paido | public (HB) | no | Greek, Eng., Jap. | 40h08m | 10h56m | 0m | 0m | 0m | 0m
Tsay | public (HB) | no | Mandarin | 132h02m | 34h07m | 2h08m | 10m | 57h31m | 28m
Tsimane | upon agreement | mostly | Tsimane | 9h30m | 37m | 23m | 11m | 28m | 0m
Vanuatu | upon agreement | no | Mixture | 2h29m | 12m | 5m | 5m | 9m | 1m
WAR2 | public (DB) | yes | English (US) | 50m | 14m | 0m | 0m | 0m | 9m
Hold-out set |
ACLEW-Random | private (DB) | yes | Mixture | 20h | 1h39m | 45m | 43m | 2h48m | 0m

In order to train our model, we gathered multiple child-centered corpora data
[21, 22, 23, 24, 25, 26, 27, 28, 29, 30] drawn from various child-centered
sources, several of which were not daylong. Importantly, the recordings used
for this work cover a wide range of environments, conditions and languages and
have been collected and annotated by numerous field researchers.

We will refer to this data set as BabyTrain, of which a broad description is
given in Table 1.

We split the BabyTrain data set into a training, development and test sets,
containing approximately 60%, 20% and 20% of the audio duration respectively.
We applied this split such that files associated to a given key child were
included in only one of the three sets, splitting children up within each of
the 8 corpora of BabyTrain. The only exception was WAR2, too small to be
divided, and therefore put in the training set in its entirety.

In order to ensure that our models generalize well enough to unseen data, and
to compare the performance with the LENA system, we kept the ACLEW-Random as a
hold-out data set.

###  2.3 Evaluation metric

For each class, we use the F-measure between precision and recall, such as
implemented in pyannote.metrics [31] to evaluate our systems:

$\text{F-measure}=\dfrac{2\times\text{precision}\times\text{recall}}{\text{precision}\times\text{recall}}$

where $\text{precision}=\text{tp}/(\text{tp}+\text{fp})$ and
$\text{recall}=\text{tp}/(\text{tp}+\text{fn})$ with:

  * •

tp the duration of true positives

  * •

fp the duration of false positives

  * •

fn the duration of false negatives

We select our models by averaging the F-measure across the 5 classes. Note
that these 5 metrics have been computed in a binary fashion, where the
predictions of our model for a given class were compared to all reference
speaker turns such as provided by the human annotations (no matter if the
latter were overlapping or not). In diarization studies, the choice of a
collar around every reference speaker turns is often made to account for
inaccuracies in the reference labels. We chose not to do so, consequently all
numbers reported in this paper can be considered as having a collar equal to
0.

###  2.4 Implementation details

Figure 1 illustrates the broad architecture used in all experiments. For
SincNet, we use the configuration proposed by the authors of the original
paper [12]. All LSTMs and inner feed-forward layers have a size of 128 and use
_tanh_ activations. The last feed-forward layer uses a sigmoid activation
function.

Data augmentation is applied directly on the waveform using additive noise
extracted from the MUSAN database [32] with a random target signal-to-noise
ratio ranging from 5 to 20 dB. The learning rate is set up by a cyclical
scheduler [33], each cycle lasting for 1.5 epoch.

Since we address the problem in a multi-label classification fashion, multiple
classes can be activated at the same time. For the reference turns, the SPEECH
class was considered to be activated whenever one (or more) of the KCHI, CHI,
FEM, MAL or UNK class was activated. The UNK class (see Table 1) corresponds
to cases when the human annotator could hear that the audio contained speech
or vocalizations, without being able to identify the voice source. This class
does contribute in activating the SPEECH class, but our model does not return
a score for it.

###  2.5 Evaluation protocol

For all experiments, the neural network is trained for 10 epochs
(approximately 2400 hours of audio) on the training set. The development set
is used to choose the actual epoch and thresholds $\\{\sigma_{j}\\}_{j=1}^{K}$
that maximizes the average F-measure between precision and recall across
classes.

We report both the in-domain performance (computed on the test set of
BabyTrain) and the out-of-domain performance (computed on the hold-out set,
ACLEW-Random). We compare our model with the LENA system on the hold-out set.

Table 2: In-domain performance in terms of F-measure between precision and recall. The ”Ave.” column represents the F-measure averaged across the 5 classes. Numbers are computed on the test set from which the Paido corpora has been removed. Performance on the development set are reported using small font size. We report two variants, the first one is based on 5 binary models trained separately on each of the class, the second one consists of a single model trained in a multitask fashion Train/Dev. | System | KCHI | OCH | MAL | FEM | SPEECH | Ave.
---|---|---|---|---|---|---|---
without Paido | binary |  76.1 79.2 |  22.5 28.7 |  37.8 38.9 |  80.2 83.5 |  88.0 89.3 |  60.9 63.9
with Paido | multi |  75.8 78.7 |  25.4 30.3 |  40.1 43.2 |  82.3 83.9 |  88.2 90.1 |  62.3 65.2
without Paido | multi |  77.3 80.6 |  25.6 30.6 |  42.2 43.7 |  82.4 84.2 |  88.4 90.3 |  63.2 65.9

##  3 Results

We evaluate two different approaches, one consisting of 5 models trained
separately for each of the class (referred as binary), and one consisting of a
single model trained jointly on all the classes (referred as multitask). At
first, both in the binary and the multitask scenario, architectures shared the
same set of hyper-parameters. Only the dimension of the output layer differed.
Results indicated that multitask approaches were significantly better than
binary ones, which seems to show that sharing weights during training helps
better learn the boundaries between the different classes.

To further improve the performance of our model, we tried multiple sets of
hyper-parameters (varying the number of filters, the number of LSTM and FF
layers, and their size). However, no significant differences have been
observed among the different architectures. The retained architecture consists
of 256 filters of length L = 251 samples, 3 LSTM layers of size 128, and 2 FF
layers of size 128.

Finally, removing Paido from the training and development set led to
improvements on the other test domains, as well as the hold-out set, while the
performance on the Paido domain remained high. Indeed, we observed a F-measure
of 99 on the KCHI class for the model trained with Paido as compared to 89 for
the model trained without it. This difference can be explained by a higher
amount of false alarms returned by the model trained without it. The Paido
domain is quite far from our target domain since it consists of laboratory
recordings of words in isolation spoken by children, and thus it is reasonable
to think that removing it leads to better models.

###  3.1 In-domain performance

Since LENA can only be evaluated in data collected exclusively with the LENA
recording device and BabyTrain contains a mixture of devices, we do not report
on LENA in-domain performance. Additionally, comparing performance on a domain
that would have been seen during the training by our model but not by LENA
would have unfairly advantaged us.

Table 2 shows results in terms of F-measure between precision and recall on
the test set for each of the 5 classes. The best performance is obtained for
the KCHI, FEM, and SPEECH classes, which correspond to the 3 classes that are
the most present in BabyTrain (See Table 1). Performance is lower for the OCH
class and MAL classes, with an F-measure of 25.6 and 42.2 respectively, most
likely due to the fact that these two classes are underrepresented in our data
set. The F-measure is lowest for the OCH class. In addition to being
underrepresented in the training set, utterances belonging to the OCH class
can easily be confused with KCHI utterances since the main feature that
differentiates these two classes is the average distance to the microphone.

The multitask model consistently outperforms binary ones. When training in a
multitask fashion, increases are higher for the lesser represented classes,
namely OCH and MAL. Additionally, removing Paido leads to an improvement of
0.9 in terms of average F-measure on the other domains.

###  3.2 Performance on the hold-out data set

Table 3: Performance on the hold-out data set in terms of F-measure between
precision and recall. ”Ave.” column represents the F-measure averaged across
the 5 classes. The hold-out data set has never been seen during the training,
neither by LENA, nor by our model.

Train/Dev. | System | KCHI | OCH | MAL | FEM | SPEECH | Ave.
---|---|---|---|---|---|---|---
english (USA) | LENA | 54.9 | 28.5 | 37.2 | 42.6 | 70.2 | 46.7
without Paido | binary | 67.6 | 23.0 | 31.6 | 62.6 | 77.6 | 52.5
with Paido | multi | 66.4 | 19.9 | 39.9 | 63.0 | 77.6 | 53.3
without Paido | multi | 68.7 | 33.2 | 42.9 | 63.4 | 78.4 | 57.3

Table 3 shows performance of LENA, our binary variant, and our multitask
variant on the hold-out data set. As observed on the test set, the model
trained in a multi-task fashion shows better performance than the models
trained in a binary fashion. Removing Paido leads to a performance increase of
4 points on the average F-measure.

Turning to the comparison with LENA, both the LENA model and our model show
lower performance for the rarer OCH and MAL classes. Our model outperforms the
LENA model by a large margin. We observe an absolute improvement in terms of
F-measure of 13.8 on the KCHI class, 4.6 on the OCH class, 5.6 on the MAL
class, 20.8 on the FEM class, and 8.1 on the SPEECH class. This leads to an
absolute improvement of 10.6 in terms of F-measure averaged across the 5
classes.

##  4 Reproducible research

All the code has been implemented using pyannote.audio [34], a python open-
source toolkit for speaker diarization. Our own code, easy-to-use scripts to
apply the pretrained model can be found on our GitHub repository
111https://github.com/MarvinLvn/voice-type-classifier, which also includes
confusion matrices and a more extensive comparison with LENA. As soon as
required agreements will be obtained, we plan to facilitate access to the data
by hosting them on Homebank.

##  5 Conclusion

In this paper, we gathered recordings drawn from diverse child-centered
corpora that are known to be amongst the most challenging audio files to
process, and proposed an open-source speech processing model that classifies
audio segments into key child vocalizations, other children vocalizations,
adult male speech, and adult female speech. We compared our approach with a
homologous system, the LENA software, which has been used in numerous child
language studies. Our model outperforms LENA by a large margin and will, we
hope, lead to more accurate observations of early linguistic environments. Our
work is part of an effort to strengthen collaborations between the speech
processing and the child language acquisition communities. The latter have
provided data as that used here, as well as interesting challenges [6, 7]. Our
paper is an example of the speech processing community returning the favor by
providing robust models that can handle spontaneous conversations in real-
world settings.

## References

  * [1] B. Hart and T. R. Risley, _Meaningful differences in the everyday experience of young American children._ Paul H Brookes Publishing, 1995.
  * [2] G. Wells, “Describing children’s linguistic development at home and at school,” _British Educational Research Journal_ , vol. 5, no. 1, 1979.
  * [3] E. Bergelson, A. Amatuni, S. Dailey, S. Koorathota, and S. Tor, “Day by day, hour by hour: Naturalistic language input to infants,” _Developmental science_ , vol. 22, no. 1, 2019.
  * [4] M. VanDam, A. Warlaumont, E. Bergelson, A. Cristia, M. Soderstrom, P. De Palma, and B. MacWhinney, “HomeBank: An Online Repository of Daylong Child-Centered Audio Recordings,” _Seminars in Speech and Language_ , vol. 37, no. 02, 2016.
  * [5] A. Le Franc, E. Riebling, J. Karadayi, Y. Wang, C. Scaff, F. Metze, and A. Cristia, “The ACLEW DiViMe: An Easy-to-use Diarization Tool,” in _Interspeech_ , 2018.
  * [6] N. Ryant, K. Church, C. Cieri, A. Cristia, J. Du, S. Ganapathy, and M. Liberman, “The Second DIHARD Diarization Challenge: Dataset, Task, and Baselines,” in _Interspeech_ , 2019, pp. 978–982. [Online]. Available: http://dx.doi.org/10.21437/Interspeech.2019-1268
  * [7] P. García, J. Villalba, H. Bredin, J. Du, D. Castan, A. Cristia, L. Bullock, L. Guo, K. Okabe, P. S. Nidadavolu _et al._ , “Speaker detection in the wild: Lessons learned from jsalt 2019,” _Odyssey_ , 2020\.
  * [8] N. Ryant, M. Liberman, and J. Yuan, “Speech activity detection on youtube using deep neural networks.” in _Interspeech_ , 2013.
  * [9] A. Mesaros, T. Heittola, A. Eronen, and T. Virtanen, “Acoustic event detection in real life recordings,” in _18th European Signal Processing Conference_ , 2010.
  * [10] M. Mulimani and S. G. Koolagudi, “Acoustic event classification using spectrogram features,” in _TENCON_ , 2018.
  * [11] D. Xu, U. Yapanel, S. Gray, and C. T. Baer, “The lena language environment analysis system: the interpreted time segments (its) file,” 2008.
  * [12] M. Ravanelli and Y. Bengio, “Speaker recognition from raw waveform with sincnet,” in _Spoken Language Technology Workshop (SLT)_ , 2018, pp. 1021–1028.
  * [13] J. Gilkerson, K. K. Coulter, and J. A. Richards, “Transcriptional analyses of the lena natural language corpus,” _Boulder, CO: LENA Foundation_ , vol. 12, p. 2013, 2008.
  * [14] H. Ganek and A. Eriks-Brophy, “The language environment analysis (lena) system: A literature review,” in _Proceedings of the joint workshop on NLP for Computer Assisted Language Learning and NLP for Language Acquisition at SLTC_ , 2016.
  * [15] M. Vandam, D. K. Oller, S. Ambrose, S. Gray, J. Richards, J. Gilkerson, N. Silbert, and M. Moeller, “Automated vocal analysis of children with hearing loss and their typical and atypical peers,” _Ear and hearing_ , vol. 36, 2015.
  * [16] N. Ferjan Ramírez, S. R. Lytle, and P. K. Kuhl, “Parent coaching increases conversational turns and advances infant language development,” _Proceedings of the National Academy of Sciences_ , vol. 117, no. 7, 2020\.
  * [17] J. Dykstra Steinbrenner, M. Sabatos-DeVito, D. Irvin, B. Boyd, K. Hume, and S. Odom, “Using the language environment analysis (lena) system in preschool classrooms with children with autism spectrum disorders,” _Autism : the international journal of research and practice_ , vol. 17, 2012.
  * [18] A. Cristia, M. Lavechin, C. Scaff, M. Soderstrom, C. Rowland, O. Räsänen, J. Bunce, and E. Bergelson, “A thorough evaluation of the language environment analysis (lena) system,” _Behavior Research Methods_ , 2019.
  * [19] J. Gilkerson, Y. Zhang, J. Richards, X. Xu, F. Jiang, J. Harnsberger, and K. Topping, “Evaluating lena system performance for chinese: A pilot study in shanghai,” _Journal of speech, language, and hearing research : JSLHR_ , vol. 58, 2015.
  * [20] M. Canault, M.-T. Le Normand, S. Foudil, N. Loundon, and H. Thai-Van, “Reliability of the language environment analysis system (lena) in european french,” _Behavior research methods_ , vol. 48, no. 3, pp. 1109–1124, 2016\.
  * [21] E. Bergelson, A. Warlaumont, A. Cristia, M. Casillas, C. Rosemberg, M. Soderstrom, C. Rowland, S. Durrant, and J. Bunce, “Starter-ACLEW,” 2017\. [Online]. Available: http://databrary.org/volume/390
  * [22] M. Canault, M.-T. Le Normand, S. Foudil, N. Loundon, and H. Thai-Van, “Reliability of the Language ENvironment Analysis system (LENA™) in European French,” _Behavior Research Methods_ , vol. 48, no. 3, Sep. 2016.
  * [23] H. Chung, E. J. Kong, J. Edwards, G. Weismer, M. Fourakis, and Y. Hwang, “Cross-linguistic studies of children’s and adults’ vowel spaces,” _The Journal of the Acoustical Society of America_ , vol. 131, no. 1, Jan. 2012. [Online]. Available: http://asa.scitation.org/doi/10.1121/1.3651823
  * [24] J. J. Holliday, P. F. Reidy, M. E. Beckman, and J. Edwards, “Quantifying the Robustness of the English Sibilant Fricative Contrast in Children,” _Journal of Speech, Language, and Hearing Research_ , vol. 58, no. 3, Jun. 2015.
  * [25] E. J. Kong, M. E. Beckman, and J. Edwards, “Voice onset time is necessary but not always sufficient to describe acquisition of voiced stops: The cases of Greek and Japanese,” _Journal of Phonetics_ , vol. 40, no. 6, Nov. 2012\.
  * [26] F. Li, “Language-Specific Developmental Differences in Speech Production: A Cross-Language Acoustic Study: Language-Specific Developmental Differences,” _Child Development_ , vol. 83, no. 4, Jul. 2012.
  * [27] G. Pretzer, A. Warlaumont, L. Lopez, and E. Walle, “Infant and adult vocalizations in home audio recordings,” 2018.
  * [28] A. Cristia and H. Colleran, “Excerpts from daylong recordings of young children learning many languages in vanuatu,” 2007.
  * [29] C. Scaff, J. Stieglitz, and A. Cristia, “Excerpts from daylong recordings of young children learning tsimane’ in bolivia,” 2007.
  * [30] J. S. Tsay, “Construction and automatization of a minnan child speech corpus with some research findings,” _IJCLCLP_ , vol. 12, 2007.
  * [31] H. Bredin, “pyannote.metrics: a toolkit for reproducible evaluation, diagnostic, and error analysis of speaker diarization systems,” in _Interspeech_ , 2017.
  * [32] D. Snyder, G. Chen, and D. Povey, “Musan: A music, speech, and noise corpus,” _arXiv preprint arXiv:1510.08484_ , 2015.
  * [33] L. N. Smith, “Cyclical learning rates for training neural networks,” in _Winter Conference on Applications of Computer Vision_ , 2017.
  * [34] H. Bredin, R. Yin, J. M. Coria, G. Gelly, P. Korshunov, M. Lavechin, D. Fustes, H. Titeux, W. Bouaziz, and M.-P. Gill, “pyannote.audio: neural building blocks for speaker diarization,” in _International Conference on Acoustics, Speech, and Signal Processing (ICASSP)_ , 2020.


# Domain Adaptation of Visual Policies with a Single Demonstration

Weiyao Wang1 and Gregory D. Hager1 This work was supported by the U.S.
National Science Foundation grants IIS-1900952 to Johns Hopkins University.1W.
Wang and Gregory D. Hager are with the Department of Computer Science, Johns
Hopkins University, Baltimore, MD 21218, USA wwang121 @ jhu.edu and hager @
cs.jhu.edu

###### Abstract

Deploying machine learning algorithms for robot tasks in real-world
applications presents a core challenge: overcoming the domain gap between the
training and the deployment environment. This is particularly difficult for
visuomotor policies that utilize high-dimensional images as input,
particularly when those images are generated via simulation. A common method
to tackle this issue is through domain randomization, which aims to broaden
the span of the training distribution to cover the test-time distribution.
However, this approach is only effective when the domain randomization
encompasses the actual shifts in the test-time distribution. We take a
different approach, where we make use of a single demonstration (a prompt) to
learn policy that adapts to the testing target environment. Our proposed
framework, PromptAdapt, leverages the Transformer architecture’s capacity to
model sequential data to learn demonstration-conditioned visual policies,
allowing for in-context adaptation to a target domain that is distinct from
training. Our experiments in both simulation and real-world settings show that
PromptAdapt is a strong domain-adapting policy that outperforms baseline
methods by a large margin under a range of domain shifts, including variations
in lighting, color, texture, and camera pose. Videos and more information can
be viewed at project webpage: https://sites.google.com/view/promptadapt.

##  I Introduction

Reinforcement learning (RL) has demonstrated impressive achievements in
complex robotic tasks that use image observations [1, 2, 3]. At the same time,
advances in simulation [4, 5] have enabled RL algorithms to overcome their
high sample complexity by training in a simulated environment before
deployment on a physical robot. However, the discrepancies between the
simulated data and the real-world, the well-known sim-to-real gap [6], often
leads to poor real-world performance [7]. This is an extreme case of the more
general problem of ensuring RL algorithms are able to accommodate domain
shifts on their inputs. The present work mainly focuses on robustness under
changes in visual appearances, such as changes in lighting and texture, a
significant category of environmental shifts within the sim-to-real gap.

A common approach to addressing distribution shift is to attempt to learn a
visual policy that is robust to appearance changes through domain
randomization [8, 9, 10, 11]. Applying a large amount of randomization
generally increases (but does not guarantee) the likelihood that visual policy
will work in a novel test environment, but can also lead to increased training
difficulty and decreased average policy performance. A variety of other
approaches have been proposed to mitigate the difficulty of policy learning
[12, 13, 14, 15, 16, 17, 18]. However accommodating large domain shifts
between the training environment and the deployment environment, which may be
unknown during training, remains a challenge.

Instead of attempting to learn a visual policy that is robust to all unknown
environmental changes, we instead pose the following question: Can a single
demonstration serve as a prompt for visual domain adaptation in visuomotor
policies? Our approach to answering this question builds on a related set of
ideas that use state-action trajectories, in the form of a demonstration, as a
form of context or "prompt" to enable rapid adaptation to new tasks [19, 20,
21]. In the case of domain shift, we formulate the problem as a demonstration-
conditioned policy distillation challenge. The demonstration trajectory
provides crucial information about both the optimal policy and the transition
dynamics within the target domain. In many real-world scenarios, a human
operator can offer this brief demonstration to illustrate how to complete the
task under the conditions of the target domain.

In brief, in our prompt-based domain adaptation framework, termed PromptAdapt,
we first train a teacher policy with a ground truth state vector as input. We
then train a Transformer-based student visual policy to imitate the teacher
policy’s behavior, conditioned on a demonstration with image observations
obtained under various domain randomization perturbations (the prompt). During
testing in a new environment, the algorithm is given a corresponding
demonstration acquired in the new environment. The agent leverages the
observations and policy information in the prompt to adapt to the domain shift
and act accordingly. This framework is attractive because the visual policy is
domain adapted with only one short trajectory segment and no explicit
finetuning. It combines policy distillation and the benefits of the
Transformer architecture to efficiently adapt to the target domain.

To validate the performance of our method, we conducted tests in both
simulated and real-world environments. Our results indicate that PromptAdapt
is highly effective in adapting to visual domain shifts, such as changes in
lighting, color, texture, and camera pose. Through extensive experiments, we
demonstrate its superior performance over a range of baseline approaches. Our
method consistently achieves higher cumulative rewards or success rates for
both in-distribution and out-of-distribution shifts from the training set,
highlighting its practicality for adapting to unknown environmental changes
that may occur in real-world deployments.

Figure 1: Left: We first train a teacher policy using privileged ground truth
state information. Right: We then distill the learned policy to a visual
input-only student policy through imitation learning. For the student policy,
our PromptAdapt architecture leverages a Transformer network to condition on a
single demonstration, efficiently adapting to the target domain during
testing. Same domain randomization function is applied to images in
demonstration and per step observation to enable demonstration conditioned
adaptation to visual appearance changes.

##  II Related Work

Visual Reinforcement Learning: Visual reinforcement learning has gained
significant attention due to its potential applicability in a wide range of
real-world tasks. Methods that combine convolutional neural networks
(ConvNets) with reinforcement learning algorithms have achieved remarkable
success in learning policies directly from raw pixel inputs. More recent work
has explored the use of unsupervised or self-supervised auxiliary tasks [22,
23, 24, 25] and data augmentations [3, 26, 27, 28, 29] to learn robust and
performing visual polices.

Imitation Learning and Policy Distillation: Imitation learning [13, 14] is an
effective approach to transfer expert knowledge to a student policy. Policy
distillation [15, 16, 17, 18] is a related technique where a student policy is
trained to imitate a teacher policy, often with the goal of compressing or
refining the teacher policy. In the context of image based RL, [12] proposes
to disentangle between policy learning and visual generalization by policy
distillation. In some approaches [17, 30, 31], teacher can access to
additional privileged information (e.g., grounded truth state) which student
policy does not have. Building upon this framework, our method extends it to
adapt visual policies using a single demonstrations.

Domain Transfer for Control Policy: Domain transfer has been extensively
studied for control policies, often in the context of sim-to-real transfer. In
such scenarios, the goal is to deploy a policy trained in a simulator onto a
real physical robot, where there may be discrepancies in physics,
configurations, and appearances. One widely used approach to bridge this gap
is domain randomization. This strategy aims to reduce the discrepancies
between the source (simulated) and target (real) domains [8, 9]. Techniques
such as [10, 11] have utilized domain randomization to enhance the
generalization capabilities of learned policies. They introduce variability in
the simulation during training to mitigate shifts in factors such as system
dynamics [30, 32] and visual appearances [33]. In addition to mere
augmentation of the training environment, domain adaptation methods [34, 35]
are explored to refine usually state-based policies using test-time trajectory
sequences. In contexts closely related to ours, focusing on adapting visual
policies, [36] employs inverse dynamics modeling with a self-supervised
learning objective to fine-tune the image encoder during test time, thereby
enhancing performance. Some recent studies have delved into the use of
generative adversarial training to aid sim-to-real transfer [37, 38, 39].

Meta-Learning: Meta-learning aims to quickly adapt to new machine learning
tasks with limited data at test time [40, 41, 42, 43]. In the context of
reinforcement learning, meta-learning has been used to enable fast adaptation
of policies to perform new tasks [43, 44, 45, 46]. Our work is related to
these approaches, as we leverage a single demonstrations to adapt visual
policies; however, our method does not make weight updates through finetuning.
Instead, we focus on the Transformer architecture as a means to effectively
condition on the demonstration data for adaptation in an in-context manner.

One-Shot Imitation Learning with Transformer: The Transformer architecture
[47, 48] has been highly successful in the field of natural language
processing and has been adapted for various other domains, including
reinforcement learning and robotic control. Originating in the realm of NLP,
the prompt-based approach [49, 50] has gained traction as an effective method
for in-context task adaptation. In this approach, a task-specific prompt is
added as a prefix to the input sequence, guiding the model toward the desired
output. This design has been adopted in the field of robot learning recently,
particularly for adaptation to new tasks encountered during testing, and is
framed within the context of one-shot imitation learning. For example, [19]
proposes a one-shot imitation learning method that employs the Transformer
architecture to learn new tasks based on a single demonstration from an
expert. Subsequent work [20] further improves generalization capabilities by
incorporating temporal contrastive learning. In the context of offline RL,
[21] extends the recently proposed Decision Transformer [51] to enable few-
shot generalization for unseen tasks.

The majority of current research in this area studies using a single
demonstration to perform unseen testing tasks. However, these approaches often
overlook the challenge of adapting to different visual domains, an obstacle
that remains significant for important applications such as sim-to-real
adaptation. Our aim is to extend the current framework of Transformer-based,
demonstration-conditioned policy adaptation to include visual domain
adaptation.

##  III Method

We frame our problem as a single demonstration conditioned domain adaptation
problem for visual policies. Formally, consider a Markov decision process
(MDP) [52] $\mathcal{M}$ parameterized by the tuple $<S,A,O,R,P,\gamma>$,
where $S$ and $A$ are the state and action spaces. $P:S\times A\times
S\rightarrow[0,1]$ is the transition function, $R:S\times
A\rightarrow\mathrm{R}$ is the scalar reward, and $\gamma$ is the discount
factor. We assume that the desired visual RL agent cannot directly access
state space $S$ but only receives higher-dimensional input $o\in O$ (e.g.,
pixel images) as observations. The source environment $\mathcal{M}_{src}$ and
target environment $\mathcal{M}_{tgt}$ differ only in the mapping from state
space $S$ to observation space $O$ due to visual appearance shift, while all
other elements remain the same.

We assume access to an $h$-step demonstration trajectory
$\tau^{*}=(o_{0}^{*},a_{o}^{*},...,o_{h-1}^{*},a_{h-1}^{*})$ in the target
environment. In this work, we aim to design an algorithm that can efficiently
leverage such a single demonstration to perform well on $\mathcal{M}_{tgt}$,
which is not known a priori.

Policy distillation: To concentrate on the issue of domain adaptation through
a single demonstration, we divide policy learning and policy adaptation under
the framework of policy distillation. In the initial stage, we train a high-
performing teacher policy $\pi_{t}(s):S\rightarrow A$ using a standard RL
algorithm directly from state space. The objective is to maximize cumulative
episodic reward in the source environment:

$J(\pi_{t})=\mathbb{E}_{\tau\sim\pi_{t}}[R(\tau)]=\mathbb{E}_{\tau\sim\pi_{t}}[\sum_{t=0}^{T}\gamma^{t}r_{t}],$
(1)

where $\tau$ is the rollout trajectory induced by executing teaching policy
$\pi_{t}$. For this stage, we choose Soft Actor-Critic (SAC) [53] as it is a
state-of-the-art RL algorithm that is widely used in continuous control tasks.

In the second stage, we train a student policy $\pi_{s}(o;\tau^{*})$ to
imitate the optimal actions taken by the teacher, given the observation and a
single demonstration $\tau^{*}$, both of which are under the same domain
randomization function $f_{DR}$ from a domain randomization function set
$F_{DR}$. This design brings demonstrations and per step observation to the
same randomized visual domain and encourages the student policy to learn to
adapt its policy based on the demonstration $\tau^{*}$.

Algorithm 1 Domain adapted visual policy via single demonstration

1:Input: $\pi_{t},\pi_{s}$: trained teacher policy and randomly initialized
student policy, $\mathcal{D}_{dagger}$: empty dagger dataset, $F$: domain
randomization function set, $N$: number of steps for each trial and $K$:
number of epochs.

2:Roll out $\pi_{t}$ to collect initial samples in $\mathcal{D}_{dagger}$

3:for $k=1,2,\dots,K$ do

4: Sample domain randomization function $f_{DR}\sim F_{DR}$

5: Roll out $\pi_{t}$ to get demonstration $\tau^{\prime}$ under $f_{DR}$

6: for $n=1,2,\dots,N$ do

7: Roll out $\pi_{s}$ to collect $o^{\prime},s^{\prime}$ samples under
$f_{DR}$

8: Add new data
$\mathcal{D}_{dagger}\leftarrow\mathcal{D}_{dagger}\cup\\{o^{\prime},s^{\prime},\tau^{\prime}\\}$

9: $\triangleright$ Collect data

10: Sample tuple $(o,s,\tau^{*})\sim\mathcal{D}_{dagger}$

11: Update $\pi_{s}$ to minimize
$\left\lVert\pi_{s}(o;\tau^{*})-\pi_{t}(s)\right\rVert^{2}$

12: $\triangleright$ Update student policy

The student policy is distilled from the teacher using the DAgger (Dataset
Aggregation) [13] imitation procedure. To begin with, we initialize a DAgger
dataset $\mathcal{D}_{dagger}$ by rolling out the teacher policy. In each
iteration, we select a domain randomization function $f_{DR}\in F_{DR}$ and
apply it to the policy rollout process to collect teacher demonstration
$\tau^{\prime}$ and student rollout observations $o^{\prime},s^{\prime}$ from
the simulator. The process alternates between (1) updating the student’s
parameters through gradient descent on a supervised regression loss, and (2)
adding more samples to $\mathcal{D}_{dagger}$ using the latest student policy.

Transformer-based prompt-conditioned visual policy: We build our model based
on the Transformer architecture. We first process the visual input through a
deep convolutional neural network (ConvNet) to extract features. Given an
input observation $o$, the ConvNet encoder $E$ generates an output
representation $E(o)\in\mathbb{R}^{d_{e}}$, where $d_{e}$ is the encoder
output dimension. We then define two embedding functions,
$f_{o}:\mathbb{R}^{d_{e}}\rightarrow\mathbb{R}^{d_{h}}$ and
$f_{a}:\mathbb{R}^{d_{a}}\rightarrow\mathbb{R}^{d_{h}}$ to embed the
observation features and actions, respectively, to embedding with dimension of
$d_{h}$. The resulting embeddings $f_{o}(E(o))$ and $f_{a}(a)$ are then used
as input to the Transformer’s encoder layer. To provide sequential information
in the trajectory, we incorporate the learnable linear positional encodings
$P\in\mathbb{R}^{(h+1)\times d_{h}}$ as in [51]. We add the positional
encodings to the input elements before they are processed by the Transformer
encoder, allowing the self-attention mechanism in the Transformer to
selectively focus on different parts of the demonstration trajectory and the
current observation.

$\displaystyle\mathrm{Input}$
$\displaystyle=\mathrm{Concat}\Bigl{(}[f_{o}(E(o_{0}^{*})),f_{a}(a_{0}^{*}),$
(2)
$\displaystyle\ldots,(f_{o}(E(o_{h-1}^{*})),f_{a}(a_{h-1}^{*})),f_{o}(E(o))]\Bigr{)}+P$

We then predict the next action to take $\hat{a}$ based on the last element of
the output from the Transformer. To train the student policy, we minimize the
$L_{2}$ distance between the predicted action $\hat{a}$ and the teacher
policy’s output $a^{*}$, which is obtained by applying the teacher policy to
the state $s$. The loss function $L$ can be formulated as:

$L(\pi_{s})=\left\lVert\hat{a}-a^{*}\right\rVert^{2}=\left\lVert\pi_{s}(o;\tau^{*})-\pi_{t}(s)\right\rVert^{2},$
(3)

where $||\cdot||^{2}$ denotes the squared $L_{2}$ distance. By minimizing the
imitation loss, the student policy learns to mimic the behavior of the teacher
policy. This learning objective results in adaptive policy capable of
effectively extracting knowledge from given demonstration and adjusting
policies accordingly.

##  IV Experiments

###  IV-A Settings

Environments: We selected four tasks from the Distracting Control Suite
(Distracting CS) [54] and further implemented two tasks for the UR5 robot in
both simulation and real world settings to benchmark our algorithm’s
performance. Simulation observations can be found in Figure 2.

Based on the DeepMind Control Suite [55], Distracting CS is a collection of
continuous control tasks designed to challenge an agent’s ability to learn in
the presence of domain shift. This is introduced by varying levels of changes
in colors, backgrounds, and camera poses, making it an ideal benchmark for
testing the generalization of reinforcement learning algorithms. We selected
four tasks: cartpole swingup, ball_in_cup catch, finger spin, and walker walk.
The evaluation metric is the episode accumulated reward. We treat the
distractions (background, color and camera pose) introduced in this benchmark
as domain randomization elements. We train with environments using random
distractions of 0.2 intensity as defined in [55]. We then test with random
distractions of either 0.2 intensity to evaluate in-distribution (out-of-
sample) adaption ability or 0.4 intensity to evaluate out-of-distribution
adaption ability.

To further evaluate the applicability of our approach to robotic arm
manipulation tasks, we conducted tests with two UR5 robot tasks: reach and
push. We have simulated environment implemented with the Mujoco physics engine
[4] and a physical UR5 robot for real-world experiments under the sim-to-real
setting. The evaluation metric is the task success rate. The success of the
reach task is defined as moving the end effector within 5 cm of a colored
circular chip, while the success of the push task is defined as pushing a
block over a finish line. The policy is trained in simulation with domain
randomization including lighting conditions, object colors, table textures,
and camera poses. During testing in simulation, we evaluate on a random
generated environments with either in-distribution or out-of-distribution
domain randomization hyperparameters to test domain adaptation abilities. For
testing on physical robot, we also replicate the in-distribution and out-of-
distribution settings with corresponding colors and camera poses to evaluate
the applicability to real-world scenarios.

Figure 2: Sample observations in simulated environments. In-distribution image
sample is shown left in each pair and out-of-distribution image sample is
shown right. First row: cartpole swingup, ball_in_cup catch; Second row:
finger spin and walker walk; Third row: UR5 reach and UR5 push.

Baseline Methods: In order to benchmark the efficacy of our method in learning
domain-adapting visual policies given demonstration, we compare it against
three baseline methods: RL with domain randomization (DR) [10], policy
distillation with domain randomization and policy distillation with model-
agnostic meta-learning (MAML) [43]. DR serves as a baseline that improves
robustness solely by expanding the input observation distribution, without
utilizing the demonstration at test time. This technique is combined with
direct image based RL with SAC [53] and policy distillation respectively. Also
used under policy distillation framework, MAML utilizes the demonstration by
adapting the policy’s weights through gradient descent on observation-action
pairs from the demonstration. We use the same policy distillation framework as
our method: imitation from trained teacher policy using the DAgger [13]
algorithm.

TABLE I: Evaluation in Distracting CS Environments by Episodic Cumulative Rewards (mean $\pm$ standard deviation) | In-Distribution Reward | Out-of-Distribution Reward
---|---|---
Task | RL | Policy Distillation | RL | Policy Distillation
| DR | DR | MAML | PA (ours) | DR | DR | MAML | PA (ours)
ball_in_cup catch | $109\pm 28$ | $718\pm 70$ | $773\pm 80$ | $\mathbf{883\pm 25}$ | $129\pm 41$ | $372\pm 61$ | $444\pm 87$ | $\mathbf{538\pm 76}$
cartpole swingup | $123\pm 30$ | $258\pm 32$ | $234\pm 75$ | $\mathbf{432\pm 195}$ | $117\pm 48$ | $222\pm 23$ | $203\pm 43$ | $\mathbf{280\pm 86}$
finger spin | $13\pm 6$ | $167\pm 65$ | $192\pm 79$ | $\mathbf{408\pm 210}$ | $12\pm 4$ | $25\pm 14$ | $39\pm 21$ | $\mathbf{103\pm 43}$
walker walk | $372\pm 304$ | $304\pm 101$ | $329\pm 91$ | $\mathbf{601\pm 106}$ | $99\pm 65$ | $145\pm 32$ | $154\pm 29$ | $\mathbf{220\pm 91}$

Algorithm Details: In our algorithm, PromptAdapt, the teacher policy is
learned using SAC [53], with the state vector serving as input. The actor and
critic networks are implemented with a multilayer perceptron (MLP) consisting
of two hidden layers, each with a size of 1024 units. The teacher policy is
trained with 500K environment steps for Disctracting CS environments and 100K
for UR5 environments. The student policy is constructed using a 3-layer
Transformer, accompanied by an 11-layer ConvNet image tokenizer and a linear
action tokenizer. To encapsulate temporal information, we stack three
consecutive frames for each observation. The student policy is trained with
100K environment steps. During testing in simulation, we use the same teacher
policy with groundtruth state access to produce demonstration trajectories;
for testing in real-world, we have human tele-operated trajectories as
demonstration.

We implement baseline methods using an 11-layer ConvNet, followed by an MLP
with two hidden layers, each of size 1024 units. We ensure that the total
number of trainable parameters for our method and both baselines are
comparable around 2.7M. Both baseline methods are trained by imitating the
same teacher policy by 100K steps as our method. All experiments are repeated
across 4 random seeds and evaluated with 20 episodes for simulated experiments
and 10 episodes for real-world experiments.

###  IV-B Results

TABLE II: Evaluation in simulated UR5 environments by task success rate | In-Distribution Reward | Out-of-Distribution Reward
---|---|---
Task | RL | Policy Distillation | RL | Policy Distillation
| DR | DR | MAML | PA (ours) | DR | DR | MAML | PA (ours)
UR5 reach | $48\pm 16$ | $90\pm 12$ | $96\pm 4$ | $\mathbf{99\pm 1}$ | $11\pm 1$ | $52\pm 10$ | $53\pm 12$ | $\mathbf{77\pm 9}$
UR5 push | $39\pm 44$ | $88\pm 8$ | $87\pm 6$ | $\mathbf{96\pm 1}$ | $9\pm 14$ | $36\pm 8$ | $38\pm 5$ | $\mathbf{68\pm 9}$

1\. Does the incorporation of a single demonstrations enhance agent
performance in both in-distribution and out-of-distribution settings?

We conduct experiments with Distracting CS benchmarks and simulated UR5 robot
tasks to evaluate our method’s performance. As can be seen from the results
presented in Table I and Table II, our method consistently outperforms
baselines of RL with DR and policy distillation with DR and MAML across all
environments, exhibiting superior performance in both in-distribution and out-
of-distribution settings. By leveraging the teacher policy embedded in
demonstrations, our method is capable of effectively capturing and exploiting
the nuances of the visual elements in new environments. The enhanced
performance in out-of-distribution settings demonstrates that the agent has
strong capability for generalizing to new scenarios.

2\. How does the proposed approach improve generalization across different
environmental elements?

We further examine the effectiveness of our method, PromptAdapt (PA), compared
to the baselines in terms of generalization over various out-of-distribution
factors in the UR5 simulated environments. As illustrated in Figure 3, our
method consistently outperforms baseline methods across multiple out-of-
distribution factors. A significant improvement is observed in the camera pose
factor, which is presumably the most challenging aspect in the domain
difference. The superior performance of PromptAdapt underscores its ability to
capture and adapt to complex changes across domains. We leave RL with DR out
of this comparison because it does not perform well in in-distribution setting
already due to the difficulty in training RL directly under heavy domain
changes.

PromptAdapt also demonstrates improved performance in other out-of-
distribution factors, such as lighting and texture. These improvements further
support that PromptAdapt can adapt to novel situations with different visual
conditions. The consistent performance gains across all out-of-distribution
factors suggest that PromptAdapt is a promising approach for creating
adaptable agents capable of handling complex tasks in diverse environments.

(a) UR5 reach

(b) UR5 push

Figure 3: Ablation study in UR5 simulation environments. Our method leads to
lower decrease in success for all out of distribution factors including
lighting, object color, table texture and camera pose. Error bar indicates one
standard deviation.

(a) In-distribution

(b) Out-of-distribution

Figure 4: Ablation study in Distracting CS over different variants in
demonstration. Figure 5: Left: Real-world observations for UR5 reach and UR5
push. Right: Experiment setup.

3\. How do different components in demonstration trajectories contribute to
the observed performance of the proposed approach?

To understand how various components in the trajectory prompt influence the
performance of PromptAdapt, we conducted an ablation study in two Distracting
CS environments, focusing on the following variants. History Trajectory:
replace the demonstration with the previous $h$ steps of student rollout;
Random Demonstration: use a random rollout instead of expert demonstration for
prompt; Demonstration without Action: use the expert demonstration but
excludes action. All variants are trained and tested both in the given
configuration.

The findings from our study, as shown in Figure 4, suggest that each of the
variants results in some degree of performance drop in both in-distribution
and out-of-distribution settings. This implies that a single demonstration
contains valuable information about the target environment that isn’t fully
captured by the history trajectory, a random demonstration, or a demonstration
without action. The History Trajectory variant, which utilizes past student
rollouts, shows a substantial decrease in performance, suggesting that these
rollouts may not fully encapsulate the specifics of the target task. The
Random Demonstration variant also shows a similar performance dip, hinting
that the quality of the demonstration, rather than merely the presence of test
domain trajectories, is critical in effective adaptation. Finally, when action
information is excluded in the Demonstration without Action variant, we
observe a slight drop in performance, indicating that action sequences in the
single demonstration also help adaptation by providing information about
transition dynamics and optimal policy. Overall, these results indicate the
value of expert demonstrations, inclusive of action sequences, in the
successful adaptation of our method.

4\. How does our approach apply to real-world?

We evaluate the real-world applicability of our method under the sim-to-real
setting for the two UR5 robot manipulations tasks compared to policy
distillation based baseline methods. Demonstration in real-world is recorded
by a human tele-operator. As demonstrated in Table III, our method
consistently outperforms the baseline approaches in the UR5 reach and UR5
push. This highlights the effectiveness of our method in handling both in-
distribution and out-of-distribution settings, leading to improved performance
under the domain shift from simulation to real-world. Figure 5 presents visual
examples of the real-world experiment.

TABLE III: Evaluation on real-world robot by success rates. | In distribution success rate
---|---
| DR | MAML | PA (ours)
UR5 reach | 48$\pm$10 | 50$\pm$8 | 80$\pm$8
UR5 push | 30$\pm$8 | 33$\pm$10 | 53$\pm$13
| Out of distribution success rate
| DR | MAML | PA (ours)
UR5 reach | 25$\pm$6 | 28$\pm$5 | 50$\pm$8
UR5 push | 13$\pm$5 | 13$\pm$10 | 33$\pm$5

##  V Limitation

Through our sim-to-real experiments, we have demonstrated that demonstrations
provided by human operators during testing can align well with those learned
via RL during training. Meanwhile, the proposed method relies on the
availability of a demonstration trajectory on the same robot to bridge the
domain gap between training and testing environments. This could limit the
applicability when only cross-embodiment demonstrations are available (e.g.,
from a different type of robot). Further analysis and solution to this problem
would be in the scope of future work.

##  VI Conclusion

In this paper, we introduced a novel approach to learn domain-adapting visual
policies. Our method, utilizing the strengths of policy distillation and the
Transformer architecture, efficiently adapts visuomotor policies to target
environments using a single demonstration. Through comprehensive experiments,
we demonstrate the effectiveness of our method across various simulated and
real-world robotics tasks. Ablation studies further illustrate the vital roles
of different components in the trajectory prompt, underscoring the significant
role of demonstrations in our approach. Our proposed method effectively
improve the adaptation of policies to novel environments, paving the way for
more efficient and effective domain-adapting visuomotor policies.

## References

  * [1] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke _et al._ , “Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation,” _arXiv preprint arXiv:1806.10293_ , 2018.
  * [2] J. Kulhánek, E. Derner, T. De Bruin, and R. Babuška, “Vision-based navigation using deep reinforcement learning,” in _2019 european conference on mobile robots (ECMR)_. IEEE, 2019, pp. 1–8.
  * [3] N. Hansen, H. Su, and X. Wang, “Stabilizing deep q-learning with convnets and vision transformers under data augmentation,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 3680–3693, 2021.
  * [4] E. Todorov, T. Erez, and Y. Tassa, “Mujoco: A physics engine for model-based control,” in _2012 IEEE/RSJ international conference on intelligent robots and systems_. IEEE, 2012, pp. 5026–5033.
  * [5] V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, and G. State, “Isaac gym: High performance gpu-based physics simulation for robot learning,” 2021.
  * [6] W. Zhao, J. P. Queralta, and T. Westerlund, “Sim-to-real transfer in deep reinforcement learning for robotics: a survey,” in _2020 IEEE symposium series on computational intelligence (SSCI)_. IEEE, 2020, pp. 737–744.
  * [7] K. Cobbe, C. Hesse, J. Hilton, and J. Schulman, “Leveraging procedural generation to benchmark reinforcement learning,” in _International conference on machine learning_. PMLR, 2020, pp. 2048–2056.
  * [8] Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and V. Lempitsky, “Domain-adversarial training of neural networks,” _The journal of machine learning research_ , vol. 17, no. 1, pp. 2096–2030, 2016.
  * [9] E. Tzeng, J. Hoffman, K. Saenko, and T. Darrell, “Adversarial discriminative domain adaptation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 7167–7176.
  * [10] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel, “Domain randomization for transferring deep neural networks from simulation to the real world,” in _2017 IEEE/RSJ international conference on intelligent robots and systems (IROS)_. IEEE, 2017, pp. 23–30.
  * [11] X. Chen, J. Hu, C. Jin, L. Li, and L. Wang, “Understanding domain randomization for sim-to-real transfer,” _arXiv preprint arXiv:2110.03239_ , 2021.
  * [12] L. Fan, G. Wang, D.-A. Huang, Z. Yu, L. Fei-Fei, Y. Zhu, and A. Anandkumar, “Secant: Self-expert cloning for zero-shot generalization of visual policies,” _arXiv preprint arXiv:2106.09678_ , 2021.
  * [13] S. Ross, G. Gordon, and D. Bagnell, “A reduction of imitation learning and structured prediction to no-regret online learning,” in _Proceedings of the fourteenth international conference on artificial intelligence and statistics_. JMLR Workshop and Conference Proceedings, 2011, pp. 627–635.
  * [14] J. Ho and S. Ermon, “Generative adversarial imitation learning,” _Advances in neural information processing systems_ , vol. 29, 2016.
  * [15] A. A. Rusu, S. G. Colmenarejo, C. Gulcehre, G. Desjardins, J. Kirkpatrick, R. Pascanu, V. Mnih, K. Kavukcuoglu, and R. Hadsell, “Policy distillation,” _arXiv preprint arXiv:1511.06295_ , 2015.
  * [16] E. Parisotto, J. L. Ba, and R. Salakhutdinov, “Actor-mimic: Deep multitask and transfer reinforcement learning,” _arXiv preprint arXiv:1511.06342_ , 2015.
  * [17] T. Chen, J. Xu, and P. Agrawal, “A system for general in-hand object re-orientation,” in _Conference on Robot Learning_. PMLR, 2022, pp. 297–307.
  * [18] H. Lai, W. Zhang, X. He, C. Yu, Z. Tian, Y. Yu, and J. Wang, “Sim-to-real transfer for quadrupedal locomotion via terrain transformer,” _arXiv preprint arXiv:2212.07740_ , 2022.
  * [19] S. Dasari and A. Gupta, “Transformers for one-shot visual imitation,” in _Conference on Robot Learning_. PMLR, 2021, pp. 2071–2084.
  * [20] Z. Mandi, F. Liu, K. Lee, and P. Abbeel, “Towards more generalizable one-shot visual imitation learning,” in _2022 International Conference on Robotics and Automation (ICRA)_. IEEE, 2022, pp. 2434–2444.
  * [21] M. Xu, Y. Shen, S. Zhang, Y. Lu, D. Zhao, J. Tenenbaum, and C. Gan, “Prompting decision transformer for few-shot policy generalization,” in _International Conference on Machine Learning_. PMLR, 2022, pp. 24 631–24 645.
  * [22] C. Finn, X. Y. Tan, Y. Duan, T. Darrell, S. Levine, and P. Abbeel, “Deep spatial autoencoders for visuomotor learning,” in _2016 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2016, pp. 512–519.
  * [23] X. Lin, H. Baweja, G. Kantor, and D. Held, “Adaptive auxiliary task weighting for reinforcement learning,” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [24] C. Lyle, M. Rowland, G. Ostrovski, and W. Dabney, “On the effect of auxiliary tasks on representation dynamics,” in _International Conference on Artificial Intelligence and Statistics_. PMLR, 2021, pp. 1–9.
  * [25] T. He, Y. Zhang, K. Ren, M. Liu, C. Wang, W. Zhang, Y. Yang, and D. Li, “Reinforcement learning with automated auxiliary loss search,” _arXiv preprint arXiv:2210.06041_ , 2022.
  * [26] I. Kostrikov, D. Yarats, and R. Fergus, “Image augmentation is all you need: Regularizing deep reinforcement learning from pixels,” _arXiv preprint arXiv:2004.13649_ , 2020.
  * [27] N. Hansen and X. Wang, “Generalization in reinforcement learning by soft data augmentation,” in _2021 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2021, pp. 13 611–13 617.
  * [28] G. Ma, Z. Wang, Z. Yuan, X. Wang, B. Yuan, and D. Tao, “A comprehensive survey of data augmentation in visual reinforcement learning,” _arXiv preprint arXiv:2210.04561_ , 2022.
  * [29] D. Bertoin, A. Zouitine, M. Zouitine, and E. Rachelson, “Look where you look! saliency-guided q-networks for visual rl tasks,” in _Thirty-sixth Conference on Neural Information Processing Systems (NeurIPS 2022)_ , 2022.
  * [30] J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter, “Learning quadrupedal locomotion over challenging terrain,” _Science robotics_ , vol. 5, no. 47, p. eabc5986, 2020.
  * [31] T. Miki, J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter, “Learning robust perceptive locomotion for quadrupedal robots in the wild,” _Science Robotics_ , vol. 7, no. 62, p. eabk2822, 2022.
  * [32] V. Lim, H. Huang, L. Y. Chen, J. Wang, J. Ichnowski, D. Seita, M. Laskey, and K. Goldberg, “Planar robot casting with real2sim2real self-supervised learning,” _arXiv preprint arXiv:2111.04814_ , 2021.
  * [33] ——, “Real2sim2real: Self-supervised learning of physical single-step dynamic actions for planar robot casting,” in _2022 International Conference on Robotics and Automation (ICRA)_. IEEE, 2022, pp. 8282–8289.
  * [34] W. Yu, J. Tan, C. K. Liu, and G. Turk, “Preparing for the unknown: Learning a universal policy with online system identification,” _arXiv preprint arXiv:1702.02453_ , 2017.
  * [35] A. Kumar, Z. Fu, D. Pathak, and J. Malik, “Rma: Rapid motor adaptation for legged robots,” _arXiv preprint arXiv:2107.04034_ , 2021.
  * [36] N. Hansen, R. Jangir, Y. Sun, G. Alenyà, P. Abbeel, A. A. Efros, L. Pinto, and X. Wang, “Self-supervised policy adaptation during deployment,” _arXiv preprint arXiv:2007.04309_ , 2020.
  * [37] K. Rao, C. Harris, A. Irpan, S. Levine, J. Ibarz, and M. Khansari, “Rl-cyclegan: Reinforcement learning aware simulation-to-real,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 11 157–11 166.
  * [38] D. Ho, K. Rao, Z. Xu, E. Jang, M. Khansari, and Y. Bai, “Retinagan: An object-aware approach to sim-to-real transfer,” 2020. [Online]. Available: https://arxiv.org/abs/2011.03148
  * [39] T. Yoneda, G. Yang, M. R. Walter, and B. Stadie, “Invariance through inference,” _arXiv preprint arXiv:2112.08526_ , 2021.
  * [40] K. Rakelly, A. Zhou, C. Finn, S. Levine, and D. Quillen, “Efficient off-policy meta-reinforcement learning via probabilistic context variables,” in _International conference on machine learning_. PMLR, 2019, pp. 5331–5340.
  * [41] T. Hospedales, A. Antoniou, P. Micaelli, and A. Storkey, “Meta-learning in neural networks: A survey,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 44, no. 9, pp. 5149–5169, 2021.
  * [42] Y. Wang, Q. Yao, J. T. Kwok, and L. M. Ni, “Generalizing from a few examples: A survey on few-shot learning,” _ACM computing surveys (csur)_ , vol. 53, no. 3, pp. 1–34, 2020.
  * [43] C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,” in _International conference on machine learning_. PMLR, 2017, pp. 1126–1135.
  * [44] Y. Duan, M. Andrychowicz, B. Stadie, O. Jonathan Ho, J. Schneider, I. Sutskever, P. Abbeel, and W. Zaremba, “One-shot imitation learning,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [45] L. Kirsch, S. van Steenkiste, and J. Schmidhuber, “Improving generalization in meta reinforcement learning using learned objectives,” _arXiv preprint arXiv:1910.04098_ , 2019.
  * [46] X. Song, Y. Yang, K. Choromanski, K. Caluwaerts, W. Gao, C. Finn, and J. Tan, “Rapidly adaptable legged robots via evolutionary meta-learning,” in _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2020, pp. 3769–3776.
  * [47] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [48] T. Lin, Y. Wang, X. Liu, and X. Qiu, “A survey of transformers,” _AI Open_ , 2022.
  * [49] P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig, “Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing,” _arXiv preprint arXiv:2107.13586_ , 2021.
  * [50] J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, Q. Le, and D. Zhou, “Chain of thought prompting elicits reasoning in large language models,” _arXiv preprint arXiv:2201.11903_ , 2022.
  * [51] L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and I. Mordatch, “Decision transformer: Reinforcement learning via sequence modeling,” _Advances in neural information processing systems_ , vol. 34, pp. 15 084–15 097, 2021.
  * [52] R. S. Sutton and A. G. Barto, _Reinforcement learning: An introduction_. MIT press, 2018.
  * [53] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel _et al._ , “Soft actor-critic algorithms and applications,” _arXiv preprint arXiv:1812.05905_ , 2018.
  * [54] A. Stone, O. Ramirez, K. Konolige, and R. Jonschkowski, “The distracting control suite–a challenging benchmark for reinforcement learning from pixels,” _arXiv preprint arXiv:2101.02722_ , 2021.
  * [55] Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. de Las Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancq, T. Lillicrap, and M. Riedmiller, “Deepmind control suite,” 2018.