11institutetext: Boston University
11email<EMAIL_ADDRESS>
# Supervised Attribute Information Removal and Reconstruction for Image
Manipulation

Nannan Li  Bryan A. Plummer

###### Abstract

The goal of attribute manipulation is to control specified attribute(s) in
given images. Prior work approaches this problem by learning disentangled
representations for each attribute that enables it to manipulate the encoded
source attributes to the target attributes. However, encoded attributes are
often correlated with relevant image content. Thus, the source attribute
information can often be hidden in the disentangled features, leading to
unwanted image editing effects. In this paper, we propose an Attribute
Information Removal and Reconstruction (AIRR) network that prevents such
information hiding by learning how to remove the attribute information
entirely, creating attribute excluded features, and then learns to directly
inject the desired attributes in a reconstructed image. We evaluate our
approach on four diverse datasets with a variety of attributes including
DeepFashion Synthesis, DeepFashion Fine-grained Attribute, CelebA and CelebA-
HQ, where our model improves attribute manipulation accuracy and top-k
retrieval rate by 10% on average over prior work. A user study also reports
that AIRR manipulated images are preferred over prior work in up to 76% of
cases111Code and models are available at https://github.com/NannanLi999/AIRR.

##  1 Introduction

Attribute manipulation translates images based on desired attributes, which
has applications to face editing [26, 32, 31], image retrieval [27, 12, 34],
and image synthesis [3, 16], among others. In these tasks, the goal is to be
able to control a specified attribute without affecting other information in
the source image. While Generative Adversarial Networks (GANs) have achieved
impressive performance on attribute manipulation, a major challenge is that
the generator tends to take a shortcut by utilizing the preserved source
attribute information instead of the target attribute for manipulation [13,
29, 18], thus causing improper image editing effects in manipulated images.
Prior work has tried to address this by adding random noise during the
reconstruction [4, 30] or learning disentangled attribute representations
which are used to manipulate the images [26, 33]. However, low-magnitude
random noise is not targeted to the source attribute, which could be
intentionally ignored by the model or inadvertently suppress key source
features. On the other hand, even with a disentangled image representation,
the correlation between attributes and relevant image content could cause
source attribute information to be hidden in the rest of the image features.
For example, attribute _formal_ in a dress is often correlated with the
dress’s _long length_.

(a) Framework of previous methods [3, 12, 34, 35, 11]. Dashed arrows mean two
different ways of obtaining the reconstructed image

(b) Pipeline of the proposed method

Figure 1: In (a), the generator like those used by [11, 3, 12, 34, 35]
incorrectly utilizes the hidden source attribute information instead of the
target attribute _sleeveless_ for image manipulation. As a result, the
manipulated image still contains the source attribute _long sleeve_ , causing
improper image editing effects. To avoid this issue, the proposed method in
(b) erases the source attribute information in the encoded features through an
attribute remover with a disentanglement loss, conditioning the manipulated
output only on the input target attribute _sleeveless_ and the attribute
excluded features

To address these issues, we propose a supervised Attribute Information Removal
and Reconstruction (AIRR) model that learns an attribute excluded
representation and reconstructs the image with desired attributes. The key
challenge is in identifying the preserved source attribute information and
decorrelating it from the image representation [15, 31, 26, 33]. Prior
research on feature disentanglement either doesn’t consider the decorrelation
[15, 31, 35], or has limitations on the number of attributes it can
decorrelate in a forward pass [26, 33], unlike our approach which can
disentangle any number of attributes. In addition, as illustrated in Figure
1a, these methods often rely on the full image information for both
manipulation and reconstruction. As mentioned earlier, this can lead to
information hiding in the manipulated image. In contrast, as shown in Figure
1b, we use our remover to erase attribute information to obtain attribute
excluded features, which are then used to directly generate both the
reconstructed and manipulated images. Since this should eliminate the
information that could potentially be hidden in the disentangled
representation, we avoid the information hiding issues in prior work.

One challenge in our approach is our reliance on being able to identify and
remove attribute information in real images. For example, although the color
_white_ appears in the background of the input images in Figure 1, a good
attribute classifier would predict that the clothing item is _gray_ and not
_white_. This means that the background information could mislead the
attribute recognition. To address this issue, we segment the object of
interest (_e.g_., using [19, 36]) to split the image encoder into two branches
for the object of interest and the background, respectively. This helps AIRR
to concentrate the manipulation on the object of interest without influencing
the background information.

Our main contributions are:

  * •

We propose the Attribute Information Removal and Reconstruction method (AIRR),
a controllable disentangled attribute manipulation framework that produces
high quality images. The key insight in AIRR is the attribute information
removal and reconstruction module that produces an attribute excluded
representation, eliminating sources of information hiding that degrades
performance in prior work.

  * •

Extensive experiments across DeepFashion Synthesis [21], DeepFashion Fine-
grained Attribute [21], CelebA [22] and CelebA-HQ [17] report that AIRR
improves the attribute manipulation accuracy and top-k retrieval rate by 10%
on average over the state-of-the-art. Moreover, we show that AIRR can
effectively control attribute strength as well as efficiently manipulating
multiple attributes in a single forward pass.

  * •

A user study further validates the effectiveness of our approach, where our
methods are shown to produce high quality images that more accurately achieve
the target attribute manipulation by up to 76% over prior work.

##  2 Related Work

Early research in attribute manipulation [7, 11] combined the target attribute
label directly with the image or image features, and decoded them into
manipulated output. However, the decoder could incorrectly use the preserved
source attribute information for image manipulation. Thus, more recent work
(including this paper), has focused on learning disentangled attribute
representations, which we will discuss in more detailed below.

Unsupervised disentanglement. Several studies explored disentanglement in the
latent space of GANs in an unsupervised manner [25, 23, 9, 28, 32]. These
methods aim to manipulate the attributes on synthetic data, where the image
content is randomly generated. In [9], the authors found that the principle
components of features on pretrained GANs represent high-level semantic
concepts. In [32], the authors introduced channel-wise disentanglement of
StyleGAN [14]. Shoshan  _et al_. [28] utilized contrastive learning to
disentangle the latent space, achieving explicit control over synthetic facial
images. However, without manual examinations on the feature space, it’s
difficult to locate the exact attribute representation that we want to
manipulate, especially for attributes with high-level semantics. Thus, in our
work we focus on cases where attributes we wish to manipulate are known,
enabling us to directly target our feature learning.

Supervised disentanglement. Supervised disentanglement methods edit real
images based on attribute annotations. Prior work on this task can be
categorized in two types: spatial disentanglement and feature disentanglement.
In methods that focus on spatial disentanglement [15, 31], attributes are
located spatially and thus disentangled in the feature map. These methods can
find attribute-specific features by an attention map, whereas attribute-
relevant information is implicitly kept and thus influences the image
manipulation. On the other hand, feature disentanglement identifies certain
features corresponding to the manipulated attribute. [35] presents a method
that learns a linear transformation function that maps StyleGAN’s latent code.
Although StyleGAN’s latent space is disentangled [1], without orthogonal
constraints, such linear combination could result in correlation between
different image attributes and content. To address this, Yang  _et al_. [33]
learned attribute relevant and irrelevant features, but each manipulated
attribute requires training its own model, which is computationally costly.
Instead, Shen  _et al_. [26] manipulated attributes with a conditional
subspace projection via Support Vector Machines (SVM), whereas the
manipulation accuracy depends on the capability of SVM and each forward pass
can control only a single attribute. In contrast, our proposed approach can
manipulate multiple source attributes in a single forward pass by utilizing
the injected attribute embedding.

Attribute manipulation in fashion. Apart from the above mentioned methods that
are mainly applied to facial attribute editing, attribute manipulation in
fashion images has also gained a lot of attention. Recent work on this topic
mainly aim to improve the image retrieval accuracy for item recommendation.
For example, researches have leveraged spatial information when manipulating
attributes [3, 2], learned a dictionary of attribute transformations [27], or
used the attribute probability distribution as an disentangled representation
for image retrieval[12]. Kwon  _et al_. [16] predicted changes to an item’s
shape as a result of changing an attribute, enabling them to make more
significant alterations to the clothing in images. However, many of these
methods also suffered from issues with disentangling attributes, often due to
misinformation hiding, which our work minimizes.

Figure 2: AIRR framework. In the AIRR generator, a given image is parsed into
an object of interest $I_{cl}$ and background $I_{bg}$ through an offline
parser [19, 36]. $I_{cl}$ and $I_{bg}$ are encoded in separate branches (Sec.
3.1). In the $I_{cl}$ branch, the source attribute information in the encoded
features are erased by an attribute remover using a disentanglement loss (Sec.
3.2). Subsequently, the source attributes in $\mathbf{A}$ and the target
attributes in $\mathbf{\bar{A}}$ are embedded into the attribute excluded
features for image reconstruction and image manipulation, respectively (Sec.
3.3)

##  3 Attribute Information Removal and Reconstruction

Given image $I$ and its attributes
$\mathbf{A}=\\{\mathbf{a_{1}},\mathbf{a_{2}},...,\mathbf{a_{n}}\\}$, where
$a_{i}$ denotes the $i$th attribute, we aim to manipulate any number of
attributes in $\mathbf{A}$. To achieve this goal, the generator first takes a
real image as input, and uses our attribute remover to decorrelate the image
attributes from the image features. The resulting attribute excluded
representation is then combined with the target attribute embeddings to
produce the manipulated output $I_{map}$. In the following, we introduce the
four components of our model: image encoder (Section 3.1), attribute remover
(Section 3.2), decoder (Section 3.3), and learning objectives (Section 3.4).
Figure 2 shows an overview of our approach.

###  3.1 Image Encoder

To concentrate the manipulation on the object that we want to manipulate, the
image encoder in AIRR is split into two branches for the object of interest
$I_{cl}$ and the background $I_{bg}$, respectively. Prior work often achieves
the segmentation of $I_{cl}$ by learning an attention map [15, 3, 2], while we
found empirically that using an offline parser [19, 36] is more accurate in
segmenting instances. This segmentation is especially helpful for images with
multiple objects, _e.g_., an image of a fashion model wearing top, leggings
and boots. As shown in Figure 2, after obtaining $I_{cl}$ and $I_{bg}$, the
image encoder encodes $I_{cl}$ into $E_{1}(I_{cl})$ in the first branch, and
$I_{bg}$ into $E_{2}(I_{bg})$ in the second branch. Later on, AIRR only
manipulates $E_{1}(I_{cl})$ without influencing the background information
$E_{2}(I_{bg})$.

###  3.2 Attribute Remover

While prior work directly used the image features $E_{1}(I_{cl})$ to generate
the image [11, 3, 12, 34, 35], in AIRR, image features $E_{1}(I_{cl})$ from
our base encoder are fed into an attribute remover to learn an attribute-
excluded representation $R(E_{1}(I_{cl}))$. The attribute remover is an
$n$-layer convolutional block that is used to decorrelate the source attribute
information from the image representation. A design requirement for the
attribute remover is that it does not have skip connections since we aim to
erase the attribute information from the encoded features, whereas skip
connections would preserve this information.

To disentangle all the source attribute information from $E_{1}(I_{cl})$, we
would need an attribute classifier to first identify these attributes. This
can be achieved by Maximum Likelihood Estimation (MLE):

${{L_{d}}\big{(}E_{1}(I_{cl})\big{)}}=-\sum\nolimits_{\mathbf{a_{i}}\in\mathbf{A}}{\mathbf{y_{i}}^{T}\log{p_{c}}(\mathbf{a_{i}}|E_{1}(I_{cl}))}$
(1)

where ${p_{c}}(\mathbf{a_{i}}|E_{1}(I_{cl}))$ is the probability distribution
of $\mathbf{a_{i}}$, and $\mathbf{y_{i}}$ is the corresponding one-hot
attribute label. We use one residual block as the attribute classifier to
predict $p_{c}(\cdot)$.

After identifying the source attributes, we can then eliminate the attribute
information in $R(E_{1}(I_{cl}))$ by minimizing their mutual information.
Alternatively, it’s easier to minimize the upper bound of this mutual
information, which is the maximum log probability in the attribute class
distribution added by a constant $c$ (See the Supplementary for proof of this
upper bound):

$\text{MI}\big{(}{\mathbf{a}}_{i},R(E_{1}(I_{cl}))\big{)}\leq
c+{{\max}_{\mathbf{a}_{i}}}\log{p_{c}}\big{(}\mathbf{a}_{i}|R(E_{1}(I_{cl}))\big{)}$
(2)

Intuitively, minimizing this upper bound gives a uniform distribution over the
attributes, meaning that the uncertainty for a specific attribute is
maximized. Therefore, the generated features would have little knowledge of
what the original attributes are. This loss function is thus defined as a
margin loss:

${L_{d}}\big{(}R({E_{1}}({I_{cl}}))\big{)}=\sum\limits_{{{\mathbf{a}}_{\mathbf{i}}}\in{\mathbf{A}}}{\max\\{}{{\max}_{\mathbf{a}_{i}}}\log{p_{c}}\big{(}{{\mathbf{a}}_{\mathbf{i}}}|R(E_{1}(I_{cl}))\big{)}-\log\frac{1}{{|{{\mathbf{a}}_{\mathbf{i}}}|}},c\\},$
(3)

where $|\mathbf{a_{i}}|$ is the number of attribute values in
$\mathbf{a_{i}}$, and $c$ indicates the proximity to a uniform distribution,
which is set to 0.01 in our experiments as we found it performed well. We
refer to ${L_{d}}(R({E_{1}}({I_{cl}})))$ as a Mutual Information Minimization
(MIM) loss. Note that our attribute classifier and remover are trained end-to-
end with our other generator and decoder components.

###  3.3 Decoder

After the attribute remover, a new learned attribute representation is
integrated with the disentangled features $R(E_{1}(I_{cl}))$ to generate the
output image. Assuming a scale embedding vector $\bm{{\beta}_{a_{i}}}$ and a
bias embedding vector $\bm{{\gamma}_{a_{i}}}$ for attribute $\mathbf{a_{i}}$,
the original image $I$ thus can be reconstructed by combining
$R\big{(}E_{1}(I_{cl})\big{)}$, $E_{2}(I_{bg})$ and the attribute embeddings,
_i.e_.,

${I_{rec}}=G\Big{(}\text{concat}\big{[}\sum\nolimits_{\mathbf{a_{i}}\in\mathbf{A}}{{\bm{\beta_{\mathbf{a_{i}}}}}\cdot
R\big{(}{E_{1}}({I_{cl}})\big{)}+\bm{{\gamma_{\mathbf{a_{i}}}}}},E_{2}(I_{bg})\big{]}\Big{)},$
(4)

where $G$ is the decoder. Similarly, given target attributes of our desired
output $\mathbf{\bar{A}}$, $R\big{(}E_{1}(I_{cl})\big{)}$ produces the
manipulated image $I_{map}$ by

${I_{map}}=G\Big{(}\text{concat}\big{[}\sum\nolimits_{\mathbf{\bar{a_{i}}}\in\mathbf{\bar{A}}}{{\bm{\beta}_{\mathbf{\bar{a_{i}}}}}}\cdot
R\big{(}{E_{1}}({I_{cl}})\big{)}+\bm{{\gamma}_{\mathbf{\bar{a_{i}}}}},E_{2}(I_{bg})\big{]}\Big{)}$
(5)

In contrast to some prior work [3, 7, 15], where the attribute embedding
vector and the encoded features are combined by concatenation, we multiply
these two features such that linear interpolation between different attribute
embeddings can better control the strength of these attributes in the output
[8, 33]. Further discussion can be found in Section 4.5.

###  3.4 Learning Objectives

Disentanglement loss.  The disentanglement loss combines the MLE loss in Eq.
(1) and the MIM loss in Eq. (3) as

$L_{d}={{L_{d}}\big{(}E_{1}(I_{cl})\big{)}}+{{L_{d}}\big{(}R\big{(}E_{1}(I_{cl})\big{)}\big{)}}$
(6)

By first identifying the source attributes in $E_{1}(I_{cl})$ and then
minimizing the attribute information in $R(E_{1}(I_{cl}))$, this
disentanglement loss enables the decoder to condition the output on the new
attribute that is injected to the generator. In Section 4.5, we also show
empirically that with the disentanglement loss, the attribute remover indeed
gets rid of all the source attribute information.

Reconstruction loss. The reconstructed image $I_{rec}$ is evaluated by its
$l_{1}$ distance to the original image $I$:

${L_{rec}}=||{I_{rec}}-I||_{1}$ (7)

Adversarial loss. The manipulated image $I_{map}$ doesn’t have a paired ground
truth of how it should look like, for which its plausibility is evaluated by
the discriminator $D$. Using LSGAN [24], the adversarial loss of the generator
can be written as

${L_{adv}^{g}}={(1-D({I_{map}}))^{2}}+{(1-D({I_{rec}}))^{2}}$ (8)

In the discriminator, this adversarial loss includes both the reconstructed
image $I_{rec}$ and the manipulated image $I_{map}$ since they are both fake
samples:

$L_{adv}^{d}=(1-D(I))^{2}+\frac{1}{2}\big{(}({D}(I_{map})^{2}+{D}(I_{rec})^{2}\big{)}$
(9)

Image attribute classification loss [6]. This loss maximizes the mutual
information between the injected attribute embedding and the generated image:

$\displaystyle L_{attr}^{g}$
$\displaystyle=-\sum\nolimits_{\mathbf{a_{i}}\in\mathbf{A}}{\mathbf{y_{i}}^{T}\log
p_{d}(\mathbf{a_{i}}|I_{rec})}-\sum\nolimits_{\mathbf{\bar{a}_{i}}\in\mathbf{\bar{A}}}{\mathbf{\bar{y}_{i}}^{T}\log
p_{d}(\mathbf{\bar{a_{i}}}|I_{map})}$ (10)

where $p_{d}(\cdot)$ is the probability distribution of attributes predicted
by a classification branch in the discriminator. For the discriminator, this
loss is defined on the real image $I$.

$L_{attr}^{d}=L_{attr}^{d}(I)=-\sum\nolimits_{\mathbf{a_{i}}\in\mathbf{A}}{\mathbf{y_{i}}^{T}\log
p_{d}(\mathbf{a_{i}}|I)}$ (11)

Perceptual loss. To further improve the quality of the generated images, a
perceptual loss is introduced in the generator as in [3]. It is based on the
distance of paired real and fake images in the CNN feature space

${L_{p}}=||\text{CNN}(I)-\text{CNN}({I_{rec}})||_{1}+||\text{CNN}({I_{ref}})-\text{CNN}({I_{map}})||_{1}$
(12)

where $I_{ref}$ is selected from the real images in the dataset to have
exactly the same attributes $\mathbf{\bar{A}}$ as $I_{map}$.

Full objective. Including all the above loss functions, the full objectives
for the generator and discriminator are

$L_{gen}=L_{adv}^{g}+\lambda_{1}{L_{d}}+\lambda_{2}{L_{attr}^{g}}+\lambda_{3}{L_{rec}}+\lambda_{4}{L_{p}}$
(13) $L_{dis}=L_{adv}^{d}+2\lambda_{2}L_{attr}^{d}$ (14)

where $\lambda_{1},\lambda_{2},\lambda_{3}$ are trade-off parameters. As in
prior work [20, 5], these parameters must be set carefully to control the
degree of disentanglement. Note that except for $L_{rec}$, these loss
functions are all symmetrical with respect to $\mathbf{A}$ and
$\mathbf{\bar{A}}$ to enforce that the manipulated image is a plausible
reconstructed image.

Table 1: Statistics of the datasets used in our experiments in Section 4

Dataset | image | #training | #test | #attributes | #attribute
---|---|---|---|---|---
size | images | images | values
DeepFashion Synthesis [21] | 128x128 | 76,979 | 2000 | 2 | 21
DeepFashion Fine-grained Attribute [21] | 256x256 | 19,000 | 1000 | 6 | 26
CelebA [22] | 128x128 | 200,599 | 2000 | 8 | 21
CelebA-HQ [17] | 1024x1024 | 29,000 | 1000 | 8 | 21

##  4 Experiments

To prove the efficiency of the proposed method, we evaluate our model on four
publicly available datasets: DeepFashion Synthesis [21], DeepFashion Fine-
grained Attribute [21], CelebA [22] and CelebA-HQ [17]. On CelebA and CelebA-
HQ, we group the attributes into 8 attribute categories and 21 attribute
values following [2]. See Table 1 for detailed statics of each dataset.

###  4.1 Implementation Details

We use the model architecture in [37] as our backbone on DeepFashion and
CelebA datasets. Following [35], on CelebA-HQ we adopt another backbone:
StyleGAN2 [14], which is better suited to high resolution images. To improve
training stability, we froze the weights of StyleGAN2’s encoder and generator.
Except for CelebA-HQ, the CNN used in Eq. (12) is a ResNet-50 model [10]
pretrained on image attribute classification task. On CelebA-HQ, we use
StyleGAN2’s encoder as the CNN in order to match the identity loss defined in
[35].

For DeepFashion the target attributes are uniformly and randomly sampled from
items in the same clothing category, _e.g_., dress and leggings. Here we did
not sample the target attributes from the whole dataset because some annotated
attributes can only appear in certain clothing categories. For example,
leggings can’t have V-neckline, and skirts can’t have long sleeve. On CelebA-
HQ, we traverse the values of the 8 attributes for each image during test for
fair comparison with existing methods that use binary attribute values [33,
35, 26].

###  4.2 Experimental Settings

Baselines. We compare our model with related approaches on attribute
manipulation: StarGAN [7], AMNet [3], FLAM [27], VPTNet [16], AttGAN [11],
Student [18], FSNet-v2 [2], CAFE-GAN [15], InterfaceGAN [26], L2M-GAN [33] and
LatentTransformer [35]. Among these approaches, FLAM, Student, InterfaceGAN
and L2M-GAN also aim to achieve feature-level disentanglement, while AMNet,
FSNet-v2 and CAFE-GAN aim to learn spatially disentangled representations. For
StarGAN, AttGAN, Student, InterfaceGAN, L2M-GAN and LatentTransformer, we used
the official implementations at author-provided links. AMNet and FLAM are
reproduced by us following the configurations provided in the corresponding
papers. Results of FashionSearchNet-v2, VPTNet and CAFE-GAN are directly
copied from the original papers. For fair comparison on CelebA-HQ, we used
StyleGAN2 encoded image features in InterfaceGAN.

Evaluation metrics. Following [3, 2], we use human evaluation and two standard
metrics to evaluate the model’s performance on attribute manipulation:
attribute manipulation accuracy and top-k retrieval. Attribute manipulation
accuracy, which is the classification accuracy of the target attribute on the
manipulated images, measures the extent to which a model can modify the target
attribute. We use a ResNet-50 model [10] pretrained on attribute
classification to evaluate the attribute manipulation accuracy on DeepFashion
and CelebA. On CelebA-HQ, the accuracy is computed using the same facial
attribute classifier as [35] for a fair comparison. Top-k retrieval, on the
other hand, evaluates both the attribute changing and preservation capability.
It is defined as the number of hits divided by the total number of queries. A
query is called a hit if any of the manipulated image’s top-k matches has
exactly the target attributes in $\mathbf{\bar{A}}$. The top-k retrieval rate
is averaged across all attributes. In all experiments, we use the deep
features in the last fully-connected layer of the attribute classifier for
image retrieval. All retrieval galleries have 20,000 images.

###  4.3 Quantitative Results

As shown in Table 2-5, AIRR outperforms the state-of-the-art by a significant
margin on most evaluation metrics. For example, Table 5 shows the average
attribute manipulation accuracy and top-k retrieval rates on CelebA-HQ, which
boost performance by more than 20% compared to existing methods. The
improvements are more obvious on additive attributes, such as _wearing hat_ ,
which reports gains over prior work by more than 40% in Table 4 and 5. Further
discussion on ablations of our model can be found in Section 4.5.

Table 2: Results on DeepFahison Synthesis. In our models,
$\lambda_{1}=0.25,\lambda_{2}=0.125,\lambda_{3}=1.0,\lambda_{4}=1.0$

Method | Manipulation Accuracy | Top-K Retrieval
---|---|---
Color |  Sleeve |  Avg. |  R@5 |  R@20
StarGAN [7] |  70.4 |  77.2 |  73.8 |  71.1 |  82.9
AMNet [3] |  74.4 |  82.1 |  78.3 |  85.1 |  90.5
AttGAN [11] |  80.2 |  91.0 |  85.6 |  90.6 |  95.2
FLAM [27] |  - |  - |  - |  26.7 |  41.3
VPTNet [16] |  - |  85.7 |  - |  - |  -
AIRR (w/o mask) |  88.8 |  92.2 |  90.5 |  94.4 |  97.1
AIRR (w/o $L_{d}$) |  93.9 |  89.5 |  91.7 |  95.0 |  97.3
AIRR (w $L_{h}$) |  89.4 |  90.5 |  90.0 |  90.8 |  92.4
AIRR |  94.1 |  96.5 |  95.3 |  97.6 |  98.8

Table 3: Results on DeepFahison Fine-grained Attributes. In our models,
$\lambda_{1}=0.05,\lambda_{2}=0.125,\lambda_{3}=2.0,\lambda_{4}=1.0$

Method | Manipulation Accuracy | Top-K Retrieval
---|---|---
Pattern |  Sleeve |  Length |  Neckline |  Material |  Style |  Avg. |  R@5 |  R@20
StarGAN [7] |  54.0 |  38.7 |  22.4 |  44.3 |  47.2 |  24.6 |  38.5 |  25.1 |  39.3
AttGAN [11] |  47.4 |  31.5 |  19.0 |  33.7 |  40.3 |  23.5 |  32.6 |  28.3 |  39.1
AMNet [3] |  53.6 |  56.5 |  24.4 |  68.3 |  47.9 |  27.4 |  46.4 |  44.1 |  47.5
FLAM [27] |  - |  - |  - |  - |  - |  - |  - |  17.6 |  29.8
AIRR (w/o mask) |  70.7 |  44.3 |  19.0 |  57.3 |  55.0 |  25.9 |  45.4 |  38.2 |  52.4
AIRR (w/o $L_{d}$) |  89.1 |  60.6 |  31.8 |  73.5 |  74.9 |  34.2 |  60.8 |  56.4 |  68.7
AIRR (w $L_{h}$) |  85.1 |  59.2 |  25.7 |  72.4 |  72.6 |  34.7 |  58.2 |  51.1 |  60.2
AIRR |  87.8 |  65.9 |  32.2 |  74.5 |  76.3 |  35.8 |  62.1 |  57.7 |  70.3

Attribute preservation analysis. To analyze what influence that changing a
specific attribute has on preserving others, we gradually increase the ratio
of manipulated images (_i.e_., the number of manipulated images divided by the
number of all test images), and observe the ratio of successfully preserved
attributes. Figure 3 provides the attribute changing rate (_i.e_., attribute
manipulation accuracy) vs. attribute preservation rate for each attribute in
the DeepFashion dataset. In each graph, the preservation rates are averaged
over all attributes excluding the target attribute. In Figure 3, our method
achieves the highest preservation rate under the same attribute changing rate,
proving its capability of controllable attribute manipulation as well as
preservation.

Table 4: Results on CelebA. In our models,
$\lambda_{1}=0.5,\lambda_{2}=0.5,\lambda_{3}=1.0,\lambda_{4}=1.0$

Method | Manipulation Accuracy | Top-K Retrieval
---|---|---
Hair Color |  Beard |  Hair Type |  Smiling |  Eyeglasses |  Gender |  Hat |  Age |  Avg. |  R@5 |  R@20
StarGAN [7] |  55.2 |  51.6 |  35.6 |  64.0 |  86.1 |  36.5 |  6.4 |  44.9 |  38.8 |  39.9 |  55.2
Student [18] |  47.7 |  43.5 |  37.3 |  60.0 |  12.1 |  42.7 |  11.5 |  39.8 |  36.8 |  38.2 |  53.9
AMNet [3] |  58.6 |  34.4 |  26.7 |  43.7 |  10.0 |  21.0 |  13.0 |  22.4 |  28.7 |  33.1 |  46.0
AttGAN [11] |  72.6 |  88.5 |  48.1 |  79.8 |  94.7 |  89.7 |  21.7 |  60.6 |  69.5 |  72.4 |  86.5
CAFE-GAN [15] |  83.6 |  40.1 |  - |  - |  - |  95.2 |  - |  88.6 |  - |  - |  -
FSNet-v2 [2] |  - |  - |  - |  - |  - |  - |  - |  - |  - |  68.0 |  77.5
AIRR(w/o mask) |  86.1 |  96.0 |  58.5 |  92.7 |  98.9 |  91.3 |  75.6 |  64.9 |  83.0 |  86.5 |  94.3
AIRR (w/o $L_{d}$) |  74.3 |  93.9 |  51.4 |  92.4 |  99.4 |  89.5 |  83.4 |  69.7 |  81.8 |  87.1 |  94.9
AIRR |  75.9 |  96.4 |  58.4 |  94.8 |  99.1 |  91.6 |  93.1 |  80.6 |  86.2 |  89.1 |  95.6

Table 5: Results on CelebA-HQ. In our models,
$\lambda_{1}=0.25,\lambda_{2}=0.125,\lambda_{3}=20.0,\lambda_{4}=10.0$

Method | Manipulation Accuracy | Top-K Retrieval
---|---|---
Hair Color |  Beard |  Hair Type |  Smiling |  Eyeglasses |  Gender |  Hat |  Age |  Avg. |  R@5 |  R@20
InterfaceGAN [26] |  38.4 |  80.8 |  36.4 |  97.7 |  29.7 |  55.8 |  2.9 |  42.0 |  48.0 |  19.1 |  38.8
LatentTrans [35] |  37.0 |  78.8 |  48.9 |  85.3 |  49.6 |  62.2 |  5.6 |  47.4 |  51.9 |  20.7 |  41.8
L2M-GAN [33] |  - |  - |  - |  89.7 |  - |  - |  - |  - |  - |  - |  -
AIRR(w/o mask+$L_{d}$) |  40.8 |  73.1 |  50.7 |  93.2 |  63.1 |  71.4 |  40.1 |  83.3 |  64.5 |  51.3 |  67.9
AIRR(w/o mask) |  54.8 |  76.9 |  58.4 |  95.4 |  88.2 |  79.0 |  49.2 |  88.0 |  73.7 |  60.9 |  75.5

Figure 3: Attribute changing rate vs. attribute preservation rate. The
interval of $y$ axis is made to be unequal for better visualization purposes

Table 6: A/B user judgements for attribute manipulation correctness on the
DeepFashion Fine-grained Attribute dataset

AIRR/StarGAN | AIRR/AttGAN | AIRR/AMNet
---|---|---
65%/35% | 61%/39% | 54%/46%

Table 7: A/B user judgements for attribute manipulation correctness on the
CelebA-HQ dataset

AIRR/InterfaceGAN | AIRR/LatentTrans
---|---
76%/24% | 70%/30%

User study. We also conducted human evaluation experiments on DeepFashion
Fine-grained Attribute and CelebA-HQ using Amazon Mechanical Turk service to
verify the quality of manipulated images. We tested on 50 images in each
dataset, and different 5 worker were assigned per image. Each worker was
presented 3 pictures: the original image, the manipulated image produced by
AIRR, and the manipulated image generated by a randomly chosen baseline
approaches in Table 7 or 7. The worker was asked to pick an image that better
converts the specified attribute in the given image. Table 7 shows that 54-65%
of workers think our method achieves better attribute manipulation on the
DeepFashion Fine-grained Attribute dataset. On CelebA-HQ, reported in Table 7,
70-76% workers voted for AIRR, verifying its improved capability of
manipulating facial attributes compared to prior work.

Figure 4: Qualitative examples on DeepFashion Synthesis. The first column
shows original images Figure 5: Qualitative examples on DeepFashion Fine-
grained Attribute Figure 6: Qualitative examples on CelebA

###  4.4 Qualitative Results

Figure 4-7 presents some qualitative examples on each dataset that we used.
For attributes that are relatively shallow and easy to learn, such as _color_
in Figure 5, all methods perform well in transforming the source attribute
into the target attribute. Whereas for attributes with relatively more
complicated semantics, _e.g_., _lattice_ in Figure 5, our method better
represents the target attribute in the generated images. In addition, it can
also be observed that due to the information hiding problem, several failures
of previous methods exhibit as visually not altering the original image, such
as AMGAN’s _To Denim_ in Figure 5 and LatentTrans’s _To Hat_ in Figure 7. On
the other hand, although our method avoids using the source attribute
information for manipulation, it’s failures can be more significant due to
excessive manipulation. For example, the last two rows of _To Floral_ in
Figure 5 alter the dresses’ appearance completely.

###  4.5 Model Analysis

Ablations of model components. To demonstrate the contribution of the
components of our proposed framework, we evaluate the performance of our model
without the disentanglement loss and the parsing mask. Tables 2-5 report
quantitative results of ablations of our model. AIRR (w/o $L_{d}$), which
disables the attribute remover, causes losses on average attribute
manipulation accuracy and top-5 retrieval on all four datasets. Especially in
CelebA-HQ, AIRR (w/o $L_{d}$) losses 9% accuracy compared to AIRR. This
suggests that disentangling attribute information by decorrelation is
effective in image manipulation. We also explored what the generated images
would look like when injecting no attribute, _i.e_., setting the target
attribute’s scale embedding vector to be $\mathbf{1}$ and the bias embedding
vector to be $\mathbf{0}$. This way we can check if the model is hiding source
attribute information in the encoded features. If it suffers from information
hiding, then the source attributes should been seen in generated images. In
Figure 8(a), without the proposed disentanglement loss, most source
attributes, including material and pattern, indeed appear in the generated
images. With the proposed attribute excluded representation, all these
attribute information is successfully removed, avoiding the information hiding
problem suffered by prior work.

In the meanwhile, AIRR (w/o mask), which removes the parsing mask along with
the second encoder, also degrades the evaluation metrics as seen in TAbles
2-5. This indicates the importance of concentrating manipulation on the object
of interest. However, we note that even without the parsing mask, our approach
still outperforms prior work on most metrics. Note that in the two ablations
of CelebA-HQ, we didn’t add the parsing mask in order to reduce the
computational costs for generating high resolution images.

We also tried replacing the proposed disentanglement loss with the honesty
loss in [4], which was introduced to avoid the general information hidden
problem when using cycle consistency. In Table 2 and 3, AIRR still outperforms
AIRR (w $L_{h}$) that adopts the honesty loss, suggesting that the proposed
disentanglement loss is more targeted to the attribute manipulation task. See
the Supplementary for more ablation results for hyperparameters used by our
model.

Interpolation of attribute values.  Linear interpolating between different
attribute embedding vectors $\bm{\beta_{a_{i}}}$ and $\bm{\gamma_{a_{i}}}$
corresponds to an interpolation between different values of the target
attribute. Take ”smile” for example, Figure 8(b) gives the outputs of
interpolating from the _not smiling_ embedding vector to the _smiling_
embedding vector. Let $c$ be the weight (_i.e_., interpolation coefficient) of
the target attribute _smiling_. The smile in generated images gradually builds
up as $c$ increases, showing a continuous control over the attribute strength.

Figure 7: Qualitative examples on CelebA-HQ

(a) Examples on images without target attributes

(b) Examples on controlling attribute strength

(c) Examples on controlling multiple attributes

Figure 8: Visualizations used for model analysis in Sec. 4.5

Controlling multiple attributes in one forward pass.  In some prior work,
_e.g_., [35, 31, 33], multi-attribute editing is often accomplished by
sequential manipulation, _i.e_., edit one attribute at a time. In contrast,
AIRR is capable of changing multiple attributes in a single forward-pass by
directly specifying the input target attributes in $\mathbf{\bar{A}}$. Figure
8(c) gives some examples on CelebA-HQ. Even manipulating 3 or 4 attributes at
the same time, our model is able to edit only the specified attributes without
influencing other information in the image.

##  5 Conclusion

In this paper, we propose Attribute Information Removal and Reconstruction
(AIRR) network for image editing. The attribute information removal and
reconstruction module in AIRR produces an attribute excluded representation,
eliminating sources of information hiding suffered by prior work. Results on
four diverse datasets including DeepFashion Synthesis, DeepFashion Fine-
grained Attribute, CelebA and CelebA-HQ, report that our model improves
attribute manipulation accuracy and top-k retrieval rate by 10% on average
over prior work. A user study also demonstrates that images with attributes
manipulated with our approach are preferred in up to 76% of cases. One
direction for future work is to explore controllable attribute manipulation in
unsupervised setting.

Acknowledgements. This material is based upon work supported, in part, by
DARPA under agreement number HR00112020054 and the National Science Foundation
under Grant No. DBI-2134696. Any opinions, findings, and conclusions or
recommendations expressed in this material are those of the author(s) and do
not necessarily reflect the views of the supporting agencies.

## References

  * [1] Abdal, R., Qin, Y., Wonka, P.: Image2stylegan++: How to edit the embedded images? In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2020)
  * [2] Ak, K.E., Lim, J.H., Sun, Y., Tham, J.Y., Kassim, A.A.: Fashionsearchnet-v2: Learning attribute representations with localization for image retrieval with attribute manipulation. arXiv preprint arXiv:2111.14145 (2021)
  * [3] Ak, K.E., Lim, J.H., Tham, J.Y., Kassim, A.A.: Attribute manipulation generative adversarial networks for fashion images. In: Proceedings of the IEEE International Conference on Computer Vision (2019)
  * [4] Bashkirova, D., Usman, B., Saenko, K.: Adversarial self-defense for cycle-consistent GANs. arXiv preprint arXiv:1908.01517 (2019)
  * [5] Burns, A., Sarna, A., Krishnan, D., Maschinot, A.: Unsupervised disentanglement without autoencoding: Pitfalls and future directions. arXiv preprint arXiv:2108.06613 (2021)
  * [6] Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., Abbeel, P.: Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In: Proceedings of the International Conference on Neural Information Processing Systems (2016)
  * [7] Choi, Y., Choi, M., Kim, M., Ha, J., Kim, S., Choo, J.: StarGAN: Unified generative adversarial networks for multi-domain image-to-image translation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018)
  * [8] Choi, Y., Uh, Y., Yoo, J., Ha, J.W.: StarGAN v2: Diverse image synthesis for multiple domains. In: Proceedings of the IEEE conference on computer vision and pattern recognition (2020)
  * [9] Härkönen, E., Hertzman, A., Lehtinen, J., Paris, S.: GANSpace: Discovering interpretable GAN controls. In: Proceedings of the IEEE Conference on Neural Information Processing Systems; (2020)
  * [10] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016)
  * [11] He, Z., Zuo, W., Kan, M., Shan, S., Chen, X.: AttGAN: Facial attribute editing by only changing what you want. IEEE transactions on image processing 28(11), 5464–5478 (2019)
  * [12] Hou, Y., Vig, E., Donoser, M., Bazzani, L.: Learning attribute-driven disentangled representations for interactive fashion retrieval. In: Proceedings of the IEEE International Conference on Computer Vision (2021)
  * [13] Hu, Q., Szabó, A., Portenier, T., Favaro, P., Zwicker, M.: Disentangling factors of variation by mixing them. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018)
  * [14] Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2019)
  * [15] Kwak, J.g., Han, D.K., Ko, H.: CAFE-GAN: arbitrary face attribute editing with complementary attention feature. In: Proceedings of the European Conference on Computer Vision (2020)
  * [16] Kwon, Y., Petrangeli, S., Kim, D., Wang, H., Swaminathan, V., Fuchs, H.: Tailor me: An editing network for fashion attribute shape manipulation. In: Proceedings of the IEEE Winter Conference on Applications of Computer Vision (2022)
  * [17] Lee, C.H., Liu, Z., Wu, L., Luo, P.: MaskGAN: Towards diverse and interactive facial image manipulation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2020)
  * [18] Lezama, J.: Overcoming the disentanglement vs reconstruction trade-off via jacobian supervision. In: International Conference on Learning Representations (2018)
  * [19] Li, P., Xu, Y., Wei, Y., Yang, Y.: Self-correction for human parsing. IEEE Transactions on Pattern Analysis and Machine Intelligence (2020). https://doi.org/10.1109/TPAMI.2020.3048039
  * [20] Liu, X., Thermos, S., Valvano, G., Chartsias, A., O’Neil, A., Tsaftaris, S.A.: Measuring the biases and effectiveness of content-style disentanglement. In: Proceedings of the British Machine Vison Conference (2021)
  * [21] Liu, Z., Luo, P., Qiu, S., Wang, X., Tang, X.: DeepFashion: Powering robust clothes recognition and retrieval with rich annotations. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (2016)
  * [22] Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: Proceedings of International Conference on Computer Vision (2015)
  * [23] Locatello, F., Bauer, S., Lucic, M., Raetsch, G., Gelly, S., Schölkopf, B., Bachem, O.: Challenging common assumptions in the unsupervised learning of disentangled representations. In: International Conference on Machine Learning (2019)
  * [24] Mao, X., Li, Q., Xie, H., Lau, R.Y., Wang, Z., Paul Smolley, S.: Least squares generative adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision (2017)
  * [25] Ramesh, A., Choi, Y., LeCun, Y.: A spectral regularizer for unsupervised disentanglement. In: International Conference on Machine Learning (2018)
  * [26] Shen, Y., Gu, J., Tang, X., Zhou, B.: Interpreting the latent space of GANs for semantic face editing. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2020)
  * [27] Shin, M., Park, S., Kim, T.: Semi-supervised feature-level attribute manipulation for fashion image retrieval. In: Proceedings of the British Machine Vison Conference (2019)
  * [28] Shoshan, A., Bhonker, N., Kviatkovsky, I., Medioni, G.: GAN-control: Explicitly controllable GANs. In: Proceedings of the IEEE International Conference on Computer Vision (2021)
  * [29] Szabo, A., Hu, Q., Portenier, T., Zwicker, M., Favaro, P.: Understanding degeneracies and ambiguities in attribute transfer. In: Proceedings of the European Conference on Computer Vision (2018)
  * [30] Usman, B., Bashkirova, D., Saenko, K.: Disentangled unsupervised image translation via restricted information flow (2021)
  * [31] Wang, R., Chen, J., Yu, G., Sun, L., Yu, C., Gao, C., Sang, N.: Attribute-specific control units in StyleGAN for fine-grained image manipulation. In: Proceedings of the ACM International Conference on Multimedia (2021)
  * [32] Wu, Z., Lischinski, D., Shechtman, E.: StyleSpace analysis: Disentangled controls for StyleGAN image generation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2021)
  * [33] Yang, G., Fei, N., Ding, M., Liu, G., Lu, Z., Xiang, T.: L2M-GAN: Learning to manipulate latent space semantics for facial attribute editing. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2021)
  * [34] Yang, X., Song, X., Han, X., Wen, H., Nie, J., Nie, L.: Generative attribute manipulation scheme for flexible fashion search. In: Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval (2020)
  * [35] Yao, X., Newson, A., Gousseau, Y., Hellier, P.: A latent transformer for disentangled face editing in images and videos. In: Proceedings of the IEEE International Conference on Computer Vision (2021)
  * [36] Yu, C., Wang, J., Peng, C., Gao, C., Yu, G., Sang, N.: Bisenet: Bilateral segmentation network for real-time semantic segmentation. In: Proceedings of the European Conference on Computer Vision (2018)
  * [37] Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision (2017)

Supplementary

##  1 Proof of the Upper Bound for Mutual Information

In Eq. (2) of the paper, we claimed that the mutual information between the
source attributes ${{\mathbf{a}}_{i}}$ and the attribute excluded features
$R(E_{1}(I_{cl}))$ is upper bounded by the maximum log probability in the
attribute distribution. We prove this claim in the following.

Let $\text{MI}({{\mathbf{a}}_{i}},R(E_{1}(I_{cl})))$ denote the mutual
information. Replacing $R(E_{1}(I_{cl}))$ with $r$ for convenience gives

$\displaystyle{\text{MI}}({{\mathbf{a}}_{i}},r)$
$\displaystyle=\sum\nolimits_{r}{\sum\nolimits_{{{\mathbf{a}}_{i}}}{p({{\mathbf{a}}_{i}},r)\log\frac{{p({{\mathbf{a}}_{i}},r)}}{{p({{\mathbf{a}}_{i}})p(r)}}}}$
(15)
$\displaystyle=\sum\nolimits_{r}{\sum\nolimits_{{{\mathbf{a}}_{i}}}{p({{\mathbf{a}}_{i}},r)\log\frac{{p({{\mathbf{a}}_{i}}|r)}}{{p({{\mathbf{a}}_{i}})}}}}$
$\displaystyle=\sum\nolimits_{r}{\sum\nolimits_{{{\mathbf{a}}_{i}}}{p({{\mathbf{a}}_{i}},r)[\log
p({{\mathbf{a}}_{i}}|r)-\log p(}}{{\mathbf{a}}_{i}})]$

Since the number of attribute values in ${{\mathbf{a}}_{i}}$ is finite, $-\log
p(\mathbf{a}_{i})$ can be upper bounded by a constant $c,c>0$:

$\displaystyle{\text{MI}}({{\mathbf{a}}_{i}},r)$
$\displaystyle\leq\sum\nolimits_{r}{\sum\nolimits_{{{\mathbf{a}}_{i}}}{p({{\mathbf{a}}_{i}},r)\log
p({{\mathbf{a}}_{i}}|r)+c\sum\nolimits_{r}{\sum\nolimits_{{{\mathbf{a}}_{i}}}{p({{\mathbf{a}}_{i}},r)}}}}\hfill$
(16)
$\displaystyle=\sum\nolimits_{r}{\sum\nolimits_{{{\mathbf{a}}_{i}}}{p({{\mathbf{a}}_{i}},r)\log
p({{\mathbf{a}}_{i}}|r)+c}}\hfill$

In the r.h.s., we can continue upper bounding ${{p({{\mathbf{a}}_{i}}|r)}}$
with the maximum probability in the distribution to make it independent of
${\mathbf{a}}_{i}$:

$\displaystyle{\text{MI}}({{\mathbf{a}}_{i}},r)$
$\displaystyle\leq\sum\nolimits_{r}{\max\limits_{{{\mathbf{a}}_{i}}}\log
p({{\mathbf{a}}_{i}}|r)\sum\nolimits_{{a_{i}}}{p({{\mathbf{a}}_{i}},r)}}+c$
(17) $\displaystyle=\sum\nolimits_{r}{p(r)\max\limits_{{{\mathbf{a}}_{i}}}\log
p({{\mathbf{a}}_{i}}|r)}+c$
$\displaystyle={\mathbb{E}_{r\sim{p(r)}}}[\max\limits_{{{\mathbf{a}}_{i}}}\log
p({a_{i}}|r)]+c,$

where $c$ is a constant. Note that we can not minimize the mutual information
itself because the joint distribution $p(\mathbf{a}_{i},r)$ is intractable.
The tightness of this upper bound depends on the distribution
$p(\mathbf{a}_{i})$ and $p(\mathbf{a}_{i}|r)$. More specifically, larger
$\min_{\mathbf{a_{i}}}p(\mathbf{a_{i}})$ gives smaller constant $c$, and
smaller $\max_{\mathbf{a_{i}}}p(\mathbf{a_{i}}|r)$ reduces the gap. The
equality is reached when $p(\mathbf{a_{i}}|r)$ is an uniform distribution.

To conclude, using an attribute classifier to estimate the above conditional
probability $p({a_{i}}|r)$, we prove that the upper bound is the maximum log
probability in the attribute distribution as in Eq. (2).

##  2 Ablations on Hyperparameters

In Figure 9, we provide the experimental results for setting different values
of the hyperparameters in Eq. (13) and (14) on CelebA. $\lambda_{1}$ to
$\lambda_{4}$ denotes the trade-off parameter for disentanglement, image
attribute prediction, image reconstruction and perceptual loss, respectively.
Figure 9(a) shows the manipulation accuracy, top-5 retrieval and top-20
retrieval rates for each parameter. The reconstruction error has a different
unit of measurement, for which we show its corresponding graph in Figure 9(b).
It can be noticed that increasing the weight (_i.e._ , $\lambda_{2}$) for the
image attribute loss improves the manipulation accuracy, whereas it can hurt
the reconstruction performance. This indicates a trade-off between successful
manipulation and qualitative reconstruction. In the paper, we chose the values
of each trade-off parameter for a balance between these two aspects.

(a) Parameter value v.s. Accuracy. Higher is better

(b) Parameter Value v.s. Reconstruction Error. Lower is better

Figure 9: Results on using different values of the hyperparameters.
$\lambda_{1}$ to $\lambda_{4}$ denotes the trade-off parameters for
disentanglement, image attribute prediction, image reconstruction and
perceptual loss, respectively

Figure 10: Additional examples on manipulating the attribute strength


# CoRL: Environment Creation and Management Focused on System
Integration111Approved for Public Release. Case Numbers: APRS-
RYZ-2023-01-00006

Justin D. Merrick222Deputy Branch Chief, Autonomous Controls Branch, 21300 8th
St. and Benjamin K. Heiner333Behavior Development Lead AACO, Autonomy
Capability Team 3, 2241 Avionics Circle. Air Force Research Laboratory,
Wright-Patterson AFB, OH, 45433  Cameron Long444Analyst, 6800 Cortona Dr Toyon
Research Corporation, Goleta, CA, 93117  Brian Stieber555Deputy Director,
Algorithm/AI Solutions, 6800 Cortona Dr and Steve Fierro666Senior Analyst,
6800 Cortona Dr Toyon Research Corporation, Goleta, CA, 93117  Vardaan
Gangal777Junior Autonomy Engineer,1415 Research Park Dr Jacobs, Dayton, OH,
45432  Madison Blake888Senior Staff Engineer, 209 Madison St. Heron Systems,
Alexandria, VA, 22314  Joshua Blackburn999Lead Engineer, 2611 Commons Blvd,
Ste 150 STR, Beavercreek, OH, 45431

###### Abstract

Existing reinforcement learning environment libraries use monolithic
environment classes, provide shallow methods for altering agent observation
and action spaces, and/or are tied to a specific simulation environment. The
Core Reinforcement Learning library (CoRL) is a modular, composable, and
hyper-configurable environment creation tool. It allows minute control over
agent observations, rewards, and done conditions through the use of easy-to-
read configuration files, pydantic [1] validators, and a functor design
pattern. Using integration pathways allows agents to be quickly implemented in
new simulation environments, encourages rapid exploration, and enables
transition of knowledge from low-fidelity to high-fidelity simulations.
Natively multi-agent design and integration with Ray/RLLib [2] at release
allow for easy scalability of agent complexity and computing power. The code
is publicly released and available at https://github.com/act3-ace/CoRL.

##  1 Introduction

Existing Reinforcement Learning (RL) software libraries can be roughly broken
into three categories and their combinations: environment API, algorithm
implementation, and integration. Integration handles the communication between
the previous two. The introduction of the OpenAI Gym environment API [3] led
to a proliferation of software packages utilizing this API [4]. Despite this
popularity, the Gym API has significant limitations. Most importantly, it
lacks effective tooling for environment creation and alteration and does not
natively support environments with multiple agents, which has led to the
creation of multi-agent specific environment APIs such as PettingZoo [5]. The
Core Reinforcement Learning library (CoRL) straddles the environment API and
integration categories, providing a modular, composable, and hyper-
configurable suite of tools for environment creation101010Code available at:
https://github.com/act3-ace/CoRL. CoRL allows users to rapidly explore complex
design spaces with minimal re-integration. The contributions of CoRL are as
follows:

  1. 1.

A suite of environment and agent creation tools,

  2. 2.

A hyper-configurable environment enabling rapid exploration for task-based RL,

  3. 3.

A flexible framework for developing these environments and the agents that
populate them,

  4. 4.

Tooling to validate configuration files and give useful feedback when files
are mis-configured, and

  5. 5.

Integration pathways enabling a dramatic reduction in development time to
transition agents into a new simulation.

The remainder of this paper compares CoRL to existing RL libraries (Section
2), expands upon the design of CoRL and how it achieves its environment
creation capabilities (Section 3), presents case studies highlighting the
composable and modular nature of CoRL (Section 4), and discusses its potential
impacts and future improvements (Section 5).

##  2 Related Works

Gym [3] is perhaps the best known environment API library currently in use.
Its quantity and variety of environments and its common sense API have made it
ubiquitous. This has also resulted in a large number of RL algorithms and RL
algorithm libraries being designed for use with Gym. Stable Baselines [6],
Tianshou [7], and RLLib [2] are just a couple of the many libraries which have
implementations that are utilize this API. Gym organizes each environment into
a single class structure, implementing four main methods and six attributes.
Of principle importance are the step method, which takes an action as an
argument and runs one time step of the environment’s dynamics, returning the
next observation, the reward for the time step, whether the episode is
complete, and an information dictionary that varies per environment; the reset
method, which sets the environment to an initial state and returns an initial
observation and an information dictionary; the action_space attribute which
describes the type, cardinality, and bounds of the actions required by the
environment; and the observation_space attribute, which returns the type,
cardinality, and bounds of the observations output by the environment. CoRL
uses this interface in its own Environment class and includes full integration
with existing environments that use the Gym API. However, CoRL eschews the
monolithic class structure for a more modular framework that allows users to
easily modify their environments via YAML configuration files. The wrappers
used to modify existing Gym environments have a similar function to these
configuration management tools in CoRL. However, the modular and configurable
nature of observations, rewards, and dones in CoRL allow complex permutations
of a single task, and implementations of new tasks, in a way that is quick,
readable, and repeatable.

Petting Zoo [5] is another environment API which extends the capabilities of
Gym to multiple agents and introduces the Agent Environment Cycle Game in
which agents act and receive reward in sequential manner. CoRL is also
natively multi-agent and capable of handling agent death and exposing complex
reward attribution. Unified Distributed Environment (UDE) [8] is also multi-
agent capable and seeks to improve on the modularity of simulation
environments through its bridge component. This is similar in concept to the
CoRL Simulator class, but UDE does not seek to create a configurable and
composable environment and instead is merely providing an integration tool.
UDE also has environment virtualization capabilities, similar to RLLib, which
CoRL does not seek to emulate. Gymnasium [9] is a fork of Gym with a slightly
different API that differentiates between "terminated", meaning an agent met
its goal or violated a constraint, and "truncated", meaning an episode met its
time limit without reaching its goal or violating a constraint. These
semantics are mirrored in CoRL through the use of a DoneStatusCode and a
dictionary of Done functors.

The Unity ML-Agents Toolkit [10] spans all three categories of library, but in
a more closed ecosystem. It uses a similar structure to and implements many of
the features of CoRL. The Agents in the ML-Agents Learning Environment are
analogous to CoRL Platforms, while the Behaviors are analogous to instances of
the Policy class. Sensors exist in both CoRL and Unity ML-Agetns in a very
similar implementation. This level of modularity is also extended to the done
state and the rewards, allowing access to the rewards and done states of
individual agents. The differences between Unity ML-Agents and CoRL lie in the
ability of CoRL to use any simulation environment, at the cost of some
integration work, and in the Glue structure of CoRL being much more flexible
and composable than the Sensor structure in Unity ML-Agents. CoRL also does
not attempt to implement any RL algorithms, relying on other libraries to
implement those in a manner consistent with the Environment class being used.

The DeepMind Control Suite (dm_control) [11] is a popular environment creation
tool that pairs the MuJoCo [12] physics library with a Python front-end that
allows user to assemble new environments and tasks in ways very similar to
CoRL. The XML specifications of the environment used in dm_control provide
precise control over the physics and objects in an environment, while CoRL’s
YAML configuration files provide the same precise control and composability
over the observations, rewards, actions, and dones. dm_control provides
similar precise control over rewards as well by the implementation of their
tolerance() function. CoRL takes a different approach in implementing some
simple reward classes and providing a framework for the user to implement
their own reward functions. The API utilized by dm_control is different from
that of Gym, which CoRL utilizes. The return from the step function returns
information about the type of episode termination, similar to that of
Gymnasium, which CoRL implements with its DoneStatusCode. The focus of
dm_control is on the creation of a simulation, while the focus of CoRL is on
the management and manipulation of the outputs of that simulation. The
slightly different API used by dm_control means that fewer algorithm libraries
support its use or do so through the use of wrappers which enforce conformity
with the Gym API.

SuperSuit [13] is an integration tool between environments and algorithms that
implements a number of more complex wrappers to Gym environments, such as
frame stacking or delaying observations. All of these wrappers use the Gym or
Gymnasium API and their functions can be or are replicated in CoRL.

##  3 Design

CoRL is designed to be modular, composable, and hyper-configurable. It
accomplishes these goals through the use of pydantic[1] validators which
specify the construction of YAML configuration files, a stateful functor
design pattern for Glues, Rewards, and Termination/Goal Critiera (i.e. Dones),
an Agent and Platform abstraction which allow hierarchies of agents to emerge
naturally, and a pluggable simulator interface. An environment class is used
to manage each of these constructs and ensure final compatibility with the
training algorithm’s interface. Fig. 1 shows a high level view of CoRL.

Figure 1: A high Level view of CoRL showing the default information flows.
(from left to right) The Training Framework is external to CoRL and must be
compatible with the Environment class. The Environment class houses and
manages all other classes. The Agent instances house Reward Functors that
utilize the observations provided by Glues to construct the observation passed
to the Training Framework. Glues utilize the measurements of the Sensors of
Platforms or pass the controls to the Platform’s Controller(s). Done Functors
evaluate Platform state and report the done status to the Environment.
Platforms are managed by the Simulator, which passes data to and from the
Simulation, which is external to CoRL.

###  3.1 Validators and Functors

Each class in CoRL has its own validator. These serve three purposes. The
first is to provide typing requirements for the class’s attributes. These
serve to guide a user in constructing a YAML configuration file for the class.
Examples of this pairing between validator and configuration file are shown in
Section 4. Second, the validator is used to perform functional checks on the
inputs provided. Third, the validators are used to interpret the configuration
arguments to generate values and complex datatypes that enable the
composability of CoRL Glues, Rewards, and Dones. This last purpose is
accomplished through the use of the Episode Parameter Provider (EPP), the
Reference Store (Section 3.4), and CoRL’s unit library. The EPP and Reference
store supply values from a distribution to the functors. The unit library
automatically converts units based on the configuration arguments.
Specifically, the units of a functor must match the type (e.g. velocity,
distance) of unit used in any wrapped or referenced classes. In addition to
implementing a number of base and derived physical units, CoRL also implements
and enforces fractions and percentages, as well as a None unit type.

The stateful functor design pattern used for CoRL’s Glues, Rewards, and Dones
combines naturally with the validators to allow a single configuration file
entry to direct the composition of multiple functors. These create a directed
acyclic graph of functors, which is used to reduce the number of instances of
identical functors. Additionally, this functor design allows the user to
operate at their preferred level of detail. A single functor can be created to
do a complex manipulation of platform state, or a chain of functors can be
used to accomplish the same calculation, allowing reuse. The statefulness of
CoRL functors also allows communication to flow between the three main types
of functors (Glues, Rewards, and Dones). Each of these classes and their
interactions is explained in greater detail in Sections 3.5 and 3.6.

###  3.2 Simulator

The Simulator class allows the CoRL Environment class (Section 3.8) to
modularly interact with a variety of simulations of varying complexity with
minimal reintegration. At its core a Simulator class looks very similar to a
Gym Environment, implementing a reset and step method to reset or advance the
simulation dynamics; however, the Simulator has attributes for frame_rate,
sim_time, and platforms. These attributes allow for the creation, destruction,
or unresponsiveness of Platforms (Section 3.3) to be communicated to the
Environment. The similarity to the Gym API allows for environment dynamics to
be implemented directly in the Simulator class for simple problems or where no
external simulation is present or necessary.

The minimal reintegration is achieved through a plugin library of Platform
Parts. This allows for a rapid and principled transition from virtual
simulation, to bench testing, to real-world testing. A Simulator class is
defined for a virtual simulation, with a separate Simulator defined for both
bench testing and real-world testing. By implementing the same types of Parts
for each Simulator, the engineer can be confident that the measurements that
the Glues create are consistent across these implementations.

###  3.3 Platforms

Platforms are an important piece of the modularity of CoRL. A platform is an
abstraction of a piece of the environment that has a measurable or modifiable
state. In the simplest implementations, this can be the entire environment.
CoRL uses this implementation for default interaction with existing Gym
environments. A more physical interpretation might be in an orbital dynamics
problem where a platform is defined as a spacecraft. In this example, the
Platform class can be used to define the core spacecraft states such as its
position and velocity, as well as how to extract them from the Simulator
class.

In all cases, Platforms have some combination of Sensors and Controllers,
which are collectively called Parts. Parts dictate how the observation or
modification of Platform states are achieved, with Sensors observing the state
of the platform for downstream use by Glues and Controllers allow modifying
the controllable states of the Platform in order to affect its future state,
as determined by the dynamics of the Simulator. Fig. 2 shows how this sequence
is used in the step method of the environment to construct the observation for
a single agent occupying a single platform.

In order to minimize re-integration efforts in the case where the simulation
dynamics change or the platform changes, Parts have properties which define
the limits, cardinality, and units of the space in which they are valid,
similar to the spaces provided in Gym. Additionally, Parts are added to a
plug-in library. This library registers a Part with a specific group name and
conditions that must be met to invoke the Part. For example, a condition might
require a specific Platform type or Simulator class be used. The functionality
behind these constructs is that Parts can be added to an Agent’s configuration
file using the group name. That group name is then referenced with the plug-in
library and the conditions for using the Parts in that group are checked until
a match is found. Then, if the Simulator or Platform is changed, the Agent
configuration file need not change, as the plug-in library search will select
the appropriate Part to attach to the Platform or Platforms the Agent is
attached to. This modularity of Parts also creates a natural pathway to
dealing with asynchronous or missing measurement problems in a localized
fashion. CoRL Sensors implement a value hold by default in the case of missing
or malformed measurements.

Figure 2: Sequence diagram for creation of an observation in the step function
of CoRL’s environment

###  3.4 Episode Parameter Provider and Reference Store

The Episode Parameter Provider (EPP) enables the environment to modify its
parameters, such as the initial state, on a per-episode basis throughout the
training process. These modifications can happen using two separate
mechanisms. First, the parameters returned by the EPP are actually
distributions rather than values, which the environment then samples to get
the parameter value. This allows the user to interject randomness into the
training process by using different values each episode. CoRL natively
provides parameters that implement constant, uniform, truncated Gaussian, and
discrete choice distributions. Furthermore, there is an abstract interface
that allows users to implement other distributions by extending the base
parameter class.

Second, the EPP contains hooks that allow it to modify the distribution of
parameters as training progresses. When defining the original parameter
distributions, the user can specify updaters on the distribution
hyperparameters, for example to increment a hyperparameter by a fixed step
size. At the end of each training iteration, the environment passes the
training result to the episode parameter provider. This allows it to modify
its internal data, such as calling the updater on the parameter distributions
that it will return to the environment. By creating subclass implementations
of the episode parameter provider, the user is able to implement various forms
of curriculum learning, such as domain randomization [14, 15] or automatic
domain randomization [16].

As an example use of the EPP, consider an extension to Cartpole [17] where the
height of the pole could be varied for each episode. In order to solve
Cartpole with an arbitrary height, the training might use the episode
parameter provider to specify this height as a uniform parameter with some
bounds. Furthermore, the upper bound on the distribution could grow larger
using an updater as part of a curriculum learning scheme.

The EPP is able to provide parameters to glues, rewards, dones, simulators,
and simulator platform initialization. If multiple elements require the same
parameter (such as a done termination value and a reward that computes
distance from that termination value), the parameter can be put into the
Reference Store. Then the relevant objects can all reference the same value
after the parameter has been sampled by the environment. The EPP is
implemented as a Ray Actor [18, 19], meaning CoRL has a strong dependency on
Ray.

###  3.5 Glues

Glues are a flexible integration class used to communicate and transform
information from the Platforms (and their Parts) to the Agents (Section 3.7)
and vice versa. When communicating from Platform to Agent, the glue instances
construct the observation that the Agent will use to calculate the rewards and
done, and which the Policy will use to determine what action(s) to take. When
communicating from Agent to Platform, the glue instances construct the action
that the Platform, and ultimately the Simulator, can use to propagate the
dynamics.

Due to their composable nature, Glues can vary greatly in complexity and may
use the environment state, platform state and sensors, reference store
parameters, and other glues in their calculations. Several base types of Glues
are included in CoRL. For the most basic functionality, the ObserveSensor and
ControllerGlue Glues connect Platform Sensors and Controllers to the Agent and
the Policy. To aid in Glue composition, wrapper Glues that can wrap one or
more other Glues and, in the case of wrapping multiple Glues, store them in
list or dictionaries, are also included. Other basic mathematical functions
such as unit vector transformation, norm calculation, projection, and
differencing are included. Examples Glues and their compositions are shown in
Section 4.

To take advantage of the directed acyclic graph created by wrapped Glues (and
eventually by Rewards and Dones, Section 3.6, as well), Glues may optionally
use an Extractor class. The extractor simplifies that configuration YAML and
is used to topologically sort the dependencies of the CoRL functors. This
reduces memory requirements and improves computation speed by eliminating
identical instances of a Glue.

###  3.6 Rewards and Dones

Figure 3: CoRL Reward types: (1) Base types - general API shared by all
rewards, (2) Task Rewards - Specific rewards utilized by the environments in
Section 4, (3) Core Types - General task independent rewards, (4) Specialized
Core Types - Specific reward implementations that provide shaping or advanced
logic for activation.

Dones are a class of functors that test whether their criteria have been met
and, if so, mark their scope done and issue a status code to indicate whether
this constitutes a win, partial win, draw, partial loss, or loss. The scope of
a particular Done can be a single agent or the environment as a whole. These
different scopes allow for a variety of done conditions to hold. For instance,
the default behavior in CoRL does not end an episode until each agent has had
a Done evaluate True. Through a configuration file, this can be changed to end
an episode as soon as any agent is done. Furthermore, with environment Dones,
called Shared Dones and which have a slightly different call signature, the
episode can be terminated regardless of the done status of particular agents.

Similar to Dones, Rewards are a class of functors that, when evaluated,
produce a scalar value for a particular component of an agent’s reward. The
rewards for each agent are eventually summed in the Environment to produce the
scalar reward that the Gym API expects, but, by calculating the component
rewards individually, and reporting them to Tensorboard, analysis of an
agent’s general behaviors can be conducted analyzed across training, similar
to the process proposed in [20]. Rewards are specifically calculated after
Dones in order to allow rewards to utilize the done status code created by any
Done functor that evaluates to True.

As with all functors in CoRL, both Rewards and Dones are composable and
configurable. Fig. 3 shows some examples of Rewards and Dones and how they can
compose. Fig. 4 shows a sequence diagram of the processing of the Dones and
Rewards as implemented in CoRL’s Environment class (Section 3.8). Similar to
Glues, Rewards and Dones may also take advantage of the graph of functors
through the Extractor class.

Figure 4: Sequence diagram for creation of rewards and dones in the step
function of CoRL’s environment

###  3.7 Agents and Policies

The CoRL Agent class implements an agent as described in [21]. Each instance
of an agent may partially or fully control any number of platforms and
contains a dictionary of Glues, which serve as the agent’s sensors and
actuators. The instance also contains a dictionary of Rewards, which the agent
uses to communicate with a Policy defined in its configuration. The Policy
serves to map the Glue observations to an action.

The structure of a Policy class must be dictated by the algorithm
implementation library and the environment API that CoRL is integrating with.
While integration with RLLib is included in the CoRL release, a Policy
implementation is not limited to that library, nor must it be compatible with
RL in general. Examples of a random action policy and a scripted policy are
included in the CoRL release.

This construction of an agent as a separate entity from a Platform or a Policy
allows a natural progression from single-agent to multi-agent environments,
both competitive and cooperative. It also allows for shared Policys among
Agents, multiple Agents utilizing the same Platform, and a robust means of
developing hierarchies of Agents. Furthermore, it minimizes the re-integration
necessary for transferring agents from learning environments to inference
environments, or from simple dynamics to complex dynamics.

###  3.8 Environment

At the opposite end of CoRL from the Simulator is the Environment class, which
integrates the Agents, Platforms, Simulator, EPP, Reference Store, Glues,
Dones, and Rewards into a cohesive whole that complies with the interfaces
expected by the training algorithm. Included in the CoRL release is a multi-
agent environment that complies with the the RLLib MultiAgentEnv API, a multi-
agent version of the Gym API. While the configuration files specify how the
environment is to be created, the Environment class processes these
configurations to initialize the component parts of the environment.

The environment is also responsible for maintaining any global Parameters and
Dones that are specified in the configuration, such as Parameters that are
applicable to multiple agents, or Dones that deal with the overall state of
the simulation as opposed to any particular Agent or Platform. A simple
example of this is a hard environment horizon. If the episode is terminated
after a fixed number of steps, the meta nature of this done condition means it
is best placed in the environment. In addition, the Environment also checks
the observation(s) obtained from the Agent(s) to ensure that the measurements
lie within the space defined by the Glues. This sanity check can be configured
to check every step, or it can be used to spot-check observations throughout
training.

###  3.9 Visualization/Debugging Tools

CoRL, through its Ray/RLLib integration, utilizes Tensorboard [22] to log and
track metrics and generate visualizations for them. Within CoRL, Tensorboard
has been configured to log and track all rewards, dones, and reference store
parameters. Additionally, the DoneStatusCode and simulator reset parameters
are also logged. All of these measurements are automatically aggregated to
maximum, minimum and mean. All parameters are also logged to a CSV file for
ease of visualization outside of Tensorboard.

The configuration of RLLib, the Environment, the Agents, and the Policies are
also saved for each trial in order to assist in debugging or determining the
differences between runs. These are stored in a JSON file. As some parameters
may change over the course of training, especially those modified by an EPP,
the information logged to Tensorboard, along with the configuration of the
agent and environment for each iteration are logged in a JSON file.

###  3.10 Evaluation Framework

After a policy has been trained, the Evaluation Framework can be used to
investigate the policy’s performance. The evaluation framework follows the
same CoRL design practices, utilizing YAML configuration files and a functor
design pattern. The framework has been constructed to enable three core
processes: Evaluate, Generate Metrics, and Visualize. Each process utilizes
separate YAML configuration files to supply the appropriate inputs and each
may be run separately or in series as an end-to-end pipeline.

Figure 5: The Evaluation Framework

The Evaluate process executes rollouts from a user-defined set of initial
conditions. To achieve this, the evaluation framework captures CoRL agent and
environment definitions and uses them with an engine to execute rollouts. The
engine is specific to the training framework used. At release, CoRL implements
an RLLib engine that is compatible with all Policy classes included in CoRL.
The trajectory resulting from a rollout is contained in an EpisodeArtifact
class which stores agent information such as observations, rewards, dones, and
platform state, as well as environment state information. An EpisodeArtifact
is generated and saved to disk for each member of the set of initial
conditions.

The Generate Metrics process computes metrics from the trajectory data
collected and stored to disk by the Evaluate process. Metric classes using the
same composable functor design pattern as Glues, Rewards, and Dones define the
quantities to be computed. These quantities can be computed directly from a
trajectory, can be aggregated across all trajectories, or can be aggregated
over quantities generated by other metrics. Some example metrics are the
number of times an agent successfully solves the task and the proportion of
total reward each reward functor contributed aggregated across a trajectory.
The evaluation framework provides a layer of abstraction for the type of
quantity returned from a metric functor, categorizing them into two types,
terminal and non-terminal. A terminal metric can be categorized as one which
can be represented as a non-container type (e.g. float, string, integer). A
non-terminal metric returns a container type (e.g. dictionary, list). This
distinction is made to assist in compatibility with non-Python visualizations.
A handful of common abstract metrics are implemented at release, but a user
can craft more specific metrics to suit their needs.

The Visualize process creates visualizations from the computed metrics. A
generic interface is provided that can be implemented to provide a custom
visualization. There are two visualizations provided, a visualization that
prints the metrics computed in a table to stdout and another that creates HTML
plots of the metrics. Multiple visualizations can be included in a single
configuration file for the Visualize process.

##  4 Case Study

Two case studies are included to illustrate the design principles of CoRL. The
first uses the OpenAI Gym CartPole environment as a simulation and uses CoRL’s
GymSimulator and its associated classes to demonstrate the modularity and
composability of CoRL Glues, Rewards, and Dones. The second is an abstraction
of the spacecraft docking problem in which the degrees of freedom have been
limited to one dimension for simplicity. This case study shows how a
simulation environment can be integrated into CoRL without first being set up
as a Gym environment.

###  4.1 CartPole and OpenAI Gym

Several variations on an agent designed to solve the CartPole-v1 Gym
environment are included in CoRL to provide concrete examples of the Gym
integration and Glue composability discussed in Section 3. The simplest
example is shown in Fig. 6. Here, a trainable agent (Line 1) is configured
with two parts, one sensor and one controller (Lines 3-6). The Sensor_State
notation in the configuration file will trigger a search of the plug-in
library for a Sensor with that name, which will yield a sensor that reports
the Gym environment state as output by the step function. A baseline EPP is
included which does not have any parameters (Line 7). Glues are added for each
of the parts (Lines 8-24), each specifying which part they are connecting to.
The ObserveSensor Glue is also configured to not normalize the observations
pulled from the Sensor_State sensor. Finally, Dones and Rewards are added from
other files. Both baseline_dones.yml and baseline_rewards.yml contain a simple
done and reward that report the done and reward provided by the Gym
environment.

⬇

"agent": "corl.agents.base_agent.TrainableBaseAgent"

"config": {

"parts": [

{"part": "Controller_Gym"},

{"part": "Sensor_State"},

],

"episode_parameter_provider": !include baseline_epp.yml,

"glues": [

{

"functor": "corl.glues.common.controller_glue.ControllerGlue",

"config": {

"controller": "Controller_Gym",

},

},

{

"functor": "corl.glues.common.observe_sensor.ObserveSensor",

"config":{

"sensor": "Sensor_State",

"normalization": {

"enabled": False

}

}

},

],

"dones": !include baseline_dones.yml,

"rewards": !include baseline_rewards.yml,

}

Figure 6: CoRL agent file for CartPole-v1 Gym environment

From this basic design it is simple to add complexity to the observation. For
instance, Fig. 7 shows the YAML configuration for adding a Glue which takes
the zero-th index of the state and subtracts it from a target value, in this
case zero, to return the difference between that state and the target value.
Lines 5-8 are functionally identical to Lines 16-20 of Fig. 6, but in this
case, the TargetValueDifference Glue has wrapped the ObserveSensor Glue in a
dictionary with the key "sensor" (Line 3-4). This configuration also includes
information on the resulting observation’s units ("N/A") and the minimum and
maximum measurements expected and allowed, which are used to construct the
Glue’s observation space (Lines 16-20).

⬇

{

"functor": "corl.glues.common.target_value_difference.TargetValueDifference",

"wrapped": {

"sensor": {

"functor": "corl.glues.common.observe_sensor.ObserveSensor",

"config":{

"sensor": "Sensor_State",

"normalization": {"enabled": False}

}

},

},

"config":{

"target_value": 0,

"index": 0,

"unit": N/A,

"limit": {

"minimum": -5000,

"maximum": 5000,

"unit": N/A

}

},

}

Figure 7: Example of Glue composability using the TargetValueDifference and
ObserveSensor Glues

Complexity can also be added to the rewards. Fig. 8 shows the configuration
for a Reward which uses the Glue from Fig. 7 and grants an exponentially
decaying reward for how far the observation of the Glue is from a target
value. This example makes use of the observation extractors discussed in
Section 3.6 to specify which values, spaces, and units to use (Lines 5-7). In
this case, the Glue from Fig. 7 is used. While that observation is already a
difference to a target value, the modularity of the Rewards and Glues allow
the same measurement to be constructed entirely in the reward, or for a
different target value than zero to be rewarded. In this case, the default for
the target_value field is zero and it is not included. Lines 9-12 specify
shaping parameters for the exponential decay as well as how to handle
situations where the observation at time $t+1$ is farther away than the
observation at time $t$.

⬇

{

"name": "OpenAIGymExtractorReward",

"functor":
"corl.rewards.exponential_decay_from_target_value.ExponentialDecayFromTargetValue",

"config": {

"observation": {

"fields": ["ObserveSensor_Sensor_StateDiff", "direct_observation_diff"]

},

"index": 0,

"eps": 1, # THIS IS RADIANS

"reward_scale": .000000000001,

"is_closer": true,

"closer_tolerance": 10,

}

}

Figure 8: Example of Reward complexity using the
ExponentialDecayFromTargetValue Reward

Other variations on the CartPole agent in Fig. 6 are included in the CoRL
repository which highlight additional composability in the Glues and Rewards.
Additionally, examples of non-operable agents, repeated observations, and
agents without the ability to affect the environment are demonstrated. The
integration of Gym wrappers and the use of keyword arguments passed to Gym
environments is also demonstrated.

###  4.2 1D Docking

The 1D Docking problem is implemented with the same structure as used in [23,
24] but represented as a simple linear ODE,

$\dot{\mathbf{x}}=A\mathbf{x}+B\mathbf{u}$ (1)

where the state $\mathbf{x}=\left[x,\dot{x}\right]$, the position and velocity
of the docking craft, the control $\mathbf{u}=[T]$, the thrust of the docking
craft, and

$\displaystyle A=\begin{bmatrix}0&1\\\ 0&0\end{bmatrix}$ $\displaystyle
B=\begin{bmatrix}0\\\ \frac{1}{m}\end{bmatrix},$ (2)

the state and input matrices.

This implementation uses the concept of an Entity class (Deputy1d) which is
roughly analogous to a Platform in CoRL and a Dynamics class
(Docking1dDynamics) which calculates $\dot{\mathbf{x}}$. The task of
implementing this simulation in CoRL then becomes the creation of a Simulator,
a Platform, and several Parts. Implementing Docking1dSimulator requires
creating concrete methods for several abstract methods defined in the CoRL
BaseSimulator, most importantly a reset and a step method. Fig. 9 shows the
reset method. Some of this method, particularly Lines 2-3 and 17-19, will be
typical of most CoRL Simulator classes. Lines 6-10, however, interface
directly with the simulation by constructing and storing the Entity class(es)
associated with the simulation while Lines 12-15 pair the Entity class(es)
with CoRL Platform(s), in particular the Docking1dPlatform created for this
environment.

⬇

def reset(self, config):

config = self.get_reset_validator(**config)

self.clock = 0.0

# construct entities ("Gets the platform object associated with each
simulation entity.")

self.sim_entities = {} # pylint: disable=attribute-defined-outside-init

for agent_id, agent_config in self.config.agent_configs.items():

agent_reset_config = config.platforms.get(agent_id, {})

self.sim_entities[agent_id] = Deputy1D(name=agent_id, **agent_reset_config)

# construct platforms ("Gets the correct backend simulation entity for each
agent.")

sim_platforms = {}

for agent_id, entity in self.sim_entities.items():

agent_config = self.config.agent_configs[agent_id]

sim_platforms[agent_id] = Docking1dPlatform(platform_name=agent_id,
platform=entity, parts_list=agent_config.parts_list)

self._state = BaseSimulatorState(sim_platforms=sim_platforms,
sim_time=self.clock)

self.update_sensor_measurements()

return self._state

Figure 9: reset method of 1D Docking Simulator

The Docking1DPlatform requires a Deputy1D Entity in its validator, and uses
the properties of this Entity as its own properties to pass along the state of
the environment. Docking1DPlatform also stores the actions given to it as
shown in Fig. 2. This allows the step function of the Simulator class, shown
in Fig. 10 to retrieve these applied actions (Line 3), apply them to the
appropriate Entity classes (Lines 4-5) and then update the sensor measurements
on each Platform with the updated state of the Entity (Line 7).

⬇

def step(self):

for agent_id, platform in self._state.sim_platforms.items():

action = np.array(platform.get_applied_action(), dtype=np.float32)

entity = self.sim_entities[agent_id]

entity.step(action=action, step_size=self.config.step_size)

platform.sim_time = self.clock

self.update_sensor_measurements()

self.clock += self.config.step_size

self._state.sim_time = self.clock

return self._state

Figure 10: step method of 1D Docking Simulator

Sensor measurements are calculated from Sensor classes which access the sensor
and velocity properties of their parent Platform. These sensors are added to
the plug-in library and associated with Docking1dSimulator and
Docking1dPlatform. Finally, a Controller is also created which wraps the
Docking1dPlatform methods to save and get actions to the abstract Controller
class methods. This Part is similarly added to the plug-in library.

Now a working interface between the simulation and the CoRL Environment and
Agent classes has been made. It can be configured with YAML files similar to
those in Section 4.1. A baseline configuration is included in the CoRL
release. More detail on the Simulator, Platform, Part interfaces can be found
in the source code or documentation (Footnote 10).

##  5 Conclusions

CoRL is a modular, composable, and hyper-configurable environment creation
tool. While some existing tools rely on monolithic implementations, CoRL
allows for minute control of an agent’s observations and actions with
modification of YAML configuration files. Other tools are tied to specific
simulation environments, CoRL allows for new simulations to be integrated with
minimal difficulty. Still others are tied only to a reinforcement learning
paradigm, CoRL can be used with RL agents trained using any framework, but can
also utilize the distributed computing of Ray to evaluate non-RL agents with
ease.

At release, CoRL provides useful examples and common implementations of many
of its classes, but can be easily extended to more advanced features. Section
3.4 discussed using the EPP to do curriculum learning, but that framework
could be expanded further to include automatic domain randomization. The EPP
could also be upgraded to allow the reference store to provide information to
any object in the environment, currently it only allows for functors to
reference it. The existing unit conversion system is very rigid. Using an
existing unit conversion library could allow users to define their own units.
RLLib [2] is incompatible with the DeepMind Control Suite [11], but
implementing a Simulator, Platform, and basic Glues could allow its popular
baseline tasks to be manipulated by CoRL. CoRL is being actively maintained
and used for a variety of research tasks at the Air Force Research Laboratory.

## References

  * Covin [2021] Covin, S., “pydantic,” , 2021.  URL https://github.com/pydantic/pydantic.
  * Liang et al. [2018] Liang, E., Liaw, R., Nishihara, R., Moritz, P., Fox, R., Goldberg, K., Gonzalez, J., Jordan, M., and Stoica, I., “RLlib: Abstractions for Distributed Reinforcement Learning,” _Proceedings of the 35th International Conference on Machine Learning_ , Proceedings of Machine Learning Research, Vol. 80, edited by J. Dy and A. Krause, PMLR, 2018, pp. 3053–3062.  URL https://proceedings.mlr.press/v80/liang18b.html.
  * Brockman et al. [2016] Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W., “OpenAI Gym,” , 2016.
  * gym [2022] “OpenAI/Gym: Dependency Graph,” , 2022.  URL https://github.com/openai/gym/network/dependents.
  * Terry et al. [2020a] Terry, J. K., Black, B., Grammel, N., Jayakumar, M., Hari, A., Sulivan, R., Santos, L., Perez, R., Horsch, C., Dieffendahl, C., Williams, N. L., Lokesh, Y., Sullivan, R., and Ravi, P., “PettingZoo: Gym for Multi-Agent Reinforcement Learning,” _arXiv preprint arXiv:2009.14471_ , 2020a.
  * Raffin et al. [2021] Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., and Dormann, N., “Stable-Baselines3: Reliable Reinforcement Learning Implementations,” _Journal of Machine Learning Research_ , Vol. 22, No. 268, 2021, pp. 1–8.  URL http://jmlr.org/papers/v22/20-1364.html.
  * Weng et al. [2022] Weng, J., Chen, H., Yan, D., You, K., Duburcq, A., Zhang, M., Su, Y., Su, H., and Zhu, J., “Tianshou: A Highly Modularized Deep Reinforcement Learning Library,” _Journal of Machine Learning Research_ , Vol. 23, No. 267, 2022, pp. 1–6.  URL http://jmlr.org/papers/v23/21-1127.html.
  * La et al. [2022] La, W. G., Muralidhara, S., Kong, L., and Nichat, P., “Unified Distributed Environment,” , 2022.
  * The Farama Foundation [2022] The Farama Foundation, “Gymnasium,” , 2022.  URL https://github.com/Farama-Foundation/Gymnasium.
  * Juliani et al. [2018] Juliani, A., Berges, V.-P., Teng, E., Cohen, A., Harper, J., Elion, C., Goy, C., Gao, Y., Henry, H., Mattar, M., and Lange, D., “Unity: A General Platform for Intelligent Agents,” , 2018.  10.48550/ARXIV.1809.02627, URL https://arxiv.org/abs/1809.02627.
  * Tunyasuvunakool et al. [2020] Tunyasuvunakool, S., Muldal, A., Doron, Y., Liu, S., Bohez, S., Merel, J., Erez, T., Lillicrap, T., Heess, N., and Tassa, Y., “dm_control: Software and tasks for continuous control,” _Software Impacts_ , Vol. 6, 2020, p. 100022.
  * Todorov et al. [2012] Todorov, E., Erez, T., and Tassa, Y., “MuJoCo: A physics engine for model-based control,” _2012 IEEE/RSJ International Conference on Intelligent Robots and Systems_ , IEEE, 2012, pp. 5026–5033.  10.1109/IROS.2012.6386109.
  * Terry et al. [2020b] Terry, J. K., Black, B., and Hari, A., “SuperSuit: Simple Microwrappers for Reinforcement Learning Environments,” _arXiv preprint arXiv:2008.08932_ , 2020b.
  * Tobin et al. [2017] Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., and Abbeel, P., “Domain randomization for transferring deep neural networks from simulation to the real world,” _2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2017, pp. 23–30.  10.1109/IROS.2017.8202133.
  * Peng et al. [2018] Peng, X. B., Andrychowicz, M., Zaremba, W., and Abbeel, P., “Sim-to-Real Transfer of Robotic Control with Dynamics Randomization,” _2018 IEEE International Conference on Robotics and Automation (ICRA)_ , 2018, pp. 3803–3810.  10.1109/ICRA.2018.8460528.
  * OpenAI et al. [2019] OpenAI, Akkaya, I., Andrychowicz, M., Chociej, M., Litwin, M., McGrew, B., Petron, A., Paino, A., Plappert, M., Powell, G., Ribas, R., Schneider, J., Tezak, N., Tworek, J., Welinder, P., Weng, L., Yuan, Q., Zaremba, W., and Zhang, L., “Solving Rubik’s Cube with a Robot Hand,” , 2019.  10.48550/ARXIV.1910.07113, URL https://arxiv.org/abs/1910.07113.
  * Barto et al. [1983] Barto, A. G., Sutton, R. S., and Anderson, C. W., “Neuronlike adaptive elements that can solve difficult learning control problems,” _IEEE Transactions on Systems, Man, and Cybernetics_ , Vol. SMC-13, 1983, pp. 834–846.
  * Team [2022] Team, R., “Ray v2 Architecture,” Tech. rep., Ray Project, October 2022.  Available at https://docs.google.com/document/d/1tBw9A4j62ruI5omIJbMxly-la5w4q_TjyJgJL_jN2fI/preview#.
  * Moritz et al. [2018] Moritz, P., Nishihara, R., Wang, S., Tumanov, A., Liaw, R., Liang, E., Elibol, M., Yang, Z., Paul, W., Jordan, M. I., et al., “Ray: A distributed framework for emerging $\\{$AI$\\}$ applications,” _13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18)_ , 2018, pp. 561–577.
  * MacGlashan et al. [2022] MacGlashan, J., Archer, E., Devlic, A., Seno, T., Sherstan, C., Wurman, P. R., and Stone, P., “Value Function Decomposition for Iterative Design of Reinforcement Learning Agents,” _arXiv preprint arXiv:2206.13901_ , 2022.
  * Russell et al. [2010] Russell, S., Russell, S., Norvig, P., and Davis, E., _Artificial Intelligence: A Modern Approach_ , Prentice Hall series in artificial intelligence, Prentice Hall, 2010.  URL https://books.google.com/books?id=8jZBksh-bUMC.
  * Abadi et al. [2015] Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., Ghemawat, S., Goodfellow, I., Harp, A., Irving, G., Isard, M., Jia, Y., Jozefowicz, R., Kaiser, L., Kudlur, M., Levenberg, J., Mané, D., Monga, R., Moore, S., Murray, D., Olah, C., Schuster, M., Shlens, J., Steiner, B., Sutskever, I., Talwar, K., Tucker, P., Vanhoucke, V., Vasudevan, V., Viégas, F., Vinyals, O., Warden, P., Wattenberg, M., Wicke, M., Yu, Y., and Zheng, X., “TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems,” , 2015.  URL https://www.tensorflow.org/, software available from tensorflow.org.
  * Cunningham et al. [2022] Cunningham, J., Ravaioli, U. J., Dunlap, K., and Hobbs, K. L., “safe-autonomy-dynamics,” , 2022.  URL https://github.com/act3-ace/safe-autonomy-dynamics.
  * Ravaioli et al. [2022] Ravaioli, U. J., Cunningham, J., McCarroll, J., Gangal, V., Dunlap, K., and Hobbs, K. L., “Safe Reinforcement Learning Benchmark Environments for Aerospace Control Systems,” _2022 IEEE Aerospace Conference (AERO)_ , 2022, pp. 1–20.  10.1109/AERO53065.2022.9843750.