# Exploring Semantic Relationships for
Unpaired Image Captioning

Fenglin Liu1, Meng Gao2, Tianhao Zhang3, Yuexian Zou1
1ADSPLAB, School of ECE, Peking University  2School of ICE, Beijing University
of Posts and Telecommunications  3Beijing Foreign Studies University
{fenglinliu98<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Recently, image captioning has aroused great interest in both academic and
industrial worlds. Most existing systems are built upon large-scale datasets
consisting of image-sentence pairs, which, however, are time-consuming to
construct. In addition, even for the most advanced image captioning systems,
it is still difficult to realize deep image understanding. In this work, we
achieve unpaired image captioning by bridging the vision and the language
domains with high-level semantic information. The motivation stems from the
fact that the semantic concepts with the same modality can be extracted from
both images and descriptions. To further improve the quality of captions
generated by the model, we propose the Semantic Relationship Explorer, which
explores the relationships between semantic concepts for better understanding
of the image. Extensive experiments on MSCOCO dataset show that we can
generate desirable captions without paired datasets. Furthermore, the proposed
approach boosts five strong baselines under the paired setting, where the most
significant improvement in CIDEr score reaches 8%, demonstrating that it is
effective and generalizes well to a wide range of models111This paper was
originally published at ICDM 2019 [1]..

###### Index Terms:

vision and language, unpaired training data, image captioning, deep neural
network

##  I Introduction

Image captioning has drawn remarkable attention in both natural language
processing and computer vision. The task, which combines image understanding
and language generation, is tough yet practical. Above all, it has various
kinds of applications such as human-robot interaction [2], text-based image
retrieval [3] and helping visually impaired people see [4], among others. The
deep neural networks, particularly those based on the encoder-decoder
framework [5, 6, 7, 8, 9, 10, 11, 12, 13], have achieved great success in
advancing the state-of-the-art.

Despite the impressive achievements of the deep learning frameworks, they
unduly rely on large-scale paired data, which is not easy to access in the
real world. Especially, when it comes to the Non-English caption systems,
image-caption pairs are labor-intensive to obtain. In recent years,
unsupervised encoder-decoder models have been proposed for neural machine
translation [14, 15, 16]. Typically, the source language and the target
language are mapped into a common latent space, where sentences with the same
semantic meaning are well aligned, thus the unsupervised translation can be
carried out. For the task of image captioning, due to the great disparities
between the vision and the language domains, and distinct features of the
modalities, unsupervised learning is even more challenging. Although unpaired
image-to-sentence translation has been explored in the literature, the
training of neural models on unpaired datasets has not yet been conducted at a
semantic level.

Figure 1: Illustration of our proposed framework. Our framework consists of a
Semantic Concept Extractor, a Semantic Relationship Explorer, a Semantic
Relationship Embedding and an Attention-based Sentence Decoder.

Recently, there have been a few works [17, 18, 19, 20, 21, 22, 23] trying to
reduce the dependence of image captioning models on paired data. Specifically,
[17] proposed a method to generate captions in a central language (Chinese)
and then translate them into a target language (English), without requirement
for image-caption pairs for the target language. However, a paired image
captioning corpus for the source language and a paired corpus for translation
are still indispensable. [18] connected the visual and the textual modalities
with visual objects (e.g., dog, mirror) and an adversarial manner. It achieved
unsupervised image captioning without any image-caption pairs. Nevertheless,
their method relied on the recognized visual objects to decide whether the
generated captions are image-related, which necessitated complex models and
schemes to obtain higher-quality image captions, e.g. image reconstruction
[18], sentence reconstruction [24] and adversarial learning [25]. Moreover,
the visual object based approach does not consider attributive words (e.g.
attributes (small), relations (standing) and color (white)), which could help
improve the image representation. [19] used the scene graph to bridge the gap
between the visual and the textual modalities. Since the scene graph
constructed a series of semantic relationship information, the model achieved
good results. However, in order to construct the semantic relationships, they
needed to use Faster-RCNN [26] as the object detector, MOTIFS [27] as the
relationship detector, and an additional classifier for attribute
identification [13] for generating the image scene graph. In all, although
existing methods for unsupervised image captioning have made appreciable
progress, they are hard to implement and still far from real world
applications.

In this paper, we bridge the gap between images and captions with semantic
concepts, which explicitly represent high-level information [28]. To solve the
challenges posed by different characteristics between visual and textual
modalities, we propose an approach to explore the relationships between
individual semantic concepts. The proposed framework (Figure 1) includes a
Semantic Concept Extractor, a Semantic Relationship Explorer, a Semantic
Relationship Embedding and an Attention-based Sentence Decoder. To begin with,
our model extracts relevant semantic concepts from the images and the
captions, respectively, which is the Semantic Concept Extractor. Then, the
Semantic Relationship Explorer, which consists of an “Attribute” Aggregator,
an “Object” Aggregator and a “Relation” Aggregator, explores effective
semantic relationships among the extracted concepts from three aspects, i.e.,
“attribute”, “object” and “relation”. Meanwhile, the Semantic Relationship
Embedding can extract principle features of the semantic relationships from
different parts, which strengthens the expressive ability of relationship
features. As a result, characteristic information of the semantic concepts is
fully exploited and the representation of the image concept features is
getting closer to the textual modality and, as input to the decoder, adopted
effectively to generate meaningful sentences. Our main contributions are as
follows:

  * •

We propose a semantic based framework for unpaired image captioning. The
framework can effectively and automatically use the semantic information to
generate high quality image captions by bridging the gap between the vision
and the language domains.

  * •

To explore the relationships between individual semantic concepts, we propose
a Semantic Relationship Explorer, which can extract both coarse-grained and
fine-grained semantic relationships to help the model generate more accurate
and meaningful captions.

  * •

Our approach proves to be effective by the experiments on the MSCOCO image
captioning dataset. We outperforms the latest unpaired image captioning
systems. More encouragingly, our method also makes improvements of up to 8% in
terms of CIDEr score for paired image captioning model, when equipping it with
our approach.

##  II Related Work

###  II-A Image Captioning

Recently, many neural model systems have been proposed for image captioning
[5, 6, 7, 8, 9, 10, 11, 12, 13]. The most advanced methods [9, 10, 11, 12, 13]
rely on the encoder-decoder framework and combine the attention mechanism [5]
to transform the image into coherent caption. However, the paired image-
sentence data for training these models is expensive to collect. The first
model under the unsupervised training setting, built by [17], aims to take the
pivot language to connect source image and target caption. Although image and
target language caption pairs are not used, their method relies on image-pivot
pairs and a pivot-target parallel translation corpus. [18] and [19] align
between the vision and the language domain in an adversarial way, so they
don’t need any paired image-sentence data. Also, using a scarce amount of
paired data as a bridge in the form of semi-supervised learning has been
proposed by [23]. In contrast to the approaches mentioned above, our approach
attempts to regard semantic concepts as a bridge and explore the semantic
relationships among them, which has not been well studied yet.

###  II-B Unsupervised Machine Translation

Unsupervised image captioning is essentially similar to unsupervised machine
translation. Nevertheless, due to the apparently different characteristics of
the image and text modality, unpaired image-to-sentence translation is more
challenging than unpaired sentence-to-sentence translation.

###  II-C Exploring Semantic Relationships

A most recent advancement [11, 13] attempts to use graph networks to
explicitly explore visual relationships by encoding scene graphs, which model
the spatial and semantic relationships of image regions based on visual
features. However, they built an additional model to explicitly predict the
relationships between visual objects with extra annotated data. Instead, our
model associates the semantic concepts based on attention, and the
relationships are implicitly modelled as weighted combinations and trained
with the captioning model.

Figure 2: Illustration of the proposed framework. The semantic concepts are
extracted by an extractor [29]. The color Green denotes the attribute words,
the color Red denotes the object words and Blue color denotes the relation
words. The Semantic Relationship Explorer consists of a Coarse-Grained
Semantic Relationship Explorer and a Fine-Grained Semantic Relationship
Explorer. The latter one includes three sub-modules: a “Attribute” Aggregator,
an “Object” Aggregator and a “Relation” Aggregator, then adaptively explores
the relationships among the extracted semantic concepts. The proposed approach
could achieve deep semantic understanding of the image and in turn generate
more complete and coherent captions.

##  III The Proposed Framework

As shown in Figure 2, the proposed framework consists of four main modules:
(1) Semantic Concept Extractor: this module is able to extract the semantic
concepts from images or sentences; (2) Semantic Relationship Explorer (Figure
3): since the extracted semantic concepts are independent and not associated
with each other, e.g., the three words riding, boy and bike, this module can
associate these words together as a phrase boy riding bike to represent a
complete semantics; (3) Semantic Relationship Embedding: this module can
strengthen the expressive ability of relationship features by extracting
different features of the explored semantic relationships from different
parts; (4) Through the above three steps, we are able to provide rich semantic
relationship information which is beneficial for the fourth module, i.e.,
Attention-based Sentence Decoder, to generate the complete and coherent image
captions under the unpaired setting. In the following parts of this section,
we will describe these steps in detail.

###  III-A Semantic Concept Extractor

In recent studies [29, 28, 7, 30, 31], the semantic concepts are introduced to
represent the explicit high-level information of the image [28]. In
particular, it consists of a series of words, including attributes (e.g.,
young, black), objects (e.g., woman, shirt) and relationships (e.g., sitting,
wearing). In this case, these semantic concept words for an image can be
generated by a Semantic Concept Extractor [28, 7]. In implementation, we adopt
a weakly-supervised method of Multiple Instance Learning (MIL) [32] to build
the Semantic Concept Extractor, following [29]. Technically, for each concept,
MIL takes the “positive” and “negative” bags of bounding boxes as input sets,
where every bag corresponds to an image. If the current concept is in the
semantic label (e.g. description and scene graph) of the image, the bag is
regarded to be positive, otherwise it is said to be negative. MIL iteratively
selects the positive instances to train an extractor for the semantic
concepts. Due to limited space, please refer to [29] for detailed explanation.
Other semantic concept extracting approaches can also be used and may produce
better results, which, however, are not the main focuses of this work.

To extract the semantic concepts of a sentence, we directly use the words,
which appear in both the ground truth and the pre-defined semantic concepts
vocab, as semantic concepts.

In addition, because semantic concepts are discrete word tokens, we adapt a
word embedding to project them into vectors
$\boldsymbol{C}_{g}=\\{\boldsymbol{c}_{1},\boldsymbol{c}_{2},\ldots,\boldsymbol{c}_{N_{g}}\\},\boldsymbol{c}_{i}\in\mathbb{R}^{e}$,
where $e$ is the dimension of word embedding. Then all words, words containing
only “attribute”, words containing only “object” and words containing only
“relation” are respectively represented as global semantic concepts
$\boldsymbol{C}_{g}\in\mathbb{R}^{N_{g}\times e}$, attribute semantic concepts
$\boldsymbol{C}_{a}\in\mathbb{R}^{N_{a}\times e}$, object semantic concepts
$\boldsymbol{C}_{o}\in\mathbb{R}^{N_{o}\times e}$ and relation semantic
concepts $\boldsymbol{C}_{r}\in\mathbb{R}^{N_{r}\times e}$, where $N_{g}$,
$N_{a}$, $N_{o}$ and $N_{r}$ represent the number of words per type.

###  III-B Semantic Relationship Explorer

In this section, we will first introduce the method we use to extract the
semantic relationships - MultiHead Attention, then we will introduce the
proposed Semantic Relationship Explorer, Figure 3 shows a sketch.

Figure 3: Illustration of the proposed Semantic Relationship Explorer. The
color Green denotes the Attribute words, the color Red denotes the Object
words and Blue color denotes the Relation words.

####  III-B1 MultiHead Attention

In order to extract the semantic relationships, we adopt multi-head attention
[33], which can calculate the association weights between each pair of the
given semantic concepts [34]. The following multi-head attention consists of
$\boldsymbol{N}$ parallel heads and each head is represented as a scaled dot-
product attention.

$\text{Att}(\text{Q,K,V})=\text{softmax}\left(\frac{\text{Q}\text{W}^{\text{Q}}(\text{K}\text{W}^{\text{K}})^{T}}{\sqrt{{d}_{k}}}\right)\text{V}\text{W}^{\text{V}}$
(1)

where $\text{Q}\in\mathbb{R}^{l\times d_{h}}$,
$\text{K}\in\mathbb{R}^{{k}\times d_{h}}$ and
$\text{V}\in\mathbb{R}^{{k}\times d_{h}}$ represent respectively the query
matrix, the key matrix and the value matrix.
$\text{W}^{\text{Q}},\text{W}^{\text{K}},\text{W}^{\text{V}}\in\mathbb{R}^{d_{h}\times
d_{k}}$ are learnable parameters and ${d}_{k}=d_{h}/{n}$, where $n$ represents
the number of heads.

Following the multi-head attention is a fully-connected feed-forward network,
which is defined as follows:

$\text{FFN}(x)=\text{ReLU}(x\text{W}_{\text{f}}+{b}_{\text{f}})\text{W}_{\text{ff}}+{b}_{\text{ff}}$
(2)

where $\text{W}_{\text{f}}$ and $\text{W}_{\text{ff}}$ denote matrices for
linear transformation; $b_{\text{f}}$ and ${b}_{\text{ff}}$ represent the bias
terms.

We use the attention-based method to make each semantic concept adaptively
find the most relevant attributes, objects and relations, and establish
related relationships, thus completing the exploration process of the semantic
relationships.

####  III-B2 Coarse-Grained Semantic Relationship Explorer

The Coarse-Grained SRE is a simple and intuitive way to explore the semantic
relationships. The relationships are built via multi-head attention on the
global semantic concepts $\boldsymbol{C}_{g}$. Particularly, we use
$\boldsymbol{C}_{g}$ as Query, Key and Value simultaneously. The result turns
out to be a set of attended semantic concepts:

$\boldsymbol{R}_{g}=\text{FFN}(\text{MultiHead}(\boldsymbol{C}_{g},\boldsymbol{C}_{g},\boldsymbol{C}_{g}))$
(3)

with Coarse-Grained SRE, all semantic concepts can attend to other semantic
concepts related to them, making it easy to capture global relationships
quickly. However, the semantic relationships extracted by Coarse-Grained SRE
are coarse-grained, because for any kind of semantic concept as a Query, it is
not necessary to treat all words as Key and Value, i.e., there is no need to
attend to the most relevant among all words. For example, when an “attribute”
word is used as a Query, it does not need to pay attention to the “relation”
words. This is because the “attribute” words are usually used to describe a
specific object, while the “relation” words are usually used to describe the
relationship among two or more objects, that is, the association between the
“attribute” words and the “relation” words is not very strong. Therefore, the
semantic relationships explored by Coarse-Grained SRE are not good due to the
noise introduced by many unrelated words.

####  III-B3 Fine-Grained Semantic Relationship Explorer

In order to solve the problems of the Coarse-Grained SRE and to extract more
fine-grained and more precise semantic relationships, we propose the Fine-
Grained Semantic Relationship Explorer (Fine-Grained SRE). Inspired by the
proposed attribute embedding, object embedding and relationship embedding in
[13], we propose “Attribute” Aggregator, “Object” Aggregator and “Relation”
Aggregator, which use attribute words, object words and relation words,
respectively, as Value and Key. In addition, we get finer and more accurate
semantic relationships by using the words of the category most relevant to
them as Query.

##### “Attribute” Aggregator

A specific object may have multiple attributes. Therefore, we have to focus on
the attributes that are most relevant to it from a set of attribute semantic
concepts. In this case, according to the attention theorem, the object
semantic concepts $\boldsymbol{C}_{o}$ are the Queries, and the attribute
semantic concepts $\boldsymbol{C}_{a}$ are the Keys and Values. Consequently,
the result turns out to be a set of attended attribute semantic concepts:

$\boldsymbol{R}_{a}=\text{FFN}(\text{MultiHead}(\boldsymbol{C}_{o},\boldsymbol{C}_{a},\boldsymbol{C}_{a}))$
(4)

Through this formula, we can get detailed object-attribute relationship
information.

##### “Object” Aggregator

For a particular relation concept, we tend to focus on seeking for two objects
that are related to each other to form the object-relationship-object, e.g.,
women-wearing-shirt. Therefore, for the “Object” Aggregator, the relation
semantic concepts should focus on the object semantic concepts. The relation
semantic concepts $\boldsymbol{C}_{r}$ serve as Queries, and the object
semantic concepts $\boldsymbol{C}_{o}$ serve as Keys and Values, which can be
defined as follows:

$\boldsymbol{R}_{o}=\text{FFN}(\text{MultiHead}(\boldsymbol{C}_{r},\boldsymbol{C}_{o},\boldsymbol{C}_{o}))$
(5)

Thus, we can get detailed object-relation-object relationship information by
the “Object” Aggregator.

##### “Relation” Aggregator

A specific object may have many relations with other objects, so we need to
focus on the relation that is most relevant to it from a set of relation
semantic concepts, e.g., woman-eating, woman-holding, woman-wearing and so on.
Therefore, for “Relation” Aggregator, we consider what relationships they are
attached to. Accordingly, the object semantic concepts $\boldsymbol{C}_{o}$
should pay attention to the relation semantic concepts $\boldsymbol{C}_{r}$,
which is defined as follows:

$\boldsymbol{R}_{r}=\text{FFN}(\text{MultiHead}(\boldsymbol{C}_{o},\boldsymbol{C}_{r},\boldsymbol{C}_{r}))$
(6)

where we can get object-relation relationship information.

###  III-C Semantic Relationship Embedding

For the semantic concepts of each image or sentence, through Coarse-Grained
SRE and Fine-Grained SRE, we can get coarse-grained relationships
$\boldsymbol{R}_{g}\in\mathbb{R}^{N_{g}\times e}$ and fine-grained
relationships $\boldsymbol{R}_{a}\in\mathbb{R}^{N_{a}\times e}$,
$\boldsymbol{R}_{o}\in\mathbb{R}^{N_{o}\times e}$ and
$\boldsymbol{R}_{r}\in\mathbb{R}^{N_{r}\times e}$, respectively. These
semantic relationships are of different quantity and contain different
information. So in order to have the same number of features for each type of
semantic relationships and to encourage the diversity of extracted main
features for decoding effectively, we use a self-attention mechanism provided
by [35]. This mechanism can be defined as follows:

$\displaystyle\boldsymbol{A}$
$\displaystyle=\text{softmax}(\boldsymbol{W}_{s_{2}}\text{tanh}(\boldsymbol{W}_{s_{1}}{\boldsymbol{R}_{k}}^{T}))$
(7)
$\displaystyle\boldsymbol{M}_{k}=\boldsymbol{A}\boldsymbol{R}_{k},k\in\\{g,a,o,r\\}$
(8)

where $\boldsymbol{W}_{s_{1}}\in\mathbb{R}^{d_{a}\times e}$ and
$\boldsymbol{W}_{s_{2}}\in\mathbb{R}^{r\times d_{a}}$ are learnable
parameters; r stands for the number of extracted features in different parts.
Through the formula, we can get different semantic relationship embeddings,
i.e., $\boldsymbol{M}_{k}\in\mathbb{R}^{r\times e}(k\in\\{g,a,o,r\\})$, which
share the same dimensions. For coarse-grained relationships embedding,
$\boldsymbol{M}_{c}=\boldsymbol{M}_{g}$. We combine $\boldsymbol{M}_{a}$,
$\boldsymbol{M}_{o}$ and $\boldsymbol{M}_{r}$ to obtain the fine-grained
relationships embedding,
$\boldsymbol{M}_{f}=[\boldsymbol{M}_{a};\boldsymbol{M}_{o};\boldsymbol{M}_{r}]$,
where [;] stands for concatenation operation.

###  III-D Attention-Based Decoder

The decoder part is essential in the way that it translates the extracted
semantic relationships into their corresponding captions. The decoder is an
attention-based LSTM [36], which takes the word embedding vector
${\boldsymbol{w}^{e}_{t}}$, concatenated with the averaged semantic concepts
vector
$\boldsymbol{c}_{avg}=\frac{1}{N_{g}}\sum_{i=1}^{N_{g}}\boldsymbol{c}_{i}$, as
input, which is denoted as
$\boldsymbol{x}_{t}=[\boldsymbol{w}^{e}_{t};\boldsymbol{c}_{avg}]$.

At each time step, when the decoder generates captions, the semantic
relationship information that needs attention varies with different time
steps. Inspired by the attention mechanism in [36], we design an attention
LSTM sentence decoder by injecting all of the coarse-grained or fine-grained
semantic relationship embedding features ($\boldsymbol{M}_{k},k\in\\{c,f\\}$)
into a LSTM with attention mechanism. Specifically, at each time step t,
according to the current hidden layer $\boldsymbol{h}_{t}\in\mathbb{R}^{{d}}$,
the attention LSTM decoder firstly generates the attention distribution over
the n (n is r for $\boldsymbol{M}_{c}$ and 3r for $\boldsymbol{M}_{f}$) parts
of semantic relationship embedding features
$\boldsymbol{M}_{k},k\in\\{c,f\\}$:

$\displaystyle\boldsymbol{\alpha}_{t}=\text{softmax}(\boldsymbol{w}_{\alpha}\text{tanh}(\boldsymbol{W}_{M}{\boldsymbol{M}_{k}}^{T}\oplus\boldsymbol{W}_{h}\boldsymbol{h}_{t})$
(9)

where $\boldsymbol{W}_{M}\in\mathbb{R}^{d_{a}\times e}$,
$\boldsymbol{W}_{h}\in\mathbb{R}^{d_{a}\times d}$ and
$\boldsymbol{w}_{\alpha}\in\mathbb{R}^{d_{a}}$ are learnable parameters;
$\oplus$ denotes the matrix-vector addition, which is calculated by adding the
vector to each column of the matrix.
$\boldsymbol{\alpha}_{t}\in\mathbb{R}^{n}$ is the attentive weight of
$\boldsymbol{M}_{k}$ and the attentive relationship vector
$\boldsymbol{r}_{t}\in\mathbb{R}^{e}$ can be defined as follows:

$\displaystyle\boldsymbol{r}_{t}=\boldsymbol{\alpha}_{t}\boldsymbol{M}_{k}$
(10)

With a further combination with $\boldsymbol{h}_{t}$, the attentive
relationship can predict the current output word:

$\displaystyle
y_{t}\sim\boldsymbol{p}_{t}=\text{softmax}(\boldsymbol{W}_{p,r}\boldsymbol{r}_{t}+\boldsymbol{W}_{p,h}\boldsymbol{h}_{t})$
(11)

where $\boldsymbol{W}_{p,r}\in\mathbb{R}^{|D|\times
e},\boldsymbol{W}_{p,h}\in\mathbb{R}^{|D|\times d}$, and each value of
$\boldsymbol{p}_{t}\in\mathbb{R}^{|D|}$ denotes the probability suggesting how
likely the corresponding word in vocabulary $D$ is the current output word.

The approach mentioned above encourages the model to exploit all the available
semantic relationship information. Thanks to the abundant and enriched
information extracted by the Semantic Relationship Explorer, the source
semantic concept information turns into a deep image understanding.

Model | BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4 | METEOR | ROUGE-L | CIDEr | SPICE
---|---|---|---|---|---|---|---|---
Language Pivoting [17] | 46.2 | 24.0 | 11.2 | 5.4 | 13.2 | - | 17.7 | -
Adversarial+Reconstruction [18] | 58.9 | 40.3 | 27.0 | 18.6 | 17.9 | 43.1 | 54.9 | 11.1
Graph-Align† [19] | 66.7 | 48.0 | 31.5 | 21.1 | 20.5 | 46.2 | 68.7 | 14.5
Graph-Align [19] | 67.1 | 47.8 | 32.3 | 21.5 | 20.9 | 47.2 | 69.5 | 15.0
Coarse-Grained SRE | 50.3 | 36.6 | 24.7 | 16.5 | 14.3 | 33.4 | 37.2 | 10.6
Fine-Grained SRE | 57.5 | 38.9 | 27.1 | 19.7 | 17.4 | 41.9 | 49.7 | 13.3
Graph-Align [19] \+ Fine-Grained SRE | 67.8 | 48.7 | 33.6 | 21.8 | 22.1 | 48.4 | 75.7 | 16.1
Table I: Performance on the MSCOCO Karpathy test split [37] under the unpaired
setting. All values are reported in percentage (%). Higher is better in all
columns. † denotes the results of the re-implemented Graph-Align for fair
comparison. The proposed Fine-Grained SRE has a better performance than the
Coarse-Grained SRE. Additionally, it is clear that the Fine-Grained SRE
approach also boosts the performance of Graph-Align, indicating that our
approach learns very effective semantic relationship information even for
scene-graphs.

##  IV Implementation Details

In this section, we will introduce how to train our model and how to use our
model to generate image captions under the unpaired settings.

###  IV-A Training Strategy

##### Training in Language Domain

Since there is no parallel data, we train the full model in the language
domain, and use the method of reconstructing the original sentence to train
the model, that is:

Sentence $\to$ Semantic Concepts $\to$ Semantic Relationships $\to$ Semantic
Relationship Embedding $\to$ Decoder $\to$ Sentence

The Semantic Relationship Explorer explores the Semantic Relationships
$\boldsymbol{R}^{S}$ from the Semantic Concepts $\boldsymbol{C}^{S}$, which
are extracted from the sentence $\boldsymbol{S}$, and obtain Semantic
Relationship Embeddings $\boldsymbol{E}^{S}$ through a self-attention
mechanism. Finally, the original sentence $\boldsymbol{S}$ is reconstructed by
an Attention-based Sentence Decoder.

##### Training Objectives

Firstly, we train the full model by minimizing the Cross-Entropy (XE) Loss
[38, 39, 40, 36, 7, 41, 42, 43]. Given a target ground truth sequence
${y}^{\text{*}}_{1:T}$ and a captioning model with parameters $\theta$, the
goal is to minimize the following cross entropy loss:

$\displaystyle{L}_{CE}(\theta)=-\displaystyle{\sum_{t=1}^{T}}\text{log}({p}_{\theta}({y}^{\text{*}}_{t}|{y}^{\text{*}}_{1:t-1}))$
(12)

Recently, reinforcement learning methods have also been widely used for
captioning model training [9, 13, 44, 11]. Consequently, we further employ a
reinforcement learning (RL) loss to improve the performance of our model. In
this case, the cross-entropy loss method is used to pre-train the model, after
which our goal is to minimize the negative expected score as:

$\displaystyle{L}_{RL}(\theta)=-{E}_{{y}_{1:T}\sim{p}_{\theta}}[r({y}_{1:T})]$
(13)

where $r$ is the score function (e.g., CIDEr). Following the Self-Critical
Sequence Training [44] (SCST), the gradient of ${L}_{RL}(\theta)$ can be
approximated by

$\displaystyle{\nabla}_{\theta}{L}_{RL}(\theta)\approx-(r({y}^{s}_{1:T})-r(\hat{y}_{1:T})){\nabla}_{\theta}log{p}_{\theta}({y}^{s}_{1:T})$
(14)

where $r({y}^{s}_{1:T})$ is the score of a sampled caption ${y}^{s}_{1:T}$ and
$r(\hat{y}_{1:T})$ suggests the baseline score of a caption which is generated
by the current model using greedy decode. Through this gradient, sampled
captions with higher CIDEr scores are more likely to be generated by the model
because their corresponding probabilities are increased. Through this
approach, we complete CIDEr optimization.

###  IV-B Inference Strategy

We use the full model to train the model in the language domain to generate
image captions. The inference process is defined as follows:

Image $\to$ Semantic Concepts $\to$ Semantic Relationships $\to$ Semantic
Relationship Embedding $\to$ Decoder $\to$ Sentence

Given an image $\boldsymbol{I}$, we can extract the Semantic Concepts
$\boldsymbol{C}^{I}$ from $\boldsymbol{I}$ through the Semantic Concept
Extractor. Then, similar to sentence reconstruction process, we use the
Semantic Relationship Explorer to extract the Semantic Relationship
$\boldsymbol{R}^{I}$, which embedded to Semantic Relationship Embeddings
$\boldsymbol{E}^{I}$. At last, $\boldsymbol{E}^{I}$ will use the decoder to
generate captions.

Model | BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4 | METEOR | ROUGE-L | CIDEr | SPICE
---|---|---|---|---|---|---|---|---
ATT-FCN [6]
Baseline | 70.9 | 53.7 | 40.2 | 30.4 | 24.3 | 53.3 | 95.6 | 18.6
\+ Proposal | 72.3 | 55.6 | 41.6 | 32.0 | 25.9 | 54.8 | 100.2 | 19.2
LSTM-A2 [7]
Baseline | 73.3 | 56.5 | 42.7 | 32.2 | 25.3 | 53.9 | 99.1 | 18.3
\+ Proposal | 77.2 | 58.4 | 44.6 | 33.1 | 25.9 | 54.2 | 103.5 | 18.7
LSTM-A3 [7]
Baseline | 73.5 | 56.6 | 42.9 | 32.4 | 25.5 | 53.9 | 99.8 | 18.5
\+ Proposal | 76.8 | 57.9 | 44.2 | 33.2 | 26.7 | 54.5 | 104.1 | 19.1
LSTM-A4 [7]
Baseline | 72.1 | 55.5 | 41.7 | 31.4 | 24.9 | 53.2 | 95.7 | 17.8
\+ Proposal | 72.8 | 56.4 | 43.4 | 32.1 | 26.6 | 54.6 | 101.4 | 18.4
LSTM-A5 [7]
Baseline | 73.4 | 56.7 | 43.0 | 32.6 | 25.4 | 54.0 | 100.2 | 18.6
\+ Proposal | 75.2 | 57.9 | 44.2 | 34.0 | 26.7 | 54.8 | 108.0 | 19.7
Table II: Evaluation of representative systems under the paired setting. The
proposed approach can further improve the already strong baselines in all
metrics, which demonstrates the generalization ability of our approach to a
wide range of existing systems. The significant improvements come from the
explored semantic relationships rather than the simple incorporation of the
semantic concepts, which indicates its effectiveness in exploring semantic
relationships.

##  V Experiments

In this section, we will firstly describe a benchmark dataset for image
captioning as well as some widely-used metrics and experimental settings. Then
we will present the proposed model.

###  V-A Datasets, Metrics and Settings

Several datasets consist of images-caption pairs. We use the popular Microsoft
COCO [46] dataset to evaluate our reported results. This dataset contains
123,287 images and each image is paired with 5 sentences. To make a fair
comparison, we use the widely-used splits in the work of Karpathy and Li [37]
to report our results. There are 5,000 images each in validation set and test
set, and 566,435 sentences for training full model.

We test the model performance with MSCOCO captioning evaluation toolkit [46].
It reports the widely-used automatic evaluation metrics SPICE, CIDEr, BLEU,
METEOR and ROUGE. SPICE [47] is based on scene graph matching and CIDEr [48]
is based on n-gram matching. These two metrics are specifically designed to
evaluate image captioning systems. They both incorporate the consensus of a
set of references for an example. BLEU [49] and METEOR [50, 51] are originally
designed for machine translation evaluation, while ROUGE [52] is proposed for
automatic evaluation of the extracted text summarization. A related research
suggests that SPICE correlates the best with human judgments and does
especially well in judging detailedness, where the other metrics present
negative correlations; CIDEr and METEOR follows with no conspicuous
superiority, then comes ROUGE-L, and BLEU-4, in that order [47, 48]. Among
them, SPICE and CIDEr are specifically designed to evaluate image captioning
systems and will be the main considered metrics.

Following convention, we replace caption words appearing less than 5 times in
the training set with the commonly unknown word token UNK. This has resulted
in 9,487 words for MSCOCO. The size of the word embedding $e$ is 512 and the
hidden size $d$ of the LSTM is 512. $N$, the number of heads in multi-head
attention is set to 8 and ${d}_{ff}$, the feed-forward network dimension is
set to 2048. Besides, the hidden layer size ${d}_{a}$ is 350. The relationship
embedding has 30 rows (the $r$), which is similar to [35]. Following [19], we
train the Semantic Concept Extractor on the Visual Genome [53] dataset. Only
top 20 semantic concepts are selected for each image. We also share the
semantic concept embedding and the input word embedding in implementation.
During the training process, firstly, we train the model with the cross-
entropy loss for 15 epochs. Then, in reinforcement learning, we train the
entire model with the batch size of 50 and use Adam for parameter
optimization. For full model, the learning rate is 4e-4. The
$\beta_{1},\beta_{2}$ are respectively set to 0.8 and 0.999. During sampling
for MSCOCO dataset, we apply beam-search with a beam size of 5.

###  V-B Results

####  V-B1 Under the Unpaired Setting

In this section, in order to show the advantages of our method, firstly we
compare our approach with the current unpaired image captioning methods,
including the recently proposed Language Pivoting [17],
Adversarial+Reconstruction [18] and Graph-Align [19], which are the most
advanced on the MSCOCO dataset in comparable settings. As shown in Table I, in
terms of SPICE, our proposed Fine-Grained SRE is competitive with all
baselines, which use complex models and schemes. This is reasonable because
both semantic concepts and semantic relationships are high-level and
understandings of image [28]. Additionally, we boost the performance of Graph-
Align, which indicates vision domain and language domain can be connected by
our approach effectively. As expected, the Fine-Grained SRE has a better
effect than the Coarse-Grained SRE. This indicates that the Fine-Grained SRE
helps improve the quality of the extracted semantic relationships by
extracting relevant relationships in more precise ranges.

####  V-B2 Under the Paired Setting

We have done experiments to further illustrate the advantages of our method.
Under the paired setting, we use five strong baselines without exploring the
semantic relationships for semantic concepts. We replace the source semantic
concept information of the baselines with extracted semantic relationship
information, which is extracted by Fine-Grained Semantic Relationship
Explorer. Since our focus is to provide semantic relationship information, we
preserve the original settings for all baselines.

ATT-FCN [6] attends semantic concepts in all decoding steps. LSTM-A [7] uses a
series of models (LSTM-A2,3,4,5) to incorporate the semantic concepts. We will
study the effect of our method on them. LSTM-A3 takes semantic concepts as
input at the first decoding step and visual features at the second step.
LSTM-A4 presents textual concepts to the decoder, leaving visual features for
the following steps; Contrary to LSTM-A3 and LSTM-A4, LSTM-A2 and LSTM-A5
reverse this order by firstly presenting visible features, respectively.

As shown in Table II, Fine-Grained SRE successfully promotes all baseline
systems. It has brought improvements up to 8% and 6% in terms of CIDEr and
SPICE, respectively, which further demonstrates its wide-range generalizing
ability and indicates its effectiveness in exploring semantic relationships,
which is less prone to the variations of model structures, hyper-parameters
(e.g., learning rate and batch-size), and learning paradigm.

To conclude, we not only performs better than existing methods under the
unpaired setting, but manage to greatly improve the baselines under the paired
setting.

Model | BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4 | METEOR | ROUGE-L | CIDEr | SPICE
---|---|---|---|---|---|---|---|---
GT-ATT [28] | 82.3 | 71.7 | 54.2 | 44.9 | 30.7 | 59.7 | 143.9 | 24.1
GT-SRE | 84.7 | 72.9 | 57.8 | 46.4 | 32.0 | 61.5 | 154.1 | 26.2
GT-SRE-Att | 90.6 | 80.4 | 65.9 | 57.2 | 35.5 | 66.9 | 161.5 | 27.4
GT-SRE-Emb-Att (Full Model) | 92.4 | 86.9 | 74.4 | 63.1 | 36.7 | 69.1 | 174.4 | 28.3
Table III: Performance of the proposed models using the ground-truth semantic concepts on MSCOCO. The attention mechanism (Att) and the embedding mechanism (Emb) can promote the baseline in all metrics. Additionally, we report the performance of a representative system ATT [28]. As we can see, our approach outperforms the ATT substantially in all metrics, which further demonstrates the effectiveness of our approach.  Methods | BLEU-4 | METEOR | ROUGE-L | CIDEr | SPICE
---|---|---|---|---|---
All | Objects | Attributes | Relations | Color | Count | Size
Graph-Align | 21.1 | 20.5 | 46.2 | 68.7 | 14.5 | 28.8 | 3.8 | 3.8 | 1.7 | 0.7 | 1.9
Graph-Align w/ “Attribute” | 21.5 | 20.9 | 47.0 | 71.2 | 15.0 | 29.2 | 5.6 | 3.4 | 3.1 | 2.6 | 2.3
Graph-Align w/ “Object” | 21.3 | 21.4 | 46.7 | 70.1 | 15.6 | 30.7 | 4.2 | 4.3 | 2.4 | 1.5 | 2.0
Graph-Align w/ “Relation” | 20.7 | 20.4 | 46.4 | 69.8 | 14.7 | 29.0 | 3.6 | 4.5 | 2.0 | 1.1 | 1.9
Full Model | 21.8 | 22.1 | 48.4 | 75.7 | 16.1 | 31.0 | 5.8 | 4.9 | 3.4 | 3.7 | 2.5
Table IV: Results of incremental analysis of our proposed approach under the unpaired setting. For a better understanding of the differences, we further list the breakdown of SPICE F-scores. “w/ Attribute”, “w/ Object” and “w/ Relation” stand for the baseline further equipped with the “Attribute” Aggregator, “Object” Aggregator and “Relation” Aggregator, respectively. The bold numbers are best numbers before applying full model. We can see that the “w/ Attribute” has a higher attributes, colors, count and size scores than the other baselines. The “w/ Object” promotes the baseline in objects scores and the “w/ Relation” reaches better scores in relations. As we can see, incorporating all the sub-modules (i.e., Full Model) directly on the baseline leads to overall improvements.  |  |  |  |
---|---|---|---|---
Reference |  a zebra grazing on grass in a green open field. |  a woman is outside playing with her dogs in the grass. |  a herd of wild animals walking across a large dry field. |  a woman playing tennis on a tennis court while people watch.
Graph-Align |  an animal standing on a field. |  a woman standing on a field. |  an animal standing on a field. |  a woman standing on a field.
w/ “Attribute” |  an animal standing on lush green grass field. |  a beautiful woman standing on a grass field. |  a herd of animals standing on a dry field. |  a woman standing on a tennis court.
w/ “Object” |  a zebra standing on a field with grass. |  a woman in shirt and dogs playing on a grass field. |  zebras and giraffes standing on a dry field. |  a woman with a racket standing on a tennis court.
w/ “Relation” |  an animal standing on a field eating grass. |  a woman wearing shirt standing on a grass field with dogs nearby. |  animals walking in a field. |  a woman holding a racket standing on a tennis court.
Full Model |  a zebra eating lush green grass in a grass field. |  a woman wearing black shirt is playing with dogs on a green grass field. |  giraffes and zebras standing on a dry grass field. |  a woman wearing white shirt holding a tennis racket on the court.
Table V: Examples of the captions generated by different methods. The first
line is the input image. The second and the third lines are the ground truth
and the captions generated by the Graph-Align. The fourth, the fifth and the
sixth lines are captions generated by adding “Attribute” Aggregator, “Object”
Aggregator and “Relation” Aggregator, respectively. From the captions, we can
find that the “Attribute” Aggregator helps the baseline to generate more
detailed captions in attributes and colors for each object. The “Object”
Aggregator results in more comprehensiveness in objects, and the “Relation”
Aggregator helps the baseline to establish more reasonable relationships among
objects. The Full Model, which incorporates all sub-modules, is able to
generate more complete captions that are detailed in the objects, attributes,
relations and colors.

##  VI Analysis

In this section, we will first analyze the contribution of each component in
the proposed method. Then examples and error analysis under the unpaired
setting are given to demonstrate the strengths and areas for improvement of
our model. The following analysis are conducted on the the proposed Fine-
Grained SRE.

###  VI-A Quantitative Analysis

The generic captioning performance can be affected by the Semantic
Relationship Embedding and Attention-based Sentence Decoder we adopted.
Therefore, we first assume that ground truth semantic concepts are fixed and
then analyze them. The results are shown in Table III. In addition, because of
our main contribution is proposing the Semantic Relationship Explorer, we
select the Graph-Align to conduct a series of experiments under real unpaired
situations to evaluate the contribution of sub-modules. The results are shown
in Table IV. Meanwhile, we have also listed results of SPICE sub-categories to
make it easier to analyze the quality and differences of the captions.

##### Contribution of Semantic Relationship Embedding and Attention-based
Sentence Decoder

As shown in Table III, the comparison of the three models indicates the upper
limit of the performance of attention mechanism (Att) and embedding mechanism
(Emb). It is demonstrated that the Attention-based Sentence Decoder (GT-SRE-
Att) outperforms the non-attention model (GT-SRE) in all metrics. This proves
that weighing over features can improve the global dependency of features. The
performance of model is greatly improved when it is combined with Semantic
Relationship Embedding (GT-SRE-Emb-Att). This indicates that Semantic
Relationship Embedding can further extract multiple features from semantic
relationships, which strengthens its representing ability.

Furthermore, our full model performs better than GT-ATT. The latter directly
uses semantic concepts at all steps instead of exploring the semantic
relationships for semantic concepts. This proves the necessity to build
relationships between semantic concepts.

##### Effect of “Attribute” Aggregator.

As shown in Table IV, if we incorporate our model with the “Attribute”
Aggregator and aggregate correlated attributes in each specific object, it
will have strengthened attribute relationship representations. Consequently,
the object united with the strengths of attribute collocations can produce a
balanced improvement. This is especially true in attributes, color and count
score. It can also promote the decoder to generate accurate captions.

##### Effect of “Object” Aggregator.

Just as we have expected, “Object” Aggregator performs well in connecting
related objects in the semantic concepts according to a relation word, which
is proved by the increased scores in objects. The object collocations use the
“relation” word as a pivot to extract constrained collocations, which distills
more detailed relationship information between objects, promoting the decoder
to generate comprehensive captions.

##### Effect of “Relation” Aggregator.

As is shown in Table IV, the base model, which directly incorporates
“Relation” Aggregator, have improved relations. This indicates that the
introduction of relation collocations has induced more powerful relation
information to the decoder. The decoder therefore establishes more fine-
grained relationships around a specific object.

##### Full Model.

As we expected, each component, with different functions, brings about
improvements from different aspects to the model. Consequently, their
advantages are united to produce abundant and enriched information. This
achieves deep image understanding, brings an overall improvement (see Table
IV) and results in generating comprehensive and detailed captions (see Figure
V).

###  VI-B Qualitative Analysis

In the Figure V, we list examples to intuitively show the differences between
models. As is shown, all models can generate fluent and descriptive sentences
of input images. However, different models present different amount of
semantic relationship information. The w/ Object has more objects but lacks
details, e.g. color and number. The w/ Attribute generates more detailed
attributes and color. The w/ Relation performs well in portraying the
relations but worse in attributes. The proposed Full Model, compared with the
models mentioned above, helps the baseline to maintain good balances as well
as generates more complete and coherent captions.

The reason for this phenomenon is “Object” Aggregator can automatically extend
its focus in a specific relation and find related objects to establish a
relationship group. “Attributes” Aggregator learns attribute-word collocations
from various visual attributes and rich semantic information. “Relation”
Aggregator learns to automatically extend its focus in a specific object and
look for related relations. Based on rich semantic relationship information,
the decoder can retrieve information through attention mechanism. It will also
decide on the most probable next word.

The qualitative analysis indicates that the semantic relationship information,
extracted by Semantic Relationship Explorer, is much more aggregated than the
source semantic concept information, ensuring the focus of current caption
generation.

Figure 4: Illustrations of the error analysis of the proposal.

###  VI-C Error Analysis

In this section, we conduct error analysis using the proposed (full) model on
the test set to provide insights on how the model may be improved. We find
some generated captions that do not meet the standard. There are mainly three
types of errors, i.e., location (45.9%), attribute (21.7%), and object
(18.4%), with the left (14.0%) falling into other categories. In the Figure 4,
we also give some examples.

Location error means that the model describes an irrelevant location
relationship for an object (e.g., mislocating banana). Attribute error occurs
when there are a lot of attributes, which can respectively establish logical
relationships with a specific object. Therefore, the model mistakenly treats
an incorrect relationship as grounded. In the given example, field can
establish a relationship with either dry or grass, but only grass field is
correct as shown in the picture. Object error occurs when there are incorrect
extracted semantic concepts. The model treats the semantic concept as grounded
in the image. In the given example, the incorrect semantic concept is man.

The challenging part is that these semantic relationships, which are extracted
by the Semantic Relationship Explorer, are logically correct, otherwise the
proposed model will choose other semantic relationships. A more powerful and
accurate Semantic Concept Extractor may be needed to solve the problem, but
it’s unlikely to be completely avoided.

##  VII Conclusions

In this work, we have proposed a framework for unpaired image captioning and a
Semantic Relationship Explorer (SRE). Without any parallel data, SRE extracts
abundant and rich semantic relationship information for image captioning. Our
method employs the semantic relationships to bridge the gap between vision and
language domains. It is a powerful basis for the image description. We also
use Semantic Relationship Embedding to get more powerful relationship
features. Meanwhile the Attention-based Sentence Decoder is adopted to improve
the global dependency of features. Our proposal is validated by experiments on
the MSCOCO image captioning dataset. Remarkably, the proposed method
outperforms the existing methods under the unpaired setting. In addition, it
manages to boost five strong baselines to achieve a significant improvement
under the paired setting. As shown in the quantitative and qualitative
analysis, generated captions achieve a great balance among “object”,
“attribute” and “relation”, which are more complete and coherent compared to
other existing models.

## References

  * [1] F. Liu, M. Gao, T. Zhang, and Y. Zou, “Exploring semantic relationships for image captioning without parallel data,” in _ICDM_ , 2019.
  * [2] A. Das, S. Kottur, K. Gupta, A. Singh, D. Yadav, J. M. F. Moura, D. Parikh, and D. Batra, “Visual dialog,” in _CVPR_ , 2017.
  * [3] X. Liu, H. Li, J. Shao, D. Chen, and X. Wang, “Show, tell and discriminate: Image captioning by self-retrieval with partially labeled data,” in _ECCV_ , 2018.
  * [4] S. Wu, J. Wieland, O. Farivar, and J. Schiller, “Automatic alt-text: Computer-generated image descriptions for blind users on a social network service,” in _CSCW_ , 2017.
  * [5] K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhudinov, R. Zemel, and Y. Bengio, “Show, attend and tell: Neural image caption generation with visual attention,” in _ICML_ , 2015.
  * [6] Q. You, H. Jin, Z. Wang, C. Fang, and J. Luo, “Image captioning with semantic attention,” in _CVPR_ , 2016.
  * [7] T. Yao, Y. Pan, Y. Li, Z. Qiu, and T. Mei, “Boosting image captioning with attributes,” in _ICCV_ , 2017.
  * [8] F. Liu, X. Ren, Y. Liu, H. Wang, and X. Sun, “simnet: Stepwise image-topic merging network for generating detailed and comprehensive image captions,” in _EMNLP_ , 2018.
  * [9] P. Anderson, X. He, C. Buehler, D. Teney, M. Johnson, S. Gould, and L. Zhang, “Bottom-up and top-down attention for image captioning and VQA,” in _CVPR_ , 2018.
  * [10] F. Liu, X. Wu, S. Ge, W. Fan, and Y. Zou, “Federated learning for vision-and-language grounding problems,” in _AAAI_ , 2020.
  * [11] T. Yao, Y. Pan, Y. Li, and T. Mei, “Exploring visual relationship for image captioning,” in _ECCV_ , 2018.
  * [12] F. Liu, X. Ren, Y. Liu, K. Lei, and X. Sun, “Exploring and distilling cross-modal information for image captioning,” in _IJCAI_ , 2019.
  * [13] X. Yang, K. Tang, H. Zhang, and J. Cai, “Auto-encoding scene graphs for image captioning,” in _CVPR_ , 2019.
  * [14] M. Artetxe, G. Labaka, E. Agirre, and K. Cho, “Unsupervised neural machine translation,” in _ICLR_ , 2018.
  * [15] G. Lample, A. Conneau, L. Denoyer, and M. Ranzato, “Unsupervised machine translation using monolingual corpora only,” in _ICLR_ , 2018.
  * [16] G. Lample, M. Ott, A. Conneau, L. Denoyer, and M. Ranzato, “Phrase-based & neural unsupervised machine translation,” in _EMNLP_ , 2018.
  * [17] J. Gu, S. R. Joty, J. Cai, and G. Wang, “Unpaired image captioning by language pivoting,” in _ECCV_ , 2018.
  * [18] Y. Feng, L. Ma, W. Liu, and J. Luo, “Unsupervised image captioning,” in _CVPR_ , 2019.
  * [19] J. Gu, S. R. Joty, J. Cai, H. Zhao, X. Yang, and G. Wang, “Unpaired image captioning via scene graph alignments,” _arXiv preprint arXiv:1903.10658_ , 2019.
  * [20] I. Laina, C. Rupprecht, and N. Navab, “Towards unsupervised image captioning with shared multimodal embeddings,” in _ICCV_ , 2019.
  * [21] J. Gao, Y. Zhou, P. L. H. Yu, and J. Gu, “Unsupervised cross-lingual image captioning,” _arXiv preprint arXiv:2010.01288_ , 2020.
  * [22] S. Cao, G. An, Z. Zheng, and Q. Ruan, “Interactions guided generative adversarial network for unsupervised image captioning,” _Neurocomputing_ , 2020.
  * [23] D. Kim, J. Choi, T. Oh, and I. S. Kweon, “Image captioning with very scarce supervised data: Adversarial semi-supervised learning approach,” in _EMNLP/IJCNLP_ , 2019.
  * [24] P. Vincent, H. Larochelle, Y. Bengio, and P. Manzagol, “Extracting and composing robust features with denoising autoencoders,” in _ICML_ , 2008\.
  * [25] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. C. Courville, and Y. Bengio, “Generative adversarial nets,” in _NIPS_ , 2014.
  * [26] S. Ren, K. He, R. B. Girshick, and J. Sun, “Faster R-CNN: towards real-time object detection with region proposal networks,” in _NIPS_ , 2015.
  * [27] R. Zellers, M. Yatskar, S. Thomson, and Y. Choi, “Neural motifs: Scene graph parsing with global context,” in _CVPR_ , 2018.
  * [28] Q. Wu, C. Shen, L. Liu, A. R. Dick, and A. van den Hengel, “What value do explicit high level concepts have in vision to language problems?” in _CVPR_ , 2016.
  * [29] H. Fang, S. Gupta, F. N. Iandola, R. K. Srivastava, L. Deng, P. Dollár, J. Gao, X. He, M. Mitchell, J. C. Platt, C. L. Zitnick, and G. Zweig, “From captions to visual concepts and back,” in _CVPR_ , 2015.
  * [30] P. Lu, L. Ji, W. Zhang, N. Duan, M. Zhou, and J. Wang, “R-VQA: learning visual relation facts with semantic attention for visual question answering,” in _KDD_ , 2018.
  * [31] D. Yu, J. Fu, T. Mei, and Y. Rui, “Multi-level attention networks for visual question answering,” in _CVPR_ , 2017.
  * [32] C. Zhang, J. C. Platt, and P. A. Viola, “Multiple instance boosting for object detection,” in _NIPS_ , 2006.
  * [33] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in _NIPS_ , 2017.
  * [34] F. Liu, Y. Liu, X. Ren, X. He, K. Lei, and X. Sun, “Aligning visual regions and textual concepts for semantic-grounded image representations,” in _NeurIPS_ , 2019.
  * [35] Z. Lin, M. Feng, C. N. dos Santos, M. Yu, B. Xiang, B. Zhou, and Y. Bengio, “A structured self-attentive sentence embedding,” in _ICLR_ , 2017.
  * [36] J. Lu, C. Xiong, D. Parikh, and R. Socher, “Knowing when to look: Adaptive attention via a visual sentinel for image captioning,” in _CVPR_ , 2017\.
  * [37] A. Karpathy and F. Li, “Deep visual-semantic alignments for generating image descriptions,” in _CVPR_ , 2015.
  * [38] J. Lu, J. Yang, D. Batra, and D. Parikh, “Neural baby talk,” in _CVPR_ , 2018.
  * [39] L. Chen, H. Zhang, J. Xiao, L. Nie, J. Shao, W. Liu, and T. Chua, “SCA-CNN: spatial and channel-wise attention in convolutional networks for image captioning,” in _CVPR_ , 2017.
  * [40] B. Yang, F. Liu, and Y. Zou, “Non-autoregressive video captioning with iterative refinement,” _arXiv preprint arXiv:1911.12018_ , 2019.
  * [41] B. Yang, Y. Zou, F. Liu, and C. Zhang, “Non-autoregressive coarse-to-fine video captioning,” in _AAAI_ , 2021.
  * [42] Z. Gan, C. Gan, X. He, Y. Pu, K. Tran, J. Gao, L. Carin, and L. Deng, “Semantic compositional networks for visual captioning,” in _CVPR_ , 2017\.
  * [43] Y. Wang, Z. Lin, X. Shen, S. Cohen, and G. W. Cottrell, “Skeleton key: Image captioning by skeleton-attribute decomposition,” in _CVPR_ , 2017.
  * [44] S. J. Rennie, E. Marcheret, Y. Mroueh, J. Ross, and V. Goel, “Self-critical sequence training for image captioning,” in _CVPR_ , 2017.
  * [45] F. Liu, M. Gao, Y. Liu, and K. Lei, “Self-adaptive scaling for learnable residual structure,” in _CoNLL_ , 2019.
  * [46] X. Chen, H. Fang, T. Lin, R. Vedantam, S. Gupta, P. Dollár, and C. L. Zitnick, “Microsoft COCO captions: Data collection and evaluation server,” _arXiv preprint arXiv:1504.00325_ , 2015.
  * [47] P. Anderson, B. Fernando, M. Johnson, and S. Gould, “SPICE: Semantic propositional image caption evaluation,” in _ECCV_ , 2016.
  * [48] R. Vedantam, C. L. Zitnick, and D. Parikh, “Cider: Consensus-based image description evaluation,” in _CVPR_ , 2015.
  * [49] K. Papineni, S. Roukos, T. Ward, and W. Zhu, “BLEU: a Method for automatic evaluation of machine translation,” in _ACL_ , 2002.
  * [50] C. Lin and E. H. Hovy, “Automatic evaluation of summaries using n-gram co-occurrence statistics,” in _HLT-NAACL_ , 2003.
  * [51] S. Banerjee and A. Lavie, “METEOR: An automatic metric for MT evaluation with improved correlation with human judgments,” in _ACL_ , 2005.
  * [52] C.-Y. Lin, “ROUGE: A package for automatic evaluation of summaries,” in _ACL_ , 2004.
  * [53] R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L. Li, D. A. Shamma, M. S. Bernstein, and F. Li, “Visual genome: Connecting language and vision using crowdsourced dense image annotations,” _IJCV_ , 2017.


ADCS Preliminary Design For GNB

Alessio Bocci 111†Corresponding Author e-mail<EMAIL_ADDRESS>†,
Giovanni Mingari Scarpello 222Second Author e-mail<EMAIL_ADDRESS>
###### Abstract

This work deals with an ADCS model for a satellite orbiting around Earth. The
object is to achieve a preliminary design and perform some analysis on it. To
do so, a GNB was selected and main properties are exploited. Previous works of
[9], [13], [14], [15] and [17] were analyzed and a synthesis was obtained;
then a suitable control system was designed to satisfy technical requirements.
Coding was performed using Matlab and Simulink.

Keywords: Attitude Determination, Attitude Control, Nanosatellite, Orbital
Perturbations, Quaternion, Two Body Problem, Euler’s Equations, Lyapunov
Function.

## List of Acronyms

ADCS: Attitude Determination and Control System

BRITE: BRIght Target Explorer

ECI: Earth Centered Inertial (Reference Frame)

EPS: Electrical Power System

FOV: Field Of View

GNB: Generic Nanosatellite Bus

OBDH: On Board Data Handling

QUEST: QUaternion ESTimator

RMS: Root Mean Square

RW: Reaction Wheel

SRP: Solar Radiation Pressure

SFL: Space Flight Laboratory

TCS: Thermal Control System

TLE: Two Line Elements

TMTC: Telemetry and Telecommand (System)

TRIAD: TRIaxial Attitude Determination

UTIAS: University of Toronto Institute for Aerospace Studies

mNm: milli-Newton-meter

mNms: milli-Newton-meter-second

##  1 Introduction

Satellites are roughly consisting of two parts, the payload and the satellite
bus. The payload is responsible for completing the primary objectives of the
mission, while the satellite bus supports the payload in its operation
(mechanical structure, EPS, TCS, OBDH, ADCS, TMTC).

In order to reduce the development cost of the bus, the Space Flight
Laboratory [3] developed the so called Generic Nanosatellite Bus (GNB)
designed to work with a wide array of payloads applying minimal changes to the
bus configuration; it is a highly capable and versatile platform designed to
handle a wide variety of missions.

###  1.1 BRITE Constellation Mission

As reported by [1], BRITE (BRIght-star Target Explorer) is a low-cost
Austrian/Canadian constellation of nanosatellites born as a collaborative
science demonstration mission; the original concept of a single satellite
photometry mission has grown to a six satellite constellation with science
teams, engineering teams and funding sources in Canada, Austria and Poland.

As detailed in [21], the aim of the mission is to perform long term stellar
observations in order to detect the variations in apparent luminosity of
certain stars to understand their history, core composition and internal
structure.

BRITE mission measurements will do research about the generation of heavy
elements, the creation of planets, and the ecology of the Universe.

Primary targets are hot luminous stars, massive stars, and stellar supernovae
and, as secondary objective, cool luminous stars, such as Red Giants, which
are rich in carbon and neutron production.

##  2 Satellite Layout

The typical configuration of the BRITE satellite is presented in Figure 1 and
Figure 2, essentially it is a 20 cm x 20 cm x 20 cm cube with a mass of 7 Kg.

The 2 mm thick panels with cross braces provide additional rigidity to the
structure supporting a mounting surface for the solar cells, most sun sensors,
the magnetometer boom, the magnetorquers and the various antennas. All other
components, such as the reaction wheels, various computers on board, battery
assemblies, radios and some of the sun sensors are mounted directly to the
traies.

In [10],[13],[14] and [15] all components, instrumentation (in particular
sensors and actuators for ADCS) and geometrical/mass properties are analyzed
and assembly and integration procedures are presented. In [15] its Inertia
Tensor is reported as:

$J=\left[\begin{array}[]{ccc}0.0465&-0.0007&0.0004\\\
-0.0007&0.0486&-0.0021\\\
0.0004&-0.0021&0.0482\end{array}\right]\quad\text{Kg}\cdot\text{m}^{2}$ (2.1)

Figure 1: BRITE Configuration (courtesy of [1]) Figure 2: BRITE Configuration
- Exploded View (courtesy of [1])

##  3 Environment of the Mission and Mathematical Models

###  3.1 Orbital Motion

From [2], [5] and [6] and is possible evaluate the spacecraft TLEs from his
catalog number. In our case the NORAD CATALOG NUMBER is 40020 and the
following orbit properties are reported in Table 1.

Perigee Altitude | Apogee Altitude | Period | Eccentricity | Inclination
---|---|---|---|---
$611$ Km |  $732$ Km |  $98.18$ min | 0.0085944 | $None$
Table 1: Orbit Properties

Therefore from TLEs is possible to evaluate the initial conditions in terms of
position and velocity for orbit propagation. Denoting with
$\bm{r}=[x,y,z]^{T}$ the position vector of the spacecraft in the ECI
reference frame, the restricted two body problem equation is:

$\ddot{\bm{r}}+\frac{\mu}{||\bm{r}||^{3}}\cdot\bm{r}=\bm{a}_{p}$ (3.1)

where $\mu$ is the Earth gravitational constant.

The $\bm{a}_{p}$ represents the perturbative accelerations acting on the
spacecraft; we considered:

  1. 1.

Perturbation due to Earth oblateness: $J_{2}$ term (see [12] page 491 equation
10.30)

  2. 2.

Perturbation due to Drag: Earth atmosphere effect (see [16] page 108 equation
3.163)

  3. 3.

Perturbation due to Solar Radiation Pressure: SRP (see [16] page 109 equation
3.167)

  4. 4.

Perturbation due to Third Body: Moon (see [12] page 530 equation 10.117)

It is important to underline that as far as SRP and Drag are concerned, the
force is directly correlated to the area seen by the relative velocity vector
and therefore is strictly connected to the attitude that the satellite is
assuming in any instant of time. We assumed the value of the Drag Coefficient
is $C_{d}=2.6$. The eclipse condition is taken into consideration through the
cylindrical approximation:

$\bm{r}\times\bm{e}_{e-s}<-\sqrt{||\bm{r}||^{2}-R_{e}^{2}}$ (3.2)

where $\bm{e}_{e-s}$ is the unit vector from the Earth to the Sun and $R_{e}$
is the Earth radius.

The $J_{2}$ effect is negligible for short time mission operations, but as
long as time passes, its effect becomes evident; in fact the orbit drifting
implies the changing in keplerian elements and in particular on the
inclination of the orbit. Therefore the Earth magnetic field sensed by the
spacecraft will be strongly affected by it.

###  3.2 Attitude Motion

The mathematical model adopted is based on Euler’s equations of motion,
denoting with $\bm{\omega}$ the body’s angular velocity vector and with $J$
its inertia tensor, we have:

$J\dot{\bm{\omega}}=(J\bm{\omega})\wedge\bm{\omega}+\bm{M}_{d}+\bm{L}$ (3.3)

In (3.3) $\bm{L}$ represents the control torque (we will see later how to
define it according to the mission purposes) while $\bm{M}_{d}$ is the
disturbance torque. We considered the following disturbances:

  1. 1.

Gravity Gradient torque (see [16] page 104 equation 3.155)

  2. 2.

Drag Torque (see [16] page 108 equation 3.164)

  3. 3.

Solar Radiation Pressure Torque (see [16] page 109 equation 3.168)

  4. 4.

Magnetic Torque (see [16] page 405 equation 11.5 for the dipole approximation
for the magnetic field and page 106 equation 3.159 for the magnetic torque)

Once (3.3) is integrated, the kinematics in terms of quaternion $\bm{q}$ is
simply:

$\dot{\bm{q}}=\frac{1}{2}U(\bm{\omega})\bm{q}$ (3.4)

where:

$U(\bm{\omega})=\left[\begin{array}[]{cccc}0&\omega_{3}&-\omega_{2}&\omega_{1}\\\
-\omega_{3}&0&\omega_{1}&\omega_{2}\\\ \omega_{2}&-\omega_{1}&0&\omega_{3}\\\
-\omega_{1}&-\omega_{2}&-\omega_{3}&0\end{array}\right]$ (3.5)

##  4 Sensors and Actuators

For attitude determination the typical GNB (see [14]) configuration is:

  1. 1.

6 Sun Sensors (one on each face): unit sphere coverage of $99.8\%$. Main
properties are reported in Table 2.

Mass | Dimensions | Accuracy | FOV | Resolution
---|---|---|---|---
$<6$ g |  $30$ mm x $30$ mm |  $<$$$ RMS |  $None$ (min) $-$ $None$ (max) |  $None$/pixel
Table 2: Sun Sensors Main Properties

  2. 2.

1 Magnetometer: it is structurally located over the body, in order to ensure
minimal measurement corruption by the residual magnetic dipole of the
satellite. This sensor can provide attitude knowledge in the range of $\pm$$$.

  3. 3.

1 Star Tracker: Miniature Star Tracker (MST) developed by AeroAstro which
meets the accuracy requirements of the BRITE mission while fitting into the
volume of only one quarter of the GNB payload bay. Main properties are
reported in Table 3.

Mass | Dimensions | Accuracy | Power | Stars Tracked
---|---|---|---|---
$375$ g |  $6$ cm x $7.62$ cm x $7.62$ cm |  $\pm 70$ arc seconds |  $<2$ W | $>9$
Table 3: Star Tracker Main Properties

For attitude control the typical GNB configuration is:

  1. 1.

3 Orthogonal Reaction Wheels: the Sinclair/SFL [4] reaction wheels are the
primary control actuator for all GNBs and are responsible of slewing the
spacecraft bus to a desired attitude. Main properties are reported in Table 4.

Mass | Dimensions | Moment of | Momentum | Maximum
---|---|---|---|---
|  | Inertia | Capacity | Torque
$185$ g |  $5$ cm x $5$ cm x $4$ cm |  $5.12\cdot 10^{-5}$ $\text{Kg}\cdot\text{m}^{2}$ |  $30$ mNms |  $2$ mNm
Table 4: Star Tracker Main Properties

  2. 2.

3 Orthogonal Magnetorquers: are the second type of control actuator used on
the GNB; they are used for the detumbling phase and for reaction wheels
desaturation. Main properties are reported in Table 5.

Mass | Dimensions | Power Consumption | Dipole Moment
---|---|---|---
$30$ g |  $8$ cm x $8$ cm x $10$ cm |  $300$ mW |  $0.12$ A$\text{m}^{2}$
Table 5: Magnetorquers Main Properties

##  5 Mission Requirements

The ADCS requirements can be formalized as:

  * •

The ADCS shall allow imaging of a target field for 15 minutes continuously for
each orbit, with a goal of imaging multiple targets each orbit

  * •

The ADCS should accommodate observations of multiple targets in a single orbit

  * •

The ADCS shall provide a pointing accuracy less then $None$ (1 arcmin RMS) on
each target

  * •

The Attitude Determination shall require less then 10 arcsec

  * •

The ADCS shall be capable of operating for the full duration of the mission (2
years)

##  6 Main Quantities Estimation and Filtering

###  6.1 Attitude Determination

As already mentioned, the spacecraft is equipped with various sensors for
attitude determination, whose features and operating mechanisms are discussed
in detail in [22].

Our implementation logic is to use the pair sun sensors - magnetometer for
attitude determination in the initial phase of detumbling in order to give to
the star tracker enough time to start correctly and in case of failure of the
latter.

This combination of sensors is also used for avoiding the Sun in order not to
damage the star tracker.

In all other cases, the attitude determination is performed by the star
tracker. Therefore this is a redundant configuration.

When using Sun sensors and magnetometer, the estimation is less precise than
star tracker one; however, if the spacecraft is not in eclipse condition, they
provide two measurements capable of determining the attitude accordingly, in
order to solve the sun sensors - magnetometer attitude determination problem,
the static algorithm TRIAD has been chosen. This method is easy and no
computational problems are present.

On the other hand the star tracker provides much more measurements at the same
time; in fact we need at least eight stars to have $90\%$ of probability to
find four stars in the FOV.

To be more realistic, we decided to build a star catalog in which we
considered eight stars; using their right ascension and declination, director
cosines of the star position vector can be evaluated.

In order to model the FOV limitation a sort "eclipse condition" (3.2) is
adopted. The algorithm logic is that of determine the stars in the FOV, then
remove the other for the next calculations to reduce the computational cost.

The chosen static determination algorithm in this case is the QUEST method
instead of Davenport’s q method since it is more precise but more
computationally expansive.

###  6.2 Angular Velocity Estimation

Since no _rate sensors_ are present the angular velocity $\bm{\omega}$ has to
be estimated ex post (this problem is also analyzed by [11]). The quaternion
kinematics can be described by:

$\dot{\bm{q}}=\frac{1}{2}\Xi(\bm{q})\bm{\omega}\quad\quad\text{with}\quad\Xi(\bm{q})=\left[\begin{array}[]{ccc}q_{4}&-q_{3}&q_{2}\\\
q_{3}&q_{4}&-q_{1}\\\ -q_{2}&q_{1}&q_{4}\\\
-q_{1}&-q_{2}&-q_{3}\end{array}\right]$ (6.1)

Equation (6.1) can be then inverted as:

$\bm{\omega}=2\Xi^{T}(\bm{q})\dot{\bm{q}}$ (6.2)

Then the derivative $\dot{\bm{q}}$ of the quaternion can be evaluated using
finite difference method.

In a same fashion the Attitude Matrix $A_{B/N}$ that represent the map from
the inertial frame ($N$) to the body ($B$) frame (direction cosines matrix)
can be computed as:

$\dot{A}_{B/N}=-[\bm{\omega}\wedge]A_{B/N}\rightarrow[\bm{\omega}\wedge]=-\dot{A}_{B/N}A_{B/N}^{T}$
(6.3)

where:

$[\bm{\omega}\wedge]=\left[\begin{array}[]{ccc}0&-\omega_{3}&\omega_{2}\\\
\omega_{3}&0&-\omega_{1}\\\ -\omega_{2}&\omega_{1}&0\end{array}\right]$ (6.4)

and finally we have:

$\bm{\omega}=(-\dot{A}_{B/N}A_{B/N}^{T})^{V},\quad\text{with}\quad(P)^{V}=[P_{32},P_{13},P_{21}]^{T}$
(6.5)

In (6.5) P is a generic matrix. Then the derivative $\dot{A}_{B/N}$ of the
Attitude Matrix can be evaluated using finite difference method.

###  6.3 Filtering

The TRIAD algorithm and the QUEST method provide a good approximation of
quaternion and attitude matrix despite the random error related to the
inaccuracy of the sensors.

On the other hand, the numerical derivative generates noisy signals that have
to be filtered. In order to eliminate high frequency noise without loosing too
much information from the signal, two different low pass filter are
implemented whose transfer functions are:

$F_{1}(s)=\frac{\tilde{\omega}_{1}}{s+\tilde{\omega}_{1}},\quad
F_{2}(s)=\frac{\tilde{\omega}_{2}^{2}}{1+2\xi\tilde{\omega}_{2}s+\tilde{\omega}_{2}^{2}}$
(6.6)

where $\tilde{\omega}_{1}$ and $\tilde{\omega}_{2}$ are the cut off
frequencies of the first order and second order low pass filter respectively
and $\xi$ is the damping ratio of the second order low pass filter.

All these parameters are optimally tuned in order to obtain the best
performances depending on each mission operation phase.

##  7 Detumbling Maneuver

The detumble mode shall reduce the satellite rotation rates after release from
the launcher, and after entering safe-hold mode, to the point where satellite
angular momentum is low enough to be absorbed by reaction wheels. For BRITE,
this is designed to recover from tumble rates up to 60 deg/s.

The control law is chosen as a combination of a bang-bang control and a pure
B-dot control law 333This is a simplified magnetic control law, more advanced
ones are present in literature such as [18]:

$\bm{L}_{det}=\begin{cases}-\bm{b}\wedge\bm{m}&\quad\text{if}\quad||\bm{\omega}||\geq
1\deg/s\\\
-k_{det}(\bm{b}\wedge\bm{\omega})\wedge\bm{b}&\quad\text{if}\quad||\bm{\omega}||<1\deg/s\end{cases}$
(7.1)

Where:

$\bm{b}=\dfrac{\bm{B}}{||\bm{B}||},\quad\bm{m}=-m_{max}\cdot\text{sign}(\dot{\bm{b}})\quad\text{and}\quad
k_{det}=\dfrac{4\pi}{T_{orb}}[1+\sin(\xi_{m})]J_{min}$ (7.2)

Where $\bm{B}$ is the Earth magnetic field vector, $\bm{m}$ is the dipole
moment of the magnetorquer, $m_{max}$ is the maximum magnetic dipole of the
magnetorquer, $T_{orb}$ is the orbital period, $\xi_{m}$ is the inclination of
the spacecraft orbit relative to the geomagnetic equatorial plane and
$J_{min}$ is the minimum principal moment of inertia. The formula for
$k_{det}$ in (7.2) can be found in [16] and a detailed proof is in [7].
Typical values of $m_{max}$ for GNB satellites are presented in [9] , in our
case $m_{max}=0.12$ $Am^{2}$.

The satellite is equipped with one star tracker, six sun sensors and one
magnetometer, therefore the implementation logic is such that the attitude
sensing is performed using sun sensors and the magnetometer for at least 60
minutes (which more or less correspond with the time the satellite is not in
eclipse) end possibly, if needed, the star tracker performs the remaining of
the attitude sensing to conclude the detumbling. This strategy is done in
order to give enough time to the star tracker to switch on correctly (maximum
time 60 minutes).

Figure 3: Detumbling Simulation Results - Angular Velocity Components
Behaviour

Figure 3 shows the behaviour of the angular velocity components, the initial
condition $\bm{\omega}(0)$ is selected as the limit one; we can see that the
asymptotic zero condition is achieved more or less in 6 hours.

Figure 4: Detumbling Simulation Results - Dipole moment components Figure 5:
Detumbling Simulation Results - Control Torque components

Figure 4 presents the behaviours of magnetic dipole $\bm{m}$ components; here
we can appreciate the difference and the switching of two methods. Figure 5
presents the required torque, called $\bm{L}$ for simplicity. Such a
simulation, with so onerous initial conditions, shows also the robustness of
the control design; in fact it is capable of detumbling successfully the
spacecraft despite the long amount of time required. Therefore less onerous
initial conditions will require less effort.

##  8 Slew and Tracking Maneuvers

After the detumbling maneuver, the spacecraft shall point to the target. For
our purposes, the goal is to point a fixed star. Two main consequences can be
identified: the _reference attitude_ (desired attitude) is not time varying
with respect the ECI reference frame and the _desired angular velocity_ during
the tracking is zero. Therefore we are allowed to formulate the tracking and
slew problem on the same time as:

$\bm{L}_{s-t}=-k_{\omega}\bm{\omega}+\bm{\omega}\wedge(J\bm{\omega})-k_{A}(A_{e}^{T}-A_{e})^{V}$
(8.1)

Where:

$(A_{e}^{T}-A_{e})^{V}=[\hat{A}_{32},\hat{A}_{13},\hat{A}_{21}]^{T},\quad\text{with}\quad\hat{A}=A_{e}^{T}-A_{e}$

Equation (8.1) can be proved as follows:

###### Proof.

Let $\bm{\omega}_{e}=\bm{\omega}-A_{e}\bm{\omega}_{d}$ the error in angular
velocity with respect to the desired set point $\bm{\omega}_{d}$. The
condition of equilibrium to be studied is $\bm{\omega}_{e}=\bm{0}$, then a
suitable Lyapunov function is:

$V(\bm{\omega}_{e})=\frac{1}{2}\bm{\omega}_{e}^{T}J\bm{\omega}_{e}+k_{A}\text{tr}(I-A_{e})$
(8.2)

which is zero for $\bm{\omega}_{e}=\bm{0}$ and $A_{e}=I$. The chosen Lyapunov
function is never less than zero because the first term is a quadratic form
and the trace surely greater than zero. Performing the time derivative of
(8.2) we have:

$\displaystyle\dot{V}(\bm{\omega}_{e})$
$\displaystyle=\bm{\omega}_{e}^{T}J\dot{\bm{\omega}}_{e}+k_{A}\text{tr}([\bm{\omega}\wedge]A_{e})$
$\displaystyle=\bm{\omega}_{e}^{T}\left[(J\bm{\omega})\wedge\bm{\omega}+\bm{L}-JA_{e}\dot{\bm{\omega}}_{d}+J[\bm{\omega}_{e}\wedge]A_{e}\bm{\omega}_{d}+k_{A}(A_{e}^{T}-A_{e})\right]$

Where the external disturbances are not considered since in general it is not
possible to compensate them. Finally we derive the general control law:

$\bm{L}=-k_{\omega}\bm{\omega}_{e}+\bm{\omega}\wedge(J\bm{\omega})+J(A_{e}\dot{\bm{\omega}}_{d}-[\bm{\omega}_{e}\wedge]A_{e}\bm{\omega}_{d})-k_{A}(A_{e}^{T}-A_{e})^{V}$
(8.3)

which has been chosen in such a way that $\dot{V}(\bm{\omega}_{e})<0$.

By setting $\bm{\omega}_{d}=\bm{0}$ in (8.3) we get (8.1) and the system is
asymptotically stable. ∎

From (8.1) we can see that first of all we perform a feedback linearization by
means of the term $\bm{\omega}\wedge(J\bm{\omega})$ to the Euler’s equations
and then two proportional actions are applied: one $k_{\omega}$ on the angular
velocity and the other $k_{A}$ on the attitude error ($\\{k_{\omega},\quad
k_{a}\\}>0$).

Equation (8.1) represents the so called desired control law, then we map it
onto actuators solving the following differential equation:

$\dot{\bm{h}}_{rw}=-\tilde{A}^{*}[\bm{L}_{s-t}+\bm{\omega}\wedge(\tilde{A}\bm{h}_{rw})]$
(8.4)

where $\tilde{A}$ is the distribution matrix of the momentum exchange devices
and $\tilde{A}^{*}$ its pseudo-inverse (since we have three orthogonal
reaction wheels $\tilde{A}=\tilde{A}^{*}=I\in\mathbb{R}^{3\times 3}$),
$\bm{h}_{rw}$ is the angular momentum vector of the reaction wheels. Equation
(8.4) can be proven as follows:

###### Proof.

The total angular momentum of the spacecraft can be expressed as:

$\bm{h}_{tot}=J\bm{\omega}+\tilde{A}\bm{h}_{rw}$ (8.5)

performing time derivative of (8.5) and remembering that matrix $J$ and
$\tilde{A}$ are not time varying, we have:

$J\dot{\bm{\omega}}+\bm{\omega}\wedge(J\bm{\omega})+\tilde{A}\dot{\bm{h}}_{rw}+\bm{\omega}\wedge(\tilde{A}\bm{h}_{rw})=\bm{L}+\bm{M}_{d}$
(8.6)

Then (8.4) is obtained inverting the relation
$\bm{L}=-\tilde{A}\dot{\bm{h}}_{rw}-\bm{\omega}\wedge(\tilde{A}\bm{h}_{rw})$ ∎

Without any loss of generality, we considered as target the star Alpha Circini
(a variable star in the faint, southern, circumpolar constellation of
Circinus) that has an _apparent magnitude_ of 3.18 - 3.21 (that was also
imaged and studied by BRITE Austria).

Alpha Circini’s properties with respect to the ECI reference frame are:

$r_{cir-sun}=54\quad\text{ly},\quad\alpha_{cir}=14h\quad 42m\quad
30.41958s,\quad\delta_{cir}=-$$\quad 58^{\prime}\quad 30.4934^{\prime\prime}$
(8.7)

where $r_{cir-sun}$ is the distance from the Sun, $\alpha_{cir}$ is the Right
Ascension and $\delta_{cir}$ is the declination of the star. Therefore from
the information in (8.7) we can easily compute the position vector with
respect to the ECI reference frame.

If we now consider the unit vector of the x-axis of the ECI reference frame
$\bm{i}=[1,0,0]^{T}$ than the quaternion to rotate $\bm{i}$ into the direction
of $\bm{r}_{cir-sun}$ can be evaluated as:

$\bm{q_{d}}=[\cos(\psi/2),\bm{u}\sin(\psi/2)]^{T},\quad\bm{u}=\frac{\bm{i}\wedge\bm{r}_{cir-
sun}}{||\bm{i}\wedge\bm{r}_{cir-
sun}||},\quad\psi=\cos^{-1}\left(\frac{\bm{i}\times\bm{r}_{cir-
sun}}{||\bm{r}_{cir-sun}||}\right)$ (8.8)

From (8.8) the desired attitude matrix and the attitude error
($A_{e}=A_{B/N}A_{d}^{T}$) can be evaluated by the usual formulas for
converting a quaternion into attitude matrix.

Finally we can evaluate the most important performance parameter for this
mission phase, the so called pointing error that we define as:

$\theta_{e}=\cos^{-1}\left(\frac{\text{tr}(A_{e})-1}{2}\right)$ (8.9)

Equation (8.9) provides a criterion for deciding if we are or not satisfying
the pointing requirement.

The initial conditions for the slew maneuver are those at the end of the
detumbling one both in terms of angular velocity and attitude quaternion.

In our design is of course necessary to provide a desaturation mechanism for
the reaction wheels; this is done by magnetotorquers. If we define $M_{max}$
the maximum torque that can be produced by a reaction wheel then the torque to
be applied to the RW and the torque to be produced by the magnetorquers to be
applied to the spacecraft are:

$\bm{T}_{rw}=-M_{max}\cdot\text{sign}(\bm{h}_{rw})\quad\text{and}\quad\bm{L}_{des}=\bm{m}_{des}\wedge\bm{B},\quad\bm{m}_{des}=M_{max}\cdot\frac{\text{sign}(\bm{h}_{rw})\wedge\bm{b}}{||\bm{B}||}$
(8.10)

Where $M_{max}$ is the maximum torque that reaction wheels can provide.
Equation (8.10) provides the desaturation mechanism and it is effective if the
magnetic field is orthogonal to the spin axis of the wheel; this yields to the
possibility of making the desaturation when we reach such a condition
preventing the saturation before it actually happens.

Figure 6: Slew and Tracking Simulation Results - Angular Velocity Components
Behaviour Figure 7: Slew and Tracking Simulation Results - Required Torque
Figure 8: Slew and Tracking Simulation Results - Pointing Accuracy Figure 9:
Slew and Tracking Simulation Results - Attitude Error

Figure 6 and Figure 7 show respectively the behaviour of the angular velocity
and torque components (in Figure 7 the torque is labelled as $L$ and not
$L_{s-t}$ as in (8.1) for simplicity); as expected $\bm{\omega}$ start with
the zero condition achieved from the detumbling and then start to increase
(thanks to the RW actions) to line up the camera with the target direction.

In Figure 8 we can see how the pointing error decreases till the required
pointing accuracy $<$$$ (in our calculation $\approx$$$), this means that the
attitude error tends to the identity matrix as shown in Figure 9.

These calculations are done considering the saturation and de-saturation
mechanism according to (8.10), however with the designed control the reaction
wheels are more or less working under the saturation limit.

##  9 Re-Pointing

The technical requirements of the ADCS subsystem establish that during one
orbital period the satellite must orient the camera towards more than one star
and keep this direction for at least 15 minutes (multiple targets each orbit).

Let us consider as the first target Alpha Circini and as second target Tania
Australis.

Tania Australis is a binary star in the constellation of Ursa Major. An
apparent visual magnitude of +3.06 places it among the brighter members of the
constellation. This star has the following properties with respect to the ECI
reference frame are:

$r_{tania-sun}=230\quad\text{ly},\quad\alpha_{tania}=10h\quad 22m\quad
19.73976s,\quad\delta_{tania}=$$\quad 29^{\prime}\quad 58.2691^{\prime\prime}$

Figure 10 , Figure 11, Figure 12 and Figure 13 show the results mentioned
above.

It can be appreciated how the control system is able to reorient the satellite
ensuring the required pointing accuracy. Also in this case the limits imposed
by the reaction wheels are respected and the attitude error converges to the
identity matrix.

Figure 10: Slew and Tracking Simulation Results - Angular Velocity Components
Behaviour (Re-pointing) Figure 11: Slew and Tracking Simulation Results -
Required Torque (Re-pointing) Figure 12: Slew and Tracking Simulation Results
- Pointing Accuracy (Re-pointing) Figure 13: Slew and Tracking Simulation
Results - Attitude Error (Re-pointing)

##  10 Conclusions

Spacecraft attitude and control in short consist of a continuous check that a
satellite points in some desired direction. But really attitude and control
should be estimated simultaneously, nevertheless they are separated to some
extent due to a "separation theorem". In such a way, for the most space
missions design, attitude determination dynamics can be designed separately
from the control. The case study presented here holds almost the main theories
and the state of the art, analyzing the transient performances of the control
system by solving, by suitable numeric algorithms, the nonlinear differential
equations modeling its dynamics and stability. After a short description of
the system features, we performed several simulations. The main concern
angular velocity, dipole moments and control torques during the tumbling.
Afterwards angular velocity, required torque and pointing accuracy have been
monitored during slew and star tracking. In any case we could ascertain that
all the transients behave in due way and extinguish within the prescribed
duration according to design specifications and according to our stability
analysis in the sense of Lyapunov. The paper can be seen as our first step on
the subject: further advanced techniques of control have been taken also into
account (see [8], [19] and [20]) and will be exploited in a future work.

## References

  * [1] –  “eoPortal Website”, https://directory.eoportal.org/web/eoportal/home
  * [2] –  “HORIZONS Web-Interface”, https://ssd.jpl.nasa.gov/horizons.cgi
  * [3] –  “SFL Website”, https://www.utias-sfl.net
  * [4] –  “Sinclair Interplanetary Website”, http://www.sinclairinterplanetary.com
  * [5] –  “Space-Track Website”, https://www.space-track.org/auth/login
  * [6] –  “T. S. Kelso “CeleStrak””, https://www.celestrak.com/NORAD/elements/
  * [7] Giulio Avanzini and Fabrizio Giulietti  “Magnetic Detumbling Of A Rigid Spacecraft”  In _Journal of guidance, control, and dynamics_ 35.4, 2012, pp. 1326–1334
  * [8] James D Biggs, Yuliang Bai and Helen Henninger  “Attitude Guidance And Tracking For Spacecraft With Two Reaction Wheels”  In _International Journal of Control_ 91.4 Taylor & Francis, 2018, pp. 926–936
  * [9] Kieran Carroll, Slavek Rucinski and Robert Zee  “Arc-Minute Nanosatellite Attitude Control: Enabling Technology For The BRITE Stellar Photometry Mission”  In _18th Annual AIAA/USU Conference on Small Satellites_ , 2004
  * [10] Guy de Carufel  “Assembly, Integration And Thermal Testing Of The Generic Nanosatellite Bus”, 2009
  * [11] Alexandre Cortiella et al.  “3CAT-2: Attitude Determination And Control System For A GNSS-R Earth Observation 6U Cubesat Mission”  In _European journal of remote sensing_ 49.1 Taylor & Francis, 2016, pp. 759–776
  * [12] Howard D Curtis  “Orbital Mechanics For Engineering Students”  Butterworth-Heinemann, 2020
  * [13] Jennifer Elliott  “The Thermal Design And Analysis Of The CanX-4/-5 And NEMO-AM Nanosatellites”, 2014
  * [14] Michael R. Greene  “The Attitude Determination And Control System Of The Generic Nanosatellite Bus”, 2009
  * [15] Bryan Johnston-Lemke  “High Performance Attitude Determination and Control for Nanosatellite Missions”, 2011
  * [16] F Landis Markley and John L Crassidis  “Fundamentals Of Spacecraft Attitude Determination And Control”  Springer, 2014
  * [17] Adam Philip  “Attitude Sensing, Actuation, And Control Of The BRITE And CanX-4 & 5 Satellites”  University of Toronto, 2008
  * [18] Antonio L Rodriquez-Vazquez, Maria A Martin-Prats and Franco Bernelli-Zazzera  “Spacecraft Magnetic Attitude Control Using Approximating Sequence Riccati Equations”  In _IEEE transactions on Aerospace and Electronic Systems_ 51.4 IEEE, 2015, pp. 3374–3385
  * [19] Emmanuel Sin et al.  “Attitude Trajectory Optimization for Agile Satellites in Autonomous Remote Sensing Constellation”  In _arXiv preprint arXiv:2102.07940_ , 2021
  * [20] Miaomiao Wang, Soulaimane Berkane and Abdelhamid Tayebi  “Nonlinear Observers Design For Vision-Aided Inertial Navigation Systems”  In _arXiv preprint arXiv:2102.05111_ , 2021
  * [21] WW Weiss et al.  “BRITE-Constellation: Nanosatellites For Precision Photometry Of Bright Stars”  In _Publications of the Astronomical Society of the Pacific_ 126.940 IOP Publishing, 2014, pp. 573
  * [22] James R Wertz  “Spacecraft Attitude Determination And Control”  Springer Science & Business Media, 2012