**footnotetext: Equal contribution. ${}^{\text{\Yinyang}}$ Equal
advising.**footnotetext: All figures and pictures by the author(s) under a CC
BY 4.0 license.

# Adversarial domain adaptation to reduce sample bias of a high energy physics
event classifier

J.M. Clavijo  P. Glaysher  J. Jitsev  and J.M. Katzy

###### Abstract

We apply adversarial domain adaptation in unsupervised setting to reduce
sample bias in a supervised high energy physics events classifier training. We
make use of a neural network containing event and domain classifier with a
gradient reversal layer to simultaneously enable signal versus background
events classification on the one hand, while on the other hand minimising the
difference in response of the network to background samples originating from
different MC models via adversarial domain classification loss. We show the
successful bias removal on the example of simulated events at the LHC with
$t\bar{t}H$ signal versus $t\bar{t}b\bar{b}$ background classification and
discuss implications and limitations of the method

##  1 Introduction

Many measurements and searches for new phenomena performed by the experiments
at the Large Hadron Collider (LHC) use a classification algorithm, such as
Boosted Decision Trees or Neural Networks, to discriminate the physics process
of interest (signal) from other physics processes with similar signature
(background). The algorithms are optimized using supervised training on
detailed Monte Carlo (MC) simulation data sets, containing samples labeled as
signal or background. The resulting classifier is applied to unlabeled data to
separate signal and background, and to measure the statistical significance of
the signal or its strength, assuming that the simulated and the real data sets
are identically distributed.

However, significant differences between domains of real and simulated data
sets always exist and the learner may pick up those domain-specific
discriminating features that perform well on classification task in one domain
while being not suitable for classification in the other, introducing a bias
via the source samples used for training when attempting to classify samples
from target domain. This problem is similar to that of visual recognition
where, for instance, training may be performed on artificially generated
images, the source domain, and applied to real photographs, the target domain.
In order to avoid training a model that is suitable for classification on the
source domain only, while failing when employed on target domain, algorithms
of domain adaptation have been developed.

In this paper we apply the method of domain adaptation to a problem of
classification on high energy physics data using a Domain Adversarial Neural
Network (DANN) [2] to classify events in the search for the
$t\bar{t}H(H\rightarrow b\bar{b})$ process at the LHC, which is very rare and
hard to separate from the abundant $t\bar{t}+\text{jets}$ background [1]. In
the cited measurement work, a classifier is trained on labeled MC predictions
to separate signal from background. The trained classifier is applied on MC
where signal and background events are mixed according to the theoretical
predicted fraction, and on data to obtain binned distributions of classifier
output. The ratio of the resulting spectra is used in a profile likelihood fit
to measure the signal ratio in data. The effect on the final result caused by
the bias for the specific MC background model of the source domain used for
training is estimated using an alternative simulation of a target domain,
based on a different physics model, which was not used for training. The
difference between the classifier outputs of the different background MCs is
taken as uncertainty on the classification in the fit. This uncertainty
happens to be the largest on the measurement, hampering the observation of the
searched process. Therefore a solution to minimize this sample bias is of high
importance. For the study presented here, the two background simulations
correspond to the different domains. The domain adaptation is applied to
reduce mentioned training bias.

The network structure consists of a common feature extractor part and separate
branches for label classification and domain adaptation, implemented via a
gradient reversal layer as presented in [2]. This network structure differs
from other adversarial approaches by including domain adaptation in the
learning process via the shared feature extractor part used by both label and
domain classifier as proposed on theoretical grounds in [3]. This way, the
network model is pushed to extract discriminant features for the
classification that are at the same time invariant to the different domains.
The use case presented here differs from [2] as we provide a set of physical
jet properties instead of images as input and the use of a bigger and more
complex data set. Additionally, we make detailed performance analyses,
evaluating the influence of several hyper-parameters and also exploring
training issues that appear for this kind of architecture.

Adversarial classifiers without domain adaptation were used in high energy
physics before, e.g. to reduce theoretical uncertainties [4], to decorrelate a
jet tagger from the jet mass [5] and to tune a classifier against a nuisance
parameter [8]. An adversarial set-up involving domain adaptation without
labels has been used for multi-class classification in a search for long lived
particles [6]. In this paper, we systematically study the algorithm taking
advantage of the two domains being labelled to control the results achieved
without use of labels. We address the challenges for learning algorithms in
domain adaptation, the dependence of the hyper-parameters specific to domain
adaptation and potential bias of the classifier. Furthermore, since the use of
domain adaptation without labels for multi-modal distributions can be a
problem as pointed out in e.g. [7], we chose to use the domain adaptation
algorithm for binary classification and proof its applicability for this use
case in contrast to the multi-class application mentioned above.

In this paper we describe the network used in Sec. 2, followed by the details
of the data sets used in Sec. 3. We systematically study the dependence on
hyper parameters in Sec. 4, including some issues observed during the
training. In Sec. 5, we expose the performance through different figure of
merits related to physics searches and we include a feasibility study for a
potential use with real unlabeled data. Finally, a summary and some
conclusions are given in Sec. 6.

##  2 The Deep Adversarial Neural Network

We follow the architecture presented in [2] with a feed-forward neural network
composed of three parts as shown in Fig. 1: a _feature extractor_ which splits
into the _label predictor_ , performing the signal-background classification,
and the _domain classifier_. Domain adaptation is enabled via an adversarial
interplay between domain classifier and feature extractor. For training and
testing we have two data sets (domains): source and target, both containing
signal and background events. The target domain is constructed as a
representative pseudo-data, meaning that it is treated as unlabeled and it has
a signal to background proportion similar to the one expected in a real data
sample. For measuring our algorithm performance we make use of the target
labels in the final test.

For the label classification we train the network only using events from the
source domain. The gate layer stops the target events propagation making the
_label predictor_ loss being evaluated only on the source events. This allows
training the network on mixed samples of both domains. The classification is
adapted to the target domain by connecting the _feature extractor_ with the
_domain classifier_ through a gradient reversal layer. This layer does nothing
during the forward propagation but inverts the sign of the gradients flowing
from domain classifier during the backpropagation. The _domain classifier_ is
trained to determine which domain the events belong to. Due to the gradient
reversal, the _feature extractor_ is pushed to form such feature
representation that do not allow to distinguish between two different domains,
thus avoiding the sample bias. As a result of such adversarial training, the
features in the last layer of the _feature extractor_ will both allow the
classification between signal and background and become domain invariant,
rendering classification model domain-independent. The gradients of the
reversal layer are scaled by the parameter $\lambda$ allowing to regularize
their influence and hence tune the importance of the label classification
versus the domain invariance.

Figure 1: Domain-adversarial network as an alternative to reduce
classification bias, adapted from [2].

In order to have balanced classes for each classification the event weights of
the source domain are scaled as required according to class ratios. For the
label predictor the weights are such that the effective number of signal and
background events are the same. For the domain classifier, the weights are
scaled to match the signal to background ratio existing in the target domain.

##  3 Data sets

The feature selection for the input of the network was inspired by the
analysis presented in [1] to separate $t\bar{t}H$ from the $t\bar{t}+b\bar{b}$
background. In total 41 geometrical and kinematic quantities are used as input
to the network, such as the angular distance between different jets and/or
leptons, the mass of various jet and lepton systems and the event topology.
The complete list of features, their correlations and the relative importance
are given in [9].

We use MC samples provided by the HepSim Group [10]. The ttH signal sample
containing $13\cdot 10^{6}$ events was generated with MadGraph [11] matched to
the Herwig6 parton shower [12]. Two background samples were generated,
significantly differing in the theoretical predictions. One, used for the
source domain, with $2\cdot 10^{6}$ events of top quark pair production with
additional light quarks using MadGraph matched to the Herwig6 merged with
$10^{7}$ top quark pair events with additional bottom-quarks using MadGraph
matched to Pythia6 [13]. The other background sample, which is used for the
target domain, contains $3\cdot 10^{7}$ events of top quark pair production in
association with bottom quark pairs, generated with the PowhegBox+OpenLoops
[14] and is matched to Pythia8 for the full event generation including the
prediction of additional light quarks.

The ATLAS detector response was simulated using Delphes simulation [15]. For
this study, reconstructed leptons, jets and bottom111$bottom$ stands for
bottom and anti-bottom quarks quark iniated jets (called $b$-jets in the
following) are used. Jets are reconstructed using the anti-kT algorithm [16]
with a radius of $R=0.4$. The identification efficiency of $b$-jets was taken
from [17], assuming the reconstructed $b$-jets to have a 70% tagging
probability with a corresponding light jet/c-jet rejection probability
parameterization.

Events selected for the neural network training were required to fulfil the
following criteria:

  * •

one electron or muon with transverse momentum $p_{T}$$\geq$ 20 GeV

  * •

at least 5 jets with $p_{T}$$\geq$ 25 GeV

  * •

at least 3 $b$-jets.

With this selection applied the source and target data sets where constructed
with $546\cdot 10^{3}$ signal each and same amount of background events, using
statistically independent events from the same simulation as signal but
different background simulations for source and target. One half from each
data set was left for testing purposes, the remaining were used for training.
For the target domain only $14368$ signal events were randomly selected for
training, to match the 5:95 ratio of signal to background estimated in real
data.

##  4 Network set-up and training

The network was implemented using the Keras v2.2.4 [18] with TensorFlow v1.12
[19] as back-end library. The training set-up is described in Sec. 4.1. A
hyper-parameter scan was done to optimize the performance of the network, as
described in Sec. 4.2. Some special considerations for the loss function and
its optimization are described in Sec. 4.3 and Sec. 4.4, respectively.

###  4.1 Training set-up

The initial weights of the network were set by the Xavier initializer, as
suggested in [20]. The number of training epochs was dynamically selected with
the following condition applied: the training were stopped if the running
average over 50 epochs in the total loss does not decrease more than 0.05%
with respect to the previous 50 epochs. This number was restricted to the
interval [200, 1000]. The lower limit was set to skip some random fluctuations
at the beginning. The upper limit is just a big value that was never reached
with the specified condition. After the training was stopped the weights of
the network in the epoch with the lowest label predictor loss were selected. A
batch size of 16384 was used. Each batch was composed by source and target
events in a 1:1 proportion. The events were randomly shuffled at each epoch,
resulting in a different batch selection each time. The _domain classifier_
and _label predictor_ outputs were set to have two neurons each, using softmax
activation function and cross-entropy loss in both (Sec. 4.3 describes an
alternative). The RMSProp Keras optimizer was used, with the parameters:
learning_rate =0.001 and rho=0.9.

###  4.2 Hyper-parameter optimization

The hyper-parameters of the network were chosen with the help of the Hyperopt
library [21], using the Tree of Parzen Estimators (TPE) algorithm implemented
on it. The number of layers in each part of the network was let vary from 1 to
8. Each layer could have a number of neurons between 5 and 100, but having a
linear behavior in each part of the network (either decreasing or increasing).
For the activation function of the hidden layers ReLU, tanh and ELU were
tested. Each of this hyper-parameters were sampled from a uniform
distribution. Additionally, the $\lambda$ parameter was sampled from a log-
uniform distribution in the range [1, 1000], with this giving more priority to
low values as these were found to give better results.

The additive inverse of the label _label predictor_ area under the receiver
operating characteristic curve for the target domain was used as the loss to
minimize. Three independent optimizations where performed in parallel in order
to have a better view of the hyper-parameter space. This also helps to detect
if the global minimum of the loss is found. Approximately 1000 iterations
where performed in each case.

By analyzing the sets of parameters with good performance and the decisions
made by the sampling algorithm, we were able to draw the following
conclusions:

  * •

The optimal number of layers in the _label predictor_ is one: only the output
layer. Two is also good in cases of a very complex _feature extractor_.

  * •

Higher complexity in the _feature extractor_ provides performance improvement
but also makes the network more prone to over-training.

  * •

The number of neurons in the last layer of the feature extractor should be at
most the same that in the input. We think this number is also related to the
correlations in the input features: a smaller number for high correlations
could provide a better optimized feature extraction.

  * •

An increase in the domain classifier complexity does not cause significant
improvements, but it needs at least a similar complexity than the feature
extractor in order to provide good corrections.

  * •

The performance with ELU and tanh as activation function for the hidden layers
was very similar. ReLU was significantly worse.

Finally the _feature extractor_ was chosen to have four layers with 20, 16, 13
and 10 neurons respectively, the _label predictor_ with only the output layer
(2 neurons) and the _domain classifier_ with four layers of 20, 35, 50 and 2
neurons respectively. The ELU activation function was used in all the hidden
layers.

Note that due to the non-deterministic nature of the training process, results
during the optimization were sometimes not representative of the behavior for
each set of hyper-parameters tested. Set-ups with higher performance were
found, but its results were not reproduced in further tests. Therefore, we
chose a configuration with stable results instead of the best one reported by
the optimization process. It also had the advantage of being not complex
enough to be affected by over-training.

###  4.3 Loss and activation functions for the outputs

The total loss of the network ($L$) is given by the sum of the individual
losses of the _label predictor_ ($L_{y}$) and _domain classifier_ ($L_{d}$):

$L=L_{y}+L_{d}$ (4.1)

The gradient reversal layer affect the backpropagation in such a way that the
gradients of the total loss with respect to the _feature extractor_ weights
($\theta_{f}$) are computed as:

$\frac{\partial L}{\partial\theta_{f}}=\frac{\partial
L_{y}}{\partial\theta_{f}}-\lambda\frac{\partial L_{d}}{\partial\theta_{f}}$
(4.2)

Two alternatives were used for computing the loss: set-up A with a softmax
activation and cross-entropy loss in both outputs, and set-up B with softmax
and cross-entropy loss in the _label predictor_ , and linear loss following
eqn.4.4 in the _domain classifier_.

The cross-entropy loss for a single event $E_{i}$ is given by:

$L_{i}=\begin{cases}-\ln(y_{i})&\text{if }E_{i}\in\text{class 1}\\\
-\ln(1-y_{i})&\text{if }E_{i}\in\text{class 0}\end{cases}$ (4.3)

where $y_{i}$ represents the network output for that event. Note that even
though we have a two-neuron output we refer to $y_{i}$ as a single value since
the second neuron behaves as 1 minus the first. Class 0 corresponds to
background and class 1 to signal for the _label predictor_. A perfect
classification yields a loss of 0, value toward which the loss is optimized.

Set-up A also uses this loss for the _domain classifier_ , with $y_{i}$ in
equation 4.3 corresponding to the _domain classifier_ output and classes 0 and
1 corresponding to target and source domains respectively. In this case,
perfect separation also results in a loss of 0 but a separation between the
domains is not intended. Instead, the network response should be the same for
both classes of events which is provided as an additional restriction via the
gradient reversal layer. The _domain classifier_ loss is minimized but, under
this restriction, the lowest achievable loss is when the response for both
classes, i.e. source and target, is $y_{i}=0.5$, resulting in a loss of
${-\ln{0.5}\approx 0.693}$. This behavior is visible in Fig. 2b, where the
predicted loss of 0.693 is reached in the first epochs and kept most of the
training. It should be noted that this poses an extra requirement on the
_feature extractor_ , which besides providing domain independent features, is
also optimised to provide features for which the _domain classifier_ output
are exactly 0.5 for all events.

We found that deviations in the output of the _domain classifier_ from the
optimal value of $y_{i}=0.5$ had severe influences on the classification in
general. Analyzing at a lower level we found that these changes were driven by
huge gradients back-propagated from the domain classifier loss, further
amplified by $\lambda$ as $\lambda>1$ was used. To avoid the change in the
gradients under $y_{i}$ deviations we tested a set-up where the derivatives of
the _domain classifier_ loss were independent of the $y_{i}$ values. To
achieve this behaviour, we removed the activation function from the _domain
classifier_ output and changed the loss to a linear function, computed for a
single event ($E_{i}$) as:

$L_{i}=\begin{cases}-y_{i}&\text{if }E_{i}\in\text{source}\\\ \
\;\,y_{i}&\text{if }E_{i}\in\text{target}\end{cases}$ (4.4)

This new set-up has also the advantage that $y_{i}$ is not limited to 0.5 in
the optimized case, since now, if the condition of no domain separation is
met, this loss has a value of 0 for any value of the _domain classifier_
output so the _feature extractor_ has more freedom during the optimization.

###  4.4 Training of the neural network

The ADAM optimizer [22], being commonly used nowadays, was used as starting
point. ADAM is an extension of RMSProp with SGD Momentum i.e. adding momentum
terms defined as decaying average of the past gradients. The momentum terms
should help to faster escape from highly sub-optimal loss regions. However,
when we used the default values of the momentum term ($\mu=0.9$) we noticed
severe oscillations of the label predictor loss, as shown in Fig. 2a. These
oscillations seem to be caused by fluctuations in the domain classifier loss
part on which the label predictor has then to react in the common effort to
minimize the global loss. We switched to RMSProp, which does not use a
momentum term, resulting in a more stable loss course during the training. We
therefore did not attempt to further use ADAM.

(a)

(b)

Figure 2: Examples of fluctuations observed during training of the network
using set-up A. (a) Training and validation loss for the label predictor using
the ADAM and RMSprop optimizers. (b) Big spikes of the label predictor and
domain classifier losses (case with $\lambda=20$).

Beside those small fluctuation described above, infrequent large spikes where
found. One of them is shown in Fig. 2b, where $L_{y}$ minimizes smoothly for
over 300 epochs but suddenly $L_{y}$ raise to huge values together with
$L_{d}$. Running 3000 independent trainings we found that these spikes appear
in around a 0.7% of the cases for set-up A and 1.6% for set-up B.

Performing analyses we found that changing the weights of the network to the
ones used 10 epochs before makes the spikes vanish. This indicates that the
cause of the spikes involves initial random fluctuation related to the
adversarial training with the gradient reversal layer. Backing up this
assumption, we also found that the frequency of these spikes increases by
increasing the value of $\lambda$.

Furthermore, comparing set-ups A and B, we found in A stronger dependence of
the learning curves on the randomization (initial weights, data shuffling,
etc.), which is demonstrated in Fig. 3. The learning curves for set-up B agree
better indicating a more stable training. They also converge faster. The
stopping criterion is reached in set-up A after around 600 epochs but after
about 400 epochs in set-up B.

(a)  (b)

Figure 3: _Label predictor_ loss in 20 different trainings starting from
different random seeds for (a) set-up A with $\lambda=20$ and (b) set-up B
with $\lambda=10^{-5}$. For better visibility, the y-axis is limited to 0.66
cutting off high fluctuations. Note that set-up B reaches better _label
predictor_ loss.

###  4.5 Tuning impact of the adversarial domain classification

The parameter $\lambda$ controls the influence of the label predictor and
domain classifier responses on the total loss. It determines how much the
responses to source and target data input produced by feature extractor should
agree. High $\lambda$ values forces a strong agreement but may impair the
ability of the feature extractor to provide useful features for the
classification, low values give more freedom for the feature representation
density distribution but might not be enough for obtaining a good agreement
between the domains and thus introduce a bias for source domain samples. To
give an example, Figure 4 shows the discriminant output for the set-up A for
values of $\lambda$ between 0 and 20. A large difference between source and
target domain feature extractor response density can be observed for
$\lambda=0$, while with increasing values of $\lambda$ the influence of the
domain classifier on the density alignment and consequently also on label
prediction increases and finally a very strong agreement between feature
extractor responses to both background samples is reached at the highest value
of $\lambda$, while label predictor performance deteriorates. The optimal
lambda value is specific to the problem and the performance measure applied as
will be discussed in the following.

(a)  (b)  (c)  (d)

Figure 4: Label predictor response to signal (blue) and background (orange)
for different values of $\lambda$. The label predictor is trained on the
source domain and applied to a statistically independent part of the source
domain (lines) and the target domain (area). Each of the distribution is
normalized to 1. (a) $\lambda=0$ (b) $\lambda=0.58$ (c) $\lambda=1.5$ ( d)
$\lambda=20$ . Discussion see text.

(a)  (b)

Figure 5: Performance measured as the area under the ROC curve (AUC) for
several values of $\lambda$. The difference between the response for source
and target is measured as the Kolmogorov-Smirnov distance. (a) set-up A (b)
set-up B. Each point represent the average over $\sim 200$ independent
training processes (with different random numbers). The error bars represent
the 15.8 and 84.2 percentiles, corresponding to $\pm 1\sigma$ in a normal
distributed variable.

##  5 Results

The performance of the network depends on the relative importance of the
adversarial branch containing domain classifier steered by the parameter
$\lambda$. As for any hyper-parameter, the values of $\lambda$ are specific to
the network architecture and data sets used and need to be determined for each
particular use case. We consider three measures of performance, demonstrating
the bias without the adversarial treatment and their improvement when the
adversarial branch is included.

First we report AUC which is a common performance measure for binary
classifiers. Since a good value for $\lambda$ was still not selected we made a
scan over a range of possible values (Fig. 5). We extend it with the
Kolmogorov-Smirnov distance as a measure of agreement between the response of
the two domains. This distance is given by the maximum absolute difference
between the cumulative distributions of the normalized _label predictor_
response for the two domains. The best choice of $\lambda$ is the value for
which the maximum source domain AUC is achieved among those with the lowest
Kolmogorov-Smirnov distance. This criterion for the optimal $\lambda$ has the
advantage that it can be computed without using the target labels, i.e. using
labeled source and unlabeled target data. To demonstrate that the criterion
for $\lambda$ selection leads to desired behavior on target data, we compute
the AUC for the target domain, as in our study target labels were provided by
the simulation. As depicted in Fig. 5, the closest match between source and
target domain and highest AUC performance is achieved when using lambda values
obtained from the criterion procedure.

With $\lambda=0$, corresponding to absence of adversarial network, an AUC on
the target domain of 0.657 is achieved. This value is improved to 0.756 using
$\lambda=20$ for set-up A, and 0.760 using $\lambda=10^{-5}$ for set-up B.
This improvements have the cost of reducing the AUC obtained for the source
domain from 0.776 in the no adversarial case, to 0.757 and 0.760 for set-ups A
and B respectively with the selected $\lambda$ values. Increasing $\lambda$
above those values only decreases the performance, but in the case of set-up B
a plateau exist such that taking $\lambda$ values up to 100 times the selected
one keeps the same performance.

(a)  (b)

Figure 6: Approximate median significance (in units of standard deviations) as
a function of $\lambda$, computed for 50000 events consisting of 5% signal and
95% background. (a) set-up A (b) set-up B. Each point represent the average
over $\sim 200$ independent trainings and the error bars represent the 15.8
and 84.2 percentiles.

To further approximate the significance as reported in Higgs discovery
searches as performance measure, we use the approximate median significance
(AMS) as proposed in [23]. This definition corresponds to a test of the signal
discovery versus background only hypothesis by taking systematic uncertainties
into account. It is calculated as:

AMS
$\displaystyle=\sqrt{\sum_{i}\left\\{2\left[(s_{i}+b_{i})\ln{s_{i}+b_{i}\over
b_{0i}}-s_{i}-b_{i}+b_{0i}\right]+{(b_{i}-b_{0i})^{2}\over\sigma_{bi}^{2}}\right\\}}$
(5.1a) $\displaystyle b_{0i}$ $\displaystyle={1\over
2}\left(b_{i}-\sigma_{bi}^{2}+\sqrt{(b_{i}-\sigma_{bi}^{2})^{2}+4(s_{i}+b_{i})\sigma_{bi}^{2}}\right)$
(5.1b)

where the sum is over the bins in the histogram of the response, $s_{i}$ and
$b_{i}$ represents the signal and background counts in the bin $i$ for the
source domain and $\sigma_{b_{i}}^{2}={1\over 2}(b_{i}-b_{i}^{\
alt.})^{2}+(0.1\ b_{i})^{2}$ is an estimator of the variance on the background
counts. The variance is computed from the difference between $b_{i}$ and the
background count for the target domain in the same bin ($b_{i}^{\ alt.}$) plus
a flat 10% uncertainty on the background, approximating the values of the
reference analysis. The AMS is a valid simplification of the significance in
the context of this paper as long as we consider only the qualitative
behavior, not the absolute values. The AMS as a function of $\lambda$ is shown
in Fig. 6. A low significance is observed in the case when the response for
both domains disagree. The significance increases with $\lambda$ until
reaching a maximum at similar positions of the maximal AUC where source and
target values agree (Fig. 5). For higher values of $\lambda$ the significance
decreases, reflecting the loss of classification power.

(a)  (b)

Figure 7: Signal purity $p_{sig}$ versus signal efficiency $\epsilon_{sig}$ of
the label predictor, calculated for the source (solid, upper line) and target
(dashed, lower line) sample, with (blue) and without (orange) the adversarial
architecture. (a) set-up A (b) set-up B.

Using the optimized $\lambda$ setting we measure the performance in terms of
signal purity, which is related to the sensitivity of the measurement. It is
defined as the ratio between number of signal events ($s$) and the total of
events ($s+b$) that fall above a specific cut in the _label predictor_
response: ${p_{sig}=\frac{s}{s+b}}$. Each possible cut corresponds to a signal
efficiency ($\epsilon_{sig}=\frac{s}{N_{s}}$), which is defined as the
fraction of signal selected ($s$) from the total number of signal events
($N_{s}$). Figure 7 shows the whole profile of the signal purity as a function
of the signal efficiency. The expected signal to background composition of
5:95 is taken into account. Classification without the adversarial part
reaches around a $9\%$ higher purity on the source domain, but very low values
for the target domain. The adversarial network yields very close values for
both source and target domains.

To give a numerical example taking the signal purity as an approximation of
the analysis sensitivity, we take the results for the source domain as the
central value and the difference between the two domains as a 1$\sigma$
uncertainty. For $\epsilon_{sig}=0.5$ we get:

  * •

no adversarial network: $p_{sig}=0.148\pm 0.069$

  * •

adversarial set-up A: $p_{sig}$= $0.137\pm 0.005$

  * •

adversarial set-up B: $p_{sig}$= $0.1369\pm 0.0004$

The relative uncertainty due to the choice of the background model on the
signal purity, ignoring other sources of uncertainty, can be improved from 47%
to 4% (0.3%) by employing the adversarial network in set-up A (set-up B).

###  5.1 Extension of the method towards training with real collision data

In this study, no labels were used for computing loss of the domain classifier
(except its alignment for signal and background ratio that was so far taken to
be the same as for the source domain). One natural extension of the method
would be to use real collision data to train domain classifier for adaptation
to real data domain. However, in real collision data the ratio of signal to
background is only known with limited precision obtained from previous
measurements or theoretical predictions. For the results presented so far, the
signal to background ratio was set to the predicted value of 5:95 in the
target domain, while scaling the source to the same ratio in the _domain
classifier_. To check the stability of our results, the dependence of the
_label predictor_ output on the chosen signal to background ratio was tested.
It was found that a change in its value had no impact, as long as it is the
same in both source and target (Fig. 8a).

However, if there is a discrepancy in the signal to background fraction
between the two domains, a small bias is introduced. This behavior is shown in
Fig. 8b, where a fixed value of 5% was used for the source signal fraction,
while varying signal fraction in the target domain. By varying the signal-to-
background ratio by a factor of two away from the ratio in the source domain,
a 1.4% bias was introduced on AUC which is however, still small compared to
case without adversarial training. It becomes therefore important to get a
good estimate for the signal to background ratio in the target domain when
using unlabeled data and to properly account for the effect of this bias on
the final result.

(a)  (b)

Figure 8: AUC ROC for the label predictor (a) as a function of the signal
fraction in source and target and (b) as a function of the varying signal
fraction in target with a fixed 5% signal fraction in the source. These plots
were produced for set-up B with $\lambda=10^{-5}$. Set-up A exhibits a similar
behavior.

We hypothesized that the gap we observe between performance for classifying
events in the source or in the target domain when signal and background ratio
do not match across domains may be caused by the shift of label distribution.
Following number of works attempting to address this issue in unsupervised
setting [7], we applied one such approach to see whether gap issue can be
tackled. The implicit alignment approach [25] points out that, among other
issues, the differences in label distribution may provide a harmful shortcut
to identify the respective domain just on the basis of differences in label
frequencies. This may strongly impair domain adaptation by ignoring actual
differences in data distributions and thus not handling properly data
distribution shift. To circumvent that, authors propose creating re-balanced
mini-batches for training domain classifier using pseudo-labels delivered by
the label predictor for target inputs, arguing for removing label frequency
differences between domains in this way. We saw however that generated pseudo
labels have very low reliability, which in turn seems to strongly impair the
composition of re-balanced mini batches and do not result in reduction of the
classification gap between domains in our case - on the contrary, the gap
falls back to the state observed without any domain adaptation. This is not
suitable for use case of real collision data adaptation in unsupervised
setting, and makes the method in the current form rely on faithful estimate of
signal to background ratio in the real collision data as pointed out above.

##  6 Conclusion

We successfully built a feed-forward fully connected adversarial neural
network for performing domain adaptation on high energy physics data to enable
event classification in the search for the $t\bar{t}H(H\rightarrow b\bar{b})$
process at the LHC. We demonstrate that adding a _domain classifier_ sub-
network with a gradient reversal layer helps removing training bias while
retaining most of the nominal classification power. We analyzed the dependence
on the hyper-parameters of the network. We studied the training stability
issues that appear due to the addition of a gradient reversal layer. We
demonstrated that by using linear activation and loss functions, stability and
convergence can be significantly improved and better performance of the
network can be achieved.

For the example use case of the ttH(bb) analysis, we demonstrate that the
adversarial domain adaptation can produce a label predictor that is almost
completely independent of the domain background model while preserving most of
the classification power for target domain. We report the improvements using
different measures. Taking the expected signal purity for a signal efficiency
of 50% as a proxy measure for the sensitivity of the analysis, the uncertainty
due to the choice of background model can be strongly reduced from a 47% to a
0.3% with the Monte Carlo samples used in this study. Significant improvements
are also reached in the approximated median significance. Although not
demonstrated, we do not expect limitations when extending this approach to
adapt to multiple alternative domains, i.e. sources of uncertainty, during
training.

Application of our approach to target samples from real collision data was
discussed where no explicit label information from target is required for
training of the _domain classifier_. For the selection of optimal value for
hyperparameter $\lambda$ that controls the impact of adversarial domain
classifier on label predictor, we designed a procedure that does not require
labeled target data. However, while per input event example labels from the
real target are not necessary for training procedure, we show that in absence
of a faithful estimate of the signal to background ratio for the real data
target domain, misalignment of the signal to background ratio between source
and target domains may lead to a small bias in the classification. This small
bias and its impact has then to be addressed in a further downstream analysis.

Using a different ratio of signal to background in source and target domains
introduces label distribution shift to the original formulation of the
problem, in addition to the already existing data distribution shift given by
the different background models in the two domains. Handling both data and
label distribution shift for domain adaptation is still a largely unresolved
problem in machine learning. For the unsupervised domain adaptation setting we
have worked with here, our observations with a recently introduced implicit
alignment approach [25] that makes use of pseudo-labels suggests that quality
of pseudo labels required for such an approach to cope with both data and
label distribution shift is not sufficient for our case. Application of our
method to real experimental collision data adaptation in unsupervised setting
in its current form will have to therefore rely on fair estimates of signal to
background ratio in the real data.

As discussed above, we see the differences in label frequency between the
domains which provides a shortcut for domain identification [25] and harms
domain adaptation, as one central issue hampering successful domain adaptation
given unknown, different signal background ratios in our case. One potential
solution that we envisage for the follow-up work may employ a network
architecture that uses yet another adversarial branch dealing explicitly with
the task to erase harmful signal-background label information from domain
classifier. Given the current progress, we anticipate that this and other
advanced approaches [26] will render our method also capable of handling label
shift as well and enable successful adaptation to real collision data in fully
unsupervised manner.

## Acknowledgments

We acknowledge that Ilyas Fatkhullin contributed at an early stage of the
analysis [24].

Data availability
The data that support the findings of this study are available upon request
from the authors.

## References

  * [1] M. Aaboud et al. [ATLAS Collaboration], Search for the standard model Higgs boson produced in association with top quarks and decaying into a $b\bar{b}$ pair in $pp$ collisions at $\sqrt{s}$ = 13 TeV with the ATLAS detector, Phys. Rev. D 97 (2018) no.7, 072016 [arXiv:1712.08895[hep-ex]].
  * [2] Y. Ganin et al., Domain-Adversarial Training of Neural Networks, Journal of Machine Learning Research 2016, vol. 17, p. 1-35 [arXiv:1505.07818 [stat.ML]] (2015).
  * [3] Ben-David, Shai and John Blitzer and Crammer, Koby and Fernando Pereira, Analysis of Representations for Domain Adaptation, Advances in Neural Information Processing Systems 19, p137–144 (2007).
  * [4] Englert, Christoph and Galler, Peter and Harris, Philip and Spannowsky, Michael, Machine learning uncertainties with adversarial neural networks Eur. Phys. J. C79,4 (2019).
  * [5] C. Shimmin et al., Decorrelated Jet Substructure Tagging using Adversarial Neural Networks, Phys. Rev. D 96, 074034 (2017), [arxiv:1703.03507 [hep-ex]].
  * [6] A. M. Sirunyan et al. [CMS], “A deep neural network to search for new long-lived particles decaying to jets,” Mach. Learn. Sci. Tech. 1 (2020), 035012 [arXiv:1912.12238 [hep-ex]].
  * [7] Conditional Adversarial Domain Adaptation Long, Mingsheng and CAO, ZHANGJIE and Wang, Jianmin and Jordan, Michael I in "Advances in Neural Information Processing Systems 31" eds. S. Bengio and H. Wallach and H. Larochelle and K. Grauman and N. Cesa-Bianchi and R. Garnett. by Curran Associates, Inc/ p1640ff (2018) [http://papers.nips.cc/paper/7436-conditional-adversarial-domain-adaptation.pdf]
  * [8] G. Louppe, M. Kagan and K. Cranmer, Learning to Pivot with Adversarial Networks, Advances in Neural Information Processing Systems 30, p981 (2017), [arxiv:1611.01046 [stat.ML]] (2016).
  * [9] P. Glaysher, J. M. Katzy and S. An, Iterative subtraction method for Feature Ranking, [arXiv:1906.05718 [physics.data-an] ](2019).
  * [10] S. V. Chekanov [HepSim Group], Advances in High Energy Physics, vol. 2015, Article ID 136093, 7, [arxiv:1403.1886 [hep-ph]].
  * [11] J. Alwall, M. Herquet, F. Maltoni, O. Mattelaer and T. Stelzer, JHEP 1106 (2011) 128, doi:10.1007/JHEP06(2011)128, [arXiv:1106.0522 [hep-ph]].
  * [12] G. Corcella, I. Knowles, G. Marchesini, S. Moretti, K. Odagiri, P. Richardson, M. Seymour, B. Webber JHEP 0101:010 (2001), [arXiv:hep-ph/0011363].
  * [13] T. Sjostrand, S. Mrenna and P. Z. Skands, JHEP 0605 (2006) 026 [hep-ph/0603175].
  * [14] T. Jezo, J. Lindert, N. Moretti, S. Pozzorini, New NLOPS predictions for $t\bar{t}+b$-jet production at the LHC, Eur. Phys, J. C78-6-502 (2018), [arXiv:1802.00426 [hep-ph]].
  * [15] J. de Favereau et al. [DELPHES 3 Collaboration], DELPHES 3, A modular framework for fast simulation of a generic collider experiment, JHEP 1402 (2014) 057, [arXiv:1307.6346 [hep-ex]].
  * [16] M. Cacciari, G. P. Salam and G. Soyez, JHEP 04 (2008), 063 [arXiv:0802.1189 [hep-ph]].
  * [17] M. Aaboud et al. [ATLAS Collaboration], Measurements of b-jet tagging efficiency with the ATLAS detector using $t\overline{t}$ events at $\sqrt{s}=13$ TeV, JHEP 1808 (2018) 089 [arXiv:1805.01845 [hep-ex]].
  * [18] F. Chollet et al., Keras, https://github.com/fchollet/keras (2015).
  * [19] M. Abadi et.al., TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems, [https://www.tensorflow.org/]
  * [20] X. Glorot, Y. Bengio, Understanding the difficulty of training deep feedforward neural networks, Proc. of 13th Intern.Conf.on Artificial Intelligence and Stat.(AISTATS) Vol.9 of JMLR (2010).
  * [21] J. Bergstra, D. Yamis, D.D. Cox, Making a Science of Model Search, In Proc. of 30th Int. Conf. on Machine Learning (ICML 2013) Volume 28 p.I-115 (2013).
  * [22] Diederik P. Kingma, Jimmy Ba, Adam a method for stochastic optimisation
  * [23] C.Adam-Bourdarios, G.Cowan, C. Germain, I.Guuon, B. Kegl, D.Rousseau, JMLR: Workshop and Conference Proceedings: 42:19-55 (2015)
  * [24] I. Fatkhullin, DESY Summer Student Program’s report (2019) [https://www.desy.de/f/students/2019/reports/Ilyas.Fatkhullin.pdf]
  * [25] X. Jiang, Q. Lao, S. Matwin, M. Havaei, Implicit class-conditioned domain alignment for unsupervised domain adaptation International Conference on Machine Learning (PMLR), 2020, 4816-4827
  * [26] V. Prabhu, S. Khare, D. Kartik, J. Hoffman, SENTRY: Selective Entropy Optimization via Committee Consistency for Unsupervised Domain Adaptation, [arXiv:2012.11460 [cs.CV]], 2020


# Multiple View Generation and Classification of Mid-wave Infrared Images
using Deep Learning

###### Abstract

We propose a novel study of generating unseen arbitrary viewpoints for
infrared imagery in the non-linear feature subspace . Current methods use
synthetic images and often result in blurry and distorted outputs. Our
approach on the contrary understands the semantic information in natural
images and encapsulates it such that our predicted unseen views possess good
3D representations. We further explore the non-linear feature subspace and
conclude that our network does not operate in the Euclidean subspace but
rather in the Riemannian subspace. It does not learn the geometric
transformation for predicting the position of the pixel in the new image but
rather learns the manifold. To this end, we use t-SNE visualisations to
conduct a detailed analysis of our network and perform classification of
generated images as a low-shot learning task.

Index Terms—  Autoencoder, view prediction, infrared imagery, manifold
learning, classification

##  1 Introduction

View synthesis has been a long-standing problem with works such as [1, 2]
notable in giving good qualitative solutions. However,majority of these works
[3, 4] involve the use synthetic data which is unlimited in quantity and can
be created with desired azimuth, elevation, degree variation and background
setting. ShapeNET [5] dataset which consists of over 8000 synthetic car and
chair models and SURREAL dataset [6] which contains synthetic human motion
sequences are two such widely used datasets. We make the problem challenging
by dealing with natural infrared images of MWIR (medium wave) frequency in the
’DSIAC-ATR Image database’ . These images are inherently complex and carry
intrinsic details such as background , clutter , moving targets, day and night
variations which are difficult to comprehend and predict.Collection of such
data is also a big hassle due to which we show that our generated images can
also be used in the training corpus for various image processing tasks
including classification. Since infrared data is extremely scarce, our study
assists in creating data as close to real world natural images. Most of the
related view generation studies have been categorized in two categories.
Geometry based where the weights of the neural network learn the geometric
transformation of a pixel between two views, the input and output view.
Second, where view prediction is seen as a learning problem and the network is
trained to understand the 3D representation of objects. Few works have
exploited the use of both approaches [1] to come up with hybrid methods of
using geometry as well as learning for view morphing. We however adopt the
learning based approach and assert via our results that our approach is not
just learning the 3D representation of objects but rather its manifold. We
believe the pixels in the input and output view are highly correlated and if
observed on a manifold lie at the same point. We make use of this assumption
and do novel view synthesis as a manifold learning task and not as a geometry
based task. We do not propose to explicitly learn the 3D information of our
model but rather we want to exploit the latent space embedding of autoencoders
to learn the underlying semantic information and pixel relation between input
and novel output views. We show via experiments that our network learns the
underlying feature subspace in the non-linear domain. Our approach is entirely
unsupervised and we can say that while learning the manifold, our network also
learns to classify objects and becomes a classification model as suggested in
[7]. We are able to generate novel views both during day and night, which is
significant in infrared imagery. We make following contributions:

  * •

We present a novel study of generating unseen views for infrared imagery.

  * •

We propose a method that creates realistic looking training images to assist
in low-shot learning and classification.

  * •

We perform experiments that show view generation is a manifold learning task
that exploits latent space embeddings.

Fig. 1: Our proposed network framework and training technique.We compute loss
between our embeddings and final generated view using mean square error.

##  2 Proposed Approach

Fig. 2: Our implemented network architecture. For both our vanilla and
predictor autoencoder, we use 4 convolutional and 4 deconvolutional layers. We
use input image size of 64 x 64.

###  2.1 Deep Convolutional Neural Network based Autoencoder framework

Feature extraction and feature selection(a NP hard problem) when done manually
pose a lot of problems. Are the features good or variable enough to assist in
network learning are questions that need addressing. Autoencoders have emerged
as an alternative to feature extraction and selection in the non-linear
feature subspace [8]. We propose a deep convolutional neural network based
autoencoder with emphasis on exploiting its latent space and learning the
manifold. Our proposed method is illustrated in Figure 1.Block 1 denotes a
vanilla autoencoder [9] which acts as a guide and paves way to formulate the
first term of our loss function. This block is trained separately and is used
to supply the ground truth embeddings for the desired viewpoints.Block 2
denotes the predictor network which is an exact replica of the first block
except that it receives an input view and produces an unseen output view. We
adapt the network proposed in [2] to create the autoencoder blocks. The
autoencoders have 4 convolutional blocks in encoder and decoder with
consecutive layers having 32, 32,64 and 64 filters respectively.The filter
size is $5\times 5$ for the first layer and then $3\times 3$ for the following
layers.We use an image size of $64\times 64$ for our network. We tried using
image size of $128\times 128$ as well but obtained better results with the
former. The encoder’s output is $4\times 4\times 64$ dimensional which we
flatten to create a 1024 long embedding representing the input view. We
introduce feature fusion in the network and pass a 5 dimensional long vector
through a 64 dimensional fully connected layer to be processed
independently.In an unsupervised setting, the 1024 embedding of our input view
which is of varying orientation, azimuth and time of day is concatenated with
our independent feature vector. This latent embedding which is now 1088
dimensional long is jointly processed using two further fully connected layers
as shown in the detailed architecture in Figure 2. We use leaky relu
activation.For the last layer, we use hyperbolic tangent since it produces
steeper slope and stronger gradients as stated by LeCun et al. [8, 10].

###  2.2 Objective function

Our objective function is a sum of two separate loss terms. The first term
aims to guide the latent space embedding to a set of values which best
describe our desired viewpoint. This also means, we must have the ground truth
for the embeddings. Section 2.1 describes how we obtain these ground truth
embeddings. Once we obtain them, we compute our first term in the loss
function $L_{e}$ .

$L_{e}=\sum\lVert e_{2}-e_{1}\rVert^{2}$ (1)

Here $e_{2}$ is the ideal embedding which we generate using our vanilla
autoencoder and $e_{1}$ is the embedding which we obtain while training our
predictor network. This is depicted in Figure 1.We use mean square error to
calculate the loss. The second term in our objective function aims to reduce
the loss between our predicted output view and its ground truth. It is given
by :

$L_{o}=\sum\lVert y_{2}-y_{1}\rVert^{2}$ (2)

where $y_{2}$ and $y_{1}$ are the ground truth and generated views
respectively. We use $L_{e}$ and $L_{o}$ to compute our final reconstruction
loss $L_{t}$ which is a sum of these two error terms as observed below.

$L_{t}=L_{e}+L_{o}$ (3) $L_{t}=\sum\lVert e_{2}-e_{1}\rVert^{2}+\sum\lVert
y_{2}-y_{1}\rVert^{2}$ (4)

Fig. 3: Qualitative results of our method. The left side of the figure shows
the single input image which is used as View 1 for our network. Top row of the
images on the right are the predicted images generated using the single input
image, the bottom row shows their corresponding ground truth views. Results
for other classes have been shared in supplementary material.

##  3 Experiments

###  3.1 Dataset

We work on ‘DSIAC-ATR Image Dataset’. The Automated Target Recognition (ATR)
Algorithm Development Image database [11] has been collected by US Army Night
Vision and Electronic Sensors Directorate (NVESD) with the intent of
supporting work in the infrared and ATR domain. This database contains MWIR
(mid-wave infrared) images of military and civilian vehicles with 10 non-human
categories. Each category has videos collected during the day and night at
different ranges (distance from the camera capturing the target). Each video
shows the object moving in two 360 degree circles. The ranges lie between
900-5000 m. This means that all of the videos have been captured from a very
far distance which makes our task of view prediction extremely difficult and
painstaking.Most of the space in the images is taken up by background and
clutter with the object of interest appearing very small and distorted. The 10
object categories consist of SUVs, pickup trucks, battle tanks and anti-
aircraft missile launchers.Since these are infrared images; all vehicles are
grayscale and show uneven heat distribution. Our work as a result is very
different from other view prediction research studies [2, 4, 1, 6, 3, 12]
which use evenly distributed colored objects, have unlimited training samples
and also have the ability to create their own azimuth, elevation and range
parameters. We on the contrary are dealing with annotations supplied with the
dataset. Hence parameters like range, azimuth, elevation , light intensity are
all beyond our control.

Model | Input | Conv layers | Average
---|---|---|---
| dimensions | encoder,decoder | error
MV3D[2] | $128\times 128$ | 5,5 | 0.0056
Ours + | $64\times 64$ | 4,4 | 0.0011
MSE loss |  |  |
Ours + | $64\times 64$ | 4,4 | 0.0009
New loss |  |  |
Table 1:  A quantitative comparison - average error with different
methods,input dimensions and objective functions.

###  3.2 Training and Implementation details

We first convert the videos from arf format to avi format which are then
converted to $640\times 580$ dimensional images [13]. These images are used to
obtain the objects of interest which are cropped to a size of $64\times 64$
removing background as much as possible. The dataset has 5 ranges, we work
with objects that are at a range of 1000 m or less so that the object is
visible. This threshold gives us approximately 180 images per object of
different azimuth. We create our dataset of views which are all 5 degrees
apart and contain images both from day and night time.These are used to create
training pairs with varying input and output viewpoints.We end with 30k
training pairs and 10k test pairs.We use Adam optimizer [14] and adaptive
learning rate methods. We use learning rate of 1e-3 initially and decrease it
to 1e-4 for the last few epochs. We reach convergence after 80 epochs.We
extract ground truth embeddings from block 1. These embeddings are then fed
with the ground truth images for view 2 in block 2 of our proposed method. Our
predictor network uses our new loss function as defined in section 2.2 to
compute the final reconstruction error given in Table 1. We perform multiple
experiments; baseline based on [2, 4, 12, 1], our new architecture with just
simple mean square error loss and then with our new objective function. We use
a batch size of 64 for all our experiments and keras with tensorflow backend
for implementation. Figure 3 illustrates our obtained qualitative results.

##  4 Evaluation

###  4.1 Classification and Low-shot learning

Studies have shown that it is difficult to judge the generated images both
qualitatively and quantitatively . Visual analysis depends on how the reader
observes and assess the image. A good evaluation metric in terms of quality is
their classification. This has given rise to the concepts of low-shot [15],
zero-shot and one-shot learning.These techniques are becoming popular because
it is difficult to collect and annotate real world datasets.We perform such a
classification experiment using 8 classes and replace class 2- BRDM2’s
training images with only our network generated images. We have already shown
in Figure 3 that our network is capable of generating multiple novel views
using a single image. We use a pretrained VGG-16 [16] network and place our
8-class classifier on top. We test using only real-world images for all
classes even BRDM2. We obtain 68% classification accuracy with very little
mispredictions for class 2- BRDM2 shown in Figure 4 as compared to a 79%
accuracy when we use real images for training affirming that our generated
images are of good quality and can be used for experimental research.

Fig. 4: The confusion matrix shows correct predictions for Class 2 ’BRDM2’-an
infantry vehicle when we use synthesized images for training.

###  4.2 Manifold learning

Manifold representations are not only convenient for visualizing high
dimensional data, but also provide insights about how different classes inter-
relate. A lower dimensional embedding of the manifold ensures that the
geodesic relations between the data points is preserved. The problem of
predicting new views from a few original views may be viewed as learning the
behavior of the manifold, and the transition from one view to another as a
movement along its surface. Thus, we interpret our results in terms of how the
original and predicted views behave in the latent space. We verify that the
predicted views and the ground truth images are “close” on the underlying
manifold. The t-SNE[17] method models each high-dimensional object by a two-or
three-dimensional point in such a way that similar objects are modeled by
nearby points and dissimilar objects are modeled by distant points with high
probability. We visualize two embeddings from our trained network, one
obtained from the output of the encoder, second obtained after we inject our
orientation parameters and process the concatenated features. Figure 5 (a)
shows our embeddings before feature fusion. We can see different object
classes, for each object class, we observe two clusters, one showing the night
and the other, the day view. The data points however seem scattered and do not
show uniformity. Figure 5 (b) shows the results for our embeddings after
feature fusion and what the network will predict given the unseen viewpoints.
It clearly depicts uniformity and precise clustering which affirms that the
network learnt the manifold of the objects.

Fig. 5: t-SNE visualizations for the two embeddings.Object classes and their
full names have been shared in the supplementary material.

##  5 Conclusion

We have performed a novel study of predicting unseen views on infrared
imagery. The use of deep learning and a CNN based autoencoder for exploiting
the latent space embeddings and predict novel views has shown promising
results. Our proposed method has provided us an effective modeling of entities
present in our dataset. The method is able to understand the difference in
object shape and class without any reliance on color. We are able to show that
the network learns the manifold and operates in riemannian feature space.
Moreover our experimental evaluations show the quality of our generated images
and the effectiveness of our technique.

Acknowledgement: The authors gratefully recognize the support of Leonardo DRS
for funding parts of this research.

## References

  * [1] Tinghui Zhou, Shubham Tulsiani, Weilun Sun, Jitendra Malik, and Alexei A Efros,  “View synthesis by appearance flow,”  in European conference on computer vision. Springer, 2016, pp. 286–301.
  * [2] Maxim Tatarchenko, Alexey Dosovitskiy, and Thomas Brox,  “Multi-view 3d models from single images with a convolutional network,”  in European Conference on Computer Vision. Springer, 2016, pp. 322–337.
  * [3] Dinghuang Ji, Junghyun Kwon, Max McFarland, and Silvio Savarese,  “Deep view morphing,”  in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 2155–2163.
  * [4] Alexey Dosovitskiy, Jost Tobias Springenberg, and Thomas Brox,  “Learning to generate chairs with convolutional neural networks,”  in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 1538–1546.
  * [5] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al.,  “Shapenet: An information-rich 3d model repository,”  arXiv preprint arXiv:1512.03012, 2015.
  * [6] Shi Jin, Ruiynag Liu, Yu Ji, Jinwei Ye, and Jingyi Yu,  “Learning to dodge a bullet: Concyclic view morphing via deep learning,”  in Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 218–233.
  * [7] Maxim Tatarchenko, Stephan R Richter, René Ranftl, Zhuwen Li, Vladlen Koltun, and Thomas Brox,  “What do single-view 3d reconstruction networks learn?,”  in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 3405–3414.
  * [8] David Charte, Francisco Charte, Salvador García, María J del Jesus, and Francisco Herrera,  “A practical tutorial on autoencoders for nonlinear feature fusion: Taxonomy, models, software and guidelines,”  Information Fusion, vol. 44, pp. 78–96, 2018.
  * [9] David Berthelot, Colin Raffel, Aurko Roy, and Ian Goodfellow,  “Understanding and improving interpolation in autoencoders via an adversarial regularizer,”  arXiv preprint arXiv:1807.07543, 2018.
  * [10] Genevieve B Orr and Klaus-Robert Müller,  Neural networks: tricks of the trade,  Springer, 2003.
  * [11] US Army Night Vis. and Elec. Sensors Directorate (NVESD),  “Atr algorithm development image database,”  Journal of machine learning research, pp. 1–10, 2012.
  * [12] Tejas D Kulkarni, William F Whitney, Pushmeet Kohli, and Josh Tenenbaum,  “Deep convolutional inverse graphics network,”  in Advances in neural information processing systems, 2015, pp. 2539–2547.
  * [13] Maliha Arif and Abhijit Mahalanobis,  “View prediction using manifold learning in non-linear feature subspace,”  in MIPPR 2019: Pattern Recognition and Computer Vision. International Society for Optics and Photonics, 2020, vol. 11430, p. 114301N.
  * [14] Diederik P Kingma and Jimmy Ba,  “Adam: A method for stochastic optimization,”  arXiv preprint arXiv:1412.6980, 2014.
  * [15] Yu-Xiong Wang, Ross Girshick, Martial Hebert, and Bharath Hariharan,  “Low-shot learning from imaginary data,”  in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 7278–7286.
  * [16] Karen Simonyan and Andrew Zisserman,  “Very deep convolutional networks for large-scale image recognition,”  arXiv preprint arXiv:1409.1556, 2014.
  * [17] Laurens van der Maaten and Geoffrey Hinton,  “Visualizing data using t-sne,”  Journal of machine learning research, vol. 9, no. Nov, pp. 2579–2605, 2008.