# Unsupervised Multi-source Domain Adaptation Without Access to Source Data

Sk Miraj Ahmed1, Equal Contribution Dripta S. Raychaudhuri1,∗ Sujoy Paul2,∗,
Work done while SP was a PhD student at UC Riverside. Samet Oymak1 Amit K.
Roy-Chowdhury1
1 University of California  Riverside  2 Google Research
{sahme047@, drayc001@, spaul003<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Unsupervised Domain Adaptation (UDA) aims to learn a predictor model for an
unlabeled domain by transferring knowledge from a separate labeled source
domain. However, most of these conventional UDA approaches make the strong
assumption of having access to the source data during training, which may not
be very practical due to privacy, security and storage concerns. A recent line
of work addressed this problem and proposed an algorithm that transfers
knowledge to the unlabeled target domain from a single source model without
requiring access to the source data. However, for adaptation purposes, if
there are multiple trained source models available to choose from, this method
has to go through adapting each and every model individually, to check for the
best source. Thus, we ask the question: can we find the optimal combination of
source models, with no source data and without target labels, whose
performance is no worse than the single best source? To answer this, we
propose a novel and efficient algorithm which automatically combines the
source models with suitable weights in such a way that it performs at least as
good as the best source model. We provide intuitive theoretical insights to
justify our claim. Furthermore, extensive experiments are conducted on several
benchmark datasets to show the effectiveness of our algorithm, where in most
cases, our method not only reaches best source accuracy but also outperforms
it.

##  1 Introduction

Deep neural networks have achieved proficiency in a multiple array of vision
tasks [11, 25, 18, 35], however, these models have consistently fallen short
in adapting to visual distributional shifts [27]. Human recognition, on the
other hand, is robust to such shifts, such as reading text in a new font or
recognizing objects in unseen environments. Imparting such robustness towards
distributional shifts to deep models is fundamental in applying these models
to practical scenarios.

Unsupervised domain adaptation (UDA) [2, 37] seeks to bridge this performance
gap due to domain shift via adaptation of the model on small amounts of
unsupervised data from the target domain. The majority of current approaches
[7, 14] optimize a two-fold objective: (i) minimize the empirical risk on the
source data, (ii) make the target and source features indistinguishable from
each other. Minimizing distribution divergence between domains by matching the
distribution statistical moments at different orders have also been explored
extensively [42, 33].

Figure 1: Problem setup. Standard unsupervised multi-source domain adaptation
(UDA) utilizes the source data, along with the models trained on the source,
to perform adaptation on a target domain. In contrast, we introduce a setting
which adapts multiple models without requiring access to the source data.

A shortcoming of all the above approaches is the transductive scenario in
which they operate, i.e., the source data is required for adaptation purposes.
In a real-world setting, source data may not be available for a variety of
reasons. Privacy and security are the primary concern, with the data possibly
containing sensitive information. Another crucial reason is storage issues,
i.e., source datasets may contain videos or high-resolution images and it
might not be practical to transfer or store on different platforms.
Consequently, it is imperative to develop unsupervised adaptation approaches
which can adapt the source models to the target domain without access to the
source data.

Recent works [21, 23] attempt this by adapting a single source model to a
target domain without accessing the source data. However, an underlying
assumption of these methods is that the most correlated source model is
provided by an oracle for adaptation purposes. A more challenging and
practical scenario entails adaptation from a bag of source models \- each of
these source domains are correlated to the target by different amounts and
adaptation involves not only incorporating the combined prior knowledge from
multiple models, but simultaneously preventing the possibility of negative
transfer. In this paper, we introduce the problem of unsupervised multi-source
adaptation without access to source data. We develop an algorithm based on the
principles of pseudo-labeling and information maximization and provide
intuitive theoretical insights to show that our framework guarantees
performance better than the best available source and minimize the effect of
negative transfer.

To solve this problem of multiple source model adaptation without accessing
the source data, we deploy Information Maximization (IM) loss [23] on the
weighted combination of target soft labels from all the source models. We also
use the pseudo-label strategy inspired from deep cluster method [4], along
with the IM loss to minimize noisy cluster assignment of the features. The
overall optimization jointly adapts the feature encoders from sources as well
as the corresponding source weights, combining which the target model is
obtained.

Main Contributions. We address the problem of multiple source UDA, with no
access to the source data. Towards solving the problem, we make the following
contributions:
$\bullet$ We propose a novel UDA algorithm which operates without requiring
access to the source data. We term it as Data frEe multi-sourCe unsupervISed
domain adaptatiON (DECISION). Our algorithm automatically identifies the
optimal blend of source models to generate the target model by optimizing a
carefully designed unsupervised loss.
$\bullet$ Under intuitive assumptions, we establish theoretical guarantees on
the performance of the target model which shows that it is consistently at
least as good as deploying the single best source model, thus, minimizing
negative transfer.
$\bullet$ We validate our claim by extensive numerical experiments,
demonstrating the practical benefits of our approach.

##  2 Related works

In this section we present a brief overview of the literature in the area of
unsupervised domain adaptation in both the single and multiple sources
scenario, as well as the closely related setting of hypothesis transfer
learning.
Unsupervised domain adaptation. UDA methods have been used for a variety of
tasks, including image classification [44], semantic segmentation [32] and
object detection [15]. Besides the feature space adaptation methods based on
the paradigms of moment matching [42, 33] and adversarial learning [7, 44],
recent works have explored pixel space adaptation via image translation [14].
All existing UDA methods require access to labeled source data, which may not
be available in many applications.
Hypothesis transfer learning. Similar to our objective, hypothesis transfer
learning (HTL) [40, 34, 1] aims to transfer learnt source hypotheses to a
target domain without access to source data. However, data is assumed to be
labeled in the target domain in contrast to our scenario, limiting its
applicability to real-world settings. Recently, [21, 23] extend the standard
HTL setting to unsupervised target data (U-HTL) by adapting single source
hypotheses via pseudo-labeling. Our paper takes this one step further by
introducing multiple source models, which may or may not be positively
correlated with the target domain.
Multi-source domain adaptation. Multi-source domain adaptation (MSDA) extends
the standard UDA setting by incorporating knowledge from multiple source
models. Latent space transformation methods [50] aim to align the features of
different domains by optimizing a discrepancy measure or an adversarial loss.
Discrepancy based methods seek to align the domains by minimizing measures
such as maximum mean discrepancy [10, 50] and Rényi-divergence [13].
Adversarial methods aim to make features from multiple domains
indistinguishable to a domain discriminator by optimizing GAN loss [47],
$\mathcal{H}-$divergence [49] and Wasserstein distance [46, 22]. Domain
generative methods [36, 24] use some form of domain translation, such as the
CycleGAN [51], to perform adaptation at the pixel level. All these methods
assume access to the source data during adaptation.

##  3 Methodology

Method |  Multiple domains |  No source data |  Source model |  Unlabeled target data
---|---|---|---|---
UDA [14] |  ✗ |  ✗ |  ✓ |  ✓
MSDA [33] |  ✓ |  ✗ |  ✓ |  ✓
HTL [40] |  ✗ |  ✓ |  ✓ |  ✗
U-HTL [23] |  ✗ |  ✓ |  ✓ |  ✓
DECISION(Ours) |  ✓ |  ✓ |  ✓ |  ✓

Table 1: Comparison to different adaptation settings by attributes
demonstrated in the paper. Our proposed setting satisfies all the criteria
desired in a holistic adaptation framework.

Problem setting. We address the problem of jointly adapting multiple models,
trained on a variety of domains, to a new target domain with access to only
samples without annotations from the target. In this work, we will be
considering the adaptation of classification models with $K$ categories and
the input space being $\mathcal{X}$. Formally, let us consider we have a set
of source models $\\{\theta_{S}^{j}\\}_{j=1}^{n}$, where the $j^{th}$ model
$\theta_{S}^{j}:\mathcal{X}\rightarrow\mathbb{R}^{K}$, is a classification
model learned using the source dataset
$\mathcal{D}_{S}^{j}=\\{x_{S_{j}}^{i},y_{S_{j}}^{i}\\}_{i=1}^{N_{j}}$, with
$N_{j}$ data points, where $x_{S_{j}}^{i}$ and $y_{S_{j}}^{i}$ denote the
$i$-th source image and the corresponding label respectively. Now, given a
target unlabeled dataset $\mathcal{D}_{T}=\\{x^{i}_{T}\\}_{i=1}^{N_{T}}$, the
problem is to learn a classification model
$\theta_{T}:\mathcal{X}\rightarrow\mathbb{R}^{K}$, using only the learned
source models, without any access to the source datasets. Note that this is
different from multi-source domain adaptation methods in literature, which
also utilize the source data while learning the target model $\theta_{T}$.

Figure 2: Overall framework of our approach: We freeze the final
classification layers of all the sources and jointly optimize for the source
feature encoders along with it’s corresponding weights to get the target
predictor by combining those.

Overall Framework. We can decompose each of the source models into two
modules: the feature extractor
$\phi_{S}^{i}:\mathcal{X}\rightarrow\mathbb{R}^{d_{i}}$ and the classifier
$\psi^{i}_{S}:\mathbb{R}^{d_{i}}\rightarrow\mathbb{R}^{K}$. Here, $d_{i}$
refers to the feature dimension of the $i$-th model while $K$ refers to the
number of categories. We aim to estimate the target model $\theta_{T}$ by
combining knowledge only from the given source models in a manner that
automatically rejects poor source models, i.e., those which are irrelevant for
the target domain.

At the core of our framework lies a model aggregation scheme [28, 13], wherein
we learn a set of weights $\\{\alpha_{i}\\}_{i=1}^{n}$ corresponding to each
of the source models, such that, $\alpha_{k}\geq 0$ and
$\sum_{k=1}^{n}\alpha_{k}=1$. These weights represent a probability mass
function over the source domains, with a higher value implying higher
transferability from that particular domain, and are used to combine the
source hypotheses accordingly. However, unlike previous works, we jointly
adapt each individual model and simultaneously learn these weights by
utilizing solely the unlabeled target instances. In what follows, we describe
our training strategy used to achieve this in detail.

###  3.1 Weighted Information Maximization

As we do not have access to the labeled source or target data, we propose to
fix the source classifiers,$\\{\psi^{i}_{S}\\}_{i=1}^{n}$, since it contains
the class distribution information of the source domain and adapt solely the
feature maps $\\{\phi_{S}^{i}\\}_{i=1}^{n}$ via the principle of information
maximization [3, 19, 30, 23]. Our motivation behind the adaptation process
stems from the cluster assumption [5] in semi-supervised learning, which
hypothesizes that the discriminative model’s decision boundaries should be
located in regions of the input space which are not densely populated. To
achieve this, we minimize a conditional entropy term (i.e., for a given input
example) [9] as follows:

$\mathcal{L}_{\text{ent}}=-\mathbb{E}_{x_{T}\in\mathcal{D}_{T}}\Big{[}\sum_{j=1}^{K}\delta_{j}(\theta_{T}(x_{T}))\log(\delta_{j}(\theta_{T}(x_{T})))\Big{]}$
(1)

where $\theta_{T}(x_{T})=\sum_{j=1}^{n}\alpha_{j}\theta_{S}^{j}(x_{T})$, and
$\delta(\cdot)$ denotes the softmax operation with
$\delta_{j}(v)=\frac{\exp(v_{j})}{\sum_{i=1}^{K}\exp(v_{i})}$ for
$v\in\mathbb{R}^{K}$. Intuitively, if a source $\theta_{S}^{j}$ has good
transferability on the target and consequently, has smaller value of the
conditional entropy, optimizing the term (1) over
$\left\\{\theta_{S}^{j},\alpha_{j}\right\\}$, will result in higher value of
$\alpha_{j}$ than rest of the weights.

While entropy minimization effectively captures the cluster assumption when
training with partial labels, in an unsupervised setting, it may lead to
degenerate solutions, such as, always predicting a single class in an attempt
to minimize conditional entropy. To control such degenerate solutions, we
incorporate the idea of class diversity: configurations in which class labels
are assigned evenly across the dataset are preferred. A simple way to encode
our preference towards class balance is to maximize the entropy of the
empirical label distribution [3] as follows,

$\mathcal{L}_{\text{div}}=\sum_{j=1}^{K}-\bar{p}_{j}\log\bar{p}_{j}$ (2)

where
$\bar{p}=\mathbb{E}_{x_{T}\in\mathcal{D}_{T}}[\delta(\theta_{T}(x_{T}))]$.
Combining the terms (1) and (2), we arrive at,

$\mathcal{L}_{\text{IM}}=\mathcal{L}_{\text{div}}-\mathcal{L}_{\text{ent}}$
(3)

which is the empirical estimate of the mutual information between the target
data and the labels under the aggregate model $\theta_{T}$. Although
maximizing this loss makes the predictions on the target data more confident
and globally diverse, it may sometime still fail to restrict erroneous label
assignment. Inspired by [23], we propose a pseudo-labeling strategy in an
effort to contain this mislabeling.

###  3.2 Weighted Pseudo-labeling

As a result of domain shift, information maximization may result in some
instances being clubbed with the wrong class cluster. These wrong predictions
get reinforced over the course of training and lead to a phenomenon termed as
confirmation bias [43]. Aiming to contain this effect we adopt a self-
supervised clustering strategy [23] inspired from the DeepCluster technique
[4].

First, we calculate the cluster centroids induced by each source model for the
whole target dataset as follows,

$\mu_{k_{j}}^{(0)}=\frac{\sum_{x_{T}\in\mathcal{D}_{T}}\delta_{k}(\hat{\theta}_{S}^{j}(x_{T}))\hat{\phi}_{S}^{j}(x_{T})}{\sum_{x_{T}\in\mathcal{D}_{T}}\delta_{k}(\hat{\theta}_{S}^{j}(x_{T}))}$
(4)

where the cluster centroid of class $k$ obtained from source $j$ at iteration
$i$ is denoted as $\mu_{k_{j}}^{(i)}$, and
$\hat{\theta}_{S}^{j}=(\psi_{S}^{j}\circ\hat{\phi}_{S}^{j})$ denotes the
source from the previous iteration. These source-specific centroids are
combined in accordance to the current aggregation weights on each source model
as follows,

$\mu_{k}^{(0)}=\sum_{j=1}^{n}\alpha_{j}\mu_{k_{j}}^{(0)}$ (5)

Next, we compute the pseudo-label of each sample by assigning it to its
nearest cluster centroid in the feature space,

$\hat{y}_{T}^{(0)}=\text{arg}\ \underset{k}{\text{min}}\
\|\hat{\theta}_{T}(x_{T})-\mu_{k}^{(0)}\|_{2}^{2}$ (6)

We reiterate this process to get the updated centroids and pseudo-labels as
follows,

$\mu_{k_{j}}^{(1)}=\frac{\sum_{x_{T}\in\mathcal{D}_{T}}\mathbbm{1}\\{\hat{y}_{T}^{(0)}=k\\}\hat{\phi}_{S}^{j}(x_{T})}{\sum_{x_{T}\in\mathcal{D}_{T}}\mathbbm{1}(\hat{y^{t_{0}}}=k)}$
(7) $\mu_{k}^{(1)}=\sum_{j=1}^{n}\alpha_{j}\mu_{k_{j}}^{(1)}$ (8)
$\hat{y}_{T}^{(1)}=\text{arg}\ \underset{k}{\text{min}}\
\|\hat{\theta}_{T}(x_{T})-\mu_{k}^{(1)}\|_{2}^{2}$ (9)

where $\mathbbm{1}(\cdot)$ is an indicator function which gives a value of $1$
when the argument is true. While this alternating process of computing cluster
centroids and pseudo-labeling can be repeated multiple times to get stationary
pseudo-labels, one round is sufficient for all practical purposes. We then
obtain the cross-entropy loss w.r.t. these pseudo-labels as follows:

$\mathcal{L}_{\text{pl}}(Q_{T},\theta_{T})=-\mathbb{E}_{x_{T}\in\mathcal{D}_{T}}\sum_{k=1}^{K}\mathbbm{1}\\{\hat{y}_{T}=k\\}\log\delta_{k}(\theta_{T}(x_{T})).$
(10)

Note that the pseudo-labels are updated regularly after a certain number of
iterations as discussed in Section 5.

###  3.3 Optimization

In summary, given $n$ source hypothesis
$\\{\theta_{S}^{j}\\}_{j=1}^{n}=\\{\psi_{S}^{j}\circ\phi_{S}^{j}\\}_{j=1}^{n}$
and target data $\mathcal{D}_{T}=\\{x^{i}_{T}\\}_{i=1}^{n_{T}}$, we fix the
classifier from each of the sources and optimize over the parameters of
$\\{\phi_{S}^{j}\\}_{j=1}^{n}$ and the aggregation weights
$\\{\alpha_{j}\\}_{j=1}^{n}$. The final objective is given by,

$\begin{split}\mathcal{L}_{tot}=\mathcal{L}_{\text{ent}}-\mathcal{L}_{\text{div}}+\lambda\mathcal{L}_{\text{pl}}\end{split}$
(11)

The above objective is used to solve the following optimization problem, —l—
{ϕ_S^j}_j=1^n,{α_j}_j=1^nL_tot α_j ≥0, ∀j ∈{1, 2, …, n} ∑_j=1^n α_j=1

Once we obtain the optimal set of $\phi_{S}^{j*}$ and $\alpha_{j}^{*}$, the
optimal target hypothesis is computed as
$\theta_{T}=\sum_{j=1}^{n}\alpha_{j}^{*}(\psi_{S}^{j}\circ\phi_{S}^{j*})$. To
solve the optimization (3.3) we follow the steps of Algorithm (1) stated
below.

Input: Trained source models
$\\{\theta_{S}^{j}\\}_{j=1}^{n}=\\{\psi_{S}^{j}\circ\phi_{S}^{j}\\}_{j=1}^{n}$,
unlabeled target data $\\{x^{i}_{T}\\}_{i=1}^{N_{T}}$,weight parameters
$\\{\alpha_{j}\\}_{j=1}^{n}$,max number of epochs $E$, regularization
parameter $\lambda$,number of batches $B$

Output: Optimal feature enocoders $\\{\phi_{S}^{j*}\\}_{j=1}^{n}$, optimal
source weights $\\{\alpha_{j}^{*}\\}_{j=1}^{n}$

Initialization: Freeze final classification layers
$\\{\psi_{S}^{j}\\}_{j=1}^{n}$, set $\alpha_{j}=1$ for all $j$

for  _$epoch=1$ to $E$_ do

Calculate pseudo-labels from equation (6)

Calculate the mean embedding $\bar{p}$ from equation (2)

for  _$iteration=1$ to $B$_ do

Sample a mini batch from target and pass it through each of the source models

calculate all the losses from equation (1),(2) and (10)

calculate total loss from equation (3)

Update the parameters in $\\{\phi_{S}^{j}\\}_{j=1}^{n}$ and
$\\{\alpha_{j}\\}_{j=1}^{n}$ from optimization(3.3)

Make $\alpha$ positive by setting $\alpha_{j}=1/(1+e^{-\alpha_{j}})$

Normalize $\alpha$ by setting $\alpha_{j}=\alpha_{j}/\sum_{i=1}^{n}\alpha_{i}$

end for

end for

Algorithm 1 Algorithm to Solve Eq. 3.3

##  4 Theoretical Insights

Theoretical motivation behind our approach. Our algorithm aims to find the
optimal weights $\\{\alpha_{j}\\}_{j=1}^{n}$ for each source and takes a
convex combination of the source predictors to obtain the target predictor.
Here, we shall show that under intuitive assumptions on the source and target
distributions, there exists a simple choice of target predictor, which can
perform better than or equal to the best source model being applied directly
on the target data.

Formally, let $L$ be a loss function which maps the pair of model-predicted
label and the ground-truth label to a scalar. Denote the expected loss over
$k$-th source distribution $Q^{k}_{S}$ using the source predictor $\theta$ via
$\mathcal{L}(Q^{k}_{S},\theta)=\mathbb{E}_{x}[L(\theta(x),y)]=\int_{x}L(\theta(x),y)Q^{k}_{S}(x)dx$.
Now let $\theta_{S}^{k}$ be the optimal source predictor given by
$\theta^{k}_{S}=\text{arg}\ \underset{\theta}{\text{min}}\
{\cal{L}}(Q^{k}_{S},\theta)\ \forall\ 1\leq k\leq n$. Let us also assume that
the target distribution is in the span of source distributions. We formalize
this by expressing the target distribution as an affine combination of source
distributions i.e.,
$Q_{T}(x)={\sum_{k=1}^{n}\lambda_{k}Q^{k}_{S}(x):\lambda_{k}\geq
0,\sum_{k=1}^{n}\lambda_{k}=1}$. Under this assumption, if we express our
target predictor as
$\theta_{T}(x)=\sum_{k=1}^{n}\frac{\lambda_{k}Q^{k}_{S}(x)}{\sum_{j=1}^{n}\lambda_{j}Q^{j}_{S}(x)}\theta_{S}^{k}(x)$,
then we establish our theoretical claim stated in Lemma 1.

######  Lemma 1.

Assume that the loss $L(\theta(x),y)$ is convex in its first argument and that
there exists a $\lambda\in\mathbb{R}^{n}$ where $\lambda\geq 0$ and
$\lambda^{\top}\mathbbm{1}=1$, such that the target distribution is exactly
equal to the mixture of source distributions, i.e.,
$Q_{T}=\sum_{i=1}^{n}\lambda_{i}Q^{i}_{S}$. Set the target predictor as the
following convex combination of the optimal source predictors

$\theta_{T}(x)=\sum_{k=1}^{n}\frac{\lambda_{k}Q^{k}_{S}(x)}{\sum_{j=1}^{n}\lambda_{j}Q^{j}_{S}(x))}\theta_{S}^{k}(x).$

Recall the pseudo-labeling loss (10). Then, for this target predictor, over
the target distribution, the unsupervised loss induced by the pseudo-labels
and the supervised loss are both less than or equal to the loss induced by the
best source predictor. In particular,

$\mathcal{L}(Q_{T},\theta_{T})\leq\underset{1\leq j\leq n}{\min}\
\mathcal{L}(Q_{T},\theta_{S}^{j}).$

Let $\alpha=\arg\min_{1\leq j\leq n}\ \mathcal{L}(Q_{T},\theta_{S}^{j})$.
Additionally, this inequality is strict if the entries of $\lambda$ are
strictly positive and there exists a source $i$ for which the strict
inequality
$\mathcal{L}(Q^{i}_{S},\theta_{S}^{i})<\mathcal{L}(Q^{i}_{S},\theta_{S}^{\alpha})$
holds.

###### Proof.

See proof in the supplementary. ∎

Observe that the expected loss $\mathcal{L}$ defined in Lemma 1 is the
supervised loss where one does have the label information. Our proposed target
predictor $\theta_{T}$ achieves a supervised loss at least as good as the best
individual source model. Importantly, the inequality is strict under a natural
mild condition: The best individual source model $\beta$ (for the target
$Q_{T}$) is strictly worse than some source model $i$ on the source
distribution $Q_{S}^{i}$. We also note the key differences between our
algorithm and the predictor in Lemma 1. In our algorithm’s combination rule,
we fine-tune the feature extractors of each source model unlike Lemma 1.
However each source has an individual weight which is agnostic to the source
data, whereas Lemma 1 uses different weights per input instance. Below we
provide an intuitive justification for choosing this input-agnostic weighting
strategy.

Since we do not know the source distributions (due to the unavailability of
source data), let us consider the least informative of all the distributions
i.e. uniform distribution for sources by the Principle of Maximum Entropy
[17]. This uniformity is assumed over the target support set $\mathcal{X}$. In
what follows, we will consider the restrictions of the source distributions to
the target support $\cal{X}$. Mathematically, our assumption is
$Q^{k}_{S}(x)=c_{k}\mathcal{U}(x)$ when restricted to the support set
$x\in\mathcal{X}$, where $c_{k}$ is a scaling factor which captures the
relative contribution of source $k$ and $\mathcal{U}(x)$ has value $1$. If we
plug this value of the distribution in the combination rule in Lemma 1, we get
$\theta_{T}(x)=\sum_{k=1}^{n}\frac{\lambda_{k}c_{k}}{\sum_{j=1}^{n}\lambda_{j}c_{j}}\theta_{S}^{k}(x)$
(see supplementary for more details). This term consisting of $\lambda_{k}$
and $c_{k}$ essentially becomes the weighting term $\alpha_{k}$ in our
algorithm. We put this value of $\theta_{T}$ to solve the optimization (3.3)
jointly with respect to this $\alpha_{k}$ and $\phi_{S}^{k}$. Thus, our
optimization will return us a favorable combination of source hypotheses,
satisfying the bounds in Lemma 1, under the uniformity assumption of source
distributions.

##  5 Experiments

Source | Method |  | mt,up,sv,sy
---
$\rightarrow$ mm
| mm,up,sv,sy
---
$\rightarrow$ mt
| mm,mt,sv,sy
---
$\rightarrow$ up
| mm,mt,up,sy
---
$\rightarrow$ sv
| mm,mt,up,sv
---
$\rightarrow$ sy
Avg.
Multiple(w) | DAN[26] | 63.7 | 96.31 | 94.2 | 62.5 | 85.4 | 80.4
DANN[6] | 71.3 | 97.6 | 92.3 | 63.5 | 85.3 | 82.0
MCD[38] | 72.5 | 96.21 | 95.3 | 78.9 | 87.5 | 86.1
CORAL[41] | 62.5 | 97.2 | 93.4 | 64.4 | 82.7 | 80.1
ADDA[44] | 71.6 | 97.9 | 92.8 | 75.5 | 86.5 | 84.8
$\text{M}^{3}$SDA-$\beta$[33] | 72.8 | 98.4 | 96.1 | 81.3 | 89.6 | 87.6
Single(w/o) | Source-best | 60.7 | 98.2 | 74.5 | 89.5 | 89.4 | 82.5
Source-worst | 21.3 | 64 | 29.3 | 7.4 | 25.7 | 29.5
SHOT[23]-best | 94.0 | 98.7 | 97.9 | 83.5 | 97.5 | 94.3
SHOT[23]-worst | 44.5 | 97.2 | 96.2 | 29.5 | 32.5 | 60.0
Multiple(w/o) | SHOT[23]-Ens | 90.4 | 98.9 | 97.7 | 58.3 | 83.9 | 85.8
DECISION(Ours) | 93.0 | 99.2 | 97.8 | 82.6 | 97.5 | 94.0

Table 2: Results on digit recognition. MT, MM, UP, SV, SY are abbreviations of
MNIST, MNIST-M, USPS, SVHN and Synthetic Digits respectively. Multiple and
Single denotes the methods which uses multiple and single sources respectively
for domain adaptation, while (w) and (w/o) are abbreviations of with source
data and without source data respectively. Source is the accuracy with the
unadapted models, whereas -best and -worst refer to the best and worst
sources.

Source | Method |  | A,D
---
$\rightarrow$ W
| A,W
---
$\rightarrow$ D
| D,W
---
$\rightarrow$ A
Avg.
Single | Source-best | 96.3 | 98.4 | 62.5 | 85.7
Source-worst | 75.6 | 80.9 | 62.0 | 72.8
SHOT [23]-best | 98.2 | 99.6 | 75.1 | 90.9
SHOT [23]-worst | 90.6 | 94.2 | 72.9 | 85.9
Multiple | SHOT [23]-Ens | 94.9 | 97.8 | 75.0 | 89.3
DECISION(Ours) | 98.4 | 99.6 | 75.4 | 91.1

Table 3: Results on Office: A,D and W are abbreviations of Amazon, DSLR and
Webcam. For single source methods, Source-best and Source-worst denote the
best and worst unadapted source models, whereas SHOT-best, SHOT-worst are the
best and worst accuracies of adapted source models.

Datasets. To test the effectiveness of our algorithm, we experiment on various
visual benchmarks described as follows.
$\bullet$ Office [14]: In this benchmark DA dataset there are three domains
under the office environment namely Amazon (A), DSLR (D) and Webcam (W) with a
total of 31 object classes in each domain.
$\bullet$ Office-Caltech [8]: This is an extension of the Office dataset, with
Caltech-256 (C) added on top of the 3 existing domains by extracting 10
classes common to all domains.
$\bullet$ Office-Home [45]: Office-Home consists of four domains, namely,
Art(Ar), Clipart(Cl), Product(Pr) and Real-world(Re). Each of these domains
contain 65 object classes.
$\bullet$ Digits: The Digits dataset is a benchmark for DA in digit
recognition. Following [33], we utilize five subsets, namely MNIST (MT), USPS
(UP), SVHN (SV), MNIST-M (MM) and Synthetic Digits (SY) for our experiments.

In all of our experiments, we take turns and fix one of the domains as the
target and the rest as the source domains. The source data is discarded after
training the source models.

Baseline Methods. We compare our method against a wide array of baselines.
Similar to our setting, SHOT [23] attempts unsupervised adaptation without
source data. However, it adapts a single source at a time. We compare against
a multi-source extension of SHOT via ensembling - we pass the target data
through each of the adapted source model and take an average of the soft
prediction to obtain the test label. In our comparisons, we name this method
SHOT-ens. We also compare against single source baselines, namely SHOT-best
and SHOT-worst, which refer to the best adapted source model and the worst one
respectively, learned using SHOT. Additionally, we run comparisons against
traditional multi-source adaptation methods $\text{M}^{3}$SDA-$\beta$[33], DAN
[26], DANN [6], MCD [38], CORAL [41], ADDA [44], DCTN[47]. All these methods,
except for SHOT, have access to source data during adaptation.

Source | Method |  | Ar,Cl,Pr
---
$\rightarrow$ Rw
| Ar,Cl,Rw
---
$\rightarrow$ Pr
| Ar,Pr,Rw
---
$\rightarrow$ Cl
| Cl,Pr,Rw
---
$\rightarrow$ Ar
Avg.
Single(w/o) | Source-best | 74.1 | 78.3 | 46.2 | 65.8 | 66.1
Source-worst | 64.8 | 62.8 | 40.9 | 53.3 | 55.5
SHOT[23]-best | 81.3 | 83.4 | 57.2 | 72.1 | 73.5
SHOT[23]-worst | 80.8 | 77.9 | 53.8 | 66.6 | 69.8
Multiple(w/o) |  SHOT[23]-Ens | 82.9 | 82.8 | 59.3 | 72.2 | 74.3
DECISION(Ours) | 83.6 | 84.4 | 59.4 | 74.5 | 75.5
Table 4: Results on Office-Home.: AR,CL,RW and PR are abbreviations of Art, Clipart,Real-world and Product. We see that our method outperforms all the baselines including the best source accuracy as well as ensemble method. The abbreviations under the column SOURCE and METHOD are same as described in Table 2. Source | Method |  | A,C,D
---
$\rightarrow$ W
| A,C,W
---
$\rightarrow$ D
| C,D,W
---
$\rightarrow$ A
| A,D,W
---
$\rightarrow$ C
Avg.
Multiple(w) |  ResNet-101[11] | 99.1 | 98.2 | 88.7 | 85.4 | 92.9
DAN[26] | 99.5 | 99.1 | 91.6 | 89.2 | 94.8
DCTN[47] | 99.4 | 99.0 | 92.7 | 90.2 | 95.3
MCD[38] | 99.5 | 99.1 | 92.1 | 91.5 | 95.6
$\text{M}^{3}$SDA-$\beta$[33] | 99.5 | 99.2 | 94.5 | 99.2 | 96.4
Single(w/o) | Source-best | 98.9 | 99.3 | 94.8 | 86.5 | 94.9
Source-worst | 86.7 | 89.8 | 89.6 | 83.2 | 87.4
SHOT-best | 99.6 | 100 | 95.8 | 95.5 | 97.7
SHOT-worst | 97.3 | 96.2 | 95.7 | 93.9 | 95.8
Multiple(w/o) | SHOT-Ens | 99.6 | 96.8 | 95.7 | 95.8 | 97.0
DECISION(Ours) | 99.6 | 100 | 95.9 | 95.9 | 98.0
Table 5: Results on Office-Caltech Dataset:A,D,C and W are abbreviations of
Amazon, DSLR, Caltech-256 and Webcam. Our method consistently outperform all
the baselines across all the domains as target.The abbreviations under the
column SOURCE and METHOD are same as described in Table 2.

###  5.1 Implementation details

Network architecture. For the object recognition tasks, we use a pre-trained
ResNet-50 [11] as the feature extractor backbone, similar to [33, 48].
Following [23, 6], we replace the penultimate fully-connected layer with a
bottleneck layer and a task specific classifier layer. Batch normalization
[16] is utilized after the bottleneck layer, along with weight normalization
[39] in the final layer. For the digit recognition task, we use a variant of
the LeNet [20] similar to [23].
Source model training. Following [23], we train the source models using smooth
labels, instead of the usual one-hot encoded labels. This increases the
robustness of the model and helps in the adaptation process by encouraging
features to lie in tight, evenly separated clusters [29]. The maximum number
of epochs for Digits, Office, Office-Home and Office-Caltech is set to 30,
100, 50 and 100, respectively. Additionally, for our experiments on digit
recognition, we resize images from each domain to 32$\times$32 and convert the
gray-scale images to RGB.
Hyper-parameters. The entire framework is trained in an end-to-end fashion via
back-propagation. Specifically, we utilize stochastic gradient descent with
momentum value $0.9$ and weight decay equalling $10^{-3}$. The learning rate
is set at $10^{-2}$ for the bottleneck and classifier layers, while the
backbone is trained at a rate of $10^{-3}$. In addition, we use the learning
rate scheduling strategy from [6], where the initial rate is exponentially
decayed as learning progresses. The batch size is set to $32$. We use
$\lambda=0.3$ for all the object recognition tasks and $\lambda=0.1$ for the
digits benchmark. For adaptation, maximum number of epochs is set to 15, with
the pseudo-labels updated at the start of every epoch. We use PyTorch [31] for
all our experiments.

###  5.2 Digit recognition

The results on digit recognition are shown in Table 2. The digit benchmark is
characterised by the presence of very poor sources in some scenarios, notably
when treating MNIST-M, SVHN or Synthetic Digits as the target domain. For
example, on SVHN as the target, the best and worst source models adapted using
SHOT [23] exhibit a performance gap of more than $50\%$. Combining these
models via uniform ensembling results in a predictor which greatly
underperforms the best adapted source. In contrast, our method restricts this
severe negative transfer via a joint adaptation over the models and the
ensembling weights, and outperforms the baseline by $\mathbf{24.3}\%$. The
corresponding increase in performance when using Synthetic Digits and MNISTM
as the target are $\mathbf{13.5}\%$ and $\mathbf{2.6}\%$ respectively.
Overall, we obtain an average increase of $\mathbf{8.2}\%$ across all the
digit adaptation tasks over SHOT-Ens. In spite of such disparities among the
sources, our framework also achieves performance at par with the best adapted
source and actually outperforms the latter on the MNIST transfer task. We also
outperform the traditional multi-source adaptation methods, which use source
data, on all the tasks by an average of $\mathbf{6.4}\%$.

###  5.3 Object recognition

Office. The results for the 3 adaptation tasks on the Office dataset are shown
in Table 3. We achieve performance at par with the best adapted source models
on all the tasks and obtain an average increase of $\mathbf{5.2}\%$ over SHOT-
Ens. In the task of adapting to the Webcam (W) domain, negative transfer from
the Amazon (A) model brings the ensemble performance down - our model is able
to prevent this, and not only outperforms the ensemble by $\mathbf{3.5}\%$ but
also achieves higher performance than the best adapted source.
Office-Home. On the Office-Home dataset, we outperform all baselines as shown
in Table 4. Across all tasks, our method achieves a mean increase in accuracy
of $\mathbf{2}\%$ over the respective best adapted source models. This can be
attributed to the relatively small performance gap between the best and worst
adapted sources in comparison to other datasets. This suggests that, as the
performance gap between the best and worst performing sources gets smaller, or
outlier sources are removed, our method can generalize even better to the
target.
Office-Caltech. The results follow a similar trend on the Office-Caltech
dataset, as shown in Table 5. With a mean accuracy of $\mathbf{98}\%$ across
all tasks, we outperform all baselines.

###  5.4 Ablation study

Contribution of each loss. Our framework is trained using a combination of
three distinct losses: $\mathcal{L}_{\text{div}}$, $\mathcal{L}_{\text{ent}}$
and $\mathcal{L}_{\text{pl}}$. We study the contribution of each component of
our framework to the adaptation task in Table 6. First, we remove both the
diversity loss and the pseudo-labeling, and train using only
$\mathcal{L}_{\text{ent}}$. Next, we add in $\mathcal{L}_{\text{div}}$ and
perform weighted information maximization. Finally, we also compare the
results of solely using $\mathcal{L}_{\text{pl}}$.

Method |  | A,D
---
$\rightarrow$ W
| A,W
---
$\rightarrow$ D
| D,W
---
$\rightarrow$ A
Avg.
$\mathcal{L}_{pl}$ | 97.6 | 98.5 | 75.3 | 90.5
$-\mathcal{L}_{ent}$ | 96.6 | 99.0 | 68.5 | 88.0
$-\mathcal{L}_{ent}+\mathcal{L}_{div}$ | 95.9 | 99.0 | 71.6 | 88.9
$-\mathcal{L}_{ent}+\mathcal{L}_{div}+\lambda\mathcal{L}_{pl}$ | 98.4 | 99.6 | 75.4 | 91.1
Table 6: Loss-wise ablation. Contribution of each component in adaptation on
the Office dataset.

Analysis on the learned weights. Our framework jointly adapts the the source
models and learns the weights on each such source. To understand the impact of
the weights, we propose to freeze the feature extractors and optimize solely
over the weights $\\{\alpha_{j}\\}_{i=1}^{n}$. Naturally, this setup yields
better performance compared to trivially assigning equal weights to all source
models, as shown in Table 7. More interestingly, the learned weights correctly
indicate which source model performs better on the target and could serve as a
proxy indicator in a model selection framework. See Figure 3.

Method |  | Ar,Cl,Pr
---
$\rightarrow$ Rw
| Ar,Cl,Rw
---
$\rightarrow$ Pr
| Ar,Pr,Rw
---
$\rightarrow$ Cl
| Cl,Pr,Rw
---
$\rightarrow$ Ar
Avg.
Source-Ens | 67.6 | 51.4 | 77.7 | 80.1 | 69.2
DECISION-weights | 68.8 | 52.3 | 79.2 | 80.4 | 70.2

Table 7: Performance on freezing backbone network on Office-Home. DECISION-
weight is optimized solely over the source weights and consistently performs
better than uniform weighting. Figure 3: Weights as model selection proxy. The
weights learnt by our framework on Office-Home correlates positively with the
unadapted source model performance. (Left axis corresponds to the red plot and
right to the blue plot, best viewed in color.)

Distillation into a single model. Since we are dealing with multiple source
models, inference time is of the order $\mathcal{O}(m)$, where $m$ is the
number of source models. If $m$ is large, this can lead to inference being
quite time consuming. To ameliorate this overhead, we follow a knowledge
distillation [12] strategy to obtain a single target model. Teacher
supervision is obtained by linearly combining the adapted models via the
learned weights. These annotations are subsequently used to train the single
student model via vanilla cross-entropy loss. Results obtained using this
strategy are presented in the supplementary.

##  6 Conclusions and Future Work

We developed a new UDA algorithm that can learn from and optimally combine
multiple source models without requiring source data. We provide theoretical
intuitions for our algorithm and verify its effectiveness in a variety of
domain adaptation benchmarks. There are multiple exciting directions to pursue
including: First, we suspect that our algorithm’s performance can be further
boosted by incorporating data augmentation techniques during training. Second,
when there are too many source models to utilize, it would be interesting to
study whether we can automatically select an optimal subset of the source
models without requiring source data in an unsupervised fashion.

Acknowledgements. This work was partially supported by the ONR grant
N00014-19-1-2264 and the NSF grants 2008020, 2046816.

## References

  * [1] Sk Miraj Ahmed, Aske R Lejbolle, Rameswar Panda, and Amit K Roy-Chowdhury.  Camera on-boarding for person re-identification using hypothesis transfer learning.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 12144–12153, 2020.
  * [2] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan.  A theory of learning from different domains.  Machine learning, 79(1-2):151–175, 2010.
  * [3] John S Bridle, Anthony JR Heading, and David JC MacKay.  Unsupervised classifiers, mutual information and’phantom targets.  In Advances in neural information processing systems, pages 1096–1101, 1992.
  * [4] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze.  Deep clustering for unsupervised learning of visual features.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 132–149, 2018.
  * [5] Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien.  Semi-supervised learning (chapelle, o. et al., eds.; 2006)[book reviews].  IEEE Transactions on Neural Networks, 20(3):542–542, 2009.
  * [6] Yaroslav Ganin and Victor Lempitsky.  Unsupervised domain adaptation by backpropagation.  In International conference on machine learning, pages 1180–1189. PMLR, 2015.
  * [7] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky.  Domain-adversarial training of neural networks.  The Journal of Machine Learning Research, 17(1):2096–2030, 2016\.
  * [8] Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman.  Geodesic flow kernel for unsupervised domain adaptation.  In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 2066–2073. IEEE, 2012.
  * [9] Yves Grandvalet and Yoshua Bengio.  Semi-supervised learning by entropy minimization.  In Advances in neural information processing systems, pages 529–536, 2005.
  * [10] Jiang Guo, Darsh J Shah, and Regina Barzilay.  Multi-source domain adaptation with mixture of experts.  arXiv preprint arXiv:1809.02256, 2018.
  * [11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.
  * [12] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network.  arXiv preprint arXiv:1503.02531, 2015.
  * [13] Judy Hoffman, Mehryar Mohri, and Ningshan Zhang.  Algorithms and theory for multiple-source adaptation.  In Advances in Neural Information Processing Systems, pages 8246–8256, 2018.
  * [14] Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell.  Cycada: Cycle-consistent adversarial domain adaptation.  In International conference on machine learning, pages 1989–1998. PMLR, 2018.
  * [15] Han-Kai Hsu, Chun-Han Yao, Yi-Hsuan Tsai, Wei-Chih Hung, Hung-Yu Tseng, Maneesh Singh, and Ming-Hsuan Yang.  Progressive domain adaptation for object detection.  In The IEEE Winter Conference on Applications of Computer Vision, pages 749–757, 2020.
  * [16] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  arXiv preprint arXiv:1502.03167, 2015.
  * [17] Edwin T Jaynes.  Information theory and statistical mechanics.  Physical review, 106(4):620, 1957.
  * [18] Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár.  Panoptic segmentation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 9404–9413, 2019.
  * [19] Andreas Krause, Pietro Perona, and Ryan G Gomes.  Discriminative clustering by regularized information maximization.  In Advances in neural information processing systems, pages 775–783, 2010.
  * [20] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.  Gradient-based learning applied to document recognition.  Proceedings of the IEEE, 86(11):2278–2324, 1998.
  * [21] Rui Li, Qianfen Jiao, Wenming Cao, Hau-San Wong, and Si Wu.  Model adaptation: Unsupervised domain adaptation without source data.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9641–9650, 2020.
  * [22] Yitong Li, David E Carlson, et al.  Extracting relationships by multi-domain matching.  In Advances in Neural Information Processing Systems, pages 6798–6809, 2018.
  * [23] Jian Liang, Dapeng Hu, and Jiashi Feng.  Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation.  arXiv preprint arXiv:2002.08546, 2020.
  * [24] Chuang Lin, Sicheng Zhao, Lei Meng, and Tat-Seng Chua.  Multi-source domain adaptation for visual sentiment classification.  In AAAI, pages 2661–2668, 2020.
  * [25] Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3431–3440, 2015.
  * [26] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan.  Learning transferable features with deep adaptation networks.  In International conference on machine learning, pages 97–105. PMLR, 2015.
  * [27] Yawei Luo, Liang Zheng, Tao Guan, Junqing Yu, and Yi Yang.  Taking a closer look at domain shift: Category-level adversaries for semantics consistent domain adaptation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2507–2516, 2019.
  * [28] Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh.  Domain adaptation with multiple sources.  In Advances in Neural Information Processing Systems, pages 1041–1048, 2009.
  * [29] Rafael Müller, Simon Kornblith, and Geoffrey E Hinton.  When does label smoothing help?  In Advances in Neural Information Processing Systems, pages 4694–4703, 2019.
  * [30] Samet Oymak and Talha Cihad Gulcu.  Statistical and algorithmic insights for semi-supervised learning with self-training.  arXiv preprint arXiv:2006.11006, 2020.
  * [31] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  In Advances in neural information processing systems, pages 8026–8037, 2019.
  * [32] Sujoy Paul, Yi-Hsuan Tsai, Samuel Schulter, Amit K Roy-Chowdhury, and Manmohan Chandraker.  Domain adaptive semantic segmentation using weak labels.  arXiv preprint arXiv:2007.15176, 2020.
  * [33] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang.  Moment matching for multi-source domain adaptation.  In Proceedings of the IEEE International Conference on Computer Vision, pages 1406–1415, 2019.
  * [34] Michaël Perrot and Amaury Habrard.  A theoretical analysis of metric hypothesis transfer learning.  In International Conference on Machine Learning, pages 1708–1717, 2015.
  * [35] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi.  You only look once: Unified, real-time object detection.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 779–788, 2016.
  * [36] Paolo Russo, Tatiana Tommasi, and Barbara Caputo.  Towards multi-source adaptive semantic segmentation.  In International Conference on Image Analysis and Processing, pages 292–301. Springer, 2019.
  * [37] Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell.  Adapting visual category models to new domains.  In European Conference on Computer Vision, pages 213–226. Springer, 2010.
  * [38] Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada.  Maximum classifier discrepancy for unsupervised domain adaptation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3723–3732, 2018.
  * [39] Tim Salimans and Durk P Kingma.  Weight normalization: A simple reparameterization to accelerate training of deep neural networks.  In Advances in neural information processing systems, pages 901–909, 2016.
  * [40] Shashank Singh, Ananya Uppal, Boyue Li, Chun-Liang Li, Manzil Zaheer, and Barnabás Póczos.  Nonparametric density estimation under adversarial losses.  In Advances in Neural Information Processing Systems, pages 10225–10236, 2018.
  * [41] Baochen Sun, Jiashi Feng, and Kate Saenko.  Return of frustratingly easy domain adaptation.  arXiv preprint arXiv:1511.05547, 2015.
  * [42] Baochen Sun, Jiashi Feng, and Kate Saenko.  Return of frustratingly easy domain adaptation.  In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, pages 2058–2065, 2016.
  * [43] Antti Tarvainen and Harri Valpola.  Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.  In Advances in neural information processing systems, pages 1195–1204, 2017.
  * [44] Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell.  Adversarial discriminative domain adaptation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7167–7176, 2017.
  * [45] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan.  Deep hashing network for unsupervised domain adaptation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5018–5027, 2017.
  * [46] Haotian Wang, Wenjing Yang, Zhipeng Lin, and Yue Yu.  Tmda: Task-specific multi-source domain adaptation via clustering embedded adversarial training.  In 2019 IEEE International Conference on Data Mining (ICDM), pages 1372–1377. IEEE, 2019.
  * [47] Ruijia Xu, Ziliang Chen, Wangmeng Zuo, Junjie Yan, and Liang Lin.  Deep cocktail network: Multi-source unsupervised domain adaptation with category shift.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3964–3973, 2018.
  * [48] Ruijia Xu, Guanbin Li, Jihan Yang, and Liang Lin.  Larger norm more transferable: An adaptive feature norm approach for unsupervised domain adaptation.  In Proceedings of the IEEE International Conference on Computer Vision, pages 1426–1435, 2019.
  * [49] Han Zhao, Shanghang Zhang, Guanhang Wu, José MF Moura, Joao P Costeira, and Geoffrey J Gordon.  Adversarial multiple source domain adaptation.  In Advances in neural information processing systems, pages 8559–8570, 2018.
  * [50] Sicheng Zhao, Bo Li, Pengfei Xu, and Kurt Keutzer.  Multi-source domain adaptation in the deep learning era: A systematic survey.  arXiv preprint arXiv:2002.12169, 2020.
  * [51] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In Proceedings of the IEEE international conference on computer vision, pages 2223–2232, 2017.

Unsupervised Multi-source Domain Adaptation Without Access to Source Data
(Supplementary Material)

##  7 Proof of Lemma 1

######  Lemma 2.

Assume that the loss $L(\theta(x),y)$ is convex in its first argument and that
there exists a $\lambda\in\mathbb{R}^{n}$ where $\lambda\geq 0$ and
$\lambda^{\top}\mathbbm{1}=1$, such that the target distribution is exactly
equal to the mixture of source distributions, i.e
$Q_{T}=\sum_{i=1}^{n}\lambda_{i}Q^{i}_{S}$. Set the target predictor as the
following convex combination of the optimal source predictors

$\theta_{T}(x)=\sum_{k=1}^{n}\frac{\lambda_{k}Q^{k}_{S}(x)}{\sum_{j=1}^{n}\lambda_{j}Q^{j}_{S}(x))}\theta_{S}^{k}(x).$

Recall the pseudo-labeling loss ($10$). Then, for this target predictor, over
the target distribution, the unsupervised loss induced by the pseudo-labels
and the supervised loss are both less than or equal to the loss induced by the
best source predictor. In particular,

$\mathcal{L}(Q_{T},\theta_{T})\leq\underset{1\leq j\leq n}{\min}\
\mathcal{L}(Q_{T},\theta_{S}^{j}).$

###### Proof.

We can see that the left hand-side of the inequality can be upper-bounded by
some loss as follows,

$\begin{split}\mathcal{L}(Q_{T},\theta_{T})=\int_{x}Q_{T}(x)L(\theta_{T}(x),y)=&\int_{x}Q_{T}(x)L\Bigg{(}\sum_{i=1}^{n}\frac{\lambda_{i}Q^{i}_{S}(x)}{\sum_{j=1}^{n}\lambda_{j}Q^{j}_{S}(x))}\theta_{S}^{i}(x),y\Bigg{)}dx\\\
&\leq\int_{x}Q_{T}(x)\sum_{i=1}^{n}\frac{\lambda_{i}Q^{i}_{S}(x)}{\sum_{j=1}^{n}\lambda_{j}Q^{j}_{S}(x))}L(\theta_{S}^{i}(x),y)dx~{}~{}\
\text{(from Jensen's inequality)}\\\
&=\int_{x}Q_{T}(x)\sum_{i=1}^{n}\frac{\lambda_{i}Q^{i}_{S}(x)}{Q_{T}(x)}L(\theta_{S}^{i}(x),y)dx~{}~{}\
\text{(from distribution assumption)}\\\
&=\sum_{i=1}^{n}\lambda_{i}\int_{x}Q^{i}_{S}(x)L(\theta_{S}^{i}(x),y)dx~{}~{}\
\text{(changing the order of summation)}\\\
&=\sum_{i}\lambda_{i}\mathcal{L}(Q^{i}_{S}(x),\theta_{S}^{i})\end{split}$ (12)

Now for the R.H.S. we can write this loss as follows,

$\begin{split}\mathcal{L}(Q_{T},\theta_{S}^{j})&=\int_{x}Q_{T}(x)L(\theta_{S}^{j}(x),y)dx\\\
&=\int_{x}\sum_{i=1}^{n}\lambda_{i}Q^{i}_{S}(x)L(\theta_{S}^{j}(x),y)dx\\\
&=\sum_{i=1}^{n}\lambda_{i}\int_{x}Q^{i}_{S}L(\theta_{S}^{j}(x),y)dx\\\
&=\sum_{i=1}^{n}\lambda_{i}\mathcal{L}(Q^{i}_{S}(x),\theta_{S}^{j})\end{split}$
(13)

Now recall from main paper that,

$\theta^{k}_{S}=\arg\min_{\theta}{\cal{L}}(Q^{k}_{S},\theta)\quad\text{for}\quad
1\leq k\leq n.$

.

This means $\theta_{S}^{i}$ is the best predictor for the source $i$, which
has distribution $Q^{i}_{S}$. Thus we find that
$\mathcal{L}(Q^{i}_{S},\theta_{S}^{i})\leq\mathcal{L}(Q^{i}_{S},\theta_{S}^{j})\
\forall j$, which implies
$\sum_{i}\lambda_{i}\mathcal{L}(Q^{i}_{S},\theta_{S}^{i})\leq\sum_{i}\lambda_{i}\mathcal{L}(Q^{i}_{S},\theta_{S}^{j})$.
This further implies that
$\mathcal{L}(Q_{T},\theta_{T})\leq\mathcal{L}(Q_{T},\theta_{S}^{j})\ \forall
j$, which in turn concludes the proof
$\mathcal{L}(Q_{T},\theta_{T})\leq\underset{1\leq j\leq n}{\text{min}}\
\mathcal{L}(Q_{T},\theta_{S}^{j})$. Finally, suppose the entries of $\lambda$
are strictly positive and let
$\beta=\arg\min_{j}\mathcal{L}(Q_{T},\theta^{j}_{S})$. Observe that, if there
is a source $i$ such that the strict inequality
$\mathcal{L}(Q^{i}_{S},\theta_{S}^{i})<\mathcal{L}(Q^{i}_{S},\theta_{S}^{\beta})$
holds, then the main claim of the lemma also becomes strict as we find

$\mathcal{L}(Q_{T},\theta_{T})\leq\sum_{i}\lambda_{i}\mathcal{L}(Q^{i}_{S},\theta_{S}^{i})<\sum_{i}\lambda_{i}\mathcal{L}(Q^{i}_{S},\theta_{S}^{\beta})\leq\min_{j}\mathcal{L}(Q_{T},\theta^{j}_{S}).$

Verbally, this strict inequality has a natural meaning that the model $j$ is
strictly worse than model $i$ for the source data $i$. ∎

##  8 Detailed steps of combination rule under source distribution uniformity
assumption

See the discussion after Lemma 1 in the main paper for reference.

$\begin{split}\theta_{T}(x)&=\sum_{k=1}^{n}\frac{\lambda_{k}Q^{k}_{S}(x)}{\sum_{j=1}^{n}\lambda_{j}Q^{j}_{S}(x)}\theta_{S}^{k}(x)\\\
&=\sum_{k=1}^{n}\frac{\lambda_{k}c_{k}\mathcal{U}(x)}{\sum_{j=1}^{n}\lambda_{j}c_{j}\mathcal{U}(x)}\theta_{S}^{k}(x)\\\
&=\sum_{k=1}^{n}\frac{\lambda_{k}c_{k}}{\sum_{j=1}^{n}\lambda_{j}c_{j}}\theta_{S}^{k}(x)\\\
\end{split}$ (14)

##  9 Additional Experiments

Figure 4: Weights as model selection proxy. The weights learnt by our
framework on Office-31 correlates positively with the unadapted source model
performance. (Left axis corresponds to the red plot and right to the blue
plot, best viewed in color.)

From Figure 4, we can clearly see that for the model which gives higher
accuracy for the unadapted scenario, it is automatically given higher
weightage by our algorithm. As a result, we can easily infer about the quality
of the source domain, in relation to the target, from the weights learnt by
our framework.

Effect of weight on pseudo-labeling. We investigate the effect of the weight
$\lambda$ on $\mathcal{L}_{\text{pl}}$. We perform experiments on the Office
dataset by varying the value of $\lambda$ and plot the results in Figure 5. As
shown in the plot, the proposed method performs best at $\lambda=0.3$

Figure 5: Effect of $\lambda$. The variations in classification as the weight
on $\mathcal{L}_{\text{pl}}$ is varied. (Best viewed in color)

Effect of outlier source models. Our method is clearly robust to outlier
source models. In Table 2 of the main paper, when MNIST-M is the target,
transferring from only USPS, leads to an extremely poor performance of
21.3$\%$ \- here, USPS is a strong outlier. Despite the presence of such a
poor source, our framework is mostly able to correctly negate the transfer
from USPS, achieving a performance of 93$\%$, close to the best source
performance of 94$\%$. On removing USPS as a source, DECISION outperforms the
best source by achieving an accuracy of 94.5$\%$. In the future, we plan to
actively use the weights to simultaneously remove poor sources while
adaptation in order to boost the performance.

DomainNet [32]: This is a relatively new and large dataset where there are six
domains under the common object categories, namely quickdraw (Q), clipart (C),
painting (P), infograph (I), sketch (S) and real (R) with a total of 345
object classes in each domain. Experimental results on this dataset are shown
in Table 8. Our method consistently outperforms the best adapted source
baselines (SHOT-best) except for infograph as a target. However the average
performance over all the domains as target is slightly less than the SHOT-Ens.
Note that for quickdraw and clipart as target, our method outperforms all the
state of the art methods including source free and with source data single and
multi source state-of-the-art DA methods.

Source | Method |  | C,P,I,S,R
---
$\rightarrow$ Q
| Q,P,I,S,R
---
$\rightarrow$ C
| Q,C,I,S,R
---
$\rightarrow$ P
| Q,C,P,S,R
---
$\rightarrow$ I
| Q,C,P,I,R
---
$\rightarrow$ S
| Q,C,P,I,S
---
$\rightarrow$ R
Avg.
Multiple(w) |  DAN[25] | 16.2 | 39.1 | 33.3 | 11.4 | 29.7 | 42.1 | 28.6
DCTN[46] | 7.2 | 48.6 | 48.8 | 23.4 | 47.3 | 53.5 | 38.1
MCD[37] | 7.6 | 54.3 | 45.7 | 22.1 | 43.5 | 58.4 | 38.6
$\text{M}^{3}$SDA-$\beta$[32] | 6.3 | 58.6 | 52.3 | 26 | 49.5 | 62.7 | 42.5
Single(w/o) | Source-best | 11.9 | 49.9 | 47.5 | 20 | 41.1 | 57.7 | 38
Source-worst | 2.3 | 12.2 | 2.2 | 1.1 | 8.7 | 4.8 | 5.2
SHOT[22]-best | 18.7 | 58.3 | 53 | 22.7 | 48.4 | 65.9 | 44.5
SHOT[22]-worst | 3.8 | 14.8 | 3.5 | 1 | 11.9 | 6.6 | 7
Multiple(w/o) |  SHOT[22]-Ens | 15.3 | 58.6 | 55.3 | 25.2 | 52.4 | 70.5 | 46.2
DECISION(Ours) | 18.9 | 61.5 | 54.6 | 21.6 | 51 | 67.5 | 45.9
Table 8: Results on DomainNet:Q,C,P,I,S and R are abbreviations of quickdraw,
clipart, painting, infograph, sketch and real.

Distillation. Our results on using the distillation strategy outlined in
Section 5.4 of the main paper are shown in Table 9. Despite the model
compression, the performance remains consistent.

Method | Office-home | Office-Caltech | Office
---|---|---|---
| Rw | Pr | Cl | Ar | A | C | D | W | A | D | W
DECISION (original) | 83.6 | 84.4 | 59.4 | 74.5 | 95.9 | 95.9 | 100 | 99.6 | 75.4 | 99.6 | 98.4
DECISION (distillation) | 83.7 | 84.4 | 59.1 | 74.4 | 96.0 | 95.7 | 99.4 | 99.6 | 75.4 | 99.6 | 98.1
Table 9: Distillation results on object recognition tasks. Performance remains
consistent across all datasets despite distilling into a single target model.


# Leveraging OpenStack and Ceph for a Controlled-Access Data Cloud

Evan F. Bollig  MN Supercomputing Institute
University of Minnesota599 Walter Library
110 Pleasant St SE MinneapolisMinnesota55455<EMAIL_ADDRESS>,  Graham T. Allan
MN Supercomputing Institute<EMAIL_ADDRESS>,  Benjamin J. Lynch  MN
Supercomputing Institute<EMAIL_ADDRESS>,  Yectli A. Huerta  MN Supercomputing
Institute<EMAIL_ADDRESS>,  Mathew Mix  MN Supercomputing Institute
<EMAIL_ADDRESS>,  Edward A. Munsell  MN Supercomputing Institute
<EMAIL_ADDRESS>,  Raychel M. Benson  MN Supercomputing Institute
<EMAIL_ADDRESS>and  Brent Swartz  MN Supercomputing Institute
<EMAIL_ADDRESS>
(2018)

###### Abstract.

While traditional HPC has and continues to satisfy most workflows, a new
generation of researchers has emerged looking for sophisticated, scalable, on-
demand, and self-service control of compute infrastructure in a cloud-like
environment. Many also seek safe harbors to operate on or store sensitive
and/or controlled-access data in a high capacity environment.

To cater to these modern users, the Minnesota Supercomputing Institute
designed and deployed Stratus, a locally-hosted cloud environment powered by
the OpenStack platform, and backed by Ceph storage. The subscription-based
service complements existing HPC systems by satisfying the following unmet
needs of our users: a) on-demand availability of compute resources; b) long-
running jobs (i.e., $>30$ days); c) container-based computing with Docker; and
d) adequate security controls to comply with controlled-access data
requirements.

This document provides an in-depth look at the design of Stratus with respect
to security and compliance with the NIH’s controlled-access data policy.
Emphasis is placed on lessons learned while integrating OpenStack and Ceph
features into a so-called “walled garden”, and how those technologies
influenced the security design. Many features of Stratus, including tiered
secure storage with the introduction of a controlled-access data “cache”,
fault-tolerant live-migrations, and fully integrated two-factor
authentication, depend on recent OpenStack and Ceph features.

OpenStack, Ceph, Protected Data, dbGaP, S3, Private Cloud, Docker, Cloud
Computing

††journalyear: 2018††copyright: rightsretained††conference: Practice and
Experience in Advanced Research Computing; July 22–26, 2018; Pittsburgh, PA,
USA††booktitle: PEARC ’18: Practice and Experience in Advanced Research
Computing, July 22–26, 2018, Pittsburgh, PA, USA††doi:
10.1145/3219104.3219165††isbn: 978-1-4503-6446-1/18/07††ccs: Security and
privacy Virtualization and security††ccs: Computer systems organization Cloud
computing††ccs: Information systems Open source software††ccs: Applied
computing Service-oriented architectures

##  1\. Introduction

Cloud Computing has taken the world by storm. For most, the Cloud promises
immediate, on-demand access to resources, self-service management of
infrastructure, plus the ability to operate with a level of agility and
isolation not possible on traditional high performance computing (HPC)
systems. Institutions like the Minnesota Supercomputing Institute
(MSI)—primary provider for research computing services at the University of
Minnesota—are fully aware of the allure of Cloud Computing to many of their
users.

For many years, MSI has sought to provide a few fully-managed HPC clusters as
a shared resource for users, unified by a single global file system, and
centralized user authentication. At present, MSI hosts two clusters, the
latest of which, Mesabi, was purchased in 2015 and is still among the top 20
university-owned supercomputers in the nation. These clusters are available to
all MSI users as batch scheduled resources, with the scheduler tracking a
number of fair-share parameters to decide how and when jobs are run. Although
Mesabi is predominantly homogenous in architecture and networking,
heterogeneity does exist to meet diverse user needs in the specific
configurations of memory (e.g., nodes range between 64 GB and 1 TB), as well
as the presence of solid-state drives and/or accelerator boards.

Whereas traditional HPC continues to satisfy the majority of needs across
research disciplines, there is a growing trend toward data-intensive work with
ever larger storage and compute requirements, as well as more stringent data-
use agreements. Data-use agreements in particular are cumbersome to satisfy on
HPC clusters where one-off modifications necessary to satisfy agency
requirements for logging, backups (or lack thereof), and isolation would
invasively impact the workflows and user-experiences of all users in the
shared resource. The overhead in additional maintenance, monitoring, and user
education may be acceptable for a small number of edge-cases, but the burden
snowballs quickly to become unsustainable.

Compounding the issue, the popularity of Cloud Computing has given rise to
more sophisticated researchers who cross the role of developer with
operations. These users, referred to as DevOps, seek an environment unchained
from managed infrastructure, where they have the rights to escalate privileges
to install software, manage firewalls, and control other aspects of system
configuration, all in the name of agility and productivity.

Facing the need to accommodate and sustain this trend, MSI designed and built
Stratus, an on-premise cloud for protected data. Stratus is a subscription-
based Infrastructure-as-a-Service that enables users to operate within their
own virtual machines (VMs). Stratus is powered by the Newton version of the
OpenStack (OpenStack, 2017) cloud platform, and is backed by the Luminous
release of Ceph (Weil et al., 2006) storage. Stratus was opened to MSI users
at the start of fiscal year 2018 (July, 2017). Beyond managing protected data,
Stratus offers three features not present in MSI’s existing
cyberinfrastructure: a) on-demand availability of compute resources with self-
service user control; b) long-running jobs (e.g., $>30$ days) that are live-
migrated between compute nodes, and persist through maintenance periods; and
c) container-based computing with Docker, which comes pre-loaded on MSI-
blessed VMs.

OpenStack was chosen for multiple reasons: a) it has built-in support for
multi-tenancy, software defined networking, logging, SSL/TLS encrypted traffic
and other features to make compliance easier; b) the modular OpenStack
services enable mix-and-match configuration and are decoupled for fault
tolerance; and c) a massive open source community distinguishes the project as
the leading software for cloud deployments across industry and academia. Ceph
is free, open source, and backed by much of the same community as OpenStack.
Ceph excels as an efficient, low-cost storage platform with the ability to
scale to many PBs. The stability, scalability, and value of OpenStack and Ceph
have been vetted by large deployments like CERN (Bell et al., 2015) and the
NSF funded JetStream (Stewart et al., 2015).

In its initial release, Stratus is designed expressly to satisfy the NIH
Genomic Data Sharing (GDS) Policy for the Database of Genotypes and Phenotypes
(dbGaP) (Mailman et al., 2007). Although other more stringent regulations and
standards exist (e.g., HIPAA, FISMA, and ITAR to name a few), tackling dbGaP
data is a good launching point for Stratus in that it addresses basic needs
for access logging, data encryption, two-factor authentication, etc.
Furthermore, it satisfies an immediate need at the University of Minnesota
where researchers have no alternative available to analyze dbGaP data.
Compliance with other regulations can be added in the future.

Stratus is most similar in spirit to the Bionimbus Protected Data Cloud (PDC)
(Heath et al., 2014); a fully GDS- and HIPAA-compliant platform at the
University of Chicago. Both Stratus and Bionimbus PDC are built on OpenStack
and have Ceph storage. Stratus storage is entirely Ceph, in contrast to the
current Bionimbus PDC, which has two object stores for protected data: 400 TB
Ceph S3 and 1.7 PB IBM CleverSafe S3. Furthermore, Bionimbus PDC is a NIH
Trusted Partner and allowed to maintain a complete persistent clone of the
dbGaP data, while Stratus presents users a multi-tiered storage environment
with a “dbGaP Cache” to store only active subsets of the dbGaP data in a
scratch-like space. When the object lifecycle is complete or data becomes
stale, they are deleted from the object store. This allows Stratus to avoid
bit rot, and conserve storage costs.

In a similar vein, the CancerCollaboratory (Inc., 2017) at the Ontario
Institute for Cancer Research, is a Ceph-backed OpenStack cloud for cancer-
related data. CancerCollaboratory is designed for maximum storage capacity,
which is used to persistently clone, among other things, dbGaP data dealing
with cancer from Bionimbus PDC.

The NIH has also sponsored three public cloud pilots: a) Broad Institute
FireCloud (FireCloud, 2017); b) Seven Bridges Cancer Genomics Cloud (Bridges,
2017); and c) the Institute for Systems Biology Cancer Genomics Cloud (for
Systems Biology, 2017). All three pilots run on public providers
(Google/Amazon), where the annual cost of computing is substantially higher
than on a local on-premise cloud. To alleviate this burden, the NIH is
currently offering credits for approved projects to spend at the public
providers.

##  2\. Stratus Cloud Platform

Stratus is a relatively small cluster, less than one rack in size. The Stratus
compute hardware is currently twenty HPE ProLiant XL230a nodes, each with two
Intel E5-2680v4 (14-core) CPUs, 256 GB of RAM, and 10 GbE networking to the
outside world. Hyper-threading is enabled for a total core count of 1120
cores. An array of eight HPE Apollo 4200 storage servers are each connected to
compute nodes via two redundant 40 GbE switches. Each storage node has 256 GB
memory, two 800 GB Intel NVMe P3700 accelerator boards, and 198 TB raw
capacity; yielding 1.5 PB raw capacity for the entire cluster. Ten additional
8-core Advanced Clustering control servers, each with 32 GB RAM, run the
OpenStack APIs, monitor storage operations and orchestrate VM lifecycles. Two
of the ten are network controllers for VM traffic and have dual 40 GbE network
connections, while the remaining eight nodes have dual Intel X520 10GbE
network, and various purposes: two to run the Ceph RADOS gateway, three Ceph
storage monitors, one primary OpenStack API and identity controller, one
OpenStack storage controller, and one telemetry and admin node.

The Stratus platform is comprised of OpenStack services: Keystone, Cinder,
Glance, Nova, Neutron, and Horizon. Keystone manages projects, groups and user
role assignments. Identity management is outsourced by Keystone to the central
UMN Shibboleth server, also requiring Duo two-factor authentication (Security,
2017). The Cinder API provides the ability to create both storage and VM boot
volumes within Ceph block device storage. The Glance API publishes a
repository of MSI-blessed machine images to all projects, though denies
permissions to create/modify images for regular users. Since Glance images
reside in block device storage, users can create boot volumes as Copy-on-Write
clones of these images, and boot VMs in seconds. Cinder snapshots of boot
volumes suffice as machine images in the user-space and count toward project
volume quotas. Nova manages virtual machines on compute nodes. VMs are run
with QEMU+KVM hardware accelerated virtualization for minimal overhead.
Although CPU pinning is a tuning option to optimize compute-intensive
workloads, it is incompatible with live-migration, which is critical for
Stratus to support long running jobs. Nova is configured to oversubscribe
hardware by a factor of 2x for vCPUs and 1x for memory. Neutron BGP dynamic
routing allows VMs to be assigned to campus routable RFC1918 IP subnets and
have their gateways automatically discovered by the network. They can then
communicate directly to other campus resources without any Network Address
Translation (NAT). Access to external resources is provided by an upstream NAT
device for outbound routes, but no ingress connections are allowed. In a
future version of OpenStack, this configuration will allow floating IP
addresses to bypass the network nodes to send packets directly to the
hypervisor host where the VM is currently located. The Horizon web interface,
which requires two-factor authentication to enter, binds these separate web
services into a single simple and intuitive portal where users can perform
self-service management of their infrastructure. For access to the OpenStack
command line interface or to connect to VMs, users must first SSH to the
Stratus Bastion server and authenticate with two factors.

##  3\. Secure VMs with MSI-Blessed Images

Figure 1. Secure VMs start with “MSI-blessed” images constructed by DiskImage
Builder. Cloud-Init and Puppet ensure that VMs are updated with the latest
security patches and configured for compliance at boot.

MSI provides a number of customized base images for users to boot when
creating a VM on Stratus. All images are Linux distributions such as Debian
and CentOS. The primary benefit from users leveraging MSI-provided customized
images, versus the vanilla images direct from Linux distributions, is that MSI
can enable basic security controls for users and have peace of mind that a VM
is protected until users opt-out (at their own risk). Generating images also
presents an opportunity to properly configure remote desktop (XRDP), and other
heavy-weight research tools such as Galaxy (Afgan et al., 2016), which is a
web-based workflow manager popular in the genomics and proteomics communities.

There are three components that contribute to the images that users interact
with as illustrated by Figure 1. First the image is built with DiskImage
Builder (Foundation, 2018), a tool created and supported by the OpenStack
community. DiskImage Builder composes images from a prescribed set of features
called Elements which each describe how to provision individual softwares,
configuration files, accounts, etc. DiskImage Builder is extensible and MSI
has generated a collection of custom Elements to setup tools like the Minio
Client and ensure that Remote Desktop, and Docker run at boot.

When a VM boots, a service runs at startup called Cloud-Init (Cloud-Init,
2018). Cloud-Init pulls metadata from the OpenStack Nova metadata server and
configures settings at first boot like IP address, user keys, etc. Cloud-init
also has flexible mechanisms (e.g., vendor data) for MSI to inject
instructions and settings. In this case, Cloud-Init forces VMs booted from
MSI-blessed images to apply system security updates. Security updates are the
first field of defense in ensuring the integrity of VMs and protecting all
users from potential compromise.

The final component contributing to secure a VM based on MSI-blessed images is
Puppet (Puppet, 2018). Puppet is provisioned in the image by a DiskImage
Builder element and runs as a daemon service on the instance after boot. It
periodically checks a master server for reference configurations and will
apply changes to the virtual machine to match the reference. This ensures a
base state on the virtual machine such as user- and group-permissions, and ssh
keys. Although Puppet is traditionally used to fully manage infrastructure,
this use-case is lightweight and intended for emergency patching when
necessary.

A strict naming convention is applied to images so users know what Elements go
into each image and the Linux distribution it is based on. For instance,
“Centos7_dbgap_blessed_desktop” describes an image based on the CentOS 7
distribution, with dbGaP related tools (e.g., gdc-client and aspera) for
pulling dbGaP data from the NIH, and Remote Desktop (“desktop”) running at
boot.

MSI currently maintains 9 images. The available Linux distributions are CentOS
7, Debian 9, and Ubuntu 16.04, and each distribution has Vanilla (i.e., the
original image provided by the the distribution) and dbGaP-blessed (MSI)
variants. For the CentOS 7 distribution, MSI also offers variants to integrate
Remote Desktop and Galaxy services.

New images are periodically released as major security issues are patched by
distributions, or new features are requested by users. The image build-process
is automated, and as new images are uploaded, old images are phased out.

##  4\. Stratus Storage

Figure 2. Stratus is sandboxed from other MSI services, but VMs have access to
three tiers of storage. Shields denote suitable for dbGaP data.

MSI selected the HPE Apollo 4200 server as the storage building block for
Stratus due to its high storage density, without being too large to assemble a
reasonable number of nodes for a viable Ceph cluster. Each node supports 24 8
TB hard drives, six 960 GB SSDs and two 800 GB NVMe flash cards in 2U of rack
space. Each individual hard drive or SSD is the basis of a single Ceph object
storage daemon (OSD). The NVMe flash provides high speed journal space for the
hard drive-based OSDs which provide the majority of the storage, reducing
their write latency. Each NVMe flash card supports journaling for 12 hard
drive OSDs. The smaller set of SSD-based OSDs contain indexes for the s3
object stores, as well as an optional high performance block storage pool.

The drives are configured on the server as individual single-drive RAID-0
units, rather than in “JBOD” pass-through mode, in order to leverage the
onboard cache of the RAID controllers. While the servers RAID controller
supports hardware encryption of the connected storage, it was decided to avoid
this in favor of Ceph’s existing support for linux standard LUKS encryption.
This was motivated by several reasons: a) this kept the setup as hardware-
agnostic as possible; b) testing showed a negligible performance impact
(roughly 1%); and c) it was felt that Ceph’s encryption key management was
superior, storing the keys away from the storage node in the Ceph monitor key-
value store.

Within Ceph, Stratus storage is partitioned into three areas: a) a 3x
replicated block device storage for Cinder volumes with 200 TB usable
capacity; b) an S3 object store for caching dbGaP data with 500 TB capacity;
and c) a persistent S3 object store for secure archive. Both b) and c) are 4:2
erasure-coded. Each of these storage classes uses its own set of underlying
ceph storage pools, on which quotas are enforced to maintain the desired
partitioning.

S3 object lifecycle (expiration) was added as a feature in Ceph Kraken, and is
essential to the design of the dbGaP Cache. Lifecycle policies are applied to
buckets and impact all objects within. Although this is still experimental,
the plan is to enforce a default expiration of 60-days, such that objects are
deleted following a First-In-First-Out policy. If greater than 80% of the
dbGaP Cache is filled, we gradually reduce this threshold until utilization
falls below the 80% mark. While a Least Recently Used lifecycle policy might
be seen as preferable, s3 storage semantics make this unlikely to be possible;
object creation being the only stored timestamp.

From the user perspective Stratus has three storage tiers, which are
illustrated in Figure 2. Volumes are the first tier, and the layer where
active data processing occurs. The second tier, secure archive, is the
combination of Stratus’ two secure object stores where users can target either
dbGaP Cache (“s3cache”) or persistent secure storage (“s3secure”). Users
interact with this tier using an s3 client; our recommendation being Minio
Client (“mc”), both for ease of use, and because recent versions support
server-side data encryption using customer-provided keys (SSE-C). A third
“general archive” tier, is MSI’s general-purpose S3 object store on a separate
dedicated Ceph cluster . It is through this general archive that data can be
brought in from other MSI resources. Stratus users can move data here when it
is sanitized and would not violate the data-use agreement to be stored in an
unprotected environment.

A typical workflow begins with using the Horizon web interface to boot a VM
with a small boot volume plus a large (e.g. 1 TB) workspace volume. Next, the
user connects to the VM via the Stratus Bastion server and transfers the
desired dbGaP data set from the NIH endpoints to the workspace volume for
analysis. The dbGaP data can optionally be copied into the s3 dbGaP cache for
later reuse without retransfer from NIH. Intermediate data produced during
analysis can also be stashed in the dbGaP Cache, while final results can be
stored in persistent secure storage. At any point during analysis, the user
can grow the size of the workspace, snapshot it, or detach it from one VM and
attach it elsewhere (e.g., if workflow stages are setup as separate VMs). When
analysis is complete, the user can post sanitized data to MSI’s general-
purpose S3 storage where they can access it from other MSI resources or share
with colleagues.

A similar workflow is adopted by users of the Galaxy machine image on Stratus.
In those cases, the Galaxy software stores data in the dbGaP Cache, and
performs read and write operations on a local volume with small quota ( 200 GB
up to 2 TB). As the quota is reached, data on the local volume is purged
automatically by Galaxy. The benefit of dbGaP Cache in this case is that users
can process significantly more data with a small VM footprint.

In future, several improvements to the storage subsystem may be investigated.
For example, Ceph’s new “bluestore” storage backend is now considered stable,
and offers potential performance improvements. The “Queens” release of
OpenStack adds native LUKS encryption of volumes, which would permit
compliance with more stringent data-use agreements by supporting customer-
specific encryption keys on active volumes. Another feature anticipated in
Queens is multi-attach volumes, which would enable users to attach their large
workspace volume to more than one VM simultaneously, and vastly simplify
multi-VM workflows. The presence of multi-attach could dramatically impact the
level of block storage utilization.

##  5\. HPC-like Performance

Once the Stratus production system was deployed, MSI invested a significant
amount of time to benchmark the system and assess the prospect of achieving
HPC-like performance on the cloud. As a baseline for HPC performance
expectations, the performance on Stratus matched against the performance on
Mesabi, MSI’s flagship HPC.

Figure 3. HPL peak performance on a bare metal HPC-node of Mesabi versus a
cloud-based VM running on Stratus.

Figure 3 compares the HPL (Petitet et al., [n. d.]) peak performance on a bare
metal node (Mesabi) and a 16 vCPU VM (Stratus) running on a 28 core Broadwell
node). Weak scaling makes the problem size per thread constant, so that the
amount of work per thread stays constant as the number of threads is
increased. Whereas strong scaling fixes the problem size, and that work is
spread over increasing numbers of threads, so the amount of work per thread
decreases as the number of threads is increased. Weak scaling used a matrix
size that used 80% of 2 GB/thread, which is calculated as
$N=\sqrt{(t*0.8*2*1024^{3})/8}$ where $t$ is the number of threads specified
for the benchmark. Note that the root is required because N represents one
side of a square matrix. The strong scaling runs used an N based on 80% of 2
GB. The choice of 2 GB/thread was arbitrary, but was chosen to fill most of
the available memory on Mesabi’s 64 GB nodes when the number of threads is
scaled up to 24, and also ensures the HPL calculation is not entirely resident
in cache.

The yellow and red curves show the strong and weak scaling peak performance
attained using HPL on a bare metal Haswell (dual 12 core 2.5 GHz Xeon E5-2680
v3) node (Mesabi), which has a peak performance of 40.0 GFLOPS/core. At 1
thread, peak performance of 91% shows very reasonable peak performance. Within
the HPC community, getting over 90% of peak for HPL performance is generally
considered excellent performance. The purple and blue curves show the strong
and weak scaling peak performance attained using HPL on a cloud based
Broadwell (dual 14 core 2.4 GHz Xeon E5-2680 v4) 16 vCPU VM (Stratus), which
has a peak performance of 46.4 GFLOPS/core. At 1 thread, peak performance of
87% shows an approximate 5% loss due to virtualization overhead relative to
the bare metal Haswell node. It is worth noting that for the Mesabi system,
lscpu reports an L3 cache of 30 MB for the system, but within the virtualized
system no L3 cache was reported. This missing tier of cache could certainly
degrade the performance by affecting the VM kernel scheduler, resulting in the
scheduler generating unnecessary inter-processor interrupts.

For the red weak scaling curve, the peak performance is maintained at about
91% as we increase the number of threads, indicating good scaling behaviour,
until 16 threads is reached, when performance reaches 86%. This reduction to
86% indicates that even the weak scaling case does not scale all the way up to
16 threads, likely due to memory/cache contention on the node. Likewise, the
blue weak scaling Stratus curve maintains good scaling behaviour at about 87%
peak performance out to 8 threads, then dips to 83% at 16 threads. The
performance loss for the weak scaling runs is shown by the black curve, i.e.
the Mesabi bare metal weak scaling peak performance minus the Stratus VM weak
scaling peak performance. It shows that weak scaling peak performance is
fairly consistent across the number of threads, at around 5% loss due to
virtualization.

For the yellow strong scaling curve, performance decreases as the number of
threads is increased, which is normal strong scaling behaviour. The strong
scaling behaviour for the Stratus VM shown by the purple curve is similar to
the bare metal Haswell node, except when the number of threads is 16, the
strong scaling peak performance reaches 59.4%, mentioned below. The
performance loss for the strong scaling runs is shown by the green curve, the
strong scaling analogue to the black curve. The performance loss for the
strong and weak scaling are consistent at about 5%, until 16 threads are
reached, when the strong scaling performance loss increases to 12%, while the
weak scaling performance loss decreases to 3%. At 16 threads, the strong
scaling on the virtualized resource is performing worse. However, this is not
of much concern since the shrinking problem size is already unable to saturate
the core, so any amount of noise in the solution is going to heavily impact
the performance. In fact, we see a significant increase in the standard
deviation as we increase the number of threads from 8 to 16.

Figure 4. Stratus storage write bandwidth for both 3x replicated block device
storage and erasure coded object storage.

Figure 4 compares the performance achieved by the 3x replicated pool to the
erasure-coded pools on the cluster. For this test, ten compute nodes were each
running the RADOS benchmark tool with eight concurrent operations. The 4:2
($k:m$) configuration of the erasure-coded pool uses 1.5 MB of space in
backend storage for every MB of data written to the pool. The 3x replicated
pool writes 3 MB of data to the HDD for every MB written. The replicated pool
outperforms the erasure-coded pool for small writes. In addition to the added
CPU load from object storage, every object stored on the erasure-coded pool
writes a total of 6 objects ($k+m$) to the HDD-backed OSDs. This leads to the
observed performance in the IOPs-limited regime. For large objects ($>1$ MB),
the erasure-coded pool finally outperforms the replicated pool. At the 4 MB
object size, the 3x replicated pool is writing 12,234 MB/s to the backend
storage ($4078*3$), and the erasure-coded pool is only writing at 9,465 MB/s
($6310*1.5$).

In terms of observed performance within VMs, a number of FIO (fio, 2018)
storage benchmarks on Stratus showed the Volume storage capable of achieving
270 MB/s in write bandwidth. This performance is impressively 12.5% higher
than the 240 MB/s write bandwidth achieved by FIO on Mesabi’s high performance
filesystem. Obviously, there are major scaling differences between the two
systems, and they have been architected for different purposes. However,
Stratus certainly demonstrates HPC-like performance in storage as well as
compute.

##  6\. Stratus Subscription

Due to the specialized nature of Stratus, the small size, and the potential
need for growth, it was decided that it should be offered as a paid service.
An annual subscription fee was developed using Cornell’s Red Cloud (Cloud,
2017) as a model. The base subscription costs less than a thousand dollars per
year, but allots users 16 vCPUs, 32 GB RAM, 2 TB of Volume storage, and access
to the shared dbGaP Cache for short-term data storage. Users are also granted
access to a number of MSI-blessed CentOS and Ubuntu base images, and sudo
permissions to install software on personal virtual machines. Additional
vCPUs, Volume storage, and persistent object storage can be added à la carte
in 1 vCPU or 1 TB/year increments over the base subscription. Subscription
pricing is calculated based on zero-profit cost recovery of the hardware,
including all network switches, and all control-, storage-, and compute-nodes.
Staff time for operational/administrative tasks and user support are also
factored into the price. Network traffic is excluded. Note that the
subscription ensures cost-recovery of the system at 85% utilization and allows
for continuous expansion as popularity grows. The presence of a fee also
ensures that most (not all) users will exercise good cloud hygiene and
actively clean-up/scale-in their resources when not needed.

This new model has three benefits: a) as the popularity of Stratus grows, cost
recovery ensures funds will be available to scale the cluster; b) it
encourages casual users to continue working on traditional HPC systems where
MSI provides cycles and limited storage for free; and c) it encourages serious
users to practice good cloud hygiene by securing the space they are paying
for, and to clean up after themselves. Although the base subscription is
generous, all VM boot volumes, data volumes and snapshots count against the 2
TB Volume allocation, and that space can fill quickly.

Subscriptions are mapped into OpenStack using built-in service quotas.
OpenStack attaches quotas to individual projects, with toggles for vCPU count,
size of volumes and snapshots, RAM, etc. Object storage quotas are applied at
the bucket-level on Ceph.

##  7\. GDS Compliance

Most cloud providers absolve themselves from responsibility of protecting
users, focusing instead on the minimum effort necessary for regulatory
compliance within the underlying hardware, and leaving users to individually
manage and vet whatever is run on top of the cloud. This makes sense in a
self-service Infrastructure-as-a-Service model, but creates a potentially
dangerous situation for novice users with protected data. Stratus was built
with an MSI-first mentality, prioritizing security of the hardware, OpenStack,
and Ceph. However, as an academic unit supporting computationally intensive
research at the University of Minnesota since 1984, MSI has a culture of going
the extra mile to train, protect, and assist. In this case, a fair amount of
effort also went into sandboxing user VMs and providing sane defaults for
security that users can opt-out of at their own risk. This included following
the best practices guide for GDS data (NIH, 2015) as a checklist. All bullets
therein were treated as required controls. Note that the NIH best practices go
beyond what most large research computing facilities can or need to do.

Figure 5. Compliance is met with an MSI-first policy that focuses on securing
MSI, OpenStack, and Ceph. Users are protected with sane defaults and network
security.

Figure 5 illustrates a sampling of protections for MSI and users as two trees.
While outside the scope of this document to exhaustively detail each measure,
there is clearly a bias toward protecting MSI. The MSI tree on the left has
two levels of nodes. The first level are five significant controls mentioned
by the best practices guide (i.e., Encryption, Logging, Principle of Least
Privilege, Access Control, and Change Management), and the second level
summarizes some of the ways we have addressed them. The majority of the GDS
controls are intangible to users, but represent substantial administrative and
operational staff effort.

Table 1. Firewall restrictions for Stratus

Scope | Ingress | Egress
---|---|---
World | None | ALL
UMN | 443, 8443 with SSL1 | ALL
Stratus | ALL via Stratus Bastion1 | ALL
Project | ALL | ALL

  * 1

requires Security Group exceptions

The USER tree offers measures that users are aware of. A few relate to GDS
controls (e.g., two-factor authentication), but others go above and beyond.
The most impactful measures deal with network traffic. In most clouds, users
get unrestricted traffic in and out of VMs. Table 1 presents firewall
rules—both ingress and egress—for VMs on Stratus. In all cases, egress traffic
is unrestricted, but ingress is only open between VMs within the same project,
or from the Stratus Bastion (so long as Security Group exceptions exist).
Traffic is blocked on all ports from the world, but when it originates from
the campus network, ports 443 and 8443 are accessible with caveats: a) SSL is
required on both, and b) users must create a Security Group exception to opt-
out of the default network protection. These rules sandbox Stratus VMs away
from other MSI services, the University, and the world. For VMs requiring
world-wide access, MSI recommends the NIH cloud pilots (FireCloud, 2017; for
Systems Biology, 2017; Bridges, 2017).

Encryption is essential for data obfuscation and protection, and it is
leveraged on many levels including full-disk encryption of storage disks,
encrypted network traffic, and user-encrypted data at rest. Access controls
limit who can interact with systems, software, and data, and also establish
how users are properly identified (e.g., with two-factors). Principle of Least
Privilege prevents administrators and users from running software, accessing
data, or completing any other tasks for which they are not approved. Logging
is mandatory and encompasses tracking account access, data access, software
installations, account escalations, violations, etc. Logging also extends to
parsing, management, and protection of log integrity. Lastly, Change
Management tracks modifications to Stratus including web service
configurations, software updates, as well as physical hardware changes.

##  8\. Conclusion

Technical details of Stratus, a subscription-based Infrastructure as a Service
for research computing, were presented herein. Stratus satisfies the immediate
need for an environment to securely process dbGaP data at the University of
Minnesota, and meets or exceeds the requirements laid out by the NIH Genomic
Data Sharing Policy.

Future work will consider compliance with other regulations, as well as
supporting use cases with unrestricted data types. This may include cluster
solutions for processing dbGaP data (e.g., projects like Kubernetes). Also,
MSI is currently piloting a general-use partition within Stratus for non-dbGaP
projects and open-access data types. Projects under the general space are
separated from dbGaP users by a second neutron provider, and subject to less
stringent access controls and firewall rules.

## References

  * (1)
  * Afgan et al. (2016) Enis Afgan, Dannon Baker, Marius van den Beek, Daniel Blankenberg, Dave Bouvier, Martin Čech, John Chilton, Dave Clements, Nate Coraor, Carl Eberhard, Björn Grüning, Aysam Guerler, Jennifer Hillman-Jackson, Greg Von Kuster, Eric Rasche, Nicola Soranzo, Nitesh Turaga, James Taylor, Anton Nekrutenko, and Jeremy Goecks. 2016\.  The Galaxy platform for accessible, reproducible and collaborative biomedical analyses: 2016 update.  _Nucleic Acids Res._ 44, W1 (2016), W3–W10.
  * Bell et al. (2015) T Bell, B Bompastor, S Bukowiec, J Castro Leon, M K Denis, J van Eldik, M Fermin Lobo, L Fernandez Alvarez, D Fernandez Rodriguez, A Marino, B Moreira, B Noel, T Oulevey, W Takase, A Wiebalck, and S Zilli. 2015.  Scaling the CERN OpenStack cloud.  _Journal of Physics: Conference Series_ 664, 2 (2015), 022003\.
  * Bridges (2017) Seven Bridges. 2017\.  Cancer Genomics Cloud.  (June 2017).  Retrieved June 1, 2017 from http://www.cancergenomicscloud.org/
  * Cloud (2017) Red Cloud. 2017\.  Cornell University Center for Advanced Computing (CAC).  (June 2017).  Retrieved June 1, 2017 from https://www.cac.cornell.edu/redcloud
  * Cloud-Init (2018) Cloud-Init. 2018\.  (March 2018).  Retrieved March 25, 2018 from http://cloudinit.readthedocs.io/en/latest/
  * fio (2018) fio. 2018.  Flexible I/O Tester.  (March 2018).  Retrieved March 26, 2018 from http://fio.readthedocs.io/en/latest/fio_doc.html
  * FireCloud (2017) FireCloud. 2017\.  FireCloud Home Page.  (June 2017).  Retrieved June 1, 2017 from https://software.broadinstitute.org/firecloud/
  * for Systems Biology (2017) Institute for Systems Biology. 2017\.  Cancer Genomics Cloud.  (June 2017).  Retrieved June 1, 2017 from http://cgc.systemsbiology.net/
  * Foundation (2018) OpenStack Foundation. 2018\.  DiskImage-Builder Documentation.  (March 2018).  Retrieved March 25, 2018 from https://docs.openstack.org/diskimage-builder/latest/
  * Heath et al. (2014) Allison P Heath, Matthew Greenway, Raymond Powell, Jonathan Spring, Rafael Suarez, David Hanley, Chai Bandlamudi, Megan E McNerney, Kevin P White, and Robert L Grossman. 2014\.  Bionimbus: a cloud for managing, analyzing and sharing large genomics datasets.  _Journal of the American Medical Informatics Association : JAMIA_ 21, 6 (11 2014), 969–975.
  * Inc. (2017) CancerCollaboraty Inc. 2017\.  Cancer Genome Collaboratory from the Ontario Institute of Cancer Research.  (June 2017).  Retrieved June 1, 2017 from https://www.cancercollaboratory.org/
  * Mailman et al. (2007) Matthew D. Mailman, Michael Feolo, Yumi Jin, Masato Kimura, Kimberly Tryka, Rinat Bagoutdinov, Luning Hao, Anne Kiang, Justin Paschall, Lon Phan, Natalia Popova, Stephanie Pretel, Lora Ziyabari, Moira Lee, Yu Shao, Zhen Y. Wang, Karl Sirotkin, Minghong Ward, Michael Kholodov, Kerry Zbicz, Jeffrey Beck, Michael Kimelman, Sergey Shevelev, Don Preuss, Eugene Yaschenko, Alan Graeff, James Ostell, and Stephen T. Sherry. 2007.  The NCBI dbGaP database of genotypes and phenotypes.  _Nature genetics_ 39, 10 (01 Oct. 2007), 1181–1186.  https://doi.org/10.1038/ng1007-1181
  * NIH (2015) NIH. 2015.  NIH Security Best Practices for Controlled-Access Data Subject to the NIH Genomic Data Sharing (GDS) Policy.  (March 2015).  Retrieved June 1, 2017 from https://www.ncbi.nlm.nih.gov/projects/gap/pdf/dbgap_2b_security_procedures.pdf
  * OpenStack (2017) OpenStack. 2017\.  Open Source Cloud Computing Software.  (June 2017).  Retrieved June 1, 2017 from https://www.openstack.org/
  * Petitet et al. ([n. d.]) A. Petitet, R. C. Whaley, Jack Dongarra, and A. Cleary. [n. d.].  HPL - A Portable Implementation of the High-Performance Linpack Benchmark for Distributed-Memory Computers.  ([n. d.]).
  * Puppet (2018) Puppet. 2018.  Puppet Documentation.  (March 2018).  Retrieved March 25, 2018 from https://puppet.com/docs
  * Security (2017) Duo Security. 2017\.  The Trusted Access Company.  (May 2017).  Retrieved June 1, 2017 from https://duo.com
  * Stewart et al. (2015) Craig A. Stewart, Timothy M. Cockerill, Ian Foster, David Hancock, Nirav Merchant, Edwin Skidmore, Daniel Stanzione, James Taylor, Steven Tuecke, George Turner, Matthew Vaughn, and Niall I. Gaffney. 2015.  Jetstream: A Self-provisioned, Scalable Science and Engineering Cloud Environment. In _Proceedings of the 2015 XSEDE Conference: Scientific Advancements Enabled by Enhanced Cyberinfrastructure_ _(XSEDE ’15)_. ACM, New York, NY, USA, Article 29, 8 pages.  https://doi.org/10.1145/2792745.2792774
  * Weil et al. (2006) Sage A. Weil, Scott A. Brandt, Ethan L. Miller, Darrell D. E. Long, and Carlos Maltzahn. 2006\.  Ceph: A Scalable, High-performance Distributed File System. In _Proceedings of the 7th Symposium on Operating Systems Design and Implementation_ _(OSDI ’06)_. USENIX Association, Berkeley, CA, USA, 307–320.  http://dl.acm.org/citation.cfm?id=1298455.1298485