# Domain Generalization via Inference-time Label-Preserving Target Projections

Prashant Pandey  1 Mrigank Raman  1 Sumanth Varambally  1 Prathosh AP1
1IIT Delhi
{bsz178495 mt1170736  mt6170855<EMAIL_ADDRESS>
###### Abstract

Generalization of machine learning models trained on a set of source domains
on unseen target domains with different statistics, is a challenging problem.
While many approaches have been proposed to solve this problem, they only
utilize source data during training but do not take advantage of the fact that
a single target example is available at the time of inference. Motivated by
this, we propose a method that effectively uses the target sample during
inference beyond mere classification. Our method has three components - (i) A
label-preserving feature or metric transformation on source data such that the
source samples are clustered in accordance with their class irrespective of
their domain (ii) A generative model trained on the these features (iii) A
label-preserving projection of the target point on the source-feature manifold
during inference via solving an optimization problem on the input space of the
generative model using the learned metric. Finally, the projected target is
used in the classifier. Since the projected target feature comes from the
source manifold and has the same label as the real target by design, the
classifier is expected to perform better on it than the true target. We
demonstrate that our method outperforms the state-of-the-art Domain
Generalization methods on multiple datasets and tasks.

##  1 Introduction

Domain shift refers to the existence of significant divergence between the
distributions of the training and the test data [40]. This causes the machine
learning models trained only on the training or the source data to perform
poorly on the test or target data. A naive way of handling this problem is to
fine-tune the model with new data which is often infeasible because of the
difficulty in acquiring labelled data for every new target domain. The class
of Domain Adaptation (DA) methods [41, 9, 14, 39, 26, 4, 31, 33, 34] tackle
this problem by utilizing the (unlabeled) target data to minimize the domain
shift; however they cannot be used when unlabeled target data is unavailable.

Domain generalization (DG) [30, 21, 10, 22, 1, 23], on the other hand, views
the problem from the following perspective: how to make a model trained on
single or multiple source domains generalize on completely unseen target
domains. These methods do so via (i) learning feature representations that are
invariant to the data domains using methods such as adversarial learning [24,
25], (ii) simulating the domain shift while learning through meta-learning
approaches [22, 1], and (iii) augmenting the source dataset with synthesized
data from fictitious target domains [43, 47]. These methods have been shown to
be effective in dealing with the problem of domain shift. However, most of the
existing methods do not utilize the test sample from the target distribution
available at the time of inference beyond mere classification. On the other
hand, it is a common experience that when humans encounter an unseen object,
they often relate it to a previously perceived similar object.

Motivated by this intuition, in this paper, we make the following
contributions towards addressing the problem of DG: (a) Given samples from
multiple source distributions, we propose to learn a source domain invariant
representation that also preserves the class labels. (b) We propose to
‘project’ the target samples to the manifold of the source-data features
before classification through an inference-time label-preserving optimization
procedure over the input of a generative model (learned during training) that
maps an arbitrary distribution (Normal) to the source-feature manifold. (c) We
demonstrate through extensive experimentation that our method achieves new
state-of-the-art performance on standard DG tasks while also outperforming
other methods in terms of robustness and data efficiency.

##  2 Prior Work

Meta-learning : Meta-learning methods aim to improve model robustness against
unseen domains by simulating domain shift during training. This is done by
splitting the training set into a meta-train and meta-test set. [22] provide a
general framework for meta-learning-based DG, where model parameters are
updated to minimize loss over the meta-train and meta-test domains in a
coordinated manner. [1] propose a pre-trained regularizer network which is
used to regularize the learning objective of a domain-independent task
network. [7] use a common feature extractor backbone network in conjunction
with several domain-specific aggregation modules. An aggregation over these
modules is performed during inference to predict the class label. [23] train
separate feature extractors and classifiers on each of the source domains and
minimize the loss on mismatched pairs of feature extractors and classifiers to
improve model robustness. [15] utilise a probabilistic meta-learning model in
which classifier parameters shared across domains are modeled as
distributions. They also learn domain-invariant representations by optimizing
a variational approximation to the information bottleneck. Since meta-learning
methods are only trained on the simulated domain shifts, they might not always
perform well on target domains that are not ‘covered’ in the simulated shifts.
Data augmentation: Augmenting the dataset with random transformations improves
generalization [13]. Commonly used augmentation techniques include rotation,
flipping, random cropping, random colour distortions, amongst others. [38] use
gradients from a domain classifier to perturb images. However, these
perturbations might not be reflective of practically observed domain shift.
[47] aim to address this issue using an adversarial procedure to train a
transformation network to produce an image translation that aims to generate
novel domains while retaining class information. [46] generate images from
pseudo-novel domains with an optimal transport based formulation while
preserving semantic information with cycle-consistency and classification
losses. [36] solve the problem of single-source DG by creating fictitious
domains using Wasserstein Auto-Encoders in a meta-learning framework. While
these generated domains differ significantly from the source domains, they
potentially do not reflect practical domain differences.
Domain-invariant representations: Another common pervasive theme in domain
generalization literature is transforming the source data into a lower-
dimensional ‘feature’ space that is invariant to domains but retains the
discriminative class information; these features are used for classification.
[10] learn an auto-encoder to extract domain invariant features by
reconstructing inter and cross domain images. [24] use adversarial auto-
encoders to align the representations from all the source domains to a
Laplacian prior using adversarial learning procedure. [6] employ episodic
training to simulate domain shift while minimizing a global class-alignment
loss and local sample-clustering objective to cluster points class-wise. [15]
learn a kernel function that minimizes mean domain discrepancy and intra-class
scatter while maximizing mean class discrepancy and multi-domain between-class
scatter. [35] propose a low-rank decomposition on the final classification
layer to identifiably learn common and specific features across domains. [37]
use domain-specific normalizations to learn representations that are domain-
agnostic and semantically discriminative.

All of the above methods require domain labels, which might not always be
viable. [5] aim to solve the problem of DG without domain labels by learning
an auxiliary task of solving jigsaw puzzles. The idea is that features learned
from such an auxiliary task will be invariant of the domains. [27] first
assigns pseudo-labels inferred by clustering the domain discriminative
features. They train a domain classifier against these pseudo-labels, which is
further used to adversarially train a domain-invariant feature extractor. [29]
use a semantic alignment loss as an additional regularizer while training the
classifier for domain invarient feature learning. [16] iteratively locate
dominant features activated on the training data using layer gradients, and
learn useful features by self-challenging. [44] learn how to generalize across
domains by simultaneously providing extrinsic supervision in the form of a
metric learning task and intrinsic supervision in terms of a self-supervised
auxiliary task

##  3 Proposed Method

Figure 1: A) We design a function $f$ (neural network $f_{\theta}$) to learn a
label-preserving metric that produces a similarity score of 1 when the ground
truth labels (given by function $g$) between a pair of images match and -1
otherwise. The function ‘sim’ refers to the cosine similarity function. B) $f$
is implemented using a neural network $f_{\theta}$. During training, the
examples from the source domains are utilized to create a source manifold
$\mathcal{Z}_{s}$ using loss $\mathcal{L_{A}}$ such that the features on the
manifold are implicitly clustered to preserve the labels of examples. C) A
classifier $\mathcal{C}_{\psi}$ and a generative model $G_{\phi}$ are trained
on the label-preserving features from manifold $\mathcal{Z}_{s}$ such that
$G_{\phi}$ learns to map a Gaussian vector $\mathbf{u}$ to a point on the
manifold $\mathcal{Z}_{s}$. D) During inference, $f_{\theta^{\ast}}$ projects
target $\mathbf{x_{t}}$ to a point $\mathbf{z_{t}}$ on the label-preserving
feature space. We propose an inference-time procedure to project the target
feature to a point $\mathbf{z_{t}^{\ast}}$ on the source manifold which is
finally classified to predict its label $\mathbf{\hat{y}_{t}}$.
$\theta^{\ast}$, $\psi^{\ast}$ and $\phi^{\ast}$ indicate that the weights of
their corresponding networks are fixed during inference.

###  3.1 Problem Setting and Method Overview

Let $\mathcal{X}$ and $\mathcal{Y}$ respectively denote data and the label
spaces. Let $\mathcal{H}$ be the space of hypotheses where each hypothesis $h$
in $\mathcal{H}$ maps points from $\mathcal{X}$ to a label in $\mathcal{Y}$. A
domain is defined by the tuple $(\mathcal{D},g_{\mathcal{D}})$ where
$\mathcal{D}$ is a probability distribution over $\mathcal{X}$ and
$g_{\mathcal{D}}$ where $g_{\mathcal{D}}:\mathcal{X}\to\mathcal{Y}$ is a
function that assigns ground-truth labels. It is generally assumed that the
ground-truth labeling function $g$ is same across all the domains. Domain
Generalization is defined as the task where there are a total of $N$ domains
out of which $|S|$ are source and $|T|$ are target domains. The source and
target domains are respectively denoted by $\mathcal{D}_{i}^{S}$, $i\in[|S|]$
and $\mathcal{D}_{j}^{T}$, $j\in[|T|]$. The objective is to train a classifier
on the source domains that predicts well on the target domain when the target
samples are not available during training.

The motivation for our method comes from the following observation: DG methods
that learn domain invariant representations do so only using the source data.
Therefore, classifiers trained on such representations are not guaranteed to
perform well on target data that is outside the source data manifolds. Hence,
performance on the target data can be improved if the target sample is
projected on to the manifold of the source features such that the ground-truth
label is preserved, before classification. To this end, we propose a three-
part procedure for domain generalization:
1\. (Training): Learn a label-preserving domain invariant representation using
source data. We first transform the data from multiple source domains into a
space where they are clustered according to class labels, irrespective of the
domains and build a classifier on these features.
2\. (Training): Learn to generate features from the domain invariant feature
manifold created from the source data by constructing a generative model on
it.
3\. (Inference): Given a test target sample, project it on to the source-
feature manifold in a label-preserving manner. This is done by solving an
inference-time optimization problem on the input space of the aforementioned
generative model. Finally, classify the projected target feature.

Note that the parameters of all the networks involved (Feature extractor,
generative model and the classifier) are learned only using the source data
and fixed during inference. Thus, our approach is well within the realm of DG
despite a label-preserving optimization problem being solved for every target
sample during inference. The overall procedure is depicted in Figure 1. In the
subsequent sections, we describe all the aforementioned components in detail.

###  3.2 Label-Preserving Transformation

####  3.2.1 Domain invariant Features

The first step in our method is to learn a feature (metric) space such that
the input images are clustered in accordance with their labels (classes)
irrespective of their domains. We explicitly construct such a feature space
$\mathcal{F}$ with a function $f:\mathcal{X}\to\mathcal{F}$ by solving the
following optimization problem.  ~f∑_j=1^N∑_i=1^N (-1)^α(i,j)∥~f(x_i) -
~f(x_j)∥^2 ∥~f(x_k)∥=1 ∀k ∈[N]

$\displaystyle\alpha(i,j)=\begin{cases}0&g(\mathbf{x}_{i})=g(\mathbf{x}_{j})\\\
1&\text{otherwise}\end{cases}$

The function $f$ is learned such that when a pair of source samples have the
same ground truth labels, the norm of the difference between their
representations under $f$ is low irrespective of their domain membership and
high when they belong to different classes. Under this formulation, the
features in the space $\mathcal{F}$ will be ‘clustered’ in accordance with
their class labels, irrespective of the domains. In fact, it can be shown that
the aforementioned $f$ minimizes the $\mathcal{H}$-divergence [2] between any
two pairs of domains.

######  Proposition 1.

The label-preserving transformation $f$ defined in Eq. 1, reduces the
$\mathcal{H}$-divergence between any two pair of domains on which it is
learned (Proof in Appendix).

In summary, the proposed feature transformation merges multiple source domains
into a single feature domain such that images that have the same labels
cluster into a group in the feature space.

####  3.2.2 Learning the $f$-function

We propose to learn ${f}$ by parameterizing it with a deep neural network,
$f_{\theta}$. It is easy to see that the objective function in Eq. 3.2.1
reduces to an optimization of cosine-similarity between the pair of samples
$f_{\theta}(\mathbf{x}_{i})$ and $f_{\theta}(\mathbf{x}_{j})$. Suppose
$\mathbf{z}_{i}=f_{\theta}(\mathbf{x}_{i})$ and
$\mathbf{z}_{j}=f_{\theta}(\mathbf{x}_{j})$ represent the feature vectors of
inputs, the cosine-similarity $\mathbf{s}_{i,j}$ between $\mathbf{z}_{i}$ and
$\mathbf{z}_{j}$ is given by,

$\mathbf{s}_{i,j}=\frac{\mathbf{z}_{i}\cdot\mathbf{z}_{j}}{\left\lVert\mathbf{z}_{i}\right\rVert\left\lVert\mathbf{z}_{j}\right\rVert}$
(1)

Note that the optimization problem in Eq. 3.2.1 seeks $\mathbf{s}_{i,j}$ to be
high when the labels are same and low when they are different (denoted by the
$\alpha(i,j)$ term in Eq. 3.2.1). Thus, we first translate $\mathbf{s}_{i,j}$
into logits for a sigmoid activation and use binary cross entropy on the
generated probabilities. However, since $-1\leq\mathbf{s}_{i,j}\leq 1$, we
scale it with a small positive constant $\tau$ (typically 0.1) to widen the
range of the generated logits. Mathematically, we can write
$\mathbf{p}_{i,j}=\text{sigmoid}(\mathbf{s}_{i,j}/\tau)$. Under this
formulation, one can treat $\mathbf{p}_{i,j}$ as a similarity score which
should be 1 if ($\mathbf{x}_{i},\mathbf{x}_{j}$) have the same label
($\alpha(i,j)=0$) and 0 otherwise. Thus, we finally use a binary cross entropy
loss $\mathcal{L_{A}}$ between $\mathbf{p}_{i,j}$ and $1-\alpha(i,j)$, to
train the $f_{\theta}$ network.

###  3.3 Inference-time Target Projections

In a DG setting, the feature transformation mentioned in the previous sections
is learned on the source domains. Let $\mathcal{Z}_{s}$ denote the manifold
created by learning such features using the source data. The classifier
$\mathcal{C}_{\psi}$ is trained on points from the source data feature
manifold $\mathcal{Z}_{s}$. Because of the domain shift, the feature
$f(\mathbf{x_{t}})$ corresponding to a test target point $\mathbf{x_{t}}$
might not fall on the source data feature manifold $\mathcal{Z}_{s}$. This
causes the classifier to fail on the target feature $f(\mathbf{x_{t}})$. To
address this issue, we propose to project or ‘push’ the target feature onto
$\mathcal{Z}_{s}$ while preserving the ground truth label of $\mathbf{x_{t}}$,
so that the classifier can better discern the class label of $\mathbf{x_{t}}$.
We propose to accomplish such a label-preserving projection by solving an
inference-time optimization problem on the input space of a generative model
trained on $\mathcal{Z}_{s}$. For convenience, we define a function
$\tilde{g}:\mathcal{F}\to\mathcal{Y}$ such that $\tilde{g}(f(x))=g(x)$

####  3.3.1 Generating the Source-Feature Manifold $\mathcal{Z}_{s}$

Once the label-preserving source-feature manifold $\mathcal{Z}_{s}$ is
obtained through $f_{\theta}$, we build a generative model on it. That is, a
transformation from samples from an arbitrary distribution, such as Normal
distribution, to the source-data feature manifold $\mathcal{Z}_{s}$ is
learned. We choose two state-of-the-art neural generative models for this
purpose: (a) Variational Auto-Encoder (VAE) [18]: In this setting, a VAE is
trained using the source-data features $\mathbf{z}$, by encoding them to
produce the latent space $\mathbf{u}\sim\mathcal{N}(0,I)$. A decoder
$G_{\phi}$ reconstructs (generates) the source-feature manifold
$\mathcal{Z}_{s}$ by minimizing a regularized norm-based loss. (b) Generative
Adversarial Networks (GAN) [11]: Here, a GAN is trained with a generator
network $G_{\phi}$ that maps an arbitrary latent space
$\mathbf{u}\sim\mathcal{N}(0,I)$ to the source-feature manifold
$\mathcal{Z}_{s}$. Note that these generative models are trained on the
source-features alone and fixed during the inference procedure. We denote the
trained generative model by $G_{\phi^{\ast}}$.

####  3.3.2 Label-preserving Projections

The final component of our method is to project the target features on to the
source-feature manifold during inference. It is to be noted that the
transformation $f_{\theta}$ is constructed such that when a pair of samples
have zero distance in that space, they have same ground truth label. That is,
if $\lVert f_{\theta}(\mathbf{x}_{1})-f_{\theta}(\mathbf{x}_{2})\rVert=0$,
then $g(\mathbf{x}_{1})=g(\mathbf{x}_{2})$. We exploit this property and solve
a (per-sample) optimization procedure on the input space of the generative
model $G_{\phi^{\ast}}(\mathbf{u})$ to obtain the target-feature projection.

Let $\mathbf{z_{t}}=f_{\theta}(\mathbf{x_{t}})$ denote the feature vector
corresponding to a test target sample $\mathbf{x_{t}}$. Our goal is to find
the projected target feature in the source-feature manifold
$\mathbf{z_{t}^{*}}\in\mathcal{Z}_{s}$ that has the same ground-truth label as
that of $\mathbf{z_{t}}$. By construction of $f_{\theta}$ (Eq. 3.2.1), the
cosine distance between $\mathbf{z_{t}}$ and $\mathbf{z_{t}^{*}}$ should be
low if their ground truth labels are to match. Based on this, we devise the
following optimization problem on the input space of $G_{\phi^{*}}$ to find
$\mathbf{z_{t}^{*}}$:

$\displaystyle\mathcal{L_{S}}$
$\displaystyle=\Bigg{[}1-{\frac{\mathbf{z_{t}}\cdot
G_{\phi^{\ast}}(\mathbf{u})}{\left\lVert\mathbf{z_{t}}\right\rVert\left\lVert
G_{\phi^{\ast}}(\mathbf{u})\right\rVert}}\Bigg{]}$ (2)
$\displaystyle\mathbf{u^{*}}=\operatorname*{arg\,min}_{{\mathbf{u}}}\mathcal{L_{S}}$
(3) $\displaystyle\mathbf{z_{t}^{*}}=G_{\phi^{\ast}}(\mathbf{u^{*}})$ (4)

The objective function in Eq. 3 seeks to find the projected target feature
$\mathbf{z_{t}^{*}}$ (via $\mathbf{u^{*}}$ from Eq. 4) whose cosine distance
is least from the true target feature $\mathbf{z_{t}}$.

####  3.3.3 Analysis and Implementation

In this section, we analyze the performance of the classifier on account of it
using the projected target instead of the real target features. We start by
upper bounding the expected value of the misclassification when the projected
target is used in the classifier, in the below preposition.

######  Proposition 2.

The expected misclassification rate obtained with a classifier $h$ when the
projected target is used instead of the true target, obeys the following
upper-bound:

$\displaystyle\mathbb{E}_{(\mathcal{D}^{T},\mathcal{D}^{T^{\ast}})}\left|\tilde{g}(\mathbf{z_{t}})-h(\mathbf{z_{t}^{*}})\right|\leq$
$\displaystyle\underbrace{\mathbb{E}_{\mathcal{D}^{T^{\ast}}}\left|\tilde{g}(\mathbf{z_{t}^{*}})-h(\mathbf{z_{t}^{*}})\right|}_{\text{i}}+\underbrace{\mathbb{E}_{(\mathcal{D}^{T},\mathcal{D}^{T^{\ast}})}\left|\tilde{g}(\mathbf{z_{t}})-\tilde{g}(\mathbf{z_{t}^{*}})\right|}_{\text{ii}}$
(5)

where $\mathcal{D}$ and $\mathcal{D}^{T^{\ast}}$ respectively denote the true
and the projected target distributions respectively (Proof in Appendix).

The term (i) in Eq. 19 is the misclassification error of $h$ on the projected
target and term (ii) is the difference between the ground truth labels of the
true and the projected targets. Given that our overall objective is to
minimize the LHS of Eq. 19, the optimization procedure in the previous section
aims to minimize term (ii) while term (i) is expected to be less since the
projected target is expected to lie on the source-feature manifold
$\mathcal{Z}_{s}$ on which the classifier is trained.

In the implementation, during inference, we optimize the objective in Eq. 3
(reducing term (ii) in Eq. 19) by gradient descent. A discussion on the choice
of stopping criteria can be found in Section 4.7. The training and the
inference procedures are detailed in Algorithm 1 and shown in Figure 1.

Training

Input: Batch size $N$, learning rate $\eta$, source data
$\\{(\mathbf{x}_{k},\mathbf{y}_{k})\\}$;

Result: Trained $f_{\theta^{\ast}}$, generative model $G_{\phi^{\ast}}$,
classifier $\mathcal{C_{\psi^{\ast}}}$

for  _ sampled minibatch $\\{(\mathbf{x}_{k},\mathbf{y}_{k})\\}^{N}_{k=1}$ _
do

for  _all $i\in\\{1,...N\\}$ and $j\in\\{1,...N\\}$ _ do

$(\mathbf{z}_{i},\mathbf{z}_{j})\leftarrow(f_{\theta}(\mathbf{x}_{i}),f_{\theta}(\mathbf{x}_{j}))$

$\mathbf{s}_{i,j}\leftarrow\frac{\mathbf{z}_{i}\cdot\mathbf{z}_{j}}{\lVert\mathbf{z}_{i}\rVert\lVert\mathbf{z}_{j}\rVert}$

$\mathbf{y}_{i,j}=\delta_{\mathbf{y}_{i},\mathbf{y}_{j}}$

$\mathbf{p}_{i,j}\leftarrow\text{sigmoid}(\mathbf{s}_{i,j}/\tau)$

end for

$\mathcal{L_{\mathcal{A}}}\leftarrow\frac{1}{N^{2}}\sum_{i=1}^{N}\sum_{j=1}^{N}\text{BCELoss}(\mathbf{p}_{i,j},\mathbf{y}_{i,j})$

$\theta\leftarrow\theta-\eta\nabla_{\theta}\mathcal{L_{A}}$

end for

Train $G_{\phi}$ and $\mathcal{C_{\psi}}$ on
$\\{(f_{\theta}(\mathbf{x}_{k}),\mathbf{y}_{k})\\}$.

Inference

Input: Target image $\mathbf{x_{t}}$, trained network $f_{\theta^{\ast}}$,
generative model $G_{\phi^{\ast}}$, classifier $\mathcal{C_{\psi^{\ast}}}$,
iteration rate $\beta$;

Result: Target label $\mathbf{\hat{y}_{t}}$

$\mathbf{z_{t}}\leftarrow f_{\theta^{\ast}}(\mathbf{x_{t}})$;

Sample $\mathbf{u}$ from $\mathcal{N}(0,I)$;

Initialize $U$ and $L$ as empty lists

for  _all $i\in\\{1,...M\\}$_ do

$\mathbf{z}\leftarrow G_{\phi^{\ast}}(\mathbf{u})$

$\mathcal{L_{S}}\leftarrow
1-\frac{\mathbf{z}\cdot\mathbf{z_{t}}}{\left\lVert\mathbf{z}\right\rVert\left\lVert\mathbf{z_{t}}\right\rVert}$

$\mathbf{u}\leftarrow\mathbf{u}-\beta\nabla_{\mathbf{u}}\mathcal{L_{S}}$

$(U[i],L[i])\leftarrow(\mathbf{u},\mathcal{L_{S}})$

end for

Smoothen $L$ by window-averaging

$\mathbf{u}^{\ast}\leftarrow U[\operatorname*{arg\,max}_{{i}}\delta^{2}L]$

$\mathbf{\hat{y}_{t}}\leftarrow\mathcal{C_{\psi^{\ast}}}(G_{\phi^{\ast}}(\mathbf{u}^{\ast}))$

Algorithm 1 Inference-time Target Projections

##  4 Experiments and Results

We have considered four standard DG datasets - PACS [21], VLCS [8], Office-
Home [42] and Digits-DG [47] to demonstrate the efficacy of our method. All
these datasets contain four domains out of which three are used as sources and
the other as a target in a leave-one-out strategy. We use a VAE as the
generative model $G_{\phi}$ in all our main results owing to its stability of
training vis-a-vis a GAN. The performance metric is the classification
accuracy and we compare against a baseline Deep All method: classifier trained
on the combined source domains without employing any DG techniques. For each
target domain, we have reported our average and standard deviation for five
independent runs of the model. We also compare our method with the existing DG
methods and report the results, dataset wise. We report the standard deviation
as 0 for models which have not reported them. For each dataset, we use the
validation set for selecting hyperparameters if it is available. Otherwise, we
split the data from source domains and use the smaller set for hyperparameter
selection. We also use data augmentation for regularizing the network
$f_{\theta}$. For further details on the datasets, machine configuration and
choice of hyperparameters, please refer to the Appendix.

Method | Art. | Cartoon | Sketch | Photo | Avg.
---|---|---|---|---|---
AlexNet
Deep All | 65.96$\pm$0.2 | 69.50$\pm$0.2 | 59.89$\pm$0.3 | 89.45$\pm$0.3 | 71.20
Jigen [5] | 67.63$\pm$0.0 | 71.71$\pm$0.0 | 65.18$\pm$0.0 | 89.00$\pm$0.0 | 73.38
MMLD [27] | 69.27$\pm$0.0 | 72.83$\pm$0.0 | 66.44$\pm$0.0 | 88.98$\pm$0.0 | 74.38
MASF [6] | 70.35$\pm$0.3 | 72.46$\pm$0.2 | 67.33$\pm$0.1 | 90.68$\pm$0.1 | 75.21
EISNet [44] | 70.38$\pm$0.4 | 71.59$\pm$1.3 | 70.25$\pm$1.4 | 91.20$\pm$0.0 | 75.86
RSC [16] | 71.62$\pm$0.0 | 75.11$\pm$0.0 | 66.62$\pm$0.0 | 90.88$\pm$0.0 | 76.05
Ours | 72.67$\pm$0.5 | 76.51$\pm$0.3 | 73.09$\pm$0.2 | 92.01$\pm$0.3 | 78.57
ResNet-18
Deep All | 77.65$\pm$0.2 | 75.36$\pm$0.3 | 69.08$\pm$0.2 | 95.12$\pm$0.1 | 79.30
MMLD [27] | 81.28$\pm$0.0 | 77.16$\pm$0.0 | 72.29$\pm$0.0 | 96.09$\pm$0.0 | 81.83
EISNet [44] | 81.89$\pm$0.9 | 76.44$\pm$0.3 | 74.33$\pm$1.4 | 95.93$\pm$0.1 | 82.15
L2A-OT [46] | 83.30$\pm$0.0 | 78.20$\pm$0.0 | 73.60$\pm$0.0 | 96.20$\pm$0.0 | 82.80
DSON [37] | 84.67$\pm$0.0 | 77.65$\pm$0.0 | 82.23$\pm$0.0 | 95.87$\pm$0.0 | 85.11
RSC [16] | 83.43$\pm$0.8 | 80.31$\pm$1.8 | 80.85$\pm$1.2 | 95.99$\pm$0.3 | 85.15
Ours | 86.39$\pm$0.3 | 81.26$\pm$0.2 | 81.79$\pm$0.1 | 97.15$\pm$0.4 | 86.65
ResNet-50
Deep All | 81.31$\pm$0.3 | 78.54$\pm$0.4 | 69.76$\pm$0.4 | 94.97$\pm$0.1 | 81.15
MASF [6] | 82.89$\pm$0.2 | 80.49$\pm$0.2 | 72.29$\pm$0.2 | 95.01$\pm$0.1 | 82.67
EISNet [44] | 86.64$\pm$1.4 | 81.53$\pm$0.6 | 78.07$\pm$1.4 | 97.11$\pm$0.4 | 85.84
DSON [37] | 87.04$\pm$0.0 | 80.62$\pm$0.0 | 82.90$\pm$0.0 | 95.99$\pm$0.0 | 86.64
RSC [16] | 87.89$\pm$0.0 | 82.16$\pm$0.0 | 83.85$\pm$0.0 | 97.92$\pm$0.0 | 87.83
Ours | 90.25$\pm$0.4 | 85.19$\pm$0.2 | 86.20$\pm$0.5 | 98.97$\pm$0.1 | 90.15

Table 1: Comparison of performance between different models on PACS [21]
dataset with AlexNet, ResNet-18 and ResNet-50 as backbones for the
$f_{\theta}$ network.

###  4.1 Multi-source Domain Generalization

PACS: The PACS dataset consists of images from Photo, Art Painting, Cartoon
and Sketch domains. We follow the experimental protocol defined in [21]. We
use ResNet-50, ResNet-18 and AlexNet as backbones for the feature extractor
network $f_{\theta}$ and train them on source domains. To perform target
projection, we learn to sample from the features produced by $f_{\theta}$ by
training a VAE on the latent space ($\mathbf{u}$). We achieve state-of-the-art
results with all three choices of backbones as shown in Table 1.

Method | Caltech | LabelMe | Pascal | Sun | Avg.
---|---|---|---|---|---
Deep All | 96.45$\pm$0.1 | 60.03$\pm$0.5 | 70.41$\pm$0.4 | 62.63$\pm$0.3 | 72.38
Jigen [5] | 96.93$\pm$0.0 | 60.90$\pm$0.0 | 70.62$\pm$0.0 | 64.30$\pm$0.0 | 73.19
MMLD [27] | 96.66$\pm$0.0 | 58.77$\pm$0.0 | 71.96$\pm$0.0 | 68.13$\pm$0.0 | 73.88
MASF [6] | 94.78$\pm$0.2 | 64.90$\pm$0.1 | 69.14$\pm$0.2 | 67.64$\pm$0.1 | 74.11
EISNet [44] | 97.33$\pm$0.4 | 63.49$\pm$0.8 | 69.83$\pm$0.5 | 68.02$\pm$0.8 | 74.67
RSC [16] | 97.61$\pm$0.0 | 61.86$\pm$0.0 | 73.93$\pm$0.0 | 68.32$\pm$0.0 | 75.43
Ours | 98.12$\pm$0.1 | 66.80$\pm$0.3 | 74.77$\pm$0.4 | 70.43$\pm$0.1 | 77.53

Table 2: Comparison of performance between different models using AlexNet
backbone on VLCS [8] dataset.

VLCS: VLCS comprises of the VOC2007(Pascal), LabelMe, Caltech and Sun domains,
all of which contain photos. We follow the same experimental setup as
mentioned in [27] where we train on three source domains with 70% data from
each and test on all the examples from the fourth target domain. We utilize
similar setup for $f_{\theta}$ and $G_{\phi}$ as in the PACS dataset with
AlexNet as the backbone. We achieve SOTA results on VLCS as evident from Table
2. We emphasize that unlike PACS dataset where the domains differ in image
styles, VLCS consists of domains that contain only photos. Thus, we
demonstrate that our method generalizes well even when the source domains are
not diverse.

Method | Artistic | Clipart | Product | Real-World | Avg.
---|---|---|---|---|---
Deep All | 52.06$\pm$0.5 | 46.12$\pm$0.3 | 70.45$\pm$0.2 | 72.45$\pm$0.2 | 60.27
D-SAM [7] | 58.03$\pm$0.0 | 44.37$\pm$0.0 | 69.22$\pm$0.0 | 71.45$\pm$0.0 | 60.77
Jigen [5] | 53.04$\pm$0.0 | 47.51$\pm$0.0 | 71.47$\pm$0.0 | 72.79$\pm$0.0 | 61.20
MMD-AAE [24] | 56.50$\pm$0.0 | 47.30$\pm$0.0 | 72.10$\pm$0.0 | 74.80$\pm$0.0 | 62.70
DSON [37] | 59.37$\pm$0.0 | 45.70$\pm$0.0 | 71.84$\pm$0.0 | 74.68$\pm$0.0 | 62.90
RSC [16] | 58.42$\pm$0.0 | 47.90$\pm$0.0 | 71.63$\pm$0.0 | 74.54$\pm$0.0 | 63.12
L2A-OT [46] | 60.60$\pm$0.0 | 50.10$\pm$0.0 | 74.80$\pm$0.0 | 77.00$\pm$0.0 | 65.60
Ours | 62.63$\pm$0.2 | 55.79$\pm$0.3 | 76.86$\pm$0.1 | 78.98$\pm$0.1 | 68.56

Table 3: Comparison of performance between different models using ResNet-18
backbone on Office-Home [42].

Method | MNIST | MNIST-M | SVHN | SYN | Avg.
---|---|---|---|---|---
Deep All | 95.24$\pm$0.1 | 58.36$\pm$0.6 | 62.12$\pm$0.5 | 78.94$\pm$0.3 | 73.66
Jigen [5] | 96.50$\pm$0.0 | 61.40$\pm$0.0 | 63.70$\pm$0.0 | 74.00$\pm$0.0 | 73.90
CCSA [29] | 95.20$\pm$0.0 | 58.20$\pm$0.0 | 65.50$\pm$0.0 | 79.10$\pm$0.0 | 74.50
MMD-AAE [24] | 96.50$\pm$0.0 | 58.40$\pm$0.0 | 65.00$\pm$0.0 | 78.40$\pm$0.0 | 74.60
CrossGrad [38] | 96.70$\pm$0.0 | 61.10$\pm$0.0 | 65.30$\pm$0.0 | 80.20$\pm$0.0 | 75.80
L2A-OT [46] | 96.70$\pm$0.0 | 63.90$\pm$0.0 | 68.60$\pm$0.0 | 83.20$\pm$0.0 | 78.10
Ours | 97.99$\pm$0.1 | 66.52$\pm$0.4 | 71.31$\pm$0.3 | 85.40$\pm$0.5 | 80.30

Table 4: Comparison of performance between different models on Digits-DG [47]
dataset.

Office-Home: Office-Home contains images from 4 domains namely Artistic,
Clipart, Product and Real-World. We follow the experimental protocol as
outlined in [7]. We utilize a ResNet-18 as backbone with two additional fully-
connected layers. The reconstruction error in VAE is minimized using both L1
and L2 loss functions. We achieve SOTA results on all the domains using a
ResNet-18 backbone as shown in Table 3. It should be noted that Clipart is a
difficult domain to generalize to as it is dissimilar to other domains. We
achieve 5.6% improvement over the nearest competitor on the Clipart domain,
showcasing the merit of our method in generalizing to dissimilar target
domains.

Digits-DG: Digits-DG is the task of digit recognition using MNIST [20],
MNIST-M [9], SVHN [32] and SYN [9] domains that differ drastically in font
style and background. We follow the experimental setup of [47] and use their
architecture for the feature extractor $f_{\theta}$. The $G_{\phi}$ network is
implemented in a similar way as it is implemented for the PACS dataset with
ResNet-18 as the backbone. Table 4 shows the performance of our method on
Digits-DG dataset compared against various SOTA methods.

(a)

(b)

(c)

(d)

Figure 2: (left to right) (a) Plot showing $\mathcal{A}$-divergence between
source and projected target features. (b) Ablation on Office-Home dataset
highlighting the relative importance of different components employed in our
method. (c) Relative performance of different methods trained on 20%, 40%, 60%
and 80% of PACS dataset. For each setting, we report average performance with
all examples from the target domains using leave-one-out strategy. (d)
Relative performance of DG methods trained on a single domain (Sketch (S)) and
tested on Photo (P), Cartoon (C) and Art (A) domains.

###  4.2 Pairwise $\mathcal{H}$-divergence

We examine the effectiveness of the proposed method in projecting target
features onto the source-feature manifold by computing a proxy measure for
$\mathcal{H}$-divergence called the $\mathcal{A}$-distance [3], that measures
the divergence between two domains. We compute the $\mathcal{A}$-distance
between features from the Sketch domain (target domain) and each of the three
source domains (Photo, Cartoon and Art), obtained in two ways - first, from a
ResNet-18 model trained on the source domains (Deep All), and second, the
features obtained after performing target projections using our method. We
compare the $\mathcal{A}$-distance obtained from these two methods in Figure
7a . It is observed that the source-target divergence is reduced substantially
compared to Deep All, suggesting the effectiveness of our projection scheme in
bringing the target points onto the source manifold.

###  4.3 Ablation Studies

To quantify the importance of the two components (i.e. the networks
$f_{\theta}$ and $G_{\phi}$), we perform ablative studies using the Office-
Home and Digits-DG datasets. For the Office-Home dataset, the Deep All model
is trained with ResNet-18 as the backbone while the backbone proposed by [47]
is used for the Deep All model on the Digits-DG dataset. We classify target
domain features extracted from the feature extractor network without the
target projection procedure as shown in Figure 7b. We observe a marginal
improvement over Deep All by classifying on the domain-invariant features
learnt by $f_{\theta}$ without using the $G_{\phi}$ network. We attribute this
to the explicit class separability imposed by the $f_{\theta}$ network in the
feature space $\mathcal{F}$. When we exclude the $f_{\theta}$ network by
training the generative model on Deep All features and perform the projection
procedure on these features, we still observe a sizable performance boost over
merely classifying with the Deep All model. This shows the effectiveness of
the target projection procedure as a sampling strategy. The best performance
is achieved with all components working together as seen in Figure 7b.

###  4.4 Sampling strategy

To quantify the effect of the specific generative model used for the
$G_{\phi}$ network, we report performance with three samplers on VLCS dataset,
namely: (a) VAE, (b) 1-Nearest Neighbor (1-NN) using the similarity metric as
in Eq. 1, (c) GAN, and show results in Table 5. It is seen that sampling using
continuous generative models (VAE and GAN) offer better performance compared
to 1-NN sampling. On domains like LabelMe and Sun, they offer improvement of
about 5.7% and 7.0% respectively over the 1-NN method of sampling. This is
attributed to the fact that generative models can approximate the true data
distribution by learning from empirical data and offer infinite sampling while
1-NN search is restricted to the existing training points only.

Method | Caltech | LabelMe | Pascal | Sun | Avg.
---|---|---|---|---|---
Deep All | 96.45 | 60.03 | 70.41 | 62.63 | 72.38
1-NN | 96.51 | 61.44 | 71.82 | 63.46 | 73.31
Ours ($G_{\phi}$ = GAN) | 97.89 | 67.18 | 74.59 | 70.28 | 77.48
Ours ($G_{\phi}$ = VAE) | 98.12 | 66.80 | 74.77 | 70.43 | 77.53

Table 5: Performance of our method on VLCS dataset with different sampling
strategies.

###  4.5 Low resource settings

In this section, we demonstrate the efficacy of our method in low-resource
settings. We show that our method generalizes well when used in scarce
resource settings and single domain DG problems, and can also easily be
extended to supervised domain adaptation with access to scant labelled target
samples. Our method has a two-fold advantage that makes it especially data-
efficient: (a) since $f_{\theta}$ is trained on pairs of images, it can learn
effectively even on small datasets (b) the generative model $G_{\phi}$ enables
infinite sampling during the target projection procedure.

(a)

(b)

Figure 3: (left) (a) Effect of deviation from the optimal number of iterations
of each target example during inference on PACS. (right) (b) Plot of
iterations vs loss for projection of a target example from Sketch domain on
source manifold ($\mathcal{Z}_{s}$) created by Photo, Art and Cartoon domains.

####  4.5.1 Scarce resource setting

We compare our method against Deep All and two state-of-the-art methods, RSC
[16] and Jigen [5]. For each source domain in PACS we train on {20%, 40%, 60%,
80%} data from each domain and test on the entire target domain. From Figure
7, it is evident that our method outperforms all the baselines at every
considered resource setting.

####  4.5.2 Supervised Domain Adaptation

This setting has also been referred to as Few-shot Domain Adaptation in [36]
and [28]. In this setting, in addition to the source data, we assume that we
have access to a limited number of labelled samples ($|\mathcal{T}|$) from the
target domain at train time. We train the feature extractor network
$f_{\theta}$, generative model $G_{\phi}$ and the classifier
$\mathcal{C_{\psi}}$ on the source data and fine tune on the target samples.
We compare our method against M-ADA [36] and FADA [28] on the Digits dataset.
The results are presented in Table 6. We outperform our nearest competitors by
a significant margin, thus highlighting the adaptability of our method to this
use-case.

Method | $|\mathcal{T}|$ | U$\rightarrow$M | M$\rightarrow$S | S$\rightarrow$M | Avg.
---|---|---|---|---|---
FADA [28] | 7 | 91.50 | 47.00 | 87.20 | 75.23
CCSA [29] | 10 | 95.71 | 37.63 | 94.57 | 75.97
M-ADA [36] | 0 | 71.19 | 36.61 | 60.14 | 55.98
7 | 92.33 | 56.33 | 89.90 | 79.52
10 | 93.67 | 57.16 | 91.81 | 80.88
Ours | 0 | 74.52 | 42.96 | 64.12 | 60.53
7 | 93.81 | 58.92 | 92.02 | 81.58
10 | 96.10 | 60.07 | 95.33 | 83.83

Table 6: Comparison of few-shot domain adaptation performance between
different models on Digits dataset (MNIST [20] (M), USPS [17] (U) and SVHN
[32] (S)).

####  4.5.3 Single Source Domain Generalization

Method | S$\,\to\,$P | S$\,\to\,$C | S$\,\to\,$A
---|---|---|---
Deep All | 29.88 | 32.47 | 30.56
Ours (w/o $G_{\phi}$) | 33.76 | 37.94 | 36.02
Ours (w/o $f_{\theta}$) | 50.39 | 66.82 | 44.94
Ours ($f_{\theta}+G_{\phi}$) | 53.82 | 70.33 | 50.61

Table 7: Ablation on Single Source DG with Sketch (S) as source and Photo (P),
Cartoon (C) and Art (A) as unseen target domains.

In single source DG, we only have access to a single domain during training
and aim to generalize to all other unseen domains. We train on the Sketch
domain of the PACS dataset and test on the other three domains (i.e. Photo,
Art Painting and Cartoon). We compare our method against Jigen [5] and RSC
[16]. The results are presented in Figure 2d. We also examine the individual
effects of each of the components $f_{\theta}$ and $G_{\phi}$ of our method in
this scenario by examining the performance without each of them (Table 7). We
observe that without $G_{\phi}$, the performance difference between Deep All
and our method is substantially lower than the improvement obtained by
performing target projections on Deep All features, highlighting the
effectiveness of the projection procedure. The best performance is obtained
when both are used together, since the projection procedure effectively
utilizes the label-preserving metric defined by $f_{\theta}$.

###  4.6 Robust Domain Generalization

We examine the robustness of our method against different types of
corruptions. We benchmark against the CIFAR-10-C dataset [12], which consists
of images with 19 corruptions types applied at five levels of severities on
the test set of CIFAR-10. We follow the protocol detailed in [36] and train
our model on the CIFAR-10 dataset using a Wide Residual Network (WRN) backbone
[45]. The results are presented severity level-wise in Table 8 and by type of
corruption in Figure 4a. Similar to 4.5.3, we observe the effectiveness of
$G_{\phi}$ in generalizing to corruptions.

Method | Level 1 | Level 2 | Level 3 | Level 4 | Level 5
---|---|---|---|---|---
ERM [19] | 87.8$\pm$0.1 | 81.5$\pm$0.2 | 75.5$\pm$0.4 | 68.2$\pm$0.6 | 56.1$\pm$0.8
M-ADA [36] | 90.5$\pm$0.3 | 86.8$\pm$0.4 | 82.5$\pm$0.6 | 76.4$\pm$0.9 | 65.6$\pm$1.2
Ours | 93.6$\pm$0.2 | 89.2$\pm$0.4 | 85.3$\pm$0.1 | 79.0$\pm$0.3 | 68.2$\pm$0.6

Table 8: Comparison of performance on CIFAR-10-C with 5 different levels of
severity. Accuracy averaged over all 19 corruption levels

(a)

(b)

Figure 4: (left) (a) Performance comparison between M-ADA and our method on 10
out of the 19 corruption types at severity level 5 from the CIFAR-10-C
dataset. (right) (b) Plot showing relative importance of each component of our
method on the CIFAR-10-C dataset. Accuracy averaged over all 19 corruption
levels.

###  4.7 Determining the stopping criteria

The inference-time iterative optimization process discussed in 3.3.3 needs a
stopping criteria, since stopping too early would not guarantee the label
preservation, while stopping too late may take the projected target to a low-
probability region on the source manifold.

We address this issue heuristically: stop the iteration process at the “elbow-
point” (maxima of the second derivative) of the loss curve as a function of
the number of iterations (denoted by $\text{n}^{\ast}$). This choice is
inspired by the observation that the elbow-point reflects the point of
diminishing returns; for a constant iteration rate ($\beta$), the loss
decreases at a slower rate beyond this point.

As empirical evidence, we vary the stopping point for each target sample by a
fixed number of iterations $\epsilon$ around the maxima $\text{n}^{\ast}$ of
the second derivative values of the sample and calculate the accuracy on the
target projections so obtained. The plot of accuracy vs $\epsilon$ is shown in
Figure 3a. We observe that the highest accuracy is obtained around
$\epsilon=0$ which indicates the correctness of the stopping criteria
$\text{n}^{\ast}$. For negative values of $\epsilon$, the accuracy drop can be
explained by inadequate label preservation, while the projected target leaves
the source manifold for positive values of $\epsilon$.

##  5 Conclusion

We propose a novel Domain Generalization technique where the source domains
are utilized to learn a domain invariant label-preserving metric space. During
inference, every target sample is projected onto this space so that the
classifier trained on the source features can generalize well on the projected
target sample. We have demonstrated that this method yields SOTA results on
Multi Source, Single Source and Robust Domain Generalization settings. In
addition, the data-efficiency of the method makes it suitable to work well in
Low Resource settings. Future iterations of work could attempt to extend this
method to Domain Generalization for Segmentation and Zero-Shot Learning.

##  Appendix A Theory

######  Proposition 3.

The label-preserving transformation $f$ defined in Eq. 1 of the main paper
reduces the $\mathcal{H}$-divergence between any pair of domains on which it
is learned. Stated explicitly:

Assume the case of Binary Classification. We assume that class prior
probabilities for each class is equal in all the source domains, i.e.

$\mathbf{P}(\mathcal{Y}_{i}=\ell)=\mathbf{P}(\mathcal{Y}_{k}=\ell)\quad\forall
i,k\in\\{1,...,|S|\\},\forall\ell\in\\{0,1\\}.$ (6)

where
$\mathbf{P}(\mathcal{Y}_{i}={\ell})=\mathop{\mathbb{E}}_{x\in\mathcal{D}_{i}^{S}}[\mathbbm{1}_{g(x)=\ell}]$
denotes the proportion of examples having label $\ell$ in source domain
$\mathcal{D}^{S}_{i}$. If there exists a metric space denoted by
$(\mathcal{F},d)$ and a transformation function $f:\mathcal{X}\to\mathcal{F}$
such that $d(f(x_{1}),f(x_{2}))=0\iff g(x_{1})=g(x_{2})$, i.e. $x_{1}$ and
$x_{2}$ have the same labels (irrespective of domain), then,

$\Delta_{\mathcal{H^{\prime}}}[f(\mathcal{D}_{i}^{S}),f(\mathcal{D}_{k}^{S})]=0\quad\forall
i,k\in\\{1,...,|S|\\}$ (7)

where $\mathcal{H^{\prime}}$ denotes the space of hypotheses
$h^{\prime}:\mathcal{F}\to\mathcal{Y}$, and $f(\mathcal{D})$ denotes the
distribution $\mathcal{D}$ under the transformation $f$.

###### Proof.

For any $i,k\in[|S|]$ and any $h^{\prime}\in\mathcal{H^{\prime}}$,

$\displaystyle\left|\mathbf{P}_{x\in\mathcal{D}_{i}^{S}}[h^{\prime}(f(x))=1]-\mathbf{P}_{x\in\mathcal{D}_{k}^{S}}[h^{\prime}(f(x))=1]\right|$
(8)
$=\left|\mathop{\mathbb{E}}_{x\in\mathcal{D}_{i}^{S}}[\mathbbm{1}_{h^{\prime}(f(x))=1}]-\mathop{\mathbb{E}}_{x\in\mathcal{D}_{k}^{S}}[\mathbbm{1}_{h^{\prime}(f(x))=1}]\right|$
(9)

Now, note that $g(x_{1})=g(x_{2})\implies d(f(x_{1}),f(x_{2}))=0\implies
f(x_{1})=f(x_{2})$ $\implies h^{\prime}(f(x_{1}))=h^{\prime}(f(x_{2}))$.
This means that all the examples of a given class are assigned the same label
$\ell$ by $h^{\prime}$, regardless of the source domain.

Depending on the labels assigned to the different classes by $h^{\prime}$,
these three cases would arise:

Case 1: $h^{\prime}(f(x))=1\quad\forall x\in\mathcal{X}$

$\implies\mathop{\mathbb{E}}_{x\in\mathcal{D}_{i}^{S}}[\mathbbm{1}_{h^{\prime}(f(x))=1}]=\int_{\mathcal{X}}\mathcal{D}_{i}^{S}(x)dx=1$
$=\int_{\mathcal{X}}\mathcal{D}_{k}^{S}(x)dx=\mathop{\mathbb{E}}_{x\in\mathcal{D}_{k}^{S}}[\mathbbm{1}_{h^{\prime}(f(x))=1}]$
(10)
$\therefore\left|\mathbf{P}_{x\in\mathcal{D}_{i}^{S}}[h^{\prime}(f(x))=1]-\mathbf{P}_{x\in\mathcal{D}_{k}^{S}}[h^{\prime}(f(x))=1]\right|=0$

Case 2: $h^{\prime}(f(x))=0\quad\forall x\in\mathcal{X}$

$\mathop{\mathbb{E}}_{x\in\mathcal{D}_{i}^{S}}[\mathbbm{1}_{h^{\prime}(f(x))=1}]=\mathop{\mathbb{E}}_{x\in\mathcal{D}_{j}^{S}}[\mathbbm{1}_{h^{\prime}(f(x))=1}]=0$
(11)
$\therefore\left|\mathbf{P}_{x\in\mathcal{D}_{i}^{S}}[h^{\prime}(f(x))=1]-\mathbf{P}_{x\in\mathcal{D}_{k}^{S}}[h^{\prime}(f(x))=1]\right|=0$

Case 3: $h^{\prime}(f(x))=1\iff g(x)=\ell$ for some $\ell\in\\{0,1\\}$

$\mathop{\mathbb{E}}_{x\in\mathcal{D}_{i}^{S}}[\mathbbm{1}_{h^{\prime}(f(x))=1}]=\mathop{\mathbb{E}}_{x\in\mathcal{D}_{i}^{S}}[\mathbbm{1}_{g(x)=\ell}]=\mathbf{P}(\mathcal{Y}_{i}={\ell})$
(12)

Similarly,

$\mathop{\mathbb{E}}_{x\in\mathcal{D}_{k}^{S}}[\mathbbm{1}_{h^{\prime}(f(x))=1}]=\mathbf{P}(\mathcal{Y}_{k}={\ell})$
(13)
$\therefore\left|\mathbf{P}_{x\in\mathcal{D}_{i}^{S}}[h^{\prime}(f(x))=1]-\mathbf{P}_{x\in\mathcal{D}_{k}^{S}}[h^{\prime}(f(x))=1]\right|\\\
=\left|\mathbf{P}(\mathcal{Y}_{i}={\ell})-\mathbf{P}(\mathcal{Y}_{k}={\ell})\right|=0$
(14)

Since,

$\left|\mathbf{P}_{x\in\mathcal{D}_{i}^{S}}[h^{\prime}(f(x))=1]-\mathbf{P}_{x\in\mathcal{D}_{k}^{S}}[h^{\prime}(f(x))=1]\right|=0\
\forall h^{\prime}\in\mathcal{H^{\prime}}$ (15)

we have,

$\Delta_{\mathcal{H^{\prime}}}[f(\mathcal{D}_{i}^{S}),f(\mathcal{D}_{k}^{S})]=\\\
\sup_{h^{\prime}\in\mathcal{H^{\prime}}}\left|\mathbf{P}_{x\in\mathcal{D}_{i}^{S}}[h^{\prime}(f(x))=1]-\mathbf{P}_{x\in\mathcal{D}_{k}^{S}}[h^{\prime}(f(x))=1]\right|=0\\\
\forall i,k\in\\{1,...,|S|\\}$ (16)

∎

We note here that in practice, the assumption $g(x_{1})=g(x_{2})\implies
d(f(x_{1}),f(x_{2}))=0$ need not necessarily hold, i.e. two different images
having the same label might not have coincident representations. However, the
optimization procedure detailed in Eq. 1 of the main paper forces
representations of the same class together into the same cluster, as described
in Section D.1, thus resulting in a higher similarity score for images with
the same label.

######  Proposition 4.

The expected misclassification rate obtained with a classifier $h^{\prime}$
when the projected target $\mathbf{z_{t}^{\ast}}$ is used instead of the true
target $\mathbf{z_{t}}$, obeys the following upper-bound:

$\displaystyle\mathbb{E}_{(\mathcal{D}^{T},\mathcal{D}^{T^{\ast}})}\left|\tilde{g}(\mathbf{z_{t}})-h^{\prime}(\mathbf{z_{t}^{*}})\right|\leq$
$\displaystyle\underbrace{\mathbb{E}_{\mathcal{D}^{T^{\ast}}}\left|\tilde{g}(\mathbf{z_{t}^{*}})-h^{\prime}(\mathbf{z_{t}^{*}})\right|}_{\text{i}}+\underbrace{\mathbb{E}_{(\mathcal{D}^{T},\mathcal{D}^{T^{\ast}})}\left|\tilde{g}(\mathbf{z_{t}})-\tilde{g}(\mathbf{z_{t}^{*}})\right|}_{\text{ii}}$
(17)

where $\mathcal{D}^{T}$ and $\mathcal{D}^{T^{\ast}}$ respectively denote the
true and the projected target distributions, respectively.

###### Proof.

Using the triangle inequality on $\mathbb{R}$, we get

$\displaystyle\left|\tilde{g}(\mathbf{z_{t}})-h^{\prime}(\mathbf{z_{t}^{*}})\right|\leq\left|\tilde{g}(\mathbf{z_{t}^{*}})-h^{\prime}(\mathbf{z_{t}^{*}})\right|+\left|\tilde{g}(\mathbf{z_{t}})-\tilde{g}(\mathbf{z_{t}^{*}})\right|$
(18)

$\mathbf{z_{t}}$ and $\mathbf{z_{t}^{\ast}}$ are random variables denoting
feature vectors corresponding to the target point and the point obtained by
projecting the target point to the source manifold. We assume that these come
from the true ($\mathcal{D}^{T}$) and projected ($\mathcal{D}^{T^{\ast}}$)
target distributions, respectively. To obtain the average misclassification
rate, we take an expectation with respect to their joint distribution.

Since expectation is a linear operation, we can take expectation over both
sides of the inequality. Doing so with respect to the joint distribution of
$(\mathcal{D}^{T},\mathcal{D}^{T^{\ast}})$ on both sides,

$\displaystyle\mathbb{E}_{(\mathcal{D}^{T},\mathcal{D}^{T^{\ast}})}\left|\tilde{g}(\mathbf{z_{t}})-h^{\prime}(\mathbf{z_{t}^{*}})\right|\leq$
$\displaystyle{\mathbb{E}_{\mathcal{D}^{T^{\ast}}}\left|\tilde{g}(\mathbf{z_{t}^{*}})-h^{\prime}(\mathbf{z_{t}^{*}})\right|}+{\mathbb{E}_{(\mathcal{D}^{T},\mathcal{D}^{T^{\ast}})}\left|\tilde{g}(\mathbf{z_{t}})-\tilde{g}(\mathbf{z_{t}^{*}})\right|}$
(19)

Note that in term (i) of the inequality, we can omit the expectation over
$\mathcal{D}^{T}$ since there is no dependence on the random variable
$\mathbf{z_{t}}$.

∎

Figure 5: Few example images from PACS (1st row), VLCS (2nd row), Office-Home
(3rd row) and Digits-DG (4th row) datasets. Figure 6: Few example images from
CIFAR-10-C dataset depicting 5 levels of severities and four of the 19
Corruption types.

##  Appendix B Datasets

We demonstrate the effectiveness of our algorithm on the following DG
datasets:

PACS: Stands for Photo, Art Painting, Cartoon and Sketch. This dataset
contains a total of 9991 images taken from different sources such as
Caltech256, Sketchy, TU-Berlin and Google Images. Each image is assigned one
out of seven possible labels namely dog, elephant, giraffe, guitar, horse,
house or person. The substantial domain shift in the dataset poses a
significant challenge to DG methods.

VLCS: Stands for VOC2007(Pascal), LabelMe, Caltech and Sun where each one of
them is a different dataset differing in the camera specifications. There are
a total of 10729 photos in the whole dataset where each photo is assigned one
out of five labels namely bird, car, chair, dog, or person.

Office-Home: This dataset is comprised of four domains namely Art, Clipart,
Product and Real-World. There are a total of 15588 images in the dataset and
each image is assigned one out of 65 classes. The Real-World images are taken
with a regular camera and the Product images are taken from a vendor websites
and thus differ in background and quality.

Digits-DG: This digit recognition (0-9) dataset contains a mixture of 4
different domains namely MNIST, MNIST-M, SVHN, and SYN that differ drastically
in font style and background. The dataset contains 24000 images.

CIFAR-10-C: This dataset is a robustness benchmark for deep learning systems.
It consists of test images of CIFAR-10 with 19 different types of corruption
having 5 different severity levels (1-5) with 5 indicating most severe. The
models are trained on CIFAR-10 and evaluated on CIFAR-10-C. Please refer to
Figure 6 for a few samples from different severity levels.

(a)

(b)

(c)

Figure 7:  Relative performance of DG methods trained on a single domain of
PACS dataset.

##  Appendix C Implementation Details

###  C.1 System Configuration and Frameworks

All the experiments are performed using an Intel Xeon processor (12 Cores)
with a base frequency of 2.7 GHz, 32GB RAM, Ubuntu OS and four NVIDIA® Tesla®
V100 (16 GB Memory) GPUs. Our implementation was written in Python 3.6.10 and
uses PyTorch version 1.6.0 running on CUDA version 10.1.

###  C.2 Architectural details

Please refer to Figure 11 for different architectures and backbones that we
employed to evaluate the performance of our method on each dataset.

###  C.3 Hyperparameter choices

We select hyperparameters based on model performance on a validation set
consisting of data from the source domains. We use the splits given with the
dataset whenever they are available. When the validation split for the source
domains are not explicitly provided, we split our training data into a small
validation set for model selection. We choose the best model based on (a) the
average loss $\mathcal{L_{A}}$ (for $f_{\theta}$) (b) the average accuracy
(for $\mathcal{C_{\psi}}$) (c) the average reconstruction/discriminator loss
(for $G_{\phi}$) on the validation set.

##### Training the network $f_{\theta}$:

We use the SGD optimizer with a learning rate of 0.001 to train the
$f_{\theta}$ network for all the datasets except for Digits-DG and CIFAR-10-C
datasets. Digits-DG dataset is trained with a learning rate of 0.05 while for
CIFAR-10-C, we train the network with an initial learning rate of 0.1.

We use the same backbone as described in [47] for Digits-DG dataset and for
CIFAR-10-C, the backbone employed is Wide Residual Network [45] (WRN) with
depth and width of 16 and 4, respectively. We train for 200 epochs with the
ResNet-18/ResNet-50 backbones, while AlexNet models are trained for 250
epochs. For the Digits-DG dataset, the $f_{\theta}$ network is trained for 100
epochs while WRN is trained for 200 epochs with a weight decay of 0.0005. In
WRN, the initial learning rate of 0.1 is reduced to 0.02, 0.004 and 0.0008 at
the 50th, 100th, and 150th epoch, respectively. As a regularization, we apply
random image augmentations, namely horizontal flip (with probability 0.5),
color jitter (with probability 0.8), greyscaling (with probability 0.2),
applying Gaussian Blur (with a kernel of size 21) and adding Gaussian noise
(with $\sigma=0.2)$. Batch size for all the backbones is fixed to be 128.

##### Training the VAE/GAN $G_{\phi}$:

VAE is trained with a learning rate of 0.005 and momentum of 0.9 using SGD
optimizer for 350 epochs in all cases except for the Digits-DG dataset, where
it is trained for 150 epochs. We train with a batch size of 64 in all
settings. We use the standard VAE objective, with a combination of both the L1
and L2 losses for the reconstruction error term. In case of GAN (on VLCS
dataset), the learning rate is set to be 0.0002 and it is trained for 450
epochs.

##### Training the Classifier $\mathcal{C}_{\psi}$:

The classifier $\mathcal{C}_{\psi}$ is trained using the Adam optimizer with a
learning rate of 0.003 for 15, 20 and 30 epochs on the Digits-DG backbone,
ResNet-based models and AlexNet-based models respectively, with a batch size
of 64 in all cases.

###  C.4 Iteration rate $\beta$

Iteration rate ($\beta$) |  Avg. number of iterations to stop |  Total iterations ($M$) | Acc. (in %)
---|---|---|---
0.05 | 465 | 20000 | 81.32
0.01 | 602 | 20000 | 81.79
0.005 | 2672 | 20000 | 80.92
0.001 | 15976 | 20000 | 81.56

Table 9: Comparison of performance on Sketch as a target domain for different
values of iteration rate $\beta$.

To project a target example $\mathbf{z_{t}}$ on to the manifold of source
features ($\mathcal{Z}_{s}$) during inference, we follow an iterative
optimization procedure where a latent vector $\mathbf{u}$ (initially drawn
from an isotropic Gaussian distribution) is optimized in the generative latent
space of $G_{\phi}$. We examine the effect of varying the iteration rate
$\beta$ in this optimization procedure through the following experiment: for a
fixed iteration rate $\beta$, selected from the values
$\\{0.05,0.01,0.005,0.001\\}$, we perform the optimization process until the
‘optimal stopping point’ (which corresponds to the “elbow” point) is reached.
We report the test accuracy in each case. As shown in Table 9, we have
validated that for a large number of iterations ($M=20000$), all reasonably
small iteration rates give consistent performance on target domains. The
second column in Table 9 reports the average of all optimal stopping points
($\text{n}^{\ast}$) of the target examples for the Sketch domain of the PACS
dataset.

Thus, we conclude that for reasonably small choice of $\beta$, the performance
of the inference-time optimization procedure is unaffected by varying $\beta$.
Hence, we fix the iteration rate to be 0.01 in all our experiments for
uniformity.

(a)

(b)

Figure 8: T-SNE plot of features obtained from mixture of PAC (Photo, Art,
Cartoon) as source domains using a) Deep All and b) $f_{\theta}$ network.

(a)

(b)

Figure 9: a) $\mathcal{A}$-distance (lower is better) between sources using
Deep All and label-preserving features from $f_{\theta}$. b) Ablation on
different components of the proposed method with Digits-DG dataset.

##  Appendix D Additional Results

###  D.1 Class-wise alignment

Despite the large inter-domain discrepancy in the PACS dataset, it is observed
that when we train the $f_{\theta}$ network on Photo, Art and Cartoon as
source domains, we obtain features that align (or cluster) class-wise which is
not the case with Deep All features on same source domains as shown by T-SNE
plots in Figure 8. This demonstrates the effectiveness of the objective of
$f_{\theta}$ network in bringing examples from the same class together in the
feature space $\mathcal{F}$ irrespective of the domains they belong to. It
helps shallow classifiers (with a single fully connected layer as in
$\mathcal{C}_{\psi}$) to distinguish between features which are more distinct
across the class-labels. For features extracted from the Deep All model, the
separation across the classes is less which makes the shallow classifier less
robust and more error-prone.

We also demonstrate the effectiveness of the $f_{\theta}$ network in reducing
source-source $\mathcal{H}$-divergence by comparing the $\mathcal{A}$-distance
between features extracted from different source domains through Deep All and
the $f_{\theta}$ network. These results are presented in 9a.

###  D.2 Low resource settings

We train different models on each of the source domains, namely Photo, Cartoon
and Art and test on the other three domains of the PACS dataset. We observe
that our method is significantly more data efficient compared to the
baselines, as shown in Figure 7, owing to the fact that $f_{\theta}$ learns
label information from pairs of images and $G_{\phi}$ facilitates projection
on the manifold of label-reserving source features.

###  D.3 Ablation with Digits-DG

We conduct ablation with Digits-DG [47] dataset by training the proposed
method with and without $f_{\theta}$ and $G_{\phi}$ networks. Figure 9b shows
the merit of the proposed components in improving the performance on the
Digits-DG dataset.

###  D.4 Robust DG

Figure 10 shows the robustness of our method against all 19 types of
corruptions from the CIFAR-10-C dataset [12]. The proposed method trained on
CIFAR-10 dataset, achieves generalization to the corrupted target images
having severity level 5.

Figure 10: Comparison of performance on all 19 Corruption types with severity
level 5.

(a) Multi-source DG on PACS with AlexNet

(b)  Multi-source DG on PACS with RestNet-18

(c) Multi-source DG on VLCS with AlexNet

(d) Multi-source DG on Office-Home with ResNet-18

(e) Multi-source DG on Digits-DG

(f) Multi-source DG on VLCS with GAN

(g) Multi-source DG on PACS with ResNet-50

(h) Robust DG on CIFAR-10-C with Wide Residual Network (WRN)

Figure 11:  Architectures used for each dataset. Conv2d, ConvT2d, MaxPool2d
and FC are Convolution 2D, Convolution Transpose 2D, 2D Max Pooling and Fully
Connected layers, respectively. $\mathcal{C}_{\psi}$ is a single hidden layer
classifier. Square bracket represents kernel size with number of output
channels written right after it. $\mathbf{u}$ represents a vector from the
latent space of $G_{\phi}$. ReLU/LeakyReLU activations are used in all blocks.
All components $f_{\theta}$, $G_{\phi}$ and $\mathcal{C}_{\psi}$ are trained
independently.

## References

  * [1] Yogesh Balaji, Swami Sankaranarayanan, and Rama Chellappa.  Metareg: Towards domain generalization using meta-regularization.  In Advances in Neural Information Processing Systems, pages 998–1008, 2018.
  * [2] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan.  A theory of learning from different domains.  Machine learning, 79(1-2):151–175, 2010.
  * [3] Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira.  Analysis of representations for domain adaptation.  In Advances in neural information processing systems, pages 137–144, 2007.
  * [4] Konstantinos Bousmalis, Nathan Silberman, David Dohan, Dumitru Erhan, and Dilip Krishnan.  Unsupervised pixel-level domain adaptation with generative adversarial networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3722–3731, 2017.
  * [5] Fabio M Carlucci, Antonio D’Innocente, Silvia Bucci, Barbara Caputo, and Tatiana Tommasi.  Domain generalization by solving jigsaw puzzles.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2229–2238, 2019.
  * [6] Qi Dou, Daniel Coelho de Castro, Konstantinos Kamnitsas, and Ben Glocker.  Domain generalization via model-agnostic learning of semantic features.  In Advances in Neural Information Processing Systems, pages 6450–6461, 2019.
  * [7] Antonio D’Innocente and Barbara Caputo.  Domain generalization with domain-specific aggregation modules.  In German Conference on Pattern Recognition, pages 187–198. Springer, 2018.
  * [8] Chen Fang, Ye Xu, and Daniel N. Rockmore.  Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), December 2013.
  * [9] Yaroslav Ganin and Victor Lempitsky.  Unsupervised domain adaptation by backpropagation.  In International conference on machine learning, pages 1180–1189, 2015.
  * [10] Muhammad Ghifary, W Bastiaan Kleijn, Mengjie Zhang, and David Balduzzi.  Domain generalization for object recognition with multi-task autoencoders.  In Proceedings of the IEEE international conference on computer vision, pages 2551–2559, 2015.
  * [11] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In Advances in neural information processing systems, pages 2672–2680, 2014.
  * [12] Dan Hendrycks and Thomas Dietterich.  Benchmarking neural network robustness to common corruptions and perturbations.  In International Conference on Learning Representations, 2019.
  * [13] Alex Hernández-García and Peter König.  Further advantages of data augmentation on convolutional neural networks.  In International Conference on Artificial Neural Networks, pages 95–103. Springer, 2018.
  * [14] Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell.  Cycada: Cycle-consistent adversarial domain adaptation.  In International conference on machine learning, pages 1989–1998, 2018.
  * [15] Shoubo Hu, Kun Zhang, Zhitang Chen, and Laiwan Chan.  Domain generalization via multidomain discriminant analysis.  In Uncertainty in artificial intelligence: proceedings of the… conference. Conference on Uncertainty in Artificial Intelligence, volume 35. NIH Public Access, 2019.
  * [16] Zeyi Huang, Haohan Wang, Eric P Xing, and Dong Huang.  Self-challenging improves cross-domain generalization.  arXiv preprint arXiv:2007.02454, 2020.
  * [17] Jonathan J. Hull.  A database for handwritten text recognition research.  IEEE Transactions on pattern analysis and machine intelligence, 16(5):550–554, 1994.
  * [18] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  arXiv preprint arXiv:1312.6114, 2013.
  * [19] Vladimir Koltchinskii.  Oracle Inequalities in Empirical Risk Minimization and Sparse Recovery Problems: Ecole d’Eté de Probabilités de Saint-Flour XXXVIII-2008, volume 2033.  Springer Science & Business Media, 2011.
  * [20] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.  Gradient-based learning applied to document recognition.  Proceedings of the IEEE, 86(11):2278–2324, 1998.
  * [21] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales.  Deeper, broader and artier domain generalization.  In Proceedings of the IEEE international conference on computer vision, pages 5542–5550, 2017.
  * [22] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales.  Learning to generalize: Meta-learning for domain generalization.  In Thirty-Second AAAI Conference on Artificial Intelligence, 2018\.
  * [23] Da Li, Jianshu Zhang, Yongxin Yang, Cong Liu, Yi-Zhe Song, and Timothy M Hospedales.  Episodic training for domain generalization.  In Proceedings of the IEEE International Conference on Computer Vision, pages 1446–1455, 2019.
  * [24] Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot.  Domain generalization with adversarial feature learning.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5400–5409, 2018.
  * [25] Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao.  Deep domain generalization via conditional invariant adversarial networks.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 624–639, 2018.
  * [26] Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I Jordan.  Unsupervised domain adaptation with residual transfer networks.  In Advances in neural information processing systems, pages 136–144, 2016.
  * [27] Toshihiko Matsuura and Tatsuya Harada.  Domain generalization using a mixture of multiple latent domains.  In AAAI, pages 11749–11756, 2020.
  * [28] Saeid Motiian, Quinn Jones, Seyed Iranmanesh, and Gianfranco Doretto.  Few-shot adversarial domain adaptation.  In Advances in Neural Information Processing Systems, pages 6670–6680, 2017.
  * [29] Saeid Motiian, Marco Piccirilli, Donald A Adjeroh, and Gianfranco Doretto.  Unified deep supervised domain adaptation and generalization.  In Proceedings of the IEEE International Conference on Computer Vision, pages 5715–5725, 2017.
  * [30] Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf.  Domain generalization via invariant feature representation.  In International Conference on Machine Learning, pages 10–18, 2013\.
  * [31] Zak Murez, Soheil Kolouri, David Kriegman, Ravi Ramamoorthi, and Kyungnam Kim.  Image to image translation for domain adaptation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4500–4509, 2018.
  * [32] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng.  Reading digits in natural images with unsupervised feature learning.  2011\.
  * [33] Pau Panareda Busto and Juergen Gall.  Open set domain adaptation.  In Proceedings of the IEEE International Conference on Computer Vision, pages 754–763, 2017.
  * [34] Prashant Pandey, Aayush Kumar Tyagi, Sameer Ambekar, and Prathosh.  Skin segmentation from nir images using unsupervised domain adaptation through generative latent search.  arXiv preprint arXiv:2006.08696, 2020.
  * [35] Vihari Piratla, Praneeth Netrapalli, and Sunita Sarawagi.  Efficient domain generalization via common-specific low-rank decomposition.  arXiv preprint arXiv:2003.12815, 2020.
  * [36] Fengchun Qiao, Long Zhao, and Xi Peng.  Learning to learn single domain generalization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12556–12565, 2020.
  * [37] Seonguk Seo, Yumin Suh, Dongwan Kim, Jongwoo Han, and Bohyung Han.  Learning to optimize domain specific normalization for domain generalization.  arXiv preprint arXiv:1907.04275, 2019.
  * [38] Shiv Shankar, Vihari Piratla, Soumen Chakrabarti, Siddhartha Chaudhuri, Preethi Jyothi, and Sunita Sarawagi.  Generalizing across domains via cross-gradient training.  In International Conference on Learning Representations, 2018.
  * [39] Baochen Sun, Jiashi Feng, and Kate Saenko.  Return of frustratingly easy domain adaptation.  In Thirtieth AAAI Conference on Artificial Intelligence, 2016.
  * [40] Antonio Torralba and Alexei A Efros.  Unbiased look at dataset bias.  In CVPR 2011, pages 1521–1528. IEEE, 2011.
  * [41] Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell.  Adversarial discriminative domain adaptation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7167–7176, 2017.
  * [42] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan.  Deep hashing network for unsupervised domain adaptation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5018–5027, 2017.
  * [43] Riccardo Volpi, Hongseok Namkoong, Ozan Sener, John C Duchi, Vittorio Murino, and Silvio Savarese.  Generalizing to unseen domains via adversarial data augmentation.  In Advances in neural information processing systems, pages 5334–5344, 2018.
  * [44] Shujun Wang, Lequan Yu, Caizi Li, Chi-Wing Fu, and Pheng-Ann Heng.  Learning from extrinsic and intrinsic supervisions for domain generalization.  arXiv preprint arXiv:2007.09316, 2020.
  * [45] Sergey Zagoruyko and Nikos Komodakis.  Wide residual networks.  In Edwin R. Hancock Richard C. Wilson and William A. P. Smith, editors, Proceedings of the British Machine Vision Conference (BMVC), pages 87.1–87.12. BMVA Press, September 2016.
  * [46] Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang.  Learning to generate novel domains for domain generalization.  arXiv preprint arXiv:2007.03304, 2020.
  * [47] Kaiyang Zhou, Yongxin Yang, Timothy M Hospedales, and Tao Xiang.  Deep domain-adversarial image generation for domain generalisation.  In AAAI, pages 13025–13032, 2020.


* Diebold and Yılmaz, (2014) Diebold, F. X. and Yılmaz, K. (2014).  On the network topology of variance decompositions: Measuring the connectedness of financial firms.  Journal of econometrics, 182(1):119–134.
  * Dolado, (1992) Dolado, J. J. (1992).  A note on weak exogeneity in var cointegrated models.  Economics Letters, 38(2):139–143.
  * Dufour, (2003) Dufour, J.-M. (2003).  Identification, weak instruments, and statistical inference in econometrics.  Canadian Journal of Economics/Revue canadienne d’économique, 36(4):767–808.
  * Eickmeier et al., (2015) Eickmeier, S., Lemke, W., and Marcellino, M. (2015).  Classical time varying factor-augmented vector auto-regressive models—estimation, forecasting and structural analysis.  Journal of the Royal Statistical Society Series A: Statistics in Society, 178(3):493–533.
  * Feldkircher et al., (2020) Feldkircher, M., Huber, F., and Pfarrhofer, M. (2020).  Factor augmented vector autoregressions, panel vars, and global vars.  Macroeconomic Forecasting in the Era of Big Data: Theory and Practice, pages 65–93.
  * Feve and Guay, (2010) Feve, P. and Guay, A. (2010).  Identification of technology shocks in structural vars.  The Economic Journal, 120(549):1284–1318.
  * Forni et al., (2017) Forni, M., Hallin, M., Lippi, M., and Zaffaroni, P. (2017).  Dynamic factor models with infinite-dimensional factor space: Asymptotic analysis.  Journal of Econometrics, 199(1):74–92.
  * Freyberger et al., (2020) Freyberger, J., Neuhierl, A., and Weber, M. (2020).  Dissecting characteristics nonparametrically.  The Review of Financial Studies, 33(5):2326–2377.
  * Fries and Zakoian, (2019) Fries, S. and Zakoian, J.-M. (2019).  Mixed causal-noncausal ar processes and the modelling of explosive bubbles.  Econometric Theory, 35(6):1234–1270.
  * Fry and Pagan, (2011) Fry, R. and Pagan, A. (2011).  Sign restrictions in structural vector autoregressions: A critical review.  Journal of Economic Literature, 49(4):938–960.
  * Gafarov et al., (2018) Gafarov, B., Meier, M., and Olea, J. L. M. (2018).  Delta-method inference for a class of set-identified svars.  Journal of Econometrics, 203(2):316–327.
  * Ganics et al., (2021) Ganics, G., Inoue, A., and Rossi, B. (2021).  Confidence intervals for bias and size distortion in iv and local projections-iv models.  Journal of Business & Economic Statistics, 39(1):307–324.
  * Giraitis et al., (2021) Giraitis, L., Kapetanios, G., and Marcellino, M. (2021).  Time-varying instrumental variable estimation.  Journal of Econometrics, 224(2):394–415.
  * Goes, (2016) Goes, C. (2016).  Institutions and growth: A gmm/iv panel var approach.  Economics letters, 138:85–91.
  * Gomez Cram and Olbert, (2023) Gomez Cram, R. and Olbert, M. (2023).  Measuring the expected effects of the global tax reform.  Review of Financial Studies.
  * Gorodnichenko and Lee, (2020) Gorodnichenko, Y. and Lee, B. (2020).  Forecast error variance decompositions with local projections.  Journal of Business & Economic Statistics, 38(4):921–933.
  * Gourieroux and Monfort, (1997) Gourieroux, C. and Monfort, A. (1997).  Time series and dynamic models, volume 3.  Cambridge University Press.
  * Gouriéroux et al., (2020) Gouriéroux, C., Monfort, A., and Renne, J.-P. (2020).  Identification and estimation in non-fundamental structural varma models.  The Review of Economic Studies, 87(4):1915–1953.
  * Greenaway-McGrevy, (2013) Greenaway-McGrevy, R. (2013).  Multistep prediction of panel vector autoregressive processes.  Econometric Theory, 29(4):699–734.
  * Greenaway-McGrevy, (2020) Greenaway-McGrevy, R. (2020).  Multistep forecast selection for panel data.  Econometric Reviews, 39(4):373–406.
  * Grigoriu, (1995) Grigoriu, M. (1995).  Linear and nonlinear systems with non-gaussian white noise input.  Probabilistic engineering mechanics, 10(3):171–179.
  * Guay, (2021) Guay, A. (2021).  Identification of structural vector autoregressions through higher unconditional moments.  Journal of Econometrics, 225(1):27–46.
  * Guo and Ling, (2023) Guo, F. and Ling, S. (2023).  Inference for the vec (1) model with a heavy-tailed linear process errors.  Econometric Reviews, 42(9-10):806–833.
  * Hamilton, (2020) Hamilton, J. D. (2020).  Time series analysis.  Princeton university press.
  * Han, (2018) Han, X. (2018).  Estimation and inference of dynamic structural factor models with over-identifying restrictions.  Journal of Econometrics, 202(2):125–147.
  * Hannadige et al., (2023) Hannadige, S. B., Gao, J., Silvapulle, M. J., and Silvapulle, P. (2023).  Forecasting a nonstationary time series using a mixture of stationary and nonstationary factors as predictors.  Journal of Business & Economic Statistics, pages 1–13.
  * Hatchondo et al., (2016) Hatchondo, J. C., Martinez, L., and Sosa-Padilla, C. (2016).  Debt dilution and sovereign default risk.  Journal of Political Economy, 124(5):1383–1422.
  * Hausman and Taylor, (1983) Hausman, J. A. and Taylor, W. E. (1983).  Identification in linear simultaneous equations models with covariance restrictions: An instrumental variables interpretation.  Econometrica: Journal of the Econometric Society, pages 1527–1549.
  * Hecq et al., (2000) Hecq, A., Palm, F. C., and Urbain, J.-P. (2000).  Permanent-transitory decomposition in var models with cointegration and common cycles.  Oxford Bulletin of Economics and Statistics, 62(4):511–532.
  * Hecq and Voisin, (2021) Hecq, A. and Voisin, E. (2021).  Forecasting bubbles with mixed causal-noncausal autoregressive models.  Econometrics and Statistics, 20:29–45.
  * Hendry, (1994) Hendry, D. F. (1994).  On the interactions of unit roots and exogeneity.  Econometric Reviews, 14(4):383–419.
  * Hsiao, (1997) Hsiao, C. (1997).  Statistical properties of the two-stage least squares estimator under cointegration.  The Review of Economic Studies, 64(3):385–398.
  * Hsu et al., (2022) Hsu, Y.-C., Lai, T.-C., and Lieli, R. P. (2022).  Counterfactual treatment effects: Estimation and inference.  Journal of Business & Economic Statistics, 40(1):240–255.
  * Huber et al., (2023) Huber, F., Krisztin, T., and Pfarrhofer, M. (2023).  A bayesian panel vector autoregression to analyze the impact of climate shocks on high-income economies.  The Annals of Applied Statistics, 17(2):1543–1573.
  * Ivanov and Kilian, (2005) Ivanov, V. and Kilian, L. (2005).  A practitioner’s guide to lag order selection for var impulse response analysis.  Studies in Nonlinear Dynamics & Econometrics, 9(1).
  * Johansen, (1995) Johansen, S. (1995).  Identifying restrictions of linear equations with applications to simultaneous equations and cointegration.  Journal of econometrics, 69(1):111–132.
  * Jorda, (2005) Jorda, O. (2005).  Estimation and inference of impulse responses by local projections.  American economic review, 95(1):161–182.
  * Kalliovirta et al., (2015) Kalliovirta, L., Meitz, M., and Saikkonen, P. (2015).  A gaussian mixture autoregressive model for univariate time series.  Journal of Time Series Analysis, 36(2):247–266.
  * Kalliovirta et al., (2016) Kalliovirta, L., Meitz, M., and Saikkonen, P. (2016).  Gaussian mixture vector autoregression.  Journal of econometrics, 192(2):485–498.
  * Kang, (2003) Kang, I.-B. (2003).  Multi-period forecasting using different models for different horizons: an application to us economic time series data.  International Journal of Forecasting, 19(3):387–400.
  * Kao et al., (2018) Kao, C., Trapani, L., and Urga, G. (2018).  Testing for instability in covariance structures.  Bernoulli, 24(1):740–771.
  * Kapetanios et al., (2019) Kapetanios, G., Marcellino, M., and Venditti, F. (2019).  Large time-varying parameter vars: A nonparametric approach.  Journal of Applied Econometrics, 34(7):1027–1049.
  * Katsouris, (2023) Katsouris, C. (2023).  Limit theory under network dependence and nonstationarity.  arXiv preprint arXiv:2308.01418.
  * Keweloh, (2021) Keweloh, S. A. (2021).  A generalized method of moments estimator for structural vector autoregressions based on higher moments.  Journal of Business & Economic Statistics, 39(3):772–782.
  * Kilian and Kim, (2011) Kilian, L. and Kim, Y. J. (2011).  How reliable are local projection estimators of impulse responses?  Review of Economics and Statistics, 93(4):1460–1466.
  * Kilian and Lewis, (2011) Kilian, L. and Lewis, L. T. (2011).  Does the fed respond to oil price shocks?  The Economic Journal, 121(555):1047–1072.
  * Kilian and Lütkepohl, (2017) Kilian, L. and Lütkepohl, H. (2017).  Structural vector autoregressive analysis.  Cambridge University Press.
  * Kim and Roubini, (2000) Kim, S. and Roubini, N. (2000).  Exchange rate anomalies in the industrial countries: A solution with a structural var approach.  Journal of Monetary economics, 45(3):561–586.
  * Komunjer and Ng, (2011) Komunjer, I. and Ng, S. (2011).  Dynamic identification of dynamic stochastic general equilibrium models.  Econometrica, 79(6):1995–2032.
  * Kong et al., (2019) Kong, J., Phillips, P. C., and Sul, D. (2019).  Weak $\sigma$-convergence: Theory and applications.  Journal of Econometrics, 209(2):185–207.
  * Koop and Korobilis, (2013) Koop, G. and Korobilis, D. (2013).  Large time-varying parameter vars.  Journal of Econometrics, 177(2):185–198.
  * Krampe et al., (2023) Krampe, J., Paparoditis, E., and Trenkler, C. (2023).  Structural inference in sparse high-dimensional vector autoregressions.  Journal of Econometrics, 234(1):276–300.
  * Kuersteiner, (2005) Kuersteiner, G. M. (2005).  Automatic inference for infinite order vector autoregressions.  Econometric Theory, 21(1):85–115.
  * Kyle et al., (2023) Kyle, A. S., Obizhaeva, A. A., and Wang, Y. (2023).  Beliefs aggregation and return predictability.  The Journal of Finance, 78(1):427–486.
  * Lai and Wei, (1982) Lai, T. L. and Wei, C. Z. (1982).  Least squares estimates in stochastic regression models with applications to identification and control of dynamic systems.  The Annals of Statistics, 10(1):154–166.
  * Lanne et al., (2023) Lanne, M., Liu, K., and Luoto, J. (2023).  Identifying structural vector autoregression via leptokurtic economic shocks.  Journal of Business & Economic Statistics, 41(4):1341–1351.
  * Lanne and Luoto, (2021) Lanne, M. and Luoto, J. (2021).  Gmm estimation of non-gaussian structural vector autoregression.  Journal of Business & Economic Statistics, 39(1):69–81.
  * (102) Lanne, M. and Lütkepohl, H. (2008a).  Identifying monetary policy shocks via changes in volatility.  Journal of Money, Credit and Banking, 40(6):1131–1149.
  * (103) Lanne, M. and Lütkepohl, H. (2008b).  A statistical comparison of alternative identification schemes for monetary policy shocks.
  * Lanne et al., (2010) Lanne, M., Lütkepohl, H., and Maciejowska, K. (2010).  Structural vector autoregressions with markov switching.  Journal of Economic Dynamics and Control, 34(2):121–131.
  * Lanne et al., (2017) Lanne, M., Meitz, M., and Saikkonen, P. (2017).  Identification and estimation of non-gaussian structural vector autoregressions.  Journal of Econometrics, 196(2):288–304.
  * Lanne and Nyberg, (2016) Lanne, M. and Nyberg, H. (2016).  Generalized forecast error variance decomposition for linear and nonlinear multivariate models.  Oxford Bulletin of Economics and Statistics, 78(4):595–603.
  * Lanne and Saikkonen, (2002) Lanne, M. and Saikkonen, P. (2002).  Threshold autoregressions for strongly autocorrelated time series.  Journal of Business & Economic Statistics, 20(2):282–289.
  * Lanne and Saikkonen, (2013) Lanne, M. and Saikkonen, P. (2013).  Noncausal vector autoregression.  Econometric Theory, 29(3):447–481.
  * Lastrapes, (2005) Lastrapes, W. D. (2005).  Estimating and identifying vector autoregressions under diagonality and block exogeneity restrictions.  Economics letters, 87(1):75–81.
  * Lee et al., (2023) Lee, S., Lee, S., and Maekawa, K. (2023).  Change point test for structural vector autoregressive model via independent component analysis.  Journal of Statistical Computation and Simulation, 93(5):687–707.
  * Leeper et al., (2013) Leeper, E. M., Walker, T. B., and Yang, S.-C. S. (2013).  Fiscal foresight and information flows.  Econometrica, 81(3):1115–1145.
  * Lewis and Syrgkanis, (2020) Lewis, G. and Syrgkanis, V. (2020).  Double/debiased machine learning for dynamic treatment effects via g-estimation.  arXiv preprint arXiv:2002.07285.
  * LI and Hui, (1989) LI, W. K. and Hui, Y. (1989).  Robust multiple time series modelling.  Biometrika, 76(2):309–315.
  * Lovcha and Perez-Laborda, (2021) Lovcha, Y. and Perez-Laborda, A. (2021).  Identifying technology shocks at the business cycle via spectral variance decompositions.  Macroeconomic Dynamics, 25(8):1966–1992.
  * Lütkepohl, (1982) Lütkepohl, H. (1982).  Non-causality due to omitted variables.  Journal of Econometrics, 19(2-3):367–378.
  * Lütkepohl, (1990) Lütkepohl, H. (1990).  Asymptotic distributions of impulse response functions and forecast error variance decompositions of vector autoregressive models.  The review of economics and statistics, pages 116–125.
  * Lütkepohl, (2002) Lütkepohl, H. (2002).  Forecasting cointegrated varma processes.  A Companion to Economic Forecasting, Blackwell, Oxford, pages 179–205.
  * Lütkepohl et al., (2021) Lütkepohl, H., Meitz, M., Netšunajev, A., and Saikkonen, P. (2021).  Testing identification via heteroskedasticity in structural vector autoregressive models.  The Econometrics Journal, 24(1):1–22.
  * Magnusson and Mavroeidis, (2014) Magnusson, L. M. and Mavroeidis, S. (2014).  Identification using stability restrictions.  Econometrica, 82(5):1799–1851.
  * Marcellino et al., (2006) Marcellino, M., Stock, J. H., and Watson, M. W. (2006).  A comparison of direct and iterated multistep ar methods for forecasting macroeconomic time series.  Journal of econometrics, 135(1-2):499–526.
  * Mariano, (2002) Mariano, R. S. (2002).  Testing forecast accuracy.  A companion to economic forecasting, 2:284–298.
  * Medeiros and Mendes, (2016) Medeiros, M. C. and Mendes, E. F. (2016).  Regularization of high-dimensional time-series models with non-gaussian and heteroskedastic errors.  Journal of Econometrics, 191(1):255–271.
  * Meitz et al., (2023) Meitz, M., Preve, D., and Saikkonen, P. (2023).  A mixture autoregressive model based on student’st–distribution.  Communications in Statistics-Theory and Methods, 52(2):499–515.
  * Meitz and Saikkonen, (2021) Meitz, M. and Saikkonen, P. (2021).  Testing for observation-dependent regime switching in mixture autoregressive models.  Journal of Econometrics, 222(1):601–624.
  * Mélard et al., (2006) Mélard, G., Roy, R., and Saidi, A. (2006).  Exact maximum likelihood estimation of structured or unit root multivariate time series models.  Computational statistics & data analysis, 50(11):2958–2986.
  * Menchetti et al., (2023) Menchetti, F., Cipollini, F., and Mealli, F. (2023).  Combining counterfactual outcomes and arima models for policy evaluation.  The Econometrics Journal, 26(1):1–24.
  * Mensi et al., (2023) Mensi, W., Aslan, A., Vo, X. V., and Kang, S. H. (2023).  Time-frequency spillovers and connectedness between precious metals, oil futures and financial markets: Hedge and safe haven implications.  International Review of Economics & Finance, 83:219–232.
  * Mertens and Ravn, (2010) Mertens, K. and Ravn, M. O. (2010).  Measuring the impact of fiscal policy in the face of anticipation: a structural var approach.  The Economic Journal, 120(544):393–413.
  * Mertens and Ravn, (2013) Mertens, K. and Ravn, M. O. (2013).  The dynamic effects of personal and corporate income tax changes in the united states.  American economic review, 103(4):1212–1247.
  * Miao et al., (2023) Miao, K., Phillips, P. C., and Su, L. (2023).  High-dimensional vars with common factors.  Journal of Econometrics, 233(1):155–183.
  * Montiel Olea and Plagborg-Moller, (2021) Montiel Olea, J. L. and Plagborg-Moller, M. (2021).  Local projection inference is simpler and more robust than you think.  Econometrica, 89(4):1789–1823.
  * Moon and Schorfheide, (2002) Moon, H. R. and Schorfheide, F. (2002).  Minimum distance estimation of nonstationary time series models.  Econometric Theory, 18(6):1385–1407.
  * Morley and Piger, (2012) Morley, J. and Piger, J. (2012).  The asymmetric business cycle.  Review of Economics and Statistics, 94(1):208–221.
  * Müller and Petalas, (2010) Müller, U. K. and Petalas, P.-E. (2010).  Efficient estimation of the parameter path in unstable time series models.  The Review of Economic Studies, 77(4):1508–1539.
  * Ng and Perron, (2001) Ng, S. and Perron, P. (2001).  Lag length selection and the construction of unit root tests with good size and power.  Econometrica, 69(6):1519–1554.
  * Nyberg and Rauhala, (2022) Nyberg, H. and Rauhala, S. (2022).  A structural vector autoregression containing positive-valued components.  Available at SSRN.
  * Olea et al., (2021) Olea, J. L. M., Stock, J. H., and Watson, M. W. (2021).  Inference in structural vector autoregressions identified with an external instrument.  Journal of Econometrics, 225(1):74–87.
  * Pagan and Pesaran, (2008) Pagan, A. R. and Pesaran, M. H. (2008).  Econometric analysis of structural systems with permanent and transitory shocks.  Journal of Economic Dynamics and control, 32(10):3376–3395.
  * Paparoditis, (2018) Paparoditis, E. (2018).  Sieve bootstrap for functional time series.  The annals of Statistics, 46(6B):3510–3538.
  * Paruolo, (1997) Paruolo, P. (1997).  Asymptotic inference on the moving average impact matrix in cointegrated 1 (1) var systems.  Econometric Theory, 13(1):79–118.
  * Paruolo, (2000) Paruolo, P. (2000).  Asymptotic efficiency of the two stage estimator in i (2) systems.  Econometric Theory, 16(4):524–550.
  * Paruolo and Rahbek, (1999) Paruolo, P. and Rahbek, A. (1999).  Weak exogeneity in i (2) var systems.  Journal of Econometrics, 93(2):281–308.
  * Pesaran et al., (2000) Pesaran, M. H., Shin, Y., and Smith, R. J. (2000).  Structural analysis of vector error correction models with exogenous i (1) variables.  Journal of econometrics, 97(2):293–343.
  * Petrova, (2019) Petrova, K. (2019).  A quasi-bayesian local likelihood approach to time varying parameter var models.  Journal of Econometrics, 212(1):286–306.
  * Petrova, (2022) Petrova, K. (2022).  Asymptotically valid bayesian inference in the presence of distributional misspecification in var models.  Journal of Econometrics, 230(1):154–182.
  * Petursson and Slok, (2001) Petursson, T. G. and Slok, T. (2001).  Wage formation and employment in a cointegrated var model.  The Econometrics Journal, 4(2):191–209.
  * Phillips, (1989) Phillips, P. C. (1989).  Partially identified econometric models.  Econometric Theory, 5(2):181–240.
  * Phillips, (1991) Phillips, P. C. B. (1991).  Optimal inference in cointegrated systems.  Econometrica: Journal of the Econometric Society, pages 283–306.
  * Phillips, (1998) Phillips, P. C. B. (1998).  Impulse response and forecast error variance asymptotics in nonstationary vars.  Journal of econometrics, 83(1-2):21–56.
  * Phillips and Hansen, (1990) Phillips, P. C. B. and Hansen, B. E. (1990).  Statistical inference in instrumental variables regression with i (1) processes.  The review of economic studies, 57(1):99–125.
  * Phillips et al., (2020) Phillips, P. C. B., Leirvik, T., and Storelvmo, T. (2020).  Econometric estimates of earth’s transient climate sensitivity.  Journal of Econometrics, 214(1):6–32.
  * Plagborg-Møller and Wolf, (2021) Plagborg-Møller, M. and Wolf, C. K. (2021).  Local projections and vars estimate the same impulse responses.  Econometrica, 89(2):955–980.
  * Plagborg-Møller and Wolf, (2022) Plagborg-Møller, M. and Wolf, C. K. (2022).  Instrumental variable identification of dynamic variance decompositions.  Journal of Political Economy, 130(8):2164–2202.
  * Pretis, (2020) Pretis, F. (2020).  Econometric modelling of climate systems: The equivalence of energy balance models and cointegrated vector autoregressions.  Journal of Econometrics, 214(1):256–273.
  * Pretis, (2021) Pretis, F. (2021).  Exogeneity in climate econometrics.  Energy Economics, 96:105122.
  * Primiceri, (2005) Primiceri, G. E. (2005).  Time varying structural vector autoregressions and monetary policy.  The Review of Economic Studies, 72(3):821–852.
  * Rehman et al., (2018) Rehman, M. U., Shahzad, S. J. H., Uddin, G. S., and Hedström, A. (2018).  Precious metal returns and oil shocks: A time varying connectedness approach.  Resources Policy, 58:77–89.
  * Rousseeuw, (1984) Rousseeuw, P. J. (1984).  Least median of squares regression.  Journal of the American statistical association, 79(388):871–880.
  * Rubio-Ramirez et al., (2010) Rubio-Ramirez, J. F., Waggoner, D. F., and Zha, T. (2010).  Structural vector autoregressions: Theory of identification and algorithms for inference.  The Review of Economic Studies, 77(2):665–696.
  * Rubio-Ramirez et al., (2005) Rubio-Ramirez, J. F., Waggoner, D. F., and Zha, T. A. (2005).  Markov-switching structural vector autoregressions: theory and application.
  * Rygh Swensen, (2022) Rygh Swensen, A. (2022).  On causal and non-causal cointegrated vector autoregressive time series.  Journal of Time Series Analysis, 43(2):178–196.
  * Safikhani et al., (2022) Safikhani, A., Bai, Y., and Michailidis, G. (2022).  Fast and scalable algorithm for detection of structural breaks in big var models.  Journal of Computational and Graphical Statistics, 31(1):176–189.
  * Saikkonen, (1995) Saikkonen, P. (1995).  Problems with the asymptotic theory of maximum likelihood estimation in integrated and cointegrated systems.  Econometric Theory, 11(5):888–911.
  * Samuelson, (1941) Samuelson, P. A. (1941).  Conditions that the roots of a polynomial be less than unity in absolute value.  The Annals of Mathematical Statistics, 12(3):360–364.
  * Sargan, (1983) Sargan, J. D. (1983).  Identification and lack of identification.  Econometrica: Journal of the Econometric Society, pages 1605–1633.
  * Sengupta and Kay, (1989) Sengupta, D. and Kay, S. (1989).  Efficient estimation of parameters for non-gaussian autoregressive processes.  IEEE Transactions on Acoustics, Speech, and Signal Processing, 37(6):785–794.
  * She et al., (2022) She, R., Mi, Z., and Ling, S. (2022).  Whittle parameter estimation for vector arma models with heavy-tailed noises.  Journal of Statistical Planning and Inference, 219:216–230.
  * Sims, (2012) Sims, E. R. (2012).  News, non-invertibility, and structural vars.  In DSGE Models in Macroeconomics: Estimation, Evaluation, and New Developments, pages 81–135. Emerald Group Publishing Limited.
  * Soderstrom et al., (1978) Soderstrom, T., Ljung, L., and Gustavsson, I. (1978).  A theoretical analysis of recursive identification methods.  Automatica, 14(3):231–244.
  * Soenen and Vander Vennet, (2022) Soenen, N. and Vander Vennet, R. (2022).  Ecb monetary policy and bank default risk.  Journal of International Money and Finance, 122:102571.
  * Stock and Watson, (2018) Stock, J. H. and Watson, M. W. (2018).  Identification and estimation of dynamic causal effects in macroeconomics using external instruments.  The Economic Journal, 128(610):917–948.
  * Tao and Yu, (2020) Tao, Y. and Yu, J. (2020).  Model selection for explosive models.  In Essays in honor of Cheng Hsiao, volume 41, pages 73–103. Emerald Publishing Limited.
  * Tchatoka and Dufour, (2013) Tchatoka, F. D. and Dufour, J.-M. (2013).  On the finite-sample theory of exogeneity tests with possibly non-gaussian errors and weak identification.
  * Vegh and Vuletin, (2015) Vegh, C. A. and Vuletin, G. (2015).  How is tax policy conducted over the business cycle?  American Economic Journal: Economic Policy, 7(3):327–370.
  * Velasco and Lobato, (2018) Velasco, C. and Lobato, I. N. (2018).  Frequency domain minimum distance inference for possibly noninvertible and noncausal arma models.  The Annals of Statistics, 46(2):555–579.
  * Vermeulen and Vansteelandt, (2015) Vermeulen, K. and Vansteelandt, S. (2015).  Bias-reduced doubly robust estimation.  Journal of the American Statistical Association, 110(511):1024–1036.
  * Virolainen, (2020) Virolainen, S. (2020).  Structural gaussian mixture vector autoregressive model with application to the asymmetric effects of monetary policy shocks.  arXiv preprint arXiv:2007.04713.
  * Watson, (1994) Watson, M. W. (1994).  Vector autoregressions and cointegration.  Handbook of econometrics, 4:2843–2915.
  * Yamamoto and Horie, (2023) Yamamoto, Y. and Horie, T. (2023).  A cross-sectional method for right-tailed panic tests under a moderately local to unity framework.  Econometric Theory, 39(2):389–411.
  * Yoon et al., (2019) Yoon, S.-M., Al Mamun, M., Uddin, G. S., and Kang, S. H. (2019).  Network connectedness and net spillover between financial and commodity markets.  The North American Journal of Economics and Finance, 48:801–818.
  * Zhang, (2023) Zhang, Y. (2023).  Statistical inference of high-dimensional vector autoregressive time series with non-iid innovations.  arXiv preprint arXiv:2310.07364.