# Mining Label Distribution Drift in Unsupervised Domain Adaptation

Peizhao Li † Brandeis University †, Indiana University - Purdue University
Indianapolis ‡
{peizhaoli<EMAIL_ADDRESS><EMAIL_ADDRESS>Zhengming Ding ‡ Brandeis
University †, Indiana University - Purdue University Indianapolis ‡
{peizhaoli<EMAIL_ADDRESS><EMAIL_ADDRESS>and Hongfu Liu † Brandeis
University †, Indiana University - Purdue University Indianapolis ‡
{peizhaoli<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Unsupervised domain adaptation targets to transfer task knowledge from labeled
source domain to related yet unlabeled target domain, and is catching
extensive interests from academic and industrial areas. Although tremendous
efforts along this direction have been made to minimize the domain divergence,
unfortunately, most of existing methods only manage part of the picture by
aligning feature representations from different domains. Beyond the
discrepancy in feature space, the gap between known source label and unknown
target label distribution, recognized as label distribution drift, is another
crucial factor raising domain divergence, and has not been paid enough
attention and well explored. From this point, in this paper, we first
experimentally reveal how label distribution drift brings negative effects on
current domain adaptation methods. Next, we propose Label distribution
Matching Domain Adversarial Network (LMDAN) to handle data distribution shift
and label distribution drift jointly. In LMDAN, label distribution drift
problem is addressed by the proposed source samples weighting strategy, which
select samples to contribute to positive adaptation and avoid negative effects
brought by the mismatched in label distribution. Finally, different from
general domain adaptation experiments, we modify domain adaptation datasets to
create the considerable label distribution drift between source and target
domain. Numerical results and empirical model analysis show that LMDAN
delivers superior performance compared to other state-of-the-art domain
adaptation methods under such scenarios.

##  1 Introduction

Figure 1: Domain adaptation with and w/o label distribution drift. Source and
target domain differ in the color of borders of circles. Circles with
different colors inside denote different categories, and the size indicates
the number of samples within that category. Straight lines denote decision
boundary learned by the classifier. Adaptation under label distribution drift
makes feature misaligned in categorical level and decision boundary unapplied
to target domain.

Domain adaptation is a fundamental research topic in the machine learning and
transfer learning field, and continually draws attention from academic and
industrial communities [27, 2]. It aims to build models on labeled source data
and related target data, then make models adapt and generalize on target
domain. Progress along this direction can be served for many downstream tasks,
including image-to-image translation [36], image segmentation [24], arrhythmia
detection [13], and so on. Different settings for domain adaptation are
applicable for complicated real-world problems [18, 20, 28, 9]. Among these
settings, unsupervised domain adaptation, containing no label but only samples
in target domain, is a challenging but practical one, owing to actual scenes
are suffering from label-scarcity.

The mitigation toward domain shift, which aims to reduce the domain divergence
between source and target, is the primary solution for unsupervised domain
adaptation problems. Existing methods [14, 21, 12, 23, 33, 22, 34] mainly
focus on alleviating negative effects brought by domain shift in feature
representations. They reduce the discrepancy by pushing feature distribution
from two separate domains close to each other. Consequently, models are
expected to be generalized favorably to a related target data distribution.
Adversarial learning is recently introduced into domain adaptation with
promising performance [12, 32, 22, 19]. By executing generated features to
confuse the discriminator, meanwhile forming the discriminator to distinguish
from source to target features, adversarial training aligns features through
the min-max game and delivers domain-invariant feature representations.

However, existing methods mainly consider feature level alignment, which is
not enough to guarantee the success of a positive adaptation. As another
component of domain shift, the disparity in distributions of labels between
source and target domain, i.e., the number of samples in each category differs
from source to target, is known as label distribution drift in corresponding
to the shift in the distribution of samples across domains. As presented in
Figure 1, label distribution drift brings negative effects from two aspects.
First, features belong to a large-size category in target domain are expected
to approach features in mismatched categories in source domain due to the
imbalanced adaptation toward label distribution. As a result, the alignment
corrupts feature representations of those misaligned samples. Second, decision
boundary of the classifier is only trained on labeled source samples, and is
not applicable to target domain when label distributions differ significantly.
These two inside reasons make the adaptation power down under label
distribution drift scenario. As the complements, empirical evidence in Section
3 also demonstrates that huge label distribution drift between two sets of
data brings a significant drop into the capacity of adaptation, and even leads
models to perform worse than none-transfer methods. More challenging,
different from sample distribution shift between domains, label distribution
cannot be aligned directly by existing methods because of the unknown target
label distribution, and it is more difficult to address this problem when a
considerable label distribution drift exists.

In this paper, we move a further step toward huge label distribution drift in
the unsupervised domain adaptation setting, and manage data shift together
with label distribution drift in a unified framework. As mentioned before,
domain adaptation with only feature space alignment is not enough. Therefore
we try to align two domains on the premise that corresponding label
distributions are roughly matched, and continually alleviate data shift and
label distribution drift simultaneously during training. To this end, we
propose the Label distribution Matching Domain Adversarial Network (LMDAN). To
be specific, we propose a novel weighting strategy for source samples re-
weighting, which mining samples that contribute to positive adaptation while
mitigating negative effects comes from aligning irrelevant classes across
domains. The proposed weighting function contributes to both the adversarial
feature alignment and classified boundary learning, hence addresses the two-
fold negative impacts brought by huge label distribution drift simultaneously.
In summary, we highlight major contributions of this paper in three folds as
follow.

  * •

We experimentally investigate the negative impact brought by label
distribution drift under current state-of-the-art domain adaptation methods,
which, to our best knowledge, has not been well explored.

  * •

We propose a label matching strategy to re-weight source samples, that enables
the source label distribution match with the unknown target one during adapted
process, and addresses the two-fold negative impacts brought by huge label
distribution drift simultaneously.

  * •

Different from experiments in previous literature, we evaluate the proposed
method on three benchmark datasets with manual modifications to simulate
considerable label distribution drift, and LMDAN achieves leading performance
compared to state-of-the-art domain adaptation methods. Additionally, we
provide comprehensive analyses for the proposed method.

##  2 Related Work

According to domain divergence, we introduce related works on unsupervised
domain adaptation in terms of feature space and label space alignment, then
highlight the differences between existing works and the proposed method.

Feature alignment aims to reduce the domain divergence in feature space.
Traditional methods construct projections for two domains, mapping two feature
distributions into the manifold space or subspace to address the domain shift
problem [14, 11, 10, 8]. Recently, Long et al. [21, 23] use deep models to
reduce the discrepancy between feature spaces in multiply layer levels.
Further, by the success of Generative Adversarial Network [15], adversarial
learning in deep models for unsupervised domain adaptation are continually
deliver favorable performance [22, 5, 34]. Ganin et al. [12] are the first to
employ adversarial learning based domain adaptation model and paves to many
following works. Tzeng et al. [32] uses two separate encoders for adaptation
while decomposing the transfer process from an end-to-end fashion. Besides,
the incorporation of conditional distribution [22, 6] in adaptation is also a
promising way to reach domain-invariant representations. Optimal transport for
domain adaptation [7, 30, 3, 1] is another interesting line of research, where
source samples are mapped into target domain with minimal cost transportation.
Although great efforts have been made to seek better feature alignment, only
handling the feature divergence is not enough to guarantee an adaptation
without any negative transfer.

Label distribution drift is another problem in domain adaptation but with less
exploration. The problem results from the divergence between known source
label and unknown target label distribution. Liang et al. [13] focus on the
negative transfer and imbalanced distributions in multi-source transfer
learning, while Ming et al. [26] exploit label and structural information
within and across domains based on maximum mean discrepancy. Similarly, Yan et
al. [33] introduce class-specific auxiliary weights into original maximum mean
discrepancy to exploit the class prior probability in source and target
domain. Moreover, Chen et al. [4] employ a simple re-weighting function in
Earth-Mover distance reduction. Although some pioneering works have touched
the label distribution drift problem, they only explore some preliminary
scenarios and leave cases with a vast gap in label distribution across
domains.

Different from existing studies, We tackle the domain adaptation problem under
considerable label distribution drift situations and conduct a comprehensive
cognition on label distribution drift from both experimentally and
methodological perspectives. Based on these, we propose a novel label matching
strategy by continually seeking for samples that are benefit to positive
adaptation and simultaneously prevent from the negative transfer.

##  3 Motivation

In this section, we firstly illustrate the label distribution drift problem in
terms of theoretical and practical perspectives, and further present our
problem formulation.

###  3.1 Notation

We start from the basic notations. Consider taking $n_{s}$ labeled samples
$\left\\{{(x_{i}^{s},y_{i}^{s})}\right\\}^{n_{s}}_{i=1}$$\in$$(\mathcal{X}\times\mathcal{Y})^{n_{s}}$
under domain distribution $\mathscr{D}^{s}$, where $\mathcal{X}$ and
$\mathcal{Y}$ denotes the corresponding input and label space, and $n_{t}$
unlabeled samples with the same label set as source samples
$\left\\{{x_{j}^{t}}\right\\}^{n_{t}}_{j=1}$$\in$$\mathcal{X}^{n_{t}}$ taken
from target domain distribution $\mathscr{D}^{t}$. The goal of unsupervised
domain adaptation is to utilize labeled source data for unlabeled target
samples predictions. Suppose a encoder $\mathcal{F}$ that is designed for
projecting samples drawn i.i.d. from $\mathscr{D}^{s}$ and $\mathscr{D}^{t}$
to a shared feature representation space.

###  3.2 Label Distribution Drift

Figure 2: Performance of DANN under varying degrees of label distribution
drift on Office-31 dataset from source domain Amazon to target domain Webcam.
The black line indicates training with the original label distribution, while
red and blue lines denote sample drop rates by 50% and 75%, respectively.
Solid lines indicate that dropped samples come from the first 15 classes for
both source and target domain, while dashed lines indicate dropped samples
come from the first 15 classes and last 16 classes in source and target
domain, respectively. The legend provides the measurement of KL-divergence
between source and target label distribution.

Tremendous efforts have been made to explore solutions for unsupervised domain
adaptation. Unfortunately, most existing studies only focus on feature space
divergence (denotes as $Div$), that is, minimizing
$Div(\mathbb{E}_{x\sim\mathscr{D}^{s}}[\mathcal{F}(x)],\mathbb{E}_{x\sim\mathscr{D}^{t}}[\mathcal{F}(x)])$,
while ignore the negative effects brought by label distribution drift.

From the theoretical perspective, a generalization bound for domain adaptation
problem towards the expected error on target samples [2] is given as follows:

$\begin{split}{\epsilon^{t}}(h)\,\leq{\,}{\epsilon^{s}}(h)\,+\,{\frac{1}{2}}d_{\mathcal{H}\Delta\mathcal{H}}(\mathscr{D}^{s},\mathscr{D}^{t})\,+\,{\lambda^{*}}\,+\,\Omega,\end{split}$
(1)

where ${\epsilon^{t}}(h)$ and ${\epsilon^{s}}(h)$ are expected errors on
target and source domain, respectively; $\mathcal{H}$ is hypothesis space,
$\lambda^{*}:={\epsilon^{s}}(h^{*})+{\epsilon^{t}}(h^{*})$ is the optimal
joint risk among source and target samples, and $\Omega$ is a constant related
to the numbers of samples, dimensions, confidence level, and VC-dimension of
$\mathcal{H}$. With the assumption that source and target label distribution
are close enough, methods with only feature alignment could achieve small
target error $\epsilon^{t}(h)$ by reducing domain distance term
$\frac{1}{2}d_{\mathcal{H}\Delta\mathcal{H}}(\mathscr{D}^{s},\mathscr{D}^{t})$.
However, as pointed out by Zhao et al. [35], when the above assumption does
not hold, the huge label distribution gap between two domains leads the joint
error term $\lambda^{*}$ increase oppositely during the optimization towards
domain distance term, and might counteract with the reduction in domain
distance term, which even increases the value of the upper bound.

The above perspective is supported by practical evidence. Figure 2 shows the
performance of the classical domain adversarial method DANN [12] on varying
degrees of label distribution drift on Office-31 dataset [31]. Solid and
dashed lines represent slight and huge label distribution drift, while all
experiments are with the same size of training data. Two observations are
quite clear: (1) Compared to the black line training on original dataset, the
solid red and blue lines deliver similar results, indicating that even if size
of classes in the same domain are imbalanced, high performance is still
achieved by DANN under the scenario of similar source and target label
distribution. Although training sets of solid lines contain different samples,
dropped samples do not bring negative effects into the performance; (2) The
gap between solid and dashed lines indicates that when there is a significant
label distribution drift between two domains, the performance drops
dramatically. The adaptation performance of DANN becomes much worse with a
larger divergence between source and target label distribution. More
experimental results reveal the negative effects from label distribution drift
can be referred in Section 5.

Based on the above theoretical analysis and practical evidence, only realizing
the alignment on feature distribution is still far away from the success of
the adaptation. The above exploration motivates us to provide the unified
problem formulations of unsupervised domain adaptation on both data
distribution shift and label distribution drift.

###  3.3 Problem Formulation

Two facing challenges for unsupervised domain adaptation problems brought by
domain divergence are as follows.

Data Distribution Shift. Usually samples and extracted feature distributions
from source to target domain are different, i.e.,
$\mathcal{P}(x^{s})\neq\mathcal{P}(x^{t})$, and prohibits models to learn a
classifier with labeled source samples that can directly applied for target
sample predictions. For this reason, feature alignment can be achieved by
minimizing the divergence

$Div(\mathbb{E}_{x^{s}\sim\mathscr{D}_{s}}[\mathcal{F}(x^{s})],\mathbb{E}_{x^{t}\sim\mathscr{D}_{t}}[\mathcal{F}(x^{t})])\
,$

or by minimization conditional distribution divergence

$Div(\mathbb{E}_{(x^{s},y^{s})\sim\mathscr{D}^{s}}[\mathcal{F}(x^{s})|y^{s}],\mathbb{E}_{(x^{t},y^{t})\sim\mathscr{D}^{t}}[\mathcal{F}(x^{t})|y^{t}]),\
\forall{y^{s}=y^{t}},$

to narrow the data distribution shift for domain adaptation.

Label Distribution Drift. Beyond the inconsistency in feature space, domain
divergence also occurs in label space, where
$\mathcal{P}(y^{s})\neq\mathcal{P}(y^{t})$. It is more challenging to handle
label distribution drift than data shift due to $\mathcal{P}(y^{t})$ is an
agnostic distribution in unsupervised domain adaptation.

When considerable label distribution divergence exists, the excessive
optimization towards feature distribution alignment will lead to the
minimization of

$Div(\mathbb{E}_{(x^{s},y^{s})\sim\mathscr{D}^{s}}[\mathcal{F}(x^{s})|y^{s}],\mathbb{E}_{(x^{t},y^{t})\sim\mathscr{D}^{t}}[\mathcal{F}(x^{t})|y^{t}]),\
\forall{y^{s}\neq{y^{t}}},$

which aligns target representations to irrelevant classes in source domain
during training. These are enable to corrupt categorical feature
representations and rise increasing predicted error on predicting $y^{\prime}$
in target domain, and further deliver negative effects to the adapted process.

Most unsupervised domain adaptation methods consider domain divergence merely
from data shift, and take no drift in label space for granted. It is not only
far away from real scenarios but also suffers from the degraded performance
due to label distribution drift (See Figure 2). Consider negative effects can
be easily accessed when the inconsistent label distribution and the alignment
between irrelevant classes across domains exist, not all samples can be used
during the training process for positive adaptation only comes from the part
of correctly matched pair of source-target samples. Consequently, we try to
exploit and emphasize the part of correctly matched samples in two domains,
while mitigating the alignment on class-mismatched samples, thus further
increase the ratio of positive adaptation and avoid the negative transfer
brought by the alignment between irrelevant categories across source and
target domains. This can be view as an unsupervised sample selection that we
are continually seeking for samples that benefit to adaptation and avoid
negative transfer concurrently during training.

Figure 3: The framework of Label distribution Matching Domain Adversarial
Network (LMDAN). Adversarial feature alignment aims to obtain domain-invariant
features and to reduce data shift, while the matching in label distribution
exploit class-wise weights. These weights dually contribute to decision
boundary adaptation and feature alignment.

##  4 Methodology

We start this section by elaborating on the proposed LMDAN framework, then
provide a detailed description for the label distribution matching and source
sample weighting strategy, followed by overall objective function and the
corresponding optimization solution for LMDAN.

###  4.1 Overview

Figure 3 shows the framework of the proposed LMDAN. It minimizes the domain
divergence embedded in feature space on the premise of close source-target
label distribution. Specifically, to align the source and target domain under
label distribution drift, LMDAN contains two interactive parts: adversarial
training that for domain-invariant features generation, and the class-wise re-
weighting strategy through the optimal assignment for source sample selection.
In adversarial feature alignment, the encoder $\mathcal{F}$ tries to extract
feature $\mathcal{F}(x^{s})$ and $\mathcal{F}(x^{t})$ from two domains and
confuse the discriminator $\mathcal{D}$, while $\mathcal{D}$ tries to
distinguish $\mathcal{F}(x^{s})$ and $\mathcal{F}(x^{t})$ from each other.
Finally, $\mathcal{F}$ is trained to map data distribution from two domains
close enough. In source samples weighting part, by adding class-wise weights
on both adversarial training and supervision on the classifier, we manipulate
feature alignment in adversarial training and decision boundary of the
classifier to tackle the label drift scenario simultaneously. The dual
weighting strategy makes the network adapt to target domain by two sides: (1)
The weighting for min-max game emphasize features in the same category get
closer across domains, at the same time mitigate the misalignment, and (2) The
weighting for classifier makes decision boundary adapt to the target label
distribution. In the following, we emphatically illustrate the source sample
weighting strategy, then provide details for adversarial training and overall
loss functions.

###  4.2 Label Distribution Matching

Label distribution matching is one of the crucial components in the LMDAN
framework. It disposes of label distribution drift towards source-target
sample matching. Here, we expect to exploit samples in the parts of classes
matched across source and target domain by the optimal assignment, then
enlarge matched classes and shrink the size of less relevant classes in source
domain. As a result, samples in source domain engage in the adversarial
feature alignment are able to approach to target domain in terms of the label
distribution, and further contribute to increase positive and mitigate
negative transfer during the training process.

To achieve this, we employ the classified probability $g$ with
$||g||_{1}$$=$$1$ of every sample to measure the degree of matching. Based on
the measurement of distance and optimal matching, mismatched pairs result in a
larger distance, while matched pairs perform inversely. Consider a cost
function $c:\mathcal{C}\times\mathcal{C}\to\mathbb{R}^{+}$ and $g_{i}^{s}$ and
$g_{j}^{t}$ the classified probabilities obtained by the classifier
$\mathcal{G}$ for source sample $x_{i}^{s}$ and target sample $x_{j}^{t}$,
respectively, and the output space $\mathcal{C}:g\in\mathcal{C}$. Based on
optimal assignment [17], we seek for a joint probability distribution $\gamma$
according to $g^{s}$ and $g^{t}$:

$\vspace{-1mm}\gamma^{*}\,=\,\mathop{\arg\min}_{\gamma\in\prod(\mathcal{C}\times\mathcal{C})}\int_{\mathcal{C}\times\mathcal{C}}c(g^{s},g^{t})d\gamma(g^{s},g^{t})\
.$ (2)

This indicates the optimal assignment based on classified probabilities from
source to target with the least cost.

As for the discrete version for implementation, we employ euclidean distance
to build the cost matrix $M=\\{m_{ij}\\}\in\mathbb{R}^{{n_{s}}\times{n_{t}}}$
between source and target domain,

$\vspace{-1.0mm}m_{ij}=c(g_{i}^{s},g_{j}^{t})=||g_{i}^{s}-g_{j}^{t}||_{2}\ ,$
(3)

and other distance functions can be used as well. Based on $M$, the optimal
assignment is written as:

$\displaystyle\gamma^{*}=\mathop{\arg\min}_{\gamma\in\mathbb{R}^{n_{s}\times{n_{t}}}}\left\langle\gamma,M\right\rangle_{F},$
(4)
$\displaystyle\textup{s.t.}:\gamma\textbf{1}_{n_{t}}=\sum^{n_{s}}_{i}{g^{s}_{i}},\
\gamma\textbf{1}_{n_{s}}=\sum^{n_{t}}_{j}{g^{t}_{j}}\ ,$

where $\left\langle\cdot,\cdot\right\rangle_{F}$ indicates Frobenius inner
product, and $\textbf{1}_{n}$ is vector of ones with $n$-dimension.

Input: $\left\\{{(x_{i}^{s},y_{i}^{s})}\right\\}^{n_{s}}_{i=1}$ and
$\left\\{{x_{j}^{t}}\right\\}^{n_{t}}_{j=1}$

Output: $\left\\{{\hat{y}_{j}^{t}}\right\\}^{n_{t}}_{j=1}$

while  _not converge_ do

Build the min-batch set {$x_{i}^{s}$, $y_{i}^{s}$, $x_{j}^{t}$};

Compute $w_{k}$ by Eq. (4), (5), &(6);

Optimize $\mathcal{D}$, $\mathcal{F}$ and $\mathcal{G}$ by Eq. (7);

end while

Predict target label by $\hat{y}_{j}^{t}$ =
$\arg\max\mathcal{G}(\mathcal{F}(x_{j}^{t}))$.

Algorithm 1 LMDAN

We then incorporate the distance within classified probabilities into the
optimal assignment plan and make the conjunct term guide class-wise weights
for each class. We obtain the weight guiding matrix
$\textbf{T}=\\{t_{ij}\\}\in\mathbb{R}^{n_{s}\times{n_{t}}}$ by

$\textbf{T}=\gamma^{*}\,\circ\,\textbf{M}\ ,$ (5)

where $\circ$ denotes the Hadamard product. By matching classification
probabilities with the minimal cost, weight guiding matrix provides guidance
for misaligned samples. Moreover, following the above step, we compute the
class-wise weight $w_{k}$ for class with index $k$ in source domain by:

$\vspace{-0.5mm}w_{k}=\Big{(}({\sum_{i=1}^{n_{s}}\mathbbm{1}_{y_{i}^{s}=k}})^{\alpha}\cdot{\sum_{i=1}^{n_{s}}t_{ij}\mathbbm{1}_{y_{i}^{s}=k}}\Big{)}^{-1},$
(6)

where $\mathbbm{1}$ is the indicator function. Note that $w_{k}$ consists of
two parts, where the first term manages the imbalanced class size within
source domain itself, and the second awards or punishes the matched or
mismatched pairs between source and target accordingly. $\alpha$ is the
parameter to control the influence of source class imbalanced scale.

Using weights in terms of categories according to the optimal matching toward
classified probabilities, we are able to distinguish classes that misaligned
and less relevant to positive transfer from well-aligned ones during training.
By re-weighting samples in source domain by class-wise weights, the size or
corresponding categories is enlarged or shrunk accordingly, then further push
the source label distribution to the unknown target one dynamically.

Table 1: Results for unsupervised domain adaptation under label distribution drift on Office-31[0.75;0.75] dataset Method |  A $\rightarrow$ W |  A $\rightarrow$ D |  W $\rightarrow$ A |  W $\rightarrow$ D |  D $\rightarrow$ A |  D $\rightarrow$ W | Average
---|---|---|---|---|---|---|---
ResNet-50 [16] |  66.1 $\pm$ 4.3 |  65.8 $\pm$ 1.5 |  53.3 $\pm$ 3.1 |  87.8 $\pm$ 2.9 |  53.0 $\pm$ 3.7 |  79.4 $\pm$ 2.6 |  67.6 $\pm$ 1.5
DANN [12] |  50.7 $\pm$ 2.6 |  54.0 $\pm$ 2.7 |  35.4 $\pm$ 3.4 |  62.6 $\pm$ 4.2 |  34.6 $\pm$ 3.8 |  56.3 $\pm$ 2.9 |  49.0 $\pm$ 0.8
JAN [23] |  51.2 $\pm$ 3.2 |  49.5 $\pm$ 2.4 |  46.1 $\pm$ 3.9 |  72.9 $\pm$ 4.1 |  40.9 $\pm$ 5.1 |  71.8 $\pm$ 2.6 |  55.4 $\pm$ 1.6
WMMD [33] |  39.1 $\pm$ 5.2 |  43.3 $\pm$ 4.1 |  38.4 $\pm$ 2.7 |  67.8 $\pm$ 4.8 |  34.1 $\pm$ 3.2 |  68.1 $\pm$ 7.1 |  48.5 $\pm$ 3.4
CDAN [22] |  65.7 $\pm$ 3.2 |  62.8 $\pm$ 4.8 |  52.5 $\pm$ 2.7 |  78.1 $\pm$ 4.7 |  39.8 $\pm$ 4.5 |  73.5 $\pm$ 4.4 |  62.1 $\pm$ 1.7
RAAN [4] |  59.4 $\pm$ 3.8 |  65.7 $\pm$ 2.9 |  48.5 $\pm$ 5.0 |  76.4 $\pm$ 3.5 |  45.8 $\pm$ 6.9 |  77.4 $\pm$ 3.6 |  62.2 $\pm$ 3.2
SymNets [34] |  57.1 $\pm$ 4.0 |  54.6 $\pm$ 2.7 |  41.9 $\pm$ 6.3 |  67.0 $\pm$ 5.1 |  32.4 $\pm$ 4.8 |  57.2 $\pm$ 6.7 |  51.7 $\pm$ 2.7
BSP [5] |  61.5 $\pm$ 2.1 |  58.9 $\pm$ 2.6 |  47.5 $\pm$ 3.2 |  85.0 $\pm$ 3.6 |  40.4 $\pm$ 2.9 |  84.1 $\pm$ 3.0 |  62.9 $\pm$ 2.2
LMDAN |  73.1 $\pm$ 1.7 |  71.0 $\pm$ 2.5 |  56.5 $\pm$ 2.4 |  84.4 $\pm$ 2.6 |  57.8 $\pm$ 4.9 |  88.8 $\pm$ 3.5 | 71.9 $\pm$ 2.1

###  4.3 Objective Function and Solution

Finally, we provide objective functions and the corresponding optimize
solution for LMDAN. We firstly calculate class-wise weights on each mini-batch
samples and then optimize toward min-max game in adversarial learning
integrates with subsequent classification by a dual weighting strategy. Loss
functions for LMDAN can be written as:

$\vspace{-1mm}\min_{\mathcal{F},\mathcal{G}}\
\mathcal{L}_{1}(\mathcal{F},\mathcal{G},\mathcal{D})+\lambda\mathcal{L}_{2}(\mathcal{F},\mathcal{D})\
\textup{and}\max_{\mathcal{D}}\mathcal{L}_{2}(\mathcal{F},\mathcal{D}),\textup{with}\\\
$ (7)
$\vspace{-1mm}\begin{split}&\mathcal{L}_{1}=\underset{(x^{s},y^{s})\sim{\mathscr{D}^{s}}}{\mathbb{E}}w_{i}\mathcal{L}\,(\mathcal{G}\,(\mathcal{F}(x^{s})),\,y^{s}),\\\
&\mathcal{L}_{2}=\underset{x^{s}\sim{\mathscr{D}^{s}}}{\mathbb{E}}w_{i}\log\left[\mathcal{D}\,(\mathcal{F}\,(x^{s}))\right]+\underset{x^{t}\sim{\mathscr{D}^{t}}}{\mathbb{E}}\log\left[1-\mathcal{D}\,(\mathcal{F}\,(x^{t}))\right],\end{split}$

where $w_{i}$ is the corresponding weight of the class where
$x^{s}\sim\mathscr{D}^{s}$ belongs to, and $\lambda$ is the trade-off
hyperparameter for classification loss and adversarial loss. In our objective
functions, the weighting strategy conducts in two places. The weighted
classifier $\mathcal{G}$ captures label distribution drift for better decision
boundary adaptation on target domain, and the weighted discriminator
$\mathcal{D}$ and encoder $\mathcal{F}$ further adjust feature alignment to
fit label distribution drift as well.

Algorithm 1 shows the complete optimization procedure. In our implementation,
we utilize cross-entropy loss as the loss function for $\mathcal{L}$, and set
the trade-off parameter $\lambda$ default to 1 for all experiments. Since the
complexity of the optimal assignment is not scalable to the whole dataset, the
mini-batch label matching is work as well. Two benefits are clear. Mini-batch
training makes the complexity of the optimal matching affordable in big data
adaptation. Besides, equivalent numbers of data points from source and target
domain can be sampled, rendering the matching and feature alignment balanced.
We use pre-trained ResNet-50 [16] as the feature extractor. Following by [12],
we set the initial learning rate $lr=0.01$ for SGD optimizer, then gradually
adjust the learning rate for the classifier by ${lr_{c}=lr(1+10p)^{-0.75}}$,
where $p$ is the training process changed from $0$ to $1$ linearly. The
learning rate for discriminator is
$lr_{d}=\frac{1-\exp(-10p)}{1+\exp(-10p)}lr$ and adjust accordingly.

##  5 Experimental Analysis

We firstly clarify the dataset details and corresponding modification for
experiments, then evaluate the performance of the proposed LMDAN by comparing
with other state-of-the-art unsupervised domain adaptation methods under the
huge label distribution drift scenario, and followed by comprehensive analyses
for LMDAN.

###  5.1 Dataset and Modification

Three widely-used real-world datasets are employed to evaluate the performance
of LMDAN and other competitive methods. (1) Office-31 [31] contains 4,652
images in total within 31 categories. The dataset contains three domains:
Amazon (A), Webcam (W), and DSLR (D), where images are broadly taken from
internet to real scenarios. (2) Visda-2017 [29] is a challenging domain
adaptation dataset, aiming to transfer knowledge from synthetic images (S) to
real images (R). It contains around 152,000 synthetic images of 3D models for
source domain, and 72,000 real images for target domain. Both two domains
contain 12 categories. (3) ImageCLEF-DA
111https://www.imageclef.org/2014/adaptation contains 600 images per domain
taken from three objects recognition datasets, Caltech-256 (C), ImageNet
ILSVRC 2012 (I), and Pascal VOC 2012 (P). In following experiments,
‘‘A$\rightarrow$W" denotes domain adaptation from source domain Amazon to
target domain Webcam.

Figure 4: Label distribution of 31 categories on original Office-31 and
modified Office-31[0.75;0.75] dataset.

Different from experiments in previous domain adaptation literature, we
simulate label distribution drift by randomly drop out 75% samples in the
first half of classes within source domain, and 75% samples in latter half of
classes in target domain, and note this modified dataset as
‘‘NAME[0.75;0.75]’’, where NAME is the name of original datasets. Figure 4
shows the label distribution on original Office-31 and modified
Office-31[0.75;0.75], respectively. The samples dropping process with
randomness is repeated five times, and we conduct experiments on all created
datasets while reporting the average performance and its fluctuation to
alleviate the sample selection bias.

Table 2: Results for unsupervised domain adaptation with label distribution drift on ImageCLEF-DA[0.75;0.75] dataset Method |  C $\rightarrow$ I |  C $\rightarrow$ P |  I $\rightarrow$ C |  I $\rightarrow$ P |  P $\rightarrow$ C |  P $\rightarrow$ I | Average
---|---|---|---|---|---|---|---
ResNet-50 [16] |  76.9 $\pm$ 3.2 |  63.8 $\pm$ 1.7 |  87.1 $\pm$ 1.8 |  71.3 $\pm$ 0.7 |  81.7 $\pm$ 3.7 |  73.4 $\pm$ 4.2 |  75.7 $\pm$ 2.1
DANN [12] |  47.4 $\pm$ 2.8 |  40.8 $\pm$ 2.7 |  55.0 $\pm$ 1.2 |  50.4 $\pm$ 2.3 |  55.0 $\pm$ 3.1 |  51.2 $\pm$ 3.6 |  50.0 $\pm$ 1.4
JAN [23] |  34.2 $\pm$ 2.8 |  27.9 $\pm$ 1.0 |  38.8 $\pm$ 3.8 |  49.0 $\pm$ 3.4 |  36.7 $\pm$ 4.4 |  44.1 $\pm$ 3.0 |  38.5 $\pm$ 0.7
WMMD [33] |  42.4 $\pm$ 1.1 |  30.4 $\pm$ 3.5 |  65.2 $\pm$ 3.9 |  70.8 $\pm$ 3.0 |  47.2 $\pm$ 3.5 |  56.4 $\pm$ 1.9 |  52.0 $\pm$ 2.9
CDAN [22] |  58.1 $\pm$ 3.8 |  52.2 $\pm$ 3.2 |  76.3 $\pm$ 4.0 |  62.7 $\pm$ 1.8 |  66.2 $\pm$ 9.5 |  59.2 $\pm$ 1.2 |  63.1 $\pm$ 2.2
RAAN [4] |  62.9 $\pm$ 1.3 |  54.6 $\pm$ 3.3 |  78.3 $\pm$ 1.7 |  63.6 $\pm$ 3.6 |  71.0 $\pm$ 6.6 |  65.4 $\pm$ 2.4 |  66.0 $\pm$ 2.3
SymNets [34] |  59.2 $\pm$ 5.0 |  53.8 $\pm$ 3.2 |  70.5 $\pm$ 3.9 |  57.2 $\pm$ 3.8 |  63.4 $\pm$ 7.6 |  54.3 $\pm$ 1.5 |  59.7 $\pm$ 1.4
BSP [5] |  52.6 $\pm$ 1.7 |  43.4 $\pm$ 2.5 |  70.5 $\pm$ 2.9 |  58.6 $\pm$ 4.6 |  67.0 $\pm$ 4.3 |  62.9 $\pm$ 1.8 |  59.2 $\pm$ 1.8
LMDAN | 79.1 $\pm$ 2.8 | 67.7 $\pm$ 2.7 | 89.8 $\pm$ 2.3 | 71.6 $\pm$ 2.8 | 88.1 $\pm$ 2.4 | 80.5 $\pm$ 1.0 | 79.5 $\pm$ 0.8

(a) Class-wise predicted accuracy.

(b) Accuracy during training.

(c) Performance on different divergences.

Figure 5: Performance of different unsupervised domain adaptation methods on
A$\rightarrow$W. (a) shows class-wise predicted accuracy on 31 classes, where
the first 15 classes have few source samples than the last 16 classes, (b)
demonstrates the variation of overall accuracy during training, and (c)
reports the performance with different levels of label distribution drift
including the original dataset, [0.25;0.25], [0.5;0.5], [0.625;0.625] and
[0.75;0.75]. X-axis denotes the KL-divergence values between source and target
label distribution corresponding to the value of 0.0390, 0.0882, 0.2817,
0.5085, and 0.8879.

###  5.2 Competitive Methods

We compare to seven recent deep unsupervised domain adaptation methods and
ResNet-50 trained only on source domain without adaptation. JAN [23] and WMMD
[33] are deep transfer models based on maximum mean discrepancy. They learn
the adaptation by aligning joint distributions of multiple domain-specific
layers. DANN [12], CDAN [22], SymNets [34], and BSP [5] are based on
adversarial training and make two feature spaces confuse the discriminator.
Further, CDAN [22] and SymNets [34] take the conditional feature distribution
into consideration and enhance the adaptation performance under normal domain
adaptation setting. Moreover, RAAN [4] reduces feature distribution divergence
by minimizing Earth-Mover distance. For all the competitive methods, we use
ResNet-50 as the feature extractor for apples to apples comparisons. We re-
implement WMMD and RAAN, and conduct experiments for the rest of the methods
with their open-source codes.

(a) ResNet-50

(b) DANN

(c) CDAN

(d) LMDAN

Figure 6: Feature space visualization on A$\rightarrow$W. Red and blue dots denote source and target samples, respectively. Table 3: Results for unsupervised domain adaptation on S$\rightarrow$R of VisDA-2017 dataset in original and [0.75;0.75]. Method | Original | [0.75;0.75]
---|---|---
ResNet-50 [16] | 44.4 |  41.0 $\pm$ 0.7
DANN [12] | 63.5 |  33.9 $\pm$ 1.3
JAN [23] | 61.6 |  27.7 $\pm$ 2.0
WMMD [33] | 45.8 |  28.4 $\pm$ 2.2
CDAN [22] | 66.8 |  38.5 $\pm$ 0.9
RAAN [4] | 59.0 |  50.9 $\pm$ 1.9
SymNets [34] | 51.6 |  22.6 $\pm$ 0.7
BSP [5] | 64.7 |  27.9 $\pm$ 3.2
LMDAN | 64.9 | 59.3 $\pm$ 1.1

###  5.3 Numerical Evaluation

Table 1 reports quantitative results for unsupervised domain adaptation on
Office-31[0.75;0.75] dataset. The performance of all competitive methods
significantly drops under the huge label distribution divergence and even
becomes worse than non-adapted ResNet-50. This indicates only aligning the
feature divergence is not enough for a positive adaptation, for label
distribution drift is also a crucial component of domain shift, and has not
been paid sufficient attention in domain adaptation area. To dispose of this
problem, LMDAN considers source-target sample pairs with different weights,
enlarges and shrinks weights for matched and mismatched samples on classified
probabilities. By this means, LMDAN outperforms other competitive methods by a
large margin. To be noticed, WMMD and RAAN also embed source sample re-
weighting strategies into training. However, their weights highly rely on
predictions of the target label distribution and make themselves struggle to
handle huge label distribution drift. Figure 5 shows more experimental details
on A$\rightarrow$W. The undesirable performance of other methods mainly
results from categories with large sizes on target domain. It is shown that
DANN and ResNet-50 return almost 0 accuracy on Class 5&15\. Thanks to label
distribution matching, LMDAN achieves much better predictions on these
categories. Moreover, Figure 5(b) shows an increasing performance of LMDAN
through iterations, indicating weighting source-target pairs by mini-batch
gradually narrows the gap in label space. It is expected to see that all
methods degrade with an increasing label distribution divergence in Figure
5(c), which demonstrate the divergence in label space has a huge impact on the
adaptation performance. LMDAN delivers robust results than others even under
huge label distribution gap, which is essential in practice due to the
agnostic target label distribution. The results in Table 2 and 3 provide more
promising results of LMDAN on ImageCLEF-DA[0.75;0.75] and
VisDA-2017[0.75;0.75].

###  5.4 Visualization

Figure 6 provides embedded feature space visualization results on
A$\rightarrow$W on Office-31[0.75;0.75] by t-SNE [25]. With huge label
distribution divergence between source and target domain, DANN and CDAN cannot
preserve the original categorical source structure due to the corruption of
representations by aligning mismatched features, and at the same time affect
source-target feature alignment dramatically. The negative effects in
adaptation further lead to inferior predicted performance on target data.
ResNet-50 preserves better source structural features but with less adaptation
on domain divergence. LMDAN re-weights samples in source domain, which not
only refines the classified decision boundary but also provides better-aligned
features.

Figure 7: Label matching on C$\rightarrow$I of ImageCLEF-DA[0.75;0.75], where
LMDAN enlarges the first six and shrinks the last six categories of source
data.

###  5.5 Distribution Matching

Figure 7 shows the effectiveness of LMDAN on label distribution matching.
ImageCLEF-DA[0.75;0.75] has few source samples in the first six categories and
but more source samples in rest categories, and target samples work inversely.
With label distribution matching in LMDAN, sizes of the first six categories
in source domain are assigned with larger weights, and this is equal to
enlarge the class size. On the contrary, the rest classes are shrunk to match
the target label distribution. Therefore, the source label distribution after
matching becomes similar to the target one. With matched distributions between
source and target domain in both feature and label space during training,
LMDAN delivers the positive transfer consistently.

###  5.6 Hyperparameter Analysis

LMDAN employs $\alpha$ as a hyper-parameter to balance the imbalance of the
source label distribution and source-target label distribution drift. Table 4
shows the performance of LMDAN on D$\rightarrow$A on modified Office-31 with
different $\alpha$. When $\alpha=0$, label distribution matching provides the
inferior performance since the poor performance from the imbalanced classifier
on target sample predictions. With an increasing $\alpha$, LMDAN gains
improvements with joint actions from source label distribution internal
balancing and source-target label distribution match; however, the first term
in Eq. (6) dominates weights when $\alpha$ goes too large. Based on the
hyperparameter analysis, we set $\alpha=2$.

Table 4: Analysis on $\alpha$ in LMDAN on D$\rightarrow$A. Divergence | $\alpha=0$ | $\alpha=1$ | $\alpha=2$ | $\alpha=4$
---|---|---|---|---
$[0.25;0.25]$ | 53.9 | 62.3 | 64.8 | 56.2
$[0.50;0.50]$ | 34.9 | 50.4 | 62.4 | 51.5
$[0.75;0.75]$ | 12.2 | 53.0 | 57.8 | 53.3

##  6 Conclusion

In this paper, we proposed Label distribution Matching Domain Adversarial
Network (LMDAN). To tackle considerable label distribution drift between two
domains, we designed the label distribution matching and weighting strategy
for source samples re-weighting, and match the known source label distribution
with the agnostic target one. These weights contributed to decision boundary
adaptation and adversarial feature alignment, thus minimized domain
divergence. Experimental results demonstrated the superior performance of
LMDAN over other methods.

## References

  * [1] Yogesh Balaji, Rama Chellappa, and Soheil Feizi.  Normalized wasserstein for mixture distributions with applications in adversarial learning and domain adaptation.  In Proceedings of The IEEE International Conference on Computer Vision, 2019.
  * [2] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan.  A theory of learning from different domains.  Machine Learning, 2010.
  * [3] Bharath Bhushan Damodaran, Benjamin Kellenberger, Remi Flamary, Devis Tuia, and Nicolas Courty.  Deepjdot: Deep joint distribution optimal transport for unsupervised domain adaptation.  In Proceedings of The European Conference on Computer Vision, 2018\.
  * [4] Qingchao Chen, Yang Liu, Zhaowen Wang, Ian Wassell, and Kevin Chetty.  Re-weighted adversarial adaptation network for unsupervised domain adaptation.  In Proceedings of The IEEE Conference on Computer Vision and Pattern Recognition, 2018.
  * [5] Xinyang Chen, Sinan Wang, Mingsheng Long, and Jianmin Wang.  Transferability vs. discriminability: Batch spectral penalization for adversarial domain adaptation.  In Proceedings of the 36th International Conference on Machine Learning, Proceedings of Machine Learning Research, 2019.
  * [6] Safa Cicek and Stefano Soatto.  Unsupervised domain adaptation via regularized conditional alignment.  In Proceedings of The IEEE International Conference on Computer Vision, October 2019.
  * [7] Nicolas Courty, Rémi Flamary, Devis Tuia, and Alain Rakotomamonjy.  Optimal transport for domain adaptation.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017\.
  * [8] Zhengming Ding and Yun Fu.  Deep transfer low-rank coding for cross-domain learning.  IEEE Transactions on Neural Networks and Learning Systems, 2019\.
  * [9] Zhengming Ding and Hongfu Liu.  Marginalized latent semantic encoder for zero-shot learning.  In Proceedings of The IEEE Conference on Computer Vision and Pattern Recognition, 2019.
  * [10] Zhengming Ding, Ming Shao, and Yun Fu.  Deep low-rank coding for transfer learning.  In Twenty-Fourth International Joint Conference on Artificial Intelligence, 2015.
  * [11] Basura Fernando, Amaury Habrard, Marc Sebban, and Tinne Tuytelaars.  Unsupervised visual domain adaptation using subspace alignment.  In Proceedings of The IEEE International Conference on Computer Vision, 2013.
  * [12] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky.  Domain-adversarial training of neural networks.  Journal of Machine Learning Research, 2016.
  * [13] Liang Ge, Jing Gao, Hung Ngo, Kang Li, and Aidong Zhang.  On handling negative transfer and imbalanced distributions in multiple source transfer learning.  Statistical Analysis and Data Mining: The ASA Data Science Journal, 2014.
  * [14] Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman.  Geodesic flow kernel for unsupervised domain adaptation.  In Proceedings of The IEEE Conference on Computer Vision and Pattern Recognition, 2012.
  * [15] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In Advances in Neural Information Processing Systems, 2014.
  * [16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of The IEEE Conference on Computer Vision and Pattern Recognition, 2016.
  * [17] Leonid V Kantorovich.  On the translocation of masses.  Journal of Mathematical Sciences, 2006.
  * [18] Elyor Kodirov, Tao Xiang, Zhenyong Fu, and Shaogang Gong.  Unsupervised domain adaptation for zero-shot learning.  In Proceedings of The IEEE International Conference on Computer Vision, 2015.
  * [19] Peizhao Li, Han Zhao, and Hongfu Liu.  Deep fair clustering for visual learning.  In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [20] Hongfu Liu, Ming Shao, and Yun Fu.  Structure-preserved multi-source domain adaptation.  In Proceedings of The IEEE 16th International Conference on Data Mining, 2016.
  * [21] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I. Jordan.  Learning transferable features with deep adaptation networks.  In Proceedings of the 32nd International Conference on Machine Learning, 2015.
  * [22] Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I Jordan.  Conditional adversarial domain adaptation.  In Advances in Neural Information Processing Systems, 2018.
  * [23] Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I Jordan.  Deep transfer learning with joint adaptation networks.  In Proceedings of The 34th International Conference on Machine Learning, 2017.
  * [24] Yawei Luo, Liang Zheng, Tao Guan, Junqing Yu, and Yi Yang.  Taking a closer look at domain shift: Category-level adversaries for semantics consistent domain adaptation.  In Proceedings of The IEEE Conference on Computer Vision and Pattern Recognition, 2019.
  * [25] Laurens van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  Journal of Machine Learning Research, 2008.
  * [26] Tzu Ming Harry Hsu, Wei Yu Chen, Cheng-An Hou, Yao-Hung Hubert Tsai, Yi-Ren Yeh, and Yu-Chiang Frank Wang.  Unsupervised domain adaptation with imbalanced cross-domain data.  In Proceedings of The IEEE International Conference on Computer Vision, 2015.
  * [27] Jialin Pan and Qiang Yang.  A survey on transfer learning.  IEEE Transactions on Knowledge and Data Engineering, 2010.
  * [28] Pau Panareda Busto and Juergen Gall.  Open set domain adaptation.  In Proceedings of The IEEE International Conference on Computer Vision, Oct 2017.
  * [29] Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko.  Visda: The visual domain adaptation challenge.  arXiv preprint arXiv:1710.06924, 2017.
  * [30] Michaël Perrot, Nicolas Courty, Rémi Flamary, and Amaury Habrard.  Mapping estimation for discrete optimal transport.  In Advances in Neural Information Processing Systems, 2016.
  * [31] Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell.  Adapting visual category models to new domains.  In Proceedings of The European Conference on Computer Vision, 2010\.
  * [32] Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell.  Adversarial discriminative domain adaptation.  In Proceedings of The IEEE International Conference on Computer Vision, 2017.
  * [33] Hongliang Yan, Yukang Ding, Peihua Li, Qilong Wang, Yong Xu, and Wangmeng Zuo.  Mind the class weight bias: Weighted maximum mean discrepancy for unsupervised domain adaptation.  In Proceedings of The IEEE Conference on Computer Vision and Pattern Recognition, 2017.
  * [34] Yabin Zhang, Hui Tang, Kui Jia, and Mingkui Tan.  Domain-symmetric networks for adversarial domain adaptation.  In Proceedings of The IEEE Conference on Computer Vision and Pattern Recognition, 2019.
  * [35] Han Zhao, Remi Tachet Des Combes, Kun Zhang, and Geoffrey Gordon.  On learning invariant representations for domain adaptation.  In Proceedings of The 36th International Conference on Machine Learning, 2019.
  * [36] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In Proceedings of The IEEE International Conference on Computer Vision, 2017.


m = N_{[]}(\bar \epsilon^2, \Phi_{\gamma},L_2(P))\le \exp(K\bar \epsilon^{-4/(2+\alpha)}).
\end{align*}

Let $\phi_1^L\le\phi_1^T,\dots,\phi_{m}^L\le\phi_m^T$ be the functions defining the $m$ brackets for $\Phi_\gamma$. Thus, for each fixed $t$ and $\phi\in\Phi_\gamma$, we have

\begin{align*}
I(\phi_i^L(v,u)\ge t )\le I(\phi(v,u)\ge t )\le I(\phi_i^T(v,u)\ge t ),
\end{align*}

for some $i=1,\dots,m$. Define $F_i^L(t)=P(\phi_i^L(V,\tilde U)\ge t)$, and let $t^L_{ij}$ for $j=1,\dots,O(\bar \epsilon^{-2})$, partition the line in segments having $F^L_i$-probability less than or equal to a fraction of $\bar \epsilon^2$. Similarly, define $F_i^T(t)=P(\phi_i^T(V,\tilde U)\ge t)$, and let $t^T_{ij}$ for $j=1,\dots,O(\bar \epsilon^{-2})$, partition the line in segments having $F^T_i$-probability less than or equal to a fraction of $\bar \epsilon^2$. Now define the following bracket for $t$:

t^L_{ij_1}\le t\le t^T_{ij_2},

where $t^L_{ij_1}$ is the largest of the $t^L_{ij}$ with the property of being less than or equal to $t$ and $t^T_{ij2}$ is the smallest of the $t^T_{ij}$ with the property of being greater than or equal to $t$. We will now show that the brackets for $\mathcal{F}$ are given by

\begin{align*}
I(\phi_i^L(v,u)\ge t_{ij_2}^T )\le I(\phi(v,u)\ge t )\le I(\phi_i^T(v,u)\ge t_{ij_1}^L).
\end{align*}

Note that, taking eventually a smaller $\gamma>0$, for $\phi_1,\phi_2\in\Phi_{\gamma}$, there exist $\phi_1(v,\cdot)^{-1}$ and $\phi_2(v,\cdot)^{-1}$, which we can denote by $\rho_1(v,t)$ and $\rho_2(v,t)$, respectively. In addition, by the regularity assumptions on $\varphi$, it is easy to show that there exists a constant $\Theta$, such that, for any integer $k\ge 1$, we have

\begin{align}
\|\rho_1-\rho_2\|_{P,k}\le \|\rho_1-\rho_2\|_{\infty}\le \Theta \|\phi_1-\phi_2\|_{1},
\end{align}

where $\|\cdot\|_{P,k}$ is the $L_k(P)$  norm. This also implies that there exists a constant $K$ such that

\begin{align*}
\|	I&(\phi_i^L(V,\tilde U)\ge t_{ij_2}^T )- I(\phi_i^T(V,\tilde U)\ge t_{ij_1}^L)\|_{P,2}^2\\
&=P(\phi_i^T(V,\tilde U)\ge t_{ij_1}^L) - P(\phi_i^L(V,\tilde U)\ge t_{ij_2}^T ) \\
&=F_i^T(t_{ij1}^L) - F_i^L(t_{ij2}^T)\\
&\le F_i^T(t)-F_i^L(t) +K \epsilon^2.
\end{align*}

Now, denote by $f_{\tilde U}(u)$ the quantity $f_{\tilde U}(u)=\text{d}P(\tilde U\le u)/\text{d}u$, which exists a.e., and by $\rho_i^L(v,t)$ and $\rho_i^T(v,t)$ the functions $\phi_i^L(v,\cdot)^{-1}$ and $\phi_i^T(v,\cdot)^{-1}$, respectively. Then, using a Taylor expansion, we obtain

\begin{align}\label{eq:_ingrid_paper_21}
F_i^T(t)-F_i^L(t) &= \int_{\mathcal{V}}\Big(P(\tilde U\ge \rho_i^T(v,t)) - P(\tilde U\ge \rho_i^L(v,t))\Big)\text{d}\mu(v)\nonumber\\
&=\int_{\mathcal{V}}f_{\tilde U}(\xi(v,t)) \big(\rho_i^T(v,t) - \rho_i^L(v,t)\big)\text{d}\mu(v),
\end{align}

where $\xi(v,t)$ is a point between $\rho_i^T(v,t)$ and $\rho_i^L(v,t)$. Therefore, for a suitable constant $J$ bounding $f_U$, we obtain

\begin{align*}
F_i^T(t)-F_i^L(t) &\le J \|\rho_i^T(v,t) - \rho_i^L(v,t)\|_{P,1}\\
&\le J  \|\rho_i^T(v,t) - \rho_i^L(v,t)\|_{P,2} \\
&\le J\Theta \bar \epsilon^2.
\end{align*}

Hence for the class $\mathcal{F}$ and for each $\bar \epsilon>0$, we have at most $O(\bar \epsilon^{-2}\exp(K\bar \epsilon^{-4/(2+\alpha)}))$ brackets in total. However, for $\bar \epsilon>1$, one bracket suffices. So, it holds

\begin{align*}
\int_{0}^\infty\sqrt{\log N_{[]}(\bar \epsilon,\mathcal{F},L_2(P))}\text{d}\bar \epsilon<\infty.
\end{align*}

This shows that the class $\mathcal{F}$ is Donsker. Together with Theorem 2.10.6 in [25], this result also implies that the following families of functions are Donsker:

\begin{align*}
\mathcal{F}_1 &=\{\mathcal{V}\times[0,\bar U]\ni (v,u)\mapsto \exp(v^\top\beta)I(\phi(v,u)\ge t)| t\in[0,\bar T],\beta\in\mathcal{B},\phi\in\Phi_\gamma\};\\
\mathcal{F}_2 &=\{\mathcal{V}\times[0,\bar U]\times\{0,1\}\ni (v,u,\Delta)\mapsto I(\phi(v,u)\ge t)I(\Delta=1)| t\in[0,\bar T],\phi\in\Phi_\gamma\};\\
\mathcal{F}_3 &=\{\mathcal{V}\times[0,\bar U]\times\{0,1\}\ni (v,u,\Delta)\mapsto v\exp(v^\top\beta)I(\phi(v,u)\ge t)I(\Delta=1)\\
&\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad| t\in[0,\bar T],\beta\in\mathcal{B},\phi\in\Phi_\gamma\}.
\end{align*}

Now, consider $\mathcal{F}_1$. What remains to show is that if $\beta_n$ is a sequence in $n$ converging to $\beta_0$ (with respect to the Euclidean norm) and $\phi_n$ is a sequence in $n$ converging to  $\varphi$  (with respect to $\|\cdot\|_1$) then

\begin{align*}
E\Bigl\{&\Bigl[\Bigl(\exp\bigl(V^\top\beta_n\bigr)I\bigl(\phi_n(V,\tilde U)\ge t\bigr) - \exp\bigl(V^\top\beta\bigr)I\bigl(\varphi(V,\tilde U)\ge t\bigr)\Bigr) \\
&- 	E\Bigl(\exp\bigl(V^\top\beta_n\bigr)I\bigl(\phi_n(V,\tilde U)\ge t\bigr) - \exp\bigl(V^\top\beta\bigr)I\bigl(\varphi(V,\tilde U)\ge t\bigr)\Bigr)\Bigr]^2\Bigr\} \xrightarrow{}0
\end{align*}

This result easily follows from the regularity of the distribution of $\tilde U$, the compactness of the support of $V$, the independence between $V$ and $\tilde U$, the definition of the norm $\|\cdot\|_1$ and the Lipschitz continuity of $v\mapsto \exp(v^\top\beta)$ in $\beta$ uniformly in $v$. More precisely, write

\begin{align*}
E\Bigl\{&\Bigl[\Bigl(\exp\bigl(V^\top\beta_n\bigr)I\bigl(\phi_n(V,\tilde U)\ge t\bigr) - \exp\bigl(V^\top\beta\bigr)I\bigl(\varphi(V,\tilde U)\ge t\bigr)\Bigr) \\
&- 	E\Bigl(\exp\bigl(V^\top\beta_n\bigr)I\bigl(\phi_n(V,\tilde U)\ge t\bigr) - \exp\bigl(V^\top\beta\bigr)I\bigl(\varphi(V,\tilde U)\ge t\bigr)\Bigr)\Bigr]^2\Bigr\} \\
&= I_1+I_2,
\end{align*}

\begin{align*}
I_1 &= E[\exp(2V^\top\beta_n)I(\phi_n(V,\tilde U)\ge t)] \\
&\quad+ E[\exp(2V^\top\beta_0)I(\varphi(V,\tilde U)\ge t)]\\ &\quad-2E[\exp(V^\top\beta_n)I(\phi_n(V,\tilde U)\ge t)\exp(V^\top\beta_n)I(\phi_n(V,\tilde U)\ge t)];
\end{align*}

\begin{align*}
I_2 &= \Big(E[(V^\top\beta_n)P(\phi_n(V,\tilde U))]-E[(V^\top\beta_0)P(\varphi(V,\tilde U))]\Big)^2.
\end{align*}

Now, using similar arguments as before, we have

\begin{align*}
\sup_{v\in\mathcal{V}}|P(\phi_n&(v,\tilde U)\ge t)-P(\varphi(v,\tilde U)\ge t)| \\
&=	\sup_{v\in\mathcal{V}}|	P(\tilde U\ge \rho_n(v,t))-P(\tilde U \ge\varrho(v,t)|\\
&\le K\|\phi_n-\varphi\|_1
\end{align*}

for a suitable constant $K$, where $\rho_n(v,t) = \phi_n(v,\cdot)^{-1}(t)$. Also,

\begin{align*}
\sup_{v\in\mathcal{V}}|\exp(v^\top\beta_n) -\exp(v^\top\beta_0) | \le K\|\beta_n-\beta_0\|,
\end{align*}

for a further constant $K$. Therefore, it is easy to see that $I_2\xrightarrow{}0$. Using the independence between $V$ and $\tilde U$, similar arguments also show that $I_1\xrightarrow{}0$.
Similar results can be shown for the families $\mathcal{F}_2$ and $\mathcal{F}_3$. Thus, from Corollary 2.3.12 in [25], we have that, for any sequence $\varsigma_n=o(1)$, it holds

\begin{align}\label{eq:final_donsker}
&\sup_{s\in[0,\bar T],\|\beta-\beta_0\|\le \varsigma_n,\|\phi-\varphi\|_{1}\le \varsigma_n} |\hat Q^\phi(s) - Q^\phi(s) - \hat Q^\varphi(s) +Q^\varphi(s)|= o_p(n^{-1/2});\nonumber\\
&\sup_{s\in[0,\bar T],\|\beta-\beta_0\|\le \varsigma_n,\|\phi-\varphi\|_{1}\le \varsigma_n} \|\hat E^\phi(g(\beta,v),s) \nonumber\\
&\quad\quad- E^\phi(g(\beta,v),s) -\hat E^\varphi(g(\beta_0,v),s) + E^\varphi(g(\beta_0,v),s) \|= o_p(n^{-1/2});\\
&\sup_{s\in[0,\bar T],\|\beta-\beta_0\|\le \varsigma_n,\|\phi-\varphi\|_{1}\le \varsigma_n} \|\hat E_1^\phi(g(\beta,v),s)) \nonumber\\
&\quad\quad- E_1^\phi(g(\beta,v),s) -\hat E_1^\varphi(g(\beta_0,v),s)+ E_1^\varphi(g(\beta_0,v),s) \|= o_p(n^{-1/2}).\nonumber
\end{align}

Now, with a similar decomposition as the one in [24] (page 97), we can write

\begin{align*}
M_n(\beta,\phi)-&M(\beta,\varphi) - (M_n(\beta,\varphi)-M(\beta_0,\varphi))\\
&=\sum_{j=1}^6D_{jn}(\beta,\phi)+\sum_{j=1}^8R_{jn} (\beta,\phi) ,
\end{align*}

where $D_{jn}(\beta,\phi) = C_{jn}(\beta,\phi) - C_{jn}(\beta_0,\varphi)$,

\begin{align*}
C_{1n}(\beta,\phi) &= \hat E_1^\phi(v,0)-E_1^\phi(v,0);\\
C_{2n}(\beta,\phi)&=-[\hat Q^\phi(0)-Q^\phi(0)]E^\phi(v\exp(v^\top\beta),0)/E^\phi(\exp(v^\top\beta),0);\\
C_{3n}(\beta,\phi)&=\int_0^{\bar T}[(\hat Q^\phi(s)-Q^\phi(s))/E^\phi(\exp(v^\top\beta),s)]\text{d}E^\phi(v\exp(v^\top\beta),s);\\
C_{4n}(\beta,\phi)&=\int_0^{\bar T}[(\hat Q^\phi(s)-Q^\phi(s))E^\phi(v\exp(v^\top\beta),s)/E^\phi(\exp(v^\top\beta),s)^2]\\
C_{5n}(\beta,\phi)&=\int_0^{\bar T}[(\hat E^\phi(v\exp(v^\top\beta),s) -E^\phi(v\exp(v^\top\beta),s))/E^\phi(\exp(v^\top\beta),s) ]\text{d}Q^\phi(s);\\
C_{6n}(\beta,\phi)&=\int_0^{\bar T}[(\hat E^\phi(\exp(v^\top\beta),s) -E^\phi(\exp(v^\top\beta),s))\\
&\quad\quad\times E^\phi(v\exp(v^\top\beta),s)/E^\phi(\exp(v^\top\beta),s)^2 ]\text{d}Q^\phi(s);
\end{align*}

\begin{align*}
R_{1n}(\beta,\phi) &= \int_0^{\bar T}[(\hat E^\phi(v\exp(v^\top\beta),s)-E^\phi(v\exp(v^\top\beta),s))/\hat E^\phi(\exp(v^\top\beta),s)]\\
&\quad\quad\quad\quad\text{d}\big(\hat Q^\phi(s) - Q^\phi(s) - \hat Q^\varphi(s) +Q^\varphi(s)\big);\\
R_{2n}(\beta,\phi) &= \int_0^{\bar T}\Big\{[(\hat E^\phi(v\exp(v^\top\beta),s)-E^\phi(v\exp(v^\top\beta_0),s))/\hat E^\phi(\exp(v^\top\beta),s)] \\
&\quad\quad-[(\hat E^\varphi(v\exp(v^\top\beta_0),s)-E^\varphi(v\exp(v^\top\beta_0),s))/\hat E^\varphi(\exp(v^\top\beta),s)]\Big\}\\
&\quad\quad\quad\quad\text{d}\big(\hat Q^\varphi(s) - Q^\varphi(s)\big);\\
R_{3n}(\beta,\phi) &= \int_0^{\bar T}[(\hat E^\phi(v\exp(v^\top\beta),s)-E^\phi(v\exp(v^\top\beta),s))\\
&\quad\quad\times(\hat E^\phi(\exp(v^\top\beta),s)-E^\phi(\exp(v^\top\beta),s))\\
&\quad\quad \times (\hat E^\phi(\exp(v^\top\beta),s)E^\phi(\exp(v^\top\beta),s))^{-1}]\text{d} \big( Q^\phi(s) - Q^\varphi(s)\big);\\
R_{4n}(\beta,\phi) &= \int_0^{\bar T}\Big\{[(\hat E^\phi(v\exp(v^\top\beta),s)-E^\phi(v\exp(v^\top\beta),s))\\
&\quad\quad\times(\hat E^\phi(\exp(v^\top\beta),s)-E^\phi(\exp(v^\top\beta),s))\\
&\quad\quad \times (\hat E^\phi(\exp(v^\top\beta),s)E^\phi(\exp(v^\top\beta),s))^{-1}]\\
&\quad\quad-[(\hat E^\varphi(v\exp(v^\top\beta_0),s)-E^\varphi(v\exp(v^\top\beta_0),s))\\
&\quad\quad\times(\hat E^\varphi(\exp(v^\top\beta_0),s)-E^\varphi(\exp(v^\top\beta_0),s))\\
&\quad\quad \times (\hat E^\varphi(\exp(v^\top\beta_0),s)E^\varphi(\exp(v^\top\beta_0),s))^{-1}]\Big\}\text{d} Q^\varphi(s);\\
R_{5n}(\beta,\phi) &= -\int_0^{\bar T}[(\hat E^\phi(\exp(v^\top\beta),s)-E^\phi(\exp(v^\top\beta),s))E^\phi(v\exp(v^\top\beta),s)\\
&\quad\quad \times (\hat E^\phi(\exp(v^\top\beta),s)E^\phi(\exp(v^\top\beta),s))^{-1}]\\
&\quad\quad\quad\quad\text{d} \big(\hat Q^\phi(s) - Q^\phi(s) - \hat Q^\varphi(s) +Q^\varphi(s)\big);\\
R_{6n}(\beta,\phi) &= -\int_0^{\bar T}\Big\{[(\hat E^\phi(\exp(v^\top\beta),s)-E^\phi(\exp(v^\top\beta),s))E^\phi(v\exp(v^\top\beta),s)\\
&\quad\quad \times (\hat E^\phi(\exp(v^\top\beta),s)E^\phi(\exp(v^\top\beta),s))^{-1}]\\
&\quad\quad -[(\hat E^\varphi(\exp(v^\top\beta_0),s)-E^\varphi(\exp(v^\top\beta_0),s))E^\varphi(v\exp(v^\top\beta_0),s)\\
&\quad\quad \times (\hat E^\varphi(\exp(v^\top\beta_0),s)E^\varphi(\exp(v^\top\beta_0),s))^{-1}]\Big\}\text{d} \big(\hat Q^\varphi(s) -Q^\varphi(s)\big);\\
R_{7n}(\beta,\phi) &= \int_0^{\bar T}[(\hat E^\phi(\exp(v^\top\beta),s)-E^\phi(\exp(v^\top\beta),s))^2E^\phi(v\exp(v^\top\beta),s)\\
&\quad\quad \times (\hat E^\phi(\exp(v^\top\beta),s)E^\phi(\exp(v^\top\beta_0),s)^2)^{-1}]\text{d} \big(Q^\phi(s) -Q^\varphi(s)\big);\\
R_{8n}(\beta,\phi) &= \int_0^{\bar T}\Big\{[(\hat E^\phi(\exp(v^\top\beta),s)-E^\phi(\exp(v^\top\beta),s))^2E^\phi(v\exp(v^\top\beta),s)\\
&\quad\quad \times (\hat E^\phi(\exp(v^\top\beta),s)E^\phi(\exp(v^\top\beta),s)^2)^{-1}]\\
&\quad\quad -[(\hat E^\varphi(\exp(v^\top\beta_0),s)-E^\varphi(\exp(v^\top\beta_0),s))^2E^\varphi(v\exp(v^\top\beta_0),s)\\
&\quad\quad \times (\hat E^\varphi(\exp(v^\top\beta_0),s)E^\varphi(\exp(v^\top\beta_0),s)^2)^{-1}]\Big\}\text{d} Q^\varphi(s).
\end{align*}

This decomposition, and the equalities in (<ref>) imply  the assertion.

(2.6) The statement can be reduced to showing that $n^{1/2}\{M_n(\beta_0,\varphi) + \Gamma_2(\beta_0,\varphi)[\hat\varphi-\varphi]\}$ can be expressed as a sum of iid terms of the form

\begin{align*}
&n^{1/2}\{M_n(\beta_0,\varphi) + \Gamma_2(\beta_0,\varphi)[\hat\varphi-\varphi]\\
&\quad\quad = n^{-1/2}\sum_{i=1}^n\eta(Y_i,\delta_i,Z_i,W_i,X_i,\Delta_i,\tilde U_i) + o_p(1).
\end{align*}

The fact that $ n^{1/2} M_n(\beta_0,\varphi)$ admits such representation is thanks to the decomposition given in (2.5). We now show that the same holds for the term $ n^{1/2}\Gamma_2(\beta_0,\varphi)[\hat\varphi-\varphi]$, whose expression is given in (<ref>).

Define $p(z,x)= P(Z=z|X=x)$ and denote by $\nu(x)$ the density of $X$. First, consider $e_1(v,0)[\hat\varphi-\varphi]$, that is

\begin{align*}
e_1(v,0)[\hat\varphi-\varphi] =- \int_{\mathcal{X}}\sum_{l=1}^{L} \frac{(z_l^\top,x^\top)^\top}{\frac{\partial}{\partial u}\varphi(z_l,x,0)}\big(\hat\varphi(z_l,x,0)-\varphi(z_l,x,0)\big)p(z_l,x)\nu(x)\text{d}x.
\end{align*}

Denote $\Gamma(\varphi^x,F^x)^{-1}_l$ the $l$th row the matrix $\Gamma(\varphi^x,F^x)^{-1}$. Given two vectors $v$ and $w$ denote by $vw$ the inner product, defined by $v^\top w$. We can write

\begin{align*}
&e_1(v,0)[\hat\varphi-\varphi] \\
&=- \int_{\mathcal{X}}\sum_{l=1}^{L} \frac{(z_l^\top,x^\top)^\top}{\frac{\partial}{\partial u}\varphi(z_l,x,0)}\Bigg((nh)^{-1}\sum_{i=1}^n K\left(\frac{x-X_i}{h}\right)\\
&\times (\Gamma(\varphi^x,F^x)^{-1})_l\left(\sum_{l=1}^L\eta^F(Y_i, \delta_i, Z_i,  W_i, \varphi(z_l,x,0), z_l, x, w_k)\right)_{k=1}^L \Bigg)p(z_l,x)\nu(x)\text{d}x \\
&=(nh)^{-1}\int_{\mathcal{X}}\sum_{i=1}^n K\left(\frac{x-X_i}{h}\right) b(Y_i,\delta_i,Z_i,W_i,x)\text{d}x+o_p(n^{-1/2}),
\end{align*}

\begin{align*}
&b(Y_i,\delta_i,Z_i,W_i,x) = -\sum_{l=1}^{L} \frac{(z_l^\top,x^\top)^\top}{\frac{\partial}{\partial u}\varphi(z_l,x,0)}\Bigg((\Gamma(\varphi^x,F^x)^{-1})_l\\
&\times\Big(\sum_{l=1}^L\eta^F(Y_i, \delta_i, Z_i,  W_i, \varphi(z_l,x,0), z_l, x, w_k)\Big)_{k=1}^L \Bigg)p(z_l,x)\nu(x).
\end{align*}

Thus, by standard change of variable and Taylor expansion, we obtain

\begin{align*}
e_1&(v,0)[\hat\varphi-\varphi] \\
&= n^{-1} \sum_{i=1}^n\int_{\mathcal{X}} b(Y_i,\delta_i,Z_i,W_i,hx+X_i)K(x)\text{d}x\\
&= n^{-1} \sum_{i=1}^n\int_{\mathcal{X}} [b(Y_i,\delta_i,Z_i,W_i,X_i) \\
&\quad\quad+ \sum_{j=1}^{\nu-1}(hx)^j\frac{\partial^j}{\partial x^j}b(Y_i,\delta_i,Z_i,W_i,X_i) +O_p(h^\nu)]K(x)\text{d}x \\
&= n^{-1} \sum_{i=1}^n b(Y_i,\delta_i,Z_i,W_i,X_i) + o_p(n^{-1/2}),
\end{align*}

where in the last equality we used that $O_p(h^\nu) = o_p(n^{-1/2})$.
Now, write  $q^\varphi(t)[\hat\varphi-\varphi] = I_1 + I_2$, where

\begin{align*}
I_1 &=\int_{\mathcal{V}}  g_0(v,t) \big(\hat\varphi(v,\rho(v,t))-\varphi(v,\rho(v,t))\big)\text{d}\mu(v);\\
I_2 &= \int_{\mathcal{V}} g_1(v,t)\big(\frac{\partial}{\partial u}\hat\varphi(v,\rho(v,t))-\frac{\partial}{\partial u}\varphi(v,\rho(v,t))\big)\text{d}\mu(v);
\end{align*}

\begin{align*}
g_0(v,t)&= \Big\{I(\varrho(v,t)<\bar U-\delta) + I(\bar U-\delta\le \varrho(v,t)\le \bar U)\frac{\bar U}{\delta}\Big\}\frac{\varrho'(v,t) \varphi''(v,\varrho(v,t))}{\varphi'(v,\varrho(v,t))} \\
&\quad+ \Big\{ I(\bar U-\delta\le \varrho(v,t)\le \bar U)\frac{\varrho(v,t)'}{\delta}\Big\} \frac{1}{(\varphi'(v,\varrho(v,t)))^2} \\
%&\quad\quad-\frac{I(\bar U\ge \varphi(v,\cdot)^{-1}(t)) \bar U^{-1}\big(1-\bar U^{-1}\varphi(v,\cdot)^{-1}(t)\big) }{\varphi'(v,\varrho(v,t))} ;\\
g_1(v,t)&=  \Big\{I(\varrho(v,t)<\bar U-\delta) + I(\bar U-\delta\le \varrho(v,t)\le \bar U)\frac{\bar U}{\delta}\Big\}\frac{1}{(\varphi'(v,\varrho(v,t)))^2}
\end{align*}

Using a similar argument as for $ e_1(v,0)[\hat\varphi-\varphi]$ we again can show that there exists a function $\tilde b(y,\delta, z, w, x)$ such that

\begin{align*}
I_1 = n^{-1} \sum_{i=1}^n\tilde b(Y_i,\delta_i,Z_i,W_i,X_i) + o_p(n^{-1/2}).
\end{align*}

We now focus on $I_2$, and we write it as $I_2 = I_{21}+ I_{22}+o_p(n^{-1/2})$, where

\begin{align*}
&I_{21}=\int_{\mathcal{X}}\sum_{l=1}^{L}g_0(z_l,x,t)\Big[(nh)^{-1}\sum_{i=1}^n K\left(\frac{x - X_i}{h}\right)\\
&\times\sum_{j=1}^L \kappa_j^l(x,\varrho(z_l,x,t))\eta_j(Y_i,\delta_i,Z_i,W_i,x,\varrho(z_l,x,t)) \\
& + (nh)^{-1}\sum_{i=1}^n  K\left(\frac{x - X_i}{h}\right) \sum_{j,k=1}^L \zeta_{jk}^l(x,\varrho(z_l,x,t))
%\frac{ I(Z_i=z_l,W_i=w_k)}{f_{Z,X,W}(z_l,x,w_k)}
&\times p(z_l,x)\nu(x)\text{d}x\\
&I_{22} = \int_{\mathcal{X}}\sum_{l=1}^{L}g_1(z_l,x,t)\Big[(nh\epsilon)^{-1}\sum_{i=1}^n  K\left(\frac{x - X_i}{h}\right)\\
&\times \int \tilde K'(s)\sum_{j,k=1}^L \zeta_{jk}^l(x,\varrho(z_l,x,t))\xi^F(Y_i, \delta_i, Z_i,  W_i,  t-s\epsilon, z_l, x, w_k)\text{d}s \Big]\\
&\times p(z_l,x)\nu(x)\text{d}x.
\end{align*}

So, again with a similar argument as before, $I_{21}$ admits such representation, and we now discuss $I_{22}$. Theorem 3 (b) of [27] implies that

\begin{align*}
&(nh)^{-1}\sum_{i=1}^n  K\left(\frac{x - X_i}{h}\right) \xi^F(Y_i, \delta_i, Z_i,  W_i,  t-s\epsilon, z_l, x, w) \\
&= (nh)^{-1}\sum_{i=1}^n  K\left(\frac{x - X_i}{h}\right) \xi^F(Y_i, \delta_i, Z_i,  W_i,  t, z, x, w) +  O_p\big((\epsilon\log n/(nh))^{1/2} +\epsilon^{1/2}h^\nu\big).
\end{align*}

Since $\int\tilde  K'(s)\text{d}s=0$, $\int s\tilde K'(s)\text{d}s=-1$ and $\int s^j\tilde K'(s)\text{d}s=0$ for $j=2,\dots,\pi$, we obtain

\begin{align*}
&I_{22} = \int_{\mathcal{X}}\sum_{l=1}^{L}g_1(z_l,x,t)\Big[(nh\epsilon)^{-1}\sum_{i=1}^n  K\left(\frac{x - X_i}{h}\right)\\
&\quad\times \int \tilde K'(s)\sum_{j,k=1}^L \zeta_{jk}^l(x,\varrho(z_l,x,t))F(t-\epsilon s,z|x,w)\text{d}s \Big]\\
&\quad\times p(z_l,x)\nu(x)\text{d}x + O_p\big((\log n/( nh\epsilon))^{1/2} +\epsilon^{-1/2}h^\nu\big)\\
&\quad\times\Big[(nh\epsilon)^{-1}\sum_{i=1}^n  K\left(\frac{x - X_i}{h}\right) \int \tilde K'(s)\sum_{j,k=1}^L \zeta_{jk}^l(x,\varrho(z_l,x,t))\big(-s\epsilon f(t,z_l|x,w_k) \\
&\quad\quad\quad\quad+\frac{1}{2}s^2\epsilon^2 f'(t,z_l|x,w_k) -\frac{1}{6} s^3\epsilon^3 f''(t,z_l|x,w_k)+\dots+ O(\epsilon^{\pi+1})\big)\text{d}s \Big]\\
&\quad\times p(z_l,x)\nu(x)\text{d}x + O_p\big((\log n/( nh\epsilon))^{1/2} +\epsilon^{-1/2}h^\nu\big))\\
&= \int_{\mathcal{X}}\sum_{l=1}^{L}g_1(z_l,x,t)\\
&\quad\times\Big[(nh)^{-1}\sum_{i=1}^n  K\left(\frac{x - X_i}{h}\right)\sum_{j,k=1}^L \zeta_{jk}^l(x,\varrho(z_l,x,t))(f(t,z_l|x,w_k) + O(\epsilon^\pi))\Big]\\
&\quad\times p(z_l,x)\nu(x)\text{d}x + O_p\big((\log n/( nh\epsilon))^{1/2} +\epsilon^{-1/2}h^\nu\big)
\end{align*}

Note that $O_p\big((\log n/(nh\epsilon ))^{1/2} +\epsilon^{-1/2}h^\nu\big) = o_p(n^{-1/2})$ since $w<2\nu u-1$. All the other terms in $ \Gamma_2(\beta_0,\varphi)[\hat\varphi-\varphi]$ can follow the same arguments, from which the assertion follows.

* $u<1/2$

* $u>1/(4\nu)$

* $w>1/(4\pi)$

* $w>u$

* $w<(1-u)/(5+2\alpha)$

* $w<u(\nu+1)/(3+\alpha)$

* $w<u\nu/\alpha$

* $w<1/(3+\alpha)$

* $u<1/(5+2\alpha)$

* $u<1/(5+2\alpha)$

* $w>u(1+\alpha)/\pi$

* $w<2\nu u-1$

results in

\begin{align*}
\frac{1}{4\nu}<u<\min(\frac{1}{2},\frac{1}{5+2\alpha})
\end{align*}

\begin{align*}
\max(\frac{u(1+\alpha)}{\pi}, u,\frac{1}{4\pi})<w<\min(\frac{1}{3+\alpha},2\nu u-1,\frac{u\nu}{\alpha},\frac{u(\nu+1)}{3+\alpha},\frac{1-u}{5+2\alpha})
\end{align*}