# Self-degraded contrastive domain adaptation for industrial fault diagnosis
with bi-imbalanced data

Gecheng Chen, Zeyu Yang, Chengwen Luo, Jianqiang Li

###### Abstract

Modern industrial fault diagnosis tasks often face the combined challenge of
distribution discrepancy and bi-imbalance. Existing domain adaptation
approaches pay little attention to the prevailing bi-imbalance, leading to
poor domain adaptation performance or even negative transfer. In this work, we
propose a self-degraded contrastive domain adaptation (Sd-CDA) diagnosis
framework to handle the domain discrepancy under the bi-imbalanced data. It
first pre-trains the feature extractor via imbalance-aware contrastive
learning based on model pruning to learn the feature representation
efficiently in a self-supervised manner. Then it forces the samples away from
the domain boundary based on supervised contrastive domain adversarial
learning (SupCon-DA) and ensures the features generated by the feature
extractor are discriminative enough. Furthermore, we propose the pruned
contrastive domain adversarial learning (PSupCon-DA) to pay automatically re-
weighted attention to the minorities to enhance the performance towards bi-
imbalanced data. We show the superiority of the proposed method via two
experiments.

###### Index Terms:

Transfer learning; industrial imbalanced data; model pruning; contrastive
learning; adversarial learning

##  I Introduction

As modern industries have grown in complexity and scale, there has been a
significant increase in attention to precise and efficient data-driven fault
diagnosis methods to guarantee reliable and stable operations. These methods
are designed to identify various fault conditions in processes or machines
using extensive data collected from sensors. Recently, deep learning (DL)
techniques, such as fully connected neural networks (FNN), and convolutional
neural networks (CNN), have been widely applied to data-driven diagnosis and
achieved distinguished performance due to their feature extraction ability and
end-to-end structures [1, 2, 3].

However, traditional DL methods only work well under a strong assumption, that
is, the training data (denoted as source domain) and the testing data (denoted
as target domain) should have the same distribution, which can hardly hold in
real-world industrial applications. For example, in an industrial production
process, the training data may be gathered under specific working conditions
but the condition of interest can be another one with a different working
load. As a result, traditional DL-based fault diagnosis methods may fail
because these two conditions have different data distributions.

On the basis of domain discrepancy, however, the industrial data also exhibits
heavy imbalance because the processes mainly run in normal conditions. In the
scenario of domain discrepancy, the impact of imbalance is two-fold: intra-
domain imbalance and inter-domain imbalance [4]. We use (a.1) in Figure 1 to
illustrate these two imbalances. In (a.1), blue and yellow samples represent
samples from two domains, separately, whereas circles and stars represent two
different classes. Intra-domain imbalance means that the amount of the normal
class (majority) can be much larger than the fault classes (minorities) in
both domains. For example, the number of blue stars is much larger than
circles. Inter-domain imbalance indicates that the probabilities of different
faults of two domains can be highly different, leading to different imbalance
ratios of these two domains. In (a.1) we can see that the circles are the
minority class in the blue domain but they are the majority class in the
yellow domain. We call the data with two kinds of imbalance bi-imbalanced
data.

In order to solve the domain discrepancy, the unsupervised domain adaptation
(UDA) has achieved great progress [5, 6]. It leverages the knowledge of the
label-rich source domain to solve a similar task on a different but related
target domain with no label. Lots of metric-based UDA approaches have been
proposed to minimize the discrepancy measured by some explicit function to
align these two domains, e.g., maximum mean discrepancy (MMD) [7] or
correlation alignment [8]. However, the pre-selected metric function may not
always be applicable, impeding the applications of metric discrepancy-based DA
approaches. Recently, domain adversarial neural networks (DANN) [9] and
related techniques have received increasing attention because they measure the
discrepancy between two domains implicitly by a domain discriminator. In
general, domain adversarial learning-based approaches train a feature
extractor and a domain discriminator adversarially to extract domain invariant
features from the two domains and construct an efficient classifier for the
target domain. However, a model well-trained based on source labels may
overfit the source domain and cannot generalize well to the target domain
[10]. Even worse, the domain adversarial learning-based methods are not
guaranteed to generate discriminative features for all samples. The main
reason is that the feature extractor can simply generate features along the
domain boundary because the goal of the feature extractor is to fool the
domain discriminator during training [11]. When the imbalance exists, the
minorities can be hardly corrected once they are along the domain boundary due
to the coverage of the majority.

Recently, several works have explored self-supervised techniques to improve
the representation learning of DANN-related work and increase the diagnosis
accuracy in a pre-train manner. Contrastive learning, originally proposed to
extract representation features from images or videos, learns effective
representations in the latent space in an unsupervised manner by enforcing the
similarity of positive pairs and enlarging the distance of negative pairs [12,
13]. As one of the most popular contrastive learning methods, simple
contrastive representation learning (SimCLR) [14] has been introduced to
domain adaptation methods for industrial fault diagnosis [15, 16]. However,
this contrastive learning-based approach is highly vulnerable to imbalanced
data and may make some minority samples mis-featured. The reason is that the
contrastive learning of minorities can be covered by the majorities under
imbalanced settings because the instance-rich class has much more separable
information [17].

Figure 1: The intuition behind the proposed method. (a.1)-(a.2) illustrates
the process of SimCLR, (b.1)-(b.2) shows the process of DANN and (c.1)-(c.3)
introduces what the proposed Sd-CDA does.

In [18], the authors prove that DNNs tend to memorize the majorities but
forget the minorities or difficult samples after pruning or compression. In
other words, for a minority sample or a sample along the decision boundary,
the difference between extracted features given by a DNN and its pruned
version should be larger compared with a normal sample. Inspired by this, we
propose a novel domain adaptation framework called Self-degraded Contrastive
Domain adaptation (Sd-CDA) for the aforementioned two challenges, which
consists of two parts: imbalance-aware contrastive learning for
representations (Ia-CLR) and boundary-aware adversarial domain adaptation (Ba-
ADA). First, to relieve the impact of bi-imbalance on contrastive learning, we
propose the imbalance-aware contrastive learning for representations (Ia-CLR)
as the pre-train procedure before domain adaptation. Since the original SimCLR
learns the feature representations via input augmentation, we can consider it
as contrastive learning at a data level. In this work, the proposed Ia-CLR
facilitates SimCLR with a model-level view by minimizing the difference
between the feature extractor and its pruned counterpart which we can consider
as a perturbed version of the original model. Given that the minorities cause
more significant differences than the majority, the Ia-CLR will attach higher
weights automatically and implicitly to the minorities during the self-
supervised training. Second, in order to solve the bi-imbalance challenge
during the adversarial training and ensure the extraction of discriminative
features, the proposed Ba-ADA first introduces the idea of supervised
contrastive learning to the discriminator of DANN to make the extracted
features of two domains more compact. By doing this, it forces the feature
extractor to generate discriminative features away from the domain boundary.
What’s more, Ba-ADA proposes a pruned supervised contrastive learning to help
detect the minorities of the target domain along the domain boundary by
pruning the discriminator. In summary, the main innovations of this work are
as follows:

  * •

Propose an imbalance-aware contrastive pre-train approach by pruning the
feature extractor to enhance the representation learning of the bi-imbalance
data on the basis of traditional constrastive learning.

  * •

Introduce the idea of supervised contrastive learning to DANN to force the
feature extractor to generate discriminative features.

  * •

Construct a pruned supervised contrastive learning strategy for the domain
discriminator to pay more attention to the minorities under bi-imbalance
during the adversarial training.

The remainder of this paper is organized as follows. We first review several
related works in Section II. Then we introduce the proposed Sd-CDA framework
in detail in Section III. In Section IV we demonstrate the superiority of the
proposed method via two cases. Conclusion remarks and discussion are given in
Section V.

##  II Related work

###  II-A Unsupervised domain adaptation

Current unsupervised domain adaptation approaches for industrial fault
diagnosis can be mainly divided into two categories: metric-based methods and
adversarial learning-based methods. Metric-based domain adaptation methods
mainly measure the discrepancy between the source and the target domain based
on several explicit metric functions, e.g., MMD, and correlation alignment
(CORAL). Lu [19] proposes a domain adaptation fault diagnosis method to reduce
the MMD between the source and the target domains. Jiao [20] introduces the
joint MMD to learn transferable features and formulates a residual domain
adaptation framework. Li [21] proposes a distribution alignment method to
reduce the difference between the two domains based on the central moment
metric. However, the pre-selected metric function may not always be applicable
and may result in significant computational costs for the neural networks.

Adversarial learning-based methods primarily achieve knowledge transfer by
implicitly measuring the difference between different domains and guiding the
model to learn domain invariant features based on the idea of Generative
Adversarial Networks (GANs). Ganin proposes the Domain Adversarial Neural
Networks (DANN) framework, introducing adversarial learning into domain
adaptation [9]. The main structure of DANN includes three parts: a feature
generator $G$, a domain discriminator $D$ and a label classifier $C$. During
the training, $G$ and $C$ are trained jointly to minimize the cross-entropy
loss on the source samples with labels. Additionally, $G$ and $D$ are trained
adversarially. $D$ tries to discriminate correctly whether a feature generated
by $G$ is from source or target sample while $G$ tries to fool $D$. By this
adversarial training manner, DANN decreases the discrepancy of the two domains
and extracts the domain invariant features for downstream tasks. Chen applies
DANN to the task of diagnosing rotating machinery faults [22]. Wang uses
Wasserstein GAN to make the adversarial learning process smoother [30]. Chai
extends the original DANN into a fine-grained version [10]. Huang builds a
domain adaptation network based on conditional adversarial learning [23].
However, the generated features are not guaranteed to be discriminative enough
even if the discriminator is confused. To solve this problem, several recent
works rely on pseudo-labels to ensure discriminative features. Lu assigns
generated features pseudo-labels based on clustering and corrects the
predicted class labels based on the pseudo-labels [15]. Jiao formulates a bi-
classifier structure to generate high-confidence pseudo-labels for the target
samples [24]. However, since the pseudo-labels are generated by the feature
generator or label classifier given by previous steps, they are not reliable
enough and may cause the training to diverge. Compared with the existing
approaches, the proposed Sd-CDA forces the feature extractor to generate
discriminative features without any pseudo-labels.

###  II-B Imbalanced domain adaptation

Current imbalanced domain adaptation methods mainly consider the intra-domain
imbalance. Based on the idea of sample generation, Zareapoor trains an
adversarial generative network model to generate minority class data and
diagnostic decisions simultaneously [25]. However, since the information of
minorities is very limited, the generated data may not match the real fault
distribution or even cause overfitting. Based on the idea of re-weighting,
Xiao combines TrAdaBoost to dynamically weight each sample point according to
the classification error of each source domain sample to address the with-
domain imbalance [26]. Based on the principle of feature transfer, Kuang re-
weights the loss of each class based on the imbalance ratio when aligning the
distributions of different domains [27]. Liu explored the sample weights based
on meta-learning [28]. However, as the target domain distribution is unknown,
the resampling methods derived from the source domain imbalance are often not
suitable for the target domain. In order to solve the inter-domain imbalance,
Wu [29] and Ding [4] align the features of every class separately via local
MMD to avoid the coverage of the majority. However, they still rely on the
unreliable pseudo-labels of the target samples to train the model. Compared
with these methods, the proposed model assigns weights to the minorities
automatically based on whether the model learns them well without any specific
functions or pseudo-labels. By doing this, the resulting weights are more
reliable and convenient to achieve.

###  II-C Contrastive learning

Deep learning relies heavily on substantial amounts of labeled training data.
However, data annotation can be costly, leading to significant interest in
self-supervised learning. Contrastive learning, an emerging self-supervised
technique, shows great promise in learning effective representations without
the need for label information [12, 13]. It learns effective visual
representations in the latent space in an unsupervised manner by enforcing the
similarity of positive pairs and enlarging the distance of negative pairs to
learn the visual representations. Simple contrastive representation learning
(SimCLR), one of the most prevalent contrastive learning methods, has
attracted a lot of attention [14]. Assume an unlabeled dataset $X$ whose data
size is $N$, SimCLR first augments it via two different augmentations and
results in $2N$ data points. For each augmented input, only the other version
from the same input is set to be the positive pair, and all the remaining
$2(N-1)$ samples are regarded as the negative pairs. The ultimate goal of
SimCLR is to minimize the Normalized Temperature-Scaled Cross-Entropy loss
(NT-Xent) which is defined as:

$\mathcal{L}_{g}=\frac{1}{N}\sum_{i=1}^{N}-\log\frac{d(v_{i},v_{i}^{+})}{d(v_{i},v_{i}^{+})+\sum_{v_{i}^{-}\in
V^{-}}d(v_{i},v_{i}^{-})},$ (1)

where $v_{i},v_{i}^{+}$ represents the projections of positive pairs,
$v_{i},v_{i}^{-}$ represents projections of negative pairs and $V^{-}$ means
the set of projections of negative pairs.
$d(\cdot,\cdot)=\exp(sim(\cdot,\cdot)/\tau)$ calculates the similarity between
two projections, $sim(\cdot,\cdot)$ represents a kernel function and $\tau$ is
the temperature parameter. On the basis of pure unsupervised contrastive
learning, Khosla proposes supervised contrastive learning (SupCon) by
considering all samples belonging to the same categories as positive against
the negatives from the remainder categories [30]. SupCon makes the samples of
different categories more compact by taking label information into account.

Recently, SimCLR has been introduced to domain adaptation approaches as a pre-
train step to avoid the overfitting of DANN towards the source domain. Lu
takes different transformed methods as pretext to pre-train the DANN model via
contrastive learning [15]. Azuma combines SimCLR with DANN and proposes the
contrastive domain adaptation framework [16]. However, contrastive learning of
minorities can be covered by the majorities under imbalanced settings because
the instance-rich class has much more separable information [17]. Compared
with the existing contrastive learning-based pre-train approaches, the
proposed method utilizes the difference between the original model and its
pruned counterpart to re-weight the samples implicitly without label
information.

##  III Methodologies

###  III-A Motivations and intuition

We use the SimCLR and DANN as examples to illustrate the impact of bi-
imbalance on contrastive learning-based pre-train methods and domain
adversarial learning in detail and introduce the main intuition of this work
in Figure 1.

The original SimCLR approach learns effective representations by maximizing
the agreement between different versions of augmented unlabeled input. Such
pre-train procedure helps the model learn effective feature extraction and
achieve fast and stable downstream convergence as shown in (a.1) - (a.2).
However this contrastive learning-based approach is highly vulnerable to
imbalanced data and may make some minority samples mis-featured as shown in
(a.2), and thus influence the downstream training. The existing approaches for
the imbalance problem rely on the label information to re-weight samples or
generate minorities. However, due to the inter-domain imbalance, the imbalance
of the target domain is unknown and cannot be estimated by the imbalance ratio
of the source domain. Based on this, a big challenge of implementing effective
contrastive learning for industrial fault diagnosis is how to solve the
possible imbalance in the data to avoid mis-featured samples without label
information of the target domain. Given that the pruned version of a neural
network tends to forget the minorities [18], it is a natural intuition to use
a pruned model to detect the minorities of both domains without the label
information as shown in the step (c.1) - (c.2).

In (b.1) - (b.2), we show the general steps of DANN: it decreases the
discrepancy between two domains to extract domain invariant features, and then
uses the classifier trained on the labeled source domain to classify the
target samples. However, the feature extractor can simply generate features
along the domain boundary because the goal of the feature extractor is to fool
the domain discriminator during training. What’s worse, the minorities can be
hardly corrected once they are along the domain boundary due to the coverage
of the majority when the imbalance exists. As for the misclassified source
samples, we can tune the sample weights of minorities to pay more attention to
the classification-related features of the minorities. However, how to force
the minorities of the target domain away from the domain boundary under bi-
imbalanced scenarios remains a big challenge. Since supervised contrastive
learning has the ability to make the samples of different categories more
compact, it holds the potential to help the domain discriminator drag the
samples along the domain boundary back. In order to handle the bi-imbalance
and enhance the detection of minorities of the target domain, we can also
implement the pruned version of the domain discriminator to help the training
as shown in (c.2) - (c.3).

###  III-B Problem formulation and overall structure

Assume that the source domain
$D_{s}=\\{(X^{i}_{s},y^{i}_{s})\\}_{i=1}^{n_{s}}$ and the target domain as
$D_{t}=\\{(X^{i}_{t})\\}_{i=1}^{n_{t}}$, where $n_{s}$ and $n_{t}$ are the
sample size of the source and the target domain, respectively,
$y^{i}_{s}\in\\{1,...,K\\}$ represents the classification label. Besides,
these two domains have different marginal distributions, i.e.,
$P(X_{s})\not=P(X_{t})$, and different conditional distributions, i.e.,
$P_{s}(y|X)\not=P_{t}(y|X)$. The main goal is to train a network for the
target task of cross-domain fault diagnosis, i.e., find accurate labels
$y^{i}_{t}$’s for the target samples. Figure 2 shows the overall structure of
the proposed method. It consists of two parts: imbalance-aware contrastive
representation learning (Ia-CLR) and boundary-aware adversarial domain
adaptation (Ba-ADA).

First, it implements contrastive learning to pre-train the feature extractor.
The original inputs, including $X_{s}^{\prime}s$ and $X_{t}^{\prime}s$, and
the corresponding augmented samples (pretext samples) constitute a new dataset
for contrastive learning so that the pre-trained feature extractor can
guarantee that its generated features are invariant to different domains. In
order to relieve the possible influence of bi-imbalance during the iteration,
we construct a new feature extractor by pruning the original one and formulate
a pruned contrastive learning to pre-train the feature extractor.

Then, it implements the boundary-aware adversarial domain adaptation to train
an efficient classifier for the target domain. In the Ba-ADA, we first train
the feature extractor, domain discriminator and label classifier together via
domain adversarial training. Then, it freezes the feature extractor and label
classifier and implements pruned supervised contrastive learning for the
domain discriminator to detect the samples along the domain boundary. By
repeating these steps, Ba-ADA can force the feature extractor to generate
informative features rather than features along the domain boundary and thus
train a more effective label classifier.

Figure 2: The overall structure of Sd-CDA. It consists of two parts:
imbalance-aware contrastive representation learning and boundary-aware
adversarial domain adaptation.

###  III-C Imbalance-aware contrastive learning of representations

In this work, we only implement one augmentation to the inputs and regard each
input with the augmented version as the positive pair. If we denote
$X=X_{s}\cap X_{t}$ and the augmented version of $X$ as $\hat{X}$, then
$(x_{i},\hat{x}_{i})$ is consider as a positive pair, where
$i=1,...,n_{s}+n_{t}$. Let’s denote the feature extractor as $G$ and the
projection head as $P$. Original SimCLR feeds the positive or negative pairs
to the same $G$ and $P$ and calculates the corresponding NT-Xent loss. In
order to focus on feature learning for minorities, we create a pruned version
of $G$ based on usual pruning tricks for neural networks [31], denoted as
$G^{p}$. The pruning process mainly removes the weights or biases with small
magnitudes or sets a small portion of the weights to zero randomly. The top
part of Figure 2 shows the detailed training process of the Ia-CLR. We first
feed the pair separately $(x_{i},\hat{x}_{i})$ into two feature extractors,
i.e., $G$ and $G^{p}$, and get a pair of generated features
$(G(x_{i}),G^{p}(\hat{x}_{i}))$. Then they get projected as
$(P(G(x_{i})),P(G^{p}(\hat{x}_{i})))$ through the same projection head $P$. We
define the Pruned Normalized Temperature-Scaled Cross-Entropy loss (PNT-Xent)
to enforce the similarity between the positive pair
$(P(G(x_{i})),P(G^{p}(\hat{x}_{i})))$. If we define $v_{i}=P(G(x_{i}))$ and
$\hat{v}_{i}^{+}=P(G^{p}(\hat{x}_{i}))$ for simplicity, then we can construct
the PNT-Xent of the whole system as follows:

$\mathcal{L}_{g}^{p}=\frac{1}{n_{s}+n_{t}}\sum_{i=1}^{n_{s}+n_{t}}-\log\frac{d(v_{i},\hat{v}_{i}^{+})}{d(v_{i},\hat{v}_{i}^{+})+\sum_{v_{i}^{-}\in
V^{-}}d(v_{i},v_{i}^{-})},$ (2)

where $V^{-}=\\{(X\backslash
v_{i})\cup(\hat{X}\backslash\hat{v}_{i}^{+}\\})\\}$ represents the set of
negative pairs. Based on [18], if $x_{i}$ belongs to the minority, the
difference between the corresponding projections $v_{i}$ and $\hat{v}_{i}^{+}$
should be larger than that of majority samples. As a result, the proportion of
the loss caused by minorities should be more significant than that of
majorities so that the training will be forced to pay more attention to
minorities automatically.

Compared with the original SimCLR, the Ia-CLR can detect minorities and
enlarge their proportion in the loss function automatically to avoid bias
toward majorities in the training process. Compared with the existing
imbalance learning approaches in supervised settings, there is no need to
search for optimal weights or pseudo-labels for the minorities to do re-
weighting or re-balancing in Ia-CLR. More importantly, Ia-CLR is highly
suitable for the UDA setting because it does not need the labels for the
target samples.

###  III-D Boundary-aware adversarial domain adaptation

The original domain adversarial learning may fail because the feature
extractor can simply generate samples along the domain boundary to fool the
domain discriminator, especially for the target samples, because we do not
have the corresponding task-specific information available for target samples.
To solve this problem, we propose the boundary-aware adversarial domain
adaptation (Ba-ADA) framework to detect the features along the domain boundary
and force the feature extractor to generate discriminative features for all
samples. The main intuition of the Ba-ADA also comes from [18], i.e.,
difficult samples are hard for DNNs to memorize well and may be easily
forgotten them the DNNs are pruned. The samples along the domain boundary are
difficult samples for $D$ with no doubt because the predictive results of
these samples can change easily with even a small disturbance of $D$.

Ba-ADA includes four parts: feature extractor $G$, domain discriminator $D$,
pruned domain discriminator $D^{p}$ and the label classifier $C$. The bottom
part of Figure 2 shows the detailed structure of Ba-ADA. The training of Ba-
ADA consists of three steps in each iteration: supervised learning, domain
adversarial training, and boundary-aware contrastive learning.

In the supervised learning stage, we use the labeled source samples to train
the feature extractor $G$ and label classifier $C$ to accurately predict
labels for the source domain based on the label classification loss
$\mathcal{L}_{l}$ defined as follows:

$\mathcal{L}_{c}=-\frac{1}{n_{s}}\sum_{i=1}^{n_{s}}\sum_{k=1}^{K}\mathbf{1}_{[y_{s}=k]}\log(C(G(x_{s}^{i}))).$
(3)

Next, we implement adversarial training to narrow the discrepancy between the
source domain and the target domain. The adversarial loss is defined as
follows:

$\mathcal{L}_{d}=-\sum_{i=1}^{n_{s}}\log(D(G(x_{s}^{i})))-\sum_{j=1}^{n_{t}}\log(1-D(G(x_{t}^{j}))).$
(4)

Last, we introduce the boundary-aware constrastive learning. As discussed
before, original adversarial training cannot detect the samples along the
domain boundary, especially for the target samples, and thus the generated
domain invariant features are not reliable. Supervised contrastive learning
(SupCon) [30] is proposed to guide the model to output similar features if the
inputs come from the same class. As a result, the SupCon has the potential to
guide the training of $D$ to push the samples along the boundary back to the
corresponding domain and make the output of $D$ given by each domain more
compact. Following this idea, we first define the supervised contrastive
domain adversarial loss (SupCon-DA). Denote the output of $D$ for $X_{s}$ and
$X_{t}$ as $U_{s}=D(G(X_{s}))$ and $U_{t}=D(G(X_{t}))$ for simplicity. Also
denote $U=U_{s}\cup U_{t}$ as the overall output space of $D$. Different from
SimCLR, here we consider all the outputs in $U_{s}$ as positive pairs mutually
and the same for $U_{t}$ to utilize the domain label information. By doing
this, the SupCon-DA loss encourages $D$ to give close outputs to all features
from the same domain and results in a more robust clustering of output space
to avoid being misled by the samples along the domain boundary generated by
$G$. Let’s use one sample $u_{s}^{i}$ in $U_{s}$ to illustrate the definition
of SupCon-DA loss:

$\mathcal{L}_{bd}(u_{s}^{i})=\frac{1}{n_{s}-1}\sum_{q\in
A_{s}(u_{s}^{i})}\log\frac{d(u_{s}^{i},q)}{\sum_{p\in
B(u_{s}^{i})}d(u_{s}^{i},p)},$ (5)

where $A_{s}(u_{s}^{i})=\\{U_{s}\backslash u_{s}^{i}\\}$ represents all the
positive samples corresponding to $u_{s}^{i}$, and
$B(u_{s}^{i})=\\{U\backslash u_{s}^{i}\\}$ represents all the other samples in
the overall space $U$. Please note that the loss 5 for the target samples also
takes the same format by switching the subscripts $s$ to $t$ only. Then we can
define the overall SupCon-DA loss of $U$ as follows:

$\mathcal{L}_{bd}=\frac{1}{n_{s}}\sum_{i=1}^{n_{s}}\mathcal{L}_{bd}(u_{s}^{i})+\frac{1}{n_{t}}\sum_{i=1}^{n_{t}}\mathcal{L}_{bd}(u_{t}^{i}).$
(6)

If we fine-tune the domain discriminator $D$ by minimizing $\mathcal{L}_{bd}$,
we can keep the predictive results of $D$ of different domains away from the
boundary effectively.

However, this SupCon-DA learning is also vulnerable to the possible bi-
imbalance just like what we discussed about SimCLR before. As the training
process goes on, the difficult samples of the majority of the two domains can
be well detected and pushed away from the boundary by $D$. However, the
difficult samples of the minority can be easily neglected because the
corresponding gradients may be diluted by the majority in the SupCon-DA loss.
As a result, we construct a pruned version of the latest $D$, denoted as
$D^{p}$, to help detect the difficult samples of the minority one step
further. The intuition behind this approach is quite simple: given a difficult
sample for the minority, the predictive domain labels of $D^{p}$ and $D$ can
be absolutely different, thus intriguing a large disturbance to the
corresponding SupCon-DA loss to draw attention from $D$. Here we use
$U_{s}^{p}=D^{p}(G(X_{s}))$, $U_{t}^{p}=D^{p}(G(X_{t}^{p}))$ and
$U^{p}=U^{p}_{s}\cup U_{t}^{p}$ to denote the outputs of $D^{p}$
correspondingly. Then we can define the Pruned Supervised Contrastive Domain
Adversarial Loss (PSupCon-DA) as:

$\mathcal{L}_{bd}^{p}=\frac{1}{n_{s}}\sum_{i=1}^{n_{s}}\mathcal{L}_{bd}(u_{sp}^{i})+\frac{1}{n_{t}}\sum_{i=1}^{n_{t}}\mathcal{L}_{bd}(u_{tp}^{i}),$
(7)

where $u_{sp}^{i}=D^{p}(G(x_{s}^{i}))$ and $u_{tp}^{i}=D^{p}(G(x_{t}^{i}))$.
Please note that the original

We fine-tune $D$ based on $\mathcal{L}_{cd}^{p}$ rather than
$\mathcal{L}_{cd}$ to solve the imbalance challenge.

###  III-E Training objectives and training procedures

In this section, we will discuss the detailed training procedures and
objectives of the proposed method. Let’s denote the parameters of feature
extractor $G$, pruned feature extractor $G^{p}$, projection head $H$, domain
discriminator $D$, pruned domain discriminator $D^{p}$ and label classifier
$C$ as $\theta_{g}$, $\theta_{g}^{p}$, $\theta_{h}$, $\theta_{d}$,
$\theta_{d}^{p}$ and $\theta_{c}$, respectively. The ultimate objective of the
proposed method can be written as:

$\min_{\theta_{g},\theta_{c}}\max_{\theta_{d}}\mathcal{L}_{g}^{p}+\lambda_{c}\mathcal{L}_{c}-\lambda_{d}\mathcal{L}_{d}+\lambda_{bd}\mathcal{L}_{bd}^{p},$
(8)

where $\lambda_{c}$, $\lambda_{d}$ and $\lambda_{bd}$ represent the weights of
different loss, respectively. Next, we will introduce the training process of
8 in detail.

In the Ia-CLR, we train the feature extractor $G$ in a self-supervised manner
to learn the feature representation of the source and the target samples,
i.e., find the optimal $\theta_{g}$ and $\theta_{h}$ to minimize the PNT-Xent
loss $\mathcal{L}_{g}^{p}$. Since $\mathcal{L}_{g}^{p}$ is calculated based on
$G^{p}$ with parameter $\theta_{g}^{p}$ we cannot backpropagate the gradient
of $\theta_{g}$ directly. As a result, we update the parameters in
$\theta_{g}$ corresponding to $\theta_{g}^{p}$ and freeze the remaining parts
(pruned parameters) in each iteration. Specifically, if we denote the
$\theta_{g}=\\{\theta_{g}^{p},\bar{\theta}_{g}^{p}\\}$ with
$\bar{\theta}_{g}^{p}$ denoting the pruned parameters, the training process of
IaCLR can be written as:

$\theta_{g}\xrightarrow{p}\\{\theta_{g}^{p},\bar{\theta}_{g}^{p}\\}$
$\theta_{g}^{p}\xleftarrow{}\theta_{g}^{p}-\omega\frac{\partial{\mathcal{L}_{g}^{p}}}{\partial\theta_{g}^{p}}$
$\theta_{h}\xleftarrow{}\theta_{h}-\omega\frac{\partial{\mathcal{L}_{g}^{p}}}{\partial\theta_{h}}$
$\theta_{g}\xleftarrow{}\\{\theta_{g}^{p},\bar{\theta}_{g}^{p}\\}$

where $\xrightarrow{p}$ represents pruning process and $\omega$ is the basic
learning rate.

In Ba-ADA, we will use $\theta_{g}$ trained by IaCLR as the initial parameters
of $G$. The training process of Ba-ADA can be divided into two parts. In the
first part, we implement the original domain adversarial learning to train
$G$, $D$ and $C$ as follows:

$\theta_{g}\xleftarrow{}\theta_{g}-\lambda_{c}\omega\frac{\partial{\mathcal{L}_{c}}}{\partial\theta_{g}}+\lambda_{d}\omega\frac{\partial{\mathcal{L}_{d}}}{\partial\theta_{g}}$
$\theta_{d}\xleftarrow{}\theta_{d}-\lambda_{d}\omega\frac{\partial{\mathcal{L}_{d}}}{\partial\theta_{d}}$
$\theta_{c}\xleftarrow{}\theta_{c}-\lambda_{c}\omega\frac{\partial{\mathcal{L}_{c}}}{\partial\theta_{c}}$

Next, we use the PSupCon-DA loss to fine-tune $D$ based on the pruned version
$D^{p}$. Similar to the IaCLR, we cannot update $D$ by minimizing
$\mathcal{L}_{bd}^{p}$ directly. Again, we denote
$\theta_{d}=\\{\theta_{d}^{p},\bar{\theta}_{d}^{p}\\}$ then the training
process of SupCon-DA can be written as:

$\theta_{d}\xrightarrow{p}\\{\theta_{d}^{p},\bar{\theta}_{d}^{p}\\}$
$\theta_{d}^{p}\xleftarrow{}\theta_{d}^{p}-\lambda_{bd}\omega\frac{\partial{\mathcal{L}_{bd}^{p}}}{\partial\theta_{d}^{p}}$
$\theta_{d}\xleftarrow{}\\{\theta_{d}^{p},\bar{\theta}_{d}^{p}\\}$

###  III-F Implementation details

In this section, we will discuss some guidelines for the choice of hyper-
parameters and model structure in the proposed method.

####  III-F1 Structure of neural networks

In Figure 2, we only use some simple structures of fully connected networks
(FNNs) to illustrate the proposed method. However, in real-world applications,
we need to design different structures applicable to data with different
properties. For example, we need to incorporate the CNN-pool module when
facing vibration signals or images but use FNNs or Resnet structures for
structured data. We will show different structures in our experiments.

####  III-F2 Pruning methods

The prevailing pruning methods include random pruning methods [32] and
magnitude-based pruning methods [33]. Random pruning methods set a proportion
of a model’s parameters to zero randomly while magnitude-based pruning methods
mask the model’s parameters with the smallest absolute values based on a
predefined proportion hyper-parameter. To maintain a stable training process
and avoid large vibration of the pruned model-related loss, we implement
magnitude-based pruning methods based on the L1 norm [31]. As for the choice
of proportion hyper-parameter $\alpha_{g}$ in IaCLR and $\alpha_{d}$ in Ba-
ADA, we have several empirical guidelines based on our experiments.

In IaCLR, a higher $\alpha_{g}$ often leads to a faster decrease of
$\mathcal{L}_{g}^{p}$ at the beginning stage because a higher $\alpha_{g}$
causes a larger disagreement between $G$ and $G^{p}$. However, the loss may
not decrease after a certain amount of iterations or even increase as the
training process, which we call the ”bottleneck” of training. This bottleneck
is due to the large difference between the feature generator and its pruned
model. We cannot guarantee converged training when the bottleneck appears. As
a result, we recommend a relatively small $\alpha_{g}$, i.e.,
$0.1<\alpha_{g}<0.3$ for Ia-CLR to maintain stable training.

In Ba-ADA, we empirically find that a larger $\alpha_{d}$ generally leads to a
higher accuracy for the the final classifier. The possible reason is due to
the motivation of PSupCon-DA, i.e., higher $\alpha_{d}$ leads to a more
significant disturbance of $\mathcal{L}_{bd}^{p}$ to remind it of more
difficult samples. However, a larger disturbance of $\mathcal{L}_{bd}^{p}$ may
cause gradient explosion and make the whole training diverge. As a result, we
recommend choosing the optimal $\alpha$ for PSupCon-DA carefully based on the
corresponding loss value of $\mathcal{L}_{bd}^{p}$. Based on our experiments,
we would recommend $0.3<\alpha_{d}<0.5$ for SupCon-DA. We will show detailed
experiments regarding the choice of $0.3<\alpha_{d}<0.5$ in Section IV.

####  III-F3 Basic learning rate $\omega$ and loss weights $\lambda_{c}$,
$\lambda_{d}$ and $\lambda_{bd}$

Different from the training of traditional neural networks, domain adaptation-
related methods cannot use accuracy on the validation set to set the learning
rate and stop criterion because we do not have labels for the target domain.
In this work, we choose the basic learning rate $\omega$ according to the loss
trend of the Ia-CLR training process. Since a higher $\omega$ also means a
faster decrease of $\mathcal{L}_{g}^{p}$ and may get bottleneck just like what
a higher $\alpha_{g}$ may cause in Ia-CLR. As a result, we will recommend a
small $\omega$ as long as the training process can proceed and the overall
training process does not exceed the time requirements.

Loss weights $\lambda_{c}$ and $\lambda_{d}$ determine the adversarial
training of $G$ and $D$. Generally, $G$ has a more complex structure than $D$
which means $\theta_{g}$ needs more training than $\theta_{d}$. As a result,
we recommend setting $\lambda_{c}>\lambda_{d}$. As for $\lambda_{bd}$, we also
recommend a relatively small value to guarantee stable training, i.e.,
$\lambda_{bd}<\lambda_{d}$.

##  IV Case studies

In this section, we will evaluate the proposed Sd-CDA on two different cases,
i.e., a mechanical rolling bearing dataset and an industrial three-phase flow
process dataset. We compare the proposed method with five different approaches
including the original CNNs, joint domain adaptation (JDA) [34], domain
adversarial neural networks (DANN) [9], contrastive domain adaptation (ConDA)
[16] and an imbalanced domain adaptation (Imba-DA) [29].

###  IV-A Mechanical rolling bearing case

The mechanical rolling bearing dataset provided by Case Western Reserve
University includes vibration signals collected from different positions of
the bearing [35], i.e., fan end and drive end, and different working speeds,
i.e., 1730 rpm, 1750 rpm, 1772 rpm and 1797 rpm. In this experiment, we choose
the data collected on the drive end with a sampling frequency of 12kHz and
consider two working speeds with the largest differences, i.e., 1730 rpm and
1797 rpm, to construct two different domain adaptation cases, i.e., 1730 rpm
$\xrightarrow{}$ 1797 rpm and 1797 rpm $\xrightarrow{}$ 1730 rpm.

Specifically, this dataset consists of a normal condition (N) and a total of
nine fault categories: three different fault modes, i.e., inner ring fault
(IR), ball fault (BF) and outer ring fault (ORF), and three different fault
diameters, i.e., 0.007 inch (S), 0.014 inch (M) and 0.028 inch (L). We
construct one balance setting and three imbalance settings to simulate the
possible bi-imbalance to test the performance of the proposed domain
adaptation method, i.e., balanced source domain to balanced target domain
(B2B), balanced source domain to imbalanced target domain (B2I), imbalanced
source domain to balanced target domain (I2B), imbalanced source domain to
imbalanced target domain (I2I). As a result, we have a total of eight
scenarios as shown in Table I. Since the original dataset is approximately
balanced, we use all the original samples of two domains to formulate the B2B
setting. As for the other three imbalanced settings, we sample the original
dataset for different categories based on the sample size as shown in Figure
3. As we can see, the B2I and I2B settings simulate the scenario with bi-
imbalance, and the bi-imbalance is much more severe in the I2I setting
setting.

TABLE I: All scenarios with different cases and different imbalance settings. Case | Source | Target | Imbalance settings
---|---|---|---
1 | 1797 rpm | 1730 rpm | B2B, B2I, I2B, I2I
2 | 1730 rpm | 1797 rpm | B2B, B2I, I2B, I2I
Figure 3: Sample size for CWRU dataset in different settings

In order to extract the information in the vibration signals, we follow [36]
to convert the original data into $18\times 18$ images by wavelet transform. A
CNN-based structure with Conv($3\times 3$, 32)-Pool-Conv($3\times 3$,
64)-Pool-Conv($3\times 3$, 128) is used as the basic feature extractor to
construct different diagnosis structures where Conv($3\times 3$, 128) means
that the kernel size is $3\times 3$ and the number of output channels is 128.
The basic domain discriminator, basic projection head and basic label
classifier are all set to FNNs with two layers and the basic learning rate
$\omega=0.001$. The implementation details of the six methods are as follows:

  * •

CNN: Basic feature extractor + basic label classifier;

  * •

JDA: Basic feature extractor + basic label classifier, regularize parameter
$\lambda=1$;

  * •

DANN: Basic feature extractor + basic label classifier and basic domain
discriminator;

  * •

ConDANN: basic feature extractor + basic projection head as the SimCLR model +
DANN;

  * •

Imba-DA: Basic feature extractor + basic label classifier and basic domain
discriminator with $\mu_{1}=0.95$ and $\mu_{2}=0.05$;

  * •

Sd-CDA: Basic feature extractor + basic projection head as the Ia-CLR mode
where the pruning method is L1-norm pruning with pruning proportion
$\alpha_{g}=0.25$. DANN with $\lambda_{c}=\lambda_{d}=1$ and
$\lambda_{bd}=10^{-7}$ as the Ba-ADA model and the L1-norm pruning with
$\alpha=0.4$ is used to prune the domain discriminator.

We average the diagnosis accuracy of two domain adaptation cases and list them
in Table II. The classic DA approaches JDA and DANN reach good accuracy for
the B2B setting but their accuracy drops dramatically for the three imbalanced
settings because they do not consider the imbalance. With the help of
contrastive learning, the ConDA gets higher accuracy compared with DANN
because the pre-tain process helps ConDA learn the feature representation more
efficiently, but it still fails for imbalanced settings. On the basis of DANN,
the imba-DA gets higher accuracy because it considers the imbalance
distributions of two domains and implements cost-sensitive re-weighting to
guide the training. However, the cost-sensitive policy cannot detect the
samples on the domain boundary, especially for the target samples. The
proposed Sd-CDA beats other DA or imbalanced DA approaches for three
imbalanced settings, especially for the I2B and I2I settings.

Figure 4 shows the detailed diagnosis accuracy for all categories under three
imbalanced settings. We can see that the MIRF and LBF faults are difficult to
detect in general. Compared with other approaches, the proposed method can get
approximately the same performance for other ’easy’ categories, but for the
’difficult’ categories, the proposed method performs much better than others.
Especially for the LBF fault, the proposed method achieves at least twice the
recognition accuracy compared to other methods. Figure 4 proves the
superiority of Ia-CLR and Ba-ADA towards the imbalanced samples.

TABLE II: Average diagnosis accuracy for different methods under four settings. Methods | B2B | B2I | I2B | I2I
---|---|---|---|---
CNN | 82.43 | 80.44 | 62.51 | 51.19
JDA | 92.34 | 84.24 | 72.01 | 54.49
DANN | 87.27 | 83.92 | 72.31 | 56.04
ConDA | 92.26 | 85.64 | 78.24 | 58.24
Imba-DA | 91.37 | 86.42 | 79.23 | 64.58
Ours | 91.26 | 90.21 | 84.92 | 69.54
Figure 4: Class-wise diagnosis accuracy for CWRU dataset.

###  IV-B Industrial three-phase flow process

Cranfield University’s three-phase flow facility is designed to deliver
precise, controlled flow rates of water, oil, and air into a pressurized
system [37]. The facility can manage either single phases of these fluids or
their combinations in specific proportions. During this experimentation, two
primary process inputs, i.e., the air and water flow rates, and the other 24
variables are considered. All data is collected with a 1 Hz frequency.
Notably, the facility allows for the selection of four distinct air flow rates
and five water flow rates, which leads to different working conditions and
data distributions. In this experiment, we choose the data under two different
water flow rate as our domain A and domain B and consider four faults. The
detailed information of the original datasets is shown in Table III. We also
follow Table I to construct two domain adaptation cases based on domain A and
domain B and test the performance of different methods under B2B, B2I, I2B,
and I2I settings. Figure 5 shows the sample ratio on the basis of the original
dataset in different imbalanced settings.

TABLE III: Dataset information of three-phase flow process | Domain A | Domain B
---|---|---
Fault class | Water flow | No. of | Water flow | No. of
| rate | samples | rate | samples
Fault 1 | 2 | 3121 | 3.5 | 3001
Fault 2 | 2 | 2181 | 3.5 | 2137
Fault 3 | 2 | 3000 | 3.5 | 3001
Fault 4 | 2 | 3000 | 3.5 | 3001
Figure 5: Sample ratio for Three-phase flow process in imbalanced settings

In this experiment, we use the FNNs to construct the feature extractor.
Specifically, an FNN(100)-Dropout(0.3)-FNN(24)-Dropout(0.3) structure is used
as the basic feature extractor. The basic domain discriminator, basic
projection head and basic label classifier are all set to FNNs with two layers
and the basic learning rate $\omega=0.0001$. The implementation details of
different methods are as follows:

  * •

FNN: Basic feature extractor + basic label classifier;

  * •

JDA: Basic feature extractor + basic label classifier, regularize parameter
$\lambda=1$;

  * •

DANN: Basic feature extractor + basic label classifier and basic domain
discriminator;

  * •

ConDANN: basic feature extractor + basic projection head as the SimCLR model +
DANN;

  * •

Imba-DA: Basic feature extractor + basic label classifier and basic domain
discriminator with $\mu_{1}=0.95$ and $\mu_{2}=0.05$;

  * •

Sd-CDA: Basic feature extractor + basic projection head as the Ia-CLR mode
where the pruning method is L1-norm pruning with pruning proportion
$\alpha=0.25$. DANN with $\lambda_{c}=1$, $\lambda_{d}=0.001$ and
$\lambda_{bd}=10^{-8}$ as the Ba-ADA model and the L1-norm pruning with
$\alpha=0.3$ is used to prune the domain discriminator.

Table IV lists the average diagnosis accuracy of two domain adaptation cases,
i.e., A to B and B to A. We can see that the proposed method achieves the best
performance in general, especially for the B2I and I2I settings. One can see
that in the I2I setting, the classific DA approaches almost fail because the
impact of imbalance hinders the classifier heavily. The proposed method gets
much higher accuracy, over twice the accuracy compared with the original DANN,
which proves that the proposed method is able to relieve the bi-imbalance and
get a satisfactory classifier.

TABLE IV: Average diagnosis accuracy for different methods under four settings Methods | B2B | B2I | I2B | I2I
---|---|---|---|---
FNN | 65.54 | 42.12 | 55.51 | 21.21
JDA | 72.32 | 40.22 | 54.01 | 22.32
DANN | 71.32 | 65.54 | 43.53 | 21.24
ConDA | 56.21 | 59.42 | 46.25 | 44.26
Imba-DA | 78.54 | 62.32 | 61.45 | 55.23
Ours | 84.23 | 66.23 | 61.25 | 58.25

Figure 6 shows the class-wise accuracy for three imbalanced settings. We can
see that almost all other methods cannot detect fault 4 but the proposed
method can achieve a certain amount of detection. As for the other three
faults in B2I and I2B settings, the proposed method gets approximately the
same good performance as other methods. In the B2I setting, DANN gets higher
accuracy for fault 1 but lower accuracy for fault 2 and fault 4 compared with
the proposed method, so the proposed method still gets better overall
accuracy. In the I2B setting, even though ConDA detects fault 1 better, it
fails for fault 3 and fault 4. In the I2I setting, the superiority of the
proposed method is quite clear for all faults.

Figure 6: Class-wise diagnosis accuracy for three-phase flow process data.

###  IV-C In-depth studies of the proposed method

####  IV-C1 Analysis for different $\alpha_{d}$ and $\lambda_{bd}$ for the
proposed method

In this experiment, we test the performance of the proposed method under
different $\alpha_{d}$ and $\lambda_{bd}$ to explore some empirical
guidelines. We use the 1797 $\xrightarrow{}$ 1730 setting to implement the
experiment.

First, we test the proposed method using different pruning proportions
$\alpha_{d}$ for the Ba-ADA, ranging from 0.1 to 0.6. In this experiment, we
use L1 pruning methods for Ia-CLR and Ba-ADA and set $\alpha_{g}=0.25$,
$\lambda_{c}=\lambda_{d}=1$ and $\lambda_{bd}=10^{-7}$. The left subfigure of
Figure 7 shows the accuracy under different $\alpha_{d}$. When $\alpha_{d}$ is
small, the accuracy increases as $\alpha_{d}$. The possible reason is that a
higher $\alpha_{d}$ will cause larger $\mathcal{L}_{bd}^{p}$ leading to a
heavier push to the discriminator to detect and pay attention to the sample
along the boundary. However, if the resulted $\mathcal{L}_{bd}^{p}$ is too
large, it may cover the gradient of $\mathcal{L}_{d}$ and $\mathcal{L}_{c}$.
As a result, we cannot reach efficient domain adaptation. More importantly,
the $\mathcal{L}_{bd}^{p}$ will remain large and cannot decrease so that the
overall training process will diverge if a large $\alpha_{d}$ is used. As a
result, the accuracy drops dramatically when $\alpha_{d}$ increases after 0.4.

Then we test the proposed method using different $\lambda_{bd}$, ranging from
$10^{-9}$ to $10^{-5}$. In this experiment, we use L1 pruning methods for Ia-
CLR and Ba-ADA and set $\alpha_{g}=0.25$, $\alpha_{d}=0.4$
$\lambda_{c}=\lambda_{d}=1$. The left subfigure of Figure 7 shows the accuracy
of this experiment. We can see that the $\lambda_{bd}$ also shows the same
trend as the $\alpha_{d}$ because a larger $\lambda_{bd}$ also leads to a
larger update step caused by $\mathcal{L}_{bd}^{p}$. Different from the impact
of different $\alpha_{d}$, the accuracy changes more heavily along
$\lambda_{bd}$. As a result, we consider $\lambda_{bd}$ as the coarse
adjustment and $\alpha_{d}$ as the fine adjustment of the proposed method. In
other words, we need to first choose a suitable $\lambda_{bd}$ to make sure a
stable changing process of $\mathcal{L}_{bd}^{p}$ and then tune the
$\alpha_{d}$ to get better accuracy.

####  IV-C2 Ablation study

We also implement ablation studies to test the efficiency of IaCLR and BaADA
separately based on the 1797 $\xrightarrow{}$ 1730 setting. The benchmark is
chosen as the original DANN. We add two key parts of the Sd-CDA step by step
on the basis of DANN and generate three situations, i.e., DANN + Ia-CLR, DANN
+ Ba-ADA, and DANN + Ia-CLR + Ba-ADA (Sd-CDA) as shown in Table V.

From Table V we can see that, these two components are all important to the
proposed Sd-CDA, i.e., they all contribute to a better diagnosis accuracy,
especially in the three situations with imbalance. By comparison, the
contribution of Ba-ADA tends to be slightly lower than Ia-CLR in B2I and I2B
situations but higher in the I2I situation. This phenomenon is quite
reasonable. In B2I and I2B situations, the most decisive question is to learn
effective feature representation to leverage the information of samples
because we have more samples. As a result, the pre-train strategy stands out.
But in the I2I situation, the imbalances are quite severe and also different
in the two domains. In this situation, how to handle the imbalance when
training a stable classifier becomes more important. This is why BaADA plays a
significant role in the I2I situation.

Figure 7: Accuracy of the proposed method under different $\alpha_{d}$ and $\lambda_{bd}$ TABLE V: Ablation study for the proposed method Ia-CLR | Ba-ADA | B2B | B2I | I2B | I2I
---|---|---|---|---|---
|  | 89.27 | 86.24 | 74.58 | 64.78
$\checkmark$ |  | 86.34 | 87.35 | 82.34 | 69.42
| $\checkmark$ | 88.25 | 86.25 | 80.05 | 74.25
$\checkmark$ | $\checkmark$ | 93.14 | 91.14 | 86.30 | 77.74

##  V Conclusion

In this work, we propose an Sd-CDA framework for the industrial fault
diagnosis tasks with both domain discrepancy and bi-imbalance challenges. It
first pre-trains the feature extractor via pruned contrastive learning to
learn efficient feature representations for the bi-imbalance data. Then it
constructs a pruned supervised contrastive domain adversarial learning
approach to avoid the learned features biased to the source domain or along
the domain boundary. Additionally, we provide several guidelines for choosing
the hyper-parameters of the proposed framework based on the empirical
experiments. A series of experiments validate the superiority of our work.

## References

  * [1] Z. Yang and Z. Ge, “On paradigm of industrial big data analytics: From evolution to revolution,” _IEEE Transactions on Industrial Informatics_ , vol. 18, no. 12, pp. 8373–8388, 2022.
  * [2] H. Wu and J. Zhao, “Deep convolutional neural network model based chemical process fault diagnosis,” _Computers & chemical engineering_, vol. 115, pp. 185–197, 2018.
  * [3] B. Song and H. Shi, “Fault detection and classification using quality-supervised double-layer method,” _IEEE Transactions on Industrial Electronics_ , vol. 65, no. 10, pp. 8163–8172, 2018.
  * [4] Y. Ding, M. Jia, J. Zhuang, Y. Cao, X. Zhao, and C.-G. Lee, “Deep imbalanced domain adaptation for transfer learning fault diagnosis of bearings under multiple working conditions,” _Reliability Engineering & System Safety_, vol. 230, p. 108890, 2023.
  * [5] C. Li, S. Zhang, Y. Qin, and E. Estupinan, “A systematic review of deep transfer learning for machinery fault diagnosis,” _Neurocomputing_ , vol. 407, pp. 121–135, 2020.
  * [6] H. Chen, H. Luo, B. Huang, B. Jiang, and O. Kaynak, “Transfer learning-motivated intelligent fault diagnosis designs: A survey, insights, and perspectives,” _IEEE Transactions on Neural Networks and Learning Systems_ , 2023.
  * [7] A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola, “A kernel two-sample test,” _The Journal of Machine Learning Research_ , vol. 13, no. 1, pp. 723–773, 2012.
  * [8] B. Sun and K. Saenko, “Deep coral: Correlation alignment for deep domain adaptation,” in _Computer Vision–ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part III 14_. Springer, 2016, pp. 443–450.
  * [9] Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. March, and V. Lempitsky, “Domain-adversarial training of neural networks,” _Journal of machine learning research_ , vol. 17, no. 59, pp. 1–35, 2016.
  * [10] Z. Chai and C. Zhao, “A fine-grained adversarial network method for cross-domain industrial fault diagnosis,” _IEEE Transactions on Automation Science and Engineering_ , vol. 17, no. 3, pp. 1432–1442, 2020.
  * [11] K. Saito, K. Watanabe, Y. Ushiku, and T. Harada, “Maximum classifier discrepancy for unsupervised domain adaptation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 3723–3732.
  * [12] M. Noroozi and P. Favaro, “Unsupervised learning of visual representations by solving jigsaw puzzles,” in _European conference on computer vision_. Springer, 2016, pp. 69–84.
  * [13] P. Bachman, R. D. Hjelm, and W. Buchwalter, “Learning representations by maximizing mutual information across views,” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [14] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in _International conference on machine learning_. PMLR, 2020, pp. 1597–1607.
  * [15] N. Lu, H. Xiao, Z. Ma, T. Yan, and M. Han, “Domain adaptation with self-supervised learning and feature clustering for intelligent fault diagnosis,” _IEEE Transactions on Neural Networks and Learning Systems_ , 2022.
  * [16] C. Azuma, T. Ito, and T. Shimobaba, “Adversarial domain adaptation using contrastive learning,” _Engineering Applications of Artificial Intelligence_ , vol. 123, p. 106394, 2023.
  * [17] Z. Jiang, T. Chen, B. J. Mortazavi, and Z. Wang, “Self-damaging contrastive learning,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 4927–4939.
  * [18] S. Hooker, A. Courville, G. Clark, Y. Dauphin, and A. Frome, “What do compressed deep neural networks forget?” _arXiv preprint arXiv:1911.05248_ , 2019.
  * [19] W. Lu, B. Liang, Y. Cheng, D. Meng, J. Yang, and T. Zhang, “Deep model based domain adaptation for fault diagnosis,” _IEEE Transactions on Industrial Electronics_ , vol. 64, no. 3, pp. 2296–2305, 2016.
  * [20] J. Jiao, M. Zhao, J. Lin, and K. Liang, “Residual joint adaptation adversarial network for intelligent transfer fault diagnosis,” _Mechanical Systems and Signal Processing_ , vol. 145, p. 106962, 2020.
  * [21] X. Li, Y. Hu, J. Zheng, M. Li, and W. Ma, “Central moment discrepancy based domain adaptation for intelligent bearing fault diagnosis,” _Neurocomputing_ , vol. 429, pp. 12–24, 2021.
  * [22] Z. Chen, G. He, J. Li, Y. Liao, K. Gryllias, and W. Li, “Domain adversarial transfer network for cross-domain fault diagnosis of rotary machinery,” _IEEE Transactions on Instrumentation and Measurement_ , vol. 69, no. 11, pp. 8702–8712, 2020.
  * [23] Z. Huang, Z. Lei, G. Wen, X. Huang, H. Zhou, R. Yan, and X. Chen, “A multisource dense adaptation adversarial network for fault diagnosis of machinery,” _IEEE Transactions on Industrial Electronics_ , vol. 69, no. 6, pp. 6298–6307, 2021.
  * [24] J. Jiao, H. Li, and J. Lin, “Self-training reinforced adversarial adaptation for machine fault diagnosis,” _IEEE Transactions on Industrial Electronics_ , 2022.
  * [25] M. Zareapoor, P. Shamsolmoali, and J. Yang, “Oversampling adversarial network for class-imbalanced fault diagnosis,” _Mechanical Systems and Signal Processing_ , vol. 149, p. 107175, 2021.
  * [26] D. Xiao, Y. Huang, C. Qin, Z. Liu, Y. Li, and C. Liu, “Transfer learning with convolutional neural networks for small sample size problem in machinery fault diagnosis,” _Proceedings of the Institution of Mechanical Engineers, Part C: Journal of Mechanical Engineering Science_ , vol. 233, no. 14, pp. 5131–5143, 2019.
  * [27] J. Kuang, G. Xu, T. Tao, and Q. Wu, “Class-imbalance adversarial transfer learning network for cross-domain fault diagnosis with imbalanced data,” _IEEE Transactions on Instrumentation and Measurement_ , vol. 71, pp. 1–11, 2021.
  * [28] X. Liu, J. Chen, K. Zhang, S. Liu, S. He, and Z. Zhou, “Cross-domain intelligent bearing fault diagnosis under class imbalanced samples via transfer residual network augmented with explicit weight self-assignment strategy based on meta data,” _Knowledge-Based Systems_ , vol. 251, p. 109272, 2022.
  * [29] Z. Wu, H. Zhang, J. Guo, Y. Ji, and M. Pecht, “Imbalanced bearing fault diagnosis under variant working conditions using cost-sensitive deep domain adaptation network,” _Expert Systems with Applications_ , vol. 193, p. 116459, 2022.
  * [30] P. Khosla, P. Teterwak, C. Wang, A. Sarna, Y. Tian, P. Isola, A. Maschinot, C. Liu, and D. Krishnan, “Supervised contrastive learning,” _Advances in neural information processing systems_ , vol. 33, pp. 18 661–18 673, 2020\.
  * [31] P. Molchanov, S. Tyree, T. Karras, T. Aila, and J. Kautz, “Pruning convolutional neural networks for resource efficient inference,” _arXiv preprint arXiv:1611.06440_ , 2016.
  * [32] S. Han, J. Pool, J. Tran, and W. Dally, “Learning both weights and connections for efficient neural network,” _Advances in neural information processing systems_ , vol. 28, 2015.
  * [33] S. Han, H. Mao, and W. J. Dally, “Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding,” _arXiv preprint arXiv:1510.00149_ , 2015.
  * [34] M. Long, J. Wang, G. Ding, J. Sun, and P. S. Yu, “Transfer joint matching for unsupervised domain adaptation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2014, pp. 1410–1417.
  * [35] W. A. Smith and R. B. Randall, “Rolling element bearing diagnostics using the case western reserve university data: A benchmark study,” _Mechanical systems and signal processing_ , vol. 64, pp. 100–131, 2015.
  * [36] S. Shao, S. McAleer, R. Yan, and P. Baldi, “Highly accurate machine fault diagnosis using deep transfer learning,” _IEEE Transactions on Industrial Informatics_ , vol. 15, no. 4, pp. 2446–2455, 2018.
  * [37] C. Ruiz-Cárcel, Y. Cao, D. Mba, L. Lao, and R. Samuel, “Statistical process monitoring of a multiphase flow facility,” _Control Engineering Practice_ , vol. 42, pp. 74–88, 2015.