# Focus-Driven Contrastive Learning for Medical Question Summarization

Ming Zhang1,2 Shuai Dou3 Ziyang Wang1,2 Yunfang Wu1,3
1MOE, Key Laboratory of Computational Linguistics, Peking University
2School of Software and Microelectronics, Peking University
3School of Computer Science, Peking University
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>*Corresponding author.

###### Abstract

Automatic medical question summarization can significantly help the system to
understand consumer health questions and retrieve correct answers. The Seq2Seq
model based on maximum likelihood estimation (MLE) has been applied in this
task, which faces two general problems: the model can not capture well
question focus and and the traditional MLE strategy lacks the ability to
understand sentence-level semantics. To alleviate these problems, we propose a
novel question focus-driven contrastive learning framework (QFCL). Specially,
we propose an easy and effective approach to generate hard negative samples
based on the question focus, and exploit contrastive learning at both encoder
and decoder to obtain better sentence-level representations. On three medical
benchmark datasets, our proposed model achieves new state-of-the-art results,
and obtains a performance gain of 5.33, 12.85 and 3.81 points over the
baseline BART model on three datasets respectively. Further human judgement
and detailed analysis prove that our QFCL model learns better sentence
representations with the ability to distinguish different sentence meanings,
and generates high-quality summaries by capturing question focus.

##  1 Introduction

Input question: consumer health question (CHQ)
---
subject: gender dysphoria message: no health care on
my son suffering from gender dysphoria what can we
do to help him he worked out of high school no problems
now not working and about shutting himself in his room
24/7 theres nothing this condition in our area we live in
[location].no help in area what can we do he has had bad
thoughts already please help us with some sort of info
thank yuo [name] [location]
Golden summary: frequently asked question (FAQ):
Where can I find information on treatment and resources
for gender dysphoria?
Summary by BART (baseline):
What are the treatments for weight loss?
Summary by our model:
What are the treatments for gender dysphoria?

Table 1: An example of medical question summarization in MeqSum dataset, where
the question focus is highlighted in green. Summaries generated by BART and
our model are also listed.

A growing number of health questions are raised by consumers on websites
nowadays, which are usually written in natural language and including detailed
and peripheral information not related to the answers. Summaries of such
questions can greatly improve the performance in retrieving relevant answers
Ben Abacha and Demner-Fushman (2019). Accordingly, the medical question
summarization task is defined as summarizing the consumer health questions
(CHQ) into frequently asked questions (FAQ), which are shorter but remain
essential information of the original question to get correct answers. An
example of medical question summarization is shown in Table 1.

The Seq2Seq neural models have been widely used in abstractive summarization
Nallapati et al. (2016); Lewis et al. (2020); Zhang et al. (2020) and show
promising potentials, and they have also been applied in medical question
summarization and achieve current state-of-the-art results. Ben Abacha and
Demner-Fushman (2019) apply the pointer-generator model for this task. Yadav
et al. (2021a) present a reinforcement learning framework with question-type
identification reward and question-focus recognition reward. Mrini et al.
(2021b) propose a multitask learning method by treating recognizing question
entailment as an auxiliary task.

Figure 1: Sketch of our proposed contrastive learning framework.
$M_{s}$,$M_{h}$ represents the memory bank that contains simple negative
samples and hard negative samples respectively. $\mathcal{R}_{f}$,
$\mathcal{R}_{c}$, $\mathcal{R}_{g}$ denotes the sentence representation of
FAQ, CHQ and generated summary. $\mathcal{L}_{ctrS}$ and $\mathcal{L}_{ctrH}$
are contrastive learning loss on simple negative samples and hard negative
samples respectively. $+$ indicates the positive sample, and $-$ indicates the
negative sample.

.

In the medical question summarization task, the input question CHQ is always
lengthy and contains redundant information, where some salient medical
entities and the semantic focus of question are vital to understand users’
intention. But it still remains a challenging task for the existing methods to
capture the question focus. As described in the example 1, the focus "gender
dysphoria" is mis-replaced by "weight loss" in the summary generated by the
fine-tuned BART, resulting in a completely different meaning from the original
sentence.

For the medical question summarization task, the generated question summary is
required to semantically close to the reference question. However, in most of
current pre-trained models such as BART Lewis et al. (2020), the model adopts
maximum likelihood estimation (MLE) and mainly focuses on the accuracy of the
prediction of masked tokens, but does not guarantee to the semantic similarity
or dissimilarity of the whole sentences. To address this issue, some previous
works adopt reinforcement learning (RL) in text summarization task Li et al.
(2019); Paulus et al. (2018), but RL suffers from the noise gradient
estimation problem Greensmith et al. (2004), which makes the training process
unstable and sensitive to hyper-parameters.

To alleviate these problems, we propose a novel question focus-driven
contrastive learning (QFCL) framework for medical question summarization, as
illustrated in Figure 1. In our model, we introduce a "double anchors"
strategy for contrastive learning, by utilizing the sentence representation of
CHQ as an anchor and the generated summary as another anchor, and regarding
the golden reference FAQ as the positive sample. In addition, we present a
"focus-driven hard negatives generator" to construct hard negative samples, by
replacing the focus phrases with other phrases sharing the same attribute.

Through contrastive learning, we minimize the distance between CHQ/generated
summary and golden reference, and maximize the distance between CHQ/generated
summary and other negative samples. By using the double anchors, our model is
able to extract sentence-level semantic features to alleviate the problem of
MLE. With the help of hard negatives generator, the model learns to pay more
attention to question focus and thus produces high quality summary.

We conduct extensive experiments on three medical question summarization
datasets: Meqsum Ben Abacha and Demner-Fushman (2019), HealthCareMagic and
iCliniq Zeng et al. (2020). Our proposed model outperforms previous best
results by a wide margin, achieving new state-of-the-art results on all three
datasets. Compared with the baseline BART, our model brings a relative
performance gain of $12.2\%$, $28.7\%$ and $9.6\%$ on Meqsum, Cliniq and
HealthcareMagic respectively. Through analysis, we prove that our model
significantly gains the power of distinguishing the semantics between
generated summaries and negative samples, and our model generates high-quality
summaries capturing more question focuses. Our data and codes are publicly
available for future research 111https://github.com/zhangming880102/QFCL.

Figure 2: The overall framework of QFCL. $\mathcal{L}_{ctrC}$ and
$\mathcal{L}_{ctrG}$ are contrastive learning loss on the two anchors
respectively.

##  2 Ralated Work

###  2.1 Medical Question Summarization

The medical question summarization task is defined by Ben Abacha and Demner-
Fushman (2019). They construct a benchmark dataset Meqsum, and apply a
pointer-generator model to generate question summary. At the question
summarization campaign of MEDIQA-21 organized by Ben Abacha et al. (2021),
almost all approaches rely on the fine-tuning of pre-trained transformer
models. Transfer learning, knowledge-base, and ensemble methods are widely
utilized by participanting teams to achieve better performance He et al.
(2021); Yadav et al. (2021b); Mrini et al. (2021c); Sänger et al. (2021). In
this paper, we also base our method on the strong pre-trained BART model.

Recently, Yadav et al. (2021a) propose a RL framework with two question-aware
semantic rewards: question-type identification reward (QTR) and question-focus
recognition reward (QFR). QTR is to identify whether the question types are
consistent with the gold question, and QFR is designed to capture question
focus. But in their work, the question types and question focuses in the
dataset should be manually labeled, which is both time-consuming and labor-
intensive for large-scale datasets such as HealthcareMagic and iCliniq.
Moreover, the RL training process is unstable. Mrini et al. (2021b) claim an
equivalence between medical question summary and recognizing question
entailment(RQE), and employ multi-task learning to train the model to not only
perform next-word-prediction but also carry question entailment recognition.
These two studies demonstrate that the pre-trained models achieve better
performance after capturing the underlying sentence semantics of generated
questions. Different from these works, we exploit contrastive learning to
obtain focus-aware question representations.

###  2.2 Contrastive Learning

Different from the traditional methods which learn representations in pixel-
level for computer vision tasks, contrastive learning encodes high-level
features to distinguish different objects and has achieved great success
Henaff (2020); Chen et al. (2020); Misra and van der Maaten (2020); He et al.
(2020), and it has also been applied in several NLP tasks such as machine
translation Pan et al. (2021), pre-training Chi et al. (2021) and question
answering Yang et al. (2021). In the field of summarization, Liu and Liu
(2021) present a contrastive framework to bridge the gap between the learning
objective and evaluation metrics, Cao and Wang (2021) design several negative
sample construction strategies to solve the factual inconsistency problem. In
contrast, we use the MoCo structure to handle with the large volume of
negative samples, and propose a new negative sample construction method.

Chen et al. (2020) prove that large size of negative samples can improve the
performance of contrastive learning, but it also brings heavy burden on
computation cost. To address this issue, He et al. (2020) propose MoCo, which
maintains a queue as the memory bank to store negative samples. MoCo adopts
two encoders with the same structure: key encoder and query encoder, where the
key encoder is momentum updated from the query encoder.

##  3 Model

Given an input question CHQ, which is written by consumers and contains
lengthy and complex information, the medical question summarization task aims
to automatically generate a question summary that is a frequently asked
question (FAQ), capturing the essential information to help efficiently
retrieve correct answers. A more detailed structure of our proposed QFCL model
is presented in Figure 2.

###  3.1 Contrastive Learning Architecture

We employ the pre-trained BART Lewis et al. (2020) as our basic model to
generate question summaries. For contrastive learning, we adopt the MoCo
architecure He et al. (2020), which contains a key encoder $E_{k}$ with the
same structure as the BART encoder $E_{q}$, and a queue to store simple
negative samples with large volume. The simple negative samples in the queue
are progressively replaced by current mini-batch of representations extracted
from the key encoder. All samples in the queue will be used as negative
samples in the next batch. In addition, QFCL employs a hard negatives
generator to generate hard negative samples.

In our model, the BART encoder $E_{q}$ and the decoder are updated via back
propagation by combining three types of loss functions, as described in the
subsequent sections. The parameters of $E_{k}$ are frozen and updated slowly
towards that of $E_{q}$:

$\theta_{k}\leftarrow m\theta_{k}+(1-m)\theta_{q}$ (1)

where $m$ is a momentum coefficient.

At the inference, only the BART encoder and decoder are retained, other parts
such as the key encoder, the queue, and the hard negatives generator are all
discarded.

###  3.2 Simple Negative Samples

In the medical question summarization task, the input question CHQ should be
semantically close to its reference summary FAQ but different from other
question summaries. Therefore, we regard the CHQ $c_{i}$ in the $i$-th pair as
the anchor, FAQ $f_{i}$ in the same pair as the positive sample and randomly
select $f_{j}$ from other different pairs to serve as simple negative samples.

Let $\mathcal{R}_{s}$ denote the average decoded output of an arbitrary
sentence $s$, the objective function of the simple contrastive learning is
defined as:

$\mathcal{L}_{ctrCS}=-log{\frac{e^{{sim(\mathcal{R}_{ci},\mathcal{R}_{fi})}/{\tau}}}{\sum_{\mathcal{R}_{fj}\in
M_{s}}{e^{{sim(\mathcal{R}_{ci},\mathcal{R}_{fj})}/{\tau}}}}}$ (2)

where $\mathcal{R}_{ci}$ indicates the sentence representation of the $i$-th
CHQ extracted from $E_{q}$, and $\mathcal{R}_{fi}$ and $\mathcal{R}_{fj}$ are
extracted from the key encoder $E_{k}$ for the $i$-th and $j$-th FAQ
respectively. The operation $sim$ is to calculate the cosine similarity,
$\tau$ is a temperature hyper-parameter. $M_{s}$ is the memory bank which
contains one positive sample and $K$ simple negative samples in the queue with
respect to an anchor.

###  3.3 Focus-Driven Hard Negative Samples

Figure 3: The method of hard negative samples generation.

The above simple negative samples are randomly selected. As claimed by
Kalantidis et al. (2020), hard negative samples that are more similar to
positive samples can facilitate the model to get better performance. Inspired
by this, we build a bridge between hard sample generation and question focus
prediction.

####  3.3.1 Question Focus Identification

As mentioned before, the question focus is essential to understand a consumer
health question. If some focus phrases are missing in the generated summary,
the semantic will drift far away from the original user’s intention. So we
construct difficult negative samples based on the question focus to enhance
contrastive learning. Specially, we replace the focus phrases with some other
phrases of the same attribution, and keep other words of the sentence
unchanged. An example of hard negative sample generation is shown in Figure 3.

One issue for our method is how to automatically annotate question focus.
Yadav et al. (2021a) manually labeled the question focus in MeqSum dataset.
However, this is quite time-consuming and labor-intensive, driving us to find
a method which can automatically mark the question focus in larger datasets,
such as HealthcareMagic and iCliniq. We analyzed the manually labeled MeqSum
dataset, and found that in 340 of the total 500 records (up to $68\%$), the
question focuses are the overlap phrases between CHQ and FAQ. Accordingly, we
hypothesize that the same phrases appearing both in the source question and
the golden summary have a high probability to be key-phrases. This idea is
also proved to be effective in Li et al. (2020).

Since the question focus is usually a phrase rather than a single word, we
need to split one sentence into phrases. We apply the chunker Akbik et al.
(2018) to the CHQ and FAQ text, and record the chunk label of each phrase.
Then the consistent phrases appearing both in CHQ and FAQ are labeled as the
question focuses.

####  3.3.2 Hard Negative Sample Generation

We constructed a dictionary by concatenating all phrases of the FAQ sentences
in the train set. To generate hard negative samples, the question focuses are
randomly replaced by other phrases of the same chunk label from the
dictionary. As shown in Figure 2, “breast cancer” is replaced by “diabetes”
since they share the same label “NP”. We repeat this process $N_{h}$ times to
construct $N_{h}$ different hard negative samples for each CHQ-FAQ pair.

####  3.3.3 Contrastive Learning on Hard Negative Samples

The sentence representation of hard sample $\mathcal{R}_{h}$ is extracted from
the key encoder $E_{k}$. We define the hard loss function of contrastive
learning as:

$\mathcal{L}_{ctrCH}=-log{\frac{e^{{sim(\mathcal{R}_{ci},\mathcal{R}_{fi})}/{\tau}}}{\sum_{\mathcal{R}_{h}\in
M_{h}}{e^{{sim(\mathcal{R}_{ci},\mathcal{R}_{h})}/{\tau}}}}}$ (3)

where $M_{h}$ denotes the memory bank containing one positive sample and
$N_{h}$ hard negative samples.

This loss function forces the model to not only shorten the distance between
CHQ and FAQ, but also expand the gap between the CHQ and hard negative
samples. In this way, we achieve the goal of making the model pay more
attention to the question focus, and obtain a focus-aware representation.

###  3.4 Contrastive Learning at Decoder

An imbalance existing in the above method is that contrastive learning is only
utilized at the encoder. We fine-tuned BART on iCliniq dataset, and found that
the decoder lacks the ability to distinguish the representations between the
generated summary and the positive samples/unrelated negative samples, as
$s_{g\\_faq}^{+}$, $s_{g\\_sim}^{-}$, $s_{g\\_hard}^{-}$ shown in Figure 4.
Therefore, we try to improve the similarity between the generated summary and
its reference FAQ, and at the same time enlarge the dis-similarity between the
generated summary and other unrelated questions.

Specially, we regard the generated summary as an extra anchor, and denote the
representation of the generated summary as $g_{i}$. Since the output summary
should be semantically consistent with the corresponding FAQ, we consider the
representation of the FAQ $f_{i}$ in the same pair as the positive sample, and
select the simple negative samples randomly from the queue and generate hard
negative samples using the hard negatives generator. The object functions of
contrast loss $\mathcal{L}_{ctrGS}$ and $\mathcal{L}_{ctrGH}$ at the decoder
end are defined in a similar style as Equation 2 and 3, except that the anchor
$c_{i}$ is replaced by another anchor $g_{i}$.

###  3.5 Overall Objective Function

For predicting next tokens in the generated summary, we use the cross entropy
loss $\mathcal{L}_{ce}$:

$\mathcal{L}_{ce}=-\frac{1}{\left|T\right|}\sum_{t\in
T}{log(p(y_{t}|x,y_{1:t-1},\theta))}$ (4)

In our model, the overall loss function consists of five parts: the cross
entropy loss $\mathcal{L}_{ce}$ and four different loss functions of
contrastive learning: $\mathcal{L}_{ctrCS}$, $\mathcal{L}_{ctrCH}$ for the
anchor at the encoder end, $\mathcal{L}_{ctrGS}$, $\mathcal{L}_{ctrGH}$ for
the anchor at the decoder end. We define the contrastive learning loss with
respect to these two anchors as:

$\begin{split}\mathcal{L}_{ctrC}&=\alpha\mathcal{L}_{ctrCS}+\beta\mathcal{L}_{ctrCH}\\\
\mathcal{L}_{ctrG}&=\alpha\mathcal{L}_{ctrGS}+\beta\mathcal{L}_{ctrGH}\\\
\end{split}$ (5)

where $\alpha$, $\beta$ are hyper-parameters to control the balance between
simple negatives and hard ones. The weights of contrastive learning loss at
the encoder and decoder are considered as equal, and the overall loss is
defined as:

$\mathcal{L}=\mathcal{L}_{ce}+\frac{1}{2}\mathcal{L}_{ctrC}+\frac{1}{2}\mathcal{L}_{ctrG}$
(6)

##  4 Experiments

###  4.1 Datasets

We conduct experiments on three English benchmark medical question
summarization datasets, including Meqsum, HealthcareMagic and iCliniq. Meqsum
is a high-quality dataset from NIH 222www.nlm.nih.gov/medlineplus, constructed
by Ben Abacha and Demner-Fushman (2019). Mrini et al. (2021a) extracted
HealthCareMagic and iCliniq datasets from MedDialog Zeng et al. (2020) , which
are collected automatically from the online healthcare service platforms
333www.healthcaremagic.com 444www.icliniq.com. MeqSum’s and HealthcareMagic’s
summaries are written by medical experts in formal style, while iCliniq’s are
patient-written. We list some statistics of these datasets in table 2.
Following previous works, we adopt ROUGE Lin
(2004)555https://pypi.org/project/py-rouge  as the evaluation metric. Our data
and codes are available at https:// github.com/zhangming880102/QFCL.

Dataset | Train | Dev | Test | Length
---|---|---|---|---
MeqSum | 400 | 100 | 500 | 60.8/10.1
HealthCareMagic | 181,122 | 22,641 | 22,642 | 82.8/9.7
iCliniq | 24,851 | 3,105 | 3,106 | 89.7/12.3

Table 2: Statistics of three medical question summarization datasets. Length
indicates the average length of CHQ/FAQ.

Model | MeqSum | iCliniq | HealthCareMagic
---|---|---|---
R1 | R2 | RL | R1 | R2 | RL | R1 | R2 | RL
ProphetNet + QTR + QFRYadav et al. (2021a) | 45.52 | 27.54 | 48.19 | - | - | - | - | - | -
MTL+Data augmentationMrini et al. (2021b) | 49.20 | 29.50 | 44.80 | 54.20 | 36.90 | 49.10 | 45.90 | 24.30 | 42.90
BART Lewis et al. (2020) | 46.17 | 28.05 | 43.75 | 48.79 | 25.47 | 44.69 | 42.33 | 23.07 | 39.60
BART + S | 49.30 | 31.78 | 46.89 | 56.58 | 36.43 | 52.06 | 44.35 | 24.73 | 41.46
BART + S + H | 49.96 | 32.72 | 47.66 | 58.26 | 40.08 | 55.34 | 45.52 | 25.71 | 42.51
BART + S + H + D (QFCL) | 51.48 | 34.16 | 49.08 | 60.09 | 43.22 | 57.54 | 46.42 | 26.47 | 43.41

Table 3: Experimental results on three medical question summarization
datasets. $S$ denotes the contrastive learning on simple negative samples at
the encoder end; $H$ denotes the contrastive learning on hard negative samples
at the encoder end; $D$ denotes the decoder end’s contrastive learning. The
top group lists the existing state-of-the-art results on three datasets, and
the bottom group shows our ablation study on different components.

###  4.2 Training Details

We utilize BART-large Lewis et al. (2020) in
huggingface666huggingface.co/facebook/bart-large as our pre-trained model. The
learning rate of BART baseline is set to 3e-5 as the same with Mrini et al.
(2021b). For contrastive learning in QFCL, the learning rate is optimized to
1e-5. Betas of Adam optimizer is set to 0.9 and 0.999. Batch size is set to
16. The number of hard negative samples $n_{h}$ is set to 64. For Moco, the
queue size $K$ is set to 4096, temperature $\tau$ is 0.07, and the momentum
coefficient $m$ is 0.999. In Equation 5, $\alpha$ and $\beta$ are set to 1 and
0.5 respectively through grid search on MeqSum development set. Experiments
were all performed on a single NVIDIA RTX 3090 GPU. The average runtimes of
each epoch for MeqSum, iCliniq and HealthcareMagic are 4.2h, 0.6h and 0.1h
respectively.

###  4.3 Overall Performance

We report our experimental results in Table 3. Our model achieves new state-
of-the-art results on all three datasets. Compared with the previous best
results, we obtain an improvement of 0.99 ROUGE-L score on MeqSum, 8.44 on
iCliniq, and 0.51 on HealthcareMagic, respectively.

MTL+Data augmentation Mrini et al. (2021b) obtains the previous state-of-the-
art results on iCliniq and HealthcareMagic, which utilizes the question
entailment data to augment summarization data. In contrast, our method doesn’t
need other classification models or external data. The work of
ProphetNet+QTR+QFR Yadav et al. (2021a) gets the previous best result on
MeqSum, which presents a reinforcement learning-based framework with question-
aware rewards. Comparing with this competitive model, our method obtains
consistent better performance on all metrics, with 2.28 improvement on R1,
4.66 improvement on R2 and 0.89 improvement on RL. We did not compare the
results of Yadav et al. (2021a) on the other two datasets, since their method
requires manually labeled question focuses and question types.

###  4.4 Ablation Study

We perform ablation study to evaluate the impacts of different components
employed in QFCL, and report the results in Table 3. In particular, for Meqsum
dataset, due to the small size which may cause the training unstable, we
conducted five separate experiments and computed the average ROUGE score of
these five checkpoints as the final result. Compared with the base BART model,
we obtain an absolute improvement of 5.33 points on average. T-test is
implemented on such five ROUGE scores and the p-value is less than 1e-2,
validating that this improvement is significant. On Cliniq the absolute
improvement is 12.85 points and on HealthcareMagic 3.81 points. In comparison
to BART, the relative improvements of our model are $12.2\%$, $28.7\%$ and
$9.6\%$ on Meqsum, Cliniq and HealthcareMagic respectively.

The results demonstrate that each component of our model is helpful. On
MeqSum, there is an increase of 3.15 points for BART+S compared to the
baseline, indicating that the contrastive learning on simple negative samples
largely improves model performance. It shows an continuous increase of 0.77
points for BART+S+H, and the highest ROUGE-L score is obtained when three
parts are all implemented in our model. It suggests that each component in
QFCL contributes positively, and metrics like ROUGE evaluating the similarity
between whole sentences benefit from our contrastive learning strategy.

###  4.5 Human Evaluation

To quantitatively assess the results, we compare our method with the baseline
BART through human judgement. We randomly selected 50 samples from each of
three datasets, and hired 3 graduate students to categorize each generated
summary into one of the following categories: ’Incorrect’, ’Acceptable’, and
’Perfect’. We compute the average number of each category, and report the
result in Table 4. The average Spearman correlation coefficient between three
annotators is 0.68, which guarantees a high quality of our annotation data.
The evaluation results show that our model generates a higher proportion of
perfect samples and a lower proportion of incorrect ones, by enhancing the
model’s ability of capturing sentence semantics and question focuses.

Model | MeqSum | iCliniq | HealthCareMagic
---|---|---|---
I | A | P | I | A | P | I | A | P
BART | 28.7 | 17.3 | 4.0 | 12.3 | 17.0 | 20.7 | 20.7 | 20.3 | 9.0
QFCL | 12.0 | 18.0 | 20.0 | 6.3 | 17.7 | 26.0 | 5.7 | 16.3 | 28.0
Table 4: Human evaluation of the summaries generated by BART and QFCL
respectively. The metric I means the number of incorrect samples, A means
acceptable, P means perfect.

###  4.6 Case Study

To clearly show the output question summary, we list two samples to compare
our model with BART in Table 5. In Case 1, BART captures the question focus
"Ampicillin" but misses "drink alcohol", and in Case 2 it misses the question
focus "breast milk". In contrast, our model successfully extract multiple
question focuses from the lengthy CHQ, and generate summaries which more
conform to the meaning of original questions.

Case1
---
CHQ | MESSAGE: Is it okay to drink alcohol in
moderation when taking Ampicillin. I was
told it negates any medical effect of the drug
FAQ | Can I drink alcohol while taking Amoxicillin?
BART | What are the side effects of Ampicillin?
QFCL | Is it okay to drink alcohol with Ampicillin?
Case2
CHQ | Hi….. I have 3 month old baby girl…… I don t
have breast milk from the beginning due to
some reason. I can not give formula milk to
baby…… So right now i m giving buffelo milk
…….. What else i should give her for better
nourishment????? ……. She has constipation
problem may be due to milk but i cant give her
breastmilk or formula ……. How to overcome
it?????……… Please help me
FAQ | Suggest ways to feed newborn other than
breast milk
BART | Suggest treatment for constipation in a child
QFCL | Suggest better nourishment for baby other
than breast milk

Table 5: Examples of generated question summaries by BART and our QFCL model.
The question focuses are highlighted. Figure 4: Correlation between sentence
representation similarities and epoch numbers on dev set. The red lines are
about the anchor CHQ. $s_{c\\_faq}^{+}$ is the average cosine similarity
between CHQ and related FAQ, $s_{c\\_sim}^{-}$ is between CHQ and simple
negative samples (other FAQs), $s_{c\\_hard}^{-}$ is between CHQ and hard
negative samples. The green lines are about the anchor of generated summary.
$s_{g\\_faq}^{+}$ is the average cosine similarity between the generated
summary and FAQ, $s_{g\\_sim}^{-}$ is between generated summary and simple
negatives, $s_{g\\_hard}^{-}$ is between generated summary and hard negatives.
The epoch number equaling 0 denotes the initial pre-trained model.

##  5 Model Analysis

###  5.1 Correlation of Sentence Representations

Since the auxiliary structures are discarded at the inference stage, we make
further analysis to check that whether the retained model has the ability to
distinguish different sentence-level semantics when facing unknown data. We
train QFCL and BART on the training set for 20 epochs and save each
checkpoint, and evaluate these checkpoints on the development set.

Four types of sentence representations are extracted from these checkpoints:
CHQ’s representation $\mathcal{R}_{c}$, FAQ’s representation
$\mathcal{R}_{f}$, hard negatives’ representation $\mathcal{R}_{h}$, and the
generated summary’s representation at decoder end $\mathcal{R}_{g}$. Then we
calculate the cosine similarity between them, and draw the relationship
between these similarity scores and the epoch numbers, as shown in Figure 4.

Regarding the anchor CHQ in the curve of iCliniq, $s_{c\\_faq}^{+}$,
$s_{c\\_sim}^{-}$ and $s_{c\\_hard}^{-}$ are very close to each other at epoch
0, suggesting that the initial encoder lacks the ability to capture different
semantics. With the increase of training steps, $s_{c\\_faq}^{+}$ changes
smoothly, while $s_{c\\_sim}^{-}$ decreases sharply to near zero and
$s_{c\\_hard}^{-}$ decreases gradually and converges at a middle level between
$s_{c\\_faq}^{+}$ and $s_{c\\_sim}^{-}$. This suggests that, powered by
contrastive learning, our model has learned to distinguish sentences of
different meanings at the encoder end.

With the generated summary as another anchor, we find out that
$s_{g\\_faq}^{+}$, $s_{g\\_sim}^{-}$, $s_{g\\_hard}^{-}$ are all near to 0
initially, which depict that the decoder is also weak in representing
sentence-level semantics. After training, $s_{g\\_faq}^{+}$ increases
significantly, $s_{g\\_hard}^{-}$ converges between $s_{g\\_faq}^{+}$ and
$s_{g\\_sim}^{-}$, and $s_{g\\_sim}^{-}$ keeps very low all the time. It
suggests that the decoder has strengthened its power to distinguish different
semantics as the same to the encoder end.

Another chart is drawn to show this relationship for BART baseline in Figure
4. The similarities between the anchor and the positive samples, negative
samples are very close, and never improve significantly with the progress of
training. This situation suggesting that the BART baseline has a relatively
weaker performance to distinguish the sentences of different meanings at both
encoder and decode, since it only focuses on the prediction of next tokens.

We also draw this correlation curve on MeqSum and HealthcareMagic. The curve
of HealthcareMagic is similar to iCliniq. On MeqSum, our model can still
distinguish sentences with different semantics better than the baseline, but
the signal is not as significant as iCliniq or HealthcareMagic due to the
limited size of training set.

Model | C1 | C2 | C3 | C4 | C5 | Mean
---|---|---|---|---|---|---
BART | 33.37 | 40.76 | 39.78 | 35.34 | 36.21 | 37.09
QFCL | 47.41 | 42.24 | 45.20 | 45.20 | 47.17 | 45.44

Table 6: Accuracy of question focuses in generated summaries. C1-C5 means 5
different checkpoints trained by each model.

###  5.2 Capturing Question Focus

To study whether our model pays more attention to the question focus, we
evaluate the accuracy of question focuses in generated summaries. We use the
sequence labeling model trained by Yadav et al. (2021a) to predict question
focuses on the MeqSum dataset, and regard the 812 predicted question focuses
in test set as the gold-standard. For QFCL and BART, we train five checkpoints
and generate summaries on these checkpoints, and compute the accuracy of
question focuses on test set. As shown in Table 6, the average accuracy is
37.09% for BART and 45.44% for QFCL. Our model exceeds the baseline by 8.35
points for question focus generation. P-value of t-test on these two sets of
results is 1.04e-3, indicating that this improvement is statistically
significant.

##  6 Conclusion

In this paper, we introduce a novel question focus-based contrastive learning
framework QFCL for medical question summarization. In the proposed model, we
adopt a "double anchor" strategy, by considering both the input question CHQ
and the generated summary as comparing anchors. And we exploit a "hard
negatives generator" to generate hard negative samples based on the question
focus. Our model significantly improves the performance on three medical
question summarization datasets, and achieves new state-of-the-art results. In
the future, we would like to find a more effective way to do question focus
recognition.

## Acknowledgement

This work is supported by the National Hi-Tech RD Program of China
(No.2020AAA0106600), the National Natural Science Foundation of China
(62076008) and the Key Project of Natural Science Foundation of China
(61936012).

##  Appendix A Ethical Consideration

The datasets used in our work are all publicly available. We used BART as our
basic model which follows the apache-2.0 license. The datasets and the method
should only be used for research purposes, not in the commercial field.

The personal information in the datasets has been hidden through
preprocessing. For example, the name of the consumer was converted to an
placeholder [name] and the address was converted to [location], as shown in
Table 1.

As the current models could not guarantee to generate summaries fully conforms
to the intention of the consumers, the method in our paper can only be used as
an auxiliary tool to avoid further misleading suggestions.

## References

  * Akbik et al. (2018) Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018.  Contextual string embeddings for sequence labeling.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 1638–1649, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  * Ben Abacha and Demner-Fushman (2019) Asma Ben Abacha and Dina Demner-Fushman. 2019.  On the summarization of consumer health questions.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2228–2234, Florence, Italy. Association for Computational Linguistics.
  * Ben Abacha et al. (2021) Asma Ben Abacha, Yassine Mrabet, Yuhao Zhang, Chaitanya Shivade, Curtis Langlotz, and Dina Demner-Fushman. 2021.  Overview of the MEDIQA 2021 shared task on summarization in the medical domain.  In _Proceedings of the 20th Workshop on Biomedical Language Processing_ , pages 74–85, Online. Association for Computational Linguistics.
  * Cao and Wang (2021) Shuyang Cao and Lu Wang. 2021.  CLIFF: Contrastive learning for improving faithfulness and factuality in abstractive summarization.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6633–6649, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Chen et al. (2020) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , pages 1597–1607. PMLR.
  * Chi et al. (2021) Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, and Ming Zhou. 2021.  InfoXLM: An information-theoretic framework for cross-lingual language model pre-training.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3576–3588, Online. Association for Computational Linguistics.
  * Greensmith et al. (2004) Evan Greensmith, Peter L Bartlett, and Jonathan Baxter. 2004.  Variance reduction techniques for gradient estimates in reinforcement learning.  _Journal of Machine Learning Research_ , 5(9).
  * He et al. (2020) Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020.  Momentum contrast for unsupervised visual representation learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9729–9738.
  * He et al. (2021) Yifan He, Mosha Chen, and Songfang Huang. 2021.  damo_nlp at MEDIQA 2021: Knowledge-based preprocessing and coverage-oriented reranking for medical question summarization.  In _Proceedings of the 20th Workshop on Biomedical Language Processing_ , pages 112–118, Online. Association for Computational Linguistics.
  * Henaff (2020) Olivier Henaff. 2020.  Data-efficient image recognition with contrastive predictive coding.  In _International Conference on Machine Learning_ , pages 4182–4192. PMLR.
  * Kalantidis et al. (2020) Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. 2020.  Hard negative mixing for contrastive learning.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 21798–21809. Curran Associates, Inc.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Li et al. (2020) Haoran Li, Junnan Zhu, Jiajun Zhang, Chengqing Zong, and Xiaodong He. 2020.  Keywords-guided abstractive sentence summarization.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 8196–8203.
  * Li et al. (2019) Siyao Li, Deren Lei, Pengda Qin, and William Yang Wang. 2019.  Deep reinforcement learning with distributional semantic rewards for abstractive summarization.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 6038–6044, Hong Kong, China. Association for Computational Linguistics.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Liu and Liu (2021) Yixin Liu and Pengfei Liu. 2021.  SimCLS: A simple framework for contrastive learning of abstractive summarization.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 1065–1072, Online. Association for Computational Linguistics.
  * Misra and van der Maaten (2020) Ishan Misra and Laurens van der Maaten. 2020.  Self-supervised learning of pretext-invariant representations.  In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 6706–6716.
  * Mrini et al. (2021a) Khalil Mrini, Franck Dernoncourt, Walter Chang, Emilia Farcas, and Ndapa Nakashole. 2021a.  Joint summarization-entailment optimization for consumer health question understanding.  In _Proceedings of the Second Workshop on Natural Language Processing for Medical Conversations_ , pages 58–65, Online. Association for Computational Linguistics.
  * Mrini et al. (2021b) Khalil Mrini, Franck Dernoncourt, Seunghyun Yoon, Trung Bui, Walter Chang, Emilia Farcas, and Ndapa Nakashole. 2021b.  A gradually soft multi-task and data-augmented approach to medical question understanding.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1505–1515, Online. Association for Computational Linguistics.
  * Mrini et al. (2021c) Khalil Mrini, Franck Dernoncourt, Seunghyun Yoon, Trung Bui, Walter Chang, Emilia Farcas, and Ndapa Nakashole. 2021c.  UCSD-adobe at MEDIQA 2021: Transfer learning and answer sentence selection for medical summarization.  In _Proceedings of the 20th Workshop on Biomedical Language Processing_ , pages 257–262, Online. Association for Computational Linguistics.
  * Nallapati et al. (2016) Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çağlar Gulçehre, and Bing Xiang. 2016.  Abstractive text summarization using sequence-to-sequence RNNs and beyond.  In _Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning_ , pages 280–290, Berlin, Germany. Association for Computational Linguistics.
  * Pan et al. (2021) Xiao Pan, Mingxuan Wang, Liwei Wu, and Lei Li. 2021.  Contrastive learning for many-to-many multilingual neural machine translation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 244–258, Online. Association for Computational Linguistics.
  * Paulus et al. (2018) Romain Paulus, Caiming Xiong, and Richard Socher. 2018.  A deep reinforced model for abstractive summarization.  In _International Conference on Learning Representations_.
  * Sänger et al. (2021) Mario Sänger, Leon Weber, and Ulf Leser. 2021.  WBI at MEDIQA 2021: Summarizing consumer health questions with generative transformers.  In _Proceedings of the 20th Workshop on Biomedical Language Processing_ , pages 86–95, Online. Association for Computational Linguistics.
  * Yadav et al. (2021a) Shweta Yadav, Deepak Gupta, Asma Ben Abacha, and Dina Demner-Fushman. 2021a.  Reinforcement learning for abstractive question summarization with question-aware semantic rewards.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 249–255, Online. Association for Computational Linguistics.
  * Yadav et al. (2021b) Shweta Yadav, Mourad Sarrouti, and Deepak Gupta. 2021b.  NLM at MEDIQA 2021: Transfer learning-based approaches for consumer question and multi-answer summarization.  In _Proceedings of the 20th Workshop on Biomedical Language Processing_ , pages 291–301, Online. Association for Computational Linguistics.
  * Yang et al. (2021) Nan Yang, Furu Wei, Binxing Jiao, Daxing Jiang, and Linjun Yang. 2021.  xMoCo: Cross momentum contrastive learning for open-domain question answering.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 6120–6129, Online. Association for Computational Linguistics.
  * Zeng et al. (2020) Guangtao Zeng, Wenmian Yang, Zeqian Ju, Yue Yang, Sicheng Wang, Ruisi Zhang, Meng Zhou, Jiaqi Zeng, Xiangyu Dong, Ruoyu Zhang, Hongchao Fang, Penghui Zhu, Shu Chen, and Pengtao Xie. 2020.  MedDialog: Large-scale medical dialogue datasets.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9241–9250, Online. Association for Computational Linguistics.
  * Zhang et al. (2020) Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020.  Pegasus: Pre-training with extracted gap-sentences for abstractive summarization.  In _International Conference on Machine Learning_ , pages 11328–11339. PMLR.


* Schröder & Cuntz (2005) Schröder K. P., Cuntz M., 2005, ApJL, 630, L73
  * Scicluna et al. (2015) Scicluna P., Siebenmorgen R., Wesson R., Blommaert J. A. D. L., Kasper M., Voshchinnikov N. V., Wolf S., 2015, A&A, 584, L10
  * Scicluna et al. (2020) Scicluna P., et al., 2020, arXiv e-prints, p. arXiv:2002.03100
  * Seifried et al. (2019) Seifried D., Walch S., Reissl S., Ibáñez-Mejía J. C., 2019, MNRAS, 482, 2697
  * Seifried et al. (2020) Seifried D., Walch S., Weis M., Reissl S., Soler J. D., Klessen R. S., Joshi P. R., 2020, MNRAS, 497, 4196
  * Soker (2002) Soker N., 2002, MNRAS, 336, 826
  * Tahani (2022) Tahani M., 2022, Frontiers in Astronomy and Space Sciences, 9, 940027
  * Thirumalai & Heyl (2013) Thirumalai A., Heyl J. S., 2013, MNRAS, 430, 1359
  * Tielens et al. (2005) Tielens A. G. G. M., Waters L. B. F. M., Bernatowicz T. J., 2005, in Krot A. N., Scott E. R. D., Reipurth B., eds, Astronomical Society of the Pacific Conference Series Vol. 341, Chondrites and the Protoplanetary Disk. p. 605
  * Tram & Hoang (2022) Tram L. N., Hoang T., 2022, Frontiers in Astronomy and Space Sciences, 9, 923927
  * Tram et al. (2020) Tram L. N., Hoang T., Soam A., Lesaffre P., Reach W. T., 2020, ApJ, 893, 138
  * Tram et al. (2021a) Tram L. N., Hoang T., Lee H., Santos F. P., Soam A., Lesaffre P., Gusdorf A., Reach W. T., 2021a, ApJ, 906, 115
  * Tram et al. (2021b) Tram L. N., et al., 2021b, ApJ, 923, 130
  * Truong et al. (2022) Truong B., et al., 2022, ApJ, 936, 101
  * Vassiliadis & Wood (1993) Vassiliadis E., Wood P. R., 1993, ApJ, 413, 641
  * Verhoelst et al. (2009) Verhoelst T., van der Zypen N., Hony S., Decin L., Cami J., Eriksson K., 2009, A&A, 498, 127
  * Vlemmings (2019) Vlemmings W., 2019, IAU Symposium, 343, 19
  * Vlemmings et al. (2006) Vlemmings W. H. T., Diamond P. J., Imai H., 2006, Nature, 440, 58
  * Vlemmings et al. (2012) Vlemmings W. H. T., Ramstedt S., Rao R., Maercker M., 2012, A&A, 540, L3
  * Vlemmings et al. (2017) Vlemmings W. H. T., et al., 2017, A&A, 603, A92
  * Wing & Lockwood (1973) Wing R. F., Lockwood G. W., 1973, ApJ, 184, 873
  * Yang et al. (2016) Yang H., Li Z.-Y., Looney L., Stephens I., 2016, MNRAS, 456, 2794
  * de Jager et al. (1988) de Jager C., Nieuwenhuijzen H., van der Hucht K. A., 1988, A&AS, 72, 259
  * van Loon et al. (2005) van Loon J. T., Cioni M. R. L., Zijlstra A. A., Loup C., 2005, A&A, 438, 273


11institutetext: Tsinghua University 22institutetext: Southern University of
Science and Technology 33institutetext: Sinosmart Inc11footnotetext: These
authors contributed equally to this work.

# Bootstrap The Original Latent: Learning a Private Model from a Black-box
Model

Shuai Wang∗ 11 Daoan Zhang∗ 22 Jianguo Zhang 22 Weiwei Zhang 33 Rui Li 11

###### Abstract

In this paper, considering the balance of data/model privacy of model owners
and user needs, we propose a new setting called Back-Propagated Black-Box
Adaptation (BPBA) for users to better train their private models via the
guidance of the back-propagated results of a Black-box foundation/source
model. Our setting can ease the usage of the foundation/source models as well
as prevent the leakage and misuse of foundation/source models. These
phenomenons are more severe and common in medical image analysis. To better
deal with the problems, we propose a new paradigm called Bootstrap The
Original Latent (BTOL) to fully utilize the foundation/source models. Our
strategy consists of a trainable adapter and a freeze-and-thaw strategy. We
apply our BTOL under BPBA and Black-box UDA on three different medical image
segmentation datasets. Experiments show that our paradigm is efficient and
robust under various settings.

##  1 Introduction

Foundation Models such as ChatGPT [20] and BEIT-3 [32] are robust and
efficient. They have an increasing power to generalize different types of
data. It is a trend that people can adapt their private data by generating a
tiny task-specific model under the instruction of the foundation models
[32][20][21]. However, making such a foundation model available to everyone is
a luxury. On the one hand, training such a big model is expensive even for
big-name companies. On the other hand, the owners should take into account
commercial considerations and data misuse. Therefore, foundation models such
as GPT31, ERNIE 3.0 [25] are only provided as a service when only black-box
API can be accessed.

Under the so-called Model-as-a-Service (MaaS) [24] scenario, users can adapt
their data via zero-shot or black-box source-free domain adaptation. These
techniques work well on language tasks but are inefficient in visual tasks
because the distribution gap in vision tasks are large [35]. Even foundation
vision models cannot generalize on such varieties of data. Thus, foundation
models can help improve the efficiency of their task-specific models for the
users[28, 27], especially for those who only have little or unannotated data.
However, it still requires a tuning process for users to adapt their data[34,
33].

The phenomenon and need in the medical image analysis scenarios are more
severe and common [17, 38, 10]. The performance of the source model can be
substantially degraded when transferred to a different distribution. Also, the
need for data/model privacy protection in the hospital is more strict than in
other scenarios[29, 30].

Thus, a common question in both MaaS and medical image analysis scenarios has
emerged: Can we balance the privacy of the foundation/source models and the
practical needs for users to train their models? A recently proposed strategy
called black-box domain adaptation [14] provides a solution for model owners
by only providing the forward propagated logits to users. In this paper, we
extend this strategy to provide both the forward and backward propagated
results. This operation can ease the users to adapt their data simultaneously
without affecting the privacy of source data and models. This proposed
paradigm is called Back-Propagated Black-Box Adaptation (BPBA).

For a better illustration, as shown in Table. 1, conventional UDA [8]
(Unsupervised Domain Adaptation) allows users to access both source data and
model. White-box UDA[13] bans the user’s access to source data to preserve
data privacy compared to conventional UDA. Black-box UDA [37] only allows the
users to achieve the forward-propagation results (logits, etc.) to preserve
the model privacy. While our BPBA additionally allows users to utilize the
back-propagated information (usually gradients) compared to Black-box UDA.
This relieves the burden for users to establish a tiny model on the private
data as well as preserves the model privacy for source model owners.

Table 1: Comparison of Different Existing Paradigms. UDA indicates unsupervised domain adaptation; Param. Avail. indicates the availability or updating of the parameters of source model; For-propa. indicates forward-propagation; Back-propa. indicates back-propagation outputs. | Source Data | Param. Avail. | For-propa. | Back-propa.
---|---|---|---|---
Conventional UDA [8] | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$
White-box UDA [13] | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$
Black-box UDA [37] | $\times$ | $\times$ | $\checkmark$ | $\times$
BPBA (Ours) | $\times$ | $\times$ | $\checkmark$ | $\checkmark$

###  1.1 BPBA and Self-supervised learning

To better understand the BPBA, we give an analysis and comparison of BPBA and
self-supervised learning [11, 6, 9, 36]. Since no source data and no ground
truth are provided, we consider the adaption problem as a conditional self-
supervised alignment problem. Both settings aim to learn good representations
from the unlabeled data for the downstream tasks. Nevertheless, in BPBA, we
have a conditional well-pre-trained model that contains lots of implicit
information which can be inherited to optimize the target model. This is the
crucial difference compared to the conventional self-supervised problem. How
to leverage the hidden information in the source model shall be considered in
the model design.

Moreover, take a brief look at traditional self-supervised learning; the
critical properties of the success of self-supervised learning strategies is
to maintain the alignment and uniformity of representations [31]. The
alignment indicates that the representations of the positive pair are close in
the latent space. The uniformity indicates that the representations should be
distributed as evenly as possible on the unit sphere. If we fail to align the
positive pair, the model will learn the sub-optimal representations. If we
lose uniformity, the model will suffer from mode collapse. In BPBA, we also
need to consider both properties when adapting unlabeled data. However, the
pre-trained model provides a strict mapping for unlabeled data, thus holding a
stable and uniform distribution in the latent space. Therefore, other than
unleashing the potential of the source model, another critical point to
optimize the model is to build a better alignment strategy to align the
unlabeled and invisible source data.

###  1.2 Bootstrap The Original Latent

To better utilize the user’s data and guarantee the alignment, we propose a
Freeze-and-thaw Adapter strategy called BTOL as shown in Fig. 1. Considering
the potential domain gap between the unseen source data/model and target data,
we design a novel cross-supervised paradigm to align the distribution via an
adapter implicitly. To fully exploit the source model and unlabeled data, a
freeze-and-thaw strategy is developed for the adapter and target model to
teach each other. Notice that our BTOL can also be utilized in the Black-box
UDA task via a simulator as proposed in the lower part of Fig. 1. Under both
Black-box UDA and BPBA settings, our BTOL can make full use of the source
model and target data.

The contributions of this paper can be summarized into three folds:

(1) We find a better solution to balance the model privacy and user needs in
both MaaS and medical image analysis by allowing users access to back-
propagated information. We extend the Black-box UDA to Back-Propagated Black-
Box Adaptation (BPBA) which can benefit both model owners and users.

(2) We propose a novel training strategy called BTOL under the proposed
setting. Our solution consists of an adapter and a freeze-and-thaw strategy.
We then broaden our strategy to the Black-box UDA task, and our model also
achieves state-of-the-art performance.

(3) Our paradigm outperforms all the Black-box UDA and White-box UDA methods
on different settings and datasets.

##  2 Methodology

In this section, we introduce the novel approach BTOL to better uncover the
potential of the source model without any manual regularizations. For
simplicity, in our method, adapters are used to narrow the distribution gap
between the unseen source data and target data, freeze-and-thaw strategy can
provide a constraint to avoid the representation collapse. To better exploit
the potential of our method, we utilize our BTOL in both the BPBA and black-
box UDA settings. Without any manual augmentations, our method can easily
outperform any existing black-box UDA methods and even some white-box UDA
methods. Detailed methods and analysis are presented below.

###  2.1 BTOL in the BPBA

We first build the BTOL strategy for BPBA, which is shown in the upper part of
Fig. 1. In BPBA, given a foundation/source model in the cloud server, which
only allows users to achieve the output logits and back-propagated gradients.
Users can utilize the two feed-backs to train their models.

As the source model is efficient only on the source domain data, we aim to
transfer the target domain to source-domain-like distribution to satisfy the
source model. Thus the source model can produce a more accurate result. We set
an adapter to deal with the domain transfer and align the distribution between
the unseen source and target domains.

We then introduce a dual EM[18] approach called the freeze-and-thaw strategy
to deal with the alignment. In the conventional EM algorithm, the expectation
step and maximization step are interleaved by recalculating the parameters and
estimating the distribution. In our method, we make an extension and
adaptation for the BPBA problem. The updating of the adapter and target model
are interleaved by freezing one module and training another one. The pseudo-
code is presented in Algorithm 1.

The whole pipeline of our strategy is straightforward and easy to understand.
We first trained the target model using the pseudo labels generated from
source models. Then we train the adapter and target model via the freeze-and-
thaw strategy. All the results will be reported in the experiments.

Figure 1: Overview of our BTOL under different settings. Upper: BPBA setting ;
lower: Black-box UDA setting

###  2.2 BTOL in the Black-box UDA

In the Black-box UDA setting, The back-propagated result is unavailable.
Existing methods can only utilize manual augmentations, label-denoising
techniques, and regularizations. They all fail to bootstrap the inner latent
information in the original source model. Under this tougher condition, we
introduce a simulator to simulate the source data distribution; then, we can
apply our freeze-and-thaw under this setting.

The simulation step is presented in the second and third steps of the lower
part in Fig. 1 after training the target model via the pseudo labels. We copy
the weights of the target model to initialize the simulator. In the second
step, we train the adapter to satisfy both the source and target models. This
step helps the adapter output a mixture of source and target domains. In the
third step, we freeze the adapter and train the simulator. This allows the
simulator to get familiar with the output of the trained adapter, which is the
mixed distribution of source and target domains. In the simulation procedure,
the ideal distribution the simulator can imitate is the average mixture of the
source and target distributions. After the simulation, we can use the trained
simulator to replace the invisible source model to execute the freeze-and-thaw
strategy.

Algorithm 1 Training of BTOL under BPBA.

1:Random initialize adapter $A$ and target model $T$; Get access to source
model $S$; The freeze-and-thaw loss is cross-entropy loss $L_{ce}$; $T$ is
number of the training epoch. $E_{1}$ and $E_{2}$ are the numbers of tiny
epoch in the training loop.

2:Target Model Initialization: Trained the target model $T$ with the pseudo
labels generated by the source model $S$ until converged.

3:Freeze-and-Thaw:

4:for $t=0,1,\ldots,T-1$ do

5: target model $T$:Freeze the parameters.

6: for $e=0,1,\ldots,E_{1}-1$ do

7: adapter $A$: Learn parameters via the gradient downloaded from source model
$S$ under the supervision from the pseudo output logits from target model $T$.

8: end for

9: adapter $A$:Freeze the parameters.

10: for $e=0,1,\ldots,E_{2}-1$ do

11: target model $T$: Learn parameters under the supervision from the pseudo
output logits from the series connection of adapter $A$ and source model $S$.

12: end for

13:end for

##  3 Experiments

###  3.1 Dataset and Evaluation Metric.

We evaluate our method on three tasks: fundus segmentation, cardiac structure
segmentation, and prostate segmentation. For the fundus segmentation task, we
use the training set of REFUGE challenge [19] as the source domain and use the
RIM-ONE-r3 [7] as the target domain. Following [4], we split the target domain
into 99/60 images for training and testing, respectively. We resize each image
to $512\times 512$ to feed the network. For the cardiac structure segmentation
task, we choose the public ACDC dataset [3] contains 200 volumes as the source
domain. For the target domain, we use the LGE dataset from Multi-sequence
Cardiac MR segmentation Challenge (MSCMR2019) [39]. We split it into 80%/20%
for training and testing. We use 2d slices for training, and all images are
resized to $192\times 192$ as the network input. As for prostate segmentation,
we use the MSD05 [1] as the source domain and Promise12 [15] as the target
domain. We also split it into 80%/20% for training and testing. For
evaluation, we use two commonly-used metrics in the medical segmentation
field: Dice Score (DSC) and Average Surface Distance (ASD). DSC measures the
overlap between prediction and ground truth, and ASD represents the
performance at the object boundary. Higher DSC and lower ASD mean better
performance.

Table 2: Experiments of fundus segmentation. The source model is trained on REFUGE challenge and the target data is RIM-ONE-r3. The baseline model is the target model which is trained on the pseudo label. Type "W" indicates the method is a white-box UDA method; the Type "B" indicates BTOL in the black-box UDA; Type "P" indicates BTOL in the BPBA. Methods | Type | Dice $\uparrow$ | ASD$\downarrow$
---|---|---|---
|  | Disc | Cup | Avg. | Disc | Cup | Avg.
Source Model | - | $88.34^{\pm 4.48}$ | $71.35^{\pm 22.75}$ | $79.85^{\pm 12.59}$ | $10.65^{\pm 4.27}$ | $10.75^{\pm 5.34}$ | $10.7^{\pm 4.18}$
Baseline Model | - | $91.34^{\pm 4.49}$ | $71.85^{\pm 17.00}$ | $81.59^{\pm 9.83}$ | $7.62^{\pm 3.65}$ | $10.64^{\pm 6.71}$ | $9.00^{\pm 3.87}$
SRDA [2] | W | $89.37^{\pm 2.70}$ | $77.61^{\pm 13.58}$ | $83.49^{\pm 8.14}$ | $9.91^{\pm 2.45}$ | $10.15^{\pm 5.75}$ | $10.03^{\pm 4.10}$
AdvEnt [26] | W | $89.73^{\pm 3.66}$ | $77.99^{\pm 21.08}$ | $83.86^{\pm 12.37}$ | $9.84^{\pm 3.86}$ | $7.57^{\pm 4.24}$ | $8.71^{\pm 8.10}$
DAE [12] | W | $89.08^{\pm 3.32}$ | $79.01^{\pm 12.82}$ | $84.41^{\pm 8.07}$ | $11.63^{\pm 6.84}$ | $10.31^{\pm 8.45}$ | $10.97^{\pm 7.65}$
DPL [4] | W | $90.13^{\pm 3.06}$ | $79.78^{\pm 11.05}$ | $84.95^{\pm 7.06}$ | $9.43^{\pm 3.46}$ | $9.01^{\pm 5.59}$ | $9.22^{\pm 4.53}$
EMD [16] | B | $90.50^{\pm 3.78}$ | $73.50^{\pm 11.56}$ | $82.00^{\pm 8.76}$ | $10.52^{\pm 4.18}$ | $7.12^{\pm 4.15}$ | $8.82^{\pm 2.59}$
BTOL(Ours) | B | $92.54^{\pm 4.08}$ | $72.86^{\pm 13.46}$ | $82.70^{\pm 8.24}$ | $8.45^{\pm 3.32}$ | $9.14^{\pm 3.64}$ | $8.80^{\pm 3.02}$
BTOL(Ours) | P | $91.52^{\pm 4.19}$ | $78.73^{\pm 11.87}$ | $\mathbf{85.13^{\pm 7.14}}$ | $7.53^{\pm 3.36}$ | $8.76^{\pm 3.66}$ | $\mathbf{8.14^{\pm 2.74}}$
Table 3: Experiments of cardiac structure segmentation and prostate
segmentation. Due to the page limitation, we choose the previous sota method
DPL for a fair comparison.

Cardiac Structure Segmentation
---
Methods | Type | Dice $\uparrow$
|  | RV | Myo | LV | Avg.
Source Model | - | $40.28^{\pm 26.73}$ | $48.83^{\pm 10.84}$ | $76.45^{\pm 10.21}$ | $55.19^{\pm 14.19}$
Baseline Model | - | $46.67^{\pm 30.49}$ | $53.95^{\pm 9.70}$ | $76.01^{\pm 9.11}$ | $58.87^{\pm 13.37}$
DPL [4] | W | $48.14^{\pm 29.45}$ | $53.76^{\pm 9.11}$ | $78.42^{\pm 8.64}$ | $60.11^{\pm 15.73}$
EMD [16] | B | $47.59^{\pm 28.46}$ | $53.67^{\pm 9.79}$ | $75.48^{\pm 9.58}$ | $58.91^{\pm 13.48}$
BTOL(Ours) | B | $47.12^{\pm 29.45}$ | $53.85^{\pm 10.15}$ | $78.45^{\pm 9.66}$ | $59.81^{\pm 12.86}$
BTOL(Ours) | P | $49.78^{\pm 25.45}$ | $54.12^{\pm 9.46}$ | $76.52^{\pm 10.40}$ | $\mathbf{60.14^{\pm 13.04}}$
Cardiac Structure Segmentation
Methods | Type | ASD $\downarrow$
|  | RV | Myo | LV | Avg.
Source Model | - | $4.50^{\pm 3.42}$ | $4.60^{\pm 2.51}$ | $5.78^{\pm 2.02}$ | $4.96^{\pm 1.74}$
Baseline Model | - | $2.03^{\pm 1.61}$ | $4.14^{\pm 1.87}$ | $5.52^{\pm 2.36}$ | $3.90^{\pm 1.28}$
DPL [4] | W | $1.55^{\pm 1.24}$ | $4.75^{\pm 2.04}$ | $4.95^{\pm 2.23}$ | $3.75^{\pm 1.20}$
EMD [16] | B | $2.12^{\pm 1.47}$ | $4.25^{\pm 1.95}$ | $5.13^{\pm 2.78}$ | $3.83^{\pm 1.41}$
BTOL(Ours) | B | $1.76^{\pm 1.45}$ | $4.55^{\pm 1.34}$ | $5.32^{\pm 2.78}$ | $3.88^{\pm 1.24}$
BTOL(Ours) | P | $1.32^{\pm 1.11}$ | $4.24^{\pm 1.92}$ | $5.45^{\pm 2.18}$ | $\mathbf{3.67^{\pm 1.14}}$

Prostate Segmentation
---
Methods | Type | Dice $\uparrow$ | ASD $\downarrow$
Source Model | - | $47.50^{\pm 26.21}$ | $9.80^{\pm 8.84}$
Baseline Model | - | $51.68^{\pm 24.56}$ | $8.99^{\pm 8.25}$
DPL [4] | W | $52.95^{\pm 23.14}$ | $7.98^{\pm 7.26}$
EMD [16] | B | $52.47^{\pm 23.18}$ | $8.11^{\pm 7.85}$
BTOL(Ours) | B | $52.40^{\pm 23.18}$ | $7.88^{\pm 7.02}$
BTOL(Ours) | P | $\mathbf{54.16^{\pm 22.75}}$ | $\mathbf{6.58^{\pm 6.98}}$

Table 4: Experiments of fundus segmentation of different backbones. The source model is DeepLabV3+ and target model is UNet[22]. The source model is trained on REFUGE challenge and the target data is RIM-ONE-r3. Experiments show that our method can work on different target model backbones. Methods | Type | Dice $\uparrow$ | ASD$\downarrow$
---|---|---|---
|  | Disc | Cup | Avg. | Disc | Cup | Avg.
Baseline Model | - | $46.39^{\pm 34.86}$ | $75.06^{\pm 27.98}$ | $60.73^{\pm 29.42}$ | $14.90^{\pm 11.52}$ | $15.61^{\pm 14.45}$ | $15.26^{\pm 8.95}$
BTOL(Ours) | B | $47.40^{\pm 31.20}$ | $76.48^{\pm 26.55}$ | $61.94^{\pm 28.47}$ | $14.56^{\pm 10.76}$ | $13.00^{\pm 12.45}$ | $13.78^{\pm 8.62}$
BTOL(Ours) | P | $49.70^{\pm 32.68}$ | $78.15^{\pm 26.80}$ | $\mathbf{63.93^{\pm 27.67}}$ | $13.56^{\pm 9.80}$ | $12.17^{\pm 10.64}$ | $\mathbf{12.87^{\pm 9.15}}$

###  3.2 Implementation Details

For all experiments, we use DeepLabV3+ [5] with MobileNetV2 [23] backbone as
the segmentation model (e.g., Target model or Simulator in Figure 1). For the
adapter, we use a neural network consisting of three blocks, and each block
includes a 2D convolution layer and Relu activation function. We use Adam
optimizer with learning rate as $1e^{-4}$ and set batch size as 8 without
specific choice. We train the target model from scratch 100 epochs in the
target model initialization stage. After that, we set $T=4$, $E_{1}=10$ and
$E_{2}=30$ in Figure 1.

###  3.3 Experimental Results

We test our method mainly on medical image datasets from different body parts
to prove efficacy and robustness, including fundus segmentation, cardiac
structure segmentation, and prostate segmentation. As presented in Table. 3,
2, When applying BTOL to black-box UDA tasks, our model can easily outperform
previous methods without any augmentations, which means our method can fully
exploit the potential information from the source model. When applying our
method to the BPBA setting without getting access to or updating the source
model parameters, our model outperforms all white-box UDA methods under all
the settings. This proves that our model can make more efficient use of the
source model, even with fewer operations. We believe that if adding more
augmentations and regularizations, our method can achieve more impressive
results.

We also test out strategy by changing the backbone of the target model as
shown in 4. We changed the target model backbone from DeepLabV3+ to UNet[22].
Our strategy can still stably outperform the baseline model, which shows our
strategy is robust and easy to use in various types of backbone.

##  4 Discussion

From the view of self-supervised learning, our BTOL can be thought of as a
conditional dual SimSiam [6]. SimSiam attributes the collapsing solutions to
the stop-gradient operation. In our method, since there is a well-trained
source model, we can thus broaden the unilateral stop-gradient operation into
the dual freeze-and-thaw strategy without considering the model collapse. This
EM-like approach can adequately utilize the source model. Moreover, our model
does not need a pair of augmented inputs like SimSiam because the source model
can provide a fixed distribution to maintain uniformity. If adding more
augmentation procedures like SimSiam, we believe our model can achieve an
improvement.

##  5 Conclusion

In this paper, we address a new setting, BPBA, to deal with the balance
between model privacy and user needs when adapting the Black-box
foundation/source models. Compared to Black-box UDA, we release the back-
propagated information for users to train their models. We also propose a new
strategy called BTOL to fully utilize the implicit information from the source
model to optimize the target model. BTOL contains an adapter and a freeze-and-
thaw strategy to cross-teach the adapter and target model. Our paradigm
outperforms all the methods on different settings. Our setting and solution
can benefit nearly all the fields which utilize the foundation/source models.
We hope our work can provide a new perspective and solution in the new era of
foundation models.

## References

  * [1] Antonelli, M., Reinke, A., Bakas, S., Farahani, K., et al.: The medical segmentation decathlon. Nature Communications 13(1) (Jul 2022)
  * [2] Bateson, M., Kervadec, H., Dolz, J., Lombaert, H., Ayed, I.B.: Source-relaxed domain adaptation for image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 490–499. Springer (2020)
  * [3] Bernard, O., Lalande, A., Zotti, C., et al.: Deep learning techniques for automatic mri cardiac multi-structures segmentation and diagnosis: is the problem solved? IEEE Transactions on Medical Imaging 37(11), 2514–2525 (2018)
  * [4] Chen, C., Liu, Q., Jin, Y., Dou, Q., Heng, P.A.: Source-free domain adaptive fundus image segmentation with denoised pseudo-labeling. In: MICCAI. pp. 225–235. Springer (2021)
  * [5] Chen, L., Zhu, Y., Papandreou, G., Schroff, F., Adam, H.: Encoder-decoder with atrous separable convolution for semantic image segmentation. In: ECCV (2018)
  * [6] Chen, X., He, K.: Exploring simple siamese representation learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 15750–15758 (2021)
  * [7] Fumero, F., Alayón, S., Sanchez, J.L., Sigut, J., Gonzalez-Hernandez, M.: Rim-one: An open retinal image database for optic nerve evaluation. In: international symposium on computer-based medical systems. pp. 1–6. IEEE (2011)
  * [8] Ganin, Y., Lempitsky, V.: Unsupervised domain adaptation by backpropagation. In: International conference on machine learning. pp. 1180–1189. PMLR (2015)
  * [9] Grill, J.B., Strub, F., Altché, F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M., et al.: Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems 33, 21271–21284 (2020)
  * [10] Guan, H., Liu, M.: Domain adaptation for medical image analysis: a survey. IEEE Transactions on Biomedical Engineering 69(3), 1173–1185 (2021)
  * [11] He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 9729–9738 (2020)
  * [12] Karani, N., Erdil, E., Chaitanya, K., Konukoglu, E.: Test-time adaptable neural networks for robust medical image segmentation. MIA 68, 101907 (2021)
  * [13] Liang, J., Hu, D., Feng, J.: Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In: International Conference on Machine Learning. pp. 6028–6039. PMLR (2020)
  * [14] Liang, J., Hu, D., Feng, J., He, R.: Dine: Domain adaptation from single and multiple black-box predictors. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8003–8013 (2022)
  * [15] Litjens, G., Toth, R., van de Ven, W., Hoeks, C., Kerkstra, S., van Ginneken, B., Vincent, G., Guillard, G., Birbeck, N., Zhang, J., et al.: Evaluation of prostate segmentation algorithms for mri: the promise12 challenge. Medical image analysis 18(2), 359–373 (2014)
  * [16] Liu, X., Yoo, C., Xing, F., Kuo, C.C.J., El Fakhri, G., Kang, J.W., Woo, J.: Unsupervised black-box model domain adaptation for brain tumor segmentation. Frontiers in Neuroscience p. 341 (2022)
  * [17] Mahmood, F., Chen, R., Durr, N.J.: Unsupervised reverse domain adaptation for synthetic medical images via adversarial training. IEEE transactions on medical imaging 37(12), 2572–2581 (2018)
  * [18] Moon, T.K.: The expectation-maximization algorithm. IEEE Signal processing magazine 13(6), 47–60 (1996)
  * [19] Orlando, J.I., Fu, H., Breda, J.B., van Keer, K., Bathula, D.R., et al.: Refuge challenge: A unified framework for evaluating automated methods for glaucoma assessment from fundus photographs. Medical image analysis 59, 101570 (2020)
  * [20] Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C.L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al.: Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155 (2022)
  * [21] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PMLR (2021)
  * [22] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: International Conference on Medical image computing and computer-assisted intervention. pp. 234–241. Springer (2015)
  * [23] Sandler, M., Howard, A.G., Zhu, M., Zhmoginov, A., Chen, L.: Mobilenetv2: Inverted residuals and linear bottlenecks. In: CVPR (2018)
  * [24] Sun, T., Shao, Y., Qian, H., Huang, X., Qiu, X.: Black-box tuning for language-model-as-a-service. In: International Conference on Machine Learning. pp. 20841–20855. PMLR (2022)
  * [25] Sun, Y., Wang, S., Feng, S., Ding, S., Pang, C., Shang, J., Liu, J., Chen, X., Zhao, Y., Lu, Y., et al.: Ernie 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation. arXiv preprint arXiv:2107.02137 (2021)
  * [26] Vu, T.H., Jain, H., Bucher, M., et al.: Advent: Adversarial entropy minimization for domain adaptation in semantic segmentation. In: CVPR. pp. 2517–2526 (2019)
  * [27] Wang, L.: Dionysus: Recovering scene structures by dividing into semantic pieces
  * [28] Wang, L., Chen, L.: Ftso: Effective nas via first topology second operator. arXiv preprint arXiv:2303.12948 (2023)
  * [29] Wang, L., Gong, Y., Ma, X., Wang, Q., Zhou, K., Chen, L.: Is-mvsnet: Importance sampling-based mvsnet. In: Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXII. pp. 668–683. Springer (2022)
  * [30] Wang, L., Gong, Y., Wang, Q., Zhou, K., Chen, L.: Flora: dual-frequency loss-compensated real-time monocular 3d video reconstruction. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 1 (2023)
  * [31] Wang, T., Isola, P.: Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In: International Conference on Machine Learning. pp. 9929–9939. PMLR (2020)
  * [32] Wang, W., Bao, H., Dong, L., Bjorck, J., Peng, Z., Liu, Q., Aggarwal, K., Mohammed, O.K., Singhal, S., Som, S., et al.: Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208.10442 (2022)
  * [33] Zeng, D., Liu, W., Chen, W., Zhou, L., Zhang, M., Qu, H.: Substructure aware graph neural networks. In: Proc. of AAAI (2023)
  * [34] Zeng, D., Zhou, L., Liu, W., Qu, H., Chen, W.: A simple graph neural network via layer sniffer. In: ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 5687–5691. IEEE (2022)
  * [35] Zhang, D., Chen, M., Li, C., Huang, L., Zhang, J.: Aggregation of disentanglement: Reconsidering domain variations in domain generalization. arXiv preprint arXiv:2302.02350 (2023)
  * [36] Zhang, D., Li, C., Li, H., Huang, W., Huang, L., Zhang, J.: Rethinking alignment and uniformity in unsupervised image semantic segmentation. arXiv preprint arXiv:2211.14513 (2022)
  * [37] Zhang, H., Zhang, Y., Jia, K., Zhang, L.: Unsupervised domain adaptation of black-box source models. arXiv preprint arXiv:2101.02839 (2021)
  * [38] Zhang, Y., Wei, Y., Wu, Q., Zhao, P., Niu, S., Huang, J., Tan, M.: Collaborative unsupervised domain adaptation for medical image diagnosis. IEEE Transactions on Image Processing 29, 7834–7844 (2020)
  * [39] Zhuang, X.: Multivariate mixture model for myocardial segmentation combining multi-source images. IEEE Transactions on Pattern Analysis and Machine Intelligence 41(12), 2933–2946 (2018)

Figure 2: The structure of Adapter in Figure 1. The Adapter consists of $k$ blocks and a residual skip connection. We set $k=3$ in all experiments. $H$,$W$,$C$ denote the height, width and the number of channels of inputs, respectively. Table 5: Results on different $k$ (the number of blocks in Adapter) on the RIM-ONE-r3 dataset. $k$ | 1 | 3 | 5
---|---|---|---
Dice (%) | 83.19$\pm$8.56 |  85.13$\pm$7.14 | 84.92$\pm$7.56