Federated and Generalized Person Re-identification
through Domain and Feature Hallucinating

Fengxiang Yang1 Zhun Zhong2 Zhiming Luo1
Shaozi Li1 Nicu Sebe2

1 Xiamen University 2 University of Trento

###### Abstract

In this paper, we study the problem of federated domain generalization (FedDG)
for person re-identification (re-ID), which aims to learn a generalized model
with multiple decentralized labeled source domains. An empirical method
(FedAvg) trains local models individually and averages them to obtain the
global model for further local fine-tuning or deploying in unseen target
domains. One drawback of FedAvg is neglecting the data distributions of other
clients during local training, making the local model overfit local data and
producing a poorly-generalized global model. To solve this problem, we propose
a novel method, called “Domain and Feature Hallucinating (DFH)”, to produce
diverse features for learning generalized local and global models.
Specifically, after each model aggregation process, we share the Domain-level
Feature Statistics (DFS) among different clients without violating data
privacy. During local training, the DFS are used to synthesize novel domain
statistics with the proposed domain hallucinating, which is achieved by re-
weighting DFS with random weights. Then, we propose feature hallucinating to
diversify local features by scaling and shifting them to the distribution of
the obtained novel domain. The synthesized novel features retain the original
pair-wise similarities, enabling us to utilize them to optimize the model in a
supervised manner. Extensive experiments verify that the proposed DFH can
effectively improve the generalization ability of the global model. Our method
achieves the state-of-the-art performance for FedDG on four large-scale re-ID
benchmarks.

Figure 1: Illustration of our solution for FedDG re-ID. In the FedDG re-ID,
each domain is considered as an individual client and we aim to train a
generalized model with these decentralized data. In our solution, we share the
domain-level feature statistics (DFS) of all clients to facilitate generalized
re-ID training. The training process contains four steps: \raisebox{-.9pt}{1}⃝
local training with DFS downloaded from server, which synthesizes novel
features with our proposed domain & feature hallucinating (DFH) to improve the
generalization of local models. \raisebox{-.9pt}{2}⃝ client-to-server
updating, which updates DFS in server side. \raisebox{-.9pt}{3}⃝ server-side
model aggregation. \raisebox{-.9pt}{4}⃝ redistributing global model and
server-side DFS. By running \raisebox{-.9pt}{1}⃝ to \raisebox{-.9pt}{4}⃝ for
several epochs, we can effectively improve the generalization of global model.

##  1 Introduction

Person re-identification (re-ID) aims at retrieving target pedestrians in a
non-overlapped camera system. Thanks to the burgeoning development of deep
learning, the accuracies of deep re-ID models have been boosted rapidly Sun et
al. (2018); Wang et al. (2018); Ye et al. (2021). Despite their promising
performance in training domains, most of them can not generalize well to
unseen domains. A plausible solution for this problem is domain generalization
(DG), which aims to learn generalized models with source domains. However,
existing advances in DG Zhao et al. (2021); Song et al. (2019); Zhou et al.
(2020); Dai et al. (2021); Guo et al. (2020) usually demand the centralization
of training data, which may bring privacy issues and thus limit the real-world
applications.

Federated learning McMahan et al. (2017); Liu et al. (2021); Yoon et al.
(2021); Li et al. (2021); Zhuang et al. (2020) is one way to alleviate the
privacy issues, which utilizes multiple clients to jointly learn a model
without exchanging their local data. Recently, Wu et al. Wu & Gong (2021a)
introduce a new paradigm, called federated domain generalized (FedDG) re-ID,
to jointly solve the generalization and privacy problems. As illustrated in
Fig. 1, FedDG re-ID treats each domain as an individual client and aims to
learn a generalized re-ID model without data transmission. Wu & Gong (2021a)
achieves this goal with FedAvg McMahan et al. (2017), which repeatedly
iterates the local training and the model aggregation processes. In addition,
to avoid the model from overfitting on the local data, Wu & Gong (2021a)
enforces the local model to mimic the predictions of the global model.
However, this approach does not explicitly consider the data distribution of
all clients during local training, hindering the further improvement of
generalization for the global model.

In this paper, we introduce a new learning tactic for FedDG re-ID, which
shares the domain-level feature statistics (DFS) of all domains without
violating data privacy. For each domain, DFS are obtained by the mean and
variance of identities. Given the shared DFS, we propose a Domain and Feature
Hallucinating (DFH) approach to generate diverse novel features, which are
used to learn generalized models. The process of our method is shown in Fig.
1, which contains four steps. In step \raisebox{-.9pt}{1}⃝, we locally train
the model of each client with the proposed DFH, where the local model and DFS
are downloaded from the server. At each training iteration, we first propose
Feature Hallucinating (FH) to diversify local features by scaling and shifting
with the shared domain statistics of all clients. Then, to further diversify
the local features, we synthesize novel domain statistics with the proposed
Domain Hallucinating (DH), which is achieved by re-weighting DFS with randomly
sampled weights. The novel statistics are also used in FH to generate novel
features. The generated novel features retain the pair-wise similarities of
the original ones, which can be utilized to improve generalization of local
models with supervised learning. After local training, the local models and
DFS are uploaded to the server (step \raisebox{-.9pt}{2}⃝) for updation. Next,
we obtain the global model by averaging local models (step
\raisebox{-.9pt}{3}⃝). Finally, the global model and DFS are redistributed to
each client for the next epoch of local training (step \raisebox{-.9pt}{4}⃝).
With several iterations, we can obtain the final global model, which is
deployed on unseen target domains. Our contributions are three-fold:

  * •

We introduce a new learning strategy for FedDG re-ID. By sharing the domain-
level feature statistics (DFS) among domains, we are able to generate features
of new distributions, without violating data privacy.

  * •

We design a Feature Hallucinating (FH) method to generate novel features with
the obtained domain statistics. The hallucinated features retain the pair-wise
similarities of the original features, enabling us to use them to optimize the
model in a supervised manner.

  * •

We propose a Domain Hallucinating (DH) method to synthesize novel domain
statistics, allowing the local models to see more diversified data
distributions under the federated learning scenario.

Extensive experiments demonstrate the advantage of our learning strategy and
DFH method in improving the generalization of the global model. Our method
achieves state-of-the-art results on re-ID benchmarks for FedDG re-ID.

##  2 Related Work

Domain Generalization Recent advances have paid more attention to training
generalizable re-ID models, such as domain adaptation (DA) Zheng et al.
(2021); Zhong et al. (2019); Yang et al. (2021); Ge et al. (2020), and domain
generalization (DG) Li et al. (2018); Zhao et al. (2021); Jin et al. (2020);
Chattopadhyay et al. (2020). DG optimizes re-ID models with several source
domains and directly deploys the obtained model to target domain without
further fine-tuning, which is a more applicable scenario than DA due to its
low dependency on training data of target domain. Most DG methods focus on the
close-set scenario Chattopadhyay et al. (2020); Li et al. (2018); Muandet et
al. (2013); Qiao et al. (2020), considering that target domains have exactly
the same IDs with the source domains and thus can hardly be applied to real-
world applications. Different from them, DG for person re-ID is a more
challenging problem because of the disjoint label space in the training and
testing set. There are many algorithms designed for training domain
generalized person re-ID Song et al. (2019); Jin et al. (2020); Liao & Shao
(2020); Zhao et al. (2021). Jin et al. Jin et al. (2020) propose to
disentangle features into ID-relevant and -irrelevant parts for subsequent
feature stylization, reconstruction, and training. Liao et al. Liao & Shao
(2020) improve the generalization of re-ID models by using their QAConv to
compute novel sample-wise similarities. Zhao et al. Zhao et al. (2021) utilize
meta-learning to improve the generalization of models and integrate memory
bank Zhong et al. (2019) into the training process to address a series of
problems raised by the open-set attribute Panareda Busto & Gall (2017) in re-
ID. These methods are successful, but they require combining several domains
to achieve good results and may cause privacy problems when transferring data
between different clients. In this paper, we consider a more practical and
challenging scenario for DG in re-ID, which optimizes re-ID models with
decentralized source data.

Federated Learning In federated learning, training data are stored in several
isolated local clients and can not be transferred to other clients to protect
data privacy. McMahan et al. propose FedAvg McMahan et al. (2017) algorithm as
the baseline method by averaging local models trained with local data and
redistributing the averaged global model to local clients for further
training. However, FedAvg may suffer from the non-IID or data heterogeneity
problem Zhao et al. (2018) where the data distribution of local clients are
different, leading to considerable performance degradation for the aggregated
model. To solve this problem, FedProx Li et al. (2020) adds a regularization
term to enforce the local model similar to the averaged global model.
Recently, many researches Feng et al. (2020); Liu et al. (2021); Ahmed et al.
(2021) extend the content of federated learning by applying it to various
types of computer vision tasks, but most of them are designed for closed-set
image classification or segmentation tasks, which have the same identities
between training and testing sets. For the open-set Panareda Busto & Gall
(2017) re-ID problem, Zhuang et al. Zhuang et al. (2020) explore the
application of federated learning in re-ID and propose two settings called
“Federated-by-camera" and “Federated-by-identity" for the federated training
of a single domain. Wu et al. Wu & Gong (2021a) propose decentralized domain
generalization for person re-ID to make the DG re-ID algorithms privacy-
preserving. In this paper, we propose a novel DFH algorithm for FedDG re-ID
problem and evaluate our model with large-scale benchmarks.

Figure 2: The overall framework of our proposed method. We propose to share
domain-level feature statistics (DFS) for local training and the overall
training can be divided into four steps. \raisebox{-.9pt}{1}⃝ Local training.
\raisebox{-.9pt}{2}⃝ Client-to-server Updating. \raisebox{-.9pt}{3}⃝ Server-
side aggregation. \raisebox{-.9pt}{4}⃝ Redistributing. The four steps are
repeated for several steps to obtain final global model for evaluation.

##  3 Methodology

Problem Definition Suppose there are $N$ labeled source domains. For a given
domain $i$ ($1\leq i\leq N$), we have $M_{i}$ training images
$\mathcal{X}_{i}=(x_{1},x_{2},...,x_{M_{i}})$ and labels
$\mathcal{Y}_{i}=(y_{1},y_{2},...,y_{M_{i}})$ from $N_{i}$ IDs to train a
local model $\phi_{i}$. The goal of FedDG re-ID is utilizing these
decentralized data to optimize a well-generalized global re-ID model $\phi$.
Since each domain is considered as an individual client, “client" and “domain"
have the same meaning in the context of FedDG re-ID. The global model is then
used to be evaluated in unseen domains.

###  3.1 Overview

The overall training process for one training epoch is shown in Fig. 2, which
contains four steps: \raisebox{-.9pt}{1}⃝ local training, \raisebox{-.9pt}{2}⃝
client-to-server updating, \raisebox{-.9pt}{3}⃝ server-side aggregation, and
\raisebox{-.9pt}{4}⃝ redistributing. Before the optimization starts, we
extract local features with corresponding local models to obtain mean and
variance vectors for each identity ($\bm{\mu}_{i}\in\mathbb{R}^{N_{i}\times
d}$ and $\bm{\sigma}_{i}^{2}\in\mathbb{R}^{N_{i}\times d}$ are matrices with
$N_{i}$ rows and $d$ columns, where $d$ is feature dimension and $N_{i}$ is
the number of IDs for domain $i$). These ID-level feature statistics (IFS) are
used to estimate domain-level feature statistics (DFS), which will be uploaded
to server for public use. In step \raisebox{-.9pt}{1}⃝, we download DFS of all
clients from the server to transform local batch features to other domains
with the proposed feature hallucinating (FH). Moreover, we also enrich the
statistical information by synthesizing novel domain statistics with our
domain hallucinating (DH). Synthesized novel DFS, combined with DFS from other
domains, are used to generate novel features with FH. These transformed
features retain the pair-wise similarities of original ones and can be
utilized to improve local generalization in a supervised manner. After
executing local training for all clients, we upload local models and new DFS
to the server to update the server-side DFS (step \raisebox{-.9pt}{2}⃝) and
obtain the global model (step \raisebox{-.9pt}{3}⃝). In step
\raisebox{-.9pt}{4}⃝, the global model and updated DFS are redistributed to
each client to prepare for the next epoch of optimization. These four steps
are repeated for several epochs to obtain the final model for evaluation.

###  3.2 Local Training

In this section, we introduce the strategy to estimate DFS for preparation,
and demonstrate the local training process with our DFH.

Estimating DFS. We model the features of a specific domain $D_{i}$ as a
Gaussian distribution, which can be parameterized by mean ($\bm{\mu}_{D_{i}}$)
and variance vectors ($\bm{\sigma}_{D_{i}}^{2}$). A simple way to estimate
them is averaging ID-level mean ($\bm{\mu}_{i}$) and variance vectors
($\bm{\sigma}_{i}^{2}$). This is because IFS roughly reflects the feature
distribution of its corresponding client by showing how the class centroids
are scattered in the feature space, and the intra-class variations for each
ID. However, the features of $D_{i}$ could be quite various, which is
generally difficult to be represented by only two vectors $\bm{\mu}_{D_{i}}$
and $\bm{\sigma}_{D_{i}}^{2}$. To solve this drawback, we additionally treat
the domain-level mean and variance vectors as another two Gaussian
distributions, i.e.,
$\bm{\mu}_{D_{i}}\sim\mathcal{N}(\widehat{\bm{\mu}}_{i},\widehat{\bm{\sigma}}_{i}^{2})$
and
$\bm{\sigma}_{D_{i}}^{2}\sim\mathcal{N}(\widetilde{\bm{\mu}}_{i},\widetilde{\bm{\sigma}}_{i}^{2})$.
We estimate them with IFS by using following equations:

$\displaystyle\widehat{\bm{\mu}}_{i}=\frac{1}{N_{i}}\sum_{j=1}^{N_{i}}\bm{\mu}_{i,j},\quad(\widehat{\bm{\sigma}}_{i})^{2}=\frac{1}{N_{i}}\sum_{j=1}^{N_{i}}\Big{(}\bm{\mu}_{i,j}-\widehat{\bm{\mu}}_{i}\Big{)}^{2},$
(3.1)
$\displaystyle\widetilde{\bm{\mu}}_{i}=\frac{1}{N_{i}}\sum_{j=1}^{N_{i}}(\bm{\sigma}_{i,j}^{2}),\quad(\widetilde{\bm{\sigma}}_{i})^{2}=\frac{1}{N_{i}}\sum_{j=1}^{N_{i}}\Big{(}\bm{\sigma}_{i,j}^{2}-\widetilde{\bm{\mu}}_{i}\Big{)}^{2},$

where $\bm{\mu}_{i,j}$ is the $j$-th row of IFS $\bm{\mu}_{i}$, denoting the
mean vector of $j$-th ID in client $i$. $\bm{\sigma}_{i,j}^{2}$ has similar
meaning. These four parameters
($\widetilde{\bm{\mu}}_{i},\widetilde{\bm{\sigma}}_{i}^{2},\widehat{\bm{\mu}}_{i},\widehat{\bm{\sigma}}_{i}^{2}$)
are the final DFS shared for local training. By sampling from these two
Gaussian distributions (i.e.,
$\mathcal{N}(\widehat{\bm{\mu}}_{i},\widehat{\bm{\sigma}}_{i}^{2})$ and
$\mathcal{N}(\widetilde{\bm{\mu}}_{i},\widetilde{\bm{\sigma}}_{i}^{2})$), we
can generate diverse mean ($\bm{\mu}_{D_{i}}$) and variance
($\bm{\sigma}_{D_{i}}^{2}$) vectors to better describe domain $i$ while
protecting the data privacy. We thereby send them to the server to improve
local training with our feature hallucinating algorithm.

Feature Hallucinating. During the local training of domain $i$, we first
download DFS from the server and sample domain-level mean and variance vectors
from them. We then sample a batch of data from $D_{i}$ to obtain local batch
features $\mathbf{f}_{i}$, and transform them to another domain $k(k\neq i)$
with our feature hallucinating algorithm (FH), formulated as:

$\mathbf{f}_{D_{k}}=\bm{\mu}_{D_{k}}+\bm{\sigma}_{D_{k}}BN(\mathbf{f}_{i}),$
(3.2)

where $\mathbf{f}_{i}\in\mathbb{R}^{N_{b}\times d}$ is a batch of feature
vectors in $D_{i}$, extracted by the local model $\phi_{i}$. $BN$ is the
pooling-5 batch-norm function of $\phi_{i}$, which transforms $\mathbf{f}_{i}$
to the standard Gaussian distribution. $\bm{\mu}_{D_{k}}$ and
$\bm{\sigma}_{D_{k}}^{2}$ are sampled from the shared DFS (i.e.,
$\bm{\mu}_{D_{k}}\sim\mathcal{N}(\widehat{\bm{\mu}}_{k},\widehat{\bm{\sigma}}_{k}^{2})$
and
$\bm{\sigma}_{D_{k}}^{2}\sim\mathcal{N}(\widetilde{\bm{\mu}}_{k},\widetilde{\bm{\sigma}}_{k}^{2}$))
and will be replicated to the same size as $\mathbf{f}_{i}$ before
computation. It should be noted that Eq. 3.2 is similar to batch-norm Ioffe &
Szegedy (2015), where the affine parameters are replaced by the statistics in
unseen domain $D_{k}$ ($\bm{\mu}_{D_{k}}$ and $\bm{\sigma}_{D_{k}}^{2}$) for
scaling and shifting. Our FH enables the local model to improve local
generalization without breaking the privacy constraint. To make a better use
of the obtained DFS, we further enrich the local features with our domain
hallucinating method for more generalized local training.

Domain Hallucinating. Our domain hallucinating (DH) aims at synthesizing novel
domain-level mean and variance vectors by re-weighting vectors from other
domains that are sampled from the shared DFS. Specifically, we sample domain
weights from the Dirichlet distribution Shu et al. (2021) and obtain the novel
statistics $\bm{\mu}_{Novel}$ and $\bm{\sigma}_{Novel}^{2}$ with the following
equations:

$\displaystyle\mathbf{w}\sim$ $\displaystyle Dir(\bm{\alpha}),$
$\displaystyle\quad\bm{\mu}_{Novel}=\sum_{i=1}^{N}w_{i}\bm{\mu}_{D_{i}},$
$\displaystyle\quad\bm{\sigma}_{Novel}^{2}=\sum_{i=1}^{N}w_{i}\bm{\sigma}_{D_{i}}^{2},$
(3.3)

where $\mathbf{w}\in\mathbb{R}^{N}$ is the domain weight vector sampled from
Dirichlet distribution. The $i$-th element of $\mathbf{w}$ is $w_{i}$, which
denotes the weight for client $i$. $\bm{\alpha}$ is the $N$-dim parameter for
Dirichlet distribution. The synthesized novel domain statistics
($\bm{\mu}_{Novel}\in\mathbb{R}^{d}$ and
$\bm{\sigma}_{Novel}^{2}\in\mathbb{R}^{d}$ ) are different from all clients
and can be used in our subsequent feature hallucinating. The domain
hallucinating is motivated by MixUp Zhang et al. (2017), which shows that the
beta-distribution is an effective to generate weights for two samples during
the sampling of weights. Dirichlet distribution is a multivariate
generalization of the beta-distribution Shu et al. (2021), enabling us to
generate weights for multiple samples ($\geq 2$). Our setting assumes that
there are multiple domains, we thus adopt Dirichlet distribution in our
method.

Algorithm 1 The Process of Local Training.

Inputs: The total client number $N$. Total image number $N_{total}$. Local
client $i$, its corresponding training data
$\mathcal{X}_{i}=(x_{1},x_{2},...,x_{N_{i}})$ and labels
$\mathcal{Y}_{i}=(y_{1},y_{2},...,y_{N_{i}})$. Local iteration number $iter$.
Outputs: Local model $\phi_{i}$.

1:// Domain Hallucinating.

2:function DH($\bm{\mu}_{D_{1}}$, $\bm{\sigma}_{D_{1}}^{2}$, …,
$\bm{\mu}_{D_{N}}$, $\bm{\sigma}_{D_{N}}^{2}$)

3: Sample $\mathbf{w}$ from Dirichlet distribution;

4: Synthesize novel domain statistics with Eq. 3.3;

5: Return $\bm{\mu}_{Novel}$ and $\bm{\sigma}_{Novel}^{2}$;

6:end function

7:// Local Training for Client $i$.

8:function LocalTrain($i$, $\widehat{\bm{\mu}}_{1}$,
$\widehat{\bm{\sigma}}_{1}^{2}$, $\widetilde{\bm{\mu}}_{1}$,
$\widetilde{\bm{\sigma}}_{1}^{2}$, …, $\widehat{\bm{\mu}}_{N}$,
$\widehat{\bm{\sigma}}_{N}^{2}$, $\widetilde{\bm{\mu}}_{N}$,
$\widetilde{\bm{\sigma}}_{N}^{2}$)

9: for $k$ in $N$ do

10: Sample $\bm{\mu}_{D_{k}}$, $\bm{\sigma}_{D_{k}}^{2}$ from
$\mathcal{N}(\widehat{\bm{\mu}}_{i},\widehat{\bm{\sigma}}_{i}^{2})$ and
$\mathcal{N}(\widetilde{\bm{\mu}}_{i},\widetilde{\bm{\sigma}}_{i}^{2})$;

11: end for

12: ($\bm{\mu}_{Novel}$, $\bm{\sigma}_{Novel}^{2}$) $\leftarrow$
DH($\bm{\mu}_{D_{1}}$, $\bm{\sigma}_{D_{1}}^{2},...,\bm{\mu}_{D_{N}}$,
$\bm{\sigma}_{D_{N}}^{2}$);

13: for $iter\\_num$ in $iter$ do

14: Sample a batch of training data from $\mathcal{X}_{i}$ and
$\mathcal{Y}_{i}$;

15: Extract batch features $\mathbf{F}_{i}$ with $\phi_{i}$;

16: Transform $\mathbf{F}_{i}$ to $\mathbf{F}_{Novel}$ and
$\mathbf{F}_{k}(k\neq i)$ with Eq. 3.2;

17: Compute Eq. 3.6 to optimize $\phi_{i}$;

18: end for

19: Return $\phi_{i}$;

20:end function

Local Training with Synthesized Features. From Eq. 3.2, we observe that FH can
transform local features to another domain’s distribution through scaling and
shifting, which retains the pair-wise similarities of the original batch
features. Therefore, we can utilize triplet loss Hermans et al. (2017) to
optimize the local model with the synthesized novel feature and enhance the
generalization of local models (optimization loss is not limited to triplet
loss). Concretely, we sample a training batch from ($\mathcal{X}_{i}$,
$\mathcal{Y}_{i}$) of client $i$ with batch size $N_{b}$, then extract
features
$\mathbf{F}_{i}=\\{\mathbf{f}_{i,1},\mathbf{f}_{i,2},...,\mathbf{f}_{i,N_{b}}\\}$
with local model $\phi_{i}$. Each feature is then transformed to novel domain
and other decentralized domains with Eq. 3.2 to obtain
$\mathbf{F}_{Novel}=\\{\mathbf{f}_{Novel,1},\mathbf{f}_{Novel,2},...,\mathbf{f}_{Novel,N_{b}}\\}$
and
$\mathbf{F}_{k}=\\{\mathbf{f}_{k,1},\mathbf{f}_{k,2},...,\mathbf{f}_{k,N_{b}}\\}(k\neq
i)$. Thus, we optimize local model with the following triplet loss:

$L_{tri}(\mathbf{F})=\frac{1}{N_{b}}\sum_{n=1}^{N_{b}}\Big{[}||\mathbf{f}_{n}^{+}-\mathbf{f}_{n}||_{2}-||\mathbf{f}_{n}^{-}-\mathbf{f}_{n}||_{2}+m\Big{]}_{+},$
(3.4)

where $\mathbf{F}$ can be $\mathbf{F}_{Novel}$ or $\mathbf{F}_{k}(k\neq i)$.
$\mathbf{f}_{n}^{+}$ and $\mathbf{f}_{n}^{-}$ are hard positive and hard
negative sample for anchor $\mathbf{f}_{n}$ within the batch. $m$ is margin,
$N_{b}$ is the batch size, and $[\cdot]_{+}$ refers to $\max(\cdot,0)$.
Besides, we also use the original local features $\mathbf{F}_{i}$ for local
optimization, formulated as:

$L_{ori}(\mathbf{F}_{i},\mathcal{Y}_{i})=L_{tri}(\mathbf{F}_{i})+L_{ce}(FC(\mathbf{F}_{i}),\mathcal{Y}_{i}),$
(3.5)

where $FC$ is a fully connected classification layer, which transforms the
input features $\mathbf{F}_{i}$ to logits. The logits and their corresponding
ground-truth labels $\mathcal{Y}_{i}$ are used to compute cross entropy loss
$L_{ce}$.

The total loss for local training is formulated as:

$L_{local}(\mathbf{F}_{i},\mathcal{Y}_{i})=L_{ori}(\mathbf{F}_{i},\mathcal{Y}_{i})+\lambda\Big{[}L_{tri}(\mathbf{F}_{Novel})+\frac{1}{N-1}\sum_{k=1,k\neq
i}^{N}L_{tri}(\mathbf{F}_{k})\Big{]},$ (3.6)

where $\lambda$ is the balancing factor. The overall process of local training
is shown in Alg. 1.

###  3.3 Subsequent Learning

Client-to-server Updating. By running local training for all clients, we
obtain well-generalized local models. Both local models and DFS (i.e.,
$\widehat{\bm{\mu}}_{i}$, $\widehat{\bm{\sigma}}_{i}^{2}$,
$\widetilde{\bm{\mu}}_{i}$, $\widetilde{\bm{\sigma}}_{i}^{2}$ for the $i$-th
domain) will be uploaded to the server. The server-side DFS will be updated by
the local DFS to prepare for the next epoch of local training. Besides, the
updated server-side DFS ensure that all the DFS used for local training are
synchronous.

Server-side Aggregation. We aggregate all local models in server-side with
weighted average. The weight for each model is decided by the relative ratio
between the number of images in current client and all clients. In detail, the
global model $\phi$ is obtained by:

$\phi=\sum_{i=1}^{N}\frac{N_{i}}{N_{total}}\phi_{i},$ (3.7)

where $N_{total}=\sum_{i=1}^{N}N_{i}$ is the total number of images in all
clients.

Redistributing. The global model $\phi$, combined with server-side DFS (i.e.,
$\widehat{\bm{\mu}}_{1}$, $\widehat{\bm{\sigma}}_{1}^{2}$,
$\widetilde{\bm{\mu}}_{1}$, $\widetilde{\bm{\sigma}}_{1}^{2}$, …,
$\widehat{\bm{\mu}}_{N}$, $\widehat{\bm{\sigma}}_{N}^{2}$,
$\widetilde{\bm{\mu}}_{N}$, $\widetilde{\bm{\sigma}}_{N}^{2}$) are
redistributed to all clients for the next epoch of federated learning. The
aforementioned four steps are repeated for several epochs to obtain the final
global model for evaluation. The overall process of FedDG re-ID is elaborated
in Alg. 2.

Algorithm 2 The Overall Federated Learning.

Inputs: Decentralized $N$ training domains and their labels
($\\{\mathcal{X}_{1},\mathcal{Y}_{1}),...,(\mathcal{X}_{N},\mathcal{Y}_{N})\\}$).
Initialized local models $\phi_{1},\phi_{2},...,\phi_{N}$. Global training
epoch $epoch$, batch-size $N_{b}$..
Outputs: Well-generalized global model $\phi$.

1:// Initialize Server-side DFS.

2:for $i$ in $N$ do

3: Obtain DFS for client-$i$ (i.e., $\widehat{\bm{\mu}}_{i}$,
$\widehat{\bm{\sigma}}_{i}^{2}$, $\widetilde{\bm{\mu}}_{i}$,
$\widetilde{\bm{\sigma}}_{i}^{2}$) and upload them to server;

4:end for

5:for $m$ in $epoch$ do

6: // Step 1: Local Training.

7: for $k$ in $N$ do

8: // Use downloaded DFS.

9: $\phi_{k}\leftarrow$ LocalTrain($k$, $\widehat{\bm{\mu}}_{1}$,
$\widehat{\bm{\sigma}}_{1}^{2}$, $\widetilde{\bm{\mu}}_{1}$,
$\widetilde{\bm{\sigma}}_{1}^{2}$, …, $\widehat{\bm{\mu}}_{N}$,
$\widehat{\bm{\sigma}}_{N}^{2}$, $\widetilde{\bm{\mu}}_{N}$,
$\widetilde{\bm{\sigma}}_{N}^{2}$);

10: end for

11: // Step 2: Client-to-server Updating.

12: Send local models to the server;

13: Update server-side DFS;

14: // Step 3: Server-side Aggregation.

15: Obtain global model $\phi$ with Eq. 3.7;

16: // Step 4: Redistributing.

17: Redistribute $\phi$ and server-side DFS to local clients;

18:end for

19:Return $\phi$;

###  3.4 Discussion on Data Privacy Constraint

FedDG re-ID prohibits the transmission of images between different clients,
which limits the application of most DG Dai et al. (2021); Zhao et al. (2021);
Zhou et al. (2020); Guo et al. (2020) algorithms. Generally speaking, the
privacy constraint refers to no data transmission. However, recent works state
that sharing averaged images Yoon et al. (2021), domain-specific classifiers
Wu & Gong (2021b) or intermediate feature distributions Luo et al. (2021); Yao
et al. (2022) will not bring the leakage of data privacy since such weak
information cannot be used to recover the original images. For example, FedMix
Yoon et al. (2021) weakens the data-privacy constraint by using the averaged
images from other clients for optimization. Wu et al. Wu & Gong (2021b)
attempt to handle the federated learning problem by jointly using the
classifiers of each local model for optimization. Luo et al. Luo et al. (2021)
propose CCVR to alleviate the heterogeneity in federated learning by sharing
the feature statistics of each class for each domain. Despite their
effectiveness in closed-set classification problem, most of them are not
suitable in FedDG re-ID. This is because each domain has completely different
IDs in FedDG re-ID. In this paper, we follow Luo et al. (2021); Yao et al.
(2022) and share domain statics between clients. Different from them, our
method shares the global statistics in the pooling-5 feature space to
synthesize novel features for optimization with re-parameterization trick
Kingma & Welling (2013), offering a plausible way to handle the heterogeneity
in the open-set FedDG re-ID.

##  4 Experiments

###  4.1 Experiment Setup

Datasets. In our experiments, we use four re-ID benchmarks, including
Market-1501 (Market) Zheng et al. (2015), CUHK02 Li & Wang (2013), MSMT-17
(MSMT) Wei et al. (2018) and CUHK03 Li et al. (2014). Market contains $1,501$
IDs ($32,668$ images) taken by $6$ cameras, of which $750$ IDs ($12,936$
images) are used for training while the other $751$ IDs for evaluation. CUHK02
has $7,264$ bounding-boxes manually cropped from $1,816$ pedestrians. MSMT
covers $126,441$ images from $4,101$ IDs captured by $15$ cameras. CUHK03
collects $28,193$ pedestrian images from $1,467$ IDs, where $1,367$ IDs are
used for training and the rest for evaluation. Note that, DukeMTMC-ReID has
been withdrawn by creators and thus is not used in this work.

Evaluation Protocol. The overall performance is evaluated with the mean
average precision (mAP) and rank-1 accuracy. We choose the final global model
for evaluation.

Implementation Details. We conduct all our experiments with ResNet-50 He et
al. (2016). During local training, we set learning rate as $1\times 10^{-3}$,
batch size $N_{b}=64$, local training iteration number $iter=200$, and the
total server-client collaborative training epochs $epoch=40$. The learning
rate will be multiplied with $0.5$ at the $20$-th and $30$-th epoch of
training. The margin in Eq. 3.4 is set to $0.5$, $\lambda$ in Eq. 3.6 is set
to $5$. We use random crop, random flip and random erasing Zhong et al. (2020)
as data augmentation during optimization and SGD is chosen as the optimizer.
We set $\bm{\alpha}$ to all-one vector. All images are resized to $256\times
128$. During testing, we extract $2048$-dim pooling-5 features for retrieval.

Table 1: Comparison with state-of-the-arts. We compare our methods with FedAvg McMahan et al. (2017), FedPav Zhuang et al. (2020), FedReID Wu & Gong (2021a) and SNR Jin et al. (2020). M: Market-1501, C2: CUHK02, C3: CUHK03, MS: MSMT-17. Sources | Methods | Target: C2 | Sources | Methods | Target: M | Sources | Methods | Target: C3
---|---|---|---|---|---|---|---|---
mAP | rank-1 | mAP | rank-1 | mAP | rank-1
MS | / | 43.8 | 41.8 | MS | / | 23.3 | 47.5 | MS | / | 18.0 | 18.5
C3 | 49.8 | 45.8 | C3 | 13.2 | 31.1 | C2 | 21.6 | 22.5
M | 42.4 | 37.9 | C2 | 18.9 | 41.2 | M | 10.2 | 11.2
MS+ C3+M | FedAvg | 61.3 | 58.6 |  MS+ C3+C2 | FedAvg | 24.6 | 48.3 |  MS+ C2+M | FedAvg | 20.3 | 22.7
FedPav | 62.6 | 59.8 | FedPav | 25.4 | 49.4 | FedPav | 22.5 | 24.3
FedReID | 64.3 | 61.2 | FedReID | 25.5 | 49.6 | FedReID | 22.4 | 23.0
SNR | 67.7 | 65.1 | SNR | 28.3 | 53.2 | SNR | 26.0 | 29.1
DFH | 71.7 | 69.5 | DFH | 31.3 | 56.5 | DFH | 27.2 | 30.5
SNR+DFH | 72.4 | 70.2 | SNR+DFH | 33.2 | 58.8 | SNR+DFH | 31.1 | 33.5

###  4.2 Comparison with State-of-the-arts

We alternately use CUHK02, Market, and CUHK03 as the target client and the
other datasets as decentralized source clients for federated learning. FedAvg
McMahan et al. (2017), FedPav Zhuang et al. (2020), FedReID Wu & Gong (2021a)
and SNR Jin et al. (2020) are the state-of-the-arts used in our experiments
for comparison. FedAvg requires the classifier in client-side and server-side
have the same number of outputs. Therefore, it will collect the total number
of IDs in all source clients as the output of classifiers for all models.
Differently, FedPav allows local models to have classifiers with a different
number of outputs, and only transmits the feature extractor of models for
aggregation. FedReID improves the accuracies of FedDG re-ID by using knowledge
distillation between the aggregated model and the local model. SNR is a single
domain generalization method, which modifies the network structure of
ResNet-50 by adding SNR module for feature disentanglement and better
training. We use SNR under the federated constraint for fair comparison. We
report all results in Tab. 1 and obtain the following conclusions: (1) Our
method achieves the state-of-the-art performance on FedDG re-ID. Concretely,
we achieve mAP=71.7% and rank-1=69.5% when testing on CUHK02, mAP=31.3% and
rank-1=56.5% when testing on Market, and mAP=27.2% and rank-1=30.5% when
testing on CUHK03. These results outperform FedPav by 9.1%, 5.9%, and 4.7% mAP
scores on the three benchmarks respectively. Besides, our method also
outperforms SNR Jin et al. (2020) by 4.0%, 3.0%, and 1.2% mAP scores on the
three datasets without changing the network’s structure, indicating the
effectiveness of our model. (2) Federated learning with multiple domains
achieves better generalization results than using a single domain. We take the
results on CUHK02 for example, the models trained with MSMT, CUHK03, and
Market respectively achieve 43.8%, 49.8%, and 42.4% mAP scores on the test set
of CUHK02, which are far less than the results of multi-source FedDG re-ID
methods like FedPav and ours. The results demonstrate that although federated
learning does not combine all training data for optimization, it can still
achieve better generalization than models with one domain for training. (3)
Our method can be flexibly integrated into some DG methods like SNR Jin et al.
(2020), and help them improve federated re-ID accuracies. As shown in Tab. 1,
SNR achieves 28.3% mAP score on Market1501. After adding our DFH into the
training process, the mAP is increased by 4.9%, which indicates the
effectiveness of DFH on SNR under federated constraint. Similar results can
also be found in the evaluation results of other benchmarks.

Table 2: Ablation study of our method. FH: Feature Hallucinating, DM: Dirichlet MixUp Shu et al. (2021), FM: Feature-level MixUp Zhang et al. (2017), DH: Our Domain Hallucinating. No. | Attributes |  MS+C2 +C3$\rightarrow$M |  MS+M +C2$\rightarrow$C3
---|---|---|---
FH | DM | FM | DH | mAP | rank-1 | mAP | rank-1
1 | $\times$ | $\times$ | $\times$ | $\times$ | 25.4 | 49.4 | 22.5 | 24.3
2 | $\checkmark$ | $\times$ | $\times$ | $\times$ | 28.4 | 51.7 | 24.1 | 25.7
3 | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | 29.1 | 52.2 | 25.6 | 28.2
4 | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | 27.8 | 50.5 | 23.6 | 25.3
5 | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | 31.3 | 56.5 | 27.2 | 30.5

###  4.3 Ablation Study

We evaluate the effectiveness of DH, FH, and compare our DFH with other
feature-level MixUp methods under the federated learning scenario. In detail,
we conduct experiments that (1) do not use DH and FH; (2) do not synthesize
novel domain with DH, but use FH to enable local model to see other clients’
feature distributions; (3) use Dirichlet-MixUp Shu et al. (2021) on
transformed features to generate novel features; (4) use Feature-level MixUp
Zhang et al. (2017) to generate novel features and sample weights from beta
distribution; (5) use DFH; All results are reported in Tab. 2.

The Effectiveness of Feature Hallucinating. From No. 1 and No. 2 of Tab. 2, we
notice that FH enables the local model to see the feature distributions of
other domains, leading to the improvement of re-ID accuracies. In detail, for
experiments utilizing Market as the target domain, we improve the mAP from
25.4% to 28.4%. Similar results can also be found on experiments when
evaluated on CUHK03, which elaborate the effectiveness of FH for federated
learning.

The Effectiveness of Domain Hallucinating. From No. 2 and No. 5 of Tab. 2, we
conclude that synthesizing novel domain statistics is also beneficial to
improving the generalization of the re-ID model. After applying both FH and DH
to the FedDG re-ID, the mAP scores are further improved from 28.4% to 31.3% on
Market. We can also receive similar results on CUHK03, which suggest the
importance of synthesizing novel domain statistics during federated learning.

Comparison between DH and Other Feature-level MixUp Methods. To clarify the
differences between our DH and other feature MixUp methods, we replace the DH
with another two methods: Dirichlet-Mixup Shu et al. (2021) and Feature-level
MixUp Zhang et al. (2017). The former approach samples weights from Dirichlet
distribution and generates novel features with the hallucinated features,
which is plausible when there are more than two features to mixup. The latter
one follows the original MixUp and samples weights from beta-distribution. By
comparing No. 3 and No. 4, we conclude that for FedDG re-ID problem with
multiple domains, it is better to sample weights from Dirichlet distribution.
Moreover, by comparing No. 3, No. 5, we notice that our DH achieves better
results than Dirichlet-Mixup. Therefore, our DFH can bring more diverse local
features during local training and can effectively improve local
generalization.

Table 3: Effectiveness of training with less source clients. Methods |  MS+C3$\rightarrow$M |  MS+C2$\rightarrow$M |  MS+M$\rightarrow$C3 |  MS+C2$\rightarrow$C3
---|---|---|---|---
mAP | rank-1 | mAP | rank-1 | mAP | rank-1 | mAP | rank-1
FedPav | 24.6 | 45.3 | 24.3 | 47.8 | 18.9 | 19.5 | 22.8 | 26.5
SNR | 26.3 | 49.7 | 28.2 | 53.6 | 20.9 | 22.0 | 23.2 | 22.4
Ours | 29.8 | 54.9 | 29.9 | 55.1 | 23.5 | 25.2 | 25.0 | 28.7

###  4.4 Sensitivity Analysis

Sensitivity to the Number of Clients. Previous experiments are all conducted
with three decentralized source domains. To explore the effectiveness of our
method when deployed on fewer clients, we report experiments with only two
source decentralized clients in Tab. 3. By observing the results, we notice
that our approach can improve the generalization of re-ID models with less
(two) decentralized domains. Concretely, DFH achieves higher re-ID accuracies
than FedPav Zhuang et al. (2020) and SNR Jin et al. (2020) on
“MS+C3$\rightarrow$M", “MS+C2$\rightarrow$M", “MS+M$\rightarrow$C3" and
“MS+C2$\rightarrow$C3". For the comparison with FedPav, DFH can consistently
improve the re-ID accuracies for about 3-5% mAP scores. The experiments of SNR
report around 2% improvements on mAP, indicating that our method is also
effective when there are fewer source domains.

Sensitivity to $\bm{\alpha}$. We explore the effect of using different
$\bm{\alpha}$ when sampling domain weight from the Dirichlet distribution
during DH. Specifically, $\bm{\alpha}$ is a $N$ dimensional vector where each
element is related to a domain. We change one of the elements in $\bm{\alpha}$
from 1 to 2 and remain the others fixed for “MS+M+C2$\rightarrow$C3" and
“MS+C3+C2$\rightarrow$M" to see how it affects the final performance. The
results are reported in Tab. 4. We find that changing one of the elements in
$\bm{\alpha}$ does not have a significant effect on the final results. In
real-world applications, we recommend setting $\bm{\alpha}$ to all-one vector
for simplicity.

Table 4: Sensitivity analysis of $\bm{\alpha}$.  $\bm{\alpha}$ | Target: C3 | $\bm{\alpha}$ | Target: M
---|---|---|---
MS | M | C2 | mAP | rank-1 | MS | C3 | C2 | mAP | rank-1
1 | 1 | 1 | 27.2 | 30.5 | 1 | 1 | 1 | 31.3 | 56.5
1 | 2 | 1 | 26.9 | 25.5 | 1 | 2 | 1 | 30.7 | 55.3
2 | 1 | 1 | 25.8 | 28.7 | 2 | 1 | 1 | 29.8 | 54.6
1 | 1 | 2 | 26.4 | 30.2 | 1 | 1 | 2 | 30.2 | 54.8

Sensitivity to $\lambda$. We investigate the effect of $\lambda$ by changing
its value from 2 to 6 with a step size of 1 and apply them in two experiments
(“MS+C2+M$\rightarrow$C3" and “MS+C2+C3$\rightarrow$M"). A higher $\lambda$
means paying more attention to the transformed features during optimization.
From Fig. 3, we can draw a conclusion that focusing on the transformed
features during optimization and using larger $\lambda$ is beneficial to
improving the generalization of models due to the novel information included
in the transformed features. But, the improvement of re-ID accuracies begins
to become small when $\lambda>5$. Therefore, we choose $\lambda=5$ for FedDG
re-ID tasks.

Figure 3: Sensitivity analysis of $\lambda$. We change the $\lambda$ from 2 to
6 with a step size of 1 to see the importance of using transformed features
during optimization.

###  4.5 Loss Function

. We want to remind the readers that the hallucinated features can not be
optimized with cross-entropy loss and local classifiers. As shown in Tab. 5,
the results of using cross-entropy loss for optimization is far less than the
triplet version. That is because FH generates features through affine
transformation, which will change the neighbors of the original ones. Since
the transformed features and the original ones have different neighbors,
making them to be similar with cross-entropy loss will back-propagate false
information to the network and reduce the discrimination of the model.

Table 5: The Necessity of Using Triplet Loss. Loss Function |  MS+M+C2$\rightarrow$C3 |  MS+C2+C3$\rightarrow$M
---|---|---
mAP | rank-1 | mAP | rank-1
Cross-Entropy | 20.4 | 23.8 | 26.3 | 50.6
Triplet | 27.2 | 30.5 | 31.3 | 56.5

###  4.6 Visualization

Convergence of DFH. We evaluate the mAP and rank-1 scores of the averaged
model every 5 epochs to check the convergence of our DFH in the whole 40
epochs. As shown in Fig. 4, the mAP and rank-1 scores become steady at the
35-th epoch in “MS+C2+M$\rightarrow$C3", while “MS+C2+C3$\rightarrow$M"
converged at the 10-th epoch.

Figure 4: Convergence of DFH.

Results of Restored Images with D2D. Transferring the raw features to
different clients may cause some information leakage because researchers have
developed algorithms like D2D Yin et al. (2020) to restore the original images
from the frozen model. It, therefore, raises questions whether sharing the
domain-level feature statistics can preserve data privacy. We conduct
experiments to restore images with the trained Market model and D2D.

The shared DFS are parameters of two Gaussians, which should be first sampled
to obtain the domain-level mean and variance. The receiver cannot even know
the number of IDs from DFS and will cause some problems for D2D to recover the
raw images. Therefore, we use IFS to restore the raw images and the mean
vectors for each ID in the Market are used to initialize the classifier during
the optimization. If IFS can successfully protect data privacy, the DFS will
also satisfy privacy constraint. The results are shown in Fig. 5. It is
obvious that the restored images are blurry and we cannot recognize the object
in them. Based on the results, we claim that transferring DFS will not violate
data privacy constraint.

Figure 5: Restored Images with D2D.

Visualization of the Hallucinated Features. To gain a better understanding of
how DFH and other feature mixup methods work in FedDG re-ID task, we conduct a
visualization experiment, which chooses MSMT, CUHK02, and Market as the
decentralized domains. We collect images from randomly selected three
identities for each domain and extract their features with the final global
model. Then, we respectively use Dirichlet-MixUp Shu et al. (2021), Feature
MixUp Zhang et al. (2017), and our DFH to generate novel features from MSMT.
All features are visualized with _t_ -SNE Van der Maaten & Hinton (2008) in
Fig. 6. We use different colors to denote different domains while different
shapes to indicate identities. In Fig. 6, we have three observations. (1) All
three methods try to improve the model’s generalization by synthesizing novel
features in the transition among different domains. Specifically, the
transformed features are located in the gap of three source domains, which
enables the local model to see more diverse samples and avoids overfitting by
smoothing the transition between different domains. (2) We should use
Dirichlet-MixUp for generalization tasks with multiple domains. As mentioned
in Shu et al. (2021), the classical MixUp only mix features from two domains
with weights sampled from beta-distribution, which can not generate diverse
features for problems with multiple domains. The comparison between Fig. 6-(a)
and (b) also demonstrates the necessity of using Dirichlet-MixUp. (3) Our DFH
can further improve the generalization of models by enlarging the intra-class
variations. In Fig. 6-(a), the transformed features nearly retain the same
pair-wise similarities of the original features, while our DFH in Fig. 6-(c)
can generate more diverse features. This is because DFH generates new features
by applying affine transformation with synthesized novel feature statistics,
which avoid applying MixUp in feature level and improve the diversity of novel
features.

Figure 6: Visualization of novel features generated by FH+Dir-MixUp,
FH+Feature-MixUp, and DFH. (a): FH+Dir-MixUp, which retains nearly the same
pair-wise similarities of local features. (b): FH+Dir-MixUp, which can only
interpolate the transition between each two domains. (c): Our DFH.

##  5 Conclusion

In this paper, we share domain-level feature statistics (DFS) of different
clients and propose domain & feature hallucinating (DFH) to improve the
generalization of FedDG re-ID. During the local training of each client, we
estimate the feature distribution of local clients with the shared DFS. Then,
we synthesize novel domain statistics by interpolating shared DFS with
randomly sampled domain weights. The novel domain statistics are utilized by
our feature hallucinating to transform local features to novel distributions,
enabling the local model to see as diverse samples as possible and improving
the generalization of local and global models. Extensive experiments
demonstrate the effectiveness of our method.

## References

  * Ahmed et al. (2021) Sk Miraj Ahmed, Dripta S Raychaudhuri, Sujoy Paul, Samet Oymak, and Amit K Roy-Chowdhury.  Unsupervised multi-source domain adaptation without access to source data.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021.
  * Chattopadhyay et al. (2020) Prithvijit Chattopadhyay, Yogesh Balaji, and Judy Hoffman.  Learning to balance specificity and invariance for in and out of domain generalization.  In _European Conference on Computer Vision_ , 2020.
  * Dai et al. (2021) Yongxing Dai, Xiaotong Li, Jun Liu, Zekun Tong, and Ling-Yu Duan.  Generalizable person re-identification with relevance-aware mixture of experts.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021.
  * Feng et al. (2020) Hao-Zhe Feng, Zhaoyang You, Minghao Chen, Tianye Zhang, Minfeng Zhu, Fei Wu, Chao Wu, and Wei Chen.  Kd3a: Unsupervised multi-source decentralized domain adaptation via knowledge distillation.  In _International Conference on Machine Learning_ , 2020.
  * Ge et al. (2020) Yixiao Ge, Feng Zhu, Dapeng Chen, Rui Zhao, and Hongsheng Li.  Self-paced contrastive learning with hybrid memory for domain adaptive object re-id.  In _Advances in Neural Information Processing Systems_ , 2020.
  * Guo et al. (2020) Jianzhu Guo, Xiangyu Zhu, Chenxu Zhao, Dong Cao, Zhen Lei, and Stan Z Li.  Learning meta face recognition in unseen domains.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2016.
  * Hermans et al. (2017) Alexander Hermans, Lucas Beyer, and Bastian Leibe.  In defense of the triplet loss for person re-identification.  _arXiv preprint arXiv:1703.07737_ , 2017.
  * Ioffe & Szegedy (2015) Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In _International Conference on Machine Learning_ , pp. 448–456, 2015.
  * Jin et al. (2020) Xin Jin, Cuiling Lan, Wenjun Zeng, Zhibo Chen, and Li Zhang.  Style normalization and restitution for generalizable person re-identification.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020.
  * Kingma & Welling (2013) Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_ , 2013.
  * Li et al. (2018) Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales.  Learning to generalize: Meta-learning for domain generalization.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2018.
  * Li et al. (2021) Qinbin Li, Bingsheng He, and Dawn Song.  Model-contrastive federated learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021.
  * Li et al. (2020) Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith.  Federated optimization in heterogeneous networks.  In _Proceedings of Machine Learning and Systems_ , 2020.
  * Li & Wang (2013) Wei Li and Xiaogang Wang.  Locally aligned feature transforms across views.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2013.
  * Li et al. (2014) Wei Li, Rui Zhao, Tong Xiao, and Xiaogang Wang.  Deepreid: Deep filter pairing neural network for person re-identification.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2014.
  * Liao & Shao (2020) Shengcai Liao and Ling Shao.  Interpretable and generalizable person re-identification with query-adaptive convolution and temporal lifting.  In _European Conference on Computer Vision_ , 2020.
  * Liu et al. (2021) Quande Liu, Cheng Chen, Jing Qin, Qi Dou, and Pheng-Ann Heng.  Feddg: Federated domain generalization on medical image segmentation via episodic learning in continuous frequency space.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021.
  * Luo et al. (2021) Mi Luo, Fei Chen, Dapeng Hu, Yifan Zhang, Jian Liang, and Jiashi Feng.  No fear of heterogeneity: Classifier calibration for federated learning with non-iid data.  In _Advances in Neural Information Processing Systems_ , 2021.
  * McMahan et al. (2017) Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas.  Communication-efficient learning of deep networks from decentralized data.  In _Artificial intelligence and statistics_ , 2017.
  * Muandet et al. (2013) Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf.  Domain generalization via invariant feature representation.  In _International Conference on Machine Learning_ , 2013.
  * Panareda Busto & Gall (2017) Pau Panareda Busto and Juergen Gall.  Open set domain adaptation.  In _Proceedings of the IEEE International Conference on Computer Vision_ , 2017.
  * Qiao et al. (2020) Fengchun Qiao, Long Zhao, and Xi Peng.  Learning to learn single domain generalization.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020.
  * Shu et al. (2021) Yang Shu, Zhangjie Cao, Chenyu Wang, Jianmin Wang, and Mingsheng Long.  Open domain generalization with domain-augmented meta-learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021.
  * Song et al. (2019) Jifei Song, Yongxin Yang, Yi-Zhe Song, Tao Xiang, and Timothy M Hospedales.  Generalizable person re-identification by domain-invariant mapping network.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019.
  * Sun et al. (2018) Yifan Sun, Liang Zheng, Yi Yang, Qi Tian, and Shengjin Wang.  Beyond part models: Person retrieval with refined part pooling (and a strong convolutional baseline).  In _European Conference on Computer Vision_ , 2018.
  * Van der Maaten & Hinton (2008) Laurens Van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  _Journal of machine learning research_ , 2008.
  * Wang et al. (2018) Guanshuo Wang, Yufeng Yuan, Xiong Chen, Jiwei Li, and Xi Zhou.  Learning discriminative features with multiple granularities for person re-identification.  In _Proceedings of the ACM International Conference on Multimedia_ , 2018.
  * Wei et al. (2018) Longhui Wei, Shiliang Zhang, Wen Gao, and Qi Tian.  Person transfer gan to bridge domain gap for person re-identification.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2018.
  * Wu & Gong (2021a) Guile Wu and Shaogang Gong.  Decentralised learning from independent multi-domain labels for person re-identification.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2021a.
  * Wu & Gong (2021b) Guile Wu and Shaogang Gong.  Collaborative optimization and aggregation for decentralized domain generalization and adaptation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021b.
  * Yang et al. (2021) Fengxiang Yang, Zhun Zhong, Zhiming Luo, Yuanzheng Cai, Yaojin Lin, Shaozi Li, and Nicu Sebe.  Joint noise-tolerant learning and meta camera shift adaptation for unsupervised person re-identification.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021.
  * Yao et al. (2022) Chun-Han Yao, Boqing Gong, Hang Qi, Yin Cui, Yukun Zhu, and Ming-Hsuan Yang.  Federated multi-target domain adaptation.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2022.
  * Ye et al. (2021) Mang Ye, Jianbing Shen, Gaojie Lin, Tao Xiang, Ling Shao, and Steven C. H. Hoi.  Deep learning for person re-identification: A survey and outlook.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2021.
  * Yin et al. (2020) Hongxu Yin, Pavlo Molchanov, Jose M Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K Jha, and Jan Kautz.  Dreaming to distill: Data-free knowledge transfer via deepinversion.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020.
  * Yoon et al. (2021) Tehrim Yoon, Sumin Shin, Sung Ju Hwang, and Eunho Yang.  Fedmix: Approximation of mixup under mean augmented federated learning.  In _International Conference on Learning Representation_ , 2021.
  * Zhang et al. (2017) Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz.  Mixup: Beyond empirical risk minimization.  In _International Conference on Machine Learning_ , 2017.
  * Zhao et al. (2018) Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra.  Federated learning with non-iid data.  _arXiv preprint arXiv:1806.00582_ , 2018.
  * Zhao et al. (2021) Yuyang Zhao, Zhun Zhong, Fengxiang Yang, Zhiming Luo, Yaojin Lin, Shaozi Li, and Nicu Sebe.  Learning to generalize unseen domains via memory-based multi-source meta-learning for person re-identification.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021.
  * Zheng et al. (2021) Kecheng Zheng, Wu Liu, Lingxiao He, Tao Mei, Jiebo Luo, and Zheng-Jun Zha.  Group-aware label transfer for domain adaptive person re-identification.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021.
  * Zheng et al. (2015) Liang Zheng, Liyue Shen, Lu Tian, Shengjin Wang, Jingdong Wang, and Qi Tian.  Scalable person re-identification: A benchmark.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2015.
  * Zhong et al. (2019) Zhun Zhong, Liang Zheng, Zhiming Luo, Shaozi Li, and Yi Yang.  Invariance matters: Exemplar memory for domain adaptive person re-identification.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019.
  * Zhong et al. (2020) Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang.  Random erasing data augmentation.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2020.
  * Zhou et al. (2020) Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang.  Learning to generate novel domains for domain generalization.  In _European Conference on Computer Vision_ , 2020.
  * Zhuang et al. (2020) Weiming Zhuang, Yonggang Wen, Xuesen Zhang, Xin Gan, Daiying Yin, Dongzhan Zhou, Shuai Zhang, and Shuai Yi.  Performance optimization of federated person re-identification via benchmark analysis.  In _Proceedings of the ACM International Conference on Multimedia_ , 2020.


# Slot Filling for Biomedical Information Extraction

Yannis Papanikolaou, Marlene Staib, Justin Grace and Francine Bennett
Healx
Cambridge
UK
(yannis.papanikolaou, marlene.staib, justin.grace<EMAIL_ADDRESS>
###### Abstract

Information Extraction (IE) from text refers to the task of extracting
structured knowledge from unstructured text. The task typically consists of a
series of sub-tasks such as Named Entity Recognition and Relation Extraction.
Sourcing entity and relation type specific training data is a major bottleneck
in domains with limited resources such as biomedicine. In this work we present
a slot filling approach to the task of biomedical IE, effectively replacing
the need for entity and relation-specific training data, allowing us to deal
with zero-shot settings. We follow the recently proposed paradigm of coupling
a Tranformer-based bi-encoder, Dense Passage Retrieval, with a Transformer-
based reading comprehension model to extract relations from biomedical text.
We assemble a biomedical slot filling dataset for both retrieval and reading
comprehension and conduct a series of experiments demonstrating that our
approach outperforms a number of simpler baselines. We also evaluate our
approach end-to-end for standard as well as zero-shot settings. Our work
provides a fresh perspective on how to solve biomedical IE tasks, in the
absence of relevant training data. Our code, models and datasets are available
at https://github.com/ypapanik/biomedical-slot-filling.

##  1 Introduction

In Information Extraction (IE) we are interested in extracting structured
knowledge from unstructured text. This structured knowledge takes most usually
the form of directed binary relations between entities, in other words triples
of the form head - relation - tail, which can then be used to populate a
Knowledge Base or a Knowledge Graph with factual information.

The standard approach to perform IE relies on a cascade of Natural Language
Processing (NLP) models. First, Named Entity Recognition (NER) is employed to
find and extract entities of interest, subsequently Entity Linking (EL) to
link the extracted entities to Knowledge Base identifiers and finally Relation
Extraction (RE) to identify existing relations between entities.

These individual sub-tasks tasks have attracted a great deal of attention in
recent years with methods and datasets fuelling further research Verga et al.
(2018); Zeng et al. (2014, 2015); Lin et al. (2016). IE is largely regarded as
a main facilitator of structured data reasoning, such as Knowledge Base
Completion.

###  1.1 Standard Information Extraction vs Slot Filling

A major bottleneck in the above approach is that all modules (NER, EL, RE)
need training data specific to the entity or relation types that we are
interested in extracting. For instance, a NER model recognizing diseases needs
training data annotated with the entity type $disease$ and so forth. The
biomedical domain is particularly affected by these limitations, given the
vast variety of entity and relation types which are commonly of interest.
Additionally, sourcing training data for each sub-task and type is expensive
and challenging, requiring subject matter experts. For reference, the UMLS
ontology contains 125 semantic (entity) types and 54 relation types.

An alternative approach to standard IE is slot filling. The way IE is
conceptualized in slot filling is highly reminiscent of open domain question
answering (QA): for a given head-relation query the retriever returns a set of
relevant passages, which are then fed to a reader model that then extracts a
matching tail entity, the answer. By following such an approach, we can deal
with zero-shot settings since, unlike standard IE, we are not seeking to
recognize specific entity types or extract specific relation types, but rather
do machine reading comprehension, that is, extract answers in response to
queries. Importantly, this approach extends to relation types that were unseen
during training, effectively reducing the need for re-training and re-
deployment of a model deployed into production.

Furthermore, standard IE requires processing of every single sentence of the
given corpus through its different modules (NER, EL, RE). In contrast, the
computational cost of slot filling is much smaller as it performs retrieval
and reading comprehension on far fewer queries to extract relations. As an
example, Hetionet Himmelstein et al. (2017) contains around 2.25M relations,
but they can be formulated in around 46k distinct queries, of the form head-
relation111In other words, if we were trying to build a KB from biomedical
text that would contain these 2.25M relations, we would require to perform
around 46k queries on our index to retrieve relevant documents..

As a final point we summarize below how the two approaches would materialize
in a production setting, to make their differences more apparent. We note that
standard IE might involve additional tasks, such as coreference resolution
(which we do not describe here for simplicity):

Standard IE:

  * •

For each sentence, recognise entities with NER model.

  * •

For each recognised entity, link to an entity identifier from a Knowledge
Base, discarding entries that cannot be linked.

  * •

For each sentence that contains more than one recognized entity, extract
relations between the entities with a RE model.

  * •

Aggregate relations per sentence, resolving potential conflicts.

Slot filling:

  * •

For each entity in the Knowledge Base and each possible relation type,
consider all possible head - relation pairs and construct the relevant
queries, in a form head - relation222With this formulation a head and a tail
can be used interchangeably, by just changing the relation type, e.g. a _drug-
treats-disease relation_ can also be cast to _disease-is treated by-drug_
without additional training data..

  * •

For each query, retrieve the top k relevant documents with a retriever model.

  * •

For each query-retrieved document pair, perform reading comprehension,
extracting zero, one or multiple answers, i.e., relation tails.

  * •

For each answer, link to an entity identifier from a Knowledge Base,
discarding entries that cannot be linked.

###  1.2 Slot Filling: General vs Biomedical Domain

Although similar in most aspects, slot filling in the general domain against
slot filling in the biomedical and more broadly the scientific domain differ
in a few key ways. The first lies in the link between relations and entities.
In the general domain, a specific relation type will often imply a specific
entity type as well, whereas this rarely holds in biomedical literature.
Consider for example a relation child-of in the general domain, where we
expect both head and tail of the relation to be entities of type person, as
opposed to a relation (up)regulate in biomedicine where the head might be gene
or drug equivalently. These nuances in the language used render the task of
slot filling more challenging in biomedicine.

Another, perhaps more critical aspect relates to retrieval and more
specifically how we build and evaluate on a retrieval dataset. In the general
domain, a slot filling query, or more broadly a question within the QA
framework, will most often have a unique answer333We are implicitly referring
only to factoid queries here which is the case for most open domain QA
datasets; queries of list type would have multiple answers in any case.,
whereas this rarely holds when mining the biomedical literature. For instance,
consider the examples illustrated in Table 1 coming from two well established
general domain benchmarks, Natural Questions Kwiatkowski et al. (2019) and
zsRE Levy et al. (2017) against two datasets from the biomedical domain,
BioASQ Tsatsaronis et al. (2015) and our slot filling dataset (BioSF).

Dataset | Query | Answer(s)
---|---|---
NQ | when is the next deadpool movie being released | May 18, 2018
NQ | what was the first capital city of australia | Melbourne
zsRE | Elmer George [SEP] spouse | Mari Hulman George
zsRE | Boone River [SEP] mouth of the watercourse | Des Moines River
BioASQ | What are the main indications of lacosamide? | ’epilepsy’, ’analgesic’
BioASQ | Which metabolite activates AtxA? | ’CO2’, ’bicarbonate’
BioSF | sildenafil [SEP] regulator | ’L765A’, ’F786A’, ’F820A’
BioSF | Amprenavir [SEP] interacts with | ’rifabutin’, ’ritonavir’
Table 1:  Examples of queries for general domain benchmarks (NQ, zsRE) vs
biomedical domain benchmarks (BioASQ, BioSF). Queries in the biomedical domain
usually involve multiple valid answers, as opposed to the general domain.

This difference has a number of implications both for training and evaluation.
With respect to training, one of the major successes of neural-based retrieval
methods has been attributed to being able to present the model with hard
negatives, i.e., examples were a previous version of the retriever (or a
simpler statistical retriever) have failed. When, for example, we have a
query-answer pair that mentions that Barack Obama’s wife is Michelle Obama,
and the model returns a passage that does not include the string "Michelle
Obama", we can relatively safely consider this a false positive and use that
passage as a hard negative. This helps the algorithm correct mistakes and
improve. In biomedicine on the other hand, if we have an example stating that
sildenafil regulates a mutation L765A, we cannot be sure that all alternative
strings extracted by the model are true negatives, as there may be other valid
answers that we cannot validate due to our Knowledge Base being incomplete.
This compromises our ability to build gold standard training data and we are
presented with a situation similar to the one encountered in distant
supervision, where unlabeled examples are considered as negatives but might be
positives in some cases. Practically, this leads to a noisy training set which
may reduce model accuracy.

During evaluation of a biomedical retriever, we encounter the same problem, in
the sense that we might obtain misleading low performance since unknown
correct passages might rank higher than the known correct ones. This leads to
an imperfect, i.e., "silver" quality, evaluation regime making it hard to
compare approaches and models.

In this work we aim to address the challenges mentioned in the two previous
subsections. Specifically,

  * •

We provide a short review of the relevant work in Section 2.

  * •

We contribute a novel formulation of biomedical IE as a slot filling task, to
address few-shot or zero-shot settings in Section 3.

  * •

We release a new benchmark for biomedical slot filling, dubbed BioSF which we
describe in Section 4.

  * •

We train a biomedical dense passage retriever along with a biomedical reading
comprehension model for slot filling, using BioSF. We provide the models
publicly.

  * •

We present an evaluation of our approach over several baselines on BioSF,
which we are able to outperform by a large margin, in Section 5.

##  2 Related Work

Recent years have witnessed a series of significant advances in the field of
QA, primarily owing to the Transformer architecture Vaswani et al. (2017) and
the BERT self-supervised pre-training paradigm Devlin et al. (2019). These
advances, both in terms of methods Chen et al. (2017); Lin et al. (2019); Guu
et al. (2020); Lewis et al. (2020b) and datasets Kwiatkowski et al. (2019);
Yang et al. (2018), motivated researchers to formulate a series of different
NLP tasks as open domain QA, including entity linking or relation extraction
Levy et al. (2017); Petroni et al. (2021). In this work we follow this
paradigm by formulating biomedical IE as a slot-filling task.

In open domain QA, given a query, a retrieval module first retrieves relevant
documents from the knowledge source (such as Wikipedia). A reading
comprehension module is then used to extract a span from the relevant
documents, the answer. The retrieval step was, up to very recently, dominated
by statistical-based approaches, namely BM25 or tf-idf Chen et al. (2017).
ORQA Lee et al. (2019b) and REALM Guu et al. (2020) have been the first neural
based methods to clearly outperform statistical based retrieval, although they
required expensive language model pre-training. Dense Passage Retrieval (DPR)
Karpukhin et al. (2020) has improved upon these methods by employing BERT-
based encoders, one for the queries and one for passages. These are jointly
optimized during training to classify passages as relevant versus irrelevant.
This approach has proved superior to other neural based approaches and has
quickly become the preferred method for open domain QA in subsequent work
Lewis et al. (2020b); Izacard and Grave (2021); Maillard et al. (2021).

Among the subsequent works, Retrieval Augmented Generation Lewis et al.
(2020b) employs an architecture based on DPR and BART Lewis et al. (2020a)
that is optimized end to end during finetuning, to retrieve relevant documents
and generate answers to queries. Fusion-in-decoder Izacard and Grave (2021)
employs DPR or BM25 as retrievers coupled with a T5 language model, to
generate answers by attending at multiple passages simultaneously. For
simplicity, we are not considering these approaches in this work, leaving
their implementation for the biomedical domain for future work.

In an effort to fuel further research on this field, Petroni et al. (2021)
introduced KILT, a new benchmark of knowledge intensive tasks, which contains
among others two slot filling datasets, zero-shot RE which was first presented
in Levy et al. (2017) and T-REx introduced by Elsahar et al. (2018). In
building our biomedical slot filling dataset we largely follow the conventions
and format of KILT, with the intention to ease experimentation.

Finally, Glass et al. (2021) have presented a RAG model specifically finetuned
for slot filling on the above datasets, showing significant improvement over
the generic alternatives, which were finetuned on Natural Questions (NQ).

##  3 Biomedical Slot Filling

Formally, let us first define the task of IE. We assume a knowledge source
$K$, consisting of passages $p_{i}$. Furthermore, we assume there exists a
Knowledge Base that contains a number of entities $e_{i}$. Our goal is to
extract from $K$ all possible triples of the form $e_{a}-r_{i}-e_{b}$ where
$r_{i}\in R$ and $R$ is the set of possible relation types. For each $e_{i}$
we assume that it has a specific entity type $e_{t}$ and that each $e_{t}$ can
be involved in a specific subset of $R$.

Slot filling further formulates the above task as follows: we first employ a
retrieval model $M_{r}$ that encodes all passages $p_{i}$ from $K$. The
encoded passages are indexed to allow fast retrieval. At inference, for each
$e_{i}$ of type $e_{t}$, we consider all possible relations from R and
construct the relevant queries $q_{i}:e_{i}-r_{i}$. Each query is then encoded
and the resulting vector is used to query the index, returning the n most
similar $p_{i}$ in terms of the maximum inner product:

$sim(q_{i},p_{i})=E_{Q}(q_{i})^{T}E_{P}(p_{i})$ (1)

where $E_{Q}$ is the query encoder and $E_{P}$ is the passage encoder.
Subsequently a reader model $M_{qa}$ takes as input the above query and each
of the retrieved passages and extracts zero, one or more spans, i.e., answers.
Valid answers are considered as those representing an entity $e_{i}$.

Here, we adopt as $M_{r}$ a neural, dense bi-encoder, namely DPR, which uses a
different encoder for passages and queries, but any type of retriever can be
used such as BM25, where $E_{Q}=E_{P}$. We initialize DPR’s encoders with the
ones presented in Karpukhin et al. (2020) which were finetuned on the NQ
benchmark. We subsequently train DPR on the dataset presented in Section 4,
with the following loss function:

$L(q_{i},p_{i}^{+},p_{i}^{-})=-log\frac{e^{sim(q_{i},p_{i}^{+})}}{e^{sim(q_{i},p_{i}^{+})}+e^{sim(q_{i},p_{i}^{-})}}$
(2)

Unlike Karpukhin et al. (2020), we assume that each training instance is a
$(q_{i},p_{i}^{+},p_{i}^{-})$ tuple where $p_{i}^{+}$ is a positive, i.e.,
relevant passage and $p_{i}^{-}$ is a negative passage.

Regarding the reader comprehension model $M_{qa}$, we employ a pretrained
BioBERT Lee et al. (2019a) model and finetune it on the dataset of Section 4.
To finetune we follow the standard approach for question answering with BERT
where the input is the concatenated query and passage separated by special
token [SEP] and the outputs are the start and end token positions within the
passage. The training objective is the sum of the log-likelihoods of the
correct start and end positions. For more details we refer the interested
reader to Devlin et al. (2019).

Dataset | relation | relation types | # instances
---|---|---|---
BioCreative V CDR Li et al. (2016) | compound-disease | 1 | 15,796
BioCreative VI ChemProt Krallinger et al. (2017) | compound-protein | 9 | 15,568
DDIExtraction 2013 Segura Bedmar et al. (2013) | drug-drug | 1 | 32,018
Table 2:  Public datasets used to build our biomedical slot filling dataset,
BioSF. The relation types for the drug-drug interactions dataset have been
merged into one relation dubbed interacts with.

##  4 Biomedical Slot Filling Dataset

In order to build a slot filling dataset for biomedicine, we resort to a
number of publicly available biomedical NER and RE datasets, summarized in
Table 2. Each instance in these datasets contains the relation triple as well
as the text where it was found, thus we can easily transform them in a
question answering-like format for slot filling. In total, we build two
datasets, one to train and evaluate the retriever and one for the reader model
respectively.

Specifically for the retriever training, we use negative, i.e., null relation
instances, as negatives. Additionally, we have used BM25 to add hard negatives
to our dataset, exactly as Karpukhin et al. (2020); Glass et al. (2021) have
done previously. Although, as mentioned above, these negatives might entail
some noise, similarly to when following a distant supervision approach we
expect the noise to cancel out overall. Both datasets with their training,
development and testing splits are released with our code. In the following,
we refer to our dataset as BioSF.

Retriever | hits@1 | hits@10 | hits@100 | index size(Gb)
---|---|---|---|---
BM25 | 21.4 | 36.1 | 60.6 | 1.1
DPR-NQ Karpukhin et al. (2020) | 5.5 | 17.2 | 37.6 | 2.9
DPR-multitask Maillard et al. (2021) | 4.2 | 14.3 | 33.8 | 2.9
DPR-zsRE Glass et al. (2021) | 7.6 | 19.6 | 37.2 | 2.9
Bio-DPR(ours) | 31.0 | 55.1 | 72.5 | 2.9
Table 3:  Evaluation results for retrieval experiments on the BioSF
development set using as content one million passages from PubMed. Values in
bold show statistically significant results in terms of z-test at p-value of
0.05, whereas for our model we show the average across five different DPR
training runs.

##  5 Experiments

In this Section we present the experiments that we conducted, followed by a
discussion on their implications. We are interested in evaluating our
biomedical DPR retriever, our biomedical slot filling reader and finally the
end to end slot filling approach.

Retriever | hits@1 | hits@10 | hits@100 | index size
---|---|---|---|---
BM25 | 11.0 | 30.3 | 56.1 | 29.4
DPR-NQ | 5.2 | 17.9 | 38.9 | 90.0
DPR-zsRE | 2.3 | 10.2 | 26.4 | 90.0
Bio-DPR(ours) | 11.5 | 33.2 | 59.1 | 90.0
Table 4:  Evaluation results for retrieval experiments on the BioSF
development set on full PubMed. Values in bold show statistically significant
results for a z-test at p-value of 0.05.

###  5.1 Retrieval

First, we are interested to understand the performance of our approach against
different baselines. To that end, we employ BM25 as well as two already
finetuned DPR retrievers from Karpukhin et al. (2020); Glass et al. (2021).
BM25 is a well established algorithm for retrieval, outperforming until very
recently more sophisticated neural-based approaches. It is also particularly
efficient and does not require any training, which makes it a very attractive
option for real-world production settings. Nevertheless, it is a statistical,
pattern matching based approach lacking the ability to learn semantics or
context.

Regarding the general domain DPR models, since they are currently state of the
art in the relevant general domain tasks, we seek to see if they can be used
successfully for the biomedical domain. Our model is trained on far less data,
which is nevertheless domain and task specific, therefore it is crucial to
understand which approach fares better.

####  5.1.1 Experimental Setup

We employ a PubMed dump from April 2020 as our knowledge sourse, filtering to
documents that have an abstract and splitting abstracts to roughly 100-token
length passages. We also use a smaller subset of one million passages, in
order to be able to search for optimal hyper-parameters and allow easy
replication of results. In that subset, we randomly sample passages and add
the gold passages from BioSF so as to make sure that a perfect retrieval
algorithm would be able to retrieve all correct passages and find the answer.
We highlight that this is an easier version of the real-world task, where the
retriever needs to search among around 29 million passages.

For BM25, we employ the anserini package Yang et al. (2017), and build a
Lucene index on the pre-processed passages, whereas we used the off the shelf
Huggingface models for the general domain DPR retrievers.

For our retriever, we train DPR on the BioSF dataset, for 40 epochs keeping
the best model in terms of the validation loss. We use a learning rate of
$3e-5$, an Adam optimizer with default options and a training batch size of 32
examples. Subsequently, we encode the passages with the trained passage
encoder. Encoding the full 29 million passages takes around 96 GPU hours on a
V100. We then build a flat FAISS Johnson et al. (2019) index for the encoded
passages.

####  5.1.2 Results

Initially, we conduct experiments on the smaller dataset that we described
above of one million passages. As we noted in Section 1.2 evaluating retrieval
for slot filling or more broadly for QA in the biomedical domain is
significantly different than in the general domain since in biomedicine a
query has in most cases multiple answers as opposed to the general domain.
Table 3 illustrates the results for this first series of experiments.

As we can see the DPR models that have been finetuned on the general domain
perform rather poorly compared to the much lighter and computationally
efficient BM25. Nevertheless, our model Bio-DPR, is substantially better than
BM25 in all cases, achieving up to 19 points of improvement (in the case of
hits@10). These results, are aligned to the results previously presented for
the general domain where BM25 has been outperformed by DPR. Nevertheless, in-
domain training data seems critical for DPR to perform well for slot filling,
a finding also shared in Maillard et al. (2021).

The same findings apply for the full PubMed knowledge source, as illustrated
in Table 4, although the improvement of our model over BM25 is much smaller
but still significant.

###  5.2 Slot Filling Reader

For the reader, we finetune a BioBERT-base and a BioBERT-large model on the
BioSF training set. We further include two baselines, one trained on the
BioASQ 8 QA dataset and one trained in the zero-shot RE (zsRE) dataset from
Levy et al. (2017). We employ these two baselines to test whether in-domain
data from a different task (BioASQ) or general domain data for the same task
(zsRE) can be helpful in learning an accurate model.

For all models, we train up to ten epochs, keeping the best performing model
on the development set, using a learning rate of $3e-5$, a batch size of $32$
and the Adam optimizer with default parameters. Table 5 presents the results.
We observe that the baselines perform rather poorly compared to the models
trained with in-domain slot filling data - a finding that highlights the
importance of building an in-domain dataset for slot-filling.

Model | Data | Exact Match(dev/test) | F1(dev/test)
---|---|---|---
BioBERT-base | BioASQ | 13.10/13.44 | 17.95/18.64
" | zsRE | 16.59/15.77 | 22.51/22.98
" | BioSF | 52.30/54.67 | 58.82/59.98
BioBERT-large | BioSF | 54.80/55.65 | 60.92/61.55
Table 5:  Evaluation results for the reader experiments on the BioSF
development and testing sets. We report the averages across five runs for each
model, results in bold show a statistically significant improvement for a
z-test at p-value of 0.05.

###  5.3 End to End Evaluation

Having evaluated both components of our approach, we now turn our attention to
the end to end setting, which simulates better a real world scenario. In this
setting, we are given a head entity and a relation and we want to correctly
extract the tail entity. To evaluate our approach in such a setting, we first
use the triples included in the BioSF test set. This dataset contains 3,171
queries with 2.35 answers, i.e. tails, per query on average.

Additionally, we would like to understand how our approach performs in the
zero-shot setting, i.e., for entities and relations that our model has not
seen during training. To this end, we employ Hetionet Himmelstein et al.
(2017), a network of biomedical knowledge assembled from 29 biomedical
Knowledge Bases, containing 24 distinct relation types. We keep nine relation
types that our models have not previously seen, e.g., "expresses",
"localizes", "treats" and randomly sample 500 queries, with 9.3 answers, i.e.
tails, per query on average. We note that this dataset differs substantially
to the previous one, in the sense that a query might have far more valid
answers. For example, some queries have more than 100 valid answers.

Setting | Dataset | end-to-end micro-recall
---|---|---
Standard | BioSF test set | 24.38
Zero-shot | Hetionet | 18.66
Table 6:  End to end evaluation of our approach on a standard as well as a
zero-shot setting.

In both cases, we first retrieve the top-100 passages for each query, from the
full PubMed knowledge source, using our bio-DPR model and subsequently we pass
all query-passage pairs through our reader model. We evaluate with micro-
recall since, as we discussed previously, there might be multiple valid
answers not contained in our KB and we aim to examine what percentage of the
KB triples we can extract from text. We note again that this is not a perfect
evaluation as, besides the issue mentioned above, there might also be triples
in Hetionet that do not appear in any sentence in the literature. Table 6
illustrates our results. The recall is substantially low, a finding that is
somewhat expected due to the imperfect nature of our evaluation setting, as
well the challenging nature of the task, especially in the zero-shot setting.
Nevertheless, we consider that these two additional datasets, will enable
further research and improved approaches. Overall, the above experiments
should be regarded as a stepping stone towards a novel paradigm for biomedical
IE, overcoming the shortcomings of the current standard approach.

##  6 Conclusions and Future Work

In this work we formulated the task of biomedical Information Extraction as a
slot filling problem. This approach aims to forgo the need for entity and
relation type specific training data, which is scarce and costly to annotate
in the biomedical domain. Additionally, this formulation allows to deal with
the addition of new relation types, without needing to re-train the relevant
models.

Additionally, we have introduced a new biomedical slot filling benchmark and
used it to train a biomedical DPR model, a dual BERT-based encoder for
retrieval, as well as a biomedical slot filling reader based on BioBERT. In a
series of experiments our approach outperforms significantly a number of
general domain baselines as well as the simpler BM25 retriever. Furthermore,
our results illustrate the importance of in-domain, task-specific training
data, in line with findings from recent works Glass et al. (2021); Maillard et
al. (2021).

In future work, we aim to focus on sequence to sequence variants of this work
such as the work in Izacard and Grave (2021), as well as to conduct a thorough
comparison of a standard biomedical IE system against our slot filling
approach.

## References

  * Chen et al. (2017) Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017.  Reading wikipedia to answer open-domain questions.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1870–1879.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , pages 4171–4186. Association for Computational Linguistics.
  * Elsahar et al. (2018) Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Laforest, and Elena Simperl. 2018.  T-REx: A large scale alignment of natural language with knowledge base triples.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , Miyazaki, Japan. European Language Resources Association (ELRA).
  * Glass et al. (2021) Michael Glass, Gaetano Rossiello, Md Faisal Mahbub Chowdhury, and Alfio Gliozzo. 2021.  Robust retrieval augmented generation for zero-shot slot filling.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 1939–1949, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020.  Realm: Retrieval-augmented language model pre-training.  _arXiv preprint arXiv:2002.08909_.
  * Himmelstein et al. (2017) Daniel Scott Himmelstein, Antoine Lizee, Christine Hessler, Leo Brueggeman, Sabrina L Chen, Dexter Hadley, Ari Green, Pouya Khankhanian, and Sergio E Baranzini. 2017.  Systematic integration of biomedical knowledge prioritizes drugs for repurposing.  _Elife_ , 6:e26726.
  * Izacard and Grave (2021) Gautier Izacard and Édouard Grave. 2021.  Leveraging passage retrieval with generative models for open domain question answering.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 874–880.
  * Johnson et al. (2019) Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019.  Billion-scale similarity search with gpus.  _IEEE Transactions on Big Data_.
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020.  Dense passage retrieval for open-domain question answering.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6769–6781.
  * Krallinger et al. (2017) Martin Krallinger, Obdulia Rabal, Saber A Akhondi, et al. 2017.  Overview of the biocreative vi chemical-protein interaction track.  In _Proceedings of the sixth BioCreative challenge evaluation workshop._
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019.  Natural questions: a benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:453–466.
  * Lee et al. (2019a) Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019a.  Biobert: pre-trained biomedical language representation model for biomedical text mining.  _arXiv preprint arXiv:1901.08746_.
  * Lee et al. (2019b) Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019b.  Latent retrieval for weakly supervised open domain question answering.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 6086–6096.
  * Levy et al. (2017) Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017.  Zero-shot relation extraction via reading comprehension.  In _Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017)_ , pages 333–342.
  * Lewis et al. (2020a) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020a.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880.
  * Lewis et al. (2020b) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020b.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  _arXiv preprint arXiv:2005.11401_.
  * Li et al. (2016) Jiao Li, Yueping Sun, Robin J Johnson, Daniela Sciaky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J Mattingly, Thomas C Wiegers, and Zhiyong Lu. 2016.  Biocreative v cdr task corpus: a resource for chemical disease relation extraction.  _Database_ , 2016.
  * Lin et al. (2019) Chen Lin, Timothy Miller, Dmitriy Dligach, Steven Bethard, and Guergana Savova. 2019\.  A bert-based universal model for both within-and cross-sentence clinical temporal relation extraction.  In _Proceedings of the 2nd Clinical Natural Language Processing Workshop_ , pages 65–71.
  * Lin et al. (2016) Yankai Lin, Shiqi Shen, Zhiyuan Liu, Huanbo Luan, and Maosong Sun. 2016.  Neural relation extraction with selective attention over instances.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , volume 1, pages 2124–2133.
  * Maillard et al. (2021) Jean Maillard, Vladimir Karpukhin, Fabio Petroni, Wen-tau Yih, Barlas Oğuz, Veselin Stoyanov, and Gargi Ghosh. 2021.  Multi-task retrieval for knowledge-intensive tasks.  _arXiv preprint arXiv:2101.00117_.
  * Petroni et al. (2021) Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, et al. 2021.  Kilt: a benchmark for knowledge intensive language tasks.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2523–2544.
  * Segura Bedmar et al. (2013) Isabel Segura Bedmar, Paloma Martínez, and María Herrero Zazo. 2013.  Semeval-2013 task 9: Extraction of drug-drug interactions from biomedical texts (ddiextraction 2013).  Association for Computational Linguistics.
  * Tsatsaronis et al. (2015) George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotis, Ioannis Partalas, Matthias Zschunke, Michael R Alvers, Dirk Weissenborn, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopoulos, et al. 2015.  An overview of the bioasq large-scale biomedical semantic indexing and question answering competition.  _BMC bioinformatics_ , 16(1):1–28.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in neural information processing systems_ , pages 5998–6008.
  * Verga et al. (2018) Patrick Verga, Emma Strubell, and Andrew McCallum. 2018.  Simultaneously self-attending to all mentions for full-abstract biological relation extraction.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , volume 1, pages 872–884.
  * Yang et al. (2017) Peilin Yang, Hui Fang, and Jimmy Lin. 2017.  Anserini: Enabling the use of lucene for information retrieval research.  New York, NY, USA. Association for Computing Machinery.
  * Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018.  Hotpotqa: A dataset for diverse, explainable multi-hop question answering.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2369–2380.
  * Zeng et al. (2015) Daojian Zeng, Kang Liu, Yubo Chen, and Jun Zhao. 2015.  Distant supervision for relation extraction via piecewise convolutional neural networks.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 1753–1762.
  * Zeng et al. (2014) Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, and Jun Zhao. 2014.  Relation classification via convolutional deep neural network.  In _Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers_ , pages 2335–2344.