11institutetext:  Department of Computer Science, Durham University, Durham,
United Kingdom
11email: {xiatian.zhang, shuang.chen<EMAIL_ADDRESS>22institutetext: Department of Computer Science, Swansea University, Swansea,
United Kingdom
22email<EMAIL_ADDRESS>
# Depth-Aware Endoscopic Video Inpainting

Francis Xiatian Zhang 11 0000-0003-0228-6359 Shuang Chen 11
0000-0002-6879-7285 Xianghua Xie 22 0000-0002-2701-8660
Hubert P. H. Shum (🖂) 11 0000-0001-5651-6039

###### Abstract

Video inpainting fills in corrupted video content with plausible replacements.
While recent advances in endoscopic video inpainting have shown potential for
enhancing the quality of endoscopic videos, they mainly repair 2D visual
information without effectively preserving crucial 3D spatial details for
clinical reference. Depth-aware inpainting methods attempt to preserve these
details by incorporating depth information. Still, in endoscopic contexts,
they face challenges including reliance on pre-acquired depth maps, less
effective fusion designs, and ignorance of the fidelity of 3D spatial details.
To address them, we introduce a novel Depth-aware Endoscopic Video Inpainting
(DAEVI) framework. It features a Spatial-Temporal Guided Depth Estimation
module for direct depth estimation from visual features, a Bi-Modal Paired
Channel Fusion module for effective channel-by-channel fusion of visual and
depth information, and a Depth Enhanced Discriminator to assess the fidelity
of the RGB-D sequence comprised of the inpainted frames and estimated depth
images. Experimental evaluations on established benchmarks demonstrate our
framework’s superiority, achieving a 2% improvement in PSNR and a 6% reduction
in MSE compared to state-of-the-art methods. Qualitative analyses further
validate its enhanced ability to inpaint fine details, highlighting the
benefits of integrating depth information into endoscopic inpainting.

###### Keywords:

Endoscopy Video Inpainting Deep Learning

##  1 Introduction

In endoscopic videos, occlusions or artifacts caused by reflections or
instrument shadows significantly degrade the visual quality. This issue is
commonly known as _corruptions_ , hiding critical anatomical details required
for endoscopy examinations and surgeries, affecting clinical decision
significantly [16]. As a technique to improve video quality by reconstructing
the corrupted regions based on the uncorrupted information, video inpainting
is introduced by [20, 21, 1, 6] into the endoscopic scenario to mitigate the
corruptions, known as endoscopic video inpainting. While these methods could
reconstruct 2D visual information in corrupted endoscopic videos, they suffer
from preserving vital 3D spatial details, resulting in the artifact and
spatial inconsistency at the inpainted regions, such low-fidelity performance
limits their reliability for clinical applications.

Employing depth map to complement 3D spatial information is wildly applied in
general video painting [15, 22, 14], which offers a promising solution to
preserve 3D spatial awareness for endoscopic video inpainting. Nevertheless,
applying this solution is hindered by three significant challenges: First, it
is not feasible to pre-acquire endoscopic depth maps, as the depth sensor is
not available in standard monocular endoscopic cameras [9]. Second, given the
learned depth features from deep-learning-based methods, simply concatenating
visual features channel-wise and using vanilla convolution for fusion [14]
fails to effectively exploit the depth representation, as they tend to capture
redundant representations from visual features [13], losing the 3D spatial
details complementing by depth maps. Third, none of these methods [15, 22, 14]
assess the 3D spatial fidelity in RGB inpainted outputs, which compromises the
reliability of inpainted content.

To address these challenges, we propose the Depth-Aware Endoscopic Video
Inpainting (DAEVI) framework that provides more reliable inpainted details for
improved clinical reference. It consists of a Spatial-Temporal Guided Depth
Estimation (STGDE) module, a Bi-Modal Paired Channel Fusion (BMPCF) module,
and a Depth-Enhanced Discriminator (DED), each designed to overcome the
respective challenge. First, our STGDE module extracts depth information
during visual feature learning to provide 3D spatial information, thus
avoiding the requirement for pre-acquired depth maps as input. Second, the
BMPCF module conducts a tailor-made feature fusion algorithm to better
correlate the 3D spatial relevancy between visual and depth features by pair-
wise fusing each visual and depth feature. Third, our DED assesses the 3D
spatial fidelity of the RGB-D sequence formed by the inpainted frames and
estimated depths, promoting realistic outputs with plausible 3D spatial
details.

Figure 1: Comparison with previous methods by Newson _et al._ [17] and Daher
_et al._ [6] on corrupted frames from the HyperKvasir dataset [4]. Red boxes
highlight significant differences. Reference frames are near frames with less
corruption. Our inpainted content is not only visually plausible but also
contextually realistic.

We evaluate our method on the HyperKvasir endoscopic video dataset [4] and
compare it with the corresponding benchmark [23, 6]. The quantitative
experiments demonstrate that our proposed DAEVI outperforms state-of-the-art
approaches [6], achieving approximately 2% better Peak Signal-to-Noise Ratio
(PSNR) and 6% lower Mean Squared Error (MSE). Our qualitative results show
that DAEVI inpaints more fine-grained details, such as microvessels and the
boundary of instruments, as depicted in Fig. 1. Furthermore, we directly apply
our DAEVI trained on HyperKvasir to the SERV-CT datasets [8], demonstrating
our method’s generalizability in endoscopic video inpainting. Our source code
is available on bluehttps://github.com/FrancisXZhang/DAEVI.

Our work contributes in several ways, as outlined below:

  1. 1.

To the best of our knowledge, DAEVI is the first endoscopic video inpainting
framework to incorporate depth information. The effectiveness is demonstrated
by comprehensive experiments.

  2. 2.

We propose a Spatial-Temporal Guided Depth Estimation module, to translate
depth representation directly from latent visual features, hence circumventing
the challenge of acquiring depth maps during endoscopic surgery.

  3. 3.

We design a Bi-Modal Paired Channel Fusion module that fuses each pair of
channels from visual and depth features, which effectively leverages 3D
spatial details in endoscopic inpainting.

  4. 4.

We introduce a Depth-Enhanced Discriminator within our end-to-end
optimization, which assesses the fidelity of the inpainted RGB-D sequence,
promoting realistic outputs with more plausible 3D spatial details.

##  2 Methodology

Given the input endoscopic video frames $X\in\mathbb{R}^{T\times H\times
W\times 3}$, we leverage the binary mask $M\in\mathbb{R}^{T\times H\times
W\times 1}$, which identifies the corrupted regions, to get the input
$X_{M}={X}\odot{M}$. After processing by our DAEVI to generate the uncorrupted
output $\hat{Y}\in\mathbb{R}^{T\times H\times W\times 3}$. $\odot$ is the
element-wise product, $H\times W$ is the spatial dimension. The whole
formulation is denoted as: ${\hat{Y}}=DAEVI({X}_{M})$.

Figure 2: The overview of our framework. First, our Spatial-Temporal Guided
Depth Estimation module translates depth information from corrupted frames
(See 2.1). Second, our Bi-Modal Paired Channel Fusion module effectively fuses
visual features with depth features (See 2.2). Third, our Depth Enhanced
Discriminator assesses the fidelity of the inpainted RGB-D sequence (See 2.3).

The overall architecture is shown in Fig. 2. First, DAEVI employs a
convolutional encoder to embed $X_{M}$ into compact visual features
$F\in\mathbb{R}^{\frac{H}{4}\times\frac{W}{4}\times 4C}$ to effectively
represent local visual features. Following this, our Spatial-Temporal Guided
Depth Estimation (STGDE) module learns multi-level visual features and
translates them into depth maps. Subsequently, the proposed Bi-Modal Paired
Channel Fusion (BMPCF) module fuses visual and depth features to obtain an
integrated representation with enhanced 3D spatial details. After that, a
convolutional decoder reconstructs the final inpainted frames $\hat{Y}$.
During training, our Depth-Enhanced Discriminator (DED) assesses the visual
and spatial fidelities of the inpainted RGB-D sequence.

###  2.1 Spatial-Temporal Guided Depth Estimation (STGDE)

Depth-aware endoscopic video inpainting faces a unique challenge in acquiring
depth data, as the standard endoscopic cameras are technically unable to
provide the raw depth [9]. To address this challenge, we propose a STGDE
module to translate depth features from latent visual features. STGDE involves
multiple transformer blocks $TB$ and a depth decoder $Dec_{D}$. Each TB is a
spatial-temporal transformer block [23] with a spatial enhancement to enhance
the local feature learning, thereby improving the representation ability.
$Dec_{D}$ aims to gather the latent visual feature across all $TBs$ for
effective depth estimation.

Specifically, after the encoder, corrupted frames $X_{M}$ are embedded as
visual features $F$, which are fed into the first transformer block $TB_{1}$.
The output from $TB_{i}$ is subsequently fed into the following $TB_{i+1}$,
where $i\in(1,{N_{s}}-1)$ and ${N_{s}}$ is the number of $TB_{s}$. The
$F^{i-1}$, as the input for each $TB_{i}$, are linearly transformed into query
$Q^{i}$, key $K^{i}$, and value $V^{i}$ separately. We introduce a $3\times 3$
depth-wise convolution $P^{\prime i}_{V}(\cdot)$ to enhance the $V^{i}$ for
better spatial feature learning:

$Q^{i},K^{i},V^{i}=P^{i}_{Q}(F^{i-1}),P^{i}_{K}(F^{i-1}),P^{i}_{V}(F^{i-1})+P^{\prime
i}_{V}(F^{i-1}),$ (1)

where $P^{i}_{Q,K,V}(\cdot)$ denote 1x1 2D convolutions. After that, we split
each of $Q^{i}$, $K^{i}$, and $V^{i}$ into $n={r_{1}}\times{r_{2}}$ smaller
patches $Q^{i}_{p}$, $K^{i}_{p}$, and $V^{i}_{p}$ $\in\mathbb{R}^{Tn\times
c\times h/r_{1}\times w/r_{2}}$, where $h/r_{1}\times w/r_{2}$ is the spatial
dimension of patches. Then we utilize the patched $Q^{i}_{p}$, $K^{i}_{p}$,
and $V^{i}_{p}$ to get the attention output $F^{i}_{att}$:

$S^{i}=Q^{i}_{p}(K^{i}_{p})^{\top}/\sqrt{r_{1}\times r_{2}\times c},$ (2)
$F^{i}_{att}=\text{softmax}(S^{i}\odot M)V^{i}_{p},$ (3)

where $S^{i}$ is the attention score, and $M$ denotes a resized binary mask
matrix indicating the currupted regions [23]. Then, after a convolutional
projection $P^{i}_{F}$ followed by a feed-forward network $FFN^{i}$, we obtain
the output $F^{i}$ of $TB_{i}$:

$F^{i}=FFN^{i}\left(P^{i}_{F}\left(F^{i}_{att}\right)\right).$ (4)

To translate depth maps $\hat{D}$ from latent visual features, we aggregate
$F_{att}$ across all $TB_{s}$ to gather the multi-layer visual representation.
After a convolutional projection $P^{i}_{D}$, the depth decoder generate the
depth maps $\hat{D}$:

$\hat{D}=Dec_{D}\left(\sum_{i=1}^{N_{s}}P^{i}_{D}\left(F^{i}_{att}\right)\right).$
(5)

###  2.2 Bi-Modal Paired Channel Fusion (BMPCF)

Endoscopic depth-aware inpainting encounters a challenge in effectively
integrating depth with visual information, as the simple channel-wise
concatenation followed by a vanilla convolution is unable to fully exploit the
correlation between depth and visual feature, especially in endoscopic scenes
such complex nature involving varied spatial structures [19].

To address this challenge and effectively enhance visual information with 3D
spatial details, we design a BMPCF module to correlate each visual and depth
feature by a tailor-made pair-wise fusion algorithm.

Specifically, given the depth maps $\hat{D}$ translated by $Dec_{D}$, we first
enlarge the channel capacity of $\hat{D}$ with a depth encoder $Enc_{D}$ to
obtain the embedded depth feature $F_{D}$, which has the same number of
channels as the STGDE’s output $F^{N_{s}}$. Then, to ensure each depth
correlates precisely to the corresponding visual feature, we sequentially
interleave the sliced $F^{N_{s}}$ and $F_{D}$ in channel-wise:

$F_{pair}[:,2i]=\begin{cases}F^{N{s}}[:,i]&\text{for }i=0,2,\ldots,c-2,\\\
F_{D}[:,i]&\text{for }i=1,3,\ldots,c-1,\end{cases}$ (6)

where $c$ is the channel number of $F^{N{s}}$ and $F_{D}$, also indicates the
number of pairs. After that, the group-wise convolution $G(\cdot)$ [12] is
employed on $F_{pair}$ to fuse each pair of visual and depth features,
producing the fused output $F_{f}\in\mathbb{R}^{T\times c\times h\times w}$:

$F_{f}=G(F_{pair}).$ (7)

In this way, each convolutional kernel facilitates the fusion between every
two adjacent channels consisting of one visual feature and one depth feature.
Subsequently, a convolutional decoder reconstructs inpainted frames $\hat{Y}$
from $F_{f}$.

###  2.3 Depth-Enhanced Discriminator (DED)

Effectively assessing the spatial fidelity is critical in endoscopic depth-
aware inpainting, as minor inaccuracies, such as incorrect anatomical details,
could significantly impact clinical decisions [7]. While Generative
Adversarial Network (GAN) strategies [10] in previous depth-aware inpainting
methods [22, 14] only enhance the fidelity of RGB content and ignore the
fidelity in 3D spatial details, resulting in unreliable outputs for clinical
reference [14]. To this end, we introduce DED during training, to
comprehensively assess the RGB-D inpainted endoscopic frames across spatial,
temporal and depth dimensions.

Specifically, we followed [5] to build our DED with 6 convolutional blocks.
The inpainted frames and depth are concatenated as the RGB-D data to be the
input of DED. The adversarial loss $L_{ADV}=L_{GEN}+L_{DED}$ is adopted from
[10]:

$\begin{split}L_{DED}=\mathbb{E}_{(D,Y)\sim P_{Data}}[Relu(1-DED([D,Y]))]\\\
\mathbb{E}_{(\hat{D},\hat{Y})\sim
P_{G}}[Relu(DED([\hat{D},\hat{Y}]))],\end{split}$ (8)
$L_{GEN}=-\mathbb{E}_{(\hat{D},\hat{Y})\sim P_{G}}[DED([\hat{D},\hat{Y}])],$
(9)

To enhance parameter optimization efficiency, we adopt an end-to-end
optimization strategy for our DAEVI. Our full loss function is as follows:

$L=\lambda_{D}L_{D}+\lambda_{I}L_{I}+\lambda_{GEN}L_{GEN}+\lambda_{P}L_{P}+\lambda_{S}L_{S},$
(10)

where $\lambda$ denotes the weight of each loss term. $L_{D}$ and $L_{I}$ are
the L1 reconstruction loss [24] for translated depth and inpainted frames,
respectively. $L_{P}$ and $L_{S}$ denote the perceptual loss and style loss,
respectively [11] (details can be found in our supplementary material). In
each iteration, $L$ and $L_{DED}$ optimize our inpainting network and our DED,
respectively.

##  3 Experiment

###  3.1 Experimental Setting

We evaluate our method against the existing benchmark established [6] on the
HyperKvasir endoscopic video dataset [4]. This dataset includes 373 videos
with a total of 889,372 frames, 343 for training and 30 for testing. Depth
ground truth is derived from a pre-trained endoscopic depth estimator [19] on
unmasked frames, which is needed only in training. Following the benchmark
setting [6], we employ their provided masks to identify corrupted regions in
frames and calculate metrics such as Peak Signal-to-Noise Ratio (PSNR),
Structural Similarity Index (SSIM), and Mean Square Error (MSE) specifically
for corrupted regions.

We configure the block number $N_{s}=8$ and set the weights for $\lambda_{D}$,
$\lambda_{P}$, $\lambda_{S}$, $\lambda_{I}$, and $\lambda_{GEN}$ to [0.1, 0.1,
250, 1, 0.01]. We employ Adam optimizer with learning rate $=$ 1e-4,
$\beta_{1}$: 0, $\beta_{2}$: 0.99. All experiments are trained on an NVIDIA
TITAN RTX 24G GPU with a batch size of 4 for 200k iterations. Training
iterations alternate between selecting 5 random or consecutive frames resized
to 288 x 288 pixels. For inference, the model processes every 5 corrupted
frames alongside 10 nearby corrupted frames sampled for reference,
reassembling them into the full video with a real-time processing speed of
approximately 0.03 seconds per frame.

Table 1: Inpainting Performance Comparison and Ablation Study. w/o STGDE: A pre-trained depth estimator [19] is leveraged for depth estimation instead of STGDE; w/o BMPCF: Simple concatenation is used for fusion instead of BMPCF; w/o DED: A standard RGB discriminator [5] is used in GAN training instead of DED. Methods |  $PSNR_{Crop}\uparrow$ |  $SSIM_{Crop}\uparrow$ |  $MSE_{Crop}\downarrow$
---|---|---|---
Arnold _et al._ [2] |  19.909 |  0.559 |  895.222
Newson _et al._ [17] |  22.27 |  0.650 |  543.636
STTN [23] |  28.683 |  0.793 |  119.541
Daher _et al._ [6] |  29.542 |  0.785 |  104.719
DAEVI (Full Framework) |  30.126 |  0.797 |  97.873
w/o STGDE |  29.801 |  0.788 |  105.150
w/o BMPCF |  29.695 |  0.791 |  103.861
w/o DED |  29.286 |  0.797 |  108.903

###  3.2 Results

Comparison with Existing Methods In Table 1, we benchmark our DAEVI against
Arnold _et al._ [2], a diffusion-based approach; Newson _et al._ [17],
employing temporal patches; STTN [23], a transformer-based method; and Daher
_et al._ [6], the state-of-the-art (SOTA) method in endoscopic video
inpainting. Our method outperforms existing methods across all metrics. These
results highlight the benefits of integrating depth information into
endoscopic video inpainting. Clinically, these improvements enhance the
visibility of endoscopic videos [21], crucial for accurate diagnostics and
surgical planning.

Qualitative Results To demonstrate our inpainting performance on corrupted
regions, we remove specular reflections and high reflection from instruments
in the HyperKvasir dataset [4] and the SERV-CT dataset, an external-body
endoscopic dataset with depth ground truth provided [8]. Figure 1 from the
HyperKvasir dataset illustrates our method more effectively restoring details
such as microvessels and the interface between instruments and organs. These
enhancements are crucial for safer and more efficient endoscopic operations
[3]. More qualitative results can be found in our supplementary material.

Figure 3: Comparison of deep learning-based inpainting performance on the
SERV-CT dataset: a) Generalization Capability, and b) Depth Information
Preservation.

Additionally, Fig. 3 a) shows our method’s ability to inpaint highly plausible
content on the SERV-CT dataset without specific fine-tuning, underscoring our
approach’s strong generalization capabilities. By applying the pre-trained
DepthNet [19] to our inpainted frames, Fig. 3 b) reveals that our method
improves depth estimation, achieving the lowest Root Mean Squared Error (RMSE)
between the ground truth depth and the depth estimation after inpainting. This
further underscores our method’s superior effectiveness in preserving 3D
spatial details compared to existing methods.

Ablation Study Our ablation study in Table 1 shows that our full framework
yields the best results, highlighting the importance of each module. Notably,
removing DED lowers MSE and PSNR but doesn’t affect SSIM, which remains
comparable to the complete framework. This may be because DED significantly
enhances 3D spatial fidelity, which does not directly correspond to the
features assessed by SSIM [18], such as contrast and luminance.

Table 2: Performance Analysis Across Depth Estimation Block Configurations. Methods |  $PSNR_{Crop}\uparrow$ |  $SSIM_{Crop}\uparrow$ |  $MSE_{Crop}\downarrow$
---|---|---|---
DAEVI (First 4 Blocks) |  29.921 |  0.792 |  100.851
DAEVI (Last 4 Blocks) |  29.900 |  0.796 |  101.313
DAEVI (All 8 Blocks) |  30.126 |  0.797 |  97.873
Table 3: Online Inference Performance Analysis Methods |  $PSNR_{Crop}\uparrow$ |  $SSIM_{Crop}\uparrow$ |  $MSE_{Crop}\downarrow$
---|---|---|---
Daher _et al._ [6] |  29.542 |  0.785 |  104.719
DAEVI |  30.126 |  0.797 |  97.873
DAEVI (Online) |  30.117 |  0.797 |  98.147

Depth Estimation Block Configuration Analysis Table 2 evaluates inpainting
performance using different configurations of the first 4, last 4, and all 8
blocks of STGCN for depth estimation within the DAEVI framework. This analysis
reveals that employing all 8 blocks, which combines both low-level and high-
level features, achieves optimal performance, highlighting the effectiveness
of our current STGDE design.

Online Inference Performance Analysis Table 3 assesses our method’s online
inference ability, which employs only past frames for reference. Our online
performance still outperforms the SOTA method [6] that uses both past and
future frames as reference frames, demonstrating its potential for live
endoscopic videos.

##  4 Conclusion and Discussions

In this paper, we propose the DAEVI framework, the first endoscopic video
inpainting framework designed to incorporate depth information to achieve
reliable 3D spatial details. This work offers a potential solution to enhance
the quality of endoscopic videos, facilitating informed clinical decision-
making. Our DAEVI leverages depth information from latent visual features of
corrupted endoscopic frames with STGDE, effectively fuses visual and depth
information with BMPCF, and assesses the fidelity of the RGB-D content with
DED. Experimental evaluations demonstrate its significant superiority compared
to existing methods.

While our framework demonstrates improvements in inpainting performance, it is
not without limitations. As our work’s focus is on corruption inpainting, the
real-world applicability of DAEVI could still be influenced by the
effectiveness of any external corruption detection backbone. Future work could
integrate existing endoscopic corruption detection methods such as semantic
segmentation [1] into our framework, optimizing both corruption detection and
inpainting tasks.

####  4.0.1 Acknowledgements

This research is supported in part by the EPSRC NortHFutures project (ref:
EP/X031012/1).

####  4.0.2

The authors declare that there are no conflicts of interest related to this
paper.

## References

  * [1] Ali, S., Zhou, F., Bailey, A., Braden, B., East, J.E., Lu, X., Rittscher, J.: A deep learning framework for quality assessment and restoration in video endoscopy. Medical Image Analysis 68, 101900 (2021)
  * [2] Arnold, M., Ghosh, A., Ameling, S., Lacey, G.: Automatic segmentation and inpainting of specular highlights for endoscopic imaging. EURASIP Journal on Image and Video Processing 2010, 1–12 (2010)
  * [3] Ben-Menachem, T., Decker, G.A., Early, D.S., Evans, J., Fanelli, R.D., Fisher, D.A., Fisher, L., Fukami, N., Hwang, J.H., Ikenberry, S.O., et al.: Adverse events of upper gi endoscopy. Gastrointestinal Endoscopy 76(4), 707–718 (2012)
  * [4] Borgli, H., Thambawita, V., Smedsrud, P.H., Hicks, S., Jha, D., Eskeland, S.L., Randel, K.R., Pogorelov, K., Lux, M., Nguyen, D.T.D., et al.: Hyperkvasir, a comprehensive multi-class image and video dataset for gastrointestinal endoscopy. Scientific data 7(1), 283 (2020)
  * [5] Chang, Y.L., Liu, Z.Y., Lee, K.Y., Hsu, W.: Free-form video inpainting with 3d gated convolution and temporal patchgan. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 9066–9075 (2019)
  * [6] Daher, R., Vasconcelos, F., Stoyanov, D.: A temporal learning approach to inpainting endoscopic specularities and its effect on image correspondence. Medical Image Analysis p. 102994 (2023)
  * [7] Dray, X., Histace, A., Robertson, A., Segui, S.: Artificial intelligence for protruding lesions. In: Artificial Intelligence in Capsule Endoscopy, pp. 121–148. Elsevier (2023)
  * [8] Edwards, P.E., Psychogyios, D., Speidel, S., Maier-Hein, L., Stoyanov, D.: Serv-ct: A disparity dataset from cone-beam ct for validation of endoscopic 3d reconstruction. Medical Image Analysis 76, 102302 (2022)
  * [9] Fu, Z., Jin, Z., Zhang, C., He, Z., Zha, Z., Hu, C., Gan, T., Yan, Q., Wang, P., Ye, X.: The future of endoscopic navigation: A review of advanced endoscopic vision technology. IEEE Access 9, 41144–41167 (2021)
  * [10] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial networks. Communications of the ACM 63(11), 139–144 (2020)
  * [11] Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14. pp. 694–711. Springer (2016)
  * [12] Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems 25 (2012)
  * [13] Li, J., Wen, Y., He, L.: Scconv: Spatial and channel reconstruction convolution for feature redundancy. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6153–6162 (2023)
  * [14] Li, S., Zhu, S., Ge, Y., Zeng, B., Imran, M.A., Abbasi, Q.H., Cooper, J.: Depth-guided deep video inpainting. IEEE Transactions on Multimedia (2023)
  * [15] Liao, M., Lu, F., Zhou, D., Zhang, S., Li, W., Yang, R.: Dvi: Depth guided video inpainting for autonomous driving. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXI 16. pp. 1–17. Springer (2020)
  * [16] Monkam, P., Wu, J., Lu, W., Shan, W., Chen, H., Zhai, Y.: Easyspec: Automatic specular reflection detection and suppression from endoscopic images. IEEE Transactions on Computational Imaging 7, 1031–1043 (2021)
  * [17] Newson, A., Almansa, A., Fradet, M., Gousseau, Y., Pérez, P.: Video inpainting of complex scenes. Siam Journal on Imaging Sciences 7(4), 1993–2019 (2014)
  * [18] Pambrun, J.F., Noumeir, R.: Limitations of the ssim quality metric in the context of diagnostic imaging. In: 2015 IEEE International Conference on Image Processing. pp. 2960–2963. IEEE (2015)
  * [19] Shao, S., Pei, Z., Chen, W., Zhu, W., Wu, X., Sun, D., Zhang, B.: Self-supervised monocular depth and ego-motion estimation in endoscopy: Appearance flow to the rescue. Medical Image Analysis 77, 102338 (2022)
  * [20] Shen, D.F., Guo, J.J., Lin, G.S., Lin, J.Y.: Content-aware specular reflection suppression based on adaptive image inpainting and neural network for endoscopic images. Computer Methods and Programs in Biomedicine 192, 105414 (2020)
  * [21] Tukra, S., Marcus, H.J., Giannarou, S.: See-through vision with unsupervised scene occlusion reconstruction. IEEE Transactions on Pattern Analysis and Machine Intelligence 44(7), 3779–3790 (2021)
  * [22] Yamashita, Y., Shimosato, K., Ukita, N.: Boundary-aware image inpainting with multiple auxiliary cues. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 619–629 (2022)
  * [23] Zeng, Y., Fu, J., Chao, H.: Learning joint spatial-temporal transformations for video inpainting. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVI 16. pp. 528–543. Springer (2020)
  * [24] Zhou, S., Li, C., Chan, K.C., Loy, C.C.: Propainter: Improving propagation and transformer for video inpainting. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 10477–10486 (2023)

  *[DAEVI]: Depth-Aware Endoscopic Video Inpainting
  *[STGDE]: Spatial-Temporal Guided Depth Estimation
  *[BMPCF]: Bi-Modal Paired Channel Fusion
  *[DED]: Depth-Enhanced Discriminator
  *[PSNR]: Peak Signal-to-Noise Ratio
  *[MSE]: Mean Squared Error
  *[GAN]: Generative Adversarial Network
  *[SOTA]: state-of-the-art


# FairVFL: A Fair Vertical Federated Learning Framework with Contrastive
Adversarial Learning

Tao Qi
Tsinghua University
Beijing, China
<EMAIL_ADDRESS>
&Fangzhao Wu
Microsoft Research Asia
Beijing China
<EMAIL_ADDRESS>
&Chuhan Wu
Tsinghua University
Beijing, China
<EMAIL_ADDRESS>
Lingjuan Lyu
Sony AI
Tokyo, Japan
<EMAIL_ADDRESS>
&Tong Xu
USTC
Hefei, China
<EMAIL_ADDRESS>
&Hao Liao
Shenzhen University
Shenzhen, China
<EMAIL_ADDRESS>
&Zhongliang Yang
Tsinghua University
Beijing, China
<EMAIL_ADDRESS>
&Yongfeng Huang
Tsinghua University
& Zhongguancun Laboratory
<EMAIL_ADDRESS>
&Xing Xie
Microsoft Research Asia
Beijing China
<EMAIL_ADDRESS>
The corresponding author.

###### Abstract

Vertical federated learning (VFL) is a privacy-preserving machine learning
paradigm that can learn models from features distributed on different
platforms in a privacy-preserving way. Since in real-world applications the
data may contain bias on fairness-sensitive features (e.g., gender), VFL
models may inherit bias from training data and become unfair for some user
groups. However, existing fair machine learning methods usually rely on the
centralized storage of fairness-sensitive features to achieve model fairness,
which are usually inapplicable in federated scenarios. In this paper, we
propose a fair vertical federated learning framework (FairVFL), which can
improve the fairness of VFL models. The core idea of FairVFL is to learn
unified and fair representations of samples based on the decentralized feature
fields in a privacy-preserving way. Specifically, each platform with fairness-
insensitive features first learns local data representations from local
features. Then, these local representations are uploaded to a server and
aggregated into a unified representation for the target task. In order to
learn a fair unified representation, we send it to each platform storing
fairness-sensitive features and apply adversarial learning to remove bias from
the unified representation inherited from the biased data. Moreover, for
protecting user privacy, we further propose a contrastive adversarial learning
method to remove private information from the unified representation in server
before sending it to the platforms keeping fairness-sensitive features.
Experiments on three real-world datasets validate that our method can
effectively improve model fairness with user privacy well-protected.

##  1 INTRODUCTION

In recent years, the explosion of data volume has enhanced the performance of
machine learning (ML) models on many tasks, e.g., personalized recommendation
and search [42, 51, 3]. In many real-world scenarios, feature fields of the
same sample may be decentralized on different platforms [49, 30]. For
instance, video watching behaviors and search engine behaviors of the same
user usually spread across different platforms [43]. To learn models from
various input features, existing machine learning methods usually rely on
centralized storage of different feature fields [32, 22]. However, the data on
different platforms is usually privacy-sensitive, and cannot be centralized
due to privacy concerns and risks [48, 37]. Thus, vertical federated learning
(VFL), which can utilize decentralized features for unified model learning in
a privacy-preserving way, has become more and more important [54, 10, 8].

Existing VFL methods usually communicate intermediate results instead of raw
data across platforms to enable models utilizing decentralized feature fields
[54, 10]. For example, Yang et al. [50] proposed a vertical federated
framework for logistic regression. They first calculated local results based
on local feature fields, and then shared encrypted local results with a server
to make predictions and learn parameters. Che et al. [8] proposed a vertical
federated multi-view learning framework to integrate decentralized medical
data. They locally encoded medical feature fields kept by multiple hospitals
into representation vectors and uploaded them to the server for diagnosis
prediction. However, since real-world data usually encodes bias on fairness-
sensitive features (e.g., genders and ages) [28, 52], VFL models may inherit
bias from data and become unfair to some user groups [46, 47].

Due to the importance of fairness in machine learning, fair ML methods have
attracted increasing attention in recent years [46, 47, 28, 52, 27]. For
example, Zafar et al. [52] proposed a fairness-aware optimization framework to
improve the model fairness, which can constrain the difference between
prediction distributions for different user groups. Li et al. [27] proposed a
fair recommendation framework, which quantifies the recommendation unfairness
on different user groups as a penalty for item re-ranking. In general, most of
the existing fair ML methods rely on the centralized storage of fairness-
sensitive features to achieve model fairness. However, in vertical federated
learning, feature fields are decentralized on different platforms [44, 15],
making it difficult to apply existing fair ML methods to improve the fairness
of VFL models. Thus, how to design a fair VFL framework that can protect user
privacy and meanwhile improve model fairness, is a rarely studied problem.

In this paper, we propose a fair vertical federated learning framework (named
FairVFL), which can improve the fairness of VFL models in a privacy-preserving
way. In FairVFL, we partition data into two groups based on their fairness
sensitivities, i.e., fairness-insensitive features that can be used for the
target task and fairness-sensitive features (e.g., genders) that should be
causally irrelevant to model predictions. Along this line, the core of FairVFL
is to learn unified and fair representations of samples based on their
features distributed across different platforms with user privacy well-
protected. Specifically, platforms with fairness-insensitive features first
learn local data representations, which are then uploaded to servers to build
unified representations. Since unified representations encode rich information
of raw data, they can be used for the target task without access to the raw
data. Besides, unified representations may inherit bias towards sensitive
features (e.g., gender stereotype) from implicit feature correlations in
biased data and hurt model fairness [46, 28, 47]. In order to further learn
fair unified representations, we apply adversarial learning to unified
representations to remove bias encoded in them. Due to the decentralized
storage of feature fields, unified representations are sent to platforms with
fairness-sensitive features to learn adversarial gradients. Besides, since
unified representations may still encode some private information of raw data
[41, 35, 34], directly sharing them among platforms may incur privacy
problems. Thus, we further use contrastive adversarial learning to remove
private information from the unified representations before sending them to
other platforms. Extensive experiments on three real-world datasets show that
our FairVFL approach can effectively improve the fairness of VFL models and
meanwhile the privacy is well protected.

##  2 RELATED WORK

###  2.1 Vertical Federated Learning

Vertical federated learning is a privacy-preserving paradigm for training ML
models from feature fields that are kept by different platforms that share the
same data ID space, and has been widely studied in recent years [44, 8, 15,
9]. For example, Wu et al. [44] proposed a privacy-preserving ad CTR
prediction framework with vertical federated learning. They proposed to
coordinate local platforms to learn behavior representations from local
features and share the behavior representations with the server to model user
interest in ads. However, since real-world data usually encode bias on
fairness-sensitive features (e.g., genders) [46, 47], VFL models may inherit
bias from data and make unfair decisions for some user groups (e.g., minority
users). To improve model fairness in VFL, Liu et al. [29] formulated the model
performance gap for different user groups as a fairness regularization and
incorporated it into the federated optimization objective. However, their
method needs to share the data grouping information with the server, which may
also leak user privacy on fairness-sensitive features (e.g., gender).
Different from these methods, we propose a unified framework that can improve
the fairness of VFL models which applies the adversarial learning to learn
fair models and employ a novel contrastive adversarial learning method to
protect user privacy.

###  2.2 Fair Machine Learning

With the increasing impacts of ML techniques on our society, the fairness of
machine learning models has attracted substantial attention [46, 23, 27, 47].
Existing fair machine learning methods are usually designed to learn unbiased
models with respect to various fairness-sensitive features (e.g., genders).
Some of them applied pre-processing techniques [6, 33, 14, 7] (e.g.,
resampling) or post-processing techniques [27, 4, 23, 19] (e.g., re-ranking)
to improve model fairness. However, these methods usually independently
optimized model accuracy and fairness, which may only achieve a sub-optimal
trade-off between accuracy and fairness [46, 47]. To better improve model
fairness, many works have been proposed to jointly optimize model accuracy and
fairness during model training [52, 16, 28, 47]. However, most of the existing
fair ML methods rely on the centralized storage of feature fields, making them
difficult to be applied in vertical federated learning. Different from these
methods, we propose a fair vertical federated learning framework, which
applies adversarial learning to improve the fairness of VFL models based on
decentralized feature fields.

##  3 Methodology

###  3.1 Problem Definition of FairVFL

In VFL, different feature fields of the same samples are decentralized on
multiple platforms. From a fairness perspective, these data can be partitioned
into two groups, i.e., fairness-insensitive features and fairness-sensitive
features. The former can be taken by models as the inputs while the latter are
expected to be causally irrelevant to the model predictions. Without loss of
generality, we assume that there are $m$ types of fairness-sensitive features,
and we denote the platform keeping the $i$-th one with type $d_{i}^{a}$ as
$P^{a}_{i}$.111To simplify the presentation, we use different symbols to
denote platforms storing different fairness-sensitive feature fields. We
remark that a platform can store multiple sensitive feature fields in
practice. Besides, we assume that fairness-insensitive feature fields are
decentralized on $n$ platforms, and we denote the $i$-th platform as
$P^{b}_{i}$. Moreover, we assume that there is a task platform $P^{t}$ keeping
labels $y$ of samples on the target task. Following existing works [44, 51],
we assume that there is a trust-worthy server $P^{w}$ for information
aggregation.

Next, we present a formal definition of privacy protection and fairness. The
privacy constraint requires that the private information in the local data of
a platform should not be disclosed to another platform. Thus, the raw data of
a local platform cannot be disclosed and the intermediate results shared with
other platforms should be carefully protected. Besides, in this paper, we
define model fairness from a casual view [28]. The counterfactual fairness
requires that the model prediction for a sample should always hold if we only
change the fairness-sensitive features. According to the theoretical analysis
in Li et al. [28], eliminating fairness-sensitive feature information in the
data representation that is used for the model prediction can achieve casual
fairness. We remark that even though fairness-sensitive features are not taken
as the inputs, the model may also inherit bias related to sensitive features
by mining inherent shortcuts from data. [46, 28, 47]. The goal of fair
vertical federated learning is to achieve a good trade-off among performance,
privacy, and fairness, i.e., improving the fairness of VFL models with minimal
performance drop in a privacy-preserving way.

###  3.2 Federated Model Serving

Figure 1: Framework of our fair vertical federated learning method (FairVFL).

The main framework of FairVFL to utilize decentralized feature fields is to
encode them into a unified representation (Fig. 1). The ML model in FairVFL is
partitioned into three parts, i.e., multiple local models
$\\{\mathcal{M}^{l}_{i}|i=1,...,n\\}$, an aggregation model $\mathcal{M}^{w}$
and a task-specific model $\mathcal{M}^{t}$, where $\mathcal{M}^{l}_{i}$ is
the $i$-th local model maintained by the platform $P^{b}_{i}$. In FairVFL,
local models are used to encode local fairness-insensitive features into local
representations. We remark that FairVFL is a general framework and the
architecture of a specific local model can be adjusted based on the input
data. For example, the local models for encoding textual search logs can be
implemented by transformers [40] or PLMs [11]. When the task platform $P^{t}$
needs to serve a target sample $s$, $P^{t}$ will first distribute the ID of
the sample to fairness-insensitive platforms $\\{P^{b}_{i}|i=1,...,n\\}$. In
the $i$-th platform $P^{b}_{i}$, the local model $\mathcal{M}^{l}_{i}$ encodes
fairness-insensitive features of the target sample $s$ stored in $P^{b}_{i}$,
and builds a local representation $\textbf{s}^{l}_{i}$. Thus, we can obtain
multiple local representations $\\{\textbf{s}^{l}_{i}|i=1,...,n\\}$, which
encode different fairness-insensitive feature fields of the target sample.
Following Wu et al. [44], these local representations are then uploaded to the
server $P^{w}$ for task prediction.

The aggregation model $\mathcal{M}^{w}$ is maintained by the server $P^{w}$.
It is used to aggregate information in local representations into a unified
representation s. Specifically, different feature fields usually have inherent
relatedness, mining which can enhance the model for the target task [3, 45].
Thus, we first apply a multi-head self-attention network [40] to capture
relatedness among local representations, where the contextual representation
of $\textbf{s}^{l}_{i}$ is denoted as $\hat{\textbf{s}}_{i}$. Then we apply an
attention network to contextual representations to model their relative
importance for representing raw data and build the unified representation
s.222The aggregation model can be also implemented by other pooling networks,
like a dense network. Since s can encode various information in decentralized
fairness-insensitive feature fields, we further upload the unified
representation s instead of raw data to the task platform $P^{t}$ to provide
information for the target task. (We apply LDP to protect s before sending it
to $P^{t}$.)

The task-specific model $\mathcal{M}^{t}$ is maintained by the task-specific
platform $P^{t}$. It is used to utilize information in the unified
representation s to make prediction for the target task:
$\hat{y}=\mathcal{M}^{t}(\textbf{s}),$ where $\hat{y}$ is the model
prediction. The architecture of the task-specific model $\mathcal{M}^{t}$ is
also adjusted based on the target task (e.g., risk prediction). For instance,
for the classification task, we can employ a dense network to implement
$\mathcal{M}^{t}$. In this way, decentralized fairness-insensitive feature
fields of a target sample can be utilized by FairVFL for the target task
without the disclosure of raw data.

###  3.3 Fair Representation Learning in FairVFL

Figure 2: The detailed workflow of contrastive adversarial learning.

As introduced in Sec. 3.2, the task prediction for each sample is computed
based on their unified representation s. Thus, the core of FairVFL to achieve
counterfactual fairness is to learn a fair and unified representation s that
does not encode fairness-sensitive bias. Although the fairness-sensitive
features are not model inputs, the model may still encode bias related to them
by mining inherent shortcuts to these features from data. Thus, we apply
adversarial learning to prevent unified representations from encoding
fairness-sensitive bias. Vanilla adversarial learning techniques usually rely
on centralized storage of input data and labels of fairness-sensitive
attributes to learn adversarial gradients for model training. However, in
vertical federated scenarios, feature fields are decentralized on different
platforms, which brings challenges to the application of existing methods. An
intuitive solution is sharing the unified representation with platforms that
store observed fairness-sensitive features $\\{P^{a}_{i}|i=1,2,...,m\\}$ to
learn corresponding adversarial gradients. However, unified representations
usually encode some private information, and directly sharing them across
different platforms may lead to user privacy leakage. To tackle this
challenge, we propose a contrastive adversarial learning technique to protect
user privacy by modifying unified representations.

As shown in Fig. 2, the core idea of the contrastive adversarial learning is
to learn a protected representation that only encodes information on a certain
fairness-sensitive feature. The protected representation can be further shared
with the platform keeping the corresponding sensitive feature to learn
adversarial gradients without leaking user privacy. Specifically, for each
fairness-sensitive feature $d^{a}_{i}$, we first map unified representation s
into a protected representation $\textbf{a}_{i}$:
$\textbf{a}_{i}=A_{i}(\textbf{s})$, where $A_{i}$ is a mapper based on MLP and
s is the preimage of $\textbf{a}_{i}$. $\textbf{a}_{i}$ is designed to only
retain information of s on the fairness-sensitive feature $d^{a}_{i}$ and
eliminate other user privacy. To achieve this goal, we propose the contrastive
adversarial learning method to make the preimage of $\textbf{a}_{i}$
indistinguishable among unified representations with the same feature
$d^{a}_{i}$. Given representations of samples in the same training batch
$\\{(\textbf{a}_{i}^{j},\textbf{s}^{j})|j=1,2,..,E\\}$, we first rank these
samples based on the relevance between their protected representations
$\textbf{a}_{i}^{j}$ and $\textbf{a}_{i}$:
$r_{j}=\textbf{a}_{i}\cdot\textbf{a}_{i}^{j}$, where $\textbf{a}_{i}^{j}$ and
$\textbf{s}^{j}$ are the protected and unified representation of the $j$-th
sample, $r_{j}$ is the relevance score, and $E$ is the batch size.333The
relevance score for the current data sample is masked as $-\infty$.
Intuitively, highly relevant data samples are likely to share the same
fairness-sensitive feature $d^{a}_{i}$. Thus, from the top $E_{i}$ samples
ranked by relevance scores, we can randomly select a unified representation
$\textbf{s}^{-}_{i}$ that is highly likely to be the same with s in
$d^{a}_{i}$, where $E_{i}$ is a hyper-parameter. $\textbf{s}^{-}_{i}$ is used
as the negative sample in contrastive learning to learn a contrastive
discriminator $D^{c}_{i}$ to classify preimage of $\textbf{a}_{i}$:

$\mathcal{L}^{p}_{i}=-\frac{1}{|\Omega|}\sum_{x\in\Omega}\log\frac{\exp(D^{c}_{i}(\textbf{a}_{i},\textbf{s}))}{\exp(D^{c}_{i}(\textbf{a}_{i},\textbf{s}))+\exp(D^{c}_{i}(\textbf{a}_{i},\textbf{s}^{-}_{i}))},$
(1)

where $D^{c}_{i}(\cdot)$ is implemented by an MLP network, $\Omega$ is the
training set and $x$ is a data sample in $\Omega$. Optimal discriminator
$\hat{D}^{c}_{i}$ is learned by minimizing $\mathcal{L}^{p}_{i}$ with fixed
representations s, $\textbf{s}^{-}_{i}$, $\textbf{a}_{i}$:

$\hat{D}^{c}_{i}=\arg\min_{D^{c}_{i}}\mathcal{L}^{p}_{i}.$ (2)

Finally, the optimal discriminator $\hat{D}^{c}_{i}$ is used to learn
contrastive adversarial loss $\mathcal{L}^{c}_{i}$ and the contrastive
adversarial gradients
$\frac{\partial\mathcal{L}^{c}_{i}}{\partial\textbf{a}_{i}}$ on
$\textbf{a}_{i}$ with the fixed optimal discriminator $\hat{D}^{c}_{i}$, s and
$\textbf{s}^{-}_{i}$:

$\mathcal{L}^{c}_{i}=-\frac{1}{|\Omega|}\sum_{x\in\Omega}\log\frac{\exp(\hat{D}^{c}_{i}(\textbf{a}_{i},\textbf{s}))}{\exp(\hat{D}^{c}_{i}(\textbf{a}_{i},\textbf{s}))+\exp(\hat{D}^{c}_{i}(\textbf{a}_{i},\textbf{s}^{-}_{i}))}.$
(3)

The contrastive adversarial gradient
$\frac{\partial\mathcal{L}^{c}_{i}}{\partial\textbf{a}_{i}}$ on
$\textbf{a}_{i}$ is further back-propagated to the mapper $A_{i}$:

$\frac{\partial\mathcal{L}^{c}_{i}}{\partial
A_{i}}=\frac{\partial\mathcal{L}^{c}_{i}}{\partial\textbf{a}_{i}}\frac{\partial\textbf{a}_{i}}{\partial
A_{i}},\quad\textbf{a}_{i}=A_{i}(\textbf{s}),$ (4)

where negative contrastive adversarial gradient
$-\frac{\partial\mathcal{L}^{c}_{i}}{\partial A_{i}}$ on $A_{i}$ is used for
model updating, and representations s and $\textbf{s}_{i}^{-}$ are fixed and
not tuned by the contrastive adversarial gradient. In this way, contrastive
adversarial gradients can enforce the mapper $A_{i}$ to protect user private
information in $\textbf{a}_{i}$.

Since user privacy can be reduced from $\textbf{a}_{i}$, we further share it
with the platform $P^{a}_{i}$ keeping label $\textbf{y}^{a}_{i}$ of the
fairness-sensitive feature $d^{a}_{i}$ to learn corresponding adversarial
gradients. The platform $P^{a}_{i}$ first employs a bias discriminator
$D^{a}_{i}$ to predict the label $\hat{\textbf{y}}^{a}_{i}$ of fairness-
sensitive feature $d^{a}_{i}$ from the protected representation
$\textbf{a}_{i}$ and obtain a bias discrimination loss $\mathcal{L}^{d}_{i}$
as follows:

$\mathcal{L}^{d}_{i}=-\frac{1}{|\Omega|}\sum_{x\in\Omega}\textbf{y}^{a}_{i}\cdot\log\hat{\textbf{y}}^{a}_{i},\quad\hat{\textbf{y}}^{a}_{i}=D^{a}_{i}(\textbf{a}_{i}),$
(5)

where $D^{a}_{i}$ is implemented by an MLP network. Moreover, besides private
information, contrastive adversarial learning may remove bias information from
$\textbf{a}_{i}$ and hurt fair representation learning. Thus, we minimize bias
discrimination loss $\mathcal{L}^{d}_{i}$ to optimize both bias discriminator
$D^{a}_{i}$ and mapper $A_{i}$:

$\hat{D}^{a}_{i},\hat{A}_{i}=\arg\min_{D^{a}_{i},A_{i}}\mathcal{L}^{d}_{i}=\arg\min_{D^{a}_{i},A_{i}}-\textbf{y}^{a}_{i}\cdot\log
D^{a}_{i}(A_{i}(\textbf{s})),$ (6)

where $\hat{D}^{a}_{i}$ is the optimal bias discriminator, $\hat{A}_{i}$ is
the $i$-th optimal mapper, and the unified representation s is fixed. Finally,
for the $i$-th fairness-sensitive feature $d^{a}_{i}$, we can obtain the
adversarial loss $\mathcal{L}^{a}_{i}$ and corresponding adversarial gradients
$\frac{\partial\mathcal{L}^{a}_{i}}{\partial\textbf{s}}$ on unified
representation s:

$\mathcal{L}^{a}_{i}=-\frac{1}{|\Omega|}\sum_{x\in\Omega}\textbf{y}^{a}_{i}\cdot\log\hat{\textbf{y}}^{a}_{i},\quad\hat{\textbf{y}}^{a}_{i}=\hat{D}^{a}_{i}(\hat{A}_{i}(\textbf{s})),$
(7)

where the optimal bias discriminator and mapper are fixed when optimizing the
unified representation s. The negative adversarial gradient
$-\frac{\partial\mathcal{L}^{a}_{i}}{\partial\textbf{s}}$ is further used to
reduce fairness-sensitive bias encoded in the unified representation s to
achieve counterfactual fairness. Besides, we can obtain the task loss
$\mathcal{L}^{t}$ based on task label $y$ and predicted label $\hat{y}$. The
task loss function is also adjusted based on the target task. Finally, we can
formulate the overall loss $\mathcal{L}$ as:

$\mathcal{L}=\mathcal{L}^{t}-\sum_{i=1}^{m}\lambda_{i}\mathcal{L}^{a}_{i}-\sum_{i=1}^{m}\gamma_{i}\mathcal{L}^{c}_{i},$
(8)

where $\mathcal{L}^{t}$, $\mathcal{L}^{a}$ and $\mathcal{L}^{c}$ represents
the training objective for model performance, fairness and privacy
respectively, $\lambda_{i}$ is the weight of loss $\mathcal{L}^{a}_{i}$ and
$\gamma_{i}$ is the weight of loss $\mathcal{L}^{c}_{i}$.

###  3.4 Federated Model Training

Next, we will introduce how we train model on decentralized fairness-sensitive
and insensitive feature fields. First, for the task loss $\mathcal{L}^{t}$,
the task platform $P^{t}$ can directly calculate gradient
$\frac{\partial\mathcal{L}^{t}}{\partial\mathcal{M}^{t}}$ on task model and
gradient $\frac{\partial\mathcal{L}^{t}}{\partial\textbf{s}}$ on unified
representation s. $\frac{\partial\mathcal{L}^{t}}{\partial\mathcal{M}^{t}}$ is
used to update $\mathcal{M}^{t}$ and
$\frac{\partial\mathcal{L}^{t}}{\partial\textbf{s}}$ is distributed to the
server $P^{w}$ for the further model updating. Second, it is usually difficult
to obtain the optimal discriminator in adversarial learning. Following
Goodfellow et al. [17], we optimize the discriminator for a single step based
on discrimination gradients before executing adversarial learning.
Specifically, for the contrastive adversarial learning, the sever first learns
contrastive discrimination gradient
$\frac{\partial\mathcal{L}_{i}^{p}}{\partial D^{c}_{i}}$ on contrastive
discriminator $D^{c}_{i}$ to update it. Then the server utilizes updated
$D^{c}_{i}$ to learn contrastive adversarial gradient
$\frac{\partial\mathcal{L}_{i}^{c}}{\partial A_{i}}$ on mapper $A_{i}$ to
update it. Third, for adversarial learning on each fairness-sensitive feature
$d^{a}_{i}$, platform $P^{a}_{i}$ first calculates bias discrimination
gradient $\frac{\partial\mathcal{L}^{d}_{i}}{\partial D^{a}_{i}}$ on the bias
discriminator $D^{a}_{i}$ and
$\frac{\partial\mathcal{L}^{d}_{i}}{\partial\textbf{a}_{i}}$ on protected
representation $\textbf{a}_{i}$. The former is used to optimize $D^{a}_{i}$.
The latter is distributed to the server $P^{w}$ to calculate gradient on
$A_{i}$: $\frac{\partial\mathcal{L}^{a}_{i}}{\partial
A_{i}}=\frac{\partial\mathcal{L}^{a}_{i}}{\partial\textbf{a}_{i}}\frac{\partial\textbf{a}_{i}}{\partial
A_{i}}$, which is used to optimize $A_{i}$. Next, the platform $P^{a}_{i}$
calculates the adversarial gradient
$\frac{\partial\mathcal{L}^{a}_{i}}{\partial\textbf{a}_{i}}$ on
$\textbf{a}_{i}$. $\frac{\partial\mathcal{L}^{a}_{i}}{\partial\textbf{a}_{i}}$
is further distributed to $P^{w}$ to calculate adversarial gradients
$\frac{\partial\mathcal{L}^{a}_{i}}{\partial\textbf{s}}$ on the unified
representation s based on gradient back propagation:
$\frac{\partial\mathcal{L}^{a}_{i}}{\partial\textbf{s}}=\frac{\partial\mathcal{L}^{a}_{i}}{\partial\textbf{a}_{i}}\frac{\partial\textbf{a}_{i}}{\partial\textbf{s}}$.
Until now, the server $P^{w}$ can further calculate overall gradients
$\frac{\partial\mathcal{L}}{\partial\textbf{s}}=\frac{\partial\mathcal{L}^{t}}{\partial\textbf{s}}-\sum_{i=1}^{m}\lambda_{i}\frac{\partial\mathcal{L}^{a}_{i}}{\partial\textbf{s}}$
on the unified representation s. The server further calculates gradient
$\frac{\partial\mathcal{L}}{\partial\mathcal{M}^{w}}$ on the aggregation
model:
$\frac{\partial\mathcal{L}}{\partial\mathcal{M}^{w}}=\frac{\partial\mathcal{L}}{\partial\textbf{s}}\frac{\partial\textbf{s}}{\partial\mathcal{M}^{a}}$
and the gradient $\frac{\partial\mathcal{L}}{\partial\textbf{s}^{l}_{i}}$ on
each local representation $\textbf{s}^{l}_{i}$:
$\frac{\partial\mathcal{L}}{\partial\textbf{s}^{l}_{i}}=\frac{\partial\mathcal{L}}{\partial\textbf{s}}\frac{\partial\textbf{s}}{\partial\textbf{s}^{l}_{i}}$.
Finally, gradients on local representations are distributed to corresponding
platforms $\\{P^{b}_{i}|i=1,2,...,n\\}$ to optimize local models
$\\{\mathcal{M}^{l}_{i}|i=1,2,...,n\\}$. The detailed process of a training
round in FairVFL is in Algorithm 1 (Supplementary).

###  3.5 Discussion on Privacy and Efficiency

Privacy: In FairVFL, each data platform locally keeps its private data and
never shares it with the outside, which can protect user privacy to some
extent. Besides, for the vertical federated model learning and inference, the
intermediate model results, i.e., data representations and model gradients,
need to be exchanged across platforms, which may lead to potential privacy
leakage. To tackle this challenge, we propose a contrastive adversarial
learning method to protect private information in data representations, which
can compare data representations with the same attribute and remove the
attribute-irrelated information in them. To further protect private
information in gradients, we can apply differential privacy techniques to
perturb exchanged gradients as standard VFL methods [48].

Efficiency: FairVFL needs multiple times of communications in a single
training round, which may arouse concerns on communication efficiency.
Fortunately, the extra communication costs are usually small. This is because
in the extra communication rounds FairVFL only needs to exchange protected
representations and their gradients, and the extra communication cost is
$\mathcal{O}(4mEH)$, where $m$ is the number of fairness-sensitive platforms,
$H$ denotes the dimensionality of protected representations, and $E$ denotes
the batch size. Since $M$, $E$, and $B$ is usually small in practice, the
corresponding cost is usually minor. For the computation efficiency, compared
with the standard VFL framework, FairVFL only increases the computation cost
of the server and platforms with fairness-sensitive attributes. They need to
extra compute the gradients on the attribute discriminators and contrastive
discriminators. Thus, the computation complexities of FairVFL and standard VFL
are $\mathcal{O}(d^{3}+D^{3})$ and $\mathcal{O}(D^{3})$ respectively, where
$d$ and $D$ denote the parameter sizes of the discriminators and the main task
model. In widely used settings, the discriminator is usually implemented by a
small FFNN network, whose parameters are usually much less than the main task
model. Thus, the computation efficiency of FairVFL is still comparable with
standard VFL methods.

##  4 Experiment

###  4.1 Datasets and Experimental Settings

We evaluate model performance and fairness on three real-world datasets. The
first one is ADULT [25], which is a widely used public dataset for fair ML
[53, 21, 36, 26]. It contains rich feature fields (e.g., education and social
relation) for income prediction. 20,000 randomly selected data samples are
used to construct training and validation dataset, and 10,000 randomly
selected data samples are used to construct test dataset. Following existing
works [46, 47], we treat gender and age as fairness-sensitive features and
utilize other feature fields for income prediction. Besides, we randomly
partition 12 types of feature fields in ADULT into three platforms to simulate
the VFL scenarios444FairVFL is feature partition agnostic and does not rely on
specific partition strategy.. The second one is NEWS, which is constructed by
user logs (i.e., news clicks, search, and web-page browsing logs) on Microsoft
News and Bing Search platforms during 6 weeks (June 23th - July 20th, 2019)
for news recommendation. These three types of data are decentralized on three
platforms under VFL settings. We randomly select 100,000 news impressions in
the first three weeks to construct the training and validation set and select
100,000 news impressions in the last week to construct the test set. Besides,
we also treat gender and age as fairness-sensitive attributes in NEWS. The
third one is CelebA [31], which is a public face attributes dataset. To
simulate the VFL setting, we use the raw images and a part of the raw
attributes (i.e., the “Straight Hair” attribute and “Wavy Hair”) as the input
features. The raw images are stored in a data platform and other input
attributes are stored in another platform. The target task is to classify
whether the input sample has the attribute “Smiling”. We also treat gender
(the “Male” attribute) as the target fairness-sensitive attribute.

In FairVFL, the local representations and unified representations are
$400$-dimensional. The protected representations for gender and age are $32$\-
and $64$-dimensional, respectively. The weights of contrastive adversarial
loss for different sensitive features are set to $0.25$. The weights of
adversarial loss for gender and age are set to $\frac{2}{3}$ and $\frac{2}{3}$
on NEWS, respectively. Besides, the weights of adversarial loss for gender and
age are set to $1e2$ and $1e1$ on ADULT, respectively. We exploit Adam
algorithm [24] for model optimization with 1$e$-4 learning rate. The size of
the mini-batch for model training is set to $32$. Besides, for both age and
gender, we set $E_{i}$ to $5$ for simplification. We also use the dropout
technique [39] with a $0.2$ drop probability to alleviate model overfitting.
Hyper-parameters are selected based on the validation dataset. Codes are
available in https://github.com/taoqi98/FairVFL.

###  4.2 Performance and Fairness Evaluation

Remark that we can achieve counterfactual fairness by reducing fairness-
sensitive features encoded in data representations that are used for the model
predictions [28]. Thus, following existing works [13, 5, 47, 28], we utilize
the ensemble of $5$ attackers to predict fairness-sensitive features from
unified representations s to evaluate the independence of fairness-sensitive
features and data representations to verify model fairness. If attackers can
distinguish fairness-sensitive bias from unified representations then we can
say the model is unfair. Thus, a lower F1 score on the fairness-sensitive
feature classification task means better model fairness. Besides, we use
accuracy and F1 score to evaluate the classification task on ADULT and CelebA.
Moreover, following these works [2, 42], we use AUC and nDCG@10 to evaluate
the news recommendation task on NEWS. Each experiment is repeated five times
and we report the average performance.

Table 1: Model performance and fairness on ADULT and NEWS.

Training Strategy | ADULT | NEWS
---|---|---
Model | Income Prediction | Model Fairness | Model | News Recommendation | Model Fairness
Accuracy | F1 | Gender F1 | Age F1 | AUC | nDCG@10 | Gender F1 | Age F1
CenTrain | MLP | 82.15$\pm$0.86 | 78.42$\pm$0.66 | 78.27$\pm$1.60 | 47.47$\pm$0.90 | NAML | 64.04$\pm$0.13 | 30.80$\pm$0.13 | 70.05$\pm$0.21 | 20.01$\pm$3.08
TabNet | 82.23$\pm$1.02 | 78.50$\pm$0.80 | 79.07$\pm$1.79 | 49.12$\pm$1.24 | LSTUR | 64.68$\pm$0.33 | 30.97$\pm$0.20 | 70.45$\pm$0.37 | 20.00$\pm$0.39
AutoInt | 82.31$\pm$1.92 | 78.49$\pm$1.50 | 79.22$\pm$0.84 | 48.99$\pm$1.25 | NRMS | 64.24$\pm$0.18 | 30.78$\pm$0.11 | 70.25$\pm$0.24 | 21.07$\pm$0.81
FairGo | MLP | 77.97$\pm$1.34 | 78.49$\pm$1.25 | 50.92$\pm$6.25 | 15.92$\pm$3.66 | NAML | 60.73$\pm$0.25 | 28.30$\pm$0.16 | 54.08$\pm$5.97 | 15.93$\pm$1.65
TabNet | 74.40$\pm$1.71 | 75.33$\pm$1.47 | 50.28$\pm$5.32 | 15.63$\pm$3.14 | LSTUR | 61.03$\pm$0.24 | 28.31$\pm$0.15 | 53.57$\pm$3.88 | 15.74$\pm$1.22
AutoInt | 76.89$\pm$1.50 | 77.31$\pm$1.27 | 50.59$\pm$4.41 | 15.73$\pm$2.27 | NRMS | 61.49$\pm$0.37 | 28.83$\pm$0.31 | 53.81$\pm$2.94 | 16.45$\pm$1.49
FairSM | MLP | 77.57$\pm$1.39 | 78.31$\pm$1.06 | 50.66$\pm$6.43 | 15.55$\pm$3.10 | NAML | 60.59$\pm$0.19 | 28.15$\pm$0.16 | 54.13$\pm$5.38 | 15.74$\pm$1.52
TabNet | 74.04$\pm$1.66 | 75.07$\pm$1.39 | 50.61$\pm$4.44 | 15.98$\pm$3.14 | LSTUR | 61.11$\pm$0.54 | 28.33$\pm$0.34 | 53.16$\pm$4.75 | 15.24$\pm$1.84
AutoInt | 76.30$\pm$2.56 | 76.88$\pm$2.05 | 50.53$\pm$5.02 | 15.50$\pm$3.10 | NRMS | 61.78$\pm$0.31 | 28.95$\pm$0.22 | 54.10$\pm$2.42 | 15.91$\pm$1.99
FairRec | MLP | 77.59$\pm$1.42 | 78.08$\pm$1.24 | 50.94$\pm$7.15 | 15.75$\pm$4.62 | NAML | 60.69$\pm$0.22 | 28.21$\pm$0.17 | 54.47$\pm$2.79 | 15.67$\pm$1.90
TabNet | 74.89$\pm$1.76 | 75.65$\pm$1.61 | 50.35$\pm$5.79 | 15.61$\pm$3.18 | LSTUR | 60.99$\pm$0.78 | 28.31$\pm$0.47 | 53.36$\pm$1.37 | 15.65$\pm$0.93
AutoInt | 76.60$\pm$1.91 | 77.17$\pm$1.54 | 50.43$\pm$7.01 | 15.83$\pm$3.96 | NRMS | 61.44$\pm$0.16 | 28.76$\pm$0.21 | 53.60$\pm$2.84 | 16.26$\pm$2.39
VFL | MLP | 81.47$\pm$2.14 | 77.82$\pm$1.57 | 79.05$\pm$0.91 | 47.48$\pm$1.25 | NAML | 63.93$\pm$0.45 | 30.75$\pm$0.45 | 69.72$\pm$0.48 | 20.09$\pm$0.86
TabNet | 81.77$\pm$1.72 | 78.09$\pm$1.35 | 78.75$\pm$0.77 | 48.36$\pm$1.32 | LSTUR | 64.39$\pm$0.32 | 30.85$\pm$0.19 | 70.07$\pm$0.37 | 19.92$\pm$1.63
AutoInt | 81.65$\pm$1.52 | 78.02$\pm$1.17 | 79.07$\pm$1.42 | 47.98$\pm$1.51 | NRMS | 64.38$\pm$0.13 | 30.93$\pm$0.11 | 70.67$\pm$0.23 | 21.41$\pm$0.66
FairVFL | MLP | 76.74$\pm$2.64 | 77.87$\pm$2.14 | 50.31$\pm$4.99 | 15.50$\pm$4.33 | NAML | 60.41$\pm$0.18 | 27.95$\pm$0.18 | 53.38$\pm$4.40 | 15.55$\pm$1.41
TabNet | 75.51$\pm$0.69 | 76.06$\pm$0.60 | 50.72$\pm$5.72 | 15.48$\pm$2.46 | LSTUR | 60.98$\pm$0.28 | 28.25$\pm$0.36 | 53.51$\pm$3.41 | 15.23$\pm$0.94
AutoInt | 76.19$\pm$0.99 | 76.86$\pm$0.85 | 50.53$\pm$4.48 | 15.22$\pm$2.93 | NRMS | 61.43$\pm$0.13 | 28.81$\pm$0.08 | 53.33$\pm$2.35 | 15.98$\pm$1.94
|  |  |  |  |  |  |  |  |  |

Table 2: Model performance and fairness on CelebA.

Method | Main Task | Fairness
---|---|---
Accuracy | F1 | GenderF1
CenTrain | 0.9180$\pm$0.0013 | 0.9180$\pm$0.0014 | 0.8706$\pm$0.0069
FairGo | 0.9096$\pm$0.0086 | 0.9095$\pm$0.0088 | 0.5159$\pm$0.0570
FairSM | 0.9032$\pm$0.0230 | 0.9027$\pm$0.0242 | 0.5259$\pm$0.0807
FairRec | 0.9070$\pm$0.0094 | 0.9067$\pm$0.0097 | 0.5241$\pm$0.0743
VFL | 0.9192$\pm$0.0020 | 0.9174$\pm$0.0021 | 0.8727$\pm$0.0056
FairVFL | 0.9045$\pm$0.0130 | 0.9042$\pm$0.0135 | 0.5143$\pm$0.0778
|  |  |

We compare our FairVFL method with several recent representative baseline
methods for fair representation learning: (1) FairGo [47]: a user-centric fair
representation learning framework that utilizes adversarial learning to reduce
fairness-sensitive bias information in representations; (2) FairSM [28]: a
unified framework to achieve counterfactual fairness on representations; (3)
FairRec [46]: an adversarial representation decomposition framework that can
decompose bias and bias-insensitive information. These methods rely on
centralized storage of features to learn fair ML models. Besides, both FairVFL
and these three methods are general frameworks and can be applied to basic
models to improve their fairness. In addition, baseline methods also include a
standard vertical federated representation learning method (VFL) without
fairness criterion [8]. Thus, to compare their effectiveness, we combine them
with several basic ML models on the three datasets. Basic models for ADULT are
implemented by three SOTA models for modeling structural features, including
MLP [18], TabNet [12] and AutoInt [38]. Basic models for NEWS are implemented
by SOTA news recommendation models, including NAML [42], LSTUR [2] and NRMS
[45]. Moreover, the VFL model for CelebA is based on ResNet-10 [20]. Due to
space limitations, detailed settings of these basic models are summarized in
the Supplementary.

Results on ADULT and NEWS are summarized in Table 1, and results on CelebA are
summarized in Table 2, from which we have several observations. First, basic
models based on vertical federated training (e.g., MLP+VFL) can achieve
similar performance with the same basic models based on centralized training
(e.g., MLP). These results show that the vertical federated learning technique
can effectively enable ML models to utilize decentralized feature fields for
target tasks without the disclose of raw data. Second, models without the
protection of fair ML methods (e.g., MLP and MLP+VFL) are biased by fairness-
sensitive features. This is because data may encode bias on sensitive features
like genders. Models may inherit bias from data and become unfair to some user
groups. Third, after applying our FairVFL method to basic models, the fairness
of these models effectively improves. This is because our FairVFL method
applies the adversarial learning on bias discrimination to unified
representations, which can effectively reduce bias encoded in them and improve
model fairness. Fourth, compared with models protected by centralized fair
machine learning methods (i.e., FairGo, FairSM and FairRec), models protected
by FairVFL can achieve similar performance and fairness. These results verify
that our FairVFL method can effectively improve the fairness of VFL models.

###  4.3 Ablation Study

In this section, we first evaluate the effectiveness of adversarial learning
in improving model fairness. Due to space limitations, we only show results on
ADULT in the following sections. Results are shown in Fig. 3, from which we
have several observations. First, after removing adversarial learning on
gender discrimination, model fairness on user gender seriously drops. This is
because users of the same gender usually have similar behavior patterns and
there may exist gender bias in data. Models may inherit gender bias from
implicit feature correlations in data and become unfair to some user groups.
To tackle this challenge, FairVFL applies adversarial learning on bias
discrimination to unified representations to reduce gender bias encoded in
them. Second, removing adversarial learning for age discrimination also
seriously hurts model fairness on user age, which further verifies the
effectiveness of adversarial learning for improving model fairness.

Figure 3: Ablation study on adversarial learning and contrastive adversarial
learning. CAL denotes contrastive adversarial learning and Random denotes
predicting private data randomly.

Next, we verify the effectiveness of contrastive adversarial learning on
privacy protection. Since unified representations include private information
of fairness-insensitive features on local platforms, communicating them across
platforms may arouse privacy concerns. Motivated by membership inference
attack methods, we evaluate the privacy protection by inferring the input
features of a unified representation from corresponding protected
representations $\\{\textbf{a}_{i}|i=1,...,m\\}$ that are shared across
platforms $\\{P^{a}_{i}|i=1,...,m\\}$. Specifically, based on the ADULT
dataset, we infer the value of input features from each $\textbf{a}_{i}$ via
$5$ attackers and average the results. A lower inference accuracy (the F1
value) means better privacy protection. We remark that for fairness
evaluation, we distinguish bias on fairness-sensitive features from unified
representations instead of the shared representations, and these sensitive
features are also not the input of representation learning. We only show
results on two representative feature fields (i.e., education, and social
relation) in Fig. 3 due to space limitations. First, without the protection of
contrastive adversarial learning, shared representations are very informative
for inferring private user information. These results verify that directly
sharing unified representations across platforms may leak user privacy.
Second, contrastive adversarial learning can effectively protect user privacy
in protected representations. This is because contrastive adversarial learning
enforces protected representations to be indistinguishable for predicting its
preimage from unified representations of the same fairness-sensitive features,
which can effectively reduce bias-irrelevant personal information in shared
representations.

Figure 4: Influence of contrastive adversarial learning weights on privacy
protection and model fairness, where contrastive adversarial loss weights for
different sensitive features are set to $\gamma_{c}$.

###  4.4 Hyper-Parameter Analysis

Next, we evaluate the influence of the weights of contrastive adversarial
learning $\\{\gamma_{i}|i=1,2,...,m\\}$ on both privacy protection and model
fairness. To simplify the analysis, we set them to the same value (denoted as
$\gamma_{c}$). The influence of $\gamma_{c}$ on privacy protection and model
fairness is shown in Fig. 4, from which we have two observations. First, with
the increase of $\gamma_{c}$, FairVFL can achieve better privacy protection
ability. This is intuitive since larger $\gamma_{c}$ makes contrastive
adversarial gradients more important to update the mappers, making them can
reduce user privacy in unified representations more effectively. Second,
larger $\gamma_{c}$ also hurts model fairness more seriously. This is because
larger $\gamma_{c}$ makes mappers pay more attention to reducing user private
information, which may hurt their abilities on retaining information of
fairness-sensitive bias information. Thus, the protected representations may
contain less bias information and hurt the effectiveness of adversarial
learning in improving model fairness. Fortunately, when $\gamma_{c}$ is small
(e.g., 0.25), FairVFL is effective in privacy protection since the performance
of user privacy inference is close to random. Besides, FairVFL can achieve
satisfied model fairness (about 0.5 and 0.15 F1 value for gender and age
prediction) when $\gamma_{c}$ is small (e.g., 0.25). This indicates that
FairVFL can achieve effective privacy protection ability without hurting model
fairness and we set $\gamma_{c}$ to $0.25$ in FairVFL.

##  5 CONCLUSION

In this paper, we propose a fair vertical federated learning framework (named
FairVFL), which can improve the fairness of VFL models in a privacy-preserving
way. The core of FairVFL is to learn fair and unified representations to
encode samples based on their decentralized feature fields. Each platform
storing fairness-insensitive features first learns local representations from
their local features. Then these local representations are uploaded to a
server to form a unified representation used for the target task. In order to
further learn fair and unified representations, we send them to platforms
storing fairness-sensitive features and apply adversarial learning to reduce
bias encoded in them. Besides, to protect shared representations from leaking
user privacy, we propose a contrastive adversarial learning method to reduce
user private information in them before sharing them with other platforms.
Experimental results on three datasets show that FairVFL is comparable with
centralized baseline fair ML methods in terms of both performance and
fairness, meanwhile effectively protects user privacy.

## Acknowledgments

This work was supported by the National Key R&D Program of China
2021ZD0113902, the National Natural Science Foundation of China under Grant
numbers 82090053, U1936126, and Tsinghua-Toyota Joint Research Fund
20213930033.

## References

  * [1]
  * An et al. [2019] Mingxiao An, Fangzhao Wu, Chuhan Wu, Kun Zhang, Zheng Liu, and Xing Xie. 2019\.  Neural news recommendation with long-and short-term user representations. In _ACL_. 336–345.
  * Arık and Pfister [2020] Sercan O Arık and Tomas Pfister. 2020.  Tabnet: Attentive interpretable tabular learning. In _AAAI_. 6679–6687.
  * Beutel et al. [2019] Alex Beutel, Jilin Chen, Tulsee Doshi, Hai Qian, Li Wei, Yi Wu, Lukasz Heldt, Zhe Zhao, Lichan Hong, Ed H Chi, et al. 2019\.  Fairness in recommendation ranking through pairwise comparisons. In _KDD_. 2212–2220.
  * Bose and Hamilton [2019] Avishek Bose and William Hamilton. 2019.  Compositional fairness constraints for graph embeddings. In _ICML_. 715–724.
  * Burke et al. [2018] Robin Burke, Nasim Sonboli, and Aldo Ordonez-Gauger. 2018\.  Balanced neighborhoods for multi-sided fairness in recommendation. In _ACM FAccT_. 202–214.
  * Calmon et al. [2017] Flavio P Calmon, Dennis Wei, Bhanukiran Vinzamuri, Karthikeyan Natesan Ramamurthy, and Kush R Varshney. 2017.  Optimized pre-processing for discrimination prevention. In _NIPS_. 3995–4004.
  * Che et al. [2021] Sicong Che, Hao Peng, Lichao Sun, Yong Chen, and Lifang He. 2021.  Federated Multi-View Learning for Private Medical Data Integration and Analysis.  _TIST_ (2021).
  * Cheng et al. [2021] Kewei Cheng, Tao Fan, Yilun Jin, Yang Liu, Tianjian Chen, Dimitrios Papadopoulos, and Qiang Yang. 2021.  Secureboost: A lossless federated learning framework.  _IEEE Intelligent Systems_ (2021).
  * Cheng et al. [2020] Yong Cheng, Yang Liu, Tianjian Chen, and Qiang Yang. 2020\.  Federated learning for privacy-preserving AI.  _Commun. ACM_ (2020), 33–36.
  * Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _NAACL_. 4171–4186.
  * Dwivedi et al. [2017] Vijay Prakash Dwivedi, Deepak Kumar Singh, Saurav Jha, et al. 2017\.  Gender Classification of Blog Authors: With Feature Engineering and Deep Learning using LSTM Networks. In _ICoAC_. 142–148.
  * Elazar and Goldberg [2018] Yanai Elazar and Yoav Goldberg. 2018.  Adversarial Removal of Demographic Attributes from Text Data. In _EMNLP_. 11–21.
  * Feldman et al. [2015] Michael Feldman, Sorelle A Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubramanian. 2015\.  Certifying and removing disparate impact. In _KDD_. 259–268.
  * Fu et al. [2021] Fangcheng Fu, Yingxia Shao, Lele Yu, Jiawei Jiang, Huanran Xue, Yangyu Tao, and Bin Cui. 2021.  VF2Boost: Very Fast Vertical Federated Gradient Boosting for Cross-Enterprise Learning. In _SIGMOD_. 563–576.
  * Ge et al. [2021] Yingqiang Ge, Shuchang Liu, Ruoyuan Gao, Yikun Xian, Yunqi Li, Xiangyu Zhao, Changhua Pei, Fei Sun, Junfeng Ge, Wenwu Ou, et al. 2021\.  Towards Long-term Fairness in Recommendation. In _WSDM_. 445–453.
  * Goodfellow et al. [2020] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2020\.  Generative adversarial networks.  _Commun. ACM_ (2020), 139–144.
  * Gorishniy et al. [2021] Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. 2021.  Revisiting Deep Learning Models for Tabular Data.  _arXiv preprint arXiv:2106.11959_ (2021).
  * Hardt et al. [2016] Moritz Hardt, Eric Price, and Nati Srebro. 2016.  Equality of opportunity in supervised learning. In _NIPS_. 3315–3323.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016\.  Deep residual learning for image recognition. In _CVPR_. 770–778.
  * Huang and Vishnoi [2019] Lingxiao Huang and Nisheeth Vishnoi. 2019.  Stable and fair classification. In _ICML_. 2879–2890.
  * Jiang et al. [2019] Di Jiang, Yuanfeng Song, Yongxin Tong, Xueyang Wu, Weiwei Zhao, Qian Xu, and Qiang Yang. 2019.  Federated Topic Modeling. In _CIKM_. 1071–1080.
  * Khademi et al. [2019] Aria Khademi, Sanghack Lee, David Foley, and Vasant Honavar. 2019\.  Fairness in algorithmic decision making: An excursion through the lens of causality. In _WWW_. 2907–2914.
  * Kingma and Ba [2015] Diederik P Kingma and Jimmy Ba. 2015.  Adam: A Method for Stochastic Optimization. In _ICLR_.
  * Kohavi et al. [1996] Ron Kohavi et al. 1996\.  Scaling up the accuracy of naive-bayes classifiers: A decision-tree hybrid.. In _KDD_. 202–207.
  * Lee et al. [2021] Joshua K Lee, Yuheng Bu, Deepta Rajan, Prasanna Sattigeri, Rameswar Panda, Subhro Das, and Gregory W Wornell. 2021.  Fair Selective Classification via Sufficiency. In _ICML_. 6076–6086.
  * Li et al. [2021a] Yunqi Li, Hanxiong Chen, Zuohui Fu, Yingqiang Ge, and Yongfeng Zhang. 2021a.  User-oriented Fairness in Recommendation. In _WWW_. 624–632.
  * Li et al. [2021b] Yunqi Li, Hanxiong Chen, Shuyuan Xu, Yingqiang Ge, and Yongfeng Zhang. 2021b.  Towards Personalized Fairness based on Causal Notion. In _SIGIR_. 1054–1063.
  * Liu et al. [2021] Changxin Liu, Zirui Zhou, Yang Shi, Jian Pei, Lingyang Chu, and Yong Zhang. 2021\.  Achieving Model Fairness in Vertical Federated Learning.  _arXiv preprint arXiv:2109.08344_ (2021).
  * Liu et al. [2020] Fenglin Liu, Xian Wu, Shen Ge, Wei Fan, and Yuexian Zou. 2020.  Federated learning for vision-and-language grounding problems. In _AAAI_. 11572–11579.
  * Liu et al. [2015] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. 2015\.  Deep learning face attributes in the wild. In _ICCV_. 3730–3738.
  * Luo et al. [2021] Xinjian Luo, Yuncheng Wu, Xiaokui Xiao, and Beng Chin Ooi. 2021\.  Feature inference attack on model predictions in vertical federated learning. In _ICDE_. 181–192.
  * Luong et al. [2011] Binh Thanh Luong, Salvatore Ruggieri, and Franco Turini. 2011\.  k-NN as an implementation of situation testing for discrimination discovery and prevention. In _KDD_. 502–510.
  * Lyu et al. [2020a] Lingjuan Lyu, Xuanli He, and Yitong Li. 2020a.  Differentially private representation for nlp: Formal guarantee and an empirical study on privacy and fairness. In _EMNLP_. 2355–2365.
  * Lyu et al. [2020b] Lingjuan Lyu, Yitong Li, Xuanli He, and Tong Xiao. 2020b.  Towards differentially private text representations. In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_. 1813–1816.
  * Madras et al. [2018] David Madras, Elliot Creager, Toniann Pitassi, and Richard Zemel. 2018.  Learning adversarially fair and transferable representations. In _ICML_. 3384–3393.
  * McMahan et al. [2017] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. 2017\.  Communication-Efficient Learning of Deep Networks from Decentralized Data. In _AISTATS_. 1273–1282.
  * Song et al. [2019] Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, and Jian Tang. 2019.  Autoint: Automatic feature interaction learning via self-attentive neural networks. In _CIKM_. 1161–1170.
  * Srivastava et al. [2014] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014.  Dropout: A simple way to prevent neural networks from overfitting.  _JMLR_ (2014), 1929–1958.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need. In _NIPS_. 6000–6010.
  * Weng et al. [2020] Haiqin Weng, Juntao Zhang, Feng Xue, Tao Wei, Shouling Ji, and Zhiyuan Zong. 2020\.  Privacy leakage of real-world vertical federated learning.  _arXiv preprint arXiv:2011.09290_ (2020).
  * Wu et al. [2019a] Chuhan Wu, Fangzhao Wu, Mingxiao An, Jianqiang Huang, Yongfeng Huang, and Xing Xie. 2019a.  Neural news recommendation with attentive multi-view learning. In _IJCAI_. 3863–3869.
  * Wu et al. [2019b] Chuhan Wu, Fangzhao Wu, Mingxiao An, Tao Qi, Jianqiang Huang, Yongfeng Huang, and Xing Xie. 2019b.  Neural news recommendation with heterogeneous user behavior. In _EMNLP_. 4876–4885.
  * Wu et al. [2020b] Chuhan Wu, Fangzhao Wu, Tao Di, Yongfeng Huang, and Xing Xie. 2020b.  FedCTR: Federated Native Ad CTR Prediction with Multi-Platform User Behavior Data.  _arXiv preprint arXiv:2007.12135_ (2020).
  * Wu et al. [2019c] Chuhan Wu, Fangzhao Wu, Suyu Ge, Tao Qi, Yongfeng Huang, and Xing Xie. 2019c.  Neural news recommendation with multi-head self-attention. In _EMNLP_. 6390–6395.
  * Wu et al. [2021b] Chuhan Wu, Fangzhao Wu, Xiting Wang, Yongfeng Huang, and Xing Xie. 2021b.  FairRec:Fairness-aware News Recommendation with Decomposed Adversarial Learning. In _AAAI_. 4462–4469.
  * Wu et al. [2021a] Le Wu, Lei Chen, Pengyang Shao, Richang Hong, Xiting Wang, and Meng Wang. 2021a.  Learning Fair Representations for Recommendation: A Graph-based Perspective. In _WWW_. 2198–2208.
  * Wu et al. [2020a] Yuncheng Wu, Shaofeng Cai, Xiaokui Xiao, Gang Chen, and Beng Chin Ooi. 2020a.  Privacy preserving vertical federated learning for tree-based models. In _VLDB_. 2090–2103.
  * Yang et al. [2019a] Kai Yang, Tao Fan, Tianjian Chen, Yuanming Shi, and Qiang Yang. 2019a.  A quasi-newton method based vertical federated learning framework for logistic regression.  _arXiv preprint arXiv:1912.00513_ (2019).
  * Yang et al. [2019b] Shengwen Yang, Bing Ren, Xuhui Zhou, and Liping Liu. 2019b.  Parallel distributed logistic regression for vertical federated learning without third-party coordinator.  _arXiv preprint arXiv:1911.09824_ (2019).
  * Yao et al. [2021] Jing Yao, Zhicheng Dou, and Ji-Rong Wen. 2021.  FedPS: A Privacy Protection Enhanced Personalized Search Framework. In _WWW_. 3757–3766.
  * Zafar et al. [2019] Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez-Rodriguez, and Krishna P Gummadi. 2019.  Fairness constraints: A flexible approach for fair classification.  _JMLR_ (2019), 2737–2778.
  * Zemel et al. [2013] Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. 2013.  Learning fair representations. In _ICML_. 325–333.
  * Zhang et al. [2021] Qingsong Zhang, Bin Gu, Cheng Deng, and Heng Huang. 2021\.  Secure Bilevel Asynchronous Vertical Federated Learning with Backward Updating. In _AAAI_. 10896–10904.

## Checklist

  1. 1.

For all authors…

    1. (a)

Do the main claims made in the abstract and introduction accurately reflect
the paper’s contributions and scope? [Yes]  See Section 1.

    2. (b)

Did you describe the limitations of your work? [Yes]  See Supplementary
Information.

    3. (c)

Did you discuss any potential negative societal impacts of your work? [N/A]
Our work focuses on privacy protection and model fairness.

    4. (d)

Have you read the ethics review guidelines and ensured that your paper
conforms to them? [Yes]

  2. 2.

If you are including theoretical results…

    1. (a)

Did you state the full set of assumptions of all theoretical results? [N/A]

    2. (b)

Did you include complete proofs of all theoretical results? [N/A]

  3. 3.

If you ran experiments…

    1. (a)

Did you include the code, data, and instructions needed to reproduce the main
experimental results (either in the supplemental material or as a URL)? [Yes]
Section 4.1 and Supplementary Information.

    2. (b)

Did you specify all the training details (e.g., data splits, hyperparameters,
how they were chosen)? [Yes]  See Section 4.1 and Supplementary Information.

    3. (c)

Did you report error bars (e.g., with respect to the random seed after running
experiments multiple times)? [Yes]  See Section 4.1.

    4. (d)

Did you include the total amount of compute and the type of resources used
(e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]  See
Supplementary Information.

  4. 4.

If you are using existing assets (e.g., code, data, models) or
curating/releasing new assets…

    1. (a)

If your work uses existing assets, did you cite the creators? [Yes]  See
Section 4.1.

    2. (b)

Did you mention the license of the assets? [Yes]  See Supplementary
Information.

    3. (c)

Did you include any new assets either in the supplemental material or as a
URL? [Yes]  See Section 4.1.

    4. (d)

Did you discuss whether and how consent was obtained from people whose data
you’re using/curating? [Yes]  See Supplementary Information.

    5. (e)

Did you discuss whether the data you are using/curating contains personally
identifiable information or offensive content? [Yes]  See Supplementary
Information.

  5. 5.

If you used crowdsourcing or conducted research with human subjects…

    1. (a)

Did you include the full text of instructions given to participants and
screenshots, if applicable? [N/A]

    2. (b)

Did you describe any potential participant risks, with links to Institutional
Review Board (IRB) approvals, if applicable? [N/A]

    3. (c)

Did you include the estimated hourly wage paid to participants and the total
amount spent on participant compensation? [N/A]

## Supplementary

### Federated Fair Model Training

The detailed workflow of our FairVFL method is summarized in Algorithm 1.

Algorithm 1 Federated Model Training in FairVFL

1: for all $i$ in $1,2,...,n$ do

2: Query $\textbf{s}^{l}_{i}$ from the platform $P^{b}_{i}$

3: end for

4: Apply $\mathcal{M}^{w}$ to learn s from
$\\{\textbf{s}^{l}_{i}|i=1,2,...,n\\}$ on $P^{w}$

5: Upload s to $P^{t}$ to calculate
$\frac{\partial\mathcal{L}^{t}}{\partial\textbf{s}}$ and
$\frac{\partial\mathcal{L}^{t}}{\partial\mathcal{M}^{t}}$

6: Use $\frac{\partial\mathcal{L}^{t}}{\partial\mathcal{M}^{t}}$ to update
$\mathcal{M}^{t}$ and distribute
$\frac{\partial\mathcal{L}^{t}}{\partial\textbf{s}}$ to $P^{w}$

7: for all $i$ in $1,2,...,m$ do

8: Map s to $\textbf{a}_{i}$ via $A_{i}$ on $P^{w}$

9: Select a contrastive negative sample $\textbf{s}^{-}_{i}$ on $P^{w}$

10: Calculate $\frac{\partial\mathcal{L}^{p}_{i}}{\partial D^{c}_{i}}$ based
on Eq. 1 to update $D^{c}_{i}$ on $P^{w}$

11: Calculate $\frac{\partial\mathcal{L}^{c}_{i}}{\partial A_{i}}$ based on
Eq. 3 to update $A_{i}$ on $P^{w}$

12: Recalculate $\textbf{a}_{i}$ via updated $A_{i}$

13: Upload $\textbf{a}_{i}$ from $P^{w}$ to $P^{a}_{i}$

14: Learn $\frac{\partial\mathcal{L}^{d}_{i}}{\partial D^{a}_{i}}$ based on
Eq. 5 to update $D^{a}_{i}$ on $P^{a}_{i}$

15: Learn $\frac{\partial\mathcal{L}^{d}_{i}}{\partial\textbf{a}_{i}}$ based
on Eq. 5 on $P^{a}_{i}$ and distribute it to $P^{w}$

16: Learn $\frac{\partial\mathcal{L}^{d}_{i}}{\partial A_{i}}$ via
$\frac{\partial\mathcal{L}^{d}}{\partial\textbf{a}_{i}}$ on $P^{w}$ to update
$A_{i}$

17: Recalculate $\textbf{a}_{i}$ via updated $A_{i}$ and upload it to
$P^{a}_{i}$

18: Learn $\frac{\partial\mathcal{L}^{a}_{i}}{\partial\textbf{a}_{i}}$ based
on Eq. 7 on $P^{a}_{i}$ and distribute it to $P^{w}$

19: Calculate $\frac{\partial\mathcal{L}^{a}_{i}}{\partial\textbf{s}}$ via
$\frac{\partial\mathcal{L}^{a}_{i}}{\partial\textbf{a}_{i}}$ on $P^{w}$

20: end for

21: Calculate $\frac{\partial\mathcal{L}}{\partial\textbf{s}}$ for s based on
Eq. 8 on $P^{w}$

22: Calculate $\frac{\partial\mathcal{L}}{\partial\mathcal{M}^{w}}$ via
$\frac{\partial\mathcal{L}}{\partial\textbf{s}}$ to update $\mathcal{M}^{w}$
on $P^{w}$

23: for all $i$ in $1,2,...,n$ do

24: Calculate $\frac{\partial\mathcal{L}}{\partial\textbf{s}^{l}_{i}}$ via
$\frac{\partial\mathcal{L}}{\partial\textbf{s}}$ on $P^{w}$ and distribute it
to $P^{b}_{i}$

25: Calculate $\frac{\partial\mathcal{L}}{\partial\mathcal{M}^{l}_{i}}$ via
$\frac{\partial\mathcal{L}}{\partial\textbf{s}^{l}_{i}}$ to update
$\mathcal{M}^{l}_{i}$ on $P^{b}_{i}$

26: end for

### Experimental Settings

Next, we will introduce the basic models used for model training in details.
Since ADULT is a tabular feature dataset, we implement three basic models for
modeling structural features [18] to predict income on ADULT: (1) MLP [18]:
converting user features into embedding vectors and using an MLP network for
income prediction. (2) TabNet [12]: using attentive feature transformer
networks to model relatedness of different features and build local
representations. (3) AutoInt [38]: applying multi-head self-attention network
to feature embeddings to model their interactions and learn their
representations. In these three methods, dimensions of feature embeddings are
set to 32, and dimensions of local representations are set to 400. In
addition, we choose three mainstream news recommendation models as basic
models for the news recommendation task on NEWS: (1) NAML [42]: applying an
attentive CNN network to learn behavior representations and an attention
network to learn user representations; (2) LSTUR [2]: proposing to learn
short-term user representations from recent user behaviors and long-term user
representations via user ID embeddings; (3) NRMS [45]: employing multi-head
self-attention networks to learn behavior and user representations. In these
three methods, three types of behavior representations (e.g., clicked news)
are set to 400-dimensional.

### Influence of Adversarial Learning

Figure 5: Influence of adversarial learning on model fairness, where
$\lambda_{g}$ and $\lambda_{a}$ are adversarial learning weights for gender
and age discrimination, respectively.

Next, we will analyze the influence of the weights of adversarial learning on
model fairness. We summarize results in Fig. 5 and have several observations.
First, with the increase of $\lambda_{g}$, model fairness on gender can
increase. This is because larger $\lambda_{g}$ makes models pay more attention
to reducing gender bias encoded in unified representations during model
training. Second, when $\lambda_{g}$ is large enough, model fairness on gender
becomes stable. This is because when $\lambda_{g}$ is large enough, gender
information in unified representations can be effectively reduced. Third, with
the increasing of $\lambda_{a}$, model fairness on age also first increases
and then converges. Similarly, this is because users with similar ages usually
have similar behaviors and may encode age bias in real-world data. Larger
$\lambda_{a}$ can more effectively prevent unified representations from
encoding age bias from data until age bias is effectively removed.

### Limitations and Future Work

Although FairVFL is effective in both model fairness and privacy protection,
FairVFL may have more communication latency than other baseline methods during
model training. This is because the adversarial learning and the contrastive
adversarial learning in FairVFL make it need to communicate with a fairness-
sensitive feature platform multiple times. In our future work, we plan to
study an asynchronous communication mechanism to reduce the communication
latency of FairVFL.