# FedAds: A Benchmark for Privacy-Preserving CVR Estimation with Vertical
Federated Learning

Penghui Wei  Alibaba Group<EMAIL_ADDRESS>,  Hongjian Dou  Alibaba
Group<EMAIL_ADDRESS>,  Shaoguo Liu∗ Alibaba Group
<EMAIL_ADDRESS>,  Rongjun Tang† The Chinese University of Hong
Kong, Shenzhen<EMAIL_ADDRESS>,  Li Liu  The Thrust of
Artificial Intelligence, The Hong Kong University of Science and Technology
(Guangzhou<EMAIL_ADDRESS>,  Liang Wang  Alibaba Group
<EMAIL_ADDRESS>and  Bo Zheng  Alibaba Group bozheng@alibaba-
inc.com

(2023)

###### Abstract.

Conversion rate (CVR) estimation aims to predict the probability of conversion
event after a user has clicked an ad. Typically, online publisher has user
browsing interests and click feedbacks, while demand-side advertising platform
collects users’ post-click behaviors such as dwell time and conversion
decisions. To estimate CVR accurately and protect data privacy better,
vertical federated learning (vFL) is a natural solution to combine two sides’
advantages for training models, without exchanging raw data. Both CVR
estimation and applied vFL algorithms have attracted increasing research
attentions. However, standardized and systematical evaluations are missing:
due to the lack of standardized datasets, existing studies adopt public
datasets to simulate a vFL setting via hand-crafted feature partition, which
brings challenges to fair comparison. We introduce FedAds, the first benchmark
for CVR estimation with vFL, to facilitate standardized and systematical
evaluations for vFL algorithms. It contains a large-scale real world dataset
collected from Alibaba’s advertising platform, as well as systematical
evaluations for both effectiveness and privacy aspects of various vFL
algorithms. Besides, we also explore to incorporate unaligned data in vFL to
improve effectiveness, and develop perturbation operations to protect privacy
well. We hope that future research work in vFL and CVR estimation benefits
from the FedAds benchmark.

Ad Ranking, Vertical Federated Learning, Deep Generative Model

†Work done during internship at Alibaba Group. ∗Correspondence to: S. Liu.

††journalyear: 2023††copyright: acmlicensed††conference: Proceedings of the
46th International ACM SIGIR Conference on Research and Development in
Information Retrieval; July 23–27, 2023; Taipei, Taiwan††booktitle:
Proceedings of the 46th International ACM SIGIR Conference on Research and
Development in Information Retrieval (SIGIR ’23), July 23–27, 2023, Taipei,
Taiwan††price: 15.00††doi: 10.1145/3539618.3591909††isbn:
978-1-4503-9408-6/23/07††ccs: Information systems Online advertising††ccs:
Computing methodologies Neural networks

##  1\. Introduction

Nowadays, more and more Web applications employ machine learning models to
provide personalized services for users to meet their preferences. Conversion
rate (CVR) estimation is a foundational module in online recommendation and
advertising systems, which aims to predict the probability of conversion event
(e.g., purchase behavior in e-commerce ads) after a user has clicked an
item/ad. It plays a crucial in candidate ranking and ad bidding strategies.

Figure 1 (a) gives an illustration of user behaviors in online advertising. If
a user browses an ad at the publisher page and then clicks on it, the user
will arrive at an ad landing page. Follow-up behaviors including conversion
decisions may happen. Therefore, the online publisher has user browsing
interests and click feedbacks, and the demand-side advertising platform
collects users’ post-click behaviors on ad pages such as dwell time and
conversion. CVR estimation models are trained using collected data about user
profiles, user behaviors and item/ad properties, with the goals of improving
user experiences and increasing platform revenue. However due to the potential
abuse and leak of user data, continued concerns regarding data privacy and
security are raised.

User behaviors collected by online publisher or advertising platform are
private information and should be protected. To estimate CVR accurately and
protect data privacy better, vertical federated learning (vFL) (Vepakomma et
al., 2018b; Yang et al., 2019) is a natural solution for model training, which
combines two sides’ advantages without exchanging raw data. As a branch of
federated learning (FL) (McMahan et al., 2017) techniques, vFL can
collaboratively train a model using feature-partitioned data from multiple
participants via exchanging intermediate results (e.g., hidden representations
and gradients) rather than raw data (e.g., features and labels). Figure 1 (b)
shows the vFL framework for training a neural network based CVR estimation
model. There are two participants, where the non-label party is online
publisher and the label party is advertising platform that owns conversion
labels. The training data for vFL is feature-partitioned: before model
training, the two participants first perform private set intersection (PSI)
(Kolesnikov et al., 2016) to obtain an aligned sample ID set, and each
sample’s features come from both non-label party (e.g., behaviors on publisher
page) and label party (e.g., behaviors on ad page).

The whole model is split into two submodels owned by non-label and label
parties respectively. During the forward pass, for a given input sample, the
non-label party’s submodel sends a hidden representation to the label party.
Then the label party combines such representation with its own one, and
produces the predicted conversion probability and cross-entropy loss according
to the sample’s conversion label. During the backward propagation, the label
party computes gradient w.r.t. the non-label party’s hidden representation and
sends it back, thus the update of non-label party’s submodel parameters
depends on the label party.

Figure 1. (a) The feedback behaviors after a user browses an ad. (b) Vertical
federated learning framework for training conversion rate estimation model.
Online publisher and advertising platform provide the collected user feedback
data and collaboratively train the model.

Both CVR estimation (Ma et al., 2018; Wei et al., 2021; Guo et al., 2021; Xu
et al., 2022) and applied vFL algorithms (Castiglia et al., [n.d.]; Kang et
al., 2022; Li et al., 2022d; He et al., 2022; Li et al., 2022c; Li et al.,
2022a; Sun et al., 2022; Fu et al., 2022) have attracted increasing research
attentions. Specifically, to improve privacy-preserving CVR estimation, two
research problems in vFL should be tackled. The first is about effectiveness:
Traditional vFL training procedure only employs aligned samples among multiple
participants, however the size of aligned samples is quite limited, which
restricts the model performance. Various approaches based on self-supervised
learning (Castiglia et al., [n.d.]; He et al., 2022) and semi-supervised
learning (Kang et al., 2022; Li et al., 2022d; Li et al., 2022c) are proposed
to explore the potential usage of a large amount of unaligned samples owned by
each party for improving vFL. The second is about privacy: Although vFL only
exchange intermediate results rather than raw features and labels, recent
studies revealed that it still suffers from privacy leakage risks such as
label inference attack, which means that a honest-but-curious non-label party
can successfully infer private labels. To defend against such attack, many
studies focus on random perturbation based approaches (Li et al., 2022a; Sun
et al., 2022; Fu et al., 2022) for protecting private label information.

However, standardized datasets and systematical evaluations for vFL algorithms
are missing: due to the lack of standardized datasets, existing studies
usually adopt public datasets to simulate a vFL experiment setting via hand-
crafted feature partition, rather than adopting datasets from real-world vFL
applications. This situation brings challenges to fair comparison of various
models and hinders further research of vFL and privacy-preserving CVR
estimation. As a result, above-mentioned vFL algorithms are not compared under
the same dataset and evaluation procedure currently. Therefore, there is a
great need for a comprehensive benchmark to facilitate vFL research.

In this paper we introduce FedAds, the first benchmark for privacy-preserving
CVR estimation with vFL, to facilitate systematical evaluations for vFL
algorithms. It contains 1) a large-scale real-world dataset from our online
advertising platform, collected from an ad delivery business relying on vFL-
based ranking models, as well as 2) systematical evaluations for both
effectiveness and privacy aspects of various neural network based vFL
algorithms through extensive experiments. Besides, to improve vFL
effectiveness, we explore to incorporate unaligned data via generating
unaligned samples’ feature representations using generative models. To protect
privacy well, we also develop perturbation based on mixup and projection
operations. We hope that future research work in both vFL algorithms and CVR
estimation benefits from our FedAds benchmark.

The main contributions of this work are:

  * •

We provide a real-world CVR estimation dataset collected from our ad delivery
business relying on vFL-based ranking models. To our knowledge, this is the
first large-scale dataset for vFL research.

  * •

We conduct systematical evaluations to recently proposed vFL algorithms for
effectiveness and privacy aspects respectively on the proposed dataset, which
promotes fair comparison of various studies.

  * •

We propose two approaches for incorporating unaligned samples and protecting
private label information in vFL respectively, and experiments on the proposed
dataset verify their performance.

##  2\. Preliminaries

###  2.1. Conversion Rate Estimation

The goal of a post-click conversion rate (CVR) estimation model $f(\cdot)$ is
to produce the conversion probability if a user has clicked an ad:
$\hat{p}_{\mathrm{CVR}}=f(\bm{x})$, where $\bm{x}$ denotes input feature of a
sample. The model $f(\cdot)$ is trained using click log
$\mathcal{D}=\left\\{\left(\bm{x},y\right)\right\\}$ with cross-entropy loss,
where $y\in\\{0,1\\}$ is the sample’s binary conversion label. CVR estimation
models are usually deployed to the ranking module of online advertising and
recommendation systems, and they are crucial for improving user experiences,
satisfying advertiser demands and increasing the revenue of advertising
platform.

###  2.2. Vertical Federated Learning

We first give a brief introduction to two-party vFL framework, and then
discuss the issues in traditional vFL.

####  2.2.1. Two-Party vFL

We consider the overall framework in Figure 1 (b). Without loss of generality,
we assume that there are two separate participants, namely non-label party and
label party. They cooperate with each other to learn a model
$f:\mathcal{X}\rightarrow\mathcal{Y}$. The feature space
$\mathcal{X}=\mathcal{X}_{\mathrm{N}}\cup\mathcal{X}_{\mathrm{L}}$ is composed
of two parts, where $\mathcal{X}_{\mathrm{N}}$ / $\mathcal{X}_{\mathrm{L}}$
represents the feature subspace owned by Non-label party / Label party. The
label party owns the one-hot label space $\mathcal{Y}$.

The whole model $f(\cdot)$ is split into two submodels $f_{\mathrm{N}}(\cdot)$
and $f_{\mathrm{L}}(\cdot)$ which are owned by non-label party and label-party
respectively:

(1) $f\coloneqq f_{\mathrm{N}}\circ f_{\mathrm{L}}\,,$

here $f_{\mathrm{N}}(\cdot)$ is the non-label party’s submodel which produces
hidden representation for each sample and sends it to label party, while
$f_{\mathrm{L}}(\cdot)$ is the label party’s submodel that produces the
predicted probability distribution.

Before model training, the two parties first perform PSI to obtain an aligned
sample set
$\mathcal{D}_{\mathrm{aligned}}=\\{(\bm{x}_{\mathrm{N}},\bm{x}_{\mathrm{L}},\bm{y})\\}$,
where $\bm{x}_{\mathrm{N}}$ and $\bm{x}_{\mathrm{L}}$ denote the sample’s
feature provided by two parties respectively, and its label $\bm{y}$ is a one-
hot vector. Consider the forward pass, for each sample the model $f(\cdot)$
outputs the predicted distribution $\hat{\bm{y}}$, and then computes the
cross-entropy loss $\mathcal{L}$:

(2)
$\bm{h}_{\mathrm{N}}=f_{\mathrm{N}}(\bm{x}_{\mathrm{N}})\,,\quad\bm{h}_{\mathrm{L}}=f_{\mathrm{L}}^{(\mathrm{b})}(\bm{x}_{\mathrm{L}})$
(3)
$\bm{l}=f_{\mathrm{L}}^{(\mathrm{t})}\bigl{(}\bm{h}_{\mathrm{N}},\bm{h}_{\mathrm{L}}\bigr{)}\,,\quad\hat{\bm{y}}=\mathrm{softmax}(\bm{l})$
(4) $\mathcal{L}=-\bm{y}^{\top}\log\hat{\bm{y}}$

where $\bm{h}_{\mathrm{N}}$ is the hidden representation that the non-label
party sends to the label party, and the output layer of
$f_{\mathrm{N}}(\cdot)$ is also known as cut layer. The label party’s submodel
$f_{\mathrm{L}}(\cdot)$ is composed of a bottom part
$f_{\mathrm{L}}^{(\mathrm{b})}(\cdot)$ and a top part
$f_{\mathrm{L}}^{(\mathrm{t})}(\cdot)$.

For the backward pass, the label party’s submodel $f_{\mathrm{L}}(\cdot)$ is
updated normally based on the gradient of loss $\mathcal{L}$ w.r.t. the
submodel parameters. To update the parameters of non-label party’s submodel
$f_{\mathrm{N}}(\cdot)$, the label party further computes the gradient w.r.t.
the hidden representation $\bm{h}_{\mathrm{N}}$, and sends it back to the non-
label party:

(5)
$\bm{g}\coloneqq\nabla_{{\bm{h}_{\mathrm{N}}}}\mathcal{L}=\frac{\partial\bm{l}}{\partial\bm{h}_{\mathrm{N}}}\frac{\partial\mathcal{L}}{\partial\bm{l}}=\frac{\partial\bm{l}}{\partial\bm{h}_{\mathrm{N}}}(\hat{\bm{y}}-{\bm{y}})\,.$

After receiving the gradient $\bm{g}$, the non-label party computes the
gradients of the remaining submodel parameters using chain rule, and thus
continues the backward pass to update the submodel $f_{\mathrm{N}}(\cdot)$.

In the above training process, the two participants do not share their raw
data (features $\bm{x}_{\mathrm{N}}$, $\bm{x}_{\mathrm{L}}$ and label
$\bm{y}$) to each other. On the contrary, they only exchange intermediate
results $\bm{h}_{\mathrm{N}}$ and $\bm{g}$.

####  2.2.2. Issues of Effectiveness and Privacy

vFL has been successfully applied in healthcare informatics (Vepakomma et al.,
2018b), computational advertising (Li et al., 2022a) and many other domains.
To further improve vFL algorithms, recent studies pay more attention to the
following perspectives: effectiveness and privacy.

Effectiveness: Limited Aligned Samples.

The training procedure of traditional vFL algorithms relies on aligned,
feature-partitioned data. That is, each participant first provides its own
sample ID set, and then a PSI process is adopt to align samples from all
participants, and finally produces the intersection sample set, namely aligned
samples. In the aligned data, for a specific sample, each participant owns a
part of feature of it, and all participants need to collaboratively train the
vFL model.

We can see that the vFL model performance greatly depends on the size of
aligned samples. However the size is usually limited, which restricts the
effectiveness of vFL models. To tackle this, a direction is to make use of
local samples of each participant, namely unaligned samples. In the case of
CVR estimation, the advertising platform usually also collects conversion
events from in-station ads (which are not displayed on the extra publishers).
These local data can be used as auxiliary training samples to enhance the
model trained on aligned samples only. To exploit such unaligned samples that
only have partly features into the vFL training framework, we focus on how to
synthesize the features of other participants.

Privacy: Potential Label Leakage.

vFL is often considered to be privacy-oriented, because during training
process participants only exchange intermediate hidden representations and
gradients rather than raw features and labels. However, recent studies
revealed that it still suffers from potential privacy leakage risks: (1) label
inference attack (Li et al., 2022a; Sun et al., 2022; Fu et al., 2022), which
means that a honest-but-curious non-label party can successfully infer private
labels, and (2) input reconstruction attack (Vepakomma et al., 2018a; Sun et
al., 2021), which means that the label party can reconstruct the raw input
features of the non-label party.

In this work we focus on defending against label inference attacks, aiming to
guarantee that the private label information owned by label party can be
protected. Specifically, although the label party only sends gradients to the
non-label party during training, from Equation 5 we can observe that the
mathematical expression of the gradient $\bm{g}$ w.r.t. hidden representation
contains label information $\bm{y}$, which results in potential label leakage.

During the training procedure of vFL models, privacy-preserving computing
techniques can be applied to protect private information. For instance, the
open-source framework EFLS111https://github.com/alibaba/Elastic-Federated-
Learning-Solution from Alibaba Group provides APIs of cryptography-based
homomorphic encryption (Zhang and Zhu, 2020) and perturbation-based
differential privacy (Abadi et al., 2016). In this work we focus on random
perturbation-based algorithms.

##  3\. Proposed Benchmark

Currently, the lack of a comprehensive benchmark brings challenges to fair
comparison of various algorithms, and also hinders further research for
tacking the effectiveness and privacy issues of vFL. To address this, we
introduce FedAds, the first benchmark for privacy-preserving CVR estimation
with vFL, to facilitate systematical evaluations for vFL algorithms.

###  3.1. Overview

The FedAds benchmark contains:

  * •

A large-scale dataset from Alibaba’s advertising platform, collected from the
log of an ad delivery business relying on vFL-based ranking models. Details in
Section 3.2.

  * •

Systematical evaluations for both effectiveness and privacy aspects of various
vFL algorithms. Details in Section 5.

We release the FedAds benchmark at the page
https://github.com/alibaba/Elastic-Federated-Learning-Solution/tree/FedAds.

###  3.2. Real-World Dataset Construction

We first introduce the background of the collected data, and then give
statistics and features of the dataset.

####  3.2.1. Data Description

The dataset is built upon the click log of our e-commerce ad delivery
business, in which both the online publisher and the advertising platform
belong to Alibaba Group. Although the two parties belong to the same company,
they still cannot share user behavior information to each other. Specifically,
the online publisher is a mobile app that contains ad positions. As shown in
Figure 2, the advertising platform bids for ad impressions in real-time, and
for each request the predicted CVR score $\hat{p}_{\mathrm{CVR}}$ is a key
factor in the bid price. If an ad from the advertising platform wins a bid, it
will be displayed to the user. The user will arrive at another e-commerce
mobile app that manages the ad landing page if he/she clicks on the ad, and
may take further behaviors such as add-to-wishlist and purchase.

The above ad delivery business is a typical application of vFL, where the
online publisher and the advertising platform collaboratively train the CVR
estimation model for ranking candidate ads and improving the delivery
performance. Therefore, we believe that further vFL research can benefit from
our benchmark.

Table 1. Statistics of the dataset. Split | # Samples | # Users | # Ads | CVR
---|---|---|---|---
Training+Test | 11.3 mil. | 4.1 mil. | 1.3 mil. | 0.6%
Training | 10.0 mil. | 3.7 mil. | 1.2 mil. | 0.6%
Test | 1.3 mil. | 0.7 mil. | 0.4 mil. | 0.6%

####  3.2.2. Dataset Construction

We built the dataset based on the above collected data. Specifically, we
collect 1-month consecutive user click events of the delivery business, and
each sample in the dataset is corresponding to a unique click event. We record
context information for each sample, such as the timestamps of request and
click event. Generally, the dataset is composed of features from both parties,
and conversion labels from label party.

Conversion label.

A sample’s label is set to 1 if the user purchases the item described by the
clicked ad, where the attribution window is set to 24 hours. Here we employ
last-touch attribution, which means that if a user clicks on the ad multiple
times and finally purchases the item, we regard that this conversion event is
attributed by the last click event.

Figure 2. Brief illustration of the ranking stage in ad delivery procedure and
user behaviors. The proposed dataset is built upon the click log collected
from this procedure.

Features and processing.

The feature set for each sample consists of two parts: one part from the label
party (i.e., advertising platform) and another one from the non-label party
(i.e., online publisher). Specifically,

  * •

Features from label party: we construct user-side, ad-side and context
features.

    * -

The user-side features contain user profile information (e.g., user ID and
activity level), as well as purchase-related behaviors such as user’s
historial purchased items.

    * -

The ad-side features contain profile information (e.g., ad ID, item ID, item’s
brand ID and price level), as well as ad statistics like historial
click/conversion count level.

    * -

Context feature is the timestamp of conversion event.

  * •

Features from non-label party: similarly, we construct user-side, ad-side and
context features.

    * -

The user-side features are click-related behaviors such as user’s historial
clicked ads.

    * -

The ad-side features are statistical information such as historial impression
count level.

    * -

Context feature is the timestamp of click event.

In summary, there are 16 features owned by label party and 7 features owned by
non-label party. For the considerations of fair comparison and removing
personal identifiable information, in our dataset we release the processed
features rather than original values. Specifically, for discrete features we
map the original values to IDs. For each continuous feature, we perform equi-
frequency discretization to transform the original values to bin IDs.

####  3.2.3. Statistics

Table 1 lists the statistics of our constructed dataset. Totally the dataset
contains 11.3 million samples, and to our knowledge it is the largest public
dataset for evaluating CVR estimation models and vFL algorithms. We split it
to training set and test set based on click timestamp, where the last week’s
samples are selected for test set. Details about how to use the dataset for
evaluating effectiveness and privacy are introduced in Section 5.

We compare our proposed dataset with current commonly-used datasets in vFL
research in Table 2. More importantly, our dataset is constructed from real
world applications, thus we do not need to simulate a vFL experiment setting.

Table 2. Comparison of datasets. Split | # Samples | Type
---|---|---
BHI (Mooney, [n.d.]) | 0.3 mil. | Image
Yahoo Answers (Rakshit, [n.d.]) | 1.5 mil. | Text
Give Me Some Credit (Kaggle, [n.d.]b) | 0.3 mil. | Tabular
Epsilon (PASCAL-Challenge-2008, [n.d.]) | 0.5 mil. | Tabular
Avazu (Kaggle, [n.d.]a) | 4 mil. | Tabular
Criteo (Labs., [n.d.]) | 4.5 mil. | Tabular
FedAds (Ours) | 11.3 mil. | Tabular

Figure 3. (a) Learning a conditional diffusion model for generating federated
embeddings of label party’s unaligned samples. (b) The Diffu-AT framework for
exploiting unaligned samples in vFL training.

##  4\. Methodology

Before the systematical evaluations of existing vFL models, we propose two
approaches for improving effectiveness and privacy.

###  4.1. Exploiting Label Party’s Unaligned Samples

As statement in Section 2.2.2, the limited size of aligned samples
$\mathcal{D}_{\mathrm{aligned}}=\\{(\bm{x}_{\mathrm{N}},\bm{x}_{\mathrm{L}},\bm{y})\\}$
restricts the effectiveness of vFL models. From the perspective of advertising
platform, a nature way to tackle this problem is to incorporate its local and
unaligned samples
$\mathcal{D}_{\mathrm{unaligned}}=\left\\{\left(\bm{x}_{\mathrm{L}}^{\mathrm{u}},\bm{y}^{\mathrm{u}}\right)\right\\}$
into the vFL training procedure.

However, the challenge of exploiting label party’s unaligned samples in vFL is
that they do not have non-label party’s features (i.e.,
$\bm{x}_{\mathrm{N}}^{\mathrm{u}}$ is missing). We propose Diffu-AT, an
enhanced vFL training framework which first generates missing feature with a
diffusion model, and then performs alternatively training to incorporate
unaligned samples into the traditional vFL framework.

####  4.1.1. Generating Federated Embedding
$\tilde{\bm{h}}_{\mathrm{L}}^{\mathrm{u}}$ with Conditional Diffusion Model

As shown in Figure 1, the key of traditional vFL training procedure is that
the non-label party sends the representation $\bm{h}_{\mathrm{N}}$ (named
federated embedding in this work) to enhance the prediction. To effectively
incorporate unaligned samples into vFL, we employ deep generative models to
synthesize federated embeddings $\tilde{\bm{h}}_{\mathrm{N}}^{\mathrm{u}}$ for
those samples.

Problem formulation.

We formulate the synthesis process is to learn a generation model
$\tilde{f}_{\mathrm{N}}(\cdot)$, which can generate a federated embedding
$\tilde{\bm{h}}_{\mathrm{N}}^{\mathrm{u}}$ given label party’s feature
$\bm{x}_{\mathrm{L}}^{\mathrm{u}}$ and label $\bm{y}^{\mathrm{u}}$, namely
$\tilde{\bm{h}}_{\mathrm{N}}^{\mathrm{u}}=\tilde{f}_{\mathrm{N}}\left(\bm{x}_{\mathrm{L}}^{\mathrm{u}},\bm{y}^{\mathrm{u}}\right)$.

Step 1: vFL pretraining.

To this end, we first perform vFL training, that is, pretrain a vFL model
$f(\cdot)$ using aligned samples $\mathcal{D}_{\mathrm{aligned}}$ with the
loss in Eq. 4. Therefore, based on the pretrained vFL model, for each aligned
sample we obtain its federated embedding ${\bm{h}}_{\mathrm{N}}$ and label
party’s representation ${\bm{h}}_{\mathrm{L}}$. Similarly, for each unaligned
sample, we obtain its label party’s representation
${\bm{h}}_{\mathrm{L}}^{\mathrm{u}}$.

Next, we use the data
$\left\\{({\bm{h}}_{\mathrm{N}},{\bm{h}}_{\mathrm{L}},\bm{y})\right\\}$ of
aligned samples to learn the generation model $\tilde{f}_{\mathrm{N}}(\cdot)$,
so as to perform inference on the data
$\left\\{\left(\bm{h}_{\mathrm{L}}^{\mathrm{u}},{\bm{y}}^{\mathrm{u}}\right)\right\\}$
of unaligned samples to generate
$\left\\{\tilde{\bm{h}}_{\mathrm{N}}^{\mathrm{u}}\right\\}$.222Note that
compared to previous problem formulation, we simplify the synthesis process
via replacing the input $\bm{x}_{\mathrm{L}}^{\mathrm{u}}$ with
$\bm{h}_{\mathrm{L}}^{\mathrm{u}}$.

Step 2: Learning a conditional diffusion model as
$\tilde{f}_{\mathrm{N}}(\cdot)$.

Inspired by recent studies on diffusion-based deep generative models (Sohl-
Dickstein et al., 2015; Ho et al., 2020; Li et al., 2022b; Gong et al., 2023),
we propose to synthesize unaligned samples’ federated embeddings via a
conditional diffusion model.

Formally, we regard the concatenation of label party’s representation
$\bm{h}_{\mathrm{L}}$ and label ${\bm{y}}$ as the condition
$\left[{\bm{y}};\bm{h}_{\mathrm{L}}\right]$. We define the forward (noising)
process with $T$ steps as:

(6) $\displaystyle
q\left(\bm{h}_{\mathrm{N},t}\mid\bm{h}_{\mathrm{N},t-1}\right)$
$\displaystyle=\mathcal{N}\left(\bm{h}_{\mathrm{N},t};\sqrt{1-\beta_{t}}\bm{h}_{\mathrm{N},t-1},\beta_{t}\mathbf{I}\right)$
$\displaystyle\bm{z}_{t}$
$\displaystyle\coloneqq\left[{\bm{y}};\bm{h}_{\mathrm{L}};\bm{h}_{\mathrm{N},t}\right]$

where $\bm{h}_{\mathrm{N},0}\coloneqq\bm{h}_{\mathrm{N}}$ and
$\bm{z}_{0}\coloneqq\left[{\bm{y}};\bm{h}_{\mathrm{L}};\bm{h}_{\mathrm{N}}\right]$,
which means that at each step we incrementally add Gaussian noise on the part
of $\bm{h}_{\mathrm{N}}$ only, while keeping the condition part unchanged.
$\beta_{t}$ is a hyperparameter that controls the degree of noise at each
step, and after $T$ steps the $\bm{h}_{\mathrm{N},T}$ is approximately
Gaussian.

Further, the reverse (denoising) process is to reconstruct the original
$\bm{z}_{0}$ given $\bm{z}_{T}$:

(7)
$p_{\Theta}(\bm{z}_{t-1}\mid\bm{z}_{t})=\mathcal{N}\left(\bm{z}_{t-1};\bm{\mu}_{\Theta}(\bm{z}_{t},t),\bm{\Sigma}_{\Theta}(\bm{z}_{t},t)\right)$

where $\bm{\mu}_{\Theta}(\bm{z}_{t},t)$ and
$\bm{\Sigma}_{\Theta}(\bm{z}_{t},t)$ are the predicted mean and standard
deviation of $p_{\Theta}(\bm{z}_{t-1}\mid\bm{z}_{t})$ respectively,
parameterized by learnable $\Theta$. The objective is to maximize the marginal
likelihood $\mathbb{E}_{q(\bm{z}_{0})}\left[\log
p_{\Theta}(\bm{z}_{0})\right]$, and it can be optimized using the variational
lower bound (Sohl-Dickstein et al., 2015). In practice, we follow DDPM (Ho et
al., 2020) to use a simplified form that the training stability has been
empirically proved:

(8)
$\mathcal{L}_{\mathrm{DDPM}}=\sum_{t=1}^{T}\mathbb{E}_{q(\bm{z}_{t}\mid\bm{z}_{0})}\left[\lVert\tilde{\bm{\mu}}_{t}(\bm{z}_{t},\bm{z}_{0})-\bm{\mu}_{\Theta}(\bm{z}_{t},t)\rVert^{2}\right]$

where
$\tilde{\bm{\mu}}_{t}(\bm{z}_{t},\bm{z}_{0})=\frac{\sqrt{\alpha_{t}}(1-\bar{\alpha}_{t-1})}{1-\bar{\alpha}_{t}}\bm{z}_{t}+\frac{\sqrt{\bar{\alpha}_{t-1}}\beta_{t}}{1-\bar{\alpha}_{t}}\bm{z}_{0}$
is the mean of posterior $q(\bm{z}_{t-1}\mid\bm{z}_{t},\bm{z}_{0})$, and here
$\alpha_{t}=1-\beta_{t},\bar{\alpha}_{t}=\prod_{i=1}^{t}\alpha_{i}$.

After learning the diffusion model, we perform conditional generation to
obtain each unaligned sample’s federated embedding
$\tilde{\bm{h}}_{\mathrm{N}}^{\mathrm{u}}$ given the conditional input
$\bm{z}_{T}=\left[{\bm{y}}^{\mathrm{u}};\bm{h}_{\mathrm{L}}^{\mathrm{u}};\bm{h}_{T}\right]$,
where $\bm{h}_{T}\sim\mathcal{N}(\bm{0},\mathbf{I})$ is the initiate state.
The generation repeats $T$ steps of denoising operation using the learned
model $\Theta$ for sampling $\bm{z}_{T-1},\bm{z}_{T-2},\ldots,\bm{z}_{0}$
using $\bm{z}_{t-1}\sim p_{\Theta}(\bm{z}_{t-1}\mid\bm{z}_{t})$. Note that at
each step, we replace the condition part of the generated $\bm{z}_{t}$ to the
original condition
$\left[{\bm{y}}^{\mathrm{u}};\bm{h}_{\mathrm{L}}^{\mathrm{u}}\right]$. Finally
at the last step we obtain
$\tilde{\bm{h}}_{\mathrm{N}}^{\mathrm{u}}\coloneqq\bm{h}_{0}$ as the
synthesized federated embedding, where $\bm{h}_{0}$ is the part in
$\bm{z}_{0}$.

####  4.1.2. Alternative Training Framework

Now we have aligned samples
$\mathcal{D}_{\mathrm{aligned}}=\\{(\bm{x}_{\mathrm{N}},\bm{x}_{\mathrm{L}},\bm{y})\\}$
as well as unaligned samples with synthesized federated embeddings
$\tilde{\mathcal{D}}_{\mathrm{unaligned}}=\left\\{\left(\tilde{\bm{h}}_{\mathrm{N}}^{\mathrm{u}},\bm{x}_{\mathrm{L}}^{\mathrm{u}},\bm{y}^{\mathrm{u}}\right)\right\\}$.

To effectively fuse them for learning an enhanced federated model that
improves effectiveness, we propose to combine the two sample set in an
alternative training fashion. As shown in Figure 3 (b), the label party
augments an auxiliary submodel $\tilde{f}_{\mathrm{L}}(\cdot)$ composed of
$\left(\tilde{f}_{\mathrm{L}}^{(\mathrm{b})},\tilde{f}_{\mathrm{L}}^{(\mathrm{t})},\tilde{f}_{\mathrm{N}}\right)$
to exploit unaligned samples, and recall that $\tilde{f}_{\mathrm{N}}(\cdot)$
is the learned diffusion model.

Our proposed vFL framework named Diffu-AT contains a federated branch
${f}(\cdot)$ and a local branch $\tilde{f}_{\mathrm{L}}(\cdot)$ for learning
from aligned samples $\mathcal{D}_{\mathrm{aligned}}$ and unaligned samples
$\tilde{\mathcal{D}}_{\mathrm{unaligned}}$ respectively in an alternative
training fashion, and their bottom parts ${f}_{\mathrm{L}}^{(\mathrm{b})}$ and
$\tilde{f}_{\mathrm{L}}^{(\mathrm{b})}$ share the parameter set. Specifically,
at each training iteration, we randomly sample a mini-batch from
$\mathcal{D}_{\mathrm{aligned}}$ or
$\tilde{\mathcal{D}}_{\mathrm{unaligned}}$, and then update the parameters of
the corresponding branch. The sampling probability of selecting a mini-batch
from $\mathcal{D}_{\mathrm{aligned}}$ is set to
$p=|{\mathcal{D}}_{\mathrm{aligned}}|/(|{\mathcal{D}}_{\mathrm{aligned}}|+|\tilde{\mathcal{D}}_{\mathrm{unaligned}}|)$.

Put all together, Algorithm 1 shows the training procedure of our Diffu-AT.
Note that for large-scale deep learning based estimation models in online
advertising and recommendation systems, the number of epochs is usually set to
one.

####  4.1.3. Online Inference

During online inference, only the federated branch $f(\cdot)$ is needed for
producing the real-time predictions, and the local branch
$\tilde{f}_{\mathrm{L}}(\cdot)$ is dropped.

We notice that some studies focus on performing online inference based on a
local model owned by label party via distilling knowledge from the federated
model, motivated by reducing response time of receiving federated embedding
from the non-label party (Li et al., 2022c; Li et al., 2022d). However, in
practice we found that the performance of the distilled local model drops
drastically compared to the federated model, which is unacceptable in
Alibaba’s advertising business.

Algorithm 1 Training Procedure of Diffu-AT

0: 1) Aligned samples
$\mathcal{D}_{\mathrm{aligned}}=\\{(\bm{x}_{\mathrm{N}},\bm{x}_{\mathrm{L}},\bm{y})\\}$
2) Unaligned samples
$\mathcal{D}_{\mathrm{unaligned}}=\left\\{\left(\bm{x}_{\mathrm{L}}^{\mathrm{u}},\bm{y}^{\mathrm{u}}\right)\right\\}$

0: The federated model $f(\cdot)$ composed of
$\left({f}_{\mathrm{L}}^{(\mathrm{b})},{f}_{\mathrm{L}}^{(\mathrm{t})},{f}_{\mathrm{N}}\right)$

1: $\triangleright$ Generating Federated Embeddings for Unaligned Samples

2: Learn a conditional diffusion model on
$\left\\{({\bm{h}}_{\mathrm{N}},{\bm{h}}_{\mathrm{L}},\bm{y})\right\\}$

3: Inference on unaligned samples
$\left\\{\left(\bm{h}_{\mathrm{L}}^{\mathrm{u}},{\bm{y}}^{\mathrm{u}}\right)\right\\}$,
generating $\left\\{\tilde{\bm{h}}_{\mathrm{N}}^{\mathrm{u}}\right\\}$

4:
$\tilde{\mathcal{D}}_{\mathrm{unaligned}}=\left\\{\left(\tilde{\bm{h}}_{\mathrm{N}}^{\mathrm{u}},\bm{x}_{\mathrm{L}}^{\mathrm{u}},\bm{y}^{\mathrm{u}}\right)\right\\}$

5: $\triangleright$ Alternative Training

6:
$p=|{\mathcal{D}}_{\mathrm{aligned}}|/(|{\mathcal{D}}_{\mathrm{aligned}}|+|\tilde{\mathcal{D}}_{\mathrm{unaligned}}|)$

7: for all epoch $e\leftarrow 1$ to $E$ do

8: for all iteration $i\leftarrow 1$ to $I$ do

9: Randomly sampling a value $p_{i}$ from $\mathcal{U}(0,1)$

10: if $p_{i}\leq p$ then

11: Yield a mini-batch from ${\mathcal{D}}_{\mathrm{aligned}}$

12: Update the parameters of the federated branch $f(\cdot)$

13: else if $p_{i}>p$ then

14: Yield a mini-batch from $\tilde{\mathcal{D}}_{\mathrm{unaligned}}$

15: Update the parameters of the local branch $\tilde{f}_{\mathrm{L}}(\cdot)$

16: end if

17: end for

18: end for

###  4.2. Defending Label Inference Attack

As statement in Section 2.2.2, due to the mathematical expression of the
gradient $\bm{g}$ w.r.t. federated embedding contains label information
$\bm{y}$, vFL models may suffer from potential label leakage, which means that
a honest-but-curious non-label party can infer private labels. We focus on
employ random perturbation methods to protect label information during vFL
training.

Given the fact that the label leakage mainly comes from the difference between
the magnitudes and directions of samples, an intuitive way to address such
problem is to applying random convex combination of gradients while
transmitting in the cut layer, also known as mixup (Zhang et al., 2018). To
alleviate label leakage, we propose a simple-yet-effective gradient mixup and
projection approach named MixPro, which performs convex combinations and
projections on in-batch sample gradients for protecting private label
information. It employs mixup operation as initial perturbation to original
gradients, and then performs projection to further remove useless information
contained in original gradients.

MixPro does not make any assumption about gradient distribution and can be
seamlessly integrated into neural network based vFL framework. Next we first
introduce the gradient mixup strategy, then adopt gradient projection to
further modify the gradient directions to achieve better privacy-preserving
performance during vFL model training.

####  4.2.1. Gradient Mixup

For a batch of samples during training, we denote
$\left\\{\bm{g}_{i}\right\\}_{i=1}^{B}$ to be the collection of gradients
w.r.t. federated embeddings at the cut layer, where $B$ is the batch size. We
formulate the mixup-based perturbed gradient of the $i$-th sample to be the
convex combination of two sample gradients in the form below:

(9) ${\bm{g}}_{mixed,i}=\lambda\cdot{\bm{g}}_{i}+(1-\lambda)\cdot{\bm{g}}_{r}$

where ${\bm{g}}_{r}$ is a random sample’s gradient from the given batch. As
stated in the original mixup method (Zhang et al., 2018), we choose
$\lambda\sim\mathrm{Beta}(\alpha,\alpha)$ and set $\lambda>0.5$, where
$\alpha$ is a hyperparameter.

Note that we set $\lambda>0.5$ since the perturbed gradients should preserve
more information from the original gradient including magnitude and direction
to maintain the prediction performance of the vFL model. We only use the
convex combination between two sample gradients in order to simplify the
calculation, and we empirically found that this strategy also achieves similar
performance with more gradients to be mixed.

Figure 4. Illustration of gradient mixup and projection operations in MixPro
for a training sample’s gradient w.r.t. the federated embedding.

####  4.2.2. Gradient Projection

The above gradient mixup strategy may still suffer from the label leakage
problem induced by the directions of gradients, since the direction
information of the original gradient remains in the perturbed gradient to some
extent.

In order to keep the directions of gradients confined to a smaller region, we
propose to further perform gradient projection on the mixed gradients. This is
inspired by the studies in multi-task learning models (Wang et al., 2021; Yu
et al., 2020), in which the gradients for different tasks are projected along
the direction of main one to avoid the conflict in gradient directions and
thus achieve better performance. Given the intuition that higher similarity of
gradients in orientation will effectively alleviate the label leakage from
sample gradients, such projection technique in multi-task optimization can be
further adopted in defending label leakage in vFL.

Specifically, we denote $\bar{\bm{g}}=\frac{1}{B}\sum_{i=1}^{B}\bm{g}_{i}$ as
the average gradient for a batch of samples. We set the goal of the cosine
similarity between the an in-batch gradient $\bm{g}_{mixed,i}$ and the average
gradient $\bar{\bm{g}}$ is that it should be larger than a pre-defined
threshold $\phi_{goal}\in[-1,1]$:

(10)
$\phi_{i}=\cos\left(\bm{g}_{mixed,i},\bar{\bm{g}}\right)\geq\phi_{goal}\,.$

For a sample where the similarity goal is not achieved (that is,
$\phi_{goal}>\phi_{i}$), the following projection operation will be applied
and we obtain the projected gradient:

$\displaystyle\bm{g}_{projected,i}=\bm{g}_{mixed,i}\ +$ (11)
$\displaystyle\frac{\lVert\bm{g}_{mixed,i}\rVert\left(\phi_{goal}\sqrt{1-(\phi_{i})^{2}}-\phi_{i}\sqrt{1-(\phi_{goal})^{2}}\right)}{\lVert\bar{\bm{g}}\rVert\sqrt{1-(\phi_{goal})^{2}}}\cdot\bar{\bm{g}}.$

And finally, the operation of our MixPro is:

(12) $\displaystyle\bm{g}_{perturbed,i}=\begin{cases}\bm{g}_{mixed,i}\ ,\
&\textrm{if}\ \phi_{i}\geq\phi_{goal}\ ,\\\ \bm{g}_{projected,i}\ ,\
&\textrm{if}\ \phi_{i}<\phi_{goal}\ .\end{cases}$

During training process, based on our defense approach MixPro, for a batch of
aligned training samples, the label party sends the perturbed gradients
$\left\\{\bm{g}_{perturbed,i}\right\\}_{i=1}^{B}$ rather than the original
ones $\left\\{\bm{g}_{i}\right\\}_{i=1}^{B}$ to the non-label party for
updating the submodel $f_{\mathrm{N}}(\cdot)$, aiming to improve the privacy
of the vFL model training.

MixPro does not make any assumption about gradient distribution and can be
seamlessly integrated into arbitrary neural network based vFL training
framework.

##  5\. Systematical Evaluations

As another contribution of our FedAds benchmark, we conduct systematical
evaluations of various vFL models for both effectiveness and privacy aspects.
For existing vFL algorithms that focus on improving effectiveness, we evaluate
their performance in Section 5.1. We then compare representative approaches
for defending label inference attack, and results are shown in Section 5.2.

###  5.1. Experiments on Effectiveness

We first introduce experimental setup, evaluation metrics and comparative
approaches in experiments, and then list implementation details and show
experimental results.

####  5.1.1. Experimental Setup.

We use our proposed dataset for evaluation. Specifically, we use 20% of the
training set as aligned samples $\mathcal{D}_{\mathrm{aligned}}$, and the rest
80% of the training set is used as unaligned samples
$\mathcal{D}_{\mathrm{unaligned}}$ by removing their features from non-label
party. The performance is evaluated using the test set.

This setup allows us to know the upper bound of exploiting unaligned samples
in vFL on our dataset: if we use the full training data as aligned samples and
train a vFL model, the model’s performance is the upper bound because we
“leak” the non-label party’s features of $\mathcal{D}_{\mathrm{unaligned}}$.

####  5.1.2. Evaluation Metrics.

We use AUC and negative log likelihood (NLL for short) as the evaluation
metrics for effectiveness. The former measures ranking performance on
candidates, and the latter reflects calibration performance of predicted
scores.

####  5.1.3. Comparative Approaches.

We compare the following approaches to evaluate effectiveness:

  * •

Local is a model trained on label party’s features only, without using any
features from non-label party.

  * •

VanillaVFL is a vFL model trained on aligned samples using the loss in
Equation 4.

  * •

HeuristicVFL further exploits label party’s unaligned samples, in which the
missing non-label party’s features are synthesized with a heuristic way: for
each unaligned sample, we retrieve the user ID from aligned samples, and
compute the averaged federated embedding of this user in VanillaVFL. Then we
perform alternative training (see Section 4.1.2).

  * •

SS-VFL (Castiglia et al., [n.d.]) exploits unaligned samples with self-
supervised learning. Each party first employs its local samples to perform
unsupervised pretraining, and then a VanillaVFL is trained as the final model.

  * •

FedCVT (Kang et al., 2022) exploits unaligned samples with both self-
supervised learning and semi-supervised learning. It first performs
unsupervised pretraining to learn a two-stream network. Then a similarity
function is used to generate unaligned samples’ features. Finally it combines
unlabeled unaligned samples and labeled aligned samples with semi-supervised
learning in a co-training fashion.333Note that the time complexity of
similarity computing process in FedCVT is too high, and thus we re-implement
it with an approximate way through randomly dropping some candidates before
similarity computing.

  * •

VFL-MPD (Li et al., 2022d) exploits unaligned samples with a specific self-
supervised learning task, where a matched pair detection task is proposed to
learn powerful representations using large size of unaligned samples.

  * •

FedHSSL (He et al., 2022) exploits unaligned samples with two-stage
pretraining. The first stage is cross-party pretraining that fits learned
representations each other. The second stage is local pretraining where the
learning objective is based on data augmentation.

  * •

JPL (Li et al., 2022c) exploits unaligned samples via synthesizing non-label
party’s features based on learning a mapping function between label party’s
features and non-label party’s features, with several constraints like
representation equivalence and label discrimination.444Note that the original
JPL approach distills the knowledge of federated model to a local model, which
results in performance drop. For fair comparison, we replace the distillation
with the alternative training as we stated in Section 4.1.2.

  * •

Diffu-AT is our proposed approach.

  * •

ORALE is a model trained on the full training set that the non-label party’s
features of unaligned samples are known in advance. Therefore its performance
is the upper bound of exploiting unaligned samples in our dataset.

Table 3. Effectiveness evaluation of comparative vFL algorithms for CVR estimation. Method |  Ranking: AUC |  Calibration: NLL
---|---|---
Label party’s features $\left\\{(\bm{x}_{\mathrm{L}},\bm{y})\right\\}$
Local | 0.609 | 0.0391
\+ vFL training
$\left\\{(\bm{x}_{\mathrm{N}},\bm{x}_{\mathrm{L}},\bm{y})\right\\}$
VanillaVFL | 0.620 | 0.0389
\+ Unaligned samples
$\left\\{\left(\bm{x}_{\mathrm{L}}^{\mathrm{u}},\bm{y}^{\mathrm{u}}\right)\right\\}$
HeuristicVFL | 0.630 | 0.0387
SS-VFL | 0.636 | 0.0381
FedCVT | 0.639 | 0.0379
VFL-MPD | 0.641 | 0.0373
FedHSSL | 0.642 | 0.0375
JPL | 0.644 | 0.0374
Diffu-AT (Ours) | 0.645 | 0.0375
Upper bound
$\left\\{\left(\bm{x}_{\mathrm{N}}^{\mathrm{u}},\bm{x}_{\mathrm{L}}^{\mathrm{u}},\bm{y}^{\mathrm{u}}\right)\right\\}$
ORALE | 0.658 | 0.0367

####  5.1.4. Implementation Details.

We choose YouTube-DNN (Covington et al., 2016) as the backbone. Each feature
is represented as an 8-dim embedding. The non-label party submodel
$f_{\mathrm{N}}$ contains an embedding table and a two-layered DNN with output
sizes of [128, 32]. The label party’s submodel contains two branches
$f_{\mathrm{L}}$ and $\tilde{f}_{\mathrm{L}}$, where they share the embedding
table and bottom part (a two-layered DNN with output sizes of [256, 128]), and
each top part is a single layer with logistic function to produce predicted
score. The two branches do not share the batch normalization operations. The
batch size is set to 256. In the conditional diffusion model, the total step
$T$ is set to 1000 as in previous work (Ho et al., 2020). For timestep $t$ we
use sine and cosine functions to encode, and the schedule for $\beta$ is
linear schedule. We implement the models using
XDL555https://github.com/alibaba/x-deeplearning and EFLS.

####  5.1.5. Experimental Results.

Table 3 shows the evaluation results for effectiveness of all the comparative
approaches. Note that 0.3% improvement on AUC can be regarded as a large
improvement in large-scale industrial datasets (Cheng et al., 2016). We
observe that VanillaVFL outperforms Local by around 1.1% AUC, which
demonstrates that incorporating non-label party’s features to perform
federated training is effective to improve the estimation performance.
Furthermore, exploiting unaligned samples usually boost the AUC and NLL
compared to VanillaVFL. For instance, the simple approach HeuristicVFL
outperforms VanillaVFL by 1.0% AUC which shows the potential of this
direction. By comparing the approaches that exploits unaligned samples, we can
see that the models considering label information of unaligned data (such as
Diffu-AT and JPL) generally perform better than the self-supervised learning
models (such as FedHSSL, VFL-MPD and SS-VFL), and thus the labeled samples
from label party are the key for improving traditional vFL.

Our proposed Diffu-AT shows the best performance on the ranking ability among
all comparative approaches, verifying that the synthesized federated
embeddings with diffusion model can enhance the representation of unaligned
samples. We also see that VFL-MPD performs the best on the calibration
performance, and we suggest that its pretraining objective obtains better
model initialization. We leave the improvement of Diffu-AT’s calibration
performance in future work.

###  5.2. Experiments on Privacy

####  5.2.1. Evaluation Metrics for Privacy.

We first introduce how a non-label party can perform label inference attack to
steal private label information, and then gives the evaluation metrics for
privacy.

Label inference attack.

Specifically, the non-label party is the attacker that performs label
inference attack. The objective of an attacker is to infer the private labels
${\bm{y}}$ owned by the label party based on exchanged federated embeddings
$\bm{h}_{\mathrm{N}}$ and/or gradients $\bm{g}$. Specifically, the attack can
be performed between two iterations during training phrase, or after the
training is finished. We assume that the non-label party is honest-but-
curious, which means that it cannot interfere with the training process (such
as sending wrong hidden representations to the label party). Under this
situation, the non-label party can employ arbitrary classifiers to infer
labels.

From Equation 5 we observe that the form of gradient $\bm{g}$ actually
contains label information ${\bm{y}}$. Besides, Sun et al. (2022) also found
that the federated embeddings gradually have correlations with labels during
vFL training. We introduce two attack strategies based on gradient and
federated embedding respectively.

  1. (1)

Gradient-based attack (Li et al., 2022a). Given the observation that a model
tends to be less confident about “a positive sample being positive” than “a
negative sample being negative”, if a sample’s gradient norm
$\lVert\bm{g}\rVert_{2}$ is larger than a threshold, the attacker infers that
it belongs to positive class.

  2. (2)

Federated embedding-based attack (Sun et al., 2022). After vFL model training,
we perform clustering on the federated embeddings to place all training
samples into two clusters. Given the prior knowledge that positive samples’
size is smaller than that of negative samples, the attacker infers that the
samples in the smaller cluster belong to the positive class.

Evaluation metrics.

For a vFL model equipping defense approach, we evaluate two aspects including
utility and privacy. The utility aspect is about the prediction performance of
the model, and we choose AUC used in previous experiments on effectiveness.

The privacy aspect evaluates the defense ability to label inference attack.
Because the evaluation relies on the attack strategy that may be stochastic to
some extent, we design a metric named $\Delta\mathrm{LeakAUC}$ to compute
relative improvement to a base model: 1) we first use the model without any
defense approach as the base, and compute the AUC value given the labels
inferred by attacker and the true labels, namely
$\mathrm{LeakAUC}_{\mathrm{base}}$. 2) Consider a model with a specific
defense approach, similarly we compute the $\mathrm{LeakAUC}_{\mathrm{exp}}$.
3) Finally we compute the relative improvement
$\Delta\mathrm{LeakAUC}=(\mathrm{LeakAUC}_{\mathrm{exp}}-\mathrm{LeakAUC}_{\mathrm{base}})/\mathrm{LeakAUC}_{\mathrm{base}}$
as the evaluation metric. For the LeakAUC metric, lower is better, because it
is expected that the attacker cannot recover true labels.

####  5.2.2. Comparative Approaches.

We compare the following random perturbation based defense approaches.

  * •

No Defense is the vFL model that does not equip any defense approach during
training.

  * •

DP (Abadi et al., 2016) employs differential privacy to obtain a generic
gradient perturbation framework, in which DP is enforced on the transmitted
information.

  * •

Marvell (Li et al., 2022a) adds Gaussian noise to perturb original gradients,
where the covariance matrices of noise distribution are computed based on
minimizing leakage level under a noise power constraint. The noise form
assumes that the original gradient also follows Gaussian distribution.

  * •

MixPro is our proposed defense approach.

Table 4. Privacy evaluation of comparative defense approaches to label inference attack. Method |  Privacy: $\Delta\mathrm{LeakAUC}$ |  Utility: AUC
---|---|---
No Defense | - | 0.620
DP | -5.7% | 0.602
Marvell | -24.7% | 0.601
MixPro (Ours) | -11.3% | 0.602

####  5.2.3. Implementation Details.

We choose the VanillaVFL model used in previous experiments on effectiveness
as the base model to compute $\mathrm{LeakAUC}_{\mathrm{base}}$, and the label
inference attack strategy is the federated embedding-based attacker. There are
two key hyperparameters in our proposed MixPro: $\alpha$ to control the
gradient mixup strategy and $\phi_{goal}$ to determine how the mixed gradient
should be projected. Generally, a larger $\alpha$ will force the mixup
strategy to show less uncertainty and the mixup weight $\lambda$ will thus
become closer to 0.5, leading to better privacy preservation but greater
compromise on AUC. A larger $\phi_{goal}$ will also narrow down the region of
gradients’ directions and provide better privacy performance with more trade-
off on AUC. In experiments we set $\alpha=0.6$ and $\phi_{goal}=\sqrt{3}/2$ by
default.

####  5.2.4. Experimental Results.

Table 4 shows the experimental results on our proposed datasets of our
proposed MixPro and other compared approaches against the label inference
attack. The straightforward DP only achieves very limited privacy performance
compared to No Defense, which means that more sophisticated approaches are
needed for protecting label information.

Marvell reduces the LeakAUC by a large margin and achieves an acceptable
level, demonstrating that the a well-designed random perturbation strategy is
very effective for defending label inference attack. Our proposed MixPro
performs much better than DP, verifying its privacy performance in vFL model
training, and Marvell is still the state-of-the-art defense approach. We
suggest that in MixPro the random sampling operation for mixup and projection
may restrict its defense ability, because the size of positive samples is very
small and thus two combined gradients usually come from the same class. We
also notice that both MixPro and Marvell result in a drop of around 2.0% AUC
compared to No Defense. Therefore, better utility-privacy trade-off is a key
direction in further defense approaches during vFL model training.

##  6\. Related Work

There are many great benchmarks for FL, such as LEAF (Caldas et al., 2018),
FedML (He et al., 2020), Flower (Beutel et al., 2020), FedScale (Lai et al.,
2022), FLamby (Terrail et al., 2022) and pFL-bench (Chen et al., 2022). They
mainly focus on horizontal FL and personalized FL, and to our knowledge no vFL
benchmarks have been proposed for fair comparing existing approaches,
especially for neural network based vFL approaches. In this work, we propose
the vFL benchmark named FedAds, which provides a large-scale dataset collected
from Alibaba’s advertising system, as well as systematical evaluations for
existing approaches. Therefore we believe that the FedAds makes good
contributions to facilitate vFL research.

##  7\. Conclusion and Future Work

We introduce FedAds, the first benchmark for privacy-preserving CVR
estimation, to facilitate systematical evaluations for vFL algorithms. It
contains 1) a large-scale real-world dataset from our online advertising
platform, collected from an ad delivery business relying on vFL-based ranking
models, as well as 2) systematical evaluations for both effectiveness and
privacy aspects of various neural network based vFL algorithms through
extensive experiments. Besides, to improve vFL effectiveness, we explore to
incorporate unaligned data via generating unaligned samples’ feature
representations using generative models. To protect privacy well, we also
develop perturbation based on mixup and projection operations. Experiments
show that they achieve reasonable performances.

In future work, we shall explore the following directions: 1) Improving the
calibration performance of vFL models (Pan et al., 2020; Wei et al., 2022). 2)
Alleviating the sample selection bias issue in CVR estimation models through
debiasing approaches (Guo et al., 2021; Xu et al., 2022) for vFL models. 3)
Improving vFL training efficiency. 4) Extending the usage of vFL fashion from
ranking stage to retrieval stage in online adverting systems.

## Acknowledgement

This work was supported by Alibaba Innovative Research project. We thank all
the anonymous reviewers for their valuable comments. We also thank Jinquan Liu
and Prof. Baoyuan Wu for assistance.

## References

  * (1)
  * Abadi et al. (2016) Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. 2016.  Deep learning with differential privacy. In _Proceedings of the 2016 ACM SIGSAC conference on computer and communications security_. 308–318.
  * Beutel et al. (2020) Daniel J Beutel, Taner Topal, Akhil Mathur, Xinchi Qiu, Titouan Parcollet, Pedro PB de Gusmão, and Nicholas D Lane. 2020\.  Flower: A friendly federated learning research framework.  _arXiv preprint arXiv:2007.14390_ (2020).
  * Caldas et al. (2018) Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Konečnỳ, H Brendan McMahan, Virginia Smith, and Ameet Talwalkar. 2018.  Leaf: A benchmark for federated settings.  _arXiv preprint arXiv:1812.01097_ (2018).
  * Castiglia et al. ([n.d.]) Timothy Castiglia, Shiqiang Wang, and Stacy Patterson. [n.d.].  Self-Supervised Vertical Federated Learning. In _Workshop on Federated Learning: Recent Advances and New Challenges (in Conjunction with NeurIPS 2022)_.
  * Chen et al. (2022) Daoyuan Chen, Dawei Gao, Weirui Kuang, Yaliang Li, and Bolin Ding. 2022.  pFL-Bench: A Comprehensive Benchmark for Personalized Federated Learning.  _arXiv preprint arXiv:2206.03655_ (2022).
  * Cheng et al. (2016) Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. 2016\.  Wide & deep learning for recommender systems. In _Proceedings of the 1st workshop on deep learning for recommender systems_. 7–10.
  * Covington et al. (2016) Paul Covington, Jay Adams, and Emre Sargin. 2016\.  Deep neural networks for YouTube recommendations. In _Proceedings of the 10th ACM conference on recommender systems_. 191–198.
  * Fu et al. (2022) Chong Fu, Xuhong Zhang, Shouling Ji, Jinyin Chen, Jingzheng Wu, Shanqing Guo, Jun Zhou, Alex X Liu, and Ting Wang. 2022.  Label inference attacks against vertical federated learning. In _31st USENIX Security Symposium (USENIX Security 22), Boston, MA_.
  * Gong et al. (2023) Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and Lingpeng Kong. 2023.  Sequence to sequence text generation with diffusion models. In _International Conference on Learning Representations_.
  * Guo et al. (2021) Siyuan Guo, Lixin Zou, Yiding Liu, Wenwen Ye, Suqi Cheng, Shuaiqiang Wang, Hechang Chen, Dawei Yin, and Yi Chang. 2021.  Enhanced Doubly Robust Learning for Debiasing Post-click Conversion Rate Estimation. In _The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_.
  * He et al. (2020) Chaoyang He, Songze Li, Jinhyun So, Xiao Zeng, Mi Zhang, Hongyi Wang, Xiaoyang Wang, Praneeth Vepakomma, Abhishek Singh, Hang Qiu, et al. 2020\.  Fedml: A research library and benchmark for federated machine learning.  _arXiv preprint arXiv:2007.13518_ (2020).
  * He et al. (2022) Yuanqin He, Yan Kang, Jiahuan Luo, Lixin Fan, and Qiang Yang. 2022.  A hybrid self-supervised learning framework for vertical federated learning.  _arXiv preprint arXiv:2208.08934_ (2022).
  * Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020.  Denoising diffusion probabilistic models.  _Advances in Neural Information Processing Systems_ 33 (2020), 6840–6851.
  * Kaggle ([n.d.]a) Kaggle. [n.d.]a.  Avazu dataset.  https://www.kaggle.com/c/avazu-ctr-prediction.
  * Kaggle ([n.d.]b) Kaggle. [n.d.]b.  Give me some credit dataset.  https://www.kaggle.com/c/GiveMeSomeCredit.
  * Kang et al. (2022) Yan Kang, Yang Liu, and Xinle Liang. 2022.  FedCVT: Semi-supervised vertical federated learning with cross-view training.  _ACM Transactions on Intelligent Systems and Technology (TIST)_ 13, 4 (2022), 1–16.
  * Kolesnikov et al. (2016) Vladimir Kolesnikov, Ranjit Kumaresan, Mike Rosulek, and Ni Trieu. 2016.  Efficient batched oblivious PRF with applications to private set intersection. In _Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security_. 818–829.
  * Labs. ([n.d.]) Criteo Labs. [n.d.].  Criteo dataset.  https://labs.criteo.com/2014/02/download-kaggle-display-advert ising-challenge-dataset/.
  * Lai et al. (2022) Fan Lai, Yinwei Dai, Sanjay Singapuram, Jiachen Liu, Xiangfeng Zhu, Harsha Madhyastha, and Mosharaf Chowdhury. 2022.  Fedscale: Benchmarking model and system performance of federated learning at scale. In _International Conference on Machine Learning_. PMLR, 11814–11827.
  * Li et al. (2022a) Oscar Li, Jiankai Sun, Xin Yang, Weihao Gao, Hongyi Zhang, Junyuan Xie, Virginia Smith, and Chong Wang. 2022a.  Label Leakage and Protection in Two-party Split Learning. In _International Conference on Learning Representations_.
  * Li et al. (2022c) Wenjie Li, Qiaolin Xia, Hao Cheng, Kouyin Xue, and Shu-Tao Xia. 2022c.  Vertical semi-federated learning for efficient online advertising.  _arXiv preprint arXiv:2209.15635_ (2022).
  * Li et al. (2022d) Wenjie Li, Qiaolin Xia, Junfeng Deng, Hao Cheng, Jiangming Liu, Kouying Xue, Yong Cheng, and Shu-Tao Xia. 2022d.  Semi-supervised cross-silo advertising with partial knowledge transfer.  _International Workshop on Trustworthy Federated Learning in Conjunction with IJCAI 2022 (FL-IJCAI’22)_ (2022).
  * Li et al. (2022b) Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori Hashimoto. 2022b.  Diffusion-LM Improves Controllable Text Generation. In _Advances in Neural Information Processing Systems_.
  * Ma et al. (2018) Xiao Ma, Liqin Zhao, Guan Huang, Zhi Wang, Zelin Hu, Xiaoqiang Zhu, and Kun Gai. 2018.  Entire Space Multi-Task Model: An Effective Approach for Estimating Post-Click Conversion Rate. In _The 41st International ACM SIGIR Conference on Research and Development in Information Retrieval_.
  * McMahan et al. (2017) Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. 2017\.  Communication-efficient learning of deep networks from decentralized data. In _Artificial intelligence and statistics_. PMLR, 1273–1282.
  * Mooney ([n.d.]) P. Mooney. [n.d.].  Breast histopathology images.  https://www.kaggle.com/datasets/paultimothymooney/breast-histopathology-images.
  * Pan et al. (2020) Feiyang Pan, Xiang Ao, Pingzhong Tang, Min Lu, Dapeng Liu, Lei Xiao, and Qing He. 2020.  Field-aware calibration: A simple and empirically strong method for reliable probabilistic predictions. In _Proceedings of The Web Conference 2020_. 729–739.
  * PASCAL-Challenge-2008 ([n.d.]) PASCAL-Challenge-2008. [n.d.].  epsilon dataset.  https://www.csie.ntu.edu.tw/ cjlin/libsvmtools/datasets/.
  * Rakshit ([n.d.]) Soumik Rakshit. [n.d.].  Yahoo answers dataset.  https://www.kaggle.com/soumikrakshit/yahoo-answers-dataset.
  * Sohl-Dickstein et al. (2015) Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. 2015.  Deep unsupervised learning using nonequilibrium thermodynamics. In _International Conference on Machine Learning_. PMLR, 2256–2265.
  * Sun et al. (2022) Jiankai Sun, Xin Yang, Yuanshun Yao, and Chong Wang. 2022\.  Label Leakage and Protection from Forward Embedding in Vertical Federated Learning.  _arXiv preprint arXiv:2203.01451_ (2022).
  * Sun et al. (2021) Jiankai Sun, Yuanshun Yao, Weihao Gao, Junyuan Xie, and Chong Wang. 2021.  Defending against reconstruction attack in vertical federated learning.  _arXiv preprint arXiv:2107.09898_ (2021).
  * Terrail et al. (2022) Jean Ogier du Terrail, Samy-Safwan Ayed, Edwige Cyffers, Felix Grimberg, Chaoyang He, Regis Loeb, Paul Mangold, Tanguy Marchand, Othmane Marfoq, Erum Mushtaq, et al. 2022\.  FLamby: Datasets and Benchmarks for Cross-Silo Federated Learning in Realistic Healthcare Settings.  _arXiv preprint arXiv:2210.04620_ (2022).
  * Vepakomma et al. (2018a) Praneeth Vepakomma, Otkrist Gupta, Abhimanyu Dubey, and Ramesh Raskar. 2018a.  Reducing leakage in distributed deep learning for sensitive health data.  _arXiv preprint arXiv:1812.00564_ 2 (2018).
  * Vepakomma et al. (2018b) Praneeth Vepakomma, Otkrist Gupta, Tristan Swedish, and Ramesh Raskar. 2018b.  Split learning for health: Distributed deep learning without sharing raw patient data.  _ICLR 2019 Workshop on AI for social good_ (2018).
  * Wang et al. (2021) Zirui Wang, Yulia Tsvetkov, Orhan Firat, and Yuan Cao. 2021\.  Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models. In _International Conference on Learning Representations_.
  * Wei et al. (2022) Penghui Wei, Weimin Zhang, Ruijie Hou, Jinquan Liu, Shaoguo Liu, Liang Wang, and Bo Zheng. 2022.  Posterior Probability Matters: Doubly-Adaptive Calibration for Neural Predictions in Online Advertising. In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 2645–2649.
  * Wei et al. (2021) Penghui Wei, Weimin Zhang, Zixuan Xu, Shaoguo Liu, Kuang-chih Lee, and Bo Zheng. 2021\.  AutoHERI: Automated Hierarchical Representation Integration for Post-Click Conversion Rate Estimation. In _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_. 3528–3532.
  * Xu et al. (2022) Zixuan Xu, Penghui Wei, Weimin Zhang, Shaoguo Liu, Liang Wang, and Bo Zheng. 2022\.  UKD: Debiasing Conversion Rate Estimation via Uncertainty-regularized Knowledge Distillation. In _Proceedings of The Web Conference 2022_.
  * Yang et al. (2019) Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. 2019\.  Federated machine learning: Concept and applications.  _ACM Transactions on Intelligent Systems and Technology (TIST)_ 10, 2 (2019), 1–19.
  * Yu et al. (2020) Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. 2020\.  Gradient surgery for multi-task learning.  _Advances in Neural Information Processing Systems_ 33 (2020), 5824–5836.
  * Zhang et al. (2018) Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. 2018.  mixup: Beyond Empirical Risk Minimization. In _International Conference on Learning Representations_.
  * Zhang and Zhu (2020) Yifei Zhang and Hao Zhu. 2020.  Additively homomorphical encryption based deep neural network for asymmetrically collaborative machine learning.  _arXiv preprint arXiv:2007.06849_ (2020).