# Deep Ensemble Shape Calibration: Multi-Field Post-hoc Calibration in Online
Advertising

Shuai Yang  Shopee Discovery AdsBeijingChina ,  Hao Yang  Shopee Discovery
AdsBeijingChina ,  Zhuang Zou  Shopee Discovery AdsBeijingChina ,  Linhe Xu
Shopee Discovery AdsBeijingChina ,  Shuo Yuan  Shopee Discovery
AdsBeijingChina and  Yifan Zeng  Shopee Discovery AdsBeijingChina

(2018)

###### Abstract.

In the e-commerce advertising scenario, estimating the true probabilities
(known as a calibrated estimate) on Click-Through Rate (CTR) and Conversion
Rate (CVR) is critical and can directly affect the benefits of the buyer,
seller and platform.

Previous research has introduced numerous solutions for addressing the
calibration problem. These methods typically involve the training of
calibrators using a validation set and subsequently applying these calibrators
to correct the original estimated values during online inference. However,
what sets e-commerce advertising scenarios is the challenge of multi-field
calibration. Multi-field calibration can be subdivided into two distinct sub-
problems: value calibration and shape calibration. Value calibration is
defined as no over- or under-estimation for each value under concerned fields
(such as the average of pCTR should equal to the CTR for value ”women’s shoes”
in the field ”category”, in the advertising context, the number of fields can
range from dozens to even hundreds). Shape calibration is defined as no over-
or under-estimation for each subset of the pCTR within the specified range
under condition of concerned fields. In order to achieve shape calibration and
value calibration, it is necessary to have a strong data utilization ability.
Because the quantity of pCTR specified range for single field-value (such as
user ID and item ID) sample is relative small, which makes the calibrator more
difficult to train. However the existing methods cannot simultaneously fulfill
both value calibration and shape calibration.

To solve these problems, we propose a new method named Deep Ensemble Shape
Calibration (DESC) 111The code is at https://github.com/HaoYang0123/DESC. We
introduce innovative basis calibration functions, which enhance both function
expression capabilities and data utilization by combining these basis
calibration functions. A significant advancement lies in the development of an
allocator capable of allocating the most suitable shape calibrators to
different estimation error distributions within diverse fields and values.
Finally, we achieve significant improvements in both public data and
industrial data, and on online experiments we get +2.5% of CVR and +4.0% of
GMV (Gross Merchandise Volume).

Multi-Field Calibration, Basis Calibration Function, Field-aware Attention

††copyright: acmcopyright††journalyear: 2018††doi:
XXXXXXX.XXXXXXX††conference: Make sure to enter the correct conference title
from your rights confirmation emai; June 03–05, 2018; Woodstock, NY††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06

##  1\. INTRODUCTION

Figure 1. Examples to show significant variations in value miscalibration
among different fields and values. Figure 2. Examples to show significant
variations in shape miscalibration among different values within the same
field.

Estimating CTR and CVR is a crucial technology in e-commerce advertising
domains (Cheng et al., 2016; Shan et al., 2016; Zhai et al., 2016; Zhu et al.,
2017; Huang et al., 2019; Wang et al., 2021; Du et al., 2021; Davidson et al.,
2010; Guo et al., 2017b; Wang et al., 2018; Bian et al., 2022; Nguyen et al.,
2023; Yang et al., 2023; Deng et al., 2021). Accurate prediction of CTR (pCTR)
and CVR (pCVR) is essential, as it necessitates precision not only in ranking
but also in absolute values.

However, recent studies (Bella et al., 2010; Guo et al., 2017a; Korb, 1999;
Ovadia et al., 2019) have revealed that numerous established machine learning
techniques, particularly deep learning methods extensively applied in the
fields of e-commerce advertising, often yield inadequately calibrated
probability predictions.

In previous research, there are many solutions to solve the calibration
problem. These methods learn calibrator through the validation set, and then
use the calibrator to correct the original estimated values in the online
inference stage. These methods can be categorized into parameter methods
(including Platt Scaling (Platt et al., 1999), Temperature scaling (Guo et
al., 2017a), Beta calibration (Kull et al., 2017), Gamma calibration (Kweon et
al., 2022) and Dirichlet calibration (Kull et al., 2019)), non-parameter
methods (such as Histogram Binnning (Zadrozny and Elkan, 2001) and Isotonic
Regression (Zadrozny and Elkan, 2002)) and hybrid methods. Hybrid methods
encompass both non-field-aware (Kumar et al., 2019; Jiang et al., 2011; Zhang
et al., 2020) and field-aware approaches (Pan et al., 2020; Wei et al., 2022;
Huang et al., 2022).

However, what sets apart e-commerce advertising scenarios is the challenge of
multi-field calibration, which means that for each pCTR range in each value
under each concerned field the pCTR should be calibrated. Multi-field
calibration can be subdivided into two distinct sub-problems: value
calibration and shape calibration. Value calibration is defined as no over- or
under-estimation each value under concerned fields (such as average of pCTR
should equal to the CTR for value ”women’s shoes” in the field ”category”, in
the e-commerce advertising context, the number of fields can range from dozens
to even hundreds). Figure 1 illustrates the significant inconsistency of over-
and under-estimation across different field values. Shape calibration is
defined as no over- or under-estimation for each subset of the pCTR within the
specified range for each value under concerned fields. Take Figure 2 as an
example, Field $f$ has two values: Value1 and Value2, and these two shapes are
significantly different. In order to achieve shape calibration and value
calibration, it is necessary to have a strong data utilization ability.
Because the samples of the specified pCTR range for single field-value (such
as user ID and item ID) are relative small, which makes the calibrator more
difficult to train. However, the existing method cannot simultaneously fulfill
both value calibration and shape calibration.

For Parameter methods, non-parameter methods and non-field-aware methods, they
involve the training of a single calibration function to address the issue of
shape miscalibration. These methods focus on shape calibration but not value
calibration.

For field-aware methods, they have different emphases, and cannot
simultaneously fulfill both value calibration and shape calibration.
NeuralCalib/Field-aware Calibration (FAC) (Pan et al., 2020) places its focus
on modeling estimation value biases across various field values, rather than
addressing shape miscalibrations. Multiple Boosting Calibration Tree (MBCT)
(Huang et al., 2022) effectively enhances the binning of field values, giving
less emphasis to shape miscalibrations. AdaCalib (Ada) (Wei et al., 2022) can
only model shape calibration and value calibration under the condition of a
single field.

An analysis of the aforementioned field-aware methods reveals several key
observations. Firstly, within a single field, there exists multiple values. In
some cases, the allocation of samples to each field value may be limited,
particularly in scenarios involving user ID, item ID and certain sparse cross-
features. Consequently, training a calibrator under such limited samples can
be challenging. Additionally, field-aware methods often employ a binning
strategy, followed by the generation of calibration parameters based on
information from two adjacent bins. This approach can lead to a sample
isolation issue wherein the parameters of a bin are exclusively influenced by
the samples within that bin and its neighboring bins, with no updates from
samples in other bins. As a result, these methods can result in suboptimal
data utilization.

To solve these above problems, we propose a new method named Deep Ensemble
Shape Calibration (DESC). There are four contributions in our work:

  * •

We redefine the multi-field calibration: perform shape calibration and value
calibration at the same time.

  * •

We propose the novel basis calibration functions, which can simultaneously
improve function expression ability and data utilization through the
combination of basis calibration functions.

  * •

We make a breakthrough in putting forward an allocator that can allocate the
most suitable shape calibrators for different estimation error distributions
on various fields and values.

  * •

Two public datasets and one industrial dataset are tested to show that our
proposed DESC method achieves significantly better performance compared with
other methods in both calibration metrics and non-calibration metrics. And we
get +2.5% of CVR and +4.0% of GMV on the online experiment.

##  2\. RELATED WORKS

With the growing emphasis on improving the reliability and accuracy of machine
learning models, several research approaches have emerged. Numerous
calibration methods focus on acquiring a mapping function to convert predicted
probabilities into observed posterior probabilities, referred to as post-hoc
calibration. These studies can be broadly classified into three categories:
non-parameter, parameter, and hybrid methods.

###  2.1. Non-parametric Methods

Non-parametric methods, in contrast, do not rely on any assumptions regarding
the distribution of estimates. Examples of non-parametric methods encompass
Histogram Binning (HB)(Zadrozny and Elkan, 2001) and Isotonic Regression (IR)
(Zadrozny and Elkan, 2002). HB involves sorting the estimated values and then
dividing them into bins of equal frequencies or intervals, utilizing the
posterior CTR of each bin as a calibration parameter. Building upon this, IR
introduces the additional constraint of order preservation. These approach can
introduce instability in the bin boundary values.

###  2.2. Parametric Methods

Parameter methods often rely on specific probability distribution assumptions
for deriving mapping functions. Platt Scaling (Platt et al., 1999),
extensively employed in binary classification calibration, assumes a Gaussian
distribution with equal variances for both positive and negative classes
(Kweon et al., 2022). For multi-class tasks, Temperature Scaling (Guo et al.,
2017a) extends this approach. Beta calibration (Kull et al., 2017), Gamma
calibration (Kweon et al., 2022) and Dirichlet calibration (Kull et al., 2019)
rely on their respective probability distributions for calibration. Parameter
methods are highly reliant on the strong distribution assumption, which can
lead to suboptimal performance if the assumption does not hold in practical
scenarios.

###  2.3. Hybrid Methods

Hybrid methods integrate non-parametric and parametric approaches. Based on
whether they are applied at the field-level, we further categorize hybrids
into two groups: non-field-aware and field-aware methods.

####  2.3.1. Non-field-aware Methods

Non-field-aware methods encompass three notable approaches: Scaling-binning
(Kumar et al., 2019), Smooth Isotonic Regression (SIR) (Jiang et al., 2011),
and Ensemble Temperature Scaling (ETS) (Zhang et al., 2020). Scaling-binning
combines the techniques of Platt Scaling and Histogram Binning. SIR employs
linear interpolation based on isotonic regression, while ETS combines multiple
temperature scalings for calibration, offering greater expressiveness than
temperature scaling. Notably, all of these methods utilize raw predicted
scores as input without taking field information into account.

####  2.3.2. Field-aware Methods

Field-aware methods integrate supplementary features to formulate calibration
functions. NeuralCalib (FAC) (Pan et al., 2020) introduces an auxiliary module
by utilizing neural networks and incorporates multiple field modeling.
However, it does not address the challenge of shape miscalibration variance
among different field values. AdaCalib (Ada) (Wei et al., 2022) learns an
isotonic function based on posterior statistics and selects the most
appropriate bin number for a single field value. However, AdaCalib does not
encompass all fields. MBCT (Huang et al., 2022) employs trees to uncover more
effective calibration across fields. However, due to its high computational
complexity, calibration trees are unable to handle sparse fields, such as user
ID and item ID, and do not adequately address discrepancies in shape
calibration.

##  3\. PROBLEM FORMULATION

In the context of online advertising systems, using CTR as an illustration
(CVR follows the same principles), we have trained a neural predictor, denoted
as $f_{uncalib}$, on a training dataset $\mathcal{D}_{train}$. This dataset
includes all field values as inputs ($x$) and click responses ($y$), where
$y=0$ represents non-click events and $y=1$ signifies click events. The neural
predictor is capable of forecasting the likelihood of a click using the
following formula.

(1) $\hat{p}_{uncalib}=f_{uncalib}(x)$

To address the under- and over-estimation issues associated with the predicted
scores generated by $f_{uncalib}$, we must train an additional calibrator,
denoted as $f_{calib}$, while considering $n$ distinct fields ($Z_{1}$,
$Z_{2}$, …, $Z_{n}$) in a validation dataset. Our objective is for $f_{calib}$
to predict the conditional expectation $\mathbf{E}[y|x]$. Subsequently, the
theoretical calibration error of $f_{calib}$ with respect to the ground-truth
under the $l_{p}$-norm is defined as:

(2)
$TCE_{p}(f_{calib})=(\mathbf{E}_{x}[|\mathbf{E}[y|x]-f_{calib}(x)|^{p}])^{\frac{1}{p}}.$

The calibrator $f_{calib}$ is considered to be perfectly calibrated when the
theoretical calibration error $TCE_{p}(f_{calib})$ is zero. However, in fact
perfect calibration is impossible in practice. Only approximate and asymptotic
grouped calibrations are possible for finite and specific partitions of
samples (Gupta et al., 2020). To test the performance of different
calibrators, we will explain some related calibration metrics in EXPERIMENTS
section.

##  4\. METHODS

Figure 3. (a) Overall architecture of DESC, its input includes the non-
calibrated pCTR and the original fields, and the final output is the
calibrated pCTR. It is end-to-end trainable. (b) Single Field Shape Calibrator
takes one field and the non-calibrated pCTR as inputs and outputs the
calibrated shape score for this field.

Under the requirement of multi-field calibration, both shape calibration and
value calibration need to be performed simultaneously. Therefore, within the
entire DESC architecture, we have designed separate modules, namely Shape
Calibrator and Value Calibrator (Figure 3), to achieve shape calibration and
value calibration. The final calibrated score is the product of these two
parts:

(3) $\hat{p}_{calib}=\mathbb{S}(x)\cdot\mathbb{V}(x).$

$\mathbb{S}(x)$ and $\mathbb{V}(x)$ refer to the shape calibrated score (the
output of Shape Calibrator) and value calibrated score (the output of Value
Calibrator), respectively. The input $x$ consists of all $n$ field values and
non-calibrated scores $\hat{p}_{uncalib}$. The training loss function is the
negative log-likelihood function.

(4) $\mathcal{L}(x,\;y)=-\frac{1}{|\mathcal{D}|}(y\cdot
log\hat{p}_{calib}+(1-y)\cdot log(1-\hat{p}_{calib}))$

In Section 4.1, we present the Shape Calibrator module, which is responsible
for achieving shape calibration. In Section 4.2, we discuss the Value
Calibrator module, which is designed to accomplish value calibration. In
Section 4.3, we elaborate on the deployment of DESC in an online setting,
outlining the necessary procedures and considerations.

###  4.1. Shape Calibrator

For multi-field calibration, the goal of Shape Calibrator is to ensure that:
given input features $x$, we need to reduce the problems of over- and under-
estimation across all intervals of pCTR (shape miscalibration).

In section 4.1.1, we pre-define a variety of basis functions to accommodate
different shape requirements. Section 4.1.2 deals with the allocation of
appropriate shape functions given specific field conditions. This section
offers separate discussions on how shape allocation is managed for regular
fields and sparse fields.

In section 4.1.3, when dealing with multiple fields, there may be conflicts in
terms of the influence of different fields on calibration. Thus, we introduce
a multi-field fusion mechanism named Multi-Field Shape Ensemble.

Collectively, sections 4.1.1 and 4.1.2 are referred to as the Single Field
Shape Calibrator (SFSC), as depicted in Figure 3b.

####  4.1.1. Shape Pre-Define

In this section, we pre-define some basis calibration functions. Then, we
merge the basis calibration functions into shape functions to enhance their
expressive ability.

Basis calibration function $\mathcal{B}(x)$ satisfies the following
conditions: 1. The function is monotonically non-decreasing and continuous in
range $(0,1)$.
2. When $x$ approaches $0^{+}$, the limit of $\mathcal{B}(x)$ is 0, and when $x$ approaches $1^{-}$, the limit of $\mathcal{B}(x)$ is 1 (Equation 5).

(5) $\lim_{x\to 0^{+}}\mathcal{B}(x)=0\ and\ \lim_{x\to
1^{-}}\mathcal{B}(x)=1$

In Equation 6, we pre-define $m$ basis functions consisting of $p$ power
functions, $l$ logarithmic functions and $s$ scaling functions. These
functions exhibit different shape characteristics, and their shapes vary when
the hyper-parameters (_e.g._ the coefficients in these basis calibration
functions) are different. Compared to segmented linear functions in
traditional calibration methods, including SIR, FAC, Ada, using basis
calibration functions doesn’t require data segmentation. In other words, shape
learning can utilize all samples for training, significantly enhancing the
data utilization.

(6)
$\begin{matrix}\par\mathcal{B}(x)=\\{\underbrace{\mathcal{B}_{1}^{power},...,\mathcal{B}_{p}^{power}}_{p\
power\
functions},\underbrace{\mathcal{B}_{1}^{log},...,\mathcal{B}_{l}^{log}}_{l\
log\
functions},\underbrace{\mathcal{B}_{1}^{scaling},...,\mathcal{B}_{s}^{scaling}}_{s\
scaling\ functions}\\}\end{matrix}$ (7)
$\mathcal{B}_{i}^{power}(x;h_{i})=x^{h_{i}}$ (8)
$\mathcal{B}_{i}^{log}(x;v_{i})=\frac{log(1+v_{i}\cdot x)}{log(1+v_{i})}$ (9)
$\mathcal{B}_{i}^{scaling}(x;a_{i})=\sigma(\sigma^{-1}(x)\cdot a_{i})\ \ ,$
(10) $where\ \sigma(x)=\frac{1}{1+exp(-x)}\ and\
\sigma^{-1}(x)=log(\frac{x}{1-x})$

$h_{i},v_{i}$ and $a_{i}$ (${h_{i},v_{i},a_{i}}\in\mathbf{R}^{+}$) are
parameters of basis calibration functions. We pre-define these parameters
using equally spaced floats (e.g. $[0.3,0.5,0.7,0.9]$) for each type of basis
calibration function.

However, $\mathcal{B}(x)$ can only represent simple shapes, and complex shapes
can be composed of simple shapes, as shown in Figure 4. Therefore, we need to
combine these basis calibration functions through weighted summation to create
shape function $\mathbb{S}_{i}(x)$ capable of representing complex shapes for
$i$-th field, as shown in Equation 11, where $\alpha_{i}^{j}$ refers to the
attention weight of $j$-th basis function for $i$-th field, which will be
explained in next section.

(11)
$\mathbb{S}_{i}(x)=\sum_{j=1}^{m}{\alpha_{i}^{j}\cdot\mathcal{B}_{j}(x)},\;\;\mathcal{B}_{j}(x)\in\mathcal{B}(x)$

Figure 4. Complex shape can be composed of simple shapes.

####  4.1.2. Shape Allocation

In shape allocation, we allocate suitable shape functions based on feature
values. These features include two parts: pCTR bucket feature and original
field features. Introducing pCTR bucket feature allows for better shape
allocation within different pCTR intervals. The specific steps are as follows:
Step1. We pre-define the embedding size for each feature. Each one-hot feature
(including pCTR bucket feature $bucket_{\hat{p}_{uncalib}}$ and original field
$Z_{i}$) is projected into a fixed-size dense embedding, such as
$\mathbf{b_{\hat{p}_{uncalib}}}$ and $\mathbf{e_{i}}$. The bucket size of
$bucket_{\hat{p}_{uncalib}}$ is assigned as 100.
Step2. We concatenate embedding vectors into a Multi-Layer Perceptron (MLP),
followed by a softmax operation, the output $\mathbf{\alpha}_{i}$ size of
softmax is $1\times m$, with $\mathbf{\alpha}_{i}^{1}$ to
$\mathbf{\alpha}_{i}^{m}$, same as the number of pre-defined basis functions.

(12)
$\mathbf{\alpha}_{i}=Softmax(MLP(\mathbf{b_{\hat{p}_{uncalib}}},\;\mathbf{e_{i}}))$

Step3. Finally, we obtain the output value of the Shape Calibrator
$\mathbb{S}_{i}(x)$ for $Z_{i}$ shown in the previous Equation 11.

In the shape allocation stage, the expressive ability of embedding directly
affects the ability of allocating shapes. For sparse fields, we enhance the
expressive ability of embeddings by using self-attention. The specific formula
is shown in following formula, where $d$ is the dimension of $\mathbf{e}$ and
$n$ is the number of fields.

(13)
$\mathbb{X}(\mathbf{e_{i}})=\sum_{j=1}^{n}{(Softmax(\frac{\mathbf{e_{i}}\cdot\mathbf{e_{j}}^{T}}{\sqrt{d}})\cdot\mathbf{e_{j}})\textbf{1}_{[i\neq
j]}}$

That means: for more similar fields, their miscalibration distributions will
be similar. If the semantics between two embeddings $\mathbf{e_{i}}$ and
$\mathbf{e_{j}}$ is similar, then the corresponding weight is also relatively
large.

(14)
$\mathbf{\alpha}_{i}=Softmax(MLP(\mathbf{b_{\hat{p}_{uncalib}}},\;\mathbf{e_{i}},\;\mathbb{X}(\mathbf{e_{i}})))$

Next, we concatenate the original embedding $\mathbf{e_{i}}$, the enhanced
output embedding $\mathbb{X}(\mathbf{e_{i}})$ , and the pCTR bucket embedding
$\mathbf{b_{\hat{p}_{uncalib}}}$. Then, following the same process as in the
shape allocation stage, we generate the attention for shape allocation
$\mathbf{\alpha}_{i}$ and perform the final fusion.

####  4.1.3. Multi-Field Shape Ensemble

Different calibration values for different fields can conflict with each
other. For example, there is a scenario where 19 to 26-year-old users demand
for women’s shoes. Under the item category field of ”women’s shoes”, the non-
calibrated model behaves 30% over-estimation of pCTR in the 0.01 to 0.03
range. However, under the user age field of ”19 to 26”, there’s a 20% under-
estimation of pCTR in the same 0.01 to 0.03 range.

Consequently, we not only need to perform shape calibration on individual
field but also need to globally harmonize the outputs of shape calibrators for
different fields. This helps reduce calibration errors at a global level.

We use Global Shape Attention to combine the output results obtained from
different fields (Figure 3a). Global Shape Attention, denoted as $\Psi$, is
derived from the Global Shape Attention Generator module (details will be
elaborated in section 4.2.2). The size of $\Psi$ is $1\times n$, with
$\Psi_{i}\in[0,1]$, and $\sum_{i=1}^{n}{\Psi_{i}}=1$. The fusion formula is as
follows, where there are a total of $n$ fields, and $\mathbb{S}_{i}(x)$
represents the output of the shape calibrator for $i$-th field.

(15) $\mathbb{S}(x)=\sum_{i=1}^{n}{\mathbb{S}_{i}(x)\cdot\Psi_{i}}$

We finally obtain the output of Global Shape Calibrator: $\mathbb{S}(x)$.

###  4.2. Value Calibrator

The goal of the Value Calibrator is to ensure that, for each sample $x$, there
is no overall over- or under-estimation (value calibration). We use all fields
for Value Calibrator to achieve the best overall performance, as described in
section 4.2.1. Considering all fields allows for the excellent allocation of
shapes for each field, we use global shape attention depicted in section
4.2.2.

####  4.2.1. Global Field Value Calibrator

Global Field Value Calibrator encompasses all the necessary information of the
fields concerned. These input fields are projected into fixed-size dense
representations, which are fed into a neural network (the form of the neural
network is not restricted, here we use MLP). Then we get the middle output
$hiddenLayer$ and the final output $\mathbb{V}(x)$ for the Global Field Value
Calibrator. Finally, by combining $\mathbb{V}(x)$ and $\mathbb{S}(x)$, we
obtain the final calibrated output
$\hat{p}_{calib}=\mathbb{S}(x)\cdot\mathbb{V}(x)$.

(16)
$hiddenLayer=MLP_{1}(Concat(\mathbf{b_{\hat{p}_{uncalib}}},\mathbf{e}_{1},\mathbf{e}_{2},...,\mathbf{e}_{n}))$
(17) $\mathbb{V}(x)=MLP_{2}(hiddenLayer)$

####  4.2.2. Global Shape Attention Generator

The Global Field Value Calibrator not only produces the final output but also
yields valuable intermediate representation. In this context, we use the
intermediate representation $hiddenLayer$ as input of Equation 18, which can
effectively learn information from all fields. After passing through a MLP and
softmax layer, it produces Global Shape Attention Score $\Psi$ with a size
equal to the number of fields $n$. In the Shape Calibration stage, the output
produced by each field $Z_{i}$ is multiplied by its corresponding score
$\Psi_{i}$ and summed to obtain the final Shape Calibration Score.

(18) $\Psi=Softmax(MLP_{3}(hiddenLayer))$

###  4.3. Online Deployment

The overall framework of the online service using the DESC calibration method
in our industrial advertising system is shown in Figure 5. When a real-time
user request comes, all candidate items are recalled and predicted with pCTRs
by a non-calibration model (ranking model). Then DESC model calibrates the
pCTRs considering different field values, including user, item, and context
features. In detail, the input $x$ of DESC has two parts: different field
values and its non-calibrated score $\hat{p}_{uncalib}$. DESC uses a light
neural network to output the calibrated score $\hat{p}_{calib}$, which is then
used to sort the final ranks for all candidate items. We have integrated the
DESC method as a plugin into the ranking model to reduce maintenance costs
while only adding a little more time for online inference.

Figure 5. Real-Time calibration system used DESC method for CXR (CTR/CVR) task
in our industrial advertising system.

##  5\. EXPERIMENTS

###  5.1. Experimental Setup

####  5.1.1. Datasets.

To validate the effectiveness of our proposed DESC method, we conduct
experiments on two public datasets (AliCCP and CRITEO for CTR prediction
tasks) and one industrial dataset (Shopee for CVR prediction task). CRITEO
display advertising data 222 https://www.kaggle.com/c/criteo-display-ad-
challenge, which consists of 46 million samples. We split the samples into 28
million samples as training set ($\mathcal{D}_{train}$), 9 million samples as
validation set ($\mathcal{D}_{dev}$) and other 9 million samples as testing
set ($\mathcal{D}_{test}$). AliCCP (Alibaba Click and Conversion Prediction)
(Ma et al., 2018) 333
https://tianchi.aliyun.com/datalab/dataSet.html?dataId=408 contains 80 million
samples. We split these samples with the proportion 2:1:1 to
$\mathcal{D}_{train}$, $\mathcal{D}_{dev}$ and $\mathcal{D}_{test}$. To test
DESC method on CVR prediction task, we collect the conversion logs from the
Shopee’s online advertising system. It contains 100 million samples, where the
first 60 million for $\mathcal{D}_{train}$, the next 20 million for
$\mathcal{D}_{dev}$ and the last 20 million for $\mathcal{D}_{test}$.

####  5.1.2. Competing Methods.

Several representative calibration methods are used as competitors. We tested
several competitive methods, including parametric methods, non-parametric
methods, non-field-aware hybrid methods and field-aware hybrid methods.
Parametric method contains Platt Scaling (PS) (Platt et al., 1999). Non-
parametric method contains Histogram Binning (HB) (Zadrozny and Elkan, 2001)
and Istonic Regression (IR) (Zadrozny and Elkan, 2002). Non-field-aware method
contains Smooth Isotonic Regression (SIR) (Jiang et al., 2011) and Ensemble
Temperature Scaling (ETS) (Zhang et al., 2020). Field-aware method contains
FAC (Pan et al., 2020), Ada (Wei et al., 2022) and MBCT (Huang et al., 2022).

We use DeepFM (Guo et al., 2017b) to train the non-calibrated models with all
fields in $\mathcal{D}_{train}$, and predict the non-calibrated scores in
$\mathcal{D}_{dev}$ and $\mathcal{D}_{test}$.

All calibration methods are trained using samples of $\mathcal{D}_{dev}$ and
are tested using samples of $\mathcal{D}_{test}$.

####  5.1.3. Parameter Configuration.

For all neural calibrators, including FAC, Ada and DESC, we use Adam as the
optimizer with a learning rate of 1e-3 and batch size of 16,384. For the
embedding size of field value $d$, both FAC and Ada are set as 256 while for
DESC, it is 128 because it obtains a better result. For FAC, we set the number
of bins to 100 while for Ada, the candidate set of bin numbers is set {2, 4,
8} following in (Wei et al., 2022). For DESC, the number of basis functions
$m$ is 48 (the numbers of the three types of basis functions are both 16), and
the numbers of fields $n$ are 26, 23 and 10 for Criteo, AliCCP and Industrial
dataset, respectively. For MBCT, we set the same hyperparameters according to
the paper.

####  5.1.4. Compared Metrics.

Two commonly used metrics, F-RCE (Field RCE) and F-ECE (Field ECE) for each
field $Z_{i}$ are used. To calculate the F-RCE of $i$-th field (shown in
Equation 19), we use the testing dataset $\mathcal{D}_{test}$ (for simplicity,
$\mathcal{D}$ refers to $\mathcal{D}_{test}$ here) consisting of
$(x^{j},y^{j})$, where $x^{j}$ and $y^{j}$ mean the input features with
different fields and clicked label of $j$-th sample. The subset of
$\mathcal{D}^{z}$ has the samples with the same value $z$ for $i$-th field.
F-RCE of $i$-th field evaluates the deviation level of each sample’s
calibrated probability $\hat{p}_{calib}^{j}$ considering $j$-th field.

Another metric is the F-ECE of $i$-th field. As shown in Equation 20, it can
be calculated for each subset $D^{z}$ of samples with the same value in $i$-th
field. We can calculate the subset of F-ECE (shown in Equation 21) by
partitioning predictions into $M$ equally-spaced bins and taking a weighted
average of the difference between bins’ accuracy and confidence. In detail,
firstly we sort the samples by the non-calibrated scores
$\hat{p}_{uncalib}^{j}$. Then all samples are grouped into $M$ interval bins.
For all samples ($B_{m}$) in $m$-th bin, we calculate the accuracy and
confidence, where the accuracy is the average of labels and the confidence is
the average of calibrated scores.

(19) $F\text{-}RCE_{i}=\frac{1}{|\mathcal{D}|}\sum_{z\in
Z_{i}}\frac{|\sum_{(x^{j},y^{j})\in\mathcal{D}^{z}}(y^{j}-\hat{p}_{calib}^{j})|}{\frac{1}{|\mathcal{D}^{z}|}\sum_{(x^{j},y^{j})\in\mathcal{D}^{z}}y^{j}}$
(20) $F\text{-}ECE_{i}@M=\frac{1}{|\mathcal{D}|}\sum_{z\in
Z_{i}}|\mathcal{D}^{z}|\cdot(F\text{-}ECE_{\mathcal{D}^{z}}@M)$ (21)
$F\text{-}ECE_{\mathcal{D}^{z}}@M=\sum_{m=1}^{M}\frac{|B_{m}|}{|\mathcal{D}^{z}|}|acc(B_{m})-conf(B_{m})|$
(22) $acc(B_{m})=\frac{1}{|B_{m}|}\sum_{j\in
B_{m}}y^{j},\;conf(B_{m})=\frac{1}{|B_{m}|}\sum_{j\in
B_{m}}\hat{p}_{calib}^{j}$

As explained that the traditional field-aware calibration methods, such as FAC
(Pan et al., 2020) and Ada (Wei et al., 2022), only consider one field to
calibrate, these methods do not take into account the influence of other
fields on the calibration results. We found that the calibration results from
these models only training on one field perform poorly on other fields. So we
use the Multi-Field RCE (MF-RCE) and Multi-Field ECE (MF-ECE) on all fields to
compare different methods as follows:

(23) $MF\text{-}RCE=\frac{1}{n}\sum_{i}F\text{-}RCE_{i}$

(24) $MF\text{-}ECE@M=\frac{1}{n}\sum_{i}F\text{-}ECE_{i}@M,$

where $n$ is the number of fields.

We use F-RCE and F-ECE as the primary empirical metrics to compare the
calibration errors of different calibration methods on one field and MF-RCE
and MF-ECE to test on all fields. Besides, we also report the overall ranking
performance by using AUC and Log-loss metrics. Since PCOC (Predicted Click
Over Click) (He et al., 2014; Graepel et al., 2010) and ECE (Expected
Calibration Error) (Huang et al., 2022) indicators can be achieved well for
all competing methods, and F-ECE and MF-ECE, as compared to the traditional
ECE, can provide a finer-grained representation of calibration error in the
context of field-aware problems, PCOC and ECE are not listed in the main
results.

###  5.2. Main Results

Table 1. Results of different methods for calibrating CTR predictive models on public and industrial datasets for one field. Type | Method | Criteo | AliCCP | Industrial Data
---|---|---|---|---
AUC $\uparrow$ |  Log-loss $\downarrow$ |  F-RCE $\downarrow$ |  F-ECE@3 $\downarrow$ |  AUC $\uparrow$ |  Log-loss $\downarrow$ |  F-RCE $\downarrow$ |  F-ECE@3 $\downarrow$ |  AUC $\uparrow$ |  Log-loss $\downarrow$ |  F-RCE $\downarrow$ |  F-ECE@3 $\downarrow$
No Calib. | N/A | 0.7949 | 0.4559 | 0.0426 | 0.0119 | 0.6130 | 0.1611 | 0.0876 | 0.0135 | 0.8613 | 0.5212 | 0.0743 | 0.0253
Non-Param | HB | 0.7944 | 0.4562 | 0.0328 | 0.0112 | 0.6128 | 0.1621 | 0.0852 | 0.0130 | 0.8610 | 0.5218 | 0.0669 | 0.0204
IR | 0.7949 | 0.4559 | 0.0320 | 0.0110 | 0.6130 | 0.1611 | 0.0843 | 0.0132 | 0.8613 | 0.5212 | 0.0665 | 0.0202
Param | PS | 0.7949 | 0.4559 | 0.0301 | 0.0105 | 0.6130 | 0.1611 | 0.0841 | 0.0128 | 0.8613 | 0.5212 | 0.0660 | 0.0201
Non-Field- Aware | SIR | 0.7949 | 0.4559 | 0.0300 | 0.0097 | 0.6130 | 0.1611 | 0.0836 | 0.0127 | 0.8613 | 0.5211 | 0.0660 | 0.0201
ETS | 0.7949 | 0.4559 | 0.0293 | 0.0095 | 0.6130 | 0.1611 | 0.0837 | 0.0127 | 0.8613 | 0.5212 | 0.0661 | 0.0199
Field- Aware | FAC | 0.7991 | 0.4523 | 0.0265 | 0.0087 | 0.6751 | 0.1571 | 0.0829 | 0.0123 | 0.8634 | 0.5203 | 0.0402 | 0.0163
Ada | 0.7992 | 0.4521 | 0.0273 | 0.0089 | 0.6752 | 0.1568 | 0.0832 | 0.0125 | 0.8639 | 0.5201 | 0.0413 | 0.0170
MBCT | 0.7951 | 0.4550 | 0.0248 | 0.0095 | 0.6742 | 0.1611 | 0.0828 | 0.0128 | 0.8621 | 0.5209 | 0.0409 | 0.0175
DESC | 0.7992 | 0.4522 | 0.0203 | 0.0080 | 0.6774 | 0.1566 | 0.0821 | 0.0120 | 0.8641 | 0.5192 | 0.0307 | 0.0132
Table 2. Results of different methods for calibrating CTR predictive models on public and industrial datasets for all fields. Type | Method | Criteo | AliCCP | Industrial Data
---|---|---|---|---
MF-RCE $\downarrow$ |  MF-ECE@3 $\downarrow$ |  MF-RCE $\downarrow$ |  MF-ECE@3 $\downarrow$ |  MF-RCE $\downarrow$ |  MF-ECE@3 $\downarrow$
No Calib. | N/A | 0.0702 | 0.0142 | 0.1492 | 0.0143 | 0.0793 | 0.0370
Non-Param | HB | 0.0692 | 0.0134 | 0.1430 | 0.0135 | 0.0779 | 0.0250
IR | 0.0688 | 0.0131 | 0.1427 | 0.0131 | 0.0766 | 0.0249
Param | PS | 0.0685 | 0.0130 | 0.1420 | 0.0129 | 0.0765 | 0.0248
Non-Field- Aware | SIR | 0.0686 | 0.0131 | 0.1422 | 0.0130 | 0.0764 | 0.0247
ETS | 0.0683 | 0.0127 | 0.1470 | 0.0129 | 0.0761 | 0.0248
Field- Aware | FAC | 0.0517 | 0.0101 | 0.1321 | 0.0120 | 0.0412 | 0.0183
Ada | 0.0501 | 0.0102 | 0.1338 | 0.0122 | 0.0415 | 0.0180
MBCT | 0.0660 | 0.0104 | 0.1357 | 0.0125 | 0.0430 | 0.0192
DESC | 0.0489 | 0.0099 | 0.1264 | 0.0115 | 0.03606 | 0.0132

####  5.2.1. Results on One Field.

As other hybrid field-aware based methods only consider one field to
calibrate, we compare our proposed method in one field in the same way. We
select the field of ”C2” (577 unique values) in Criteo, ”853” (39,979 unique
values) in AliCCP and ”item ID” (5 million unique values) in Industrial data
to perform our experiments. As shown in Table 1, the AUCs of all calibrators
are higher than the uncalibrator while all other calibration-related metrics
(F-ECE, F-RCE) of calibrators are smaller than the uncalibrator, which shows
that the calibration methods can improve the accuracy of pCTR while
maintaining a certain increase in ranking performance. Specially, by fusing
multiple fields in shape allocation and shape augmentation, our proposed DESC
can achieve much smaller calibration errors compared to other competitors even
in one field evaluation. It’s worth explaining that DESC performs much better
than other competitors in industrial data than public data. The reason is that
DESC behaves better in calibration performance in this scene where the data
sparsity is more serious in practical industrial data.

####  5.2.2. Results on All Fields.

As DESC can consider the effects of different fields on calibration, we need
to evaluate its performance on all fields. As shown in Table 2, the conclusion
is consistent with the smallest calibration errors on all three datasets.

###  5.3. In-Depth Analysis

####  5.3.1. Model Structure Ablation Experiment for DESC

We perform some model structure ablation experiments to validate the
significance and effectiveness.

(1) Ablation Experiment of Shape Calibrator. To evaluate the effectiveness of
the Shape Calibrator and Value Calibrator, we delete the corresponding module
and left the other parts unchanged. Firstly, we delete the Shape Calibrator
sub-module, the AUC reduces to 0.6602 while the metrics of MF-RCE and MF-ECE@3
on all fields are slightly larger than DESC with 0.1434 and 0.0120 (first line
on Table. 3).

(2) Ablation Experiment of Value Calibrator. Secondly, we delete another sub-
module, the Value Calibrator. The phenomenon is almost the same, with 0.6761
of AUC, and 0.1398 and 0.0119 of MF-RCE and MF-ECE@3 on all fields (second
line on Table. 3).

(3) Ablation Experiment of Multi-Field Shape Ensemble. In METHODS section, we
have emphasized that shape calibration needs to consider multiple fields
because different fields and values can have numerous different shapes. Then,
we only use mean-pooling to compute the final predicted result considering
different fields ($\Psi_{i}$ is all the same with $1/n$ in Equation 15). The
results are still inferior as shown in the third line in Table. 3.

(4) Ablation Experiment of pCTR Bucket Feature. The pCTR bucket feature is
concatenated with the field embedding to calculate the shape allocation
weights in DESC (Equation 14). Then we delete the pCTR bucket feature (fourth
line on Table. 3) to verify that different pCTR values can also have effects
on the calibration shape.

(5) Ablation Experiment of Embedding Augmentation. To enhance the expressive
ability of embeddings, the self-attention mechanism is used. Then we delete
this part by using the original Equation 12 rather than Equation 14. The fifth
line on Table. 3 shows that the embedding augmentation mechanism can enhance
the representational power of embedding.

Table 3. Ablation study of DESC on AliCCP. Type |  AUC $\uparrow$ |  Log-loss $\downarrow$ |  MF-RCE $\downarrow$ |  MF-ECE@3 $\downarrow$
---|---|---|---|---
w/o Shape Calibrator | 0.6602 | 0.1581 | 0.1434 | 0.0120
w/o Value Calibrator | 0.6761 | 0.1567 | 0.1398 | 0.0119
w/o Multi-Field Shape Ensemble | 0.6755 | 0.1568 | 0.1324 | 0.0116
w/o pCTR Bucket Feature | 0.6764 | 0.1567 | 0.1391 | 0.0120
w/o Embedding Augment | 0.6764 | 0.1567 | 0.1398 | 0.0119
DESC | 0.6774 | 0.1566 | 0.1264 | 0.0115

####  5.3.2. DESC exhibits a stronger data utilization capability.

In this part, we analyze the data utilization capabilities of different
calibrators from both a global perspective (overall down sampling) and an
individual perspective (field value sample quantity).

(1) Overall Down Sampling  We further conducted experiments to compare the
calibration performance of four different hybrid field-aware methods (FAC,
Ada, MBCT and DESC) across various sampling ratios. As shown in Figure 6, DESC
exhibits a distinct lower MF-ECE in the context of varying sampling ratios. In
a global perspective, this demonstrates that DESC exhibits a stronger data
utilization capability.

Figure 6. Down sampling analysis in CRITEO.

(2) Field Value Sample Quantity. We analysis the performance differences among
MBCT, FAC, Ada and DESC at sample quantity aspect. We set the quantity of
samples with a specific field value (for the convenience of display, we use
$log_{10}(x)$) as the x-axis and $EER_{Calib}$ ($\mathbf{E}$xcept Calibration
$\mathbf{E}$rror $\mathbf{R}$atio) as the y-axis (the smaller $EER_{Calib}$
is, the better DESC is compared with other calibrators), as shown in Figure 7.

(25) $EER_{Calib}=\frac{ECE_{DESC}}{ECE_{Calib}},Calib\in\\{MBCT,FAC,Ada\\}$

ECE (Expected Calibration Error) (Huang et al., 2022) is used as the
calibration error to explain the results. The ECE value ranges from
$(0,+\infty]$. The higher ECE value indicates higher calibration error and
worse performance. We can find that for the samples with small sized of
number, $EER$ is less than 1, which means, from an individual perspective,
that DESC possesses a stronger data utilization capability.

Figure 7. Performance differences for the sample quantity of the field value.

####  5.3.3. DESC Can Adapt to Complex Miscalibration Shapes.

We analysis the performance differences among MBCT, FAC, Ada and DESC at shape
complexity.

From the perspective of pCTR, we divide the estimated pCTR values into bins
either with equal frequency or equal interval, and each bin can calculate the
PCOC (Predicted Click Over Click) (He et al., 2014; Graepel et al., 2010) to
describe the over- and under-estimation within that bin.

For some cases, the over- and under-estimation within each bin are close,
while in other cases, there is a significant difference in over- and under-
estimation for each bin. We define the ”Miscalibration Complexity” metric to
characterize the disparity in over- and under-estimation across different pCTR
bins. We selected all field values for a subset of fields. For field $Z$ with
a value $v$, the samples are divided into $Q$ ($Q>1$) bins. The miscalibration
complexity is denoted as $\mathbb{D}_{Z,v}$ as follows:

(26) $\mathbb{D}_{Z,v}=\frac{1}{Q-1}\sum_{bin=1}^{Q-1}{\lvert
PCOC_{bin+1}-PCOC_{bin}\rvert}.$

In general, the higher the miscalibration complexity, the higher the disparity
in over- and under-estimation within the bins. For a calibrator, a higher
miscalibration complexity indicates a greater calibration challenge. On one
hand, it requires higher precision in shaping the accuracy of the calibration,
and on the other hand, an excessive reliance on posterior data can compromise
the generalization capability of the calibrator.

We still use the ECE metric to characterize the calibration error of different
calibrators under the specific field value. We compare MBCT, FAC, Ada, and
DESC and obtain the $EER_{Calib}$.

Figure 8. Performance differences for miscalibration complexity aspect.

For each calibrator and each field value, there is a corresponding
$\mathbb{D}_{Z,v}$ value as the x-axis and $EER_{Calib}$ as the y-axis. As
shown in Figure 8, we can observe that in the high $\mathbb{D}_{Z,v}$ range,
all $EER_{Calib}$ values are less than 1, which indicates that DESC always
performs better in conditions of complex shapes compared to other methods.

###  5.4. Online Results

To prove the effectiveness of the proposed method, we further conduct real-
world online experiments. We feed item ID with 5 million unique values,
category ID with 5,000 unique values, bidding type, time, raw predict score
and other features into DESC. Then the model is deployed for online pCVR
calibration on Shopee’s advertising system. For the online A/B test, we build
two experimental buckets, each with ten percent of the traffic. One bucket was
configured with SIR (Jiang et al., 2011) as the control group, while the other
was configured with DESC as the experimental group. We use the two significant
important business metrics CVR (Conversion Rate) and GMV (Gross Merchandise
Volume) to evaluate our calibration effectiveness. The online results over a
7-day period indicated that DESC brings +2.5% on CVR and +4.0% on GMV compared
to SIR. These improvements verify the effectiveness and business values of
DESC in practical advertising system.

##  6\. DISCUSSION

We propose a new calibration approach named DESC. Firstly, we put forward
multiple types of functions (_e.g._ , power function, scaling function,
logarithmic function) as the basis functions. Secondly, we allocate the
appropriate basis functions to combine the shape function given the specific
field and specific value. Finally, all fields are concatenated and used to
calculate the weight for the calibration value from each field, which can
further improve the accuracy of pCTR on multiple fields. Both offline and
online experiments verify that DESC achieves significant improvement. In
future work, we should explore the performance of our proposed method on other
machine learning tasks except CXR task. Also, we need to study the reasons of
miscalibration error in these machine learning tasks, especially for CXR task,
and find new technology (_e.g._ data augment, multi-tasking learning) to
further improve the performance.

## References

  * (1)
  * Bella et al. (2010) Antonio Bella, Cèsar Ferri, José Hernández-Orallo, and María José Ramírez-Quintana. 2010.  Calibration of machine learning models.  In _Handbook of Research on Machine Learning Applications and Trends: Algorithms, Methods, and Techniques_. IGI Global, 128–146.
  * Bian et al. (2022) Weijie Bian, Kailun Wu, Lejian Ren, Qi Pi, Yujing Zhang, Can Xiao, Xiang-Rong Sheng, Yong-Nan Zhu, Zhangming Chan, Na Mou, et al. 2022\.  CAN: feature co-action network for click-through rate prediction. In _Proceedings of the fifteenth ACM international conference on web search and data mining_. 57–65.
  * Cheng et al. (2016) Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. 2016\.  Wide & deep learning for recommender systems. In _Proceedings of the 1st workshop on deep learning for recommender systems_. 7–10.
  * Davidson et al. (2010) James Davidson, Benjamin Liebald, Junning Liu, Palash Nandy, Taylor Van Vleet, Ullas Gargi, Sujoy Gupta, Yu He, Mike Lambert, Blake Livingston, et al. 2010\.  The YouTube video recommendation system. In _Proceedings of the fourth ACM conference on Recommender systems_. 293–296.
  * Deng et al. (2021) Chao Deng, Hao Wang, Qing Tan, Jian Xu, and Kun Gai. 2021.  Calibrating user response predictions in online advertising. In _Machine Learning and Knowledge Discovery in Databases: Applied Data Science Track: European Conference, ECML PKDD 2020, Ghent, Belgium, September 14–18, 2020, Proceedings, Part IV_. Springer, 208–223.
  * Du et al. (2021) Chao Du, Zhifeng Gao, Shuo Yuan, Lining Gao, Ziyan Li, Yifan Zeng, Xiaoqiang Zhu, Jian Xu, Kun Gai, and Kuang-Chih Lee. 2021.  Exploration in online advertising systems with deep uncertainty-aware learning. In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_. 2792–2801.
  * Graepel et al. (2010) Thore Graepel, Joaquin Quinonero Candela, Thomas Borchert, and Ralf Herbrich. 2010.  Web-scale bayesian click-through rate prediction for sponsored search advertising in microsoft’s bing search engine. Omnipress.
  * Guo et al. (2017a) Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017a.  On calibration of modern neural networks. In _International conference on machine learning_. PMLR, 1321–1330.
  * Guo et al. (2017b) Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. 2017b.  DeepFM: a factorization-machine based neural network for CTR prediction.  _arXiv preprint arXiv:1703.04247_ (2017).
  * Gupta et al. (2020) Chirag Gupta, Aleksandr Podkopaev, and Aaditya Ramdas. 2020.  Distribution-free binary classification: prediction sets, confidence intervals and calibration.  _Advances in Neural Information Processing Systems_ 33 (2020), 3711–3723.
  * He et al. (2014) Xinran He, Junfeng Pan, Ou Jin, Tianbing Xu, Bo Liu, Tao Xu, Yanxin Shi, Antoine Atallah, Ralf Herbrich, Stuart Bowers, et al. 2014\.  Practical lessons from predicting clicks on ads at facebook. In _Proceedings of the eighth international workshop on data mining for online advertising_. 1–9.
  * Huang et al. (2022) Siguang Huang, Yunli Wang, Lili Mou, Huayue Zhang, Han Zhu, Chuan Yu, and Bo Zheng. 2022.  MBCT: Tree-Based Feature-Aware Binning for Individual Uncertainty Calibration. In _Proceedings of the ACM Web Conference 2022_. 2236–2246.
  * Huang et al. (2019) Tongwen Huang, Zhiqi Zhang, and Junlin Zhang. 2019.  FiBiNET: combining feature importance and bilinear feature interaction for click-through rate prediction. In _Proceedings of the 13th ACM Conference on Recommender Systems_. 169–177.
  * Jiang et al. (2011) Xiaoqian Jiang, Melanie Osl, Jihoon Kim, and Lucila Ohno-Machado. 2011.  Smooth isotonic regression: a new method to calibrate predictive models.  _AMIA Summits on Translational Science Proceedings_ 2011 (2011), 16.
  * Korb (1999) Kevin B Korb. 1999.  Calibration and the evaluation of predictive learners. In _Proceedings of the Sixteenth International Joint Conference on Artificial Intelligence_. Citeseer, 73–77.
  * Kull et al. (2019) Meelis Kull, Miquel Perello Nieto, Markus Kängsepp, Telmo Silva Filho, Hao Song, and Peter Flach. 2019.  Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration.  _Advances in neural information processing systems_ 32 (2019).
  * Kull et al. (2017) Meelis Kull, Telmo Silva Filho, and Peter Flach. 2017.  Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers. In _Artificial Intelligence and Statistics_. PMLR, 623–631.
  * Kumar et al. (2019) Ananya Kumar, Percy S Liang, and Tengyu Ma. 2019.  Verified uncertainty calibration.  _Advances in Neural Information Processing Systems_ 32 (2019).
  * Kweon et al. (2022) Wonbin Kweon, SeongKu Kang, and Hwanjo Yu. 2022.  Obtaining Calibrated Probabilities with Personalized Ranking Models. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 36. 4083–4091.
  * Ma et al. (2018) Xiao Ma, Liqin Zhao, Guan Huang, Zhi Wang, Zelin Hu, Xiaoqiang Zhu, and Kun Gai. 2018.  Entire space multi-task model: An effective approach for estimating post-click conversion rate. In _The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval_. 1137–1140.
  * Nguyen et al. (2023) Dang Minh Nguyen, Chenfei Wang, Yan Shen, and Yifan Zeng. 2023.  LightSAGE: Graph Neural Networks for Large Scale Item Retrieval in Shopee’s Advertisement Recommendation. In _Proceedings of the 17th ACM Conference on Recommender Systems_. 334–337.
  * Ovadia et al. (2019) Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. 2019.  Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift.  _Advances in neural information processing systems_ 32 (2019).
  * Pan et al. (2020) Feiyang Pan, Xiang Ao, Pingzhong Tang, Min Lu, Dapeng Liu, Lei Xiao, and Qing He. 2020.  Field-aware calibration: a simple and empirically strong method for reliable probabilistic predictions. In _Proceedings of The Web Conference 2020_. 729–739.
  * Platt et al. (1999) John Platt et al. 1999\.  Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods.  _Advances in large margin classifiers_ 10, 3 (1999), 61–74.
  * Shan et al. (2016) Ying Shan, T Ryan Hoens, Jian Jiao, Haijing Wang, Dong Yu, and JC Mao. 2016.  Deep crossing: Web-scale modeling without manually crafted combinatorial features. In _Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining_. 255–262.
  * Wang et al. (2018) Jizhe Wang, Pipei Huang, Huan Zhao, Zhibo Zhang, Binqiang Zhao, and Dik Lun Lee. 2018.  Billion-scale commodity embedding for e-commerce recommendation in alibaba. In _Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining_. 839–848.
  * Wang et al. (2021) Zhiqiang Wang, Qingyun She, and Junlin Zhang. 2021.  MaskNet: Introducing feature-wise multiplication to CTR ranking models by instance-guided mask.  _arXiv preprint arXiv:2102.07619_ (2021).
  * Wei et al. (2022) Penghui Wei, Weimin Zhang, Ruijie Hou, Jinquan Liu, Shaoguo Liu, Liang Wang, and Bo Zheng. 2022.  Posterior Probability Matters: Doubly-Adaptive Calibration for Neural Predictions in Online Advertising. In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 2645–2649.
  * Yang et al. (2023) Hao Yang, Ziliang Wang, Weijie Bian, and Yifan Zeng. 2023.  Practice on Effectively Extracting NLP Features for Click-Through Rate Prediction. In _Proceedings of the 32nd ACM International Conference on Information and Knowledge Management_. 4887–4893.
  * Zadrozny and Elkan (2001) Bianca Zadrozny and Charles Elkan. 2001.  Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In _Icml_ , Vol. 1. 609–616.
  * Zadrozny and Elkan (2002) Bianca Zadrozny and Charles Elkan. 2002.  Transforming classifier scores into accurate multiclass probability estimates. In _Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining_. 694–699.
  * Zhai et al. (2016) Shuangfei Zhai, Keng-hao Chang, Ruofei Zhang, and Zhongfei Mark Zhang. 2016.  Deepintent: Learning attentions for online advertising with recurrent neural networks. In _Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining_. 1295–1304.
  * Zhang et al. (2020) Jize Zhang, Bhavya Kailkhura, and T Yong-Jin Han. 2020.  Mix-n-match: Ensemble and compositional methods for uncertainty calibration in deep learning. In _International conference on machine learning_. PMLR, 11117–11128.
  * Zhu et al. (2017) Han Zhu, Junqi Jin, Chang Tan, Fei Pan, Yifan Zeng, Han Li, and Kun Gai. 2017.  Optimized cost per click in taobao display advertising. In _Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining_. 2191–2200.