# Loss Functions and Metrics in Deep Learning

Juan R. Terven
CICATA-Qro
Instituto Politecnico Nacional
Mexico
<EMAIL_ADDRESS>
& Diana M. Cordova-Esparza
Facultad de Informática
Universidad Autónoma de Querétaro
Mexico
<EMAIL_ADDRESS>
& Alfonso Ramirez-Pedraza
Visión Robótica
Centro de Investigaciones en Óptica A.C.
Mexico
<EMAIL_ADDRESS>
& Edgar A. Chavez-Urbiola
CICATA-Qro
Instituto Politecnico Nacional
Mexico
<EMAIL_ADDRESS>

###### Abstract

One of the essential components of deep learning is the choice of the loss
function and performance metrics used to train and evaluate models. This paper
reviews the most prevalent loss functions and performance measurements in deep
learning. We examine the benefits and limits of each technique and illustrate
their application to various deep-learning problems. Our review aims to give a
comprehensive picture of the different loss functions and performance
indicators used in the most common deep learning tasks and help practitioners
choose the best method for their specific task.

_Keywords_ Deep learning $\cdot$ loss functions $\cdot$ performance metrics

###### Contents

  1. 1 Introduction
  2. 2 Loss Functions vs. Performance Metrics
    1. 2.1 Properties of loss functions
  3. 3 Regression
    1. 3.1 Regression Loss Functions
      1. 3.1.1 Mean Squared Error (MSE)
      2. 3.1.2 Mean Absolute Error (MAE)
      3. 3.1.3 Huber Loss
      4. 3.1.4 Log-Cosh Loss
      5. 3.1.5 Quantile Loss
      6. 3.1.6 Poisson Loss
    2. 3.2 Regression Performance Metrics
      1. 3.2.1 Root Mean Squared Error (RMSE)
      2. 3.2.2 Mean Absolute Percentage Error (MAPE)
      3. 3.2.3 Symmetric Mean Absolute Percentage Error (SMAPE)
      4. 3.2.4 Coefficient of Determination $R^{2}$
      5. 3.2.5 Adjusted $R^{2}$
  4. 4 Classification
    1. 4.1 Classification Loss Functions
      1. 4.1.1 Binary Cross-Entropy Loss (BCE)
      2. 4.1.2 Weighted Binary Cross Entropy (WBCE)
      3. 4.1.3 Categorical Cross-entropy Loss (CCE)
      4. 4.1.4 Sparse Categorical Cross-entropy Loss
      5. 4.1.5 Cross-Entropy loss with label smoothing
      6. 4.1.6 Focal loss
      7. 4.1.7 Hinge Loss
    2. 4.2 Classification Performance Metrics
      1. 4.2.1 Confusion Matrix
      2. 4.2.2 Accuracy
      3. 4.2.3 Precision
      4. 4.2.4 Recall, Sensitivity, or True Positive Rate (TPR)
      5. 4.2.5 Precision-Recall Tradeoff
      6. 4.2.6 F1-score
      7. 4.2.7 F2-score
      8. 4.2.8 Specificity
      9. 4.2.9 False Positive Rate (FPR)
      10. 4.2.10 Negative Predictive Value (NPV)
      11. 4.2.11 True Discovery Rate (TDR)
      12. 4.2.12 False Discovery Rate (FDR)
      13. 4.2.13 Precision-Recall Curve
      14. 4.2.14 Area Under the Receiver Operating Characteristic curve (AUC-ROC)
  5. 5 Object Detection
    1. 5.1 Object Detection Loss Functions
      1. 5.1.1 Smooth L1 Loss
      2. 5.1.2 Intersection over Union (IoU) Loss
      3. 5.1.3 YOLO Loss
    2. 5.2 Object Detection Metrics
      1. 5.2.1 Average Precision (AP)
      2. 5.2.2 Average Recall (AR)
  6. 6 Image Segmentation
    1. 6.1 Segmentation Loss Functions
      1. 6.1.1 Cross Entropy Loss for Segmentation
      2. 6.1.2 Intersection Over Union (IoU) loss for segmentation
      3. 6.1.3 Dice Loss
      4. 6.1.4 Tversky loss
      5. 6.1.5 Lovász Loss
    2. 6.2 Segmentation Metrics
      1. 6.2.1 Pixel Accuracy
      2. 6.2.2 Boundary F1 Score (BF)
      3. 6.2.3 Panoptic Quality (PQ)
  7. 7 Face Recognition
    1. 7.1 Face Recognition Loss Functions and Metrics
      1. 7.1.1 Softmax Loss
      2. 7.1.2 A-Softmax Loss
      3. 7.1.3 Center Loss
      4. 7.1.4 CosFace: Large-Margin Cosine Loss
      5. 7.1.5 ArcFace. Additive Angular Margin Loss
      6. 7.1.6 Triplet Loss
      7. 7.1.7 Contrastive Loss
      8. 7.1.8 Circle Loss
      9. 7.1.9 Barlow Twins Loss
      10. 7.1.10 SimSiam Loss
  8. 8 Image Generation
    1. 8.1 Image Generation Loss functions
      1. 8.1.1 Reconstruction Loss
      2. 8.1.2 Kullback-Leibler Divergence Loss
      3. 8.1.3 Adversarial Loss
      4. 8.1.4 Wasserstein loss
      5. 8.1.5 Negative Log-likelihood in Normalizing Flows
      6. 8.1.6 Contrastive Divergence
    2. 8.2 Image Generation Metrics
      1. 8.2.1 Peak Signal-to-Noise Ratio (PSNR)
      2. 8.2.2 Structural Similarity Index (SSIM)
      3. 8.2.3 Inception Score (IS)
      4. 8.2.4 Fréchet Inception Distance (FID)
  9. 9 Discussion
  10. 10 Conclusion
  11. 11 Acknowledgments

## Acronyms

AP

    Average Precision
AUC-ROC

    Area under the Receiver Operating Characteristic curve
BCE

    Binary Cross Entropy
BF

    Boundary F1 Score
CCE

    Categorical Cross Entropy
COCO

    Common Objects in Context
FDR

    False Discovery Rate
FID

    Fréchet Inception Distance
FPR

    False Positive Rate
IoU

    Intersection Over Union
IS

    Inception Score
KL

    Kullback-Leibler
MAE

    Mean Absolute Error
MAPE

    Mean Absolute Percentage Error
MSE

    Mean Square Error
NPV

    Negative Predictive Value
PQ

    Panoptic Quality
PSNR

    Peak Signal-to-Noise Ratio
RMSE

    Root Mean Square Error
SMAPE

    Symmetric Mean Absolute Percentage Error
SSIM

    Structural Similarity Index
TDR

    True Discovery Rate
TPR

    True Positive Rate
VOC

    Visual Object Classes
WBCE

    Weighted Binary Cross Entropy
YOLO

    You Only Look Once

##  1 Introduction

Deep Learning has become a powerful tool for solving complex problems in
various fields, such as image [1, 2, 3, 4, 5] and speech recognition [6, 7, 8,
9, 10], natural language processing [11, 12, 13, 14, 15], and computer vision
[16, 17, 18]. One of the critical components of Deep Learning is the choice of
the loss function and performance metrics used to train and evaluate models.
Loss functions measure how well a model can approximate the desired output,
while performance metrics evaluate the model’s ability to make accurate
predictions on unseen data.

Selecting a suitable loss function and performance metric is crucial for
achieving good performance in deep learning tasks. However, with a wide range
of options available, it can be challenging for practitioners to choose the
most appropriate method for their specific task.

This paper aims to comprehensively review the most commonly used loss
functions and performance metrics in deep learning. We will discuss the
advantages and limitations of each method and provide examples of their
application in various Deep Learning tasks. We begin by discussing regression
and classification’s most commonly used loss functions, including mean squared
error, cross-entropy, and hinge loss. Then, we explain their advantages and
limitations and when they are typically used. For example, mean squared error
is widely used for regression tasks, while cross-entropy is used for
classification tasks. We will also examine more complex tasks such as object
detection, segmentation, face recognition, and image generation.

Along the way, we review the most commonly used performance metrics in each
category, explaining how these metrics are calculated, their advantages and
limitations, and when they are typically used.

##  2 Loss Functions vs. Performance Metrics

A loss function and a performance metric are both used to evaluate the
performance of a deep learning model, but they serve different purposes.

A loss function is used during training to optimize the model’s parameters. It
measures the difference between the predicted and expected outputs of the
model, and the goal of training is to minimize this difference.

On the other hand, a performance metric is used to evaluate the model after
training. It measures how well the model can generalize to new data and make
accurate predictions. Performance metrics also compare different models or
configurations to determine the best-performing one.

The following list describes the common differences between loss functions and
performance metrics:

  * •

Optimization vs. Evaluation: As mentioned previously, loss functions optimize
the model’s parameters during training. In contrast, performance metrics
evaluate the model’s performance after training.

  * •

Model-Dependence: Loss functions depend on the model’s architecture and the
specific task. Performance metrics, however, are less dependent on the model’s
architecture and can be used to compare different models or configurations of
a single model.

  * •

Minimization vs. Maximization: The goal of training a deep learning model is
to minimize the loss function. However, evaluating a model aims to maximize
the performance metric —except for error performance metrics such as Mean
Squared Error.

  * •

Interpretation: Loss functions can be challenging to interpret as their values
are often arbitrary and depend on the specific task and data. On the other
hand, performance metrics are often more interpretable as they are used across
different tasks.

###  2.1 Properties of loss functions

The loss functions have a series of properties that need to be considered when
selected for a specific task:

  1. 1.

Convexity: A loss function is convex if any local minimum is the global
minimum. Convex loss functions are desirable because they can be easily
optimized using gradient-based optimization methods.

  2. 2.

Differentiability: A loss function is differentiable if its derivative with
respect to the model parameters exists and is continuous. Differentiability is
essential because it allows the use of gradient-based optimization methods.

  3. 3.

Robustness: Loss functions should be able to handle outliers and not be
affected by a small number of extreme values.

  4. 4.

Smoothness: Loss function should have a continuous gradient and no sharp
transitions or spikes.

  5. 5.

Sparsity: A sparsity-promoting loss function should encourage the model to
produce sparse output. This is useful when working with high-dimensional data
and when the number of important features is small.

  6. 6.

Multi-modality: A loss function is considered multi-modal if it has multiple
global minima. Multi-modal loss functions can be useful for tasks requiring
the model to learn multiple data representations.

  7. 7.

Monotonicity: A loss function is monotonous if its value decreases as the
predicted output approaches the true output. Monotonicity ensures that the
optimization process is moving toward the correct solution.

  8. 8.

Invariance: A loss function is invariant if it remains unchanged under
particular input or output transformations. Invariance is valuable when
working with data that may be transformed in various ways, such as rotation,
scaling, or translation.

The following sections review the loss functions and performance metrics for
common deep learning tasks. Table 1 summarizes common vision-related tasks
with their common loss functions used and performance metrics.

Table 1:
Loss functions and performance metrics for deep learning tasks. Deep Learning Task |  Loss Functions |  Performance Metrics
---|---|---
Regression |  MSE (3.1.1), MAE (3.1.2)
Huber loss (3.1.3), Log-Cosh (3.1.4)
Quantile loss (3.1.5)
Poisson loss (3.1.6) |  MSE (3.1.1), MAE (3.1.2)
RMSE (3.2.1) , MAPE (3.2.2)
SMAPE (3.2.3)
$R^{2}$ (3.2.4), Adjusted $R^{2}$ (3.2.5)
Binary Classification |  BCE (4.1.1)
WBCE (4.1.2)
Hinge loss (4.1.7)
Focal loss (4.1.6) |  Accuracy (4.2.2), Precision (4.2.3)
Recall (4.2.4), F1-Score (4.2.6)
AUC-ROC (4.2.14)
PR Curve (4.2.13)
Multi-Class Classification |  CCE (4.1.3)
Sarse CCE (4.1.4)
CCE w/label smoothing (4.1.5)
Focal loss (4.1.6)
Hinge loss (4.1.7) |  Accuracy (4.2.2)
Precision (4.2.3)
Recall or TPR (4.2.4)
F1-Score (4.2.6), F2-Score (4.2.7)
PR Curve (4.2.13)
Object Detection |  Smooth L1 (5.1.1)
IoU loss (5.1.2)
Focal loss (4.1.6)
YOLO loss (5.1.3) |  AP (5.2.1)
AR (5.2.2)
Semantic Segmentation |  CCE
IoU loss (5.1.2)
Dice Loss (6.1.3)
Tversky loss (6.1.4)
Lovasz loss (6.1.5) |  IoU (5.1.2),
Pixel Accuracy (6.2.1),
AP (5.2.1)
BF (6.2.2)
Instance Segmentation |  CCE (4.1.3)
IoU loss (6.1.2)
Smooth L1 (5.1.1) |  AP (5.2.1)
Panoptic Segmentation |  CCE (4.1.3)
Dice Loss (6.1.3) |  PQ (6.2.3)
Face Recognition |  A-Softmax (7.1.2)
Center loss (7.1.3)
CosFace (7.1.4)
ArcFace (7.1.5)
Triplet loss (7.1.6)
Contrastive loss (7.1.7)
Circle loss (7.1.8) |  Accuracy (4.2.2)
Precision (4.2.3)
Recall (4.2.4)
F1-Score (4.2.6)
Image Generation |  Adversarial Loss (8.1.3)
Reconstruction loss (8.1.1)
KL Divergence (8.1.2)
Wasserstein Loss (8.1.4)
Constrastive Divergence (8.1.6) |  PSNR (8.2.1)
SSIM (8.2.2)
IS (8.2.3),
FID (8.2.4)
Image-to-Image Translation |  Adversarial Loss (8.1.3),
Cycle-Consistency Loss |  Perceptual Measures, MOS
Style Transfer |  Content Loss,
Style Loss |  Perceptual Measures, MOS
Image Super-resolution |  MSE (3.1.1),
Perceptual Loss |  PSNR (8.2.1), SSIM (8.2.2)
Depth Estimation |  MSE (3.1.1) |  RMSE (3.2.1), Absolute Relative
Depth Error
Pose Estimation |  MSE (3.1.1) |  PCK, OKS
Optical Character Recognition (OCR) |  CCE (4.1.3),
CTC Loss |  Accuracy (4.2.2), Precision (4.2.3),
Recall (4.2.4), F1-Score (4.2.6)

##  3 Regression

Regression is a supervised learning problem in machine learning that aims to
predict a continuous output value based on one or more input features. Common
regression models include linear regression, polynomial regression, and
regression trees.

_Linear regression_ assumes a linear relationship between the independent and
dependent variables. It is represented by the equation

$\hat{y}=\beta_{0}+\beta_{1}x_{1}+\beta_{2}x_{2}+\dots+\beta_{n}x_{n},$ (1)

where $\hat{y}$ is the predicted value, $\beta_{0}$ is the intercept or bias,
$\beta_{1},\beta_{2},...,\beta_{n}$ are the coefficients or weights
corresponding to the input features or independent variables
$x_{1},x_{2},...,x_{n}$ and $n$ is the number of input features.

The goal is to find the bias and the coefficient values that minimize the
difference between the predicted and actual values, usually using a loss
function such as Mean Squared Error (MSE) or Mean Absolute Error (MAE).

In _polynomial regression_ , the relationship between the independent variable
$x$ and the dependent variable $y$ is modeled as an $n^{th}$ degree
polynomial. This is useful for capturing complex, non-linear relationships
between input and output variables. The general form of a polynomial
regression equation is given by

$\hat{y}=\beta_{0}+\beta_{1}x+\beta_{2}x^{2}+\beta_{3}x^{3}+\dots+\beta_{n}x^{n},$
(2)

where $\hat{y}$ is the predicted value, $\beta_{0}$ is the intercept or bias,
$\beta_{1},\beta_{2},...,\beta_{n}$ are the coefficients corresponding to the
powers of $x$ and $n$ is the degree of the polynomial.

Like linear regression, the objective is to find the bias and the coefficients
that minimize the difference between the predicted and the actual values.
However, high-degree polynomials tend to overfit when the model becomes
excessively complex such that it performs well on training data but poorly on
unseen or test data.

_Regression trees_ , on the other hand, are a type of decision tree where the
output is a continuous variable. Unlike linear and polynomial regression
models that establish a single prediction equation, regression trees split the
input space into smaller regions where a simple model is used. The tree is
built during training through a process known as binary recursive
partitioning. The output for a new instance is predicted by traversing the
tree until a leaf node is reached. The value associated with the leaf node is
typically the mean target value of the training samples in this node. Unlike
polynomial regression, this model can capture complex, non-linear
relationships and interactions between features without specifying them
explicitly. However, regression trees can also overfit the training data if
not properly pruned or controlled, leading to poor generalization performance
on new, unseen data. Figure 1 shows a regression tree.

Figure 1: Regression Tree.

Regression is used in various domains, including finance, healthcare, social
sciences, sports, and engineering. Some practical applications include house
price prediction [19], energy consumption forecasting [20], healthcare and
disease prediction [21], stock price forecasting [22], and customer lifetime
value prediction [23].

In the following subsections, we will review the most common lost functions
and performance metrics used for regression.

###  3.1 Regression Loss Functions

Table 2 shows the common loss functions used for regression and their
applications.

The following subsections describe each of these loss functions in more
detail.

Table 2: Loss Functions and their applications in regression tasks. Loss Function |  Applications
---|---
Mean Squared Error (MSE) |  Linear Regression, Ridge Regression, Lasso Regression, Neural Networks, Support Vector Regression, Decision Trees, Random Forests, Gradient Boosting
Mean Absolute Error (MAE) |  Quantile Regression, Robust Regression, L1 Regression, Neural Networks, Decision Trees, Random Forests, Gradient Boosting
Huber Loss |  Robust Linear Regression, Robust Neural Networks, Gradient Boosting, Random Forests
Log-Cosh Loss |  Robust Regression, Neural Networks, Gradient Boosting
Quantile Loss |  Quantile Regression, Distributional Regression, Extreme Value Prediction
Poisson Loss |  Poisson Regression, Count Data Prediction, Generalized Linear Models, Neural Networks, Gradient Boosting

####  3.1.1 Mean Squared Error (MSE)

The Mean Square Error (MSE) measures the average of the squared differences
between the predicted values and the true values [24]. The MSE loss function
can be defined mathematically as

$MSE=\frac{1}{n}\sum_{i=1}^{n}(y_{i}-\hat{y_{i}})^{2},$ (3)

where $n$ is the number of samples, $y_{i}$ is the true value of the $i^{th}$
sample and $\hat{y_{i}}$ is the predicted value of the $i^{th}$ sample.

The MSE loss function has the following properties:

  * •

Non-negative: Since the differences between the predicted and actual values
are squared, MSE is always non-negative. A value of 0 indicates a perfect fit,
while larger values correspond to higher discrepancies between predictions and
actual values.

  * •

Quadratic: MSE is a quadratic function of the prediction errors, which means
it places more emphasis on larger errors than smaller ones. This property
makes it sensitive to outliers and can lead to models that prioritize reducing
large errors over smaller ones.

  * •

Differentiable: MSE is a smooth and continuous function for the model
parameters. This property allows for the efficient computation of gradients,
which is essential for optimization algorithms like gradient descent.

  * •

Convex: MSE is a convex function, which means it has a unique global minimum.
This property simplifies the optimization process, as gradient-based
optimization techniques can converge to the global minimum without getting
trapped in local minima. However, for deep neural networks, the error
landscape is generally non-convex due to the multiple layers of non-linear
activation functions, leading to a complex and highly non-linear optimization
problem.

  * •

Scale-dependent: The value of MSE depends on the scale of the target variable,
making it difficult to compare the performance of models across different
problems or target variable scales. For this purpose, researchers often use
the root mean squared error (RMSE) or mean squared percentage error (MSPE).

The MSE, also called L2 loss, is computationally simple. However, it is not
robust to outliers due to the square of the error term. Thus if the data
includes outliers, it is better to use another loss function, such as Mean
Absolute Error (MAE) which is more robust to outliers, or Huber Loss, which is
a combination of MSE and MAE. The MSE is also used as a performance metric.

####  3.1.2 Mean Absolute Error (MAE)

The Mean Absolute Error (MAE) is another commonly used loss function in
regression problems. It measures the average of the absolute differences
between the predicted values and the true values [25]. The MAE loss can be
defined as

$MAE=\frac{1}{n}\sum_{i=1}^{n}|y_{i}-\hat{y_{i}}|,$ (4)

where $n$ is the number of samples, $y_{i}$ and $\hat{y_{i}}$ are the true and
predicted value of the $i^{th}$ sample.

The MAE loss function has the following properties:

  * •

Non-negative: Like MSE, MAE is always non-negative because it takes the
absolute value of the differences between predicted and actual values. A value
of 0 indicates a perfect fit, while larger values correspond to higher
discrepancies between predictions and actual values.

  * •

Linear: MAE is a linear function of the prediction errors, which treats all
errors equally regardless of their magnitude. This property makes MAE less
sensitive to outliers than MSE, as it does not disproportionately emphasize
large errors.

  * •

Robust: Due to its linear nature and reduced sensitivity to outliers, MAE is
considered a more robust loss function than MSE. This makes it suitable for
applications where the presence of outliers is expected or the distribution of
errors is not symmetric.

  * •

Non-differentiable: Although MAE is continuous, it is not differentiable when
the prediction error is zero due to the absolute value function. This property
can complicate the optimization process for specific algorithms, particularly
those relying on gradient-based techniques. However, subgradient methods[26,
27, 28, 29] can be employed to overcome this issue.

  * •

Convex: MAE is a convex function, which means it has a unique global minimum.
This property simplifies the optimization process, as gradient-based
optimization techniques can converge to the global minimum without getting
trapped in local minima. Like the MSE, the MAE is non-convex for Deep neural
networks due to the multiple layers with non-linear activation functions.

  * •

Scale-dependent: Like MSE, the value of MAE depends on the scale of the target
variable, making it difficult to compare the performance of models across
different problems or target variable scales. To address this issue,
researchers often use scale-invariant metrics such as mean absolute percentage
error (MAPE) or normalized mean absolute error (NMAE) to compare models across
different scales or units.

The MAE, called L1 loss, is often used as an evaluation metric. It is
computationally simple and easy to understand, but it does not have the smooth
and differentiable property of the MSE and is not sensitive to outliers.

####  3.1.3 Huber Loss

The Huber loss combines the properties of both Mean Squared Error (MSE) and
Mean Absolute Error (MAE). Huber loss is designed to be more robust to
outliers than MSE while maintaining smoothness and differentiability [30]. The
Huber loss function is defined as

$L(y,\hat{y})=\begin{cases}\frac{1}{2}(y-\hat{y})^{2}&\text{for
}|y-\hat{y}|\leq\delta\\\
\delta(|y-\hat{y}|-\frac{1}{2}\delta)&\text{otherwise},\end{cases}$ (5)

where $y$ is the true value, $\hat{y}$ is the predicted value, and $\delta$ is
a user-specified threshold value.

When the error is small, the Huber loss function behaves like the MSE loss
function, and when the error is large, the Huber loss function behaves like
the MAE loss function. This property makes the Huber loss function more robust
to outliers than the MSE loss function, as it is less sensitive to large
errors.

The Huber loss function is differentiable, which makes it suitable for use in
gradient-based optimization algorithms such as stochastic gradient descent
(SGD). It is commonly used in linear regression and time series forecasting,
as it can handle outliers and noise in the data. It is also used in robust
optimization problems where the data may contain outliers or noise.

The threshold $\delta$ can be chosen empirically by trying different values
and evaluating the model’s performance. However, common practice is to set
$\delta$ to a small value if the data has a lot of noise and to a large value
if the data has outliers.

####  3.1.4 Log-Cosh Loss

The Log-Cosh loss function is smooth and differentiable. It is commonly used
in regression problems where the data may contain outliers or noise [31]. The
Log-Cosh loss is defined as

$L(y,\hat{y})=\frac{1}{n}\sum_{i=1}^{n}\log(\cosh(y_{i}-\hat{y}_{i})),$ (6)

where $y$ is the true value, $\hat{y}$ is the predicted value and $n$ is the
number of samples.

One of the advantages of the log-cosh loss function is that it is less
sensitive to outliers than the mean squared error (MSE), as it is not affected
by extreme data values. However, it is more sensitive to small errors than the
Huber loss.

####  3.1.5 Quantile Loss

Also known as quantile regression loss, this function is often used for
predicting an interval instead of a single value [32]. If we denote the
quantile as $q$ where $0<q<1$, and the predicted and actual values as
$\hat{y}$ and $y$ respectively, then the quantile loss is given by

$L(y,\hat{y})=q\cdot\max(y-\hat{y},0)+(1-q)\cdot\max(\hat{y}-y,0)$ (7)

$\max(a,b)$ represents the maximum of $a$ and $b$. The expression $y-\hat{y}$
is used when the prediction underestimates, and $\hat{y}-y$ is used when the
prediction overestimates. The loss is scaled by $q$ for underestimations and
$(1-q)$ for overestimations.

Note that when $q=0.5$, the quantile loss is equivalent to the Mean Absolute
Error (MAE), making it a generalization of MAE that allows for asymmetric
penalties for underestimations and overestimations.

Overestimation occurs when a model’s prediction exceeds the actual value.
Underestimation is the opposite of overestimation. It occurs when a model’s
prediction is lower than the actual value.

Practical examples of quantile regression include:

Financial Risk Management: To estimate Value-at-Risk (VaR) and Conditional
Value-at-Risk (CVaR), which are measures of financial risk used in risk
management. These quantile-based measures help to understand the potential for
extreme losses [33].

Supply Chain and Inventory Management: Predicting demand for products can
benefit from quantile loss as it can give a range of potential demand rather
than a single point, which can help manage inventory and reduce stockouts or
overstock situations [34].

Energy Production: To predict power output, having a range of potential
outputs to manage grid stability [35].

Economic Forecasting: Predicting economic indicators can use quantile
regression to give a range of possible values, which can help planning and
policy-making [36].

Weather Forecasting: Can be useful for predicting variables like temperature
or rainfall, where providing a range can be more informative than a single-
point estimate [37, 38].

Real Estate Pricing: Predicting the price of a property within a range can be
more useful than predicting a single price [39].

Healthcare: Quantile regression can predict a range of possible patient
outcomes based on a set of features, which can assist doctors in making more
informed decisions [40].

####  3.1.6 Poisson Loss

Poisson loss is used in regression tasks when the target variable represents
count data and is assumed to follow a Poisson distribution. The Poisson loss
is derived from the negative log-likelihood of the Poisson distribution. It
maximizes the likelihood of observing the count data given the predicted
values [41]. It s defined as

$L(y,\hat{y})=\frac{1}{n}\sum_{i=1}^{n}(\hat{y}_{i}-y_{i}\log(\hat{y}_{i})),$
(8)

where $y_{i}$ represents the actual target value, $\hat{y}_{i}$ is the
predicted value, and $n$ is the number of samples.

When applying the Poisson loss function to model count data, we must ensure
that the predicted values are non-negative since negative counts are not
meaningful in real-world scenarios. To achieve this, it is common to use a
link function that transforms the linear combination of input features to a
non-negative output, which can then be interpreted as the expected count.

A link function is a mapping from the linear predictor to the predicted value.
In the context of Poisson regression, the exponential function is a common
choice for the link function because it guarantees non-negative outputs. The
exponential function has the following form:

$\hat{y}_{i}=\exp(\mathbf{w}^{\top}\mathbf{x}_{i}+b),$ (9)

where $\mathbf{w}$ is a vector of weights, $\mathbf{x}_{i}$ is a vector of
input features for the $i$-th observation, and $b$ is the bias term.

Using the exponential function as a link function, we ensure that the
predicted values $\hat{y}_{i}$ are always non-negative. In this case, the
Poisson loss function can be written as

$L(y,\hat{y})=\frac{1}{n}\sum_{i=1}^{n}\left(\exp(\mathbf{w}^{\top}\mathbf{x}_{i}+b)-y_{i}\log(\exp(\mathbf{w}^{\top}\mathbf{x}_{i}+b))\right)$
(10)

The Poisson distribution is typically used for modeling the number of times an
event occurred in an interval. Here are some examples of applications where
Poisson loss can be useful.

Traffic Modeling: Poisson regression can predict the number of cars that pass
through a toll booth during a given time interval based on factors like the
time of day, day of the week, and weather conditions [42].

Healthcare: Epidemiology can predict the number of disease cases in different
regions based on variables like population density, vaccination rates, and
social behavior patterns [43].

Insurance: In the insurance industry, it can be used to model claim counts for
certain types of insurance policies [44].

Customer Service: Poisson regression can be used to predict the number of
calls that a call center receives during different times of the day, to aid in
staff scheduling [45].

Internet Usage: It can be used to model the number of website visits or clicks
on an ad during a given time interval to help understand user behavior and
optimize ad placement [46].

Manufacturing: It can predict the number of defects or failures in a
manufacturing process, helping in quality control and maintenance planning
[47].

Crime Analysis: Poisson regression can be used to model the number of
occurrences of certain types of crimes in different areas to help in police
resource allocation and crime prevention strategies [48].

###  3.2 Regression Performance Metrics

Table 3 shows the most common metrics used in regression tasks. The following
sections delve into more details on each of these metrics skipping the mean
square error (MSE) and the mean absolute error (MAE) because they are the same
discussed previously as loss functions.

Table 3: Common performance metrics used in regression. Performance Metric |  Applications
---|---
Mean Squared Error (MSE) |  General-purpose regression, model selection,
|  optimization, linear regression, neural networks
Root Mean Squared Error (RMSE) |  General-purpose regression, model selection,
|  optimization, linear regression, neural networks
Mean Absolute Error (MAE) |  General-purpose regression, model selection,
|  optimization, robustness to outliers,
|  time series analysis
R-squared (R²) |  Model evaluation, goodness-of-fit, linear regression,
|  multiple regression
Adjusted R-squared |  Model evaluation, goodness-of-fit, linear regression,
|  multiple regression with many predictors
Mean Squared Logarithmic Error |  Forecasting, model evaluation, skewed
(MSLE) |  target distributions, finance, sales prediction
Mean Absolute Percentage Error |  Forecasting, model evaluation, time series analysis,
(MAPE) |  business analytics, supply chain optimization

####  3.2.1 Root Mean Squared Error (RMSE)

The Root Mean Square Error (RMSE) is the square root of the mean squared error
(MSE) defined as

$RMSE=\sqrt{\frac{1}{n}\sum_{i=1}^{n}(y_{i}-\hat{y}_{i})^{2}},$ (11)

where $y_{i}$ is the true value, $\hat{y}_{i}$ is the predicted value, and $n$
is the number of samples.

The RMSE measures the average deviation of the predictions from the true
values. This metric is easy to interpret because it is in the same units as
the data. However, it is sensitive to outliers. Lower RMSE values indicate
better model performance, representing smaller differences between predicted
and actual values.

####  3.2.2 Mean Absolute Percentage Error (MAPE)

The Mean Absolute Percentage Error (MAPE) measures the average percentage
error of the model’s predictions compared to the true values. It is defined as

$MAPE=\frac{1}{n}\sum_{i=1}^{n}\frac{|y_{i}-\hat{y}_{i}|}{y_{i}}\times 100,$
(12)

where $y_{i}$ is the true value, $\hat{y}_{i}$ is the predicted value, and $n$
is the number of samples.

One of the advantages of using MAPE is that it is easy to interpret, as it is
expressed in percentage terms. It is also scale-independent, which can be used
to compare models across different scales of the target variable. However, it
has two limitations: it can produce undefined results when $y_{i}$ is zero and
is sensitive to outliers.

####  3.2.3 Symmetric Mean Absolute Percentage Error (SMAPE)

The Symmetric Mean Absolute Percentage Error (SMAPE) is a variation of the
Mean Absolute Percentage Error (MAPE) commonly used to evaluate the accuracy
of predictions in time series forecasting [49]. SMAPE is defined as

$SMAPE=\frac{2}{n}\sum_{i=1}^{n}\frac{|y_{i}-\hat{y}_{i}|}{|y_{i}|+|\hat{y}_{i}|}*100,$
(13)

where $y_{i}$ is the true value, $\hat{y}_{i}$ is the predicted value, and $n$
is the number of samples.

One of the advantages of using SMAPE is that it is symmetric, which means that
it gives equal weight to over-predictions and under-predictions. This is
particularly useful when working with time series data, where over-predictions
and under-predictions may have different implications, and SMAPE helps to
ensure that the model is equally penalized for both types of errors, leading
to better overall performance in terms of how well it meets the business needs
or objectives. However, SMAPE has some limitations; for example, it can
produce undefined results when both $y_{i}$ and $\hat{y}_{i}$ are zero and can
be sensitive to outliers.

The implications of over-predictions and under-predictions varied depending on
the application. In the following, we discuss real-world examples.

Inventory Management: Over-predicting demand can lead to excess inventory,
which ties up capital and can result in waste if products expire or become
obsolete. Under-predicting demand can lead to stockouts, lost sales, and
damage to customer relationships [50]. A symmetric error measure like SMAPE
penalizes both cases because over-prediction and under-prediction have costly
implications.

Energy Demand Forecasting: Over-prediction of energy demand can cause
unnecessary production, leading to waste and increased costs. Under-prediction
can lead to insufficient power generation, resulting in blackouts or the need
for expensive on-demand power generation [51].

Financial Markets: In financial markets, over-prediction of a stock price
might lead to unwarranted investments resulting in financial loss, while
under-prediction might result in missed opportunities for gains [52].

Sales Forecasting: Over-prediction of sales could lead to overstaffing,
overproduction, and increased costs, while under-prediction could lead to
understaffing, missed sales opportunities, and decreased customer satisfaction
[53].

Transportation and Logistics: Over-predicting the demand for transportation
might lead to underutilized vehicles or routes, resulting in unnecessary
costs. Under-predicting demand might lead to overcrowding and customer
dissatisfaction [54].

####  3.2.4 Coefficient of Determination $R^{2}$

The Coefficient of Determination ($R^{2}$), measures how well the model can
explain the variation in the target variable [55]. $R^{2}$ is defined as the
proportion of the variance in the target variable that the model explains. It
ranges from 0 to 1, where 0 means that the model does not explain any
variation in the target variable, and one means that the model explains all
the variation in the target variable.

The formula for R-squared is

$R^{2}=1-\frac{\sum_{i=1}^{n}(y_{i}-\hat{y}i)^{2}}{\sum{i=1}^{n}(y_{i}-\bar{y})^{2}},$
(14)

where $y_{i}$ is the true value, $\hat{y}_{i}$ is the predicted value,
$\bar{y}$ is the mean of the true values, and $n$ is the number of samples.

Benefits and Limitations of R-squared

Some of the main benefits of $R^{2}$ are:

  1. 1.

Measures the relationship between the model and the response variable:
R-squared describes the strength of the relationship between the model and the
response variable on a convenient 0 – 1 scale.

  2. 2.

Interpretable: It can be more interpretable than other statistics because it
provides a percentage that can be intuitively understood.

  3. 3.

Helps in model selection: If we have two models, we can compare their
R-squared values as a part of the selection process. The model with the higher
R-squared could indicate a better fit.

The limitations of $R^{2}$ include:

  1. 1.

Misleading with non-linear relationships: $R^{2}$ works as intended in a
simple linear regression model with one explanatory variable but can be
misleading with more complex, nonlinear, or multiple regression models.

  2. 2.

Influenced by the number of predictors: $R^{2}$ always increases as we add
more predictors to a model, even if they are unrelated to the outcome
variable. This can lead to overly complex models that overfit the data. This
is the benefit of the adjusted $R^{2}$, which adjusts the $R^{2}$ value based
on the number of predictors in the model.

  3. 3.

Sensitive to outliers: $R^{2}$ is sensitive to outliers.

  4. 4.

Does not check for biased predictions: $R^{2}$ cannot determine whether the
coefficient estimates and predictions are biased, which is to say, whether the
predictions systematically over or underestimate the actual values.

  5. 5.

Limitation with small sample sizes: When the sample size is small, the $R^{2}$
value might be unreliable. It can be artificially high or low and might not
represent the true strength of the relationship between the variables.

####  3.2.5 Adjusted $R^{2}$

Adjusted $R^{2}$ is a modified version of $R^{2}$ that has been adjusted for
the number of predictors in the model. It increases only if the new term
improves the model more than would be expected by chance. It decreases when a
predictor improves the model by less than expected by chance [56]. The
adjusted R-squared is defined as

$\text{Adjusted }R^{2}=1-\left(\frac{(1-R^{2})(n-1)}{n-k-1}\right),$ (15)

where $n$ is the number of observations, $k$ is the number of predictors. The
adjustment is a penalty for adding unnecessary predictors to the model. This
penalty increases with the increase in the number of predictors. This is
particularly useful in multiple regression, where several predictors are used
simultaneously.

The Adjusted $R^{2}$ is often used for model comparison, as it won’t
necessarily increase with adding more variables to the model, unlike regular
$R^{2}$. It is useful when we need to compare models of different sizes.
Unlike $R^{2}$, its value can be negative, meaning that the model is a poor
fit for the data.

##  4 Classification

Classification is a supervised machine learning task in which a model is
trained to predict the class or category of a given input data point.
Classification aims to learn a mapping from input features to a specific class
or category.

There are different classification tasks, such as binary classification,
multi-class classification, and multi-label classification. Binary
classification is a task where the model is trained to predict one of two
classes, such as "spam" or "not spam," for an email. Multi-class
classification is a task where the model is trained to predict one of the
multiple classes, such as "dog," "cat," and "bird," for an image. Multi-label
classification is a task where the model is trained to predict multiple labels
for a single data point, such as "dog" and "outdoor," for an image of a dog in
the park.

Classification algorithms can be based on techniques such as decision trees,
Naive Bayes, k-nearest neighbors, Support Vector Machines, Random Forest,
Gradient Boosting, Neural Networks, and others.

###  4.1 Classification Loss Functions

Several loss functions can be used for classification tasks, depending on the
specific problem and algorithm. In the following sections, we describe the
most common loss functions used for classification:

####  4.1.1 Binary Cross-Entropy Loss (BCE)

The Binary Cross Entropy (BCE), also known as log loss, is a commonly used
loss function for binary classification problems. It measures the
dissimilarity between the predicted probability of a class and the true class
label [57]. Cross-entropy is a well-known concept in information theory
commonly used to measure the dissimilarity between two probability
distributions. In binary classification, the true class is usually represented
by a one-hot encoded vector, where the true class has a value of 1, and the
other class has a value of 0. The predicted probability is represented by a
vector of predicted probabilities for each class, where the predicted
probability of the true class is denoted by $p(y=1|x)$ and the predicted
probability of the other class is denoted by $p(y=0|x)$.

The loss function is defined as

$L(y,p)=-(y\log(p)+(1-y)\log(1-p))$ (16)

Which intuitively can be split into two parts:

$\begin{cases}-log(p)&\text{if }y=1\\\ -log(1-p)&\text{if }y=0,\end{cases}$
(17)

where $y$ is the true class label (0 or 1) and $p$ is the predicted
probability of the positive class. The loss function is minimized when the
predicted probability $p$ equals the true class label $y$.

The binary cross-entropy loss has several desirable properties, such as being
easy to compute, differentiable, and providing a probabilistic interpretation
of the model’s output. It also provides a smooth optimization surface and is
less sensitive to outliers than other loss functions. However, it is sensitive
to the class imbalance problem, which occurs when the number of samples of one
class is significantly greater than the other. We can use the _Weighted Binary
Cross Entropy_ for these cases.

####  4.1.2 Weighted Binary Cross Entropy (WBCE)

Variation of the standard binary cross-entropy loss function, where the weight
of each sample is considered during the loss calculation. This is useful in
situations where the distribution of the samples is imbalanced [58].

In the standard binary cross-entropy loss, the loss is calculated as the
negative log-likelihood of the true labels given the predicted probabilities.
In the Weighted Binary Cross Entropy (WBCE), a weight is assigned to each
sample, and the loss for each sample is calculated as

$L=-(w_{i}\cdot ylog(p)+w_{i}(1-y)log(1-p)),$ (18)

where $w_{i}$ is the weight assigned to the $i^{th}$ sample, $y$ is the true
label, and $p$ is the predicted probability of the positive class.

By assigning a higher weight to samples from under-represented classes, the
model is encouraged to pay more attention to these samples, and the model’s
overall performance can be improved.

####  4.1.3 Categorical Cross-entropy Loss (CCE)

The Categorical Cross Entropy (CCE), also known as the negative log-likelihood
loss or Multi-class log loss, is a function used for multi-class
classification tasks. It measures the dissimilarity between the predicted
probability distribution and the true distribution [59].

Given the predicted probability distribution, it is defined as the average
negative log-likelihood of the true class. The formula for categorical cross-
entropy loss is expressed as

$L=-\frac{1}{N}\sum_{i=1}^{N}\sum_{j=1}^{C}y_{i,j}log(p_{i,j}),$ (19)

where $N$ is the number of samples, $C$ is the number of classes, $y$ is the
true label, and $p$ is the predicted probability of the true class. The loss
is calculated for each sample and averaged over the entire dataset.

The true label is a one-hot encoded vector in traditional categorical cross-
entropy loss, where the element corresponding to the true class is one, and
all other elements are 0. However, in some cases, it is more convenient to
represent the true class as an integer, where the integer value corresponds to
the index of the true class leading to the sparse categorical cross-entropy
loss discussed next.

####  4.1.4 Sparse Categorical Cross-entropy Loss

Variation of the categorical cross-entropy loss used for multi-class
classification tasks where the classes are encoded as integers rather than
one-hot encoded vectors [59]. Given that the true labels are provided as
integers, we directly select the correct class using the provided label index
instead of summing over all possible classes. Thus the loss for each example
is calculated as

$H(y,\hat{y})=-\log(\hat{y}_{i,y_{i}})$ (20)

And the final sparse categorical cross-entropy loss is the average over all
the samples:

$H(Y,\hat{Y})=-\frac{1}{n}\sum_{i=1}^{n}\log(\hat{y}_{i,y_{i}}),$ (21)

where $y_{i}$ is the true class of the $i$-th sample and $\hat{y}_{i,y_{i}}$
is the predicted probability of the $i$-th sample for the correct class
$y_{i}$.

####  4.1.5 Cross-Entropy loss with label smoothing

In the Cross-Entropy loss with label smoothing, the labels are _smoothed_ by
adding a small value to the true label and subtracting the same value from all
other labels. This helps reduce the model’s overconfidence by encouraging it
to produce more uncertain predictions [60, 61].

The motivation behind this is that when training a model, it is common to
become over-confident in its predictions, particularly when trained on a large
amount of data. This overconfidence can lead to poor performance on unseen
data. Label smoothing helps to mitigate this problem by encouraging the model
to make less confident predictions.

The formula for the Cross-Entropy loss with label smoothing is similar to the
standard categorical cross-entropy loss but with a small epsilon added to the
true label and subtracted from all other labels. The formula is given by

$L(y,\hat{y})=-\sum_{c=1}^{C}\left[(1-\epsilon)y_{c}\log\hat{y}_{c}+\frac{\epsilon}{C}\log\hat{y}_{c}\right],$
(22)

where $y$ is the true label, $\hat{y}$ is the predicted label, $C$ is the
number of classes, and $\epsilon$ is the smoothing value. Typically,
$\epsilon$ is set to a small value, such as 0.1 or 0.2.

Label smoothing does not always improve performance, and it is common to
experiment with different epsilon values to find the best value for a specific
task and dataset.

####  4.1.6 Focal loss

The focal loss introduced by Tsung-Yi Lin et al. [62] is a variation of the
standard cross-entropy loss that addresses the issue of class imbalance, which
occurs when the number of positive samples (objects of interest) is much
smaller than the number of negative samples (background). In such cases, the
model tends to focus on the negative samples and neglect the positive samples,
leading to poor performance. The focal loss addresses this issue by down-
weighting the easy negative samples and up-weighting the hard positive
samples.

The focal loss is defined as

$FL(p_{t})=-\alpha_{t}(1-p_{t})^{\gamma}log(p_{t}),$ (23)

where $p_{t}$ is the predicted probability for the true class, $\alpha_{t}$ is
a weighting factor that controls the importance of each example, and $\gamma$
is a focusing parameter that controls the rate at which easy examples are
down-weighted.

The weighting factor $\alpha_{t}$ is usually set to the inverse class
frequency to balance the loss across all classes. The focusing parameter
$\gamma$ is typically set to a value between 2 and 4 to give more weight to
hard examples.

In the original paper, the authors used a sigmoid activation function for
binary classification and the cross-entropy loss for multi-class
classification. The focal loss is combined with these loss functions to
improve the performance of object detection and semantic segmentation models.

In recent works, focal loss has been used in object detection, semantic [63],
instance segmentation [64], and human pose estimation [65].

####  4.1.7 Hinge Loss

Hinge loss is a popular function used for _maximum-margin_ classification,
commonly used for support vector machines (SVMs) for example in _one-vs-all_
classification where we classify an instance as belonging to one of many
categories and situations where we want to provide a margin of error [66].

The hinge loss function for an individual instance can be represented as

$L(y,f(x))=\max(0,1-y\cdot f(x)),$ (24)

where $y$ is the true label of the instance, which should be -1 or 1 in a
binary classification problem. $f(x)$ is the predicted output for the instance
$x$. The raw margin is $y\cdot f(x)$.

The hinge loss is 0 if the instance is on the correct side of the margin. The
loss is proportional to the distance from the margin for data on the wrong
side of the margin.

###  4.2 Classification Performance Metrics

Table 4 summarizes the common metrics used for classification. The following
sections will delve into each of these metrics.

Table 4: Metrics used in classification task. Common Name | Other Names | Abbr | Definitions | Interpretations
---|---|---|---|---
True Positive | Hit | TP | True Sample | Correctly labeled
|  |  | labeled true | True Sample
True Negative | Rejection | TN | False Sample | Correctly labeled
|  |  | labeled false | False sample
False Positive | False alarm | FP | False sample | Incorrectly labeled
| Type I Error |  | labeled True | False sample
False Negative | Miss, | FN | True sample | Incorrectly label
| Type II Error |  | labeled false | True sample
Recall | True Positive | TPR | TP/(TP+FN) |  $\%$ of True samples
| Rate |  |  | correctly labeled
Specificity | True Negative | SPC, | TN/(TN+FP) |  $\%$ of False samples
| Rate | TNR |  | correctly labeled
Precision | Positive Pre- | PPV | TP/(TP+FP) |  $\%$ of samples labeled
| dictive Value |  |  | True that really are True
Negative Predi- |  | NPV | TN/(TN+FN) |  $\%$ of samples labeled
ctive Value |  |  |  | False that really are False
False Negative |  | FNR | FN/(TP+FN)= |  $\%$ of True samples
Rate |  |  | 1-TPR | incorrectly labeled
False Positive | Fall-out | FPR | FP/(FP+FN)= |  $\%$ of False samples
Rate |  |  | 1-SPC | incorrectly labeled
False Discovery |  | FDR | FP/(TP+FP)= |  $\%$ of samples labeled
Rate |  |  | 1-PPV | True that are really False
True Discovery |  | TDR | FN/(TN+FN)= |  $\%$ of samples labeled
Rate |  |  | 1-NPV | False that are really True
Accuracy |  | ACC | $\frac{(TP+TN)}{(TP+TN+FP+FN)}$ | Percent of samples
|  |  |  | correctly labeled
F1 Score |  | F1 | $\frac{(2*TP)}{((2*TP)+FP+FN)}$ | Approaches 1 as
|  |  |  | errors decline

####  4.2.1 Confusion Matrix

The confusion matrix is used to define a classification algorithm’s
performance. It contains the number of true positives (TP), true negatives
(TN), false positives (FP), and false negatives (FN) that result from the
algorithm. The confusion matrix for a binary classification problem is
represented in a 2x2 table as shown in Table 5.

Table 5: Confusion Matrix | Predicted Positive | Predicted Negative
---|---|---
Actual Positive | True Positive (TP) | False Negative (FN)
Actual Negative | False Positive (FP) | True Negative (TN)

For example, consider a binary classification problem where the algorithm
tries to predict whether an image contains a cat. The confusion matrix for
this problem would look like Table 6:

Table 6: Confusion Matrix | Predicted as Cat | Predicted as Not Cat
---|---|---
Actual Cat | TP | FN
Actual Not Cat | FP | TN

Where:

  * •

TP: the number of images correctly classified as cats.

  * •

TN: the number of images correctly classified as not cats.

  * •

FP: the number of images incorrectly classified as cats.

  * •

FN: the number of images incorrectly classified as not cats.

Using the values in the confusion matrix, we can calculate performance metrics
such as accuracy, precision, recall, and F1-score.

####  4.2.2 Accuracy

Accuracy is a commonly used metric for object classification. It is the ratio
of correctly classified samples to the total number of samples [67].
Mathematically, it can be represented as

$Accuracy=\frac{Number\ of\ Correctly\ Classified\ Samples}{Total\ Number\ of\
Samples}$ (25)

Accuracy can be expressed in terms of the confusion matrix values as

$Accuracy=\frac{TP+TN}{TP+FP+TN+FN}$ (26)

It is a simple and intuitive metric, but it can be misleading when the class
distribution is imbalanced, as it tends to favor the majority class. For
example, let’s assume that we want to predict the presence of cancer in a
cell. If for every 100 samples, only one contains cancer, a useless model that
always predicts "No cancer" will have an accuracy of 99%. Other metrics, such
as precision, recall, or F1-score, are more appropriate in these cases.

####  4.2.3 Precision

Precision measures the accuracy of positive predictions. It is defined as the
number of true positive predictions divided by the number of true positive
predictions plus the number of false positive predictions [68].
Mathematically, it can be represented as

$Precision=\frac{TP}{TP+FP},$ (27)

where $TP$ is the number of true positive predictions, and $FP$ is the number
of false positive predictions.

Precision is useful when the cost of a false positive prediction is high, such
as in medical diagnosis or fraud detection. A high precision means the model
is not generating many false positives, so the predictions are reliable.
However, it is important to note that precision is not the only metric to
consider when evaluating a model’s performance, as high precision can also be
achieved by a model that is not generating many positive predictions at all,
which would result in a low recall.

####  4.2.4 Recall, Sensitivity, or True Positive Rate (TPR)

The recall metric, also known as sensitivity or True Positive Rate (TPR),
measures the proportion of true positive instances (i.e., instances correctly
classified as positive) out of the total number of positive instances [68].
Mathematically, it is represented as

$Recall=\frac{TruePositives}{TruePositives+FalseNegatives}$ (28)

It measures how well the model can identify all the positive instances in the
dataset. A high recall value indicates the model has fewer false negatives,
meaning it can correctly identify the most positive instances. However, a high
recall value does not necessarily mean the model has a high precision, as the
number of false positives can also influence it.

####  4.2.5 Precision-Recall Tradeoff

The precision-recall tradeoff refers to the inverse relationship between
precision and recall. As one metric increases, the other tends to decrease.

Imagine a machine learning model trying to predict whether an email is spam.
If the model is tuned to be very conservative and only marks an email as spam
when confident, it is likely to have high precision (i.e., if it marks an
email as spam, it is very likely to be spam). However, this conservative
approach means it will probably miss many spam emails it is unsure about,
leading to a lower recall.

Conversely, if the model is tuned to be liberal and marks emails as spam more
freely, it will probably identify most spam emails correctly, leading to a
high recall. However, this approach will also incorrectly mark many non-spam
emails as spam, leading to a lower precision.

This tug-of-war between precision and recall is the crux of the tradeoff. An
optimal balance between the two must be found depending on the use case. For
instance, in a medical context, a high recall might be prioritized to ensure
that all possible disease cases are identified, even at the expense of some
false positives. On the other hand, a spam detection system might aim for high
precision to avoid annoying users with wrongly classified emails, accepting
that some spam messages might slip through.

The precision-recall tradeoff is a crucial consideration when tuning machine
learning models. Maximizing both metrics is only sometimes possible; thus, a
balance must be struck based on the requirements and constraints of the
specific application.

####  4.2.6 F1-score

The F1 score combines precision and recall to provide a single value
representing a classification model’s overall performance [68]. It is defined
as the harmonic mean of precision and recall computed as

$F1=2*\frac{precision\cdot recall}{precision+recall}$ (29)

The F1 score considers both the model’s ability to correctly identify positive
examples (precision) and the ability of the model to identify all positive
examples in the dataset (recall). A higher F1 score indicates that the model
has a better balance of precision and recall, whereas a low F1 score indicates
that the model may have a high precision or recall but not both.

It is particularly useful when the class distribution is imbalanced, or we
want to give equal weight to precision and recall.

####  4.2.7 F2-score

The F2 score is a variation of the F1 score, with more weight given to the
recall metric. The F2 score is the harmonic mean of precision and recall, with
a weighting factor of 2 for recall [68]. The formula for the F2 score is

$F2=(1+2^{2})\frac{Precision*Recall}{2^{2}*Precision+Recall}$ (30)

Like the F1 score, the F2 score ranges from 0 to 1, with a higher score
indicating better performance. However, the F2 score places a greater emphasis
on recall, making it useful when it is important to minimize false negatives.
For example, a false negative could mean a patient is not diagnosed with a
serious disease in medical diagnosis, so the F2 score is often used in such
scenarios [69].

####  4.2.8 Specificity

Specificity, also known as the true negative rate (TNR), is a metric that
measures the proportion of actual negatives that are correctly identified as
negatives by a classification model. It is defined as the number of true
negatives (TN) divided by the number of true negatives plus the number of
false positives (FP) [70]. The formula for specificity is

$Specificity=\frac{TN}{TN+FP}$ (31)

This metric is particularly useful in medical diagnostic testing, where it is
important to minimize the number of false positives to avoid unnecessary
treatments or interventions. High specificity indicates that the model is good
at identifying negatives and has a low rate of false positives.

It is often used with the Recall or TPR to evaluate the overall performance of
a classification model.

####  4.2.9 False Positive Rate (FPR)

The False Positive Rate (FPR) is used to evaluate the proportion of false
positives (i.e., instances that are incorrectly classified as positive) to the
total number of negatives (i.e., instances that are correctly classified as
negative). It is also known as the Type I Error rate, which complements the
Specificity metric.

Formally, the FPR is calculated as

$FPR=\frac{FP}{FP+TN}$ (32)

FPR directly relates to the threshold classifying instances as positive or
negative. A lower threshold will increase the number of false positives and
thus increase the FPR, while a higher threshold will decrease the number of
false positives and decrease the FPR.

In practice, the FPR is often plotted on the x-axis of a Receiver Operating
Characteristic (ROC) curve to visualize the trade-off between the TPR and FPR
for different classification thresholds. See section 4.2.14 for more details.

####  4.2.10 Negative Predictive Value (NPV)

The Negative Predictive Value (NPV) measures the proportion of negative cases
that are correctly identified as such [70]. It is calculated as

$NPV=\frac{TN}{TN+FN}$ (33)

The NPV is useful when the cost of a false negative (i.e., an actual negative
case being classified as positive) is high. For example, a false negative
result in medical diagnostics can delay treatment or even death. In such
cases, a high NPV is desired.

The NPV is not affected by the prevalence of the condition in the population,
whereas other metrics, such as sensitivity and specificity, are. This makes
the NPV a useful metric for evaluating the performance of a classifier when
the class distribution is imbalanced.

The NPV can be interpreted as the complement of the false positive rate (FPR)

$NPV=1-FPR$ (34)

####  4.2.11 True Discovery Rate (TDR)

True Discovery Rate (TDR) evaluates the proportion of true positive
predictions a model makes among all the positive predictions. It is also known
as the Positive Predictive Value (PPV) or precision of the positive class
[70]. TDR is calculated as

$TDR=\frac{TP}{TP+FP}$ (35)

TDR is a useful metric for evaluating the performance of a model in situations
where the number of false positive predictions is high, and the number of true
positive predictions is low. It is particularly useful in high-dimensional
datasets where the number of features is large and the number of positive
observations is low. TDR can provide a more accurate picture of the model’s
performance than accuracy or recall in such cases.

There may be a trade-off between TDR and recall in some cases: TDR may be low
when the recall is high, and vice versa. Therefore, it’s important to consider
both TDR and recall when evaluating the performance of a model.

####  4.2.12 False Discovery Rate (FDR)

The False Discovery Rate (FDR) measures the proportion of false positives
among all positive predictions made by a classifier [70]. It is defined as

$FDR=\frac{FP}{TP+FP}$ (36)

The FDR can be an alternative to the False Positive Rate (FPR) when the cost
of false positives and true negatives differs. It is particularly useful in
cases where the number of false positives is more critical than the number of
false negatives, such as in medical testing or fraud detection. A lower FDR
value indicates that the classifier makes fewer false positive predictions.

####  4.2.13 Precision-Recall Curve

The precision-recall (PR) curve is a graphical representation of the trade-off
between precision and recall for different threshold values of a classifier.
Precision is the proportion of true positive predictions out of all positive
predictions, while recall is the proportion of true positive predictions out
of all actual positive instances. The precision-recall curve plots precision
on the y-axis and recall on the x-axis for different threshold values of the
classifier [68].

Computing the Precision-Recall Curve

  1. 1.

Start with a binary classifier that can predict a binary outcome and estimate
the probability of the positive class. These probabilities are also known as
_scores_.

  2. 2.

For every possible threshold (from 0 to 1) on these scores, calculate the
Precision (see Section 4.2.3) and the Recall (see Section 4.2.4).

  3. 3.

Plot a curve with Recall on the X-axis and Precision on the Y-axis. Figure
2(a) shows an example of Precision-Recall curves for three models.

Interpretation of the Precision-Recall Curve

Figure 2(a) shows the precision/recall curves for three models trained on the
same data. The dashed line shows the ideal performance. Each model reports its
Average Precision metric (see Section 5.2.1 for more details on Average
Precision). In the following, we explain how to interpret PR curves.

_The closer the curve is to the top-right corner, the better the model’s
performance._ Ideal performance is indicated by a point at (1,1), which
signifies perfect precision (no false positives) and recall (no false
negatives). If the curve is closer to the top-right corner of the plot, it
indicates that the model achieves a good balance of precision and recall for
most threshold settings.

_The area under the curve (AUC-PR)_ provides a single-number summary of the
information in the curve. The maximum possible AUC is 1, which corresponds to
a perfect classifier. A random classifier will have an AUC of 0.5. A model
with a higher AUC is generally considered better.

_Steepness of the curve_. Ideally, we want the recall to increase quickly as
precision decreases slightly, resulting in a steep curve. This steepness
reflects a good balance between precision and recall. If the curve is less
steep, we are losing a lot of precision for small increases in recall.

_Comparison of different models_. We can compare the PR curves of different
models to understand their performance. If the PR curve of one model is
entirely above that of another, it indicates superior performance across all
thresholds.

####  4.2.14 Area Under the Receiver Operating Characteristic curve (AUC-ROC)

The Area Under the Receiver Operating Characteristic Curve (AUC-ROC) is a
commonly used performance metric for evaluating the performance of binary
classification models [68]. It measures the ability of the model to
distinguish between positive and negative classes by plotting the true
positive rate (TPR) against the false positive rate (FPR) at various threshold
settings. The AUC-ROC is a value between 0 and 1, with 1 indicating a perfect
classifier and a value of 0.5 indicating a classifier that performs no better
than random guessing.

The Area under the Receiver Operating Characteristic curve (AUC-ROC) offers a
single-value summary of the model’s performance across all possible threshold
values. This measure is particularly valuable when comparing the performance
of different models, as its assessment is independent of threshold choice.

However, in cases where the positive and negative class distributions are
significantly imbalanced, the AUC-ROC, while still applicable, may not provide
the most accurate performance representation. With a heavy imbalance, the ROC
curve can appear overly optimistic, as a low false positive rate can still
mean a large number of false positives if the total count of actual negatives
is high, resulting in a misleadingly high AUC-ROC value.

In such imbalanced scenarios, the Precision-Recall (PR) curve and its
corresponding area under the curve (AUC-PR) can often provide a more nuanced
and accurate performance assessment. As PR curves focus more on detecting
positive instances, often the minority class in an imbalanced dataset, they
can deliver a more insightful evaluation of a model’s ability to detect
positive instances, providing a more relevant representation of the model’s
performance.

Computing the ROC Curve

  1. 1.

Start with a binary classifier that can predict a binary outcome and estimate
the probability of the positive class. These probabilities are also known as
_scores_.

  2. 2.

For every possible threshold (from 0 to 1) on these scores, calculate the TPR
(see Section 4.2.4) and the FPR (see Section 4.2.9).

  3. 3.

Plot a curve with FPR on the X-axis and TPR on the Y-axis. Figure 2(b) shows
an example of ROC curves for three models.

Interpretation of the ROC Curve

Figure 2(b) shows the ROC curves for three models trained on the same data.
The dashed line shows random performance. Each model reports its Area under
the curve (AUC) in the legend. In the following, we explain how to interpret
ROC curves.

_TPR and FPR on each axis:_ The True Positive Rate (TPR) is used for the
vertical axis. It measures the proportion of actual positives that are
correctly identified as such. The False Positive Rate (FPR), also known as the
fall-out or Probability of False Alarm, measures the proportion of actual
negatives that are incorrectly identified as positives. The ROC curve plots
the TPR vs. FPR at different classification thresholds. Lowering the
classification threshold classifies more items as positive, thus increasing
both False Positives and True Positives.

_Area Under the ROC Curve (AUC-ROC):_ AUC provides an aggregate performance
measure across all possible classification thresholds. AUC-ROC of a model
equals the probability that the model will rank a randomly chosen positive
instance higher than a randomly chosen negative instance. Hence, the higher
the AUC-ROC score, the better the model (from 0 to 1).

_Diagonal line equals random guess:_ The diagonal line in the ROC curve plot
has an AUC of 0.5 and represents a model with no discriminatory ability, i.e.,
one that predicts positives and negatives at random.

_Towards the top-left corner:_ The more the curve sits in the top-left corner,
the better the classifier, as it means the True Positive Rate is high and the
False Positive Rate is low.

_Compare Models:_ ROC curves are useful for comparing different models. The
model with a higher AUC and its curve towards the top-left corner is generally
considered better.

(a) Precision/Recall curve

(b) ROC Curve

Figure 2: Precision/Recall and ROC Curves. (a) Shows the precision/recall
curves for three models trained on the same data. The dashed line shows the
ideal performance. Each model reports its Average Precision metric. (b) Shows
the ROC curves for three models trained on the same data. The dashed line
shows random performance. Each model reports its Area under the curve (AUC) in
the legend

##  5 Object Detection

Object detection in deep learning is a computer vision technique that involves
localizing and recognizing objects in images or videos. It is common in
various applications such as autonomous driving [71, 72, 73, 74], surveillance
[75, 76, 77], human-computer interaction [78, 79, 80, 81], and robotics [82,
83, 84, 85]. Object detection involves identifying the presence of an object,
determining its location in an image, and recognizing the object’s class.

###  5.1 Object Detection Loss Functions

Since object detection involves localization (regression) and recognition
(classification), object detection systems use a combination of multiple loss
functions. Among these loss functions, we find:

  * •

Multi-Class Log Loss (also known as Cross-Entropy Loss): It is used for the
multi-class classification part of the object detector. Penalizes the
difference between the predicted class probabilities and the ground truth
class labels.

  * •

Smooth L1 Loss: It is used for the regression part of the object detector. It
aims to reduce the mean absolute error between the predicted and ground truth
bounding box coordinates.

  * •

IoU Loss: It calculates the Intersection Over Union (IoU) between the
predicted bounding box and the ground truth bounding box and penalizes the
difference between the predicted IoU and the ground truth IoU.

  * •

Focal Loss: It is used to overcome the problem of class imbalance and focuses
on the misclassified samples. It penalizes the samples that are easily
classified with high confidence and gives more weight to the samples that are
difficult to classify.

  * •

YOLO Loss: It is used for the You Only Look Once (YOLO) object detection
family of algorithms and combines the prediction of bounding box coordinates,
objectness scores, and class probabilities.

In the following sections, we will delve into the loss functions that we have
not touched on before or are defined differently.

####  5.1.1 Smooth L1 Loss

The smooth L1 loss, also known as the smooth mean absolute error (SMAE) loss,
is a commonly used loss function in object detection tasks; it was introduced
in Fast R-CNN [86].

The smooth L1 loss is a modification of the mean absolute error (MAE) loss
that aims to balance between being too sensitive to outliers and insensitive
to small errors. The formula for the smooth L1 loss is given by

$L=\begin{cases}0.5*(x_{i}-y_{i})^{2}&\text{if }|x_{i}-y_{i}|<1\\\
|x_{i}-y_{i}|-0.5&\text{otherwise},\end{cases}$ (37)

where $x_{i}$ and $y_{i}$ are the predicted and ground truth values,
respectively.

It is commonly used in the region proposal network (RPN) part of the two-stage
object detectors [86, 87] to regulate the regression of the bounding box
coordinates outperforming the mean square error (MSE) loss in terms of both
accuracy and efficiency.

####  5.1.2 Intersection over Union (IoU) Loss

Intersection Over Union (IoU) is a metric used in object detection that
measures the overlap between two bounding boxes. Figure 3 depicts the IoU
metric used in object detection. The IoU between two bounding boxes is
calculated as

$IoU=\frac{area\ of\ intersection}{area\ of\ union}$ (38)

The IoU loss function is defined as

$L=1-IoU$ (39)

This function encourages the predicted bounding boxes to overlap highly with
the ground truth bounding boxes. A high IoU value indicates that the predicted
bounding box is close to the ground truth, while a low IoU value indicates
that the predicted bounding box is far from the ground truth.

The IoU loss function is commonly used for one-stage detectors [88, 89] as
part of a multi-task loss function that includes a classification loss and a
localization loss.

Figure 3: Intersection Over Union (IoU). a) The IoU is calculated by dividing
the intersection of the two boxes by the union of the boxes; b) examples of
three different IoU values for different box locations.

####  5.1.3 YOLO Loss

The You Only Look Once (YOLO) loss function is used in the YOLO object
detection architecture. It was introduced by Redmon et al. in [90]. The YOLO
loss function is a multi-part loss that consists of three components:

  1. 1.

Localization loss: This component penalizes the network for misprediction of
the object’s coordinates in the image. It is calculated as the mean squared
error between the predicted and ground-truth bounding box coordinates.

  2. 2.

Confidence loss: This component penalizes the network for not detecting an
object even when one is present. It is a binary cross-entropy loss calculated
between the predicted objectiveness score and the ground-truth label.

  3. 3.

Classification loss: This component penalizes the network for misclassifying
the object. It is a multi-class cross-entropy loss calculated between the
predicted class scores and the ground-truth label.

The total YOLO loss is the weighted sum of these three components. Figure 4
explains the full YOLO loss function.

Figure 4: The YOLO Loss function comprises three parts: a localization loss, a
confidence loss, and a classification loss.

###  5.2 Object Detection Metrics

To compute the metrics in object detection, we also compute the True
Positives, False Positives, True Negatives, and False Negatives. The
definitions of these metrics are based on the IoU score as follows:

True Positives in object detection: The match between the predicted location
of an object and its actual location is measured using an Intersection Over
Union (IoU) score. The IoU score ranges from 0 to 1, with a score of 1
indicating a perfect match between the predicted and ground-truth locations.
Since a perfect match is hard to achieve, we define a threshold value to
determine whether a prediction is a true positive. Common values for the
threshold are 0.25, 0.5, and 0.75. These thresholds are not fixed and can be
adjusted based on the application’s requirements. If the IoU score between the
predicted and ground-truth boxes is greater than or equal to the defined
threshold, the prediction is considered a true positive.

False Positive in object detection: Occurs when the model predicts the
presence of an object, but the object is not present in the image. This
affects the precision metric.

False Negative in object detection: Occurs when the model fails to detect an
object that is present in an image. This affects the recall metric.

True Negative in object detection: Refers to a case where the object detector
correctly determines that an object is not present in an image.

Common IoU thresholds for object detection:

  * •

0.5: A threshold of 0.5 is commonly used as a balanced threshold for object
detection. A predicted bounding box is considered a true positive if its IoU
with the ground truth bounding box is greater than or equal to 0.5.

  * •

0.75: A threshold of 0.75 is used for applications that require higher
precision, such as autonomous driving, where false positive detections can
lead to critical consequences.

  * •

0.25: A threshold of 0.25 is used for applications that require higher recall,
such as medical image analysis, where missing detections can lead to an
incorrect diagnosis.

The common object detection metrics are:

  * •

Average Precision (AP).

  * •

Intersection over union (IoU). See details in section 5.1.2.

  * •

Precision-Recall Curve. See details in section 4.2.13.

####  5.2.1 Average Precision (AP)

Object detection models must identify and localize multiple object categories
in an image. The AP metric addresses this by calculating each category’s
Average Precision (AP) separately and then taking the mean of these APs across
all categories (that is why it is also called mean average precision or mAP).
This approach ensures that the model’s performance is evaluated for each
category individually, providing a more comprehensive assessment of the
model’s overall performance.

To accurately localize objects in images, AP incorporates the Intersection
over Union (IoU) to assess the quality of the predicted bounding boxes. As
described previously, IoU is the ratio of the intersection area to the union
area of the predicted bounding box and the ground truth bounding box (see
Figure 3). It measures the overlap between the ground truth and predicted
bounding boxes. The COCO benchmark considers multiple IoU thresholds to
evaluate the model’s performance at different levels of localization accuracy.

The two most common object detection datasets are The Pascal Visual Object
Classes (VOC) [91] and Microsoft Common Objects in Context (COCO) [92]. The AP
is computed differently in each of these. In the following, we describe how it
is computed on each dataset.

#### VOC Dataset

This dataset includes 20 object categories. To compute the AP in VOC, we
follow the next steps:

  1. 1.

_Compute Intersection over Union (IoU)_ : For each detected object, compute
the IoU with each ground truth object in the same image (refer to section
5.1.2 for more details).

  2. 2.

_Match Detections and Ground Truths:_ For each detected object, assign it to
the ground truth object with the highest IoU, if the IoU is above the
threshold.

  3. 3.

_Compute Precision and Recall:_ For each category, calculate the precision-
recall curve by varying the confidence threshold of the model’s predictions
(refer to section 4.2.13 for more details). This results in a set of
precision-recall pairs.

  4. 4.

_Sort and interpolate with 11-points:_ Sort the precision-recall pairs by
recall in ascending order. Then, for each recall level $r$ in the set
$\\{0,0.1,0.2,...,1.0\\}$, find the highest precision $p(r)$ for which the
recall is at least $r$. This is known as interpolated precision. This process
results in a precision-recall curve that is piecewise constant and
monotonically decreasing.

  5. 5.

_Compute Area Under Curve (AUC)_ : The Average Precision is then defined as
the area under this interpolated precision-recall curve. Since the curve is
piecewise constant, this can be computed as a simple sum: $AP=sum(p(r)/N)$,
where the sum is over the $N$ recall levels, and $p(r)$ is the interpolated
precision at recall level $r$.

#### Microsoft COCO Dataset

This dataset includes 80 object categories and uses a more complex method for
calculating AP. Instead of using an 11-point interpolation, it uses a
101-point interpolation, i.e., it computes the precision for 101 recall
thresholds from 0 to 1 in increments of 0.01. Also, the AP is obtained by
averaging over multiple IoU values instead of just one, except for a common AP
metric called $AP_{50}$, which is the AP for a single IoU threshold of 0.5.
Table 7 shows all the metrics used to evaluate models in the COCO dataset. The
steps for computing AP in COCO are the following:

  1. 1.

_Compute the Intersection over Union (IoU):_ For each detected object, compute
the IoU with each ground truth object in the same image.

  2. 2.

_Match Detections and Ground Truths:_ For each detected object, assign it to
the ground truth object with the highest IoU, if this IoU is above the
threshold.

  3. 3.

_Compute Precision and Recall:_ For each possible decision threshold
(confidence score of the detection), compute the precision and recall of the
model. This results in a set of precision-recall pairs.

  4. 4.

_Interpolate Precision:_ For each recall level $r$ in the set
$\\{0,0.01,0.02,...,1.00\\}$ (for the 101-point interpolation used in COCO),
find the maximum precision $p(r)$ for which the recall is at least $r$. This
is known as interpolated precision.

  5. 5.

_Compute Area Under Curve (AUC):_ The Average Precision is then defined as the
area under this interpolated precision-recall curve. Since the curve is a
piecewise constant, this can be computed as a simple sum: $AP=sum(p(r))/101$,
where the sum is over the 101 recall levels, and $p(r)$ is the interpolated
precision at recall level $r$.

  6. 6.

_Average over IoU Thresholds:_ Repeat steps 2-5 for different IoU thresholds
(e.g., 0.5, 0.55, 0.6, …, 0.95) and average the AP values.

  7. 7.

_Average over Categories:_ Repeat steps 2-6 for each category and average the
AP values. This is to prevent categories with more instances from dominating
the evaluation.

  8. 8.

_Average over Object Sizes:_ Finally, you can compute AP for different object
sizes (small, medium, large) to see how well the model performs on different
sizes of objects.

####  5.2.2 Average Recall (AR)

Average Recall (AR) is used to evaluate the performance of object detection
models. Unlike Precision or Recall, defined at a particular decision
threshold, Average Recall is computed by averaging recall values at different
levels of Intersection over Union (IoU) thresholds and, if needed, at
different maximum numbers of detections per image. This metric is commonly
used to report COCO data results [92, 93].

The general steps to compute AR are the following:

  1. 1.

_Compute the Intersection over Union (IoU):_ For each detected object, compute
the IoU with each ground truth object in the same image.

  2. 2.

_Match Detections and Ground Truths:_ For each ground truth object, find the
detected object with the highest IoU. If this IoU is above a certain
threshold, the detection is considered a true positive, and the ground truth
is _matched_. Each ground truth can only be matched once.

  3. 3.

_Compute Recall:_ For each image, recall is the number of matched ground
truths divided by the total number of ground truths.

  4. 4.

_Average over IoU Thresholds:_ Repeat steps 2 and 3 for different IoU
thresholds (e.g., from 0.5 to 0.95 with step size 0.05), and average the
recall values.

  5. 5.

_Average over Max Detections:_ Repeat steps 2-4 for different maximum numbers
of detections per image (e.g., 1, 10, 100), and average the recall values.
This step is necessary because allowing more detections per image can
potentially increase recall but at the cost of potentially more false
positives.

  6. 6.

_Average over Images:_ Finally, compute the average recall over all the images
in the dataset.

For COCO, the Average Recall measure can also be computed separately for
different object sizes (small, medium, and large) to evaluate how well the
model works for objects of different sizes.

Table 7: COCO Evaluation Metrics. Average Precision (AP) |
---|---
AP |  $\%$ AP at IoU=.50:.05:.95 (primary challenge metric)
$AP^{IoU=.50}$ |  $\%$ AP at IoU=.50 (PASCAL VOC metric)
$AP^{IoU=.75}$ |  $\%$ AP at IoU=.75 (strict metric)
AP Across Scales: |
$AP^{small}$ |  $\%$ AP for small objects: area $<32^{2}$
$AP^{medium}$ |  $\%$ AP for medium objects: $32^{2}<$ area $<96^{2}$
$AP^{large}$ |  $\%$ AP for large objects: area $>96^{2}$
Average Recall (AR): |
$AR^{max=1}$ |  $\%$ AR given 1 detection per image
$AR^{max=10}$ |  $\%$ AR given 10 detections per image
$AR^{IoU=100}$ |  $\%$ AR given 100 detection per image
AR Across Scales: |
$AR^{small}$ |  $\%$ AR for small objects: area $<32^{2}$
$AR^{medium}$ |  $\%$ AR for medium objects: $32^{2}<$ area $<96^{2}$
$AR^{large}$ |  $\%$ AR for large objects: area $>96^{2}$

##  6 Image Segmentation

Image segmentation aims to assign a label or category to each pixel in the
image, effectively segmenting the objects at a pixel level. Segmentation is
usually performed using deep learning models trained to classify each pixel in
the image based on its features and context. Segmentation methods are mainly
classified into three categories: semantic segmentation [94, 95, 96, 97, 98,
99, 100, 101], instance segmentation [102, 103, 104, 105, 106], and panoptic
segmentation [107, 108, 109, 110, 111].

_Semantic Segmentation_ studies the uncountable stuff in an image. It analyzes
each image pixel and assigns a unique class label based on the texture it
represents. In a street image, the semantic segmentation’s output will assign
the same label to all the cars and the same image to all the pedestrians; it
cannot differentiate the objects separately.

_Instance Segmentation_ deals with countable things. It can detect each object
or instance of a class present in an image and assigns it to a different mask
or bounding box with a unique identifier.

_Panoptic Segmentation_ presents a unified segmentation approach where each
pixel in a scene is assigned a semantic label (due to semantic segmentation)
and a unique instance identifier (due to instance segmentation).

Segmentation applications include scene understanding [112, 113, 114, 115,
116], medical image analysis [117, 118, 119, 120, 121, 122, 123], robotic
perception [124, 125, 126, 127, 128], autonomous vehicles [129, 130, 131, 132,
133, 134], video surveillance [135, 136, 137, 138, 139], and augmented reality
[140, 141, 142, 143, 144].

###  6.1 Segmentation Loss Functions

Common loss functions include cross-entropy loss, Intersection over union
(IoU) loss, Focal loss, Dice loss, Tversky loss, and Lovász Loss. The
following sections will describe these loss functions and their applications.

####  6.1.1 Cross Entropy Loss for Segmentation

The cross-entropy loss for segmentation measures the dissimilarity between the
predicted and ground truth segmentation maps. The cross-entropy loss is
calculated by comparing the predicted and ground truth segmentation maps
pixel-by-pixel [95]. It is defined as the negative log-likelihood of the
ground truth segmentation map given the predicted segmentation map. The cross-
entropy loss is calculated using the following formula:

$-\frac{1}{N}\sum\limits_{i=1}^{N}\sum\limits_{c=1}^{C}y_{i,c}log(p_{i,c}),$
(40)

where $N$ is the total number of pixels in the image, $C$ is the number of
classes, $y$ is the ground truth segmentation map, and $p$ is the predicted
segmentation map. The values of $y$ and $p$ should be between 0 and 1 and sum
up to 1. The lower the cross-entropy loss, the better the prediction.

####  6.1.2 Intersection Over Union (IoU) loss for segmentation

The Intersection Over Union (IoU) loss is a commonly used loss function and
evaluation metric for semantic segmentation tasks. The goal is to predict a
per-pixel segmentation mask for a given image. The IoU loss is also known as
the Jaccard loss or Jaccard Index (JI), and is defined as the ratio of the
intersection of the predicted and ground-truth masks to the union of the
predicted and ground-truth masks. The IoU loss is calculated per-pixel basis,
and the final loss is the average IoU across all pixels in the image.

The IoU loss can be defined mathematically as:

$IoU=\frac{1}{n}\sum_{i=1}^{n}\frac{y_{i}\cap\hat{y}_{i}}{y_{i}\cup\hat{y}_{i}},$
(41)

where $y_{i}$ is the ground-truth mask for pixel $i$, $\hat{y}_{i}$ is the
predicted mask, $y_{i}\cap\hat{y}_{i}$ is the intersection of the ground-truth
and predicted masks, and $y_{i}\cup\hat{y}_{i}$ is the union of the ground-
truth and predicted masks.

IoU is commonly used in various semantic segmentation works as a loss function
[145, 146] and as an evaluation metric[94, 147].

####  6.1.3 Dice Loss

The Dice loss, also known as the Dice similarity coefficient [148], is used to
evaluate the similarity between the predicted segmentation mask and the ground
truth mask. The loss function is defined as

$L=1-\frac{2\cdot intersection(pred,gt)}{|pred|+|gt|},$ (42)

where $pred$ is the predicted segmentation mask, $gt$ is the ground truth
segmentation mask, $intersection(pred,gt)$ is the number of pixels that are in
the intersection of the predicted and ground truth masks, and $|pred|$ and
$|gt|$ are the total number of pixels in the predicted and ground truth masks,
respectively.

Dice loss is widely used in medical imaging, where the goal is to segment
structures in images with high precision.

####  6.1.4 Tversky loss

The Tversky loss [149] is a variation of the Dice loss, commonly used in image
segmentation tasks. It is defined as

$Tversky(A,B)=\frac{|A\cap B|}{|A\cap B|+\alpha|A\backslash
B|+\beta|B\backslash A|},$ (43)

where $A$ and $B$ are the predicted and ground truth segmentation masks,
respectively, and $\alpha$ and $\beta$ are user-defined hyperparameters that
control the weighting of false positives and false negatives.

This loss function is similar to the Dice loss, but it allows the assignment
of different weights to false positives and false negatives, which can be
useful in certain scenarios where the imbalance between the two types of
errors is significant.

####  6.1.5 Lovász Loss

The main idea behind the Lovász Loss[150] is to optimize the IoU by optimizing
the Jaccard index or IoU between the predicted segmentation and the ground-
truth segmentation. This loss function is particularly useful in image
segmentation tasks where the Intersection-over-Union (IoU) score is highly
important. The Lovász Loss is defined as the sum of the predicted segmentation
mask and the ground-truth segmentation mask, weighted by the Jaccard index as
follows:

$L=-\frac{1}{N}\sum_{i=1}^{N}Jaccard(p_{i},y_{i})\log(p_{i}),$ (44)

where $N$ is the number of pixels in the image, $p$ is the predicted
segmentation mask, and $y$ is the ground-truth segmentation mask.

The Lovász loss provides a differentiable surrogate for the non-differentiable
IoU metric, allowing it to be optimized directly.

###  6.2 Segmentation Metrics

The common metrics for evaluating segmentation are the Mean Intersection over
union (mIoU), pixel accuracy, Average Precision (AP) (refer to section 5.2.1),
BF score, and Panoptic Quality. The following sections will explain each,
skipping IoU, and AP already discussed.

####  6.2.1 Pixel Accuracy

Measures the proportion of correctly classified pixels in the whole image. It
is calculated by dividing the number of correctly classified pixels by the
total number of pixels in the image. The formula for pixel accuracy is

$Pixelaccuracy=\frac{\text{Number of correctly classified pixels}}{\text{Total
number of pixels in image}}$ (45)

Like regular accuracy, pixel accuracy can be misleading when the class
imbalance is high, as it does not consider false positives or negatives. This
is why other metrics, such as Intersection over Union (IoU) or Dice
coefficient, are commonly used to evaluate image segmentation models.

####  6.2.2 Boundary F1 Score (BF)

The Boundary F1 Score (BF) [151], often abbreviated as BF score, is a metric
used for evaluating image segmentation quality, particularly when the
precision of the boundary location is important. The BF score applies the
concept of the F1 score to the segmentation boundaries rather than the segment
regions themselves.

The computation of the BF score involves the following steps:

  1. 1.

For each segment in the ground truth, find the closest predicted segment
(according to some distance measure, often the mean shortest distance between
the boundaries of the two segments).

  2. 2.

Compute the precision as the proportion of predicted segments close enough to
a ground truth segment. Formally, this is $P=TP/(TP+FP)$, where $TP$ (True
Positive) is the number of predicted segments close enough to a ground truth
segment, and $FP$ (False Positive) is the number of predicted segments not
close enough to any ground truth segment.

  3. 3.

Compute the recall as the proportion of ground truth segments close enough to
a predicted segment. Formally, this is $R=TP/(TP+FN)$, where $FN$ (False
Negative) is the number of ground truth segments not close enough to any
predicted segment.

  4. 4.

The BF score is then the harmonic mean of precision and recall:

$F\\_score=\frac{2*Precision*Recall}{Precision+Recall}$ (46)

A threshold is typically defined as _close enough_ when matching predicted
segments to ground truth segments. The BF score ranges from 0 to 1, with 1
indicating a perfect match between the predicted and ground truth boundaries.

####  6.2.3 Panoptic Quality (PQ)

Panoptic Quality (PQ) is a metric proposed for evaluating panoptic
segmentation tasks [107].

The Panoptic Quality metric is defined as:

$PQ=\frac{\Sigma_{(p,g)\in
TP}IoU(p,g)}{|TP|}\times\frac{|TP|}{|TP|+\frac{1}{2}|FP|+\frac{1}{2}|FN|},$
(47)

where

  * •

IoU(p, g) denotes the intersection-over-union of prediction p and ground truth
g.

  * •

TP (True Positive) is a set of matched pairs of predicted and ground truth
segments.

  * •

FP (False Positive) is a set of predicted segments not matched with any ground
truth segment.

  * •

FN (False Negative) is a set of ground truth segments not matched with any
predicted segment.

The PQ metric ranges between 0 and 1, where 1 signifies a perfect
segmentation. It is a product of two terms:

The first term, called _segmentation quality_ (SQ), calculates the average IoU
of the correctly predicted segments (true positives). This term measures how
accurately each detected object has been segmented and thus rewards prediction
quality.

The second term, called _Recognition Quality_ (RQ), calculates the ratio of
the number of true positives to the total number of segments. This term
measures how accurately each object has been recognized.

This metric can be more informative than mean IoU when dealing with complex
scenes containing multiple object instances per class, as it considers both
the segmentation quality and the correct recognition of distinct instances.

##  7 Face Recognition

Face recognition aims to accurately match an individual’s face in an image or
video to a corresponding entry in a database of faces. This task is often
performed using deep learning algorithms, such as Convolutional Neural
Networks (CNNs) or Transformers [152], that are trained on large datasets of
face images. The algorithms are trained to extract features from images faces
and then use these features to recognize faces that match those in the
database. Face recognition has many applications, including security [153,
154], social media [155], and biometric identification systems [156, 157].

###  7.1 Face Recognition Loss Functions and Metrics

The loss functions used in face recognition are typically aimed at preserving
the structure and similarity of the input faces. They can be divided into two
classes: loss functions based on classification and loss functions based on
representation learning. Common loss functions based on classification are
softmax loss, A-softmax loss, Center loss, Large-Margin cosine loss, and
Additive Angular Margin loss. On the other hand, loss functions based on
representation learning are Triplet loss, Contrastive loss, Circle loss, and
the Barlow twins loss.

The metrics commonly used for face recognition are the same as the ones used
for classification, such as accuracy, precision, recall, F1-score, ROC, etc.

In the following subsections, we will describe each of these loss functions.

####  7.1.1 Softmax Loss

The Softmax Loss computes the cross-entropy between the predicted class
probabilities and the true class label, followed by the logarithmic operation
to convert the negative log-likelihood into a loss value. The final loss is
obtained by summing the cross-entropy over all classes and all samples.

Let’s denote the weight vectors for each class (or, in this case, each face
identity) as $W=\\{w_{1},w_{2},...,w_{n}\\}$, where $n$ is the number of
classes (or identities). For a given input image $x$ of class $y$, the linear
classifier would compute a score $f(x,W)=Wx+b$, where $b$ is the bias term.

The softmax function converts these scores into probabilities. The probability
of the $i^{th}$ class is computed as follows:

$P(y=i|x;W)=\frac{e^{w_{i}^{T}x+b_{i}}}{\sum_{j=1}^{n}e^{w_{j}^{T}x+b_{j}}}$
(48)

The softmax loss (also known as the cross-entropy loss) for an input-output
pair $(x,y)$ is defined as the negative log-likelihood of the correct class,
which we can be expressed as

$L_{i}=-\log(P(y=y_{i}|x;W))=-f_{y_{i}}+\log\sum_{j=1}^{n}e^{f_{j}},$ (49)

where $f_{y_{i}}$ is the score for the correct class, and $f_{j}$ are the
scores for all classes. The total loss for a batch of data is the mean of
$L_{i}$ over all the examples in the batch.

The disadvantage of this loss function is that it does not have fine-grained
control over the intra-class and inter-class distances that come in handy for
face recognition purposes.

####  7.1.2 A-Softmax Loss

The A-Softmax loss [158], also known as the SphereFace loss, was designed to
address the limitations of the traditional softmax loss by considering the
angular information between the features of face images and their
corresponding labels. The A-Softmax loss aims to maximize the inter-class
separability and minimize the intra-class variations of face features.

Given a weight matrix $W$, an input feature vector $x$, and a margin parameter
$m$, the SphereFace loss is calculated as follows:

1\. Compute the Normalized weight matrix $W_{norm}$ as

$W_{norm}=\frac{W}{||W||}$ (50)

where each column $w_{i}$ in $W_{norm}$ is a unit vector, i.e., $||w_{i}||=1$.
The normalization operation makes the weights for each class lie on the unit
hypersphere.

2\. Compute the margin $M$ to be applied to the angles:

$M=(m-1)\cdot y_{true}+1$ (51)

In this equation, $y_{true}$ is the true class label. If $y_{true}$ equals 1,
$M$ equals $m$, and if $y_{true}$ equals 0, $M$ equals 1.

3\. Compute the cosine of the angle $\theta$ between the feature vector $x$
and the weight vector:

$cos(\theta)=\frac{W_{norm}\cdot x}{||x||},$ (52)

where $||\cdot||$ denotes the L2 norm.

4\. Compute the new angle $\theta^{\prime}$ and its cosine after applying the
margin:

$\displaystyle\theta^{\prime}=\theta\cdot M$ (53) $\displaystyle
cos(\theta^{\prime})=cos(\theta^{\prime})$

5\. Compute the new prediction $y_{pred}^{\prime}$ by rescaling with the norm
of $x$:

$y_{pred}^{\prime}=||x||\cdot cos(\theta^{\prime})$ (54)

6\. Finally, compute the SphereFace loss $L$, which is the negative log-
likelihood of the true class:

$L=-\log\frac{\sum y_{true}\cdot e^{y_{pred}^{\prime}}}{\sum
e^{y_{pred}^{\prime}}}$ (55)

Here, the summation is taken over all classes. The numerator is the
exponentiated prediction for the true class, and the denominator is the sum of
exponentiated predictions for all classes.

Compared to the traditional softmax loss, the A-Softmax loss produces more
discriminative and compact features, improving face recognition performance.

####  7.1.3 Center Loss

The center loss [159] aims to increase the inter-class variance while reducing
the intra-class variance by penalizing the distances between the features of a
sample and the center of its corresponding class. The center loss is inspired
by the idea of having a center for each class in the feature space, and the
loss function encourages the features of the same class to be close to its
center. The center loss is defined as the Euclidean distance between the
feature of a sample and the center of its class in the feature space.

The center loss is usually added to the main loss function in the training
process, and it is defined as

$L_{center}=\frac{1}{2}\sum_{i=1}^{n}\left|\mathbf{x_{i}}-\mathbf{c_{y_{i}}}\right|_{2}^{2},$
(56)

where $\mathbf{x_{i}}$ is the feature representation of the $i^{th}$ sample,
$y_{i}$ is its corresponding class label, $\mathbf{c_{y_{i}}}$ is the center
of class $y_{i}$, and $n$ is the number of samples.

####  7.1.4 CosFace: Large-Margin Cosine Loss

CosFace loss, also known as the Large Margin Cosine Loss, maximizes the
decision margin in the cosine space to further enhance the discriminative
power of the deeply learned features.

The cosine of the angle between the feature vector $x_{i}$ and the weight
$w_{j}$ of the $j^{th}$ class is given by

$cos\theta_{j}=\frac{w_{j}^{T}x_{i}}{||w_{j}||||x_{i}||},$ (57)

where $||.||$ denotes the l2 norm.

The CosFace method adds a cosine margin $m$ to the target logit, so the
modified cosine of the angle $\theta_{y_{i}}$ corresponding to the ground
truth class $y_{i}$ is given by:

$cos\theta_{y_{i}}-m$ (58)

Then the CosFace loss for an input-output pair $(x_{i},y_{i})$ is defined as

$L_{i}=-\log\frac{e^{s(cos\theta_{y_{i}})-m}}{e^{s(cos\theta_{y_{i}})-m}+\sum_{j\neq
y_{i}}^{n}e^{scos\theta_{j}}},$ (59)

where $s$ is a scaling factor.

One advantage of this function is that the cosine function’s non-monotonicity
does not create a problem here, unlike SphereFace. Also, because the feature
vector is normalized, the model must learn better separation of the angles as
it cannot reduce loss by learning a different norm [160].

####  7.1.5 ArcFace. Additive Angular Margin Loss

The ArcFace loss [161] enhances the discriminative power of the softmax loss
by adding an angular margin penalty to the target logit.

The ArcFace method adds an additive angular margin $m$ to the target logit, so
the modified cosine of the angle $\theta_{y_{i}}$ corresponding to the ground
truth class $y_{i}$ is given by

$cos(\theta_{y_{i}}+m)$ (60)

Then the ArcFace loss for an input-output pair $(x_{i},y_{i})$ is defined as

$L_{i}=-\log\frac{e^{scos(\theta_{y_{i}}+m)}}{e^{scos(\theta_{y_{i}}+m)}+\sum_{j\neq
y_{i}}^{n}e^{scos\theta_{j}}},$ (61)

where $s$ is a scaling factor. The margin $m$ can be interpreted as an
additional arc length on the hypersphere of radius $s$. Experiments show
better inter-class discrepancy than Triplet Loss while having about the same
intra-class similarity [160].

####  7.1.6 Triplet Loss

Probably the best-known loss function for face recognition. The idea behind
the triplet loss [162, 163] is to train the model to distinguish between a
positive pair of images (two images of the same person) and a negative pair of
images (two images of different persons).

Given an anchor image, $A$, a positive image, $P$, and a negative image, $N$,
(see Figure 5), the loss is calculated as the distance between the anchor
image and positive image and the distance between the anchor image and
negative image, plus a margin. The equation is defined as

$L_{triplet}=\max{0,\left\lVert\textbf{f}_{A}-\textbf{f}_{P}\right\rVert^{2}_{2}-\left\lVert\textbf{f}_{A}-\textbf{f}_{N}\right\rVert^{2}_{2}+\alpha},$
(62)

where $\textbf{f}_{A}$, $\textbf{f}_{P}$, and $\textbf{f}_{N}$ are the
embeddings of the anchor image, positive image, and negative image,
respectively, and $\alpha$ is a hyperparameter known as the margin. By
minimizing this loss, the embeddings of the positive images get closer to each
other than the embeddings of the negative images.

Figure 5: Triplet Loss. Given an anchor image, a positive image, and a
negative image, the triplet loss computes the distance between the embeddings
of the anchor image and positive image and the distance between the embeddings
of the anchor image and negative image. By minimizing this loss, the
embeddings of the positive images get closer to each other than the embeddings
of the negative images.

####  7.1.7 Contrastive Loss

The contrastive loss [164] learns a feature representation or image embedding
that projects similar images close to each other and dissimilar images far
apart. This loss is based on a Siamese architecture of the neural network. The
contrastive loss function is defined as:

$L=\frac{1}{N}\sum_{i=1}^{N}y_{i}\left|f(x_{i}^{a})-f(x_{i}^{p})\right|_{2}^{2}+(1-y_{i})\max\left(0,m-\left|f(x_{i}^{a})-f(x_{i}^{n})\right|_{2}^{2}\right),$
(63)

where $f(x_{i}^{a})$ is the deep feature representation of the anchor image
$x_{i}^{a}$, $f(x_{i}^{p})$ is the deep feature representation of the positive
image $x_{i}^{p}$, $f(x_{i}^{n})$ is the deep feature representation of the
negative image $x_{i}^{n}$. $y_{i}$ is a binary label indicating whether the
anchor and positive images are similar (1) or dissimilar (0), $m$ is a margin
hyperparameter, and $N$ is the number of triplets in the batch.

The margin $m$ controls how hard the model should work to push dissimilar
embeddings apart. Extending a trained model for new/unseen classes is easy
because it learns to create a semantic representation of the image rather than
classify it among a predetermined set of classes.

One limitation of this loss is that the margin $m$ is the same constant for
all dissimilar pairs, which implicitly pushes the model to have the same
distance between all dissimilar pairs, even if some are more dissimilar [160,
165]. A second limitation is that the absolute notion of similar and
dissimilar pairs is not transferable from one context to another [166].

####  7.1.8 Circle Loss

The Circle Loss [167] pushes positive pairs closer and negative pairs farther
away while maintaining a _circle-like_ decision boundary. This is achieved by
adding margins to positive and negative pairs, enlarging the intra-class
variance for negative pairs, and reducing the intra-class variance for
positive pairs. By doing so, Circle Loss can effectively handle imbalanced
data and complex distributions, which are common challenges in face
recognition tasks. The circle loss equation can be expressed as

$\displaystyle\alpha_{pos_{i}}$
$\displaystyle=\max(O_{pos_{i}}-s_{pos_{i}},0)$ (64)
$\displaystyle\alpha_{neg_{j}}$
$\displaystyle=\max(O_{neg_{j}}-s_{neg_{j}},0)$ $\displaystyle sum_{pos}$
$\displaystyle=\sum_{i}e^{-\gamma\cdot\alpha_{pos_{i}}\cdot_{pos_{i}}}$
$\displaystyle sum_{neg}$
$\displaystyle=\sum_{j}e^{\gamma\cdot\alpha_{neg_{j}}\cdot s_{neg_{j}}}$
$\displaystyle L$ $\displaystyle=\log(1+sum_{pos}\cdot sum_{neg})$

Where:

  * •

$s_{pos_{i}}$ and $s_{neg_{j}}$ represent the pairwise similarity between the
positive and negative pairs. The positive pairs belong to the same class, and
the negative pairs belong to different classes.

  * •

$O_{pos_{i}}$ and $O_{neg_{j}}$ represent the user-defined margins for
positive and negative pairs, respectively. $O_{pos}$ is a margin that should
be smaller than the expected similarity of positive pairs, while $O_{neg}$
should be larger than the expected similarity of negative pairs. Thus, you
want to choose $O_{pos}$ and $O_{neg}$ such that $O_{pos}<O_{neg}$.

  * •

$\alpha_{pos_{i}}$ and $\alpha_{neg_{j}}$ are slack variables that ensure the
positive similarities are larger than $O_{pos_{i}}$ and the negative
similarities are smaller than $O_{neg_{j}}$. This is achieved by setting the
minimum value to 0, ignoring similarities that have already met the margin
requirement.

  * •

$sum_{pos}$ and $sum_{neg}$ are exponentiated and scaled sums of the positive
and negative similarities, respectively. The exponential function ensures that
all values are positive and emphasize larger values, while $\gamma$ is a
scaling factor that controls the rate at which the emphasis increases. The
slack variables $\alpha_{pos_{i}}$ and $\alpha_{neg_{j}}$ are used in the
exponent to give more weight to pairs far from meeting the margin requirement.

  * •

Finally, $L$ is the Circle Loss computed as the logarithm of 1 plus the
product of $sum_{pos}$ and $sum_{neg}$. This encourages both $sum_{pos}$ and
$sum_{neg}$ to be small, which in turn encourages the positive similarities to
be large and the negative similarities to be small. Adding one inside the
logarithm ensures that the argument is always positive, and the logarithm
itself helps dampen the effect of large values and reduce the effect of
outliers.

The circle loss has a more definite convergence target than the triplet loss
because there is a single point in the ($s_{neg}$, $s_{pos}$) space toward
which the optimization is driven ($O_{neg}$, $O_{pos}$). However, choosing
($O_{neg}$, $O_{pos}$) is arbitrary. In practice, it is common to use cross-
validation or a separate validation set to tune these hyperparameters.

A common strategy is to start with small margins and gradually increase them.
If the margins are too large, the model struggles to learn; if they are too
small, it learns embeddings that are not discriminative enough. In some
implementations of Circle Loss, $O_{pos}$ and $O_{neg}$ are not independent
but related by a constant margin $m$, which is set to ensure a sufficient gap
between positive and negative pairs. In this case, we only need to tune one of
the margins or the gap $m$.

####  7.1.9 Barlow Twins Loss

The Barlow Twins loss [168] is a self-supervised learning approach. The key
idea is to make the outputs of a two-twin network, processing two different
views of the same image, as similar as possible while reducing the redundancy
between the dimensions of these representations. It encourages the network to
learn highly informative features about the input and non-redundant.

Given the batch size $N$ and the dimensionality $D$ of the embeddings, the
network processes two different augmentations of the same input data,
producing the embeddings $z_{a}$ and $z_{b}$. These embeddings are then
normalized to have zero mean and unit variance along the batch dimension.

The computation of the Barlow Twins loss can be formulated as follows: 1\.
Compute the cross-correlation matrix $C$:

$C=\frac{1}{N}z_{a_{norm}}^{T}\cdot z_{b_{norm}}$ (65)

2\. Compute the difference matrix $C_{diff}$ between the cross-correlation
matrix and an identity matrix and square it:

$C_{diff}=(C-I)^{2}$ (66)

3\. Scale the off-diagonal elements of $C_{diff}$ by a factor $\lambda$:

$C_{diff_{ij}}=\begin{cases}C_{diff_{ij}},&\text{if }i=j\\\ \lambda
C_{diff_{ij}},&\text{if }i\neq j\end{cases}$ (67)

4\. Finally, the Barlow Twins loss $L$ is the sum of all elements in the
updated $C_{diff}$:

$L=\sum_{i,j}C_{diff_{ij}}$ (68)

By backpropagating this loss and updating the model’s parameters, the network
is trained to minimize redundancy and increase the similarity between two
different views of the same image, learning more robust and informative
features as a result.

This method does not require a fixed number of classes and does not suffer
from data expansion as it does not require explicit negative examples.
However, the model in the paper required large dimensionality of the final
representation for good performance, and the performance is not robust to
remove certain distortions to the inputs [160].

####  7.1.10 SimSiam Loss

SimSiam [169] is a self-supervised learning aimed at learning representations
by pushing two views of the same image to be as similar as possible. While it
is not explicitly designed for face recognition, it can be used to learn
meaningful representations of faces.

Given two different augmentations $x_{1}$ and $x_{2}$ of the same image, they
are forwarded through a neural network and a prediction head to obtain the
features $z_{1},z_{2}$ and the predictions $p_{1},p_{2}$.

The loss is defined as the negative cosine similarity between the predictions
and the features of the different augmentations.

$L=-\frac{1}{2}[\frac{(p_{1}^{T}z_{2})}{(||p_{1}||\cdot||z_{2}||)}+\frac{(p_{2}^{T}z_{1})}{(||p_{2}||\cdot||z_{1}||)}],$
(69)

where T denotes transpose, and $||\cdot||$ denotes the L2 norm.

This loss encourages the model to make the predictions $p_{1}$ and $p_{2}$ as
similar as possible to the features $z_{2}$ and $z_{1}$, respectively.

An important part of the SimSiam approach is a stop-gradient operation applied
to $z_{2}$ in the first term and $z_{1}$ in the second term of the loss
function. This means that the gradients are not backpropagated through these
variables during training. The stop-gradient operation is critical to avoid
the model collapsing into trivial solutions where the features and the
predictions are the same.

The advantages of the SimSiam loss are:

  1. 1.

_No Negative Pair:_ Unlike contrastive learning methods that require negative
examples, SimSiam does not require any. This simplifies the model training
process and can make it more efficient.

  2. 2.

_Stop-Gradient Operation:_ The use of stop-gradient operation in SimSiam makes
it less prone to collapsing into trivial solutions. This is a significant
advantage because many self-supervised learning models struggle with this
problem.

  3. 3.

_Simplicity:_ SimSiam is simpler compared to other self-supervised learning
methods. It uses a symmetric architecture and a simple loss function which
encourages two views of the same image to have similar representations.

The disadvantages include:

  1. 1.

_Hyperparameter Sensitivity:_ SimSiam has a few crucial hyperparameters, such
as the learning rate and weight decay, which require careful tuning to get the
best performance. Incorrect settings can significantly degrade the model’s
performance.

  2. 2.

_Dependence on Data Augmentation:_ The success of SimSiam, like many other
self-supervised learning models, heavily relies on the choice and quality of
data augmentations. This requires domain knowledge and potentially significant
effort to determine the most effective augmentations.

  3. 3.

_Non-semantic Features:_ One common issue with self-supervised learning
methods, including SimSiam is that the features learned may not necessarily be
semantically meaningful. They are good at capturing low-level features but may
not be as effective in capturing high-level semantic information.

##  8 Image Generation

Image generation in deep learning involves using artificial neural networks to
generate new images. This task has been revolutionized by developing various
models such as Variational Autoencoders (VAEs) [170, 171, 172, 173],
Generative Adversarial Networks (GANs) [174, 175, 176, 177, 178], Normalized
Flow models (NFs) [179, 180, 181, 182, 183], Energy-Based Models (EBMs) [184,
185, 186, 187, 188, 189], and Diffusion Models [190, 191, 192, 193, 194].
These models allow the generation of high-quality images that can be used in
various applications such as image super-resolution [195, 196, 197, 198],
denoising [199, 200, 201], inpainting [202, 203, 204, 205], and style transfer
[206, 207, 208, 209].

_Variational Autoencoders (VAEs)_ are generative models that use deep learning
techniques to create new data and learn latent representations of the input
data. VAEs consist of two primary components: an encoder and a decoder. The
encoder takes input data and compresses it into a lower-dimensional latent
space, capturing the essential features of the data. This is typically a
probabilistic process, producing a mean and standard deviation representing
potential latent space values distribution. The decoder then takes a point
from this latent space distribution and reconstructs the original data. The
entire process is trained in such a way as to minimize the difference between
the original and the reconstructed data, as well as to ensure the latent space
approximates a standard Gaussian distribution. VAEs can generate new data by
feeding the decoder points sampled from the latent space.

_Generate Adversarial Networks (GANs)_ involve two neural networks, a
Generator, and a Discriminator, playing a game against each other. The
Generator tries to create data similar to the training data, while the
Discriminator tries to distinguish between the real and generated data.
Through this process, both networks improve: the Generator learns to produce
increasingly realistic data, while the Discriminator becomes better at
distinguishing between real and artificial data. This adversarial process
continues until an equilibrium is reached, at which point the Generator is
producing realistic data and the Discriminator is, at best, randomly guessing
whether the data is real or generated. This equilibrium is conceptually
referred to as a Nash equilibrium in game theory [210].

_Normalizing Flow Models (NFs)_ can construct complex probability
distributions by transforming a simple base distribution through a sequence of
invertible and differentiable transformations, or _flows_. These flows warp
and twist the data space, allowing the model to generate diverse outputs. The
parameters of these transformations are learned from data using maximum
likelihood estimation. An advantage of Normalizing Flows over other generative
models is their ability to provide an exact and tractable likelihood for a
given sample, enabling efficient sampling and density estimation. However,
they can be computationally intensive to train due to the need to compute and
backpropagate through the Jacobian of the flows.

_Energy-Based Models (EBMs)_ learn a scalar energy function to distinguish
real data points from unlikely ones, assigning lower energy values to points
similar to the training data and higher values otherwise. A neural network
often parameterizes this function and learns from the data. Sampling in EBMs
is typically done via Markov Chain Monte Carlo (MCMC) [211] methods, producing
samples distributed according to the learned energy function. While EBMs can
represent a wide variety of data distributions, they can be challenging to
train due to the intractability of the partition function and the
computational expense of MCMC sampling.

_Diffusion models_ can create new data by simulating a random process,
specifically a diffusion process. The process starts with a simple data
distribution (e.g., Gaussian noise) and gradually transforms it towards the
target data distribution, like a particle undergoing diffusion or a random
walk. This transformation is controlled by a neural network trained to model
the reverse process, taking real-world data and applying a series of
transformations to reduce it to noise. During generation, an initial noise
sample is iteratively updated using this reverse process model, running the
process backward, leading to a sample from the target data distribution. This
approach allows the generation of complex and high-quality data, such as
images, by simulating a smooth transition from noise to the desired data.

The following sections will review the common lost functions and performance
metrics used for image generation.

###  8.1 Image Generation Loss functions

The loss function in a VAE consists of the reconstruction loss and the
Kullback-Leibler (KL) (refer to section 8.1.2). The reconstruction loss
measures how well the decoded data matches the original input data. The KL
divergence measures how much the learned distribution in the latent space
deviates from a target distribution, usually a standard normal distribution.
KL-divergence is used as a regularization term to ensure that the
distributions produced by the encoder remain close to a unit Gaussian,
penalizing the model if the learned distributions depart from it.

The most common loss function used in GANs is the adversarial loss, which is
the sum of the cross-entropy loss between the generator’s predictions and the
real or fake labels. More recently, WGAN [212] applied the Wasserstein
distance as an alternative to training GANs to improve the stability and avoid
mode collapse that occurs when the generator network stops learning the
underlying data distribution and begins to produce a limited variety of
outputs, rather than a diverse range of outputs that accurately represent the
true data distribution.

Normalizing Flows are typically trained using maximum likelihood estimation.
Given a dataset, the aim is to maximize the log-likelihood of the data under
the model by minimizing the negative log-likelihood of the data.

During training, Energy-based models (EBMs) minimize a loss function that
encourages the energy function to assign lower energy values to data points
from the training data and higher energy values to other points. Different
types of EBMs use different loss functions, such as Contrastive Divergence
(CD)[213], Maximum Likelihood Estimation (MLE)[214], and Noise-Contrastive
Estimation (NCE) [215].

Diffusion models use a denoising loss function based on the Mean Absolute
Error (MAE) or the Mean Squared Error (MSE) between the original and
reconstructed data.

In the next sections, we describe in detail each of these losses.

####  8.1.1 Reconstruction Loss

The purpose of the reconstruction loss is to ensure that the decoder network
of the VAE can reconstruct an input image that is as close as possible to the
original image. In essence, the reconstruction loss compares the original
image to the reconstructed image and penalizes any difference between the two.
The reconstruction loss is calculated as the mean squared error (MSE) between
the original image and the reconstructed image, which is defined as follows

$L_{recon}=\frac{1}{N}\sum_{i=1}^{N}||x_{i}-\hat{x_{i}}||^{2},$ (70)

where $x_{i}$ is the original image, $\hat{x_{i}}$ is the reconstructed image,
and $N$ is the number of images.

Another metric that can be used for reconstruction loss is binary cross-
entropy (BCE), a common choice for binary images. The BCE loss can be defined
as:

$L_{BCE}=-\sum_{i=1}^{n}y_{i}\log(\hat{y_{i}})+(1-y_{i})\log(1-\hat{y_{i}}),$
(71)

where $y_{i}$ is the binary value of the original image and $\hat{y_{i}}$ is
the binary value of the generated image.

In the context of deep learning, reconstruction loss has been used in various
image generation tasks such as image restoration, image generation, and image
synthesis.

####  8.1.2 Kullback-Leibler Divergence Loss

The KL divergence loss, also known as the KL divergence loss, measures the
difference between the predicted probability distribution and the true
probability distribution of the classes [216]. The KL divergence loss ensures
that the predicted probabilities are close to the true probabilities, which
can be useful in cases where the true probabilities are known or can be
approximated.

The KL divergence loss is defined as

$KL(p||q)=\sum_{i=1}^{n}p(x_{i})log(\frac{p(x_{i})}{q(x_{i})}),$ (72)

where $p(x_{i})$ is the true probability of class $x_{i}$ and $q(x_{i})$ is
the predicted probability of class $x_{i}$.

The KL divergence loss is often used in generative models such as variational
autoencoders, which aim to approximate the true data distribution. It is also
used in reinforcement learning to ensure that the agent’s learned policy is
close to the optimal policy.

One disadvantage of using the KL divergence loss is that it is sensitive to
zero probabilities in the true probability distribution. This can lead to
numerical instability and can make the loss function difficult to optimize. To
overcome this issue, a common practice is to add a small value (e.g., $1e-7$)
to the true probability distribution to avoid zero probabilities.

####  8.1.3 Adversarial Loss

The adversarial loss is the main function used in Generative Adversarial
Networks (GANs). It is based on the idea of a minimax game between the two
neural networks. The adversarial loss is used to train the generator, defined
as the negative log-likelihood of the discriminator’s output.

Goodfellow et al. first introduced the adversarial loss in [174]. The authors
showed that this loss function leads to the convergence of the generator to a
Nash equilibrium, where the generator produces realistic images that the
discriminator cannot distinguish from real images.

The adversarial loss can be defined as:

$L_{adv}(G,D)=-\mathbb{E}{x\sim p{data}(x)}[\log D(x)]-\mathbb{E}_{z\sim
p_{z}(z)}[\log(1-D(G(z)))],$ (73)

where $G$ is the generator network, $D$ is the discriminator network, $x$ are
real images from the data distribution $p_{data}(x)$, $z$ are random noise
inputs from the noise distribution $p_{z}(z)$, and $G(z)$ are generated
images.

The adversarial loss encourages the generator to generate indistinguishable
images from real images and the discriminator to correctly classify real
images and generated images. The training process alternates between updating
the generator and the discriminator until the generator produces realistic
images.

####  8.1.4 Wasserstein loss

The Wasserstein loss [212] is used in generative models, such as Generative
Adversarial Networks (GANs), to measure the difference between the real and
generated distributions. The Wasserstein loss is the minimum work required to
transform one probability distribution into another. The amount of work is the
sum of the distances between each sample multiplied by their probability mass.
The Wasserstein loss can be written as

$W(p_{data},p_{gen})=\inf_{\gamma\in\Gamma(p_{data},p_{gen})}\int_{x,y}||x-y||d\gamma(x,y)$
(74)

where $p_{data}$ and $p_{gen}$ represent the real and generated distributions,
respectively. $\Gamma(p_{data},p_{gen})$ is the set of joint distributions
with marginals $p_{data}$ and $p_{gen}$, and $\gamma(x,y)$ is the joint
distribution of $(x,y)$.

The Wasserstein loss has been widely used in GANs for its ability to provide a
more stable training process and to avoid the mode collapse problem, where the
generator produces a limited number of similar outputs.

####  8.1.5 Negative Log-likelihood in Normalizing Flows

In Normalizing Flows, the objective is to learn a transformation (or a
sequence of transformations) that maps a complex data distribution to a
simpler distribution, for example, a standard Gaussian. The transformations
are chosen such that their Jacobian determinant is easy to compute, which
allows using the change of variables formula to compute the likelihood of the
data under the model.

Given a data point $x$, let’s denote $z=f_{\theta}(x)$ as the mapping of $x$
to the base distribution under the flows parameterized by $\theta$, and
$p_{z}(z)$ as the density of the base distribution. Then, the log-likelihood
of $x$ under the model is given by

$\log p_{\theta}(x)=\log p_{z}(f_{\theta}(x))-\log\left|\det\frac{\partial
f_{\theta}(x)}{\partial x}\right|$ (75)

The second term is the log absolute determinant of the Jacobian of the
transformation, which corrects for the change in volume due to the
transformation.

The loss function that we minimize during training is the negative log-
likelihood of the data. If our dataset consists of $N$ points
$x_{1},x_{2},\ldots,x_{N}$, then the loss function $L(\theta)$ is expressed as

$L(\theta)=-\frac{1}{N}\sum_{i=1}^{N}\log
p_{\theta}(x_{i})=-\frac{1}{N}\sum_{i=1}^{N}\left[\log
p_{z}(f_{\theta}(x_{i}))-\log\left|\det\frac{\partial
f_{\theta}(x_{i})}{\partial x_{i}}\right|\right]$ (76)

In practice, stochastic gradient descent or a variant minimizes this loss
function and learns the parameters $\theta$ of the flows.

Normalizing Flows are computationally intensive to train, due to the need to
compute and backpropagate through the Jacobian of the flows. For this reason,
methods such as RealNVP [179] and Glow [180] are designed so that the
determinant of the Jacobian can be computed efficiently.

####  8.1.6 Contrastive Divergence

Contrastive Divergence (CD) can be used in training Energy-Based Models
(EBMs), specifically for estimating the log-likelihood gradient. As with RBMs,
the objective is to maximize the likelihood of the data under the model. The
key difference is that the energy function is defined over the visible and
hidden units in EBMs, whereas it’s defined over the joint configuration of
visible and hidden units in RBMs.