# Streamlined Framework for Agile Forecasting Model Development towards
Efficient Inventory Management

Jonathan Hans Soeseno  Sergio González  Trista Pei-Chun Chen  AI Center,
Inventec Corporation, Taipei, Taiwan

###### Abstract

This paper proposes a framework for developing forecasting models by
streamlining the connections between core components of the developmental
process. The proposed framework enables swift and robust integration of new
datasets, experimentation on different algorithms, and selection of the best
models. We start with the datasets of different issues and apply pre-
processing steps to clean and engineer meaningful representations of time-
series data. To identify robust training configurations, we introduce a novel
mechanism of multiple cross-validation strategies. We apply different
evaluation metrics to find the best-suited models for varying applications.
One of the referent applications is our participation in the intelligent
forecasting competition held by the United States Agency of International
Development (USAID). Finally, we leverage the flexibility of the framework by
applying different evaluation metrics to assess the performance of the models
in inventory management settings.

###### keywords:

Demand forecasting, Model development, Model selection, Forecasting
competitions , Feature engineering , Time series.

##  1 Introduction

Logistics is defined as the coordination of information, physical, and
financial flows to and from trading partners and consumers. An inventory
requires capital investment to build and stock finished goods. Inventory
management considers several complex variables, including time, cost,
location, transport, and risks. Logistics inventory management is a long-
standing challenge that is essential for various aspects of life. One concrete
example is its impact on global health [6, 49]. In this setting, adequate
inventory management translates to demand satisfaction of medical products,
which could potentially save lives, prevent unwanted pregnancies, and improve
the general well-being of people’s lives. Another example in an industrial
setting can be drawn by considering a manufacturing company that further
assists companies in manufacturing their products [54]. Here, each company may
request hundreds or even thousands of products. Inventory management plays a
crucial role in reducing costs and mitigating the loss of potential income.
Many factors affect the results of inventory management solutions. One
prominent factor is the ability to forecast future demand accurately. Accurate
demand forecasts, in turn, facilitate better management of inventory
logistics.

Therefore, forecasting algorithms may be used to perform the analysis. As a
considerably popular field, there are many state-of-the-art algorithms that
are mainly categorized into statistical and AI-based approaches. Statistical
approaches often focus on modeling the temporal relation of a given time-
series data [9, 30]. Thus, they are robust and accurate. However, these
approaches rely heavily on data pre-processing steps to ensure the underlying
statistical constraints, such as stationarity (i.e., time-series statistics
are constant over time), seasonality, and the trend of an autoregressive
integrated moving average (ARIMA) model [9]. Because of this requirement,
statistical approaches are often used to model specific time-series data and
are therefore difficult to maintain at scale.

By contrast, AI-based approaches utilize machine learning and deep learning
models to solve tasks that may involve larger datasets [32, 22, 38]. Recently,
AI-based forecasting approaches have become more prominent in accommodating
multiple time series using a single model [8, 42]. This technique, known as
cross-learning, allows the model to share helpful information across time-
series data [50]. It also simplifies the development and deployment processes.
However, similar to all other deep learning approaches, AI-based forecasting
approaches struggle with smaller datasets, creating the necessity to tune and
hand-engineer complex data representations.

With such a wide variety of algorithm choices available, the problem of
maximizing performance then translates to searching for the most suitable
algorithm for the task at hand, often described with a recorded dataset.
Unfortunately, developing a forecasting model is not trivial, mainly because
of the complexity of its components. For instance, the datasets may consist of
various issues and formats, while the forecasting algorithms require specific
feature representations and proper hyperparameter tuning [12, 47]. In
addition, it would still be necessary to define a metric to select the best
model [13, 36], which varies significantly depending upon the intended
application of the models. For example, custom evaluation metrics based on an
inventory simulation process may be considered to estimate the performance of
an accurate model given an inventory management setting.

To address these issues, we propose a streamlined framework that organizes
each component in a modular manner. Therefore, it allows users to quickly
iterate through combinations of algorithms and datasets in a one-to-many or
many-to-many fashion. With numerous possible combinations of feature
representations, model architecture, training setup, and other relevant
hyperparameters, a robust strategy is required to choose models with the best
generalization properties. Hence, the framework enables the use of novel
multiple cross-validation strategies built on top of existing ones. In
addition, given the flexibility of the framework in the application of various
evaluation metrics with guaranteed reliability and reproducibility, it enables
us to extract insights from each model and select the most suitable model or
improve the model design depending on the requirements.

To evaluate the effectiveness of our framework, we perform several experiments
with four real-life forecasting problems. One of the referents is our
participation in the intelligent forecasting competition organized by USAID.
We begin by comparing the strategy of learning one model for each time series
and cross-learning a single model to predict multiple series. Further, we
empirically demonstrate the advantages of combining multiple cross-validation
strategies to tune robust models with better generalization. Finally, we
investigate the performance of such accurate forecasting models in the
inventory management setting by applying metrics derived from an inventory
simulation process. We summarize our contributions as follows:

  * 1.

A streamlined framework that allows users to quickly iterate through different
AI model training configurations by swiftly integrating various datasets,
algorithms, and metrics.

  * 2.

A scheme to facilitate a robust search for the most optimal training
configurations through the integration of multiple cross-validation
strategies.

  * 3.

A comprehensive evaluation of forecasting models based on user-defined
metrics, including conventional and inventory specific metrics.

The remainder of this paper is organized as follows. Section 2 presents the
background and related works on forecasting approaches, tuning and model
selection, and inventory management techniques. Section 3 describes in detail
the modules of the proposed streamlined forecasting framework, as well as its
main strengths and applications. Section 4 discusses our study case on the
intelligent forecasting competition by USAID. In Section 5, we empirically
analyze the design of our framework in terms of the use of cross-learning,
integration of multiple cross-validation strategies, and its applicability to
inventory management settings. Finally, Section 6 concludes our study and
discusses practical future work.

##  2 Related works

Forecasting [30, 9] is one of the most attractive research fields in
statistics and machine learning because of its impact on organizational
decision-making. In particular, forecasting is beneficial in financial,
economic, and operational processes [39], such as demand forecasting and
inventory management [55, 54]. A vast majority of forecasting research is
focused on the pursuit of economic profit. However, the knowledge learned in
the past few years is also applicable to general and social goods [49]. As
forecasting practitioners, we have the responsibility of executing the same.
For example, demand forecasting insights that are generally utilized by
retailers or manufacturing companies were instead used for humanitarian aid in
[37, 3, 10].

Therefore, practitioners aim to develop robust and accurate forecasting
algorithms. Section 2.1 addresses these forecasting approaches from the
perspective of statistical and AI-based models. In Section 2.2, we discuss the
importance of hyperparameter tuning on model performance, as well as the
metrics and validation strategies used in forecasting scenarios. Finally,
Section 2.3 examines the proposals for the transition of forecasting models
into inventory management decisions.

###  2.1 Forecasting algorithms

Over the years, the forecasting community has questioned which approaches are
more suitable for different time series forecasting problems [26, 29, 42]. The
superiority of statistical forecasting algorithms over AI-based approaches is
still an open discussion [19, 53]. The former includes classical and well-
known approaches, such as ARIMA [9], exponential smoothing [17, 18], and
Croston’s methods [14, 1], among others. The latter group involves the use of
deep learning [22, 38] and machine learning [32] algorithms along with feature
engineering [24].

In this discussion, Makridakis et al. [40] and Gilliland [19] based on the
results of the M4 competition, claimed that the more complex AI-based
approaches do not necessarily outperform simpler statistical forecasting
models. However, more recent studies [53, 8] and the M5 competition [43, 42]
demonstrated a superior performance by the AI-based models. Recent AI-based
approaches have the ability to learn a single model from multiple time series
data to forecast individual time series [50]. This mechanism, referred to as
cross-learning, is one of the contributors to the improvement in performance
in the latest competitions [52, 50, 8].

Among the AI-based approaches, recurrent neural networks [38], particularly
long short-term memory (LSTM) [25], and gradient boosting decision trees
(GBDT) [15, 16], in this case LightGBM [33], have been the most successful
algorithms for forecasting problems [8]. LSTMs are neural networks with added
cells that facilitate learning from sequence or temporal data. LightGBM is
Microsoft’s GBDT implementation with leaf-wise tree growth and histogram-based
cut selection, which makes it one of the most efficient and effective GBDTs
[21].

In addition to statistical and AI-based models, hybrid algorithms combine
classic forecasting approaches with the latest AI-based techniques [52, 44,
59]. As a reference, NeuralProphet [59, 58] integrates the classical
decomposition time series analysis of Facebook’s Prophet [57] and AR-Net [60].
AR-Net is a neural network architecture that performs an autoregressive
procedure similar to ARIMA but offers more options for scalability.

###  2.2 Hyperparameter tuning: metrics and validation strategies

Forecasting algorithms, especially AI-based algorithms, such as LSTM or GBDT,
consist of a considerable number of hyperparameters that are required to be
tuned to access their full potential [47]. Additionally, the correct selection
of the trained models through different validation strategies is essential to
achieve generalization and avoid overfitting [12]. This process can be
considerably costly, both computationally and for practitioners who analyze
the resultant models. LSTM and GBDT were not ranked as the best forecasting
approaches until they were computationally optimized [8].

Tunability and model selection is a complex stage of AI modeling with a
considerable research interest, which has motivated meta-learning and AutoML
[28, 63]. This interest is extendable to forecasting scenarios [45, 2].
However, it becomes even more difficult to forecast problems due to the lack
of a clear consensus on the metrics and validation strategies. Additionally,
the selection of engineered features commonly forms part of the tuning stage
of AI forecasting models.

Over the years, multiple metrics have been proposed to evaluate the accuracy
of these forecasting models [31, 36], including scale-dependent regression
metrics, such as mean absolute error (MAE) and mean squared error (MSE);
percentage metrics, such as symmetric mean absolute percentage error (SMAPE);
and, native forecasting metrics, such as mean absolute scaled error (MASE). In
forecasting model validation, three different groups of strategies outshine
the rest [13]. Out-of-sample validation retains a fragment at the end of the
time series as the validation set [56]. Prequential or time series cross-
validation uses an increasing window, which defines the training and
validation sets, and repeats the process several times while increasing the
window sizes. K-fold cross-validation divides the time series into equal parts
from which one is used for validation, while the rest is included in the
training set [4, 5].

###  2.3 From forecasting to inventory management

Once the models are developed, the forecasts can be applied to inventory
management decisions. This transition might not be straightforward, as the
most accurate models do not necessarily lead to the most suitable inventory
management [35, 20, 34]. Therefore, various approaches have been proposed
following two main paradigms, depending on whether they consider forecasting
and inventory management separate problems or a single learning process [27,
20]. In the first paradigm, the demand distribution is generally assumed or
roughly estimated using the forecasts and their errors [51, 27]. The second
paradigm simultaneously optimizes the forecasting models and decisions. Some
examples of these approaches are the minimization of the holding and shortage
costs in newsvendor problems [27, 48], consideration of demand uncertainty in
Bayesian forecasting models [46], and inclusion of simulation optimization
[35]. These approaches might not be as generalist as desired. That is, they
usually are defined on a particular inventory setting, or limited to specific
forecasting models.

Our approach is distinct from the existing inventory optimization approaches,
as we do not impose any distributions like the first paradigm; nor do we
perform any type of inventory optimization. Instead, we aim to select the best
configurations of the forecasting models based on their performance in terms
of inventory management metrics obtained through a simulation process.

##  3 Method

We aim to enable an agile development of forecasting models on a large scale.
However, it is a challenging task because the development process may require
several stages to work coherently. Therefore, we have designed a streamlined
framework that allows us to robustly and quickly develop our model by
seamlessly integrating various datasets, model architectures, and evaluation
metrics.

We begin by designing pre-processing modules for each dataset that apply the
most appropriate data cleaning procedures and enable a uniform interface
across the datasets. Next, we engineer lagged, pattern, and statistical
features to allow improved time-series data representation for the models and
store the cleaned data alongside the engineered feature representations in our
dataset repository. We further utilize the repository of datasets and
algorithms to train our model and tune the hyperparameters to satisfy user-
defined evaluation metrics. Figure 1 provides an overview of the developmental
procedures of our framework.

The pre-processing step to enable a uniform and clean interface across
different datasets is explained in Section 3.1. Section 3.2 addresses the
utilization of this uniform interface to build improved feature sets to
represent the time series. Next, Sections 3.3 and 3.4 discuss the procedure
used to train and tune the models to satisfy the evaluation metrics. Finally,
in Section 3.5, we highlight the main strengths of the proposed streamlined
framework and introduce real-world applications.

Figure 1: The proposed framework streamlines all development components of
forecasting models.

###  3.1 Uniform interface for datasets

In this step, the goal is to pre-process datasets obtained from various data
sources, which are likely to consist of different sizes, formats, and issues.
It is possible to design a single pipeline that is suitable for a majority of
these datasets. However, doing so would introduce substantial complexity for
accommodating large dataset variations. Such a tremendous effort would still
not be useful when we consider new datasets. Therefore, to avoid additional
and unnecessary complexity, we instead design a pre-processing module for each
dataset.

These pre-processing modules, further referred to as data loaders, enable us
to apply the most appropriate pre-processing steps for each dataset. For
instance, removing missing values from one dataset, or designing imputation
strategies on another dataset. However, because the datasets might contain
different information presented in various formats, the subsequent modules
would require a specific interface to access each of them. Therefore, in
addition to applying cleaning and pre-processing steps, we design the data
loaders to provide a uniform interface to access the time series identifier,
timestep, and target. We consider these features essential while developing
forecasting models.

###  3.2 Descriptive representation of time series

The uniform interfaces provided by the data loaders enable us to perform
feature engineering over each dataset. This procedure aims to engineer
meaningful representations that allow AI models to understand time-series data
better. For instance, instead of asking the model to memorize the past eight
recorded values, we can explicitly feed them as inputs. Doing so allows the AI
model to focus more on interpreting, rather than memorizing. This strategy is
beneficial while dealing with smaller datasets, as specific patterns might
require larger datasets to extract.

The feature engineering procedure considers three different types of features.
The lagged features are implemented by directly shifting the target variable
backward in time. The pattern features focus on describing the shape of the
time-series data within a given window. Depending on the recording frequency
of the dataset, the module extracts information, such as the percentage of
zero values, last-observed non-zero values, trends, and occurrences of up-down
patterns. Finally, we compute the statistic features of the recorded values
within a time series window, such as the mean, minimum, maximum, and standard
deviation.

This step results in feature sets that are stored inside the dataset
repository to accompany the cleaned dataset. It is possible to include other
features obtained from external sources into this repository. One of such
occasions was during our participation in the USAID intelligent forecasting
competition, where we incorporated world population data in addition to the
three time-series features.

###  3.3 Training and tuning AI models

Given multiple datasets and state-of-the-art forecasting algorithms available
in our repositories, we aim to develop a model that offers the best
performance. To this end, we need to maximize the model’s performance by
tuning the hyperparameters that control its capacity, training strategy,
learning objective, and overall performance. In addition to the large number
of hyperparameters, we are also required to carefully select the features from
the datasets, as some features may be crucial in improving the performance of
the model, while others might introduce unnecessary intricacies.

Combining the variation of the hyperparameters with the dataset features could
quickly generate a massive number of configurations. For example, consider ten
hyperparameters with three possible values each. When we combine them with 40
unique feature sets, the number of combinations could easily reach millions.
Given more complex model architectures with more hyperparameters, it would be
impractical to exhaustively attempt all the configurations because of a
significant computational resource and time requirement. Although impractical,
it is still beneficial to optimize the training process and leverage from
parallelization opportunities to sample and iterate through many
configurations, albeit not all of them. By analyzing different configurations
at such a scale, we can gain more insight into the problem and thus design
better data representations or model architectures.

Figure 2: Illustration of time-series cross-validation (top) and k-fold cross-
validation (bottom).

Therefore, it is necessary to select configurations that yield desirable
performance and robust models. To do so, we incorporated two different
validation strategies: time series and k-fold cross-validation (see the
illustration in Figure 2). The time series cross-validation is performed on
the time axis. Our k-fold cross-validation is performed across different time
series, contrary to traditional k-fold strategies. In each fold, we exclude
certain time series from training to constitute the validation set.

The main idea of cross-validation is to evaluate the performance of a
particular model against all seen portions of the datasets, leading to a more
reliable performance estimation. However, using only one cross-validation
strategy might not be sufficient when multiple time series exist. By
exclusively using the time series cross-validation, we only evaluate the
performance of the model in dealing with future unseen data points and ignore
generalization over different patterns of other time series. In contrast,
using only the k-fold cross-validation, we only examine the generalization on
other time series. Therefore, to ensure the selection of robust models, we
consider the application of both cross-validation strategies, simultaneously.
We accommodate the top configurations that are highly ranked by time series
and k-fold cross-validations. Further, we demonstrate the effectiveness of
using multiple cross-validation strategies in Section 5.3.

###  3.4 Evaluation metric

Each training configuration yields a forecasting model. With a large pool of
models obtained from the hyperparameter tuning process, we intend to reliably
evaluate and select the best-performing model. To this end, we separate the
evaluation metrics from the hyperparameter tuning process. This separation
enables us to apply multiple evaluation metrics to a specific training
configuration, and therefore, allows us to select the model that best fits the
intended application. We include a few conventional metrics [31, 36], such as
MAE, RMSE, and MASE, defined as follows:

$MAE=\frac{1}{N}\sum_{t=1}^{N}|y_{t}-\hat{y}_{t}|$ (1)

$RMSE=\sqrt{\frac{1}{N}\sum_{t=1}^{N}(y_{t}-\hat{y}_{t})^{2}}$ (2)

$MASE=\frac{\frac{1}{N}\sum_{t=1}^{N}|y_{t}-\hat{y}_{t}|}{\frac{1}{N-1}\sum_{t=2}^{N}|y_{t}-y_{t-1}|}$
(3)

Here, $y_{t}$ and $\hat{y}_{t}$ denote the ground-truth target and model
prediction at time $t$, respectively. These conventional metrics measure the
accuracy of the models based on how close the predictions are to the ground-
truth targets. In addition, the framework can accommodate more complex metrics
on inventory performance [11], such as the days-of-inventory (DOI) and
stockout rate (SR). The DOI evaluates the average time that the inventory is
stored before sale. The SR measures the ability to satisfy demand based on the
available inventory. These are defined as follows:

$DOI=\frac{\mathit{Avg.\ Inventory\ Cost}}{\mathit{Cost\ of\ Goods\
Sold}}\times\mathit{Days}$ (4) $SR=\frac{\mathit{Unfulfilled\
Demand}}{\mathit{Total\ Demand}}$ (5)

We formulate an inventory simulation to compute these inventory metrics. The
simulation considers the recorded sales at each time step as the optimal
inventory replenishment. Under this condition, the inventory would be consumed
and replenished with the same amount, leaving zero inventory at the end of
every timestep. Thus, the replenishment is instant, similar to [23]. From
here, we can substitute the replenishment with the predicted sales at timestep
while retaining the original sales as the consumption of the inventory. In
this setting, we are able to obtain a more refined insight into how a specific
model performs in a given inventory task. For example, underestimating future
demands would lead to a higher SR, while overestimation would result in a
higher DOI.

We specifically choose the DOI and SR metrics, as they represent a crucial
trade-off in the inventory management setting. Note that minimizing either of
the metrics would lead to a catastrophic impact on the other. For instance,
while minimizing the DOI of the inventory level, there exists a degenerate
solution with zero inventory level at all times. However, doing so would
result in a very high SR. The opposite occurs when we attempt to minimize the
SR. We can maintain the inventory at its maximum capacity to meet every
demand, but doing so would result in an extremely high DOI. Figure 3 provides
a visual illustration of the DOI and SR trade-off, where Figure 3 (a)
represents the ideal trade-off, while Figure 3 (b) and (c) represent the
trivial cases.

Figure 3: Illustration of optimization trade-off between DOI and SR (a) ideal;
(b) low SR; (c) low DOI.

###  3.5 Streamlined AI model framework

Given the model development components, we arrange them into a streamlined
framework, beginning from pre-processing raw datasets, up until the evaluation
metrics. We further optimize the framework to quickly iterate through multiple
training configurations involving various model hyperparameters and feature
sets. This functionality enables three different application scenarios.

The first scenario compares several models over a specific dataset to search
for the model with the highest prediction accuracy. One of the actual
applications of this scenario is our participation in the USAID intelligent
forecasting competition, where we were required to fit and compare as many
models as possible over a single dataset. In Section 5, we discuss the design
choices for our best model. Another scenario is when a new model architecture
is included in the model repository. Before adding such a model to our
collection, we aim to understand its behavior in different settings of the
datasets. In this case, we would obtain a single model fitted over several
datasets. The last scenario is to benchmark different models over different
datasets in an N-model and N-dataset fashion. This scenario is primarily
applicable when writing research papers in which such a comprehensive
benchmark is important to highlight the contribution of the study.

These three different applications of the streamlined framework contribute to
its versatility and applicability in multiple occasions. This versatility
eventually led to the development of new models, which were then added back
into the repository, further enriching our model collection. Therefore,
prototyping solutions for future projects becomes easier because we might
obtain a reasonably adequate baseline model performance by going through the
models in our repository.

##  4 Case study

In July 2020, we participated in an intelligent forecasting competition hosted
by the USAID. The competition aimed to solve global health challenges by
applying accurate forecasting algorithms for predicting the contraceptive
demand and assisting with its distribution in Cote d’Ivoire. The competition
was particularly challenging as it involved sizable time series data from 151
sites and 10 products, meaning that we were required to forecast the demand
for more than one thousand unique time series. Given the monthly recording of
logistics information, our goal was to predict the demand for the next three
months.

To develop our models, we integrated the competition problem into our
streamlined framework. By leveraging the capacities of our framework, we were
able to quickly prototype and identify strong base models to work with. In
Section 5, we discuss the design choices for the proposed solution, beginning
from training procedure, validation strategy, and model selection. We also
included additional experiments to demonstrate the performance of such models
when applied to an inventory management setting.

##  5 Experiments

In this section, our goal is to evaluate the critical decisions that
contributed to building our solution for the USAID competition rather than
describing its exact implementation. For this purpose, we included three
additional time-series datasets, each carrying a unique property of a real-
world scenario. Using these datasets, we first assessed our decision to use a
single model for all available time series (cross-learning) rather than
developing a specific model for each of them. Next, we demonstrated a robust
validation strategy for tuning hyperparameters of the models. Finally, we
evaluated the best models selected via conventional metrics in an inventory
management setting.

Table 1: Brief description of each cleaned dataset included in our experiments. Name | Frequency | Length | #Time-Series
---|---|---|---
USAID Contraceptive | Monthly | 42 | 762
Dairy Products | Weekly | 484 | 5
Walmart Sales | Weekly | 278 | 30
Kaggle Store Demand | Weekly | 261 | 500

###  5.1 Datasets

To investigate the effectiveness of the design choices of our USAID
competition solution, we considered three additional datasets, in addition to
the USAID contraceptive dataset. These datasets are publicly available. The
USAID Contraceptive dataset111https://zindi.africa/competitions/usaids-
intelligent-forecasting-challenge-model-future-contraceptive-use/data [61]
posed a challenging real-world problem because it contains various unique time
series recorded at a monthly frequency. Note that the publicly available USAID
contraceptive dataset is slightly different from the one used in the
competition. The Dairy Products dataset222https://mpr.datamart.ams.usda.gov/
[62] records the weekly consumption of five dairy products in the United
States. This dataset represents a situation in which there are a few time
series, each with a high temporal granularity. We also included the Walmart
Sales dataset333https://www.kaggle.com/c/m5-forecasting-accuracy/data of the
M5 forecasting competition [41, 43]. We aggregated the recorded sales quantity
by product group and site to achieve a dataset containing a sizable number of
unique time series. Next, the Kaggle Store Demand
dataset444https://www.kaggle.com/c/demand-forecasting-kernels-only/data [7, 8]
consists of 500 products with weekly recording frequency, resulting in the
most extensive dataset used in our experimental setting. We reserved three
months of data points from each dataset as a separate-unseen test subset that
is used to evaluate the models. Table 1 provides a detailed description of
each dataset.

###  5.2 Single forecasting model using cross-learning

The challenges of the USAID intelligent forecasting competition were related
to the large number of time series. Although the logistics data were recorded
across four years, from 2016 to 2020, they only have a monthly frequency. This
implies that we were required to handle various unique time series, each of
which is relatively short. This problem became more complicated with missing
values and intermittent demand in large portions of the time-series data.

Table 2: Comparison between NeuralProphet and LightGBM using single and multiple model strategy.  Metric | Model | USAID | Dairy | Walmart | Kaggle
---|---|---|---|---|---
MAE | NeuralProphet | 14.5859 | 5935891.8020 | 809.0128 | 31.5787
LightGBM-Multi | 13.6166 | 5559450.1860 | 734.5936 | 32.2843
LightGBM-Single | 11.3646 | 5742281.9593 | 590.5159 | 30.3952
RMSE | NeuralProphet | 31.7738 | 10784627.0688 | 1516.4113 | 46.6282
LightGBM-Multi | 29.2691 | 9022726.5642 | 1472.2758 | 42.8675
LightGBM-Single | 26.2613 | 9068631.7436 | 1032.0995 | 43.4232
MASE | NeuralProphet | 1.5528 | 0.9199 | 1.2261 | 1.0803
LightGBM-Multi | 1.4205 | 0.9145 | 1.0559 | 1.0863
LightGBM-Single | 1.1616 | 0.9433 | 0.8969 | 1.0136

Statistical forecasting models are generally applied by relying on information
from individual time series. Thus, we tuned multiple models equivalent to the
number of series, further referred to as the multiple-model setup. Performing
hyperparameter tuning on all the models would be infeasible because each set
of hyperparameters needs to be applied across all models. An alternative
solution was to consider AI-based forecasting models similar to the winning
solution of the M5 forecasting competition. Hence, we accommodated several
time series in a single-model setup, also referred to as cross-learning. This
setup offers an immediate benefit where the AI model can learn patterns across
different time series, which was highly suitable for the USAID contraceptive
dataset. We leveraged the number of time series available in the dataset while
removing the complexity of training one model for each. Therefore, iterating
through a significantly larger number of hyperparameters became more
straightforward.

We compared the two setups, single (cross-learning) and multiple models, using
NeuralProphet [57, 59] and LightGBM [33]. NeuralProphet only accepts a single
time series dataset as the input. Therefore, it is limited to the multiple-
model setup. The same inherent constraint would also apply to the statistical
models. By contrast, we implemented both single and multiple model strategies
for LightGBM. Here, LightGBM-Single represents the experimental setting in
which we use the LightGBM with the single-model setup. The LightGBM-Multi
further represents an experiment in which we assign one model for each time
series.

As expected, the multiple-model setup yields more accurate predictions for
datasets containing few, but long time-series data. The multiple-model setup
introduces more model parameters to learn the pattern of each time series,
which eventually translate to better accuracy. However, doing so might lead to
severe overfitting problems when the number of samples is insufficient. This
observation is highlighted by the lower errors in the dairy products dataset,
but higher in the USAID dataset (see Table 2). As also observed by Spiliotis
et al. [53], Bojer & Meldgaard [8], Makridakis et al. [42], the single-model
(i.e., cross-learning) setup enables the algorithm to extract and share
underlying patterns across all available time series in the dataset. This
ability helps the model tackle the main challenges of the USAID dataset
because it compensates the low temporal granularity with the number of time
series.

Table 3: Comparison between different cross-validation strategies on MAE,
RMSE, and MASE.

Metric Model CV USAID Dairy Walmart Kaggle Valid Test Valid Test Valid Test
Valid Test MAE LGBM TS 10.1634 11.8230 7227133.4748 5756410.0923 564.9514
584.4730 35.0562 22.5980 KFold 10.1651 11.8315 7237897.8031 5740567.3344
584.9778 583.0270 35.1731 22.4844 Both 10.1514 11.8093 7207972.2908
5685080.2655 566.4891 575.7160 35.0449 22.4148 LSTM TS 11.8949 13.0825
7552314.1023 6305839.8578 663.6378 710.5549 34.8898 28.9381 KFold 11.9651
12.9743 7609099.4352 6265020.7515 678.8025 709.8739 35.8380 27.7679 Both
11.8843 12.9477 7519657.1878 6214551.6917 656.8424 695.0386 34.4677 27.5839
RMSE LGBM TS 26.2004 26.0029 12255448.2921 8791346.7036 910.2738 1069.3699
46.1781 30.3938 KFold 26.2146 26.0558 12281805.1658 8785695.6910 934.3512
1084.9685 46.3201 30.2436 Both 26.1487 25.9906 12229970.6146 8759418.5526
909.6075 1066.7050 46.1634 30.1608 LSTM TS 27.7264 27.1020 12616295.4649
9822605.7652 1103.1000 1263.4808 46.6570 39.0600 KFold 27.8067 27.0024
12724024.0935 9749545.8742 1134.8565 1251.3391 47.9005 37.4096 Both 27.7019
26.9958 12568829.5047 9697273.7960 1088.5752 1228.7584 46.0689 37.0581 MASE
LGBM TS 0.8698 1.1973 0.8654 1.0141 0.9251 0.8906 1.3546 0.7637 KFold 0.8921
1.2075 0.8663 1.0153 0.9820 0.8679 1.3590 0.7600 Both 0.8646 1.1861 0.8641
1.0020 0.9349 0.8575 1.3544 0.7578 LSTM TS 1.1932 1.3487 0.9066 1.1320 1.0602
1.1276 1.3471 0.9691 KFold 1.2170 1.3055 0.9125 1.1259 1.0848 1.1264 1.3845
0.9308 Both 1.1929 1.3013 0.9031 1.1114 1.0505 1.1032 1.3317 0.9242

Figure 4: Average MASE of cross-validation strategies computed from the top-k
configurations of LightGBM.

###  5.3 Multiple validation strategy

To evaluate the effectiveness of using the multiple cross-validation strategy,
we sampled 3 K configurations for each LightGBM and LSTM, both with the
single-model setup. For brevity, we have omitted the single-model suffix from
the name of the models. Figure 4 provides a quick overview of the prediction
accuracy of the top-k configurations according to MASE because MASE is a scale
independent metric that allows us to compare various datasets. As highlighted
by the performance of LightGBM on the Kaggle dataset, by examining a smaller
number of experiments, the top configurations exhibit a larger performance
variance on the unseen test set. Therefore, it is essential to consider
additional configurations when choosing the best model, as the top models are
sensitive to small changes or even suffer from overfitting issues.

Next, we sliced Figure 4 by observing the top-1000 configurations and
presented the results in Table 3. As shown in the table, using the time series
cross-validation, we can identify configurations that transfer well between
seen validation and the unseen test set. However, since the time series cross-
validation only evaluates the configurations exclusively based on predicting
future values, it fails to generalize across different time series. This
problem is highlighted by the best performance on the validation set, which
does not transfer to the test portion of the Walmart dataset.

Our multiple cross-validation strategy incorporated both time series and
k-fold cross-validation. Therefore, it can robustly identify configurations
that can accurately predict future values while maintaining generalization for
patterns across time series. As a result, we can consistently identify
configurations that adapt well to the test set for all datasets, even when
there exists a significant discrepancy between the validation and test, as
highlighted by the Kaggle dataset (see Table 3). This gap between the
validation and test set is expected, as we only reserve a small number of data
points within the unseen test set.

###  5.4 Assessing forecasting models for inventory management setting

Next, we investigated the performance of the models chosen using conventional
regression metrics when applied to the inventory management setting. To this
end, we applied the DOI and SR inventory metrics (defined in Equations 4 and
5, respectively) in addition to the conventional metrics. As demonstrated in
Table 4, selecting accurate models based on MASE led to a decent trade-off
between DOI and SR. This property is expected as, by providing accurate
predictions, the model performs relatively well on both DOI and SR. Similar
behavior was found with the other conventional metrics (MAE and RMSE), which
were omitted for the sake of brevity.

Although the conventional metrics allow us to choose accurate models, they do
not indicate whether the errors stem from under, or overestimation of future
demands. Moreover, they do not guarantee that the chosen model satisfies the
desired inventory constraints. For example, in the global health setting
represented by the USAID contraceptive dataset, we might favor models with the
best SR to ensure that customers’ demands were met. We can incorporate this
user preference while choosing the best models by leveraging the flexibility
of the framework in applying various evaluation metrics. In summary, training
the models with the conventional metrics would enable them to provide an
accurate estimate, while the DOI and SR would select models with predictions
that favor the inventory setting, rather than pure accuracy.

Table 4: Performance analysis of the best models based on MASE, DOI, and SR.  Dataset | Best of | LightGBM | LSTM
---|---|---|---
MASE | SR (%) | DOI | MASE | SR (%) | DOI
USAID | MASE | 1.1365 | 45.8812 | 28.3914 | 1.2576 | 41.4141 | 39.2569
SR (%) | 1.3384 | 34.1126 | 42.6878 | 1.3571 | 37.6723 | 44.7749
DOI | 1.3859 | 61.6276 | 20.0022 | 1.3994 | 48.9253 | 33.5962
Dairy | MASE | 0.8482 | 3.9102 | 7.2427 | 0.9915 | 3.9031 | 7.3538
SR (%) | 0.9686 | 3.3607 | 7.3982 | 1.2849 | 3.0090 | 7.7407
DOI | 0.9463 | 4.8160 | 7.1397 | 1.2195 | 5.9958 | 7.1614
Walmart | MASE | 0.8207 | 3.4908 | 6.8754 | 0.9750 | 3.5652 | 6.9265
SR (%) | 0.8317 | 3.4210 | 6.8890 | 1.1509 | 2.6013 | 7.1289
DOI | 3.4659 | 21.4416 | 5.5041 | 2.7908 | 17.0041 | 5.8377
Kaggle | MASE | 0.6522 | 3.2703 | 6.9292 | 0.7656 | 3.1037 | 7.0138
SR (%) | 0.7620 | 2.3686 | 7.1183 | 1.4179 | 1.2799 | 7.6287
DOI | 1.5432 | 8.2296 | 6.7172 | 1.5374 | 10.2071 | 6.4397

##  6 Conclusion

We proposed a streamlined framework to facilitate the development of
forecasting models in an agile fashion by streamlining the core components of
the process. The framework is equipped with the functionality of applying
suitable pre-processing; therefore, it can resolve variations in format,
issues, and features into a uniform interface. This uniform interface for all
datasets is proven to be an essential aspect of the framework, as it allows
users to extract better feature representations of time-series data and
enables the same procedure to fit different algorithms. Users can further
iterate through various training configurations, including feature
representations, algorithms, and other hyperparameters. Furthermore, we
included a novel combination of time series and k-fold cross-validation
strategies into the framework to robustly identify configurations with the
finest generalization properties. In addition, the framework integrates custom
evaluation metrics to fit the task at hand. An example of this evaluation
flexibility is the possibility of assessing the forecasting models directly in
an inventory management setting through a simulation process. With these
capabilities, the framework enables users to effortlessly incorporate new
datasets, experiment on different algorithms, and select the best model
configurations using the desired evaluation metrics.

Throughout the experiments, we discussed the design decisions and the
application of our framework on different forecasting datasets. The USAID
forecasting competition is our referent study case. We also demonstrated the
advantages of cross-learning a single model from multiple time series.
Additionally, the comparison of different validation strategies highlighted
that our combination of multiple validations is beneficial for the selection
of the best models. Finally, we evaluated the models with the highest accuracy
using inventory metrics, thus providing insights into their performance in the
inventory management setting, which allowed us to impose preferences on model
selection.

The proposed framework prompts interesting research possibilities. For
example, we could extend the application of 1-model and N-datasets to achieve
better model generalization or even improve the model performance over shorter
time series data. Another interesting direction would be to promote the sample
diversity of an ensemble process considering subsets of the time-series data,
similar to the idea of our adapted k-fold cross-validation.

## References

  * Babai et al. [2019] Babai, M. Z., Dallery, Y., Boubaker, S., & Kalai, R. (2019).  A new method to forecast intermittent demand in the presence of inventory obsolescence.  International Journal of Production Economics, 209, 30–41.
  * Bakhashwain & Sagheer [2021] Bakhashwain, N., & Sagheer, A. (2021).  Online tuning of hyperparameters in deep LSTM for time series applications.  International Journal of Intelligent Engineering and Systems, 14, 212–220.
  * Behl & Dutta [2019] Behl, A., & Dutta, P. (2019).  Humanitarian supply chain management: a thematic literature review and future directions of research.  Annals of Operations Research, 283, 1001–1044.
  * Bergmeir & Benítez [2012] Bergmeir, C., & Benítez, J. M. (2012).  On the use of cross-validation for time series predictor evaluation.  Information Sciences, 191, 192–213.
  * Bergmeir et al. [2018] Bergmeir, C., Hyndman, R. J., & Koo, B. (2018).  A note on the validity of cross-validation for evaluating autoregressive time series prediction.  Computational Statistics & Data Analysis, 120, 70–83.
  * Bhaskar et al. [2020] Bhaskar, S., Tan, J., Bogers, M. L., Minssen, T., Badaruddin, H., Israeli-Korn, S., & Chesbrough, H. (2020).  At the epicenter of covid-19–the tragic failure of the global supply chain for medical supplies.  Frontiers in public health, 8, 821.
  * Bojer & Meldgaard [2018] [dataset] Bojer, C. S., & Meldgaard, J. P. (2018).  Kaggle Store Demand dataset.  Kaggle repository.  URL: https://www.kaggle.com/c/demand-forecasting-kernels-only/data (Accessed on 10 December 2021).
  * Bojer & Meldgaard [2021] Bojer, C. S., & Meldgaard, J. P. (2021).  Kaggle forecasting competitions: An overlooked learning opportunity.  International Journal of Forecasting, 37, 587–603.
  * Box et al. [2015] Box, G. E., Jenkins, G. M., Reinsel, G. C., & Ljung, G. M. (2015).  Time series analysis: forecasting and control.  John Wiley & Sons.
  * Burba et al. [2021] Burba, D., Soeseno, J. H., & Chen, T. C. (2021).  Forecasting with artificial intelligence: From manufacturing logistics management to global health forecasting.  In International Workshop on Forecasting for Social Good (F4SG 2021).
  * Cannon [2008] Cannon, A. R. (2008).  Inventory improvement and financial performance.  International Journal of Production Economics, 115, 581–593.
  * Cawley & Talbot [2010] Cawley, G. C., & Talbot, N. L. (2010).  On over-fitting in model selection and subsequent selection bias in performance evaluation.  The Journal of Machine Learning Research, 11, 2079–2107.
  * Cerqueira et al. [2020] Cerqueira, V., Torgo, L., & Mozetič, I. (2020).  Evaluating time series forecasting models: An empirical study on performance estimation methods.  Machine Learning, 109, 1997–2028.
  * Croston [1972] Croston, J. D. (1972).  Forecasting and stock control for intermittent demands.  Journal of the Operational Research Society, 23, 289–303.
  * Friedman [2001] Friedman, J. H. (2001).  Greedy function approximation: a gradient boosting machine.  Annals of statistics, (pp. 1189–1232).
  * Friedman [2002] Friedman, J. H. (2002).  Stochastic Gradient Boosting.  Computional Statistics & Data Analysis, 38, 367–378.
  * Gardner Jr [1985] Gardner Jr, E. S. (1985).  Exponential smoothing: The state of the art.  Journal of forecasting, 4, 1–28.
  * Gardner Jr [2006] Gardner Jr, E. S. (2006).  Exponential smoothing: The state of the art—part ii.  International journal of forecasting, 22, 637–666.
  * Gilliland [2020] Gilliland, M. (2020).  The value added by machine learning approaches in forecasting.  International Journal of Forecasting, 36, 161–166.
  * Goltsos et al. [2021] Goltsos, T. E., Syntetos, A. A., Glock, C. H., & Ioannou, G. (2021).  Inventory–forecasting: mind the gap.  European Journal of Operational Research, .
  * González et al. [2020] González, S., García, S., Del Ser, J., Rokach, L., & Herrera, F. (2020).  A practical tutorial on bagging and boosting based ensembles for machine learning: Algorithms, software tools, performance study, practical perspectives and opportunities.  Information Fusion, 64, 205–237.
  * Goodfellow et al. [2016] Goodfellow, I., Bengio, Y., & Courville, A. (2016).  Deep learning.  MIT press.
  * Guo et al. [2014] Guo, X., Liu, C., Xu, W., Yuan, H., & Wang, M. (2014).  A prediction-based inventory optimization using data mining models.  In 2014 Seventh International Joint Conference on Computational Sciences and Optimization (pp. 611–615).  IEEE.
  * Guyon et al. [2008] Guyon, I., Gunn, S., Nikravesh, M., & Zadeh, L. A. (2008).  Feature extraction: foundations and applications volume 207.  Springer.
  * Hochreiter & Schmidhuber [1997] Hochreiter, S., & Schmidhuber, J. (1997).  Long short-term memory.  Neural computation, 9, 1735–1780.
  * Hong et al. [2019] Hong, T., Xie, J., & Black, J. (2019).  Global energy forecasting competition 2017: Hierarchical probabilistic load forecasting.  International Journal of Forecasting, 35, 1389–1399.
  * Huber et al. [2019] Huber, J., Müller, S., Fleischmann, M., & Stuckenschmidt, H. (2019).  A data-driven newsvendor problem: From data to decision.  European Journal of Operational Research, 278, 904–915.
  * Hutter et al. [2019] Hutter, F., Kotthoff, L., & Vanschoren, J. (2019).  Automated Machine Learning.  Springer.
  * Hyndman [2020] Hyndman, R. J. (2020).  A brief history of forecasting competitions.  International Journal of Forecasting, 36, 7–14.
  * Hyndman & Athanasopoulos [2018] Hyndman, R. J., & Athanasopoulos, G. (2018).  Forecasting: principles and practice.  Melbourne, Australia: OTexts.
  * Hyndman & Koehler [2006] Hyndman, R. J., & Koehler, A. B. (2006).  Another look at measures of forecast accuracy.  International journal of forecasting, 22, 679–688.
  * Jordan & Mitchell [2015] Jordan, M. I., & Mitchell, T. M. (2015).  Machine learning: Trends, perspectives, and prospects.  Science, 349, 255–260.
  * Ke et al. [2017] Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., & Liu, T.-Y. (2017).  Lightgbm: A highly efficient gradient boosting decision tree.  Advances in neural information processing systems, 30, 3146–3154.
  * Kourentzes et al. [2021] Kourentzes, N., Svetunkov, I., & Trapero, J. R. (2021).  Connecting forecasting and inventory performance: a complex task.  Available at SSRN 3878176, .
  * Kourentzes et al. [2020] Kourentzes, N., Trapero, J. R., & Barrow, D. K. (2020).  Optimising forecasting models for inventory planning.  International Journal of Production Economics, 225, 107597.
  * Koutsandreas et al. [2021] Koutsandreas, D., Spiliotis, E., Petropoulos, F., & Assimakopoulos, V. (2021).  On the selection of forecasting accuracy measures.  Journal of the Operational Research Society, (pp. 1–18).
  * Van der Laan et al. [2016] Van der Laan, E., van Dalen, J., Rohrmoser, M., & Simpson, R. (2016).  Demand forecasting and order planning for humanitarian logistics: An empirical assessment.  Journal of Operations Management, 45, 114–122.
  * Lim & Zohren [2021] Lim, B., & Zohren, S. (2021).  Time-series forecasting with deep learning: a survey.  Philosophical Transactions of the Royal Society A, 379, 20200209.
  * Makridakis et al. [2020a] Makridakis, S., Hyndman, R. J., & Petropoulos, F. (2020a).  Forecasting in social settings: The state of the art.  International Journal of Forecasting, 36, 15–28.
  * Makridakis et al. [2018] Makridakis, S., Spiliotis, E., & Assimakopoulos, V. (2018).  Statistical and machine learning forecasting methods: Concerns and ways forward.  PloS one, 13, e0194889.
  * Makridakis et al. [2020b] [dataset] Makridakis, S., Spiliotis, E., & Assimakopoulos, V. (2020b).  Walmart Sales dataset from M5 forecating accuracy competition.  Kaggle repository.  URL: https://www.kaggle.com/c/m5-forecasting-accuracy/data (Accessed on 10 December 2021).
  * Makridakis et al. [2021a] Makridakis, S., Spiliotis, E., & Assimakopoulos, V. (2021a).  The M5 accuracy competition: Results, findings and conclusions.  International Journal of Forecasting, .
  * Makridakis et al. [2021b] Makridakis, S., Spiliotis, E., & Assimakopoulos, V. (2021b).  The M5 competition: Background, organization, and implementation.  International Journal of Forecasting, .
  * Montero-Manso et al. [2020] Montero-Manso, P., Athanasopoulos, G., Hyndman, R. J., & Talagala, T. S. (2020).  FFORMA: Feature-based forecast model averaging.  International Journal of Forecasting, 36, 86–92.
  * Petropoulos et al. [2018] Petropoulos, F., Kourentzes, N., Nikolopoulos, K., & Siemsen, E. (2018).  Judgmental selection of forecasting models.  Journal of Operations Management, 60, 34–46.
  * Prak & Teunter [2019] Prak, D., & Teunter, R. (2019).  A general method for addressing forecasting uncertainty in inventory models.  International Journal of Forecasting, 35, 224–238.
  * Probst et al. [2019] Probst, P., Boulesteix, A.-L., & Bischl, B. (2019).  Tunability: importance of hyperparameters of machine learning algorithms.  The Journal of Machine Learning Research, 20, 1934–1965.
  * Punia et al. [2020] Punia, S., Singh, S. P., & Madaan, J. K. (2020).  From predictive to prescriptive analytics: A data-driven multi-item newsvendor model.  Decision Support Systems, 136, 113340.
  * Rostami-Tabar et al. [2021] Rostami-Tabar, B., Ali, M. M., Hong, T., Hyndman, R. J., Porter, M. D., & Syntetos, A. (2021).  Forecasting for social good.  International Journal of Forecasting, .
  * Semenoglou et al. [2021] Semenoglou, A.-A., Spiliotis, E., Makridakis, S., & Assimakopoulos, V. (2021).  Investigating the accuracy of cross-learning time series forecasting methods.  International Journal of Forecasting, 37, 1072–1084.
  * Silver et al. [2016] Silver, E. A., Pyke, D. F., & Thomas, D. J. (2016).  Inventory and production management in supply chains.  CRC Press.
  * Smyl [2020] Smyl, S. (2020).  A hybrid method of exponential smoothing and recurrent neural networks for time series forecasting.  International Journal of Forecasting, 36, 75–85.
  * Spiliotis et al. [2020] Spiliotis, E., Makridakis, S., Semenoglou, A.-A., & Assimakopoulos, V. (2020).  Comparison of statistical and machine learning methods for daily sku demand forecasting.  Operational Research, (pp. 1–25).
  * Syntetos et al. [2016] Syntetos, A. A., Babai, Z., Boylan, J. E., Kolassa, S., & Nikolopoulos, K. (2016).  Supply chain forecasting: Theory, practice, their gap and the future.  European Journal of Operational Research, 252, 1–26.
  * Syntetos et al. [2009] Syntetos, A. A., Boylan, J. E., & Disney, S. M. (2009).  Forecasting for inventory planning: a 50-year review.  Journal of the Operational Research Society, 60, S149–S160.
  * Tashman [2000] Tashman, L. J. (2000).  Out-of-sample tests of forecasting accuracy: an analysis and review.  International journal of forecasting, 16, 437–450.
  * Taylor & Letham [2018] Taylor, S. J., & Letham, B. (2018).  Forecasting at scale.  The American Statistician, 72, 37–45.
  * Triebe [2021] Triebe, O. (2021).  NeuralProphet GitHub.  URL: https://github.com/ourownstory/neural_prophet (Accessed on 27 September 2021).
  * Triebe et al. [2020] Triebe, O., Hewamalage, H., Laptev, N., Lima, I., & Rajagopal, R. (2020).  NeuralProphet: A scalable and extensible time-series forecasting package.  In 40th International Symposium on Forecasting (ISF 2020).
  * Triebe et al. [2019] Triebe, O., Laptev, N., & Rajagopal, R. (2019).  AR-Net: A simple auto-regressive neural network for time-series.  arXiv preprint arXiv:1911.12436, .
  * United States Agency of International Development [2020] [dataset] United States Agency of International Development (2020).  USAID Contraceptive dataset.  Zindi repository.  URL: https://zindi.africa/competitions/usaids-intelligent-forecasting-challenge-model-future-contraceptive-use/data (Accessed on 10 December 2021).
  * United States Department of Agriculture [2020] [dataset] United States Department of Agriculture (2020).  Dairy Products dataset.  USDA datamart repository.  URL: https://mpr.datamart.ams.usda.gov/ (Accessed on 10 December 2021).
  * Zöller & Huber [2021] Zöller, M.-A., & Huber, M. F. (2021).  Benchmark and survey of automated machine learning frameworks.  Journal of Artificial Intelligence Research, 70, 409–472.


# Age Prediction From Face Images Via Contrastive Learning

Yeongnam Chae Poulami Raha Mijung Kim Björn Stenger
Rakuten Institute of Technology  Rakuten Group  Inc.
{yeongnam.chae, poulami.raha, mijung.a.kim<EMAIL_ADDRESS>
## Abstract

This paper presents a novel approach for accurately estimating age from face
images, which overcomes the challenge of collecting a large dataset of
individuals with the same identity at different ages. Instead, we leverage
readily available face datasets of different people at different ages and aim
to extract age-related features using contrastive learning. Our method
emphasizes these relevant features while suppressing identity-related features
using a combination of cosine similarity and triplet margin losses. We
demonstrate the effectiveness of our proposed approach by achieving state-of-
the-art performance on two public datasets, FG-NET and MORPH II.

##  1 Introduction

Age estimation from facial images has found application in various fields.
However, the different attributes in a face image present distinct visual
features that can be challenging to disentangle. While facial feature
extractors have been pre-trained for specific tasks such as face recognition,
they are not well-suited for tasks like age estimation. Collecting a large
dataset of face images of the same individuals at different ages is more
difficult than collecting a dataset of different individuals. Therefore, the
key question is how to develop a method for learning aging-related features
that are not influenced by identity-related features.

Face age estimation methods based on convolutional neural networks have made
significant progress and can be grouped into three categories: classification
[9, 20, 34, 42], regression [12, 13, 14, 25, 30] and ranking [5, 6, 7, 8]
approaches. Recently, self-supervised [4] and attention-based approaches [39]
have been proposed. However, most of these techniques have relied on
information from individual face images, causing the model to be biased
towards features associated with attributes like identity, which hinders the
model from focusing on relevant but sparse features related to age, such as
small wrinkles or skin texture. Moreover, some studies have explored
comparative approaches [1, 18, 22, 32], which aim to learn ranking or
transformation information based on relative age differences. In contrast, the
proposed approach mainly focuses on how to emphasize sparser features related
to age while penalizing non-relevant features related to identity through
contrasting the features extracted from images from the same age group.

Figure 1: Contrastive Learning for Age Estimation using cosine similarity loss
for positive samples with the same age as the anchor image and triplet margin
loss for negative samples with a different age.

We present a new approach for face age estimation that leverages contrastive
learning. Our method aims to suppress identity-related features while
emphasizing age-related ones. To learn identity-independent age features, we
use triplets of images. Given an anchor face image, we sample two images - one
of the same age (positive sample) but with a different identity, and one of a
different age (negative sample). By comparing the anchor image to both the
positive and negative samples, we jointly minimize a cosine similarity loss
and a triplet margin loss, see Figure 1. Owing to the large number of possible
triplet samples, the method is data-efficient and able to learn age prediction
from small datasets without resorting to additional data sources, as shown in
the experiments on public datasets.

##  2 Method

We use contrastive learning to suppress identity-related features in order to
compare face images with the same age but different identity. We extract the
facial feature vector from an anchor image, ${\bf x}_{a}$, and a positive
sample, ${\bf x}_{p}$. To ensure that the positive sample ${\bf x}_{p}$ of the
given anchor ${\bf x}_{a}$ is free from identity-related features, we select
it from a set of face images ${\cal P}({\bf x}_{a})$ that have the same age
but different identity labels:

${\cal P}({\bf x}_{a})=\\{{\bf
x}_{p}~{}|~{}y_{p}=y_{a}\;\mathrm{and}\;i_{p}\neq i_{a}\\},$ (1)

where $i_{a}$ and $i_{p}$ are the identity labels and $y_{a}$ and
$y_{p}\in\\{1,2,...,A\\}$ are age labels of the anchor and the positive
sample, respectively. $A$ is the maximum age label in a dataset. We extract
ResNet-18 features ${\bf f}_{a}=R({\bf x}_{a})$, ${\bf f}_{p}=R({\bf x}_{p})$,
where ${\bf f}\in\mathbb{R}^{D}$ are features before the last fully connected
(FC) layer. $D=512$ is the dimension of the feature vector. We compute the
cosine similarity loss $L_{c}$ [29] between features ${\bf f}_{a}$ and ${\bf
f}_{p}$:

$L_{c}(\mathbf{f}_{a},\mathbf{f}_{p})=\frac{{\mathbf{f}_{a}}^{T}\mathbf{f}_{p}}{\|\mathbf{f}_{a}\|\|\mathbf{f}_{p}\|}.$
(2)

The cosine distance has been widely used to compare face images or features
[10, 17, 19, 24, 27, 40].

The probabilities computed in the final softmax output of the network for the
anchor image define a distribution over age values. Let $s_{a,j}$ denote the
probability that sample ${\bf x}_{a}$ has age label $j$. We compute a mean
loss, $L_{m}$, and a variance loss, $L_{v}$ [31]:

$L_{m}(\mathbf{s}_{a})=\sum_{j=1}^{A}{js_{a,j}}-y_{a}\;,$ (3)
$L_{v}(\mathbf{s}_{a})=\sum_{j=1}^{A}{s_{a,j}}\left(j-\sum_{k=1}^{A}{k\,s_{a,k}}\right)^{2}\;.$
(4)

### Ternary contrast via triplet margin loss.

To use contrastive learning for different ages, we adopt a triplet margin loss
[37]. The triplet margin loss is a loss function where the anchor image is
compared with a positive and a negative sample. The distance from the anchor
to the positive sample is minimized, while the distance to the negative sample
is maximized. We use a negative sample ${\bf x}_{n}$ with a different age and
identity to contrast with the positive sample and the anchor.

${\cal N}({\bf x}_{a})=\\{{\bf x}_{n}~{}|~{}y_{n}\neq
y_{a}\;\textrm{and}\;i_{n}\neq i_{a}\\}$ (5)

The triplet loss is applied to the softmax probability, $\mathbf{s}$ as
follows:

$L_{t}(\mathbf{s}_{a},\mathbf{s}_{p},\mathbf{s}_{n})=\max(\|\mathbf{s}_{a}-\mathbf{s}_{p}\|^{2}-\|\mathbf{s}_{a}-\mathbf{s}_{n}\|^{2}+\alpha,0),$
(6)

where $\mathbf{s}_{n}$ is softmax probability of the negative image and
$\alpha$ is a margin between positive and negative samples. The overall loss
function is defined as:

$L_{\textrm{total}}=L_{s}+\lambda_{m}L_{m}+\lambda_{v}L_{v}+\lambda_{c}L_{c}+\lambda_{t}L_{t}\;,$
(7)

where $L_{s}$ denotes the softmax loss, $\lambda$ denote hyper-parameters that
balance the influence of each loss term. We empirically set $\lambda_{m}$ to
0.2 and $\lambda_{v}$ to 0.05. We experimentally evaluate the values of the
binary and ternary loss terms, $\lambda_{c}$ and $\lambda_{t}$.

##  3 Experiments

### Datasets & protocols.

The MORPH II dataset is a face dataset, containing 55,134 images of 13,618
individuals. Ages range from 16 to 77 with a median age of 33. To be
consistent with prior work, the five-fold random split (RS) and five-fold
subject exclusive (SE) protocols are used in the experiments [35].

The FG-NET dataset contains 1,002 face images from 82 individuals with ages
ranging from 0 to 69 years [33]. We evaluate using the commonly used leave-
one-person-out (LOPO) protocol. Table 1 shows the age distributions in the
MORPH II and FG-NET datasets.

Table 1: Age distributions in MORPH II and FG-NET. Dataset | 0-19 | 20-39 | 40-59 |  $>$60
---|---|---|---|---
MORPH II | 7,469 | 31,682 | 15,649 | 334
FG-NET | 710 | 223 | 61 | 8

### Implementation details & evaluation metric.

We first perform facial alignment of all images using five landmarks detected
with MTCNN [44]. These aligned face images are then normalized to a size of
$256\times 256\times 3$. Subsequently, we extract features from the normalized
face images by utilizing a ResNet-18 model that has been pre-trained on
ImageNet. We apply a series of augmentation techniques, including random
affine transformations with slight variations, random vertical flips, and
random crops to $224\times 224$ during the training phase. We optimize the
model parameters using the Adam optimizer with an initial learning rate of
0.001 and train for 100 epochs, using a batch size of 64. As error metric we
use the mean absolute error (MAE), which is defined as the L1 distance between
the predicted age, $\widehat{y}_{i}$, of image ${\bf x}_{i}$ and its ground-
truth age, $y_{i}$:

$\mathrm{MAE}=\frac{1}{N}\sum_{i=1}^{N}{|\widehat{y}_{i}-y_{i}|}\;.$ (8)

Table 2: Evaluation on MORPH II. Mean Absolute Error (MAE) comparison following the RS evaluation protocol. RS(I) indicates pre-training on ImageNet, RS(F) on other large face datasets, including IMDB-WIKI [36], MSCeleb(†) [15], and FaceAugmentation(‡) [26]. Parameters: (binary) $\lambda_{c}=10$, $\lambda_{t}=0$ (ternary) $\lambda_{c}$ = 10, $\lambda_{t}=1$.  Method | RS(I) | RS(F) | Year
---|---|---|---
CRCNN[1] | 3.74 |  | 2016
OR-CNN[30] | 3.27 |  | 2016
DEX[36] | 3.25 | 2.68 | 2016
ODFL[23] | 3.12 |  | 2017
ARN[3] | 3.00 |  | 2017
Ranking-CNN[8] | 2.96 |  | 2017
AP[45] | 2.87 | 2.52 | 2017
M-LSDML[22] |  | 2.89‡ | 2018
RCL[32] |  | 2.46 | 2018
SVRT[18] |  | 2.38† | 2018
MV[31] | 2.41 | 2.16 | 2018
C3AE[43] | 2.78 | 2.75 | 2019
BridgeNet[21] |  | 2.38 | 2019
ADVL[41] |  | 1.94 | 2020
NRLD[11] | 2.35 | 1.81 | 2020
OCCO[4] | 2.29 |  | 2021
ADPF[39] | 2.54 |  | 2022
Ours (binary loss) | 2.14 |  |
Ours (ternary loss) | 2.20 |  |
Table 3:  Evaluation on MORPH II. Mean Absolute Error (MAE) comparison following the SE evaluation protocol. Methods | SE(I) | SE(F) | Year
---|---|---|---
DIF[16] | 3.00 |  | 2018
RCL[32] |  | 2.88 | 2018
SVRT[18] |  | 2.87† | 2018
MV[31] | 2.80 | 2.79 | 2018
Ours (binary loss) | 2.43 |  |
Ours (ternary loss) | 2.37 |  |
Table 4: Evaluation on FG-NET in terms of Mean Absolute Error (MAE) following the LOPO evaluation protocal. LOPO(I) indicates pre-training on ImageNet, LOPO(F) on the large IMDB-WIKI [36] dataset. Parameters: (binary) $\lambda_{c}=1$, $\lambda_{t}=0$ (ternary) $\lambda_{c}$ = 1, $\lambda_{t}=1$ Methods | LOPO(I) | LOPO(F) | Year
---|---|---|---
DEX[36] | 4.63 | 3.09 | 2016
CRCNN[1] | 4.13 |  | 2016
RCL[32] |  | 4.21 | 2018
MV[31] | 4.10 | 2.68 | 2018
C3AE[43] | 4.09 | 2.95 | 2019
BridgeNet[21] |  | 2.56 | 2019
NRLD[11] | 3.23 | 2.55 | 2020
AVDL[41] |  | 2.32 | 2020
ADPF[39] | 2.86 |  | 2022
Ours (binary loss) | 2.42 |  |
Ours (ternary loss) | 2.31 |  |

(a) FG-NET (b) MORPH II (c) Grad-CAM Visualizations

Figure 2:  Example age prediction results on (a) FG-NET and (b) MORPH II.
Numbers below each image show predicted and ground truth age. Top two rows
show accurate estimation results, the bottom row shows poor estimates. (c) A
Grad-CAM analysis on MOPRH II samples shows that our method better focuses on
features extracted from the face region compared to the Mean Variance method
[31].

### Results on MORPH II.

Table 2 shows the results following the RS protocol. The proposed model
achieves an MAE of 2.14 when using binary contrast ($\lambda_{c}=10$,
$\lambda_{t}=0$). It performs best among all approaches that do not use
external data for pre-training. Several approaches employ model pre-trained on
large face datasets like IMDB-WIKI [36], MS Celeb 1M [15], or FaceAugmentation
[26]. Results following the SE evaluation protocol on the MORPH II dataset are
shown in Table 3. The proposed model achieves an MAE of 2.37 when using
ternary contrast ($\lambda_{c}$ = 10, $\lambda_{t}=1$). In the SE protocol,
images of individuals who appear in the training set are excluded from the
test set. We observe that in this case the ternary loss improves the
performance. The coefficients of the individual loss terms are discussed in
the ablation study.

### Results on FG-NET.

As shown in Table 4, our method shows good performance on the FG-NET dataset.
We observe that our model shows competitive performance with models that use
an external dataset, IMDB-WIKI [36]. By sampling triplets of face images, the
proposed method achieves excellent performance on FG-NET without any
additional data.

Examples of age estimation results on FG-NET and MORPH II are shown in Fig. 2.
The proposed method performs robustly for various age ranges. Poor estimates
are typically caused by poor image quality.

### Identity invariance.

To measure the dependence on face identiy, we compare the feature variance
when keeping the identity fixed. A low variance indicates a larger dependency
on identiy and vice versa. We calculate the mean variance of the extracted
feature $\mathbf{f}$ and $\mathbf{s}$ by identity and compare it with the Mean
Variance (MV) method [31] on FG-NET and MORPH II datasets. As shown in Table
5, the feature extracted by the proposed method show higher variance than the
MV method for the same identity.

Table 5: Mean variance for fixed identity. Dataset | Methods | MV | Ours
---|---|---|---
FG-NET | $\mu(V_{\mathbf{f}})$ | 0.18 | 6.16
FG-NET | $\mu(V_{\mathbf{s}})$ | 36.03 | 43.71
MORPH II | $\mu(V_{\mathbf{f}})$ | 0.18 | 7.22
MORPH II | $\mu(V_{\mathbf{s}})$ | 35.87 | 44.44

For the qualitative study, a Grad-CAM [38] comparison between the MV method
[31] and the proposed method on the MORPH II dataset is shown in Fig. 2. MV
focuses on the entire face area, whereas the proposed model concentrates
mainly on forehead regions for people in their 30s and 40s. In case of teens
and 20s, it focuses on areas around the nose and mouth. The eye related
features like eye shape, eye size and eyebrow shape are reported as
distinctive feature for face identification than other facial features [2]. On
the other hand, features related to wrinkles around the forehead, nose and
mouth region are important features for age estimation [28]. As shown in Fig.
2, the proposed method makes model less dependent on the identity related
features and emphasize the features related to the age.

### Ablation study on loss functions.

To evaluate the combination of different loss functions we study binary and
ternary contrast by including or excluding the triplet margin loss. In
addition, we compare cosine similarity with a Kullback-Leibler (KL) divergence
loss. Cosine similarity is a common to measure of feature similarity while the
KLD loss is used for continuous distributions. We adopt the KLD loss on the
softmax probabilities for anchor and positive samples:

$L_{\mathrm{KLD}}(\mathbf{s}_{a},\mathbf{s}_{p})=\frac{1}{A}\sum_{j=1}^{A}{\mathbf{s}_{p,j}(\log{\mathbf{s}_{p,j}}-\mathbf{s}_{a,j})}\;.$
(9)

The MAE results for different losses are shown in Table 6. For both binary and
ternary contrast, the KLD loss performs worse than the cosine similarity loss.
The triplet margin loss improves the performance on small datasets such as FG-
NET. The cosine similarity loss improves the accuracy when there is sufficient
training data such as in the MORPH II dataset. Overall, cosine similarity loss
with triplet margin loss shows the best performance for both small and large
datasets.

Figure 3: Ablation on loss weights. MAE on MORPH II (SE) for varying
$\lambda_{c}$ and $\lambda_{t}$. (left) varying $\lambda_{c}$ with
$\lambda_{t}=0$, (right) varying $\lambda_{t}$ with $\lambda_{c}=8$ or $10$.

### Loss coefficients.

In order to select weights for the individual loss terms, we evaluate
different weight combinations. We fix coefficients $\lambda_{m}$,
$\lambda_{v}$ for the mean and variance terms to 0.2 and 0.05, respectively.
We change the coefficient of the cosine similarity loss, $\lambda_{c}$ without
negative channel and the triplet margin loss (i.e. $\lambda_{t}=0$). We change
it from 0 to 10 because an absolute scale of the cosine similarity loss is ten
times smaller than either mean loss or variance loss. We measure the MAE with
the SE protocol on the MORPH II dataset, see Fig. 3 (a). We observe that
cosine similarity shows good performance for values $\lambda_{c}=8$ and
$\lambda_{c}=10$. We vary the triplet loss coefficient, $\lambda_{t}$ from 0
to 5. We measure the MAE on MORPH II (SE protocol). As shown in Fig. 3 (b),
the cosine similarity loss is a poor choice ($\lambda_{t}=0$). Compared with
the triplet margin loss, the cosine similarity loss has a relatively larger
influence on the performance of the model. The minimum MAE is obtained for
$\lambda_{t}=1$ and $\lambda_{c}=10$.

Table 6: Ablation study. MAE for different loss terms. Best and second-best
results are shown in bold and underlined.

Loss | FG-NET | MORPH II | MORPH II
---|---|---|---
(LOPO) | (RS) | (SE)
MV | 4.10 | 2.41 | 2.80
MV + KLD | 2.43 | 2.34 | 2.85
MV + Cosine | 2.42 | 2.19 | 2.54
MV + Triplet | 2.30 | 2.49 | 2.88
MV + KLD + Triplet | 2.33 | 3.13 | 2.85
MV + Cosine + Triplet | 2.31 | 2.24 | 2.50

##  4 Conclusion

We introduced a method for age estimation from face images via contrastive
learning from triplets of face images. Our proposed approach focuses on
sparser features that are more relevant to age by penalizing non-relevant
features that are associated with identity. To encourage the similarity of
positive samples, we leverage cosine similarity, and we employ a ternary loss
to increase the distance to negative samples. Experiments on the MORPH II and
FG-NET datasets demonstrated the effectiveness of our proposed method, which
achieved state-of-the-art results.

## References

  * [1] Fatma S Abousaleh, Tekoing Lim, Wen-Huang Cheng, Neng-Hao Yu, M Anwar Hossain, and Mohammed F Alhamid.  A novel comparative deep learning framework for facial age estimation.  EURASIP Journal on Image and Video Processing, 2016(1):1–13, 2016\.
  * [2] Naphtali Abudarham and Galit Yovel.  Reverse engineering the face space: Discovering the critical features for face identification.  Journal of Vision, 16(3), 2016.
  * [3] Eirikur Agustsson, Radu Timofte, and Luc Van Gool.  Anchored regression networks applied to age estimation and super resolution.  In ICCV, pages 1652–1661, 2017.
  * [4] Weiwei Cai and Hao Liu.  Occlusion contrasts for self-supervised facial age estimation.  In Multimedia Understanding with Less Labeling on Multimedia Understanding with Less Labeling, pages 1–7. 2021.
  * [5] Kuang-Yu Chang and Chu-Song Chen.  A learning framework for age rank estimation based on face images with scattering transform.  IEEE Transactions on Image Processing, 24(3):785–798, 2015.
  * [6] Kuang-Yu Chang, Chu-Song Chen, and Yi-Ping Hung.  A ranking approach for human ages estimation based on face images.  In 2010 20th International Conference on Pattern Recognition, pages 3396–3399, 2010.
  * [7] Kuang-Yu Chang, Chu-Song Chen, and Yi-Ping Hung.  Ordinal hyperplanes ranker with cost sensitivities for age estimation.  In CVPR 2011, pages 585–592, 2011.
  * [8] Shixing Chen, Caojin Zhang, Ming Dong, Jialiang Le, and Mike Rao.  Using ranking-cnn for age estimation.  In CVPR, pages 742–751, 2017.
  * [9] Mohammad Mahdi Dehshibi and Azam Bastanfard.  A new algorithm for age recognition from facial images.  Signal Processing, 90(8):2431–2444, 2010.
  * [10] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou.  Arcface: Additive angular margin loss for deep face recognition.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4690–4699, 2019.
  * [11] Zongyong Deng, Mo Zhao, Hao Liu, Zhenhua Yu, and Feng Feng.  Learning neighborhood-reasoning label distribution (nrld) for facial age estimation.  In IEEE Int. Conf. Multimedia and Expo (ICME), pages 1–6, 2020\.
  * [12] F. Dornaika, SE. Bekhouche, and I. Arganda-Carreras.  Robust regression with deep cnns for facial age estimation: An empirical study.  Expert Systems with Applications, 141:112942, 2020.
  * [13] Yun Fu and Thomas S. Huang.  Human age estimation with regression on discriminative aging manifold.  IEEE Transactions on Multimedia, 10(4):578–584, 2008.
  * [14] Guodong Guo, Yun Fu, Charles R. Dyer, and Thomas S. Huang.  Image-based human age estimation by manifold learning and locally adjusted robust regression.  IEEE TIP, 17(7):1178–1188, 2008.
  * [15] Yandong Guo, Lei Zhang, Yuxiao Hu, Xiaodong He, and Jianfeng Gao.  Ms-celeb-1m: A dataset and benchmark for large-scale face recognition.  In European conference on computer vision, pages 87–102. Springer, 2016.
  * [16] Hu Han, Anil K. Jain, Fang Wang, S. Shan, and Xilin Chen.  Heterogeneous face attribute estimation: A deep multi-task learning approach.  IEEE TPAMI, 40:2597–2609, 2018.
  * [17] Yuge Huang, Yuhan Wang, Ying Tai, Xiaoming Liu, Pengcheng Shen, Shaoxin Li, Jilin Li, and Feiyue Huang.  Curricularface: adaptive curriculum learning loss for deep face recognition.  In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5901–5910, 2020.
  * [18] Woobin Im, Sungeun Hong, Sung-Eui Yoon, and Hyun S Yang.  Scale-varying triplet ranking with classification loss for facial age estimation.  In Asian Conference on Computer Vision, pages 247–259. Springer, 2018.
  * [19] Minchul Kim, Anil K Jain, and Xiaoming Liu.  Adaface: Quality adaptive margin for face recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18750–18759, 2022.
  * [20] A. Lanitis, C. Draganova, and C. Christodoulou.  Comparing different classifiers for automatic age estimation.  IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 34(1):621–628, 2004.
  * [21] Wanhua Li, Jiwen Lu, Jianjiang Feng, Chunjing Xu, Jie Zhou, and Qi Tian.  Bridgenet: A continuity-aware probabilistic network for age estimation.  In CVPR, pages 1145–1154, 2019.
  * [22] Hao Liu, Jiwen Lu, Jianjiang Feng, and Jie Zhou.  Label-sensitive deep metric learning for facial age estimation.  IEEE Transactions on Information Forensics and Security, 13(2):292–305, 2017.
  * [23] Hao Liu, Jiwen Lu, Jianjiang Feng, and Jie Zhou.  Ordinal deep feature learning for facial age estimation.  In IEEE Int. Conf. Automatic Face and Gesture Recognition, pages 157–164, 2017.
  * [24] Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song.  Sphereface: Deep hypersphere embedding for face recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 212–220, 2017.
  * [25] Yangjing Long.  Human age estimation by metric learning for regression problems.  In Int. Conf. Computer Graphics, Imaging and Visualization, pages 343–348, 2009.
  * [26] Iacopo Masi, Anh Tu an Trãn, Tal Hassner, Jatuporn Toy Leksut, and Gérard Medioni.  Do we really need to collect millions of faces for effective face recognition?  In European Conference on Computer Vision (ECCV), October 2016.
  * [27] Qiang Meng, Shichao Zhao, Zhida Huang, and Feng Zhou.  Magface: A universal representation for face recognition and quality assessment.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14225–14234, 2021.
  * [28] Choon-Ching Ng, Moi Hoon Yap, Nicholas Costen, and Baihua Li.  Will wrinkle estimate the face age?  In 2015 IEEE International Conference on Systems, Man, and Cybernetics, pages 2418–2423, 2015.
  * [29] Hieu V Nguyen and Li Bai.  Cosine similarity metric learning for face verification.  In ACCV, pages 709–720. Springer, 2010.
  * [30] Zhenxing Niu, Mo Zhou, Le Wang, Xinbo Gao, and Gang Hua.  Ordinal regression with multiple output cnn for age estimation.  In CVPR, pages 4920–4928, 2016.
  * [31] Hongyu Pan, Hu Han, Shiguang Shan, and Xilin Chen.  Mean-variance loss for deep age estimation from a face.  In CVPR, pages 5285–5294, 2018.
  * [32] Hongyu Pan, Hu Han, Shiguang Shan, and Xilin Chen.  Revised contrastive loss for robust age estimation from face.  In 2018 24th International Conference on Pattern Recognition (ICPR), pages 3586–3591. IEEE, 2018.
  * [33] Gabriel Panis, Andreas Lanitis, Nicolas Tsapatsoulis, and Timothy Cootes.  An overview of research on facial aging using the fg-net aging database.  IET Biometrics, 5, May 2015.
  * [34] KBRK Ramesha, KB Raja, KR Venugopal, and LM Patnaik.  Feature extraction based face recognition, gender and age classification.  International Journal on Computer Science and Engineering,, 2:14–23, 2010.
  * [35] K. Ricanek and T. Tesafaye.  Morph: a longitudinal image database of normal adult age-progression.  In 7th International Conference on Automatic Face and Gesture Recognition (FGR06), pages 341–345, 2006.
  * [36] Rasmus Rothe, Radu Timofte, and Luc Van Gool.  Deep expectation of real and apparent age from a single image without facial landmarks.  IJCV, 126:144–157, 2016.
  * [37] Florian Schroff, Dmitry Kalenichenko, and James Philbin.  Facenet: A unified embedding for face recognition and clustering.  In CVPR, pages 815–823, 2015.
  * [38] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In ICCV, pages 618–626, 2017.
  * [39] Haoyi Wang, Victor Sanchez, and Chang-Tsun Li.  Improving face-based age estimation with attention-based dynamic patch fusion.  IEEE Transactions on Image Processing, 2022.
  * [40] Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu.  Cosface: Large margin cosine loss for deep face recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5265–5274, 2018.
  * [41] Xin Wen, Biying Li, Haiyun Guo, Zhiwei Liu, Guosheng Hu, Ming Tang, and Jinqiao Wang.  Adaptive variance based label distribution learning for facial age estimation.  Berlin, Heidelberg, 2020. Springer-Verlag.
  * [42] Zhiguang Yang and Haizhou Ai.  Demographic classification with local binary patterns.  In Seong-Whan Lee and Stan Z. Li, editors, Advances in Biometrics, pages 464–473, Berlin, Heidelberg, 2007. Springer Berlin Heidelberg.
  * [43] Chao Zhang, Shuaicheng Liu, Xun Xu, and Ce Zhu.  C3ae: Exploring the limits of compact model for age estimation.  In CVPR, pages 12579–12588, 2019.
  * [44] Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, and Yu Qiao.  Joint face detection and alignment using multitask cascaded convolutional networks.  IEEE Signal Processing Letters, 23(10):1499–1503, 2016.
  * [45] Yunxuan Zhang, Li Liu, Cheng Li, and Chen Change Loy.  Quantifying facial age by posterior of age comparisons.  In BMVC, 2017.