# A Comprehensive Sustainable Framework for Machine Learning and Artificial
Intelligence

Roberto Pagliari  Peter Hill  Po-Yu Chen Corresponding Author. Email: po-
<EMAIL_ADDRESS>Maciej Dabrowny  Tingsheng Tan  Francois Buet-
Golfouse  JPMorgan  Imperial College London

###### Abstract

In financial applications, regulations or best practices often lead to
specific requirements in machine learning relating to four key pillars:
fairness, privacy, interpretability and greenhouse gas emissions. These all
sit in the broader context of sustainability in AI, an emerging practical AI
topic. However, although these pillars have been individually addressed by
past literature, none of these works have considered all the pillars. There
are inherent trade-offs between each of the pillars (for example, accuracy vs
fairness or accuracy vs privacy), making it even more important to consider
them together. This paper outlines a new framework for Sustainable Machine
Learning and proposes FPIG, a general AI pipeline that allows for these
critical topics to be considered simultaneously to learn the trade-offs
between the pillars better. Based on the FPIG framework, we propose a meta-
learning algorithm to estimate the four key pillars given a dataset summary,
model architecture, and hyperparameters before model training. This algorithm
allows users to select the optimal model architecture for a given dataset and
a given set of user requirements on the pillars. We illustrate the trade-offs
under the FPIG model on three classical datasets and demonstrate the meta-
learning approach with an example of real-world datasets and models with
different interpretability, showcasing how it can aid model selection.

##  1 Introduction

Artificial Intelligence has become an emerging tool essential for all
financial sectors [37, 61, 53, 57].

However, the characterisation of AI extends beyond the realm of technology and
permeates into the precincts of infrastructure [26] and ideology [44], leading
to an opacity around the concept of AI [40]. This nebulous nature of AI
magnifies the challenges of effectively understanding and governing it while
underscoring the need for malleability and interdisciplinary dialogue in AI
ethics and governance. Consequently, this discourse does not gravitate towards
a rigid definition of AI; rather, it embraces its polysemous essence and
explores AI as a complex system [16].

The current landscape of AI ethics frameworks [32, 39] is peppered with a
proliferation of proposed principles and a conspicuous absence of uniformity
across these frameworks. The initial environmental rights and climate justice
movements were driven by the United Nations Climate Change Conferences, and
Sustainable Development Goals (SDGs) [6], along with the environmental, social
and corporate governance (ESG) frameworks [4]. Unfortunately, while the
environmental implications of AI are gradually entering the discourse [56],
the broader concept of sustainability in AI appears to be largely overlooked
[34]. Recent literature only fostered a narrow vision of sustainable AI [58,
62], neglecting the interconnected nature of various AI governance challenges.
A holistic view of sustainable AI should be an amalgamation of three
intertwined pillars: economic, environmental and social, necessitating a
complex systems approach [27].

Financial institutions have particular duties in relation to AI that need to
be paid close attention to. The Information Commissioner’s Office (ICO) has
strict guidance on AI regarding interpretability, data protection and privacy
[8]. Additionally, there have been several recent developments from
significant organisations relating to AI regulations, bolstering the
importance of sustainable AI’s key features. For example, the European Union
has proposed the AI Act [7], a European law on AI. The Bank of England has
also recently updated their model risk management framework [1], outlining the
expectations of the Prudential Regulation Authority (PRA) for banks’
management of model risk. They indicate the need for a robust approach to
model risk and discuss five key principles within their framework, from
governance to validation. The particular focus on model risk mitigants
indicates the importance for banks to consider factors such as the
interpretability and fairness of their models as part of the model selection
stage. In addition, the FCA has also recently updated their consumer duty
expectations [3], raising the standards required by financial institutions
from previous expectations. With the growing role AI is having, there is an
increased expectation from the FCA for consumers to be at the forefront of
model design. The EU AI act [5] has also been proposed recently, focusing on
the risk of AI applications and categorising AI use into four risk levels, as
well as imposing increased documentation and validation of models.

In this paper, we first advocate for the adoption of the principles of
sustainability science to AI, analysing AI through the lens of an
unsustainable system. We then propose a Fair, Private, Interpretable and Green
(FPIG) framework to address the above-mentioned pillars. These four features
(as illustrated in Fig 1) are tightly associated with the concept of
sustainability but, to the best of our knowledge, have yet to be tackled
together under a single framework.

In the FPIG framework, we first propose to integrate fairness [35] into our ML
objective function to reduce the loss disparity across groups. This approach
allows us to change the level of fairness required in our training, varying
from a standard optimisation (with no additional fairness constraints) to a
multi-objective scenario, where trade-offs across different metrics form the
Pareto front. Secondly, we integrate the concept of differential privacy
during the model training process. By adding noise during training, we ensure
that good models are obtained for all other dimensions across varying degrees
of differential privacy.

The carbon dioxide (CO2) emission during model training and inference pipeline
is tracked and monitored by an independent software package, CodeCarbon [2].
Ultimately, we propose a new meta-learning algorithm that helps users to find
better AI models and hyperparameters (e.g., number of neural network layers)
given a dataset and the three sustainability goals plus model
interpretability.

Figure 1: Privacy, fairness, accuracy, interpretability and GHG emission are
the practical sustainability metrics that drive the three pillars of
sustainability.

To demonstrate the effectiveness of the FPIG framework, we evaluate it with
five independent datasets and four distinctive types of machine learning (ML)
models, varying in interpretability. We compute the results across the
different pillars. Our evaluation reveals common trade-offs from training
models using our frameworks, such as trade-offs between accuracy, fairness and
privacy. We also indicate the significant features when considering a meta-
learning approach, allowing us to estimate the impact on accuracy, fairness,
privacy and carbon emissions of training different models without needing to
undertake the cost of actually training them. We demonstrate the usefulness of
this approach on a particular example.

The rest of this paper is organised as the following. A brief overview of the
related work is given in Section 2. We then present the FPIG framework in
Section 3. The experiment results with five distinctive datasets are presented
in Section 4, and the paper is ultimately concluded in Section 5.

##  2 Sustainability in Artificial Intelligence and Machine Learning

The notion of "sustainable AI" has been proposed by a variety of researchers
and practitioners [25, 54] intending to emphasise the interconnection between
AI and sustainability [58]. Nevertheless, the term "sustainable" is frequently
interpreted as synonymous with "environmentally friendly." For instance, the
"Sustainable AI" manifesto issued by Facebook AI [62] is solely focused on
diminishing carbon emissions from AI systems whilst vowing to "advance the
field of AI in an environmentally responsible manner". This exemplifies the
challenge of encouraging stakeholders to embrace a multifaceted perspective on
sustainability rather than confining it merely to environmental aspects.

#### Fairness

is one of the most important sustainability metrics according to SDGs.
Fairness in ML models refers to the absence of bias or discrimination in the
predictions and decisions made by the models [50, 59, 15]. Technically
speaking, fairness involves identifying and addressing biases in the data used
to train the models and the algorithms themselves. Techniques such as pre-
processing the data to remove biased patterns, using specialised algorithms
that explicitly consider fairness constraints during training, and employing
fairness metrics to evaluate model performance can help achieve fairness in ML
models [17, 18]. Recent research has defined different fairness metrics for AI
[50, 59, 15]. Among these definitions, group fairness metrics such as
demographic parity [19], equalised odds [35], and social fairness ensures that
different groups based on a protected attribute are treated equally. The
impossibility theorems on fairness [41, 23] show that these definitions cannot
all be satisfied at once. There is also individual fairness and counterfactual
fairness [42], which are proposed to ensure fairness at individual levels.
Fairness can be considered in both a supervised learning or an unsupervised
learning setting. Recent research also investigated in incorporating multiple
fairness objectives into ML models given a desired level of fairness, using
group functionals [17, 18].

|c|c|c| Model Explainability Tunable Hyperparameters Linear Regression 1
Regularisation strength Tree 1 Max depth Random Forest 2 Number of estimators
Max depth
Max rows to subsample XGBoost 2 Number of estimators
Max depth

Learning rate

Fraction of columns to subsample

Max rows to subsample

L1 regularisation
L2 regularisation
Minimum loss reduction for partition
Balance between positive and negative samples Neural Network 3 Number of
layers
Layer size

Table 1: Models and associated tunable hyperparameters used in our
benchmarking study. Note that we consider a subset of the hyperparameters that
could potentially be tuned, and thus this is not an exhaustive list. A self-
defined measure of explainability from $1$ to $3$ is included, with $1$ being
the most and $3$ least explainable.

Privacy in ML models pertains to preserving the confidentiality and security
of sensitive data used for training and inference [55]. Privacy protection
involves implementing mechanisms that prevent unauthorised access, use, or
disclosure of personal information. Techniques like data anonymisation,
encryption, and secure multi-party computation can be employed to protect
privacy in ML models. Differential privacy (DP) has also been regarded as the
gold standard in academia as it provides a well-defined theoretical guarantee.
It can be applied to ensure that individual data points are not
distinguishable, thereby safeguarding privacy while maintaining the utility of
the models [9]. Adding noise during training is one way to incorporate DP into
ML models. For example, Differentially private stochastic gradient descent
(DP-SGD) [30] makes deep learning models differentially private by modifying
the mini-batch stochastic optimisation process during gradient descent [21,
20, 38]. Other approaches incorporate DP to data synthesis [45, 60] and than
train the model on the DP synthetic data, making these models more robust
against DP attacks with exponentially many queries.

(a) Adult Income

(b) COMPAS

(c) LSAC

(d) Loan Default

(e) Support 2

Figure 2: The moving average (we consider a rolling $500$ trial period.) of
Accuracy, Group Disparity, and Emissions metrics over time (number of trials),
for (a) Adult income, (b) COMPAS, (c) LSAC, (d) Loan Default, (e) Support2
datasets.

Interpretability in ML models refers to understanding and explaining the
reasoning behind the model’s predictions or decisions [51]. Interpretability
techniques involve feature importance analysis, rule-based approaches, and
model-agnostic techniques[28]. These techniques provide insights into the
factors influencing the model’s output and enable humans to comprehend and
validate the decision-making process. Also, layer-wise relevance propagation
(LRP) or attention mechanisms can help identify relevant features or parts of
the input contributing to the model’s predictions. A few works investigated
the trade-offs between computational efficiency and model explainability [33,
46].

Greenhouse Gas (GHG) emissions associated with ML models refer to the carbon
footprint generated during the model’s lifecycle, including data processing,
training, inference, and deployment [36]. At a technical level, reducing GHG
emissions involves optimising the computational resources used for training by
employing energy-efficient hardware and algorithms. Techniques like model
compression, which reduces the model’s size or complexity, can also contribute
to lower energy consumption during inference [22]. Additionally, adopting
hardware acceleration techniques, using distributed computing, and leveraging
renewable energy sources can help minimise the environmental impact of ML
models [43].

While there exists extensive literature working on improving model efficiency,
which ultimately leads to lower greenhouse gas (GHG) emissions, explainable AI
(xAI) has become an emerging area that attracts research [28, 51]. A few works
investigated the trade-offs between computational efficiency and model
explainability [33, 46]. Nevertheless, to our knowledge, this paper is the
first to introduce a framework encompassing all four sustainability goals,
including fairness, privacy, model interpretability and low GHG emissions.

##  3 The FPIG Framework

We propose a framework incorporating the four sustainability features into the
model training pipeline. Specifically, we are optimizing along different
dimensions (e.g., model performance, explainability, carbon emission and
fairness with some level of privacy).

###  3.1 Single-Objective Optimization

Traditionally, the hyper-parameters of a machine learning model are tuned to
maximize one metric of interest, for example, the area under the curve. Once
the metric of interest is defined, the objective is to minimize the quantity
in Eq. (1):

minimize $\displaystyle f(\textbf{x})$ (1) subject to
$\displaystyle\textbf{x}\in\mathbf{X}$

where $\textbf{x}\in\mathbf{X}\subseteq\mathbb{R}^{d}$ is the set of $d$
hyper-parameters, $\mathbf{X}\subseteq\mathbb{R}^{d}$ is the search space and
$f(\cdot)$ is the objective function, for example, a loss function to be
minimized.

In the case of a single-objective scenario, such as the one shown in Eq. (1),
the Tree-structured Parzen Estimator (TPE), originally proposed for neural
networks [14], is widely used for optimizing a wide number of machine learning
models. The key idea is to separate the likelihood function $p(\mathbf{x}|y)$
in two components so as to identify which region the best hyper-parameters are
likely to be in:

$p(\mathbf{x}|y)=\begin{cases}l(\mathbf{x})&\text{if }y<y^{*}\\\
g(\mathbf{x})&\text{if }y\geq y^{*}\end{cases}$ (2)

where $y^{*}$ is usually a quantile of the observed values $y$ (e.g, $80\%$),
and $l(\cdot)$ and $g(\cdot)$ are the probability density functions formed
using the observations $\\{\mathbf{x}^{(i)}\\}$ below and above $y^{*}$,
respectively. This methodology begins with several random observations
$\\{\mathbf{x}\\}^{(i)}$ and proceeds iteratively by adding one observation at
a time such that the expected improvement is maximized. As shown in [14], the
expected improvement is proportional to

$\text{EI}_{y^{*}}(\mathbf{x})\propto\left[y^{*}+\frac{g(\mathbf{x})}{l(\mathbf{x})}(1-y^{*})\right]^{-1}.$

In other words, the aim is to sample with higher probability under
$l(\mathbf{x})$ (i.e., the portion of the density with the most promising
hyper-parameters) and lower probability under $g(\mathbf{x})$.

###  3.2 Multi-Objective Optimization

In a multi-objective scenario, we are interested in the minimization (or
maximization) of many objectives that usually conflict. The optimization
problem is defined as follows:

minimize $\displaystyle
f(\mathbf{x})=(f_{1}(\textbf{x}),\dots,f_{n}(\textbf{x}))$ (3) subject to
$\displaystyle\textbf{x}\in\mathbf{X}$

where, as in Eq. (1), $\textbf{x}\in\mathbf{X}\subseteq\mathbb{R}^{d}$ is the
set of $d$ hyper-parameters, $\mathbf{X}\subseteq\mathbb{R}^{d}$ is the search
space. The difference is that this time, we define a vector of cutoffs
$\mathbf{Y}^{*}=(y_{1}^{*},\dots,y_{n}^{*})$, that is, a cutoff for every
objective, and the TPE estimator is generalized a follows:

$p(\mathbf{x}|\mathbf{y})=\begin{cases}l(\mathbf{x})&\text{if
}\mathbf{y}\succ\mathbf{Y^{*}}\cup\mathbf{y}\parallel\mathbf{Y^{*}}\\\
g(\mathbf{x})&\text{if }\mathbf{Y^{*}}\succeq\mathbf{y}\end{cases}$ (4)

where the $\succ$, $\succeq$ and $\parallel$ operators denote dominant, weakly
dominant and non-comparable relationships, respectively, as per [52]. This
time, as shown in Eq. (4), the most "promising" solutions are those that
dominate the cutoff $\mathbf{Y^{*}}$ or those that are not comparable to
$\mathbf{Y^{*}}$. The less promising models are those that are weakly
dominated by $\mathbf{Y^{*}}$ and, thus, contribute to forming the density
function of the least "promising" hyper-parameters $g(\mathbf{x})$. Splitting
the data into two sets is, in this instance, achieved via the Hype method
[12]; however, the methodology is, in principle, the multi-objective
equivalent of Eq. (1).

|c|c|c|c|c|c|c|c| Dataset Best Model w.r.t. Model Architecture Accuracy Group
Disparity Differential Privacy Explainability Carbon Emissions Adult income
[13] Accuracy xgboost 0.861 0.167 10.5 2.0 $3.92\times 10^{-6}$ Fairness
xgboost 0.767 0.000 10.0 2.0 $3.18\times 10^{-6}$ Carbon Emissions decision
tree 0.726 0.446 10.5 1.0 $\mathbf{2.26\times 10^{-6}}$ Equal Importance
xgboost 0.767 0.000 10.0 2.0 $3.05\times 10^{-6}$ COMPAS [11] Accuracy
decision tree 0.671 0.095 10.5 1.0 $1.65\times 10^{-7}$ Fairness xgboost 0.460
0.000 0.5 2.0 $6.59\times 10^{-7}$ Carbon Emissions decision tree 0.630 0.079
1.5 1.0 $\mathbf{7.00\times 10^{-8}}$ Equal Importance logistic regression
0.614 0.007 0.5 1.0 $5.02\times 10^{-7}$ LSAC [24] Accuracy neural network
0.949 0.003 10.5 3.0 $9.71\times 10^{-5}$ Fairness xgboost 0.946 0.000 2.0 2.0
$7.58\times 10^{-7}$ Carbon Emissions xgboost 0.946 0.000 2.0 2.0
$\mathbf{7.58\times 10^{-7}}$ Equal Importance xgboost 0.946 0.000 2.0 2.0
$7.58\times 10^{-7}$ Loan Default [64] Accuracy neural network 0.999 0.013
10.5 3.0 $2.56\times 10^{-4}$ Fairness decision tree 0.745 0.000 10.5 1.0
$3.59\times 10^{-6}$ Carbon Emissions decision tree 0.745 0.000 10.5 1.0
$\mathbf{3.59\times 10^{-6}}$ Equal Importance random forest 0.848 0.0003 10.5
2.0 $1.44\times 10^{-5}$ Support2 [64] Accuracy xgboost 0.977 0.0259 10.5 2.0
$3.01\times 10^{-6}$ Fairness xgboost 0.255 0.000 2.5 2.0 $1.64\times 10^{-6}$
Carbon Emissions decision tree 0.913 0.021 10.5 1.0 $\mathbf{6.16\times
10^{-7}}$ Equal Importance decision tree 0.956 0.005 10.5 1.0 $7.02\times
10^{-7}$

Table 2: The best models across different datasets, concerning different
sustainability metrics. We consider accuracy, fairness, carbon emissions and
an equal importance approach, as defined in Section 4.3. The results shown are
on an out-of-sample test set.

###  3.3 Incorporating Fairness

There exist several metrics for quantifying fairness and algorithmic bias of
machine learning models. Amongst them, the most popular and often considered
are equalized odds, equal opportunity, and demographic parity [49]. Without
loss of generality, in our framework, we optimized for demographic parity,
which is satisfied when the condition below holds true:

$P(\hat{Y}|A=0)=P(\hat{Y}|A=1).$

11121321In other words, the protected attribute $A$ (e.g., sex or age) does
not influence the model’s outcome. In reality, demographic parity can never be
exactly zero because the protected attribute $A$ usually correlates with other
features the model uses. Hence the objective is minimizing group disparity
$f(\hat{Y},A)$ as

$f(\hat{Y},A)=\left|P(\hat{Y}|A=0)-P(\hat{Y}|A=1)\right|.$ (5)

###  3.4 Incorporating Privacy

To include privacy in our framework, we consider the concept of differential
privacy [29, 30], which we use to include privacy guarantees in the model.

######  Definition 1.

A randomized mechanism $\mathcal{M}:\mathcal{D}\rightarrow\mathcal{R}$
satisfies $(\epsilon,\delta)$-differential privacy if for any two adjacent
inputs $d,d^{\prime}\in\mathcal{D}$, and any $S\subset\mathcal{R}$ fulfil the
inequality below:

$\mathbb{P}(\mathcal{M}(d)\in S)\leq
e^{\epsilon}\mathbb{P}(\mathcal{M}(d^{\prime})\in S)+\delta.$ (6)

To incorporate Differential Privacy into diverse model architectures, we
didn’t take the common route where privacy is applied via training, such as
the popular Differentially-Private Stochastic Gradient Descent (DP-SGD) [9].
Instead, we exploit the idea that converts data into differentially private
synthetic data, which can be exploited by different model architectures
universally. Various DP data synthesis approaches were proposed for different
modalities. For example, [60] and [47] can generate DP synthetic data for
tabular and images, respectively.

In this work, we adopted DPView [45], a state-of-the-art DP-aware high-
dimensional data synthesis for tabular data. For a given privacy requirement
$\epsilon$, it utilises the domain size of attributes and the correlation
among attributes to analytically optimise both privacy budget allocation and
consistency in producing synthetic data points. Their evaluation demonstrated
that the approach is versatile (when applied to tabular data from vest
applications) and can effectively preserve model utilities. Compared to
traditional gradient-based approaches (e.g., DP-SGD [9]) where privacy can
still be breached by querying the model multiple times and the total privacy
budgets are required to split across users, DPView is more robust as it does
not suffer from the same issues since noises are directly applied to data
instead of the model during training.

(a) Loan

(b) COMPAS

(c) LSAC

Figure 3: Scatter plots of accuracy against group disparity for different
datasets for all $2000$ trials. Different models are shown in different
colours, and different levels of carbon emissions are shown using different
sizes.

###  3.5 Evaluating GHG Emissions

We included a GHG emission tracking with CodeCarbon [48] throughout our
training and inference pipeline to monitor the carbon emissions from each
model training. It helps to track the overall carbon emission by accumulating
the power consumption of individual hardware components and converting it into
GHG emission based on the energy mixture of local power grids. At the end of
the training, the overall GHG emission amount is output along with the model
parameters.

##  4 Evaluation

This section presents an experimental analysis of an AI pipeline implementing
the proposed FPIG framework. All the code is implemented in Python 3.9, and
the experiments are performed on an 16 CPU instance consisting of 64GB memory.
The evaluation is divided into two parts. Firstly, we run $2000$ trials for
each dataset using different models and hyperparameters using the FPIG
framework. Using results across all trials, we then identify the relationships
between the key metrics (accuracy, fairness, emissions). Secondly, we use the
results of the trials to develop a _sustainable meta learning_ algorithm,
aiming to learn the relationship between the meta-features of the model and
dataset used, and the outputted accuracy, fairness and emissions.

###  4.1 Dataset, Differential Privacy and Protected Attributes

We included the five public datasets in our evaluation:

  * •

Adult Income [13] is a public multivariate social dataset for annual income
classification (i.e., if annual income is above 50K). It comprises 48,842
records with 14 attributes such as education, occupation, and work class.

  * •

COMPAS Recidivism Racial Bias [11] is a popular commercial algorithm judges
and parole officers use to score criminal defendants’ recidivism likelihood.
This dataset compares the algorithm outputs and the ground truths, which shows
that the algorithm is biased in favour of white defendants and against black
inmates, based on a 2-year follow-up study.

  * •

LSAC [24] is a public dataset originally collected for a ’LSAC National
Longitudinal Bar Passage Study’ study. It includes background information and
if (and how) candidates passed the bar exam to become lawyers in the United
States.

  * •

Loan Default [64] is a public multivariate financial dataset for loan default
classification. It includes 139,202 records with 34 attributes such as income,
gender, and loan purpose. Note that we randomly sampled 40,000 records from
this dataset in our evaluation.

  * •

Support2 [31] is a public multivariate health dataset for predicting survival
over a $180$-day period for seriously ill hospitalized adults. It comprises
9,105 records and 42 attributes, such as age, sex, and follow-up days.

Each dataset is split into a training set (70%) and a test set (30%). We then
apply DPView [45] to each training set and generate twenty Differentially
Private (DP) synthetic data (having the same number of data points as the
training sets) with different DP levels $\epsilon=[0.5,10.0]$, where smaller
$\epsilon$ indicates higher DP. Also, we only consider one protected
attribute, that is gender, for all five datasets. We only consider binary
protected attributes (thus generating two groups in each case) and measure
fairness via disparity, i.e., the absolute difference in the average loss
between both groups. This is for simplicity, but our approach can easily carry
over to other metrics and more complex settings.

###  4.2 Models and their Parameters

We built the FPIG framework using various models with varying degrees of
complexity. We consider a range of hyperparameters for each model, as detailed
below. Varying the model complexity and parameters will give different
performance metrics and, better or worse, fairness and GHG emissions. Complex
models like Neural Networks are expected to worsen fairness due to
overfitting. The trade-offs between fairness and accuracy, as well as GHG
emissions, should always be considered when designing new models. Table 2
illustrates the models and associated hyperparameters used in our study. We
also include a self-defined measure of each model’s explainability. This
ranges from $1$ to $3$, with $1$ being the most explainable model and $3$
least explainable.

###  4.3 Search Space and Pareto-Front Analysis

We exploited Optuna [10], a state-of-the-art hyper-parameter optimization
package, as our optimization engine to find the Pareto Fronts of the
objectives defined in the FPIG framework. For each dataset, we run $2000$
trials. For each trial, we select a value between $0.5$ and $10$ for
differential privacy, where $10$ indicates lower differential privacy and vice
versa. We then use the respective differentially private dataset based on this
value. We also include the option of no differential privacy. The objectives
are listed below:

  * •

Fairness: we exploited demographic parity [49] as our fairness metrics. The
objective is to minimise the group disparity $f$ in Equation (5).

  * •

Explainability: ML models are assigned an ordinal value $/{0,1,2,3/}$ to
illustrate their explainability as in Table 2, the lowest indicating the
simplest models. Models range from decision logistic regression to decision
trees, random forests, XGBoost, and neural networks.

  * •

Carbon emission: CodeCarbon [48] is used to measure GHG emission in practical
settings. The GHG emission is reported in the unit of the kilogram.

  * •

Accuracy: given that all the five tasks are classification, we use
classification accuracy $[0,1]$ as the performance metric evaluating model
utility.

  * •

Equal Importance: To find the best models with respect to the accuracy,
fairness and carbon emissions, we consider a naive approach for trading off
each of them with equal importance. To define this, we scale the values of
each metric to be between $0$ and $1$ to obtain $v^{D}_{i,j}$, the scaled
value for each trial $i\in\\{1,\cdots,2000\\}$ for each metric
$j\in\\{\text{Accuracy, Fairness, Carbon Emissions}\\}$, for each dataset $D$.
The best trial is defined to be

$i^{D}=\text{argmin}_{i}\Big{(}(1-v^{D}_{i,\text{Accuracy}})+v^{D}_{i,\text{Fairness}}+v^{D}_{i,\text{Carbon
Emissions}}\Big{)},$ (7)

as we wish to maximise Accuracy, whilst minimising Fairness and Carbon
Emissions.

###  4.4 Optimisation under the FPIG Framework

We use Optuna to search through the hyperparameter space and find the Pareto-
Frontier. Table 3.2 summarises the best models and their overall performances
concerning the performance metrics - Accuracy, Fairness, Carbon Emissions -
and their corresponding model Explinabilities and Differential Privacy levels.
We further illustrate (in a rolling average of 500 trials) the trade-offs
between the three performance metrics over time in Figure 2. Below, we
summarise our observations.

#### Although the degree may vary, the trade-offs between objectives always
exist.

The results showed that Accuracy can always trade for Fairness across all five
datasets. This observation aligns with the heuristic, where better fairness
usually leads to worse model utility. However, the degree of trade-offs
varies. As can be seen in Table 3.2, Adult Income demonstrates a small
accuracy reduction from 0.861 to 0.767 when improving the group disparity from
0.167 to 0.0, whilst Support2 shows significant accuracy reductions from 0.977
to 0.255 when improving the group disparity from 0.0259 to 0.0. Similarly, we
can train a model with lower Carbon emissions by trading Accuracy and
Fairness. Compared to Accuracy, the trade-offs between Carbon Emission and
Fairness are more significant across all five datasets. For example, when
applying the model with the lowest Carbon Emission model to the Adult income
dataset, the Accuracy and Group Disparity were reduced by 0.135 and increased
by 0.279, respectively, compared to the optimal for Accuracy and Fairness.

#### Multiple objectives can be improved during model tuning.

Although trade-offs (disregarding their degree) are seen across all datasets
tested as shown in Table 3.2, we also observed that multiple objectives could
still be improved simultaneously. First, the models selected by optimising the
Equal Importance (Eq. (7)) demonstrate balanced performances between all three
metrics - Accuracy, Fairness and Carbon Emission, indicating that it is
possible to find a good solution across all datasets. For example, we find a
model in which the Accuracy and Fairness are reduced by only 0.057 and 0.007,
respectively, with the COMPAS dataset. Second, from Figure 2 we observed that
the trade-offs between objectives varies as per dataset. For example, Accuracy
and Group Disparity are positively correlated (i.e., higher accuracy and lower
fairness) in Adult Income and COMPAS datasets, whilst the same correlation in
the LSAC and Support 2 datasets is negative. Although it is possible to
improve multiple objectives simultaneously, the trajectory toward optimal
(concerning the surrogate objective, such as the Equal Importance in Eq. (7))
can vary as per dataset.

#### The models yielding lower Group Disparity are usually more deferentially
private.

Heuristically, we know that protected attributes (e.g., gender in our
experiments) could potentially be utilised to identify individuals. Therefore,
differential privacy (DP) could also be improved when building fair models for
those protected attributes. Our experiment results shown in Table 3.2 confirm
the above hypothesis. The best model concerning fairness always provides
better DP (fulfilling smaller privacy budget $\epsilon$) compared to other
scenarios.

|c|c|c|c|c|c| Model Inputs Dataset
Adult COMPAS Loan LSAC Support2 Logistic Regression 0.518 0.439 -0.348 0.300
-0.202 Decision Tree 0.472 0.383 -0.608 0.253 -0.321 Random Forest -0.072
0.526 -0.612 0.137 -0.054 XGBoost 0.519 0.871 -0.283 -0.093 -0.188 Neural
Network 0.485 0.449 0.493 -0.570 0.028

Table 3: The correlation between Accuracy and Group Disparity.

#### Simpler models are usually more explainable and carbon-friendly.

This observation reinforces our heuristic regarding the trade-offs between
model complexity and explainability. As seen in Table 3.2, the best models for
Carbon Emission adopt the decision tree architecture (i.e., more explainable
and easy to compute) across four datasets. In contrast, Neural Network (NN)
and XGBoost, having extensive capability to approximate any continuous
function with lower Explainability, achieved the best Accuracy across four
datasets. Further observations regarding the impact of model architecture will
be presented in the next subsection.

###  4.5 The Impact of Model Architecture

The choice of model architecture also plays a significant role when searching
for better solutions under the FPIG framework. The best model architecture
varies significantly across datasets. We further see this in Figure 3, where
we compare accuracy and fairness for the three datasets - Loan Default, COMPAS
and LSAC. We plot a line that best fits each model type and dataset. As can be
seen, the trade-offs not only vary in between datasets but also differ between
model architectures. The correlations between Accuracy and Group Disparity
when applying different model architectures across the five datasets shown in
Table 4.4 also support this observation. In the COMPAS dataset, we see similar
behaviour across all model types. As we increase Accuracy, this must come at
the expense of Fairness, as Group Disparity also increases. This is shown by
the positive gradient of the lines of best fit in Figure 3 and the positive
correlations in Table 4.4. In contrast, the correlation becomes negative when
applying NNs to the LSAC dataset, which means that the NN models could improve
Accuracy and Group Disparity simultaneously compared to other model
architectures. This difference is less obvious in Loan Default and COMPAS
datasets.

Model Inputs | Sustainability Feature Coefficients
---|---
Accuracy | Group Disp. | GHG
No DP Applied | 1.118 | 0.843 | -0.153
Classifier NN (y/n) | 0.537 | -0.200 | 1.902
DP 5.5 (y/n) | 0.446 | 0.008 | 0.008
DP 7.0 (y/n) | 0.402 | 0.215 | -0.018
DP 10.0 (y/n) | 0.326 | 0.180 | 0.091
Dataset Column Number | 0.322 | -0.416 | 0.006
DP 3.0 (y/n) | 0.306 | 0.326 | 0.018
# of categorical features | 0.069 | -0.413 | 0.071
# of NN layers | -0.024 | 0.021 | 0.417
Size of NN layer | -0.067 | -0.006 | 0.102
Feature cardinality | -0.094 | 0.317 | -0.016
Variance of target | -0.461 | -0.149 | 0.034
E-net $R^{2}$ On Test Set | 0.6967 | 0.3799 | 0.6806
Table 4: Meta-learning model: Ridge Regression $(\alpha=1)$ of accuracy,
fairness (group disparity) and GHG emissions on selected features of the
combined datasets. Coefficients impose the importance of each feature on the
metric of interest. Accuracy and fairness are sensitive to differential
privacy (lower is more differentially private), whilst GHG emissions depend on
the size of the original training set and the model used.

###  4.6 Sustainable Meta Learning

In the previous subsection, we studied the trade-offs between sustainable
objectives and realised that the preferred hyperparameters and model
architectures vary as per dataset. A single solution does not exist that fits
all scenarios. Following the methodology in [63], we trained regression models
$\mathcal{M}_{i}$ for each of
$i\in[\text{accuracy},\text{disparity},\text{emissions}]$ that learn the
relationship between the key objectives (i.e. accuracy, group disparity, and
GHG emission), based on features of the dataset $d_{X}$ (e.g., number of
features, number of entries) as well as features on the model and training
$d_{m}$ ( e.g., hyperparameters of the model architecture, number of training
epochs) and finally privacy level requirements, $d_{p}$. This is trained based
on the trained models from the FPIG framework using Optuna. We aim to offer
users a framework to determine which architecture and "sustainable
hyperparameters" to pick given requirements before training, which is
typically time-consuming and leads to extensive energy consumption and GHG
emissions. This is the first step towards developing broader frameworks and
attempting to define a meta-learning approach that will allow for a more
automated ML system.

Table 4 summarises the results by running separate regression models against
each objective of interest and showing the learnt coefficients of selected
inputs. We list the most important features in the Table. Our experiment
demonstrates the relationship between dataset, model hyperparameters and
sustainability features. We observe that using a less differentially private
dataset increases the accuracy, thus illustrating the trade-off between
accuracy and privacy previously discussed. We also note that using neural
networks tends to increase accuracy, whilst using a dataset with a large
variance in the target will decrease the model’s accuracy. Further, group
disparity tends to decrease with more privacy. Using no differential privacy
is the biggest factor in having a larger group disparity, with a coefficient
of 0.843. Lastly, we note that using Neural Networks has the largest impact on
GHG Emissions, thus increasing with model complexity. There doesn’t seem to be
any significant relationship between privacy requirements and carbon
emissions.

Algorithm 1 demonstrates how the meta learning algorithms can be used in
practice. This takes in a dataset $X$, a set of candidate model architectures
$d^{(k)}_{m}$ for $k=1,\cdots,K$, along with user requirements $\tau$ on the
minimum accuracy, maximum disparity and maximum emissions the user is willing
to accept. The algorithm then returns only the model architectures whose
estimated metrics sit within the thresholds, based on the meta learning
models. From this, a user can select their chosen model based on preference
across the metrics.

0: Dataset $X$, meta-learning models $\mathcal{M}_{i}$ for
$i\in[\text{accuracy},\text{disparity},\text{emissions}]$, user requirements
$\tau=[\tau_{\text{acc}},\tau_{\text{disp}},\tau_{\text{em}}]$, privacy
requirement $d_{p}$, candidate model architectures $d^{(k)}_{m}$ for
$k=1,\cdots,K$.

Compute $d_{x}$ for dataset $X$

for $k=1,\cdots,K$ do

Compute  $\displaystyle m^{(k)}_{\text{acc}}$ $\displaystyle=$
$\displaystyle\mathcal{M}_{\text{accuracy}}[d_{x},d_{p},d^{(k)}_{m}]$
$\displaystyle m^{(k)}_{\text{disp}}$ $\displaystyle=$
$\displaystyle\mathcal{M}_{\text{disparity}}[d_{x},d_{p},d^{(k)}_{m}]$
$\displaystyle m^{(k)}_{\text{em}}$ $\displaystyle=$
$\displaystyle\mathcal{M}_{\text{emissions}}[d_{x},d_{p},d^{(k)}_{m}]$

end for

return $\\{d^{(k)}_{m}$: $m^{(k)}_{\text{acc}}\geq\tau_{\text{acc}}$,
$m^{(k)}_{\text{disp}}\leq\tau_{\text{disp}}$,
$m^{(k)}_{\text{em}}\leq\tau_{\text{em}}\\}$

Algorithm 1 Candidate Model Evaluation

##  5 Conclusion

This paper introduces the FPIG framework for Sustainable Machine Learning,
considering a multi-objective optimisation problem involving accuracy,
fairness, privacy, explainability and carbon emissions. We demonstrate this
approach on five datasets and show the trade-offs between these sustainable
objectives and the possibility of finding models to balance these trade-offs.
We also extended these observations by building a meta-learning approach to
predict these key metrics based on the dataset and model characteristics. The
results further validate our above observations and show that fairness, as one
of the sustainable objectives, is more data-dependent, meaning that it is more
difficult to provide guarantees by selecting suitable model architectures and
corresponding hyperparameters.

## References

  * [1] Bank of england model risk management framework.  https://www.bankofengland.co.uk/prudential-regulation/publication/2023/may/model-risk-management-principles-for-banks-ss/.  Accessed: 2023-07-24.
  * [2] Codecarbon.  https://codecarbon.io/.  Accessed: 2023-07-24.
  * [3] Fca consumer duty.  https://www.fca.org.uk/firms/consumer-duty.  Accessed: 2023-07-24.
  * [4] Environmental, social, and corporate governance.  https://en.wikipedia.org/wiki/Environmental,_social,_and_corporate_governance.  Accessed: 2023-07-24.
  * [5] Eu ai act.  https://artificialintelligenceact.eu/the-act/.  Accessed: 2023-07-24.
  * [6] United nation’s 17 goals for sustainable development.  https://sdgs.un.org/goals.  Accessed: 2023-07-24.
  * [7] The eu aritificial intelligence act.  https://artificialintelligenceact.eu/.  Accessed: 2023-07-24.
  * [8] Information commissioner’s office.  https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/artificial-intelligence/.  Accessed: 2023-07-24.
  * Abadi et al. [2016] M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang.  Deep learning with differential privacy.  In _Proceedings of the 2016 ACM SIGSAC conference on computer and communications security_ , pages 308–318, 2016.
  * [10] T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama.  Optuna: A next-generation hyperparameter optimization framework.  In _Proc. SIGKDD 2019_.
  * Angwin et al. [2016] J. Angwin, J. Larson, S. Mattu, and L. Kirchner.  Compas.  Kaggle, 2016.  URL: https://www.kaggle.com/datasets/danofer/compass.
  * Bader and Zitzler [2011] J. Bader and E. Zitzler.  Hype: An algorithm for fast hypervolume-based many-objective optimization.  _Evolutionary computation_ , 19(1):45–76, 2011.
  * Becker and Kohavi [1996] B. Becker and R. Kohavi.  Adult.  UCI Machine Learning Repository, 1996.  DOI: https://doi.org/10.24432/C5XW20.
  * Bergstra et al. [2011] J. Bergstra, R. Bardenet, Y. Bengio, and B. Kégl.  Algorithms for hyper-parameter optimization.  _Advances in neural information processing systems_ , 24, 2011.
  * Berk et al. [2018] R. Berk, H. Heidari, S. Jabbari, M. Kearns, and A. Roth.  Fairness in criminal justice risk assessments: The state of the art.  _Sociological Methods & Research_, Aug. 2018.  10.1177/0049124118782533.
  * Bossel [2018] H. Bossel.  _Modeling and Simulation_.  A K Peters/CRC Press, Oct. 2018.  10.1201/9781315275574.
  * Buet-Golfouse and Utyagulov [2022] F. Buet-Golfouse and I. Utyagulov.  Towards fair unsupervised learning.  In _Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency_ , pages 1399–1409, 2022.
  * Buet-Golfouse and Utyagulov [2023] F. Buet-Golfouse and I. Utyagulov.  Fairness trade-offs and partial debiasing.  In _Asian Conference on Machine Learning_ , pages 112–136. PMLR, 2023.
  * Calders and Verwer [2010] T. Calders and S. Verwer.  Three naive bayes approaches for discrimination-free classification.  _Data Mining and Knowledge Discovery_ , 21(2):277–292, 2010.  10.1007/s10618-010-0190-x.
  * Chen et al. [2020] D. Chen, T. Orekondy, and M. Fritz.  Gs-wgan: A gradient-sanitized approach for learning differentially private generators.  _Advances in Neural Information Processing Systems_ , 33:12673–12684, 2020.
  * Chen et al. [2022] J.-W. Chen, C.-M. Yu, C.-C. Kao, T.-W. Pang, and C.-S. Lu.  Dpgen: Differentially private generative energy-guided network for natural image synthesis.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8387–8396, 2022.
  * Choudhary et al. [2020] T. Choudhary, V. Mishra, A. Goswami, and J. Sarangapani.  A comprehensive survey on model compression and acceleration.  _Artificial Intelligence Review_ , 53:5113–5155, 2020.
  * Chouldechova [2017] A. Chouldechova.  Fair prediction with disparate impact: A study of bias in recidivism prediction instruments.  _Big data_ , 5(2):153–163, 2017.
  * Clair [2022] A. Clair.  LSAC.  Kaggle, 2022.  URL: https://www.kaggle.com/code/eds8531/lsac-dataset-eda-and-predictions.
  * Coeckelbergh [2021] M. Coeckelbergh.  AI for climate: freedom, justice, and other ethical and political challenges.  _AI and Ethics_ , 1(1):67–72, Feb. 2021.  ISSN 2730-5961.  10.1007/s43681-020-00007-2.  URL https://doi.org/10.1007/s43681-020-00007-2.
  * Crawford [2021] K. Crawford.  _The atlas of AI: Power, politics, and the planetary costs of artificial intelligence_.  Yale University Press, 2021.
  * De Vries [2023] B. J. De Vries.  _Sustainability science_.  Cambridge University Press, 2023.
  * Dwivedi et al. [2023] R. Dwivedi, D. Dave, H. Naik, S. Singhal, R. Omer, P. Patel, B. Qian, Z. Wen, T. Shah, G. Morgan, et al.  Explainable ai (xai): Core ideas, techniques, and solutions.  _ACM Computing Surveys_ , 55(9):1–33, 2023.
  * Dwork et al. [2006] C. Dwork, F. McSherry, K. Nissim, and A. Smith.  Calibrating noise to sensitivity in private data analysis.  In _Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings 3_ , pages 265–284. Springer, 2006.
  * Dwork et al. [2014] C. Dwork, A. Roth, et al.  The algorithmic foundations of differential privacy.  _Foundations and Trends® in Theoretical Computer Science_ , 9(3–4):211–407, 2014.
  * et al [1995] C. J. A. F. et al.  A controlled trial to improve care for seriously iii hospitalized patients.  _Management Review Quarterly_ , 1995.
  * Fjeld et al. [2020] J. Fjeld, N. Achten, H. Hilligoss, A. Nagy, and M. Srikumar.  Principled Artificial Intelligence: Mapping Consensus in Ethical and Rights-Based Approaches to Principles for AI, Jan. 2020.  URL https://papers.ssrn.com/abstract=3518482.
  * Graziani et al. [2023] M. Graziani, L. Dutkiewicz, D. Calvaresi, J. P. Amorim, K. Yordanova, M. Vered, R. Nair, P. H. Abreu, T. Blanke, V. Pulignano, et al.  A global taxonomy of interpretable ai: unifying the terminology for the technical and social sciences.  _Artificial intelligence review_ , 56(4):3473–3504, 2023.
  * Hagendorff [2022] T. Hagendorff.  Blind spots in ai ethics.  _AI and Ethics_ , 2(4):851–867, 2022.
  * Hardt et al. [2016] M. Hardt, E. Price, and N. Srebro.  Equality of opportunity in supervised learning.  _Advances in neural information processing systems_ , 29, 2016.
  * Henderson et al. [2020] P. Henderson, J. Hu, J. Romoff, E. Brunskill, D. Jurafsky, and J. Pineau.  Towards the systematic reporting of the energy and carbon footprints of machine learning.  _The Journal of Machine Learning Research_ , 21(1):10039–10081, 2020.
  * Hilpisch [2020] Y. Hilpisch.  _Artificial Intelligence in Finance_.  O’Reilly Media, 2020.
  * Horigome et al. [2023] H. Horigome, H. Kikuchi, and C.-M. Yu.  Local differential privacy protocol for making key–value data robust against poisoning attacks.  In _International Conference on Modeling Decisions for Artificial Intelligence_ , pages 241–252. Springer, 2023.
  * Jobin et al. [2019] A. Jobin, M. Ienca, and E. Vayena.  The global landscape of AI ethics guidelines.  _Nature Machine Intelligence_ , 1(9), Sept. 2019.  10.1038/s42256-019-0088-2.
  * Katz [2020] Y. Katz.  _Artificial Whiteness: Politics and Ideology in Artificial Intelligence_.  Columbia University Press, Nov. 2020.
  * Kleinberg et al. [2017] J. Kleinberg, S. Mullainathan, and M. Raghavan.  Inherent trade-offs in the fair determination of risk scores.  In _Proceedings of the 8th Innovations in Theoretical Computer Science Conference_ , 2017.  10.4230/LIPIcs.ITCS.2017.43.
  * Kusner et al. [2017] M. J. Kusner, J. Loftus, C. Russell, and R. Silva.  Counterfactual fairness.  _Advances in neural information processing systems_ , 30, 2017.
  * Lai et al. [2020] J.-P. Lai, Y.-M. Chang, C.-H. Chen, and P.-F. Pai.  A survey of machine learning models in renewable energy predictions.  _Applied Sciences_ , 10(17):5975, 2020.
  * Lanier and Weyl [2015] J. Lanier and G. Weyl.  AI is An Ideology, Not A Technology.  _Wired_ , Mar. 2015.  ISSN 1059-1028.  URL https://www.wired.com/story/opinion-ai-is-an-ideology-not-a-technology/.  Section: tags.
  * Lin et al. [2022] C.-H. Lin, C.-M. Yu, and C.-Y. Huang.  Dpview: Differentially private data synthesis through domain size information.  _IEEE Internet of Things Journal_ , 9(17):15886–15900, 2022.
  * Lin and Gao [2022] K. Lin and Y. Gao.  Model interpretability of financial fraud detection by group shap.  _Expert Systems with Applications_ , 210:118354, 2022.
  * Lin et al. [2024] Z. Lin, S. Gopi, J. Kulkarni, H. Nori, and S. Yekhanin.  Differentially private synthetic data via foundation model APIs 1: Images.  In _Proc. ICLR 2024_ , 2024.
  * Lottick et al. [2019] K. Lottick, S. Susai, S. A. Friedler, and J. P. Wilson.  Energy usage reports: Environmental awareness as part of algorithmic accountability.  _arXiv preprint arXiv:1911.08354_ , 2019.
  * Mehrabi et al. [2021] N. Mehrabi, F. Morstatter, N. Saxena, K. Lerman, and A. Galstyan.  A survey on bias and fairness in machine learning.  _ACM computing surveys (CSUR)_ , 54(6):1–35, 2021.
  * Narayanan [2018] A. Narayanan.  Translation tutorial: 21 fairness definitions and their politics.  In _Proceedings of the Conference on Fairness, Accountability and Transparency_ , FAT* 18, New York, USA, 2018.
  * Novakovsky et al. [2023] G. Novakovsky, N. Dexter, M. W. Libbrecht, W. W. Wasserman, and S. Mostafavi.  Obtaining genetics insights from deep learning via explainable artificial intelligence.  _Nature Reviews Genetics_ , 24(2):125–137, 2023.
  * Ozaki et al. [2020] Y. Ozaki, Y. Tanigaki, S. Watanabe, and M. Onishi.  Multiobjective tree-structured parzen estimator for computationally expensive optimization problems.  In _Proceedings of the 2020 genetic and evolutionary computation conference_ , pages 533–541, 2020.
  * Pallathadka et al. [2023] H. Pallathadka, E. H. Ramirez-Asis, T. P. Loli-Poma, K. Kaliyaperumal, R. J. M. Ventayen, and M. Naved.  Applications of artificial intelligence in business management, e-commerce and finance.  _Materials Today: Proceedings_ , 80:2610–2613, 2023.
  * Sætra [2021] H. S. Sætra.  Ai in context and the sustainable development goals: Factoring in the unsustainability of the sociotechnical system.  _Sustainability_ , 13(4):1738, 2021.
  * Solove [2008] D. J. Solove.  Understanding privacy.  2008\.
  * Strubell et al. [2019] E. Strubell, A. Ganesh, and A. McCallum.  Energy and policy considerations for deep learning in nlp.  _arXiv preprint arXiv:1906.02243_ , 2019.
  * Umamaheswari et al. [2023] S. Umamaheswari, A. Valarmathi, et al.  Role of artificial intelligence in the banking sector.  _Journal of Survey in Fisheries Sciences_ , 10(4S):2841–2849, 2023.
  * Van Wynsberghe [2021] A. Van Wynsberghe.  Sustainable ai: Ai for sustainability and the sustainability of ai.  _AI and Ethics_ , 1(3):213–218, 2021.
  * Verma and Rubin [2018] S. Verma and J. Rubin.  Fairness definitions explained.  In _Proceedings of the International Conference on Software Engineering_ , pages 1–7, New York, NY, USA, 2018. ACM.  10.1145/3194770.3194776.
  * [60] G. Vietri, C. Archambeau, S. Aydore, W. Brown, M. Kearns, A. Roth, A. Siva, S. Tang, and S. Z. Wu.  Private synthetic data for multitask learning and marginal queries.  _Proc. NeurIPS 2022_.
  * Weber et al. [2023] P. Weber, K. V. Carl, and O. Hinz.  Applications of explainable artificial intelligence in finance—a systematic review of finance, information systems, and computer science literature.  _Management Review Quarterly_ , pages 1–41, 2023.
  * [62] K. Wu, E. Wu, and G. Kreiman.  Learning scene gist with convolutional neural networks to improve object recognition.  In _Proc. CISS 2018_.
  * Yang et al. [2019] C. Yang, Y. Akimoto, D. W. Kim, and M. Udell.  Oboe: Collaborative filtering for automl model selection.  In _Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining_, pages 1173–1183, 2019.
  * Yasser [2020] M. H. Yasser.  Loan default dataset.  Kaggle, 2020.  URL: https://www.kaggle.com/datasets/yasserh/loan-default-dataset.


Rate of change of _varentropy_ On the rate of change of _Varentropy_ for
Markov diffusion processesSupported by the NYUAD grant
76/71260/ADHPG/AD356/51119. Michele Pavon111Mathematics Program, Division of
Science, New York University, Abu Dhabi, UAE<EMAIL_ADDRESS>; local free energy ; space-time harmonic function ; Markov diffusion
60J60;62B10;94A17 November12, 2023 0 2020 0 10.1214/YY-TN Relying on the
reverse-time space-time harmonic property of the ratio of two solutions of the
Fokker-Plank equation, we establish an explicit formula for derivative of the
_varentropy_ for a Markov diffusion process. The formula involves a nonlinear
function of the local free energy $\ln(p_{t}(x)/\bar{p}(x))$.We then verify
that our formula yields the correct result in the simple case of a scalar
Gaussian diffusion. In the latter case, varentropy is exponentially decaying
to zero.

##  1 Introduction

Entropy concentration phenomena have been long studied, see. e.g. [6]. In [3],
concentration of information for convex measures was studied through the
variance of the information content, namely $-logp(X)$ where $p$ is the
density of the probability distribution of the random vector $X$. In the [12],
the cutoff phenomenon was explicitly connected to an entropy concentration
principle for non-negatively curved Markov chains. The central indicator of
the entropy concentration property is the (worst-case) Varentropy, namely the
variance of a relative information content with respect to the invariant
measure.

We study here the evolution of varentropy in the simple case of a Markov
diffusion process with constant diffusion coefficient. Using a reverse-time
space-time harmonic property of the ratio of two solutions of the Fokker-Plank
equation, we obtain an explicit formula for the derivative of the varentropy
which involves a nonlinear function of the local free energy
$\ln(p_{t}(x)/\bar{p}(x))$, $\bar{p}$ being the invariant density. We then
show that the formula applied to the scalar, Gaussian case yields the correct
result.

The paper is outlined as follows. In Section 2, we recall some results by
Föllmer dating back to the 1980’s. In Section 3 we recall some reverse-time
properties of the ratio of two solutions of a Fokker-Planck equation. In
Section 4, we present our main result Theorem 4.1 on the time derivative of
the Varentropy. In the final section, we show that the formula obtained in
Section 4 yields the correct result when applied to a scalar, Gaussian
diffusion.

##  2 Finite-energy diffusions

Let ${\cal W}_{x}$ be Wiener measure starting at $x$ on
$\Omega=C([0,T],\mbox{\BB R}^{n})$. Let

${\cal W}=\int{\cal W}_{x}dx$

be stationary Wiener measure. We denote by $\mathcal{D}$ the family of
probability measures on $\Omega$ which are equivalent to stationary Wiener
measure. Let $\mathcal{F}_{t}$ and $\mathcal{G}_{t}$ be the $\sigma$-algebras
of events observable up to time $t$ and from time $t$ on, respectively. By
Girsanov’s theorem, any $P\in\mathcal{D}$ has a forward drift $\beta_{t}^{P}$
measurable with respect to $\mathcal{F}_{t}$ and a backward drift
$\gamma_{t}^{P}$ measurable with respect to $\mathcal{G}_{t}$. That means
that, under $P$, the coordiante process admits for $0\leq s<t\leq T$ the
representations

$\displaystyle X_{t}-X_{s}$ $\displaystyle=$
$\displaystyle\int_{s}^{t}\beta_{\tau}^{P}d\tau+W_{t}-W_{s},$ (1)
$\displaystyle X_{t}-X_{s}$ $\displaystyle=$
$\displaystyle\int_{s}^{t}\gamma_{\tau}^{P}d\tau+\bar{W}_{t}-\bar{W}_{s}.$ (2)

Here $W$ and $\bar{W}$ are standard $n$-dimensional Wiener processes adapted
to $(\mathcal{F}_{t})$ and to $(\mathcal{G}_{t})$, respectively. Moreover, the
two drifts satisfy the condition

$P\left[\int_{0}^{T}|\beta^{P}_{t}|^{2}dt\right]=P\left[\int_{0}^{T}|\gamma^{P}_{t}|^{2}dt\right]=1,$

see e.g [7, Theorem 7.11]. By the Girsanov theory, we have $P$ a.s.

$\displaystyle\frac{dQ}{dP}(X)$ $\displaystyle=$
$\displaystyle\frac{q_{0}(X_{0})}{p_{0}(X_{0})}\exp\left[\int_{0}^{T}(\beta^{Q}_{t}-\beta^{P}_{t})dX_{t}+\frac{1}{2}\left(\beta^{P}_{t}-\beta^{Q}_{t}\right)^{2}dt\right]$
(3) $\displaystyle=$
$\displaystyle\frac{q_{T}(X_{T})}{p_{T}(X_{T})}\exp\left[\int_{0}^{T}(\gamma^{Q}_{t}-\gamma^{P}_{t})dX_{t}+\frac{1}{2}\left(\gamma^{P}_{t}-\gamma^{Q}_{t}\right)^{2}dt\right].$

######  Proposition 2.1.

Let $P,Q\in\mathcal{D}$ have one-time density $p_{t}$ and $q_{t}$,
$t\in[0,T]$, respectively. If $\gamma^{P}_{t}=\gamma^{Q}_{t}$ for $t\in[0,T]$,
$\frac{q_{t}}{p_{t}}(X_{t})$ is a $(P,\mathcal{F}_{t})$ martingale on $[0,T]$.
If $\beta^{P}_{t}=\beta^{Q}_{t}$ for $t\in[0,T]$, $\frac{q_{t}}{p_{t}}(X_{t})$
is a reverse-time $(P,\mathcal{G}_{t})$ martingale on $[0,T]$.

######  Proof 2.2.

Suppose $\gamma^{P}_{t}=\gamma^{Q}_{t}$ for $t\in[0,T]$. By (3), under $P$,

$\mbox{\BB
E}\left\\{\frac{dQ}{dP}|\mathcal{F}_{t}\right\\}=\frac{dQ_{0t}}{dP_{0t}}=\frac{q_{t}}{p_{t}}(X_{t}).$

Being the conditional expectation of a fixed random variable with respect to
an increasing family of $\sigma$-algebras, $\frac{q_{t}}{p_{t}}(X_{t})$ is a
supermantingale. Since it has constant expectation, it is actually a
martingale. The other half is proven similarly.

##  3 Markov diffusion processes

Let $\\{X_{t};t\geq 0\\}$ be a $\mbox{\BB R}^{n}$-valued Markov diffusion
process defined on a probability space $(\Omega,\cal F,\mbox{\BB P})$ with
forward It$\hat{o}$ differential

$dX_{t}=b_{+}(t,X_{t})dt+\sigma dW_{t},$ (4)

where $W$ is a standard $n$-dimensional Wiener process. Under rather weak
assumptions [8, 5], $X$ possesses on $t>0$ also a reverse-time differential

$dX_{t}=b_{-}(t,X_{t})dt+\sigma d\bar{W}_{t}.$ (5)

Here $\bar{W}$ is another $n$-dimensional standard Wiener process with
$\bar{W}_{s}-\bar{W}_{t}$ independent of $X_{\tau}$ whenever $s<t\leq\tau$.
Here and in the sequel $dt>0$ so that, if $X$ is a finite-energy diffusion,
i.e.

$\mbox{\BB E}\left\\{\int_{t_{0}}^{t_{1}}\|b_{+}\|^{2}\right\\}<\infty,$

we get the two drifts as Nelson’s conditional derivatives, namely

$\displaystyle b_{+}(t,X_{t})$ $\displaystyle=$ $\displaystyle\lim_{dt\searrow
0}E\left\\{\frac{X_{t+dt}-X_{t}}{dt}|X_{\tau},t_{0}\leq\tau\leq t\right\\},$
$\displaystyle b_{-}(t,X_{t})$ $\displaystyle=$ $\displaystyle\lim_{dt\searrow
0}E\left\\{\frac{X_{t}-X_{t-dt}}{dt}|X_{\tau},t\leq\tau\leq t_{1}\right\\},$

the limits being in $L^{2}_{n}(\Omega,\cal f,\mbox{\BB P})$ and $dt>0$ for
both formulas. Moreover, we have Nelson’s duality formula

$b_{-}(t,X_{t})=b_{+}(t,X_{t})-\sigma^{2}\nabla\ln p_{t}(X_{t})\>{\rm a.s.}$
(6)

where $p_{t}$ is the probability density of $X_{t}$ satisfying (at least
weakly) the Fokker-Planck equation

$\frac{\partial p}{\partial t}+\nabla\cdot(b_{+}p)-\frac{\sigma^{2}}{2}\Delta
p=0.$ (7)

From (6), we also have

$\frac{\partial p}{\partial t}+\nabla\cdot(b_{-}p)+\frac{\sigma^{2}}{2}\Delta
p=0.$ (8)

There are two change of variables formulae related to (4) and (5). Let
$f:\mbox{\BB R}^{n}\times[t_{0},t_{1}]\rightarrow\mbox{\BB R}$ be twice
continuously differentiable with respect to the spatial variable and once with
respect to time. Then:

$\displaystyle
f(X_{t},t)-f(X_{s},s)=\int_{s}^{t}\left(\frac{\partial}{\partial\tau}+b_{+}(\tau,X_{\tau})\cdot\nabla+\frac{\sigma^{2}}{2}\Delta\right)f(X_{\tau},\tau)d\tau$
(9) $\displaystyle+\int_{s}^{t}\sigma\nabla f(X_{\tau},\tau)\cdot dW_{\tau},$
(10) $\displaystyle
f(X_{t},t)-f(X_{s},s)=\int_{s}^{t}\left(\frac{\partial}{\partial\tau}+b_{-}(\tau,X_{\tau})\tau)\cdot\nabla-\frac{\sigma^{2}}{2}\Delta\right)f(X_{\tau},\tau)d\tau$
(11) $\displaystyle+\int_{s}^{t}\sigma\nabla f(X_{\tau})\cdot
d\bar{W}_{\tau}.$ (12)

The stochastic integrals appearing in (10) and (12) are a (forward) Ito
integral and a backward Ito integral, respectively, see [9] for the details.
Suppose that $b_{+}(t,x)=b_{+}(x)$ is such that (7) has $\bar{p}$ as invariant
solution. Then, a direct calculation [10] shows that $\bar{p}/p_{t}$ is
reverse-time space time harmonic, namely

$\left(\frac{\partial}{\partial\tau}+b_{-}(\tau,x)\cdot\nabla-\frac{\sigma^{2}}{2}\Delta\right)\left(\frac{\bar{p}}{p_{t}}\right)=0.$
(13)

It follows that $M_{t}:=\frac{\bar{p}}{p_{t}}(X_{t})$ is a reverse-time local
martingale with respect to the decreasing family ${\cal
F}_{t}^{+}(X)=\sigma\left\\{X_{\tau};\tau\geq t\right\\}$. This follows also
from Proposition 2.1 for diffusions with diffusion coefficient $\sigma^{2}$.
In view of (12), the reverse-time Ito differential of $M_{t}$ is

$dM_{t}=\sigma\nabla\left(\frac{\bar{p}}{p_{t}}\right)(X_{t})d\bar{W}_{t}.$
(14)

##  4 Varentropy

Define $-\ln p_{t}(x)$ as the local entropy. We mention that $-\log p_{t}$,
where $p_{t}$ satisfies a Fokker-Planck equation, may be viewed as the value
function of a stochastic control problem. In particular, in [10, p.194],
$-\log p(x,t)$ was named local entropy and various of its properties were
established. The corresponding optimal control is related to the so-called
score-function $\nabla\log p(x,t)$ of generative models of machine learning
based on flows. Local entropy was recently rediscovered in connection with an
attempt to smooth the energy landscape of deep neural networks, see [11] and
references therein.

Let $\psi(t,x)=\ln(p_{t}/\bar{p})$ be the local free energy as in [10]. Then

$\mbox{\BB E}\\{\psi(t,X_{t})\\}=\mbox{\BB D}(p_{t}\|\bar{p})=\int_{\mbox{\bb
R}^{n}}\ln\frac{p_{t}}{\bar{p}}(x)p_{t}(x)dx,$

the relative entropy or divergence or Kullback-Leibler index between $p_{t}$
and $\bar{p}$. Moreover, the free energy decays according to [4]

$\frac{d}{dt}\mbox{\BB E}\\{\psi(t,X_{t})\\}=\frac{d}{dt}\mbox{\BB
D}(p_{t}\|\bar{p})=-\frac{\sigma^{2}}{2}\int_{\mbox{\bb
R}^{n}}\|\nabla\ln\frac{p_{t}}{\bar{p}}(x)\|^{2}p_{t}(x)dx.$ (15)

Define the Varentropy ${\cal V}_{KL}(p_{t}\|\bar{p})$ as in [12]

$\displaystyle{\cal V}_{KL}(p_{t}\|\bar{p})$ $\displaystyle=$
$\displaystyle\int_{\mbox{\bb
R}^{n}}\left(\ln\frac{p_{t}(x)}{\bar{p}(x)}-\mbox{\BB
D}(p_{t}\|\bar{p})\right)^{2}p_{t}(x)dx$ $\displaystyle=$
$\displaystyle\mbox{\BB E}\left[\left(\psi(t,X_{t})-\mbox{\BB
E}(\psi(t,X_{t})\right)^{2}\right]=\mbox{\BB
E}\left\\{\psi(t,X_{t})^{2}\right\\}-\mbox{\BB E}\\{\psi(t,X_{t})\\}^{2}.$

Hence, in view of (15),

$\displaystyle\frac{d}{dt}{\cal V}_{KL}(p_{t}\|\bar{p})$ $\displaystyle=$
$\displaystyle\frac{d}{dt}\mbox{\BB
E}\left\\{\psi(t,X_{t})^{2}\right\\}-\frac{d}{dt}\left[\mbox{\BB
D}(p_{t}\|\bar{p})^{2}\right]$ (16) $\displaystyle=$
$\displaystyle\frac{d}{dt}\mbox{\BB
E}\left\\{\psi(t,X_{t})^{2}\right\\}+\mbox{\BB
D}(p_{t}\|\bar{p})\sigma^{2}\int_{\mbox{\bb
R}^{n}}\|\nabla\ln\frac{p_{t}}{\bar{p}}(x)\|^{2}p_{t}(x)dx.$

To understand the rate of change of the Varentropy, we need therefore to study

$\frac{d}{dt}\mbox{\BB
E}\left\\{\psi(t,X_{t})^{2}\right\\}=\frac{d}{dt}\mbox{\BB E}\left\\{(-\ln
M_{t})^{2}\right\\},$

where we recall that $M_{t}:=\frac{\bar{p}}{p_{t}}(X_{t})$.

######  Theorem 4.1.

Let $\psi(t,x)=\ln(p_{t}/\bar{p})$ .The rate of change of the Varentropy is
given by

$\frac{d}{dt}{\cal V}_{KL}(p_{t}\|\bar{p})=\sigma^{2}\mbox{\BB
E}\left\\{\left[-\psi(t,X_{t})-1+\mbox{\BB
E}\\{\psi(t,X_{t})\\}\right]\|\nabla\psi(t,X_{t})\|^{2}\right\\}.$ (17)

######  Proof 4.2.

We first compute the reverse-time differential of $N_{t}=(-\ln M_{t})^{2}=(\ln
M_{t})^{2}=\left[\ln\frac{p_{t}}{\bar{p}}(X_{t})\right]^{2}=\psi(t,X_{t})^{2}$.
Note that $f(x)=(\ln x)^{2}$ is not convex nor concave on $x>0$ and we have

$f^{\prime}(x)=2\ln x\cdot\frac{1}{x},\quad f^{\prime\prime}(x)=\frac{2(1-\ln
x)}{x^{2}}.$

By (14) and the generalization of (12), we get

$\displaystyle dN_{t}$ $\displaystyle=$
$\displaystyle\frac{df}{dx}(M_{t})dM_{t}-\frac{1}{2}\frac{d^{2}f}{dx^{2}}(M_{t})d\langle
M\rangle_{t}=-\frac{1-\ln
M_{t}}{M_{t}^{2}}\sigma^{2}\left\|\nabla\frac{p_{t}}{\bar{p}}\right\|^{2}dt+2\ln
M_{t}\frac{1}{M_{t}}\nabla\frac{\bar{p}}{p_{t}}\sigma d\bar{W}_{t}$ (18)
$\displaystyle=$
$\displaystyle\sigma^{2}\left(-\ln\frac{p_{t}}{\bar{p}}(X_{t})-1\right)\|\nabla\ln\frac{p_{t}}{\bar{p}}(X_{t})\|^{2}dt+2\ln\frac{p_{t}}{\bar{p}}(X_{t})\nabla\ln\frac{p_{t}}{\bar{p}}(X_{t})\sigma
d\bar{W}_{t}.$

By a locatization argument, the stochastic integral

$\int_{t_{0}}^{t_{1}}2\ln\frac{p_{t}}{\bar{p}}(X_{t})\nabla\ln\frac{p_{t}}{\bar{p}}(X_{t})\sigma
d\bar{W}_{t}$

has zero expectation. Thus, in view of (16) and (18), we get

$\displaystyle\frac{d}{dt}{\cal V}_{KL}(p_{t}\|\bar{p})$ $\displaystyle=$
$\displaystyle\frac{d}{dt}\mbox{\BB E}\left[\left(\psi(t,X_{t})-\mbox{\BB
E}(\psi(t,X_{t})\right)^{2}\right]$ (19) $\displaystyle=$
$\displaystyle\frac{d}{dt}\mbox{\BB E}\left\\{N_{t}\right\\}+\mbox{\BB
D}(p_{t}\|\bar{p})\sigma^{2}\int_{\mbox{\bb
R}^{n}}\|\nabla\ln\frac{p_{t}}{\bar{p}}(x)\|^{2}p_{t}(x)dx$ $\displaystyle=$
$\displaystyle\sigma^{2}\mbox{\BB
E}\left\\{\left(-\ln\frac{p_{t}}{\bar{p}}(X_{t})-1\right)\|\nabla\ln\frac{p_{t}}{\bar{p}}(X_{t})\|^{2}\right\\}+\mbox{\BB
D}(p_{t}\|\bar{p})\sigma^{2}\int_{\mbox{\bb
R}^{n}}\|\nabla\ln\frac{p_{t}}{\bar{p}}(x)\|^{2}p_{t}(x)dx$ $\displaystyle=$
$\displaystyle\sigma^{2}\mbox{\BB
E}\left\\{\left(-\ln\frac{p_{t}}{\bar{p}}(t,X_{t})-1+\mbox{\BB
D}(p_{t}\|\bar{p})\right)\|\nabla\ln\frac{p_{t}}{\bar{p}}(X_{t})\|^{2}\right\\}$
$\displaystyle=$ $\displaystyle\sigma^{2}\mbox{\BB
E}\left\\{\left[-\psi(t,X_{t})-1+\mbox{\BB
E}\\{\psi(t,X_{t})\\}\right]\|\nabla\psi(t,X_{t})\|^{2}\right\\}.$

##  5 The scalar Gaussian case

Consider the case of a scalar, zero-mean, Gaussian diffusion $\\{X_{t};t\geq
0\\}$ with forward It$\hat{o}$ differential

$dX_{t}=-\frac{1}{2}X_{t}dt+dW_{t}$

and initial condition $X_{0}$ distributed according to

$p_{0}(x)=\frac{1}{\sigma_{0}\sqrt{2\pi}}\exp\left[-\frac{x^{2}}{\sigma_{0}^{2}}\right].$

In this simple case, the time derivative of Varentropy can be computed
explicitly. The variance $\sigma_{t}^{2}$ of $X_{t}$ satisfies

$\frac{d}{dt}(\sigma_{t}^{2})=-\sigma_{t}^{2}+1,\quad\lim_{t\rightarrow\infty}\sigma^{2}_{t}=\bar{\sigma}^{2}=1.$

Thus the invariant density is simply the standard Normal

$\bar{p}(x)=\frac{1}{\sqrt{2\pi}}\exp\left[-x^{2}\right].$

Then

$\psi(t,x)=\ln(p_{t}/\bar{p})(x)=\ln\left\\{\frac{1}{\sigma_{t}}\exp\left[-\frac{x^{2}}{2}\left(-1+\frac{1}{\sigma_{t}^{2}}\right)\right]\right\\}=-\ln\sigma_{t}-\frac{x^{2}}{2}\left(\frac{1-\sigma_{t}^{2}}{\sigma_{t}^{2}}\right)$

It follows that

$\mbox{\BB
E}\\{\psi(t,X_{t})\\}=-\ln\sigma_{t}-\frac{1}{2}\left(1-\sigma_{t}^{2}\right).$

Observe that

$\dot{\sigma}_{t}=\frac{d}{dt}\left(\sqrt{\sigma_{t}^{2}}\right)=\frac{1}{2}\frac{1}{\sqrt{\sigma_{t}^{2}}}\frac{d}{dt}\left(\sigma^{2}_{t}\right)=\frac{1}{2\sigma_{t}}\left(1-\sigma_{t}^{2}\right).$

Then

$\frac{d}{dt}\mbox{\BB
E}\\{\psi(t,X_{t})\\}=-\frac{\dot{\sigma}_{t}}{\sigma_{t}}+\frac{1}{2}\frac{d}{dt}\left(\sigma_{t}^{2}\right)=-\frac{1}{2\sigma_{t}^{2}}\left(1-\sigma_{t}^{2}\right)+\frac{1}{2}\left(1-\sigma_{t}^{2}\right)=-\frac{1}{2}\frac{\left(1-\sigma_{t}^{2}\right)^{2}}{\sigma_{t}^{2}}.$

Recal that the Varentropy is

${\cal V}_{KL}(p_{t}\|\bar{p})=\mbox{\BB
E}\left\\{\psi(t,X_{t})^{2}\right\\}-\mbox{\BB E}\\{\psi(t,X_{t})\\}^{2}.$

We compute first $\mbox{\BB E}\left\\{\psi(t,X_{t})^{2}\right\\}$. We have

$\psi(x,t)^{2}=\left[-\ln\sigma_{t}-\frac{x^{2}}{2}\left(\frac{1-\sigma_{t}^{2}}{\sigma_{t}^{2}}\right)\right]^{2}=\left(\ln\sigma_{t}\right)^{2}+\ln\sigma_{t}x^{2}\left(\frac{1-\sigma_{t}^{2}}{\sigma_{t}^{2}}\right)+\frac{x^{4}}{4}\left(\frac{1-\sigma_{t}^{2}}{\sigma_{t}^{2}}\right)^{2}.$

As $X_{t}$ ia a centered Gaussian $\mbox{\BB
E}\\{X_{t}^{4}\\}=3\sigma_{t}^{4}$. Hence, we get

$\mbox{\BB
E}\left\\{\psi(x,t)^{2}\right\\}=\left(\ln\sigma_{t}\right)^{2}+\ln\sigma_{t}\left(1-\sigma_{t}^{2}\right)+\frac{3}{4}\left(1-\sigma_{t}^{2}\right)^{2}.$

Thus,

$\displaystyle{\cal V}_{KL}(p_{t}\|\bar{p})$ $\displaystyle=$
$\displaystyle\left(\ln\sigma_{t}\right)^{2}+\ln\sigma_{t}\left(1-\sigma_{t}^{2}\right)+\frac{3}{4}\left(1-\sigma_{t}^{2}\right)^{2}-\left[\ln\sigma_{t}+\frac{1}{2}\left(1-\sigma_{t}^{2}\right)\right]^{2}$
$\displaystyle=$
$\displaystyle\left(\ln\sigma_{t}\right)^{2}+\ln\sigma_{t}\left(1-\sigma_{t}^{2}\right)+\frac{3}{4}\left(1-\sigma_{t}^{2}\right)^{2}-\left(\ln\sigma_{t}\right)^{2}-\ln\sigma_{t}\left(1-\sigma_{t}^{2}\right)-\frac{1}{4}\left(1-\sigma_{t}^{2}\right)^{2}$
$\displaystyle=$ $\displaystyle\frac{1}{2}\left(1-\sigma_{t}^{2}\right)^{2}.$

It follows that

$\frac{d}{dt}{\cal
V}_{KL}(p_{t}\|\bar{p})=-\left(1-\sigma_{t}^{2}\right)\frac{d}{dt}\left(\sigma_{t}^{2}\right)=-\left(1-\sigma_{t}^{2}\right)^{2}=-2{\cal
V}_{KL}(p_{t}\|\bar{p}).$ (20)

Hence, Varentropy is exponentially decaying according to

${\cal V}_{KL}(p_{t}\|\bar{p})={\cal
V}_{KL}(p_{0}\|\bar{p})e^{-2t}=\frac{1}{2}\left(1-\sigma_{0}^{2}\right)^{2}e^{-2t}.$

We now try to compute the Varentropy time derivative using formula (19)

$\frac{d}{dt}{\cal V}_{KL}(p_{t}\|\bar{p})=\mbox{\BB
E}\left\\{\left[-\psi(t,X_{t})-1+\mbox{\BB
E}\\{\psi(t,X_{t})\\}\right]\left\|\frac{\partial\psi(t,X_{t})}{\partial
x}\right\|^{2}\right\\}.$ $\displaystyle\frac{d}{dt}{\cal
V}_{KL}(p_{t}\|\bar{p})$ $\displaystyle=$ $\displaystyle\mbox{\BB
E}\left\\{\left[\ln\sigma_{t}+\frac{X_{t}^{2}}{2}\left(\frac{1-\sigma_{t}^{2}}{\sigma_{t}^{2}}\right)-1-\ln\sigma_{t}-\frac{1}{2}\left(1-\sigma_{t}^{2}\right)\right]\cdot\left[X_{t}^{2}\left(\frac{1-\sigma_{t}^{2}}{\sigma_{t}^{2}}\right)^{2}\right]\right\\}$
$\displaystyle=$ $\displaystyle\mbox{\BB
E}\left\\{\frac{X_{t}^{4}}{2}\left(\frac{1-\sigma_{t}^{2}}{\sigma_{t}^{2}}\right)^{3}\right\\}-\mbox{\BB
E}\left\\{X_{t}^{2}\left(\frac{1-\sigma_{t}^{2}}{\sigma_{t}^{2}}\right)^{2}\right\\}-\frac{1}{2}\mbox{\BB
E}\left\\{X_{t}^{2}\left(1-\sigma_{t}^{2}\right)\left(\frac{1-\sigma_{t}^{2}}{\sigma_{t}^{2}}\right)^{2}\right\\}$
$\displaystyle=$
$\displaystyle\frac{3\sigma^{4}_{t}}{2}\left(\frac{1-\sigma_{t}^{2}}{\sigma_{t}^{2}}\right)^{3}-\frac{\left(1-\sigma_{t}^{2}\right)^{2}}{\sigma_{t}^{2}}-\frac{1}{2}\frac{\left(1-\sigma_{t}^{2}\right)^{3}}{\sigma_{t}^{2}}$
$\displaystyle=$
$\displaystyle\frac{3}{2\sigma_{t}^{2}}\left(1-\sigma_{t}^{2}\right)^{3}-\frac{\left(1-\sigma_{t}^{2}\right)^{2}}{\sigma_{t}^{2}}-\frac{1}{2}\frac{\left(1-\sigma_{t}^{2}\right)^{3}}{\sigma_{t}^{2}}$
$\displaystyle=$
$\displaystyle\frac{\left(1-\sigma_{t}^{2}\right)^{3}}{\sigma_{t}^{2}}-\frac{\left(1-\sigma_{t}^{2}\right)^{2}}{\sigma_{t}^{2}}$
$\displaystyle=$ $\displaystyle-\left(1-\sigma_{t}^{2}\right)^{2}$

which coincides with (20).

##  6 Closing comments

Further study, possibly based on functional inequalities, is needed to
establish if (19) implies that Varentropy ${\cal V}_{KL}(p_{t}\|\bar{p})$, in
analogy to relative entropy $\mbox{\BB D}(p_{t}\|\bar{p})$, is always
decreasing for Markov diffusions like in the Gaussian case. If not, there
could be cases where Varentropy attains a maximum at some time $t_{M}$ after
which it decreases to zero in analogy to what is observed in some Markov
chains models [12].

## References

  * [1] Föllmer, H.: Time reversal on Wiener space, in: Stochastic Processes - Mathematics and Physics , Lect. Notes in Math. 1158, Springer-Verlag, New York,1986, p. 119. MR0838561
  * [2] Föllmer, H.: Random fields and diffusion processes, in: Ècole d’Ètè de Probabilitès de Saint-Flour XV-XVII, edited by P. L. Hennequin, Lecture Notes in Mathematics, Springer-Verlag, New York, 1988, vol.1362,102-203. MR0983373
  * [3] Fradelizi, M., Li, J., Madiman, M.: Concentration of information content for convex measures, Electron. J. Probab. 25, Paper No. 20, 22, 2020. MR4073681
  * [4] Graham, R.: Path integral methods in nonequilibrium thermodynamics and statistics, in Stochastic Processes in Nonequilibrium Systems, L. Garrido, P. Seglar and P.J.Shepherd Eds., Lecture Notes in Physics 84, Springer-Verlag, New York, 1978, 82-138. MR0532361
  * [5] Haussmann, U. G. and Pardoux, E.: Time reversal of diffusions, The Annals of Probability, 1188-1205, 1986. MR0866342
  * [6] Jaynes, E. T.: Papers on Probability, Statistics and Statistical Physics, R.D. Rosenkranz ed., Dordrecht, 1983. MR0786828
  * [7] Liptser R. S. and Shiryaev, A. N.: Statistics of Random Processes I, General Theory, Springer-Verlag, 1977. MR0474486
  * [8] Nelson, E.: Dynamical Theories of Brownian Motion, Princeton University Press, Princeton, 1967. MR0214150
  * [9] Nelson, E.: Stochastic mechanics and random fields, inÈcole d’Ètè de Probabilitès de Saint-Flour XV-XVII, edited by P. L. Hennequin, Lecture Notes in Mathematics, Springer-Verlag, New York, 1988, Vol.1362, pp. 428-450. MR0983376
  * [10] Pavon, M.: Stochastic control and nonequilibrium thermodynamical systems, Appl. Math. and Optimiz. 19 1989, 187-202. MR0962891
  * [11] Pavon, M.: On Local Entropy, Stochastic Control, and Deep Neural Networks, IEEE Control Systems Letters, 7, 2022, 437-441. MR4523720
  * [12] Salez, J.: Cutoff for non-negatively curved Markov chains, ArXiv e-prints, arXiv: 2102.05597v2.