# Active Inference Meeting Energy-Efficient Control of Parallel and Identical
Machines

Yavar Taheri Yeganeh
Politecnico di Milano
<EMAIL_ADDRESS>
&Mohsen Jafari
Rutgers University
<EMAIL_ADDRESS>
&Andrea Matta
Politecnico di Milano
<EMAIL_ADDRESS>

###### Abstract

We investigate the application of active inference in developing energy-
efficient control agents for manufacturing systems. Active inference, rooted
in neuroscience, provides a unified probabilistic framework integrating
perception, learning, and action, with inherent uncertainty quantification
elements. Our study explores deep active inference, an emerging field that
combines deep learning with the active inference decision-making framework.
Leveraging a deep active inference agent, we focus on controlling parallel and
identical machine workstations to enhance energy efficiency. We address
challenges posed by the problem’s stochastic nature and delayed policy
response by introducing tailored enhancements to existing agent architectures.
Specifically, we introduce multi-step transition and hybrid horizon methods to
mitigate the need for complex planning. Our experimental results demonstrate
the effectiveness of these enhancements and highlight the potential of the
active inference-based approach.

_Keywords_ Active Inference $\cdot$ Probabilistic Deep Learning $\cdot$
Reinforcement Learning $\cdot$ Energy-Efficient Control $\cdot$ Manufacturing
Systems

##  1 Introduction

Active inference (AIF), an emerging field inspired by the principles of
biological brains, offers a promising alternative for decision-making models.
It unifies perception, learning, and decision-making under the free energy
principle (FEP), which formulates neuronal inference and learning under
uncertainty [1]. Accordingly, the brain is modeled through levels of
(variational) Bayesian inference [2], minimizing prediction errors by
leveraging a generative model of the world while considering uncertainties.
This framework enables the development of agents that can calibrate their
models and make decisions without complete knowledge of system dynamics.
Significant progress has been made in applying active inference across various
domains, including robotics, autonomous driving, and healthcare [3, 4, 5],
showcasing its ability to handle complex decision-making tasks in dynamic
environments.

Recently, the manufacturing industry’s focus on energy efficiency has
intensified due to its significant contribution to global energy consumption.
Addressing energy efficiency at the machine level has become critical, with
energy-efficient scheduling (EES) and energy-efficient control (EEC)
strategies emerging as key approaches to reducing environmental impact [6].
While traditional EEC methods often necessitate complete system knowledge,
reinforcement learning (RL) has shown potential in optimizing manufacturing
processes without prior system knowledge [7, 8]. However, RL agents may
struggle to rapidly adjust their policies to changing conditions.

This research aims to build upon advancements in active inference-based
decision-making [9, 10] and apply it to EEC in manufacturing systems,
demonstrating its potential and advancing the understanding of active
inference in complex environments. The existing active inference agents often
rely on extensive search algorithms during planning and make decisions based
on immediate next predictions [9], which pose challenges in the context of the
EEC problem. By employing deep active inference as the decision-making
algorithm, we introduce tailored enhancements, such as multi-step transition
and hybrid horizon methods, to address the challenges posed by the problem’s
stochastic nature and delayed policy response. Our experimental results
highlight the effectiveness of these enhancements and underscore the potential
of the active inference-based approach. The remainder of the paper is
organized as follows: we begin by concisely introducing the EEC problem and
the manufacturing system under study. We then present an overview of the
formalization of active inference, describe the agent, evaluate its
performance, and discuss future directions.

##  2 Application Overview

Active inference has proven effective in various applications commonly
associated with decision-making processes in biological agents, such as humans
and animals. These applications primarily involve visual sensory output as
observations. For instance, Fountas et al. (2020) [9] tested their agent on
tasks like Dynamic dSprites [11] and Animal-AI [12], which can be performed by
biological agents simply. Additionally, applications in robotics [13, 14]
(e.g., manipulation [4]) align with tasks that human agents can typically
perform naturally. This effectiveness stems from active inference being a
theory of decision-making for biological agents [2]. However, certain
applications, such as the control of industrial systems, can present complex
challenges. While the decision-making processes and existing applications
mentioned above may not be straightforward, human agents may struggle to
devise effective policies for these more intricate problems.

###  2.1 EEC in Manufacturing Systems

EEC gaining prominence in both academic and industrial circles within
manufacturing systems. It offers substantial energy savings across three key
control levels: component, machine, and production system. At its core, EEC
involves managing the power consumption state of objects based on
environmental conditions. Objects are kept fully operational when their
functions are needed and transitioned to low power states when not in use,
though this poses challenges due to the unpredictable nature of environmental
demands and the penalties incurred during state transitions. These penalties
include both time wasted during transition, when the object adds no value, and
the energy consumed during the transition process. A comprehensive and recent
literature review on this topic can be found in [15].

###  2.2 System Description

The specific system under study is a manufacturing workstation consisting of a
single upstream buffer serving multiple parallel machines as depicted in Fig.
1. Key characteristics of this system include a finite buffer capacity,
stochastic arrival of parts, and identical machines operating in a mass-
production context. Each machine can exist in various states including
working, standby, startup, and failed, with further subdivisions within the
working state to denote idle and busy states. Additionally, machines are
subject to stochastic failures and repairs, each with expected durations,
further complicating control strategies.

Figure 1: Layout of parallel machines.

Power consumption varies based on the machine’s state, with different power
requirements for standby, startup, idle, and busy states. For instance,
standby and failed states require minimal power consumption, while idle and
startup states demand higher levels. Furthermore, the highest power
consumption occurs during the busy state when the machine is fully operational
and actively processing parts. Managing these power variations efficiently is
crucial for effective energy-efficient control in manufacturing systems.

##  3 Deep Active Inference Agent

Active inference is a unifying theory that integrates inference, perception,
and action by emphasizing the dependence of observations on actions [16]. To
effectively manage observations toward preferred states, the optimization of
actions plays a crucial role [16]. This concept was originally proposed as a
framework for understanding how organisms actively control and navigate their
environment by iteratively updating their beliefs and actions based on sensory
evidence [2]. The FEP [17, 18] is at the core of active inference, paving the
way for creating a mathematical model, and there are even experimental
evidences supporting it [19]. We modify the proposed agent by Fountas et al.
(2020) [9], which exhibited notable capabilities and performance when compared
against three benchmark model-free RL algorithms in two image applications.

###  3.1 Active Inference Formalism

Active inference agents employ an integrated probabilistic framework
consisting of an internal generative model [10] coupled with inference
mechanisms to represent and interact with the world. Similar to RL the agent
interacts with the environment, but using three random variables representing
observation, latent state, and action (i.e., $(o_{t},s_{t},a_{t})$ at time
$t$). The framework assumes a Partially Observable Markov Decision Process
(POMDP) [20, 10, 21]. The generative model of the agent, parameterized with
$\theta$, is defined over these variables (i.e.,
$P_{\theta}(o_{1:t},s_{1:t},a_{1:t-1})$) [9]. Generally, the agent acts to
reduce surprise, which can be quantified by $-\log P_{\theta}(o_{t})$.
Specifically, there are two steps for the agent while interacting with the
world [2, 9] as follows:

1) The agent calibrates its generative model through fitting predictions and
improve its representation of the world. This is done by minimizing
Variational Free Energy (VFE), which is similar to surprise of predictions in
connection with the actual observations [9, 22, 21], as follows:

${\theta}^{*}=\arg\min_{\begin{subarray}{c}\theta\end{subarray}}\left(\mathbb{E}_{Q_{\phi}(s_{t},a_{t})}\left[\log
Q_{\phi}(s_{t},a_{t})-\log P_{\theta}(o_{t},s_{t},a_{t})\right]\right)\ .$ (1)

This objective function is commonly known as the negative evidence lower bound
(ELBO) [23], which is the upper bound for $-\log P_{\theta}(o_{t})$. It is
also used as a foundation for training variational autoencoders [24].

2) The agent makes decisions (i.e., chooses actions) in active inference based
on the accumulated negative Expected Free Energy (EFE or $G$):

$P(\pi)\stackrel{{\scriptstyle}}{{=}}{\sigma}(-G(\pi))\ {=}\
\sigma\left(-\sum_{\tau>t}{G}(\pi,\tau)\right)\ ,$ (2)

where $\sigma(\cdot)$ represents the _Softmax_ function, and $\pi$ (i.e.,
policy) denotes the sequence of actions. The EFE encompasses minimizing
surprise regarding preferred observations111Here, the surprise refers to the
preference, while in VFE, surprise pertains to the actual observation used for
calibrating the model., exploring uncertainty, and reducing uncertainty about
model parameters [9]. The EFE for $\tau\geq t$ can be formulated as follows
[25]:

$G(\pi,\tau)=\mathbb{E}_{{P}(\sigma_{\tau}|s_{\tau},\theta)}\mathbb{E}_{{\cal
Q}_{\phi}(s_{\tau},\theta|\pi)}\left[\log
Q_{\phi}(s_{\tau},\theta|\pi)-\log{P}(o_{\tau},s_{\tau},\theta|\pi)\right]\ .$
(3)

Fountas et al., (2020) [9] provided a derivation [25] for calculating the EFE
in Eq. 3 at each time step:

$\displaystyle G(\pi,\tau)$ $\displaystyle=-\mathbb{E}_{\tilde{Q}}\left[\log
P(o_{\tau}|\pi)\right]$ (4a)
$\displaystyle\quad+\mathbb{E}_{\tilde{Q}}\left[\log Q(s_{\tau}|\pi)-\log
P(s_{\tau}|o_{\tau},\pi)\right]$ (4b)
$\displaystyle\quad+\mathbb{E}_{\tilde{Q}}\left[\log
Q(\theta|s_{\tau},\pi)-\log P(\theta|s_{\tau},o_{\tau},\pi)\right].$ (4c)

They expanded the formalism, leading to a tractable estimate for EFE that is
both interpretable and calculable [9]:

$\displaystyle G(\pi,\tau)$
$\displaystyle=-\mathbb{E}_{Q(\theta|\pi)Q(s_{\tau}|\theta,\pi)Q(o_{\tau}|s_{\tau},\theta,\pi)}\left[\log\textstyle
P(o_{\tau}|\pi)\right]$ (5a)
$\displaystyle\quad+\mathbb{E}_{Q(\theta|\pi)}\left[\,\mathbb{E}_{Q(o_{\tau}|\theta,\pi)}H(s_{\tau}|o_{\tau},\pi)-H(s_{\tau}|\pi)\right]$
(5b)
$\displaystyle\quad+\mathbb{E}_{Q(\theta|\pi)Q(s_{\tau}|\theta,\pi)}H(o_{\tau}|s_{\tau},\theta,\pi)-\mathbb{E}_{Q(s_{\tau}|\pi)}H(o_{\tau}|s_{\tau},\pi)\
.$ (5c)

This paved the way for establishing a unified formalism for computing
decisions in Eq. 2. Accordingly, actions are connected to perception, which is
achieved through Bayesian inference, based on the EFE in Eq. 5.

Through this formalism that leads to calculating the EFE (i.e., Eq. 5), we can
interpret the contribution of each element [9]: The first term (i.e., Eq. 5a)
is analogous to reward in RL as it is the surprise222Here, instead of
maximizing cumulative rewards, the focus is on minimizing the surprise, which
quantifies the extent of deviation (i.e., misalignment) between the prediction
and the preferred observation. of the prediction considering preferred
observation. The second term (i.e., Eq. 5b) represents state uncertainty,
which is mutual information between the agent’s beliefs about state before and
after prediction. This term also shows a motivation to explore areas of the
environment that resolve state uncertainty [9]. The third term (i.e., Eq. 5c)
represents uncertainty about model parameters considering new observations.
This term is also referred active learning, novelty, or curiosity [9]. In
fact, model parameters (i.e., $\theta$), particularly contribute to making
predictions, including generation of the next states.

Figure 2: The illustration depicts two views of the active inference
framework: general steps on the left and active inference elements on the
right.

In summary, the framework (as depicted in Fig. 2) is realized through a
mathematical formalism in the following manner. The observation is fed as
input, which propagates through the model to create perception (i.e.,
beliefs), which include generating future states. It is to facilitate the
calculation of EFE (in Eq. 5) integrated into the planner derived from policy
(in Eq. 2) to act on the environment. After obtaining the next observation the
VFE (in Eq. 1) can be calculated, which calibrate (i.e., learning) the model
based on the matching the new observation with the prediction. In fact, every
time the framework goes through the loop the model is optimized based on the
VFE form the previous loop, then the optimized model is used to for the rest.

###  3.2 Architecture

An agent within the active inference framework needs different modules, which
are entangled within the framework. Amortization [26, 27] is introduced into
the formalism (in Sec. 3.1) to scale-up the realization [9]. The formalism is
inherently probabilistic and parameterized with with two sets,
$\theta=\\{{\theta}_{s},{\theta}_{o}\\}$ representing generative and
$\phi=\\{{\phi}_{s}\\}$ recognition elements [9]. Using the following
parameterized modules the formalism (in Sec. 3.1) can be calculated: Encoder
(i.e., $Q_{\phi_{s}}(s_{t})$), an amortized inference of the hidden state
(i.e., an inference network [28] providing a mapping between the observation,
$\tilde{o}_{t}$, and a distribution for its corresponding hidden state).
Transition (i.e., ${P}_{\theta_{s}}({s}_{t+1}|\tilde{s}_{t},\tilde{a}_{t})$),
which generates a distribution for the next hidden state based on both a
sampled action and the current hidden state. Decoder (i.e.,
${P}_{\theta_{o}}({o}_{t+1}|\tilde{s}_{t+1})$) generates a distribution for
prediction based on the sampled hidden state.

Neural networks can facilitate the realization of these modules by
representing a mapping between a sample and the respective distribution. In
fact, parameters of a pre-selcted (e.g., Gaussian) distribution can be
approximated. Here, we model the state space specifically with a multivariate
Gaussian distribution, assuming no covariance (i.e., diagonal Gaussian). Using
the VFE (i.e., Eq. 1), all the three networks can be trained in an end to end
fashion. Aside from action and the transition, which can even be considered
integrated within the state space, the structure bears a resemblance to a
variational autoencoder [24]. It is noteworthy that training for the both
include optimizing ELBO, as specified in Eq. 1.

EncoderTransitionDecoder$\tilde{o}_{t}$$\tilde{o}_{t+1}$$\tilde{o}_{t}$$P_{\theta_{s}}(s_{t+1}|\tilde{s}_{t},\tilde{a}_{t})$$P_{\theta_{o}}(o_{t+1}|\tilde{s}_{t+1})$$Q_{\phi_{s}}(s_{t})$$\tilde{a}_{t}\
,\ \Delta_{t}\ $ Figure 3: The agent’s architecture and generative framework
resemble that of a VAE. The line on top represents the agent simulating the
future and making a prediction, while on the bottom, the agent receives a new
observation after $\Delta_{t}$ of taking an action, $\tilde{a}_{t}$ .

Utilizing the mentioned architecture (also depicted in Fig. 3), the agent
would be able to calculate the EFE in Eq. 5 for a given policy (i.e., $\pi$),
which is a sequence of actions. Thus, we can calculate the probabilities for
selecting actions through Eq. 2. In order to make better decisions, the agent
can plan ahead by simulating future trajectories using the architecture
mentioned above. However, as the policy space grows exponentially into the
future, it is infeasible to evaluate all possible scenarios. Fountas et al.
(2020) [9] introduced two approaches to alleviate the obstacle. 1) They used
the standard Monte-Carlo Tree Search (MCTS) [29, 30], a tree-based search
algorithm, which selectively explore promising trajectories in a restricted
manner. 2) They introduced another recognition module [31, 32, 27],
parameterized with ${\phi}_{a}$ as follows: Habit (i.e.,
$Q_{\phi_{a}}(a_{t})$), an amortized inference of actions (i.e., an inference
network [28] providing a mapping between a sampled hidden state,
$\tilde{s}_{t}$, and normalized, through _Softmax_ function, probabilities for
the actions). Habit is also realized through a neural network, which
approximates the posterior distribution over actions (i.e.,
${P}({a}_{t}|{s}_{t})$) using the prior ${P}({a}_{t})$ that is obtained from
the MCTS [9]. This network is trained to reproduce the last action sampled
form the planner, given the last state. This is similar to the fast and
habitual decision-making in biological agents [33].

Fountas et al. (2020) [9] followed the standard four steps of MCTS [34], which
let them to restrict and prioritize the trajectories that should be evaluated.
Iterativly a weighted tree that has memory updates the visited states. During
each loop, a path from the existing tree (towards a leaf node) is selected
(i.e., selection) based on the following upper bound confidence:

$U(s_{t},a_{t})=\tilde{G}(s_{t},a_{t})+c_{\mathrm{explore}}\cdot
Q_{\phi_{a}}(a_{t}|s_{t})\cdot\frac{1}{1+N(a_{t},s_{t})}$ (6)

Where $\tilde{G}(s_{t},a_{t})$ represents the algorithm’s current estimation
for the EFE, while $N(a_{t},s_{t})$ denotes the number of times a node (i.e.,
a state and action pair) in the tree has been visited during the tenure, along
with an exploration hyperparameter, $c_{\mathrm{explore}}$.Then, starting from
the leaf and for every possible action (i.e., expansion), the EFE is
calculated for a fixed number of future steps (i.e., simulation). This value
is finally added to all nodes along the path to calculate the average of
$\tilde{G}(s_{t},a_{t})$ (i.e., backpropagation). After all, Actions are
sampled from the probabilities created by
$P(a_{t})=\frac{N(a_{t},s_{t})}{\sum_{j}N(a_{t},j,s_{t})}$ (where
$P(a_{t})=\sum_{\pi:a_{1}=a_{t}}P(\pi)$), which is proportional to the number
of times a node is visited. The planning process continues until a maximum
number of loops (i.e., a hyperparameter) is reached, or a condition, i.e.,
$\text{max}P(a_{t})-\text{mean}P(a_{t})>T_{dec}$, is met, indicating that the
planning is finished. Fountas et al. (2020) [9] further employed
$Q_{\phi_{a}}(a_{t})$ (i.e., habit) to modulate the state space, motivated by
incorporating uncertainty. The divergence between the policy obtained from the
planner (i.e., MCTS) and the habit (i.e.,
$D_{t}=D_{\mathrm{KL}}\left[Q_{\phi_{a}}(a_{t})\left|\right|P(a_{t})\right]$)
can serve as a loss function for the habit. This divergence also represents a
type of uncertainty in the state space, preventing the habit from being
identical to the policy [35, 36]. Therefore, they used $D_{t}$ in a logistic
function as follows:

$\bm{\omega}_{t}=\,\frac{\alpha}{1\,+\,e^{-\,\frac{b-D_{t-1}}{c}}}\,+\,d\ .$
(7)

The monotonically decreasing pattern establishes a reciprocal connection
between $D_{t-1}$ and $\bm{\omega}_{t}$, i.e., the state precision, using
hyperparameters $\\{\alpha,b,c,d\\}$. Altogether, the transition (i.e.,
${P}_{\theta_{s}}({s}_{t}|{s}_{t-1},{a}_{t-1})$) is modeled with the
distribution $\mathcal{N}(\mu,\,\sigma^{2}/\bm{\omega}_{t})$ [9]. This
precision is analogous to $\beta$ in $\beta\text{-VAE}$ [11], effectively
promoting disentanglement of the state space by the encoder [9].

To facilitate the computations and effectively estimating different elements
within the framework, Fountas et al. (2020) [9] used several levels of Monte-
Carlo simulations (i.e., sampling based estimations). In addition to MCTS, all
terms in the EFE (Eq. 5), including the use of MC dropout [37] for model
parameters (i.e., 5c), are estimated in this manner [9].

###  3.3 Enhancements

We explore various aspects to design the agent and leverage the formalism and
existing architecture outlined by Fountas et al. (2020) [9] as a foundation.
First, we examine how features and requirements related to the problem under
study can influence the agent. Subsequently, we propose solutions to address
these issues, introducing a coherent agent design.

####  3.3.1 Exploring the Application

The simulation of the system, described in Section 2.1, is designed to
replicate features of an industrial system. It utilizes Poisson processes [38]
(i.e., exponential distributions) for machine state transitions and part
arrivals [7]. The _event-driven_ steps employed by Loffredo et al. (2023) [7,
8] trigger decisions after a system state transition rather than at fixed
intervals, proving effective for controlling working machines. This problem
can be viewed as either continuous-time stochastic control or a discrete-time
Markov Chain process [39]. Continuous-time modeling requires making time
intervals visible to the agent for both machines and subsequent observations,
whereas discrete-time modeling allows the agent to learn dynamics by observing
transitions to create probabilities for different transitions. There is a
variable time interval between subsequent observations (i.e., $\Delta t$ as
depicted in Fig. 3). This variability requires synchronizing the transition
for prediction (i.e., $P_{\theta_{o}}(o_{t+1}|\tilde{s}_{t+1})$) with the next
observation (i.e., $\tilde{o}_{t+1}$) in the continuous-time model.
Incorporating continuous-time facilitates neural network function
approximation for longer $\Delta t$ during planning. However, using
continuous-time can further complicate the prediction structure since
residence times for machine states exist in observation. Here, we utilize
discrete-time _event-driven_ steps that simplify this process compared to the
continuous-time approach.

The stochastic nature and the integral and continuous form of the reward
functions for the system under study [7, 8] mean decision impacts may not be
immediately observable, aligning with POMDPs. The system has a delay in
responding to the policy, which we call a _long/delayed impact horizon_ ,
particularly with respect to reward. This is an important distinction from
environments evaluated by Fountas et al. (2020) [9]. The agent proposed by
them focuses on planning based on immediate next predictions. The problem at
hand is continuous with no terminal state, and the span over which reward
functions are integrated may encompass a few thousand steps. This complicates
planning and highlights the need for less computational cost.

####  3.3.2 Experience Replay

The generative model encompasses the core of active inference [17, 2, 9] and
predictive coding [16]. Therefore, the performance of any active inference-
based agent heavily relies on its accuracy. To improve model training, we
introduce experience replay [40] using a memory that stores
$(o_{t},a_{t},o_{t+1})$ at different steps. During training, we sample a batch
of experiences from the memory and ensure the latest experience is also
included. However, for all the batched experiences, we utilize
$\bm{\omega}_{t}$ based on the latest experience.

####  3.3.3 Hybrid Horizon

To address the limitations arising from the short horizon of the EFE, which
relies on immediate next predictions, we propose augmenting the planner with
an auxiliary term to account for longer horizons. Q-learning [41] and its
enhanced variant, deep Q-learning, can serve as model-free planners with
longer horizons, leveraging rewards to update the Q-value for a state-action
pair [42]. Loffredo et al. (2023) [9] demonstrated that deep Q-learning can
achieve near-optimal performance for the systems under study. Accordingly, we
modify $Q_{\phi_{a}}(a_{t})$ to represent amortized inference of actions,
mapping observations $\tilde{o}_{t}$ (or sampled predictions) to normalized
action probabilities using a _Softmax_ function and training it with deep
Q-learning updates based on rewards coming from experience replay. We
introduce a hyperparameter $\gamma$ to balance the contributions of long and
short horizons. Thus, we arrive at a new formulation for the planner to
incorporate longer horizons:

$P(a_{t})=\gamma\cdot
Q_{\phi_{a}}(a_{t})+(1-\gamma)\cdot{\sigma}\left(-G(\pi)\right).$ (8)

The resulting combination achieves a controlled balance between the EFE for
the short horizon terms, which incorporates uncertainties, and the long
horizon term. We further utilize the new $Q_{\phi_{a}}(a_{t})$ to modulate the
agent’s state uncertainty based on Eq. 7. In fact, $D_{t}$ (i.e.,
$D_{\mathrm{KL}}\left[Q_{\phi_{a}}(a_{t})\left|\right|P(a_{t})\right]$)
represents the discrepancy between the long horizon and the combined policy,
reflecting a form of knowledge gap for the agent.

####  3.3.4 Multi-Step Transition and Planning

Given the stochastic nature and _long impact horizon_ of the system under
study, a one-step transition (as depicted in Fig. 3) may not result in
significant changes in observation and state, leading to indistinguishable EFE
terms. Therefore, the model should learn transitions beyond one step and
predict further into the future to distinguish the impact of different
policies. We modify the transition module to allow multiple steps, controlled
by a hyperparameter (e.g., $s=90$), enabling multi-step transitions given the
policy (i.e., sequence of actions). Representing the sequence of actions in a
policy as a one-hot vector can be high-dimensional, so we utilize integer
encodings as an approximation. This is feasible since the actions (or number
of machines) are less categorical and can be considered rather continuous in
this case. During planning, we utilize repeated actions in the transition for
each action and calculate the EFE accordingly. This method assesses the impact
of actions over a short period, using repeated action simulations. This
approximation helps to distinguish different actions over a horizon based on
the EFE. Thus, even a single multi-step transition can serve as a simple and
computationally efficient planner. For deeper simulations, it can be combined
with MCTS of repeated transitions. Alternatively, it can be combined with a
less expensive planner that starts with repeated transitions for each action,
followed by simulating until a specific depth using the following policy:

$\tilde{P}(\pi,a_{\tau})=(1-c_{\mathrm{explore}})\cdot
Q_{\phi_{a}}(a_{\tau})+c_{\mathrm{explore}}\cdot\sigma\left(-\log
P(o_{\tau}|\pi)\right),$ (9)

to calculate the EFE of the final state or trajectory. After several loops,
the accumulated EFE for each action can be used in Eq. 8.

##  4 Results

To demonstrate the potential of our methodology, we concentrate our
experiments on controlling a real industrial workstation comprising six
parallel-identical machines with an upstream capacity of 10, as outlined in
[7]. All the stochastic processes characterizing the workstation follow
Poisson distributions, and thus are exponentially distributed with different
expected values. We first introduce the preference function, then describe the
setup and training, and finally present the numerical performance of our
agent333The source code will be available on GitHub at
https://github.com/YavarYeganeh/AIF_Meeting_EEC..

###  4.1 Preference Function

Active inference involves an agent acting to achieve its preferred
observation, similar to the concept of a setpoint in control theory [35, 16].
Consequently, the agent possesses an internal function to quantify the
proximity of the prediction to the preferred state. This function is important
for the agent’s performance. While it correlates with the reward function in
reinforcement learning, it is based on a different philosophy: a control
setpoint rather than the cumulative reward in the MDP framework of
reinforcement learning [42]. Instead of the reward function used by Loffredo
et al. (2023) [8], we propose a different preference function for the multi-
objective optimization of the system under study. This function aligns with
the concept of a _long/delayed impact horizon_ system for our agent to
control, accounting for the average performance of the system over a fixed
time span ($t_{s}$444In our implementation, we use the closest recorded
timestamp from the system, respecting the fixed time span., e.g., 8 hours)
leading up to the observation at ($t$). It includes terms for production,
energy consumption, and a combined term as follows:

$\displaystyle R_{\text{production}}$
$\displaystyle=\frac{T_{\text{current}}}{T_{\text{max}}}$ (10a) $\displaystyle
R_{\text{energy}}$ $\displaystyle=1-\frac{E_{\text{avg}}}{E_{\text{max}}}$
(10b) $\displaystyle R$ $\displaystyle=\phi\cdot
R_{\text{production}}+\left(1-\phi\right)\cdot R_{\text{energy}}\ ,$ (10c)

where:

  * •

$\phi$ is the weighting coefficient balancing production and energy
consumption. It is set close to 1 (0.97 in our implementation) to ensure that
the agent does not significantly reduce production.

  * •

$T_{\text{current}}=\frac{NP(t)-NP(t-t_{s})}{t_{s}}$ represents the throughput
within the past $t_{s}$ period, where $NP(t)$ is the number of parts produced
within this period.

  * •

$T_{\text{max}}$ is the maximum achievable throughput, occurring under the
_ALL ON_ policy [8].

  * •

$E_{\text{avg}}=\frac{C(t)-C(t-t_{s})}{t_{s}}$ represents the average energy
consumption over the past $t_{s}$ period, where $C(t)$ is the total energy
consumption within this period.

  * •

$E_{\text{max}}$ is the maximum theoretical consumption rate of the system,
pertaining to all machines operating in their busy state.

###  4.2 Agent Setup and Training

We adhere to the MC sampling methodology for calculating EFE, as well as most
agent configurations, as outlined by Fountas et al. (2020) [9]. Notably, we
employ Bernoulli and Gaussian distributions to model prediction and state
distributions, respectively. We introduced a modification by utilizing
activation functions for the encoder and transition networks. The output of
these two networks generates means and variances representing Gaussian
distributions for the state. We applied the _Tangent Hyperbolic_ function for
the means and the _Sigmoid_ function, multiplied by a factor, $\lambda_{s}$,
between 1 and 2, for the variances. This enforces further regularization and
stability for the state space to prevent unbounded values, which need to be
fitted into a normal distribution. In our implementation, in contrast to [9],
which does not use activations for the state, these activations are essential
for learning an effective policy.

Our agent’s observation comprises buffer levels and machine states, all one-
hot encoded. Similarly, we incorporate the three reward terms in Eq. 10,
predicting only their means without pre-defining a distribution. The agent’s
preference (i.e., $P(o_{\tau}|\pi)$) corresponds to the prediction of the
combined reward term for the system, which the agent seeks to maximize. Given
the composition of binary and continuous components of the observation, the
loss of the VAE’s reconstruction is equally scaled aggregation of binary
cross-entropy and mean square error. Additionally, due to the one-hot
structure of the observation, we sample the predictions, excluding the reward
terms, which are treated as means, to be fed into the decoder during the
calculation of EFE. As we validate the performance of a control system for an
industrial application, we use a single system during each training epoch,
with an experience replay size of 200. These systems are initialized with a
random policy for one day of simulation, after removing the profile from one
day of warm-up simulation using the _ALL ON_ policy. We trained our agents on-
policy, following a similar approach to the algorithm proposed by Fountas et
al. (2020) [9], but with adjustments to accommodate the modifications we made,
including multi-step transition, experience replay, and a hybrid planner.

###  4.3 Agent Performance

To assess the efficacy of our agents, we tested their performance several
times during different training epochs, each on independent systems
initialized with a random agent after warm-up, similar to those used for
training. We simulated the interaction between the agent and the system, which
was randomly controlled at the beginning, for one day of simulation time. The
combined reward term (in Eq. 10) at the end of the test simulation was
extracted as the performance measure, with a time span of $t_{s}$ (i.e., 8
hours in this case). Our agent, similar to [9], has a large set of
hyperparameters, which proved to be sensitive, especially those related to the
state space of the model. We considered both 1-step transitions and multi-step
transitions, taking repeated actions during planning for each of the possible
actions (i.e., determining how many machines to keep _ON_) to then calculate
their EFE. Fig. 4 presents the comparison for a single set of hyperparameters,
particularly $\lambda_{s}=1$, except $s$, across different $\gamma$. It shows
the suitability of the multi-step transition and simple repeated planner as
well as hybrid horizon.

Figure 4: Comparison of test rewards during training of agents with 90-step
transition against 1-step transition, when $\lambda_{s}=1$.

Since the state should fit the normal distribution, $\lambda_{s}=1$ for
variance will target areas of the input domain of the _Sigmoid_ function that
are saturated and have smaller gradients. To address this, we increased
$\lambda_{s}$ to 1.5, where the _Sigmoid_ function has larger gradients. This
results in higher rewards even for very small $\gamma$ (i.e., 0.05), as
presented in Fig. 5.

Figure 5: The test performance of the agent with 90-step transition and
repeated actions for planning, when $\lambda_{s}=1.5$ and $\gamma=0.05$,
replicated with 10 different random seeds. A: Average reward. B: Average
planner distribution for different actions. C: Average EFE’s term 2 for
different actions.

This performance is primarily coming from the second term of the EFE (i.e.,
Eq. 5b), which serves as an _intrinsic reward_ [21], as demonstrated in Fig.
5C. The other two terms are less distinguishable for different actions on
average. This suggests that the agent differentiates between various actions
in its state space to achieve high rewards. In fact, the _long impact horizon_
and stochasticity, as well as our approximations in transition and planning,
hinder the predictive power of the agent, but it managed to infer the impact
of different actions in its state space. It is also worth noting that our
agent quickly converges to high rewards but may experience instability and
loss of control if training continues for (a long time), due to a
_catastrophic_ increase in the loss, particularly the reconstruction term of
the generative model. This necessitates early stopping mechanisms for training
and the introduction of regularization elements (e.g., dropout and
normalization) to prevent or mitigate the issue.

##  5 Conclusion and Future Work

The results of our study demonstrate the effectiveness of our proposed
modifications for the active-inference-inspired agent. Notably, a single
multi-transition lookahead with repeated actions, coupled with a hybrid
horizon, can achieve high rewards without the need for extensive planning
algorithms like MCTS. This is particularly notable considering the unique
challenges posed by the application under study, which demands deep lookahead
into the future to distinguish different actions, while stochasticity presents
a challenge. The significant impact of hyperparameters on agent performance
emphasizes the importance of meticulous tuning for optimal results and
effective control. This highlights the potential of our methodology to tackle
the EEC problem and similar applications characterized by a _long/delayed
impact horizon_ and high stochasticity.

Improving the methodology can involve enhancing the generative model, the core
of active inference, especially by introducing recurrent transitions or
enhancing predictive capabilities. Integration of diffusion-based generative
models instead of VAEs [5] is also a promising direction. The framework and
formalism of active inference agents show promise for non-stationary
scenarios, where model-free agents may struggle to adapt swiftly. Future
research will concentrate on improving the agent, extending experimental
validation, and tailoring the methodology to non-stationary scenarios,
leveraging the strengths of active inference to develop more robust and
efficient decision-making algorithms.

## References

  * Friston [2010] Karl Friston.  The free-energy principle: a unified brain theory?  _Nature reviews neuroscience_ , 11(2):127–138, 2010.
  * Parr et al. [2022] Thomas Parr, Giovanni Pezzulo, and Karl J Friston.  _Active inference: the free energy principle in mind, brain, and behavior_.  MIT Press, 2022.
  * Pezzato et al. [2023] Corrado Pezzato, Carlos Hernández Corbato, Stefan Bonhof, and Martijn Wisse.  Active inference and behavior trees for reactive action planning and execution in robotics.  _IEEE Transactions on Robotics_ , 39(2):1050–1069, 2023.
  * Schneider et al. [2022] Tim Schneider, Boris Belousov, Georgia Chalvatzaki, Diego Romeres, Devesh K Jha, and Jan Peters.  Active exploration for robotic manipulation.  In _2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 9355–9362. IEEE, 2022.
  * Huang et al. [2024] Yufei Huang, Yulin Li, Andrea Matta, and Mohsen Jafari.  Navigating autonomous vehicle on unmarked roads with diffusion-based motion prediction and active inference.  _arXiv preprint arXiv:2406.00211_ , 2024.
  * Loffredo et al. [2024] Alberto Loffredo, Nicla Frigerio, Ettore Lanzarone, and Andrea Matta.  Energy-efficient control in multi-stage production lines with parallel machine workstations and production constraints.  _IISE Transactions_ , 56(1):69–83, 2024.
  * Loffredo et al. [2023a] Alberto Loffredo, Marvin Carl May, Louis Schäfer, Andrea Matta, and Gisela Lanza.  Reinforcement learning for energy-efficient control of parallel and identical machines.  _CIRP Journal of Manufacturing Science and Technology_ , 44:91–103, 2023a.  ISSN 1755-5817.
  * Loffredo et al. [2023b] Alberto Loffredo, Marvin Carl May, Andrea Matta, and Gisela Lanza.  Reinforcement learning for sustainability enhancement of production lines.  _Journal of Intelligent Manufacturing_ , pages 1–17, 2023b.
  * Fountas et al. [2020] Z. Fountas, Noor Sajid, Pedro A. M. Mediano, and Karl J. Friston.  Deep active inference agents using monte-carlo methods.  _ArXiv_ , abs/2006.04176, 2020.
  * Da Costa et al. [2023] Lancelot Da Costa, Noor Sajid, Thomas Parr, Karl Friston, and Ryan Smith.  Reward maximization through discrete active inference.  _Neural Computation_ , 35(5):807–852, 2023.
  * Higgins et al. [2016] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner.  beta-vae: Learning basic visual concepts with a constrained variational framework.  In _International conference on learning representations_ , 2016.
  * Crosby et al. [2019] Matthew Crosby, Benjamin Beyret, and Marta Halina.  The animal-ai olympics.  _Nature Machine Intelligence_ , 1(5):257–257, 2019.
  * Lanillos et al. [2021] Pablo Lanillos, Cristian Meo, Corrado Pezzato, Ajith Anil Meera, Mohamed Baioumy, Wataru Ohata, Alexander Tschantz, Beren Millidge, Martijn Wisse, Christopher L Buckley, et al.  Active inference in robotics and artificial agents: Survey and challenges.  _arXiv preprint arXiv:2112.01871_ , 2021.
  * Da Costa et al. [2022] Lancelot Da Costa, Pablo Lanillos, Noor Sajid, Karl Friston, and Shujhat Khan.  How active inference could help revolutionise robotics.  _Entropy_ , 24(3):361, 2022.
  * Renna and Materi [2021] Paolo Renna and Sergio Materi.  A literature review of energy efficiency and sustainability in manufacturing systems.  _Applied Sciences_ , 11(16):7366, 2021.
  * Millidge et al. [2022] Beren Millidge, Tommaso Salvatori, Yuhang Song, Rafal Bogacz, and Thomas Lukasiewicz.  Predictive coding: towards a future of deep learning beyond backpropagation?  _arXiv preprint arXiv:2202.09467_ , 2022.
  * Friston et al. [2010] Karl Friston, Francesco Rigoli, Dimitri Ognibene, Christoph Mathys, Thomas Fitzgerald, and Giovanni Pezzulo.  Action and behavior: A free-energy formulation.  _Biological Cybernetics_ , 102(3):227–260, 2010\.
  * Millidge [2021] Beren Millidge.  Applications of the free energy principle to machine learning and neuroscience.  _arXiv preprint arXiv:2107.00140_ , 2021.
  * Isomura et al. [2023] Takuya Isomura, Kiyoshi Kotani, Yasuhiko Jimbo, and Karl J Friston.  Experimental validation of the free-energy principle with in vitro neural networks.  _Nature Communications_ , 14(1):4547, 2023.
  * Kaelbling et al. [1998] Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra.  Planning and acting in partially observable stochastic domains.  _Artificial intelligence_ , 101(1-2):99–134, 1998\.
  * Paul et al. [2023] Aswin Paul, Noor Sajid, Lancelot Da Costa, and Adeel Razi.  On efficient computation in active inference.  _arXiv preprint arXiv:2307.00504_ , 2023.
  * Sajid et al. [2022] Noor Sajid, Francesco Faccio, Lancelot Da Costa, Thomas Parr, Jürgen Schmidhuber, and Karl Friston.  Bayesian brains and the rényi divergence.  _Neural Computation_ , 34(4):829–855, 2022.
  * Blei et al. [2017] David M Blei, Alp Kucukelbir, and Jon D McAuliffe.  Variational inference: A review for statisticians.  _Journal of the American statistical Association_ , 112(518):859–877, 2017.
  * Kingma and Welling [2013] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_ , 2013.
  * Schwartenbeck et al. [2019] Philipp Schwartenbeck, Johannes Passecker, Tobias U Hauser, Thomas HB FitzGerald, Martin Kronbichler, and Karl J Friston.  Computational mechanisms of curiosity and goal-directed exploration.  _elife_ , 8:e41703, 2019.
  * Gershman and Goodman [2014] Samuel Gershman and Noah Goodman.  Amortized inference in probabilistic reasoning.  In _Proceedings of the annual meeting of the cognitive science society_ , volume 36, 2014.
  * Marino et al. [2018] Joe Marino, Yisong Yue, and Stephan Mandt.  Iterative amortized inference.  In _International Conference on Machine Learning_ , pages 3403–3412. PMLR, 2018.
  * Margossian and Blei [2023] Charles C Margossian and David M Blei.  Amortized variational inference: When and why?  _arXiv preprint arXiv:2307.11018_ , 2023.
  * Silver et al. [2017] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al.  Mastering the game of go without human knowledge.  _nature_ , 550(7676):354–359, 2017.
  * Coulom [2006] Rémi Coulom.  Efficient selectivity and backup operators in monte-carlo tree search.  In _International conference on computers and games_ , pages 72–83. Springer, 2006.
  * Piché et al. [2018] Alexandre Piché, Valentin Thomas, Cyril Ibrahim, Yoshua Bengio, and Chris Pal.  Probabilistic planning with sequential monte carlo methods.  In _International Conference on Learning Representations_ , 2018.
  * Tschantz et al. [2020] Alexander Tschantz, Beren Millidge, Anil K Seth, and Christopher L Buckley.  Control as hybrid inference.  _arXiv preprint arXiv:2007.05838_ , 2020.
  * Van Der Meer et al. [2012] Matthijs Van Der Meer, Zeb Kurth-Nelson, and A David Redish.  Information processing in decision-making systems.  _The Neuroscientist_ , 18(4):342–359, 2012.
  * Świechowski et al. [2023] Maciej Świechowski, Konrad Godlewski, Bartosz Sawicki, and Jacek Mańdziuk.  Monte carlo tree search: A review of recent modifications and applications.  _Artificial Intelligence Review_ , 56(3):2497–2562, 2023.
  * Friston et al. [2017] Karl Friston, Thomas FitzGerald, Francesco Rigoli, Philipp Schwartenbeck, and Giovanni Pezzulo.  Active inference: a process theory.  _Neural computation_ , 29(1):1–49, 2017.
  * Byers and Serences [2012] Anna Byers and John T Serences.  Exploring the relationship between perceptual learning and top-down attentional control.  _Vision research_ , 74:30–39, 2012.
  * Gal and Ghahramani [2016] Yarin Gal and Zoubin Ghahramani.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In _international conference on machine learning_ , pages 1050–1059. PMLR, 2016.
  * Kingman [1992] John Frank Charles Kingman.  _Poisson processes_ , volume 3.  Clarendon Press, 1992.
  * Ross [2014] Sheldon M Ross.  _Introduction to probability models_.  Academic press, 2014.
  * Mnih et al. [2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al.  Human-level control through deep reinforcement learning.  _nature_ , 518(7540):529–533, 2015.
  * Watkins and Dayan [1992] Christopher JCH Watkins and Peter Dayan.  Q-learning.  _Machine learning_ , 8:279–292, 1992.
  * Sutton and Barto [2018] Richard S Sutton and Andrew G Barto.  _Reinforcement learning: An introduction_.  MIT press, 2018.


#  BankNote-Net: Open dataset for assistive universal currency recognition

Felipe Oviedo<EMAIL_ADDRESS>Microsoft AI for Good Research Lab1
Microsoft WayRedmondWAUSA98052 ,  Srinivas Vinnakota  Microsoft1 Microsoft
WayRedmondWAUSA98052 ,  Eugene Seleznev  Microsoft Seeing AI1 Microsoft
WayRedmondWAUSA98052 ,  Hemant Malhotra  Microsoft1 Microsoft
WayRedmondWAUSA98052 ,  Saqib Shaikh  Microsoft Seeing AI1 Microsoft
WayRedmondWAUSA98052 and  Juan Lavista Ferres  Microsoft AI for Good Research
Lab1 Microsoft WayRedmondWAUSA98052

(2021)

###### Abstract.

Millions of people around the world have low or no vision. Assistive software
applications have been developed for a variety of day-to-day tasks, including
optical character recognition, scene identification, person recognition, and
currency recognition. This last task, the recognition of banknotes from
different denominations, has been addressed by the use of computer vision
models for image recognition. However, the datasets and models available for
this task are limited, both in terms of dataset size and in variety of
currencies covered. In this work, we collect a total of 24,826 images of
banknotes in variety of assistive settings, spanning 17 currencies and 112
denominations. Using supervised contrastive learning, we develop a machine
learning model for universal currency recognition. This model learns compliant
embeddings of banknote images in a variety of contexts, which can be shared
publicly (as a highly-compressed vector representation), and can be used to
train and test specialized downstream models for any currency, including those
not covered by our dataset or for which only a few real images per
denomination are available (few-shot learning). We deploy a variation of this
model for public use in the last version of the Seeing AI app developed by
Microsoft. We share our encoder model and the embeddings as an open dataset in
our BankNote-Net repository.

assistive technology, vision impairment, low vision, blindness, computer
vision, machine learning, contrastive learning

††copyright: acmcopyright††journalyear: 2021††doi: XXXXXXX.XXXXXXX††ccs:
Human-centered computing Accessibility technologies

##  1\. Introduction

According to recent data (Bourne et al., 2021), 43 million individuals
worldwide are estimated to suffer blindness and over 200 million people suffer
moderate or severe vision impairment. 55% of people in this group are women
and 89% live in low and middle-income countries. Due to population growth and
higher incidence of certain pathologies (Bourne et al., 2021), the number of
people affected by blindness and moderate to severe vision loss is expected to
increase to over 61 million and 235 million people by 2050, respectively.

Given the magnitude of the problem, the software community has developed
numerous mobile applications and wearable technologies to aid visually
impaired individuals in a multitude of tasks (Manduchi and Kurniawan, 2018;
Tapu et al., 2020; Hakobyan et al., 2013), including navigation, public
transport use, scene understanding, person recognition, currency note
recognition, etc. Seeing AI (Microsoft, 2022), a major application in the
space developed by Microsoft, is a pioneer of applying AI to assist
individuals with low and no vision and currently has 100 thousand average
monthly users in 80 countries, supporting vision-to-speech in 19 languages.
One common task of assistive applications consists in the recognition (i.e.
multi-class classification) of banknotes of different denominations for a
specific currency. Machine learning (ML) is often applied for this task.
Several ML models have been develop for particular currencies (Raval and Shah,
2019), including Indian Rupee (Veeramsetty et al., 2020; Upadhyay et al.,
2020; Jain et al., 2021; Rahman et al., 2014), US dollar (Hasanuzzaman et al.,
2012; Gai et al., 2013; Paisios et al., 2011), Canadian dollar (Paisios et
al., 2011), Euro (Raval and Shah, 2018), Bangladesh Taka (Islam et al., 2021),
Thailand Baht (Takeda et al., 2003), Egyptian Pound (Semary et al., 2015),
Pakistan Rupee (Imad et al., 2020), among others (Raval and Shah, 2019). The
techniques employed in previous works include both classical computer vision
and machine learning approaches, such as LDA (Rahman et al., 2014), feature
transformations and template-matching (Raval and Shah, 2018; Gai et al., 2013;
Semary et al., 2015), support vector machines (Upadhyay et al., 2020), and
modern deep learning approaches (Islam et al., 2021; Veeramsetty et al., 2020;
Takeda et al., 2003; Imad et al., 2020).

Either for training or for testing purposes, these models rely on a relatively
large dataset of labelled banknote images spanning all the denominations of
certain currency. If the model is going to be deployed in a mobile assistive
application, as is common for the banknote recognition task, production
quality models require well-curated datasets in a variety of real assistive
scenarios, including images with poor focus, various backgrounds, occlusions,
blur, varying orientation and perspective, etc. The lack of comprehensive
public datasets has made limited the development of these models to a few
particular countries and currencies.

In this work, our goal is to facilitate the training of machine learning
models for the recognition of any banknote in assistive settings. For this
purpose, we collect, curate and label BankNote-Net: the largest open dataset
of banknote images for accessibility to date, composed of a total of 24,826
images of front and back banknote faces across 17 currencies and 112
denominations. To approximate its use by an individual with low or no vision,
the images are captured in a variety of orientations, lightning, background,
occlusion and image quality conditions. Using supervised contrastive learning
(Khosla et al., 2020), we use this dataset to train a universal deep learning
model for multi-class classification spanning the 17 currencies and 224
classes (2 faces per each one of the 112 denominations). As the reproduction
of high-resolution currency images is regulated in some jurisdictions, such as
as England (of England, 2022) or Japan (of Japan, 2022), we use this model to
learn highly compressed and compliant embeddings for the images. Using leave-
one-group-out cross validation, we demonstrate these embeddings are useful for
pre-training models for currencies beyond those covered by our dataset for
both the abundant data setting and the few-shot learning case. Figure 1
summarizes our approach. These embeddings, along with the pre-trained encoding
model, are shared as a public dataset in our BankNote-Net repository. In
addition, we use our dataset for successful validation and deployment of a
variation of our model in production, as part of the most recent version of
the Seeing AI app (Microsoft, 2022).

Figure 1. Our approach: we use a convolutional neural network (MobileNet V2)
as an encoder for our assistive banknote images. Using supervised contrastive
learning, the models learns an highly-compressed and descriptive embedding for
each image. We use these embeddings for downstream tasks, such as recognition
of Yen banknotes, as shown in the figure. Our final dataset is composed of the
learned embeddings for 17 currencies and 224 classes. ”Dense Classifier”
corresponds to a fully-connected neural network, and ”GAP” corresponds to a
Global Average Pooling layer.

##  2\. Data collection and curation

Deploying reliable ML models for banknote recognition requires images of
banknotes in real assistive scenarios, i.e. varying background, illumination,
blur, orientation, occlusion and focus conditions. Although, high quality
scans of multiple currencies are available in banknote collectors’ websites
such as (BankNote.sw, 2022; Database, 2022), abundant data of real usage
scenarios for assistive applications is very limited. In consequence, we
design our data collection process with this accessibility application in
mind.

The final data collection process consisted of the following steps:

  1. (1)

Currency selection: we selected the particular currencies to collect according
to population and geographic diversity, and according to areas with high
number of users of the Microsoft Seeing AI app.

  2. (2)

Recruitment: for each specific currency, we on-boarded consenting volunteers
or Amazon Mechanical Turk (Amazon, 2022) workers to perform data collection.

  3. (3)

Training: each worker or volunteer was provided with a brochure of
requirements for the desired pictures along with example pictures. This
brochure is included in Appendix A.

  4. (4)

Data collection: each worker or volunteer captured and labelled up around 100
images for each orientation (front or back) of each denomination of a specific
currency. Each picture was captured with a square aspect ratio to facilitate
preprocessing. The images of each currency were captured exclusively by 1 or 2
volunteers or workers located in the country where a specific currency is
used; this decision was made to improve data completeness across all
denominations of a given currency.

  5. (5)

Data validation and preparation: the collected data was validated manually by
the authors to check for inconsistencies in labelling or limited image
diversity. Then, the file formats were standardized and all images were
resized to 224 x 224 pixels.

In total, 24,826 images were collected from 17 currencies. Figure 2a presents
the distribution of collected images across currencies, the distribution of
images per class, and multiple examples of the collected images. The images
span 112 denominations and 224 classes (each denomination maps to two classes,
corresponding to the front and back faces of the banknote). The mean number of
images per class is 110, as shown in Figure 2. For banknotes that have more
than a one version in circulation, we defined each banknote version as a
different class. Appendix B has a list of all denominations and classes in our
dataset.

Figure 2. Overview of BankNote-Net: a) Total number of images per currency. b)
Distribution of number of images per class (combination of denomination and
front/back side, e.g. ”5 USD - Front Side” is a class). c) Example of images
in the dataset, collected in diverse accessibility scenarios.

##  3\. Data Representation and Evaluation

The BankNote-Net dataset can be used for various computer vision tasks,
including fine-tuning of recognition models for those currencies included in
the dataset, robust validation of recognition models, and transfer learning
and few-shot learning to novel currencies. For this purpose, we encode all
BankNote-Net images as embeddings and make them publicly available. In the
following subsections, we present our encoder model and demonstrate the
utility of our embeddings.

###  3.1. Learning banknote representations

We are interested in learning a compact and compliant representation of
banknote images that can be used for downstream image recognition tasks. For
this purpose, we train a neural network image encoder using a supervised
contrastive learning loss (Khosla et al., 2020). The supervised contrastive
learning loss combines a contrastive loss with a cross entropy loss to learn
improved discriminative representations across all classes. Since most
assistive applications are deployed in mobile devices, we use the MobileNet V2
architecture (Sandler et al., 2018) ($\alpha=1.3$, global average pooling for
feature extraction, Image-Net weights, temperature for supervised contrastive
loss T=$0.05$, ADAM optimizer with $\textit{lr}=5e^{-5}$ and
$\textit{epochs}=150$) as the encoder backbone, as shown in Figure 1. The
encoder outputs a 256-dimensional vector embedding $v\in\mathbb{R}^{1\times
256}$. During training with the supervised contrastive loss, we find
beneficial to freeze the bottom layers in the backbone architecture up to
layer $n=96$. $n$, along all other discussed hyper-parameters, are chosen by
hyperparameter tuning of the mean AUC across all classes on a 10% random
validation set, using the Optuna package (Akiba et al., 2019). During
training, we use random rotation, cropping and channel shift to augment the
data.

For each image in our dataset, this model returns a highly-informative and
compact embedding, which can be used for downstream tasks, such as multi-class
classification, transfer learning or few-shot learning. Our machine learning
model is summarized in Figure 1.

###  3.2. Leave-currency-out cross validation

We perform a leave-one-group-out cross validation procedure to confirm the
predictive power of our embeddings. For this, we hold out one of the 17
currencies in the dataset at a time, and train our encoder with the remaining
currencies. For each held-out currency, we split data according to two kinds
of splits that we called: full split and few-shot split. The full split
corresponds to a 80%-20% random train-test split, representing a setting with
500-1500+ images per denomination. The few-shot split corresponds to a 10%-90%
train-test split, representing a few-shot learning setting with 5-20 training
images per class.

For each one these held-out data splits, we train an encoder model and use it
as a feature extractor for the testing currency. On top of this model, we
train a classification head consisting of a global average pooling layer and 2
fully connected layers ($\textit{dropout rate}=0.5$, Adam optimizer,
$\textit{lr}=1e-03$, $\textit{epochs}=50$). For each test currency, we average
the results over 5 random seeds. It is important to emphasize that the test
currency is never used during the training and hyperparameter tuning of the
encoder. We evaluate three different encoder initialization scenarios:

  * •

BankNote-Net: Pre-training encoder with Banknote-Net embeddings of all
training currencies, leaving out testing currency. Frozen encoder, training
and testing of top layers on test currency.

  * •

ImageNet: Pre-training encoder with ImageNet. Frozen encoder, training and
testing of top layers on test currency.

  * •

Random: Training of encoder and top layers from scratch, all layers trainable.

Figure 3 compares the accuracy, recall and precision for the three
initialization scenarios and the two kinds of data splits. We observe that the
Banknote-Net embeddings lead to superior and less variable performance in both
the full and few-shot splits. In the few-shot split, we observe a notable
boost in performance, confirming the generalization power of our approach to
unseen currencies, even when data is scarce. We note that the accuracy and
recall using BankNote-Net weights outperform those models that use other
initialization, including ImageNet. The precision for the BankNote-Net
initialization is comparable to ImageNet, and is significantly superior to
training the model from scratch. The performance on unseen currency data is
comparable to models trained exclusively on a single currency, such as (Jain
et al., 2021; Imad et al., 2020; Islam et al., 2021; Paisios et al., 2011).

Figure 3. Distribution of accuracy, recall and precision for all test
currencies after leave-currency-out cross validation. We use a particular
initialization (BankNote-Net, ImageNet, Random) and a specific train-test
split (Full-split or Few-Shot split).

###  3.3. Public embeddings

As discussed, the reproduction of high-resolution banknote images is regulated
in some jurisdictions, such as as England (of England, 2022) or Japan (of
Japan, 2022). Thus, we make our dataset public in the form of embeddings,
which are distilled and compliant representations of the banknote images.
Although our original images are only 224x224 pixels in size and are not high
resolution scans of banknotes, we provide the following analysis for the
benefit of the reader.

Our embeddings can be seen as a lossy hash of the collected images. To
empirically demonstrate this, we use Shannon coding theorem (Cheng et al.,
2020; Shannon, 2001). According to the theorem, the maximum lossless
compressed image size $C$ using pixel-wise 8-bit compression is:

(1) $C\leq S\>\frac{H}{8}$

Here, S is the original image size (in bytes) and H is the Shannon entropy of
an image.

Modern lossless image compression algorithms, such as LZMA and DEFLATE,
exploit the local structure in a image to surpass the limit defined by
Equation 1 (Cheng et al., 2020). If the image compression algorithm is allowed
to be lossy, as in JPEG, it can be further compressed; however, the image will
not be able to be decompressed without resolution losses. Our embedding
technique compresses the image significantly way beyond commonly used lossy
compression algorithms, which in turn makes high resolution reconstruction of
an image from the embedding infeasible.

We illustrate this fact in Figure LABEL:embedding. In the top figure we
present the t-SNE representation for the embeddings of 5 major currencies in
the dataset. The points naturally cluster around currencies, which confirms
the discriminative power of our embeddings. In the bottom figure, we present
the original file size (in kilobytes (KB), before downsizing to 224x224
pixels) of each image in the dataset according to its entropy. The Shannon
coding theorem sets a theoretical limit for the compressed data points as
described in Equation 1. As discussed, lossless JPEG surpasses this limit,
specially for cases with low image entropy. The 50% lossy JPEG image improves
on the lossless JPEG compression, with most images in the dataset located
below the Shannon coding bounds. Our BankNote-Net embeddings surpass the best
compression ratio by 1-2 orders of magnitude. This significant compression
level, combined with the initial image resizing step, makes lossless and high
resolution reconstructions of the images in the dataset infeasible.

Figure 4. Top: t-SNE representation of the embeddings of 5 major currencies in
the BankNote-Net dataset. The embeddings naturally cluster individual
currencies together, aiding predictive performance of models learned on the
BankNote-Net data. Other currencies in the dataset are omitted for plot
readability. Bottom: Compressed size (in Kilobytes) for each image in the
dataset as a function of entropy. Our encoded (BankNote-Net) representation is
several orders of magnitude smaller than lossy and lossless compression
algorithms, making high-resolution reconstruction infeasible.

##  4\. Open Dataset and Production Deployment

Originally, the Seeing AI app supported recognition of 7 currencies that were
deployed and serviced as independent models for each currency. With Banknote-
Net, we substantially expanded the currencies covered by the Seeing AI app, by
using the dataset to train and validate a variation of the universal model for
all 17 supported currencies. In this case, we perform further hyperparameter
tuning to maximize the model precision for predictions with confidence of 0.99
or above. We choose this approach to minimize false positives during assistive
use. This single universal model is deployed in the last version of the Seeing
AI mobile app.

The embeddings, labels and the pre-trained MobileNet V2 encoder model
developed during work are available in the BankNote-Net repository.

##  5\. Discussion and Conclusions

In this work, we collect and make public a large dataset of banknote images
for assistive recognition. To share the dataset, we use supervised contrastive
learning to get learn compliant and highly-compressed embeddings for each of
the 24,826 images and the 17 currencies in the dataset. We demonstrate the
power of these embeddings to improve the predictive performance of currency
recognition models for assistive applications, including the few-shot
scenario. Additionally, we demonstrate that the embeddings cannot be feasibly
decompressed to high-resolution images.

Our dataset has several limitations to consider. In particular, most of the
images of a given currency were collected by 1-2 volunteers or workers.
Although the workers were clearly aiming to optimize diversity of settings,
backgrounds and orientations for the images, the final diversity of the
dataset can be limited by bias during data collection, inherent specifications
of the mobile camera employed, limited background for image capture, etc. This
constitutes an important trade-off we made for the data collection process,
and we chose to perform it in this way to maximize completeness across the
denominations within a given currency. Regardless, we show that the learned
embeddings have substantial generalization power, including to currencies not
covered by the BankNote-Net dataset. This makes the dataset very useful for
future applications, for instance to extend the model to non-supported
currencies or to include new versions of banknotes that were not considered in
the dataset. In the future, we may consider expanding the dataset to achieve a
broader diversity of data collectors.

Another important consideration is that the deployment of the model in
production may require the optimization of particular metric of interest
beyond precision and recall, e.g. the high confidence precision mentioned in
Section 4. In this case, the dataset user may benefit from re-training or
fine-tuning the encoder on a representative validation dataset to achieve good
performance on the custom metric.

###### Acknowledgements.

The authors thank Ria Sankar and Heila Precel for helpful discussion. We
acknowledge the contributions of all data collectors across the world.

##  6\. Data and Code Availability

The BankNote-Net dataset, along with an encoder, labels and usage examples is
available as an open dataset under the MIT and CDLA-Permissive-2.0 licenses in
the BankNote-Net repository.

## References

  * (1)
  * Akiba et al. (2019) Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019.  Optuna: A next-generation hyperparameter optimization framework. In _Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining_. 2623–2631.
  * Amazon (2022) Amazon. 2022.  Amazon Mechanical MTurk.  https://www.mturk.com/.
  * BankNote.sw (2022) BankNote.sw. 2022\.  baknote.ws.  http://www.banknote.ws/.
  * Bourne et al. (2021) Rupert Bourne, Jaimie D Steinmetz, Seth Flaxman, Paul Svitil Briant, Hugh R Taylor, Serge Resnikoff, Robert James Casson, Amir Abdoli, Eman Abu-Gharbieh, Ashkan Afshin, et al. 2021\.  Trends in prevalence of blindness and distance and near vision impairment over 30 years: an analysis for the Global Burden of Disease Study.  _The Lancet Global Health_ 9, 2 (2021), e130–e143.
  * Cheng et al. (2020) Xinghua Cheng, Zhilin Li, et al. 2020\.  How does Shannon’s source coding theorem fare in prediction of image compression ratio with current algorithms?  (2020).
  * Database (2022) IBNS Database. 2022\.  IBNS.  https://www.theibns.org/.
  * Gai et al. (2013) Shan Gai, Guowei Yang, and Minghua Wan. 2013.  Employing quaternion wavelet transform for banknote classification.  _Neurocomputing_ 118 (2013), 171–178.
  * Hakobyan et al. (2013) Lilit Hakobyan, Jo Lumsden, Dympna O’Sullivan, and Hannah Bartlett. 2013.  Mobile assistive technologies for the visually impaired.  _Survey of ophthalmology_ 58, 6 (2013), 513–528.
  * Hasanuzzaman et al. (2012) Faiz M Hasanuzzaman, Xiaodong Yang, and YingLi Tian. 2012\.  Robust and effective component-based banknote recognition for the blind.  _IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews)_ 42, 6 (2012), 1021–1030.
  * Imad et al. (2020) Muhammad Imad, Farhat Ullah, Muhammad Abul Hassan, et al. 2020\.  Pakistani Currency Recognition to Assist Blind Person Based on Convolutional Neural Network.  _Journal of Computer Science and Technology Studies_ 2, 2 (2020), 12–19.
  * Islam et al. (2021) Md Islam, Mohiuddin Ahmad, Akash Shingha Bappy, et al. 2021\.  Real-Time Bangladeshi Currency Recognition Using Faster R-CNN Approach for Visually Impaired People.  In _Communication and Intelligent Systems_. Springer, 147–156.
  * Jain et al. (2021) Abhishek Jain, Paras Jain, and Vikas Tripathi. 2021\.  Automated Currency Recognition Using Neural Networks.  In _Research in Intelligent and Computing in Engineering_. Springer, 835–841.
  * Khosla et al. (2020) Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020.  Supervised contrastive learning.  _arXiv preprint arXiv:2004.11362_ (2020).
  * Manduchi and Kurniawan (2018) Roberto Manduchi and Sri Kurniawan. 2018.  _Assistive technology for blindness and low vision_.  CRC Press.
  * Microsoft (2022) Microsoft. 2022\.  Seeing AI app.  https://www.microsoft.com/en-us/ai/seeing-ai.
  * of England (2022) Bank of England. 2022\.  Using images of banknotes — Bank of England.  https://www.bankofengland.co.uk/banknotes/using-images-of-banknotes.
  * of Japan (2022) Bank of Japan. 2022\.  Laws concerning Banknote Reproduction : Bank of Japan.  https://www.boj.or.jp/en/note_tfjgs/note/security/gizo0410a.htm/.
  * Paisios et al. (2011) Nektarios Paisios, Alex Rubinsteyn, Vrutti Vyas, and Lakshminarayanan Subramanian. 2011\.  Recognizing currency bills using a mobile phone: an assistive aid for the visually impaired. In _Proceedings of the 24th annual ACM symposium adjunct on User interface software and technology_. 19–20.
  * Rahman et al. (2014) Shafin Rahman, Prianka Banik, and Shujon Naha. 2014\.  LDA based paper currency recognition system using edge histogram descriptor. In _2014 17th International Conference on Computer and Information Technology (ICCIT)_. IEEE, 326–331.
  * Raval and Shah (2018) Vishwas Raval and Apurva Shah. 2018.  HORBoVF—A Novel Three-Level Image Classifier Using Histogram, ORB and Dynamic Bag of Visual Features. In _International Conference on ISMAC in Computational Vision and Bio-Engineering_. Springer, 895–902.
  * Raval and Shah (2019) Vishwas Raval and Apurva Shah. 2019.  A Comparative analysis on currency recognition systems for iCu e and an improved grab-cut for android devices.  In _Emerging Technologies in Data Mining and Information Security_. Springer, 279–286.
  * Sandler et al. (2018) Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. 2018.  Mobilenetv2: Inverted residuals and linear bottlenecks. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 4510–4520.
  * Semary et al. (2015) Noura A Semary, Sondos M Fadl, Magda S Essa, and Ahmed F Gad. 2015\.  Currency recognition system for visually impaired: Egyptian banknote as a study case. In _2015 5th International Conference on Information & Communication Technology and Accessibility (ICTA)_. IEEE, 1–6.
  * Shannon (2001) Claude Elwood Shannon. 2001\.  A mathematical theory of communication.  _ACM SIGMOBILE mobile computing and communications review_ 5, 1 (2001), 3–55.
  * Takeda et al. (2003) Fumiaki Takeda, Lalita Sakoobunthu, and Hironobu Satou. 2003\.  Thai banknote recognition using neural network and continues learning by DSP unit. In _International Conference on Knowledge-Based and Intelligent Information and Engineering Systems_. Springer, 1169–1177.
  * Tapu et al. (2020) Ruxandra Tapu, Bogdan Mocanu, and Titus Zaharia. 2020\.  Wearable assistive devices for visually impaired: A state of the art survey.  _Pattern Recognition Letters_ 137 (2020), 37–52.
  * Upadhyay et al. (2020) Anand Upadhyay, Shashank Shukla, and Verlyn Pinto. 2020\.  Indian Currency Recognition Using Radial Basis Function and Support Vector Machine. In _International Conference on Intelligent Computing and Smart Communication 2019_. Springer, 1049–1056.
  * Veeramsetty et al. (2020) Venkataramana Veeramsetty, Gaurav Singal, and Tapas Badal. 2020\.  Coinnet: platform independent application to recognize Indian currency notes using deep learning techniques.  _Multimedia Tools and Applications_ 79 (2020), 22569–22594.

##  Appendix A Appendix A

The data collection instructions and image examples shared with data
collectors are included below.

##  Appendix B Appendix B

List of classes in the BankNote-Net dataset, composed (in order) of a currency
Forex abbreviation, denomination, face (front or back) and, if applicable,
variation of banknote.