# JITA4DS: Disaggregated execution of Data Science Pipelines between the Edge
and the Data Centre

Genoveva Vargas-Solar (1) French Council of Scientific Research (CNRS)-LIRIS
(1),
University of Arizona (2)
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Ali Akoglu (2) French Council of Scientific
Research (CNRS)-LIRIS (1),
University of Arizona (2)
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Md Sahil Hassan (2) French Council of Scientific
Research (CNRS)-LIRIS (1),
University of Arizona (2)
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### keywords:

Disaggregated Data Centers, Data Science Pipelines, Edge Computing.

### Abstract

This paper targets the execution of data science (DS) pipelines supported by
data processing, transmission and sharing across several resources executing
greedy processes. Current data science pipelines environments provide various
infrastructure services with computing resources such as general-purpose
processors (GPP), Graphics Processing Units (GPUs), Field Programmable Gate
Arrays (FPGAs) and Tensor Processing Unit (TPU) coupled with platform and
software services to design, run and maintain DS pipelines. These one-fits-all
solutions impose the complete externalization of data pipeline tasks. However,
some tasks can be executed in the edge, and the backend can provide just in
time resources to ensure ad-hoc and elastic execution environments.

This paper introduces an innovative composable ”Just in Time Architecture” for
configuring DCs for Data Science Pipelines (JITA-4DS) and associated resource
management techniques. JITA-4DS is a cross-layer management system that is
aware of both the application characteristics and the underlying
infrastructures to break the barriers between applications,
middleware/operating system, and hardware layers. Vertical integration of
these layers is needed for building a customizable Virtual Data Center (VDC)
to meet the dynamically changing data science pipelines’ requirements such as
performance, availability, and energy consumption. Accordingly, the paper
shows an experimental simulation devoted to run data science workloads and
determine the best strategies for scheduling the allocation of resources
implemented by JITA-4DS.

##  1 Introduction

Data infrastructures such as Google, Amazon, eBay, and E-Trade are powered by
data centers (DCs) with tens to hundreds of thousands of computers and storage
devices running complex software applications. Existing IT architectures are
not designed to provide an agile infrastructure to keep up with the rapidly
evolving next-generation mobile, big data, and data science pipelines demands.
These applications are distinct from the ”traditional” enterprise ones because
of their size, dynamic behavior, and nonlinear scaling and relatively
unpredictable growth as inputs being processed. Thus, they require continuous
provisioning and re-provisioning of DC resources [13, 18, 28] given their
dynamic and unpredictable changes in the Service Level Objectives (SLOs)
(e.g., availability response time, reliability, energy).

This paper targets the execution of data science (DS) pipelines supported by
data processing, transmission and sharing across several resources executing
greedy processes. A Data Science Pipeline consists of a set of data processing
tasks organized as a data flow defining the data dependencies among the tasks
and a control flow defining the order in which tasks are executed. Current
data science pipelines environments promote high-performance cloud platforms
as backend support for completely externalizing their execution. These
platforms provide various infrastructure services with computing resources
such as general-purpose processors (GPP), Graphics Processing Units (GPUs),
Field Programmable Gate Arrays (FPGAs) and Tensor Processing Unit (TPU)
coupled with platform and software services to design, run and maintain DS
pipelines. These one-fits-all solutions impose the complete externalization of
data pipeline tasks that assume (i) reliable and completely available network
connection; (ii) can be energy and economically consuming, allocating large
scale resources for executing pipelines tasks. However, some tasks can be
executed in the edge, and the backend can provide just in time resources to
ensure ad-hoc and elastic execution environments.

Our research investigates architectural support, system performance metrics,
resource management algorithms, and modeling techniques to enable the design
of composable (disaggregated) DCs. The goal is to design an innovative
composable ”Just in Time Architecture” for configuring DCs for Data Science
Pipelines (JITA-4DS) and associated resource management techniques. DCs
utilize a set of flexible building blocks that can be dynamically and
automatically assembled and re-assembled to meet the dynamic changes in
workload’s Service Level Objectives (SLO) of current and future DC
applications. DCs configured using JITA-4DS provide ad-hoc environments
efficiently and effectively meet the continuous changes in data-driven
applications or workloads (e.g., data science pipelines). To assess
disaggregated DC’s, we study how to model and validate their performance in
large-scale settings. Accordingly, the paper shows an experimental simulation
devoted to run data science workloads and determine the best strategies for
scheduling the allocation of resources implemented by JITA-4DS.

The remainder of the paper is organized as follows. Section 2 discusses
related work identifying certain drawbacks and issues we believe remain open.
Section 3 JITA-4DS the just in time edge-based data science pipeline,
execution environment proposed in this paper. Section 4 describes preliminary
results regarding the use of JITA-4DS for executing data sciencce workloads.
Finally Section 5 concludes the paper and discusses future work.

##  2 Related Work

The work introduced in this paper is related to two types of approaches: (i)
disaggregated data centers willing to propose alternatives to one fits all
architectures; and (ii) data science pipelines’ execution platforms relying on
cloud services for running greedy data analytics tasks.

###  2.1 Disaggregated data centers

Disaggregation of IT resources has been proposed as an alternative
configuration for data centers. Compared to the monolithic server approach, in
a disaggregated data center, CPU, memory and storage are separate resource
blades interconnected via a network. The critical enabler for the
disaggregated data center is the network and management software to create the
logical connection of the resources needed by an application [27]. The
industry has started to introduce systems that support a limited
disaggregation capability. For example, the Synergy system by Hewlett Packard
Enterprise (HPE) [5], and the Unified Computing System (UCS) [10] M series
servers by Cisco are two commercial examples of composable infrastructures.

HPE Synergy allows the CPU to be decoupled from the storage and memory, but
components remain physically close together. Liqid [6] provides proprietary
software that dynamically composes physical servers from the pools of bare
metal resources such as compute pool, graphics pool, storage pool, and
networking pool. DriveScale’s composable infrastructure [2] collocates the
cluster of diskless compute units and storage units within a rack. The work
introduced in [27] proposes a disaggregated data center network architecture,
with a scheduling algorithm designed for disaggregated computing.

####  2.1.1 Data Science Environments

Data analytics stacks environments provide the underlying infrastructure for
managing data, implementing data processing workflows to transform them, and
executing data analytics operations (statistics, data mining, knowledge
discovery, computational science processes). For example, the Berkeley Data
Analytics Stack (BDAS) from the AMPLAb project is a multi-layered architecture
that provides tools for virtualizing resources, addressing storage, data
processing and querying as underlying tools for big data-aware applications.
AsterixDB from the Asterix project is a full-fledged big data stack designed
as a scalable, open-source Big Data Management System (BDMS [1]).

Cloud based Data Science Environments Three families of environments provide
tools to explore, engineer and analyse data collections. (i) data analytics
stacks; They are notebook oriented environments externalised on the cloud. A
notebook is a JSON document, following a versioned schema, and containing an
ordered list of input/output cells which can contain code, text using Markdown
[7] mathematics, plots and rich media. They provide data labs and environments
with libraries for defining and executing notebooks. Examples of existing data
labs are Kaggle [4] and CoLab [3] from Google, and Azure Notebooks from
Microsoft Azure [8].

####  2.1.2 Externalised Data Programming environments

can be installed locally or externalized in clouds. This environments are
based on the notion of notebook that Thereby, experiments can become
completely and absolutely replicable. Data Labs with Runtime Environments
allow data scientists to find and publish data sets, explore and build models
in a web-based data-science environment, and work with other data scientists
and machine learning engineers. Data labs offer a public data platform, a
cloud-based workbench for data science, and Artificial Intelligence
experiments.

Platforms for custom modelling provide a suite of machine learning tools
allowing developers with little experience to train high-quality models. Tools
are provided as services by commercial cloud providers that include storage,
computing support and environments for training and enacting greedy artificial
intelligence (AI) models. The leading vendors providing this kind of platforms
are Amazon Sage Maker, Azure ML Services, Google ML Engine and IBM Watson ML
Studio.

All except Azure ML Service provide built-in machine learning and artificial
intelligence algorithms, prediction tools like linear regression, and
operation for processing data structures such as tabular data representations.
They also support the most commonly used machine learning and artificial
intelligence libraries and frameworks for executing tasks that can be wrapped
as pipelines.

Machine Learning and Artificial Intelligence Studios give an interactive,
visual workspace to build, test, and iterate on analytics models and develop
experiments. An experiment consists of data sets that provide data to
analytical modules connected to construct an analysis model. An experiment can
be created from scratch or derived from an existing sample experiment as a
template. Data sets and analysis modules can be drag-and-dropped onto an
interactive canvas, connecting them together to form an experiment, which can
be executed on a machine learning runtime (cloud) environment.

Machine learning runtime environments provide the tools needed for executing
machine learning workflows, including data stores, interpreters and runtime
services like Spark, Tensorflow and Caffe for executing analytics operations
and models. The most prominent studios are, for example, Amazon Machine
Learning, Microsoft Artificial Intelligence and Machine Learning Studio, Cloud
Auto ML, Data Bricks ML Flow and IBM Watson ML Builder.

Each provide different families of built-in models such as classification,
regression, clustering, anomaly detection, recommendation and ranking.

####  2.1.3 Serverless and Edge based Approaches

Serverless computing is a computing execution model where the cloud provider
allocates machine resources on demand. Serverless computing does not hold
resources in volatile memory; computing is done in short bursts with the
results persisted to storage. Developers of serverless applications do not
address capacity planning, configuration, management, maintenance, fault
tolerance, or scaling of containers, VMs, or physical servers. Serverless
vendors offer compute runtimes, also known as Function as a Service (FaaS)
platforms, which execute application logic but do not store data. Kubeless and
Fission are two Open Source FaaS platforms which run with Kubernetes.

Amazon AWS Lambda proposes an abstract serverless computing model supported by
additional AWS serverless tools such as AWS Serverless Application Model (AWS
SAM) Amazon CloudWatch. Google Cloud Platform created Google Cloud Functions.
IBM offers IBM Cloud Functions in public, IBM Cloud and IBM Cloud Code Engine.
Microsoft Azure offers Azure Functions, offered both in the Azure public cloud
or on-premises via Azure Stack. Cloudflare offers Cloudflare Workers.

Edge computing [14] is a distributed computing paradigm that brings
computation and data storage closer to the sources of data. The increase of
devices at the edge of the network produces a massive amount of data - storing
and using all that data in cloud data centers pushes network bandwidth
requirements to the limit. The aim is to move the computation away from data
centers towards the edge of the network, exploiting smart objects, mobile
phones, or network gateways to perform tasks and provide services on behalf of
the cloud [26]. One definition of edge computing is any computer program that
delivers low latency nearer to the requests [15]. Moving services to the edge
enables content caching, service delivery, persistent data storage resulting
in better response times and transfer rates.

####  2.1.4 Discussion

Machine learning studios address the analytics and data management divide with
integrated backends for efficient execution of analytics activities pipelines
allocating the necessary infrastructure (CPU, FPGA, GPU, TPU) and platform
(Spark, Tensorflow) services. These descriptions provide insight to data
scientists about the size of the data collections, licences, provenance as
well as data structure and content distributions that can be visually
observed. Curated data collections associated with a search engine can be
shared and used in target data science projects. Data labs offer storage space
often provided by a cloud vendor (e.g., users of CoLab use their google drive
storage space for data collections, notebooks and results). Execution
environments associate computing resources for executing notebooks that use
curated data collections. Other open and private spaces can be coupled and
used as persistence support for decoupling the enactment environment from the
data persistence support (e.g., data can be initially stored in github and
then uploaded to the data lab for analytics purposes). In both cases available
space depends on the type of subscription to the data lab. Machine learning
studios address the analytics and data management divide with integrated
backends for dealing with efficient execution of analytics activities
pipelines allocating the necessary infrastructure (CPU, FPGA, GPU, TPU) and
platform (Spark, Tensorflow) services.

These environments provide resources (CPU, storage and main memory) for
executing data science tasks. These tasks are repetitive, process different
amounts of data and require storage and computing support. Data science
projects have life cycle phases that imply in-house small scale execution
environments, and they can evolve into deployment phases where they can touch
the cloud and the edge resources. Therefore, they require underlying elastic
architectures that can provide resources at different scales. Disaggregated
data centers solutions seem promising for them. Our work addresses the
challenges implied when coupling disaggregated solutions with data science
projects.

##  3 JITA-4DS: Just in time Edge Based Data Science Pipelines Execution

The Just in Time Architecture for Data Science Pipelines (JITA-4DS),
illustrated in Figure 1, is a cross-layer management system that is aware of
both the application characteristics and the underlying infrastructures to
break the barriers between applications, middleware/operating system, and
hardware layers. Vertical integration of these layers is needed for building a
customizable Virtual Data Center (VDC) to meet the dynamically changing data
science pipelines’ requirements such as performance, availability, and energy
consumption.

Our approach to design and development of the proposed JITA is illustrated in
Figure 1.

Figure 1: Just in Time Architecture for Data Science Pipelines - JITA-4DS

The overall JITA design approach focusses on the following interrelated
research tasks:

  1. 1.

JITA middleware to support building VDCs that are optimized for each
application type.

  2. 2.

Development of AVDM system that will support efficient and dynamic management
of VDC software and hardware building blocks.

JITA-4DS benefits from virtualization techniques in the following aspects.
Virtual Machine (VM) technologies such as VMware [11] and Xen [9] provide a
flexible management platform useful for both the encapsulation of application
execution environments and the aggregation and accounting of resources
consumed by an application [24]. Consequently, VMs have been widely used to
provide a layer that is well-positioned in the hardware/software stack of
computer systems. Virtual Machine (VM) technologies provide fine-grain
resource monitoring and control capabilities necessary for the JITA-4DS
approach. Indeed,

JITA-4DS fully exploits the virtualization from the virtual machine (VM) level
into the VDC level (e.g., fine-grain resource monitoring and control
capabilities). JITA-4DS can build a VDC that can meet the application SLO,
such as execution performance and energy consumption to execute data science
pipelines. The selected VDC, then, is mapped to a set of heterogeneous
computing nodes such as GPPs, GPUs, TPUs, special-purpose units (SPUs) such as
ASICs and FPGAs, along with memory and storage units.

JITA-4DS encourages a novel resource management methodology that is based on
the time dependent value of service (VoS) metric to guide the assignment of
resources to each VDC that maximizes the overall system-wide VoS metric. To
predict the execution time and energy consumption of each application type, we
use statistical and data mining techniques [23, 21, 20, 22], which represent
the execution time and energy consumption as a function of the VDC resources.
A complete study of these aspects for JITA-4DS have been described in [12].

DS pipelines running on top of JITA-4DS VDC’s apply sets of big data
processing operators to stored data and streams produced by the Internet of
Things (IoT) farms (see the upper part of Figure 1). In the JITA-4DS approach,
the tasks composing a data science pipeline are executed by services that
implement big data operators. The objective is to execute as just in time
edge-based processes (similar to lambda functions), and they interact with the
VDC underlying services only when the process to execute needs more resources.
This means that services are running on the edge, on processing entities with
different computing and storage capacities. They can totally or partially
execute their tasks on the edge and/or on the VDC.

This, in turn, creates the need for novel resource management approaches in
streaming-based data science pipelines (see experimental discussion in Section
4). These approaches should support and satisfy the data management strategy
and stream exchange model between producers and consumers, invoke tasks with
the underlying exchange model constraints on the compute and storage resources
in the suitable form and modality and meet multi-objective competing
performance goals. Next, we describe the architecture of big data operators,
and we show how they interact with the VDC. Later we will introduce our
resource management approach for JITA-4DS.

The data management strategy stream exchange model. thanks to the following
interrelated research thrusts: (1) JITA Design Approach, and (2) Modeling,
Analysis, and Simulation of JITA.

###  3.1 Big Data/Stream producing and processing services

We assume that services that run on the edge produce and process data in batch
or as streams. Data and stream processing services implement operators to
support the analysis (machine learning, statistics, aggregation, AI) and
visualise big data/streams produced in IoT environments. As shown in Figure 1,
data and stream producing services residing on edge rely on underlying
message-based communication layers for transmitting them to processing and
storage services. These services can reside on edge or a VDC. A data/stream
service implements simple or complex analytics big data operations (e.g.,
fetch, sliding window, average, etc.). Figure 2 shows the general architecture
of a streaming service.

Figure 2: Architecture of a Big data/stream processing service

A service consists of three key components, Buﬀer Manager, Fetch and Sink, and
OperatorLogic. The service logic is based on a scheduler that ensures the
recurrence rate in which the analytics operation implemented by the service is
executed. Stream/data processing is based on unlimited consumption of data
ensured by the component Fetch that works if streams are notiﬁed by a
producer. This speciﬁcation is contained in the logic of the components
OperatorLogic and Fetch. As shown in the ﬁgure, service communicates
asynchronously with other micro-services using a message oriented middleware.
As data is produced, the service fetches and copies the data to an internal
buﬀer. Then, depending on its logic, it applies a processing algorithm and
sends the data to the services connected to it. The general architecture of a
service is specialized in concrete services implementing the most popular
aggregation operations. These services can process data and streams on edge or
a VDC.

These services are able to:

  * -

process data and streams on the edge or on a VDC on-line using tree window
based strategies [19, 17] (tumbling, sliding and landmark) well known in the
stream processing systems domain;

  * -

combine stream histories with continuous ﬂows of streams of the same type (the
average number of connections to Internet by Bob of the last month until the
next hour).

Since RAM assigned to a service might be limited, and in consequence its
buﬀer, every service implements a data management strategy by collaborating
with the communication middleware and with the VDC storage services to exploit
buﬀer space, avoiding losing data, and processing and generating results on
time. Big stream/data operators combine stream processing and storage
techniques tuned depending on the number of things producing streams, the pace
at which they produce them, and the physical computing resources available to
process them online (on edge and VDC) and deliver them to the consumers (other
services). Stores are distributively installed on edge and on the VDC.

Services adopt the tuple oriented data model as stream exchange model with the
IoT environment producing streams and the services. A stream is represented as
a series of attribute value couples where values are of atomic types (integer,
string, char, ﬂoat. From a service point of view, a stream is a series of
attribute value couples where values are of atomic types (integer, string,
char, ﬂoat). We tuples are time-stamped, where assume that one of the
attributes of the tuple corresponds to its time-stamp. the time-stamp
represents the time of arrival of the stream to the communication
infrastructure. Data processing tasks navigate through the structure of the
tuple for accessing attribute values.

###  3.2 Interval oriented storage support for consuming streams

Big stream/data operators combine stream processing and storage techniques
tuned depending on the number of things producing streams, the pace at which
they produce them, and the physical computing resources available for
processing them on-line (on edge and VDC) and delivering them to consumers
(other services). A service that aggregates historical data and streams
includes a component named HistoricFetch. The component HistoricFetch is
responsible for performing a one-shot query for retrieving stored data
according to an input query. Stores are distributively installed on edge and
on the VDC. As described above, we have implemented a general/abstract micro-
service that contains a Fetch and Sink micro-services. The historical fetch
component has been specialized to interact with two stores that can be
distributedly installed on the edge and on the VDC.

  * -

InfluxDB: a time series system accepting temporal queries, useful for
computing time tagged tuples.

  * -

Cassandra: a key-value store that provides non-temporal read/write operations
that might be interesting for storing huge quantities of data.

###  3.3 Edge based Data Science (DS) Pipelines

are expressed by a series of data processing operations applied to
streams/data stemming from things, stores or services. A DS pipeline is
implemented by mashing up services implementing operators based on a
composition operation that connects them by expressing a data flow (IN/OUT
data). Aggregation (min, max, mean) and analytics (k-means, linear regression,
CNN) services can be composed with temporal windowing services (landmark,
sliding) that receive input data from storage support or a continuous data
producer for instance, a thing. The connectors are Fetch, and Sink services
that determine the way services exchange data from/to things, storage systems,
or other services (on-demand or continuous). Services can be hybrid (edge and
VDC) services depending on the number of underlying services (computing,
memory, storage) required. To illustrate the use of a JITA-4DS, we introduce
next a use case that makes full use of edge and VDC services configured ad-hoc
for the analysis requirements.

###  3.4 Use Case: Analysing the connectivity of a connected society

The experiment scenario aims at analyzing the connectivity of the connected
society. The data is used then to answer queries such as:

  * -

Connection speed of one internet provider: Am I receiving the network speed I
am paying for all the time?

  * -

Internet availability with different Internet providers: Which are the periods
of the day in which I can upload/download files at the highest speed using
different network providers?

Figure 3: Services architecture for analyzing society connectivity with the
neubot dataset.

The data set used was produced in the context of the Neubot project. Neubot is
a project on measuring the Internet from the edges by the Nexa Center for
Internet and Society at Politecnico di Torino (https://www.neubot.org/). It
consists of network tests (e.g., download/upload speed over HTTP) realized by
different users in different locations using an application that measures the
network service quality delivered by different Internet connection types. The
Neubot data collection was previously used in the context of the FP7 project
S2EUNET. The type of queries implemented as data science pipelines were the
following:

    EVERY 60 seconds compute the max value of download_speed
    of the last 3 minutes
    FROM Ψcassandra database neubot series speedtests and streaming
    RabbitMQ queue neubotspeed

    EVERY Ψ5 minutes compute the mean of the download_speed
    of the last 120 days
    FROM Ψcassandra database neubot series speedtests and streaming
    rabbitmq queue neubotspeed

    EVERY 30 seconds compute the mean value of upload_speed
     starting 10 days ago
    FROM Ψcassandra database neubot series speedtests and streaming
    rabbitmq queue neubotspeed

We built an IoT farm for deploying our experiment and implemented a
distributed version of the IoT environment on a clustered version of RabbitMQ.
This setting enabled us to address a scaleup setting regarding several data
producers (things) deployed on edge. We installed aggregation operators as
services distributed on the things and an edged execution environment deployed
on servers deployed in different devices. The challenge is to consume streams,
create a history of connectivity information, and then combine these
voluminous histories with new streams to answer the queries. Depending on the
observation window size, the services access the observations stored as post-
mortem data sets from stores at the VDC level and connect to online producers
currently observing their connections (on edge). For example, the second query
observes a window of 10 days size. Our services could deal with histories
produced in windows of size 10 days or even 120 days. Such massive histories
could be combined with recent streams and produce reasonable response times
(order of seconds).

##  4 Runtime Emulation Environment

To conduct experiments on the DS workloads, we configure a runtime environment
that models the flexible heterogeneous resource pool and schedulers. This
environment allows us sweep across the design search space of varying resource
combinations and scheduling policies. For this purpose, we leverage the
Compiler-Integrated Runtime environment [25] that is an open-source Linux
based user-space runtime framework.

The integrated compiler in this environment converts the applications into a
Directed Acyclic Graph (DAG) representation, where

  * -

A node is called a task representing a function used in the application domain
(e.g., k-means).

  * -

An edge of the DAG represents a predecessor-successor dependency between the
task nodes of an application.

The compiler then generates a “flexible” binary structure for the runtime to
invoke each DAG node on any of the available compute resources.

The runtime executes as a daemon process and consists of three key components:

  * -

Application manager: parses the DAG and prepares handles for each kernel in
the “flexible-binary” structure.

  * -

Workload manager: schedules the tasks of applications on available PEs based
on the user-defined scheduling policy and manages the data transfers to and
from the PEs.

  * -

Resource manager: monitors the state of the PEs in the target hardware
configuration and maintains coordination with the workload manager.

Using this runtime environment, we modelled a hierarchical resource pool,
described next.

###  4.1 Hierarchical Resource Pool: a JITA4-DS instance

As shown in Figure 4, the hierarchical resource pool consists of two layers-
the frontend, and the backend, with heterogeneous computing resources.

The frontend represents the low-power computing resources present at the edge,
such as Nvidia Volta GPU and ARM CPU cores.

The backend represents the high-performance compute resources such as Intel
Xeon CPU cores, Nvidia Tesla V100 GPUs and Xilinx Alveo FPGA platforms.

The runtime is launched as a daemon process on one of the Xeon CPU cores on
the backend platform and provided with the list of available frontend and
backend resources. We assume that the user is submitting a job to the server
(backend). The data flow of this job starts on the edge (frontend), where
real-time data from the sensors are captured. This raw data are processed
through the user-defined DS workflow.

Job submission involves providing the application task flow DAG generated from
the DS workflow and the flexible binaries to enable execution on the resources
of the two-layer architecture. This DAG and flexible binary file are obtained
by compiling the source code of DS workflow through the runtime environment-
integrated compiler toolchain. At this stage, the runtime manager determines
the mapping of individual tasks on the DAG to processing elements based on the
state of computing resources at both the frontend (edge) and backend (server).

This application submission process allows users to submit any desired number
of application instances to the runtime, either all instances submitted at
once or submitted with a periodic delay between each instance.

Upon receiving a job consisting of several tasks at the runtime, and after raw
data from the edge sensors are captured, the runtime can schedule each task on
either one of the heterogeneous resources available at the backend platform.
This task involves requesting input data from and sending output data to the
edge device or decide to schedule the task on one of the edge resources, which
alleviates the overhead of communicating with the backend and exchanging
necessary input-output data.

To enable the runtime schedulers in making an informed decision, each task in
the DAG file is assigned an expected execution time on the supported compute
platforms based on historical data. The expected execution time of the tasks
on the frontend resources denotes only the execution time, whereas, for
backend resources, this time consists of both execution time and data
communication overhead.

Figure 4: Diagram of Hierarchical architecture model using our runtime
environment.

###  4.2 Executing Data Science Workloads

We use the Data Science (DS) workloads as the application to run on the
hierarchical architecture for our experiments. The DS workload (see Figure 5)
consists of 16 task nodes, including: frequently used data science functions
such as SQL Transform, data summarization, column selection in dataset,
filter-based feature selection, k-means clustering, time series anomaly
detection, sweep clustering, train clustering model etc.

Figure 5: Data Science Workload.

We assume that there is historical execution time data for each task node on
each of the compute resources modeled. To further consider the communication
cost, we assume the data rate of the communication channel to be 12 Mbps [16]
and calculate the data transfer overhead by using the volume of input and
output data of each task that needs to be transferred.

With the experimental setup, we can conduct elaborate experiments to get
answers to some crucial questions. For example,

  * RQ1

Is it beneficial to hand off all the tasks of given workloads to the backend
resources.

  * RQ2

Should some of the tasks be assigned to the frontend resources instead?

  * RQ3

Can the communication cost of executing some tasks on the backend resources
grow large enough for the runtime to decide to run it on a frontend resource?

In order to find these answers, we conduct two experiments on the hierarchical
architecture model in our runtime environment.

####  4.2.1 Experiment 1: Sweeping across resource pool configurations

In the first experiment, we fix the scheduling policy to Earliest Finish Time
(EFT) and sweep across the combinatorial search space of resource pool
configurations. Each resource pool configuration refers to a unique
combination of front and backend heterogeneous resources with a specific count
of each resource type.

For our experiment, we vary the number of ARM cores and Xeon CPUs between 1 to
3 each, whereas we keep the number of Volta GPUs, Tesla V100 GPUs and Xilinx
Alveo FPGAs fixed at 1. On each configuration, we run 100 instances of the DS
workload submitted to the runtime at once. Upon receiving the execution time
of the DS workload on all the tested configurations, we identify the
configuration that gives the minimum execution time. Once this configuration
is identified, we use it to conduct a second experiment.

Figure 6: Execution time of 100 DS workflow instances for different resource
pool configurations. First 9 columns of barchart also consist of 1 Nvidia
Volta GPU (frontend), 1 Nvidia Tesla V100 GPU (backend) and 1 Xilinx Alveo
FPGA (backend). The 10th configuration uses 3 ARM CPUs and 1 Volta GPU. The
11th configuration uses 3 Xeon CPUs, 1 Nvidia Tesla V100 GPU, and 1 Xilinx
Alveo FPGA.

The result of experiment 1 is presented in Figure 6. This bar chart plots
varying resource pool configurations along X-axis and their corresponding
execution times for processing 100 instances of DS-workflow along Y-axis. The
first 9 configurations (from left) include 1 Nvidia Volta GPU (at frontend), 1
Nvidia Tesla GPU and 1 Xilinx Alveo FPGA (at backend), along with the varying
number of ARM and Xeon cores denoted by the label of the barplot. In the
remaining two configurations, the _Edge only_ configuration consists of 3 ARM
CPU cores and 1 Nvidia Volta GPU only, and executes all of the DS workload
instances at the frontend. The _Server only_ configuration consists of 3 Xeon
CPU cores, 1 Nvidia Tesla GPU and 1 Xilinx Alveo FPGA, and executes the entire
application at the backend after collecting input data from frontend. In this
plot, the two largest execution times are resulted by the _Edge only_ or
_Server only_ configurations, whereas the remaining configurations consisting
of a mixture of edge and server resources demonstrate lower execution times.
The _Edge only_ configuration consists of low-power compute resources, which
cause the overall workload execution on the edge slower. On the other hand,
the _Server only_ configuration relies on the frontend to send larger amount
of input data at the very beginning of workload execution, which increases the
execution time significantly. Another observation we make here is that, with
increasing number of available parallel resources (CPU, GPU or FPGA), the
execution time reduces. Therefore, the lowest execution time is delivered by
the configuration which holds the maximum number of resources of all types
considered in this experiment (3 ARM, 1 Volta, 3 Xeon, 1 Tesla, 1 Alveo).
Hence we choose this configuration to be used in conducting the second
experiment.

Depending on the above observations, we can answer the research questions as
follows-

  * RQ1

In terms of execution time, it is not beneficial to hand off all the tasks of
a given workload to the backend resources. The initial data transfer overhead
of large raw data from frontend to backend increases the execution time
significantly, compared to if the data was to be used to run all the tasks at
the frontend. Additionally, it is not beneficial to execute the entire
workload on the frontend, as the frontend is constrained in terms of
computation power. Hence, a mixture of both frontend and backend resources can
be more beneficial.

  * RQ2

Yes, to reduce the execution time, the data transfer overhead should be
reduced, therefore some of the tasks should be executed at the frontend.

  * RQ3

Yes, in some scenarios the communication cost of transferring the data from
frontend to backend might grow large enough for the runtime to decide to run
those tasks on the frontend. This can be seen in the configurations with a
mixture of frontend and backend resources. In these configurations the runtime
decides to offload some tasks onto the frontend resources, which eventually
reduces the execution time of the workload significantly (by upto 57%)
compared to running the entire workload on backend.

####  4.2.2 Experiment 2: Sweeping across scheduling policies

In this experiment, we sweep across different scheduling policies by running
100 instances of the DS workload submitted at once, using three different
scheduling policies- ‘EFT’, ‘Earliest Task First (ETF)’, and ’Round Robin
(RR)’. The EFT and ETF are sophisticated scheduling policies that take into
account the hierarchy of the resource pool, expected execution time and data
communication overhead of the underlying tasks in order to make scheduling
decisions, while the RR is a simple scheduler that assigns tasks to resources
in a round robin manner. We aim to identify the scheduling policy that gives
the lowest execution time and highest mean resource utilization on the optimal
configuration from experiment 1.

This configuration includes 3 ARM CPU cores and 1 Nvidia Volta GPU at the
frontend, along with 3 Xeon CPU cores, 1 Nvidia Tesla GPU and 1 Xilinx Alveo
FPGA at the backend. We identify the resource utilization as the fraction of
application execution time during which a particular resource is busy
executing tasks. We take the mean of resource utilization of all resources to
obtain the mean resource utilization.

Figure 7: Resource pool configuration: 3 ARM CPU cores, 1 Nvidia Volta GPU
(Frontend), 3 Xeon CPU Cores, 1 Nvidia Tesla GPU and 1 Xilinx Alveo FPGA
(Backend). Using EFT, ETF and RR schedulers, (a) presents execution time of
100 DS-workflow jobs, and (b) presents average resource utilization.

Figure 7(a) and (b) presents bar plots with the schedulers plotted along
X-axis, and the execution time and mean resource utilization plotted along
Y-axis respectively. Both these figures show that ETF and EFT schedulers
perform very closely in terms of execution time and mean resource utilization.
From Figure 7(a), we notice that both ETF and EFT schedulers reduce the
execution time by around 57% compared to the RR scheduler. Furthermore,
looking at Figure 7(b), we notice that both ETF and EFT schedulers increase
the mean resource utilization by upto around 21% compared to the RR scheduler.
These results demonstrate that the runtime using hierarchy aware schedulers
can significantly improve the execution time and mean resource utilization
compared to the simpler scheduling heuristics.

####  4.2.3 Discussion

In the JITA-4DS environment, the resource management problem can be more
complex and requires the design of new heuristics. The computing resources
allocated to the VDC for a given class of applications are a heterogeneous
mixture of different processing devices (different CPUs, different GPUs,
different accelerators, etc.) with various execution performance and energy
consumption characteristics. They depend on each of the specific applications
being executed by that VDC. Our future work will consider this variables in
the decision making process. For example, several aspects remain open, like
the ad-hoc design of the JITA-4DS resource management system for a VDC built
from a fixed set of components. The design of a JITA-4DS instance is
determined by the execution time and energy consumption cost, and resources
requirements of a data science pipeline. Therefore, it is necessary to
dynamically identify the configuration choices for a given pipeline and define
VDC resources’ effective resource allocation strategies. In general, for
determining the dynamic resources requirements of data science pipelines at
runtime, it is necessary to consider two challenges. First, calculate a VDC-
wide Value of Service (VoS) for a given interval of time, weigh individual
values of various instances of pipelines. We have started providing a study on
VoS for JITA-4DS [12], we will integrate these observations in further
experiments on the current emulation. Second, propose objective functions that
can guide heuristics to operate in the large search space of resource
configurations. The objective is to derive possible adequate allocations of
the shared and fixed VDC resources for several instances of data science
pipelines. We have observed that decisions must be made regarding the resource
management system for JITA-4DS to divide the shared, fixed resource pool
across different VDCs to maximize the overall system-wide VoS. All of the
above single VDC challenges still apply and interact across VDCs. Additional
problems, such as determining when resources should be reallocated across VDCs
and do so in an online fashion, must be addressed. This includes the
methodologies for reassigning resources that do not interfere with currently
executing applications on different VDCs affected by the changes and measuring
and accounting for the overhead of establishing new VDC configurations.

##  5 Conclusion and Future Work

This paper introduced JITA-4DS, a virtualized architecture that provides a
disaggregated data center solution ad-hoc for executing DS pipelines requiring
elastic access to resources. DS pipelines process big streams and data
coordinating operators implemented by services deployed on edge. Given that
operators can implement greedy tasks with computing and storage requirements
beyond those residing on edge, they interact with VDC services. We have set
the first simulation setting to study resources delivery in JITA-4DS.

We are currently addressing challenges of VDCs management in simpler
environments, on cloud resource management heuristics, big data analysis, and
data mining for performance prediction. We give the first experimental results
of these aspects. To simulate, evaluate, analyze, and compare different
heuristics, we will further build simulators for simpler environments and
combine open-source simulators for different levels of the JITA-4DS hierarchy.
We are currently defining a ”benchmark” with different types of data science
workloads that share data collections and functions and study complex resource
allocation patterns.

## References

  * [1] ASTERIX open-source big data management system.  https://asterixdb.apache.org.  Accessed: 2021-07-24.
  * [2] DriveScale Composable Platform.
  * [3] Google Colab.  https://colab.research.google.com.  Accessed: 2021-07-24.
  * [4] Google Kaggle.  http://www.kaggle.com.  Accessed: 2021-07-24.
  * [5] HPE Synergy.
  * [6] Liqid Composable Infrastructure.
  * [7] MARKDOWN.  https://guides.github.com/features/mastering-markdown/.  Accessed: 2021-07-24.
  * [8] Microsoft Azure Notebooks.  https://notebooks.azure.com.  Accessed: 2021-07-24.
  * [9] The Hypervisor (x86 & ARM).
  * [10] Unified Computing.
  * [11] VMware vSphere Hypervisor.
  * [12] Ali Akoglu and Genoveva Vargas-Solar.  Putting data science pipelines on the edge.  arXiv preprint arXiv:2103.07978, 2021.
  * [13] Hao Chen, Yijia Zhang, Michael C. Caramanis, and Ayse K. Coskun.  Energyqare: Qos-aware data center participation in smart grid regulation service reserve provision.  ACM Trans. Model. Perform. Eval. Comput. Syst., 4(1), January 2019\.
  * [14] Andy Davis, Jay Parikh, and William E Weihl.  Edgecomputing: extending enterprise applications to the edge of the internet.  In Proceedings of the 13th international World Wide Web conference on Alternate track papers & posters, pages 180–187, 2004.
  * [15] Pedro Garcia Lopez, Alberto Montresor, Dick Epema, Anwitaman Datta, Teruo Higashino, Adriana Iamnitchi, Marinho Barcellos, Pascal Felber, and Etienne Riviere.  Edge-centric computing: Vision and challenges, 2015.
  * [16] Adnan Ghayas.  Average 4g lte speed: How fast is 4g lte?, 2021.
  * [17] Lukasz Golab and M Tamer Özsu.  Data stream management.  Synthesis Lectures on Data Management, 2(1):1–73, 2010.
  * [18] Ram Srivatsa Kannan, Lavanya Subramanian, Ashwin Raju, Jeongseob Ahn, Jason Mars, and Lingjia Tang.  Grandslam: Guaranteeing slas for jobs in microservices execution frameworks.  EuroSys ’19, New York, NY, USA, 2019. Association for Computing Machinery.
  * [19] Jürgen Krämer and Bernhard Seeger.  Semantics and implementation of continuous sliding window queries over data streams.  ACM Transactions on Database Systems (TODS), 34(1):4, 2009.
  * [20] Nirmal Kumbhare, Ali Akoglu, Aniruddha Marathe, Salim Hariri, and Ghaleb Abdulla.  Dynamic power management for value-oriented schedulers in power-constrained hpc system.  Parallel Computing, 99:102686, 2020.
  * [21] Nirmal Kumbhare, Aniruddha Marathe, Ali Akoglu, Salim Hariri, and Ghaleb Abdulla.  Adaptive power reallocation for value-oriented schedulers in power-constrained hpc.  In 2019 20th International Conference on Parallel and Distributed Computing, Applications and Technologies (PDCAT), pages 133–139. IEEE, 2019.
  * [22] Nirmal Kumbhare, Aniruddha Marathe, Ali Akoglu, Howard Jay Siegel, Ghaleb Abdulla, and Salim Hariri.  A value-oriented job scheduling approach for power-constrained and oversubscribed hpc systems.  IEEE Transactions on Parallel and Distributed Systems, 31(6):1419–1433, 2020.
  * [23] Nirmal Kumbhare, Cihan Tunc, Dylan Machovec, Ali Akoglu, Salim Hariri, and Howard Jay Siegel.  Value based scheduling for oversubscribed power-constrained homogeneous hpc systems.  In 2017 International Conference on Cloud and Autonomic Computing (ICCAC), pages 120–130. IEEE, 2017.
  * [24] Dylan Machovec, Bhavesh Khemka, Nirmal Kumbhare, Sudeep Pasricha, Anthony A Maciejewski, Howard Jay Siegel, Ali Akoglu, Gregory A Koenig, Salim Hariri, Cihan Tunc, Michael Wright, Marcia Hilton, Rajendra Rambharos, Christopher Blandin, Farah Fargo, Ahmed Louri, and Neena Imam.  Utility-based resource management in an oversubscribed energy-constrained heterogeneous environment executing parallel applications.  In Parallel Computing, volume 83, pages 48–72, Apr. 2019.
  * [25] Joshua Mack et al.  User-Space Emulation Framework for Domain-Specific SoC Design.  In 2020 IEEE Int. Parallel and Distrib. Process. Symp. Workshops), pages 44–53, 2020.
  * [26] Massimo Merenda, Carlo Porcaro, and Demetrio Iero.  Edge machine learning for ai-enabled iot devices: A review.  Sensors, 20(9):2533, 2020.
  * [27] A. D. Papaioannou, R. Nejabati, and D. Simeonidou.  The benefits of a disaggregated data centre: A resource allocation approach.  In 2016 IEEE Global Communications Conference (GLOBECOM), pages 1–7, 2016.
  * [28] Xiaolong Xu, Wanchun Dou, Xuyun Zhang, and Jinjun Chen.  EnReal: An energy-aware resource allocation method for scientific workflow executions in cloud environment.  IEEE Transactions on Cloud Computing, 4(2):166–179, Sep. 2015.


# Efficient training of lightweight neural networks using Online Self-Acquired
Knowledge Distillation

Maria Tzelepi and Anastasios Tefas
Department of Informatics
Aristotle University of Thessaloniki
Thessaloniki, Greece
Email<EMAIL_ADDRESS>
###### Abstract

Knowledge Distillation has been established as a highly promising approach for
training compact and faster models by transferring knowledge from heavyweight
and powerful models. However, KD in its conventional version constitutes an
enduring, computationally and memory demanding process. In this paper, Online
Self-Acquired Knowledge Distillation (OSAKD) is proposed, aiming to improve
the performance of any deep neural model in an online manner. We utilize k-nn
non-parametric density estimation technique for estimating the unknown
probability distributions of the data samples in the output feature space.
This allows us for directly estimating the posterior class probabilities of
the data samples, and we use them as soft labels that encode explicit
information about the similarities of the data with the classes, negligibly
affecting the computational cost. The experimental evaluation on four datasets
validates the effectiveness of proposed method.

###### Index Terms:

Knowledge Distillation, Online Knowledge Distillation, Self-Distillation

##  I Introduction

Deep learning models [1] have been extensively utilized over the recent years
so as to resolve a broad range of computer vision tasks [2]. However, their
superior performance usually stems from their depth and complexity. This
constitutes a major impediment on applying these models for real-time
applications as well as on devices with limited computational resources such
as mobile phones and embedded systems. Thus, an apparent need for developing
compact yet effective models has been arisen. Among other solutions proposed
to achieve this goal [3], Knowledge Distillation (KD) [4] has emerged as a
very effective way to address this issue.

KD, in its seminal approach, describes the procedure where the knowledge of a
high-capacity model, known as teacher, which exhibits high performance, is
transferred to a more compact and faster model, known as student. The student
model is trained to regress the so-called soft labels produced by the teacher
by raising the temperature of the softmax activation function on the output
layer of the network. These soft labels convey more information of the way the
model learns to generalize, as compared to the hard labels, trying implicitly
to reveal similarities over the data. Other approaches include the transfer of
knowledge from teachers to students of identical capacity [5, 6, 7]. The
latter procedure is known as self-distillation.

KD methods can be classified into two broad categories: online and offline KD.
The latter is what has already been described as the multi-stage procedure of
first training a more complex and powerful teacher model and then distilling
the knowledge to a simpler student model by training it to mimic the teacher.
On the contrary, online KD describes the procedure where the teacher and the
student models are trained simultaneously, that is without the stage of pre-
training the teacher network. Online KD includes works proposing to train
multiple models mutually from each other [8], as well as works proposing to
build ensembles of multiple identical branches of a target model in order to
build a strong teacher and distill the knowledge from the teacher to the
target model [9].

In this work, we pursue the direction of online KD through a different
perspective: We propose a single-stage self-acquired knowledge distillation
approach, namely Online Self-Acquired Knowledge Distillation (OSAKD) for
improving the performance of any deep neural model in an online manner. The
motivation behind this work is to train effectively small networks with an
additional supervision that conveys extra knowledge beyond the hard labels
from the model itself and also in an online manner. That is, in a typical
classification problem the softmax classifier, which produces a probability
distribution over the classes, inherently suppresses all the activations
excluding the highest one. Conventional KD methods manifest that the class
probability distribution of a strong teacher model provides useful information
on the similarities of the data with all the classes. Thus, instead of
training a simpler student model merely with the hard labels, it is useful to
maintain the similarities of the samples with the classes [4]. In addition, in
[6] it is shown that useful information about the similarities of the samples
with the classes can be obtained even by transferring the knowledge through
the class probability distribution from a teacher of identical capacity to
student. Furthermore, in [10] it is stated that small networks usually have
the same representation capacity as large networks, however they are harder to
train, compared to large networks. Thus, taking the aforementioned
observations into consideration, the question that arises is how can someone
efficiently train small yet effective networks, deriving additional
information about the similarities between the classes from the model itself
and also in an online manner.

Towards this end, we utilize the k-nn non-parametric density estimation
technique [11] for estimating the unknown probability distributions of the
data samples in the feature space of the output layer. In this way, we are
able to directly estimate the posterior class probabilities of the data
samples, and use them as soft labels. The estimated soft labels explicitly
encode information on the similarity of each training sample with the classes,
while it is expected that as the training progresses we learn more and more
reliable and meaningful soft labels, since they are driven by the supervised
loss. This argument is also experimentally validated. Furthermore, it should
be emphasized that as opposed to a general classification task where when
estimating the posterior class probabilities some errors lead to the
accumulation of errors, and the goal is to minimize them, in our case, propose
to estimate the posterior class probabilities utilizing the k-nn density
estimation technique, so as to use them as soft labels, that is as an
auxiliary task to the conventional supervised loss. Thus, our goal is not to
directly minimize the estimation error for them, but rather to continuously
estimating and considering them through the training process, so as to assist
the main classification objective.

The intuition of the proposed online distillation method, considering a
probabilistic view of KD is as follows: Deep neural models transform the
probability distribution of the data, layer by layer, learning increasingly
complex layer representations. Considering a multi-class classification
problem, a conventional supervised loss, forces the data representations in
the output layer of a neural model to become one-hot representations. However,
trying to convert the complex data representations to one-hot representations
usually leads to over-training and also requires deeper and more complicated
models. Thus, in this work considering the neighborhood of each sample as it
is explained in the subsequent Section, we produce the soft labels that encode
the similarity of each sample to each of the classes. That is, the hard label
objective forces all the data representations belonging to a certain class to
concentrate one point. On the contrary, each data representation is forced to
approach its own soft label encoding its similarity with the classes of the
problem. In this way, we learn representations that recover the similarities
of the samples with the classes, improving the generalization ability of the
model.

The main contributions of this paper can be summarized as follows:
-We propose a novel Online Self-Acquired Knowledge Distillation method.
-The OSAKD method acquires the soft labels from the model itself in an online manner, without requiring the utilization of multiple models or copies of the model like the most online KD methods, rendering it more efficient.
-The proposed method is model-agnostic, that is, it is applicable to several neural network architectures.
-The experimental evaluation indicates the OSAKD method can improve the classification performance of simple and deeper models.

The reminder of the manuscript is organized as follows. Section II discusses
relevant works. The proposed OSAKD method is presented in detail in Section
III. Subsequently, in Section IV the experiments conducted to validate the
proposed method are provided, and finally the conclusions are drawn in Section
V.

##  II Prior Work

In this Section we discuss prior studies on self-distillation and online KD.
Regarding the first case, a self-distillation approach where a teacher model
is initially trained, and then after its convergence, an identical student
model is trained with both the goals of the hard labels and matching the
output of the teacher model, however without softening the logits (i.e., the
inputs to the final softmax activation function) by raising the temperature,
is proposed in [6]. Similarly, a target model is trained with a conventional
supervised loss, the self-discovered knowledge is extracted, and in the second
training stage, the model is trained with both the supervised and the
distillation losses in [7].

In this paper, a self-distillation method is also proposed. However, a key
attribute of the proposed method is that the knowledge is distilled within the
same model online. The proposed approach does not use the aforementioned
multiple stages of the training pipeline, which renders it more efficient. It
also uses soft labels explicitly, recovering the similarities of each training
sample with the classes.

Subsequently, several works proposing online distillation have been also
recently proposed. A method namely codistillation has been proposed in [12].
It improves the accuracy by proposing to train $k$ copies of a target model in
parallel, by adding a distillation term to the loss function of the $i$-th
model to match the average prediction of the other models. A quite similar
approach is proposed in [8] where an ensemble of students teach each other
throughout the training process. That is, each student is trained with a
conventional supervised learning loss, and a distillation loss that aligns
each student's class posterior with the class probabilities of other students.
In this way, each model acts as a teacher of the other models. In this
approach, as opposed to the aforementioned codistillation method [12],
different networks can be used for the mutual training.

Furthermore, an online distillation approach where a multi-branch version of
the network is built by adding identical branches each of which constitutes an
independent classification model with shared low level layers, and a strong
teacher model is created utilizing a gated logit ensemble of the multiple
branches, in [9]. Each branch is trained with the conventional classification
loss and the distillation loss which regresses the teacher's prediction
distributions. Finally, a recent work [13], combines the previous works [8]
and [9] proposing an online mutual knowledge distillation method for enhancing
both the performance of the fusion module and the sub-networks. That is, when
different sub-networks are used, the sub-networks are trained similar to [8],
while when identical sub-networks are used, the low level layers are shared,
and a multi-branch architecture similar to [9] is used.

Contrariwise, we propose a self-supervised online KD methodology which allows
synchronous model updating, without the need of building multiple identical
models, or using multiple possibly different models to mutually teach each
other, which comes with additional computational cost.

##  III Proposed Method

We consider a $C$-class classification problem, and the labeled data
${\\{\mathbf{x}_{i},\mathbf{y}_{i}\\}}_{i=1}^{N}$, where
$\mathbf{x}_{i}\in\Re^{D}$ an input vector and $D$ its dimensionality, while
$\mathbf{y}_{i}\in\mathcal{Z^{C}}$ corresponds to its $C$-dimensional one-hot
class label vector. For an input space $\mathcal{X}\subseteq\Re^{D}$ and an
output space $\mathcal{F}\subseteq\Re^{C}$, we consider as
$\phi(\cdot\,;\mathcal{W}):\mathcal{X}\rightarrow\mathcal{F}$ a deep neural
network with $N_{L}\in\mathbb{N}$ layers, and set of parameters
$\mathcal{W}=\\{\mathbf{W}_{1},\dots,\mathbf{W}_{N_{L}}\\}$, where
$\mathbf{W}_{L}$ are the weights of a specific layer $L$, which transforms its
input vector to a $C$-dimensional probability vector. That is,
$\phi(\mathbf{x}_{i}\,;\mathcal{W})\in\mathcal{F}$ corresponds to the output
vector of $\mathbf{x}_{i}\in\mathcal{X}$ given by the network $\phi$ with
parameters $\mathcal{W}$.

Thus, considering the typical classification problem, we seek for the
parameters $\mathcal{W^{*}}$ that minimize the cross entropy loss,
$\mathcal{L}_{ce}$, between the output vector
$\phi(\mathbf{x}_{i}\,;\mathcal{W})$ and the one-hot class label vector
$\mathbf{y}_{i}$:

$\mathcal{W^{*}}=\operatorname*{arg\,min}_{\mathcal{W}}\sum_{i=1}^{N}\mathcal{L}_{ce}\big{(}\mathbf{y}_{i},\phi(\mathbf{x}_{i}\,;\mathcal{W})\big{)},$
(1)

The cross entropy loss for a sample $i$ is formulated as:

$\mathcal{L}_{ce}(\mathbf{y}_{i},\mathbf{z}_{i})=\sum_{m=1}^{C}y_{i}^{m}\log(z_{i}^{m}),$
(2)

where $y_{i}^{m}$ is the $m$-th element of $\mathbf{y}_{i}$ one-hot label
vector, and $z_{i}^{m}$ is defined as the output of the softmax operation on
the $C$-dimensional network's output:

$z_{i}^{m}=\frac{\exp({\phi(\mathbf{x}_{i}\,;\mathcal{W}})^{m})}{\sum_{j=1}^{C}\exp({\phi(\mathbf{x}_{i}\,;\mathcal{W})^{j})}}.$
(3)

In this work, we propose to distill additional knowledge online from the model
itself throughout the network's training. To this end, we propose to utilize
k-nn non-parametric density estimation [11] for estimating the unknown
probability distributions of the data samples in the output space
$\mathcal{F}\subseteq\Re^{C}$. Generally, the idea of non-parametric density
estimation relies on the fact that the probability $P$ that a vector
$\mathbf{x}$ will fall in a region $R$ is given by:
$P=\int_{R}p(\mathbf{x}^{\prime})d\mathbf{x}^{\prime}$. Hence, $P$ is a
smoothed version of the density function $p(\mathbf{x})$ and this smoothed
value of $p$ can be estimated by estimating the probability $P$. Considering,
that $N$ samples ${\\{\mathbf{x}_{i}}\\}_{i=1}^{N}$ are drawn independently
and identically distributed according to the probability law $p(\mathbf{x})$,
then the probability that $k$ of these $N$ fall in $R$ is given by the
binomial law: $P_{k}={N\choose k}P^{k}(1-P)^{N-k}$, and the expected value for
$k$ is given by: $\mathbb{E}[k]=NP$.

Moreover, this binomial distribution for $k$ peaks very sharply about the
mean, so that it is expected that the ratio $\frac{k}{N}$ will be a very good
estimate for the probability $P$, and therefore for the smoothed density
function. The larger the $N$, the more accurate the estimate is. Assuming also
that $p(\mathbf{x})$ is continuous and that the region $R$ is so small that
$p$ does not considerably vary within it, we can arrive at the following
equation: $\int_{R}p(\mathbf{x}^{\prime})d\mathbf{x}^{\prime}\simeq
p(\mathbf{x})V$, where $\mathbf{x}$ is a point within $R$ and $V$ is the
volume enclosed by $R$. By combining the previous equations we arrive at the
following estimate for $p(\mathbf{x})$: $p(\mathbf{x})=\frac{\frac{k}{N}}{V}$.

In the k-nn density estimation we determine the number of nearest neighbors,
$k$, and we adjust the volume, as compared to the parzen windows method where
for a fixed-length volume $V$ we observe how many points $k$ fall into the
region.

The key advantage of the k-nn density estimation method is that it allow us to
directly estimate the posterior probabilities $P(c_{m}|\mathbf{x})$ of the
class being $c_{m},m=\\{1,\cdots,C\\}$, from a set of $N$ labelled data by
using the samples to estimate the densities involved, [11]. That is, assuming
that we place a cell of volume $V$ around $\mathbf{x}$ and capture $k$
samples, $k_{m}$ of which belong to the class $c_{m}$. Then, the estimate for
the probability $p(\mathbf{x}|c_{m})$ is:
$p(\mathbf{x}|c_{m})=\frac{k_{m}}{N_{m}V}$, where $N_{m}$ is the number of
samples which belong to the class $c_{m}$, and similarly the unconditional
density is estimated as: $p(\mathbf{x})=\frac{k}{NV}$, and the priors can be
approximated by: $P(c_{m})=\frac{N_{m}}{N}$.

Thus, the posterior probabilities using the Bayes rule are given by:

$P(c_{m}|\mathbf{x})=\frac{p(\mathbf{x}|c_{m})P(c_{m})}{p(\mathbf{x})}=\frac{\frac{k_{m}}{N_{m}V}\frac{N_{m}}{N}}{\frac{k}{NV}}=\frac{k_{m}}{k}.$
(4)

Therefore, for a specific number of nearest neighbors, $k$, the posterior
probabilities can be estimated and used as soft labels for the network's
training. That is, the soft label for a sample $i$ is formulated as:
$\mathbf{s}_{i}=\big{[}\frac{k_{1}}{k},\frac{k_{2}}{k},\dots,\frac{k_{C}}{k}\big{]}.$

Thus, in the proposed distillation training procedure, we seek for the
parameters $\mathcal{W^{*}}$ that minimize the overall loss of cross entropy,
$\mathcal{L}_{ce}$ and self-distillation, $\mathcal{L}_{sd}$:

$\mathcal{W^{*}}=\operatorname*{arg\,min}_{\mathcal{W}}\sum_{i=1}^{N}[\lambda\mathcal{L}_{ce}\big{(}\mathbf{y}_{i},\phi(\mathbf{x}_{i}\,;\mathcal{W})\big{)}+(1-\lambda)\mathcal{L}_{sd}\big{(}\mathbf{s}_{i},\phi(\mathbf{x}_{i}\,;\mathcal{W})\big{)}],$
(5)

where $\lambda$ balances the importance between predicting the hard labels in
$\mathbf{y}_{i}$ and regressing the soft labels in $\mathbf{s}_{i}$. Either
Kullback-Leibler (KL) divergence or the Mean Squared Error (MSE) between the
soft labels, and the actual predictive vectors can be utilized to train the
network. In this work, we utilize MSE.

Consequently, the OSAKD training procedure is as follows. The input images are
fed to the network, and for each sample the class predictions are produced.
Subsequently, the soft labels are computed based on the neighborhood of each
sample, according to the procedure described above. Then, the network is
trained using the cross entropy loss with the hard labels, and concurrently
using the distillation loss so as to regress the produced soft labels,
enforcing it to regard the similarity of each sample with the classes.

##  IV Experiments

Four datasets were used to validate the performance of the proposed method.
That is, Cifar-10 [14], Street View House Numbers (SVHN) [15], Fashion MNIST
[16], and Tiny ImageNet 111https://tiny-imagenet.herokuapp.com/. The
descriptions of the models' architectures used for each dataset follow bellow.
Test accuracy was used to evaluate the proposed method. Each experiment is
repeated five times and the mean value and the standard deviation are
reported, considering the maximum value of test accuracy for each experiment.
The curves of mean test accuracy are also provided. Finally, the sum of
floating point operations (FLOPs) is used to evaluate the complexity of the
proposed method.

###  IV-A CNN Models

In the case of the Cifar-10, and SVHN datasets a simple CNN model is utilized,
consisting of five layers; two convolutional layers with 6 filters of size
$5\times 5$ and 16 filters of size $5\times 5$ respectively, followed by a
Rectified Linear Unit (ReLU) [17] activation, and three fully connected layers
($128\times 64\times 10$). The convolutional layers are followed by a $2\times
2$ max-pooling layer with a stride of 2. In the first two fully connected
layers the activation function is the ReLU, while the last output layer is a
10-way softmax layer which produces a distribution over the 10 class labels of
the utilized datasets. In the case of the Fashion MNIST dataset a simple
architecture is also utilized, consisting of two convolutional with 20 filters
of size $5\times 5$ and 50 filters of size $5\times 5$ followed by a ReLU
activation, and two fully connected layers ($64\times 10$). The convolutional
layers are followed by $2\times 2$ max-pooling layer with a stride of 2. In
the fist fully connected layer a ReLU activation is applied, while the last
output layer is a 10-way softmax layer. Finally, in the challenging case of
Tiny ImageNet dataset the common ResNet-50 [18] architecture is utilized,
without utilizing any pre-trained model in order to avoid image resizing. It
is noteworthy that the goal of this work is not to provide state-of-the-art
models, but rather to to evaluate the effect of the proposed online self-
distillation method on training lightweight model that can be effectively
deployed on embedded and mobile devices. To this end, we use the
aforementioned simple CNN architectures in three out of four cases, while we
also utilize a common powerful network in the case of Tiny ImageNet,
validating our claim that the proposed method is model agnostic. Finally, for
comparison purposes against previous online KD approaches, we also utilize
ResNet-32 [18] to perform experiments on Cifar-10 dataset.

###  IV-B Implementation Details

The mini-batch gradient descent is used for the networks' training,
considering mini-batch size of 64 samples. The learning rate is set to
$10^{-3}$, and the momentum is 0.9. The models are trained on an NVIDIA
GeForce GTX 1080 with 8GB of GPU memory for 100 epochs. The parameter
$\lambda$ in eq. (5) for controlling the balance between the contributing
losses, is set to 0.9 for all the utilized datasets except for the Tiny
ImageNet where it is set to 0.99.

###  IV-C Experimental Results

TABLE I: Test Accuracy for all the utilized datasets Method | Cifar-10 | SVHN | Fashion MNIST | Tiny ImageNet
---|---|---|---|---
w/o OSAKD | 64.734% $\pm$ 0.654% | 88.706% $\pm$ 0.306% | 91.214% $\pm$ 0.141% | 31.050% $\pm$ 1.550%
OSAKD-8NN | 66.734% $\pm$ 0.086% | 89.567% $\pm$ 0.436% | 91.49% $\pm$ 0.006% | 31.282% $\pm$ 0.506%
OSAKD-16NN | 65.97% $\pm$ 0.733% | 89.207% $\pm$ 0.39% | 91.418% $\pm$ 0.124% | 31.916% $\pm$ 0.982%

The evaluation results are presented in Table I. We use two different values
of nearest neighbors (NN), that is 8NN and 16NN (abbreviated as OSAKD-8NN and
OSAKD-16NN respectively), and we compare their performance with the baseline
model, without applying knowledge distillation (abbreviated as w/o OSAKD).
Best results are printed in bold.

First, it can be observed that the proposed method improves the baseline
performance, for various network architectures, varying from lightweight ones
(for example the model used in the three first datasets) to heavyweight ones
(i.e. ResNet-50 in the Tiny ImageNet dataset), and also for various problems,
varying from binary to multi-class problems. It can also been observed that
even slightly better performance for the 10-class problems is also achieved
considering 8NN. However, studying the marginal case -based on the number of
classes (i.e. 200 classes)- of Tiny ImageNet, we can draw the conclusion that
in problems consisting of considerably more classes, computing the soft labels
using more nearest neighbors allows for producing more reliable soft labels
that convey more useful information about the similarities of the samples with
the classes.

Figs 1(a)-1(d) illustrate the comparisons of the mean test accuracy over the
epochs of training of the proposed method considering 8NN and 16NN for all the
utilized datasets. The enhanced performance of the OSAKD method is illustrated
in the Figures. Furthermore, the generalization ability of the proposed online
distillation method is clearly depicted. In addition, it is demonstrated that
the proposed method provides in general more stable performance, as compared
to the baseline of training without distillation. Another observation that can
be made from the Figures, is that as the training process progresses, more
meaningful soft labels are produced, since as we have already stated the soft
labels are driven by the supervised loss, and we expect to produce more
reliable soft labels throughout the training procedure.

((a)) Cifar-10

((b)) SVHN

((c)) Fashion MNIST

((d)) Tiny ImageNet

Figure 1: Test accuracy throughout the training epochs.

Furthermore, we compare the proposed methods with state-of-the-art online
distillation methods. More specifically, we utilize a common architecture,
that is ResNet-32 [18], we apply the proposed online distillation method on
Cifar-10 dataset, and compare the performance with competitive online
distillation methods, [9, 13]. For fair comparisons we follow the same
training setup as in [9, 13]. That is, we use the SGD with Nesterov momentum
and set the momentum to 0.9. The initial learning rate is set to 0.1 and drops
to 0.01 at 50% training and to 0.001 at 75%. The network is trained for 300
epochs.

We compare the proposed method with ONE [9] and FFL [13]. It should be noted
that since the proposed method is a unique branch method that does not use
multiple branches of the network, we use only two sub-networks in all the
competitive approaches, similar to [13], for as much as possible fair
comparisons. Thus, we compare the OSAKD method with ONE distillation method,
considering the average performance of the two branches, and correspondingly
with the FFL-S distillation method considering the average performance of the
two sub-networks. It should also be noted that the number of parameters in
both FFL-S and ONE cases in the test phase is identical to the OSAKD case,
since the additional branches in both cases as well as the fusion module in
FFL-S are removed during the test phase. Evaluation results are presented in
Table II. As it is demonstrated, the proposed online distillation method is
superior over competitive online distillation methods.

TABLE II: Comparisons against online distillation methods on Cifar-10 utilizing the ResNet-32 architecture. Method | Test Accuracy
---|---
ResNet-32 | 93.07% $\pm$ 0.17%
ONE [9] | 93.76%$\pm$ 0.12%
FFL-S [13] | 93.81% $\pm$ 0.12%
OSAKD | 93.93% $\pm$ 0.09%

Finally, we evaluate the complexity of the proposed method using the sum of
floating point operations (FLOPs) in one forward pass on a fixed input size.
We use the ResNet-32 model on the Cifar-10 dataset. In order to highlight the
effectiveness of the proposed method we compare the complexity with the most
famous offline KD method, [4]. In this case, we use as teacher the stronger
ResNet-110 model. The OSAKD method requires 0.07 GFLOPs, while the offline KD
requires 0.33 GFLOPs, rendering the proposed online distillation method as
significantly more efficient as compared to the conventional offline
methodology. We should finally note that competitive online distillation
methods that utilize multiple branches or copies of a given network, require
at least two times more FLOPs than the proposed one. That is, the proposed
online distillation method is also more efficient as compared to competitive
online methods.

##  V Conclusions

In this paper, we proposed a novel single-stage online self-distillation
approach, namely Online Self-Acquired Knowledge Distillation (OSAKD). The
proposed method utilizes the k-nn non-parametric density estimation in order
to estimate the unknown density distributions of the data samples in the
output feature space. In this way, we are able to directly estimate the
posterior class probabilities, revealing explicitly the similarities of the
data samples with each class. This approach, allows us for deriving additional
knowledge directly from the data, without affecting the model architecture by
adding multiple branches or employing multiple models, and at the same time in
a single stage training pipeline. The experimental evaluation indicates the
effectiveness of the proposed method to improve the performance of any model,
regardless its complexity.

## Acknowledgment

This research has been partially financially supported by General Secretariat
for Research and Technology (GSRT) and the Hellenic Foundation for Research
and Innovation (HFRI) (Scholarship Code: 2826.) and the European Union's
Horizon 2020 research and innovation programme under grant agreement No 871449
(OpenDR). This publication reflects the authors' views only. The European
Commission is not responsible for any use that may be made of the information
it contains.

## References

  * [1] L. Deng, ``A tutorial survey of architectures, algorithms, and applications for deep learning,'' _APSIPA Transactions on Signal and Information Processing_ , vol. 3, 2014.
  * [2] Y. Guo, Y. Liu, A. Oerlemans, S. Lao, S. Wu, and M. S. Lew, ``Deep learning for visual understanding: A review,'' _Neurocomputing_ , vol. 187, pp. 27 – 48, 2016.
  * [3] Y. Cheng, D. Wang, P. Zhou, and T. Zhang, ``A survey of model compression and acceleration for deep neural networks,'' 2017.
  * [4] G. Hinton, O. Vinyals, and J. Dean, ``Distilling the knowledge in a neural network,'' 2015.
  * [5] J. Yim, D. Joo, J. Bae, and J. Kim, ``A gift from knowledge distillation: Fast optimization, network minimization and transfer learning,'' in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017.
  * [6] T. Furlanello, Z. C. Lipton, M. Tschannen, L. Itti, and A. Anandkumar, ``Born again neural networks,'' in _ICML_ , 2018.
  * [7] X. Lan, X. Zhu, and S. Gong, ``Self-referenced deep learning,'' in _Proceedings of the Asian Conference on Computer Vision_ , 2018, pp. 284–300.
  * [8] Y. Zhang, T. Xiang, T. M. Hospedales, and H. Lu, ``Deep mutual learning,'' in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018.
  * [9] x. lan, X. Zhu, and S. Gong, ``Knowledge distillation by on-the-fly native ensemble,'' in _Proceedings of the Advances in Neural Information Processing Systems 31_ , S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, Eds., 2018, pp. 7517–7527.
  * [10] J. Ba and R. Caruana, ``Do deep nets really need to be deep?'' in _Proceedings of the Advances in Neural Information Processing Systems 27_ , 2014, pp. 2654–2662.
  * [11] R. O. Duda, P. E. Hart, and D. G. Stork, _Pattern classification_. John Wiley & Sons, 2012.
  * [12] R. Anil, G. Pereyra, A. Passos, R. Ormandi, G. E. Dahl, and G. E. Hinton, ``Large scale distributed neural network training through online distillation,'' 2018.
  * [13] J. Kim, M. Hyun, I. Chung, and N. Kwak, ``Feature fusion for online mutual knowledge distillation,'' 2019.
  * [14] A. Krizhevsky and G. Hinton, ``Learning multiple layers of features from tiny images,'' Citeseer, Tech. Rep., 2009.
  * [15] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng, ``Reading digits in natural images with unsupervised feature learning,'' 2011.
  * [16] H. Xiao, K. Rasul, and R. Vollgraf, ``Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms,'' 2017.
  * [17] V. Nair and G. E. Hinton, ``Rectified linear units improve restricted boltzmann machines,'' in _Proceedings of the international conference on machine learning_ , 2010, pp. 807–814.
  * [18] K. He, X. Zhang, S. Ren, and J. Sun, ``Deep residual learning for image recognition,'' in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 770–778.