marginparsep has been altered.
topmargin has been altered.
marginparwidth has been altered.
marginparpush has been altered.
The page layout violates the ICML style. Please do not change the page layout,
or include packages like geometry, savetrees, or fullpage, which change it for
you. We’re not able to reliably undo arbitrary changes to the style. Please
remove the offending package(s), or layout-changing commands and try again.

PolyThrottle: Energy-efficient Neural Network Inference on Edge Devices

Minghao Yan 1  Hongyi Wang 2  Shivaram Venkataraman 1

###### Abstract

As neural networks (NN) are deployed across diverse sectors, their energy
demand correspondingly grows. While several prior works have focused on
reducing energy consumption during training, the continuous operation of ML-
powered systems leads to significant energy use during inference. This paper
investigates how the configuration of on-device hardware—elements such as GPU,
memory, and CPU frequency, often neglected in prior studies, affects energy
consumption for NN inference with regular fine-tuning. We propose
PolyThrottle, a solution that optimizes configurations across individual
hardware components using Constrained Bayesian Optimization in an energy-
conserving manner. Our empirical evaluation uncovers novel facets of the
energy-performance equilibrium showing that we can save up to 36 percent of
energy for popular models. We also validate that PolyThrottle can quickly
converge towards near-optimal settings while satisfying application
constraints.

††footnotetext:  1University of Wisconsin-Madison 2Carnegie Mellon University.
Correspondence to: Minghao Yan<EMAIL_ADDRESS>
Figure 1: Figure illustrating the overall workflow of PolyThrottle. The
optimizer first identifies the optimal hardware configuration for a given
model. When new data arrives, the inference server handles the inference
requests. Upon receiving a fine-tuning request, our performance predictor
estimates whether time-sharing inference and fine-tuning workloads would
result in SLO violations. Then the predictor searches for feasible adjustments
to meet the SLO constraints. If such adjustments are identified, the system
implements the changes and schedules fine-tuning requests until completion.

##  1 Introduction

The rapid advancements in neural networks and their deployment across various
industries have revolutionized multiple aspects of our lives. However, this
sophisticated technology carries a drawback: high energy consumption which
poses serious sustainability and environmental challenges Anderson et al.
(2022); Gupta et al. (2022); Cao et al. (2020); Anthony et al. (2020).
Emerging applications such as autonomous driving systems and smart home
assistants require real-time decision-making capabilities jet , and as we
integrate NNs into an ever-growing number of devices, their collective energy
footprint poses a considerable burden to our environment Wu et al. (2022);
Schwartz et al. (2020); Lacoste et al. (2019). Moreover, considering that many
devices operate on battery power, curbing energy consumption not only
alleviates environmental concerns but also prolongs battery life, making low-
energy NN models highly desirable for numerous use cases.

In prior literature, strategies for reducing energy consumption revolve around
designing more efficient neural network architectures Howard et al. (2017);
Tan & Le (2019), quantization Kim et al. (2021); Banner et al. (2018);
Courbariaux et al. (2015; 2014); Gholami et al. (2021), or optimizing maximum
GPU frequency You et al. (2022); Gu et al. (2023). From our experiments, we
make new observations about the tradeoffs between energy consumption,
inference latency, and various other hardware configurations. Memory
frequency, for example, emerges as a significant contributor to energy
consumption (as shown in Figure 3), beyond the commonly investigated
relationship between maximum GPU compute frequency and energy consumption.
Table 2 shows that even with optimal maximum GPU frequency, we can save up to
$25\%$ energy by further tuning memory frequency. In addition, minimum GPU
frequency also proves to be of importance in certain cases, as shown in Figure
4 and Table 3.

We also observe that a simple linear relationship falls short of capturing the
tradeoff between energy consumption, neural network inference latency, and
hardware configurations. The complexity of this tradeoff is illustrated by the
Pareto Frontier in Figure 2. This nuanced interplay between energy consumption
and latency poses a challenging question: How can we find a near-optimal
configuration that closely aligns with this boundary?

Designing an efficient framework to answer the above question is challenging
due to the large configuration space, the need to re-tune each model and
hardware, and frequent fine-tuning operations. A naive approach, such as grid
search, is inefficient and can take hours to find the optimal solution for a
given model and desired batch size on a given hardware. The uncertainty in
inference latency, especially at smaller batch sizes Gujarati et al. (2020),
further exacerbates the challenge. Furthermore, given that distinct hardware
platforms and NN models display unique energy consumption patterns (Section
3), relying on a universally applicable pre-computed optimal configuration is
not feasible. Every deployed device must be equipped to quickly identify its
best configuration tailored to its specific workload. Finally, in production
environments, daily fine-tuning is often necessary to adapt to a dynamic
external environment and integrate new data Cai et al. (2019; 2020). This
demands a mechanism that can quickly adjust configurations to complete fine-
tuning requests in time while ensuring the online inference workloads meet
Service Level Objectives (SLOs).

In this paper, we explore the interplay between inference latency, energy
consumption, and hardware frequency and propose PolyThrottle as our solution.
PolyThrottle takes a holistic approach, optimizing various hardware components
and batch sizes concurrently to identify near-optimal hardware configurations
under a predefined latency SLO. PolyThrottle complements existing efforts to
reduce inference latency, including pruning, quantization, and knowledge
distillation. We use Constrained Bayes Optimization with GPU, memory, CPU
frequencies, and batch size as features, and latency SLO as a constraint to
design an efficient framework that automatically adjusts configurations,
enabling convergence towards near-optimal settings. Furthermore, PolyThrottle
uses a performance prediction model to schedule fine-tuning operations without
disrupting ongoing online inference requests. We integrate PolyThrottle into
Nvidia Triton on Jetson TX2 and Orin and evaluate on state-of-the-art CV and
NLP models, including EfficientNet and Bert Tan & Le (2019); Devlin et al.
(2018).

To summarize, our key contributions include:

1\. We examine the influence of hardware components beyond GPUs on energy
consumption, delineate new tradeoffs between energy consumption and inference
performance, and reveal new possibilities for optimization.

2\. We construct an adaptive framework that efficiently finds energy-optimal
hardware configurations. To accomplish this, we employ Constrained Bayesian
Optimization.

3\. We develop a performance model to capture the interaction between
inference and fine-tuning processes. We use this model to schedule fine-tuning
requests and carry out real-time modifications to meet inference SLOs.

4\. We implement and evaluate PolyThrottle on a state-of-the-art inference
server on Jetson TX2 and Orin. With minimal overheads, PolyThrottle reduces
energy consumption per query by up to $36\%$.

Figure 2: Left figure shows the Pareto Frontier of energy vs. latency tradeoff
for various batch sizes on EfficientNet B7 on Jetson Orin. Right figure shows
the Pareto Frontier of energy vs. latency tradeoff for various batch sizes on
EfficientNet B4 on Jetson TX2. Each data point in this plot is representative
of a unique hardware configuration, and each line corresponds to a batch size.
The figure shows that the tradeoff does not always conform to the same pattern
across varied hardware platforms and models.

##  2 Motivation

Many deep neural networks have been deployed on edge devices to perform tasks
such as image classification, object detection, and dialogue systems.
Scenarios including smart home assistants He et al. (2020), inventory and
supply chain monitoring jet , and autopilot Gog et al. (2022) often use
battery-based devices that contain GPUs to perform the aforementioned tasks.
In these scenarios, pre-trained models are installed on the devices where the
inference workload is deployed.

Prior works have focused on optimizing the energy consumption of GPUs Wang et
al. (2020b; 2021); Tang et al. (2019); Strubell et al. (2019); Mei et al.
(2017) in cloud scenarios Qiao et al. (2021); Wan et al. (2020); Hodak et al.
(2019) and training settings Wang et al. (2020a); Peng et al. (2019); Kang et
al. (2022). On-device inference workloads exhibit different characteristics
and warrant separate attention. In this section, we outline previous efforts
in optimizing on-device neural network inference and discuss our approach to
holistically optimize energy consumption.

###  2.1 On-device Neural Network Deployment

Prior work in optimizing on-device neural network inference focuses on
quantization Kim et al. (2021); Banner et al. (2018); Courbariaux et al.
(2015; 2014); Gholami et al. (2021), designing hardware-friendly network
architectures Xu et al. (2019); Lee et al. (2019); Sanh et al. (2019); Touvron
et al. (2021); Howard et al. (2019), and leveraging hardware components
specific to mobile settings, such as DSPs Lane & Georgiev (2015). Our work
explores an orthogonal dimension and aims to answer a different question:
Given a neural network to deploy on a specific device, how can we tune the
device to reduce energy consumption?

In our work, we focus on edge devices that contain CPUs, memory, and GPUs.
These devices are generally more powerful than DSPs often found on mobile
devices. One such example is the Nvidia Jetson series, which is capable of
handling a wide array of applications, ranging from AI to robotics and
embedded IoT solutions jet . The devices also come with dynamic voltage and
frequency scaling (DVFS) capabilities that allow for the optimization of power
consumption and thermal management during complex computational tasks. The
Jetson series features a unified memory shared by both CPU and GPUs. We refer
to the operating frequency of CPU, GPU, and shared memory as CPU frequency,
GPU frequency, and memory frequency in this paper.

Case Study on Inventory Management: To understand the system requirements in
edge NN inference, we next describe a case study of how NNs are deployed in an
inventory management company. From our conversations, Company A works with
Customer B to deploy neural networks on edge devices to optimize inventory
management. To comply with regulations and protect privacy, data from each
inventory site are required to be stored locally. The vast difference in the
layout of the inventories makes it impossible to pre-train the model on data
from every warehouse. Therefore, these devices come with a pre-trained model
based on data from a small sample of inventories, which may have significantly
different layouts and external environments compared to the actual deployment
venue. Consequently, daily fine-tuning is required to enhance performance in
the deployed sites, as the environment continually evolves. Similar arguments
apply to smart home devices, where a model is pre-trained on selected
properties, but the deployed households may be much more diverse. To address
privacy concerns, on-device fine-tuning of neural networks is preferred, as it
keeps sensitive data locally. Therefore, edge devices often need to run both
inference and periodic fine-tuning. Combining multiple workloads on edge
devices can lead to SLO violations due to interference and increased energy
use.

###  2.2 Holistic Energy Consumption Optimization

Some recent works have explored reducing energy consumption by optimizing for
batch size and GPU maximum frequency You et al. (2022); Nabavinejad et al.
(2021); Komoda et al. (2013); Gu et al. (2023) and developing power models for
modern GPUs Kandiah et al. (2021); Hong & Kim (2010); Arafa et al. (2020);
Lowe-Power et al. (2020). In this work, we argue that other hardware
components also cause energy inefficiency and require separate optimization.
We perform a grid search over GPU, memory, and CPU frequencies and various
batch sizes to examine the Pareto frontier of inference latency and energy
consumption. Figure 2 shows the tradeoff between the per-query energy
consumption and inference latency (normalized to the optimal latency) on
Jetson TX2 and Jetson Orin. Each point in the figure represents the optimal
configuration that we find through grid search under a given inference latency
budget and batch size. As Figure 2 shows, the Pareto frontier is not smooth
globally and is difficult to capture by a simple model, which warrants more
sophisticated optimization techniques to quickly converge to a hardware
configuration that lies on the Pareto Frontier Censor (1977).

Zeus You et al. (2022) attempts to reduce the energy consumption of neural
network training by changing the GPU power limit and tuning training batch
size. PolyThrottle also includes these two factors. In Zeus You et al. (2022),
the focus is on training workloads in data center settings, where batch size
tuning helps achieve an accuracy threshold in an energy-efficient way. We
include batch size as part of PolyThrottle as it provides a trade-off between
inference latency and throughput. Our empirical evaluation reveals new avenues
available for optimization which complicates the search space, as we describe
next.

##  3 Opportunities

In this section, we perform empirical experiments to uncover new opportunities
for optimizing energy use in NN inference. As discussed in Section 2, prior
work did not study how memory frequency, minimum GPU frequency, and CPU
frequency play a role in energy consumption. This is partially limited by
hardware constraints. Specialized power rails need to be built into the device
during manufacturing to enable accurate measurement of energy consumption
associated with each component. We leverage two Jetson developer kits, TX2 and
Orin, which offer native support for component-wise energy consumption
measurement and frequency tuning, to study how these frequencies impact
inference latency and energy consumption in modern deep learning workloads. We
find that the default frequencies are much larger than optimal and throttling
all these frequency knobs offer energy consumption reduction with minimal
impact on inference latency.

Figure 3 illustrates the energy optimization landscape when varying GPU and
memory frequencies, without imposing any constraints on latency SLO. The plot
reveals that, without any other constraints, the energy optimization landscape
generally exhibits a bowl shape. However, this shape varies depending on the
models, devices, and other hyperparameters, such as batch sizes (See Appendix
B for more results). Next, we dive into how each hardware component affects
inference energy consumption.

Setup:  Experiments in this section are performed with 16-bit floating point
number precision, as it has been demonstrated to have minimal impact on model
accuracy in practice. We use Bert and EfficientNet models and vary the
EfficientNet model size between B0, B4, B7 (Table 4).

Figure 3: This figure shows per query energy cost as we vary the GPU frequency
and memory frequency for EfficientNet B4 on Jetson TX2 versus varying memory
and GPU frequency with batch size fixed at 1.

Memory frequency experiment: For each model, we fix the GPU frequency at the
optimal frequency determined by grid-search of all possible frequency
configurations. We then examine the tradeoff between inference latency and
energy consumption as we progressively throttle memory frequency. The range of
available memory frequencies can be found in Table 1.

Results:  Table 2 reveals that memory frequency plays a vital role in reducing
energy consumption. The savings provided by memory frequency tuning are
similar and consistent across models on both hardware platforms, ranging from
approximately $12\%$ to $25\%$. This indicates that the default memory
frequency is higher than optimal for modern Deep Learning workloads. For heavy
workloads such as Bert, memory tuning can account for the majority of the
energy consumption reduction. This can be partially attributed to the memory-
bound nature of Transformer-based models Ivanov et al. (2021). Our result
demonstrates that systems that aim to optimize energy use in neural network
inference need to take memory frequency into account.

Table 1: This table shows the energy frequency range for Jetson TX2 and Jetson Nano in MHz. Device | Min GPU | Max GPU | Min Mem | Max Mem
---|---|---|---|---
TX2 | 114.75 | 1300.5 | 40.8 | 1866
Orin | 114.75 | 1300.5 | 204 | 3199
Table 2: This table shows the optimal memory frequency (in MHz) and the corresponding energy savings for various models on Jetson TX2. B0/B4/B7 represent different models in the Efficient Net series. Model | Energy Reduction | Optimal Mem Freq
---|---|---
B0 | 14.9% | 1331
B4 | 14.3% | 1331
B7 | 12.0% | 1331
Bert Base | 25.4% | 1062

CPU Frequency Experiment:  CPUs are only used for data pre-processing. Thus,
we first measure the time spent in the data processing part of the inference
pipeline. Next, we measure the energy saved by throttling the CPU frequency
and assess the inference latency slowdown caused by reducing CPU frequency.
The data preprocessing we perform is standard in almost all image processing
and object detection pipelines, where we read the raw image file, convert it
to an RGB scale, resize it, and reorient it to the desired input resolution
and data layout.

Results:  The preprocessing time across different EfficientNet models remains
constant since the operations performed are identical. As a result, the
relative impact of CPU tuning on overall energy consumption depends on the
ratio between preprocessing time and inference time. As the model size
increases and inference duration increases, the influence of CPU tuning on
overall energy consumption decreases. We observe that on both Jetson TX2 and
Orin platforms, CPU tuning can decrease preprocessing energy consumption by
approximately $30\%$. Depending on the model, quantization level, and batch
size, this results in up to a $6\%$ reduction in overall energy consumption.

Table 3: This table shows the optimal minimum GPU frequency (MHz) and the corresponding energy savings for various models with 16-bit floating-point precision on Jetson Orin. B0/B4/B7 represent different models in the Efficient Net series. |  |  | Optimal Min
---|---|---|---
Model | Size | Energy Reduction | GPU Freq
B0 | 5.3M | 47.6% | 1236
B4 | 19M | 29.1% | 1033
B7 | 66M | 8.8% | 1134
Bert Base | 110M | 1.1% | 217
Figure 4: This figure shows per query energy cost as we vary the minimum and
maximum GPU frequency. As we increase the minimum GPU frequency, energy cost
decreases.

Minimum GPU frequency experiment: We maintain the default hardware
configuration and only adjust the minimum GPU frequency on Jetson Orin.
Increasing the minimum GPU frequency forces the GPU DVFS mechanism to operate
within a smaller range. We scale the model from EfficientNet B0 to
EfficientNet B7 to illustrate the effect of the GPU minimum frequency on
inference latency.

Results:  Table 3 indicates that tuning the minimum GPU frequency can
significantly reduce energy consumption when the workload cannot fully utilize
the computational power of the hardware. Notably, energy consumption and
inference latency are reduced by forcing the GPU to operate at a higher
frequency. This differs from the tradeoff observed in other experiments, where
we exchange inference latency for lower energy consumption. Tuning minimum GPU
frequency can nearly halve the energy consumption for small models. As
computational power becomes saturated with increasing model size, the return
on tuning the minimum GPU frequency diminishes.

Figure 4 shows the per query energy cost as we vary the minimum and maximum
GPU frequency. It shows that increasing the minimum GPU frequency from the
default minimum leads to lower energy costs and inference latency.

##  4 Architecture Overview

To take advantage of the opportunities described in the previous section, we
design PolyThrottle, a system that navigates the tradeoff between latency SLO,
batch size, and energy. PolyThrottle optimizes for the most energy-efficient
hardware configurations under performance constraints and handles scheduling
of on-device fine-tuning.

Figure 1 shows a high-level overview of PolyThrottle’s workflow. In a
production environment, sensors on the edge devices continuously collect data
and send the data to the deployed model for inference. In the meantime, to
adapt to a changing environment and data patterns, these data are also saved
for fine-tuning later. Due to the limited computation resources on these edge
devices, fine-tuning workloads are often scheduled in conjunction with the
continuously running inference requests. To address the challenges in model
deployment on edge devices, PolyThrottle consists of two key components:

1\. An optimization framework that finds optimal hardware configurations for a
given model under predetermined SLOs using few samples.

2\. A performance predictor and scheduler to dynamically schedule fine-tuning
requests and adjust for the optimal hardware configuration while satisfying
SLO.

PolyThrottle tackles these challenges separately. Offline, we automatically
find the best CPU frequency, GPU frequency, memory frequency, and recommended
batch size for inference requests that satisfy the latency constraints while
minimizing per-query energy consumption. We discuss the details of the
optimization procedure in Section 5. We also show that our formulation can
find near-optimal energy configurations in a few minutes using just a handful
of samples. Compared to the lifespan of long-running inference workloads, the
overhead is negligible.

The optimal configuration is then installed on the inference server. At
runtime, the client program processes the input and sends inference requests
to the inference server. Meanwhile, if there are pending fine-tuning requests,
the performance predictor predicts the inference latency when running
concurrent fine-tuning, and decides whether it is possible to satisfy the
latency SLO if fine-tuning is scheduled concurrently. A detailed discussion on
performance prediction can be found in Section 6. The scheduler then decides
what the new configuration that can satisfy the latency SLO while minimizing
per-query energy consumption is. If such a configuration is attainable, it
will schedule fine-tuning requests iteration-by-iteration until all pending
requests are finished.

Online vs. Offline: Adjusting the frequency of each hardware component entails
writing to one or multiple hardware configuration files, a process that takes
approximately 17ms each. On Jetson TX2 and Orin, each CPU core, GPU, and
memory has a separate configuration file that determines operating frequency.
As a result, setting the operating frequencies for CPUs, GPU, and memory could
require up to 150ms. This duration could exceed the latency SLO for many
applications, and this is without accounting for the additional overhead of
completing frequency changes. Since the latency SLO for a specific workload
does not change frequently, PolyThrottle determines the optimal hardware
configuration before deployment and only performs online adjustments to
accommodate fine-tuning workloads.

##  5 Problem Formulation: Two-phase Tuning

Our objective is to automatically find the optimal hardware configurations
that minimize energy consumption while satisfying latency SLOs. Formally, we
are solving the optimization problem:

$\displaystyle\min\quad$ $\displaystyle~{}f(x_{CPU},\ x_{GPU_{min}},\
x_{GPU_{max}},\ x_{Mem},\ b)$ $\displaystyle\mathrm{s.t.}\quad$
$\displaystyle~{}t(x_{CPU},\ x_{GPU_{min}},\ x_{GPU_{max}},\ x_{Mem},\ b)\leq
c$

where $f$ and $t$ represent the energy consumption and latency associated with
a workload under the given hardware configurations and batch size. We use
$x_{CPU},\ x_{GPU_{min}},\ x_{GPU_{max}},x_{Mem}$ to denote the frequency
limit of CPU, GPU, and memory on the device, and use $b$ to denote the maximum
batch size. We use $c$ to denote the inference latency SLO limit, which is set
by application users. This optimization problem is challenging on several
fronts:

1\. The search space is large, and performing a grid search will take hours
depending on the model size. On TX2 and Orin, there are 5005 and 1820 points
in the grid if we allow 5 different batch sizes. An exhaustive search would
take 14 and 5 hours, respectively.

2\. To satisfy latency constraints, it is hard to decouple each dimension and
optimize them separately as they jointly affect inference latency in a non-
trivial way.

3\. The optimization landscape may vary across models and devices.

We observe that CPU frequency can be decoupled from GPU frequency, memory
frequency, and batch size, as it mainly affects the preprocessing latency and
energy consumption. We pipeline the requests so different requests can use CPU
and GPU resources at the same time to increase inference throughput.

Based on this observation, we propose a two-phase hardware tuning framework,
where CPU tuning is done separately from tuning other hardware components. The
challenge that remains is to efficiently optimize for an unknown function with
noise. As shown in Figure 3 and 4, the performance of neural network inference
with changes in memory and GPU frequency is difficult to predict, therefore, a
good solution must be able to handle the variance while converging to a near-
optimal configuration in a sample-efficient fashion. This requires the method
to adaptively balance the tradeoff between exploration and exploitation. To
solve this, we formulate the optimization problem as a Bayesian Optimization
problem and leverage recent advances in the field to incorporate the SLO
constraints unique to our setting.

###  5.1 Constrained Bayesian Optimization

Bayesian Optimization is a prevalent method for hyperparameter tuning
Kandasamy et al. (2020); Klein et al. (2015), as it can optimize various
black-box functions. This method is especially advantageous when evaluating
the objective function is expensive and requires a substantial amount of time
and resources.

However, some applications may involve constraints that must be satisfied in
addition to optimizing the objective function. Constrained Bayesian
Optimization (CBO) Gardner et al. (2014) is an extension of Bayesian
Optimization that tackles this challenge by incorporating constraints into the
optimization process.

In CBO, the objective function and constraints are treated as distinct
functions. The optimization algorithm seeks to identify the set of input
parameters that maximize the objective function while adhering to the
constraints. These constraints are usually expressed as inequality constraints
that must be satisfied during the optimization process. The expected
constrained improvement acquisition function in CBO is defined as follows:
$EI_{C}(\hat{x})=PF(\hat{x})\times EI(\hat{x})$.

Here $EI(\hat{x})$ represents the expected improvement (EI) Brochu et al.
(2010) within an unconstrained Bayesian Optimization scenario, while
$PF(\hat{x})$ is a univariate Gaussian cumulative distribution function,
delineating the anticipated probability of whether $\hat{x}$ can fulfill the
constraints. Intuitively, EI chooses the next configuration by optimizing the
expected improvement relative to the best recently explored configuration. In
PolyThrottle, we choose EI since our empirical findings and corroborations
from additional studies Alipourfard et al. (2017) show that EI performs better
than other widely-used acquisition functions Snoek et al. (2012).

CBO Gardner et al. (2014) also employs a joint prior distribution over the
objective and constraint functions that captures their correlation structure.
This joint prior is constructed by assuming that the objective and constraint
functions are drawn from a multivariate Gaussian distribution with a
parameterized mean vector and covariance matrix. These hyperparameters are
learned from data using maximum likelihood estimation.

During the optimization process, the algorithm uses this joint prior to
compute an acquisition function that balances exploration (sampling points
with high uncertainty) and exploitation (sampling points where the objective
function is expected to be low and subject to feasibility constraints). The
algorithm then selects the next point to evaluate based on this acquisition
function. During each iteration, the algorithm will test whether the selected
configuration violates any of the given constraints and take the result into
account for the next iteration. Encoding more system-specific hints as
constraints can be of independent research interests, however, we show in
Section 7 that the current formulation performs well under a variety of
scenarios.

##  6 Modeling Workload Interference

Consider the case where we run an inference workload and aim to support fine-
tuning without interfering with the online inference process. When a fine-
tuning request arrives, we need to decide if it is possible to execute the
fine-tuning request without violating inference SLOs. Time-sharing has been
the default method for sharing GPU workloads. In time-sharing, shared
workloads use different time slices and alternate GPU use between them.
Recently, CUDA streams, Multiprocess Service (MPS) NVIDIA (2023a) and MIG
NVIDIA (2023b) have been proposed to perform space-sharing on GPUs. However,
these approaches are not supported on edge GPU devices Bai et al. (2020); Zhao
et al. (2023); Yu & Chowdhury (2020); Wu et al. (2021). Given this setup, we
propose building a performance model that can predict the inference latency in
the presence of fine-tuning requests and only execute fine-tuning requests if
the predicted latency can satisfy SLO.

Feature selection:  To build the performance model, we leverage the following
insights to select features: 1. In convolutional neural networks, the 2D
convolution layers’ performance largely determines the overall performance of
the network. Its latency is correlated to the number of floating point
operations (FLOPs) required during forward / backward propagation. 2. The
ratio between the number of FLOPs and the number of memory accesses, also
known as arithmetic intensity, together with total FLOPs, encapsulates whether
a neural network is compute-bound or memory-bound. Using these insights, we
add the following features to our model: Inference FLOPs, Inference Arithmetic
Intensity, Fine-tuning FLOPs, Fine-tuning Arithmetic Intensity, and Batchsize.

Model selection:  We propose using a linear model to predict inference latency
when a fine-tuning workload is running concurrently on the same device. The
model aims to capture how the proposed variables affect the resource
contention between the inference workload and the fine-tuning workload, and
therefore, affect the inference latency. The proposed model can be summarized
as follows:

$\displaystyle\text{Inference time}=$
$\displaystyle\theta_{0}+\theta_{1}\times\text{FLOPs}_{inf}+\theta_{2}\times
AI_{inf}$ $\displaystyle+\theta_{3}\times\text{FLOPs}_{ft}+\theta_{4}\times
AI_{ft}$ $\displaystyle+\theta_{5}\times\text{Batchsize}\vspace{-0.3cm}$

Given the above performance model, we use a Non-negative Least Squares (NNLS)
solver to find the model that best fits the training data. An advantage of
NNLS for linear models is that we can solve this with very few training data
points Venkataraman et al. (2016). We collect a few samples on the provided
model by varying the inference and fine-tuning batch sizes and the output
dimension, which captures various fine-tuning settings. This model is used as
part of the workload scheduler during deployment to predict whether it is
possible to schedule a fine-tuning request.

Fine-tuning scheduler:  During inference, when there are outstanding fine-
tuning requests, PolyThrottle uses the model to decide whether it is possible
to schedule the request online without violating the SLO. When the model finds
a feasible configuration, it adjusts accordingly until either all pending
requests are finished or a new latency constraint is imposed.

##  7 Experiments

Table 4: This table shows the scaling pattern of the EfficientNet model family. Model | Input dim / width | Width coef | Depth coef
---|---|---|---
B0 | 224 $\times$ 224 | 1.0 | 1.0
B4 | 380 $\times$ 380 | 1.4 | 1.8
B7 | 600 $\times$ 600 | 2.0 | 3.1

###  7.1 Setup

Hardware Platform: Our experiments are conducted on the Jetson TX2 Developer
Kit and Jetson Orin Developer Kit. To assess the energy consumption of our
program, we employ the built-in power monitors on the Jetson TX2 and Jetson
Orin Developer Kits. We also cross-validate our measurements with an external
digital multimeter (See Appendix A for more details on hardware and energy
measurement).

Workload Selection: We base our experiments on the EfficientNet family and
Bert models Tan & Le (2019); Devlin et al. (2018). EfficientNet is chosen not
only for its status as a state-of-the-art convolutional network in on-device
and mobile settings but also for its principled approach to scaling the width,
depth, and resolution of convolution layers. Table 4 summarizes the scaling
pattern of EfficientNet from the smallest B0 to the largest B7. We select Bert
to investigate energy usage patterns in a Transformer-based model Wolf et al.
(2020), where the workload is more memory-bounded compared to convolution-
based neural networks. Bert and its variants Kim et al. (2021); Devlin et al.
(2018); Sanh et al. (2019); Tambe et al. (2021) are widely used for Question
and Answering tasks Rajpurkar et al. (2016), making it applicable for numerous
edge devices, such as smart home assistants and smart speakers.

Dataset:  We evaluate PolyThrottle on real-world traffic streams data Shen et
al. (2019) and sample frames uniformly to feed into EfficientNet. For Bert, we
evaluate on SQuAD Rajpurkar et al. (2016) for Question Answering. Note that
datasets do not affect PolyThrottle’s performance since inference latency
would not change significantly across datasets once the model is chosen.

Implementation:  PolyThrottle is built on the Nvidia Triton inference server.
To maximize performance, we generate TensorRT kernels that profile various
data layouts and tiling strategies to identify the fastest execution graph for
a given hardware platform. Our modules include a Bayesian optimizer for
determining the best configuration, an inference client responsible for
preprocessing and submitting requests to the inference server, and a
performance predictor module integrated into the inference client for
scheduling fine-tuning requests. We maintain separate queues for inference and
fine-tuning requests.

###  7.2 Efficiently Searching for Optimal Configuration

In this experiment, we carry out an extensive empirical analysis of tuning
various models across different hardware configurations while also adjusting
the quantization level. We perform a grid search on EfficientNet B0, B4, B7,
and Bert Base to examine the potential energy savings and identify the optimal
GPU and memory frequencies for each model. We also adjust the quantization
level for each tested model. We evaluate 16-bit and 32-bit floating point
(FP16/FP32) precision. The optimal energy consumption and configuration
referenced later in this section use the results obtained here as the baseline
and optimal solution. Having obtained the optimal frequency using grid search,
we next evaluate the average number of attempts it takes for PolyThrottle to
find a solution within $5\%$ of the optimal solution. We compare our
Constrained Bayesian Optimization (CBO) formulation against Random Search
(RS).

Figure 5: This figure compares search efficiency between Constrained Bayesian
Optimization and Random Search. The y-axis represents the number of attempts
it takes to find a near-optimal configuration and the x-axis represents the
deployed and associated quantization level. The first row corresponds to the
setting where we set a latency target but restrict the batch size to 1. The
second row where we relax the latency constraint and allow batching inference
requests.

Experiment Settings:  We measure the average number of attempts needed to find
a near-optimal configuration. For Random Search, we calculate the expected
number of trials needed to find a near-optimal configuration based on the grid
size by computing the fraction of near-optimal configurations and taking the
reciprocal. For CBO, we set the $\xi$ parameter associated with the Expected
Improvement function to 0.1 and initial random samples to be 5, which we find
to work well across different models and hardware platforms. We conduct two
experiments where we set different inference latency constraints, the results
can be found in Figure 5:

1\. We restrict inference latency to close to the optimal latency ($20\%$). In
this setting, the tight latency constraints make it impossible to batch the
inference query, essentially reducing the search space for the optimal
configuration.

2\. In the second benchmark, we relax the inference latency constraint to
include the configurations that provide the lowest energy-per-query in Figure
2. In this setting, we need to explore the batch size dimension to find the
configuration that minimizes energy. We test on EfficientNet B0, B4, and B7,
as well as Bert Base on both Jetson TX2 and Jetson Orin.

Results:  Figure 5 shows that CBO outperforms RS in both scenarios. Since CBO
models the relationship between hardware configuration and latency, it can
find a near-optimal solution with only $5$ to $15$ samples. In the second
scenario, the performance of RS deteriorates as it is unable to leverage the
relationship between latency and batch size when dealing with a
multiplicatively increasing search space. Overall CBO takes 3-10x fewer
samples in the second setting. The overhead of performing CBO is also minimal.
As shown in Figure 5, CBO only requires around 15 samples to find a near-
optimal solution and the optimization procedure can be completed in a few
minutes. In cases where a new model is deployed, only a few minutes of
overhead are needed to find optimal configurations for the new model.

It is important to note that though RS might achieve performance comparable to
CBO under certain conditions, this result is merely the expected value and the
variance of RS is large. For instance, if 10 out of 200 configurations are
near-optimal, the expected number of trials needed to reach a near-optimal
configuration is 20, with a standard deviation of 19.49. Consequently, it’s
plausible that even after 40 trials, RS might still fail to identify a near-
optimal configuration. On the other hand, the standard deviation of CBO is
smaller; in all experiments, CBO’s standard deviations are less than 3.

Table 5: This table shows the SLO violation rate and job completion time of various scheduling strategies for Jetson Orin on EfficientNetB7. The baseline shows the SLO violation rate without fine-tuning. Method | Workload | SLO violation
---|---|---
Greedy | Uniform | 37.91%
Adaptive | Uniform | 2.08%
Greedy | Poisson | 60.41%
Adaptive | Poisson | 5.42%
Greedy | Twitter | 22.0%
Adaptive | Twitter | 5.8%
Baseline | Uniform | 0.4%
Baseline | Poisson | 1.67%
Baseline | Twitter | 3.8%
Table 6: This table shows the SLO violation rate and job completion time of various scheduling strategies for Jetson TX2 on EfficientNetB4. The baseline shows the SLO violation rate without fine-tuning. Method | Workload | SLO violation
---|---|---
Greedy | Uniform | 16%
Adaptive | Uniform | 5.5%
Greedy | Poisson | 35.4%
Adaptive | Poisson | 7.4%
Greedy | Twitter | 35.0%
Adaptive | Twitter | 7.5%
Inference Only | Uniform | 1%
Inference Only | Poisson | 3.5%
Inference Only | Twitter | 5.3%

###  7.3 Workload-aware Fine-tuning Scheduling

Next, we evaluate how well PolyThrottle handles fine-tuning requests alongside
inference. The central question we aim to address is whether our performance
predictor can effectively identify and adjust accordingly when the SLO
requirement is at risk of being violated, and if reducing the inference batch
size and trading throughput can satisfy the latency SLO. To simulate this
scenario, we generate two distinct inference arrival patterns (Uniform and
Poisson) and use the publicly available Twitter trace twi (2018) and compare
our adaptive scheduling approach to greedy scheduling, where a fine-tuning
request is scheduled as soon as it arrives. The three arrival patterns
represent scenarios that are highly controlled and bursty, respectively. In
this context, we contrast PolyThrottle’s adaptive scheduling mechanism with
the greedy scheduling approach to assess the efficacy of PolyThrottle in
meeting the desired SLO requirement.

Experiment Settings:  We evaluate on both synthetic and real workloads. For
synthetic workloads, we generate a stream of inference requests using both
Uniform and Poisson distributions. For real-world workload, we first uniformly
sample a day of Twitter streaming traces and then compute the variance of
requests during each minute. We then picked the segment with the highest
variance to test PolyThrottle’s capability in handling request bursts twi
(2018); Romero et al. (2021). On Jetson Orin, we replay the stream for 30
seconds and measure the SLO violation rate during the replay using
EfficientNet B7. Since each burst only lasts for a few seconds, this suffices
to capture many bursts in the workload. We find that running the experiment
for longer durations produces similar results. We set the fine-tuning batch
size to 64, the number of fine-tuning iterations to 10, SLO to 0.7s, the
output dimension to 1000, and an average of 8 inference requests per second.
On Jetson TX2, we do the same experiment on EfficientNet B4. Due to memory
constraints, we perform the fine-tuning batch size to 8, SLO to 1s, the output
dimension to 100, and an average of 4 inference requests per second. We select
a less performative model on TX2 to meet a reasonable SLO target (under 1s).
The number of fine-tuning iterations is chosen based on the duration of the
replay. We then measure the energy costs when deploying PolyThrottle at the
default and optimal hardware frequency, respectively, to measure how much
energy we save during this period. The optimal hardware frequency is obtained
from results in Section 7.2.

For greedy scheduling, we employ a standard drop policy Crankshaw et al.
(2017); Shen et al. (2019), whereby a request is dropped if it has already
exceeded its deadline. In the adaptive setting, we use the predictor to
determine whether to drop an inference request. We also replay the inference
request stream without fine-tuning requests to serve as a baseline.

Results:  Table 5 and 6 show the SLO violation rates under various workloads
and latency targets. The findings indicate that greedy scheduling may lead to
significant SLO violations owing to the interference introduced by the fine-
tuning workload. In contrast, PolyThrottle’s adaptive scheduling mechanism
demonstrates the ability to achieve low SLO violation rates by dynamically
adjusting configurations. The baseline figures in the table represent SLO
violation rates in the absence of interference from fine-tuning requests.

Inherent variance in neural network inference resulted in $1\%$ of SLO
violations in the case of Uniform distribution. However, bursts in the Poisson
distribution and the Twitter workload generated more SLO violations.
PolyThrottle’s adaptive scheduling mechanism significantly reduces the SLO
violation rate, meeting the SLO requirements while concurrently handling fine-
tuning requests. Nevertheless, in several instances, we were unable to achieve
near-zero SLO violation rates. This limitation can be attributed to the
granularity of scheduling as we process the current batch of requests over an
extended timespan due to interference from the fine-tuning workload.

We also reduce energy consumption by $14\%$ on EfficientNet B7 on Jetson Orin
and by $23\%$ on EfficientNet B4 on Jetson TX2 across the workloads. We show
in Appendix D how PolyThrottle reacts to changing SLOs when there are
outstanding fine-tuning requests.

##  8 Conclusion

In this work, we examine the unique characteristics of energy consumption in
neural network inference, especially for edge devices. We identified unique
tradeoffs and dimensions between energy consumption and inference latency SLOs
and empirically demonstrated hidden components in optimizing energy
consumption. We then propose an optimization framework that automatically and
holistically tunes various hardware components to find a configuration aligned
with the Pareto Frontier. We empirically verify the effectiveness and
efficiency of PolyThrottle. PolyThrottle also adapts to the need for fine-
tuning and proposes a simple performance prediction model to adaptively
schedule fine-tuning requests while keeping the online inference workload
under the inference latency SLO whenever possible. We hope our study sheds
more light on the hidden dimension of NN energy optimization.

## References

  * (1) Jetson partner solutions ebook.  URL https://resources.nvidia.com/en-us-jetson-success-stories/jetson-partner-solutions-ebook?lx=XRDs_y.
  * twi (2018) Twitter streaming traces, 2018.  URL https://archive.org/details/archiveteam-twitter-stream-2018-04.
  * Alipourfard et al. (2017) Alipourfard, O., Liu, H. H., Chen, J., Venkataraman, S., Yu, M., and Zhang, M.  Cherrypick: Adaptively unearthing the best cloud configurations for big data analytics.  In _NSDI_ , volume 2, pp. 4–2, 2017.
  * Anderson et al. (2022) Anderson, T., Belay, A., Chowdhury, M., Cidon, A., and Zhang, I.  Treehouse: A case for carbon-aware datacenter software.  2022\.
  * Anthony et al. (2020) Anthony, L. F. W., Kanding, B., and Selvan, R.  Carbontracker: Tracking and predicting the carbon footprint of training deep learning models.  2020\.
  * Arafa et al. (2020) Arafa, Y., ElWazir, A., ElKanishy, A., Aly, Y., Elsayed, A., Badawy, A., Chennupati, G., Eidenbenz, S., and Santhi, N.  Verified instruction-level energy consumption measurement for nvidia gpus.  2020\.
  * Bai et al. (2020) Bai, Z., Zhang, Z., Zhu, Y., and Jin, X.  Pipeswitch: Fast pipelined context switching for deep learning applications.  In _Proceedings of the 14th USENIX Conference on Operating Systems Design and Implementation_ , pp. 499–514, 2020.
  * Banner et al. (2018) Banner, R., Hubara, I., Hoffer, E., and Soudry, D.  Scalable methods for 8-bit training of neural networks.  _Advances in neural information processing systems_ , 31, 2018.
  * Brochu et al. (2010) Brochu, E., Cora, V. M., and De Freitas, N.  A tutorial on bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning.  _arXiv preprint arXiv:1012.2599_ , 2010.
  * Cai et al. (2019) Cai, H., Wang, T., Wu, Z., Wang, K., Lin, J., and Han, S.  On-device image classification with proxyless neural architecture search and quantization-aware fine-tuning.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops_ , pp. 0–0, 2019.
  * Cai et al. (2020) Cai, H., Gan, C., Zhu, L., and Han, S.  Tinytl: Reduce memory, not parameters for efficient on-device learning.  _Advances in Neural Information Processing Systems_ , 33:11285–11297, 2020.
  * Cao et al. (2020) Cao, Q., Balasubramanian, A., and Balasubramanian, N.  Towards accurate and reliable energy measurement of nlp models.  In _Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing_ , 2020.
  * Censor (1977) Censor, Y.  Pareto optimality in multiobjective problems.  _Applied Mathematics and Optimization_ , 4(1):41–59, 1977.
  * Courbariaux et al. (2014) Courbariaux, M., Bengio, Y., and David, J.-P.  Training deep neural networks with low precision multiplications.  _arXiv preprint arXiv:1412.7024_ , 2014.
  * Courbariaux et al. (2015) Courbariaux, M., Bengio, Y., and David, J.-P.  Binaryconnect: Training deep neural networks with binary weights during propagations.  _Advances in neural information processing systems_ , 28, 2015.
  * Crankshaw et al. (2017) Crankshaw, D., Wang, X., Zhou, G., Franklin, M. J., Gonzalez, J. E., and Stoica, I.  Clipper: A low-latency online prediction serving system.  In _NSDI_ , volume 17, pp. 613–627, 2017.
  * Devlin et al. (2018) Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Gardner et al. (2014) Gardner, J. R., Kusner, M. J., Xu, Z. E., Weinberger, K. Q., and Cunningham, J. P.  Bayesian optimization with inequality constraints.  In _ICML_ , volume 2014, pp. 937–945, 2014.
  * Gholami et al. (2021) Gholami, A., Kim, S., Dong, Z., Yao, Z., Mahoney, M. W., and Keutzer, K.  A survey of quantization methods for efficient neural network inference.  _arXiv preprint arXiv:2103.13630_ , 2021.
  * Gog et al. (2022) Gog, I., Kalra, S., Schafhalter, P., Gonzalez, J. E., and Stoica, I.  D3: a dynamic deadline-driven approach for building autonomous vehicles.  In _Proceedings of the Seventeenth European Conference on Computer Systems_ , pp. 453–471, 2022.
  * Gu et al. (2023) Gu, D., Xie, X., Huang, G., Jin, X., and Liu, X.  Energy-efficient gpu clusters scheduling for deep learning.  _arXiv preprint arXiv:2304.06381_ , 2023.
  * Gujarati et al. (2020) Gujarati, A., Karimi, R., Alzayat, S., Hao, W., Kaufmann, A., Vigfusson, Y., and Mace, J.  Serving $\\{$DNNs$\\}$ like clockwork: Performance predictability from the bottom up.  In _14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20)_ , pp. 443–462, 2020.
  * Gupta et al. (2022) Gupta, U., Kim, Y. G., Lee, S., Tse, J., Lee, H.-H. S., Wei, G.-Y., Brooks, D., and Wu, C.-J.  Chasing carbon: The elusive environmental footprint of computing.  _IEEE Micro_ , 42(4):37–47, 2022.
  * He et al. (2020) He, C., Li, S., So, J., Zeng, X., Zhang, M., Wang, H., Wang, X., Vepakomma, P., Singh, A., Qiu, H., et al.  Fedml: A research library and benchmark for federated machine learning.  _arXiv preprint arXiv:2007.13518_ , 2020.
  * Hodak et al. (2019) Hodak, M., Gorkovenko, M., and Dholakia, A.  Towards power efficiency in deep learning on data center hardware.  In _IEEE International Conference on Big Data_ , 2019.
  * Hong & Kim (2010) Hong, S. and Kim, H.  An integrated gpu power and performance model.  In _ISCA_ , 2010.
  * Howard et al. (2019) Howard, A., Sandler, M., Chu, G., Chen, L.-C., Chen, B., Tan, M., Wang, W., Zhu, Y., Pang, R., Vasudevan, V., et al.  Searching for mobilenetv3.  pp. 1314–1324, 2019.
  * Howard et al. (2017) Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  _arXiv preprint arXiv:1704.04861_ , 2017.
  * Ivanov et al. (2021) Ivanov, A., Dryden, N., Ben-Nun, T., Li, S., and Hoefler, T.  Data movement is all you need: A case study on optimizing transformers.  _Proceedings of Machine Learning and Systems_ , 3:711–732, 2021.
  * Kandasamy et al. (2020) Kandasamy, K., Vysyaraju, K. R., Neiswanger, W., Paria, B., Collins, C. R., Schneider, J., Poczos, B., and Xing, E. P.  Tuning hyperparameters without grad students: Scalable and robust bayesian optimisation with dragonfly.  _The Journal of Machine Learning Research_ , 21(1):3098–3124, 2020.
  * Kandiah et al. (2021) Kandiah, V., Peverelle, S., Khairy, M., Pan, J., Manjunath, A., Rogers, T. G., Aamodt, T. M., and Hardavellas, N.  Accelwattch: A power modeling framework for modern gpus.  In _MICRO_ , 2021.
  * Kang et al. (2022) Kang, D.-K., Lee, K.-B., and Kim, Y.-C.  Cost efficient gpu cluster management for training and inference of deep learning.  _Energies_ , 15(2):474, 2022.
  * Kim et al. (2021) Kim, S., Gholami, A., Yao, Z., Mahoney, M. W., and Keutzer, K.  I-bert: Integer-only bert quantization.  pp. 5506–5518. PMLR, 2021.
  * Klein et al. (2015) Klein, A., Bartels, S., Falkner, S., Hennig, P., and Hutter, F.  Towards efficient bayesian optimization for big data.  In _NIPS 2015 Bayesian Optimization Workshop_ , 2015.
  * Komoda et al. (2013) Komoda, T., Hayashi, S., Nakada, T., Miwa, S., and Nakamura, H.  Power capping of cpu-gpu heterogeneous systems through coordinating dvfs and task mapping.  In _2013 IEEE 31st International Conference on computer design (ICCD)_. IEEE, 2013.
  * Lacoste et al. (2019) Lacoste, A., Luccioni, A., Schmidt, V., and Dandres, T.  Quantifying the carbon emissions of machine learning.  _arXiv preprint arXiv:1910.09700_ , 2019.
  * Lane & Georgiev (2015) Lane, N. D. and Georgiev, P.  Can deep learning revolutionize mobile sensing?  In _Proceedings of the 16th international workshop on mobile computing systems and applications_ , pp. 117–122, 2015.
  * Lee et al. (2019) Lee, J., Chirkov, N., Ignasheva, E., Pisarchyk, Y., Shieh, M., Riccardi, F., Sarokin, R., Kulik, A., and Grundmann, M.  On-device neural net inference with mobile gpus.  _arXiv preprint arXiv:1907.01989_ , 2019.
  * Lowe-Power et al. (2020) Lowe-Power, J., Ahmad, A. M., Akram, A., Alian, M., Amslinger, R., Andreozzi, M., Armejach, A., Asmussen, N., Beckmann, B., Bharadwaj, S., et al.  The gem5 simulator: Version 20.0+.  _arXiv preprint arXiv:2007.03152_ , 2020.
  * Mei et al. (2017) Mei, X., Wang, Q., and Chu, X.  A survey and measurement study of gpu dvfs on energy conservation.  _Digital Communications and Networks_ , 3(2):89–100, 2017.
  * Nabavinejad et al. (2021) Nabavinejad, S. M., Reda, S., and Ebrahimi, M.  Batchsizer: Power-performance tradeoff for dnn inference.  In _Proceedings of the 26th Asia and South Pacific Design Automation Conference_ , 2021.
  * NVIDIA (2023a) NVIDIA.  Stream management, 2023a.  URL https://docs.nvidia.com/cuda/cuda-runtime-api.
  * NVIDIA (2023b) NVIDIA.  Nvidia multi-instance gpu, 2023b.  URL https://docs.nvidia.com/datacenter/tesla/mig-user-guide/index.html.
  * Peng et al. (2019) Peng, Y., Zhu, Y., Chen, Y., Bao, Y., Yi, B., Lan, C., Wu, C., and Guo, C.  A generic communication scheduler for distributed dnn training acceleration.  In _SOSP_ , 2019.
  * Qiao et al. (2021) Qiao, A., Choe, S. K., Subramanya, S. J., Neiswanger, W., Ho, Q., Zhang, H., Ganger, G. R., and Xing, E. P.  Pollux: Coadaptive cluster scheduling for goodput-optimized deep learning.  In _OSDI_ , 2021.
  * Rajpurkar et al. (2016) Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P.  Squad: 100,000+ questions for machine comprehension of text.  In _EMNLP_ , 2016.
  * Romero et al. (2021) Romero, F., Li, Q., Yadwadkar, N. J., and Kozyrakis, C.  $\\{$INFaaS$\\}$: Automated model-less inference serving.  In _2021 USENIX Annual Technical Conference (USENIX ATC 21)_ , pp. 397–411, 2021.
  * Sanh et al. (2019) Sanh, V., Debut, L., Chaumond, J., and Wolf, T.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.  _arXiv preprint arXiv:1910.01108_ , 2019.
  * Schwartz et al. (2020) Schwartz, R., Dodge, J., Smith, N. A., and Etzioni, O.  Green ai.  _Commun. ACM_ , 63(12):54–63, 2020.
  * Shen et al. (2019) Shen, H., Chen, L., Jin, Y., Zhao, L., Kong, B., Philipose, M., Krishnamurthy, A., and Sundaram, R.  Nexus: A gpu cluster engine for accelerating dnn-based video analysis.  In _Proceedings of the 27th ACM Symposium on Operating Systems Principles_ , pp. 322–337, 2019.
  * Snoek et al. (2012) Snoek, J., Larochelle, H., and Adams, R. P.  Practical bayesian optimization of machine learning algorithms.  _Advances in neural information processing systems_ , 25, 2012.
  * Strubell et al. (2019) Strubell, E., Ganesh, A., and McCallum, A.  Energy and policy considerations for deep learning in nlp.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , 2019.
  * Tambe et al. (2021) Tambe, T., Hooper, C., Pentecost, L., Jia, T., Yang, E.-Y., Donato, M., Sanh, V., Whatmough, P., Rush, A. M., Brooks, D., et al.  Edgebert: Sentence-level energy optimizations for latency-aware multi-task nlp inference.  In _MICRO_ , 2021.
  * Tan & Le (2019) Tan, M. and Le, Q.  Efficientnet: Rethinking model scaling for convolutional neural networks.  pp. 6105–6114. PMLR, 2019.
  * Tang et al. (2019) Tang, Z., Wang, Y., Wang, Q., and Chu, X.  The impact of gpu dvfs on the energy and performance of deep learning: An empirical study.  In _Proceedings of the Tenth ACM International Conference on Future Energy Systems_ , 2019.
  * Touvron et al. (2021) Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jégou, H.  Training data-efficient image transformers & distillation through attention.  pp. 10347–10357. PMLR, 2021.
  * Venkataraman et al. (2016) Venkataraman, S., Yang, Z., Franklin, M., Recht, B., and Stoica, I.  Ernest: Efficient performance prediction for $\\{$Large-Scale$\\}$ advanced analytics.  In _13th USENIX Symposium on Networked Systems Design and Implementation (NSDI 16)_ , pp. 363–378, 2016.
  * Wan et al. (2020) Wan, C., Santriaji, M., Rogers, E., Hoffmann, H., Maire, M., and Lu, S.  Alert: Accurate learning for energy and timeliness.  In _ATC_ , 2020.
  * Wang et al. (2021) Wang, F., Zhang, W., Lai, S., Hao, M., and Wang, Z.  Dynamic gpu energy optimization for machine learning training workloads.  _IEEE Transactions on Parallel and Distributed Systems_ , 2021.
  * Wang et al. (2020a) Wang, G., Venkataraman, S., Phanishayee, A., Devanur, N., Thelin, J., and Stoica, I.  Blink: Fast and generic collectives for distributed ml.  In _Proceedings of Machine Learning and Systems_ , 2020a.
  * Wang et al. (2020b) Wang, Y., Wang, Q., Shi, S., He, X., Tang, Z., Zhao, K., and Chu, X.  Benchmarking the performance and energy efficiency of ai accelerators for ai training.  In _20th IEEE/ACM International Symposium on Cluster, Cloud and Internet Computing (CCGRID)_ , 2020b.
  * Wolf et al. (2020) Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Scao, T. L., Gugger, S., Drame, M., Lhoest, Q., and Rush, A.  Transformers: State-of-the-art natural language processing.  In _EMNLP_ , 2020.
  * Wu et al. (2022) Wu, C.-J., Raghavendra, R., Gupta, U., Acun, B., Ardalani, N., Maeng, K., Chang, G., Aga, F., Huang, J., Bai, C., Gschwind, M., Gupta, A., Ott, M., Melnikov, A., Candido, S., Brooks, D., Chauhan, G., Lee, B., Lee, H.-H., Akyildiz, B., Balandat, M., Spisak, J., Jain, R., Rabbat, M., and Hazelwood, K.  Sustainable ai: Environmental implications, challenges and opportunities.  In _Proceedings of Machine Learning and Systems_ , 2022.
  * Wu et al. (2021) Wu, X., Rao, J., Chen, W., Huang, H., Ding, C., and Huang, H.  Switchflow: preemptive multitasking for deep learning.  In _Proceedings of the 22nd International Middleware Conference_ , pp. 146–158, 2021.
  * Xu et al. (2019) Xu, M., Liu, J., Liu, Y., Lin, F. X., Liu, Y., and Liu, X.  A first look at deep learning apps on smartphones.  In _The World Wide Web Conference_ , WWW ’19, pp. 2125–2136, 2019\.
  * You et al. (2022) You, J., Chung, J.-W., and Chowdhury, M.  Zeus: Understanding and optimizing gpu energy consumption of dnn training.  _arXiv preprint arXiv:2208.06102_ , 2022.
  * Yu & Chowdhury (2020) Yu, P. and Chowdhury, M.  Fine-grained gpu sharing primitives for deep learning applications.  _Proceedings of Machine Learning and Systems_ , 2:98–111, 2020.
  * Zhao et al. (2023) Zhao, Y., Liu, X., Liu, S., Li, X., Zhu, Y., Huang, G., Liu, X., and Jin, X.  Muxflow: Efficient and safe gpu sharing in large-scale production deep learning clusters.  _arXiv preprint arXiv:2303.13803_ , 2023.

##  Appendix A Hardware Details

###  A.1 Jetson platform details

The Jetson TX2 Developer Kit features a 256-core NVIDIA Pascal GPU, a Dual-
Core NVIDIA Denver 2 64-bit CPU, a Quad-Core ARM Cortex-A57 MPCore CPU, and
8GB of 128-bit LPDDR4 memory with 59.7 GB/s bandwidth. The kit’s maximum power
consumption is 15W. The Jetson Orin Developer Kit includes a 2,048-core NVIDIA
Ampere GPU with 64 Tensor Cores and a 12-core Arm CPU. This kit comes with
32GB of 256-bit LPDDR5 memory, featuring a 204.8GB/s bandwidth, and has a
maximum power consumption of 60W.

###  A.2 Power consumption measurement

The Nvidia Jetson TX2 Developer Kit allows for separate measurements of GPU,
CPU, DDR, and total energy consumption, while the Jetson Orin uses the built-
in tegrastats module for measuring power usage across hardware components. Due
to power rail design limitations, GPU power usage on the Jetson Orin can only
be measured alongside SoC power usage.

On Jetson TX2, we measure power usage by querying the total power input. We
then average the peak power consumption to obtain the power usage during
inference. Then we compute the energy cost for each inference request by
multiplying the power and the inference time. On Jetson Orin, we leverage the
existing tegrastats tool and repeatedly query tegrastats at a fixed interval
(50ms). We then sum up each component’s power consumption to obtain the
overall power consumption, before multiplying the power and the inference time
to obtain the energy cost for each inference request. To obtain a steady
reading, we send 1000 inference requests for each hardware configuration for
every model that we test.

We cross-validate our measurements using a USB digital multimeter capable of
transmitting data to computer software in real-time via Bluetooth. The
measurements obtained from the multimeter generally align with those from the
internal power rails on Jetson Kits, although external measurements are
consistently around $10\%$ higher than Jetson internal measurements. This
discrepancy may be attributable to unaccounted factors in the power rail
design. We opted to use internal measurements since they provide component-
specific readings, whereas the multimeter can only measure overall energy
consumption. Moreover, the multimeter supports one measurement per second,
while Jetson tools allow for millisecond-scale measurements, which are better
suited to inference workloads.

###  A.3 Measurement Overhead

Since we repeatedly query the power input or built-in power management tool,
we want to understand whether these queries affect total energy consumption.
We use a USB digital multimeter capable of transmitting data to computer
software in real-time via Bluetooth. We then run our inference program with
and without querying the power input or the power management tool. We find
that the power consumption reported by the multimeter increases around $5\to
10\%$, depending on the base power consumption. We observe that this increment
is near constant across different models and runs and therefore we believe
using internal measurement as described in the section above will not affect
our findings. The multimeter cannot provide the precision and flexibility we
need to measure the energy cost of inference, which often operates at a
millisecond scale.

##  Appendix B Experimental Results

In this section, we further demonstrate the tradeoff between memory frequency
and maximum GPU frequency by presenting an array of results. These results
underline the interesting observation that the energy consumption patterns may
vary for the same model operating on different devices. Furthermore, even for
the same model-device pairing, the optimization landscape can be significantly
influenced by the batch size. This underlines the complexities of energy
optimization and the need for an adaptive framework that can take these
factors into account. Figures $6-12$ show the energy consumption patterns of
EfficientNet and Bert on Jetson TX2 and Orin under various batch sizes. Table
7 shows the optimal CPU frequency and corresponding energy consumption
reduction in image preprocessing.

Figure 6: This figure shows per query energy cost as we vary the GPU frequency and memory frequency for Bert at FP16 on Jetson TX2 versus varying Memory and GPU frequency with batch size fixed at 1. Figure 7: This figure shows per query energy cost as we vary the GPU frequency and memory frequency for Bert at FP32 on Jetson TX2 versus varying Memory and GPU frequency with batch size fixed at 1. Figure 8: This figure shows per query energy cost as we vary the GPU frequency and memory frequency for Bert at FP16 on Jetson TX2 versus varying Memory and GPU frequency with batch size fixed at 8. Figure 9: This figure shows per query energy cost as we vary the GPU frequency and memory frequency for EfficientNet B4 at FP16 on Jetson TX2 versus varying Memory and GPU frequency with batch size fixed at 16. Figure 10: This figure shows per query energy cost as we vary the GPU frequency and memory frequency for EfficientNet B7 at FP16 on Jetson TX2 versus varying Memory and GPU frequency with batch size fixed at 16. Figure 11: This figure shows per query energy cost as we vary the GPU frequency and memory frequency for EfficientNet B7 at FP16 on Jetson Orin versus varying Memory and GPU frequency with batch size fixed at 8. Figure 12: This figure shows per query energy cost as we vary the GPU frequency and memory frequency for EfficientNet B7 at FP16 on Jetson Orin versus varying Memory and GPU frequency with batch size fixed at 1. Figure 13: This figure shows per query energy cost as we vary the GPU frequency and memory frequency for EfficientNet B4 at FP16 on Jetson Orin versus varying Memory and GPU frequency with batch size fixed at 8. Table 7: This table shows how CPU frequency (MHz) affects energy consumption for image processing on Jetson TX2 and Jetson Orin. |  | Optimal Min
---|---|---
Device | Energy Reduction | CPU Freq
Orin | 28.6% | 900
TX2 | 29.4% | 1000

##  Appendix C Arithmetic intensity

The arithmetic intensity of a 2D convolution layer can be computed by the
following equation:

$\displaystyle AI_{conv}=\frac{N\cdot K\cdot P\cdot Q\cdot C\cdot R\cdot
S}{N\cdot C\cdot H\cdot W+K\cdot C\cdot R\cdot S+N\cdot K\cdot P\cdot Q}$ (1)

The notations used in equation 1 can be found in table 8.

Table 8: The notation table defines the variable used in equation 1 Variable | Definition
---|---
$N$ | Batch size
$C$ | Input number of channels
$H$ | Input tensor height
$W$ | Input tensor width
$K$ | Output number of channels
$P$ | Output tensor height
$Q$ | Output tensor width
$R$ | Convolution filter height
$S$ | Convolution filter width

The FLOPs term captures the total computation of each workload, while the
arithmetic intensity term captures how much computation power and memory
bandwidth will affect the final performance. Combining the aforementioned
features with an intercept term, which captures the fixed overhead in neural
network inference, we can build a model that predicts inference latency if the
hardware operating frequency is stable.

##  Appendix D Predictor Analysis

We vary the latency SLO to assess how the predictor schedules the fine-tuning
requests. We replay a 60-second stream where we initially set the latency SLO
to 250ms for the first half (30 seconds), and then increase it to 700ms for
the remainder. As shown in Figure 14, under stringent latency conditions, the
predictor deduces that it is impractical to schedule fine-tuning requests
while adhering to the latency SLO, hence no fine-tuning requests are
scheduled. Conversely, when the latency SLO is more relaxed, the predictor
determines that it is feasible to schedule fine-tuning requests and
sequentially schedules each request once the preceding one is completed and
has issued a completion signal.

Figure 14: This figure shows the inference latency SLO and when fine-tuning
requests are scheduled.