tcb@breakable

# Toward Sustainable GenAI using Generation Directives for Carbon-Friendly
Large Language Model Inference

Baolin Li1, Yankai Jiang1, Vijay Gadepally2, Devesh Tiwari1
1 Northeastern University, 2 MIT

###### Abstract

The rapid advancement of Generative Artificial Intelligence (GenAI) across
diverse sectors raises significant environmental concerns, notably the carbon
emissions from their cloud and high performance computing (HPC)
infrastructure. This paper presents Sprout, an innovative framework designed
to address these concerns by reducing the carbon footprint of generative Large
Language Model (LLM) inference services. Sprout leverages the innovative
concept of “generation directives” to guide the autoregressive generation
process, thereby enhancing carbon efficiency. Our proposed method meticulously
balances the need for ecological sustainability with the demand for high-
quality generation outcomes. Employing a directive optimizer for the strategic
assignment of generation directives to user prompts and an original offline
quality evaluator, Sprout demonstrates a significant reduction in carbon
emissions by over 40% in real-world evaluations using the Llama2 LLM and
global electricity grid data. This research marks a critical step toward
aligning AI technology with sustainable practices, highlighting the potential
for mitigating environmental impacts in the rapidly expanding domain of
generative artificial intelligence.

##  I Introduction

The emergence of Generative Artificial Intelligence (GenAI) has significantly
impacted various sectors such as scientific discovery, engineering, law, and
finance [1, 2, 3], signaling a major shift in how challenges and tasks are
approached in these fields. This technology’s ability to produce novel content
from existing data has cemented its popularity in datacenters worldwide.
However, the AI boom, driven by the demand for GenAI, has prompted concerns
over its environmental impact, particularly in terms of carbon emissions
associated with the energy-intensive nature of these technologies. OpenAI’s
reported pursuit of trillions in investment for AI chips [4], destined for
cloud and high performance computing (HPC) datacenters, underscores the scale
of infrastructure expansion required to support GenAI’s growth. With global
datacenter energy consumption projected to more than double from 460 TWh in
2022 to 1000 TWh by 2026 [5], the consequent surge in electricity generation
to power these facilities could contribute to 8% of global carbon emissions
within a decade [6], highlighting the urgent need for sustainable practices in
the rapidly expanding realm of artificial intelligence.

Generative Large Language Models (LLMs), such as ChatGPT, experience over 1
billion visits monthly, underscoring an urgent need for research focused on
minimizing their environmental impact. Training these models requires
extensive compute cycles and corresponding carbon footprint. However, it is
the inference processes of these LLMs that are poised to become the
predominant source of emissions, according to various prior studies [7, 8, 9].
Unlike traditional natural language understanding models that predict a single
masked word or sentiment, generative LLMs are even more carbon-demanding as
they perform iterative predictions for each request until reaching a
predefined token or iteration limit. Despite the critical nature of this
issue, there’s a noticeable gap in research dedicated to reducing carbon
emissions specifically from the inference operations of generative language
models. Addressing this gap is crucial for making GenAI advancements
sustainable and environmentally responsible.

In this paper, we design Sprout as the first work to address the
sustainability challenges in running a generative LLM inference service.
Various previous works have attempted to reduce the carbon footprint of
machine learning (ML) applications in cloud and HPC datacenters [8, 10, 11,
12], but none has designed optimizations tailored to LLM inference which is
becoming a dominant workload in information technology and requires immediate
intervention to reduce its carbon footprint. The following summarized the
insights behind Sprout and its key contributions.

Introduction of generation directives to LLM inference for carbon saving.
Previous works have identified the opportunity to manipulate the number of
parameters in the model to save energy and carbon [13, 10, 14], while Sprout
is the first work to identify that in generative language model inference, its
autoregressive generation pattern presents a unique opportunity outside of the
dimension that any previous work has explored. Sprout introduces the concept
of “generation directives”, a strategy to indirectly manipulate the number of
autoregressive inference iterations while providing high-quality content
generation. For example, a directive can guide the model to provide a concise
response, saving significant carbon from generating a long sequence while
still being accurate. Identifying the variability in the carbon intensity of
the electricity generation and the diverse requirements of different tasks,
Sprout can leverage different generation directives to minimize the carbon
footprint of LLM inference with a guarantee of generation quality.

Design and implementation of carbon-optimized generation directive
configuration for LLM inference.  We present Sprout, an innovative carbon-
aware generative language model inference framework designed to reduce carbon
footprint through the strategic use of token generation directives while
maintaining high-quality outputs. From the selection of directive levels based
on electricity grid carbon intensity and user behavior variability, Sprout
introduces a linear programming approach for system-level optimization,
balancing carbon savings with generation quality. Sprout identifies the
difficulty in retrieving generation quality feedback, and implements an
automatic offline quality assessment mechanism to ensure the framework’s
decisions are informed by up-to-date generation quality.

Evaluation of Sprout with real-world LLM and electricity grid operators.  Our
extensive evaluation of the Sprout system demonstrates its effectiveness in
reducing carbon emissions of LLM inference services by more than 40% while
still achieving high generation quality. We evaluate the system using
production software setup, the latest open-source Llama2 LLM, representative
corpus to synthesize user prompts, and real carbon intensity traces from
multiple global electricity grid operator regions. Our real-system prototype
demonstrates that Sprout is superior to its competitors and is well-aligned to
a hypothetical yet unattainable Oracle scheme. These results suggest Sprout
offers a meaningful step forward in making LLM inference systems more
environmentally friendly, contributing to the ongoing effort to align GenAI
technology with sustainable practices.

##  II Background and Motivation

###  II-A Background

Carbon footprint of an inference request.  The carbon footprint is a metric
for quantifying the amount of greenhouse gas emissions (gCO2) generated. When
requesting a service from a cloud and HPC datacenter (e.g., HTTP requests),
its carbon footprint comprises the operational carbon and embodied carbon. The
operational carbon comes from the electricity grid that powers the datacenter,
which powers the hardware (e.g., GPUs) that serves the request. The carbon
intensity (denoted as $CO_{2}^{\text{Intensity}}$) of the grid, representing
the amount of CO2 emission per energy usage (gCO2/kWh), reflects the
“greenness” of the energy source. For example, wind turbines have lower carbon
intensity than coal power plants. Due to the difference in availability and
stability of renewable energy, carbon intensity varies significantly over time
and across geographical regions. The carbon intensity is the multiplier to the
request’s energy consumption when quantifying its operational carbon
footprint.

Embodied carbon (denoted as $CO_{2}^{\text{Embed}}$) represents the carbon
emissions associated with the manufacturing and packaging of computer
components, effectively “embodied” within the device itself. For an inference
request processed by a computing device, its share of embodied carbon is
proportional to the execution time relative to the device’s overall
operational lifespan. More detailed information about the embodied and
operational carbon footprint in sustainable computing is available in previous
works [15, 6]. The total carbon footprint of serving an inference request,
$C_{\text{req}}$, can be formally expressed as:

$\displaystyle C_{\text{req}}=CO_{2}^{\text{Intensity}}\cdot
E_{\text{req}}+\frac{CO_{2}^{\text{Embed}}}{T_{\text{life}}}\cdot
T_{\text{req}}$ (1)

Here, $E_{\text{req}}$ and $T_{\text{req}}$ represent the energy consumption
and execution time for the request, respectively, with $T_{\text{life}}$
indicating the assumed device lifespan, set to five years for this analysis.
Given that the lifespan of the device significantly exceeds any single
request’s execution time, operational carbon predominantly dictates the total
carbon footprint, except in scenarios where $CO_{2}^{\text{Intensity}}$
approaches zero.

Figure 1: The auto-regressive generation process of generative language model
inference.

Generative language model inference.  Transformers have revolutionized
language models, enabling systems like BERT [16] to predict missing words
within sentences, thus enhancing our understanding of language contexts. Yet,
with the rise of applications such as ChatGPT, generative large language
models have taken center stage. These models diverge from mere language
understanding by engaging in autoregressive token generation – taking a user
prompt and iteratively predicting subsequent tokens until an end-of-sequence
(EOS) token emerges or a predefined limit is reached, as shown in Fig. 1. A
key component supporting this process is the KV cache, which stores key and
value vectors from previously processed tokens. This mechanism allows for
subsequent tokens to be processed with attention scores computed against all
prior KV vectors without the need for recomputation, enabling the LLM to
efficiently generate significantly more tokens than the input prompt. As a
result, the computational and carbon footprint during the inference phase is
primarily driven by token generation, rather than the initial pre-filing phase
to populate the input prompt’s KV vectors [17]. Note that in the context of
this work, all LLMs we refer to are generative models. For a deeper dive into
the intricacies of LLM inference, readers are encouraged to consult previous
works [18, 19].

###  II-B Discoveries and Opportunities

In this section, we introduce a unique discovery for generative language model
inference that distinguishes this paper from all previous works and discuss
how Sprout can exploit it to save inference carbon footprint.

Figure 2: Two factors that impact a request’s carbon footprint during LLM
inference: (a) the number of model parameters and (b) the number of generated
tokens.

Takeaway 1. The carbon footprint of LLM inference depends on not only the
model size but also the number of tokens generated, presenting a new
opportunity to reduce carbon without reducing the model size.

In Fig. 2 (a), we demonstrate how the carbon footprint of LLM inference
changes with model size, showcasing examples with the Llama2 model at 7
billion and 13 billion parameters. Previous studies, such as those by INFaaS
[14], Clover [10], and ALERT [13], have delved into optimizing model
parameters to reduce costs, carbon, and energy consumption. Yet, our research
uncovers a previously unexplored factor in generative AI that significantly
influences carbon emissions: the number of tokens generated in response to a
prompt.

In Fig. 2 (b), we execute a series of input prompts on the Llama2 13B model
and observe that there is a strong linear correlation between the total carbon
emission and the volume of tokens generated from request. Despite initial
computations to pre-fill the KV cache with key and value vectors from the
input prompt, we show that the overall carbon emission of a request is largely
dictated by the quantity of generated tokens. This revelation opens up a novel
pathway for optimizing the carbon efficiency of generative language model
inference. Rather than downsizing the model and potentially compromising its
contextual understanding capabilities, maintaining the model’s size while
focusing on generating fewer, more concise tokens can be the key step toward
more sustainable GenAI.

Figure 3: (a) Using generation directives can control the number of generated
tokens while providing accurate responses. (b) Hosting larger models (e.g.,
Llama2 13B) with generation directives is better than hosting smaller models
(e.g., Llama2 7B) in terms of both carbon emission and correctness.

Takeaway 2. Incorporating generation directives into prompts can significantly
reduce the carbon footprint by enabling concise yet accurate responses.

To control the length of token generation by a LLM, we introduce a novel
concept termed ”generation directive,” defined as follows:

###### Definition 1

A generation directive is an instruction associated with a prompt input that
dictates the manner in which a generative language model produces tokens for
the prompt. Each generation directive level specifies a pre-defined text
sequence that acts as this guiding instruction.

Similar to a compiler directive, which orchestrates the program compilation
process, a generation directive strategically influences the LLM’s token
generation for an input prompt (details in Sec. III-E). In Fig. 3 (a), we show
a prompt from the popular MMLU task [20]. Without using specific directives
(denoted as level L0), the Llama2 13B model defaults to generating an
extensive number of tokens to elucidate the selection. However, such detailed
background information may not always align with user preferences.
Implementing a generation directive at level L1, designed to prompt the LLM
toward brief responses, ensures both brevity and correctness. This application
demonstrates a significant reduction in carbon emissions from generated
tokens, as evidenced previously in Fig. 2 (b). Since the generation-directive-
induced tokens are stored in the KV cache (Sec. II-A) to incur minimal
additional emissions, a generation directive would significantly enhance the
carbon efficiency of generative language model inference.

Fig. 3 (b) demonstrates that employing generation directives with a larger
model (13B, L1) significantly outperforms smaller models (7B, L0) in both
carbon efficiency and the accuracy of generated content. This is attributed to
the larger model’s superior contextual understanding, which, when combined
with concise generation directives, retains its comprehensive knowledge base
without unnecessary verbosity. This approach not only reduces the carbon
footprint but also ensures high-quality outputs, highlighting the strategic
advantage of optimizing response generation over simply reducing model size.

Figure 4: Applying generation directives across different applications reveals
variability in sensitivity to these directives, impacting both carbon
emissions and the accuracy of the generated content.

Figure 5: System Design Overview of Sprout.

Takeaway 3: The impact of employing generation directives on carbon emissions
and accuracy differs across user prompts, presenting a systemic challenge in
optimally utilizing these directives, particularly in the context of
fluctuating carbon intensity.

We have shown the effectiveness of generation directives on the MMLU task but
in reality, the user can submit all kinds of prompts. In Fig. 4, we show the
impacts of different generation directives (L0, L1, L2) on different tasks
including science knowledge [21] and trivia knowledge [22]. We can observe
that both the amount of carbon emission and the generation’s correctness rate
vary with the task. The findings indicate that while directives promoting
concise responses may decrease accuracy in complex, multi-step reasoning
tasks, they could enhance it in tasks where answers are directly inferable
from the prompt or learned context. Therefore, a system configuring the
generation directives must have a generation quality evaluator (Sec. III-A) to
provide feedback when the user prompts’ sensitivity to directive levels varies
over time.

In addition, all the CO2 emissions in this section are shown with a constant
carbon intensity of 100 gCO2/kWh and power usage effectiveness (PUE) of 1.2,
while in the real world, the carbon intensity changes all the time from the
varying energy source mixture [23, 24]. Responding to these challenges, we
design Sprout, a generative language model inference framework that takes
advantage of generation directives to dynamically optimize the carbon
footprint while guaranteeing high-quality generations.

##  III Design

###  III-A System Overview

Sprout is designed as the first carbon-aware generative language model
inference framework, utilizing token generation directives to minimize carbon
footprint while ensuring high-quality content generation. Fig. 5 shows a brief
design overview of Sprout. Once the users prompts are assigned to an inference
server by the load balancer, the prompts need to be tokenized into numerical
representations. In this phase, a generation directive selector 1 assigns a
directive level to each prompt, integrating it into the tokenized input. The
policy to assign the directive levels is determined by the Sprout’s token
generation directive optimizer 2 (Sec. III-B), which is based on the current
electricity grid’s carbon intensity and the token generation quality and
carbon feedback.

To retrieve the local carbon intensity, we can access third-party API
endpoints such as Electricity Maps [25]. To enable inference carbon feedback,
we can monitor the datacenter PUE and device energy with tools such as nvidia-
smi to record the GPU power and processing time of requests and save the logs
to the database. However, obtaining the token generation quality feedback is a
different process from the above metrics. After autoregressive inference
concludes on the inference server 3, the generated tokens are detokenized and
sent back to the user clients, while simultaneously, the request and node
monitoring logs are archived in the database. A generation quality evaluator 4
then extracts a sample of prompts from the database, generates responses for
each at all generation directive levels, and identifies the directive level
that yields the best response for each request.

Determining the optimal level for quality generation presents a challenge due
to the subjective nature of preference and the absence of a definitive best
response for user prompts, making manual evaluation by humans impractical.
Following a methodology from recent research [26], an LLM-based automatic
evaluator, rather than human evaluators, is employed to assess generation
feedback, aligning with common academic and industry practices [27, 28, 29].
This evaluator consults an auto-evaluation LLM 5 to gauge its preference on
the responses, logging these preferences back into the database. The whole
process happens offline, and since the evaluation process also incurs carbon
emission, Sprout’s opportunistic evaluation invoker 6 (Sec. III-C) ensures the
evaluations are carried out only as necessary and during periods of low carbon
intensity.

Next, we explore the foundational mechanisms of Sprout, focusing on its
strategic use of generation directives via the token generation directive
optimizer to both reduce the carbon footprint and ensure the generation of
high-quality content.

###  III-B Generation Directive Optimizer

Section II-B illustrates that while employing generation directives to reduce
token output in the autoregressive process is beneficial for lowering carbon
emissions, it poses a risk to content quality. Two key external factors
further complicate this balance: the regional carbon intensity powering the
datacenter, which directly affects the efficacy of carbon savings, and the
nature of user prompts, which influences the impact of generation directives
on both emissions and content quality. To address these challenges, Sprout’s
optimizer is designed to dynamically adjust to fluctuations in carbon
intensity and the variability of user prompt tasks. In scenarios of low carbon
intensity, Sprout prioritizes directives that enhance content quality,
leveraging the reduced carbon cost of generating new tokens. Conversely, under
high carbon intensity, it opts for directives that may slightly compromise
quality but significantly reduce emissions. This strategic approach underpins
the mathematical formulation of the Sprout optimizer, ensuring that it aligns
with the dual objectives of environmental sustainability and content quality.

Optimization variable.  The core challenge lies in selecting the optimal
generation directive for each user request to minimize carbon emissions while
ensuring satisfactory generation quality. However, optimizing directive levels
on a per-prompt basis introduces several practical complications: (i)
Dimensionality challenge: optimizing for a per-prompt basis brings up the
dimensionality challenge as the number of dimensions equals the number of
requests at each optimization step. (ii) Computational overhead: the
optimization is in the critical path before the autoregressive inference
starts and could introduce significant overhead as tens to hundreds of new
requests can arrive every second. Solving a high-dimension optimization
problem would require significant compute cycles that delay the time to first
token (TTFT). (iii) Predictability issues: anticipating the specific impact of
each directive level on carbon emissions and content quality for individual
prompts is challenging. While general trends can be inferred from historical
data, the unique nature of each prompt means accurate predictions are only
feasible post-inference.

Considering the outlined design challenges, Sprout adopts a system-level
optimization strategy for generation directive levels, rather than an
impractical per-prompt optimization. It achieves this by determining the
probability of selecting each directive level for any user prompt received.
Let’s denote $n$ as the total number of available generation directive levels.
The optimization variable, represented as
$\mathbf{x}=[x_{0},x_{1},\ldots,x_{n-1}]^{T}$, defines $x_{i}\in[0,1]$ as the
probability of applying the $i$-th directive level to any prompt, with $x_{0}$
representing the baseline directive L0 (indicating no directive). To ensure
every prompt receives a directive level, the condition
$\sum_{i=0}^{n-1}x_{i}=1$ must be satisfied. This system-wide probabilistic
approach to directive selection, while not optimizing for individual prompts,
is shown to achieve carbon savings close to those of an impractical per-prompt
Oracle optimizer, as detailed in Sec. V.

Objective function.  The primary goal of Sprout is to minimize the carbon
footprint associated with each inference request. The objective function,
$f(\mathbf{x})$, encapsulates the expected carbon footprint of an inference
request, detailed in Eq.1 (Sec. II-A). It incorporates: (i) the current
regional carbon intensity ($k_{0}$ in $gCO_{2}/kWh$), obtained via API; (ii)
the prorated per-second embodied carbon of the inference hardware through its
device lifetime ($k_{1}$ in $gCO_{2}/s$); and (iii) the profiles of energy
consumption ($\mathbf{e}$) and processing time ($\mathbf{p}$) for requests
employing various generation directive levels. The vectors
$\mathbf{e}=[e_{0},e_{1},\ldots,e_{n-1}]^{T}$ and
$\mathbf{p}=[p_{0},p_{1},\ldots,p_{n-1}]^{T}$ represent the average energy (in
kWh) and processing time (in seconds), respectively, for recent requests at
each directive level, retrievable from the database. Following Eq. 1, the
formula for calculating the expected carbon emissions of an inference request
is thus given by:

$\displaystyle
f(\mathbf{x})=k_{0}\cdot\mathbf{e}^{T}\mathbf{x}+k_{1}\cdot\mathbf{p}^{T}\mathbf{x},$
(2)

where $\mathbf{x}$ denotes the probabilities of selecting each directive level
across all user prompts.

Generation quality constraints.  The last piece of information the optimizer
needs is quality feedback. The generation quality evaluator reports the auto-
evaluation LLM’s preference on which directive level is the best for all
sampled requests. Let $\mathbf{q}=[q_{0},q_{1},\ldots,q_{n-1}]^{T}$ where
$q_{i}\in[0,1]$ denote the preference rate of each directive level reported by
the evaluator. For example, if $\mathbf{q}=[0.5,0.3,0.2]^{T}$, it means 50% of
the time, the auto-evaluator prefers the response generated using directive
L0, 30% of the time by L1 and 20% of the time by L2. We can denote the
expected generation quality as $\mathbf{q}^{T}\mathbf{x}$. During the
optimization, we need to make sure the preference rate does not deviate beyond
a threshold of $\xi\in[0,1]$ away from the $q_{0}$ generation baseline using
directive $L0$. In addition, Sprout designs the actual quality deviation from
$q_{0}$ to vary based on the current carbon intensity – when the carbon
intensity is low, the constraint should be more strictly enforced (deviation
closer to 0) since renewable energy is abundant in the grid to support high-
quality generation, and vice versa, during high carbon intensity periods, the
deviation should be closer to $\xi$. This can be formulated as an inequality
constraint:

$\displaystyle\mathbf{q}^{T}\mathbf{x}\geq(1-\frac{k_{0}-k_{0}^{\text{min}}}{k_{0}^{\text{max}}-k_{0}^{\text{min}}}\cdot\xi)\cdot
q_{0}$ (3)

where $k_{0}^{\text{min}}$ and $k_{0}^{\text{max}}$ are the known historical
minimum and maximum carbon intensities, respectively. The parameter $\xi$,
adjustable according to system requirements, facilitates a balance between
carbon footprint and content quality. For Sprout’s evaluation (detailed in
Sec. V), we set $\xi$ to 0.1. This setting dictates that no matter how high
the carbon intensity is, the system must select directive levels that ensure
the auto-evaluation LLM’s preference for generated content remains at least
90% as favorable as it would be using the baseline directive L0.

Problem formulation.  We can construct the overall optimization problem using
the objective function in Eq. 2 and the constraint in Eq. 3 along with other
inherent constraints of $\mathbf{x}$. For simplicity, we replace the right-
hand side of Eq. 3 with scalar $q_{lb}$ to represent the quality lower bound.
We have

$\displaystyle\min_{\mathbf{x}\in\mathbb{R}^{n}}\begin{aligned}
&f(\mathbf{x})\end{aligned}$ (4) $\displaystyle\text{s.t.
}\;\mathbf{q}^{T}\mathbf{x}\geq q_{lb},$ (5)
$\displaystyle\;\;\;\;\;\;\;\forall i,\;0\leq x_{i}\leq 1,$ (6)
$\displaystyle\;\;\;\;\;\;\sum_{i=0}^{n-1}x_{i}=1$ (7)

where the inequality constraint Eq. 6 indicates that the probability of each
level is within the range of 0 to 1, and the equality constraint Eq. 7
indicates that all probabilities sum to 1. We can observe that the objective
function Eq. 2 is linear because both $\mathbf{e}^{T}$ and $\mathbf{p}^{T}$
are constants to the optimization variable $\bm{x}$. In addition, all the
constraints in Eq. 5, 6 and 7 are all linear to $\mathbf{x}$. Therefore, we
have mapped the optimal generation directive level configuration problem to a
linear programming problem and we can use the HiGHS dual simplex solver [30]
to find the optimal solution for $\mathbf{x}$.

###  III-C Opportunistic Offline Quality Assessment

In Eq. 5, Sprout relies on the $\mathbf{q}^{T}$ vector to impose the quality
constraint. As a carbon-friendly generative language model inference
framework, Sprout not only cares about the carbon footprint of the inference
server but also the quality evaluation process, especially when the auto-
evaluation LLM can have $>10\times$ number of parameters than the inference
model (e.g., the GPT-4 model with a Mixture-of-Experts architecture is
estimated to have 220B parameters per expert [31] comparing against the Llama2
13B model). Note that the quality evaluation is not in the critical path of
online inference serving because it is not latency-critical and thus can be
done offline in a different server as an application decoupled from inference.

Consequently, Sprout adopts an opportunistic approach to performing generation
quality evaluations, triggering them based on specific carbon intensity
thresholds of the evaluation server. This strategy is informed by the premise
that (i) access to the auto-evaluation LLM might be facilitated exclusively
via third-party APIs, such as OpenAI’s API, and (ii) the volume of evaluation
requests remains constant across cycles, as detailed in Sec. III-E. This
method ensures that Sprout’s carbon footprint overhead from the quality
evaluation is minimized.

When deciding on whether to evaluate at the current time $t$, it’s critical to
weigh the carbon intensity of the LLM at the current moment, denoted as
$k_{2}^{(t)}$, against the time elapsed since the last evaluation at $t_{0}$.
Direct and frequent evaluations can lead to unnecessary carbon emissions
without significant benefit, whereas delayed evaluations can undermine the
optimizer’s reliability, as the $\mathbf{q}^{T}$ vector becomes outdated (Sec.
III-B). To mitigate these issues, we first enforce a grace period to ensure
the evaluation does not occur too frequently, then introduce an urgency
multiplier to the carbon intensity to capture the increasing need for re-
evaluation as time progresses. The urgency-adjusted carbon intensity
$k_{2}^{\prime(t)}$ can be expressed as

$\displaystyle k_{2}^{\prime(t)}=e^{-\beta(t-t_{0})}\cdot k_{2}^{(t)}$ (8)

Figure 6: Process in selecting the generation quality evaluation opportunity
(marked as a golden star). The grace period is depicted by a red area, while
the carbon intensity threshold is indicated by a green horizontal line. (a)
For an offline evaluation to be deemed appropriate, the urgency-adjusted
carbon intensity, $k_{2}^{\prime(t)}$, must fall within the green zone.
Instances marked by two red crosses, despite showing a positive second-order
derivative, do not qualify for evaluation due to their positioning outside the
eligible range. (b) Even if carbon intensity stays high all the time, the
increasing evaluation urgency ensures that offline evaluation always occurs.

The urgency parameter, $\beta$, determines the rate at which the evaluation
interval incurs penalties over time, ensuring that the value of immediate
evaluation – offering a timely update to the $\mathbf{q}^{T}$ vector in
Sec.III-B – is weighed against waiting for potentially lower future carbon
intensities. Setting $\beta$ to 0.028, for instance, halves the urgency-
adjusted carbon intensity, $k_{2}^{\prime(t)}$, relative to the actual carbon
intensity, $k_{2}^{(t)}$, after a 24-hour lapse without evaluation. An offline
evaluation is initiated under three conditions: (i) $t_{s}$ represents a local
minimum for $k_{2}^{\prime(t)}$, indicating a positive second-order derivative
at that point; (ii) a grace period has elapsed since the last evaluation;
(iii) the urgency-adjusted carbon intensity at $t_{s}$,
$k_{2}^{\prime(t_{s})}$, falls below a predefined threshold, such as 50% of
the historical maximum carbon intensity. This evaluative mechanism,
illustrated in Fig. 6, highlights moments of evaluation marked by stars in two
different cases, underlining the proactive approach of Sprout in scheduling
evaluations with consideration for both carbon intensity and the need for
timely quality feedback.

###  III-D Miscellaneous Design Considerations

Role of auto-evaluation LLM.  The auto-evaluation LLM, boasting orders of
magnitude more parameters than the inference model, might seem like an ideal
choice for processing user prompts. However, utilizing a giant model like
GPT4, with its estimated 1.76 trillion parameters [31], entails considerable
development, training, and deployment resources, making it impractical for
most organizations due to high costs and environmental impact. Also, directly
serving millions of user prompts on such a model incurs significantly more
carbon emissions than a model with billions of parameters. Therefore, for most
cases, it is better to fine-tune an open-sourced model like Llama2 to tailor
to the user targets and use third-party LLMs like GPT4 for occasional quality
feedback.

There may be instances where the auto-evaluator’s preferences diverge from an
individual user’s expectations, as users might have varying inclinations
toward the conciseness or detail of responses. In such cases, the inference
service could proactively notify users when responses are condensed due to
elevated carbon intensity levels, subsequently inquiring about their
preference for more detailed answers. Should a user client express a
preference for depth, Sprout can then specifically mark this preference by
applying the baseline directive, L0, to all their future prompts, ensuring
tailored responses that align more closely with their expectations.

Number of evaluation samples.  According to the sample size theory in [32], at
each quality evaluation, with a confidence level of 95%, if we sample 500 user
prompts, the maximum margin of error is only 4.4%. Therefore, we use a fixed-
sized 500 request samples to provide generation quality feedback. This fixed
sample size of 500 is chosen for generating quality feedback within Sprout,
considering its minimal impact relative to the total volume of prompts
processed during the evaluation period. Consequently, the carbon emissions
associated with these evaluations are deemed negligible and are not factored
into the carbon footprint reduction strategy detailed in Sec. III-B.

###  III-E Implementation

Figure 7: Sprout implements generation directive level assignment as LLM
system prompts.

Applying generation directive levels.  The inference service provider
specifies the number of directive levels and the actual directive to apply for
each level. Sprout implements the generation directives as the system prompt
alongside the user prompt, as the system prompt is widely accepted as a
prompting format compatible with leading AI platforms like OpenAI ChatML [33],
Llama [34], Anthropic Claude [35], MistralAI [36], etc. Figure 7 illustrates
Sprout’s method of incorporating a specific directive, such as the text from
level L1, directly into the inference request as a system prompt. When a
system prompt already exists within a user prompt, Sprout precedes it with the
selected generation directive, ensuring seamless integration.

Inference server and monitoring.  Sprout seamlessly integrates with existing
inference server setups by processing system prompts together with user
prompts, avoiding the need for infrastructure alterations. Mirroring industry-
standard LLM inference practices, the server incorporates vLLM [17] for its
high-throughput and efficient KV cache management and utilizes FlashAttention
[37] to streamline self-attention computations at the CUDA kernel level. To
accurately log execution metrics as outlined in Eq. 2, the CarbonTracker [38]
package has been adapted to monitor each inference processing node,
facilitating the calculation of $\mathbf{e}^{T}$ and $\mathbf{p}^{T}$ vectors
essential for optimizing Sprout’s operation.

Figure 8: A simplified example of Sprout’s quality evaluation query. Box 1
represents the instructions and outputs generated using different directives,
box 2 represents the quality evaluator template, and box 3 represents the
query in ChatML [33] format to the auto-evaluation LLM.

Automatic quality evaluation.  We extend the AplacaEval [39] project to build
Sprout’s quality evaluator. Specifically, we generalized the auto-annotator to
be able to query the auto-evaluation LLM to select the best one from an
arbitrary number of generations, each corresponding to a specific generation
directive level. We also implemented shuffling of the directive-guided
generations to remove position bias in the query. The evaluator is
meticulously implemented to prompt the auto-evaluation LLM to generate minimal
tokens – just enough to identify the preferred output prior to the EOS token.
This design is both carbon-efficient and cost-effective as commercial LLMs
charge based on the number of tokens generated. A simplified example is shown
in Fig. 8 where when we send the query to auto-evaluation LLM, it will
generate “Output (1)” as the preferred output. We have manually examined the
preference of several auto-evaluation LLMs (GPT-4, GPT-4 Turbo, GPT-3.5 Turbo)
and confirm that the evaluator accurately identifies the correct response in
over 97% of cases.

##  IV Methodology

Experiment setup.  The experiments are carried out on a testbed comprising two
nodes, each equipped with two NVIDIA A100 40GB Tensor Core GPUs and two AMD
EPYC 7542 CPUs. The Llama2 13B model, a prominent large language model
released by Meta [40], is utilized to establish the inference server, with
each GPU hosting a model instance within its 40GB HBM memory. To assess
Sprout’s efficiency, three levels of generation directives are implemented: L0
as the default baseline with no directives, L1 for “brief” generation, and L2
for “very brief” generation. GPT-4, accessed via the OpenAI API, serves as the
auto-evaluation LLM for offline quality assessments.

TABLE I: Language modeling tasks to evaluate Sprout.

Dataset | Description | Task
---|---|---
Alpaca [41] |  Instructions generated by OpenAI’s text-davinci-003 | Instruction tuning
GSM8K [42] |  Grade school math problems |  Arithmetic and multi-step reasoning
MMLU [20] |  Massive multitask language understanding |  Multiple-choice questions
Natural Questions [43] |  Real-user questions from Google |  Question answering
ScienceQA [21] |  School science subjects (e.g., Biology/Physics/Chemistry) |  Multiple-choice science questions
TriviaQA [22] |  Trivia questions collected by trivia enthusiasts |  Reading comprehension

Sprout is evaluated using six diverse language modeling datasets, detailed in
Table I. These datasets span various fields and applications, serving as
critical benchmarks in performance evaluations for leading large language
models, including Llama [40], Claude [44], Mixtral [45], GPT [46], and Gemini
[47]. To simulate realistic user prompts for the inference server, the
construction of tasks is guided by user request patterns from the Alibaba
Platform for AI trace [48], ensuring the evaluation comprehensively represents
the workload encountered in practical scenarios.

TABLE II: Different geographical regions and their minimum and maximum annual
carbon intensity.

Region | abbr. | Operator | Annual Min/Max
---|---|---|---
Texas (US) | TX |  Electric Reliability Council of Texas (ERCOT) | 124 / 494 (gCO2/kWh)
California (US) | CA |  California Independent System Operator (CISO) | 55 / 331 (gCO2/kWh)
South Australia | SA |  Australian Energy Market Operator (AEMO) | 10 / 526 (gCO2/kWh)
Netherland | NL | TenneT | 23 / 463 (gCO2/kWh)
Great Britain | GB |  National Grid Electricity System Operator (ESO) | 24 / 282 (gCO2/kWh)

The evaluation of Sprout extends across five grid operation regions in various
countries, as described in Table II. Given the variability in carbon intensity
by region, this diversity enables a comprehensive assessment of Sprout’s
performance in differing environmental contexts. The study uses carbon
intensity data from February, June, and October of 2023, sourced from
Electricity Maps [25] at hourly intervals, to gauge Sprout’s adaptability to
fluctuating carbon intensity levels across these regions. Despite the offline
evaluation LLM not being sensitive to latency and thus not requiring proximity
to users – allowing it to be located in any global data center with the lowest
carbon footprint – for a more cautious approach, we assume it resides in the
same region as the inference server.

Competing schemes.  Sprout is evaluated alongside five distinct strategies,
detailed as follows:

Base.  This is the baseline strategy that represents a vanilla LLM inference
system, it does not explore the opportunity of generation directives discussed
in Sec. II-B.

CO2_Opt.  This represents a scheme that aggressively minimizes CO2 emissions
without considering the generation quality. Specifically, it will always use
the generation directive level that yields the lowest carbon footprint for all
prompts.

Model_Opt.  This scheme is an implementation of the idea to adjust the
underlying model parameters to achieve optimization goals from previous works
[14, 10, 13]. Unaware of the generation directives, this scheme uses inference
model variants (i.e., Llama2 7B and 13B) as optimization variables since model
variants also introduce the trade-offs between carbon and generation quality.
The scheme represents the optimal model variant selection for the user
prompts.

Sprout_Sta.  This is a static version of Sprout, applying a single, month-long
optimal generation directive configuration identified through offline
analysis, without dynamic adjustments based on real-time carbon intensity and
generation feedback. The best static configuration is determined by sweeping
the possible static configurations.

Oracle.  This is an impractical scheme based on oracle information. It assumes
the inference carbon emission on every generation directive level is known
ahead of time for all user prompts, and knows the exact generation quality
feedback for future prompts instead of relying on sampling. It does not suffer
from any profiling overheads and sampling inaccuracies.

Metrics.  The evaluation of Sprout centers on two primary metrics: the carbon
footprint associated with each inference request and the quality of the
content it generates. The carbon footprint metric accounts for the CO2
emissions associated with each inference, averaged for comparison against the
default operation represented by Base. The generation quality is measured from
the auto-evaluation LLM’s preference, normalized against Base’s performance.
For instance, if the auto-evaluator shows a preference for Sprout’s responses
48% of the time versus 52% for Base, Sprout’s normalized generation preference
score would be 92.3%.

Next, we thoroughly evaluate how Sprout can contribute to sustainable GenAI
using its directive-guided LLM inference.

##  V Evaluation

###  V-A Effectiveness of Sprout

Figure 9: Sprout archives significant carbon savings while preserving
generation quality across all geographical regions.

Sprout consistently achieves substantial carbon savings while adhering to
generation quality standards in diverse geographical regions. According to
Fig. 9, Sprout’s application of optimized generation directives can reduce
carbon emissions by up to 60%. With a preference deviation ($\xi=0.1$) set
from the baseline in Eq. 3, generation preferences across all regions remain
above the 90% mark, notably reaching over 95% in South Australia alongside a
carbon saving exceeding 40%. From an inference service provider perspective,
according to a recent survey [9], deploying OpenAI’s ChatGPT service
necessitates around 29K NVIDIA A100 GPUs, equating to an energy consumption of
564 MWh daily. In the Azure West US region of California [49], this translates
to monthly CO2 emissions of 3,266 tonnes. Adopting Sprout could result in a
monthly carbon reduction of 1,903 tonnes—equivalent to offsetting the carbon
footprint of flying 6,358 passengers from New York to London [50].

Figure 10: Sprout excels when competing against competitive strategies and is
closest to Oracle

Sprout surpasses competing methods, closely aligning with the Oracle standard.
Fig. 10 illustrates Sprout’s performance against competing strategies outlined
in Sec. IV, showcasing its proximity to the ideal Oracle in both carbon
savings and normalized generation preference across all regions. Here,
vertical lines denote the upper bound of generation preference in our
evaluation, while horizontal lines indicate the upper bound of carbon savings.
Unlike CO2_Opt, which prioritizes carbon reduction at the expense of
generation quality, Sprout maintains a balance closer to Base. While
Model_Opt, Sprout_Sta, and Sprout exhibit similar preferences, Model_Opt falls
short in carbon savings, highlighting the limitations of optimizing solely
based on inference model variants [14, 10, 13]. In contrast to its static
version Sprout_Sta, Sprout demonstrates that its dynamic approach to
generation directives yields results nearer to the Oracle benchmark,
underscoring the effectiveness of adaptive configurations.

###  V-B Mechanisms behind Sprout’s Effectiveness

Figure 11: Cumulative distribution function of per-request carbon emission
normalized to Base when the environmental carbon intensity varies.

Next, the inference carbon footprint is analyzed from the perspective of
individual user requests, as depicted in Fig. 11, which presents the empirical
cumulative distribution function (CDF) for 10K requests across three
environmental carbon intensities: 200, 300, and 400 gCO2/kWh. The x-axis
scales the CO2 emissions of each request relative to executions on the Base
system. Since we only show CO2 per request, CO2_Opt is the best among all the
schemes – 80% of requests have used less than 30% of the Base carbon emission.
When carbon intensity increases, Sprout’s CDF moves closer and closer to
CO2_Opt, indicating that Sprout’s optimizer is adapting to the regional carbon
intensity since the gain from using more concise directives gets amplified by
higher carbon intensities (Sec. II-A). Specifically, when carbon intensity is
200 gCO2/kWh, 40% of Sprout’s requests have used less than 40% of the carbon
footprint than Base; when it increases to 400 gCO2/kWh, about 75% of Sprout’s
requests have less than 40% of Sprout’s carbon footprint. Unlike CO2_Opt and
Sprout_Sta, which do not adjust based on carbon intensity and thus maintain
constant CDF curves, Sprout exhibits a dynamic adaptation, aligning it closely
with the Oracle benchmark in a per-request analysis.

Figure 12: Sprout configures the generation directive levels (represented as
pie charts) according to quality evaluator preference and carbon intensity.

Fig. 12 illustrates Sprout’s adaptive use of generation directive levels
across different scenarios, represented through pie charts. During period 0,
on average, the carbon intensity is 100 gCO2/kWh and nearly half of the
evaluations prefer the L0 directive. Consequently, Sprout allocates L0 to 91%
of the prompts in total, reflecting its high preference rate. In period 1,
with rising carbon intensity, there’s a noticeable shift toward employing more
L1 and L2 directives, aligning with environmental considerations. Period 2
presents unchanged carbon intensity but altered user preferences, leading to
an increased preference for L0 by the evaluator and a corresponding adjustment
in Sprout’s directive assignments from 73% to 85% toward L0. Finally, in
period 3, a significant change in user behavior emerges, showing a clear
preference for L1 and L2 directives. This shift, coupled with the benefits of
carbon savings at elevated carbon intensities, leads Sprout to primarily
assign L1 and L2 directives.

Figure 13: Without the offline evaluator, Sprout misses the chance to leverage
requests amenable to concise directive levels, thus forfeiting potential
benefits in carbon savings and generation preference simultaneously.

The offline quality evaluator is key to Sprout’s effectiveness as we explain
in Fig. 13. To show the necessity of the quality evaluator, we select Sprout-
friendly prompts which are prompts whose shorter responses are on average more
preferred by the auto-evaluator than their default responses, and mix them
with unfriendly prompts (shorter responses are less preferred by auto-
evaluator than default responses). Over time, we vary the proportion of these
two types of prompts, and we can observe that when the portion of friendly is
high, Sprout without the evaluator will miss out on the opportunity to save
more carbon while achieving higher evaluator preference at the same time. As
we can see around hour 22, the normalized preference is above 100%, meaning
the auto-evaluation LLM prefers Sprout’s generation over the default
generation more than 50% of the time. The offline evaluator’s low carbon
overhead is also a key reason why Sprout can save so much carbon with the
evaluator, as we discuss next.

Figure 14: (a). Sprout’s offline evaluator has negligible carbon emission
overhead. (b). Violin plot of regional carbon intensity distribution, and the
carbon intensity where Sprout invokes offline evaluation (marked as red line).

In Fig. 14 (a), we show the carbon overhead of Sprout’s offline evaluator.
Since GPT-4 is only accessible from third-party API, we use the following
numbers to estimate the offline evaluation carbon footprint. GPT-4 is
speculated to use a mixture-of-experts (MoE) architecture, and during
inference, only one expert is active. Thus, the model size is equivalent to
one expert that has 220B parameters, which can be hosted on 16 A100 GPUs. With
the measured average API accessing time of 500ms, we assume all 16 GPUs are
running at max power (250W), under no network delay and no batched processing.
Despite our conservative estimation where in reality the GPU generation time
is much shorter than 500ms (network latency, pre- and post-processing) and
multiple requests can be processed simultaneously in a batch, the overhead in
Fig. 14(a) serving 30 requests per second (RPS) [17] is still well below 1%
for all regions. The negligible carbon impact stems from (i) strategically
timing evaluations to coincide with periods of low carbon intensity as shown
in Fig. 14 (b), and (ii) configuring the request to the auto-evaluation LLM in
such a way that it generates only a minimal number of tokens for assessment,
as detailed in Sec. III-E.

###  V-C Robustness and Implications

Figure 15: Sprout remains effective in all geographical regions during
different seasons.

Finally, we assess the robustness of Sprout and its broader implications. Fig.
15 presents an evaluation of Sprout across various periods of 2023,
demonstrating its consistent efficacy across different seasons. Sprout
consistently enables the inference server to achieve over 40% carbon emission
savings while sustaining high levels of generation quality.

Figure 16: Pareto front of Sprout across geographical regions when varying the
preference coefficient. Sprout can still reduce 40% of carbon emission under
strict generation preference constraints.

Sprout offers operators the ability to balance carbon savings against quality
through the adjustable parameter $\xi$. Fig. 16 illustrates the Pareto front
demonstrating the trade-off between carbon savings and normalized generation
preference as $\xi$ is varied. Remarkably, even when tightening the generation
preference criterion to 95% (indicating the evaluator prefers Sprout’s
generation over the default 48.7% of the time), Sprout consistently secures
over 40% carbon savings across all regions.

Sprout stands as the inaugural approach to utilizing generation directives for
configuring generative LLM inference, with a particular emphasis on advancing
sustainability within GenAI by addressing carbon emissions. This strategy
opens up extensive possibilities beyond its current focus. For instance, using
generation directives can significantly enhance LLM inference throughput,
thereby reducing the number of GPU servers needed to achieve specific rates of
requests per second (RPS). This efficiency translates into reduced capital
expenses for building LLM inference infrastructure and lowers the embodied
carbon associated with manufacturing the GPU servers.

##  VI Related Work

Sustainable computing.  With the recent rise of interest and urgency toward
reducing the carbon footprint of information technology, Totally Green [51]
and ACT [6] introduced carbon modeling frameworks from system and architecture
perspective. Based on the carbon modeling, Sustainable AI [8], Sustainable HPC
[24], and Chien et al. [52] have explored various carbon trade-offs in
designing and operating large-scale computer systems. Various works have
analyzed the recent trend and the future of AI development’s impact on carbon
emission [53, 54, 55, 56, 11]. Sprout is motivated by these works and takes
the effort a step further to LLM inference application. While systems like
Ecovisor [57], Dodge et al.[12], Clover[10], and Carbon Explorer [23] have
been designed to adapt to varying carbon intensities, they have not been
specifically optimized for LLM inference workloads. Luccioni et al [58] and
Chien et al. [7] have characterized the carbon profile and challenges from LLM
inference, while Sprout has designed an end-to-end framework that leverages
generation directives to address these climate challenges.

Large language model inference.  Generative LLM inference is distinct from
conventional ML inference due to its substantial parameter size, the extensive
use of self-attention operations, and the autoregressive generation process.
As such, the optimization strategies commonly applied in general ML inference
contexts [14, 10, 13, 59, 60, 61, 62] are not ideally suited for generative
LLMs. This gap has spurred the creation of dedicated LLM inference serving
frameworks [63, 64, 65, 66, 67], with notable examples like Orca [18], which
introduces iteration-level batching, and vLLM [17], known for its efficient KV
cache management via paging. Sprout is engineered for compatibility with these
specialized frameworks, with its effectiveness stemming not from the
particulars of the inference server’s setup but from the strategic use of
token generation directives.

The surge in LLM inference popularity has prompted a diverse range of research
on performance and memory optimization, exploring strategies like sparsity and
pruning [68, 69], speculative decoding [70, 71], GPU kernel tiling and fusion
[37, 72], disk offloading [73, 74], and mixture-of-experts approaches [75,
76]. These advancements are crucial for facilitating the deployment of larger
LLMs to a broader audience. However, the environmental implications of these
technologies are equally important. While LLMCarbon [77] offers carbon
footprint predictions to help researchers gauge the environmental impact of
LLMs prior to training, Sprout stands out as the first work to tackle the
carbon footprint challenge of generative LLM inference using a novel
generation directive mechanism.

##  VII Conclusion

This paper introduced Sprout, an innovative framework designed to enhance the
sustainability of generative AI by creating a carbon-aware inference service
for generative language models. Utilizing the novel concept of generation
directives, Sprout significantly optimizes the carbon footprint associated
with LLM inference. Our evaluation, conducted using a Llama2 inference server
and a GPT-4 quality evaluator, demonstrates that Sprout can reduce the carbon
footprint of inference activities by over 40% in various global regions.
Through the development of Sprout, we seek to pave the way for a greener
future in generative AI, stimulating further research into minimizing the
environmental impacts in the era of rapid AI advancements.

## Acknowledgments

This material is based upon work supported by the Assistant Secretary of
Defense for Research and Engineering under Air Force Contract No.
FA8702-15-D-0001, and United States Air Force Research Laboratory Cooperative
Agreement Number FA8750-19-2-1000. Any opinions, findings, conclusions, or
recommendations expressed in this material are those of the author(s) and do
not necessarily reflect the views of the Assistant Secretary of Defense for
Research and Engineering, or the United States Air Force. The U.S. Government
is authorized to reproduce and distribute reprints for Government purposes
notwithstanding any copyright notation herein.

## References

  * [1] J. Jumper, R. Evans, A. Pritzel, T. Green, M. Figurnov, O. Ronneberger, K. Tunyasuvunakool, R. Bates, A. Žídek, A. Potapenko _et al._ , “Highly accurate protein structure prediction with alphafold,” _Nature_ , vol. 596, no. 7873, pp. 583–589, 2021.
  * [2] N. Pierce and S. Goutos, “Why law firms must responsibly embrace generative ai,” _Available at SSRN 4477704_ , 2023.
  * [3] B. Chen, Z. Wu, and R. Zhao, “From fiction to fact: the growing role of generative ai in business and finance,” _Journal of Chinese Economic and Business Studies_ , vol. 21, no. 4, pp. 471–496, 2023.
  * [4] Fortune. (2024) Sam altman seeks trillions of dollars to reshape business of chips and ai. [Online]. Available: https://fortune.com/2024/02/12/sam-altman-7-trillion-ai-chips-grind-for-future-substack/
  * [5] IEA. (2024) Electricity 2024, analysis and forecast to 2026. [Online]. Available: https://www.iea.org/reports/electricity-2024
  * [6] U. Gupta, M. Elgamal, G. Hills, G.-Y. Wei, H.-H. S. Lee, D. Brooks, and C.-J. Wu, “Act: Designing sustainable computer systems with an architectural carbon modeling tool,” in _Proceedings of the 49th Annual International Symposium on Computer Architecture_ , 2022, pp. 784–799.
  * [7] A. A. Chien, L. Lin, H. Nguyen, V. Rao, T. Sharma, and R. Wijayawardana, “Reducing the carbon impact of generative ai inference (today and in 2035),” in _Proceedings of the 2nd Workshop on Sustainable Computer Systems_ , 2023, pp. 1–7.
  * [8] C.-J. Wu, R. Raghavendra, U. Gupta, B. Acun, N. Ardalani, K. Maeng, G. Chang, F. Aga, J. Huang, C. Bai _et al._ , “Sustainable ai: Environmental implications, challenges and opportunities,” _Proceedings of Machine Learning and Systems_ , vol. 4, pp. 795–813, 2022.
  * [9] A. de Vries, “The growing energy footprint of artificial intelligence,” _Joule_ , vol. 7, no. 10, pp. 2191–2194, 2023.
  * [10] B. Li, S. Samsi, V. Gadepally, and D. Tiwari, “Clover: Toward sustainable ai with carbon-aware machine learning inference service,” in _Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis_ , 2023, pp. 1–15.
  * [11] T. Anderson, A. Belay, M. Chowdhury, A. Cidon, and I. Zhang, “Treehouse: A case for carbon-aware datacenter software,” _ACM SIGENERGY Energy Informatics Review_ , vol. 3, no. 3, pp. 64–70, 2023.
  * [12] J. Dodge, T. Prewitt, R. Tachet des Combes, E. Odmark, R. Schwartz, E. Strubell, A. S. Luccioni, N. A. Smith, N. DeCario, and W. Buchanan, “Measuring the carbon intensity of ai in cloud instances,” in _Proceedings of the 2022 ACM conference on fairness, accountability, and transparency_ , 2022, pp. 1877–1894.
  * [13] C. Wan, M. Santriaji, E. Rogers, H. Hoffmann, M. Maire, and S. Lu, “$\\{$ALERT$\\}$: Accurate learning for energy and timeliness,” in _2020 USENIX annual technical conference (USENIX ATC 20)_ , 2020, pp. 353–369.
  * [14] F. Romero, Q. Li, N. J. Yadwadkar, and C. Kozyrakis, “$\\{$INFaaS$\\}$: Automated model-less inference serving,” in _2021 USENIX Annual Technical Conference (USENIX ATC 21)_ , 2021, pp. 397–411.
  * [15] U. Gupta, Y. G. Kim, S. Lee, J. Tse, H.-H. S. Lee, G.-Y. Wei, D. Brooks, and C.-J. Wu, “Chasing carbon: The elusive environmental footprint of computing,” in _2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA)_. IEEE, 2021, pp. 854–867.
  * [16] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [17] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” in _Proceedings of the 29th Symposium on Operating Systems Principles_ , 2023, pp. 611–626.
  * [18] G.-I. Yu, J. S. Jeong, G.-W. Kim, S. Kim, and B.-G. Chun, “Orca: A distributed serving system for $\\{$Transformer-Based$\\}$ generative models,” in _16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22)_ , 2022, pp. 521–538.
  * [19] I. Gim, G. Chen, S.-s. Lee, N. Sarda, A. Khandelwal, and L. Zhong, “Prompt cache: Modular attention reuse for low-latency inference,” _arXiv preprint arXiv:2311.04934_ , 2023.
  * [20] D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, “Measuring massive multitask language understanding,” _arXiv preprint arXiv:2009.03300_ , 2020.
  * [21] P. Lu, S. Mishra, T. Xia, L. Qiu, K.-W. Chang, S.-C. Zhu, O. Tafjord, P. Clark, and A. Kalyan, “Learn to explain: Multimodal reasoning via thought chains for science question answering,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 2507–2521, 2022.
  * [22] M. Joshi, E. Choi, D. S. Weld, and L. Zettlemoyer, “Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension,” _arXiv preprint arXiv:1705.03551_ , 2017.
  * [23] B. Acun, B. Lee, F. Kazhamiaka, K. Maeng, U. Gupta, M. Chakkaravarthy, D. Brooks, and C.-J. Wu, “Carbon explorer: A holistic framework for designing carbon aware datacenters,” in _Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2_ , 2023, pp. 118–132.
  * [24] B. Li, R. Basu Roy, D. Wang, S. Samsi, V. Gadepally, and D. Tiwari, “Toward sustainable hpc: Carbon footprint estimation and environmental implications of hpc systems,” in _Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis_ , 2023, pp. 1–15.
  * [25] E. Maps. (2024) Electricity Maps Live 24/7. [Online]. Available: https://app.electricitymaps.com/map
  * [26] Y. Dubois, C. X. Li, R. Taori, T. Zhang, I. Gulrajani, J. Ba, C. Guestrin, P. S. Liang, and T. B. Hashimoto, “Alpacafarm: A simulation framework for methods that learn from human feedback,” _Advances in Neural Information Processing Systems_ , vol. 36, 2024.
  * [27] Y. Liu, D. Iter, Y. Xu, S. Wang, R. Xu, and C. Zhu, “G-eval: NLG evaluation using gpt-4 with better human alignment,” in _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , H. Bouamor, J. Pino, and K. Bali, Eds. Singapore: Association for Computational Linguistics, Dec. 2023, pp. 2511–2522. [Online]. Available: https://aclanthology.org/2023.emnlp-main.153
  * [28] Y. Bai, J. Ying, Y. Cao, X. Lv, Y. He, X. Wang, J. Yu, K. Zeng, Y. Xiao, H. Lyu _et al._ , “Benchmarking foundation models with language-model-as-an-examiner,” _Advances in Neural Information Processing Systems_ , vol. 36, 2024.
  * [29] MistralAI. (2024) Employ another llm for evaluation. [Online]. Available: https://docs.mistral.ai/guides/prompting-capabilities/#employ-another-llm-for-evaluation
  * [30] Q. Huangfu and J. J. Hall, “Parallelizing the dual revised simplex method,” _Mathematical Programming Computation_ , vol. 10, no. 1, pp. 119–142, 2018.
  * [31] Wikipedia. (2024) Gpt4. [Online]. Available: https://en.wikipedia.org/wiki/GPT-4
  * [32] J. Charan and T. Biswas, “How to calculate sample size for different study designs in medical research?” _Indian journal of psychological medicine_ , vol. 35, no. 2, pp. 121–126, 2013.
  * [33] OpenAI. (2024) Chat markup language. [Online]. Available: https://github.com/MicrosoftDocs/azure-docs/blob/main/articles/ai-services/openai/includes/chat-markup-language.md
  * [34] FacebookResearch. (2024) Inference code for llama models. [Online]. Available: https://github.com/facebookresearch/llama
  * [35] Anthropic. (2024) System prompts. [Online]. Available: https://docs.anthropic.com/claude/docs/how-to-use-system-prompts
  * [36] Huggingface. (2024) Mistral-7b-instruct-v0.1. [Online]. Available: https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1
  * [37] T. Dao, “Flashattention-2: Faster attention with better parallelism and work partitioning,” _arXiv preprint arXiv:2307.08691_ , 2023.
  * [38] L. F. W. Anthony, B. Kanding, and R. Selvan, “Carbontracker: Tracking and predicting the carbon footprint of training deep learning models,” _arXiv preprint arXiv:2007.03051_ , 2020.
  * [39] X. Li, T. Zhang, Y. Dubois, R. Taori, I. Gulrajani, C. Guestrin, P. Liang, and T. B. Hashimoto, “Alpacaeval: An automatic evaluator of instruction-following models,” https://github.com/tatsu-lab/alpaca_eval, 2024.
  * [40] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale _et al._ , “Llama 2: Open foundation and fine-tuned chat models,” _arXiv preprint arXiv:2307.09288_ , 2023.
  * [41] R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto, “Alpaca: A strong, replicable instruction-following model,” _Stanford Center for Research on Foundation Models. https://crfm. stanford. edu/2023/03/13/alpaca. html_ , vol. 3, no. 6, p. 7, 2023.
  * [42] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano _et al._ , “Training verifiers to solve math word problems,” _arXiv preprint arXiv:2110.14168_ , 2021.
  * [43] T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee _et al._ , “Natural questions: a benchmark for question answering research,” _Transactions of the Association for Computational Linguistics_ , vol. 7, pp. 453–466, 2019.
  * [44] Anthropic. (2024) Introducing the next generation of claude. [Online]. Available: https://www.anthropic.com/news/claude-3-family
  * [45] A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. l. Casas, E. B. Hanna, F. Bressand _et al._ , “Mixtral of experts,” _arXiv preprint arXiv:2401.04088_ , 2024.
  * [46] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat _et al._ , “Gpt-4 technical report,” _arXiv preprint arXiv:2303.08774_ , 2023.
  * [47] G. Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth _et al._ , “Gemini: a family of highly capable multimodal models,” _arXiv preprint arXiv:2312.11805_ , 2023.
  * [48] Q. Weng, W. Xiao, Y. Yu, W. Wang, C. Wang, J. He, Y. Li, L. Zhang, W. Lin, and Y. Ding, “$\\{$MLaaS$\\}$ in the wild: Workload analysis and scheduling in $\\{$Large-Scale$\\}$ heterogeneous $\\{$GPU$\\}$ clusters,” in _19th USENIX Symposium on Networked Systems Design and Implementation (NSDI 22)_ , 2022, pp. 945–960.
  * [49] Microsoft. (2024) Azure global infrastructure experience. [Online]. Available: https://datacenters.microsoft.com/globe/explore?info=region_westus
  * [50] ICAO. (2024) International civil aviation organization carbon emissions calculator. [Online]. Available: https://www.icao.int/environmental-protection/Carbonoffset
  * [51] J. Chang, J. Meza, P. Ranganathan, A. Shah, R. Shih, and C. Bash, “Totally green: evaluating and designing servers for lifecycle environmental impact,” _ACM SIGPLAN Notices_ , vol. 47, no. 4, pp. 25–36, 2012.
  * [52] M. Dietrich and A. Chien, “Navigating dennard, carbon and moore: Scenarios for the future of nsf advanced computational infrastructure,” in _Practice and Experience in Advanced Research Computing_ , 2022, pp. 1–6.
  * [53] D. Patterson, J. Gonzalez, Q. Le, C. Liang, L.-M. Munguia, D. Rothchild, D. So, M. Texier, and J. Dean, “Carbon emissions and large neural network training,” _arXiv preprint arXiv:2104.10350_ , 2021.
  * [54] D. Patterson, J. Gonzalez, U. Hölzle, Q. Le, C. Liang, L.-M. Munguia, D. Rothchild, D. R. So, M. Texier, and J. Dean, “The carbon footprint of machine learning training will plateau, then shrink,” _Computer_ , vol. 55, no. 7, pp. 18–28, 2022.
  * [55] R. Schwartz, J. Dodge, N. A. Smith, and O. Etzioni, “Green ai,” _Communications of the ACM_ , vol. 63, no. 12, pp. 54–63, 2020.
  * [56] E. Strubell, A. Ganesh, and A. McCallum, “Energy and policy considerations for deep learning in nlp,” in _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , 2019, pp. 3645–3650.
  * [57] A. Souza, N. Bashir, J. Murillo, W. Hanafy, Q. Liang, D. Irwin, and P. Shenoy, “Ecovisor: A virtual energy system for carbon-efficient applications,” in _Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2_ , 2023, pp. 252–265.
  * [58] A. S. Luccioni, S. Viguier, and A.-L. Ligozat, “Estimating the carbon footprint of bloom, a 176b parameter language model,” _Journal of Machine Learning Research_ , vol. 24, no. 253, pp. 1–15, 2023.
  * [59] Y. Wang, K. Chen, H. Tan, and K. Guo, “Tabi: An efficient multi-level inference system for large language models,” in _Proceedings of the Eighteenth European Conference on Computer Systems_ , 2023, pp. 233–248.
  * [60] S. Choi, S. Lee, Y. Kim, J. Park, Y. Kwon, and J. Huh, “Serving heterogeneous machine learning models on $\\{$Multi-GPU$\\}$ servers with $\\{$Spatio-Temporal$\\}$ sharing,” in _2022 USENIX Annual Technical Conference (USENIX ATC 22)_ , 2022, pp. 199–216.
  * [61] B. Li, S. Samsi, V. Gadepally, and D. Tiwari, “Kairos: Building cost-efficient machine learning inference systems with heterogeneous cloud resources,” in _Proceedings of the 32nd International Symposium on High-Performance Parallel and Distributed Computing_ , 2023, pp. 3–16.
  * [62] D. Crankshaw, X. Wang, G. Zhou, M. J. Franklin, J. E. Gonzalez, and I. Stoica, “Clipper: A $\\{$Low-Latency$\\}$ online prediction serving system,” in _14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17)_ , 2017, pp. 613–627.
  * [63] W. Cui, Z. Han, L. Ouyang, Y. Wang, N. Zheng, L. Ma, Y. Yang, F. Yang, J. Xue, L. Qiu _et al._ , “Optimizing dynamic neural networks with brainstorm,” in _17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23)_ , 2023, pp. 797–815.
  * [64] R. Y. Aminabadi, S. Rajbhandari, A. A. Awan, C. Li, D. Li, E. Zheng, O. Ruwase, S. Smith, M. Zhang, J. Rasley _et al._ , “Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale,” in _SC22: International Conference for High Performance Computing, Networking, Storage and Analysis_. IEEE, 2022, pp. 1–15.
  * [65] Z. Zhou, X. Wei, J. Zhang, and G. Sun, “$\\{$PetS$\\}$: A unified framework for $\\{$Parameter-Efficient$\\}$ transformers serving,” in _2022 USENIX Annual Technical Conference (USENIX ATC 22)_ , 2022, pp. 489–504.
  * [66] X. Miao, C. Shi, J. Duan, X. Xi, D. Lin, B. Cui, and Z. Jia, “Spotserve: Serving generative large language models on preemptible instances,” _arXiv preprint arXiv:2311.15566_ , 2023.
  * [67] Y. Sheng, S. Cao, D. Li, B. Zhu, Z. Li, D. Zhuo, J. E. Gonzalez, and I. Stoica, “Fairness in serving large language models,” _arXiv preprint arXiv:2401.00588_ , 2023.
  * [68] Z. Liu, J. Wang, T. Dao, T. Zhou, B. Yuan, Z. Song, A. Shrivastava, C. Zhang, Y. Tian, C. Re _et al._ , “Deja vu: Contextual sparsity for efficient llms at inference time,” in _International Conference on Machine Learning_. PMLR, 2023, pp. 22 137–22 176.
  * [69] E. Frantar and D. Alistarh, “Sparsegpt: Massive language models can be accurately pruned in one-shot,” in _International Conference on Machine Learning_. PMLR, 2023, pp. 10 323–10 337.
  * [70] Y. Leviathan, M. Kalman, and Y. Matias, “Fast inference from transformers via speculative decoding,” in _International Conference on Machine Learning_. PMLR, 2023, pp. 19 274–19 286.
  * [71] C. Chen, S. Borgeaud, G. Irving, J.-B. Lespiau, L. Sifre, and J. Jumper, “Accelerating large language model decoding with speculative sampling,” _arXiv preprint arXiv:2302.01318_ , 2023.
  * [72] N. Zheng, H. Jiang, Q. Zhang, Z. Han, L. Ma, Y. Yang, F. Yang, C. Zhang, L. Qiu, M. Yang _et al._ , “Pit: Optimization of dynamic sparse deep learning models via permutation invariant transformation,” in _Proceedings of the 29th Symposium on Operating Systems Principles_ , 2023, pp. 331–347.
  * [73] Y. Sheng, L. Zheng, B. Yuan, Z. Li, M. Ryabinin, B. Chen, P. Liang, C. Ré, I. Stoica, and C. Zhang, “Flexgen: High-throughput generative inference of large language models with a single gpu,” in _International Conference on Machine Learning_. PMLR, 2023, pp. 31 094–31 116.
  * [74] S. Rajbhandari, O. Ruwase, J. Rasley, S. Smith, and Y. He, “Zero-infinity: Breaking the gpu memory wall for extreme scale deep learning,” in _Proceedings of the international conference for high performance computing, networking, storage and analysis_ , 2021, pp. 1–14.
  * [75] J. Li, Y. Jiang, Y. Zhu, C. Wang, and H. Xu, “Accelerating distributed $\\{$MoE$\\}$ training and inference with lina,” in _2023 USENIX Annual Technical Conference (USENIX ATC 23)_ , 2023, pp. 945–959.
  * [76] L. Xue, Y. Fu, Z. Lu, L. Mai, and M. Marina, “Moe-infinity: Activation-aware expert offloading for efficient moe serving,” _arXiv preprint arXiv:2401.14361_ , 2024.
  * [77] A. Faiz, S. Kaneda, R. Wang, R. Osi, P. Sharma, F. Chen, and L. Jiang, “Llmcarbon: Modeling the end-to-end carbon footprint of large language models,” _arXiv preprint arXiv:2309.14393_ , 2023.