#  APIServe: Efficient API Support for Large-Language Model Inferencing

Reyna Abhyankar  Zijian He  Vikranth Srivatsa  Hao Zhang  Yiying Zhang

###### Abstract

Large language models are increasingly integrated with external tools and APIs
like ChatGPT plugins to extend their capability beyond language-centric tasks.
However, today’s LLM inference systems are designed for standalone LLMs. They
treat API calls as new requests, causing unnecessary recomputation of already
computed contexts, which accounts for 37-40% of total model forwarding time.
This paper presents APIServe, the first LLM inference framework targeting API-
augmented LLMs. APIServe minimizes the GPU resource waste caused by API calls
and dedicates saved memory for serving more requests. APIServe improves the
overall serving throughput by 1.6$\times$ and completes 2$\times$ more
requests per second compared to the state-of-the-art LLM inference systems.

Machine Learning Systems, LLM Inference, API Augmented LLM

rm

University of California, San Diego

††footnotetext: Equal contribution

##  1 Introduction

Large language models (LLMs) (OpenAI et al., 2023; Touvron et al., 2023) have
shown immense potential in various applications, such as natural language
understanding and content generation. Nonetheless, LLMs alone can only
generate text. To extend LLMs’ capabilities to handle more diverse and open-
ended tasks, a recent trend is to augment LLMs with external tools (Parisi et
al., 2022; Schick et al., 2023; Hao et al., 2023; Mialon et al., 2023) such as
arithmetic calculation (Hao et al., 2023) and virtual-environment interaction
(Shridhar et al., 2020), or interact with humans and the other environments in
real-time, such as chatgpt-plugins. (OpenAI, 2023). More generally, we can
view the use of tools and the interaction with humans or the environment all
as LLMs calling APIs.

APIs have unique properties that bring new problems to LLM inference systems.
First, API calls intercept or preempt regular LLM decoding, i.e., when an API
call is made, the normal decoding phase is paused and can only resume when the
API returns. Second, an API’s execution time varies drastically across API
types and requests. For example, a calculator API finishes in less than 1 ms,
yet it may take humans more than 1 minute to digest and type a subsequent chat
prompt. The uncertain API execution time complicates request scheduling.
Third, unlike regular requests, the context (i.e., KV caches) cannot be used
for a paused request during API calls but will be needed upon API return. It
is unclear how to handle temporarily unused context. Adding to the complexity
is the high variation in context length and API execution time across
requests.

Unfortunately, existing LLM inference systems (Kwon et al., 2023; Yu et al.,
2022; Li et al., 2023; Aminabadi et al., 2022; NVIDIA, 2023; Agrawal et al.,
2023) fall short in serving LLMs with API calls. They typically interpret an
API call within a request as a termination signal – they conclude the ongoing
request at the point of an API call, discard its context, relegate the task to
the API provider, and subsequently re-initiate a new request upon receiving
the API’s output. Consequently, when a request frequently invokes API calls,
for each API call, the initial prompt and all tokens generated before the API
call are formed as a new request submitted to the inference system, causing
substantial recomputation of keys and values for all previously processed
tokens. This recomputation wastes GPU resources, which could otherwise be used
to serve new requests.

In contrast to the aforementioned API handling approach (which we refer to as
Discard), an alternative approach is to preserve a request’s context during
the API call and resume the request when the API call returns. This approach,
which we call Preserve, avoids recomputation but occupies GPU memory for the
entire API execution time. This GPU memory could accommodate other requests’
context and increase overall throughput (Kwon et al., 2023).

To avoid recomputation and memory preservation, another possible approach to
handle APIs is to swap contexts to CPU memory when an API is called (we call
it Swap). Although Swap avoids recomputation and GPU memory wastage, with
limited GPU-CPU link bandwidth, foreground tasks (normal forwarding) could be
bottlenecked by waiting for swapping to finish. The GPU memory could be used
for processing more requests.

How should an LLM inference system efficiently support API calls? To answer
this question, we first study the computational patterns of LLM augmented with
six typical types of APIs: arithmetic, question-and-answer, virtual
environment, chatbot, image generation, and text-to-speech transformation. We
then evaluate how well the state-of-the-art LLM inference system, vLLM (Kwon
et al., 2023) (which performs Discard), and the two possible extensions
(Preserve and Swap) perform on these APIs.

Based on our findings, we propose APIServe, the first LLM inference framework
targeting API-augmented LLMs. The core idea of APIServe, which we call min-
waste preemption, is to minimize GPU memory waste caused by API calls so that
the same amount of GPU resources can be used to serve more requests. To this
end, APIServe incorporates three contributions. First, we deduce three waste-
calculation equations to quantify the GPU memory waste of Discard, Preserve,
and Swap.

Second, we improve individual preemption techniques to reduce or eliminate
their memory waste. For Swap, we propose swap pipelining, which overlaps
swapping and foreground computation in a model-layer-by-layer manner.
Additionally, we split the swapping of a sequence into multiple iterations
††An iteration is a single model forwarding pass. so that each iteration’s
swap needs stays within what the GPU-CPU link can sustain (i.e., a swap
budget). As a result, APIServe completely eliminates GPU memory waste of Swap.
For Discard, we observe that certain GPU processing capacity is unused by
running requests at each iteration because of the imbalanced
memory/computation requirements of decoding. We utilize this unused capacity
for recomputation. We split requests’ contexts into chunks, each small enough
to fit within the unused capacity and recomputed in one iteration.

Finally, APIServe dynamically chooses the preemption and resuming strategies
within and across requests to minimize overall GPU memory waste while ensuring
fairness. For preemption scheduling, we sort API-preempted requests by their
potential memory waste and assign the swap budget to requests with the highest
potential waste. For the remaining requests, we discard or preserve their
context based on the waste comparison. For resuming scheduling, we assign the
swap budget to swapped-out requests with the earliest original arrival times
(FCFS). We also schedule discarded requests, preserved requests, and non-API
waiting requests according to FCFS, adding requests until the GPU’s processing
capacity is reached.

We implement APIServe on top of vLLM (Kwon et al., 2023), a state-of-the-art
LLM inference system. We evaluate APIServe, vLLM (Discard), Preserve, and Swap
on A100 GPUs using two LLMs (GPT-J-6B (Wang & Komatsuzaki, 2021) and
Vicuña-13B (Chiang et al., 2023)) and the six API types we study. Overall,
APIServe sustains 1.6$\times$ higher serving load than vLLM while maintaining
similar latency per token generation. APIServe also achieves over 2$\times$
more completed requests per second. We will open source APIServe soon.

##  2 API-Augmented LLMs

This section first discusses popular API-augmented LLM frameworks and then
presents properties of typical APIs from our empirical study.

###  2.1 API-Augemented LLM Frameworks

To extend LLMs’ ability to undertake more types of tasks, various approaches
have been proposed to augment an LLM with tool, or API, uses. Below, we
briefly describe typical API uses in two broad categories.

The first type of APIs are non-LLM tools called during the decoding phase of
an LLM, usually aimed to extend the type of tasks the LLM can handle. These
APIs range from simple tools like calculator (Wolfram, 2023) and calendar to
real-world interactions like information retrieval (Baeza-Yates et al., 1999)
and restaurant reservation (OpenTable, 2023). Additionally, an API can invoke
another machine-learning model such as translation (Costa-jussà et al., 2022)
and question-answering (QA) (Izacard et al., 2022). An LLM is usually fine-
tuned to generate appropriate API calls automatically during the decoding
phase, with frameworks like TaLM(Parisi et al., 2022), ToolkgenGPT(Hao et al.,
2023), and Toolformer(Schick et al., 2023).

The second type augments a single triggering of an LLM by making a sequence of
calls to the same LLM or to other tools. For example, to continuously interact
with a human, a chatbot can take rounds of back-and-forth chats by calling an
LLM at each step and maintaining a chat history (Greyling, 2023). Another
popular use case is decomposing a complex task into multiple steps. For
example, Chain-of-Thoughts (Wei et al., 2022; Zhang et al., 2022) breaks one
request into a chain of “thought” steps, each being a new prompt to an LLM
that continues the history of thoughts. Similarly, ReAct (Yao et al., 2022)
uses a chain of reasoning and action to accomplish complex tasks. To assist
users in expressing such augmented LLM usages, LangChain (Chase, 2022),
Gorilla (Patil et al., 2023), and SGLang (Zheng et al., 2023b) provide
programming models for users to trigger LLMs or tools in a task flow.

###  2.2 API Calls and Their Properties

To understand LLM APIs and design inference systems for them, we first examine
a set of representative APIs, including arithmetic (math), question-and-answer
(QA), virtual environment (VE), chatbot, image generation, and text-to-speech
(TTS), to answer three key questions: 1) how long does an API call take? 2)
how many API calls are generated for a request? and 3) what is the context
length when an API call is made? Table 1 summarizes our results with averages
and variations. We leave other metrics like API returned token length,
detailed CDF distributions, and our evaluation methodology to the Appendix.

Type | Exec Time (sec) | Num API Calls | Context Len
---|---|---|---
Math | (9e-5, 6e-5) | (3.75, 1.3) | (1422, 738)
QA | (0.69, 0.17) | (2.52, 1.73) | (1846, 428)
VE | (0.09, 0.014) | (28.18, 15.2) | (2185, 115)
Chatbot | (28.6,15.6)* | (4.45,1.96) | (753, 703)
Image |  (20.03, 7.8)$\dagger$ | (6.91,3.93)* | (1247,792)
TTS |  (17.24,7.6)$\dagger$ | (6.91,3.93)* | (1251,792)

Table 1: API Properties Each cell shows (mean, variance) of API calls of the
type. * estimated behavior. $\dagger$ partially estimated.

Arithmetic (Math). To solve complex math problems, an LLM can be augmented to
deconstruct the problem and call a calculator API step-by-step. We evaluate
this use case with the GSM8K-XL (Hao et al., 2023) dataset, which contains
8.5K high-quality grade school math problems. As expected, the calculator
API’s execution times are short, an average of 0.2 ms. It has a fairly large
context length when APIs are called since we must prompt the LLM with
demonstrations of solving problems step-by-step.

Knowledge-based question and answer (QA). To allow pre-trained LLMs to access
wider sets of knowledge, one can augment the LLM to call knowledge-based QA
APIs that retrieve information from a rich dataset. We use the Multihop QA
Wikipedia (Yang et al., 2018) dataset to evaluate this API use. These APIs,
while having longer context lengths for questions, provide relatively quick
yet variable execution times due to the non-deterministic network
communication latency with Wikipedia.

Virtual Environment (VE). An LLM can be augmented to interact with virtual
environments and accomplish complex tasks by performing gradual steps (Gu et
al., 2022; Huang et al., 2022; Hu et al., 2023). To evaluate VE APIs, we use
the ALFWorld dataset (Shridhar et al., 2020), an interactive environment that
aligns text descriptions and commands with a physically embodied robotic
simulation. This VE API has a short and stable execution time due to a locally
executed and embodied text-based environment. The context length is long due
to the instructions needed to prompt the LLM to understand VE action
sequences.

Chatbot. A popular task for LLMs is a chatbot, which interacts with human with
a sequence of chats. Here, we view human responses as an API, as the time that
humans use to respond essentially “leaves,” or preempts, regular LLM
processing, but the chat history should be kept as a context for following
chats. To evaluate chatbots, we use the ShareGPT dataset (Zheng et al.,
2023a), which contains crowd-sourced real-user ChatGPT conversations. We
estimate human response time (i.e., the API execution time) as the summation
of human scanning time (Brysbaert, 2019) of the previous chat response and
typing time (Ma et al., 2015) of the next prompt. The context lengths vary
greatly due to the variety of human chat sequences. Accordingly, the Chatbot
API execution time also has a high variance.

Image Generation. An LLM can be augmented to generate a sequence of images
when a human gradually refines features through multiple prompts (e.g., adding
details to a face depiction). This involves two APIs: one for image generation
and another for receiving user prompts. The latter API type is similar to
Chatbot’s. To understand this use case, we use ChatGPT to create a dataset by
generating a series of image-generation prompts, each triggering a call to the
Stable Diffusion model (Rombach et al., 2021). The execution time of the image
generation API has high variation due to variations in executing a diffusion
model and the variation in human response time. The average context length is
shorter than the Chat API due to smaller input prompts to the diffusion API.

Text-To-Speech (TTS). As demonstrated by the ChatGPT Whisper API support
(Brockman et al., 2023), TTS can be integrated with an LLM to communicate with
a human. Similar to our image-generation dataset, we use ChatGPT to generate a
series of prompts, each triggering an API call to the Bark TTS model (AI,
2023). Similar to image generation, we estimate the human response time but
measure actual TTS execution time. TTS’s behavior also has high variation in
API execution time due to variations in model execution time and user response
time.

Summary and Insights. Overall, we find API execution time to be highly
dependent on API types and exhibit a clear difference between short-running
(Math, QA, VE) and long-running (Chatbot, Image, TTS) ones. The short-running
APIs are fully automated with no human interaction. Most of them also exhibit
small variations (except for QA, which interacts with the network). APIs that
involve human interaction and/or calling another large model are long-running
with high variations. The short/long execution times and their variation imply
that no single API preemption strategy is universally applicable, but an
inference system could utilize API type as a hint for execution time. Context
lengths for all the APIs are large, implying potential high GPU memory wasted.
Context lengths also see high variation, complicating the handling of them at
the run time.

Figure 1: APIServe and Alternative Approaches. Vertical: GPU resources
occupied or wasted by API and other requests. Horizontal: timeline divided by
iterations (dotted vertical lines). Hatch parts represent memory waste.
APIServe adaptively combines preserve, our optimized discard, and our
optimized swap. All green parts represent APIServe’s techniques.

##  3 Existing LLM Inference Systems

This section discusses related works in existing LLM inference systems and why
simple extensions to them are insufficient for API handling, as shown in
Figure 1.

###  3.1 LLM Inference Systems

Recent years have seen the rise of inference systems that are designed
specifically for LLMs. Unlike APIServe, none of the existing inference systems
are designed for handling APIs, and all suffer from unoptimized performance,
as we will show in Section 3.2.

Some of them aim to improve overall inference throughput with better
scheduling strategies. For example, Orca (Yu et al., 2022) proposes iteration-
level scheduling where, at the end of each model forward pass, Orca’s
scheduler is invoked to form a new batch for the next forward pass. Such
iteration-level scheduling strategies improve GPU utilization, resulting in
higher inference throughput. APIServe also adopts iteration-level scheduling
but with a unique and new scheduling strategy — at each iteration, APIServe
makes a decision for requests calling and returning from APIs in a manner that
minimizes GPU memory wastage based on their API call properties. Apart from
FCFS which Orca and others use, alternative request scheduling policies such
as multi-level feedback queue in FastServe (Wu et al., 2023) have been
proposed. We leave the comparison of these scheduling policies to future work.

Another optimization target is GPU memory usage. vLLM (Kwon et al., 2023)
proposes the concept of paged attention, where KV caches are treated as
virtual memory that can map to non-contiguous physical GPU memory. Because of
the flexibility in virtual memory, vLLM achieves better utilization of GPU
memory, thereby improving overall inference performance. APIServe also
utilizes paged attention to efficiently use GPU physical memory. Unlike vLLM,
which is agnostic to APIs, APIServe adaptively preserves, discards, or swaps
KV cache of API-preempted requests to minimize memory waste. Other memory-
efficient techniques, such as prefix sharing in SGLang (Zheng et al., 2023b),
are orthogonal and can potentially be added to APIServe.

Yet another set of LLM inference systems focuses on the imbalanced computation
needs between the prefill and decoding stages. Sarathi (Agrawal et al., 2023)
is a system that proposes the technique of chunked prefill, which splits
prompt tokens into chunks, each merged with other decoding requests to form
one iteration’s batch. APIServe’s chunking splits sequences for recomputation
and swapping to fully utilize GPU and GPU-CPU-link resources while minimizing
memory waste. Other proposals on prefill-decoding optimization like DistServe
(Zhong et al., 2024) are orthogonal and can potentially be added to APIServe.

###  3.2 Limitations of Inference Systems and Extensions

We now qualitatively and quantitatively analyze the limitations of existing
LLM inference systems and naive extensions to them for API support. We
quantify the impact of these approaches on serving speed with our proposed GPU
memory waste calculation.

Discard-based approaches. Today’s LLM inference systems are not designed for
APIs. To use them for APIs, one would need to treat the API call preemption as
the end of the request and re-initiate a request when the API returns. With
today’s inference systems using the FCFS scheduling policy, they would
schedule these re-initiated requests at the end of the waiting queue and
starve API requests. An easy fix to this scheduling problem is using an API
request’s original arrival time when re-inserting the resumed request to the
waiting queue (we call this scheme ImprovedDiscard).

Discard and ImprovedDiscard both require the recomputation of all context
tokens and incur GPU memory waste from two aspects. First, the recomputing
itself consumes memory not used for producing any new tokens. Assuming the
context for request $i$ when calling API $j$ has $C_{i}^{j}$ tokens, each
token’s KV value occupies $M$ memory, and recomputation takes
$T_{fwd}(C_{i}^{j})$ time, where $T_{fwd}$ is a mapping from the number of
scheduled tokens in a batch to the execution time of that iteration. Then the
memory wasted for the recomputation is $T_{fwd}(C_{i}^{j})\times
C_{i}^{j}\times M$. Second, recomputing the context increases iteration time,
as an iteration now needs to finish all the recomputation alongside model
forwarding of other running requests. The memory occupied by the other running
requests is wasted during the additional iteration time, $T_{fwd}(C_{i}^{j})$.
Thus, the memory waste for this reason is $T_{fwd}(C_{i}^{j})\times
C_{other}\times M$, where $C_{other}$ is the sum of the contexts of other
requests. The total memory waste for Discard and ImprovedDiscard is:

$\begin{split}\text{WasteDiscard}_{i}^{j}&=T_{fwd}(C_{i}^{j})\times
C_{i}^{j}\times M\\\ &+T_{fwd}(C_{i}^{j})\times C_{other}\times M\\\
\end{split}$ (1)

In practice, our evaluation shows that Discard incurs a GPU resource wastage
(GB*min) of 27%, and 37-40% of the total model forwarding time is spent on
recomputation with a mixed workload containing all six APIs in §2.

Preserve-based approach. Instead of discarding, one could preserve the context
of a request when its API call executes. This Preserve strategy avoids
recomputation cost and can immediately resume a request when its API call
returns. However, the preserved context wastes GPU memory for the entire API
execution time. The preserve waste $P_{i}^{j}$ for request $i$ when calling
API $j$ is the API execution time, $T^{j}$, multiplied by the amount of GPU
memory held by the request’s context.

$\text{WastePreserve}_{i}^{j}=T_{API}^{j}\times C_{i}\times M$ (2)

Our measured GPU memory waste with real APIs is surprisingly high: nearly half
of the GPU memory is occupied by interrupted requests for more than 60% of the
time.

Swap-based approach. To avoid recomputation overhead in Discard and memory
waste in Preserve, one potential way is to swap all context data from GPU to
CPU memory when an API call happens and swap it back to GPU upon the API’s
return. The straightforward implementation of Swap performs swapping in a
synchronous manner by launching CUDA kernels for the movement, moving out/in
the data, and then letting computation utilize the memory space. Swapping can
take an extremely long time with huge contexts because of the volume of data
to be moved and the kernels to be launched. To elaborate on the latter,
multiple kernels need to be launched, each for a discontinuous physical memory
region. With PagedAttention (Kwon et al., 2023), the context of an API request
can scatter across physical memory regions, causing high kernel launching
time.

Similar to Discard, Swap incurs memory waste from two sources. First, the
memory space being swapped is wasted during the swap time, accounting for
$T_{swap}(C_{i}^{j})\times C_{i}^{j}\times M$, where $T_{swap}$ is a mapping
from the the number of tokens to swap to the correcponsing swapping latency.
Second, Swap could increase the iteration time, causing all running requests
to wait for the swapping to finish and do nothing. This waste is
$T_{swap}(C_{i}^{j})\times C_{other}\times M$. The total waste doubles to
account for swapping in and out:

$\begin{split}\text{WasteSwap}_{i}^{j}&=2\times T_{swap}(C_{i}^{j})\times
C_{batch}\times M\end{split}$ (3)

Our evaluation shows that Swap wastes 26% GPU resources, and over 25% of the
total execution time is spent on waiting for swap with the mixed workload.

##  4 APIServe

This section presents APIServe and its min-waste preemption designs, also
illustrated in Figure 1.

###  4.1 Swap Pipelining and Chunking

We discuss how APIServe avoids swap memory waste.

Swap pipelining and overlapping.

Swap performs swapping in a serial manner and blocks foreground computation
for the entire swap time. To mitigate this problem, we propose to perform
swapping in a pipelined manner and in the background. Specifically, APIServe
views each model layer’s swapping as one pipeline stage and pipelines kernel
launching, data movement, and foreground model forwarding. For example, when
we launch the swap kernel for layer $i+2$, we move the context for layer
$i+1$, and layer $i$’s context has been freed and used for normal forwarding.

Swap chunking. To further reduce memory waste, we propose to chunk swap out
and swap in across multiple iterations so that in each iteration, the swap
latency can be hidden by overlapping it with model forwarding. We obtain
$T_{fwd}$ through offline profiling and calculate $T_{swap}$ based on the
swapping bandwidth and the per-token memory requirements $M$. At iteration $i$
with batch size $B_{i}$, we set $T_{swap}(N_{i})=T_{fwd}(B_{i})$ and solve for
$N_{i}$, which we call the swap limit. This overlaps swapping with model
forwarding.

Determining swap-in and swap-out budget. At every iteration, there can be
tokens waiting to be swapped out and swapped in. APIServe determines how much
bandwidth to give to swap out and to swap in at each iteration by maximizing
inference throughput (i.e., the number of tokens that can be added to an
iteration’s processing) while guaranteeing the following criteria: 1) the
total amount of swap-in and swap-out tokens should be at most $N_{i}$; 2) the
amount of swapped-out memory should not exceed the amount of free CPU memory
plus the swapped-in memory; and 3) the amount of swapped-in memory and memory
for newly scheduled tokens (i.e., our maximizing target) should not exceed the
amount of swapped-out memory and free GPU memory. We use the obtained swap-in
and swap-out budget when deciding per-request preemption schedule in §4.3.

###  4.2 Recomputation Chunking

Unlike Swap, which consumes GPU-CPU-link bandwidth and can be hidden from
foreground GPU tasks, the recomputation of discarded tokens requires GPU
resources that cannot be avoided. However, we make a key observation that
decoding and recomputation have complimentary resource requirements, with
decoding requiring less GPU core resources (for one query token) than
recomputation (queries for the entire context length) for the same amount of
memory. Thus, a batch of decoding requests usually cannot fill all GPU cores
before running out of GPU memory. Mixing decoding and recomputation requests
can increase GPU core utilization while staying within the GPU memory
boundary. The main challenge to solve is how much recomputation to mix with
decoding.

Discard and ImprovedDiscard impose the recomputation of an entire request in
one iteration. Recomputation of long context can easily add too much
computation burden than what a GPU can handle, causing significantly increased
iteration time. As a result, long $T_{fwd}(C_{i}^{j})$ causes huge
$WasteDiscard$ in Equation 1.

To mitigate this issue, we propose an adaptive technique that separates the
recomputation of a context sequence into chunks, each added to one iteration
without going beyond what the GPU can sustain. We observe that for a given
model architecture, the number of query tokens that all the GPU cores can
process in parallel is limited and fixed. We call this number the GPU
saturation point, $S$, expressed as a number of query tokens. Processing more
query tokens beyond $S$ increases iteration time without improving serving
throughput. APIServe obtains $S$ from offline profiling and sets the chunk
size as $S$ minus the running group size.

The memory waste of our chunked recomputation mechanism can be calculated in
two parts, similar to Equation 1. For the recomputation itself, we add one
chunk of memory waste per iteration that lasts until all chunks have been
recomputed, which essentially cuts the waste of Discard’s all-recomputation-
at-once scheme by half (i.e., $T_{fwd}(C_{i}^{j})\times C_{i}^{j}\times M/2$),
as illustrated in Figure 1. The second part of waste comes from other
requests’ occupied memory during recomputation-added iteration time. This
waste amounts to $n\times T_{fwd}(\frac{C_{i}^{j}}{n})\times C_{other}\times
M$, where $n$ is the number of iterations to finish recomputing the request
and $T_{fwd}(\frac{C_{i}^{j}}{n})$ is the per-chunk added iteration time.
Adding these two parts, we have the total recomputation memory waste of
chunked recomputation as

$\begin{split}\text{WasteChunkD}_{i}^{j}&=\frac{T_{fwd}(C_{i}^{j})\times
C_{i}^{j}\times M}{2}\\\ &+n\times T_{fwd}(\frac{C_{i}^{j}}{n})\times
C_{other}\times M\end{split}$ (4)

Compared to Equation 1, the left term (recomputation itself) cuts the
corresponding term of Discard by half, and the right term (other requests) is
no larger than the right term of Discard because $n\times
T_{fwd}(\frac{C_{i}^{j}}{n})\leq T_{fwd}(C_{i}^{j})$. Meanwhile, APIServe
increases GPU core utilization and further improves overall serving
throughput.

###  4.3 Inter-Request Action Decision

Scheduling API-preempted requests. We now discuss how APIServe schedules API-
preempted requests using a hybrid of preserve, our improved swap, and our
improved discard when considering multiple requests together. First, for each
request $i$ preempted by API $j$, we calculate its memory waste as the min of
WastePreserve (Equation 2) and WasteChunkDiscard (Equation 4):

$\text{Waste}_{i}^{j}=\textbf{min}(\text{WastePreserve}_{i}^{j},\text{WasteChunkD}_{i}^{j})$
(5)

Next, we sort all API-preempted requests in descending order of their memory
waste. We swap out context from these requests according to this order until
we run out of swap-out budget. For the remaining API-preempted requests, we
preserve or discard their remaining context based on its decision from
Equation 5 (i.e. to preserve or discard).

Scheduling API-returned and other waiting requests. At the beginning of each
iteration, APIServe determines what requests to insert into the batch for the
iteration. To facilitate this decision, APIServe maintains three queues: a
running queue containing currently running requests in the system, a swap
queue containing requests that have been swapped out and have returned from
API calls, and a waiting queue. The waiting queue contains discarded requests
that have returned from API calls, new requests received by APIServe that have
never been served, and previously running requests that have been descheduled
due to lack of GPU resources. The latter two types are the same as today’s
inference systems. Each queue is sorted by its requests’ original arrival
times.

At each iteration, we choose requests from the waiting queue in the FCFS
(First-Come First-Serve) order until the GPU saturation point is reached; FCFS
ensures fairness and avoids starvation. Requests chosen can be any of the
three aforementioned waiting types. If it is a discarded request, the
iteration will perform the recomputation of the scheduled number of tokens. If
this request has only been partially recomputed, it remains in the wait queue
with the remaining tokens. At each iteration, we also choose requests from the
swap queue in the FCFS order until the swap-in budget is reached. We maintain
and schedule a separate swap queue because the swap-in budget is additional to
GPU resources and should always be utilized by API requests as much as the
budget allows.

Figure 2: Normalized Latency against Request Arrival Rate. Lower right is
better, i.e., sustains higher serving load.

Figure 3: Throughput against Request Arrival Rate  Throughput expressed as
completed requests per second. Higher is better.

###  4.4 API Execution Time Estimation

The calculation of memory waste for preserving context (Equation 2) requires
having a value for the API execution time. For API types with large variations
in execution times and for API types not profiled offline, we propose a
dynamic estimation method by setting $\hat{T}_{API}=t_{now}-t_{call}$ where
$t_{now}$ is the current time updated at each iteration and $t_{call}$ is the
time when the API was invoked. Effectively, the longer a request is paused,
the larger API time it gets assigned. Our evaluation shows that APIServe using
this estimation method achieves 93% of the performance compared with using an
oracle for API execution time.

##  5 Evaluation Results

We implement APIServe on top of vLLM and reuse its PagedAttention technique.
We compare APIServe to vanilla vLLM, ImprovedDiscard, Preserve, and Swap, as
discussed in §3.2. To evaluate APIServe and the baselines, we augment the
6B-parameter GPT-J model (Wang & Komatsuzaki, 2021) and the 13B Vicuna model
(Chiang et al., 2023) with APIs presented in §2. We run augmented GPT-J on one
NVIDIA A100 GPU on an AWS p4d instance. For Vicuna, we use two environments:
running on a single A100 GPU and distributed on two A100 GPUs in one p4d
instance. To mimic real-world serving scenarios that often receive different
types of requests, we use a request dataset that merges the six API types
presented in §2 by uniformly sampling requests from them.

###  5.1 End-to-End Performance

We first compare the end-to-end performance of APIServe and the baselines.
Following recent LLM inference research papers (Kwon et al., 2023; Yu et al.,
2022), we first report the serving throughput as normalized latency (i.e., the
median of every request’s end-to-end latency divided by its output length)
when varying request load (number of requests arrived per second). Under the
same normalized latency, a higher-throughput system should sustain a higher
request rate. Apart from normalized latency, we also report the number of
finished requests per second. Figure 2 and Figure 3 show these results for
APIServe and the baselines on the 6B and 13B models. For 13B, we report both a
single GPU and two GPU executions.

6B model results. APIServe outperforms baselines across all metrics. APIServe
sustains up to 1.6$\times$ higher request arrival rate at the same normalized
latency on the 6B model compared to vLLM. Under the same request rate,
APIServe’s normalized latency is 1.9$\times$-5.7$\times$ lower. The vanilla
vLLM suffers from high recomputation waste and delays in scheduling API-
returned requests per FCFS policy. Delayed execution greatly impacts vanilla
vLLM’s number of requests finished per second.

ImprovedDiscard maintains an API request’s original arrival time. Thus, both
its normalized latency and finished requests per second are better than
vanilla vLLM. However, ImprovedDiscard still incurs the same recomputation
overhead and underperforms APIServe on all metrics.

Preserve is the best among all the baselines, showing that for our mixed
workload, the preserving overhead is lower than the recomputation overhead for
most requests. Yet, Preserve is worse than APIServe on all metrics.

The overall results for Swap are similar to ImprovedDiscard. Both baselines
avoid GPU memory waste during API execution time. While ImprovedDiscard causes
running requests to wait for the recomputation requests in an iteration, Swap
causes running requests to wait for swapping out/in requests in an iteration.
These overheads have similar effects on the final performance metrics.

APIServe outperforms all of the baselines, ImprovedDiscard, Preserve, and Swap
mainly because of our min-waste-based schedule. Moreover, APIServe improves
the recomputation and swap mechanisms over ImprovedDiscard and Swap with our
chunking and pipelining techniques. Compared to them, APIServe eliminates over
60% of GPU waste caused by recomputation and 96% of GPU waste for swapping.

13B model results. When running the 13B model on a single GPU, APIServe
outperforms all the baselines on normalized latency but with smaller
improvements than the 6B model. A larger model’s weights occupy more GPU
memory, leaving less memory for KV caches. Moreover, a larger model’s normal
forwarding is longer and contributes to most of the normalized latency. Thus,
the room for improvement is small in general. Still, we serve up to
1.25$\times$ higher request rates.

A more practical scenario for large model inference is a distributed execution
environment where the model is distributed to multiple GPUs, usually with
tensor parallelism (Shoeybi et al., 2020). We evaluate the 13B model on two
GPUs in the same server. APIServe outperforms all baselines by a larger margin
than when running the 13B on a single GPU. APIServe serves up to 1.8$\times$
higher request arrival rates and a 1.6$\times$-10$\times$ improvement in
normalized latency compared to vLLM.

Single-API workloads. Apart from the mixed workload, we evaluate APIServe on
two single-API workloads: QA and Chatbot (§2). APIServe outperforms vLLM by up
to 2.3$\times$ and 1.9$\times$ on normalized latency with QA and Chatbot
respectively. APIServe’s improvement with QA is lager because most QA API
calls are short and have smaller waste when using preserve than discard.

###  5.2 Performance Deep Dive

To further understand APIServe’s benefits, we break down its techniques by
adding them one at a time to vanilla vLLM. We report the normalized latency
and GPU memory waste under the load of 2 requests per second for the 6B model
in Figure 4. Other request loads have similar trends.

Figure 4: APIServe Technique Breakdown.

We first improve Discard (vanilla vLLM) by maintaining the original API
request arrival time as in ImprovedDiscard, which reduces normalized latency
by 24.5%. We then add APIServe’s recomputation chunking, resulting in 7.8%
more improvement. Next, we add APIServe’s budgeted swapping (discarding once
the limit is reached). This results in an added improvement of 12.7%. Then, we
add preserve and use a simple heuristic to decide between discard and
preserve: discard for interactive (or long-running) APIs and preserve for
automated (short-running) APIs. This addition has a 46.1% improvement, mainly
because of the added support for Preserve. Finally, adding min-waste-based
adaptive schedule (i.e., the whole APIServe) results in an additional 46.4%
improvement. Each of APIServe’s techniques reduces GPU memory waste, with the
whole APIServe only having 0.69% waste. This demonstrates the effectiveness of
our min-waste preemption principle.

##  6 Conclusion

We introduced APIServe, an LLM inference framework optimized for API calls. By
minimizing GPU memory waste as a unified objective, APIServe delivers
1.25$\times$-2$\times$higher request arrival rates and achieves
1.6$\times$-10$\times$lower normalized latency compared to state-of-the-art
inference systems.

## References

  * Agrawal et al. (2023) Agrawal, A., Panwar, A., Mohan, J., Kwatra, N., Gulavani, B. S., and Ramjee, R.  Sarathi: Efficient llm inference by piggybacking decodes with chunked prefills.  _arXiv preprint arXiv:2308.16369_ , 2023.
  * AI (2023) AI, S.  Bark: Text-to-speech model.  https://github.com/suno-ai/bark, 2023.
  * Aminabadi et al. (2022) Aminabadi, R. Y., Rajbhandari, S., Awan, A. A., Li, C., Li, D., Zheng, E., Ruwase, O., Smith, S., Zhang, M., Rasley, J., et al.  Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale.  In _SC22: International Conference for High Performance Computing, Networking, Storage and Analysis_ , pp. 1–15. IEEE, 2022.
  * Baeza-Yates et al. (1999) Baeza-Yates, R., Ribeiro-Neto, B., et al.  _Modern Information Retrieval_ , volume 463.  ACM Press, New York, 1999.
  * Brockman et al. (2023) Brockman, G., Eleti, A., Georges, E., Jang, J., Kilpatrick, L., Lim, R., Miller, L., and Pokrass, M.  Introducing chatgpt and whisper apis.  OpenAI Blog, March 1 2023.  URL https://openai.com/blog/introducing-chatgpt-and-whisper-apis.
  * Brysbaert (2019) Brysbaert, M.  How many words do we read per minute? a review and meta-analysis of reading rate, Apr 2019.  URL osf.io/preprints/psyarxiv/xynwg.
  * Chase (2022) Chase, H.  LangChain, October 2022.  URL https://github.com/langchain-ai/langchain.
  * Chiang et al. (2023) Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J. E., Stoica, I., and Xing, E. P.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.  URL https://lmsys.org/blog/2023-03-30-vicuna/.
  * Costa-jussà et al. (2022) Costa-jussà, M. R., Cross, J., Çelebi, O., Elbayad, M., Heafield, K., Heffernan, K., Kalbassi, E., Lam, J., Licht, D., Maillard, J., Sun, A., Wang, S., Wenzek, G., Youngblood, A., Akula, B., Barrault, L., Gonzalez, G. M., Hansanti, P., Hoffman, J., Jarrett, S., Sadagopan, K. R., Rowe, D., Spruit, S., Tran, C., Andrews, P., Ayan, N. F., Bhosale, S., Edunov, S., Fan, A., Gao, C., Goswami, V., Guzmán, F., Koehn, P., Mourachko, A., Ropers, C., Saleem, S., Schwenk, H., and Wang, J.  No language left behind: Scaling human-centered machine translation, 2022.
  * Greyling (2023) Greyling, C.  When using the chatgpt api, users will have to manage the context.  Medium, March 6 2023.  URL https://cobusgreyling.medium.com/when-using-the-chatgpt-api-users-will-have-to-manage-the-context-ba5869238913.
  * Gu et al. (2022) Gu, J., Stefani, E., Wu, Q., Thomason, J., and Wang, X.  Vision-and-language navigation: A survey of tasks, methods, and future directions.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, 2022.  doi: 10.18653/v1/2022.acl-long.524.  URL http://dx.doi.org/10.18653/v1/2022.acl-long.524.
  * Hao et al. (2023) Hao, S., Liu, T., Wang, Z., and Hu, Z.  Toolkengpt: Augmenting frozen language models with massive tools via tool embeddings.  _arXiv preprint arXiv:2305.11554_ , 2023.
  * Hu et al. (2023) Hu, Y., Lin, F., Zhang, T., Yi, L., and Gao, Y.  Look before you leap: Unveiling the power of gpt-4v in robotic vision-language planning, 2023.
  * Huang et al. (2022) Huang, W., Abbeel, P., Pathak, D., and Mordatch, I.  Language models as zero-shot planners: Extracting actionable knowledge for embodied agents, 2022.
  * Izacard et al. (2022) Izacard, G., Lewis, P., Lomeli, M., Hosseini, L., Petroni, F., Schick, T., Dwivedi-Yu, J., Joulin, A., Riedel, S., and Grave, E.  Atlas: Few-shot learning with retrieval augmented language models, 2022.
  * Kwon et al. (2023) Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I.  Efficient memory management for large language model serving with pagedattention.  In _Proceedings of the 29th Symposium on Operating Systems Principles_ , pp. 611–626, 2023.
  * Li et al. (2023) Li, Z., Zheng, L., Zhong, Y., Liu, V., Sheng, Y., Jin, X., Huang, Y., Chen, Z., Zhang, H., Gonzalez, J. E., et al.  Alpaserve: Statistical multiplexing with model parallelism for deep learning serving.  _arXiv preprint arXiv:2302.11665_ , 2023.
  * Ma et al. (2015) Ma, Z., Edge, D., Findlater, L., and Tan, H. Z.  Haptic keyclick feedback improves typing speed and reduces typing errors on a flat keyboard.  In _2015 IEEE World Haptics Conference (WHC)_ , pp. 220–227. IEEE, 2015.
  * Mialon et al. (2023) Mialon, G., Dessì, R., Lomeli, M., Nalmpantis, C., Pasunuru, R., Raileanu, R., Rozière, B., Schick, T., Dwivedi-Yu, J., Celikyilmaz, A., Grave, E., LeCun, Y., and Scialom, T.  Augmented language models: a survey, 2023.
  * NVIDIA (2023) NVIDIA.  Fastertransformer.  https://github.com/NVIDIA/FasterTransformer, 2023.
  * OpenAI (2023) OpenAI.  ChatGPT plugins.  https://openai.com/blog/chatgpt-plugins, March 2023.
  * OpenAI et al. (2023) OpenAI, :, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., Avila, R., Babuschkin, I., Balaji, S., Balcom, V., Baltescu, P., Bao, H., Bavarian, M., Belgum, J., Bello, I., Berdine, J., Bernadett-Shapiro, G., Berner, C., Bogdonoff, L., Boiko, O., Boyd, M., Brakman, A.-L., Brockman, G., Brooks, T., Brundage, M., Button, K., Cai, T., Campbell, R., Cann, A., Carey, B., Carlson, C., Carmichael, R., Chan, B., Chang, C., Chantzis, F., Chen, D., Chen, S., Chen, R., Chen, J., Chen, M., Chess, B., Cho, C., Chu, C., Chung, H. W., Cummings, D., Currier, J., Dai, Y., Decareaux, C., Degry, T., Deutsch, N., Deville, D., Dhar, A., Dohan, D., Dowling, S., Dunning, S., Ecoffet, A., Eleti, A., Eloundou, T., Farhi, D., Fedus, L., Felix, N., Fishman, S. P., Forte, J., Fulford, I., Gao, L., Georges, E., Gibson, C., Goel, V., Gogineni, T., Goh, G., Gontijo-Lopes, R., Gordon, J., Grafstein, M., Gray, S., Greene, R., Gross, J., Gu, S. S., Guo, Y., Hallacy, C., Han, J., Harris, J., He, Y., Heaton, M., Heidecke, J., Hesse, C., Hickey, A., Hickey, W., Hoeschele, P., Houghton, B., Hsu, K., Hu, S., Hu, X., Huizinga, J., Jain, S., Jain, S., Jang, J., Jiang, A., Jiang, R., Jin, H., Jin, D., Jomoto, S., Jonn, B., Jun, H., Kaftan, T., Łukasz Kaiser, Kamali, A., Kanitscheider, I., Keskar, N. S., Khan, T., Kilpatrick, L., Kim, J. W., Kim, C., Kim, Y., Kirchner, H., Kiros, J., Knight, M., Kokotajlo, D., Łukasz Kondraciuk, Kondrich, A., Konstantinidis, A., Kosic, K., Krueger, G., Kuo, V., Lampe, M., Lan, I., Lee, T., Leike, J., Leung, J., Levy, D., Li, C. M., Lim, R., Lin, M., Lin, S., Litwin, M., Lopez, T., Lowe, R., Lue, P., Makanju, A., Malfacini, K., Manning, S., Markov, T., Markovski, Y., Martin, B., Mayer, K., Mayne, A., McGrew, B., McKinney, S. M., McLeavey, C., McMillan, P., McNeil, J., Medina, D., Mehta, A., Menick, J., Metz, L., Mishchenko, A., Mishkin, P., Monaco, V., Morikawa, E., Mossing, D., Mu, T., Murati, M., Murk, O., Mély, D., Nair, A., Nakano, R., Nayak, R., Neelakantan, A., Ngo, R., Noh, H., Ouyang, L., O’Keefe, C., Pachocki, J., Paino, A., Palermo, J., Pantuliano, A., Parascandolo, G., Parish, J., Parparita, E., Passos, A., Pavlov, M., Peng, A., Perelman, A., de Avila Belbute Peres, F., Petrov, M., de Oliveira Pinto, H. P., Michael, Pokorny, Pokrass, M., Pong, V., Powell, T., Power, A., Power, B., Proehl, E., Puri, R., Radford, A., Rae, J., Ramesh, A., Raymond, C., Real, F., Rimbach, K., Ross, C., Rotsted, B., Roussez, H., Ryder, N., Saltarelli, M., Sanders, T., Santurkar, S., Sastry, G., Schmidt, H., Schnurr, D., Schulman, J., Selsam, D., Sheppard, K., Sherbakov, T., Shieh, J., Shoker, S., Shyam, P., Sidor, S., Sigler, E., Simens, M., Sitkin, J., Slama, K., Sohl, I., Sokolowsky, B., Song, Y., Staudacher, N., Such, F. P., Summers, N., Sutskever, I., Tang, J., Tezak, N., Thompson, M., Tillet, P., Tootoonchian, A., Tseng, E., Tuggle, P., Turley, N., Tworek, J., Uribe, J. F. C., Vallone, A., Vijayvergiya, A., Voss, C., Wainwright, C., Wang, J. J., Wang, A., Wang, B., Ward, J., Wei, J., Weinmann, C., Welihinda, A., Welinder, P., Weng, J., Weng, L., Wiethoff, M., Willner, D., Winter, C., Wolrich, S., Wong, H., Workman, L., Wu, S., Wu, J., Wu, M., Xiao, K., Xu, T., Yoo, S., Yu, K., Yuan, Q., Zaremba, W., Zellers, R., Zhang, C., Zhang, M., Zhao, S., Zheng, T., Zhuang, J., Zhuk, W., and Zoph, B.  Gpt-4 technical report, 2023.
  * OpenTable (2023) OpenTable.  New: Chatgpt restaurant recs, powered by opentable, March 23 2023.  URL https://www.opentable.com/blog/chatgpt/.
  * Parisi et al. (2022) Parisi, A., Zhao, Y., and Fiedel, N.  Talm: Tool augmented language models.  _arXiv preprint arXiv:2205.12255_ , 2022.
  * Patil et al. (2023) Patil, S. G., Zhang, T., Wang, X., and Gonzalez, J. E.  Gorilla: Large language model connected with massive apis.  _arXiv preprint arXiv:2305.15334_ , 2023.
  * Rombach et al. (2021) Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B.  High-resolution image synthesis with latent diffusion models, 2021.
  * Schick et al. (2023) Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Zettlemoyer, L., Cancedda, N., and Scialom, T.  Toolformer: Language models can teach themselves to use tools.  _arXiv preprint arXiv:2302.04761_ , 2023.
  * Shoeybi et al. (2020) Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B.  Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020.
  * Shridhar et al. (2020) Shridhar, M., Yuan, X., Côté, M.-A., Bisk, Y., Trischler, A., and Hausknecht, M.  Alfworld: Aligning text and embodied environments for interactive learning.  _arXiv preprint arXiv:2010.03768_ , 2020.
  * Touvron et al. (2023) Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023.
  * Wang & Komatsuzaki (2021) Wang, B. and Komatsuzaki, A.  GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.  https://github.com/kingoflolz/mesh-transformer-jax, May 2021.
  * Wei et al. (2022) Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in Neural Information Processing Systems_ , 35:24824–24837, 2022.
  * Wolfram (2023) Wolfram, S.  Chatgpt gets its ’wolfram superpowers’!  Stephen Wolfram Writings, 2023.  URL https://writings.stephenwolfram.com/2023/03/chatgpt-gets-its-wolfram-superpowers/.
  * Wu et al. (2023) Wu, B., Zhong, Y., Zhang, Z., Huang, G., Liu, X., and Jin, X.  Fast distributed inference serving for large language models, 2023.
  * Yang et al. (2018) Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W. W., Salakhutdinov, R., and Manning, C. D.  Hotpotqa: A dataset for diverse, explainable multi-hop question answering.  _arXiv preprint arXiv:1809.09600_ , 2018.
  * Yao et al. (2022) Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y.  React: Synergizing reasoning and acting in language models.  _arXiv preprint arXiv:2210.03629_ , 2022.
  * Yu et al. (2022) Yu, G.-I., Jeong, J. S., Kim, G.-W., Kim, S., and Chun, B.-G.  Orca: A Distributed Serving System for Transformer-Based Generative Models.  In _16th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’22)_ , Carlsbad, CA, July 2022.
  * Zhang et al. (2022) Zhang, Z., Zhang, A., Li, M., and Smola, A.  Automatic chain of thought prompting in large language models, 2022.
  * Zheng et al. (2023a) Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., and Stoica, I.  Judging llm-as-a-judge with mt-bench and chatbot arena, 2023a.
  * Zheng et al. (2023b) Zheng, L., Yin, L., Xie, Z., Huang, J., Sun, C., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., Barrett, C., and Sheng, Y.  Efficiently programming large language models using sglang, 2023b.
  * Zhong et al. (2024) Zhong, Y., Liu, S., Chen, J., Hu, J., Zhu, Y., Liu, X., Jin, X., and Zhang, H.  Distserve: Disaggregating prefill and decoding for goodput-optimized large language model serving, 2024.

Appendix

Figure 5: CDF Results of Short APIs. Each line plots the CDF distribution of
all the calls of one API type.

Figure 6: CDF Results of Long APIs. Each line plots the CDF distribution of
all the calls of one API type.

Figure 7: Time-to-first-token (TTFT) with 6B and 13B models on the mixed
workload

## API Properties and Dataset Generation

We analyze the properties of different datasets to evaluate the effectiveness
of our inference system. We consider datasets from arithmetic, a knowledge-
based question and answering system, a multi-step chat bot, and an embodied
virtual environment. The dataset demonstrates different execution time,
frequency of API calls, and context lengths. We augment the datasets with
relevant APIs from calculator, wikipedia, and a virtual environment gym.

We measure the API execution time, the number of API calls, the number of
returned tokens of an API call, and the context length when an API is called.
Figures 5 and 6 present the detailed CDF results the four metrics for short-
running APIs (math, QA, VE) and long-running APIs (chatbot, image, TTS).
Below, we briefly introduce each API type and discuss their properties.

We use the ReAct framework to prompt the LLM to call a Wikipedia endpoint with
different questions. The responses from the Wikipedia endpoint are
postprocessed. We use the live Wikipedia API endpoint for our benchmark and
For each call, we retrieve a summary of the relevant Wikipedia page, which we
post-process by limiting the size of the responses to fit within the maximum
model sequence length.

To evaluate VE, we prompt GPT-4 LLM with an initial virtual environment and
use the ReAct event loop to repeatedly interact with the environment. For our
evaluation, we truncate to fit within the context length.

We prompt GPT-4 to generate complex Stable Diffusion prompts at a certain
size. Each prompt will trigger an API call of the Stable Diffusion model
(Rombach et al., 2021) to generate an image. Each round with the LLM involves
a user prompt with normal decoding and then a diffusion call. The number of
calls total is modeled around the chat dataset. We estimate the number of
image generation rounds (i.e., number of API calls) to be the same as the
number of chatting rounds in Chatbot. As there is no standard way of returning
tokens from an image generation, we use a short, constant-length sentence
describing the generated image as returned tokens. For API execution time, we
measure the actual Stable Diffusion call and estimate the human response time
in the same as as Chatbot; the sum of them at each round is shown in Table 1
and Figure 6. For the context and return length, we provide a mixture of
ShareGPT dataset (Zheng et al., 2023a) and the real prompts.

We use a methodology similar to image generation to understand TTS-augmented
LLMs.

Figure 8: Overall APIServe Architecture.

## Implementation Details

We implement APIServe on top of vLLM and build APIServe, an inference system
designed for API-augmented LLMs. As shown in Figure 8, APIServe includes an
API executor that performs different API calls, an iterative-level scheduler
that forms a batch of waiting requests and (partial) paused requests, a swap
manager that facilitates GPU/CPU memory swapping, a running status monitor
that monitors the resource usage and size of running requests, a waste
estimator that calculates GPU memory waste for each API-paused request, and an
offline profiler that collects basic systems metrics before starting serving.

At the end of each iteration, the scheduler gathers all requests that trigger
API calls, adds them to the paused-request queue, swaps as much context
(computed keys and values) of paused requests as allowed to CPU memory,
determines whether to preserve or discard the remaining context of each paused
request based on the GPU memory waste amount calculated by the waste
estimator, and calls the actual API. When an API call finishes, APIServe
determines how many swapped-out or discarded tokens to swap in or recompute in
the next iteration. It also determines how many waiting requests (non-API
paused requests) to schedule in the next iteration. When all the context of a
paused request has been restored, APIServe adds the tokens returned by the API
call to the resumed request.

## Additional Evaluation Results

A major distinction between typical inference workloads and tool-augmented
workloads is the added latency from API execution. We observe that 31% percent
of the median request end-to-end time is spent waiting on its API to return.
This means requests take longer to complete, which can add significant latency
for those in the waiting queue. Hence, we place an emphasis on improving time
to first token (TTFT) in addition to improving serving throughput as measured
by vLLM and Orca. We vary request arrival rate and collect median TTFT as well
as median normalized end-to-end latency. Normalized latency is measured by
dividing the request end-to-end time by its output length. In our metric, we
remove a request’s API execution time from its end-to-end latency because the
user perceives the API execution, which means the system is not penalized. All
experiments use 30-minute traces. The results on our 6B, 13B, and distributed
13B models are shown in Figure 7.