# A Trade-off Analysis of Replacing Proprietary LLMs with Open Source SLMs in
Production

Chandra Irugalbandara  Jaseci Labs
<EMAIL_ADDRESS>Ashish Mahendra  Jaseci Labs
<EMAIL_ADDRESS>Roland Daynauth  University of Michigan
<EMAIL_ADDRESS>Tharuka Kasthuri Arachchige  Jaseci Labs
<EMAIL_ADDRESS>Krisztian Flautner  University of Michigan
<EMAIL_ADDRESS>Lingjia Tang  University of Michigan
Jaseci Labs
<EMAIL_ADDRESS>Yiping Kang  University of Michigan
Jaseci Labs
<EMAIL_ADDRESS>Jason Mars  University of Michigan
Jaseci Labs
<EMAIL_ADDRESS>
###### Abstract

Many companies rely on APIs of managed AI models such as OpenAIs GPT-4 to
create AI-enabled experiences in their products. Along with the benefits of
ease of use and shortened time to production, this reliance on proprietary
APIs has downsides in terms of model control, performance reliability, up-time
predictability, and cost. At the same time, there has been a flurry of open
source small language models (SLMs) that have been made available for
commercial use. However, their readiness to replace existing capabilities
remains unclear, and a systematic approach to test these models is not readily
available.

In this paper, we present a systematic evaluation methodology for, and
characterization of, modern open source SLMs and their trade-offs when
replacing a proprietary LLM APIs for a real-world product feature. We have
designed SLaM, an automated analysis tool that enables the quantitative and
qualitative testing of product features utilizing arbitrary SLMs. Using SLaM,
we examine both the quality and the performance characteristics of modern SLMs
relative to an existing customer-facing OpenAI-based implementation. We find
that across 9 SLMs and 29 variants, we observe competitive quality-of-results
for our use case, significant performance consistency improvement, and a cost
reduction of 5x–29x when compared to OpenAI GPT-4.

###### Index Terms:

Deep Learning, Language Models, Characterization

##  I Introduction

Generative AI (GenAI), particularly generative large language models (LLMs),
has recently seen a surge in popularity within both academic and industry
communities. This surge is largely attributed to OpenAI’s launch of ChatGPT
[1] and its groundbreaking performance and capabilities across a wide range of
natural language generation tasks. These tasks range from intelligent chatbots
and specialized document creation to coding assistance, among others. Indeed,
this emerging class of AI has disrupted the commercial AI landscape as they
have become a top strategic priority for many startups and established
corporations [2].

Generative LLMs take natural language input (prompts) and generate responses
following the instructions and information provided in the input prompt. These
models are transformer neural networks with billions to trillions of
parameters and are prohibitively expensive to train and serve. Consequently,
many companies turn to cloud API offerings to develop their next-generation
GenAI features and products.

OpenAI’s GPT models, especially GPT-4, have emerged as a favored choice for
their cutting-edge capabilities and developer-friendly interface [3],
facilitating rapid prototyping and short time-to-market. However, reliance on
proprietary cloud APIs to access LLMs’ capabilities presents numerous
challenges and drawbacks.

Issues such as inconsistent request latencies and outages during peak usage
times have been noted with OpenAI’s APIs [4]. Furthermore, the per-request
pricing model can accumulate significant costs over time, which is
particularly challenging for startups. Additionally, the proprietary nature of
cloud APIs limits developers from customizing models with their data to better
suit specific use cases through fine-tuning.

Meanwhile, the AI community has seen the emergence of numerous open-source
language models of different sizes available for commercial use. Additionally,
several quantization techniques to make these models even smaller have
recently been published and applied to these models [5][6]. We categorize
these models as Small Language Models (SLMs). These SLMs have shown promising
performance, comparable to larger LLMs in certain benchmarks [7][8][9].
However, the readiness of SLMs to replace OpenAI’s APIs in production settings
remains unclear, particularly regarding response quality, performance, and
cost-effectiveness. Additionally, a methodology and tooling for evaluating
SLMs for a particular product feature is not readily available.

In this work, we present SLaM, a systematic methodology and automated tooling
for evaluating open-source SLMs compared to proprietary LLM API services. SLaM
(1) automates the download and hosting of any set of HuggingFace models into a
hosted environment, (2) provide human-in-the-loop tooling for crowd source
blind testing of model responses, and (3) includes a range of fully automated
AI based model evaluation pipelines. Using SLaM, we characterize modern SLMs
and their quantized variants through a case study of replacing a production
application’s existing OpenAI GPT-4 API-based feature. We examines four
questions relevant to the viability of migrating to self-hosted SLMs:

  1. 1.

Is the quality of modern SLMs good enough for users?

  2. 2.

How well can AI-assisted tooling automate the process of identifying SLM
alternatives?

  3. 3.

What are the latency implications of self-hosted SLMs in a utility-based cloud
environment such as AWS?

  4. 4.

What are the cost trade-offs of self-hosted SLMs compared to using OpenAIs
GPT-4 APIs?

Utilizing the SLaM evaluation methodology and its accompanying toolset, we
perform an analysis of a suite of 9 unique SLMs and their quantized versions,
totaling 29 different models. We evaluate applying these models to an existing
in-production GPT4-based “pep talk” feature in a productivity tool, using user
data on tasks to deliver daily, intelligent encouragement. We show that open-
source SLMs can indeed effectively replace OpenAI’s models in a production
environment. Specifically, we find that SLMs, compared to OpenAI, can generate
responses with similar levels of quality, provide a similar yet more reliable
latency performance, and reduce costs by up to 23X. This paper makes the
following contributions:

  * •

We introduce a systematic methodology and process for evaluating a range of
open-source SLMs for product feature implementation in comparison to
proprietary LLM cloud APIs.

  * •

We design and develop the SLaM tool, which support human-in-the-loop and AI
automated evaluation process. We plan to open-source SLaM tool and the results
presented in this paper upon publication.

  * •

We present a characterization of modern quantized SLMs through a case study of
replacing GPT-4 in a real-world application. We find SLMs are not just indeed
viable alternatives to proprietary APIs but has advantages.

Figure 1: Brief history of evolution of language models.

Figure 2: OpenAI APIs status, captured on 12/14/23

Figure 3: Architecture of the SLaM Evaluation tool

Figure 4: The screenshot of SLaM Tool for human evaluation of model responses

##  II Background: The Recent Evolution of LLMs

###  II-A Large Language Models

LLMs are language models that are pre-trained on large datasets with deep
learning techniques to process and comprehend natural language. Because of
their outstanding performance in a variety of applications, LLMs are gaining
popularity in both academia and industry [10]. LLMs are capable of performing
a variety of tasks, such as text generation, translation, and summarization,
as well as real-world applications, such as virtual assistants, chatbots, and
language translation systems [11].

In contrast to the conventional approach for NLP tasks, which involves fine-
tuning models through supervised learning on small, task-specific datasets,
LLMs can effectively perform a wide range of tasks based on instructions in
the input (prompts). By providing LLMs with instructions and/or examples of
the desired outcome, LLMs can perform tasks for which they were not explicitly
trained for.

Figure 1 visualizes a brief history of language models. Vaswani et al. [12]
first introduced in 2017 the transformer architecture and self-attention
mechanism. This innovation greatly improved performance across various NLP
applications. GPT-2 [13], introduced in 2019, is a large transformer-based
model with 1.5 billion parameters, employing self-attention mechanisms for
language understanding. Megatron-LM (2019) [14] surpasses GPT-2 with 8.3
billion parameters. This extensive size enables Megatron-LM to capture and
generate more intricate linguistic patterns. On March 14, 2023, OpenAI
announced the release of GPT-4, their latest LLM. This state-of-the-art LLM
powers many of OpenAI’s popular AI applications, including the widely used
ChatGPT, and brings much attention to the research of LLMs. [15]

###  II-B Impact of LLMs

LLMs are revolutionizing the tech industry by altering how we work and
interact with information. Significant increases in efficiency and the
establishment of new job categories are possible with the integration of LLMs.
LLMs have been integrated into various applications, including virtual
assistants, language translation, content writing, and scientific research,
where they assist in generating human-like text. LLMs can handle routine
inquiries, freeing human agents to handle more complex issues. Over 2 million
developers have adopted LLMs for their applications. [16].

###  II-C OpenAI APIs

OpenAI APIs have quickly emerged as the preferred cloud options for LLMs
inference. While OpenAI’s APIs provide convenient access to powerful language
models such as GPT-4, reliance on cloud APIs and models present several
challenges for developers, including lack of model control, unreliable uptime
and potential significant cost. Furthermore, cloud APIs also limit the
developer’s ability to fine-tune the models with custom data to tailor to
specific tasks

###  II-D Open-source Language Models

There have been a surge of open-source language models released by the
research community and industry companies. Figure 1 highlights the timeline of
several key open-source LLMs release since the launch of ChatGPT. These LLMs
are generally smaller in size, making them viable model to self-host. In
addition, several quantization techniques have been proposed to make these
models even smaller [17][18]. The quantization process entails mapping the
model’s weights from data types with higher precision (16bit) to ones with
lesser precision (4bit, 3bit, 2bit). By doing this, the model is effectively
compressed, which lowers memory consumption.

###  II-E Developing with LLMs

The open-source community has created new infrastructure and tools to make
developing with LLM an easy process, further accelerate the adoption of LLMs
into modern applications and products. LangChain [19] is a platform that
assists developers in creating and distributing reasoning applications with
natural language models that are context-aware. LlamaIndex is another
noteworthy tool. It is a data platform that links large language models (LLMs)
to bespoke data sources for various use cases, including structured analytics
and chatbots enhanced with data. It offers data ingestion, indexing, query
interface, and knowledge agents for various data types and jobs.

##  III Problem: Realizing the “Pep-Talk” Feature

This paper investigates the feasibility of replacing an LLM (via OpenAI APIs)
with SLMs in a production use case. In particular, we are interested in the
response quality of SLMs compared to LLMs. In addition, we aim to understand
whether SLMs can address and mitigate key limitations when relying on LLMs for
production use, specifically cost and performance (latency etc.).

This study evaluates a real LLM use case in a production application. Here, we
first describe the details of the application and the LLM use case. We then
present the key limitations of OpenAI APIs and several research questions we
aim to answer when comparing SLMs with LLMs.

###  III-A Product Feature Case Study

The application in this case study is a personal task-management and
productivity application that has been in production for over a year. In this
application (available on both desktop and mobile devices), users can list the
tasks they plan to work on each day. The tasks can be from different life
domains, work, personal health, finances, etc. Users can also set their
longer-term goals and daily habits. The platform records what they have
accomplished daily, their progress towards their goals, habits completion,
etc. Users use this software to manage their tasks and stay organized,
focused, and productive. A number of AI features help the user achieve their
objectives.

One of the platform’s key features is the “Daily Pep Talk”. When a user logs
in to the application at the beginning of their day, the platform provides an
encouraging message based on what they have accomplished the day before, what
they plan to do today, and their progress toward their goals. The user’s
activity from the previous day, along with their upcoming plan for the coming
day are combined with additional context and instructions into a prompt. This
prompt is used as input to an LLM to instruct it to generate the appropriate
response, which is then shown to the user. We show the response template
below. The [placeholders] are replaced with user specific data in production

You are given below a description of the tasks and goals in my todo list.
Always answer the query using the provided context information, and not prior
knowledge.

Some rules to follow:

1\. Avoid statements like ’Based on the context, …’ or ’The context
information …’ or anything along those lines.

Context information is below.

———————

Today is 2023-11-13.

The last day I logged in was 2023-11-10 and I completed the following tasks on
that day:

[LIST OF TASKS COMPLETED]

Here are today’s focused tasks. These are tasks that I think are important:

[LIST OF TASKS PLANNED]

Here are the rituals that are scheduled for today. These are recurring tasks
that help me build and maintain good habits or work/life responsibilities that
happen regularly:

[LIST OF RECURRING TASKS SCHEDULED]

I have also set the following goals for myself for this week. These are
overarching objectives I want to accomplish in this week:

[LIST OF USER SET GOALS]

———————

Given the context information, answer the query.

Query: Imagine you are my personal assistant, generate a short briefing for me
at the start of my day. In the briefing, summarize what I completed in the
previous day and then give me a preview of the key activities for today. In
this briefing, consider my goals for this week and tell me if my focused tasks
and rituals are aligned with those goals. Carefully evaluate the associations
between the tasks and goals and describe the tasks based on how related you
think they are. Note that it is possible that a task is not directly
associated with any goals. Reference the specific tasks mentioned in the
context and generate this briefing in a single, naturally flowing narrative.
Avoid simply listing out tasks one by one. Use a motivating and encouraging
tone. Keep your response within 4 sentences.

Answer:

This method of including user- and feature-specific information and detailed
instructions describing the scenario in the input to an LLM is commonly called
”prompt engineering.” Prompt engineering is currently the most popular method
of adapting a pre-trained LLM to a specific use case [20, 21].

Other applications have also integrated AI components, such as an intelligent
assistant directly relying on OpenAI APIs [22, 16]. However, as we have
discovered in production, relying on GPT-4 via OpenAI to provide the LLM
service poses several challenges.

###  III-B Challenges with LLM APIs

In the currently deployed version of the application, we use the GPT-4 model
from OpenAI’s cloud service. While the OpenAI APIs provide state-of-the-art
performance and its easy-to-use cloud APIs reduce time-to-market, our
experience indicates three key limitations when using them in production
deployments.

Performance and Reliability Developers and users of OpenAI have reported a
large variability in query latency [23], ranging from sub 1s to 4 - 5 minutes.
In addition to latency variability, OpenAI has reported frequent API outages
since its launch. Figure 4 shows a snapshot of the OpenAI cloud platform
status as of December 13th, 2023. In November alone, it experienced four major
outages and three partial outages, some of which lasted more than 3 hours.
Given that OpenAI APIs may experience unpredictable latency and stability,
these factors pose considerable risks for deployment in a production
environment.

Token Usage Limits The token limits in the OpenAI API can be detrimental to
production deployments. Currently, there is a limit of 1000 tokens per minute
for OpenAI and 2000 tokens per minute in Azure GPT-4 endpoints. During our
deployment, even with a mandatory wait time of 10 seconds in between requests,
we still encountered issues of exceeding token limit and requests are dropped
by OpenAPI without processing. As a result, the actual resulting latency of
those requests increased from 300 milliseconds to more than 5 seconds after
subsequent retrying. This situation is not well suited for a production
environment.

Cost As of December 2023, OpenAI’s API for GPT-4, their most advanced and
capable LLM, is at $0.03 per 1K input tokens and $0.06 per 1K output tokens.
Assuming each request consists of 1000 input tokens and 1000 output tokens,
one request costs $0.09. An average traffic of 1000 requests per day would
lead to the cost of $2700 per month, or $32,400 per year. This can easily
scale up to $1,000,000/month when the request load goes above 360,000
requests/day. In other words, if each user only sends five queries/day,
spending $12,000,000/year for openAI API can only support around 71,000 active
users. This is a significant expense for companies, especially for start-ups.

###  III-C Replacing OpenAI with SLMs

To address these limitations, it is imperative for us to investigate the
feasibility of replacing LLMs via cloud APIs with Small Language Models (SLMs)
that can be self-hosted to reduce cost and increase performance
predictability. In the rest of the paper, we set out to answer the following
questions:

  * •

What is the right process for evaluating SLMs to replace LLMs via APIs for a
production use case? How much can this process be automated?

  * •

Are SLMs capable of generating responses that are of similar quality as
OpenAI’s GPT-4?

  * •

How do SLMs in production perform in terms of latency performance and
consistency compared to OpenAI APIs?

  * •

What are the cost trade-offs of self-hosted SLMs vs. OpenAI GPT-4?

Figure 5: Human Evaluation

Figure 6: GPT-4 Based Evaluation

Figure 7: Semantic Similarity Evaluation

##  IV SLAM Methodology and Tool

To answer our research questions, we present the SLaM toolset, a novel
platform to evaluate the performance of SLM vs. LLM across a wide range of
metrics (response quality, latency distribution, availability, etc) in
production use cases. One of the key contributions of SLaM is to automate the
evaluation process, including the setup, response generations, measuring,
benchmarking, and comparison between SLM and LLM models in a production
environment, significantly reducing the manual effort required in such a
process.

###  IV-A SLaM Architecture and Components

SLaM is designed to facilitate a comprehensive analysis of various language
models. It is composed of several components and functionality. Figure 4
illustrates the architecture of SLaM.

  * •

Automating Hosting and Configuration -  SLaM first automatically downloads the
models of interest from the Hugging Face model repository[24] and hosts the
models in the AWS cloud. It sets the models up for experiments and
evaluations.

  * •

Quality Evaluation by Human - Establish baseline For the baseline
measurements, SLaM facilitates crowd-sourcing humans to evaluate model
response quality. For a given prompt, SLaM collects the responses by multiple
models using that prompt and provides a UI that allows a user (human
evaluator) to rate those responses. It then aggregates these ratings from a
pool of users. Users are only provided with responses and not the model
information to ensure their ratings are unbiased. Figure 4 presents the UI
screenshot for users to rate the responses of a model.

  * •

Automated Quality Evaluation To reduce the effort needed for human evaluation,
SLaM provides a comprehensive set of novel, automated approaches to evaluate
response quality. This includes Similarity Scorer and GPT-4-based AI scoring.
Similarity Scorer is designed to provide a quantitative assessment of how
closely the responses from various language models match a benchmark response
using a range of similarity metrics, capturing both semantic and syntactic
elements of the language. The next section will provide more details on our
automated evaluation methodology.

  * •

Automated Performance Evaluation In addition to model response quality, SLaM
also automatically provides performance evaluation (query latency
distribution, etc) of the models. It measures the query latency distribution
over 24 hrs (and longer if needed), sampling multiple queries every hour to
get a diurnal pattern of the query latency distribution. Section VI presents
more details on the performance evaluation.

###  IV-B SLaM Response Quality Evaluation Methodology

To evaluate the efficacy and accuracy of the Small Language Models, SLaM has
designed and included a range of evaluation techniques from human-based
evaluations to various automated AI-driven analyses.

####  IV-B1 Human-based Evaluation

A User is presented with prompts and model-generated responses in UI for
rating the response. To prevent bias, the model that generated the response is
not disclosed to the evaluator. This ensures that ratings are based solely on
the response quality, not on preconceived notions about a model’s
capabilities. Responses are presented in randomized order which further helps
in reducing evaluation bias. The user’s task is to evaluate the quality of
each response based on relevance and accuracy on a scale of 0 to 10. Figure 7
depicts the evaluation process.

####  IV-B2 GPT-4-based evaluation

To automate the quality evaluation process, we have designed multiple
techniques that leverage an LLM model, e.g., GPT-4. Later in Section VI, we
evaluate the efficacy of these GPT-4-based techniques.

  * •

GPT-Scorer: GPT-4 receives the prompt and responses from all language models.
SLaM then prompts GPT-4 to give a rate from 0 to 10 to the responses from
various models. The output of GPT-4 assigns a quality score to each response.
Promp template for this method is:

PROMPT: prompt

RESPONSE:response

Rate the response on a scale of 0 to 10 (0 being the worst and 10 being the
best). Rate the response based on how well it answers the prompt. Reason step
by step and then give a score. Only give a number between 0 and 10.

  * •

GPT-Comparer: This methodology utilizes GPT-4 to analyze and score responses
from various models while comparing the specific model response with a
reference response (GPT-4 response). The prompt template is:

REFERENCE RESPONSE: reference_response

TARGET RESPONSE:target_response

Rate the target response on a scale of 0 to 10 on how much close the target
response to the reference response (0 being the completely different and 10
being the very close in terms of meaning and objective).

Output format

Reason: Reasoning…

Score: Rating

A variation of this is GPT-Comparer with Reasoning: In addition to asking for
just a 0-10 rating, SLaM asks GPT-4 for a rationale behind each score,
providing valuable interpretive feedback.

  * •

GPT-4 Multi-Choice Selector In this variation, GPT-4 is provided with multiple
responses (e.g., four responses from 4 different models) for the same prompt.
GPT-4 then is asked to select the response most appropriate or best fitting.
Prompt template:

PROMPT:

prompt

RESPONSES:

responses

What is the best response? Reason why and give the choice between 1 and
num_responses.

Output format

Reason: Reasoning…

Choice: Choice

####  IV-B3 Similarity score ranking

In addition to LLM-based auto-evaluation metrics, we then designed a
Similarity Scorer to provide a quantitative assessment of how closely the
responses for various language models match a reference response. A set of
similarity metrics are being used to rank the models. These similarity scores
are either based on standard techniques from literature or similarity metrics
leveraging state-of-the-art embeddings. The evaluation results are presented
in Section VI.

  * •

TF-IDF [25]: A standard similarity metric used in information retrieval to
assess how relevant a word is to a response in a collection of responses.

  * •

SBERT [26]: SBERT provides semantically meaningful sentence embeddings,
enabling the computation of cosine similarity between responses.

  * •

USE-QA: Utilizes embeddings from USE-QA model [27] for similarity comparison,
a commonly used metric for assessing responses in a question-answering format.

  * •

GPT4 Embedding: By leveraging the embeddings from GPT-4, this method conducts
a cosine similarity analysis between the responses.

  * •

SEM-BLEU: Combines the semantic analysis of SBERT with the BLEU [28] score’s
syntactic evaluation. The score is generated by calculating the arithmetic
mean of the SBERT and BLEU scores.

The goal of SLaM’s automated quality evaluation methodology is to provide
relevant metrics to assess the response quality of small language models
without human ratings. We have designed a set of methodologies and metrics
towards the goal. Each of these methodologies captures some aspects of the
response quality. We will compare and discuss these metrics in Section VI.

##  V Experimental Setup

###  V-A OpenAI API Setup

For OpenAI APIs, we set the temperature parameter as 0.7 and use the default
for the rest configurations. The temperature parameter influences the
randomness of the generated responses. The connection to the OpenAI API was
established through the ChatOpenAI wrapper, a component of the Langchain
framework, streamlining interactions with the OpenAI API within our
experimental setup.

###  V-B SLMs Setup

Self-hosted AWS instance In our experiments, we used the g4.dn.2xlarge AWS EC2
instance. This instance uses an Intel Xeon processor with 8 virtual CPUs,
operating at 2.5GHz, 32GB of memory, and a single Nvidia T4 GPU with 16GB of
dedicated memory.

Frameworks & Libraries We employed Ollama to handle model loading,
downloading, and inference tasks. Ollama operates as a distinct server
responsible for managing model-related tasks, providing a set of API endpoints
for integration with the application backend. The application backend utilizes
the user graph to perform retrieval operations, initiating requests to Ollama
for inference. The resulting responses were then sent back to the SLaM UI
(built with Streamlit), which presented the outputs to the human reviewers.
Langchain is used to connect Ollama and OpenAI chat models. As illustrated in
Figure 4, this integrated framework facilitated efficient experimentation and
evaluation of SLMs.

###  V-C SLMs evaluated in this work

This experiment focused on widely available open-source language models. We
used the top models from the Hugging Face LLM Leaderboard [29].

The selected models for our experimentation include Starling-lm:7b, Mistral-
instruct:7b, OpenChat:7b, Zephyr:7b, Stablelm-zephyr:3b, Orca-mini:3b,
Vicuna:7b, Orca2:7b, neuralChat:7b, and Llama2-chat:7b. These models were
representative of state-of-the-art performance in the field. We include
several quantized versions of these models as well, for their benefit of
smaller model size and lower memory requirements. Table I shows details about
the models studied.

Model |  # bits |  # Params (B) |  Model Size (GB) |  Memory Usage (MB)
---|---|---|---|---
llama2:7b-chat |  4 |  7 |  3.8 |  6933
llama2:7b-chat-q2_K |  2 |  7 |  2.8 |  5567
llama2:7b-chat-q3_K_L |  3 |  7 |  3.6 |  6271
mistral:7b-instruct |  4 |  7 |  4.1 |  4795
mistral:7b-instruct-q2_K |  2 |  7 |  3.1 |  3897
mistral:7b-instruct-q3_K_L |  3 |  7 |  3.8 |  4601
neural-chat:7b |  4 |  7 |  4.1 |  5183
neural-chat:7b-v3.2-q2_K |  2 |  7 |  3.1 |  4285
neural-chat:7b-v3.2-q3_K_L |  3 |  7 |  3.8 |  4601
openchat:7b-v3.5 |  4 |  7 |  4.1 |  5183
openchat:7b-v3.5-q2_K |  2 |  7 |  3.1 |  4285
openchat:7b-v3.5-q3_K_L |  3 |  7 |  3.8 |  4989
orca-mini:3b |  4 |  3 |  2 |  3089
orca2:7b |  4 |  7 |  3.8 |  6527
orca2:7b-q2_K |  2 |  7 |  2.8 |  5567
orca2:7b-q3_K_L |  3 |  7 |  3.6 |  6271
stablelm-zephyr:3b |  4 |  3 |  1.6 |  3443
stablelm-zephyr:3b-q2_K |  2 |  3 |  1.2 |  3251
stablelm-zephyr:3b-q2_K_L |  3 |  3 |  1.5 |  3507
starling-lm:7b |  4 |  7 |  4.1 |  5183
starling-lm:7b-alpha-q2_K |  2 |  7 |  3.1 |  4285
starling-lm:7b-alpha-q3_K_L |  3 |  3 |  3.8 |  4989
vicuna:7b |  4 |  7 |  3.8 |  5371
vicuna:7b-q2_K |  2 |  7 |  2.8 |  4411
vicuna:7b-q2_K_L |  4 |  3 |  3.6 |  5115
zephyr:7b-beta |  4 |  7 |  4.1 |  5183
zephyr:7b-beta-q2_K |  2 |  7 |  3.1 |  4285
zephyr:7b-beta-q3_K_L |  3 |  7 |  3.8 |  4989
|  |  |  |
TABLE I: SLMs studied in this work.

##  VI Evaluation

We leverage the SLaM methodology and toolset to investigate replacing OpenAI
LLMs with self-hosted SLMs for the production use case described in Section
III. In this section, we present the result of this case study and key
observations and insights.

###  VI-A Can SLMs generate responses as high-quality as OpenAI LLMs?

Figure 8: Response quality of GPT-4 and SLMs, as rated by human reviewers.
Score distribution (boxes) and mean score (line) of each model are shown.

We first investigate the quality of the responses generated by the SLMs and
compare them against OpenAI GPT-4. Figure 8 shows the scores of each model’s
responses rated by the team of human reviewers, following the evaluation
process depicted in Figure 7. The boxes show the distribution of scores of
each model, and the line represents the mean scores of each model. We observe
that, while GPT-4 (1st box on the left) has the highest mean score (5.9),
several SLMs achieved scores very close to GPT-4, including Vicuna:7b-q3
(5.6), Starling-lm:7b-q4 (5.1) and neural-chat:7b-q2 (5.3). This demonstrates
that certain SLMs can indeed generate responses with quality competitive to
OpenAI LLMs. In particular, we find that many quantized models perform
similarly or even better than their base models. This is particularly
encouraging for applying SLMs in production because of the smaller size of
quantized LLMs.

To demonstrate the response quality of SLMs, we showcase below sample
responses of the daily peptalk feature from GPT-4 and two of the highest-rated
SLMs, Starling-LM:7B and Vicua:7b-q3. The responses have been updated
minimally to comply with double-blind.

GPT-4 – Good morning! You had a productive day last time, accomplishing both
your daily rituals. Today, we’re starting strong with planning the day, a task
that aligns perfectly with your goal to get back into your daily routine.
Focused tasks such as stress testing the openAI assistants API, preparing a
demo for the RAG prototype, and improving the pep talk use cases, will help
you dive deep into local LLMs and get the pep talk AI feature ready for
productization. Don’t forget your self-care tasks as well such as scheduling
your orthopedic appointment, which addresses your goal to take care of your
lower back issues. Let’s make today a great one!

starling-lm:7b – Great job on completing the dailies on November 10th! Today,
you have a full schedule ahead of you with key activities such as stress
testing the OpenAI assistant API, preparing a demo for the RAG prototype,
quantization and LORA of a model, and trying out GPT-4 Turbo. To align with
your goals for this week, ensure that improving the pep talk use cases remains
a priority and consider scheduling your orthopedic appointment sooner rather
than later. Your daily rituals like planning, lifting, and medium scanning are
crucial to maintaining good habits and staying focused. While some tasks may
not be directly related to your goals, they contribute to overall progress and
personal growth. Let’s make today a productive one and keep moving forward
with determination and enthusiasm!

vicuna:7b-q3 – Good morning! Based on your previous log-in and completed
tasks, you were able to complete the daily meetings, which is fantastic.
Today’s focused tasks include planning for the day, stress testing the openAI
assistants API, preparing a demo for the RAG prototype, trying quantization
and LORA a model, trying out GPT-4-turbo and DALL-E 3, scheduling acura
service, improving the pep talk use cases, sending the consultation report,
working on the scallop project, and scheduling an orthopedic appointment. Your
rituals for today include Daily, Zero Inbox, Review Elastic Logs, and lifting
weights. These recurring tasks help you build good habits and maintain your
workflow responsibilities. In addition, you set a few goals for yourself this
week, which are to get the pep talk AI feature ready for productization, get
back into your daily routine and workflow, take care of lower back issues, and
dive deeper into local LLMs. Based on your focused tasks and rituals, it seems
that you have a great start to your day ahead. You are taking care of some
important projects and maintaining good habits. Good luck with achieving your
goals this week!

In addition to the top-performing models, the majority of the SLMs achieve
similar performance and have scores close to the top models. This suggests
that most of the SLMs studied can generate responses with quality close to
OpenAI LLMs. On the other hand, we do observe that a small subset of SLMs
exhibit clearly inferior response quality compared to the rest of the group.
Specifically, the orca2:7b models (base and quantized) and stablelm-
zephyr:3b-q3 scored significantly worse than the rest.

Key Observation: Many SLMs, including their quantized versions, are able to
generate responses of similar quality as OpenAI GPT-4, while a small set of
SLMs are the clear low-performing options compared to the rest of the group.

###  VI-B Automated Evaluation via GPT-4

Human evaluation can be labor-intensive and time-consuming and are not always
viable. We explore the feasibility of leveraging GPT-4 to automate the
response quality scoring. We compare three ”GPT Evaluators”, including scoring
(same as human), comparison, and multiple choice. Details of these evaluators
are described in Section IV-B2

Figure 9: Distribution of scores given to 10 samples of responses from each
model by the GPT-4 scorer using reasoning evaluation methods against models.
Figure 10: Jaccard index of bottom 10 SLMs ranked by each automated evaluation
method compared to human evaluation.

GPT-Scorer Figure 9 shows the scores of response from each model as rated by a
GPT-4 model with the instruction prompt shown in Section IV-B2. The boxes show
the distribution of scores, and the two lines show the mean score of human
evaluation and GPT-Scorer. We observe that GPT-Scorer evaluation follows a
similar trend as the human evaluation. Furthermore, it is able to identify the
same subset of inferior models as the human evaluation. This suggests that
prompting GPT-4 to score model response quality in the same style as human
evaluation provides conclusions similar to those of human evaluation.

Key Observation: Scoring responses with prompting using GPT-4 arrives at
similar evaluation outcome as human evaluation.

GPT Evaluator and Similarity Scoring We now compare across the automated
evaluator, including the GPT-based evaluators and similarity scoring method.
As discussed in Section VI-A, many SLMs provide competitive response quality
to OpenAI while a small subset of SLMs appear clearly inferior. Being able to
quickly identify these models and remove them from consideration can greatly
accelerate the SLM model evaluation and selection process. Therefore, we focus
on characterizing how well each auto-evaluation method can identify low-
performing SLMs. Figure 10 shows the Jaccard index of each automated
evaluation method (5 similarity scoring and 4 GPT-based) for the bottom 10
SLMs in their ranking compared to the human evaluation. A higher Jaccard index
for an automated evaluation method indicates that it ranks the model more
similar to that of human rankings. We observe that sem-bleu has the highest
Jaccard index, meaning that its judgment of low-performing models aligns with
human ranking the most. Sem-bleu, as described in Section IV-B3, combines
sentence-level (SBERT) and token-level (BLEU) similarity.

Key Observation: Automated evaluation methods can effectively identify low-
performing models, accelerating the process of SLMs evaluation. Specifically,
Sem-bleu provides rankings closest to human evaluation.

Figure 11: Latency per token of SLMs and OpenAI GPT-4 API. The distribution of
10 requests is shown here.

Figure 12: Latency per token of SLMs and OpenAI GPT-4 API. The distribution of
10 requests is shown here.

Figure 13: Number of tokens generated by SLMs and OpenAI GPT-4 API.
Distribution of 10 requests with the same prompt shown here.

###  VI-C Are SLMs faster than openAI APIs?

Per-request Latency – Figure 13 shows the per-request latency of the SLMs and
OpenAI GPT-4. Distributions of 10 requests with the same input are shown here.
While GPT-4 (1st row) has the lowest per-request latency, most SLMs provide
competitive latency performance. Specifically, the mean request latency of
mistral:7b-instruct, orca-mini:3b and stablelm-zephyr:3b are within sub 1
second of GPT-4’s performance.

Per-token Latency – Language models generate the output response one token at
a time, and therefore, the end-to-end request latency is influenced by the
number of tokens the model generates. To further understand the performance
characteristics of these SLMs, we characterize the number of tokens in the
response generated by each model (Figure 13) and the per-token latency of each
model (Figure 13). Figure 13 shows that the Orca 2 models and StableLM-
zephyr:3b-q2 have large variations in their response length. This explains the
wide distribution of per-request latency for these models observed in Figure
13. GPT-4’s response length has a much lower variance than the SLMs,
indicating its responses more consistently follow the prompt’s instructions.
Furthermore, Figure 13 shows that many SLMs, such as StableLM-zephyr-3b,
mistral-7b, and starlingLM, outperform OpenAI GPT-4 on per-token latency.

Key Observation: Self-hosted SLMs can achieve similar or better latency
performance as OpenAI GPT-4 in per token generation. GPT-4 is more consistent
in following response length control instructions in the prompt, providing
more consistent performance on the request level, while SLMs have a wider
distribution of per-request latency.

###  VI-D Are self-hosted SLMs more reliable than openAI?

Figure 14: Latency distribution of OpenAI’s API over the course of a 24-hour
period. Figure 15: Latency distribution of OpenAI’s GPT-4 API, Starling-lm:7b,
and Orca-mini:3b over the course of a 24-Hour Period.

We characterize the variability in the performance of OpenAI APIs and self-
hosting SLMs at different times of the day. Figure 14 shows how the request
latency of OpenAI API varies over the course of 24 hours. Ten requests are
measured at each hour, and their distributions are shown as box plots. The
same input and request parameters are used across all requests. We observe
significant variations in request latency for OpenAI at different times of the
day, where per request latency ranges from 3.4 seconds to 8.6 seconds. This
large variability can present a significant challenge in production settings.

Similarly, Figure 15 shows the latency variability over 24 hours of four
selected SLMs and compares them against OpenAI GPT-4. StarlingLMs are selected
because their response quality are among the best of the SLMs and Orca-mini-3b
is selected because its small size and yet strong results in terms of response
quality. When measuring the SLMs performance, we warm up the machine with 10
requests first and we measure the latency of the subsequent 10 requests. We
observe that the self-hosted SLMs have a significantly more consistent request
latency over the course of the 24 hours and they also have a narrower
distribution from request to request within the same hour.

Key Observation: Self-hosting SLMs can significantly reduce performance
variability in production compared to OpenAI APIs.

###  VI-E Are SLMs more cost-effective than OpenAI’s cloud service?

We compute the throughput of each model when running on an AWS instance at 80%
utilization. The per-1K token results are shown in Figure 16. The
corresponding OpenAI cost is $0.09. Figure 17 shows the cost reduction
resulting from running the SLMs instead of the OpenAI model. We picked 80%
utilization to be conservative in our comparison. The use of autoscaling
monitors utilization over time and automatically adds compute resources as
needed, but its use also implies that an individual node will never be fully
utilized. If the number of inferences is low, the OpenAI service could have a
cost advantage. However, the EC2 instance could schedule containers unrelated
to inferencing and amortize its cost in that case. Overall, we see a cost
reduction of 5x to 23x, depending on the model used.

Figure 16: Cost per 1K token generated on each model. GPT-4 incurs a cost of
$0.03 per input token and $0.06 per completion token, whereas the self-hosted
SLM models only incur costs associated with AWS EC2. We assume 80% utilization
of the node for cost comparisons.  Figure 17: Cost reduction of each SLM
against the OpenAI’s GPT-4 running on a g4dn.xlarge node in AWS at 80%
utilization.

##  VII Related Works

We discuss prior work that addresses the challenges associated with large
language models (LLMs) and ones that introduce innovative hardware solutions
to accelerate them.

Several work has investigated the performance of machine learning models
focusing on the training aspects on various hardware designs [30, 31, 32].

XRBench[33] is a benchmarking suite for evaluating machine learning workloads
related to extended reality. It assesses model dependency and concurrency for
testing the real-time use of multi-model multi-task workloads. The concept of
multi-DNN workloads is further developed by [34] which introduces sub-
accelerators to efficiently manage multiple subtasks concurrently. Xu et al.
[35] discuss the challenges of tuning large language models (LLMs) like GPT-3
and GPT-4 due to their immense sizes and the inaccessibility of their weights.
Their proposed Super In-Context Learning (SuperICL) enables black-box LLMs to
collaborate with locally finetuned smaller models. Liang, et. al[36].
demonstrate that LLMs can assist in developing quantum computing architectures
on par with sophisticated quantum architecture search.

There are system design innovations that support and accelerate
LLMs[37][38][39]. For example, both FLAT [40] and SPRINT [41] address the
computational challenges of attention mechanisms in machine learning. FLAT
introduces tailored dataflow optimization for attention mechanisms without
altering their functionality. SPRINT is an accelerator that leverages the
inherent parallelism of ReRAM crossbar arrays to compute attention scores
approximately, and SUGAR is a Self-Adaptive Reconfigurable Array (SARA) with
an integrated recommendation neural network (ADAPTNET) that enables runtime
reconfiguration for optimized performance in accelerator architectures[42].
Wang et al [43] propose a novel technique to reduce data communication
overheads in large deep learning models by overlapping communication with
computation. This technique decomposes an identified original communication
collective and the dependent computation operation into a sequence of fine-
grained operations. Rouhani et al. [44] introduce Block Data Representations
(BDR), a framework for exploring and evaluating a wide spectrum of narrow-
precision formats for deep learning. [45] presents a framework to evaluate
hardware designs for accelerating LLM workloads.

There are also architectural innovations to accelerate ML and LLM models.
VEGETA[46] is an ISA extension for accelerating the computation of sparse
matrix multiplication on the CPU for improving DNN workloads. Norm et al [47]
discuss the design and implementation of Google’s domain-specific architecture
(DSA) for machine learning models, including LLMs. They introduce SparseCores,
dataflow processors that accelerate models that rely on embeddings. The
performance, scalability, efficiency, and availability of TPU v4 make it an
ideal vehicle for LLMs. Also, Seongmin et al [48] presents a multi-FPGA
acceleration appliance that executes GPT-2 model inference end-to-end with low
latency and high throughput. Cong et al. [49] proposes OliVe, an
algorithm/architecture co-designed solution that adopts an outlier-victim pair
(OVP) quantization that can be efficiently integrated into existing hardware
accelerators using systolic arrays and tensor cores to achieve speedup and
energy reduction with superior model accuracy.

##  VIII Conclusion

We show that while LLMs, like GPT-4 have excellent breadth of capabilities,
their power can be approximated with smaller, faster, and cheaper models for
domain-specific tasks. We evaluate the trade-offs on the part of a commercial
application that generates a daily ”pep talk” to its user based on their past
behavior and future tasks. While GPT-4 achieves the highest accuracy across a
broad range of models, as judged by a human panel, most small language models
come close to it at one-fifth to one-twentythird of the cost (when running in
AWS). The SLMs also benefit from more predictable performance and lower
latency. Our evaluation is facilitated by an automated methodology called
SLaM, which not only collects data from humans and from multiple LLM-based
auto-evaluation techniques but also compares them with state-of-the-art
embeddings-based similarity metrics.

## References

  * [1] “OpenAI blog: ChatGPT,” https://openai.com/blog/chatgpt, 2023.
  * [2] M. Dowling and B. Lucey, “Chatgpt for (finance) research: The bananarama conjecture,” _Finance Research Letters_ , vol. 53, p. 103662, 2023.
  * [3] OpenAI. (Year Accessed) Introducing ChatGPT and Whisper APIs. [Online]. Available: https://openai.com/blog/introducing-chatgpt-and-whisper-apis
  * [4] The Register. (2023) Outage hits OpenAI’s ChatGPT as servers stopped by ’Claude’ glitch. [Online]. Available: https://www.theregister.com/2023/11/08/outage_chatgpt_openai_claude/
  * [5] H. Shen, H. Chang, B. Dong, Y. Luo, and H. Meng, “Efficient llm inference on cpus,” _arXiv preprint arXiv:2311.00502_ , 2023.
  * [6] H. Chang, H. Shen, Y. Cai, X. Ye, Z. Xu, W. Cheng, K. Lv, W. Zhang, Y. Lu, and H. Guo, “Effective quantization for diffusion models on cpus,” _arXiv preprint arXiv:2311.16133_ , 2023.
  * [7] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier _et al._ , “Mistral 7b,” _arXiv preprint arXiv:2310.06825_ , 2023.
  * [8] G. Wang, S. Cheng, X. Zhan, X. Li, S. Song, and Y. Liu, “Openchat: Advancing open-source language models with mixed-quality data,” _arXiv preprint arXiv:2309.11235_ , 2023.
  * [9] S. Mukherjee, A. Mitra, G. Jawahar, S. Agarwal, H. Palangi, and A. Awadallah, “Orca: Progressive learning from complex explanation traces of gpt-4,” _arXiv preprint arXiv:2306.02707_ , 2023.
  * [10] Y. Chang and at el., “A survey on evaluation of large language models,” 2023.
  * [11] M. Ollivier and at el., “A deeper dive into chatgpt: history, use and future perspectives for orthopaedic research,” _Knee Surgery, Sports Traumatology, Arthroscopy: Official Journal of the ESSKA_ , vol. 31, no. 4, pp. 1190–1192, 2023.
  * [12] A. Vaswani and at el., “Attention is all you need,” 2023.
  * [13] A. Radford and at el., “Language models are unsupervised multitask learners,” 2019. [Online]. Available: https://api.semanticscholar.org/CorpusID:160025533
  * [14] M. Shoeybi and at el., “Megatron-lm: Training multi-billion parameter language models using model parallelism,” 2020.
  * [15] OpenAI, “Gpt-4 technical report,” _ArXiv_ , vol. abs/2303.08774, 2023. [Online]. Available: https://arxiv.org/abs/2303.08774
  * [16] “Join us for openai’s first developer conference,” 2023. [Online]. Available: https://openai.com/blog/announcing-openai-devday
  * [17] Z. L. et al., “Llm-qat: Data-free quantization aware training for large language models,” 2023.
  * [18] J. Liu, R. Gong, X. Wei, Z. Dong, J. Cai, and B. Zhuang, “Qllm: Accurate and efficient low-bitwidth quantization for large language models,” 2023.
  * [19] L. AI, “Langchain: Building applications with llms through composability,” https://github.com/langchain-ai/langchain, 2023, accessed: Dec. 15, 2023.
  * [20] “Emnlp: Prompt engineering is the new feature engineering,” 2022. [Online]. Available: https://www.amazon.science/blog/emnlp-prompt-engineering-is-the-new-feature-engineering
  * [21] P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig, “Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing,” _ACM Comput. Surv._ , vol. 55, no. 9, jan 2023. [Online]. Available: https://doi.org/10.1145/3560815
  * [22] “Gpt-3 powers the next generation of apps,” 2023. [Online]. Available: https://openai.com/blog/gpt-3-apps
  * [23] “Openai: why are the api calls so slow? when will it be fixed,” 2023. [Online]. Available: https://community.openai.com/t/openai-why-are-the-api-calls-so-slow-when-will-it-be-fixed/148339/68?page=4
  * [24] HuggingFace, “Hugging Face Models,” https://huggingface.co/models, 2023, accessed: Dec. 15, 2023.
  * [25] C. D. Manning, _An introduction to information retrieval_. Cambridge university press, 2009.
  * [26] N. Reimers and I. Gurevych, “Sentence-bert: Sentence embeddings using siamese bert-networks,” _CoRR_ , vol. abs/1908.10084, 2019. [Online]. Available: http://arxiv.org/abs/1908.10084
  * [27] D. Cer, Y. Yang, S.-y. Kong, N. Hua, N. Limtiaco, R. S. John, N. Constant, M. Guajardo-Cespedes, S. Yuan, C. Tar _et al._ , “Universal sentence encoder,” _arXiv preprint arXiv:1803.11175_ , 2018.
  * [28] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , 2002, pp. 311–318.
  * [29] HuggingFaceH4, “Open LLM Leaderboard,” https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard, 2023, accessed: Dec. 15, 2023.
  * [30] W. Won, T. Heo, S. Rashidi, S. Sridharan, S. Srinivasan, and T. Krishna, “Astra-sim2.0: Modeling hierarchical networks and disaggregated systems for large-model training at scale,” in _IEEE International Symposium on Performance Analysis of Systems and Software, ISPASS 2023, Raleigh, NC, USA, April 23-25, 2023_. IEEE, 2023, pp. 283–294. [Online]. Available: https://doi.org/10.1109/ISPASS57527.2023.00035
  * [31] D. Moolchandani, J. Kundu, F. Ruelens, P. Vrancx, T. Evenblij, and M. Perumkunnil, “Amped: An analytical model for performance in distributed training of transformers,” in _IEEE International Symposium on Performance Analysis of Systems and Software, ISPASS 2023, Raleigh, NC, USA, April 23-25, 2023_. IEEE, 2023, pp. 306–315. [Online]. Available: https://doi.org/10.1109/ISPASS57527.2023.00037
  * [32] J. Gómez-Luna, Y. Guo, S. Brocard, J. Legriel, R. Cimadomo, G. F. Oliveira, G. Singh, and O. Mutlu, “Evaluating machine learningworkloads on memory-centric computing systems,” in _2023 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS)_ , 2023, pp. 35–49.
  * [33] H. Kwon, K. Nair, J. Seo, J. Yik, D. Mohapatra, D. Zhan, J. Song, P. Capak, P. Zhang, P. Vajda _et al._ , “Xrbench: An extended reality (xr) machine learning benchmark suite for the metaverse,” _Proceedings of Machine Learning and Systems_ , vol. 5, 2023.
  * [34] H. Kwon, L. Lai, M. Pellauer, T. Krishna, Y.-H. Chen, and V. Chandra, “Heterogeneous dataflow accelerators for multi-dnn workloads,” in _2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA)_. IEEE, 2021, pp. 71–83.
  * [35] C. Xu and at el., “Small models are valuable plug-ins for large language models,” _ArXiv_ , vol. abs/2305.08848, 2023.
  * [36] Z. Liang, J. Cheng, R. Yang, H. Ren, Z. Song, D. Wu, X. Qian, T. Li, and Y. Shi, “Unleashing the potential of llms for quantum computing: A study in quantum architecture design,” _arXiv preprint arXiv:2307.08191_ , 2023.
  * [37] Z. Wu, L. Y. Dai, A. Novick, M. Glick, Z. Zhu, S. Rumley, G. Michelogiannakis, J. Shalf, and K. Bergman, “Peta-scale embedded photonics architecture for distributed deep learning applications,” _Journal of Lightwave Technology_ , 2023.
  * [38] F. Blanaru, A. Stratikopoulos, J. Fumero, and C. Kotselidis, “Enabling pipeline parallelism in heterogeneous managed runtime environments via batch processing,” in _Proceedings of the 18th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments_ , 2022, pp. 58–71.
  * [39] J. Song, J. Yim, J. Jung, H. Jang, H.-J. Kim, Y. Kim, and J. Lee, “Optimus-cc: Efficient large nlp model training with 3d parallelism aware communication compression,” in _Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2_ , 2023, pp. 560–573.
  * [40] S.-C. Kao, S. Subramanian, G. Agrawal, A. Yazdanbakhsh, and T. Krishna, “Flat: An optimized dataflow for mitigating attention bottlenecks,” in _Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2_ , 2023, pp. 295–310.
  * [41] A. Yazdanbakhsh and et al., “Sparse attention acceleration with synergistic in-memory pruning and on-chip recomputation,” in _2022 55th IEEE/ACM International Symposium on Microarchitecture (MICRO)_. Los Alamitos, CA, USA: IEEE Computer Society, oct 2022, pp. 744–762. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/MICRO56248.2022.00059
  * [42] A. Samajdar, M. Pellauer, and T. Krishna, “Self-adaptive reconfigurable arrays (sara): Using ml to assist scaling gemm acceleration,” _arXiv preprint arXiv:2101.04799_ , 2021.
  * [43] S. Wang and at el., “Overlap communication with dependent computation via decomposition in large deep learning models,” in _Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1_ , ser. ASPLOS 2023. Association for Computing Machinery, 2022, p. 93–106.
  * [44] D. Rouhani and et al., “With shared microexponents, a little shifting goes a long way,” in _Proceedings of the 50th Annual International Symposium on Computer Architecture_ , ser. ISCA ’23. New York, NY, USA: Association for Computing Machinery, 2023. [Online]. Available: https://doi.org/10.1145/3579371.3589351
  * [45] H. Zhang, A. Ning, R. Prabhakar, and D. Wentzlaff, “A hardware evaluation framework for large language model inference,” 2023.
  * [46] G. Jeong, S. Damani, A. R. Bambhaniya, E. Qin, C. J. Hughes, S. Subramoney, H. Kim, and T. Krishna, “Vegeta: Vertically-integrated extensions for sparse/dense gemm tile acceleration on cpus,” in _2023 IEEE International Symposium on High-Performance Computer Architecture (HPCA)_. IEEE, 2023, pp. 259–272.
  * [47] N. Jouppi and et al., “Tpu v4: An optically reconfigurable supercomputer for machine learning with hardware support for embeddings,” in _Proceedings of the 50th Annual International Symposium on Computer Architecture_ , ser. ISCA ’23. Association for Computing Machinery, 2023.
  * [48] S. Hong and at el., “Dfx: A low-latency multi-fpga appliance for accelerating transformer-based text generation,” in _2022 IEEE Hot Chips 34 Symposium (HCS)_ , 2022, pp. 1–17.
  * [49] C. Guo and et al., “Olive: Accelerating large language models via hardware-friendly outlier-victim pair quantization,” in _Proceedings of the 50th Annual International Symposium on Computer Architecture_ , ser. ISCA ’23. ACM, Jun. 2023. [Online]. Available: http://dx.doi.org/10.1145/3579371.3589038