# Text me the data: Generating Ground Pressure Sequence from Textual
Descriptions for HAR
††thanks: The research was supported by the BMBF (German Federal Ministry of
Education and Research) in the project VidGenSense (01IW21003).

Lala Shakti Swarup Ray  DFKI
Kaiserslautern, Germany
<EMAIL_ADDRESS>Bo Zhou  RPTU Kaiserslautern-Landau & DFKI
Kaiserslautern, Germany
<EMAIL_ADDRESS>Sungho Suh  RPTU Kaiserslautern-Landau & DFKI
Kaiserslautern, Germany
<EMAIL_ADDRESS>Lars Krupp  RPTU Kaiserslautern-Landau & DFKI
Kaiserslautern, Germany
<EMAIL_ADDRESS>Vitor Fortes Rey  RPTU Kaiserslautern-Landau & DFKI
Kaiserslautern, Germany
<EMAIL_ADDRESS>Paul Lukowicz  RPTU Kaiserslautern-Landau & DFKI
Kaiserslautern, Germany
<EMAIL_ADDRESS>
###### Abstract

In human activity recognition (HAR), the availability of substantial ground
truth is necessary for training efficient models. However, acquiring ground
pressure data through physical sensors itself can be cost-prohibitive, time-
consuming. To address this critical need, we introduce Text-to-Pressure (T2P),
a framework designed to generate extensive ground pressure sequences from
textual descriptions of human activities using deep learning techniques. We
show that the combination of vector quantization of sensor data along with
simple text conditioned auto regressive strategy allows us to obtain high-
quality generated pressure sequences from textual descriptions with the help
of discrete latent correlation between text and pressure maps. We achieved
comparable performance on the consistency between text and generated motion
with an R squared value of 0.722, Masked R squared value of 0.892, and FID
score of 1.83. Additionally, we trained a HAR model with the the synthesized
data and evaluated it on pressure dynamics collected by a real pressure sensor
which is on par with a model trained on only real data. Combining both real
and synthesized training data increases the overall macro F1 score by 5.9
percent.

###### Index Terms:

HAR, Generative model, GPT, Pressure Sensor, Synthetic dataset

##  I Introduction

Ground pressure data collected from humans plays an indispensable role in a
wide array of applications, spanning from Human Activity Recognition (HAR) [1]
to gait analysis [2], healthcare monitoring [3], rehabilitation programs [4],
robotics development [5], and human-robot interaction [6]. However, a
significant challenge arises when it comes to collecting ground pressure data,
as it traditionally relies on physical sensors such as pressure pads, smart
mats, and pressure insoles. These conventional sensing devices present
inherent limitations related to portability, versatility and durability that
makes the data collection process both time-consuming and resource-intensive.
This obstacle becomes particularly daunting when striving to amass a diverse
and substantial dataset encompassing various users and a multitude of activity
variations.

To address this challenge, researchers have explored innovative approaches to
synthesize pressure data from readily available modalities like RGB images [7]
and depth images [8] but those models need real data from another modality as
input making the whole purpose redundant. Another intriguing approach involves
using activity labels themselves to generate pressure sensor data. Recent
advancements in generative models, especially within the realm of
transformers, have revolutionized the field of data generation from textual
descriptions. Large Language Models (LLMs) like CLIP [9] and the GPT series
[10] have emerged as game-changers. These LLMs, equipped with billions of
parameters, exhibit remarkable capabilities in comprehending and generating
text. Importantly, they can transform the text into various data formats,
including images [11] and 3D pose data [12]. This transformative technology
empowers researchers and practitioners by enabling the creation of diverse and
high-quality datasets directly from text, expanding the boundaries of what AI
can accomplish. In the context of ground pressure sensors, LLMs offer the
potential to convert textual descriptions of human activities into synthetic
ground pressure data. This approach not only addresses the challenges of slow
and costly data collection but also serves as valuable training material for
models like T2P. By adopting this method, T2P can learn to generate highly
accurate ground pressure sequences from text, significantly enhancing their
effectiveness in HAR tasks.

To summarize, this paper makes the following contributions:

  * •

A method of sensor data synthesis from texts that enables training a sensor-
based HAR system (dynamic planar pressure) that outperforms a model trained
with real data, validated with a real sensor system.

  * •

A large synthetic dataset consisting of 240 activity descriptions, with 720
unique pressure map sequences of 86,400 frames and a real dataset with eight
activity descriptions from ten individuals, totaled 16,256 pressure frames.

##  II Approach

###  II-A Dataset Synthesis Pipeline

Figure 1: Data generation pipeline that combines GPT-4, T2M-GPT, SinMDM,
joint2SMPL, and PresSim to generate pressure map sequences from activity
description for training T2P.

To generate the extensive dataset encompassing four distinct activity
categories – basic, dance, yoga, and workout, all revolving around lower body
movements and semi-static actions, a meticulous and intricate procedure was
employed. This comprehensive process hinged on the utilization of state of the
art technologies, including GPT-4 [13], T2M-GPT [14], SinMDM [15], joint2smpl
[16], and PresSim [7], as illustrated in Figure 1.

The primary rationale behind opting for synthesized motion as opposed to
utilizing pre-existing motion datasets such as HumanML3D [17] or the KIT-ML
dataset [18] is rooted in the unique nature of the target pressure data we
aimed to generate. This data pertains to an 80x28 array of pressure-sensing
mats, which permits only limited movement across its surface. Straying from
the defined surface wouldn’t capture the desired pressure data accurately.

The initial step involved the manual collection of sixty distinct motion types
associated with each activity class. Subsequently, we employed prompt
engineering with GPT-4 to deconstruct these activities into their elemental
motions, stripping away any external environmental context. This was achieved
by assigning ”role” and ”content” and providing T2M-GPT examples as reference
prompts. For instance, an activity like ”Push-ups” would be transformed into
an activity description like ”A person assumed an all-fours position, facing
downwards, then lowered their upper body toward the ground and lifted it back
up using only their arms.”

Following this, T2M-GPT[14] was employed to translate these activity
descriptions into 3D kinematic pose sequences. Each pose sequence comprised 22
joints and 120 frames for each motion. To maintain the quality of the
generated motions, they underwent manual inspection to identify and eliminate
any inaccuracies or errors.

To introduce diversity and variability into the dataset, SinMDM [15] is
harnessed to generate three unique motion sequences for each activity, with a
random number of frames, resulting in a total of 720 3D pose sequences
spanning 86400 frames. This approach significantly enriched the dataset by
introducing complexity and diversity.

To prepare these generated 3D poses for use with PresSim, they needed to be
converted into volumetric poses in SMPL format [19]. To achieve this, we
employed the joint2smpl technique [16], which expertly converted the 3D pose
sequences into the desired SMPL format.Finally, PresSim was utilized to
transform these volumetric poses into 720 distinct pressure sequences. This
comprehensive process yielded a dataset that included 240 unique activity
descriptions and a total of 86,400 frames.

###  II-B Validation Data Collection

We also collected a real dataset to validate our model which is explained in
detail in Section III. We used a Fitness-mat from Sensing.Tex [20] with a
$80\times 28$ sensor grid having a sensing area of $560\times 1680$ $mm$ with
separate sensor area of $12\times 16$ $mm$ with measuring range of 0-5000
millimeters of mercury (mmHg). The hardware consists of a non-elastic, non-
slippery yoga mat made with Thermoplastic Polyolefins(TPE) polymers having a
range of operation from 5% – 70% humidity and 15${}^{\circ}C$ – 45
${}^{\circ}C$. The detailed data statistics are provided in Table I.

TABLE I:  Data statistics including subject mass ($kilogram$), height ($centimeter$) and gender. Subject | Mass ($kg$) | Height ($cm$) | Gender | Frames
---|---|---|---|---
1 | 57.7 | 162 | Female | 802
2 | 94.7 | 172 | Male | 1247
3 | 76.3 | 183 | Male | 1045
4 | 74.3 | 175 | Male | 1031
5 | 53.3 | 153 | Female | 743
6 | 62.3 | 173 | Male | 865
7 | 77.2 | 184 | Male | 1069
8 | 72.2 | 178 | Male | 989
9 | 60.2 | 156 | Male | 839
10 | 67.3 | 178 | Female | 932
Total | - | - | - | 16256

###  II-C Architecture

Figure 2: Illustration of the T2P architecture that utilizes Variational auto-
encoder (VQ-VAE) and auto-regressive transformer (T2P) module to train the
model.

#### Vector Quantized Variational Autoencoder (VQ-VAE)

Inspired by T2M-GPT, we discretized the Pressure map sequence by employing a
VQ-VAE given in Figure 2 to create a dictionary for the pressure sequences.
This process facilitates the discrete representation of continuous pressure
data, enhancing its suitability for modeling. The resulting dictionary serves
as a compact and structured representation of intricate pressure information.
The autoencoder follows a ResNet-inspired simple 1D CNN blocks architecture
with residual blocks (RB) in both encoder and decoder, maintaining the same
input and output size for the Pressure map sequence. ReLu activation is
utilized in the network, and a vector quantizer is inserted between the
encoder and decoder to convert extracted features into vector sequences.
Training involves a combination of reconstruction loss ($L_{\text{r}}$) and
quantization loss ($L_{\text{q}}$). To enhance the robustness of our
dictionary, scheduled annealing is implemented. Scheduled annealing
dynamically adjusts the weights for reconstruction ($w_{\text{r}}$) and
quantization ($w_{\text{q}}$) throughout training. The weights are defined as
functions of training step ($t$) and contribute to the total loss ($L(t)$),
balancing reconstruction and quantization losses.

$L(t)=w_{\text{r}}(t)\cdot L_{\text{r}}(t)+w_{\text{q}}(t)\cdot
L_{\text{q}}(t)$

Additionally, we employ exponential moving average (EMA) to stabilize training
and enhance the quality of learned representations in the dictionary. EMA
prevents rapid changes in the dictionary, ensuring stability and meaningful
feature capture from the input data, thereby contributing to improved training
and generative capabilities.

$\text{EMA}_{t}=(1-\alpha)\cdot\text{EMA}_{t-1}+\alpha\cdot\text{Dictionary}_{t}$

Here, $\text{EMA}_{t}$ represents the Exponential Moving Average at time $t$,
$\alpha$ is the smoothing factor for EMA (computed as $\alpha=\frac{2}{N+1}$
with $N$ as a hyper-parameter), and $\text{Dictionary}_{t}$ denotes the
dictionary entries at training step $t$.

#### T2P

In T2P, we harnessed the power of a pre-trained CLIP encoder to derive
meaningful text embeddings from natural language text, denoted as
$E_{\text{CLIP}}(T)$, forming the foundation for our subsequent modeling
efforts. The pressure sequences, denoted as $P=[p_{1},p_{2},\ldots,p_{T}]$,
were then transformed into a sequence of indices,
$P^{\prime}=[p^{\prime}_{1},p^{\prime}_{2},\ldots,p^{\prime}_{T/l},\text{End}]$,
where $p^{\prime}_{i}\in[1,2,\ldots,p^{\prime}_{T/l}]$. Here, $T/l$ indicates
the division of the original sequence into segments, each of length $l$, with
indices $p^{\prime}_{i}$ drawn from a pre-learned dictionary. Mathematically,
this conversion process is represented as:

$P\rightarrow
P^{\prime}=[p^{\prime}_{1},p^{\prime}_{2},\ldots,p^{\prime}_{T/l},\text{End}]$

The obtained indices $p^{\prime}_{i}$, coupled with the text conditioning
$E_{\text{CLIP}}(T)$, serve as crucial inputs to our auto-regressive
transformer to generate a sequence of vectors. At each step, the model
predicts the next index based on prior predictions and the text conditioning,
with the ”End” token signaling the sequence termination. Mathematically, the
auto-regressive sequence generation process for synthesized pressure vectors
$P^{\prime\prime}=[p^{\prime\prime}_{1},p^{\prime\prime}_{2},\ldots,\text{End}]$,
given text conditioning $E_{\text{CLIP}}(T)$, is expressed as:

$E_{\text{transformer}}(P^{\prime},E_{\text{CLIP}}(T))\rightarrow[p^{\prime\prime}_{1},p^{\prime\prime}_{2},\ldots,\text{End}]$

The resulting vector sequences are then transformed back into pressure
sequences using the pre-trained VQ-VAE decoder.

##  III Experimental Results

#### Implementation details

The VQ-VAE model trained on all 86,400 pressure frames using 10-fold cross-
validation. Early stopping and a decaying learning rate scheduler ($1e^{-3}$)
were employed to prevent overfitting. For T2M, the dataset was divided into
66,200 training sets, 5,120 test sets, and 15,080 validation sets. Both
models, implemented in PyTorch 2.0, were trained on a cluster with 128GB
memory, four processors, and two Nvidia RTX A6000 graphics cards.

#### Qualitative analysis

Figure 3: visualization of T2P results for two activities walk back and sit.

In Figure 3, we showcase generated Pressure sequences paired with
corresponding text prompts. GPT-4 prompt engineering is employed to convert
labels into text prompts, which are then used as input for T2P to generate the
pressure sequence. While acknowledging that our method may not achieve a
perfect representation of the underlying motion details, it effectively
communicates the original activity label through the resulting pressure
sequences.

#### Quantitative analysis

We used FID metrics that measures the similarity between real and generated
pressure maps, and binarized R2 that assesses the accurateness of point of
contact in our generated data.In our ablation studies, the VQ-VAE was enhanced
with residual blocks (RB), scheduled annealing (SA), and exponential moving
averages (EMA), resulting in a notable 1.802 improvement in the FID score II.
Comparisons were made with a baseline T2P model that uses extracted features
instead of quantized vectors to train the autorgressive transformer, showing
the latter’s effectiveness despite the complexity of pressure data III. Our
model was also compared with PressureTransferNet, another model to generate
synthetic pressure sequence from existing pressure using style transfer
techniques, revealing a 0.102 improvement in binarized $R^{2}$ score without
the need for physical pressure data. Evaluation of T2P using real pressure
data from an 80x28 pressure sensing mattress involved creating synthetic
pressure sequences for diverse activity labels. The HAR model, trained on a
combination of synthetic and real data, outperformed models trained solely on
real data, highlighting the significance of data diversity and augmentation.
This approach, leveraging synthetic data for controlled exploration and
enriching training datasets, enhances model accuracy and adaptability to real-
world pressure data complexities and variations IV.

TABLE II: Ablation Studies on VQ-VAE using RB, SA and EMA. Model | Dictionary size | FID
---|---|---
VQ-VAE | 128 | 3.542
VQ-VAE | 256 | 2.012
VQ-VAE | 512 | 1.813
VQ-VAE + RB | 512 | 0.767
VQ-VAE + RB + SA | 512 | 0.336
VQ-VAE + RB + SA + EMA | 512 | 0.011
TABLE III: Baseline vs. VQ-VAE T2P using $R^{2}$, Binarized $R^{2}$, and FID Training approach | Binarized $R^{2}$ | $R^{2}$ | FID
---|---|---|---
Baseline T2P | 0.635 | 0.574 | 7.311
VQ-VAE T2P | 0.892 | 0.722 | 1.830
TABLE IV: F1 score of SHAR model trained on synthetic data vs. real data. HAR training set | macro F1-score
---|---
Synthetic (T2P) generated data | 0.502 $\pm$ 0.031
Real pressure data | 0.742 $\pm$ 0.038
Synthetic (PressureTransferNET + T2P) data | 0.737 $\pm$ 0.035
Real+ synthetic (PressureTransferNET + T2P) data | 0.801 $\pm$ 0.027

##  IV Conclusion

In this work, we have introduced T2P, an innovative framework that combines
vector quantization of sensor data with a simple auto-regressive method, to
generate high-quality ground pressure sequences that exhibit a discrete latent
correlation between text and pressure maps. Our method demonstrates remarkable
results, with $R^{2}$ of 0.722, binarized $R^{2}$ of 0.892, and FID of 1.83,
indicating strong consistency between the text and the generated pressure.
Furthermore, we leveraged the synthesized data to train a HAR model, which
increased the overall macro F1 score by approximately 5.9 percent when trained
with a combination of both synthetic and real data. These findings underscore
the potential of this framework to enhance the robustness of HAR models in the
pressure sensor domain, showcasing the effectiveness of leveraging current
large language models to generate synthetic data for practical applications.
Moreover, the versatility of this workflow extends beyond ground pressure
data, as it can be applied to other sensor modalities, offering a promising
avenue for generating sensor data from textual descriptions across various
domains.

## References

  * [1] Y. Hou, L. Wang, R. Sun, Y. Zhang, M. Gu, Y. Zhu, Y. Tong, X. Liu, Z. Wang, J. Xia _et al._ , “Crack-across-pore enabled high-performance flexible pressure sensors for deep neural network enhanced sensing and human action recognition,” _ACS nano_ , vol. 16, no. 5, 2022.
  * [2] B. Ali, N. Faramarzi, U. Farooq, H. C. Bidsorkhi, A. G. D’Aloia, A. Tamburrano, and M. S. Sarto, “Graphene-based smart insole sensor for pedobarometry and gait analysis,” _IEEE Sensors Letters_ , 2023.
  * [3] K. Meng, X. Xiao, W. Wei, G. Chen, A. Nashalian, S. Shen, X. Xiao, and J. Chen, “Wearable pressure sensors for pulse wave monitoring,” _Advanced Materials_ , vol. 34, no. 21, p. 2109357, 2022.
  * [4] Y. Han, A. Varadarajan, T. Kim, G. Zheng, K. Kitani, A. Kelliher, T. Rikakis, and Y.-L. Park, “Smart skin: Vision-based soft pressure sensing system for in-home hand rehabilitation,” _Soft Robotics_ , vol. 9, no. 3, pp. 473–485, 2022.
  * [5] X. Fu, J. Dong, L. Li, L. Zhang, J. Zhang, L. Yu, Q. Lin, J. Zhang, C. Jiang, J. Zhang _et al._ , “Fingerprint-inspired dual-mode pressure sensor for robotic static and dynamic perception,” _Nano Energy_ , vol. 103, p. 107788, 2022.
  * [6] X. Fan, D. Lee, L. Jackel, R. Howard, D. Lee, and V. Isler, “Enabling low-cost full surface tactile skin for human robot interaction,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 2, pp. 1800–1807, 2022.
  * [7] L. S. S. Ray, B. Zhou, S. Suh, and P. Lukowicz, “Pressim: An end-to-end framework for dynamic ground pressure profile generation from monocular videos using physics-based 3d simulation,” in _2023 IEEE International Conference on Pervasive Computing and Communications Workshops and other Affiliated Events (PerCom Workshops)_. IEEE, 2023, pp. 484–489.
  * [8] H. M. Clever, P. L. Grady, G. Turk, and C. C. Kemp, “Bodypressure-inferring body pose and contact pressure from a depth image,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 45, no. 1, pp. 137–153, 2022.
  * [9] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark _et al._ , “Learning transferable visual models from natural language supervision,” in _International conference on machine learning_. PMLR, 2021, pp. 8748–8763.
  * [10] L. Floridi and M. Chiriatti, “Gpt-3: Its nature, scope, limits, and consequences,” _Minds and Machines_ , vol. 30, pp. 681–694, 2020.
  * [11] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in _IEEE/CVF conference on computer vision and pattern recognition_ , 2022.
  * [12] J. Kim, J. Kim, and S. Choi, “Flame: Free-form language-based motion synthesis & editing,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 37, no. 7, 2023, pp. 8255–8263.
  * [13] OpenAI, “Gpt-4 technical report,” 2023.
  * [14] J. Zhang, Y. Zhang, X. Cun, S. Huang, Y. Zhang, H. Zhao, H. Lu, and X. Shen, “T2m-gpt: Generating human motion from textual descriptions with discrete representations,” _arXiv preprint arXiv:2301.06052_ , 2023.
  * [15] S. Raab, I. Leibovitch, G. Tevet, M. Arar, A. H. Bermano, and D. Cohen-Or, “Single motion diffusion,” _arXiv preprint arXiv:2302.05905_ , 2023.
  * [16] X. Zuo, S. Wang, J. Zheng, W. Yu, M. Gong, R. Yang, and L. Cheng, “Sparsefusion: Dynamic human avatar modeling from sparse rgbd images,” _IEEE Transactions on Multimedia_ , vol. 23, pp. 1617–1629, 2021.
  * [17] C. Guo, S. Zou, X. Zuo, S. Wang, W. Ji, X. Li, and L. Cheng, “Generating diverse and natural 3d human motions from text,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2022, pp. 5152–5161.
  * [18] M. Plappert, C. Mandery, and T. Asfour, “The KIT motion-language dataset,” _Big Data_ , vol. 4, no. 4, pp. 236–252, dec 2016. [Online]. Available: http://dx.doi.org/10.1089/big.2016.0028
  * [19] M. Loper, N. Mahmood, J. Romero, G. Pons-Moll, and M. J. Black, “Smpl: A skinned multi-person linear model,” in _Seminal Graphics Papers: Pushing the Boundaries, Volume 2_ , 2023, pp. 851–866.
  * [20] S. T. Team, “Stretchable electronics, smarter connectivity,” 2023. [Online]. Available: http://sensingtex.com/


# From Words to Watts: Benchmarking the Energy Costs of Large Language Model
Inference
††thanks: DISTRIBUTION STATEMENT A. Approved for public release. Distribution
is unlimited. This material is based upon work supported by the Assistant
Secretary of Defense for Research and Engineering under Air Force Contract No.
FA8702-15-D-0001, and United States Air Force Research Laboratory Cooperative
Agreement Number FA8750-19-2-1000. Any opinions, findings, conclusions or
recommendations expressed in this material are those of the author(s) and do
not necessarily reflect the views of the Assistant Secretary of Defense for
Research and Engineering, or the United States Air Force. The U.S. Government
is authorized to reproduce and distribute reprints for Government purposes
notwithstanding any copyright notation herein.

Siddharth Samsi1§, Dan Zhao2, Joseph McDonald1, Baolin Li3, Adam Michaleas1,
Michael Jones1, William Bergeron1, Jeremy Kepner1, Devesh Tiwari3, Vijay
Gadepally1
1 MIT, 2 NYU, 3 Northeastern University

###### Abstract

Large language models (LLMs) have exploded in popularity due to their new
generative capabilities that go far beyond prior state-of-the-art. These
technologies are increasingly being leveraged in various domains such as law,
finance, and medicine. However, these models carry significant computational
challenges, especially the compute and energy costs required for inference.
Inference energy costs already receive less attention than the energy costs of
training LLMs—despite how often these large models are called on to conduct
inference in reality (e.g., ChatGPT). As these state-of-the-art LLMs see
increasing usage and deployment in various domains, a better understanding of
their resource utilization is crucial for cost-savings, scaling performance,
efficient hardware usage, and optimal inference strategies.

In this paper, we describe experiments conducted to study the computational
and energy utilization of inference with LLMs. We benchmark and conduct a
preliminary analysis of the inference performance and inference energy costs
of different sizes of LLaMA—a recent state-of-the-art LLM—developed by Meta AI
on two generations of popular GPUs (NVIDIA V100 & A100) and two datasets
(Alpaca and GSM8K) to reflect the diverse set of tasks/benchmarks for LLMs in
research and practice. We present the results of multi-node, multi-GPU
inference using model sharding across up to 32 GPUs. To our knowledge, our
work is the one of the first to study LLM inference performance from the
perspective of computational and energy resources at this scale.

###### Index Terms:

Large Language Models, Natural Language Processing, Inference, Green AI, LLM,
NLP, Deep Learning, Distributed Computing, Energy, Sustainability

##  I Introduction

Generative models (GenAI) are able to produce new content from synthesizing
text, images, and audio from which it’s trained on. While GenAI is not
entirely new, the recent application and broad availability of this technology
via tools such as Stable Diffusion [1], OpenAI’s ChatGPT, Google’s Bard and
integration into the Microsoft Bing search engine has captured the imagination
of the world and led to a massive surge in interest in deploying these types
of models across a variety of domains ranging such as education, government,
engineering, law, finance and many more.

The popularity of these models has also put a spotlight on many societal
concerns stemming from their usage. From ethical concerns ranging from
violations of copyright laws [2, 3] to safety concerns arising from the fact
that these models are capable of hallucinating or fabricating information,
concerns about these models in the educational and medical domain [4, 5],
their carbon footprint, and many more.

In this paper, we focus primarily on understanding the significant amount of
resources—time, computation, and energy—required for using and deploying some
of the large language models (LLM) like those that underlie ChatGPT, Bard,
etc. Several prior works have estimated the compute and energy costs of
training language models. Works like [6] discuss the carbon footprint of
language models such as BERT, ELMo, and precursors to larger models such as
GPT-3 and GPT-4 that power some of the popular AI chatbots today. Others have
also looked to larger language models; for instance, the largest NVIDIA
Megatron-LM model required 3,072 A100 GPUs [7, 8, 9] for its training. While
the complete details (time and resources used) of compute required for
training GPT-3/4 are not available, several estimates for training[10, 11] and
inference are publicly available. As industry attempts to shore up competitive
moats and restrict information regarding their underlying LLM technologies,
these details can become less reliable and available. Compounding this issue,
estimates for inference are even less readily available [12] despite their
significant share of energy costs and their likely larger impact on the
environment [13]—especially since model inference calls can occur more
frequently than training/fine-tuning for real-world deployments and
applications.

We present the results of our inference experiments on LLaMA [14]: an open
sourced pre-trained large language models by Meta AI. The LLaMA model is
available in a number of sizes but, in most cases, its larger variants
typically require multiple high-end GPUs for both training and inference
(assuming no further compression/distillation). While our emphasis is on
characterizing the compute performance and energy used for multi-node, multi-
GPU inference, we also include results from single node instances using
smaller variants of the model as a baseline comparison. We hope our work will
help illustrate some of the compute performance and energy utilization
characteristics of LLM inference. We also hope that our experiments, analysis,
and data on real-world hardware will spur further analysis, benchmarking, and
more open dissemination of the systematic performance characteristics for a
wider range of large models—especially under different kinds of hardware,
data, and optimization strategies.

##  II Overview of Large Language Models

The landscape of large language models (LLMs) and large foundation models
(LFMs) has seen explosive growth in both the speed of development as well as
complexity of ever larger models. Over the past several years, competition has
been fierce and the pace un-relenting as AI research groups across private
companies and academic institutions have developed new models whose
performance continues to improve on a wide suite of natural language
benchmarks but still requires significant amounts of compute and energy. We
provide a brief overview of LLMs and LFMs below along with details around the
specific LLM we use for our analysis.

Figure 1: Development paths of LLMs: A tree diagram illustrating the
development of language models and foundation models from 2017 to early 2023.
Pink branches indicate encoder-type language models, green indicates encoder-
decoder hybrid models, and the dark grey indicates decoder-style models. The
bar-plot on the bottom right tallies the number of open/closed source models
developed by different companies/institutions. We study LLaMA (outlined by the
red arrow and red circle in the diagram above) as an example of one of the
more recent, modern, and state-of-the-art LLMs whose size/complexity resemble
Google’s Bard and OpenAI’s GPT-4, all three of which were released around the
same time (spring 2023). Original figure from [15].

###  II-A Large Language Models & Large Foundation Models

As seen in Fig. 1, many different LLMs and foundation models exist—each with
their own respective training setup, architectural modifications, purposes or
use-cases, etc. Large language models and foundation models are best known for
their sheer size, resource intensity (i.e., the amount of computational
resources required for training/inference), and their impressive capabilities
in tasks that include, but may not be limited to, natural language.

Typically, LLMs refer to language models containing on the order of hundreds
of millions to billions of parameters that are trained on extremely large
datasets of text. These models are also typically based on some variant of the
original transformer architecture [16] usually leveraging the decoder half or
a hybrid encoder-decoder architecture. Large language models can be considered
a subset of large foundation models; whereas LLMs focus almost exclusively on
language data for their inputs and outputs, large foundation models include
models that allow for multiple modalities such as image and text (e.g., GPT-4)
or other modalities such as image generation (e.g., Stable Diffusion) or video
generation (e.g., MidJourney). We refer to [17] for a comprehensive review of
the broad classes of GenAI and their capabilities.

###  II-B LLaMA

Developed by Meta AI and released in February of 2023, LLaMA [14] (Large
Language Model Meta AI) is a large language model (LLM) that relies on the
traditional transformer architecture originally introduced in [16]. Most
notably, the performance of LLaMA rivaled or exceeded that of GPT-3 on many
NLP benchmarks and remains competitive with other state-of-the-art LLMs [14].
Like other LLMs, LLaMA was pre-trained on a large collection of data including
but not limited to CommonCrawl, Github, Wikipedia, etc. As of spring 2023,
alongside other recently timed releases of state-of-the-art LLMs such as
Google’s Bard and OpenAI’s GPT-4, LLaMA is competitive in its state-of-the-art
performance across multiple tasks, making it an ideal workhorse for
realistically studying and benchmarking inference.

LLaMA comes in four sizes characterized by the number of parameters: 7 billion
(LLaMA 7B), 13 billion (LLaMA 13B), 33 billion (LLaMA 33B) and 65 (LLaMA 65B).
LLaMA’s model weights, across all of its variants, were publicly released
under a non-commercial license, making it one of only a select few modern,
state-of-the-art LLMs that have been publicly available.

To best understand the realities that lie behind the energy costs and
throughput of state-of-the-art LLM inference, we focus our analysis on the
largest available version of LLaMA—namely, LLaMA 65B. We also conduct analysis
comparing the 7B and 13B LLaMA variants to establish the baseline performance
of the smaller variants of the LLaMA model.

The largest model we focus our analysis on, LLaMA 65B, is a 65 billion
parameter model with an effective model dimension of 8,192 and a total of 80
layers and 64 attention heads, trained over 1.4 trillion tokens. By focusing
on the largest 65B version, we also hope to study inference at its fullest
scale, controlling for and benchmarking phenomena that we may not observe on
LLMs of smaller size or complexity. This way, we can realistically benchmark
and study the dynamics, as well as the implications, of inference energy costs
and through-put on a scale consistent with state-of-the-art LLMs that we see
and use today.

##  III Experimental Setup

We conducted our experiments on the MIT Supercloud high-performance computing
(HPC) system[18]. This heterogeneous HPC cluster consists of 448 compute nodes
with dual Intel Xeon Gold 6248 CPUs with 384 GB of RAM and two NVIDIA Volta
V100 GPUs with 32 GB of memory per node. Each node on the system has two
independent back-end fabrics: a 100 Gb/s Intel Omnipath as well as a 25 Gb/s
Ethernet interconnect using Mellanox ConnectX-4 adapters with all servers
connected to a single, non-blocking Arista DCS-7516 Ethernet core switch. The
GPUs, Omnipath, and Ethernet cards are all connected to PCIe slots that route
directly to the Xeon processors without any intermediary PCIe switches. All
experiments in this paper exclusively used the 25 Gb/s Ethernet interconnect.
The system also includes 480 CPU-only nodes with Intel Xeon Platinum 8260
processors. In addition, four nodes with NVIDIA A100 GPUs were also available
for experiments described in this paper. A summary of the hardware is shown in
Table I. All experiments described in this paper were run exclusively on
NVIDIA GPUs.

TABLE I: Compute node configurations: This table lists the types of hardware used for inference evaluations in our experiments. Each node consists of 2 CPUs and 2 GPUs in the configuration listed below. All GPUs are from NVIDIA. CPU |  | GPU
---|---|---
Type | Memory | TDP |  | Type | Memory | TDP
| (GB) | (W) |  |  | (GB) | (W)
Intel Xeon |  |  |  |  |  |
Gold 6248 | 384 | 150 |  | V100 | 32 | 250
Intel Xeon |  |  |  |  |  |
Platinum 8358 | 503 | 240 |  | A100 | 80 | 300

###  III-A Models

Experiments were performed using open-source implementation of the pre-trained
LLaMA 65B model available via request from Meta [14] and evaluation scripts
available via GitHub [19]. This implementation of the model uses Pytorch and
the FairScale [20] library to enable model sharding across multiple GPUs and
nodes. For the models, we use a decoder temperature setting $\tau=0.8$ and a
top-$p$ value of 0.95 in attempts to align our settings with the general range
of values that are typically used. In future work, we aim to study how varying
decoding temperature, top-$p$, and other hyper-parameters may affect compute
performance and energy usage during inference. While our main focus is on
LLaMA 65B, we also examine LLaMA 7B and LLaMA 13B to characterize inference
performance and energy under the bare minimum settings/resources required to
run these models.

###  III-B Datasets

We used two datasets to evaluate inference performance. The first is an
instruction following dataset used to fine-tune the Alpaca [21] model (from
here on, this dataset is referred to as “Alpaca” in our paper which is not to
be confused with the Alpaca model). This Alpaca dataset consists of 52,000
instruction-following tasks, instructions/questions where some have example
inputs and some do not, that the model is asked to answer. The second dataset
is GSM8K [22], consisting of 8,500 human crafted grade school math problems.
The goal of using these two datasets is two-fold: (1) to evaluate the model on
a diverse set of tasks in NLP and (2) evaluate how different types of data and
its underlying dynamics can impact energy and inference performance. While
natural language is more common in LLM usage and in LLM training data,
increasingly new capabilities have been demonstrated in LLMs, including the
ability to solve simple mathematical problems, provide/correct examples of
code, and more. Math questions also differ considerably from questions posed
in natural language which can result in smaller context windows,
inputs/outputs of differing lengths, number of decoded tokens, etc. This, in
turn, may impact inference performance in either throughput rates or energy
costs. For this reason, our benchmarking experiments are conducted on both
datasets.

For both datasets, we sample 4,096 inputs for our inference experiments. Using
the entirety of the datasets would only serve to increase inference time and
energy used for the experimentation unreasonably and did not provide any
significant benefits to the study.

###  III-C Evaluation

Our goal is to evaluate the inference performance, latency, and inference
energy costs of LLaMA 65B as a representative large language model that
requires sharding across multiple GPUs. We intend this to be a preliminary
analysis that will help guide more in-depth experiments and benchmarking for
our future work. Our analysis also includes limited analysis of smaller LLaMA
variants to illustrate inference performance and energy trade-offs in bare-
minimum hardware settings: namely, LLaMA 7B and 13B. While we do not control
for the correctness/quality of the outputs or the complexity of the
inputs/outputs in studying trade-offs between inference energy and
performance, we hope to account for this as an ablative study in future work.
Similarly, we do not perform a comprehensive evaluation with different
optimization techniques or inference settings available for LLMs such as
modeling query arrival rates, model quantization, continuous batching, etc.
which we also leave for future work.

Inference performance is measured in terms of rates: words, tokens, and
responses per second or, equivalently, the number of words, tokens, and
responses generated per second. When running inference with LLaMA, the model
generates a string of text for each input until the length of the text hits a
maximum generation length or a stop-word is encountered. The number of words
are calculated by counting the number of words present in the output by
splitting each output string on spaces. The number of tokens is calculated
using LLaMA’s own default tokenizer by counting the number of tokens in the
tokenized output. Lastly, the number of responses per second or the response
rate is calculated using the total number of responses and the total time
needed to run inference over the input data.

We monitor GPUs using the nvidia-smi [23] and NVIDIA DCGM [24] utilities to
study GPU utilization, energy, power draw, etc. during our experiments. The
nvidia-smi utility is used to capture GPU usage over time at 100ms intervals
and the DCGM monitoring tool is used to capture aggregate GPU energy in Joules
for the rank-0 node. For a multi-node, multi-GPU model, we multiply the rank-0
energy by the number of nodes used. Maximum power draw on GPUs is capped at
250 Watts unless otherwise stated. Due to limits on resource availability, we
mainly use V100 GPUs for larger-scale distributed experiments (i.e., for 8,
16, and 32 shards) and A100 GPUs for smaller scale experiments.

Inference energy metrics are calculated by combining the inference metrics
above with the energy data collected from our GPUs using NVIDIA’s utilities
described above. Specifically, energy per second is defined as the total
aggregate GPU energy spent from a single experiment/job (across all shards)
divided by the total run time of that experiment/job in seconds. A single
experiment/job denotes a single run through all 4,096 prompts under a
specified batch size. Energy per token and energy per response are similarly
defined as total energy divided by the number of decoded output tokens and the
number of responses as defined above, respectively.

##  IV Results

###  IV-A Baselines: LLaMA 7B, 13B, & 65B

####  IV-A1 Inference Performance

We begin our analysis with a baseline comparison of LLaMA 65B with smaller-
scale LLaMA models: LLaMA 7B and 13B. The goal is to understand the following:
what do inference performance and energy trade-offs look like for the
different sizes of LLaMA under the bare-minimum set of resources required to
have them running inference? This question can be important for researchers
and users who have may not have limitless computational resources and hardware
acceleration or may be constrained in terms of GPU memory, etc.

Given the sizes of the models, the size of the data, and the hardware memory
limits, we only show results from experiments that were possible for a given
combination of parameters (i.e., for some models, certain combinations of
batch size and number of shards are infeasible due to memory limits of the
underlying GPUs). Table II shows the bare minimum hardware requirements for
each LLaMA variant and the maximum batch size possible for each combination,
assuming no further model compression, optimization, quantization,
distillation etc.

TABLE II: Baseline configurations for LLaMA 7B, 13B, and 65B: This table lists the bare minimum hardware required for different models and the maximum batch size possible given the bare minimum hardware for a max response length of 256. These limits are imposed by a combination of GPU memory, model size, response length and the number of GPUs. While the 65B model can sharded across 6 V100 GPUs, we use 8 since the model architecture makes it better suited for balanced sharding across 8 GPUs. Model Size | V100 32GB |  | A100 80GB
---|---|---|---
| Count | Max. Batch size |  | Count | Max. Batch size
7B | 1 | 64 |  | 1 | 64
13B | 2 | 64 |  | 1 | 64
65B | 8 | 64 |  | 4 | 128

With these limits in mind, we present the inference performance of LLaMA 7B,
13B, and 65B on the Alpaca and GSM8K datasets with the bare minimum hardware
settings in Figure 2. The plots in Figure 2 show a baseline comparison of
inference performance of the three LLaMA variants on both the V100 and A100
GPUs respectively. For each model, in line with the spirit of the bare minimum
settings, inference is done with a batch size of 64 and an maximum generation
length of 256. The 7B model was run on a single GPU and 13B on two GPUs in
each case whereas the 65B model was run on 8 V100 GPUs and 4 A100 GPUs
respectively due to the size of the model and available memory on the GPU(s).

As expected, we observe that the A100 outperforms V100 on both the Alpaca and
GSM8K datasets: particularly for the smaller LLaMA 7B and 13B, we see anywhere
from a 2 times (7B) to a 1.25 times increase (13B) in inference latency on the
A100 when compared to the V100 across words per second, tokens per second, and
responses per second. Faster response rates and inference are likely due to
the fact that the number of computations, directly related to the number of
parameters of said model, involved in the 7B and 13B models are significantly
lower than the 65B model. We do note that for LLaMA 65B, we see a much smaller
improvement in using the A100 over the V100; however, since the 65B model
requires sharding across two (A100) or four (V100) compute nodes at the
mininum, this could result in additional latency to each forward pass of the
model, explaining the smaller improvements. We also observe that while LLaMA
7B exhibits a considerable improvement in inference throughput on both Alpaca
and GSM8K with the A100, the improvement is much larger for Alpaca than GSM8K.
This can also be attributed to the different complexities of inputs from each
dataset.

(a) Results from the Alpaca dataset.

(b) Results from GSM8K dataset

Figure 2: Baseline comparison of inference performance/latency between LLaMA
7B, 13B and 65B: inference performance comparisons on the minimum set of
hardware required to run inference (see Table II) across model sizes and
between V100s and A100s.

####  IV-A2 Inference Energy

Figure 3 shows a comparison of the energy per second required to run inference
on LLaMA 7B, 13B, and 65B, with different GPUs under the same bare minimum
hardware settings as the above. For both the Alpaca and GSM8K datasets, we see
that there is a considerable increase in the energy per second across all
LLaMA sizes when using the A100 over the V100 where the most considerable
increase is for the smallest 7B model. Although Figure 2 shows a considerable
increase in inference throughput from using the A100, Figure 3 shows us that
this improvement does not come for free: it comes at an increased energy cost
per second. Moreover, for the largest LLaMA 65B, it is less clear whether the
increased inference energy per second (Figure 3) is worth the small
improvement in inference throughput in terms of words/token/responses per
second (Figure 2).

Figure 3: Baseline energy per second (Watts) estimates of performing inference
with LLaMA 7B, 13B, and 65B: inference energy comparisons on the minimum set
of hardware/settings required (see Table II) with Alpaca and GSM8K on a log-
scale. Color indicates device (V100/A100), bars indicate average quantities
and lines indicate error bars. Energy is averaged over maximum generation
lengths of 256, 512, and 1024 due to near-identical energy/size trends for
each generation length.

###  IV-B Energy per Second: LLaMA 65B

We first take a look at the amount of energy inference costs per unit time in
seconds. Figures 4 and 5 show a more in-depth look of the energy inference
costs of LLaMA 65B across different batch sizes and degrees of sharding.
Specifically, Figure 4 shows energy costs for maximum generation length 512
and Figure 5 shows energy costs for 1024.

Figure 4: Energy per second (Watts) estimates of LLaMA 65B across batch sizes
of 64/128/256/256 and 8/16/32 shards for max generation length 512: inference
energy estimates on Alpaca and GSM8K on log-scale. Color indicates batch size.

Figure 5: Energy per second (Watts) estimates of LLaMA 65B across batch sizes
of 64/128/256/512 and 8/16/32 shards for max generation length 1024: inference
energy estimates on Alpaca and GSM8K on log-scale. Color indicates batch size.

Overall, we see an average increase in energy per second with the number of
shards. While there is a slight correlation as energy per second increases
with increasing batch size, increasing the number of shards always increases
the wattage. Indeed, the energy per second increases with the number of shards
even at the same batch size (e.g., the energy of inference at batch size 64,
going from 16 shards to 32 shards). For both datasets, increasing the max
generation length from 512 to 1024 does seem to increase the energy per second
for each batch size within each shard configuration, but the overall effect is
less clear or consistent. Overall, we see that the energy per second for
inference with LLaMA 65B is on the order of 300 Watts to 1 Kilowatt from the
lower shard configuration of 8 GPUs to the higher end of 32 GPUs.

###  IV-C Energy per Decoded Token: LLaMA 65B

Moving on to energy per each decoded output token, we see that in Figures 6
and 7 that energy per token tends to follow a similar pattern in relation to
the number of shards: as the number of shards increases, the energy per output
token also increases. However, we see little change in the average energy per
token between max generation length 512 and 1024. For instance, with length
512, we see that it takes about 3-4 Joules for a output token, which is
approximately the same amount for length 512. As with energy per second, max
generation length seems to have a negligible effect on energy costs from 512
to 1024. Interestingly, there appears to be an exception for the GSM8K math
problem dataset; there exists a “sweet spot” at 16 shards where continuously
increasing the batch size can actually reduce the energy per token at max
generation length 512. However, this disappears under max generation length
1024 where increasing the batch size increases the energy per token. The
definitive existence of this sweet spot for datasets of differing
styles/complexities, or others like it, will require more experimentation and
benchmarking to establish.

Figure 6: Energy per output token estimates of LLaMA 65B across batch sizes of
64/128/256/512 and 8/16/32 shards for max generation length 512: inference
energy estimates on Alpaca and GSM8K on log-scale. Color indicates batch size.

Figure 7: Energy per output token estimates of LLaMA 65B across batch sizes of
64/128/256/512 and 8/16/32 shards for max generation length 1024: inference
energy estimates on Alpaca and GSM8K on log-scale. Color indicates batch size.

###  IV-D Energy per Response: LLaMA 65B

Figures 8 and 9 show energy metrics in terms of responses from the 65B model.
Like before, we see that increasing the number of shards still tends to
increase the energy costs of inference per response most overall while
increasing the maximum generation length from 512 (Figure 8) to 1024 (Figure
9) does not induce a clear or significant effect in inference energy costs.
Also like before, while we see slight increases in energy costs per response
generated within a shard configuration as batch size increases, but not
consistently or significantly. Again, we see that for GSM8K, at max generation
length 512, increasing the batch size while keeping the number of shards fixed
at 16 is associated with a decrease in energy per response, which is
consistent with what we observed in energy per tokens in the same setting.

Figure 8: Energy per response estimates of LLaMA 65B across batch sizes of
64/128/256/512 and 8/16/32 shards for max generation length 512: inference
energy estimates on Alpaca and GSM8K on log-scale. Color indicates batch size.

Figure 9: Energy per response estimates of LLaMA 65B across batch sizes of
64/128/256/512 and 8/16/32 shards for max generation length 512: inference
energy estimates on Alpaca and GSM8K on log-scale. Color indicates batch size.

###  IV-E Effects of GPU Power Capping on LLaMA 65B

Power consumption in AI is an increasingly important concern. In prior work,
we have shown [25] that power capping GPUs during training of language models
such as BERT [26] is an effective way of reducing the energy consumed training
these models. While the work in [25] focused on model training, in this paper,
we focus on inference. In order to study the effect of power capping on
inference using large language models, we ran a limited set of experiments
using LLaMA 65B. We ran the 65B model on four 80GB A100 GPUs with the power
cap set at 250W, 175W and 150W.

Table III shows the relative change in total inference time, energy and token
rate under power cap conditions. Results shown here are calculated relative to
a power cap of 250W. For a 30% reduction in power from 250W to 175W, the
inference time increases by an average of 6.7% for a corresponding average
reduction in total energy by 23.21%. However, a reduction in power cap to 150W
results in a much more significant (19.49%) increase in average inference
time. These results show that power capping as an energy savings intervention
can be effective when applied appropriately. A static power cap for all GPU
workloads may not show the same effectiveness depending on the task and
additional experimentation is required to make broader recommendations.

Output | Time |  | Energy |  | Token Rate
---|---|---|---|---|---
length | % change |  | % change |  | % change
| 175W | 150W |  | 175W | 150W |  | 175W | 150W
256 | 6.23 | 15.33 |  | -21.82 | -32.76 |  | -5.87 | -13.15
512 | 6.51 | 21.70 |  | -23.95 | -34.66 |  | -6.11 | -17.83
1024 | 7.40 | 21.65 |  | -23.87 | -34.59 |  | -6.89 | -17.80
TABLE III: Effects of GPU power capping on LLaMA 65B inference: This table
shows the relative performance of the LLaMA 65B model on the GSM8k dataset
with a batch size of 64 and output lengths of 256, 512, 1024 using NVIDIA A100
GPUs. The GPUs were power capped at 250W, 175W and 150W. Results shown here
are relative to model performance at 250W to stay consistent with the settings
in the rest of the experiments described here.

###  IV-F GPU Resource Utilization under Distributed Inference

Finally, we briefly examine the average GPU resource utilization by the 65B
LLaMA model when running model sharded inference. For the sake of simplicity,
we only consider a batch size of 64 and a maximum generated output length of
256. For this configuration, we ran on four A100 GPUs and 8, 16, 32 V100 GPUs.
These results are summarized in Tables IV and V. In all cases, the streaming
multiprocessors (SM) utilization as reported by the DCGM utility was observed
to be in the 94%-95% range. For the A100 GPUs, the average SM utilization
rises to 98% when the maximum generated output length is increased to 2048.
Given that the model is sharded in a manner that enables us to load it fully
in GPU memory and run inference on a non-trivial amount of data, we expect
memory utilization to be low depending on the specific model parameters and
input sizes used. Thus, on the four 80GB A100 nodes, the memory utilization
varies between 23%-27% depending the maximum generated output length. This
under-utilization of memory implies that it may be possible to co-locate
multiple models on the same set of GPUs to increase aggregate throughput and
potentially reduce cloud compute costs or improve system utilization at a
supercomputer center. With new GPU sharing capabilities such as Multi-Process
Service (MPS) [27] and Multi-Instance GPU (MIG) [28], a single GPU may be
shared by diverse workloads for an overall improvement in system throughput as
shown in recent work [29]. The optimal GPU configuration for sharing LLMs and
other workloads is a part of our future work in this area.

Model Shards | Output Length | Max. Memory Util. | Avg. SM Util.
---|---|---|---
4 | 256 | 23.36 | 95.00
4 | 512 | 24.54 | 98.81
4 | 1024 | 24.85 | 98.85
4 | 2048 | 27.00 | 98.00
TABLE IV: A100 Utilization: This table shows GPU utilization for 80GB A100 GPUs and LLaMA 65B with 4 shards, batch size of 64 averaged across both datasets used in this paper. Model Shards | Output Length | Max. Memory Util. | Avg. SM Util.
---|---|---|---
8 | 256 | 24.25 | 94.75
16 | 256 | 13.33 | 95.00
32 | 256 | 6.66 | 95.66
TABLE V: V100 Utilization: This table shows GPU utilization for 32GB V100 GPUs
and LLaMA 65B with 8, 16, 32 shards, a batch size of 64 and maximum generated
output length of 256 averaged across both datasets used in this paper. We
limit this result to an ouptut length of 256 because longer outputs on 8 V100
GPUs are not possible given memory limits of the GPU.

##  V Discussion

In this paper, we show the results of benchmarking a representative large
language model on NVIDIA GPUs. We show baseline results from smaller models
(7B, 13B) and compare them against the largest available version (65B) of
LLaMA. We also examine the inference performance and energy across distributed
settings and different configurations by varying model parameters, input data,
and hardware configurations. By comparing a natural language instruction
following dataset (Alpaca) and a mathematical question-answer dataset (GSM8K),
we also find that the complexity of the input dataset can affect the model
performance for a given set of hyper-parameters and hardware configuration.

Given the size of LLMs and the limits imposed by current hardware, inference
with large models can impose onerous requirements. For example, we find that,
at a minimum, 8 V100 GPUs each with 32 GB of RAM or 4 A100 GPUs each with 80GB
of memory are required for any meaningful inferences with the 65B LLaMA model.
In each case among our experiments, we shard the model evenly across all GPUs
in order to fit the model/data; however, this results in only 20%-25% of the
GPU memory being utilized at any given time. This over-provisioning of
resources represents new opportunities for resource sharing across multiple
workloads in the latest NVIDIA GPUs. The Multi-Process Service (MPS) [27] and
Multi-Instance GPU (MIG) [28] are new capabilities that enable GPU sharing
across different workloads. Although identifying the optimal MPS or MIG
configuration for a given set of workloads is challenging, recent work [29]
has developed new techniques to exploit these capabilities in order to
dynamically partition GPU resources. This opens up the potential to optimally
partition high-end GPUs such as the A100s or H100s to co-locate multiple LLMs
for inference—with the potential of only minimal degradation to computational
performance.

Finally, as AI compute requirements have increased, there is an increasing
focus on approaches to reduce the carbon and energy footprints of datacenters
by making larger models leaner or more efficient. Approaches such as model
quantization, distillation, sparsification, etc. are being developed to reduce
the compute required for AI along with the development of custom, energy-
efficient hardware for inference and training. However, simple interventions
like GPU power capping is available to be deployed today—our preliminary
analysis with LLM inference in this paper suggests that power capping can be
an effective tool for reducing inference energy. If applied at the datacenter-
scale, this intervention has the potential to reduce overall energy usage in
the long-run as new approaches are developed to address the energy consumption
of AI compute.

As part of our future plans, we aim to conduct similar experiments on other
open-source, large language models along with more in-depth characterization
of compute and energy for not just inference, but also for the training/fine-
tuning of these models. It is our hope that this paper provides a baseline for
inference with LLMs and fosters a broader discussion of the challenges and
opportunities in this field.

## Acknowledgements

The authors acknowledge the MIT SuperCloud [18] and Lincoln Laboratory
Supercomputing Center for providing HPC and consultation resources that have
contributed to the research results reported within this paper. The authors
acknowledge the MIT SuperCloud team: William Arcand, William Bergeron, Chansup
Byun, Michael Houle, Anna Klein, Peter Michaleas, Lauren Milechin, Julie
Mullen, Albert Reuther, Antonio Rosa, and Charles Yee. The authors also wish
to acknowledge the following individuals for their contributions and support:
Bob Bond, Allan Vanterpool, Tucker Hamilton, Jeff Gottschalk, Mike Kanaan,
Charles Leiserson, Dave Martinez, Steve Rejto, Marc Zissman.

## References

  * [1] Stability-AI, “Stable Diffusion,” https://github.com/Stability-AI/StableDiffusion, 2023.
  * [2] D. Foster, _Generative deep learning_. ” O’Reilly Media, Inc.”, 2022.
  * [3] Z. Epstein, A. Hertzmann, the Investigators of Human Creativity _et al._ , “Art and the science of generative ai,” _Science_ , vol. 380, no. 6650, pp. 1110–1111, 2023. [Online]. Available: https://www.science.org/doi/abs/10.1126/science.adh4451
  * [4] B. Meskó and E. J. Topol, “The imperative for regulatory oversight of large language models (or generative ai) in healthcare,” _npj Digital Medicine_ , vol. 6, no. 1, p. 120, 2023.
  * [5] H. Zohny, J. McMillan, and M. King, “Ethics of generative ai,” _Journal of Medical Ethics_ , vol. 49, no. 2, pp. 79–80, 2023. [Online]. Available: https://jme.bmj.com/content/49/2/79
  * [6] E. Strubell, A. Ganesh, and A. McCallum, “Energy and policy considerations for deep learning in NLP,” in _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. Florence, Italy: Association for Computational Linguistics, Jul. 2019, pp. 3645–3650.
  * [7] M. Shoeybi, M. Patwary, R. Puri _et al._ , “Megatron-lm: Training multi-billion parameter language models using model parallelism,” 2020.
  * [8] D. Narayanan, M. Shoeybi, J. Casper _et al._ , “Efficient large-scale language model training on gpu clusters using megatron-lm,” 2021.
  * [9] “Nvidia/megatron-lm: Ongoing research training transformer models at scale,” https://github.com/NVIDIA/Megatron-LM, 2023.
  * [10] J. Sevilla, L. Heim, A. Ho _et al._ , “Compute trends across three eras of machine learning,” 2022.
  * [11] D. Patel, “The ai brick wall – a practical limit for scaling dense transformer models, and how gpt 4 will break past it,” https://www.semianalysis.com/p/the-ai-brick-wall-a-practical-limit, 2023\.
  * [12] R. Desislavov, F. Martínez-Plumed, and J. Hernández-Orallo, “Trends in ai inference energy consumption: Beyond the performance-vs-parameter laws of deep learning,” _Sustainable Computing: Informatics and Systems_ , vol. 38, p. 100857, 2023.
  * [13] D. Zhao, N. C. Frey, J. McDonald _et al._ , “A green(er) world for a.i.” in _2022 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW)_ , 2022, pp. 742–750.
  * [14] H. Touvron, T. Lavril, G. Izacard _et al._ , “Llama: Open and efficient foundation language models,” 2023.
  * [15] “Different development paths of llms,” https://www.interconnects.ai/p/llm-development-paths.
  * [16] A. Vaswani, N. Shazeer, N. Parmar _et al._ , “Attention is all you need,” 2017\.
  * [17] R. Gozalo-Brizuela and E. C. Garrido-Merchan, “Chatgpt is not all you need. a state of the art review of large generative ai models,” 2023.
  * [18] A. Reuther, J. Kepner, C. Byun _et al._ , “Interactive supercomputing on 40,000 cores for machine learning and data analysis,” in _2018 IEEE High Performance extreme Computing Conference (HPEC)_. IEEE, 2018, pp. 1–6.
  * [19] Facebook Research, online, 2023. [Online]. Available: {https://github.com/facebookresearch/llama}
  * [20] FairScale authors, “Fairscale: A general purpose modular pytorch library for high performance and large scale training,” https://github.com/facebookresearch/fairscale, 2021.
  * [21] R. Taori, I. Gulrajani, T. Zhang _et al._ , “Stanford alpaca: An instruction-following llama model,” https://github.com/tatsu-lab/stanford_alpaca, 2023.
  * [22] K. Cobbe, V. Kosaraju, M. Bavarian _et al._ , “Training verifiers to solve math word problems,” _arXiv preprint arXiv:2110.14168_ , 2021.
  * [23] NVIDIA. Nvidia-smi. [Online]. Available: http://developer.download.nvidia.com/compute/DCGM/docs/nvidia-smi-367.38.pdf
  * [24] ——. Nvidia data center GPU manager (dcgm). [Online]. Available: https://developer.nvidia.com/dcgm
  * [25] J. McDonald, B. Li, N. Frey _et al._ , “Great power, great responsibility: Recommendations for reducing energy for training language models,” in _Findings of the Association for Computational Linguistics: NAACL 2022_ , 2022, pp. 1962–1970.
  * [26] J. Devlin, M.-W. Chang, K. Lee _et al._ , “BERT: Pre-training of deep bidirectional transformers for language understanding,” in _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. Minneapolis, Minnesota: Association for Computational Linguistics, Jun. 2019, pp. 4171–4186. [Online]. Available: https://aclanthology.org/N19-1423
  * [27] NVIDIA, “Multi-Process Service,” https://docs.nvidia.com/deploy/mps/, 2023\.
  * [28] NVIDIA, “NVIDIA Multi Instance GPU User Guide,” https://docs.nvidia.com/datacenter/tesla/mig-user-guide/, 2023.
  * [29] B. Li, T. Patel, S. Samsi _et al._ , “Miso: exploiting multi-instance gpu capability on multi-tenant gpu clusters,” in _Proceedings of the 13th Symposium on Cloud Computing_ , 2022, pp. 173–189.