# LLeMpower: Understanding Disparities in the Control and Access of Large
Language Models

Vishwas Sathish*, Hannah Lin*, Aditya K Kamath*, Anish Nyayachavadi
{vsathish, hannahyl, akkamath<EMAIL_ADDRESS>
###### Abstract

Large Language Models (LLMs) are a powerful technology that augment human
skill to create new opportunities, akin to the development of steam engines
and the internet. However, LLMs come with a high cost. They require
significant computing resources and energy to train and serve. Inequity in
their control and access has led to concentration of ownership and power to a
small collection of corporations. In our study, we collect training and
inference requirements for various LLMs. We then analyze the economic
strengths of nations and organizations in the context of developing and
serving these models. Additionally, we also look at whether individuals around
the world can access and use this emerging technology. We compare and contrast
these groups to show that these technologies are monopolized by a surprisingly
few entities. We conclude with a qualitative study on the ethical implications
of our findings and discuss future directions towards equity in LLM access.

##  1 Introduction

LLMs have become ubiquitous in AI research and are gradually permeating our
daily lives. We have seen LLMs impact developments in many different fields,
including journalism [1], medicine [2], biology [3], chemistry [4], law [5],
education [6, 7, 8], software engineering [9, 10] and robotics [11, 12]. Their
widespread adoption promises faster delivery of consumer products and
services. Analyzing the potential for generative AI to transform a wide range
of industries, Mckinsey & Company predict that LLMs will impact productivity
and the global economy by trillions of dollars [13]. However, will this
economic growth be experienced by all or only a few? We approach this question
by examining the economic accessibility of LLMs to various groups of people
around the world and by discussing the ethical implications of our findings.

With the emergence of the Transformer architecture [14] and its subsequent
language model BERT [15], LLMs have been increasing exponentially in size
[16]. Each larger model has led to new model capabilities, and recent scaling
laws have confirmed that increasing model size leads to more powerful LLMs
[17, 18]. However, this rapid growth comes at the cost of increasing energy
and hardware requirements. When models grow larger, more compute and memory is
needed to store model parameters and compute weight updates. Moreover, LLMs
require specialized hardware such as GPUs and TPUs, which come with exorbitant
price tags. As shown in Table 1, training GPT-4 took an estimated $100
million. Serving GPT-4 for inference requires roughly $900 per day. The high
compute costs of LLMs create a barrier to access for those with lower
financial resources.

Due to this financial barrier, future breakthroughs in training and serving
LLMs are restricted to a few small groups. For example, although journalism
has been shown to benefit from LLMs [19], newsrooms in Africa lack financial
resources to train and use dedicated LLMs [20]. Similarly, while some have
been able to fine-tune their own LLM for use in medical settings [21], many
hospitals do not have the adequate funding required to fine-tune such models
for large-scale use. Observing these disparities, we analyze the economic
resources of several groups around the world, and compare them to the costs of
developing and operating LLMs.

We choose to focus our work on a few key stakeholders: _Private sector
organizations and industries_ are the largest group of LLM developers that
will ultimately affect the global economy. Whether they effectively
incorporate and sell LLM-based services will depend on advancements in
_research institutions_. As products based off LLMs are released,
_individuals_ will start to form a significant portion of their user base. As
such, we look at the distribution of economic resources across (1) small and
large corporations, (2) research institutions, and (3) individuals by country.

Figure 1: We compare the costs of 6 LLM models ranging from small to large
with the economic resources of companies, research labs, and individuals.
(Logos from [22, 23, 24].)

Our work uses the GPU compute cost for these models as a proxy for their
practical deployment. We do not consider the costs for hiring engineers, data
scientists and miscellaneous overhead costs that may be necessary in training
or using an LLM. We start with this assumption from the observation that the
current costs for training LLMs far dwarf other operational costs. Our
contributions can be summarized as follows:

  * •

Tabulate the training and inference costs for a few common LLMs. Collect the
raw data for financial resources of a range of companies, research
institutions, and individuals, from reputed resources available on the
internet.

  * •

Compare the LLM training costs to the financial resources of companies and
research institutions to determine which groups have the ability to dictate
the future of this technology.

  * •

Analyze how accessible LLMs are to individuals of different countries based on
economic and language barriers.

  * •

Provide a discussion on ideas for increasing accessibility to LLMs globally.

We begin by discussing the related work on LLM equity in Section 2. We then
describe the methodology used for selecting models and populations to analyze
(Section 3). The results of our analysis are presented in Section 4, and we
follow with a discussion of the implications of our findings. We then briefly
mention our suggestions for next steps to diffuse the concentration of power
and bridge LLM equity across groups (Section 5).

##  2 Related Work

In this section, we briefly discuss prior research work around the wider topic
of LLMs and their socio-economic impact. Following the discussion in the
introduction, the widespread adoption of LLMs promise faster delivery of
consumer products and services powered by AI. This rapid growth comes at the
cost of increased energy consumption, hardware resource requirements and human
labor.

Public Availability: While the tech industry mega-corporations have
traditionally held a monopoly over these large-scale models, several
competitive, open-source alternatives have emerged. Both industry [25, 26, 27]
as well as collaborative efforts [28] have open-sourced LLMs aiming to improve
public access. However, these alternatives still require significant computing
and energy resources to match the state-of-the-art performance.

Cost and Energy: Several works [29, 30] have analyzed the financial costs
required to train and operate LLMs without delving deep into their long term
ethical implications. Other works [31, 32, 33] have looked at the energy costs
of these models and the potential environmental impact. Compared to these, we
analyze the cost and energy usage from a collective control perspective,
asking ethical questions such as, will the future of LLMs be democratic,
fostering a society where everyone can use and develop them? If not, which
groups of population and organizations determine who has access? and how will
this impact the broader society in the long run?

LLM Cost Reduction: Recently, several papers have analyzed methods to improve
the efficiency of serving LLMs. One such method is quantization which reduces
the precision of LLM weights [34]. While this reduces the required memory and
compute resources needed to serve LLMs [35], it also reduces accuracy and
precision, leading to concerns about model quality and equity. For example,
models that use quantization may be less powerful compared to models that do
not. In addition, quantization is only used for LLM serving, whereas training
still requires significant computational resources [36].

LLMs, Society, and Ethics: Attempts have been made in prior work to analyse
the influence of AI models in shaping society [37, 33]. Several works have
noted the dominance of English in Language Models, making them almost
exclusive to European and North American Societies [38, 39]. Ethical concerns
have been raised over the exposure of toxic online content among underpaid
moderators, and the significant impact on their mental health during the data
collection process for these LLM pipelines [40]. Moreover, AI technology is
believed to be moving towards a "Turing trap" with excessive focus on human-
like AI, with an eventual goal of replacing human labour. This process
centralises power towards those who control the technology [41]. Hence, it is
essential to analyse the distribution of LLM access across different socio-
economic groups to promote equity in access to this powerful technology. To
date, there has been limited research in this area [42]. Our project aims to
fill this gap by drawing relevant insights about the ethics of inequity of
this technology.

##  3 Methodology

In this section, we outline how we identify and analyze the financial
requirements needed for training and serving LLMs. We also detail our method
of quantifying distribution of financial resources around the world. We rely
entirely on published and open data.

Table 1: Details of LLMs used for our analysis. Model | Size (Params) | Release | Cost | Performance
---|---|---|---|---
Train ($) | Serve ($/day) | (MMLU %)
T5 [43] | Small (11B) | Oct 2019 | 2 Mil | 36 | 25.9
LLaMA-2 [26] | Small (7B) | July 2023 | 5.52 Mil | 18 | 45.3
GPT-3 [44] | Medium (175B) | June 2020 | 5 Mil | 180 | 43.9
BLOOM [28] | Medium (176B) | July 2022 | 5 Mil | 180 | 39.1
PaLM [45] | Large (540B) | April 2022 | 23.1 Mil | 500 | 71.3
GPT-4 [46] | Large (1000B) | March 2023 | 100 Mil | 900 | 86.4

###  3.1 Models

We focus on 6 different models spanning three categories: 1 Small ($\sim$10B
parameters), 2 Medium ($\sim$100B parameters), and 3 Large LLMs (>500B
parameters). We only consider models that we have observed to be commonly used
or cited by the general public.

Cost: Table 1 contains our list of LLMs and their associated costs. To
identify the training and serving costs of these models, we analyze the
resource data published by model creators. For unknown training costs, we base
our analysis on GPU rental prices on the cloud [47]. We use an optimistic
estimate of $\$1.5/GPU/hr$ [48, 49] for a single $A100$ GPU served on the
cloud, and multiply that with the GPU hours required for training the model.
For example, LLaMA-2 13B required over 368,000 GPU hours to train [50] making
the total training cost $368000*1.5\simeq\$5.52M$. An alternative approach
would be to calculate costs if organizations performed the training in-house.
This would require calculating the costs of purchasing GPUs, constructing a
datacenter, and operating the datacenter, which is complicated and error-
prone.

We estimate the inference costs by computing the memory required to load the
model, the number of A100 GPUs (80GB) required to satisfy the memory
constraints [51], and the daily cost of serving these models on the cloud. For
example, inference with GPT-3 requires 175 billion 16-bit floating point
parameters. The total number of A100s required would be $(175\
billion*(16/8))/80\simeq 5\ GPUs$. The cost of serving this for 1 day would be
$5*\$1.5*24=\$180$.

Performance: To compare the performance of these models, we rely on the MMLU
[52] benchmark. This benchmark consists of multiple choice tests spanning 57
different domains, intending to holistically capture the knowledge and
reasoning capabilities of different models. A score of 25% implies that a
model is as good as random chance.

Table 1 contains the publicly reported performance of T5 [53], GPT-3 [52],
BLOOM [54], LLaMA-2 [26], PaLM [53], and GPT-4 [46]. These numbers are
gathered from different sources and the exact evaluation setup may not match.
However, this gives us a rough scale to compare the models.

###  3.2 Populations

For this project, we choose stakeholders which we divide into three
categories:

  1. 1.

Businesses: As LLMs have powerful capabilities in a wide range of tasks, from
text translation [55, 56] to mathematical reasoning [57], many companies will
want to adopt LLMs and build businesses around them. Those with the resources
to train their own LLMs will hold a significant advantage in integrating AI
into their products over companies that do not have such resources.

  2. 2.

Research Institutions: The ability for researchers to participate in
development of NLP and AI algorithms ensure that LLMs properly serve their
communities. But this depends largely on the accessibility of LLMs for their
research. If LLMs are not accessible to localised researchers, it is difficult
for the overall development of LLM to align with the needs of specific
communities.

  3. 3.

Individuals: Many LLM-based products, such as ChatGPT, Gemini, and Bing Chat,
are extremely useful in everyday life. They have the power to significantly
increase the productivity of skilled workers in domains like writing,
journalism and even research. However, to access the powerful capabilities of
LLMs, individuals need to pay usage fees. The price tag that comes with LLM
inference plays a significant role in determining which individuals will be
end users of LLMs.

For businesses, we omit the larger corporations that already hold significant
monopoly and instead evaluate financial resources by comparing the venture
capital (VC) funding across different countries. VC funding allows us to
capture the financing power of small business that have high growth potential.
We extract our VC funding data from [58] and [59]. To examine the financial
resources of research institutions, we use national research budgets as a
proxy as obtained from the World Bank [60]. We acknowledge that research
budgets may not accurately describe the amount a country will spend on
developing LLMs as certain countries may prioritize other research goals, but
we begin with these numbers as a starting point for analyzing how likely a
country could invest in AI endeavours. We finally look at how accessible LLMs
would be to individuals of different nations. For this, we take the average
and median income of people around the world [61]. Similar to research
budgets, this allows us to see how likely the average citizen of the country
would have access to this technology.

##  4 Results

(a) Training costs as a fraction of VC funding.

(b) Countries and their VC funding for 2022.

(c) Training costs as a fraction of national research budgets.

(d) LLM subscription costs as a fraction of mean/median income.

Figure 2: Model training and subscription costs for startups, research
institutions.

###  4.1 Businesses

Figures 2(a) and 2(b) show VC funding budget compared to the costs of training
small, medium, and large LLMs. These figures focuses on early and late-stage
startups in Europe and North America in 2022. From Figure 2(a), we notice that
United States and Canada are the only nations that could train GPT-4 with less
than 25% of their total seed fund investments. In Figure 2(b), we use a log
scale to represent the investments on the Y-axis due to their highly skewed
nature. The United States receives over 100 times more VC funding than France
or Italy and is particularly capable of training the largest LLMs.

###  4.2 Research Institutions

We analyzed 149 countries whose national research budgets were available. To
avoid the impact of temporary economic downturns, we picked the largest amount
that each country spent on research within the span of 1996 - 2021. Figure
2(c) shows the fraction of the research budgets that these countries would
have to spend to be able to afford training the various models. 58 countries
would not be able to train GPT-4 even if they spent their entire research
budget. Practically, countries will have various different research
priorities, like healthcare, agriculture, etc., which will divert certain
amounts of their budget. Only 4 countries were able to afford training GPT-4
using less that 0.1% of their research budget, namely USA, China, Japan, and
Germany.

###  4.3 Individuals

Based on the subscription fee for ChatGPT Plus [62], we estimate the cost of
using LLM services to be $20/month. We compare this pricing to the median and
mean income of countries listed in [61]. If individuals are willing to place
their entire monthly salary into buying LLM services, people from all
countries (present in [61]) can afford LLM services on an average. However, it
is more realistic to estimate that individuals are willing to put around 10%
of their monthly income into subscription services according to recent
analysis [63]. At this rate, only individuals living in the top 56% of
countries can afford LLM services.

##  5 Discussion

(a) LLM training and inference viability across continents.

(b) MMLU performance of GPT-4 by language. Each country is shaded by its MMLU
score for its national language.

Figure 3: Distribution of model training and subscription costs and model
language capabilities around the world.

###  5.1 Who holds control of LLMs?

From our data in Section 4, we see that the majority of financial viability
for both training and operating LLMs are concentrated in wealthy nations, such
as the United States. Figure 3(a) combines data from Figure 2 by showing the
average amount remaining if countries spend up to a generous 50% of their VC
funding and research budgets on training models and if individuals spend up to
10% of their income on LLM subscriptions. The figure highlights the disparity
of financial viability between different countries and presents the
concentration of VC, research, and individual financial viability in Asian,
North American, and European countries. This inequality is also reflected in
the distribution of Massive Multitask Language Understand (MMLU) performance
around the world. The distribution of MMLU performance for GPT-4 in Figure
3(b) shows the majority of LLM language power lies in English-speaking
countries [46].

###  5.2 Expanding LLMs to All

#### Why concentration of LLM power is concerning:

The high compute costs of LLMs has led to a concentration of LLM control among
the most wealthy countries. This raises the question: In what ways have the
strong biases in access and control of these models impacted society? Upon
analysis, we find that despite efforts from several wealthy corporations to
produce LLMs that are equitable and fair, these statistical models have been
found to produce and amplify social biases [64]. Many LLMs hold cultural
values and political biases reflecting those of model trainers [65]. The
occurrence of hate speech, misinformation, and gender or cultural biases can
differ in LLMs depending on how training data is collected and how the model
is trained [66, 67]. If LLM training is constrained to a few most highly-
resourced countries, LLMs may misrepresent minorities and show biases against
those from less affluent countries.

Another important observation to consider in the discussion around equity is
that the top LLM-based services now are offered through payment plans. As
highlighted in Figure 2(d), these pay-walled services are not accessible to
all individuals. Pay-for-use LLMs may further increase the economic divide in
the world, especially given the high potential for LLMs to increase
productivity for users who have access [68]. If only the rich are able to
access LLMs and improve their productivity, will we exacerbate the current
inequality in the world economy?

#### Biases in language:

Figure 3(b) shows the MMLU performance of GPT-4, the latest and the most
powerful language model across different languages. MMLU is the measure of the
model’s multitask accuracy across 57 tasks [52]. Blue shows the regions (and
their official language) with the highest MMLU scores. It is clear from the
figure that English dominates as the language that performs best at multiple
language tasks. The disparity in language performance brings us to several
concerns. What are the implications of LLM impact on other highly used
languages like Urdu, Thai, Hindi, etc.? Would people using these languages not
be able to make effective use of the technology? What would the future look
like when most languages are left behind by such a powerful technology?

#### Expanding LLM accessibility:

Given the hurdle that LLM compute costs create for accessing LLMs and the
ethical concerns they bring, here we offer a few ideas on how we might
increase accessibility of LLMs.

_1\. Invest in developing methods to decrease the compute costs of LLMs._
Some research has explored how to improve LLM training and serving efficiency,
such as model quantization [34], distillation [69], and pruning [70]. We
encourage the community to continue investing in such methods and identifying
ways to further decrease LLM compute costs and to make AI more green [71]. It
is crucial for us to consider how we can increase accessibility for LLMs today
before expanding the models and making them more resource intensive and more
difficult for others to train and serve.

_2\. Encourage cross-cultural and international collaborations in development
of LLMs._
Much of LLM training today is concentrated within select companies within
certain countries (Figures 2, 3). We can lessen this by incorporating cross-
cultural co-design and collaboration. Cross-cultural development and research
can help create models with an increased understanding of the different
cultures of the end users. Increasing diversity in the LLM research ecosystem
may also help researchers gain a deeper understanding of compute constraints
within low-resourced communities and the potential impacts of decisions being
made during LLM training.

_3\. Advocate for policies and initiatives that increase fair access to LLMs_
Ultimately, changes in who holds power over LLMs will require policies and
intiatives that enforce fair practice and deployment of LLMs. We call the
community to look for ways to enact policies that will allow for increased LLM
access, such as ensuring LLM companies develop fair payment plans that
consider economic viability globally and not just among the wealthy.

###  5.3 Limitations

We chose to rely on existing datasets to glean financial data as outline in
Section 3, and our results only capture groups and countries covered in those
datasets. For individual income and academic research institutions, we take
aggregates. We acknowledge that the choice of categories and aggregation has
implications and that there are a wide variance of economic conditions within
each group. We encourage further research to better capture the nuances within
each group and their implications on LLM accessibility. With the introductory
analysis and awareness on LLM access discussed in this article, we hope to
provide an inspiring starting point for AI startups and researchers, to
address these issues more concretely.

##  6 Conclusion

We have found that access to training and inference for LLMs is highly
concentrated within the hands of a few countries. With the exorbitant costs of
training LLMs, companies, research institutions, and individuals with the
highest economic earnings represent the majority of those with abilities to
train LLMs. Similarly, costs for inference constrain using LLMs to those in
the top earning countries. This leads to a concerning concentration of LLM
power among a small group. We discuss several implications of the observed
inequity and pose important questions to consider looking forward. We propose
initial ideas for how to combat this monopolization. We also encourage the
research community to continue examining ways to decrease the costs of
training LLMs and to allow those in lower-resourced countries easier access to
LLM inference.

## References

  * [1] Zijian Ding et al.  “Harnessing the Power of LLMs: Evaluating Human-AI Text Co-Creation through the Lens of News Headline Generation”, 2023  arXiv:2310.10706 [cs.CL]
  * [2] Arun James Thirunavukarasu et al.  “Large language models in medicine”  Nature medicine, 2023
  * [3] Shreyas Bhat Brahmavar et al.  “Generating Novel Leads for Drug Discovery using LLMs with Logical Feedback”  In _bioRxiv_ Cold Spring Harbor Laboratory, 2023, pp. 2023–09
  * [4] Kevin Maik Jablonka et al.  “14 examples of how LLMs can transform materials science and chemistry: a reflection on a large language model hackathon”  In _Digital Discovery_ 2.5 Royal Society of Chemistry, 2023, pp. 1233–1250
  * [5] Jiaxi Cui et al.  “Chatlaw: Open-source legal large language model with integrated external knowledge bases”  In _arXiv preprint arXiv:2306.16092_ , 2023
  * [6] Steven Moore et al.  “Empowering education with llms-the next-gen interface and content generation”  In _International Conference on Artificial Intelligence in Education_ , 2023, pp. 32–37  Springer
  * [7] Changrong Xiao et al.  “Evaluating reading comprehension exercises generated by LLMs: A showcase of ChatGPT in education applications”  In _Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023)_ , 2023, pp. 610–625
  * [8] Runchu Tian et al.  “DebugBench: Evaluating Debugging Capability of Large Language Models”  In _arXiv preprint arXiv:2401.04621_ , 2024
  * [9] Xinyi Hou et al.  “Large language models for software engineering: A systematic literature review”  In _arXiv preprint arXiv:2308.10620_ , 2023
  * [10] Ramakrishna Bairi et al.  “Codeplan: Repository-level coding using llms and planning”  In _arXiv preprint arXiv:2309.12499_ , 2023
  * [11] Peng Shu et al.  “LLMs for Coding and Robotics Education”  In _arXiv preprint arXiv:2402.06116_ , 2024
  * [12] Fanlong Zeng et al.  “Large language models for robotics: A survey”  In _arXiv preprint arXiv:2311.07226_ , 2023
  * [13] Michael Chui et al.  “The economic potential of Generative AI: The Next Productivity Frontier”  In _McKinsey & Company_ McKinsey & Company, 2023  URL: https://www.mckinsey.com/capabilities/mckinsey-digital/our-insights/the-economic-potential-of-generative-AI-the-next-productivity-frontier#introduction
  * [14] Ashish Vaswani et al.  “Attention is All you Need”  In _Advances in Neural Information Processing Systems_ 30 Curran Associates, Inc., 2017  URL: https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf
  * [15] Jacob Devlin et al.  “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”, 2019  arXiv:1810.04805 [cs.CL]
  * [16] Shervin Minaee et al.  “Large Language Models: A Survey”  In _arXiv preprint arXiv:2402.06196_ , 2024
  * [17] Jared Kaplan et al.  “Scaling laws for neural language models”  In _arXiv preprint arXiv:2001.08361_ , 2020
  * [18] Jordan Hoffmann et al.  “Training compute-optimal large language models”  In _arXiv preprint arXiv:2203.15556_ , 2022
  * [19] Savvas Petridis et al.  “Anglekindling: Supporting journalistic angle ideation with large language models”  In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ , 2023, pp. 1–16
  * [20] Allen Munoriyarwa, Sarah Chiumbu and Gilbert Motsaathebe  “Artificial intelligence practices in everyday news production: The case of South Africa’s mainstream newsrooms”  In _Journalism Practice_ 17.7 Taylor & Francis, 2023, pp. 1374–1392
  * [21] Zeming Chen et al.  “MEDITRON-70B: Scaling Medical Pretraining for Large Language Models”, 2023  arXiv:2311.16079 [cs.CL]
  * [22] OpenAI  “Brand guidlines”  URL: https://openai.com/brand
  * [23] Wikipedia, the free encyclopedia  “Google PaLM Logo”, 2023  URL: https://en.m.wikipedia.org/wiki/File:Google_PaLM_Logo.svg/
  * [24] Meta  “Meta brand resources and guidelines”  URL: https://about.meta.com/brand/resources/meta/company-brand/
  * [25] Hugo Touvron et al.  “LLaMA: Open and Efficient Foundation Language Models”, 2023  arXiv:2302.13971 [cs.CL]
  * [26] Susan Zhang et al.  “OPT: Open Pre-trained Transformer Language Models”, 2022  arXiv:2205.01068 [cs.CL]
  * [27] Hugo Touvron et al.  “Llama 2: Open Foundation and Fine-Tuned Chat Models”, 2023  arXiv:2307.09288 [cs.CL]
  * [28] BigScience Workshop et al.  “BLOOM: A 176B-Parameter Open-Access Multilingual Language Model”, 2023  arXiv:2211.05100 [cs.CL]
  * [29] Wayne Xin Zhao et al.  “A Survey of Large Language Models”, 2023  arXiv:2303.18223 [cs.CL]
  * [30] Humza Naveed et al.  “A Comprehensive Overview of Large Language Models”, 2023  arXiv:2307.06435 [cs.CL]
  * [31] Ahmad Faiz et al.  “LLMCarbon: Modeling the end-to-end Carbon Footprint of Large Language Models”, 2024  arXiv:2309.14393 [cs.CL]
  * [32] Emma Strubell, Ananya Ganesh and Andrew McCallum  “Energy and Policy Considerations for Deep Learning in NLP”, 2019  arXiv:1906.02243 [cs.CL]
  * [33] Roy Schwartz et al.  “Green AI”  In _CoRR_ abs/1907.10597, 2019  arXiv: http://arxiv.org/abs/1907.10597
  * [34] Miguel Carreira Neves  “What are quantized llms?”  In _TensorOps_ TensorOps, 2023  URL: https://www.tensorops.ai/post/what-are-quantized-llms
  * [35] Yilong Zhao et al.  “Atom: Low-bit Quantization for Efficient and Accurate LLM Serving”, 2023  arXiv:2310.19102 [cs.LG]
  * [36] Tim Dettmers et al.  “QLoRA: Efficient Finetuning of Quantized LLMs”, 2023  arXiv:2305.14314 [cs.LG]
  * [37] Rishi Bommasani et al.  “On the Opportunities and Risks of Foundation Models”, 2022  arXiv:2108.07258 [cs.LG]
  * [38] Regina Ta and Nicol Turner Lee  “How language gaps constrain generative AI development”, https://www.brookings.edu/articles/how-language-gaps-constrain-generative-ai-development/
  * [39] Krithika Ramesh, Sunayana Sitaram and Monojit Choudhury  “Fairness in Language Models Beyond English: Gaps and Challenges”, 2023  arXiv:2302.12578 [cs.CL]
  * [40] Niamh Rowe  “‘It’s destroyed me completely’: Kenyan moderators decry toll of training of AI models”  The Guardian, 2023  URL: https://www.theguardian.com/technology/2023/aug/02/ai-chatbot-training-human-toll-content-moderator-meta-openai
  * [41] Erik Brynjolfsson  “The Turing Trap: The Promise and Peril of Human-Like Artificial Intelligence”, 2022  arXiv:2201.04200 [econ.GN]
  * [42] Emily M. Bender et al.  “On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?”  In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ , FAccT ’21  Virtual Event, Canada: Association for Computing Machinery, 2021, pp. 610–623  DOI: 10.1145/3442188.3445922
  * [43] Colin Raffel et al.  “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer”  In _CoRR_ abs/1910.10683, 2019  arXiv: http://arxiv.org/abs/1910.10683
  * [44] Tom B. Brown et al.  “Language Models are Few-Shot Learners”, 2020  arXiv:2005.14165 [cs.CL]
  * [45] Aakanksha Chowdhery et al.  “PaLM: Scaling Language Modeling with Pathways”, 2022  arXiv:2204.02311 [cs.CL]
  * [46] OpenAI et al.  “GPT-4 Technical Report”, 2023  arXiv:2303.08774 [cs.CL]
  * [47] Or Sharir, Barak Peleg and Yoav Shoham  “The Cost of Training NLP Models: A Concise Overview”  In _CoRR_ abs/2004.08900, 2020  arXiv: https://arxiv.org/abs/2004.08900
  * [48] “A100 GPU Cloud Availability and Pricing”  GPU Utils, 2024  URL: https://gpus.llm-utils.org/a100-gpu-cloud-availability-and-pricing/
  * [49] “On-demand GPU cloud pricing”  Lambda Labs, 2024  URL: https://lambdalabs.com/service/gpu-cloud#pricing
  * [50] “LLaMA2: Model Details”  Meta Research, 2024  URL: https://github.com/facebookresearch/llama/blob/main/MODEL_CARD.md
  * [51] “Optimizing LLMs for Speed and Memory”  HuggingFace, 2024  URL: https://huggingface.co/docs/transformers/v4.35.0/llm_tutorial_optimization
  * [52] Dan Hendrycks et al.  “Measuring Massive Multitask Language Understanding”, 2021  arXiv:2009.03300 [cs.CY]
  * [53] Hyung Won Chung et al.  “Scaling Instruction-Finetuned Language Models”, 2022  arXiv:2210.11416 [cs.LG]
  * [54] Shijie Wu et al.  “BloombergGPT: A Large Language Model for Finance”, 2023  arXiv:2303.17564 [cs.LG]
  * [55] Wenhao Zhu et al.  “Multilingual machine translation with large language models: Empirical results and analysis”  In _arXiv preprint arXiv:2304.04675_ , 2023
  * [56] Longyue Wang et al.  “Document-level machine translation with large language models”  In _arXiv preprint arXiv:2304.02210_ , 2023
  * [57] Shima Imani, Liang Du and Harsh Shrivastava  “MathPrompter: Mathematical Reasoning using Large Language Models”, 2023  arXiv:2303.05398 [cs.CL]
  * [58] “Venture capital investments (market statistics)”  OECD, 2022  URL: https://data-explorer.oecd.org/vis?lc=en&tm=invest&pg=0&snb=107&vw=tb&df[ds]=dsDisseminateFinalDMZ&df[id]=DSD_VC%40DF_VC_INV&df[ag]=OECD.SDD.TPS&df[vs]=1.0&pd=2007%2C&dq=...USD_EXC.A&ly[rw]=BUSINESS_DEVELOPMENT_STAGE&ly[cl]=TIME_PERIOD&ly[rs]=REF_AREA&to[TIME_PERIOD]=false
  * [59] “Locations funding heatmap)”  dealroom.co, 2023  URL: https://app.dealroom.co/curated-heatmaps/funding/location/f/growth_stages/
  * [60] “Research and development expenditure (% of GDP)”  World Bank, 2024  URL: https://data.worldbank.org/indicator/GB.XPD.RSDV.GD.ZS
  * [61] “Median Income by Country 2024”  World Population Review  URL: https://worldpopulationreview.com/country-rankings/median-income-by-country
  * [62] OpenAI  “Introducing ChatGPT Plus”  URL: https://openai.com/blog/chatgpt-plus
  * [63] “Subscription Service Statistics and Costs”  C+R Research  URL: https://www.crresearch.com/blog/subscription-service-statistics-and-costs/
  * [64] Isabel O. Gallegos et al.  “Bias and Fairness in Large Language Models: A Survey”, 2023  arXiv:2309.00770 [cs.CL]
  * [65] Ruibo Liu et al.  “Quantifying and alleviating political bias in language models”  In _Artificial Intelligence_ 304 Elsevier, 2022, pp. 103654
  * [66] Shangbin Feng et al.  “From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models”, 2023  arXiv:2305.08283 [cs.CL]
  * [67] Esin Durmus et al.  “Towards Measuring the Representation of Subjective Global Opinions in Language Models”, 2023  arXiv:2306.16388 [cs.CL]
  * [68] Jesse G Meyer et al.  “ChatGPT and large language models in academia: opportunities and challenges”  In _BioData Mining_ 16.1 Springer, 2023, pp. 20
  * [69] Jianping Gou et al.  “Knowledge distillation: A survey”  In _International Journal of Computer Vision_ 129 Springer, 2021, pp. 1789–1819
  * [70] Xinyin Ma, Gongfan Fang and Xinchao Wang  “Llm-pruner: On the structural pruning of large language models”  In _Advances in neural information processing systems_ 36, 2024
  * [71] Roy Schwartz et al.  “Green ai”  In _Communications of the ACM_ 63.12 ACM New York, NY, USA, 2020, pp. 54–63