# A Software Engineering Perspective on Testing Large Language Models:
Research, Practice, Tools and Benchmarks

Sinclair Hudson<EMAIL_ADDRESS>University of
TorontoTorontoOntarioCanada ,  Sophia Jit<EMAIL_ADDRESS>University of TorontoTorontoOntarioCanada ,  Boyue Caroline Hu
<EMAIL_ADDRESS>University of TorontoTorontoOntarioCanada and  Marsha
Chechik<EMAIL_ADDRESS>University of TorontoTorontoOntarioCanada

(2024)

###### Abstract.

Large Language Models (LLMs) are rapidly becoming ubiquitous both as stand-
alone tools and as components of current and future software systems. To
enable usage of LLMs in the high-stake or safety-critical systems of 2030,
they need to undergo rigorous testing. Software Engineering (SE) research on
testing Machine Learning (ML) components and ML-based systems has
systematically explored many topics such as test input generation and
robustness. We believe knowledge about tools, benchmarks, research and
practitioner views related to LLM testing needs to be similarly organized. To
this end, we present a taxonomy of LLM testing topics and conduct preliminary
studies of state of the art and practice approaches to research, open-source
tools and benchmarks for LLM testing, mapping results onto this taxonomy. Our
goal is to identify gaps requiring more research and engineering effort and
inspire a clearer communication between LLM practitioners and the SE research
community.

LLMs, Software Engineering for Machine Learning, Testing

††copyright: acmcopyright††journalyear: 2024††doi:
10.1145/1122445.1122456††conference: International Workshop on Software
Engineering in 2030; November 2024; Puerto Galinàs (Brazil)††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06

##  1\. Introduction

Large Language Models (LLMs) are a type of Machine Learning (ML) designed to
understand and generate human-like text. They are gaining interest from both
researchers and practitioners due to their outstanding performance in various
domains, such as language translation and medical text analysis (Chang et al.,
2023). Given the growing usage of LLMs as components in software systems, the
question of how to specify the expected properties of such systems (e.g.,
correctness or fairness) and effectively test them becomes very important.
Given the speed with which LLMs have taken center stage, successes and gaps in
LLM testing approaches and practices have not yet been systematized in a
clear, principled, and comprehensive way, leading to difficulties in applying
them in practice (Yang et al., 2024).

LLMs are a particular type of ML model, and testing of ML models has been
studied extensively in software engineering literature. Specifically, Zhang
et. al (Zhang et al., 2022) proposed a taxonomy of ML testing, with the
following categories: (1) testing workflows (how to test), e.g., test oracle
generation; (2) testing components (where to test), e.g., data and learning
program testing; (3) testing properties (what to test), e.g., privacy and
fairness; and (4) application scenarios, e.g., autonomous driving (see Fig.
1).

Our vision. The current research and industrial practices on LLM testing
should be organized through the SE lens, i.e., organizing existing methods
with topics of ML testing in SE. This organization should facilitate the
identification of gaps and support communication among stakeholders, including
LLM testing researchers, tool developers, and users. Ultimately, this should
contribute to the safer usage of LLMs in software in high-stake domains by
2030.

To validate our vision, we present a preliminary study of current research
methods, benchmarks, testing tools, and industry practices for LLM testing,
aiming to answer two research questions: (RQ1:) To what extent do current
open-source benchmarks, tools, and online discussions addressing LLM testing
cover research topics in the ML testing taxonomy? (RQ2:) What are the
remaining gaps preventing practitioners from effectively applying LLM testing
research?

The rest of this paper is organized as follows: We first extend the ML testing
taxonomy of (Zhang et al., 2022) with LLM-specific research topics and then
identify topics in ML testing not yet addressed by LLM testing (Sec. 2). Using
this taxonomy, we then provide a brief survey of open-source benchmarks and
tools (Sec. 3 and 4) and present a preliminary examination of LLM testing in
practice through discussions on Reddit, where practitioners share their
experiences with testing LLMs as well as post and answer questions about
testing (Sec. 5). We summarize our findings and our vision in Sec. 6.

##  2\. LLM Testing Research

for tree=my node, l sep+=5pt, grow’=east, edge=gray, thick, parent
anchor=east, child anchor=west, edge path= [draw, edge] (!u.parent anchor) –
+(10pt,0) —- (.child anchor)edge label; , if=isodd(n_children()) for children=
if=equal(n,(n_children(”!u”)+1)/2)calign with current [ML Testing [Testing
workflow (how to test) [Benchmarks, draw=blue [Composite benchmarks,
color=blue] [Multi-modal benchmarks, color=blue] [Specific benchmarks,
color=blue] [General benchmarks, color=blue] ] [Debug and repair] [Bug report
analysis] [Test prioritization and reduction] [Test adequacy generation] [Test
oracle identification, outer color=green!20, draw=blue] [Test input
generation, draw=blue]] [Testing components (where to test) [Framework
testing] [Learning program testing, draw=blue] [Data testing]] [Testing
properties (what to test) [Correctness, outer color=green!20, inner
color=orange!70, draw=blue [Test for Hallucination, outer color=green!20]]
[Privacy] [Interpretability] [Fairness, draw=blue] [Efficiency] [Robustness
and security, draw=blue] [Model relevance, outer color=orange!20]]
[Application Scenario [Natural language inference, draw=blue] [Machine
translation, draw=blue] [Autonomous driving, outer color=gray!20] [Natural
science & engineering, color=blue] [Social science, color=blue] [Medical
applications, color=blue] [Agent applications, color=blue] [Other
applications, color=blue] ] ]

Legend: ML testing topics, LLM-specific topics,  for tree=my node, l sep+=5pt,
grow’=east, edge=gray, thick [Covered by LLM research, draw=blue] for tree=my
node, l sep+=5pt, grow’=east, edge=gray, thick [Not relevant for LLMs, outer
color=gray!20] for tree=my node, l sep+=5pt, grow’=east, edge=gray, thick
[benchmark strengths, outer color=orange!20] for tree=my node, l sep+=5pt,
grow’=east, edge=gray, thick [open-source tool strengths, outer
color=green!20]

Figure 1.  Taxonomy of ML testing in SE (Zhang et al., 2022) with additional
LLM-specific topics.

In this section, we extend our taxonomy with research on LLM testing in Fig.
1. Due to the recent drastic increase in LLM-based research, we selected two
most recent comprehensive literature reviews on LLM evaluation (Chang et al.,
2023; Zhao et al., 2023).

ML Testing Topics Covered by LLM research. Several ML testing topics have been
explored by LLM research, shown as blue outline in Fig. 1. For testing
workflow, LLM-based applications are commonly evaluated with benchmarks (Chang
et al., 2023), synthetic data, and human annotations (Zhao et al., 2023)
corresponding to test input and oracle generation. Unlike ML testing, existing
work on LLM evaluation has primarily focused on learning program testing,
i.e., testing trained LLMs, and mostly testing w.r.t. their specific
applications (Chang et al., 2023), e.g., natural language processing,
reasoning, medical usage, etc. Testing properties studied by LLM research
include robustness, fairness, and trustworthiness (Chang et al., 2023). It has
been shown that LLMs can fabricate false information that sounds plausible,
i.e., ”hallucinate” (Huang et al., 2023). As a result, correctness testing
also checks whether the model outputs strictly factual information. Several
techniques have been developed for this, including factual consistency tests
(Laban et al., 2023; Li et al., 2023), uncertainty-based tests (Huang et al.,
2023), and black-box tests (Manakul et al., 2023; Adlakha et al., 2023; Luo et
al., 2023). Hallucination is also a studied failure mode in machine
translation, ”where translations are grammatically correct but unrelated to
the source sentence” (Müller et al., 2020; Raunak et al., 2021). This
definition differs slightly from hallucination in the context of LLMs, where
there is no source sentence and the truthfulness of the generated text is of
principle concern (Huang et al., 2023).

New Entries: LLM-specific Topics. In addition to “general” ML testing topics,
research in LLM testing has also explored topics specific to LLMs, shown as
blue text in Fig. 1. First, due to the capabilities of LLMs being suitable for
a large variety of tasks, LLMs have been tested for correctness, robustness,
fairness, trustworthiness (Chang et al., 2023). In addition, LLMs have been
deployed in systems for various other domains including medical and social
science, or other specific applications such as personality tests (Bodroza et
al., 2023).

Conclusion. Addressing RQ1, our initial review indicates that several SE
testing topics are relevant but have yet to receive sufficient attention in
LLM testing research.

Threats to Validity. We looked at only two literature reviews, although they
were comprehensive and recent at the time of writing.

##  3\. LLM Evaluation Benchmarks

Research in LLMs has primarily relied on benchmarks for evaluation. Popular
LLMs such as GPT-4 and Gemini use multiple benchmarks to communicate and
compare their performance across a wide range of tasks and domains (OpenAI et
al., 2024; Team et al., 2023). In this section, we analyze the OpenCompass
GitHub repository (Contributors, 2023), a popular collection of 76 publicly
available benchmarks, aiming to identify under-explored LLM testing topics.
OpenCompass was picked as the highest-starred repository of LLM benchmarks
found by searching ”llm evaluation” in GitHub.

Benchmark Coverage. The studied benchmarks included general benchmarks, e.g.,
SQuAD2.0 (Rajpurkar et al., 2018) developed for general language
understanding, _specific_ ones for particular tasks, e.g., CMB (Wang et al.,
2024b) developed for medical applications, and _multi-modal_ benchmarks for
tasks concerning both images and text, e.g., MMBench (Liu et al., 2023a) for
vision language models. 74 out of 76 benchmarks are dedicated to testing model
correctness across different capabilities including mathematical reasoning,
content summary, and code generation (Contributors, 2023). The remaining two
benchmarks, CrowS-Pairs (Nangia et al., 2020) and AdvGLUE (Wang et al., 2022),
test model fairness and robustness, respectively. Certain benchmarks such as
SciBench (Wang et al., 2024c), HumanEval (Chen et al., 2021), and CMB (Wang et
al., 2024b) can be used to assess model relevance to certain application
scenarios. There were no benchmarks in the collection dedicated to privacy,
security, efficiency, or interpretability. Some applications, e.g., open-ended
chatbots, require that the LLM be tested on a wide variety of tasks, to see
how well it responds. Composite benchmarks in OpenCompass, such as MMBench
(Liu et al., 2023a) and SummEdits (Laban et al., 2023), are collections of
different task-specific benchmarks, developed to address this challenge. By
providing consistent testing harnesses (Liu et al., 2023c; Srivastava et al.,
2023), composite benchmarks allow practitioners and researchers to understand
an LLM’s model relevance to open-ended applications.

Conclusion. Our analysis of OpenCompass determined that almost all benchmarks
are dedicated to testing model correctness and relevance to certain
application domains. Benchmarks for privacy, security, robustness, efficiency,
interpretability, and fairness do exist outside of OpenCompass (Wang et al.,
2024a), though more work is needed to increase their adoption and coverage. To
answer RQ1, benchmarks alone cover a few topics of the ML testing taxonomy
sufficiently.

Threats to Validity. We have considered only one benchmark collection. It is
by far the most popular on GitHub, and therefore representative of today’s
state of practice for benchmarks dedicated to LLM testing. OpenCompass focuses
on general LLM capabilities, and as a result benchmarks focused on specific
use-cases, such as CodeXGLUE (Lu et al., 2021) are not considered.

##  4\. Open-Source Testing Tools

Table 1.  Open-source tools covering the different topics of ML testing. Tools
analysed in March 2024, with coverage based on the tool’s official
documentation.

Tool | GitHub Stars |  Debug and repair |  Test prior. and reduct. |  Test adequacy generation |  Test oracle identification |  Test input generation |  Framework testing |  Learning program testing |  Data testing |  Privacy |  Interpretability |  Fairness |  Efficiency |  Robustness and security |  Model relevance |  Correctness |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
LangChain Evals (Chase, 2022) | 79674 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  Legend:
Giskard (AI, 2024b) | 2705 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  : focus of the tool, 3+ tests or features
promptfoo (Promptfoo, 2024) | 2141 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  : addressed by 1-2 tests or features
TruLens-Eval (TruEra, 2024) | 1456 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  : no related functionality in the tool
DeepEval (AI, 2024a) | 1416 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |

Open-source tools give practitioners access to testing techniques established
in research. In this section, we align the functionalities of open-source
tools with topics outlined in our taxonomy (Fig. 1) to pinpoint gaps in
applying LLM testing research in practice.

Tool Selection. Aiming to identify popular open-source testing tools used by
practitioners, we searched for “llm evaluation” on Github and then ranked the
results by the number of GitHub stars, which indicates practitioner interest
and usage. We only considered repositories with a focus on testing and
evaluation, excluding model zoos, tools for running benchmarks, and other LLM
repositories without prominent testing packages. We also limited our analysis
to generic tools for text-in, text-out applications that are applicable to a
variety of LLM deployments, such as summarization, question-answering, and
text generation. The search yielded eight LLM testing repositories with over
1000 stars and proper documentation, of which three contained only
functionalities implemented by other repositories and thus were excluded. Tbl.
1 lists the selected five tools and the taxonomy topics (see Fig. 1) they
address.

Strengths of Open-Source Tools. _Test oracle identification_ and _tests for
correctness_ were addressed by all tools studied, highlighted with green in
Fig. 1. _Test oracles_ have been a common focus of many open-source tools,
including regular expressions or other naive string matching functions
(Promptfoo, 2024; Chase, 2022). Additionally, LLMs or Retrieval Augmented
Generation (RAG) systems have been used as test oracles for testing LLMs,
i.e., “LLM-as-a-judge” or “GEval” (Liu et al., 2023b). The focus on test
oracles arises from the fact that the input and output of LLMs consist of
unrestricted natural language text. This transforms the task of identifying
the correct output into a natural language understanding challenge, involving
interpreting whether the output conveys the intended meaning (Liu et al.,
2023b). In terms of _correctness_ , LLM outputs can be considered incorrect
for multiple nuanced reasons, based on the requirements of the system. For
example, since both politeness and factual accuracy are important for the
correctness of a customer service LLM, responses that are rude but valid, or
polite but wrong are considered incorrect. Tests have been implemented to
check that responses contain specific, factual information and no
hallucinations (OpenAI, 2024b; Promptfoo, 2024; TruEra, 2024; AI, 2024a).
Additionally, there are tests dedicated to checking abstract writing
properties, such as ”concise”, ”uncontroversial”, or ”sensitive” (Chase,
2022). Finally, there are tests for checking whether the output conforms to a
specified structure and format (Chase, 2022; Promptfoo, 2024), for example,
JSON.

Gaps in Open-Source Tools. Several research topics were not addressed by any
of the studied tools. We denote them by empty circles in Tbl. 1. Out of them,
model relevance has been addressed by benchmarks (see Sec. 3). None of the
surveyed tools have implemented infrastructure to prioritize tests or skip
redundant tests, despite the inference of LLMs with millions of parameters
being very expensive. Therefore, test reduction and prioritization for LLMs
remains a promising area for future work, both in research and development.
Additionally, the original training corpora of the LLMs are often unknown to
LLM practitioners, and presumably immense in size, posing challenges for data,
learning program, and framework testing. Nevertheless, LLM practitioners and
researchers can still debug LLMs with tests and repair bugs with fine-tuning,
either through an API (OpenAI, 2024a) or locally with custom-built corpora and
learning programs (run.ai, 2024; HuggingFace, 2024). However, these workflows
are not implemented by any tools we studied, and thus are important directions
for future work.

Conclusion. By mapping open-source tools to topics on the taxonomy, we
discovered that test oracle identification and correctness are well-addressed
by open-source tools, even though comparisons between different test oracles
for different tasks are yet to be made. Furthermore, most other ML testing
topics remain unaddressed, namely, debugging and repair, test prioritization
and reduction, test adequacy generation, framework testing, learning program
testing, data testing, and model relevance. To answer RQ1 and RQ2, open-source
tools cover only a small portion of the topics in the ML testing taxonomy, and
this lack of tooling directly prevents practitioners from effectively applying
LLM testing research.

Threats to Validity. Our tool survey consists of only five of today’s most
popular tools, selected though only one search query on GitHub, and thus is
clearly incomplete. Furthermore, the functionalities of the tools are
determined solely based on documentation, potentially providing limited
insights compared to thorough experimentation and code review.

##  5\. Practice of LLM Testing

In this section, we examine how LLM testing is done in practice by analyzing
testing-related discussions on online forums. By comparing discussions with
topics from our taxonomy and testing tools, we identify potential gaps in
current testing practices.

Data. We focused on Reddit, which is one of the most popular online forums
(Eghtesadi and Florea, 2020), because its subreddits are issue-oriented forums
widely used and accessible to a broad audience (Krohn and Weninger, 2022).
Specifically, we selected the subreddit r/LocalLlama because of its singular
focus on LLMs and popularity on the website – since its creation in March
2023, it has become one of the most popular subreddits on the forum, ranking
among the top five percent in size111Although the stated objective of this
subreddit is to ”discuss about Llama, the large language model created by Meta
AI”, in practice, posts range from topics such as technology policy, news,
model and code development, to popular LLMs released by firms such as OpenAI
and Anthropic.. Subreddit submissions and their corresponding comments (from
March 2023 to January 2024) were collected from Academic Torrents
(stuck_in_the_matrix, [n.d.]; Watchful1, [n.d.]). To characterize
conversations occurring on the platform, submissions and comments were
combined, such that the unit of analysis is a subreddit thread. The final data
set consisted of 15,209 submissions and 11,377 comments across the subreddit,
which amounted to 11,344 threads222# of threads is greater than # submissions
since the threads account for partial conversations where the submissions may
have been removed from the data. (submissions and corresponding comments).

Testing-related Keywords. To examine how subreddit users from diverse
backgrounds discuss LLM testing, we first searched the data for keywords
associated with testing software systems (Braiek and Khomh, 2019; Grosse and
Duvenaud, 2014; Murphy et al., 2008; Xie et al., 2011; Dwarakanath et al.,
2018), machine learning (Goodfellow et al., 2014; Kurakin et al., 2016; Madry
et al., 2017; Tramèr et al., 2017), LLMs (Wang et al., 2024d), and the names
of the testing tools in Tbl. 1. These keywords were derived from the
corresponding literature, and thus differ from the taxonomy proposed in Fig.
1. Tbl. 3 reports the frequency and percent of the testing-specific keywords
found in the data. Keywords related to testing appeared in approximately 0.8%
of the total threads. Next, “unit test” (N=38) and notions of “toxicity”
(N=29) are the most frequently discussed testing keywords, while other
traditional software (e.g., “test oracle” (N = 0), “functional test” (N = 0))
and machine learning (e.g.,“adversarial attack” (N = 5), “adversarial example”
(N = 0)) testing keywords appeared in less than 0.05% of the data, if at all.
The testing tools in Tbl. 1 similarly proved unpopular amongst subreddit users
and collectively appeared in less than 0.02% of the data.

Analysis of Discussions. To better characterize the testing-related
conversations, we conducted semantic thematic analysis (Clarke and Braun,
2021) for identifying and analyzing patterns, themes, and insights within data
containing testing-related keywords. The quality of these conversations varied
across posts and ranged from general discussion of testing, to particular
methods for doing so in practice. We noticed that the former was particularly
obvious for “toxicity”, which was frequently mentioned in discussions of toxic
or biased model outputs and performance reports of newly released models. In
contrast, discussion of unit testing proved more actionable with many users
reporting the tests they personally developed to evaluate models or those
reported in the literature. Despite the relative popularity of unit testing on
the subreddit, there appeared to be a lack of coherent guidance or steps on
how to practically test LLMs. For example, in response to a post asking
“Prompt Engineering Seems Like Guesswork - How To Evaluate LLM Application
Properly?”, some users responded by noting methods such as unit tests, while
others reported processes they independently developed. For example, the quote
in panel A of Tbl. 2 notes a combinatorial testing approach with user
evaluation. In contrast, panel B highlights a hierarchical method developed by
another user which seemingly draws from both software engineering (e.g, unit
testing and user acceptance testing) and ML practices (e.g., evaluation on
benchmarks).

Table 2. Example Quotes from r/LocalLama.

A. Example of Combinatorial Testing |  B. Example of a Hierarchical Testing Method
---|---
“This is what I did. Write a set of prompts in the domains I’m interested in, a list of models, a list of generation presets. Run all combinations (prompts x models x presets) of settings, then rate outputs on a 4-point scale. I’m doing it the hard way and manually rating outputs. Then I can calculate relative scores for specific domains, models, presets, or any combination of them, and see what works best for what….” |  “I’ve been working in a similar setup (chatbot evaluation) for my company and our take is to consider 3 tiers of evaluations:  • Tier 1: evaluate predicted vs expected across a reference dataset, using classical textcomparison metrics… and semantic embedding similarity… • Tier 2: evaluate prediction against expected properties (ex: schema compliance, lack of toxicity, robustness, factuality)… • Tier 3: Look at user feedback (implicit or explicit) to see how your model is performing vis a vis the real user experience…”

Conclusion. The apparent novelty of many of the methods noted on r/LocalLlama
suggests both the utility of SE and ML approaches to testing LLMs, and
potential discrepancies between theory and practice. In addition to explicitly
drawing on methods such as unit and integration testing to test LLMs, the
methods proposed by users are ad-hoc and experimental. Interestingly, we
noticed that the discussions do not explicitly appeal to SE or ML
terminologies. Thus, the mentioned testing methods differ from both the
academic literature and the testing tools in Tab. 1. Our results show a large
gap between research and practice, which highlights the necessity for better
communication between researchers and practitioners.

Threats to Validity. The analysis of the subreddit, r/LocalLama, serves as a
preliminary examination of user discussions of LLM testing. Specifically, the
search terms used in the analysis were limited, potentially resulting in the
omission of relevant keywords pertinent to testing discussions. Additionally,
the diverse user group on the subreddit may not fully represent professionals
actively engaged in the testing field, thus impacting the generalizability of
the findings.

Table 3. Frequency and Proportion of Keywords in Data.

Testing Approaches | Testing Tools
---|---
Keyword _a_ | Frequency | Percentage _b_ | Keyword | Frequency | Percentage _a_
unit test | 38 | 0.33 | promptfoo | 1 | 0.01
toxicity | 29 | 0.26 | TruLens-Eval | 1 | 0.01
integration test | 8 | 0.07 | LangChain Evals | 0 | 0.00
conciseness | 6 | 0.05 | Giskard | 0 | 0.00
adversarial train | 5 | 0.04 | DeepEval | 0 | 0.00
adversarial attack | 5 | 0.04 |  |  |

_a_ Keywords _adversarial example/sample_ ,
_mutation/functional/metamorphic/property based/bias test_ , _test oracle_ and
_neuron coverage_ in the Testing Approaches have 0 percentage and thus are
omitted.
_b_ Percent of the total number of subreddit threads (N = 11,344).

##  6\. Conclusion and Future Outlook

In this paper, we proposed a vision for organizing and unifying efforts on LLM
testing from both research and industry practices through a taxonomy of ML
testing in SE. We presented a preliminary study on current research methods,
open-source benchmarks, testing tools, and online discussions by
practitioners.

Our results allow us to answer RQ1: a large portion of LLM testing research is
not yet incorporated into publicly available tools and benchmarks, leading to
future engineering and research directions. Regarding the answer to RQ2, we
noticed that developers online do not explicitly reference SE or ML testing
topics but are incorporating SE testing techniques into their practices.
Although preliminary, these results show promising prospects for organizing
existing methods using taxonomies such as the one in Fig. 1, identifying gaps
and future directions, and promoting communication and collaboration between
researchers and industry practitioners.

Our study has many limitations: not only did we analyze only _some_
benchmarks, tools and online discussion forums, the pace of change of SE
engineering will soon render our March 2024 snapshot and conclusions based on
it, obsolete. Future work needs to increase the scope of research
publications, tools, benchmarks and practitioner online discussions (and also
interview studies) as well as make frequent updates to the knowledge gathered.
As applications of LLMs continue to grow, in 2030, the taxonomy of LLM testing
should continue to expand, helping to connect research and industry practices.
Structuring tools and benchmarks following this taxonomy facilitates user
access to testing techniques tailored to their application requirements.
Ultimately, we hope our work will facilitate testing of LLMs in a clear,
principled, and comprehensive way, enabling usage of LLMs in high-stake or
safety-critical systems.

## Acknowledgement

The authors would like to thank the reviewers of Software Engineering 2030 for
their valuable feedback and comments.

## References

  * (1)
  * Adlakha et al. (2023) Vaibhav Adlakha, Parishad BehnamGhader, Xing Han Lu, Nicholas Meade, and Siva Reddy. 2023.  Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answering.  arXiv:2307.16877 [cs.CL]
  * AI (2024a) Confident AI. 2024a.  DeepEval.  https://github.com/confident-ai/deepeval?tab=readme-ov-file.
  * AI (2024b) Giskard AI. 2024b.  Giskard.  https://github.com/Giskard-AI/giskard?tab=readme-ov-file.
  * Bodroza et al. (2023) Bojana Bodroza, Bojana M. Dinic, and Ljubisa Bojic. 2023.  Personality testing of GPT-3: Limited temporal reliability, but highlighted social desirability of GPT-3’s personality instruments results.  arXiv:2306.04308 [cs.AI]
  * Braiek and Khomh (2019) Houssem Ben Braiek and Foutse Khomh. 2019.  TFCheck: A tensorflow library for detecting training issues in neural network programs. In _2019 IEEE 19th international conference on software quality, reliability and security (QRS)_. IEEE, 426–433.
  * Chang et al. (2023) Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. 2023.  A Survey on Evaluation of Large Language Models.  arXiv:2307.03109 [cs.CL]
  * Chase (2022) Harrison Chase. 2022.  LangChain.  https://github.com/langchain-ai/langchain
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021.  Evaluating Large Language Models Trained on Code.  _CoRR_ abs/2107.03374 (2021).  arXiv:2107.03374 https://arxiv.org/abs/2107.03374
  * Clarke and Braun (2021) Victoria Clarke and Virginia Braun. 2021.  Thematic analysis: a practical guide.  (2021).
  * Contributors (2023) OpenCompass Contributors. 2023.  OpenCompass: A Universal Evaluation Platform for Foundation Models.  https://github.com/open-compass/opencompass.
  * Dwarakanath et al. (2018) Anurag Dwarakanath, Manish Ahuja, Samarth Sikand, Raghotham M Rao, RP Jagadeesh Chandra Bose, Neville Dubash, and Sanjay Podder. 2018.  Identifying implementation bugs in machine learning based image classifiers using metamorphic testing. In _Proceedings of the 27th ACM SIGSOFT international symposium on software testing and analysis_. 118–128.
  * Eghtesadi and Florea (2020) Marzieh Eghtesadi and Adrian Florea. 2020.  Facebook, Instagram, Reddit and TikTok: a proposal for health authorities to integrate popular social media platforms in contingency planning amid a global pandemic outbreak.  _Canadian Journal of Public Health_ 111 (2020), 389–391.
  * Goodfellow et al. (2014) Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 2014.  Explaining and harnessing adversarial examples.  _arXiv preprint arXiv:1412.6572_ (2014).
  * Grosse and Duvenaud (2014) Roger B Grosse and David K Duvenaud. 2014.  Testing mcmc code.  _arXiv preprint arXiv:1412.5218_ (2014).
  * Huang et al. (2023) Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2023.  A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions.  arXiv:2311.05232 [cs.CL]
  * HuggingFace (2024) HuggingFace. 2024.  peft.  https://github.com/huggingface/peft.
  * Krohn and Weninger (2022) Rachel Krohn and Tim Weninger. 2022.  Subreddit links drive community creation and user engagement on Reddit. In _Proceedings of the International AAAI Conference on Web and Social Media_ , Vol. 16. 536–547.
  * Kurakin et al. (2016) Alexey Kurakin, Ian Goodfellow, and Samy Bengio. 2016.  Adversarial machine learning at scale.  _arXiv preprint arXiv:1611.01236_ (2016).
  * Laban et al. (2023) Philippe Laban, Wojciech Kryściński, Divyansh Agarwal, Alexander R. Fabbri, Caiming Xiong, Shafiq Joty, and Chien-Sheng Wu. 2023.  LLMs as Factual Reasoners: Insights from Existing Benchmarks and Beyond.  arXiv:2305.14540 [cs.CL]
  * Li et al. (2023) Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2023.  HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models.  arXiv:2305.11747 [cs.CL]
  * Liu et al. (2023c) Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. 2023c.  AgentBench: Evaluating LLMs as Agents.  _arXiv preprint arXiv: 2308.03688_ (2023).
  * Liu et al. (2023a) Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. 2023a.  MMBench: Is Your Multi-modal Model an All-around Player?  _arXiv:2307.06281_ (2023).
  * Liu et al. (2023b) Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023b.  G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment.  arXiv:2303.16634 [cs.CL]
  * Lu et al. (2021) Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu. 2021.  CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation.  arXiv:2102.04664 [cs.SE]
  * Luo et al. (2023) Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2023.  ChatGPT as a Factual Inconsistency Evaluator for Text Summarization.  arXiv:2303.15621 [cs.CL]
  * Madry et al. (2017) Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2017.  Towards deep learning models resistant to adversarial attacks.  _arXiv preprint arXiv:1706.06083_ (2017).
  * Manakul et al. (2023) Potsawee Manakul, Adian Liusie, and Mark J. F. Gales. 2023.  SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models.  arXiv:2303.08896 [cs.CL]
  * Müller et al. (2020) Mathias Müller, Annette Rios, and Rico Sennrich. 2020.  Domain Robustness in Neural Machine Translation. In _Proceedings of the 14th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track)_ , Michael Denkowski and Christian Federmann (Eds.). Association for Machine Translation in the Americas, Virtual, 151–164.  https://aclanthology.org/2020.amta-research.14
  * Murphy et al. (2008) Christian Murphy, Gail E Kaiser, and Lifeng Hu. 2008.  Properties of machine learning applications for use in metamorphic testing.  (2008).
  * Nangia et al. (2020) Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020.  CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Online.
  * OpenAI (2024a) OpenAI. 2024a.  _Fine-tuning_.  https://platform.openai.com/docs/guides/fine-tuning
  * OpenAI (2024b) OpenAI. 2024b.  OpenAI Evals.  https://github.com/openai/evals.
  * OpenAI et al. (2024) OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O’Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. 2024.  GPT-4 Technical Report.  arXiv:2303.08774 [cs.CL]
  * Promptfoo (2024) Promptfoo. 2024.  promptfoo.  https://github.com/promptfoo/promptfoo?tab=readme-ov-file.
  * Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018.  Know What You Don’t Know: Unanswerable Questions for SQuAD.  arXiv:1806.03822 [cs.CL]
  * Raunak et al. (2021) Vikas Raunak, Arul Menezes, and Marcin Junczys-Dowmunt. 2021.  The Curious Case of Hallucinations in Neural Machine Translation. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (Eds.). Association for Computational Linguistics, Online, 1172–1183.  https://doi.org/10.18653/v1/2021.naacl-main.92
  * run.ai (2024) run.ai. 2024.  _LLaMA 2 Fine Tuning_.  https://www.run.ai/guides/generative-ai/llama-2-fine-tuning#howto
  * Srivastava et al. (2023) Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane, Anantharaman S. Iyer, Anders Andreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhlmüller, Andrew Dai, Andrew La, Andrew Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakaş, B. Ryan Roberts, Bao Sheng Loe, Barret Zoph, Bartłomiej Bojanowski, Batuhan Özyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ekmekci, Bill Yuchen Lin, Blake Howald, Bryan Orinion, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Argueta, César Ferri Ramírez, Chandan Singh, Charles Rathkopf, Chenlin Meng, Chitta Baral, Chiyu Wu, Chris Callison-Burch, Chris Waites, Christian Voigt, Christopher D. Manning, Christopher Potts, Cindy Ramirez, Clara E. Rivera, Clemencia Siro, Colin Raffel, Courtney Ashcraft, Cristina Garbacea, Damien Sileo, Dan Garrette, Dan Hendrycks, Dan Kilman, Dan Roth, Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Moseguí González, Danielle Perszyk, Danny Hernandez, Danqi Chen, Daphne Ippolito, Dar Gilboa, David Dohan, David Drakard, David Jurgens, Debajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes, Diganta Misra, Dilyar Buzan, Dimitri Coelho Mollo, Diyi Yang, Dong-Ho Lee, Dylan Schrader, Ekaterina Shutova, Ekin Dogus Cubuk, Elad Segal, Eleanor Hagerman, Elizabeth Barnes, Elizabeth Donoway, Ellie Pavlick, Emanuele Rodola, Emma Lam, Eric Chu, Eric Tang, Erkut Erdem, Ernie Chang, Ethan A. Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Eunice Engefu Manyasi, Evgenii Zheltonozhskii, Fanyue Xia, Fatemeh Siar, Fernando Martínez-Plumed, Francesca Happé, Francois Chollet, Frieda Rong, Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Germán Kruszewski, Giambattista Parascandolo, Giorgio Mariani, Gloria Wang, Gonzalo Jaimovitch-López, Gregor Betz, Guy Gur-Ari, Hana Galijasevic, Hannah Kim, Hannah Rashkin, Hannaneh Hajishirzi, Harsh Mehta, Hayden Bogar, Henry Shevlin, Hinrich Schütze, Hiromu Yakura, Hongming Zhang, Hugh Mee Wong, Ian Ng, Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kernion, Jacob Hilton, Jaehoon Lee, Jaime Fernández Fisac, James B. Simon, James Koppel, James Zheng, James Zou, Jan Kocoń, Jana Thompson, Janelle Wingfield, Jared Kaplan, Jarema Radom, Jascha Sohl-Dickstein, Jason Phang, Jason Wei, Jason Yosinski, Jekaterina Novikova, Jelle Bosscher, Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel, Jesujoba Alabi, Jiacheng Xu, Jiaming Song, Jillian Tang, Joan Waweru, John Burden, John Miller, John U. Balis, Jonathan Batchelder, Jonathan Berant, Jörg Frohberg, Jos Rozen, Jose Hernandez-Orallo, Joseph Boudeman, Joseph Guerr, Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan, Katerina Ignatyeva, Katja Markert, Kaustubh D. Dhole, Kevin Gimpel, Kevin Omondi, Kory Mathewson, Kristen Chiafullo, Ksenia Shkaruta, Kumar Shridhar, Kyle McDonell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang, Liam Dugan, Lianhui Qin, Lidia Contreras-Ochando, Louis-Philippe Morency, Luca Moschella, Lucas Lam, Lucy Noble, Ludwig Schmidt, Luheng He, Luis Oliveros Colón, Luke Metz, Lütfi Kerem Şenel, Maarten Bosma, Maarten Sap, Maartje ter Hoeve, Maheen Farooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, Maria Jose Ramírez Quintana, Marie Tolkiehn, Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew L. Leavitt, Matthias Hagen, Mátyás Schubert, Medina Orduna Baitemirova, Melody Arnaud, Melvin McElrath, Michael A. Yee, Michael Cohen, Michael Gu, Michael Ivanitskiy, Michael Starritt, Michael Strube, Michał Swędrowski, Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Mitch Walker, Mo Tiwari, Mohit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, Mukund Varma T, Nanyun Peng, Nathan A. Chi, Nayeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts, Nick Doiron, Nicole Martinez, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish Shirish Keskar, Niveditha S. Iyer, Noah Constant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung, Paul Pu Liang, Paul Vicol, Pegah Alipoormolabashi, Peiyuan Liao, Percy Liang, Peter Chang, Peter Eckersley, Phu Mon Htut, Pinyu Hwang, Piotr Miłkowski, Piyush Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, Qing Lyu, Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Raefer Gabriel, Rahel Habacker, Ramon Risco, Raphaël Millière, Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymaekers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan LeBras, Rosanne Liu, Rowan Jacobs, Rui Zhang, Ruslan Salakhutdinov, Ryan Chi, Ryan Lee, Ryan Stovall, Ryan Teehan, Rylan Yang, Sahib Singh, Saif M. Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel R. Bowman, Samuel S. Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebastian Gehrmann, Sebastian Schuster, Sepideh Sadeghi, Shadi Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixiang Shane Gu, Shubh Pachchigar, Shubham Toshniwal, Shyam Upadhyay, Shyamolima, Debnath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo-Hwan Lee, Spencer Torene, Sriharsha Hatwar, Stanislas Dehaene, Stefan Divic, Stefano Ermon, Stella Biderman, Stephanie Lin, Stephen Prasad, Steven T. Piantadosi, Stuart M. Shieber, Summer Misherghi, Svetlana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq Ali, Tatsu Hashimoto, Te-Lin Wu, Théo Desbordes, Theodore Rothschild, Thomas Phan, Tianle Wang, Tiberius Nkinyili, Timo Schick, Timofei Kornev, Titus Tunduny, Tobias Gerstenberg, Trenton Chang, Trishala Neeraj, Tushar Khot, Tyler Shultz, Uri Shaham, Vedant Misra, Vera Demberg, Victoria Nyamai, Vikas Raunak, Vinay Ramasesh, Vinay Uday Prabhu, Vishakh Padmakumar, Vivek Srikumar, William Fedus, William Saunders, William Zhang, Wout Vossen, Xiang Ren, Xiaoyu Tong, Xinran Zhao, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yangqiu Song, Yasaman Bahri, Yejin Choi, Yichi Yang, Yiding Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yufang Hou, Yuntao Bai, Zachary Seid, Zhuoye Zhao, Zijian Wang, Zijie J. Wang, Zirui Wang, and Ziyi Wu. 2023.  Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models.  arXiv:2206.04615 [cs.CL]
  * stuck_in_the_matrix ([n.d.]) RaiderBDev stuck_in_the_matrix, Watchful1. [n.d.].  Reddit comments/submissions 2005-06 to 2023-12.  ([n. d.]).
  * Team et al. (2023) Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy Lillicrap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Isard, Paul R. Barham, Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, George Tucker, Enrique Piqueras, Maxim Krikun, Iain Barr, Nikolay Savinov, Ivo Danihelka, Becca Roelofs, Anaïs White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Mehran Kazemi, Lucas Gonzalez, Misha Khalman, Jakub Sygnowski, Alexandre Frechette, Charlotte Smith, Laura Culp, Lev Proleev, Yi Luan, Xi Chen, James Lottes, Nathan Schucher, Federico Lebron, Alban Rrustemi, Natalie Clay, Phil Crone, Tomas Kocisky, Jeffrey Zhao, Bartek Perz, Dian Yu, Heidi Howard, Adam Bloniarz, Jack W. Rae, Han Lu, Laurent Sifre, Marcello Maggioni, Fred Alcober, Dan Garrette, Megan Barnes, Shantanu Thakoor, Jacob Austin, Gabriel Barth-Maron, William Wong, Rishabh Joshi, Rahma Chaabouni, Deeni Fatiha, Arun Ahuja, Ruibo Liu, Yunxuan Li, Sarah Cogan, Jeremy Chen, Chao Jia, Chenjie Gu, Qiao Zhang, Jordan Grimstad, Ale Jakse Hartman, Martin Chadwick, Gaurav Singh Tomar, Xavier Garcia, Evan Senter, Emanuel Taropa, Thanumalayan Sankaranarayana Pillai, Jacob Devlin, Michael Laskin, Diego de Las Casas, Dasha Valter, Connie Tao, Lorenzo Blanco, Adrià Puigdomènech Badia, David Reitter, Mianna Chen, Jenny Brennan, Clara Rivera, Sergey Brin, Shariq Iqbal, Gabriela Surita, Jane Labanowski, Abhi Rao, Stephanie Winkler, Emilio Parisotto, Yiming Gu, Kate Olszewska, Yujing Zhang, Ravi Addanki, Antoine Miech, Annie Louis, Laurent El Shafey, Denis Teplyashin, Geoff Brown, Elliot Catt, Nithya Attaluri, Jan Balaguer, Jackie Xiang, Pidong Wang, Zoe Ashwood, Anton Briukhov, Albert Webson, Sanjay Ganapathy, Smit Sanghavi, Ajay Kannan, Ming-Wei Chang, Axel Stjerngren, Josip Djolonga, Yuting Sun, Ankur Bapna, Matthew Aitchison, Pedram Pejman, Henryk Michalewski, Tianhe Yu, Cindy Wang, Juliette Love, Junwhan Ahn, Dawn Bloxwich, Kehang Han, Peter Humphreys, Thibault Sellam, James Bradbury, Varun Godbole, Sina Samangooei, Bogdan Damoc, Alex Kaskasoli, Sébastien M. R. Arnold, Vijay Vasudevan, Shubham Agrawal, Jason Riesa, Dmitry Lepikhin, Richard Tanburn, Srivatsan Srinivasan, Hyeontaek Lim, Sarah Hodkinson, Pranav Shyam, Johan Ferret, Steven Hand, Ankush Garg, Tom Le Paine, Jian Li, Yujia Li, Minh Giang, Alexander Neitz, Zaheer Abbas, Sarah York, Machel Reid, Elizabeth Cole, Aakanksha Chowdhery, Dipanjan Das, Dominika Rogozińska, Vitaly Nikolaev, Pablo Sprechmann, Zachary Nado, Lukas Zilka, Flavien Prost, Luheng He, Marianne Monteiro, Gaurav Mishra, Chris Welty, Josh Newlan, Dawei Jia, Miltiadis Allamanis, Clara Huiyi Hu, Raoul de Liedekerke, Justin Gilmer, Carl Saroufim, Shruti Rijhwani, Shaobo Hou, Disha Shrivastava, Anirudh Baddepudi, Alex Goldin, Adnan Ozturel, Albin Cassirer, Yunhan Xu, Daniel Sohn, Devendra Sachan, Reinald Kim Amplayo, Craig Swanson, Dessie Petrova, Shashi Narayan, Arthur Guez, Siddhartha Brahma, Jessica Landon, Miteyan Patel, Ruizhe Zhao, Kevin Villela, Luyu Wang, Wenhao Jia, Matthew Rahtz, Mai Giménez, Legg Yeung, Hanzhao Lin, James Keeling, Petko Georgiev, Diana Mincu, Boxi Wu, Salem Haykal, Rachel Saputro, Kiran Vodrahalli, James Qin, Zeynep Cankara, Abhanshu Sharma, Nick Fernando, Will Hawkins, Behnam Neyshabur, Solomon Kim, Adrian Hutter, Priyanka Agrawal, Alex Castro-Ros, George van den Driessche, Tao Wang, Fan Yang, Shuo yiin Chang, Paul Komarek, Ross McIlroy, Mario Lučić, Guodong Zhang, Wael Farhan, Michael Sharman, Paul Natsev, Paul Michel, Yong Cheng, Yamini Bansal, Siyuan Qiao, Kris Cao, Siamak Shakeri, Christina Butterfield, Justin Chung, Paul Kishan Rubenstein, Shivani Agrawal, Arthur Mensch, Kedar Soparkar, Karel Lenc, Timothy Chung, Aedan Pope, Loren Maggiore, Jackie Kay, Priya Jhakra, Shibo Wang, Joshua Maynez, Mary Phuong, Taylor Tobin, Andrea Tacchetti, Maja Trebacz, Kevin Robinson, Yash Katariya, Sebastian Riedel, Paige Bailey, Kefan Xiao, Nimesh Ghelani, Lora Aroyo, Ambrose Slone, Neil Houlsby, Xuehan Xiong, Zhen Yang, Elena Gribovskaya, Jonas Adler, Mateo Wirth, Lisa Lee, Music Li, Thais Kagohara, Jay Pavagadhi, Sophie Bridgers, Anna Bortsova, Sanjay Ghemawat, Zafarali Ahmed, Tianqi Liu, Richard Powell, Vijay Bolina, Mariko Iinuma, Polina Zablotskaia, James Besley, Da-Woon Chung, Timothy Dozat, Ramona Comanescu, Xiance Si, Jeremy Greer, Guolong Su, Martin Polacek, Raphaël Lopez Kaufman, Simon Tokumine, Hexiang Hu, Elena Buchatskaya, Yingjie Miao, Mohamed Elhawaty, Aditya Siddhant, Nenad Tomasev, Jinwei Xing, Christina Greer, Helen Miller, Shereen Ashraf, Aurko Roy, Zizhao Zhang, Ada Ma, Angelos Filos, Milos Besta, Rory Blevins, Ted Klimenko, Chih-Kuan Yeh, Soravit Changpinyo, Jiaqi Mu, Oscar Chang, Mantas Pajarskas, Carrie Muir, Vered Cohen, Charline Le Lan, Krishna Haridasan, Amit Marathe, Steven Hansen, Sholto Douglas, Rajkumar Samuel, Mingqiu Wang, Sophia Austin, Chang Lan, Jiepu Jiang, Justin Chiu, Jaime Alonso Lorenzo, Lars Lowe Sjösund, Sébastien Cevey, Zach Gleicher, Thi Avrahami, Anudhyan Boral, Hansa Srinivasan, Vittorio Selo, Rhys May, Konstantinos Aisopos, Léonard Hussenot, Livio Baldini Soares, Kate Baumli, Michael B. Chang, Adrià Recasens, Ben Caine, Alexander Pritzel, Filip Pavetic, Fabio Pardo, Anita Gergely, Justin Frye, Vinay Ramasesh, Dan Horgan, Kartikeya Badola, Nora Kassner, Subhrajit Roy, Ethan Dyer, Víctor Campos, Alex Tomala, Yunhao Tang, Dalia El Badawy, Elspeth White, Basil Mustafa, Oran Lang, Abhishek Jindal, Sharad Vikram, Zhitao Gong, Sergi Caelles, Ross Hemsley, Gregory Thornton, Fangxiaoyu Feng, Wojciech Stokowiec, Ce Zheng, Phoebe Thacker, Çağlar Ünlü, Zhishuai Zhang, Mohammad Saleh, James Svensson, Max Bileschi, Piyush Patil, Ankesh Anand, Roman Ring, Katerina Tsihlas, Arpi Vezer, Marco Selvi, Toby Shevlane, Mikel Rodriguez, Tom Kwiatkowski, Samira Daruki, Keran Rong, Allan Dafoe, Nicholas FitzGerald, Keren Gu-Lemberg, Mina Khan, Lisa Anne Hendricks, Marie Pellat, Vladimir Feinberg, James Cobon-Kerr, Tara Sainath, Maribeth Rauh, Sayed Hadi Hashemi, Richard Ives, Yana Hasson, YaGuang Li, Eric Noland, Yuan Cao, Nathan Byrd, Le Hou, Qingze Wang, Thibault Sottiaux, Michela Paganini, Jean-Baptiste Lespiau, Alexandre Moufarek, Samer Hassan, Kaushik Shivakumar, Joost van Amersfoort, Amol Mandhane, Pratik Joshi, Anirudh Goyal, Matthew Tung, Andrew Brock, Hannah Sheahan, Vedant Misra, Cheng Li, Nemanja Rakićević, Mostafa Dehghani, Fangyu Liu, Sid Mittal, Junhyuk Oh, Seb Noury, Eren Sezener, Fantine Huot, Matthew Lamm, Nicola De Cao, Charlie Chen, Gamaleldin Elsayed, Ed Chi, Mahdis Mahdieh, Ian Tenney, Nan Hua, Ivan Petrychenko, Patrick Kane, Dylan Scandinaro, Rishub Jain, Jonathan Uesato, Romina Datta, Adam Sadovsky, Oskar Bunyan, Dominik Rabiej, Shimu Wu, John Zhang, Gautam Vasudevan, Edouard Leurent, Mahmoud Alnahlawi, Ionut Georgescu, Nan Wei, Ivy Zheng, Betty Chan, Pam G Rabinovitch, Piotr Stanczyk, Ye Zhang, David Steiner, Subhajit Naskar, Michael Azzam, Matthew Johnson, Adam Paszke, Chung-Cheng Chiu, Jaume Sanchez Elias, Afroz Mohiuddin, Faizan Muhammad, Jin Miao, Andrew Lee, Nino Vieillard, Sahitya Potluri, Jane Park, Elnaz Davoodi, Jiageng Zhang, Jeff Stanway, Drew Garmon, Abhijit Karmarkar, Zhe Dong, Jong Lee, Aviral Kumar, Luowei Zhou, Jonathan Evens, William Isaac, Zhe Chen, Johnson Jia, Anselm Levskaya, Zhenkai Zhu, Chris Gorgolewski, Peter Grabowski, Yu Mao, Alberto Magni, Kaisheng Yao, Javier Snaider, Norman Casagrande, Paul Suganthan, Evan Palmer, Geoffrey Irving, Edward Loper, Manaal Faruqui, Isha Arkatkar, Nanxin Chen, Izhak Shafran, Michael Fink, Alfonso Castaño, Irene Giannoumis, Wooyeol Kim, Mikołaj Rybiński, Ashwin Sreevatsa, Jennifer Prendki, David Soergel, Adrian Goedeckemeyer, Willi Gierke, Mohsen Jafari, Meenu Gaba, Jeremy Wiesner, Diana Gage Wright, Yawen Wei, Harsha Vashisht, Yana Kulizhskaya, Jay Hoover, Maigo Le, Lu Li, Chimezie Iwuanyanwu, Lu Liu, Kevin Ramirez, Andrey Khorlin, Albert Cui, Tian LIN, Marin Georgiev, Marcus Wu, Ricardo Aguilar, Keith Pallo, Abhishek Chakladar, Alena Repina, Xihui Wu, Tom van der Weide, Priya Ponnapalli, Caroline Kaplan, Jiri Simsa, Shuangfeng Li, Olivier Dousse, Fan Yang, Jeff Piper, Nathan Ie, Minnie Lui, Rama Pasumarthi, Nathan Lintz, Anitha Vijayakumar, Lam Nguyen Thiet, Daniel Andor, Pedro Valenzuela, Cosmin Paduraru, Daiyi Peng, Katherine Lee, Shuyuan Zhang, Somer Greene, Duc Dung Nguyen, Paula Kurylowicz, Sarmishta Velury, Sebastian Krause, Cassidy Hardin, Lucas Dixon, Lili Janzer, Kiam Choo, Ziqiang Feng, Biao Zhang, Achintya Singhal, Tejasi Latkar, Mingyang Zhang, Quoc Le, Elena Allica Abellan, Dayou Du, Dan McKinnon, Natasha Antropova, Tolga Bolukbasi, Orgad Keller, David Reid, Daniel Finchelstein, Maria Abi Raad, Remi Crocker, Peter Hawkins, Robert Dadashi, Colin Gaffney, Sid Lall, Ken Franko, Egor Filonov, Anna Bulanova, Rémi Leblond, Vikas Yadav, Shirley Chung, Harry Askham, Luis C. Cobo, Kelvin Xu, Felix Fischer, Jun Xu, Christina Sorokin, Chris Alberti, Chu-Cheng Lin, Colin Evans, Hao Zhou, Alek Dimitriev, Hannah Forbes, Dylan Banarse, Zora Tung, Jeremiah Liu, Mark Omernick, Colton Bishop, Chintu Kumar, Rachel Sterneck, Ryan Foley, Rohan Jain, Swaroop Mishra, Jiawei Xia, Taylor Bos, Geoffrey Cideron, Ehsan Amid, Francesco Piccinno, Xingyu Wang, Praseem Banzal, Petru Gurita, Hila Noga, Premal Shah, Daniel J. Mankowitz, Alex Polozov, Nate Kushman, Victoria Krakovna, Sasha Brown, MohammadHossein Bateni, Dennis Duan, Vlad Firoiu, Meghana Thotakuri, Tom Natan, Anhad Mohananey, Matthieu Geist, Sidharth Mudgal, Sertan Girgin, Hui Li, Jiayu Ye, Ofir Roval, Reiko Tojo, Michael Kwong, James Lee-Thorp, Christopher Yew, Quan Yuan, Sumit Bagri, Danila Sinopalnikov, Sabela Ramos, John Mellor, Abhishek Sharma, Aliaksei Severyn, Jonathan Lai, Kathy Wu, Heng-Tze Cheng, David Miller, Nicolas Sonnerat, Denis Vnukov, Rory Greig, Jennifer Beattie, Emily Caveness, Libin Bai, Julian Eisenschlos, Alex Korchemniy, Tomy Tsai, Mimi Jasarevic, Weize Kong, Phuong Dao, Zeyu Zheng, Frederick Liu, Fan Yang, Rui Zhu, Mark Geller, Tian Huey Teh, Jason Sanmiya, Evgeny Gladchenko, Nejc Trdin, Andrei Sozanschi, Daniel Toyama, Evan Rosen, Sasan Tavakkol, Linting Xue, Chen Elkind, Oliver Woodman, John Carpenter, George Papamakarios, Rupert Kemp, Sushant Kafle, Tanya Grunina, Rishika Sinha, Alice Talbert, Abhimanyu Goyal, Diane Wu, Denese Owusu-Afriyie, Cosmo Du, Chloe Thornton, Jordi Pont-Tuset, Pradyumna Narayana, Jing Li, Sabaer Fatehi, John Wieting, Omar Ajmeri, Benigno Uria, Tao Zhu, Yeongil Ko, Laura Knight, Amélie Héliou, Ning Niu, Shane Gu, Chenxi Pang, Dustin Tran, Yeqing Li, Nir Levine, Ariel Stolovich, Norbert Kalb, Rebeca Santamaria-Fernandez, Sonam Goenka, Wenny Yustalim, Robin Strudel, Ali Elqursh, Balaji Lakshminarayanan, Charlie Deck, Shyam Upadhyay, Hyo Lee, Mike Dusenberry, Zonglin Li, Xuezhi Wang, Kyle Levin, Raphael Hoffmann, Dan Holtmann-Rice, Olivier Bachem, Summer Yue, Sho Arora, Eric Malmi, Daniil Mirylenka, Qijun Tan, Christy Koh, Soheil Hassas Yeganeh, Siim Põder, Steven Zheng, Francesco Pongetti, Mukarram Tariq, Yanhua Sun, Lucian Ionita, Mojtaba Seyedhosseini, Pouya Tafti, Ragha Kotikalapudi, Zhiyu Liu, Anmol Gulati, Jasmine Liu, Xinyu Ye, Bart Chrzaszcz, Lily Wang, Nikhil Sethi, Tianrun Li, Ben Brown, Shreya Singh, Wei Fan, Aaron Parisi, Joe Stanton, Chenkai Kuang, Vinod Koverkathu, Christopher A. Choquette-Choo, Yunjie Li, TJ Lu, Abe Ittycheriah, Prakash Shroff, Pei Sun, Mani Varadarajan, Sanaz Bahargam, Rob Willoughby, David Gaddy, Ishita Dasgupta, Guillaume Desjardins, Marco Cornero, Brona Robenek, Bhavishya Mittal, Ben Albrecht, Ashish Shenoy, Fedor Moiseev, Henrik Jacobsson, Alireza Ghaffarkhah, Morgane Rivière, Alanna Walton, Clément Crepy, Alicia Parrish, Yuan Liu, Zongwei Zhou, Clement Farabet, Carey Radebaugh, Praveen Srinivasan, Claudia van der Salm, Andreas Fidjeland, Salvatore Scellato, Eri Latorre-Chimoto, Hanna Klimczak-Plucińska, David Bridson, Dario de Cesare, Tom Hudson, Piermaria Mendolicchio, Lexi Walker, Alex Morris, Ivo Penchev, Matthew Mauger, Alexey Guseynov, Alison Reid, Seth Odoom, Lucia Loher, Victor Cotruta, Madhavi Yenugula, Dominik Grewe, Anastasia Petrushkina, Tom Duerig, Antonio Sanchez, Steve Yadlowsky, Amy Shen, Amir Globerson, Adam Kurzrok, Lynette Webb, Sahil Dua, Dong Li, Preethi Lahoti, Surya Bhupatiraju, Dan Hurt, Haroon Qureshi, Ananth Agarwal, Tomer Shani, Matan Eyal, Anuj Khare, Shreyas Rammohan Belle, Lei Wang, Chetan Tekur, Mihir Sanjay Kale, Jinliang Wei, Ruoxin Sang, Brennan Saeta, Tyler Liechty, Yi Sun, Yao Zhao, Stephan Lee, Pandu Nayak, Doug Fritz, Manish Reddy Vuyyuru, John Aslanides, Nidhi Vyas, Martin Wicke, Xiao Ma, Taylan Bilal, Evgenii Eltyshev, Daniel Balle, Nina Martin, Hardie Cate, James Manyika, Keyvan Amiri, Yelin Kim, Xi Xiong, Kai Kang, Florian Luisier, Nilesh Tripuraneni, David Madras, Mandy Guo, Austin Waters, Oliver Wang, Joshua Ainslie, Jason Baldridge, Han Zhang, Garima Pruthi, Jakob Bauer, Feng Yang, Riham Mansour, Jason Gelman, Yang Xu, George Polovets, Ji Liu, Honglong Cai, Warren Chen, XiangHai Sheng, Emily Xue, Sherjil Ozair, Adams Yu, Christof Angermueller, Xiaowei Li, Weiren Wang, Julia Wiesinger, Emmanouil Koukoumidis, Yuan Tian, Anand Iyer, Madhu Gurumurthy, Mark Goldenson, Parashar Shah, MK Blake, Hongkun Yu, Anthony Urbanowicz, Jennimaria Palomaki, Chrisantha Fernando, Kevin Brooks, Ken Durden, Harsh Mehta, Nikola Momchev, Elahe Rahimtoroghi, Maria Georgaki, Amit Raul, Sebastian Ruder, Morgan Redshaw, Jinhyuk Lee, Komal Jalan, Dinghua Li, Ginger Perng, Blake Hechtman, Parker Schuh, Milad Nasr, Mia Chen, Kieran Milan, Vladimir Mikulik, Trevor Strohman, Juliana Franco, Tim Green, Demis Hassabis, Koray Kavukcuoglu, Jeffrey Dean, and Oriol Vinyals. 2023.  Gemini: A Family of Highly Capable Multimodal Models.  arXiv:2312.11805 [cs.CL]
  * Tramèr et al. (2017) Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick McDaniel. 2017.  Ensemble adversarial training: Attacks and defenses.  _arXiv preprint arXiv:1705.07204_ (2017).
  * TruEra (2024) TruEra. 2024.  TruLens.  https://github.com/truera/trulens.
  * Wang et al. (2024a) Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, Sang T. Truong, Simran Arora, Mantas Mazeika, Dan Hendrycks, Zinan Lin, Yu Cheng, Sanmi Koyejo, Dawn Song, and Bo Li. 2024a.  DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models.  arXiv:2306.11698 [cs.CL]
  * Wang et al. (2022) Boxin Wang, Chejian Xu, Shuohang Wang, Zhe Gan, Yu Cheng, Jianfeng Gao, Ahmed Hassan Awadallah, and Bo Li. 2022.  Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models.  arXiv:2111.02840 [cs.CL]
  * Wang et al. (2024d) Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, and Qing Wang. 2024d.  Software testing with large language models: Survey, landscape, and vision.  _IEEE Transactions on Software Engineering_ (2024).
  * Wang et al. (2024b) Xidong Wang, Guiming Hardy Chen, Dingjie Song, Zhiyi Zhang, Zhihong Chen, Qingying Xiao, Feng Jiang, Jianquan Li, Xiang Wan, Benyou Wang, and Haizhou Li. 2024b.  CMB: A Comprehensive Medical Benchmark in Chinese.  arXiv:2308.08833 [cs.CL]
  * Wang et al. (2024c) Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R. Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. 2024c.  SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models.  arXiv:2307.10635 [cs.CL]
  * Watchful1 ([n.d.]) RaiderBDev Watchful1. [n.d.].  Reddit comments/submissions 2024-01.  ([n. d.]).
  * Xie et al. (2011) Xiaoyuan Xie, Joshua WK Ho, Christian Murphy, Gail Kaiser, Baowen Xu, and Tsong Yueh Chen. 2011.  Testing and validating machine learning classifiers by metamorphic testing.  _Journal of Systems and Software_ 84, 4 (2011), 544–558.
  * Yang et al. (2024) Jingfeng Yang, Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng, Haoming Jiang, Shaochen Zhong, Bing Yin, and Xia Hu. 2024.  Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond.  _ACM Trans. Knowl. Discov. Data_ (feb 2024).  https://doi.org/10.1145/3649506 Just Accepted.
  * Zhang et al. (2022) Jie M. Zhang, Mark Harman, Lei Ma, and Yang Liu. 2022.  Machine Learning Testing: Survey, Landscapes and Horizons.  _IEEE Trans. Software Eng._ 48, 2 (2022), 1–36.  https://doi.org/10.1109/TSE.2019.2962027
  * Zhao et al. (2023) Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. 2023.  A Survey of Large Language Models.  arXiv:2303.18223 [cs.CL]


#  Physical effects of gravitational waves: pedagogical examples

Hayato Motohashi  Division of Liberal Arts, Kogakuin University, 2665-1
Nakano-machi, Hachioji, Tokyo 192-0015, Japan  Teruaki Suyama  Department of
Physics, Tokyo Institute of Technology, 2-12-1 Ookayama, Meguro-ku, Tokyo
152-8551, Japan

###### Abstract

General relativity describes gravitation in terms of the geometry of
spacetime. It predicts the existence of gravitational waves (GWs) that stretch
and compress spacetime and were detected recently by state-of-the-art
interferometer observations. Yet, for those who are not familiar with general
relativity, it may be difficult to understand how the GWs actually stretch and
compress spacetime. In this paper, after reviewing the fact that the effects
of GWs can be understood as a force in Newtonian mechanics, we consider
extreme and readily perceivable situations where large-amplitude GWs pass
through the human body and the Earth, and demonstrate that GWs cause phenomena
that commonly occur in daily life by back-of-the-envelope calculations. Our
analysis provides intellectual and pedagogical materials for understanding the
nature of GWs for nonexperts in general relativity.

##  I Introduction

On February 11, 2016, the LIGO Scientific Collaboration and Virgo
Collaboration announced the first direct detection of gravitational waves
(GWs) generated by the coalescence of binary black holes (BHs) Abbott:2016blz
; LIGOpress . It was a historical milestone for mankind to finally prove the
prediction of Einstein’s general relativity proposed about 100 years ago. GWs
are ripples in spacetime. Now, it is almost common knowledge not only among
scientists but also the general public that spacetime is stretched and
compressed upon the passage of GWs. Many people are interested in GWs and
general relativity; hence, it is time to think about how to cultivate its
intuitive understanding Schutz:2021xns . For a good review on the history of
GWs, see Cervantes-Cota:2016zjc .

Compared with, for example, electromagnetic waves, there are several obstacles
to people being aware of the effects of GWs in their daily lives. First of
all, it is difficult to imagine what happens when GWs stretch and compress
spacetime. In particular, a common misconception is as follows: “Rulers should
also be stretched and compressed similarly to space. Thus, are GWs not
physically observable inherently?” Actually this is not the case, and GWs do
lead to physically observable effects. In principle, one can notice the
passage of GWs if their amplitude is sufficiently large to cause visible
distortions (see Fig. 1). Although this conceptual issue has already been
addressed in depth in the literature (e.g., see Misner:1973prb ; Creighton ;
Maggiore:2007ulw ; Saulson:1997ck ), in the next section, we briefly provide
the physical explanation of the reason why GWs can yield observable effects,
which is basically a restatement of what is already known. Nevertheless, we
believe that it helps readers confronted with this conceptual question
recognize the reality of GWs and easily understand the subsequent sections
where we study how the GWs with extreme amplitudes cause visible effects on
daily phenomena.

The second reason why people do not commonly think about GWs as much as they
do about electromagnetic waves is their extremely small amplitude. For
instance, in the case of GWs emitted from a typical binary BH merger at a
cosmological distance, the change in distance between the Earth and the Sun is
merely about one-tenth of the Bohr radius! It is this smallness of GW
amplitude that had prevented direct detection for about 100 years and rendered
GWs irrelevant to daily physical phenomena.

Figure 1:  Artistic image of extremely exaggerated physical effects of GWs.
Owing to the passage of GWs, the Earth is stretched and compressed in a
direction perpendicular to the propagation of GWs. In addition to this effect,
two effects on the light arriving at the observer, i.e., color change of the
light due to the red-/blue-shift and the change in light trajectories, also
distort the appearance of the Earth.

Although these peculiar circumstances have led to the general public being
less aware of GWs, given their scientific significance, it is still
regrettable if only the experts of general relativity can appreciate their
physical properties*1*1*1There are several educational papers that aim to
spread the basic concept and recent experimental achievements of general
relativity and GWs Schutz:1984nf ; Farr:2011cw ; Burko:2016vnu ;
Mathur:2016cox ; Hilborn:2017liy . . Fortunately, the physical effects of GWs
can be understood through the familiar Newtonian mechanics Misner:1973prb ;
Maggiore:2007ulw . The aim of this paper is to provide intellectual and
pedagogical materials for understanding the nature of GWs within the framework
of Newtonian mechanics and at the level of the back-of-the-envelope
calculation, by which students who have learned Newtonian mechanics but are
not familiar with relativity can obtain a clear view of the physical effects
of GWs. To this end, we consider extreme and readily perceivable situations.
As it is often the case when learning a particular subject in physics, ideal
examples enable us to easily understand the essence of the physical effects of
GWs without being concerned with non-GW effects that are unnecessary for the
current purpose but may become dominant in more realistic situations. We
perform a quantitative analysis of several parameter regions in the
amplitude–frequency space in which GWs would visibly affect us. Additionally,
considering a BH binary system as the source of GWs, we shall estimate the
corresponding BH mass and distance from the Earth.

We stress that our results should not be regarded as a warning of the risk of
damage to human beings posed by GWs in actual daily life. As we shall see
below, GWs may readily and perceivably exert their effects only at a
sufficiently close distance from massive BHs.

##  II Physical effects of GWs: general arguments

Before presenting our clear examples of the physical effects of GWs, in this
section we provide a general argument on how we can, in principle, detect GWs
by addressing the following conceptual question that people who are not
familiar with general relativity might ask: “If the passage of GWs causes
nothing but the stretching and the compression of space, do they also stretch
and compress a ruler for measuring the change in the size of the body
simultaneously, making the effects of GWs unobservable?” Although the answer
to this question is already provided in several reports (e.g., Misner:1973prb
; Creighton ; Maggiore:2007ulw ; Saulson:1997ck ), we consider it useful not
to skip this issue but to recapitulate the essential point of the answer here
in order for readers to understand the background and hence the subsequent
results of our analyses.

In a nutshell, an essential point in any experiment to detect the physical
effects of GWs is the existence of an “absolute ruler”. Here, the absolute
ruler refers to the ruler (not any ordinary ruler) that does not change upon
the passage of GWs. The effects of GWs can then be perceived as the change of
the system measured using the absolute ruler.

The absolute ruler is a set of fundamental physical constants that are
literally absolute and do not change in the presence of GWs. The relevant
physical constants depend on the types of system and measurement considered.
Here, we focus on the following two cases of physical phenomena caused by GWs
and clarify how GWs induce physical effects that can be detected.

  1. 1.

Variation of light travel time

  2. 2.

Variation of the size of a deformable or semirigid body with time

The relevant physical constants in each phenomenon are the speed of light $c$
(case 1) and those used to determine the electromagnetic force at the
microscopic level, such as the electric charge (case 2).

Case 1 is actually implemented in current laser interferometers (such as
LIGO). In laser interferometers, the distance $L$ between the beam splitter
and the mirror changes upon the passage of GWs, and consequently the travel
time of the laser light over that distance, which the detector actually
measures, also changes. Although the propagation speed of the laser light is
not affected by GWs, the wavelength $\lambda$ is stretched and compressed by
them; hence, the laser light is red-/blue-shifted. Thus, the ratio $L/\lambda$
remains constant. Nevertheless, this does not mean that the laser
interferometer cannot detect GW signals since it can detect the change in
light travel time Saulson:1997ck ; Farr:2011cw .

In case 2, the physical phenomena induced by the distortion of a deformable
body or vibrations of a semirigid body are measured. To understand this
qualitatively, let us model the material of a body as a collection of point
masses connected by springs. This model provides a convenient alternative to
the electromagnetic force acting between atoms. The properties of the springs,
such as their natural length and restoring force, are determined by physical
constants, such as the electric charge and electron mass, and are not affected
by GWs. When GWs pass through, the GWs change the length of the springs from
their natural length. For the atoms and springs, this action is perceived as a
physical force because natural length is absolute. As a result, the atoms
oscillate around the equilibrium position, which, at the macroscopic level,
can be seen as vibrations of the body.

For case 2, we can derive the effective force induced by GWs without resorting
to general relativity. Consider a system consisting of two point masses
floating in space, each with mass $m$ and separated by a distance $L$. When a
gravitational wave with amplitude $h(t)$ passes through the system, the
distance between the masses (as measured by an absolute ruler) becomes
$\sqrt{1+h(t)}L$ according to general relativity. If we assume that $h(t)\ll
1$, which is valid in most cases explored in this paper, we can use the
binomial approximation $(1+\epsilon)^{1/2}\approx 1+\frac{1}{2}\epsilon$ to
say that the distance is approximately $[1+\frac{1}{2}h(t)]L$, meaning that an
observer with an absolute ruler would say that each mass is oscillating toward
or away from the system’s center of mass with an acceleration of
$a=\frac{1}{2}\ddot{h}(\frac{1}{2}L)$ (since each mass is $\frac{1}{2}L$ from
the center) = $\frac{1}{4}\ddot{h}L$. However we might interpret this in
general relativity, from a Newtonian perspective, this acceleration must be
caused by a force, meaning that each mass behaves as if it were experiencing a
force of magnitude

$F=\frac{1}{4}m{\ddot{h}}(t)L$ (1)

toward or away from each the system’s center of mass. If the two masses
happened to be connected by a spring, this effective force would measurably
compress or expand the spring: indeed, the spring would behave as if it were
being compressed by a total force of $F=\frac{1}{2}m{\ddot{h}}(t)L$.

The above discussion demonstrates that the stretching and compression of space
by GWs do yield observable physical effects. In the following, we focus on
case 2 and provide three extreme examples (destruction of organs, earthquakes,
and tides) with which readers unfamiliar with general relativity can imagine
the physical effects of GWs intuitively in terms of Newtonian mechanics. As
the waveform of a GW, we consider a monochromatic wave given by

$h(t)=h_{0}\cos(2\pi f_{\rm gw}t).$ (2)

##  III Example 1: Organ destruction

The first example we study is the potential destruction of the human body
caused by passing GWs. With the typical realistic strain, the effect of GWs on
the human body is totally negligible. However, as we have discussed in §II,
for a sufficiently large amplitude of GWs, in principle, GWs could destroy
organs or tissues inside our body. Let us provide an order-of-magnitude
estimation of the stretching of GWs at which organs are destroyed by
determining the force exerted by GWs of various amplitudes and frequencies on
the parts of an organ and comparing against the maximum force the organ can
tolerate without being destroyed.

In the order-of-magnitude estimation, we may replace an organ with two massive
objects separated by a distance equivalent to the size of the organ and solve
Eq. (1) as a force acting on the organ. Taking $L=10~{}{\rm cm}$ and
$m=0.1~{}{\rm kg}$ as representative values for an organ (e.g., heart) and
scaling frequency $f_{\rm gw}$ relative to $100~{}{\rm Hz}$, which is the
typical frequency for an event measured by LIGO, Virgo, or KAGRA, we find that
the amplitude of the total compressive force on the organ is approximately
given by

$F\approx 10^{3}~{}{\rm N}~{}{\left(\frac{f_{\rm gw}}{100~{}{\rm
Hz}}\right)}^{2}~{}h_{0}.$ (3)

Roughly speaking, an organ will collapse if more than $\sim 10^{3}~{}{\rm N}$
is exerted on it Rosen:2008 . By adopting this criterion, we find that an
organ is destroyed if the GW amplitude $h_{0}$ satisfies

$h_{0}\gtrsim{\left(\frac{f_{\rm gw}}{100~{}{\rm Hz}}\right)}^{-2}.$ (4)

The red solid line in Fig. 2 shows the boundary of this equality as a function
of $f_{\rm gw}$. The lower limit of $f_{\rm gw}$ around $10^{2}~{}{\rm Hz}$
originates from the criterion $h_{0}<1$ for which our linear approximation is
valid. Cases where $h_{0}>1$ are beyond the scope of this paper. The upper
limit of $f_{\rm gw}$ is the inverse of the time that a sound wave travels
across an organ:

$f_{\rm gw}<\frac{c_{\rm o}}{L}=10^{5}~{}{\rm Hz}~{}{\left(\frac{c_{\rm
o}}{10^{3}~{}{\rm m/s}}\right)}{\left(\frac{L}{10~{}{\rm cm}}\right)}^{-1},$
(5)

where $c_{\rm o}$ is the speed of sound passing across the organ. Above this
frequency, the force from GWs changes its direction before it affects the
whole organ, and it is not clear whether the organ is destroyed even if the
force given by Eq. (3) is applied. Hence, in our analysis, we consider the
frequency range that satisfies this condition. In Fig. 2, the boundaries
$h_{0}=1$ and $f_{\rm gw}=10^{5}~{}{\rm Hz}$ are shown as red dashed lines.

Figure 2:  GW amplitudes $h_{0}$ sufficiently strong to cause large effects in
terms of frequency $f_{\rm gw}$. The solid lines indicate the threshold beyond
which human lives would be threatened, whereas the dashed lines indicate the
possible boundaries of the validity of the approximations we employed. The
red, green, and blue lines are, respectively, the thresholds, beyond which the
GWs cause organ destruction, major earthquakes, and large tides.

##  IV Example 2: Earthquakes

Next, we consider an earthquake that would be induced by the passage of GWs. A
pioneering work was performed by Dyson, who calculated the seismic response of
the Earth to the passage of 1 Hz GWs Dyson:1997gv . Constraints using
potential seismic signatures have been extensively studied Coughlin:2014sca .
Dyson’s computation heavily relies on tensorial equations and general
relativity, which are beyond the scope of this paper. Instead, here, we
consider a back-of-the-envelope calculation. Quite remarkably, results of our
crude calculation reproduce those of Dyson’s computation within a factor of
${\cal O}(1)$.

An earthquake is simply the vibration of the ground propagating as a sound
wave. When a GW of frequency $f_{\rm gw}$ is passing through the Earth, a
sound wave with the same frequency is induced and propagates. Let us model the
motion of a segment of the Earth medium with a harmonic oscillator in which a
point mass is attached to a spring of length $L$. As we have discussed in §II,
GWs induce the acceleration of the harmonic oscillator of the order of
$\pi^{2}f_{\rm gw}^{2}Lh_{0}$. From the shape of the plane wave $e^{2\pi
ix/\lambda_{\rm e}}$, where $\lambda_{\rm e}$ is the wavelength of a
transverse seismic wave passing through the Earth, the phases of the ground
motion at different locations are nearly coherent (i.e., the phase difference
is $<2\pi$) if the separation is less than $\lambda_{\rm e}$. In other words,
modeling the vibration of the ground as oscillations of the harmonic
oscillator is valid for the segment of the Earth medium with a (typical) size
$\lesssim\lambda_{\rm e}$. This means that we should take $L$ to be
$\lambda_{\rm e}$. Then, the acceleration of the ground is estimated as

$a=\pi^{2}c_{\rm e}f_{\rm gw}h_{0},$ (6)

where $c_{\rm e}=\lambda_{\rm e}f_{\rm gw}$ is the speed of a transverse
seismic wave passing through the Earth, which is typically $3\times 10^{3}$
m/s. Note also that since $f_{\rm gw}=c/\lambda_{\rm gw}=c_{\rm
e}/\lambda_{\rm e}$ it holds that $\lambda_{\rm e}\ll\lambda_{\rm gw}$. The
acceleration (6) is the same as that obtained by Dyson up to a factor of
${\cal O}(1)$ Dyson:1997gv .

The induced earthquake will be catastrophic if the acceleration is greater
than the gravitational acceleration $g\approx 9.8$ m/s2. Thus, by equating
acceleration (6) to $g$, we obtain the critical GW amplitude $h_{0}$ that
yields a serious earthquake:

$h_{0}=\frac{g}{\pi^{2}c_{\rm e}f_{\rm gw}}.$ (7)

The green solid line in Fig. 2 shows $h_{0}$ that causes an earthquake with
ground acceleration comparable to the gravitational acceleration. As in
Dyson:1997gv , we focus on the GW frequency range $1~{}{\rm mHz}<f_{\rm
gw}<1~{}{\rm kHz}$. In this range, it is a good approximation to treat the
surface of the Earth as a flat plane since the wavelength of the induced
seismic waves is much smaller than the size of the Earth and larger than the
surface irregularities.

##  V Example 3: Tides

The third effect we consider is tides, i.e., the rise and fall of sea levels,
which are usually generated by the gravitational forces from the Moon and Sun.
Similarly, when GWs pass through the Earth, they can generate tides. Tides
involve very complicated processes and it is difficult to analyze those
generated by the passage of GWs. As in the previous two examples, we will
evaluate tides by a crude approximation. To this end, we treat the Earth as a
rock sphere completely encompassed by a water shell of uniform depth $H$. In
this model, we assume that the rock sphere is so rigid that it is not deformed
by GWs and only the water shell is freely deformed (since it is liquid) by the
force given by Eq. (1).

For this model, the first task is to determine the length scale $L$. We make a
rough estimate; if the period of GWs is longer than one day, the ocean on the
global scale will coherently respond to GWs similarly to the case in response
to the tidal force generated by the Moon’s gravitational force. In this case,
the natural scale of $L$ will be the Earth radius $R_{\oplus}$. Assuming
$L=R_{\oplus}$, the force from the GWs exerted on the volume element $\Delta
V$ is given by

$F\simeq\pi^{2}f_{\rm gw}^{2}\rho_{\rm w}R_{\oplus}h_{0}\Delta V,$ (8)

where $\rho_{\rm w}$ is the density of water. Conversely, if the period of GWs
is much shorter than one day but longer than the time scale of the sound wave
traveling across the depth $H$, the response of the sea level will not be the
same as that in the low-frequency case.

To understand this situation, let us consider the motion of a harmonic
oscillator with a periodic external force as a simple model:
$m{\ddot{x}}=-kx+m\omega^{2}h_{0}L\sin(\omega t)$. Here, $x$ and the external
force are assumed to represent the change in sea level and the effect by GWs,
respectively. Note that we have taken the coefficient of the external force to
be $m\omega^{2}h_{0}L$; with this expression, $h_{0}L$ does not depend on
$\omega$ and is simply proportional to $h_{0}$ [see Eq. (1)]. Assuming
$x=x_{0}\sin(\omega t)$, $x_{0}$ is given by

$x_{0}=\frac{h_{0}L\omega^{2}}{\omega_{0}^{2}-\omega^{2}},$ (9)

where $\omega_{0}=\sqrt{k/m}$. In the low-frequency limit
$\omega\ll\omega_{0}$, we have $x_{0}=h_{0}L\omega^{2}/\omega_{0}^{2}$,
namely, $x_{0}$ is determined by the balance between the term $-kx$ and the
external force. On the other hand, in the high-frequency limit
$\omega\gg\omega_{0}$, we have $x_{0}=-h_{0}L$. Apparently, this might look
inconsistent with the standard result of the forced harmonic oscillator, for
which the amplitude goes to zero in the high-frequency limit. The point is
that, as stressed above, the GW driving force actually increases with
frequency for a given GW amplitude $h_{0}$, and this leads to the conclusion
that in the high-frequency limit $x_{0}$ becomes a constant value $-h_{0}L$
independent of the frequency of the external force $\omega$. Given these
observations, it would be reasonable to consider that the tide induced by GWs
becomes independent of $f_{\rm gw}$ for $f_{\rm gw}\gg f_{\rm day}=1/{\rm
day}$.

The next nontrivial task is to estimate the magnitude of the tide induced by
the force given in Eq. (8), which, in principle, can be done by solving the
shallow-water equation. However, even at the level of the order-of-magnitude
estimate, how this can be achieved is nontrivial. To overcome this issue, we
simply compare force (8) with the tidal force induced by the Moon. For $f_{\rm
gw}<f_{\rm day}$, we assume that the level of the tide induced by GWs is the
same as that of an ordinary tide if the two forces have the same magnitude.
The tidal force from the Moon acting on the volume element $\Delta V$ is
estimated as

$F_{\rm L}\simeq\frac{GM_{\rm L}\rho_{\rm w}\Delta V}{D^{3}}R_{\oplus}.$ (10)

Here, $M_{\rm L}$ is the lunar mass, and $D$ is the lunar distance. By
equating Eq. (8) to Eq. (10), we obtain the minimum $h_{0}$ that yields a tide
comparable to the ordinary one as

$h_{0}=\frac{GM_{\rm L}}{\pi^{2}D^{3}f_{\rm gw}^{2}}.$ (11)

On the other hand, for $f_{\rm gw}\gg f_{\rm day}$, on the basis of the above
argument on the forced oscillator, we assume that the critical $h_{0}$ is
given by the right-hand side of Eq. (11) with $f_{\rm gw}$ being replaced by
$f_{\rm day}$.

The blue solid line in Fig. 2 shows $h_{0}$ that causes a tide comparable to
that caused by the Moon. The increase in sea level to this amount will
threaten the lives of many people residing in coastal regions. The upper limit
for $f_{\rm gw}\lesssim 1~{}{\rm Hz}$ is imposed by the condition for the
validity of the shallow-water approximation that the time for sound to pass
across the depth of the ocean is shorter than the GW frequency.

##  VI GWs from BH binaries

In the previous sections, we considered extreme situations where the GWs cause
visible effects on humans by causing stretching and shrinking of space, and we
obtained the parameter regions in the amplitude–frequency space, as shown in
Fig. 2. Now it is interesting to consider converting the parameter regions for
the GWs to those for BH binaries. The strongest GWs are emitted during the
last moment of the merging of BHs in a binary system. Therefore, it is natural
to try to relate GWs to BH binary systems in such a way.

We assume that a BH binary system consists of two equal-mass BHs each of which
has mass $M$ and is located at the distance $r$ from the Earth. Precise
modeling of the GW waveform from such a phase requires numerous computations
based on general relativity. Since we are working at the level of the order-
of-magnitude estimate, we do not use the results obtained from full
computations but make a rough estimate of the GW amplitude and frequency as
follows.

At the last inspiral phase, the distance between BHs is comparable to the size
of the BHs, i.e., the Schwarzschild radius $r_{\rm s}=2GM/c^{2}$, and the BHs
are orbiting at a relativistic speed. In this extreme situation, the
disturbance of the gravitational field caused by the motion of the BHs, which
propagates outward as GWs, would be comparable to the gravitational field
generated by the BHs when they are not moving. Thus, it is natural to think
that the GW amplitude around the BHs is also ${\cal O}(1)$. The GW amplitude
attenuates inversely proportionally to the propagation distance. Thus, by
denoting $r$ as the distance to the GW source, we find that the GW amplitude
$h_{0}$ on the Earth would be on the order of

$h_{0}\simeq\frac{2GM}{c^{2}r}.$ (12)

The period of GWs is given by the time scale of the change of the GW source,
which is $r_{\rm s}/c$. Therefore, the frequency of the GWs would be on the
order of

$f_{\rm gw}\simeq\frac{c}{r_{\rm s}}=\frac{c^{3}}{2GM}.$ (13)

This clearly shows that there is a one-to-one correspondence between $f_{\rm
gw}$ and $M$.

In the previous sections, we estimated the critical values of $h_{0}$ above
which phenomena catastrophic to humans occur at various values of $f_{\rm
gw}$. By using Eq. (13) to replace $f_{\rm gw}$ with $M$, we can draw curves
corresponding to the critical values of $h_{0}$ in the $M$–$r$ plane, as shown
in Fig. 3. As expected, the results in Fig. 3 suggest that the GWs would cause
serious damage only if too-heavy BHs merged at too-close distances from the
Earth. It would be a good exercise to think about the farthest and/or lightest
parameter value for each case. For instance, let us consider the farthest
parameter value $(r,M)\sim(10^{15}{\rm km},10^{10}M_{\odot})$ for the tide. It
means that a large tide would be induced by GWs if two supermassive BHs each
with a mass of $\sim 10^{10}M_{\odot}$ merged at about 100 light years from
the Earth. To get a sense of this situation, let us compare the distance and
the mass $(r,M)\sim(10^{15}{\rm km},10^{10}M_{\odot})$ of a BH with those of
Sgr A∗, which is the supermassive BH at the center of the Milky Way galaxy.
The mass of Sgr A∗ is $4.2\times 10^{6}M_{\odot}$ and the distance is
$2.7\times 10^{4}$ light years from the Earth Abuter2019 . Therefore, even the
farthest parameter value means that too-heavy BHs merged at too-close
distances from the Earth. If such supermassive BHs existed, the Solar System
would be destroyed by their gravitational force far before they merge.
Therefore, the earthquakes and tides generated by GWs are totally irrelevant
to reality. However, when space travel to a BH binary systems becomes popular
in the future, the effect of GWs on organs may be relevant during such travel.
In that case, the red boundary in Fig. 3 must be included as a caution in the
flight plan for astronauts.

Figure 3:  Parameter region corresponding to Fig. 2 in terms of the BH mass
$M$ and the distance $r$ from the Earth assuming that GWs are emitted at the
last inspiral stage of two BHs of equal mass. The BH mass $M$ is normalized by
the solar mass $M_{\odot}\approx 1.99\times 10^{30}$kg.

##  VII Conclusion

GWs are ripples of spacetime traveling across the universe. Not only spacetime
but also everything in nature are stretched and compressed by GWs. Therefore,
in principle, they could cause some large effects. After clarifying the
conceptual issue about the detectability of GWs and translating the effects of
GWs into the language of Newtonian mechanics, we performed the order-of-
magnitude estimation for visible phenomena, such as organ destruction,
earthquakes, and tides, that would be caused by large-amplitude GWs. We
obtained the amplitude–frequency space in which GWs would cause some large
effects. As a reference, we converted the parameter regions to those in BH
mass–distance space. These are interesting examples highlighting the nature of
GWs and are useful for gaining some understanding of the theory of relativity.

###### Acknowledgements.

H.M. was supported by Japan Society for the Promotion of Science (JSPS)
Grants-in-Aid for Scientific Research (KAKENHI) Grants No. JP18K13565 and No.
JP22K03639. T.S. was supported by the MEXT Grant-in-Aid for Scientific
Research on Innovative Areas No. 17H06359, No. 19K03864, and No. 21H05453.

## References

  * (1) LIGO Scientific, Virgo collaboration, B. Abbott et al., Observation of Gravitational Waves from a Binary Black Hole Merger, Phys. Rev. Lett. 116 (2016) 061102 [arXiv:1602.03837].
  * (2) http://mediaassets.caltech.edu/gwave.
  * (3) B. F. Schutz, Intuition in Einsteinian Physics, arXiv:2106.01820.
  * (4) J. L. Cervantes-Cota, S. Galindo-Uribarri and G.-F. Smoot, A Brief History of Gravitational Waves, Universe 2 (2016) 22 [arXiv:1609.09400].
  * (5) C. W. Misner, K. S. Thorne and J. A. Wheeler, Gravitation. W. H. Freeman, San Francisco, 1973.
  * (6) J. D. E. Creighton and W. G. Anderson, Gravitational-wave physics and astronomy: An introduction to theory, experiment and data analysis, Wiley Series in Cosmology. Wiley-VCH, 2011.
  * (7) M. Maggiore, Gravitational Waves. Vol. 1: Theory and Experiments, Oxford Master Series in Physics. Oxford University Press, 2007.
  * (8) P. R. Saulson, If light waves are stretched by gravitational waves, how can we use light as a ruler to detect gravitational waves?, Am. J. Phys. 65 (1997) 501.
  * (9) B. F. Schutz, Gravitational waves on the back of an envelope, Am. J. Phys. 52 (1984) 412.
  * (10) B. Farr, G. Schelbert and L. Trouille, Gravitational Wave Science in the High School Classroom, Am. J. Phys. 80 (2012) 898 [arXiv:1109.3720].
  * (11) L. M. Burko, Gravitational Wave Detection in the Introductory Lab, Phys. Teacher 55 (2017) 288 [arXiv:1602.04666].
  * (12) H. Mathur, K. Brown and A. Lowenstein, An analysis of the LIGO discovery based on Introductory Physics, Am. J. Phys. 85 (2017) 676 [arXiv:1609.09349].
  * (13) R. C. Hilborn, Gravitational waves from orbiting binaries without general relativity, Am. J. Phys. 86 (2018) 186 [arXiv:1710.04635].
  * (14) J. Rosen, J. Brown, S. De, M. Sinanan and B. Hannaford, Biomechanical properties of abdominal organs in vivo and postmortem under compression loads, Journal of Biomechanical Engineering 130 (2008) 021020.
  * (15) F. Dyson, Seismic response of the earth to a gravitational wave in the 1-Hz band, Astrophys. J. 156 (1969) 529.
  * (16) M. Coughlin and J. Harms, Upper Limit on a Stochastic Background of Gravitational Waves from Seismic Measurements in the Range 0.05–1 Hz, Phys. Rev. Lett. 112 (2014) 101102 [arXiv:1401.3028].
  * (17) GRAVITY collaboration, R. Abuter et al., A geometric distance measurement to the Galactic Center black hole with 0.3% uncertainty, Astron. Astrophys. 625 (2019) L10 [arXiv:1904.05721].