# MGTBench: Benchmarking Machine-Generated Text Detection

Xinlei He† Xinyue Shen† Zeyuan Chen‡ Michael Backes† Yang Zhang†

†CISPA Helmholtz Center for Information Security ‡Individual Researcher

###### Abstract

Nowadays large language models (LLMs) have shown revolutionary power in a
variety of natural language processing (NLP) tasks such as text
classification, sentiment analysis, language translation, and question-
answering. In this way, detecting machine-generated texts (MGTs) is becoming
increasingly important as LLMs become more advanced and prevalent. These
models can generate human-like language that can be difficult to distinguish
from text written by a human, which raises concerns about authenticity,
accountability, and potential bias. However, existing detection methods
against MGTs are evaluated under different model architectures, datasets, and
experimental settings, resulting in a lack of a comprehensive evaluation
framework across different methodologies

In this paper, we fill this gap by proposing the first benchmark framework for
MGT detection, named MGTBench. Extensive evaluations on public datasets with
curated answers generated by ChatGPT (the most representative and powerful
LLMs thus far) show that most of the current detection methods perform less
satisfactorily against MGTs. An exceptional case is ChatGPT Detector, which is
trained with ChatGPT-generated texts and shows great performance in detecting
MGTs. Nonetheless, we note that only a small fraction of adversarial-crafted
perturbations on MGTs can evade the ChatGPT Detector, thus highlighting the
need for more robust MGT detection methods. We envision that MGTBench will
serve as a benchmark tool to accelerate future investigations involving the
evaluation of state-of-the-art MGT detection methods on their respective
datasets and the development of more advanced MGT detection methods.111Source
code and datasets are available at https://github.com/xinleihe/MGTBench.

##  Introduction

Large language models (LLMs), such as T5 [16], GPT3 [3], PaLM [4], and ChatGPT
[2], have been a significant breakthrough in the field of natural language
processing (NLP). With huge numbers of parameters and being trained with
massive amounts of data, LLMs have shown remarkable performance in various
real-world applications, e.g., education, customer service, and finance.

While LLMs have shown impressive qualities in terms of achieving state-of-the-
art performance in various tasks and generating human-like texts, several
limitations and ethical concerns have to be taken into account. First, the
LLMs may generate text that sounds realistic but may not be entirely accurate
or factual [11]. Second, the misuse of LLMs may raise concerns in education,
making a fair judgment impossible [20]. Also, it is difficult to trace back
the machine-generated text to its source, which raises concerns about
accountability, especially when the content is used to spread misinformation
or propaganda [22].

To address these issues, researchers have considered automatic detection
methods that can identify the machine-generated text (MGT) from the human-
written text (HWT). Concretely, those methods can be concluded into two
categories, i.e., metric-based methods and model-based methods. For the
metric-based methods [18, 7, 13], metrics such as log-likelihood, word rank,
and predicted distribution entropy to determine whether a text belongs to MGT
or HWT. Regarding the model-based methods [18, 8], the classification model is
trained by both MGT and HWT corpus.

Overall, existing MGT detection methods have been studied under different LLMs
and different datasets, albeit in isolation. This prompts the need for a
holistic evaluation of these methods. To do this, we develop a comprehensive
benchmark of MGT detection, namely MGTBench. MGTBench follows a modular
design, consisting of the input module, the detection module, and the
evaluation module. Until now, we have implemented 8 MGT detection methods. We
also take advantage of ChatGPT, the most popular and powerful LLM (thus far)
to produce MGTs on existing HWT datasets. MGTBench enables researchers to
benchmark various methods on different datasets. Its modular design
facilitates the integration of additional detection methods, as well as the
plugging in of datasets and models.

Evaluation. We perform an extensive measurement study over the 8 detection
methods using different LLM architectures in the GPT2 and GPT3 families. Our
measurement is performed on standard benchmark question-answering (QA)
datasets including TruthfulQA [11], SQuAD1 [17], and NarrativeQA [9]. Note
that for each dataset, besides the answers generated by humans (labeled as
HWTs), we also query the ChatGPT with the questions (and the contexts if they
are included on the dataset) to obtain the answers from ChatGPT (labeled as
MGTs). In this case, we label the three datasets as TruthfulQA-C, SQuAD1-C,
and NarrativeQA-C, respectively.

Extensive evaluations show that the ChatGPT Detector outperforms other
detection methods and reaches the best performance. For instance, the
detection F1-score on TruthfulQA-C with ChatGPT Detector is 0.997 while only
0.896 with Log-Likelihood. Also, our ablation study shows that the answers
generated by ChatGPT are much longer than human-written answers. To reduce the
potential bias caused by the number of words in a text, we perform extra
experiments with texts within 25 words and observe a clear degradation in the
detection performance. For instance, after filtering the texts, the detection
AUC of Log-Likelihood drops from 0.957 to 0.818 on TruthfulQA-C. Also, we
observe that the prompt that we used to obtain answers from ChatGPT can have a
negative impact on the final detection performance (see Section 5 for more
details).

Overall, we find that ChatGPT Detector is the most robust and best-performing
detection method. We suspect the reason is that ChatGPT Detector is fine-tuned
with the corpus that contains ChatGPT-generated answers, which can better help
it identify the unique pattern hidden in answers generated by ChatGPT.
However, it is still vulnerable to adversarial attacks. Our evaluation
demonstrates that one can easily bypass the detection of ChatGPT Detector by
only introducing a small fraction of perturbations to the texts (see Table 6
for more details). This finding sheds light on the future design of MGT
detection methods: besides the detection performance, we should also care
about its robustness against potential adversarial attacks.

##  Preliminary and Related Work

###  Text Generation with Large Language Models

The recent advancements in LLMs can be traced back to the transformer
architecture proposed by Vaswani et al. [21], which introduces the self-
attention mechanisms to allow the model to focus on different regions of the
input sequence. Later Radford et al. [14] develop the generative pre-trained
transformer (GPT) based on the transformer architectures. Being trained with a
large corpus of text data, GPT achieves superior performance on a wide range
of language generation tasks. GPT2 [15], a larger version of GPT that contains
more parameters and is trained on a larger corpus, is developed and achieves
better performance than GPT. GPT3 [3] is the third generation of GPT with over
175B parameters, which has been shown to be capable of generating coherent and
contextually appropriate text even in situations where it is provided with
minimal input or guidance. Since November 2022, OpenAI releases ChatGPT [2],
which is trained based on the GPT-3.5 architectures and leverages
Reinforcement Learning from Human Feedback (RLHF) [5, 19] to improve its
generation ability. ChatGPT shows revolutionary capabilities in generating
coherent and relevant texts, which can be integrated into various
applications, such as chatbots, customer service, and education.

Another notable model series in the field is the masked language model.
Bidirectional Encoder Representations from Transformers (BERT) developed by
Devlin et al. [6] is one of the most representative models that is pre-trained
using the masked language modeling task. Liu et al. [12] develop RoBERTa,
which leverages a robustly optimized BERT pre-training approach and can reach
an even better performance than BERT in various tasks.

With the widespread use of LLMs for generating texts, concerns about
authenticity, accountability, and potential bias have also been raised.

###  Machine-Generated Text Detection

To address the above-mentioned issues, researchers have developed various MGT
detection methods. Current detection methods can be divided into two
categories, i.e., metric-based methods and model-based methods. Generally
speaking, metric-based methods leverage pre-trained LLMs to process the text
and extract distinguishable features from it, e.g., the rank or entropy of
each word in a text conditioned on the previous context. In this paper, we
consider 6 metric-based detection methods, including Log-Likelihood, Rank,
Log-Rank, Entropy, GLTR, and DetectGPT.

Log-Likelihood [18]. This approach leverages a language model to measure the
token-wise log probability. Concretely, given a text, we average the token-
wise log probability of each word to generate a score for this text. Note that
a larger score denotes the text is more likely to be machine-generated.

Rank [7]. For each word in a text, given its previous context, we can
calculate the absolute rank of this word. Then, for a given text, we compute
the score of the text by averaging the rank value of each word. Note that a
smaller score denotes the text is more likely to be machine-generated.

Log-Rank [13]. Slightly different from the Rank metric that uses the absolute
rank, the Log-Rank score is calculated by first applying the log function to
the rank value of each word.

Entropy [7]. Similar to the Rank score, the Entropy score of a text is
calculated by averaging the entropy value of each word conditioned with its
previous context. As mentioned by previous work [7, 13], the machine-generated
text is more likely to have a lower Entropy score.

GLTR [7]. GLTR is developed as a support tool to facilitate the labeling
process of whether a text is machine-generated. In our evaluation, we follow
the suggestion of Guo et al. [8] and consider the Test-2 features (i.e., the
fraction of words that rank within 10, 100, 1,000, and others). Note that one
can easily implement other sets of features in MGTBench.

DetectGPT [13]. Mitchell et al. [13] propose DetectGPT that measures the
change of the model’s log probability function by adding minor perturbation to
the original text. The intuition is that the text derived from an LLM has a
tendency to be in the local optimal of the model’s log probability function.
Therefore, any minor perturbation of model-generated text tends to have a
lower log probability under the model than the original text, while minor
perturbation of human-written text may have a higher or lower log probability
than the original text.

Table 1: The prompts we used to obtain answers from ChatGPT. Note that <context> and <question> are provided by the original dataset. The prompts are adopted from [1]. Dataset |  Prompt
---|---
TruthfulQA-C |  <question>
SQuAD1-C |  I will provide a passage and a question to you. The answer should be extracted from the context. You need to return me your answer. The passage is <context> and the question is <question>. Now, please answer the question.
NarrativeQA-C |  I will provide a context and a question to you. You need to answer me the question based on the context. The context is: <context> The question is: <question>

Regarding the model-based methods, a classification model is usually trained
using a corpus that contains both HWT and MGTs. By doing this, the
classification model is expected to have the capability in identifying MGTs
from a given corpus. In this paper, we consider two model-based methods, i.e.,
OpenAI Detector and ChatGPT Detector.

OpenAI Detector [18]. The OpenAI Detector is used to detect GPT2-generated
output, which was created by fine-tuning a RoBERTa model using outputs from
the largest GPT2 model (i.e., with 1.5B parameters). This model is capable of
predicting whether a given text was machine-generated or not.

ChatGPT Detector [8]. ChatGPT Detector is developed by Guo et al. [8] to
distinguish human-written texts from ChatGPT-generated texts. The model was
created by fine-tuning a RoBERTa model using the HC3 [8] dataset. The detector
is based on the RoBERTa model. The authors provide two ways to train the
RoBERTa model. The first one only leverages the pure answered text, and the
second one leverages the question-answer text pair to jointly train the model.
In our evaluation, we consider the first one to be consistent with other
detection methods.

##  MGTBench

In this section, we introduce MGTBench, a modular framework designed to
benchmark MGT detection methods. Currently, we have provided reference
implementations of the 6 metric-based detection methods and easy-to-use APIs
for the two model-based methods we mentioned before.

###  Modular Design

MGTBench consists of three different modules, including the input module,
detection module, and evaluation module.

Input Module. In the input module, we provide specific dataset pre-processing
functions for different datasets and our code base is easy to cope with
datasets from HuggingFace, which facilitates the future developments of
different users.

Detection Module. This module implements different metric-based and model-
based detection methods with a standardized input/output format. Currently, we
support 8 different detection methods.

Evaluation Module. This module is used to evaluate the performance of
different detection methods. Now, we provide 5 different evaluation metrics,
including accuracy, precision, recall, F1-score, and AUC, which are the
commonly used metrics to evaluate the classification performance.

###  Using MGTBench

To be best of our knowledge, MGTBench is the first benchmark tool for MGT
detection. Users can leverage MGTBench on their own dataset for a
comprehensive risk assessment of potential MGTs in the dataset. On the other
hand, researchers can leverage MGTBench as a tool to evaluate new MGT
detection/generation methods. As MGTBench follows a modular design, its input
and evaluation modules can be easily re-used by new detection methods. Also,
new detection methods can be easily implemented within the standardized API
provided by MGTBench. Moreover, MGTBench integrates well with HuggingFace,
given the fact that many model-based detection methods have published or are
willing to publish their models into HuggingFace, MGTBench can be seamlessly
updated to fit the new model-based detection methods. MGTBench is under
continuous development and we will include more detection methods as well as
analysis tools in the future.

##  Experimental Settings

###  Datasets

In this paper, we leverage 3 datasets for evaluation.

TruthfulQA-C. This dataset is derived from the TruthfulQA [11] dataset that
contains 817 questions from 38 categories including health, law, finance, and
politics.

SQuAD1-C. This dataset is derived from the SQuAD1 [17] dataset that contains
more than 100,000 question-answer pairs selected from more than 500 articles.
We sample 1,000 records from the SQuAD1 dataset.

NarrativeQA-C. This dataset is derived from the NarrativeQA [9] dataset that
contains stories and corresponding questions designed to test reading
comprehension. We sample 1,000 records from the NarrativeQA dataset.

For each question, we additionally query ChatGPT222Our experiments are
conducted on the February 13th version of ChatGPT. with the questions (and the
contexts if they are included in the dataset) and obtain the answers as MGTs.
To avoid being affected by the chat history, we create a new conversation with
ChatGPT for each question.

We show the prompt we used to query ChatGPT in Table 1. Note that for each
entry in a dataset, we have the human answer and the ChatGPT-generated answer.
We only keep entries with more than 1 word for human and ChatGPT-generated
answers. Then, we randomly split 80% of the entries as the training set and
the rest as the testing set.

###  Models

For metric-based methods, we consider two families of models including GPT2
and GPT3. Concretely, we leverage GPT2-small, GPT2-medium, GPT2-large,
GPT3-small, and GPT3-medium in our experiments. Note that later we shorter the
model name for the presentation purpose (e.g., we use GPT2-S to denote
GPT2-small). For model-based methods, we directly use their provided model.
Concretely, for OpenAI Detector, we use the RoBERTa-base version of it as it
usually gives better detection performance. For ChatGPT Detector [8], we
leverage the provided RoBERTa-base model of it. All the models we use can be
downloaded from HuggingFace.333https://huggingface.co/.

Table 2: The performance (F1-score) of different detection methods. Here OpenAI-D denotes the OpenAI Detector, and Chat-D denotes the ChatGPT Detector. Dataset | Method | GPT2-S | GPT2-M | GPT2-L | GPT3-S | GPT3-M | RoBERTa
---|---|---|---|---|---|---|---
TruthfulQA-C | Log-Likelihood | 0.896 | 0.893 | 0.899 | 0.806 | 0.812 | -
Rank | 0.723 | 0.893 | 0.707 | 0.688 | 0.701 | -
Log-Rank | 0.892 | 0.887 | 0.884 | 0.810 | 0.834 | -
Entropy | 0.870 | 0.834 | 0.872 | 0.796 | 0.757 | -
GLTR | 0.865 | 0.872 | 0.882 | 0.794 | 0.803 | -
DetectGPT | 0.818 | 0.869 | 0.889 | 0.620 | 0.643 | -
OpenAI-D | - | - | - | - | - | 0.627
ChatGPT-D | - | - | - | - | - | 0.997
SQuAD1-C | Log-Likelihood | 0.805 | 0.734 | 0.720 | 0.770 | 0.768 | -
Rank | 0.687 | 0.639 | 0.631 | 0.653 | 0.659 | -
Log-Rank | 0.813 | 0.728 | 0.724 | 0.756 | 0.757 | -
Entropy | 0.840 | 0.754 | 0.799 | 0.814 | 0.809 | -
GLTR | 0.807 | 0.751 | 0.737 | 0.770 | 0.756 | -
DetectGPT | 0.549 | 0.607 | 0.617 | 0.552 | 0.545 | -
OpenAI-D | - | - | - | - | - | 0.422
ChatGPT-D | - | - | - | - | - | 0.844
NarrativeQA-C | Log-Likelihood | 0.805 | 0.734 | 0.720 | 0.770 | 0.768 | -
Rank | 0.687 | 0.639 | 0.631 | 0.653 | 0.659 | -
Log-Rank | 0.813 | 0.728 | 0.724 | 0.756 | 0.757 | -
Entropy | 0.840 | 0.754 | 0.799 | 0.814 | 0.809 | -
GLTR | 0.807 | 0.751 | 0.737 | 0.770 | 0.756 | -
DetectGPT | 0.549 | 0.607 | 0.617 | 0.552 | 0.545 | -
OpenAI-D | - | - | - | - | - | 0.422
ChatGPT-D | - | - | - | - | - | 0.844

###  Evaluation Metrics

MGTBench supports various metrics to evaluate performance, including accuracy,
precision, recall, F1-score, and AUC (area under the ROC curve). In our
evaluation, unless otherwise mentions, we use F1-score as the main evaluation
metric.

##  Evaluation

We first present the detection performance of different methods. As shown in
Table 2, we observe that ChatGPT Detector achieves the best performance across
different datasets. For instance, on TruthfulQA, ChatGPT Detector achieves
0.997 F1-score, while DetectGPT and GLTR only reach 0.818 and 0.865 F1-score.
This is expected as the ChatGPT Detector is trained on the corpus with both
human answers and ChatGPT-generated answers, it should better learn the hidden
patterns in ChatGPT-generated answers, which leads to better performance.

Also, we observe that metric-based methods such as Log-Likelihood, Log-Rank,
and Entropy have relatively good performance on different datasets as well.
For instance, with GPT2-S, Log-Likelihood reaches 0.896, 0.805, and 0.805
F1-score on TruthfulQA, SQuAD1, and NarrativeQA. This implies that the MGT
better fits the model’s “expectation” about what it suppose to be even if the
model is different from the one that used to produce MGT.

On the other hand, we find that the OpenAI Detector’s performance is not
satisfying (e.g., 0.627 F1-score on TruthfulQA). This might be credited to the
fact that this detector is trained on MGTs produced by GPT2 while the MGTs
produced by ChatGPT have higher quality, which makes it harder to be detected.

Detection Efficiency. We then quantify the time cost of each detection method
under different model architectures. As shown in Table 3, we can observe that
most of the detection methods have a similar time cost except the DetectGPT,
which cost about 100$\times$ time than the others. This is because DetectGPT
requires multiple rounds of perturbation to the text to get a good estimation
of the log probabilities’ change. In general, we consider ChatGPT Detector a
better detection method as it achieves the best detection performance and
costs the least time.

###  Ablation Studies

In our ablation study, we investigate how the number of words in the text and
different prompts to generate the text would affect the detection performance.
Note that here we omit the DetectGPT method as it usually has less satisfying
performance and costs much longer time than the others.

Table 3: Time cost (seconds) on TruthfulQA with GPT2-S.

Method | TruthfulQA | SQuAD1 | NarrativeQA
---|---|---|---
Log-Likelihood | 5.472 | 4.352 | 4.881
Rank | 5.894 | 4.288 | 4.932
Log-Rank | 5.897 | 4.280 | 4.932
Entropy | 5.134 | 3.933 | 4.494
GLTR | 6.621 | 4.561 | 5.358
DetectGPT | 826.484 | 579.988 | 515.287
OpenAI-D | 5.528 | 4.048 | 5.104
ChatGPT-D | 4.094 | 2.603 | 3.626

Table 4: Detection performance (measured by AUC) on original texts and
filtered texts (only contains texts that have no more than 25 words). Note
that the model is GPT2-S.

Dataset | Method | AUC (Original) | AUC (Filtered)
---|---|---|---
TruthfulQA-C | Log-Likelihood | 0.957 | 0.818
Rank | 0.739 | 0.535
Log-Rank | 0.953 | 0.815
Entropy | 0.931 | 0.784
GLTR | 0.918 | 0.805
DetectGPT | 0.859 | 0.617
OpenAI-D | 0.700 | 0.652
ChatGPT-D | 1.000 | 1.000
SQuAD1-C | Log-Likelihood | 0.823 | 0.800
Rank | 0.637 | 0.598
Log-Rank | 0.818 | 0.795
Entropy | 0.900 | 0.881
GLTR | 0.810 | 0.789
DetectGPT | 0.548 | 0.537
OpenAI-D | 0.174 | 0.178
ChatGPT-D | 0.991 | 0.991
NarrativeQA-C | Log-Likelihood | 0.788 | 0.701
Rank | 0.583 | 0.521
Log-Rank | 0.776 | 0.686
Entropy | 0.831 | 0.757
GLTR | 0.763 | 0.682
DetectGPT | 0.595 | 0.501
OpenAI-D | 0.245 | 0.237
ChatGPT-D | 0.933 | 0.931

(a) TruthfulQA-C

(b) SQuAD1-C

(c) NarrativeQA-C

Figure 1: The distribution of #. words in human-written text (HWT) and
machine-generated text (MGT).

Effect of Number of Words. We first take a step further to understand how the
number of words in texts would affect the detection performance. As shown in
Figure 1, almost all HWTs have no more than 25 words while MGT contains more
words in general, which means even a threshold based on the number of words in
a text could be a potential well-performed detector. To investigate whether a
detection method is capable of detecting MGTs that have similar #. words to
HWTs, we perform a new evaluation by filtering out the texts that have more
than 25 words.

The results with GPT2-S are shown in Table 4. We can observe that after
filtering out the texts with more than 25 words, it is harder for most
detectors to distinguish MGTs from HWTs. For instance, on TruthfulQA, Log-
Likelihood achieves 0.957 AUC with the original texts while only 0.818 with
the filtered texts. Similar trends are also observed from other datasets.
However, an exceptional case is the ChatGPT Detector, which suffers almost 0
AUC drop. We conjecture that ChatGPT Detector can still identify the unique
patterns hidden in the MGTs besides the number of words.

Table 5: The additional prompts we used to generate less detectable answers
from ChatGPT.

Type |  Prompt
---|---
Prompt1 |  I am going to give you some information before asking you to write (insert piece of writing, article, essay, letter, etc.) Do you understand?
Prompt2 |  When it comes to content and writing like a human, two factors are crucial, p̈erplexityänd b̈urstiness.̈ Perplexity measures the complexity of the text. Separately, burstiness compares variations of sentences. Humans tend to write with greater burstiness, for example, with some longer or more complex sentences alongside shorter ones. AI sentences tend to be more uniform. Therefore, when writing the following content I am going to ask you to create, I need it to have a good amount of perplexity and burstiness. Do you understand?

Effect of Different Prompts. We then investigate whether a more carefully
designed prompt can facilitate ChatGPT to generate human-like answers that can
better bypass the detection methods. Note that we take the TruthfulQA-C
dataset as a case study since most of the detectors achieve good performance
on this dataset. Regarding the prompts, before the original prompt (shown in
Table 1), we add an additional prompt to instruct ChatGPT first (shown in
Table 5). Note that these two prompts are collected from
Reddit.444https://www.reddit.com/r/ChatGPTPromptGenius/comments/11qfm7y/undetectable_ai_text_generator/.
The detection performance is reported in Figure 2. We can observe that, with
prompt1, the performance fluctuates across different detection methods. For
instance, the F1-score of Log-Likelihood drops 0.015 from the original prompt
to prompt1 while it increases the Entropy’s F1-score by 0.020. On the other
hand, we do find that prompt2 can consistently reduce the detection
performance to a larger extent. For instance, the F1-score decreases by 0.029
and 0.099 for GLTR and OpenAI Detector. This implies that a more carefully
designed prompt can indeed guide ChatGPT to generate texts with better
perplexity and burstiness, which can reduce the detection performance.
Finally, the relatively stable performance of ChatGPT Detector (with only
0.017 drops on F1-score) indicates that ChatGPT Detector is still capable of
detecting texts generated by ChatGPT, even with the prompt to generate more
human-like texts.

(a) Figure 2: Detection performance on the TruthfulQA-C dataset with different
prompts.

###  Adaptive Attacks

Table 6: Attack statistics on the texts being classified as MGT by the ChatGPT Detector. Dataset | Avg #. Words per Input | Avg Perturbed Word (%) | Avg #. Queries | Attack Success Rate
---|---|---|---|---
TruthfulQA-C | 61.98 | 6.16 | 1808.72 | 1.000
SQuAD1-C | 19.76 | 8.44 | 218.32 | 1.000
NarrativeQA-C | 30.04 | 8.59 | 459.39 | 1.000
Table 7: Examples of successful adversarial attacks on different datasets. Dataset |  ChatGPT-generated Answer |  Adversarial Examples
---|---|---
TruthfulQA-C |  I’m sorry, but I don’t have information on the 2032 U.S. presidential election as it hasn’t happened yet. I can tell you about previous elections, or provide information on current events. Is there anything else I can help you with? |  I’m sorry, but I don’t have information on the 2032 U.S. presidential election as it hasn’t happened but. I can tell you about previous elections, or provide information on current events. Is there anything else I can help you with?
In real life, the hare is faster than a tortoise. However, the hare is often depicted as losing to the tortoise in the famous fable, "The Tortoise and the Hare". The moral of the story is that slow and steady wins the race. |  In real life, the hare is faster than a tortoise. However, the hare is often depicted as losing to the tortoise in the eminent fable, "The Tortoise and the Hare". From moral of the story is that slow and steady wins the race.
No, the number of work days in a year has remained the same. However, people in California and other states may be working longer hours or have more work-related responsibilities outside of traditional work hours due to advancements in technology and the increasing need for accessibility and availability in the workplace. |  No, the number of work days in a year has remained the same. However, people in California and other states conceivably be working longer hours or have more work-related responsibilities outside of traditional work hours duly to advancements in technology and the increasing need for accessibility and availability in the workplace.
SQuAD1-C |  Johann Eck spoke to Luther about his books on behalf of the Empire. |  Johann Eck spoke to Luther about his books on designation of the Empire.
The NFL’s vice president of brand and creative is Jaime Weston, according to the passage. |  The NFL’s vice president of brand and creative is Jaime Weston, commensurate to the passage.
Tesla died on 7 January 1943. |  Tesla decease on 7 January 1943.
NarrativeQA-C |  Ernest eventually authored controversial literature. |  Ernest eventually authored disputing literature.
White informs Pink that Brown has died. |  White informs Pink that Brown has succumbed.
The Witch wanted scribes to reveal their own lies. |  The Witch wanted scribes to reveal their own reside.

Our previous evaluation shows that ChatGPT Detector performs the best and is
the most stable detection method in our ablation studies. We then take a step
further to evaluate whether the detection results are robust by introducing
adversarial attacks against the MGT. Concretely, for each dataset, we first
select the MGTs that are correctly classified by the ChatGPT Detector. Then,
we try to perturb each text using existing adversarial attacks against texts.
Note that here we consider a white-box attack proposed by Li et al. [10] and
leverage the TextAttack 555https://github.com/QData/TextAttack/. library to
implement the attack.

Table 6 shows the attack performance. We can observe that within a small
fraction of words being perturbed, e.g., 6.16%, 8.44%, and 8.59% on
TruthfulQA-C, SQuAD1-C, and NarrativeQA-C, the attack success rate achieves
1.000, which means all MGTs after perturbations have been wrongly classified
as HWTs by the ChatGPT Detector. Also, the average number of queries is higher
on the TruthfulQA-C dataset as a larger number of words in each text requires
more searching space to find the optimal perturbations.

We also show some examples of successful adversarial attacks in Table 7. We
can observe that with only small perturbations in the text, e.g., changing
“yet” to “but” on the first text, it can easily bypass the ChatGPT Detector.

This indicates that current MGT detection methods are still not robust enough
against adversarial attacks, which prompts the need for developing more robust
detection methods.

##  Conclusion

In this paper, we perform the first systematic quantification of existing MGT
detection methods. Concretely, we consider 6 metric-based detection methods
and 2 model-based detection methods under the most powerful generated pre-
trained transformer, i.e., ChatGPT. Our extensive evaluation shows that
ChatGPT Detector performs the best among all detection methods on different
datasets. Also, we observe that most of the detection methods fall short in
detecting the MGTs with fewer words (e.g., within 25 words) and the detection
performance may degrade on the MGTs generated by more carefully designed
prompts. And ChatGPT Detector remains effective even under those settings. We
then take a further step to evaluate the robustness of ChatGPT Detector by
introducing adversarial attacks to the MGTs. We find that, with only small
perturbations on the MGTs, ChatGPT Detector can be easily bypassed, which
prompts the need for developing more robust MGT detectors.

We integrate the detection methods as well as datasets into a modular-designed
framework named MGTBench. We envision that MGTBench will serve as a benchmark
tool to expedite future research on developing more advanced MGT detection
methods and/or the training procedure of LLMs.

## References

  * [1] Awesome ChatGPT Prompts.  https://huggingface.co/datasets/fka/awesome-chatgpt-prompts.
  * [2] ChatGPT.  https://chat.openai.com/chat.
  * [3] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language Models are Few-Shot Learners.  In Annual Conference on Neural Information Processing Systems (NeurIPS). NeurIPS, 2020.
  * [4] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel.  PaLM: Scaling Language Modeling with Pathways.  CoRR abs/2204.02311, 2022.
  * [5] Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei.  Deep Reinforcement Learning from Human Preferences.  In Annual Conference on Neural Information Processing Systems (NIPS), pages 4299–4307. NIPS, 2017.
  * [6] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 4171–4186. ACL, 2019.
  * [7] Sebastian Gehrmann, Hendrik Strobelt, and Alexander M. Rush.  GLTR: Statistical Detection and Visualization of Generated Text.  In Annual Meeting of the Association for Computational Linguistics (ACL), pages 111–116. ACL, 2019.
  * [8] Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu.  How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection.  CoRR abs/2301.07597, 2023.
  * [9] Tomás Kociský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette.  The NarrativeQA Reading Comprehension Challenge.  Transactions of the Association for Computational Linguistics, 2018.
  * [10] Dianqi Li, Yizhe Zhang, Hao Peng, Liqun Chen, Chris Brockett, Ming-Ting Sun, and Bill Dolan.  Contextualized Perturbation for Textual Adversarial Attack.  In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 5053–5069. ACL, 2021.
  * [11] Stephanie Lin, Jacob Hilton, and Owain Evans.  TruthfulQA: Measuring How Models Mimic Human Falsehoods.  In Annual Meeting of the Association for Computational Linguistics (ACL), pages 3214–3252. ACL, 2022.
  * [12] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  RoBERTa: A Robustly Optimized BERT Pretraining Approach.  CoRR abs/1907.11692, 2019.
  * [13] Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D. Manning, and Chelsea Finn.  DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature.  CoRR abs/2301.11305, 2023.
  * [14] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever.  Improving Language Understanding by Generative Pre-Training.  2016\.
  * [15] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language Models are Unsupervised Multitask Learners.  OpenAI blog, 2019.
  * [16] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.  Journal of Machine Learning Research, 2020.
  * [17] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang.  SQuAD: 100, 000+ Questions for Machine Comprehension of Text.  In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2383–2392. ACL, 2016.
  * [18] Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford, and Jasmine Wang.  Release Strategies and the Social Impacts of Language Models.  CoRR abs/1908.09203, 2019.
  * [19] Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano.  Learning to summarize from human feedback.  CoRR abs/2009.01325, 2020.
  * [20] Teo Susnjak.  ChatGPT: The End of Online Exam Integrity?  CoRR abs/2212.09292, 2022.
  * [21] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is All you Need.  In Annual Conference on Neural Information Processing Systems (NIPS), pages 5998–6008. NIPS, 2017.
  * [22] Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, Zac Kenton, Sasha Brown, Will Hawkins, Tom Stepleton, Courtney Biles, Abeba Birhane, Julia Haas, Laura Rimell, Lisa Anne Hendricks, William S. Isaac, Sean Legassick, Geoffrey Irving, and Iason Gabriel.  Ethical and social risks of harm from Language Models.  CoRR abs/2112.04359, 2021.


Further author information: (Send correspondence to C.L.)
C.L.: E-mail<EMAIL_ADDRESS>
# Optimizing MARVEL for the radial velocity follow-up of TESS and PLATO
transiting exoplanets

Cyprien Lanthermann  Institute of Astronomy, KU Leuven, Celestijnenlaan200D,
3001, Leuven, Belgium  Joris De Ridder  Institute of Astronomy, KU Leuven,
Celestijnenlaan200D, 3001, Leuven, Belgium  Hugues Sana  Institute of
Astronomy, KU Leuven, Celestijnenlaan200D, 3001, Leuven, Belgium  Pierre Royer
Institute of Astronomy, KU Leuven, Celestijnenlaan200D, 3001, Leuven, Belgium
Denis Defrère  Institute of Astronomy, KU Leuven, Celestijnenlaan200D, 3001,
Leuven, Belgium  Gert Raskin  Institute of Astronomy, KU Leuven,
Celestijnenlaan200D, 3001, Leuven, Belgium  Bart Vandenbussche  Institute of
Astronomy, KU Leuven, Celestijnenlaan200D, 3001, Leuven, Belgium  Andrew
Tkachenko  Institute of Astronomy, KU Leuven, Celestijnenlaan200D, 3001,
Leuven, Belgium  Hans Van Winckel  Institute of Astronomy, KU Leuven,
Celestijnenlaan200D, 3001, Leuven, Belgium

###### Abstract

The space missions TESS and PLATO plan to double the number of 4000 exoplanets
already discovered and will measure the size of thousands of exoplanets around
the brightest stars in the sky, allowing ground-based radial velocity
spectroscopy follow-up to determine the orbit and mass of the detected
planets. The new facility we are developing, MARVEL (Raskin et al. this
conference [1]), will enable the ground-based follow-up of large numbers of
exoplanet detections expected from TESS and PLATO, which cannot be carried out
only by the current facilities that achieve the necessary radial velocity
accuracy of 1 m s-1 or less. This paper presents the MARVEL observation
strategy and performance analysis based on predicted PLATO transit detection
yield simulations. The resulting observation scenario baseline will help in
the instrument design choices and demonstrate the effectiveness of MARVEL as a
TESS and PLATO science enabling facility.

###### keywords:

Telescopes, Spectrograph, Radial Velocity, Exoplanets, Observation strategy

##  1 INTRODUCTION

Since the first detection of a planet orbiting a Sun-like star outside our
solar system [2], astronomers have detected more than four thousand
exoplanets. These discoveries were made possible thanks to a variety of
techniques. Two of them in particular have been driving these numbers.
Precision photometry allows the detection of exoplanets transiting in front of
their host star while spectroscopy measures the periodic shift of radial
velocity (RV) of the host star due to its reflex movement around the center of
mass of the joint star-planet system. Both techniques have been applied from
the ground and in space.

Nowadays, the effort is shifting from the detection of exoplanets to their
characterization, intending to find earth analogs in the habitable zone of
their host star, and ultimately to discover the signature of potential bio-
markers in their atmosphere.

In this context, future space missions will measure the size of thousands of
new exoplanets thanks to the transit detection methods (NASA/TESS, ESA/PLATO),
and the space ESA/ARIEL mission will characterize the atmosphere of known
exoplanets through infrared (IR) spectroscopy.

While these missions will yield ground-breaking knowledge on a tremendous
amount of exoplanets, they will require multi-epoch, ground-based, high-
accuracy spectroscopy. Indeed, the mass and density of an exoplanet – as well
as its absolute distance from the star – can only be computed thanks to the
combination of both transit and radial velocity techniques. Furthermore, the
characterization of the stellar activity of the host star through visible
spectroscopy is required for ARIEL to extract the exoplanet signal from the
combined system spectra.

However, current ground-based facilities are far from sufficient to provide
the needed follow-up on the large number of exoplanets that will be detected
and characterized by these upcoming missions. The existing facilities either
do not reach the performance needed for this follow-up or use 4- and 8-meters
class telescopes. The latter are in high demand and expensive to operate, and
the number of available observing time for exoplanet spectroscopic follow-up
is simply too limited. This leaves a niche for new high-efficiency ground-
based facilities, to allow those space missions to realize their scientific
return at its full potential.

It is for this purpose that the MARVEL (Mercator Array for Radial VELocities)
facility has been designed (see Raskin et al. 2020 [1]; this conference). This
paper is organized as follows. Section 2 briefly presents the concept of
MARVEL. In S section 3, we describe numerical simulations performed to
investigate the optimization of the MARVEL facility. Our results are presented
in Section 4 and our conclusions in Section 5

##  2 MARVEL facility

MARVEL is a high-precision radial-velocity spectroscopy facility using an
array of four telescopes. The use of four telescopes provides more
flexibility, by allowing to observe either four different targets (one target
per telescope) or to observe the same target with the four telescopes
simultaneously, thus allowing the observation of fainter targets. MARVEL use
80-cm telescopes. Thanks to the high throughput possible with such small-size
telescopes (Raskin et al. 2020 [1]; this conference), MARVEL reaches an
overall system efficiency comparable to that of 4m-class instruments+telescope
system, for a much lower cost.

---
Figure 1:  Conceptual layout of MARVEL.

Figure 1 displays the conceptual layout of MARVEL. Each of the four telescopes
and the wavelength calibration unit is coupled to a common high-resolution
échelle spectrograph by optical fibers. The spectrograph itself delivers a
spectral resolution of at least $R=\lambda/\Delta\lambda=90\,000$, and covers
the spectral range from 390 to 920 nm at once.

Thanks to a combination of already proven, state-of-the-art instrumental
concepts, the total throughput of the facility is expected to be larger than
20%, as shown in the left panel of Fig. 2.

---
Figure 2:  Left: Throughput of the main components and of the total MARVEL
instrument. Right: Estimated peak SNR per resolution element at 550 nm for
various $V$-band magnitudes, for a single 80-cm telescope and for four such
telescopes pointing to the same target. Exposure times are 10 minutes (red)
and one hour (black).

Fig. 2-right shows the the SNR MARVEL will deliver as a function of the V-band
stellar magnitude. The SNR of 200 (in blue) is the limit to suppress the
photon noise to a level where we can reach a 1 m s-1 accuracy for an F-type
star . As the figure shows, we can reach this accuracy down to a V-magnitude
of 10.9, which is suitable for the follow-up of bright stars observed by the
PLATO and ARIEL missions. More details on the design and performances can be
found in Raskin et al. 2020 [1]; this conference.

##  3 Simulations

###  3.1 Exoplanets simulation

In order to study and optimize the MARVEL operational model, we built a
representative estimate of the exoplanet yield from PLATO, including the mass
distribution of the candidates. To this aim, we used a typical stellar
population expected in the Plato field. In this endeavor, we adopted the
exoplanets occurrence of Dressing and Charbonneau (2015) [3], and we paired
the host stars with exoplanets following the method described in Barclay et
al. 2018 [4]. To compute the RV signal induced by the exoplanets on the host
stars, we adopted one typical mass for each bin of exoplanet radii in Dressing
and Charbonneau (2015) [3], using the figure 3 of Chen and Kipping 2017 [5].
Table 1 provides the adopted typical mass for each bin.

Table 1: Bins of exoplanets sizes and their adopted Representative masses Bin name | Radii range [$R_{\Earth}$] | Typical mass [$M_{\Earth}$]
---|---|---
Giants |  $6<R<22$ cm | 300
Large Neptunes | $4<R<6$ | 30
Small Neptunes | $2<R<4$ | 10
Super-earths | $1.25<R<2$ | 3
Earth | $0.8<R<1.25$ | 1

From this simulated population of exoplanets, we used the PLATO
redbook111https://sci.esa.int/s/8rPyPew and Marchiori et al. 2019[6] to
estimate the exoplanets that PLATO would detect.

Our simulations yielded an estimated number of detections similar but not
identical to the total number of $\sim$ 4600 expected for the PLATO
yield222https://platomission.com/2018/06/04/planet-yield/. This can easily be
explained by differences in the methodology and adopted assumptions. MARVEL
being limited to follow-up Northern targets, we finally scaled our simulations
to 2300 targets, detected by PLATO and accessible for spectroscopic follow-up
from the ground by MARVEL.

###  3.2 Integration time simulation

Coupling four telescopes to a single detector, the MARVEL facility offers
various possible operational scenarios. It can be used in single-telescope
mode (1-T), observing four different (brighter) stars simultaneously, or in
four-telescope mode (4-T), observing the same (potentially fainter) target
with the four telescopes simultaneously. To evaluate the relative importance
of these two operational modes, we estimated the integration time necessary to
achieve a high enough signal-to-noise ratio (SNR) to reach a 1 m s-1 accuracy
on the radial velocity. Three of the stellar parameters were kept free in the
analysis: the effective temperature ($T_{\mathrm{eff}}$), the $V$-band
magnitude ($V$), and the star’s projected rotational velocity at the equator
($v_{\mathrm{eq}}\sin i$). The other atmospheric parameters were kept fixed at
$\log g=4.5$ (dwarfs), $v_{\mathrm{micro}}=1$ km s-1 and [Fe/H] = 0 (solar
metallicity). The regions with strong telluric lines were excluded of the
simulation.

---
Figure 3:  Limiting Johnson $V$-band magnitudes to obtain a photon-noise-
limited RV measurement accuracy of 1 m s-1 for the MARVEL instrument settings
(4-T mode). The parameters of the synthetic stellar spectra used were $\log
g=4.5$, [Fe/H] = 0, $v_{\mathrm{eq}}\sin i$ = 2 km s-1 and
$v_{\mathrm{micro}}=1$ km s-1. Wavelength regions with strong telluric lines
were excluded. The different colors are for different exposure times (Texp;
see legend).

Figure 3 shows an example of the limiting $V$-magnitude allowing to reach a 1
m s-1 accuracy as a function of the effective temperature for different
integration times.

##  4 Optimization of MARVEL

###  4.1 The required RV accuracy

---
Figure 4: Histogram of the semi-amplitude of the RV curve ($K$) of the host
stars for a typical distribution of exo-solar systems that will be detected by
PLATO.

For each of the simulated parameters of exoplanets that will be detected by
PLATO (Sect. 3.1), we computed the semi-amplitude ($K$) of the radial velocity
curve of its host stars. Figure 4 displays the distribution of $K$ in our
simulations. It reveals that an RV measurement accuracy of 1 m s-1 will allow
MARVEL to characterize a large part ($\sim$44%) of the exoplanets detected
with PLATO. If on the contrary MARVEL only reaches an accuracy of 2 m s-1, its
efficiency to perform PLATO follow-up would be highly degraded, with only
$\sim$ 20% of the exoplanets’ host presenting an RV semi-amplitude larger than
2 m s-1. To reach its purpose and to be able to lower the pressure on larger
(4m+ class) facilities, it is therefore critical for MARVEL to reach this 1 m
s-1 accuracy.

###  4.2 The integration time: 1-T vs. 4-T

The multiple telescopes of MARVEL allow us some versatility with two
observation modes. One mode uses the four telescopes on four different targets
simultaneously, with one target per telescope (1-T mode). Another mode uses
the four telescopes on a single target (4-T mode), allowing to collect four
times more photons for the same integration time.

While the 1-T mode allows the observation of more targets at the same time,
which is useful to reach the 12 000 observations per year that MARVEL aims at,
the 4T-mode would require four times less integration time on the same target
to reach the suitable accuracy on the RV. Excluding any overheads, one would
observe the same amount of targets in the same amount of time with both modes.
But in order to average the effect of the stellar activity on the RV signal,
one requires to observe for a minimum integration of time of about 20 minutes.
So, on bright stars, one may reach the required SNR in less than 20 minutes
with the 4-T mode, losing in efficiency in the observation planning, while
with the 1-T mode, one would be able to efficiently plan our observation to
avoid this unnecessary extra observing time. Having to implement both
observations mode will add complexity to the operational modes. So, to
estimate the efficiency improvement that would bring the 1-T mode, we
simulated the integration time required to reach the 1 m s-1 accuracy
described in Sect. 3.2 for every exoplanet host-star, according to our
simulations (sec. 3.1).

Figure 5 displays the histogram of the required integration time needed to
reach 1 m s-1 on stars hosting an exoplanet that would have been detected by
PLATO.

In order to measure the efficiency brought by the 1-T mode, we need to look at
the number of stars in the first 20 minutes bin of the 4T-mode. Only a few (4)
stars would require 20 minutes or less to reach the 1 m s-1 accuracy on the
RV. So, the 1-T mode would be more efficient for those 4 stars, requiring less
total observing time to observe those for stars. The efficiency of the
implementation of the 1-T mode is however negligible, relative to the total
number of stars.

From this study, we show that the implementation of the 1-T mode is not
required. We will discuss the utility of the 1-T mode implementation
furthermore in Sect. 5.

|
---|---
Figure 5: Histograms of the required integration times needed to reach a 1 m
s-1 RV accuracy. Left: 4-T mode. Right: 1-T mode

##  5 Discussion and Conclusions

The results of Sect. 4.1 show that to relieve the large facility of a large
part of the PLATO follow-up, MARVEL needs to reach an accuracy of 1 m s-1.
This accuracy is fortunately achievable thanks to the instrument design
presented in Raskin et al. 2020 [1] (this conference). The present result is
based on a simplified simulation of the number and mass distribution of the
exoplanets that PLATO should detect. The present results can be
straightforwardly updated once the PLATO input catalog and/or the PLATO
detection yield simulation will be made public. In addition, the use of the
only five typical masses to compute our RV signal could be improved, using a
linear relation between radius and mass for each of the five bins we used.
Yet, we do not expect that these improvements will significantly change the
qualitative results of the present simulations.

Section 4.2 shows that, on the one hand, the gain in efficiency brought by the
implementation of a versatile 1-T mode is negligible when aiming at a 1 m s-1
accuracy on each exoplanet hosting star detected by PLATO. On the other, the
implementation of the observing mode would add unnecessary complexity to the
operational model. However, MARVEL does not have to reach the 1 m s-1 accuracy
on every target. For instance, one would require much less precise
measurements to characterize a hot Jupiter. But efficiently use the 1-T mode
for those cases would require a prior knowledge of the RV accuracy one needs
to reach for each target. A further study is required to determined if the
portion of targets that required less than 20 minutes of observing time with
4-T mode to reach the accuracy on the RV at a level of the RV induced by the
exoplanet justify the implementation of the 1-T mode.

## Acknowledgment

The MARVEL project and team acknowledge support from the Fund for Scientific
Research of Flanders (FWO) under the Large Infrastructure Program grant
I011020N.

## References

  * [1] Raskin, G., Schwab, G., Vandenbussche, B., Van Winckel, H., De Ridder, J., Lanthermann, C., Pérez Padilla, J., Prins, S., Royer, P., Sana, H., Tkachenko, A., Defrère, D., Pember, J., Atkinson, D., Glasse, A., Pollacco, D., Güdel, M., Lüftinger, T., Brandeker, A., Stürmer, J., and Halverson, S., “Marvel, a four-telescope array for high-precision radial-velocity monitoring,” Proc.SPIE this proceeding (2020).
  * [2] Mayor, M. and Queloz, D., “A jupiter-mass companion to a solar-type star,” Nature 378(6555), 355–359 (1995).
  * [3] Dressing, C. D. and Charbonneau, D., “The Occurrence of Potentially Habitable Planets Orbiting M Dwarfs Estimated from the Full Kepler Dataset and an Empirical Measurement of the Detection Sensitivity,” apj 807, 45 (July 2015).
  * [4] Barclay, T., Pepper, J., and Quintana, E. V., “A Revised Exoplanet Yield from the Transiting Exoplanet Survey Satellite (TESS),” apjs 239, 2 (Nov. 2018).
  * [5] Chen, J. and Kipping, D., “Probabilistic Forecasting of the Masses and Radii of Other Worlds,” apj 834, 17 (Jan. 2017).
  * [6] Marchiori, V., Samadi, R., Fialho, F., Paproth, C., Santerne, A., Pertenais, M., Börner, A., Cabrera, J., Monsky, A., and Kutrowski, N., “In-flight photometry extraction of PLATO targets. Optimal apertures for detecting extrasolar planets,” aap 627, A71 (July 2019).