# GPT-4 vs. Human Translators: A Comprehensive Evaluation of Translation
Quality Across Languages, Domains, and Expertise Levels

Jianhao Yan1,2 Pingchuan Yan3∗ Yulong Chen4∗
Judy Li5 Xianchao Zhu5 Yue Zhang${}^{2,6,\text{{\char 0\relax}}}$
1 Zhejiang University 2 School of Engineering, Westlake University

3 University College London 4 University of Cambridge 5 Lan-Bridge Group

6 Institute of Advanced Technology, Westlake Institute for Advanced Study

<EMAIL_ADDRESS>
These authors contributed equally to this work.

###### Abstract

This study comprehensively evaluates the translation quality of Large Language
Models (LLMs), specifically GPT-4, against human translators of varying
expertise levels across multiple language pairs and domains. Through carefully
designed annotation rounds, we find that GPT-4 performs comparably to junior
translators in terms of total errors made but lags behind medium and senior
translators. We also observe the imbalanced performance across different
languages and domains, with GPT-4’s translation capability gradually weakening
from resource-rich to resource-poor directions. In addition, we qualitatively
study the translation given by GPT-4 and human translators, and find that
GPT-4 translator suffers from literal translations, but human translators
sometimes overthink the background information. To our knowledge, this study
is the first to evaluate LLMs against human translators and analyze the
systematic differences between their outputs, providing valuable insights into
the current state of LLM-based translation and its potential limitations.

##  1 Introduction

Recent studies show that LLMs can serve as a strong translation system and a
good substitute for NMT models Jiao et al. (2023a); Wang et al. (2023a); Enis
and Hopkins (2024); Huang et al. (2023); Wu et al. (2024a); Hendy et al.
(2023); Peng et al. (2023). For example, Jiao et al. (2023a) and Wang et al.
(2023a) find that GPT-4 can outperform commercial machine translation systems
via automatic and human evaluations. Such impressive results have hastened a
wide range of applications, including the use of GPT-4 for literary
translation Wu et al. (2024b).

Despite their impressive capabilities, the nature of LLM output compared to
human translators remains unclear. This raises two critical questions: (1)
_How do LLMs compare to human experts in translation quality?_ and (2) _Are
there fundamental differences in their outputs?_ These inquiries are
particularly relevant in light of recent research demonstrating significant
distinctions between LLM-generated and human-generated texts in general Li et
al. (2023); Bao et al. (2023). Such findings suggest that even if LLMs produce
high-quality translations, their outputs may possess unique characteristics
that distinguish them from human-produced translations.

To determine where LLMs fall within the spectrum of human translation
proficiency, which ranges from novice translators to seasoned professionals,
we study the problem by taking the current representative LLM, i.e., GPT-4,
and comparing it against human translators with different expertise. We first
conduct a preliminary study comparing human translations against GPT-4
translations, finding that _even experts cannot reach a consensus on which
translation is better_. Given these findings, we take a finer-grained
evaluation across different languages and domains, so that translation quality
can be better calibrated and systematic differences can be measured. Our
evaluation covers three language pairs from resource-rich to resource-poor,
i.e., Chinese$\leftrightarrow$English, Russian$\leftrightarrow$English, and
Chinese$\leftrightarrow$Hindi, and three domains, i.e., News, Technology, and
Biomedical. Given a source sentence, we ask junior, medium, and senior
translators and GPT-4 to generate the corresponding translation in the target
language. Then given each translation pair, we hire independent expert
annotators to label the errors in the target sentence under the MQM schema
Freitag et al. (2021). We find that GPT-4 reaches a comparable performance to
junior translators in the perspective of total errors made, and lags behind
senior ones with a considerable gap.

Our further analyses and qualitative studies show that there are imbalanced
performances for different languages and domains. From resource-rich to
resource-poor directions, GPT-4’s translation capability gradually weakens.
For resource-rich directions like Chinese$\leftrightarrow$English, GPT-4
performs comparably with junior translators and even close to medium
translators, but in Chinese$\leftrightarrow$Hindi, it even lags behind our
baseline system. The weaknesses mentioned above are also general shortcomings
of large models and reflect that although large models have achieved universal
translation with a focus on one language, translation between low-resource
languages remains a relative weakness.

To our knowledge, we are the first to evaluate LLMs against human translators
and analyze the systematic differences between LLMs and human translators.

##  2 Related Work

#### Benchmarking LLMs

Previous studies have benchmarked LLMs on various NLP tasks. Xu et al. (2020)
benchmark several LLMs on Chinese text, evaluating their Chinese ability. Ye
et al. (2024) assess LLMs through Question Answering (QA), MMLU Hendrycks et
al. (2021), and other metrics. From these tests, LLMs with larger scales are
generally proved to be more accurate except for certain tasks. Yuan et al.
(2023) demonstrates that LLMs perform well in long-context understanding and
are more capable with Out-of-Distribution, which means LLMs have a certain
degree of generalization ability.

Further to the MT field, Jiao et al. (2023b) find that GPT-4 performed
competitively with other SotA translation products. Wang et al. (2023a)
further investigated the capability of GPT-4 in document-level translation,
the results show that GPT-4 performs better than commercial translation
products and document NMT methods. Compared to them, our work empirically
shows that GPT-4 is comparable to junior human translators.

#### LLMs as Human Experts

Due to the great capacities of GPT-4 over traditional NLP models, researchers
have investigated and compared the performance of GPT-4 as human experts in
multiple NLP tasks. Zhu et al. (2024) highlight that GPT-4 and GPT-4-turbo
show top performance on a Chinese financial language understanding task. Liu
et al. (2023b) find the LLMs can be beneficial to biomedical NLP tasks. Goyal
et al. (2022) compare GPT models with several summarization models and humans,
and find that GPT can generate summaries preferred by humans. In AI for
education area, Nguyen and Allan (2024) show GPT-4’s can provide teaching
feedback for students. Maloney et al. (2024) find that GPT-4 shows close
performance compared with human participants in coordination games. Siu (2023)
show that GPT-4 is comparable to humans on technical translation tasks. Bojic
et al. (2023) find that GPT-4 can outperform human experts on linguistic
pragmatic tasks. In clinical diagnostics, Han et al. (2023) find that GPT-4
can give comparable performance to humans, and GPT-4v (vision version) can
even outperform human experts.

#### Human Evaluation for MT

Graham et al. (2013) first propose Direct Assessment (DA), which uses a
continuous score from 0 to 100 to represent the quality of a hypothesis. DA
has been adopted in WMT translation tasks for the past few years Farhad et al.
(2021); Kocmi et al. (2022, 2023). MQM Lommel et al. (2014), the annotation
used in this paper, is another widely used annotation scheme Klubička et al.
(2018); Rei et al. (2020a). It requires the annotators to annotate the error
span for each hypothesis and is shown to be more accurate and reliable than DA
Freitag et al. (2021). Thus, it is utilized in the metrics tasks of 2022 and
2023 WMT challenges Freitag et al. (2022, 2023).

#### Human Parity

The human parity for machine translation systems is first claimed by Hassan et
al. (2018), which describes a comparable performance on the WMT 2017 news
translation task from Chinese to English when compared to professional human
translations. However, this claim is challenged by the following research,
raising concerns about the limited scope of human parity. These limitations
include the expertise of human evaluators (Fischer and L"̈aubli, 2020), the
origin and quality of source sentences (Toral et al., 2018; Kim et al., 2023),
the limited scenario of comparison (Poibeau, 2022) and difficulty of
translation (Graham et al., 2020), indicating significant gaps between NMT
models and the professional translators. In this work, we evaluate whether the
SOTA LLM GPT-4 performs comparable to professional translators and what
differs between human translators and LLMs. With the above lessons in mind, we
address these limitations by hiring expert annotators, avoiding target-origin
source text, manually evaluating source sentences, and covering high-resource
to low-resource language pairs and various domains.

##  3 Preliminary Study

This section presents our preliminary study. We aim to first compare GPT-4
translations with human translations qualitatively, in a coarse manner. Our
comparison is simple and direct. We sample human-translated texts and prompt
GPT-4 to translate the same source sentence. Then, we ask expert annotators to
determine which translation is better.

Particularly, to have a quick overview of the qualities of human translations
against GPT-4 translations, we first utilize COMET-
QE111Unbabel/wmt23-cometkiwi-da-xl to score our in-house Chinese to English
human-translated documents, and select two documents with the highest score
and the lowest score. Note that our in-house translated documents are all
translated by professional translators. In this way, we gather 40 pairs of
translations from professional translators and GPT-4, respectively. Recent
findings Freitag et al. (2021) have demonstrated that crowd-sourced human
ratings are less reliable for high-quality MT evaluation. Thus, we hire six
expert annotators to compare the two translations and select the better
translations they find. We randomly shuffle the GPT-4 and human translations
to prevent annotators from identifying GPT-4.

The average win rate of GPT is 15.5/40 (36.25%). It looks like a clear win for
human translators, but when delving deeper, we find that the expert annotators
have a low ratio of agreement with each other. In Table 1, most annotators
only agree with each other at around 60% (the baseline is 50%) of an agreed
winner at each source sentence. We further conduct a significance test and
only annotator B finds human translation significantly better than GPT’s
translation and other annotators have high p-values. Given annotators’
expertise and our task is straightforward, these results indicate that even
expert annotators find it difficult to agree on which translation is better,
and GPT-generated translations might have different advantages against human-
generated ones. These results motivate us to conduct a finer-grained and
comprehensive evaluation to reveal the systematic difference between GPT-4 and
human translations.

Annotators | A | B | C | D | E | F
---|---|---|---|---|---|---
A | 100.0 | 57.5 | 65.0 | 65.0 | 62.5 | 67.5
B | - | 100.0 | 52.5 | 52.5 | 50.0 | 50.0
C | - | - | 100.0 | 65.0 | 82.5 | 67.5
D | - | - | - | 100.0 | 57.5 | 62.5
E | - | - | - | - | 100.0 | 70.0
F | - | - | - | - | - | 100.0
p-value | 1.000 | 0.038 | 0.268 | 0.081 | 0.154 | 0.875

Table 1: Ratio(%) of agreed winner across expert annotators and significance p-value for binomial test. P-value < 0.05 denotes a significant difference between GPT-4 and Human. Type | Error Name |  Explanations
---|---|---
Accuracy | Mistranslation |  Translation does not accurately represent the source.
Addition |  Information not present in the source.
MT Hallucination |  Information that has nothing related to source; or gibberish; or repeats
Omission |  Missing content from the source.
Untranslated |  Not translated.
Wrong Name Entity and Term |  Wrong usage of NE and Terminology.
Fluency | Grammar |  Problems with grammar of target language.
Punctuation |  Incorrect punctuation (for locale or style)
Spelling |  Incorrect spelling or capitalization.
Register |  Wrong grammatical register (e.g., inappropriately informal pronouns).
Inconsistent Style |  Internal inconsistency ( not related to terminology )
Unnatural Flow |  Translations that are too literal or sound unnatural.
Other | Non-translation |  -
Table 2: Error category and explanations. We mainly follow the guidelines from
Unbabel, and merge some errors to reduce the efforts for annotators to
understand the annotation system. Concrete examples for each error category
can be found in the Appendix.

##  4 Main Experimental Setup

Motivated by the results from our preliminary study, we conduct a
comprehensive and fine-grained evaluation, for revealing the systematic
difference between humans and GPTs. Specifically, we employed the widely
recognized Multidimensional Quality Metrics (MQM) framework Lommel et al.
(2014) and compared human translators with varying levels of expertise to
GPT-4. Our evaluation spans multiple languages and domains, aiming to furnish
broad insights into these comparisons.

###  4.1 Data Collection

We collect multilingual and multi-domain source sentences. Our multilingual
evaluation data contains six language directions, covering high resource to
low resource, including English to Chinese, Chinese to English, English to
Russian, Russian to English, English to Hindi, and Hindi to English.

For general domain Chinese$\Leftrightarrow$English and
English$\Leftrightarrow$Russian, we sample source sentences from test sets of
WMT2023 and WMT2022, respectively. For Chinese$\Leftrightarrow$Hindi, we
extract source news text from public websites. For multi-domain evaluation
data, we evaluate two domains, i.e., biomedical and technology and we evaluate
Chinese to English. The source sentences are extracted news texts from public
websites. We ensure that all sources are source language origin to avoid the
effect of translationese. We manually evaluate all source sentences for these
tasks and ensure the source sentences are not too easy or too short. Finally,
each task contains 200 sentences, making our evaluation a total of 1600
sentences.

###  4.2 Human Translators and Machine Translators

We ask different human translators to translate our source sentences into the
target language. Translators are of three different levels of expertise,
categorized into junior-level, medium-level, and senior-level translators. The
level of expertise is ranked by in-house criteria covering the translators’
educational background, translation experience, and practical proficiency. See
Appendix A for more details. For a fair comparison, we request the experts not
use machine translation or GPTs as assistance. For all directions except Zh-Hi
and Hi-Zh, we collect three human translation results from each level of
expertise. For Zh-Hi and Hi-Zh, we only have medium-level and senior-level
translators due to the scarcity of translators.

Except for human translators, we use gpt-4-1106-preview, the current state-of-
the-art large language model released by OpenAI and Seamless M4T Communication
et al. (2023) as the representative of traditional machine translations to
complement our experiments. We directly prompt GPT-4 to obtain the
translation, as it is the most common practice for normal users, the easiest
to reproduce, and to avoid confusion by various techniques.

###  4.3 Prompt Search

Previous study Zhao et al. (2021); Liu et al. (2023a) shows that different
prompts with LLMs can result in distinctive performance. Thus, we collect
three candidate prompts used in previous research Xu et al. (2023); Jiao et
al. (2023a) and use COMET-QE Rei et al. (2020b) to select the best prompt to
make the best use of GPT-4, as shown in Table 3. In particular, we use these
three prompts to prompt GPT-4 to translate 100 source sentences in our
Chinese-to-English test set and adopt COMET-QE to evaluate the quality of
translations. We find that the third prompt yields the best performance, and
hence we adopt this prompt for all following experiments.

Prompt |  COMET
---|---
Please translate the following sentence from Chinese into English. Your language and style should align with the language conventions of a native speaker. \n{SOURCE}\n |  0.775
You are an expert translator for translating Chinese to English. Your language and style should align with the language conventions of a native speaker. \n[Chinese]: {SOURCE}\n[English]: |  0.755
Please provide the English translation for these sentences. Your language and style should align with the language conventions of a native speaker. \n{SOURCE}\n |  0.780
Table 3: Taking Chinese to English as an example, our three prompts and
corresponding scores with COMET-QE. {SOURCE} represents the source sentence to
be translated.

###  4.4 Annotation Protocol

To evaluate the results of candidates’ systems, we hire experts to annotate
the errors of translations blindly. The annotation platform is Doccano
Nakayama et al. (2018), and the error tags are made according to MQM
standards. MQM requires the annotators to annotate the span of errors in each
hypothesis. All hypotheses of the same source sentence are shown to the
annotator together to help decide which is better. We have 13 error categories
and two severities, as shown in Table 2. Our categorization for errors mostly
follows Unbabel’s practice 222https://help.unbabel.com/hc/en-
us/articles/6444304419479-Annotation-Guidelines-Typology-3-0 and we focus on
most common error types. Each tag has subtags with two severities, i.e., Minor
or Major. A screenshot of the annotation system is given in Figure 5.

For each task, we first ask the two expert annotators to carefully read our
manual and conduct a training round on the first 10 groups of translations.
Then, we manually check these annotations to provide feedback and ask the two
annotators to check their disagreements and revise their results. After two
rounds of such training processes, we ask the annotators to finish the
remaining sentences without knowing each other’s results.

After the first round of annotation, we conduct a second round to further
refine the evaluation results. In particular, we hire another two experts for
each task and show them the previous annotation results. They are asked to
approve and make necessary modifications to previous round annotations.

Task | Cohen Kappa(Segment) | Krippendorffs(Span)
---|---|---
Reference, Re-Annotated by Freitag et al. (2021)
WMT 2020 En-De | 0.208 | 0.456
WMT 2021 En-De | 0.230 | 0.501
Ours
General Zh-En | 0.257 | 0.436
General En-Zh | 0.544 | 0.579
General En-Ru | 0.461 | 0.566
General Ru-En | 0.341 | 0.875
General Zh-Hi | 0.256 | 0.443
General Hi-Zh | 0.234 | 0.495
Technology Zh-En | 0.306 | 0.581
Biomedical Zh-En | 0.373 | 0.616
Average | 0.321 | 0.555

Table 4: Cohen Kappa (segment-level) and Krippendorffs’ Alpha (span-level)
agreement of our annotations.

###  4.5 Inter-Annotator Agreement

Error annotation with MQM is challenging, and previous work demonstrates that
the agreement scores between MQM annotations are relatively low Lommel et al.
(2014). Reasons for this could be disagreement on precise spans and ambiguous
error categorization Lommel et al. (2014). Despite the low agreement scores,
MQM is more reliable than other evaluation protocols like Direct Assessment
Freitag et al. (2021).

To compute inter-annotator agreement for MQM, we employ segment-level Cohen’s
Kappa Cohen (1960) and span-level Krippendorff’s alpha (Krippendorff, 1980).
For reference, we calculate the agreement on the annotated results of the 2020
and 2021 WMT English-to-German tasks by Freitag et al. (2021). Our IAA results
are shown in Table 4. Thanks to our two-round annotation process, our IAA
scores show a favorable agreement, indicating a good annotation quality.

##  5 Main Results

###  5.1 Overall Results

#### Analysis of Error Severity

The upper part of Figure 1 plots the averaged number of errors of different
systems and translators. Compared to our MT baseline (seamless), GPT-4 has
much fewer errors. _It performs almost as well as the junior-level translator
at the level of total errors_ , as GPT-4 is annotated with only slightly more
minor and major errors than junior translators. However, GPT-4 still has clear
performance gaps between medium or senior human translators, as it makes
considerably more mistakes than experienced translators. To our knowledge, we
are the first to report how GPT-4 is on translation against human translators.

Figure 1: _Upper_ : Error severity for each system. The gray line represents
the standard deviation for each system across tasks. _Bottom_ : Error category
analysis for each system.

#### Analysis of Error Categories

Furthermore, we plot the categories of errors in the bottom part of Figure 1.
Compared with junior human translators, GPT-4 makes more errors in the
accuracy of translations, which accounts for most of the disparity.
Interestingly, GPT-4 surpasses junior translators in fluency issues, denoting
a better capability of language usage.

In addition, Figure 2 shows the top 5 categories of errors made by different
systems. ‘Mistranslation’ is the most frequent error made by all systems.
Improving much over the seamless baseline, GPT-4 makes comparable numbers of
‘Mistranslation’ with junior and medium human translators.

For all translators, ‘Unnatural Flow’ is among the most frequent errors.
Seamless, GPT-4, and junior translators have similar levels of ‘Unnatural
Flow’, indicating possible issues of literal translation and not following
language conventions. In contrast, medium and senior translators are annotated
with significantly fewer errors of ‘Unnatural Flow’.

In addition, we notice even though GPT-4 makes much fewer ‘Wrong Name
Entity(NE)’ errors compared to Seamless, which could be beneficial because of
its huge knowledge acquired in the pre-training stage, it still has a gap
compared to human translators.

Finally, we notice that GPT-4 does not have Omission or Addition problems in
its top-5 errors, whereas even senior translators have Addition errors.

Figure 2: Top 5 categories of errors made by each system.

(a) Chinese$\leftrightarrow$English

(b) English$\leftrightarrow$Russian

(c) Chinese$\leftrightarrow$Hindi

Figure 3: Error category results for each language. Each sub-figure is the
average over two directions. We only include ‘Major’ errors here to highlight
the most severe problems. Higher values indicate more errors and the number
after each error type is the maximum number of that error.

###  5.2 Detailed Results for Each Language

In Figure 3, we present detailed results for each language pair, averaged over
two directions.

#### English-Chinese

From Figure 3(a), GPT-4 shows the great capability of translating English to
Chinese and vice versa. From the radar chart, we can see that GPT-4 makes
almost the same or slightly fewer semantic errors (Omission, Addition, and
Mistranslation errors) than Junior and Senior translators. Especially
mistranslation errors, which are generally considered most semantically
detrimental, are better than junior and senior translators. For omission and
addition errors, GPT-4 reaches almost the same level as senior translators.
However, GPT-4 made significantly more lexical, stylistic, and grammatical
errors than human translators do. The error distribution of translation of
GPT-4 meets our expectations, as in the absence of reference, GPT-4 will
translate unfamiliar words directly and literally instead of seeking online
materials or other forms of help like human translators. Furthermore, due to
the complexity and variability of Chinese, the translation of entity names or
proper nouns is usually not one-to-one, two above reasons together cause the
inferiority of the performance of GPT-4 in these aspects.

#### English-Russian

For the English-Russian translation tasks, GPT-4 made slightly more semantic
errors but the number of mistranslation errors made by GPT-4 is almost at the
same level as medium and senior translators. However, GPT-4 generally made
less stylistic, grammatical, and wrong name entity & term than junior
translators. The English-Russian translation tasks are quite challenging and
the performance of translators varies significantly, but GPT-4 still maintains
the average level overall.

#### Hindi-Chinese

As the low-resource language pair we evaluate, GPT-4 demonstrates the worst
performance across evaluated translators. We observe that GPT4 is inferior to
our MT baseline. This may be due to the small portion of Hindi and Chinese
corpora in its pre-training dataset. Specifically, making the most
‘Mistranslation’ errors of GPT-4 indicates a distance away from the language
understanding of human translators. As a comparison, SeamlessM4T performs
better in both semantic and lexical errors.

#### Discussion

Our results here manifest an imbalance of multilinguality for LLMs Wang et al.
(2023b). Our results imply that GPT-4 can serve as a reliable translator for
resource-high such as Chinese to English but is doubtful for low-resource
directions like Chinese-Hindi. In the low-resource scenario, machine
translator is more reliable.

(a) General news domain.

(b) Technology domain.

(c) Biomedical domain.

Figure 4: Error category results for different domains in Chinese-to-English.
We only include ‘Major’ errors here to highlight the most severe problems.
Higher values indicate more errors and the number in the bracket is the
maximum number of that error.

###  5.3 Detailed Results for Different Domains

Figure 4 presents our results for different domains in Chinese-to-English
translation. We compare three different domains, including news, technology,
and biomedical.

#### General News Domain

GPT-4 performs worse in the news domain than human translators of three
levels. The number of semantic errors made by GPT-4 is quite close to junior
and medium translators. Nonetheless, GPT-4 made more lexical and grammatical
errors compared to human translators. We hypothesize the reasons for the
situation described above to happen are mainly because of the literariness and
timeliness. Because GPT-4 is not able to access the online materials to
confirm the name of a specific entity or event.

#### Technology Domain

The performance of GPT-4 is relatively close to medium-level translators.
Except for the Wrong Name Entity & Terms, GPT-4 makes almost the same or even
fewer errors than medium-level translators across all aspects. Specifically,
the number of semantic errors made by GPT-4 is almost the same to medium-level
translators and it makes much fewer structural and grammatical errors. It
means that in this field, GPT-4 might understand the original text better than
junior or medium-level translators and be able to conduct a translation that
is more in line with the original meaning.

#### Biomedical Domain

Similar to the technology domain, the qualities of the translations made by
GPT-4 and medium-level translators stand at the same level. Despite slightly
more Wrong Name Entity & Terms errors made, GPT-4 performs better than junior
and medium-level translators in other aspects.

#### Discussion

For specific domains like technology, we show that GPT-4 is comparable with
junior/medium translators. We still notice a similar imbalance issue as in the
multilingual setting, but GPT-4’s performance is not as sensitive as in the
change of language.

Source | 巨人网络有限公司
---|---
GPT-4 | Giant Network Group Inc.
Human | Giant Interactive Group Inc.
Table 5: Named Entity cases.

###  5.4 Case Study

We also qualitatively understand the difference between the translations given
by GPT-4 and human translators.

#### Literal Translations

Among the error cases, the typical one is literal translations. Specifically,
we find that GPT-4 sometimes translates with semantically correct, but in-
native and literal translations. This is problematic with named entities,
especially those occurring less frequently. As shown in Table 5, when not
knowing the correct translation of ‘巨人网络有限公司’, GPT-4 translates the term word
by word. However, the issue of name entities occurs less for human
translators, partially because they would google it to find the correct
translation. Thus, this issue might be resolved by incorporating web-search
into agent-like translation Feng et al. (2024); Wu et al. (2024c).

Except for named entities, we notice that the literal translation causes
Unnatural Flows. As shown in Table 6, when translating ‘It’s just a white
screen’, GPT-4 translates the phrase to ‘它(it)只是(is just)一个(a)白屏(white
screen)’, but human translator translates this phrase to “页面显示空白(The page
display is white)”, which represents a preciser meaning and follows local
conventions.

Source |  It’s just a white screen or it times out loading it, or the page becomes unresponsive!
---|---
GPT-4 |  它只是一个白屏，要么是加载时超时，要么页面变得无响应了！
Human |  页面要么显示空白，要么加载超时或是无响应。
Table 6: Unnatural-Flow cases. Red represents the literal translation and green is more natural and native in Chinese. Source |  He has health concerns atm but we also have Daley entering his 2nd year and is a decent safety net.
---|---
GPT-4 |  他目前有健康问题，但我们还有戴利进入他的第二年，他是一个不错的安全保障。
Human |  他目前有健康问题。不过，戴利两岁了，是个不错的备选人。
Table 7: Human imagination cases. Red denotes the imagined part.

#### Human Imagination

We find human translators also have drawbacks compared to the GPT-4
translator. When the source sentence contains insufficient information to
translate, human translators tend to fill the gap by imagination or
overthinking. An example is given in Table 7. The translator wrongly
understands the phrase ‘entering his 2nd year’ as Daley is a two-year-old
baby, but the sentence describes a 2nd-year player for sports. This may be due
to daily language habits, misunderstanding, or not paying attention, and could
be related to the hallucination Zhang et al. (2023) of LLMs. GPT-4’s literal
translation helps in this, as it keeps faithful to the source sentence. This
also aligns with our findings in Section 5.1 that GPT-4 has fewer Additions or
Omissions.

##  6 Conclusion

In this study, we comprehensively evaluated the translation quality of GPT-4
against human translators of varying expertise levels across multiple language
pairs and domains. Our findings showed that GPT-4 performs comparably to
junior translators in terms of total errors made but lags behind medium and
senior translators. We also notice that GPT-4’s translation capability
gradually weakens from resource-rich to resource-poor language pairs.
Qualitative analysis revealed that GPT-4 tends to produce more literal
translations compared to human translators but suffers less from imagined
information.

The results of this study demonstrate that GPT-4 has made significant strides
in approaching human-level translation quality, as well as highlighting the
nuanced difference between them. This suggests promising opportunities for
collaboration and enhancement of translation workflows. As research continues
to advance, we anticipate that LLMs will become increasingly valuable tools in
the translation industry, working alongside human translators to improve
productivity, efficiency, and overall translation quality.

##  7 Limitations

Our work is limited in the following aspects: (1) We benchmark GPT-4 for
translation tasks, as it is a representative large language model and shows
state-of-the-art performance for many text-based tasks. However, our
evaluations can be extended to other LLMs such as Claude-3. (2) Our evaluation
covers three languages and six directions from resource-rich to resource-poor.
However, for other languages, there might be linguistic-specific phenomena
that are not covered in this paper.

## References

  * Bao et al. (2023) Guangsheng Bao, Yanbin Zhao, Zhiyang Teng, Linyi Yang, and Yue Zhang. 2023.  Fast-detectgpt: Efficient zero-shot detection of machine-generated text via conditional probability curvature.  _arXiv preprint arXiv:2310.05130_.
  * Bojic et al. (2023) Ljubisa Bojic, Predrag Kovacevic, and Milan Cabarkapa. 2023.  Gpt-4 surpassing human performance in linguistic pragmatics.  _arXiv preprint arXiv:2312.09545_.
  * Cohen (1960) Jacob Cohen. 1960.  A coefficient of agreement for nominal scales.  _Educational and psychological measurement_ , 20(1):37–46.
  * Communication et al. (2023) Seamless Communication, Loïc Barrault, Yu-An Chung, Mariano Cora Meglioli, David Dale, Ning Dong, Paul-Ambroise Duquenne, Hady Elsahar, Hongyu Gong, Kevin Heffernan, John Hoffman, Christopher Klaiber, Pengwei Li, Daniel Licht, Jean Maillard, Alice Rakotoarison, Kaushik Ram Sadagopan, Guillaume Wenzek, Ethan Ye, Bapi Akula, Peng-Jen Chen, Naji El Hachem, Brian Ellis, Gabriel Mejia Gonzalez, Justin Haaheim, Prangthip Hansanti, Russ Howes, Bernie Huang, Min-Jae Hwang, Hirofumi Inaguma, Somya Jain, Elahe Kalbassi, Amanda Kallet, Ilia Kulikov, Janice Lam, Daniel Li, Xutai Ma, Ruslan Mavlyutov, Benjamin Peloquin, Mohamed Ramadan, Abinesh Ramakrishnan, Anna Sun, Kevin Tran, Tuan Tran, Igor Tufanov, Vish Vogeti, Carleigh Wood, Yilin Yang, Bokai Yu, Pierre Andrews, Can Balioglu, Marta R. Costa-jussà, Onur Celebi, Maha Elbayad, Cynthia Gao, Francisco Guzmán, Justine Kao, Ann Lee, Alexandre Mourachko, Juan Pino, Sravya Popuri, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, Paden Tomasello, Changhan Wang, Jeff Wang, and Skyler Wang. 2023.  Seamlessm4t: Massively multilingual & multimodal machine translation.
  * Enis and Hopkins (2024) Maxim Enis and Mark Hopkins. 2024.  From llm to nmt: Advancing low-resource machine translation with claude.  _arXiv preprint arXiv:2404.13813_.
  * Farhad et al. (2021) Akhbardeh Farhad, Arkhangorodsky Arkady, Biesialska Magdalena, Bojar Ondřej, Chatterjee Rajen, Chaudhary Vishrav, Marta R Costa-jussa, España-Bonet Cristina, Fan Angela, Federmann Christian, et al. 2021.  Findings of the 2021 conference on machine translation (wmt21).  In _Proceedings of the Sixth Conference on Machine Translation_ , pages 1–88. Association for Computational Linguistics.
  * Feng et al. (2024) Zhaopeng Feng, Yan Zhang, Hao Li, Bei Wu, Jiayu Liao, Wenqiang Liu, Jun Lang, Yang Feng, Jian Wu, and Zuozhu Liu. 2024.  Tear: Improving llm-based machine translation with systematic self-refinement.
  * Fischer and L"̈aubli (2020) Lukas Fischer and Samuel L"̈aubli. 2020.  What’s the difference between professional human and machine translation? a blind multi-language study on domain-specific MT.  In _Proceedings of the 22nd Annual Conference of the European Association for Machine Translation_ , pages 215–224, Lisboa, Portugal. European Association for Machine Translation.
  * Freitag et al. (2021) Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021.  Experts, errors, and context: A large-scale study of human evaluation for machine translation.  _Transactions of the Association for Computational Linguistics_ , 9:1460–1474.
  * Freitag et al. (2023) Markus Freitag, Nitika Mathur, Chi-kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Tom Kocmi, Frederic Blain, Daniel Deutsch, Craig Stewart, Chrysoula Zerva, Sheila Castilho, Alon Lavie, and George Foster. 2023.  Results of WMT23 metrics shared task: Metrics might be guilty but references are not innocent.  In _Proceedings of the Eighth Conference on Machine Translation_ , pages 578–628, Singapore. Association for Computational Linguistics.
  * Freitag et al. (2022) Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and André F. T. Martins. 2022.  Results of WMT22 metrics shared task: Stop using BLEU – neural metrics are better and more robust.  In _Proceedings of the Seventh Conference on Machine Translation (WMT)_ , pages 46–68, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  * Goyal et al. (2022) Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022.  News summarization and evaluation in the era of gpt-3.  _arXiv preprint arXiv:2209.12356_.
  * Graham et al. (2013) Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013.  Continuous measurement scales in human evaluation of machine translation.  In _Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse_ , pages 33–41, Sofia, Bulgaria. Association for Computational Linguistics.
  * Graham et al. (2020) Yvette Graham, Christian Federmann, Maria Eskevich, and Barry Haddow. 2020.  Assessing human-parity in machine translation on the segment level.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 4199–4207, Online. Association for Computational Linguistics.
  * Han et al. (2023) Tianyu Han, Lisa C Adams, Keno Bressem, Felix Busch, Luisa Huck, Sven Nebelung, and Daniel Truhn. 2023.  Comparative analysis of gpt-4vision, gpt-4 and open source llms in clinical diagnostic accuracy: A benchmark against human expertise.  _medRxiv_ , pages 2023–11.
  * Hassan et al. (2018) Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mu Li, et al. 2018.  Achieving human parity on automatic chinese to english news translation.  _arXiv preprint arXiv:1803.05567_.
  * Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021.  Measuring massive multitask language understanding.
  * Hendy et al. (2023) Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023.  How good are gpt models at machine translation? a comprehensive evaluation.  _arXiv preprint arXiv:2302.09210_.
  * Huang et al. (2023) Hui Huang, Shuangzhi Wu, Xinnian Liang, Bing Wang, Yanrui Shi, Peihao Wu, Muyun Yang, and Tiejun Zhao. 2023.  Towards making the most of llm for translation quality estimation.  In _CCF International Conference on Natural Language Processing and Chinese Computing_ , pages 375–386. Springer.
  * Jiao et al. (2023a) Wenxiang Jiao, Wenxuan Wang, Jen tse Huang, Xing Wang, Shuming Shi, and Zhaopeng Tu. 2023a.  Is chatgpt a good translator? yes with gpt-4 as the engine.
  * Jiao et al. (2023b) Wenxiang Jiao, Wenxuan Wang, Jen tse Huang, Xing Wang, Shuming Shi, and Zhaopeng Tu. 2023b.  Is chatgpt a good translator? yes with gpt-4 as the engine.
  * Kim et al. (2023) Ahrii Kim, Yunju Bak, Jimin Sun, Sungwon Lyu, and Changmin Lee. 2023.  The suboptimal wmt test sets and its impact on human parity.  _Preprints_.
  * Klubička et al. (2018) Filip Klubička, Antonio Toral, and Víctor M Sánchez-Cartagena. 2018.  Quantitative fine-grained human evaluation of machine translation systems: a case study on english to croatian.  _Machine Translation_ , 32(3):195–215.
  * Kocmi et al. (2023) Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, et al. 2023.  Findings of the 2023 conference on machine translation (wmt23): Llms are here but not quite there yet.  In _Proceedings of the Eighth Conference on Machine Translation_ , pages 1–42.
  * Kocmi et al. (2022) Tom Kocmi, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Thamme Gowda, Yvette Graham, Roman Grundkiewicz, Barry Haddow, et al. 2022.  Findings of the 2022 conference on machine translation (wmt22).  In _Proceedings of the Seventh Conference on Machine Translation (WMT)_ , pages 1–45.
  * Krippendorff (1980) Klaus Krippendorff. 1980.  Validity in content analysis.  _Computerstrategien für die Kommunikationsanalyse_ , 69:45p.
  * Li et al. (2023) Yafu Li, Qintong Li, Leyang Cui, Wei Bi, Longyue Wang, Linyi Yang, Shuming Shi, and Yue Zhang. 2023.  Deepfake text detection in the wild.  _arXiv preprint arXiv:2305.13242_.
  * Liu et al. (2023a) Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023a.  Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.  _ACM Computing Surveys_ , 55(9):1–35.
  * Liu et al. (2023b) Zhengliang Liu, Tianyang Zhong, Yiwei Li, Yutong Zhang, Yi Pan, Zihao Zhao, Peixin Dong, Chao Cao, Yuxiao Liu, Peng Shu, et al. 2023b.  Evaluating large language models for radiology natural language processing.  _arXiv preprint arXiv:2307.13693_.
  * Lommel et al. (2014) Arle Lommel, Maja Popovic, and Aljoscha Burchardt. 2014.  Assessing inter-annotator agreement for translation error annotation.  In _MTE: Workshop on Automatic and Manual Metrics for Operational Translation Evaluation_ , pages 31–37. Language Resources and Evaluation Conference Reykjavik.
  * Maloney et al. (2024) Laurence T Maloney, Maria F Dal Martello, Vivian Fei, and Valerie Ma. 2024.  A comparison of human and gpt-4 use of probabilistic phrases in a coordination game.  _Scientific reports_ , 14(1):6835.
  * Nakayama et al. (2018) Hiroki Nakayama, Takahiro Kubo, Junya Kamura, Yasufumi Taniguchi, and Xu Liang. 2018.  doccano: Text annotation tool for human.  Software available from https://github.com/doccano/doccano.
  * Nguyen and Allan (2024) Ha Nguyen and Vicki Allan. 2024.  Using gpt-4 to provide tiered, formative code feedback.  In _Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1_ , pages 958–964.
  * Peng et al. (2023) Keqin Peng, Liang Ding, Qihuang Zhong, Li Shen, Xuebo Liu, Min Zhang, Yuanxin Ouyang, and Dacheng Tao. 2023.  Towards making the most of chatgpt for machine translation.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 5622–5633.
  * Poibeau (2022) Thierry Poibeau. 2022.  On" human parity" and" super human performance" in machine translation evaluation.  In _Language Resource and Evaluation Conference_.
  * Rei et al. (2020a) Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020a.  COMET: A neural framework for MT evaluation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2685–2702, Online. Association for Computational Linguistics.
  * Rei et al. (2020b) Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020b.  Comet: A neural framework for mt evaluation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2685–2702.
  * Siu (2023) Sai Cheong Siu. 2023.  Chatgpt and gpt-4 for professional translators: Exploring the potential of large language models in translation.  _Available at SSRN 4448091_.
  * Toral et al. (2018) Antonio Toral, Sheila Castilho, Ke Hu, and Andy Way. 2018.  Attaining the unattainable? reassessing claims of human parity in neural machine translation.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 113–123, Brussels, Belgium. Association for Computational Linguistics.
  * Wang et al. (2023a) Longyue Wang, Chenyang Lyu, Tianbo Ji, Zhirui Zhang, Dian Yu, Shuming Shi, and Zhaopeng Tu. 2023a.  Document-level machine translation with large language models.
  * Wang et al. (2023b) Wenxuan Wang, Wenxiang Jiao, Jingyuan Huang, Ruyi Dai, Jen-tse Huang, Zhaopeng Tu, and Michael R Lyu. 2023b.  Not all countries celebrate thanksgiving: On the cultural dominance in large language models.  _CoRR_.
  * Wu et al. (2024a) Minghao Wu, Thuy-Trang Vu, Lizhen Qu, George Foster, and Gholamreza Haffari. 2024a.  Adapting large language models for document-level machine translation.  _arXiv preprint arXiv:2401.06468_.
  * Wu et al. (2024b) Minghao Wu, Yulin Yuan, Gholamreza Haffari, and Longyue Wang. 2024b.  (perhaps) beyond human translation: Harnessing multi-agent collaboration for translating ultra-long literary texts.  _arXiv preprint arXiv:2405.11804_.
  * Wu et al. (2024c) Minghao Wu, Yulin Yuan, Gholamreza Haffari, and Longyue Wang. 2024c.  (perhaps) beyond human translation: Harnessing multi-agent collaboration for translating ultra-long literary texts.
  * Xu et al. (2023) Haoran Xu, Young Jin Kim, Amr Sharaf, and Hany Hassan Awadalla. 2023.  A paradigm shift in machine translation: Boosting translation performance of large language models.
  * Xu et al. (2020) Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, et al. 2020.  Clue: A chinese language understanding evaluation benchmark.  _arXiv preprint arXiv:2004.05986_.
  * Ye et al. (2024) Fanghua Ye, Mingming Yang, Jianhui Pang, Longyue Wang, Derek F. Wong, Emine Yilmaz, Shuming Shi, and Zhaopeng Tu. 2024.  Benchmarking llms via uncertainty quantification.
  * Yuan et al. (2023) Lifan Yuan, Yangyi Chen, Ganqu Cui, Hongcheng Gao, Fangyuan Zou, Xingyi Cheng, Heng Ji, Zhiyuan Liu, and Maosong Sun. 2023.  Revisiting out-of-distribution robustness in nlp: Benchmark, analysis, and llms evaluations.
  * Zhang et al. (2023) Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. 2023.  Siren’s song in the ai ocean: a survey on hallucination in large language models.  _arXiv preprint arXiv:2309.01219_.
  * Zhao et al. (2021) Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021.  Calibrate before use: Improving few-shot performance of language models.  In _International conference on machine learning_ , pages 12697–12706. PMLR.
  * Zhu et al. (2024) Jie Zhu, Junhui Li, Yalong Wen, and Lifan Guo. 2024.  Benchmarking large language models on cflue–a chinese financial language understanding evaluation dataset.  _arXiv preprint arXiv:2405.10542_.

##  Appendix A Expertise of Human Annotators

To categorize translators into junior, medium, or senior levels, we have
established a comprehensive set of criteria that take into account various
factors indicative of a translator’s expertise and experience. These factors
include the translator’s educational background, particularly the prestige of
the institution from which they graduated, as well as their length of service
in the translation industry, the duration of their translation career, the
number of translations completed, and any professional certifications they
have obtained. To ensure the ongoing competence of our translators, we conduct
quarterly assessments to evaluate their performance. For instance, to be
classified as a senior-level translator, an individual must possess a minimum
of ten years of translation experience, demonstrate exceptional proficiency by
achieving a score of 99% on our assessments, and hold the distinguished
CATTI++ translation certification. By considering these stringent criteria, we
aim to maintain a highly qualified and skilled pool of translators across all
levels of expertise.

##  Appendix B Annotation Requirements

###  B.1 Error Types

Our annotation system is built upon the open-sourced doccano system
333https://github.com/doccano/doccano. In Figure 5, we provide a screenshot of
our annotation system. For each source sentence, outputs for different systems
are given and the annotators can select spans of the text and annotate the
error type and severity.

Figure 5: A screenshot of the Doccano annotation system we use.

##  Appendix C Detailed Explanation and Guidance for Each Error Types

Our evaluation protocol largely follows the MQM criteria released by
Unbabel444. We provide a detailed annotation manual for annotators, including
an explanation for each error type as well as illustrative examples for error
types. It is included in the following:

###  C.1 Annotation Requirements

The minimum unit that can be selected and annotated is a whole word, a
whitespace, a punctuation mark, or an isolated character. In the following
example, the version in French has an extra exclamation mark, so it’s
necessary to annotate it as a Punctuation error:

_[EN] Thank you very much._

_[FR] Merci beaucoup!_

Wrong selection $\rightarrow$Merci [beaucoup!]PUNCTUATION Correct selection
$\rightarrow$Merci beaucoup[!]PUNCTUATION

If the issue occurs in a multiword expression, you will need to select the
whole expression; if, for example, an entire sentence was translated and it
shouldn’t have been, you should select the entire sentence.

In the following example, we have an Unnatural Flow error:

_[EN] Hi, Mary here._

_[ES] Hola, Mary aquí._

Wrong selection $\rightarrow$ Hola, [Mary aquí.]UNNATURAL FLOW Correct
selection $\rightarrow$ Hola, [Mary aquí]UNNATURAL FLOW.

###  C.2 Error Types

#### Accuracy

  * •

Mistranslation

    * –

Description: Translation does not accurately represent the source.

    * –

Example:  _[EN] It has to be done by the book._ _[FR] Il doit être fait [par
le livre]MISTRANSLATION_ _[Reason] The word-for-word translation into French
doesn’t work._

  * •

Addition

    * –

Description: Information not present in the source.

    * –

Example:  _[EN] That way you can be sure that you were the one who made the
changes._ _[ES] Así puedes estar seguro de que fuiste tú quien hizo [todos
ADDItIoN los cambios._ _[Reason] [Todos] (meaning ’all’ in Spanish) is not
present in the source and it is incorrectly added in the target text._

  * •

MT Hallucination

    * –

Description: information that has nothing related to source; or gibberish; or
repeats

    * –

Example:  _[EN] You can send us a follow-up email at this address [EMAIL]._
_[ES] [Hágame saber si tiene alguna otra pregunta]MT HALLUCINATION.]_
_[Reason]: The Spanish translation reads please let me know if you have any
other questions and it’s grammatically correct and fluent, but it has no
relation at all with the source.]_

  * •

Omission

    * –

Description: Missing content from the source.

    * –

Example:  _[EN] We do not have much information on this._ _[FR] Nous ne
disposons pas [] OMISSION beaucoup d’informations à ce sujet._ _[Reason]: The
French sentence requires the preposition [de] (disposer de)._

  * •

Untranslated

    * –

Description: Not translated.

    * –

Example:  _[EN] How To Make Pizza Dough_ _[FR] Comment faire de [Pizza
Dough|UNTRANSLATED_ _[Reason]: [Pizza Dough] is not a named entity and is
untranslated in the French version._

  * •

Wrong Name Entity & Term

    * –

Description: Wrong usage of NE and Terminology.

    * –

Example:  _[EN] Dear Wiley,_ _[IT] Gentile [Wilar WRONG NAMED ENTITY,_
_[Reason]: The name in the Italian version doesn’t match the original._

#### Fluency

  * •

Grammar

    * –

Description: Problems with grammar of target language.

    * –

Example:  _[EN] I understand that you want to check in online._ _[CS] chàpu,
ze se chcete [odbavení]gRAMMaR online._ _[Reason]: Wrong part of speech makes
the sentence ungrammatical in Czech._

  * •

Punctuation

    * –

Description: incorrect punctuation (for locale or style).

    * –

Example:  _[EN] Original copy of the Proof of Purchase or Invoice (not a
screenshot):_ _[PT] C’opia original do comprovante de compra ou nota fiscal
(não uma captura de tela)[.]PUNCTUATION_ _[Reason]: There’s a period instead
of a colon in the Brazilian Portuguese version of this sentence._

  * •

Spelling

    * –

Description: incorrect spelling or capitalization.

    * –

Example:  _[EN] This sort of damage is not covered under the warranty, but we
will seek assistance from a higher support and see what we can do regarding
this issue._ _[IT] Questo tipo di danno non è coperto dalla garanzia, ma
chiederò comunque aiuto ai responsabili dell’assistenza per capire che cosa
[Zi]SPELLING può fare per quanto riguarda questo problema._ _[Reason]: There’s
a typo in the sentence in Italian: the word [zi] should be [si] instead._

  * •

Register

    * –

Description: Wrong grammatical register (e.g., inappropriately informal
pronouns).

    * –

Example:  _[EN] Wishing you a great day ahead._ _[DE] Ich wünsche
[Ihnen]REGISTER einen schönen Tag._ _[Reason]: The required register for the
German translation is Informal but the pronoun [Inhen] is Formal._

  * •

Inconsistent Style

    * –

Description: internal inconsistency (not related to terminology).

    * –

Example:  _[EN] Please click on this link. […] This link will expire in 24
hours._ _[NN] Klikk på denne [lenken].[…]Denne [linken]INCONSISTENCY utloper
om 24 timer._ _[Reason]: Both [lenk] and [link] are correct in Norwegian, but
in the same document, only one should be used. Note: this is a single error,
not two_

  * •

Unnatural Flow

    * –

Description: translations that are too literal or sound unnatural.

    * –

Example:  _[EN] Zebras are ideal for animal matching._ _[DE] [Zebras sind
ideal, um bestimmte Tiere zu finden]UNNATURAL FLOW._ _[Reason] The German
translation sounds too literal, it reads like a translation, using the verb
[finden] (finding) as a translation for matching. The verb matching should be
translated as [detektieren] (detect) to read as if it was originally written
in the target language: [Zebras sind ein ideales Beispiel zur Detektion von
Wildtieren.]_

#### Other

  * •

Non-translation

##  Appendix D Extra Details

###  D.1 Translation Prompt in Preliminary Study

In two experiments, the translation prompt we use is as follows:

  * •

Please translate the following sentences from <SRC_LANG> to <TGT_LANG>. Ensure
line alignment across the document while maintaining the fluency of overall
translation.

The prompt asks GPT4 to maintain the sentence alignment of the given document,
so each sentence can be aligned back to its source sentence while being
translated at the document level. In practice, we find most times GPT4 can
follow our instructions. Occasionally, it fails to keep the sentence structure
of the document and merges some sentences in one row. In these cases, we
manually split the merged sentences.

###  D.2 Model and Decoding

For GPT-4, we use greedy search for decoding, to ensure the reproducibility of
the results. For SeamlessM4T, we use the 2.3B version of
_seamlessM4T_v2_large_ and adopt beam search with beam size 5.


# Testing a pulsating binary model for long secondary periods in red variables

J.D. Nie  Department of Astronomy, Beijing Normal University, Beijing 100875,
China<EMAIL_ADDRESS>X.B. Zhang  National Astronomical
Observatories, Chinese Academy of Sciences, Beijing 100012, China
<EMAIL_ADDRESS>B.W. Jiang  Department of Astronomy, Beijing Normal
University, Beijing 100875, China<EMAIL_ADDRESS>
###### Abstract

The origin of the long secondary periods (LSPs) in red variables remains a
mystery up to now, although there exist many models. The light curves of some
LSPs stars mimic an eclipsing binary with a pulsating red giant component. To
test this hypothesis, the observational data of two LSP variable red giants,
77.7795.29 and 77.8031.42, discovered by the MACHO project from the LMC, are
collected and analyzed. The probable eclipsing features of the light curves
are simulated by the Wilson-Devinney (W-D) method. The simulation yields a
contact and a semidetached geometry for the two systems, respectively. In
addition, the pulsation constant of the main pulsating component in each
binary system is derived. By combining the results of the binary model and the
pulsation component, we investigate the feasibility of the pulsating binary
model. It is found that the radial velocity curve expected from the binary
model has a much larger amplitude than the observed one and a period double
the observed one. Furthermore, the masses of the components based on the
density derived from the binary orbit solution are too low to be compatible
with both the evolutionary stage and the high luminosity. Although the
pulsation mode identified by the pulsation constant which is dependent on the
density from the binary-model is consistent with the first or second overtone
radial pulsation, we conclude that the pulsating binary model is a defective
model for the LSP.

binaries: close - galaxies: individual (LMC) - stars: late-type - stars:
oscillations

##  1 INTRODUCTION

Among the variable red giant stars, one sub-type exhibits long secondary
periods (LSPs). The light curves of these stars exhibit not only a short
primary period but also a long secondary period, which is approximately nine
times longer than the short one. This phenomenon has been known for several
decades (Payne-Gaposhkin, 1954; Houk, 1963). Some samples of these LSPs
variables are shown in Kiss et al. (1999). An interest in the stars with the
LSPs has been renewed by the study of Wood et al. (1999). This paper shows
that in the LMC, $\sim$25 % of all variable asymptotic giant branch (AGB)
stars show LSP roughly nine times longer than the short primary period which
is typically $\sim$ 30–200 days. Meanwhile, a study of the bright local
pulsating red giants indicates that at least one-third of these stars exhibit
LSPs (Percy et al., 2004). Soszynski also gives $\sim$30% of pulsating red
giants in the LMC with LSPs (Soszyński et al., 2007). In the period-luminosity
(P-L) diagram, it is interesting to see that the LSP variables follow a
distinct sequence (sequence D), which is roughly parallel to the radial
pulsation sequences A, B, and C for variable red giants.

The LSPs present in variable red giants have attracted a lot of attention
since their discovery, but their origin still remains mysterious. Since the
LSPs are several times longer than the fundamental radial periods, they could
not be caused by normal radial pulsations. Moreover, Wood et al. (1999) and
Wood et al. (2004) note that the LSPs can not be explained by $g^{+}$ mode for
the oscillatory $g^{+}$ mode is evanescent in convective region and it is
unlikely to be observable in a red giant because of its convective envelope.
The $g^{-}$ mode is dynamically unstable in the convection envelope, and
unable to lead to any oscillation. Regarding the nonradial $p$ modes, their
periods are rather shorter than those of the fundamental radial modes, so they
can not explain the LSPs either. Alternatively, if the LSPs are caused by some
strange modes, they would be extremely damped and should not be seen (Wood,
2000). One more possible explanation is the rotating spheroid model, which can
explain the shape of the velocity curve, but there is no reason for the
rotating period to bring about the observed P-L relation.

It seems that variable red giants with long secondary periods (hereinafter
referred to as LSPVs) can not be easily interpreted as pulsating red giants.
In this situation, a hypothesis of binarity arose. It has been suggested that
the sequence D stars could be components of close binary systems, and the LSPs
could be interpreted as the light variations caused by ellipsoidal binary
motions or eclipses. Soszyński et al. (2004, 2007) find that the sequence D
stars overlap with and have a direct continuation of the sequence E stars that
are mostly confirmed to be binaries. Radial velocity variations on the time-
scale of LSPs have also been measured in a number of sequence D stars, such
radial velocity variations being a requirement if the sequence D stars are
binaries (Hinkle et al., 2002; Olivier & Wood, 2003; Wood et al., 2004;
Nicholls et al., 2009). Some observational arguments supporting the binary
hypothesis have already been reviewed by Soszyński (2007).

After examining the light curves of LSPVs collected from the MACHO database,
we note that many of them show some nearly regular and stable, eclipse-like
light variations at the long secondary periods with a large amplitude in
comparison with that of the primary pulsation. It seems that the LSPs can be
easily interpreted as eclipses by an orbiting component. If this can be shown
to be the case, it would be direct evidence to support the binary hypothesis
for the sequence D stars. To test this idea, we propose a model – pulsating
binary, i.e. a binary system with a pulsating component. We begin with some
very probable eclipsing LSPs candidates whose light curves look like eclipsing
binary with primary and secondary minima. The light curves of two LSP stars
are collected and analyzed by using the Wilson-Devinney ( W-D, hereafter) code
and the power-spectrum method. Afterwards, the proposed eclipsing and
pulsating nature as well as their evolutionary properties are discussed.

##  2 THE DATA

Since the LSPs are sometimes as long as 1500 days, the candidate selection is
based on the long-term MACHO survey. The MACHO project is a microlensing
experiment that monitored numerous stars in the LMC, SMC and Galactic Bulge
over a time of $\sim$ 3000 days. As the by-products of this survey, a large
number of variables were discovered and monitored, including many LSPVs. From
this database, two LSPVs in the LMC are selected as our program objects. Their
F.T.S (field.title.sequence) numbers are 77.7795.29 and 77.8031.42,
respectively. They are chosen as the working sample for the following reasons:
(1) The time base of observations for both of the stars is long enough to
cover at least a couple of LSPs. (2) Their light variations are both eclipse-
like and symmetric over the long secondary period, and have distinct short
primary variation. (3) These two stars are quite different from each other not
only in their LSPs but also in the shapes of the light curves, they are
expected to be different types of binaries.

The MACHO data are taken in non-standard red and blue bandpasses. The adopted
transformation from the MACHO instrumental photometry to Kron-Cousins $R$ and
$V$ are given in Alcock et al. (1999). However, it does not work well for our
candidates in practice and brings about visible dispersion for the photometric
points. Analyzing the data sets, we find the reason mainly being that the
observed color index term “$(V_{M,t}-R_{M,t})$” in Alcock et al. (1999) brings
in large dispersion as the raw red and blue magnitudes transfer their errors
to each other. This term requires both measurements to be good at each time
when adopting the transformation. In addition, the transformation has a term
involving airmass. When the value of the airmass is greater than 2.0, the
correction for the atmosphere extinction is rather uncertain. We adopt another
transformation to avoid the above problems as Struble et al. (2006):

$\displaystyle\ R=R_{M,t}+23.90+0.1825(V_{M,t}-R_{M,t})$ (1a) $\displaystyle
V=V_{M,t}+24.22-0.1804(V_{M,t}-R_{M,t}),$ (1b)

where the term “$V_{M,t}-R_{M,t}$” is the mean color index for a red giant
instead of an individual measurement. Since color indexes of the red giants
have a small scattering of about 0.2 magnitude, the adoption of the mean
multiplied by a factor of 0.18 brings about a dispersion of only 0.04 mag,
much smaller than the photometric uncertainty. The light curves of
standardized $R$ and $V$ magnitudes for our two candidates are shown in Fig.
1.

OGLE–II (Udalski et al., 1997; Szymański, 2005) also provides the light
variation data for red giants in the $I$ band, and the time span of the light
curve overlaps the last half of the MACHO data, with $\sim$ 300 days extension
which is not long for the LSP. Besides and more importantly, the data are much
sparser than the MACHO in the overlapping time range and it is of little help
in analyzing the short primary pulsation. Thus we don’t include the OGLE–II
data for following analysis.

##  3 THE BINARY MODEL

Assuming that the phenomenon of the LSPs is caused by an orbiting component,
we can employ some binary simulation method to analyze the LSPs light curve.
The W-D method, which consists of two main FORTRAN programs – LC and DC (short
for Light Curve and Differential Correction, respectively), is a simulation
method producing the photometric solution for the binary system.

The 2007 version of the W-D code is used to analyze the eclipse-like light
curves in $R$ and $V$ bandpasses of our two candidates. Nonlinear limb-
darkening via a logarithmic form along with many other features (Wilson &
Devinney, 1971; Wilson, 1979, 1990; Kallrath et al., 1998) are used in the
code. Considering the likely close distance between the components, the effect
of reflection is taken into account.

In computing the photometric solution, the important parameters adopted in the
DC program are as follows. The orbital period for adjustment is computed by
the PDM (Phase Dispersion Minimization) method (Stellingwerf, 1978) and the
Period04 software for double-check at the beginning. The value is about 970
days for 77.7795.29 and 1700 days for 77.8031.42. Note that these orbital
periods are twice the normally adopted value for the LSP, the latter
corresponding to one cycle of long period variation apparent in the light
curve. The temperatures of the two primary stars are both set to 3311 K
according to Fox & Wood (1982) for long period stars. The initial bolometric
(X1,X2,Y1,Y2) and monochromatic (x1,y1,x2,y2) limb-darkening coefficients of
the components are taken from Van Hamme (1993). The gravity darkening
exponents are set to 0.32 for both the primary and the secondary component for
convective envelopes according to Lucy (1967). The bolometric albedos are
taken as A1=0.4 and A2=0.5 following Rucinski (1969).

The preliminarily estimated parameters are put in the first iteration of the
DC code and after several iterations, a converged solution is reached. The
photometric solutions are shown in Table 1, and the synthesized $R$ band light
curves computed from the LC code are shown in Fig. 1 together with the
observed curves of the two candidates. The photometric solutions in the table
provide insight into the orbital dynamics of the binary system. The orbit
period ($P_{\rm orb}$) adjusted by the DC code is consistent with the value
from the PDM and Period04 method. The large angles of inclination ($i$), and
argument of periastron ($\omega$) describe the orientation of the binary
orbit, while the value of $e$ reveals a nearly circular orbit. Moreover, we
find that the secondary component is a low mass M-type star, according to the
mass ratio and effective temperature. All the binary modes, including detached
mode, semidetached mode, and contact mode in the code are tried in the
solution-seeking procedure, and it is found that only one mode can reach a
converged solution for each candidate. For 77.7795.29, a contact configuration
converges, and for 77.8031.42, a semidetached system with the primary
component filling its Roche lobe converges. This implies that both of them
have a mass exchange with their components due to Roche lobe overflow. No
converged solutions were found for systems showing purely ellipsoidal
variations i.e. systems with an invisible secondary companion.

Using the photometric solutions, the theoretical radial velocities can be also
simulated by the LC code. We try this in order to compare them with some
observational facts. The bottom panel of Fig. 1 shows the synthesized velocity
curves for the primary components. The full amplitude of the velocity
variation is about 11.0 km s-1 for 77.7795.29, and 9.0 km s-1 for 77.8031.42,
where the value of the binary system mass is taken from later Section 5 (but
see discussion there for the problem of mass). Yet all the studies of radial
velocity for LSPVs (Hinkle et al., 2002; Olivier & Wood, 2003; Wood et al.,
2004; Nicholls et al., 2009) show typical amplitudes of 3-4 km s-1 and no
objects with full velocity amplitude greater than 7 km s-1 are found.
Moreover, the synthesized velocity curves of the primary components have a
period which is twice of the light variation period. In contrast, all the
observed radial velocities for the LSPVs have the same period as that of the
light variation. The synthesized radial velocities for the second stars are
also obtained from the LC code, and we find that the full amplitude is about
18.5 km s-1 for 77.7795.29 and 18.3 km s-1 for 77.8031.42. The large velocity
separations between the primary and secondary stars would have been readily
seen, while in the existing spectral observations there is no indication of
this. Therefore the synthesized velocity curve based on the binary model is
inconsistent with the observation.

In addition, we should note that, among all the parameters derived from the DC
code, some are highly reliable, including the orbital period, mass ratio and
temperature difference if we admit the binary hypothesis, and some are
relatively uncertain, such as the effective temperatures of the two
components. Several parameters derived from the LC code, such as the mass and
radius, are particularly uncertain for there are no radial velocities
available for comparison, so they are not listed in Table 1. The synthesized
radial velocities produced by the LC code are also very uncertain.

##  4 THE INTRINSIC PULSATION

For an eclipsing binary with a pulsating (primary) component, the observed
light curve could be approximately interpreted as (Zhang et al., 2009):

$l_{\rm obs}=l_{1}f_{\rm pul}+l_{2},$ (2)

where $l_{1}$ and $l_{2}$ are the calculated brightness contributed by the
primary and secondary components respectively, and $f_{\rm pul}$ denotes the
pulsating variation of $l_{1}$. With the derived binary photometric solution,
we could compute the brightness of the binary components separately at each
observation epoch by using the LC code in the W-D code package, from which the
“pure” pulsation light variation of the primary star could then be extracted.
However, we do not use this approach, because the intrinsic pulsating
amplitude of the primary component is so large that the theoretically
synthesized light curve computed by the LC code would have up-down fluctuation
and the extracted light curve would not be exactly equivalent to the true
intrinsic pulsation variations. Instead, we make use of the original data sets
to analyze the primary pulsation.

Period04, a commonly used technique for time series analysis, which utilizes
Fourier transforms as well as multiple-least-squares algorithms, is applied to
the power spectrum analysis. In doing that, we use the $R$ band time series
data which are brighter and with higher photometric accuracy than in the $V$
band. We select only those peaks in the power spectrum with signal to noise
ratio (S/N) larger than 4.0 for further discussion. Fig. 2 and Fig. 3 show the
first four-frequency solution of the data sets in the $R$ band before and
after subtraction of the most prominent remaining frequency (prewhitening). At
the top of the two figures the spectral window based on the epoches of
available observations is displayed. We notice that the alias patterns,
including the 1 c/d daily alias, are quite low in power. The next four
patterns in both figures are the step-by-step power spectrum derived from the
pulsation data. The fitting is stopped when the light curve is well fitted.
For 77.7795.29, nine frequencies are obtained, by using most parts of the
MACHO data while the last part of the MACHO data is dropped because of a large
gap. Among all the nine frequencies obtained, we find that the main power
spectrum are dominated at $f_{1}=0.0020$ c/d and $f_{2}=0.0100$ c/d. $f_{1}$
is two times the assumed orbital frequency ($f_{0}=1/970=0.00103$ c/d), it is
not accepted as the real pulsating frequency, so only $f_{2}$ remains as the
intrinsic pulsating frequency. The other derived frequencies are found to be
related to $f_{0}$ and $f_{2}$ as follows: $f_{1}=2f_{0}$,
$f_{3}=f_{2}+6f_{0}$, $f_{4}=f_{0}$, $f_{5}=4f_{0}$, $f_{6}=f_{2}+1/365$,
$f_{7}=f_{2}+7f_{0}$, $f_{8}=3f_{0}$, $f_{9}=5f_{0}$, all of them are aliases.
For the other star 77.8031.42, we obtain $f_{2}=0.0112$ c/d, the real
pulsating frequency, and the others have the relations of: $f_{1}=2f_{0}$,
$f_{3}=4f_{0}$, $f_{4}=f_{2}+0.0004$, $f_{5}=3f_{6}$, $f_{6}=2f_{0}+f_{8}$,
$f_{7}=4f_{6}$, $f_{8}=1/365$, $f_{9}=f_{2}+10f_{0}$, $f_{10}=f_{2}-0.0004$,
$f_{11}=f_{6}+f_{8}$, $f_{12}=10f_{0}$, $f_{13}=12f_{0}$, $f_{14}=f_{0}$,
where $f_{0}$ is the assumed orbital frequency. Here, 0.0004 c/d, may be the
rotation frequency of the LSPVs, in the theoretical range of 2400–10000 days
for the rotation period of red giants (Kiss et al., 2000; Wood et al., 2004).
Based on the above analysis, only one short primary pulsation is found for our
two candidates respectively and the result is consistent with previous result
from Wood et al. (1999) for LSPs stars.

The eclipsing light curve synthesis has provided us some important parameters
of the binary system such as the mass ratio and the unified radius
$r_{1}=R_{1}/A$ which normalizes the radius to the semi-major axis (actually
the accurate ratio can be obtained by the LC code). With these values, the
mean density of the pulsating primary component can be precisely determined.
From the Kepler’s law

$P_{\rm orb}^{2}=\frac{4\pi^{2}A^{3}}{G(M_{1}+M_{2})}$ (3)

and the density in solar units

$\rho_{1}/\rho_{\sun}=\frac{M_{1}/R_{1}^{3}}{M_{\sun}/R_{\sun}^{3}},$ (4)

we get:

$\rho_{1}/\rho_{\sun}=\frac{4\pi^{2}R_{\sun}^{3}}{M_{\sun}G(1+q)r_{1}^{3}P_{\rm
orb}^{2}},$ (5)

where $P_{\rm orb}$ is the orbit period of the binary system.

Substituting the values of $P_{\rm orb}$, $q$ and $r_{1}$ from the W-D code
analysis of the light curve into Eq. (5), the mean density is computed, being
$8.47\times 10^{-8}$ and $3.67\times 10^{-8}$ in solar unit respectively,
which agrees with the density of red giants. Then the pulsation constant can
be easily calculated from: $Q=P_{\rm{pul}}(\rho_{1}/\rho_{\sun})^{1/2}$, where
$P_{\rm pul}$ is the “pure” pulsation period. The pulsation constant $Q$ is an
important parameter to tell the intrinsic oscillation mode of a pulsating
star, especially in normal radial mode. The $Q$ value calculated on the binary
hypothesis, is shown in Table 3 and equal to 0.029 and 0.017 for our two
candidates, respectively.

The values of $r1$ calculated by the LC code in Table 3, 0.47 and 0.44
respectively, are consistent with that estimated in Soszyński et al. (2007)
which notices $R_{1}/A\approx 0.4$ in the binary scenario. The result of mode
identification shows agreement with the theoretical expectation for red giants
(Fox & Wood, 1982) and reveals that 77.7795.29 is a first overtone pulsating
star while 77.8031.42 is a second overtone pulsating star. The pulsation
properties derived from a binary model are reasonable for a pulsating red
giant.

##  5 THE EVOLUTIONARY PROPERTY

A further analysis of the pulsating binary model is carried out from the
evolutionary property of the components, as the stellar properties of the red
giants are rarely known, by making use of the results derived in Section 4.
From the 2MASS near-infrared photometric data and the empirical bolometric
correction factor $BC_{\rm K}$ as a function of $(J-K)$ (Bessell & Wood,
1984), and with a distance modulus of 18.54 for the LMC, the bolometric
magnitude can be calculated, and the luminosity can thus be obtained. Applying
the formula $L=4\pi\sigma R^{2}T_{\rm{eff}}^{4}$, the equilibrium radius of
the primary star can be computed. Here, $T_{\rm{eff}}$ is an assumed value,
3311 K, which is the mean temperature for long period variables in LMC. We do
not calculate the temperature from the color index because the secondary
component is not understood well and its contribution to the observed color
index is not known. With the mean density and the radius, the mass of the
primary star is derived. All these values are listed in Table 3.

The luminosity of the primary stars in the table is a few thousands of solar
luminosity, consistent with the red giant luminosity. The radii of both
candidate stars are on the order of hundreds of solar radius, which are
consistent with red giants. However, the masses of the two LSPVs are too low,
and the masses of their components would be even lower, according to the mass
ratios in Table 1. Moreover, these low mass stars have very high luminosities,
which seems impossible. For the star 77.7795.29, the mass is 0.31$M_{\sun}$
and its luminosity is 2502$L_{\sun}$. In the age of universe, such a low mass
star is impossible to evolve up to the red giant branch. For the secondary
component, the mass is about 0.188$M_{\sun}$, while its luminosity is
1607$L_{\sun}$, the value of which is computed from the data of
“$L_{1}/(L_{1}+L_{2})$” in Table 1. It is hard to imagine how such low mass
stars in a binary system exchange their masses. It is difficult to find an
evolutionary path to satisfy this situation. This problem is also present in
the star 77.8031.42, with a mass of 0.36$M_{\sun}$ and a luminosity of
4839$L_{\sun}$, whose component has about 0.175$M_{\sun}$ and 3790$L_{\sun}$.
If a reasonable mass is expected, the density should be upgraded by a factor
of four or the radius be upgraded by a factor of 1.4. Recalling the
determination of the density from Eq. (5), the density is calculated from the
parameters, $q$, $r_{1}$ and $P_{\rm orb}$ all of which have relatively small
errors which will not bring about a density uncertainty of a factor of four.
Besides, the density is reasonable for a red giant. On the other hand, the
radius is derived from the luminosity and effective temperature. The
luminosity has uncertainty from the bolometric correction and the neglect of
interstellar extinction, and the effective temperature also has some
uncertainty. If the luminosity is higher or the effective temperature lower,
the radius would be larger and so the mass. However, if the mass is indeed
higher, the system would then have a large separation from Eq.(3) and an even
more serious problem with the large velocity amplitude. In summary, the
pulsating binary model requires a low mass incompatible with an evolved
stellar stage having the observed high luminosity.

##  6 DISCUSSION AND CONCLUSION

The origin of the LSPs is unknown and there exist many explanations. The light
curves of some LSPVs are eclipse-like and it seems that this phenomenon is due
to an invisible component orbiting around the pulsating red giant. To test
this hypothesis, we propose a model – pulsating binary, and select two LSP
stars to analyze their orbit motions and pulsation nature.

On the assumption of binarity, we simulate the photometric light curves of the
systems by using the W-D method. The photometric solutions give us a
configuration for the binary system: a contact system for 77.7795.29 and a
semidetached system for 77.8031.42 with its Roche lobe fully filled. It means
that the LSP star may have strong interaction and very probably mass transfer
with the other component via Roche lobe overflow. However, Roche lobe overflow
will rule out the binary hypothesis if we accept the view of Wood et al.
(2004), which argues that the mass transfer of Roche lobe will result in a
short merger timescale of about 1000 yrs. Moreover, the calculated effective
temperatures and the “$L_{1}/(L_{1}+L_{2})$” values in Table 1 suggest that
the secondary star is also a red giant with a similar temperature and
luminosity to that of the primary red giant. This would lead to a double-lined
spectroscopic binary. However, no observer of radial velocities in these
systems has reported seeing spectral lines from the secondary star (Hinkle et
al., 2002; Olivier & Wood, 2003; Wood et al., 2004; Nicholls et al., 2009).
There are also some serious problems about the synthesized velocity curves.
The simulation produces one cycle of velocity curve for two cycles of the
light curve, while the observed velocities show one cycle of the radial
velocity curve for one cycle of the light curve. The synthesized full velocity
amplitudes for the star 77.7797.29 and 77.8031.42 are much greater than the
typical values of other LSPVs. The full amplitudes of the secondary components
are even larger and would lead to great velocity separations between the
primary and the secondary stars, which have never been found by spectral
observations.

Fourier analysis is applied to investigate the intrinsic oscillation of the
LSPVs over the raw data sets. Using the parameters ($R_{1}/A$ and $q$)
obtained from the W-D code, the mean densities of the LSPVs are deduced and
they are consistent with the red giant phase. Then the pulsation constants are
obtained by using the classical equation
$Q=P_{\rm{pul}}(\rho_{1}/\rho_{\sun})^{1/2}$ to identify the pulsation mode.
For star 77.7795.29, one pulsating frequency is detected and it is caused by
the first overtone radial pulsation, and for star 77.8031.42, the only
pulsating frequency is caused by the second overtone radial pulsation. These
agree with the theoretical value for red giants and the conclusion of Wood et
al. (1999).

The stellar properties of LSPVs are derived by using the information from the
pulsating binary model. We calculate the bolometric magnitude, luminosity,
radius and mass of the primary star, and find some of them conflict very
seriously with the evolutionary properties of red giants. In particular, the
masses for the star 77.7795.29 and 77.8031.42 are both less than
0.4$M_{\sun}$, and their luminosities are too high for such low masses. It is
difficult to imagine how such low mass stars could have such large
luminosities. The situation gets even worse if we apply the mass ratio and the
value of “$L_{1}/(L_{1}+L_{2})$” to calculate the mass and luminosity of the
secondary star.

Therefore, the radial velocities and the masses computed from the pulsating
binary model do not agree with some observations and facts about red giants.
We conclude that the model “pulsating binary” has some deficiencies in dealing
with the observed properties of LSPVs and that the binary hypothesis for
explaining the LSPs seems unreasonable.

The authors are very grateful to Prof. Peter Wood for very constructive
suggestions and helpful discussion. They also thank the anonymous referee for
helpful suggestions to improve the work significantly. This research is
supported by the National Natural Science Foundation of China (NSFC) through
grant 10778601 and the Ministry of Science and Technology of the People’s
Republic of China through grant 2007CB815406.

## References

  * Alcock et al. (1999) Alcock, C., et al. 1999, PASP, 111, 1539
  * Bessell & Wood (1984) Bessell, M. S., & Wood, P. R. 1984, PASP, 96, 247
  * Derekas et al. (2006) Derekas, A., Kiss, L. L., Bedding, T. R., Kjeldsen, H., Lah, P., & Szabó, Gy. M. 2006, ApJ, 650, L55
  * Fox & Wood (1982) Fox, M. W., & Wood, P. R. 1982, ApJ, 259, 198
  * Hinkle et al. (2002) Hinkle, K. H., Lebzelter, T., Joyce, R. R., & Fekel, F. C. 2002, AJ, 123, 1002
  * Houk (1963) Houk, N. 1963, AJ, 68, 253
  * Kallrath et al. (1998) Kallrath, J., Milone, E. F., Terrell, D., & Young, A. T. 1998, ApJ, 508, 308
  * Kiss et al. (1999) Kiss, L. L., Szatmáry, K., Cadmus Jr, R. R., & Mattei, J. A. 1999, A&A, 346, 542
  * Kiss et al. (2000) Kiss, L. L., Szatmáry, K., Szabó, G., & Mattei, J. A. 2000, A&AS, 145, 283
  * Lucy (1967) Lucy, L. B. 1967, Z. Astrophys, 65, 89
  * Nicholls et al. (2009) Nicholls, C. P., Wood, P. R., Cioni, M.-R. L., & Soszyński, I. 2009, MNRAS, 399, 2063
  * Olivier & Wood (2003) Olivier, E. A., & Wood, P. R., 2003, ApJ, 584, 1035
  * Payne-Gaposhkin (1954) Payne-Gaposhkin, C. 1954, Harvard Ann., 113, 4
  * Percy et al. (2004) Percy, J. R., Bakos, A. G., Besla, G., Hou, D., Velocci, V., & Henry, G. W. 2004, in ASP Conf. Ser. 310, Variable Stars in the Local Group, ed. D. W. Kurtz & K. R. Pollard (San Francisco. CA: ASP), 348
  * Rucinski (1969) Rucinski, S. M. 1969, Acta Astron., 19, 245
  * Soszyński et al. (2004) Soszyński, I., et al. 2004, Acta Astron., 54, 347
  * Soszyński (2007) Soszyński, I. 2007, ApJ, 660, 1486
  * Soszyński et al. (2007) Soszyński, I., et al. 2007, Acta Astron., 57, 201
  * Stellingwerf (1978) Stellingwerf, R. F. 1978, ApJ, 224, 953
  * Struble et al. (2006) Struble, M. F., Galatola, A.,Faccioli, L., Alcock, C., & Cruz, K. 2006, AJ, 131, 2196
  * Szymański (2005) Szymański, M. K. 2005, Acta Astron, 55, 43
  * Udalski et al. (1997) Udalski, A., Kubiak, M., Szymański, M. 1997, Acta Astron, 47, 319
  * Van Hamme (1993) Van Hamme, W. 1993, AJ, 106, 209
  * Wilson & Devinney (1971) Wilson, R. E., & Devinney, E. J. 1971, ApJ, 166, 605
  * Wilson (1979) Wilson, R. E. 1979, ApJ, 234, 1054
  * Wilson (1990) Wilson, R. E. 1990, ApJ, 356, 613
  * Wood et al. (1999) Wood, P. R., et al. 1999, in IAU Symp. 191, AGB stars, ed. T. Le Bertre, A. Lèbre, & C. Waelkens (San Francisco: ASP), 151
  * Wood (2000) Wood, P. R. 2000, PASA, 17, 18
  * Wood et al. (2004) Wood, P. R., Olivier, E. A., & Kawaler, S. D. 2004, ApJ, 604, 800
  * Wood (2006) Wood, P. R. 2006, Mem. S. A. It., 77, 76
  * Zhang et al. (2009) Zhang, X. B., Zhang, R. X., Li, Q. S. 2009, RAA, 9, 422

Figure 1: Light and velocity curves synthesized by the W-D code for 77.7795.29
and 77.8031.42. Also shown are the observed $R$ and $V$ band light curves from
the MACHO project. Red asterisk represents the observed data point in $R$ band
while the blue one is the $V$ band data point. The black solid line in the $R$
band panel denotes the theoretical synthesized light curve. The bottom panel
is the synthesized velocity curve.

Figure 2: Power spectrum and light curve of the primary star 77.7795.29 in
binary system. The top five panels show power spectra and the bottom panel
shows the light curve in the $R$ band. The black solid light curve is the fit
of frequency solution described in Sect. 4 of the text.

Figure 3: The same as Fig. 2, but for the star 77.8031.42.

Table 1: Photometric solution for stars 77.7795.29 and 77.8031.42. | 77.7795.29 |  | 77.8031.42
---|---|---|---
classification | contact |  | semidetached
$P_{\rm orb}$ (day) | 970.9 |  | 1700.9
$T_{1}$ (K) | 3311(assumed) |  | 3311(assumed)
$T_{2}$ (K) | 3284 |  | 3544$\pm{48}$
$e$ | 0.0016$\pm{0.0008}$ |  | 0.0123$\pm{0.0164}$
$i$ (degree) | 62.49$\pm{0.27}$ |  | 63.19$\pm{2.22}$
$\omega$ | 3.64$\pm{0.20}$ |  | 2.91
$q=m_{2}/m_{1}$ | 0.608$\pm{0.003}$ |  | 0.486$\pm{0.029}$
$\varphi_{1}$ | 2.8359$\pm{0.0041}$ |  | 2.8748
$\varphi_{2}$ | 2.8359 |  | 2.8943 $\pm{0.055}$
$L_{1}/(L_{1}+L_{2})$ (V band) | 0.609 |  | 0.561

Table 2: Results of Fourier analysis for stars 77.7795.29 and 77.8031.42. |  | 77.7795.29 |  |  |  | 77.8031.42 |  |
---|---|---|---|---|---|---|---|---
F | Frequency | Amplitude | Phase | S/N | Frequency | Amplitude | Phase | S/N
| (c/d) | (mag) |  |  | (c/d) | (mag) |  |
$f_{1}$ | 0.0020 | 0.2138 | 0.3144 | 21.6729 | 0.0011 | 0.1464 | 0.46905 | 19.3482
$f_{2}$ | 0.0100 | 0.0529 | 0.6576 | 22.4815 | 0.0112 | 0.0770 | 0.7190 | 44.7148
$f_{3}$ | 0.0162 | 0.0327 | 0.0015 | 13.9081 | 0.0022 | 0.0613 | 0.2773 | 35.6114
$f_{4}$ | 0.0011 | 0.0363 | 0.9538 | 15.4348 | 0.0116 | 0.0446 | 0.7607 | 25.9384
$f_{5}$ | 0.0041 | 0.0280 | 0.6792 | 11.9004 | 0.0120 | 0.0404 | 0.6926 | 23.4930
$f_{6}$ | 0.0130 | 0.0205 | 0.8848 | 8.7069 | 0.0040 | 0.0260 | 0.1102 | 15.1249
$f_{7}$ | 0.0174 | 0.0189 | 0.4320 | 8.1379 | 0.0162 | 0.0267 | 0.1793 | 15.4983
$f_{8}$ | 0.0031 | 0.0199 | 0.0318 | 8.0322 | 0.0029 | 0.0351 | 0.8120 | 20.3622
$f_{9}$ | 0.0049 | 0.0199 | 0.6315 | 8.4640 | 0.0169 | 0.0224 | 0.8329 | 13.0511
$f_{10}$ |  $\cdots$ |  $\cdots$ |  $\cdots$ |  $\cdots$ | 0.0108 | 0.0311 | 0.7704 | 18.1114
$f_{11}$ |  $\cdots$ |  $\cdots$ |  $\cdots$ |  $\cdots$ | 0.0070 | 0.0287 | 0.0900 | 16.6748
$f_{12}$ |  $\cdots$ |  $\cdots$ |  $\cdots$ |  $\cdots$ | 0.0057 | 0.0236 | 0.3752 | 13.7059
$f_{13}$ |  $\cdots$ |  $\cdots$ |  $\cdots$ |  $\cdots$ | 0.0065 | 0.0242 | 0.2452 | 14.0897
$f_{14}$ |  $\cdots$ |  $\cdots$ |  $\cdots$ |  $\cdots$ | 0.0006 | 0.0206 | 0.8211 | 11.9582

Table 3: Parameters of the primary stars  Star | Frequency | $R_{1}/A$ | $\rho_{1}/\rho_{\sun}$ | Q | Mode | $M_{bol}$ | $L/L_{\sun}$ | $R/R_{\sun}$ | $M/M_{\sun}$
---|---|---|---|---|---|---|---|---|---
| (c/d) |  |  |  |  | (mag) |  |  |
77.7795.29 | 0.0100 | 0.47 | 8.47E-8 | 0.029 | 1H | -3.75 | 2502 | 153 | 0.31
77.8031.42 | 0.0112 | 0.44 | 3.66E-8 | 0.017 | 2H | -4.46 | 4839 | 213 | 0.36