# From Effectiveness to Efficiency: Comparative Evaluation of Code Generated
by LCGMs for Bilingual Programming Questions

Weipeng Jiang1, Xuanqi Gao1, Juan Zhai2, Shiqing Ma2, Xiaoyu Zhang1, Chao
Shen13  1School of Cyber Science and Engineering
Xi’an Jiaotong University
lenijwp, gxq2000<EMAIL_ADDRESS><EMAIL_ADDRESS>2Manning
College of Information & Computer Sciences
University of Massachusetts, Amherst
juanzhai<EMAIL_ADDRESS>
###### Abstract

Large Code Generation Models (LCGMs) have garnered significant attention and
achieved promising results across various programming tasks. However, concerns
arise regarding performance when using non-English prompts, as these models
are primarily trained on English-centric corpora, and most programming
language tokens resemble English. Existing benchmarks often rely on English
programming questions and limited manual unit test cases, inadequately
assessing LCGM-generated code quality. This paper investigates code quality
differences, specifically effectiveness and efficiency, when employing
different natural languages as inputs, focusing on Chinese and English due to
their prominent corpora and LCGM availability. Evaluating LCGM-generated code
quality under bilingual inputs presents three challenges: (1) lack of high-
quality bilingual programming question datasets, (2) insufficient unit test
cases for comprehensive correctness verification, and (3) limited support for
comparing generated code performance. To address these challenges, we curated
a test suite of 52 bilingual programming questions and developed automated
input generators for each. We enhanced correctness verification by sampling
larger unit test cases and estimated code performance by profiling execution
time relative to input size growth. Using this framework, we conducted an
empirical study on six state-of-the-art LCGMs. The results revealed that LCGM-
generated code exhibits varying bilingual correctness on an average of 10.5%
of tasks, with 39.5% of correct code showing diverse bilingual performance
differences. Our findings suggested LCGMs may not consistently generate high-
quality code across different languages, providing insights for future
research directions.

###### Index Terms:

Large Code Generation Models, Code Quality, Effectiveness and Efficiency,
Bilingual Evaluation

11footnotetext: 3: Chao Shen is the corresponding author. Email:
<EMAIL_ADDRESS>
##  I INTRODUCTION

The rapid advancement of Large Code Generation Models (LCGMs) has
revolutionized the field of software engineering, offering promising solutions
to a wide range of programming tasks. These models, built on the architecture
of large language models and trained on vast corpora of source code, have
demonstrated remarkable capabilities in generating functional code snippets
[1, 2], assisting with programming tasks [3, 4, 5], and even automating entire
software development workflows [6, 7].

While LCGMs support multiple input languages, facilitating developers to
describe requirements in their preferred natural language, a critical question
arises concerning the consistency of code generation capabilities across
languages. This concern is particularly significant as LCGMs are predominantly
trained on English-centric corpora, potentially impacting their performance
when prompted with inputs in other languages. Furthermore, the fact that most
programming languages originate from English, e.g., using English words as
reserved keywords, further exacerbates this concern.

Existing evaluation benchmarks for LCGMs primarily focus on English
programming questions and rely on a limited number of manually crafted unit
test cases for code correctness validation [8, 9], lacking a comprehensive
assessment of code quality, especially regarding generation effectiveness and
efficiency. The absence of diverse input languages in these benchmarks hinders
understanding the performance of LCGMs when prompted with various natural
languages. Moreover, measuring the efficiency of the generated code itself is
a challenging problem.

This paper aims to address this research gap by conducting the first extensive
empirical study to comprehensively observe and compare the code quality
differences of mainstream LCGMs on multilingual programming questions.
Specifically, we focus on English and Chinese, as these languages have a
significant presence in the programming community and a wealth of available
LCGMs and corpora [10]. By conducting a comparative analysis of the LCGM-
generated code for bilingual programming questions, we aim to shed light on
the potential disparities in code quality and provide insights into the
factors influencing these differences. We design the following three research
questions to assess the differences in LCGM-generated code across English and
Chinese:

  * •

(RQ1) Are LCGM-generated code equally effective when using bilingual
programming questions?

  * •

(RQ2) Are LCGM-generated code equally efficient when using bilingual
programming questions?

  * •

(RQ3) What is the impact of prompting on bilingual quality differences across
languages?

Evaluating the quality of LCGM-generated code under bilingual inputs poses
several challenges. Firstly, there is a scarcity of high-quality bilingual
parallel programming questions datasets that can serve as a reliable
benchmark. Secondly, the limited number of unit test cases used in existing
benchmarks may not adequately ensure the thorough correctness verification of
the generated code [11]. Lastly, there is a lack of support for measuring and
comparing the performance of the generated code, which is crucial for
assessing their efficiency.

To address these challenges, we have carefully curated a comprehensive test
suite of 52 bilingual programming questions through a rigorous manual
preparation process. Furthermore, we develop automated input generators for
each programming question, which facilitate the generation of a variety of
test cases. We improve the correctness verification process by extensively
sampling larger unit test cases using our input generator. To measure the
performance of the generated code, we implement a profiling mechanism that
measures the code execution time relative to the growth of the input size.
This allows us to compare the efficiency of the generated code across
different languages and input sizes.

Using this evaluation framework, we conduct an empirical study on six state-
of-the-art LCGMs. According to our results, we summarize a series of findings
and implications that may guide future research on the consistency of LCGM
multilingual capabilities. We highlight some key findings as follows:

  1. 1.

Existing LCGMs exhibit significant disparities in both effectiveness and
efficiency when generating code for programming questions separately presented
in English and Chinese.

  2. 2.

As the temperature parameter increases, the quality of code generated by most
LCGMs tends to shift from favoring Chinese towards favoring English.

  3. 3.

Different prompting strategies significantly impact the bilingual code quality
generated by LCGMs, highlighting the potential of prompting to alleviate the
cross-linguistic performance gap, suggesting it as a potential area for
further research.

Overall, our work confirms that LCGMs do not consistently generate high-
quality code across different languages, highlighting the need for further
investigation into the factors influencing these disparities.

Contributions. The contributions of this paper are as follows:

  * •

We conduct the first study specifically focusing on investigating the code
quality differences of LCGMs on English-Chinese bilingual programming
questions.

  * •

We collect a dataset of 52 bilingual programming questions and perform a
systematic evaluation on six mainstream LCGMs, observing that the LCGM-
generated code shows different bilingual correctness on an average of 10.5%
programming tasks, where 39.5% correct code also exhibits diverse bilingual
performance differences.

  * •

We summarize a series of findings and implications, then provide guidance and
inspiration for future research.

  * •

We release our dataset and evaluation framework [12], aiming to facilitate
further research on multilingual code generation quality assessment.

##  II BACKGROUND

Large Language Models (LLMs) denote language models characterized by billions
of parameters, meticulously trained on extensive textual corpora to excel in
various language processing assignments. Particularly, Large Code Generation
Models undergo training and fine-tuning procedures specifically designed for
programming languages, thereby augmenting their proficiency in executing
software engineering tasks [13, 14]. The success of these models can be
attributed to their training on vast corpora of code and natural language
data, which allows them to learn the intricate patterns, syntax, and semantics
of programming languages.

Thus, the quality of the training corpus of LCGMs is a critical factor in
determining the accuracy, diversity, and contextual relevance of the code they
generate. The training corpora used for these LCGMs typically consist of a
diverse collection of open-source code repositories, such as those hosted on
GitHub [15]. These repositories encompass a wide range of programming
languages, e.g., Python, Java, C++, and JavaScript. However, it is important
to recognize that these repositories are English-heavy, i.e., project
descriptions, functional documentations, code comments, and issue discussions
are always written in English, as it is the predominant language used in
global software development [10]. Additionally, many programming languages,
such as Python and Java, have syntax and keywords that closely resemble
English words and phrases [16]. Consequently, the datasets used for training
LCGMs are heavily skewed towards English-language content. This linguistic
disparity raises concerns regarding the effectiveness of LCGMs in supporting
and generating code for tasks presented in languages other than English,
particularly those with distinct linguistic structures. Naturally, this leads
to an urgent need for an in-depth analysis of whether LCGMs exhibit consistent
capabilities and quality in generating code across different languages.
Unfortunately, existing code generation evaluation benchmarks (e.g., HumanEval
[8] and MBPP [9]) fail to address this question, as they predominantly focus
on English programming challenges. Besides, these benchmarks focus on
providing unit test cases to verify the correctness of code generated by
LCGMs. In software engineering practice, assessing code quality often requires
consideration of additional factors beyond correctness, such as code
efficiency.

##  III STUDY DESIGN

Figure 1: The workflow of our evaluation.

###  III-A Goals and Research Questions

To investigate the performance disparities of LCGMs in generating code across
English and other languages, we select Chinese as a comparative case study.
Chinese is one of the most widely spoken languages globally and also one of
the most popular languages among software developers and targeted users of
LCGMs. Owing to the rapid growth of the Chinese software industry [17], the
scale of Chinese corpora (e.g., GitHub repositories with Chinese
documentation) is substantial, relative to other non-English languages. As
such, most LCGMs show strong performance on leaderboards related to Chinese
programming tasks [18]. Moreover, Chinese exhibits significant linguistic
divergence from English in terms of its writing system, grammatical structure,
and lexical inventory [19], presenting a distinct challenge for LCGMs
predominantly trained on English-heavy corpora. Our study aims to conduct a
fine-grained comparative analysis of the code generation quality (i.e., both
effectiveness and efficiency) of existing LCGMs when presented with
programming tasks in English and Chinese.

However, this investigation encounters several challenges. Firstly, there is a
shortage of high-quality bilingual parallel programming question datasets.
Existing datasets mainly provide questions in English. Secondly, current
benchmarks for evaluating code effectiveness often rely on a limited number of
unit test cases, which have been proven to be insufficient [11]. Furthermore,
those benchmarks do not emphasize measuring the efficiency of the generated
code, resulting in a lack of readily available measurement tools. To address
these challenges, we propose a novel evaluation framework in this paper that
includes a carefully curated bilingual programming question dataset. This
framework enables us to validate the correctness of the generated code and
estimate its execution performance under various inputs, thereby facilitating
the assessment of the effectiveness and efficiency of LCGMs in generating code
for bilingual programming questions and further exploring the quality
differences. Ultimately, based on the proposed framework, we aim to answer the
following research questions:

  * •

RQ1: Are LCGM-generated code equally effective when using bilingual
programming questions?

  * •

RQ2: Are LCGM-generated code equally efficient when using bilingual
programming questions?

  * •

RQ3: What is the impact of prompting on bilingual quality differences across
languages?

###  III-B Evaluation Framework

####  III-B1 Design Overview

To enable simultaneous measurement of the effectiveness and efficiency of
LCGM-generated code for bilingual programming questions, we propose a
framework, consisting of four steps: dataset collection, code generation,
correctness verification, and performance estimation. Figure 1 illustrates our
framework. We begin by selecting appropriate programming questions and
canonical solutions from mainstream LCGMs benchmarks, and meticulously
preparing bilingual (English-Chinese) versions. We also construct an input
generator to automatically create test inputs of various sizes for each
question ( 1). Next, we input the bilingual programming questions into the
LCGM, considering different prompting methods ( 2). This allows us to
investigate the impact of prompting on the quality of generated code in both
languages. To assess the effectiveness(RQ1), we verify the correctness of the
generated code by comparing its output results with the canonical solution
using unit test cases from the input generator ( 3). To evaluate the
efficiency(RQ2), we collect “input size-execution time” pairs for functionally
correct code by running them with different input sizes and fitting them to
common time complexity expressions to estimate the average execution
performance ( 4). Besides, this framework also supports the investigation of
the impact of prompting by allowing customizing various prompts used by code
generation (RQ3). With the verified correctness and average execution
performance, we gain insights into the quality of LCGM-generated code.

####  III-B2 Dataset Collection

Existing benchmarks usually provide many meticulously crafted programming
tasks for evaluating the correctness of LCGM-generated code, which typically
include task descriptions, program entries, input/output examples, canonical
solutions, and a few unit test cases. We use three popular datasets, HumanEval
[8], MBPP [9], and LeetCode [20] as the initial question list, where requiring
LCGMs to fill in Python code segments based on specific task descriptions and
program entry points. To keep the quality and suitability of the test suite,
we then filter programming questions based on the following criteria: (i)
ensuring that the tasks are of moderate difficulty level to prevent LCGMs from
struggling to generate correct answers in both bilingual inputs, (ii) ensuring
that the tasks have multiple potential solutions to increase the likelihood of
observing performance differences, and (iii) ensuring that the tasks take a
variable with varying dimensionality as inputs, e.g., an array or list, where
the execution time of the solutions is strongly correlated with the
dimensionality of the input variable. After careful screening, we ultimately
select 52 programming questions, covering a diverse range of problem types,
such as dynamic programming, string processing, array enumeration, and more.
Although this dataset is only half the size of HumanEval, it is sufficient to
show the quality differences in LCGM-generated code for bilingual inputs, as
demonstrated in § IV.

The questions collected from HumanEval and MBPP are initially in English. Two
authors proficient in both languages meticulously translate them into Chinese.
For questions from LeetCode, which provides official bilingual versions, we
adopt them directly. We thoroughly check all bilingual question pairs to
ensure their consistency in task descriptions and adherence to the respective
language conventions. Additionally, for each programming question, we provide
an automated input sample generator, denoted as $G$. It takes an input size
parameter $n_{i}$ that determines the dimensionality of the generated
variables. These variables are tailored to act as unit test inputs for the
specific programming questions. By varying the size parameter, $G$ can
generate a wide range of inputs, from small and simple cases to large and
complicated ones.

Our test suite can be formally represented as $\mathcal{D}=\langle
Q,S,G\rangle$, where $Q$ is the programming question, $S$ is the canonical
solution, and $G$ is the test input generator. $Q$ consists of a task
description, an entry point function name, and reference input/output. For
each question $q\in Q$, we have two bilingual parallel versions, $q_{en}$ and
$q_{zh}$, presented in English and Chinese, respectively. $S$ is the canonical
solution of the given programming questions, which is used to produce ground
truth outputs for correctness verification. $G$ is a crucial component,
serving as the input generator. By leveraging $G$, we can automatically
generate a diverse set of test cases, saving time and effort compared to
manually generating inputs while maintaining high quality and coverage.

####  III-B3 Code Generation

While LCGMs demonstrate impressive code generation capabilities, accurately
extracting the desired code from their responses remains an engineering
challenge. Typically, many LCGMs struggle to precisely terminate their
generation process after completing the target function, often continuing to
generate extraneous code until reaching the maximum token limit. Directly
executing such code in a sandbox environment, e.g., provided by HumanEval,
frequently triggers parsing errors or interference from the additional
functions. To minimize this interference, we perform an Abstract Syntax Tree
(AST) based analysis on the generated results. By traversing the AST, we
identify and extract only the code segments corresponding to the target
function specified by the entry point. This targeted extraction ensures that
only the code essential to the function’s execution is retained, thereby
reducing the likelihood of errors that may arise from the inclusion of non-
essential or erroneous code segments. This preprocessing is crucial in
accurately assessing the correctness of the LCGMs’ outputs and ensuring that
the evaluation is not compromised by the LCGMs’ tendency to generate
superfluous code.

####  III-B4 Correctness Verification

As discussed earlier, only a few test cases in existing benchmarks are often
insufficient to guarantee thorough functionality correctness judgement, such a
concern is also highlighted in recent studies [11]. To ensure the adequacy of
unit testing, we automatically generate larger-size test cases for correctness
verification. As aforementioned, our collected dataset includes an input
generator, which can generate random inputs of a specified size. With the
canonical solution $S$, we can automatically synthesize reference outputs,
i.e., ground truth. Therefore, we can conduct more comprehensive unit testing.
We draw inspiration from HumanEval’s approach, which involves executing the
code in a sandbox environment and verifying correctness by comparing the
output of the LCGM-generated code with the synthesized ground truth. By
comparing the differences in functionality correctness of LCGM-generated code
across bilingual versions of the same programming question, we can conduct
further statistical analysis to support the investigation of RQ1.

Furthermore, we filter out the functionally correct code based on extensive
unit testing. Prioritizing correctness verification lays a solid foundation
for subsequent performance estimation, as discussing the performance merits of
code with inherent functional errors lacks practical significance.

####  III-B5 Performance Estimation

TABLE I: Common time complexity types. Type | Expression
---|---
Constant | $time=a(sec)$
Logarithmic | $time=a*log(n)+b(sec)$
Linear | $time=a*log(n)+b(sec)$
Linearithmic | $time=a*n*log(n)+b(sec)$
Quadratic | $time=a*n^{2}+b(sec)$
Cubic | $time=a*n^{3}+b(sec)$
Exponential | $time=a^{n}+b(sec)$

Accurately characterizing and evaluating the execution performance or time
complexity of arbitrary code is often challenging, especially when the LCGM-
generated code may contain external function calls. In our framework, we
estimate the execution performance by sampling a large number of inputs and
observing the changes in the execution time of the LCGM-generated code as the
input size grows. Recall that with the input generator $G$, we can sample test
inputs of various sizes for the under-measuring code by generating input
variables with different dimensions, thereby enabling us to profile actual
execution time for LCGM-generate code across different input sizes.

Initially, when constructing the input generator, we empirically set an
initial maximum input size $N_{max}$ for each programming question, taking
into account the complexity of the canonical solution. This value is set to be
sufficiently large, allowing us to divide it into many segments to sample
different input sizes. However, a problem arises as the LCGM-generate code
could have a higher complexity than the canonical solution, leading to
excessively long execution times and making the overall evaluation time
prohibitively expensive. To address this problem, we design an adaptive
mechanism for adjusting the maximum input size. Following HumanEval, we first
set a maximum time limit $T_{max}$ for code execution. Subsequently, we
examine the execution of each LCGM-generated code to determine if it triggers
a timeout when processing inputs of size $N_{max}$. In the event of a timeout,
we iteratively decrease $N_{max}$ by half until the execution satisfies the
$T_{max}$ constraint.

With the maximum input size $N_{max}$, we divide it into $seg$ equal parts and
obtain an increasing sequence of input sizes:

$N={n_{i}|n_{i}=i\cdot\frac{N_{max}}{seg},i=1,2,\dots,seg}$

For each input size $n_{i}$, we perform $k$ samplings, record the execution
time of each run, and take the average as the execution time estimate $t_{i}$
under that input size. This way, we obtain a set of “input size-execution
time” pairs:

$NT={(n_{i},t_{i})|n_{i}\in
N,t_{i}=\frac{1}{k}\sum_{j=1}^{k}t_{ij},i=1,2,\dots,seg}$

where $t_{ij}$ represents the execution time of the $j$-th sampling when the
input size is $n_{i}$.

We present a summary of seven prevalent time complexity types, as illustrated
in Table I. To determine the most matching time complexity type and
corresponding mathematical expression, we employ the least squares method to
perform a regression analysis [21] on the time series data $(NT)$. This
approach enables us to identify the closest matching complexity type.
Additionally, we assign a score ranging from 1 to 7 to each time complexity
type, following the order presented in Table I. Lower scores indicate higher
efficiency in terms of actual code execution. With this scoring mechanism, we
can quantitatively compare the efficiency of two code pieces in actual
sampling-based execution.

###  III-C Measurement Metrics

In this study, we focus on measuring the bilingual quality differences in
LCGM-generated code.

####  III-C1 Effectiveness

The effectiveness is measured through the correctness verification procedure.
The function $CorVer$ takes a problem $q$ as input and returns a boolean
value. It returns $True$ if the problem $q$ has been correctly solved by the
target LCGM, and $False$ otherwise. Thus, we define the following evaluation
metrics:

$\bullet$ $\mathbf{CR_{en}}$ (Correctness Rate in English): This metric
represents the proportion of programming questions in English for which the
target LCGM can generate correct code. The specific calculation can be
formalized as:

$\text{CR}_{en}=\frac{|{q\in Q:\text{CorVer}(q_{en})=True}|}{|Q|}$

$\bullet$ $\mathbf{CR_{zh}}$ (Correctness Rate in Chinese): Similarly, this
metric represents the proportion of programming questions in Chinese for which
the target LCGM can generate correct code. It is calculated as:

$\text{CR}_{zh}=\frac{|{q\in Q:\text{CorVer}(q_{zh})=True}|}{|Q|}$

$\bullet$ $\mathbf{CR_{bi}}$ (Correctness Rate in Bilingual): This metric
represents the proportion of programming questions for which the target LCGM
can generate correct code in both languages. It is calculated as:

$\text{CR}_{bi}=\frac{|{q\in
Q:\text{CorVer}(q_{en})\wedge\text{CorVer}(q_{zh})=\text{True}}|}{|Q|}$

$\bullet$ $\mathbf{CDR}$ (Correctness Difference Rate): This metric represents
the proportion of programming questions for which the target LCGM can generate
correct code only for one specific language. It is calculated as:

$\text{CDR}=\frac{|{q\in
Q:\text{CorVer}(q_{en})\oplus\text{CorVer}(q_{zh})=\text{Ture}}|}{|Q|}$

where $\oplus$ is the XOR operator.

####  III-C2 Efficiency

As aforementioned, we evaluate the efficiency differences on the programming
questions subset where LCGMs can generate correct solutions for both bilingual
inputs. Firstly, we define the subset of correctly solved programming
questions as $Q^{\prime}\subseteq Q$, which satisfies

$\forall q\in
Q^{\prime},\text{CorVer}(q_{en})\wedge\text{CorVer}(q_{zh})=\text{True}$

Besides, we introduce $SComplex$ as a function that inputs a programming
question $q$ and returns the score of the estimated complexity for the LCGM-
generated code. The score is a value from 1 to 7, with lower scores indicating
higher efficiency in terms of actual code execution. The measurement metrics
for efficiency are as follows:

$\bullet$ $\mathbf{PAR_{en}}$ (Performance Advantage Rate in English): This
metric represents the proportion of programming questions in $Q^{\prime}$ for
which the LCGMs generate code with better estimated time complexity (lower
$SComplex$ value) in English compared to Chinese. It is calculated as:

$\text{PAR}_{en}=\frac{|{q\in
Q^{\prime}:\text{SComplex}(q_{en})<\text{SComplex}(q_{zh})}|}{|Q^{\prime}|}$

$\bullet$ $\mathbf{PAR_{zh}}$ (Performance Advantage Rate in Chinese): This
metric represents the proportion of programming questions in $Q^{\prime}$ for
which the LCGMs generate code with better estimated time complexity (lower
$SComplex$ score) in Chinese compared to English. It is calculated as:

$\text{PAR}_{zh}=\frac{|{q\in
Q^{\prime}:\text{SComplex}(q_{zh})<\text{SComplex}(q_{en})}|}{|Q^{\prime}|}$

$\bullet$ $\mathbf{PDR}$ (Performance Difference Rate): This metric represents
the proportion of programming questions in $Q^{\prime}$ for which the LCGMs
generate code with significantly different estimated time complexities
(different $SComplex$ scores) between English and Chinese. It is calculated
as:

$\text{PDR}=\frac{|{q\in
Q^{\prime}:\text{SComplex}(q_{en})\neq\text{SComplex}(q_{zh})}|}{|Q^{\prime}|}$

###  III-D Target LCGMs

We select the following six LCGMs in our study, considering: 1) English and
Chinese are well-supported; 2) they top mainstream evaluation benchmarks and
leaderboards (for Python code generation); 3) they are representative and
diverse in architecture, i.e., originating from different model families; and
4) due to our GPU constraints, the open-source models do not exceed 15B.

  * •

CodeGen2.5 [22]: Building upon the popular CodeGen2 family of models,
CodeGen2.5 with 7B parameters is on par with 15B parameter LCGMs and has
undergone additional training on Python. It is developed by Salesforce.

  * •

StarCoder [23]: Trained on 80+ programming languages, StarCoder supports a
novel combination of capabilities and architectural features unavailable in
other open LCGMs. It has 15B parameters.

  * •

CodeGeeX [24]: CodeGeeX is a multilingual code generation model implemented
based on the ChatGLM2 architecture. It has 6B parameters and particularly
supports bilingual input in English and Chinese.

  * •

CodeLlama [25]: Developed by Meta, CodeLlama is a foundational model with
various parameter size versions. We select the 7B version for our study.

  * •

DeepSeek-Coder [26]: This model comprises a series of code language models
trained from scratch on both 87% code and 13% natural language in English and
Chinese. It supports project-level code completion and infilling. We choose
the 6.7B version here.

  * •

GPT-3.5-Turbo [27]: Developed by OpenAI and empowering ChatGPT, GPT-3.5-Turbo,
although not specifically trained for code generation tasks, can still assist
in programming tasks due to its powerful natural language understanding and
generation capabilities.

###  III-E Parameter Configuration

In our experiments, the maximum time limit $T_{max}$ is 6 seconds, the number
of sampling $k$ is 100, and the number of segments $seg$ is 50. Another
important parameter when utilizing LCGMs is the temperature ($t$), which
typically ranges from 0 to 1. Lower temperature values result in more
deterministic and concentrated outputs, while higher values lead to more
diverse and exploratory generations. We select two common temperature values,
0.2 and 0.8, to better observe the impact of this parameter on the quality of
the generated code [28].

##  IV RESULTS AND ANALYSIS

###  IV-A RQ1: Study on Effectiveness.

Design. In this RQ, we investigate the capability and differences of LCGMs in
generating correct code for bilingual programming questions. Considering the
stochastic nature of the LCGMs’ generation process, we independently execute
the generation process 20 times for each LCGM on each programming question. If
at least one piece of generated code passes the correctness verification, we
consider the LCGM capable of correctly answering the question. That is, we use
the pass@20 metric to evaluate the correctness of the generated code. The
specific evaluation metrics are discussed in § III-C.

TABLE II: Effectiveness Measurement Results. | t = 0.2 | t = 0.8
---|---|---
LCGM | $CR_{en}$ | $CR_{zh}$ | $CR_{bi}$ | $CDR$ | $CR_{en}$ | $CR_{zh}$ | $CR_{bi}$ | $CDR$
CodeGeeX | 0.60 | 0.54 | 0.54 | 0.06 | 0.62 | 0.63 | 0.58 | 0.10
CodeGen2.5 | 0.52 | 0.52 | 0.46 | 0.12 | 0.46 | 0.48 | 0.40 | 0.13
CodeLlama | 0.50 | 0.54 | 0.46 | 0.12 | 0.52 | 0.50 | 0.46 | 0.10
DeepSeek-Coder | 0.63 | 0.58 | 0.54 | 0.13 | 0.71 | 0.65 | 0.63 | 0.10
StarCoder | 0.48 | 0.52 | 0.44 | 0.12 | 0.58 | 0.56 | 0.52 | 0.10
GPT-3.5-Turbo | 0.58 | 0.64 | 0.58 | 0.08 | 0.7 | 0.66 | 0.67 | 0.04
Average | 0.55 | 0.56 | 0.50 | 0.11 | 0.60 | 0.58 | 0.54 | 0.10

Results. Table II presents the results of our investigation into the bilingual
code generation abilities of LCGMs. The table is divided into two sections,
representing different generation temperatures ($t=0.2$ and $t=0.8$) used
during code generation. Each row corresponds to a specific LCGM, and the
columns display various metrics.

Columns $\text{CR}_{en}$ and $\text{CR}_{zh}$ demonstrate the proportions of
correctly solved programming questions by LCGMs for English and Chinese
inputs, respectively. We observe an overall improvement in the ability to
generate correct code as the temperature increases from 0.2 to 0.8 for most
LCGMs (except CodeGen2.5). We use green and red to highlight the LCGMs with
the highest code correctness. Green indicates that the model has higher
correctness for the questions in English, while red indicates that the model
has higher correctness for Chinese questions. Obviously, the advantage of
generating correct code is skewed towards English, i.e., when $t=0.2$, only
two models gain an advantage in English, but when $t=0.8$, four models gain an
advantage in English.

Finding 1: When the temperature is lower, most LCGMs exhibit higher code
generation effectiveness in Chinese programming questions, while at higher
temperatures, most LCGMs demonstrate higher effectiveness in English
programming questions.

The column $CDR$ demonstrates the disparities in LCGMs’ ability to generate
correct code on two language versions of the same programming question.
Particularly, it directly illustrates the extent to which LCGMs can generate
correct code for one language while producing incorrect code for another. The
results indicate that, across both temperatures, these LCGMs exhibit
inconsistencies in their ability to generate correct code on over 10% of the
programming questions, on average.

Finding 2: LCGMs show inconsistency in generating correct codes for more than
10% bilingual programming questions.

In our experiments, GPT-3.5-Turbo and CodeGeeX emerge as the top performers
among the six LCGMs we evaluate, demonstrating the highest pass rates for code
generation in both English and Chinese individually, as well as achieving the
lowest discrepancy in bilingual effectiveness. It is worth noting that, when
$t=0.2$, CodeGeeX outperforms GPT-3.5-Turbo in both the correctness and
consistency of generating codes for Chinese programming questions. This could
be attributed to the fact that CodeGeeX is developed by a Chinese company and
has undergone more training in Chinese corpora.

###  IV-B RQ2:Study on Efficiency.

TABLE III: Efficiency Measurement Results. | t = 0.2 | t = 0.8
---|---|---
LCGM | $PAR_{en}$ | $PAR_{zh}$ | $PDR$ | $PAR_{en}$ | $PAR_{zh}$ | $PDR$
CodeGeeX | 0.17 | 0.17 | 0.34 | 0.34 | 0.2 | 0.54
CodeGen2.5 | 0.2 | 0.23 | 0.43 | 0.11 | 0.25 | 0.36
CodeLlama | 0.13 | 0.27 | 0.4 | 0.21 | 0.14 | 0.34
DeepSeek-Coder | 0.2 | 0.23 | 0.43 | 0.24 | 0.11 | 0.34
StarCoder | 0.17 | 0.24 | 0.41 | 0.26 | 0.13 | 0.39
GPT-3.5-Turbo | 0.06 | 0.29 | 0.35 | 0.22 | 0.19 | 0.41
Average | 0.16 | 0.24 | 0.39 | 0.23 | 0.17 | 0.40

Design. In this RQ, we focus on the programming questions that each LCGM can
correctly solve in both English and Chinese versions. For each question, we
collect the corresponding LCGM-generated correct code in both languages. It is
important to note that a single programming question may have multiple correct
generated solutions. To represent the performance level of the code generated
by LCGMs for each programming question in the respective language version, we
select the best-performing code from the set of correct solutions for each
language. By choosing the most efficient code generated for each language
version of each programming question, we can effectively compare which
language enables the LCGM to generate the most optimized code. This approach
allows us to assess the impact of language on the performance of the generated
code and determine if there are any significant differences between the
English and Chinese versions.

Figure 2: Performance Evaluation Results for Each LCGM.

Results. Figure 2 shows the selected set of programming questions for each
LCGM and the number of questions for which each language generates higher-
performance code. The overlapping part in the middle represents questions
where the generated code performance is at the same level for both languages.
It is evident that for each LCGM, there is a significant proportion of
questions where the performance of generated code in one language is superior
to that in another language. Table III further presents the specific
evaluation metrics. $PDR$ provides a detailed measurement of this difference.
Experimental results show that the average $PDR$ is over 39% during various
temperatures, which indicates that the efficiency of code generated by LCGMs
may have greater inconsistency than effectiveness, warranting attention.
Finding 3: LCGMs exhibit differences in the efficiency of code generation for
bilingual programming questions: for over 39% of the correctly solved
questions, the efficiency level of the generated code varies between the two
languages.

$PAR_{en}$ and $PAR_{zh}$ represent the proportion of programming questions
where these LCGMs gain an advantage in each language, as highlighted. It can
be observed that when $t=0.2$, five LCGMs generate code with better
performance for Chinese input; when $t=0.8$, five LCGMs generate code with
better performance for English inputs. In other words, as $t$ increases, the
advantage shifts from Chinese to English, which is consistent with our
observations in RQ1. Figure 3 further illustrates the distribution of the
performance advantages of the generated code for each LCGM. The horizontal
axis represents the number of LCGMs that generate more efficient code in the
Chinese version of the question, while the vertical axis represents the number
of LCGMs that generate more efficient code in the English version of the
question. Experimental results show that when $t=0.2$, the darker shaded areas
in the graph indicate a stronger preference for Chinese.

Finding 4: With increasing temperatures, the advantage of generating more
efficient code shifts from Chinese to English inputs.

However, we observe that CodeGeeX and GPT-3.5-Turbo exhibit the lowest
efficiency differences when $t=0.2$, and show the highest differences as $t$
increase to 0.8. In contrast, the efficiency differences for the other four
LCGMs decrease. Despite GPT-3.5-Turbo’s superior performance in code
generation effectiveness and the lowest bilingual discrepancy, its efficiency
in code generation is not as commendable. This suggests that while correctness
is a crucial aspect, the efficiency and other quality attributes of code
generation by these applications warrant greater attention.

Finding 5: While an LCGM’s consistency in generating effective code for
bilingual programming questions is notable, it does not necessarily ensure a
high level of consistency in the efficiency of the generated code.

(a) t = 0.2 (b) t = 0.8

Figure 3: The distribution of bilingual performance advantages for six LCGMs.

###  IV-C RQ3:The Impact of Prompting

TABLE IV: Different Prompting Methods. Prompting Type | Detailed Prompts
---|---
Directly Input(DI) | {question}
Basic Prompt(BP) | # Complete the following Python code without any tests or explanation\n {question}
Quality-aware Prompt(QA) | # Complete the following Python code by  a concise and efficient implementation.\n {question}
CoT Prompt(CoT) | # Complete the following Python code without any tests or explanation,  let’s think step by step\n {question}
Quality-aware CoT Prompt(QA+CoT) | # Complete the following Python code by  a concise and efficient implementation,  let’s think step by step\n {question}

(a) Impact on effectiveness

(b) Impact on efficiency

Figure 4: Impact of Prompting on Bilingual Code Generation.

Figure 5: Examples of Bilingual Quality Differences

Design. In this RQ, we investigate the impact of different prompts on the
quality of LCGM-generate code in bilingual programming questions. As shown in
Table IV, in previous experiments, we directly fed those questions into LCGMs.
Now, we introduce different prompting paradigms to provide clearer and more
specific instructions, aiming to improve the quality of code generation.
First, we experiment with a basic prompt, which is commonly utilized by some
instruction-tuned LCGMs to enhance generation capability [23]. Building upon
this, we further attempt to introduce quality-aware information into the
prompt, explicitly instructing the model to generate “concise and efficient”
code, to improve the correctness and efficiency of the generated code.
Moreover, the Chain-of-Thought (CoT) [29] prompting method has recently been
widely proven to significantly improve the quality of generation results. A
typical example is adding statements like “Let’s think step by step” in the
tail of the prompt, guiding the model to perform step-by-step thinking and
reasoning. Therefore, we also incorporate the CoT prompting method in our
experiments. Finally, to comprehensively consider both code quality and the
logical coherence of the generation process, we experiment with the
combination of quality-aware information and the CoT prompting method.

Results. Figure 4 presents the impact of different prompting methods on the
effectiveness and efficiency of LCGM-generated code based on our proposed
metrics. Firstly, we observe that different prompting methods have varying
degrees of impact on each metric, and this impact varies across different
LCGMs. From an effectiveness perspective, using these diverse prompts may lead
to a decrease in the ability of LCGMs to generate correct code in both Chinese
and English languages compared to directly inputting programming questions,
especially for CodeGeeX and StarCoder. From an efficiency standpoint, prompts
also have varying degrees of influence. For example, at $t=0.2$, it can be
observed that diverse prompts significantly bias code generation towards
English in models like CodeGen2.5, CodeLlama, DeepSeek-Coder, and StarCoder.
Moreover, the impact of increasing temperature $t$ on the effectiveness and
efficiency of code generation varies across different prompts on LCGMs. For
example, as $t$ changes, the code generation effectiveness of prompts “QA” and
“QA+CoT” fluctuates significantly, while prompts ’DI’ and ’BP’ show minimal
sensitivity to changes in $t$. We can also observe that Finding 1 might not
hold for other prompts. For example, increasing the temperature when using the
prompt “BP” tends to result in higher generation effectiveness for Chinese
questions rather than higher effectiveness for English. Overall, different
prompting strategies affect the quality differences in code generation by
LCGMs in bilingual contexts, and this difference is often influenced by
temperature, with varying effects on different LCGMs.  Finding 6: Prompting
differentially affects the effectiveness and efficiency of bilingual code
generation by LCGMs, with varying impacts across LCGMs and temperature
settings.

However, an intriguing point is that CoT shows significant potential in
eliminating the efficiency discrepancies in code generation by LCGMs on
bilingual dataset. It can be observed that, for CodeGen2.5 and StarCoder,
using CoT prompts as input reduces the efficiency difference metric to 0
(applicable to both $t=0.2$ and $t=0.8$). However, this effect is not observed
in other LCGMs. Nonetheless, this highlights the potential importance of
prompting methods in reducing bias in LCGMs’ bilingual capabilities, recalling
a very promising direction for future research.  Finding 7: The CoT prompting
shows significant potential in enhancing the efficiency of bilingual code
generation.

###  IV-D Case Study

Figure 5 illustrates a pair of practical examples for GPT-3.5-Turbo on
bilingual programming tasks. In the first example in Figure 5(a),
GPT-3.5-Turbo generates the correct code for the Chinese input but fails to do
so for the English input. Specifically, the solution “return total//n” (marked
in red) does not align with the task description of returning the result ”in a
rounded integer.” This further emphasizes the need for an interpretability-
driven comprehensive study regarding which key entities or information from
input problems are captured by LCGMs for different input languages.

Moreover, we find that even when LCGMs generate functionally correct code for
both English and Chinese inputs, there can still be significant differences in
the efficiency of the generated code. As shown in the second example in Figure
5(b), the code generated for the Chinese input is less efficient than that for
the English input. The average execution time complexity of the code generated
for the Chinese input is $O(n^{2})$ due to the presence of nested loops. In
contrast, the code generated for the English input has an average time
complexity of only $O(n\ast\log(n))$, thanks to the more concise and efficient
library call of “sort()”. This may inspire us that the subjective enthusiasm
of LCGMs for library API calls may affect the overall efficiency of their
generated code. It is worth noting that to better control variables and ensure
that the Chinese-English differences in the input exist only in the
programming question descriptions, we used the same English prompt for code
generation in both languages.

###  IV-E Discussions and Suggections

Comprehensive Multilingual Code Generation Quality Assessment: Based on the
results from RQ1 and RQ2, our study has revealed significant disparities in
the effectiveness and efficiency of LCGMs in generating code for bilingual
(Chinese and English) programming. It is reasonable to infer that such
differences are not confined to these two languages but may represent a
broader phenomenon. Emerging research has increasingly investigated the
multilingual understanding and generation capabilities of Large Language
Models [30], and similar scrutiny must be directed towards LCGMs. Moreover,
effectiveness and efficiency are only two aspects of code quality, other
important attributes such as security, readability, and maintainability also
deserve consideration. Therefore, expanding the scope of quality assessment to
include these additional attributes is a valuable direction for future
research.

Developing Better Prompting Methods. According to RQ3, the impact of different
prompting strategies on the quality of LCGM-generated code is substantial.
Prompting techniques are pivotal in the era of LLMs, and well-crafted prompts
can significantly enhance the quality of generation [31]. Investigating how to
construct superior prompts that enable LCGMs to consistently produce high-
quality code across different languages is of paramount importance.

Adaptive Temperature Tuning. It has been observed that as the temperature
parameter increases, LCGMs exhibit varying performance across languages, i.e.,
most LCGMs tend to prefer generating code of higher quality in English.
Identifying the optimal temperature settings that allow LCGMs to perform best
for each language could be a key factor in improving multilingual code
generation.

In summary, one important message from this study is that the software
engineering community should establish benchmarks and frameworks for
evaluating the multilingual code generation capabilities of LCGMs, considering
different code quality assessment metrics, input languages, prompting methods,
and more. As an initial step, we have released our framework [12] and strongly
encourage our readers to contribute more datasets and code quality
measurements.

###  IV-F Threats to Validity

Limited Dataset: The dataset utilized in this study, despite careful
designing, is limited to 52 bilingual programming questions, which may not
sufficiently cover the diverse range of tasks LCGMs encounter in practical
applications. Furthermore, the dataset only includes English and Chinese,
potentially failing to capture the performance variations of LCGMs across a
wider array of languages. This limitation stems from the requirement of strong
language proficiency to ensure the quality of multilingual parallel data.

Limited LCGMs: The selected six LCGMs represent the current technology but
don’t cover all models or training methods. However, the selected LCGMs serve
as the foundation for many advanced LCGMs. Our evaluation method is universal
and extendable to a broader range of models.

Evaluation Metrics: The metrics employed in this study primarily focus on code
correctness and efficiency, potentially neglecting other crucial aspects of
code quality, such as readability, maintainability, and adherence to coding
standards. The absence of these factors in the evaluation process may restrict
the comprehensive understanding of the LCGM’ performance.

To mitigate the threats and foster further research, we make our collected
dataset and evaluation framework publicly available [12]. This initiative aims
to empower researchers to expand the study by incorporating diverse
programming tasks, languages, and quality assessment metrics, as well as to
include a broader range of LCGMs in the comparison.

##  V RELATED WORK

###  V-A Large Code Generation Models

LLMs have achieved impressive results in code processing, with models ranging
from general-purpose to specialized architectures. General-purpose LLMs, such
as LLaMA [32], Llama2 [33], LaMDA [34], and PaLM [35], can generate code
thanks to pre-training on text data that includes code. For example, Llama2
contains 328GB of code from GitHub. Even smaller models like Phi-1.5 (1.3
billion parameters) can demonstrate coding proficiency [36].

Language models with additional pre-training on code have achieved new
benchmarks. Codex [8] fine-tuned GPT-3 on 100B code tokens, and PaLM-Coder
[35] extended pre-training with 7.8 billion tokens of code. CodeLlama [25]
enhanced Llama2 by pre-training it on 500 billion code tokens, and CodeFuse-
CodeLlama [37], a fine-tuned version of CodeLlama, surpasses even GPT-4’s
performance.

Specialized architecture models use different architectures to generate code.
CodeT5+ [38] uses an encoder-decoder architecture with contrastive objectives,
while CodeFusion [39] introduces diffusion models to code modeling. These
models can outperform other LLMs on code synthesis datasets.

###  V-B Testing and Evaluation for Code Generation Model

Over the years, the software engineering community has proposed various
evaluation tasks to assess code models, which have been consolidated into
benchmarks such as CodeXGLUE [40] and CodeBLEU [41]. However, with the
introduction of HumanEval [8] and the increasing capability of code generation
models, the focus has shifted towards functional correctness as a more
appropriate metric for evaluation [42, 43, 9]. This has led to the development
of metrics such as pass@k [8] and passn@k [44], which assess the model’s
ability to generate code that passes unit tests. Several benchmarks have been
created to evaluate code synthesis, including HUMANEVAL and MBPP [9] for
Python, as well as benchmarks targeting other programming languages such as
SQL (Spider[45]), C++, JavaScript, and Go (HUMANEVAL-X [24]), C++ and Java
(CodeContests [44]), and a multilingual extension of HUMANEVAL and MBPP
(MultiPL-E [46]). In addition to function-level evaluation, CrossCodeEval [47]
and CoderEval [48] have introduced benchmarks for cross-file and project-level
code completion. Besides correctness evaluation, some works are dedicated to
testing and assessing the robustness of LCGMs [49, 50, 51, 52, 53].

##  VI CONCLUSION

In this paper, we conduct a pioneering empirical study on the bilingual code
generation quality of Large Code Generation Models (LCGMs). While LCGMs have
revolutionized software engineering tasks such as code generation, their
performance on non-English languages, particularly Chinese, has not been
systematically investigated. We collect a dataset containing 52 bilingual
parallel programming tasks and evaluate the code generated by six state-of-
the-art LCGMs in terms of effectiveness and efficiency. Our findings reveal
that there exist quality differences between LCGM-generated code for English
and Chinese programming questions. This work underscores the urgency for
further research and development to ensure consistent code generation across
languages.

## References

  * [1] P. Yin and G. Neubig, “A syntactic neural model for general-purpose code generation,” _arXiv preprint arXiv:1704.01696_ , 2017.
  * [2] S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg _et al._ , “Sparks of artificial general intelligence: Early experiments with gpt-4,” _arXiv preprint arXiv:2303.12712_ , 2023.
  * [3] Y. Wan, Z. Zhao, M. Yang, G. Xu, H. Ying, J. Wu, and P. S. Yu, “Improving automatic source code summarization via deep reinforcement learning,” in _Proceedings of the 33rd ACM/IEEE international conference on automated software engineering_ , 2018, pp. 397–407.
  * [4] Y. Liang and K. Zhu, “Automatic generation of text descriptive comments for code blocks,” in _Proceedings of the AAAI conference on artificial intelligence_ , vol. 32, no. 1, 2018.
  * [5] S. Kotsiantis, V. Verykios, and M. Tzagarakis, “Ai-assisted programming tasks using code embeddings and transformers,” _Electronics_ , vol. 13, no. 4, p. 767, 2024.
  * [6] R. Pudari and N. A. Ernst, “From copilot to pilot: Towards ai supported software development,” _arXiv preprint arXiv:2303.04142_ , 2023.
  * [7] C. Qian, X. Cong, C. Yang, W. Chen, Y. Su, J. Xu, Z. Liu, and M. Sun, “Communicative agents for software development,” _arXiv preprint arXiv:2307.07924_ , 2023.
  * [8] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman _et al._ , “Evaluating large language models trained on code,” _arXiv preprint arXiv:2107.03374_ , 2021.
  * [9] J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le _et al._ , “Program synthesis with large language models,” _arXiv preprint arXiv:2108.07732_ , 2021.
  * [10] F. Tomassetti and M. Torchiano, “An empirical assessment of polyglot-ism in github,” in _Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering_ , 2014, pp. 1–4.
  * [11] J. Liu, C. S. Xia, Y. Wang, and L. Zhang, “Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation,” _Advances in Neural Information Processing Systems_ , vol. 36, 2024.
  * [12] “Our code,” https://github.com/lenijwp/BilingualCodeEval, 2024.
  * [13] G. Sridhara, S. Mazumdar _et al._ , “Chatgpt: A study on its utility for ubiquitous software engineering tasks,” _arXiv preprint arXiv:2305.16837_ , 2023.
  * [14] A. Mastropaolo, S. Scalabrino, N. Cooper, D. N. Palacio, D. Poshyvanyk, R. Oliveto, and G. Bavota, “Studying the usage of text-to-text transfer transformer to support code-related tasks,” in _2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)_. IEEE, 2021, pp. 336–347.
  * [15] M. Allamanis, E. T. Barr, P. Devanbu, and C. Sutton, “A survey of machine learning for big code and naturalness,” _ACM Computing Surveys (CSUR)_ , vol. 51, no. 4, pp. 1–37, 2018.
  * [16] J. F. Pane, B. A. Myers _et al._ , “Studying the language and structure in non-programmers’ solutions to programming problems,” _International Journal of Human-Computer Studies_ , vol. 54, no. 2, pp. 237–264, 2001.
  * [17] T. t. p. g. i. S. a. n. l. f. t. i. g. b. c. o. c. D. to varying update cycles and S. C. D. M. u.-t.-D. D. T. R. in the Text, “Topic: Software industry in China,” https://www.statista.com/topics/11244/software-industry-in-china/.
  * [18] L. Fu, H. Chai, S. Luo, K. Du, W. Zhang, L. Fan, J. Lei, R. Rui, J. Lin, Y. Fang _et al._ , “Codeapex: A bilingual programming evaluation benchmark for large language models,” _arXiv preprint arXiv:2309.01940_ , 2023.
  * [19] T. McEnery and R. Xiao, _Corpus-based contrastive studies of English and Chinese_. Routledge, 2010.
  * [20] “Leetcode,” https://leetcode.com/problemset/, 2024.
  * [21] N. R. Draper and H. Smith, _Applied regression analysis_. John Wiley & Sons, 1998, vol. 326.
  * [22] E. Nijkamp, H. Hayashi, C. Xiong, S. Savarese, and Y. Zhou, “Codegen2: Lessons for training llms on programming and natural languages,” _arXiv preprint_ , 2023.
  * [23] R. Li, L. B. Allal, Y. Zi, N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. Li, J. Chim _et al._ , “Starcoder: may the source be with you!” _arXiv preprint arXiv:2305.06161_ , 2023.
  * [24] Q. Zheng, X. Xia, X. Zou, Y. Dong, S. Wang, Y. Xue, L. Shen, Z. Wang, A. Wang, Y. Li _et al._ , “Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x,” in _Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_ , 2023, pp. 5673–5684.
  * [25] B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y. Adi, J. Liu, T. Remez, J. Rapin _et al._ , “Code llama: Open foundation models for code,” _arXiv preprint arXiv:2308.12950_ , 2023.
  * [26] D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y. Wu, Y. Li _et al._ , “Deepseek-coder: When the large language model meets programming–the rise of code intelligence,” _arXiv preprint arXiv:2401.14196_ , 2024.
  * [27] “Openai api,” https://openai.com/product, 2024.
  * [28] “Openai api refernce,” https://platform.openai.com/docs/api-reference/completions/create, 2024.
  * [29] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou _et al._ , “Chain-of-thought prompting elicits reasoning in large language models,” _Advances in neural information processing systems_ , vol. 35, pp. 24 824–24 837, 2022.
  * [30] W. Zhang, M. Aljunied, C. Gao, Y. K. Chia, and L. Bing, “M3exam: A multilingual, multimodal, multilevel benchmark for examining large language models,” _Advances in Neural Information Processing Systems_ , vol. 36, 2024.
  * [31] P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig, “Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing,” _ACM Computing Surveys_ , vol. 55, no. 9, pp. 1–35, 2023.
  * [32] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar _et al._ , “Llama: Open and efficient foundation language models,” _arXiv preprint arXiv:2302.13971_ , 2023.
  * [33] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale _et al._ , “Llama 2: Open foundation and fine-tuned chat models,” _arXiv preprint arXiv:2307.09288_ , 2023.
  * [34] R. Thoppilan, D. De Freitas, J. Hall, N. Shazeer, A. Kulshreshtha, H.-T. Cheng, A. Jin, T. Bos, L. Baker, Y. Du _et al._ , “Lamda: Language models for dialog applications,” _arXiv preprint arXiv:2201.08239_ , 2022.
  * [35] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann _et al._ , “Palm: Scaling language modeling with pathways,” _Journal of Machine Learning Research_ , vol. 24, no. 240, pp. 1–113, 2023.
  * [36] Y. Li, S. Bubeck, R. Eldan, A. Del Giorno, S. Gunasekar, and Y. T. Lee, “Textbooks are all you need ii: phi-1.5 technical report,” _arXiv preprint arXiv:2309.05463_ , 2023.
  * [37] B. Liu, C. Chen, C. Liao, Z. Gong, H. Wang, Z. Lei, M. Liang, D. Chen, M. Shen, H. Zhou _et al._ , “Mftcoder: Boosting code llms with multitask fine-tuning,” _arXiv preprint arXiv:2311.02303_ , 2023.
  * [38] Y. Wang, W. Wang, S. Joty, and S. C. Hoi, “Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,” _arXiv preprint arXiv:2109.00859_ , 2021.
  * [39] M. Singh, J. Cambronero, S. Gulwani, V. Le, C. Negreanu, and G. Verbruggen, “Codefusion: A pre-trained diffusion model for code generation,” _arXiv preprint arXiv:2310.17680_ , 2023.
  * [40] S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. Clement, D. Drain, D. Jiang, D. Tang _et al._ , “Codexglue: A machine learning benchmark dataset for code understanding and generation,” _arXiv preprint arXiv:2102.04664_ , 2021.
  * [41] S. Ren, D. Guo, S. Lu, L. Zhou, S. Liu, D. Tang, N. Sundaresan, M. Zhou, A. Blanco, and S. Ma, “Codebleu: a method for automatic evaluation of code synthesis,” _arXiv preprint arXiv:2009.10297_ , 2020.
  * [42] B. Roziere, M.-A. Lachaux, L. Chanussot, and G. Lample, “Unsupervised translation of programming languages,” _Advances in neural information processing systems_ , vol. 33, pp. 20 601–20 611, 2020.
  * [43] D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. Song _et al._ , “Measuring coding challenge competence with apps,” _arXiv preprint arXiv:2105.09938_ , 2021.
  * [44] Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. Dal Lago _et al._ , “Competition-level code generation with alphacode,” _Science_ , vol. 378, no. 6624, pp. 1092–1097, 2022.
  * [45] T. Yu, R. Zhang, K. Yang, M. Yasunaga, D. Wang, Z. Li, J. Ma, I. Li, Q. Yao, S. Roman _et al._ , “Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task,” _arXiv preprint arXiv:1809.08887_ , 2018.
  * [46] F. Cassano, J. Gouwar, D. Nguyen, S. Nguyen, L. Phipps-Costin, D. Pinckney, M.-H. Yee, Y. Zi, C. J. Anderson, M. Q. Feldman _et al._ , “Multipl-e: a scalable and polyglot approach to benchmarking neural code generation,” _IEEE Transactions on Software Engineering_ , 2023.
  * [47] Y. Ding, Z. Wang, W. Ahmad, H. Ding, M. Tan, N. Jain, M. K. Ramanathan, R. Nallapati, P. Bhatia, D. Roth _et al._ , “Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion,” _Advances in Neural Information Processing Systems_ , vol. 36, 2024.
  * [48] H. Yu, B. Shen, D. Ran, J. Zhang, Q. Zhang, Y. Ma, G. Liang, Y. Li, Q. Wang, and T. Xie, “Codereval: A benchmark of pragmatic code generation with generative pre-trained models,” in _Proceedings of the 46th IEEE/ACM International Conference on Software Engineering_ , 2024, pp. 1–12.
  * [49] A. Mastropaolo, L. Pascarella, E. Guglielmi, M. Ciniselli, S. Scalabrino, R. Oliveto, and G. Bavota, “On the robustness of code generation techniques: An empirical study on github copilot,” in _2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)_. IEEE, 2023, pp. 2149–2160.
  * [50] G. Yang, Y. Zhou, W. Yang, T. Yue, X. Chen, and T. Chen, “How important are good method names in neural code generation? a model robustness perspective,” _ACM Transactions on Software Engineering and Methodology_ , 2023.
  * [51] S. Wang, Z. Li, H. Qian, C. Yang, Z. Wang, M. Shang, V. Kumar, S. Tan, B. Ray, P. Bhatia _et al._ , “Recode: Robustness evaluation of code generation models,” _arXiv preprint arXiv:2212.10264_ , 2022.
  * [52] Z. Li, C. Wang, Z. Liu, H. Wang, D. Chen, S. Wang, and C. Gao, “Cctest: Testing and repairing code completion systems,” in _2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)_. IEEE, 2023, pp. 1238–1250.
  * [53] Z. Yang, Z. Sun, T. Z. Yue, P. Devanbu, and D. Lo, “Robustness, security, privacy, explainability, efficiency, and usability of large language models for code,” 2024.


My results support Mejía-Restrepo et al. (2018b) results and confirm that the
virial factor, $f$, depends on FWHM. In this paper,
$f\propto\mathit{FWHM}(H{\small\beta})^{-1.34\pm 0.37}$. The BLR is a non-
spherical region. The BLR region has the disk-like geometry.

I suggest that some WLQs are normal quasars in a reactivation stage. However,
the shielding gas is another plausible explanation for the WLQs. Apart from
the previous mentioned finding that WLQs are QSOs in a reactivation phase,
along with the development of the BLR, the shielding gas seems like a good
explanation of the nature of WLQs.

The abnormal absorption of SDSS J110511.15+530806.5 quasar is explained in the
two Gaussian functions description. The fitting procedure of the model
redden*AGNSED*gabs*gabs concluded with the global parameters:

  * •

$M_{\mathrm{BH}}$= $(3.522\pm 0.010)\times 10^{9}$,

  * •

$\dot{m}$= $0.274\pm 0.001$,

  * •

cos $i$= $0.841\pm 0.003$,

  * •

$a_{*}$ = $0.36\pm 0.02$

The corona and the warm skin investigation stipulated in:

  * •

$kT_{e,hot}$ = 100 keV, $\Gamma_{hot}$ = $1.300^{+1.545}_{-1.300}$, $R_{hot}$
= $58.54\pm 3.36$ $r_{g}$,

  * •

$kT_{e,warm}$ = $0.356^{+4.876}_{-0.350}$ keV, $\Gamma_{warm}$ = $2.722\pm
0.153$, $R_{warm}$ = $430.0^{+50.0}_{-11.0}$ $r_{g}$.

In summary:

  1. 1.

The investigation of the Weak Emission-line Quasars.

     * •

The continuum fit method is an unbiased way to describe the global parameters,

     * •

The flatness of the BLR in weak emission-line quasars plays a more important
role than the inclination,

     * •

The correction of the SMBH masses based on FWHM estimation is needed,

     * •

A similar or even the same behavior of normal AGNs and WLQs suggests that both
kinds of sources have the same dim nature of the velocity component and
similar geometry of the BLR.

  2. 2.

The spectrum of SDSS J110511.15+530806.5 with extensive absorption.

     * •

The nature of the deep absorption of the J1105 found an explanation in the two
Gaussian function with $\sigma=332$ Å, EW = 714 Åand $\sigma=375$ Å, EW = 124
Å,

     * •

Column density of the first Gaussian with log$(N_{j})$ of C 4 =
$17.027^{+0.080}_{-0.010}$ $cm^{-2}$ or log$(N_{j})$ of Mg 2
$16.003^{+0.080}_{-0.010}$ $cm^{-2}$,

     * •

Column density of the second Gaussian with log$(N_{j})$ of C 4 =
$16.266^{+0.065}_{-0.080}$ cm-2 or log$(N_{j})$ of Mg 2
$15.242^{+0.065}_{-0.080}$ cm-2

     * •

The explanation of the origin of the absorption in J1105 by strong
outflow/wind and/or intervening material along the line of sight (Fig. 6.5 and
Fig. 6.6)

###  7.1 Future work/prospect of WLQs research

I will focus on describing the AD continua models for the broader sample to
constrain $M_{\mathrm{BH}}$ and $\dot{m}$ for WLQs. It will include a recently
developed sample of X-ray WLQs (e.g. Luo et al. 2015; Ni et al. 2018, 2020;
Timlin et al. 2020). Using more sophisticated codes (e.g. hardening factor,
the influence of the corona) – such as: QSOSED, KerrBB from XSPEC package – I
will develop a grid of models with a range of $M_{\mathrm{BH}}$, $\dot{m}$,
and spin parameters. Exploiting available data I will scour databases to
create a SED of each WLQ. The potential new observations for WLQs are crucial
for the disk-fitting method to have a spectrum with UV points. It allows us to
observed the bend point in the SED which is important for the model
constraints. The Swift’s Ultraviolet/Optical Telescope (UVOT) and/or Hubble
Space Telescope (HST) time will be significant.

Figure 7.1: Baldwin effect. Blue dots, sample of normal Quasars. Red area
expected range of WLQs.

The Baldwin effect is the relation between the luminosity of AGNs and
emission-lines in the electromagnetic spectra. It is an important feature in
the description of quasars and AGNs because the value of EW is related to
$\dot{m}$ and/or spin of BH. The results from the first task will be used to
determine the Baldwin effect for the broader WLQs sample. I will determine the
Baldwin effect by a using broader sample of WLQs. Previous attempts have been
done; however, I will use corrected $\dot{m}$. I will include new properties
in the model description such as: spectral hardening factor, the effect of
self-irradiation, and the effect of limb-darkening. I am aiming to see an
anti-correlation between EW of lines e.g (C 4) and $L/L_{Edd}$ relation in WLQ
(Figure 7.1). However, due to the weakness of C 4 line, I expect to observe
shifted anti-correlation which is different than for normal quasars (red area
in Figure 7.1).

Additionally, I will explore the influence of radiative efficiency on this in
a broader WLQs sample. I will inspect the wide range of $\eta$ parameter,
which influences the $L/L_{Edd}$ and thus the spin of BH. As a result, the
shape of SED will be more constrained due to the strong dependency on
$L/L_{Edd}$. In order to constraint/establish the nature of WLQs well, more
research is needed. I would expand the WLQs sample and use the recent X-ray
development in the WLQs matter.

## References

  * Abazajian et al. (2009) Abazajian, K. N., Adelman-McCarthy, J. K., Agüeros, M. A., Allam, S. S., Allende Prieto, C., An, D., Anderson, K. S. J., Anderson, S. F., Annis, J., Bahcall, N. A., & et al. 2009, ApJS, 182, 543
  * Abramowicz et al. (1988) Abramowicz, M. A., Czerny, B., Lasota, J. P., & Szuszkiewicz, E. 1988, ApJ, 332, 646
  * Andika et al. (2020) Andika, I. T., Jahnke, K., Onoue, M., Bañados, E., Mazzucchelli, C., Novak, M., Eilers, A.-C., Venemans, B. P., Schindler, J.-T., Walter, F., Neeleman, M., Simcoe, R. A., Decarli, R., Farina, E. P., Marian, V., Pensabene, A., Cooper, T. M., & Rojas, A. F. 2020, ApJ, 903, 34
  * Antonucci (2002) Antonucci, R. 2002, in Astronomical Society of the Pacific Conference Series, Vol. 284, IAU Colloq. 184: AGN Surveys, ed. R. F. Green, E. Y. Khachikian, & D. B. Sanders, 147
  * Arnaud (1996) Arnaud, K. A. 1996, in Astronomical Society of the Pacific Conference Series, Vol. 101, Astronomical Data Analysis Software and Systems V, ed. G. H. Jacoby & J. Barnes, 17
  * Bañados et al. (2014) Bañados, E., Venemans, B. P., Morganson, E., Decarli, R., Walter, F., Chambers, K. C., Rix, H. W., Farina, E. P., Fan, X., Jiang, L., McGreer, I., De Rosa, G., Simcoe, R., Weiß, A., Price, P. A., Morgan, J. S., Burgett, W. S., Greiner, J., Kaiser, N., Kudritzki, R. P., Magnier, E. A., Metcalfe, N., Stubbs, C. W., Sweeney, W., Tonry, J. L., Wainscoat, R. J., & Waters, C. 2014, AJ, 148, 14
  * Baskin & Laor (2005) Baskin, A. & Laor, A. 2005, MNRAS, 356, 1029
  * Bentz & Manne-Nicholas (2018) Bentz, M. C. & Manne-Nicholas, E. 2018, ApJ, 864, 146
  * Bentz et al. (2009) Bentz, M. C., Walsh, J. L., Barth, A. J., Baliber, N., Bennert, V. N., Canalizo, G., Filippenko, A. V., Ganeshalingam, M., Gates, E. L., Greene, J. E., Hidas, M. G., Hiner, K. D., Lee, N., Li, W., Malkan, M. A., Minezaki, T., Sakata, Y., Serduke, F. J. D., Silverman, J. M., Steele, T. N., Stern, D., Street, R. A., Thornton, C. E., Treu, T., Wang, X., Woo, J.-H., & Yoshii, Y. 2009, ApJ, 705, 199
  * Bian & Zhao (2002) Bian, W. & Zhao, Y. 2002, A&A, 395, 465
  * Bianchi et al. (2017) Bianchi, L., Shiao, B., & Thilker, D. 2017, ApJS, 230, 24
  * Blandford et al. (2019) Blandford, R., Meier, D., & Readhead, A. 2019, ARA&A, 57, 467
  * Blandford & McKee (1982) Blandford, R. D. & McKee, C. F. 1982, ApJ, 255, 419
  * Blandford et al. (1990) Blandford, R. D., Netzer, H., Woltjer, L., Courvoisier, T. J. L., & Mayor, M. 1990, Active Galactic Nuclei (Springer)
  * Bondi (1952) Bondi, H. 1952, MNRAS, 112, 195
  * Cao (2009) Cao, X. 2009, MNRAS, 394, 207
  * Cao & Li (2008) Cao, X. & Li, F. 2008, MNRAS, 390, 561
  * Capellupo et al. (2015) Capellupo, D. M., Netzer, H., Lira, P., Trakhtenbrot, B., & Mejía-Restrepo, J. 2015, MNRAS, 446, 3427
  * Cardelli et al. (1989) Cardelli, J. A., Clayton, G. C., & Mathis, J. S. 1989, ApJ, 345, 245
  * Castignani et al. (2013) Castignani, G., Haardt, F., Lapi, A., De Zotti, G., Celotti, A., & Danese, L. 2013, A&A, 560, A28
  * Cochran (1952) Cochran, W. G. 1952, The Annals of Mathematical Statistics, 23, 315
  * Collin & Kawaguchi (2004) Collin, S. & Kawaguchi, T. 2004, A&A, 426, 797
  * Collin et al. (2006) Collin, S., Kawaguchi, T., Peterson, B. M., & Vestergaard, M. 2006, A&A, 456, 75
  * Collinson et al. (2015) Collinson, J. S., Ward, M. J., Done, C., Land t, H., Elvis, M., & McDowell, J. C. 2015, MNRAS, 449, 2174
  * Collinson et al. (2017) Collinson, J. S., Ward, M. J., Landt, H., Done, C., Elvis, M., & McDowell, J. C. 2017, MNRAS, 465, 358
  * Czerny (2019) Czerny, B. 2019, Open Astronomy, 28, 200
  * Czerny et al. (2016) Czerny, B., Du, P., Wang, J.-M., & Karas, V. 2016, ApJ, 832, 15
  * Czerny et al. (2011) Czerny, B., Hryniewicz, K., Nikołajuk, M., & Sądowski, A. 2011, MNRAS, 415, 2942
  * Czerny et al. (2004a) Czerny, B., Li, J., Loska, Z., & Szczerba, R. 2004a, MNRAS, 348, L54
  * Czerny et al. (2004b) —. 2004b, MNRAS, 348, L54
  * Czerny et al. (2003) Czerny, B., Nikołajuk, M., Różańska, A., Dumont, A. M., Loska, Z., & Zycki, P. T. 2003, A&A, 412, 317
  * Czerny et al. (2019) Czerny, B., Wang, J.-M., Du, P., Hryniewicz, K., Karas, V., Li, Y.-R., Pand a, S., Sniegowska, M., Wildy, C., & Yuan, Y.-F. 2019, ApJ, 870, 84
  * Decarli et al. (2008) Decarli, R., Dotti, M., Fontana, M., & Haardt, F. 2008, MNRAS, 386, L15
  * DeGraf et al. (2015) DeGraf, C., Di Matteo, T., Treu, T., Feng, Y., Woo, J. H., & Park, D. 2015, MNRAS, 454, 913
  * Denney et al. (2009) Denney, K. D., Peterson, B. M., Pogge, R. W., Adair, A., Atlee, D. W., Au-Yong, K., Bentz, M. C., Bird, J. C., Brokofsky, D. J., Chisholm, E., Comins, M. L., Dietrich, M., Doroshenko, V. T., Eastman, J. D., Efimov, Y. S., Ewald, S., Ferbey, S., Gaskell, C. M., Hedrick, C. H., Jackson, K., Klimanov, S. A., Klimek, E. S., Kruse, A. K., Ladéroute, A., Lamb, J. B., Leighly, K., Minezaki, T., Nazarov, S. V., Onken, C. A., Petersen, E. A., Peterson, P., Poindexter, S., Sakata, Y., Schlesinger, K. J., Sergeev, S. G., Skolski, N., Stieglitz, L., Tobin, J. J., Unterborn, C., Vestergaard, M., Watkins, A. E., Watson, L. C., & Yoshii, Y. 2009, ApJ, 704, L80
  * Denney et al. (2010) —. 2010, ApJ, 721, 715
  * Diamond-Stanic et al. (2009) Diamond-Stanic, A. M., Fan, X., Brandt, W. N., Shemmer, O., Strauss, M. A., Anderson, S. F., Carilli, C. L., Gibson, R. R., Jiang, L., Kim, J. S., Richards, G. T., Schmidt, G. D., Schneider, D. P., Shen, Y., Smith, P. S., Vestergaard, M., & Young, J. E. 2009, ApJ, 699, 782
  * Dobrzycki et al. (2007) Dobrzycki, A., Nikolajuk, M., Bechtold, J., Ebeling, H., Czerny, B., & Różańska, A. 2007, A&A, 476, 1205
  * Done & Krolik (1996) Done, C. & Krolik, J. H. 1996, ApJ, 463, 144
  * Fan et al. (1999) Fan, X., Strauss, M. A., Gunn, J. E., Lupton, R. H., Carilli, C. L., Rupen, M. P., Schmidt, G. D., Moustakas, L. A., Davis, M., Annis, J., Bahcall, N. A., Brinkmann, J., Brunner, R. J., Csabai, I., Doi, M., Fukugita, M., Heckman, T. M., Hennessy, G. S., Hindsley, R. B., Ivezić , Ž., Knapp, G. R., Lamb, D. Q., Munn, J. A., Pauls, A. G., Pier, J. R., Rockosi, C. M., Schneider, D. P., Szalay, A. S., Tucker, D. L., & York, D. G. 1999, ApJ, 526, L57
  * Fausnaugh et al. (2017) Fausnaugh, M. M., Grier, C. J., Bentz, M. C., Denney, K. D., De Rosa, G., Peterson, B. M., Kochanek, C. S., Pogge, R. W., Adams, S. M., Barth, A. J., Beatty, T. G., Bhattacharjee, A., Borman, G. A., Boroson, T. A., Bottorff, M. C., Brown, J. E., Brown, J. S., Brotherton, M. S., Coker, C. T., Crawford, S. M., Croxall, K. V., Eftekharzadeh, S., Eracleous, M., Joner, M. D., Henderson, C. B., Holoien, T. W. S., Horne, K., Hutchison, T., Kaspi, S., Kim, S., King, A. L., Li, M., Lochhaas, C., Ma, Z., MacInnis, F., Manne-Nicholas, E. R., Mason, M., Montuori, C., Mosquera, A., Mudd, D., Musso, R., Nazarov, S. V., Nguyen, M. L., Okhmat, D. N., Onken, C. A., Ou-Yang, B., Pancoast, A., Pei, L., Penny, M. T., Poleski, R., Rafter, S., Romero-Colmenero, E., Runnoe, J., Sand, D. J., Schimoia, J. S., Sergeev, S. G., Shappee, B. J., Simonian, G. V., Somers, G., Spencer, M., Starkey, D. A., Stevens, D. J., Tayar, J., Treu, T., Valenti, S., Van Saders, J., Villanueva, S., J., Villforth, C., Weiss, Y., Winkler, H., & Zhu, W. 2017, ApJ, 840, 97
  * Fitzpatrick (1999) Fitzpatrick, E. L. 1999, PASP, 111, 63
  * Forster et al. (2001) Forster, K., Green, P. J., Aldcroft, T. L., Vestergaard, M., Foltz, C. B., & Hewett, P. C. 2001, ApJS, 134, 35
  * Francis et al. (1991) Francis, P. J., Hewett, P. C., Foltz, C. B., Chaffee, F. H., Weymann, R. J., & Morris, S. L. 1991, ApJ, 373, 465
  * García et al. (2013) García, J., Dauser, T., Reynolds, C. S., Kallman, T. R., McClintock, J. E., Wilms, J., & Eikmann, W. 2013, ApJ, 768, 146
  * Gaskell (2009) Gaskell, C. M. 2009, New Astronomy Reviews, 53, 140
  * Gelman et al. (2004) Gelman, A., Carlin, J. B., Stern, H. S., & Rubin, D. B. 2004, Bayesian Data Analysis, 2nd edn. (Chapman and Hall/CRC)
  * Greene & Ho (2004) Greene, J. E. & Ho, L. C. 2004, ApJ, 610, 722
  * Griesmann & Kling (2000) Griesmann, U. & Kling, R. 2000, ApJ, 536, L113
  * Haardt & Maraschi (1991) Haardt, F. & Maraschi, L. 1991, ApJ, 380, L51
  * Heintz et al. (2018) Heintz, K. E., Fynbo, J. P. U., Ledoux, C., Jakobsson, P., Møller, P., Christensen, L., Geier, S., Krogager, J. K., & Noterdaeme, P. 2018, A&A, 615, A43
  * Hewett et al. (1995) Hewett, P. C., Foltz, C. B., & Chaffee, F. H. 1995, AJ, 109, 1498
  * Hewett et al. (2001) —. 2001, AJ, 122, 518
  * Hryniewicz et al. (2010) Hryniewicz, K., Czerny, B., Nikołajuk, M., & Kuraszkiewicz, J. 2010, MNRAS, 404, 2028
  * Hubeny et al. (2000) Hubeny, I., Agol, E., Blaes, O., & Krolik, J. H. 2000, ApJ, 533, 710
  * Hutchings & Giasson (2001) Hutchings, J. B. & Giasson, J. 2001, PASP, 113, 1205
  * Ji et al. (2015) Ji, T., Zhou, H., Jiang, P., Wang, T., Ge, J., Wang, H., Komossa, S., Hamann, F., Zuther, J., Liu, W., Lu, H., Zuo, W., Yang, C., & Yuan, W. 2015, ApJ, 800, 56
  * Jiang et al. (2013) Jiang, P., Zhou, H., Ji, T., Shu, X., Liu, W., Wang, J., Dong, X., Bai, J., Wang, H., & Wang, T. 2013, AJ, 145, 157
  * Jiang et al. (2014) Jiang, Y.-F., Stone, J. M., & Davis, S. W. 2014, The Astrophysical Journal, 796, 106
  * Just et al. (2007) Just, D. W., Brandt, W. N., Shemmer, O., Steffen, A. T., Schneider, D. P., Chartas, G., & Garmire, G. P. 2007, ApJ, 665, 1004
  * Kaspi et al. (2005) Kaspi, S., Maoz, D., Netzer, H., Peterson, B. M., Vestergaard, M., & Jannuzi, B. T. 2005, ApJ, 629, 61
  * Kaspi et al. (2000) Kaspi, S., Smith, P. S., Netzer, H., Maoz, D., Jannuzi, B. T., & Giveon, U. 2000, ApJ, 533, 631
  * Knigge et al. (2008) Knigge, C., Scaringi, S., Goad, M. R., & Cottis, C. E. 2008, MNRAS, 386, 1426
  * Kratzer & Richards (2015) Kratzer, R. M. & Richards, G. T. 2015, AJ, 149, 61
  * Krolik (1999) Krolik, J. H. 1999, Active galactic nuclei : from the central black hole to the galactic environment (Princeton University Press)
  * Kubota & Done (2018) Kubota, A. & Done, C. 2018, MNRAS, 480, 1247
  * Laor et al. (2006) Laor, A., Barth, A. J., Ho, L. C., & Filippenko, A. V. 2006, ApJ, 636, 83
  * Laor & Davis (2011) Laor, A. & Davis, S. W. 2011, MNRAS, 417, 681
  * Leighly et al. (2007a) Leighly, K. M., Halpern, J. P., Jenkins, E. B., & Casebeer, D. 2007a, ApJS, 173, 1
  * Leighly et al. (2007b) Leighly, K. M., Halpern, J. P., Jenkins, E. B., Grupe, D., Choi, J., & Prescott, K. B. 2007b, ApJ, 663, 103
  * Li et al. (2020) Li, Q., Wang, R., Fan, X., Wu, X.-B., Jiang, L., Bañados, E., Venemans, B., Shao, Y., Li, J., Zhang, Y., Zhang, C., Wagg, J., Decarli, R., Mazzucchelli, C., Omont, A., & Bertoldi, F. 2020, ApJ, 900, 12
  * Liu et al. (2015) Liu, W.-J., Zhou, H., Ji, T., Yuan, W., Wang, T.-G., Jian, G., Shi, X., Zhang, S., Jiang, P., Shu, X., Wang, H., Wang, S.-F., Sun, L., Yang, C., Liu, B., & Zhao, W. 2015, ApJS, 217, 11
  * Liu & Zhang (2011) Liu, Y. & Zhang, S. N. 2011, ApJ, 728, L44
  * Loska et al. (2004) Loska, Z., Czerny, B., & Szczerba, R. 2004, MNRAS, 355, 1080
  * Luo et al. (2015) Luo, B., Brandt, W. N., Hall, P. B., Wu, J., Anderson, S. F., Garmire, G. P., Gibson, R. R., Plotkin, R. M., Richards, G. T., Schneider, D. P., Shemmer, O., & Shen, Y. 2015, ApJ, 805, 122
  * Marculewicz & Nikolajuk (2020) Marculewicz, M. & Nikolajuk, M. 2020, ApJ, 897, 117
  * Marlar et al. (2018) Marlar, A., Shemmer, O., Anderson, S. F., Brand t, W. N., Diamond-Stanic, A. M., Fan, X., Luo, B., Plotkin, R. M., Richards, G. T., Schneider, D. P., & Wu, J. 2018, ApJ, 865, 92
  * McDowell et al. (1995) McDowell, J. C., Canizares, C., Elvis, M., Lawrence, A., Markoff, S., Mathur, S., & Wilkes, B. J. 1995, ApJ, 450, 585
  * Mejía-Restrepo et al. (2018a) Mejía-Restrepo, J. E., Lira, P., Netzer, H., Trakhtenbrot, B., & Capellupo, D. M. 2018a, Nature Astronomy, 2, 63
  * Mejía-Restrepo et al. (2018b) —. 2018b, Nature Astronomy, 2, 63
  * Meusinger & Balafkan (2014) Meusinger, H. & Balafkan, N. 2014, A&A, 568, A114
  * Meusinger et al. (2016) Meusinger, H., Schalldach, P., Mirhosseini, A., & Pertermann, F. 2016, A&A, 587, A83
  * Morton (2003) Morton, D. C. 2003, ApJS, 149, 205
  * Narayan & Yi (1994) Narayan, R. & Yi, I. 1994, ApJ, 428, L13
  * Netzer (2013) Netzer, H. 2013, The Physics and Evolution of Active Galactic Nuclei (Cambridge University Press; 1 edition)
  * Netzer (2015) —. 2015, ARA&A, 53, 365
  * Netzer & Trakhtenbrot (2014) Netzer, H. & Trakhtenbrot, B. 2014, MNRAS, 438, 672
  * Ni et al. (2018) Ni, Q., Brandt, W. N., Luo, B., Hall, P. B., Shen, Y., Anderson, S. F., Plotkin, R. M., Richards, G. T., Schneider, D. P., Shemmer, O., & Wu, J. 2018, MNRAS, 480, 5184
  * Ni et al. (2020) Ni, Q., Brandt, W. N., Yi, W., Luo, B., Timlin, J. D., I., Hall, P. B., Liu, H., Plotkin, R. M., Shemmer, O., Vito, F., & Wu, J. 2020, ApJ, 889, L37
  * Nikolajuk et al. (2004) Nikolajuk, M., Czerny, B., Różańska, A., & Dumont, A.-M. 2004, Nuclear Physics B Proceedings Supplements, 132, 201
  * Nikołajuk et al. (2006) Nikołajuk, M., Czerny, B., Ziółkowski, J., & Gierliński, M. 2006, MNRAS, 370, 1534
  * Nikołajuk & Walter (2012) Nikołajuk, M. & Walter, R. 2012, MNRAS, 420, 2518
  * Novikov & Thorne (1973) Novikov, I. D. & Thorne, K. S. 1973, in Black Holes (Les Astres Occlus), 343–450
  * Onken et al. (2004) Onken, C. A., Ferrarese, L., Merritt, D., Peterson, B. M., Pogge, R. W., Vestergaard, M., & Wandel, A. 2004, ApJ, 615, 645
  * Padovani et al. (2017) Padovani, P., Alexander, D. M., Assef, R. J., De Marco, B., Giommi, P., Hickox, R. C., Richards, G. T., Smolčić, V., Hatziminaoglou, E., Mainieri, V., & Salvato, M. 2017, A&A Rev., 25, 2
  * Page & Thorne (1974) Page, D. N. & Thorne, K. S. 1974, ApJ, 191, 499
  * Page et al. (2014) Page, M. J., Yershov, V., Breeveld, A., Kuin, P., Mignani, R., Smith, P. J., Rawlings, J., Oates, S., Siegel, M., & Roming, P. 2014, in Proceedings of Swift: 10 Years of Discovery (SWIFT 10, 37
  * Pancoast et al. (2014) Pancoast, A., Brewer, B. J., Treu, T., Park, D., Barth, A. J., Bentz, M. C., & Woo, J.-H. 2014, MNRAS, 445, 3073
  * Peterson (1993) Peterson, B. M. 1993, PASP, 105, 247
  * Peterson (1997) —. 1997, An Introduction to Active Galactic Nuclei (Cambridge, New York Cambridge University Press)
  * Peterson (2004) Peterson, B. M. 2004, in IAU Symposium, Vol. 222, The Interplay Among Black Holes, Stars and ISM in Galactic Nuclei, ed. T. Storchi-Bergmann, L. C. Ho, & H. R. Schmitt, 15–20
  * Peterson (2014) —. 2014, Space Sci. Rev., 183, 253
  * Peterson et al. (2004) Peterson, B. M., Ferrarese, L., Gilbert, K. M., Kaspi, S., Malkan, M. A., Maoz, D., Merritt, D., Netzer, H., Onken, C. A., Pogge, R. W., Vestergaard, M., & Wandel, A. 2004, ApJ, 613, 682
  * Petrucci et al. (2018) Petrucci, P. O., Ursini, F., De Rosa, A., Bianchi, S., Cappi, M., Matt, G., Dadina, M., & Malzac, J. 2018, A&A, 611, A59
  * Plotkin et al. (2015) Plotkin, R. M., Shemmer, O., Trakhtenbrot, B., Anderson, S. F., Brandt, W. N., Fan, X., Gallo, E., Lira, P., Luo, B., Richards, G. T., Schneider, D. P., Strauss, M. A., & Wu, J. 2015, ApJ, 805, 123
  * Polletta et al. (2007) Polletta, M., Tajer, M., Maraschi, L., Trinchieri, G., Lonsdale, C. J., Chiappetti, L., Andreon, S., Pierre, M., Le Fèvre, O., Zamorani, G., Maccagni, D., Garcet, O., Surdej, J., Franceschini, A., Alloin, D., Shupe, D. L., Surace, J. A., Fang, F., Rowan-Robinson, M., Smith, H. E., & Tresse, L. 2007, ApJ, 663, 81
  * Prochaska et al. (2011) Prochaska, J. X., Weiner, B., Chen, H. W., Cooksey, K. L., & Mulchaey, J. S. 2011, ApJS, 193, 28
  * Richards et al. (2003) Richards, G. T., Hall, P. B., Vanden Berk, D. E., Strauss, M. A., Schneider, D. P., Weinstein, M. A., Reichard, T. A., York, D. G., Knapp, G. R., Fan, X., Ivezić, Ž., Brinkmann, J., Budavári, T., Csabai, I., & Nichol, R. C. 2003, AJ, 126, 1131
  * Różańska et al. (2015) Różańska, A., Malzac, J., Belmont, R., Czerny, B., & Petrucci, P. O. 2015, A&A, 580, A77
  * Runnoe et al. (2014) Runnoe, J. C., Brotherton, M. S., DiPompeo, M. A., & Shang, Z. 2014, MNRAS, 438, 3263
  * Sadowski et al. (2009) Sadowski, A., Abramowicz, M. A., Bursa, M., Kluźniak, W., Różańska, A., & Straub, O. 2009, A&A, 502, 7
  * Saez et al. (2021) Saez, C., Brandt, W. N., Bauer, F., Chartas, G., Misawa, T., Hamann, F., & Gallagher, S. 2021, arXiv e-prints, arXiv:2106.06567
  * Schlegel et al. (1998) Schlegel, D. J., Finkbeiner, D. P., & Davis, M. 1998, ApJ, 500, 525
  * Shakura & Sunyaev (1973) Shakura, N. I. & Sunyaev, R. A. 1973, A&A, 500, 33
  * Shankar et al. (2009) Shankar, F., Weinberg, D. H., & Miralda-Escudé, J. 2009, ApJ, 690, 20
  * Shemmer et al. (2009) Shemmer, O., Brandt, W. N., Anderson, S. F., Diamond-Stanic, A. M., Fan, X., Richards, G. T., Schneider, D. P., & Strauss, M. A. 2009, ApJ, 696, 580
  * Shemmer et al. (2010) Shemmer, O., Trakhtenbrot, B., Anderson, S. F., Brandt, W. N., Diamond-Stanic, A. M., Fan, X., Lira, P., Netzer, H., Plotkin, R. M., Richards, G. T., Schneider, D. P., & Strauss, M. A. 2010, ApJ, 722, L152
  * Shen (2013) Shen, Y. 2013, Bulletin of the Astronomical Society of India, 41, 61
  * Shen et al. (2008) Shen, Y., Greene, J. E., Strauss, M. A., Richards, G. T., & Schneider, D. P. 2008, ApJ, 680, 169
  * Shen et al. (2019) Shen, Y., Grier, C. J., Horne, K., Brandt, W. N., Trump, J. R., Hall, P. B., Kinemuchi, K., Starkey, D., Schneider, D. P., Ho, L. C., Homayouni, Y., I-Hsiu Li, J., McGreer, I. D., Peterson, B. M., Bizyaev, D., Chen, Y., Dawson, K. S., Eftekharzadeh, S., Green, P. J., Guo, Y., Jia, S., Jiang, L., Kneib, J.-P., Li, F., Li, Z., Nie, J., Oravetz, A., Oravetz, D., Pan, K., Petitjean, P., Ponder, K. A., Rogerson, J., Vivek, M., Zhang, T., & Zou, H. 2019, ApJ, 883, L14
  * Shen & Ho (2014) Shen, Y. & Ho, L. C. 2014, Nature, 513, 210
  * Shen et al. (2011a) Shen, Y., Richards, G. T., Strauss, M. A., Hall, P. B., Schneider, D. P., Snedden, S., Bizyaev, D., Brewington, H., Malanushenko, V., Malanushenko, E., Oravetz, D., Pan, K., & Simmons, A. 2011a, ApJS, 194, 45
  * Shen et al. (2011b) —. 2011b, ApJS, 194, 45
  * Shen et al. (2011c) —. 2011c, ApJS, 194, 45
  * Sivia & Rawlings (2009) Sivia, D. S. & Rawlings, S. G. 2009, Bayesian Methods in Cosmology (Cambridge University Press), 36–56
  * Skrutskie et al. (2006) Skrutskie, M. F., Cutri, R. M., Stiening, R., Weinberg, M. D., Schneider, S., Carpenter, J. M., Beichman, C., Capps, R., Chester, T., Elias, J., Huchra, J., Liebert, J., Lonsdale, C., Monet, D. G., Price, S., Seitzer, P., Jarrett, T., Kirkpatrick, J. D., Gizis, J. E., Howard, E., Evans, T., Fowler, J., Fullmer, L., Hurt, R., Light, R., Kopan, E. L., Marsh, K. A., McCallon, H. L., Tam, R., Van Dyk, S., & Wheelock, S. 2006, AJ, 131, 1163
  * Spergel et al. (2007) Spergel, D. N., Bean, R., Doré, O., Nolta, M. R., Bennett, C. L., Dunkley, J., Hinshaw, G., Jarosik, N., Komatsu, E., Page, L., Peiris, H. V., Verde, L., Halpern, M., Hill, R. S., Kogut, A., Limon, M., Meyer, S. S., Odegard, N., Tucker, G. S., Weiland , J. L., Wollack, E., & Wright, E. L. 2007, ApJS, 170, 377
  * Stone & Richards (2019) Stone, R. B. & Richards, G. T. 2019, MNRAS, 488, 5916
  * Sun & Malkan (1989) Sun, W.-H. & Malkan, M. A. 1989, ApJ, 346, 68
  * Timlin et al. (2020) Timlin, J. D., Brandt, W. N., Ni, Q., Luo, B., Pu, X., Schneider, D. P., Vivek, M., & Yi, W. 2020, MNRAS, 492, 719
  * Trakhtenbrot & Netzer (2012) Trakhtenbrot, B. & Netzer, H. 2012, MNRAS, 427, 3081
  * Véron-Cetty & Véron (2001) Véron-Cetty, M. P. & Véron, P. 2001, A&A, 374, 92
  * Vestergaard (2003) Vestergaard, M. 2003, ApJ, 599, 116
  * Vestergaard & Osmer (2009) Vestergaard, M. & Osmer, P. S. 2009, ApJ, 699, 800
  * Vestergaard & Peterson (2006) Vestergaard, M. & Peterson, B. M. 2006, ApJ, 641, 689
  * Vestergaard & Wilkes (2001) Vestergaard, M. & Wilkes, B. J. 2001, ApJS, 134, 1
  * Wang et al. (2009) Wang, J.-G., Dong, X.-B., Wang, T.-G., Ho, L. C., Yuan, W., Wang, H., Zhang, K., Zhang, S., & Zhou, H. 2009, ApJ, 707, 1334
  * Wang et al. (2019) Wang, S., Shen, Y., Jiang, L., Horne, K., Brandt, W. N., Grier, C. J., Ho, L. C., Homayouni, Y., I-Hsiu Li, J., Schneider, D. P., & Trump, J. R. 2019, ApJ, 882, 4
  * Weymann et al. (1979) Weymann, R. J., Williams, R. E., Peterson, B. M., & Turnshek, D. A. 1979, ApJ, 234, 33
  * Wickramasinghe & Hoyle (1998) Wickramasinghe, N. C. & Hoyle, F. 1998, Ap&SS, 259, 379
  * Wild et al. (2008) Wild, V., Kauffmann, G., White, S., York, D., Lehnert, M., Heckman, T., Hall, P. B., Khare, P., Lundgren, B., Schneider, D. P., & vanden Berk, D. 2008, MNRAS, 388, 227
  * Wildy et al. (2018) Wildy, C., Czerny, B., & Kuźmicz, A. 2018, ApJ, 861, 54
  * Wills & Browne (1986) Wills, B. J. & Browne, I. W. A. 1986, ApJ, 302, 56
  * Winkler (1992) Winkler, H. 1992, MNRAS, 257, 677
  * Wright et al. (2010) Wright, E. L., Eisenhardt, P. R. M., Mainzer, A. K., Ressler, M. E., Cutri, R. M., Jarrett, T., Kirkpatrick, J. D., Padgett, D., McMillan, R. S., Skrutskie, M., Stanford, S. A., Cohen, M., Walker, R. G., Mather, J. C., Leisawitz, D., Gautier, Thomas N., I., McLean, I., Benford, D., Lonsdale, C. J., Blain, A., Mendez, B., Irace, W. R., Duval, V., Liu, F., Royer, D., Heinrichsen, I., Howard, J., Shannon, M., Kendall, M., Walsh, A. L., Larsen, M., Cardon, J. G., Schick, S., Schwalm, M., Abid, M., Fabinsky, B., Naes, L., & Tsai, C.-W. 2010, AJ, 140, 1868
  * Wu et al. (2012a) Wu, J., Brandt, W. N., Anderson, S. F., Diamond-Stanic, A. M., Hall, P. B., Plotkin, R. M., Schneider, D. P., & Shemmer, O. 2012a, ApJ, 747, 10
  * Wu et al. (2011) Wu, J., Brandt, W. N., Hall, P. B., Gibson, R. R., Richards, G. T., Schneider, D. P., Shemmer, O., Just, D. W., & Schmidt, S. J. 2011, ApJ, 736, 28
  * Wu et al. (2012b) Wu, X.-B., Hao, G., Jia, Z., Zhang, Y., & Peng, N. 2012b, AJ, 144, 49
  * Yu & Tremaine (2002) Yu, Q. & Tremaine, S. 2002, MNRAS, 335, 965
  * Yuan & Narayan (2004) Yuan, F. & Narayan, R. 2004, ApJ, 612, 724
  * Zhang et al. (2000) Zhang, S. N., Cui, W., Chen, W., Yao, Y., Zhang, X., Sun, X., Wu, X.-B., & Xu, H. 2000, Science, 287, 1239
  * Zombeck (2007) Zombeck, M. 2007, Handbook of Space Astronomy and Astrophysics: Third Edition (Cambridge University Press)