# Automated Essay Scoring Using Grammatical Variety and Errors
with Multi-Task Learning and Item Response Theory

Kosuke Doi Katsuhito Sudoh Satoshi Nakamura
Nara Institute of Science and Technology
{doi.kosuke.de8, sudoh<EMAIL_ADDRESS>

###### Abstract

This study examines the effect of grammatical features in automatic essay
scoring (AES). We use two kinds of grammatical features as input to an AES
model: (1) grammatical items that writers used correctly in essays, and (2)
the number of grammatical errors. Experimental results show that grammatical
features improve the performance of AES models that predict the holistic
scores of essays. Multi-task learning with the holistic and grammar scores,
alongside using grammatical features, resulted in a larger improvement in
model performance. We also show that a model using grammar abilities estimated
using Item Response Theory (IRT) as the labels for the auxiliary task achieved
comparable performance to when we used grammar scores assigned by human
raters. In addition, we weight the grammatical features using IRT to consider
the difficulty of grammatical items and writers’ grammar abilities. We found
that weighting grammatical features with the difficulty led to further
improvement in performance.111The code is publicly available at
https://github.com/ahclab/aes-grammar-mtl-irt.

Automated Essay Scoring Using Grammatical Variety and Errors
with Multi-Task Learning and Item Response Theory

Kosuke Doi Katsuhito Sudoh Satoshi Nakamura Nara Institute of Science and
Technology {doi.kosuke.de8, sudoh<EMAIL_ADDRESS>

##  1 Introduction

Automated Essay Scoring (AES) is a task that automatically grades essays.
Essay assignments are widely used in language tests and classrooms to assess
learners’ writing abilities, while grading them takes time and effort for
human raters. Maintaining inter- and intra-rater reliability is another issue
associated with human scoring. AES can help alleviate these problems and has
been attracting more attention in recent years.

The grading methods for essays can be roughly categorized into two types:
holistic scoring and analytic scoring. The former assigns a single score to an
essay based on its overall performance, while the latter assigns different
scores to various aspects of the essay, such as grammar, vocabulary, content,
or organization (Weigle, 2002). However, rubrics for holistic scoring
typically contain descriptions of several aspects of writing used in analytic
scoring (_e.g._ , TOEFL iBT Independent Writing Rubric).

Among those aspects, we focus on grammatical features, inspired by the
research on criterial features for the levels of the Common European
Frameworks of References (CEFR) (Council of Europe, 2001) in L2 English
(Hawkins and Filipović, 2012). The CEFR, one of the influential frameworks in
language teaching, describes language abilities in functional terms (_i.e._ ,
can-do statements, such as “Can write short, simple essays on topics of
interest”). However, it is grammatical items and lexis that realize the
functions written in can-do statements. To fully develop and elaborate their
ideas in essays, they need to use a wide range of grammatical items. In fact,
grammar plays an important role in essay scoring. Researches on writing in the
second language acquisition field have been focusing on syntactic
complexity222Syntactic complexity refers to the extent to which a learner can
use a wide variety of both basic and sophisticated structures (Wolfe-Quintero
et al., 1998). and accuracy (see Kuiken, 2023; Housen et al., 2012).

Hawkins and Filipović (2012) identified grammatical items that learners at a
certain level and higher can use correctly and items that learners at a
certain level are prone to making mistakes in. It is known that human raters
look for those features consciously or unconsciously when they evaluate
learners’ performance, and explicit use of grammatical features in AES will
improve model performance.

Grammatical features have been used in many feature-engineering AES models
(see Ke and Ng, 2019) as well as in hybrid models, which incorporate
handcrafted features into deep neural network AES models (Dasgupta et al.,
2018; Uto et al., 2020; Bannò and Matassoni, 2022). In Yannakoudakis et al.
(2011), features representing grammatical structures were used together with
other linguistic features. However, in many previous studies (_e.g._ ,
Vajjala, 2018; Uto et al., 2020), grammatical items used correctly were
aggregated into measures of grammatical complexity (_e.g._ , ratio of
dependent clauses per clauses; see Wolfe-Quintero et al., 1998) rather than
individual grammatical items (_e.g._ , adverbial clause _if_ , adverbial
clause _so that_) even though the difficulties of individual grammatical items
are different.

In this paper, we propose to use individual grammatical items as inputs to
hybrid AES models that predict holistic scores, and leverage the models to
incorporate the variety of grammatical items in grading essays. We also use
frequencies of grammatical errors corrected by a modern grammatical error
correction model (GECToR-large; Tarnavskyi et al., 2022) as model inputs. The
grammatical features are combined with an essay representation and passed into
a fully connected feed-forward neural network to predict the score of an input
essay. Our models used BERT (Devlin et al., 2019) to learn essay
representations following the current state-of-the-art AES models (Yang et
al., 2020; Cao et al., 2020; Wang et al., 2022).

To utilize grammatical features more effectively, we develop a multi-task
learning framework that jointly learns to predict holistic scores and grammar
scores of essays. We use two types of grammar scores: (1) scores assigned to
essays by human raters and (2) writers’ latent abilities estimated based on
patterns of grammar usage using Item Response Theory (IRT) (Lord, 1980). Note
that teacher labels are not necessary for estimating the latent abilities
using IRT.

IRT estimates not only each writer’s ability but also the characteristics of
each item (_i.e._ , individual grammatical item), such as discrimination and
difficulty parameters. Therefore, we use these IRT parameters to weight
grammatical items (_e.g._ , award writers who use a difficult grammatical
item; see Section 3.1.2).

In summary, the contributions of this paper are as follows:

  * •

We propose to use individual grammatical items and grammatical errors as
inputs to AES models, and leverage the models to consider grammar use in
predicting holistic scores of essays.

  * •

We develop a multi-task learning framework that jointly learns to predict
holistic scores and grammar scores of essays.

  * •

We apply IRT to writers’ grammar usage patterns and (1) use estimated latent
abilities for multi-task learning, and (2) use IRT parameters to weight
grammatical items when we feed them to AES models.

  * •

We show the effectiveness of incorporating grammatical features into BERT-
hybrid AES models. Our method shows a significant advantage on some essay
assignments in the Automated Student Assessment Prize (ASAP)
dataset333https://www.kaggle.com/c/asap-aes.

##  2 Related Work

###  2.1 Automated Essay Scoring

Early AES models predict essay scores using handcrafted features (see Ke and
Ng, 2019). For example, e-rater Burstein et al. (2004) uses 12 features,
including grammatical errors and lexical complexity measures. Yannakoudakis et
al. (2011) automatically extracted various linguistic features, including
grammatical structures, using a parser. These features were weighted and used
to train SVM ranking models. Vajjala (2018) reported that measures of
grammatical complexity and errors were assigned large weights among various
linguistic features.

Recently, a deep neural network-based approach has become popular. AES models
based on RNN (Taghipour and Ng, 2016), Bi-LSTM Alikaniotis et al. (2016), and
pre-trained language models (Nadeem et al., 2019; Yang et al., 2020; Cao et
al., 2020; Wang et al., 2022) have been proposed. In addition, a hybrid model,
which incorporates handcrafted features into a deep neural network-based
model, has been proposed (Dasgupta et al., 2018; Uto et al., 2020; Bannò and
Matassoni, 2022).

AES using a large language model has also been explored. Mizumoto and Eguchi
(2023) demonstrated that using linguistic features in GPT-3 improved AES
performance. Yancey et al. (2023) reported that providing a small number of
scoring examples to GPT-4 led to comparable performance to models trained on
hundreds of thousands of data based on 85 language features.

This study examines the effect of explicitly considering grammatical features
in a hybrid AES model by incorporating individual grammatical items as model
inputs and weighting them using IRT parameters.

###  2.2 Multi-Task Learning

Multi-task learning (MTL) (Caruana, 1997) is a method that improves the
generalization performance of the main task by training a single model to
perform multiple tasks simultaneously. MTL has been used in previous studies
in AES, and shown to be effective. Cummins et al. (2016) used MTL to overcome
the lack of task-specific data in the ASAP dataset by treating each essay
prompt as a different task. Xue et al. (2021) also trained a model jointly on
eight different prompts in the ASAP dataset using BERT.

There are also studies that have performed MTL with other NLP tasks. Cummins
and Rei (2018) trained an LSTM jointly on grammatical error detection and AES.
While the error detection task in Cummins and Rei (2018) required the model to
predict whether a particular token was errorful, ones in Elks (2021) require
to (1) predict a sentence contains errors or (2) classify tokens by a type of
error (_e.g._ , correct, lexical, form). Other auxiliary tasks used in
previous studies include morpho-syntactic labeling, language modeling, and
native language identification (Craighead et al., 2020), sentiment analysis
(Muangkammuen and Fukumoto, 2020), predicting the level of each token (Elks,
2021), and predicting span, type, and quality of argumentative elements (Ding
et al., 2023).

In this paper, we train models jointly on holistic scores and grammar scores.
This is similar to AES models that predict multiple essay traits
simultaneously (Mathias and Bhattacharyya, 2020; Hussein et al., 2020; Mim et
al., 2019; Ridley et al., 2021), but the difference between them and ours is
that we explicitly incorporate grammatical features to a model, which are
related to the score to be predicted.

###  2.3 Item Response Theory

IRT is a probabilistic model that has been widely used in psychological and
educational measurement (Hambleton et al., 1991). An IRT model expresses the
probability of a correct response to a test item as a function of the item
parameters, which represent the characteristics of the item, and the ability
parameter, which represents the ability of the examinee.

Previous studies in AES used IRT to mitigate raters’ bias (Uto and Okano,
2021), integrate prediction scores from various AES models (Aomi et al., 2021;
Uto et al., 2023), and predict multiple essay traits (Uto, 2021; Shibata and
Uto, 2022). These studies employed a multidimensional IRT model since
unidimensionality cannot be assumed for the subject to which IRT is applied.

In contrast, we regard individual grammatical items as test items, assuming
that whether grammar items are used correctly constitutes grammar ability
(_i.e._ , satisfy the assumption of unidimensionality). We model writers’
grammar ability using two-parameter logistic model (Lord, 1952), formulated by
the following equation:

$P_{ij}(\theta_{i})=\frac{1}{1+\exp(-Da_{j}(\theta_{i}-b_{j}))}$ (1)

where $P_{ij}(\theta_{i})$ is the probability that the writer $i$ with ability
$\theta_{i}$ uses the grammatical item $j$ correctly, $a_{j}$ is the
discrimination parameter for item $j$, and $b_{j}$ is the difficulty parameter
for item $j$. $D$ is a scaling factor and set to $1.0$ in this paper.

##  3 Proposed Method

###  3.1 Grammatical Features

The Common European Frameworks of References (CEFR) (Council of Europe, 2001)
is an internationally recognized framework for language proficiency. It
divides proficiency into six levels ranging from A1 (beginner) to C2
(advanced). Due to the language-neutral nature of the CEFR, what grammatical
and lexical properties learners develop across the CEFR levels has been
studied language by language.

Such properties (criterial features) in English have been identified by
English Profile Programme (Hawkins and Filipović, 2012). Criterial features
refer to linguistic properties that are characteristic and indicative of L2
proficiency levels and that distinguish higher levels from lower (_ibid_).
They identified positive linguistic features (PFs; grammatical items that
learners can use correctly at a certain level and higher) and negative
linguistic features (NFs; grammatical items that learners at a certain level
are prone to making mistakes in) in relation to the CEFR levels.

Based on the analyses of human raters’ grading performance in actual exams,
Hawkins and Buttery (2009) have argued that they develop clear intuitions
about these properties. We expect that allowing a model to learn grammar
representations using grammatical features would improve the AES performance.
Table 1 shows PFs and NFs used in our experiments. The following sections
describe them in detail.

Features |  Descriptions
---|---
type256 |  256 grammatical items, whether a writer use the items
err54 |  54 types of errors, # of errors
multiply_b |  Modify type256 with item difficulty
prob |  Replace elements in type256 with the probabilities of using the items correctly
multiply_prob |  Weight type256 with the probabilities
add_prob |  Consider both the actual use (type256) and the probabilities
Table 1: Grammatical features used in our experiments. The number of errors is
relative freq. per 100 words.

####  3.1.1 Positive Linguistic Features

PFs were extracted using a toolkit for frequency analysis of grammatical
items, which is provided by the CEFR-J Grammar Profile (Ishii and Tono, 2018).
It extracts 501 grammatical items in text based on regular expressions and
calculates the frequencies of them. We converted the frequencies into the
256-dimensional vector (type256) based on CEFR-J Grammar Profile for
Teachers444https://www.cefr-j.org/download.html#cefrj_grammar
The toolkit distinguishes the same items in different sentence types such as
the affirmative or negative, while CEFR-J Grammar Profile for Teachers does
not. as $\bm{g_{i}}=\\{g_{i1},g_{i2},...,g_{i256}\\}$. Each dimension
corresponds to a grammatical item, and $g_{ij}=1$ if the writer $i$ used the
item $j$ in the essay, and 0 if not.

####  3.1.2 PFs Weighted using IRT Parameters

Researches on criterial features have shown that learners master more and more
grammatical items across the CEFR levels, but type256 does not consider the
difficulties of the items. Therefore, we weight them using the IRT parameters.

We transform $g_{ij}$ in the following four ways:

multiply_b:

$g^{\prime}_{ij}=g_{ij}\times b_{j}$

prob:

$g^{\prime}_{ij}=P_{ij}(\hat{\theta}_{i})$

multiply_prob:

$g^{\prime}_{ij}=g_{ij}\times P_{ij}(\hat{\theta}_{i})$

add_prob:

$g^{\prime}_{ij}=\alpha g_{ij}+(1-\alpha)P_{ij}(\hat{\theta}_{i})$

where $\hat{\theta}_{i}$ is the grammatical ability of the writer $i$
estimated based on the patterns of grammar usage using IRT, and $\alpha$ is a
weighting parameter. $\alpha$ was set to $0.5$ in this paper.

multply_b aims to consider the difficulty of items by multiplying the
difficulty parameter for the item. However, writers might not have used some
grammatical items because of the essay topic although they had enough
abilities to do. Therefore, we use $P_{ij}(\hat{\theta}_{i})$, which shows the
probability that the writer $i$ with ability $\hat{\theta}_{i}$ can use the
item $j$ correctly. In prob $g_{ij}$ is replaced with
$P_{ij}(\hat{\theta}_{i})$, while in multiply_prob and add_prob the two values
are combined to consider both the ability of writers and the actual use in
essays.

IRT parameters were estimated independently from the prediction of essay
scores. The IRT parameters were frozen during the training of scoring models.

####  3.1.3 Negative Linguistic Features

We calculated the number of grammatical errors per 100 words as NFs.
Specifically, we created the 54-dimensional vector (err54) based on error tags
assigned by ERRANT (Bryant et al., 2017)555Based on all possible combinations
of the error types and categories. We tried the 24-dimensional vector, which
was based on the error types, but the 54-dimensional vector improved the model
performance more.. We used GECToR-large (Tarnavskyi et al., 2022) to correct
grammatical errors in essays.

###  3.2 Model Architecture

Figure 1: The architectures of proposed models

Our model takes a batch of essays and grammatical features as input and
predicts the holistic scores of the essays. We prepare a model that takes only
a batch of essays as input for a baseline. Essay representations are obtained
from the [CLS] token of the BERT model.

Grammatical features are used in the four settings shown in Figure 1. In cat,
we concatenate the essay representation and the vector of grammatical
features, and feed it to a fully connected feed-forward neural network (FFNN).
In net, we first feed the vector of grammatical features to an FFNN and
concatenate the representation from the final layer with the essay
representation. In multi, we perform multi-task learning with the model
architecture of net. The FFNN in multi consists of shared layers only, and
does not have task-specific layers666We tried models with task-specific
layers, but the performance was worse than ones without them.. In dual, the
predicted values for the auxiliary task are output from the FFNN for
grammatical features.

As the labels for the auxiliary task in multi and dual, we used grammar scores
assigned to essays by human raters, which is available in ASAP and ASAP++
dataset (Mathias and Bhattacharyya, 2018), and grammar abilities estimated
using IRT. Grammar abilities can be estimated from writers’ grammar usage
patterns without any teacher labels.

##  4 Experiments

###  4.1 Data and Evaluation

We used the ASAP and the ASAP++ dataset in our experiments. The ASAP consists
of essays for eight different prompts, with holistic scores for Prompts 1-6
and analytic scores for Prompts 7-8. In Prompt 7 and 8, the weighted sum of
the analytic scores constitutes the total score, which is the target of
prediction by our models. ASAP++ includes analytic scores of essays for Prompt
1-6. We developed AES models that predict the holistic score for each essay
prompt. From analytic scores, we only used ones related to
grammar777Conventions for Prompt 1, 2, 7, and 8. Language for Prompt 3-6..

We evaluated the scoring performance of our models using the Quadratic
Weighted Kappa (QWK) on the ASAP dataset. Following the previous studies, we
adopted 5-fold cross validation with 60/20/20 split for train, development,
and test sets, which was provided by Taghipour and Ng (2016).

###  4.2 Settings

As explained in Section 3.2, we developed our AES models based on BERT. We
used bert-base-uncased provided by Hugging
Face888https://github.com/huggingface/transformers. The maximum input length
was set to 512.

We normalized essay scores in the range of $[-1,1]$. The mean squared error
(MSE) loss was employed for both the main and auxiliary tasks. We updated the
parameters for the FFNN and the BERT layers. The number of hidden layers in
the FFNN for grammatical features was set to 3, and the number of the nodes in
the hidden layer to one-half the dimension of the grammatical features. The
number of hidden layers in the FFNN on the top was set to
$\\{1,2,3,4,5,7,10\\}$ for cat, $\\{1,2,3\\}$ for net, and $\\{2,3\\}$ for
multi and dual and we chose the value that achieved the best QWK score on the
development set for Prompt 1. The number of the nodes was set to 512. For both
FFNNs, we adopted relu as the activation function and set the dropout ratio to
0.2. In multi and dual, we tried $\\{0.8,0.6\\}$ for the weights of the loss
function for the main task. We used Adam optimizer (Kingma and Ba, 2015) with
a learning rate of 1e-5. We trained models with the batch size
$\\{4,8,16,32\\}$ for 10 epochs. In the following sections, we report the
scores on test sets for the batch size with the highest QWK on the development
set for each essay prompt. The scores are the average of three experiments
with different seed values.

##  5 Results

###  5.1 Hyperparameters for Each Model Architecture

| # of hidden layers
---|---
Model | 1 | 2 | 3 | 4 | 5 | 7 | 10
baseline | .813 | – | – | – | – | – | –
cat | .792 | .825 | .814 | .813 | .801 | .766 | .722
net | .812 | .824 | .817 | – | – | – | –
multi–hum (0.8) | – | .819 | .827 | – | – | – | –
multi–hum (0.6) | – | .804 | .812 | – | – | – | –
dual–hum (0.8) | – | .816 | .824 | – | – | – | –
dual–hum (0.6) | – | .820 | .819 | – | – | – | –
Table 2: Comparison of the number of hidden layers in FFNN on the top
(type256, Prompt 1, QWK dev)

Using type256 for the grammar features, we searched for the optimal
hyperparameters for each model architecture. Table 2 shows the QWK results on
the development set of Prompt 1 when we changed the number of hidden layers in
the FFNN on the top. When the number of hidden layers was set to 1 in cat, QWK
was lower than the baseline (.792 vs. .813). QWK became the highest when the
number of hidden layers was set to 2, while it got lower as the number of
hidden layers increased. In net, the architecture with 2 hidden layers
achieved the highest QWK. In both multi--hum and dual--hum999“–hum” represents
that grammar scores assigned by human raters were used. “–irt” is added when
grammar abilities estimated using IRT are used., QWK became the highest when
the number of hidden layers was set to 3 and the weight of the loss for the
main task to 0.8. In the subsequent experiments, we trained models using these
hyperparameters.

###  5.2 Comparison among Model Architectures

| Prompt |
---|---|---
Model | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | avg.
baseline | .799 | .662 | .662 | .804 | .801 | .809 | .821 | .726 | .760
\+ type256 |  |  |  |  |  |  |  |  |
cat | .819 | .674 | .675 | .801 | .809 | .809 | .830 | .721 | .767
net | .814 | .679 | .678 | .810 | .806 | .806 | .831 | .737 | .770
multi–hum | .816 | .678 | .683 | .812 | .810 | .811 | .830 | .746 | .773
dual–hum | .818 | .673 | .687 | .819 | .807 | .813 | .833 | .750 | .775
Table 3: Comparison among model architectures (type256, QWK test)

Using type256 for the grammar features, we compared the model performance
among the four model architectures. Table 3 shows the QWK results on the test
set of all prompts. By using type256, the average QWK score for all essays
improved in all proposed models, compared to the baseline (See avg. in Table
3).

In cat, however, the QWK scores did not improve in three prompts (Prompt 4, 6,
and 8), which suggests that simple concatenation of essay representations and
grammatical features was not sufficient enough to take advantage of the
information that the grammatical features have. In net, only Prompt 6 did not
improve from the baseline, and it seems effective to feed the grammatical
features to an FFNN before concatenating with essay representations.

The QWK scores for the models with the auxiliary task (multi--hum and dual--
hum) were higher than the others. Even when looking at the QWK scores for each
essay prompt individually, the scores improved for all prompts. These results
suggest that multi-task learning with grammar scores is effective to take
advantage of grammatical features.

Dual--hum achieved the best performance among the four proposed architectures.
In dual--hum, grammar scores were predicted from the final layer of the FFNN
for grammatical features (see Figure 1), which might let the model learn
better representations for grammatical features.

Since the dual--hum model performed the best, we conducted the subsequent
experiments using the setting.

###  5.3 Comparison of Grammatical Features

| Prompt |
---|---|---
Features | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | avg.
baseline | .799 | .662 | .662 | .804 | .801 | .809 | .821 | .726 | .760
multi–ffnn1 | .803 | .680 | .659 | .797 | .802 | .806 | .827 | .723 | .762
multi–ffnn3 | .812 | .671 | .684 | .812 | .805 | .812 | .831 | .748 | .772
type256 | .818 | .673 | .687 | .819 | .807 | .813 | .833 | .750 | .775
| (\+ .019) | (+ .011) | (\+ .025) | (+ .015) | (+ .006) | (+ .004) | (\+ .012) | (\+ .024) | (+ .015)
err54 | .815 | .672 | .689 | .813 | .805 | .812 | .832 | .756 | .774
| (+ .016) | (+ .010) | (\+ .027) | (+ .009) | (+ .004) | (+ .003) | (+ .011) | (\+ .030) | (+ .014)
type256 + err54 | .821 | .673 | .689 | .815 | .810 | .805 | .834 | .752 | .775
| (\+ .022) | (+ .011) | (\+ .027) | (+ .011) | (+ .009) | (- .004) | (\+ .013) | (\+ .026) | (+ .015)
multiply_b | .811 | .680 | .701 | .818 | .813 | .821 | .829 | .759 | .779
| (+ .012) | (+ .018) | (\+ .039) | (+ .014) | (+ .012) | (+ .012) | (+ .008) | (\+ .033) | (+ .019)
prob | .820 | .661 | .682 | .813 | .807 | .808 | .834 | .752 | .772
| (\+ .021) | (- .001) | (\+ .020) | (+ .009) | (+ .006) | (- .001) | (\+ .013) | (\+ .026) | (+ .012)
multiply_prob | .826 | .662 | .678 | .815 | .813 | .809 | .827 | .746 | .772
| (\+ .027) | ($\pm$ 0) | (+ .016) | (+ .011) | (+ .012) | ($\pm$ 0) | (+ .006) | (\+ .020) | (+ .012)
add_prob | .812 | .674 | .682 | .806 | .799 | .812 | .827 | .757 | .771
| (+ .013) | (+ .012) | (\+ .020) | (+ .002) | (- .002) | (+ .003) | (+ .006) | (\+ .031) | (+ .011)
Yang et al. (2020) | .817 | .719 | .698 | .845 | .841 | .847 | .839 | .744 | .794
| (+ .017) | (\+ .040) | (+ .019) | (\+ .023) | (\+ .038) | (\+ .050) | (+ .004) | (+ .019) | (\+ .026)
Cao et al. (2020) | .824 | .699 | .726 | .859 | .822 | .828 | .840 | .726 | .791
| (- .002) | (+ .001) | (+ .017) | (\+ .037) | (- .002) | (- .001) | (+ .011) | (- .017) | (+ .006)
Wang et al. (2022) | .834 | .716 | .714 | .812 | .813 | .836 | .839 | .766 | .791
Table 4: Comparison among grammatical features (dual–hum, QWK test). The
numbers in parentheses indicate the improvement from the baseline. The numbers
in parentheses for Yang et al. (2020) and Cao et al. (2020) are the
improvement from their baseline, which is equivalent to ours (RegressionOnly
and BERT (individual), respectively; n/a for Wang et al. (2022)).

Using the dual--hum setting, we compared the effectiveness of different
grammatical features. Table 4 shows the QWK results on the test sets when we
trained models using different grammatical features.101010The QWK results for
the auxiliary task are shown in Appendix A.

##### PFs and NFs

In the previous section, we showed that positive linguistic features (PFs;
type256) improved the AES performance. From the Table 4, we can see that
negative linguistic features (NFs; err54) also improved the model performance
(see avg.). Even on a per-prompt basis, the QWK scores were higher for all
prompts than those in the baseline.

Combining the PFs and the NFs (type256 + err54) also resulted in an
improvement in AES performance. However, the average QWK score (.775) was
almost same as that for type256 and err54, and no synergistic effect was
observed by using both the PFs and the NFs. We just concatenated the vectors
of the two features before feeding the features to the FFNN for grammatical
features, and there might be more effective ways to combine them.

##### PFs weighted using IRT parameters

We further explored the effectiveness of PFs by weighting them using IRT
parameters (see Section 3.1.2). When we considered the difficulties of
individual grammatical items (multiply_b), the QWK score became the highest
among all settings. On the other hand, modifying type256 with the probability
that a writer with a certain grammar ability uses the grammatical item
correctly did not help to improve AES performance. Although the QWK scores got
higher than that for the baseline, they were lower than that for type256. The
results suggest that it is more important to capture what items the writer
actually used in the essay than what items the writer seemed able to use.

##### Effect of Grammatical Features

To verify that the score improvement came from the addition of grammatical
features rather than multi-task learning, we trained models with the auxiliary
task but without using grammatical features. The number of hidden layers in
the FFNN on the top was set to 1 (multi--ffnn1; same as the baseline) and 3
(multi--ffnn3; the best parameter for multi--hum; see Section 5.1). The QWK
scores for multi--ffnn1 and multi--ffnn3 were higher than that of the
baseline, but lower than those of the models with grammatical features (Table
4). The results show that both multi-task learning and grammatical features
contributed to improve the model performance. In addition, the significant
improvement on multi--ffnn3 suggests that adding layers on the top of BERT
would be effective in multi-task learning.

##### Scoring examples

We show some examples from the fold 2 of Prompt 1 (Table 5). The true scores
of the four examples are 10, and are written in roughly the same number of
words.

In ID 1382, a relatively wide variety of grammatical items were used (10.18
items per 100 words, while the average for essays with true score of 10
included in the fold 2 test set was 9.86). The model trained using type256
captured the characteristic and predicted the correct score.

On the other hand, for ID 377 and 104, the model trained using type256
assigned lower scores than the baseline because of the limited variety of
grammatical items in the essays. Note that the prediction improved in ID 377,
while it got worse in ID 104.

In ID 1097, our model did not perform well. Although a relatively wide variety
of grammatical items were used, the predicted score was lower than that of the
baseline.111111See Appendix B for the confusion matrix on all the data points.

|  | Grammatical items | Predicted score
---|---|---|---
Essay ID | # words | # type | per 100 | baseline | type256
1382 | 442 | 45 | 10.18 | 9 | 10
377 | 480 | 47 | 9.79 | 12 | 11
104 | 405 | 38 | 9.38 | 9 | 8
1097 | 421 | 42 | 9.98 | 9 | 8
Table 5: Scoring examples. The true scores of the four examples are 10. Per
100 represents the number of different grammar items used per 100 words.

##### Comparison with existing models

The QWK scores for the state-of-the-art AES models are also shown in Table 4.
The average QWK score of our models (the highest at .779) was not as high as
those of the existing models. In some prompts, there seemed to be differences
in baseline QWK scores between the previous studies and ours, and we made
comparisons based on the improvement from each baseline121212We re-implemented
$\text{R}^{2}$ BERT (Yang et al., 2020), but our re-implementation of the
model did not achieve as good scores as those reported in their paper.
Furthermore, we trained models using grammatical features with the loss
combination proposed by them (_i.e._ , regression and ranking loss), which
resulted in lower QWK scores than our baseline..

In Prompt 1, 3, 7, and 8, our proposed models showed a greater improvement in
the QWK scores compared to Yang et al. (2020) and Cao et al. (2020). In these
four prompts, the scores themselves of our models were also competitive with
those of the existing models. Cao et al. (2020) achieved the state-of-the-art
results in Prompt 3, 4, and 7, but the improvements from their baselines were
relatively small in the other prompts.

However, our proposed methods were less effective for Prompt 2, 4, 5, and 6,
which resulted in lower average QWK scores than the existing models. To
identify when the proposed methods were effective, we examined the
characteristics of the essays, such as the type of essays, the average number
of words in essays, the correlation coefficient between holistic scores and
the grammar ability parameter $\theta$ and between human-annotated grammar
scores and $\theta$, and the variance of $\theta$, but none of them could
provide a satisfactory explanation. We need further investigation and it might
help to improve the performance on the prompts where our methods were less
effective.

###  5.4 Using the IRT Ability Parameter for the Auxiliary Task

| Prompt |
---|---|---
Model | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | avg.
multi-irt | .819 | .669 | .697 | .811 | .813 | .821 | .839 | .757 | .778
dual-irt | .805 | .678 | .686 | .807 | .808 | .816 | .831 | .742 | .772
Table 6: QWK results of the models using the IRT ability parameter for the
auxiliary task (QWK test)

In Section 5.2, we demonstrated that dual--hum model achieved the best
performance among the four proposed architectures. However, the architecture
requires grammar scores annotated by human raters. Therefore we employed
grammar abilities estimated using IRT, which requires no human-annotated
labels, as the teacher signals.

Table 6 shows that multi--irt and dual--irt models achieved comparable
performance to the models that used human-annotated score. In general,
analytical scoring is more time-consuming than holistic scoring, and grammar
scores, which are one of the analytical scores, are not always available in a
dataset. A method that improves AES performance without the additional human-
annotated labels has practical value. Another advantage of using IRT for our
AES models is that we can provide the characteristics of grammatical items
(_i.e._ , discrimination and difficulty) as well as essay scores.

##  6 Conclusions

This study examined the effectiveness of using grammatical features in AES
models. Specifically, we fed two kinds of features: (1) grammatical items that
writers used correctly in essays (PFs), and (2) the number of grammatical
errors (NFs). We showed that both PFs and NFs improved the model performance,
but combining them did not result in further improvement. The experimental
results suggest that multi-task learning would be effective to take advantage
of the information that the grammatical features have. One of the future
directions could be exploring effective ways to combine PFs and NFs to improve
the model performance since the way in this study was a simple concatenation
of the two vectors (_e.g._ , to learn representations for PFs and NFs in
different networks and combine them). Another direction would be to examine
the effectiveness of adding our grammatical features in AES using a large
language model. It potentially improves the scoring performance in zero-
and/or few-shot settings Mizumoto and Eguchi (2023). Furthermore, in order to
have more interpretable models, it would be beneficial to analyze how much
individual grammatical features contribute to model’s score prediction. The
insights delivered by interpretable models can help practitioners in
education.

We also weighted PFs in several ways using IRT parameters and found that
considering the difficulties of grammatical items would improve the model
performance. In addition, we used the ability parameter $\theta$ as teacher
signals for the auxiliary task in multi-task learning. Although no human-
annotated labels are required to estimate the IRT parameters, the model
trained with the ability parameter achieved comparable performance to the
model trained with grammar scores annotated by human raters. In this study,
IRT parameters were estimated based on grammatical items that writers used in
their essays. In the future, we will apply IRT to both PFs and NFs to model
writers’ grammar abilities.

##  7 Limitations

Our proposed methods showed significant advantage on some essay prompts in the
ASAP dataset, while they were less effective on the other prompts. Further
investigation is necessary to clarify what kind of essays our proposed methods
would be effective to. An analysis of the effectiveness of grammatical
features on different prompts will also provide additional insights into the
variation of model behavior across different prompts.

There are also some limitations related to the extraction of grammatical
features. First, the toolkit provided by the CEFR-J Grammar Profile extracts
grammatical items based on sophisticated regular expression patterns, which
was written by a linguist. It would be quite challenging to prepare a similar
toolkit in other languages. Bannò and Matassoni (2022) let a model predict the
frequencies of grammatical errors from essay representations, which can be
applicable to PFs, but the approach requires human-annotated labels to train a
model. Another approach is to extract grammatical features based on cross-
linguistically consistent annotations such as Universal Dependencies. It makes
easier to use grammatical features in other languages, while it remains
challenging to extract ones related to parts of speech and/or morphological
features rather than dependencies (_e.g._ , present perfect in English).

Second, there could be errors in the extraction using regular expressions and
the same is true for grammatical error correction. Experiments using
grammatical features annotated by humans would help reveal the influence of
errors in feature extraction.

Third, our method requires explicitly extracting grammatical features at test
time as well as at training time. An alternative would be to develop a multi-
task learning framework where a model is trained to reconstruct grammatical
features at training time and then run the trained model on unparsed test data
(_e.g._ , Andersen et al., 2021).

## Acknowledgments

A part of this work was supported by JSPS KAKENHI Grant Numbers JP21H05054 and
by JST, the establishment of university fellowships towards the creation of
science technology innovation, Grant Number JPMJFS2137.

## References

  * Alikaniotis et al. (2016) Dimitrios Alikaniotis, Helen Yannakoudakis, and Marek Rei. 2016.  Automatic text scoring using neural networks.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 715–725, Berlin, Germany. Association for Computational Linguistics.
  * Andersen et al. (2021) Øistein E Andersen, Zheng Yuan, Rebecca Watson, and Kevin Yet Fong Cheung. 2021.  Benefits of alternative evaluation methods for automated essay scoring.  In _Proceedings of the 14th International Conference on Educational Data Mining (EDM21)_ , pages 856–864.
  * Aomi et al. (2021) Itsuki Aomi, Emiko Tsutsumi, Masaki Uto, and Maomi Ueno. 2021.  Integration of automated essay scoring models using item response theory.  In _Artificial Intelligence in Education: 22nd International Conference, AIED 2021, Utrecht, The Netherlands, June 14–18, 2021, Proceedings, Part II_ , pages 54–59.
  * Bannò and Matassoni (2022) Stefano Bannò and Marco Matassoni. 2022.  Cross-corpora experiments of automatic proficiency assessment and error detection for spoken English.  In _Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022)_ , pages 82–91, Seattle, Washington. Association for Computational Linguistics.
  * Bryant et al. (2017) Christopher Bryant, Mariano Felice, and Ted Briscoe. 2017.  Automatic annotation and evaluation of error types for grammatical error correction.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 793–805, Vancouver, Canada. Association for Computational Linguistics.
  * Burstein et al. (2004) Jill Burstein, Martin Chodorow, and Claudia Leacock. 2004.  Automated essay evaluation: The criterion online writing service.  _AI Magazine_ , 25(3):27–36.
  * Cao et al. (2020) Yue Cao, Hanqi Jin, Xiaojun Wan, and Zhiwei Yu. 2020.  Domain-adaptive neural automated essay scoring.  In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , page 1011–1020.
  * Caruana (1997) Rich Caruana. 1997.  Multitask learning.  _Machine Learning_ , 28:41–75.
  * Council of Europe (2001) Council of Europe. 2001.  _Common European Framework of Reference for Languages: Learning, Teaching, Assessment_.  Cambridge University Press, Cambridge.
  * Craighead et al. (2020) Hannah Craighead, Andrew Caines, Paula Buttery, and Helen Yannakoudakis. 2020.  Investigating the effect of auxiliary objectives for the automated grading of learner English speech transcriptions.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2258–2269, Online. Association for Computational Linguistics.
  * Cummins and Rei (2018) Ronan Cummins and Marek Rei. 2018.  Neural multi-task learning in automated assessment.  _arXiv_ , arXiv: 1801.06830.
  * Cummins et al. (2016) Ronan Cummins, Meng Zhang, and Ted Briscoe. 2016.  Constrained multi-task learning for automated essay scoring.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 789–799, Berlin, Germany. Association for Computational Linguistics.
  * Dasgupta et al. (2018) Tirthankar Dasgupta, Abir Naskar, Lipika Dey, and Rupsa Saha. 2018.  Augmenting textual qualitative features in deep convolution recurrent neural network for automatic essay scoring.  In _Proceedings of the 5th Workshop on Natural Language Processing Techniques for Educational Applications_ , pages 93–102, Melbourne, Australia. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Ding et al. (2023) Yuning Ding, Marie Bexte, and Andrea Horbach. 2023.  Score it all together: A multi-task learning study on automatic scoring of argumentative essays.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 13052–13063, Toronto, Canada. Association for Computational Linguistics.
  * Elks (2021) Tim Elks. 2021.  Using transfer learning to automatically mark L2 writing texts.  In _Proceedings of the Student Research Workshop Associated with RANLP 2021_ , pages 51–57, Online. INCOMA Ltd.
  * Hambleton et al. (1991) Ronald K. Hambleton, Hariharan Swaminathan, and H. Jane Rogers. 1991.  _Fundamentals of Item Response Theory_.  SAGE Publications, California.
  * Hawkins and Buttery (2009) John A Hawkins and Paula Buttery. 2009.  Using learner language from corpora to profi le levels of profi ciency: insights from the english profi le programme.  In Lynda Taylor and Cyril J Weir, editors, _Language Testing Matters Investigating the Wider Social and Educational Impact of Assessment - Proceedings of the ALTE Cambridge Conference April 2008_ , pages 158–175. Cambridge University Press.
  * Hawkins and Filipović (2012) John A Hawkins and Luna Filipović. 2012.  _Criterial Features in L2 English: Specifying the Reference Levels of the Common European Framework_.  Cambridge University Press, Cambridge.
  * Housen et al. (2012) Alex Housen, Folkert Kuiken, and Ineke Vedder, editors. 2012.  _Dimensions of L2 Performance and Proficiency: Complexity, Accuracy and Fluency in SLA_.  John Benjamins, Amsterdam.
  * Hussein et al. (2020) Mohamed A. Hussein, Hesham A. Hassan, and Mohammad Nassef. 2020.  A trait-based deep learning automated essay scoring system with adaptive feedback.  _International Journal of Advanced Computer Science and Applications_ , 11(5):287–293.
  * Ishii and Tono (2018) Yasutake Ishii and Yukio Tono. 2018.  Investigating japanese EFL learners’ overuse/underuse of english grammar categories and their relevance to CEFR levels.  In _Proceedings of the 4th Asia Pacific Corpus Linguistics Conference (APCLC 2018)_ , pages 160–165.
  * Ke and Ng (2019) Zixuan Ke and Vincent Ng. 2019.  Automated essay scoring: A survey of the state of the art.  In _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence_ , pages 33–40.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _Proceedings of the 3rd International Conference on Learning Representations_.
  * Kuiken (2023) Folkert Kuiken. 2023.  Linguistic complexity in second language acquisition.  _Linguistics Vanguard_ , 9(s1):83–93.
  * Lord (1952) Frederic M. Lord. 1952.  _A theory of test scores (Psychometric Monograph No. 7)_.  Psychometric Society, Iowa City.
  * Lord (1980) Frederic M. Lord. 1980.  _Applications of Item Response Theory To Practical Testing Problems_.  Routledge, New York.
  * Mathias and Bhattacharyya (2018) Sandeep Mathias and Pushpak Bhattacharyya. 2018.  ASAP++: Enriching the ASAP automated essay grading dataset with essay attribute scores.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , Miyazaki, Japan. European Language Resources Association (ELRA).
  * Mathias and Bhattacharyya (2020) Sandeep Mathias and Pushpak Bhattacharyya. 2020.  Can neural networks automatically score essay traits? In _Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 85–91, Seattle, WA, USA → Online. Association for Computational Linguistics.
  * Mim et al. (2019) Farjana Sultana Mim, Naoya Inoue, Paul Reisert, Hiroki Ouchi, and Kentaro Inui. 2019.  Unsupervised learning of discourse-aware text representation for essay scoring.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop_ , pages 378–385, Florence, Italy. Association for Computational Linguistics.
  * Mizumoto and Eguchi (2023) Atsushi Mizumoto and Masaki Eguchi. 2023.  Exploring the potential of using an ai language model for automated essay scoring.  _Research Methods in Applied Linguistics_ , 2(2):100050.
  * Muangkammuen and Fukumoto (2020) Panitan Muangkammuen and Fumiyo Fukumoto. 2020.  Multi-task learning for automated essay scoring with sentiment analysis.  In _Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing: Student Research Workshop_ , pages 116–123, Suzhou, China. Association for Computational Linguistics.
  * Nadeem et al. (2019) Farah Nadeem, Huy Nguyen, Yang Liu, and Mari Ostendorf. 2019.  Automated essay scoring with discourse-aware neural models.  In _Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 484–493, Florence, Italy. Association for Computational Linguistics.
  * Ridley et al. (2021) Robert Ridley, Liang He, Xin-yu Dai, Shujian Huang, and Jiajun Chen. 2021.  Automated cross-prompt scoring of essay traits.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 35(15):13745–13753.
  * Shibata and Uto (2022) Takumi Shibata and Masaki Uto. 2022.  Analytic automated essay scoring based on deep neural networks integrating multidimensional item response theory.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 2917–2926, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  * Taghipour and Ng (2016) Kaveh Taghipour and Hwee Tou Ng. 2016.  A neural approach to automated essay scoring.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 1882–1891, Austin, Texas. Association for Computational Linguistics.
  * Tarnavskyi et al. (2022) Maksym Tarnavskyi, Artem Chernodub, and Kostiantyn Omelianchuk. 2022.  Ensembling and knowledge distilling of large sequence taggers for grammatical error correction.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3842–3852, Dublin, Ireland. Association for Computational Linguistics.
  * Uto (2021) Masaki Uto. 2021.  A multidimensional generalized many-facet rasch model for rubric-based performance assessment.  _Behaviormetrika_ , 48:425–457.
  * Uto et al. (2023) Masaki Uto, Itsuki Aomi, Emiko Tsutsumi, and Maomi Ueno. 2023.  Integration of prediction scores from various automated essay scoring models using item response theory.  _IEEE Transactions on Learning Technologies_ , 16(6):983–1000.
  * Uto and Okano (2021) Masaki Uto and Masashi Okano. 2021.  Learning automated essay scoring models using item-response-theory-based scores to decrease effects of rater biases.  _IEEE Transactions on Learning Technologies_ , 14(6):763–776.
  * Uto et al. (2020) Masaki Uto, Yikuan Xie, and Maomi Ueno. 2020.  Neural automated essay scoring incorporating handcrafted features.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 6077–6088, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Vajjala (2018) Sowmya Vajjala. 2018.  Automated assessment of Non-Native learner essays: Investigating the role of linguistic features.  _International Journal of Artificial Intelligence in Education_ , 28(1):79–105.
  * Wang et al. (2022) Yongjie Wang, Chuang Wang, Ruobing Li, and Hui Lin. 2022.  On the use of bert for automated essay scoring: Joint learning of multi-scale essay representation.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3416–3425, Seattle, United States. Association for Computational Linguistics.
  * Weigle (2002) Sara Cushing Weigle. 2002.  _Assessing Writing_.  Cambridge University Press, Cambridge.
  * Wolfe-Quintero et al. (1998) Kate Wolfe-Quintero, Shunji Inagaki, and Hae-Young Kim. 1998.  _Second language development in writing : measures of fluency, accuracy, & complexity_.  University of Hawai’i Press, Honolulu.
  * Xue et al. (2021) Jin Xue, Xiaoyi Tang, and Liyan Zheng. 2021.  A hierarchical bert-based transfer learning approach for multi-dimensional essay scoring.  _IEEE Access_ , 9:125403–125415.
  * Yancey et al. (2023) Kevin P. Yancey, Geoffrey Laflair, Anthony Verardi, and Jill Burstein. 2023.  Rating short L2 essays on the CEFR scale with GPT-4.  In _Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023)_ , pages 576–584, Toronto, Canada. Association for Computational Linguistics.
  * Yang et al. (2020) Ruosong Yang, Jiannong Cao, Zhiyuan Wen, Youzheng Wu, and Xiaodong He. 2020.  Enhancing automated essay scoring performance via fine-tuning pre-trained language models with combination of regression and ranking.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1560–1569, Online. Association for Computational Linguistics.
  * Yannakoudakis et al. (2011) Helen Yannakoudakis, Ted Briscoe, and Ben Medlock. 2011.  A new dataset and method for automatically grading ESOL texts.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 180–189, Portland, Oregon, USA. Association for Computational Linguistics.

| Prompt |
---|---|---
Model | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | avg.
type256 | 0.032 | -0.007 | 0.050 | -0.007 | 0.001 | 0.000 | 0.000 | 0.079 | 0.016
err54 | -0.002 | 0.017 | -0.003 | 0.014 | -0.007 | 0.003 | -0.012 | 0.045 | 0.008
type256+err54 | 0.148 | 0.003 | 0.085 | 0.001 | 0.000 | 0.001 | -0.002 | 0.110 | 0.039
multiply_b | 0.015 | -0.003 | -0.002 | -0.023 | 0.000 | 0.000 | 0.012 | -0.003 | -0.001
prob | 0.052 | -0.025 | 0.028 | 0.000 | 0.000 | 0.000 | 0.000 | 0.046 | 0.008
multiply_prob | 0.097 | 0.000 | 0.003 | 0.000 | 0.000 | 0.000 | 0.007 | 0.059 | 0.018
add_prob | 0.053 | -0.023 | 0.050 | -0.002 | 0.004 | 0.000 | 0.039 | 0.004 | 0.011
Table 7: QWK results for the auxiliary task on the test set (models shown in
Table 4)

##  Appendix A QWK Results for the Auxiliary Task

Table 7 shows the QWK score for the auxiliary task (_i.e._ , predicting
grammar score). The QWK scores were generally low, and some of them were
negative. We observed that the models output scores close to the mode value in
each prompt. One of the possible reasons is the relatively low weight for loss
function for the auxiliary task (_i.e._ , 0.2). However, when we assigned a
higher weight for the auxiliary task (_i.e._ , 0.4), the model prediction for
the main task got worse. Further consideration is necessary for predicting
multiple essay traits simultaneously (_e.g._ , Ridley et al., 2021; Shibata
and Uto, 2022).

##  Appendix B Detailed Results of Model Predictions

Detailed scoring performance of the model trained using type256 is shown in
Figure 2. The values in the confusion matrices are the sum of all experiments
(_i.e._ , 5-fold cross validation and three experiments with different seed
values).

Figure 2: Scoring performance of the model trained using type256

Figure 2: Scoring performance of the model trained using type256 (_cont._)


# A Heintze-Karcher type inequality in hyperbolic space

Yingxiang Hu  School of Mathematical Sciences, Beihang University, Beijing
100191, P.R. China<EMAIL_ADDRESS>,  Yong Wei  and  Tailong Zhou
School of Mathematical Sciences, University of Science and Technology of
China, Hefei 230026, P.R. China<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract.

In this paper, we prove a new Heintze-Karcher type inequality for shifted mean
convex hypersurfaces in hyperbolic space. As applications, we prove an
Alexandrov type theorem for closed embedded hypersurfaces with constant
shifted $k$th mean curvature in hyperbolic space. Furthermore, a uniqueness
result for $h$-convex hypersurfaces satisfying certain curvature equations is
obtained.

###### Key words and phrases:

Heintze-Karcher inequality, shifted principal curvatures, unit normal flow,
hyperbolic space

##  1\. Introduction

The classical Alexandrov’s theorem [Alexandrov1956] states that any closed,
embedded hypersurface in Euclidean space with constant mean curvature is a
round sphere. Alexandrov’s method is based on the maximum principle for
elliptic equations. This result essentially improves previous results due to
Süss [Suss1952] and Hsiung [Hsiung1954], which require the additional
assumption of convexity or star-shapedness on the hypersurface. Later, Reilly
[Reilly1977] provided a new proof of the Alexandrov’s theorem by using his
famous integral formula. Another proof of Alexandrov’s theorem was also
provided by Montiel and Ros [Montiel-Ros1991], by combining the Minkowski
formulae with the following Heintze-Karcher inequality.

######  Theorem A.

Let $\Omega$ be a bounded domain with smooth boundary $\Sigma=\partial\Omega$
in Euclidean space $\mathbb{R}^{n+1}$. Assume that the mean curvature $H$ of
$\Sigma$ is positive, then

$\displaystyle\int_{\Sigma}\frac{1}{H}d\mu\geq\frac{n+1}{n}\operatorname{Vol}(\Omega).$
(1.1)

Equality holds in (1.1) if and only if $\Sigma$ is umbilic.

The proof of Theorem A is inspired by a direct integral method due to Heintze
and Karcher [Heintze-Karcher1978], using the unit normal flow in Euclidean
space. Montiel and Ros [Montiel-Ros1991] also proved an analogue of the
Alexandrov’s theorem in hyperbolic space and in the open hemisphere. However,
it remains unknown for a long time whether the Heintze-Karcher inequality
(1.1) can be generalized to space forms. A breakthrough in this direction was
made by Brendle [Bre13], who utilized the unit normal flow with respect to a
conformal metric to establish the Heintze-Karcher type inequality in certain
warped product manifolds. In particular, Brendle proved the following result.

######  Theorem B.

[Bre13] Let $\Omega$ be a bounded domain with smooth boundary
$\Sigma=\partial\Omega$ in hyperbolic space $\mathbb{H}^{n+1}$. Fix a point
$o\in\mathbb{H}^{n+1}$ and $V=\cosh r(x)$, where $r(x)=d(x,o)$ is the distance
to $o$. Assume that the mean curvature $H$ of $\Sigma$ is positive, then

$\displaystyle\int_{\Sigma}\frac{V}{H}d\mu\geq\frac{n+1}{n}\int_{\Omega}Vd\operatorname{vol}.$
(1.2)

Equality holds in (1.2) if and only if $\Sigma$ is umbilic.

This inequality (1.2) was also proved later using the generalized Reilly’s
formula, see [Qiu-Xia2015, Li-Xia2019].

The main purpose of this paper is to establish the following new Heintze-
Karcher type inequality for hypersurfaces with mean curvature $H>n$ in
hyperbolic space.

######  Theorem 1.1.

Let $\Omega$ be a bounded domain with smooth boundary $\Sigma=\partial\Omega$
in hyperbolic space $\mathbb{H}^{n+1}$. Fix a point $o\in\mathbb{H}^{n+1}$ and
$V(x)=\cosh r(x)$, where $r(x)=d(o,x)$ is the distance to this point $o$.
Assume that the mean curvature of $\Sigma=\partial\Omega$ satisfies $H>n$,
then

$\displaystyle\int_{\Sigma}\frac{V-V_{,\nu}}{H-n}d\mu\geq\frac{n+1}{n}\int_{\Omega}Vd\operatorname{vol},$
(1.3)

where $V_{,\nu}=\langle\overline{\nabla}V,\nu\rangle=\langle\sinh
r\partial_{r},\nu\rangle$ is the support function of $\Sigma$ and $\nu$
denotes the unit outward normal of $\Sigma$. Equality holds in (1.3) if and
only if $\Sigma$ is umbilic.

For the curve case (i.e., $n=1$), the inequality (1.3) was proved by Li and Xu
[Li-Xu2022, Proposition 8.2]. However, it seems difficult to generalize their
proof to the hypersurfaces in $\mathbb{H}^{n+1}$ $(n\geq 2)$. Our proof of
Theorem 1.1 relies on the unit normal flow, whose solution is a family of
level sets $\\{v=t\\}$. Here $v(x)=d(x,\Sigma)$ is the distance from
$x\in\Omega$ to the hypersurface $\Sigma$. Since the distance function $v$ is
merely Lipschitz continuous, we define the following geometric quantity

$Q(t)=e^{(n+1)t}\left[\int_{\Sigma_{t}^{\ast}}\frac{V-V_{,\nu}}{H-n}d\mu-\frac{n+1}{n}\int_{\\{v>t\\}}Vd\operatorname{vol}\right].$

where $\Sigma_{t}^{\ast}$ is the smooth part of $\\{v=t\\}$ and $\\{v>t\\}$ is
the super-level set. Combining with the Minkowski type formula on
$\Sigma_{t}^{\ast}$ for a.e. $t\in(0,T)$ (Lemma 3.4), we prove that $Q(0)\geq
Q(t)$ for all $t\in(0,T)$ (Proposition 3.5). Finally, Theorem 1.1 follows from
the asymptotic estimate $\liminf_{t\rightarrow T}Q(t)\geq 0$.

Next, we will provide two applications of the Heintze-Karcher type inequality
(1.3) in Theorem 1.1. For this, we recall the definition of shifted $k$th mean
curvature. Given a smooth hypersurface $\Sigma$ in $\mathbb{H}^{n+1}$, we
denote its principal curvatures as $\kappa=(\kappa_{1},\cdots,\kappa_{n})$.
The shifted principal curvatures are defined as
$\tilde{\kappa}=(\widetilde{\kappa}_{1},\cdots,\widetilde{\kappa}_{n})=(\kappa_{1}-1,\cdots,\kappa_{n}-1)$.
Then the shifted $k$th mean curvature $E_{k}(\widetilde{\kappa})$ of $\Sigma$
is defined as the normalized $k$th elementary symmetric function of
$\tilde{\kappa}$. These definitions arise naturally in the context of
horospherically convex ($h$-convex) geometry of hypersurfaces in
$\mathbb{H}^{n+1}$ (see [ACW2021, Esp09]).

The first application of Theorem 1.1 is an Alexandrov-type theorem for smooth
hypersurface with constant shifted $k$th mean curvature in hyperbolic space.

######  Theorem 1.2.

Let $\Sigma$ be a smooth closed and embedded hypersurface in
$\mathbb{H}^{n+1}$. Let $k\in\\{2,\cdots,n\\}$. If the shifted $k$th mean
curvature $E_{k}(\widetilde{\kappa})$ is constant, then it is a geodesic
sphere.

For $k=1$, Theorem 1.2 reduces to the original Alexandrov theorem for smooth
embedded hypersurface with constant mean curvature in hyperbolic space
[Montiel-Ros1991].

For the second application of Theorem 1.1, we recall the following uniqueness
result for $h$-convex domains by Li and Xu [Li-Xu2022, Proposition 8.1].

######  Theorem C.

[Li-Xu2022] Let $n\geq 2$ and $k\in\\{1,\cdots,n\\}$. Assume that $\chi(s)$ is
a smooth, positive and monotone non-decreasing function defined on
$\mathbb{R}^{+}$. Let $\Omega$ be a smooth uniformly h-convex bounded domain
in $\mathbb{H}^{n+1}$ which contains the origin in its interior. If
$\Sigma=\partial\Omega$ satisfies the following equation

$\displaystyle E_{k}(\widetilde{\kappa})=\chi(V-V_{,\nu}),$

then it must be a geodesic sphere. Moreover, if $\chi$ is strictly increasing,
then it is a geodesic sphere centered at the origin.

Here we say that a smooth bounded domain in hyperbolic space is $h$-convex
(resp. uniformly $h$-convex), if the principal curvatures of its boundary
satisfy $\kappa_{i}\geq 1$ (resp. $\kappa_{i}\geq 1+\varepsilon$, for some
constant $\varepsilon>0$). The $h$-convex domains in hyperbolic space are very
natural geometric objects and have received much attentions in recent years,
see e.g. [BM99, CabezasRivas-Miquel2007, Wang-Xia2014, AW2018, ACW2021, Wang-
Wei-Zhou2022, HL22, HLW20].

We can use Theorem 1.1 to remove the condition “the origin lies in the
interior of $\Omega$” in the statement of Theorem C and obtain the following
uniqueness result.

######  Theorem 1.3.

Let $n\geq 2$. Assume that $\chi(s)$ is a smooth, positive and monotone non-
decreasing function defined on $\mathbb{R}^{+}$. Let $\Omega$ be a smooth
bounded and uniformly h-convex domain in $\mathbb{H}^{n+1}$. For each
$k=1,\cdots,n$, if $\Sigma=\partial\Omega$ satisfies

$\displaystyle E_{k}(\widetilde{\kappa})=\chi(V-V_{,\nu}),$ (1.4)

then $\Sigma$ is a geodesic sphere. Moreover, if $\chi$ is strictly
increasing, then it is a geodesic sphere centered at the origin.

We note that if the origin lies in the interior of $\Omega$, then the uniform
$h$-convexity of $\Sigma=\partial\Omega$ implies that $\Sigma$ is star-shaped
with respect to the origin. The star-shapedness property is crucial to
establish the uniqueness result in Theorem C, see [Li-Xu2022, p.97].

The paper is organized as follows. In Section 2, we give some preliminaries on
closed hypersurfaces in hyperbolic space. In Section 3, we use the unit normal
flow in hyperbolic space to prove Theorem 1.1. In Section 4, we give the
proofs of Theorems 1.2 and 1.3.

######  Acknowledgments.

This work is supported by National Key Research and Development Program of
China 2021YFA1001800 and 2020YFA0713100, and NSFC grant No.12101027.

##  2\. Preliminaries

In this section, we collect some preliminaries on elementary symmetric
functions and geometry of hypersurfaces in hyperbolic space.

###  2.1. Elementary symmetric functions

We first review some properties of elementary symmetric functions. See
[Guan14] for more details.

Let $\sigma_{m}:\mathbb{R}^{n}\rightarrow\mathbb{R}$ be the $m$th elementary
symmetric function defined by

$\displaystyle\sigma_{m}(\lambda)=\sum_{i_{1}<\cdots<i_{m}}\lambda_{i_{1}}\cdots\lambda_{i_{m}},$

where $\lambda=(\lambda_{1},\cdots,\lambda_{n})\in\mathbb{R}^{n}$. Similarly,
the normalized $m$th elementary symmetric function is defined by

$\displaystyle E_{m}(\lambda)=\frac{1}{\binom{n}{m}}\sigma_{m}(\lambda).$

We also set $\sigma_{0}(\lambda)=E_{0}(\lambda)=1$ and
$\sigma_{m}(\lambda)=E_{m}(\lambda)=0$ for $m>n$ by convention. For a
symmetric matrix $A=(A_{i}^{j})$, we set

$\sigma_{m}(A)=\frac{1}{m!}\delta_{j_{1}\cdots j_{m}}^{i_{1}\cdots
i_{m}}A_{i_{1}}^{j_{1}}\cdots A_{i_{m}}^{j_{m}},$

where $\delta_{j_{1}\cdots j_{m}}^{i_{1}\cdots i_{m}}$ is the generalized
Kronecker symbol.

If $\lambda(A)=(\lambda_{1}(A),\cdots,\lambda_{n}(A))$ are the eigenvalues of
$A$, then $E_{m}(A)=E_{m}(\lambda(A))$. The Gårding cone is defined by

$\Gamma_{m}^{+}=\\{\lambda\in\mathbb{R}^{n}:~{}\sigma_{i}(\lambda)>0,1\leq
i\leq m\\}.$

A symmetric matrix $A$ is said to belong to $\Gamma_{m}^{+}$ if its
eigenvalues $\lambda(A)\in\Gamma_{m}^{+}$. Denote by

$\displaystyle(\dot{E}_{m})_{i}^{j}(A)=\frac{\partial E_{m}(A)}{\partial
A_{j}^{i}}=\frac{m(n-m)!}{n!}\delta_{ii_{1}\cdots i_{m-1}}^{jj_{1}\cdots
j_{m-1}}A_{j_{1}}^{i_{1}}\cdots A_{j_{m-1}}^{i_{m-1}}.$

If $A$ is a diagonal matrix with
$\lambda(A)=(\lambda_{1},\cdots,\lambda_{n})$, then

$\displaystyle(\dot{E}_{m})_{i}^{j}(A)=\frac{\partial
E_{m}}{\partial\lambda_{i}}(\lambda)\delta_{i}^{j}.$

The following formulae for the elementary symmetric functions are well-known.

######  Lemma 2.1.

[Hardy-Littlewood-Polya1934, Reilly1973] We have

$\displaystyle\sum_{i,j}(\dot{E}_{m})_{i}^{j}(A)A_{j}^{i}=$ $\displaystyle
mE_{m}(A),$ (2.1)
$\displaystyle\sum_{i,j}(\dot{E}_{m})_{i}^{j}(A)\delta_{j}^{i}=$
$\displaystyle mE_{m-1}(A),$ (2.2)
$\displaystyle\sum_{i,j}(\dot{E}_{m})_{i}^{j}(A)(A^{2})_{j}^{i}=$
$\displaystyle nE_{1}(A)E_{m}(A)-(n-m)E_{m+1}(A),$ (2.3)

where $(A^{2})_{j}^{i}=\sum_{k}A_{j}^{k}A_{k}^{i}$.

######  Lemma 2.2.

[Guan14, Lemma 2.5] If $\lambda\in\Gamma_{m}^{+}$, then the following Newton-
MacLaurin inequality holds

$\displaystyle E_{m}(\lambda)\leq E_{1}(\lambda)E_{m-1}(\lambda).$ (2.4)

Equality holds if and only if $\lambda=cI$ for some constant $c>0$, where
$I=(1,\cdots,1)$.

###  2.2. Hypersurfaces in hyperbolic space

Fix a point $o\in\mathbb{H}^{n+1}$, we take $r(x)=d(o,x)$. Using the geodesic
polar coordinates with respect to $o$, the hyperbolic space $\mathbb{H}^{n+1}$
can be expressed as a warped product $[0,\infty)\times\mathbb{S}^{n}$ equipped
with the metric

$\overline{g}=dr^{2}+\lambda(r)^{2}g_{\mathbb{S}^{n}},$

where $g_{\mathbb{S}^{n}}$ is the round metric of $\mathbb{S}^{n}$ and
$\lambda(r)=\sinh r$. Then $\lambda^{\prime}(r)=\cosh r$.

Let $\Sigma$ be a smooth hypersurface in $\mathbb{H}^{n+1}$ with unit outward
normal $\nu$. Let $e_{1},\cdots,e_{n}$ be an orthonormal basis of the tangent
space of $\Sigma$. Denote $\overline{\nabla}$ as the Levi-Civita connection on
$\mathbb{H}^{n+1}$. Then the induced metric $g$ of $\Sigma$ is
$g_{ij}=\overline{g}(e_{i},e_{j})$, and the second fundamental form
$h=(h_{ij})$ of $\Sigma$ in $\mathbb{H}^{n+1}$ is given by

$\displaystyle
h_{ij}=h(e_{i},e_{j})=\bar{g}(\overline{\nabla}_{e_{i}}\nu,e_{j}).$

The principal curvatures $\kappa=(\kappa_{1},\cdots,\kappa_{n})$ of $\Sigma$
are the eigenvalues of the Weingarten matrix $(h_{i}^{j})=(g^{jk}h_{ki})$,
where $(g^{ij})$ is the inverse matrix of $(g_{ij})$. The mean curvature of
$\Sigma$ is defined as

$H=g^{ij}h_{ij}=\sum_{i=1}^{n}\kappa_{i}.$

Without loss of generality, we assume that $e_{1},\cdots,e_{n}$ are the
corresponding principal directions with respect to the principal curvatures
$\kappa_{1},\cdots,\kappa_{n}$.

######  Lemma 2.3.

Let $\Sigma$ be a smooth hypersurface in $\mathbb{H}^{n+1}$. Denote $V=\cosh
r$ and
$V_{,\nu}=\langle\overline{\nabla}V,\nu\rangle=\langle\lambda\partial_{r},\nu\rangle$.
Then we have

$\displaystyle\nabla_{i}V=\langle\overline{\nabla}V,e_{i}\rangle=\langle\lambda(r)\partial_{r},e_{i}\rangle,\quad\nabla_{i}V_{,\nu}=\langle\lambda\partial_{r},h_{i}^{j}e_{j}\rangle,$
(2.5)

and

$\displaystyle\nabla_{i}\nabla_{j}V=Vg_{ij}-V_{,\nu}h_{ij},$ (2.6)
$\displaystyle\nabla_{i}\nabla_{j}V_{,\nu}=\langle\lambda\partial_{r},\nabla
h_{ij}\rangle+Vh_{ij}-V_{,\nu}(h^{2})_{ij},$ (2.7)

where $\\{e_{1},\cdots,e_{n}\\}$ is a local orthonormal frame on $\Sigma$ and
$(h^{2})_{ij}=h_{i}^{k}h_{kj}$. Furthermore, if the hypersurface $\Sigma$ is
closed, then for each $\varepsilon\in\mathbb{R}$, there holds

$\displaystyle\int_{\Sigma}(V-\varepsilon
V_{,\nu})E_{k-1}(A)d\mu=\int_{\Sigma}V_{,\nu}E_{k}(A)d\mu,\quad k=1,\cdots,n,$
(2.8)

where $A=(h_{j}^{i}-\varepsilon\delta_{j}^{i})$.

###### Proof.

(2.5) – (2.7) are well-known, see e.g. [Guan-Li2014]. To prove (2.8), by (2.1)
– (2.2) and (2.6), we have

$\displaystyle\sum_{i,j}(\dot{E}_{k})_{i}^{j}(A)\nabla^{i}\nabla_{j}V=$
$\displaystyle\sum_{i,j}(\dot{E}_{k})_{i}^{j}(A)(V\delta_{j}^{i}-V_{,\nu}h_{j}^{i})$
$\displaystyle=$
$\displaystyle\sum_{i,j}(\dot{E}_{k})_{i}^{j}(A)\left[(V-\varepsilon
V_{,\nu})\delta_{j}^{i}-V_{,\nu}(h_{j}^{i}-\varepsilon\delta_{j}^{i})\right]$
$\displaystyle=$ $\displaystyle k\Big{(}(V-\varepsilon
V_{,\nu})E_{k-1}(A)-V_{,\nu}E_{k}(A)\Big{)}.$ (2.9)

Note that $A_{ij}=h_{ij}-\varepsilon g_{ij}$ is a Codazzi tensor, i.e.,
$\nabla_{\ell}A_{ij}$ is symmetric in $i,j,\ell$. It follows that
$(\dot{E}_{k})_{i}^{j}(A)$ is divergence free. The equation (2.8) follows from
integration by parts and the divergence free property of
$(\dot{E}_{k})_{i}^{j}(A)$. ∎

Along the general flow

$\displaystyle\frac{\partial}{\partial t}X=\mathcal{F}\nu$ (2.10)

for hypersurfaces in hyperbolic space $\mathbb{H}^{n+1}$, where $\nu$ is the
unit outward normal and $\mathcal{F}$ is the speed function of the smooth part
of the evolving hypersurface $\Sigma_{t}$, we have the following evolution
equations (see e.g. [HP99, Theorem 3.2]).

######  Proposition 2.4.

Along the flow (2.10) in $\mathbb{H}^{n+1}$, we have

$\displaystyle\partial_{t}g_{ij}=$ $\displaystyle 2\mathcal{F}h_{ij},$ (2.11)
$\displaystyle\partial_{t}d\mu_{t}=$ $\displaystyle\mathcal{F}Hd\mu_{t},$
(2.12) $\displaystyle\partial_{t}\nu=$ $\displaystyle-\nabla\mathcal{F},$
(2.13) $\displaystyle\partial_{t}h_{i}^{j}=$
$\displaystyle-\nabla^{j}\nabla_{i}\mathcal{F}-\mathcal{F}((h^{2})_{i}^{j}-\delta_{i}^{j}),$
(2.14)

where $(h^{2})_{i}^{j}=h_{i}^{k}h_{k}^{j}$, and $d\mu_{t}$ is the area form on
$\Sigma_{t}^{\ast}$.

##  3\. Proof of Theorem 1.1

Let $\Omega$ be a bounded domain in $(\mathbb{H}^{n+1},\overline{g})$ with
smooth boundary $\Sigma=\partial\Omega$. Let $\nu$ denote the outward unit
normal to $\Sigma$. Throughout this section, we assume that the mean curvature
of $\Sigma$ satisfies $H>n$ with respect to this choice of unit normal.

Denote by $X:\Sigma\times[0,\infty)\rightarrow\overline{\Omega}$ the normal
exponential map with respect to $\overline{g}$. For each point $y\in\Sigma$,
the curve $t\mapsto X(y,t)$ is a geodesic which has unit speed, and

$\displaystyle\left\\{\begin{aligned} \left.\frac{\partial}{\partial
t}X(y,t)\right|_{t=0}=&-\nu(y),\\\ X(y,0)=&y.\end{aligned}\right.$ (3.1)

For each point $p\in\overline{\Omega}$, we denote by $v(p)=d(p,\Sigma)$ the
geodesic distance to $\Sigma$ and $\\{v=0\\}=\Sigma$. We define

$\displaystyle A=$
$\displaystyle\left\\{(x,t)\in\Sigma\times[0,\infty):v(X(x,t))=t\right\\},$

and

$\displaystyle A^{\ast}=$
$\displaystyle\left\\{(x,t)\in\Sigma\times[0,\infty):(x,t+\delta)\in
A~{}\text{for some $\delta>0$}\right\\}.$

The following proposition follows from the definition of segment domain of a
Riemannian manifold, see [Bre13, Proposition 3.1].

######  Proposition 3.1.

[Bre13] The set $A$ and $A^{\ast}$ have the following properties:

  1. $(i)$

If $(x,t_{0})\in A$, then $(x,t)\in A$ for all $t\in[0,t_{0}]$.

  2. $(ii)$

The set $A$ is closed, and we have $X(A)=\overline{\Omega}$.

  3. $(iii)$

The set $A^{\ast}$ is an open subset of $\Sigma\times[0,\infty)$, and the
restriction $X|_{A^{\ast}}$ is a diffeomorphism.

The set $G=X(A^{\ast})$ is the largest open subset of $\Omega$ such that for
every $x$ in $G$, there is a unique closest point $y\in\Sigma$ to $x$.
Moreover, the boundary $\Sigma$ is smooth, and thus the distance function
$d(\cdot,\Sigma):\overline{\Omega}\rightarrow[0,\infty)$ belongs to
$C^{\infty}(G\cup\Sigma)\cap\operatorname{Lip}(\Omega)$. The set
$C=\Omega\backslash G$ is called the cut locus of $\Sigma$. A result of Li and
Nirenberg [Li-Nirenberg2005, Corollary 1.6] states that the $n$-dimensional
Hausdorff measure of the cut-locus $C$ of the hypersurface $\Sigma$ is finite,
i.e., $\mathcal{H}^{n}(C)<\infty$.

Consider a geodesic from a point $y\in\Sigma$ going into $\Omega$ in the
normal direction until it first hits a point in $x=m(y)$ in $C$. The point
$m(y)$ is called the cut point of $y$ on $\Sigma$, which means that if we go
beyond the point $x$ on the geodesic to any other point $x^{\prime}$, then
$x^{\prime}$ has a closer point $y^{\prime}\in\Sigma$ such that
$d(x^{\prime},y^{\prime})<d(x^{\prime},y)$. The cut locus of $\Sigma$ is the
collection of all these cut points $m(y)$ for all $y\in\Sigma$. An equivalent
way to define the cut point of $y\in\Sigma$ is as follows: The set of $t>0$
satisfying $d(X(y,t),\Sigma)=t$ is either $(0,\infty)$ or
$(0,\widetilde{t}(y)]$ for some $0<\widetilde{t}(y)<\infty$. In the latter
case, $\widetilde{m}(y)=X(y,\widetilde{t}(y))$ is the cut point of
$y\in\Sigma$, and the collection of $\widetilde{m}(y)$ for all $y\in\Sigma$,
denoted as $\widetilde{C}$, is also called the cut locus of $\Sigma$. These
two definitions are the same, i.e., $\widetilde{m}(y)=m(y)$ for all
$y\in\Sigma$ and $\widetilde{C}=C$, see [Li-Nirenberg2005, Section 4].

According to [Li-Nirenberg2005, Theorem 1.5], as $\Sigma$ is compact, the
function $d(y,m(y))$ is Lipschitz continuous in $y\in\Sigma$, and hence
$\widetilde{t}(y)=d(y,X(y,\widetilde{t}(y)))=d(y,\widetilde{m}(y))=d(y,m(y))$
is also Lipschitz continuous in $y\in\Sigma$. Let
$T=\max_{y\in\Sigma}\\{\widetilde{t}(y)\\}<\infty$. Then we have $0\leq
v(x)\leq T<\infty$ for all $x\in\overline{\Omega}$. For each $t\in[0,T]$, we
define

$\displaystyle\Sigma_{t}^{\ast}=X(A^{\ast}\cap(\Sigma\times\\{t\\})).$

Note that $\Sigma_{t}^{\ast}$ is a smooth hypersurface which is contained in
the level set $\\{v=t\\}$. Then we have $\Sigma_{0}^{\ast}=\Sigma=\\{v=0\\}$,
and $\Sigma_{T}^{\ast}=\emptyset$ due to the definition of $A^{\ast}$. In
particular, $\Sigma_{t}^{\ast}=\\{v=t\\}\backslash C$ for all $0<t<T$ and

$\Omega\backslash C=X(A^{\ast})=\bigcup_{0<t<T}\Sigma_{t}^{\ast}.$

Denote by $H$ and $h$ the mean curvature and the second fundamental form of
the smooth hypersurface $\Sigma_{t}^{\ast}$ in hyperbolic space.

######  Lemma 3.2.

The mean curvature of $\Sigma_{t}^{\ast}$ satisfies $H>n$ and the following
differential inequality

$\displaystyle\frac{\partial}{\partial
t}\left(\frac{V-V_{,\nu}}{H-n}\right)\leq-\frac{H}{n}\frac{V-V_{,\nu}}{H-n}.$
(3.2)

###### Proof.

Using Proposition 2.4 with $\mathcal{F}=-1$, the mean curvature of
$\Sigma_{t}^{\ast}$ evolves by

$\displaystyle\frac{\partial}{\partial t}H=|h|^{2}-n.$

By the trace inequality $|h|^{2}\geq H^{2}/n$, we obtain

$\displaystyle\frac{\partial}{\partial t}(H-n)\geq\frac{1}{n}(H-n)(H+n)$ (3.3)

at each point on $\Sigma_{t}^{\ast}$. Since the mean curvature of the initial
hypersurface $\Sigma$ satisfies $H>n$, applying the comparison principle to
(3.3) we conclude that $H>n$ on $\Sigma_{t}^{\ast}$ for each $t\in[0,T)$.

To prove the differential inequality (3.2), we use the evolution equation
(2.13) of the unit normal $\nu$ to calculate that

$\displaystyle\frac{\partial}{\partial t}V=$
$\displaystyle\langle\overline{\nabla}V,-\nu\rangle=-V_{,\nu},$ (3.4)
$\displaystyle\frac{\partial}{\partial t}V_{,\nu}=$
$\displaystyle\langle\overline{\nabla}_{-\nu}(\lambda\partial_{r}),\nu\rangle+\langle\lambda\partial_{r},\partial_{t}\nu\rangle=-\lambda^{\prime}=-V,$
(3.5)

where we used the fact that $\lambda\partial_{r}$ is a conformal Killing
vector field, that is,

$\langle\overline{\nabla}_{X}(\lambda\partial_{r}),Y\rangle=\lambda^{\prime}\langle
X,Y\rangle$ (3.6)

for any tangent vector fields $X,Y$ of $\mathbb{H}^{n+1}$ (see [Bre13, Lemma
2.2]). Since

$\displaystyle V-V_{,\nu}=\cosh r-\sinh
r\langle\partial_{r},\nu\rangle\geq\cosh r-\sinh r>0,$

it follows from (3.3)–(3.5) that

$\displaystyle\frac{\partial}{\partial t}\left(\frac{V-V_{,\nu}}{H-n}\right)=$
$\displaystyle\frac{1}{H-n}\frac{\partial}{\partial
t}(V-V_{,\nu})-\frac{V-V_{,\nu}}{(H-n)^{2}}\frac{\partial}{\partial t}H$
$\displaystyle\leq$ $\displaystyle-\frac{H}{n}\frac{V-V_{,\nu}}{H-n}$

at each point on $\Sigma_{t}^{\ast}$. This completes the proof. ∎

######  Corollary 3.3.

The function $t\mapsto\mu(\Sigma_{t}^{\ast})$ is monotone decreasing.

###### Proof.

On $\Sigma_{t}^{\ast}$, the area form $d\mu_{t}$ satisfies

$\displaystyle\frac{\partial}{\partial t}d\mu_{t}=-Hd\mu_{t}\quad\text{on
$\Sigma_{t}^{\ast}$}.$ (3.7)

Since $H>n$ on $\Sigma_{t}^{\ast}$, (3.7) implies that the area form of
$\Sigma_{t}^{\ast}$ is monotone decreasing in $t$. Moreover, the set
$\\{x\in\Sigma:(x,t)\in A^{\ast}\\}$ becomes smaller as $t$ increases. Then
the assertion follows. ∎

The following lemma is crucial in the proof of Theorem 1.1.

######  Lemma 3.4.

For a.e. $t\in(0,T)$, there holds

$\displaystyle\int_{\Sigma_{t}^{\ast}}V_{,\nu}d\mu=(n+1)\int_{\\{v>t\\}}Vd\operatorname{vol}.$
(3.8)

###### Proof.

Let $(\mathbb{B}^{n+1},\delta)$ be the unit open ball centered at the origin
in $\mathbb{R}^{n+1}$ equipped with the Euclidean metric $\delta$. We use the
Poincaré ball model to represent the hyperbolic space $\mathbb{H}^{n+1}$, that
is,

$\displaystyle(\mathbb{H}^{n+1},\overline{g})=\left(\mathbb{B}^{n+1},f^{2}\delta\right),\quad
f^{2}=\dfrac{4}{(1-|x|_{\delta}^{2})^{2}},$

where $|x|_{\delta}=\tanh\left(\frac{r}{2}\right)\in[0,1)$ and $r=r(x)$ is the
hyperbolic distance to the origin $o$. It is easy to check that $f=\cosh r+1$.

Let $\Omega$ be the bounded domain in $\mathbb{H}^{n+1}$ with smooth boundary
$\Sigma$, then it can be also viewed as a smooth bounded domain in
$\mathbb{B}^{n+1}$. For each $0<t<T$, the level set $\\{v=t\\}$, which is the
topological boundary of $\\{v>t\\}$, is a solution of the unit normal flow
(3.1).

Claim: For a.e. $t\in(0,T)$, there holds

$\displaystyle\mathcal{H}^{n}(\\{v=t\\}\backslash\Sigma_{t}^{\ast})=0.$ (3.9)

###### Proof of Claim.

It follows from [Li-Nirenberg2005, Corollary 1.6] that
$\mathcal{H}^{n}(C)<\infty$. Suppose (3.9) does not hold, there exists
uncountably many $\\{t_{\alpha}\\}_{\alpha\in\Lambda}\in(0,T)$ such that
$\mathcal{H}^{n}(\\{v=t\\}\backslash\Sigma_{t_{\alpha}}^{\ast})>0$. It follows
that

$\displaystyle\mathcal{H}^{n}(C)=$
$\displaystyle\mathcal{H}^{n}\left(\bigcup_{0<t<T}(\\{v=t\\}\backslash\Sigma_{t}^{\ast})\right)$
$\displaystyle\geq$
$\displaystyle\sum_{\alpha\in\Lambda}\mathcal{H}^{n}(\\{v=t\\}\backslash\Sigma_{t_{\alpha}}^{\ast})=\infty,$

which contradicts with $\mathcal{H}^{n}(C)<\infty$. ∎

We continue the proof of Lemma 3.4. Since $v$ is Lipschitz continuous, by
[Maggi2012, Theorem 13.1 and Example 13.3] we conclude that for a.e.
$t\in(0,T)$, the super-level set $\\{v>t\\}$ is an open set of finite
perimeter in $\mathbb{R}^{n+1}$. The reduced boundary
$\partial^{\ast}\\{v>t\\}$ of $\\{v>t\\}$ exists and it is always a subset of
the topological boundary $\\{v=t\\}$, see [Maggi2012, Remark 15.3]. Moreover,
by the coarea formula for Lipschitz functions, we have

$\displaystyle\mathcal{H}^{n}(\partial^{\ast}\\{v>t\\})=\mathcal{H}^{n}(\\{v=t\\}),\quad\text{for
a.e. $t\in[0,T)$},$ (3.10)

see [Maggi2012, Theorem 18.1 and Remark 18.2]. Recall that the reduced boudary
$\partial^{\ast}\\{v>t\\}$ is the union of countably many compact subsets
$K_{i}$ of $C^{1}$-hypersurfaces such that
$\partial^{\ast}\\{v>t\\}=N\cup\bigcup_{i\geq 1}K_{i}$, where
$\mathcal{H}^{n}(N)=0$. It follows from (3.9) and (3.10) that for a.e.
$t\in(0,T)$, $\\{v=t\\}=\Sigma_{t}^{\ast}\cup Z$ with $\mathcal{H}^{n}(Z)=0$
and hence $\\{v=t\\}$ is an open set with almost $C^{1}$-boundary.

Let $\\{E_{i}\\}_{i=1}^{n+1}$ be an orthonormal basis in
$(\mathbb{B}^{n+1},\delta)$. Then
$\\{\overline{E}_{i}=f^{-1}E_{i}\\}_{i=1}^{n+1}$ is an orthonormal basis of
$(\mathbb{B}^{n+1},f^{2}\delta)$. Denote by $\overline{\nabla}$ (resp.
$\nabla^{\delta}$) and $\overline{\Delta}$ (resp. $\Delta_{\delta}$) the
gradient and the Laplacian with respect to $\overline{g}=f^{2}\delta$ (resp.
$\delta$). For any function $F\in C^{\infty}(\mathbb{B}^{n+1},f^{2}\delta)$,
it can also be considered as a function $F\in
C^{\infty}(\mathbb{B}^{n+1},\delta)$. Then there holds

$\displaystyle\nabla^{\delta}F=\sum_{i=1}^{n+1}E_{i}(F)E_{i}=f^{2}\sum_{i=1}^{n+1}\overline{E}_{i}(F)\overline{E}_{i}=f^{2}\overline{\nabla}F.$

Using the transformation law of the Laplacian for $\overline{g}=f^{2}\delta$,
one has

$\displaystyle\overline{\Delta}V=f^{-2}\left(\Delta_{\delta}V+(n-1)d\ln
f(\nabla^{\delta}V)\right).$ (3.11)

We also have

$\displaystyle d\operatorname{vol}=f^{n+1}d\operatorname{vol}_{\delta},\quad
d\mu=f^{n}d\mu_{\delta},\quad\nu=f^{-1}\nu_{\delta},$ (3.12)

where $d\operatorname{vol}_{\delta}$ is Euclidean volume form, $d\mu_{\delta}$
is the area form and $\nu_{\delta}$ is the unit outward normal of $\Sigma$ in
$(\mathbb{B}^{n+1},\delta)$.

Using the Gauss-Green theorem on open sets with almost $C^{1}$-boundary in
Euclidean space [Maggi2012, Theorem 9.6], there holds

$\displaystyle\int_{\Sigma_{t}^{\ast}}f^{n-1}dV(\nu_{\delta})d\mu_{\delta}=$
$\displaystyle\int_{\\{v>t\\}}\operatorname{div}^{\delta}(f^{n-1}\nabla^{\delta}V)d\operatorname{vol}_{\delta}$
$\displaystyle=$
$\displaystyle\int_{\\{v>t\\}}\left(f^{n-1}\Delta_{\delta}V+(n-1)f^{n-2}df(\nabla^{\delta}V)\right)d\operatorname{vol}_{\delta}.$
(3.13)

Applying (3.11) and (3.12) to (3), we obtain

$\displaystyle\int_{\\{v>t\\}}\overline{\Delta}Vd\operatorname{vol}=$
$\displaystyle\int_{\\{v>t\\}}\left(f^{n-1}\Delta_{\delta}V+(n-1)f^{n-2}df(\nabla^{\delta}V)\right)d\operatorname{vol}_{\delta}$
$\displaystyle=$
$\displaystyle\int_{\Sigma_{t}^{\ast}}f^{n-1}dV(\nu_{\delta})d\mu_{\delta}$
$\displaystyle=$
$\displaystyle\int_{\Sigma_{t}^{\ast}}\langle\overline{\nabla}V,\nu\rangle
d\mu.$

Since $\overline{\Delta}V=(n+1)V$ (which follows from the conformal property
(3.6), see also [Montiel-Ros1991]), we obtain the equation (3.8). ∎

Next, we introduce the quantity

$\displaystyle
Q(t)=e^{(n+1)t}\left[\int_{\Sigma_{t}^{\ast}}\frac{V-V_{,\nu}}{H-n}d\mu-\frac{n+1}{n}\int_{\\{v>t\\}}Vd\operatorname{vol}\right].$
(3.14)

We have the following monotonicity property.

######  Proposition 3.5.

For any $t\in[0,T)$, there holds

$\displaystyle Q(0)\geq Q(t).$

###### Proof.

By the coarea formula, we have

$\displaystyle\int_{\\{v>t\\}}Vd\operatorname{vol}=\int_{X(A^{\ast}\cap(\Sigma\times(t,T)))}Vd\operatorname{vol}=\int_{t}^{T}\int_{\Sigma_{\tau}^{\ast}}Vd\mu
d\tau,$

and hence

$\displaystyle\frac{d}{dt}\int_{\\{v>t\\}}Vd\operatorname{vol}=-\int_{\Sigma_{t}^{\ast}}Vd\mu_{t}.$
(3.15)

Using (3.2), (3.7) and (3.15), we obtain

$\displaystyle\limsup_{h\searrow 0}\frac{1}{h}\left(Q(t)-Q(t-h)\right)$
$\displaystyle\leq$
$\displaystyle(n+1)Q(t)+e^{(n+1)t}\Bigg{(}\int_{\Sigma_{t}^{\ast}}\frac{\partial}{\partial
t}\left(\frac{V-V_{,\nu}}{H-n}\right)d\mu_{t}$
$\displaystyle\qquad-\int_{\Sigma_{t}^{\ast}}\frac{V-V_{,\nu}}{H-n}Hd\mu_{t}+\frac{n+1}{n}\int_{\Sigma_{t}^{\ast}}Vd\mu_{t}\Bigg{)}$
$\displaystyle\leq$
$\displaystyle\frac{n+1}{n}e^{(n+1)t}\left(\int_{\Sigma_{t}^{\ast}}V_{,\nu}d\mu_{t}-(n+1)\int_{\\{v>t\\}}Vd\operatorname{vol}\right).$
(3.16)

By Lemma 3.4, the last line of (3) vanishes for a.e. $t\in[0,T)$. It follows
that for any $t\in[0,T)$,

$\displaystyle Q(t)-Q(0)\leq$
$\displaystyle\int_{0}^{t}\frac{n+1}{n}e^{(n+1)\tau}\left(\int_{\Sigma_{\tau}^{\ast}}V_{,\nu}d\mu_{t}-(n+1)\int_{\\{v>\tau\\}}Vd\operatorname{vol}\right)d\tau$
$\displaystyle\leq$ $\displaystyle 0.$

∎

Now we can complete the proof of Theorem 1.1.

###### Proof of Theorem 1.1.

By Proposition 3.5, we have

$\displaystyle\int_{\Sigma}\frac{V-V_{,\nu}}{H-n}d\mu-\frac{n+1}{n}\int_{\Omega}Vd\operatorname{vol}=Q(0)\geq\liminf_{t\rightarrow
T}Q(t),$ (3.17)

where we used the fact that $\Sigma=\Sigma_{0}^{\ast}$ and $\Omega=\\{v>0\\}$.
To prove the desired inequality (1.3), it suffices to show that
$\liminf_{t\rightarrow T}Q(t)\geq 0$.

Since $V-V_{,\nu}>0$ and $H>n$ on $\Sigma_{t}^{\ast}$ (by Lemma 3.2), we
deduce that for a.e. $t\in[0,T)$,

$\displaystyle Q(t)\geq$
$\displaystyle-\frac{n+1}{n}e^{(n+1)T}\int_{\\{v>t\\}}Vd\operatorname{vol}$
$\displaystyle=$
$\displaystyle-\frac{n+1}{n}e^{(n+1)T}\int_{X(A^{\ast}\cap(\Sigma\times(t,T)))}Vd\operatorname{vol}$
$\displaystyle=$
$\displaystyle-\frac{n+1}{n}e^{(n+1)T}\int_{t}^{T}\int_{\Sigma_{\tau}^{\ast}}Vd\mu
d\tau.$

As $d\mu_{t}$ is monotone decreasing by (3.7), $V=\cosh r$ is uniformly
bounded on $\Omega$ and $T<\infty$, we conclude that

$\liminf_{t\rightarrow T}Q(t)\geq 0.$

If equality holds in (1.3), then the equality holds in the differential
inequality (3.2), which implies that $\Sigma_{t}^{\ast}$ is totally umbilical
for all $t\in[0,T)$. Therefore, the initial hypersurface
$\Sigma=\Sigma_{0}^{\ast}$ is umbilical. This completes the proof of Theorem
1.1. ∎

##  4\. Proof of Theorems 1.2 and 1.3

###### Proof of Theorem 1.2.

Fix $k\in\\{2,\cdots,n\\}$. Since $\Sigma$ is compact, there exists an
elliptic point such that all principal curvatures $\kappa_{i}>1$ for
$i=1,\cdots,n$ and hence the constant $E_{k}(\widetilde{\kappa})$ is positive.
By continuity of the principal curvatures, we have
$\widetilde{\kappa}\in\Gamma_{k}^{+}$. Then we have

$\displaystyle\int_{\Sigma}(V-V_{,\nu})\frac{E_{k-1}(\widetilde{\kappa})}{E_{k}(\widetilde{\kappa})}d\mu=$
$\displaystyle\frac{1}{E_{k}(\widetilde{\kappa})}\int_{\Sigma}(V-V_{,\nu})E_{k-1}(\widetilde{\kappa})d\mu$
$\displaystyle=$
$\displaystyle\frac{1}{E_{k}(\widetilde{\kappa})}\int_{\Sigma}V_{,\nu}E_{k}(\widetilde{\kappa})d\mu$
$\displaystyle=$ $\displaystyle\int_{\Sigma}V_{,\nu}d\mu,$ (4.1)

where in the second equality we used Minkowski identity (2.8). By the
Minkowski formula (see [Montiel-Ros1991]), we also have

$\int_{\Sigma}V_{,\nu}d\mu=(n+1)\int_{\Omega}Vd\mathrm{vol},$ (4.2)

where $\Omega$ is the domain enclosed by $\Sigma$. Then the Heintze-Karcher
type inequality (1.3) and the Newton-MacLaurin inequality (2.4) imply

$\displaystyle\int_{\Sigma}(V-V_{,\nu})\frac{E_{k-1}(\widetilde{\kappa})}{E_{k}(\widetilde{\kappa})}d\mu\geq$
$\displaystyle\int_{\Sigma}\frac{V-V_{,\nu}}{E_{1}(\widetilde{\kappa})}d\mu$
$\displaystyle\geq$ $\displaystyle(n+1)\int_{\Omega}Vd\mathrm{vol}$
$\displaystyle=$ $\displaystyle\int_{\Sigma}V_{,\nu}d\mu.$ (4.3)

Combining (4) and (4), we conclude that the equality holds in the Newton-
MacLaurin inequality (2.4), so $\Sigma$ is totally umbilical and it is a
geodesic sphere. ∎

###### Proof of Theorem 1.3.

As the hypersurface $\Sigma$ is uniformly $h$-convex, we have
$\widetilde{\kappa}_{i}>0$, $E_{k}(\widetilde{\kappa})>0$ and $\frac{\partial
E_{k}(\widetilde{\kappa})}{\partial\widetilde{\kappa}_{i}}>0$. We deduce from
the Heintze-Karcher type inequality (1.3) and (4.2) that

$\displaystyle 0\leq$
$\displaystyle\int_{\Sigma}\left(\frac{V-V_{,\nu}}{E_{1}(\widetilde{\kappa})}-V_{,\nu}\right)d\mu$
$\displaystyle\leq$
$\displaystyle\int_{\Sigma}\frac{(V-V_{,\nu})E_{k-1}(\widetilde{\kappa})-V_{,\nu}E_{k}(\widetilde{\kappa})}{E_{k}(\widetilde{\kappa})}d\mu$
$\displaystyle=$
$\displaystyle\int_{\Sigma}\frac{1}{\chi}\Big{(}(V-V_{,\nu})E_{k-1}(\widetilde{\kappa})-V_{,\nu}E_{k}(\widetilde{\kappa})\Big{)}d\mu,$
(4.4)

where the second inequality follows from $V>V_{,\nu}$ and the Newton-MacLaurin
inequality (2.4), and the last equality follows from the curvature equation
(1.4). Using (2.2) with $\varepsilon=1$, we can rewrite the last line of (4)
as

$\displaystyle\int_{\Sigma}\frac{1}{\chi}\Big{(}(V-V_{,\nu})E_{k-1}(\widetilde{\kappa})-V_{,\nu}E_{k}(\widetilde{\kappa})\Big{)}d\mu$
$\displaystyle=$
$\displaystyle\frac{1}{k}\int_{\Sigma}\frac{1}{\chi}\sum_{i=1}^{n}\frac{\partial
E_{k}(\widetilde{\kappa})}{\partial\widetilde{\kappa}_{i}}\nabla_{i}\nabla_{i}Vd\mu$
$\displaystyle=$
$\displaystyle\frac{1}{k}\int_{\Sigma}\frac{\chi^{\prime}}{\chi^{2}}\sum_{i=1}^{n}\frac{\partial
E_{k}(\widetilde{\kappa})}{\partial\widetilde{\kappa}_{i}}\nabla_{i}V\nabla_{i}(V-V_{,\nu})d\mu$
$\displaystyle=$
$\displaystyle-\frac{1}{k}\int_{\Sigma}\frac{\chi^{\prime}}{\chi^{2}}\sum_{i=1}^{n}\frac{\partial
E_{k}(\widetilde{\kappa})}{\partial\widetilde{\kappa}_{i}}\widetilde{\kappa}_{i}|\nabla_{i}V|^{2}d\mu\leq
0,$ (4.5)

where we used integration by parts and the divergence-free property of
$E_{k}(\widetilde{\kappa})$ in the second equality, and the third equality
follows from (2.5). The last inequality is due to $\chi>0$ and
$\chi^{\prime}\geq 0$.

Combining (4) and (4), we see that both the Heintze-Karcher type inequality
(1.3) and Newton-MacLaurin inequality (2.4) must achieve the equality. This
implies that $\Sigma$ is a geodesic sphere. Furthermore, if $\chi^{\prime}>0$,
the last inequality of (4) implies that $\nabla_{i}V=0$ for all
$i=1,\cdots,n$. Then the unit normal $\nu$ is proportional to the radial
direction $\partial_{r}$, and hence $\Sigma$ must be a geodesic sphere
centered at the origin. ∎

## References