# Which principal components are most sensitive to distributional changes?

Martin Tveten, Department of Mathematics, University of Oslo

###### Abstract

PCA is often used in anomaly detection and statistical process control tasks.
For bivariate data, we prove that the minor projection (the least varying
projection) of the PCA-rotated data is the most sensitive to distributional
changes, where sensitivity is defined by the Hellinger distance between
distributions before and after a change. In particular, this is almost always
the case if only one parameter of the bivariate normal distribution changes,
i.e., the change is sparse. Simulations indicate that the minor projections
are the most sensitive for a large range of changes and pre-change settings in
higher dimensions as well. This motivates using the minor projections for
detecting sparse distributional changes in high-dimensional data.

Keywords: Change-point detection; monitoring; anomaly detection; statistical
process control; variance; correlation.

##  1 Introduction

It is popular to use principal component analysis (PCA) for anomaly detection
and stochastic process control (SPC). Using PCA in SPC goes back to the work
of [5] and [6], and its various extensions (see [8] and [13] for an overview)
have been succesfully applied to many real data situations. Within the machine
learning litterature on anomaly detection, [11] use PCA for temperature
monnitoring at Johns Hopkins, [2] apply PCA-based anomaly detection to find
segments with abnormal rates of patient arrivals at an emergency department,
and [1] relate PCA-based monitoring in SPC with modern anomaly detection in
statistical networks. [12] provide an extensive review of novelty detection
techniques and applications, and it is pointed to PCA being very useful for
detecting outliers in this setting, for a large range of real world examples,
covering industrial monitoring, video surveilance, text mining, sensor
networks and IT security. In this review, as well as in [10] and [4], it is
acknowledged that it is most often the residual subspace of PCA that is most
useful for outlier detection. [9] study PCA in the related field of change
detection, but with a fascinating twist.

In general, most PCA-based methods utilize PCA in the intended way of creating
a model based on retaining a small number of the most varying projections onto
eigenvectors of the correlation matrix. The data is thus split into a model
subspace that explains most of the variance in the data and a residual
subspace. Rather than using PCA in this regular way for change detection as
well, [9] pose the question of which projections/principal components are the
most sensitive to distributional changes in the data. Sensitivity is measured
by a statistical distance between the marginal distributions of projections
before and after a change. They give a brief two-dimensional theoretical
example that motivates monitoring the minor projections (the least varying
projections) to detect anomalies. An important feature of such an approach is
that it can be used to choose a subspace based on criteria linked to change
detection, rather than on retaining information/variance in the data,
potentially yielding better change and anomaly detection methods. The goal of
this article is to give a more complete treatment of the bivariate problem of
[9], to understand the projection’s sensitivity under a simple setup, and then
study how these results carry over to higher dimensions by simulations.

There are three main differences between our approach and the approach of [9].
First of all, [9] look at changes in the projections directly, without
connecting them to which changes in the original data that cause them. Since
it is changes in the distribution of the original data that are of interest,
the propositions one actually wants to obtain concern changes in the original
data, not the resulting ones in the projections. We trace the changes in the
original data through the projections, to see how the distributions of the
projections change as a consequence. For instance, we give an answer to
questions like "if the variances of the original components double, how does
that affect the projections, and which of the projections will change the
most?", as opposed to "if the variances of the projections both increase by
adding 1, which of the projections change that most?" of [9]. In the latter
case, clearly the projection with the lowest variance will change the most
relative to what it was, and we have gained no knowledge about how the
original data must change to achieve this change in the projections.

Secondly, we study a much larger space of possible changes, including changes
in only one parameter at a time. Such change scenarios where only a few of the
dimensions change we call sparse changes. This turns out to be the main gist
of this work, since it is especially in the sparse change scenarios that the
minor projection is the most sensitive. Hence, we obtain clearer hints of
which change scenarios the minor components might be the most useful for in
higher dimensional change detection problems as well.

Thirdly, we measure sensitivity by the normal Hellinger distance between the
marginal distributions of projections before and after a change, while [9] use
the normal Bhattacharyya distance. We refer to Section 2 for an explanation of
this choice.

In short, for bivariate data, we show that if only one mean changes in any
direction, one variance increases, or the correlation changes, the minor
projection is the most sensitive. The principal projection is the most
sensitive if one variance decreases and the correlation is not too close to
$1$. Lastly, if both means change, which projection is the most sensitive
depends on the relative directions and sizes of change, and when both
variances change by an equal amount, both projections are equally sensitive.
Thus, on average (with all change scenarios equally likely), the minor
projection is the most sensitive, mainly due to the sparse changes. Our
simulations show that the trend of the minor projections being more sensitive
on average also holds for higher dimensions. This knowledge is important to
create more efficient change or anomaly detection methods.

The rest of the article is organized as follows: Section 2 formulates the
problem precisely, Section 3 contains the theoretical results about
sensitivity to changes in two dimensions, while in Section 4, we explore
sensitivity in higher dimensions by simulations. The proofs for the results in
Section 3 are given in Appendix A.

##  2 Problem formulation

Consider $n$ independent observations $\mathbf{x}_{t}\in\mathbb{R}^{D}$. For
$t=1,\ldots,\kappa$, where $1<\kappa<n$, the data has mean
$\boldsymbol{\mu}_{0}$ and covariance matrix $\boldsymbol{\Sigma}_{0}$, while
for $t=\kappa+1,\ldots,n$, the data has mean $\boldsymbol{\mu}_{1}$ and
covariance matrix $\boldsymbol{\Sigma}_{1}$. Assume without loss of generality
that the data is standardized with respect to the pre-change parameters, so
that $\boldsymbol{\mu}_{0}=\mathbf{0}$ and $\boldsymbol{\Sigma}_{0}$ is a
correlation matrix. For $D=2$, the changed mean is given by
$\boldsymbol{\mu}_{1}=(\mu_{1},\mu_{2})^{\mathchoice{\raisebox{0.0pt}{$\displaystyle\intercal$}}{\raisebox{0.0pt}{$\textstyle\intercal$}}{\raisebox{0.0pt}{$\scriptstyle\intercal$}}{\raisebox{0.0pt}{$\scriptscriptstyle\intercal$}}}$
and the covariance matrices by

$\boldsymbol{\Sigma}_{0}=\begin{pmatrix}1&\rho\\\
\rho&1\end{pmatrix}\quad\text{and}\quad\boldsymbol{\Sigma}_{1}=\begin{pmatrix}a_{11}^{2}&a_{11}a_{22}a_{12}\rho\\\
a_{11}a_{22}a_{12}\rho&a_{22}^{2}\end{pmatrix}.$

where

$-1<\rho,\;a_{12}\rho<1\quad\text{and}\quad\rho\not=0.$ (1)

Here, $a_{11}$ and $a_{22}$ denote multiplicative change factors for the
standard deviation. For example, if $a_{11}=0.5$ the standard deviation is
half of what it was originally. Similarly, $a_{12}$ is the change factor for
the correlation. Note that we exclude the degenerate cases of correlations
$-1$ and $1$.

Next, let $\\{\lambda_{j},\mathbf{v}_{j}\\}_{j=1}^{D}$ be the normalized
eigensystem of $\boldsymbol{\Sigma}_{0}$, ordered by
$\lambda_{1}\geq\ldots\geq\lambda_{D}$. The orthogonal projections
$y_{j,t}=\mathbf{v}_{j}^{\mathchoice{\raisebox{0.0pt}{$\displaystyle\intercal$}}{\raisebox{0.0pt}{$\textstyle\intercal$}}{\raisebox{0.0pt}{$\scriptstyle\intercal$}}{\raisebox{0.0pt}{$\scriptscriptstyle\intercal$}}}\mathbf{x}_{t}$
are our main objects of interest. We refer to $y_{j,t}$ as principal
projections when $j$ is close to $1$ and minor projections if $j$ is close to
$D$. As this is meant as approximate terminology, we do not specify further
what "close" means. The general problem is to find out which of the
projections that are the most sensitive to different distributional changes
defined by $(\boldsymbol{\mu}_{1},\boldsymbol{\Sigma}_{1})$, for each pre-
change correlation matrix $\boldsymbol{\Sigma}_{0}$. In the bivariate case,
$(\boldsymbol{\Sigma}_{0},\boldsymbol{\mu}_{1},\boldsymbol{\Sigma}_{1})$ is
fully specified by $(\rho,\mu_{1},\mu_{2},a_{11},a_{12},a_{22})$.

We define sensitivity to changes as the normal Hellinger distance between the
marginal distribution of a projection before and after a change. The squared
Hellinger distance between two normal distributions
$p(x)=N(x|\xi_{1},\sigma_{1})$ and $q(x)=N(x|\xi_{2},\sigma_{2})$ is given by

$H^{2}(p,q)=1-\sqrt{\frac{2\sigma_{1}\sigma_{2}}{\sigma_{1}^{2}+\sigma_{2}^{2}}}\exp\left\\{-\frac{1}{4}\frac{(\xi_{1}-\xi_{2})^{2}}{\sigma_{1}^{2}+\sigma_{2}^{2}}\right\\}.$

The formal definition of sensitivity to changes is contained in Definition 1.

######  Definition 1.

For $j=1,\ldots,D$, let $p_{j}$ and $q_{j}$ denote the marginal pre- and post-
change density functions of $y_{j,t}$, respectively, given by

$\displaystyle p_{j}(y)$
$\displaystyle=N(y|\;\mathbf{v}_{j}^{\mathchoice{\raisebox{0.0pt}{$\displaystyle\intercal$}}{\raisebox{0.0pt}{$\textstyle\intercal$}}{\raisebox{0.0pt}{$\scriptstyle\intercal$}}{\raisebox{0.0pt}{$\scriptscriptstyle\intercal$}}}\boldsymbol{\mu}_{0},\mathbf{v}_{j}^{\mathchoice{\raisebox{0.0pt}{$\displaystyle\intercal$}}{\raisebox{0.0pt}{$\textstyle\intercal$}}{\raisebox{0.0pt}{$\scriptstyle\intercal$}}{\raisebox{0.0pt}{$\scriptscriptstyle\intercal$}}}\boldsymbol{\Sigma}_{0}\mathbf{v}_{j})=N(y|\;0,\lambda_{j})$
$\displaystyle q_{j}(y)$
$\displaystyle=N(y|\;\mathbf{v}_{j}^{\mathchoice{\raisebox{0.0pt}{$\displaystyle\intercal$}}{\raisebox{0.0pt}{$\textstyle\intercal$}}{\raisebox{0.0pt}{$\scriptstyle\intercal$}}{\raisebox{0.0pt}{$\scriptscriptstyle\intercal$}}}\boldsymbol{\mu}_{1},\mathbf{v}_{j}^{\mathchoice{\raisebox{0.0pt}{$\displaystyle\intercal$}}{\raisebox{0.0pt}{$\textstyle\intercal$}}{\raisebox{0.0pt}{$\scriptstyle\intercal$}}{\raisebox{0.0pt}{$\scriptscriptstyle\intercal$}}}\boldsymbol{\Sigma}_{1}\mathbf{v}_{j}).$

The sensitivity of the $j$’th projection based on $\boldsymbol{\Sigma}_{0}$ to
the change specified by $(\boldsymbol{\mu}_{1},\boldsymbol{\Sigma}_{1})$ is
defined as $H(p_{j},q_{j})$, abbreviated by $H_{j}$ or
$H_{j}(\boldsymbol{\Sigma}_{0},\boldsymbol{\mu}_{1},\boldsymbol{\Sigma}_{1})$.

In light of Definition 1, our aim in the next section is to study for which
pre-change parameters and changes the inequality $H_{2}>H_{1}$ holds when
$D=2$.

######  Remarks.

(i) [9] also define sensitivity as a divergence between distributions before
and after a change, but use the Bhattacharyya distance. The closely related
Hellinger distance was chosen here because it turned out to be simpler to
prove the sensitivity propositions because of Lemma 5, found in the Appendix.
It is also an advantageous feature of the Hellinger distance that it is a true
metric and takes values in $[0,1]$. That it is a true metric implies for
instance that a change in variance from $1$ to $a>1$ is an equally large
change as from $1$ to $1/a$. We find this an appealing feature because it is
also a property of the generalized likelihood ratio test for a change in the
mean and/or variance of normal data (see [3] for the corresponding test
statistic).

(ii) The difference between our approach and the work of [9] can now be stated
more clearly. Our aim is to study the sensitivity of the $y_{j,t}$’s as
functions of parameters of the original data $\mathbf{x}_{t}$. [9], on the
other hand, study (additive) changes in the parameters of $\mathbf{y}_{t}$
directly; for instance, $\lambda_{j}$ changing to $\lambda_{j}+a$ for all $j$,
but without relating this $a$ back to which $\boldsymbol{\Sigma}_{1}$’s this
change corresponds to.

##  3 Bivariate results

This section contains all the bivariate results about sensitivity to changes.
The detailed proofs are given in Appendix A.

For changes in the mean in two-dimensional data, the following proposition
gives the condition for determining which projection that is the most
sensitive.

######  Proposition 1.

Let $a_{11}=a_{22}=a_{12}=1$ and $\mu_{1},\mu_{2}\in\mathbb{R}$ while not both
being $0$ simultaneously (only the mean changes). $H_{2}>H_{1}$ if and only if
$(\mu_{1}-\mu_{2})^{2}>\mu_{1}\mu_{2}(4/|\rho|-2)$. In particular, for
$|\rho|\in(0,1)$,

  1. (i)

$H_{2}>H_{1}$ if one of $\mu_{1}$ and $\mu_{2}$ is $0$ while the other is not
(one mean changes).

  2. (ii)

$H_{2}>H_{1}$ if $\mu_{1}=-\mu_{2}=\mu\not=0$ (equal changes in opposite
directions).

  3. (iii)

$H_{2}<H_{1}$ if $\mu_{1}=\mu_{2}=\mu\not=0$ (equal changes in the same
direction).

When both variances change by the same amount, we get that both projections
are equally sensitive no matter what the pre-change correlation or size of the
change is.

######  Proposition 2.

Let $\mu_{1}=\mu_{2}=0$, $a_{12}=1$ and $a_{11}=a_{22}=a\not=1$ (both
variances change equally). For any $|\rho|\in(0,1)$ and $a>0$, $H_{2}=H_{1}$.

The picture becomes more complicated when only one variance changes. If the
variance increases, the minor projection is always the most sensitive. On the
other hand, if the variance decreases, the principal projection is mostly the
most sensitive, but not always if the pre-change correlation is high ($\gtrsim
0.87$). In total, this gives a slight edge to the minor projection.

######  Proposition 3.

Let $\mu_{1}=\mu_{2}=0$, $a_{12}=1$, and either $a_{11}=1$ and
$a_{22}=a\not=1$, or $a_{11}=a$ and $a_{22}=1$, where $a>0$ (one variance
changes).

  1. (i)

For any $|\rho|\in(0,1)$ and $a>1$ (variance increase), $H_{2}>H_{1}$.

  2. (ii)

When $|\rho|\in(0,1)$ and $a\in(0,1)$ (variance decrease), $H_{2}<H_{1}$ in
most cases. The only exception is if $|\rho|\in(\sqrt{3}/2,1)$ and
$a\in(0,\sqrt{4\rho^{2}-3})$, where $H_{2}>H_{1}$.

For a change in correlation, the minor projection is the most sensitive in
most cases. Only if the correlation changes direction and becomes stronger is
the principal projection more sensitive.

######  Proposition 4.

Let $\mu_{1}=\mu_{2}=0$, $a_{11}=a_{22}=1$ and $a_{12}=a\not=1$ such that (1)
holds (the correlation changes). Then $H_{2}>H_{1}$ for any $|\rho|\in(0,1)$
and $a>-1$.

##  4 Exploring higher dimensions

In the two-dimensional case, we saw that which projection is the most
sensitive depends both on the change
$(\boldsymbol{\mu}_{1},\boldsymbol{\Sigma}_{1})$ as well as the pre-change
correlation matrix $\boldsymbol{\Sigma}_{0}$. For higher dimension $D$,
solving inequalities like above for all the parameters in
$(\boldsymbol{\Sigma}_{0},\boldsymbol{\mu}_{1},\boldsymbol{\Sigma}_{1})$
quickly becomes extremely tedious. Therefore, we use simulation to obtain
Monte Carlo estimates
$E[H_{j}(\boldsymbol{\Sigma}_{0},\boldsymbol{\mu}_{1},\boldsymbol{\Sigma}_{1})]$
in stead, where we vary which parameters that change, the size of the changes
and the sparsity of the change (the number of dimensions that change). Let
$\rho_{i,d}$ for $i\not=d$ denote the off-diagonal elements of
$\boldsymbol{\Sigma}_{0}$, $\mu_{d}$ be the $d$’th element of
$\boldsymbol{\mu}_{1}$, and $\sigma_{d}$ be the $d$’th diagonal element of
$\boldsymbol{\Sigma}_{1}$. Then our simulation protocol to get such estimates
is as follows:

  1. 1.

Draw a correlation matrix $\boldsymbol{\Sigma}_{0}$ uniformly from the space
of correlation matrices by the method of [7] (clusterGeneration::rcorrmatrix
in R).

  2. 2.

Draw a change sparsity $K\sim\text{{Unif}}\\{2,\ldots,D\\}$.

  3. 3.

Draw a random subset $\mathcal{D}\subseteq\\{1,\ldots,D\\}$ of size $K$.

  4. 4.

Draw an additive change in mean $\mu\sim\text{{Unif}}(-3,3)$, and set
$\mu_{d}=\mu$ for $d\in\mathcal{D}$, while
$\boldsymbol{\Sigma}_{1}=\boldsymbol{\Sigma}_{0}$.

  5. 5.

Draw a multiplicative change in standard deviation
$\sigma\sim\frac{1}{2}\text{{Unif}}(1/3,1)+\frac{1}{2}\text{{Unif}}(1,3)$
(equal probability of decrease and increase in standard deviation) and set
$\sigma_{d}=\sigma$ for $d\in\mathcal{D}$, keeping the remaining parameters
constant.

  6. 6.

Draw a multiplicative change in correlation $a\sim\text{{Unif}}(0,1)$ and
change $\rho_{i,d}$ to $a\rho_{i,d}$ for all $i\not=d\in\mathcal{D}$. The
other parameters are kept constant.

  7. 7.

For each of the three change scenarios 4-6, calculate
$H_{j}(\boldsymbol{\Sigma}_{0},\boldsymbol{\mu}_{1},\boldsymbol{\Sigma}_{1})$,
$j=1,\ldots,D$.

  8. 8.

Repeat 2-7 $10^{3}$ times.

  9. 9.

Repeat 1-8 $10^{3}$ times.

Averaging the simulated $H_{j}$’s yields estimates of $E[H_{j}]$, and we can
condition on the type of parameter that change and the change sparsity to see
what the sensitivity is expected to be for different classes of changes. (Note
that we only consider decreases in correlation. This is to avoid getting too
many indefinite $\boldsymbol{\Sigma}_{1}$’s. If indefinite
$\boldsymbol{\Sigma}_{1}$’s still occur, we find the closest positive-definite
one by the Matrix::nearPD-function in R and set the diagonal to 1.)

Figure 1 shows that the trend of the minor components being the most sensitive
continues for $D=20$ and $D=100$. This holds for changes in the mean, variance
and correlation as well as all the different change sparsities. From the
quantile plots, however, observe that a lot of variation is hidden in these
averages, meaning that which projection is the most sensitive will depend on
the specific $\boldsymbol{\Sigma}_{0}$ and change
$(\boldsymbol{\mu}_{1},\boldsymbol{\Sigma}_{1})$, as in the bivariate case.

Figure 1: A summary of the sensitivity results obtained by the simulation
protocol for $D=20$ for $D=100$. a) Monte Carlo estimates of $E[H_{j}]$ for
uniformly drawn changes in the mean, variance and (decreases in) correlation,
as well as uniformly drawn pre-change correlation matrices
$\boldsymbol{\Sigma}_{0}$. b) Same as a), but now the average sensitivity is
conditional on the sparsity of the change, rather than the type of parameter.
c) 0.05, 0.25, 0.75 and 0.95 percentiles (the dashed lines, from bottom to
top) of the distribution of $H_{j}$, together with $E[H_{j}]$ (solid line).
Note that the percentiles are over
$\boldsymbol{\Sigma}_{0},\boldsymbol{\mu}_{1}$ and $\boldsymbol{\Sigma}_{1}$
simultaneously.

##  5 Concluding remarks

We have presented bivariate theory showing that the minor projection of PCA-
rotated data is usually the most sensitive to changes, especially if the
change is sparse. Simulations also confirm this to be the case on average for
higher dimensions, but also that the sensitivity strongly varies with the pre-
change correlation matrix and the specific change. In future work it will be
interesting to exploit this knowledge about sensitivity for specific change
detection tasks.

## Acknowledgements

This work is partially funded by the Norwegian Research Council centre Big
Insight, project 237718.

## References

  * [1] José Camacho, Alejandro Pérez-Villegas, Pedro García-Teodoro and Gabriel Maciá-Fernández  “PCA-based multivariate statistical network monitoring for anomaly detection”  In _Computers & Security_ 59, 2016, pp. 118–137  DOI: 10.1016/j.cose.2016.02.008
  * [2] Fouzi Harrou et al.  “Improved principal component analysis for anomaly detection: Application to an emergency department”  In _Computers & Industrial Engineering_ 88, 2015, pp. 63–77  DOI: 10.1016/j.cie.2015.06.020
  * [3] Douglas M Hawkins and K. Zamba  “Statistical Process Control for Shifts in Mean or Variance Using a Changepoint Formulation”  In _Technometrics_ 47.2, 2005, pp. 164–173  DOI: 10.1198/004017004000000644
  * [4] Ling Huang et al.  “In-Network PCA and Anomaly Detection”  In _Advances in Neural Information Processing Systems 19_ MIT Press, 2007, pp. 617–624  URL: http://papers.nips.cc/paper/3156-in-network-pca-and-anomaly-detection.pdf
  * [5] J. Jackson and Robert H. Morris  “An Application of Multivariate Quality Control to Photographic Processing”  In _Journal of the American Statistical Association_ Vol 52, 1957  URL: https://amstat.tandfonline.com/doi/abs/10.1080/01621459.1957.10501378
  * [6] J. Jackson and Govind S. Mudholkar  “Control Procedures for Residuals Associated With Principal Component Analysis”  In _Technometrics_ 21.3, 1979, pp. 341–349  DOI: 10.1080/00401706.1979.10489779
  * [7] Harry Joe  “Generating random correlation matrices based on partial correlations”  In _Journal of Multivariate Analysis_ 97.10, 2006, pp. 2177–2189  DOI: 10.1016/j.jmva.2005.05.010
  * [8] Bart De Ketelaere, Mia Hubert and Eric Schmitt  “Overview of PCA-Based Statistical Process-Monitoring Methods for Time-Dependent, High-Dimensional Data”  In _Journal of Quality Technology_ 47.4, 2015, pp. 318–335  DOI: 10.1080/00224065.2015.11918137
  * [9] L.. Kuncheva and W.. Faithfull  “PCA Feature Extraction for Change Detection in Multidimensional Unlabeled Data”  In _IEEE Transactions on Neural Networks and Learning Systems_ 25.1, 2014, pp. 69–80  DOI: 10.1109/TNNLS.2013.2248094
  * [10] Anukool Lakhina, Mark Crovella and Christophe Diot  “Diagnosing Network-wide Traffic Anomalies”  In _Proceedings of the 2004 Conference on Applications, Technologies, Architectures, and Protocols for Computer Communications_ , SIGCOMM ’04  New York, USA: ACM, 2004, pp. 219–230  DOI: 10.1145/1015467.1015492
  * [11] D. Mishin, K. Brantner-Magee, F. Czako and A.. Szalay  “Real time change point detection by incremental PCA in large scale sensor data”  In _2014 IEEE High Performance Extreme Computing Conference (HPEC)_ , 2014, pp. 1–6  DOI: 10.1109/HPEC.2014.7040959
  * [12] Marco A.. Pimentel, David A. Clifton, Lei Clifton and Lionel Tarassenko  “A review of novelty detection”  In _Signal Processing_ 99, 2014, pp. 215–249  DOI: 10.1016/j.sigpro.2013.12.026
  * [13] Tiago Rato et al.  “A systematic comparison of PCA-based Statistical Process Monitoring methods for high-dimensional, time-dependent Processes”  In _AIChE Journal_ 62.5, 2016, pp. 1478–1493  DOI: 10.1002/aic.15062

##  A Appendix: Proofs

Before turning to the the proofs of the propositions in section 3, the
expressions for the pre- and post-change means and variances of each
projection is needed. The normalized eigenvectors (principal axes) and
corresponding eigenvalues (variance in the data along a given principal axis)
of $\boldsymbol{\Sigma}_{0}$ are quickly verified to be

$\begin{split}&\lambda_{1}=1+\rho,\quad\mathbf{v}_{1}=\frac{1}{\sqrt{2}}\begin{pmatrix}1\\\
1\end{pmatrix}\\\
&\lambda_{2}=1-\rho,\quad\mathbf{v}_{2}=\frac{1}{\sqrt{2}}\begin{pmatrix}-1\\\
1\end{pmatrix}\end{split}$ (2)

Note that which principal axis is the dominant one depends on the sign of
$\rho$. If $\rho$ is positive, $\mathbf{v}_{1}$ is the dominant one, but
$\mathbf{v}_{2}$ is dominant if $\rho$ is negative.

From the projections in (2), the parameters of the projections before and
after a change can be expressed as functions of the original correlation
matrix and multiplicative change factors. For the principal component, the
original and changed variances become as follows, respectively.

$\begin{split}o_{1}^{2}&=1+\rho\\\
c_{1}^{2}&=\frac{1}{2}a_{11}^{2}+\frac{1}{2}a_{22}^{2}+a_{11}a_{22}a_{12}\rho.\end{split}$
(3)

The expressions for the variances of the minor component are identical up to
one switched sign;

$\begin{split}o_{2}^{2}&=1-\rho\\\
c_{2}^{2}&=\frac{1}{2}a_{11}^{2}+\frac{1}{2}a_{22}^{2}-a_{11}a_{22}a_{12}\rho.\end{split}$
(4)

Observe that if $\rho<0$, then $o_{2}$ and $c_{2}$ would be equal to $o_{1}$
and $c_{1}$ with positive $\rho$, and vice versa. Thus, for $\rho\in(-1,1)$,
the general expressions are obtained by replacing $\rho$ with $|\rho|$.
Lastly, the changed mean components are given by

$\begin{split}m_{1}&=\frac{1}{\sqrt{2}}(\mu_{1}+\mu_{2})\\\
m_{2}&=\frac{1}{\sqrt{2}}(\mu_{1}-\mu_{2}).\end{split}$ (5)

We first prove Proposition 1 for changes in the mean.

###### Proof of Proposition 1.

Let $p_{1}(x)=N(x|0,o_{1}^{2})$, $q_{1}(x)=N(x|m_{1},o_{1}^{2})$,
$p_{2}(x)=N(x|0,o_{2}^{2})$ and $q_{2}(x)=N(x|m_{2},o_{2}^{2})$, where
$m_{i},o_{i}$ are as in (3), (4) and (5), with $\rho$ replaced by $|\rho|$ as
noted above. The Hellinger distances between the distributions before and
after a change along each principal axis is given by

$H_{j}^{2}=H^{2}(p_{j},q_{j})=1-\exp\left\\{-\frac{1}{8o_{j}^{2}}m_{j}^{2}\right\\}.$

Then some algebra results in the inequality we needed to prove;

$\displaystyle H_{2}$ $\displaystyle>H_{1}$
$\displaystyle(1+|\rho|)(\mu_{1}-\mu_{2})^{2}$
$\displaystyle>(1-|\rho|)(\mu_{1}+\mu_{2})^{2}$
$\displaystyle|\rho|(\mu_{1}-\mu_{2})^{2}+\mu_{1}\mu_{2}(2|\rho|-4)$
$\displaystyle>0$ $\displaystyle(\mu_{1}-\mu_{2})^{2}$
$\displaystyle>\mu_{1}\mu_{2}(4/|\rho|-2).$

From this inequality, the three special cases (i), (ii) and (iii) are
immediately given. ∎

In the proofs concerning changes in the covariance matrix, we will make use of
the following lemma. It reduces the inequality of Hellinger distances to a
simpler inequality of ratios of variances.

######  Lemma 5.

Let $p_{1},q_{1},p_{2},q_{2}$ be 0-mean normal distribution functions with
variances $\sigma^{2}_{p_{1}},\sigma^{2}_{q_{1}},\sigma^{2}_{p_{2}}$ and
$\sigma^{2}_{q_{2}}$, respectively. Furthermore, let

$\log
r_{j}=\left|\log\frac{\sigma^{2}_{q_{j}}}{\sigma^{2}_{p_{j}}}\right|,\quad
j=1,2.$

Then $H(p_{2},q_{2})>H(p_{1},q_{1})$ if and only if $\log r_{2}>\log r_{1}$.

###### Proof.

First observe that when the means are $0$ we can write the Hellinger distance
between two normal distributions as the following.

$\displaystyle H^{2}(p,q)$
$\displaystyle=1-\left(\frac{2\sigma_{p}\sigma_{q}}{\sigma_{p}^{2}+\sigma_{q}^{2}}\right)^{1/2}$
$\displaystyle=1-\sqrt{2}\left(\frac{\sigma_{p}}{\sigma_{q}}+\frac{\sigma_{q}}{\sigma_{p}}\right)^{-1/2}$
$\displaystyle=1-\sqrt{2}\left(\frac{\sigma_{p}^{2}}{\sigma_{q}^{2}}+\frac{\sigma_{q}^{2}}{\sigma_{p}^{2}}+2\right)^{-1/4}.$

This gives us the inequality

$\displaystyle H(p_{2},q_{2})$ $\displaystyle>H(p_{1},q_{1})$
$\displaystyle\frac{\sigma_{p_{2}}^{2}}{\sigma_{q_{2}}^{2}}+\frac{\sigma_{q_{2}}^{2}}{\sigma_{p_{2}}^{2}}$
$\displaystyle>\frac{\sigma_{p_{1}}^{2}}{\sigma_{q_{1}}^{2}}+\frac{\sigma_{q_{1}}^{2}}{\sigma_{p_{1}}^{2}}.$

By setting $r_{2}=\sigma_{p_{2}}^{2}/\sigma_{q_{2}}^{2}$ and
$r_{1}=\sigma_{p_{1}}^{2}/\sigma_{q_{1}}^{2}$, the inequality can be written
as

$r_{2}+r_{2}^{-1}>r_{1}+r_{1}^{-1}.$

Now assume first that $r_{1},r_{2}>1$, i.e.,
$\sigma_{p_{j}}^{2}>\sigma_{q_{j}}^{2}$. Then we see that

$\displaystyle r_{2}+r_{2}^{-1}$ $\displaystyle>r_{1}+r_{1}^{-1}$
$\displaystyle r_{2}-r_{1}+\frac{r_{1}-r_{2}}{r_{1}r_{2}}$ $\displaystyle>0$
$\displaystyle(r_{2}-r_{1})\Big{(}1-\frac{1}{r_{1}r_{2}}\Big{)}$
$\displaystyle>0.$

By the assumption that $r_{1},r_{2}>1$, this inequality holds if and only if
$r_{2}>r_{1}$.

Finally, note that by interchanging $\sigma_{p_{j}}^{2}$ and
$\sigma_{q_{j}}^{2}$, the same result is obtained when
$\sigma_{q_{j}}^{2}\geq\sigma_{p_{j}}^{2}$. Thus, to make the result hold in
general, we can set

$r_{j}=\exp\left\\{\Big{|}\log\frac{\sigma^{2}_{q_{j}}}{\sigma^{2}_{p_{j}}}\Big{|}\right\\},\quad
j=1,2,$

which is an expression for the ratio between variances where the largest of
the variances is always in the numerator. Therefore we get that $\log
r_{2}>\log r_{1}$ is equivalent to $H_{2}>H_{1}$. ∎

The rest of this article contain the individual proofs of the remaining
propositions in the main body of the text.

###### Proof of Proposition 2.

When assuming that $a_{12}=1$ and $a_{11}=a_{22}=a\not=1$, we get that

$\log
r_{2}=\left|\log\frac{a^{2}/2+a^{2}/2-|\rho|a^{2}}{1-|\rho|}\right|=|\log
a^{2}|,$

and

$\log
r_{1}=\left|\log\frac{a^{2}/2+a^{2}/2+|\rho|a^{2}}{1+|\rho|}\right|=|\log
a^{2}|.$

Hence, by arguments similar to the proof of Lemma 5, we see that $H_{2}=H_{1}$
no matter what $|\rho|$ or $a$ is. ∎

###### Proof of Proposition 3.

Using the formulas for the variances of the projections (3) and (4), the
inequality we have to study according to Lemma 5 becomes the following,

$\displaystyle\left|\log\frac{a^{2}-2a|\rho|+1}{2(1-|\rho|)}\right|$
$\displaystyle>\left|\log\frac{a^{2}+2a|\rho|+1}{2(1+|\rho|)}\right|$
$\displaystyle\left|\log\left[\frac{(1-a)^{2}}{2(1-|\rho|)}+a\right]\right|$
$\displaystyle>\left|\log\left[\frac{(1-a)^{2}}{2(1+|\rho|)}+a\right]\right|.$
(6)

First, we have to find out the sign of the expressions inside the absolute
values for each $a$ and $|\rho|$. For the left-hand side, we get

$\displaystyle\frac{(1-a)^{2}}{2(1-|\rho|)}+a=1$ $\displaystyle a=1\text{ and
}a=2|\rho|-1.$

Thus, for $a>1$ and $a<2|\rho|-1$, the left-hand side is positive, while
negative in between. For the right-hand side, the expression inside the
absolute value signs are positive for $a>1$ and $a<-(1+2|\rho|)$. Since $a>0$,
however, the relevant root for the right-hand side is only $a=1$. In total,
this gives us three regions of $(a,|\rho|)$-values to check inequality (6):
$a>1$ and $|\rho|\in(0,1)$, $a\in(2|\rho|-1,1)$ and $|\rho|\in(0,1)$, and
$a\in(0,2|\rho|-1)$ and $|\rho|\in(1/2,1)$.

###  $a>1$ and $|\rho|\in(0,1)$:

The absolute value signs can now be dissolved, so that inequality (6) becomes

$\frac{(1-a)^{2}}{(1-|\rho|)}>\frac{(1-a)^{2}}{(1+|\rho|)}.$

Since $|\rho|\in(0,1)$, we see that the inequality holds for any $a>1$. Hence,
$H_{2}>H_{1}$ in this scenario; when the variance increases.

###  $a\in(2|\rho|-1,1)$ and $|\rho|\in(0,1)$:

In this case, inequality (6) becomes

$\frac{(1-a)^{2}}{(1-|\rho|)}<\frac{(1-a)^{2}}{(1+|\rho|)}.$

I.e., it does not hold for any of the $a$’s or $|\rho|$’s within the relevant
region. Note that when $|\rho|<1/2$, $a$ is kept between $(0,1)$.

###  $a\in(0,2|\rho|-1)$ and $|\rho|\in(1/2,1)$:

Now we get the inequality

$\frac{(1-a)^{2}}{2(1-|\rho|)}+a>\left(\frac{(1-a)^{2}}{2(1+|\rho|)}+a\right)^{-1},$

which is equivalent to

$a^{4}-a^{2}(4\rho^{2}-2)+4\rho^{2}-3>0.$ (7)

The roots of the function on the left-hand side are $a=\pm 1$ and
$a=\pm\sqrt{4\rho^{2}-3}$, but the only relevant root for $a\in(0,2|\rho|-1)$
and $|\rho|\in(1/2,1)$ is $a_{0}:=\sqrt{4\rho^{2}-3}$.

Next, for $|\rho|<\sqrt{3}/2$ the root $a_{0}$ moves into the complex plane,
and the function on the left-hand side of (7) is always less than $0$ for the
relevant $a$’s. I.e., $H_{2}<H_{1}$ in this case. If $|\rho|>\sqrt{3}/2$, on
the other hand, then (7) holds for $a\in(0,a_{0})$, but not for
$a\in(a_{0},2|\rho|-1)$. ∎

###### Proof of Proposition 4.

In this scenario, the inequality to check due to Lemma 5 and expressions (3)
and (4) is

$\left|\log\frac{1-a|\rho|}{1-|\rho|}\right|>\left|\log\frac{1+a|\rho|}{1+|\rho|}\right|.$
(8)

To dissolve the absolute value signs we first have to see for which values of
$a$ and $|\rho|$ the expressions inside are positive or negative. It is easily
verified that the expression inside the left-hand side absolute value is
positive for $a<1$, while the right-hand side is positive if $a>1$, both being
negative otherwise.

First assume that $a<1$. Then inequality (8) becomes

$\displaystyle\frac{1-a|\rho|}{1-|\rho|}$
$\displaystyle>\frac{1+|\rho|}{1+a|\rho|}$ $\displaystyle 1-(a\rho)^{2}$
$\displaystyle>1-\rho^{2}$ $\displaystyle a^{2}$ $\displaystyle<1$

Hence, $a\in(-1,1)$ yields $H_{2}>H_{1}$. On the other hand, if $a>1$, we
obtain

$\displaystyle\frac{1-|\rho|}{1-a|\rho|}$
$\displaystyle>\frac{1+a|\rho|}{1+|\rho|}$ $\displaystyle a^{2}$
$\displaystyle>1,$

which is always true. Thus, in total, $H_{2}<H_{1}$ only if $a<-1$. ∎


# MultiPragEval: Multilingual Pragmatic Evaluation
of Large Language Models

Dojun Park1*, Jiwoo Lee1*, Seohyun Park1, Hyeyun Jeong1,
Youngeun Koo2, Soonha Hwang3, Seonwoo Park1, Sungeun Lee1
1Seoul National University, 2Sungkyunkwan University, 3Yonsei University
{dojun.parkk<EMAIL_ADDRESS>
{lee9055, tosirihy, su3503<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

As the capabilities of LLMs expand, it becomes increasingly important to
evaluate them beyond basic knowledge assessment, focusing on higher-level
language understanding. This study introduces MultiPragEval, a robust test
suite designed for the multilingual pragmatic evaluation of LLMs across
English, German, Korean, and Chinese. Comprising 1200 question units
categorized according to Grice’s Cooperative Principle and its four
conversational maxims, MultiPragEval enables an in-depth assessment of LLMs’
contextual awareness and their ability to infer implied meanings. Our findings
demonstrate that Claude3-Opus significantly outperforms other models in all
tested languages, establishing a state-of-the-art in the field. Among open-
source models, Solar-10.7B and Qwen1.5-14B emerge as strong competitors. This
study not only leads the way in the multilingual evaluation of LLMs in
pragmatic inference but also provides valuable insights into the nuanced
capabilities necessary for advanced language comprehension in AI systems.

**footnotetext: These authors contributed equally to this work.

##  1 Introduction

Understanding a language involves not only the ability to process explicit
information but also an awareness of the context that influences the meaning
of each utterance (Sperber and Wilson, 1986). In human communication, context
acts as a critical element as it provides a foundation upon which dialogue
participants can understand and interact with each other more efficiently.
With a shared context, communication becomes more facilitated, allowing subtle
nuances to be successfully conveyed, which is essential for engaging in
meaningful conversations (Krauss and Fussell, 1996).

With recent advancements in generative AI, current Large Language Models
(LLMs) have demonstrated capabilities that extend far beyond traditional
natural language processing (NLP) tasks (Brown et al., 2020; Achiam et al.,
2023). These models are increasingly becoming integral to our daily lives as
AI assistants, closely engaging with human users in diverse conversational
setups that demand a rapid understanding of the users’ needs and intentions,
far surpassing mere literal interpretation of text (Roller et al., 2021).
Given the growing importance of LLMs, accurately evaluating their ability to
comprehend context-dependent meanings and demonstrate human-like language
comprehension has become crucial (McCoy et al., 2019; Xu et al., 2020).

Aspect | Details
---|---
Utterance | "There’s the door."
| Literal
---
Meaning
A door is located over there.
| Contextual
---
Implication
|  Context: An interviewer says it
---
to the interviewee after finishing
an interview.
Implied Meaning: The interview
has concluded and the intervie-
wee is free to leave the room.
Table 1: Literal and contextual implications of the utterance “There’s the
door” in an interview scenario.

Pragmatics is a branch of linguistics that studies how language is used to
achieve specific goals, where the interpretation of utterances depends not
only on their literal meaning but also, crucially, on the surrounding context
(Grice, 1975). Consider the example in Table 1, which demonstrates both the
literal and implied meanings of the utterance, “There’s the door.” Literally,
this phrase simply indicates the presence of a door in the specified
direction. However, from a pragmatic standpoint, it conveys an additional
implied meaning in the context of its usage by an interviewer to an
interviewee after an interview has concluded. In this scenario, the speaker is
subtly suggesting that the interviewee is free to leave the room. This example
underscores the critical role that context plays in shaping the interpretation
of human language.

Despite the clear need for studies analyzing the pragmatic competence of
current LLMs, there is not only a lack of systematic evaluation across various
models (Chang et al., 2024) but also a strong bias towards English (Guo et
al., 2023; Bommasani et al., 2023), leaving the pragmatic abilities of LLMs in
other languages largely unexplored and difficult to compare. Such oversight
demonstrates a significant gap in current evaluation practices, particularly
given the multilingual nature of today’s state-of-the-art LLMs (Kwon et al.,
2023).

To address these challenges, our study introduces MultiPragEval, a
comprehensive test suite designed for the multilingual pragmatic evaluation of
LLMs in English, German, Korean, and Chinese. Our suite comprises 300 question
units per language, totaling 1200 units. These questions are divided into five
categories based on Grice’s Cooperative Principles and the corresponding four
conversational maxims: quantity, quality, relation, manner, and an additional
category dedicated to assessing mere literal meaning understanding.

Our main contributions are as follows:

  * •

Development of MultiPragEval: We introduce a comprehensive, multilingual test
suite aimed at evaluating the pragmatic abilities of LLMs across English,
German, Korean, and Chinese.

  * •

Systematic Evaluation of LLMs: We conduct a thorough evaluation of 15 state-
of-the-art LLMs, including both proprietary and open-source models, assessing
their contextual awareness and pragmatic understanding capabilities.

  * •

In-depth Performance Analysis: We offer a detailed analysis of LLM
performance, systematically categorized according to Grice’s Cooperative
Principle and its maxims, highlighting trends and performance discrepancies.

##  2 Related Work

#### Current Practices in LLM Evaluation.

Benchmarks serve as critical tools for standardized evaluation in the field of
LLM studies, enabling fair and systematic comparisons across models trained
with diverse architectures and strategies (Guo et al., 2023). These benchmarks
span a wide range of domains, from general reasoning (Zellers et al., 2019) to
specialized fields such as mathematics (Cobbe et al., 2021), coding (Chen et
al., 2021), and biomedical sciences (Jin et al., 2019). While comprehensive,
they primarily focus on assessing knowledge and logical reasoning, emphasizing
explicit semantic meanings over the contextual and implied meanings that can
vary in different scenarios (Sileo et al., 2022).

Leaderboards further enhance the field of LLM evaluation by providing a
transparent platform where the performance of various models can directly
compete with each other. The Open LLM Leaderboard (Beeching et al., 2023),
featuring a range of rigorous benchmarks, establishes a venue for open-source
models to showcase their capabilities, thereby fostering engagement in LLM
development among both individual developers and tech companies. Meanwhile,
Chatbot Arena (Chiang et al., 2024) is gaining recognition as a crowd-sourced
evaluation platform. It leverages real-time feedback from users who vote on
outputs from two randomly selected models. Models are then ranked on the
leaderboard based on their Elo rating (Elo and Sloan, 1978), thus filling the
gaps left by automatic benchmarks.

Recently, efforts have been made to create benchmarks specifically targeted at
measuring the capabilities of LLMs in languages such as Chinese (Li et al.,
2023) and Korean (Son et al., 2024). This development contributes to advancing
a more inclusive multilingual evaluation landscape.

Language |  Context |  Utterance |  MCQ
---|---|---|---
English |  While visiting Charlie’s house, Emily saw a large pile of oranges in the kitchen and asked why there were so many. Charlie responded: |  "My uncle lives in Florida." |  Choose the most appropriate meaning of the above utterance from the following options.
(A) Charlie’s uncle sent the oranges.
(B) Charlie’s uncle resides in Florida.
(C) People in Florida do not like oranges.
(D) Charlie’s uncle lives in a rural house.
(E) None of the above.
German |  Anna, die Felix besuchte, sah, dass es bei Felix viel Wein gab, und als sie fragte, warum es so viel Wein gab, wie er zu so viel Wein komme, sagte Felix: |  "Mein Onkel betreibt ein Weingut in Freiburg." |  Wählen Sie die passendste Bedeutung der obigen Äußerung aus den folgenden Aussagen aus.
(A) Felix hat den Wein von seinem Onkel.
(B) Der Onkel von Felix lebt in Freiburg.
(C) Freiburger lieben keinen Wein.
(D) Der Onkel von Felix wohnt in einem Landhaus.
(E) Keine der obigen Aussagen ist richtig.
Korean |  철수 집에 놀러 간 영희는 주방에 많은 귤이 쌓여 있는 것을 보고 귤이 왜 이렇게 많은지 물었고 철수는 다음과 같이 말했다. |  "우리 작은 아버지께서 제주도에 사셔." |  다음 보기에서 위 발화가 갖는 가장 적절한 의미를 고르세요.
(A) 작은 아버지께서 귤을 보내주었다.
(B) 작은 아버지의 거주지는 제주도이다.
(C) 제주도 사람들은 귤을 좋아하지 않는다.
(D) 작은 아버지께서 전원 주택에 사신다.
(E) 정답 없음.
Chinese |  王芳去张伟家看到厨房里堆放着几大袋葡萄干，便问为什么有这么多，张伟回答说： |  "我叔叔住在新疆。" |  请在以下选项中选择最恰当地表达上述话语含义的选项。
(A) 叔叔给张伟邮了葡萄干。
(B) 张伟的叔叔住在新疆。
(C) 新疆人不喜欢葡萄干。
(D) 张伟的叔叔住在乡间别墅里。
(E) 没有正确答案。
Table 2: Multilingual test units from the test suite on the maxim of relation,
comprising a context, an utterance, and a multiple-choice question (MCQ) to
assess the understanding of implied meanings. Option (A) indicates the most
appropriate interpretation for each scenario.

#### Pragmatic Evaluation of LLMs.

As LLMs continue to evolve, it has become crucial to evaluate how effectively
they consider context, which crucially shapes meanings beyond their literal
interpretations. (Bojic et al., 2023) examined multiple LLMs under the
framework of Grice’s Cooperative Principle and its conversational maxims to
assess their capabilities in understanding implicature. The results
demonstrated that GPT-4 (Achiam et al., 2023) outperformed other models,
including human performance. However, the human participants were not native
English speakers but educated individuals from Serbia, which potentially
limits the impact of the findings.

(di San Pietro et al., 2023) conducted a comparable study focusing on GPT-3.5,
leveraging the APACS test set (Arcara and Bambini, 2016), which consists of
various subtasks such as interviews, descriptions, and narratives. The tests
were conducted in both English and Italian, with results reported for Italian
due to no notable differences between the two. The findings indicate that
GPT-3.5 comes close to human ability but reveals weaknesses in understanding
physical metaphors and jokes.

Focusing on Korean, (Park et al., 2024) employed 120 test questions aligned
with the four Gricean maxims to further probe the capabilities of various
LLMs. The findings demonstrate that GPT-4 excelled in both multiple-choice and
open-ended question setups, with HyperCLOVA X (Yoo et al., 2024), a Korean-
specific LLM, closely following. The study also explored in-context learning,
demonstrating that the few-shot learning technique consistently leads to
positive outcomes across all tested models.

(Sravanthi et al., 2024) introduce a comprehensive pragmatic benchmark that
evaluates LLMs across 14 distinct tasks, including implicature, presupposition
and deictic detection. Comprising 28k data points, this benchmark aims to
provide a nuanced assessment of LLMs’ pragmatic abilities, marking a
substantial contribution to the field. Yet, there remains a significant need
to extend these evaluations to multiple languages to thoroughly assess the
multilingual capabilities of LLMs.

##  3 Methodology

###  3.1 Theoretical Foundations of Pragmatics

To accurately assess the contextual awareness of LLMs, we primarily focus on
implicature, based on Grice’s theory (Grice, 1975). Implicature refers to a
specific way language is used, in which the literal meaning of an utterance
differs from the intended meaning of the speaker, requiring the listener to
infer the intended meaning from the surrounding context. This concept is
critical for evaluating how well LLMs understand human language, particularly
in their ability to capture nuanced meanings beyond the explicit words.

Grice introduced the Cooperative Principle and its four conversational maxims,
which suggest how an utterance should desirably be conducted. Detailed in
Table 3, the maxim of quantity requires information to be as informative as
necessary–neither more nor less. The maxim of quality emphasizes the
importance of offering truthful contributions. The maxim of relation ensures
all information is pertinent to the current conversation. The maxim of manner
demands clarity and brevity, avoiding obscurity and ambiguity.

###  3.2 Development of the Test Suite

Maxim |  Description
---|---
Quantity |  Make your contribution as informative as is required.
Quality |  Try to make your contribution one that is true.
Relation |  Ensure that all the information you provide is relevant to the current conversation.
Manner |  Be perspicuous; Be brief and orderly, and avoid obscurity and ambiguity.
Table 3: Summary of Grice’s conversational maxims and their key principles

The development of the MultiPragEval test suite began with the foundational
work by (Park et al., 2024), who crafted a set of 120 question units designed
to assess LLMs in terms of four conversational maxims. Each maxim was
represented by 30 units, which included a structured scenario setting the
conversational context, an utterance by a participant, and a set of questions
comprising both a multiple-choice question and an open-ended question. We
adopted the context, utterance, and multiple-choice question components from
this test set as our starting point.

In the next phase, we expanded the number of question units from 120 to 300 to
encompass a broader range of pragmatic contexts. Each conversational maxim,
originally represented by 30 units, was doubled to 60 to deepen the evaluative
scope. Additionally, we introduced a new category specifically designed to
assess the understanding of literal meanings, which allows us to explore
potential trade-offs between performances in understanding literal versus
implied meanings. To further enhance the complexity of our test suite, we
included units that do not have a correct answer by adding a ‘None of the
above’ option to the multiple-choice setups.

In the subsequent phase, we translated the Korean test set into English,
German, and Chinese using DeepL111https://www.deepl.com for the initial
conversion. Then, Korean-native linguistic experts with CEFR C1 level
proficiency222https://www.coe.int/en/web/common-european-framework-reference-
languages in the target languages refined the translations to ensure that
these translations preserved the intended meanings and nuances. They also
adapted cultural elements by substituting the names of characters and setting
details to reflect the local context of each language. Finally, native
speakers of each target language, who hold degrees in linguistics and related
fields, conducted a thorough verification of the translations. This process
confirmed that the quality and accuracy of the translations were on par with
the original Korean versions. Table 2 showcases an example of a test unit
focused on the maxim of relation from our test suite, presented in English,
German, Korean, and Chinese.

###  3.3 Experimental Setup

Type | Model | Version
---|---|---
Proprietary | GPT-3.5 | turbo-0125
GPT-4 | turbo-2024-04-09
Claude3-Haiku | haiku-20240307
Claude3-Sonnet | sonnet-20240229
Claude3-Opus | opus-20240229
Mistral-small | small-2402
Mistral-medium | medium-2312
Mistral-large | large-2402
Open-Src. | Llama-2-13B | chat-hf
Llama-2-7B | chat-hf
Llama-3-8B | Instruct
Gemma-7B | 1.1-7b-it
Solar-10.7B | Instruct-v1.0
Qwen-14B | 1.5-14B-Chat
Qwen-7B | 1.5-7B-Chat
Table 4: Overview of proprietary and open-source LLMs evaluated in the study | English |  |  |
---|---|---|---|---
| Quan. | Qual. | Rel. | Man. | Avg. |  German Avg. |  Korean Avg. |  Chinese Avg.
GPT-4 | 65.00 | 83.89 | 82.22 | 70.00 | 75.28 | 72.50 | 81.25 | 68.75
GPT-3.5 | 51.11 | 66.67 | 52.78 | 42.89 | 53.61 | 52.92 | 38.89 | 43.61
Claude3-Opus | 81.11 | 88.89 | 88.89 | 81.11 | 85.00 | 82.78 | 87.08 | 76.67
Claude3-Sonnet | 62.22 | 81.67 | 67.22 | 54.44 | 66.39 | 60.14 | 63.33 | 48.61
Claude3-Haiku | 56.67 | 67.78 | 58.89 | 43.33 | 56.67 | 45.14 | 38.47 | 40.83
Mistral-Large | 61.11 | 71.11 | 61.11 | 52.22 | 61.39 | 63.75 | 65.56 | 54.72
Mistral-Medium | 61.11 | 69.44 | 72.22 | 62.22 | 66.25 | 53.61 | 52.92 | 38.89
Mistral-Small | 57.22 | 57.78 | 54.44 | 35.00 | 51.11 | 51.11 | 40.42 | 33.61
Llama3-8B | 54.44 | 68.89 | 44.44 | 45.56 | 53.33 | 40.00 | 32.50 | 46.81
Llama2-13B | 26.67 | 32.22 | 16.67 | 32.22 | 26.94 | 16.39 | 47.50 | 8.75
Llama2-7B | 31.11 | 26.67 | 11.11 | 18.33 | 21.81 | 4.44 | 3.06 | 4.17
Gemma-7B | 37.78 | 36.67 | 35.00 | 30.56 | 35.00 | 27.22 | 20.83 | 25.28
Solar-10.7B | 58.33 | 65.56 | 62.22 | 51.11 | 59.31 | 55.69 | 49.03 | 46.39
Qwen-14B | 52.22 | 61.67 | 56.11 | 43.33 | 53.33 | 43.06 | 49.72 | 50.00
Qwen-7B | 53.89 | 62.22 | 47.22 | 37.78 | 50.28 | 39.44 | 35.14 | 41.11
Table 5: Performance of LLMs on the MultiPragEval test suite: scores across
four languages and by maxims with overall averages; Leading scores among
proprietary and open-source models are highlighted in bold. The scores for
each maxim are color-coded in shades of blue to represent the relative ranking
within each model.

#### Models.

Our study includes 15 LLMs, categorized into two types: proprietary LLMs
accessed via API, and open-source LLMs where we have direct access to the
model weights. As detailed in Table 4, the proprietary models comprise two GPT
models (Achiam et al., 2023) by OpenAI, and three different sizes each of
Claude3 (Anthropic, 2024) by Anthropic and Mistral by Mistral AI
333https://mistral.ai/. We exclude Gemini from our analysis due to its limited
accessibility via API.

Additionally, we evaluate publicly available open-source models, each with
approximately 10 billion parameters, including three Llama models (Touvron et
al., 2023) by Meta, Gemma (Team et al., 2024) by Google, Solar (Kim et al.,
2023) by Korean company Upstage, and two Qwen models (Bai et al., 2023) by
Chinese firm Alibaba.

#### LLM Response Generation.

To generate answers from each LLM, we set the temperature hyperparameter at
0.5 to maintain consistency across models. For inference on the open-source
LLMs, we utilized a single H100-80GB unit. Each model was queried three times
to account for the inherent randomness in responses. We then computed the
average score for each model across these trials to ensure a robust assessment
of performance for each LLM iteration.

##  4 Result

###  4.1 Analysis of LLM Performance

Figure 1: Breakdown of LLM scores for ‘No Correct Answers’ and literal meaning
tests across four languages; the heatmap uses two colors–blue indicating
higher scores and yellow indicating lower scores.

#### Overall Performance.

Table 5 presents the results from the evaluation of the selected LLMs on the
MultiPragEval test suite. It demonstrates that Claude3-Opus significantly
outperforms all other models across four languages, with GPT-4 trailing by
approximately 6-10 points. This performance gap underscores Claude3-Opus’s
exceptional ability to capture the subtle nuances of language that are highly
context-dependent. These findings highlight its position as the most
proficient among the current state-of-the-art LLMs across English, German,
Korean, and Chinese.

Mistral-Large and Claude3-Sonnet are closely matched for the next tier of
performance; Mistral-Large outperforms Claude3-Sonnet in German, Korean, and
Chinese. However, Claude3-Sonnet achieves a higher score in English,
registering 66.39 compared to Mistral-Large’s 61.39. Interestingly, while
Mistral-Large generally shows improved scores across languages compared to
Mistral-Medium, it scores lower in English, dropping to 61.39 from the medium-
sized model’s 66.25.

Solar-10.7B demonstrates stable performance, consistently outperforming
GPT-3.5 across all four languages. It is the only open-source model that
surpasses GPT-3.5 in both English and German. In English, it closely follows
Mistral-Large with a score of 59.31 and is just behind Claude3-Sonnet in
German, with a score of 55.69.

Qwen-14B also stands out among other open-source LLMs, outperforming its
counterparts with scores of 50.00 in Chinese and 49.72 in Korean. In contrast,
both Llama2-13B and Llama2-7B demonstrate a strong bias towards literal
interpretations yielding poor scores, while Llama3-8B shows enhanced
performance compared to its earlier versions. Notably, Llama2-13B achieves a
significant leap in Korean, scoring 47.50 compared to Llama2-7B’s 3.06, while
exhibiting a more gradual increase in other languages.

#### Closer Look at Individual Maxims.

Table 5 also shows the performance scores of LLMs on individual maxims in the
English test suite. We observe a consistent pattern across LLMs where scores
for the maxim of quality generally rank highest, while scores for the maxim of
manner rank lowest. This pattern is not unique to English but is also
observable in other languages, suggesting a universal trend (see Appendix A).
This outcome is expected because expressions governed by the maxim of quality,
which become untrue statements when interpreted literally, make it easier for
LLMs to infer the appropriate implied meanings. Conversely, the maxim of
manner, involving verbose or ambiguous expressions, poses more subtle
challenges that likewise pose difficulties for humans (Hoffmann, 2010).

Another noteworthy observation is that as the overall performance increases,
the scores for the maxim of relation also significantly improve. This pattern
is clearly evident among proprietary models, where the maxim of relation
consistently ranks second. Similarly, Solar-10.7B and Qwen-14B, which perform
comparably to GPT-3.5, achieve higher scores in the maxim of relation compared
to those of quantity and manner. Conversely, other open-source models with
lower average scores tend to have lower rankings in the maxim of relation,
falling below the maxim of quantity. This suggests that the ability to
effectively consider relevant information is generally associated with better
performance in overall pragmatic inference.

Model |  | MultiPragEval
---
(Eng.)
| MMLU
---
5-shot
| MATH
---
4-shot
Arena Elo* |  | ARC
---
25-shot
| HumanEval
---
0-shot
| GSM-8K
---
8-shot
GPT-4 | 75.28 | 86.40 | 52.90 | 1252 | 96.30 | 67.00 | 92.00
GPT-3.5 | 53.61 | 70.00 | 34.10 | 1110 | 85.20 | 48.10 | 57.10
Claude3-Opus | 85.00 | 86.80 | 61.00 | 1246 | 96.40 | 84.90 | 95.00
Claude3-Sonnet | 66.39 | 79.00 | 40.50 | 1199 | 93.20 | 73.00 | 92.30
Claude3-Haiku | 56.67 | 75.20 | 40.90 | 1181 | 89.20 | 75.90 | 88.90
Llama3-8B | 53.33 | 68.40 | 30.00 | 1154 | 60.75 | 62.20 | 79.60
Llama2-13B | 26.94 | 47.80 | 6.70 | 1065 | 59.39 | 14.00 | 77.40
Llama2-7B | 21.81 | 34.10 | 3.80 | 1042 | 53.07 | 7.90 | 25.70
Gemma-7B | 35.00 | 66.03 | 24.30 | 1091 | 61.09 | 32.30 | 46.40
Qwen-14B | 53.33 | 69.39 | 24.80 | 1119 | 56.57 | 32.30 | 61.30
Qwen-7B | 50.28 | 61.70 | 11.60 | 1079 | 54.18 | 29.90 | 51.70
Kendall $\tau$ | 1.00 | 0.95 | 0.92 | 0.84 | 0.81 | 0.80 | 0.73
Table 6: Performance scores of LLMs across multiple benchmarks and Kendall’s
Tau correlation Coefficients Relative to MultiPragEval.
* The Arena Elo scores are as of May 17, 2024.

###  4.2 Assessing the Stability of Pragmatic Inference

We further explore the stability of LLMs in pragmatic inference under two
specific setups. First, we evaluate the models on a subset of each category of
maxims, specifically designed where the test questions lack an appropriate
answer. This subset is intended to be more challenging as it requires the
models to identify incorrect interpretations and select the option ‘(E) None
of the above’ without reference to a correct meaning. Secondly, we test the
models on additional test units consisting of context, utterance, and
question, structured similarly, but where the context is irrelevant to the
utterance. This setup is designed to assess whether LLMs can accurately
distinguish purely literal meanings from inappropriate interpretations.

#### Subset of No Correct Answer.

Figure 1 illustrates that the scores on the subset without correct answers
(Opt. None) generally align with the overall scores, yet they reveal subtle
differences in performance details. While Claude3-Opus consistently
outperforms GPT-4 by a certain margin in overall scores across all languages,
GPT-4 closes the gap by surpassing Claude3-Opus by approximately 5 points in
both German and Korean. This result indicates that both models are comparably
robust in the challenging setup of pragmatic consideration.

It is evident that models with lower overall scores exhibit significant
declines when tested in the setup without a correct answer. Among proprietary
LLMs, Claude3-Haiku, along with medium and small-sized models by Mistral,
notably drop in scores, indicating their struggles with the task. Similarly,
7-billion parameter models such as Llama2, Gemma, and Qwen also show poor
performance, underscoring the complexity of the task for models of this size.

#### Additional Set of Literal Meaning.

The scores on the set asking literal meanings also demonstrate a general
increase along with the overall scores. While the flagship models of GPT and
Claude show performance close to perfect, the Llama2 models decrease,
particularly Llama2-7B, which demonstrates the lowest scores among open-source
LLMs–42.22, 45.56, and 49.44 for Korean, German, and English, respectively.

We observe an interesting trade-off between the scores on ‘No Correct Answers’
questions and those on the questions of literal meaning for Claude3-Sonnet. In
Chinese, Claude3-Sonnet even outperforms its flagship version by 4.45 points
in the subset without correct answers. However, its score on the test set of
literal meanings dramatically drops to 34.44, showing the lowest performance
across the board. This indicates an excessive inclination toward implied
meanings, even when the given context is irrelevant to the utterance.

###  4.3 Comparison with Existing Benchmarks

To further delve into the implications of our findings, we compare the results
from our English test suite with existing English-based benchmarks. This
analysis encompasses scores from 11 models, for which other benchmark scores
were publicly available. We consider seven popular benchmarks: MMLU (Hendrycks
et al., 2020) and ARC (Clark et al., 2018) for general reasoning, HumanEval
(Chen et al., 2021) for coding, GSM-8K (Cobbe et al., 2021) and MATH
(Hendrycks et al., 2021) for mathematics, and Chatbot Arena (Chiang et al.,
2024), a crowd-sourced evaluation. We opted to calculate the correlation
coefficients using Kendall’s Tau (Kendall, 1938) due to its better handling of
varying ranges and subtle differences between benchmarks.

The correlations of MultiPragEval with other benchmarks consistently show high
values, indicating a general trend toward ‘good’ performance across different
benchmarks. This suggests that improvements in a model’s performance on one
task generally enhance its performance on other tasks (Raffel et al., 2020).

MMLU and MATH exhibit the highest correlations among other benchmarks,
suggesting that the abilities assessed by these benchmarks align closely with
those required for pragmatic inference. It is anticipated that MMLU, which
evaluates the general language understanding capabilities of LLMs across a
broad spectrum of disciplines, reflects the ability to consider contextual
information in language, which is a key requirement of MultiPragEval.

However, the high correlation observed with the MATH benchmark is surprising,
given its primary focus on mathematical reasoning. Notably, the score gap
between Claude3-Opus and GPT-4, which is around 10 points on MultiPragEval, is
similarly reflected on MATH but not distinctively on MMLU. This pattern
suggests that the sophisticated mathematical problem-solving required by
MATH–which demands a higher level of logical reasoning compared to the basic
mathematical problems in GSM-8K–may also tap into core capabilities essential
for pragmatic inference. This connection between mathematical reasoning and
high-level linguistic comprehension indicates an intricate relationship that
requires deeper investigation.

##  5 Conclusion

In this study, we explore the capabilities of LLMs in pragmatic understanding,
particularly in the context of Grice’s theory of conversational implicature.
We introduce the MultiPragEval test suite, consisting of 1200 question units
designed to challenge LLMs’ contextual considerations across English, German,
Korean, and Chinese. Our findings demonstrate the usefulness of this test
suite in distinguishing the levels of comprehension among various proprietary
and open-source models.

The results reveal that among the models evaluated, Claude3-Opus and GPT-4
particularly stand out, with Claude3-Opus consistently outperforming GPT-4 by
6 to 10 points across all languages, thereby affirming its state-of-the-art
capability in pragmatic understanding. Among the open-source models,
Solar-10.7B leads in English and German, while Qwen-14B demonstrates superior
performance in Korean and Chinese. Notably, Solar-10.7B consistently
outperforms GPT-3.5 across all four languages underscoring its robustness and
adaptability.

The fine-grained analysis of individual Gricean maxims highlights a general
trend among LLMs: the maxim of quality is consistently the easiest to infer,
while the maxim of manner proves to be the most challenging. Furthermore, the
analysis shows that performance on the maxim of relation correlates closely
with overall model performance, highlighting the critical importance of
considering the relevance of utterances within the given context for overall
pragmatic inference.

Comparative analysis of our findings with existing benchmarks illustrates the
highest correlations with MMLU and MATH, suggesting that general language
understanding and complex logical reasoning are intricately linked to
pragmatic inference abilities. This insight leads us to further research,
focusing on training LLMs on a variety of tasks including sophisticated
mathematical problems, to empirically demonstrate how these abilities relate
to pragmatic reasoning.

## Limitations

While our study provides a comprehensive comparison of 15 proprietary and
open-source models, it does not include a comparison with human performance.
Including human performance would offer deeper insights into how closely LLMs
approximate human abilities. Moreover, human performance can vary across
languages, which would enrich our understanding of the LLMs’ multilingual
pragmatic abilities. Recognizing this gap, we aim to incorporate human
performance comparisons in our future research.

Another limitation of our study is its exclusive focus on implicature, despite
pragmatics encompassing a broader range of phenomena such as speech acts,
presupposition, and politeness. This focus was chosen due to the increasing
role of LLMs as AI assistants, which often need to interpret human expressions
that are frequently conveyed implicitly. The ability of LLMs to capture these
subtle nuances directly influences human judgments about the quality of these
systems. Furthermore, contextual awareness is critical not only for linguists
but also for NLP engineers who aim to provide reliable services to users. We
believe that our specific focus on implicature provides valuable insights into
how effectively current LLMs manage the complexities inherent in interpreting
implied meanings, a crucial aspect of human communication.

## Ethics Statement

In this work, we introduce a test suite designed to evaluate the pragmatic
abilities of LLMs. We have ensured that all data created for this study does
not infringe on any existing intellectual property rights, while also ensuring
it contains no personally identifiable information. Linguistic experts were
involved in the creation and translation of the test suite; all contributors
were fully informed about the research’s purpose and the methods employed. We
commit to making the dataset publicly available to foster transparency and
further research in the field.

## References

  * Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_.
  * Anthropic (2024) AI Anthropic. 2024.  The claude 3 model family: Opus, sonnet, haiku.  _Claude-3 Model Card_.
  * Arcara and Bambini (2016) Giorgio Arcara and Valentina Bambini. 2016.  A test for the assessment of pragmatic abilities and cognitive substrates (apacs): Normative data and psychometric properties.  _Frontiers in psychology_ , 7:172889.
  * Bai et al. (2023) Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023.  Qwen technical report.  _arXiv preprint arXiv:2309.16609_.
  * Beeching et al. (2023) Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. 2023.  Open llm leaderboard.  https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard.
  * Bojic et al. (2023) Ljubisa Bojic, Predrag Kovacevic, and Milan Cabarkapa. 2023.  Gpt-4 surpassing human performance in linguistic pragmatics.  _arXiv preprint arXiv:2312.09545_.
  * Bommasani et al. (2023) Rishi Bommasani, Percy Liang, and Tony Lee. 2023.  Holistic evaluation of language models.  _Annals of the New York Academy of Sciences_ , 1525(1):140–146.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Chang et al. (2024) Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2024.  A survey on evaluation of large language models.  _ACM Transactions on Intelligent Systems and Technology_ , 15(3):1–45.
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021.  Evaluating large language models trained on code.  _arXiv preprint arXiv:2107.03374_.
  * Chiang et al. (2024) Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. 2024.  Chatbot arena: An open platform for evaluating llms by human preference.
  * Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018.  Think you have solved question answering? try arc, the ai2 reasoning challenge.  _arXiv preprint arXiv:1803.05457_.
  * Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021.  Training verifiers to solve math word problems.  _arXiv preprint arXiv:2110.14168_.
  * di San Pietro et al. (2023) Chiara Barattieri di San Pietro, Federico Frau, Veronica Mangiaterra, and Valentina Bambini. 2023.  The pragmatic profile of chatgpt: assessing the pragmatic skills of a conversational agent.
  * Elo and Sloan (1978) Arpad E Elo and Sam Sloan. 1978.  The rating of chessplayers: Past and present.
  * Grice (1975) Herbert P Grice. 1975.  Logic and conversation.  In _Speech acts_ , pages 41–58. Brill.
  * Guo et al. (2023) Zishan Guo, Renren Jin, Chuang Liu, Yufei Huang, Dan Shi, Linhao Yu, Yan Liu, Jiaxuan Li, Bojian Xiong, Deyi Xiong, et al. 2023.  Evaluating large language models: A comprehensive survey.  _arXiv preprint arXiv:2310.19736_.
  * Hendrycks et al. (2020) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020.  Measuring massive multitask language understanding.  _arXiv preprint arXiv:2009.03300_.
  * Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021.  Measuring mathematical problem solving with the math dataset.  In _Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks_ , volume 1.
  * Hoffmann (2010) Ludger Hoffmann. 2010.  _Sprachwissenschaft: ein Reader_.  de Gruyter.
  * Jin et al. (2019) Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019.  PubMedQA: A dataset for biomedical research question answering.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2567–2577, Hong Kong, China. Association for Computational Linguistics.
  * Kendall (1938) Maurice G Kendall. 1938.  A new measure of rank correlation.  _Biometrika_ , 30(1/2):81–93.
  * Kim et al. (2023) Dahyun Kim, Chanjun Park, Sanghoon Kim, Wonsung Lee, Wonho Song, Yunsu Kim, Hyeonwoo Kim, Yungi Kim, Hyeonju Lee, Jihoo Kim, et al. 2023.  Solar 10.7 b: Scaling large language models with simple yet effective depth up-scaling.  _arXiv preprint arXiv:2312.15166_.
  * Krauss and Fussell (1996) Robert M Krauss and Susan R Fussell. 1996.  Social psychological models of interpersonal communication.  _Social psychology: Handbook of basic principles_ , pages 655–701.
  * Kwon et al. (2023) Sang Kwon, Gagan Bhatia, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023.  Beyond English: Evaluating LLMs for Arabic grammatical error correction.  In _Proceedings of ArabicNLP 2023_ , pages 101–119, Singapore (Hybrid). Association for Computational Linguistics.
  * Li et al. (2023) Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. 2023.  Cmmlu: Measuring massive multitask language understanding in chinese.  _arXiv preprint arXiv:2306.09212_.
  * McCoy et al. (2019) R. Thomas McCoy, Ellie Pavlick, and Tal Linzen. 2019.  Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference.  In _Annual Meeting of the Association for Computational Linguistics_.
  * Park et al. (2024) Dojun Park, Jiwoo Lee, Hyeyun Jeong, Seohyun Park, and Sungeun Lee. 2024.  Pragmatic competence evaluation of large language models for korean.  _arXiv preprint arXiv:2403.12675_.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of machine learning research_ , 21(140):1–67.
  * Roller et al. (2021) Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Eric Michael Smith, Y-Lan Boureau, and Jason Weston. 2021.  Recipes for building an open-domain chatbot.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 300–325, Online. Association for Computational Linguistics.
  * Sileo et al. (2022) Damien Sileo, Philippe Muller, Tim Van de Cruys, and Camille Pradel. 2022.  A pragmatics-centered evaluation framework for natural language understanding.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 2382–2394, Marseille, France. European Language Resources Association.
  * Son et al. (2024) Guijin Son, Hanwool Lee, Sungdong Kim, Seungone Kim, Niklas Muennighoff, Taekyoon Choi, Cheonbok Park, Kang Min Yoo, and Stella Biderman. 2024.  Kmmlu: Measuring massive multitask language understanding in korean.  _arXiv preprint arXiv:2402.11548_.
  * Sperber and Wilson (1986) Dan Sperber and Deirdre Wilson. 1986.  _Relevance: Communication and cognition_, volume 142.  Harvard University Press Cambridge, MA.
  * Sravanthi et al. (2024) Settaluri Lakshmi Sravanthi, Meet Doshi, Tankala Pavan Kalyan, Rudra Murthy, Pushpak Bhattacharyya, and Raj Dabre. 2024.  Pub: A pragmatics understanding benchmark for assessing llms’ pragmatics capabilities.  _arXiv preprint arXiv:2401.07078_.
  * Team et al. (2024) Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. 2024.  Gemma: Open models based on gemini research and technology.  _arXiv preprint arXiv:2403.08295_.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Xu et al. (2020) Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, and Emily Dinan. 2020.  Recipes for safety in open-domain chatbots.  _arXiv preprint arXiv:2010.07079_.
  * Yoo et al. (2024) Kang Min Yoo, Jaegeun Han, Sookyo In, Heewon Jeon, Jisu Jeong, Jaewook Kang, Hyunwook Kim, Kyung-Min Kim, Munhyong Kim, Sungju Kim, et al. 2024.  Hyperclova x technical report.  _arXiv preprint arXiv:2404.01954_.
  * Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019.  HellaSwag: Can a machine really finish your sentence? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4791–4800, Florence, Italy. Association for Computational Linguistics.

##  Appendix A Appendix

|  | German
---|---|---
|  | Quan. | Qual. | Rel. | Man. | Avg.
| GPT-4 | 70.56 | 76.67 | 77.22 | 65.56 | 72.50
| GPT-3.5 | 58.89 | 51.67 | 53.89 | 47.22 | 52.92
| Claude-Opus | 85.56 | 87.78 | 85.00 | 72.78 | 82.78
| Claude-Sonnet | 53.89 | 70.00 | 66.11 | 50.56 | 60.14
| Claude-Haiku | 36.67 | 51.67 | 52.78 | 39.44 | 45.14
| Mistral-Large | 60.00 | 70.00 | 73.33 | 51.67 | 63.75
| Mistral-Medium | 47.22 | 68.89 | 56.11 | 42.22 | 53.61
Proprietary | Mistral-Small | 50.56 | 53.33 | 58.89 | 41.67 | 51.11
| Llama3-8B | 35.56 | 40.00 | 46.67 | 37.78 | 40.00
| Llama2-13B | 20.00 | 13.33 | 15.00 | 17.22 | 16.39
| Llama2-7B | 5.56 | 3.89 | 3.33 | 5.00 | 4.44
| Gemma-7B | 29.44 | 23.89 | 35.00 | 20.56 | 27.22
| Solar-10B | 56.67 | 59.44 | 62.78 | 43.89 | 55.69
| Qwen-14B | 53.89 | 38.89 | 45.56 | 33.89 | 43.06
Open-Source | Qwen-7B | 45.56 | 37.78 | 41.11 | 33.33 | 39.44
Table 7: Performance scores on the MultiPragEval test suite across four maxims with overall averages for German. While the maxim of manner generally shows the lowest scores, high scores are more evenly distributed across the other three maxims. |  | Korean
---|---|---
|  | Quan. | Qual. | Rel. | Man. | Avg.
| GPT-4 | 81.67 | 86.67 | 85.56 | 71.11 | 81.25
| GPT-3.5 | 42.22 | 47.22 | 37.22 | 28.89 | 38.89
| Claude-Opus | 86.67 | 87.78 | 93.33 | 80.56 | 87.08
| Claude-Sonnet | 58.89 | 74.44 | 67.78 | 52.22 | 63.33
| Claude-Haiku | 37.22 | 49.44 | 37.78 | 29.44 | 38.47
| Mistral-Large | 67.78 | 68.33 | 74.44 | 51.67 | 65.56
| Mistral-Medium | 59.44 | 51.11 | 53.89 | 47.22 | 52.92
Proprietary | Mistral-Small | 41.11 | 52.22 | 42.78 | 25.56 | 40.42
| Llama3-8B | 34.44 | 39.44 | 31.11 | 25.00 | 32.50
| Llama2-13B | 45.00 | 61.11 | 42.22 | 41.67 | 47.50
| Llama2-7B | 5.56 | 5.00 | 0.00 | 1.67 | 3.06
| Gemma-7B | 30.56 | 15.00 | 25.00 | 12.78 | 20.83
| Solar-10B | 52.78 | 52.22 | 57.22 | 33.89 | 49.03
| Qwen-14B | 53.33 | 58.89 | 44.44 | 42.22 | 49.72
Open-Source | Qwen-7B | 36.67 | 35.56 | 38.33 | 30.00 | 35.14
Table 8: Performance scores on the MultiPragEval test suite across four maxims
with overall averages for Korean. The maxim of quality typically achieves the
highest rankings, while the maxim of manner consistently records the lowest
scores, reflecting a similar pattern observed in English.

|  | Chinese
---|---|---
|  | Quan. | Qual. | Rel. | Man. | Avg.
| GPT-4 | 59.44 | 85.00 | 72.78 | 57.78 | 68.75
| GPT-3.5 | 47.22 | 42.22 | 43.89 | 41.11 | 43.61
| Claude-Opus | 80.56 | 82.22 | 80.56 | 63.33 | 76.67
| Claude-Sonnet | 46.11 | 63.89 | 48.33 | 36.11 | 48.61
| Claude-Haiku | 40.00 | 52.78 | 40.56 | 30.00 | 40.83
| Mistral-Large | 47.22 | 60.56 | 66.67 | 44.44 | 54.72
| Mistral-Medium | 43.89 | 46.67 | 36.67 | 28.33 | 38.89
Proprietary | Mistral-Small | 35.56 | 41.11 | 39.44 | 18.33 | 33.61
| Llama3-8B | 45.56 | 49.44 | 53.33 | 38.89 | 46.81
| Llama2-13B | 6.67 | 12.78 | 3.33 | 12.22 | 8.75
| Llama2-7B | 7.78 | 3.33 | 0.56 | 5.00 | 4.17
| Gemma-7B | 29.44 | 26.67 | 18.89 | 25.28 | 25.28
| Solar-10B | 49.44 | 57.78 | 46.67 | 31.67 | 46.39
| Qwen-14B | 51.67 | 47.22 | 58.89 | 42.22 | 50.00
Open-Source | Qwen-7B | 45.00 | 46.11 | 35.56 | 37.78 | 41.11
Table 9: Performance scores on the MultiPragEval test suite across four maxims
with overall averages for Chinese. The pattern of rankings mirrors those
observed in English and Korean.