# CeCNN: Copula-enhanced convolutional neural networks in joint prediction of
refraction error and axial length based on ultra-widefield fundus images

Chong Zhong† Yang Li† Danjuan Yang  Meiyan Li  Xingtao Zhou  Bo
<EMAIL_ADDRESS>[ Catherine C<EMAIL_ADDRESS>[
A.H. Welsh  School of Data Science, Fudan University presep=, ]e1  Department
of Applied Mathematics, The Hong Kong Polytechnic University presep=, ]e2
College of Business and Economics, Australian National University  Eye
Institute and Department of Ophthalmology, Eye & ENT Hospital, Fudan
University

###### Abstract

Ultra-widefield (UWF) fundus images are replacing traditional fundus images in
screening, detection, prediction, and treatment of complications related to
myopia because their much broader visual range is advantageous for highly
myopic eyes. Spherical equivalent (SE) is extensively used as the main myopia
outcome measure, and axial length (AL) has drawn increasing interest as an
important ocular component for assessing myopia. Cutting-edge studies show
that SE and AL are strongly correlated. Using the joint information from SE
and AL is potentially better than using either separately. In the deep
learning community, though there is research on multiple-response tasks with a
3D image biomarker, dependence among responses is only sporadically taken into
consideration. Inspired by the spirit that information extracted from the data
by statistical methods can improve the prediction accuracy of deep learning
models, we formulate a class of multivariate response regression models with a
higher-order tensor biomarker, for the bivariate tasks of regression-
classification and regression-regression. Specifically, we propose a copula-
enhanced convolutional neural network (CeCNN) framework that incorporates the
dependence between responses through a Gaussian copula (with parameters
estimated from a warm-up CNN) and uses the induced copula-likelihood loss with
the backbone CNNs. We establish the statistical framework and algorithms for
the aforementioned two bivariate tasks. We show that the CeCNN has better
prediction accuracy after adding the dependency information to the backbone
models. The modeling and the proposed CeCNN algorithm are applicable beyond
the UWF scenario and can be effective with other backbones beyond ResNet and
LeNet.

Copula,

Convolutional Neural Network,

Multi-task learning,

Myopia,

Ultra-widefield fundus image,

3D medical image object,

###### keywords:

, , , , , , , and

††${\dagger}$: Co-first authors. ††${\ddagger}$: Co-corresponding authors.

##  1 Introduction

Ocular pathologies are highly prevalent globally, often resulting in visual
impairment. The most recent epidemiological data indicates that at least 2.2
billion individuals worldwide present with impairments in visual acuity
(Organization et al.,, 2019). The most prevalent form of visual impairment
worldwide is myopia, estimated to affect approximately 130 million individuals
globally (Holden et al.,, 2016). According to the World Health Organization,
myopia exhibits high prevalence in high-income nations within the Asia-Pacific
and East Asian regions, with estimated rates of 53.4% and 51.6% respectively
(Organization et al.,, 2019). Particularly, in China, myopia is reported to
affect around 600 million individuals, including more than 100 million primary
and secondary school students (Wang et al.,, 2022). High myopia is often
associated with axial length elongation and numerous complications including
cataract genesis, retinal detachment, macular pathologies, and optic
neuropathy (Ikuno,, 2017). Given the widespread prevalence of myopia and its
significant impact on health, there is an evident need for increased myopia
screening.

In practice, the shortage of trained human resources and sustained financial
resources pose great challenges to implementing myopia screening (Organization
et al.,, 2019). It has been estimated that the ophthalmologist-to-patient
ratio in China and India is nearly 1:100,000 and most of the ophthalmological
resources are in urban areas, making people living in rural and remote areas
less likely to visit an ophthalmologist (Sharma et al.,, 2012; Zhang et al.,,
2021). The lack of medical resources and the severe shortage of
ophthalmologists make early screening difficult in remote areas. Fortunately,
in recent years, deep learning techniques applied to fundus images have aided
the diagnosis and assessment of several diseases including diabetic
retinopathy (Cen et al.,, 2021; Kim et al.,, 2021) and glaucoma (Kim et al.,,
2020; Li et al.,, 2021). For myopia screening, there have been some related
deep learning studies on predicting the diagnostically important quantities,
the spherical equivalent (SE) and the axial-length (AL) separately (Yang et
al., (2022); Oh et al., (2023)).

However, to the best of our knowledge, there is no work that jointly predicts
the SE and AL, and uses these to classify high myopia status based on fundus
images. Separately predicting SE and AL by multiple branches of convolutional
neural networks (CNNs) may produce unsatisfactory accuracy and hinder its
application in clinical practice. To fill this gap, in this paper, we propose
a copula-enhanced convolutional neural network (CeCNN) framework to assist
myopia screening based on the ultrawide field (UWF) fundus images collected in
The Eye, Ear, Nose and Throat (Eye and ENT) Hospital of Fudan University from
2015 to 2019. The proposed CeCNN framework is flexible and general; it can be
nested within popular backbone CNNs and can effectively improve the accuracy
in the prediction of SE, AL, and the classification of high myopia, compared
with the backbone CNNs. Our study reveals that statistical toolkits help us
understand the information within data objects better and promote feature
extraction, and therefore, improve the performance of machine learning models.

There are three motivations for the current study, the ophthalmological
practice that screens myopia through cutting-edge UWF images, joint prediction
of highly correlated outcomes, and enhancement of deep learning toolkits by
incorporating data information extracted by statistical approaches.

###  1.1 Motivation 1: UWF fundus images

Our research is motivated by the successful use of fundus images in the
clinical practice of ophthalmology. In clinical settings, typical fundus
findings of myopia include a tessellated fundus, lacquer cracks, and focal or
diffuse chorioretinal atrophy (Ohno-Matsui et al.,, 2019). As a result, fundus
imaging is crucial for detecting complications related to myopia and may help
identify factors that predict the progression of myopia (Ang et al.,, 2019).

There are two main types of fundus images: regular fundus images and ultrawide
field (UWF) fundus images (Midena et al.,, 2022). Compared with the UWF fundus
images, regular fundus images have some limitations.

Figure 1: Advantages of UWF imaging in myopia-related pathology

The visual range of the regular fundus image is confined to a mere 30∘–75∘,
offering limited insights into the peripheral retina. Moreover, regular fundus
photography is a labor-intensive process demanding substantial photographic
expertise and patient cooperation (Aggarwal et al.,, 2017). In contrast, the
UWF fundus image offers a broader 200∘ view, making it especially advantageous
for highly myopic eyes given the elevated occurrence of peripheral retinal
lesions, including lattice degeneration and retinal breaks (Ang et al.,,
2019). Figure 1 illustrates some advantages of UWF imaging. In the figure, the
orange dashed circle represents the area covered by regular fundus images. It
is evident that UWF captures a significantly larger area compared to
conventional fundus images. Additionally, the area within the yellow circle
indicates lesions caused by peripheral laser spots, the region within the blue
circle shows extreme peripheral chorioretinal atrophy, and the area within the
green circle contains pigmentary degeneration lesions. These lesions are all
associated with myopia, and UWF can encompass a greater area related to
myopia-associated pathologies. Hence, UWF fundus images provide much richer
information than regular fundus images.

Since 2017, deep learning with UWF fundus image has been applied to diagnosing
ophthalmic diseases such as glaucoma, age-related macular degeneration, and
diabetic retinopathy; see Yang et al., (2021) for review. For myopia, we may
be the first to apply deep learning to UWF fundus images to jointly predict
AL, SE, and/or high myopia status.

###  1.2 Motivation 2: Highly correlated SE and AL

In clinical practice, SE is a key measure of the degree of myopia and
determines high myopia status. Hence, predicting SE from the UWF fundus image
is an indispensable task. In addition to SE, AL may be another highly relevant
measure for myopia. The existing literature has shown that AL is an important
ocular component which combines the information of anterior chamber depth,
lens thickness, and vitreous chamber depth (Meng et al.,, 2011; Tideman et
al.,, 2016). An earlier study showed that AL holds significant value in
predicting the onset of myopia within 2–4 years before onset (Mutti et al.,,
2007). When investigating refractive-error disorders, utilizing AL as a
distinct trait could avoid some confounders (Meng et al.,, 2011). In
ophthalmological practice, the association between SE and AL has been
recognized as AL has been mostly regarded as a predictor for SE in high myopia
screening among children (Zhang et al., (2011); Zadnik et al., (2015)).
Although AL can be measured with high precision, the measurement is costly and
time-consuming. Thus, predicting AL based on the UWF fundus image is
meaningful. Nonetheless, existing deep learning studies with UWF fundus images
either predict the AL only (Oh et al.,, 2023), or ignore the correlation
between AL and SE and predict them separately (Yang et al.,, 2022).

As a matter of fact, there is a strong correlation between SE and AL. As shown
by Figure 2, the scatter plot (left subfigure) reveals linear dependence
between AL and SE, and the elliptical contour plot (right subfigure) indicates
that the two quantities may follow a bivariate Gaussian distribution with a
non-identity correlation matrix. These characteristics motivate us to model
the two responses SE and AL jointly.

Figure 2: Left: scatter plot; right:contour plot; x axis: AL; y axis: SE.

In addition to the two continuous quantities SE and AL, high myopia status is
another essential item to be checked in myopia screening since high myopia may
lead to myopic maculopathy and substantially increase the risk of blindness
(Iwase et al.,, 2006). To this end, in our real-world application, we
categorize our tasks into regression-classification tasks and regression-
regression tasks, for predicting AL (continuous variable) and high myopia
status (binary variable), and predicting SE and AL, respectively. Since high
myopia status is closely related to SE and so is the AL, we are driven to
jointly model the two responses in both tasks.

###  1.3  Motivation 3: Incorporating dependence into deep learning models

In multi-task learning on medical images, to the best of our knowledge,
dependence across the responses is rarely taken into consideration. A possible
reason is that in point estimation of multivariate Gaussian mean, the maximum
likelihood estimator is unrelated to the covariance/dependence structure.
Hence, researchers tend to model the total loss as a weighted sum of task-
specific losses such as when modeling the heteroscedastic uncertainty among
the multiple tasks (Kendall et al.,, 2018) and using Pareto weighted multi-
task learning (Lin et al.,, 2019), and ignore dependence modeling. Another
reason may be that modeling dependence in deep learning models is non-trivial.
For example, the dependence between continuous and discrete responses in
regression and classification tasks may not be expressed through learnable
parameters, and these may be difficult to learn by tuning due to a lack of
guidance and computational infeasibility. Furthermore, incorporating
dependence into a deep learning model may be sensitive to the data size.
Specifically, the UWF image data was captured in The Eye and ENT Hospital of
Fudan University, one of very few hospitals in mainland China with the
expensive UWF imaging devices from Optomap Daytona. The data size is further
limited by the high cost of diagnostic tests, making it easy to encounter the
overfitting issue. Overfitting means that the dependence structure is falsely
described by a huge number of redundant model parameters so the training
procedure cannot incorporate the dependence information.

###  1.4 Multi-response learning tasks with 3D image biomarkers

In this paper, we consider a multi-response deep learning model with 3D image
biomarkers to address our problem. Let
$\bm{Y}_{i}=(y_{i,1},\ldots,y_{i,p_{1}},y_{i,p_{1}+1},\ldots,y_{i,p_{1}+p_{2}})$
be a $p$-dimensional response vector, where $y_{ij}\in\mathcal{R}$ for $j\leq
p_{1}$ and $y_{ij}\in\\{0,1\\}$ for $p_{1}<j\leq p_{1}+p_{2}$. That is, there
are $p_{1}$ regression tasks and $p_{2}$ binary classification tasks. Let
$\mathcal{X}_{i}\in\mathbb{R}^{k_{1}\times\ldots\times k_{J}}$ be a $J$-order
tensor, and $\mathcal{G}:\mathbb{R}^{k_{1}\times\ldots\times
k_{J}}\to\mathbb{R}^{p}$ be an unknown non-linear regression function, for
$i=1,\ldots,N$. We formulate the multiple-task supervised learning model for
multiple dependent responses and a tensor object explanatory variable as

$\displaystyle
E(\bm{Y}_{i}|\mathcal{X}_{i})=\mathcal{G}(\mathcal{X}_{i}):=(g_{1}(\mathcal{X}_{i}),\ldots,g_{p}(\mathcal{X}_{i})).$
(1)

Here $g_{j}$ are unknown single output functions.

We model the unknown regression functions $(g_{1},\ldots,g_{p})$ as $p$
branches of CNNs, which are commonly used in deep learning to address
supervised learning tasks with tensor-variate image data objects. Although
there are various types of CNNs (Simonyan and Zisserman,, 2014; He et al.,,
2016; Huang et al.,, 2017, among others), they share a similar architecture.
In general, the architecture of a CNN consists of convolution layers, pooling
layers, and fully connected layers (LeCun et al.,, 2015). The raw images are
first fed into the convolution layer to extract the local features of the
images. The local features are expressed in many feature maps by a number of
filters (convolution kernels) using a convolution operator. The features
extracted by a convolution layer are then aggregated in a pooling layer to
merge similar features. After the above two procedures, the induced feature
maps are imported to another convolution layer followed by a pooling layer
through some non-linear activation function, say, the ReLU activation function
(Fukushima,, 1975). After a series of convolution/pooling layers, the induced
features are redirected to a series of fully connected layers as ordinary deep
neural networks (DNNs) to obtain the final outputs.

In general, a CNN has a huge number of parameters contained in the numerous
hidden layers. Fitting a CNN is equivalent to optimizing a specific loss over
these parameters. In the existing literature for multi-branch CNNs, the
overall loss is usually defined as the sum of the empirical losses on all
branches: the mean square error (MSE) loss of the CNNs for regression tasks
and the cross entropy loss of the CNNs for classification tasks (e.g. Liu et
al.,, 2019; Li et al.,, 2023, among others). However, from a statistical
viewpoint, optimizing such losses ignores the inherent dependence among the
responses $(y_{i1},\ldots,y_{ip})$, incurring model misspecification that
results in suboptimal prediction performance. This motivates us to consider a
modified loss that captures the dependence structure among the multiple
responses. As far as we know, there is no deep learning literature that can
capture an unknown but general dependence structure in different tasks, though
there is some work that considers temporal dependence across responses in
facial expression analysis (Walecki et al.,, 2017). Particularly, if there are
both classification and regression tasks, the dependence between binary and
continuous responses cannot be expressed through a simple joint density
function, posing a challenge to modeling the dependence structure.

In this paper, the proposed CeCNN framework sharply enhances the prediction
capability of traditional CNNs in multi-response learning. The essence of the
CeCNN is a so-called copula-likelihood loss that efficiently captures the
dependence structure across multiple responses through a Gaussian copula
model. The CeCNN framework comprises three modules, a warm-up module that
trains the backbone CNNs under empirical losses, a copula estimation module
that estimates the parameters in the Gaussian copula based on the results of
the warm-up module, and finally, a CeCNN module that trains the backbone CNNs
under the derived copula-likelihood loss. The CeCNN framework enjoys great
generalizability in the sense that it is flexible to various backbone CNNs. In
the application to the UWF fundus image data, CeCNN dramatically improves the
prediction capability of several backbone CNNs in both regression and
classification tasks, demonstrating the value of our method.

The rest of the paper is organized as follows. Sections 2 and 3 introduce how
CeCNN works in regression-classification and regression-regression tasks,
respectively. Section 4 presents the performance of CeCNN in prediction and
classification on the UWF fundus image dataset. Section 5 carries out
simulations on synthetic datasets to demonstrate CeCNN. Section 6 concludes
the paper with a brief discussion about future work.

##  2 Regression-classification task

Due to the substantial severity of high myopia in practical scenarios and the
significant role of AL in myopia prediction, this section treats the specific
regression-classification task with two responses, AL and high myopia status.
In medical practice, a patient is usually diagnosed with high myopia if the SE
exceeds -8.0 diopters (D) (Kobayashi et al.,, 2005). That is, under model 1,
for the $i$th subject, the response vector is $\bm{Y}_{i}=(y_{i1},y_{i2})$,
where $y_{i1}\in R_{+}$ and $y_{i2}\in\\{0,1\\}$ denote the AL and the status
of high myopia (1: high myopia; 0, otherwise) of the $i$th individual,
respectively. The explanatory variable is
$\mathcal{X}_{i}\in\mathbb{R}^{224\times 224\times 2}$, a UWF fundus image
stored in red and green with $224\times 224$ channel-wised pixels.

In the deep learning literature, the selected loss is usually the mean square
error (MSE) for continuous responses in regression tasks, and the cross
entropy for binary responses in binary classification tasks. Simply optimizing
the empirical losses for the CNNs for the two responses separately treats
these responses as independent and thus can produce suboptimal prediction
results when dependence is present. Hence, we replace the usual losses with a
single copula-likelihood loss so that the fitted CNNs can incorporate the
dependence among responses. We construct the copula-likelihood loss for the
regression-classification task in subsection 2.1 and summarize the CeCNN
procedure in subsection 2.2.

The copula model (Sklar, (1959); Murray et al., (2013)) is an effective model
for the dependence between variables. A $p$-dimensional copula $C$ is a
distribution function on $[0,1]^{p}$, where each univariate marginal
distribution is uniform on $[0,1]$. Any joint distribution $F$ can be
completely specified by its marginal distributions and a copula, that is,
there exists a copula $C$ such that

$F(y_{1},\ldots,y_{p})=C\\{F_{1}(y_{1}),\ldots,F_{p}(y_{p})\\},$

where $F_{j}$ denotes the $j$th marginal CDF of $y_{j}$ for $j=1,\ldots,p$.
Let $\Phi$ be the CDF of the standard normal distribution and
$\Gamma\equiv(\gamma_{tj})_{p\times p}$ be a correlation matrix, where the
elements $\gamma_{tj}$ are defined as

$\displaystyle\gamma_{tj}=corr(\Phi^{-1}\\{F_{t}(y_{t})\\},\Phi^{-1}\\{F_{j}(y_{j})\\}),$
(2)

representing the linear correlation between two Gaussian scores
$(\Phi^{-1}\\{F_{t}(y_{t})\\},\Phi^{-1}\\{F_{j}(y_{j})\\})$. Equipped with the
correlation matrix $\Gamma$, a Gaussian copula is given as

$\displaystyle
C(\bm{y}|\Gamma)=\Phi_{p}(\Phi^{-1}\\{F_{1}(y_{1})\\},\ldots,\Phi^{-1}\\{F_{p}(y_{p})\\}|\Gamma),$

where $\Phi_{p}(\cdot|\Gamma)$ denotes the $p$-dimensional standard Gaussian
CDF with correlation matrix $\Gamma$. Under the Gaussian copula, the
coefficient $\gamma_{tj}$ has a clear interpretation: when the marginal
distributions $F_{t}$ and $F_{j}$ are Gaussian, $\gamma_{tj}$ is exactly the
Pearson correlation between $y_{t}$ and $y_{j}$; when $F_{t}$ and $F_{j}$ are
both continuous but non-Gaussian, $\gamma_{tj}$ is called the van der Waerden
coefficient; when $F_{t}$ and $F_{j}$ are both discrete, the interpretation of
$\gamma_{tj}$ depends on the data types; see Song, (2007, Section 6.3) for
comprehensive descriptions.

###  2.1 Copula-likelihood loss

We begin by modeling the marginal distribution of $y_{i1}|\mathcal{X}_{i}$.
Let $f_{1}(\cdot;g_{1}(\mathcal{X}_{i}))$ be the conditional density of
$y_{i1}|\mathcal{X}_{i}$ so that
$E(y_{i1}|\mathcal{X}_{i})=g_{1}(\mathcal{X}_{i})$. In this subsection, we
assume $f_{1}$ is a Gaussian density to ease notation. That is,
$y_{i1}|\mathcal{X}_{i}\sim N(g_{1}(\mathcal{X}_{i}),\sigma^{2})$. Next, let
the marginal distribution of $y_{i2}|\mathcal{X}_{i}$ be

$\displaystyle
y_{i2}|\mathcal{X}_{i}\sim\text{Bernoulli}[\text{Sigmoid}\\{g_{2}(\mathcal{X}_{i})\\}],$

where $\text{Sigmoid}(z)=1/{(1+e^{-z})}$. In statistics, the Sigmoid function
is the inverse of the usual logit link function; in deep learning it
represents a fully connected sigmoid layer applied after learning $g_{2}$ to
transform the outputs to probabilities.

Let $\rho\equiv
corr(\Phi^{-1}\\{F_{1}(y_{i1})\\},\Phi^{-1}\\{F_{2}(y_{i2})\\}|\mathcal{X}_{i})$
be the conditional correlation between the two Gaussian scores
$\Phi^{-1}\\{F_{1}(y_{i1})\\}$ and $\Phi^{-1}\\{F_{2}(y_{i2})\\}$ (the
antidiagonal of $\Gamma$). Let $\mu_{i1}=g_{1}(\mathcal{X}_{i})$ be the
conditional expectation of $y_{i1}$,
$\mu_{i2}=\text{Sigmoid}\\{g_{2}(\mathcal{X}_{i})\\}$ be the conditional
probability that $y_{i2}=1$, and $z_{i1}=(y_{i1}-\mu_{i1})/\sigma$ be the
standardized residual of $y_{i1}$. The partial derivative of the Gaussian
copula with respect to the continuous coordinate $y_{i1}$ is

$\displaystyle\begin{aligned} \frac{\partial}{\partial
y_{i1}}C(y_{i1},y_{i2}|\Gamma)&\propto\frac{1}{\sqrt{1-\rho^{2}}}\int_{-\infty}^{\Phi^{-1}(\mu_{i2})}\exp\left\\{-\frac{1}{2}(z_{i1},s)\Gamma^{-1}(z_{i1},s)^{T}+\frac{1}{2}z_{i1}^{2}\right\\}ds\\\
&=\frac{1}{\sqrt{1-\rho^{2}}}\int_{-\infty}^{\Phi^{-1}(\mu_{i2})}\exp\left\\{-\frac{z_{i1}^{2}-2\rho
z_{i1}s+s^{2}}{2(1-\rho^{2})}+\frac{z_{i1}^{2}}{2}\right\\}ds\\\
&=\frac{1}{\sqrt{1-\rho^{2}}}\int_{-\infty}^{\Phi^{-1}(\mu_{i2})}\exp\left\\{-\frac{(\rho^{2}z_{i1}^{2}-2\rho
z_{i1}s+s^{2})}{2(1-\rho^{2})}\right\\}ds\\\
&\propto\Phi\left(\frac{\Phi^{-1}(\mu_{i2})-\rho
z_{i1}}{\sqrt{1-\rho^{2}}}\right)\equiv
C^{*}(\mu_{i2},z_{i1}|\rho).\end{aligned}$ (3)

A more general form of (3) for an arbitrary number of regression and
classification tasks can be found in Song et al., (2009). By taking the value
of $y_{i2}$ to be either $0$ or $1$, the logarithm of the joint density of
$(y_{i1},y_{i2})$ under the Gaussian copula is

$\displaystyle l(y_{i1},y_{i2}|\mathcal{X}_{i})=\log
N(y_{i1};\mu_{i1},\sigma^{2})+y_{i2}\log
C^{*}(\mu_{i2},z_{i1}|\rho)+(1-y_{i2})\log\\{1-C^{*}(\mu_{i2},z_{i1}|\rho)\\}.$

Then, the copula-likelihood loss is minus the log-likelihood which is

$\displaystyle\begin{aligned}
\mathcal{L}_{1}&(g_{1},p_{2}|\\{\bm{Y}_{i}\\}_{i=1}^{n},\\{\mathcal{X}_{i}\\}_{i=1}^{n},\rho,\sigma)=\frac{1}{2\sigma^{2}}\sum_{i=1}^{n}(y_{i1}-\mu_{i1})^{2}\\\
&-\left\\{\sum_{i=1}^{n}[y_{i2}\log
C^{*}(\mu_{i2},z_{i1}|\rho)+(1-y_{i2})\log\\{1-C^{*}(\mu_{i2},z_{i1}|\rho)\\}]\right\\}.\end{aligned}$
(4)

Note that on the right hand side (RHS) of (4), the first summand is related to
$g_{1}$ only, while the second summand associates $\mu_{i1}$ and $\mu_{i2}$,
or equivalently, $g_{1}$ and $g_{2}$ through a parameter $\rho$.
Mathematically, when $\rho=0$, $C^{*}(\mu_{i2},z_{i1}|\rho)$ becomes
$\mu_{i2}=\text{Sigmoid}\\{g_{2}(\mathcal{X}_{i})\\}$, implying that the
second summand at the RHS of equation (4) reduces to a pure cross entropy loss
of $g_{2}$ only. Hence, the two nonparametric functions $g_{1}$ and $g_{2}$
(expressed in CNNs) can be learned separately. That is, by fixing $\rho=0$,
loss (4) simplifies to the sum of two empirical losses, MSE and cross entropy,
as in common deep learning practice. However, when $\rho\not=0$, $g_{1}$ and
$g_{2}$ cannot be separated but have to be optimized jointly. Therefore, both
$\sigma$ and $\rho$ have a substantial impact on the optimization of (4), and
their effects cannot be ignored. From a statistical perspective, fixing
$\rho=0$ corresponds to treating $y_{i1}$ and $y_{i2}$ as conditionally
independent, a strong assumption violating the data ground truth. In contrast,
through the Gaussian copula, we can characterize the dependence of $y_{i1}$
and $y_{i2}$ by the two parameters $\rho$ and $\sigma$, each of which has a
clear interpretation.

Marginally, $\rho$ measures the correlation between the $z_{i1}$, the
standardized residuals of $g_{1}(\mathcal{X}_{i})$, and the Gaussian scores
$\Phi^{-1}[\text{Sigmoid}\\{g_{2}(\mathcal{X}_{i})\\}]$. In this sense,
specifying $\rho$ is essential to fitting the two CNNs $g_{1}$ and $g_{2}$.
Meanwhile, note that in (4), $\sigma^{-1}$ acts as the weight balancing the
MSE loss and the cross-entropy-like loss. A smaller $\sigma$ puts a higher
weight on the MSE loss and hence, optimizing (4) may reduce the MSE prediction
error if the variance of the continuous responses $y_{i1}$ is small. This
explains why in our application, in the regression-classification task of
predicting AL and the status of high myopia, the CeCNN enjoys a much smaller
MSE in the prediction of AL than the backbone CNNs.

###  2.2 End-to-end CeCNN

We provide the whole procedure for learning the end-to-end CeCNN in this
subsection. The overall CeCNN architecture has three modules, the warm-up CNN,
estimation of the copula parameters, and the CeCNN. The three modules are
summarized in Algorithm 1. The plot of the CeCNN architecture can also be
viewed in Fig. 5 in subsection 4.2.

Algorithm 1 End-to-end CeCNN (regression-classification task)

1:Training data $\\{\mathcal{X}_{i},\bm{Y}_{i}=(y_{i1},y_{i2})\\}_{i=1}^{n}$.

2:CeCNN $(\hat{g}_{1},\hat{p}_{2})$.

3:

4:Module 1: Warm-up CNN

5:

6:$g_{j}=\text{Conv}_{j}(1,\ldots,k_{1})\circ\text{Pool}_{j}(1,\ldots,k_{2})\circ\text{F-C}_{j}(1,\ldots,k_{3})$,
for $j=1,2$.

7:$\text{F-C}_{1}(k_{3})=\bm{w}_{k_{3}}^{T}\bm{z}_{k_{3}-1}+b_{k_{3}}=g_{1}(\mathcal{X}_{i})$.

8:$\hat{g}_{1}^{0}=\arg\min
n^{-1}\sum_{i=1}^{n}(y_{ij}-g_{j}(\mathcal{X}_{i}))^{2}$.

9:$\text{F-C}_{2}(k_{3}-1)=\bm{w}_{k_{3}-2}^{T}\bm{z}_{k_{3}-2}+b_{k_{3}-1}\equiv
g_{2}(\mathcal{X}_{i})$,
$\text{F-C}_{j}(k_{3})=\text{Sigmoid}(g_{2}(\mathcal{X}_{i}))\equiv
p_{2}(\mathcal{X}_{i})$.

10:$\hat{p}_{2}^{0}=\arg\min-\sum_{i=1}^{n}[y_{ij}\log(p_{2}(\mathcal{X}_{i}))+(1-y_{i})\log(1-p_{2}(\mathcal{X}_{i}))]$.

11:

12:Module 2: Estimation of copula parameters

13:

14:$e_{i}=y_{i1}-\hat{g}_{1}^{0}(\mathcal{X}_{i})$,
$\hat{\sigma}=\text{sd}(\bm{e})$,
$\hat{\rho}=corr(\bm{y}_{1},\Phi^{-1}(\hat{p}_{2}^{0}(\mathcal{X})))$.

15:

16:Module 3: CeCNN

17:

18:Determine the copula-likelihood
$\mathcal{L}_{1}(g_{1},p_{2}|\mathcal{X},\bm{Y},\hat{\rho},\hat{\sigma})$ in
the form of (4).

19:Obtain $(\hat{g}_{1},\hat{p}_{2})=\arg\min\mathcal{L}_{1}$.

Module 1 includes the basic construction of multi-branch CNNs under two
empirical losses. Without loss of generality, we assume that all branches of
the CNN share the same number of convolution (Conv) layers $k_{1}$, pooling
(Pool) layers $k_{2}$, and fully connected (F-C) layers $k_{3}$. Whether they
share the same convolution or pooling layers depends on the architecture of
the backbone CNN employed by the user. The last F-C layer distinguishes the
tasks of regression or classification by using linear functions and the
sigmoid transformation, respectively (in lines 2 and 4, respectively). Then
all the numerous parameters included in the very deep layers are updated by
the Adam algorithm (Kingma and Ba,, 2014) to optimize the losses presented in
lines 3 and 5, respectively.

In Module 2, we estimate the two copula parameters $\sigma$ and $\rho$ based
on the warm-up CNN fitted in Module 1. To estimate $\sigma$, we compute the
residuals from the warm-up CNN $e_{i}=y_{i1}-\hat{g}_{1}^{0}(\mathcal{X}_{i})$
and then the sample standard deviation of the residuals. Since $y_{i1}$ is
Gaussian, the warm-up CNN $\hat{g}_{1}^{0}$ under the MSE loss is, marginally,
the maximum likelihood estimator of $g_{1}$. Hence, the sample standard error
of the residuals is exactly the maximum likelihood estimator of $\sigma$. For
estimation of the correlation $\rho$, we use the Pearson correlation between
the Gaussian scores $y_{i1}$ and
$\Phi^{-1}(\text{Sigmoid}(g_{2}(\mathcal{X}_{i})))$ (note that the responses
$y_{i1}$ are Gaussian). In summary, the copula parameters $\sigma$ and
$\Gamma$ are estimated as

$\hat{\sigma}=\text{sd}(\bm{e}),~{}\hat{\Gamma}\equiv(\hat{\rho}^{|t-j|})_{2\times
2},~{}\hat{\rho}=corr(\bm{y}_{1},\Phi^{-1}(\text{Sigmoid}(g_{2}(\mathcal{X})))).$

Finally, in Module 3, we determine the copula-likelihood loss based on the
estimated copula parameters and use the backbone CNNs from Module 1 to train
the CeCNN under the copula-likelihood loss. Note that in this Module, CeCNN
does not rely on any specific architecture in the backbone CNN, and thus,
there is great flexibility in the choice of different CNNs.

##  3 Regression-regression task

This section treats the specific regression-regression task of predicting the
clinically important, highly correlated responses SE and AL (see Figure 1),
using the proposed CeCNN.

Let $\bm{Y}_{i}\in\mathbb{R}^{p}$. We rewrite model (1) as the following
regression model

$\displaystyle\bm{Y}_{i}=\mathcal{G}(\mathcal{X}_{i})+\bm{\epsilon}_{i},$ (5)

where $\bm{\epsilon}_{i}=(\epsilon_{i1},\ldots,\epsilon_{ip})$ is a
$p$-dimensional noise vector, $\bm{\epsilon}_{i}\perp\mathcal{X}_{i}$,
$E(\epsilon_{ij})=0$ for all $1\leq j\leq p$ and
$\bm{\epsilon}_{i}\perp\bm{\epsilon}_{i^{\prime}}$. Under model (5), given
$\mathcal{X}_{i}$, the dependence among $(Y_{i1},\ldots,Y_{ip})$ is equivalent
to the dependence among the noise $(\epsilon_{i1},\ldots,\epsilon_{ip})$.
Therefore, the joint distribution of $\bm{Y}_{i}|\mathcal{X}_{i}$ can be
obtained from the joint distribution of $\bm{\epsilon}_{i}$. Using a Gaussian
copula, we obtain

$\displaystyle
F(y_{i1},\ldots,y_{ip}|\mathcal{X}_{i},\Gamma)=C_{p}(F_{\epsilon_{1}}(y_{i1}-g_{1}[(\mathcal{X}_{i})]\\},\ldots,F_{\epsilon_{p}}(y_{ip}-g_{p}[(\mathcal{X}_{i})]\\}|\Gamma),$
(6)

where $F_{\epsilon_{j}}$ denotes the marginal CDF of $\epsilon_{j}$ and
$\Gamma$ is the correlation matrix of $(\epsilon_{1},\ldots,\epsilon_{p})$.

###  3.1 Two types of model error

For the regression-regression task, the key is to specify the marginal CDFs
$F_{\epsilon_{j}}$ of the model error. In this subsection, we study two types
of model error.

Gaussian error We first consider the Gaussian case where marginally, all model
errors $\epsilon_{j}\sim N(0,\sigma_{j}^{2})$, where $\sigma_{j}$ is an
unknown scale parameter. Then the likelihood contribution of
$\bm{Y}_{i}|\mathcal{X}_{i}$ reduces to the multivariate Gaussian density.
Hence, for some training data
$(\\{\bm{Y}_{i}\\}_{i=1}^{n},\\{\mathcal{X}_{i}\\}_{i=1}^{n})$, the copula-
likelihood loss may be defined as

$\displaystyle\mathcal{L}_{2}(\\{g_{j}\\}_{j=1}^{p}|\\{\bm{Y}_{i}\\}_{i=1}^{n};\\{\mathcal{X}_{i}\\}_{i=1}^{n})=-\sum_{i=1}^{n}\log
N_{p}\\{(y_{i1}-g_{1}[(\mathcal{X}_{i})],\ldots,y_{ip}-g_{p}[(\mathcal{X}_{i})]);\bm{0}_{p},\Sigma\\},$
(7)

where $N_{p}(\cdot;\bm{0}_{p},\Sigma)$ denotes the density of the p-variate
Gaussian distribution with mean $\bm{0}_{p}$ and covariance matrix $\Sigma$.
Mathematically, $\Sigma=diag(\sigma_{1},\ldots,\sigma_{p})\Gamma
diag(\sigma_{1},\ldots,\sigma_{p})\equiv(\sigma_{tj})_{p\times
p}=\gamma_{tj}\sigma_{t}\sigma_{j}$. If the correlation matrix $\Gamma$ is an
identity matrix, the copula-likelihood loss (7) simply reduces to the sum of
the empirical MSE loss and some constant. Therefore, the copula-likelihood
loss (7) is a generalization of the empirical MSE loss.

Before optimizing the loss (7) with respect to each CNN $g_{i}$, one has to
estimate the covariance matrix $\Sigma$, or equivalently, the correlation
$\Gamma$ and the marginal standard deviations
$(\sigma_{1},\ldots,\sigma_{p})$. Intuitively, their estimates can be obtained
from the empirical correlation matrix and marginal standard deviations of the
residuals from warm-up CNNs. Let $(\hat{g}_{01},\ldots,\hat{g}_{0p})$ be $p$
branches of CNNs trained with the empirical MSE loss for
$(g_{1},\ldots,g_{p})$ on the training dataset
$(\\{\bm{Y}_{i}\\}_{i=1}^{n},\\{\mathcal{X}_{i}\\}_{i=1}^{n})$. Using the
residuals $(e_{i1},\ldots,e_{ip})$ for each observation, where
$e_{ij}=y_{ij}-\hat{g}_{0j}(\bm{X}_{i})$, we obtain estimates of the copula
parameters $\Gamma$ and $\sigma_{j}$ as

$\hat{\Gamma}\equiv(\hat{\gamma}_{tj})_{p\times
p}=corr(\\{e_{it}\\}_{i=1}^{n},\\{e_{ij}\\}_{i=1}^{n}),~{}\hat{\sigma}_{j}=\text{sd}(\\{e_{ij}\\}_{i=1}^{n}).$

Here $corr$ represents the Pearson correlation. With the above estimates, we
can train new CNNs by optimizing the copula-likelihood loss through commonly
used stochastic optimizers such as the stochastic gradient descent and the
Adam algorithm (Kingma and Ba,, 2014).

Nonparametric error We consider next a more general case where the marginal
distribution of error $\epsilon_{j}$ is unknown. In this case, we estimate the
marginal CDF $F_{\epsilon_{j}}$ for $j=1,\ldots,p$, then estimate $\Gamma$,
and finally we combine them to derive the copula-likelihood loss.

Again, we begin with the warm-up CNNs with MSE loss and obtain the residuals
$(e_{i1},\ldots,e_{ip})$ on the training dataset. The estimator of the
marginal CDF $F_{\epsilon_{j}}$ is given by the empirical CDF of the
residuals. To make sure $E(\epsilon_{j})=0$, we use the mean-centered
residuals $\tilde{e}_{i1},\ldots,\tilde{e}_{ip}$ in the empirical CDF

$\hat{F}_{\epsilon_{j}}(t)=\frac{1}{n}\sum_{i=1}^{n}I(\tilde{e}_{ij}\leq
t),~{}j=1,\ldots,p.$

To facilitate using a stochastic optimizer, we smooth the empirical CDF with a
Gaussian kernel so that

$\displaystyle\tilde{F}_{\epsilon_{j}}(t)=\frac{1}{n}\sum_{i=1}^{n}\Phi\\{(t-\tilde{e}_{ij})/\psi_{0}\\},~{}\tilde{f}_{\epsilon_{j}}(t)=\frac{1}{n\psi_{0}}\sum_{i=1}^{n}\phi\\{(t-\tilde{e}_{ij})/\psi_{0}\\},$
(8)

where $\psi_{0}>0$ is a tuning parameter.

Once we obtain the empirical estimates $\tilde{F}_{\epsilon_{j}}$, the
correlation matrix $\Gamma$ is directly given by (2) so that

$\hat{\gamma}_{sj}=corr(\\{\Phi^{-1}(\tilde{F}_{\epsilon_{s}}(\tilde{e}_{is}))\\}_{i=1}^{n},\\{\Phi^{-1}(\tilde{F}_{\epsilon_{j}}(\tilde{e}_{ij}))\\}_{i=1}^{n}).$

That is, the Pearson correlation between the two Gaussian scores of the
smoothed empirical CDFs of the centered residuals.

With the above estimates, according to Song et al., (2009, eq. (7)), the
copula-likelihood loss is given (up to a constant) as

$\displaystyle\mathcal{L}_{3}(\\{g_{j}\\}_{j=1}^{p}|\\{\bm{Y}_{i}\\}_{i=1}^{n};\\{\mathcal{X}_{i}\\}_{i=1}^{n})=-\sum_{i=1}^{n}\frac{1}{2}\bm{q}_{i}^{T}(\bm{I}_{p}-\Gamma^{-1})\bm{q}_{i}-\sum_{i=1}^{n}\sum_{j=1}^{p}\log\\{\tilde{f}_{\epsilon_{j}}(y_{ij}-g_{j}(\mathcal{X}_{i}))\\},$
(9)

where
$\bm{q}_{i}=(\Phi^{-1}\\{\tilde{F}_{\epsilon_{1}}[y_{i1}-g_{1}(\mathcal{X}_{i})]\\},\ldots,\Phi^{-1}\\{\tilde{F}_{\epsilon_{p}}[y_{ip}-g_{p}(\mathcal{X}_{i})]\\})^{T}$.
Obviously, if $\Gamma=\bm{I}_{p}$ (no correlation between the responses), loss
(9) reduces to the sum of minus the log densities $\tilde{f}_{\epsilon_{j}}$.

###  3.2 End-to-end CeCNN

Finally, we summarize the CeCNN for the regression-regression task in
Algorithm 2. In this task, the CeCNN again has the three-module structure with
different copula parameters. For Gaussian error, the copula parameters are the
marginal SDs $\sigma_{j}$ and the Pearson correlation $\gamma_{tj}$; for
nonparametric error, the copula parameters contain an infinite dimensional
parameter $f_{\epsilon_{j}}$ and the transformed correlation $\gamma_{tj}$. An
overview of the CeCNN architecture is shown in Fig. 5 in subsection 4.2.

Algorithm 2 End-to-end CeCNN (regression-regression task)

1:Training data
$\\{\mathcal{X}_{i},\bm{Y}_{i}=(y_{i1},\ldots,y_{ip1})\\}_{i=1}^{n}$.

2:Trained CeCNN $(\hat{g}_{1},\ldots,\hat{g}_{p})$.

3:

4:Module 1: Warm-up CNN

5:

6:for $j=1,\ldots,p$ do

7:
$g_{j}=\text{Conv}_{j}(1,\ldots,k_{1})\circ\text{Pool}_{j}(1,\ldots,k_{2})\circ\text{F-C}_{j}(1,\ldots,k_{3})$.

8:
$\text{F-C}_{j}(k_{3})=\bm{w}_{k_{3}}^{T}\bm{z}_{k_{3}-1}+b_{k_{3}}=g_{j}(\mathcal{X}_{i})$;

9: $\hat{g}_{j}^{0}=\arg\min
n^{-1}\sum_{i=1}^{n}(y_{ij}-g_{j}(\mathcal{X}_{i}))^{2}$.

10:end for

11:

12:Module 2: Copula parameters estimation

13:

14:if Gaussian then

15: Obtain residuals $e_{ij}=y_{ij}-\hat{g}_{j}^{0}(\mathcal{X}_{i})$.

16: $\hat{\sigma}_{j}=\text{sd}(\bm{e}_{ij})$,
$\hat{\gamma}_{tj}=corr(\bm{e}_{it},\bm{e}_{ij})$.

17:else if Non-Gaussian then

18: $\tilde{e}_{ij}=e_{ij}-\bar{e}_{j}$,
$\tilde{F}_{\epsilon_{j}}(t)=\frac{1}{n}\sum_{i=1}^{n}\Phi_{\psi_{0}}(t-\tilde{e}_{ij}),~{}\tilde{f}_{\epsilon_{j}}(t)=\frac{1}{n}\sum_{i=1}^{n}\phi_{\psi_{0}}(t-\tilde{e}_{ij})$

19:
$\hat{\gamma}_{tj}=corr(\Phi^{-1}(\tilde{F}_{\epsilon_{t}}(\tilde{\bm{e}}_{it})),\Phi^{-1}(\tilde{F}_{\epsilon_{j}}(\tilde{\bm{e}}_{ij})))$

20:end if

21:

22:Module 3: CeCNN

23:

24:if Gaussian then

25: Define the loss $\mathcal{L}_{2}$ the same as (7).
$(\hat{g}_{1},\ldots,\hat{g}_{p_{1}})=\arg\min\mathcal{L}_{2}$.

26:else if Non-gaussian then

27: Define the loss $\mathcal{L}_{3}$ the same as (9).
$(\hat{g}_{1},\ldots,\hat{g}_{p_{1}})=\arg\min\mathcal{L}_{3}$.

28:end if

##  4 Application to the UWF fundus image dataset

For our proposed CeCNN model, in the context of the regression-classification
task, we predict both the axial length and whether the patient has high
myopia. Regarding the regression-regression problem, we predict the axial
length and refractive error.

###  4.1 Data Preparation and Descriptive Analysis

The data collection process involved capturing 987 fundus images from the left
eyes of 987 patients using the Optomap Daytona scanning laser ophthalmoscope
(Daytona, Optos, UK). The data collection period extended from November 2015
to January 2019, and was conducted at The Eye and ENT Hospital of Fudan
University. For this study, only patients with myopia in both eyes were
included, and left eyes were chosen to avoid correlation between data from
using both the same patient’s eyes.

All enrolled patients sought refractive surgery treatment and were exclusively
myopia patients. To ensure homogeneity and accuracy, individuals with other
ocular conditions such as cataract, vitreoretinal diseases, or glaucoma, as
well as those with a history of trauma or previous ocular surgery, were
excluded from the dataset. For image selection, we required the fovea to be
positioned at the center of the image and applied the criterion of
gradability. Images were considered gradable if there was no blurring in the
optic disc or foveal area, and if less than 50% of the peripheral retinal area
was obscured by eyelids or eyelashes. The UWF fundus images obtained during
the study were exported in JPEG format and compressed to a resolution of 224 x
224 pixels to facilitate subsequent analysis.

Response variables The response variables included in the models include two
continuous responses: SE and AL, and one binary response indicating whether
the patient has high myopia or not. SE and AL are recorded during the data
collecting process, and high myopia is defined according to SE value. Due to
the commonly observed visual dysfunction in myopic eyes with a refractive
error exceeding -8.0 diopters (D), an eye is considered to have high myopia
when its refractive error (the value of SE) surpasses -8.0 D. This specific
value is used as the defining criterion for high myopia (Kobayashi et al.,,
2005). We will predict both SE and AL values (a pure regression task), and
also predict AL value and myopia status (a regression-classification task).
Table LABEL:table:stats_SE_AL summarizes the statistical information on SE and
AL. It is clear from the table that SE and AL exhibit a strong linear
correlation, suggesting that it is valuable to incorporate their dependence.

Table 1: Statistical information of SE and AL Variables | Range | Mean | Sd | Correlation coefficient
---|---|---|---|---
SE | (4.25, 23) | 11.12 | 3.57 | 0.79
AL | (24.22, 34.23) | 27.84 | 1.69

###  4.2 Structure of CeCNN

In order to demonstrate the adaptability of our model to different deep
learning architectures, we selected LeNet (LeCun et al.,, 1998) and ResNet-18
(He et al.,, 2016) as our backbone models. LeNet represents one of the
simplest, original CNN models, while ResNet is acknowledged as one of the most
effective and prominent CNN models in the field.

Simplified backbone models to overcome overfitting The conventional ResNet18
model contains over ten million parameters. Similarly, when we used the
traditional LeNet structure, the parameter count exceeded five million.
However, our dataset comprises only a few hundred images. This discrepancy
posed a significant challenge in the form of severe overfitting.

To mitigate the risk of overfitting during the training process, we adopted
the common strategy of simplifying the neural network architecture.
Specifically, we removed the last two CNN blocks from ResNet18 and increased
the filter sizes of the convolutional and pooling layers of LeNet. This
substantial reduction in the parameter count effectively prevented
overfitting. After simplification, the parameter count of ResNet18 reduced
from over ten million to around six hundred thousand. Similarly, the parameter
count of the simplified LeNet decreased from over five million to just over
sixty thousand. Fig. 3 and Fig. 4 show the architecture of the simplified
versions of LeNet and ResNet18 respectfully. Fig. 3 was created using (Parmar
et al.,, 2018).

Regarding the regression-regression tasks, we observed that during the
training process, when the backbone model exhibited severe overfitting,
whether it was ResNet or LeNet serving as the backbone model, the CeCNN
framework would fail to extract effective dependence information from the
residuals. This made CeCNN ineffective. For regression-classification tasks,
when LeNet was used as the backbone model, overfitting still made CeCNN
ineffective. However, when ResNet was used as the backbone model, CeCNN
remained effective. As a result, for regression-regression tasks, we used
simplified versions of both LeNet and ResNet, and for regression-
classification tasks, we applied the simplified version of LeNet and the
original, unsimplified version for ResNet.

Figure 3: Architecture of simplified LeNet Figure 4: Architecture of
simplified ResNet18

Architecture of CeCNN The complete architecture for applying the CeCNN model
to the UWF dataset is illustrated in Fig. 5. Initially, we trained the
backbone models and obtained the residuals from their outputs (Module 1).
Then, using both the outputs and residuals, we computed estimates of the
copula parameters and the correlation matrix (Module 2). With the correlation
matrix and residuals, we proceeded to train our CeCNN models with the proposed
copula loss (Module 3). When training our CeCNN model, we used the network
parameters from the previously pre-trained CNN backbone models as the initial
parameters of CeCNN.

Figure 5: Architecture of proposed CeCNN.

###  4.3 Prediction Performance Evaluation

The dataset of 987 fundus images was partitioned into the training data set,
the validation set, and the testing set, with a ratio of 6:2:2. To mitigate
bias in model evaluation and obtain more reliable estimates of the results, 10
rounds of 5-fold cross-validation were employed. For the pure regression task,
we calculated the Root Mean Square Error (RMSE) and the Mean Absolute Error
(MAE), while for the regression-classification task, we computed the RMSE,
Area Under the Curve (AUC) and accuracy. We assessed the predictive
performance of our CeCNN in comparison to our backbone models, aiming to show
that the proposed techniques enhance the performance of the original models.

Results of regression-classification tasks For the regression-classification
tasks, the results in Fig. 6 and Fig. 7 show the enhanced predictive
performance of our CeCNN models for both classification and regression tasks.
When ResNet is used as the backbone model, the accuracy of predicting high
myopia improves by 0.927%, but the AUC decreases by 0.065%. The RMSE for
predicting AL improves by 0.913%. On the other hand, with LeNet as the
backbone model, the accuracy of predicting high myopia decreases by 0.145%.
However, the AUC increases by 1.227%. Simultaneously, the MSE for predicting
AL improves by 1.223%.

Figure 6: Box plots of MSE, Accuracy and AUC in 10 rounds of 5-fold validation
of the regression-classification tasks for LeNet as the backbone model. Figure
7: Box plots of MSE, Accuracy and AUC in 10 rounds of 5-fold validation of the
regression-classification tasks for ResNet as the backbone model.

Results of regression-regression tasks The results for the regression-
regression tasks are shown in Fig. 8 and Fig. 9. Our CeCNN models demonstrate
improved predictive performance for both SE and AL on average. Specifically,
under the Gaussian setting, the RMSE of AL improves by an average of 10.378%
and the RMSE of SE improves by an average of 0.717%, with LeNet as the
backbone. On the other hand, the RMSE of AL improves by an average of 3.683%
and the RMSE of SE improves by an average of 2.232%, with ResNet as the
backbone.

In the semi-parametric scenario, the RMSE of AL improves by an average of
9.905% and the RMSE of SE improves by an average of 0.528%, with LeNet as the
backbone. On the other hand, the RMSE of AL improves by an average of 3.860%
and the RMSE of SE improves by an average of 2.268%, with ResNet as the
backbone.

Figure 8: Box plots of RMSE and MAE in 10 rounds of 5-fold validation of the
regression-regression tasks for LeNet as the backbone model. Figure 9: Box
plots of RMSE and MAE in 10 rounds of 5-fold validation of the regression-
regression tasks for ResNet as the backbone model.

##  5 Analysis on synthetic data

We carried out simulations of synthetic datasets for each task to demonstrate
the superiority of the proposed CeCNN. To match our application scenarios, in
both regression-classification and regression-regression tasks, we consider
correlated bivariate responses and a single image covariate. To simplify the
ground truth model, we compress the image covariate to a grey matrix. We
present the simulation details for the regression-classification and
regression-regression tasks in subsections 5.1 and 5.2, respectively.

###  5.1 Regression-classification task

On this task, the synthetic image covariate is generated as a grey matrix
$X_{i}\in\mathbb{R}^{9\times 9}$ that can be divided into nine independent
blocks of $3\times 3$ square matrices:

$X=\begin{pmatrix}X_{1,1}&X_{1,2}&X_{1,3}\\\ X_{2,1}&X_{2,2}&X_{2,3}\\\
X_{3,1}&X_{3,2}&X_{3,3}\end{pmatrix},~{}X_{t,s}=\begin{pmatrix}X_{t,s}^{(1,1)}&X_{t,s}^{(1,2)}&X_{t,s}^{(1,3)}\\\
X_{t,s}^{(2,1)}&X_{t,s}^{(2,2)}&X_{t,s}^{(2,3)}\\\
X_{t,s}^{(3,1)}&X_{t,s}^{(3,2)}&X_{t,s}^{(3,3)}\end{pmatrix},~{}t,s=1,2,3.$

For $k,l=1,2,3$, we set $X_{3,3}^{(k,l)}\sim N(1,0.5^{2})$ as elements of the
dark block and $X_{t,s}^{(k,l)}\sim N(0,1^{2})$ as elements of bright blocks,
for $(t,s)\not=(3,3)$. Based on this covariate, we generated the synthetic
continuous response $y_{1}\in\mathbb{R}$ and the synthetic binary response
$y_{2}\in\\{0,1\\}$ using model (1).

For each block $X_{t,s}$, we define the block operators
$S_{t,s},S^{*}_{t,s}:\mathbb{R}^{3\times 3}\to\mathbb{R}$ for responses
$y_{1}$ and $y_{2}$, respectively. Then the two responses were generated as

$\displaystyle\begin{aligned} y_{1}\sim N\left\\{\sum_{1\leq t,s\leq
3}S_{t,s}(X_{t,s}),1^{2}\right\\},~{}y_{2}\sim\text{Bern}\left\\{\text{Sigmoid}[\sum_{1\leq
t,s\leq 3}S^{*}_{t,s}(X_{t,s})]\right\\}.\end{aligned}$ (10)

For the block operators $S_{t,s}$, we set

$\displaystyle\begin{aligned} S_{1,1}=\sum_{1\leq l,k\leq
3}\text{tanh}(X_{1,1}^{(k,l)}),~{}S_{2,2}=\sum_{1\leq l,k\leq
3}X_{2,2}^{(k,l)},~{}S_{3,3}=\text{tanh}\left(\sum_{1\leq l,k\leq
3}X_{3,3}^{(k,l)}\right),\end{aligned}$

and $S_{t,s}=0$ for other blocks. Under this setting, the function
$g_{1}=\sum_{1\leq t,2,\leq 3}S_{t,s}$ is a nonlinear function that can be
modeled by a CNN. For the block operators $S^{*}_{t,s}$, we set

$S^{*}_{2,2}=S_{2,2}=\sum_{1\leq l,k\leq 3}X_{2,2}^{(k,l)}$

and $S^{*}_{t,s}=0$ for other blocks so that $E(y_{2})=1/2$. The two responses
are dependent in this case since they share the same block operator on block
$X_{2,2}$

###  5.2 Regression-regression task

For the regression-regression task, we generate the synthetic image covariates
in a similar way to that of the regression-classification task. For this task,
we set five dark blocks $X_{t,s}^{(k,l)}\sim N(1,0.5^{2})$ for $k,l=1,2,3$ and
$(t,s)\in\\{(1,1),(2,2),(3,3),(1,3),(3,1)\\}$. The remaining blocks are set to
be the aforementioned bright blocks $N(0,1^{2})$. The responses are generated
from model (5), where the model error
$\bm{\epsilon}_{i}=(\epsilon_{i1},\epsilon_{i2})\sim MVN(\bm{0}_{2},\Sigma)$
with

$\displaystyle\Sigma=\begin{pmatrix}1&0\\\
0&2\end{pmatrix}\begin{pmatrix}1&0.7\\\
0.7&1\end{pmatrix}\begin{pmatrix}1&0\\\ 0&2\end{pmatrix}.$

This covariance setting yields a strong correlation and two different levels
of marginal variation (variances of 1 and 4), corresponding to the AL and SE,
respectively. We set the nonlinear regression functions $g_{1}$ and $g_{2}$ as
the sum of block operators so that

$\displaystyle g_{1}(X)=\sum_{1\leq t,s\leq
3}S_{t,s}(X_{t,s}),~{}g_{2}(X)=\sum_{1\leq t,s\leq 3}S_{t,s}^{*}(X_{t,s}).$

The block operators $S_{t,s}$ and $S_{t,s}^{*}$ are defined as

$\displaystyle S_{1,1}=\sum_{1\leq l,k\leq
3}\text{tanh}(X_{1,1}^{(k,l)}),~{}S_{2,2}=\sum_{1\leq l,k\leq
3}X_{2,2}^{(k,l)},~{}S_{3,3}=\text{tanh}\left(\sum_{1\leq l,k\leq
3}X_{3,3}^{(k,l)}\right),$ $\displaystyle S_{1,3}^{*}=\sum_{1\leq l,k\leq
3}\text{tanh}(X_{1,1}^{(k,l)}),~{}S_{2,2}^{*}=\sum_{1\leq l,k\leq
3}X_{2,2}^{(k,l)},~{}S_{3,1}^{*}=\text{tanh}\left(\sum_{1\leq l,k\leq
3}X_{3,3}^{(k,l)}\right),$

and $S_{t,s}=S_{t,s}^{*}$ for the other blocks. Under this setting, marginally
we have $E(y_{1}|X)=E(y_{2}|X)$ and the two regression functions $g_{1}$ and
$g_{2}$ overlap.

###  5.3 Backbone model

A similar framework of CeCNN to that in Fig. 5 was used except that a much
simpler CNN model was chosen as the backbone model for simulation. The
backbone model consisted of two basic convolutional layers and two fully
connected layers. The detailed architecture of the backbone model is shown in
Fig. 10. For regression-classification tasks, we used MSE, AUC, and accuracy
as the metrics to evaluate the results. For regression-regression tasks, we
used RMSE as the metric for assessing outcomes.

Figure 10: The architecture of CNN used in simulations.

###  5.4 Results of simulations

In total, we generated $n=10,000$ synthetic observations in our simulations.
We present the simulation results in Fig. 11 and Fig. 12. We compare the
proposed CeCNN model with the backbone model only. From Fig. 11, we find that
CeCNN improves prediction performance on both regression and classification
tasks. Particularly, the improvement in the MSE metric on the regression task
is more apparent than in the classification task. A possible reason is that
the classification task is much easier in this setting since the function
$g_{2}$ is actually a linear function. For regression-regression tasks, both
AL and SE show remarkable improvements, with AL showing greater improvement
than SE.

Figure 11: Box plots of MSE, Accuracy and AUC in 10 rounds of 5-fold
validation of the regression-classification tasks for simulation data. Figure
12: Box plots of RMSE and MAE in 10 rounds of 5-fold validation of the
regression-regression tasks for simulation data.

##  6 Discussion

In this paper, we analyze one of the earliest batches of UWF fundus image data
from mainland China. We succeed in enhancing the capability of deep learning
models to predict AL, SE, and myopia status through our proposed CeCNN
framework. In a multi-task learning scenario, CeCNN incorporates the
dependence between responses into backbone CNN models through a Gaussian
copula model. The use of the copula model captures the inherent dependence
structure efficiently and enhances the prediction capability of deep learning
models. We verify the flexibility of CeCNN using two different backbone
CNNs,ResNet and Lenet. We also provide a semi-parametric alternative to the
Gaussian copula for the regression-regression task, making CeCNN robust to the
marginal distributions of model error.

The copula parameters may be unlearnable in the sense that they cannot be
updated together with other CNN parameters in the training step. Instead, we
use a warm-up CNN model to estimate them. Note that one should try to
circumvent the overfitting issue in the warm-up step. Recall that in both (4)
and (7), estimation of copula parameters depends on the model residuals of the
regression task. An overfitted warm-up model means that all regression
residuals obtained in the training procedure highly concentrate around zero
and these residuals cannot provide adequate information to estimate the copula
parameters. With ill-estimated copula parameters, CeCNN cannot be properly
trained, let alone enhance prediction capability.

We adopt the strategy of reducing the model size, i.e. the number of CNN
parameters, to circumvent overfitting; refer to subsection 4.2. Though there
are other (maybe better) alternative techniques to overcome overfitting such
as regularization and data augmentation (Bejani and Ghatee,, 2021), we are
limited by the lack of available UWF fundus image data since (the Eye and ENT
Hospital of Fudan University is among the few pioneer hospitals in mainland
China who deployed the Optomap Daytona devices). The huge gap between the
model size and the number of subjects makes neither regularization nor data
augmentation feasible as the former is sensitive to the tuning parameter and
the latter is expensive without any guarantee of quality. Luckily, mainland
China has bee promoting the idea that departments of ophthalmology nationwide
should facilitate myopia diagnostics with UWF fundus images. Once more data
from multiple centers are collected, it is worthwhile and doable to explore
techniques to really resolve overfitting highly over-parameterized models.

As shown in the paper, the CeCNN framework can be equipped with different
types of backbone CNNs. Indeed, CeCNN does not construct a new architecture
for a neural network; rather, it formulates a new loss that can be installed
into any existing backbone CNNs. Furthermore, the proposed framework can be
associated with various deep learning models like generative adversarial
networks (Goodfellow et al.,, 2020) and transformers (Dosovitskiy et al.,,
2020). In this sense, we share a similar spirit to Kim and Wang, (2023) in
adapting a new loss to enhance deep learning models, though they focus on
adversarial learning for multi-class classification, and we focus on multi-
task learning.

Bo Fu and Yang Li are partially supported by National Natural Science
Foundation of China (Grant No. 71991471). Chong Zhong is partially supported
by a grant (GRF1531519) from the Research Grants Council of the Hong Kong SAR
and ZZPC, PolyU. Catherine C. Liu is partially supported by a grant
(GRF15301123) from the Research Grants Council of the Hong Kong SAR.

## References

  * Aggarwal et al., (2017) Aggarwal, K., Agarwal, A., Deokar, A., Singh, R., Bansal, R., Sharma, A., Sharma, K., Dogra, M. R., and Gupta, V. (2017).  Ultra-wide field imaging in paradoxical worsening of tubercular multifocal serpiginoid choroiditis after the initiation of anti-tubercular therapy.  Ocular Immunology and Inflammation.
  * Ang et al., (2019) Ang, M., Wong, C. W., Hoang, Q. V., Cheung, G. C. M., Lee, S. Y., Chia, A., Saw, S. M., Ohno-Matsui, K., and Schmetterer, L. (2019).  Imaging in myopia: potential biomarkers, current challenges and future developments.  British Journal of Ophthalmology, 103(6):855–862.
  * Bejani and Ghatee, (2021) Bejani, M. M. and Ghatee, M. (2021).  A systematic review on overfitting control in shallow and deep neural networks.  Artificial Intelligence Review, pages 1–48.
  * Cen et al., (2021) Cen, L.-P., Ji, J., Lin, J.-W., Ju, S.-T., Lin, H.-J., Li, T.-P., Wang, Y., Yang, J.-F., Liu, Y.-F., Tan, S., et al. (2021).  Automatic detection of 39 fundus diseases and conditions in retinal photographs using deep neural networks.  Nature communications, 12(1):4828.
  * Dosovitskiy et al., (2020) Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020).  An image is worth 16x16 words: Transformers for image recognition at scale.  In International Conference on Learning Representations.
  * Fukushima, (1975) Fukushima, K. (1975).  Cognitron: A self-organizing multilayered neural network.  Biological cybernetics, 20(3-4):121–136.
  * Goodfellow et al., (2020) Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2020).  Generative adversarial networks.  Communications of the ACM, 63(11):139–144.
  * He et al., (2016) He, K., Zhang, X., Ren, S., and Sun, J. (2016).  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778.
  * Holden et al., (2016) Holden, B. A., Fricke, T. R., Wilson, D. A., Jong, M., Naidoo, K. S., Sankaridurg, P., Wong, T. Y., Naduvilath, T. J., and Resnikoff, S. (2016).  Global prevalence of myopia and high myopia and temporal trends from 2000 through 2050.  Ophthalmology, 123(5):1036–1042.
  * Huang et al., (2017) Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K. Q. (2017).  Densely connected convolutional networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700–4708.
  * Ikuno, (2017) Ikuno, Y. (2017).  Overview of the complications of high myopia.  Retina, 37(12):2347–2351.
  * Iwase et al., (2006) Iwase, A., Araie, M., Tomidokoro, A., Yamamoto, T., Shimizu, H., Kitazawa, Y., Group, T. S., et al. (2006).  Prevalence and causes of low vision and blindness in a japanese adult population: the tajimi study.  Ophthalmology, 113(8):1354–1362.
  * Kendall et al., (2018) Kendall, A., Gal, Y., and Cipolla, R. (2018).  Multi-task learning using uncertainty to weigh losses for scene geometry and semantics.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7482–7491.
  * Kim and Wang, (2023) Kim, J. and Wang, X. (2023).  Robust sensible adversarial learning of deep neural networks for image classification.  The Annals of Applied Statistics, 17(2):961–984.
  * Kim et al., (2020) Kim, K. E., Kim, J. M., Song, J. E., Kee, C., Han, J. C., and Hyun, S. H. (2020).  Development and validation of a deep learning system for diagnosing glaucoma using optical coherence tomography.  Journal of clinical medicine, 9(7):2167.
  * Kim et al., (2021) Kim, K. M., Heo, T.-Y., Kim, A., Kim, J., Han, K. J., Yun, J., and Min, J. K. (2021).  Development of a fundus image-based deep learning diagnostic tool for various retinal diseases.  Journal of Personalized Medicine, 11(5):321.
  * Kingma and Ba, (2014) Kingma, D. P. and Ba, J. (2014).  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980.
  * Kobayashi et al., (2005) Kobayashi, K., Ohno-Matsui, K., Kojima, A., Shimada, N., Yasuzumi, K., Yoshida, T., Futagami, S., Tokoro, T., and Mochizuki, M. (2005).  Fundus characteristics of high myopia in children.  Japanese journal of ophthalmology, 49:306–311.
  * LeCun et al., (2015) LeCun, Y., Bengio, Y., and Hinton, G. (2015).  Deep learning.  nature, 521(7553):436–444.
  * LeCun et al., (1998) LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. (1998).  Gradient-based learning applied to document recognition.  Proceedings of the IEEE, 86(11):2278–2324.
  * Li et al., (2023) Li, X., Lv, S., Li, M., Zhang, J., Jiang, Y., Qin, Y., Luo, H., and Yin, S. (2023).  Sdmt: Spatial dependence multi-task transformer network for 3d knee mri segmentation and landmark localization.  IEEE Transactions on Medical Imaging.
  * Li et al., (2021) Li, Z., Guo, C., Lin, D., Nie, D., Zhu, Y., Chen, C., Zhao, L., Wang, J., Zhang, X., Dongye, M., et al. (2021).  Deep learning for automated glaucomatous optic neuropathy detection from ultra-widefield fundus images.  British journal of ophthalmology, 105(11):1548–1554.
  * Lin et al., (2019) Lin, X., Zhen, H.-L., Li, Z., Zhang, Q.-F., and Kwong, S. (2019).  Pareto multi-task learning.  Advances in neural information processing systems, 32.
  * Liu et al., (2019) Liu, M., Zhang, J., Adeli, E., and Shen, D. (2019).  Joint classification and regression via deep multi-task multi-channel learning for Alzheimer’s disease diagnosis.  IEEE Transactions on Biomedical Engineering, 66(5):1195–1206.
  * Meng et al., (2011) Meng, W., Butterworth, J., Malecaze, F., and Calvas, P. (2011).  Axial length of myopia: a review of current research.  Ophthalmologica, 225(3):127–134.
  * Midena et al., (2022) Midena, E., Marchione, G., Di Giorgio, S., Rotondi, G., Longhin, E., Frizziero, L., Pilotto, E., Parrozzani, R., and Midena, G. (2022).  Ultra-wide-field fundus photography compared to ophthalmoscopy in diagnosing and classifying major retinal diseases.  Scientific Reports, 12(1):19287.
  * Murray et al., (2013) Murray, J. S., Dunson, D. B., Carin, L., and Lucas, J. E. (2013).  Bayesian Gaussian copula factor models for mixed data.  Journal of the American Statistical Association, 108(502):656–665.
  * Mutti et al., (2007) Mutti, D. O., Hayes, J. R., Mitchell, G. L., Jones, L. A., Moeschberger, M. L., Cotter, S. A., Kleinstein, R. N., Manny, R. E., Twelker, J. D., and Zadnik, K. (2007).  Refractive error, axial length, and relative peripheral refractive error before and after the onset of myopia.  Investigative ophthalmology & visual science, 48(6):2510–2519.
  * Oh et al., (2023) Oh, R., Lee, E. K., Bae, K., Park, U. C., Yu, H. G., and Yoon, C. K. (2023).  Deep learning-based prediction of axial length using ultra-widefield fundus photography.  Korean journal of ophthalmology: KJO, 37(2):95.
  * Ohno-Matsui et al., (2019) Ohno-Matsui, K., Fang, Y., Shinohara, K., Takahashi, H., Uramoto, K., and Yokoi, T. (2019).  Imaging of pathologic myopia.  The Asia-Pacific Journal of Ophthalmology, 8(2):172–177.
  * Organization et al., (2019) Organization, W. H. et al. (2019).  World report on vision.
  * Parmar et al., (2018) Parmar, C., Barry, J. D., Hosny, A., Quackenbush, J., and Aerts, H. J. (2018).  Data analysis strategies in medical imaging.  Clinical cancer research, 24(15):3492–3499.
  * Sharma et al., (2012) Sharma, T., Raman, R., John, S., and Badrinath, S. (2012).  Telescreening for diabetic retinopathy in south india.  Digital Teleretinal Screening: Teleophthalmology in Practice, pages 137–142.
  * Simonyan and Zisserman, (2014) Simonyan, K. and Zisserman, A. (2014).  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556.
  * Sklar, (1959) Sklar, M. (1959).  Fonctions de repartition an dimensions et leurs marges.  Publ. inst. statist. univ. Paris, 8:229–231.
  * Song, (2007) Song, P. X.-K. (2007).  Correlated data analysis: modeling, analytics, and applications.  Springer.
  * Song et al., (2009) Song, P. X.-K., Li, M., and Yuan, Y. (2009).  Joint regression analysis of correlated data using gaussian copulas.  Biometrics, 65(1):60–68.
  * Tideman et al., (2016) Tideman, J. W. L., Snabel, M. C., Tedja, M. S., Van Rijn, G. A., Wong, K. T., Kuijpers, R. W., Vingerling, J. R., Hofman, A., Buitendijk, G. H., Keunen, J. E., et al. (2016).  Association of axial length with risk of uncorrectable visual impairment for europeans with myopia.  JAMA ophthalmology, 134(12):1355–1363.
  * Walecki et al., (2017) Walecki, R., Pavlovic, V., Schuller, B., Pantic, M., et al. (2017).  Deep structured learning for facial action unit intensity estimation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3405–3414.
  * Wang et al., (2022) Wang, S., Dong, W., Chen, Y., Yi, Z., and Zhong, J. (2022).  An automatic early screening system of eye diseases using uwf fundus images based on deep neural networks.  Intelligence-Based Medicine, 6:100079.
  * Yang et al., (2022) Yang, D., Li, M., Li, W., Wang, Y., Niu, L., Shen, Y., Zhang, X., Fu, B., and Zhou, X. (2022).  Prediction of refractive error based on ultrawide field images with deep learning models in myopia patients.  Frontiers in medicine, 9:834281.
  * Yang et al., (2021) Yang, J., Fong, S., Wang, H., Hu, Q., Lin, C., Huang, S., Shi, J., Lan, K., Tang, R., Wu, Y., et al. (2021).  Artificial intelligence in ophthalmopathy and ultra-wide field image: A survey.  Expert Systems with Applications, 182:115068.
  * Zadnik et al., (2015) Zadnik, K., Sinnott, L. T., Cotter, S. A., Jones-Jordan, L. A., Kleinstein, R. N., Manny, R. E., Twelker, J. D., Mutti, D. O., et al. (2015).  Prediction of juvenile-onset myopia.  JAMA ophthalmology, 133(6):683–689.
  * Zhang et al., (2011) Zhang, M., Gazzard, G., Fu, Z., Li, L., Chen, B., Saw, S. M., and Congdon, N. (2011).  Validating the accuracy of a model to predict the onset of myopia in children.  Investigative ophthalmology & visual science, 52(8):5836–5841.
  * Zhang et al., (2021) Zhang, W., Dai, Y., Liu, M., Chen, Y., Zhong, J., and Yi, Z. (2021).  Deepuwf-plus: automatic fundus identification and diagnosis system based on ultrawide-field fundus imaging.  Applied Intelligence, 51:7533–7551.


# Large Language Models for Semantic Monitoring of Corporate Disclosures: A
Case Study on Korea’s Top 50 KOSPI Companies

Junwon Sung*<EMAIL_ADDRESS>Woojin Heo<EMAIL_ADDRESS>Yunkyung Byun
<EMAIL_ADDRESS>Youngsam Kim
EG Asset Pricing, Republic of Korea<EMAIL_ADDRESS>
###### Abstract

In the rapidly advancing domain of artificial intelligence, state-of-the-art
language models such as OpenAI’s GPT-3.5-turbo and GPT-4 offer unprecedented
opportunities for automating complex tasks. This research paper delves into
the capabilities of these models for semantically analyzing corporate
disclosures in the Korean context, specifically for timely disclosure. The
study focuses on the top 50 publicly traded companies listed on the Korean
KOSPI, based on market capitalization, and scrutinizes their monthly
disclosure summaries over a period of 17 months. Each summary was assigned a
sentiment rating on a scale ranging from 1(very negative) to 5(very positive).
To gauge the effectiveness of the language models, their sentiment ratings
were compared with those generated by human experts. Our findings reveal a
notable performance disparity between GPT-3.5-turbo and GPT-4, with the latter
demonstrating significant accuracy in human evaluation tests. The Spearman
correlation coefficient was registered at 0.61, while the simple concordance
rate was recorded at 0.82. This research contributes valuable insights into
the evaluative characteristics of GPT models, thereby laying the groundwork
for future innovations in the field of automated semantic monitoring.

##  1 Introduction

In recent years, large language models like OpenAI’s ChatGPT have caught
people’s attention [5]. What sets these models apart is their ability to learn
from the context they’re given, making them incredibly versatile when it comes
to handling new data [2, 9]. This means they are great for a wide range of
language tasks, without needing specialized training data or fine-tuning [7,
6, 8, 13].

This is especially useful for tasks that require keeping up with constantly
changing information, like monitoring the news in real-time. In our study, we
are using these advanced models to analyze the sentiment in corporate
announcements. This is an intricate task because these announcements are
usually filled with complex data that takes time and expertise to understand.
We are specifically looking at how well OpenAI’s latest models, like
GPT-3.5-turbo and GPT-4, can perform sentiment analysis on corporate
announcements from Korean companies.

Conceptually, corporate disclosure can be broadly categorized into two types:
periodic and continuous disclosures [11]. Periodic disclosures encompass
routine reports that companies are obligated to submit to the pertinent
regulatory bodies. These reports often encompass financial statements,
quarterly updates, and annual summaries. Conversely, continuous disclosures
encompass significant information that emerges outside the regular reporting
timetable and could potentially influence the valuation of a company’s shares
or other financial instruments. In South Korea, these ongoing reports are
termed ‘timely disclosures,’ while in the United States, they are denoted as
‘current disclosures’ or simply 8-K reports.

Our monitoring system focuses on continuous disclosures, given their often
immediate impact on a company’s financial standing and, subsequently, investor
decisions. This type of disclosure may include, but is not limited to, merger
announcements, changes in executive leadership, regulatory investigations, or
any other material events that could affect the stock price or investor
perception. Given the increasing importance of continuous corporate disclosure
in shaping investor sentiment and market dynamics, the ability to rapidly and
accurately assess these communications is crucial. Traditional methods of
sentiment analysis often involve manual annotations or rely on simpler
algorithms that may not capture the complex events often found in these
announcements [10, 1]. The limitations of these approaches become even more
pronounced when dealing with non-English languages, where cultural and
linguistic subtleties can significantly impact the interpretation of
sentiment.

Our study aims to fill a gap in the existing literature by leveraging the
capabilities of state-of-the-art language models to perform sentiment analysis
on Korean corporate announcements. We seek to answer the following research
questions:

  1. 1.

How effective are large language models, specifically GPT-3.5-turbo and GPT-4,
in analyzing sentiment in corporate announcements?

  2. 2.

What challenges or limitations are associated with using large language models
like GPT-3.5-turbo and GPT-4 for sentiment analysis in the context of
corporate announcements?

The second question aims to uncover any potential drawbacks or limitations of
using these advanced models for this specific task. While these models offer a
range of capabilities, they are not without challenges. These could include
computational constraints or the risk of model biases affecting the analysis.
Understanding these challenges is essential for evaluating the practicality
and reliability of using large language models for real-time semantic
monitoring.

By addressing these research questions, we aim to offer a balanced view of
both the capabilities and limitations of using large language models for
sentiment analysis in Korean corporate announcements. This should contribute
to both academic discussions and provide actionable insights for industry
practitioners.

##  2 Methodology

###  2.1 Data Collection

  * •

Targeted Enterprises: The top 50 companies by market capitalization were
selected from the Korea KOSPI as of June 28, 2023. This decision was based on
their significant influence on the market and the general interest in these
leading companies.

  * •

Data Collection Period: From January 1, 2022, to May 31, 2023.

  * •

Data Collection Methodology: The dataset for this study was sourced from the
Korea Investor’s Network for Disclosure System (KIND). To maintain coherence
and relevance, the disclosures were summarized into single sentences using the
GPT-3.5 model prior to the sentiment rating process. As noted in the
introduction, periodic disclosures were excluded. Specifically, fair reports,
business reports, semi-annual reports, and quarterly reports were not
included. It is worth noting that the average token length for these periodic
disclosures is 65,657, whereas the token length for timely disclosures
averages 2,172.

###  2.2 Data Preprocessing

Data obtained from corporate disclosures was systematically converted into a
monthly time-series format to enable effective sentiment analysis, as depicted
in Table 2. Each data point includes key elements such as the date, time,
title, and a succinct summary of the disclosure’s content. The collection
process is limited to a maximum of 15 disclosures per month for each company.
In cases where a company releases more than 15 disclosures within a single
month, only the most recent 15 are chosen for analysis. This limitation is
imposed primarily due to the context-length constraints of the language models
being used. For example, if a company were to issue daily disclosures from
June 1 to June 30, analytically, it would be more valuable to consider the
disclosures from June 16 to June 30 rather than the initial 15 from June 1 to
June 15.

###  2.3 Utilization of the GPT Model

The GPT model was directed to assign scores between 1 and 5 based on criteria
delineated in Table 1.

Score |  Criteria
---|---
1 (Very Negative) |  The company’s overall situation is very unfavorable, indicating a decline in revenue and profit. Financial conditions are unstable, market share is decreasing, and there are concerns about the ability of management and social responsibility. The future outlook in this situation is highly uncertain, facing threats to the company’s sustainability.
2 (Negative) |  The company’s condition is unfavorable, but certain improvements are possible. The trend of declining revenue and profit continues, and financial conditions are unstable. Market share may vary depending on competitive situations, and evidence of innovation or growth potential is limited. The outlook for the future is not very bright.
3 (Neutral) |  The company’s situation has not changed significantly, indicating that revenue and profit are stable. Financial conditions are stable, and competitiveness in the market is consistently maintained. Innovation and growth potential are average, and the future outlook remains stable without significant changes.
4 (Positive) |  The company is showing significant revenue and growth, indicating that it is being operated well overall. Financial conditions are positive, and there is a trend of increasing market share. There are positive expectations regarding innovation and growth potential, and the outlook for the future is positive.
5 (Very Positive) |  The company is achieving explosive revenue and profit, occupying an outstanding position in the market as a result. Financial conditions are very stable, and market share is dominant. The company possesses excellent innovation and growth potential, and the expectations for the future are very high.
Table 1: Scoring criteria (common for GPT model and human participants)

As illustrated in Table 1, the model was prompted to evaluate several key
factors, including the company’s financial health, market share, and growth
potential. Using the OpenAI API, both the prompt (Table 1) and summarized data
(Table 2) were dispatched to the model, from which the GPT’s rating was then
retrieved.

Date | Time |  Details
---|---|---
2023-06-13 | 16:30 |  Additional Listing (Domestic CB Conversion): CJ CGV Co., Ltd. has additionally listed 383 registered common shares. The issuance price for the 6th time is 26,600 KRW, and for the 9th time is 22,000 KRW. The issuance period is from May 16 to 31, 2023. The dividend base date is January 1, 2023. The capital increase method is a domestic CB conversion, and the listing date is June 16, 2023.
2023-06-20 | 15:49 |  Capital Increase Decision: CJ CGV decided through a board meeting on June 20, 2023, to increase capital by issuing a total of 74,700,000 common shares. They plan to raise 100 billion KRW for facility funds, 90 billion KRW for operational funds, and 380 billion KRW for debt repayment. The issue price of new shares is 7,630 KRW per share. The assignment date for the new shares is July 31, 2023, and the listing date is expected to be September 27, 2023.
2023-06-20 | 16:04 |  Loan Decision: A decision was made to lend money to CJ CGV’s Hong Kong corporation, CGI HOLDINGS LIMITED. The loan amount is 102.456 billion KRW with an interest rate of 7.37%. The loan period is from June 20, 2023, to December 20, 2023. This loan is an extension of an existing loan for the improvement of the subsidiary’s financial structure. The board decision date is June 20, 2023.
2023-06-20 | 16:09 |  Transactions with Affiliates: CJ CGV conducted product and service transactions with its affiliate, CJ OliveNetworks, in the third quarter of 2023. The transaction amount totals 12.349 billion KRW, which is 1.75% of the previous fiscal year’s sales. The transaction details include software and other service contracts, and the contract method is by mutual agreement.
2023-06-20 | 16:12 |  Bond Warning: CJ CGV 35CB (new type) (KR6079161C75) is designated as a bond of concern for investment. This is due to its closing price falling below 80% of its face value.
2023-06-27 | 17:04 |  Additional Listing (Domestic CB Conversion): CJ CGV has additionally listed 378 registered common shares. The issuance prices for the 6th and 9th issues are 26,600 KRW and 22,000 KRW, respectively. The issuance date ranges from June 8 to 15, 2023. The dividend calculation date is January 1, 2023. The method of capital increase is domestic CB conversion, and the listing date is June 30, 2023.
2023-06-30 | 15:50 |  Decision to Provide Collateral for Others: The company decided to provide collateral for CGI Holdings Limited’s borrowings of 26.256 billion KRW from KEB Hana Bank Hongkong Branch. The collateral amount is 29.343 billion KRW, which is 7.46% of the company’s equity capital of 393.089 billion KRW. The collateral provision period is from June 30 to September 27, 2023, and the collateral property is in KRW deposit.
Table 2: CJ CGV prompt data as of end of June

To ensure a uniform and comprehensive analysis, responses from the GPT models
are formatted as depicted in Table 3. For enhanced interpretability, each
generated response is structured to incorporate a brief rationale along with
its corresponding evaluation score.

Rating Score |  Reasons for the score
---|---
2 (Negative) |  CJ CGV is making efforts to secure funds through additional listing and paid-in capital increases. However, given the extension of loans to affiliates, the use of capital increase funds for debt repayment, and the forecast for designation as an investment cautionary stock, the company’s financial status is perceived as unstable. Such circumstances could increase uncertainty about the company’s future growth and potentially weaken its competitiveness.
Table 3: Response example from the GPT-4 model.

###  2.4 Evaluation Method

In our research, we specifically selected the two language models:

  * •

ChatGPT-3.5-turbo-16K

  * •

GPT-4

For the evaluation, both the GPT models and human assessors scrutinized a set
of 815 evaluation queries related to the top 50 companies listed on the KOSPI
index. These queries were assessed using a standardized 1-5 point scale. To
ensure consistency, the prompted queries were kept uniform across all
evaluations. It should be noted that the queries were translated into English
to optimize the performance of the model, a technique commonly employed to
enhance accuracy.

For human evaluation, we collaborated with two highly proficient experts
boasting over a decade of experience within the realm of financial data
analysis. These experts conducted individual assessments of the data,
employing identical criteria to those utilized for evaluating the GPT models.
To ensure a fluid and efficient evaluation process, we equipped the experts
with a user-friendly web-based interface, as depicted in Figure 1.

We utilized Cohen’s Kappa statistic to measure the degree of consistency
between human evaluators, yielding a value of 0.352. This result indicates a
fair level of inter-rater agreement. Additionally, the simple agreement rate
was observed to be 68%, further substantiating the reliability of the
assessment. The scores attributed to each query by the two experts were summed
and subsequently averaged. Any fractional values resulting from this averaging
process were methodically rounded down to the nearest tenth, reflecting a
conservative approach to the rating values.

Figure 1: Disclosure rating screen for human raters

##  3 Experimental Results

###  3.1 Conditions for Rating Adjustments

Previous study has shown that GPT model summaries of public disclosures could
emphasize a positive or negative tone more than human intuition might suggest
[6]. Therefore, this study considered the possibility of bias in the sentiment
score assigned by the GPT model. To evaluate the effect of this potential
bias, we constructed several artificial rating adjustment conditions as
follows:

  * •

Condition 1: No adjustments. This serves as a control group, with no
adjustments made, to have a baseline performance metric.

  * •

Condition 2: Subtract 1 point if the GPT score is 4 or above. This adjustment
helps balance the model’s tendency to be overly positive, making its
evaluations more aligned with human judgment.

  * •

Condition 3: Add 1 point if the GPT score is 2 or lower. By adding a point for
lower scores, you are testing the reverse hypothesis – that the model might be
unduly negative or understate negative tones.

  * •

Condition 4: Add 1 point if the GPT score is 2 or below, and subtract 1 point
if the score is 4 or above. This combines Conditions 2 and 3 to examine
whether the model has both overestimation and underestimation biases at the
same time.

###  3.2 Results of the Conditions

The correlation and concordance rate results for each condition are presented
in Table 4. In Condition 1, The concordance rate is higher in ChatGPT-3.5 than
in GPT-4, yet GPT-4 has a higher Spearman and Kendall coefficient. The GPT-4
model records the highest correlation in Condition 2 (0.82 agreement rate,
0.61 Spearman, and 0.59 Kendall). However, both GPT models show reduced
correlations compared to the baseline condition in Condition 3 and Condition
4.

The key findings from these results are summarized as follows:

  * •

The GPT-4 model demonstrates higher correlations than GPT-3.5 in all
conditions.

  * •

The GPT-4 model in Condition 2 shows the highest performance across all
settings for every measure.

  * •

The effects of rating adjustments are clearly observed in correlation
measures.

The third point on the previous list emphasizes a significant imbalance within
the rating scales. As depicted in Figure 5, human evaluators assigned roughly
75%(615/815) of their ratings to a score of 3. This skew allowed the GPT
models to achieve relatively high performance in terms of concordance rate
under Condition 4, even while registering the lowest correlation values.

The performance pattern of the ChatGPT-3.5 model differs subtly from that of
the GPT-4 model. Specifically, ChatGPT-3.5 achieves its peak concordance rate
under Condition 2, while it registers the highest correlation values in
Condition 1. This variation could suggest that the ChatGPT-3.5 model’s ratings
are less consistent than those generated by the GPT-4 model. For a detailed
view of the rating distribution across various conditions for the models,
please refer to Figures 5 through 5.

Condition | GPT Model | Concordance Rate | Spearman | Kendall
---|---|---|---|---
Condition 1 | ChatGPT-3.5 | 0.60 | 0.48 | 0.46
| GPT-4 | 0.43 | 0.57 | 0.54
Condition 2 | ChatGPT-3.5 | 0.77 | 0.42 | 0.41
| GPT-4 | 0.82 | 0.61 | 0.59
Condition 3 | ChatGPT-3.5 | 0.59 | 0.40 | 0.39
| GPT-4 | 0.39 | 0.51 | 0.48
Condition 4 | ChatGPT-3.5 | 0.76 | 0.13 | 0.13
| GPT-4 | 0.79 | 0.42 | 0.40
Table 4: Performance of GPT models

Figure 2: Condition 1

Figure 3: Condition 2

Figure 4: Condition 3

Figure 5: Condition 4

##  4 Discussions

ChatGPT-3.5 VS. GPT-4. While ChatGPT-3.5 shows a higher concordance rate than
the GPT-4 model in Condition 1 (0.6 versus 0.43), GPT-4 ultimately outshines
ChatGPT-3.5 when comparing their peak performance rates in Condition 2 (0.82
versus 0.77). Notably, the correlations between GPT-4 and human ratings
consistently outperform those of ChatGPT-3.5, indicating that GPT-4 offers
superior consistency. This outcome is not unexpected, given GPT-4’s superior
performance records in tasks related to common sense reasoning [9].

Rating Adjustments. The empirical finding [6] that the GPT models might over-
emphasize positive tones seems validated in our results. Figures 2 and 3
clearly illustrate that both GPT models have a tendency to overestimate
positive disclosures when compared to human evaluators. Interestingly, this
tendency is not symmetrical. As a result, Conditions 4 and 5 yielded
suboptimal performance for the GPT models overall. The GPT-4 model under
Condition 2 aligns most closely with the human evaluators and exhibits the
highest levels of accuracy and correlations in our results. Our empirical
study suggests that prompt engineering alone may be insufficient to address
the domain-specific nuances of financial texts like corporate disclosures.
This is because large language models may struggle to accurately estimate the
impact of such financial events.

Limitations and Challenges. While this study marks a significant stride in the
application of Large Language Models (LLMs) like GPT to corporate disclosures,
several challenges and limitations must be acknowledged. First and foremost,
the GPT models operated without the advantage of background knowledge about
the companies being analyzed. This presented a significant drawback, as it
limited the depth of their analysis and their ability to understand context.
Without knowledge of a company’s history, market position, or unique financial
complexities, the insights generated by the models were inherently limited.

Secondly, the absence of external sources such as financial data in tables or
news articles further constrained the models’ performance. Financial reports
are often intricate documents accompanied by various supplementary data and
market analyses. The inability to integrate and interpret these additional
resources may have prevented a more comprehensive and nuanced evaluation of
corporate disclosures. Furthermore, the mathematical capabilities of the GPT
models present another set of limitations. While they can perform basic
arithmetic and some algebraic operations, their ability to understand and
analyze complex financial formulas or perform advanced statistical analyses is
limited [4]. This is especially pertinent when evaluating corporate financial
reports, which often require a sophisticated understanding of accounting
principles and mathematical models for accurate interpretation. However, it is
worth noting that some of these limitations can be mitigated by employing
expert libraries for the problems that GPT models struggle with [12]. This
hybrid approach could potentially offer a more accurate and nuanced analysis,
bringing together the text-processing strengths of GPT models with the
computational rigor of specialized libraries.

Lastly, LLMs like ChatGPT are subject to modifications, which could lead to
inconsistent performance [3]. This inherent risk of variability must be
addressed to ensure the stability of any monitoring system that relies on
these models.

Future Work. There is significant room for growth and expansion of this
research. Potential avenues include:

  * •

Incorporating the GPT-4 model for the summarization process, hypothesizing
that its advanced capabilities might yield even more accurate summaries than
ChatGPT-3.5-turbo.

  * •

Developing mechanisms to feed contextual information about companies to the
GPT models, allowing for a richer and more nuanced analysis.

  * •

Incorporating external data sources into the analysis, such as financial data
tables and news articles, for a more comprehensive sentiment analysis.

##  5 Conclusion

In conclusion, our study demonstrates the potential of large language models,
such as ChatGPT-3.5-turbo and GPT-4, in performing sentiment analysis on
corporate announcements from Korea’s top 50 KOSPI companies. These models
exhibit varying degrees of success in evaluating the sentiment of continuous
disclosures. Through empirical analysis, we observed that GPT-4 outperforms
ChatGPT-3.5-turbo in terms of both consistency and correlation with human
evaluations, indicating its superior ability to comprehend and assess complex
financial language. The study also highlights the importance of addressing
biases and limitations in LLMs’ output, as evidenced by the need for rating
adjustments to align model-generated scores with human judgment. However,
challenges like the lack of background knowledge, the absence of external data
integration remind us of the evolving nature of LLMs’ capabilities. Future
research should explore approaches to improve contextual understanding,
integrate external data, and enhance mathematical analysis within the
framework of sentiment monitoring in the financial domain. This research
contributes to a deeper understanding of the capabilities and limitations of
LLMs for real-time semantic monitoring and presents valuable insights for both
academia and industry practitioners engaged in sentiment analysis of corporate
disclosures.

## References

  * [1] Sudeep R. Bapat, Saumya Kothari, and Rushil Bansal.  Sentiment analysis of esg disclosures on stock market, 2022.
  * [2] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners, 2020.
  * [3] Lingjiao Chen, Matei Zaharia, and James Zou.  How is chatgpt’s behavior changing over time?, 2023.
  * [4] Simon Frieder, Luca Pinchetti, Alexis Chevalier, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Christian Petersen, and Julius Berner.  Mathematical capabilities of chatgpt, 2023.
  * [5] Will Douglas Heaven.  The inside story of how chatgpt was built from the people who made it.  Technical report, MIT Technology Review, 2023.
  * [6] Alex Kim, Maximilian Muhn, and Valeri Nikolaev.  Bloated disclosures: Can chatgpt help investors process financial information?  arXiv preprint arXiv:2306.10224, 2023.
  * [7] Alex Kim, Maximilian Muhn, and Valeri Nikolaev.  Can gpt-4 support analysis of textual data in tasks requiring highly specialized domain expertise?  arXiv preprint arXiv:2306.10224, 2023.
  * [8] Jessica López Espejel, El Hassane Ettifouri, Mahaman Sanoussi Yahaya Alassan, El Mehdi Chouham, and Walid Dahhane.  Gpt-3.5 vs gpt-4: Evaluating chatgpt’s reasoning performance in zero-shot learning.  arXiv preprint arXiv:2305.12477, 2023.
  * [9] OpenAI.  Gpt-4 technical report, 2023.
  * [10] Sridhar Ravula.  Bankruptcy prediction using disclosure text features, 2021.
  * [11] Jin-Young Yang.  Continuous disclosure practices in the korean equity market.  Macroeconomy and Financial Markets, 2013.
  * [12] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao.  React: Synergizing reasoning and acting in language models, 2023.
  * [13] Wenxuan Zhang, Yue Deng, Bing Liu, Sinno Jialin Pan, and Lidong Bing.  Sentiment analysis in the era of large language models: A reality check.  arXiv preprint arXiv:2305.15005, 2023.

## Appendix: Individual results of the KOSPI 50 Companies

* Concordance Rate and correlation index of the companies in condition 2.
* NaN: When assessing the rating relationship between humans and models, identical values on one or both sides prevent accurate ranking and can lead to undefined correlation coefficients like Spearman or Kendall.

Company Name | Concordance Rate | Spearman | Kendall
---|---|---|---
HD Korea Shipbuilding | 0.65 | 0.31 | 0.3
HD Hyundai Heavy Industries | 0.71 | 0.56 | 0.55
HMM | 0.92 | 0.85 | 0.83
KB Financial | 0.82 | 0.64 | 0.62
LG | 0.75 | -0.04 | -0.04
LG Household & Health Care | 0.81 | 0.74 | 0.72
LG Energy Solution | 0.69 | NaN | NaN
LG Innotek | 0.82 | 0.7 | 0.69
LG Electronics | 0.88 | 0.58 | 0.56
LG Chem | 0.82 | 0.71 | 0.7
NAVER | 0.94 | NaN | NaN
POSCO Holdings | 0.82 | 0.77 | 0.74
S-Oil | 0.71 | 0.59 | 0.57
SK | 0.88 | NaN | NaN
SK Innovation | 0.69 | 0.39 | 0.38
SK Telecom | 1 | NaN | NaN
SK Hynix | 0.94 | 0.91 | 0.9
Korea Zinc | 0.88 | 0.71 | 0.7
Kia | 0.65 | 0.34 | 0.32
IBK | 0.75 | 0.58 | 0.56
Daewoo Shipbuilding | 0.71 | 0.5 | 0.47
Korean Air | 0.88 | 0.77 | 0.76
Doosan Mobility Innovation | 0.56 | 0.28 | 0.28
Lotte Chemical | 0.82 | 0.31 | 0.31
Meritz Financial | 0.81 | 0.45 | 0.45
Samsung SDI | 0.88 | 0.69 | 0.67
Samsung C&T | 0.76 | -0.04 | -0.04
Samsung Biologics | 0.65 | 0.39 | 0.39
Samsung Life | 0.94 | 0.82 | 0.81
Samsung SDS | 0.88 | 0.69 | 0.68
Samsung Electro-Mechanics | 0.88 | 0.77 | 0.75
Samsung Electronics | 0.94 | 0.89 | 0.87
Samsung Fire & Marine Insurance | 0.88 | 0.77 | 0.76
Celltrion | 0.76 | 0.74 | 0.72
Shinhan Financial | 0.88 | 0.69 | 0.68
Woori Financial | 0.94 | 0.86 | 0.86
Kakao | 0.71 | 0.38 | 0.37
Kakao Bank | 0.82 | 0.45 | 0.45
KT | 0.82 | 0.55 | 0.55
KT&G | 0.92 | 0.92 | 0.9
Krafton | 0.94 | 0.88 | 0.87
POSCO International | 0.71 | 0.28 | 0.26
POSCO Futurem | 0.75 | 0.61 | 0.6
Hana Financial | 0.87 | 0.77 | 0.76
HYBE | 0.92 | 0.88 | 0.87
Korea Electric Power | 0.88 | 0.61 | 0.6
Hanwha Solutions | 0.94 | 0.73 | 0.72
Hyundai Glovis | 0.81 | 0.61 | 0.6
Hyundai Mobis | 0.82 | 0.52 | 0.5
Hyundai Motor | 0.88 | 0.69 | 0.68