11institutetext: Tecnologico de Monterrey, Escuela de Ingenieria y Ciencias
Monterrey, Nuevo León, México 22institutetext: Universidad Panamericana,
Facultad de Ingeniería
Josemaría Escrivá de Balaguer 101, Aguascalientes, 20290, México
33institutetext: Université de Lorraine and CNRS, CRAN UMR 7039, 2 avenue de
la Forêt de Haye, F-54518 Vandœuvre-Lès-Nancy Cedex44institutetext: Hôpital
Ambroise Paré, 9 av. Charles de Gaulle, F-92100 Boulogne-Billancourt

# A Novel Hybrid Endoscopic Dataset for Evaluating Machine Learning-based
Photometric Image Enhancement Models

Axel García-Vega 11 Ricardo Espinosa 2233 Gilberto Ochoa-Ruiz Thomas Bazin
1144 Luis Eduardo Falcón-Morales  11 Dominique Lamarque 44 Christian Daul 44

###### Abstract

Endoscopy is the most widely used medical technique for cancer and polyp
detection inside hollow organs. However, images acquired by an endoscope are
frequently affected by illumination artefacts due to the enlightenment source
orientation. There exist two major issues when the endoscope’s light source
pose suddenly changes: overexposed and underexposed tissue areas are produced.
These two scenarios can result in misdiagnosis due to the lack of information
in the affected zones or hamper the performance of various computer vision
methods (e.g., SLAM, structure from motion, optical flow) used during the non
invasive examination. The aim of this work is two-fold: i) to introduce a new
synthetically generated data-set generated by a generative adversarial
techniques and ii) and to explore both shallow based and deep learning-based
image-enhancement methods in overexposed and underexposed lighting conditions.
Best quantitative results (i.e., metric based results), were obtained by the
deep-learnnig -based LMSPEC method, besides a running time around 7.6 fps.

Data available at: https://data.mendeley.com/datasets/3j3tmghw33/1

###### Keywords:

Medical imaging Image enhancement Endoscopy

##  1 Introduction

###  1.1 Medical Context

Endoscopy is the most effective and common used examination tool to prevent
colon cancer by screening for lesions. This technique is used to examine the
human colon through a flexible tube called endoscope, while a camera attached
at the tip gathers visual information in real-time. Additionally to the
camera, a point light provides the lighting source during the surgery.

On the other hand, Minimally Invasive Surgery (MIS) is a type of endoscopic
procedure that is increasingly becoming a mainstream medical procedure,
overtaking traditional surgical operations and treatments. Such endoscopic
intervention entails a less traumatic experience and less pain for the
patient, quicker recovery after surgery and shortened hospital stays [9].

Both endoscopic examinations and MIS procedures require a great deal of
expertise from the surgeon or specialist and demand complicated training and
certifications. Despite of this, there is ample evidence in the literature
that during the vast majority of endoscopic examinations, a great deal of
regions of interest are missed [13], which poses a serious problem, as they
might contain suspicious regions or other lesions such as polyps. Moreover,
such issues make the inspection by the physician a strenuous and time
consuming task. In general, these problems stem from how rapidly the lighting
conditions in the endoscopic video can change from frame to frame (see Fig.
1). In fact, a highly non-linear illumination response produces endoscopic
frames which are highly lit in some sections (overexposed) and poorly lit
(underexposed) in other areas.

Additionally to affecting strongly the efficiency in the lesion detection by
the doctor, the highly changing illumination conditions in endoscopic settings
also hampers the performance of AI-based tools that are being increasingly
developed for Computer-aided Detection (CADe) and Computer-aided Diagnosis
(CADx) applications and in MIS and laparoscopy, among other areas of research,
as we will discuss next.

Figure 1: Light condition changes in sequential video frames. Frame $1$ shows
a normal condition. Later frames show poor conditions while light points to
the closest object.

###  1.2 Motivation for our proposal

In recent years computer vision (CV) has been playing significant role in
endoscopic explorations aided by novel deep learning (DL) techniques, which
give computers the visual and temporal learning abilities to understand
complex surgical procedures in hollow organs [9]. The integration of such
techniques is currently being investigated also for laparoscopy and MIS
applications (i.e., such as instrument tracking, endoscopic view enhancement
and suspicious lesion tracking [9]). Among these applications, 3D surface
reconstruction seems to be a promising solution for the poor depth perception
which prevents the full-fledged adoption of the above-mentioned techniques in
MIS and Robotic-assisted surgery (RAS).

However, as with other applications of computer vision in endoscopy, proper
illumination conditions is an aspect of utmost importance for attaining a high
performance in applications such a CADx and such reconstruction methods. In
the latter case, the techniques traditionally employed for recovering 3D
information, such as Simultaneous Localization and Mapping (SLAM) [7],
Structure for Motion (SfM) and Optical Flow (OF) [15], work poorly on
endoscopic images. This is due to strong photo-metric variations caused by
moving light sources, moist surfaces, occlusions as well as reflections for
surgical instruments that provoke underexposed and overexposed video frames
[13].

As a matter of fact, 3D reconstruction techniques rest on the constraint of
brightness constancy [15]. This constraint assumes that the values in
intensity of the pixels remain constant or with a few variations, which
usually holds in applications in natural imagery (i.e., photography) [1] or
autonomous driving [8]. However, in endoscopic procedures, as Figure 1 shows,
the brightness constancy assumption does not hold due to the numerous
illuminations variations when the camera and light move through the organs.

Therefore, the use of image enhancement techniques for pre-processing such
images is a mandatory step to carry out a robust 3D reconstruction [18] and to
develop reliable and robust CADe/CADx tools.

###  1.3 Contributions and organization of the article

Over the decades, multiple image enhancement techniques have been explored for
illumination adaptation, exposure correction and high-dynamic-range tone
mapping in several applications, to cite a few. Nevertheless, most of the
proposed methods for exposure correction have been designed to deal either
with low-light high-light settings separately, due to the different
characteristics of these enhancing tasks in different areas. In enhancing
endoscopic images, these methods have presented a general lack of robustness
due to the fact that in endoscopy images both problems (under/over exposure)
are present simultaneously.

Additionally, as we will discuss next, most of the machine learning based
approaches to IE require large amounts of paired-data images (i.e., corrupted
and non-corrupted ground truth images) for training and testing purposes.
However, to the best of our knowledge, such a large database does not exist
for testing image enhancement algorithms in endoscopic images.

Therefore, in order to mitigate the problems related to large variations in
illumination in endoscopic interventions, in this paper we present novel
synthetic dataset for testing and evaluating endoscopic image enhancement
methods. In order to attain this goal, of our proposal consists of the
following contributions:

  * •

1\. We introduce a novel dataset containing synthetically generated over and
under exposure frames, in tandem with their respective ground truth images. We
believe that this dataset can serve as a cornerstone for testing well known IE
methods and provide a baseline for future developments in ML for endoscopy and
in DL-based image enhancement methods.

  * •

2\. We perform a thorough evaluation of traditional and DL-based image
enhancement (IE) methods in order to highlight the importance of the ground
truth data and the scarsity of bidirectional (under/over exposure) methods.

The rest of this paper is organized as follows. In Section 3 we present the
datasets from which the raw data was obtained, besides the methodology
utilized for creating the synthetic dataset and both traditional and DL-based
methods for enhancing the synthetic data. In Section 4 we provide a thorough
description of the data preparation and training setup for our experiments.
Also, we present a brief context about metrics implemented and their
respective results. Finally, in Section 5, we give our conclusion and an ideas
for future work.

##  2 State of the art

Mainly, image enhancement consists of two main domains i) spatial domain that
involves direct manipulation of pixels, and ii) frequency domain that involves
Fourier transformation of the input image [22]. IE methods improve the
perception of the image using spatial domain, frequency domain and a
combination of both methods in order to output a modified image on its
contrast, hue, or brightness.

These methods have been approached using either traditional or deep learning
techniques. Some examples of the traditional methods group on the spatial
domain are Histogram Equalization (HE) [19], Dehazing-based methods [17] and
curve adjustment approaches [11]. Another category of image enhancement
approaches are those based on Retinex theory, which decompose the images in
reflection and illumination components [22]; examples of these methods are
Single Scale Retinex (SSR) [16] and multi-scale Retinex (MSR) [16]. These
methods suffer from a limitation in model capacity for the decomposition and
it is difficult to implement successfully in endoscopy scenarios due to the
challenging lighting changes.

Recently, deep learning based methods have been explored for achieving more
accurate and real-time image enhancement procedures with successful results.
However, most the works in the literature have been oriented to the low-light
image enhancement such as deep-learning networks based on the Retinex
assumption, which combines Retinex assumption with the Convolutional Neural
Networks (CNN) [22].

Furthermore, although these methods have shown great promise in natural
images, many of them cannot be directly applied to endoscopic images or
videos. First of all, most of the existing methods can either enhance either
under-or over-exposed images but not both. For instance, Deep UPE [20], or the
work presented in [25, 26] are examples of methods that can be applied
successfully in under-exposed natural images, whereas just a few methods have
been explored to correct both under-exposed and over-exposed images. Recently,
Afifi et al. [1] proposed a novel deep learning based technique to enhance
images in both directions under- and over-exposure in a supervised manner.

Herein, we benchmark this technique against traditional methods in order to
assess our real-synthetic paired dataset and highlight the need for reliable
and standard datasets for future DL-based IE methods in endoscopic procedures.
Nonetheless, another requirement for developing usable IE methods in endoscopy
is related their inference time or latency. Most of the works found in the
literature take several seconds to enhance a single image, hampering their use
in real-time image processing applications such as video processing of
underwater [12], X-ray imaging [14] and restoration in video endoscopy [9] .
Endoscopy is an application area which requires dynamic and real-time image
enhancement, which additionally needs to be accurate in the sense of not
introducing any artifacts and we believe that the work presented in [1] can
serve a launching pad to satisfy these requirements.

Figure 2 shows some examples of endoscopic video frames with some of these
examples. These dramatic changes impede the development of robust computer
vision methods for CADe and CADx among other tasks such as MIS or CIS
(computer integrated surgery). Besides, it has been demonstrated that image
enhancement pre-processing techniques can significantly boost the performance
of 3D reconstruction pipelines using endoscopic images [26].

As detecting artifacts such as under and over exposed frames is so important
for a variety of applications, and in other to improve the applicability of
CADe and CADx application endoscopy, several datasets and challenges have have
been proposed [2, 3]. The focus of these challenges has been to foster the
development of real-time methods for detecting artefacts such as bubbles,
instruments, blood or even some lesions, where under and overexposure play a
major role, as they are typically discarded in many automated procedures.
However, many computer vision algorithms such as SfM or SLAM require as many
frames as possible for maximizing the quality of the obtained results and as
such, discarding such frames is not an option and image enhancement is thus
necessary.

An additional problem is that, to the best of our knowledge, datasets
containing both pairs of ground truth (clean images) and corrupted images
(with exposure errors) does not exist in for endoscopic imaging. This is in
stark contrast to the more conventional/mainstream computational photography
research field, in which several standard datasets have been proposed and are
widely used for testing new algorithms and architectures [22, 6]

Thus, in order to develop and test new image enhancement methods in endoscopy
is necessary to develop and test such dataset from the ground up. The
contribution of the work presented in herein is thus creation of such a novel
real-synthetic paired dataset. For doing so, we leverage three existing
endoscopic datasets and a Generative Adversarial Networks (GANs) approach for
transferring under and over exposed over clean images. In this manner, we have
both a reference image and a corrupted images over which reference-based
metrics (i.e. SSIM) can be applied in order to assess the performance of newly
developed enhancement methods.

##  3 Data and Methodology

###  3.1 Data

As briefly discussed above, the data for our experiments was obtained from
three different sources, none of them containing enough information for doing
image enhancement with deep learning-based methods, i.e., these are not
datasets with paired frames for applying methods where we have access to the
corresponding ground truth. The chosen datasets are: the EAD2020 Challenge
dataset [2], EAD 2.0 from EndoCV2022 Challenge dataset [3] and the HyperKvasir
dataset [5]. They were selected for our experiments as they contain single
frames and also sequential frames from several hollow organs. Figure 2 shows
examples of a normal frame (without exposure error) and two examples
containing exposure problems that were extracted from the datasets.

(a) Without exposure error.

(b) Overexposure.

(c) Underexposure.

Figure 2: Example of endoscopic frames exhibiting the types of artefacts that
we are are interested on removing through image enhancement methods

####  3.1.1 EAD2020 Challenge Dataset

The Endoscopy Computer Vision Challenge (EndoCV2020) addressed two challenges
both focused on finding novel methods for detection and segmentation in the
endoscopic videos: Endoscopy Artifact Detection (EAD2020) and Endoscopy
Disease Detection (EDD2020) [2] challenges. The goal of the EDD2020 sub-
challenge was to develop methods for detecting and segmenting visible
diseases. For this purpose, a full dataset comprised of images, annotations
and masks was created from video frames and annotated by experts from various
institutions. However, the nature of the challenge involves only classes
related to diseases. On the other hand, EAD2020 challenge comprises a variety
of endoscope positions, organs, disease/abnormality and image artefacts. All
frames were annotated by experts with 8 different classes; in particular, two
classes are of interest to our research: contrast and saturation artefacts.
These classes denote what is well-known in photography as overexposure and
underexposure errors but in specific areas on the image. From the 2531 images,
only 770 frames were labeled with underexposure (contrast) and 249 with
overexposure (saturation). Figure 2 shows two examples from the dataset
containing these two kinds of artifacts.

####  3.1.2 EAD2.0 Dataset

Similarly to EAD2020 Challenge Dataset in Sec. 3.1.1, EAD2.0 Dataset [3]
consists of multi-center and diverse population sub-datasets with tasks for
detection and segmentation but focus on assessing generalizability of
algorithms. This dataset contains more sequence/video data and multi-modality
data from different centers. Particularly, the dataset consists of 24
sequences with 1106 video frames with multi-instance labels for training.
However, unlike the EAD2020, EAD2.0 dataset was not labeled with any class
related to underexposure errors. In Section 3.2.1 we will explain briefly the
way we filtered out the dataset in order to have images with exposure errors.

####  3.1.3 HyperKvasir Datset

Borgli et. al in [5] presented the largest image and video dataset of the
gastrointestinal tract available in literature with 110,079 images and 374
videos. The data was collected during real gastro- and colonoscopy
examinations. The dataset is split into labeled images (10,662) and unlabeled
images (99,417). Since this dataset had no a specific purpose, data was not
annotated, and in fact the only labels that it contains are related to the
organ type that the video frames comes from. In present work we only take the
set of labeled images.

###  3.2 Methodology

Given a set of raw endoscopic frames collected from the three datasets
previously mentioned, our pipeline aims to i) train an object detector to
classify frames with and without exposure errors over unlabeled datasets (see
Fig. 3(a)), ii) manually filter out non-informative frames, iii) train a GAN
for creating synthetic frames with exposure errors (see Fig. 3(b)), and iv)
quantitatively and qualitatively assess our dataset on various types of image
enhancement methods: traditional algorithms based on histogram equalization,
models based on Retinex theory and deep neural network architectures.

####  3.2.1 Data Preparation

We discarded non-informative frames. For instance, fully dark, bright or
blurry images, were filtered out. After this process, we ended up with 7,064
images, 1049 from EAD2020, 654 from EAD2.0 and 5361 from HyperKvasir.As we
mentioned before, the EAD2020 dataset already contains annotations about
exposure errors of our concern, hence we trained a YOLOv4 object detector [4]
with data from EAD2020 [2] and we obtained reliable results on detecting
exposure errors. Finally, the model was applied to detect those instances in
EAD2.0 [3] and HyperKvasir [5]. Therefore, as shown in Figure 3, the object
detector acts as a frame classifier, since it outputs frames without exposure
errors, and with over and under exposure errors.

(a) Object detector for classifying frames.

(b) Pipeline for creating our real-synthetic endoscopy dataset for IE.

Figure 3: Overall pipeline of the methodology

####  3.2.2 Image-to-Image Translation

Image-to-image translation is an application of GANs which manage to translate
from one representation of an image to another, or rather it does style
transfer. For instance, a scene may be rendered by a grayscale image, RGB
image or edge sketches [8].

One of the contributions of the present work is to use image-to-image
translation [8] to take normal endoscopic frames and transfer to style with
over- and underexposure. For implementing this task, we used the CycleGAN
architecture [27] since the main problem to tackle is the lack of paired data,
thus CycleGAN is a promising method for working with our unpaired data.

####  3.2.3 Image Enhancement

One of the concerns of this work has been to find image enhancement methods
that perform the restoration of both types of exposure errors, since these
problems are often found simultaneously in endoscopic examinations. Contrary
to other methods in the literature, the model proposed by Afifi et al. [1]is
capable of handling both problems in natural images with accurate results and
with a very low latency and we will used as a baseline for comparison
traditional image processing algorithms as well as for future developments.

The approach proposed by the authors, known as Learning Multi-Scale Photo
Exposure Correction (LMSPEC) extracts random patches of three different sizes
(128x128, 256x256 and 512x512) and decomposes each patch in four-level
frequencies, i.e., it makes a Laplacian Pyramid (LP) of four levels. Then,
each level of the resulting LP is utilized as input of a set of sequential
U-Net-like sub-networks. For each patch, an L1 loss-based loss functions is
proposed, aimed to store global color information besides detail information.
A discriminator network is also for computing an adversarial loss aimed to
preserve realism in the corrected patch. In this work, we implemented a
quantitative and qualitative benchmark using i) traditional methods: RLBHE
[28] and FHSABP [19] based on histogram equalization, and LIME [10] and DUAL
[24] based on Retinex theory, and ii) DL-based method: LMSPEC [1].

###  3.3 Training Setup

All our models were trained and the experiments were executed on an NVIDIA
DGX-1 system with eight Tesla V100 GPUs.

####  3.3.1 YOLOv4 setup

Frames annotated by experts provided on the EAD2020 challenge [2] allowed us
to train YOLOv4 object detector (for classifying frames without exposure
errors, under- and over-exposed) with 90% for training and 10% for testing.
For avoiding over-fitting, data augmentation techniques such as rotations,
splits, blurry, and hue change, were applied. The hyper-parameters were set as
follows: the number of classes were 2 (overexposure and underexposure), thus
the number of filters on each convolutional layer was 30. The training steps
was set to 6000 with an initial step decay learning rate of 0.01 and divided
by factor of 10 at the 4,800 steps and 5,400 steps. The momentum was set as
0.9 and weight decay as 0.0005.

####  3.3.2 CycleGANs setup

For the first experiment all frames with exposure errors, i.e., $1,296$
overexposed and $1,289$ underexposed, were taken as adversarial discriminators
$D_{Y}$. On the other hand, the $4,478$ normal frames were split up into 65%
for training and 35 %for testing. The hyper-parameters were set for both
experiments as follows: 150 epochs, ADAM optimizer, learning rate of 0.0002,
decay $\beta_{1}$ of $0.5$, decay $\beta_{2}$ of 0.999, starting decay from
epoch $100$, cycle loss weight $\lambda_{cyc}$ of $10$ and identity loss
weight $\lambda_{id}$ of 5.

####  3.3.3 LMSPEC setup

We perform the same split for manage both exposure errors, 70% for training,
27% for testing and 3% for validation. We only perform experiments on patches
with dimensions 128x128 and 256x256. The hyper-parameters were setup for both
experiments as follows: ADAM optimizer, decay rate $\beta_{1}$ of $0.9$, decay
$\beta_{2}$ of 0.999, a learning rate of 0.0001 for the generator and 0.00001
for the discriminator; for patches with dimension 128x128 we used a mini-batch
size of 32, 40 epochs and at epoch 20 the learning rate was decayed by factor
of 0.5. For patches with dimension 256x256 we used a mini-batch size of 8, 30
epochs and each 10 epochs the learning rate was decayed by the same factor;
the adversarial loss was activated only for patches with dimensions 256x256
after 15 epochs.

###  3.4 Metrics

One of the drawbacks of implementing a framework like ours, which is based in
GANs and synthetic datasets, is that there are not other works to compare
against our results. Therefore, for evaluating our image enhancement results
we make use of well-known of reference-based metrics, thus the need for a
paired dataset with ground truth images. For instance, in [23] P. Welander et
al. performed a perceptual study for evaluating the realism on the resulting
synthetic images. Unlike them, herein we performed a perceptual study for
evaluating the quality of the generated images. First of all, we use the
Structure Similarity Index (SSIM) for filtering out similar frames, whereas
the Main Square Error (MSE) on the gray-scale domain was used for filtering
out very darkened/brightened frames, or for discarding frames without changes
on light conditions after the GAN style transfer.

On the other hand, for evaluating our image enhancement experiments we
implemented a statistical analysis, ground truth dependent evaluation. We used
the Mean Squared Error (MSE) to evaluate the pixel-wise average squared errors
in the image. We also make use of of the Peak Signal-to-Noise Ratio (PSNR) and
of Structural Similarity Index Metric (SSIM) [21] for evaluating
quantitatively the quality of the enhanced images results.

##  4 Experiments and Results

We implemented three main experiments. i) we created a frames classifier by
training and testing YOLOv4 object detector, ii) we created a paired real-
synthetic dataset by training and testing CycleGANs, and iii) we utilized our
dataset for exposure correction with traditional methods, and training and
testing LMSPEC to highlight the importance of a paired data.

###  4.1 Results

After implementing the CycleGAN model, we obtained 1,564 frames for each type
of exposure error. However, some frames were not useful or sufficiently
informative and thus we carried out a statistical analyses based on the MSE
and SSIM metrics that were used to establish boundaries that separate
informative from non-informative frames. For our dataset, the range for the
SSIM was between $0.6\leq SSIM<0.9$ ; on the other hand, the value of the MSE
over grayscale levels boundaries was $100<MSE<1,500$ for overexposure and
$100<MSE<750$ for underexposure errors. After this procedure, we finally
obtained a dataset with 985 underexposed and 1,231 overexposed frames ground
truth and corrupted images.

Table 1: Results for full reference-based quality experiments. | Overexposure | Underexposure |
---|---|---|---
Method | MSE$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | MSE$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | Inference time
LIME | 0.048 | 8.566 | 0.597 | 0.054 | 17.331 | 0.699 | 44.0236
DUAL | 0.043 | 0.907 | 0.726 | 0.053 | 20.012 | 0.708 | 30.9965
FHSABP | 0.036 | 16.021 | 0.631 | 0.034 | 18.195 | 0.633 | 0.2953
RLBHE | 0.051 | 19.425 | 0.746 | 0.055 | 21.053 | 0.723 | 0.2996
LMSPEC | 0.046 | 24.061 | 0.812 | 0.964 | 23.863 | 0.793 | 0.1316

(a) Overexposure.

(b) Underexposure.

Figure 4: Reference-based PSNR and SSIM evaluation results.

Table 1 reports the quantitative results of each method applied to our
dataset, in which numbers in bold were the best results for each metric. As it
can be inferred from the table, LMSPEC achieves a very competitive performance
for both types of exposure errors in terms of of noise removal and the
preservation of the structure of the image.

On the other hand, Figure 4 shows distribution and density of the computed
metrics (in the form of box and plots) obtained for every applied enhanced
method. It turns out that LMSPEC got higher values for both compared metrics,
and the results tend to be more robust as they show very low variance. Figure
5 shows qualitatively results of enhancement with all methods.

This qualitative comparison is very much required as by comparing methods in
terms of MSE and SSIM might suggest that the the Retinex Theory-based DUAL
method perform as well as LMSPEC. However, as it can be observed in Figure 5,
this is far from the truth, as a comparison between the 4th and 7th columns
shows: DUAL is in general not able to remove overexposure artefacts, and in
some cases it makes the problem worse.

Another advantage of LMSPEC over DUAL and other more traditional methods
pertains the processing or inference time. As it can be observed in Table 1,
more traditional methods require very long processing times (44 and 30 seconds
in average for image for LIME and DUAL, respectively) whereas LMSPEC requires
0.13 seconds, attaining a very good performance (around $7.6fps$).

Figure 5: Results of our methods: first column real images, second column
GANs-made synthetic images with exposure error, and third column the image
corrected. First pair of rows, we show two overexposed cases, and second pair
underexposed cases.

##  5 Conclusion and Future Work

In this paper we have proposed a pipeline for generating a novel dataset which
aim is to provide a baseline for comparing image enhancement methods. The
dataset was built leveraging images frm other publicly available datasets for
finding suitable uncorrupted and under and over exposed images using an object
detector. This intermediate dataset in the used for transferring the desired
artifacts into the uncorrupted images, effectively enabing us to create a
paired dataset that can be used for reference-based testing purposes. After a
filtering process, this dataset was validated by expert endoscopists in our
team.

However, some of the synthetic images still present drawbacks such as noise,
color changes and other types distortions. We consider that a longer training,
in tandem with a larger dataset or data augmentation techniques would
alleviate this issue. Nonetheless, we demonstrate that this dataset is
versatile enough for testing traditional and more advanced image enhancement
methods such as LMSPEC.

As our results suggest, this method has shown remarkable results, but further
tests and improvements are necessary to make it useful for endoscopic. For
instance, the model can introduce some artifacts in some images; this aspect
needs to be properly characterized and taken into account for designing novel
loss functions (i.e. perceptual loss) that preserve color and texture more
effectively. Secondly, there are some areas of improvement in order to make
this model to be able to run in real time (i.e., 24 FPS).

## Acknowledgments

The authors wish to thank the AI Hub and the CIIOT at ITESM for their support
for carrying the experiments reported in this paper in their NVIDIA’s DGX
computer.

## References

  * [1] Afifi, M., Derpanis, K.G., Ommer, B., Brown, M.S.: Learning multi-scale photo exposure correction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9157–9167 (2021)
  * [2] Ali, S., Dmitrieva, M., Ghatwary, N., Bano, S., Polat, G., Temizel, A., Krenzer, A., Hekalo, A., Guo, Y.B., Matuszewski, B., et al.: A translational pathway of deep learning methods in gastrointestinal endoscopy. arXiv preprint arXiv:2010.06034 (2020)
  * [3] Ali, S., Ghatwary, N.: Endoscopic computer vision challenges 2.0, https://endocv2022.grand-challenge.org/
  * [4] Bochkovskiy, A., Wang, C.Y., Liao, H.Y.M.: Yolov4: Optimal speed and accuracy of object detection. arXiv preprint arXiv:2004.10934 (2020)
  * [5] Borgli, H., Thambawita, V., Smedsrud, P.H., Hicks, S., Jha, D., Eskeland, S.L., Randel, K.R., Pogorelov, K., Lux, M., Nguyen, D.T.D., Johansen, D., Griwodz, C., Stensland, H.K., Garcia-Ceja, E., Schmidt, P.T., Hammer, H.L., Riegler, M.A., Halvorsen, P., de Lange, T.: HyperKvasir, a comprehensive multi-class image and video dataset for gastrointestinal endoscopy 7 (2020)
  * [6] Bychkovsky, V., Paris, S., Chan, E., Durand, F.: Learning photographic global tonal adjustment with a database of input / output image pairs. In: The Twenty-Fourth IEEE Conference on Computer Vision and Pattern Recognition (2011)
  * [7] Chen, L., Tang, W., John, N.W., Wan, T.R., Zhang, J.J.: Slam-based dense surface reconstruction in monocular minimally invasive surgery and its application to augmented reality. Computer methods and programs in biomedicine 158, 135–146 (2018)
  * [8] Elgendy, M.: Deep learning for vision systems. Simon and Schuster (2020)
  * [9] Fu, Z., Jin, Z., Zhang, C., He, Z., Zha, Z., Hu, C., Gan, T., Yan, Q., Wang, P., Ye, X.: The future of endoscopic navigation: A review of advanced endoscopic vision technology. IEEE Access 9, 41144–41167 (2021)
  * [10] Guo, X., Li, Y., Ling, H.: Lime: Low-light image enhancement via illumination map estimation. IEEE Transactions on image processing 26(2), 982–993 (2016)
  * [11] Huang, S.C., Cheng, F.C., Chiu, Y.S.: Efficient contrast enhancement using adaptive gamma correction with weighting distribution. IEEE transactions on image processing 22(3), 1032–1041 (2012)
  * [12] Li, C., Guo, C., Ren, W., Cong, R., Hou, J., Kwong, S., Tao, D.: An underwater image enhancement benchmark dataset and beyond. IEEE Transactions on Image Processing 29, 4376–4389 (2019)
  * [13] Ma, R., Wang, R., Pizer, S., Rosenman, J., McGill, S.K., Frahm, J.M.: Real-time 3d reconstruction of colonoscopic surfaces for determining missing regions. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 573–582. Springer (2019)
  * [14] Naidu, S., Parvatkar, P., Quadros, A., Kumar, K.C., Natekar, A., Aswale, S.: Medical image enhancement based on statistical and image processing techniques
  * [15] Phan, T.B., Trinh, D.H., Wolf, D., Daul, C.: Optical flow-based structure-from-motion for the reconstruction of epithelial surfaces. Pattern Recognition 105, 107391 (2020)
  * [16] Rahman, Z.u., Jobson, D.J., Woodell, G.A.: Multi-scale retinex for color image enhancement. In: Proceedings of 3rd IEEE international conference on image processing. vol. 3, pp. 1003–1006. IEEE (1996)
  * [17] Savelli, B., Bria, A., Galdran, A., Marrocco, C., Molinara, M., Campilho, A., Tortorella, F.: Illumination correction by dehazing for retinal vessel segmentation. In: 2017 IEEE 30th International Symposium on Computer-Based Medical Systems (CBMS). pp. 219–224. IEEE (2017)
  * [18] Shao, S., Pei, Z., Chen, W., Zhu, W., Wu, X., Sun, D., Zhang, B.: Self-supervised monocular depth and ego-motion estimation in endoscopy: appearance flow to the rescue. Medical image analysis 77, 102338 (2022)
  * [19] Wang, C., Peng, J., Ye, Z.: Flattest histogram specification with accurate brightness preservation. IET Image Processing 2(5), 249–262 (2008)
  * [20] Wang, R., Zhang, Q., Fu, C.W., Shen, X., Zheng, W.S., Jia, J.: Underexposed photo enhancement using deep illumination estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6849–6857 (2019)
  * [21] Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing 13(4), 600–612 (2004)
  * [22] Wei, C., Wang, W., Yang, W., Liu, J.: Deep retinex decomposition for low-light enhancement. CoRR
  * [23] Welander, P., Karlsson, S., Eklund, A.: Generative adversarial networks for image-to-image translation on multi-contrast mr images-a comparison of cyclegan and unit. arXiv preprint arXiv:1806.07777 (2018)
  * [24] Zhang, Q., Nie, Y., Zheng, W.S.: Dual illumination estimation for robust exposure correction. In: Computer Graphics Forum. vol. 38, pp. 243–252. Wiley Online Library (2019)
  * [25] Zhang, Y., Guo, X., Ma, J., Liu, W., Zhang, J.: Beyond brightening low-light images. International Journal of Computer Vision 129(4), 1013–1037 (2021)
  * [26] Zhang, Y., Wang, S., Ma, R., McGill, S.K., Rosenman, J.G., Pizer, S.M.: Lighting enhancement aids reconstruction of colonoscopic surfaces. In: International Conference on Information Processing in Medical Imaging. pp. 559–570. Springer (2021)
  * [27] Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE international conference on computer vision. pp. 2223–2232 (2017)
  * [28] Zuo, C., Chen, Q., Sui, X.: Range limited bi-histogram equalization for image contrast enhancement. Optik 124(5), 425–431 (2013)


# Cybonto: Towards Human Cognitive Digital Twins for Cybersecurity

Tam n. Nguyen  Department of Management Information Systems
University of Arizona
<EMAIL_ADDRESS>
linkedin.com/in/tamcs/

###### Abstract

Cyber defense is reactive and slow. On average, the time-to-remedy is hundreds
of times larger than the time-to-compromise. In response to the expanding
ever-more-complex threat landscape, Digital Twins (DTs) and particularly Human
Digital Twins (HDTs) offer the capability of running massive simulations
across multiple knowledge domains. Simulated results may offer insights into
adversaries’ behaviors and tactics, resulting in better proactive cyber-
defense strategies. For the first time, this paper solidifies the vision of
DTs and HDTs for cybersecurity via the Cybonto conceptual framework proposal.
The paper also contributes the Cybonto ontology, formally documenting 108
constructs and thousands of cognitive-related paths based on 20 time-tested
psychology theories. Finally, the paper applied 20 network centrality
algorithms in analyzing the 108 constructs. The identified top 10 constructs
call for extensions of current digital cognitive architectures in preparation
for the DT future.

###### Index Terms:

Cybersecurity, Artificial General Intelligence, Human Behavior Modeling,
Cognitive Systems, Cognitive Twins, Digital Twins

##  I Introduction

Humans are recognized to be the weakest link in the cybersecurity defense
chain [1, 2]. Insider threat incidents cost both small and large companies
billions of dollars annually [3]. Cyber defenders are reactive and slow. On
average, hackers need 15 hours to compromise a system while defenders need 200
to 300 days to discover a breach [2]. Meanwhile, the cybersecurity threat
landscape keeps expanding. Cyber defenders respond by enlisting inter-
discipline knowledge from numerous fields such as math, psychology, and
criminology [4, 5, 6, 2]. In such a climate, Digital Twins (DTs) and
incredibly Human Digital Twins (HDTs) offer the capability of running large-
scale simulations across multiple knowledge domains to improve proactive
cyber-defense strategies.

Digital Twins are computational models of physical systems, including humans.
The DT market is rapidly growing at a compound annual rate of 45.4% [7].
Notably, massive DT projects such as the British National Digital Twin [8] are
being built. Within the intertwined DT networks, individual smart DTs such as
HDTs should be capable of not only executing mimetic behaviors but also having
local and global awareness, self-learning, and self-optimizing [7].

For the first time, this paper proposes a grounded vision on how DTs and HDTs
can be applied towards cybersecurity. The main goal is making a case for
expanding current digital cognitive architectures that will be at the hearts
of future HDTs. The paper unified twenty most cybersecurity-relevant finalists
from over seventy behavioral psychology theories. The theory-informed
knowledge and other cybersecurity constructs were then encoded as the novel
Cybonto ontology. Analyzing the Cybonto ontology informed the Cybonto
conceptual framework.

The key contributions are as followed. The Cybonto conceptual framework
solidifies the vision of how human cognitive digital twins and digital twin
systems can be leveraged to design proactive cybersecurity strategies. The
Cybonto ontology provides research-based guidance on 108 constructs and
thousands of possible paths among them. Analyzing the ontology’s cognitive
core using more than 20 network centrality algorithms yields Behavior,
Arousal, Goals, Perception, Self-efficacy, Circumstances, Evaluating,
Behavior-Controlabiity, Knowledge, and Intentional Modality as the top 10 most
influential constructs. These results call for the expansion of current
cognitive architectures to better fit their future employments in DT systems.

##  II Literature Review

The concept of HDTs previously appeared in human-computer interaction studies.
In comparison with traditional models, HDTs for digital twin systems have
broader scopes with emphasis on both behavioral and cognitive activities. The
work of Somers et al. is an excellent example in which HDT acts as a sensible
personal assistant in organizing social events [9]. Notably, the HDT did not
explicitly ask potential event participants for their preferences. Instead, it
observed the people’s social dimensions and then modeled the cognitive
processes underlying an expert event planner’s decisions.

Zhang et al. [10] describes HDTs’ self-awareness as a continuous process that
involves dynamic knowledge acquisition and utilization. Numerous feedback
loops will be needed. Well-designed ontologies are essential for information
exchanges among different models [11, 12]. Compared with an application
ontology, a reference ontology is supposed to be much more canonical and
reusable [13]. Ontological reusability begins with the adoption of a top-level
ontology. Key papers in cognitive frameworks and cybersecurity ontologies are
as followed.

###  II-A Cognitive Frameworks

ACT-R [14] is representative of the psychological modeling group with Clarion
and Epic as other members. SOAR [15] is representative of the agent
functionality-focused group that also includes Sigma, Lida, Icarus, and
Companions. ACT-R and SOAR differ on architectural constraints, memory
retrieval, conflict resolution strategies, and exhaustive processing [16].
ACT-R sequential architecture forces developers to watch out for bottlenecks
while SOAR’s parallel architecture is more relaxed [16]. ACT-R provides two
options for resolving conflicts, while SOAR offers none.

Both SOAR and ACT-R share the same general cognitive cycle and common
architectural modules such as perception, short-term memory, declarative
learning, declarative long-term memory, procedural long-term memory,
procedural learning, action selection, and action. While ACT-R, SOAR, and
other cognitive systems rely on symbolic input/output and rule database, their
symbols may contain statistical metadata, and their architectures do allow the
integration of deep learning systems.

###  II-B Cybersecurity Ontologies

Ontologies are essential for symbolic operations, the building of a knowledge
base, and explainability. Ontologies can be manually build from scratch [17,
18] or be automatically extracted [19, 20]. DOLCE111https://lnkd.in/gTFR8Wt
vs. BFO222https://basic-formal-ontology.org/ highlights the importance of
ontological commitments. DOLCE is grounded in natural language while BFO is
grounded in the real world [21]. Because objects can be conceptual or actual
in a language-based ontology, there is always a risk of one actual object
being recognized as two or more different conceptual objects.

Oltramari et al. [22] introduced Cratelo, which has DOLCE as the top-level
ontology. The ontology’s human behavioral structures are confined within the
cyber operation scope. Costa et al. [23] used the natural language processing
approach in building their Insider Threat Indicator Ontology (ITIO). The
ontology inherited considerable amounts of language ambiguity and did not
support the identification of deeper behavioral structures. In 2019, Greitzer
et al. [24] built upon their 2016’s work and introduced the Sociotechnical and
Organizational Factors for Insider Threat (SOFIT). Due to the absence of a
top-level ontology and the behavioral language that leans heavily towards
organizational insider threat activities, SOFIT is an application ontology
rather than a reference ontology. Greitzer et al. [24] also admitted that
ontology validation exercises only covered 10% of the ontology.

Meanwhile, Donalds and Osei-Bryson [25] reported that cybersecurity ontologies
have been insufficient due to fragmentation, incompatibility, and inconsistent
use of terminologies. The team proposed a cybercrime classification ontology
structured around attack events [25]. While the ontology provides a holistic,
multi-perspective view regarding cybercrime attacks, its behavioral components
are limited and lack theoretical grounding.

TABLE I: Ranking of Top 25 Cybersecurity Related Behavioral Theories Code | Theory Name | Year | Citations |  | Author
---
i-10
| G-scholar
---
Results
| Cyber-
---
security
Impres-
sions
| Cyber-
---
security
Density
| Crimi-
---
nology
Impres-
sions
| Fitted
---
Cita-
tions
/Year
| Final
---
Score
28 | Protection Motivation Theory | 1997 | 1211 | n/a | 10500 | 10 | 9 | 7 | 0 | 6.5
27 | Prospect Theory | 2013 | 68709 | 354 | 66200 | 8 | 1 | 6 | 10 | 6.3
15 | General Theory of Crime | 1990 | 14412 | 91 | 13500 | 9 | 1 | 10 | 1 | 5.3
35 | Self-Efficacy Theory | 1999 | 91578 | n/a | 212000 | 9 | 0 | 6 | 5 | 5
44 | Social Norms Theory | 2005 | 534 | n/a | 47400 | 7 | 9 | 2 | 0 | 4.5
2 | Affective Events Theory | 1996 | 5587 | 66 | 6880 | 10 | 1 | 6 | 0 | 4.3
10 | Differential Association Theory | 1995 | 148 | 31 | 10700 | 9 | 1 | 7 | 0 | 4.3
12 | Extended Parallel Processing Model | 1992 | 3453 | n/a | 412 | 7 | 4 | 6 | 0 | 4.3
14 | Focus Theory of Normative Conduct | 1991 | 3116 | n/a | 6220 | 6 | 10 | 1 | 0 | 4.3
8 | Containment Theory | 1961 | 638 | 43 | 2240 | 9 | 1 | 6 | 0 | 4
49 | Theory of Planned Behaviour | 1991 | 88719 | 217 | 85800 | 9 | 1 | 3 | 3 | 4
41 | Social Identity Theory | 1979 | 25410 | n/a | 66200 | 7 | 0 | 7 | 1 | 3.8
18 | Goal Setting Theory | 1990 | 14489 | 294 | 51700 | 6 | 1 | 7 | 1 | 3.8
51 |  | Transtheoretical Model of
---
Behaviour Change
1997 | 7842 | 307 | 35900 | 6 | 0 | 7 | 0 | 3.3
34 | Self-Determination Theory | 2012 | 1879 | 319 | 165000 | 8 | 0 | 4 | 0 | 3
22 | Operant Learning Theory | 1965 | 22236 | 461 | 40500 | 7 | 1 | 4 | 0 | 3
39 | Social Cognitive Theory | 2001 | 21385 | n/a | 162000 | 8 | 0 | 3 | 1 | 3
5 | Change Theory | 1958 | 74 | n/a | 54700 | 8 | 0 | 2 | 0 | 2.5
23 |  | Precaution Adoption Process
---
Approach
1988 | 1804 | 102 | 2590 | 6 | 1 | 3 | 0 | 2.5
11 | Diffusion of Innovations | 1962 | 125502 | n/a | 96700 | 4 | 1 | 3 | 2 | 2.5
9 | Control Theory | 1982 | 3113 | 381 | 11500 | 6 | 1 | 1 | 0 | 2
33 | Risk as Feelings Theory | 2001 | 6796 | 313 | 550 | 5 | 2 | 1 | 0 | 2
43 | Social Learning Theory | 1941 | 3761 | n/a | 145000 | 2 | 0 | 6 | 0 | 2
21 | Norm Activation Theory | 1992 | 18807 | 234 | 4610 | 5 | 1 | 1 | 1 | 2
45 | Technology Acceptance Model | 2003 | 33231 | 123 | 48100 | 2 | 3 | 1 | 2 | 2
Figure 1: Conbonto Conceptual Framework

###  II-C Open Problems

While massive DT projects are underway, digital cognitive twin development is
pale in comparison, and HDT for cybersecurity is non-existent. This paper
examined both ACT-R and SOAR published research repositories and found no
cybersecurity dedicated track with topics such as cybersecurity, online
ethical decisions, cyber criminology, or cyber attack/defense simulations.
There is no grounded vision on how powerful DT systems with HDTs may improve
proactive cybersecurity defenses. Recommended exploring questions are (i) What
are the types of HDT (malicious hackers, groups as single HDT, defenders,etc.)
to be built? (ii) What will HDT for cybersecurity feedback loops look like?
(iii) How will existing cognitive architectures be extended to best facilitate
those feedback loops? (iv) What shall we learn from our continuous observation
of those HDTs?

Figure 2: Conbonto Ontology - Behavioral/Cognitive’s Horizontal Relationships
Visualized

Current cybersecurity-related cognitive models focus on narrow use cases and
are far from the HDTs that can automatically interact with other DTs while
building up their own awareness. For the main reason, existing cognitive
architectures do not provide enough granularity. This leads to further
problems with multi-modal understanding and meta-cognition. For example,
current long-term memory architecture can be further divided into experiences
and beliefs. It is possible for two persons sharing a strong belief to have
different interpretations of the same data. One may be significantly
influenced by a past experience. Additionally, having access to too much data
due to lack of granularity will lead to cognitive bottlenecks at system
levels. Deciding which chunks of data to be loaded, excluded or be permanently
erased from memory remains a challenge.

Finally, we do not have a reference ontology for documenting and sharing
behavioral-cybersecurity knowledge. Existing cybersecurity ontologies that
have behavioral components are mostly application ontologies with none or weak
ontological commitments. Such ontologies will not fit for use in massive and
complex DT systems.

##  III The Cybonto Conceptual Framework

The novel Cybonto conceptual framework aims to provide general directions on
answering the previously-mentioned questions regarding the vision of DTs and
HDTs for cybersecurity. The framework targets the cognitive process of a
malicious actor as an HDT within a DT system. Cognitive space is defined by
the behavioral/cognitive component of the Cybonto ontology. The action space
is limited by the HDT’s set of encoded actions, its ability to improvise new
moves, and the other DTs’ interaction interfaces. In the beginning, fifty
theories were picked from the behavioral/cognitive psychology body of
knowledge. Each theory was ranked based on its ability to generate research,
relevancy to cybersecurity and criminology, and consistency. Table I presents
the top 25 theories.

Then, each theory was codified into tuples of (entity, ”influence”
relationship, entity). The combination of 20 codified top theories formed the
Cybonto cognitive core ontology with over 100 constructs. Full description of
Cybonto in RDF store, Neo4J relational database, and other documentation are
available at Cybonto-1.0 Github repository
333https://github.com/Cybonto/CYBONTO-1.0. The Cybonto conceptual framework
was formed upon analysis of the Cybonto ontology. Figure 1 presents the
Cybonto conceptual framework with three environment types and four groups of
digital twins (DTs).

The internal environment (INE) is private to each DT. It contains both
cognitive components and non-cognitive components. Opposite to the internal
environment is the societal environment (SOE) where everything is public. In
between, the in-group environment (IGE) connects INE with SOE. All
environments follow Bronfenbrenner’s Ecological System Theory [26] which
describes influences as progressive, varying, and reciprocal forces among
individuals and environments. For example, a seemingly distant public event
may still be able to affect certain private mental processes.

The IEG and the SOE are relative to the targeted HDT. The IEG is equivalent to
Bronfenbrenner’s micro- and meso-systems. The microsystem is the most
inﬂuential external environment with members such as family, close friends,
school, lovers, and mentors. SOE is equivalent to Bronfenbrenner’s Exo-,
Macro-, and Chrono-systems. The Cybonto conceptual framework requires four
representatives from four DT groups. We need one attacker HDT and one defender
HDT. Unlike traditional models to which data and feature specifications were
explicitly fed, an attacker HDT must collect the data by itself. Group-related
data cannot be inferred if the fundamental group structure is not met. Hence,
we then need at least two more DTs to present IEG and SOE identities.

An HDT can perform two main types of behaviors: the artifact creating/altering
behavior and the non-artifact behavior. An artifact can range from a piece of
code to a complex non-cognitive digital twin. Viewing a malware’s codes is a
non-artifact behavior, while running the codes can be an artifact-altering
behavior if the codes make changes to other artifacts. The perceptual layer
sits on the border between the internal and external environments (IEG and
SOE). Different perceptual layers in combination with different cognitive
systems will have different perceptions of the same data streams. Refined
perceptions constitute only a small part of a digital cognitive system. The
Cybonto ontology details thousands of cognitive paths for processing initial
perceptions. The result of a cognitive processing chain will be either a non-
artifact behavior or an artifact creating/altering behavior. The behaviors
(data streams) will be observed by other HDTs, and a new round of feedback
loops begins. It is essential to note that a behavior can be kept secret
within the in-group environment.

In this framework: (a) HDTs have the complete freedom to interact with other
DTs per published protocols, and automatically seek whatever data is made
available to them. (b) By releasing their behaviors, HDTs generate new data,
which may then be consumed by other HDTs. (c) The cognitive architecture
within each HDT determines its cognitive capabilities which should include
awareness and adaptation. (d) Cybonto DT simulation’s objectives should be
more about discovering new knowledge (the Why and How) rather than mining
specific data (the What).

##  IV Analysis of The Cybonto Ontology

Cybonto elected the Basic Formal Ontology (BFO) as its top-level ontology from
more than thirty candidates. BFO is the only top-level ontology that adopts
materialism, commits to actual-world possibilia, and has an intensional
criterion of identity. The Cybonto Core (the behavioral/cognitive component)
is grounded further by employing the Mental Functioning (MF) as its mid-level
ontology. MF follows best practices outlined by the OBO Foundry and aligns
with other projects in the Cognitive Atlas - a state-of-the-art collaborative
knowledge-base in Cognitive Science [27].

Materialism views the world as a collection of materialized objects existing
in space and time [21] \- a core principle in DT strategies. Committing to
materialism through BFO offers a fundamental distinction in the way Cybonto
represents mental constructs. For centuries, cognitive activities were
considered abstract particulars that could only be described through
languages. This tradition is the reason why most behavioral components in
cybersecurity ontologies are language-based. Recent breakthroughs in the
brain-machine interface such as those of Neuralink [28] enables measurements
of brain activities that correspond to certain cognitive constructs.
Therefore, it is now possible to ground behavioral/cognitive ontologies in
materialism. Cybonto rejects conceptual objects, different linguistic
descriptions of the same actual objects, process-based objects, and
qualitative object labels that cannot be measured in real life.

Figure 2 shows the network of Cybonto’s horizontal relationships. Each node
size equals the log scale of the node’s page rank. A darker link color
indicates a higher link value. Nodes were automatically arranged in a multi-
circle layout with higher betweenness centrality nodes closer to the center.
Figure 3 shows the most popular entities based on different network centrality
scores.

Figure 3: Most Influential Constructs Based on Six Types of Centralities

Top Authority Central (AC) constructs receive influence from constructs that
have the most influence on others. Top Betweenness Central (BC) constructs are
the ones that sit in the shortest paths among other constructs. BC constructs
can serve either as bridges or gatekeepers of other constructs and processes.
Top Eigenvector Central (EC) constructs are the leaders of their own cliques.
A clique is a group of constructs in which each member has relationships with
the others. In the context of the cognitive digital twin, a clique may
represent a strong cognitive/behavioral pattern. Not only the top EC
constructs are well-connected with their own clique members, they also have
relationships with other cliques.

Contribution Centrality is Eigenvector Centrality on inverse-Jaccard weighted
values of the input networks. A link between two constructs has the most
contribution weight when the neighbors of one end are most different from the
neighbors at the other end. Top Out-degree Central (OC) constructs have the
most out-links (influencing) to others while top In-coming Central (IC)
constructs are influenced by the most important in-coming neighbors. The top
PageRank constructs have relationships (whether in-coming or out-going) with
the most influential neighbors.

The top 10 constructs across 20 network centrality measures are Behavior,
Arousal, Goals, Perception, Self-efficacy, Circumstances, Evaluating,
Behavior-Controlabiity, Knowledge, and Intentional Modality. In this list,
only Behaviors, Goals, Perception, Evaluating and Knowledge are parts of
existing digital cognitive architectures, although some are not explicitly
implemented. It is possible that before this study, influential cognitive
structures have been studied per independent use-cases and thus could not
collectively attract attention from conservative cognitive system designers.
Now with a bird-eye view across 20 behavioral theories, these top 10
constructs deserve better attention.

Within cognitive architectures, we may consider implementing Goals, Knowledge,
Perception, and Evaluating explicitly and with finer granularity. For example,
Perception is more than short-lived sensory perception. For example, Alice
perceived Bob as a nice guy, and such perception persists whether Bob is with
Alice or not. Additionally, we should consider adding Arousal and Intentional
Modality. Although Arousal is a non-cognitive construct, it is ranked in the
second place and influences several cognitive constructs within the top 10,
such as Evaluating and Intentional Modality. Unfortunately, the current state
of research regarding Arousal as a part of a digital cognitive process is
almost non-existent. SOAR-related research results show a few papers studying
the effects of general emotions. ACT-R research repository shows just four
papers studying the effects of Arousal on memory management.

The Circumstance is another non-cognitive construct with significant influence
on behavioral outcomes. The paper recommends expanding the existing Mental
Image module in existing cognitive architectures to include non-physical
environment variables such as urgency, group dynamics, and social sentiments.
Finally, the paper recommends a new component - Imagining - to enable the HDT
to run its own situational simulations and reason about possible
circumstances.

##  V Conclusion

Once massive non-cognitive digital twin systems are brought online, adding
human cognitive digital twins will be the only logical next step. The vision
of letting human digital twins run free in a digital twin world (and observing
them) is realistic and offers a new paradigm in knowledge mining. The Cybonto
conceptual framework demonstrates how such an ecosystem can be leveraged for
shaping proactive cybersecurity defense strategies. Notably, HDTs are
fundamentally different from deep learning models. Most cognitive systems can
combine human cognitive reasoning (symbolic) with deep learning models (sub-
symbolic). Cognitive reasoning with good enough granularity and a well-
designed ontology allows us to observe and more importantly, to understand
what the digital twins are doing. Hence, the paper also proposes the Cybonto
ontology as specific recommendations on how existing cognitive systems can be
expanded. Future work may involve further framework development, fine-tuning
and expanding the ontology, and building a malicious HDT for demonstration
purposes.

## References

  * [1] B. Bulgurcu, H. Cavusoglu, and I. Benbasat, “Information Security Policy Compliance: An Empirical Study of Rationality-Based Beliefs and Information Security Awareness,” _MIS Quarterly_ , vol. 34, 2010.
  * [2] R. A. Maalem Lahcen, B. Caulkins, R. Mohapatra, and M. Kumar, “Review and insight on the behavioral aspects of cybersecurity,” _Cybersecurity_ , vol. 3, no. 1, 2020.
  * [3] Ponemon Institute, “2020 Costs of Insider Threat Global Report,” Ponemon, Tech. Rep., 2020.
  * [4] K. M. Carley, “Social cybersecurity: an emerging science,” _Computational and Mathematical Organization Theory_ , vol. 26, no. 4, pp. 365–381, 2020. [Online]. Available: https://doi.org/10.1007/s10588-020-09322-9
  * [5] J. Li, K. Larsen, and A. Abbasi, “Theoryon: A design framework and system for unlocking behavioral knowledge through ontology learning,” _MIS Quarterly: Management Information Systems_ , vol. 44, no. 4, 2020.
  * [6] M. Välja, F. Heiding, U. Franke, and R. Lagerström, “Automating threat modeling using an ontology framework: Validated with data from critical infrastructures,” _Cybersecurity_ , vol. 3, no. 1, 2020.
  * [7] P. Eirinakis, K. Kalaboukas, S. Lounis, I. Mourtos, J. M. Rozanec, N. Stojanovic, and G. Zois, “Enhancing Cognition for Digital Twins,” _Proceedings - 2020 IEEE International Conference on Engineering, Technology and Innovation, ICE/ITMC 2020_ , 2020.
  * [8] Centre for Digital Built Britain, “National Digital Twin Programme,” 2019\. [Online]. Available: https://www.cdbb.cam.ac.uk/what-we-do/national-digital-twin-programme
  * [9] S. Somers, A. Oltramari, and C. Lebiere, “Cognitive twin: A cognitive approach to personalized assistants,” _CEUR Workshop Proceedings_ , vol. 2600, 2020.
  * [10] N. Zhang, R. Bahsoon, and G. Theodoropoulos, “Towards Engineering Cognitive Digital Twins with Self-Awareness,” _IEEE Transactions on Systems, Man, and Cybernetics: Systems_ , vol. 2020-Octob, pp. 3891–3896, 2020.
  * [11] Z. Ma, M. J. Schultz, K. Christensen, M. Værbak, Y. Demazeau, and B. N. Jørgensen, “The application of ontologies in multi-agent systems in the energy sector: A scoping review,” _Energies_ , vol. 12, no. 16, pp. 1–31, 2019.
  * [12] M. Bienvenu, B. T. Cate, C. Lutz, and F. Wolter, “Ontology-based data access: A study through disjunctive datalog, CSP, and MMSNP,” _ACM Transactions on Database Systems_ , vol. 39, no. 4, 2014.
  * [13] R. Arp, _Building ontologies with Basic Formal Ontology_ , B. Smith, A. D. Spear, A. D. Spear, d. IEEE Xplore (Online Service), and p. MIT Press, Eds. Cambridge, Massachusetts, 2015.
  * [14] J. R. Anderson, M. Matessa, and C. Lebiere, “ACT-R: A Theory of Higher Level Cognition and Its Relation to Visual Attention,” _Human–Computer Interaction_ , vol. 12, no. 4, pp. 439–462, 12 1997. [Online]. Available: https://doi.org/10.1207/s15327051hci1204_5
  * [15] J. E. Laird, _The Soar Cognitive Architecture_. The MIT Press, 2019.
  * [16] R. M. Jones, C. Lebiere, and J. A. Crossman, “Comparing modeling idioms in ACT-R and SOAR,” _Proceedings of ICCM 2007 - 8th International Conference on Cognitive Modeling_ , pp. 49–54, 2007.
  * [17] M. Uschold and M. King, “Towards a Methodology for Building Ontologies,” in _Towards a Methodology for Building Ontologies_ , 1995.
  * [18] M. Uschold, M. King, S. B. R. House, S. Moralee, and Y. Zorgios, “The Enterprise Ontology,” _The Knowledge Engineering Review_ , vol. 13, pp. 31–89, 1995. [Online]. Available: https://dl.acm.org/citation.cfm?id=976226
  * [19] A. Maedche, B. Motik, and L. Stojanovic, “Managing multiple and distributed ontologies on the Semantic Web,” _VLDB Journal_ , vol. 12, no. 4, pp. 286–302, 11 2003.
  * [20] P. Haase and L. Stojanovic, “Consistent evolution of OWL ontologies,” in _Lecture Notes in Computer Science_ , vol. 3532. Springer Verlag, 2005, pp. 182–197.
  * [21] C. Partridge, A. Mitchell, A. Cook, D. Leal, J. Sullivan, and M. West, “A survey of Top-Level Ontologies - to inform the ontological choices for a Foundation Data Model,” The Centre for Digital Built Britain, Tech. Rep., 2020\. [Online]. Available: www.digitaltwinhub.co.uk.
  * [22] A. Oltramari, L. F. Cranor, R. J. Walls, and P. McDaniel, “Building an ontology of cyber security,” _CEUR Workshop Proceedings_ , vol. 1304, pp. 54–61, 2014.
  * [23] D. L. Costa, M. J. Albrethsen, M. L. Collins, S. J. Perl, G. J. Silowash, and D. L. Spooner, “An Insider Threat Indicator Ontology,” _Carnegie Mellon University_ , no. CMU/SEI-2016-TR-007, pp. 1–87, 2016. [Online]. Available: https://bit.ly/3uHkpx1
  * [24] F. L. Greitzer, J. D. Lee, J. Purl, and A. K. Zaidi, “Design and Implementation of a Comprehensive Insider Threat Ontology,” _Procedia Computer Science_ , vol. 153, pp. 361–369, 2019. [Online]. Available: https://doi.org/10.1016/j.procs.2019.05.090
  * [25] C. Donalds and K. M. Osei-Bryson, “Toward a cybercrime classification ontology: A knowledge-based approach,” _Computers in Human Behavior_ , vol. 92, no. July 2018, pp. 403–418, 2019. [Online]. Available: https://doi.org/10.1016/j.chb.2018.11.039
  * [26] U. Bronfenbrenner, “Ecological systems theory.” in _Six theories of child development: Revised formulations and current issues._ London, England: Jessica Kingsley Publishers, 1992, pp. 187–249.
  * [27] J. Hastings and W. Ceusters, “Representing mental functioning: Ontologies for mental health and disease,” _… Mental Functioning, …_ , pp. 1–5, 2012.
  * [28] E. Musk and Neuralink, “An integrated brain-machine interface platform with thousands of channels,” _bioRxiv_ , p. 703801, 7 2019. [Online]. Available: https://www.biorxiv.org/content/10.1101/703801v2


# BoRA: Bayesian Hierarchical Low-Rank Adaption for Multi-task Large Language
Models

Simen Eide  University of Oslo  Arnoldo Frigessi

(July 2024)

###### Abstract

This paper introduces Bayesian Hierarchical Low-Rank Adaption (BoRA), a novel
method for finetuning multi-task Large Language Models (LLMs). Current
finetuning approaches, such as Low-Rank Adaption (LoRA), perform exeptionally
well in reducing training parameters and memory usage but face limitations
when applied to multiple similar tasks. Practitioners usually have to choose
between training separate models for each task or a single model for all
tasks, both of which come with trade-offs in specialization and data
utilization.

BoRA addresses these trade-offs by leveraging a Bayesian hierarchical model
that allows tasks to share information through global hierarchical priors.
This enables tasks with limited data to benefit from the overall structure
derived from related tasks while allowing tasks with more data to specialize.
Our experimental results show that BoRA outperforms both individual and
unified model approaches, achieving lower perplexity and better generalization
across tasks. This method provides a scalable and efficient solution for
multi-task LLM finetuning, with significant practical implications for diverse
applications.

##  1 Introduction

Large Language Models (LLM) have become a popular model for solving a wide
range of generative text problems. LLMs are typically pretrained on a large
and diverse dataset of texts, and then fine-tuned on a new data set of texts,
dedicated to a specific task. The fine-tuning process is often done by
maximizing the next token log likelihood of the task-specific dataset, and the
resulting model is then used to make predictions and generations for new data.
A popular fine-tuning method is Low-Rank Adaption (LoRA), which reduces the
number of trainable parameters by a factor of up to 10’000 and the GPU memory
requirement by a factor of 3, while still outperforming other fine-tuning
techniques ([3]).

When performing fine-tuning of LLMs, the practitioner often wants to solve
multiple similar tasks at the same time: A media organization might want to
use an LLM to generate various textual elements to generate headlines,
keywords, and summaries for a given news article. A website might have
multiple chatbot tasks answering user inquiries across various topics. And
finally, a scriptwriting AI might want to generate dialogues for different
characters in a movie, each with their unique style and vocabulary.

In these multi-task cases, the practitioner can either (1) train one model for
each task and dataset separately, or (2) train one model for all tasks and
condition the specific output wanted (e.g [7]). There is a trade-off between
these two approaches: Train each task separately and the model will be able to
specialize on each task, but it may suffer when limited data are available for
some task. Train one model for all tasks, and the model will be able to share
information between tasks, but it may not be able to specialize on each task
due to limited model capacity.

In this paper we suggest a way to circumvent these trade-offs and presents a
generalization of the two LoRA fine-tuning methods for multi-task problems,
which we call Bayesian Hierarchical Low-Rank Adaption method (BoRA). The BoRA
method allows tasks to share information between each other through sharing a
set of global hierarchical prior parameters, in a Bayesian hierarchical
setting. This means that tasks with little data can benefit from the global
structure across related tasks, while tasks with more data can specialize on
their specific task. We show in Section 3.4 that BoRA can be seen as a
generalization of the two approaches above.

To test our method, we evaluate the BoRA on the next token generation problem
of parliament speeches letting each parliament representative be a task. In
our setting, BoRA is superior to both conventional approaches described above
and suggests a way to mitigate the trade-off between the two.

The paper is organized as follows: Section 2 presents related works. Section 3
presents BoRA, the Hierarchical LoRA method; Section 4 describes the
experimental setup, Section 5 presents the results, and Section 6 discusses
the results and the implications of the new method.

##  2 Related works

We group related work into two areas: multi-task learning and fine-tuning
methods of large language models. Autoregressive LLMs and Bayesian modeling
are described in Section 3.

###  2.1 Multi-task Learning

Multi-task Learning [1] is a popular machine learning approach for training
one model on multiple tasks. A common approach is to share the lower layers of
a neural network between tasks, and have separate parameters for the last (or
last few) layers for each task. Our paper suggests a different approach, where
we share information between tasks through a hierarchical prior over the task-
specific parameters.

###  2.2 Finetuning methods

A popular approach to finetuning neural networks involves adjusting only the
top layer (or the n top layers) of a pre-trained neural network (e.g. [8]).
These techniques exploit that features learned in the lower layers of the
network are general and transferable, while the top layers are more task-
specific. This approach has been widely adopted in neural networks. However,
in this paper we focus on the LoRA method, which instead allows us to train a
low rank decomposition of the entire network.

There are several variations of the LoRA algorithm. The Sparse Mixture of Low
Rank Adapters (SIRA) [10], for instance, creates a Mixture of Expert on the
low-rank parameters. Another variation, the Weight Decomposed Low-Rank
Adaption (DoRA) [2], decomposes the low-rank parameters into scale and unit
norm direction parameters. There is also the Efficient Low Rank Adaption
(LoRA+) [2], which improves the existing LoRA optimization algorithm by
adjusting the learning rate of the low-rank parameters. Lastly, MoRA [4]
decompose the low-rank parameters using learnable square matrices with non-
parameter based compressions and decompressions. All these methods are
designed to improve the performance of language models similar to LoRA, and
they can be used in conjunction with the proposed Bayesian Hierarchical LoRA
method.

##  3 Method: Hierarchical LLM

This section describes the Hierarchical LoRA method for multi-task problems.
We begin by defining the task and data structure, followed by an introduction
to pretrained autoregressive language models. Next, we describe the LoRA
method, and finally, we present the proposed Bayesian Hierarchical LoRA model.

###  3.1 Task and Data Structure

We define our study across $D$ distinct tasks, denoted by $d=1,2,...,D$. For
each task $d$, we have a dataset $\mathcal{D}_{d}$ containing $N_{d}$
documents, with each document $n=1,...,N_{d}$ consisting of a sequence of
$W_{n}$ tokens. This dataset structure can be formally represented as:

$\mathcal{D}=\\{\mathcal{D}_{d}\\}_{d=1}^{D}=\\{\\{w_{d,n,1:W_{n}}\\}_{n=1}^{N_{d}}\\}_{d=1}^{D}$
(1)

In this notation, $w_{d,n,1:W_{n}}$ represents the token sequence in the
$n^{th}$ document of the $d^{th}$ task. For simplification in subsequent
discussions, we will omit the indices $d$ and $n$ (and write $w_{i}$) when
their reference is evident from the context.

###  3.2 Pretrained Autoregressive Language Model

Our foundational model is a pretrained autoregressive language model, denoted
by its parameters $\theta_{full}$ and its architecture, referred to as a Large
Language Model ($LLM$). The parameters $\theta_{full}$ has been obtained from
pretraining on a large and diverse dataset, distinct from our current dataset
$\mathcal{D}$. The model’s probability distribution is given by:

$P(w_{1:W}|\theta_{full})=\prod_{i=1}^{W-1}LLM(w_{i+1}|w_{1:i})$ (2)

where $LLM(w_{i+1}|w_{1:i})$ is the probability of the next token $w_{i+1}$
given the previous tokens $w_{1:i}$ in the document. We omit to write
$\theta_{full}$ explicitly in the LLM, as the autoregressive language model
always depend on these parameters.

###  3.3 Low-Rank Adaption (LoRA)

A Large Language Model contains many linear layers, and the LoRA method
reduces the number of trainable parameters by reparameterizing the weight
parameters ($W\in\mathbb{R}^{n_{1}\times n_{2}}$, $W\subset\theta$) of these
layers into the original pretrained weights $W_{full}$ and a low-rank
decomposition $A$ and $B$ ([2]):

$W=W_{full}+\frac{\alpha}{r}BA$ (3)

where $W_{full}\in\mathbb{R}^{n_{1}\times n_{2}}$ is the original pretrained
weight, $B\in\mathbb{R}^{n_{1}\times r}$ and $A\in\mathbb{R}^{r\times n_{2}}$
are the low-rank factors, $n_{1}$, $n_{2}$ are the input and output dimension
of the original weight matrix in the linear layer,
$r\ll\text{min}(n_{1},n_{2})$ is the low rank dimension and $\alpha$ is a
scaling hyperparameter.

During LoRA fine-tuning, the low-rank factors $A$ and $B$ are learned
(estimated) from data, while the original pretrained weights $W_{full}$ are
kept fixed to their pretrained value.

###  3.4 The Bayesian Hierarchical LoRA model

In the hierarchical LoRA model, we introduce a LoRA parameter set for each
task $d$, denoted as $\theta_{d}$, containing all the low-rank decomposition
matricies $A_{d}$ and $B_{d}$ from each decomposed layer in the Large Language
Model. The likelihood of the data $\mathcal{D}$ given the task parameters
$\theta_{1:D}$ is then given by (combining equation 1 and 2):

$\displaystyle\text{L}(\mathcal{D}|\theta_{1},\theta_{2},\ldots,\theta_{D})=\prod_{d=1}^{D}L(\mathcal{D}_{d}|\theta_{d})$
$\displaystyle=\prod_{d=1}^{D}\prod_{n=1}^{N_{d}}\prod_{i=1}^{W_{n}-1}\text{LLM}(w_{d,n,i+1}|w_{d,n,1:i};\theta_{d}).$
(4)

Tasks will share some knowledge with each other, and we model this by
introducing a Gaussian hierarchical prior over the task parameters
$\theta_{1:D}$, given by

$P(\theta_{1:D}|\Theta,\tau)=\prod_{d=1}^{D}N(\theta_{d};\Theta,\frac{1}{\tau}I)$
(5)

where $\Theta$ is a set of hierarchical mean parameters, $I$ is a unit
diagonal matrix and $\tau\geq 0$ is a scalar precision hyperparameter
controlling how similar the task parameters $\theta_{d}$ should be to the
hierarchical mean parameters $\Theta$. We let $\Theta$ have a uniform
hyperprior (i.e. $P(\Theta)=1$). The prior assumes that the task parameters
share a common origin from the common Gaussian prior, because the tasks are
related: input data come from similar distributions and the output tasks have
similar aspects. The precision hyperparameter $\tau$ controls how much the
task parameters $\theta_{d}$ are a priori expected to be allowed to deviate
from the hierarchical mean parameters $\Theta$. This is a measure for how much
structure and information the tasks share with each other, and should be set
by the practitioner based on her’s prior knowledge.

The posterior distribution of the hierarchical model is then given by
combining equation 3.4 and 5:

$P(\theta_{1:D}|\mathcal{D},\tau)\propto\prod_{d=1}^{D}\text{L}(\mathcal{D}_{d}|\theta_{d})\cdot
P(\theta_{d}|\Theta,\tau)$ (6)

The resulting BoRA model solves each task based on its data size and
similarity to the other tasks: If a task has few data points, the posterior in
equation 6 will be dominated by the prior term and the task parameters will be
close to the global hierarchical parameters and borrow information and
structure from these. On the other hand, if a task has a lot of data points,
the likelihood will dominate the posterior and the task parameters will be
allowed to learn more about the specific task and rely less on the
hierarchical parameters.

The hierarchical LoRA model can be seen as a generalization of the two
conventional approaches for multi-task problems: If we let $\tau$ go to zero,
the probability distribution of the prior (eq. 5) will become a flat uniform
prior and the problem will be equivalent to training each task independently.
On the other hand, if we let $\tau$ go to infinity, the prior will become a
strong constraint on the task parameters not allowing them to deviate from the
global parameters, and the problem will be equivalent to training one model
for all tasks.

###  3.5 Optimization

Given the precision hyperparameter $\tau$, we want to find the maximum a
posteriori (MAP) estimate of the hierarchical model. We will do this by using
AdamW ([6]), a gradient-based optimization algorithm, optimizing over the task
parameters $\theta_{1:D}$ and the hierarchical mean parameters $\Theta$ using
the posterior in equation 6. This implies that the task parameters with have
gradients both towards minimizing the convential LLM loss (represented by the
likelihood term) and the hierarchical mean parameters (originating from the
prior). The hierarchical mean parameters $\Theta$ will have gradients towards
the average of the task parameters, and learn to represent the global
structure of the tasks.

An important observation is that the gradient of the prior term in equation 6
is proportional to the precision hyperparameter $\tau$. This means that the
optimization will need more steps to converge for larger values of $\tau$, as
the task parameters will be more constrained towards the hierarchical mean
parameters, and a posteriori dependent on each other. Therefore, we adjust the
learning rate of the AdamW optimizer to be proportional to the precision
hyperparameter $\tau$, and we find in our experiments that this causes the
optimizations to converge at a similar rate for different values of $\tau$.

##  4 Experimental setup

Figure 1: Number of training examples for each task.

To test our method, we optimize the model with a range of different precision
hyperparameters $\tau$, and evaluate the perplexity on the test set for each
task. Perplexity is defined as the exponentiated average negative log-
likelihood per token, and is a common metric for evaluating language models. A
lower perplexity indicates a higher likelihood and a better model. We use the
Talk of Norway Dataset [5], which consists of speeches of Norwegian parliament
politicians. Speakers come from different political parties and geographical
areas, but the tasks will still share a common domain, represented by
parliamentary speeches in Norway over a short period of time. Therefore, we
can consider different speakers of this dataset as different tasks.
Specifically, we select chronologically the 25 first speakers that have above
100 speeches in the dataset. This gives us a total of 25 tasks with samples
ranging from 110 to 477 documents, and an average of 202 documents per task.
For some speakers we have significantly less data compared to others. The
distribution of samples per task can be seen in Figure 1. We leave 33% of the
data for the test set.

We select a model that has been pretrained on a large and diverse dataset, but
has not seen the specific tasks that we want to evaluate it on. We will use
the ‘opt-350m’ model [9], which is predominantly trained on English, and has
only seen a small amount of Norwegian data from the CommonCrawls dataset.

##  5 Results

The overall results of the hierarchical LoRA model on the Talk of Norway
Dataset are shown in Table 1 and Figure 2. We see that the model achieves the
best test perplexity of 12.82 when using a precision parameter of $100$. The
improvement is largest from training the models independently ($\tau=0$), and
smaller when comparing to the limiting case when all task specific model
parameters are constraint to be equal ($\tau=10000$).

Learning Rate | Precision ($\tau$) | Test Perplexity
---|---|---
$10^{-4}$ | 0 | 16.80
$10^{-5}$ | $10^{0}$ | 16.59
$10^{-4}$ | $10^{1}$ | 12.85
$10^{-3}$ | $10^{2}$ | 12.82
$10^{-2}$ | $10^{3}$ | 13.26
$10^{-1}$ | $10^{4}$ | 13.91

Table 1: Empirical results of the hierarchical LORA finetuned on the three
tasks. The perplexity is calculated on the test set.

Figure 2: Plot of Precision vs Perplexity. The thick black line is the overall
test perplexity over all tasks, and the thinner grey lines are the test
perplexity for each individual task.

If we consider each task separately (Table 2), we see that all tasks benefit
from a hierarchical model compared to both the case when all models are
trained independently ($\tau=0$) and the limiting case when all task specific
model parameters are constraint to be equal ($\tau=10000$). Figure 3 shows the
relative improvement of each task when using the hierarchical model compared
to the two cases versus their training data size. There is an indication that
the improvement from training separately is larger for tasks with less data,
and we do not see this for the ”one model” case. A precision parameter of
$100$ gives the best perplexity for all our tasks.

Figure 3: Task dataset size versus the relative improvement in perplexity of
the best performing hierarchical model ($\tau=100$) compared to the case when
all models are trained independently ($\tau=0$) in blue, and compared to the
limiting case when all task specific model parameters are constraint to be
equal ($\tau=10000$) in red. The figure shows that all tasks benefit from
sharing parameters with the global model, and indicates that the tasks with
less data benefit more than the tasks with more data.

##  6 Discussion

Our results in Figure 2 show that the hierarchical LoRA model can improve the
test perplexity for all tasks compared to both training the models
independently and training one model for all tasks. This confirms our
hypothesis that BoRA can be a useful method for multi-task problems, and that
it can mitigate the trade-off between training each task separately and
training one model for all tasks. It is interesting to note the BoRA also
helps for the tasks with relatively more data.

We observe in Figure 4 that the distance between the task parameters and the
global prior increases with increasing task dataset size. This is as expected
in a hierarchical model, as tasks with more data will have a larger likelihood
term in the posterior and will be allowed to specialize more on their specific
task during training.

We find no relationship between dataset size and the final perplexity (Figure
5). One could expect that tasks with more data are able to learn more about
their specific task and achieve a lower perplexity, but this is not the case
in our experiments. There are many other task specific factors such as under-
represented dialects and political views that can influence the final
perplexity. Controlling for these factors is difficult and out of scope for
this paper, but we believe that these factors may be the reason for this
result.

However, when considering the relative improvement of the best performing
model, the results in Figure 3 show tasks with less data may benefit more than
the tasks with more data. Although a weak signal, this is an expected result,
as the hierarchical model allows tasks with little data to borrow information
and structure from the global model, while tasks with more data can specialize
on their specific task.

In principle, one could adopt a fully Bayesian approach, instead than the
Maximum a Posteriori maximization, which would tell us how much we can trust
the hierarchical model. Such an approach would require the use of Markov Chain
Monte Carlo (MCMC), Variational Inference or other methods. This would require
to sample from the posterior distribution of the hierarchical model, and would
allow to obtain uncertainty estimates for the task parameters. However, MCMC
methods are computationally heavy, and it is not clear to us how to best
sample from the posterior distribution of the language model in an efficient
and scalable way. A middle ground between a full Bayesian approach and the MAP
estimate is to be Bayesian for a few parameters. For instance, instead of our
approach where we maximize the likelihood of the validation set with respect
to the precision parameter $\tau$, one could be Bayesian and integrate over
its empirical posterior distribution obtained from the MCMC. This would give
us an estimate of how well the tasks share information with each other in the
specific problem. We leave these questions for future work.

Figure 4: The figure shows the L2-distance between each task’s adapter weight
and the global prior on the y-axis, and the number of training examples on the
x-axis. We see that, as expected, tasks with more training data have a larger
distance to the global prior.

Figure 5: Training size vs perplexity of the best performing hierarchical
model ($\tau=100$). The figure shows that there is no strong relationship
between dataset size and final perplexity.

##  7 Conclusion

This paper presented a new generalization of the LoRA fine-tuning method for
multi-task problems, which we call BoRA, Bayesian Hierarchical Low-Rank
Adaption for Multi-task Large Language Models. The hierarchical model allows
tasks to share information between each other through a set of global
hierarchical prior parameters, and can be seen as a generalization of the two
conventional approaches for multi-task problems. We show that BoRA can improve
the test perplexity for all tasks compared to both training the models
independently and training one model for all tasks, and that it can mitigate
the trade-off between the two approaches. We propose to adapt the learning
rate to the design parameter that controls the trade off between likelihood
and prior. Future work includes investigating the effect that BoRA has on more
tasks and datasets, investigating how the relative capacity of the global
model affects the performance, and investigating the full posterior
distribution of the model.

## References

  * [1] Rich Caruana  “Multitask Learning”  In _Machine learning 28_ , 1997, pp. 41–75  DOI: 10.1023k
  * [2] Soufiane Hayou, Nikhil Ghosh and Bin Yu  “LoRA+: Efficient Low Rank Adaptation of Large Models”  arXiv, 2024  arXiv: http://arxiv.org/abs/2402.12354
  * [3] Edward J. Hu et al.  “LoRA: Low-Rank Adaptation of Large Language Models”  arXiv, 2022  DOI: 10.48550/arXiv.2106.09685
  * [4] Ting Jiang et al.  “MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning”  arXiv, 2024  arXiv: http://arxiv.org/abs/2405.12130
  * [5] Emanuele Lapponi, Martin G. Søyland, Erik Velldal and Stephan Oepen  “The Talk of Norway: a richly annotated corpus of the Norwegian parliament, 1998–2016”  In _Language Resources and Evaluation_ 52.3, 2018, pp. 873–893  DOI: 10.1007/s10579-018-9411-5
  * [6] Ilya Loshchilov and Frank Hutter  “Decoupled Weight Decay Regularization”  arXiv, 2019  DOI: 10.48550/arXiv.1711.05101
  * [7] Text-to-text Transformer et al.  “Exploring the limits of transfer learning with a unified text-to-text transformer”  In _arXiv_ 21, 2019, pp. 1–67  arXiv:1910.10683
  * [8] Jason Yosinski, Jeff Clune, Yoshua Bengio and Hod Lipson  “How transferable are features in deep neural networks?”  In _Advances in Neural Information Processing Systems_ 27, 2014  arXiv: http://arxiv.org/abs/1411.1792
  * [9] Susan Zhang et al.  “OPT: Open Pre-trained Transformer Language Models”  arXiv, 2022  DOI: 10.48550/arXiv.2205.01068
  * [10] Yun Zhu et al.  “SiRA: Sparse Mixture of Low Rank Adaptation”  arXiv, 2023  arXiv: http://arxiv.org/abs/2311.09179

##  Appendix A Task specific test perplexities

Table 2 shows the test perplexities for each task when using BoRA with
different precision hyperparameters $\tau$.

Table 2: Test perplexity for each task and precision hyperparameter $\tau$

tau | 0 | 1 | 10 | 100 | 1000 | 10000
---|---|---|---|---|---|---
Task |  |  |  |  |  |
abid q raja | 15.56 | 15.20 | 12.57 | 11.60 | 11.91 | 12.41
akhtar chaudhry | 14.78 | 14.37 | 12.01 | 11.43 | 11.66 | 12.06
aksel hagen | 15.88 | 15.30 | 12.68 | 11.84 | 12.26 | 12.61
alf egil holmelid | 16.96 | 16.86 | 13.97 | 12.66 | 13.35 | 14.51
anders b werp | 16.86 | 16.35 | 13.04 | 11.87 | 12.54 | 13.00
andré n skjelstad | 15.53 | 15.23 | 12.59 | 11.58 | 11.98 | 12.41
andré oktay dahl | 16.01 | 15.85 | 13.50 | 12.55 | 12.88 | 13.48
anette trettebergstuen | 15.98 | 15.73 | 13.43 | 12.57 | 12.90 | 13.28
anita apelthun sæle | 25.44 | 25.42 | 21.04 | 19.21 | 20.16 | 21.37
anne tingelstad wøien | 16.41 | 16.23 | 13.60 | 12.43 | 12.90 | 13.54
anniken huitfeldt | 14.44 | 14.39 | 12.33 | 11.66 | 12.05 | 12.53
ansgar gabrielsen | 16.10 | 16.16 | 14.07 | 13.23 | 13.68 | 14.19
arild grande | 16.65 | 16.29 | 13.31 | 12.21 | 12.56 | 13.08
arne lyngstad | 15.57 | 15.28 | 12.12 | 11.25 | 11.66 | 12.08
arve kambe | 17.78 | 17.40 | 14.24 | 13.11 | 13.43 | 13.79
asmund kristoffersen | 17.15 | 16.75 | 13.63 | 12.61 | 13.03 | 13.61
audun lysbakken | 15.21 | 15.25 | 13.22 | 12.48 | 12.67 | 13.13
bendiks h arnesen | 14.34 | 14.23 | 12.04 | 11.11 | 11.42 | 11.97
bente thorsen | 16.35 | 15.88 | 12.87 | 11.60 | 12.28 | 13.06
bjørg tørresdal | 16.02 | 15.65 | 12.78 | 11.58 | 11.92 | 12.46
bjørn hernæs | 16.33 | 15.94 | 13.68 | 12.76 | 13.08 | 13.58
bjørn jacobsen | 22.28 | 22.87 | 19.54 | 18.52 | 19.05 | 20.53
bjørn lødemel | 22.91 | 24.09 | 19.65 | 18.08 | 19.39 | 21.08
bjørn tore godal | 16.57 | 16.30 | 14.55 | 14.02 | 14.22 | 14.48
borghild tenden | 14.42 | 14.84 | 12.12 | 11.10 | 11.68 | 12.41