# DLVGen: A Dual Latent Variable Approach to Personalized Dialogue Generation

Jing Yang Lee1 , Kong Aik Lee2 and Woon Seng Gan1
1School of Electrical and Electronic Engineering, Nanyang Technological
University, Singapore
2Institute for Infocomm Research, A*Star, Singapore
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
https://orcid.org/0000-0002-3611-3512 https://orcid.org/0000-0001-9133-3000
https://orcid.org/0000-0002-7143-1823

###### Abstract

The generation of personalized dialogue is vital to natural and human-like
conversation. Typically, personalized dialogue generation models involve
conditioning the generated response on the dialogue history and a
representation of the persona/personality of the interlocutor. As it is
impractical to obtain the persona/personality representations for every
interlocutor, recent works have explored the possibility of generating
personalized dialogue by finetuning the model with dialogue examples
corresponding to a given persona instead. However, in real-world
implementations, a sufficient number of corresponding dialogue examples are
also rarely available. Hence, in this paper, we propose a Dual Latent Variable
Generator (DLVGen) capable of generating personalized dialogue in the absence
of any persona/personality information or any corresponding dialogue examples.
Unlike prior work, DLVGen models the latent distribution over potential
responses as well as the latent distribution over the agent’s potential
persona. During inference, latent variables are sampled from both
distributions and fed into the decoder. Empirical results show that DLVGen is
capable of generating diverse responses which accurately incorporate the
agent’s persona.

##  1 INTRODUCTION

Personalized dialogue generation refers to the task of generating coherent,
fluent dialogue consistent with a specific persona or personality. The
generation of personalized dialogue is key to achieving natural, human-like
dialogue. Conventionally, this task requires a representation of the
interlocutor’s persona or personality. This representation can either be
explicitly provided in the form of a textual persona description, which
consists of multiple persona statements, or via personality related metadata
such as gender, age etc. Recent work have also explored inferring the persona
statement directly from the dialogue context. The persona/personality
representation, along with the dialogue context, is then used to condition the
decoder.

Due to the difficulty of crafting persona descriptions and personality
representations in a practical setting, the Persona Agnostic Meta-Learning
(PAML) framework (Madotto et al.,, 2019) and the Multi-Task Meta-Learning
(MTML) framework (Lee et al.,, 2021) were proposed. Both PAML and MTML
involved pretraining the model via meta-learning and finetuning with dialogue
examples corresponding to the dialogue agent’s persona. Similarly, (Zheng et
al., 2019b, ) also introduced a pretraining and finetuning approach for
persona-sparse dialogue which features an attention routing structure and a
learned persona attribute embedding. However, even though these methods do not
require the explicit provision of any persona/personality information,
multiple dialogue examples corresponding to the dialogue agent’s persona is
still needed to finetune the model. This is impractical as corresponding
dialogue examples are rarely available in real-world implementations. Hence,
in this paper, we propose a novel Dual Latent Variable Generator (DLVGen) for
personalized dialogue generation, which does not rely on any
persona/personality information or on any corresponding dialogue examples.
Instead, DLVGen generates the response given only the dialogue context.

A common issue with regard to personalized dialogue agents is the low response
diversity. Prior work address this issue by exploring the application of
latent variable models, specifically the Conditional Variational Auto Encoder
(CVAE) (Sohn et al.,, 2015). These approaches involved modelling the potential
dialogue responses as a latent Gaussian distribution, which improves diversity
due to the introduced stochasticity. Typically, the persona or personality
information and dialogue history are used to generate the latent Gaussian
distribution. During inference, the persona or personality vector, the
dialogue context, along with a latent variable sampled from the latent
distribution, are passed to the decoder for response generation. The Persona-
CVAE framework (Song et al.,, 2019) aims to generate responses which
incorporate information from the provided textual persona description, which
is incorporated into the decoding process via a multi-hop attention mechanism.
On the other hand, the Persona-Aware Variational Response Generator
(PAGenerator) (Wu et al.,, 2020) relies on user embeddings trained
concurrently with the CVAE. In addition to the CVAE, the basic Variational
Auto Encoder (VAE) and the Wasserstein Auto Encoder (WAE) have also been
applied to personalized dialogue generation in a similar manner (Chan et al.,,
2019). The Common Sense and Persona Aligned Chatbot (COMPAC) (Majumder et
al.,, 2020), on the other hand, relies on a latent variable to select the
appropriate persona information from a set of expanded persona descriptions.

Unlike the aforementioned models, DLVGen involves generating two latent
Gaussian distributions: the latent distribution over the agent’s potential
persona (defined by multiple persona statements in the persona description),
and the latent distribution over potential dialogue responses. While it has
been established that modelling the latent distribution over responses would
increase response diversity, we hypothesize that modelling latent distribution
over the agent’s potential persona would result in responses which incorporate
a range of potential persona information inferred from the dialogue context.
We utilize the persona description only during training of a neural network
that is tasked with generating the latent distribution over the agent’s
potential persona. To the best of our knowledge, this is the first attempt at
modelling the latent distribution over the agent’s _potential_ persona. For
this paper, our contributions are three-fold:

  1. 1.

We propose the DLVGen framework which leverages persona descriptions only
during training. Unlike prior frameworks, the proposed framework models both
the latent distribution over potential dialogue responses as well as the
latent distribution over the agent’s potential persona.

  2. 2.

We introduce a variance regularization technique which involves maximizing or
minimizing the variance of the distribution over the agent’s potential persona
and the distribution over potential responses respectively. In particular,
evaluation results reveal that minimizing the variance of the distribution
over potential responses improves the persona consistency of the generated
responses.

  3. 3.

We present a selection framework based on lexical diversity to select the
final response from a pool of generated responses. Generally, we find that the
response selected via lexical diversity selection demonstrate greater persona
consistency and diversity.

##  2 METHODOLOGY

###  2.1 Task Definition

In this paper, we will tackle the task of generating personalized dialogue
responses in the absence of any persona/personality information or
corresponding dialogue examples. Hence, during inference, the model is
expected to generate personalized dialogue given only the dialogue context,
which consists of all previous utterances in the conversation. In other words,
the model is expected to infer the agent’s persona from only the dialogue
context. For this task, the generated response is expected to be diverse as
well as persona consistent i.e., accurately reflect the agent’s persona.
However, in cases where no persona information can be inferred from the
dialogue context, the responses generated should aim to be persona neutral.
This means that, as far as possible, responses should not contain any personal
information which could potentially contradict the agent’s persona.

For our experiments, we utilize the ConvAI2 dialogue corpus. We chose the
ConvAI2 corpus as the textual persona descriptions, which consists of multiple
persona statements, can be used during training. Also, the available persona
descriptions allows us to evaluate the amount of persona information that is
accurately reflected in the generated response relatively easily.

###  2.2 Dual Latent Variable Generator

Figure 1: Flowchart depicting the architecture of the DLVGen model. The dashed
lines and solid lines represents the connections that occur only during
training and inference respectively. The dotted bidirectional connections
indicate the KL divergence computation. $\oplus$ represents the concatenate
operation.

Essentially, the proposed Dual Latent Variable Generator (DLVGen) framework
requires learning two distinct networks to model the latent distribution over
potential dialogue responses as well as the latent distribution over the
agent’s potential persona. During inference, latent variables are sampled from
both distributions and fed to the decoder, which consists of a GPT-2
pretrained language model (Radford and Wu,, 2019). Randomly sampling the
latent distribution over responses would improve response diversity due to the
stochasticity introduced by the latent variable. Similarly, randomly sampling
the latent distribution over the agent’s persona would incorporate a wide
range of potential persona information (inferred from the dialogue context) in
the generated response. The induced stochasticity would also reduce the
likelihood of the model incorporating the same persona information in multiple
responses, further contributing to the overall response diversity. In our
approach, the persona statements describing each interlocutor is only utilized
during training. An overview of the model is provided in Figure 1.

In our discussion, $x$ represents the dialogue context which consists of all
the prior utterances in the dialogue history. $p$ refers to all statements in
the provided textual persona description. $y$ denotes the dialogue response.
During inference, the dialogue response is generated based on the dialogue
context $x$, the latent variable sampled from the distribution over the
agent’s potential persona $z_{p}$, and the latent variable sampled from the
distribution over potential dialogue responses $z_{r}$. Since $z_{p}$ and
$z_{r}$ each represent different aspects of the generated response ($z_{p}$
encompasses the persona information and $z_{r}$ captures information regarding
the flow of the dialogue), we will regard $z_{p}$ and $z_{r}$ as independent
i.e., $z_{p}\perp\\!\\!\\!\perp z_{r}$. The persona information and dialogue
flow information, which are both derived from the dialogue context, are
disentangled via the prior and recognition networks introduced in the
remainder of this section. Hence, by assuming $z_{p}$ and $z_{r}$ are
independent random variables, the generation process can be expressed via the
following conditional distribution
$p(y,z_{p},z_{r}|x)=p(y|x,z_{p},z_{r})p(z_{p}|x)p(z_{r}|x)$. We also assume
that $z_{p}$ and $z_{r}$ can be modelled by multivariate isotropic Gaussian
distributions:

$p(z_{p}|x)=N(\mu_{p},\sigma_{p}^{2}\textbf{I})$ (1)
$p(z_{r}|x)=N(\mu_{r},\sigma_{r}^{2}\textbf{I})$ (2)

where $\mu_{p},\mu_{r}$ and $\sigma_{p}^{2},\sigma_{r}^{2}$ refer to the mean
and variance of the distribution over the potential persona and the
distribution over potential responses respectively. Hence, to approximate
$p(z_{p}|x)$ and $p(z_{r}|x)$, we define 2 prior networks
$p_{\theta_{p}}(z_{p}|x)$ and $p_{\theta_{r}}(z_{r}|x)$, which are single-
layer Multi-Layer Perceptrons (MLPs) parameterized by $\theta_{p}$ and
$\theta_{r}$. $p_{\theta_{p}}(z_{p}|x)$ and $p_{\theta_{r}}(z_{r}|x)$ are
termed the persona prior network and the response prior network respectively.
Before being fed to the prior networks, a GPT-2 pretrained model (Radford and
Wu,, 2019) is used to obtain $\bar{x}$, $\bar{y}$ and $\bar{p}$, the sequence
embeddings of the dialogue context $x$, response label $y$ and persona
descriptions $p$ respectively. Utilizing the persona and response prior
networks, the following means $\mu_{p},\mu_{r}$ and variances
$\sigma_{p}^{2},\sigma_{r}^{2}$ are derived as follows:

$\begin{bmatrix}\mu_{p}\\\
log(\sigma_{p}^{2})\end{bmatrix}=W_{p}\begin{bmatrix}\bar{x}\end{bmatrix}+b_{p}$
(3) $\begin{bmatrix}\mu_{r}\\\
log(\sigma_{r}^{2})\end{bmatrix}=W_{r}\begin{bmatrix}\bar{x}\end{bmatrix}+b_{r}$
(4)

where $W_{p},W_{r}$ refer to the weight vectors and $b_{p},b_{r}$ refer to the
bias vectors corresponding to the persona and response prior networks.

Additionally, to approximate the posterior of $p(z_{p}|x)$ and $p(z_{r}|x)$,
we introduce latent variables $z^{\prime}_{p}$ and $z^{\prime}_{r}$ as well as
latent distributions $q(z^{\prime}_{p}|x,p)$ and $q(z^{\prime}_{r}|x,y)$,
which are generated by 2 recognition networks
$q_{\phi_{p}}(z_{p}^{\prime}|x,p)$ and $q_{\phi_{r}}(z_{r}^{\prime}|x,y)$
(defined as single-layer MLPs parametrized by $\phi_{p}$ and $\phi_{r}$)
respectively. $q_{\phi_{p}}(z^{\prime}_{p}|x,p)$ is termed the persona
recognition network and $q_{\phi_{r}}(z^{\prime}_{r}|x,y)$ is termed the
response recognition network. Similarly, the sequence embeddings $\bar{x}$,
$\bar{y}$ and $\bar{p}$ are fed to the recognition networks. The persona and
response recognition networks define the approximate posterior
$q(z^{\prime}_{p}|x,p)$ and $q(z^{\prime}_{r}|x,y)$ via generating the
respective means $\mu^{\prime}_{p},\mu^{\prime}_{r}$ and variances
$\sigma^{\prime 2}_{p},\sigma^{\prime 2}_{r}$:

$\begin{bmatrix}\mu^{\prime}_{p}\\\ log(\sigma^{\prime
2}_{p})\end{bmatrix}=W^{\prime}_{p}\begin{bmatrix}\bar{x}\\\
\bar{p}\end{bmatrix}+b^{\prime}_{p}$ (5) $\begin{bmatrix}\mu^{\prime}_{r}\\\
log(\sigma^{\prime
2}_{r})\end{bmatrix}=W^{\prime}_{r}\begin{bmatrix}\bar{x}\\\
\bar{y}\end{bmatrix}+b^{\prime}_{r}$ (6)

where $W_{p}^{\prime},W_{r}^{\prime}$ refer to the weight vectors and
$b_{p}^{\prime},b_{r}^{\prime}$ refer to the bias vectors corresponding to the
persona and response recognition networks. The KL divergence between the prior
and recognition networks will be included in the final loss function. It
should be noted that for the persona prior network and the persona recognition
network, before being fed into the GPT-2 pretrained model to obtain the
embedding, the utterances corresponding to the opposing interlocutor (the
user) are masked. This prevents the generated Gaussian distribution from
modelling the persona of the opposing interlocutor.

To ensure end-to-end differentiability, latent variables are sampled using the
reparameterization trick (Kingma and Welling,, 2014). During training, the
latent variables $z^{\prime}_{p}$ and $z^{\prime}_{r}$ are sampled from the
persona and response recognition networks. However, during inference, the
latent variables $z_{p}$ and $z_{r}$ are sampled from the persona and response
prior networks. After sampling, the latent variables $z^{\prime}_{p}$ and
$z^{\prime}_{r}$ (training) or $z_{p}$ and $z_{r}$ (inference) are
concatenated and fed to a linear layer ($W_{LV}$) before being passed to the
decoder, which constitutes a GPT-2 pretrained language model. The resultant
representation is added to the GPT-2 input, which consists of the token
embedding and the positional encoding, at every decoding step. The persona
description $p$ is only used during training to learn the latent distribution
over the the agent’s potential persona. A simple figure depicting this process
in provided in Figure 2.

Since maximizing the conditional log likelihood during training requires an
intractable marginalization over $z_{p}$ and $z_{r}$, DLVGen is trained via
the Stochastic Gradient Variational Bayes (SGVB) framework (Kingma and
Welling,, 2014) by maximizing the variational lower bound. Also, the Bag-of-
Words (BoW) loss (Zhao et al.,, 2017) is incorporated into the loss function
to address the vanishing latent variable problem. Essentially, SGVB involves
maximizing the variational lower bound of $p(y|x)$:

$\begin{split}&L(\theta_{p},\theta_{r},\phi_{p},\phi_{r},x,y,p,z_{p},z_{r})=\\\
&E_{q_{\phi_{p}}(z_{p}^{\prime}|x,p);q_{\phi_{r}}(z_{r}^{\prime}|x,y)}[log\>p(y|x,z_{p},z_{r})]\\\
&-KL(q_{\phi_{p}}(z^{\prime}_{p}|x,p)||p_{\theta_{p}}(z_{p}|x))\\\
&-KL(q_{\phi_{r}}(z^{\prime}_{r}|x,y)||p_{\theta_{r}}(z_{r}|x))\\\
&+E_{q_{\phi_{p}}(z_{p}^{\prime}|x,p);q_{\phi_{r}}(z_{r}^{\prime}|x,y)}[log\>p(y_{bow}|x,z_{p},z_{r})]\end{split}$
(7)

where $KL(\cdot)$ refers to the KL divergence and $y_{bow}$ represents the
response bag-of-words.

Figure 2: Diagram depicting the GPT-2 decoder during inference. After the
latent variables are concatenated and fed into a linear layer $W_{LV}$, the
resultant combined latent variable is added to the positional encoding and
token embedding at every decoding step. $x_{0}$ and $x_{t}$ represent the
first and last token embedding of the dialogue context. $y_{0}$ and $y_{t}$
represent the first and last token of the generated response.

###  2.3 Variance Regularization

Additionally, we introduce a novel regularization technique based on the
variances of distributions generated by the prior networks i.e., $p(z_{p}|x)$
and $p(z_{r}|x)$. Essentially, we either maximize the variance of the latent
distribution over the agent’s potential persona $\sigma_{p}^{2}$, or minimize
the variance of the latent distribution over potential responses
$\sigma_{r}^{2}$. Specifically, this is realized by either adding the
euclidean norm of the log-variance of $p(z_{r}|x)$, $log(\sigma_{r}^{2})$, or
the inverse of the euclidean norm of the log-variance of $p(z_{p}|x)$,
$log(\sigma_{p}^{2})$. The inverse of the euclidean norm of the log-variance
can also be interpreted as the precision of the distribution. The variance
regularization terms can be expressed as:

$R_{r}=\lambda_{r}\>\lVert log(\sigma_{r}^{2})\lVert$ (8)
$R_{p}=\lambda_{p}\lVert\frac{1}{log(\sigma_{p}^{2})}\lVert$ (9)

where $R_{r}$ refers to the loss term which minimizes the variance of the
distribution over potential responses, and $R_{p}$ refers to the loss term
which maximizes the variance of the distribution over the agent’s potential
persona. $\lambda_{r}$ and $\lambda_{p}$ are penalty terms to be tuned. Then,
the 2 variance regularization terms can be individually added to the loss
function. Alternatively, both terms can be added to the loss function to form
the final loss function $\bar{L}$:

$\begin{split}\bar{L}&(\theta_{p},\theta_{r},\phi_{p},\phi_{r},x,y,p,z_{p},z_{r})=\\\
&L(\theta_{p},\theta_{r},\phi_{p},\phi_{r},x,y,p,z_{p},z_{r})\\\
&+R_{r}(\sigma_{r}^{2})+R_{p}(\sigma_{p}^{2})\end{split}$ (10)

By maximizing the variance of the latent distribution over the agent’s
potential persona, we hope to encourage the persona prior network to model a
wider range of potential persona information, which would further improve
response diversity. On the other hand, by minimizing the variance of the
latent distribution over potential responses, we aim to constrain the
variability of the sampled random variables, improving the contextual
coherence of the generated responses.

###  2.4 Lexical Diversity Selection

Lexical diversity is a measure of how many different lexical words appear in a
given text. Unlike filler words such as ‘and’ or ‘the’, lexical words are
words that convey information or meaning (‘cat’,‘computer’,‘sky’ etc.) . By
sampling the Gaussian distributions $N$ times, $N$ different responses can be
generated. Hence, we introduce a method of selecting the best response from
the pool of $N$ generated responses. Intuitively, we assume that lexically
diverse sentences make for more informative responses. Hence, to quantify the
lexical diversity of the generated responses, we utilize the Measure of
Textual Lexical Diversity (MTLD) and the Moving Average Type Token Ratio
(MATTR) scores (Fergadiotis et al.,, 2015). Obtaining the MTLD involves
computing the Token-Type Ratio (TTR) for sequentially larger segments of the
sentence. For each segment, when the TTR drops below a predefined threshold
$h$, a count is incremented by 1. Then, the total number of tokens in the
generated response is divided by the final count. The final MTLD is obtained
by averaging the forward and backward MTLD scores. The MATTR, on the other
hand is based on averaging the TTR of successive segments of the generated
response with a fixed window size $w$. After the computing the MATTR and MTLD,
we select the response with the highest combined score. This is summarized in
the following expression:

$\underset{r\in R}{\operatorname{argmax}}(0.1M_{1}^{h}(r)+M_{2}^{w}(r))$ (11)

where $M_{1}^{h}$ and $M_{2}^{w}$ refer to the functions for calculating the
MTLD score (with threshold $h$) and MATTR (with window size $w$) score
respectively. $R$ refers to the pool of generated responses. The MTLD score
was penalized by a factor of 0.1 to prevent it from overwhelming the MATTR
score. The proposed lexical diversity selection method is relatively
straightforward to implement and can be applied on top of any latent variable
dialogue model.

##  3 EXPERIMENT

###  3.1 Corpus

We evaluate our approach on the ConvAI2 personalized dialogue corpus (Dinan et
al.,, 2019). Since the ConvAI2 corpus is based on the PersonaChat corpus
(Zhang et al.,, 2018), both corpora are structurally similar. Both corpora
provide personalized dialogues as well as persona descriptions in the form of
several sentences. However, the ConvAI2 corpus features numerous crowd-sourced
rewrites and rephrases, which increases the overall task difficulty. In
ConvAI2, the training set $\textbf{D}_{train}$ contains 17,878 dialogues from
1155 unique personas, while the test set (validation set is used as the test
set in our experiments as actual test set is hidden) $\textbf{D}_{test}$
contains 1000 dialogues from 100 unique personas.

###  3.2 Implementation

For our experiments, we utilize the PyTorch, ParlAI, and HuggingFace
libraries. During training, the Adam optimizer (learning rate = 0.0001) is
used with a batch size of 16. The size of latent variables $z_{p}$, $z_{r}$,
$z^{\prime}_{p}$, $z^{\prime}_{r}$ are fixed at 256. We utilize the GPT-2
pretrained language model (Radford and Wu,, 2019) from HuggingFace to obtain
the context, persona and response embeddings. The decoder also consists of a
GPT-2 model (12 layers, 768 dimensional hidden state, 12 heads, 117 million
parameters).

For variance regularization, the values of $\lambda_{r}$ and $\lambda_{p}$ was
set to 0.5 and 1.0 respectively. For the MTLD and MATTR computation during
lexical diversity selection, $h$ and $w$ were fixed at 0.72 and 4
respectively. Responses are generated via beam search with a beam size of 3.
$N$ is also set to 3.

###  3.3 Baselines

To evaluate our proposed model on the task described in section 2.1, we
implement the following baselines:

  1. 1.

PAML Following Madotto et al., (2019), we pretrain a standard transformer with
PAML. However, using the ConvAI2 corpus, the model was applied to the task
defined in section 2.1 instead (PAML was originally evaluated on the
PersonaChat corpus), which involved directly evaluating the pretrained
transformer without further finetuning.

  2. 2.

MTML Following Lee et al., (2021), we pretrain a standard transformer with
MTML ($\alpha=0.8$). Similarly, using the ConvAI2 corpus, the model was
applied to the task defined in section 2.1 (MTML was also originally evaluated
on the PersonaChat corpus), which involved directly evaluating the pretrained
transformer without further finetuning.

  3. 3.

GPT-2 Similar to TransferTransfo (Wolf et al.,, 2019). However, instead of
GPT, GPT-2 was finetuned for dialogue generation. Since the model was applied
to the task defined in section 2.1, the decoder input consists of only the
dialogue context. The persona description was not utilized.

  4. 4.

CVAE Similar to the CVAE-based dialogue model proposed by Zhao et al., (2017),
which involves only generating and sampling from the latent distribution over
responses. However, our implementation utilizes the GPT-2 pretrained model
instead of the bidirectional GRU during decoding as well as to generate the
sequence embeddings. Additionally, since the model was applied to the task
defined in section 2.1, the persona description was not used. Instead, the
prior network generates the latent Gaussian distribution based on only the
dialogue context, and the decoder input consists of only the sampled latent
variable and the dialogue context.

  5. 5.

DLVGen A model which consists of our proposed DLVGen architecture with a GPT-2
decoder. We also implement additional variations of this model which include
each of the variance regularization terms $R_{r}$ and $R_{p}$, as well as the
combination of both terms. Additionally, we implement the lexical diversity
selection (LS) for all variants.

###  3.4 Evaluation

In our experiments, $N$ was set to 3 i.e., the Gaussian distributions were
sampled 3 times to generate 3 responses. For the latent variable models (CVAE
and DLVGen), the final automatic and human evaluation scores are obtained by
averaging the individual scores from the $N$ responses generated from each
dialogue example in $\textbf{D}_{test}$.

####  3.4.1 Automatic Metrics

Table 1:  Results on the ConvAI2 dialogue corpus. $R_{r}$, $R_{p}$ and $LS$ refer to the variance regularization terms presented in Equation 5 and 6, and the lexical diversity selection respectively. | C-score | Distinct-1 | Distinct-2 |  | Persona
---
Consistency
Naturalness | Engagingness
PAML | -0.029 | 0.089 | 0.182 | 0.041 | 0.141 | 0.064
MTML | -0.055 | 0.118 | 0.268 | 0.057 | 0.116 | 0.083
GPT-2 | -0.011 | 0.097 | 0.177 | -0.079 | 0.378 | 0.114
CVAE | -0.046 | 0.152 | 0.422 | 0.012 | 0.227 | 0.091
DLVGen | 0.048 | 0.151 | 0.426 | 0.105 | 0.276 | 0.128
+LS | 0.049 | 0.367 | 0.757 | 0.123 | 0.356 | 0.117
+$R_{r}$ | 0.075 | 0.171 | 0.474 | 0.226 | 0.325 | 0.104
+$R_{r}$+LS | 0.081 | 0.390 | 0.809 | 0.269 | 0.367 | 0.151
+$R_{p}$ | 0.024 | 0.161 | 0.448 | 0.093 | 0.181 | 0.134
+$R_{p}$+LS | 0.017 | 0.370 | 0.775 | 0.142 | 0.262 | 0.148
+$R_{r}$+$R_{p}$ | 0.054 | 0.171 | 0.474 | 0.173 | 0.279 | 0.174
+$R_{r}$+$R_{p}$+LS | 0.062 | 0.386 | 0.801 | 0.214 | 0.369 | 0.145

We evaluate the generated responses with the following automatic metrics:

  1. 1.

Distinct 1 & 2 Distinct-1 and 2 scores quantify the diversity of a generated
response. The Distinct-1 and Distinct-2 scores accounts for the number of
distinct 1-grams and 2-grams in the generated response respectively. A higher
Distinct-1 or 2 score would indicate greater response diversity.

  2. 2.

C-score The C-score (Madotto et al.,, 2019) reveals the extent to which the
persona is accurately reflected in the generated response. Essentially, it is
generated via a BERT model, which is finetuned to indicate if the generated
response entails, contradicts or is independent to each statement in the
persona description. A higher C-score would indicate a larger amount of
accurately incorporated persona information.

####  3.4.2 Human Evaluation

We engaged 5 graduates to evaluate the various generated responses based on 3
criteria which are similar to the criteria used during human evaluation in
prior work(Lee et al.,, 2021; Madotto et al.,, 2019):

  1. 1.

Persona Consistency The individuals were told to rate the responses in terms
of the amount of persona information present in the generated response on a
scale from -1 to 1. A score of -1 would indicate a contradiction with the
corresponding persona description and a score of 1 would indicate an accurate
incorporation of persona information.

  2. 2.

Naturalness The individuals were told to rate the generated responses
according to human-likeness on a scale from -1 to 1. This criteria accounts
for both the diversity and general fluency of the generated response. A score
of 1 would be assigned to a perfectly fluent response that is indiscernible
from a human response. A score of -1 would indicate an awkwardly phrased
response with multiple grammatical and lexical errors.

  3. 3.

Engagingness Individuals were told to rate the generated responses in terms of
how engaging the generated response was, or how well the response attempts to
continue the conversation, on a scale from -1 to 1. A score of 1 would
indicate a response which follows the logical flow of the dialogue and aims to
continue the conversation. A score of -1 would be assigned to a laconic
response that is contextually incoherent.

##  4 RESULTS & DISCUSSION

Table 2:  Dialogue responses generated by DLVGen+$R_{r}$+LS. $Ref$ indicates
the response label. Since $N=3$, $r_{1},r_{2},r_{3}$ refer to the 3 responses
generated after sampling the latent distributions 3 times. The response
selected by the lexical diversity selector is in bold. Persona
---
i have been trying all types of food everywhere i go.
hey there i’m 23 and i love food.
i’ve been traveling the world for a years.
i also like to cook but i’m not very good at it.
i own a yacht and i rent it out when i’m not using it.
Dialogue Context
User: | hello. how are you today ?
Agent: | good today . just cooking some mexican food. i cooking but am not very good.
User: | oh i like mexican food , but my favorite food are cheeseburgers.
Agent: | i inherited some money and bought a yacht to travel, i try different foods traveling
User: | i help out at a soup kitchen since i grew up poor
Responses
$Ref:$ | cheeseburgers are great , i try all kinds of foods everywhere i go , gotta love food.
$r_{1}:$ | i love to cook , i am a chef.
$r_{2}:$ | i love to cook.
$r_{3}:$ | i love to travel , i have been to many countries.

###  4.1 Results

The automatic and human evaluation results attained by the models described in
section 3.4 are displayed in Table 1. An example of dialogue responses
generated by the best performing DLVGen variant (DLVGen+$R_{r}$+LS) is
provided in Table 2.

###  4.2 Discussion

Based on the C-scores and Persona Consistency scores attained, it can be
easily observed that our proposed model incorporates the corresponding persona
information to a larger extent compared to PAML, MTML, GPT-2 and CVAE. All
DLVGen variants achieved higher C-scores and Persona Consistency scores.
However, while the addition of the variance regularization term $R_{r}$
generally improves the C-score/Persona Consistency score, the introduction of
$R_{p}$ leads to a slight drop in the same metrics. We suspect that this is
because a large variance would increase the chances of modelling incorrect or
contradictory persona traits, which would negatively impact the persona
consistency of the responses. DLVGen variant that achieved the highest C-score
and Persona Consistency score was DLVGen+$R_{r}$+LS. Also, it can be observed
that the DLVGen model variants with lexical diversity selection achieved
better C-scores/ Persona Consistency scores. This indicates that lexically
diverse responses tend to contain more persona information.

It should be noted that the C-scores and Persona Consistency scores obtained
are dependant on the length of the dialogue context. A longer dialogue context
(greater number of utterances) would usually imply greater persona consistency
and larger C-score. This is expected as a short dialogue context would not
have sufficient persona information to be inferred by the persona prior
network. For these cases, we observe that there is a tendency for the model to
incorporate a randomly generated persona, which could be due to the failure of
the persona prior network to model any persona information from the context.
As a result, the generated response might contradict the agent’s persona and
negatively impact the C-score as well as Persona Consistency score. Ideally,
in such cases, the model should be able to recognize that there is
insufficient persona information in the dialogue context, and the generated
response should be largely persona neutral (section 2.1) while remaining
contextual and fluent. Addressing this issue could be a direction for future
research. A potential approach could involve estimating the uncertainty, which
could be represented by the variance, of the persona prior network, and
incorporating the uncertainty estimate in the decoding process.

Upon closer inspection of the responses generated by each of model, we notice
that out of the responses generated by GPT-2 and CVAE, few attempt to
incorporate the persona information. Instead, a significant fraction of
responses were relatively short and generic. On the other hand, responses
generated by DLVGen are more likely to a attempt some form of persona
incorporation. It should be highlighted that even though it is less likely for
generic responses to achieve high C-scores, it is also less likely for them to
be assigned negative C-scores. A potential avenue for future work could
involve designing a metric which penalizes generic responses in addition to
inaccurate responses.

From our experimental results, the GPT-2 base model attained noticeably lower
Distinct 1 and 2 scores compared to the latent variable models. This follows
previous work which reported an increase in response diversity due to the
stochasticity introduced by the latent variable (Zhao et al.,, 2017; Song et
al.,, 2019). Furthermore, the DLVGen model variants with lexical diversity
selection achieved better Distinct-1 and 2 scores compared to all other
models. This is not surprising since lexical diversity would generally imply
general diversity. The improvements on both diversity and persona consistency
demonstrate the efficacy of our lexical diversity selection approach.

When it comes to the Naturalness, we can observe that the usage of the GPT-2
pretrained language model resulted in higher scores. Both PAML and MTML, which
utilize standard Transformers, attained noticeably lower Naturalness scores
compared to all other models. This is to be expected as pretrained language
models such as GPT-2 have demonstrated high language understanding and
generation capability even without any finetuning. We can also observe that
most of the latent variable models attained comparable Naturalness scores. The
only exception is the inclusion of the variance regularization term $R_{p}$,
which results in a lower Naturalness score. The GPT-2 base model, however,
maintains an edge over the latent variable models in terms on Naturalness
score. This could be attributed to the short, generic responses generated by
the base model, which are relatively more fluent and human-like. On the other
hand, when it comes to the Engagingness score, all implemented models achieved
relatively poor performance. Relatively few responses attempt to continue the
conversation. Instead, most responses are largely informative.

##  5 RELATED WORK

###  5.1 Latent Variable Models

Latent variable models are a category of models that involve inferring latent
random variables from a group of observable variables (Verbeke and
Molenberghs,, 2017). Latent variable models such as the Variational Auto
Encoder (VAE) (Kingma and Welling,, 2014) and the Conditional Variational Auto
Encoder (CVAE) (Sohn et al.,, 2015) have been applied to the task of open-
domain dialogue generation, where the potential dialogue responses are
modelled as a latent Gaussian distribution (Li et al.,, 2020; Shen et al.,,
2018; Zhao et al.,, 2017; Serban et al.,, 2017). In addition to personalized
dialogue generation (examples provided in the introduction), CVAEs have been
applied to conditional dialogue generation tasks such as emotional dialogue
generation (Liu et al.,, 2021; Asghar et al.,, 2020; Zhou and Wang,, 2018) as
well as topical dialogue generation (Wang et al.,, 2020).

###  5.2 Personalized Dialogue

Over the past few years, there has been numerous publications exploring
various approaches to the task of personalized dialogue generation. As
mentioned in the introduction, there are numerous approaches to the task of
personalized dialogue generation. A popular approach typically involves
conditioning dialogue responses on the dialogue context in addition to textual
persona descriptions (Lee et al.,, 2021; Na et al.,, 2021; Liu et al.,, 2020;
Majumder et al.,, 2020; Madotto et al.,, 2019; Wolf et al.,, 2019; Zheng et
al.,, 2020; Chan et al.,, 2019; Song et al.,, 2019). This approach focuses on
generating responses which incorporate the provided persona information.
Another approach involves incorporating personality or persona related
metadata into the decoding process (Qian et al.,, 2018). Some approaches also
involve implicitly learning personality user embeddings (Zheng et al., 2019a,
; Wu et al.,, 2020; Al-Rfou et al.,, 2016; Li et al.,, 2016). Another approach
entails inferring the dialogue agent’s personality directly from the dialogue
history (Zheng et al., 2019b, ; Su et al.,, 2019). Typically, for this
approach, the primary aim is to train the agent to mimic the dialogue style of
the interlocutor.

##  6 CONCLUSION

In this paper, we introduced DLVGen, a dual latent variable model which models
the potential responses and the agent’s potential persona as latent Gaussian
distributions. Through our experiments, we find that responses generated by
DLVGen effectively incorporates persona information inferred from the dialogue
context. We also introduced a variance regularization technique and lexical
diversity selection method which improves the quality of the generated
responses in terms of both persona consistency and human-likeness. However, an
area for improvement is the relatively poor engagingness of the dialogue.
Encouraging the generation of persona consistent, diverse yet engaging open-
domain dialogue is a potential avenue for future research. A possible approach
involves designing an objetcive function which explicitly accounts for the
engagingness of the generated response. The dialogue model could then be
trained on both objective functions via a multi-task learning framework.

## REFERENCES

  * Al-Rfou et al., (2016) Al-Rfou, R., Pickett, M., Snaider, J., Sung, Y., Strope, B., and Kurzweil, R. (2016).  Conversational contextual cues: The case of personalization and history for response ranking.  CoRR, abs/1606.00372.
  * Asghar et al., (2020) Asghar, N., Kobyzev, I., Hoey, J., Poupart, P., and Sheikh, M. B. (2020).  Generating emotionally aligned responses in dialogues using affect control theory.
  * Chan et al., (2019) Chan, Z., Li, J., Yang, X., Chen, X., Hu, W., Zhao, D., and Yan, R. (2019).  Modeling personalization in continuous space for response generation via augmented Wasserstein autoencoders.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1931–1940, Hong Kong, China. Association for Computational Linguistics.
  * Dinan et al., (2019) Dinan, E., Logacheva, V., Malykh, V., Miller, A., Shuster, K., Urbanek, J., Kiela, D., Szlam, A., Serban, I., Lowe, R., Prabhumoye, S., Black, A. W., Rudnicky, A., Williams, J., Pineau, J., Burtsev, M., and Weston, J. (2019).  The second conversational intelligence challenge (convai2).
  * Fergadiotis et al., (2015) Fergadiotis, G., Wright, H. H., and Green, S. B. (2015).  Psychometric evaluation of lexical diversity indices: Assessing length effects.  Journal of Speech, Language, and Hearing Research, 58(3):840–852.
  * Kingma and Welling, (2014) Kingma, D. P. and Welling, M. (2014).  Auto-encoding variational bayes.
  * Lee et al., (2021) Lee, J. Y., Lee, K. A., and Gan, W. S. (2021).  Generating personalized dialogue via multi-task meta-learning.  In Proceedings of the 25th Workshop on the Semantics and Pragmatics of Dialogue - Full Papers, Potsdam, Germany. SEMDIAL.
  * Li et al., (2020) Li, C., Gao, X., Li, Y., Peng, B., Li, X., Zhang, Y., and Gao, J. (2020).  Optimus: Organizing sentences via pre-trained modeling of a latent space.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4678–4699, Online. Association for Computational Linguistics.
  * Li et al., (2016) Li, J., Galley, M., Brockett, C., Spithourakis, G., Gao, J., and Dolan, B. (2016).  A persona-based neural conversation model.  In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 994–1003, Berlin, Germany. Association for Computational Linguistics.
  * Liu et al., (2021) Liu, M., Bao, X., Liu, J., Zhao, P., and Shen, Y. (2021).  Generating emotional response by conditional variational auto-encoder in open-domain dialogue system.  Neurocomputing, 460:106–116.
  * Liu et al., (2020) Liu, Q., Chen, Y., Chen, B., Lou, J.-G., Chen, Z., Zhou, B., and Zhang, D. (2020).  You impress me: Dialogue generation via mutual persona perception.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1417–1427, Online. Association for Computational Linguistics.
  * Madotto et al., (2019) Madotto, A., Lin, Z., Wu, C.-S., and Fung, P. (2019).  Personalizing dialogue agents via meta-learning.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5454–5459, Florence, Italy. Association for Computational Linguistics.
  * Majumder et al., (2020) Majumder, B. P., Jhamtani, H., Berg-Kirkpatrick, T., and McAuley, J. (2020).  Like hiking? you probably enjoy nature: Persona-grounded dialog with commonsense expansions.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9194–9206, Online. Association for Computational Linguistics.
  * Na et al., (2021) Na, Y., Park, J., and Sohn, K.-A. (2021).  Is your chatbot perplexing?: Confident personalized conversational agent for consistent chit-chat dialogue.  In Proceedings of the 13th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART,, pages 1226–1232. INSTICC, SciTePress.
  * Qian et al., (2018) Qian, Q., Huang, M., Zhao, H., Xu, J., and Zhu, X. (2018).  Assigning personality/profile to a chatting machine for coherent conversation generation.  In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18, pages 4279–4285. International Joint Conferences on Artificial Intelligence Organization.
  * Radford and Wu, (2019) Radford, A. and Wu, J. (2019).  Rewon child, david luan, dario amodei, and ilya sutskever. 2019.  Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9.
  * Serban et al., (2017) Serban, I. V., Sordoni, A., Lowe, R., Charlin, L., Pineau, J., Courville, A., and Bengio, Y. (2017).  A hierarchical latent variable encoder-decoder model for generating dialogues.  In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI’17, page 3295–3301. AAAI Press.
  * Shen et al., (2018) Shen, X., Su, H., Niu, S., and Demberg, V. (2018).  Improving variational encoder-decoders in dialogue generation.  In AAAI.
  * Sohn et al., (2015) Sohn, K., Lee, H., and Yan, X. (2015).  Learning structured output representation using deep conditional generative models.  In Cortes, C., Lawrence, N., Lee, D., Sugiyama, M., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc.
  * Song et al., (2019) Song, H., Zhang, W., Cui, Y., Wang, D., and Liu, T. (2019).  Exploiting persona information for diverse generation of conversational responses.  In IJCAI.
  * Su et al., (2019) Su, F.-G., Hsu, A. R., Tuan, Y.-L., and Lee, H.-Y. (2019).  Personalized Dialogue Response Generation Learned from Monologues.  In Proc. Interspeech 2019, pages 4160–4164.
  * Verbeke and Molenberghs, (2017) Verbeke, G. and Molenberghs, G. (2017).  Modeling through latent variables.  Annual Review of Statistics and Its Application, 4(1):267–282.
  * Wang et al., (2020) Wang, Y., Si, P., Lei, Z., and Yang, Y. (2020).  Topic enhanced controllable cvae for dialogue generation (student abstract).  Proceedings of the AAAI Conference on Artificial Intelligence, 34:13955–13956.
  * Wolf et al., (2019) Wolf, T., Sanh, V., Chaumond, J., and Delangue, C. (2019).  Transfertransfo: A transfer learning approach for neural network based conversational agents.  CoRR, abs/1901.08149.
  * Wu et al., (2020) Wu, B., Li, M., Wang, Z., Chen, Y., Wong, D. F., Feng, Q., Huang, J., and Wang, B. (2020).  Guiding variational response generator to exploit persona.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 53–65, Online. Association for Computational Linguistics.
  * Zhang et al., (2018) Zhang, S., Dinan, E., Urbanek, J., Szlam, A., Kiela, D., and Weston, J. (2018).  Personalizing dialogue agents: I have a dog, do you have pets too?  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2204–2213, Melbourne, Australia. Association for Computational Linguistics.
  * Zhao et al., (2017) Zhao, T., Zhao, R., and Eskenazi, M. (2017).  Learning discourse-level diversity for neural dialog models using conditional variational autoencoders.  In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 654–664, Vancouver, Canada. Association for Computational Linguistics.
  * (28) Zheng, Y., Chen, G., Huang, M., Liu, S., and Zhu, X. (2019a).  Personalized dialogue generation with diversified traits.  CoRR, abs/1901.09672.
  * Zheng et al., (2020) Zheng, Y., Zhang, R., Huang, M., and Mao, X. (2020).  A pre-training based personalized dialogue generation model with persona-sparse data.  Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):9693–9700.
  * (30) Zheng, Y., Zhang, R., Mao, X., and Huang, M. (2019b).  A pre-training based personalized dialogue generation model with persona-sparse data.  CoRR, abs/1911.04700.
  * Zhou and Wang, (2018) Zhou, X. and Wang, W. Y. (2018).  Mojitalk: Generating emotional responses at scale.  ArXiv, abs/1711.04090.


# Identification of Stone Deterioration Patterns with Large Multimodal Models.

Daniele Corradetti${}^{a)\,b)}$, José Delgado Rodriguesc)

ABSTRACT - The conservation of stone-based cultural heritage sites is a
critical concern for preserving cultural and historical landmarks. With the
advent of Large Multimodal Models, as GPT-4omni (OpenAI), Claude 3 Opus
(Anthropic) and Gemini 1.5 Pro (Google), it is becoming increasingly important
to define the operational capabilities of these models. In this work, we
systematically evaluate the abilities of the main foundational multimodal
models to recognise and classify anomalies and deterioration patterns of the
stone elements that are useful in the practice of conservation and restoration
of world heritage. After defining a taxonomy of the main stone deterioration
patterns and anomalies, we asked the foundational models to identify a curated
selection of 354 highly representative images of stone-built heritage,
offering them a careful selection of labels to choose from. The result, which
varies depending on the type of pattern, allowed us to identify the strengths
and weaknesses of these models in the field of heritage conservation and
restoration.

##  I Introduction

The preservation of cultural heritage sites is a critical challenge in the
fields of archaeology, historic preservation, and conservation science. Stone
monuments, buildings, and artefacts are subject to a wide range of
deterioration processes over time, including weathering, erosion, biological
growth, salt crystallisation, and human-induced damage (Doehne & Price, 2010).
Identifying and classifying these deterioration patterns is essential for
developing effective conservation strategies and interventions.

In recent years, advances in artificial intelligence and machine learning have
opened up new possibilities for automated analysis of stone-built heritage. In
particular, the development of large multimodal models (LMM) - i.e., AI
systems that can process and generate multiple types of data, including text,
images, and audio (Yin et al., 2023) - has the potential to revolutionise the
way we study and preserve the world’s stone-built heritage. Indeed, it is
important to immediately note the difference between these models and those
resulting from specific neural networks dedicated to identifying specific
deterioration patterns as they were usually developed prior to 2022. Indeed,
these neural networks were classical examples of narrow artificial
intelligence (Morris et al., 2023) visual recognition systems created ad hoc
for specific stone deterioration patterns through transfer learning from
neural networks structured for specific segmentation and classification
problems (most existing studies have focused on narrow, specialised tasks like
crack detection or material classification, often using small datasets and
custom-built algorithms cfr. Oses et al, 2014; Câmara et al., 2023; Mishra &
Lourenço, 2024). On the other hand, the current LMMs, such as OpenAI’s
GPT-4omni, Anthropic’s Claude 3 Opus, and Google’s Gemini 1.5 Pro, aim to
achieve a general model capable not only of recognising all specific
deterioration patterns with the same model but also of performing complex
diagnoses through linguistic operations or special cognitive architectures and
potentially suggesting specific interventions (e.g., Saab et al., 2024; Brown
et al., 2020).

Despite the extensively studied abilities of these LMMs, the specific skills
and limitations of models for heritage conservation applications have not been
systematically evaluated. The increasing integration of these models into
human operational workflows thus necessitates an extensive and systematic
evaluation of how state-of-the-art multimodal models perform on a diverse
range of stone deterioration patterns and anomalies, using large,
representative image sets.

In this work, we aim to address this gap by conducting a rigorous evaluation
of three leading multimodal models - GPT-4omni (OpenAI), Claude 3 Opus
(Anthropic), and Gemini 1.5 Pro (Google) - on the task of recognising and
classifying stone deterioration patterns relevant to world heritage
conservation. We define a taxonomy of stone deterioration patterns based on
the internationally recognised literature (ICOMOS-ISCS 2008), but adapted to
suit new AI interpretation tools. We then curate a dataset of 354 high-quality
images exemplifying these deterioration patterns on stone heritage sites
around the world. Using carefully designed prompts and label sets, we assess
each model’s performance in identifying the deterioration patterns present in
each image. Images and code for the test are available (see Section 5 Data
Availability) in order to allow replication of the test to other foundational
models or dedicated artificial cognitive entities (Fig. I.1).

Figure I.1: Implemented workflow followed in this study.

The results obtained are extremely informative, showing a significant
variation in accuracy depending on the pattern analysed. While some patterns
are identified with impressive accuracy, others seem invisible to LMMs or
indistinguishable from other patterns. A hypothesis formed in our work group
is that the lack of specific training has led large language models (LLMs ) to
adopt the common, non-technical meanings of the words used to identify these
patterns. The training of LMMs involves the semantic association between words
and images through a process of text, video, and audio encoding into the same
semantic space, then training often uses supervised learning and contrastive
learning strategies (Radford et al., 2021). However, the absence of a special
attention to the technical meanings that certain words have in the context of
conservation of cultural heritage leads to a confusion of terms and thus to a
generic and unspecific diagnosis. A structural problem is then the difficulty
of distinguishing certain specific patterns from mere photographs, for
instance when a tactile evaluation is required for their identification. In
our group, the opportunity to test current LMMs on such deterioration patterns
was discussed. The positive response was dictated by the need to have a
realistic overview of the possibility of using these tools profitably and
integrating them into the concrete activities of restoration and conservation
of heritage where the distinction of these patterns is still hoped for if not
required. Furthermore, we also want to stress out that the implemented test
constitutes just a first evaluation on foundational models, without
considering any finetuning or additional architectural solution around those
models. In fact, we think that the results were encouraging, suggesting that
specific training sessions on dedicated datasets could improve drastically the
results. Finally, it is worth noting that, given the rapid evolution of the
field, the aforementioned LMMs have been trained on a large portion of the
existing non-synthetic data. Given the scarcity of original data, subsequent
generations of LMMs are increasingly oriented towards training-based on
synthetic data generated by previous versions of LMMs (Choenni et al. 2023;
Gholami et al. 2023; Li et al., 2023). While this type of training can be very
effective in teaching general skills to the models, it is possible that a
cognitive weakness in the training of the actual generation of LMMs will
likely remain influencing the successive generations, if not properly
addressed.

##  II Evaluating the Large Multimodal Models

A Large Multimodal Model is an AI system capable of processing and
understanding multiple modalities of information, such as text, images, audio,
and video. These models aim to mimic human-like perception integrating the
complementary nature of different data types. Recent advancements in deep
learning, particularly transformer architectures, have enabled the development
of large-scale multimodal models that can perform a wide range of tasks across
various domains (Vaswani et al. 2017). Multimodal models learn joint
representations of different modalities (Devlin et al., 2018), allowing to
capture the complex relationships and dependencies between them. This enables
the models to perform cross-modal reasoning, generation, and retrieval tasks
(Radford et al., 2021). Even more so, in the last generation of multimodal
models, where all inputs and outputs are processed by the same neural network,
in comparison with the previous generation where the multimodality was
achieved by integrating different neural network each one trained on a
different modality (OpenAI, 2024). The versatility of LMMs has led to their
application in a wide range of domains, including healthcare, education,
entertainment, and conservation science. For our purposes, applications in
healthcare are inspirational, where LMMs have been used for medical image
analysis, disease diagnosis, and patient monitoring (Gemini Team, 2024). While
LMMs offer immense potential for automated analysis and preservation of stone
monuments, buildings, and artefacts, it is also clear that before considering
their application, a preliminary systematic evaluation across various
deterioration categories is necessary. In this work, we decided to evaluate
foundational multimodal models, i.e., without specific training and without
additional systems or architectures, in order to recognise the native
abilities of such models. The results of this study, therefore, represent a
starting point and an evaluation of the native knowledge of the various
models. Numerous benchmarkings were inspirational, but we mainly drew
inspiration from the GPQA model (Rein et al. 2023), for which the objective of
the test is to evaluate very advanced skills through the help of experts in
the field. Given the multimodal nature of the model, the choice of modality
and evaluation method was taken into consideration. After careful reflection
and preliminary tests, we considered that the most important and
discriminating ability to evaluate would consist in visually recognising stone
deterioration patterns (Antol et al., 2015). In fact, from our preliminary
analyses, we noticed that closed-form questions of a theoretical nature in the
form of audio or text have a very high percentage of correct answers and do
not guarantee a real applicability of the model to concrete cases. At the same
time, we estimated that visual evaluation could take place optimally through
the administration of photographic images representative of the various
patterns to the models.

###  II-A Preparation of the test

Deterioration patterns are the visible signs of the processes that continually
take place in any built object. By their nature, deterioration processes are
not visible and it is through the signs they impose or originate that
scientists and professionals interpret and resolve them. In this sense, the
correct observation and identification of existing patterns is the first and
most important action that conservation professionals must take care of.
Traditionally, this step is carried out through direct inspection of the
object, which involves exhaustive observation of exposed surfaces and detailed
documentation of all relevant data collected. Having automatic processing of
observational data and correct identification of deterioration patterns would
be a relevant improvement in the preparation of conservation interventions,
which could end up being a high cost/benefit option that would contribute to
making conservation activities more affordable and attractive. The objective
of this project is to obtain an AI tool capable of correctly identifying
deterioration patterns, graphically documenting their distribution throughout
the object, and calculating areas and estimating costs to carry out the
corresponding conservation intervention. Before embarking on designing a
dedicated AI tool to specifically address the project’s objectives, we decided
to systematically evaluate the performance of three of the last generation
LMMs on stone deterioration patterns recognition, which required a
comprehensive benchmarking study. The preparation of the test involved two key
steps: creating a taxonomy of stone deterioration patterns and selecting a
representative set of images for each pattern. In some preliminary trials,
LMMs were allowed to use an open taxonomy, which showed that patterns were
described in colloquial terms that varied from case to case, even when they
described the same pattern.

####  II-A1 Creating the Taxonomy

To avoid ambiguity in interpretations as much as possible and to focus the
description made by LMMs, a comprehensive taxonomy of stone deterioration
patterns relevant to world heritage conservation has been defined. It is
largely based on the ICOMOS-ISCS Glossary of deterioration patterns (ICOMOS
2008), with some adaptations to better meet the needs of this activity,
following the consensus gathered in a quick consultation with some
professionals in the area. After careful evaluation of the many terms
currently used to describe such patterns, the following list was selected:

> _Abrasion, adherent deposit, algae, alveolisation, biological colonisation,
> black crust, blistering, chipping, contour spalling, corrosion of inserted
> elements, crack, craquele, dark diffuse biocolonisation, deformation,
> degraded joint filling, detachment of mortar layer, differential erosion,
> discolouration, efflorescence, encrustation, erosion, film, flaking,
> fracture, fragmentation, gap, graffiti, granular disintegration, lichens,
> loose deposit, misalignment elements, moist area, moss, open joint, patina,
> perforation, pitting, plant, powdering, soiling, spalling, staining,
> sugaring, thin black deposit, unaesthetic joint filling, unaesthetic patch
> repair_.

Figure II.1: Example of the deterioration patterns used for benchmarking

Fig. II.1 illustrates some of the deterioration patterns used to test the
models in the present study. This list covers a wide range of stone
deterioration patterns, including the major categories of damage signs
commonly encountered in stone-built objects: patterns linked to visual
disturbances; patterns representing erosion or mass losses; patterns with
direct or indirect connection with structural stability issues. As discussed
later in this article, LMMs showed great difficulties in identifying some
chosen patterns, but no effort was made here to adapt or change the number and
type of patterns to see how such adaptations could impact the capabilities of
the tested models. However, such adaptations are not to be excluded in future
tests if deemed useful to benefit the models learning capacity and to improve
the produced outputs.

####  II-A2 Selecting the Images

The next step in preparing the test was to curate a dataset of high-quality
images representative of each stone deterioration pattern in our taxonomy. We
started with a large archive of over 8,000 images collected from various
sources, including field surveys and conservation reports. The images depict
stone heritage sites from around the world, spanning different historical
periods, architectural styles, and geological contexts. Although they cover a
wide variety of patterns, the images were not taken with this study in view
and are therefore certainly not the ideal set that one could hope to test. To
select the most suitable images for our testing, we established a set of
criteria. First, each image had to clearly contain the “Target Pattern” as a
major occurrence in its contents. Second, the images had to be of sufficient
resolution and quality to allow for detailed analysis by the LMMs. Third, we
aimed to include a diverse range of stone types, surface textures, and
environmental conditions to assess the models’ robustness and generalisation
capabilities. It is worth noting that while each image was selected with a
specific Target Pattern in mind, they always contain other patterns that the
models were asked to identify as well.

###  II-B Selection of the models

For this study, we selected three state-of-the-art foundational multimodal
models released in 2024: OpenAI GPT-4omni (OpenAI, 2024), Anthropic Claude 3
Opus (Anthropic, 2024), and Google Gemini 1.5 Pro (Google, 2024). These models
were chosen based on their very strong performance on all general benchmarks
(e.g., MMLU, GPQA, ARC-Challenge, etc.) and their ability to handle a wide
range of tasks across different domains. OpenAI GPT-4omni is an extension of
the GPT (Generative Pre-trained Transformer) architecture, which has shown
remarkable success in natural language processing tasks. GPT-4omni
incorporates visual processing capabilities, enabling it to understand and
generate both text and images. The model has been trained on a vast amount of
web-scale data, processes multiple data types simultaneously, thus improving
its ability to understand and generate diverse outputs. Having real-time
responses, being scalable and an improved efficiency, this model is naturally
one of the best candidates for deployment of real applications to cultural
heritage conservation tasks. Claude 3 Opus is Anthropic’s next generation
model offering enhanced capabilities in multimodal processing. Claude 3 Opus
builds upon the successes of the original Claude model, which demonstrated
strong performance on language understanding and generation tasks. Moreover,
Anthropic is known for its focus on ethical and robust AI systems. As
GTP4-omni, Claude 3 Opus processes various visual formats, making it
potentially suitable for detailed heritage site analysis. Google Gemini 1.5
Pro is the most advanced model of Google’s Gemini series of multimodal models.
It shows excellent results on a vast range of tasks, including image
captioning, visual question answering, and image-text retrieval. Being
released by Google, Gemini 1.5 Pro is probably the easiest model to scale and
the most cost-effective, which thus makes it one of the best candidates for
deploying robust solutions with possibly high-volume of inference requests.
Indeed, Gemini 1.5 Pro has a context window of 1 million token, which would
translate in a 40 minutes long video, the possible result of a drone
inspection of a site (Google, 2024).

###  II-C Implementation of the Test

To evaluate the selected multimodal models on the task of stone deterioration
patterns recognition, we implemented an automatised testing pipeline (Fig.
I.1) which included a simple preprocessing and a prompt engineering feature.
The first step in the testing pipeline was to preprocess the curated dataset
of images in a way compatible with the input size of the selected models. To
get accurate and concise responses from the models, we used a prompt that
allowed the model to provide as many patterns as necessary up to five, but
only using the list of stone patterns defined in our taxonomy. We then
utilized the official APIs provided by OpenAI, Anthropic, and Google to
interact with their respective multimodal models. The model’s response,
containing the identified patterns, was then received and processed. Finally,
the image filename, true pattern label, and the model’s identified patterns
were stored in a results list for the performance evaluation.

##  III Results

After the model response, the results were collected and then manually
validated. The first action was to confirm whether or not the “target pattern”
was one of the identified patterns. Then, the original image was opened to
check which of the identified patterns were actually present in the image. The
first relevant result is expressed in the rate of success the models showed in
identifying the “Target Pattern” (Table I and Fig. III.1). Indeed, it is
important for us to be sure that the most evident or relevant deterioration
pattern is systematically identified by the LMMs.

Figure III.1: Rate of success of each model on the “Target deterioration
pattern”. _Yellow_ is GPT-4omni, _light blue_ Gemini 1.5 Pro and _blue_ is
Claude 3 Opus Figure III.2: Success rate in the identification of the presence
of a deterioration pattern (as openly chosen by the models). In the image the
color _yellow_ represents the model GPT-4omni, the color _light blue_
represents Gemini 1.5 Pro and the _blue_ is for Claude 3 Opus.
_Model_ | _Targets correctly identified_ | _Total possible_ | _Success rate (%)_
---|---|---|---
GPT-4omni | 149 | 354 | 42.1%
Gemini 1.5 Pro | 138 | 354 | 39%
Claude 3 Opus | 86 | 354 | 24.3%

Table I: Success rates on the identification of the target deterioration
pattern
_Model_ | _Deterioration patterns correctly identified_ | _Total identification_ | _Success rate (%)_
---|---|---|---
GPT-4omni | 677 | 1031 | 65.6%
Gemini 1.5 Pro | 741 | 1066 | 69.5%
Claude 3 Opus | 745 | 1263 | 58.9%
Table II: Success rates in the identification of openly chosen patterns

Besides the capability to identify the target patterns, it was also considered
as relevant to know the rate of success in identifying non-target patterns
present in the images, as openly taken by the model from the prescribed list
of patterns (Table II and Fig. III.2).

Figure III.3: Example of additional queries to visualise the answers
reproducibility.

###  III-A Target Patterns

In this test results show (Table I) that the best performing model was that
from OpenAI, i.e., the GPT-4omni model, which identified the target
deterioration pattern in 42.1% of the cases, followed by Gemini 1.5 Pro, which
scored a 38.9% rate of success, while Claude 3 Opus scored only 24.3%.

Results also show (Fig. III.2) that GPT-4omni, despite its overall best
performance, was not the best performing model across all target patterns.
Indeed, Gemini 1.5 Pro scored way better than GPT-4omni in identifying
numerous patterns (e.g. “abrasion”, “alveolisation”, “open joint”, etc.) and
even Claude 3 Opus, the least performing, out passed it in identifying
“craquele” and “alveolisation”. To a certain extent, this can be taken as an
indication of the lack of reliability of the models for this specific use.

As a general rule of thumb, we saw that LMMs are better in the spontaneous
identification of patterns in an open way than in identifying the target
pattern allocated to each image (Table 2). In a simple assessment, this
discrepancy seems to be explicable by the fact that in reality they answer two
distinct questions. The target pattern answers to this implicit question: “is
this specific pattern present?”, while the open identification answers a
different one: “are any of these 46 patterns present?”. The first set of
answers in Table III.1 suggests that the models are still far from being
reliable and that they fall short in guaranteeing that they are able to
identify any specific deterioration pattern. On the other hand, the results in
Table 2 show that the models already have a vast vocabulary and identification
processes incorporated, although they still seem not sufficiently structured
and robust, meaning that, for the most part, they are unusable for practical
purposes.

Overall, these first results mean that existing models, despite their enormous
capabilities, are not prepared to solve problems in the practice of
conservation and restoration of built heritage. In fact, for any image
identified as a paradigm of a given pattern of deterioration, only when the
success rate falls very close to 100% will it be considered ready to be used
for practical purposes.

Besides the rate of success in identifying the target pattern, it is also of
interest the models’ overall success rates in identifying the presence of
patterns freely taken from the open list. In this case Gemini 1.5 Pro scored a
better result, with 11 patterns accurately chosen against 5 from GPT-4omni.
Claude 3 Opus has 9 accurately chosen patterns, but 6 of them were not in the
short list of allowed patterns, for instance “window”, “eroded bricks”, etc.
In strict terms, they could be taken as erroneous identifications, but we
opted to leave them as produced to illustrate this peculiar behaviour of the
model.

###  III-B Interpretability

An important element in developing and deploying reliable and robust AI models
for the conservation and restoration of stone involves understanding the
consistency and reliability in diagnostic capabilities. This is particularly
true for LMMs, whose results are not deterministic but have a certain degree
of variability depending on the ’temperature’ at which the model is queried.
To better understand the process of pattern recognition and its robustness, we
asked GPT-4omni to highlight in red the areas affected by the pattern for
which the model had obtained a high identification score, i.e., the presence
of algae in the illustrated example (Fig. III.3). In this figure, the first
image is the original, part of the test, while the other three are the results
obtained in three separate sessions with the same prompt, i.e.,

> prompt=’’’Colour in red the parts that are subject to algae pathology’’’.

As can be seen, the model is not consistent for the same prompt, providing
very different results. On one hand, this is encouraging, suggesting that
there is significant room for improvement by providing the model with specific
fine-tuning, precise instructions, and diagnostic guardrails. On the other
hand, our analysis highlights the need for the construction of a specific
cognitive architecture, in the absence of which the results of the
foundational model appear to be of weak consistency and thus of questionable
reproducibility.

##  IV Conclusions and Future Developments

The study carried out with a selection of deterioration patterns typically
found in built heritage objects allowed us to verify that the LMMs tested here
were not specifically trained for the conservation and restoration environment
of built heritage. Despite the enormous capabilities that can be attributed to
them, the success rates in identifying patterns are still far from what will
be required of them as working instruments to produce usable results. The
three LMMs tested have a broad vocabulary relevant to the area, but lack the
“understanding” of concepts and terminology specific to the conservation and
restoration domain. Therefore, a specific improvement process will be
necessary to bring them to a level of proficiency compatible with the needs of
professionals in this field of activity.

##  V Data Avalaibility

All data, test images, code and results are available at the url:
GitHub.com/DCorradetti/REDAI_Id_Pattern

##  VI Acknowledgements

This study was carried out under the REDAI project and the Authors thank all
team REDAI for the input and fruitful discussions but most of all José Paulo
Costa for the conceptualisation of the project.

##  VII References

  * •

Anthropic. (2024, March 4). _Introducing the next generation of Claude_. url:
https://www.anthropic.com/news/claude-3-family, retrieved May 27, 2024.

  * •

Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., &
Parikh, D. (2015). _VQA: Visual question answering_. In Proceedings of the
IEEE International Conference on Computer Vision (pp. 2425-2433).
https://doi.org/10.1109/ICCV.2015.279

  * •

Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., …
& Amodei, D. (2020). _Language models are few-shot learners_. arXiv preprint
arXiv:2005.14165.

  * •

Camara, A., de Almeida, A., Caçador, D., & Oliveira, J. (2023). _Automated
methods for image detection of cultural heritage: Overviews and perspectives_.
Archaeological Prospection, 30(2), 153–169. https://doi.org/10. 1002/arp.1883

  * •

Choenni S. , Busker T. & Bargh M. S. , _Generating Synthetic Data from Large
Language Models_ , 2023 15th International Conference on Innovations in
Information Technology (IIT), Al Ain, United Arab Emirates, 2023, pp. 73-78,
doi: 10.1109/IIT59782.2023.10366424.

  * •

Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). _Bert: Pre-training
of deep bidirectional transformers for language understanding_. arXiv preprint
arXiv:1810.04805.

  * •

Doehne, E., & Price, C. A. (2010). _Stone conservation: An overview of current
research_. Getty Publications.

  * •

Gemini Team, Google. (2024). _Advancing multimodal medical capabilities of
Gemini_. arXiv preprint arXiv:2405.03162. https://arxiv.org/pdf/2405.03162

  * •

Gholami, S., & Omar, M. (2023). _Does Synthetic Data Make Large Language
Models More Efficient?_. ArXiv, abs/2310.07830.
https://doi.org/10.48550/arXiv.2310.07830.

  * •

Google. (2024, February 15). _Introducing Gemini 1.5, Google’s next-generation
AI model_. https://blog.google/technology/ai/google-gemini-next-generation-
model-february-2024/ retrieved May 27, 2024.

  * •

Li, Z., Zhu, H., Lu, Z., & Yin, M. (2023). _Synthetic Data Generation with
Large Language Models for Text Classification: Potential and Limitations_.
ArXiv, abs/2310.07849

  * •

Mishra, M., & Lourenço, P. B. (2024). _Artificial intelligence-assisted visual
inspection for cultural heritage: State-of-the-art review_. Journal of
Cultural Heritage, 66, 536-550. https://doi.org/10.1016/j.culher.2024.01.005

  * •

Morris, M. R., Sohl-dickstein, J., Fiedel, N., Warkentin, T., Dafoe, A.,
Faust, A., … & Legg, S. (2023). _Levels of AGI: Operationalizing Progress on
the Path to AGI_. arXiv preprint arXiv:2311.02462.

  * •

OpenAI (2024, May 13) _Hello GPT-4o,_ url: https://openai.com/index/hello-
gpt-4o/ retrieved May 27, 2024.

  * •

ICOMOS-ISCS. (2008). _Illustrated glossary on stone deterioration patterns_.
ICOMOS International Scientific Committee for Stone (ISCS).

  * •

Oses, Noelia; Dornaika, Fadi; Moujahid, Abdelmalik (2014). _Image-Based
Delineation and Classification of Built Heritage Masonry_. Remote Sensing,
6(3), 1863–1889. doi:10.3390/rs6031863

  * •

Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S.,
Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., & Sutskever, I.
(2021). _Learning Transferable Visual Models From Natural Language
Supervision_. In International Conference on Machine Learning.
https://api.semanticscholar.org/CorpusID:231591445

  * •

Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J.,
Michael, J., & Bowman, S. R. (2023). _GPQA: A Graduate-Level Google-Proof Q &A
Benchmark_. arXiv preprint arXiv:2311.12022. https://arxiv.org/abs/2311.12022

  * •

Saab, K., Tu, T., Weng, W.-H., Tanno, R., Stutz, D., Wulczyn, E., Zhang, F.,
Strother, T., Park, C., Vedadi, E., Chaves, J. Z., Hu, S.-Y., Schaekermann,
M., Kamath, A., Cheng, Y., Barrett, D. G. T., Cheung, C., Mustafa, B., Palepu,
A., … Natarajan, V. (2024). _Capabilities of Gemini Models in Medicine_.
arXiv. https://arxiv.org/abs/2404.18416

  * •

Sánchez-Aparicio, L. J., Del Pozo, S., Ramos, L. F., Arce, A., & Fernandes, F.
M. (2018). _Heritage site preservation with combined radiometric and geometric
analysis of TLS data_. Automation in Construction, 85, 24-39.
https://doi.org/10.1016/j.autcon.2017.09.023

  * •

Sánchez-Aparicio, L. J., Masciotta, M. G., GarcÃa-Alvarez, J., Ramos, L. F.,
Oliveira, D. V., MartÃn-Jiménez, J. A., … & Monteiro, P. (2020). _A digital
twin approach towards the assessment of seismic vulnerability of historic
masonry buildings_. In C. Calderini, M. G. Masciotta, & L. Mazzolani (Eds.),
_12th International Conference on Structural Analysis of Historical
Constructions_ (pp. 1743-1761). https://doi.org/10.1007/978-3-030-60796-5_139

  * •

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N.,
… & Polosukhin, I. (2017). _Attention is all you need_. In Advances in neural
information processing systems (pp. 5998-6008).

  * •

Yin S., Fu C., Zhao S., Li K., Sun X., Xu T., Chen E. (2023) _A Survey on
Multimodal Large Language Models_ arXiv:2306.13549
https://doi.org/10.48550/arXiv.2306.13549

a) Departamento de Matemática,

Universidade do Algarve,

Campus de Gambelas,

8005-139 Faro, Portugal

<EMAIL_ADDRESS>
b) Grupo de Física Matemática

Instituto Superior Técnico

Av. Rovisco Pais

1049-001

Lisboa Portugal
c) Consultant in Conservation of Cultural Heritage

<EMAIL_ADDRESS>