# OrthoSeisnet: Seismic Inversion through Orthogonal Multi-scale Frequency
Domain U-Net for Geophysical Exploration

Supriyo Chakraborty a, Aurobinda Routrayb, Sanjay Bhargav Dharavathc , and
Tanmoy Damd CONTACT Supriyo Chakraborty. Email:
<EMAIL_ADDRESS>aDept. ATDC, IIT Kharagpur, Kharagpur, India;
bDept. of Electrical Engineering, IIT Kharagpur, Kharagpur, India cDept.
Physics, IIT Kharagpur, Kharagpur, India India dDept. MAE, NTU Singapore,
Singapore

###### Abstract

Seismic inversion is crucial in hydrocarbon exploration, particularly for
detecting hydrocarbons in thin layers. However, the detection of sparse thin
layers within seismic datasets presents a significant challenge due to the
ill-posed nature and poor non-linearity of the problem. While data-driven deep
learning algorithms have shown promise, effectively addressing sparsity
remains a critical area for improvement. To overcome this limitation, we
propose OrthoSeisnet, a novel technique that integrates a multi-scale
frequency domain transform within the U-Net framework. OrthoSeisnet aims to
enhance the interpretability and resolution of seismic images, enabling the
identification and utilization of sparse frequency components associated with
hydrocarbon-bearing layers. By leveraging orthogonal basis functions and
decoupling frequency components, OrthoSeisnet effectively improves data
sparsity. We evaluate the performance of OrthoSeisnet using synthetic and real
datasets obtained from the Krishna-Godavari basin. Orthoseisnet outperforms
the traditional method through extensive performance analysis utilizing
commonly used measures, such as mean absolute error (MAE), mean squared error
(MSE), and structural similarity index (SSIM)
https://github.com/supriyo100/Orthoseisnet.

###### keywords:

Seismic inversion, OrthoSeisnet, UNet, Frequency domain, Thin-Layer detection,
Kernel modelling, multi-scale resolution, attention, unsupervised learning.

††articletype: Research Paper

##  1 Introduction

Seismic inversion is a pivotal technique in exploring oil and gas reservoirs,
providing a detailed model of the subsurface. By employing sophisticated
algorithms, seismic inversion estimates reflectivity based on seismic trace
data, offering vital insights into geophysical formations. These insights
enable direct forecasts of lithology, porosity, and other hydrocarbon
indications, enhancing our understanding of subsurface formation dynamics Sen
(2006).

Seismic traces, representing acoustic waves propagating through the
subsurface, are influenced by the medium’s absorption, reflection, and
refraction properties as shown by Russell (1988). In the study of sub-strata
geology, seismic traces are inverted to acoustic impedance. However, this
process encounters challenges due to the ill-posedness, non-uniqueness, and
band-limited nature of seismic trace data. Traditional methods, including
deconvolution and regularization techniques like Tikhonov regularization and
total variation regularization (TV), have been employed to address these
challenges in Shuey (1985); Gholami (2015).

A multifaceted approach has emerged to tackle inversion challenges in seismic
data analysis. Diverse strategies, including regularization-based techniques,
dynamic warping, and multi-scale inversion methods in Burstedde Ghattas
(2009); Gregor LeCun (2010); Lin Huang (2013); Bunks . (1995); Chakraborty
Routray (2020); Chakraborty . (2022), play a crucial role in mitigating the
ill-posedness of inverse problems. Additionally, geostatistical seismic
inversion methods, utilizing stochastic sequential simulation and co-
simulation for model perturbation and updates have addressed the challenges
associated with seismic data.

Integrating deep learning (DL) into seismic data analysis represents a
transformative shift in this field. DL, encompassing deep neural networks
(DNNs) and convolutional neural networks (CNNs), showcases remarkable
proficiency in handling spatially structured multidimensional arrays. This
versatility extends across various applications, including seismic inversion,
velocity analysis, and seismic facies classification by Alfarraj AlRegib
(2019); Biswas . (2019); Feng (2020); Jin . (2019); Mosser . (2018); Di .
(2021); Liu . (2021); X. Li . (2021). Notably, Das et al. (2019) employed a 1D
CNN for seismic impedance inversion Das Pollack (2019).

However, the advantages of DL are accompanied by challenges, primarily
stemming from the insufficiency of labelled data. To overcome this hurdle,
strategies such as transfer learning, data augmentation, and synthetic data
generation have been employed, underscoring the ongoing efforts to address
data scarcity in seismic data analysis Mosser . (2018); Das Pollack (2019);
Sun . (2021). Notably, the introduction of physics-driven neural networks by
Karpatne et al. (2017) seek to enhance model-data relationships by leveraging
unlabeled data Karpatne . (2018).

Introduction of neural network methods, such as recurrent neural networks
(RNNs) and 1D CNNs, have demonstrated success in cases constrained by well
logs Jiang . (2022); Tao . (2022); He . (2022); Cui . (2022); Gadylshin .
(2022); Das Pollack (2019); Y. Wang . (2022); Gao . (2022); X. Wu . (2021);
Yuan . (2019); B. Wu . (2020); Sun . (2021).

Addressing challenges specific to seismic impedance inversion, a novel
approach involves the introduction of a self-attention U-Net, leveraging the
noise resistance inherent in CNNs Tao . (2023). Persisting data scarcity,
especially in well logs, necessitates innovative techniques such as data
augmentation and semi-supervised learning Ding . (2017); Chen . (2018).
Physics-guided CNNs, incorporating wave-propagation physics during training,
have been introduced for pre-stack and post-stack inversion in Biswas .
(2019). Furthermore, adopting a closed-loop CNN structure, alternating between
inversion and forward modeling, showcases the potential of deep learning to
overcome traditional limitations in C. Wang . (2019).

The application of DL in seismic inversion extends across acoustic impedance
inversion, pre-stack elastic and lithological parameter inversion, and full
waveform inversion by Alfarraj AlRegib (2019); Das Pollack (2019); B. Wu .
(2020). 2D network-based inversion methods, including U-Net, Attention Gate
(AG), and unstructured convolutional neural network (UCNN), have emerged as an
effective strategy to address spatial correlations, contributing to the
development of more robust and efficient seismic inversion within the realm of
deep learning Ronneberger . (2015); Cao . (2022); Ning . (2023).
Interpretation of seismic data includes post-stack seismic inversion (PSTM) to
resolve a clear picture of possible hydrocarbon deposition. Besides data
scarcity, isolating thin layers is a challenge, as addressed above.

Deep learning (DL) holds significant promise for seismic impedance inversion
but grapples with challenges related to limited training data and the
identification of thin layers. 2-D networks emerge as a solution, leveraging
spatial continuity to address data scarcity and incorporating sparsity for
more effective thin-layer identification. The sparsity property reduces
computational complexity and enhances model performance since thin layers are
sporadically distributed within the overall geological structure. The
presented paper aims to overcome these challenges, recognizing that
identifying thin layers is pivotal in improving inversion accuracy and
boosting success rates in hydrocarbon exploration.

In thin layer detection, the orthogonality inherent in the UNet architecture
represents sparse layers within the seismic trace. The paper introduces a
novel approach using multi-scale 2D frequency domain transformation for multi-
trace inversion to achieve orthogonality. Applying the inverse to the
frequency domain representations of each layer in the UNet makes the resulting
time domain representations sparse, with most values being zero. This
intentional sparsity reduces the model’s computational complexity, enhancing
its accuracy and efficiency. The UNet architecture, complemented by techniques
like early stopping, skip connections with multi-scale 2D frequency domain
transformation, and normalization, facilitates the efficient estimation of
sparse layers from seismic trace images. The methodological contributions in
this paper can be outlined as follows:

  1. 1.

Introducing a novel approach that employs multi-scale 2D frequency domain
transformation within the UNet architecture, facilitating an enhanced sparsity
representation.

  2. 2.

Incorporating multi-trace inversion using a 2D CNN within the UNet block,
ensuring lateral continuity among traces is preserved.

  3. 3.

Conducting extensive synthetic and real data experiments to validate the
proposed algorithms.

In Section II, we review related works in the field of Deep Learning (DL)
applied to PSTM (Pre-Stack Time Migration) seismic data. The methodology is
presented in Section III, followed by experimental results in Section IV.
Section V contains the discussion, and the conclusion is presented in Section
VI.

##  2 Background

Recent seismic inversion advances are attributed to applying deep learning
(DL) techniques, recognized for capturing intricate nonlinear relationships in
seismic data LeCun . (2015). DL, distinguished from traditional model-driven
methods, utilizes adjustable parameters for learning from training datasets
and exhibits remarkable performance in diverse domains LeCun . (2015).

DL’s success extends to geophysical applications, showcasing potential in
learning geological formations Sun . (2020) and fault detection X. Wu .
(2019). Convolutional Neural Networks (CNNs) Dam . (2021, 2022) demonstrate
promise in predicting acoustic impedance (AI), which is pivotal for reservoir
characterization. However, one-dimensional CNNs face challenges in real data
due to poor lateral continuity Das Pollack (2019). To address this, a physics-
guided CNN tailored for pre-stack inversion has been proposed Biswas . (2019).

DL-based seismic inversion encounters a challenge in the need for substantial
labeled data. Mitigating this, semi-supervised CNNs leverage labelled and
unlabeled data, improving results by utilizing abundant unlabeled seismic data
Alfarraj AlRegib (2019). Domain adaptation networks, incorporating unlabeled
data during training, enhance performance Q. Wang . (2022).

Studies contribute to DL advancements in seismic inversion, including a semi-
supervised sequence modelling approach for elastic impedance inversion
Alfarraj AlRegib (2019) and InversionNet, a real-time full-waveform inversion
method incorporating CNNs and continuous conditional random fields (CRFs) for
improved efficiency and accuracy Y. Wu Lin (2018). Comparative studies provide
insights into different DL methods for seismic impedance inversion SB. Zhang .
(2022). A deep-learning inversion framework emphasizes its capability to learn
complex features and enhance accuracy S. Li . (2019). Exploration extends to
three-dimensional (3D) approaches in seismic inversion, exemplified by 3D
full-waveform inversion in the time-frequency domain applied to field data
Tran . (2020). An encoder-decoder architecture efficiently handles large-scale
3D seismic inversion Gelboim . (2022).

DL techniques are harnessed in specific applications, such as salt body
classification for salt interpretation and reservoir characterization ul Islam
(2020). An autoencoder-based model introduces dimensionality reduction for
large-dimensional seismic inversion, enhancing efficiency while preserving
accuracy Gao . (2020). State-of-the-art works introduce novel techniques,
including a seismic inversion method based on two-dimensional CNNs and domain
adaptation to tackle domain mismatch between training and testing data Q. Wang
. (2022). A multichannel seismic impedance inversion approach based on
Attention U-Net incorporates attention mechanisms to capture important
features and improve accuracy Ning . (2023). These studies demonstrate the
potential of DL techniques in seismic inversion, showcasing improvements in
inversion accuracy, computational efficiency, and the ability to handle
complex subsurface structures. DL-based seismic inversion can capture complex
nonlinear relationships, allowing for more accurate and reliable inversion
results compared to traditional model-driven methods. Current literature has
yet to target sparseness. Specifically, we have introduced an orthogonal
seismic inversion network (Orthoseisnet) that uses the Unet backbone with
multi-scale frequency domain transformation to target thin layers for higher
resolution in PSTM seismic data.

##  3 Methodology

This section discusses seismic inversion theory, showing challenges in
inversion and traditional ways to estimate optimized solutions. The
architecture modification of the Unet backbone with a multi-scale frequency
domain using FFT and IFFT is discussed in detail.

###  3.1 Theory

A seismic impedance inversion is a foremost tool for exploring hydrocarbon in
seismic exploration. By convention, we can define forward and inverse problems
as:

  * •

Forward problem: Given a model of the subsurface, generate seismic traces.

  * •

Inverse Problem:  Inverse problem: Given seismic traces, estimate the
subsurface model.

####  3.1.1 Forward problem

The forward model for seismic trace generation can be represented as:

$S(t)=w(t)\ast r(t)+\in(t)$ (1)

where $S(t)$ is the seismic trace, $w(t)$ is the wavelet, $r(t)$ is the
reflectivity, and $\in(t)$ is the noise. The wavelet gives the shape of the
seismic signal. At the same time, reflectivity represents the changes in
acoustic impedance in the sub-surface, whereas noise is the random function
representing the errors in the dataset. The reflectivity can be calculated
using the following equation:

$r(t)=\frac{{di(t_{p})}}{{dt}}\frac{1}{{2i_{p}(t)}}=\frac{{\Delta\ln(i{p_{i}})}}{2}$
(2)

Where the acoustic primary impedance$(ip)$ is given by:

$i_{p}=\nu*\rho$ (3)

where $\nu$ is primary velocity and $\rho$ is the density.

####  3.1.2 Inverse problem

For the inverse problem, we estimate reflectivity from a given seismic trace
data considering a dictionary of wavelets. Due to its non-unique and under-
determined nature, inversion is ill-posed. To make the inverse problem well-
posed, we need to add regularization terms to the objective function.
Regularization terms penalize solutions inconsistent with prior knowledge or
assumptions about the subsurface. The objective function minimizes the error
by optimizing the model based on converging criteria, resulting in a stable
solution:

$\phi(m)=\left\|{d-Gm}\right\|_{2}^{2}+\chi{\left\|m\right\|_{1}}$ (4)

Where the objective function is $\phi(m)$, $d$ is the observed seismic data,
$G$ is the measurement matrix, $m$ is the model reflectivity, and $\chi$ is a
regularization term. The choice of regularization term depends on the desired
solution properties, such as $L_{1}$ norm, total variation, or smoothness
constraints. The solution to the optimization problem is the estimated model
parameters $\mathbf{m}^{\star}$ that best fit the observed data and satisfy
the regularization constraints.

###  3.2 Network Architecture

This paper uses data-driven methods to solve the reflectivity inversion
problem using Unet for feature learning and extraction. The network maps
seismic trace to impedance, and algorithm output behaves like inversion but is
constrained solely by data. The representation ability of DL is used as black
boxes as it helps to predict the network relationship between seismic trace
and impedance. The network architecture described above follows a U-Net-like
structure commonly used in image segmentation tasks. The architecture consists
of a contraction and expansive paths, as shown in Figure 1. In Fourier-
DeepONet Zhu . (2023), Fourier, along with Unet, was used to create
architecture, but the network has several Unet architectures, increasing
complexity and possibility of over-fitting. Our work aims to keep the network
light while resolving high and low-frequency features. Figure 1 shows
Orthoseisnet architecture, which orthogonalizes frequency domain components to
recover sparse features. The FFT-iFFT block includes R, which is the
parameterization in Fourier space.

Figure 1: Feature Extraction Module for Multi-scale Frequency domain Unet

The model is compiled using the Adam optimizer, mean squared error (MSE), SSIM
and other losses. The accuracy metric is also calculated. Unet input
$\mathbf{X}$ and output $\mathbf{Y}$ can be shown as below:

$\mathbf{X,Y}\in\mathbb{R}^{H\times W\times C}$ (5)

Where H, W, and C are the feature graph’s height, width, and number of
channels, respectively. The overall data flow is shown in Figure 1 details,
which are given in architecture Table 1.

####  3.2.1 Encoder Layer

The convolutional Layer is of Kernel size: $3\times 3$ starting with filters:
$16$, using activation function: $\tanh$. Output tensor:
$\mathbf{Y^{L}}\in\mathbb{R}^{H\times W\times 16}$. The height $H$ and width
$W$, is processed through the following operations:

$Y_{m}=\mathcal{}{m}(\tanh\left(\text{Conv2D}(\mathcal{D}\left\\{\tanh\left(\text{Conv2D}(X,K)\right)\right\\},K^{\prime})\right))$
(6)

$\mathbb{f}=\mathcal{F}^{-1}\left\\{\mathcal{F}(Y_{m})\right\\}$ (7)

where maxpooling $\mathcal{}{m}$ dropout $\mathcal{D}(0.1-0.3)$, and
convolution is given by :

$Y^{L}1_{i,j,k}=\tanh\left(\sum_{k^{\prime}=0}^{2}\sum_{l^{\prime}=0}^{2}X_{2i+k^{\prime},2j+l^{\prime},1}\cdot
K_{k^{\prime},l^{\prime},1,k}\right)$ (8)

####  3.2.2 Multi-scale Frequency Domain Transform Layer

: Certainly! Here are the LaTeX equations for the provided descriptions:

A) 2D Fourier transform $\mathbb{(}F)$is applied to the local 2D Convolution
layer output. Input: $\mathbf{Y}$, Output:
$\mathcal{F}\in\mathbb{C}^{\frac{H}{2}\times\frac{W}{2}\times 16}$ (complex-
valued)

$\mathcal{F}_{u,v,k}=\frac{1}{\frac{H}{2}\cdot\frac{W}{2}}\sum_{x=0}^{\frac{H}{2}-1}\sum_{y=0}^{\frac{W}{2}-1}Yp_{x,y,k}\exp\left(-j2\pi\left(\frac{ux}{\frac{H}{2}}+\frac{vy}{\frac{W}{2}}\right)\right)$
(9)

Where $u=0,1,...,\frac{H}{2}$ and $v=u=0,1,...,\frac{W}{2}$

B) Parameterizing weight tensor (R) with $\kappa$ directly by its Fourier
coefficients:

$\left(\mathcal{K}\left(v_{l}\right)\right)(x)=\mathcal{F}^{-1}\left(R\cdot\mathcal{F}\left(v_{l}\right)\right)(x),\quad\forall_{x}\in
D$ (10)

where $\mathcal{F}$ denotes the Fourier transform of a function
$f:D\rightarrow\mathbb{R}^{c}$ and $\mathcal{F}^{-1}$ is its inverse.
C)Expressing multiplication by the learnable weight tensor $R$:

$\left(R\cdot\mathcal{F}\left(v_{l}\right)\right)_{k,i}=\sum_{j=1}^{c}R_{k,i,j}\left(\mathcal{F}\left(v_{l}\right)\right)_{k,j},\quad\forall
k=1..k_{\max},i=1..c$ (11)

D)Followed by Inverse 2D Multi-scale Frequency Domain Transform Layer
$\mathbb{F^{-1}}$ : Input: $\mathcal{F}$ \- Output:
$\mathbb{f}\in\mathbb{C}^{\frac{H}{2}\times\frac{W}{2}\times 16}$ (complex-
valued)

$\mathbb{f}_{x,y,k}=\sum_{u=0}^{\frac{H}{2}-1}\sum_{v=0}^{\frac{W}{2}-1}\mathcal{F}_{u,v,k}\exp\left(j2\pi\left(\frac{ux}{\frac{H}{2}}+\frac{vy}{\frac{W}{2}}\right)\right)$
(12)

The output of $1^{st}$ Unet encoding layer is fed to the next layer. Where the
complex-valued output of the previous layer is used as the absolute value
input to the first convolutional layer in the next encoding layer. The output
tensor of this convolutional layer, denoted as $\mathbf{YL1}$, is computed as:

$\mathbf{YL1}{i,j,k}=\tanh\left(conv2D(Y_{m},K)\right)\left|\mathbb{f}\right|$
(13)

Where $\mathbf{YL1}$ represents the output tensor of the first convolutional
layer in the next encoding layer, the last encoding layer consists of two
conv2d layers (activation=’tanh’, kernel size=$3\times 3$) with a dropout
layer between them.

####  3.2.3 Decoder Layer

${x^{i+1}}=\tanh\left\\{\mathcal{D}\left\\{\text{ReLU}\left(\text{Conv2D}\left(\mathbb{C}\left\\{\text{Conv2D}\left(\mathbb{Z}L2_{i,j,k}\cdot
K\right)\right\\}\right)\right)\right\\}\right\\}$ (14)

Where Concatenation is $\mathbb{C}$ and $\mathbb{Z}L2_{i,j,k}$ is given by

$\mathbb{Z}L2_{i,j,k}=\left(\mathbb{C}\left\\{\text{Conv2D}\left({W^{t}}\cdot{x_{i}}\right)\right\\}\right)$
(15)

ReLU activation is used with a Convolution-2D layer with a dropout
$\mathbb{D}$ layer with $0.1to0.3$. The Multi-scale Frequency Domain Transform
Layer gives the final layer for every decoder layer:

$\mathcal{F}_{u,v,k}=\frac{1}{\frac{H}{2}\cdot\frac{W}{2}}\sum_{x=0}^{\frac{H}{2}-1}\sum_{y=0}^{\frac{W}{2}-1}\mathbb{Z}L2_{x,y,k}\exp\left(-j2\pi\left(\frac{ux}{\frac{H}{2}}+\frac{vy}{\frac{W}{2}}\right)\right)$
(16)

Inverse 2D Multi-scale Frequency Domain Transform Layer:

$\mathbb{f}_{x,y,k}=\sum_{u=0}^{\frac{H}{2}-1}\sum_{v=0}^{\frac{W}{2}-1}\mathcal{F}_{u,v,k}\exp\left(j2\pi\left(\frac{ux}{\frac{H}{2}}+\frac{vy}{\frac{W}{2}}\right)\right)$
(17)

The absolute value from the Multi-scale Frequency layer is used as input to
the next decoder layer as

$\mathbb{Z}L1^{{}^{\prime}}_{i,j,k}=\tanh\left\\{\text{Conv2D}\left\\{Z\cdot
K\right\\}\left|\mathbb{f}_{x,y,k}\right|\right\\}$ (18)

The final layer uses soft-max to output the estimated resolved image.

Table 1: Orthoseisnet Network Architecture Block | Layer | Unit | Parameters
---|---|---|---
Input | 0 | Seismic 2D ($128\times 128\times 1$) | 0
Enc1 | 1 | Conv2D(16, (3,3),tanh)+GN+L1 | 160+32
| 2 | Dropout, Conv2D+GN | 2320+32
| 3-4 | MaxPooling, FFT, iFFT | 0
Enc2 | 5-8 | Enc(32) abs(iFFT) | 14016
Enc3 | 9-12 | Enc(64) abs(iFFT) | 55680
Enc4 | 13-16 | Enc(128) abs(iFFT) | 221952
Enc5 | 17 | Conv2D(256) abs(iFFT) | 295168
Dec1 | 18 | Concat | 1.312M
| 19 | Conv2DTrans(128)+GN | 110976
| 20 | Dropout, Conv2D | 147584
| 21 | FFT, iFFT |
Dec2 | 22-26 | Dec(64), Dec(32), Dec(16) | 37k, 2320
Dec3 | 22-26 | Dec(64), Dec(32), Dec(16) | 37k, 2320
Dec4 | 22-26 | Dec(64), Dec(32), Dec(16) | 37k, 2320
|  | Total Parameters | 1,940,817

26816 This architecture Table 1 is designed for processing seismic data,
leveraging conventional convolutional layers and operations in the frequency
domain (FFT/iFFT). The table Table 1 offers insights into the structure and
complexity of the Orthoseisnet neural network.

###  3.3 Loss function

The impact of the loss function, network architecture, and data type on the
inversion resolution of complex stratigraphic boundaries has been thoroughly
investigated S. Li . (2019); Liu . (2021); Y. Wu Lin (2019); Araya-Polo .
(2019). A novel approach combining the structural similarity index (SSIM) and
L1 norm was proposed to improve the accuracy of extracting construction
boundaries and geological structures. Incorporating SSIM into the loss
function significantly enhances the inversion resolution S. Li . (2019).

The proposed approach utilizes a mixed loss function that combines mean
squared error (MSE) and SSIM to optimize pixel and local patch velocity
misfits simultaneously. Synthetic and real data were used to validate this
approach, which outperformed using only MSE as the loss function.

In conclusion, this study thoroughly investigates the impact of the loss
function, network architecture, and data type on the inversion resolution of
complex stratigraphic boundaries. The proposed approach, which integrates SSIM
into the loss function, demonstrates its effectiveness in accurately
extracting geological structures and improving the inversion resolution.

Different loss functions, including MSE, Mean Absolute Error (MAE), and SSIM,
were considered in the U-Net architecture for pixel-wise evaluation. Ms-SSIE
incorporates multi-scale analysis, improving image structure sensitivity,
scaling, and translation robustness, and higher correlation with human
perception of image quality Liu . (2021).

Mean Absolute Error (MAE) measures the average absolute difference between a
dataset’s predicted and actual values. It is a robust metric that indicates
the model’s prediction accuracy Y. Wu Lin (2019).

For a dataset with $N$ samples, the MAE is calculated as:

$\text{MAE}=\frac{1}{N}\sum_{i=1}^{N}|\hat{y}_{i}-y_{i}|$ (19)

Here, $\hat{y}_{i}$ represents the predicted value for the $i$-th sample, and
$y_{i}$ represents the corresponding actual value. The absolute differences
between predicted and actual values are summed and then averaged across all
samples to obtain the MAE.

The MSE loss is represented as Y. Wu Lin (2019):

$\mathcal{L}_{\text{MSE}}=\frac{1}{N}\sum_{i=1}^{N}\|\hat{y}_{i}-y_{i}\|^{2}$
(20)

The Structural Similarity Index (SSIM) measures the similarity between two
images, considering contrast and structure. The SSIM index produces a value
between -1 and 1, where 1 indicates perfect similarity. SSIM loss for
$I(i,j),K(i,j)$ Araya-Polo . (2019) is represented as SSIM in the equation
below:

$SSIM=\frac{{(2\cdot\mu_{I(i,j)}\cdot\mu_{K(i,j)}+C_{1})\cdot(2\cdot\sigma_{I(i,j),K(i,j)}+C_{2})}}{{(\mu_{I(i,j)}^{2}+\mu_{K(i,j)}^{2}+C_{1})\cdot(\sigma_{I(i,j)}^{2}+\sigma_{K(i,j)}^{2}+C_{2})}}$
(21)

Where $I(i,j)$ and $K(i,j)$ are the pixel values of the two images being
compared, $\mu_{I(i,j)}$ and $\mu_{K(i,j)}$ are the means of the pixel values
in the local windows of images $I$ and $K$ and $\sigma_{I(i,j),K(i,j)}$ is the
covariance of the pixel values in the local windows of images $I$ and $K$.

$\sigma_{I(i,j)}^{2}$ and $\sigma_{K(i,j)}^{2}$ are the variances of pixel
values in the local windows of images $I$ and $K$. $C_{1}$ and $C_{2}$ are
constants added to avoid instability when the denominators are near zero. The
advantage of using SSIM in seismic inversion, particularly in the context of a
U-Net-based model, lies in its ability to capture perceptual differences
between images, unlike traditional pixel-wise metrics such as MSE.

##  4 Experiment

The proposed OrthoSeisnet network is validated on synthetic and real seismic
trace data.

###  4.1 Synthetic data

We conducted experiments using the Marmousi synthetic dataset, known for its
geological complexity Figure 2. The dataset comprises 13,200 traces with 1001
samples per trace at a 1 ms time interval. Experiments were implemented in
Python 3.6 using TensorFlow on Google Colab TPUs. The network, OrthoSeisnet,
had input/output sizes of $128\times 128$, an initial learning rate of
$1\times 10^{-2}$, and was trained for 100 epochs. OrthoSeisnet has 1.9
million parameters, with pre-training taking 0.9 min/sample, fine-tuning
taking 0.8 min, and an inference time of 0.09 seconds.

OrthoSeisnet consistently outperformed existing methods with increased
training data. Noisy data experiments at signal-to-noise ratios
($\mathcal{SNR}$) of $10$ dB, $20$ dB, and $30$ dB for elastic parameter
inversion ($V_{p}$, $V_{s}$, $\rho$) favored OrthoSeisnet over model-driven
approaches. MSE values confirmed the superiority of data-driven methods, with
OrthoSeisnet exhibiting the smallest MSE. In conclusion, despite increased
computational demands, the Marmousi synthetic dataset served as an effective
testing ground for data-driven models, particularly OrthoSeisnet, showcasing
their potential in seismic inversion tasks.

Figure 2:  2D Marmousi2 Synthetic Dataset Figure 3:  Marmousi2 Vs.
Orthoseisnet Vs Unet Trace No 2, 400, 200 Figure 4:  Marmousi2 Vs.
Orthoseisnet Vs Unet Trace No 800, 1000, 2000

Figure 5: Krishna-Godavari basin Courtesy: National data repository, India

###  4.2 Real Data Experiment

The data provided by the Oil and Natural Gas Corporation of India(ONGC) is 3D
Post-stack seismic trace data under the GEOPIC project. The data is $150sq.km$
with $12$ wells in the Krishna-Godavari basin. This basin is an extensive
deltaic plain formed by the Krishna and Godavari rivers in Andhra Pradesh and
the adjoining areas of the Bay of Bengal, shown in Figure 5. The basin is a
proven petroliferous basin located on the east coast of India with sediments
about 5 km thick ranging in age from Late Carboniferous to Pleistocene. The
major geo-morphological units of the basin are upland plains, coastal plains,
recent floods, and delta plains. Geological surveys, gravity-magnetic surveys,
and seismic coverage have been carried out in the area. More than $225$
prospects have been probed by drilling over $557$ exploratory wells, and
hydrocarbon accumulations have been proven in $75$ of these prospects. The
basin’s tectonic history comprises the rift, passive margin, compressional,
and post-compressional stages. The data provided is shown in the table above.
The survey area of the data can be seen if the Figure 5 Input size of
$128\times 128$ is used just like synthetic data but across the Inline and
Crosslines, as shown in the data below.

Table 2: KG Basin Dataset Details Parameter | Value
---|---
Inline Range | 70-585
Crossline Range | 17-1093
Bin Size | 17.5m$\times$17.5m
Sampling Interval | 2ms
Data Length | 0-2sec
Figure 6: (A-D) marmousi2 data 0dB, 10dB, 20dB, 30dB noise (E-H) Orthoseisnet 0dB, 10dB, 20dB, 30dB noise (I-L) Unet 0dB, 10dB, 20dB, 30dB 1D CNN 0dB, 10dB, 20dB, 30dB noise. Table 3: Table 1: Performance Comparison of Different Networks on Synthetic Noisy Dataset  Data- |  |  |  |
---|---|---|---|---
set | Network |  MAE | MSE | SSIM
Synthe- |  |  |  |
tic (0db) |  |  |  |
noise | OrthoSeisnet(ours) |  0.043 | 0.005 | 0.991
| Inversion-net Y. Wu Lin (2019) |  0.95 | 1.261 | 0.846
| Unet Ronneberger . (2015) |  2.43 | 3.12 | 0.623
| CNN Alfarraj AlRegib (2019) |  2.65 | 3.22 | 0.636
Synthe- |  |  |  |
tic (10db) |  |  |  |
noise | OrthoSeisnet(ours) |  0.083 | 0.012 | 0.855
| Inversion-net Y. Wu Lin (2019) |  0.861 | 1.0597 | 0.701
| Unet Ronneberger . (2015) |  2.74 | 3.331 | 0.602
| CNN Alfarraj AlRegib (2019) |  2.78 | 3.35 | 0.611
Synthe- |  |  |  |
tic (20db) |  |  |  |
noise | OrthoSeisnet(ours) |  0.112 | 0.021 | 0.803
| Inversion-net Y. Wu Lin (2019) |  0.871 | 1.083 | 0.635
| Unet Ronneberger . (2015) |  3.15 | 3.56 | 0.542
| CNN Alfarraj AlRegib (2019) |  3.09 | 3.42 | 0.533
Synthe- |  |  |  |
tic (30db) |  |  |  |
noise | OrthoSeisnet(ours) |  0.134 | 0.032 | 0.772
| Inversion-net Y. Wu Lin (2019) |  0.889 | 1.118 | 0.608
| Unet Ronneberger . (2015) |  3.52 | 3.89 | 0.511
| CNN Alfarraj AlRegib (2019) |  3.76 | 4.16 | 0.513

A comparative analysis of synthetic data at various noise levels is conducted
for OrthoSeisnet, Inversionnet Y. Wu Lin (2018), and Basis Pursuit (BPI) R.
Zhang Castagna (2011). Table 3 presents the performance of these networks on
noisy synthetic datasets, with a specific focus on OrthoSeisnet. The
discussion centers on the improvements observed in the loss metrics (MAE, MSE,
SSIM) compared to Inversionnet and SeisInvnet. The table provides a
comprehensive comparison across different noise levels (0db, 10db, 20db, 30db)
and evaluates Mean Absolute Error (MAE), Mean Squared Error (MSE), and
Structural Similarity Index (SSIM). OrthoSeisnet consistently outperforms
Inversion-net, Unet, and SeisInvnet, showcasing lower MAE and MSE values and
higher SSIM values. This emphasizes OrthoSeisnet’s superior accuracy and
robustness in handling synthetic noisy datasets, making it a promising choice
for seismic data prediction. The results indicate that OrthoSeisnet
effectively preserves structural patterns, as reflected in its high SSIM
values approaching 1. The table offers a clear and concise overview of each
network’s performance metrics under different noise conditions.

Figure 4 compares OrthoSeisnet with the latest state-of-the-art algorithm
concerning different noise levels in the Marmousi2 synthetic model. The
clarity of layers from OrthoSeisnet shows the algorithm’s effectiveness.
Figure 7. shows a bigger view of the synthetic model at 20db noise.

It’s important to note that the improvements mentioned above are specific to
the respective metrics (MAE, MSE, SSIM). They are calculated by comparing the
values of OrthoSeisnet to Inversionnet. For the noisy synthetic datasets with
10 dB, 20 dB, and 30 dB noise levels, you can perform similar calculations to
determine the percentage improvement in the loss metrics for OrthoSeisnet
compared to Inversionnet and SeisInvnet.

In summary, OrthoSeisnet exhibits improvement in the loss metrics (MAE, MSE,
SSIM) compared to Inversionnet and SeisInvnet on noisy synthetic datasets,
with percentage improvements varying across the metrics.

###  4.3 Experiment on Real data

OrthoSeisnet on real data improves the $figure$ and $Table2$.

Figure 7: Comparison between Original Seismic Data and Orthoseisnet at Inline
Sections: (a-b) Inline 39, (c-d) Inline 277, (e-f) Inline 316, and (g-h)
Inline 460.

Comparison with the well-log of KG basin well GS-29-10, DWN-S-1 and DWN-Q-1
and their neighbouring original seismic trace to estimated trace is shown in
Figure 8.

Figure 8: Original seismic data Vs. Orthoseisnet at Inline with neighbouring Well 1-3  Table 4: Performance Comparison of Different Networks on Real Data Network | MSE | MAE | SSIM | R2
---|---|---|---|---
BPI R. Zhang Castagna (2011) | 0.349 | 0.6781 | 0.7454 | 0.7654
CNN Alfarraj AlRegib (2019) | 1.152 | 3.58 | 0.6377 | 0.7171
Unet Ronneberger . (2015) | 0.8541 | 2.964 | 0.6891 | 0.7382
Inversion-net Y. Wu Lin (2019) | 0.2346 | 1.391 | 0.7830 | 0.8141
OrthoSeisnet (Ours) | 0.02678 | 1.102 | 0.884 | 0.8945

Table LABEL:Table:_Real_Data_comparison provides a performance comparison of
different networks on real data. The networks evaluated are OrthoSeisnet,
Inversionnet, and SeisInvnet. The evaluation metrics used are Mean Absolute
Error (MAE), Mean Squared Error (MSE) and Structural Similarity Index (SSIM).
The performance comparison of different networks on real data reveals that
OrthoSeisnet surpasses its counterparts, achieving the lowest Mean Squared
Error (MSE) of 0.02678 and Mean Absolute Error (MAE) of 1.102. Additionally,
its Structural Similarity Index (SSIM) of 0.884 indicates superior capture of
structural patterns. These metrics collectively highlight OrthoSeisnet’s
exceptional performance, outclassing BPI, CNN, Unet, and Inversion-net in
minimizing errors and accurately representing real data patterns.

###  4.4 Epoch Vs. Loss

Figure 9: Epoch Vs. Loss

The relationship between the number of epochs and the corresponding loss
function values in the OrthoSeisnet model can be summarized as follows.
Initially, during the first 40 epochs, the loss decreases significantly and
converges faster. However, the model requires more than 100 epochs to achieve
further convergence beyond this point. This can be seen in figure 9

##  5 Discussion

The performance comparison of OrthoSeisnet Inversionnet and BPI on real
seismic data evaluated their effectiveness using metrics such as Mean Absolute
Error (MAE), Mean Squared Error (MSE), Structural Similarity Index (SSIM),

The results establish the superiority of OrthoSeisnet over Inversionnet and
SeisInvnet regarding prediction accuracy and capturing structural patterns.
OrthoSeisnet outperforms the other networks with lower MAE and MSE values,
indicating reduced errors, and achieves a higher SSIM value, indicating better
similarity in structural patterns.

In contrast, Inversionnet and SeisInvnet exhibit higher MAE and MSE values,
indicating larger prediction errors. These networks’ SSIM values are slightly
lower than OrthoSeisnet’s. The proposed algorithm, OrthoSeisnet, incorporates
basis orthogonalization with Fourier transform and utilizes different
frequency-band weighting to improve the detection of thin layers in the
substrate. OrthoSeisnet’s integration of an orthogonal base with 2D Fourier
transform enhances structural resolution for detecting sparse thin layers. A
comparison with existing algorithms demonstrates the superior ability of the
proposed approaches to isolate layer information and achieve better results.

Overall, the discussion highlights the superior performance of OrthoSeisnet in
lower MAE and MSE values, indicating reduced errors while demonstrating
competitive SSIM values. These findings provide valuable insights into the
effectiveness of deep learning-based approaches in addressing the challenges
posed by sparse seismic data and improving the accuracy of capturing
structural information.

##  6 Conclusion

To conclude, this study establishes the effectiveness of OrthoSeisnet in
addressing the challenges associated with sparse seismic data, thereby
improving the accuracy and fidelity of seismic imaging. The performance
comparison against Inversionnet and SeisInvnet demonstrates the superiority of
OrthoSeisnet in lower MAE and MSE values, indicating reduced errors while
showcasing competitive SSIM values.

The proposed algorithm, OrthoSeisnet, which incorporates basis
orthogonalization with Fourier transform and utilizes different frequency-band
weighting, exhibits improved performance in detecting thin layers and
isolating layer information. This research contributes to advancing deep
learning techniques in seismic data analysis, offering the potential for
further enhancements in subsurface imaging and exploration.

Overall, the results of this study demonstrate the significant potential of
OrthoSeisnet in enhancing the accuracy and reliability of seismic data
interpretation, paving the way for future advancements in seismic imaging
technologies.

##  7 Acknowledgements

The author wants to thank ONGC for the Krishna Godavari basin data received
under the GEOPIC project. We are also grateful for the advanced system
provided by ONGC, INDIA, to perform high G-flops operations.

## References

  * Alfarraj AlRegib (2019) alfarraj2019semisupervisedAlfarraj, M. AlRegib, G.  2019\.  Semisupervised sequence modeling for elastic impedance inversion Semisupervised sequence modeling for elastic impedance inversion.  Interpretation73SE237–SE249.
  * Araya-Polo . (2019) araya2019fastAraya-Polo, M., Adler, A., Farris, S. Jennings, J.  2019\.  Fast and accurate seismic tomography via deep learning Fast and accurate seismic tomography via deep learning.  Deep learning: Algorithms and applications Deep learning: Algorithms and applications ( 129–156).  Springer.
  * Biswas . (2019) biswas2019prestackBiswas, R., Sen, MK., Das, V. Mukerji, T.  2019\.  Prestack and poststack inversion using a physics-guided convolutional neural network Prestack and poststack inversion using a physics-guided convolutional neural network.  Interpretation73SE161–SE174.
  * Bunks . (1995) bunks1995multiscaleBunks, C., Saleck, F., Zaleski, S. Chavent, G.  1995\.  Multiscale seismic waveform inversion Multiscale seismic waveform inversion.  Geophysics601457–1473.
  * Burstedde Ghattas (2009) burstedde2009algorithmicBurstedde, C. Ghattas, O.  2009\.  Algorithmic strategies for full waveform inversion: 1D experiments Algorithmic strategies for full waveform inversion: 1d experiments.  Geophysics7437–46.
  * Cao . (2022) Cao2022Cao, JJ., Yao, G. da Silva, NV.  2022\.  Interpolation of Irregularly Sampled Noisy Seismic Data with the Nonconvex Regularization and Proximal Method Interpolation of Irregularly Sampled Noisy Seismic Data with the Nonconvex Regularization and Proximal Method.  Pure and Applied Geophysics1792663–678.  10.1007/s00024-021-02936-3
  * Chakraborty Routray (2020) chakraborty2020multiChakraborty, S. Routray, A.  2020\.  Multi-trace Inversion with Selective Wedge to Increase Thin Layer Resolution in Horizontal Layer Multi-trace inversion with selective wedge to increase thin layer resolution in horizontal layer.  2020 11th International Conference on Computing, Communication and Networking Technologies (ICCCNT) 2020 11th international conference on computing, communication and networking technologies (icccnt) ( 1–5).
  * Chakraborty . (2022) chakraborty2022identificationChakraborty, S., Routray, A. Tewari, RC.  2022\.  Identification of Thin Layer Via Source Wave-Field Dictionary Learning Identification of thin layer via source wave-field dictionary learning.  IECON 2022–48th Annual Conference of the IEEE Industrial Electronics Society Iecon 2022–48th annual conference of the ieee industrial electronics society ( 1–6).
  * Chen . (2018) chen2018semiChen, G., Liu, L., Hu, W. Pan, Z.  2018\.  Semi-supervised object detection in remote sensing images using generative adversarial networks Semi-supervised object detection in remote sensing images using generative adversarial networks.  Proceedings of the IEEE International Geoscience and Remote Sensing Symposium Proceedings of the ieee international geoscience and remote sensing symposium ( 2503–2506).  10.1109/IGARSS.2018.8519132
  * Cui . (2022) cui2022mapCui, S., Ma, A., Zhang, L., Xu, M. Zhong, Y.  2022\.  MAP-Net: SAR and Optical Image Matching via Image-Based Convolutional Network With Attention Mechanism and Spatial Pyramid Aggregated Pooling Map-net: Sar and optical image matching via image-based convolutional network with attention mechanism and spatial pyramid aggregated pooling.  IEEE TGRS601–13.
  * Dam . (2021) dam2021improvingDam, T., Anavatti, SG. Abbass, HA.  2021\.  Improving ClusterGAN Using Self-AugmentedInformation Maximization of Disentangling LatentSpaces Improving clustergan using self-augmentedinformation maximization of disentangling latentspaces.  arXiv preprint arXiv:2107.12706.
  * Dam . (2022) dam2022latentDam, T., Ferdaus, MM., Pratama, M., Anavatti, SG., Jayavelu, S. Abbass, H.  2022\.  Latent preserving generative adversarial network for imbalance classification Latent preserving generative adversarial network for imbalance classification.  2022 IEEE International Conference on Image Processing (ICIP) 2022 ieee international conference on image processing (icip) ( 3712–3716).
  * Das Pollack (2019) das2019convolutionalDas, V. Pollack, A.  2019\.  Convolutional neural network for seismic impedance inversion Convolutional neural network for seismic impedance inversion.  Geophysics84R869–R880.
  * Di . (2021) di2021seismicDi, H., Chen, X., Maniar, H. Abubakar, A.  2021\.  Seismic acoustic impedance estimation by learning from sparse wells via deep neural networks Seismic acoustic impedance estimation by learning from sparse wells via deep neural networks.  82nd EAGE Annual Conference & Exhibition 82nd eage annual conference & exhibition ( 2021, 1–5).
  * Ding . (2017) ding2017augmentationDing, J., Li, X. Gudivada, VN.  2017\.  Augmentation and evaluation of training data for deep learning Augmentation and evaluation of training data for deep learning.  IEEE International Conference on Big Data (Big Data) Ieee international conference on big data (big data) ( 2603–2611).
  * Feng (2020) feng2020unsupervisedFeng, R.  2020\.  Unsupervised learning elastic rock properties from pre-stack seismic data Unsupervised learning elastic rock properties from pre-stack seismic data.  Journal of Petroleum Science and Engineering192107237.
  * Gadylshin . (2022) gadylshin2022numericalGadylshin, K., Vishnevsky, D. Gadylshina, K.  2022\.  Numerical dispersion mitigation neural network for seismic modeling Numerical dispersion mitigation neural network for seismic modeling.  Geophysics87T237–T249.
  * Gao . (2020) gao2020largeGao, Z., Li, C., Liu, N., Pan, Z., Gao, J. Xu, Z.  2020\.  Large-dimensional seismic inversion using global optimization with autoencoder-based model dimensionality reduction Large-dimensional seismic inversion using global optimization with autoencoder-based model dimensionality reduction.  IEEE Transactions on Geoscience and Remote Sensing5921718–1732.
  * Gao . (2022) gao2022globalGao, Z., Yang, W., Tian, Y., Li, C., Jiang, X., Gao, J. Xu, Z.  2022\.  Global optimization with deep-learning-based acceleration surrogate for large-scale seismic acoustic-impedance inversion Global optimization with deep-learning-based acceleration surrogate for large-scale seismic acoustic-impedance inversion.  Geophysics87R35–R51.  10.1190/geo2021-0477.1
  * Gelboim . (2022) gelboim2022encoderGelboim, M., Adler, A., Sun, Y. Araya-Polo, M.  2022\.  Encoder–Decoder Architecture for 3D Seismic Inversion Encoder–decoder architecture for 3d seismic inversion.  Sensors23161.
  * Gholami (2015) gholami2015nonlinearGholami, A.  2015\.  Nonlinear multichannel impedance inversion by total-variation regularization Nonlinear multichannel impedance inversion by total-variation regularization.  Geophysics80R217–R224.
  * Gregor LeCun (2010) gregor2010learningGregor, K. LeCun, Y.  2010\.  Learning fast approximations of sparse coding Learning fast approximations of sparse coding..  Presented at the 27th Int. Conf. Machine Learning
  * He . (2022) he2022swinHe, X., Zhou, Y., Zhao, J., Zhang, D., Yao, R. Xue, Y.  2022\.  Swin Transformer Embedding UNet for Remote Sensing Image Semantic Segmentation Swin transformer embedding unet for remote sensing image semantic segmentation.  IEEE TGRS601–15.
  * Jiang . (2022) jiang2022convolutionalJiang, J., Ren, H. Zhang, M.  2022\.  A Convolutional Autoencoder Method for Simultaneous Seismic Data Reconstruction and Denoising A convolutional autoencoder method for simultaneous seismic data reconstruction and denoising.  IEEE GRSL191–5.
  * Jin . (2019) jin2019dunetJin, Q., Meng, Z., Pham, TD., Chen, Q., Wei, L. Su, R.  2019\.  DUNet: A deformable network for retinal vessel segmentation Dunet: A deformable network for retinal vessel segmentation.  Knowledge-Based Systems178149–162.
  * Karpatne . (2018) karpatne2018machineKarpatne, A., Ebert-Uphoff, I., Ravela, S., Babaie, HA. Kumar, V.  2018\.  Machine learning for the geosciences: Challenges and opportunities Machine learning for the geosciences: Challenges and opportunities.  IEEE Transactions on Knowledge and Data Engineering3181544–1554.
  * LeCun . (2015) lecun2015deepLeCun, Y., Bengio, Y. Hinton, G.  2015\.  Deep learning Deep learning.  nature5217553436–444.
  * S. Li . (2019) li2019deepLi, S., Liu, B., Ren, Y., Chen, Y., Yang, S., Wang, Y. Jiang, P.  2019\.  Deep-learning inversion of seismic data Deep-learning inversion of seismic data.  arXiv preprint arXiv:1901.07733.
  * X. Li . (2021) li2021consecutivelyLi, X., Wu, B., Zhu, X. Yang, H.  2021\.  Consecutively missing seismic data interpolation based on coordinate attention unet Consecutively missing seismic data interpolation based on coordinate attention unet.  IEEE Geoscience and Remote Sensing Letters191–5.
  * Lin Huang (2013) lin2013ultrasoundLin, Y. Huang, L.  2013\.  Ultrasound waveform tomography with a modified total-variation regularization scheme Ultrasound waveform tomography with a modified total-variation regularization scheme.  Proceedings of the SPIE Medical Imaging Conference. Proceedings of the spie medical imaging conference.
  * Liu . (2021) liu2021deepLiu, B., Yang, S., Ren, Y., Xu, X., Jiang, P. Chen, Y.  2021\.  Deep-learning seismic full-waveform inversion for realistic structural models Deep-learning seismic full-waveform inversion for realistic structural models.  Geophysics861R31–R44.
  * Mosser . (2018) mosser2018rapidMosser, L., Kimman, W., Dramsch, J., Purves, S., De la Fuente Briceño, A. Ganssle, G.  2018\.  Rapid seismic domain transfer: Seismic velocity inversion and modeling using deep generative neural networks Rapid seismic domain transfer: Seismic velocity inversion and modeling using deep generative neural networks.  80th eage conference and exhibition 2018 80th eage conference and exhibition 2018 ( 2018, 1–5).
  * Ning . (2023) ning2023multichannelNing, J., Li, S., Wei, Z. Yang, X.  2023\.  Multichannel seismic impedance inversion based on Attention U-Net Multichannel seismic impedance inversion based on attention u-net.  Frontiers in Earth Science111104488.
  * Ronneberger . (2015) ronneberger2015uRonneberger, O., Fischer, P. Brox, T.  2015October 8-12, 2023.  U-net: Convolutional networks for biomedical image segmentation U-net: Convolutional networks for biomedical image segmentation.  International Conference on Medical Image Computing and Computer-Assisted Intervention. International conference on medical image computing and computer-assisted intervention.  Germany.
  * Russell (1988) russell1988introductionRussell, BH.  1988\.  Introduction to seismic inversion methods Introduction to seismic inversion methods ( 2).  SEG Books.
  * Sen (2006) sen2006seismicSen, MK.  2006\.  Seismic inversion Seismic inversion.  Society of Petroleum Engineers Richardson, TX.
  * Shuey (1985) shuey1985simplificationShuey, RT.  1985\.  A simplification of the Zoeppritz equations A simplification of the zoeppritz equations.  Geophysics50609–614.
  * Sun . (2021) sun2021physicsSun, J., Innanen, KA. Huang, C.  2021\.  Physics-guided deep learning for seismic inversion with hybrid training and uncertainty analysis Physics-guided deep learning for seismic inversion with hybrid training and uncertainty analysis.  Geophysics863R303–R317.
  * Sun . (2020) sun2020theorySun, J., Niu, Z., Innanen, KA., Li, J. Trad, DO.  2020\.  A theory-guided deep-learning formulation and optimization of seismic waveform inversion Theory-guided DL and seismic inversion A theory-guided deep-learning formulation and optimization of seismic waveform inversion theory-guided dl and seismic inversion.  Geophysics852R87–R99.
  * Tao . (2023) tao2023acousticTao, L., Ren, H. Gu, Z.  2023\.  Acoustic Impedance Inversion from Seismic Imaging Profiles Using Self Attention U-Net Acoustic impedance inversion from seismic imaging profiles using self attention u-net.  Remote Sensing154891.
  * Tao . (2022) tao2022seismicTao, L., Ren, H., Ye, Y. Jiang, J.  2022\.  Seismic Surface-Related Multiples Suppression Based on SAGAN Seismic surface-related multiples suppression based on sagan.  IEEE GRSL193006605.
  * Tran . (2020) tran20203dTran, KT., Nguyen, TD., Hiltunen, DR., Stokoe, K. Menq, F.  2020\.  3D full-waveform inversion in time-frequency domain: Field data application 3d full-waveform inversion in time-frequency domain: Field data application.  Journal of Applied Geophysics178104078.
  * ul Islam (2020) ul2020usingul Islam, MS.  2020\.  Using deep learning based methods to classify salt bodies in seismic images Using deep learning based methods to classify salt bodies in seismic images.  Journal of Applied Geophysics178104054.
  * C. Wang . (2019) wang2019computationalWang, C., Zhang, D., Zhang, J. Liu, H.  2019\.  Computational seismic data processing with neural networks and sparse regularization Computational seismic data processing with neural networks and sparse regularization.  Geophysics832V105–V116.
  * Q. Wang . (2022) wang2022seismicWang, Q., Wang, Y., Ao, Y. Lu, W.  2022\.  Seismic inversion based on 2D-CNNs and domain adaption Seismic inversion based on 2d-cnns and domain adaption.  IEEE Transactions on Geoscience and Remote Sensing601–12.
  * Y. Wang . (2022) wang2022physicsWang, Y., Wang, Q., Lu, W. Li, H.  2022\.  Physics-Constrained Seismic Impedance Inversion Based on Deep Learning Physics-constrained seismic impedance inversion based on deep learning.  IEEE GRSL197503305.
  * B. Wu . (2020) wu2020seismicWu, B., Meng, D., Wang, L., Liu, N. Wang, Y.  2020\.  Seismic impedance inversion using fully convolutional residual network and transfer learning Seismic impedance inversion using fully convolutional residual network and transfer learning.  IEEE Geoscience and Remote Sensing Letters172140–2144.  10.1109/LGRS.2019.2963106
  * X. Wu . (2019) wu2019faultnet3dWu, X., Shi, Y., Fomel, S., Liang, L., Zhang, Q. Yusifov, AZ.  2019\.  FaultNet3D: Predicting fault probabilities, strikes, and dips with a single convolutional neural network Faultnet3d: Predicting fault probabilities, strikes, and dips with a single convolutional neural network.  IEEE Transactions on Geoscience and Remote Sensing57119138–9155.
  * X. Wu . (2021) wu2021deepWu, X., Yan, S., Bi, Z., Zhang, S. Si, H.  2021\.  Deep learning for multidimensional seismic impedance inversion Deep learning for multidimensional seismic impedance inversion.  Geophysics86R735–R745.  10.1190/geo2020-0236.1
  * Y. Wu Lin (2018) wu2018inversionnetWu, Y. Lin, Y.  2018\.  InversionNet: A real-time and accurate full waveform inversion with CNNs and continuous CRFs Inversionnet: A real-time and accurate full waveform inversion with cnns and continuous crfs.  arXiv preprint arXiv:1811.07875.
  * Y. Wu Lin (2019) wu2019inversionnetWu, Y. Lin, Y.  2019\.  InversionNet: An efficient and accurate data-driven full waveform inversion Inversionnet: An efficient and accurate data-driven full waveform inversion.  IEEE Transactions on Computational Imaging6419–433.
  * Yuan . (2019) yuan2019impedanceYuan, S., Wang, S., Luo, Y., Wei, W. Wang, G.  2019\.  Impedance inversion by using the low-frequency full-waveform inversion result as an a priori model Impedance inversion by using the low-frequency full-waveform inversion result as an a priori model.  Geophysics82R149–R164.  10.1190/geo2018-0242.1
  * R. Zhang Castagna (2011) zhang2011Zhang, R. Castagna, J.  2011\.  Seismic sparse-layer reflectivity inversion using basis pursuit decomposition Seismic sparse-layer reflectivity inversion using basis pursuit decomposition.  Geophysics76R147–R158.  10.1190/geo2011-0103.1
  * SB. Zhang . (2022) zhang2022comparisonZhang, SB., Si, HJ., Wu, XM. Yan, SS.  2022\.  A comparison of deep learning methods for seismic impedance inversion A comparison of deep learning methods for seismic impedance inversion.  Petroleum Science1931019–1030.
  * Zhu . (2023) zhu2023fourierZhu, M., Feng, S., Lin, Y. Lu, L.  2023\.  Fourier-DeepONet: Fourier-enhanced deep operator networks for full waveform inversion with improved accuracy, generalizability, and robustness Fourier-deeponet: Fourier-enhanced deep operator networks for full waveform inversion with improved accuracy, generalizability, and robustness.  arXiv preprint arXiv:2305.17289.


11institutetext:  Department of Language Science and Technology, Saarland
University, Germany
11email<EMAIL_ADDRESS>22institutetext: Smart Data Analytics
Group (SDA), University of Bonn, Germany
22email: {dubey<EMAIL_ADDRESS>33institutetext: Fraunhofer
IAIS, Dresden, Germany
33email<EMAIL_ADDRESS>
# VANiLLa : Verbalized Answers in
Natural Language at Large Scale

Debanjali Biswas Equal contribution11 Mohnish Dubey * 2233
Md Rashad Al Hasan Rony 33 Jens Lehmann 2233

###### Abstract

In the last years, there have been significant developments in the area of
Question Answering over Knowledge Graphs (KGQA). Despite all the notable
advancements, current KGQA datasets only provide the answers as the direct
output result of the formal query, rather than full sentences incorporating
question context. For achieving coherent answers sentence with the question’s
vocabulary, template-based verbalization so are usually employed for a better
representation of answers, which in turn require extensive expert
intervention. Thus, making way for machine learning approaches; however, there
is a scarcity of datasets that empower machine learning models in this area.
Hence, we provide the VANiLLa dataset which aims at reducing this gap by
offering answers in natural language sentences. The answer sentences in this
dataset are syntactically and semantically closer to the question than to the
triple fact. Our dataset consists of over 100k simple questions adapted from
the CSQA and SimpleQuestionsWikidata datasets and generated using a semi-
automatic framework. We also present results of training our dataset on
multiple baseline models adapted from current state-of-the-art Natural
Language Generation (NLG) architectures. We believe that this dataset will
allow researchers to focus on finding suitable methodologies and architectures
for answer verbalization.

###### Keywords:

Answer Representation Natural Language Generation Dataset

Resource Type: Dataset
Website and documentation: https://sda.tech/projects/vanilla/
Permanent URL: https://figshare.com/articles/Vanilla_dataset/12360743

##  1 Introduction

The goal of Question Answering over Knowledge Graphs (KGQA) is to retrieve the
answer from a knowledge graph (KG) for a given natural language question. A
conventional solution to KGQA is to translate natural language questions to a
formal query, which then returns the intended answer when executed over the
KG. The primary KGQA data sets, such as SimpleQuestion [2], WebQuestions [28],
LC-QuAD [24] and KGQA systems [12] consider the formal query response as the
final answer to the question. KGQA challenges such as QALD [25] consider the
formal query or their response for evaluations. However, these responses might
leave the final end-user unsatisfied with the answer representation.

Figure 1: Examples from Vanilla Dataset

For example,for the question $Q_{NL}$ “Whose girlfriend was Suzi Perry?”, a
KGQA system could answer the question based on the KG triple $T$ <Suzi Perry>
<unmarried partner> <Nicki Knowles>. The formal query (SPARQL) $Q_{NL}$ for
this question could be ‘SELECT ?ans WHERE {<Suzi Perry> <unmarried partner>
?ans}’ 111equivalent wikidata SPARQL is ‘SELECT ?ans WHERE { wd:Q7651082
wdt:P451 ?ans}’ , and the final answer retrieved $A_{R}$ would be ‘Nicki
Knowles’. If Question Answering system further verbalizes the answers to ‘Suzi
Perry was girlfriend of Nicki Knowles. ‘ by incorporating $Q_{NL}$ context,
the end-user will feel more satisfied with answer as a sentence when compared
to receive the answer just as a date.

A simple solution for verbalising answers is to design a template-based
solution. However, the template solution can be expensive as it requires field
experts to create the template. Moreover, the coverage is limited to queries
matching the provided templates and the templates may be too inflexible to
take particular grammar constructs, aggregate queries and context into
account. In particular in multi-turn dialogues, templates can be repetitive
and lead to unnatural conversations.

For this reason, machine learning techniques have been investigated for answer
verbalization. These techniques, in turn, depend primarily on large-scale
training data. At present, the datasets that can be applied for answer
verbalization tasks are mostly RDF verbalization datasets like WebNLG [5].
Whereas in VQuAnDa dataset [14] (we discuss this dataset in detail in section
4), they verbalized the answer based on the SPARQL query. Using such RDF/Query
verbalization technique, we can verbalize the triple, but they might be not
apt for the answer representation as they do not take the question context
into account. As an example, consider the upper question in Figure 1. In this
case, the triple verbalization $A_{T}$ ‘Suzi Perry is the unmarried partner of
Nicki Knowles.’ is less suitable than the human like response "Suzi Perry was
girlfriend of Nicki Knowles", which uses the word "girlfriend" from the user
query. In order to spur research into such context-driven answer
verbalizations for KGQA, we present the VANiLLa: ‘Verbalized Answers in
Natural Language at Large scale‘ where the verbalized answer sentences are
generated by using the context of question to encourage human-like responses.

The following are key contributions of this work:

  * •

Provision of a large-scale dataset of over 100k questions along with their
corresponding verbalized answers.

  * •

We present a cost-efficient semi-automatic framework for generating large-
scale answer verbalization datasets, which was also used to generate VANiLLa.

  * •

We provide multiple baseline models based on the conventional sequence-to-
sequence framework as reference for future research.

  * •

We also provide empirical proof that answer verbalizations are better suited
response and provide more context compared to RDF triple verbalizations.

This paper is organised into the following sections:  (2) Problem Statement,
(3) Dataset Generation Workflow,  (4) Characteristics of the Dataset,  (5)
Evaluation results,  (6) Potential impact of this datset along with some
Related Work, and  (7) Conclusion and Future Work.

##  2 Problem Statement

Formally, KGQA solutions are based on the semantic matching of a Natural
Language Question $Q_{NL}$ to KG triples. The goal is usually to generate
Formal expression $Q_{FL}$, such that $Q_{NL}$ and $Q_{FL}$ have an equivalent
semantic interpretation. Results retrieved through a formal query (often
SPARQL in KGQA) are often either entities, literals or boolean values (which
is a special case of a literal that we treat separately) or a list of the
previous items. Giving these as response directly to a user question is
sufficient to answer the question. However, the responses are often not human
friendly and it is difficult for a user to assess whether the question was
correctly understood. Thus, the need for verbalizing answers in natural
language in two fold:  (i) to provide some context of the question in the
answer to enable users to verify the provided answer [7], and  (ii) to mimic
natural human conversations.  From this perspective, an extended model to
generate the final representation of answer in the QA pipeline is recommended
as shown in Figure: 2.

In this model, given a language question $Q_{NL}$ KGQA aims to retrieve a
natural language answer $A_{NL}$ to model a transformation function $f_{kgqa}$
such that :

$f_{kgqa}:Q_{NL}\mapsto A_{NL}$
$(Q_{NL})^{C_{q}}\equiv(A_{NL})^{C_{a}}$

$C_{q}$ is the linguistic and semantic context of the natural language
question
$C_{a}$ is the linguistic and semantic context of the natural language answer

Within the scope of this paper, we do not retrieve an answer from a KG for a
given question. Rather, we focus on verbalizing the answers retrieved through
the $Q_{FL}$ corresponding to the $Q_{NL}$. We describe this model as follows:

Given a user question $Q_{NL}$ and an answer retrieved $A_{R}$ by a QA system
using $Q_{FL}$. The task of the model $f$ is to generate verbalized answer
$A_{V}$ such that the context of the question is preserved and we can obtained
a more human-like response. Formally,

$A_{V}=f(Q_{NL},A_{R})$ (1)

Figure 2: The QA pipeline for generating the answer in natural language

We further introduce our hypothesis regarding the semantic closeness of
natural language question(input to KGQA) and natural language answer(output of
KGQA).

####  2.0.1 Hypothesis H1:

For a given question $Q_{NL}$ and the retrieved answer $A_{R}$ from KG triples
$T$, the verbalized answer $A_{V}$ with question context is semantically and
syntactically closer to the question $Q_{NL}$ in comparison with that of the
KG triple verbalization $A_{T}$ and $Q_{NL}$. Formally,

$Sim(A_{V},Q_{NL})>Sim(A_{T},Q_{NL})$ (2)

Similarly, for a given question $Q_{NL}$ and the retrieved answer $A_{R}$ from
KG triples $T$, the RDF verbalization $A_{T}$ is semantically and
syntactically closer to the triple $T$ in comparison with that of the answer
verbalization $A_{V}$ and $T$. Formally,

$Sim(A_{T},T)>Sim(A_{V},T)$ (3)

where, $Sim$ is the similarity measure.

For example, in context of equation 2, Similarity ($Sim$) between ‘Brad Pitt
was born in Shawnee’ and ‘Where was Brad Pitt born?’ is greater then
Similarity between ‘The place of birth of Brad Pitt is Shawnee.’ and ‘Where
was Brad Pitt born?’. We empirically prove this hypothesis with Experiment 1
in Section: 5.1.

##  3 Dataset Design and Characteristics

###  3.1 Dataset Generation Workflow

Our dataset, VANiLLa (Verbalized Answers in Natural Language at Large scale)
consists of simple questions $Q_{NL}$, selected from the CSQA [21] and
SimpleQuestionsWikidata [8] datasets and answers $A_{R}$, labels fetched from
the Wikidata [27] Knowledge Graph, along with their answers in natural
language $A_{V}$, generated using Amazon Mechanical Turk (AMT). We adapt our
dataset from conventional Question Answering over Knowledge Graph (KGQA)
datasets on Wikidata for the purpose of offering the possibility of extending
them with verbalizations. The dataset creation process workflow uses a semi-
automatic framework, as described in Figure 3, consists of four stages:  (i)
Pre-Processing Stage,  (ii) AMT Experiment Stage,  (iii) Reviewer, and  (iv)
Post-Processing Stage.

Figure 3: Dataset Generation Workflow

####  3.1.1 Preprocessing Stage

In this stage, we first collect all simple questions from CSQA222Note: We
choose only direct questions where the KG entity directly exists in the
question and no co-referencing is required. and SimpleQuestionsWikidata
datasets. While filtering questions, we ignore questions which produce more
than one entity as an answer. This filtering process also generates a list of
predicates present in the question for each dataset. On the basis of these
predicates, we group the questions for a particular predicate to form a list.
We then fetch labels for entities and relations present in the question and
the answer from Wikidata. These labels along with the question lists are sent
to the question selected process. The purpose of this pre-processing step is
two fold:  (i) reducing the cost of crowd-sourcing the dataset from scratch,
and  (ii) opening the possibility of extending the CSQA and
SimpleQuestionsWikidata or SimpleQuestions dataset with the verbalized
answers.

####  3.1.2 Question Selection Process

The selection process starts by picking unique questions from the list of
questions for a particular relation or predicate. Only a single instance of
all the similar questions are selected in this process for the AMT experiment,
thereby, reducing the cost of verbalization. Later, we may use the
verbalizations of these selected questions as templates to automatically
generate large scale data without the requirement of manual workforce. As
illustrated in Algorithm 1, the process takes $Q$ as input, which is a list of
all filtered questions for a particular relation and outputs $S$, a list of
selected questions for a particular relation. The first step is to copy the
first question into the selected list $S$. Then we check for the 4-gram
similarity between the other questions in list $Q$ with all the selected
questions in list $S$ and continue adding dissimilar questions into list $S$,
generating patterns of questions for each predicate.

Algorithm 1 VANiLLa: Question Selection Process

Input: $Q=\\{Q_{1},Q_{2},....,Q_{n}\\}$
Output: $S=\\{S_{1},S_{2},....,S_{m}\\}$

1: $S\leftarrow Q_{1}$

2: for all $Q_{i}\in Q$ do

3: Check 4-gram similarity between $Q_{i}$ and $S_{i}$ $\forall S_{i}\in S$

4: if $Q_{i}\,and\,S_{i}\,are\,not\,similar\,\forall\,S_{i}\in S$ then

5: Add $Q_{i}$ to $S$

6: end if

7: end for

To check the similarity between two questions, we first mask the entities in
both the questions and then find 4-grams for both questions. If any of the
4-grams match, then the questions are similar, otherwise they are dissimilar.
For example, $S_{1}$ is "Who is the wife of Barack Obama?" and $Q_{1}$ is "Who
is the wife of Donald Trump?". The 4-grams for $S_{1}$ and $Q_{1}$ are {"who",
"is", "the", "wife"} and {"is", "the", "wife", "of"}. Hence, $S_{1}$ and
$Q_{1}$ are similar. Now, if we check $S_{1}$ with another question say
$Q_{2}$, which is "Who is married to Barack Obama?". The 4-grams for $Q_{2}$
are {"who", "is", "married", "to"} and do not match with $S_{1}$ . Hence, they
are dissimilar. Therefore, $Q_{2}$ is added into the selected list $S$.

####  3.1.3 AMT Experiment Stage

In this stage, the human workers or turkers working on AMT are provided with a
question $Q_{NL}$ from the selected list $S$ and its answer label $A_{R}$
retrieved from Wikidata [8]. The workers are instructed to rewrite the answers
in natural language sentences so as to generate the answer sentence $A_{V}$
without re-writing the answer label. Thus, the crowd-sourcing task aims to
verbalize the answers in natural language by incorporating the context of
question without seeing the knowledge graph triples. The turkers are provided
with explicit directions to verbalize the answer using the context of the
question $Q_{NL}$. They are also presented with adequate examples to
understand the task well.

####  3.1.4 Reviewer

A manual reviewing step is implemented to check for grammatical errors and
invalid answers verbalized in the AMT stage. One native English speaking
reviewer along with the help of the grammar checking
tool333https://app.grammarly.com/, made the necessary corrections.

####  3.1.5 Post-Processing Stage

The last stage of the workflow yields a cost-effective extension of the
dataset by utilizing the set of verbalizations received from the previous
stage. The Dataset Extension Process searches for all similar question for
every question in the verbalized dataset using 4-gram similarity and then
calls the Answer Generation Process which in turn is employing the verbalized
answer from the AMT experiment as a template to generate the answer in natural
language for all the other similar questions. Due to these actions, we can
provide a large-scale dataset with minimal cost.

###  3.2 Dataset Characteristics

Our dataset VANiLLa consists of a total of 107166 simple questions, with 85732
in training set and 21434 in test set. One significant feature of our dataset
is that it is KG-independent and hence flexible for training verbalization
models for any KG. VANiLLa covers a wide range of relations (300+) providing a
significant amount of variations to the dataset. The different tags of the
dataset are:

  * •

question-id: an unique identification number for a dataset instance

  * •

question: question $Q_{NL}$

  * •

answer: retrieved answer $A_{R}$

  * •

answer_sentence: verbalized answer in natural language $A_{V}$

However, some drawbacks of our dataset include the non-availability of
multiple answer type questions, complex questions and paraphrased questions.

###  3.3 Reusability

Our dataset can be used in multiple research areas. The straight forward
utilization is in the domain of KGQA pipeline where the final representation
of the answer can be modified into a natural language sentence. This also
applies for Dialog systems where the user not only expects more human-like
conversation but also a scope for answer verification from the question’s
point of view and for continuing the conversation. The dataset is KG
independent, hence there is a possibility of using it for text based QA
systems as well to generate answers in natural language. Also, our semi-
automatic framework can be used for a cost-effective extension of the dataset.
In addition, with some further investigation our dataset can be adopted in the
form of multi-turns for assisting conversation type QA systems.

###  3.4 Availability and Sustainability

We have published the dataset at
figshare444https://figshare.com/articles/Vanilla_dataset/12360743 under CC BY
4.0555https://creativecommons.org/licenses/by/4.0/ license to support
sustainability. Figshare ensures data persistence and public availability,
whereby guaranteeing all time accessibility of the dataset irrespective of the
running status of our servers. The figshare project includes the training and
the test sets of our dataset.

Our baseline models and data creation framework is readily available as a open
source repository666https://github.com/AskNowQA/VANiLLa under a GPL
3.0777https://www.gnu.org/licenses/gpl-3.0.html license. We intend to actively
track feature requests and bug reports using our Github repository.

##  4 Potential Impact and Related Work

To understand the impact of our dataset we first showcase the previous works
that have been done in this domain. We report some of the resources which are
currently available for verbalizing answers. GENQA [29] and COREQA [11] are
two datasets in Chinese language which provide answers in the form of
sentences rather than a concise answer. To the best of our knowledge, VQuAnDa
[14] is the first and only dataset in English language to provide answer
verbalization. The dataset contains questions from LC-QuAD [24] on DBpedia as
the target KG. The advantage of this dataset is doubled due to the
availability of SPARQL representation for questions. Thus, it is also useful
for model’s verbalizing answers from a SPARQL query. The dataset was aimed to
enable users to validate the information generated from QA systems by
incorporating not only the retrieved information but also additional
characteristics indicating the process of answer retrieval. Nevertheless, this
dataset does not cover key features of the user question to mimic human
conversation in addition to enabling answer validation.

VANiLLa is a large scale dataset with over 100k question answer pairs compared
to the 5k pairs in the VQuAnDa dataset. VQuAnDa aims to generate
verbalizations using the SPARQL query and find the position of the answer in
the natural language sentence hence in return has to implement a slot filling
approach to achieve the final response. Whereas VANiLLa promotes an end-to-end
approach by avoiding any post processing of the generated response. The
VQuAnDa dataset generation requires a SPARQL query, whereas the VANiLLa
generation workflow only requires answers. This showcases that Vanilla’s
architecture is KG-independent, and one could extend the VANiLLa dataset
generation workflow to a QA dataset with no KG or formal query in the
background. In this regard, our dataset would impact the following sectors of
the NLP community:

  * Natural Language Generation:

Popular NLG tasks and dataset on RDF verbalzation, such as WebNLG[5], do not
take into consideration the question elements when reconstructing the facts in
natural language. Hence, the sentences generated involve relevant information
from the facts without any context of the question. From this standpoint, our
dataset would open up new paradigm in NLG research.

  * Question Answering and Dialog Systems:

Different QA datasets based on Text, such as SQuAD [19] and NaturalQuestions
[15], and KG, such as SimpleQuestion [2], WebQuestionSP [28],
ComplexWebQuestions [23], QALD-9 [25], LC-QuAD 1.0 [24], LC-QuAD 2.0 [9],
exists. Furthermore, datasets are also adapted into conversation or dialog
format with multiple utterances such as, QuAC [4], CoQA [20], CSQA [21], SQA
[13]. Recent trend in QA datasets are based on the following aspects:  (1)
expanding the size of dataset,  (2) extending the dataset with more variations
in question type such as aggregations, boolean queries, etc.,  (3) enhancing
question complexity using compound features like comparison or unions,  (4)
enhancements covering multiple utterances in the form of conversations.

At present, the datasets associated with the task of QA or dialog systems
provide concise or to-the-point answers for questions posed by users. However,
verbalization of answers, which enable a more human-like response by
incorporating additional information, are neglected in these datasets. From
this viewpoint, VANiLLa will provide a more contextual and human-like response
to user questions, using elements from the question to generate the verbalized
answers. Furthermore, it is beneficial in dialog systems to have a continuous
flow in the conversation, by enabling users with more content for the next
utterance.

##  5 Evaluation

In this section, we report the different experiments we have performed for the
assessment of the quality of our dataset and for comparing the performance of
our baseline models.

###  5.1 Experiment: 1

####  5.1.1 Objective:

A more human-like response to a user question $Q_{NL}$ would mean that the
generated response in natural language provides some context of the question.
From this point of view, we perform this Experiment to show that the
verbalized answers $A_{V}$ in our dataset preserves the semantic and syntactic
meaning of the question more compared to the relevant RDF triples
verbalization $A_{T}$. Additionally, we also indicate that a response
generated by verbalizing the RDF triple $A_{T}$ would be semantically and
syntactically more close to the $T$ in contrast to the that of $A_{V}$ and
$T$. Concisely, we evaluate hypothesis H1 from section: 2.0.1.

####  5.1.2 Experimental Settings:

We randomly sample 500 questions from our dataset to perform this experiment.
For similarity comparison purposes with the natural language answers in our
dataset and user question, we generated the verbalization of the RDF triples
associated with these questions with the help of a simple AMT experiment where
the turkers were asked to rewrite the triples in the form of natural language
sentences without any information about the questions.

To analyze the quality of our dataset, we calculate the similarity measure
$Sim$:  (i) as the Levenshtein distance using Fuzzywuzzy, denoting the
"syntactic similarity" and  (ii) as the cosine vector similarity using BERT
embeddings, denoting the "semantic similarity",  of $T$ and $Q_{NL}$ with both
$A_{V}$ and $A_{T}$. We record the mean $\mu$ and standard deviation $\sigma$
for all the comparisons over the randomly sampled data.

---
Figure 4: Syntactic and Semantic Comparison of $T$ and $Q_{NL}$ with both
$A_{V}$ and $A_{T}$

####  5.1.3 Experimental Results:

The results of our experiment are depicted in Table: 1. On the basis of
Levenshtein distance, we observe that the syntactic similarity between the
verbalized RDF triple $A_{T}$ and the RDF triple $T$ is more than the
similarity between $T$ and the verbalized answer $A_{V}$, i.e.,
$Sim(A_{T},T)>Sim(A_{V},T)$. In contrary, the syntactic similarity between
$A_{V}$ and the user question $Q_{NL}$ is more than the similarity $Q_{NL}$
and the verbalized RDF triple $A_{T}$, i.e.,
$Sim(A_{V},Q_{NL})>Sim(A_{T},Q_{NL})$. We also observe the same in the case of
the cosine similarity measure. Thus, empirically proving hypothesis H1.

|  |  |  | Levenshtein Ratio |  | Cosine-Similarity
---|---|---|---|---|---|---
|  |  |  | $\mathbf{A_{T}}$ | $\mathbf{A_{V}}$ |  | $\mathbf{A_{T}}$ | $\mathbf{A_{V}}$
$\mathbf{T}$ |  | $\mu$ |  | 0.8602 | 0.6139 |  | 0.8441 | 0.7965
| $\sigma$ |  | 0.1066 | 0.1779 |  | 0.0793 | 0.0992
$\mathbf{Q_{NL}}$ |  | $\mu$ |  | 0.4750 | 0.6519 |  | 0.3528 | 0.3574
| $\sigma$ |  | 0.1018 | 0.1504 |  | 0.1064 | 0.1056
Table 1: Comparing $A_{V}$, $A_{T}$, $Q_{NL}$ and $T$ based on semantic cosine
vector and syntactic Levenshtein distance similarity

Additionally, Figure: 4 illustrates the results of the experiment calculating
the Levenshtein distances. In the first plot, we compare the Levenshtein
distance between ($T$, $A_{T}$) and ($T$, $A_{V}$) over the 500 randomly
sampled questions. The plot clearly depicts that the purple line indicating
the distance between ($T$, $A_{T}$) is higher than the green line indicating
the distance between ($T$, $A_{V}$) at most data point, thus demonstrating
that the pair ($T$, $A_{V}$) is syntactically less similar than the ($T$,
$A_{T}$). Furthermore in the second plot, we compare the Levenshtein distance
between ($Q_{NL}$, $A_{T}$) and ($Q_{NL}$, $A_{V}$), which clearly depicts the
red line indicating the distance between ($Q_{NL}$, $A_{V}$) is higher than
the blue line indicating the distance between ($Q_{NL}$, $A_{T}$). Thus,
verifying that $A_{V}$ is syntactically more similar to $Q_{NL}$ than $A_{T}$.
However, in both case there are a few points of exception where the similarity
measures are the same. This is due to questions in the dataset which contain
same surface labels as in the RDF triples.

###  5.2 Experiment: 2

####  5.2.1 Objective:

The objective of this experiment is to record the performance of our baseline
models on VANiLLa dataset based on certain evaluation criteria.

####  5.2.2 Baseline models:

This section presents an overview of our baseline models used for evaluating
and assessing the quality of the data. We decided to use some conventional
sequence-to-sequence models following the underlying Encoder-Decoder pipeline
[22, 3]. The question $Q_{NL}$ and the retrieved answer $A_{T}$ are separated
with the separation "$<$sep$>$" tag to form the input to the encoder and the
decoder produces the verbalized answer in natural language $A_{V}$ as the
output. The inputs to both the encoder and the decoder end with the end-of-
sequence "$<$eos$>$" tag and the output from the decoder is appended with the
start-of-sequence "$<$sos$>$" tag before the start of prediction. The four
baseline models are as follows:

  * Seq2Seq model with attention mechanism:

Sequence-to-Sequence model with attention mechanism [1] proposed by Bahdanau
simultaneously learns to align and translate by employing a bi-directional RNN
as the encoder and using a weighted sum of the vector representation generated
at the encoder level as the context in the decoder. These sequence-to-sequence
models have been widely used in the NLP community for tasks such as Machine
Translation, Question Answering, etc.

  * Convolution based Encoder-Decoder model:

Facebook AI introduced the use of CNN in sequence learning [10]. The concept
is to implement a fully convolutional architecture, by incorporating CNN at
both encoder and decoder level. These models implement a position embedding,
which provides an understanding about the portion of input or output sequence
currently in consideration, along with a multi-step attention.

  * Transformer:

Transformers are novel architectures that solves sequence-to-sequence tasks
quite nicely because of its ability to handle long-term dependencies.
Recently, Transformers achieved state-of-the art in wide range of NLP tasks
such as Natural Language Generation, Machine Translation. The architecture of
transformer was first proposed by Vaswani et al. [26]. Followed by that many
pre-trained transformer models are proposed such as BERT [6], GPT [18],
RoBERTa [16], which are trained on a large number of data. For simplicity, we
use a simple transformer model for the evaluation of our dataset. Multi-headed
attention and positional-encoding are used in this transformer model. Multi-
headed attention helps to capture global dependencies between input and output
and positional-encoding encodes information about the relative or absolute
position of each token in the sequence.

####  5.2.3 Dataset:

We use our dataset VANiLLa with a 80-20 split to generate the training and
test sets. The training set consists of 85732 examples whereas the test set
consists of 21434 examples. We also implement a $k$-cross validation with
$k=5$ on training set for generating the validation set in both cases.

####  5.2.4 Experimental Settings:

We use standard dropout value of 0.5 for the recurrent layers, 0.25 for
convolution layers and 0.1 for multi-head attention mechanism. The hidden
layers dimensions for transformers is set to 200 with embedding dimension of
200, while for the other baselines are set the hidden dimension to 512 and
embedding dimension to 300. In the case of Convolution based Encoder Decoder
model, we utilize 10 layers of convolutions with kernel size 3. For the multi-
headed attention mechanism in Transformer model, we set 8 as the number of
head with 4 layers. For the sake of consistency, we train all the models for 7
epochs on the our dataset, with Adam optimizer and cross entropy loss
function.

####  5.2.5 Evaluation Criteria:

We implement the following criteria for the evaluation and comparison of our
baseline models:

  * Criteria#1

Perplexity: Perplexity defines the quality of predicting real sentences for a
probability model. A good model will have a lower perplexity indicating a high
probability of prediction. It is the average branching factor for predicting
the next word. It is calculated with the help of cross-entropy as follows:

$PPL=2^{H(p,q)}$ (4)

where, $H(p,q)$ is the cross-entropy of the model which tries to learn the
probability distribution $q$ close to the unknown probability distribution of
the data $p$.

  * Criteria#2

Modified Precision: Modified Precision is the sum of the clipped n-gram counts
of the generated sequence or hypothesis divided by the total number of n-grams
in the hypothesis. The range of the modified precision is between 100 and 0
with 100 being the best score. This scores depicts two aspects:  (i) adequacy
(generated hypothesis sentence contains similar words as that in the reference
sentence), and  (ii) fluency (the n-gram matches between the hypothesis and
reference).

  * Criteria#3

BLEU: The BLEU score, Bilingual Evaluation Understudy, is a modified precision
score [17] combined with a brevity penalty for penalizing shorter generation.
Hence, the BLEU score checks for matches in length, order and word choice
between the hypothesis and the references. The range of BLEU score is between
100 and 0 with 100 being the best score.

Figure 5: Perplexity(PPL) per Epoch during training Baseline Model |  | PPL | Precision | BLEU
---|---|---|---|---
Seq2Seq with attention |  | 27.91 | 19.84 | 16.66
CNN Enc-Dec |  | 87.67 | 70.50 | 15.42
Transformer |  | 12.10 | 76.00 | 30.80
Table 2: Evaluating Baseline Models on VANiLLa Dataset (No. of Epochs: 7)

####  5.2.6 Experimental Results:

The perplexity value on the validation set after every epoch for all our
baselines are plotted in Figure: 5 and the test results of the three baseline
models trained on the two sizes of our dataset, VANiLLa has been reported in
table: 2. The results clearly indicate that the transformer model performs the
best among the three baselines with perplexity value of 12.10 and BLEU score
of 30.80.

The transformer model puts attention on the whole sentence instead of
attention on word by word, hence capturing the context better. Additionally,
transformer handles the relationship between words in a sentence by positional
encoding and multi-headed attention mechanism [26]. These characteristics
helps transformer model achieve better result than other baseline models.

###  5.3 Experiment: 3

####  5.3.1 Objective:

Empirically it is evident that the transformer model is the best amongst the
baseline models. Hence, in this experiment we examine the improvement of the
evaluation metrics for the transformer model trained on the combination of
VANiLLa and VQuAnDa [14] datasets (5000 question-answer pairs).

Training Set |  | Test Set |  | PPL | Precision | BLEU
---|---|---|---|---|---|---
VANiLLa + VQuAnDa |  | VQuAnDa |  | 132.26 | 80.45 | 21.61
| VANiLLa |  | 9.05 | 73.63 | 35.70
VQuAnDa |  | VQuAnDa |  | 84.88 | 75.05 | 26.25
Table 3: Evaluating Transformer Model on VANiLLa and VQuAnDa Dataset (No. of
Epochs 50)

####  5.3.2 Experimental Settings:

In this experiment we combine the training data of VANiLLa with the training
data of VQuAnDa dataset and feed it into the transformer model. Although, we
combine the training data for the training phase but we test the model on both
the VANiLLa and VQuAnDa separately. Note that the majority of the combined
data is from VANiLLa as the number of data in VANiLLa is 20 times more than
VQuAnDa.

####  5.3.3 Experimental Results:

The combined data used for training improves the precision score (80.45) of
VQuAnDa test data by a fair margin. Additionally, transformer model achieves
best BLEU score on VANiLLa test set. So, it is evident that this combined data
training approach helps in creating a strong baseline for future research.
Furthermore, our baseline transformer model with the suggested parameters
(described in the experimental settings) achieve SOTA in BLEU score on VQuAnDa
data which is 26.25 where in VQuAnDa paper BLEU is reported 18.18.

##  6 Conclusion and Future Work

We introduce the VANiLLa dataset with the intent to support the verbalization
of answers in the KGQA task. The dataset consisting of over 100k examples was
generated semi-automatically by adapting questions from other datasets such as
CSQA and SimpleQuestionsWikidata and then verbalizing the answers by crowd-
sourcing. We empirically show that the verbalizations in our dataset are
syntactically and semantically closer to the question than to the KG triple
labels. Additionally, we evaluate multiple baseline models based on generic
sequence to sequence models already in use in the NLP community. These
baseline models present a reference point for future research. The
experimental results of these baselines indicate that there still exists
substantial room for enhancement for solving the task of verbalizing answers.

One major use case of answer verbalizations are dialogue systems. We believe
that our dataset along with our design framework presents a valuable
contribution in this field. In the future, we plan to continuously aid
researchers with more variations and expansions of our dataset.

## References

  * [1] D. Bahdanau, K. Cho, and Y. Bengio.  Neural machine translation by jointly learning to align and translate, 2014.
  * [2] A. Bordes, N. Usunier, S. Chopra, and J. Weston.  Large-scale simple question answering with memory networks, 2015.
  * [3] K. Cho, B. van Merrienboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio.  Learning phrase representations using rnn encoder-decoder for statistical machine translation, 2014.
  * [4] E. Choi, H. He, M. Iyyer, M. Yatskar, W.-t. Yih, Y. Choi, P. Liang, and L. Zettlemoyer.  Quac: Question answering in context.  In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2174–2184, 2018.
  * [5] E. Colin, C. Gardent, Y. M’rabet, S. Narayan, and L. Perez-Beltrachini.  The WebNLG challenge: Generating text from DBPedia data.  In Proceedings of the 9th International Natural Language Generation conference, pages 163–167, Edinburgh, UK, Sept. 5-8 2016. Association for Computational Linguistics.
  * [6] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [7] D. Diefenbach, Y. Dridi, K. D. Singh, and P. Maret.  Sparqltouser: Did the question answering system understand me?  In BLINK/NLIWoD3@ISWC, 2017.
  * [8] D. Diefenbach, T. P. Tanon, K. D. Singh, and P. Maret.  Question answering benchmarks for wikidata.  In Proceedings of the ISWC 2017 Posters & Demonstrations and Industry Tracks co-located with 16th International Semantic Web Conference , Vienna, Austria, 2017.
  * [9] M. Dubey, D. Banerjee, A. Abdelkawi, and J. Lehmann.  LC-QuAD 2.0: A Large Dataset for Complex Question Answering over Wikidata and DBpedia, pages 69–78.  10 2019.
  * [10] J. Gehring, M. Auli, D. Grangier, D. Yarats, and Y. N. Dauphin.  Convolutional sequence to sequence learning.  In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1243–1252. JMLR. org, 2017.
  * [11] S. He, C. Liu, K. Liu, and J. Zhao.  Generating natural answers by incorporating copying and retrieving mechanisms in sequence-to-sequence learning.  In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 199–208, Vancouver, Canada, July 2017. Association for Computational Linguistics.
  * [12] K. Höffner, S. Walter, E. Marx, R. Usbeck, J. Lehmann, and A. N. Ngomo.  Survey on challenges of question answering in the semantic web.  Semantic Web, 8(6):895–920, 2017.
  * [13] M. Iyyer, W.-t. Yih, and M.-W. Chang.  Search-based neural structured learning for sequential question answering.  In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1821–1831, Vancouver, Canada, July 2017. Association for Computational Linguistics.
  * [14] E. Kacupaj, H. Zafar, M. Maleshkova, and J. Lehmann.  Vquanda: Verbalization question answering dataset.  In European Semantic Web Conference, 2020.
  * [15] T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, M. Kelcey, J. Devlin, K. Lee, K. N. Toutanova, L. Jones, M.-W. Chang, A. Dai, J. Uszkoreit, Q. Le, and S. Petrov.  Natural questions: a benchmark for question answering research.  Transactions of the Association of Computational Linguistics, 2019\.
  * [16] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  arXiv preprint arXiv:1907.11692, 2019.
  * [17] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu.  Bleu: A method for automatic evaluation of machine translation.  In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL, USA, 2002. Association for Computational Linguistics.
  * [18] A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever.  Improving language understanding by generative pre-training.  URL https://s3-us-west-2. amazonaws. com/openai-assets/researchcovers/languageunsupervised/language understanding paper. pdf, 2018.
  * [19] P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang.  Squad: 100,000+ questions for machine comprehension of text.  In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, 2016.
  * [20] S. Reddy, D. Chen, and C. D. Manning.  Coqa: A conversational question answering challenge.  Transactions of the Association for Computational Linguistics, 7:249–266, 2019.
  * [21] A. Saha, V. Pahuja, M. M. Khapra, K. Sankaranarayanan, and S. Chandar.  Complex sequential question answering: Towards learning to converse over linked question answer pairs with a knowledge graph.  2018\.
  * [22] I. Sutskever, O. Vinyals, and Q. V. Le.  Sequence to sequence learning with neural networks.  In Advances in neural information processing systems, pages 3104–3112, 2014.
  * [23] A. Talmor and J. Berant.  The web as a knowledge-base for answering complex questions.  In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 641–651, 2018.
  * [24] P. Trivedi, G. Maheshwari, M. Dubey, and J. Lehmann.  Lc-quad: A corpus for complex question answering over knowledge graphs.  In The Semantic Web - ISWC - 16th International Semantic Web Conference, Vienna, Austria, Proceedings, Part II. Springer, 2017.
  * [25] R. Usbeck, R. Gusmita, M. Saleem, and A.-C. Ngonga Ngomo.  9th challenge on question answering over linked data (qald-9).  11 2018.
  * [26] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin.  Attention is all you need.  In Advances in neural information processing systems, pages 5998–6008, 2017.
  * [27] D. Vrandečić and M. Krötzsch.  Wikidata: A free collaborative knowledge base.  Communications of the ACM, 57:78–85, 2014.
  * [28] W.-t. Yih, M. Richardson, C. Meek, M.-W. Chang, and J. Suh.  The value of semantic parse labeling for knowledge base question answering.  In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 201–206, Berlin, Germany, Aug. 2016. Association for Computational Linguistics.
  * [29] J. Yin, X. Jiang, Z. Lu, L. Shang, H. Li, and X. Li.  Neural generative question answering.  In Proceedings of the Workshop on Human-Computer Question Answering, pages 36–42, San Diego, California, June 2016. Association for Computational Linguistics.