11institutetext: 1Aarhus University, 2University of Copenhagen

# Text-Driven Stylization of Video Objects

Sebastian Loeschcke1 Serge Belongie2 Sagie Benaim2

# Text-Driven Stylization of Video Objects

Sebastian Loeschcke1 Serge Belongie2 Sagie Benaim2

###### Abstract

We tackle the task of stylizing video objects in an intuitive and semantic
manner following a user-specified text prompt. This is a challenging task as
the resulting video must satisfy multiple properties: (1) it has to be
temporally consistent and avoid jittering or similar artifacts, (2) the
resulting stylization must preserve both the global semantics of the object
and its fine-grained details, and (3) it must adhere to the user-specified
text prompt. To this end, our method stylizes an object in a video according
to two target texts. The first target text prompt describes the global
semantics and the second target text prompt describes the local semantics. To
modify the style of an object, we harness the representational power of CLIP
to get a similarity score between (1) the local target text and a set of local
stylized views, and (2) a global target text and a set of stylized global
views. We use a pretrained atlas decomposition network to propagate the edits
in a temporally consistent manner. We demonstrate that our method can generate
consistent style changes over time for a variety of objects and videos, that
adhere to the specification of the target texts. We also show how varying the
specificity of the target texts and augmenting the texts with a set of
prefixes results in stylizations with different levels of detail. Full results
are given on our project webpage: https://sloeschcke.github.io/Text-Driven-
Stylization-of-Video-Objects/.

###### Keywords:

Video Editing, Text-Guided Stylization, CLIP

##  1 Introduction

Manipulating semantic object entities in videos using human instructions
requires skilled workers with domain knowledge. We seek to eliminate these
requirements by specifying a desired edit or stylization through an easy,
intuitive, and semantic user instruction in the form of a text-prompt.

However, manipulating video content semantically is a challenging task. One
challenge is in generating consistent content or style changes in time, that
adhere to the target text specification. Another challenge is to manipulate
the content of an object such that it preserves the content of the original
video and the global semantics while also adhering to fine-grained details in
the target text.

|  |
---|---|---
|  |
Input video | “Swan made out of cactus” | “Swan with crocodile skin”
Figure 1: Two representative video frames and the edited video frames together
with the global target text.

In recent years, advances in computational methods emerged that enable
manipulation of appearances and style in images and allow novice users to
perform realistic image editing. These methods include manipulation tools that
use natural language (text prompts) to express the desired stylization of
images or 3D objects [18, 6]. The text-driven manipulation is facilitated by
recent developments in models for joint embeddings of text and images, e.g the
Contrastive Language Image Pretraining (CLIP [23]) model. Instead of
manipulating images or 3D objects, we use CLIP in the context of video
manipulation. This is not a straightforward task since simply maximizing the
semantic (CLIP-based) similarity between a valid target text and each 2D frame
in the video often leads to degenerate solutions. Also, applying methods for
image manipulation to each frame in a video results in edits that lack
temporal consistency.

The recently introduced Neural Layered Atlases (NLA) work [14], demonstrates
the ability to separate a moving object in a video from its background by
decomposing the video into a set of 2D atlases. Each atlas provides a unified
image representation of an object or background over the video. Edits applied
to the image representation are automatically mapped back to the video in a
temporally consistent manner. However, editing an image still requires manual
effort and editing skills from the user. Another problem with this approach is
that the 2D atlas representation can be hard to edit due to local
deformations.

We propose a method for performing intuitive and consistent video editing with
multiple capabilities by using the representational power of CLIP to express a
desired edit through a text-prompt. An example could be to change the style of
a swan swimming in a lake according to a target text: “A swan with cactus
skin.” Text is easily modifiable and allows users to express complex and
abstract stylizations intuitively. Using text to express edits reduces the
need for manual editing skills and also avoids the problems related to
deformation in the 2D atlas representations. An illustration is shown in Fig.
1.

To apply temporally consistent edits to an object in a video, our method uses
the atlas decomposition method presented in NLA [14]. We train a generator on
a single input video by sampling local and global views of each frame in the
video and applying various augmentations to each view. Our method uses a
global loss that compares each global view with a global target text and a
local loss that compares each local view with a local target text. The global
loss then focuses on the global semantics and the local views focus on the
fine-grained details. To regularize our learning, we use a sparsity loss that
encourages sparse representation and a temporal triplet loss that encourages
frames that are close in time to be similar in CLIP’s embedding space.

We demonstrate that our method results in natural and consistent stylizations
of objects for a diverse set of videos and target texts. We show how varying
the specificity of both the local and global target texts varies the
stylization and how augmenting the target texts with neutral prefixes can
result in more detailed stylizations. We also demonstrate that our global loss
focuses on the global semantics and the local losses on the fine-grained
details.

##  2 Related Work

Our work is related to video editing works and also to text-based stylization
works.

###  2.1 Video editing

Unlike images, editing or stylizing objects in videos requires the ability to
handle temporal consistency. One natural approach is to propagate the edits
from one frame to the next. Video Propagation Networks [10] use a bilateral
network to connect pixels in consecutive frames and an adaption network to
refine the pixels. Other approaches use optical flow to propagate edits made
on a few key-frames [28]. These approaches work well when there is a clear
correspondence between frames, but have difficulties, e.g., when the video
contains occlusions.

To address occlusion challenges, recent work has used deep learning
approaches, e.g. self-supervised methods for learning visual correspondence
from unlabeled videos [9, 29]. Instead, our work uses the representation
proposed by Neural Layered Atlases (NLA) [14], which decomposes a video into a
set of layered 2D atlases. Each atlas provides a unified representation of the
appearance of an object or background throughout the video. Similar to NLA,
Deformable Sprites [30] decomposes a video into a texture atlas that captures
an object’s motion across the entire video. Their method allows for temporally
consistent video edits, by applying edits to the decomposed atlas. In contrast
to NLA, they use optical flow to compute foreground objects instead of a
pretrained segmentation network to get the segmentation mask for an object in
the input video. However, both NLA and Deformable Sprites only allow for basic
manual editing. We use NLA’s atlas separation method for objects in videos,
but unlike NLA, we allow for text-driven stylization.

###  2.2 Text-based stylization

Our work bears similarities to recent image and 3D manipulation techniques
that edit style and appearances through natural language descriptions. These
descriptions are often embedded with the Contrastive Language Image
Pretraining (CLIP) [23] model, a multi-modal embedding model that learns an
image-text embedding space. Recent work used CLIP together with pretrained
generative networks for image editing and stylization [1, 21, 3, 5, 6, 8, 15].
For example, StyleCLIP [21], and StyleGAN-NADA [8] both use a pretrained
StyleGAN [12] and CLIP to perform image editing, either by using CLIP to
control a latent code or to adapt an image generator to a specific domain [8].
Other examples include using StyleGAN and CLIP for image stylization [5] or
Paint by Word [3] which uses CLIP paired with StyleGAN2 [13] and BigGAN [4] to
enable “painting” images in the style of a text prompt. Usually, pretrained
generators only work well for the specific domain they are trained on. In
contrast, our method does not require a pretrained generator. We train our own
generator on the set of video frames we wish to stylize.

Other examples of semantic text-guided manipulation in the context of 3D
objects include 3DStyleNet [31], a method for changing the geometric and
texture style of 3D objects, ClipMatrix [11] which uses text-prompts to create
digital 3D creatures, and methods for generating 3D voxels using CLIP [27].
Another line of recent work uses joint-embedding architectures [26, 25, 24,
19] for image generation, e.g., DALL-E [25] and its successor, DALL-E 2 [24],
which can also be used for stylizing images. DALL-E 2 uses a two-stage model,
where a CLIP image embedding is generated using a text prompt and a decoder is
then used to generate an image conditioned on the generated image embedding.
Training joint embedding architectures requires enormous datasets and many
training hours. Instead of training on a large dataset, we train on a set of
frames for a single video and use augmentations to extract many different
views of the input frames. As opposed to all the abovementioned techniques, we
work on videos.

Another line of work uses CLIP without relying on a pretrained generator, e.g.
given a natural language input, CLIPdraw [7] synthesizes novel drawings, and
CLIPstyler [17] stylizes images. Similarly, Texts2Mesh [18] does not rely on a
pretrained generator. In Text2Mesh, the CLIP embedding space is used to enable
text-driven editing of 3D meshes. Text2Mesh uses a multi-layer perceptron
(MLP) to apply a stylization to $(x,y,z)$-coordinates of an input mesh. The
neural optimization process of the MLP is guided by a semantic loss that
computes the similarity between multiple augmented 2D views embedded with CLIP
and a target text. Similarly, we do not rely on a pretrained generator.

Our work was developed concurrently to Text2Live [2], which shares many of the
same goals and methods as our work. Similarly to our method, Text2Live uses a
pretrained Neural Layered Atlases (NLA) model to separate a moving object in a
video from its background. Text2Live train a generator to apply text-driven
local edits to a single frame and use the NLA model to map the edits back to
the input video in a temporally consistent manner. In contrast to our
approach, Text2Live does not directly generate the edited output. Instead, it
generates an edit layer that is composited with the original input.

##  3 Method

We wish to apply natural and temporally consistent stylizations to objects in
videos using a natural language text prompt as guidance. To change the style
of an object to conform with a target text prompt in a temporally consistent
manner, we build on top of a Layered Neural Atlas method [14], which separates
the appearance of an object in a video from its background. We then use a pre-
trained text-image multimodal embedding of CLIP [23] in a set of objectives.
Minimizing this set of objectives aims at matching the style of a foreground
object in a video with that of a target text. The objectives include a global
and local objective. The global objective focuses on the global semantics by
maximizing the similarity between the global views and a target text that
relates to the underlying content. Instead, the local objective focuses on the
fine-grained details, by maximizing the similarity between the local views
with a target text that relates to local semantics of the stylization. We add
a sparsity loss, similar to a $L_{1}$-regularization term, that encourages the
predicted foreground color values to be minimal. Additionally, we add a
temporal triplet loss that encourages the embeddings of frames that are close
in time to also be close in CLIP’s embedding space. We begin by describing the
method of CLIP [23] and that of Neural Layered Atlases (NLA) on which our
method is based. We then describe the training and loss formulations used by
our method.

###  3.1 CLIP

CLIP is a multi-modal embedding method that trains an image encoder $E_{img}$
and a text encoder $E_{txt}$ to match between the embeddings of corresponding
image-text pairs using a contrastive loss formulation. This loss formulation
optimizes the similarity between corresponding image-text pair $T$ and $I$.
More specifically, $I$ and $T$ are first embedded:

$I_{emb}=E_{img}(I)\in\mathbb{R}^{512},\quad
T_{emb}=E_{txt}(T)\in\mathbb{R}^{512}$

The similarity between I and T is then measured by
$\operatorname{sim}(I_{emb},T_{emb})$ where
$\operatorname{sim}(a,b)=\frac{a\cdot b}{|a||b|}$, is the cosine similarity.

###  3.2 Neural Layered Atlases (NLA)

Figure 2: Our stylization pipeline. In the first step, we train the network
using the NLA procedure [14] to reconstruct input video frames. We then
finetune the editing atlas $A$ using our approach described in Fig. 3. In our
stylization pipeline, we create a set of cropped input video frames
$Q_{Crop}$. This set is passed through a stylization model to create a
foreground and background atlas. A set of stylized frames $Q_{Style}$ is
produced by $\alpha$-blending the predicted atlases. All weights of the MLPs
are frozen except for the editing atlas MLP $A$ which is finetuned. A closer
look at how our editing atlas is trained is given in Fig. 3. Figure 3:
Finetuning the editing atlas. As described in the stylization pipeline (Fig.
2), we use our stylization pipeline to get a set of stylized frames
$Q_{Style}$. To this end, we finetune our editing atlas, which is part of the
stylization model. We sample $n_{Global}$ global views $I^{Global}$ and
$n_{Local}$ local views $I^{Local}$. The set of images $I^{Global}$ are then
augmented using random perspectives and random background removal and used
together with the global target text $T_{Global}$ to compute a global loss
(Eq. 3). Three global augmented images are used to compute the temporal loss
(Eq. 4). Similarly, the $I^{Local}$ images are augmented and used together
with the local target text $T_{Local}$ to compute the local loss (Eq. 2).
Lastly, a sparsity loss (Eq. 5) is computed from the stylized frames
$Q_{Style}$.

Neural Layered Atlases (NLA) [14] decompose a video into a set of layered 2D
atlases. Each atlas provides a unified representation of the appearance of an
object or the background throughout the video. NLA use two mapping networks
$M_{f}$ and $M_{b}$, where each takes a pixel and time location $(x,y,t)$ in
the video as input and outputs the corresponding 2D $(u,v)$-coordinate in each
atlas:

$M_{f}(p)=(u_{f}^{p},u_{f}^{p}),\quad M_{b}(p)=(u_{b}^{p},u_{b}^{p})$

An atlas network $A$ takes the predicted 2D $(u,v)$-coordinate as input and
outputs the atlas’s RGB color at that location. Additionally, all pixel
coordinates are fed into the Alpha MLP network $M_{\alpha}$ which outputs the
opacity of each atlas at that location. The RGB color can then be
reconstructed at each pixel location by alpha-blending the predicted atlas
points according to the opacity value predicted by $M_{\alpha}$.

NLA enables consistent video editing. First, each atlas is discretized into an
image. A user can then manually apply edits using an editing program. These
edits are mapped back to the input video using the computed $(u,v)$-mapping.
To get the reconstructed color $c^{p}$ for pixel $p$, the color of the
predicted foreground color $c_{f}^{p}$, background color $c_{b}^{p}$ and
predicted opacity value $\alpha^{p}$ of the edited atlas are blended as
follows:

$c^{p}=(1-\alpha^{p})c_{b}^{p}+\alpha^{p}c_{f}^{p}$

###  3.3 Our Stylization Pipeline

Instead of having to manually apply edits to the discretized atlases as in
[14], we wish to apply edits automatically using a target text prompt.
Specifically, we are interested in modifying the RGB values such that they
conform with a target text. We focus on the Atlas MLP $A$, since $A$ makes the
RGB predictions. Instead of using a single atlas MLP $A$ for all atlases, we
create two copies of the atlas MLP after pre-training the NLA network, one
editing atlas MLP for the foreground object we want to stylize ($A$) and one
for all other atlases ($A_{b}$). We then freeze the weights of $M_{b}$,
$M_{f}$, $M_{\alpha}$, $A_{b}$ and finetune the weights of $A$. While the goal
in NLA is to optimize the model to produce the best reconstruction, we instead
want to create a stylization that conforms with the inputted target text.

Our stylization pipeline is illustrated in Fig. 2. The input is a set of raw
frames $Q_{Raw}$. Since we know the position of the object in the video from
the $\alpha$-map, we can compute a bounding box containing the whole object.
Once we have the bounding box, we crop each frame in $Q_{Raw}$ such that it
only contains the content within the bounding box plus a small margin. All the
cropped frames $Q_{Cropped}$ are passed through a pre-trained NLA model, where
all MLP weights have been frozen, except for the weights of the $A$ MLP. The
NLA method produces a set of stylized frames $Q_{Style}$.

To fine-tune the weights of $A$, we sample training batches in both time and
space. Our sampling method is illustrated in Fig. 3. First, we sample a set
$Q_{Sample}$ uniformly at random among all frames in the input video and pass
them through the stylization pipeline (Fig. 2) to create a set $Q_{Style}$.

For each frame in $Q_{Style}$ we sample $n_{Global}$ views $I^{Global}$ and a
set of $n_{Local}$ views $I^{Local}$. Each of the $n_{Global}$ views is
produced by sampling a crop with a size in the range $[0.9,1.0]$ of the
original frame size. Each of the $n_{Local}$ views is produced by sampling a
crop with a size in the range $[0.1,0.5]$ of the original frame size. To
ensure the local views contain the object we want to stylize, we use the
$\alpha$-map of the frame to determine the position of the object in the
frame. We then sample until we get $n_{Local}$ views where at least
$\frac{1}{3}$ of the sampled view is part of the object.

Once the local and global views have been sampled, we apply a random
perspective transformation and a random background removal which with some
probability $p$ removes the background (details in Appendix Sec. 6.2).
Additionally, each augmented frame is normalized with the same mean and
standard deviation as used to train the CLIP model [23].

Our objective function is composed of three main losses defined in CLIP’s
feature space: (1) $L_{\text{Local}}$ which focuses on local semantics, (2)
$L_{\text{Global }}$ which focuses on global semantics, and (3) $L_{Temp}$
which encourages temporal consistency. Additionally, we use the regularization
term $L_{\text{sparsity }}$ introduced in NLA [14], which encourages sparse
representations. In all loss terms, when we compute the similarity between a
text and each sampled view, we use the average embedding across all views:

$I_{emb}^{Local}=\frac{1}{n_{Local}}\sum_{i=1}^{n_{Local}}E_{img}\left(I_{i}^{\text{Local}}\right),I_{emb}^{Global}=\frac{1}{n_{Global}}\sum_{i=1}^{n_{Global}}E_{img}\left(I_{i}^{\text{Global}}\right)$
(1)

#### Local loss

$L_{\text{Local}}$ is applied to all views in $I^{\text{Local}}$. The goal is
to modify the image, such that the local details conform with a target text
$T_{\text{Local}}$:

$L_{\text{Local}}=1-\operatorname{sim}\left(I_{emb}^{Local},E_{txt}\left(T_{\text{Local}}\right)\right)$
(2)

where $\operatorname{sim}(a,b)=\frac{a\cdot b}{|a||b|}$ is the cosine
similarity, and $E_{txt}$ denote CLIP’s pre-trained text encoder. The local
views have a more zoomed-in view of the object we are stylizing. Additionally,
the local target text $T_{Local}$ contains local specific semantics, e.g.
“rough cactus texture.” Hereby, the local loss can focus on the texture and
fine-grained details of the stylization we apply to the input video.

#### Global loss

The global loss is applied to views in $I^{\text{Global}}$ that all include
the entire object being stylized. The intended goal is that the global loss
will preserve the overall context. In the target text $T_{\text{Global}}$, we
include words that describe the global context of the object we are trying to
stylize, e.g. “A swan made of cactus.” The global loss formulation is then
given by:

$L_{\text{Global}}=1-\operatorname{sim}\left(I_{emb}^{Global},E_{txt}\left(T_{\text{Global
}}\right)\right)$ (3)

#### Temporal loss

We use a triplet loss to include a temporal aspect and enforce that
consecutive frames should be more similar in CLIP’s embedding space than
frames that are further apart. To compute the temporal loss, we sample three
frames $t_{1},t_{2,},t_{3}$, where we have that $t_{1}<t_{2,}<t_{3}$ w.r.t.
the order of the frames in the input video. We then enforce that the
similarity between the sampled global views of $t_{1}$ and $t_{2}$ in the CLIP
embedding space should be greater than the similarity between $t_{1}$ and
$t_{3}$: Let
$I_{emb({t_{1}})}^{Global},{}I_{emb({t_{2}})}^{Global},I_{emb({t_{3}})}^{Global}$
denote the average embedded global views (computed in Eq. 1) for each of the
three frames. Then we compute the triplet loss $L_{Temp}$ as follows:

$\begin{split}\operatorname{Sim}_{t_{1}t_{3}-t_{1}t_{2}}&=\operatorname{sim}\left(I_{emb({t_{1}})}^{Global},I_{emb({t_{3}})}^{Global}\right)-\operatorname{sim}\left(I_{emb({t_{1}})}^{Global},I_{emb({t_{2}})}^{Global}\right)\\\
L_{Temp}&=\lambda_{Temp}\cdot\max\left(0,\operatorname{Sim}_{t_{1}t_{3}-t_{1}t_{2}}\right)\end{split}$
(4)

where $\lambda_{Temp}$ is a weighting of the temporal loss. If the frames are
further away from each other, the temporal loss should contribute less to the
overall loss. For this reason, we weigh the contribution of the temporal loss
by a Gaussian probability density function $g$ with a mean equal to zero and a
standard deviation of five. We compute the weight of a triplet by applying $g$
to the difference between $t_{1}$ and $t_{3}$:

$\lambda_{Temp}=g(t_{3}-t_{1})$

#### Sparsity loss

We use the same sparsity loss as in NLA [14]. Its intended function is to
encourage points that are mapped to the background atlas to have a zero value
in the foreground atlas, e.g. if a point $p$ is mapped to the background
atlases it should not contain information about the foreground atlas.

$L_{\text{sparsity }}=\left\|\left(1-\alpha^{P}\right)c_{f}^{P}\right\|$ (5)

where $c_{f}^{P}$ is the predicted color at $p$ for the foreground layer and
$\alpha^{P}$ is the opacity value at location $p$.

#### Full objective

The full loss term that is minimized is represented as:

$L=\lambda_{Sparsity}L_{\text{Sparsity }}+\lambda_{Temp}L_{\text{Temp
}}+\lambda_{Local}L_{\text{Local}}+\lambda_{Global}L_{\text{Global }}$ (6)

where $\lambda_{Local}$,$\lambda_{Global}$, $\lambda_{Temp}$,
$\lambda_{Sparsity}$ are hyperparameters used to control the weighting of each
loss term. As default $\lambda_{Local}=\lambda_{Global}=1$, while
$\lambda_{Temp}$ and and $\lambda_{Sparsity}$ vary depending on the input
video.

##  4 Experiments

We evaluate our method on a set of videos from the DAVIS dataset [22] across a
diverse set of target text prompts. Our goal is to perform consistent and
natural video editing. For this purpose, we present both a quantitative and
qualitative evaluation of our results and perform a careful ablation study of
each loss term in our objective function.

In Sec. 4.1 we demonstrate the capabilities of our method by showing various
stylizations for different videos. We also present a quantitative evaluation,
where we compare our method to an image baseline method applied to each frame
in the video. In Sec. 4.2 we demonstrate that the specificity of text prompts
influences the details of the results. In Sec. 4.3 we demonstrate how text
augmentation affects the results of the stylizations. In Sec. 4.4 we conduct a
series of ablations on our loss terms and demonstrate how the local and global
losses focus on different semantics. Finally, in Sec. 4.5 we illustrate some
of the limitations of our method.

|  |
---|---|---
|  |
“Swan” | “Shiny metal swan” | “Swan with wood bark skin”
|  |
|  |
“Boat” | “Shiny aluminum fishing boat” | “Boat made of copper”
|  |
|  |
“Dog” | “Dog with zebra fur” | “Golden dog”
Figure 4: Example results. Two representative frames from each edited video together with the global target text. | Q1 (Realism) | Q2 (Matching Text)
---|---|---
Blended Diffusion [1] |  $2.22$ ($\pm 1.08$) |  $2.10$ ($\pm 1.00$)
Ours |  $\mathbf{3.47}$ ($\pm 1.10$) |  $\mathbf{3.94}$ ($\pm 0.99$)
Table 1: Mean opinion scores (1-5) and standard deviation for Q1 and Q2.

###  4.1 Varied stylizations

In Fig. 4 we illustrate our video stylization method applied to three videos
and three texts. All local target texts used for these examples are similar to
the global target text, but without containing the information about the
underlying content, e.g., for the swan, the local target text is “metal skin.”
The results show that we can apply temporally consistent stylizations that
adhere to the target text specification. The swan example shows fine-grained
details of the target texts and also preserves the underlying content of the
swan. In the boat example, the stylization captures the details of the target
text. The boat has a texture similar to shiny aluminum and also has something
that looks like a fishing net at the end of the boat. The dog example shows
that our method can apply a realistic and consistent stylization to a video
containing occlusions.

We quantify the effectiveness of our method by comparing it to an image
baseline applied to each frame in a video input. As a baseline, we use a
pretrained Blended-diffusion (BF) [1] model with standard configurations. The
model takes as input an image, a ROI mask, and a target text. BF performs
local (region-based) edits based on a target text description and the ROI
mask. We conduct a user study to evaluate the perceived quality of the
stylized outputs generated by both the BF model and our method, and the degree
to which the outputs adhere to the global target text. Our user study
comprises $50$ users and $20$ stylized videos, each with a different target
text. For each video and target text combination, the users are asked to
assign a score ($1$-$5$) to two factors: (Q1) “How realistic is the video?,”
(Q2) “does the {object} in the video adhere to the text {content}. For Q1 we
make it clear to the user that “realistic” refers to the quality of the video
content. The results are shown in Table 1.

|  |  |  |
---|---|---|---|---
(a) | (b) | (c) | (d) | (e)

|  |  |  |
---|---|---|---|---
(a) | (b) | (c) | (d) | (e)

Figure 5: Target text specificity. Each example shows a representative frame
from the video. The experiment shows the specificity of a target affects the
stylization. Global target text prompts, row one: (a) “Armor,” (b) “Iron
armor,” (c) “Medieval iron armor,” (d) “Suit of shiny medieval iron armor,”
(e) “Full plate shiny medieval iron armor,” row two: (a) “Boat made of wood,”
(b) “Boat made of dark walnut wood (c) “Fishing boat made of wood,” (d) “Old
fishing boat made of wood,” (e) “Fishing boat made of wood planks.”

###  4.2 Prompt specificity

We demonstrate that varying the specificity of the target text prompt affects
the level of detail in the stylization. Our experiment is motivated by recent
work on prompt engineering [32] that shows how slight changes in the target
text can have a big impact on the CLIP similarity between a text and an image.
Fig. 5 shows an increasing level of detail for two videos and two target
texts. The target text specificity not only influences the level of detail, it
also makes it easier for CLIP to navigate in its embedding space. In the swan
example in column (a), we have “Swan with an armor,” which is a more ambiguous
target compared to the other swan examples with more detailed target texts and
stylizations. We hypothesize that this is because several stylizations can
satisfy the more simple target text, while a more specific target text narrows
down the set of possible directions in CLIP’s embedding space. The swan
examples in columns (d) and (e) indicate that CLIP has some understanding of
the different body parts of the swan. The “full plate” target text (d) covers
the entire head of the swan while the “suit” of armor in column (e) has a
clear cut around the head of the swan.

|  |  |  |
---|---|---|---|---
(a) | (b) | (c) | (d) | (e)

|  |  |  |
---|---|---|---|---
(a) | (b) | (c) | (d) | (e)

Figure 6: Prefix augmentations - varying number of prefixes to sample from in each iteration. Each figure shows a representative frame from each video and experiment configuration. The examples in row one use the texts: $T_{Global}$:“Origami swan with white paper skin,” $T_{Local}$: “Origami white paper skin,” and in row two: $T_{Global}$:“Dog with bengal tiger fur,” $T_{Local}$: “Bengal tiger fur.” (a) no prefixes, (b) $4$ local, no global, (c) $4$ global, no local (d) $4$ global & $4$ local, (e) $8$ global & $8$ local. The specific prefixes are described in Sec. 6.3. |  |  |  |
---|---|---|---|---
(a) | (b) | (c) | (d) | (e)
Figure 7: Ablation on each loss term \- All experiments were run with the same seed and with the texts: $T_{Global}$: “A swan with crocodile skin,” $T_{Local}$: “Crocodile skin.” Each figure shows a representative frame from a video, where we ablate one of our loss terms in Eq. 6. (a) All losses, (b) w/o local loss, (c) w/o global loss, (d) w/o temporal loss, (e) w/o sparsity loss. |  |
---|---|---
(a) | (b) | (c)
Figure 8: Global and local semantics. A representative frame from each of the
edited video frames. The texts used are: (a) $T_{Global}$: “Swan with cactus
skin,” $T_{Local}$: “Cactus skin,” (b) $T_{Global}$: “Swan with cactus skin,”
$T_{Local}$: “Rough cactus skin,” (c) $T_{Global}$: “Swan made out of cactus,”
$T_{Local}$: “Catus skin.” (a) and (b) have the same global target text while
(a) and (c) have the same local target texts. In (b) the local details have
changed to a more rough texture. In (c) the global swan semantics are better
preserved.

###  4.3 Text augmentation

We add textual augmentation to our method to address some of the challenges
with prompt engineering. Inspired by Zhou et al. [32], we add neutral prefixes
to both the local and global target texts, e.g., “a photo of a {}.” We then
sample a new prefix each iteration for each of the target texts as a form of
regularization. Fig. 6 illustrates our text augmentation experiment. We
demonstrate that using prefixes increases the robustness and quality of the
results. In each experiment and each iteration, we sample one prefix among a
set of prefixes (details in appendix Sec. 6.3) for both the global and local
target texts. In this experiment, we vary the number of prefixes to sample
from and show that using an equal amount of prefixes for both the local and
global target texts produces better quality results than using no prefixes. In
both the swan and dog examples for columns (e) and (d) that use multiple
prefixes, we see that the stylizations are more detailed than the examples in
columns (a-c), e.g., in the dog example (d) and (e) the tiger fur has white
pigments around the belly area and also more natural tiger stripes.

###  4.4 Ablation study

We validate each term in our objective function (Eq. 6) with an ablation study
illustrated in Fig. 7. In (a), we see that all losses combined result in a
natural stylization that shows clear characteristics of a swan with crocodile
skin. In (b) we see that without the local loss the results lack fine-grain
details. It is still evident what the intended stylization was but the
crocodile texture is not as clear as in (a). The results in (c) show that
without the global loss the global semantics are less clear, e.g., the swan’s
neck is not stylized as detailed as the rest of the body. In (d), we see that
without the temporal loss, we get more edits outside the mask of the object.
In (e), we see that without the sparsity loss, the stylization is very noisy.

|  |
---|---|---
(a) | (b) | (c)
Figure 9: Limitations. A representative frame from each of the edited video
frames. The texts used are: (a) $T_{Global}$: “Swan with cactus skin,”
$T_{Local}$: “Cactus skin,” (b) $T_{Global}$: “Boat made out of chocolate,”
$T_{Local}$: “Chocolate texture,” (c) $T_{Global}$: “Dog with zebra fur,”
$T_{Local}$: “Zebra fur.” In (a), the cactus texture is applied like photos of
cactus. In (b) the chocolate boat has a sailing ship printed at the end of the
boat. In (c), the dog has a face of a dog on its haunches.

In Fig 8 we illustrate how the local loss affects the fine-grained details and
the global loss affects the global semantics. We use (a) as a baseline and
then vary the local target texts in (b) and the global target texts in (c). In
(b) we see, how changing the local target text to include “rough” affects the
details of the cactus texture. In (c) we see how the global semantics of the
swan’s body become more realistic as an effect of changing the global target
text.

###  4.5 Limitations

Fig. 9 illustrates some of the limitations of our method. During training our
method occasionally starts to overfit and produce unintended stylizations. In
(a), we see that the body of the swan contains photo-like cacti instead of
natural cactus texture. In (b) and (c), we see how our model has used the
contexts of the global target text. In (b), a sailing ship has been added to
the end of the boat and in (c), a face of a dog has been added.

Our method is limited to text-prompts that do not entail features that cross
between the decomposed atlas layers. Some changes are best realized through a
shape change, e.g, “Swan with long hair.”, which is not currently possible in
our framework. Other limitations of our method include stylizations that focus
too much on some property of the target texts, e.g., we experienced that the
global target texts and similar variants “Swan with strawberry texture skin,”
gave noisy results, where the swan was colored red.

##  5 Conclusion

We considered the problem of developing intuitive and semantic control for
consistent editing and styling of objects in videos. This problem poses a
challenge in generating consistent content and style changes over time while
being able to produce fine-grained details and preserve the global semantics.
We proposed a method that uses CLIP [23] and the video object representation
[14] to stylize objects in videos by using both a global target text, to
control the global semantics of the stylization, and a local target text, to
control the fine-grained details. We demonstrated that the specificity and the
prefixes of the target texts can have a significant impact on the details
produced by our method’s stylization. In future work, it would be interesting
to investigate the limitations of CLIP. A model that is able to generate fine-
grained stylizations of videos and images could be leveraged to create data
augmentations in other learning settings. Another line of future work is to
extend our model to be able to generate shape changes or even new objects from
scratch.

### Acknowledgement

This research was supported by the Pioneer Centre for AI, DNRF grant number
P1. We would like to thank Ira Assent for the helpful discussions.

## References

  * [1] Avrahami, O., Lischinski, D., Fried, O.: Blended diffusion for text-driven editing of natural images. CoRR abs/2111.14818 (2021), https://arxiv.org/abs/2111.14818
  * [2] Bar-Tal, O., Ofri-Amar, D., Fridman, R., Kasten, Y., Dekel, T.: Text2live: Text-driven layered image and video editing (2022). https://doi.org/10.48550/ARXIV.2204.02491, https://arxiv.org/abs/2204.02491
  * [3] Bau, D., Andonian, A., Cui, A., Park, Y., Jahanian, A., Oliva, A., Torralba, A.: Paint by word. CoRR abs/2103.10951 (2021), https://arxiv.org/abs/2103.10951
  * [4] Brock, A., Donahue, J., Simonyan, K.: Large scale GAN training for high fidelity natural image synthesis. CoRR abs/1809.11096 (2018), http://arxiv.org/abs/1809.11096
  * [5] Chefer, H., Benaim, S., Paiss, R., Wolf, L.: Image-based clip-guided essence transfer. CoRR abs/2110.12427 (2021), https://arxiv.org/abs/2110.12427
  * [6] Crowson, K., Biderman, S., Kornis, D., Stander, D., Hallahan, E., Castricato, L., Raff, E.: Vqgan-clip: Open domain image generation and editing with natural language guidance (2022). https://doi.org/10.48550/ARXIV.2204.08583, https://arxiv.org/abs/2204.08583
  * [7] Frans, K., Soros, L.B., Witkowski, O.: Clipdraw: Exploring text-to-drawing synthesis through language-image encoders (2021). https://doi.org/10.48550/ARXIV.2106.14843, https://arxiv.org/abs/2106.14843
  * [8] Gal, R., Patashnik, O., Maron, H., Chechik, G., Cohen-Or, D.: Stylegan-nada: Clip-guided domain adaptation of image generators. CoRR abs/2108.00946 (2021), https://arxiv.org/abs/2108.00946
  * [9] Jabri, A., Owens, A., Efros, A.: Space-time correspondence as a contrastive random walk. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., Lin, H. (eds.) Advances in Neural Information Processing Systems. vol. 33, pp. 19545–19560. Curran Associates, Inc. (2020), https://proceedings.neurips.cc/paper/2020/file/e2ef524fbf3d9fe611d5a8e90fefdc9c-Paper.pdf
  * [10] Jampani, V., Gadde, R., Gehler, P.V.: Video propagation networks. CoRR abs/1612.05478 (2016), http://arxiv.org/abs/1612.05478
  * [11] Jetchev, N.: Clipmatrix: Text-controlled creation of 3d textured meshes. CoRR abs/2109.12922 (2021), https://arxiv.org/abs/2109.12922
  * [12] Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. CoRR abs/1812.04948 (2018), http://arxiv.org/abs/1812.04948
  * [13] Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T.: Analyzing and improving the image quality of stylegan. CoRR abs/1912.04958 (2019), http://arxiv.org/abs/1912.04958
  * [14] Kasten, Y., Ofri, D., Wang, O., Dekel, T.: Layered neural atlases for consistent video editing. CoRR abs/2109.11418 (2021), https://arxiv.org/abs/2109.11418
  * [15] Kim, G., Ye, J.C.: Diffusionclip: Text-guided image manipulation using diffusion models. CoRR abs/2110.02711 (2021), https://arxiv.org/abs/2110.02711
  * [16] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: Bengio, Y., LeCun, Y. (eds.) 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings (2015), http://arxiv.org/abs/1412.6980
  * [17] Kwon, G., Ye, J.C.: Clipstyler: Image style transfer with a single text condition. CoRR abs/2112.00374 (2021), https://arxiv.org/abs/2112.00374
  * [18] Michel, O., Bar-On, R., Liu, R., Benaim, S., Hanocka, R.: Text2mesh: Text-driven neural stylization for meshes. CoRR abs/2112.03221 (2021), https://arxiv.org/abs/2112.03221
  * [19] Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., Chen, M.: GLIDE: towards photorealistic image generation and editing with text-guided diffusion models. CoRR abs/2112.10741 (2021), https://arxiv.org/abs/2112.10741
  * [20] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., Chintala, S.: Pytorch: An imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems 32, pp. 8024–8035. Curran Associates, Inc. (2019), http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf
  * [21] Patashnik, O., Wu, Z., Shechtman, E., Cohen-Or, D., Lischinski, D.: Styleclip: Text-driven manipulation of stylegan imagery. CoRR abs/2103.17249 (2021), https://arxiv.org/abs/2103.17249
  * [22] Pont-Tuset, J., Perazzi, F., Caelles, S., Arbelaez, P., Sorkine-Hornung, A., Gool, L.V.: The 2017 DAVIS challenge on video object segmentation. CoRR abs/1704.00675 (2017), http://arxiv.org/abs/1704.00675
  * [23] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I.: Learning transferable visual models from natural language supervision. CoRR abs/2103.00020 (2021), https://arxiv.org/abs/2103.00020
  * [24] Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., Chen, M.: Hierarchical text-conditional image generation with clip latents (2022). https://doi.org/10.48550/ARXIV.2204.06125, https://arxiv.org/abs/2204.06125
  * [25] Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., Sutskever, I.: Zero-shot text-to-image generation. CoRR abs/2102.12092 (2021), https://arxiv.org/abs/2102.12092
  * [26] Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E., Ghasemipour, S.K.S., Ayan, B.K., Mahdavi, S.S., Lopes, R.G., Salimans, T., Ho, J., Fleet, D.J., Norouzi, M.: Photorealistic text-to-image diffusion models with deep language understanding (2022). https://doi.org/10.48550/ARXIV.2205.11487, https://arxiv.org/abs/2205.11487
  * [27] Sanghi, A., Chu, H., Lambourne, J.G., Wang, Y., Cheng, C.Y., Fumero, M., Malekshan, K.R.: Clip-forge: Towards zero-shot text-to-shape generation (2021). https://doi.org/10.48550/ARXIV.2110.02624, https://arxiv.org/abs/2110.02624
  * [28] Texler, O., Futschik, D., Kucera, M., Jamriska, O., Sochorová, S., Chai, M., Tulyakov, S., Sýkora, D.: Interactive video stylization using few-shot patch-based training. CoRR abs/2004.14489 (2020), https://arxiv.org/abs/2004.14489
  * [29] Wang, X., Jabri, A., Efros, A.A.: Learning correspondence from the cycle-consistency of time. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2019)
  * [30] Ye, V., Li, Z., Tucker, R., Kanazawa, A., Snavely, N.: Deformable sprites for unsupervised video decomposition (2022). https://doi.org/10.48550/ARXIV.2204.07151, https://arxiv.org/abs/2204.07151
  * [31] Yin, K., Gao, J., Shugrina, M., Khamis, S., Fidler, S.: 3dstylenet: Creating 3d shapes with geometric and texture style variations. CoRR abs/2108.12958 (2021), https://arxiv.org/abs/2108.12958
  * [32] Zhou, K., Yang, J., Loy, C.C., Liu, Z.: Learning to prompt for vision-language models. CoRR abs/2109.01134 (2021), https://arxiv.org/abs/2109.01134

##  6 Appendix

###  6.1 Training details

We train on a single GPU (RTX $6000$). Our method is implemented using the
PyTorch framework [20] and will be made available. We use an ADAM optimizer
[16] with an initial learning rate of $1^{-4}$ and decay the learning rate by
a factor of $0.9$ every $200$ iterations. Our method takes about $40$ minutes
and $2000$ iterations with a batch size of three input frames, each of size
$432$x$768$, sampled from a video of maximally $70$ frames. High-quality
results usually appear after $10-20$ minutes.

All images are normalized and resized to $224$x$224$ before being passed to
the CLIP model. We normalization with mean $(0.48145466,0.4578275,0.40821073)$
and standard deviation $(0.26862954,0.26130258,0.27577711)$, which is the same
as CLIP was trained with. We use the pretrained “ViT-L/14” CLIP model loaded
from OpenAi’s GitHub page.

###  6.2 Image augmentation

  * •

Random Crops: For the global views we crop the reconstructed frame with a
random scaling in the range: $[0.9,1.0]$ and for the local views we use a
scaling in the range $[0.1,0.5]$. For both global and local views we use a
random aspect ratio in the range $[0.8,1.2]$. The local crops have a chance of
not including the object we want to edit, e.g. a small crop from the bounding
box containing the Swan in Fig. 4 has a chance of only containing the
background. To combat this, we use the $\alpha$-map to locate the object, and
sample local crops until we get a crop that contains at least $\frac{1}{3}$ of
the object.

  * •

Random Perspective Transformation: With probability $\frac{1}{2}$, we sample a
number $d$ uniformly at random in the range: $[0.1,0.5]$ and use $d$ as the
distortion scale.

  * •

Random Background Removal: We locate the foreground object using the
$\alpha$-map. Then, with probability $\frac{1}{2}$ we color all background
pixels black.

###  6.3 Text augmentations

In our text augmentation we randomly sample one of the following prefixes in
each iteration:

  1. 1.

“a photo of a {}”

  2. 2.

“a {}”

  3. 3.

“an image of a {}”

  4. 4.

“the {}”

  5. 5.

“image of a {}”

  6. 6.

“image of the {}”

  7. 7.

“photo of a {}”

  8. 8.

“photo of the {}”

All prefixes are neutral and intended to not change the semantics of the
stylization. In the text augmentation experiments (Sec. 4.3), if we use four
prefixes, these are the first four in the list above.


11institutetext: IIIT Hyderabad, India22institutetext: 22email:
<EMAIL_ADDRESS>33email:
<EMAIL_ADDRESS>
# A framework for syntactic and semantic quality evaluation of ontologies

Vivek Iyer 1122 0000-0003-4451-8293 Lalit Mohan Sanagavarapu 1122
0000-0003-0745-1042 Y Raghu Reddy 1133 0000-0003-2280-5400

###### Abstract

The increasing focus on Web 3.0 is leading to automated creation and
enrichment of ontologies and other linked datasets. Alongside automation,
quality evaluation of enriched ontologies can impact software reliability and
reuse. Current quality evaluation approaches oftentimes seek to evaluate
ontologies in either syntactic (degree of following ontology development
guidelines) or semantic (degree of semantic validity of enriched
concepts/relations) aspects. This paper proposes an ontology quality
evaluation framework consisting of: (a) SynEvaluator and (b) SemValidator for
evaluating syntactic and semantic aspects of ontologies respectively.
SynEvaluator allows dynamic task-specific creation and updation of syntactic
rules at run-time without any need for programming. SemValidator uses Twitter-
based expertise of validators for semantic evaluation. The efficacy and
validity of the framework is shown empirically on multiple ontologies.

###### Keywords:

Ontology Quality Evaluation Syntactic Evaluation Semantic Validation
Crowdsourcing Twitter-based expertise

##  1 Introduction

The exponential increase in Internet users over the past decade has led to
generation of large volume of data. Web 3.0, otherwise commonly referred to as
Semantic web, seeks to represent internet data as knowledge through knowledge
graphs, ontologies and other knowledge systems [4]. These representations
enable knowledge integration, semantic ambiguity resolution, information
extraction, decision making, reasoning and many other use cases relevant to
the building of ‘intelligent’ software systems. Ontologies, in particular,
store domain-specific knowledge, and represent this knowledge through
concepts, relations, axioms and instances. They contain a formal structure and
achieve a certain level of rigor due to the presence of rules and constraints.
Ontologies are rarely static in nature. The range and the depth of the
knowledge stored are enriched over time. This impacts a wide variety of
software applications that utilize ontologies for reasoning, decision-making,
question-answering, etc. Ontology enrichment is thus a crucial step in the
ontology engineering process.

Traditionally, ontologies are created and managed by knowledge engineers and
domain experts resulting in high costs due to the expert human labour
involved. Automated or semi-automated approaches to ontology enrichment are
increasingly popular, driven by increased availability of domain-relevant
internet data and improvements in natural language processing and machine
learning models [36]. Research on ontology learning (both creation and
enrichment) snowballed in the last two decades [20], [36], [15], with
increased focus on fully automated Deep Learning based approaches [30]. Given
the variety in ontology enrichment approaches, it is important to evaluate the
quality of the enriched ontology.

Figure 1: Syntactic quality violation: No domain or range for property Figure
2: Semantic violation: Invalid enriched concept

Ontology evaluation approaches can broadly be divided into: manual, automated
and semi-automated approaches. In general, ontology evaluation happens on one
of two aspects: syntactic quality, or semantic quality. We define syntactic
quality of an ontology as a measure of its adherence to ontology development
guidelines or rules. For example, One such rule could necessitate presence of
both domain and range elements in properties. Examples of other rules or
guidelines could include explicit declaration of equivalent and inverse
properties, presence of annotations [34], following of unique naming
conventions [23], etc. Figure 1 shows, in Turtle syntax [3], a property
without a defined range element - thus violating the rule that necessitates
the presence of both domain and range elements. Semantic quality deals with
validity of enriched concepts, relations and instances. Figure 2 shows an
example ontology enriched with concepts extracted from a sentence to emphasize
the need for evaluation of enriched ontologies. The previously existing
concepts are shown in blue, the valid enriched concepts in green and the
invalid enriched ones in red. Ontology enrichment algorithms using Hearst
patterns [14] could mistakenly detect ‘antivirus’ as a type of ‘malware’. In
such cases, before creating a final ontology, the enriched ontology needs to
be validated for semantic quality, to accept or reject the enriched concepts,
properties and instances.

A variety of metric-based methods were proposed to evaluate various syntactic
quality-based characteristics of ontologies [9], [19], [32]. Publicly
available tools were proposed that allow users to evaluate syntactical quality
of ontologies using pre-defined metrics [18], [22] or rules [28], [31].
However, they offer limited customization and flexibility to the user for
creating task-specific rules for evaluation, even more so for non-programmers.
In regards to semantic evaluation, researchers have traditionally employed
domain experts [21], while in this decade, crowdsourced validators [17], [24]
are being used for semantic ontology validation. This paper proposes a
customizable and scalable framework that evaluates syntactic and semantic
aspects of ontology quality using SynEvaluator and SemValidator respectively.

  * •

$SynEvaluator$: a tool that uses a rule-creation framework for allowing users
to non-programatically create rules during run-time and to set task-specific
priorities for these rules.

  * •

$SemValidator$: a tool that uses crowdsourcing for validation of semantic
quality of enriched ontologies. In this paper, a Twitter-based expertise
estimation algorithm is used to weight validators’ decisions.

The source code for SynEvaluator and SemValidator is available on
GitHub111https://github.com/Remorax/SynEvaluator/,222https://github.com/Remorax/SemValidator/.
They are also deployed on Heroku as web-
applications333https://synevaluator.herokuapp.com/,444http://semvalidator.herokuapp.com/.

The rest of the paper is structured as follows: Section 2 describes related
work in ontology quality evaluation. The proposed framework constituting of
SynEvaluator and SemValidator is shown in section 3. Section 4 details the
experiments done to show the efficacy and accuracy of the framework, through
these tools. SynEvaluator is tested for its utility and accuracy for
implementing syntactical quality evaluation rules by comparing it against a
popular syntactic quality evaluation tool, OOPS! [27]. The efficacy of
$SemValidator$ is shown by conducting crowdsourced survey involving 28
validators on two popular ontologies, Stanford Pizza [8] and Information
Security ontology [10]. Accuracy of TweetExpert algorithm on responses to both
of these ontologies using multiple ML regression algorithms is shown. Finally,
section 5 summarizes the contributions and suggests possible future directions
of research.

##  2 Related Work

Ontology quality evaluation approaches can be broadly classified into a)
syntactic and b) semantic quality evaluation approaches. Syntactic evaluation
approaches primarily evaluate structural aspects of an ontology based on
ontology development guidelines, common pitfalls, structural metrics etc.
OntoClean [13], one of the earliest known works in this area, proposed a
methodology for validating adequacy of relationships in an ontology based on
notions drawn from philosophy such as essence, identity, and unity. Similarly,
OntoQA [32] proposed ontology evaluation on the basis of schema metrics and
instance metrics. They stated that ‘goodness’ or ‘validity’ of an ontology
varies between different users and domains. Gangemi et al. [11] proposed
structural, functional and usability-related measures using O2 and oQual, a
meta-ontology and an ontology model for ontology evaluation and validation.
Burton et al. [5] proposed an ontology evaluation model based on semiotic
theory. In order to apply the metrics proposed in these works, tools such as
S-OntoEval [7] drawn from semiotic theory and AktiveRank [1] that ranked
ontologies based on structural metrics like class match measure, density
measure etc. were proposed. However, the tools proposed in these articles are
either closed-source prototypes or theoretical frameworks and are not publicly
available.

There are also a few publicly available closed-source ontology (syntactic)
evaluation tools, such as OOPS! [28], DoORS [22] and OntoMetrics [18].
OntOlogy Pitfall Scanner (OOPS!) evaluates ontologies on the basis of
established ontology quality rules related to human understanding, logical
consistency, real world representation and modelling issues, and manually
assigned priorities. DoORS, evaluates ontologies based on metrics drawn from
Semiotic Theory while OntoMetrics uses metrics proposed in OntoQA, [11]. These
tools, however are not flexible or customizable, and evaluate ontologies using
a fixed set of rules or metrics. Users are unable to create/update customized
rules and set task-specific priorities, which can be crucial as requirements
and application scenarios vary. The available open source implementations [33]
require creation of new rules and priorities via programming which can be
daunting for non-programmers.

Semantic evaluation approaches focus on semantic validity of concepts and
relationships in an ontology. Traditionally, it has been formulated as a task
requiring simple accept/reject decisions from domain experts [2]. In the past
few years, a good number of crowdsourced ontology evaluation approaches have
been proposed. Hanika et al. [35] have developed the UComp Protégé plugin to
provide a platform for crowdsourced workers to validate classes, subclasses,
properties and instances. Kiptoo et al. [16] use crowdsourcing for axiom and
assertion verification in ontologies as well as for verification of subclass-
superclass relations by Amazon Mechanical Turks [24]. Pittet et al. used
crowdsourced workers to propose changes related to addition, deletion and
substitution errors [25]. Zhang et al. [37] used crowdsourced workers to
obtain written feedback (comments/suggestions/references) for making final
validation decisions. Requiring complex tasks (such as making data quality
decisions or requiring written feedback) from crowdsourced workers can be
expensive and unscalable as the size of ontology and/or number of workers
increases. Some approaches [35], [17], [25] used quality control mechanisms
like majority voting that are debatable. Noy et al. [24] addressed this by
using qualification questions and spam filtering techniques. While these
mechanisms eliminate spammers, it may not be applicable where large number of
workers have some degree of knowledge but only few are experts. Therefore, an
assessment of domain expertise on a continuous scale would be useful as a
quality control metric. Further, an integrated quality evaluation framework
that seeks to evaluate ontologies on both syntactic and semantic aspects, and
also addresses the above-mentioned problems would be useful for a holistic and
integrated approach to quality evaluation of enriched ontologies.

##  3 Proposed Framework

In this paper, an ontology evaluation framework that combines automated
syntactic evaluation and human-centric semantic evaluation is proposed.
SynEvaluator aims at increasing user flexibility by allowing customized rule
creation at runtime, as well as scalability (with respect to user base) by
proposing an approach that removes the need for programming. SemValidator
proposes a crowdsourcing based approach that uses the validators’ Twitter
profiles for quality control.

Figure 3: Work flow for semantic and syntactic quality evaluation of
ontologies.

The framework’s work flow is shown in Figure 3. The input to quality
evaluation process is an enriched ontology. The ontology may have been
enriched with concepts, relations and instances using some automated and semi-
automated algorithms. The ontology may contain syntactic quality errors (due
to violation of ontology development guidelines) and/or semantic errors (due
to wrongly enriched concepts). In the figure, for clarity, concepts and
relationships with potential syntactic violations are bordered in yellow, an
those that do not contain such violations are outlined in blue. Also, concepts
that potentially contain semantic violations are highlighted in yellow colour
and concepts that are semantically valid are highlighted in green.

An ontology engineer uploads an enriched ontology to SynEvaluator, and creates
syntactic quality evaluation rules. The rules created using a theoretical rule
creation framework are applied on the parsed ontology object through
SynEvaluator’s ontology evaluation module. This returns a list of detected
violations and the elements causing these violations. Using these elements as
suggestions, the ontology engineer can fix violations in a iterative manner.
The iterations may be repeated as needed. Then, the ontology engineer can
provide this ontology as input to SemValidator so that it can be validated for
semantic quality. As part of semantic validation, the ontology is provided to
crowdsourced validators who give their accept or reject validation decisions
for each of the enriched concepts, relations and instances. Simultaneously, an
estimate of the domain knowledge of each of these validators is calculated
from their Twitter profile using the TweetExpert algorithm. These scores
alluded to as ‘TweetExpert scores’, are used as a quality control mechanism to
ensure that the decisions of crowdsourced validators are given weightages
according to their knowledge of the ontology domain. Finally, the output of
this algorithm is used to take the final accept/reject decision for each
enriched element, resulting in an ontology with both good syntactic and
semantic quality.

###  3.1 Stage 1: SynEvaluator

In this section, the underlying terminology used in SynEvaluator and rule
creation framework is illustrated with examples. Further, the implementation
of SynEvaluator as a web application is detailed. Finally, the section ends
with a discussion on potential benefits and limitations of SynEvaluator.

####  3.1.1 Defining the Rule Creation Framework

SynEvaluator allows users to create rules at run-time. The rules are
constructed from individual components like “Subjects", “Clauses" or “Operator
Expressions". The operational definitions of these and other relevant terms
are :

  * •

Ontological Element: Refers to any element that forms a constituent part of an
ontology. It refers to any primary component (classes, individuals,
properties), their related elements (subclasses, domains, ranges), annotations
(labels, descriptions, comments) or attributes (ID, language, namespace).

  * •

Rule: Refers to a sequence of one or more clauses, optionally connected by one
or more operator expressions that returns either one or more ontological
elements or a boolean value.

  * •

Clause: Refers to a transformation applied on an ontological element(s) to
return either one or more ontological elements or a boolean value.

  * •

Operator Expression: An expression used to compare and/or connect non-empty
sequences of clauses to return a boolean value.

  * •

Subject: Refers to an ontological element (typically primary components such
as classes, individuals and properties), that is subjected to transformations
carried out through sequential clauses to form a rule.

Figure 4: Structure of supported expressions

Using these concepts, the expressions supported by SynEvaluator are formally
defined as shown in Figure 4. The notation used is similar to $RegEx$ notation
with $*$ denoting zero to infinity, $+$ denoting one to infinity, and $?$
denoting zero or one occurrences. A ‘Subject’ comprises the beginning and is
always the first keyword in a rule in out proposed framework. It goes through
a series of transformations as defined by sequences of clauses. Clauses can be
of of two types: a) Extractive Clauses and b) Functional Clauses. Extractive
Clauses consist of (Predicate, Object) pairs that use the Predicate to execute
a transformation on the return value from the previous clause using the Object
as argument. More specifically, object specifies the type of element
‘extracted’ by the predicate, and elements satisfying this (Predicate, Object)
pair are returned as output. Functional clauses, on the other hand, consist of
(Predicate, Function) pairs that involve executing a function of type
described by predicate on the return value from the previous clause. These
clauses typically check for existence of a certain functional property and
thus return a boolean value in response.

Currently, two kinds of functional properties are supported: i) ontological
(or structural) properties, that execute ontology-level functions (such as
uniqueness, validity, consistency etc) on ontological elements, and ii)
linguistic properties that linguistically analyze text (such as checking for
polysemes, conjunctions etc) returned from previous clauses. In case of
‘False’ value returned by a Functional Clause or an empty list (no matching
elements) returned by an Extractive Clause, that particular ontological
element is returned as an element containing a violation.

Extractive Clauses can also be chained together to form clause sequences.
Since functional clauses return a boolean value, they cannot be chained
further. Clause sequences can be compared through Operator Expressions.
Operator Expressions essentially consist of a ‘Predicate’ indicating operator
type, followed by an ‘Operator’ keyword. Operator Expressions comprise of two
main categories of operators, namely: (a) Logical Operators and (b)
Comparative Operators. Logical Operators like ‘And’, ‘Or’ and ‘Not’ are used
to create logical combinations of clause sequences. Comparative Operators like
‘Equality’, ‘Inverse’ and ‘Synonymy’ are used to compare return values.

Table 1: Keywords for different expression types in the proposed framework Subject | Extractive Clause | Functional Clause | Operator Expression
---|---|---|---
Predicate | Predicate | Predicate
Ontology Metadata | Has Related Element (1) | Has Ontological Property (1) | Uses Comparative Operator (1)
Ontological Element | Has Attribute (2) | Has Linguistic Property (2) | Uses Logical Operator (2)
Class | Object | Function | Operator
Instance | Domain (1) | ID Consistency (1) | Equality (1)
Property | Subclass (1) | Uniqueness (1) | Inverse (1)
Object Property | Disjoint Class (1) | Text Validity (1) | And (2)
Datatype Property | ID (2) | Contains Polysemes (2) | Or (2)
Symmetric Property | Language (2) | Contains Conjunctions (2) | Not (2)

Table 1 summarizes keywords supported by the proposed framework. Note that the
table lists 8 Subject keywords, while for Extractive Clauses, Functional
clauses and operator expressions it lists 2 predicates and 5 objects,
functions and operators respectively. This is due to differing syntax followed
by each expression type. Also, every predicate has a list of valid
Objects/Functions/Operators. This is shown in the table through bracketed
numbering. For example, the valid Predicates for ‘Has Attribute‘ are ‘ID‘ and
‘Language‘. The complete list of supported keywords is provided over
here555https://bit.ly/3zfdI8f.

####  3.1.2 Examples of Created Rules

Figure 5 shows examples of 2 rules. In both examples 1 and 2, ‘Property’ is
the ‘Subject’ of the rule. ‘hasRelatedElement Domain’, ‘hasRelatedElement
Range’ and ‘hasOntologicalProperty Uniqueness’ are all clauses. In rule 1, the
‘hasRelatedElement Domain’ clause carries out a transformation that uses the
‘hasRelatedElement’ predicate to extract ‘Domain’ elements. A similar clause
is used for extracting ‘Range’ elements. These two clauses (or clause
sequences of length are combined using the operator expression
‘usesLogicalOperator And’. This rule thus necessitates non-null values for
both Domain and Range elements for each element of ‘Subject’, in this case,
‘Property’. Properties that do not contain both elements are therefore
returned as ontological elements containing violations.

Figure 5: Examples of rules implemented by SynEvaluator

Example 2 shows a rule where clauses have been chained together to constitute
a clause sequence of length 2. This sequence consists of the extractive clause
‘hasRelatedElement Domain’ followed by functional clause
‘hasOntologicalProperty Uniqueness’. The extractive clause essentially
extracts Domain element(s) of each Property. Then, the functional clause
applies ‘Uniqueness’ function on ontological elements returned by previous
clause with function type defined by Predicate “hasOntologicalProperty". In
case of non-existence of domain for a particular property or existence of
multiple domains, this rule would return that property as containing a
violation due to violation in first and second clauses respectively.

####  3.1.3 Proposing SynEvaluator: the final web application

Figure 6: The SynEvaluator interface. Subjects are highlighted in blue,
Clauses in yellow, Operator Expressions in green and Rule Priorities in red.
Among Clauses, Extractive Clauses are shown in solid lines while Functional
Clauses are outlined with dotted lines.

The rule creation framework proposed above is used to create a web application
for use by the ontology engineer. The primary interface of this application,
SynEvaluator, is shown in Figure 6. It allows the users (ontology engineers)
to use dropdown menus to create functional and extractive clauses, operator
expressions and thus, rules, as well as set priorities for these rules. Users
can choose between ‘Low’, ‘Medium’ and ‘High’ priorities based on the task-
specific importance of the rule. Figure 8 shows, with the help of an activity
diagram, how a user could create an appropriate rule using SynEvaluator.
Finally, after uploading the enriched ontology and creating the rules,
SynEvaluator parses the ontology using its parsing module and executes the
created rules (Figure 3). Post evaluation, the user is presented with a list
of violating elements along with count and priority of each valid rule.

####  3.1.4 Benefits and Limitations

The proposed framework makes it significantly easier for non-programmers to
create customised rules dynamically. Also, compared to previous quality
evaluation tools, due to the framework’s ability to reuse keywords to create
new rules, the developer effort required to hard-code rules is minimised.
Another major benefit is that the proposed framework can potentially be used
to query over entire OWL language. This can be done as any ontological
element/attribute can be extracted using extractive clauses and the
appropriate function executed on them. Lastly, due to functional clauses, it
is possible to execute ontological or ontology-level functions like normal
query languages and use linguistic analysis on text. This is particularly
useful in quality evaluation while applying appropriateness checks on IDs or
labels.

Figure 7: More examples of rules supported by SynEvaluator

In Figure 5 (Example 1), it can be seen that properties with missing domain or
range, a common quality violation, could be detected through a combination of
logical operators and extractive clauses. Another common violation is related
to the absence of annotations (comments, descriptions, etc.) for an
ontological element [34]. This can be converted into a rule, once again
through Extractive Clauses as shown in Figure 7 (Example 1). A third quality
violation would be when similar (or synonymous) classes are incorrectly
defined as equivalent classes [23]. This can be defined as in Figure 7
(Example 2), where the comparative operator ‘Dissimilarity’ is used to test
for semantic similarity through cosine similarity of label embeddings. As
mentioned previously, it is also possible to perform linguistic tests unlike
other query languages. A basic example of this is detection of conjunctions in
a label as shown in Figure 7(Example 3). This is useful to identify quality
violations where different concepts are merged in the same class using
conjunctions [28]. A more advanced example of a linguistic test would be a
rule detecting polysemous elements (Figure 7, Example 4). This is useful in
detecting violating elements that have labels denoting differing conceptual
ideas in differing senses. SynEvaluator implements this check through the use
of WordNet’s synsets to find out how many senses a word can have. Once these
rules are created, the ontology engineer can add domain/range elements;
annotations; remove synonymous equivalent classes and fix classes with
conjunctions and polysemes as appropriate. SynEvaluator can thus help in
fixing structural and linguistic quality violations.

The current version of SynEvaluator has a few limitations. It is currently
only possible to chain clauses together or use operators to compare chained
clauses. As a result, it is not possible to create rules with multiple lines.
One major consequence of this is that variable assignment is not supported,
and it is not possible to create a variable in one line and refer to it in
another, as part of the same rule. Aggregation operators, such as ‘Count‘ or
‘Sum‘, are currently not supported either. Finally, it is not possible to
create rules that require reasoning. The described limitations shall be
addressed in future iterations of SynEvaluator. In spite of the limitations,
the current framework (as shown in Experiments section) is still able to
support creation of the majority of quality evaluation rules.

Figure 8: Activity Diagram for creating a rule in SynEvaluator

###  3.2 Stage 2: SemValidator

SemValidator uses a crowdsourced approach to semantically evaluate ontologies.
The key feature of SemValidator is that it does not require ontology
engineers, domain experts or knowledge of OWL language for validation. This is
useful in a crowdsourced setting, where validators may have varying degrees of
expertise and knowledge. If further necessitates the use of appropriate
quality control mechanisms. To ensure quality, SemValidator uses TweetExpert
algorithm to calculate expertise score of a crowdsourced validator, which is
then used to weigh their decisions. This section describes the approach used
by TweetExpert and justifies the choices made. This is then followed by a
discussion on the assumptions made by TweetExpert and the feasibility of the
assumptions in the context of crowdsourcing. The section finally ends with a
description of the implementation of SemValidator and how it can be used by
crowdsourced validators.

####  3.2.1 TweetExpert algorithm

TweetExpert algorithm takes Twitter usernames of validators as input. For each
of the validators, it calculates two scores: a) a ‘TweetSim’ score and b) a
‘FriendSim’ score. A ‘TweetSim’ score is intended to assess the similarity of
the validator’s tweets to the domain of the ontology while ‘FriendSim’
estimates the domain similarity of the validator’s friends (pages they are
following). To calculate the ‘TweetSim’ score, the validator’s ‘n’ most recent
tweets are extracted from their profile and their semantic similarity with the
domain keyword is computed using the Universal Sentence Encoder (USE) [6].
Here the ‘domain keyword’ is manually chosen as the word most relevant to the
domain of the ontology. For instance, “Pizza" for Stanford Pizza ontology [8]
and “Information Security" for the ISO 27001 based “Information Security"
ontology [10]. The current implementation uses only one keyword, but it is
possible to compute similarity scores with multiple keywords and average these
scores for greater accuracy. The similarities are sorted in decreasing order.
Out of n similarities, the top-K similarities are chosen. These top-K
similarities are then averaged to yield ‘TweetSim’ score. A similar approach
is used to calculate ‘FriendSim’ score. The ‘m’ most recent friends are
extracted to calculate their ‘TweetSim’ scores. After sorting in decreasing
order, the top-K most similar scores are averaged to yield ‘FriendSim’ score.
The reason behind extracting the most recent tweets and friends is to get a
better estimate of the validator’s current knowledge and interests. On the
other hand, a top-K average helps in both filtering out occasional out-of-
domain tweets from domain experts and smoothening out the effects of
coincidental in-domain outliers from non-experts. The value of K is thus
appropriately empirically chosen such that it is large enough to not include
in-domain outliers from laymen, but small enough to exclude any out-of-domain
tweets from experts. The pseudocode for TweetExpert is shown in Algorithm 1.

Result: TweetExpert score

Function   _calculate_tweet_similarity(_$username$_)_:

user_tweets := extract_tweets(username)

tweet_similarities := calculate_USE_similarity(user_tweets, domain_name)

best_tweet_similarities := get_top_K_tweets(tweet_similarities)

tweet_similarity_score = average(best_tweet_similarities)

return tweet_similarity_score

Function   _calculate_friend_similarity(_$username$_)_:

user_friends := extract_friends(username)

user_friends_scores := foreach  _ $friend\in user\\_friends$_ do
calculate_tweet_similarity(friend)

best_friend_scores := get_top_K’_friends(user_friends_scores)

friend_similarity_score = average(best_friend_scores)

return friend_similarity_score

tweet_sim := calculate_tweet_relevance(username)

friend_sim := calculate_friend_relevance(username)

score := ML_predict_score(tweet_sim, friend_sim)

Algorithm 1 TweetExpert algorithm

Finally, the calculated ‘TweetSim’ and ‘FriendSim’ scores are input to a pre-
trained Machine Learning regression algorithm that predicts the final
$TweetExpert$ score using the two scores as feature vectors. The current
implementation uses Epsilon-Support Vector Regression (SVR), since it was
experimentally found to yield best results (shown in Experiments section).
However, the system uses strategy software design pattern [29] which enables
easy interchange of regression algorithms. The TweetExpert score is calculated
for each of the validators by repeating the process described above. The final
decision is taken using a weighted majority voting algorithm with the
TweetExpert scores being used as weights. The TweetExpert scores provide a way
to estimate a confidence value for the decisions input by each validator. This
is particularly crucial as a quality control metric in non-probabilistic
sampling techniques like crowdsourcing where number of validators can grow
exponentially in count and diversity.

####  3.2.2 Assumptions

SemValidator makes reasonably grounded assumptions to establish efficacy and
suitability in a crowdsourced setting. For example, about 49% of crowdsourced
workers whose primary source of income is Amazon Mechanical Turk, a popular
crowdsourcing platform, are in the 18-29 age
range666https://pewrsr.ch/3vX5q2G. As of February 2021, about 42% of Americans
in the 18-29 age range use Twitter, with this age group being the most active
demographic on Twitter777https://bit.ly/3z5IDn9. It is also assumed that in
order for expertise estimation to work, Twitter users tweet reasonably
frequently. The average number of tweets per day per user, according to a 2016
study888https://bit.ly/3fSjWmA, is 4.422, which translates to over 1600 tweets
a year. This is typically sufficient since $K\sim 50-100$. Finally,
SemValidator assumes workers have public Twitter profiles, which would enable
tweets and friends of a worker to be extracted. This assumption is predicated
upon a recent 2019 survey999https://bit.ly/3pmMc3O that showed 87% of Twitter
users in USA have public accounts. Also, as part of future work, SemValidator
would allow login through Facebook and Linkedin as well and a similar
algorithm for expertise estimation could be used. This is expected to further
increase the applicability of this expertise-based approach for crowdsourcing.

####  3.2.3 Proposing SemValidator: the final validation workbench

Figure 9: The main validation interface of SemValidator. The Stanford Pizza
ontology has been uploaded on SemValidator, with its concepts in blue and the
enriched concepts in green. One of the enriched concepts, “Tandoori Pizza" has
been selected, with options to “Accept" and “Reject" it.

The proposed workflow is used to develop a validation workbench for
crowdsourced workers that allows for accepting or rejecting enriched concepts,
relations and instances in an enriched ontology. The main validation interface
of this workbench, called SemValidator, is shown in Figure 9. This application
integrates Twitter authentication and uses the TweetExpert algorithm for
calculating validator expertise. SemValidator allows for two types of users:
(a) the administrator and (b) the validator. The administrator, typically the
ontology engineer, can upload/delete ontologies to be validated and also
access decisions made by validators. When the validator selects an ontology to
validate, the ontology is served using the WebVOWL ontology visualization
software. The enriched concepts and instances are highlighted in green, and on
selection, enable the validators to select accept/reject decisions
accordingly. Enriched relations, also in green, may also be accepted/rejected
independent of the concepts they relate. The validators’ decisions are
recorded by SemValidator and logged to a database. The administrator can
download this database after the crowdsourcing survey is completed, evaluate
validator expertise using their Twitter usernames and then make final
accept/reject decisions.

The syntactic and semantic evaluation aspects of the framework may be used
independently of each other. However, utilizing the framework in an integrated
manner as shown in Figure 3 is expected to give best results. This way,
syntactical violations can be detected easily and in an automated and
customizable manner, while semantic violations can now be detected more
accurately by crowdsourced validators. The resulting ontology has enhanced
syntactic and semantic ontological quality and is now fit for reuse.

##  4 Experiments

Stanford Pizza and ISO-IEC 27001 Information Security ontologies are evaluated
using the proposed framework to demonstrate ontology quality evaluation. Since
the focus of this work is on enriched ontologies, these are manually enriched
with concepts, properties and instances before quality evaluation. The RDF
triples used to enrich Pizza and Information Security ontologies are provided
over here101010https://bit.ly/3z5lGR1. Only 5 triples were chosen for this
iteration, considering that ontology enrichment is as an iterative process and
the count of triples per iteration is expected to be of this order. For Pizza,
the domain-specific webpages used for extraction consisted of culinary
articles111111https://bit.ly/3yYGGZP and food travel
blogs121212https://bit.ly/3iiEtCm. For Information Security they consisted of
informative articles and product pages by Cisco131313https://bit.ly/3wXRHZj,
Barracuda141414https://bit.ly/34PIUN9 and
SearchSecurity151515https://bit.ly/3wXRJQV. Please note that some relevant
data in this section is shown through external links due to space constraints.

###  4.1 Syntactic Quality Evaluation using SynEvaluator

This section attempts to evaluate the applicability and accuracy of
SynEvaluator in creating rules and detecting violations respectively. It is
hypothesized that SynEvaluator’s reusable theoretical framework for priority-
specific rule creation increases its applicability for diverse range of tasks,
but without compromising on violation-detection accuracy. To prove these
claims, SynEvaluator is compared against OOPS! [27], a popular, publicly
available SOTA tool that allows qualitative rule evaluation using rules.
Moreover, OOPS! compiles 41 most commonly observed pitfalls drawn from several
popular works on ontology quality evaluation [12], [23], [34] and the ones
described in the OOPS! catalogue [26] are chosen as rules for implementation.
This allows appropriate assessment of applicability for rule creation, while
also allowing accuracy evaluation by comparing SynEvaluator’s detected
violations to that of OOPS! Note that while SynEvaluator’s suitability for
rule creation is being assessed here by comparing with rule-based evaluation
approaches, it cannot be compared yet with other contemporary works focusing
on metric-based evaluation. This is being planned as part of future work by
introducing metric-to-rule conversion, which would allow metrics to be framed
as rules. One possible way this could be done is by adding conditions, such as
comparison operators, after metrics to form rules.

Table 2: No of pitfalls implemented by SynEvaluator (SE) Vs OOPS! SE OOPS! |  | Implemented
---
| Not implemented
---
| Implemented
---
29 | 4
| Not implemented
---
1 | 7

SynEvaluator’s framework can be used to successfully formulate 30 of the 41
pitfalls compiled in the catalogue, compared to OOPS! which can automate 33.
As shown in Table 2, the majority of pitfalls (29 of 41) can be successfully
implemented by both OOPS! and SynEvaluator. 7 cannot be implemented by either
of them, and this involves rules that require human knowledge and reasoning
(such as overspecialization of ontology hierarchy, usage of wrong relations,
etc.). 1 of the pitfalls (involving linguistic polysemy detection) could be
implemented by SynEvaluator but not by OOPS!. There are 4 pitfalls implemented
by OOPS! which SynEvaluator cannot check. This includes rules that check for
undeclared disjoint concepts, undeclared transitive properties, equivalent
classes, etc. Supporting such rules involves a higher degree of ontological
reasoning that SynEvaluator is incapable of. Nevertheless, SynEvaluator’s
ability to implement vast majority of quality evaluation rules off-the-shelf
in a customizable manner increases confidence in its applicability for future
evaluation tasks that may involve more rules.

For comparing accuracy of violation detection, SynEvaluator’s reported
violations are compared with OOPS! [27]. Given the absence of ground truth and
the widespread popularity of OOPS!, it is assumed that OOPS! uses valid, non-
erroneous rule checks for the implemented pitfalls. It can be observed that
the violations reported by SynEvaluator match those of OOPS! in all pitfalls
except for P22. OOPS! mentions that these could be due to inconsistent naming
conventions, however SynEvaluator is unable to detect any such errors in the
ontologies. This may be a result of the incorrect naming checks currently
used. But for the other pitfalls, both tools returning the same number of
violations suggests that for the implemented framework, SynEvaluator is
accurate in evaluating rules and detecting pitfalls.

###  4.2 Semantic Quality Validation using SemValidator

SemValidator uses a crowdsourced survey to semantically validate the enriched
Pizza and Information Security ontologies. Crowdsourced validators were
invited by tweeting a description of the task along with the link to the
SemValidator application. Among the 31 validators that participated in the
survey, 3 had private Twitter accounts so their responses were discarded. The
28 validators with public accounts included 5 validators with a background in
Cybersecurity, 4 in Artificial Intelligence, 5 in Software Engineering, 4 in
Healthcare, 3 in Food domain (such as chefs and culinary experts), 3 in
Literature, 2 in Politics and 1 each in Fashion and Pure Science. These
diverse bunch of validators were asked to self-identify their domain of
expertise for statistical purposes before participating in the survey. To
preserve validator anonymity, the anonymized results of the survey are
provided here161616https://bit.ly/3pvzbFh. After completion of the survey for
both ontologies, ‘TweetSim’ and ‘FriendSim’ scores were calculated for all 28
validators using the TweetExpert algorithm, described previously. TweetExpert
took 1̃5 seconds to execute for our values of $K=20$ and $K^{\prime}=5$ per
validator profile. Finally, TweetExpert score was calculated by experimenting
with 4 standard regression algorithms: Linear Regression, Random Forest
Regression, K-Nearest Neighbours (KNN) Regression and Epsilon-Support Vector
Regression (SVR). A standard majority voting where all decisions have equal
weightage is considered as a naive baseline.

The input features for each algorithm were ‘TweetSim’ and ‘FriendSim’ scores
of a validator and the label was the ratio of correct answers (to total
answers) given by them. This was done to ensure validators with more correct
answers received higher ‘TweetExpert’ scores using similarity scores as
feature vectors. Since a gold standard for the correct answers did not exist,
CISOs were asked to manually validate the enriched triples from the
information security domain, while the authors themselves validated the
triples from the pizza domain. To determine the best among the 4 regression
algorithms for score prediction, 7-fold cross validation was carried out. The
28 responses (now reduced to feature vectors and labels) available for both
ontologies were divided into 7 folds of 4 responses each. 5 folds were used
for training, 1 for validation and 1 for testing. For each chosen test fold,
immediately succeeding fold was chosen as the validation set and all the other
folds were used for training.

Different sets of hyperparameters were considered for each regression
algorithm171717https://bit.ly/3wUa9SA. The best set of hyperparameters for
each algorithm was determined by selecting the algorithm with highest accuracy
on the validation set, calculated by determining the proportion of correctly
answered questions to the total number of questions. The predicted answer to a
question is determined by using weighted majority voting algorithm on the
predicted expertise scores in the validation set. The higher the number of
correct answers, the closer the predicted scores are to their actual value.
The same procedure was followed for calculating the accuracy on the test
folds. Accuracy obtained on test and validation sets for each of these
algorithms is shown in Table 3 for both ontologies. Epsilon Support Vector
Regression was found to yield best results on both datasets.

To determine whether training score prediction algorithms requires feature
vectors from the same domain, an SVR model is trained on a more generic
dataset, i.e the Pizza dataset and tested on Information Security, a niche
dataset. The model hyperparameters are chosen as the ones that performed best
on the Information Security dataset earlier. Varying percentages of the Pizza
dataset are used to construct training subsets, to observe variation in test
accuracy with subset size. To ensure randomness while choosing these subsets,
10 experimental trials were conducted where the dataset is shuffled before
each trial and trained on the first $x\%$ of samples, where $x$ is the
percentage of the training dataset chosen. The results are shown in Table 4.
The accuracy monotonically increases with increase in size of training subset.
The results suggest that a model pre-trained to predict the expertise score on
one domain could be reused multiple times in different domains, even if they
are niche.

Table 3: Mean Validation and Test Accuracy scores on Pizza and Security datasets Algorithm | Pizza | Security
---|---|---
Val | Test | Val | Test
Majority Voting | 71.43 | 71.43 | 51.43 | 51.43
Linear Regression | 85.71 | 83.57 | 68.57 | 71.43
Random Forest | 80.0 | 81.43 | 68.57 | 74.28
KNN | 85.71 | 80.0 | 77.14 | 77.14
SVR | 88.57 | 85.71 | 80.0 | 80.0
Table 4: Mean accuracy scores on test dataset (Information Security) with variation in % of training dataset (Pizza). Dataset% | 10% | 25% | 50% | 75% | 100%
---|---|---|---|---|---
Accuracy | 70% | 76% | 86% | 92% | 100%

Table 3 shows that the best-performing algorithm (TweetExpert with SVR
followed by weighted majority voting) significantly outperforms naive majority
voting. It can be observed that naive majority voting gives particularly bad
results for Information Security, a relatively niche domain, than Pizza, a
domain known more to laymen as well. When replacing naive majority voting with
TweetExpert + SVR, a drastic increase in performance in Information Security
(55.55%) vs Pizza (20%) can be observed. Even the worst performing regression
algorithm gives an increase of 38.8% and 14% respectively. As a result, it can
be inferred that estimating expertise using TweetExpert can be particularly
useful for quality control in niche domains.

##  5 Conclusion and Future Work

Ontology evaluation is a critical stage of any ontology engineering process.
In this paper, SynEvaluator and SemValidator are proposed for syntactic and
semantic evaluation of an enriched ontology respectively. Although the work
focuses on an enriched ontology, it can be easily extended for evaluation of
any generic ontology.

SynEvaluator automatically evaluates ontologies using customised rules created
dynamically at runtime, and improves on previous quality evaluation approaches
in a variety of ways. Firstly, it offers greater flexibility to the user in
terms of creating, updating and deleting custom rules as well as setting
priorities. Secondly, it proposes rule creation by the usage of a novel
theoretical framework that factors rules into sequences of ‘clauses’ and
‘operator expressions’. This facilitates creation of an interactive interface
that makes it easier for non-programmers to dynamically create rules. In
addition, chaining together independent keywords can facilitate creation of a
large number of rules without requiring additional developer programming.
SemValidator improves on previously proposed crowdsourced ontology validation
approaches by incorporating a Twitter-based quality control mechanism. The
TweetExpert algorithm is proposed for calculating the expertise score of a
validator using the tweets and friends extracted from their Twitter profile as
input.

The efficacy of SynEvalautor was shown by implementing rules to detect
pitfalls and the accuracy of the detected violations was compared with
publicly available OOPS! tool. Semantic quality evaluation using SemValidator
is performed on both Pizza and Information security ontologies with the help
of a crowdsourced survey of 28 validators. The experimental results showed a
significantly higher than naive majority voting.

The experimental results are encouranging but also can be used as an aid to
further extend the research work. For example, SynEvaluator can be expanded to
support additional, more complex operations such as using parantheses,
arithmetic operators, variable assignment etc. Although the TweetExpert
algorithm used by SemValidator currently calculates expertise using only
tweets and friends as feature vectors, it could be extended to use additional
information such as Twitter lists, followers, tweet metadata, etc.

## References

  * [1] Alani, H., Brewster, C., Shadbolt, N.: Ranking ontologies with AKTiveRank. In: International Semantic Web Conference. pp. 1–15. Springer (2006)
  * [2] Amardeilh, F., Laublet, P., Minel, J.L.: Document Annotation and Ontology Population from Linguistic Extractions. In: Proceedings of the 3rd International Conference on Knowledge Capture. pp. 161–168 (2005)
  * [3] Beckett, D., Berners-Lee, T., Prud’hommeaux, E., Carothers, G.: Rdf 1.1 turtle. World Wide Web Consortium pp. 18–31 (2014)
  * [4] Berners-Lee, T., Hendler, J., Lassila, O.: The Semantic Web. Scientific American 284(5), 34–43 (2001)
  * [5] Burton-Jones, A., Storey, V.C., Sugumaran, V., Ahluwalia, P.: A semiotic metrics suite for assessing the quality of ontologies. Data & Knowledge Engineering 55(1), 84–102 (2005)
  * [6] Cer, D., Yang, Y., Kong, S.y., Hua, N., Limtiaco, N., John, R.S., Constant, N., Guajardo-Céspedes, M., Yuan, S., Tar, C., et al.: Universal Sentence Encoder. arXiv preprint arXiv:1803.11175 (2018)
  * [7] Dividino, R.Q., Romanelli, M., Sonntag, D., et al.: Semiotic-based Ontology Evaluation Tool (S-OntoEval). In: LREC (2008)
  * [8] Drummond, N.: Stanford pizza ontology. https://protege.stanford.edu/ontologies/pizza/pizza.owl, accessed: 2021-03-28
  * [9] Duque-Ramos, A., Fernández-Breis, J.T., Iniesta, M., Dumontier, M., Aranguren, M.E., Schulz, S., Aussenac-Gilles, N., Stevens, R.: Evaluation of the OQuaRE Framework for Ontology Quality. In: Expert Systems with Applications. vol. 40, pp. 2696–2703. Elsevier (2013)
  * [10] Fenz, S., Goluch, G., Ekelhart, A., Riedl, B., Weippl, E.: Information security fortification by ontological mapping of the iso/iec 27001 standard. In: 13th Pacific Rim International Symposium on Dependable Computing (PRDC 2007). pp. 381–388. IEEE (2007)
  * [11] Gangemi, A., Catenacci, C., Ciaramita, M., Lehmann, J.: A theoretical framework for ontology evaluation and validation. In: SWAP. vol. 166, p. 16 (2005)
  * [12] Gómez-Pérez, A.: Evaluation of taxonomic knowledge in ontologies and knowledge bases (1999)
  * [13] Guarino, N., Welty, C.: Evaluating ontological decisions with ontoclean. Communications of the ACM 45(2), 61–65 (2002)
  * [14] Hearst, M.A.: Automatic Acquisition of Hyponyms from Large Text Corpora. In: Proceedings of the 14th conference on Computational linguistics. vol. 2, pp. 539–545. Association for Computational Linguistics (1992)
  * [15] Iyer, V., Mohan, L., Bhatia, M., Reddy, Y.R.: A survey on ontology enrichment from text. In: Proceedings of the 16th International Conference on Natural Language Processing. pp. 95–104. NLP Association of India, International Institute of Information Technology, Hyderabad, India (Dec 2019), https://aclanthology.org/2019.icon-1.11
  * [16] Kiptoo, C.C.: Ontology enhancement using crowdsourcing: a conceptual architecture. International Journal of Crowd Science (2020)
  * [17] Kontokostas, D., Zaveri, A., Auer, S., Lehmann, J.: Triplecheckmate: A tool for crowdsourcing the quality assessment of linked data. In: International Conference on Knowledge Engineering and the Semantic Web. pp. 265–272. Springer (2013)
  * [18] Lantow, B.: OntoMetrics: Putting Metrics into Use for Ontology Evaluation. In: KEOD. pp. 186–191 (2016)
  * [19] Lozano-Tello, A., Gómez-Pérez, A.: Ontometric: A Method to Choose the Appropriate Ontology. In: Journal of Database Management. vol. 2, pp. 1–18. IDEA Group Publishing (2004)
  * [20] Maedche, A., Staab, S.: Ontology Learning for the Semantic Web. Intelligent Systems 16(2), 72–79 (2001)
  * [21] Makki, J., Alquier, A.M., Prince, V.: An nlp-based ontology population for a risk management generic structure. In: Proceedings of the 5th international conference on Soft computing as transdisciplinary science and technology. pp. 350–355 (2008)
  * [22] McDaniel, M., Storey, V.C., Sugumaran, V.: Assessing the Quality of Domain Ontologies: Metrics and an Automated Ranking System. Data & Knowledge Engineering 115, 32–47 (2018)
  * [23] Noy, N.F., McGuinness, D.L., et al.: Ontology development 101: A guide to creating your first ontology (2001)
  * [24] Noy, N.F., Mortensen, J., Musen, M.A., Alexander, P.R.: Mechanical turk as an ontology engineer? using microtasks as a component of an ontology-engineering workflow. In: Proceedings of the 5th Annual ACM Web Science Conference. pp. 262–271 (2013)
  * [25] Pittet, P., Barthélémy, J.: Exploiting users’ feedbacks: Towards a task-based evaluation of application ontologies throughout their lifecycle. In: International conference on knowledge engineering and ontology development. vol. 2 (2015)
  * [26] Poveda, M.: Catalogue of common pitfalls. http://oops.linkeddata.es/catalogue.jsp, accessed: 2021-03-28
  * [27] Poveda-Villalón, M., Gómez-Pérez, A., Suárez-Figueroa, M.C.: Oops!: A pitfall-based system for ontology diagnosis. In: Innovations, Developments, and Applications of Semantic Web and Information Systems, pp. 120–148. IGI Global (2018)
  * [28] Poveda-Villalón, M., Suárez-Figueroa, M.C., Gómez-Pérez, A.: Validating Ontologies with OOPS! In: International Conference on Knowledge Engineering and Knowledge Management. pp. 267–281. Springer (2012)
  * [29] Richard, E., RALPH, H., JOHNSON, R., et al.: Design patterns: Elements of reusable object-oriented software (1995)
  * [30] Sanagavarapu, L.M., Iyer, V., Reddy, Y.R.: OntoEnricher: A Deep Learning Approach for Ontology Enrichment from Unstructured Text. arXiv preprint arXiv:2102.04081 (2021)
  * [31] Schober, D., Tudose, I., Svatek, V., Boeker, M.: Ontocheck: verifying ontology naming conventions and metadata completeness in protégé 4. In: Journal of biomedical semantics. vol. 3, pp. 1–10. BioMed Central (2012)
  * [32] Tartir, S., Arpinar, I.B., Sheth, A.P.: Ontological Evaluation and Validation. In: Theory and applications of ontology: Computer applications, pp. 115–130. Springer (2010)
  * [33] Tibaut, A.: Ontology Evaluation. https://github.com/atibaut/ontology-evaluation (Online; accessed on 15-Mar-2021)
  * [34] Vrandečić, D.: Ontology Evaluation. In: Handbook on Ontologies, pp. 293–313. Springer (2009)
  * [35] Wohlgenannt, G., Sabou, M., Hanika, F.: Crowd-based ontology engineering with the ucomp protégé plugin. Semantic Web 7(4), 379–398 (2016)
  * [36] Wong, W., Liu, W., Bennamoun, M.: Ontology Learning from Text: A Look Back and into the Future. ACM Computing Surveys (CSUR) 44(4), 1–36 (2012)
  * [37] Zhang, Y., Saberi, M., Chang, E.: Semantic-based lightweight ontology learning framework: a case study of intrusion detection ontology. In: Proceedings of the international conference on web intelligence. pp. 1171–1177 (2017)