# Interval Logic Tensor Networks

Samy Badreddine1 Gianluca Apriceno2,3 Andrea Passerini3 Luciano Serafini2
1Sony AI, Tokyo, Japan
2Fondazione Bruno Kessler, Trento, Italy
3University of Trento, Trento, Italy
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

In this paper, we introduce Interval Real Logic (IRL), a two-sorted logic that
interprets knowledge such as sequential properties (traces) and event
properties using sequences of real-featured data. We interpret connectives
using fuzzy logic, event durations using trapezoidal fuzzy intervals, and
fuzzy temporal relations using relationships between the intervals’ areas. We
propose Interval Logic Tensor Networks (ILTN), a neuro-symbolic system that
learns by propagating gradients through IRL. In order to support effective
learning, ILTN defines smoothened versions of the fuzzy intervals and temporal
relations of IRL using softplus activations. We show that ILTN can
successfully leverage knowledge expressed in IRL in synthetic tasks that
require reasoning about events to predict their fuzzy durations. Our results
show that the system is capable of making events compliant with background
temporal knowledge.

##  1 Introduction

Event detection (ED) from sequences of data is a critical challenge in various
fields, including surveillance (Clavel et al., 2005), multimedia processing
(Xiang and Wang, 2019; Lai, 2022), and social network analysis (Cordeiro and
Gama, 2016). Neural network-based architectures have been developed for ED,
leveraging various data types such as text, images, social media data, and
audio. Integrating commonsense and structural knowledge about events and their
relationships can significantly enhance machine learning methods for ED. For
example, in analyzing a soccer match video, the knowledge that a red card
shown to a player is typically followed by the player leaving the field can
aid in event detection. Additionally, knowledge about how simple events
compose complex events is also useful for complex event detection.

Background knowledge has been shown to improve the detection of complex events
especially when training data is limited (Yin et al., 2020). Some approaches
show how knowledge expressed in first-order logic (Vilamala et al., 2023;
Apriceno et al., 2021, 2022) can be exploited for complex event detection.
Other approaches use temporal logic, such as LTLf, to embed temporal
properties in deep-learning architectures processing image sequences (Umili et
al., 2022).

To the best of our knowledge, all existing methods that incorporate background
temporal knowledge in event detection adopt a point-wise approach, defining
events based on properties that hold (or do not hold) at specific time points
during the event’s duration. However, the knowledge representation and formal
ontology literature advocates for event-centric representations, where events
are treated as ”first-class citizens” with properties that cannot be expressed
solely in terms of time-point properties (Kowalski and Sergot, 1986; Allen,
1983; Mueller, 2008).

The traditional perspective of event representation characterizes events as
crisp entities and represents the duration of an event, which is the time span
during which it occurs, as a convex subset of integers or real numbers.
However, this approach does not account for events that have smooth beginnings
or endings, such as a snowfall. Furthermore, even crisp events can benefit
from fuzzy semantics in representing relations between them. For example, the
statement ”Darwin (1809-1882) lived before Einstein (1879-1955)” is not as
true as ”March 13 comes before March 14,” but it is also not entirely false.
To address this limitation, knowledge representation formalisms have been
proposed for fuzzy intervals and fuzzy relations between them (Ohlbach, 2004;
Schockaert et al., 2008).

This paper introduces a novel logical framework that enables the specification
of dynamically changing propositions, as well as properties and relations
between events. We refer to this framework as Interval Real Logic, which is an
extension of Real Logic (Badreddine et al., 2022). This logic is designed to
capture knowledge properties and relations between objects that evolve over
time, including properties and relations between events. Interval Real Logic
is interpreted in the domain of real-data sequences, where objects are
associated with trajectories, and events are associated with the objects that
participate in the event, as well as the temporal interval during which the
event occurs.

In addition, the paper introduces the differentiable implementation of
Interval Real Logic in a neuro-symbolic architecture, Interval Logic Tensor
Networks (ILTN), to detect events from data sequences using background
knowledge expressed in Interval Real Logic. To effectively propagate gradients
through the logic, we propose modified trapezoidal fuzzy membership functions
and temporal relations for fuzzy intervals that overcome vanishing gradient
issues. We present a prototype implementation of ILTN and conduct basic
experiments that yield promising and positive results.

The rest of the paper is organised as follows: Section 2 presents related work
on fuzzy temporal knowledge and neuro-symbolic approaches for event detection.
Section 3 defines the language and the semantics of ILTN. Section 4 defines
fuzzy trapezoidal intervals and their temporal relations. In Section 5, the
neural architecture used to predict fuzzy events is described. In section 6,
the results on artificial experiments are discussed. Finally, in Section 7
conclusions are drawn and directions for future works are briefly outlined.

##  2 Related Work

Modeling and reasoning about temporal knowledge is a well-studied problem
(Kahn and Gorry, 1977; Allen, 1983; Allen and Hayes, 1985; Jong et al., 1999).
Temporal logics like Linear Temporal Logic (LTL) (Pnueli, 1977) and
Computational Tree Logic (CTL) (Clarke and Emerson, 1982) assume that the
underlying (temporal) information is crisp, and do not consider that the
knowledge may be characterized by vagueness and uncertainty. Following the
seminal work of Zadeh (1965) on fuzzy sets, different works have been proposed
to model both vagueness and uncertainty of temporal knowledge when this is
expressed in terms of events and their relations via a fuzzy interval-based
temporal model (Dubois and Prade, 1989; Nagypal and Motik, 2003; Ohlbach,
2004; Schockaert and Cock, 2008). These works however are not capable of
processing low level information in an efficient way, and do not consider any
learning. Indeed, fuzzy event recognition applications (Kapitanova et al.,
2012; Dima et al., 2012; Muduli et al., 2018) simply rely on a (fuzzy) rule-
based decision system.

Recently, neuro-symbolic approaches (Hitzler and Sarker, 2022), which
integrate sub-symbolic and symbolic reasoning and allow to effectively
integrate learning and reasoning, have been applied in the context of event
recognition. A common solutions consists in introducing a symbolic layer
refining the output of a pre-trained neural network (Khan et al., 2019a, b;
Xing et al., 2019; Vilamala et al., 2019; Gómez et al., 2020). In (Xing et
al., 2020), the symbolic layer is replaced by a neural network trained via
knowledge distillation to emulate symbolic reasoning. The drawback is that
this ”neuro-symbolic” layer has to be re-trained from scratch even for a
slight change of the knowledge. More recently, fully end-to-end differentiable
neuro-symbolic architectures have been proposed, by encoding temporal
reasoning primitives into existing frameworks like DeepProbLog (Vilamala et
al., 2021; Apriceno et al., 2021) or Learning Modulo Theories (Apriceno et
al., 2022). However, all these approaches reason in terms of time points,
making them incapable of fully expressing the properties of temporal events.
The solution we propose here aims to overcome these limitations by directly
focusing on temporal intervals.

LTN (Badreddine et al., 2022) is an end-to-end neuro-symbolic approach based
on fuzzy logic where prior domain knowledge is expressed in terms of Real
Logic formulas and interpreted using fuzzy logic semantics. LTN has been
applied successfully to solve structured tasks like semantic image
interpretation (Donadello et al., 2017) and to improve state of the art object
classifiers (Manigrasso et al., 2021). A first temporal extension of LTN has
been proposed by Umili et al. (2022), where Linear Temporal Logic over finite
traces (LTLf) formulas are translated to fuzzy deterministic automaton and
applied to solve a sequence classification task. However, as for the other
previously mentioned neuro-symbolic approaches, LTLf reasons in terms of time
points and thus shares their limitations. By extending LTN to deal with
(fuzzy) interval logic primitives we aim to allow them to effectively and
efficiently process temporal sequences towards complex event recognition.

##  3 Interval Real Logic

Let $\mathcal{L}_{t}$ be a first-order language that includes terms referring
to the _trajectories_ of objects over time. The syntax for terms and formulas
in $\mathcal{L}_{t}$ follows the standard syntax of first-order logic.

Similarly, let $\mathcal{L}_{e}$ be a first-order language, referred to as the
_language of events_ , which includes a set of symbols $e_{1},e_{2},\dots$
each associated with an arity $m\geq 0$. The terms of $\mathcal{L}_{e}$ are
expressed in the form $e(t_{1},\dots,t_{n})$ if $e$ has arity $m$ and
$t_{i}$’s are terms in $\mathcal{L}_{t}$. Intuitively, $e(t_{1},\dots,t_{m})$
denotes an event that involves $t_{1},\dots,t_{m}$ as participants.
Additionally, we assume that $\mathcal{L}_{e}$ contains the set of binary
predicates that correspond to binary relations between events.

######  Example 1.

Suppose that we want to describe the events that happen when two particles
move in a 2D space as shown in Figure 1.

$x$$z$$0$$2$$4$$6$$8$$10$$0$$2$$4$$\bullet$$\bullet$$\bullet$$\bullet$$\bullet$$\bullet$$\bullet$$\bullet$$\bullet$$\bullet$$\bullet$$\bullet$$\bullet$$\bullet$$\bullet$$\bullet$$\bullet$$\bullet$$\bullet$$\bullet$$\bullet$$\bullet$$\bullet$$\bullet$$\bullet$$\bullet$$\bullet$$\bullet$$\bullet$$\bullet$$\bullet$$\bullet$$0^{\prime\prime}$$5^{\prime\prime}$$10^{\prime\prime}$$15^{\prime\prime}$$20^{\prime\prime}$$0^{\prime\prime}$$5^{\prime\prime}$$10^{\prime\prime}$ab
Figure 1:  Trajectories of two particles in a 2D space. Several events happen
over time. For example, around time $5^{\prime\prime}$, the two particles
intersect. From time $0^{\prime\prime}$ to $10^{\prime\prime}$, particle $a$
rises whereas particle $b$ accelerates from left to right. Additionally, over
the whole trajectory, particle $a$ is doing a jump while changing color.

$\mathcal{L}_{t}$ and $\mathcal{L}_{e}$ are used conjointly to describe Figure
1. In $\mathcal{L}_{t}$, the two particles are denoted by constants a and b.
Unary predicates such as blue, red, and violet are included to describe the
particles’ colors over time. The atomic formula blue(a) expresses that a is
blue, with its truth value being time-dependent. To describe the proximity of
the particles, $\mathcal{L}_{t}$ uses the binary predicate close, and
close(a,b) is true around time $5^{\prime\prime}$ and false otherwise.

In $\mathcal{L}_{e}$, event symbols are used to describe the events in the
figure. For example, e0(a) can denote the jump of particle a, e1(a) can denote
the color change of a, and e2(a,b) can denote the event of a and b
intersecting. Predicates and functions on events are also included in
$\mathcal{L}_{e}$. For example, unary predicates on events can be used to
specify their types, as in the formula Jump(e0(a)), which states that e0(a) is
of type jump, and ChangeOfColor(e1(a)), which states that e1(a) is of type
color change.

We require that $\mathcal{L}_{e}$ contains the unary functions on events and
the binary relations of events shown in Table 1. In the table and in the rest
of the paper we use $\epsilon$ (possibly with indices) to denote an event term
$e(t_{1},\dots,t_{m})$.

Function symbols of $\mathcal{L}_{e}$
---
$\operatorname{\textsf{before}}(\epsilon)$ | what happens before the starting of $\epsilon$
$\operatorname{\textsf{after}}(\epsilon)$ | what happens after the end of $\epsilon$
$\operatorname{\textsf{start}}(\epsilon)$ | the starting of $\epsilon$
$\operatorname{\textsf{end}}(\epsilon)$ | the end of $\epsilon$
$[i,j]$ | for $i\leq j\in\mathbb{N}$
Allen’s predicate symbols of $\mathcal{L}_{e}$
$\epsilon_{1}\ \textsf{bf}\ \epsilon_{2}$ |  $\epsilon_{1}$ happens before $\epsilon_{2}$
$\epsilon_{1}\ \textsf{af}\ \epsilon_{2}$ |  $\epsilon_{1}$ happens after $\epsilon_{2}$
$\epsilon_{1}\ \textsf{mt}\ \epsilon_{2}$ |  $\epsilon_{2}$ happens immediately after $\epsilon_{1}$
$\epsilon_{1}\ \textsf{ol}\ \epsilon_{2}$ | the end of $\epsilon_{1}$ overlaps the start of $\epsilon_{2}$
$\epsilon_{1}\ \textsf{st}\ \epsilon_{2}$ |  $\epsilon_{1}$ is a starting part of $\epsilon_{2}$
$\epsilon_{1}\ \textsf{dr}\ \epsilon_{2}$ |  $\epsilon_{1}$ happens during $\epsilon_{2}$
$\epsilon_{1}\ \textsf{fin}\ \epsilon_{2}$ |  $\epsilon_{1}$ is an ending part of $\epsilon_{2}$
$\epsilon_{1}\ \textsf{eq}\ \epsilon_{2}$ |  $\epsilon_{1}$ is equal to $\epsilon_{2}$
Other predicate symbols of $\mathcal{L}_{e}$
$\operatorname{\textsf{H}}(\epsilon)$ | the event $\epsilon$ actually happened
$\epsilon_{1}\ \textsf{in}\ \epsilon_{2}$ |  $\epsilon_{1}$ is contained in $\epsilon_{2}$
Table 1: Basic functions and relations on events

Finally, $\mathcal{L}_{t}$ contains a unary predicate $\mathrm{Active}$ that
takes as input an event term. Intuitively, $\mathrm{Active}(\epsilon)$ returns
for every time step of the sequence if the event is running or not.

######  Example 2.

Following are some examples of formulas in $\mathcal{L}_{t}$ and
$\mathcal{L}_{e}$. The atomic formula

$\displaystyle\textsf{Sunny(weather)}\rightarrow\textsf{Happy(John)}$

is an example of a $\mathcal{L}_{t}$ formula that states that John is happy
whenever it is sunny. This formula is evaluated along two traces, one for the
weather and one for John, and can take different values at different time
points.

The following $\mathcal{L}_{e}$ formula

$\displaystyle\operatorname{\textsf{H}}(\textsf{e1(John,Mary)})\wedge\textsf{Meeting(e1(John,Mary))}$

states that a meeting between John and Mary happened.

The $\mathcal{L}_{t}$ formula

$\displaystyle\mathrm{Active}(\textsf{e1(John,Mary)})\rightarrow\textsf{Happy(John)}\wedge\textsf{Happy(Mary)}$

expresses that during the meeting between John and Mary, they were both happy.

The $\mathcal{L}_{e}$ formula

$\displaystyle\forall_{t}x,y.\textsf{Meeting}(\textsf{e1}(x,y))\rightarrow\textsf{e1}(x,y)=\textsf{e1}(y,x)$

states that in a meeting event, the roles of the participants are symmetric.
Notice that the quantification is on trace variables (not on the events). This
is highlighted by the index $t$ of the universal quantifier.

Finally, the $\mathcal{L}_{e}$ formula

$\displaystyle\forall_{e}x.\textsf{Meeting}(x)\rightarrow\exists_{e}y.\textsf{PrepareAgenda}(y)\wedge
x\ \textsf{bf}\ y$

expresses that before every meeting there should be an event that is the
preparation of the agenda. In this case, the quantification is on event
variables, indicated by the index $e$ of the quantifier.

The semantics of the trace logic $\mathcal{L}_{t}$ and the event-based logic
$\mathcal{L}_{e}$ are defined in the context of a linear discrete structure,
which models the progression of time. We use the natural numbers $\mathbb{N}$
with the standard order $<$ as the reference structure for time.

###  3.1 Trace Semantics

In $\mathcal{L}_{t}$, terms are interpreted as (possibly infinite) sequences
of data, called _trajectories_. For each time point $i\in\mathbb{N}$, an
$\mathcal{L}_{t}$ term corresponds to a feature vector in $\mathbb{R}^{n}$.
Specifically, a trajectory is a function
$\bm{t}:\mathbb{N}\rightarrow\mathbb{R}^{n}$ that assigns a feature vector in
$\mathbb{R}^{n}$ to every time point. We denote the set of trajectories with
features in $\mathbb{R}^{n}$ as $\mathbb{T}^{n}$. Trace variables in
$\mathcal{L}_{t}$ refer to variables of individuals and are associated with
batches of traces. Constants and closed terms (i.e., terms without variables)
in $\mathcal{L}_{t}$ are interpreted as single traces.

Formulas in $\mathcal{L}_{t}$ are evaluated at all time instants. For every
time $i\in\mathbb{N}$, an $\mathcal{L}_{t}$ formula is associated with a truth
value in the range $[0,1]$ that represents the level of truth of the formula
at that time. As a result, an $\mathcal{L}_{t}$ formula is interpreted as a
sequence of truth values, which we refer to as a function from $\mathbb{N}$ to
$[0,1]$. The set of such functions is denoted as $\mathbb{B}$.

The formal definition of the semantics for $\mathcal{L}_{t}$ is based on a
_grounding_ function $\mathcal{G}$ that must satisfy the following conditions:

  * •

for every variable $x$ in $\mathcal{L}_{t}$,
$\mathcal{G}(x)\in(\mathbb{T}^{n})^{b}$ is a batch of trajectory with the
integer size $b\geq 1$,

  * •

for every constant $c\in\mathcal{L}_{t}$, $\mathcal{G}(c)\in\mathbb{T}^{n}$ is
a single trajectory,

  * •

for every function $f\in\mathcal{L}_{t}$, with arity equal to $m$,
$\mathcal{G}(f):\mathbb{T}^{n_{1}}\times\dots\times\mathbb{T}^{n_{m}}\rightarrow\mathbb{T}^{n}$,
that is $\mathcal{G}(f)$ maps to a function that takes $m$ input trajectories
and returns a trajectory,

  * •

for every predicate $p\in\mathcal{L}_{t}$, with arity equal to $m$,
$\mathcal{G}(p):\mathbb{T}^{n_{1}}\times\dots\times\mathbb{T}^{n_{m}}\rightarrow\mathbb{B}$,
that is $\mathcal{G}(p)$ maps to a function that takes $m$ input trajectories
and outputs a function from time points to truth values in $[0,1]$.

Propositional connectives are interpreted according to fuzzy logic semantics
which is applied point-wise. For example, if $\phi$ and $\psi$ are
$\mathcal{L}_{t}$-formulas, then
$\mathcal{G}(\phi\wedge\psi)=T(\mathcal{G}(\phi),\mathcal{G}(\psi))=\\{T(\mathcal{G}_{i}(\phi),\mathcal{G}_{i}(\phi))\\}_{i\in\mathbb{N}}$,
where $T$ is a t-norm such as the product t-norm. Universal and existential
quantifiers are interpreted as aggregation operators. For example,
$\mathcal{G}(\forall x\phi(x))=\\{\prod_{1\leq j\leq
b}\mathcal{G}_{i}(\phi(\mathcal{G}_{j}(x)))\\}_{i\in\mathbb{N}}$.

Finally, we allow a special predicate that maps from events to
$\mathcal{L}_{t}$:

  * •

for every event $\epsilon$,
$\mathcal{G}(\mathrm{Active}(\epsilon)):\mathbb{E}^{\bm{n}}\rightarrow\mathbb{B}$;
$i\mapsto T(\mathcal{I}(\epsilon)(i),\operatorname{\textsf{H}}(\epsilon))$
where $T$ is a t-norm. The functions $\mathcal{I}$ and
$\operatorname{\textsf{H}}$, as well as the notation $\mathbb{E}^{\bm{n}}$,
are defined in Section 3.2. Intuitively, $\mathrm{Active}(\epsilon)$ maps an
event to a boolean trajectory that states _when_ and _if_ the event happens at
each timepoint of the trajectory.

###  3.2 Event Semantics

An event is seen as a potentially infinite sequence of data, (i.e., a
trajectory) and a mask that indicates the duration of the event. Formally, an
event
$\epsilon\in\mathbb{T}^{n_{1}}\times\dots\times\mathbb{T}^{n_{m}}\times\mathbb{B}$
consists of $m$ traces, which are the traces of the objects involved in the
event $\epsilon$, and a boolean trace that indicates when the event is active.
Specifically, let $\mathcal{I}(\epsilon)$ denote the boolean trace
$\mathbb{B}$ that is the activation sequence of $\epsilon$. If
$\bm{n}=(n_{1},\dots,n_{m})$, we denote $\mathbb{E}^{\bm{n}}$ as
$\mathbb{T}^{n_{1}}\times\dots\times\mathbb{T}^{n_{m}}\times\mathbb{B}$, which
represents the space of events involving $m$ objects, each with features in
$\mathbb{R}^{n_{i}}$. The formal semantics of $\mathcal{L}_{e}$ is defined in
reference to the definition of an event provided in Guarino et al. (2022) and
is given in terms of a function $\mathcal{G}$ that satisfies the following
restrictions.

  * •

For every event term $e(t_{1},\dots,t_{m})$,
$\mathcal{G}(e(t_{1},\dots,t_{m}))\in\mathbb{E}^{\bm{n}}$ where
$\bm{n}=(n_{1},\dots,n_{m})$ and $\mathcal{G}(t_{i})\in\mathbb{R}^{n_{i}}$ for
$1\leq i\leq m$,

  * •

for every $[i,j]\in\mathbb{N}$,
$\mathcal{G}(i)=\\{\mathbbm{1}_{n\in[i,j]}\\}_{n\in\mathbb{N}}$,

  * •

for every function symbol $f\in\mathcal{L}_{e}$, with arity equal to $m$,
$\mathcal{G}(f):(\mathbb{E}^{\bm{n}_{1}}\times\dots\times\mathbb{E}^{\bm{n}_{m}})\rightarrow\mathbb{E}^{\bm{n}_{1}\cdots\bm{n}_{m}}$,

  * •

for every predicate symbol $p\in\mathcal{L}_{e}$, with arity equal to $m$,
$\mathcal{G}(p):(\mathbb{E}^{\bm{n}_{1}}\times\dots\times\mathbb{E}^{\bm{n}_{m}})\rightarrow[0,1]$.

Connectives in $\mathcal{L}_{e}$ are interpreted using fuzzy semantics. For
example,
$\mathcal{G}(\phi_{1}\wedge\phi_{2})=T(\mathcal{G}(\phi_{1}),\mathcal{G}(\phi_{2}))$
where $T$ is a t-norm. Quantifiers of events are interpreted by aggregation
functions.

######  Example 3.

The first segment of the grounding $\mathcal{G}$ of the particle $a$ of Figure
1 is shown in Figure 2

$\displaystyle\begin{array}[]{|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|}\hline\cr\rotatebox{90.0}{$i$}&&&0&1&2&3&4&5&6\\\
\hline\cr\hbox{\multirowsetup\rotatebox{90.0}{$\mathcal{G}(\textsf{e2(a,b)})$}}&\hbox{\multirowsetup$\mathcal{G}(\textsf{a})$}&x&2.0&2.04&2.15&2.33&2.57&2.88&3.24\\\
&&y&1.0&1.47&1.93&2.36&2.76&3.12&3.43\\\ &&r&1.0&0.95&0.9&0.85&0.8&0.75&0.7\\\
&&g&0.0&0.0&0.0&0.0&0.0&0.0&0.0\\\ &&b&0.0&0.05&0.1&0.15&0.2&0.25&0.3\\\
\cline{2-10}\cr&\hbox{\multirowsetup$\mathcal{G}(\textsf{b})$}&x&1.0&1.02&1.08&1.18&1.32&1.5&1.72\\\
&&y&4.0&4.0&4.0&4.0&4.0&4.0&4.0\\\ &&r&1.0&0.95&0.9&0.85&0.8&0.75&0.7\\\
&&g&0.0&0.05&0.1&0.15&0.2&0.25&0.3\\\ &&b&0.0&0.0&0.0&0.0&0.0&0.0&0.0\\\
\cline{2-10}\cr&\mathcal{I}&&\leavevmode\hbox to8.94pt{\vbox
to0.4pt{\pgfpicture\makeatletter\hbox{\hskip 0.2pt\lower-0.2pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{
}\nullfont\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{{}}{}
{}{{}}{}{}{}{}{{}}{}\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgffillcolor}{rgb}{1,0,0}\pgfsys@color@rgb@fill{1}{0}{0}\pgfsys@invoke{
}{}\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@lineto{0.0pt}{0.0pt}\pgfsys@lineto{8.5359pt}{0.0pt}\pgfsys@lineto{8.5359pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{8.5359pt}{0.0pt}\pgfsys@fillstroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}&\leavevmode\hbox
to8.94pt{\vbox to3.25pt{\pgfpicture\makeatletter\hbox{\hskip
0.2pt\lower-0.2pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{
}\nullfont\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{{}}{}
{}{{}}{}{}{}{}{{}}{}\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgffillcolor}{rgb}{1,0,0}\pgfsys@color@rgb@fill{1}{0}{0}\pgfsys@invoke{
}{}\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@lineto{0.0pt}{2.84544pt}\pgfsys@lineto{8.5359pt}{2.84544pt}\pgfsys@lineto{8.5359pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{8.5359pt}{2.84544pt}\pgfsys@fillstroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}&\leavevmode\hbox
to8.94pt{\vbox to8.94pt{\pgfpicture\makeatletter\hbox{\hskip
0.2pt\lower-0.2pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{
}\nullfont\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{{}}{}
{}{{}}{}{}{}{}{{}}{}\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgffillcolor}{rgb}{1,0,0}\pgfsys@color@rgb@fill{1}{0}{0}\pgfsys@invoke{
}{}\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@lineto{0.0pt}{8.5359pt}\pgfsys@lineto{8.5359pt}{8.5359pt}\pgfsys@lineto{8.5359pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{8.5359pt}{8.5359pt}\pgfsys@fillstroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}&\leavevmode\hbox
to8.94pt{\vbox to13.2pt{\pgfpicture\makeatletter\hbox{\hskip
0.2pt\lower-0.2pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{
}\nullfont\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{{}}{}
{}{{}}{}{}{}{}{{}}{}\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgffillcolor}{rgb}{1,0,0}\pgfsys@color@rgb@fill{1}{0}{0}\pgfsys@invoke{
}{}\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@lineto{0.0pt}{12.80365pt}\pgfsys@lineto{8.5359pt}{12.80365pt}\pgfsys@lineto{8.5359pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{8.5359pt}{12.80365pt}\pgfsys@fillstroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}&\leavevmode\hbox
to8.94pt{\vbox to14.63pt{\pgfpicture\makeatletter\hbox{\hskip
0.2pt\lower-0.2pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{
}\nullfont\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{{}}{}
{}{{}}{}{}{}{}{{}}{}\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgffillcolor}{rgb}{1,0,0}\pgfsys@color@rgb@fill{1}{0}{0}\pgfsys@invoke{
}{}\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@lineto{0.0pt}{14.22638pt}\pgfsys@lineto{8.5359pt}{14.22638pt}\pgfsys@lineto{8.5359pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{8.5359pt}{14.22638pt}\pgfsys@fillstroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}&\leavevmode\hbox
to8.94pt{\vbox to7.51pt{\pgfpicture\makeatletter\hbox{\hskip
0.2pt\lower-0.2pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{
}\nullfont\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{{}}{}
{}{{}}{}{}{}{}{{}}{}\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgffillcolor}{rgb}{1,0,0}\pgfsys@color@rgb@fill{1}{0}{0}\pgfsys@invoke{
}{}\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@lineto{0.0pt}{7.11319pt}\pgfsys@lineto{8.5359pt}{7.11319pt}\pgfsys@lineto{8.5359pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{8.5359pt}{7.11319pt}\pgfsys@fillstroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}&\leavevmode\hbox
to8.94pt{\vbox to0.4pt{\pgfpicture\makeatletter\hbox{\hskip
0.2pt\lower-0.2pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{
}\nullfont\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{{}}{}
{}{{}}{}{}{}{}{{}}{}\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgffillcolor}{rgb}{1,0,0}\pgfsys@color@rgb@fill{1}{0}{0}\pgfsys@invoke{
}{}\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@lineto{0.0pt}{0.0pt}\pgfsys@lineto{8.5359pt}{0.0pt}\pgfsys@lineto{8.5359pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{8.5359pt}{0.0pt}\pgfsys@fillstroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}\\\
\hline\cr\end{array}$ Figure 2: Grounding traces and events for the particle
example. e2(a,b) is the event of the two particles intersecting.

Examples of function symbols from $\mathcal{L}_{e}$ include
$\operatorname{\textsf{before}}(\epsilon)$ or
$\operatorname{\textsf{start}}(\epsilon)$, whereas examples of predicate
symbols of $\mathcal{L}_{e}$ include $\operatorname{\textsf{H}}(\epsilon)$
(unary symbol) and $\epsilon_{1}\ \textsf{bf}\ \epsilon_{2}$ (binary symbol in
infix notation). These symbols are intuitively described in Table 1. Their
actual grounding is discussed in Section 4.

##  4 Fuzzy Intervals and Relations

As previously mentioned, $\mathcal{I}(\epsilon)$ denotes the activation
sequence of an event in $\mathbb{B}$. Notice that $\mathcal{I}(\epsilon)$ is a
fuzzy subset of $\mathbb{N}$. A requirement imposed in Guarino et al. (2022)
is that $\mathcal{I}(\epsilon)$ must be an interval, i.e., a convex subset of
time points. However, in this paper, we consider the fact that such an
interval is a fuzzy interval. Therefore, we propose imposing constraints on
the shape of such a subset to be a _trapezoidal fuzzy number_ (Abbasbandy and
Hajjari, 2009).

######  Definition 1 (Fuzzy interval).

A fuzzy interval is a fuzzy set $I:\mathbb{R}\rightarrow[0,1]$ such that there
exists $a\leq b\leq c\leq d\in\mathbb{R}$.

$\displaystyle I(x)$ $\displaystyle=\begin{cases}\frac{x-a}{b-a}&\mbox{if
$x\in(a,b)$,}\\\ 1&\mbox{if $x\in[b,c]$,}\\\ \frac{x-d}{c-d}&\mbox{if
$x\in(c,d)$,}\\\ 0&\mbox{otherwise.}\end{cases}$ (1)

We also allow special cases of semi-infinite intervals, which we use to define
the $\operatorname{\textsf{before}}$ and $\operatorname{\textsf{after}}$
operators in Section 4.1.

Left-infinity

A left-infinite fuzzy interval is characterized by the parameters
$I=\\{x\mid-\infty,-\infty,c,d\\}$.

Right-infinity

A right-infinite fuzzy interval is characterized by the parameters $I=\\{x\mid
a,b,+\infty,+\infty\\}$.

######  Example 4.

$x$$1$$0$$I\\{x\mid 1,2,5,7\\}$$1$$2$$5$$7$

With this restriction, we impose that the activation function of every event
$\mathcal{I}(\epsilon)$ is such that there is a trapezoidal fuzzy interval
$I=\\{x\mid a,b,c,d\\}$ such that $\mathcal{I}_{n}(\epsilon)=I(n)$ for every
$n\in\mathbb{N}$. For ease of notation, in the rest of the paper, we will
commonly denote an interval simply by its four parameters $I=(a,b,c,d)$.

###  4.1 Basic Operations on Fuzzy Intervals

To provide the semantics for the functions and relations of $\mathcal{L}_{e}$,
we first define a set of basic operations on fuzzy intervals. Our operations
are inspired by Ohlbach (2004) who defines such operations on any convex and
non-convex interval. We simply specialize them on trapezoidal fuzzy intervals.

####  4.1.1 Duration

The duration of a trapezoidal fuzzy interval $A=(a,b,c,d)$, denoted by $|A|$
is equal to $\int_{-\infty}^{+\infty}A(x)dx$. If $A$ is finite then
$|A|=\frac{(c-b)+(d-a)}{2}$, otherwise $|A|=\infty$.

####  4.1.2 Before and After

If $A=(a,b,c,d)$ then
$\operatorname{\textsf{before}}(A)=(-\infty,-\infty,a,b)$ and
$\operatorname{\textsf{after}}(A)=(c,d,+\infty,+\infty)$ and

$0$$1$$A$$\operatorname{\textsf{before}}(A)$$\operatorname{\textsf{after}}(A)$

####  4.1.3 Start and End

If $A=(a,b,c,d)$ is left-finite then $\operatorname{\textsf{start}}(A)$ is
defined as $(\chi-\frac{\delta}{2},\chi,\chi,\chi+\frac{\delta}{2})$, such
that $\chi=\frac{a+b}{2}$ and $\delta=\max(\frac{b-a}{2},\delta_{\min})$ where
$\delta_{\min}$ is a small positive value to account for the crisp case $a=b$.

Similarly,
$\operatorname{\textsf{end}}(A)=(\chi-\frac{\delta}{2},\chi,\chi,\chi+\frac{\delta}{2})$
such that $A$ is right-finite, $\chi=\frac{c+d}{2}$ and
$\delta=\max(\frac{d-c}{2},\delta_{\min})$.

$0$$1$$A$$\operatorname{\textsf{start}}(A)$$\operatorname{\textsf{end}}(A)$

###  4.2 Relations between Fuzzy Intervals

Ohlbach (2004) defines interval-interval relations by computing the integral
of point-interval relations over the points in a set. To avoid the complexity
associated with the integrals, and to be more compliant with Allen’s
definition in the crisp case, we define new relations based on simplified
containment ratios.

$\displaystyle A\ \textsf{in}\ B$ $\displaystyle\coloneqq\frac{|A\cap
B|}{|A|}$ $\displaystyle A\ \textsf{eq}\ B$ $\displaystyle\coloneqq A\
\textsf{in}\ B\land B\ \textsf{in}\ A$ $\displaystyle A\ \textsf{bf}\ B$
$\displaystyle\coloneqq A\ \textsf{in}\ \operatorname{\textsf{before}}(B)$
$\displaystyle A\ \textsf{af}\ B$ $\displaystyle\coloneqq B\ \textsf{in}\
\operatorname{\textsf{after}}(A)$ $\displaystyle A\ \textsf{mt}\ B$
$\displaystyle\coloneqq\operatorname{\textsf{end}}(A)\ \textsf{eq}\
\operatorname{\textsf{start}}(B)$ $\displaystyle A\ \textsf{st}\ B$
$\displaystyle\coloneqq\operatorname{\textsf{start}}(A)\ \textsf{eq}\
\operatorname{\textsf{start}}{B}\wedge\operatorname{\textsf{end}}(A)\
\textsf{bf}\ \operatorname{\textsf{end}}(B)$ $\displaystyle A\ \textsf{dr}\ B$
$\displaystyle\coloneq\operatorname{\textsf{start}}(A)\ \textsf{af}\
\operatorname{\textsf{start}}(B)\land\operatorname{\textsf{end}}(A)\
\textsf{bf}\ \operatorname{\textsf{end}}(B)$ $\displaystyle A\ \textsf{fin}\
B$ $\displaystyle\coloneqq\operatorname{\textsf{start}}(A)\ \textsf{af}\
\operatorname{\textsf{start}}{B}\wedge\operatorname{\textsf{end}}(A)\
\textsf{eq}\ \operatorname{\textsf{end}}(B)$ $\displaystyle A\ \textsf{ol}\ B$
$\displaystyle\coloneqq\operatorname{\textsf{start}}(A)\ \textsf{bf}\
\operatorname{\textsf{start}}(B)\land\operatorname{\textsf{start}}(B)\
\textsf{bf}\ \operatorname{\textsf{end}}(A)$ $\displaystyle\ \ \ \
\land\operatorname{\textsf{end}}(A)\ \textsf{bf}\
\operatorname{\textsf{end}}(B)$

In these definitions, the temporal relations take precedence over the fuzzy
conjunction $\land$. For general fuzzy intervals, $|A\cap B|$ can be hard to
compute. However, with trapezoidal intervals, the calculation of $|A\cap B|$
is derivable analitically by solving simple linear constraints system. We show
how this is done in the following subsection.

###  4.3 Area Intersection

Let us calculate $\operatorname{Area}(A\cap B)$ for any two finite intervals
$A=(a,b,c,d)$ and $B=(a^{\prime},b^{\prime},c^{\prime},d^{\prime})$. Without
loss of generality, suppose that $a\leq a^{\prime}$. Developing an explicit
formula to compute $\operatorname{Area}(A\cap B)$ is not immediate as the
shape of $A\cap B$ can be a polygon with a varying number of edges (at most
6). For example:

$0$$1$$A$$B$$a^{\prime}$$b^{\prime}$$c^{\prime}$$d^{\prime}$$a$$b$$c$$d$

We propose to first determine the vertices of the shape $A\cap B$, and then
compute the area of the shape using the shoelace formula.

##### Empty intersection

First, we dismiss the case $d\leq a^{\prime}$, in which the two intervals do
not intersect. $\operatorname{Area}(A\cap B)=0$.

$0$$1$$A$$B$

In the rest of the section, we assume that an intersection always exists.

##### Bottom vertices

We call bottom vertices of the shape $A\cap B$, the ones on the line $y=0$.
There are always two. As $a\leq a^{\prime}$, $(a^{\prime},0)$ is always a
vertex of the shape. The second vertex is $(\min(d,d^{\prime}),0)$.

##### Top vertices

We call top vertices the ones on the line $y=1$. There can be zero, one, or
two top vertices that delimit $A\cap B$, as shown in the below figures:

$0$$1$$A$$B$$0$$1$$A$$B$$0$$1$$A$$B$

If $c<b^{\prime}$ or $b>c^{\prime}$, there are zero top vertices.

If $b^{\prime}=c$, the only top vertex is $(c,1)$. If $b=c^{\prime}$, the only
top vertex is $(b,1)$.

In other cases, there are always two top vertices $(\max(b,b^{\prime}),1)$ and
$(\min(c,c^{\prime}),1)$.

##### Side vertices

To determine the side vertices that delimit $A\cap B$, we compute the
intersection of the lines drawn by the edges of each trapezoid over the whole
$xy$ plane. Then, we keep the intersections where $y\in[0,1]$. For example, in
the below figure, there is only one intersection that defines a side vertex of
$A\cap B$:

$0$$1$$A$$B$$L_{A}$$R_{A}$$L_{B}$$R_{B}$

Let us denote $L_{A}\equiv y=\frac{x-a}{b-a}$ the line drawn by the left side
of $A$, and $R_{A}\equiv y=\frac{x-d}{c-d}$ the line drawn by the right side
of $A$. Similarly, we have $L_{B}\equiv
y=\frac{x-a^{\prime}}{b^{\prime}-a^{\prime}}$ and $R_{B}\equiv
y=\frac{x-d^{\prime}}{c^{\prime}-d^{\prime}}$ defined on $B$.

We are interested in finding the four intersections $L_{A}\cap L_{B}$,
$L_{A}\cap R_{B}$, $R_{A}\cap L_{B}$, and $R_{A}\cap R_{B}$. Each is easy to
determine by solving the system of two equations associated with the pair of
lines. For example, $L_{A}\cap L_{B}$ is the point
$(\frac{ab^{\prime}-ba^{\prime}}{a-b+b^{\prime}-a^{\prime}},\frac{a-a^{\prime}}{a-b+b^{\prime}-a^{\prime}})$.

Once we have determined the intersections, we keep the ones where $y\in[0,1]$
to define the vertices of $A\cap B$.

Let us cover some of the edge cases about these intersections. Firstly, any of
the edge lines can be vertical if the trapezoid is crisp on that edge. For
example, if $a=b$, $L_{A}$ is defined by the equation $x=a$. Regardless, the
method is the same: we simply use this vertical equation in the system of two
equations. Secondly, it is possible that there are no side vertices if some
lines are parallel. For example, $L_{A}\cap L_{B}$ gives no solution if
$a-b=a^{\prime}-b^{\prime}$ (or infinite solutions if the lines are the same).
In such cases, we ignore the pair of parallel lines. Finally, it is also
possible that a side vertex is a top or bottom vertex if the lines intersect
on $y=0$ or $y=1$.

##### Area calculation

Once we have determined all the vertices $(x_{i},y_{i})$ of $A\cap B$,
arranged in a counter-clockwise sequence of points, we can calculate the area
using the shoelace formula:

$\operatorname{Area}(A\cap
B)=\frac{1}{2}\sum_{i=1}^{n}(y_{i}+y_{i+1})(x_{i}-x_{i+1})$ (2)

##### Semi-infinite intervals

We sometimes have to compute the area intersection in cases where $A$ is left-
infinite or $B$ is right-infinite (for example, when using the operators bf or
af ). However, we can turn these semi-infinite intervals to finite intervals
such that the area calculation is unchanged. If $A$ is left-infinite, we can
replace the infinite parameters with any $a\leq a^{\prime}$ and $b\leq
b^{\prime}$. Similarly, if $B$ is right-infinite, we can replace the infinite
parameters with any $c^{\prime}\geq c$ and $d^{\prime}\geq d$. Doing so, we
can reuse the method highlighted above.

##  5 Architecture

The main objective of introducing Interval Real Logic (IRL) is to use it to
impose temporal constraints in a neural architecture for Event Detection.
Given a temporal data sequence $\bm{u}=\\{u_{i}\\}_{i=0}^{T}$ we define a
neural architecture, called _Interval Logic Tensor Networks (ILTN)_ that is
capable to recognize _if_ and _when_ a set of events
$\epsilon_{1},\dots,\epsilon_{k}$ happens in the sequence, under the
hypothesis, that certain constraints expressed in IRL are (softly) satisfied.

We implemented a first simple prototype of ILTN in TensorFlow as a wrapper of
LTN (Badreddine et al., 2022). The code is available at
https://github.com/sbadredd/interval-ltn. The present section describes
important design choices that enable the architecture. In the description, we
concentrate only on the temporal prediction and not on the classification of
the events.

###  5.1 Neural Architecture

Figure 3(b) illustrates how neural networks are used to ground any event
$\epsilon$. An event is characterized by two elements: a truth degree
$\operatorname{\textsf{H}}(\epsilon)$ indicating if the event happens, and by
a trapezoid interval defining the membership function and when it happens.

Let us call _logits_ the vector of raw (non-normalized) predictions output by
the neural model, as is common in the machine learning literature. The truth
degree $\operatorname{\textsf{H}}(\epsilon)$ is easily implemented using a
single logit node which is then passed to a sigmoid normalization function and
constrained in the interval $[0,1]$.

Directly defining the parameters $(a,b,c,d)$ of the interval is difficult as
the semantic constraint $a\leq b\leq c\leq d$ is hard to implement in a neural
architecture. Instead, our neural architecture predicts the four values
$(a,b-a,c-b,d-c)$. The only semantical constraint on these values is that each
is positive. This is easily implemented using four logit nodes which are then
passed through softplus activations.

(a) $p(\bm{u})$

(b) $\epsilon(\bm{u})$

Figure 3: Implementation of a temporal predicate symbol from $\mathcal{L}_{t}$
(left) and of an event symbol (fuzzy interval and happening predicate) from
$\mathcal{L}_{e}$ (right). Examples of sequential neural architectures are
Recurrent Neural Networks or Transformers.

###  5.2 Smooth Membership Functions

We notice an important vanishing gradient issue with trapezoidal interval
functions. If $x$ is in the flat regions $I(x)=0$ or $I(x)=1$, then
$\partialderivative{I(x)}{x}=0$. To account for this, we define $I_{\sim}$, a
smooth version of the membership function (1):

$\displaystyle I_{\sim}(x)$
$\displaystyle=\begin{cases}\operatorname{\mathrm{s}_{+}}(x-a)&\mbox{if $x\leq
a$,}\\\ \operatorname{\mathrm{s}_{+}}(\max(b-x,x-c))&\mbox{if $b<x\leq c$,}\\\
\operatorname{\mathrm{s}_{+}}(d-x)&\mbox{if $d<x$,}\\\
I(x)&\mbox{otherwise.}\end{cases}$ (3)

Where $\operatorname{\mathrm{s}_{+}}$ is the softplus function defined by:

$\displaystyle\operatorname{\mathrm{s}_{+}}(x\mid\beta)$
$\displaystyle=\frac{1}{\beta}\log\left(1+e^{\beta x}\right)$ (4)
$\displaystyle\partialderivative{\operatorname{\mathrm{s}_{+}}(x\mid\beta)}{x}$
$\displaystyle=\frac{1}{1+\exp(-\beta x)}$ (5)

Notice that, in (3), the inputs to the $\operatorname{\mathrm{s}_{+}}$
function are all negative values. Intuitively, looking at the graph of
softplus in Figure 4, $\operatorname{\mathrm{s}_{+}}$ applied to negative
values outputs a value that tends to zero with non-negative gradients.

We use $I$ and $I_{\sim}$ to define an artificial operator with distinct
properties in the forward pass and backward pass of the computational graph.
111See also https://www.tensorflow.org/api_docs/python/tf/custom_gradient. Let
$\epsilon=e(t_{1},\dots,t_{m})$ be an event term associated with an interval
$I$ and a corresponding smooth version $I_{\sim}$. We use:

$\displaystyle\epsilon(x)$ $\displaystyle=I(x)$ (6)
$\displaystyle\partialderivative{\epsilon(x)}{x}$
$\displaystyle=\partialderivative{I_{\sim}(x)}{x}$ (7)

The motivation is demonstrated in Figure 5. The backward pass
$\partialderivative{I_{\sim}(x)}{x}$ has non-zero gradients everywhere that
push $x$ to fit in the center of the interval. The forward pass remains the
accurate evaluation $I(x)$.

Finally, we use the parameter $\beta$ to ensure the accuracy of the operator.
For example, for large negative differences $x-a$, the output of
$\operatorname{\mathrm{s}_{+}}(x-a)$ gets very small and can become zero
because of the way computers approximate real numbers. In float32 precision
format, this happens with $x-a>90$ and $\beta=1$. In such cases, the gradients
still vanish. We avoid this issue by setting $\beta=\frac{1}{T}$, where $T$ is
the largest time difference occuring in our data, or in other words $T$ is the
length of the trace in the experiment.

$-4$$-3$$-2$$-1$$0$$1$$2$$3$$4$$0$$0.5$$1$$1.5$$2$$2.5$$3$$3.5$$4$$4.5$$\beta=1$$\beta=2$$\beta=0.5$
(a) $\operatorname{\mathrm{s}_{+}}(x\mid\beta)$

$-4$$-3$$-2$$-1$$0$$1$$2$$3$$4$$0$$0.2$$0.4$$0.6$$0.8$$1$ (b)
$\partialderivative{\operatorname{\mathrm{s}_{+}}(x\mid\beta)}{x}$

Figure 4: The softplus function. Figure 5: Smooth membership function. The
forward pass uses $I(x)$ (top left). The backward pass uses
$\partialderivative{I_{\sim}(x)}{x}$ (bottom right).

###  5.3 Smooth Relations

Let $A=(a,b,c,d)$ and $B=(a^{\prime},b^{\prime},c^{\prime},d^{\prime})$ be two
trapezoids. Without loss of generality, suppose that $a\leq a^{\prime}$.
Similarly to how membership functions has zero gradients on some parts of the
domain, the relations in 4.2 have vanishing gradients in two situations. The
first is when $A\ \textsf{in}\ B=0$ and the trapezoids do not intersect. In
other words, when $d<a^{\prime}$. The second is when $A\ \textsf{in}\ B=1$ and
$A$ is fully contained in $B$. In other words, when $a>a^{\prime}$,
$b>b^{\prime}$, $c<c^{\prime}$, and $d<d^{\prime}$.

Again, we solve this by defining a smooth operator for the backward pass
relying on the softplus operator:

$(A\ \textsf{in}\
B)_{\sim}=\begin{cases}\operatorname{\mathrm{s}_{+}}(d-a)&\mbox{if
$d<a^{\prime}$,}\\\
\operatorname{\mathrm{s}_{+}}(a^{\prime}-a+d-d^{\prime})&\mbox{if $A$ fully in
$B$,}\\\ A\ \textsf{in}\ B&\mbox{otherwise.}\end{cases}$ (8)

with the non-vanishing derivatives on the trapezoid edges of $A$ and $B$. We
use $A\ \textsf{in}\ B$ in the forward pass of the computational graph and
$\partialderivative{(A\ \textsf{in}\ B)_{\sim}}{x}$ in the backward pass,
where $x$ is any parameter defining $A$ or $B$. Finally, we still set
$\beta=\frac{1}{T}$ for the softplus function.

Task | Initial Conditions |  Setting |  Constraints | Result
---|---|---|---|---
T1 |  |  • $B$ trainable, • $A$ and $C$ fixed. |  1. $|B|\approx 2$ 2. $B\ \textsf{af}\ A$ 3. $B\ \textsf{bf}\ C$ |
T2 |  |  • $B$ trainable, • $A$ and $C$ fixed. |  1. $|B|\approx 1.5$ 2. $B\ \textsf{st}\ C$ |
T3 |  |  • $A$ trainable, • $B$ fixed. |  1. $A\ \textsf{ol}\ B$ 2. $A(3)$ 3. $\lnot A(2)$ |
T4 |  |  • $x$ trainable, • $A$ fixed. |  1. $\operatorname{\textsf{end}}(A)(x)$ |
Table 2: Experiments

##  6 Experiments

We test the system on synthetic tasks that require a combination of learning
and reasoning about temporal relations between fuzzy intervals. Let
$\phi_{1},\dots,\phi_{n}$ be $n$ constraints written in Interval Real Logic
defining a knowledge base $\mathcal{K}$. Like in LTN (Badreddine et al.,
2022), the grounding of the knowledge base defines a satisfaction level to
maximise. Optimising by gradient descent, we have the following loss function:

$L(\mathcal{K},\theta)-(\mathcal{G}(\phi_{1},\theta)\land\dots\land\mathcal{G}(\phi_{n},\theta))$
(9)

where $\phi_{i}$’s are $\mathcal{L}_{e}$ formulas and $\theta$ is a set of
trainable parameters used to define the grounding. We focus the experimental
study on the training of events with constraints written using
$\mathcal{L}_{e}$, which is the main innovation of this paper. Specifically,
we focus on learning parameters that define the fuzzy trapezoid intervals of
events.

Table 2 displays a list of training experiments where ILTN maximizes the
satisfaction of temporal constraints. All tasks are trained using the Adam
optimizer (Kingma and Ba, 2015) with a learning rate of $0.1$. In T1, T2, T3,
and T4, the results are obtained after training for 50, 500, 5000, and 200
training steps, respectively. For the logical operators, we use the product
t-norm $u\land v=uv$ and the standard negation $\lnot u=1-u$.

We highlight the following features:

  * •

In T1, T2 and T3, the system learns fuzzy intervals,

  * •

In T4, the system learns a time point value $x$,

  * •

T1, T2, and T3 display constraints using Allen’s relational symbols af , bf ,
st , and ol ,

  * •

T3 and T4 display constraints using membership functions,

  * •

T4 displays a constraint using a functional symbol
$\operatorname{\textsf{end}}$,

  * •

In T1 and T2, $u\approx v$ is a smooth equality predicate implemented as
$\exp(-|u-v|)\in[0,1]$.

###  6.1 Challenges for Future Work

In all tasks, the system learns to update the event groundings to satisfy the
knowledge base. The experiments demonstrate that ILTN can successively
backpropagate gradients through the Interval Real logic. Nevertheless, we
highlight three limitations of our experiments that future work should
explore.

Firstly, early stopping was an important factor in our experiments. Continuing
training after reaching the maximal satisfaction level could sometimes lead to
worsening the results. This is likely due to the smoothing of operators for
obtaining non-vanishing gradients. This feature is important early in
training. However, once a constraint is satisfied, having vanishing gradients
is acceptable. Future work could explore reducing or stopping the smoothening
of constraints when their satisfaction levels are high.

Secondly, the Adam optimizer is traditionally used with learning rates in the
order of $0.001$. In comparison, the learning rate of $0.1$ used to train our
synthetic tasks is unusually high. A lower learning rate led to experiments
not converging fast enough. There is likely a scaling issue in the gradients
of some operations. This could also explain why T3 required more training
steps than the other tasks to reach convergence: this is the only task that
mixes relational operators ( ol ) and membership functions. The two have
gradients scaling differently which can challenge the training. Future work
should analyse further the gradient properties of each temporal operator.

Thirdly, the present experiments do not showcase yet the power of learning
events that depend on input features. For example, in Figure 3(b), the present
tasks only learn trapezoid logits that define a trapezoid number. There is no
sequential data in input and neural architecture that builds on top of it.
Future work should explore more elaborate tasks employing such architectures.

##  7 Conclusions

In this paper, we introduce Interval Real Logic (IRL), a two-sorted logic that
enables the prediction of properties that evolve within a set of data
sequences (traces) and properties of events that occur within the sequences.
IRL semantics are defined in terms of sequences of real feature vectors, and
connectives and quantifiers are interpreted using fuzzy logic. We represent
event duration through trapezoidal fuzzy intervals, and fuzzy temporal
relations are defined based on the relationships between the intervals’ areas
and their intersections.

We also present Interval Logic Tensor Networks (ILTN), a neuro-symbolic system
that leverages background knowledge expressed in IRL to predict the fuzzy
duration of events. To prevent vanishing gradient during learning, we use
softplus functions to smooth both events and their relations. We evaluate
ILTN’s performance on four tasks with different temporal constraints and show
that it is capable of making events compliant with background knowledge in all
four tasks.

In Section 6.1, we suggest several directions for future research. One
promising avenue would be to test ILTN on more realistic and complex
scenarios, such as those involving real-world data. Scalability is a
significant challenge for neuro-symbolic frameworks for event recognition,
which we have hoped to address by representing events as a whole, rather than
using a point-wise approach.

## References

  * Clavel et al. [2005] C. Clavel, T. Ehrette, and G. Richard.  Events detection for an audio-based surveillance system.  In _2005 IEEE International Conference on Multimedia and Expo_ , pages 1306–1309, 2005.  doi: 10.1109/ICME.2005.1521669.
  * Xiang and Wang [2019] Wei Xiang and Bang Wang.  A survey of event extraction from text.  _IEEE Access_ , 7:173111–173137, 2019.
  * Lai [2022] Viet Dac Lai.  Event extraction: A survey.  _arXiv preprint arXiv:2210.03419_ , 2022.
  * Cordeiro and Gama [2016] Mário Cordeiro and João Gama.  _Online Social Networks Event Detection: A Survey_ , pages 1–41.  Springer International Publishing, Cham, 2016.  ISBN 978-3-319-41706-6.  doi: 10.1007/978-3-319-41706-6˙1.  URL https://doi.org/10.1007/978-3-319-41706-6_1.
  * Yin et al. [2020] Shujuan Yin, Weizhong Zhao, Xingpeng Jiang, and Tingting He.  Knowledge-aware few-shot learning framework for biomedical event trigger identification.  In _2020 IEEE International Conference on Bioinformatics and Biomedicine (BIBM)_ , pages 375–380, 2020.  doi: 10.1109/BIBM49941.2020.9313195.
  * Vilamala et al. [2023] Marc Roig Vilamala, Tianwei Xing, Harrison Taylor, Luis Garcia, Mani Srivastava, Lance Kaplan, Alun Preece, Angelika Kimmig, and Federico Cerutti.  Deepprobcep: A neuro-symbolic approach for complex event processing in adversarial settings.  _Expert Systems with Applications_ , 215:119376, 2023.
  * Apriceno et al. [2021] Gianluca Apriceno, Andrea Passerini, and Luciano Serafini.  A neuro-symbolic approach to structured event recognition.  In Carlo Combi, Johann Eder, and Mark Reynolds, editors, _28th International Symposium on Temporal Representation and Reasoning, TIME 2021, September 27-29, 2021, Klagenfurt, Austria_ , volume 206 of _LIPIcs_ , pages 11:1–11:14. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2021.  doi: 10.4230/LIPIcs.TIME.2021.11.  URL https://doi.org/10.4230/LIPIcs.TIME.2021.11.
  * Apriceno et al. [2022] Gianluca Apriceno, Andrea Passerini, and Luciano Serafini.  A neuro-symbolic approach for real-world event recognition from weak supervision.  In _TIME_ , volume 247 of _LIPIcs_ , pages 12:1–12:19. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2022.
  * Umili et al. [2022] Elena Umili, Roberto Capobianco, and Giuseppe De Giacomo.  Grounding ltlf specifications in images.  In _NeSy_ , volume 3212 of _CEUR Workshop Proceedings_ , pages 45–63. CEUR-WS.org, 2022.
  * Kowalski and Sergot [1986] Robert Kowalski and Marek Sergot.  A logic-based calculus of events.  _New generation computing_ , 4:67–95, 1986.
  * Allen [1983] James F. Allen.  Maintaining knowledge about temporal intervals.  _Commun. ACM_ , 26(11):832–843, 1983.  doi: 10.1145/182.358434.  URL http://doi.acm.org/10.1145/182.358434.
  * Mueller [2008] Erik T Mueller.  Event calculus.  _Foundations of Artificial Intelligence_ , 3:671–708, 2008\.
  * Ohlbach [2004] Hans Jürgen Ohlbach.  Relations between fuzzy time intervals.  In _Proceedings. 11th International Symposium on Temporal Representation and Reasoning, 2004. TIME 2004._ , pages 44–51. IEEE, 2004.  doi: 10.1109/TIME.2004.1314418.
  * Schockaert et al. [2008] Steven Schockaert, Martine De Cock, and Etienne E Kerre.  Fuzzifying allen’s temporal interval relations.  _IEEE Transactions on Fuzzy Systems_ , 16(2):517–533, 2008.
  * Badreddine et al. [2022] Samy Badreddine, Artur d’Avila Garcez, Luciano Serafini, and Michael Spranger.  Logic tensor networks.  _Artif. Intell._ , 303:103649, 2022.  doi: 10.1016/j.artint.2021.103649.  URL https://doi.org/10.1016/j.artint.2021.103649.
  * Kahn and Gorry [1977] Kenneth Kahn and G.Anthony Gorry.  Mechanizing temporal knowledge.  _Artificial Intelligence_ , 9(1):87–108, 1977\.  ISSN 0004-3702.  doi: https://doi.org/10.1016/0004-3702(77)90015-7.  URL https://www.sciencedirect.com/science/article/pii/0004370277900157.
  * Allen and Hayes [1985] James F. Allen and Patrick J. Hayes.  A common-sense theory of time.  In Aravind K. Joshi, editor, _Proceedings of the 9th International Joint Conference on Artificial Intelligence. Los Angeles, CA, USA, August 1985_ , pages 528–531. Morgan Kaufmann, 1985.  URL http://ijcai.org/Proceedings/85-1/Papers/101.pdf.
  * Jong et al. [1999] Woei Tzy Jong, Yuh Shin Shiau, Yih Jen Horng, Hsin Horng Chen, and Shyi Ming Chen.  Temporal knowledge representation and reasoning techniques using time petri nets.  _IEEE Trans. Syst. Man Cybern. Part B_ , 29(4):541–545, 1999.  doi: 10.1109/3477.775271.  URL https://doi.org/10.1109/3477.775271.
  * Pnueli [1977] Amir Pnueli.  The temporal logic of programs.  In _FOCS_ , pages 46–57. IEEE Computer Society, 1977.
  * Clarke and Emerson [1982] Edmund M. Clarke and E. Allen Emerson.  Design and synthesis of synchronization skeletons using branching time temporal logic.  In Dexter Kozen, editor, _Logics of Programs_ , pages 52–71, Berlin, Heidelberg, 1982. Springer Berlin Heidelberg.  ISBN 978-3-540-39047-3.
  * Zadeh [1965] L.A. Zadeh.  Fuzzy sets.  _Information and Control_ , 8(3):338–353, 1965\.  ISSN 0019-9958.  doi: https://doi.org/10.1016/S0019-9958(65)90241-X.  URL https://www.sciencedirect.com/science/article/pii/S001999586590241X.
  * Dubois and Prade [1989] Didier Dubois and Henri Prade.  Processing fuzzy temporal knowledge.  _IEEE Trans. Syst. Man Cybern._ , 19(4):729–744, 1989.  doi: 10.1109/21.35337.  URL https://doi.org/10.1109/21.35337.
  * Nagypal and Motik [2003] Gabor Nagypal and Boris Motik.  A fuzzy model for representing uncertain, subjective, and vague temporal knowledge in ontologies.  In Robert Meersman, Zahir Tari, and Douglas C. Schmidt, editors, _On The Move to Meaningful Internet Systems 2003: CoopIS, DOA, and ODBASE - OTM Confederated International Conferences, CoopIS, DOA, and ODBASE 2003, Catania, Sicily, Italy, November 3-7, 2003_ , volume 2888 of _Lecture Notes in Computer Science_ , pages 906–923. Springer, 2003.  doi: 10.1007/978-3-540-39964-3“˙57.  URL https://doi.org/10.1007/978-3-540-39964-3_57.
  * Schockaert and Cock [2008] Steven Schockaert and Martine De Cock.  Temporal reasoning about fuzzy intervals.  _Artif. Intell._ , 172(8-9):1158–1193, 2008.
  * Kapitanova et al. [2012] Krasimira Kapitanova, Sang H. Son, and Kyoung-Don Kang.  Using fuzzy logic for robust event detection in wireless sensor networks.  _Ad Hoc Networks_ , 10(4):709–722, 2012.  ISSN 1570-8705.  doi: https://doi.org/10.1016/j.adhoc.2011.06.008.  URL https://www.sciencedirect.com/science/article/pii/S1570870511001326.  Advances in Ad Hoc Networks (II).
  * Dima et al. [2012] Sofia-Maria Dima, Dimitris Tsitsipis, Christos Antonopoulos, John Gialelis, and Stavros Koubias.  Flogera — a fuzzy logic event recognition algorithm in a wsn environment.  In _2012 8th International Wireless Communications and Mobile Computing Conference (IWCMC)_ , pages 850–855, 2012.  doi: 10.1109/IWCMC.2012.6314315.
  * Muduli et al. [2018] Lalatendu Muduli, Prasanta K. Jana, and Devi Prasad Mishra.  Wireless sensor network based fire monitoring in underground coal mines: A fuzzy logic approach.  _Process Safety and Environmental Protection_ , 113:435–447, 2018.  ISSN 0957-5820.  doi: https://doi.org/10.1016/j.psep.2017.11.003.  URL https://www.sciencedirect.com/science/article/pii/S0957582017303804.
  * Hitzler and Sarker [2022] Pascal Hitzler and Md Kamruzzaman Sarker.  _Neuro-Symbolic Artificial Intelligence: The State of the Art_.  IOS Press, 2022.  ISBN 978-1-64368-244-0.
  * Khan et al. [2019a] Abdullah Khan, Loris Bozzato, Luciano Serafini, and Beatrice Lazzerini.  Visual Reasoning on Complex Events in Soccer Videos Using Answer Set Programming.  In Diego Calvanese and Luca Iocchi, editors, _GCAI 2019. Proceedings of the 5th Global Conference on Artificial Intelligence, Bozen/Bolzano, Italy, 17-19 September 2019_ , volume 65, pages 42–53. EasyChair, 2019a.  doi: 10.29007/pjd4.
  * Khan et al. [2019b] Abdullah Khan, Luciano Serafini, Loris Bozzato, and Beatrice Lazzerini.  Event Detection from Video Using Answer Set Programing.  In Alberto Casagrande and Eugenio G. Omodeo, editors, _Proceedings of the 34th Italian Conference on Computational Logic, Trieste, Italy, June 19-21, 2019_ , volume 2396 of _CEUR Workshop Proceedings_ , pages 48–58. CEUR-WS.org, 2019b.  URL http://ceur-ws.org/Vol-2396/paper25.pdf.
  * Xing et al. [2019] Tianwei Xing, Marc R. Vilamala, Luis Garcia, Federico Cerutti, Lance Kaplan, Alun Preece, and Mani Srivastava.  DeepCEP: Deep Complex Event Processing Using Distributed Multimodal Information.  In _IEEE International Conference on Smart Computing, SMARTCOMP 2019, Washington, DC, USA, June 12-15, 2019_ , pages 87–92. IEEE, 2019.  doi: 10.1109/SMARTCOMP.2019.00034.
  * Vilamala et al. [2019] Marc Roig Vilamala, Liam Hiley, Yulia Hicks, Alun D. Preece, and Federico Cerutti.  A pilot study on detecting violence in videos fusing proxy models.  In _22th International Conference on Information Fusion, FUSION 2019, Ottawa, ON, Canada, July 2-5, 2019_ , pages 1–8. IEEE, 2019.  URL https://ieeexplore.ieee.org/document/9011329.
  * Gómez et al. [2020] José Roldán Gómez, Juan Boubeta-Puig, José Luis Martínez, and Guadalupe Ortiz.  Integrating complex event processing and machine learning: An intelligent architecture for detecting iot security attacks.  _Expert Syst. Appl._ , 149:113251, 2020.  doi: 10.1016/j.eswa.2020.113251.  URL https://doi.org/10.1016/j.eswa.2020.113251.
  * Xing et al. [2020] Tianwei Xing, Luis Garcia, Marc Roig Vilamala, Federico Cerutti, Lance M. Kaplan, Alun D. Preece, and Mani B. Srivastava.  Neuroplex: learning to detect complex events in sensor networks through knowledge injection.  In Jin Nakazawa and Polly Huang, editors, _SenSys ’20: The 18th ACM Conference on Embedded Networked Sensor Systems, Virtual Event, Japan, November 16-19, 2020_ , pages 489–502. ACM, 2020.  doi: 10.1145/3384419.3431158.  URL https://doi.org/10.1145/3384419.3431158.
  * Vilamala et al. [2021] Marc Roig Vilamala, Tianwei Xing, Harrison Taylor, Luis Garcia, Mani B. Srivastava, Lance M. Kaplan, Alun D. Preece, Angelika Kimmig, and Federico Cerutti.  Using deepproblog to perform complex event processing on an audio stream.  _CoRR_ , abs/2110.08090, 2021.  URL https://arxiv.org/abs/2110.08090.
  * Donadello et al. [2017] Ivan Donadello, Luciano Serafini, and Artur S. d’Avila Garcez.  Logic tensor networks for semantic image interpretation.  In Carles Sierra, editor, _Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19-25, 2017_ , pages 1596–1602. ijcai.org, 2017.  doi: 10.24963/ijcai.2017/221.  URL https://doi.org/10.24963/ijcai.2017/221.
  * Manigrasso et al. [2021] Francesco Manigrasso, Filomeno Davide Miro, Lia Morra, and Fabrizio Lamberti.  Faster-ltn: A neuro-symbolic, end-to-end object detection architecture.  In _ICANN (2)_ , volume 12892 of _Lecture Notes in Computer Science_ , pages 40–52. Springer, 2021.
  * Guarino et al. [2022] Nicola Guarino, Riccardo Baratella, and Giancarlo Guizzardi.  Events, their names, and their synchronic structure.  _Applied Ontology_ , pages 1–35, 2022.
  * Abbasbandy and Hajjari [2009] Saeid Abbasbandy and T Hajjari.  A new approach for ranking of trapezoidal fuzzy numbers.  _Computers & mathematics with applications_, 57(3):413–419, 2009.
  * Kingma and Ba [2015] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In Yoshua Bengio and Yann LeCun, editors, _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , 2015.  URL http://arxiv.org/abs/1412.6980.