34th Symposium on Naval Hydrodynamics

Washington, DC, USA, 26th June–1st July 2022

High-Fidelity Simulation and Novel Data Analysis of the Bubble Creation and
Sound Generation Processes in Breaking Waves

Q. Gao1,∗, G. B. Deane2, S. Basak1, U. Bitencourt1, and L. Shen1

(1Department of Mechanical Engineering and St. Anthony Falls Laboratory,
University of Minnesota, Minneapolis, MN 55455, USA, 2Marine Physical
Laboratory, Scripps Institution of Oceanography, University of California at
San Diego, La Jolla, CA 92093, USA)

## Abstract

Recent increases in computing power have enabled the numerical simulation of
many complex flow problems that are of practical and strategic interest for
naval applications. A noticeable area of advancement is the computation of
turbulent, two-phase flows resulting from wave breaking and other multiphase
flow processes such as cavitation that can generate underwater sound and
entrain bubbles in ship wakes, among other effects. Although advanced flow
solvers are sophisticated and are capable of simulating high Reynolds number
flows on large numbers of grid points, challenges in data analysis remain.
Specifically, there is a critical need to transform highly resolved flow
fields described on fine grids at discrete time steps into physically resolved
features for which the flow dynamics can be understood and utilized in naval
applications. This paper presents our recent efforts in this field. In
previous works, we developed a novel algorithm to track bubbles in breaking
wave simulations and to interpret their dynamical behavior over time (Gao et
al., 2021a). We also discovered a new physical mechanism driving bubble
production within breaking wave crests (Gao et al., 2021b) and developed a
model to relate bubble behaviors to underwater sound generation (Gao et al.,
2021c). In this work, we applied our bubble tracking algorithm to the breaking
waves simulations and investigated the bubble trajectories, bubble creation
mechanisms, and bubble acoustics based on our previous works.

## Introduction

Bubbles play critical roles in ocean–atmosphere processes and naval
applications, such as ship hydrodynamics. Specifically, it is important to
understand the creation mechanisms, transport behaviors, and acoustic
properties of bubbles because they can affect the drag on ship hulls and
ships’ acoustic and optical signatures. Consequently, numerous experimental
and numerical studies have been conducted on the bubbles in breaking waves and
their applications to ship hydrodynamics. For example, Deane & Stokes (2002),
Wang et al. (2016), Deike et al. (2016), Gao et al. (2018), Yu et al. (2020),
and Chan et al. (2021) investigated the bubble size spectrum in breaking waves
or free-surface turbulence and demonstrated that super-Hinze-scale bubbles
(bubbles whose radii exceed the Hinze scale) follow a $-10/3$ power-law
scaling. Super-Hinze-scale bubbles are created by a fragmentation cascade
process (Garrett et al., 2000), where large bubbles break up into smaller
bubbles due to turbulent fluctuations. Furthermore, Castro et al. (2014) and
Li et al. (2016) investigated bubble distributions with regard to ship
hydrodynamics.

The above works have advanced our understanding of the physics of bubbly flows
in the context of breaking waves and ship hydrodynamics. However, some
challenges remain in this field. For example, these works studied the
statistics of bubbles at only single instances in time because of the
limitation of the conventional bubble identification algorithm, which cannot
track bubbles temporally and thus cannot detect their evolutionary behavior
over time. Therefore, the fragmentation, coalescence, and trajectories of
bubbles were not captured in previous studies. Recently, we developed a novel
algorithm (Gao et al., 2021a) to track bubbles and detect their evolutionary
behaviors over time by making connections among bubbles between sequential
time instances. Specifically, our algorithm classifies bubble behaviors into
five categories. Based on these bubble behaviors, the algorithm derives
constraints on the bubble volumes, velocities, and positions at two sequential
time instances and then tries to make connections among the bubbles at these
successive instances in time by selecting the minimum value within a network
of error functions defined by these constraints. By following the networks
among bubbles over time, this algorithm is capable of temporally tracking
bubbles to obtain their trajectories. Moreover, the algorithm can be used to
identify newly formed bubbles when they are not associated with any bubbles at
the previous time step. The algorithm here is limited to binary fragmentation
and binary coalescence events. In other words, the algorithm cannot detect the
event that a bubble breaks up into three and more (multiple fragmentation) or
the event that three or more bubbles merge into one (multiple coalescence).
This is because establishing network becomes much complex when considering
multiple fragmentation and coalescence.

To ascertain the mechanism responsible for creating bubbles, previous studies
have focused primarily on the bubble fragmentation cascade process. During the
bubble fragmentation cascade process, large bubbles are fragmented into
smaller ones due to turbulent fluctuation. Recently, Gao et al. (2021b)
studied the bubble generation dynamics in a breaking wave. In their work, they
identified a bubble formation mechanism called air cylinder instability and
provided a formula expressing its theoretical dispersion. Different from the
fragmentation cascade process, the cylinder instability produces bubbles
through an interface instability. They compared simulation results with the
theory and found good agreement. However, some questions regarding the bubble
creation mechanism remain. For example, how many air cylinders are created in
a breaking wave, and how many bubbles are created when an air cylinder
breakup? Furthermore, how important is the cylinder instability mechanism to
the overall formation of bubbles? In the present work, we examine the wave
breaking process, bubble creation events, and the bubble size spectrum to
quantify the importance of the air cylinder instability to the production of
bubbles to some extent.

In addition, the abovementioned bubble tracking and event detection algorithm
can be used to identify bubble creation events, which is useful for
calculating wave noise. In ocean waves, wave noise is generated by newly
formed bubbles. Gao et al. (2021c) developed a simulation framework for
calculating the sound radiated by newly formed bubbles within a breaking wave
crest. The framework contains three components: (1) a two-phase flow solver,
(2) a bubble event detection algorithm, and (3) a wave noise model for
calculating bubble acoustics. In the present work, we apply this framework and
investigate the wave noise spectrogram.

The remainder of this paper is organized as follows. First, we describe the
numerical method, including the two-phase flow solver, bubble tracking
algorithm, and wave noise model. Then, we discuss the bubble entrainment
process, bubble size spectrum, bubble trajectories, and bubble acoustics.
Finally, some conclusions are drawn.

## Numerical Method

### High-fidelity simulation of wave breaking and bubble dynamics

The simulations are performed by solving the incompressible Navier–Stokes
equations on a fixed Eulerian grid, where the air and water phases are treated
as a coherent system with varying physical properties. The governing equations
are

$\frac{D\bm{u}}{Dt}=-\frac{1}{\rho}\left(\nabla
p+\rho\bm{g}+\nabla\cdot\left(2\mu\bm{D}\right)+\sigma\kappa\delta\left(\bm{x}_{s}\right)\right)$
(1)

and

$\nabla\cdot\bm{u}=0,$ (2)

where $\bm{u}$ is the velocity, $\rho$ is the density, $p$ is the pressure,
$\bm{g}$ is the acceleration due to gravity, $\mu$ is the dynamic viscosity,
$\bm{D}\equiv\left(\nabla\bm{u}+\nabla\bm{u}^{T}\right)/2$ is the deformation
tensor, $\sigma$ is the coefficient of surface tension, $\kappa$ is the
interface curvature, $\delta$ is the Dirac delta function, and $\bm{x}_{s}$
denotes the position on the air–water interface. The in-house WOW code used
here implements the coupled level set and volume of fluid (CLSVOF) method
(Sussman & Puckett, 2000) to capture the air–water interface. In the level set
(LS) method, $\phi$ is the signed LS distance function, which is positive in
water and negative in air and takes a value of $0$ at the air–water interface.
In the volume of fluid (VOF) method, the volume fraction function $F$ is
defined as the ratio of the volume of water in a grid cell to the overall grid
cell volume. The CLSVOF method utilizes the advantages of both the LS and the
VOF methods to precisely describe the interface geometry and ensure the
conservation of mass. The governing equations for the LS and volume fraction
functions are

$\frac{\partial\phi}{\partial t}+\nabla\cdot\left(\bm{u}\phi\right)=0$ (3)

and

$\frac{\partial F}{\partial t}+\nabla\cdot\left(\bm{u}F\right)=0,$ (4)

respectively. Using the LS method, the density and viscosity are expressed as

$\rho=H\left(\phi\right)\rho_{w}+\left(1-H\left(\phi\right)\right)\rho_{a}$
(5)

and

$\mu=H\left(\phi\right)\mu_{w}+\left(1-H\left(\phi\right)\right)\mu_{a},$ (6)

where $H$ is a smoothed Heaviside function and the subscripts ‘$a$’ and ‘$w$’
denote air and water, respectively. The solution procedure for the two-phase
flow solver consists of three key steps: (i) updating the fluid density and
viscosity using equations (5) and (6) based on the LS function; (ii) obtaining
the velocity and pressure fields by solving the Navier–Stokes equations (1)
and (2); and (iii) advancing the LS and volume fraction functions using
equations (3) and (4), respectively. More information about the two-phase flow
simulation method can be found in Yang et al. (2018) and Gao et al. (2021a).

Figure 1:  Visualization of a wave simulation with $S=0.38$ at different
times. (a): A jet forms on the wave crest. (b)–(c): The overturning wave crest
strikes the wave surface, producing multiple cavities and splash jets.
(d)–(e): Cavities break up into bubbles close to the wave surface. (f):
Bubbles are generated near the wave surface and quickly merge with the
surface. (g)–(h): The breaking wave entrains another group of air cylinders
and bubbles that quickly merge with the surface.

The canonical problem of a Stokes wave with a large initial steepness is
analyzed here. The wavelength $\lambda$ is $25\,\mathrm{cm}$. The simulation
domain is set to be $\lambda\times\lambda\times\lambda/2$. Periodic boundary
conditions are applied in the streamwise and spanwise directions, while a free
slip boundary condition is adopted in the vertical direction. Three cases are
analyzed with different initial wave steepnesses of $S=0.38$, $0.45$, and
$0.55$ corresponding to types of waves from a spilling breaker to a strong
plunging breaker. The density ratio $\rho_{a}/\rho_{w}$ and viscosity ratio
$\mu_{a}/\mu_{w}$ are set to $0.0012$ and $0.0154$, respectively. The Reynolds
number, Froude number, and Weber number are set as $10^{4}$, $1$, and
$8417.5$, respectively. The simulation time step is $dt=4\times
10^{-5}T\approx 1.6\times 10^{-5}\,\mathrm{s}$, where $T$ is the wave period.
A uniform-resolution $512\times 512\times 256$ grid is used, and the grid size
is approximately $0.5\,\mathrm{mm}$. The time step for tracking bubble over
time or detecting bubble events are selected to be $\Delta t=500dt\approx
8\times 10^{-3}\mathrm{s}$. Higher-resolution cases will be performed in
future studies with an increase in computational power or with more advanced
numerical algorithms, for example, adaptive mesh refinement (Zeng et al.,
2022).

### Bubble identification and tracking

The breaking wave surface and bubbles can be illustrated by the zeroth
isosurface of the LS function. To identify bubbles from the LS function field
and calculate the bubble statistics, the connected component labeling (CCL)
algorithm (Hermann, 2010; Tomar et al., 2010; Wang et al., 2016; Deike et al.,
2016; Bakshi et al., 2016; Hsiao et al., 2017; Gao et al., 2018; Hendrickson
et al., 2019) has been employed in many studies. In the simulations, bubbles
comprise grid cells with air surrounded by water. The CCL method uses a
breadth-first search (BFS) algorithm to find and tag the grid cells that
contain air and are connected with each other. Although the CCL method has
been successfully applied to resolve many physical problems, it can identify
bubbles within the flow field at only a single time instance. In contrast,
tracking bubbles and detecting their evolutionary behaviors (such as their
fragmentation and coalescence) over time, an algorithm whose capabilities
exceed those of the conventional CCL method is required.

Figure 2:  Visualization of a wave simulation with $S=0.45$ at different
times. (a): A jet forms on the wave crest. (b)–(c): The overturning wave crest
strikes the wave surface, producing multiple cavities and splash jets.
(d)–(e): Large cavities break up into large bubbles due to the instability of
the cylinder wall, and larger bubbles fragment into smaller ones in a
fragmentation cascade. (f): Larger bubbles rise toward the wave surface faster
than smaller bubbles, leaving the latter suspended underwater for a relatively
long time.

Here, we track bubbles and detect their behaviors using a novel algorithm
called the optimal network (ON) method, which was proposed in our recent work
(Gao et al., 2021a). The ON method classifies bubble behaviors into five
categories: continuity, fragmentation, coalescence, formation, and extinction.
Continuity means that a bubble is only transported by the flow and that its
volume does not change. Fragmentation is the process wherein a larger bubble
breaks up into smaller ones, while coalescence is the reverse process of
fragmentation; both fragmentation and coalescence are assumed to be binary.
Finally, formation and extinction represent a bubble forming and disappearing,
respectively, at the wave surface. The ON method tracks bubbles and detects
bubble events by making connections among bubbles at sequential time
instances. Based on the detected bubble events, we can derive the relationship
among bubble volumes, velocities, and positions at two sequential time
instances. For example, assuming continuity between two sequential time
instances, two bubbles with volumes $V_{i}$ and $V_{i+1}$, velocities
$\bm{u}_{i}$ and $\bm{u}_{i+1}$, and positions $\bm{x}_{i}$ and $\bm{x}_{i+1}$
satisfy the equations

Figure 3:  Visualization of a wave simulation with $S=0.55$ at different
times. See the caption of figure 2 for further details.

$V_{i}=V_{i+1}$ (7)

and

$\bm{x}_{i}V_{i}+\frac{1}{2}\left(V_{i}\bm{u}_{i}+V_{i+1}\bm{u}_{i+1}\right)\Delta
t=\bm{x}_{i+1}V_{i+1},$ (8)

where $\Delta t$ is the time interval between the two time instances.
Similarly, we can also obtain equations for the binary fragmentation and
binary coalescence processes. Based on the constraints imposed on the
positions, velocities, and volumes of bubbles between sequential time
instances, the ON method defines error functions, namely, pseudodistance error
functions. The pseudodistance error function for continuity is defined as

$\Theta_{CN}=\alpha_{1}\Delta_{p}+\alpha_{2}\Delta_{V}+\alpha_{3}\Delta_{l},$
(9)

with

$\Delta_{p}^{2}=\left\|\bm{x}_{i}+\frac{1}{2}\bm{u}_{i}\Delta
t-\left(\bm{x}_{i+1}-\frac{1}{2}\bm{u}_{i+1}\Delta t\right)\right\|^{2},$ (10)

Figure 4:  (a): Bubble events and cylinders in a simulated breaking wave with
$S=0.38$. (b): Average bubble eccentricity as a function of time. In (a),
circles denote bubble events, circle colors denote the event type, and size
indicates the magnitude of eccentricity. Circles are plotted horizontally with
time and vertically with the volume-equivalent radius. Purple diamonds denote
cylinders, and their size indicates the cross-sectional area-averaged radius.
Diamonds are plotted horizontally by the formation time and vertically by the
volume-equivalent radius. The lifetime of a cylinder is indicated by a purple
‘x’ connected to a diamond with a horizontal dotted line. In (b), blue and red
lines denote the average bubble eccentricity with and without considering air
cavities.

$\Delta_{V}=\frac{\left|V_{i+1}-V_{i}\right|}{\left(V_{i}V_{i+1}\right)^{1/3}},$
(11)

and

$\Delta_{l}^{2}=\left\|\bm{x}_{i}+\frac{1}{2}\bm{u}_{i}\Delta
t-\left(\bm{x}_{i+1}-\frac{1}{2}\bm{u}_{i+1}\Delta t\right)\right\|^{2},$ (12)

where $\alpha_{1}$, $\alpha_{2}$, and $\alpha_{3}$ are free parameters (good
values for these parameters are $0.45$, $0.45$, and $0.1$, respectively). A
pseudodistance error function contains three terms: $\Delta_{p}$,
$\Delta_{V}$, and $\Delta_{l}$. The first two terms come from equations (8)
and (7). The third term is used to ensure that the events can occur only
locally. Similarly, we can define pseudodistance error functions for binary
fragmentation and binary coalescence. Here, the formulas for $\Delta_{p}^{2}$
and $\Delta_{l}^{2}$ are the same for the continuity event. However, this is
not the case for fragmentation and coalescence. More details about the formula
and their discussion can be found in Gao el al. (2021a). All pseudodistance
error functions should take minimum values when the bubble events are
correctly identified. The ON method makes connections by selecting the minima
from a set of pseudodistance error functions. More information about
establishing these networks can be found in Gao et al. (2021a). The robustness
of our bubble tracking algorithm has been tested extensively with various
simulation cases, and the results show that the accuracies for continuity,
binary fragmentation, and binary coalescence are 99.5%, 90%, and 95%,
respectively.

Figure 5:  (a): Bubble events and cylinders in a simulated breaking wave with
$S=0.45$. (b): Average bubble eccentricity as a function of time. See the more
detailed description in the caption of figure 4 for the meanings of the symbol
colors and sizes.

### Sound generated by bubbles

The underwater noise created within a breaking wave is calculated based on the
wave noise model proposed by Deane (1997) and Deane & Stokes (2010). The wave
noise model assumes that newly formed bubbles created throughout the
acoustically active period of wave breaking are the dominant source of sound
and therefore relates the production of sound with these newly formed bubbles,
which can be identified using the ON method. In addition, bubble damping
effects on underwater noise are considered in this work using the e-folding
length and correction factors. More information on the calculation of the
sound generated by bubbles can be found in Gao et al., (2021b).

## Results

### Bubble entrainment procedure

Three cases, viz., $S=0.38$, $S=0.45$, and $S=0.55$, representing types of
waves from a spilling breaker to a plunging breaker, are examined here.
Visualizations of the breaking wave surface and bubble entrainment process for
each of the three cases are plotted in figures 1, 2, and 3.

The wave breaking and bubble entrainment processes are similar among the three
wave slopes simulated herein. Initially, a jet forms on the wave crest. Then,
the overturning wave crest strikes the wave surface, producing multiple
cavities and splash jets. Filaments and cavities eventually break up into
bubbles due to the instability of air cylinders, and larger bubbles fragment
into smaller bubbles via a fragmentation cascade. Finally, larger bubbles rise
toward the wave surface faster than the smaller bubbles, leaving the latter
suspended underwater for a relatively long time. With increasing wave
steepness, the wave breaking process becomes more energetic, leading to the
creation of more bubbles. In addition, bubbles are entrained at shallower
depths when $S=0.38$ than when $S=0.55$. As shown in figures 1, 2, and 3, two
bubble creation mechanisms are observed: air filament/cavity breakup and
turbulence fragmentation cascade. The air filament/cavity breakup happens at
the early stage of the breaking waves when the turbulence is weak and produces
bubbles of various sizes, which is followed by the process that relatively
large bubbles fragment into smaller ones due to turbulence fluctuations.
Comparing the simulation results for different wave steepness cases, the
turbulence intensity increases as the wave steepness increases. This might
imply that the fragmentation cascade process is more important for the high
steepness case, whereas the air filament/cavity breakup seems more important
for the low steepness case. For example, for the low wave steepness case
($S=0.38$), from figure 1, most of the bubbles are created by the air
filament/cavity breakup. It is very difficult to develop an algorithm that can
robustly distinguish the bubbles created by different mechanisms. Therefore,
in this work, we only provide qualitative descriptions of the two bubble
creation mechanisms. Detailed quantitative and accurate analysis will be in
the future study.

Figure 6:  (a): Bubble events and cylinders in a simulated breaking wave with
$S=0.55$. (b): Average bubble eccentricity as a function of time. See the more
detailed description in the caption of figure 4 for the meanings of the symbol
colors and sizes.

Bubble creation events can be robustly and accurately detected by the ON
method. The bubble creation events detected by the ON method for the cases
$S=0.38$, $S=0.45$, and $S=0.55$ are plotted in figures 4(a), 5(a), and 6(a),
respectively. Three categories for bubble behaviors, namely, binary
fragmentation, binary coalescence, and other processes, and two categories for
cavity behaviors, namely, creation and extinction, are considered here. A
cavity is defined as an air cylinder that spans across the whole simulation
domain in the spanwise direction. A cavity is represented by a purple diamond
connected to an ‘x’ with a horizontal dotted line, where the diamond and ‘x’
denote creation and extinction events, respectively, and the sizes of the
diamond and ‘x’ represent the cross-sectional area-averaged radius of the
cylinder. Blue, red, and yellow circles represent binary fragmentation events,
binary coalescence events, and other processes, respectively. These other
processes, including multiple fragmentation, multiple coalescence, and direct
entrainment, are the events that the ON method cannot identify. The circle
size denotes the bubble eccentricity, which is defined as

$e=\frac{d_{max}}{r_{v}},$ (13)

where $d_{max}$ is the maximum distance from a point in the bubble to the
center of the volume-equivalent spherical bubble and $r_{v}$ is the volume-
equivalent radius of the bubble. The eccentricity describes the degree to
which the shape of the bubble is irregular and takes a value of $1$ for a
perfectly spherical bubble. The dashed lines in figures 4(a), 5(a), and 6(a)
show a scale equal to $2.5$ grid lengths. Note that for bubbles smaller than
this scale, the physics of the surface tension become inaccurate in the
simulation. Here we still plotted the data points for bubbles smaller than
this scale, however, caution should be taken and higher resolution simulations
are needed for dealing with these bubbles.

Figure 7:  Identified bubbles and cavities in a breaking wave ($S=0.55$). The
yellow circles and squares denote identified bubbles and air cylinders,
respectively. (a)–(c): Air cylinders break up into multiple bubbles. (d):
Larger bubbles fragment into smaller bubbles, which is referred to as a bubble
fragmentation cascade. (e)–(f): Degassing process.

Figure 4 demonstrates the entrainment of a few air cylinders, followed by two
clusters of bubbles, which are created by two groups of entrained air
cylinders. Some air cylinders are not identified as air cavities because they
do not span across the whole domain in the spanwise direction. Most bubbles
are created by other processes because they are associated mainly with
cylinder instability and thus cannot be identified by the ON method. Compared
to those when $S=0.38$, the breaking waves are more energetic and produce more
bubbles when $S=0.45$ and $S=0.55$. A number of cylinders are entrained at the
onset of wave breaking and subsequently disappear when many bubbles are
created by binary fragmentation, binary coalescence, and other processes.
Initially, bubbles are created with irregular shapes, but they become more
spherical over time. Smaller bubbles tend to have small eccentricities because
surface tension is more important on small scales. Figure 6 clearly shows
that, as expected, the largest bubble size decreases over time, which is the
result of a fragmentation cascade.

Figure 8:  Ensemble-averaged bubble size spectrum. Color denotes time. The
dotted line and dashed–dotted line correspond to the power-law scalings of
$-3/2$ and $-10/3$ representing the theoretical predictions for sub-Hinze-
scale bubbles and super-Hinze-scale bubbles, respectively.

The average bubble eccentricities as a function of time for the cases
$S=0.38$, $S=0.45$, and $S=0.55$ are plotted in figures 4(b), 5(b), and 6(b),
respectively. The average eccentricity is defined as

$\bar{e}=\frac{\sum{r_{v}e}}{\sum{r_{v}}},$ (14)

where $\sum$ denotes the summation over all the bubbles in the simulation
domain. In figures 4(b), 5(b), and 6(b), the blue and red lines represent the
average eccentricities with and without cavities, respectively. A comparison
between panels (a) and (b) in figures 4, 5, and 6 generally suggests that the
eccentricity peaks correspond to cylinder breakup and bubble creation events.
Then, the average eccentricity decreases after the peak, indicating that the
bubbles are entrained with irregular shapes and become more spherical over
time. This result reveals that cylinder instability might be critical for the
transition from the two-dimensional wave field to the three-dimensional
turbulent field of a breaking wave.

### Bubble size spectrum

The bubble size spectrum is an important physical quantity for quantitatively
describing bubble entrainment. Garrett et al. (2000) stated that breaking
waves initially entrain large bubbles, which subsequently break up into
smaller bubbles. This process is referred to as a bubble fragmentation
cascade. Considering that bubble entrainment is associated with the average
rates of the supply of air and energy dissipation, Garrett et al. (2000)
conducted a dimensional analysis and proposed that the bubble size spectrum
shows a $-10/3$ power-law scaling for bubbles with radii greater than the
Hinze scale, which is the scale at which the bubble surface tension balances
the turbulent fluctuation.

Figure 9:  Oblique view of the bubble trajectories for the case $S=0.38$. Only
bubble trajectories with bubble radii greater than $0.5\,\mathrm{mm}$ and
residence times greater than $0.04T$ are plotted here. Line color and width
denote the bubble creation time and bubble size, respectively. The scatters in
the bottom panel represent the bubble bursting locations, and the scatter size
is proportional to the bubble size. Figure 10:  Top view of the bubble
trajectories for the case $S=0.38$. For more detailed descriptions of the
meanings of the line color, line width, scatter location, and scatter size,
see the caption of figure 9.

Deane & Stokes (2002) conducted experiments and verified the $-10/3$ power-law
scaling for super-Hinze-scale bubbles (bubbles whose radii exceed the Hinze
scale) at the end of the active phase of air entrainment. Moreover, they found
that sub-Hinze-scale bubbles (bubbles whose radii are smaller than the Hinze
scale) follow a $-3/2$ power-law scaling. However, the detailed mechanism of
sub-Hinze-scale bubble entrainment remains unclear. Figure 7 shows the bubble
entrainment process, and figure 8 shows the corresponding ensemble-averaged
bubble size spectrum. The definition of the bubbles used in calculating the
size spectrum is consistent with the circles in figure 6(a). The fragmentation
model proposed in Garrett et al. (2000), which leads to a -10/3 power-law
scaling for the bubble size spectrum, embodies an average air supply rate or a
source volume due to the entrainment process during wave breaking. The recent
work of Gaylo et al. (2021) defines an entrainment size distribution of bubble
sources which reproduces the same -10/3 scaling for the equilibrium bulk
bubble size spectra (for super-Hinze scale bubbles) via fragmentation
cascading, in the weak volume injection regime. In our current study, using
the CCL algorithm mentioned in the preceding section, we are able to identify
the spanwise air cylinders or cavities entrained by the breaking wave during
initial stages, and treat them to be the primary source volumes which undergo
breakup to generate bubbles. However, demarcating and estimating the bubble
injection size distribution due to air entrainment in a breaking wave, and
studying its subsequent evolution into the bulk bubble size spectrum by
employing the bubble identification and tracking subroutines, is a future
topic of interest.

Figure 11:  Oblique view of the bubble trajectories for the case $S=0.45$. For
more detailed descriptions of the meanings of the line color, line width,
scatter location, and scatter size, see the caption of figure 9.

In figure 7, the yellow circles and rectangles correspond to bubbles and air
cylinders, respectively. Air cavities and filaments are entrained in the early
stage of wave breaking (see figures 7(a) and 7(b)). At $t=0.84T$ and
$t=1.20T$, the slopes of the bubble size spectrum (black line and purple line,
respectively, in figure 8) do not follow the $-10/3$ power-law scaling for
super-Hinze-scale bubbles because some of the air filaments have broken up
into bubbles of different scales due to the abovementioned cylinder
instability (Gao et al., 2021b), which is completely different from a
fragmentation cascade. At $t=1.48T$, $t=1.76T$, and $t=2.04T$, air cavities
break up and create large bubbles, and some of these large bubbles fragment
into smaller bubbles; this phenomenon represents the fragmentation cascade
process described above. In addition, because large bubbles have higher
ascension speeds than small bubbles, some of these large bubbles rise up and
burst on the wave surface. The turbulence saturation hypothesis (Deane et al.
2016) implies that the bubble Hinze scale shows less dependence on the wave
scales and turbulence energy dissipation. Therefore, we expect that the Hinze
scale in this study is about $1\,\mathrm{mm}$. We can clearly see that the
bubble size spectrum for a fragmentation cascade shows a $-10/3$ power-law
scaling for bubbles greater than the Hinze scale, which is consistent with the
conclusions in the previous literature. The bubble size spectrum at the Hinze
scale seems showing a transition power-law scaling. This point has been
verified by the grid convergence study in the authors’ previous work and will
not be repeated here. More information can be found in figure 8 in Gao et al.
(2021b).

Figure 12:  Top view of the bubble trajectories for the case $S=0.45$. For
more detailed descriptions of the meanings of the line color, line width,
scatter location, and scatter size, see the caption of figure 9.

At $t=3.00T$, the slope of the size spectrum is steeper than $-10/3$ (red line
in figure 8) because the large bubbles have disappeared, leaving only small
bubbles in the water that persist for a relatively long time. The temporal
evolution of the bubble size spectrum shows that the bubble creation mechanism
due to cylinder instability (Gao et al., 2021) has significant impacts on the
bubble size spectrum through the following aspects. First, the bubble size
spectrum at the early wave breaking stage ($t=0.84T$ and $t=1.20T$) does not
follow the $-10/3$ power-law scaling because the bubble creation mechanism is
not a turbulence-driven fragmentation cascade. We discover that some air
cylinders break up into small bubbles, which might indicate that cylinder
instability is critical during this stage. Second, as shown in figure 8, the
two peaks near $r=5\,\mathrm{mm}$ and $r=8\,\mathrm{mm}$ in the bubble size
spectrum at $t=1.48T$ (blue line) are created by the breakup of an air cavity
(see figure 7(c)). Finally, the relatively large bubbles created by the
breakup of an air cavity are further broken up through a fragmentation
cascade, indicating that these large bubbles are important in the bubble size
spectrum at the end of the acoustically active phase.

Figure 13:  Oblique view of the bubble trajectories for the case $S=0.55$. For
more detailed descriptions of the meanings of the line color, line width,
scatter location, and scatter size, see the caption of figure 9. Figure 14:
Top view of the bubble trajectories for the case $S=0.55$. For more detailed
descriptions of the meanings of the line color, line width, scatter location,
and scatter size, see the caption of figure 9.

Bubble trajectories play important roles in many ocean–atmosphere processes,
such as air–sea gas transfer and sea spray generation. Compared with
nonbreaking waves, breaking waves and bubbles increase the air–water interface
area, resulting in an enhanced air–sea gas transfer rate. In addition, bubbles
burst on the wave surface at the end of some bubble trajectories and produce
film drops and jet drops. For these and other reasons, detailed investigations
of bubble trajectories are necessary for ongoing research. Nevertheless, there
is no literature reporting the trajectories of bubbles in breaking waves using
a direct numerical simulation because of the difficulties in tracking bubbles
and detecting their evolving behavior over time. However, the recent progress
afforded by the bubble tracking algorithm developed by Gao et al. (2021a),
namely, the ON method, enables us to accurately and robustly detect and track
bubble events. To identify bubble trajectories, the connections among
continuous bubbles are established over time.

Figure 15:  Oblique view of the bubble trajectories for the case $S=0.55$ with
bubble locations shifted according to their creation time. For more detailed
descriptions of the meanings of the line color, line width, scatter location,
and scatter size, see the caption of figure 9.

The oblique and top views of the bubble trajectories for the cases $S=0.38$,
$S=0.45$, and $S=0.55$ are plotted in figures 9–14. Only the trajectories of
bubbles whose radii exceed $0.5\,\mathrm{mm}$ and whose residence times are
greater than or equal to $0.04T$ ($T$ is the wave period) are plotted here.
The color signifies the bubble creation time. The line width and scatter size
denote the bubble size, and the scatter location reflects the bubble bursting
location. As shown in figures 9 and 10, only a few bubbles are created in the
water, and the bubble trajectories are relatively short and straight. For the
cases $S=0.45$ and $S=0.55$, more bubbles are created than in the case
$S=0.38$, and the bubble trajectories are more chaotic and arcuate with some
trajectories exhibiting rotation. The bubble trajectories have three unique
features: parallel curves, rotational structures, and tails pointing upward
and forward. For example, we observe the parallel green lines at
$x/\lambda\approx 0.9$ in figures 11 and 12 and the parallel blue lines at
$x/\lambda\approx 0.3$ in figures 13 and 14. The parallel structures show that
the bubbles are created at almost the same time with similar sizes and are
distributed uniformly in the spanwise direction, supporting the hypothesis of
bubble production by the cylinder instability mechanism described in Gao et
al. (2021b). In contrast, the rotational orange curves at $x/\lambda\approx 1$
in figure 13 are the result of the interaction between the bubble trajectories
and underwater flow vortex structures. In addition, owing to buoyancy, some
bubbles rise up while being transported; as a result, the trajectory tails of
some large bubbles point upward and forward. These bubbles are generally the
large ones that the vortex structures are not able to capture. If we shift the
bubble trajectories location according to their creation time, figure 13 can
be replotted as figure 15, where the bubble and air cylinder events can be
clearly observed.

Figure 16:  Interaction between the bubble trajectories and flow field for the
case $S=0.38$. The velocity field is plotted as the vectors with the spanwise-
averaged components $u$ and $v$. The averaged wave surface is plotted as the
black line, which is the zeroth isosurface of the spanwise-averaged LS
function. The colored lines are the bubble trajectories. For more detailed
descriptions of the meanings of the color and width of the trajectories, see
the caption of figure 9. Figure 17:  Interaction between the bubble
trajectories and flow field for the case $S=0.45$. For more details about the
meanings of the vectors and lines, see the caption of figure 16.

To check the interactions between the bubble trajectories and flow field, the
spanwise-averaged velocities and the bubble trajectories are plotted in
figures 16, 17, and 18 for the cases $S=0.38$, $S=0.45$, and $S=0.55$,
respectively. For the small wave steepness case ($S=0.38$), as shown in figure
16, bubbles are entrained at shallow depths and quickly merge with the wave
surface, resulting in relatively short, straight trajectories. To illustrate
the vortex structures in breaking waves, a straightforward way is to use the
vorticity contours. However, to show the colored trajectories and their
interaction with large vortex structure clearly, here, we use the velocity
vectors to show the large vortex structures. As shown in figures 17 and 18,
for the cases $S=0.45$ and $S=0.55$, the rotational vortex structures clearly
affect the transport of bubbles and produce rotational bubble trajectories.
Nevertheless, in this work, we only qualitatively describe the interactions
between the bubble trajectories and the flow field, and more detailed research
is needed in the future.

Figure 18:  Interaction between the bubble trajectories and flow field for the
case $S=0.55$. For more details about the meanings of the vectors and lines,
see the caption of figure 16.

### Wave noise spectrogram

The bubble creation events for the case $S=0.55$ are shown in figure 6. Bubble
acoustics are strongly associated with bubble creation events. Based on a
dataset of bubble creation events and using the wave noise calculation method
of Gao et al. (2021c), we calculate and investigate the wave noise spectrogram
for the case $S=0.55$. The spectrogram is plotted in figure 19. The wave noise
energy is concentrated in the time domain during
$0.3\,\mathrm{s}$–$0.8\,\mathrm{s}$, which is the acoustically active phase,
and in the frequency domain within $330\,\mathrm{Hz}$–$6600\,\mathrm{Hz}$,
which corresponds to bubble radii ranging from $10\,\mathrm{mm}$ to
$0.5\,\mathrm{mm}$. At approximately $0.6\,\mathrm{s}$, the high intensity of
low-frequency wave noise indicates the large bubbles produced by air cavities.
In figure 19, we plot a solid line denoting the $2.5$ grid length and
highlight that the wave frequencies above this line are not accurate due to
the increasingly inaccurate surface tension calculation attributable to the
grid resolution limitation. More results and the comparison between simulation
results with experimental datasets on wave noise power spectral density can be
found in Gao et al. (2021c).

Figure 19:  Wave noise spectrogram for the case $S=0.55$.

## Discussions

In this work, we investigated bubble creation mechanisms and bubble acoustic
characteristics for breaking waves. Specifically, a new model was developed to
relate bubble behaviors to underwater sound generation, and new physical
insight was found that drives bubble production within breaking wave crests in
this work. Our bubble acoustic model and the new insights on bubble creation
mechanisms can also be applied to other bubble-associated physical phenomena,
for example, breaking wave and bubble generation induced by vortex/interface
interaction or wave–body interaction. Hendrickson et al. (2022) developed an
air entrainment volume for the quasi-two-dimensional interactions of rising
surface-parallel vorticity with an air–water interface and applied their model
to the quasi-steady wave breaking behind a fully submerged horizontal circular
cylinder. Some air cylinders (air filament or cavity) similar to what we
observed in the breaking waves, can also be found in their simulations (See
figures 3 and 8 in Hendrickson et al., 2022). This implies that our theory for
predicting the bubble creation induced by air cylinder breakup is applicable
to their simulations. It will be also interesting to check bubble acoustics of
the wave–body interaction using our bubble acoustic model in future work.

The ON method can only detect continuity, binary fragmentation, and binary
coalescence. Therefore, in this work, we have only shown the results regarding
that can be robustly obtained by the ON method, for example, the bubble
trajectories and the bubble creation rate (wave spectrogram). However, bubbles
produced by cylinder instability involve multiple fragmentation, where the ON
method fails. Thus, we cannot distinguish the bubbles produced by different
mechanisms, such as bubble fragmentation cascade and cylinder instability,
using the ON method. A more advanced algorithm that can track bubble multiple
fragmentation and multiple coalescence is needed to provide more details on
the bubble statistics created by cylinder instability quantitatively.

The wave profile in the initial condition is uniform in the spanwise
direction. It is interesting to know the bubble creation and bubble statistics
if the perturbation is added along the spanwise direction. If the perturbation
is small, the most unstable mode of the perturbation grows fastest based on
the stability theory and the bubble produced by air cylinder instability
should still follow our air cylinder breakup theory. However, if the
perturbation is large enough, the final bubble sizes should depend on the
initial perturbation heavily. However, we expect that the perturbation might
influence the number of bubbles created, but it should not affect the bubble
size spectrum power-law scaling slope, because the power-law scaling seems a
strong and uniform law across a wide range of wave scales and various types of
breaking waves verified both experimentally and numerically. Deike et al.
(2016) performed breaking wave simulations with spanwise perturbation in the
initial wave profile. Their results show that the perturbation has little
effects on the bubble dynamics and the bubble statistics converge to the
power-law scaling. More simulations and studies will be performed to
investigate the initial perturbation effects in future work.

## Conclusions

In this work, we adopted a novel algorithm to track bubbles and detect their
evolutionary behaviors over time during breaking wave simulations and
investigated various bubble creation mechanisms and bubble trajectories. The
results show that instability along the boundary of an air cylinder is an
important mechanism for bubble creation and has significant effects on the
initial breaking wave process and bubble size spectrum. Bubble trajectories
show three unique patterns in breaking waves. Specifically, we observed
trajectories with rotational motion, trajectories pointing forward and upward,
and parallel trajectories. Finally, we investigated the acoustics of newly
formed bubbles and calculated the noise spectrogram radiated by a breaking
wave. In future work, we will conduct more detailed studies on bubble creation
mechanisms, bubble trajectories interacting with breaking wave flow fields,
and wave noise.

## Acknowledgements

The UMN team’s research was previously supported partially by the Gulf of
Mexico Research Initiative and the Office of Naval Research (Grant number
N00014-16-1-2192, Program Manager Dr. Reginald Beach) and is currently
sponsored by the Office of Naval Research (Grant number N00014-22-1-2481,
Program Manager Dr. Woei-Min Lin). GBD’s research is supported by the Office
of Naval Research (Grant number N00014-17-1-2633, Program Manager Dr. Kyle
Becker).

## References

  * [1] Bakshi, A., Altantzis, C., Bates, R.B. and Ghoniem, A.F., “Multiphase-flow Statistics using 3D Detection and Tracking Algorithm (MS3DATA): Methodology and application to large-scale fluidized beds”, Chemical Engineering Journal, Vol. 293, 2016, pp.355-364.
  * [2] Castro, A.M., Li, J., Hyman, M. and Carrica, P.M., “Turbulent and cavity free surface bubble entrainment with application to ship hydrodynamics”, Proceedings of the 30th Symposium on Naval Hydrodynamics, 2014, Hobart, Tasmania, Australia.
  * [3] Chan, W.H.R., Johnson, P.L., Moin, P. and Urzay, J., ”The turbulent bubble break-up cascade. Part 2. Numerical simulations of breaking waves”, Journal of Fluid Mechanics, Vol. 912, 2021, A43.
  * [4] Deane, G.B., “Sound generation and air entrainment by breaking waves in the surf zone”, The Journal of the Acoustical Society of America, Vol. 102(5), 1997, pp.2671-2689.
  * [5] Deane, G.B. and Stokes, M.D., “Scale dependence of bubble creation mechanisms in breaking waves”, Nature, Vol. 418(6900), 2002, pp.839-844.
  * [6] Deane, G.B. and Stokes, M.D., “Model calculations of the underwater noise of breaking waves and comparison with experiment”, The Journal of the Acoustical Society of America, Vol. 127(6), 2010, pp.3394-3410.
  * [7] Deane, G.B., Stokes, M.D. and Callaghan, A.H., “The saturation of fluid turbulence in breaking laboratory waves and implications for whitecaps”, Journal of Physical Oceanography, 46(3), 2016, pp.975-992.
  * [8] Deike, L., Melville, W.K. and Popinet, S., “Air entrainment and bubble statistics in breaking waves”, Journal of Fluid Mechanics, Vol. 801, 2016, pp.91-129.
  * [9] Gao, Q., Deane, G. and Shen, L., “Air entrainment and underwater noise of breaking waves”, Bulletin of the American Physical Society, 63, 2018.
  * [10] Gao, Q., Deane, G.B., Liu, H. and Shen, L., “A robust and accurate technique for Lagrangian tracking of bubbles and detecting fragmentation and coalescence”, International Journal of Multiphase Flow, Vol. 135, 2021a, p.103523.
  * [11] Gao, Q., Deane, G.B. and Shen, L., “Bubble production by air filament and cavity breakup in plunging breaking wave crests”, Journal of Fluid Mechanics, Vol. 929, 2021b, A44.
  * [12] Gao, Q., Shen, L. and Deane, G.B., “A numerical simulation framework for bubbly flow and sound generation in laboratory-scale breaking waves”, JASA Express Letters, Vol. 1(10), 2021c, p.100801.
  * [13] Garrett, C., Li, M. and Farmer, D., “The connection between bubble size spectra and energy dissipation rates in the upper ocean”, Journal of Physical Oceanography, Vol. 30(9), 2000, pp.2163-2171.
  * [14] Gaylo, D.B., Hendrickson, K. and Yue, D.K., “Effects of power-law entrainment on bubble fragmentation cascades”, Journal of Fluid Mechanics, Vol. 917, 2021, R1.
  * [15] Hsiao, C.T., Ma, J. and Chahine, G.L., “Multiscale two-phase flow modeling of sheet and cloud cavitation”, International Journal of Multiphase Flow, Vol. 90, 2017, pp.102-117.
  * [16] Hendrickson, K., Weymouth, G.D., Yu, X. and Yue, D.K.P., “Wake behind a three-dimensional dry transom stern. Part 1. Flow structure and large-scale air entrainment”, Journal of Fluid Mechanics, Vol. 875, 2019, pp.854-883.
  * [17] Hendrickson, K., Yu, X. and Yue, D.K.P., “Modelling entrainment volume due to surface-parallel vortex interactions with an air–water interface”, Journal of Fluid Mechanics, Vol. 938, 2022, A12.
  * [18] Herrmann, M., “A parallel Eulerian interface tracking/Lagrangian point particle multi-scale coupling procedure”, Journal of Computational Physics, Vol. 229(3), 2010, pp.745-759.
  * [19] Li, J., Castro, A.M. and Carrica, P., “Modeling bubble entrainment for ship flows: Model improvements, validation and grid convergence”, Proceedings of the 31st Symposium on Naval Hydrodynamics, 2016, Monterrey, CA, USA.
  * [20] Sussman, M. and Puckett, E.G., ”A coupled level set and volume-of-fluid method for computing 3D and axisymmetric incompressible two-phase flows”, Journal of Computational Physics, Vol. 162(2), 2000, pp.301-337.
  * [21] Tomar, G., Fuster, D., Zaleski, S. and Popinet, S., “Multiscale simulations of primary atomization”, Computers & Fluids, Vol. 39(10), 2010, pp.1864-1874.
  * [22] Wang, Z., Yang, J. and Stern, F., “High-fidelity simulations of bubble, droplet and spray formation in breaking waves”, Journal of Fluid Mechanics, Vol. 792, 2016, pp.307-327.
  * [23] Yang, Z., Deng, B.Q. and Shen, L., “Direct numerical simulation of wind turbulence over breaking waves”, Journal of Fluid Mechanics, Vol. 850, 2018, pp. 120-155.
  * [24] Yu, X., Hendrickson, K. and Yue, D.K., “Scale separation and dependence of entrainment bubble-size distribution in free-surface turbulence”, Journal of Fluid Mechanics, Vol. 885, 2020, R2.
  * [25] Zeng, Y., Xuan, A., Blaschke, J. and Shen, L., “A parallel cell-centered adaptive level set framework for efficient simulation of two-phase flows with subcycling and non-subcycling”, Journal of Computational Physics, Vol. 448, 2022, p.110740.


Foundations of the Socio-physical Model of Activities (SOMA) for Autonomous Robotic AgentsThis work was funded by the German Research Foundation (DFG) as part of Collaborative Research Center (SFB) 1320 EASE – Everyday Activity Science and Engineering, University of Bremen (<http://www.ease-crc.org/>), subprojects H2, P1 and R1.

Foundations of SOMA

Daniel Beßler1 Robert Porzel2 Mihai Pomarlan3 Abhijit Vyas1 Sebastian Höffner2 Michael Beetz1 Rainer Malaka2 John Bateman3

D. Beßler et al.

Department of Artificial Intelligence,
University of Bremen, 28359 Bremen, Germany Digital Media Lab,
University of Bremen, 28359 Bremen, Germany Department of Linguistics,
University of Bremen, 28359 Bremen, Germany

In this paper, we present foundations of the Socio-physical Model of Activities (SOMA). SOMA represents both the physical as well as the social context of everyday activities. Such tasks seem to be trivial for humans, however, they pose severe problems for artificial agents. For starters, a natural language command requesting something will leave many pieces of information necessary for performing the task unspecified. Humans can solve such problems fast as we reduce the search space by recourse to prior knowledge such as a connected collection of plans that describe how certain goals can be achieved at various levels of abstraction. Rather than enumerating fine-grained physical contexts SOMA sets out to include socially constructed knowledge about the functions of actions to achieve a variety of goals or the roles objects can play in a given situation. As the human cognition system is capable of generalizing experiences into abstract knowledge pieces applicable to novel situations, we argue that both physical and social context need be modeled to tackle these challenges in a general manner. This is represented by the link between the physical and social context in SOMA where relationships are established between occurrences and generalizations of them, which has been demonstrated in several use cases that validate SOMA.

§ INTRODUCTION

In spite of undoubtedly being ubiquitous, the domain of everyday activities poses considerable challenges.
Many people perform activities such as cooking or cleaning almost every day.
This includes to
select and manipulate ingredients, use tools and devices, arrange the prepared dishes for serving and clean up afterwards – and do it all quick and robust without recourse to an advanced computational theory.
Further, the amount of information provided in a description of a task – such as a natural language command requesting its completion – is much less than the amount of information needed to perform the task. This
raises the question how humans are able to decide so quickly what to do next, despite ambiguity and underspecification.

Lenat and Feigenbaum observe that more knowledge implies less search [1]. Knowledge of many possible plans, as well as knowledge of the world in general, seems to be the secret of human performance. There is no algorithmic reason why tomatoes and oregano go well together, or why a raw egg must be handled with care. The cook simply has to know these things. Such knowledge of the world is taught, observed, and then ingrained by practice. As Anderson observes, an agent has a great deal of knowledge [of everyday activities], which comes as a result of the activity being common [2]. As human beings, we acquire such knowledge naturally over the course of our lives.
Furthermore, to provide a better understanding of our subsequent discussion,
we will briefly introduce some of the essential concepts and data structures that drive our research efforts.
A lot of what we learn, we learn by doing, or by watching others. This suggests that a robot must have mechanisms to organize and interpret observations, either of its own behavior or of other agents, into structures that are then amenable for other computational tasks. To this end, the EASE-CRC put forth the concept of narratively-enabled episodic memories (NEEMs) as a representation of the physical context. NEEMs are comprehensive logs of raw sensor data, actuator control histories and perception events, all semantically annotated with information about what the robot is doing and why using the terminology provided by SOMA.

The computational tasks that must be solved when acting in the physical world are often very complex and beyond what is thought to be tractable. This, however, is only the case when these problems are regarded in their full generality and not for restricted versions of these problems. However, the knowledge representing such pragmatic solutions goes beyond modeling physical events and requires models of the social context by means of which the physical events can be realized and interpreted.
To represent this, the EASE-CRC has put forth the concept of pragmatic everyday activity manifolds (PEAMs), which are descriptions of a problem subspace that are more amenable to solving. Depending on the larger problem kind,  PEAMs can, for example, be expressed as plans with some placeholders for subtasks, using vague parameterizations for controllers and heuristic strategies to classify objects by their  dispositions.

The overall goal of our research is to enable robotic agents to perform everyday activities with similar robustness and flexibility as human agents do. Given this aim of the EASE-CRC, we must, in some sense, get the robot to know what humans know about the world, at least as it pertains to everyday activities. This presents several challenges, beyond the scope of what needs to be known to represent such intricate and extensive domain.
There lies the question of how to represent and structure this knowledge in order to realize a similar robustness, flexibility and efficiency in performance.
In addition, there are challenges concerning
the acquisition and learnability of the corresponding structures. In this paper, we will focus on how this knowledge is represented.
We will describe an upper level ontology together with several ontology modules aimed to address this general ontology design challenge.
The ontology is openly available [<https://github.com/knowrob/ease_ontology/tree/ekaw202>].

Individual applications of our model have been published in dedicated venues and will be referenced and summarized to validate the contribution of this work.
As depicted in Figure <ref>, our focus lies on representing both the physical context of realized everyday activities (NEEMs), as well as interpretations thereof as the social context (PEAMs).

mynode/.style=draw=none,minimum width=2.4cm,minimum height=0.6cm,text width=2.2cm, align=center,
myconcept/.style=mynode,rounded corners,
mysection/.style=draw=red,minimum width=4.1cm,minimum height=4.0cm,text width=4.1cm, align=center,
[myconcept,physical] (EV) Event;
[myconcept,social,right=0.5cm of EV] (DESCR) Description;
(EV.north east) edge[myedge,bend left=20]
node[mylabel,above] interpretation
(DESCR.north west);
(DESCR.south west) edge[myedge,bend left=20]

node[mylabel,below] realization
(EV.south east);

[mysection,left=0.0cm of EV.center,yshift=1.5cm,
fill=ground!20!white,draw=ground] (LEFT) ;
[mysection,right=0.0cm of DESCR.center,yshift=1.5cm,
fill=descriptive!20!white,draw=descriptive] (RIGHT) ;
[mylabel,anchor=west,yshift=-0.2cm,xshift=0.1cm] at (EV.west -| LEFT.west) (PC) physical context;
[mylabel,anchor=north east,xshift=-0.1cm] at (PC.north east -| RIGHT.east) social context;

[below=0.2cm of RIGHT.north,anchor=north,

(an action
(type pouring-onto)
(some substance))
(some container))
(some surface))))

[below=0.2cm of LEFT.north,anchor=north,inner sep=0,draw=ground]

[mynode,fill=gray!20!white,draw=gray,yshift=1.8cm,text width=6.2cm] (SOMA) at ($(EV)!0.5!(DESCR)$) SOMA;

SOMA represents physical and social context, and supports robotic agents in interpreting observed events, and realizing abstract descriptions.

§ RELATED WORK

Knowledge representation and reasoning in autonomous robot control is a fairly extensive field of research with developments in both service and industrial robotics.
Olivares et al. provide a comprehensive comparison of different approaches [3].

One example in the industrial robotics domain is the ROSETTA project [4, 5, 6].
Its initial scope was reconfiguration and adaptation of robot-based manufacturing cells, however, the authors have, since then, further developed their activity modeling for coping with a wider range of industrial tasks.

Other authors have focused on modeling
industrial task structure, part geometry features, or task teaching from examples [7, 8, 9, 10, 11].
Compared to the everyday activity domain, industrial tasks considered in above works are more structured, and less demanding in terms of flexibility.

An approach to activity modeling in the service robotics domain is presented by Tenorth and Beetz [12].
The scope of their work is similar to ours as the authors also consider how activity knowledge can be used to fill knowledge gaps in abstract instructions given to a robotic agent performing everyday activities.
However, the scope of the work presented here is wider, as we also consider how activity knowledge can be used for the interpretation of observations.
Our activity modeling is further more detailed in terms of activity structure as we also consider the processes and states that occur during an activity.

Another difference is that, in their modeling, there is no distinction between physical and social context, and therefore less expressivity compared to our model.

A more general approach to activity modeling for robotic agents is presented by the IEEE-RAS working group ORA [13].
The group has the goal of defining a standard ontology for various sub-domains of robotics, including a model for object manipulation tasks.
It has defined a core ORA ontology [14], as well as additional modules for industrial tasks such as kitting [15].
In terms of methodology, we differ in foundational assumptions we assert, which has important consequences on the structure of our ontology, modeling workflow, and inferential power.
In the case of ORA, the SUMO upper-level ontology is used as foundational layer.
Compared to SUMO, we use a richer axiomatization of entities on the foundational layer,
and put particular emphasis on the distinction between physical and social activity context.

§ OVERVIEW

In this section, we will provide a broad overview about SOMA.
First, we will discuss the scope of our work which is rooted in autonomous robotics in Section <ref>.
We will then elaborate on ontological commitments in SOMA, and in particular about commitments that help coping with both physical and social context in Section <ref>.
Finally, we will provide an overview about how SOMA is organized into different ontological modules in Section <ref>.

§.§ Scope

The broad scope of our work is everyday object manipulation tasks in autonomous robot control, and in particular the motion and force characteristics of objects that interact with each other.
The research question driving us is whether a single general control program can be written that can generate adequate behavior in many different contexts: for different tasks, objects, and environments.

One of the challenges is that, using such a general plan, the agent needs to fill the knowledge gaps between abstract instructions included in the plan and the realization of context specific behavior. That is, for example, the many ways of how humans perform a pouring task depending on the source from which is poured, the destination, and the substance that is to be poured.
Another challenge is that object manipulation tasks may fail if the agent does not perform the motions competently and well. This is caused by the agent choosing inappropriate parametrization of its control-level functions.

The employment of a general plan thus requires an abstract task and object model, and a mechanism to apply this abstract knowledge in situational context.
To achieve this, an agent needs to be equipped with the necessary common-sense and intuitive physics knowledge, which is what SOMA attempts.

§.§ Foundational Commitments

We decided to base our model on the DOLCE+DnS Ultralite (DUL) foundational framework [16].
This decision is greatly motivated by their underlying ontological commitments.

Firstly, DUL is not a revisionary model, but seeks to express stands that shape human cognition. Furthermore it assumes a reductionist approach – rather than capturing, for example, the flexibility of our usage of objects via multiple inheritance in a multiplicative manner, we commit to a reduced ground classification and use a descriptive approach for handling this flexibility. For this a primary branch of the ontology represents the ground physical model, e.g. objects and actions, while a secondary branch represents the social model, e.g. roles and tasks. All entities in the social branch would not exist without humans, i.e. they constitute social objects that represent concepts about or descriptions of ground elements.

Every axiomatization in the physical branch can, therefore, be regarded as expressing some physical context whereas axiomatizations in the descriptive social branch are used to express social contexts. A set of dedicated relations is provided that connect both branches. For example, as detailed in Section <ref>, the relation classifies connects ground objects, e.g. a hammer, with the roles they can play, i.e. potential classifications. Thus, we can state that a hammer can in some context be conceptualized as a murder weapon, a paper weight or a door stopper. Nevertheless, neither its ground ontological classification as a tool will change nor will hammers be subsumed as kinds of door stoppers, paper weights or weapons via multiple inheritance. Following a quick overview of the central modules of SOMA where these commitments apply, we will provide detailed examples of where and how our commitments apply in Sections <ref> and <ref>.

§.§ Module overview

SOMA is organized in several modules that conceptualize different aspects of physical and social activity context (Figure <ref>). The different modules correspond to different event types (ACT, PROC, STATE), objects that participate in the activity (OBJ), and execution context (EXEC). The scope of each of these modules is outlined below.

[owlclass] (SOMA)
SOMAphysical contextsocial     context ;
[owlclass,below right=1.0cm of SOMA,xshift=-2.5cm,yshift=0.4cm] (OBJ)  OBJObject, DispositionDesign, Affordance ;
[owlclass,below left=1.0cm of SOMA,xshift=2.5cm,yshift=0.4cm] (EXEC)  EXECGroundingBinding ;
[owlclass,above=2.5cm of SOMA,yshift=-0.4cm] (ACT)
ACTActionPlan ;
[owlclass,above left=1.0cm of SOMA,xshift=2.5cm,yshift=-0.4cm] (PROC)  PROCProcessProgression ;
[owlclass,above right=1.0cm of SOMA,xshift=-2.5cm,yshift=-0.4cm] (STATE)
STATEStateConfiguration ;
(SOMA) edge[myedge] (OBJ);
(SOMA) edge[myedge] (EXEC);
(SOMA) edge[myedge] (ACT);
(SOMA) edge[myedge] (PROC);
(SOMA) edge[myedge] (STATE);

The modular organization of SOMA. Each module defines concepts and relationships used to represent physical (orange) and social (purple) activity context.

The scope of the OBJ module is the contextualization of physical objects, and their qualities. The module includes two taxonomies used to classify objects: an object taxonomy in the grounded branch, and a role taxonomy in the descriptive branch. It further includes a taxonomy of dispositions to represent the potential of using an object in some way, and a design taxonomy used to categorize objects based on function, structure, and aesthetics.

The scope of the ACT module is the contextualization of actions. Actions are events performed by an agent (physical context), and structured by a plan that is executed by the agent (social context). Primarily, this module defines a taxonomy of actions in the descriptive branch. It further includes concepts and relations used to represent the structure of an action, and the situational context of its execution.

The purpose of the PROC module is the contextualization of processes. Processes are interpreted as events that are considered in their evolution. This is, e.g., the case for motions. A process taxonomy is defined in this module, as well as concepts that describe their evolution, and situational context.

States are the third type of events considered in our modeling.
For example, objects being in contact with each other is interpreted as a state.
The difference between a state and a process is that, when considering time slices  of the event, for states, these time slices always have the same type as the state (states are homeomeric), but for processes this is not the case.
The STATE module defines a state taxonomy, a general concept used to conceptualize states, and concepts used to represent situational context.

The EXEC module covers representations of execution contexts. It defines concepts related to parameterizing and connecting tasks. It provides ways to describe how outcomes of some tasks relate to inputs of others, and therefore characterize how tasks depend on, or enable one another.

§ OBJECT REPRESENTATION IN SOMA

One of the reasons that everyday activity is such a hard problem is the immense amount of variations an unrestricted environment may have, and the resulting potentials of interaction for an agent. Each type of object needs to be handled differently depending on its properties.
However, object manipulation tasks are often defined independent of the type of object that is manipulated. It is thus crucial for an agent that copes with these variations to have an abstract object model, and a mechanism for applying abstract object knowledge to novel situations.

The main link between physical objects and activities in SOMA is that objects participate in events in the physical branch, and that the social branch represents the interpretation of their participation. This is elaborated in Section <ref>.
Next, in Section <ref>, we will discuss how objects are organized along their design, and how design knowledge is used to cope with general tasks.
However, the agent might further need to find suitable candidate objects to perform a task by reasoning about which objects have the potential to be used in a certain way.
We employ an object disposition model for that purpose which is discussed in Section <ref>.

§.§ Object Types

For the classification of objects we employ the  pattern provided by the foundational layer. Roles are  and, as such, reside in the  branch of DUL. For human agents the ascription of roles to entities comes very natural. He is a student does not imply an isa or instanceof relation between some male individual and a student class. It is rather meant that at this point of his life the individual plays the role of a student, which, however, can and will change over time. Therefore, a classifies relation is used to express that someone can be classified as a student or something can be classified as a container.

This role pattern is of paramount importance, especially in the modeling of affordance discussed below in Section <ref>. In the model presented herein, we import the roles that have been established in the field of frame semantics [17]. The selectional restrictions imposed by the classifies relation are used in a number of reasoning processes ranging from natural language understanding to tool selection. As certain roles can only classify physical agents or specific types of designed artifacts these axiomatizations provide substantial information about context dependent meaning of objects.

§.§ Object Designs

The organization of objects along a taxonomy is difficult as objects can be categorized in many ways. A notion of design is useful to capture object categories corresponding to structural, functional, or aesthetic patterns.

Designs are in particular useful to conceptualize refunctionalized entities, and to support an agent to hypothesize unknown functions served by an entity. For example, a wooden pallet can be reused for the construction of furniture such as sofa, bed etc. The categorization of objects along their design can be employed in order to allow the use of more general plans, where, instead of object types, the plan refers to structure, aesthetics, or function.

Within the scope of SOMA, the  concept belongs to the social branch. SOMA considers structural, functional and aesthetic aspects of design. A design describes classes of objects that host a common design-relevant quality. These qualities are dispositional, geometrical, and aesthetic aspects of the object. This corresponds to our design categorization into functional, structural, and aesthetic design. Each  concept defines restrictions on the corresponding quality type that needs to be fulfilled by any object described by the design. These restrictions also represent sufficient conditions under which an object is thought to be described by the design which allows the classification of entities given their design pattern can be detected.

In the scope of this work, we only consider functional aspects of objects. These are represented using a model of dispositional qualities which is discussed next.

§.§ Object Dispositions

Objects are important to an agent because they allow it to perform, or prevent it from performing, actions to achieve its goals. The notion of “affordance” was put forth by Gibson to cover this concept [18]:

“The affordances of the environment are what it offers the animal, what it provides or furnishes, either for good or ill.”[18]

However, though evidently useful as a way to organize actionable knowledge about the world [19], affordances proved very difficult to model ontologically. Several approaches have been proposed, such as regarding affordances as qualities [20] or as events [21].

Nonetheless, we think these approaches are not satisfactory. Affordances are relational, characterizing a potential interaction of several objects, and therefore should not be treated as either a quality belonging to an object, nor as an event. We do recognize that some qualitative aspects of objects contribute to affordances, however, which is why we constructed our model around the interplay of Turvey's notion of disposition [22] and Gibson's notion of affordance.

In SOMA, the Disposition concept is defined as an object quality that allows an object to
participate in events that realize an affordance.

The Affordance concept itself, however, is defined as the relational context holding between several objects that play different roles such as being the “bearer”, “trigger”, or “background” of an affordance.

Our modeling allows us, via a mixture of DL and other reasoning mechanisms such as simulation, to answer several interesting questions such as what affordances might an object provide in some combination with others, what objects might, or probably would not, be able to provide a given affordance, what combinations of objects would work towards providing an affordance etc.

For more details on our disposition and affordance model, we invite the reader to consult our previous work on this topic [23].

§ EVENT REPRESENTATION IN SOMA

The information gap between an instruction given to an embodied agent and the way it has to move its body to successfully execute the instruction is often immense. Consider, for example, a recipe for cooking noodles that contains an instruction to boil water in a pot. It is simple to decompose this instruction into several steps with individual sub-goals such as finding pot and tap, placing the pot underneath the tap, and filling the pot with water. However, the more difficult problem is how the agent has to move its body in each step such that the goal is achieved, and unwanted side-effects are avoided.
This is, for example, to avoid spillage of the water when the valve is opened too much, or when the agent moves too fast with the water-filled pot.

Little variations in motion behavior may have drastic consequences in tasks that require delicate interaction. It is thus essential for agents performing actions in the physical world to reason about how they should move to achieve their goals in an appropriate, flexible and robust manner which is an unsolved problem for the general case.

SOMA attempts to support an agent facing this problem by equipping it with knowledge about relationships between abstract descriptions and their realization.
The support is twofold. First, the agent may employ more general plans where informational gaps are filled by reasoning over knowledge represented with SOMA. Second, the agent may employ SOMA for understanding and generalizing observations. This means that agents can interact safer in environments with incomplete information, and that they can learn general patterns from specific situations.
An illustrative example of the representation of a pouring plan in OWL Manchester Syntax is provided in Figure <ref>.

[notitle, colback=descriptive!20!white, colframe=descriptive, boxrule=0.2mm, arc=0.0mm, boxsep=0mm, top=1mm, bottom=0mm]
Individual: PouringPlan_0
Types: Plan, Description
Facts: defines Pouring_0,
hasPhase Approaching_0,
hasPhase Tilting_0
Individual: Pouring_0
Types: Task, Concept
Facts: usesRole Patient_0,
usesRole Source_0,
usesRole Destination_0,
startedBy Approaching_0

Individual: Approaching_0
Types: Motion Type, Concept
Facts: usesRole Destination_1,
overlapsWith Tilting_0
Individual: Tilting_0
Types: Motion Type, Concept
Facts: usesRole Patient_1
Individual: Binding_1
Types: RoleBinding, Description
Facts: hasBinding Source_0,
hasBinding Patient_1

An example of how plans are represented in SOMA.

In the following, we will describe the representation of activities in SOMA in more detail. We will first introduce our hierarchical organization of tasks, processes and states in Section <ref>, and, second, how they are decomposed into phases with explicit goals and individual knowledge pre-conditions in Section <ref>. Finally, we will discuss our modeling of force characteristics

in Section <ref>.

§.§ Event Types

One of the most important demands on a cognitive system is to reason about actions; colloquially speaking, an agent constantly asks itself what to do, and how to do it. This however opens up another question, namely what exactly is the entity that the agent represents – an actual event, or an interpretation of one.

As an example, consider this scenario: a robot moves toward a table carrying a plate. Midway, its gripper releases, dropping the plate, which shatters against the floor. Perhaps the robot had to transport the plate to the table, and it failed to do so; or perhaps it was required to drop the plate as part of some material test, and the table was just there for some other reason. Just by observation of the action,

other interpretive context which includes knowledge of what the robot was told to do, there is no reliable way to tell. The failed transport interpretation does seem more likely a priori, but only because we have more often seen people tell robots to transport plates rather than break them; we still make use of an expected interpretive context.

As a result, we do not define a taxonomy of action events in our ontology, but rather of tasks that are used to conceptualize actions.
For example, the Grasping concept is defined as task in SOMA, and it is used for the classification of events that are interpreted as an intentional grasping activity.
This classification pattern between events and their conceptualization is provided by the foundational layer of SOMA.

However, within the foundational layer, this pattern is only instantiated for actions and their conceptualization. In our modeling, motions of an agent and other processes, as well as state events are used to structure an activity. Thus, we also need to represent processes and states in the ground and the descriptive ontology.
The same pattern applies: the concepts Process and State are defined in the ground ontology, and their conceptualization in the descriptive ontology, and a relationship between both branches is established through the aforementioned classification pattern.

§.§ Event Phases

Actions in SOMA are composed of distinct phases. Each phase has its individual goal, and requires a different movement strategy to be executed successfully. The phases correspond to different stages of an object manipulation task, usually separated through contact events.
Flanagan et al. have pointed out the importance of contact events in object manipulation tasks [24]. The authors have shown that contact events cause a distinct pattern in sensory events, and that they can be used as sensorimotor control points for aligning and comparing predictions with actual sensory events.
Another justification is that humans have shown to direct their gaze to contact points when they perform object manipulation tasks, or when they observe another agent performing a task.

The structure of activities in SOMA is governed by a set of design patterns applied to different levels of granularity. At its core, SOMA activity modeling builds on top of the basic plan ontology design pattern that represents plans and their execution.
The pattern further defines that an execution is a situation that satisfies the description of the plan.
However, the pattern is defined too specific for the scope of this work, as we also want to provide descriptive context for states and processes.
Hence, we generalize this pattern such that it can be instantiated in different levels of granularity in the activity model. This generalized pattern is displayed in Figure <ref>. Its purpose is to link the grounded with the descriptive branch.

In SOMA, this pattern is instantiated for actions, states and processes:

* An Action is described in a Plan which is a description having an explicit goal. A plan satisfies situations that include action sequences that match the structure of the plan, such situations are called Plan Executions;

* a State is described in a Configuration which includes constraints on regions of entities and relationships between them. A configuration satisfies situations in which all constraints of the configuration are satisfied; and

* a Process is described in a Process Flow which is a description of the progression of the process. A process flow satisfies situations that include a process that progresses in the described way.

[owlclass,physical] (SIT)

* $(\exists \texttt{satisfies}.\texttt{Description})$

[owlclass,social,below=0.6cm of SIT] (DESCR)

* $(\forall \texttt{defines}.\texttt{EventType})$

[owlclass,social,right=3.4cm of DESCR.north east,anchor=north] (CON)

* $(\forall \texttt{classifies}.\texttt{Event})$

[owlclass,physical,right=3.4cm of SIT.north east,anchor=north] (EV)

* $(\forall \texttt{hasSetting}.\texttt{Situation})$

(SIT) edge[myedge] (DESCR);
(DESCR) edge[myedge] (CON);
(CON) edge[myedge] (EV);
(EV) edge[myedge] (SIT);

The "Interpretation" pattern used to link the physical branch of SOMA (top) with social context (bottom).

SOMA represents the structure of an activity using a sequence pattern based on Allen's interval calculus [25]. Allen's calculus defines thirteen relations between time intervals including before, after, overlaps, and meets. This is useful to, on the one hand, represent precedence of one phase strictly following the other, and, on the other hand, it allows to cope with concurrency in the sequence. We apply this algebra to event types that are defined within the descriptive context of a plan or process flow. However, reasoning about sequences is not well supported in OWL. Instead, we employ a specialized reasoner that internally builds a graph that encodes sequence relations, and which can be queried to infer whether some sequence relation exists between two entities or not.

Another aspect of activity structure can be captured by SOMA in, what we call, execution contexts. These are representations of how different phases of an activity constrain each other depending on conditions encountered in the activity execution. In particular, we define the  concept as identity constrain representing that a parameter or role grounding is the same in different phases, however potentially being classified differently.

Knowledge about the structure of activities can be employed by an agent in both directions: for planning an activity, and for interpreting observed events. Planning can be seen as a mapping from the descriptive to the grounded branch of SOMA, while interpretation maps the other way.
For embodied agents, planning goes beyond mere decomposition of an activity into steps, the agent may further need to decided what objects it should use, how it should move, with what speed, and how much force it should apply when getting into contact with some object.
SOMA may be employed by the agent to find potential sequences of steps and motions to execute a task, to support finding potential objects playing some role during the activity, and to constrain the values of parameters of a task.
Interpretation of events, on the other hand, is difficult because the intentions of other agents are not known. However, it is often possible to detect contact events, types of motions, and states reliably. These can  be used as tokens for an activity parser that uses SOMA as a grammar, this will be described in more detail in Section <ref>.

Knowledge Pre-Conditions.
In order to execute a motion, an agent has to invoke one of its control routines with a set of arguments. Higher-level routines may have a notion of object, but at a lower-level all boils down to numbers such as with what effort the robot moves, how fast, etc. SOMA allows to define constraints for both cases: for the types of objects that can play a role during the action, and for the value of parameters. This is done by using restrictions on what types of objects or regions can be classified by some role or parameter. This information is used to reduce the search space for doing an appropriate object or parameter selection (Section <ref>).

A goal is a description of a desired situation, and it is achieved only if the situational context, after the execution has been finished, satisfies this description. One would say then that the action was executed successfully.
SOMA is more specific about what it means to execute an action successfully as it decomposes it into processes and states where the goal of the task is that the progression of processes evolves, and that state changes occur as described.
Particularly important are the contact states in object manipulation tasks, as they represent control points for the agent when generating or observing behavior.

§.§ Event Force Characteristics

A contact state is an indicator for whether objects are touching each other or not. Patterns of such states are useful for distinguishing between categories of activities. However, different activities may cause the same pattern while their goal is different, or even the opposite of each other. This is, for example, the case for pulling and holding. Both cause the same pattern of an endeffector getting into contact with another object. But the force characteristics are different: the goal of a pulling task is to overcome the inertial force of the object to set it into motion, and the goal of a holding task is to neutralize any external force that would set the object into motion.

Another aspect is that an agent performing such a task needs to decide how much force to apply. In order to make this decision it is valuable to know what the intended force-related consequences are.

SOMA supports the representation of force characteristics using Talmy's notion of force dynamics [26]. Talmy distinguishes between two entities that participate in force dynamical processes: the Agonist, and the Antagonist. An agonist is the subject of a force dynamical expression, while the antagonist is the opposing force in the expression. Each expression has an intrinsic force tendency either to set the agonist into motion, or to keep it resting. Whether the tendency can be realized or not depends on which of the two entities is the stronger entity.

§ APPLICATIONS OF SOMA

The applicability of SOMA in the domain of everyday activities in autonomous robot control has been demonstrated in several scientific publications. We will summarize some of them below in order to provide evidence that SOMA contributes to coping with the complexities of everyday activities [please note that SOMA is referred to as "EASE ontology" or "NEEM narrative" in the listed works, as we only recently have named our ontology "SOMA".].
To conclude this section, we will elaborate on best practices in ontology engineering that we have established for collaborative development of SOMA.

mynode/.style=outer,fill=pastel1,draw=pastel3,text badly centered,text width=2.0cm,inner sep=3pt,minimum height=0.5cm,
mylarge/.style=mynode,text width=5.0cm,
mytitle/.style=font=,draw=pastel5,fill=pastel1,inner sep=0.04cm,

[mynode,physical] (EV) objects;
[mynode,social,right=0.8cm of EV] (PL) plans;
[inner sep=0,yshift=1.8cm] at ($(EV)!0.5!(PL)$) (EXEC)

[mytitle,physical] at (EXEC.north) Execution;
[mylarge,yshift=-1.0cm] at ($(EV)!0.5!(PL)$) (DIS) dispositional matching;

[] (EXEC.south -| EV.north) edge[myedge] node[left,mylabel,xshift=-0.1cm]perception
[] (EXEC.south -| PL.north) edge[myedge] node[right,mylabel,xshift=0.1cm,yshift=0.02cm]context
[] (EV.south) edge[myedge] node[left,mylabel,xshift=-0.1cm]facts
(EV.south |- DIS.north);
[] (PL.south) edge[myedge] node[right,mylabel,xshift=0.1cm]facts
(PL.south |- DIS.north);
[] (DIS) edge[myedge] node[right,mylabel,yshift=-0.5cm]selection

[mynode,physical] (EV) events;
[mynode,social,right=0.8cm of EV] (PL) plans;
[inner sep=0,yshift=1.8cm] at ($(EV)!0.5!(PL)$) (VR)

[mytitle,physical] at (VR.north) Observation;
[mylarge,yshift=-1.0cm] at ($(EV)!0.5!(PL)$) (MAP) activity parsing;

[] (VR.south -| EV.north) edge[myedge] node[left,mylabel,xshift=-0.1cm]tokens
[] (EV.south) edge[myedge] node[left,mylabel,xshift=-0.1cm]facts
(EV.south |- MAP.north);
[] (PL.south) edge[myedge] node[right,mylabel,xshift=0.1cm]facts
(PL.south |- MAP.north);
[pastel5,thick,dashed,latex-latex] (EV) – (PL);

SOMA is used for object selection (left); and for activity interpretation (right).

Each manipulation task involves a set of objects that are classified by certain roles during the task. The roles of a task are conceptualized by constraints on the class of objects that may play the role during task execution.
We have shown examples on how dispositional qualities in SOMA can be axiomatized that allow to define general plans that refer to dispositional classes of objects [23].

This dispositional object selection is depicted in the left part of Figure <ref>.

Grounding task parameters often requires predictive models which can be trained over instances of successful performance.
Such experiential knowledge is in particular useful to learn context-dependent plan specializations. That is, how the parameters of the plan can be constrained within the scope of some context to reduce the search space of parameter selection during plan execution. The learning problem is then defined with respect to a contextual pattern, and experiential samples are only considered when their contextualization matches this pattern.
We have demonstrated this capability in another work where a robot learns to execute a general fetch and place plan based on experience acquired through the execution of more constrained tasks [27].

Learning mechanisms often require large amounts of training data. This raises the question how huge amounts of experiential knowledge can be acquired. One modality for acquisition is observation of other agents where the intention of the agent is unknown.
In order to acquire experiential knowledge from observations, we have developed an activity parser that is used to find possible interpretations for observed patterns of occurrences such as that objects get into contact with each other, or that the state of an object changed [28].
The grammar used by the parser is a library of plans represented using SOMA.
This is depicted in the right part of Figure <ref>.
Each plan describes the structure of a task by axiomatizing occurrences of steps using Allen's interval algebra such that one formal plan can match many sequences of observed events.

In another work, we have provided more details about how the social context in SOMA can be grounded in data structures of a game engine [29].
The game engine implements an immersive virtual reality environment with photo realistic rendering and state of the art physics engine. Users perform object manipulation tasks while interactions, states, and motions are monitored, and used as tokens by the activity parser.
The goal is to generate large amounts of experiential knowledge that implicitly encodes common sense and naive physics knowledge, and to learn from these episodes.

Our modeling of tasks

also helps to disambiguate vague natural language commands a human user might give to a robot. SOMA defines concepts such as  and  which allows us to model how tasks relate to and depend on one another, and thus define execution contexts containing not just information about a task's parametrization, but also information about what other tasks it should enable. We use such execution contexts to set up simulation scenarios in which to test task executions and thus select among several interpretations of a vague natural language command [30].

Currently, SOMA contains more than 3000 asserted facts across all previously mentioned modules.
To make downstream work easier, we combine all modules into one overarching ontology. However, an ontology of SOMA's size is difficult to maintain: It is always possible that changes render the ontology inconsistent.

To prevent these kinds of errors, we employ strict workflows and adhere to best practices in software engineering. We use the version control system git to make it easy to share small patches of the ontology among our team and apply the patches after at least one member of our team reviewed the changes. Even the best review might not catch subtle inconsistencies. To catch such errors, we automatically deploy the ontology with each change set to a continuous integration server that tests for inconsistencies by using an OWL reasoner.

§ CONCLUSION

In this paper, we have proposed SOMA, a novel activity ontology for robotic agents that combines several established ontology design patterns with models of human cognition to cope with the challenges of everyday activities in a general manner. The SOMA ontology has been validated through a set of use-cases that demonstrate its applicability for reasoning about tasks in the scope of autonomous robot control. This is done by representing both the physical and the social context of an activity, and by establishing relationships between both contextualizations. These representations are used by robotic agents to fill knowledge gaps in general plans applicable to many situations, and to generate context-specific behavior. Another aspect is that agents may as well use SOMA for the representation of observed actions performed by another agent, and for reasoning about their social contextualization.
We believe that such an expressive activity representation is an important vehicle for transforming robots from just performing a task to mastering the corresponding activity.

[1]
Lenat, D.B., Feigenbaum, E.A.:
On the thresholds of knowledge.
Artificial Intelligence 47(1) (1991)  185 – 250

[2]
Anderson, J.E.:
Constraint-Directed Improvisation for Everyday Activities.
PhD thesis, The University of Manitoba (Canada) (1995) AAINN99082.

[3]
Olivares-Alarcos, A., Beßler, D., Khamis, A., Gonçalves, P., Habib, M.,
Bermejo, J., Barreto, M., Diab, M., Rosell, J., Quintas, J., Olszewska, J.,
Nakawala, H., Pignaton de Freitas, E., Gyrard, A., Borgo, S., Alenyà, G.,
Beetz, M., Li, H.:
A review and comparison of ontology-based approaches to robot
The Knowledge Engineering Review 34 (12 2019)

[4]
Patel, R., Hedelind, M., Lozan-Villegas, P.:
Enabling robots in small-part assembly lines: The "rosetta approach"
- an industrial perspective.
In: ROBOTIK, VDE-Verlag (2012)

[5]
Malec, J., Nilsson, K., Bruyninckx, H.:
Describing assembly tasks in declarative way.
In: IEEE/ICRA Workshop on Semantics. (2013)

[6]
Stenmark, M., Malec, J., Nilsson, K., Robertsson, A.:
On distributed knowledge bases for robotized small-batch assembly.
IEEE Transactions on Automation Science and Engineering
12(2) (2015)  519–528

[7]
Balakirsky, S., Kootbally, Z., Kramer, T., Pietromartire, A., Schlenoff, C.,
Gupta, S.:
Knowledge driven robotics for kitting applications.
Robot. Auton. Syst. 61(11) (November 2013)  1205–1214

[8]
Balakirsky, S.:
Ontology based action planning and verification for agile
Robotics and Computer-Integrated Manufacturing 33(Supplement
C) (2015)  21 – 28 Special Issue on Knowledge Driven Robotics and

[9]
Polydoros, A.S., Großmann, B., Rovida, F., Nalpantidis, L., Krüger,
Accurate and versatile automation of industrial kitting operations
with skiros.
In: Towards Autonomous Robotic Systems - 17th Annual Conference
(TAROS). (2016)  255–268

[10]
Kootbally, Z., Schlenoff, C., Lawler, C., Kramer, T., Gupta, S.:
Towards robust assembly with knowledge representation for the
planning domain definition language (pddl).
Robot. Comput.-Integr. Manuf. 33(C) (June 2015)  42–55

[11]
Perzylo, A., Somani, N., Profanter, S., Kessler, I., Rickert, M., Knoll, A.:
Intuitive instruction of industrial robots: Semantic process
descriptions for small lot production.
In: IEEE/RSJ International Conference on Intelligent Robots and
Systems (IROS). (2016)  2293–2300

[12]
Tenorth, M., Beetz, M.:
Representations for robot knowledge in the knowrob framework.
Artificial Intelligence (2015)

[13]
Schlenoff, C., Prestes, E., Madhavan, R., Goncalves, P., Li, H., Balakirsky,
S., Kramer, T., Miguelanez, E.:
An IEEE standard ontology for robotics and automation.
In: IEEE Int. Conf. on Intelligent Robots and Systems (IROS).
(2012)  1337–1342

[14]
Prestes, E., Carbonera, J.L., Fiorini, S.R., Jorge, V.A.M., Abel, M., Madhavan,
R., Locoro, A., Goncalves, P., Barreto, M.E., Habib, M., Chibani, A.,
Gérard, S., Amirat, Y., Schlenoff, C.:
Towards a core ontology for robotics and automation.
Robotics and Autonomous Systems 61(11) (2013)  1193 – 1204
Ubiquitous Robotics.

[15]
Fiorini, S.R., Carbonera, J.L., Gonçalves, P., Jorge, V.A., Rey, V.F.,
Haidegger, T., Abel, M., Redfield, S.A., Balakirsky, S., Ragavan, V., Li, H.,
Schlenoff, C., Prestes, E.:
Extensions to the core ontology for robotics and automation.
Robot. Comput.-Integr. Manuf. 33(C) (June 2015)  3–11

[16]
Masolo, C., Borgo, S., Gangemi, A., Guarino, N., Oltramari, A.:
WonderWeb deliverable D18 ontology library (final).
Technical report, IST Project 2001-33052 WonderWeb: Ontology
Infrastructure for the Semantic Web (2003)

[17]
Baker, C.F., Fillmore, C.J., Lowe, J.B.:
The berkeley framenet project.
In: Proceedings of the 36th Annual Meeting of the Association for
Computational Linguistics and 17th International Conference on Computational
Linguistics - Volume 1. ACL ’98/COLING ’98, USA, Association for
Computational Linguistics (1998)  86–90

[18]
Gibson, J.J.:
The Ecological Approach to Visual Perception.
Psychology Press Classic Editions (1979)

[19]
Yamanobe, N., Wan, W., Ramirez-Alpizar, I.G., Petit, D., Tsuji, T., Akizuki,
S., Hashimoto, M., Nagata, K., Harada, K.:
A brief review of affordance in robotic manipulation research.
Journal of the Robotics Society of Japan 36 (01 2018)

[20]
Ortmann, J., Kuhn, W.:
Affordances as qualities.
In: Proceedings of the 2010 Conference on Formal Ontology in
Information Systems: Proceedings of the Sixth International Conference (FOIS
2010), Amsterdam, The Netherlands, The Netherlands, IOS Press (2010)

[21]
Moralez, L.A.:
Affordance ontology: towards a unified description of affordances as
Res. Cogitans 7(1) (2016)  35–45

[22]
Turvey, M.T.:
Ecological foundations of cognition: Invariants of perception and
American Psychological Association (1992)

[23]
Beßler, D., Porzel, R., Pomarlan, M., Beetz, M., Malaka, R., Bateman, J.:
A Formal Model of Affordances for Flexible Robotic Task Execution.
In: European Conference on Artificial Intelligence (ECAI). (2020)
accepted for publication.

[24]
Flanagan, J.R., Bowman, M.C., Johansson, R.S.:
Control strategies in object manipulation tasks.
Current Opinion in Neurobiology (2006)

[25]
Allen, J.F.:
Maintaining knowledge about temporal intervals.
In: In Communications of the ACM, Volume 26 Issue 11. (1983)

[26]
Talmy, L.:
Toward a Cognitive Semantics. Volume 2: Typology and Process in
Concept Structuring.
Language, Speech, and Communication. MIT Press, Cambridge, MA (2000)

[27]
Koralewski, S., Kazhoyan, G., Beetz, M.:
Self-specialization of general robot plans based on experience.
Robotics and Automation Letters (2019)

[28]
Beßler, D., Porzel, R., Mihai, P., Beetz, M.:
Foundational models for manipulation activity parsing.
In Jung, T., tom Dieck, M.C., Rauschnabel, P.A., eds.: Augmented
Reality and Virtual Reality.
Springer (2019) 978-3-030-37869-1.

[29]
Haidu, A., Beßler, D., Bozcuoglu, A.K., Beetz, M.:
Knowrob-sim — game engine-enabled knowledge processing for
cognition-enabled robot control.
In: International Conference on Intelligent Robots and Systems
(IROS), Madrid, Spain, IEEE (2018)

[30]
Bateman, J., Pomarlan, M., Kazhoyan, G.:
Embodied contextualization: Towards a multistratal ontological
Applied Ontology 14(4) (2019)  379–413