# Locked and Unlocked Smooth Embeddings of Surfaces

David Eppstein Department of Computer Science, University of California,
Irvine. This work was inspired by discussions at the 3rd Virtual Workshop on
Computational Geometry, held in March 2022, for which we thank the organizers
and participants. Research supported in part by NSF grant CCF-2212129.

###### Abstract

We study the continuous motion of smooth isometric embeddings of a planar
surface in three-dimensional Euclidean space, and two related discrete
analogues of these embeddings, polygonal embeddings and flat foldings without
interior vertices, under continuous changes of the embedding or folding. We
show that every star-shaped or spiral-shaped domain is unlocked: a continuous
motion unfolds it to a flat embedding. However, disks with two holes can have
locked embeddings that are topologically equivalent to a flat embedding but
cannot reach a flat embedding by continuous motion.

##  1 Introduction

Much research in computational geometry has concerned smooth deformations of a
shape that preserve its geometric structure. This includes bloomings,
continuous and collision-free unfoldings from polyhedra to flat nets that
preserve the shape of each face [21, 6, 18, 9, 26, 15], the carpenter’s rule
problem on continuous collision-free motions that straighten a polygonal chain
while preserving segment lengths [22, 8, 20], and the closely related study of
continuous rigid motions of single-vertex origami patterns [23, 19, 1]. When
the carpenter’s rule problem is generalized to to more complex linkages or
three dimensions it can have _locked_ configurations, unable to reach a
straightened configuration even though there is no topological obstacle to
their reconfiguration [16, 4, 5, 11, 3, 7]. Demaine, Devadoss, Mitchell, and
O’Rourke studied “folded states” of simple planar polygons in 3d, consisting
of a surface-distance-preserving mapping to 3D together with a consistent
“local stacking order” at parts of the polygon that are mapped onto each
other. As they show, any folded state can be continuously transformed to any
other folded state: the configuration space of these states is connected [12,
10].

In this work we examine reconfigurability for three natural restricted forms
of folded states:

  * •

Smooth embeddings into $\mathbb{R}^{3}$, where the embedded surface is doubly
differentiable (having a tangent plane everywhere) without self-contact.

  * •

Polygonal (piecewise linear) embeddings into $\mathbb{R}^{3}$ without interior
vertices, so all “fold lines” where the mapping is not locally linear extend
entirely across the surface. There should be finitely many connected linear
pieces and no self-contact.

  * •

Planar folded states (flat foldings) without interior vertices. We again
require that the mapping be piecewise linear with finitely many pieces and
that the fold lines extend entirely across the surface.

At each interior point of a smoothly embedded flat surface in $\mathbb{R}^{3}$
that is not locally flat, the surface bends along a straight “bend line” that
continues to the surface’s boundary [14]. As an everyday example of this
phenomenon, when holding a slice of pizza at its crust, keeping the crust flat
allows the slice to droop, but bending it extends rigid bend lines lengthwise
through the slice, preventing drooping [25]. Our restriction against interior
vertices of polygonal embeddings and flat foldings provides a combinatorial
model of the same phenomenon. We have studied flat foldings with this
restriction (but not their reconfiguration) in previous work [13].

In all three models of bending and folding we allow continuous motions that
stay in the same model; in particular, in the polygonal embedding model, folds
may “roll” along the surface rather than remaining fixed in place. Our folded
states are special cases of the ones previously considered by Demaine et al.
[12, 10], and we retain their notion of a continuous motion as a mapping from
the closed unit interval $[0,1]$ to folded states that is continuous under the
sup-norm of the distances of mapped points and (for flat foldings) consistent
with respect to the local stacking order. The initial configuration of a
motion is the mapping for the parameter value $0$, and the final configuration
is the mapping for the parameter value $1$. For all three of our restricted
models of folded states, we prove the following new results:

  * •

A compact subset of the plane with a continuous shrinking motion has a
connected space of restricted folded states: every folded state can be
continuously unfolded to a flat state. These sets are topological disks and
include the star-shaped domains.

  * •

There exist subsets of the plane, topologically equivalent to a disk with two
holes, that can be locked: they have embeddings that are topologically
equivalent (ambient isotopic) to their flat embedding, but cannot be
continuously deformed to become flat.

##  2 Shapes that can shrink into themselves

Figure 1: A polygon that has a continuous shrinking motion into itself, but
that is not star-shaped.

A _star-shaped_ polygon, or more generally a star-shaped domain, is a subset
$S$ of the plane such that, with an appropriate choice of one point of the
plane to be the origin, every scaled copy $pS$ for $p\in[0,1]$ is a subset of
$S$ itself. These are widely studied in computational geometry, and can be
recognized in linear time [17]. However, these are not the only shapes that
have a continuous shrinking motion of scaled copies of the shape into
themselves. Fig. 1 depicts a different type of continuous shrinking motion, in
which the shape spirals inwards while shrinking. Such a motion can be
described by coordinatizing the plane by complex numbers, again for an
appropriately chosen origin (the center of the spiral motion), choosing a
complex number $q$ of absolute value less than one, and considering the family
of scaled copies $e^{p}qS$ for $p\in[0,\infty)$. The linear shrinking motion
of star-shaped domains is a special case of this type of motion in which $q$
is a positive real number. If any shape $S$ has any continuous shrinking
motion of its scaled copies into itself, the start of the motion can be
approximated to first order by an inward-spiraling shrinking motion of this
type, which can then be continued to a complete inward-spiraling shrinking
motion of the same shape. In this sense, this family of continuous shrinking
motions is completely general, in the sense that all shapes with a continuous
shrinking motion have a continuous shrinking motion of this type, although we
will not use this fact. Following Aharonov et al. [2], we call a compact set
$S$ that has a continuous shrinking motion of this type a _spiral-shaped
domain_.

In an inward-spiraling shrinking motion, each point of the set $S$ traces out
a logarithmic spiral, which meets every ray from the center of the motion in a
fixed angle $\theta$. The existence of a spiraling motion for a given simple
polygon and a fixed choice of $\theta$ can be tested by intersecting polygonal
regions, one for each edge, that describe the set of locations for the center
where a spiral of this angle would leave the edge towards the interior of the
polygon, rather than towards the exterior. This characterization leads to a
polynomial-time algorithm for testing the existence of an inward-spiraling
motion, by continuously varying $\theta$ over the range $(-\pi,\pi)$ and
determining the combinatorial changes in the corresponding intersection of
polygonal regions. It is plausible that finding a feasible angle $\theta$ and
a center point for that choice of $\theta$ is an LP-type problem of bounded
dimension, solvable in linear time, but we have not proved this.

The main results of this section are that every spiral-shaped domain $S$ has
connected spaces of smooth embeddings, polygonal embeddings without interior
vertices, and flat-foldings without interior vertices. Equivalently, every
embedding can be continuously unfolded. The simplest case concerns smooth
embeddings.

######  Theorem 2.1.

Every smooth embedding of a spiral-shaped domain has a continuous motion,
through smooth embeddings, to a flat embedding.

######  Proof 2.2.

Let $S$ be the given spiral-shaped domain, and $f:S\to\mathbb{R}^{3}$ be its
smooth embedding. Parameterize an inward-spiraling shrinking motion of $S$ as
$s_{i}:S\to S$ where $i\in(0,1]$, $s_{1}$ is the identity, and each $s_{i}$
scales $S$ by a factor of $i$, converging as $i\to 0$ to a single central
point of the motion (which may or may not be on the boundary of $S$).

Our proof converts this parameterized family of scalings to a parameterized
family of smooth embeddings of $S$ at a single scale, by composing $s_{i}$,
$f$, and a function that expands $\mathbb{R}^{3}$ by a factor of $1/i$ to
restore the original size of $S$. The obvious expansion function
$\mathbb{R}^{3}$ by $(x,y,z)\mapsto(x/i,y/i,z/i)$ does not work, because of
the following issues:

  * •

Composing $s_{i}$, $f$, and an expansion by $1/i$ does provide a continuous
motion of smooth embeddings on the half-open interval $(0,1]$, whose curvature
tends towards zero as the parameter goes to zero. However, we need a
continuous motion on the closed interval $[0,1]$ for which the limiting
embedding at parameter value zero exists and is completely flat.

  * •

When the origin of $\mathbb{R}^{3}$ does not belong to all of the rescaled and
smoothly embedded copies of $S$, the composition with $\mathbb{R}^{3}$ by
$(x,y,z)\mapsto(x/i,y/i,z/i)$, as $i\to 0$, will produce smooth embeddings of
$S$ whose distance from the origin is inversely proportional to $i$,
preventing them from having a limit. We can prevent this by choosing
coordinates for $\mathbb{R}^{3}$ that have as their origin $f(p)$, where $p$
is the limit point of the inward-spiraling shrinking motion on $S$. In this
way, the composition of $s_{i}$, $f$, and an expansion by $1/i$ will act as
the identity on this point, and more strongly will preserve the tangent plane
of the surface at that point.

Figure 2: A spiral-shaped domain between two logarithmic spirals, whose
shrinking motion cannot be shortcut to a linear shrinking motion.

  * •

This still does not complete the proof, because a spiral inward-shrinking
motion of the domain, composed with $f$ and $(x,y,z)\mapsto(x/i,y/i,z/i)$,
will cause the embeddings to rotate at increasing speed as $i\to 0$,
preventing the continuous motion from having a flat limiting surface at $i=0$.
For the polygonal domain of Fig. 1, this problem can be circumvented by
switching from the spiral inward-shrinking motion to a linear scaling
transformation in $S$, once the scaled copies of $S$ become small enough that
this linear scaling stays entirely within $S$. However, for some other shapes,
such as the domain between two logarithmic spirals depicted in Fig. 2,
switching to linear scaling is never possible.

Instead, we address this third issue by choosing a reference vector tangent to
$S$ at $f(p)$. We compose $s_{i}$, $f$, an expansion by $1/i$, and a rotation
of $\mathbb{R}^{3}$ (with axis perpendicular to the tangent plane at $f(p)$)
that restores this vector to its original direction.

Let $f_{i}$ denote the resulting composition of $s_{i}$, $f$, an expansion of
$\mathbb{R}^{3}$ with a careful choice of origin, and a rotation that restores
the original directions of vectors in the tangent plane to the embedded
surface. Then $f_{i}$, for values of $i$ in the half-open interval $(0,1]$,
describes a continuous motion with $f_{1}=f$ as one endpoint of the motion.
The maximum curvature of the surface $f_{i}(S)$ equals $i$ times the maximum
curvature within the $i$-scaled copy of $S$ within $f(S)$, which in the limit
becomes arbitrarily close to $i$ times the curvature at $p$ in $f(S)$ and
therefore has limiting value zero. The transformations $f_{i}$ preserve the
tangent plane to the surface and directions within the tangent plane.

For each point $q\in S$, $f_{i}(q)$ can be obtained by the exponential map:
follow a curve on $f_{i}(S)$ of length $|p-q|$, starting from $p$, in the
direction given by the image of the tangent vector $q-p$. This length and
direction are invariant through the motion, and as $i\to 0$ the curvature of
this path approaches zero. Therefore, $f_{i}$ converges pointwise to a flat
embedding $f_{0}(S)$, obtained by the exponential map on the tangent plane of
$f(S)$ at $p$. Appending $f_{0}$ to our continuous sequence of smooth
embeddings $f_{i}$ for $i\in(0,1]$ gives us a continuous sequence on
$i\in[0,1]$, flat at $i=0$ and equal to our starting embedding at $i=1$, which
therefore shows that these two embeddings are reconfigurable to each other.

This proof uses compactness to ensure that the limit point of the spiral
shrinking motion is asymptotically flat and that sufficiently small copies of
the entire domain fit into any neighborhood of that point. Non-compact
surfaces can have self-similar embeddings, smooth everywhere except the limit
point, that are invariant under shrinking and re-expansion. For polygonal
embeddings we handle the same issue of avoiding self-similar embeddings
differently, using the requirement that these embeddings have finitely many
connected linear pieces.

######  Theorem 2.3.

Every polygonal embedding without interior vertices of a bounded spiral-shaped
domain has a continuous motion, through polygonal embeddings without interior
vertices, to a flat embedding.

######  Proof 2.4.

The same idea as above comes close to working: compose the inward-spiraling
shrinking motion of the domain, the initial embedding $f$, an expansion of
$\mathbb{R}^{3}$ centered at the limit point of the motion, and a rotation of
$\mathbb{R}^{3}$ that cancels any spinning motion the inward-spiraling motion
might have. However, the limit point $p$ of the inward-spiraling shrinking
motion might be a point on a fold line of the polygonal embedding, or worse,
it might be a boundary point of $S$ where multiple fold lines meet. In this
case, there is not a unique tangent plane of $f(S)$ at $p$, and when the same
composition can be made to have a limit, this limit will be folded at $p$ in
the same way as it was in $f(p)$ rather than being flat.

To address these issues, when $p$ is a folding point of the embedding $f(S)$,
we choose one of the polygonal faces incident to $p$ to determine the tangent
plane for the previous construction. Then as above we compose the inward-
spiraling shrinking motion of the domain, the initial embedding $f$, an
expansion of $\mathbb{R}^{3}$ centered at the limit point of the motion, and a
rotation of $\mathbb{R}^{3}$ that cancels any spinning motion of this tangent
plane. The resulting composition defines a continuous motion over the half-
open interval of parameter values $(0,1]$, which can be extended with a well-
defined limit at $0$, a polygonal folded state that has the same fold lines
and fold angles as $f(S)$ at $p$ and is flat everywhere else.

We distinguish three cases:

  * •

If $p$ is not a folding point of the embedding $f(S)$, then the previous proof
applies directly.

  * •

If $p$ belongs to a single fold line of the embedding $f(S)$, then we can
concatenate two continuous motions. First we perform the motion from the
previous proof that transforms $f(S)$ into a folded state that contains this
fold line. Because this folded state has only one fold line, it cannot self-
intersect, and forms a valid polygonal embedding. Next, we perform an
additional continuous motion that linearly changes the angle of this fold from
its initial value to $\pi$ (unfolded into a flat angle). Again, none of the
intermediate states of this second continuous motion can self-intersect.

  * •

In the remaining case, $p$ is a boundary point of $S$ that belongs to multiple
fold lines within $S$. For a line from $p$ to intersect $S$ in a line segment,
it must necessarily be the case that the inward-spiraling shrinking motion is
actually the linear shrinking motion of a star-shaped domain. The _link_ of
this folded state at $p$, the intersection of $f_{0}(S)$ with a small sphere
centered at $p$, is a polygonal chain consisting of arcs of great circles.
Because it remains invariant throughout the motion, it does not self-
intersect. Any two distinct points of $S$ at distance $d$ from $p$ lie on
distinct points of a scaled copy of this link, on a sphere of radius $d$, and
for this reason cannot coincide. Therefore, the folded state at $f_{0}$ is
again a valid polygonal embedding.

By known results on the spherical carpenter’s rule problem, there exists a
continuous motion of the link, as a polygonal chain of fixed-length great-
circle arcs on the sphere, from its folded state to a completely flat state.
This motion induces a continuous motion of $f_{0}$ to a flat-folded state
[23]. Performing the continuous motion from $f(S)$ to $f_{0}(S)$, and then
using this carpenter’s rule solution on the resulting single-vertex surface,
produces a combined continous motion from $f(S)$ to a flat state.

To apply the same method to flat foldings without interior vertices, we cannot
use the carpenter’s rule problem, as 1d flat-folded polygonal chains with
fixed edge lengths cannot change continuously. Instead, we use a one-
dimensional version of Theorem 2.3:

######  Lemma 2.5.

Let $P$ be line segment, folded flat by a piecewise-isometry $f:P\to R$ with a
finite number of fold points and a consistent above-below relation for points
with the same image. Then there exists a continuous motion of flat foldings of
$P$ that transforms folding $f$ into an unfolded state.

######  Proof 2.6.

Chose any point $p$ of $P$ that is not a fold point, and form a continuous
family of one-dimensional folded states of $P$, $f_{i}(P)$ for $i\in(0,1]$, by
scaling $P$ by a factor of $i$ centered at $p$, applying $f$, and scaling the
result by a factor of $1/i$ centered at $f(p)$. When $i$ becomes less than the
distance from $p$ to the nearest fold, the result will be an unfolding of $P$,
so this provides a continuous transformation from $f$ to an unfolding.

######  Theorem 2.7.

Every flat folding without interior vertices of a bounded spiral-shaped domain
has a continuous unfolding motion through foldings of the same type.

######  Proof 2.8.

The proof follows the same outline as Theorem 2.3, combining shrinking and
unshrinking to reach a folded state in which all folds pass through the center
$p$ of the spiral shrinking transformations. If the result has a single fold
through $p$ we roll this fold to the boundary of $S$ rather than changing its
fold angle. If $p$ lies on the boundary of the domain and belongs to multiple
fold lines, we apply Lemma 2.5 in place of the spherical carpenter’s rule.

##  3 An instructive example

Figure 3: Two disks connected by a knotted band. This surface can be flattened
by a continuous motion of smooth embeddings.

Fig. 3 shows two large disks connected by a short thin band, embedded with the
disks spread flat and the band tied into an open overhead knot. If the disks
could be crumpled, it would be easy to untie and flatten, by crumpling the
disks into small enough balls that they could be passed through the knot, and
then uncrumpling. However, our model of smooth surface embeddings does not
allow crumpling. In every smooth embedding, the center point of each disk lies
in a flat subset of the disk with large diameter: either a diameter of the
disk, or a triangular subset of the disk with the vertices of this triangle on
the boundary of the disk.

Rolling up either disk around a diameter makes that diameter act like a rigid
line segment, but gives the rest of the disk a smaller overall shape. Rolling
both disks in this way can produce an embedding like the locked polygonal
chain with long “knitting needles” at its ends from Figure 1 of Biedl et al.
[4]. Any other rigid configuration for the two disks would be similarly
locked. However, this surface is unlocked! It can be unfolded through the
following sequence of transformations:

  * •

Let $D$ be a diameter of one of the two disks, touching the boundary of the
disk at its attachment point $p$ with the knotted band. Roll up the disk
around $D$, starting at one of the points of the disk that is farthest from
$D$, and leaving the semicircle opposite that point exposed on the outside of
the roll.

  * •

Poke $p$ into the hole made by the knotted band, so that if the rolled-up disk
around $D$ were not rigid, it could pass through the hole and untie the knot.
However, because $D$ is made rigid by the bending of the embedding as it rolls
around $D$, only the very end of diameter $D$ near $p$ can pass into the hole.

  * •

Continuously spin the parallel family of bend lines on the rolled-up disk, so
that it rolls up around a different diameter than $D$. Choose the direction of
spin that causes $p$ to travel along the exposed semicircle along the rolled-
up disk. As it does so, this will allow more of the diameter of the rolled-up
disk to poke through the hole in the knot.

  * •

When the bend lines have spun on the disk through an angle of $\pi$, causing
$p$ to reach the other end of the exposed semicircle, the rolled-up diameter
will have traveled all of the way through the hole in the knot, which will
become unknotted.

  * •

Unroll the disk so that it lies flat with the rest of the surface.

Thus, although bend lines of a smoothly embedded surface are rigid, the
underlying pattern of these lines on the surface can change continuously,
complicating the search for a proof that a surface is locked.

##  4 A locked surface

Figure 4: A flat surface with two holes and a configuration that cannot be
flattened. Intuitively, the two interlocked loops prevent the rolled center
region from unrolling, the bend lines of the roll make it act like a rigid
rod, and the length of this rod prevents the loops from being pulled around
its ends. For the way the two loops on the hidden side of the roll interlock,
see Fig. 5.

Fig. 4 depicts a smoothly-embedded unit square with two small loops on
midpoints of opposite sides, wrapped into a spiral roll with the loops
interlocked. Fig. 5 provides a cutaway view showing how the loops interlock.

Figure 5: Cutaway view of the two loops and centerline of the rolled-up part
of the surface of Fig. 4, showing their topological equivalence to the
Borromean rings

This rolled-up and interlocked surface is topologically equivalent (ambient
isotopic) to a flattened surface. As can be seen in Fig. 5, the interlocking
pattern is that of the Borromean rings, three unknotted loops in space that
cannot be separated from each other, in which any two of the loops become
unlinked if the third loop is removed. In the figure, the role of one of these
three loops is taken by the center of the spiral roll, which does not actually
form a loop, so this embedding is not topologically linked. From the
arrangement of the loops in Fig. 5, it is possible to unlink it by pulling the
right half of the red band to the right and up, around the right end of the
yellow spiral center, passing this part of the red band around the right half
of the blue band, and then passing the same part of the red band back to the
right and down around the right end of the yellow spiral center. This sequence
of motions reverses the front-back order of the two red-blue crossings on the
right half of the red band, after which it is straightforward to flatten the
whole surface. We have also verified that this configuration is topologically
unlocked using a physical model of two rubber bands attached to a pen.

As a smoothly-embedded surface, Fig. 4 is locked: it cannot be flattened while
preserving its geometry. To prove this, we use three interlocked properties of
the embedding that, like the Borromean rings, are interlinked: as long as any
two of the properties remain valid, the third one must remain valid as well,
so none of the three properties can be the first to break in any continuous
motion of the embedding. The first two properties are parameterized by a
parameter $\varepsilon$, which we will be able to make arbitrarily small by
making the length of the loops sufficiently small:

  1. $A$.

The two loops are bounded within distance $\varepsilon$ of each other.

  2. $B$.

The nearest bend line on the square to its center point has distance
$\leq\varepsilon$ to the center point, and crosses the top and bottom sides of
the square at distance $\geq 1/4-\varepsilon$ from its left and right sides.

  3. $C$.

The nearest bend line on the square to its center point can be completed into
a loop by a curve, on a sphere with it as diameter, forming Borromean rings
with the other two loops.

Figure 6: Bend lines for a deformed version of Fig. 4 in which the (heavier)
bend line through the center of the square crosses the top and bottom sides of
the square at distance $1/4$ from the sides. For these bend lines, the two
loops still meet near the center of the rolled surface.

The reason for the $1/4$ in property $B$ is that it is possible to deform Fig.
4, keeping the two loops interlocked in the center of the roll, so that the
center bend line crosses the top and bottom sides of the square at distance
$1/4$, as shown in Fig. 6. However, as we will prove, it is not possible to
move this center bend line significantly farther from vertical.

######  Lemma 4.1 ($A\wedge B\Rightarrow C$).

For all sufficiently small loop lengths $\delta$ and all sufficiently small
$\varepsilon$, continuous motion through states where $A(\varepsilon)$ and
$B(\varepsilon)$ hold, starting from a state where $C$ holds, cannot reach a
state where $C$ does not hold.

######  Proof 4.2.

Let $b$ be the bend line of property $B$. Then (Fig. 7) the attachment point
$p$ of the right loop must lie within the intersection of two spheres centered
at the endpoints of $b$, with radii equal to the distances from $p$ to those
endpoints, because the embeddings we consider are not allowed to increase
distances. Similarly, the attachment point of the left loop must lie within
the intersection of another two spheres. By property $A$, these attachment
points must be near each other, and until $C$ stops holding, they must also
lie near line $b$. This limits their nearby locations to points of line $b$
that are far from its endpoints on the square, so they cannot reach the sphere
through the endpoints of $b$ on which the connecting curve of property $C$
lies.

As the smooth embedding deforms continuously, the bend line nearest the center
point can change, but (as long as $B$ continues to hold) only by small
amounts, and the connecting curve can be changed by similar small amounts to
maintain property $C$. As the other two loops cannot reach the sphere
containing this curve, they cannot cross this curve (even though it does not
form a physical obstacle to them) and cannot change the knotted topology that
it forms with them.

######  Lemma 4.3 ($A\wedge C\Rightarrow B$).

For all sufficiently small $\varepsilon$ there exists a $\delta$ such that,
for loop lengths less than $\delta$, continuous motion through states where
$A(2\varepsilon)$ and $C$ hold, starting from a state where $B(\varepsilon)$
holds, cannot reach a state where $B(\varepsilon)$ does not hold.

######  Proof 4.4.

Let $b$ be the bend line nearest the center of the square. While $A$ and $C$
hold, $b$ must pass through the Borromean link formed by it and the two loops,
and so (if the center point itself does not lie on a bend line) it must lie on
a flattened part of the surface whose width is at most proportional to the
loop length. Therefore, $b$ is close to the center point, as part of property
$B$ demands. For a bend line that is close to the center point, the same
reasoning used in Fig. 7 and Lemma 4.1 shows that it must be at distance at
least $1/4-\varepsilon$ from the left and right sides, for otherwise the two
intervals on this bend line where the left and right loop attachment points
must be near would not intersect.

######  Lemma 4.5 ($B\wedge C\Rightarrow A$).

For all sufficiently small $\varepsilon$ there exists a $\delta$ such that,
for loop lengths less than $\delta$, continuous motion through states where
$B(2\varepsilon)$ and $C$ hold, starting from a state where $A(\varepsilon)$
holds, cannot reach a state where $A(\varepsilon)$ does not hold.

Figure 7: For a bend line through the square’s center, the attachment point of
the right loop must be in the shaded intersection of circles centered at the
bend endpoints, to avoid having greater distance to those endpoints than in
the flattened surface.

######  Proof 4.6.

By assumption $C$, the two loops of the surface and a third loop formed by
bend line $b$ form Borromean rings, and by assumption $B$, line $b$ is near
vertical and near the square’s center, forcing the attachment points of the
two loops to be far from the endpoints of $b$.

Each loop of the surface has small diameter, so if the two loops could be far
from each other it would be possible to make two small spheres (of radius
$\varepsilon$), one containing each loop. The loop containing $b$ lies on a
straight line within each of these two spheres. Although Borromean rings can
have two loops in two disjoint spheres [24] (Fig. 8), the third Borromean ring
cannot pass through either sphere as a straight line. If it could, we could
deform a loop within its sphere to a circle (as it is not pairwise linked with
the line passing through the sphere) and span it by a disk not crossed by the
other two loops, impossible for the Borromean rings. This contradiction shows
that the two loops cannot be separated by a distance larger than
$\varepsilon$, as stated in property $A$.

These properties together prove that the surface of Fig. 4 is locked: for
versions of this surface with short enough loops, it is impossible to deform
it as a smoothly embedded surface to its flattened state. The same is true for
the same reasons for approximations to this surface by polyhedral embeddings
or flat foldings without interior vertices. As a result, we have the following
theorem:

Figure 8: Borromean rings with links in separate spheres

######  Theorem 4.7.

For smooth embeddings, polyhedral embeddings without interior vertices, and
flat foldings without interior vertices, there exist flat surfaces with the
topology of a disk with two holes that are ambient isotopic to their flattened
form but cannot reach that form by a continuous sequence of folded states
staying within the same class of folded states.

######  Proof 4.8.

Choose a sufficiently small $\delta>0$ and $\varepsilon>0$ for the lemmas
above. The surface of Fig. 4, in the configuration of the figure, has
properties $A(\varepsilon)$, $B(\varepsilon)$, and $C$. As it continuously
moves, all three properties remain true; none can be the first to fail,
because at the instant it failed, the weaker properties $A(2\varepsilon)$ and
$B(2\varepsilon)$ would still be true, which by the lemmas would imply all
three of $C$, $A(\varepsilon)$, and $B(\varepsilon)$. However, the flattened
configuration does not obey the three properties, so it cannot be reached.

##  5 Conclusions and open problems

We have shown that, for smooth embeddings, polyhedral embeddings without
interior vertices, and flat foldings without interior vertices, every spiral-
shaped domain can be flattened. However, there exist more complex planar
shapes whose configuration spaces are disconnected: they have locked
embeddings that, although topologically equivalent, cannot be flattened.

Is every topological disk flattenable? Can the method that we used to flatten
Fig. 3 be generalized to other disks? What about surfaces with a single hole?
Additionally, we have only investigated the existence of flattenings, but not
their algorithmic complexity. For polyhedral embeddings and flat foldings, how
hard is it to determine whether a continuous flattening exists? What about for
smooth embeddings represented as a piecewise cylindrical and conical surface?

## References

  * [1] Zachary Abel, Jason Cantarella, Erik D. Demaine, David Eppstein, Thomas C. Hull, Jason S. Ku, Robert J. Lang, and Tomohiro Tachi.  Rigid origami vertices: conditions and forcing sets.  Journal of Computational Geometry, 7(1):171–184, 2016.  doi:10.20382/jocg.v7i1a9.
  * [2] Dov Aharonov, Mark Elin, and David Shoikhet.  Spiral-like functions with respect to a boundary point.  Journal of Mathematical Analysis and Applications, 280(1):17–29, 2003.  doi:10.1016/S0022-247X(02)00615-7.
  * [3] Brad Ballinger, David Charlton, Erik D. Demaine, Martin L. Demaine, John Iacono, Ching-Hao Liu, and Sheung-Hung Poon.  Minimal locked trees.  In Frank K. H. A. Dehne, Marina L. Gavrilova, Jörg-Rüdiger Sack, and Csaba D. Tóth, editors, Algorithms and Data Structures, 11th International Symposium, WADS 2009, Banff, Canada, August 21-23, 2009. Proceedings, volume 5664 of Lecture Notes in Computer Science, pages 61–73. Springer, 2009.  doi:10.1007/978-3-642-03367-4_6.
  * [4] Therese Biedl, Erik D. Demaine, Martin L. Demaine, Sylvain Lazard, Anna Lubiw, Joseph O’Rourke, Mark Overmars, Steve Robbins, Ileana Streinu, Godfried Toussaint, and Sue Whitesides.  Locked and unlocked polygonal chains in three dimensions.  Discrete & Computational Geometry, 26(3):269–281, 2001.  doi:10.1007/s00454-001-0038-7.
  * [5] Therese Biedl, Erik D. Demaine, Martin L. Demaine, Sylvain Lazard, Anna Lubiw, Joseph O’Rourke, Steve Robbins, Ileana Streinu, Godfried Toussaint, and Sue Whitesides.  A note on reconfiguring tree linkages: trees can lock.  Discrete Applied Mathematics, 117(1-3):293–297, 2002.  doi:10.1016/S0166-218X(01)00229-3.
  * [6] Therese Biedl, Anna Lubiw, and Julie Sun.  When can a net fold to a polyhedron?  Computational Geometry, 31(3):207–218, 2005.  doi:10.1016/j.comgeo.2004.12.004.
  * [7] Robert Connelly, Erik D. Demaine, Martin L. Demaine, Sándor P. Fekete, Stefan Langerman, Joseph S. B. Mitchell, Ares Ribó, and Günter Rote.  Locked and unlocked chains of planar shapes.  Discrete & Computational Geometry, 44(2):439–462, 2010.  doi:10.1007/s00454-010-9262-3.
  * [8] Robert Connelly, Erik D. Demaine, and Günter Rote.  Straightening polygonal arcs and convexifying polygonal cycles.  Discrete & Computational Geometry, 30(2):205–239, 2003.  doi:10.1007/s00454-003-0006-7.
  * [9] Erik D. Demaine, Martin L. Demaine, Vi Hart, John Iacono, Stefan Langerman, and Joseph O’Rourke.  Continuous blooming of convex polyhedra.  Graphs and Combinatorics, 27(3):363–376, 2011.  doi:10.1007/s00373-011-1024-3.
  * [10] Erik D. Demaine, Satyan L. Devadoss, Joseph S. B. Mitchell, and Joseph O’Rourke.  Continuous foldability of polygonal paper.  In Proceedings of the 16th Canadian Conference on Computational Geometry, CCCG’04, Concordia University, Montréal, Québec, Canada, August 9-11, 2004, pages 64–67, 2004.  URL: https://www.cccg.ca/proceedings/2004/55.pdf.
  * [11] Erik D. Demaine, Stefan Langerman, Joseph O’Rourke, and Jack Snoeyink.  Interlocked open and closed linkages with few joints.  Computational Geometry, 26(1):37–45, 2003.  doi:10.1016/S0925-7721(02)00171-2.
  * [12] Erik D. Demaine and Joseph S. B. Mitchell.  Reaching folded states of a rectangular piece of paper.  In Proceedings of the 13th Canadian Conference on Computational Geometry, University of Waterloo, Ontario, Canada, August 13-15, 2001, pages 73–75, 2001.  URL: https://erikdemaine.org/papers/PaperReachability_CCCG2001/.
  * [13] David Eppstein.  Realization and connectivity of the graphs of origami flat foldings.  Journal of Computational Geometry, 10(1):257–280, 2019.  doi:10.20382/jocg.v10i1a10.
  * [14] Dmitry Fuchs and Serge Tabachnikov.  Lecture 14: Paper Möbius band.  In Mathematical Omnibus: Thirty Lectures on Classic Mathematics, pages 199–206. American Mathematical Society, Providence, Rhode Island, 2007.  doi:10.1090/mbk/046.
  * [15] Yue Hao, Yun hyeong Kim, and Jyh-Ming Lien.  Synthesis of fast and collision-free folding of polyhedral nets.  In Proceedings of the 2nd ACM Symposium on Computational Fabrication, June 2018.  doi:10.1145/3213512.3213517.
  * [16] John Hopcroft, Deborah Joseph, and Sue Whitesides.  Movement problems for 2-dimensional linkages.  SIAM Journal on Computing, 13(3):610–629, 1984.  doi:10.1137/0213038.
  * [17] D. T. Lee and F. P. Preparata.  An optimal algorithm for finding the kernel of a polygon.  Journal of the ACM, 26(3):415–421, 1979.  doi:10.1145/322139.322142.
  * [18] Ezra Miller and Igor Pak.  Metric combinatorics of convex polyhedra: Cut loci and nonoverlapping unfoldings.  Discrete & Computational Geometry, 39(1-3):339–388, 2008.  doi:10.1007/s00454-008-9052-3.
  * [19] Gaiane Panina and Ileana Streinu.  Flattening single-vertex origami: The non-expansive case.  Computational Geometry, 43(8):678–687, 2010.  arXiv:1003.3490, doi:10.1016/j.comgeo.2010.04.002.
  * [20] John Pardon.  On the unfolding of simple closed curves.  Transactions of the American Mathematical Society, 361(4):1749–1764, 2009.  doi:10.1090/S0002-9947-08-04781-8.
  * [21] Guang Song and Nancy M. Amato.  A motion-planning approach to folding: From paper craft to protein folding.  IEEE Transactions on Robotics and Automation, 20(1):60–71, February 2004.  doi:10.1109/tra.2003.820926.
  * [22] Ileana Streinu.  A combinatorial approach to planar non-colliding robot arm motion planning.  In Proceedings of the 41st IEEE Symposium on Foundations of Computer Science, pages 443–453, 2000.  doi:10.1109/SFCS.2000.892132.
  * [23] Ileana Streinu and Walter Whiteley.  Single-vertex origami and spherical expansive motions.  In Discrete and Computational Geometry: Japanese Conference, JCDCG 2004, Tokyo, Japan, October 8-11, 2004, Revised Selected Papers, volume 3742 of Lecture Notes in Computer Science, pages 161–173. Springer-Verlag, 2005.  doi:10.1007/11589440_17.
  * [24] Morwen B. Thistlethwaite.  On the algebraic part of an alternating link.  Pacific Journal of Mathematics, 151(2):317–333, 1991.  doi:10.2140/pjm.1991.151.317.
  * [25] Nicholas Turner, Bill Goodwine, and Mihir Sen.  A review of origami applications in mechanical engineering.  Proceedings of the Institution of Mechanical Engineers, Part C, 230(14):2345–2362, August 2015.  doi:10.1177/0954406215597713.
  * [26] Zhonghua Xi and Jyh-Ming Lien.  Continuous unfolding of polyhedra – a motion planning approach.  In 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, September 2015.  doi:10.1109/iros.2015.7353828.


# Research and Practice of Delivering Tabletop Exercises

Jan Vykopal  0000-0002-3425-0951 Masaryk UniversityFaculty of
InformaticsBrnoCzech Republic<EMAIL_ADDRESS>,  Pavel Čeleda
0000-0002-3338-2856 Masaryk UniversityFaculty of InformaticsBrnoCzech Republic
<EMAIL_ADDRESS>,  Valdemar Švábenský  0000-0001-8546-280X Masaryk
UniversityFaculty of InformaticsBrnoCzech Republic<EMAIL_ADDRESS>,
Martin Hofbauer  0009-0005-3998-9164 Masaryk UniversityFaculty of
InformaticsBrnoCzech Republic<EMAIL_ADDRESS>and  Martin Horák
0000-0002-1835-6465 Masaryk UniversityFaculty of InformaticsBrnoCzech Republic
<EMAIL_ADDRESS>
(2024)

###### Abstract.

Tabletop exercises are used to train personnel in the efficient mitigation and
resolution of incidents. They are applied in practice to support the
preparedness of organizations and to highlight inefficient processes. Since
tabletop exercises train competencies required in the workplace, they have
been introduced into computing courses at universities as an innovation,
especially within cybersecurity curricula. To help computing educators adopt
this innovative method, we survey academic publications that deal with
tabletop exercises. From papers we identified and examined, we selected papers
for a detailed review. The results show that the existing research deals
predominantly with exercises that follow a linear format and exercises that do
not systematically collect data about trainees’ learning. Computing education
researchers can investigate novel approaches to instruction and assessment in
the context of tabletop exercises to maximize the impact of this teaching
method. Due to the relatively low number of published papers, the potential
for future research is immense. Our review provides researchers, tool
developers, and educators with an orientation in the area, a synthesis of
trends, and implications for further work.

tabletop exercise, incident response, experiential learning, cybersecurity,
hands-on training, systematic literature review

††journalyear: 2024††copyright: rightsretained††conference: Proceedings of the
2024 Innovation and Technology in Computer Science Education V. 1; July 8–10,
2024; Milan, Italy††booktitle: Proceedings of the 2024 Innovation and
Technology in Computer Science Education V. 1 (ITiCSE 2024), July 8–10, 2024,
Milan, Italy††doi: 10.1145/3649217.3653642††isbn:
979-8-4007-0600-4/24/07††ccs: General and reference Surveys and
overviews††ccs: Social and professional topics Computing education

##  1\. Introduction

The Computing Curricula 2020 report (CC2020) (Force, 2020) responds to the
need for graduates who are effective in their work roles and tasks. It
promotes using _competencies_ instead of knowledge to describe computing
curricula. Competency augments knowledge with its skilled application
motivated by the purpose of accomplishing a task. According to CC2020, the
knowledge of subject matter and related skills are as important as analytical
and critical thinking, collaboration and teamwork, and dispositions such as
being responsible and flexible or having self-confidence and self-control.

Computing educators research and use various forms of experiential learning to
prepare students for their future careers, such as simulations, team projects,
or industry internships. We highlight _tabletop exercises_ (TTX) as an
efficient method for gaining competencies relevant to work roles and tasks
centered around analysis, decision making, and communication with various
parties. In particular, these exercises are relevant for cybersecurity and IT
governance courses (ENISA, 2009), addressing incident response methodologies
outlined in frameworks like COBIT (Control OBjectives for Information and
related Technology) and ITIL (Information Technology Infrastructure Library).
While tabletop exercises are common in professional settings, they have not
been widely used in computing courses. We believe that these exercises have a
great potential for innovating teaching practice in higher education
institutions.

This paper examines the development and state of the art of TTX as presented
at various academic venues. The core contribution is a systematic review of
research papers to understand state of the art. Our work is useful to various
target groups. For educators, it shows examples of approaches and exercises
and presents practical recommendations. For researchers, it provides an
overview of methods used in exercises and implications for further research.
Finally, developers of educational tools can be inspired by the existing tools
and implications for future work.

##  2\. Background

A tabletop exercise is a form of a teaching activity aimed at training teams
in responding to crisis situations (Grance et al., 2006). It involves
simulating a crisis within the context of business operations in an
organization, such as a ransomware attack on the company infrastructure or
exfiltration of sensitive information. The team members (exercise
participants) assume different roles in the organization, such as chief
security officer or cybersecurity incident responder (Angafor et al., 2023).
During the exercise, they discuss how to effectively respond to the crises
while adhering to processes and regulations. Instructors facilitate these
discussions and provide a debriefing after the TTX. The exercise usually lasts
a few hours or days at most.

TTXs are driven by injects, pre-scripted messages, which can take the form of
an e-mail or a news article, provided to trainees during the exercise. The
purpose of injects is to advance the exercise and stimulate further actions
and discussions. For instance, injects can notify teams of a data breach in
their organization, requiring them to respond appropriately (Grance et al.,
2006).

TTXs focus on communication, coordination, and collaboration (Angafor et al.,
2023), not particular technical skills as it is the case during hands-on
training in an emulated IT environment (such as a cybersecurity lab (Huff et
al., 2023) or cyber range (Yamin et al., 2020)). The nature of TTXs allows to
conduct them using pen and paper or simple online office applications (such as
Microsoft Forms), making TTXs relatively cost-effective. However, assessing
the trainees is not automated – it requires instructors’ manual effort, which
is highly time-consuming. It may take days or weeks until the trainees receive
feedback from instructors, which lowers the effectiveness of the exercise.

TTXs share certain traits with some other forms of active learning (Sanders et
al., 2017) but also have unique characteristics. For instance, course projects
in software development or Process oriented guided inquiry learning (POGIL)
(Kussmaul, 2012) also involve student teams that work together. In contrast,
TTXs do not feature a clearly specified and structured assignment. Moreover,
student teams in a TTX are not guided by instructors but decide on their own
about the current priorities and tasks to complete. Last, TTXs intentionally
overwhelm trainees with numerous pieces of information and inputs to simulate
a stressful emergency situation.

TTXs in the cybersecurity domain can be conducted either independently or as a
part of complex exercises involving technical skills (denoted as cyber defense
exercises or red vs. blue team exercises). Locked Shields (NATO Cooperative
Cyber Defence Centre of Excellence, 2023) and Cyber Europe (for
Cybersecurity), 2023) are exercises combining TTXs with technical training.
Both are centered around a background story resembling a recent real crisis or
attack campaign (Vykopal et al., 2017; Østby et al., 2019). Participants
representing one organization or country are assigned different roles and
divided into teams. While some teams exercise mainly technical skills in an
emulated IT environment, others are engaged in decision-making processes,
standard operational procedures, or communication in a local and international
context. Locked Shields is the largest global defense exercise organized by
the NATO Cooperative Cyber Defence Centre of Excellence in Tallinn, Estonia,
since 2010. Cyber Europe is a series of pan-European exercises developed for
IT security, business continuity, and crisis management teams by European
Union Agency for Cybersecurity (ENISA), since 2010.

Diverse commercial solutions are available to address various aspects of
emergency preparation. Some vendors, such as PreparedEx (PreparedEx, 2023) or
Emergency Solutions International (Inc., 2023), provide comprehensive TTXs for
crises, training emergency responders, and highlighting inefficiencies in
processes and inadequate capabilities. Numerous commercial services (e.g.,
(CrowdStrike, 2023; Ltd., 2023; RedLegg, 2023; Consulting, 2023)) focus on
validating incident response and business continuity plans in IT operations or
cybersecurity and are often part of broader services portfolios. Additionally,
there are platform-as-a-service solutions (namely (™, 2023; Cinten, 2023;
eeedo inc, 2023; Avalias, 2023)), all focused on crisis management in general.
Furthermore, specialized platforms, such as Cyber Crisis Simulator (Labs,
2023) or open-source OpenEx (Filigran, 2024), are designed specifically for IT
and cybersecurity crisis management.

National or international authorities, such as NIST (Grance et al., 2006),
FEMA (Federal Emergency Management Agency, 2020), ENISA (ENISA, 2009), or ISO
(ISO, 2013) published standards and guides on complex exercises, which also
include TTXs or can be applied to them.

To conclude, a TTX is an established training method used in practice, yet
mostly outside university settings. Our review maps the state of the art based
on academic publications on this topic.

##  3\. Previous Literature Reviews

In the academic literature, there is one review of TTXs in the cybersecurity
domain and three papers in the subject area of healthcare.

Angafor et al. (2020b) reviewed academic and commercial product literature on
TTXs used for training cybersecurity incident response teams. The scope of our
review is wider; we searched for papers in broad subject areas of computer
science and engineering. Also, our review captures recent research and trends
since 2020 when the other review was published. The overlap of this and their
study is only two papers (namely P4 and P7, see details in Table 1).

Mahdi et al. (Mahdi et al., 2023) reviewed disaster preparation exercises
conducted by academic healthcare institutions. Based on the reviewed
literature, the authors concluded that TTXs are the easiest to organize,
conduct, and evaluate, while also useful in evaluating emergency response
protocols and their subsequent improvement.

Evans (Evans, 2019) surveyed healthcare literature for using TTXs in nursing
education. The opportunity to identify knowledge gaps, as well as knowledge
gain, was reported. The author also proposed a list of considerations for
exercise development.

Finally, Frégeau et al. (Frégeau et al., 2020) published a scoping protocol
for a review that maps the uses of TTXs in healthcare. However, the review
itself has not been published yet.

##  4\. Method of Conducting the Review

We follow the guidelines for a systematic literature review (SLR) (Kitchenham
and Charters, 2007) and a systematic mapping study (Petersen et al., 2008,
2015). This section presents the SLR protocol, which specifies the research
questions, search process (see Figure 1), and criteria for including the
discovered papers.

Figure 1. Steps of the systematic literature review and the number of
processed papers.

Diagram showing the linear progress of the steps of the literature review.

    TITLE-ABS-KEY( "table* exercise" OR "communication exercise" ) AND ( LIMIT-TO ( SUBJAREA, "COMP" ) OR LIMIT-TO ( SUBJAREA, "ENGI" ) )

Figure 2. The query for the automated paper search for papers in Scopus
(Elsevier, 2024).

Text-based search query input in the Scopus database.

###  4.1. Research Questions

Our literature review examines five research questions:

  1. (1)

What formats of tabletop exercises are used?
Namely how the exercises are prepared, delivered and what tools are applied.

  2. (2)

Who are the participants of the exercises?
What are the target groups of the exercises (trainees) and by whom are they
organized (instructors, designers)?

  3. (3)

How are the exercises developed, assessed, and evaluated?
We will examine methods for exercise preparation, assessment of trainees, and
evaluation of the exercise itself.

  4. (4)

How are the research results applied in practice?
Do the publications provide any supplementary materials or artifacts for other
educators?

  5. (5)

What are the future research directions and challenges?
What is the ongoing and long-term goal?

###  4.2. Paper Search and Automated Filtering

We used the Scopus citation database of peer-reviewed literature (Elsevier,
2024). Scopus indexes a representative portion of the databases of individual
publishers, including ACM Digital Library, IEEExplore, SpringerLink, or
Elsevier ScienceDirect. We did not use Google Scholar since it additionally
indexes many non-peer-reviewed papers.

When constructing the search query, we focused on the subject area of computer
science and engineering and used “tabletop exercise” as the primary keyword.
We also sought not to miss potential alternative names for tabletop exercises,
such as “simulation” or “communication exercises”. After several pilot
searches, we concluded with the query presented in Figure 2. The asterisk is a
wildcard to match both adjectives “table top” and “table-top”. However, the
wildcard is not used for the stem of the word “communication” because the
pilot search yielded numerous false positives. Similarly, we omitted
“simulation” matching a huge number of papers unrelated to education and
training. The search was case-insensitive.

We started the search on May 11, 2023, and then subscribed to Scopus e-mail
notifications that informed us about newly indexed papers, which we gradually
added to the candidate set. We stopped adding new candidates on July 19, 2023.
In total, we found candidate papers.

###  4.3. Inclusion and Exclusion Criteria

After multiple pilot tests, we set ten criteria.

  1. (1)

Papers must deal with TTXs in IT or operational technology (OT) operations or
security.

  2. (2)

Papers describing exercises and tools in other domains, such as transportation
or utilities, without considering cybersecurity or IT/OT operations aspects,
are excluded.

  3. (3)

We include papers written in English with _full text available_ 111We consider
a paper to be available if we can access it using tens of licensed electronic
resources our institution has access to, or if the full text is freely
available online.. We do _not set any page limit_ to include short papers,
which report ongoing and future work at the time of their publication.

  4. (4)

Other types of documents yielded by the automated search, such as conference
reviews or records, are excluded.

  5. (5)

Papers must describe _exercising or supporting the exercise of a complex
process or standard operation procedures in a safe, simulated environment_.
For instance, this includes experience reports with lessons learned or
descriptions of tools supporting the preparation, execution, and analysis of
TTXs.

  6. (6)

Papers that describe only exercising essential communication and writing
skills (e.g., technical writing, proposal paper, presentation and its
delivery) are excluded.

  7. (7)

Papers must report on an exercise involving _teams or groups, not only
individuals_.

  8. (8)

The paper must support an _educational goal_ , for example, to train or assess
participants, or help instructors conduct the exercise, or understand the
learning processes.

  9. (9)

We exclude papers that use TTXs solely as an evaluation tool in non-
educational settings, such as for testing of software or IT systems.

  10. (10)

_Generic methodologies_ applied or applicable to TTXs in IT and OT operations
and security are included.

Two authors of this paper each screened all candidate papers and applied these
criteria independently. The authors followed a simple algorithm for screening
the paper content (Švábenský et al., 2022):

⬇

for each paper in the candidate set:

read the title and abstract

decide for inclusion or exclusion

if decision cannot be made:

read the introduction and conclusion

decide for inclusion or exclusion

if decision cannot be made:

skim-read the rest of the paper

decide for inclusion or exclusion

###  4.4. Selecting Papers for Review

After both authors finished their reading, they compared candidate papers they
identified after applying the inclusion and exclusion criteria. If both
authors decided to include the paper, it was selected for review. If both
decided not to include the paper, it was excluded. In case of a disagreement,
the authors discussed their views and agreed on the final decision.

In the first round, the authors agreed that out of 140 candidate papers, nine
(6.4%) were fitting the selection criteria. The authors had opposite opinions
on seven (5%) papers. Out of the 16 papers, which passed the first round, the
authors selected papers and rejected two papers in the second selection round.
Their inter-rater agreement (Krippendorff, 2004) measured by Krippendorff’s
$\alpha$ for nominal data was 0.69, which is substantial. The coefficient was
calculated using the Python NLTK module (Natural Language Toolkit
Project(2023), NLTK).

##  5\. Results

This section presents data extracted from reading the full texts of the papers
to answer the research questions posed in Section 4.1. Table 1 summarizes the
goals and types of the reviewed papers. The summary of the papers is provided
in Section 6. Further, we refer to the selected papers using arbitrary
numbered identifiers $P_{x}$.

Table 1. Overview of the reviewed papers ordered by the year of publication. Paper Type: SW = software tool, E = exercise instance (run of a particular exercise), F = exercise format (method for conducting the exercise), O = other topic (see Goal). Paper ID | Year | Paper Type | Goal of the Paper
---|---|---|---
P1 (Marshall, 2009) | 2009 | SW, F | Describe a tool for planning complex functional and tabletop exercises
P2 (Ottis, 2014) | 2014 | F, E | Describe a format of lightweight exercises
P3 (Edzén, 2014) | 2014 | O | Propose design ideas helping to incorporate more problems having no obvious solution into TTXs
P4 (Line and Moe, 2015) | 2015 | E | Study challenges when performing TTXs
P5 (Makrodimitris and Douligeris, 2015) | 2015 | O | Review of Hermes, MSB and ENISA exercise methodologies w.r.t. the ISO standard 22398
P6 (Janssen and Vreugdenhil, 2015) | 2015 | O, E | Introduce a method for trainee assessment
P7 (Brilingaitė et al., 2017) | 2017 | SW, F, E | Present a web-based environment for conducting TTXs
P8 (Taniuchi et al., 2019) | 2019 | E, O | Study errors made by TTX facilitators when interacting with trainees
P9 (Kopustinskas et al., 2020) | 2020 | E | Report takeaways from a particular exercise
P10 (Angafor et al., 2020a) | 2020 | O | Investigate the current cybersecurity skills gap and how TTXs can fill it
P11 (Ota et al., 2022) | 2022 | F, E | Describe three exercises developed by the authors
P12 (Angafor et al., 2023) | 2023 | O, E | Report experience from a virtual incident response tabletop exercise
P13 (Skytterholm and Hotvedt, 2023) | 2023 | O, F | Investigate the characteristics of a realistic and expedient exercise scenario
P14 (Zacharis and Patsakis, 2023) | 2023 | SW, F | Apply machine learning to unstructured information sources to generate exercise content

###  5.1. RQ1: What Formats of Exercises Are Used?

From seven papers (namely P1, P2, P4, P7, P12, P13, P14) out of eight that
contain enough information about the exercise format, we see that the
exercises are designed as a series of injects (events, problems, or
situations, see Section 2). These injects form a scenario that is unknown to
the exercise participants beforehand. The injects are provided by exercise
facilitators to participants who discuss appropriate actions, processes, or
best practices within their team or with all participants.

The eighth paper (P11) reports two kinds of exercises that extend this common
format. One TTX uses cards that drive the exercise itself (attack cards) or
stimulate participants to think about several options (action cards, situation
awareness cards, and information-sharing cards). Another TTX in P11 tasked
participants to create a scheme depicting information and workflows within an
organization affected by an incident.

Only four papers mention the use of any software tool during the exercise. P7
presents a web application for delivering a TTX involving various roles
according to a scenario defined by the facilitator. Participants use a
graphical interface to respond to presented injects. They can submit a short
description of how they would solve the inject, inform other roles about it,
or delegate the resolution to other roles. P11 mentions an exercise where
participants use software to interact with virtual participants. Participants
choose a response to presented injects from a limited set of predefined
actions. P12 reports a remote exercise facilitated through Microsoft Teams web
conferencing application. P1, P6, and P14 use a software tool before or after
the actual exercise; see Section 5.3.

The exercises last from several hours (P2, P13, P14) through one day (P4, P7,
P8, P9, P11, P12) to a few days (P6).

###  5.2. RQ2: Who Are the Exercise Participants?

Trainees come from diverse sectors. The most frequent were critical
infrastructure organizations, such as energy distribution operators (P4), a
water management centre (P6), industrial control systems stakeholders (P8,
P11), or oil and gas suppliers (P9, P13). Two exercises were carried out for
university students (P2, P7). One exercise was conducted for a large law
enforcement organization (P12). The number of trainees ranged from 20 to 108.

Exercises were designed by representatives of national or transnational
authorities (P4, P9) or academic staff (P2, P8, P11, P12). The type of the
organizing entity determines the target group and its diversity (employees of
one organization, multiple organizations in a single country or multiple
organizations from multiple countries).

###  5.3. RQ3: How Are the Exercises Developed, Assessed, and Evaluated?

#### Development

There is no prevailing trend in the process of exercise preparation in the
reviewed papers, even though guidelines (Federal Emergency Management Agency,
2020; ENISA, 2009; Grance et al., 2006) and standards (ISO, 2013) had already
been published.

The exercises in P4 and P7 were designed and conducted using guidelines from
NIST (Grance et al., 2006) and ENISA (ENISA, 2009), respectively.

P1 introduces a web-based collaborative tool for designers of cybersecurity
TTXs, enabling scenario creation based on TTX objectives following a national
guideline for exercise development (Federal Emergency Management Agency,
2020). It supports various user roles like Scenario Designer, Subject Matter
Expert, or Observer, in producing a detailed list of scenario events.

P5 studies the compliance of three guidelines from the European authorities
for designing and conducting exercises with the international standard ISO
22398 (ISO, 2013).

P13 investigates the characteristics of a realistic and expedient scenario in
the field of industrial control systems. The paper lists 21 criteria and eight
exercise topics based on these scenarios.

P3 argues that current tabletop exercises do not accurately reflect the
reality of uncertain and complex problems that do not have obvious solutions.
It presents three design ideas for designing more efficient exercises. First,
the focus should be on unsolved problems that the participants themselves come
up with. Second, problems should be tamed during the exercise by the
participants instead of during the planning phase by the designer. Third, the
participants should use existing plans and experience from previous
emergencies to resolve the problems in collaboration.

P14 employs machine learning to generate scenarios of cybersecurity exercises.
First, a corpus of publicly available articles about cybersecurity incidents
is created and annotated to detect threat actors, incidents, and victims.
Then, an exercise designer provides inputs such as a keyword or a sector for
generating a graph of an incident for a created exercise. The graph is then
enhanced using information mined from existing cybersecurity taxonomies.
Finally, the graph is transformed to text using GPT-2.

#### Assessment

The assessment of exercise participants (trainees and/or facilitators) is
addressed only in three papers.

P6 proposes a method and a tool for structured assessment of trainees. First,
observable behavior in exercise subgoals must be defined in the exercise
preparation. During the exercise, human observers record five performance
aspects (timeliness, accuracy, relevance, completeness, and cost-
effectiveness) in four phases of the OODA loop (Observe, Orient, Decide, Act)
(Richards, Chet, 2020), and the tool produces a score capturing trainees’
performance.

Exercises reported in P4 include unstructured assessment of trainees’ actions
by the observers and facilitators after the exercise.

P8 studies errors made by exercise facilitators during their interactions with
groups of trainees. P8 recognizes the error of _commission_ and the error of
_omission_ , which can negatively affect the progress of a trainee group in
the exercise.

#### Evaluation

Six papers addressed the evaluation of the exercise itself. None reported the
use of specific qualitative or quantitative research methods.

In P4 and P11, the evaluation is conducted as a reflection of trainees and a
discussion with the exercise designers and facilitators after the exercise.

In the exercise described in P2, teams of trainees compete against themselves
in opposing roles of attackers and defenders. The feedback session is then
used to uncover the goals and motivations of the other role. Also, the
instructors ask for immediate feedback on the exercise format.

P7 presents summary counts of trainees’ actions in the exercise platform and
messages sent during the exercise.

P9 reports the evaluation that has two phases. Firstly, the trainees shared
comments about what they learned at the end of the exercise. Afterward,
experts invited by the exercise organizers surveyed the trainees and evaluated
the outputs produced during the exercise.

P12 discusses responses from a trainee survey, which relates the exercise
content to incident response and disaster recovery plans of a particular
organization where the exercise was held.

###  5.4. RQ4: How Are the Results Applied?

No reviewed paper refers to supplementary materials (such as software tool
implementation or exercise scenario) that other educators can directly use. In
particular, P1, P6, P7, and P14 describe software tools, but the papers do not
refer to any materials, such as software repositories or websites presenting
the tools. P13 only outlines eight scenarios as an inspiration for creating a
new exercise.

A few papers distill recommendations and lessons learned from TTX preparation
or delivery applicable to other exercises. P2 lists steps describing the
preparation of a lightweight TTX and the most common problems encountered. P3
provides three design ideas for more realistic and efficient exercises (see
Section 5.3). P4 states recommendations for conducting exercises.

Finally, P5 shows that the methodology by ENISA (ENISA, 2009) is the most
compliant with the ISO 22398 standard out of the three reviewed methodologies,
yet not entirely.

###  5.5. RQ5: What Are the Future Directions?

The reviewed papers reported diverse future work. P2 suggests developing more
detailed instructions, creating mock scenarios, and formalizing and publishing
student feedback. P3 advises the development of an online tool for running a
TTX, evaluating the proposed design ideas, and studying how TTXs are used in
incident recovery. P4 recommends studying best practices and challenges of
organizations conducting TTXs regularly and studying real-life incident
responses to design more useful future exercises. P6, which deals with
assessment, plans to replace binary scoring with a four-point scale in the
presented method. Since P7 piloted the exercise with university students,
future work includes conducting the exercise with professionals working in the
target industry. In addition, it propounds the development of an assessment of
trainee stress levels, which can then be used to adapt the exercise
progression dynamically. P8 mentions sharing the instances of errors among
facilitators. While P9 simply recommends conducting more exercises, P11
suggests developing a method to demonstrate the risks due to failure to act.
P12 plans to improve the presented exercise using lessons learned from its
first run and develop a training program for staff at all organizational
levels. P13 would like to test the proposed criteria for realistic and
expedient scenarios in practice when deploying the proposed scenarios.
Finally, P14 lists several concrete directions to improve the tool for
generating exercise content.

##  6\. Discussion

This section provides takeaways from the review of the selected papers. It
helps the reader understand the greater themes of the results presented in
Section 5. The review limitations are also discussed.

###  6.1. Summary of the Observed Trends

The topics in the identified papers differ widely. Most papers are reports
from runs of a particular exercise or describe an exercise format. The typical
TTX format is based on pieces of information gradually provided to the
trainees by exercise facilitators. The trainees discuss and respond to the
information in teams. The exercises are usually held for tens of trainees on
site for several hours or days, with little support from dedicated software
tools.

The explicit learning phase usually happens at the end or after the exercise,
when trainees reflect on their decisions with other participants or when the
previously unknown parts of the scenario are disclosed to them. Other types of
assessment or feedback are rare because the complexity and labor of the manual
preparation of the exercises hinder instructors from focusing on trainee
assessment and exercise evaluation. Out of only three papers (P4, P6, and P8)
that addressed assessment, only one (P6) suggested a method that goes further
than unstructured assessment by the observers and facilitators. Similarly, the
evaluation of the exercise itself is also underdeveloped. No paper uses
advanced feedback methods but trainees’ reflections, discussions, or surveys.

The reviewed papers did not provide many actionable artifacts or supplementary
materials (such as software supporting exercise preparation and delivery,
exercise playbooks, scenarios, or checklists) for others considering
conducting their own exercise. The absence of publicly available materials
prevents educators from adopting the concept of tabletop exercises as a
teaching method.

The current practice of conducting TTXs relies on manual preparation of
exercise content (namely injects that form exercise scenarios), with no
automation and limited future reusability. We believe this practice can be
improved by leveraging large language models for semi-automated content
generation (P14) or employing dedicated software for TTX preparation and
delivery (see Section 5.1).

Finally, future directions in the papers often mention evaluating proposed
methods by field tests or running the exercise for more trainees or different
target groups. These plans indicate that the described methods and exercises
should be developed further and provide new research opportunities in
computing education.

###  6.2. Limitations

This review is limited by narrowing its scope by only using the Scopus
citation database of the peer-reviewed literature. This decision was explained
in Section 4.2. Also, when using Scopus, we did not have to deal with
duplicate records, which may occur when using multiple other databases.

The second limitation is keyword choice. Using other keywords may have led us
to find different papers fitting the selection criteria. However, pilot
searches with related terms, such as “crisis exercise” returned an excessive
number of unrelated papers (see Section 4.2).

Finally, the data extraction was done manually, which may lead to
misinterpretation of the presented information. To mitigate this risk, we
performed cross-author discussion and validation.

##  7\. Conclusions and Future Directions

While tabletop exercises have been conducted for over a decade, often by
governmental and military agencies, the number of published papers has grown
only in recent years. The organizing entities have developed their guidelines
and summarized the best practices, but there is no widely used terminology,
textbooks, or tools for the exercise development, delivery, or evaluation.
This situation is one of the causes why TTXs are not yet widely used in
computing education, even though they can help develop competencies and
dispositions relevant to students’ future careers as IT professionals.

To the best of our knowledge, this paper is the first systematic review of
TTXs in the context of IT operations and security. We examined 140 papers and
reviewed 14 of them in detail. The structured information extracted from the
identified papers is published as supplementary material (Vykopal et al.,
2024), which also includes a review of 16 practical implementations that did
not fit this paper’s page limit.

This review implies several possible directions for future research and
practice. First, novel formats of the exercises can be proposed to make them
more realistic but still lightweight on resources and effort required for
their development and delivery. For instance, the current discussion-based
approach (“What would your team/organization do when a certain event
happened?”) could be innovated by letting the trainees perform the actual
simulated actions (such as asking another team/external party to do something)
and observe whether they do what was expected by the exercise scenario based
on training objectives. Another example is an investigation of opportunities
and limits of remote or hybrid exercise that would enable more trainees to
participate and ease traveling and logistics.

The next direction is to start using dedicated software tools in the TTX
preparation, delivery, and evaluation, such as INJECT Exercise Platform
(Švábenský et al., 2024). Automated tools can substantially increase exercise
scalability and lower the preparation effort. Not only can they enable the
participation of more trainees (regardless of whether the TTX is delivered
onsite or remotely), but they also reduce the workload of exercise designers
and facilitators. Moreover, researchers may explore employing machine learning
for trainee assessment or leverage large language models to generate exercise
content.

Next, developers can provide a dedicated platform for designing and conducting
the exercise, which would assist the facilitators or even automate some of
their tasks (such as playing the role of some simulated actors in the
exercise). Moving the exercise from pen and paper or online forms and
documents to the dedicated platform would enable the collection of data about
trainees’ interactions during the exercise and further analysis for the
assessment of trainees and evaluation of the exercise. Finally, the exercise
designers would strongly benefit from releasing and sharing tangible outputs,
such as software tools, scenarios, or checklists.

###### Acknowledgements.

This research was supported by the Open Calls for Security Research 2023–2029
(OPSEC) program granted by the Ministry of the Interior of the Czech Republic
under No. VK01030007 – Intelligent Tools for Planning, Conducting, and
Evaluating Tabletop Exercises.

## References

  * (1)
  * Angafor et al. (2020a) Giddeon N. Angafor, Iryna Yevseyeva, and Ying He. 2020a.  Bridging the Cyber Security Skills Gap: Using Tabletop Exercises to Solve the CSSG Crisis. In _Serious Games_. Springer International Publishing, Cham, 117–131.  https://doi.org/10.1007/978-3-030-61814-8_10
  * Angafor et al. (2020b) Giddeon N. Angafor, Iryna Yevseyeva, and Ying He. 2020b.  Game-based learning: A review of tabletop exercises for cybersecurity incident response training.  _Security and Privacy_ 3, 6 (2020), e126.  https://doi.org/10.1002/spy2.126
  * Angafor et al. (2023) Giddeon N. Angafor, Iryna Yevseyeva, and Leandros Maglaras. 2023\.  Scenario-based incident response training: lessons learnt from conducting an experiential learning virtual incident response tabletop exercise.  _Information & Computer Security_ 31, 4 (2023).  https://doi.org/10.1108/ICS-05-2022-0085
  * Avalias (2023) Avalias. 2023.  Avalanche TTX.  Retrieved January 17, 2024 from https://www.avalias.com/products/avalanche-ttx
  * Brilingaitė et al. (2017) Agnė Brilingaitė, Linas Bukauskas, Virgilijus Krinickij, and Eduardas Kutka. 2017.  Environment for Cybersecurity Tabletop Exercises. In _11th European Conference on Games Based Learning_. Graz, Austria, 47–55.  https://www.researchgate.net/publication/320244434_Environment_for_Cybersecurity_Tabletop_Exercises
  * Cinten (2023) Cinten. 2023.  Crisis Management.  Retrieved January 17, 2024 from https://www.cinten.com/crisis-management-digital-simulations-performance-analytics
  * Consulting (2023) Cyber Security Operations Consulting. 2023\.  _Cybersecurity TTX and Incident Response_.  CyberSecOp.  Retrieved January 17, 2024 from https://cybersecop.com/tabletop-exercise/cybersecurity-tabletop-exercise-services
  * CrowdStrike (2023) CrowdStrike. 2023\.  _CrowdStrike TTX_.  CrowdStrike.  Retrieved January 17, 2024 from https://www.crowdstrike.com/services/prepare/tabletop-exercise/
  * Edzén (2014) Svante Edzén. 2014\.  Table-Top Exercises for Emergency Management: Tame Solutions for Wicked Problems. In _2014 47th Hawaii International Conference on System Sciences_. IEEE, USA, 1978–1985.  https://doi.org/10.1109/HICSS.2014.250
  * eeedo inc (2023) eeedo inc. 2023\.  _Crisis & Preparedness Cooperation Exercise Simulator_.  eeedo inc.  Retrieved January 17, 2024 from https://eee.do/crisis-and-preparedness-cooperation-exercise-simulation-system/
  * Elsevier (2024) Elsevier. 2024\.  Scopus.  Retrieved January 17, 2024.  https://www.scopus.com
  * ENISA (2009) ENISA. 2009.  _Good Practice Guide on National Exercises_.  Technical Report. European Union Agency for Cybersecurity.  https://www.enisa.europa.eu/publications/national-exercise-good-practice-guide
  * Evans (2019) Cathleen A. Evans. 2019\.  Tabletop exercises in the nursing classroom: An introduction for nurse educators.  _Nursing Forum_ 54, 4 (2019), 669–674.  https://doi.org/10.1111/nuf.12394
  * Federal Emergency Management Agency (2020) Federal Emergency Management Agency. 2020\.  _Homeland Security Exercise and Evaluation Program (HSEEP)_.  Technical Report. Federal Emergency Management Agency.  https://www.fema.gov/sites/default/files/2020-04/Homeland-Security-Exercise-and-Evaluation-Program-Doctrine-2020-Revision-2-2-25.pdf
  * Filigran (2024) Filigran. 2024.  _OpenEx Platform_.  Filigran.  Retrieved January 17, 2024 from https://github.com/OpenEx-Platform/openex
  * for Cybersecurity) (2023) ENISA (European Union Agency for Cybersecurity). 2023.  Cyber Europe.  Online, accessed January 17, 2024.  https://www.enisa.europa.eu/topics/training-and-exercises/cyber-exercises/cyber-europe-programme
  * Force (2020) CC2020 Task Force. 2020\.  _Computing Curricula 2020: Paradigms for Global Computing Education_.  Association for Computing Machinery, New York, NY, USA.  https://doi.org/10.1145/3467967
  * Frégeau et al. (2020) Amélie Frégeau, Alexis Cournoyer, Marc-André Maheu-Cadotte, Massimiliano Iseppon, Nathalie Soucy, Julie St-Cyr Bourque, Sylvie Cossette, Véronique Castonguay, and Richard Fleet. 2020\.  Use of tabletop exercises for healthcare education: a scoping review protocol.  _BMJ Open_ 10, 1 (2020).  https://doi.org/10.1136/bmjopen-2019-032662
  * Grance et al. (2006) Tim Grance, Tamara Nolan, Kristin Burke, Rich M. Dudley, Gregory C. White, and Travis Good. 2006\.  _Guide to Test, Training, and Exercise Programs for IT Plans and Capabilities: Recommendations of the National Institute of Standards and Technology_.  Technical Report. National Institute of Standards and Technology.  https://doi.org/10.6028/NIST.SP.800-84
  * Huff et al. (2023) Philip Huff, Sandra Leiterman, and Jan P. Springer. 2023\.  Cyber Arena: An Open-Source Solution for Scalable Cybersecurity Labs in the Cloud. In _Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1_ _(SIGCSE 2023)_. Association for Computing Machinery, New York, NY, USA, 221–227.  https://doi.org/10.1145/3545945.3569828
  * Inc. (2023) Emergency Solutions International Inc. 2023\.  _ESI Exercises_.  ESI.  Retrieved January 17, 2024 from https://esintl.ca/exercise-and-evaluation/
  * ISO (2013) ISO. 2013.  _Societal security — Guidelines for exercises_.  Standard. International Organization for Standardization, Geneva, CH.
  * Janssen and Vreugdenhil (2015) Astrid Janssen and Hanneke Vreugdenhil. 2015.  Objective oriented exercise evaluation with TARCK-it. In _12th Proceedings of the International Conference on Information Systems for Crisis Response and Management, Krystiansand, Norway, May 24-27, 2015_. ISCRAM Association, Kristiansand, Norway.  http://idl.iscram.org/files/astridjanssen/2015/1223_AstridJanssen+HannekeVreugdenhil2015.pdf
  * Kitchenham and Charters (2007) Barbara Kitchenham and Stuart Charters. 2007.  _Guidelines for performing Systematic Literature Reviews in Software Engineering_.  Technical Report. EBSE.
  * Kopustinskas et al. (2020) Vytis Kopustinskas, Bogdan Vamanu, Marcelo Masera, Rimantas Šikas, Julia Vainio, Romualdas Petkevičius, and Lawrence Walzer. 2020\.  Tabletop Exercise as a Tool to Foster Resilience in the Energy Sector: Lessons Learned in the Baltic States. In _Proceedings of the 30th European Safety and Reliability Conference and 15th Probabilistic Safety Assessment and Management Conference_. Research Publishing Services, Italy, 255–262.  https://doi.org/10.3850/978-981-14-8593-0_4019-cd
  * Krippendorff (2004) Klaus Krippendorff. 2004\.  Reliability in content analysis: Some common misconceptions and recommendations.  _Human communication research_ 30, 3 (2004), 411–433.  https://doi.org/10.1111/j.1468-2958.2004.tb00738.x
  * Kussmaul (2012) Clifton Kussmaul. 2012\.  Process Oriented Guided Inquiry Learning (POGIL) for Computer Science. In _Proceedings of the 43rd ACM Technical Symposium on Computer Science Education_ _(SIGCSE ’12)_. Association for Computing Machinery, New York, NY, USA, 373–378.  https://doi.org/10.1145/2157136.2157246
  * Labs (2023) Immersive Labs. 2023\.  _Cyber Crisis Simulator_.  Immersive Labs.  Retrieved January 17, 2024 from https://www.immersivelabs.com/platform/cyber-crisis-simulator/
  * Line and Moe (2015) Maria B. Line and Nils B. Moe. 2015.  Understanding Collaborative Challenges in IT Security Preparedness Exercises. In _ICT Systems Security and Privacy Protection_. Springer, Cham, 311–324.  https://doi.org/10.1007/978-3-319-18467-8_21
  * Ltd. (2023) Micro Minder Ltd. 2023\.  _Cybersecurity Tabletop Exercise Services_.  MCS.  Retrieved January 17, 2024 from https://www.micromindercs.com/cybersecuritytabletopexercise
  * Mahdi et al. (2023) Syed S. Mahdi, Hafsa A. Jafri, Raheel Allana, Gopi Battineni, Mariam Khawaja, Syeda Sakina, Daniyal Agha, Kiran Rehman, and Francesco Amenta. 2023.  Systematic review on the current state of disaster preparation Simulation Exercises (SimEx).  _BMC Emergency Medicine_ 23, 1 (2023), 52\.  https://doi.org/10.1186/s12873-023-00824-8
  * Makrodimitris and Douligeris (2015) Georgios Makrodimitris and Christos Douligeris. 2015.  Towards a Successful Exercise Implementation – A Case Study of Exercise Methodologies. In _Human Aspects of Information Security, Privacy, and Trust_. Springer International Publishing, Cham, 207–218.  https://doi.org/10.1007/978-3-319-20376-8_19
  * Marshall (2009) Jim Marshall. 2009\.  The Cyber Scenario Modeling and Reporting Tool (CyberSMART). In _2009 Cybersecurity Applications & Technology Conference for Homeland Security_, Vol. 1. IEEE, USA, 305–309.  https://doi.org/10.1109/CATCH.2009.46
  * NATO Cooperative Cyber Defence Centre of Excellence (2023) NATO Cooperative Cyber Defence Centre of Excellence. 2023.  Locked Shields.  Online, accessed January 17, 2024.  https://ccdcoe.org/exercises/locked-shields
  * Natural Language Toolkit Project(2023) (NLTK) Natural Language Toolkit (NLTK) Project. 2023\.  Source code for nltk.metrics.agreement.  Online, accessed January 17, 2024.  http://www.nltk.org/_modules/nltk/metrics/agreement.html
  * Ota et al. (2022) Yuitaka Ota, Erika Mizuno, Tomomi Aoyama, Yoshihiro Hashimoto, Ichiro Koshijima, Haruna Asai, and Shiho Taniuchi. 2022\.  Designing Framework for Tabletop Exercise to Promote Resilience Against Cyber Attacks.  In _14th International Symposium on Process Systems Engineering_. Elsevier, Japan, 1471–1476.  https://doi.org/10.1016/B978-0-323-85159-6.50245-1
  * Ottis (2014) Rain Ottis. 2014\.  Light Weight Tabletop Exercise for Cybersecurity Education.  _Journal of Homeland Security and Emergency Management_ 11 (12 2014), 579–592.  Issue 4. https://doi.org/10.1515/jhsem-2014-0031
  * Petersen et al. (2008) Kai Petersen, Robert Feldt, Shahid Mujtaba, and Michael Mattsson. 2008.  Systematic Mapping Studies in Software Engineering. In _Proceedings of the 12th International Conference on Evaluation and Assessment in Software Engineering_ (Italy) _(EASE’08)_. BCS Learning & Development Ltd., Swindon, UK, 68–77.  https://doi.org/10.14236/ewic/EASE2008.8
  * Petersen et al. (2015) Kai Petersen, Sairam Vakkalanka, and Ludwik Kuzniarz. 2015\.  Guidelines for conducting systematic mapping studies in software engineering: An update.  _Information and Software Technology_ 64 (2015), 1 – 18.  https://doi.org/10.1016/j.infsof.2015.03.007
  * PreparedEx (2023) LLC PreparedEx. 2023\.  _PreparedEx Services_.  PreparedEx, LLC.  Retrieved January 17, 2024 from https://preparedex.com/services/
  * RedLegg (2023) RedLegg. 2023.  _RedLegg TTX_.  GLW Specialty.  Retrieved January 17, 2024 from https://www.redlegg.com/advisory-services/tabletop-exercise
  * Richards, Chet (2020) Richards, Chet. 2020\.  Boyd’s OODA loop.  _Necesse_ 5 (2020), 142–165.  https://hdl.handle.net/11250/2683228
  * Sanders et al. (2017) Kate Sanders, Jonas Boustedt, Anna Eckerdal, Robert McCartney, and Carol Zander. 2017\.  Folk Pedagogy: Nobody Doesn’t Like Active Learning. In _Proceedings of the 2017 ACM Conference on International Computing Education Research_ (Tacoma, Washington, USA) _(ICER ’17)_. Association for Computing Machinery, New York, NY, USA, 145–154.  https://doi.org/10.1145/3105726.3106192
  * Skytterholm and Hotvedt (2023) Andrea Skytterholm and Guro Hotvedt. 2023.  Criteria for Realistic and Expedient Scenarios for Tabletop Exercises on Cyber Attacks Against Industrial Control Systems in the Petroleum Industry. In _Proceedings of the International Conference on Cybersecurity, Situational Awareness and Social Media_. Springer Nature, Singapore, 39–54.  https://doi.org/10.1007/978-981-19-6414-5_3
  * Taniuchi et al. (2019) Shiho Taniuchi, Tomomi Aoyama, Haruna Asai, and Ichiro Koshijima. 2019.  Training Cyber Security Exercise Facilitator: Behavior Modeling Based on Human Error. In _Advances in Human Factors in Cybersecurity_. Springer International Publishing, Cham, 138–148.  https://doi.org/10.1007/978-3-319-94782-2_14
  * Švábenský et al. (2022) Valdemar Švábenský, Jan Vykopal, Pavel Čeleda, and Lydia Kraus. 2022.  Applications of educational data mining and learning analytics on data from cybersecurity training.  _Education and Information Technologies_ 27, 9 (2022), 12179–12212.  https://doi.org/10.1007/s10639-022-11093-6
  * Švábenský et al. (2024) Valdemar Švábenský, Jan Vykopal, Martin Horák, Martin Hofbauer, and Pavel Čeleda. 2024\.  From Paper to Platform: Evolution of a Novel Learning Environment for Tabletop Exercises. In _Proceedings of the 29th Conference on Innovation and Technology in Computer Science Education_ _(ITiCSE ’24)_. Association for Computing Machinery, New York, NY, USA, 7 pages.  https://doi.org/10.1145/3649217.3653639
  * Vykopal et al. (2024) Jan Vykopal, Pavel Čeleda, Valdemar Švábenský, Martin Hofbauer, and Martin Horák. 2024\.  Dataset: Research and Practice of Delivering Tabletop Exercises.  https://gitlab.fi.muni.cz/inject/papers/2024-iticse-research-practice.
  * Vykopal et al. (2017) Jan Vykopal, Martin Vizvary, Radek Oslejsek, Pavel Celeda, and Daniel Tovarnak. 2017.  Lessons Learned From Complex Hands-on Defence Exercises in a Cyber Range. In _2017 IEEE Frontiers in Education Conference (FIE)_. IEEE, Indianapolis, USA, 1–8.  https://doi.org/10.1109/FIE.2017.8190713
  * Yamin et al. (2020) Muhammad Mudassar Yamin, Basel Katt, and Vasileios Gkioulos. 2020\.  Cyber ranges and security testbeds: Scenarios, functions, tools and architecture.  _Computers & Security_ 88 (2020), 101636.  https://doi.org/10.1016/j.cose.2019.101636
  * Zacharis and Patsakis (2023) Alexandros Zacharis and Constantinos Patsakis. 2023.  AiCEF: an AI-assisted cyber exercise content generation framework using named entity recognition.  _International Journal of Information Security_ 22, 5 (2023), 1333–1354.  https://doi.org/10.1007/s10207-023-00693-z
  * Østby et al. (2019) Grethe Østby, Kieren N. Lovell, and Basel Katt. 2019\.  EXCON Teams in Cyber Security Training. In _2019 International Conference on Computational Science and Computational Intelligence (CSCI)_. IEEE, Las Vegas, USA, 14–19.  https://doi.org/10.1109/CSCI49370.2019.00010
  * ™ (2023) Conducttr ™. 2023\.  Crisis simulation platform.  Retrieved January 17, 2024 from https://www.conducttr.com/