# Benchmarking Actor-Critic Deep Reinforcement Learning Algorithms for
Robotics Control with Action Constraints

Kazumi Kasaura∗1, Shuwa Miura∗2, Tadashi Kozuno1, Ryo Yonetani1, Kenta
Hoshino3, Yohei Hosoe4 1 KK, TK, and RY are with OMRON SINIC X Corporation,
Hongo, Bunkyo-ku, Tokyo, Japan. {kazumi.kasaura, tadashi.kozuno,
<EMAIL_ADDRESS>2 SM is with Manning College of Information and
Computer Sciences University of Massachusetts Amherst, Amherst, Massachusetts.
This work was done while he was a research intern at OMRON SINIC X
Corporation<EMAIL_ADDRESS>3 KH is with Graduate School of Informatics,
Kyoto University, Sakyo-ku, Kyoto, Japan<EMAIL_ADDRESS>4 YH is
with Graduate School of Engineering, Kyoto University, Nishikyo-ku, Kyoto,
Japan<EMAIL_ADDRESS>∗Equal contribution.

###### Abstract

This study presents a benchmark for evaluating action-constrained
reinforcement learning (RL) algorithms. In action-constrained RL, each action
taken by the learning system must comply with certain constraints. These
constraints are crucial for ensuring the feasibility and safety of actions in
real-world systems. We evaluate existing algorithms and their novel variants
across multiple robotics control environments, encompassing multiple action
constraint types. Our evaluation provides the first in-depth perspective of
the field, revealing surprising insights, including the effectiveness of a
straightforward baseline approach. The benchmark problems and associated code
utilized in our experiments are made available online at github.com/omron-
sinicx/action-constrained-RL-benchmark for further research and development.

###### Index Terms:

reinforcement learning, action constraints, safety

##  I Introduction

Action-constrained reinforcement learning (RL) imposes explicit constraints on
actions taken by the learning system. These constraints can come from a
variety of sources, such as physical limitations of robots (e.g., torque or
power limits) [1] and safety considerations. In manufacturing applications,
for example, action constraints can prevent robot arms from hitting obstacles
[2] or moving beyond designated boundaries [3]. Other examples include
collision avoidance for autonomous vehicles [4] and maintenance for energy-
efficient building operations [5]. For real systems, it is important that each
action should satisfy constraints throughout the training process, not just at
the end, to guarantee its feasibility [2].

Many works have explored how to introduce action constraints to existing deep
RL algorithms [2, 6, 7, 5, 8]. Deep RL algorithms, which use neural network
policies, have achieved success in various continuous control tasks [9, 10].
However, without explicit constraints, agents can try to execute infeasible
actions in the environment. Existing work has addressed this problem by
projecting the outputs of policies onto feasible actions, ensuring that
constraints are satisfied before actions are executed. Examples of algorithms
that adopt this approach encompass differentiable optimization layers [2, 6,
4, 7, 11, 5], Neural Frank-Wolfe Policy Optimization (NFWPO) [8], and
$\alpha$-projection [12]. Despite the growing interest in action-constrained
RL, a comprehensive comparison of these algorithms has not been conducted.

In this paper, we present the first benchmark study for the existing action-
constrained RL algorithms. Our primary contribution is the evaluation of
action-constrained RL algorithms in terms of learning performance and
computational time requirements, laying the foundation for future research in
this domain. To this end, our evaluation uses various simulated, and thus
easy-to-reproduce, robotics control tasks from MuJoCo [13] and PyBullet-Gym
[14] on OpenAI gym [15] and evaluate existing work with several different
types of action constraints. Our evaluation centers on off-policy actor-critic
deep RL algorithms, such as Deep Deterministic Policy Gradients (DDPG) [9] and
Soft Actor-Critic (SAC) [10], due to their superior sample efficiency. Actor-
critic algorithms [16] involve training both an actor and a critic. The actor
adjusts policy parameters using the policy gradient, while the critic learns
the Q-function.

In addition to evaluating existing action-constrained RL algorithms, we also
introduce several variants of these algorithms in our evaluation. One of these
variants is a simple method that treats action constraints as part of the
state transition of the environment, serving as a baseline for comparison with
other approaches to action constraints. We also propose an alternative action
mapping method called radial squashing, which can be seen as a natural variant
of $\alpha$-projection. In our evaluation, we use a variant of DDPG called
Twin Delayed DDPG (TD3) [17] and SAC as the base deep RL algorithms. DDPG
variants are common choices for this type of problem [7, 8], while the use of
action constraints with SAC has not been previously explored and requires
nontrivial modifications to the original algorithm. We describe how to
incorporate action constraints into SAC.

Our experimental results suggest that a simple approach that trains the critic
with pre-projected actions is empirically shown to achieve performance
comparable to that of specialized algorithms. Moreover, the results also show
that alternative mapping techniques ($\alpha$-projection and radial squashing)
can achieve competitive performance compared to the existing algorithms, while
requiring less computation time. On the other hand, the results show that the
use of differentiable optimization layers, a common method for action-
constrained RL, does not outperform the simple baseline and can take a long
time to run.

##  II Background

###  II-A Action-Constrained Reinforcement Learning

We consider an RL problem, where an agent interacts with its environment
modeled by a Markov decision process (MDP). An MDP is represented by a tuple
$M=\langle\mathcal{S},\mathcal{A},T,d_{0},d_{R},\gamma\rangle$, where
$\mathcal{S}$ and $\mathcal{A}$ in this work are continuous state and action
spaces, respectively. $T$ is a conditional density function describing the
dynamics of the environment ($S_{t+1}\sim T(S_{t},A_{t})$), and $d_{0}$ is an
initial state distribution. $d_{R}$ describes how rewards are generated
($R_{t}\sim d_{R}(S_{t},A_{t},S_{t+1}))$, and $\gamma$ is a discount factor.
Given the current state, an agent determines what action to take based on its
_policy_ parameterized by $\theta$, which is either deterministic (represented
by $\mu_{\theta}$; mapping a state to an action) or stochastic
($\pi_{\theta}$; mapping the state to the probability distribution on
$\mathcal{A}$). The goal of RL is to find a policy that maximizes the expected
discounted return
$J(\pi_{\theta})=\mathbb{E}[\sum_{t=0}^{\infty}\gamma^{t}R_{t}|d_{0},\pi_{\theta}]$.
The _action value function_ or _Q-function_ for policy $\pi$ is defined as
$Q^{\pi}(s,a)=\mathbb{E}[\sum_{t=0}^{\infty}\gamma^{t}R_{t}|S_{0}=s,A_{0}=a,\pi]$.

In this paper, we consider action-constrained RL problems, where for each
state $s\in S$, there is a feasible set of actions
$\mathcal{A}_{s}\subseteq\mathcal{A}$. As in most previous work, we assume
$\mathcal{A}_{s}$ to be known apriori and characterized by linear [2, 7] or
convex constraints [8, 5]. Note that in action-constrained RL, actions are
required to satisfy constraints throughout training.111The exception can be
found in [18], where constraints are not enforced during training. Action
constraints can arise from a variety of sources such as physical limitations
of robots [1]. Such constraints can also be used to guarantee the safety of
the system. For example, several previous studies have used Control Barrier
Functions (CBF) [19] as action constraints [4, 20].

#### Box Constraints

Existing deep RL algorithms such as DDPG and SAC, by default, are limited to
handle a specific type of action constraints called _box constraints_ , which
have the form $a_{i}\in[-a_{i}^{\max},a_{i}^{\max}]$ for each $i$th dimension
of the action space. Input limits like box constraints are almost ubiquitous
in continuous control tasks as used in all control tasks from MuJoCo[13] in
OpenAI gym [15]. Existing implementations of deep RL such as [21] commonly
handle such box constraints using a hyperbolic tangent function
($\tanh(x)=\frac{e^{x}-e^{-x}}{e^{x}+e^{-x}}$) as the final activation layer
of policies, which we refer to as _squashing_. As the outputs of $\tanh$ range
from $-1$ to $1$, we can enforce the box constraints by scaling the outputs by
$a_{i}^{\max}$. While squashing is simple and effective, it cannot handle more
complex constraints such as upper-bounds on the weighted sum of $a_{i}$.

###  II-B Policy Gradient Algorithms

Policy gradient algorithms are widely used for continuous control tasks. These
algorithms adjust policy parameters $\theta$ using the gradient estimate of
the objective function $J(\pi_{\theta})$. For deterministic policies
$\mu_{\theta}$, the deterministic policy gradient [22] is given by:

$\displaystyle\textstyle\nabla_{\theta}J(\mu_{\theta})$
$\displaystyle=\mathbb{E}_{s\sim\rho^{\mu_{\theta}}}[\nabla_{a}Q^{\mu_{\theta}}(s,a)|_{a=\mu_{\theta}(s)}\nabla_{\theta}\mu_{\theta}(s)],$
(1)

where $\rho^{\mu_{\theta}}$ is a discounted state distribution under
$\mu_{\theta}$.

Deep Deterministic Policy Gradient (DDPG) [9] is a model-free actor-critic
algorithm that combines the deterministic policy gradients with function
approximation using neural networks. DDPG uses off-policy data (replay buffer)
to train the critic ($Q_{w}$ where $w$ represents the weights) and uses the
critic to train the actor ($\mu_{\theta}$). For each policy update step, a
minibatch of transitions ($\\{s_{i},a_{i},r_{i},s^{\prime}_{i}\\}_{i=1}^{N}$)
is sampled from the replay buffer. Then DDPG updates its critic in the
following direction:

$\textstyle\nabla_{w}\frac{1}{N}\sum_{i=1}^{N}(Q_{w}(s_{i},a_{i})-y(s^{\prime}_{i},r_{i}))^{2}$
(2)

where $y(r_{i},s_{i})=r_{i}+\gamma
Q_{w^{\prime}}(s^{\prime}_{i},\mu_{\theta^{\prime}}(s^{\prime}_{i}))$. The
target networks, represented by $Q_{w^{\prime}}$ and $\mu_{\theta^{\prime}}$,
have distinct parameters $w^{\prime}$ and $\theta^{\prime}$ and serve to
stabilize the training process. DDPG next updates its actor using the
following estimate for the policy gradient:

$\textstyle\nabla_{\theta}J(\mu_{\theta})\approx\frac{1}{N}\sum_{i=1}^{N}\nabla_{a}Q_{w}(s_{i},a)|_{a={\mu_{\theta}(s_{i})}}\nabla_{\theta}\mu_{\theta}(s_{i})$
(3)

As described above, DDPG handles box constraints via squashing.

Twin Delayed DDPG (TD3) [17] is a refined version of DDPG that addresses the
approximation errors often encountered in DDPG. TD3 introduces several
improvements over the original DDPG algorithm. One notable enhancement is the
_clipped double-Q_ trick, which employs the minimum value of two Q-function
outputs to decrease overestimation bias, thereby enhancing the stability and
performance of the algorithm.

Soft Actor Critic (SAC) [10] is a model-free actor-critic algorithm that
optimizes stochastic policies with the following entropy-regularized
objective:

$\textstyle\mathbb{E}[\sum_{t=0}^{\infty}\gamma^{t}R_{t}+\alpha
H(\pi(\cdot|S_{t}))|d_{0},\pi]$ (4)

where $H$ is the entropy and $\alpha>0$ is an entropy coefficient. The entropy
regularization term in the SAC algorithm encourages exploration during
training by promoting a more stochastic policy. The critic in SAC learns the
Q-function incorporating the entropy term (the soft Q-function), which is
trained using the clipped double-Q trick as in TD3. The actor is then updated
using:

$\textstyle\nabla_{\theta}\frac{1}{N}\sum_{i=1}^{N}\min_{j=1,2}Q_{w_{j}}(s_{i},\tilde{a}_{\theta}(s_{i}))-\alpha\pi_{\theta}(\tilde{a}_{\theta}(s_{i})|s_{i})$
(5)

where $\tilde{a}_{\theta}(s_{i})$ is sampled from a squashed Gaussian policy
$\pi_{\theta}(\cdot|s_{i})$ using the _reparametrization trick_ [10].

(a) Closest point projection

(b) $\alpha$-projection

(c) Radial squashing

Figure 1: Illustrations of different mappings used in the paper. $a$ and
$a^{\prime}$ are actions before mappings. $P_{s}(a)$ and $P_{s}(a^{\prime})$
are the corresponding actions after mappings. $\mathcal{A}_{s}$ is the set of
feasible actions. $c_{s}$ is the Chebyshev center of $\mathcal{A}_{s}$.

Figure 2: Illustration of the volume element $\differential{\sigma}$ of
$\partial\mathcal{A}_{s}$ and the differential solid angle
$\differential{\Omega}$ corresponding to it

##  III Algorithms for Action-Constrained RL

In this section, we overview algorithms for action-constrained RL. To handle
non-trivial constraints, most methods such as differentiable optimization
layers [2, 6, 4, 7, 5], NFWPO [8], and $\alpha$-projection [12], use a mapping
to feasible action sets
($P:\mathcal{S}\times\mathcal{A}\rightarrow\mathcal{A}_{s}$). 222Note that
mappings discussed here are on actions. This contrasts with projected gradient
algorithms, where the parameters ($\theta$) are projected to satisfy the
constraints. With neural network policies, projecting $\theta$ so that
$\mu_{\theta}$ satisfies constraints in every continuous state, in general, is
not trivial. This is because the outputs of policies before the mapping,
$\mu_{\theta}(s)$, do not necessarily satisfy action constraints. Then, rather
than the original outputs, the corresponding feasible actions after the
mapping, $P(s,\mu_{\theta}(s))$ or also denoted as $P_{s}(\mu_{\theta}(s))$,
are executed in the environment.

In what follows, we first present existing algorithms that use the closest
projection to feasible actions (Sec. III-A). Then we describe other algorithms
that use different mappings in Sec. III-B. Unless specified otherwise, our
discussion in this section assumes the use of DDPG.

###  III-A Algorithms Based on the Closest Point Projection

In this section, we describe a family of algorithms based on the _closest
point projection_ , i.e., projecting the outputs of policies to the closest
feasible actions typically in terms of the Euclidean norm. The closest point
projection $P_{s}$ is defined by the following optimization problem:

$\displaystyle
P_{s}(\mu_{\theta}(s))=\operatorname*{arg\,min}_{a\in\mathcal{A}_{s}}||a-\mu_{\theta}(s)||_{2}.$
(6)

The problem is quadratic programming for linear constraints and convex
programming for convex constraints. The following algorithms differ in how the
actor and critic are updated in the presence of the projection.

####  III-A1 Training Critic with Projected Actions

A baseline algorithm in some previous work uses the closest point projection
and trains the critic with the projected actions [2, 4, 7, 8]. Namely, it
trains the critic $Q_{w}$ in Eq. (2) with $a_{i}=P_{s}(\mu_{\theta}(s_{i}))$.
The algorithm, however, keeps the actor’s update rule unchanged. In the case
of DDPG, the actor is updated in the direction of
$\nabla_{\theta}Q_{w}(s_{i},\mu_{\theta}(s_{i}))$ (Eq. (3)). Intuitively, the
actor is updated obliviously to the projection.

As previous work [7] points out, however, updating the actor in
$\nabla_{\theta}Q_{w}(s_{i},\mu_{\theta}(s_{i}))$ is not theoretically
principled. This is because Q-values for pre-projected actions
($Q^{\mu}(s,\mu_{\theta}(s_{i}))$ are ill-defined when $\mu_{\theta}(s_{i})$
are infeasible. The approach also has a practical issue. Since the critic is
only trained with projected actions ($P_{s}(\mu_{\theta}(s_{i}))$), the critic
is unlikely to learn action values for pre-projected actions
($\mu_{\theta}(s_{i})$).

The problem is that, since a parameterized policy $\mu_{\theta}$ is now
combined with the projection $P$, policy gradients should be computed for
$\nabla_{\theta}J(P_{s}\circ\mu_{\theta}$) instead of for
$\nabla_{\theta}J(\mu_{\theta})$, as:

$\displaystyle\mathbb{E}_{s\sim\rho^{P\circ\mu}}[\nabla_{a}Q^{P\circ\mu}(s,a)|_{a=P_{s}(\mu_{\theta}(s))}\nabla_{\theta}P_{s}\circ\mu_{\theta}(s)]$
(7)
$\displaystyle=\mathbb{E}_{s\sim\rho^{P\circ\mu}}[\nabla_{a}Q^{P\circ\mu}(s,a)|_{a=P_{s}(\mu_{\theta}(s))}\frac{dP_{s}}{d\mu_{\theta}(s)}\nabla_{\theta}\mu_{\theta}(s)]$
(8)

where $\frac{dP_{s}}{d\mu_{\theta}(s)}$ is the Jacobian of the projection
$P_{s}$ with respect to suggested actions.

####  III-A2 Differentiable Optimization Layer

To compute estimates for the policy gradient with projections in Eq. (8), most
previous work has used _differentiable optimization layers_ as the final layer
of the policy network [2, 6, 7, 5]. OptLayer [2] combined a differentiable
quadratic programming (QP) layer [23] with deep RL to handle linear
constraints. PROF [5] used differentiable convex optimization layers [24] to
handle a class of convex programming called disciplined convex programming.
Differentiable optimization layers enable the calculation of projection
gradients with respect to suggested actions $\frac{dP_{s}}{d\mu_{\theta}(s)}$,
which changes the actor’s update to:

$\textstyle\frac{1}{N}\sum_{i=1}^{N}\nabla_{a}Q_{w}(s_{i},a)|_{a={P_{s}(\mu_{\theta}(s_{i}))}}\frac{dP_{s}}{d\mu_{\theta}(s)}\nabla_{\theta}\mu_{\theta}(s_{i}).$
(9)

Note that the actor is now updated in a way that incorporates the effect of
the projection, i.e., $\frac{dP_{s}}{d\mu_{\theta}(s)}$.

Existing methods using differentiable optimization layers share some common
challenges. Unlike the original actor update in Eq. (3), the new update rule
in Eq. (9) additionally involves projections ($P_{s}$) and the Jacobian
($\frac{dP_{s}}{d\mu_{\theta}(s)}$), which incurs non-negligible computational
overhead to processing each state in a mini-batch. More importantly, closest
point projection with differentiable optimization layers can suffer from the
_zero-gradient issue_. That is, since many actions initially violating
constraints can be projected to the same action, the gradients with respect to
$\theta$ tend to vanish when $\mu_{\theta}$ proposes an action violating
constraints. For example, in Fig. 1(a), both $a=\mu_{\theta}(s)$ and
$a^{\prime}=\mu_{\theta}(s)^{\prime}$ are projected to the same action. Since
the gradient of the projection is zero at $s$, the gradient of the policy
$\frac{dP_{s}}{d\mu_{\theta}(s)}\nabla_{\theta}\mu_{\theta}(s)$ also vanishes
at $s$. As a result, the actor ends up completely wasting the sample. To
alleviate the zero-gradient issue, some methods introduce penalty terms for
constraint violations [2, 5].

####  III-A3 Neural Frank-Wolfe Policy Optimization (NFWPO)

To overcome the zero-gradient issue, NFWPO [8] proposes to decouple the
projection from actor updates. This is in contrast to previous methods using
differentiable optimization layers, where $\frac{dP_{s}}{d\mu_{\theta}(s)}$ is
a part of the actor update in Eq. (9). For training the critic $Q_{w}$, NFWPO
is identical to DDPG, and uses Eq.(2) with projected actions. Regarding the
actor training, NFWPO initially determines a reference action
$a_{s}\in\mathcal{A}_{s}$ for each state $s\in S$ in a mini-batch via the
Frank-Wolfe algorithm [25]:

$\textstyle a_{s}=P_{s}(\mu_{\theta}(s)+\alpha(c_{s}-P_{s}(\mu_{\theta}(s))))$
(10)

Here, $\alpha$ represents the Frank-Wolfe learning rate, and

$\textstyle c_{s}=\operatorname*{arg\,max}_{c\in\mathcal{A}_{s}}\langle
c,\nabla_{a}Q_{w}(s,a)|_{a=P_{s}(\mu_{\theta}(s))}\rangle$ (11)

Subsequently, the policy parameters $\theta$ are updated to minimize the
distance between $\mu_{\theta}(s)$ and the reference action $a_{s}$. By doing
so, NFWPO avoids computing $\frac{dP_{s}}{d\mu_{\theta}(s)}$ and policy
gradients, thereby overcoming the zero-gradient issue. Without any action
constraints, NFWPO is equivalent to DDPG [8].

####  III-A4 Training Critic with Pre-Projected Actions

In addition to the existing methods above, we consider another simple, yet
surprisingly effective baseline that trains the actor and the critic with
_pre-projected_ , possibly infeasible actions $\mu_{\theta}(s)$, while the
projection of the actions to feasible ones is done as a part of state
transitions of the environment. In this method, the critic learns
$\tilde{Q}^{\mu}=Q^{\mu}\circ P$ instead of $Q^{\mu}$, as it is defined even
for infeasible actions. In other words, the critic now learns action-values
for suggested actions, given that suggested actions might be projected to
feasible actions. Then, the actor is updated using Eq. (3). This approach is
computationally more efficient than other techniques using differentiable
optimization or NFWPO, since the projection is performed only during rollouts.
Moreover, this approach can be easily combined with other RL algorithms,
including SAC.

Note that this baseline has several known limitations. As actions are
projected to feasible ones as part of state transitions, agents must learn the
effects of projection through experienced transitions and rewards. Moreover,
this approach can suffer from the zero-gradient issue and may require
additional penalty terms to alleviate the problem.

###  III-B Other Mapping Techniques

In this section, we present several other differentiable mapping techniques.
Similar to using differentiable optimization layers (Sec III-A2), the
algorithms in this section employ differentiable mappings to feasible actions
as the final layers of policies. However, the techniques presented here do not
necessarily map actions to the closest feasible ones. The key insight is that
since RL algorithms can learn to adjust their suggestions to achieve better
performance, there is no inherent need for using the closest point projection.

####  III-B1 $\alpha$-Projection Layer

$\alpha$-projection [12] can be used as an alternative to the closet point
projection. $\alpha$-projection assigns an interior point
$c_{s}\in\mathrm{Int}(\mathcal{A}_{s})$ for each state $s\in\mathcal{S}$. As
Fig. 1(b) shows, given a suggested action $a\in\mathcal{A}$,
$\alpha$-projection moves $a$ toward $c_{s}$ until it reaches
$\mathcal{A}_{s}$. Formally, it picks an action on the ray from $c_{s}$ to $a$
that is closest to $a$ (each $a\in\mathcal{A}_{s}$ is mapped to itself). Since
points on different rays are mapped to different points, the gradient does not
vanish in the example.

As originally studied in [12], for linear-inequality constraints, a closed-
form expression exists for $\alpha$-projection and can be implemented as an
additional neural network layer. The actor is then updated using Eq. (9),
where $P$ is replaced with $\alpha$-projection. It is also possible to extend
the $\alpha$-projection to problems with an elliptical constraint, which has
the form $(a-c(s))^{\mathsf{T}}Q(s)(a-c(s))-b(s)$ where $Q(s)$ is a positive-
semi-definite matrix.

Note that $\alpha$-projection assumes the existence of an interior point
$c_{s}\in\mathrm{Int}(\mathcal{A}_{s})$ such that, for any point
$x\in\mathcal{A}_{s}$, the segment between $c_{s}$ and $x$ is contained in
$\mathcal{A}_{s}$. For linear constraints, we use the Chebychev center of
$\mathcal{A}_{s}$ as $c_{s}$ as in [12]. During training of the actor,
$\alpha$-projection must be performed for each state in a minibatch. Computing
$c_{s}$ for each state can be time-consuming, so we store the previously
computed Chebyshev centers in the replay buffer and reuse them during
training. For problems with an elliptical constraint, we use $c(s)$ as
$c_{s}$.

####  III-B2 Radial Squashing Layer

We also propose an alternative mapping to feasible actions called _radial
squashing_ , which can also be implemented as a neural network layer. Instead
of clipping the ray from $c_{s}$ as in $\alpha$-projection, radial squashing
shrinks actions into the feasible set of actions around its center $c_{s}$, as
illustrated in Fig. 1(c). Let $a$ be the given point and $b$ be the
intersection of the boundary of $\mathcal{A}_{s}$ and the ray from $c_{s}$ to
$a$. The radial squashing layer maps:

$a\mapsto c_{s}+\tanh\left(\|a-c_{s}\|/\|b-c_{s}\|\right)(b-c_{s}).$ (12)

Note that, this mapping is differentiable at $c_{s}$ (the Jacobian becomes the
identity matrix there) and gradients never vanish for any direction.

###  III-C Incorporating Action Constraints to SAC

Although SAC has been a popular algorithm for general RL problems and can be
easily combined with the approach described in III-A4, combining it with
mapping techniques for action-constrained RL is not trivial. This is mainly
due to the entropy term in Eq. (5) that requires the calculation of the
probability density of a policy after actions are mapped to feasible ones. In
general, for a random variable $X$ with a probability density function $p$,
the probability density function $q$ of $f(X)$ after a differentiable mapping
$f$ is given by:

$q(f(X))=|\det J_{f}(X)|^{-1}p(X),$ (13)

when $\det J_{f}(X)\neq 0$, where $J_{f}(X)$ is the Jacobian of $f$.

####  III-C1 Radial Squashing with SAC

Combining radial squashing with SAC is straightforward since the determinant
of Jacobian never vanishes. To evaluate the change in probability density by
the radial squashing, it is enough to calculate the Jacobian of Eq. (12) as
follows:

$(a-c_{s})\left(\mathrm{grad}\,\frac{\tanh
L}{L}\right)^{\mathsf{T}}+\frac{\tanh L}{L}I,$ (14)

where $I$ is the identity matrix and $L:=\|a-c_{s}\|/\|b-c_{s}\|$ is a
function of $a$ that depends on the constraint.

####  III-C2 $\alpha$-projection with SAC

On the other hand, combining the $\alpha$-projection layer with SAC is non-
trivial because the probability density after projection may become infinity
(i.e., the determinant of Jacobian may be zero.) To overcome this difficulty,
we consider two probability density functions for the policy after projection:
a $d$-dimensional density function on the interior of $\mathcal{A}_{s}$ and a
$(d-1)$-dimensional density function on the boundary of $\mathcal{A}_{s}$,
where $d$ is the dimension of $\mathcal{A}_{s}$. The entropy of the
distribution is defined by the sum of the entropies for these distributions on
the interior and boundary. Mathematically, we define the measure of
$\mathcal{A}_{s}$ as the sum333The weight of the entropy (or the measure) for
the boundary in this sum is arbitrary and not scale-invariant. It can be
considered as a hyperparameter of this algorithm. In this paper, we simply
take one. Note that the action space is scaled in the implementation of these
algorithms. of the standard measure of the interior of $\mathcal{A}_{s}$ and
that of the boundary of $\mathcal{A}_{s}$ as a Riemannian submanifold of the
Euclidean space [26].

While the probability density for the interior of $\mathcal{A}_{s}$ remains
unchanged by the projection, we need to integrate the probability density
before projection for the boundary of $\mathcal{A}_{s}$. Let $p$ be the
probability density function before projection, and let $q$ be the desired
probability density function on the boundary after projection. We use the
spherical coordinate system with origin $c_{s}$. Let
$b\in\partial\mathcal{A}_{s}$ be the point on the boundary, and let
$\differential{\sigma}$ be the volume element of $\partial\mathcal{A}_{s}$ at
$b$. We must integrate $p$ on the solid cone projected to
$\differential{\sigma}$:

$q\differential{\sigma}=\int_{r_{0}}^{\infty}pr^{d-1}\differential{r}\differential{\Omega},$
(15)

where $r_{0}:=\|b-c_{s}\|$ and $\differential{\Omega}$ is the differential
solid angle corresponding to $\differential{\sigma}$ (see Fig. 2). This can be
calculated by
$\differential{\Omega}=r_{0}^{-1}\cos\theta\differential{\sigma}$, where
$\theta$ is the angle between the vector $b-c_{s}$ and the normal vector of
$\partial\mathcal{A}_{s}$ at $b$. Since $p$ is a Gaussian distribution for
SAC, it is enough to calculate an integration of form $\int
r^{d-1}\exp(-(Ar+B)^{2})\differential{r}$ after completing the square. We can
get its explicit expression using $\exp(-(Ar+B)^{2})$ and
$\mathrm{erf}(Ar+B)$, where $\mathrm{erf}$ is the error function, by applying
partial integration repeatedly.

####  III-C3 Remark on Differential Optimization Layer

Combining differentiable optimization layers with SAC, on the other hand, may
not be feasible. This is because closest point projection can map a region
with more than one dimension to one point, making it difficult to determine
the change in probability distribution.

###  III-D ConstraintNet

Another possible approach for handling action-constraints is to modify the
output layer of the actor network based on the type of constraints.
ConstraintNet [27] uses a convex combination of $n$ given vertices as the
output layer when the feasible set of actions is a convex polytope and the
vertices of the polytope are known. However, this approach has some
limitations: the number of vertices in a polytope can be exponential in the
dimension of the action, and it is not straightforward to extend this approach
to state-dependent constraints. Therefore, we did not include this approach in
our evaluations.

##  IV Experiments

In this section, we conduct an empirical evaluation of action-constrained RL
algorithms on various simulated control tasks from PyBullet-Gym [14] and
MuJoCo [13] in OpenAI Gym [15]. We additionally assess the running time of
each algorithm in Sec. IV-D.

###  IV-A Algorithms

We compare the following 13 algorithms in total.

  * •

TD3 family:

    * $\circ$

DPro: TD3 with critic trained using projected actions (Sec. III-A1)

    * $\circ$

DPro+: DPro with the penalty term (see below)

    * $\circ$

DPre: TD3 with pre-projected actions (Sec. III-A4)

    * $\circ$

DPre+: DPre with penalty term

    * $\circ$

DOpt: TD3 with optimization layer (Sec. III-A2)

    * $\circ$

DOpt+: DOpt with penalty term

    * $\circ$

NFW: NFWPO (Sec. III-A3) with TD3 techniques (clipped double Q learning,
target policy smoothing and delayed policy update)

    * $\circ$

DAlpha: TD3 with $\alpha$-projection (Sec. III-B1)

    * $\circ$

DRad: TD3 with radial squashing (Sec. III-B2)

  * •

SAC family:

    * $\circ$

SPre: SAC with pre-projected actions (Sec. III-A4)

    * $\circ$

SPre+: SPre with penalty term

    * $\circ$

SAlpha: SAC with $\alpha$-projection (Sec. III-C2)

    * $\circ$

SRad: SAC with radial squashing (Sec. III-C1)

DPro+, DPre+, DOpt+, and SPre+ introduce the penalty term for constraint
violations discussed in Sec. III-A2 or Sec. III-A4. Specifically, we added
$\|\max\\{Ax-b,\mathbf{0}\\}\|$ for linear constraints $Ax\leq b$, where $A$
is normalized so that the norm of each row is $1$, as in [2]. For elliptical
constraint $(x-c)^{\mathsf{T}}Q(x-c)\leq b$,
$\max\\{\sqrt{(x-c)^{\mathsf{T}}Q(x-c)}-\sqrt{b},0\\}$ was added where $Q$ is
normalized so that $\tr Q$ equals to the dimension. In case there exist both
constraints, we take the sum of two penalties.

Hyperparameters | Reacher | Hopper | Others
---|---|---|---
| TD3 | SAC | TD3 | SAC | TD3 | SAC
Discount factor | 0.98 | 0.99 | 0.99
Net. arch. 1st | 400 | 400 | 400 | 256 | 400 | 256
Net. arch. 2nd | 300 | 300 | 300 | 256 | 300 | 256
Batch size | 100 | 256 | 256 | 256 | 100 | 256
Learning rate | 1e-3 | 7.3e-4 | 3e-4 | 3e-4 | 1e-3 | 3e-4
Buffer size | 2e5 | 3e5 | 1e6 | 1e6 | 1e6 | 1e6
Target Update Ratio | 0.005 | 0.02 | 0.005 | 0.005 | 0.005 | 0.005
Action noise | 0.1 | - | 0.1 | - | 0.1 | -
FW learning rate | 0.05 | - | 0.01 | - | 0.01 | -
Learning starts | 1e5 | 1e5 | 1e5 | 1e5 | 1e5 | 1e5
Use SDE | - | True | - | False | - | False
Train freq. | 1 epi. | 8 steps | 1 step | 1 step | 1 epi. | 1 step
Gradient steps | -1 | 8 | 1 | 1 | -1 | 1

TABLE I: Hyperparameters used in the experiments.

###  IV-B Implementation Details

We adapted the implementations of TD3 and SAC in Stable Baselines 3 (SB3)
[21], a popular reinforcement learning library using PyTorch [28]. We use
tuned hyperparameters in RL Baselines Zoo [29] for each base algorithm (TD3 or
SAC) and each environment. For NFW, the Frank-Wolfe learning rate is tuned to
$0.05$ in Reacher and $0.01$ in other environments. Table I shows the used
hyperparameters. For train frequency, ‘1 epi.’ means that the model is updated
every episode. Similarly, ‘1 step’ means that the model is updated every step.
For gradient steps, ‘-1’ means to do as many gradient steps as steps done in
the environment during the rollout.

We also used gurobi [30] for solving linear or quadratic programming and
cvxpylayers [24] for differentiable optimization layers. To address potential
failures of the optimization solver for projections from distant points, we
applied the squashing technique to each coordinate of the neural network
outputs before projection, as done in [8], for methods utilizing pre-projected
actions, NFWPO and the optimization layer.

###  IV-C Evaluations with Various Environments and Constraints

Figure 3: Learning curves for R+L2

Figure 4: Learning curves for HC+O with batch size $\geq 100$

Environment | Name | Constraint
---|---|---
Reacher | R+N | No additional constraint
| R+L2 | $a_{1}^{2}+a_{2}^{2}\leq 0.05$
| R+O03 | $\sum_{i=1}^{2}|w_{i}a_{i}|\leq 0.3$
| R+O10 | $\sum_{i=1}^{2}|w_{i}a_{i}|\leq 1.0$
| R+O30 | $\sum_{i=1}^{2}|w_{i}a_{i}|\leq 3.0$
| R+M | $\sum_{i=1}^{2}\max\\{w_{i}a_{i},0\\}\leq 1.0$
| R+T | $a_{1}^{2}+2a_{1}(a_{1}+a_{2})\cos\theta_{2}$
|  | $+(a_{1}+a_{2})^{2}\leq 0.05$
HalfCheetah | HC+O | $\sum_{i=1}^{6}|w_{i}a_{i}|\leq 20$
| HC+MA | $w_{1}a_{1}\sin(\theta_{1}+\theta_{2}+\theta_{3})$
|  | $+w_{4}a_{4}\sin(\theta_{4}+\theta_{5}+\theta_{6})\leq 5$
Hopper | H+M | $\sum_{i=1}^{3}\max\\{w_{i}a_{i},0\\}\leq 10$
| H+O+S |  $\sum_{i=1}^{3}|w_{i}a_{i}|\leq 10$, $\sum_{i=1}^{3}a_{i}^{2}\sin^{2}\theta_{i}\leq 0.1$
Walker2d | W+M | $\sum_{i=1}^{6}\max\\{w_{i}a_{i},0\\}\leq 10$
| W+O+S |  $\sum_{i=1}^{6}|w_{i}a_{i}|\leq 10$, $\sum_{i=1}^{6}a_{i}^{2}\sin^{2}\theta_{i}\leq 0.1$
TABLE II: Experiment environments and constraints

In this experiment, we compared algorithms on the Reacher in PyBullet-Gym and
Hopper, Walker2d, and HalfCheetah in MuJoCo444The descriptions of the tasks
are available at www.gymlibrary.dev. with various constraints. Each action is
represented by a vector $(a_{1},\ldots,a_{d})$ corresponding to torques given
to $d$ joints, where $d=2$ for Reacher, $d=3$ for Hopper and $d=6$ for
Walker2d and HalfCheetah. Let $\theta_{1},\ldots,\theta_{d}$ and
$w_{1},\ldots,w_{d}$ be the angle and the angular velocity of joints
respectively.

In addition to the original box constraint $-1\leq a_{i}\leq 1$ for each $i$,
we consider constraints shown in Table II. Note that the constraint for R+L2
and HC+O are the same as those in [8]. For R+N, R+L2, R+O03, R+O10, R+O30,
R+T, HC+O, H+O+S and W+O+S, the center ($c_{s}$) of the feasible actions is at
the origin, but that is not necessarily for other cases.

For each combination of algorithms and constraints, we ran the algorithm with
$10$ different seeds. The number of timesteps is $3\times 10^{5}$ in Reacher
and $10^{6}$ in other three environments as in [29]. In each run, we evaluated
five episodes and took their mean in every $5,000$ timesteps. After each run,
we evaluated $50$ episodes again for the best parameters and consider their
mean as the final result.

| TD3 Family | SAC Family
---|---|---
Conditions |  DPro |  DPro+ |  DPre |  DPre+ |  DOpt |  DOpt+ |  NFW |  DAlpha |  DRad |  SPre |  SPre+ |  SAlpha |  SRad
R+N |  17.57 | Same as DPro due to no constraints |  18.47 |  17.40 |  17.34 |  18.61 |  Same |  15.01 |  18.47
|  $\pm 0.27$ |  |  |  |  |  |  $\pm 0.22$ |  $\pm 0.38$ |  $\pm 0.24$ |  $\pm 0.23$ |  SPre |  $\pm 2.00$ |  $\pm 0.26$
R+L2 |  -4.65 |  9.90 |  17.81 |  19.70 |  19.12 |  19.15 |  19.70 |  18.46 |  18.76 |  18.84 |  19.45 |  18.97 |  15.69
|  $\pm 0.59$ |  $\pm 2.69$ |  $\pm 0.33$ |  $\pm 0.30$ |  $\pm 0.30$ |  $\pm 0.51$ |  $\pm 0.33$ |  $\pm 0.86$ |  $\pm 0.37$ |  $\pm 0.27$ |  $\pm 0.29$ |  $\pm 0.31$ |  $\pm 0.46$
R+O03 |  17.32 |  18.74 |  18.17 |  18.19 |  17.72 |  18.55 |  18.86 |  18.20 |  17.88 |  18.78 |  15.82 |  18.58 |  19.11
|  $\pm 0.37$ |  $\pm 0.32$ |  $\pm 0.30$ |  $\pm 0.33$ |  $\pm 0.42$ |  $\pm 0.26$ |  $\pm 0.38$ |  $\pm 0.29$ |  $\pm 0.32$ |  $\pm 0.35$ |  $\pm 0.43$ |  $\pm 0.27$ |  $\pm 0.30$
R+O10 |  17.44 |  17.78 |  17.32 |  17.76 |  18.01 |  17.80 |  18.70 |  17.52 |  17.98 |  18.30 |  18.37 |  18.29 |  18.82
|  $\pm 0.53$ |  $\pm 0.34$ |  $\pm 0.31$ |  $\pm 0.34$ |  $\pm 0.18$ |  $\pm 0.42$ |  $\pm 0.30$ |  $\pm 0.50$ |  $\pm 0.23$ |  $\pm 0.35$ |  $\pm 0.32$ |  $\pm 0.32$ |  $\pm 0.33$
R+O30 |  17.59 |  17.14 |  17.32 |  17.15 |  16.48 |  16.42 |  18.44 |  17.07 |  16.40 |  18.51 |  18.41 |  16.90 |  17.51
|  $\pm 0.32$ |  $\pm 0.44$ |  $\pm 0.36$ |  $\pm 0.36$ |  $\pm 0.60$ |  $\pm 0.44$ |  $\pm 0.21$ |  $\pm 0.32$ |  $\pm 0.50$ |  $\pm 0.25$ |  $\pm 0.26$ |  $\pm 0.79$ |  $\pm 0.88$
R+M |  17.90 |  17.70 |  17.58 |  17.82 |  17.54 |  17.73 |  18.53 |  17.89 |  17.48 |  18.29 |  18.57 |  16.18 |  18.80
|  $\pm 0.29$ |  $\pm 0.33$ |  $\pm 0.28$ |  $\pm 0.35$ |  $\pm 0.34$ |  $\pm 0.29$ |  $\pm 0.31$ |  $\pm 0.31$ |  $\pm 0.43$ |  $\pm 0.39$ |  $\pm 0.31$ |  $\pm 1.86$ |  $\pm 0.25$
R+T |  0.35 |  18.51 |  15.95 |  18.53 | Unavailable |  17.97 |  18.34 |  17.79 |  18.19 |  18.49 |  14.94 |  -2.26
|  $\pm 2.34$ |  $\pm 0.26$ |  $\pm 1.29$ |  $\pm 0.34$ |  |  |  $\pm 0.35$ |  $\pm 0.31$ |  $\pm 0.30$ |  $\pm 0.41$ |  $\pm 0.30$ |  $\pm 2.72$ |  $\pm 0.44$
HC+O |  6062 |  6759 |  8255 |  8475 |  - |  - |  6213 |  9165 |  8736 |  9448 |  9973 |  9646 |  7904
|  $\pm 405$ |  $\pm 352$ |  $\pm 474$ |  $\pm 426$ |  |  |  $\pm 216$ |  $\pm 212$ |  $\pm 315$ |  $\pm 154$ |  $\pm 223$ |  $\pm 265$ |  $\pm 323$
HC+MA |  9668 |  9853 |  9780 |  10003 | Unavailable |  7166 |  9898 |  9515 |  10079 |  10318 |  10235 |  8369
|  $\pm 282$ |  $\pm 367$ |  $\pm 403$ |  $\pm 394$ |  |  |  $\pm 321$ |  $\pm 134$ |  $\pm 170$ |  $\pm 443$ |  $\pm 328$ |  $\pm 202$ |  $\pm 324$
H+M |  3151 |  3258 |  3354 |  3295 |  - |  - |  3309 |  3265 |  3305 |  3253 |  3278 |  3180 |  3190
|  $\pm 103$ |  $\pm 55$ |  $\pm 10$ |  $\pm 35$ |  |  |  $\pm 20$ |  $\pm 19$ |  $\pm 16$ |  $\pm 60$ |  $\pm 23$ |  $\pm 54$ |  $\pm 58$
H+O+S |  987 |  2245 |  3179 |  3261 | Unavailable |  1785 |  3260 |  3251 |  3257 |  2992 |  2867 |  3300
|  $\pm 78$ |  $\pm 233$ |  $\pm 30$ |  $\pm 34$ |  |  |  $\pm 316$ |  $\pm 26$ |  $\pm 26$ |  $\pm 19$ |  $\pm 300$ |  $\pm 290$ |  $\pm 21$
W+M |  1479 |  4694 |  4460 |  4898 |  - |  - |  4454 |  4071 |  4341 |  4446 |  4595 |  4268 |  4319
|  $\pm 280$ |  $\pm 169$ |  $\pm 84$ |  $\pm 155$ |  |  |  $\pm 51$ |  $\pm 190$ |  $\pm 146$ |  $\pm 131$ |  $\pm 147$ |  $\pm 158$ |  $\pm 107$
W+O+S |  856 |  931 |  3959 |  4102 | Unavailable |  3408 |  3377 |  3467 |  3828 |  4127 |  4105 |  3961
|  $\pm 70$ |  $\pm 113$ |  $\pm 77$ |  $\pm 100$ |  |  |  $\pm 300$ |  $\pm 37$ |  $\pm 65$ |  $\pm 87$ |  $\pm 79$ |  $\pm 76$ |  $\pm 96$
(batch size = 16) |
HC+O |  4108 |  5303 |  5826 |  6787 |  7053 |  7306 |  5984 |  7245 |  7387 |  8059 |  7071 |  7748 |  7459
|  $\pm 399$ |  $\pm 115$ |  $\pm 366$ |  $\pm 378$ |  $\pm 239$ |  $\pm 362$ |  $\pm 319$ |  $\pm 191$ |  $\pm 204$ |  $\pm 259$ |  $\pm 762$ |  $\pm 245$ |  $\pm 169$
H+M |  3166 |  3260 |  3193 |  3247 |  3154 |  3290 |  3041 |  3262 |  3221 |  3303 |  3211 |  3249 |  3243
|  $\pm 70$ |  $\pm 28$ |  $\pm 61$ |  $\pm 62$ |  $\pm 122$ |  $\pm 33$ |  $\pm 52$ |  $\pm 31$ |  $\pm 49$ |  $\pm 19$ |  $\pm 58$ |  $\pm 34$ |  $\pm 52$
W+M |  2138 |  3021 |  3409 |  3767 |  2837 |  3292 |  3644 |  3483 |  4013 |  3483 |  3600 |  3672 |  3431
|  $\pm 251$ |  $\pm 176$ |  $\pm 267$ |  $\pm 246$ |  $\pm 431$ |  $\pm 506$ |  $\pm 128$ |  $\pm 117$ |  $\pm 137$ |  $\pm 182$ |  $\pm 135$ |  $\pm 106$ |  $\pm 182$
TABLE III: Average Rewards.

Table III shows the average values and standard errors of rewards over the
$10$ seeds, where the top three algorithms for each constraint are emphasized
with bold text. Due to the limitation of cvxpylayer, DOpt and DOpt+ cannot be
applied to some constraints.555Constraints should adhere to the disciplined
parameterized programming rules [24]. So we excluded these algorithms from
some results and display “unavailable” in the table. Furthermore, since DOpt
and DOpt+ are too computationally expensive to run with the tuned batch size
for MuJoCo environments, we excluded them and additionally ran all algorithms
including them with batch size $16$ for constraints for which DOpt and DOpt+
are available. The bottom three rows of the table shows the results of
additional experiments. Fig. 4 and Fig. 4 present the learning curves averaged
over the $10$ seeds for R+L2 and HC+O, respectively.

###  IV-D Measurements of Training Runtime

Figure 5: Training runtime for HalfCheetah. The light areas of the bars show
the time on a CPU and the dark areas show the time on a GPU.

Finally, we measured the runtime of each algorithm for the first $1000$
gradient steps of the training procedure in the HalfCheetah environment, with
batch sizes $1,16$ or $100$. Fig. 5 summarizes the runtimes in seconds
averaged over 10 trials with different seeds.

###  IV-E Findings and Discussion

####  IV-E1 Training the critic using pre-projected actions is a good baseline
especially with penalty terms

We found that DPro, i.e., training the critic using projected actions,
demonstrated limited performance compared to DPre with pre-projected actions
in HalfCheetah and Walker environments, despite that it was used as a baseline
in some previous work [7, 8]. This could be due to its fundamental defect
discussed in Section III-A1 and indicates that _pre-projected actions should
be used for training the critic_. DPre+, on the other hand, achieved
performance comparable to other TD3 variants. We contend that both SPre and
SPre+, which are SAC algorithms utilizing pre-projected actions, serve as
viable initial selections among the evaluated algorithms. As demonstrated in
Table III, these methods consistently rank within the top three across most
conditions, while also offering the advantages of relatively low
implementation efforts and computational runtime as shown in Fig. 5

####  IV-E2 The use of optimization layers and NFWPO comes with significant
runtime overheads

DOpt requires significant computation time, as clearly shown in Fig. 5.
Nevertheless, it does not demonstrate significantly better performance than
DPre+, DAlpha or DRad.

While NFW achieved the best learning performance among the TD3 family in the
Reacher environment, its performance is not significantly better in other
environments. On the other hand, it also requires a considerably large runtime
as the batch size becomes larger.

####  IV-E3 Mapping techniques can be less expensive alternatives to
optimization layers

SAlpha and SRad demonstrated good performance for a small batch size at the
cost of 140-150% runtime overheads compared to SPre+. Also, DAlpha and DRad
performed on par (if not better) with DOpt and DOpt+, while requiring much
less runtime. Nonetheless, under certain conditions (R+L2 and R+T), SRad
exhibited subpar performance. Investigating the reasons behind this
underperformance will be the subject of future research.

##  V Conclusion

In this paper, we compared variants of TD3 and SAC on a variety of continuous
control tasks in the presence of action constraints. Our evaluation includes
new variants of the existing action-constrained RL algorithms. Our benchmark
evaluation has led to the following main findings. 1) Training the critic with
pre-projected actions is a good baseline, especially with penalty terms. 2)
The use of optimization layers and NFWPO comes with significant runtime
overheads. 3) Mapping techniques are useful alternatives to optimization
layers. A complete implementation of our benchmark evaluation is available
online at github.com/omron-sinicx/action-constrained-RL-benchmark.

## References

  * [1] Y. Fujita and S. Maeda, “Clipped Action Policy Gradient,” in _International Conference on Machine Learning_ , 2018, pp. 1597–1606.
  * [2] T.-H. Pham, G. De Magistris, and R. Tachibana, “OptLayer - Practical Constrained Optimization for Deep Reinforcement Learning in the Real World,” in _2018 IEEE International Conference on Robotics and Automation_. IEEE Press, 2018, pp. 6236–6243.
  * [3] S. Gu, E. Holly, T. Lillicrap, and S. Levine, “Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates,” in _2017 IEEE International Conference on Robotics and Automation_. IEEE Press, 2017, pp. 3389–3396.
  * [4] R. Cheng, G. Orosz, R. M. Murray, and J. W. Burdick, “End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks,” in _Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence_ , 2019, pp. 3387–3395.
  * [5] B. Chen, P. L. Donti, K. Baker, J. Z. Kolter, and M. Bergés, “Enforcing Policy Feasibility Constraints through Differentiable Projection for Energy Optimization,” in _Proceedings of the Twelfth ACM International Conference on Future Energy Systems_. ACM, 2021, pp. 199–210.
  * [6] G. Dalal, K. Dvijotham, M. Vecerik, T. Hester, C. Paduraru, and Y. Tassa, “Safe Exploration in Continuous Action Spaces,” 2018, arXiv:1801.08757.
  * [7] A. Bhatia, P. Varakantham, and A. Kumar, “Resource Constrained Deep Reinforcement Learning,” in _Proceedings of the Twenty-Ninth International Conference on Automated Planning and Scheduling_ , 2019, pp. 610–620.
  * [8] J.-L. Lin, W.-T. Hung, S. Yang, P.-C. Hsieh, and X. Liu, “Escaping from Zero Gradient: Revisiting Action-Constrained Reinforcement Learning via Frank-Wolfe Policy Optimization,” in _Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence_ , 2021.
  * [9] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” in _Proceedings of the Fourth International Conference on Learning Representations 2016_ , 2016.
  * [10] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor,” in _Proceedings of the Thirty-Fifth International Conference on Machine Learning_. PMLR, 2018, pp. 1861–1870.
  * [11] Y. Chow, O. Nachum, A. Faust, E. Duenez-Guzman, and M. Ghavamzadeh, “Lyapunov-based Safe Policy Optimization for Continuous Control,” 2019, arXiv:1901.10031.
  * [12] S. Sanket, A. Sinha, P. Varakantham, P. Andrew, and M. Tambe, “Solving Online Threat Screening Games using Constrained Action Space Reinforcement Learning,” 2020, pp. 2226–2235.
  * [13] E. Todorov, T. Erez, and Y. Tassa, “Mujoco: A physics engine for model-based control,” in _2012 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, 2012, pp. 5026–5033.
  * [14] B. Ellenberger, “PyBullet gymperium,” https://github.com/benelot/pybullet-gym, 2018–2019.
  * [15] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, “Openai gym,” 2016.
  * [16] V. Konda and J. Tsitsiklis, “Actor-critic algorithms,” _Advances in neural information processing systems_ , vol. 12, 1999.
  * [17] S. Fujimoto, H. van Hoof, and D. Meger, “Addressing Function Approximation Error in Actor-Critic Methods,” Oct. 2018, arXiv:1802.09477 [cs, stat].
  * [18] J. Li, D. Fridovich-Keil, S. Sojoudi, and C. J. Tomlin, “Augmented Lagrangian Method for Instantaneously Constrained Reinforcement Learning Problems,” in _Proceeding of the Sixtieth IEEE Conference on Decision and Control_. IEEE, 2021, pp. 2982–2989.
  * [19] A. D. Ames, S. Coogan, M. Egerstedt, G. Notomista, K. Sreenath, and P. Tabuada, “Control Barrier Functions: Theory and Applications,” in _Proceeding of the Eighteenth European Control Conference_ , 2019, pp. 3420–3431.
  * [20] M. Pereira, Z. Wang, I. Exarchos, and E. Theodorou, “Safe Optimal Control Using Stochastic Barrier Functions and Deep Forward-Backward SDEs,” in _Proceedings of the 2020 Conference on Robot Learning_. PMLR, 2021, pp. 1783–1801.
  * [21] A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dormann, “Stable-baselines3: Reliable reinforcement learning implementations,” _Journal of Machine Learning Research_ , vol. 22, no. 268, pp. 1–8, 2021\.
  * [22] D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. Riedmiller, “Deterministic policy gradient algorithms,” in _Proceedings of the Thirty-First International Conference on International Conference on Machine Learning_. JMLR, 2014, pp. 387–395.
  * [23] B. Amos and J. Z. Kolter, “OptNet: Differentiable optimization as a layer in neural networks,” in _Proceedings of the Thirty-Fourth International Conference on Machine Learning_. JMLR, 2017, pp. 136–145.
  * [24] A. Agrawal, B. Amos, S. Barratt, S. Boyd, S. Diamond, and J. Z. Kolter, “Differentiable convex optimization layers,” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [25] M. Frank and P. Wolfe, “An algorithm for quadratic programming,” _Naval Research Logistics Quarterly_ , vol. 3, no. 1-2, 1956.
  * [26] J. M. Lee and J. M. Lee, _Smooth manifolds_. Springer, 2012.
  * [27] M. Brosowsky, F. Keck, O. Dünkel, and M. Zöllner, “Sample-Specific Output Constraints for Neural Networks,” in _Proceedings of the Thirty-Fifth AAAI Conference on Artificial Intelligence_ , 2021, pp. 6812–6821.
  * [28] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga _et al._ , “Pytorch: An imperative style, high-performance deep learning library,” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [29] A. Raffin, “RL baselines3 zoo,” https://github.com/DLR-RM/rl-baselines3-zoo, 2020.
  * [30] B. Bixby, “The gurobi optimizer,” _Transp. Re-search Part B_ , vol. 41, no. 2, pp. 159–178, 2007.


# To Classify is to Interpret: Building Taxonomies from Heterogeneous Data
through Human-AI Collaboration

Sebastian Meier<EMAIL_ADDRESS>0000-0002-7466-4059 Potsdam
University of Applied Sciences, Interaction Design LabKiepenheuerallee
5PotsdamBrandenburgGermany14469 and  Katrin Glinka<EMAIL_ADDRESS>0000-0002-4232-8907 Freie Universität BerlinBerlinGermany

(2023)

###### Abstract.

Taxonomy building is a task that requires interpreting and classifying data
within a given frame of reference, which comes to play in many areas of
application that deal with knowledge and information organization. In this
paper, we explore how taxonomy building can be supported with systems that
integrate machine learning (ML). However, relying only on black-boxed ML-based
systems to automate taxonomy building would sideline the users’ expertise. We
propose an approach that allows the user to iteratively take into account
multiple model’s outputs as part of their sensemaking process. We implemented
our approach in two real-world use cases. The work is positioned in the
context of HCI research that investigates the design of ML-based systems with
an emphasis on enabling human-AI collaboration.

Human-AI Collaboration

††copyright: acmcopyright††journalyear: 2023††doi:
XXXXXXX.XXXXXXX††conference: Make sure to enter the correct conference title
from your rights confirmation emai; September 03-06, 2023; Rapperswil,
Switzerland††booktitle: Mensch und Computer 2023††price: 15.00††isbn:
978-1-4503-XXXX-X/18/06††ccs: Human-centered computing Interaction design
theory, concepts and paradigms††ccs: Human-centered computing Interactive
systems and tools††ccs: Human-centered computing Graphical user
interfaces††ccs: Computing methodologies Natural language processing††ccs:
Computing methodologies Neural networks

##  1\. Introduction

Human sensemaking and knowledge production are fundamentally dependent on
classification (Bowker and Star, 1999). Taxonomies serve this purpose as
structured classification schemes that adhere to domain-specific standards.
The importance of organizing, segmenting, and classifying data is especially
obvious in light of the ever growing amount of information that is being
created, aggregated, and made available through specialized data repositories
or on the Internet. In light of the amount and heterogeneity of the available
data, classification can hardly be addressed by means of manual-cognitive
processing alone. Systems that integrate machine learning (ML) are able to
process large amounts of data and, thus, can help with the task of
classification and organization. However, delegating this task to ML-based
systems in their entirety would mean that we sideline human interpretation and
rely on the output of black-boxed systems that reproduce language ideologies
and representational harms (see, e.g., (Blodgett et al., 2020)). As an attempt
to highlight the interpretative character of classification and taxonomy
building, we propose to design ML-based systems that enable human-AI
collaboration. Such systems are designed with the goal to effectively combine
human competencies and computational capabilities (see, e.g.,(Shneiderman and
Maes, 1997; Terveen, 1995)). Our approach enables domain experts to
iteratively interact with the suggestions of the system while retaining
interpretative authority. We report on the concept and implementation of this
approach that we realized for two real-world use cases.111The code of both use
case is available under open source licenses. An interactive online demo with
code instructions can be found here: https://github.com/sebastian-
meier/muc23-human-ai-demo In the following, we situate our work in the context
of related research on human-AI collaboration.

##  2\. Related Work

Although data and classifications make up a huge part of the infrastructures
that mediate interactions, the creation of taxonomies has not been
foregrounded in Human-Computer Interaction (HCI) research (Feinberg et al.,
2014). Nonetheless, research from HCI informed our exploration of how we can
design ML-based systems that support taxonomic classification. We propose to
design systems for iterative taxonomic classification in such a way that ML-
outputs are framed as suggestions that need to be scrutinized by human
interpretation in order to contribute to sense making (see, e.g., (Baumer et
al., 2020)). Likewise, we seek to balance the agency distribution between ML-
based systems and their users, instead of aiming at full automation (Jiang et
al., 2021). In this regard, our approach is informed by research into human-AI
collaboration that is concerned with conceptualizing and designing
interactions between humans and “AI” in such a way that the complementary
capacities of humans and technology contribute to achieving a shared goal
(Shneiderman and Maes, 1997; Terveen, 1995; Wang et al., 2020). Instead of
aiming for automation, we conceptualize taxonomy building as an interactive
and interpretative practice that integrates human competencies, machine
learning, and information visualization. Our approach foregrounds the
interpretative aspect of data generation (Feinberg, 2017) \- both on the side
of the human as well as on the side of the computer.

##  3\. Problem Space

One important step in the processing and organization of data is
classification. Classification is often done within a taxonomy and alongside
domain-specific standards. Generally speaking, taxonomies are controlled
vocabularies that are intertwined with data standardization practices. Our
work addresses two common challenges that arise in this context. For one, we
might be confronted with data that is being created dynamically and within a
frame of reference that is not clearly pre-defined. Consequently, patterns of
meaning that could be grouped under one or more classifiers only emerge over
time. Additionally, in cases where such dynamically created data does not
relate to a clearly defined frame of reference, we cannot base our
classification on already existing taxonomies. We address this challenge with
a use case on the classification of crowd-sourced questions submitted to a
citizen-driven science agenda setting project in subsection 4.1. For another,
even when working within an already established taxonomy, data work is often
done manually by humans. This means that the data created might not strictly
follow the established taxonomy or classification system. As Bowker and Star
have pointed out , there is “a permanent tension between attempts at universal
standardization of lists and the local circumstances of those using them”
(Bowker and Star, 1999, p.139), which leads to modification of the
classification system. Even in cases where taxonomies serve the needs of those
who use them, errors can occur. Thus, the existence of a taxonomy alone does
not guarantee consistent and clean data. We address this challenge with a use
case on the taxonomic classification of open government data as part of an
attempt of homogenizing the metadata in a step towards functional linked data
(Berners-Lee, 2006) in subsection 4.2.

##  4\. Use Cases

In the following, we briefly describe both use cases in terms of context,
intended user groups, underlying dataset and purpose of taxonomic
classification. In both implementations, our approach aims at making large
datasets usable (e.g. explore, search, find) for users that do not have a
background in data science or ML.

###  4.1. Use Case I - Crowd-sourced Questions

Germany’s Federal Ministry of Education and Research (BMBF) planned to deploy
a large-scale participation project in 2022 (see (of Education and Research,
[n. d.])) to develop a citizen-driven science agenda informed by the citizen
submitted questions. The first use case was an initial prototype developed as
a testbed for the future platform. A similar project had been launched by the
Research Foundation Flanders (FWO) in Belgium in 2018. The FWO project amassed
10.559 questions (see ((2018a), FWO)). Our prototype aimed at helping the
project managers of the BMBF to organize the submitted questions by way of
taxonomic classification. The resulting taxonomy should, for one, provide the
Federal Ministry with an overview of the submitted questions. For another, it
should allow citizens to explore the already submitted questions and provide
them with a frame of reference for their own questions during the submission
process. We used the questions from the FWO survey as a sample dataset (see
((2018b), FWO)). While the overall frame of reference is ‘science’, citizens
were free to ask any questions that they believe science can (or should be
able to) answer. Therefore, the result is a heterogeneous dataset in regards
to language (choice of words, style of writing, etc.) as well as content and
length (from 9 characters to 274). One could try to impose academic taxonomies
(i.e., relating to different fields of science) onto the questions. However,
this would disregard the citizens’ perspective and expertise. In contrast, our
system encourages the building of a new taxonomy based on the citizen’s
questions. In summary, the system’s intended users are project managers of the
Federal Ministry. The dataset in question is dynamically generated by
citizens. In this regard, the full extent of the dataset and its content are
unknown to the users.

###  4.2. Use Case II - Open Government Data - Metadata

Open government data (OGD) has become a cornerstone of government transparency
around the world. To organize the datasets, metadata standards have been
created by European and German working groups (e.g., INSPIRE (Commission,
2023) and DCAT-AP (Commision, 2022)). Each of these metadata standards contain
controlled vocabularies to classify each dataset’s theme or category222INSPIRE
themes (Commission, 2013b) (level 1) and codes (Commission, 2013a) (level 2),
DCAT-AP: Data/MDR themes (Commission, 2022). Additionally, datasets can be
described through keywords and tags, for which no pre-defined taxonomy exists.
As a result of the hierarchical structure of the federal government in
Germany, OGD is collected and provided through various bodies within this
structure (e.g. federal states, municipalties and government institutions) and
distributed across various organizational and technological infrastructures.
While guidelines for using the mentioned standards exist, they are not
enforced and data is often not validated. Therefore, the ‘theme’ attributes
are rarely used correctly (left empty, false entries, typing errors, etc.).
The result is a very heterogeneous dataset. For our use case, we took all the
data that could not be linked with the official taxonomies and combined the
‘themes’ entries with the ‘keywords’ entries (as for most cases the themes
field was empty). This resulted in 154.605 descriptors. To reduce the number
of unique descriptors, we pre-processed the data in five steps: 1) removing
items with only numbers or dates, 2) removing items that only contain stop-
words, 3) removing items matching OpenStreetMap place names333Although place
names seem like a useful descriptor, they are not allowed as themes for
datasets within the established metadata standards. 4) merging of small typos
through a) removing space surrounding descriptors, b) conversion to all lower-
case characters and c) Levenshtein distance and fingerprinting measures
(inspired by OpenRefine (Community, 2022)), 5) removal of descriptors used
only three times or less. This resulted in 16.207 presumably unique
descriptors. This second use case aims at enabling the intended users (ODG
experts) to explore the descriptors and re-classify them in order to build a
taxonomy that conforms to the existing metadata standards. The intended user
group for this use case are OGD experts who have no expert knowledge of ML.

##  5\. Proposed approach

Generally speaking, building and applying a taxonomy for a dataset involves
trying to find commonalities between items. Identifying commonalities hinges
on first establishing a guiding (and meaningful) principle for the assessment
of ‘similarity’ between items. In both of our use cases, we only work with
text-based items. For this purpose, we deem text-based content similarity a
meaningful principle of similarity. A variety of NLP and ML approaches exist
that enable this type of content similarity analysis (see subsection 5.1).
General purpose models make NLP readily applicable to a range of tasks, even
when there is no domain-specific model. However, this also means that the
text-to-vector embeddings will contain a level of uncertainty. Automatically
building a taxonomy based on such imperfect classifications made by a black-
boxed ML model would not emphasize the interpretative aspect of data
generation and taxonomy building, which we consider to be a crucial element of
human sensemaking and knowledge production. Furthermore, even with general
purpose models, different versions exist in which certain parameters can be
adjusted. In the following, we describe how our proposed approach frames the
models’ output as suggestions that the user can interact with. In regards to
use case II, we also illustrate how our approach foregrounds the impact of
different model parameters on the system’s output. We lay out the
technological and design elements of our system in reference to existing
approaches in the following.

###  5.1. Embedding Visualization Systems

Interactions with text-to-vector embeddings can be found in many applications
(e.g. search engines or recommender systems highlighting related items). In
those systems, the user only interacts with the data items, while the
underlying embeddings are hidden (see Figure 1 left). Alternatively, embedding
visualizations allow users to explore the embeddings visually, search related
items, or to use clustering algorithms, just to name a few. Although not
usually present in common real-world implementations, such visualisations of
embeddings are not a novel approach. Existing examples include Google’s
embedding projector (Tensorflow, 2022) or Ji et al.’s visual analysis tool of
cluster methods (Ji et al., 2019). However, existing visualization approaches
and systems do not connect the interactions with the embeddings in such a way
that the user can perform data tasks (i.e. alter, adjust, or re-assign
classifications) on the underlying dataset (see Figure 1 middle).

Figure 1. Interactions with text-to-vector embeddings in common
implementations, embedding visualisation systems, and our use cases

The graphic compares the interactions with text-to-vector embeddings in a)
common implementations, b) embedding visualisation systems, and c) the use
cases described in this paper. For all three interaction sets, the graphic
uses arrows to highlight the direction of interactions. In the first case -
the common real world implementations - the users interact with the data item
to achieve a task (e.g., search). To do so, the users interact with an
abstracted visual user interface. The AI-support (e.g., based on text-to-
vector embeddings) that leads to the final interaction result is hidden
underneath the user interface. In the second case - the embedding
visualisation system - the users interact with visualizations of the
embeddings of the data items. Users gain insights into the data items or
rather their embeddings (e.g. clusters) through interaction with the
visualization. The interactions are usually not connected with a third system.
Therefore, the interaction and insights stay in a closed loop for visual
exploration of embeddings. In the last case - the use cases described in this
paper - the users interact with the the data items and their embeddings and
are enabled to build a taxonomy for the data items. Data items and their
embeddings are visually connected. Thereby, the AI-generated embeddings help
in the actual user task (taxonomy building) while remaining visible (unlike
the first case).

In contrast, our system enables interaction with the embedding visualizations
to structure and organize data items, i.e., the embeddings and their
visualisation are used to perform a task on the dataset itself (see Figure 1
right). This way, visualizations not only provide users with an overview of
what the dataset is about, but also constitute a crucial component in the
workflow and sensemaking process. The embedding visualizations are not a
monolithic and static component of the interface, but rather a canvas on which
the users can iteratively and interactively build a taxonomy - guided by their
own interpretation of the system’s suggestions that they can relate to the
underlying texts or entries.

###  5.2. Data-Processing Pipeline and Machine Learning

In both use cases we calculate the similarity of 1) short paragraphs (use case
I) and 2) category descriptors (use case II), of one to n-words length, with
text-to-vector approaches. The default model used is TensorFlow’s universal-
sentence-encoder v4 (USE) (see (Cer et al., 2018; Google, 2022b)). However,
since the data from our use cases is in Dutch (use case I) and German (use
case II), we added translation into English to the pipeline (Google, 2022c).
While translation adds to the uncertainty of the pipeline, the improved
precision and performance of USE for English justified this trade-off.

With use case II, which focuses on OGD experts as user groups, we decided to
allow the users to interact with more models and added USE “Large”, Standard
English and Multilingual (Yang et al., 2019) as well as Neural-Net Language
Models (NNLM) for English and German (Bengio et al., 2003)(Google, 2022a). The
result is a 512-(USE) or 128-dimensional (NNLM) vector.

For the 2D visualisation, we applied dimensionality reduction methods to the
vectors (use case I: T-SNE (Van der Maaten and Hinton, 2008)), use case II:
T-SNE, PCA and MDS). Users can furthermore alter dimensionality reduction
parameters. For the expert users in use case II, we also applied the DBSCAN
(Ester et al., 1996) clustering algorithm in order to provide visual cues that
would help them to identify potential clusters within the individual embedding
sets. The reason behind providing this range of models and parameters in use
case II is that each step, especially the dimensionality reduction, introduces
uncertainty and variability. This way, the variety and inconclusiveness of ML-
based similarity analysis is made visible to the users.

To achieve fast interactions, we ran all the above steps in advance, which
generated 448 different model outputs. To give users in both use cases an
additional method to identify similar items, we are using a VPTree (Yianilos,
1993) implementation to allow users to quickly browse the full embeddings.

###  5.3. Interaction Flow Concept and Interface Design

Figure 2. Iterative selection process, based on similarity measures on the
text-to-vector embeddings

The graphic shows the process of the iterative selection process, based on
similarity measures on the text-to-vector embeddings. From left to right the
steps are described. 1. the overall data set is shown, 2. users can select a
starting point from either the embeddings or a text search of the data items,
3. the result is the initial selection of items, 4. based on this selection,
similar items (based on embeddings) are shown to the user as potential
candidates, 5. the user can then select good candidates and remove bad ones.
This new selection is then used as a new starting point for repeating the
steps 4 or 5, in order to turn this process into an incrementally improving
and growing selection process.

While the underlying data processing- and NLP-pipelines are the same, the
interactive graphical user interfaces (GUIs) explore two distinct directions,
1) a simplified interactive GUI for the project managers (use case I), and 2)
a more advanced interactive GUI for OGD-experts (use case II) (see Appendix A
for annotated screenshots of both GUIs).

The interaction flow concept of both GUIs is focused on an iterative process
that refines the selection of potentially similar items to be considered in
the taxonomic classification (see Figure 2). Starting with a data item
selected by the user, potentially similar candidates (based on the VPTree) are
suggested to the user in list and point cloud form. This allows the user to
select the best candidates for the new taxonomy group. Beyond VPTree, users
can draw a polygon-hull to select points from the visualisation. This then
increases the list of potential candidates. As more and more manually matched
candidates are selected for the cluster, the search for additional potential
candidates is expanded. The users can also remove candidates and add them on a
‘ignore’ list, which will be taken into account by subsequent searches.

For the second use case, the users can visually analyse the current selection
of candidates across multiple models’ embeddings (see subsection 5.1).
Thereby, we allow users to visually search for potential clusters in different
settings (see Figure 3). By integrating multiple embeddings, we also believe
to work towards increasing awareness for the biases, uncertainty and
variability in model outputs by emphasizing embeddings as suggestions -
suggestions that ultimately need human interpretation to successfully
contribute to taxonomic classification of heterogeneous data.

Figure 3. Extension of fig. 2, by integrating multiple embeddings and enabling
users to compare and switch between models

The graphic extends the previous process of the iterative selections. In the
previous example, users could refine their selection through suggestions of
the text-to-vector embeddings, but the embeddings were fixed. In other words:
they were based on one specific embedding process. In the extension of this
process in use case II, which is illustrated in the graphic, the users can
switch between various embeddings for the data set. Depending on the selected
model, different patterns of similarity emerge. Users can then visually
compare the output of different models and reflect on the influence of model
choice.

##  6\. Discussion

The two use cases provided us with a frame of reference for the development of
two prototypes through which we explore the design space for taxonomic
classification through human-AI collaboration. Usually, in applications with
high levels of automation, the user is only presented with one outcome, which
obfuscates how much the prediction depends on model configuration. In such
cases, the model’s output acts as a normative statement. The problems with
overreliance in AI and automation bias are well known and are being addressed,
for example, in the context of research into explainable AI (XAI) (Buçinca et
al., 2021). However, research also suggests that explanations might even have
a worsening effect, in the sense that offering explanations could lead to even
more overreliance and trust in AI (Bansal et al., 2021). We propose to take an
alternative approach that tries to balance levels of automation (see, e.g.,
(Mackeprang et al., 2019)) and combine it with an emphasis on the variability
of model outputs. With this, we seek to frame the model’s output as an
interpretative statement. We do so by integrating faceted linked views, small
multiples, and visual analytics and, thereby, make the embedding
visualisations part of the interface instead of hiding them underneath.

##  7\. Limitations and Future Work

As a next step we have to evaluate our approach through user studies in order
to confirm whether and to what extent the intended effects also hold true in a
user study. While we acknowledge this obvious limitation, we consider the work
presented in this paper as a starting point for further research. With these
studies we seek to investigate if the facilitation of user reflection
regarding the interpretative aspect of taxonomy building, were realized.

In regards to technological and design limitations, we had to take a pragmatic
approach towards the selection of models and parameters. Since use case I is
focused on project managers of the BMBF, we decided to simplify the GUI. This,
in turn, counteracts our intention to confront the users with a variety of
system outputs. In use case II, however, we offer 448 model outputs, which is
too much to continuously visually compare. This bares the question of how to
select the “best” models for the users to choose from, i.e., how to balance
visual overload on the one hand while enabling visual exploratory analysis on
the other hand.

After refining our approach, we plan to transfer it to other domains (e.g.
digital cultural heritage). We believe that our approach can contribute to the
effort of data organization while at the same time raising awareness for the
variability and uncertainty of ML model outputs, the role of interpretation
and importance of human sensemaking.

###### Acknowledgements.

The work presented in this paper was made possible through funding by the
Federal Ministry of Education and Research (BMBF) and the Federal Ministry for
Economic Affairs and Climate Action (BMWK).

## References

  * (1)
  * Bansal et al. (2021) Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. 2021\.  Does the Whole Exceed Its Parts? The Effect of AI Explanations on Complementary Team Performance. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ (Yokohama, Japan) _(CHI ’21)_. Association for Computing Machinery, New York, NY, USA, Article 81, 16 pages.  https://doi.org/10.1145/3411764.3445717
  * Baumer et al. (2020) Eric P S Baumer, Drew Siedel, Lena McDonnell, Jiayun Zhong, Patricia Sittikul, and Micki McGee. 2020\.  Topicalizer: reframing core concepts in machine learning visualization by co-designing for interpretivist scholarship.  _Human–Computer Interaction_ 35, 5–6 (Apr 2020), 452–480.  https://doi.org/10.1080/07370024.2020.1734460
  * Bengio et al. (2003) Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003.  A Neural Probabilistic Language Model.  _J. Mach. Learn. Res._ 3, null (mar 2003), 1137–1155.
  * Berners-Lee (2006) Tim Berners-Lee. 2006\.  _Linked Data_.  https://www.w3.org/DesignIssues/LinkedData.html
  * Blodgett et al. (2020) Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020.  Language (Technology) is Power: A Critical Survey of “Bias” in NLP. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Online, 5454–5476.  https://doi.org/10.18653/v1/2020.acl-main.485
  * Bowker and Star (1999) Geoffrey C. Bowker and Susan Leigh Star. 1999.  _Sorting things out: classification and its consequences_.  MIT Press, Cambridge, Mass.
  * Buçinca et al. (2021) Zana Buçinca, Maja Barbara Malaya, and Krzysztof Z. Gajos. 2021.  To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-Assisted Decision-Making.  _Proc. ACM Hum.-Comput. Interact._ 5, CSCW1, Article 188 (apr 2021), 21 pages.  https://doi.org/10.1145/3449287
  * Cer et al. (2018) Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2018.  Universal Sentence Encoder.  _CoRR_ abs/1803.11175 (2018).  arXiv:1803.11175 http://arxiv.org/abs/1803.11175
  * Commision (2022) European Commision. 2022\.  _DCAT Application Profile for data portals in Europe_.  https://joinup.ec.europa.eu/collection/semantic-interoperability-community-semic/solution/dcat-application-profile-data-portals-europe/release/211
  * Commission (2013a) European Commission. 2013a.  _INSPIRE code list register_.  https://inspire.ec.europa.eu/codelist
  * Commission (2013b) European Commission. 2013b.  _INSPIRE theme register_.  https://inspire.ec.europa.eu/theme
  * Commission (2022) European Commission. 2022\.  _RDF: Data Themes_.  http://publications.europa.eu/resource/authority/data-theme
  * Commission (2023) European Commission. 2023\.  _INSPIRE KNOWLEDGE BASE_.  https://inspire.ec.europa.eu/
  * Community (2022) Open Refine Developement Community. 2022\.  _Clustering Methods In-depth_.  https://docs.openrefine.org/technical-reference/clustering-in-depth
  * Ester et al. (1996) Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu. 1996.  A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise. In _Proceedings of the Second International Conference on Knowledge Discovery and Data Mining_ (Portland, Oregon) _(KDD’96)_. AAAI Press, 226–231.
  * Feinberg (2017) Melanie Feinberg. 2017\.  A Design Perspective on Data. In _Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems_ (Denver, Colorado, USA) _(CHI ’17)_. Association for Computing Machinery, New York, NY, USA, 2952–2963.  https://doi.org/10.1145/3025453.3025837
  * Feinberg et al. (2014) Melanie Feinberg, Daniel Carter, and Julia Bullard. 2014\.  A Story without End: Writing the Residual into Descriptive Infrastructure. In _Proceedings of the 2014 Conference on Designing Interactive Systems_ (Vancouver, BC, Canada) _(DIS ’14)_. Association for Computing Machinery, New York, NY, USA, 385–394.  https://doi.org/10.1145/2598510.2598553
  * (19) Flanders Scientific Research Fund (FWO). 2018a.  _Vraag voor de wetenschap - Vlaamse Wetenschapsagenda_.  https://www.vraagvoordewetenschap.be/
  * (20) Flanders Scientific Research Fund (FWO). 2018b.  _Vraag voor de wetenschap - Vlaamse Wetenschapsagenda_.  https://www.vraagvoordewetenschap.be/p/thema_zoek?c=&search=&order=vvdw_answered
  * Google (2022a) Google. 2022a.  _Tensorflow Hub - nnlm_.  https://tfhub.dev/google/collections/nnlm/1
  * Google (2022b) Google. 2022b.  _Tensorflow Hub - universal-sentence-encoder_.  https://tfhub.dev/google/collections/universal-sentence-encoder/1
  * Google (2022c) Google. 2022c.  _Translation AI_.  https://cloud.google.com/translate
  * Ji et al. (2019) Xiaonan Ji, Han-Wei Shen, Alan Ritter, Raghu Machiraju, and Po-Yin Yen. 2019.  Visual Exploration of Neural Document Embedding in Information Retrieval: Semantics and Feature Selection.  _IEEE Transactions on Visualization and Computer Graphics_ 25, 6 (2019), 2181–2192.  https://doi.org/10.1109/TVCG.2019.2903946
  * Jiang et al. (2021) Jialun Aaron Jiang, Kandrea Wade, Casey Fiesler, and Jed R. Brubaker. 2021.  Supporting Serendipity: Opportunities and Challenges for Human-AI Collaboration in Qualitative Analysis.  _Proc. ACM Hum.-Comput. Interact._ 5, CSCW1, Article 94 (April 2021), 23 pages.  https://doi.org/10.1145/3449168
  * Mackeprang et al. (2019) Maximilian Mackeprang, Claudia Müller-Birn, and Maximilian Timo Stauss. 2019.  Discovering the Sweet Spot of Human-Computer Configurations: A Case Study in Information Extraction.  _Proc. ACM Hum.-Comput. Interact._ 3, CSCW, Article 195 (nov 2019), 30 pages.  https://doi.org/10.1145/3359297
  * of Education and Research ([n. d.]) Federal Ministry of Education and Research. [n. d.].  _Wissenschaftsjahr 2022 – Nachgefragt!_ https://www.wissenschaftsjahr.de/2022/fragen
  * Shneiderman and Maes (1997) Ben Shneiderman and Pattie Maes. 1997.  Direct manipulation vs. interface agents.  _Interactions_ 4, 6 (Nov 1997), 42–61.  https://doi.org/10.1145/267505.267514
  * Tensorflow (2022) Tensorflow. 2022\.  _Tensorboard_.  https://github.com/tensorflow/tensorboard
  * Terveen (1995) Loren G. Terveen. 1995\.  Overview of human-computer collaboration.  _Knowledge-Based Systems_ 8, 2 (Apr 1995), 67–81.  https://doi.org/10.1016/0950-7051(95)98369-H
  * Van der Maaten and Hinton (2008) Laurens Van der Maaten and Geoffrey Hinton. 2008.  Visualizing data using t-SNE.  _Journal of machine learning research_ 9, 11 (2008).
  * Wang et al. (2020) Dakuo Wang, Elizabeth Churchill, Pattie Maes, Xiangmin Fan, Ben Shneiderman, Yuanchun Shi, and Qianying Wang. 2020.  From Human-Human Collaboration to Human-AI Collaboration: Designing AI Systems That Can Work Together with People. In _Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems_ (Honolulu, HI, USA) _(CHI EA ’20)_. Association for Computing Machinery, New York, NY, USA, 1–6.  https://doi.org/10.1145/3334480.3381069
  * Yang et al. (2019) Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, Gustavo Hernandez Abrego, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2019\.  Multilingual Universal Sentence Encoder for Semantic Retrieval.  arXiv:1907.04307 [cs.CL]
  * Yianilos (1993) Peter N. Yianilos. 1993\.  Data Structures and Algorithms for Nearest Neighbor Search in General Metric Spaces. In _Proceedings of the Fourth Annual ACM-SIAM Symposium on Discrete Algorithms_ (Austin, Texas, USA) _(SODA ’93)_. Society for Industrial and Applied Mathematics, USA, 311–321.

##  Appendix A Appendix: Annotated Screenshots

Figure 4. Annotated screenshot from use case I of the question overview (left)
for selecting the starting point for the construction of a new class and the
manual cluster view (right) for iteratively selecting class candidates

The figure shows two screenshots and corresponding explanations of the first
prototype for the iterative taxonomy building (use case I). The interface
design is minimalist using mostly black, white and gray tones, red is used as
a highlight color. On the left side, a paginated list of question with an
additional search slot allows users to navigate the data set of questions and
select one of the questions as a prompt to start the taxonomy building
process. From this “starting point” the user switches to the second sceen
(displayed on the right side of the graphic). This screen shows the actual
interactive interface. It starts with a typical TSNE scatterplot that shows
all data points. The starting point is highlighted in red. Users can select
additional candidates of ‘similar’ items from this scatter plot or draw shapes
around additional points to be included as potential candidates for the new
taxonomy. The rest of the screen is split into four parts (organized in list
form): 1. “Cluster members”: include items selected for the new taxonomy, 2.
“Possible Candidates”: include suggestions from the distance analysis within
the embeddings, 3. “Possible candidates from draw selection”: include
candidates selected through the scatter plot interactions, 4. “Ignored
candidates”: include candidates that the user excluded from one of the
candidate lists. Each list is accompanied by buttons allowing the user to move
items between those lists. Thereby, users can iteratively refine the
suggestions by the system.

Figure 5. Annotated screenshot of the expert interface for use case II

The figure shows a screenshot with explanations for the various elements of
the user interface. The interface is also presented in a minimalist design,
using three shades of light blue, as well as black and white. Interactive
buttons are presented in grey. The interface consist of an upper and lower
area. In the upper area, 14 circular scatter plots, typical for TSNE
visualisations, are shown in two rows that each contain 7 “miniature” scatter
plots. The points in the plots are coloured based on an automated clustering
process. Three dots between the first and second row of 7 scatter plots
indicate that in the full interface there would be even more plots that the
user can chose from. Each plot represents one embedding process, a label
underneath each plot describes the model (e.g. universal sentence encoder,
multi language, cosine, 128 output vectors). The lower part shows a big TSNE
plot in the middle (which was selected from one of the “miniature” scatter
plots above). This big plot is surrounded by interactive panels. Starting with
a panel of system-generated clusters, each cluster has an id and a color,
corresponding to the currently selected model in the middle. Below the cluster
list is the currently selected cluster and the data items belonging to the
class. The items within this selected cluster are highlighted on the big as
well as on the “miniature” plots, allowing the user to compare the models. The
next panel shows more information (all attributes and id) on an individual
item selected by the user. Below are the neighbour items to that item
displayed. The last panel shows new clusters that were built by the user. To
build new clusters, each panel includes buttons that allow the user to select
all or individual items from a custom selection from the plot, neighbours or
existing clusters.