# Constrained Dynamic Movement Primitives for
Safe Learning of Motor Skills

Seiji Shaw†, Devesh K. Jha‡, Arvind U. Raghunathan‡, Radu Corcodel‡,
Diego Romeres‡, George Konidaris† and Daniel Nikovski‡ †Department of Computer
Science, Brown University, Providence, Rhode Island.
<EMAIL_ADDRESS>‡ Mitsubishi Electric Research Labs (MERL),
Cambridge, MA USA<EMAIL_ADDRESS>
###### Abstract

Dynamic movement primitives are widely used for learning skills which can be
demonstrated to a robot by a skilled human or controller. While their
generalization capabilities and simple formulation make them very appealing to
use, they possess no strong guarantees to satisfy operational safety
constraints for a task. In this paper, we present constrained dynamic movement
primitives (CDMP) which can allow for constraint satisfaction in the robot
workspace. We present a formulation of a non-linear optimization to perturb
the DMP forcing weights regressed by locally-weighted regression to admit a
Zeroing Barrier Function (ZBF), which certifies workspace constraint
satisfaction. We demonstrate the proposed CDMP under different constraints on
the end-effector movement such as obstacle avoidance and workspace constraints
on a physical robot. A video showing the implementation of the proposed
algorithm using different manipulators in different environments could be
found here https://youtu.be/hJegJJkJfys.

##  I Introduction

Programming robotic arm manipulators for tasks that require complex, dynamic
movements is often difficult and time-consuming. Learning from Demonstration
(LfD) offers a more efficient approach to teach motor skills to robots [1, 2],
since expert demonstrations can be directly used to learn a suitable
representation of the movement to solve the given task. However, it is
desirable that these approaches can provide some guarantees regarding
operational constraints, which are more difficult to specify from expert
demonstration.

One of the most popular LfD approaches is the dynamic movement primitive
(DMP), a nonlinear dynamical system that can fit expert demonstration
trajectories by decoupling a nonlinear forcing function from the nominal
attraction behavior. Since their introduction by Schaal et al, [3], DMPs have
been applied to learning a wide range of skills [4, 5, 6, 7, 8]. While they
can be reparameterized by their start and goal positions and preserve the
qualitative shape of the trajectory, incorporating arbitrary operational
constraints on the trajectory is very difficult. Without care, a new start or
goal pose may cause the robot to collide with itself or the environment.
Consequently, working with DMPs in arbitrary environments require careful
engineering to ensure that the computed trajectories stay collision-free and
within workspace bounds.

For dynamical systems like DMPs, safety can be defined by whether the
trajectories of the system remain within a specified ‘safety set.’ Such a set
can be defined by removing obstacles from the set that defines the robot’s
workspace or by joint inequality constraints derived joint limits on a
manipulator arm. Ames et al. [9] introduce Zeroing-Barrier Functions (ZBFs), a
formalism that can certify whether a dynamical system is forward-invariant in
a safety set, i.e. whether the state of the system is always in the safety
set. A key challenge is to find a suitable barrier function that can be used
to certify satisfaction of all constraints desired by the user.

We present Constrained DMPs (CDMPs) which allow constraint satisfaction for
the motor skills learned using DMPs. Given a user-defined ZBF that specifies
the safety set in the robot workspace, we optimize a set of parameters to the
DMP’s nonlinear forcing function to guarantee that the DMP trajectory lies
within the safety set. We cast the optimization as a nonlinear program which
can be solved using off-the-shelf non-linear program solvers such as IPOPT
[10, 11]. Figure 1 shows a motivating example where an initial demonstration
was provided in an obstacle-free environment. The DMP computed using the
demonstration collides with a novel obstacle during execution. The algorithm
proposed in this paper is able to compute a collision-free trajectory.

Figure 1: A simple block stacking environment using the proposed CDMP
algorithm. The demonstration (top row) was provided in a collision-free
environment. A box obstacle is introduced during execution and a simple DMP
collides (middle row) with the obstacle during execution. The proposed CDMP
(bottom row) is able to successfully avoid the novel obstacle.

Contributions. This paper presents the following contributions:

  1. 1.

A novel formulation for Constrained DMPs which can satisfy operational
constraints to ensure safe reproduction of motor skills using demonstrations.

  2. 2.

Demonstrations on several examples where we can successfully compute CDMPs
which satisfy operational constraints in the robot workspace.

Compared to the past work on incorporating collision avoidance constraints in
DMPs [12, 13, 14, 15], we are able to provide guarantees for safety set
invariance for the new trajectories, which encompass obstacle-avoidance,
workspace constraints, etc.

##  II Related Work

Learning from demonstration, or imitation learning, is an active area of
research in the robot learning community [1]. In general, there are two common
behavior representations that are trained by LfD: a stochastic policy or a
dynamical or geometric representation of a concrete trajectory. A case of the
latter, DMPs have been widely successful in learning motor skills for robots
because they present a simple dynamical system that can encode a wide range of
motor skills.

DMPs enjoy wide application in a variety of robot tasks, but are unable to
incorporate operational constraints with strong guarantees. Collision
avoidance is usually incorporated by combining DMPs with artificial potential
fields (or using other similar heuristics) [12, 13]. In general, the technique
lacks a formulation that can naturally incorporate any form of positional
constraint without the time-consuming experimental tuning processes or risks
of local minima that artificial potential field-like approaches possess ([13],
Sec. V-B). More recently, there has been some work on using DMPs with sampling
based methods to allow collision avoidance [16]. These methods use sampling to
find a state from which the DMP can avoid obstacles. In contrast, in the
proposed method, we perturb the learned forcing function to satisfy
constraints on the robot’s workspace.

We also draw inspiration from approaches in safety-critical control that uses
optimization-based approaches to constrain the control inputs of a dynamical
system to maintain strong guarantees of the system’s safety. At the center of
these approaches are control Lyapunov functions (CLFs) [17] and control
barrier functions (CBFs) [9, 18, 19], which certify convergence and safety set
forward-invariance, respectively. In a similar vein, Tedrake et al. [20]
compute Lyapunov-stable regions for locally-defined LQR controllers via sum-
of-squares optimization. Since a DMP is just a nonlinear dynamical system, we
leverage Ames et al.’s zeroing barrier function (ZBF) formalism (which Ames et
al. extends to control-affine systems to define CBFs) to certify the forward-
invariance of a DMP in a predetermined safety set [9, 18].

A key insight of our work is to perturb the weights of an already-learned DMP
to admit an already-existing ZBF so that the DMP is guaranteed to be forward-
invariant in a user-defined safety set. There is a large body of work that
constrains more general parameterizable dynamical systems via Lyapunov
functions [21, 22, 23] and contraction mappings [24]. While these techniques
guarantee convergence of the dynamical system, such methods cannot maintain
the DMP’s ability to generalize motions over changes in the attraction point,
a property that our approach preserves. Nomotista et al. [25] leverage a
diffeomorphism to map non-convex obstacles into a complex space, where they
constrain dynamics of a control system via CBF. Khansari-Zadeh et al. [26]
propose a real-time dynamical-reshaping approach for avoidance of convex
obstacles. While these approaches work for obstacles that follow their
assumptions, it is unclear if such an approach can be modified for avoidance
of non-compact sets as well. Ohnishi et al. [27] consider CBFs for learning
agents to avoid more abstract task-specific constraints for more agents whose
dynamics are learned via reinforcement learning. We solve a task-specific
learning-by-demonstration problem instead.

##  III Background

In this section, we review the mathematical details of DMPs and ZBFs relevant
to our construction of CDMP.

###  III-A Dynamic Movement Primitives

DMPs were first introduced by Schaal et al. [3]. To remove explicit time
dependency, they use a canonical system to keep track of the progress through
the learned behavior:

$\tau\dot{s}=-\alpha_{s}s$ (1)

where $s=1$ at the start of DMP execution (and $\alpha_{s}>0$) and $\tau>0$
specifies the rate of progress through the DMP.

To capture attraction behavior, DMPs use a spring-damper system (the
transformation system) with an added nonlinear forcing term. Writing the DMP
equations as a system of coupled first-order ODEs yields:

$\displaystyle\tau\dot{z}$ $\displaystyle=\alpha_{z}(\beta_{z}(g-y)-z)+f(s)$
(2) $\displaystyle\tau\dot{y}$ $\displaystyle=z$ (3)

where $g$ denotes the goal pose. The forcing term is defined as a radial-basis
function:

$\displaystyle f(s)$
$\displaystyle=\frac{\sum_{i=1}^{N}w_{i}\psi_{i}(s)}{\sum_{i=1}^{N}\psi_{i}(s)}$
(4) $\displaystyle\psi_{i}(s)$ $\displaystyle=\exp{(-h_{i}(s-c_{i})^{2})}$ (5)

where $h_{i}$ and $c_{i}$ denote the width and center of the Gaussian basis
functions, respectively. The forcing term is learned from the demonstration by
solving a locally weighted regression to fit the demonstration provided by an
expert.

###  III-B Zeroing Barrier Functions

Since DMPs are formulated as an autonomous nonlinear dynamical system, we can
certify them for safety set forward-invariance using Ames et al.’s zeroing
barrier function formalism [18, 9].

First, let $h(x):\mathbb{R}^{n}\to\mathbb{R}$ be a continuous differentiable
function. Define set $\mathcal{C}$ to be the following super-level set:

$\displaystyle\mathcal{C}$ $\displaystyle=\\{x\in\mathbb{R}^{n}:h(x)\geq 0\\}$
(6) $\displaystyle\partial\mathcal{C}$
$\displaystyle=\\{x\in\mathbb{R}^{n}:h(x)=0\\}$ (7)
$\displaystyle\text{Int}(\mathcal{C})$
$\displaystyle=\\{x\in\mathbb{R}^{n}:h(x)>0\\}$ (8)

Assume we have a nonlinear dynamical system of the form:

$\dot{x}=f(x)$ (9)

We call $h(x)$ ZBF if the following inequality holds:

$\dot{h}(x)\geq-\alpha(h(x))$ (10)

where $\alpha:\mathbb{R}\to\mathbb{R}$ is a class-$\mathcal{K}$ function [9].
Like Ames et al., we consider the specific case

$\dot{h}(x)\geq-\gamma h(x).$ (11)

with constant $\gamma>0$. Given a dynamical system (9), if we can find a valid
ZBF $h(x)$(or alternatively, change our dynamical system to admit an existing
ZBF), then our dynamical system $\dot{x}=f(x)$ is forward-invariant in the set
$\mathcal{C}$, i.e. if $x_{0}\in\mathcal{C}$ and $\dot{x}=f(x)$ then:

$x\in\mathcal{C},\mbox{ }\forall t\in[0,\infty)$ (12)

##  IV Constrained Dynamic Movement Primitives

We now present our formulation of the constrained DMP, which we cast as a
nonlinear optimization problem.

The primary insight behind our proposed formulation of CDMPs is to take an
existing DMP with a forcing function learned from an expert trajectory, and
then optimize perturbations to this forcing term (Sec. IV-A) so that the DMP
dynamical system admits a ZBF that certifies trajectory generation within a
pre-defined safety set. This safety set (and ZBF) is constructed by composing
signed-distance fields from primitive convex polytopes (Sec. IV-C).

###  IV-A CDMPs as a Nonlinear Optimization Problem

As was explained earlier in Section III-A, the system of equations and the
forcing function for DMP could be written and expressed by the following
system of equations:

$\begin{bmatrix}\dot{s}\\\ \dot{z}\\\ \dot{y}\\\
\end{bmatrix}=\frac{1}{\tau}\begin{bmatrix}-\alpha_{s}s\\\
\left(\alpha_{z}(\beta_{z}(g-y)-z)+f(s)\right)\\\ z\\\ \end{bmatrix}$ (13)

where

$f(s)=\frac{\sum_{i=1}^{N}w_{i}\psi_{i}(s)}{\sum_{i=1}^{N}\psi_{i}(s)}$ (14)

is the forcing function expressed as sum of radial basis functions and which
are learned from the provided expert demonstration.

We introduce additional parameters to the DMP dynamical systems that can be
optimized to allow constraint satisfaction. In the proposed formulation, CDMPs
are computed by optimizing a set of perturbations $\\{\zeta_{i}\\}$ of the
original (regressed) weights $\\{w_{i}\\}$ of the learning forcing function to
obey the ZBF inequality for forward-invariance in the user-defined safety set.
More formally, we define the perturbed forcing function as follows:

$\tilde{f}(s)=\frac{\sum_{i=1}^{N}(w_{i}-\zeta_{i})\psi_{i}(s)}{\sum_{i=1}^{N}\psi_{i}(s)}$
(15)

where $\zeta_{i}$ are the decision variables optimized for the formulated
optimization problem. We note that this is only one possible way to perturb
the DMP forcing function, which we based on its most common representation as
a radial-basis function in the DMP literature.

We now introduce the formulation of the nonlinear optimization problem to
compute the parameter set $p=\\{\zeta_{i}\\}$:

$\min_{x,p}\int_{t_{0}}^{t_{f}}c(x(t),p)dt+\phi(x(t_{f}),p)$ (16)

subject to the dynamic constraints

$\displaystyle\tau\begin{bmatrix}\dot{s}\\\ \dot{z}\\\ \dot{y}\\\
\end{bmatrix}+\begin{bmatrix}\alpha_{s}s\\\
-\left(\alpha_{z}(\beta_{z}(g-y)-z)+\tilde{f}(s)\right)\\\ -z\\\
\end{bmatrix}$ $\displaystyle=\mathbf{0}$ (17) $\displaystyle\dot{h}(x)+\gamma
h(x)$ $\displaystyle\geq 0$ (18) $\displaystyle x(t_{0})=x_{0}$ (19)

where $x=[s,z,y]^{T}$ and $p$ is the set of parameters (including the
$\zeta_{i}$). Note that the function $c(\cdot,\cdot,\cdot)$ represents the
cost functional for the optimization problem. The simplest cost function can
penalize the L2-norm of the decision variables $\zeta_{i}$.

The problem formulation represented by Equations (16)-(19) is converted to a
finite dimensional discretized problem using PyRoboCOP [11], which is then
transcribed into a nonlinear program. The resulting NLP is then solved using
IPOPT in the PyRoboCOP environment.

###  IV-B Perturbation from Original Trajectory

Another important consideration during computation of constrained DMP is to
limit the amount of the deviation of the CDMP from the original DMP
trajectory. The design of CDMP could be seen as a trade-off between constraint
satisfaction and the original forcing function. This trade-off could be
controlled using a hyperparameter which constrains the maximum allowable
deviation between the original DMP and CDMP, which can be added as another
constraint to the trajectory optimization problem. More formally, let us
denote the original DMP trajectory as $\\{\tilde{y}(t)\\}$,
$t\in[t_{0},t_{f}]$ and the hyperparameter for the deviation from the original
trajectory as $\epsilon$. Then, the additional constraint could be represented
as follows:

$||y(t)-\tilde{y}(t)||_{2}\leq\epsilon$ (20)

With this additional deviation constraint, the optimization problem can then
be written as follows:

$\min_{x,p}\int_{t_{0}}^{t_{f}}c(x(t),p)dt+\phi(x(t_{f}),p)$ (21)

s.t., (17)-(19)

$||y(t)-\tilde{y}(t)||_{2}\leq\epsilon$ (22)

Depending on the value of $\epsilon$, we can obtain a family of CDMPs that
will allow different amount of deviation of the new trajectory from the
original DMP trajectory.

###  IV-C ZBFs from Signed-Distance Functions

Given an obstacle set $\Omega$ with boundary $\partial\Omega$ in
$\mathbb{R}^{3}$, we use the usual definition of a signed distance function
(SDFs) $\sigma:\mathbb{R}^{3}\to\mathbb{R}$ to the boundary of the obstacle
[28]:

$\sigma(x)=\begin{cases}d(x,\partial\Omega),&\text{if }x\notin\Omega,\\\
-d(x,\partial\Omega),&\text{if }x\in\Omega.\end{cases}$

and where the distance (or metric) $d(x,\partial\Omega)$ is defined by

$d(x,\partial\Omega)=\inf_{y\in\partial\Omega}d(x,y).$

We know that $|\nabla\sigma|=1$ wherever this gradient is well-defined (and
for convex polytopes, this is true whenever $\sigma(x)>0$). We also know that
the implicit surface embedded in $\mathbb{R}^{3}$ defined by the set of all
points $x\in\mathbb{R}^{3}$ where $\sigma(x)=0$ is also the boundary of the
obstacle, and $\sigma(x)<0$ whenever $x$ is inside the obstacle. Thus, SDFs
fulfill the requirements for a function to designate a safety set (what we
denote as $h(x)$ above) and certify that a DMP trajectory is also invariant in
the safety set $\Omega^{c}$.

Given two SDFs $\sigma_{1}$ and $\sigma_{2}$ representing obstacles
$\Omega_{1}$ and $\Omega_{2}$, we would typically find the SDF of
$\Omega_{1}\cup\Omega_{2}$ by taking their minimum
$\sigma(p)=\min(\sigma_{1}(p),\sigma_{2}(p))$. However, this poses problems of
differentiability, which is necessary for IPOPT to solve our optimization
problem. We resolve this issue by computing a twice-differentiable lower-bound
approximation of the minimum function as shown in [29], which is necessary
because the ZBF constraint contains a first-order derivative of ZBF $h(x)$.

$\displaystyle\operatorname{smin}(d_{1},d_{2},k)=$
$\displaystyle\min(d_{1},d_{2})$
$\displaystyle-k\left(\frac{1}{6}\right)\left(\frac{\max(k-|d_{1}-d_{2}|,0.0)}{k}\right)^{3}$

where $k$ can be interpreted as a blending radius. With repeated application
of the $\operatorname{smin}$ operation, we can have a twice-differentiable SDF
union of all the obstacles known in the workspace.

We then state the following lemma, given the informal arguments above about
our current ZBF constructions:

######  Lemma IV.1.

Let $\dot{x}=g(x)$ be a DMP, $\gamma>0$, $h:\mathbb{R}^{3}\to\mathbb{R}$ be a
ZBF, and $\mathcal{C}=\\{x\in\mathbb{R}^{3}:h(x)>0\\}$. If there exists
$\zeta_{1},...,\zeta_{n}$ such that the constrained DMP $\dot{x}=\tilde{g}(x)$
admits the ZBF inequality $\dot{h}(x)+\gamma h(x)\geq 0$ along trajectory
$x(t)$, then CDMP $\tilde{g}$’s trajectory x(t) is forward-invariant in
$\mathcal{C}$.

###### Proof.

Application of Ames et al.’s Proposition B.1 [9]. ∎

As an implication of the above lemma, we can claim that if our solver is able
to find feasible solution to (16)- (19) to satisfy the ZBF condition, the
optimized CDMP trajectory is guaranteed to stay within the safety set.

##  V Experiments and Results

We tested our algorithm in numerical and physical experiments to understand
its efficacy. The numerical examples are used to verify that our method could
be used with a wide variety of ZBFs (and thus safety sets), test our method’s
performance against existing methods for set-invariance of DMPs, and
understand its computational efficiency. Our physical experiments showed
feasibility of our algorithm in a real-world setting. All the experiments
presented in the next Section are performed using a Python-based interface to
IPOPT presented in [11].

###  V-A Numerical Experiments

We first present numerical experiments to verify that CDMPs are forward-
invariant in the safety set as defined by the ZBF. We recorded an initial
demonstration end-effector trajectory kinesthetically taught by a human expert
in $\mathbb{R}^{3}$ (see Figure 2). Then, we impose a runtime constraint that
the robot trajectory needs to follow state constraints of the type
$x(t)\in(\mathcal{X}(t))^{c}$, where $\mathcal{X}(t)$ could be non-compact and
could be defined by a user depending on a task of interest. This kind of
constraint satisfaction arises in robotic tasks where the task-specific
constraint requires that the robot should always stay in a certain set during
the task. An example of such a task could be an assembly task where the end-
effector needs to always stay in a pre-defined set with respect to the part
positions. As could be seen in Figure 2, the CDMP trajectory is able to
satisfy the specified constraint. Furthermore, we can also easily specify and
satisfy time-dependent state bounds for the trajectory which might be useful
for a lot of tasks. However, we skip these results for brevity.

Figure 2: CDMP trajectory constrained to lie within a cone. The ZBF was
constructed using the SDFs of a circles arranged along an axis, their radii
paramaterized by $z$. (Best seen in color.)

###  V-B Comparison to Dynamic Artificial Potential Fields

We compare our method to the dynamic (velocity-dependant) artificial potential
field method introduced in [13] using a generated straight-line trajectory. As
with many potential field methods, it is not difficult to construct an
adversarial obstacle configuration that causes the DMP to be trapped in a
local minimum (see Fig. 3). Even with these obstacles, our CDMP method is able
to successfully reach the goal. This demonstrates that our method is
fundamentally different from DMPs perturbed with potential field methods, and
can allow for workspace constraint satisfaction in a more principled manner.

Figure 3: CDMP and DMP with a dynamic potential field learned from generated
straight-line demonstration. The former is able to reach the goal, while the
latter is caught in a local minimum. (Best seen in color.)

###  V-C Computational Efficiency

Apart from the complexity of the constraints, the computational time for CDMP
depends on the size of the NLP created using the original demonstration
trajectory. The size of the problem depends on the number of collocation
points for the trajectory. We note that constraints are only enforced at the
collocation points when the CDMP optimization problem is transformed into a
finite dimensional NLP, so the constraints may be violated between the
collocation points if the discretization is too coarse. We evaluate the
computational efficiency of our algorithm by varying the number of collocation
points during optimization (Fig. 4). All experiments were run using
unoptimized code on a Lenovo Thinkpad equipped with an Intel i7-8565U
processor and 16Gb of RAM. While fewer collocation points result in faster
compute times, the trajectory in between collocation points is more likely to
intersect with obstacles (Fig. 5). As a result of this trade-off, the correct
number of collocation points should be chosen based on the problem. Figure 4
shows a non-linear trend between the number of collocation points and the
computational time. This is not surprising since we usie an interior-point
method (IPM) for a non-convex optimization, where the computational time can
increase sharply with problem size.

Figure 4: Experimental runtime of wall-jump problem. Figure 5: While $n=10$
collocation points was much faster to compute (Fig. 4), the trajectory was
still in collision with the obstacle since the ZBF was too coarsely enforced
relative to $n=50$. Figure 6: This shows movement of pieces on a chess board
using CDMP. The top row shows that the original DMP collides with an obstacle
on the way while the CDMP (bottom row) is able to avoid the obstacle during
movement.

###  V-D Physical Robot Experiments

We tested our CDMP on two different physical robots in three different
domains: a wall-hopping domain (see Figure 1), a chess-piece moving domain
(see Figure 6), and a whiteboard drawing domain (see Figure 7). We test the
overall efficacy of our approach in the wall-hopping and chess-moving domains,
and the drawing domain tests the CDMP’s ability to preserve the qualitative
geometry of the original DMP trajectory. A video showing the implementation of
the proposed algorithm in these environments could be found here
https://youtu.be/hJegJJkJfys.

In the wall and chess domains, we use a Kinova Gen3 robot. In these
experiments, a user kinesthetically demonstrates the robot a movement in a
workspace free of obstacles. To test the CDMP’s ability to generate collision-
free trajectories, we introduce new obstacles in the environment. The
environment for the wall-hopping domain can be seen in Figure 1, where the
CDMP finds a collision-free trajectory. Similarly, for the chessboard
environment in Figure 6, we provided an initial demonstration to move the
knight piece. While DMP (top row) fails in the presence of a new obstacle,
CDMP (bottom row) is able to compute a new non-colliding trajectory. More
experiments are shown in the supplementary video.

For the drawing experiments, we use a Mitsubishi Electric Assista industrial
manipulator arm equipped with a force/torque (F/T) sensor at the wrist. The
F/T sensor is used to design an admittance controller [30] to move the robot
in a kinesthetic teaching mode for contact-rich tasks. This controller is used
to provide demonstrations on a white-board to draw several shapes such as a
rectangle and a triangle (see Figure 7). Then, we add some constraint sets in
the original shapes, which the robot should avoid while trying to re-create
the shape. The initial demonstration and the final CDMP trajectories for the
experiments could be seen in Figure 7. In all cases, the CDMP is able to find
feasible solutions while preserving the geometric shape of the drawings in
both the experiments.

(a) Drawing a rectangle.

(b) Drawing a triangle.

Figure 7: The proposed CDMP algorithm is successfully able to draw
demonstrated shapes in the presence of constraints.

##  VI Conclusion and Future Work

In this paper, we propose a method for incorporating operational constraints
while generating trajectories for generalizing to novel initial and goal
locations. Our algorithm, constrained dynamic movement primitives, can
incorporate task and environmental constraints when generalizing to novel
conditions. The proposed CDMP was demonstrated on several examples for
collision avoidance in the presence of obstacles of different shapes and
sizes, and also preserves the qualitative geometry of the trajectory.

Even though the formulation can, in theory, include complex operational
constraints like self-collision, joint limits, etc., we need computationally
efficient and smooth barrier function representation of these constraints to
be able to compute safe trajectories. In the future, we would like to use the
proposed formulation to consider more constraints past end-effector safety
sets that can be more easily expressed in different parameterizations of the
robot configuration space, such as joint limits, self-collision avoidance,
etc. These constraints result in a complex, constrained optimization problem
which needs additional, non-trivial work to find a suitable zeroing barrier
function. Another limitation is the computational efficiency with respect to
the size of the problem. In this paper, we use IPOPT to solve the resulting
optimization which can not make efficient use of warm starting. In our future
work, we plan to integrate our problem with an active-set solver to improve
the computational efficiency.

## References

  * [1] B. D. Argall, S. Chernova, M. Veloso, and B. Browning, “A survey of robot learning from demonstration,” _Robotics and autonomous systems_ , vol. 57, no. 5, pp. 469–483, 2009.
  * [2] H. Ravichandar, A. S. Polydoros, S. Chernova, and A. Billard, “Recent advances in robot learning from demonstration,” _Annual Review of Control, Robotics, and Autonomous Systems_ , vol. 3, pp. 297–330, 2020.
  * [3] S. Schaal, “Dynamic movement primitives-a framework for motor control in humans and humanoid robotics,” in _Adaptive motion of animals and machines_. Springer, 2006, pp. 261–280.
  * [4] M. Saveriano, F. J. Abu-Dakka, A. Kramberger, and L. Peternel, “Dynamic movement primitives in robotics: A tutorial survey,” _arXiv preprint arXiv:2102.03861_ , 2021.
  * [5] K. Muelling, J. Kober, and J. Peters, “Learning table tennis with a mixture of motor primitives,” in _2010 10th IEEE-RAS International Conference on Humanoid Robots_ , 2010, pp. 411–416.
  * [6] K. Mülling, J. Kober, O. Kroemer, and J. Peters, “Learning to select and generalize striking movements in robot table tennis,” _The International Journal of Robotics Research_ , vol. 32, no. 3, pp. 263–279, 2013\.
  * [7] D. K. Jha, D. Romeres, W. Yerazunis, and D. Nikovski, “Imitation and supervised learning of compliance for robotic assembly,” in _2022 European Control Conference (ECC)_ , 2022, pp. 1882–1889.
  * [8] M. Sharma, K. Zhang, and O. Kroemer, “Learning semantic embedding spaces for slicing vegetables,” _arXiv preprint arXiv:1904.00303_ , 2019.
  * [9] A. D. Ames, X. Xu, J. W. Grizzle, and P. Tabuada, “Control barrier function based quadratic programs for safety critical systems,” _IEEE Transactions on Automatic Control_ , vol. 62, no. 8, pp. 3861–3876, 2016.
  * [10] A. Wächter and L. T. Biegler, “On the implementation of an interior-point filter line-search algorithm for large-scale nonlinear programming,” _Mathematical programming_ , vol. 106, no. 1, pp. 25–57, 2006.
  * [11] A. U. Raghunathan, D. K. Jha, and D. Romeres, “PYROBOCOP : Python-based robotic control & optimization package for manipulation and collision avoidance,” _CoRR_ , vol. abs/2106.03220, 2021.
  * [12] A. Rai, F. Meier, A. Ijspeert, and S. Schaal, “Learning coupling terms for obstacle avoidance,” in _2014 IEEE-RAS International Conference on Humanoid Robots_. IEEE, 2014, pp. 512–518.
  * [13] D.-H. Park, H. Hoffmann, P. Pastor, and S. Schaal, “Movement reproduction and obstacle avoidance with dynamic movement primitives and potential fields,” in _Humanoids 2008-8th IEEE-RAS International Conference on Humanoid Robots_. IEEE, 2008, pp. 91–98.
  * [14] H. Tan, E. Erdemir, K. Kawamura, and Q. Du, “A potential field method-based extension of the dynamic movement primitive algorithm for imitation learning with obstacle avoidance,” in _2011 IEEE International Conference on Mechatronics and Automation_. IEEE, 2011, pp. 525–530.
  * [15] F. Stulp, E. Oztop, P. Pastor, M. Beetz, and S. Schaal, “Compact models of motor primitive variations for predictable reaching and obstacle avoidance,” in _2009 9th IEEE-RAS International Conference on Humanoid Robots_. IEEE, 2009, pp. 589–595.
  * [16] S. Sobti, R. Shome, S. Chaudhuri, and L. E. Kavraki, “A sampling-based motion planning framework for complex motor actions,” in _2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2021, pp. 6928–6934.
  * [17] A. D. Ames, K. Galloway, K. Sreenath, and J. W. Grizzle, “Rapidly exponentially stabilizing control lyapunov functions and hybrid zero dynamics,” _IEEE Transactions on Automatic Control_ , vol. 59, no. 4, pp. 876–891, 2014.
  * [18] A. D. Ames, S. Coogan, M. Egerstedt, G. Notomista, K. Sreenath, and P. Tabuada, “Control barrier functions: Theory and applications,” in _2019 18th European control conference (ECC)_. IEEE, 2019, pp. 3420–3431.
  * [19] M. Z. Romdlony and B. Jayawardhana, “Uniting control lyapunov and control barrier functions,” in _53rd IEEE Conference on Decision and Control_. IEEE, 2014, pp. 2293–2298.
  * [20] R. Tedrake, I. R. Manchester, M. Tobenkin, and J. W. Roberts, “Lqr-trees: Feedback motion planning via sums-of-squares verification,” _The International Journal of Robotics Research_ , vol. 29, no. 8, pp. 1038–1052, 2010\.
  * [21] S. M. Khansari-Zadeh and A. Billard, “Learning stable nonlinear dynamical systems with gaussian mixture models,” _IEEE Transactions on Robotics_ , vol. 27, no. 5, pp. 943–957, 2011.
  * [22] ——, “Learning control lyapunov function to ensure stability of dynamical system-based robot reaching motions,” _Robotics and Autonomous Systems_ , vol. 62, no. 6, pp. 752–765, 2014.
  * [23] N. Figueroa and A. Billard, “A physically-consistent bayesian non-parametric mixture model for dynamical system learning,” in _Proceedings of The 2nd Conference on Robot Learning_ , ser. Proceedings of Machine Learning Research, A. Billard, A. Dragan, J. Peters, and J. Morimoto, Eds., vol. 87. PMLR, 29–31 Oct 2018, pp. 927–946. [Online]. Available: https://proceedings.mlr.press/v87/figueroa18a.html
  * [24] H. Ravichandar, I. Salehi, and A. Dani, “Learning partially contracting dynamical systems from demonstrations,” in _Conference on Robot Learning_. PMLR, 2017, pp. 369–378.
  * [25] G. Notomista and M. Saveriano, “Safety of dynamical systems with multiple non-convex unsafe sets using control barrier functions,” _IEEE Control Systems Letters_ , vol. 6, pp. 1136–1141, 2021.
  * [26] S. M. Khansari-Zadeh and A. Billard, “A dynamical system approach to realtime obstacle avoidance,” _Autonomous Robots_ , vol. 32, no. 4, pp. 433–454, 2012\.
  * [27] M. Ohnishi, G. Notomista, M. Sugiyama, and M. Egerstedt, “Constraint learning for control tasks with limited duration barrier functions,” _Automatica_ , vol. 127, p. 109504, 2021.
  * [28] M. W. Jones, J. A. Baerentzen, and M. Sramek, “3d distance fields: A survey of techniques and applications,” _IEEE Transactions on visualization and Computer Graphics_ , vol. 12, no. 4, pp. 581–599, 2006.
  * [29] I. Quilez, “3d distance functions.” [Online]. Available: https://iquilezles.org/www/articles/distfunctions/distfunctions.htm
  * [30] D. K. Jha, D. Romeres, S. Jain, W. Yerazunis, and D. Nikovski, “Design of adaptive compliance controllers for safe robotic assembly,” _arXiv preprint arXiv:2204.10447_ , 2022.


# Improved Membership Inference Attacks against Language Classification Models
††thanks: This work was performed as part of the NEMECYS project, which is co-
funded by the European Union under grant agreement ID 101094323, by UK
Research and Innovation (UKRI) under the UK government’s Horizon Europe
funding guarantee grant numbers 10065802, 10050933 and 10061304, and by the
Swiss State Secretariat for Education, Research and Innovation (SERI).

Shlomit Shachor & Natalia Razinkov & Abigail Goldsteen
Data Security and Privacy
IBM Research
Haifa, Israel
<EMAIL_ADDRESS>
###### Abstract

Artificial intelligence systems are prevalent in everyday life, with use cases
in retail, manufacturing, health, and many other fields. With the rise in AI
adoption, associated risks have been identified, including privacy risks to
the people whose data was used to train models. Assessing the privacy risks of
machine learning models is crucial to enabling knowledgeable decisions on
whether to use, deploy, or share a model. A common approach to privacy risk
assessment is to run one or more known attacks against the model and measure
their success rate. We present a novel framework for running membership
inference attacks against classification models. Our framework takes advantage
of the ensemble method, generating many specialized attack models for
different subsets of the data. We show that this approach achieves higher
accuracy than either a single attack model or an attack model per class label,
both on classical and language classification tasks.

##  1 Introduction

Artificial intelligence (AI) systems have become prevalent in everyday life.
AI is used in retail, security, manufacturing, health, finance, and many more
sectors to improve or even replace existing processes. However, with the rise
in AI adoption, different risks associated with AI have been identified,
including privacy risks to the people whose data was used to train the models.
In addition to fundamental societal harm, these risks can result in negative
brand reputation, lawsuits, and fines. This has given rise to the notion of
Trustworthy or Responsible AI.

A key aspect of Responsible AI is the ability to assess (and later mitigate)
these risks. Assessing the privacy risk of machine learning (ML) models is
crucial to enable well-informed decision-making about whether to use a model
in production, share it with third parties, or deploy it in customers’ homes.
The most prevalent approach to privacy risk assessment is to run one or more
known attacks against the model and measure how successful they are in leaking
personal information.

The most common attack used in model assessment is called _membership
inference_. Membership inference (MI) attacks aim to violate the privacy of
individuals whose data was used in training an ML model by attempting to
distinguish between samples that were part of a target model’s training data
(called members) and samples that were not (non-members), based on the model’s
outputs. These can be class probabilities or logits (for classification
models), the model’s loss, or activations from internal layers of the model
(in white-box attacks). Most attacks choose one or more of these features and
train a binary classifier to try to distinguish between members and non-
members.

In this paper, we present a novel framework for MI attacks against
classification models that takes advantage of the ensemble method to generate
many specialized attack models for different subsets of the data. We show that
this approach outperforms existing approaches based on a single attack model
or an attack model per class label, both on classical and language
classification tasks.

In the realm of large language models (LLM), membership inference can be
assessed for different phases of the model’s development, namely the pre-
training and fine-tuning stages. Pre-training is largely performed on publicly
available datasets, and the data used to train a model is often also public
knowledge. Fine-tuning is typically performed on a smaller, proprietary
dataset. It is therefore more common to look at the fine-tuning phase in the
context of MI attacks. However, our framework is not limited to this scenario
and can also evaluate pre-trained models if necessary.

Moreover, our approach is adapted to work with both language models that have
an explicit classification head and generative models that can respond to
classification prompts or instructions, as can be seen in the evaluation
section with the flan-UL2 model.

Our method can cater for both privacy audit mode, in which an organization
assesses the privacy vulnerability of their own models, and attack mode, where
the real training data is unknown to the attacker. For the latter, a preceding
step of generating shadow models and data is required.

This paper starts by surveying relevant prior work in Section 2. Next, we
describe our framework for improved membership inference based on small
specialized attack models in Section 3. We present our evaluation results in
Section 4. We discuss those results in Section 5 and conclude in Section 6.

##  2 Related Work

There are several types of privacy (inference) attacks against ML models,
including membership inference, attribute inference, model inversion, database
reconstruction, and most recently, training data extraction from generative
models. The most commonly researched and employed attack is the membership
inference attack, with dozens of papers published each year Hu et al. (2022),
and implementations being made available in open-source privacy assessment
frameworksKumar & Shokri (2020), Nicolae et al. (2018).

MI attacks attempt to distinguish between members, which were part of a target
model’s training data, and non-members. MI attacks have been extensively
studied in the context of classification models and in the black-box setting,
where the model internals are unknown to the attacker. The first MI attacks
were either threshold-based Yeom et al. (2018) or employed binary classifiers
trained to distinguish between members and non-members based on model outputs
Shokri et al. (2017). For example, these outputs may include class
probabilities or logits (for classification models), the model’s loss, and
possibly also activations from internal layers of the model (in white-box
attacks) Nasr et al. (2019). To generate labeled (member/non-member) data to
train the attack classifier, without knowledge of the true member samples of
the attacked model, shadow models are commonly used Shokri et al. (2017).

In the past few years, investigations have begun into MI in the context of
large language models (LLM), starting with embedding models and masked
language models Song & Raghunathan (2020); Mahloujifar et al. (2021);
Mireshghallah et al. (2022). Shejwalkar et al. (2021) looked at a similar
setting as ours, focusing on NLP classification models. They proposed mostly
threshold-based attacks, examining different features that can be used to
distinguish between members and non-members. Jagannatha et al. (2021) focused
specifically on language models that were fine-tuned for the medical domain,
including classification tasks such as MedNLI, employing both black-box and
white-box attacks. Their black-box attack applied thresholds to the training
error of samples. More recently, Likelihood Ratio Attacks (LiRA) have been
proposed, which compare target model scores to those obtained from a reference
model trained on similar data Carlini et al. (2022). Mattern et al. (2023)
tried to relieve the assumption that an adversary has access to samples
closely resembling the original training data by utilizing synthetically
generated neighbor texts.

Some works on MI in the language domain differentiate between sample-level MI,
which treats each text sequence/document in the training data separately, and
user-level MI, which groups together samples originating from the same person
or source. In this work, we focus solely on the sample level, which is usually
considered a harder problem.

Most existing approaches to MI that employ a classification model use either a
single attack model for the entire dataset or a separate attack model per
class. Ensembles have been used in a few cases in the context of adversarial
(evasion) attacks to generate more robust adversarial examples Cai et al.
(2023), Fu & Cui (2023). Salem et al. (2018) used multiple shadow models to
compensate for a lack of knowledge of the target model algorithm; however,
these multiple models were only used to generate multiple shadow datasets,
which were then combined to train the attack model.

##  3 The Framework

We propose a method for performing membership inference attacks by splitting
the initial member and non-member datasets into multiple, small non-
overlapping subsets, used to train different attack models. Thus, multiple
specialized attack models are generated for small pieces of the data, where
each model is best in identifying membership leakage for that piece. To find
the best possible attack model for each subset, many different combinations of
model type, scaling, and input features are tried. The best combination is
selected, based on highest attack accuracy. Aggregating the results from those
multiple attack models can better reveal the real leakage of the target model.

The source of the member and non-member data input to this process is
irrelevant and can come from either shadow datasets (in attack mode) or from
the actual training and test sets of the target model (in audit mode).

Our method can be used for any model that can perform classification tasks.
This includes classical ML models, such as a decision tree or random forest,
language classification models, and even generative models that were fine-
tuned for text classification tasks.

###  3.1 Use of small specialized attacks

The high-level flow of the proposed framework is depicted in Figure 1. The
first step is to split the member and non-member datasets into non-overlapping
subsets and randomly assign member/non-member pairs from those subsets. In
subsequent phases, each pair serves to generate a specialized attack model
(the pairs remain constant).

Each pair is then split into two halves, one for fitting the attack model and
the second for inferring membership. This splitting process is done multiple
times for each pair. Our experiments show that even for the same pair of
member and non-member subsets, the half used for training the attack model has
significant impact on the model’s ability to infer membership. Thus, we
perform this split multiple times, eventually using the attack that achieves
the best result (highest leakage). This allows us to generate attack models
that are even more specialized and powerful.

For each split, the data is passed through the target model and various
features calculated based on the model outputs. The features may include
predicted labels, losses, class probabilities or class-scaled probabilities,
entropy, modified entropy Shejwalkar et al. (2021), etc. The list of features
to use in each attack may be pre-configured or optimized based on the best
accuracy for each pair.

The features are then scaled using different types of scalers (e.g., robust,
min-max) and then used to train multiple types of binary classifiers (e.g.,
random forest, k-nearest neighbors, decision tree, etc.) with possibly
different combinations of input features. The specific combination of scaler,
classifier, and features is selected to achieve the best attack accuracy
(worst-case privacy leakage) for each pair. This process yields different
attack models (based on different combinations) for each subset. The results
of all of these attacks are aggregated by averaging the results, so that it
fairly represents the accuracy on all of the data assessed.

In addition, this whole flow can be performed multiple times (called
instances) on different random samples of the entire member and non-member
datasets. The sample size can be substantially smaller than the entire dataset
size, which is especially beneficial when the datasets are very large. This
improves the performance of the overall assessment process, while providing
good coverage of the data. For each instance of the flow that is run, the
results of all subset pairs are averaged, and finally the results of all
instances are aggregated. This aggregation can be done in different ways. In
the evaluation section, we show aggregation based on average attack results;
in the appendix, we show it based on best attack results.

Even though our main goal is to perform privacy evaluation of models (audit
mode), the framework can also be used in attack mode where the true membership
status of samples is unknown when performing inference. In this case, typical
ensemble aggregation methods, such as majority voting, can be used. The
parameters controlling the process such as the subset size, number of subsets,
number of runs per subset, number of instances, and type of aggregation can be
easily configured.

Figure 1: High-level overview of the framework for small specialized MIA
models.

##  4 Evaluation

Since we are targeting the privacy audit scenario and want to simplify the
evaluation, we use the known training and test data of the model when
conducting our experiments. All results presented in this section were
performed in the same manner to enable a fair comparison.

We experimented with several LLM architectures and datasets: two
classification models from huggingface - textattack/bert-base-uncased-SST-2,
fine-tuned on glue-SST2111https://huggingface.co/datasets/glue/viewer/sst2 (BB
SST2), and textattack/roberta-base-CoLA, fine-tuned on glue-
CoLA222https://huggingface.co/datasets/glue/viewer/cola (RB CoLA); one
generative model - google/flan-ul2 trained with glue-CoLA (F-UL2 CoLA) and
glue-SST2 (F-UL2 SST2); and a roberta-base model fined-tuned with the rotten
tomatoes dataset333https://huggingface.co/datasets/rotten_tomatoes (RB RT). To
evaluate the effectiveness of our method on models with a privacy defense, we
also evaluated the roberta-base model after applying differentially private
fine-tuning using DP LoRA Yu et al. (2022) with $\epsilon=2$ (RB-DP RT).
Details on the accuracy of each of these models can be found in Appendix A.

We compare our method both to training a single attack model on the entire
dataset and class-based attacks, where a separate attack model is trained per
class label.

In our experiments, we set the subset size to 50, the number of runs to 5, the
number of instances to 50, and the size of the member and non-member samples
for each instance to 1000 each (872 for SST-2). For each model and its
corresponding data, we conducted six experiments: using a single attack model
for the entire dataset (S-CL01) and a single model for class 0 (S-CL0) and for
class 1 (S-CL1); and using many small specialized attack models for the entire
dataset (M-CL01), for class 0 (M-CL0), and for class 1 (M-CL1). In these
experiments, we used the following features to train the attack models: true
labels, predicted labels, class-scaled probabilities, class-scaled logits,
losses, and modified entropy.

For google/flan-UL2, a generative model, we used commonly available prompts
for CoLA and SST2 (presented in Appendix B), requesting the model to classify
the linguistic correctness of a sentence or its sentiment, respectively. We
instructed the model to generate a score structure instead of just text, and
used low temperature mode to ensure determinism. The score structure was used
to calculate the features mentioned above (e.g., probabilities and entropy).
In addition, we calculated the perplexity for each of the choices (”positive”
and ”negative” for SST2) and used it as an additional feature.

###  4.1 Results

The average accuracy and area under the receiver operating characteristic
curve (AUC-ROC) scores across all instances are presented in Table 1. Best
scores across all instances are presented in Appendix C.

Our experiments show improvements of between 3%-12% in accuracy and AUC-ROC
for all models, datasets, and attacks tested, even when compared with class-
based methods. Surprisingly, for the defended model (RB-DP RT), even though
previous attacks seem mostly mitigated with scores ranging from 0.52 to 0.55
(which is very close to random guessing), our attack is still able to achieve
a significant advantage of up to 0.65.

It is also worth noting that the many small attack method outperforms class-
based attacks (S-CL0, S-CL1) by 3-10% even when applied to the whole dataset,
regardless of class (CL01). In some cases, the combination of many attacks
with the class-based approach provides an additional 1% gain.

Table 1: Average accuracy (AUC-ROC) across all instances for single vs. many attack models Model | S-CL01 | M-CL01 | S-CL0 | M-CL0 | S-CL1 | M-CL1
---|---|---|---|---|---|---
RB CoLA | 0.58 (0.59) | 0.65 (0.66) | 0.66 (0.67) | 0.69 (0.71) | 0.56 (0.57) | 0.63 (0.64)
BB SST2 | 0.56 (0.57) | 0.64 (0.65) | 0.59 (0.6) | 0.64 (0.66) | 0.55 (0.56) | 0.62 (0.64)
F-UL2 CoLA | 0.57 (0.58) | 0.66 (0.67) | 0.61 (0.65) | 0.67 (0.68) | 0.56 (0.57) | 0.65 (0.66)
F-UL2 SST2 | 0.61 (0.63) | 0.66 (0.68) | 0.61 (0.62) | 0.65 (0.67) | 0.63 (0.67) | 0.67 (0.7)
RB RT | 0.64 (0.66) | 0.68 (0.71) | 0.64 (0.66) | 0.68 (0.71) | 0.66 (0.68) | 0.7 (0.73)
RB-DP RT | 0.52 (0.53) | 0.64 (0.65) | 0.54 (0.54) | 0.63 (0.64) | 0.54 (0.55) | 0.63 (0.64)

##  5 Discussion

In this framework and its evaluation, several design choices were made based
on logical or performance reasons. For example, the pairs of member and non-
member subsets were assigned randomly and fixed throughout the experiment
(within a single instance). Testing all possible combinations of pair
assignments to find the best match would likely increase the attack’s success
rate even more. Another possibility is analyzing the dataset to try to find
characteristics that can be leveraged when splitting the data and assigning
pairs.

Moreover, when combining the results of the different subsets, we always used
averaging to enable a fair comparison between the attacks run on the entire
dataset and the small attacks. However, it is also possible to choose the
worst subset.

Varying the input features to the attack did not have a significant effect on
the success rate. Rather, the main advantage stems from the use of many
different attacks for different data subsets and the specialization of the
attack models.

##  6 Conclusion and Future Work

We presented a novel method for running membership inference attacks that
divides the data into small subsets and trains specialized attack models for
each subset. This method significantly outperforms attack models trained on
the entire data or per class label. It can be applied to both classical models
as well as large language models that perform classification tasks, and even
succeeds in attacking models defended using differential privacy.

During our experimentation, we saw indications that the prompt used when
assessing generative models has a significant effect on the success rate of
the attack. We plan to investigate this further and perhaps add it as an
additional source of variability in the framework.

Moreover, we plan to check the viability of this approach for other types of
privacy attacks such as attribute inference and other types of models besides
classification.

## References

  * Cai et al. (2023) Zikui Cai, Yaoteng Tan, and M Salman Asif.  Ensemble-based blackbox attacks on dense prediction.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 4045–4055, 2023.
  * Carlini et al. (2022) Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer.  Membership inference attacks from first principles.  In _2022 IEEE Symposium on Security and Privacy (SP)_ , pp. 1897–1914. IEEE, 2022.
  * Fu & Cui (2023) Zhongwang Fu and Xiaohui Cui.  Elaa: An ensemble-learning-based adversarial attack targeting image-classification model.  _Entropy_ , 25(2):215, 2023.
  * Hu et al. (2022) Hongsheng Hu, Zoran Salcic, Lichao Sun, Gillian Dobbie, Philip S. Yu, and Xuyun Zhang.  Membership inference attacks on machine learning: A survey.  _ACM Comput. Surv._ , 54(11s), sep 2022.  ISSN 0360-0300.  doi: 10.1145/3523273.  URL https://doi.org/10.1145/3523273.
  * Jagannatha et al. (2021) Abhyuday Jagannatha, Bhanu Pratap Singh Rawat, and Hong Yu.  Membership inference attack susceptibility of clinical language models.  _arXiv preprint arXiv:2104.08305_ , 2021.
  * Kumar & Shokri (2020) Sasi Kumar and Reza Shokri.  Ml privacy meter: Aiding regulatory compliance by quantifying the privacy risks of machine learning.  In _Workshop on Hot Topics in Privacy Enhancing Technologies (HotPETs)_ , 2020.
  * Mahloujifar et al. (2021) Saeed Mahloujifar, Huseyin A Inan, Melissa Chase, Esha Ghosh, and Marcello Hasegawa.  Membership inference on word embedding and beyond.  _arXiv preprint arXiv:2106.11384_ , 2021.
  * Mattern et al. (2023) Justus Mattern, Fatemehsadat Mireshghallah, Zhijing Jin, Bernhard Schölkopf, Mrinmaya Sachan, and Taylor Berg-Kirkpatrick.  Membership inference attacks against language models via neighbourhood comparison.  _arXiv preprint arXiv:2305.18462_ , 2023.
  * Mireshghallah et al. (2022) Fatemehsadat Mireshghallah, Kartik Goyal, Archit Uniyal, Taylor Berg-Kirkpatrick, and Reza Shokri.  Quantifying privacy risks of masked language models using membership inference attacks.  _arXiv preprint arXiv:2203.03929_ , 2022.
  * Nasr et al. (2019) Milad Nasr, Reza Shokri, and Amir Houmansadr.  Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning.  In _2019 IEEE symposium on security and privacy (SP)_ , pp. 739–753. IEEE, 2019.
  * Nicolae et al. (2018) Maria-Irina Nicolae, Mathieu Sinn, Minh Ngoc Tran, Beat Buesser, Ambrish Rawat, Martin Wistuba, Valentina Zantedeschi, Nathalie Baracaldo, Bryant Chen, Heiko Ludwig, Ian Molloy, and Ben Edwards.  Adversarial robustness toolbox v1.2.0.  _CoRR_ , 1807.01069, 2018.  URL https://arxiv.org/pdf/1807.01069.
  * Salem et al. (2018) Ahmed Salem, Yang Zhang, Mathias Humbert, Pascal Berrang, Mario Fritz, and Michael Backes.  Ml-leaks: Model and data independent membership inference attacks and defenses on machine learning models.  _arXiv preprint arXiv:1806.01246_ , 2018.
  * Shejwalkar et al. (2021) Virat Shejwalkar, Huseyin A Inan, Amir Houmansadr, and Robert Sim.  Membership inference attacks against nlp classification models.  In _NeurIPS 2021 Workshop Privacy in Machine Learning_ , 2021.
  * Shokri et al. (2017) R. Shokri, M. Stronati, C. Song, and V. Shmatikov.  Membership inference attacks against machine learning models.  In _2017 IEEE Symposium on Security and Privacy (SP)_ , pp. 3–18, Los Alamitos, CA, USA, may 2017. IEEE Computer Society.  doi: 10.1109/SP.2017.41.  URL https://doi.ieeecomputersociety.org/10.1109/SP.2017.41.
  * Song & Raghunathan (2020) Congzheng Song and Ananth Raghunathan.  Information leakage in embedding models.  In _Proceedings of the 2020 ACM SIGSAC conference on computer and communications security_ , pp. 377–390, 2020.
  * Yeom et al. (2018) Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha.  Privacy risk in machine learning: Analyzing the connection to overfitting.  In _2018 IEEE 31st Computer Security Foundations Symposium (CSF)_ , pp. 268–282, 2018.  doi: 10.1109/CSF.2018.00027.
  * Yu et al. (2022) Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A. Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, Sergey Yekhanin, and Huishuai Zhang.  Differentially private fine-tuning of language models.  In _ICLR_ , 2022.

##  Appendix A Details on tested models

Table 2 presents the accuracy (rounded to three digits) of all evaluated
models for the test and training data. For the SST2 and CoLA datasets, we used
the validation set as test data for our evaluation, due to a lack of true
labels in the test datasets.

Table 2: Accuracy of the evaluated models on train and test data Model | Train | Test
---|---|---
RB CoLA | 0.948 | 0.85
BB SST2 | 0.986 | 0.924
F-UL2 CoLA | 0.93 | 0.864
F-UL2 SST2 | 0.96 | 0.964
RB RT | 1.0 | 0.877
RB-DP RT | 0.917 | 0.876

##  Appendix B Prompts used for performing classification tasks in
google/flan-UL2

For CoLA: ”Sentence: {sentence}. Would a linguist rate this sentence to be
acceptable linguistically? Options: acceptable, unacceptable. Answer:”

For SST2: ”Sentence: {sentence}. What is the sentiment of this sentence?
Options: positive, negative. Answer:”.

##  Appendix C Best attack results across instances

Table 3 and Table 4 present the best accuracy and AUC-ROC scores of all the
instances in our experiments.

Table 3: Best accuracy across all instances for single vs. many small attack models Model | S-CL01 | M-CL01 | S-CL0 | M-CL0 | S-CL1 | M-CL1
---|---|---|---|---|---|---
RB CoLA | 0.6 | 0.66 | 0.7 | 0.73 | 0.58 | 0.65
BB SST2 | 0.58 | 0.66 | 0.61 | 0.67 | 0.58 | 0.66
F-UL2 CoLA | 0.59 | 0.68 | 0.66 | 0.71 | 0.58 | 0.68
F-UL2 SST2 | 0.63 | 0.68 | 0.64 | 0.69 | 0.67 | 0.71
RB RT | 0.66 | 0.7 | 0.68 | 0.72 | 0.68 | 0.72
RB-DP RT | 0.54 | 0.66 | 0.57 | 0.66 | 0.57 | 0.67
Table 4: Best AUC-ROC across all instances for single vs. many small attack models Model | S-CL01 | M-CL01 | S-CL0 | M-CL0 | S-CL1 | M-CL1
---|---|---|---|---|---|---
RB CoLA | 0.62 | 0.68 | 0.72 | 0.76 | 0.62 | 0.68
BB SST2 | 0.61 | 0.68 | 0.64 | 0.69 | 0.6 | 0.67
F-UL2 CoLA | 0.61 | 0.69 | 0.67 | 0.73 | 0.61 | 0.69
F-UL2 SST2 | 0.66 | 0.72 | 0.67 | 0.72 | 0.7 | 0.75
RB RT | 0.68 | 0.73 | 0.69 | 0.74 | 0.71 | 0.75
RB-DP RT | 0.56 | 0.68 | 0.58 | 0.67 | 0.59 | 0.68