Contraction Theory with Inequality Constraints

Winfried Lohmiller and Jean-Jacques Slotine

Nonlinear Systems Laboratory

Massachusetts Institute of Technology

Cambridge, Massachusetts, 02139, USA

{wslohmil<EMAIL_ADDRESS>
###### Abstract

This paper extends continuous contraction theory of nonlinear dynamical
systems to systems with nonlinear inequality constraints. It shows that the
contraction behaviour of the constrained dynamics is given by the covariant
derivative of the system dynamics from the original contraction theorem [5],
plus the second covariant derivative of the active inequality constraint. It
is shown that this approach works both for a step Lagrangian constraint term
at a first-order system as well for a Dirac Lagrangian constraint term for a
second-order system.

Practical applications include controllers constrained to an operational
envelope, trajectory control with moving obstacles, and a classical Lagrangian
interpretation of the single and two slit experiments of quantum mechanics.

##  1 Introduction

This paper extends contraction theory [5] of unconstrained $n$-dimensional
nonlinear dynamics in covariant form,

${\bf M}({\bf x},t)\ \dot{\bf x}={\bf f}({\bf x},t)$ (1)

where ${\bf M}({\bf x},t)$ is a Riemannian metric, to systems with
$j=1,...,J$-dimensional inequality constraints

$g_{j}({\bf x},t)\leq 0$ (2)

Note that the above ’covariant’ form [9] corresponds to the $n$-dimensional
’contravariant’ [9] dynamics

$\dot{\bf x}\ =\ {\bf f}^{\prime}({\bf x},t)$

with ${\bf f}={\bf M}{\bf f}^{\prime}$, which is used in the original
contravariant contraction theorem [5]. This paper will use covariant
coordinates since the constraint terms introduced later will be covariant.
While both representations are fully equivalent, in this case covariant
coordinates are simpler.

This paper computes continuous-time contraction rates in section 2.1 for ${\bf
M}={\bf I}$ and in section 2.2 for a general metric ${\bf M}({\bf x},t)$.
Finally the result is applied to second-order collsion dynamics in section
2.3. The discussion is illustrated with simple applications, including

  * •

trajectory control considering moving obstacles as constraint

  * •

second-order controller operation in a constrained operational envelope

  * •

a Lagrangian collision interpretation of the single and double slit experiment
of quantum mechanics

A discrete-time version of the approach is derived in [8], along with an
application to piece-wise linear neuronal networks.

##  2 Continuous-time constrained dynamics

A constraint $j$ in equation (2) can have only an impact on the dynamics (1)
if the inequality turns in an equality $g_{j}({\bf x},t)=0$ , leading to the
following definition.

###### Definition 1

The set of active constraints $\mathcal{A}({\bf x},t)\subset\\{1,...,J\\}$ (2)
contains the elements $j$ which are on the boundary of the original constraint

$g_{j}({\bf x},t)=0$

The constrained dynamic equations (1, 2) are then of the form [2]

${\bf M}\dot{\bf x}={\bf f}({\bf x},t)+\sum_{all\
j\in\mathcal{A}}\frac{\partial g_{j}}{\partial{\bf x}}\lambda_{j}$ (3)

All constraints $g_{j},j\in\mathcal{A}$ are not violated at $t+dt$ if [2]

$\displaystyle\dot{g}_{j}=\frac{\partial{g}_{j}}{\partial{\bf x}}^{T}{\bf
M}^{-1}\left({\bf f}({\bf x},t)+\sum_{all\ j\in\mathcal{A}}\frac{\partial
g_{j}}{\partial{\bf x}}\lambda_{j}\right)+\frac{\partial{g}_{j}}{\partial t}$
$\displaystyle\leq$ $\displaystyle 0$ (4)

leading to

###### Definition 2

The set of Lagrange multipliers $\lambda_{j},j\in\mathcal{A}({\bf x},t)$ of
(2) are defined as

$\displaystyle\lambda_{j}$ $\displaystyle\leq$
$\displaystyle-\sum_{k\in\mathcal{A}({\bf
x},t)}\frac{\frac{\partial{g}_{k}}{\partial{\bf x}}^{T}{\bf M}^{-1}{\bf
f}({\bf x},t)+\frac{\partial{g}_{k}}{\partial
t}}{\frac{\partial{g}_{j}}{\partial{\bf x}}^{T}{\bf M}^{-1}\frac{\partial
g_{k}}{\partial{\bf x}}}$

where we assume without loss of generality

  * •

that all corners are obtuse, i.e. $\frac{\partial{g}_{j}}{\partial{\bf
x}}^{T}{\bf M}^{-1}\frac{\partial g_{k}}{\partial{\bf x}}({\bf x},t)\geq 0$
and

  * •

that $\frac{1}{\frac{\partial{g}_{j}}{\partial{\bf x}}^{T}{\bf
M}^{-1}\frac{\partial g_{k}}{\partial{\bf x}}}$ is replaced with the Moore-
Penrose inverse [10] if $\frac{\partial{g}_{j}}{\partial{\bf x}}^{T}{\bf
M}^{-1}\frac{\partial g_{k}}{\partial{\bf x}}$ does not have full rank.

Note that possible acute corners can be locally avoided by replacing the acute
corner with two obtuse corners at the same ${\bf x},t$. Also note that
alternatively the linear perogramming (LP) definition of [7] may be used.

Let us now introduce a virtual displacement between two neighbouring
trajectories, constrained by $g_{j}({\bf x},t)=0$. This virtual displacement
has to be parallel to $g_{j}({\bf x},t)=0$, i.e. orthogonal to the normals
$\frac{\partial g_{j}}{\partial{\bf x}}$, which implies on the constraint

$\displaystyle{\bf\delta x}$ $\displaystyle=$ $\displaystyle{\bf
G}_{\parallel}({\bf x},t)\ {\bf\delta x}^{\ast}$ $\displaystyle{\bf
G}_{\parallel}^{T}{\bf G}_{\parallel}$ $\displaystyle=$ $\displaystyle{\bf I}$
$\displaystyle{\bf G}_{\parallel}^{T}\ \frac{\partial g_{j}}{\partial{\bf x}}$
$\displaystyle=$ $\displaystyle{\bf 0}\ \forall j\in\mathcal{A}({\bf x},t))$
(5)

where ${\bf\delta x}^{\ast}$ is the reduced virtual displacement, which is of
dimension $n$ minus the number of active constraints in $\mathcal{A}$. Note
that all neighbouring trajectory steps $(\dot{\bf x}+\delta\dot{\bf x})dt$
around the main trajectory step $\dot{\bf x}dt$ arrive at the same time on the
constraint since ($\delta\dot{\bf x}\ dt$) is a second-order term. Hence all
neighbouring trajectories are constrained at the same time with (5).

The following two subsections analyse the contraction behaviour first for
${\bf M}={\bf I}$ and then for the general case.

###  2.1 Basic case

The constrained dynamic equations (3) can be rewritten for ${\bf M}={\bf I}$
as

$\dot{\bf x}={\bf f}({\bf x},t)+\sum_{all\ j\in\mathcal{A}({\bf
x},t)}\frac{\partial g_{j}}{\partial{\bf x}}\lambda_{j}={\bf f}({\bf
x},t)+\sum_{j=1}^{J}\frac{\partial g_{j}}{\partial{\bf
x}}step(g_{j})\lambda_{j}$

with the Heaviside step function and Dirac impulse

$\displaystyle\theta(g_{j})$ $\displaystyle=$
$\displaystyle\left\\{\begin{array}[]{l}0\text{\ \ for \ \ }g_{j}<0\\\
1\text{\ \ for \ \ }g_{j}=0\\\ undefined\text{\ \ for \ \
}g_{j}>0\end{array}\right.$ (9) $\displaystyle\delta(g_{j})$ $\displaystyle=$
$\displaystyle\frac{\partial step(g_{j})}{\partial g_{j}}$ (10)

whose variation is

$\displaystyle{\bf\delta\dot{x}}\ $ $\displaystyle=$
$\displaystyle\frac{\partial{\bf f}}{\partial{\bf x}}{\bf\delta
x}+\sum_{j=1}^{J}\left(\left(\frac{\partial^{2}g_{j}}{\partial{\bf
x}^{2}}\lambda_{j}+\frac{\partial g_{j}}{\partial{\bf
x}}\frac{\partial\lambda_{j}}{\partial{\bf
x}}^{T}\right)\theta(g_{j})+\frac{\partial g_{j}}{\partial{\bf
x}}\delta(g_{j})\lambda_{j}\frac{\partial g_{j}}{\partial{\bf
x}}^{T}\right){\bf\delta x}$

Multiplying the above with $dt$ exactly at an activation of a constraint leads
with $\lambda_{j}dt=\frac{dg_{j}}{\frac{\partial{g}_{j}}{\partial{\bf
x}}^{T}\frac{\partial g_{j}}{\partial{\bf x}}}$ of Definition 2 to

$\displaystyle{\bf\delta{x}}_{t+dt}\ $ $\displaystyle=$
$\displaystyle\left({\bf I}-\sum_{allj\in\mathcal{A}}\frac{\frac{\partial
g_{j}}{\partial{\bf x}}\frac{\partial g_{j}}{\partial{\bf
x}}^{T}}{\frac{\partial{g}_{j}}{\partial{\bf x}}^{T}\frac{\partial
g_{k}}{\partial{\bf x}}}\ \right){\bf\delta{x}}_{t}+dt\left(\frac{\partial{\bf
f}}{\partial{\bf
x}}+\sum_{allj\in\mathcal{A}}\left(\frac{\partial^{2}g_{j}}{\partial{\bf
x}^{2}}\lambda_{j}+\frac{\partial g_{j}}{\partial{\bf
x}}\frac{\partial\lambda_{j}^{T}}{\partial{\bf x}}\right)\right){\bf\delta x}$

Performing the activation of the constraints sequentially (i.e. at least in
infinitesimal time steps) we can see that the virtual displacement ${\bf\delta
x}$ are set to ${\bf G}_{\parallel}{\bf\delta x}^{\ast}$ when a constraint is
activated. The other terms can be neglected at the constraint activation since
they are multiplied with $dt$. Now the squared virtual length dynamics can be
computed outside the activation of a constraint as

$\displaystyle\frac{1}{2}\ \frac{d}{dt}\left({\bf\delta x}^{T}{\bf\delta
x}\right)$ $\displaystyle=$ $\displaystyle\ {\bf\delta x}^{\ast T}{\bf
G}_{\parallel}^{T}\left(\frac{\partial{\bf f}}{\partial{\bf
x}}+\sum_{allj\in\mathcal{A}}\frac{\partial^{2}g_{j}}{\partial{\bf
x}^{2}}\lambda_{j}\right)_{H}{\bf G}_{\parallel}{\bf\delta x}^{\ast}$

where we used ${\bf G}_{\parallel}^{T}\frac{\partial g_{j}}{\partial{\bf
x}}\frac{\partial\lambda_{j}}{\partial{\bf x}}^{T}={\bf 0}$ since on the
constraint the first two terms vanish and outside the constraint the last term
vanishes.

Thus the dynamics of ${\bf\delta x}^{T}{\bf\delta x}$ is composed of
exponentially convergent continuous segments and an enforcement of ${\bf\delta
x}$ to ${\bf G}_{\parallel}{\bf\delta x}^{\ast}$ at the activation of a
constraint.

###  2.2 More general metrics

We now extend the results of section 2.1 to a general metric ${\bf M}({\bf
x},t)$, rather than identity. We recall the standard definition [9] of the
covariant derivative operator $\nabla_{H}$ in a Riemann space with an
underlying metric ${\bf M}({\bf x},t)$.

###### Definition 3

The covariant derivatives of a scalar $a({\bf x},t)$ or a vector ${\bf a}({\bf
x},t)$ are defined as

$\displaystyle\nabla_{M}\ a$ $\displaystyle=$ $\displaystyle\frac{\partial
a}{\partial{\bf x}}\ =\ \nabla a$ $\displaystyle(\nabla_{M}\ {\bf a})_{ij}$
$\displaystyle=$ $\displaystyle\frac{\partial{\bf a}_{i}}{\partial{\bf
x}_{j}}-\sum_{k=1}^{n}\gamma_{ij}^{k}\ {\bf a}_{k}$

where $\ \gamma_{ij}^{k}\ $ is the Christoffel term for the metric ${\bf
M}({\bf x})$,

$\gamma_{ij}^{k}({\bf x},t)=\frac{1}{2}\sum_{l=1}^{n}\left(\frac{\partial
M_{il}}{\partial x^{j}}+\frac{\partial M_{jl}}{\partial x^{i}}-\frac{\partial
M_{ij}}{\partial x^{l}}\right)M_{kl}^{-1}$

The constrained dynamic equations (3) can be rewritten as

${\bf M}\dot{\bf x}={\bf f}({\bf x},t)+\sum_{all\
j\in\mathcal{A}}\frac{\partial g_{j}}{\partial{\bf x}}\lambda_{j}={\bf f}({\bf
x},t)+\sum_{j=1}^{J}\frac{\partial g_{j}}{\partial{\bf
x}}step(g_{j})\lambda_{j}$

whose variation is with (10)

$\displaystyle\frac{\partial{\bf M}}{\partial{\bf x}}\ {\bf\delta x}\ \dot{\bf
x}+{\bf M}{\bf\delta\dot{x}}$ $\displaystyle=$
$\displaystyle\frac{\partial{\bf f}}{\partial{\bf x}}{\bf\delta
x}+\sum_{j=1}^{J}\left(\left(\frac{\partial^{2}g_{j}}{\partial{\bf
x}^{2}}\lambda_{j}+\frac{\partial g_{j}}{\partial{\bf
x}}\frac{\partial\lambda_{j}}{\partial{\bf
x}}\right)\theta(g_{j})+\frac{\partial g_{j}}{\partial{\bf
x}}\delta(g_{j})\lambda_{j}\frac{\partial g_{j}}{\partial{\bf
x}}\right){\bf\delta x}$

Multiplying the above with $dt$ exactly at an activation of a constraint leads
with $\lambda_{j}dt=\frac{dg_{j}}{\frac{\partial{g}_{j}}{\partial{\bf
x}}^{T}{\bf M}^{-1}\frac{\partial g_{j}}{\partial{\bf x}}}$ of Definition 2 to

$\displaystyle\frac{\partial{\bf M}}{\partial{\bf x}}\ {\bf\delta x}\ \dot{\bf
x}dt+{\bf M}{\bf\delta{x}}_{t+dt}$ $\displaystyle=$ $\displaystyle\left({\bf
M}-\sum_{allj\in\mathcal{A}}\frac{\frac{\partial g_{j}}{\partial{\bf
x}}\frac{\partial g_{j}}{\partial{\bf
x}}^{T}}{\frac{\partial{g}_{j}}{\partial{\bf x}}^{T}{\bf M}^{-1}\frac{\partial
g_{j}}{\partial{\bf x}}}\ \right){\bf\delta{x}}_{t}+dt\left(\frac{\partial{\bf
f}}{\partial{\bf
x}}+\sum_{allj\in\mathcal{A}}\left(\frac{\partial^{2}g_{j}}{\partial{\bf
x}^{2}}\lambda_{j}+\frac{\partial g_{j}}{\partial{\bf
x}}\frac{\partial\lambda_{j}^{T}}{\partial{\bf x}}\right)\right){\bf\delta x}$

Performing the activation of the constraints sequentially (i.e. at least in
infinitesimal time steps) we can see that the virtual displacement ${\bf\delta
x}$ are set to ${\bf G}_{\parallel}{\bf\delta x}^{\ast}$ when a constraint is
activated. The other terms can be neglected at the constraint activation since
they are multiplied with $dt$. Now the squared virtual length dynamics can be
computed outside the activation of a constraint as

$\displaystyle\frac{1}{2}\ \frac{d}{dt}\left({\bf\delta x}^{T}{\bf
M}{\bf\delta x}\right)$ $\displaystyle=$ $\displaystyle\ {\bf\delta x}^{\ast
T}{\bf G}_{\parallel}^{T}\left(\nabla_{M}\ {\bf
f}+\frac{1}{2}\frac{\partial{\bf M}}{\partial t}+\sum_{all\
j\in\mathcal{A}}\lambda_{j}\nabla_{M}^{2}g_{j}\right)_{H}{\bf
G}_{\parallel}{\bf\delta x}^{\ast}$ (11)

where we used ${\bf G}_{\parallel}^{T}\frac{\partial g_{j}}{\partial{\bf
x}}\frac{\partial\lambda_{j}}{\partial{\bf x}}={\bf 0}$ since on the
constraint the first two terms vanish and outside the constraint the last term
vanishes.

Thus the dynamics of ${\bf\delta x}^{T}{\bf\delta x}$ is composed of
exponentially convergent continuous segments and an enforcement of ${\bf\delta
x}$ to ${\bf G}_{\parallel}{\bf\delta x}^{\ast}$ at the activation of a
constraint.

Summarizing the above leads to:

###### Theorem 1

Consider the continuous dynamics

${\bf M}\dot{\bf x}={\bf f}({\bf x},t)+\sum_{all\
j\in\mathcal{A}}\lambda_{j}\frac{\partial g_{j}}{\partial{\bf x}}$ (12)

within the metric ${\bf M}({\bf x},t)$ constrained by $j=1,...,J$ inequality
constraints

$g_{j}({\bf x},t)\leq 0$

equivalent to the subset $\mathbb{G}^{n}\subset\mathbb{R}^{n}$. The set of
active constraints $\mathcal{A}$ and Lagrange multipliers $\lambda_{j}$ are
given in Definition 1 and 2.

The distance $s=\min\int_{{\bf x}(s)={\bf x}_{1}}^{{\bf
x}_{2}}\sqrt{\delta{\bf x}^{T}{\bf M}\delta{\bf x}}$ within $\mathbb{G}^{n}$
from any trajectory ${\bf x}_{1}(t)$ to any other trajectory ${\bf x}_{2}(t)$
converges exponentially to $0$ with an exponential convergence rate
$\leq\max_{along\ s}(\lambda_{\max}({\bf x},t)),\geq\min_{along\
s}(\lambda_{\min}({\bf x},t))$ with

$\lambda_{\min}{\bf G}_{\parallel}^{T}{\bf M}{\bf G}_{\parallel}\leq{\bf
G}_{\parallel}^{T}\left(\nabla_{M}\ {\bf f}+\frac{1}{2}\frac{\partial{\bf
M}}{\partial t}+\sum_{all\
j\in\mathcal{A}}\lambda_{j}\nabla_{M}^{2}g_{j}\right)_{H}{\bf
G}_{\parallel}\leq\lambda_{\max}{\bf G}_{\parallel}^{T}{\bf M}{\bf
G}_{\parallel}$ (13)

with the covariant derivative of Definition 3 and the constraint tangential
space ${\bf G}_{\parallel}$ from equation (5).

In addition the activation of a constraint discontinuously sets the virtual
displacement ${\bf\delta x}$ to ${\bf G}_{\parallel}{\bf\delta x}^{\ast}$.

Note that the inequality constraint in Theorem 13 can be replaced with an
equality constraint if also the corresponding inequality in Definition 2 is
replaced with an equality.

Also note that all lemmas of the original contraction theorem in [5] as
adaptive control, observer design, combination principles etc. can be applied
to Theorem 13 as well.

In addition note that [11] assessed the contraction behavior of nonlinear
equality constraints. Initial principles for convex constraints and
contraction theory were illustrated in [6, 12]. The following examples
illustrate the application of Theorem 13 to different constrained dynamical
systems:

  * ###### Example 2.1

:  Consider the inequality constraint in Figure 1

$g_{j}=x_{1}^{2}+x_{2}\leq 0$

Figure 1: Inequality constraint

Regions of different signs of $g_{j}$ are shown in Figure 1, along with the
gradient

$\frac{\partial g_{j}}{\partial{\bf x}}=\left(\begin{array}[]{c}2x_{1}\\\
1\end{array}\right)$

The constrained virtual displacement (5) is

${\bf\delta x}=\frac{1}{\sqrt{4x_{1}^{2}+1}}\left(\begin{array}[]{c}-1\\\
2x_{1}\end{array}\right)\delta x^{\ast}$

We consider now the divergent flow field

$\left(\begin{array}[]{c}\dot{x}_{1}\\\
\dot{x}_{2}\end{array}\right)=\left(\begin{array}[]{c}x_{1}\\\
x_{2}+1\end{array}\right)$

We compute the Lagrange multiplier of Definition 2 on the parabolic constraint
$x_{1}^{2}=-x_{2}$ as

$\displaystyle\lambda_{j}\ $ $\displaystyle=$
$\displaystyle-\left(\frac{\partial g_{j}}{\partial{\bf x}}^{T}\frac{\partial
g_{j}}{\partial{\bf x}}\right)^{-1}\left(\frac{\partial g_{j}}{\partial{\bf
x}}^{T}{\bf f}+\frac{\partial g_{j}}{\partial t}\right)$ (17) $\displaystyle=$
$\displaystyle-\frac{1}{4x_{1}^{2}+1}\left(\begin{array}[]{cc}2x_{1}&1\end{array}\right)\left(\begin{array}[]{c}x_{1}\\\
x_{2}+1\end{array}\right)=-\frac{x_{1}^{2}+1}{4x_{1}^{2}+1}\leq 0$

I.e. the trajectories stay on the parabola once they moved on the parabola.

We can bound with Theorem 13 the contraction behaviour (13) of the constrained
dynamics (12)

$\displaystyle\frac{1}{2}\ \frac{d}{dt}\left({\bf\delta x}^{T}{\bf\delta
x}\right)$ $\displaystyle\leq$ $\displaystyle{\bf\delta
x}^{T}\left(\left(\begin{array}[]{cc}1&0\\\
0&1\end{array}\right)+\lambda_{j}\left(\begin{array}[]{cc}2&0\\\
0&0\end{array}\right)\right){\bf\delta x}$ (22)

The second term is only active on the 1-dimensional parabola $\mathcal{A}$ and
decreases the divergence rate on the parabola. $\Box$

  * ###### Example 2.2

:  Consider a moving circular obstacle

$g_{j}=-(x_{1}-a(t))^{2}-(x_{2}-b(t))^{2}+1\leq 0$

The gradient at the constraint is

$\frac{\partial g_{j}}{\partial{\bf x}}=-2\left(\begin{array}[]{c}x_{1}-a\\\
x_{2}-b\end{array}\right)$

The constrained virtual displacement (5) is hence

${\bf\delta
x}=\frac{1}{\sqrt{(x_{1}-a)^{2}+(x_{2}-b)^{2}}}\left(\begin{array}[]{c}x_{2}-b\\\
-(x_{1}-a)\end{array}\right)\delta x^{\ast}$

We consider now a robot which moves according to the convergent flow field

$\left(\begin{array}[]{c}\dot{x}_{1}\\\
\dot{x}_{2}\end{array}\right)=-\left(\begin{array}[]{c}x_{1}\\\
x_{2}\end{array}\right)$

We compute the Lagrange multiplier of Definition 2 on the circular constraint
$(x_{1}-a)^{2}+(x_{2}-b)^{2}=1$ as

$\displaystyle\lambda_{j}$ $\displaystyle=$
$\displaystyle\min\left(-\left(\frac{\partial g_{j}}{\partial{\bf
x}}^{T}\frac{\partial g_{j}}{\partial{\bf x}}\right)^{-1}\left(\frac{\partial
g_{j}}{\partial{\bf x}}^{T}{\bf f}+\frac{\partial g_{j}}{\partial
t}\right),0\right)$ (26) $\displaystyle=$
$\displaystyle\min\left(-\frac{1}{2}\frac{1}{(x_{1}-a)^{2}+(x_{2}-b)^{2}}\left(\begin{array}[]{cc}x_{1}-a&x_{2}-b\end{array}\right)\left(\begin{array}[]{c}x_{1}+\dot{a}\\\
x_{2}+\dot{b}\end{array}\right),0\right)$

We can upper and low bound with Theorem 13 the contraction behaviour (13) of
the constrained dynamics (12) as

$\frac{1}{2}\ \frac{d}{dt}\left({\bf\delta x}^{T}{\bf\delta
x}\right)=-(1+2\lambda_{j})\ {\bf\delta x}^{T}{\bf\delta x}$

The flow is globally contracting with $-(1+2\lambda_{j})$. Neighbouring
trajectories exponentially converge with $e^{-\int(1+2\lambda_{j})dt}$.

Figure 2 shows such a moving circle with velocity
$v=\sqrt{\dot{a}^{2}+\dot{b}^{2}}$. The blue flow field is the unconstrained
dynamics. The red flow is the Lagrangian normalisation and the black flow
field, from which the contraction behaviour is computed in (13), is the flow
on the constraint. $\Box$

Figure 2: Contraction flow field around moving circle

###  2.3 Hamiltonian collision dynamics

In the following we show that the Lagrangian step function (10) in the normal
direction $\frac{\partial g_{j}}{\partial{\bf x}}$ of the $j=1,...,J$
inequality constraints

$g_{j}({\bf q},t)\leq 0$

in the Hamiltonian dynamics of Theorem 13

$\displaystyle\dot{\bar{\bf p}}$ $\displaystyle=$
$\displaystyle-\frac{\partial h}{\partial{\bf q}}$ $\displaystyle\dot{\bf q}$
$\displaystyle=$ $\displaystyle\frac{\partial h}{\partial\bar{\bf
p}}+\sum_{j=1}^{J}\lambda_{j}\theta(g_{j})\frac{\partial g_{j}}{\partial{\bf
q}}$

with $h=V({\bf q},t)+\frac{1}{2}\bar{\bf p}^{T}{\bf H}({\bf q})\bar{\bf p}$ is
equivalent to a Dirac constraint force in the momentum dynamics. In the above
the Langrangian $\lambda_{j}$ is a velocity increment according to Definition
2

$\displaystyle\lambda_{j}$ $\displaystyle\leq$
$\displaystyle-\sum_{k\in\mathcal{A}({\bf
x},t)}\frac{\frac{\partial{g}_{k}}{\partial{\bf x}}^{T}\bar{\bf
p}+\frac{\partial{g}_{k}}{\partial t}}{\frac{\partial{g}_{j}}{\partial{\bf
x}}^{T}\frac{\partial g_{k}}{\partial{\bf x}}}$

The equal sign holds for a plastic collision, where the orthogonal velocity
component to the constraint is set to zero. The $<$ holds for a partially
elastic collision where the trajectory bounces back orthogonal to the
constraint.

The position dynamics of the above is equivalent to the position dynamics of

$\displaystyle\dot{\bf p}$ $\displaystyle=$ $\displaystyle-\frac{\partial
h}{\partial{\bf
q}}+\sum_{j=1}^{J}\left(\lambda_{j}\delta(g_{j})\dot{g}_{j}\frac{\partial
g_{j}}{\partial{\bf q}}+\theta(g_{j})\frac{d}{dt}(\lambda_{j}\frac{\partial
g_{j}}{\partial{\bf q}})\right)$ $\displaystyle\dot{\bf q}$ $\displaystyle=$
$\displaystyle\frac{\partial h}{\partial\bar{\bf p}}$

At the time instance of the collision the term
$\frac{d}{dt}(\lambda_{j}\frac{\partial g_{j}}{\partial{\bf q}})dt$ is
neglegible. Outside the collisiton time instance the constraint force
vanishes, i.e. the last term above can be neglected. Summarizing the above
leads to:

###### Theorem 2

The Hamiltonian dynamics

$\displaystyle\dot{\bar{\bf p}}$ $\displaystyle=$
$\displaystyle-\frac{\partial h}{\partial{\bf q}}$ $\displaystyle\dot{\bf q}$
$\displaystyle=$ $\displaystyle\frac{\partial h}{\partial\bar{\bf
p}}+\sum_{j=1}^{J}\lambda_{j}\theta(g_{j})\frac{\partial g_{j}}{\partial{\bf
q}}$ (27)

with $h=V({\bf q},t)+\frac{1}{2}\bar{\bf p}^{T}{\bf H}({\bf q})\bar{\bf p}$
constrained by $j=1,...,J$ inequality constraints

$g_{j}({\bf q},t)\leq 0$ (28)

with set of active constraints $\mathcal{A}$ and Lagrange multipliers
$\lambda_{j}$ in Definition 1 and 2 is equivalent to

$\displaystyle\dot{\bf p}$ $\displaystyle=$ $\displaystyle-\frac{\partial
h}{\partial{\bf
q}}+\sum_{j=1}^{J}\lambda_{j}\delta(g_{j})\dot{g}_{j}\frac{\partial
g_{j}}{\partial{\bf q}}$ $\displaystyle\dot{\bf q}$ $\displaystyle=$
$\displaystyle\frac{\partial h}{\partial{\bf p}}$ (29)

where (27) has a step velocity increment whereas (29) has a Dirac force (see
(10)) at the collision time instance.

The following examples illustrate the above for typical constrained
Hamiltonian dynamics:

  * ###### Example 2.3

:  Consider the second-order dynamics

$\displaystyle\left(\begin{array}[]{c}\dot{v}\\\
\dot{x}\end{array}\right)=\left(\begin{array}[]{cc}-1&-1\\\
1&0\end{array}\right)\left(\begin{array}[]{c}v\\\
x\end{array}\right)-\left(\begin{array}[]{c}u(t)\\\ 0\end{array}\right)$ (38)

constrained with a partially elastic collision forcce of Theorem 2 to an
operational envelope of

$-2<x<2\Leftrightarrow\ g_{1}=x-2<0,\ g_{2}=-x+2<0$

Contraction behaviour can e.g. been seen in the metric ${\bf
M}=\left(\begin{array}[]{cc}2&1\\\ 1&2\end{array}\right)$ in which the
constrained dynamics of Theorem 13 is

$\displaystyle{\bf M}\left(\begin{array}[]{c}\dot{v}\\\
\dot{x}\end{array}\right)=\left(\begin{array}[]{cc}-1&-2\\\
1&-1\end{array}\right)\left(\begin{array}[]{c}v\\\
x\end{array}\right)-\left(\begin{array}[]{c}2\\\
1\end{array}\right)u(t)+\left(\begin{array}[]{c}0\\\
1\end{array}\right)\lambda_{j}\theta(g_{j})$ (49)

The contraction rate is here given by the symmetric part of the Jacobian

$\det\left(\left(\begin{array}[]{cc}-1&-2\\\ 1&-1\end{array}\right)_{s}-{\bf
M}\lambda\right)=0$

which implies the contraction rate $\lambda=-\frac{1}{2}$. The Lagrange
multiplier of Definition 2 is $\lambda_{j}=-\frac{3}{2}v$ for a plastic
collision. Fur a fully elastic collision it would be $\lambda_{j}=-3v$. The
dynamics (49) is illustrated in Figure 3, where the unconstrainted dynamics is
blue and the constraint dynamics is red.

Figure 3: Constrained second-order dynamics with a plastic velocity increment
at the constraint

We assess now the equivalent position dynamics with a Dirac constraint force
by multiplying from the left with ${\bf M}^{-1}$ and using Theorem 2

$\displaystyle\left(\begin{array}[]{c}\dot{v}\\\ \dot{x}\end{array}\right)$
$\displaystyle=$ $\displaystyle\left(\begin{array}[]{cc}-1&-1\\\
1&0\end{array}\right)\left(\begin{array}[]{c}v\\\
x\end{array}\right)-\left(\begin{array}[]{c}u(t)\\\
0\end{array}\right)+\frac{1}{3}\left(\begin{array}[]{cc}2&-1\\\
-1&2\end{array}\right)\left(\begin{array}[]{c}0\\\
1\end{array}\right)\lambda_{j}\theta(g_{j})$ (62) $\displaystyle=$
$\displaystyle\left(\begin{array}[]{cc}-1&-1\\\
1&0\end{array}\right)\left(\begin{array}[]{c}v\\\
x\end{array}\right)-\left(\begin{array}[]{c}u(t)\\\
0\end{array}\right)+\left(\begin{array}[]{c}-\frac{1}{3}\theta(g_{j})+\frac{2}{3}\delta(g_{j})\dot{g}_{j}\\\
0\end{array}\right)\lambda_{j}$ (71)

with the plastic Dirac impulse force $\lambda_{j}\delta(g_{j})\dot{g}_{j}$
whose transfered impulse over the collistion time instance is
$\lambda_{j}\delta(g_{j})\dot{g}_{j}dt=\lambda_{j}$. The collision dynamics
(71) with a plastic collision velocity increment $\lambda_{j}=-\frac{3}{2}v$
is shown in figure 3. The equivalent plastic force dynamics is shown in figure
4.

The fully elastic collision with $\lambda_{j}=-3v$ is shown in figure 5. At
the activation of the constraint term the velocity jumps from $\pm v$ to $\mp
v$ not changing the contraction field around $v,x$ in the equivalent position
dynamics of figure 5.

Figure 4: Constrained second-order dynamics with a plastic Dirac force at the
constraint Figure 5: Constrained second-order dynamics with a fully elastic
Dirac force at the constraint

Note that many practical systems have similar linear constraints, such as end
positions or actuator limits. $\Box$

In a constrained Riemann space, as in Theorem 13, the path of minimum length
between two points can be shaped by the constraint. For instance, with an
identity metric, it needs not be a straight line segment. For a Lagrangian

$L=\frac{1}{2}\dot{\bf x}^{T}{\bf M}({\bf x})\dot{\bf x}$

the motion [9, 13] between two connecting points ${\bf x}_{1}(t_{1})$ and
${\bf x}_{2}(t_{2})$ is always a minimization of

$\int_{{\bf x}_{1}(t_{1})}^{{\bf x}_{2}(t_{2})}Ldt$

or equivalently

$\int_{{\bf x}_{1}(t_{1})}^{{\bf x}_{2}(t_{2})}\sqrt{L}dt=\int_{{\bf
x}_{1}(t_{1})}^{{\bf x}_{2}(t_{2})}\sqrt{\delta{\bf x}^{T}{\bf M}({\bf
x})\delta{\bf x}}$

so that for a pure kinetic energy the classical Lagrangian dynamics always
corresponds to the shortest connection between two points. As we now
illustrate, this leads to a new classical interpretation of the duality of
particles and waves, by considering the multiple shortest connections between
two points in a constrained Riemann space.

  * ###### Example 2.4

:  Let us now consider the double slit experiment in Figure 6. We define the
geodesic distance on $\mathbb{G}^{2}=\mathbb{R}^{2}\setminus\mathbb{E}^{2}$
rather than on $\mathbb{R}^{2}$, thus excluding the red double slit wall
obstacle $\mathbb{E}^{2}$ in Figure 6. The shortest connecting path
$s=\min\int_{{\bf x}(s)={\bf x}_{1}}^{{\bf x}_{2}}\sqrt{\delta{\bf x}^{\ast
T}\delta{\bf x}}$ (dashed line) from ${\bf x}_{1}$ to ${\bf x}_{2}$ has
corner, marked as star in Figure 6, in the two slits.

A Hamiltonian point mass with energy $h=\frac{1}{2}{\bf p}^{T}{\bf H}({\bf
q}){\bf p}$ will get according to Theorem 2 a constraint Dirac collision force
$\lambda_{j}\delta(g_{j})\dot{g}_{j}\frac{\partial g_{j}}{\partial{\bf q}}$ at
the star corner within the Hamiltonian collision dynamics

$\displaystyle\dot{\bf p}$ $\displaystyle=$ $\displaystyle-\frac{\partial
h}{\partial{\bf
q}}+\sum_{j=1}^{J}\lambda_{j}\delta(g_{j})\dot{g}_{j}\frac{\partial
g_{j}}{\partial{\bf q}}$ $\displaystyle\dot{\bf q}$ $\displaystyle=$
$\displaystyle\frac{\partial h}{\partial{\bf p}}$ (72)

    * –

This Dirac constraint force can bend the trajectory around the corner which is
called difraction. Since the corner has normals in all directions it implies
that the trajectory can bend in any direction behind the wall as long as the
constraint is not violated.

    * –

Multiple solutions of the deterministic Hamiltonian dynamics (72) exist here
because the Dirac collision force is not Lipschitz continuous. These multiple
solutions of a deterministic Hamiltonian (72) are intepreted in quantum
physics to be a stochastic distribution of waves.

    * –

There are two paths of shortest distance in Figure 6, going through slit 1 and
slit 2. Both paths have a different distance and travel time. Hence both paths
have a different phase angle which allows to compute the quantum state
superposition of two particles with a phase shift according to Feynmans path
integral formulation [4]. Note the Fenymans path integral is just used here to
compute the stochastic distribution of the multiple solutions of a
deterministic Hamiltonian (72).

From this point of view, the only difference of quantum physics to a classical
Hamiltonian dynamics is the fact that classical Hamiltonians are defined in
unconstrained Riemann spaces whereas quantum physics is defined in constrained
Riemann spaces (72). These constraints then imply multiple solutions, due to
the non-Lipschitz Dirac collision force (72), which is interpreted as non-
determinism in quantum physics.

Figure 6: Constraint Hamiltonian interpretation of double slit experiment

$\Box$

  * ###### Example 2.5

:  Let us now consider the single slit experiment in Figure 7. We define the
geodesic distance on $\mathbb{G}^{2}=\mathbb{R}^{2}\setminus\mathbb{E}^{2}$
rather than on $\mathbb{R}^{2}$, thus excluding the red double slit wall
obstacle $\mathbb{E}^{2}$ in Figure 6. The shortest connecting path
$s=\min\int_{{\bf x}(s)={\bf x}_{1}}^{{\bf x}_{2}}\sqrt{\delta{\bf x}^{\ast
T}\delta{\bf x}}$ (dashed line) from ${\bf x}_{1}$ to ${\bf x}_{2}$ has two
corners, marked as star in Figure 6, in the slit.

A Hamiltonian point mass with energy $h=\frac{1}{2}{\bf p}^{T}{\bf H}({\bf
q}){\bf p}$ will get according to Theorem 2 a constraint Dirac collision force
$\lambda_{j}\delta(g_{j})\dot{g}_{j}\frac{\partial g_{j}}{\partial{\bf q}}$ at
the star corner within the Hamiltonian collision dynamics

$\displaystyle\dot{\bf p}$ $\displaystyle=$ $\displaystyle-\frac{\partial
h}{\partial{\bf
q}}+\sum_{j=1}^{J}\lambda_{j}\delta(g_{j})\dot{g}_{j}\frac{\partial
g_{j}}{\partial{\bf q}}$ $\displaystyle\dot{\bf q}$ $\displaystyle=$
$\displaystyle\frac{\partial h}{\partial{\bf p}}$ (73)

    * –

This Dirac constraint force can bend the trajectory within the slit, which is
called difraction. Since the corner has normals in all directions it implies
that the trajectory can bend in any direction behind the wall as long as the
constraint is not violated. The collision force applies to particles on the
left, right and middle of the slit due to the size of the particle.

    * –

Multiple solutions of the deterministic Hamiltonian dynamics (73) exist here
since the Dirac collision force is not Lipschitz continuous. These multiple
solutions of a deterministic Hamiltonian (73) are interpreted in quantum
physics to be a stochastic distribution of waves.

    * –

There are several paths of shortest distance in Figure 6, going through the
slit of finite size. All paths have a different distance and travel time.
Hence all paths have a different phase angle which allows to compute the
quantum state superposition of two particles with a phase shift according to
the Feynman path integral formulation [4]. Note the Fenyman path integral is
just used here to compute the stochastic distribution of the multiple
solutions of a deterministic Hamiltonian (73).

From this point of view, the only difference of quantum physics to a classical
Hamiltonian dynamics is the fact that classical Hamiltonians are defined in
unconstrained Riemann spaces whereas quantum physics is defined in constrained
Riemann spaces (73). These constraints then imply multiple solutions, due to
the non-Lipschitz Dirac collision force (73), which is interpreted as non-
determinism in quantum physics.

Figure 7: Constraint Hamiltonian interpretation of single slit experiment

$\Box$

##  3 Summary

This paper extends continuous contraction theory of non-linear dynamics to
non-linear inequality constraints:

  * •

The contraction behaviour of the constrained dynamics is the covariant
derivative of the system dynamics, from the original contraction theorem [5],
plus the second covariant derivative of the inequality constraint on the
constraint as defined in Theorem 13.

  * •

For Hamiltonian dynamics a Dirac collision force in Theorem 2 has an
equivalent position dynamics to the velocity constraint dynamics of heorem 13.

Practical applications include controllers constrained to an operational
envelope, trajectory control considering moving obstacles as constraints, and
a classical Hamiltonian interpretation of the single and two slit experiments
of quantum mechanics, among others.

Example 6 and Example 7 showed that in this context the only difference
between quantum physics and classical Hamiltonian physics is the fact that
classical Hamiltonians are defined in unconstrained Riemann spaces leading to
a single deterministic solution of the Lipschitz Hamiltonian dynamics. The
non-Lipschitz Dirac collision force, which is introduced by constraints in
quantum physics, leads to a set of multiple solutions of a deterministic
Hamiltonian 72, which is intepreted as non-determinism in quantum physics.

The results of this paper are extended to the discrete-time case in [8]. This
allows contraction of discrete-time learning in piece-wise linear neuronal
networks to be assessed.

Acknowledgements We thank Paul Schiefer and Lukas Huber for the simulations of
this paper. We also thank Philipp Gassert for fruitful discussions in the
context of this paper.

## References

  * [1] Bronstein, Semendjajew, Taschenbuch der Mathematik, Teubner, 1991.
  * [2] Bryson A., Yu-Chi H., Applied Optimal Control, Taylor and Francis, 1975.
  * [3] https://en.wikipedia.org/wiki/Double-slit_experiment
  * [4] Feynman’s path integral formulation https://en.wikipedia.org/wiki/Path_integral_formulation
  * [5] Lohmiller, W., and Slotine, J.J.E., On Contraction Analysis for Nonlinear Systems, Automatica, 34(6), 1998.
  * [6] Lohmiller, W., and Slotine, J.J.E., Nonlinear Process Control Using Contraction Theory, A.I.Ch.E. Journal, March 2000.
  * [7] Lohmiller, W., and Slotine, J.J.E., Contraction Theory with Inequality Constraints, arXiv:1804.10085, 2023.
  * [8] Lohmiller, W., Gassert P. and Slotine, J.J.E., MinMax Networks, arXiv, 2023
  * [9] Lovelock D., and Rund, H., Tensors, Differential Forms, and Variational Principles, Dover, 1989.
  * [10] Moore-Penrose Inverse https://en.wikipedia.org/wiki/Moore–Penrose_inverse
  * [11] Nguyen, H.D., Vu, T.L., Slotine, J.J.E., and Turitsyn, K., ”Contraction Analysis of Nonlinear DAE Systems,” I.E.E.E. Transactions on Automatic Control, 2020
  * [12] Tabareau, N., Bennequin, D., Berthoz A., Slotine J.J.E., and Girard, B., ”Geometry of the Superior Colliculus Mapping and Efficient Oculomotor Computation,” Biological Cybernetics, 97(4), 2007
  * [13] Tolman R.C., Relativity Thermodynamics and Cosmology, Dover Publications, 1934.


Researchers are increasingly using language models (LMs) for text annotation. These approaches rely only on a prompt telling the model to return a given output according to a set of instructions. The reproducibility of LM outputs may nonetheless be vulnerable to small changes in the prompt design. This calls into question the replicability of classification routines. To tackle this problem, researchers have typically tested a variety of semantically similar prompts to determine what we call “prompt stability." These approaches remain ad-hoc and task specific. In this article, we propose a general framework for diagnosing prompt stability by adapting traditional approaches to intra- and inter-coder reliability scoring. We call the resulting metric the Prompt Stability Score (PSS) and provide a Python package  for its estimation. Using six different datasets and twelve outcomes, we classify $>$150k rows of data to: a) diagnose when prompt stability is low; and b) demonstrate the functionality of the package. We conclude by providing best practice recommendations for applied researchers.

§ INTRODUCTION

Given recent advances in natural language processing, scholars have begun debating whether human coders may be replaced by machine-learning tools [Gilardi et al., 2023, Grossmann et al., 2023]. Specifically, commentary has focused on large language models (LMs), which rely on minimal researcher input and model training for the classification of different types of data [Ziems et al., 2024, Törnberg, 2023]. Given the power of many LMs, researchers have proposed that so-called zero-shot approaches are capable of overcoming even demanding classification tasks [Le Mens et al., 2023, Gilardi et al., 2023]. Importantly, these approaches represent a departure from more conventional training and testing paradigms in supervised learning: we can now accurately classify data just by telling a model what we want it to do.

Zero- or few-shot designs require just a “prompt" to guide the model in its classification decisions. This prompt often takes the form of a short phrase or set of phrases indicating the desired output [Törnberg, 2023]. Using these techniques, a large number of contributions have now demonstrated that LMs are capable of diverse classification or annotation tasks [Gilardi et al., 2023, Törnberg, 2023, Bang et al., 2023, Qin et al., 2023, Goyal et al., 2022, Chiang and Lee, 2023, Grossmann et al., 2023, Ziems et al., 2024]. All of these approaches rely on some minimal prompt design to classify the data in question.

We argue that the use of LMs to classify data in a zero- or few-shot setting has obvious analogies with the use of human coders. That is, we can understand an individual LM as equivalent to an individual expert or trained human coder. Research designs relying on content analysis of textual or other documentary data have traditionally relied on measures of inter- or intr-acoder reliability [Lombard et al., 2002]. These are important for determining construct validity, addressing inherent subjectivity biases involved in coding certain types of data, and ensuring that a given design can be replicated.

Absent from recent contributions, however, is any equivalent concern with reliability scores when using LMs [Reiss, 2023, Ollion et al., 2023, Sclar et al., 2023]. This is important for several reasons. First, many researchers are now adopting LMs as classifiers in the absence of validation data. But even if zero- or few-shot approaches are successful with one prompt design, this does not mean they will be successful with another design. Second, even when researchers do validate by using ground-truth data, we do not know how vulnerable classifications might be to small changes in the prompt design. For both of these first two reasons, future research may therefore struggle to replicate a given approach. Third, researchers can use our approach to determine the stability of a given prompt design prior to validation. That is, if a given prompt is highly unstable, the researcher has the chance to reconsider the prompt design before undertaking any validation steps such as manually coding substantial amounts of data.

In this article, we outline a technique to test for what we refer to as prompt stability. By this we mean the consistency of model classifications between individual runs of the same prompt and individual runs of different (but semantically similar) prompts. We do so by building an algorithm that automates the generation of semantically similar prompts from a baseline prompt provided by the user. We then feed our data to the LM and check for the consistency of model classifications both within the same prompt and across semantically similar prompts. In this way, we are able to calculate an equivalent of inter- and intra-coder reliability with LMs, which we refer to as the Prompt Stability Score (PSS). This represents an important diagnostic for applied researchers who will be able to determine whether their chosen LM is likely vulnerable to instability as a result of prompt design or unobserved model features.

§ CODER RELIABILITY AND PROMPT STABILITY

As a point of departure, it is helpful to think through the parallels between traditional inter- and intra-coder reliability and prompt stability. First, we consider the classic understanding of reliability using human coders. Here, researchers ask human coders to classify (“code") the same data once or multiple times. In this way, we are able to tell whether the same coder classifies the same data in the same way each time (intra-coder reliability) and whether different coders classify the same data in the same way each time (inter-coder reliability). We can summarize the advantages of using such a procedure as: objectivity, validity, and replicability [O’Connor and Joffe, 2020, Belur et al., 2021, Lamprianou, 2023]. That is, we can be confident that classifications are not vulnerable to subjectivity bias, that the construct being measured is likely valid, and that the end result is replicable. For LMs, we have to reconceptualize each of these benefits when examining prompt stability. We summarize these comparisons in Table <ref>.

In Table <ref> we note that when coder reliability is high, we have some indication that coders are not reliant on subjective judgment but are able to categorize some data based on some existing criteria. Of course, we do not know if the human coders are all consistently biased in one direction, which would result in high reliability even if responses are subjective. When prompt stability is high, we also do not know whether this is because we are targeting some objective quantity or the LM has some engineered bias toward a particular “point of view" such that it consistently reproduces the same “subjective" classification (e.g., what constitutes “hateful" speech) with either the same or similar prompt. When coder reliability is high, we have some indication that we are targeting a valid construct, which has some identifiable and understandable definition. When prompt stability is high, we also have some indication that we are classifying a valid construct which the LM can “understand." For both cases, though, it is worth noting that high reliability/stability is a necessary but not a sufficient condition for validity: a human coder may default to consistently responding “Don't know" if the construct is unclear and an LM may behave similarly . Finally, both high reliability and high stability means other researchers and other LMs are more likely to be able to reproduce our classifications.

Aspect     Coder Reliability     Prompt Stability

Objectivity     High reliability indicates that coding decisions are not overly reliant on subjective judgment.     High stability may indicate that biases in the model do not significantly affect classifications.

Validity     High reliability suggests the construct being measured is valid.     High stability suggests the construct being measured is valid.

Replicability     High reliability means other researchers can reproduce the same classifications.     High stability means other language models can reproduce the same classifications.

Comparison of coder reliability and prompt stability in research contexts

It is important to detail these comparisons, first, so that we take into consideration the close parallels between human-coded and machine-coded research designs and, second, so that we recognize the difference between reliability/stability and accuracy. Low reliability/stability is a good indication that accuracy will be low were we to repeat the same or similar analysis. But high reliability/stability could mean either the human/model is consistently wrong or that the human/model is producing accurate classifications.
In order to calculate prompt stability, we propose testing: 1) multiple iterations of the same prompt; 2) multiple iterations of different prompts. We refer to the first as intra-prompt stability and to the second as inter-prompt stability. The first of these is directly analogous to intra-coder reliability: the same human (LM) is coding the same data multiple times. The second requires some reconceptualization. Inter-coder reliability refers to different humans coding the same data multiple times. Inter-prompt stability refers to different prompts coding the same data multiple times. Overall, our approach is capturing two separate things: 1) how stable the LM results are over repeated iterations of the same prompt; and 2) how stable the LM results are across different prompts.

In many applications, researchers may find that they have low inter- or intra-coder reliability for the outcome of interest. Reasons for low  reliability scores using human coders might have to do with: ambiguous coding criteria; complexity of the underlying data; complexity of the construct; and coder skill or training. When transported to the context of LMs, the reasons for low prompt stability have some crossovers. But we again have to reconceptualize the potential underlying reasons for low prompt stability. We summarize these in Table <ref>.

Aspect     Coder Reliability     Prompt Stability

Criteria Ambiguity     Ambiguous coding instructions lead to inconsistent interpretations.     Ambiguous prompt design can lead to inconsistent language model outputs.

Data Complexity     Complex, messy, or inappropriate data can lead to unreliable classifications by coders.     Complex, messy, or inappropriate data may result in unreliable classifications by the language model.

Outcome Complexity     The construct may lack a stable definition for reliable human coding.     The construct may lack a stable definition for reliable language model classification.

Skill/Capability     Varying coder experience, skill, or training may lead to unreliable classifications.     Variability in language model capability may lead to unreliable classifications.

Factors affecting low coder reliability and prompt stability in research contexts

The approach we describe below allows the applied researcher to examine the stability of classifications across multiple runs of the same or different prompt. If prompt stability is consistently low for the same LM, this is an indication that the researcher confronts one of the first three reasons. If the prompt stability is considerably lower for one LM compared to another, this is indication that the researcher confronts the fourth reason. In summary, we can see that there are distinct parallels between traditional approaches to coder reliability scoring and prompt stability. In what follows, we describe our approach for estimating what we refer to as Prompt Stability Scores (PSS).

§ CURRENT APPROACHES

Existing work related to the reliability of LM output falls into three camps. A first group of research examines variation in LM accuracy across different versions of the same prompt as a method to probe LM uncertainty [Chen and Mueller, 2023, Shin et al., 2020, Portillo Wightman et al., 2023, Zhao et al., 2021, Arora et al., 2022, Chen et al., 2023, Schick and Schütze, 2021, Tian et al., 2023, Feffer et al., 2024]. The aim here is to optimize models by using the observed stability and accuracy of outputs to improve model performance. This is particularly necessary with more recent “closed-source" models for which we do not have access to mdoel weights. By utilizing the observed consistency of prompt outputs, researchers have attempted to derive a proxy for uncertainty [Portillo Wightman et al., 2023, Arora et al., 2022].

A second body of research focuses on providing strategies to craft better prompts. Here, notable contributions include Chain-of-Thought prompting where LMs are prompted to break down a problem into a series of reasoning steps [Wei et al., 2022]; Generated-Knowledge prompting where the LM is first encouraged to produce some knowledge about the problem before responding to the task [Liu et al., 2021]; as well as extensions of CoT prompting such as Tree-of-Thought [Long, 2023]. Similar to the first body of work described above, recent contributions have attempted to iterate over these prompt designs to select the “best" outcome according to some popularity metric [Wang et al., 2022, Madaan et al., 2023, Perez et al., 2021]. For an effective survey of these approaches see [Liu et al., 2023].

In a third body of work, applied researchers have demonstrated that LMs are capable of outperforming humans across a variety of classification and annotation tasks [Gilardi et al., 2023, Törnberg, 2023, Bang et al., 2023, Qin et al., 2023, Goyal et al., 2022, Chiang and Lee, 2023, Grossmann et al., 2023, Ziems et al., 2024]. Yet some more recent contributions have demonstrated a notable lack in the stability of outcomes when prompting LMs to classify social scientific constructs [Dentella et al., 2023, Bisbee et al., 2023]. Scholars have tried to overcome potential problems of prompt instability by using e.g., “prompt perturbation," and selecting the most common output from a set of semantically similar prompts [Törnberg, 2023, Ziems et al., 2024]. Others provide dedicated routines for overcoming potential problems with superficial prompt formatting and show the LMs can be extremely sensitive to seemingly trivial features such as punctuation [Sclar et al., 2023]. To date, social scientists lack any coherent approach or software to tackle this problem, however. Against this backdrop, [Ziems et al., 2024] argue that new validation approaches are required to avoid another replication crisis in the social sciences (see also [Bail, 2024]).

In sum, existing approaches to diagnosing potential prompt instability remain ad hoc, task specific, and do not speak to more familiar questions of coder reliability in social scientific research. We therefore aim to provide a general diagnostic approach for applied researchers. It is worth noting that the problem we aim to overcome has parallels previously popular approaches to the analysis of text such as topic modelling. Here, despite the apparent potential of topic modelling for the topical summary of text, researchers were unaware how vulnerable their conclusions might be to trivial variations in text preprocessing decisions. Against this backdrop, [Denny and Spirling, 2018] provided a general approach to assessing the potential importance of preprocessing decisions in determining outcomes. We argue that zero-shot learning has this vulnerability in common with unsupervised approaches to text analysis. We aim to provide a general solution to diagnosing how vulnerable our conclusions are to prompt design decisions and a best practice guide for applied researchers. In so doing, we are able to advance computational text analysis approaches by foregrounding techniques for determining validity [Baden et al., 2022].

§ DESIGN

§.§ Data

We use six different datasets to validate our approach and we specify two versions of the outcome for each of these datasets. These are described in Table <ref>. The “Analysis name" column is how we go on to refer to these data-outcome pairs below when discussing results. The data were selected for: a) being relevant to social science research;  b) spanning diverse types of construct; c) being adaptable to multiple operationalizations of that construct. The first dataset comes from [Van Vliet et al., 2020] where we filter for all content posted by United States Senators on Twitter (now X) in the two months prior to the 2020 US Election.[This is the same data used in [Törnberg, 2023]]. The second is a set of UK party political manifestos taken from [Benoit et al., 2016]. The third is a set of New York Times news articles about a diverse range of topics taken from [Young and Soroka, 2012]. The fourth is a set of tweets expressing stances toward individuals or topics in US politics [Ziems et al., 2024].[These originally derive from the SemEval-2016 dataset provided by [Mohammad et al., 2016].] The fifth is a set of open-ended survey responses from the British Election Study used in [Mellon et al., 2024]. And the sixth is a series of text profiles of individual voters taken from the American National Election Study used in [Argyle et al., 2023].

Dataset     Source     Outcomes     Analysis name     Relevant variation

US Senator Tweets     van Vliet et al. (2020)

* Binary: Republican/Democrat

* Binary: Populist/Not Populist

* Tweets (Rep. Dem.)

* Tweets (Populism)

* Construct validity/Outcome complexity

NYT News Articles     Benoit et al. (2016)

* Categorical: Positive/Negative/Neutral (All Articles)

* Categorical: Positive/Negative/Neutral (Short Articles)

* News

* News (Short)

* Data complexity/Suitability

UK Manifestos     Young & Soroka (2012)

* Binary: Left-wing/Right-wing

* Integer: 1-10 Ideology Scale

* Manifestos

* Manifestos Multi

* Construct validity/Outcome complexity

US Politics Tweets     Ziems et al. (2024)

* Binary: Stance For/Against

* Categorical: For/Against/None

* Stance

* Stance (Long)

* Data complexity/Suitability

BES Survey Responses     Mellon et al. (2024)

* Categorical: 6 categories

* Categorical: 12 categories

* MII

* MII (Long)

* Data complexity/Suitability

ANES Voter Profiles     Argyle et al. (2023)

* Binary: Biden/Trump (8 Variables)

* Binary: Biden/Trump (6 Variables)

* Synthetic

* Synthetic (Short)

* Construct validity/Outcome complexity

Summary of datasets, outcomes, and relevant variations used in our approach.

Overall, these datasets span a broad range of outcome types (constructs) and forms (operationalizations) relevant to social science. By including a broad range of data, we hope to show the adaptability of our proposed software to multiple questions. We also designed these datasets—and their modifications—to cover the range of data reasons for potential low coder reliability described in Table <ref>. We modify the data complexity or suitability in the US Senator Tweets, NYT News Articles, US Politics Tweets, BES Survey Responses, and ANES Voter Profiles; we modify the construct validity or outcome complexity in the US Senator Tweets, UK Manifestos, US Politics Tweets, and BES Survey Responses. This is what we refer to as “Relevant variation" in Table <ref> and below.

For the US Senator Tweets, we prompt the LM to classify whether or not the tweeter is Republican or Democrat and whether the tweet expresses populist language or or not. The logic of this task is to determine the stability of the LM for a task that involves classifying short pieces of text for an outcome relevant to the domain (tweets by Republicans or Democrats) and an outcome that is less clearly relevant to the domain (tweets that may or may not contain Populist language). For the NYT articles, we prompt the LM to classify if the overall tone of the article is positive, negative or neutral for all articles in the dataset and then for shorter articles ($<=$ 1000 words) only. The logic of this second task is to determine the stability of the LM when tasked with classifying more versus less textual information. For the UK Manifestos data, we prompt the LM to label the content as left-wing or right-wing then on a ten-point ideology scale. The motivation for this task is to determine LM stability when classifying the outcome as binary or interval. For the US Politics Tweets we filter out tweets with classification “None" and prompt the LM to produce a binary output whereas for the second run we retain the None category. Here, the intuition is that LM classifications may become less stable as more in-between cases are introduced. For the BES Survey Responses data, we prompt the LM to classify the open-ended survey responses as one of six categories that appear in the data then as one of 12 categories that appear in the data. Here, the logic is to assess model stability when the range of options is clearly defined versus when ther range of options contains some overlap (e.g., here: Economy-general versus Inflation versus Living costs). Finally, for the ANES Voter Profiles data, we take all eight variables used in the published research when assessing the reliability of silicon election study samples. For the first outcome, we use all eight variables including the explicitly political variables that ask survey respondents whether they are Republican or Democrat and their ideological outlook; for the second we omit these two variables. The logic of this test is to determine the stability of outcomes when omitting key information (here: variables) in the data to be classified.

The remaining two reasons for low coder reliability, of course, have to do with the coding criteria and coder training. In an LM context, we are here referring to the prompts we use when classifying data and the power of the LM itself. In the next section, we describe our method for testing the stability of prompts and prompt variants when annotating data. In the main analyses we only use one LM so we leave for future research any test of prompt stability across LMs with different capabilities. Finally, it is worth noting that for each of these datasets and outcomes, we do not use the exact same prompt design as the original authors. As such, our tests are not to be understood as an explicit test of the replicability of any of these published findings.

§.§ Method

§.§.§ Intra-prompt stability

For each of our datasets, we begin with an original prompt and target construct. For example, in the case of the US politics tweets, we instruct the LM to classify each tweet according to whether it thinks it was authored by a Democrat or Republican Senator. For the intra-prompt stability approach,  we simply take the original prompt and classify the same $n$ rows of data 20 times.[It is worth noting that this doesn’t merely capture the classifications of a given model at a given temperature, as the model can produce different probability distribution for the same prompt when run multiple times.] For each iteration, we then calculate a cumulative reliability score. This reliability score is simply the overall Krippendorff's Alpha (KA) for each iteration, which we refer to as the intra-prompt stability score or intra-PSS. Here, we aim to mimic a standard research design where different coders are asked to code the same data $n$ times. In the examples below, we reclassify the same randomly sampled 100 rows of data twenty times for each dataset-outcome pairing. I.e., for each data-outcome pairing, we classify 2000 rows of data.

Krippendorff's Alpha is estimated using the following formula:

\[
\alpha = 1 - \frac{D_o}{D_e}
\]

where \(D_o\) is the observed disagreement and \(D_e\) is the expected disagreement. For binary data, these are computed as follows:

\[
D_o = \frac{1}{N} \sum_{c=1}^{n} \sum_{k=1}^{K} \sum_{l=1}^{K} o_{ck} \cdot o_{cl} \cdot d_{kl}
\]

\[
D_e = \frac{1}{N(N-1)} \sum_{k=1}^{K} \sum_{l=1}^{K} e_{k} \cdot e_{l} \cdot d_{kl}
\]

where: \(o_{ck}\) is the number of coders who assigned the value \(k\) to the item \(c\); \(o_{cl}\) is the number of coders who assigned the value \(l\) to the item \(c\); \(e_{k}\) is the expected number of occurrences of value \(k\); \(e_{l}\) is the expected number of occurrences of value \(l\); \(d_{kl}\) is the distance between values \(k\) and \(l\); \(n\) is the total number of items; \(K\) is the number of unique values (categories); and \(N\) is the total number of coder assignments.[The formula for nominal data treats all outcomes as equally distant to each other. I.e., the distance between two different codings of the same data is 1 and 0 otherwise. The formula for interval data uses the squared difference between values as a measure of distance. ]

For our purposes, let $D$ be the dataset with $n$ rows. For each row $i \in \{1, 2, \ldots, n\}$ and for each iteration $j \in \{1, 2, \ldots, 20\}$, let $C_{i,j}$ be the classification given by the LM. The intra-PSS for each iteration $j$ is computed as:

\[
PSS_j = \text{PSS}(C_{1,j}, C_{2,j}, \ldots, C_{n,j})
\]

The cumulative reliability score after $k$ iterations is given by:

\[
PSS_{\text{cumulative}, k} = \frac{1}{k} \sum_{j=1}^{k} PSSj
\]

where $k \in \{1, 2, \ldots, 20\}$. To calculate the confidence interval for the intra-PSS estimates at each iteration, we use a bootstrap approach detailed in full in the Appendix.

§.§.§ Inter-prompt stability

We then use  [Zhang et al., 2019] to automate a series of semantically similar prompts. For each generation, we increase the “temperature" of the paraphrasing engine. The temperature in fine-tuned Transformer models like  controls how “conservative" or “creative" the model is. In effect, these parameters are increasing or decreasing acceptable variance in models logits under the hood. By increasing the temperature, we are effectively widening the acceptable variance in the logits of the next token.

We use this feature of generative models to produce a series of increasingly semantically dissimilar prompts. The intuition here is that a higher temperature will produce prompts that are more semantically distant from the original prompt. We then iterate through different versions of the prompt for each temperature and classify the data \( n \) times across each prompt. For each temperature, we are then able to plot the inter-prompt stability scores (inter-PSS). We use the same bootstrapping approach as before with one difference: here, we estimate the bootstrapped confidence intervals within each individual temperature. We expect that as the prompt becomes more semantically distant from the original, the variance in the prompt stability scores will also increase given the decrease in description precision of coding guidelines (prompts). In the examples below, we generate ten prompt variations for each temperature. For each data-outcome pairing, we randomly sample 100 rows for classification and we iterate across twelve temperature from 0.1 to 5.0. In sum, for each dataset, we code 1000 rows for each temperature (10 variations $\times$ 100 rows of data) for twelve temperatures (i.e., 1000 $\times$ 12 $=$ 12000).

Formally, let \( D \) be the dataset with \( n \) rows. For each prompt variation \( p_v \in \{1_1, 1_2, \ldots, 1_{10}, 2_1, 2_2, \ldots, P_{10}\} \), where \( P \) is the number of generated prompts across varying temperatures, and for each temperature \( j \in \{1, 2, \ldots, 12\} \), let \( C_{i,j}^{(p_v)} \) be the classification given by the LM for row \( i \) and temperature \( j \) using prompt variation \( p_v \).

The inter-PSS for each temperature \( j \) using prompt variation \( p_v \) is computed as:

\[
PSS_j^{(p_v)} = \text{PSS}(C_{1,j}^{(p_v)}, C_{2,j}^{(p_v)}, \ldots, C_{n,j}^{(p_v)})
\]

The overall stability score for prompt variation \( p_v \) after \( k \) temperatures is given by:

\[
PSS_{\text{avg}, k}^{(p_v)} = \frac{1}{k} \sum_{j=1}^{k} PSS_j^{(p_v)}
\]

where \( k \in \{1, 2, \ldots, 12\} \). Within each temperature, we estimate \( PSS_j^{(p_v)} \) across the multiple temperature settings and then average the \( PSS \) within each temperature. This is repeated for each prompt variation within each temperature to estimate the final inter-PSS. To calculate the confidence interval for inter-PSS within each individual prompt variation \( p_v \), we again use a bootstrap approach detailed in full in the Appendix.

A full list of the original prompts and prompt variants that we test is reproduced in Appendix Sections <ref> and <ref>. Note that, for each prompt, we also specify a , which specifies the type of output the model should provide (e.g., binary, interval). This remains constant across all variations of the prompts we use for the inter-prompt stability estimates. The rationale for this is simple: we are interested in the consistency of results across variations in target construct descriptions(i.e., prompts)—not in the format of the output. By fixing the format of the output, we aim to isolate variation in classification stability that results from semantic variation in the prompt and not stochastic differences in output formatting.

In the main results below, we use GPT-3.5 () from OpenAI. Importantly, our software package is LM agnostic meaning it is operable across multiple LMs. In the main software package, we include examples of how to implement the main functions using a range of open-source models including Llama3, Gemma, and Mistral.

§ RESULTS

We visualize the results of our tests in Figures <ref> and <ref>. We see that, for the intra-prompt stability approach, we see relatively stable outputs across all datasets. For most, the averaged intra-PSS is above 0.8 across all runs. This is largely as expected. Remember that here we are using prompts that are close to the original prompts used in the published researcher where these routines were shown to be successful. As the sample size increases with the number of iterations, the width of the bootstrapped confidence intervals naturally decreases. Some notable trends also emerge. Two datasets where the construct being coded is particularly unclear have among the lowest average PSS of all datasets: the populism outcome for US Senator tweets and the most important issue outcome for the BES survey responses. Similarly, two datasets where the data to be classified are complex, messy or unsuitable appear among the lowest-ranked for average PSS: the Biden/Trump outcome for the shortened synthetic data and the sentiment outcome for the News articles that include very long articles. By contrast, those datasets where the outcome construct is clearly delimited, the data do not contain potentially overlapping codings, and the data is suitable and contains key information are among the most stable. This is the case for the most important issue dataset that uses only six categories, the synthetic data that includes information about voter political characteristics, the manifestos data with interval coding, and the Republican versus Democrat coding for US senator tweets.

Combined intra-prompt stability scores.

Combined inter-prompt stability scores, ordered by average score.

As for the inter-prompt stability approach, we see first that the stability of outputs generally decreases as temperature increases. This is as expected: as the prompt becomes more semantically distant from the original, we expect the stability of classifications to reduce as a result of the degradation in quality of coding guidelines (prompts). Once again, some common patterns emerge. Those datasets where we see the least (most) stable PSS across temperatures are the datasets where either the data are (well-) ill-suited or the outcome construct is (precisely) imprecisely defined. It is also notable that for some of the data-outcome pairings, we see that PSS rapidly decreases even at low temperatures. This is the case with all of the data-outcome pairings in the bottom row of Figure <ref>.

§.§ Additional metrics

In addition to estimating and plotting the intra- inter-prompt stability scores, we provide a series of additional metrics derived from these scores. Table <ref> provides details of these. We include these additional metrics within the package functionality.

For intra-prompt stability, the Weighted Mean calculates the average stability score with more emphasis on the initial iterations (estimated as $\text{PSS} \times \frac{1}{\text{iteration}}$). The Iteration Threshold metric determines the number of iterations required to reach a stable PSS score, defined as when the lower bound of the bootstrapped confidence intervals exceed a specific threshold. The AUC-PSS assesses the overall stability over multiple iterations by calculating the area under the curve of PSS scores versus iterations (using the trapezoidal rule). Finally, the Variance score provides an indication of the fluctuation in stability scores across iterations, with lower variance suggesting more consistent model performance.

For the inter-prompt stability approach, the Weighted Mean is designed to give more weight to lower temperatures (estimated as $\text{PSS} \times \frac{1}{\text{temperature}}$), reflecting the stability closer to the original prompt. The Temperature Threshold metric identifies the point at which the bootstrapped upper bound of the PSS drops below a specific threshold, highlighting the temperature at which the model's responses become significantly unstable. The AUC-PSS measures the area under the curve of PSS scores versus temperatures (using the trapezoidal rule), with a larger area indicating greater overall stability. The Variance metric captures the variability of scores across different temperatures, offering insight into model consistency.

Metric     Description

Intra-PSS Metrics

Weighted mean     The average Intra-Prompt Stability Score, giving more weight to lower iterations, as these are closer to what actual practitioners would do.

Iteration Threshold     The number of iterations required to reach a stable Krippendorff's Alpha (KA) score, when the lower bound of the PSS confidence intervals at a given iterations exceeds a certain threshold  (here: .8).

AUC-PSS     The area under the curve of PSS scores vs. iterations, with a larger area indicating better overall stability.

Variance     The variance of PSS scores across all iterations, with lower variance suggesting more consistent performance.

PSS Metrics

Weighted mean     The average Inter-Prompt Stability Score, giving more weight to lower temperatures, as these are closer to the original prompt.

Temperature threshold     The temperature at which the upper bound of the PSS drops below a certain threshold (here: 0.8).

AUC-PSS     The area under the curve of PSS vs. temperature, with a larger area indicating better overall stability.

Variance     The variance of PSS scores across all temperatures, capturing the overall consistency of the stability scores.

Descriptions of the metrics used for Prompt Stability Score (PSS) and Intra-Prompt Stability Score (Intra-PSS).

We estimate each of these metrics in Figures <ref> and <ref>. Naturally, the rank order between comparable metrics (Weighted mean and AUC-PSS; Variance and Iteration/Temperature threshold) is similar but not the same. Taken these metrics together, we get a fuller picture of prompt stability both within and between prompts.

Additional metrics for intra-prompt stability routine. Note that for the purposes of visualization, the iteration threshold for Synthetic (short) shows max. threshold + 1 as this data never converged to a score above .8 after 20 iterations.

Additional metrics for inter-prompt stability routine. Note that for the purposes of visualization, the temperature threshold for MII shows max. threshold + 1 as this data never dropped below .8 after all temperatures.

We next go on to anticipate and answer several questions that applied researchers may have when implementing the approach we describe above.

§.§ Are the generated prompts actually reliable?

In the approach described above, we automate prompt generation with a machine-learning paraphraser tool across multiple temperatures. We use this technique as we expect that, as we increase the temperature, the prompt will become more semantically distant from the original and less stable. This is informative because the researcher can then observe at what point the classification procedure breaks down. This therefore provides a useful diagnostic for understanding when and why the omission or inclusion of certain information in a prompt reduces stability.

One objection to this approach is that we don't have the ability to control the prompt generation ourselves as researchers. To this, we have two responses. First, taking away control of the generation of prompts also has advantages. A well-known problem in ostensibly unsupervised machine-learning methods is the tendency of researchers to re-tune parameters until they results conform with their priors [Chang et al., 2009, Grimmer and Stewart, 2013]. By automating the generation of prompts, we reduce these researcher degrees of freedom and discourage practitioners from selecting only variations of well-performing prompts that give a false picture of prompt stability. Second, we do include in the package functionality the option to edit prompts when the automated prompts are not appropriate for the task at hand. We nonetheless caution researchers that, when editing prompts, they should: a) have good reason to do so (e.g., automation not working as prompt too long or breaking down at low temperatures because of early omissions of one crucial word); b) be fully transparent in the edits that have taken place; c) provide full code necessary to reproduce the prompt stability analysis undertaken.

A second objection is that the prompts generated are in some other way faulty, meaning that it is the proposed routine that is at fault for giving a faulty impression of instability. That is, if the prompts are just bad prompts (especially at low temperatures), then it should be unsurprising that stability is low. To assuage these concerns, we print every single prompt and variant used in our analyses in the Appendix to this article in Sections <ref> and <ref>.

§.§ What constitutes acceptable stability?

In our study, we refrain from recommending a specific cutoff or acceptable threshold for overall prompt stability. The commonly accepted metric of 0.8 is frequently used to signify “very good" agreement in the calculation of Krippendorff's Alpha (though see [Krippendorff, 2004]). We suggest that if a prompt does not achieve an intra-prompt PSS above 0.8 after the initial run, the researcher should investigate the underlying causes of this variance. For inter-PSS, we advise that if the score falls below 0.8 at a particular temperature (and especially at low temperatures), the researcher should scrutinize the prompt variant to determine its semantic validity, ensuring it retains the same substantive meaning as the original prompt. If it does have the same substantive meaning and still breaks down, this is strong indication that the routine is unstable and therefore less likely to replicate with e.g., a different LM or adapted prompt design in future research.

§.§ Parameter stability

We also know that LMs may be more less vulnerable to output variations based on the tuning of key model parameters. The most important of these are the  setting of the model and the  setting. Both of these parameters have an influence on how conservative the model is when producing its output.[For models like OpenAI's GPT-3.5 and GPT-4, the user can also adapt the “Frequency penalty" and “Presence penalty." Both of these control, in different ways, the creativity of the model by preventing it from using the same phrases or producing the same topics respectively. For classification tasks, it is less common for users to adapt these. For this reason, and for reasons of parsimony, we hold these parameters constant. A low temperature is generally recommended for text annotation. In our examples we hold it constant at .1. We also keep  at its default setting of 1.] As a result, researchers should also be aware that the stability and accuracy of their outcomes may depend on these parameters. We leave the task of determining the stability of outcomes when iterating over these additional parameters to future research.

§ CONCLUSION

In this article, we have set out a self-contained approach for testing the stability of a given prompt design when using LMs in social science research. We include in our dedicate  Python package a set of functions that can reproduce all of the analyses described above. Scratch code for this functionality is listed in Appendix listings <ref> and <ref>. This is important because the few existing approaches to cognate problems remain ad-hoc and task specific. Our approach, by contrast, is generalizable, open-source, and adaptable to diverse applications in the social sciences (and beyond). This approach will be valuable for two main reasons: 1) for determining the likely reproducibility of a given approach that relies on LMs; 2) for determining whether or not a given prompt design is even feasible given the target construct prior to undertaking laborious validation steps.

To achieve this, we classified $>$ 150k rows of data across six datasets and twelve different outcomes. This allowed us not only to test the performance of the software package we developed but also to begin to diagnose when prompt stability tends to be low. We find that when data is inappropriate for the task at hand or when the target construct is under-specified, prompt stability tends to be low. We further show that we can derive a series of useful additional metrics taken from the PSS to summarize overall model stability.

Of course, there remain important avenues for future research. Our approach does not adapt or iterate through other important model parameters such as the temperature of the actual classification engine or  settings. We also do not compare the performance of different LMs—though our software package is inter-operable across multiple LMs, requiring only two parameters in a basic annotation function. Finally, we only utilize one paraphrasing engine. Future applications may consider either alternative paraphrasing approaches or leveraging the power of other LMs to automate the prompt variants.

[Gilardi et al., 2023]
Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli.
Chatgpt outperforms crowd workers for text-annotation tasks.
Proceedings of the National Academy of Sciences, 1200 (30):0 e2305016120, 2023.

[Grossmann et al., 2023]
Igor Grossmann, Matthew Feinberg, Dawn C. Parker, Nicholas A. Christakis, Philip E. Tetlock, and William A. Cunningham.
Ai and the transformation of social science research.
Science, 3800 (6650):0 1108–1109, 2023.
URL <https://www.science.org/doi/abs/10.1126/science.adi1778>.

[Ziems et al., 2024]
Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang.
Can large language models transform computational social science?
Computational Linguistics, 500 (1):0 237–291, 2024.

[Törnberg, 2023]
Petter Törnberg.
How to use llms for text analysis.
arXiv preprint arXiv:2307.13106, 2023a.

[Le Mens et al., 2023]
Gaël Le Mens, Balázs Kovács, Michael T Hannan, and Guillem Pros.
Uncovering the semantics of concepts using gpt-4.
Proceedings of the National Academy of Sciences, 1200 (49):0 e2309350120, 2023.

[Törnberg, 2023]
Petter Törnberg.
Chatgpt-4 outperforms experts and crowd workers in annotating political twitter messages with zero-shot learning.
arXiv preprint arXiv:2304.06588, 2023b.

[Bang et al., 2023]
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al.
A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity.
arXiv preprint arXiv:2302.04023, 2023.

[Qin et al., 2023]
Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang.
Is chatgpt a general-purpose natural language processing task solver?
arXiv preprint arXiv:2302.06476, 2023.

[Goyal et al., 2022]
Tanya Goyal, Junyi Jessy Li, and Greg Durrett.
News summarization and evaluation in the era of gpt-3.
arXiv preprint arXiv:2209.12356, 2022.

[Chiang and Lee, 2023]
Cheng-Han Chiang and Hung-yi Lee.
Can large language models be an alternative to human evaluations?
arXiv preprint arXiv:2305.01937, 2023.

[Lombard et al., 2002]
Matthew Lombard, Jennifer Snyder-Duch, and Cheryl Campanella Bracken.
Content analysis in mass communication: Assessment and reporting of intercoder reliability.
Human communication research, 280 (4):0 587–604, 2002.

[Reiss, 2023]
Michael V. Reiss.
Testing the reliability of chatgpt for text annotation and classification: A cautionary remark, 2023.

[Ollion et al., 2023]
Etienne Ollion, Rubing Shen, Ana Macanovic, and Arnault Chatelain.
Chatgpt for text annotation? mind the hype!, October 2023.
URL <http://dx.doi.org/10.31235/osf.io/x58kn>.

[Sclar et al., 2023]
Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr.
Quantifying language models' sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting.
arXiv preprint arXiv:2310.11324, 2023.

[O’Connor and Joffe, 2020]
Cliodhna O’Connor and Helene Joffe.
Intercoder reliability in qualitative research: debates and practical guidelines.
International journal of qualitative methods, 19:0 1609406919899220, 2020.

[Belur et al., 2021]
Jyoti Belur, Lisa Tompson, Amy Thornton, and Miranda Simon.
Interrater reliability in systematic review methodology: exploring variation in coder decision-making.
Sociological methods & research, 500 (2):0 837–865, 2021.

[Lamprianou, 2023]
Iasonas Lamprianou.
Measuring and visualizing coders’ reliability: New approaches and guidelines from experimental data.
Sociological Methods & Research, 520 (1):0 525–553, 2023.

[Chen and Mueller, 2023]
Jiuhai Chen and Jonas Mueller.
Quantifying Uncertainty in Answers from any Language Model and Enhancing their Trustworthiness, October 2023.
URL <http://arxiv.org/abs/2308.16175>.
arXiv:2308.16175 [cs].

[Shin et al., 2020]
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh.
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts, November 2020.
URL <http://arxiv.org/abs/2010.15980>.
arXiv:2010.15980 [cs].

[Portillo Wightman et al., 2023]
Gwenyth Portillo Wightman, Alexandra Delucia, and Mark Dredze.
Strength in Numbers: Estimating Confidence of Large Language Models by Prompt Agreement.
In Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing (TrustNLP 2023), pages 326–362, Toronto, Canada, 2023. Association for Computational Linguistics.
URL <https://aclanthology.org/2023.trustnlp-1.28>.

[Zhao et al., 2021]
Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh.
Calibrate Before Use: Improving Few-Shot Performance of Language Models, June 2021.
URL <http://arxiv.org/abs/2102.09690>.
arXiv:2102.09690 [cs].

[Arora et al., 2022]
Simran Arora, Avanika Narayan, Mayee F. Chen, Laurel Orr, Neel Guha, Kush Bhatia, Ines Chami, Frederic Sala, and Christopher Ré.
Ask Me Anything: A simple strategy for prompting language models, November 2022.
URL <http://arxiv.org/abs/2210.02441>.
arXiv:2210.02441 [cs].

[Chen et al., 2023]
Yanda Chen, Chen Zhao, Zhou Yu, Kathleen McKeown, and He He.
On the Relation between Sensitivity and Accuracy in In-context Learning, February 2023.
URL <http://arxiv.org/abs/2209.07661>.
arXiv:2209.07661 [cs].

[Schick and Schütze, 2021]
Timo Schick and Hinrich Schütze.
Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference, January 2021.
URL <http://arxiv.org/abs/2001.07676>.
arXiv:2001.07676 [cs].

[Tian et al., 2023]
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D. Manning.
Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback, October 2023.
URL <http://arxiv.org/abs/2305.14975>.
arXiv:2305.14975 [cs].

[Feffer et al., 2024]
Michael Feffer, Ronald Xu, Yuekai Sun, and Mikhail Yurochkin.
Prompt exploration with prompt regression.
arXiv preprint arXiv:2405.11083, 2024.

[Wei et al., 2022]
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.
Chain-of-thought prompting elicits reasoning in large language models.
Advances in Neural Information Processing Systems, 35:0 24824–24837, 2022.

[Liu et al., 2021]
Jiacheng Liu, Alisa Liu, Ximing Lu, Sean Welleck, Peter West, Ronan Le Bras, Yejin Choi, and Hannaneh Hajishirzi.
Generated knowledge prompting for commonsense reasoning.
arXiv preprint arXiv:2110.08387, 2021.

[Long, 2023]
Jieyi Long.
Large language model guided tree-of-thought.
arXiv preprint arXiv:2305.08291, 2023.

[Wang et al., 2022]
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou.
Self-consistency improves chain of thought reasoning in language models.
arXiv preprint arXiv:2203.11171, 2022.

[Madaan et al., 2023]
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al.
Self-refine: Iterative refinement with self-feedback.
arXiv preprint arXiv:2303.17651, 2023.

[Perez et al., 2021]
Ethan Perez, Douwe Kiela, and Kyunghyun Cho.
True few-shot learning with language models.
Advances in neural information processing systems, 34:0 11054–11070, 2021.

[Liu et al., 2023]
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig.
Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.
ACM Computing Surveys, 550 (9):0 1–35, 2023.

[Dentella et al., 2023]
Vittoria Dentella, Fritz Günther, and Evelina Leivada.
Systematic testing of three language models reveals low language accuracy, absence of response stability, and a yes-response bias.
Proceedings of the National Academy of Sciences, 1200 (51):0 e2309583120, 2023.

[Bisbee et al., 2023]
James Bisbee, Joshua Clinton, Cassy Dorff, Brenton Kenkel, and Jennifer Larson.
Artificially precise extremism: how internet-trained llms exaggerate our differences.
SocArXiv Preprint, 2023.
URL <https://doi.org/10.31235/osf.io/5ecfa>.

[Bail, 2024]
Christopher A Bail.
Can generative ai improve social science?
Proceedings of the National Academy of Sciences, 1210 (21):0 e2314021121, 2024.

[Denny and Spirling, 2018]
Matthew J Denny and Arthur Spirling.
Text preprocessing for unsupervised learning: Why it matters, when it misleads, and what to do about it.
Political Analysis, 260 (2):0 168–189, 2018.

[Baden et al., 2022]
Christian Baden, Christian Pipal, Martijn Schoonvelde, and Mariken AC G van der Velden.
Three gaps in computational text analysis methods for social sciences: A research agenda.
Communication Methods and Measures, 160 (1):0 1–18, 2022.

[Van Vliet et al., 2020]
Livia Van Vliet, Petter Törnberg, and Justus Uitermark.
The twitter parliamentarian database: Analyzing twitter politics across 26 countries.
PLoS one, 150 (9):0 e0237073, 2020.

[Benoit et al., 2016]
Kenneth Benoit, Drew Conway, Benjamin E Lauderdale, Michael Laver, and Slava Mikhaylov.
Crowd-sourced text analysis: Reproducible and agile production of political data.
American Political Science Review, 1100 (2):0 278–295, 2016.

[Young and Soroka, 2012]
Lori Young and Stuart Soroka.
Affective news: The automated coding of sentiment in political texts.
Political communication, 290 (2):0 205–231, 2012.

[Mohammad et al., 2016]
Saif Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry.
Semeval-2016 task 6: Detecting stance in tweets.
In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 31–41, San Diego, California, 2016. Association for Computational Linguistics.

[Mellon et al., 2024]
Jonathan Mellon, Jack Bailey, Ralph Scott, James Breckwoldt, Marta Miori, and Phillip Schmedeman.
Do ais know what the most important issue is? using language models to code open-text social survey responses at scale.
Research & Politics, 110 (1):0 20531680241231468, 2024.

[Argyle et al., 2023]
Lisa P Argyle, Ethan C Busby, Nancy Fulda, Joshua R Gubler, Christopher Rytting, and David Wingate.
Out of one, many: Using language models to simulate human samples.
Political Analysis, 310 (3):0 337–351, 2023.

[Zhang et al., 2019]
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu.
Pegasus: Pre-training with extracted gap-sentences for abstractive summarization, 2019.

[Chang et al., 2009]
Jonathan Chang, Sean Gerrish, Chong Wang, Jordan Boyd-Graber, and David Blei.
Reading tea leaves: How humans interpret topic models.
Advances in neural information processing systems, 22, 2009.

[Grimmer and Stewart, 2013]
Justin Grimmer and Brandon M Stewart.
Text as data: The promise and pitfalls of automatic content analysis methods for political texts.
Political analysis, 210 (3):0 267–297, 2013.

[Krippendorff, 2004]
Klaus Krippendorff.
Reliability in content analysis: Some common misconceptions and recommendations.
Human communication research, 300 (3):0 411–433, 2004.