## The energy spectrum of a quantum vortex loop
moving in a long pipe

S.V. Talalov

Dept. of Applied Mathematics, Togliatti State University,
14 Belorusskaya str., Tolyatti, 445020 Russia.
<EMAIL_ADDRESS>
###### Abstract

In this study we consider the closed vortex filament that moves in a pipe of
length $L>>R_{1}$, where $R_{1}$ means inner radius of the pipe. The vortex
filament is described in the Local Induction Approximation. We quantize this
dynamical system and calculate the spectrum both circulation and energy. In
the study, we focus on the case $L\to\infty$.

keywords: local induction approximation; vortex filament quantization;
extended Galilei group.

##  1 Introduction

###

Usually, quantization of a complex dynamical system is an ambiguous procedure.
The development of quantum theory really confirms Dirac’s famous words:
”…methods of quantization are all of the nature of practical rules, whose
application depends on consideration of simplicity” [1]. For instance, the
quantization of canonically equivalent but different sets of classical
variables can lead to physically different results at the quantum level. This
is true even for simple systems from a text-book: so, the quantization of a
harmonic oscillator in terms of action-angle variables demonstrates many
unexpected and interesting results [2]. As a rule, the problem arises when we
quantize a complex dynamic system, especially in the cases where experiments
cannot confirm or disprove the final results for the spectrum of observable
values.

Our study is devoted to the quantum description of the perturbed vortex ring
of radius $R$. This ring (vortex filament) moves in the domain $D\subset{\sf
R}_{3}$ that is defined by the conditions

$\rho~{}\leq~{}R_{1}\,,\qquad-\frac{L}{2}~{}<~{}z~{}<~{}\frac{L}{2},\qquad\qquad
R_{1}<<L\,,\quad R<R_{1}\,.$ (1)

Cylindrical coordinate system $(\rho,\varphi,z)$ with corresponding basis
$(\,\boldsymbol{n}_{\rho}\,,\boldsymbol{n}_{\varphi}\,,\boldsymbol{n}_{z}\,)$
is associated with domain (1) naturally; the $z$ axis ($\boldsymbol{n}_{z}$)
is directed along the pipe. In fact, we will consider the asymptotic case
$L\to\infty$, taking into account that the value $L<\infty$ in reality. Of
course, the theory of quantum vortices has been around for decades. A lot of
articles are devoted to this issue. Here we note only some fundamental books
of [3, 4, 5] that reflect different periods of research and approaches to the
problem.

For our further studies, we use a representation for smooth closed filament of
length $S$ in the form

${\boldsymbol{r}}(\tau,\xi)~{}=~{}{\boldsymbol{q}}+{R}\int\limits_{0}^{2\pi}\left[\,{\xi-\eta}\,\right]{\boldsymbol{j}}(\tau,\eta)d\eta\,,\qquad
R=\frac{S}{2\pi}\,,\qquad\xi=\frac{s}{R}\,,$ (2)

where parameter $\tau$ describes the curve evolution. Vector
${\boldsymbol{q}}={\boldsymbol{q}}(\tau)$ defines the position of the filament
(for example, the center of the circular vortex ring) in some coordinate
system. The notation $[\,x\,]$ means the integer part of the number
$x/{2\pi}$:

$[\,0\,]=0\,,\qquad[\,x+2\pi\,]=[\,x\,]+1\,,\qquad\forall\,x\,.$ (3)

Vector ${\boldsymbol{j}}(\tau,\eta)$ is the unit tangent (affine) vector for
this filament. Let the vector ${\boldsymbol{r}}(\tau,\xi)$ satisfies the Local
Induction Equation (LIE, see [6] for example):

$\partial_{\tau}{\boldsymbol{r}}(\tau,\xi)~{}=~{}\frac{\alpha}{R}\,\partial_{\xi}{\boldsymbol{r}}(\tau,\xi)\times\partial_{\xi}^{\,2}{\boldsymbol{r}}(\tau,\xi)\,.$
(4)

Evolution parameter $\tau$ was defined here as $\tau={t\Gamma}/{4\pi R^{2}}$,
where $t$ is real time and symbol $\Gamma$ means the circulation. Let’s note
that we have redefined the evolution parameter for the curve (2) only. Any
physical quantity (for example, circulation $\Gamma$) that is not associated
directly with curve (2) as a geometric object, remains unchanged.
Dimensionless parameter $\alpha$ depends on the filament core radius $a<<R$ as
well as some details of regularization procedure that has been fulfilled to
deduce equation (4) (see, for example, [6, 7]). In the future, we will exclude
this parameter from consideration by redefining of the parameter $\tau$:
$\tau\to\alpha\tau$ Then, the function ${\boldsymbol{j}}(\tau,\eta)$ satisfies
the equation for continuous Heisenberg spin chain:

$\partial_{\tau}{\boldsymbol{j}}(\tau,\xi)~{}=~{}{\boldsymbol{j}}(\tau,\xi)\times\partial_{\xi}^{\,2}{\boldsymbol{j}}(\tau,\xi)\,.$
(5)

##  2 Classical dynamic system

###

We consider the small ($\varepsilon<<1$) perturbation
${\boldsymbol{j}}_{p}(\tau,\xi)$ of the vortex ring
${\boldsymbol{j}_{0}}(\xi)$:

$\boldsymbol{j}(\tau,\xi)~{}=~{}{\boldsymbol{j}_{0}}(\xi)+\varepsilon{\boldsymbol{j}}_{p}(\tau,\xi)\,,\qquad\qquad{\boldsymbol{j}}_{p}(\tau,\xi+2\pi)={\boldsymbol{j}}_{p}(\tau,\xi)\,,$
(6)

where unperturbed vector $\boldsymbol{j}_{0}(\xi)$ is a tangent vector for the
circular filament:

$\boldsymbol{j}_{0}(\xi)~{}=~{}\boldsymbol{e}_{\phi}\,.$

The vectors
$({\boldsymbol{e}_{\rho}}\,,\boldsymbol{e}_{\phi}\,,{\boldsymbol{e}_{z}}\,)$
denote the local cylindrical basis that is associated naturally with this
circular filament. Please note that this basis differs from the basis
$(\,\boldsymbol{n}_{\rho}\,,\boldsymbol{n}_{\varphi}\,,\boldsymbol{n}_{z}\,)$
which was previously associated with domain $D$.

Additionally, we suppose that the excitations (6) be transverse:

${\boldsymbol{j}}_{p}(\tau,\xi){\boldsymbol{j}_{0}}(\xi)~{}\equiv~{}0\,$ (7)

The small perturbation of the circular vortex filament without the restriction
(7) has been studied in the work [8]. Various aspects of the theory related to
vortex ring oscillations have also been studied in the works [9, 10]. The
small perturbations of straight vortex filaments were studied in the work
[11].

Condition (7) leads to decomposition

${\boldsymbol{j}}_{p}(\tau,\xi)~{}=~{}j_{\rho}(\tau,\xi){\boldsymbol{e}}_{\rho}+j_{z}(\tau,\xi){\boldsymbol{e}_{z}}\,,$
(8)

For convenience, we define the complex-valued function

${\mathfrak{J}}(\tau,\xi)~{}=~{}j_{\rho}(\tau,\xi)+{\rm i}j_{z}(\tau,\xi)\,.$

This function allows us define the complex amplitudes
${\mathfrak{j}}_{\,n}(\tau)$:

${\mathfrak{J}}(\tau,\xi)~{}=~{}\sum_{n}{\mathfrak{j}}_{\,n}(\tau)\,e^{\,i\,\,n\xi}\,.$
(9)

Taking into account the terms of the order $\varepsilon$ only, we can deduce
that amplitude ${\mathfrak{J}}(\tau,\xi)$ satisfies the linear equation [12]

$\partial_{\tau}{\mathfrak{J}}=-i\partial_{\xi}^{\,2}{\mathfrak{J}}-\frac{i}{2}\Bigl{(}{\mathfrak{J}}-\overline{\,{\mathfrak{J}}\,}\,\Bigr{)}\,.$
(10)

Consequently,

${\mathfrak{J}}(\tau,\xi)~{}=~{}\sum_{n}{\mathfrak{j}}_{\,n}\,e^{\,i\,[\,n\,\xi+n\sqrt{n^{2}-1}\,\tau\,]}\,,$
(11)

where the amplitudes ${\mathfrak{j}}_{\,n}$ satisfy the condition
$\overline{\,{\mathfrak{j}}\,}_{\,-n}~{}=~{}2\left[n\sqrt{n^{2}-1}-n^{2}+{1}/{2}\right]{\mathfrak{j}}_{\,n}\,.$
Thus, we can declare the vector $\boldsymbol{q}$ and amplitudes
${\mathfrak{j}}\,_{\,-n}$, where $n=0,1,2,\dots$, as independent variables.
These variables describe the dynamics of small-perturbed circle-like curve
${\boldsymbol{r}}(\tau,\xi)$ in accordance with Eq.(4). Note that we still
describe the evolution of curve (2) as a formal geometric object only.
According to our assumption, the curve ${\boldsymbol{r}}(\tau,\xi)$, is a
vortex filament. However, variables $\boldsymbol{q}$ and
${\mathfrak{j}}\,_{\,-n}$ do not take into account the movement of the
surrounding fluid any way. That is why we include the circulation $\Gamma$ as
an additional independent variable in our theory. Therefore, we describe the
dynamics of a filament by the following set ${\cal A}$ of the independent
variables:

${\cal A}~{}=~{}\\{\,\Gamma\,,\boldsymbol{q}\,,\boldsymbol{j}(\xi)\,\\}\,.$

Although these variables take into account the movement of the surrounding
fluid in a minimal way, they are inconvenient for subsequent quantization.
Therefore, we replace the set ${\cal A}$ by the set ${\cal A}^{\prime}$, where

${\cal A}~{}\subset~{}{\cal
A}^{\,\prime}~{}=~{}\bigl{\\{}\,{\boldsymbol{p}}\,,{\boldsymbol{q}}\,;{\boldsymbol{j}}(\xi)\,\bigr{\\}}\,.$
(12)

Inclusion ${\cal A}~{}\subset~{}{\cal A}^{\,\prime}$ is performed by means of
a well-known hydrodynamic formula

${\boldsymbol{p}}~{}=~{}\frac{\varrho_{0}}{2}\,\int\,\boldsymbol{r}\times\boldsymbol{w}(\boldsymbol{r})\,dV\,,$
(13)

where constant $\varrho_{0}$ means a fluid density and vector-function
$\boldsymbol{w}(\boldsymbol{r})$ means the vorticity of the vortex filament.
This function is calculated as

${\boldsymbol{w}}(\boldsymbol{r})~{}=~{}\Gamma\int\limits_{0}^{2\pi}\,\hat{\delta}(\boldsymbol{r}-\boldsymbol{r}(\xi))\partial_{\xi}{\boldsymbol{r}}(\xi)d\xi\,.$
(14)

Compact form for the formulas (13) and (14) is more convenient. Taking into
account Eq.(2), we can write:

${\boldsymbol{p}}~{}=~{}\varrho_{0}{R}^{2}\Gamma{\boldsymbol{f}}\,,\qquad{\boldsymbol{f}}~{}=~{}\frac{1}{2}\iint\limits_{0}^{2\pi}\left[\,{\xi-\eta}\,\right]\,{\boldsymbol{j}}(\eta)\times{\boldsymbol{j}}(\xi)d\xi
d\eta\,.$ (15)

It is clear that the replacement $\Gamma\to{\boldsymbol{p}}$ leads to the
certain constraints on the set ${\cal A}^{\,\prime}$ in general. This issue
has been investigated in the article [12] in detail. For our subsequent
purposes, we assume ${\mathfrak{j}}\,_{\,0}=0$, just as
${\mathfrak{j}}\,_{\,-1}=0$ (see Eq. (11)). The last assumption makes
independent momentum variations $\delta{\boldsymbol{p}}$ and small variations
${\boldsymbol{j}}_{p}$: so, the equality

${\boldsymbol{p}}~{}=~{}\pi\varrho_{0}{R}^{2}\Gamma{\boldsymbol{e}}_{z}$ (16)

is fulfilled in the case ${\mathfrak{j}}\,_{\,-1}=0$.

We are considering a non-relativistic theory; therefore, it is natural to
declare the Galilei group ${\mathcal{G}}_{3}$ as the space-time symmetry group
of our model. Moreover, we will consider the central extension
$\widetilde{\mathcal{G}}_{3}$ of this group here111We use one parameter
central extension.. Corresponding central charge $m_{0}$ (this constant is
interpreted as a ”mass”) completes the list of fundamental constants of the
theory. The other constants are as follows: the fluid’s density $\varrho_{0}$,
the speed of sound in this fluid $v_{0}$. In order to simplify some formulas,
we will use the auxiliary constants $R_{0}=\sqrt[3\,]{m_{0}/\varrho_{0}}$,
$~{}t_{0}=R_{0}/v_{0}$ and ${\cal E}_{0}=m_{0}v_{0}^{2}$ along with constants
$\varrho_{0}$, $v_{0}$ and $m_{0}$.

The presence of the central extended Galilei group allows us to apply the
group-theoretical approach to the definition of the energy of thin vortex
filament. Indeed, the Lee algebra of the group $\widetilde{\mathcal{G}}_{3}$
has three Cazimir functions:

${\hat{C}}_{1}=m_{0}{\hat{I}}\,,\quad{\hat{C}}_{2}=\left({\hat{M}}_{i}-\sum_{k,j=x,y,z}\epsilon_{ijk}{\hat{P}}_{j}{\hat{B}}_{k}\right)^{2}\quad{\hat{C}}_{3}=\hat{H}-\frac{1}{2m_{0}}\sum_{i=x,y,z}{\hat{P}}_{i}^{\,2}\,,$

where ${\hat{I}}$ is the unit operator, ${\hat{M}}_{i}$, $\hat{H}$,
${\hat{P}}_{i}$ and ${\hat{B}}_{i}$ ($i=x,y,z$) are the respective generators
of rotations, time and space translations and Galilean boosts. As it is well
known, the function ${\hat{C}}_{3}$ can be interpreted as an ”internal energy
of the particle”. In our model, the internal degrees of freedom are described
by the function ${\mathfrak{J}}(\tau,\xi)$. This function is invariant under
Galilei transformation; that is why
${\hat{C}}_{3}={\hat{C}}_{3}({\mathfrak{J}})$ in general. Let us postulate a
formula for the Cazimir functions ${\hat{C}}_{3}$:

${\hat{C}}_{3}~{}=~{}{\mathcal{E}}_{0}\sum_{n>1}|\,{\mathfrak{j}}_{\,-n}|^{2}n\sqrt{n^{2}-1}\,.$

This formula will be justified after determining the Hamiltonian structure of
the theory. Therefore, the following expression for the energy ${\mathcal{E}}$
is natural for our model:

${\mathcal{E}}({\boldsymbol{p}}\,;{\mathfrak{j}}\,)=\frac{{\boldsymbol{p}}^{\,2}}{2m_{0}}+{\mathcal{E}}_{0}\sum_{n>1}|\,{\mathfrak{j}}_{\,-n}|^{2}n\sqrt{n^{2}-1}\,.$
(17)

Let us define the Hamiltonian structure.

  * •

The set ${\cal A}^{\,\prime}$ is interpreted as a phase space ${\mathcal{H}}$
quite naturally: ${\mathcal{H}}={\mathcal{H}}_{3}\times{\mathcal{H}}_{j}$. The
space ${\mathcal{H}}_{3}$ is parametrized by the variables ${\boldsymbol{q}}$
and ${\boldsymbol{p}}$; this space is interpreted as a phase space of $3D$
free structureless particle. The phase space ${\mathcal{H}}_{j}$ is
parametrized by the complex variables ${\mathfrak{j}}_{\,-n}$,
$\overline{\,\mathfrak{j}}_{\,-n}$ ($n=0,1,\dots$).

  * •

Poisson structure:

$\displaystyle\\{p_{i}\,,q_{j}\\}$ $\displaystyle=$
$\displaystyle\delta_{ij}\,,\qquad i,j=x,y,z\,,$ (18)
$\displaystyle\\{{\mathfrak{j}}_{\,m},\overline{\,\mathfrak{j}\,}_{\,n}\\}$
$\displaystyle=$
$\displaystyle(i/{\mathcal{E}}_{0}t_{0})\,\delta_{mn}\,,\qquad
m,n=-1,-2,\dots$ (19)

All other brackets vanish. The real variable ${\mathfrak{j}}_{\,0}$ annulates
all brackets. Thus, the Poisson structure of the theory is degenerate in
general: the value ${\mathfrak{j}}_{\,0}$ marks the symplectic sheets where
the structure will be non-degenerate.

  * •

Hamiltonian

$H({\boldsymbol{p}}\,;{\mathfrak{j}}\,)~{}=~{}{\mathcal{E}}({\boldsymbol{p}}\,;{\mathfrak{j}}\,)\,,$
(20)

where the function ${\mathcal{E}}({\boldsymbol{p}}\,;{\mathfrak{j}})$ was
defined by the formula (17).

Hamiltonian (20) defines the evolution of the filament ${\boldsymbol{r}}(\xi)$
in ”conditional” time $t^{\\#}=t_{0}\tau$. It is not difficult to make sure
that the Hamiltonian (20) and the Poisson brackets (18), (19) generate
dynamics

$\displaystyle{\boldsymbol{q}}(0)$ $\displaystyle\longrightarrow$
$\displaystyle{\boldsymbol{q}}(t^{\\#})~{}=~{}{\boldsymbol{q}}(0)~{}+~{}\frac{\boldsymbol{p}}{m_{0}}\,t^{\\#}\,,$
(21) $\displaystyle{\mathfrak{j}}_{-n}(0)$ $\displaystyle\longrightarrow$
$\displaystyle{\mathfrak{j}}_{-n}(t^{\\#})~{}=~{}{\mathfrak{j}}_{-n}(0)\,e^{\,-i\,n\sqrt{n^{2}-1}\,(t^{\\#}/t_{0})\,}$

Among other things, these formulas demonstrate the fulfilled extension

$\bigl{\\{}\,{\boldsymbol{q}}\,;{\boldsymbol{j}}(\xi)\,\bigr{\\}}~{}\longrightarrow~{}\bigl{\\{}\,{\boldsymbol{p}}\,,{\boldsymbol{q}}\,;{\boldsymbol{j}}(\xi)\,\bigr{\\}}\,$

of the original dynamic system (4). Indeed, in accordance with Eq.(4)

$\frac{d{\boldsymbol{q}}}{dt^{\\#}}~{}=~{}\frac{R}{t_{0}}\,{\boldsymbol{e}}_{z}\,.$

As opposed to Eq.(21), this formula corresponds to the following dependence
${\boldsymbol{q}}={\boldsymbol{q}}(t^{\\#})$:

${\boldsymbol{q}}(0)~{}\longrightarrow~{}{\boldsymbol{q}}(t^{\\#})~{}=~{}{\boldsymbol{q}}(0)~{}+~{}{R}\,(t^{\\#}/t_{0})\,{\boldsymbol{e}}_{z}\,.$

##  3 Quantization and energy spectrum

###

Let’s quantize the constructed dynamic system. Firstly, we define a Hilbert
space $\boldsymbol{H}$ of the quantum states. The structure of the phase space
${\mathcal{H}}$ makes the following construction natural:

$\boldsymbol{H}=\boldsymbol{H}_{3}\otimes\boldsymbol{H}_{j}\,,$ (22)

where the symbol $\boldsymbol{H}_{3}$ denotes the Hilbert space of a free
structureless particle in the space ${\sf R}_{3}$ (the space $L^{2}({\sf
R}_{3})$ in our case) and symbol $\boldsymbol{H}_{j}$ denotes the Fock space
for the infinite number of the harmonic oscillators. This space is formed by
the basic vectors

$\bigl{(}\hat{a}^{+}(n_{1})\bigr{)}^{s_{1}}\dots\bigl{(}\hat{a}^{+}(n_{N})\bigr{)}^{s_{N}}|0\rangle\,,\qquad
n_{i}\not=n_{j}\quad for\quad i\not=j\,,$

where oscillators are numbered with numbers $n_{1},\dots,n_{N}$,
$N=0,1,2,\dots$ and symbols $s_{j}$ denote corresponding occupation numbers.
Creation and annihilation operators in the space $\boldsymbol{H}_{j}$ have
standard commutation relations

$[\,\hat{a}(m),\hat{a}^{+}(n)]=\delta_{mn}\hat{I}_{j}\,,\qquad\hat{a}(m)|\,0\rangle=0\,,\qquad
m,n=1,2,\dots\,,\qquad|\,0\rangle\in\boldsymbol{H}_{j}\,,$

where the operator $\hat{I}_{j}$ is unit operator in the space
$\boldsymbol{H}_{j}$.

Our postulate of quantization is following:

$\boldsymbol{q}~{}\to~{}\boldsymbol{q}\otimes\,\hat{I}_{j}\,,\qquad\boldsymbol{p}~{}\to~{}-i\hbar\nabla\otimes\,\hat{I}_{j}\,,\qquad{\mathfrak{j}}_{\,-n}~{}\to~{}\sqrt{\frac{\hbar}{t_{0}{\mathcal{E}}_{0}}}\,\Bigl{(}\hat{I}_{3}\otimes\,\hat{a}(n)\Bigr{)}\,,$

where $n=1,2,\dots\,$. The unit operator $\hat{I}_{3}$ acts in the space
$L^{2}({\sf R}_{3})$. In order to downsize the formulas, we will not write the
constructions $(\dots\otimes\,\hat{I}_{j})$ and $(\hat{I}_{3}\otimes\,\dots)$
explicitly, hoping that this simplification will not lead to
misunderstandings. As usual in a field theory, we postulate the normal
ordering for the operators $\hat{a}(m)$ and $\hat{a}^{+}(n)$ when we quantize
any products of the variables ${\mathfrak{j}}_{\,-n}$ and
$\overline{\,{\mathfrak{j}}\,}_{\,-m}$. For example,

$|{\mathfrak{j}}_{\,-n}|^{2}~{}\to~{}\frac{\hbar}{t_{0}{\mathcal{E}}_{0}}\hat{a}^{+}(n)\hat{a}(n)\,.$

In this study we consider our system in the domain (1) only. Before
calculating the energy spectrum, we quantize circulation $\Gamma$. Let’s pay
attention that our approach allows us to find the spectrum of the value
$\Gamma$ using conventional methods of quantum theory. Usually, the spectrum
of this physical value is postulated, similar to the quantization rules in the
old Bohr quantum mechanics. Arguments in favor of generalizing the standard
expression for quantum circulation were discussed in more detail in the
author’s article [13].

Squaring the Eq. (16), we have $SO(3)$ \- invariant expression

$\boldsymbol{p}^{2}~{}-~{}\pi^{2}\varrho_{0}^{2}R^{4}\Gamma^{2}~{}=~{}0\,.$

Therefore, quantized circulation values are the eigenvalues of the spectral
problem in the space $L^{2}({\sf R}_{3})$:

$\left[-\hbar^{2}\Delta~{}-~{}\pi^{2}\varrho_{0}^{2}R^{4}\Gamma^{2}\,\right]\,\Psi_{\Gamma}(\boldsymbol{r})~{}=~{}0\,,\qquad\Psi_{\Gamma}(\boldsymbol{r})\in
L^{2}({\sf R}_{3})\,.$ (23)

Solving this simple spectral problem, we find the spectrum of the value
$\Gamma$:

$\Gamma_{n,m,k}~{}=~{}\frac{\hbar}{\pi\varrho_{0}R^{2}}\sqrt{~{}\biggl{(}\frac{\pi
n}{L}\biggr{)}^{2}+\biggl{(}\frac{\zeta_{\,k}^{(m)}}{R_{1}}\biggr{)}^{2}~{}}\,,\qquad
n,m=1,2,\dots,\quad k=0,1,2,\dots\,.$ (24)

The number $\zeta_{k}^{(m)}$ means $m$-th zero of the Bessel function
$J_{k}(x)$.

Let us investigate some limiting cases for the formula (24).

  1. 1.

The value $L$ (”length of the pipe”) is finite and be comparable with the pipe
radius $R_{1}$, quantum numbers $n$ and $k$ are small. Because the asymptotic
behavior

$\zeta_{k}^{(m)}~{}\longrightarrow~{}-\frac{\pi}{4}+\frac{\pi k}{2}+\pi
m\,,\qquad m\to\infty$

takes place, the formula (24) has the following asymptotic behavior for large
values of the number $m$:

$\Gamma_{n,m,k}~{}\longrightarrow~{}\Gamma_{m}~{}=~{}\frac{\hbar\,m}{\varrho_{0}R^{2}R_{1}}\,.$

  2. 2.

The value $L$ is very large, quantum numbers $n$, $m$ and $k$ take arbitrary
but finite values. In this case circulation $\Gamma$ takes the values within
the bounded interval $(\,\Gamma_{min},\Gamma_{max}\,)$. Obviously, the value
$\Gamma_{min}=\hbar\zeta_{\,0}^{(0)}/\pi\varrho_{0}R^{2}R_{1}$. The gap
between adjacent values is as follows:

$\delta_{n}\Gamma_{n,m,k}~{}\equiv~{}\Gamma_{n+1,m,k}-\Gamma_{n,m,k}~{}=~{}\frac{\pi\hbar
R_{1}}{2\varrho_{0}R^{2}}\,\frac{2n+1}{\zeta_{\,k}^{(m)}L^{2}}~{}+~{}{\cal
O}\left(\frac{1}{L^{4}}\right)\,.$ (25)

To define the value $\Gamma_{max}$, we postulate that the vortex momentum
$\boldsymbol{p}$ is bounded in our theory:

$|\,\boldsymbol{p}\,|~{}<~{}m_{0}v_{0}\,.$

This restriction can be justified by the fact that we consider only subsonic
fluid movements. Taking into account Eq. (16), we deduce

$\Gamma_{max}~{}=~{}\frac{m_{0}v_{0}}{\pi\varrho_{0}R^{2}}~{}=~{}\hbar\frac{{\sf
k}_{max}}{\pi\varrho_{0}R^{2}}\,,$ (26)

where the number ${\sf k}_{max}=m_{0}v_{0}/\hbar$ is maximal De Broglie wave
number. Thus, we have following formula for the case $L\to\infty$:

$\Gamma~{}=~{}\frac{\gamma\,\hbar}{\pi\varrho_{0}R^{2}R_{1}}\,,$ (27)

where the real number $\gamma\in(\,\zeta_{\,0}^{(0)}\,,{\sf k}_{max}R_{1})$.
Formula (27) allows us to interpret the number $\gamma$ as a ”dimensionless
quantum circulation”. We also assume that $\zeta_{\,0}^{(0)}<<{\sf
k}_{max}R_{1}$ in this model.

Our next step is to find the energy spectrum of the dynamical system under
consideration. Here we need to remember that energy is a physical value that
is associated with time translations. Therefore, in the end, we must consider
the evolution of the vortex loop in real time $t$ instead of ”conditional”
time $t^{\\#}$:

$t^{\\#}~{}\to~{}t~{}=~{}\frac{4\pi R^{2}}{\alpha\,t_{0}\Gamma}\,t^{\\#}\,.$
(28)

First, we consider the $t^{\\#}$ \- evolution. Quantized Hamiltonian (20) has
following form:

$\widehat{H}~{}=~{}-\frac{\hbar^{2}}{2m_{0}}\Delta~{}+~{}\frac{\hbar}{t_{0}}\sum_{n>1}n\sqrt{n^{2}-1}\,\hat{a}^{+}(n)\hat{a}(n)\,.$
(29)

Spectral problem

$\widehat{H}|\Psi\rangle~{}=~{}E^{\\#}|\Psi\rangle$

has following solutions:

$\boldsymbol{H}\ni|\Psi\rangle~{}\equiv~{}|\Psi(\Gamma;\ell,s_{\ell})\rangle~{}=~{}|\Psi_{\Gamma}\rangle|\ell,s_{\ell}\rangle\,,\qquad\quad|\ell,s_{\ell}\rangle=\bigl{(}\hat{a}^{+}(\ell)\bigr{)}^{s_{\ell}}|0\rangle\in\boldsymbol{H}_{j}\,,$

where the vector $|\Psi_{\Gamma}\rangle\in L^{2}({\sf R}_{3})$ is the
eigenvector of the spectral problem (23) that corresponds to certain
eigenvalue $\Gamma$. Taking into account formula (23), the eigenvalues
$E^{\\#}_{nmk}(\ell,s_{\ell})$ are written as follows:

$E^{\\#}_{nmk}(\ell,s_{\ell})~{}=~{}\frac{(\pi\varrho_{0}R^{2})^{2}}{2m_{0}}\Gamma_{n,m,k}^{\,2}~{}+~{}\frac{\hbar}{t_{0}}\,\ell\sqrt{\ell^{2}-1}\,s_{\ell}\,,$
(30)

where numbers $\Gamma_{n,m,k}$ have been defined in accordance with formula
(24). Next, we will consider the case $L\to\infty$ only. Since this case has
been investigated above (see Eq.(27), we can write down the formula for the
energy
$E^{\\#}(\gamma;\ell,s_{\ell})=\lim_{L\to\infty}E^{\\#}_{nmk}(\ell,s_{\ell})$:

$E^{\\#}(\gamma;\ell,s_{\ell})~{}=~{}\frac{\gamma^{2}\hbar^{2}}{2m_{0}R_{1}^{\,2}}~{}+~{}\frac{\hbar}{t_{0}}\,\ell\sqrt{\ell^{2}-1}\,s_{\ell}\,,\qquad\gamma\in(\zeta_{\,0}^{(0)}\,,{\sf
k}_{max}R_{1})\,.$ (31)

Let us consider the $t^{\\#}$ \- evolution of any vector
$|\Psi\rangle\in{\boldsymbol{H}}$:

$\exp\left(\frac{i\widehat{H}t^{\\#}}{\hbar}\right)|\Psi\rangle~{}=~{}\sum_{\Gamma;\ell,s_{\ell}}C_{\gamma;\ell,s_{\ell}}\exp\left(\frac{iE^{\\#}(\gamma;\ell,s_{\ell})t^{\\#}}{\hbar}\right)|\Psi(\Gamma;\ell,s_{\ell})\rangle\,,$
(32)

where
$C_{\gamma;\ell,s_{\ell}}=\langle\Psi(\Gamma;\ell,s_{\ell})|\Psi\rangle$.
Next, we restore the real time $t$ in this formula using Eq.(28).

Figure 1: The energy spectrum for the case $L\to\infty$.

We assume that following equality takes place for the real energy
$E(\gamma;\ell,s_{\ell})$:

$E^{\\#}(\gamma;\ell,s_{\ell})t^{\\#}~{}=~{}E(\gamma;\ell,s_{\ell})t\,.$

Therefore, the ”real - time” evolution of any vector
$|\Psi\rangle\in{\boldsymbol{H}}$ is written as

$|\Psi\rangle~{}\longrightarrow~{}|\Psi(t)\rangle~{}=~{}\sum_{\Gamma;\ell,s_{\ell}}C_{\gamma;\ell,s_{\ell}}\exp\left(\frac{iE(\gamma;\ell,s_{\ell})\,t}{\hbar}\right)|\Psi(\Gamma;\ell,s_{\ell})\rangle\,,$

where

$E(\gamma;\ell,s_{\ell})~{}=~{}\frac{\alpha\,\hbar^{2}}{4\pi^{2}\varrho_{0}R^{4}R_{1}}\left(\frac{R_{0}}{2R_{1}}\,\frac{\gamma^{3}}{{\sf
k}_{max}R_{1}}~{}+~{}\gamma\,\ell\sqrt{\ell^{2}-1}\,s_{\ell}\right)\,.$ (33)

Finally, the numbers $E(\gamma;\ell,s_{\ell})$ give the energy spectrum of our
quantum system. The dependence $E=E(\gamma;\ell,s_{\ell})$ is shown in Fig.1
for some conventional scale units.

Of course, the limit $L=\infty$ is unattainable really. Therefore, the
”dimensionless circulation” $\gamma$ takes on a value in discrete set
$G_{\epsilon}=(\gamma_{1},\gamma_{2},\dots,\gamma_{n},\dots)$ only. This set
$G_{\epsilon}$ is the $\epsilon$-net for the interval
$(\zeta_{\,0}^{(0)}\,,{\sf k}_{max}R_{1})$. In accordance with formulas (25)
and (27) parameter $\epsilon$ that characterizes the set $G_{\epsilon}$ is
defined as follows:

$\epsilon~{}=~{}\epsilon(m,n,k)~{}=~{}\frac{\pi^{2}}{4}\,\frac{2n+1}{\zeta_{\,k}^{(m)}}\left(\frac{R_{1}}{L}\right)^{2}\,.$

Thus, the curves that were shown in Fig.1 are not continuous. Really, these
curves consist of many nearby points.

##  4 Concluding remarks

###

As it seems, proposed quantum system is ”inverse realization” of Lord Kelvin’s
old idea that any particles can be considered as some kind of vortex-like
structures [14]. In our approach, we describe the closed vortex filament as
some structured particle. In a sense, the fluid in which the vortex in
question moves plays the role of ”aether”. Note that such ”aether” is quite
real here, and not hypothetical. It is known that the standard approach to the
energy definition of a thin ($a\to 0$) vortex filament leads to problems due
to the divergence of integrals. In this study, we applied a group-theoretical
approach to the definition of energy. Unfortunately, the author did not find
any experimental studies, the results of which could be compared with the
formula (33). Hopefully, such studies will appear in the future.

## References

  * [1] Dirac P.A.M. Generalized hamiltonian dynamics. Canadian Journal of Mathematics. V.2. No 2. pp. 129 - 148 (1950).
  * [2] Kastrup H. A. Annalen der Physik. 519. Issues 7-8. 439 - 528 (2007).
  * [3] R.J. Donnely. Quantum Vortices in Helium II Cambrige Univ. Press. (1991).
  * [4] S.K. Nemirovskii. Gydrodynamics of quantum fluids. Waves, vortices, turbulence. Part II. Quantum vortices, superfluid turbulence. Novisibirsk: Siberian Branch of the Russian Academy of Sciences (2016) In Russian, ISBN 978-5-7692-1482-0
  * [5] Sonin E.B. Dynamics of Quantized Vortex in Superfluids. Cambridge University Press (2016).
  * [6] P.G. Saffman, Vortex dynamics. Cambrige Univ. Press, (1992).
  * [7] S.V. Alekseenko, P.A. Kuibin, V.L. Okulov, Theory of concentrated vortices. Springer-Verlag, Berlin Heidelberg (2007).
  * [8] Ricca R.L. Chaos. 3. Issue 1. 83–91 (1993)
  * [9] V.F. Kop’ev, S.A. Chernyshev. Vortex ring oscillations, the development of turbulence in vortex rings and generation of sound. Phys. Usp. 43 663–690 (2000).
  * [10] L. Kiknadze, Yu. Mamaladze. Journ. Low Temperature Physics. V. 126. Nos 1/2. Jan. (2002).
  * [11] A.J. Majda and A.L. Bertozzi. Vorticity and Incompressible Flow.  Cambridge Univ. Press. (2002).
  * [12] S.V. Talalov. Eur. Journ. Mech B/Fluids. 92. pp. 100 - 106. (2022). arXiv: math-ph/2112.04859v1.
  * [13] S.V. Talalov. Phys. Rev. Fluids. 8. 034607 (2023)
  * [14] W. Thomson. On vortex atoms. Proc. R. Soc. Edinburgh 6, 94 (1869).


#  Re-Simulation-based Self-Supervised Learning for Pre-Training Foundation
Models

P. Harris  Massachusetts Institute of Technology, Cambridge, United States
Institute for Artificial Intelligence and Fundamental Interactions, Cambridge,
United States  M. Kagan111Corresponding authors<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>SLAC National Accelerator
Laboratory, Stanford, United States  J. Krupa∗ Massachusetts Institute of
Technology, Cambridge, United States  Institute for Artificial Intelligence
and Fundamental Interactions, Cambridge, United States  B. Maier∗ I-X,
Imperial College, London, United Kingdom  N. Woodward  Massachusetts Institute
of Technology, Cambridge, United States

###### Abstract

Self-Supervised Learning (SSL) is at the core of training modern large machine
learning models, providing a scheme for learning powerful representations that
can be used in a variety of downstream tasks. However, SSL strategies must be
adapted to the type of training data and downstream tasks required. We propose
RS3L, a novel simulation-based SSL strategy that employs a method of _re-
simulation_ to drive data augmentation for contrastive learning. By
intervening in the middle of the simulation process and re-running simulation
components downstream of the intervention, we generate multiple realizations
of an event, thus producing a set of augmentations covering all physics-driven
variations available in the simulator. Using experiments from high-energy
physics, we explore how this strategy may enable the development of a
foundation model; we show how R3SL pre-training enables powerful performance
in downstream tasks such as discrimination of a variety of objects and
uncertainty mitigation. In addition to our results, we make the RS3L dataset
publicly available for further studies on how to improve SSL strategies.

##  1 Introduction

Self-Supervised Learning (SSL) has been a key driver in recent machine
learning (ML) advancements; through the use of data augmentation, SSL
algorithms rely on pseudo labels to create representations of the data that
can be highly useful and efficiently adapted for downstream tasks.
Accordingly, the resulting SSL trained model can act as a powerful _foundation
model_ [1], defined here as a model that is ``pre-trained'' on a generic task,
in this case one defined with SSL, and capable of being used in a variety of
downstream tasks. Foundation models have quickly arisen as a strategy for
describing complex tasks, especially in computer vision (e.g., [2]), natural
language processing (e.g., [3]), and speech recognition (e.g., [4]). More
recently, foundation models for machine learning in science have become an
important and active area of research (for instance in biology [5], chemistry
[6, 7, 8, 9], differential equation solving [10, 11], and cosmology [12, 13])

SSL benefits from unlabeled training and from the creation of a relation
between data augmentations. Leveraging vast unlabeled datasets can enable SSL
to outperform supervised learning. In this work, we focus on contrastive
learning, a variant of SSL. We propose a new strategy for data augmentation
and study how it performs within the SimCLR framework, which was first
introduced in [14]. In SimCLR, the learning objective is to map a data point
and its augmentation(s) to similar representations, while pushing different
data points toward differing representations. In contrastive learning, the
quality of the learning task is highly dependent on the set of augmentations.
Domain completeness, i.e., when the augmentation set covers all possible
variations in the data, can lead to better representation learning because the
model can gain a full view of plausible data during learning.

To that end, we propose a simulation-based strategy to obtain an augmented
data set that is as domain complete as possible and only limited by the
knowledge encapsulated in a simulator. More specifically, in settings with
stochastic simulators, we propose a strategy whereby we _intervene_ in the
middle of a simulation process, fix the upstream generated latent state of an
event, and _re-simulate_ downstream components to sample augmentations from
the set of all plausible observations for a given latent state. From a
programmatic standpoint, the first step fixes all initial conditions and
stochastic latent variables in the simulation up to the intervention point,
while the second step re-samples the stochastic latent variables in the
simulation after the intervention point and thus generates a new stochastic
output of the simulation conditioned on the fixed latent state at the
intervention point. We dub our new strategy RS3L ([”rIz9l], ``Re-simulation-
based self-supervised representation learning''). We will show that this
method allows for powerful contrastive pre-training that, through altering the
simulator settings for re-simulation, also aids in the mitigation of
uncertainties arising from domain shift between potentially imperfect
simulations and real data.

To test our method, we will focus our experiments on High Energy Physics
(HEP), where high-fidelity simulators are readily available (see [15] for an
overview). In our case, the fixed latent state is represented by the
elementary particle generated in a hard-scattering process, which can be
described with perturbative quantum field theory. We then re-simulate the
steps in which secondary particles are created through radiation off of the
particles produced in the hard scattering, the subsequent hadronization of
secondary particles into stable, bound particle states, and then the stable
particle interaction with the detector material. In the following, this step
is referred to as _parton showering_.

In particular, we will focus on learning representations of jets, which are
highly energetic, collimated streams of secondary particles. Jets are core
objects in HEP; they are produced from the showering and hadronization of
high-energy quarks and gluons or through the decay of heavy elementary
particles like the Higgs boson, the top quark, or heavy weak gauge bosons.
Jets are featured in many theories for new physics beyond the Standard Model.
Hence, the ability to ``tag'' a jet, i.e., to correctly and robustly classify
the particle which initiated the jet, is crucial for the success of the
physics program of experiments like ATLAS [16] and CMS [17] at the Large
Hadron Collider (LHC) [18] and future high energy collider experiments. Our
choice of re-simulating the parton shower is motivated by a desire to
characterize the progenitor of a jet, while capturing the large variability
that results for each parton shower. That is, the re-simulated shower defines
an augmentation to the existing shower that we claim captures all the physics
that we desire downstream.

By sampling from our high-fidelity simulator to obtain augmented versions of
jets, our work presents an evolution of the methods developed in JetCLR [19],
which focused on empirical methods of augmentation generation, whereas our
work develops a domain-complete augmentation set based concretely on the
physics in the simulation. As a result, we outline a strategy through
augmentations within SSL to pre-train a latent space that captures the most
salient, physically motivated, features of a simulation. Within high energy
physics, other SSL methods have also been explored for foundation model pre-
training. For instance, the masked modeling strategies developed in BERT [20]
and BEiT [21] have been adapted for set-type data in order to mask and predict
particles in a jet [22], and to mask and predict particle-type information
[23]. Supervised pre-training and fine-tuning strategies have also been
explored for high energy physics contexts [24]. A similar strategy has been
also pursued within astrophysics [25].

Accordingly, the contributions of this paper are as follows:

  * •

The development of a methodology for re-simulation through intervening in the
simulation chain. This has the benefit of generating plausible and physically-
motivated downstream outcomes that serve as augmentations in a contrastive
pre-training, enabling learning of the salient components of physics
simulations.

  * •

The creation of a large open dataset for common development in the
community222https://doi.org/10.5281/zenodo.10633815.

  * •

A systematic study of the gains of re-simulation-driven contrastive learning
against fully-supervised learning strategies.

While the focus of this work will be on jets produced at the LHC, related
strategies can be extended to other domains where simulation is present.

##  2 Results

Our re-simulation strategy aims to develop a physically motivated pre-training
of a model that we dub the RS3L backbone. This model represents data in a
multi-dimensional space rich in information that is common between an object
and its augmentations, leading to a localization of the object within the
space. Through this strategy, we can further reduce the dimensionality of the
overall space, leading to a low-dimensional representation that aims to
capture the key properties of the high-level objects. This strategy is
illustrated in Figure 1, which is described in the following.

Figure 1: Illustration of the RS3L setup, including downstream re-simulation,
sampling, graph computation, and the construction of positive and negative
pairs. These are then used in a contrastive loss function aiming to align
positive pairs and push negative pairs apart.

###  2.1 The RS3L backbone

Events are produced by a high-fidelity stochastic simulation of the hard
scattering of elementary particles (in our case, outgoing particles can be
Higgs bosons, quarks, or gluons). Different configurations for the parton
showering and hadronization are employed during re-simulation, each giving an
augmented realization, i.e., a different jet, of the same initial parton. In
the following, the Pythia8 [26] parton shower and hadronization simulation
with standardized settings derived from tuning to CMS experimental data [27]
is used to generate the _nominal_ scenario.

Domain completeness can be achieved by considering a set of augmentations
which represents the full knowledge embedded in our high-fidelity simulator.
Accordingly, we perform augmentations that span an array of plausible
experimental outcomes, namely:

  1. 1.

In-domain augmentation by keeping the simulator settings fixed but re-sampling
with a different numerical seed

  2. 2.

Out-of-domain augmentation by either (i) varying the primary simulator
settings within well-motivated bounds, in our case altering the probability
for final-state radiation (FSR) branchings in the parton showering step, or
(ii) using a different simulator, in our case a different parton shower model
(Herwig7 [28]).

The above set captures both 1) implicit uncertainty in the definition of a
quark or gluon parton shower, which results in a cascade of particles that can
vary in number of particles and distribution of energy across the particles,
and 2) the variations within parton shower simulations that are known to cover
the observed variations in the data. The augmentations themselves can be
interpreted physically as a construction of the parton ``wave function''
before its collapse into a measured state. The resulting space thus aims to
project away the imperfect modeling of the parton shower process and of the
statistical variability in our simulator, including quantum effects and ill-
defined hidden parameters.

Given a nominal jet, we randomly sample one of its augmentations from the
above set to build a _positive pair_. All other possible jet pairs in the
sample are considered _negative pairs_. We then process each jet with the
backbone, which we define using graph-based architectures with built-in graph
convolutions and message-passing. Graph neural networks (GNNs) have seen large
success over other ML algorithms like dense or convolutional neural networks
in the analysis of LHC data due to the point-cloud nature of the particles
that comprise a jet or an event (for an extensive review, see [29]). Finally,
each jet is embedded into an $N$-dimensional latent space. We use $N=8$
dimensions for the latent space as a balance between expressiveness and
computational complexity, although higher-dimensional spaces are possible333We
observed that networks trained with 128 and 512 output dimensions performed
similarly in the downstream tasks.. The 8D jet representations are then used
in the contrastive loss function based on SimCLR. The loss function aims at
aligning the 8D vectors of jets building a positive pair, and pushes apart
jets from negative pairs. A temperature parameter $\tau$ regulates the
relative importance of these two simultaneous training objectives and is set
to 0.1. All training details are provided in Appendix A.

###  2.2 Understanding the contrastive space

Of the eight dimensions, the most discriminating dimension between Higgs
bosons jets and quarks and gluons is shown on the left of Figure 2. As a
qualitative statement on the robustness of the space we derive through RS3L,
we also show in the same figure an equivalent plot for the variable $N_{2}$,
which is a theory-motivated function of the energy flow of particles in the
jet [30]. The latter represents a ratio between the compatibilities of the jet
to have one or, respectively, two prongs, which are regions of collimated
radiation. As a powerful jet tagging variable, $N_{2}$ is commonly used in
searches for processes involving hadronically decaying Higgs bosons [31, 32,
33] or (additional) weak gauge bosons [34, 35] to reduce the overwhelming
background where jets arise from quantum chromodynamics (QCD) interactions. In
both plots, the distributions of Higgs jets are shown in magenta and the
distributions of QCD jets (jets initiated from quarks and gluons) are shown in
other colors (see also Appendix A). The main (upper) panel shows the RS3L or,
respectively, $N_{2}$ feature for the nominal parton shower scenario, while
the bottom panels show the ratios of the varied distributions to the nominal
distribution.

Figure 2: (Left) One of eight features derived in the RS3L pre-training.
(Right) Jet substructure variable $N_{2}$.

In these plots, smaller ratios indicate smaller differences in the learned
representation for jets with different simulator configurations, and therefore
smaller systematic uncertainties arising from the parton showering model
chosen. Large trends are observed in these ratio when comparing different
simulator configurations in the case of $N_{2}$, while the trends are
mitigated in the RS3L space thanks to the employed self-supervision strategy
of aligning nominal jets with their augmented versions. The one-dimensional
distributions for all RS3L features are included in Appendix A.

Figure 3: Corner plots for the eight outputs of RS3L, split up into Higgs
boson (reds) and QCD jets (blues). Only small correlations are observed among
feature pairs.

Both the RS3L feature and $N_{2}$ provide discrimination power between Higgs
and QCD jets. This class separation in RS3L comes about despite the network
not receiving any information about the true class of the jet. Rather, it
emerges dynamically through self-supervision resulting from fulfilling the
contrastive training objective.

The corner plots in Figure 3 show the correlation between the eight features
derived through RS3L, split up into Higgs and QCD classes. The Pearson
correlation coefficient [36] is provided for each feature pair. The largest
(anti-)correlation for feature pairs describing Higgs boson jets is $-0.16$,
while for QCD jets it is 0.49. Overall, the eight features are largely
uncorrelated, indicating a near orthonormal feature set to characterize the
jets was found, in particular the ones coming from Higgs bosons.

Figure 4: 2D visualization of the 8D RS3L space, derived via t-SNE
dimensionality reduction. Top: A good class separation is seen between Higgs
jets and QCD (quark and gluon) jets. Bottom left: Jets shown by parton shower
model for Pythia8 and Herwig7 for a RS3L space trained with Herwig7
augmentations. Bottom left: The same for a RS3L space trained without Herwig7
augmentations. The congruence of the different parton shower models is visibly
worse in the right-hand scenario.

Finally, we visually probe the entire jet representation contained in the 8D
RS3L space by performing a reduction to two dimensions using the t-SNE
algorithm [37]. The plot on the top of Figure 4 shows this 2D space for
gluons, quarks, and Higgs jets. Again, strong separation between Higgs and QCD
jets as well as strong clustering for a given class is visible444Notably, the
model tries to push apart jets from two different Higgs bosons. However, as
all Higgs bosons share common characteristics, this objective is hard to meet,
leading to the observed clustering.. On the bottom left, we show the space
divided into jets showered with Pythia8 and, respectively, Herwig7. At the
population level, it is clear that different showering configurations occupy
similar regions in the RS3L space and, hence, our approach provides decent
robustness against domain shift. The bottom right of Figure 4 shows the t-SNE
2D space of a RS3L training that did not include Herwig7 augmentations.
Compared to the RS3L space obtained with all augmentations, this space has
visibly smaller overlap (congruence) between Pythia8-showered and
Herwig7-showered jets, speaking to larger uncertainties incurred by domain
shift.

###  2.3 Fine-tuning on top of the RS3L backbone

We now study the use of RS3L as a backbone for in-distribution and out-of-
distribution classification tasks, and we quantify performance and
uncertainties. For all fine-tunings, we consider a sample comprising the same
types of augmentations used for training the respective RS3L space. We then
compare the RS3L pre-training + fine-tuning performance to the performance of
a fully-supervised network with the same architecture. For a fair comparison,
we also train the fully-supervised network on a sample comprising the same
types of augmentations as were used to train the RS3L space. The fully-
supervised network, therefore, learns to marginalize over the augmentations,
not explicitly exploiting the information that the nominal and augmented jet
come from the same initial particle555We point out that the standard approach
for fully supervised algorithms at experiments such as ATLAS and CMS is to
train on samples comprising only jets from a nominal parton shower
configuration..

####  2.3.1 In-distribution classification and robustness

Figure 5: Tagging performance on the Higgs vs. QCD classification task. All numbers are obtained on a sample comprising only jets from the default parton shower scenario. Fine-tunings on top of a RS3L space that was trained with 5M jets are shown in light and dark blue, for graphs that have floating and fixed weights, respectively. The number in parentheses indicates the fine-tuning dataset size. Fully-supervised networks with various training sizes are shown in shades of grey. Three repeated trainings with different random initialization are used to compute an average and standard deviation for each ROC. These are shown as bands. Note that fixed-weight trainings do not have any appreciable uncertainty. Table 1: QCD rejection rates for various training configurations and Higgs efficiencies. The bolded numbers indicate the network with the best performance at a given Higgs efficiency. The uncertainties are calculated using the standard deviation of repeated trainings with different random initializations. Higgs efficiency | 0.3 | 0.5 | 0.7
---|---|---|---
$N_{2}$ | 29 | 16 | 8
Fine-tuned (3M, fixed) | $140\pm 1$ | $64\pm 0$ | $39\pm 0$
Fine-tuned (3M, floating) | $\mathbf{1325\pm 31}$ | $380\pm 4$ | $\mathbf{135\pm 1}$
Fully-supervised (1M) | $834\pm 11$ | $257\pm 6$ | $96\pm 2$
Fully-supervised (3M) | $1070\pm 29$ | $317\pm 3$ | $115\pm 1$
Fully-supervised (8M) | $1312\pm 15$ | $\mathbf{381\pm 7}$ | $134\pm 1$

The most straightforward application of RS3L is a binary classification fine-
tuning to separate the same jet classes that are used to train the RS3L space.
Thus, we consider the canonical HEP classification task of discriminating
between jets originating from Higgs bosons and jets originating from QCD, a
prerequisite for the precise understanding of Higgs boson couplings to bottom
quarks [38, 39, 40, 41, 42].

Two variants are considered for fine-tuning the RS3L space: (i) The weights of
the backbone model are kept fixed, and a task-specific head consisting of a
one-layer multilayer perceptron (MLP) is added, whose parameters are trained
on the task at hand. (ii) The weights of the backbone model are used as
initial parameters but are allowed to be adjusted (``floated'') together with
the ones of the task head.

A comparison of RS3L spaces fine-tuned for a Higgs vs. QCD classification task
is presented in Figure 5, which shows the receiver operating characteristic
(ROC) curve calculated from _nominal jets only_. The QCD rejection rate (1/
false positive rate) is shown in Table 1 for various Higgs tagging
efficiencies, again computed from only nominal jets. Compared to a fully-
supervised network trained on 8M jets, a fine-tuning using 3M jets on top of a
5M RS3L space yields a similar performance. However, if we instead match the
yields of the training dataset of the fully-supervised approach and the fine-
tuning (i.e., both training on 3M labeled samples), a large gain in
performance is observed for the RS3L + fine-tuning strategy. This indicates
that a common, large pre-training allows for efficient fine-tuning on
comparably much smaller dataset sizes than a fully supervised setup would use.

Furthermore, it is important to note that all training configurations that use
floating networks are more performant than networks trained with fixed weights
on top of the RS3L backbone. This is to be expected because in the fixed
weight case the model is severely restricted as the only trainable parameters
belong to the MLP head. Lastly, all networks outperform $N_{2}$ by a wide
margin.

In addition to background rejection, it is necessary to examine the classifier
under other metrics. In particular, we consider how the network response
varies under different parton shower configurations used in the simulator
(which we call the network's ``robustness''). The robustness is quantified by
calculating the Wasserstein distance [43, 44, 45] between the one-dimensional
distributions of the classifier output of nominal and, respectively, augmented
jets. The distances are presented in Table 2. Each column shows the distance
between the distribution for augmented jets (indicated by column title) and
nominal jets. A lower Wasserstein distance indicates a network that is less
sensitive to the showering configurations. As expected, the distance between
nominal jets and jets re-showered with just a different numerical seed is
smaller than the distance between nominal jets and any other augmentation
(``FSR up'', ``FSR down'', or ``Herwig''). Especially the distance between
nominal jets and Herwig7-showered jets is significant.

Comparing the different training strategies, the most robust networks are
trained using the RS3L method. With a fixed-weight approach, the Wasserstein
distance is significantly smaller than for all other networks. This is a
direct benefit of the RS3L pre-training strategy of minimizing the distance
between positive pairs. Including Herwig7 augmentations in RS3L or,
respectively, full supervision greatly reduces the distances compared to
setups without Herwig7 augmentations. Interestingly, we observe that already
including seed and/or FSR augmentations in RS3L helps fine-tuned networks to
be more robust against Herwig7-induced domain shift compared to fully
supervised strategies. Whether this indicates that a gain in robustness is
potentially attainable simply by training the backbone with re-seeded jets
merits further investigation that we leave to future studies.

The least robust network is a fully-supervised network trained only on jets
with re-seeded configurations (i.e., no systematic variations are seen during
training). The deterioration is particularly significant in Herwig7. More
robust networks are derived by including all variations in the fine-tuning
step. As expected, fully-supervised networks are more robust when the training
dataset is increased.

Table 2: Wasserstein distances between distributions of tagger output for
nominal and augmented jets. Each column indicates the distances between the
nominal and the respective augmentation. The number of training events is
included for each training setup. Fine-tunings are performed on a 5M RS3L pre-
training with either fixed or floating backbone weights. The inclusion of
specific augmentations (e.g. ``seed'' or ``FSR'') in the training setup
indicates that the pre-training and fine-tuning or supervision is performed on
that set of augmentations. Uncertainties are derived from three independent
trainings.

Training setup | Seed | FSR up | FSR down | Herwig
---|---|---|---|---
$N_{2}$ |  1.51 $\times 10^{-4}$ |  2.49 $\times 10^{-3}$ |  2.98 $\times 10^{-3}$ |  1.22 $\times 10^{-2}$
Fine-tuned (3M, fixed, all) |  (8.63 $\pm$ 0.23) $\times 10^{-4}$ |  ($\mathbf{4.85\pm 0.11)\times 10^{-4}}$ |  ($\mathbf{1.02\pm 0.02)\times 10^{-3}}$ | $\mathbf{(1.08\pm 0.01)\times 10^{-3}}$
Fine-tuned (3M, floating, all) |  (8.23 $\pm$ 0.24) $\times 10^{-4}$ |  (9.09 $\pm$ 0.50) $\times 10^{-4}$ |  (1.32 $\pm$ 0.01) $\times 10^{-3}$ |  (1.40 $\pm$ 0.04) $\times 10^{-2}$
Fine-tuned (3M, floating, seed+FSR) | $(\mathbf{7.87\pm 0.32)\times 10^{-4}}$ | $(9.80\pm 0.27)\times 10^{-4}$ | $(1.22\pm 0.02)\times 10^{-3}$ | $(3.58\pm 0.04)\times 10^{-2}$
Fine-tuned (3M, floating, seed) | $(8.51\pm 0.19)\times 10^{-4}$ | $(9.50\pm 0.18)\times 10^{-4}$ | $(1.36\pm 0.03)\times 10^{-3}$ | $(3.35\pm 0.09)\times 10^{-2}$
Fully-supervised (1M, all) |  (9.55 $\pm$ 0.89) $\times 10^{-4}$ |  (1.33 $\pm$ 0.04) $\times 10^{-3}$ |  (1.45 $\pm$ 0.05) $\times 10^{-3}$ |  (2.08 $\pm$ 0.05) $\times 10^{-2}$
Fully-supervised (3M, all) |  (8.87 $\pm$ 0.20) $\times 10^{-4}$ |  (1.14 $\pm$ 0.06) $\times 10^{-3}$ |  (1.48 $\pm$ 0.09) $\times 10^{-3}$ |  (2.18 $\pm$ 0.15) $\times 10^{-2}$
Fully-supervised (8M, all) | $(8.20\pm 0.51)\times 10^{-4}$ | $(9.91\pm 0.33)\times 10^{-4}$ | $(1.29\pm 0.03)\times 10^{-3}$ | $(1.32\pm 0.06)\times 10^{-2}$
Fully-supervised (8M, seed+FSR) | $(8.68\pm 0.56)\times 10^{-4}$ | $(9.95\pm 0.06)\times 10^{-4}$ | $(1.18\pm 0.02)\times 10^{-3}$ | $(3.87\pm 0.04)\times 10^{-2}$
Fully-supervised (8M, seed) | $(8.44\pm 0.15)\times 10^{-4}$ | $(1.15\pm 0.01)\times 10^{-3}$ | $(1.32\pm 0.03)\times 10^{-3}$ | $(4.47\pm 0.32)\times 10^{-2}$

####  2.3.2 Out-of-distribution classification task

We now shift to out- task to address the degree to which a general
representation of showering jets was achieved through the RS3L pre-training.
To investigate this, we fine-tune RS3L for a classification task that falls
outside of the distributions used for pre-training, namely the task of
discriminating between QCD jets and jets from ``hadronic'' W boson decays.
Contrary to the Higgs boson decays, jets from W boson decays do not feature
$B$ hadrons666The authors note that the absolute tagging performance on the W
vs. QCD and Higgs vs. QCD datasets cannot be compared directly because of the
composition of the sample, see A. The experimental signature therefore falls
out of the distribution of the Higgs jets used to train the RS3L backbone.

We follow the same procedure as for the in-distribution classification studies
by considering the same pre-trained RS3L backbone but now fine-tuned for W vs.
QCD classification. We compare with networks fully-supervised from scratch for
W vs. QCD classification. The results of this study are shown in Table 3.
Networks fine-tuned show significant improvement (as much as 10%) in
background rejection over fully-supervised networks when trained on the same
number of examples.

In terms of robustness, the Wasserstein distances between nominal and
augmented jets are shown in terms of QCD background rejection in Table 4 and
as a ROC in Figure 6. The training setups with RS3L are found to be
significantly more robust.

Table 3: QCD rejection rates for various training configurations and W efficiencies. The bolded numbers indicate the network with the best performance at a given W efficiency. W efficiency | 0.3 | 0.5 | 0.7
---|---|---|---
Training setup | 1/(QCD efficiency)
Fine-tuned (1M, floating, all) | 1589 $\pm$ 88 | 438 $\pm$ 9 | 134 $\pm$ 2
Fine-tuned (3M, floating, all) | $\mathbf{1928\pm 31}$ | $\mathbf{504\pm 3}$ | $\mathbf{147\pm 1}$
Fully-supervised (1M, all) | 1288 $\pm$ 102 | 357 $\pm$ 16 | 114 $\pm$ 2
Fully-supervised (3M, all) | 1763 $\pm$ 18 | 459 $\pm$ 1 | 137 $\pm$ 2
Figure 6: Tagging performance on the W vs. QCD classification task. RS3L
spaces that are fine-tuned for the task are shown in red. Fully-supervised
networks with various dataset training sizes are shown in shades of grey.
Table 4: Wasserstein distances between distributions of tagger output for
nominal and augmented jets in the out-of-domain case. Each column indicates
the distances between the nominal and the respective augmentation. The
distance is calculated over a sample comprising equal amounts of W and QCD
jets. The number of training events is included for each training setup.
Finally, uncertainties are derived through measuring the Wasserstein distance
for repeated trainings using the same configuration with random
initialization.

Training setup | Seed | FSR up | FSR down | Herwig
---|---|---|---|---
Fine-tuned W vs QCD (1M, floating, all) | $\mathbf{(2.48\pm 0.95)\times 10^{-4}}$ | $(5.18\pm 0.03)\times 10^{-3}$ | $(6.53\pm 0.01)\times 10^{-3}$ | $(4.07\pm 0.05)\times 10^{-2}$
Fine-tuned W vs QCD (3M, floating, all) | $(3.26\pm 0.24)\times 10^{-4}$ | $\mathbf{(4.79\pm 0.09)\times 10^{-3}}$ | $\mathbf{(6.08\pm 0.01)\times 10^{-3}}$ | $\mathbf{(3.58\pm 0.06)\times 10^{-2}}$
Fully-supervised W vs QCD (1M, all) | $(3.09\pm 0.39)\times 10^{-4}$ | $(5.02\pm 0.19)\times 10^{-3}$ | $(6.28\pm 0.32)\times 10^{-3}$ | $(4.51\pm 0.39)\times 10^{-2}$
Fully-supervised W vs QCD (3M, all) | $(4.08\pm 0.24)\times 10^{-4}$ | $(5.22\pm 0.01)\times 10^{-3}$ | $(6.70\pm 0.13)\times 10^{-3}$ | $(4.33\pm 0.19)\times 10^{-2}$

##  3 Outlook

RS3L is a novel strategy that combines the concept of re-simulation with a
contrastive loss function to drive self-supervised representation learning.
The framework provides a natural way for creating a powerful foundation model
which can be used for various downstream tasks such as classification and
uncertainty mitigation. By encapsulating systematic uncertainties as well as
the stochastic variability of the simulation into data augmentations, a latent
space with improved robustness compared to other learning strategies can be
obtained.

RS3L has been successfully applied to the canonical task of jet tagging, which
is the ability to identify the type of elementary particle at the origin of
the evolution of a particle shower in detectors at the Large Hadron Collider.
Here, the algorithm revealed identical performance in the limit of large-
enough training statistics to state-of-the-art deep learning-based jet taggers
that separate jets from Higgs bosons and from QCD partons, improved
performance at smaller dataset sizes, and improved robustness against
detrimental effects arising from systematic uncertainties. Additionally, it
shows excellent transferrability to out-of-distribution tasks, thus having
large potential to increase the efficiency of deep learning trainings in high
energy physics when employed as a common pre-training.

Enforcing domain completeness through a strategy to embed the known modeling
uncertainties within the self-supervised space directly maps this work to the
quality of the simulation used at training. Improved simulators can lead to
improved embeddings, which, in turn, lead to a better understanding of the
underlying physics. As a result, R3SL can adapt to the next generation of
simulation modeling that stands to be substantially better than the current
simulation. Through self-supervised approaches, we can continually improve the
quality of the physics that is extracted, provided we continue to improve the
physics model in the samples used for the self-supervision.

In future work, alternative self-supervised learning approaches to the
employed SimCLR strategy, such as VICReg [46], SimSiam [47], and BarlowTwins
[48] should be explored to study the practical impact of precise components of
their loss functions. Additionally, the dataset size for the pre-training
should be varied to systematically study the regimes in which the performances
of SSL, fine-tuning, and fully-supervised strategies may saturate.

## Data Availability

The RS3L dataset is published at https://doi.org/10.5281/zenodo.10633815.

##  4 Acknowledgments

The authors would like to thank M. Pierini and S. Mishra-Sharma for fruitful
discussions. The authors thank the CERN storage team for the creation and
maintenance of a dedicated storage space for this project. The trainings for
this study have been performed on the MIT Satori and subMIT clusters.

BM acknowledges the support of the Alexander von Humboldt foundation and of
Schmidt Futures. MK is supported by the US Department of Energy (DOE) under
grant DE-AC02-76SF00515. PH and JK are supported by the Institute for
Artificial Intelligence and Fundamental Interactions (IAIFI) under the NSF
grant #PHY-2019786 and the Accelerated AI Algorithms for Data Driven Discovery
Grang (A3D3) under NSF grant #PHY-2117997.

##  Appendix A Methods

###  A.1 Data augmentations and training input

Hard-scattering events for $\text{p}\text{p}\to\text{j}\text{Z}$ and
$\text{p}\text{p}\to\text{H}\text{Z}$, with
$\text{H}\to\text{b}\bar{\text{b}}$ and $\text{Z}\to\nu\bar{\nu}$, from which
we sample the jets for training the networks, are generated at a center-of-
mass energy of $\sqrt{s}=13\,\text{TeV}$ using MadGraph5_aMC@NLO v3.4.0 [49]
with the NNPDF30_nlo_nf_5_pdfas parton distribution function [50] at leading-
order accuracy in QCD. The j and, respectively, the Higgs boson are generated
within a $p_{\mathrm{T}}$ range of 400-600 GeV and within
$\absolutevalue{\eta}<0.1$. The hard-scattering events are then passed through
a parton shower simulation and are reconstructed using the Delphes v3.4.3pre1
detector simulation [51] with a CMS detector-like geometry. For gluons and the
Higgs, the first splitting or, respectively, decay is generated at matrix-
element level with a minimum $\Delta
R=\sqrt{(\Delta\eta)^{2}+(\Delta\phi)^{2}}>0.001$. All further splittings are
generated by the subsequent parton shower.

In the nominal configuration, Pythia8 v8.244 with the CP5 tune [27] is used
for parton showering, hadronization, and modelling of the underlying event. No
effects from pile-up are simulated. Four augmentations of each nominal jet are
created via re-simulation: (i) the same settings as in the nominal
configuration are used, but the partons are passed through Pythia8 using a
different random initial seed; (ii) the value for the parton shower
renormalization scale used in the determination of final-state radiation is
multiplied by a factor $\sqrt{2}$; (iii) the same scale is multiplied by a
factor $1/\sqrt{2}$; (iv) Herwig7 with the default tune is used for parton
showering and hadronization. In all parton shower configurations, initial-
state radiation has been disallowed to focus solely radiation effects of the
final state.

Jets are finally clustered from the Delphes E-Flow candidates using the
anti-$k_{t}$ algorithm [52] with a radius parameter $R=0.8$. The 100
highest-$p_{\mathrm{T}}$ E-Flow candidates per jet are used for training the
networks. For jets to be considered in the training and evaluation, one QCD
parton (gluon or quark) or a Higgs boson along with its two decay b quarks
have to be matched to the jets, i.e., fulfill $\Delta R<0.8$, with
$\Delta\eta$ and $\Delta\phi$ being the difference in pseudorapidity $\eta$
and, respectively, in azimuthal angle $\phi$ between the parton and the jet
axis. Jets with a transverse momentum of $p_{\mathrm{T}}>450$ GeV, a mass of
at least $10\,\mathrm{GeV}$, and $|\eta|<0.1$ are kept. These requirements are
also imposed for all augmentations of a given nominal jet.

The RS3L space is trained with 5M events using with a 50%/50% admixture of QCD
jets and Higgs jets. During the training of the network, the gradients get
updated after each minibatch consisting of 200 jet samples. For each nominal
jet, a systematic variation of the jet is included in the same minibatch,
selected at random among the four possible variations at random to complete
the (anchor, augmentation) pairs. The composition of nominal jets in the
training set is always exactly 50%, while the remaining 50% is allocated to
the augmentations used in training.

###  A.2 Network architecture

We use a graph neural network that employs a stack of DynamicEdgeConv [53]
network layers for enriching the particle features with information from
neighboring particles. The $k$ nearest neighbors are determined dynamically,
i.e, in the latent feature space obtained after the initial embedding or,
respectively, after each graph convolution. A BERT-like transformer
architecture [54] adapted from [55] has been used to cross-check the results
and yields comparable performance. We point out that an optimization of the
architecture used for the demonstration of the contrastive approach goes
beyond a proof-of-concept and, thus, beyond the scope of this article. The
architecture used in the RS3L pre-training is:

$\displaystyle h_{\mathrm{embed}}=\mathrm{MLP}_{\mathrm{embed}}(X)$
$\displaystyle
h_{\mathrm{DGC1}}=\textsc{DynamicEdgeConv}(h_{\mathrm{embed}}|k=24)$
$\displaystyle
h_{\mathrm{DGC2}}=\textsc{DynamicEdgeConv}(h_{\mathrm{DGC1}}|k=24)$
$\displaystyle
h_{\mathrm{DGC3}}=\textsc{DynamicEdgeConv}(h_{\mathrm{DGC2}}|k=24)$
$\displaystyle
h_{\mathrm{enc}}=\mathrm{MLP}_{\mathrm{enc}}(h_{\mathrm{DGC3}})$
$\displaystyle\bm{z}=\textsc{GlobalSumPool}(h_{\mathrm{enc}})\ .$ (1)

The input feature set $X$ is given by 14 features per particle, listed in Tab.
5. These input features are not normalized over a minibatch, but their
magnitudes are comparable among the features and $\mathcal{O}(1)$, thereby
regularizing the network and leading to a similar effect as a pre-
transformation that enforces a mean of 0 and standard deviation of 1.

For embedding the 14 particle features into a higher-dimensional latent space,
two fully-connected layers with dimensions (128/128) are used in
$\textsc{MLP}_{\text{embed}}$. After a series of three convolutions with
$k=24$, $\mathrm{MLP}_{\mathrm{enc}}$ consists of four fully-connected layers
with dimensions (64/32/32/8). Outputs of all neurons in both MLPs are
activated with an exponential linear unit [56]. The final building block,
GlobalSumPool, aggregates the eight features of each particle in the jet from
the last layer of $\mathrm{MLP}_{\mathrm{enc}}$ to obtain the final eight
features $\bm{z}$ characterizing the jet in the RS3L space.

Table 5: Description of the particle input features.  Particle feature | Description
---|---
$\log{p_{\text{T}}}$ | logarithm of the transverse momentum
$\log{(p_{\text{T}}/p_{\text{T}}^{\text{Jet}})}$ | logarithm of $p_{\text{T}}$ normalized w.r.t. the jet
$\log{E}$ | logarithm of the energy
$\log{(E/E^{\text{Jet}})}$ | logarithm of $E$ normalized w.r.t. the jet
$\Delta\eta$ | difference in pseudorapidity w.r.t. the jet
$\Delta\phi$ | difference in $\phi$ w.r.t. the jet
$\Delta R$ | distance in $\eta$-$\phi$ plane from the jet axis
$\tanh d_{0}$ | hyperbolic tangent of the transverse impact parameter
$\tanh d_{z}$ | hyperbolic tangent of the longitudinal impact parameter
isPhoton | 1 if particle is a photon, else 0
isMuon | 1 if particle is a muon, else 0
isElectron | 1 if particle is an electron, else 0
isCH | 1 if particle is a charged hadron, else 0
isNH | 1 if particle is a neutral hadron, else 0

The loss function employed is based on the SimCLR framework introduced in [14]
and has first been used for jet physics in [19]. It is given by

$\mathcal{L}=-\log{\frac{e^{s(\bm{z}_{i},\bm{z}_{i}^{\prime})/\tau}}{\sum_{i\neq
j\,\in\,\mathrm{minibatch}}\left[e^{s(\bm{z}_{i},\bm{z}_{j})/\tau}+e^{s(\bm{z}_{i},\bm{z}_{j}^{\prime})/\tau}\right]}}$
(2)

with $s(\bm{z}_{i},\bm{z}_{j})$ being the cosine similarity between negative
jet pairs $i$ and $j$ or, in the case of positive pairs, the nominal jet $i$
and the augmented jet $i^{\prime}$:

$s(\bm{z}_{i},\bm{z}_{j})=\frac{\bm{z}_{i}\cdot\bm{z}_{j}}{|\bm{z}_{i}||\bm{z}_{j}|}=\cos\theta_{ij}.$
(3)

The network will try to push apart dissimilar jets (negative pairs) on the
$N-1$-dimensional hypersphere with unit radius given by the $N$ input
features, while trying to align similar jets (positive pairs) and characterize
them by similar features. The competition between these two effects is
regulated by a temperature parameter $\tau$. We studied this parameter based
on the performance obtained when using the RS3L space to distinguish Higgs
from QCD jets. Specifically, we trained RS3L spaces with $\tau$ ranging from
$0.05$ to $0.5$ and fine-tuned these networks on the Higgs vs. QCD
classification task with a fixed backbone. A fixed backbone offers the best
insight into the performance of the RS3L pre-training step because it projects
jets into a one-dimensional classification space using only the information
derived during the self-supervised step. We observed that the Higgs vs. QCD
binary classification performance was optimal with $\tau=0.1$. The full set of
observed features is shown included in Figure 7.

Figure 7: The eight contrastive features derived during the self-supervision
step.

Figure 8 illustrates the convergence of the RS3L space as a function of
training time. In the top panel, the cosine similarity is shown for nominal
and augmented jets (averaged over each class). This is analogous to the
positive pair component of the loss function, the pairs that the network will
attempt to make parallel in the space. The network makes augmentations nearly
parallel to their augmentations, with average angles between $0.74$ and
$0.89$. We observe that the network further aligns Higgs vectors over time,
while slightly mis-aligning QCD vectors over time. In the bottom panel, the
cosine similarity between the average vectors of the two summary classes
(Higgs and QCD) is shown. At the beginning of the training, Higgs and QCD
occupy roughly the same region of the space, however as the network converges
the populations are pushed to lie on opposing ends of the space, despite not
knowing about their different class nature. In all plots, the bands are
indicative of averaging over 3 repeated RS3L trainings with random
initialization.

The performance of the RS3L space fine-tuned on the in-domain Higgs vs. QCD
classification task is shown in ROC curves Figure 9. A separate ROC curve is
shown for Higgs separation from each individual QCD subprocesses. Note that
the training sample consists of an admixture with similar weightings of all
QCD subprocesses. The left (right) shows a graph with floating (fixed).
Superior performance is observed for a graph with floating weights.

Figure 8: Metrics pertaining to the convergence of the RS3L training as a
function of epoch. Top panel: the average of cosine similarity between the
positive pairs (anchor jet and augmented jet). Bottom panel: the cosine
similarity between the average Higgs vector and average QCD vector. The
variation, indicated by the error bands, is computed over three RS3L
trainings.

Figure 9: ROC curves showing the tagging performance of RS3L fine-tunings on
the in-domain Higgs vs. QCD classification task. Left: graph fine-tuned with
floating weights. Right: graph fine-tuned with fixed weights.

###  A.3 Fine-tuning and fully-supervised trainings

The MLP placed on top of the RS3L architecture is given by one additional
fully-connected layer with 8 input dimensions and a single sigmoid activation
function at the output. The backbone network architecture used in the fine-
tuning is the exact same as the fully-supervised. The only difference between
the approaches is that in the fine-tuning method the base graph trained with
RS3L is loaded as a hot-start for the classification and the MLP is added on
top of the backbone network. We consider two configurations of fine-tuning the
RS3L space: a ``fixed'' graph (where only the MLP parameters are floating) and
a ``floating'' graph (where all the MLP and graph parameters are floating).
The admixture of QCD/H/W, as well as augmentations composition, are always
consistent between the RS3L fine-tuning and full supervision. The RS3L space
and classification steps are trained on independent datasets.

In each case, the network is trained repeatedly (at least three times) to gain
a measure of whether a result is statistically significant. The numbers in
ROCs and tables indicate the averages of these repeated trainings. The
difference between subsequent training runs is observed to be quite small.

## References

  * Bommasani  _et al._ [2022] R. Bommasani _et al._ , ``On the opportunities and risks of foundation models,'' (2022), arXiv:2108.07258 [cs.LG] .
  * Tong  _et al._ [2022] Z. Tong, Y. Song, J. Wang, and L. Wang, ``Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training,'' (2022), arXiv:2203.12602 [cs.CV] .
  * Brown  _et al._ [2020] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, ``Language models are few-shot learners,'' (2020), arXiv:2005.14165 [cs.CL] .
  * wen Yang  _et al._ [2021] S. wen Yang, P.-H. Chi, Y.-S. Chuang, C.-I. J. Lai, K. Lakhotia, Y. Y. Lin, A. T. Liu, J. Shi, X. Chang, G.-T. Lin, T.-H. Huang, W.-C. Tseng, K. tik Lee, D.-R. Liu, Z. Huang, S. Dong, S.-W. Li, S. Watanabe, A. Mohamed, and H. yi Lee, ``Superb: Speech processing universal performance benchmark,'' (2021), arXiv:2105.01051 [cs.CL] .
  * Rives  _et al._ [2021] A. Rives, J. Meier, T. Sercu, S. Goyal, Z. Lin, J. Liu, D. Guo, M. Ott, C. L. Zitnick, J. Ma, and R. Fergus, Proceedings of the National Academy of Sciences 118, e2016239118 (2021).
  * Ross  _et al._ [2022] J. Ross, B. Belgodere, and V. Chenthamarakshan, Nature Machine Intellegence 4, 1256–1264 (2022).
  * Pan [2023] J. Pan, Nature Communication Science 3 (2023).
  * Irwin  _et al._ [2022] R. Irwin, S. Dimitriadis, J. He, and E. J. Bjerrum, Machine Learning: Science and Technology 3, 015022 (2022).
  * Ahmad  _et al._ [2022] W. Ahmad, E. Simon, S. Chithrananda, G. Grand, and B. Ramsundar, ``Chemberta-2: Towards chemical foundation models,'' (2022), arXiv:2209.01712 [cs.LG] .
  * Subramanian  _et al._ [2023] S. Subramanian, P. Harrington, K. Keutzer, W. Bhimji, D. Morozov, M. Mahoney, and A. Gholami, ``Towards foundation models for scientific machine learning: Characterizing scaling and transfer behavior,'' (2023), arXiv:2306.00258 [cs.LG] .
  * McCabe  _et al._ [2023] M. McCabe, B. R.-S. Blancard, L. H. Parker, R. Ohana, M. Cranmer, A. Bietti, M. Eickenberg, S. Golkar, G. Krawezik, F. Lanusse, M. Pettee, T. Tesileanu, K. Cho, and S. Ho, ``Multiple physics pretraining for physical surrogate models,'' (2023), arXiv:2310.02994 [cs.LG] .
  * Lanusse  _et al._ [2023] F. Lanusse, L. Parker, S. Golkar, M. Cranmer, A. Bietti, M. Eickenberg, G. Krawezik, M. McCabe, R. Ohana, M. Pettee, B. R.-S. Blancard, T. Tesileanu, K. Cho, and S. Ho, ``Astroclip: Cross-modal pre-training for astronomical foundation models,'' (2023), arXiv:2310.03024 [astro-ph.IM] .
  * Walmsley  _et al._ [2022] M. Walmsley, I. V. Slijepcevic, M. Bowles, and A. M. M. Scaife, ``Towards galaxy foundation models with hybrid contrastive learning,'' (2022), arXiv:2206.11927 [cs.CV] .
  * Chen  _et al._ [2020] T. Chen, S. Kornblith, M. Norouzi, and G. E. Hinton, CoRR abs/2002.05709 (2020), 2002.05709 .
  * Campbell  _et al._ [2022] J. M. Campbell _et al._ , in  _Snowmass 2021_ (2022) arXiv:2203.11110 [hep-ph] .
  * Aad  _et al._ [2008] G. Aad _et al._ (ATLAS), JINST 3, S08003 (2008).
  * Chatrchyan  _et al._ [2008] S. Chatrchyan _et al._ (CMS), JINST 3, S08004 (2008).
  * Evans and Bryant [2008] L. Evans and P. Bryant, JINST 3, S08001 (2008).
  * Dillon  _et al._ [2022] B. M. Dillon, G. Kasieczka, H. Olischlager, T. Plehn, P. Sorrenson, and L. Vogel, SciPost Phys. 12, 188 (2022).
  * Devlin  _et al._ [2019] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, ``Bert: Pre-training of deep bidirectional transformers for language understanding,'' (2019), arXiv:1810.04805 [cs.CL] .
  * Bao  _et al._ [2022] H. Bao, L. Dong, S. Piao, and F. Wei, ``Beit: Bert pre-training of image transformers,'' (2022), arXiv:2106.08254 [cs.CV] .
  * Heinrich  _et al._ [2024] L. Heinrich, T. Golling, M. Kagan, S. Klein, M. Leigh, M. Osadchy, and J. A. Raine, ``Masked particle modeling on sets: Towards self-supervised high energy physics foundation models,'' (2024), arXiv:2401.13537 [hep-ph] .
  * Kishimoto  _et al._ [2023] T. Kishimoto, M. Morinaga, M. Saito, and J. Tanaka, ``Pre-training strategy using real particle collision data for event classification in collider physics,'' (2023), arXiv:2312.06909 [hep-ex] .
  * Vigl  _et al._ [2024] M. Vigl, N. Hartman, and L. Heinrich, ``Finetuning foundation models for joint analysis optimization,'' (2024), arXiv:2401.13536 [hep-ph] .
  * Akhmetzhanova  _et al._ [2024] A. Akhmetzhanova, S. Mishra-Sharma, and C. Dvorkin, Mon. Not. Roy. Astron. Soc. 527, 7459 (2024), arXiv:2308.09751 [astro-ph.CO] .
  * Sjöstrand  _et al._ [2015] T. Sjöstrand, S. Ask, J. R. Christiansen, R. Corke, N. Desai, P. Ilten, S. Mrenna, S. Prestel, C. O. Rasmussen, and P. Z. Skands, Comp. Phys. Comm. 191, 159 (2015).
  * Sirunyan  _et al._ [2020a] A. M. Sirunyan _et al._ (CMS), Eur. Phys. J. C 80, 4 (2020a), arXiv:1903.12179 [hep-ex] .
  * Bellm  _et al._ [2017] J. Bellm, S. Gieseke, D. Grellscheid, P. Kirchgaeßer, F. Loshaj, G. Nail, A. Papaefstathiou, S. Plätzer, R. Podskubka, M. Rauch, C. Reuschle, P. Richardson, P. Schichtel, M. H. Seymour, A. Siódmok, and S. Webster, ``Herwig 7.1 release note,''  (2017).
  * Shlomi  _et al._ [2020] J. Shlomi, P. Battaglia, and J.-R. Vlimant, Machine Learning: Science and Technology 2, 021001 (2020).
  * Moult  _et al._ [2016] I. Moult, L. Necib, and J. Thaler, Journal of High Energy Physics 2016 (2016), 10.1007/jhep12(2016)153.
  * Sirunyan  _et al._ [2019a] A. M. Sirunyan _et al._ (CMS), Eur. Phys. J. C 79, 280 (2019a), arXiv:1811.06562 [hep-ex] .
  * Sirunyan  _et al._ [2020b] A. M. Sirunyan _et al._ (CMS), JHEP 03, 025 (2020b), arXiv:1908.01713 [hep-ex] .
  * Sirunyan  _et al._ [2020c] A. M. Sirunyan _et al._ (CMS), JHEP 12, 085 (2020c), arXiv:2006.13251 [hep-ex] .
  * Sirunyan  _et al._ [2018a] A. M. Sirunyan _et al._ (CMS), JHEP 01, 097 (2018a), arXiv:1710.00159 [hep-ex] .
  * Sirunyan  _et al._ [2019b] A. M. Sirunyan _et al._ (CMS), Phys. Rev. D 100, 112007 (2019b), arXiv:1909.04114 [hep-ex] .
  * Kirch [2008] W. Kirch, ed., ``Pearson's correlation coefficient,'' in  _Encyclopedia of Public Health_ (Springer Netherlands, Dordrecht, 2008) pp. 1090–1091.
  * van der Maaten and Hinton [2008] L. van der Maaten and G. Hinton, Journal of Machine Learning Research 9, 2579 (2008).
  * Aaboud  _et al._ [2018] M. Aaboud _et al._ (ATLAS), Phys. Lett. B 786, 59 (2018), arXiv:1808.08238 [hep-ex] .
  * Sirunyan  _et al._ [2018b] A. M. Sirunyan _et al._ (CMS), Phys. Rev. Lett. 121, 121801 (2018b), arXiv:1808.08242 [hep-ex] .
  * Aad  _et al._ [2021a] G. Aad _et al._ (ATLAS), Phys. Lett. B 816, 136204 (2021a), arXiv:2008.02508 [hep-ex] .
  * Aad  _et al._ [2021b] G. Aad _et al._ (ATLAS), Eur. Phys. J. C 81, 178 (2021b), arXiv:2007.02873 [hep-ex] .
  * Tumasyan  _et al._ [2023] A. Tumasyan _et al._ (CMS), (2023), arXiv:2312.07562 [hep-ex] .
  * Vaser$\check{\mathrm{s}}$te$\breve{\i}$n [1969] L. N. Vaser$\check{\mathrm{s}}$te$\breve{\i}$n, Probl. Peredachi Inf. 5 (1969).
  * Kantorovich [1960] L. V. Kantorovich, Management Science 6, 366 (1960).
  * Ramdas  _et al._ [2015] A. Ramdas, N. Garcia, and M. Cuturi, (2015), arXiv:1509.02237 [math.ST] .
  * Bardes  _et al._ [2021] A. Bardes, J. Ponce, and Y. LeCun, CoRR abs/2105.04906 (2021), 2105.04906 .
  * Chen and He [2020] X. Chen and K. He, CoRR abs/2011.10566 (2020), 2011.10566 .
  * Zbontar  _et al._ [2021] J. Zbontar, L. Jing, I. Misra, Y. LeCun, and S. Deny, CoRR abs/2103.03230 (2021), 2103.03230 .
  * Alwall  _et al._ [2014] J. Alwall, R. Frederix, S. Frixione, V. Hirschi, F. Maltoni, O. Mattelaer, H.-S. Shao, T. Stelzer, P. Torrielli, and M. Zaro, JHEP 2014 (2014), 10.1007/jhep07(2014)079.
  * Ball  _et al._ [2015] R. D. Ball, , V. Bertone, S. Carrazza, C. S. Deans, L. D. Debbio, S. Forte, A. Guffanti, N. P. Hartland, J. I. Latorre, J. Rojo, and M. Ubiali, JHEP 2015 (2015), 10.1007/jhep04(2015)040.
  * de Favereau  _et al._ [2014] J. de Favereau, C. Delaere, P. Demin, A. Giammanco, V. Lemaitre, A. Mertens, and M. Selvaggi, JHEP 2014, 57 (2014).
  * Cacciari  _et al._ [2008] M. Cacciari, G. P. Salam, and G. Soyez, JHEP 04, 063 (2008).
  * Wang  _et al._ [2018] Y. Wang, Y. Sun, Z. Liu, S. E. Sarma, M. M. Bronstein, and J. M. Solomon, CoRR abs/1801.07829 (2018), 1801.07829 .
  * Devlin  _et al._ [2018] J. Devlin, M. Chang, K. Lee, and K. Toutanova, CoRR abs/1810.04805 (2018), 1810.04805 .
  * Maier  _et al._ [2022] B. Maier, S. M. Narayanan, G. de Castro, M. Goncharov, C. Paus, and M. Schott, Mach. Learn. Sci. Tech. 3, 025012 (2022).
  * Clevert  _et al._ [2016] D.-A. Clevert, T. Unterthiner, and S. Hochreiter, in  _ICLR (Poster)_ (2016).