# Experimental Observation of Space-Charge Field Screening of a Relativistic
Particle Bunch in Plasma

L. Verra<EMAIL_ADDRESS>INFN Laboratori Nazionali di Frascati, Via
Enrico Fermi 54, 00044 Frascati, Italy  M. Galletti  University of Rome Tor
Vergata, Via Della Ricerca Scientifica 1, 00133 Rome, Italy  INFN Tor Vergata,
Via Della Ricerca Scientifica 1, 00133 Rome, Italy  NAST Center, Via Della
Ricerca Scientifica 1, 00133 Rome, Italy  R. Pompili  INFN Laboratori
Nazionali di Frascati, Via Enrico Fermi 54, 00044 Frascati, Italy  A. Biagioni
INFN Laboratori Nazionali di Frascati, Via Enrico Fermi 54, 00044 Frascati,
Italy  M. Carillo  University of Rome Sapienza, Piazzale Aldo Moro 5, 00185
Rome, Italy  A. Cianchi  University of Rome Tor Vergata, Via Della Ricerca
Scientifica 1, 00133 Rome, Italy  INFN Tor Vergata, Via Della Ricerca
Scientifica 1, 00133 Rome, Italy  NAST Center, Via Della Ricerca Scientifica
1, 00133 Rome, Italy  L. Crincoli  INFN Laboratori Nazionali di Frascati, Via
Enrico Fermi 54, 00044 Frascati, Italy  A. Curcio  INFN Laboratori Nazionali
di Frascati, Via Enrico Fermi 54, 00044 Frascati, Italy  F. Demurtas
University of Rome Tor Vergata, Via Della Ricerca Scientifica 1, 00133 Rome,
Italy  G. Di Pirro  INFN Laboratori Nazionali di Frascati, Via Enrico Fermi
54, 00044 Frascati, Italy  V. Lollo  INFN Laboratori Nazionali di Frascati,
Via Enrico Fermi 54, 00044 Frascati, Italy  G. Parise  University of Rome Tor
Vergata, Via Della Ricerca Scientifica 1, 00133 Rome, Italy  D. Pellegrini
INFN Laboratori Nazionali di Frascati, Via Enrico Fermi 54, 00044 Frascati,
Italy  S. Romeo  INFN Laboratori Nazionali di Frascati, Via Enrico Fermi 54,
00044 Frascati, Italy  G.J. Silvi  University of Rome Sapienza, Piazzale Aldo
Moro 5, 00185 Rome, Italy  F. Villa  INFN Laboratori Nazionali di Frascati,
Via Enrico Fermi 54, 00044 Frascati, Italy  M. Ferrario  INFN Laboratori
Nazionali di Frascati, Via Enrico Fermi 54, 00044 Frascati, Italy

###### Abstract

The space-charge field of a relativistic charged bunch propagating in plasma
is screened due to the presence of mobile charge carriers. We experimentally
investigate such screening by measuring the effect of dielectric wakefields
driven by the bunch in a uncoated dielectric capillary where the plasma is
confined. We show that the plasma screens the space-charge field and therefore
suppresses the dielectric wakefields when the distance between the bunch and
the dielectric surface is much larger than the plasma skin depth. Before full
screening is reached, the effects of dielectric and plasma wakefields are
present simultaneously.

Introduction.— One of the fundamental properties of plasma is the ability to
screen electromagnetic fields, due to the presence of mobile charge carriers
(plasma electrons and ions) [1]. The propagation of electromagnetic waves in
plasma has been extensively studied [2], especially in the context of plasma
generation [3], heating and confinement [4], requiring complex and dedicated
experimental setups [5]. In the case of a relativistic charged particle bunch
travelling through a plasma with density $n_{pe}$, the plasma electrons
rapidly move to compensate for the almost purely transverse space-charge field
of the bunch. This effect is the cause of phenomena such as plasma wakefields
[6, 7].

Because of the screening operated by the plasma, the space-charge field of the
bunch is exponentially damped with the radial distance $r$ as $E_{r}\propto
e^{-r/\delta}$ [8], where the characteristics length
$\delta=c\sqrt{m_{e}\varepsilon_{0}/n_{pe}e^{2}}$ is the plasma skin depth
($c$ the speed of light, $\varepsilon_{0}$ the vacuum permittivity, $m_{e}$
and $e$ the electron rest mass and charge, respectively). Hence, the plasma
effectively screens the space-charge field of the bunch when $r\gg\delta$.
Despite the physics of the process being well known, no direct experimental
evidence of plasma screening has been provided so far for the fields generated
by a relativistic charged particle bunch.

In this Letter, we present a non-invasive measurement of plasma screening of
the space-charge field of a relativistic electron bunch. We show that
screening causes the suppression of dielectric wakefields in an uncoated
dielectric capillary, where the plasma is generated, when the distance between
the bunch and the capillary surface is much larger than the plasma skin depth.
Moreover we observe, for the first time to our knowledge, the simultaneous
effect of dielectric and plasma wakefields, when full plasma screening is not
achieved.

A relativistic, charged bunch drives wakefields when its space-charge field
interacts with a slow-wave structure or medium with a large index of
refraction. Consequently, longitudinal and transverse dielectric wakefields
(also known as Cherenkov wakefields) are generated [9, 10, 11, 12]. Dielectric
transverse wakefields are axis-symmetric with respect to the center of the
dielectric structure. Therefore, they induce a dipolar effect when the bunch
trajectory is not aligned to the longitudinal axis of the structure [13, 14,
15, 16, 17, 18]. The transverse wake potential is calculated as the
convolution of the transverse wake function $w$ (depending on the geometry,
the dielectric material and the bunch transverse offset with respect to the
center of the structure) with the longitudinal charge density distribution of
the bunch $n_{b}(t)$.

Thus, when a bunch travels with transverse offset $(x,y)$, parallel to its
axis, it drives transverse dipolar wakefields along the bunch described by
[16]: $W_{\perp}(t)=w(x,y)\int_{0}^{t}n_{b}(t)dt$, where the bunch front is at
$t=0$. The amplitude of the wakefields follows the same trend as the running
integral of the bunch charge: particles in the back of the bunch are deflected
more strongly than those in the front. The polarity of $W_{\perp}$ is such
that the trailing particles are pulled further towards the dielectric material
[19].

Dielectric capillaries are common tools for generating plasmas in plasma
wakefield accelerators (PWFA), as they enable gas injection in a high-vacuum
environment [20]. The plasma is generated by ionizing the gas injected in the
capillary with a high-voltage discharge applied between electrodes at each end
[21], or with an ionization field such as a high-intensity laser pulse [22] or
a relativistic charged particle bunch [23]. Capillaries are also employed
because they allow shaping the transverse and longitudinal profiles of the
plasma density [24], and locally injecting different gas species for various
injection schemes [25].

In a PWFA based on dielectric capillaries, both plasma and dielectric
wakefields can be present at the same time, as the space-charge electric field
of the driver bunch may interact with both plasma and dielectric material [26,
27]. In particular, the possible misalignment of the trajectory with respect
to the capillary longitudinal axis (e.g., due to jitters or to the finite
accuracy of the alignment tools) may induce a deflection to the bunch
trajectory. This could have detrimental effects on the propagation of the
bunches and the acceleration quality, by inducing transverse phenomena such as
beam-breakup and hosing instability [28, 29]. However, we show in the present
work that high-density plasma suppresses the dielectric wakefields driven in
the capillary, by screening the space-charge field of the bunch.

Experimental Setup.— We performed the measurements at the SPARC$\\_$LAB
facility [30] (see Fig. 1). An electron ($e^{-}$) bunch is generated by
illuminating a copper cathode within a radiofrequency (RF) gun with an
ultraviolet laser pulse. The bunch is accelerated in an RF linac to $E\sim
80\,$MeV and focused with transverse root mean square (rms) radius $\sim
0.4\,$mm at the entrance of a dielectric capillary. A set of electromagnetic
quadrupoles focuses the bunch on a scintillating screen $d=5.3\,$m downstream
of the capillary exit for imaging. A transverse deflection structure (TDS)
introduces a head-to-tail vertical correlation to the bunch [31], allowing to
obtain time-resolved images of the bunch at the screen position. The
normalized emittance of the bunch is $\sim 2.4\,$µm (measured with quadrupole
scan technique) and rms duration $\sim 1.8\,$ps. We use a magnetic dipole and
an additional scintillating screen to measure the beam energy.

Figure 1: Schematic of the SPARC_LAB experimental setup (not to scale). The
$e^{-}$ bunch propagates from right to left.

The dielectric capillary has radius $R_{c}=1\,$mm and length $L=10\,$cm. The
plasma is generated by a discharge pulse ($\sim 455$ A peak current) flowing
through the capillary after the introduction of hydrogen gas (${\sim
10\,}$mbar) through a high-speed solenoid valve. The capillary is installed in
a vacuum chamber connected to the linac with a windowless, three-stage
differential pumping system. The latter ensures to maintain $10^{-8}\,$mbar in
the linac, while flowing the gas, and to preserve the quality of the electron
bunch. By varying the delay between the bunch arrival time and the peak of the
discharge pulse we vary $n_{pe}$, due to recombination of the plasma. We
measure the average $n_{pe}$ with the Stark broadening technique [32] between
0.65 and 2.65 µs after the peak of the discharge pulse and we extrapolate the
values of $n_{pe}$ (with $\sim 10\%$ uncertainty) at the delays used in the
experiment (between 5.60 and 10.10 µs) by fitting the measured values with an
exponential decay function [33]. The plasma electron density is essentially
constant along the capillary (see Supplemental Material [34]), but non-
uniformities are present at the two extremese because the gas flow expands in
the vacuum chamber. However, as no capillary material is present, there is
also no dielectric wakefield effect along the entrance and exit ramps. Thus,
in the following we neglect these non-uniformities.

Current discharge is particularly suited for generating plasma in our
experimental conditions because the moderate energy and current of the $e^{-}$
bunch are not sufficient to field-ionize hydrogen gas. Moreover, it is
preferable over ionization with a laser pulse because it generates an almost
uniform transverse plasma density profile and it requires simpler
synchronization and spatial alignemnt methods.

In the following, we maintain the beam trajectory aligned through the center
of the quadrupole magnets, and we record images on the screen with the TDS
turned on, and the dipole turned off. To study the effect of dielectric
wakefields and plasma screening, we shift the capillary with a stepper motor
in the horizontal direction (perpendicular to the streaking direction of the
TDS, see Supplemental Material [34]) and we measure the induced deflection
along the bunch on time-resolved images while varying the plasma electron
density and, therefore, the plasma skin depth.

Experimental Results.— Figure 2 shows single-event, time-resolved images of a
285 pC $e^{-}$ bunch after travelling along the longitudinal axis of the
dielectric capillary (a), and with a horizontal offset $X=0.375\,$mm (b). The
bunch propagates from right to left (bunch front at $t=0$). The red points
show the centroid position along the bunch, obtained by fitting the transverse
projection of each longitudinal slice with a Gaussian function. In the aligned
case, the centroid position is essentially constant along the bunch and the
variations are much smaller than the bunch transverse size. The average
centroid position obtained from 100 consecutive images (red points, left-hand
side vertical axis in Fig. 2(c), the error bars represent the standard
deviation) is not correlated with the running sum of the bunch charge (black
line, right-hand side vertical axis, the shaded area is the standard
deviation) because $w(0,0)=0$.

In the misaligned case (Fig. 2(b)), on the contrary, the centroid is clearly
deflected, with increasing displacement along the bunch. Figure 2(d) shows
that the centroid position and the running sum follow the same trend along the
bunch. This confirms the expectation that the amplitude of the transverse
wakefields (and therefore of the transverse deflection) at any time $t$ along
the bunch depends on the amount of charge ahead of it, in agreement with the
formulation of $W_{\perp}(t)$.

The amplitude of the transverse wakefields $W_{\perp}$ reaches a maximum at
the back of the bunch. We calculate the average wakefield potential
experienced by the particles in the last slice of the bunch ($t\sim 6.7\,$ps)
as $W_{\perp}=x(t)E/edL\sim 0.4\,$MV/m. We also note that, in both cases, the
transverse size slightly increases along the bunch. This is due to the fact
that the dielectric wakefields also have a quadrupolar (defocusing) component,
growing in amplitude along the bunch.

When introducing plasma with $n_{pe}=1.0\cdot 10^{16}\,$cm-3 while
$X=0.375\,$mm (Fig. 3(a), same misalignment as Fig. 2(b)), the dielectric
wakefields are suppressed because the plasma effectively screens the space-
charge field of the bunch, since the plasma skin depth $\delta=0.053\,$mm is
much shorter than the distance between bunch and the dielectric surface
$R_{c}-X=0.625\,$mm. Thus, the average position of the centroid along the
bunch (green points in (c)) is in agreement with the aligned case with no
plasma in the capillary (blue points, same dataset as Fig. 2(c)).

Figure 2:  (a,b): single-event, time-resolved images of the 285 pC $e^{-}$
bunch for $X=0$ (aligned case) and $X=0.375\,$mm, respectively. The bunch
propagates from right to left. Red points indicate the centroid position of
each longitudinal slice. (c,d): average centroid position along the bunch
obtained from 100 consecutive events (red points, left-hand side vertical
axis; error bars are the standard deviation), corresponding to (a) and (b),
respectively. Each black line shows the average running sum of the charge
along the bunch (right-hand side vertical axis, the shaded area represents the
standard deviation).

Slices at $t>3\,$ps have a wider transverse distribution because the bunch
also drives plasma wakefields, that are axis-symmetric with respect to the
bunch propagation axis. Additionally, the bunch self-focuses due to the effect
of the magnetic field generated by the bunch current itself, that is not
compensated by the space-charge force, which is neutralized by the plasma [35,
36, 37]. As the bunch is progressively focused, it leaves the plasma with
increasing divergence along the bunch, and possibly larger emittance, causing
the transverse size to increase at the downstream screen. Because of this
effect, we calculate the position of the centroid only for $t<4.6\,$ps.

On the contrary, when $n_{pe}=0.038\cdot 10^{16}\,$cm-3 (Fig. 3(b)), plasma
screening is not effective. In this case, $\delta=0.237\,$mm is not short
enough to operate full screening, and the space-charge field of the bunch
still reaches the dielectric surface, generating dielectric wakefields that
affect the trajectory of the trailing part of the bunch. Thus, the horizontal
position of the centroid along the bunch (red points in Fig. 3(c)) varies, in
agreement with the case with no plasma (black points, same dataset as Fig.
2(d)). We note that, in this case, dielectric wakefields (causing the centroid
deflection) and plasma wakefields (causing the defocusing effect at the
screen) are present at the same time.

Figure 3:  (a,b): single-event, time-resolved images of the $e^{-}$ bunch with
$X=0.375\,$mm (as in Fig. 2(b)), and plasma with $n_{pe}=1.0$ and $0.038\cdot
10^{16}\,$cm-3, respectively. (c): average centroid position along the bunch
for $X=0$ and $n_{pe}=0$ (blue points), $X=0.375\,$mm and $n_{pe}=0$ (black
points), $n_{pe}=0.038\cdot 10^{16}\,$cm-3 (red points) and
$n_{pe}=10^{16}\,$cm-3 (green points).

Increasing $n_{pe}$ decreases the distance from the propagation axis beyond
which the space-charge field of the bunch is screened. Thus, for a small
misalignment of the bunch trajectory with respect to the center of the
dielectric capillary (i.e., a large distance from the surface), a lower
$n_{pe}$ is sufficient to suppress the dielectric wakefields than for a large
misalignment (i.e., a short distance from the surface).

Figure 4 shows the position of the centroid at $t=3.6$ ps (a), 4.1 ps (b) and
4.6 ps (c) behind the front of the bunch as a function of $n_{pe}$, for three
misalignment distances (see Legend). The displacement, which is due to the
effect of the dielectric wakefields, decreases when increasing $n_{pe}$
because the amplitude of the space-charge field reaching the dielectric
surface is progressively more screened by plasma. The trend is in good
agreement with the typical exponential decay expected from plasma screening.
The solid lines show the result of the fit for each dataset, where the
distance from the bunch to the dielectric surface is considered as a free
parameter.

For $X=0.375\,$mm (black points), full screening (i.e., centroid position in
agreement with the aligned case with no plasma) occurs for $n_{pe}>0.7\cdot
10^{16}\,$cm-3 (black dashed vertical line), corresponding to
$\delta<0.063\,$mm, that is $\sim 10\,$times shorter than the distance between
bunch and capillary surface $R_{c}-X=0.625\,$mm. For smaller misalignments,
screening occurs at lower $n_{pe}$: for $R_{c}-X=0.750\,$mm, $n_{pe}>0.5\cdot
10^{16}\,$cm-3 ($\delta<0.075\,$mm, red points and dashed vertical line); and
for $R_{c}-X=0.875\,$mm, $n_{pe}>0.24\cdot 10^{16}\,$cm-3 ($\delta<0.109\,$mm,
blue points and dashed vertical line). As expected from the screening process,
$\delta$ must be much shorter than the distance between the bunch and
dielectric surface to obtain full screening. We also note that the ratio
$(R_{c}-X)/\delta$ at full screening increases when the misalignment
increases. This is likely due to the finite transverse size of the bunch, as
some particles are closer to the dielectric material than those at the bunch
center.

As the bunch travels in plasma, it can drive plasma wakefields that also have
a transverse (periodically focusing or defocusing) component [6, 7], generally
growing along the bunch (see Fig. 3(a,b)). However, the fact that the
transition to full screening occurs at the same $n_{pe}$ for all delays along
the bunch, as highlighted by the vertical lines in Fig. 4, indicates that the
screening of the space-charge field is independent of the amplitude of the
plasma wakefields. The displacement is larger for larger misalignments, at any
$t$ along the bunch where the plasma screening is not fully effective because
$w(x,y)$ increases.

Figure 4:  Average centroid position $x$ as a function of $n_{pe}$ for three
misalignment distances at longitudinal slices $t=3.6$ ps (a), $4.1$ ps (b),
$4.6\,$ps (c) behind the front of the bunch. Dashed vertical lines indicate
the values of $n_{pe}$ after which $x$ is in agreement with the aligned case
with no plasma, for each misalignment distance.

The experimental evidence we present in this work will have significant
implications for designing and operating plasma-based accelerators. In
particular, the direct observation of space-charge screening in plasma could
be employed for managing misalignments and optimizing plasma conditions to
minimize undesired wakefield effect.

Considering future PWFA, the operational $n_{pe}$ and the transverse size of
the capillary must be chosen so that the dielectric wakefields induced by
misalignments (e.g., caused by jitters and finite accuracy of the alignment
tools) are effectively screened by the plasma. In the case of
EuPRAXIA@SPARC_LAB [38], the diameter of the dielectric capillary will be 2 mm
and the operational density $n_{pe}=10^{16}\,$cm-3. The maximum misalignment
allowed to avoid the dielectric wakefields effect would therefore be even
larger than $X=0.375\,$mm, as shown in this study, which is much larger than
the state-of-the-art capabilities to align and control the trajectory of
relativistic particle bunches (e.g., better than 20 µm in our experiment). The
measurement we present in this work could also be extended to a plasma
electron density diagnostics, by measuring the maximum misalignment for which
full screening of the space-charge field occurs.

Summary.— We investigated, with a non-invasive measurement, one of the
fundamental characteristics of plasma, that is the screening of
electromagnetic fields. We showed, using a plasma wakefield accelerator based
on a dielectric capillary, that plasma screens the space-charge field of a
relativistic $e^{-}$ bunch by measuring the suppression of dielectric
wakefields. The effect of transverse dielectric wakefields increases along the
bunch, following the same trend as the running sum of the bunch charge, in
agreement with the theory. We observed a progressively increasing screening
effect when increasing the plasma electron density, and therefore decreasing
the plasma skin depth. We measured full screening to occur along the entire
bunch when the distance from the bunch propagation axis to the dielectric
surface is more than 10 times longer than the plasma skin depth. When full
screening is not achieved, dielectric and plasma wakefields are present
simultaneously. We also discussed the implications for future plasma-based
accelerators based on dielectric capillaries.

###### Acknowledgements.

This work has been partially supported by the European Commission in the
Seventh Framework Programme, grant agreement 312453-EuCARD-2, the European
Union Horizon 2020 research and innovation program, grant agreement no. 653782
(EuPRAXIA) and the INFN with the GRANT73/PLADIP and SL COMB2FEL grants. We
thank G. Grilli and T. De Nardis for the development of the HV discharge
pulser, F. Anelli for the technical support and M. Zottola for the
experimental chamber installation.

## References

  * Chen [2016] F. F. Chen,  _Introduction to Plasma Physics and Controlled Fusion_ (Springer International Publishing, 2016).
  * Otsuka  _et al._ [2015] F. Otsuka, T. Hada, S. Shinohara, and T. Tanikawa, Penetration of a radio frequency electromagnetic field into a magnetized plasma: one-dimensional pic simulation studies, Earth, Planets and Space 67, 85 (2015).
  * Shinohara and Shamrai [2002] S. Shinohara and K. P. Shamrai, Effect of electrostatic waves on a rf field penetration into highly collisional helicon plasmas, Thin Solid Films 407, 215 (2002), proceedings of the 14th Symposium on Plasma Science for Materials.
  * Shoji [1980] T. Shoji, Description of radio-frequency plugging and heating in terms of plasma impedance, Journal of the Physical Society of Japan 49, 327 (1980), https://doi.org/10.1143/JPSJ.49.327 .
  * Whelan and Stenzel [1981] D. A. Whelan and R. L. Stenzel, Electromagnetic-wave excitation in a large laboratory beam-plasma system, Phys. Rev. Lett. 47, 95 (1981).
  * Chen  _et al._ [1985] P. Chen, J. M. Dawson, R. W. Huff, and T. Katsouleas, Acceleration of electrons by the interaction of a bunched electron beam with a plasma, Physical Review Letters 54, 693 (1985).
  * Keinigs and Jones [1987] R. Keinigs and M. E. Jones, Two‐dimensional dynamics of the plasma wakefield accelerator, The Physics of Fluids 30, 252 (1987).
  * Jackson [1962] J. D. Jackson,  _Classical Electrodynamics_ (John Wiley & Sons, 1962).
  * Bane  _et al._ [1985] K. L. Bane, P. B. Wilson, and T. Weiland, Wake Fields and Wake Field Acceleration, AIP Conf. Proc. 127, 875 (1985).
  * Gai  _et al._ [1988] W. Gai, P. Schoessow, B. Cole, R. Konecny, J. Norem, J. Rosenzweig, and J. Simpson, Experimental demonstration of wake-field effects in dielectric structures, Physical Review Letters 61, 2756 (1988).
  * Keinigs  _et al._ [1989] R. Keinigs, W. Peter, and M. E. Jones, A comparison of the dielectric and plasma wakefield accelerators, Physics of Fluids B: Plasma Physics 1, 1872 (1989), https://pubs.aip.org/aip/pfb/article-pdf/1/9/1872/12643661/1872_1_online.pdf .
  * Ng [1990] K.-Y. Ng, Wake fields in a dielectric-lined waveguide, Phys. Rev. D 42, 1819 (1990).
  * Park and Hirshfield [2000] S. Y. Park and J. L. Hirshfield, Theory of wakefields in a dielectric-lined waveguide, Physical Review E 62, 1266 (2000).
  * Baturin and Kanareykin [2014] S. S. Baturin and A. D. Kanareykin, Cherenkov radiation from short relativistic bunches: General approach, Physical Review Letters 113, 214801 (2014).
  * Bettoni  _et al._ [2016] S. Bettoni, P. Craievich, A. A. Lutman, and M. Pedrozzi, Temporal profile measurements of relativistic electron bunch based on wakefield generation, Physical Review Accelerators and Beams 19, 021304 (2016).
  * Craievich and Lutman [2017] P. Craievich and A. A. Lutman, Effects of the quadrupole wakefields in a passive streaker, Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 865, 55 (2017).
  * O’Shea  _et al._ [2020] B. D. O’Shea, G. Andonian, S. S. Baturin, C. I. Clarke, P. D. Hoang, M. J. Hogan, B. Naranjo, O. B. Williams, V. Yakimenko, and J. B. Rosenzweig, Suppression of deflecting forces in planar-symmetric dielectric wakefield accelerating structures with elliptical bunches, Physical Review Letters 124, 104801 (2020).
  * Saveliev  _et al._ [2022] Y. Saveliev, T. J. Overton, T. H. Pacey, N. Joshi, S. Mathisen, B. D. Muratori, N. Thompson, M. P. King, and G. Xia, Experimental study of transverse effects in planar dielectric wakefield accelerating structures with elliptical beams, Physical Review Accelerators and Beams 25, 081302 (2022).
  * Chao [1993] A. W. Chao,  _Physics of collective beam instabilities in high-energy accelerators_ (John Wiley & Sons, Inc., 1993).
  * Leemans  _et al._ [2006] W. P. Leemans, B. Nagler, A. J. Gonsalves, C. Tóth, K. Nakamura, C. G. R. Geddes, E. Esarey, C. B. Schroeder, and S. M. Hooker, Gev electron beams from a centimetre-scale accelerator, Nature Physics 2, 696 (2006).
  * Spence and Hooker [2000] D. J. Spence and S. M. Hooker, Investigation of a hydrogen plasma waveguide, Phys. Rev. E 63, 015401 (2000).
  * Keldysh [1965] L. V. Keldysh, Ionization in the Field of a Strong Electromagnetic Wave, J. Exp. Theor. Phys. 20, 1307 (1965).
  * O’Connell  _et al._ [2006] C. L. O’Connell, C. D. Barnes, F.-J. Decker, M. J. Hogan, R. Iverson, P. Krejcik, R. Siemann, D. R. Walz, C. E. Clayton, C. Huang, D. K. Johnson, C. Joshi, W. Lu, K. A. Marsh, W. Mori, M. Zhou, S. Deng, T. Katsouleas, P. Muggli, and E. Oz, Plasma production via field ionization, Phys. Rev. ST Accel. Beams 9, 101301 (2006).
  * Leemans  _et al._ [2014] W. P. Leemans, A. J. Gonsalves, H.-S. Mao, K. Nakamura, C. Benedetti, C. B. Schroeder, C. Tóth, J. Daniels, D. E. Mittelberger, S. S. Bulanov, J.-L. Vay, C. G. R. Geddes, and E. Esarey, Multi-gev electron beams from capillary-discharge-guided subpetawatt laser pulses in the self-trapping regime, Physical Review Letters 113, 245002 (2014).
  * Chen  _et al._ [2012] M. Chen, E. Esarey, C. B. Schroeder, C. G. R. Geddes, and W. P. Leemans, Theory of ionization-induced trapping in laser-plasma accelerators, Physics of Plasmas 19, 033101 (2012).
  * C. Li [2013] C. T. C. Li, H. Zha, Calculation of Wakefield in Plasma-Filled Dielectric Capillaries Generated by a Relativistic Electron Beam, in  _Proc. IPAC’13_ , International Particle Accelerator Conference (JACoW Publishing, Geneva, Switzerland, 2013).
  * Biagioni  _et al._ [2018] A. Biagioni, M. Anania, M. Bellaveglia, E. Brentegani, G. Castorina, E. Chiadroni, A. Cianchi, D. Di Giovenale, G. Di Pirro, H. Fares, L. Ficcadenti, F. Filippi, M. Ferrario, A. Mostacci, R. Pompili, J. Scifo, B. Spataro, C. Vaccarezza, F. Villa, and A. Zigler, Wake fields effects in dielectric capillary, Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 909, 247 (2018), 3rd European Advanced Accelerator Concepts workshop (EAAC2017).
  * Whittum  _et al._ [1991] D. H. Whittum, W. M. Sharp, S. S. Yu, M. Lampe, and G. Joyce, Electron-hose instability in the ion-focused regime, Physical Review Letters 67, 991 (1991).
  * Nechaeva  _et al._ [2024] T. Nechaeva, L. Verra, J. Pucek, L. Ranc, M. Bergamaschi, G. Zevi Della Porta, P. Muggli, R. Agnello, C. C. Ahdida, C. Amoedo, Y. Andrebe, O. Apsimon, R. Apsimon, J. M. Arnesano, V. Bencini, P. Blanchard, P. N. Burrows, B. Buttenschön, A. Caldwell, M. Chung, D. A. Cooke, C. Davut, G. Demeter, A. C. Dexter, S. Doebert, J. Farmer, A. Fasoli, R. Fonseca, I. Furno, E. Granados, M. Granetzny, T. Graubner, O. Grulke, E. Gschwendtner, E. Guran, J. Henderson, M. A. Kedves, S.-Y. Kim, F. Kraus, M. Krupa, T. Lefevre, L. Liang, S. Liu, N. Lopes, K. Lotov, M. Martinez Calderon, S. Mazzoni, K. Moon, P. I. Morales Guzmán, M. Moreira, N. Okhotnikov, C. Pakuza, F. Pannell, A. Pardons, K. Pepitone, E. Poimenidou, A. Pukhov, S. Rey, R. Rossel, H. Saberi, O. Schmitz, E. Senes, F. Silva, L. Silva, B. Spear, C. Stollberg, A. Sublet, C. Swain, A. Topaloudis, N. Torrado, M. Turner, F. Velotti, V. Verzilov, J. Vieira, C. Welsch, M. Wendt, M. Wing, J. Wolfenden, B. Woolley, G. Xia, V. Yarygova, and M. Zepp (AWAKE Collaboration), Hosing of a long relativistic particle bunch in plasma, Phys. Rev. Lett. 132, 075001 (2024).
  * Ferrario  _et al._ [2013] M. Ferrario, D. Alesini, M. Anania, A. Bacci, M. Bellaveglia, O. Bogdanov, R. Boni, M. Castellano, E. Chiadroni, A. Cianchi, S. Dabagov, C. De Martinis, D. Di Giovenale, G. Di Pirro, U. Dosselli, A. Drago, A. Esposito, R. Faccini, A. Gallo, M. Gambaccini, C. Gatti, G. Gatti, A. Ghigo, D. Giulietti, A. Ligidov, P. Londrillo, S. Lupi, A. Mostacci, E. Pace, L. Palumbo, V. Petrillo, R. Pompili, A. Rossi, L. Serafini, B. Spataro, P. Tomassini, G. Turchetti, C. Vaccarezza, F. Villa, G. Dattoli, E. Di Palma, L. Giannessi, A. Petralia, C. Ronsivalle, I. Spassovsky, V. Surrenti, L. Gizzi, L. Labate, T. Levato, and J. Rau, SPARC_LAB present and future, Nuclear Instruments and Methods in Physics Research Section B: Beam Interactions with Materials and Atoms 309, 183 (2013).
  * Alesini  _et al._ [2006] D. Alesini, G. Di Pirro, L. Ficcadenti, A. Mostacci, L. Palumbo, J. Rosenzweig, and C. Vaccarezza, Rf deflector design and measurements for the longitudinal and transverse phase space characterization at sparc, Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 568, 488 (2006).
  * Qian  _et al._ [2010] M. Qian, C. Ren, D. Wang, J. Zhang, and G. Wei, Stark broadening measurement of the electron density in an atmospheric pressure argon plasma jet with double-power electrodes, Journal of Applied Physics 107, 063303 (2010).
  * Shpakov  _et al._ [2019] V. Shpakov, M. P. Anania, M. Bellaveglia, A. Biagioni, F. Bisesto, F. Cardelli, M. Cesarini, E. Chiadroni, A. Cianchi, G. Costa, M. Croia, A. Del Dotto, D. Di Giovenale, M. Diomede, M. Ferrario, F. Filippi, A. Giribono, V. Lollo, M. Marongiu, V. Martinelli, A. Mostacci, L. Piersanti, G. Di Pirro, R. Pompili, S. Romeo, J. Scifo, C. Vaccarezza, F. Villa, and A. Zigler, Longitudinal phase-space manipulation with beam-driven plasma wakefields, Physical Review Letters 122, 114801 (2019).
  * [34] See Supplemental Material at [URL will be inserted by publisher] for details on the experimental setup .
  * Chen [1987] P. Chen, A possible final focusing mechanism for linear colliders, Particle Accelerators 20, 171 (1987).
  * Barov and Rosenzweig [1994] N. Barov and J. B. Rosenzweig, Propagation of short electron pulses in underdense plasmas, Phys. Rev. E 49, 4407 (1994).
  * Verra  _et al._ [2023] L. Verra, E. Gschwendtner, and P. Muggli, Focusing of a long relativistic proton bunch in underdense plasma,  _2022 IEEE Advanced Accelerator Concepts Workshop (AAC)_ , arXiv preprint arXiv:2302.04051 10.48550/arxiv.2302.04051 (2023).
  * R. W. Assmann, R. W. et al. [2020] R. W. Assmann, R. W. et al., Eupraxia conceptual design report, The European Physical Journal Special Topics 229, 3675 (2020).


11institutetext: JPMorgan Chase & Co., 25 Bank St, London E14 5JP, UK

# A Baseline Generative Probabilistic Model for Weakly Supervised Learning
††thanks: Supported by organization JPMorgan Chase & Co.

Georgios Papadopoulos ✉  0009-0009-9732-3423 Fran Silavong  Sean Moran

###### Abstract

Finding relevant and high-quality datasets to train machine learning models is
a major bottleneck for practitioners. Furthermore, to address ambitious real-
world use-cases there is usually the requirement that the data come labelled
with high-quality annotations that can facilitate the training of a supervised
model. Manually labelling data with high-quality labels is generally a time-
consuming and challenging task and often this turns out to be the bottleneck
in a machine learning project. Weakly Supervised Learning (WSL) approaches
have been developed to alleviate the annotation burden by offering an
automatic way of assigning approximate labels (pseudo-labels) to unlabelled
data based on heuristics, distant supervision and knowledge bases. We apply
probabilistic generative latent variable models (PLVMs), trained on heuristic
labelling representations of the original dataset, as an accurate, fast and
cost-effective way to generate pseudo-labels. We show that the PLVMs achieve
state-of-the-art performance across four datasets. For example, they achieve
22% points higher F1 score than Snorkel in the _class-imbalanced_ Spouse
dataset. PLVMs are plug-and-playable and are a drop-in replacement to existing
WSL frameworks (e.g. Snorkel) or they can be used as baseline high-performance
models for more complicated algorithms, giving practitioners a compelling
accuracy boost.

###### Keywords:

Weakly Supervised Learning Generative Models Probabilistic Models.

##  1 Introduction

In recent years, weakly supervised learning (WSL) has emerged as an area of
increasing interest among machine learning practitioners and researchers. This
interest has been driven by the need to automate the process of applying deep
learning models to unlabelled real-world data, thus making manual annotations
unnecessary and expensive. For example, medical doctors may wish to use
machine learning (ML) models to improve the detection of intracranial
haemorrhage (ICH) on head computed tomography (CT) scans [24], but current
datasets are often large and unlabelled, making the application of ML the
models difficult.

Various research teams, including Snorkel and Flying Squid [20, 19, 21, 34],
have developed methods to address this labelling problem, with the overarching
goal of reducing the cost of labelling for large datasets by hand. These WSL
methods automate the otherwise tedious and costly manual labelling process by
sourcing prior information from Subject Matter Experts (SMEs), which is used
to create labelling functions $\lambda$ that are applied to the data. The
output of this approach is typically a binary sparse matrix (labelling matrix)
$\Lambda$.

Overall, the increasing interest in WSL reflects the potential of this
approach to enable the more efficient and effective use of machine learning
models on real-world data, even when labelled data is scarce or expensive to
obtain. By leveraging SME guidance and prior knowledge, WSL methods offer a
promising avenue for automating the labelling process, reducing costs, and
enabling more widespread adoption of ML models in a range of applications.

We present a straightforward algorithm to create dichotomous classes on
unlabelled datasets. Like [20], our method utilizes labelling functions
$\lambda$ derived from Subject Matter Expert (SME) domain knowledge to
programmatically annotate previously unlabelled data. The resulting
annotations are represented as a labelling matrix $\Lambda$. Our approach
relies on the assumption that the sparse input matrix $\Lambda$ contains
sufficient information for robust model creation. Specifically, we propose to
use a probabilistic generative latent variable model, Factor Analysis (FA), to
map dependencies among the elements of the labelling matrix and generate a
1-dimensional latent factor $z$. We dichotomize the latent variable $z$ using
the median and assign each group of observations to a binary class.

Our approach addresses the negative impact of class imbalance and label
abstentions on existing WSL methods. We provide empirical evidence for the
superior performance of the FA model compared to the state-of-the-art model,
Snorkel, across three publicly available datasets and one internal curated
dataset. We also compare the performance of FA with two more complex
generative probabilistic latent variable models: Gaussian process latent
variable models (GPLVM) with Sparse Variational Gaussian Processes (SVGP) and
Variational Inference - Factor Analysis (VI-FA).

We show that FA as a WSL model outperforms other methods in Table 3, where it
achieved accuracy of 95% for the source code classification task, 86% in the
YouTube Spam dataset, 86% in the Spouse dataset, and 65% in Goodreads dataset.

To summarise, the contribution of this paper is the following:

  * •

Impact of class imbalance: We study the impact of class imbalance and label
abstentions on existing WSL models [21]. This is not only an academic problem
but also a common occurrence in real-world data and applied cases. We
empirically illustrate this negative effect on three publicly available
datasets, YouTube Spam dataset [1], Spouse dataset [21], and Goodreads dataset
[29, 30].

  * •

Stronger performance: As a solution, we propose to leverage FA for a new WSL
method that outperforms current state-of-the-art models, including Snorkel
[21], as well as the benchmark probabilistic algorithms GPLVM-SVGP and VI-FA
in terms of _both_ performance and resilience to class imbalance.

  * •

Robustness and Causality: We demonstrate the robustness of the proposed FA
model under small datasets, class-imbalance and label abstentions. Also, it is
proven that FA models offer causality between the labelling functions and the
true labels [9].

  * •

Industrial Applicability: We applied our method on internal data (JPMorgan)
and evaluated our model in real-world cases (source code) by communicating
with SMEs (firm engineers).We show our method scales well in industrial
settings, is plug-and-play, and highly robust and accurate.

##  2 Related Work

WSL, as a research area, has become widely popular and has experienced a
wealth of publications; with many culminating to end-to-end production systems
[19, 21]. Some real-world examples, from a diversified domain, which WSL
methods have been applied, include healthcare [24, 8, 25, 10, 11], human
posturing and ergonomic studies [14, 35, 5], multimedia and sound [18, 15,
23], dataset querying [33], in business studies and behavioural analysis [16,
12, 27], and autonomous driving [31].

In our paper, we draw motivation from recent research on data programming and
matrix completion methods for WSL. Specifically, in [22] the authors use
conditionally independent and user defined labelling functions with a
probabilistic model optimised using the log-maximum likelihood and gradient
descent methods. The true class label for a data point is modeled as a latent
variable that generates the observed, noisy labels. After fitting the
parameters of this generative model on unlabeled data, a distribution over the
latent, true labels can then be inferred. [2] expand the previous research by
adding an L1 regulariser to the [22]’s formula. The team created a first end-
to-end programmatic pipeline by incorporating findings from the two previous
papers, named Snorkel [20, 19, 34]. They also replaced the sampling of the
posterior from a graphical model with a matrix completion approach [19].

The main shortcomings of the probabilistic approach that [22, 2, 19] are using
are that is mathematically quite complex (for example the works of [22, 2]).
Also, challenging to implement as a plug-and-play solution on industrial scale
projects. Finally, as we demonstrate in Sections 3 and 5, fails to perform
under class-imbalance and small datasets.

One way to address the class imbalance performance problem and simplify the
algorithms came from [28]. The authors presented a structure learning method
that relies on robust Principal Component Analysis (PCA) to estimate the
dependencies among the different weakly supervision sources. They show that
the dependency structure of generative models significantly affects the
quality of the generated labels. This, thematically, is quite similar to our
work. The main differentiating factor is that in [28] they use PCA as a method
to replace the lower rank and the sparse matrix from their previous work [20];
whereas, we propose to use FA as the _entire_ WSL model.

Our approach, compared to [28], allows users to plug-and-play any latent
probabilistic models, without further modification. Another major difference
is that our approach (FA) considers independent diagonal noise compared to
spherical noise of the PCA, therefore as a model is better suited to map
causality amongst the labelling functions ($\lambda$) and the ground truth
[9].

##  3 Model Formulation

In this paper we follow a two-step approach. Initially, we utilise heuristic
labelling function techniques based on [19] to create a sparse labelling
matrix $\Lambda$. In the second step, we map the relationships (Fig. 1) among
the labelling functions using FA. Our approach has the benefit that can be
expanded using any probabilistic generative latent variable model such as
GPLVM.

(a) YouTube Spam

(b) Spouse

(c) Goodreads

Figure 1: The covariance heatmaps of the three labelling matrices ($\Lambda$).
Each heatmap is a dataset. We observe that the labelling functions $\lambda$
are independent. There is no strong relationship amongst them. But FA (and
PLVM in general) are able to capture the causal relationship and approximate a
true latent factor.

###  3.1 Labelling Functions

Labelling functions $\lambda$, as described in [22], are user-defined
programmatic items that each incorporate the SME’s knowledge in a binary form
$\lambda\in\\{0,1\\}$ or $\\{-1\\}$ if the function $\lambda$ is considered as
abstain; where no relevant information is present. The goal of this process is
to build a large set of approximate labels [2]. Effectively, rather than hand-
labelling training data, SMEs can write labelling functions instead. To this
end, as a data programming approach, labelling functions offer model
flexibility by programmatically expressing various weakly supervision sources,
including patterns, heuristics, external knowledge bases, and more.

###  3.2 Factor Analysis

Our objective is to enhance the generative methodology underlying Snorkel [21]
with a more straightforward approaching. In a related work, [28] applied a
robust PCA to improve Snorkel results. However, our method differs from theirs
in that they used PCA to initialize parameters for their probabilistic model,
whereas we replace the scalable matrix completion algorithm with a
probabilistic generative latent variable model (PLVM).

The use of generative latent models to extract underlying components from data
has been extensively researched and documented [4, 6, 3, 17]. A standard
generative latent model is Factor Analysis (FA), which is closely related to
Probabilistic PCA [26]. By leveraging the FA model, we aim to simplify the
underlying probabilistic complexity of Snorkel and improve its performance on
unlabelled datasets.

###  3.3 Weakly Supervision with Factor Analysis

Given an observed dataset $X\in\mathbb{R}^{n\times d}$, we utilise the
labelling function $\lambda$ capabilities from Snorkel [21] to create a binary
labelling matrix $\Lambda$. Labelling functions are user-defined programmatic
items that scan the underlying data $X$ and result in the labelling matrix
$\Lambda(X)$. The labelling matrix is a $n{\times}m$ sparse matrix with $m$
the number of labelling functions $\lambda$, $n$ the number of data-points in
the data $X$ and values $\Lambda\in\\{0,1,-1\\}$. The Factor Analysis (FA)
model captures the dominant dependencies amongst the data and subsequently
finds a lower dimensional probabilistic description. FA can also be used for
classification as they can model class conditional densities [6]. In brief,
the idea behind FA is that we have an observed dataset $\Lambda$ that is a
linear representation of a latent factor $z$

$\Lambda=Wz+c+\epsilon$ (1)

$W$ is the loading matrix with dimensions $m\times k$ with $k$ the dimensions
of the latent factor $z$ with $k{\ll}d$, $c$ is a centred constant bias term
and $\epsilon=\mathcal{N}(\epsilon|0,\Psi)$ is the Gaussian distributed noise
of the model with $\Psi$ the $m{\times}m$ diagonal matrix. As a reminder $m$
is the number of columns/labelling functions in our observed data $\Lambda$
and $n$ the number of observed data-points. Probabilistically, this formula
takes the form of the likelihood $p(\Lambda|z)$:

$\displaystyle p(\Lambda|z)$ $\displaystyle=\mathcal{N}(\Lambda|Wz+c,\Psi)$
(2) $\displaystyle\propto exp(-\frac{1}{2}(\Lambda-Wz-c)^{T}\Psi^{-1}(\Lambda-
Wz-c))$

the prior $p(z)$ of the Bayesian model is:

$p(z)=\mathcal{N}(z|0,I)\propto exp(-\frac{1}{2}z^{T}z)$ (3)

This means that the centre of the factor $z$, due to its prior, will be
constraint around 0. The next step to construct a full Bayesian model is to
add the marginal $p(\Lambda)$:

$p(\Lambda)=\int p(\Lambda|z)p(z)dz=\int\mathcal{N}(\Lambda|c,WW^{T}+\Psi)$
(4)

The posterior $p(z|\Lambda)$:

$\displaystyle p(z|\Lambda)=$
$\displaystyle\frac{p(\Lambda|z)p(z)}{p(\Lambda)}=\mathcal{N}(z|m,V)$ (5)
$\displaystyle m=GW^{T}\Psi^{-1}(\Lambda-c)$ $\displaystyle
V=G+\mathbb{E}[z]\mathbb{E}[z]^{T}$

with $G=(I+W^{T}\Psi^{-1}W)^{-1}$.

The log-likelihood of this model is:

$\displaystyle\mathcal{L}(\Lambda|W,z,\Psi)$
$\displaystyle=-\frac{1}{2}trace((\Lambda-c)^{T}\Sigma^{-1}(\Lambda-c))$ (6)
$\displaystyle-\frac{N}{2}log(2\pi)-\frac{1}{2}log|\Sigma|$

with $\Sigma=WW^{T}+\Psi$; where $\Psi$ the $m\times m$ noise diagonal matrix,
$WW^{T}$ the $m\times m$ weights (loadings) matrix, and $\Sigma$ the $m\times
m$ covariance matrix of the labelling data $\Lambda$.

Thus, the variance of the observed data ($\Lambda$) consists of a rank one
component $WW^{T}$ originating from the joint dependence of the indicators on
the latent factor $z$. Together with a full rank diagonal matrix $\Psi$,
arising from the presence of noise, as it is an approximation of the latent
variable.

###  3.4 Other Probabilistic Generative Latent Variable Models

In addition to Factor Analysis, we have also explored two alternative models
in the family of PLVM. Specifically, we built a variational inference version
of the Factor Analysis (VI-FA) using Tensorflow and the Adam optimiser, and we
also put together a version of the GPLVM and SVGP models from GPflow.

VI-FA model: For this model, we followed a similar process as for the
probabilistic PCA [6] but using an independent variance for each data
dimension $m$ (see Eq. 2). To infer the posterior distribution of the latent
variable model we utilise variational inference. We approximate the posterior
$p(W,z,\Psi|\Lambda)$ (see Eq. 5) using a variational distribution
$q(W,z,\Psi)$ with parameters $\theta$. To find $\theta$ we minimise the KL
divergence between the approximate distribution $q$ and the posterior,
$KL(q(W,z,\Psi)|p(W,z,\Psi|\Lambda))$, which is to maximise the ELBO.

GPLVM model: For the latter method, we trained a GPLVM model on the labelling
matrix ($\Lambda$). By its nature, a GPLVM model can be interpreted as a
generalisation of probabilistic PCA [6], where instead of optimising the
linear mappings ($W$ in Eq. 1) we optimise the latent variable $z$. In other
words, it maps the connection between latent data $z$ and observable data
$\Lambda$ using Gaussian-process priors. Overall, the log-likelihood from Eq.
6 becomes

$\displaystyle\mathcal{L}(\Lambda|W,z,K)$
$\displaystyle=-\frac{1}{2}tr((\Lambda-c)K^{-1}(\Lambda^{T}-c))$ (7)
$\displaystyle-\frac{N}{2}log(2\pi)-\frac{1}{2}log|K|$

with $K$ as the Gaussian process kernel.

During inference, the model accepts new (test) latent $z^{*}$-data and
predicts the observable data $\Lambda^{*}$ by computing the two moments, mean
and standard deviation. But, for our approach we need to be able to accept new
observable data $\Lambda^{*}$ and predict the latent $z^{*}$-data. Similar to
any other non-linear kernel-based model it is difficult for the GPLVM to be
used as a dimensionality reduction tool that accepts test data. This is
because it is challenging to invert the mapping between $z$ and observable $X$
(or $\Lambda$ in our case). Various approaches have been proposed that involve
learned pre-images and auxiliary models [32, 13, 7].

After training the GPLVM (Radial Basis Function kernel), we use an auxiliary
Bernoulli regression model (SVGP) with Matern52 kernel to create the mapping
between the latent target variable $z$ and the covariates of the regression
model $\Lambda$. Then, for new data $\Lambda^{*}$ we use the SVGP model for
predicting $z^{*}$.

##  4 Datasets

In this section, we describe the _four_ datasets used to evaluate the model
performance between Snorkel and PLVMs. Three of them are publicly available
and commonly used in the field of weakly supervised learning, and one is
internally sourced. Table 1 provides the summary statistics.

Table 1: Dataset Statistics. $\lambda$ is the labelling function. Absent,
shows the number of rows $n$ in the labelling matrix $\Lambda$ that have all
the columns $m$ assigned as absent $\\{-1\\}$. *For the Spouse dataset we do
not have the target values for the training data, only for the test sub-set.
In the table we use the test data information. For the training data
($n=22,254$) the number of absent rows is $n=16,520$ or 74%.

|  | Number of |  |
---|---|---|---|---
|  | Positive | Negative | Absent | $\lambda$
Source Code | Balanced | 127 | 123 | 0 | 3
Spam | Balanced | 831 | 755 | 230 | 9
Spouse | Unbalanced* | 218 | 2,483 | 1,951 | 9
Goodreads | Unbalanced | 514,778 | 281,293 | 691,795 | 5

YouTube Spam Comments: We use YouTube comments dataset, originally introduced
in [1]. The comments were collected via the YouTube API from five of the ten
most viewed videos on YouTube in the first half of 2015. The training data
have $n=1,586$ YouTube video messages and the test data size is $n=250$. [21]
created the labelling functions that include 5 keyword-based, 1 regular
expression-based, 1 heuristic, 1 complex preprocessors, and 2 third-party
model rules.

Spouse Dataset: This dataset is constructed by [21] to identify mentions of
spouse relationships in a set of news articles from the Signal Media. The data
is split between $n=22,254$ training samples and $n=2,701$ testing samples.
There are 9 heuristic and NLP related labelling functions
111https://github.com/snorkel-team/snorkel-
tutorials/blob/master/spouse/spouse˙demo.ipynb. The ground truth labels for
the training set are not available. Therefore, we are unable to check for
class imbalance or the accuracy of the model on the training set.

Goodreads Dataset: We use the Goodreads dataset, from [29] and [30]. This data
is a smaller sample from the original dataset and contains $n=794,294$
training records and $n=44,336$ test records, collected from $876,145$
Goodreads’ users (with detailed meta-data). We followed the same experiment
settings222https://github.com/snorkel-team/snorkel-
tutorials/blob/master/recsys/recsys_tutorial.ipynb defined by Snorkel, where
the task is to predict whether a user favours the book or not given the
interaction and metadata as context.

Source Code Dataset: In addition to the natural language based tasks, we have
also created a pipeline and evaluated our proposed method in an industrial
setting at JPMorgan; on a binary classification task in the field of Machine
Learning on Source Code (MLonCode). The objective was to predict the label of
each function/method within a set of source code repositories. To the best of
our knowledge, this is the first attempt of applying weakly supervised
learning on source code. We internally curated $n=250$ functions and asked
experienced senior software engineers to construct three labelling functions.
The three labelling functions represent empirical methods that the engineers
would have used if they were to manually assess the quality of the code of the
function/method. This results in a class balanced source code dataset as
indicated in Table 1.

##  5 Experiments

Our aim is to validate the three main hypothesis of the paper: 1) the factor
analysis model can be used for binary classification tasks; 2) the labelling
matrix that contains the observable variables ($\Lambda$) of the model is the
sufficient statistics of the model; 3) using PLVMs we achieve better results
compared to existing methodologies. We ran our experiments using the following
configurations: MacBook Pro 2019, Python 3.7.10, Snorkel 0.9.7, Sklearn 1.7.0,
Tensorflow 2.6.0, Tensorflow-probability 0.13.0. The FA method that is used in
Sklearn follows the SVD approach from [3, p. 448]. For the alternative models
that we used, Variational Inference Factor Analysis (VI-FA) and Gaussian
process latent variable models - Sparse Variational Gaussian process (GPLVM-
SVGP), we relied on Tensorflow and GPflow 2.2.1. All our models and data
shuffling were set with random key $\\{123\\}$.

(a) YouTube Spam

(b) Spouse

(c) Goodreads

Figure 2: Jointplots between the two factors from the FA model. The top and
right sides of the plots illustrate the distribution of each factor for the
respective test set. The dotted line shows the median of the first factor,
which is the one we used to dichotomise it and infer the classes during
inference. The median is calculated on the training data factor. We observe
that utilizing the first factor is sufficient to separate the two labels (blue
and orange).

Selection of classification threshold: To create binary values from the latent
factor $z$ we tested a series of thresholding methods. The best approach was
to dichotomise the normally distributed test factor $z*$ with a median
computed from the training factor $z$. Figure 2 shows that the first factor’s
median (dotted line) of the YouTube Spam data divides the two separate groups
accurately. For the Spouse and the Goodreads datasets, the median threshold
again separates the two groups but not as effectively as in the previous case.
Table 2 shows the performance of using median, mean and Youden’s J statistics
as the threshold, where median achieves the best results. Thus, we propose to
use median as the thresholding method.

Table 2: Threshold selection. The table shows the accuracy scores for each
threshold choice. The CDF threshold was calculated using the Youden’s J
statistic after we transformed the test $z*$ with the normal CDF.
|  | Median | Mean | CDF
---|---|---|---|---
Source Code |  | 0.95 | 0.92 | 0.95
Spam |  | 0.86 | 0.74 | 0.85
Spouse |  | 0.86 | 0.92 | 0.90
Goodreads |  | 0.63 | 0.39 | 0.62

YouTube Spam comments: The YouTube Spam data [1] is a balanced dataset, with
positive class numbers (1) $n=831$ and negative class numbers (0) $n=755$
(Table 1). The Snorkel and FA models achieve close results, with accuracy
$86\%$ for both methods, precision $83\%$ for Snorkel and $85\%$ for FA,
recall $88\%$ and $84\%$, and F1 score $86\%$ and $85\%$.

Table 3: Accuracy, precision, recall, and F1 metrics for Source Code
Classification, YouTube spam, spouse and Goodreads datasets, comparing the
Snorkel approach against the FA WSL model. Each model is trained on the
training dataset and evaluated on the test set. Bold numbers indicates the
best performance.

| Source Code | Spam Dataset (NLP) | Spouse Dataset (NLP) | Goodreads Dataset (Recommender systems)
---|---|---|---|---
| Snorkel | Factor Analysis | Snorkel | Factor Analysis | Snorkel | Factor Analysis | Snorkel | Factor Analysis
Accuracy | 0.92 | 0.95 | 0.86 | 0.86 | 0.54 | 0.86 | 0.53 | 0.63
Precision | 0.90 | 0.97 | 0.83 | 0.85 | 0.12 | 0.32 | 0.66 | 0.65
Recall | 0.95 | 0.93 | 0.88 | 0.84 | 0.72 | 0.64 | 0.56 | 0.95
F1 | 0.93 | 0.95 | 0.86 | 0.85 | 0.20 | 0.42 | 0.61 | 0.77

Table 4: Accuracy, precision, recall, and F1 metrics for Source Code
Classification, YouTube spam, spouse and Goodreads datasets, comparing the FA
WSL model against the VI-FA and GPLVM-SVGP models. The performance has been
measured on the test sample of each dataset. Bold numbers indicates the best
performance.

| Source Code Classification | Spam Dataset (NLP) | Spouse Dataset (NLP) | Goodreads Dataset (Recommender systems)
---|---|---|---|---
| Factor Analysis | VI-FA | GPLVM | Factor Analysis | VI-FA | GPLVM | Factor Analysis | VI-FA | GPLVM | Factor Analysis | VI-FA | GPLVM
Accuracy | 0.95 | 0.92 | 0.92 | 0.86 | 0.70 | 0.82 | 0.86 | 0.78 | 0.09 | 0.63 | 0.60 | 0.63
Precision | 0.97 | 0.89 | 0.89 | 0.85 | 0.89 | 0.79 | 0.32 | 0.22 | 0.08 | 0.65 | 0.64 | 0.64
Recall | 0.93 | 0.95 | 0.95 | 0.84 | 0.42 | 0.85 | 0.64 | 0.70 | 0.99 | 0.95 | 0.89 | 0.96
F1 | 0.95 | 0.92 | 0.92 | 0.85 | 0.58 | 0.82 | 0.42 | 0.34 | 0.15 | 0.77 | 0.74 | 0.77

Spouse Dataset: In the Spouse dataset [21], the FA achieves much higher
performance compared to the Snorkel model. The dataset suffers from _severe_
class-imbalance and a large number of absent labelled classes, as shown in
Table 1. Specifically, from the $n=22,254$ training data, 74% or $n=16,520$
observations have absent values ($\lambda=-1$) in all $m=9$ labelling
functions.

The FA model shows its strength on this type of dataset that have high number
of absent items and experience class-imbalance. In terms of accuracy, Snorkel
scores 54%, whereas our model attains an impressive 86%. On the other hand, in
recall, Snorkel shows a score of 72% and our model 64% (Table 3).

Goodreads Dataset: This is the largest dataset we used for model training and
predictions [29, 30]. Similarly to Spouse data, Goodreads is a class-
imbalanced dataset and it exhibits a considerable amount (87% of the
observations) of absent labelled items (Table 1). In Table 3, FA beats Snorkel
predictions on almost every classification metric, namely 10%+ accuracy, 1.69x
recall and 16% higher F1 score, and achieves marginally lower precision (1%).

Source Code Classification: Table 3 shows the classification performance of
Snorkel and FA when evaluated against the truth data. Specifically, the
accuracy is $92\%$ for Snorkel and $95\%$ for the FA model; the precision is
$90\%$ for Snorkel and $97\%$ for the FA; recall is $95\%$ and $93\%$
subsequently; and finally the F1 score is $93\%$ for Snorkel and $95\%$ for
the FA.

Class Imbalance and Abstentions: To examine the relationship between model
performance and class imbalance, we first quantify the class imbalance by
computing the absolute percentage difference between positive and negative,
$\frac{|{n_{pos}-n_{neg}}|}{n_{pos}+n_{neg}}$ of each dataset, where $n_{pos}$
and $n_{neg}$ refer to the number of positive and negative class respectively.
We then compared this to the result stated in Table 3. In Figure 3, we observe
promising evidence to suggest that as the extent of class imbalance increases,
the performance of Snorkel drops, whereas Factor Analysis model does not.

Figure 3: Relation between the accuracy and absolute percentage difference of
positive and negative classes representing the extent of class imbalance. We
see evidence that the accuracy of Snorkel decreases with the extent of class
imbalance, whereas Factor Analysis does not. FA also consistently outperforms
Snorkel, when we compare the model accuracy for binary classification tasks on
the four datasets: Youtube Spam (triangle), Spouse (star), Goodreads (cross),
Internal Source Code (circle) and the extent of class imbalance.

The effect of class imbalance and abstentions can also be viewed in Figure 2.
In the Spouse _(a)_ and in the Goodreads _(b)_ figures, we observe that
abstentions make more challenging for the model to dichotomise efficiently the
classes. Nevertheless, the FA method performs much more accurately compared to
Snorkel (Table 3) or the other two PLVM models (Table 4).

In general, abstentions and class-imbalance are two critical issues when we
build a WSL pipeline. The probabilistic mechanism of FA, and how it maps the
dependencies across the functions in the labelling matrix $\Lambda$, weaken
the impact of these two problems significantly.

Robustness: We studied how the model performance changes when we vary the size
of the training data. Figure 4 shows that Factor Analysis achieves a $1.7x$
and 14% higher accuracy in Spouse and Goodreads test set with only 10 training
datapoints, and 4% higher in YouTube Spam test set with 30 training
datapoints.

(a) YouTube Spam

(b) Spouse

(c) Goodreads

Figure 4: Comparison of classification performance of Snorkel and Factor
Analysis (our proposed approach) in terms of accuracy. We randomly selected
$n$ sample as the training set forming the labelling matrix $\Lambda$, where
$n\in\\{10,20,30,40,50,60\\}$ and evaluated against their respective test set.
Factor Analysis achieves significant higher accuracy than Snorkel with merely
10 training samples in the Spouse and Goodreads datasets, and 30 training
samples for the YouTube Spam dataset illustrating the robustness of our
approach.

##  6 Benefits of the model

The benefits of using the benchmark model to replace the pipeline of the
Snorkel algorithm, include the speed, the robustness of the results on
unbalanced datasets, the causality that FA offers [9], and the explainability
of the model compared to the Snorkel probabilistic approach. Explainability,
as in terms of the FA model is a method that has been tested, evaluated, and
used for years in the field of social sciences and the underlying mechanisms
have been studied extensively. Finally, as we demonstrated in the evaluation
of source code in JPMorgan, our approach can be easily integrated into
existing machine learning workflows using standard widely-used libraries in
Python (scikit) or R (psych). This allows users to leverage their existing
knowledge and resources to quickly adopt and integrate the new framework into
their applications.

##  7 Limitations

While the proposed method offers several benefits, there are some limitations
that require further investigation in future research. The model’s inability
to perform on multi-label datasets, which are commonly encountered in many
real-world applications. Multi-label datasets involve instances that can be
assigned multiple labels or categories simultaneously, making them more
complex than single-label datasets. Unfortunately, the model developed in this
research was not able to effectively handle this type of data. We attempted to
address this issue by increasing the number of principal components in the
model output, but this did not yield significant improvements in performance.

##  8 Discussion and Future Direction

We introduced generative probabilistic latent variable models (Factor
Analysis) as a novel approach to solve weakly supervised learning tasks. Our
method, by using high-level domain knowledge from Subject Matter Experts,
accomplishes high quality results and can be an excellent choice for
approximating true labels on unlabelled data. We provided evidence that Factor
Analysis is resilient to class imbalanced datasets as indicated by the
significant improvement to the classification performance. Finally, we tested
the effect of sparse data resources by varying the number of data-points used
to train the generative model and we showed that with a minimum number of
points our approach can attain high performance. For future work, we hope to
expand the generative probabilistic latent variable models into a multi-class
domain and explore our approach to other weakly supervised learning tasks.

####  8.0.1 Acknowledgements

We want to thank the reviewers for their time, effort, and the very
constructive feedback and advice. Our aim was to try and incorporate as many
of their suggestions as possible considering the time. If some of their
suggestions are not present (e.g. more datasets) is purely because of the
limited timeframe. The readers can find the relevant code, as soon as it
becomes available, at the JPMorgan Github https://github.com/jpmorganchase
under the repository name weakly-supervision and the branch ecml-experiments.

## References

  * [1] Alberto, T.C., Lochter, J.V., Almeida, T.A.: TubeSpam: Comment spam filtering on YouTube. In: 2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA). IEEE (Dec 2015). https://doi.org/10.1109/icmla.2015.37, https://doi.org/10.1109/icmla.2015.37
  * [2] Bach, S.H., He, B., Ratner, A., Ré, C.: Learning the structure of generative models without labeled data. In: Proceedings of the 34th International Conference on Machine Learning - Volume 70. p. 273–282. ICML’17, JMLR.org (2017)
  * [3] Barber, D.: Bayesian Reasoning and Machine Learning. Cambridge University Press (2012)
  * [4] Bartholomew, D.J.: The foundations of factor analysis. Biometrika 71(2), 221–232 (1984). https://doi.org/10.1093/biomet/71.2.221, https://doi.org/10.1093/biomet/71.2.221
  * [5] Bazavan, E.G., Zanfir, A., Zanfir, M., Freeman, W.T., Sukthankar, R., Sminchisescu, C.: Hspace: Synthetic parametric humans animated in complex environments (2022)
  * [6] Bishop, C.M.: Pattern Recognition and Machine Learning. Springer (2006)
  * [7] Dai, Z., Damianou, A., Gonzalez, J., Lawrence, N.D.: Variationally auto-encoded deep Gaussian processes. In: Larochelle, H., Kingsbury, B., Bengio, S. (eds.) Proceedings of the International Conference on Learning Representations. vol. 3. Caribe Hotel, San Juan, PR (2016), http://inverseprobability.com/publications/dai-variationally16.html
  * [8] Dunnmon, J.A., Ratner, A.J., Saab, K., Khandwala, N., Markert, M., Sagreiya, H., Goldman, R., Lee-Messer, C., Lungren, M.P., Rubin, D.L., Ré, C.: Cross-modal data programming enables rapid medical machine learning. Patterns 1(2), 100019 (May 2020). https://doi.org/10.1016/j.patter.2020.100019, https://doi.org/10.1016/j.patter.2020.100019
  * [9] Fabrigar, L.R., Wegener, D.T., MacCallum, R.C., Strahan, E.J.: Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods 4(3), 272–299 (Sep 1999). https://doi.org/10.1037/1082-989x.4.3.272, https://doi.org/10.1037/1082-989x.4.3.272
  * [10] Fries, J.A., Steinberg, E., Khattar, S., Fleming, S.L., Posada, J., Callahan, A., Shah, N.H.: Ontology-driven weak supervision for clinical entity classification in electronic health records. Nature Communications 12(1) (Apr 2021). https://doi.org/10.1038/s41467-021-22328-4, https://doi.org/10.1038/s41467-021-22328-4
  * [11] Goswami, M., Boecking, B., Dubrawski, A.: Weak supervision for affordable modeling of electrocardiogram data (2022)
  * [12] Jain, N.: Customer sentiment analysis using weak supervision for customer-agent chat (2021)
  * [13] Lawrence, N.D., Quiñonero-Candela, J.: Local distance preservation in the GP-LVM through back constraints. In: Proceedings of the 23rd international conference on Machine learning - ICML '06. ACM Press (2006). https://doi.org/10.1145/1143844.1143909, https://doi.org/10.1145/1143844.1143909
  * [14] Liu, Z., Zhu, X., Yang, L., Yan, X., Tang, M., Lei, Z., Zhu, G., Feng, X., Wang, Y., Wang, J.: Multi-Initialization Optimization Network for Accurate 3D Human Pose and Shape Estimation, p. 1976–1984. Association for Computing Machinery, New York, NY, USA (2021), https://doi.org/10.1145/3474085.3475355
  * [15] Manco, I., Benetos, E., Quinton, E., Fazekas, G.: Learning music audio representations via weak language supervision (2021)
  * [16] Mathew, J., Negi, M., Vijjali, R., Sathyanarayana, J.: DeFraudNet: An end-to-end weak supervision framework to detect fraud in online food delivery. In: Machine Learning and Knowledge Discovery in Databases. Applied Data Science Track, pp. 85–99. Springer International Publishing (2021). https://doi.org/10.1007/978-3-030-86514-6_6, https://doi.org/10.1007/978-3-030-86514-6_6
  * [17] Murphy, K.P.: Machine Learning: A Probabilistic Perspective. MIT Press (2012)
  * [18] Rao, V.R., Khalil, M.I., Li, H., Dai, P., Lu, J.: Decompose the sounds and pixels, recompose the events (2021)
  * [19] Ratner, A., Hancock, B., Dunnmon, J., Goldman, R., Ré, C.: Snorkel metal: Weak supervision for multi-task learning. In: Proceedings of the Second Workshop on Data Management for End-To-End Machine Learning. DEEM’18, Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3209889.3209898, https://doi.org/10.1145/3209889.3209898
  * [20] Ratner, A., Bach, S.H., Ehrenberg, H., Fries, J., Wu, S., Ré, C.: Snorkel: rapid training data creation with weak supervision. The VLDB Journal 29(2-3), 709–730 (Jul 2019). https://doi.org/10.1007/s00778-019-00552-1, https://doi.org/10.1007/s00778-019-00552-1
  * [21] Ratner, A., Hancock, B., Dunnmon, J., Sala, F., Pandey, S., Ré, C.: Training complex models with multi-task weak supervision. Proceedings of the AAAI Conference on Artificial Intelligence 33, 4763–4771 (Jul 2019). https://doi.org/10.1609/aaai.v33i01.33014763, https://doi.org/10.1609/aaai.v33i01.33014763
  * [22] Ratner, A.J., De Sa, C.M., Wu, S., Selsam, D., Ré, C.: Data programming: Creating large training sets, quickly. In: Lee, D., Sugiyama, M., Luxburg, U., Guyon, I., Garnett, R. (eds.) Advances in Neural Information Processing Systems. vol. 29. Curran Associates, Inc. (2016), https://proceedings.neurips.cc/paper/2016/file/6709e8d64a5f47269ed5cea9f625f7ab-Paper.pdf
  * [23] Reddy, R.G., Rui, X., Li, M., Lin, X., Wen, H., Cho, J., Huang, L., Bansal, M., Sil, A., Chang, S.F., Schwing, A., Ji, H.: Mumuqa: Multimedia multi-hop news question answering via cross-media knowledge extraction and grounding (2021)
  * [24] Saab, K., Dunnmon, J., Goldman, R., Ratner, A., Sagreiya, H., Ré, C., Rubin, D.: Doubly weak supervision of deep learning models for head CT. In: Lecture Notes in Computer Science, pp. 811–819. Springer International Publishing (2019). https://doi.org/10.1007/978-3-030-32248-9_90, https://doi.org/10.1007/978-3-030-32248-9_90
  * [25] Saab, K., Dunnmon, J., Ré, C., Rubin, D., Lee-Messer, C.: Weak supervision as an efficient approach for automated seizure detection in electroencephalography. npj Digital Medicine 3(1) (Apr 2020). https://doi.org/10.1038/s41746-020-0264-0, https://doi.org/10.1038/s41746-020-0264-0
  * [26] Tipping, M.E., Bishop, C.M.: Probabilistic principal component analysis. JOURNAL OF THE ROYAL STATISTICAL SOCIETY, SERIES B 61(3), 611–622 (1999)
  * [27] Tseng, A., Sun, J.J., Yue, Y.: Automatic synthesis of diverse weak supervision sources for behavior analysis (2021)
  * [28] Varma, P., Sala, F., He, A., Ratner, A., Re, C.: Learning dependency structures for weak supervision models. In: Chaudhuri, K., Salakhutdinov, R. (eds.) Proceedings of the 36th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 97, pp. 6418–6427. PMLR (09–15 Jun 2019), https://proceedings.mlr.press/v97/varma19a.html
  * [29] Wan, M., McAuley, J.: Item recommendation on monotonic behavior chains. In: Proceedings of the 12th ACM Conference on Recommender Systems. ACM (Sep 2018). https://doi.org/10.1145/3240323.3240369, https://doi.org/10.1145/3240323.3240369
  * [30] Wan, M., Misra, R., Nakashole, N., McAuley, J.: Fine-grained spoiler detection from large-scale review corpora. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics (2019). https://doi.org/10.18653/v1/p19-1248, https://doi.org/10.18653/v1/p19-1248
  * [31] Weng, Z., Varma, P., Masalov, A., Ota, J., Re, C.: Utilizing weak supervision to infer complex objects and situations in autonomous driving data. In: 2019 IEEE Intelligent Vehicles Symposium (IV). IEEE (Jun 2019). https://doi.org/10.1109/ivs.2019.8814147, https://doi.org/10.1109/ivs.2019.8814147
  * [32] Weston, J., Schölkopf, B., Bakir, G.: Learning to find pre-images. In: Thrun, S., Saul, L., Schölkopf, B. (eds.) Advances in Neural Information Processing Systems. vol. 16. MIT Press (2004), https://proceedings.neurips.cc/paper/2003/file/ac1ad983e08ad3304a97e147f522747e-Paper.pdf
  * [33] Wolfson, T., Berant, J., Deutch, D.: Weakly supervised mapping of natural language to sql through question decomposition (2021)
  * [34] Zhang, J., Yu, Y., Li, Y., Wang, Y., Yang, Y., Yang, M., Ratner, A.: WRENCH: A comprehensive benchmark for weak supervision. In: Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (2021), https://openreview.net/forum?id=Q9SKS5k8io
  * [35] Zheng, J., Shi, X., Gorban, A., Mao, J., Song, Y., Qi, C.R., Liu, T., Chari, V., Cornman, A., Zhou, Y., Li, C., Anguelov, D.: Multi-modal 3d human pose estimation with 2d weak supervision in autonomous driving (2021)

## Disclaimer

This paper was prepared for informational purposes by the Applied Innovation
of AI (AI2) and Global Technology Applied Research center of JPMorgan Chase &
Co. This paper is not a product of the Research Department of JPMorgan Chase &
Co. or its affiliates. Neither JPMorgan Chase & Co. nor any of its affiliates
makes any explicit or implied representation or warranty and none of them
accept any liability in connection with this paper, including, without
limitation, with respect to the completeness, accuracy, or reliability of the
information contained herein and the potential legal, compliance, tax, or
accounting effects thereof. This document is not intended as investment
research or investment advice, or as a recommendation, offer, or solicitation
for the purchase or sale of any security, financial instrument, financial
product or service, or to be used in any way for evaluating the merits of
participating in any transaction.