# Supplemental material for “Detection of Spin Coherence in Cold Atoms via
Faraday Rotation Fluctuations”

Maheswar Swar  Dibyendu Roy  Subhajit Bhar  Sanjukta Roy  Saptarishi Chaudhuri
Raman Research Institute, C. V. Raman Avenue, Sadashivanagar, Bangalore
560080, India

This supplemental material provides details of our experimental set-up,
theoretical modeling, and data analysis methods. We also discuss the
dependence of the Faraday rotation fluctuations signal on the polarization
states of the Raman fields.

##  .1 Description of the experimental set-up

In the present Letter, we have reported the results from two sets of
experiments: one in a thermal atomic vapor to develop and characterize the
measurement of Faraday rotation fluctuations signal from a coherently driven
system, and another to perform the main study of the extraction of spin
relaxation rate from a cloud of cold 85Rb atoms. Here, we provide details of
the second set-up. We magneto-optically trap neutral 85Rb atoms inside a glass
cell maintaining background pressure less than $10^{-10}$ millibar using
standard laser cooling and trapping techniques. The cooling beams were
generated from an external cavity diode laser (ECDL) and frequency stabilized
to the 12 MHz red detuned with respect to $5S_{1/2},F=3$ $\rightarrow$
$5P_{3/2},F^{\prime}=4$ (D2) transition. The ‘repumping’ laser beams were
derived from another ECDL and frequency stabilized to the $5S_{1/2},F=2$
$\rightarrow$ $5P_{1/2},F^{\prime}=3$ (D1) transition. A pair of magnetic
coils in near ideal anti-Helmholtz configuration produces the required spatial
magnetic field gradient. We coincided the optical and magnetic field centers
with the center of the vacuum chamber with an accuracy of $\sim$ 30 $\mu$m. We
use three independent detection techniques: absorption imaging, fluorescence
imaging, and probe absorption for characterizing the cold atoms and alignment
of the optical fields for the experiments.

Figure S1: The absorption image of magneto-optically trapped (MOT) cold 85Rb
atomic cloud. The top and right insets show the column density profiles (red)
and fit to Gaussian (black) of the trapped cloud.

In Fig. S1, we show a typical absorption image of our cold cloud, which gives
a good measurement of the total number and the spatial distribution of the
atoms. Using a separate time-of-flight measurement, we obtain the temperature
of the atomic cloud to be $\sim$ 150 $\mu$K. We typically trap more than 107
atoms with a Gaussian full width at half maxima (FWHM) of $\sim$ 4 mm.

The probe laser field is generated from yet another ECDL, and its frequency is
monitored using a high precision wavelength meter (Highfinesse, WSU2) with an
absolute frequency accuracy of 1 MHz. The collimated probe beam with a
Gaussian waist diameter of 70 $\mu$m was sent through the atomic cloud along
$\hat{x}$ at $y=0$, and the $z$ position of the probe beam was varied for
different sets of measurements.

The strength and orientation of the magnetic field $B$ vary within the atomic
cloud along the probe field direction ($x$-axis) as shown in Fig. S2. In this
configuration, the magnetic field exists only on the x-z plane, and we
calculate the magnetic field at each position using a solution of the elliptic
equations James Simpson and Youngquist (2001). We separately measure the field
components ($B_{x},B_{y},B_{z}$) using a Hall probe magnetometer (LakeShore)
for comparison and calibration purposes. The angle $\theta_{B}(x)$ between the
local magnetic field and probe laser propagation direction is position-
dependent.

The Raman radiation fields for coherent driving were derived from the same
ECDL, which provides the cooling laser light. Two independent acousto-optic
modulators (AOMs) were applied to produce the Raman fields with controllable
frequency difference (denoted as $\delta_{12}$ in the main text). The radio
frequency signal sent into the AOMs can be tuned using two voltage-controlled
crystal oscillators. The control voltage was sent using an FPGA board with a
vertical resolution of 12 bits. The long-term relative frequency ‘jitter’ of
$\delta_{12}$ was measured to be $\sim$ 7 kHz (for measurement duration of 8
seconds). This ‘jitter’ limits our measurement precision, which we wish to
reduce to a few Hz level using ultra-stable reference sources in our future
upgrading of the experimental set-up. However, this relative frequency
stability is adequate for the first sets of measurements of the Faraday
rotation fluctuations signal from the cold atoms. The Raman fields were
spatially mode cleaned using PM fibers, expanded to a Gaussian waist diameter
of 6 mm, and combined in a non-polarizing cube beam splitter (NPBS) before
sending through the cold atomic cloud. The polarization of the Raman fields
can be independently varied, employing the combination of half-wave plates
(HWPs) and quarter-wave plates (QWPs). We typically manage to obtain a
polarization purity of the Raman fields $>$ 99$\%$.

Figure S2: The $x$-dependence of the total magnetic field and the angle
$\theta_{B}(x)$ between the local magnetic field and the probe laser
propagation at $y=0$ and $z=900~{}\mu$m.

##  .2 Details of the theoretical modeling

Intrinsic spin noise spectrum: In the absence of driving by the Raman fields
$(\Omega=0)$, the rubidium atoms are in thermal equilibrium, and the
populations in their hyperfine ground level Zeeman states fluctuate over time
due to thermal (and quantum) fluctuations. Such equilibrium population
fluctuations generate an intrinsic spin noise (SN) in the atomic vapor. When
$\Omega=0$, only the lower two levels of the three-level system (3LS)
participate in the equilibrium spin dynamics detected by the probe beam.
Therefore, we can derive the SN spectrum by including a noise term in the
master equations of the density matrix elements of these levels. Thus, we
write

$\displaystyle\frac{d\tilde{\rho}_{21}}{dt}$ $\displaystyle=$ $\displaystyle
i(\omega_{2}-\omega_{1}+i\gamma_{21})\tilde{\rho}_{21}+\eta(t),$ (S1)

where we assume the noise $\eta(t)$ to be a Gaussian white noise with zero
mean and
$\langle\eta(t^{\prime})\eta(t^{\prime\prime})\rangle=N_{2}\gamma_{21}\delta(t^{\prime}-t^{\prime\prime})$,
where $N_{2}$ is number of atoms within the measurement region. We then get
the power spectrum $P(\omega)|_{\Omega=0}$ of the spontaneous spin
fluctuations as:

$\displaystyle P(\omega)|_{\Omega=0}$ $\displaystyle=$
$\displaystyle\frac{1}{2\pi}\int_{-\infty}^{\infty}dt\>e^{i\omega
t}\langle\tilde{\rho}_{21}^{\dagger}(t)\tilde{\rho}_{21}(0)\rangle$ (S2)
$\displaystyle=$ $\displaystyle\frac{1}{\pi}{\rm
Re}\Big{[}\int_{0}^{\infty}dt\>e^{i\omega
t}\langle\tilde{\rho}_{21}^{\dagger}(t)\tilde{\rho}_{21}(0)\rangle\Big{]}$
$\displaystyle=$
$\displaystyle\frac{N_{2}}{2\pi}\frac{\gamma_{21}}{\gamma_{21}^{2}+(\omega-\omega_{2}+\omega_{1})^{2}},$

where $\omega$ is the spectral frequency.

Raman-driven power spectrum: In the presence of the Raman fields, the atoms
are driven out of thermal equilibrium. We again write phenomenological master
equations for the evolution of various components of the density matrix of the
3LS for the Hamiltonian $\mathcal{H}$ in the main text. Since the intrinsic
fluctuations (related to the equilibrium noise, e.g., $\eta(t)$ above) only
form a broad background in the measured power spectrum for a relatively strong
driving by the Raman fields, we drop these noise terms from the following
master equations to be able to extract an analytical expression for the
measured power spectrum. These master equations are written in the rotating
frame by rewriting the elements of coherence as
$\rho_{31}(t)=\tilde{\rho}_{31}(t)e^{i\omega_{s_{1}}t},\rho_{32}(t)=\tilde{\rho}_{32}(t)e^{i\omega_{s_{2}}t},\rho_{21}(t)=\tilde{\rho}_{21}(t)e^{i(\omega_{s_{1}}-\omega_{s_{2}})t}$,
where $\tilde{\rho}_{31}(t),\tilde{\rho}_{32}(t)$ and $\tilde{\rho}_{21}(t)$
are the density matrix elements in the laboratory frame, and $\rho_{ij}$ are
those in the rotated frame. We further take the following limits for the
relaxation rates,
$\gamma_{13}=\gamma_{23}=\gamma,\gamma_{12}=\gamma_{21}=\gamma^{\prime},\gamma_{31}=\gamma_{32}=0$
and $\gamma^{\prime}\ll\gamma$, to simplify the master equations.

$\displaystyle\frac{d\rho_{11}}{dt}$ $\displaystyle=$
$\displaystyle\gamma(1-\rho_{11}-\rho_{22})-i\Omega_{13}(\rho_{13}-\rho_{31}),$
(S3) $\displaystyle\frac{d\rho_{22}}{dt}$ $\displaystyle=$
$\displaystyle\gamma(1-\rho_{11}-\rho_{22})-i\Omega_{23}(\rho_{23}-\rho_{32}),$
(S4) $\displaystyle\frac{d\rho_{13}}{dt}$ $\displaystyle=$
$\displaystyle-(\gamma+i\Delta_{13})\rho_{13}-i\Omega_{23}\rho_{12}$ (S5)
$\displaystyle-$ $\displaystyle i\Omega_{13}(2\rho_{11}-1+\rho_{22}),$
$\displaystyle\frac{d\rho_{31}}{dt}$ $\displaystyle=$
$\displaystyle-(\gamma-i\Delta_{13})\rho_{31}+i\Omega_{23}\rho_{21}$ (S6)
$\displaystyle+$ $\displaystyle i\Omega_{13}(2\rho_{11}-1+\rho_{22}),$
$\displaystyle\frac{d\rho_{23}}{dt}$ $\displaystyle=$
$\displaystyle-(\gamma+i\Delta_{23})\rho_{23}-i\Omega_{13}\rho_{21}$ (S7)
$\displaystyle-$ $\displaystyle i\Omega_{23}(2\rho_{22}-1+\rho_{11}),$
$\displaystyle\frac{d\rho_{32}}{dt}$ $\displaystyle=$
$\displaystyle-(\gamma-i\Delta_{23})\rho_{32}+i\Omega_{13}\rho_{12}$ (S8)
$\displaystyle+$ $\displaystyle i\Omega_{23}(2\rho_{22}-1+\rho_{11}),$
$\displaystyle\frac{d\rho_{12}}{dt}$ $\displaystyle=$
$\displaystyle-i(\Delta_{13}-\Delta_{23}-i\gamma^{\prime})\rho_{12}+i\Omega_{13}\rho_{32}$
(S9) $\displaystyle-$ $\displaystyle i\Omega_{23}\rho_{13},$
$\displaystyle\frac{d\rho_{21}}{dt}$ $\displaystyle=$ $\displaystyle
i(\Delta_{13}-\Delta_{23}+i\gamma^{\prime})\rho_{21}-i\Omega_{13}\rho_{23}$
(S10) $\displaystyle+$ $\displaystyle i\Omega_{23}\rho_{31}.$

We apply these equations to investigate how the Raman fields affect the
coherence $\tilde{\rho}_{21}(t)$ between the ground levels. From the above set
of equations, we find $\rho_{21}(t)$ at the steady-state by setting
$d\rho_{ij}(t)/dt=0$. Since we ignore the noise terms in the above master
equations in the leading order of $\Omega_{13},\Omega_{23}$, we rewrite the
power spectrum defined in the main text without the noise averaging:

$\displaystyle P(\omega)$ $\displaystyle=$
$\displaystyle\frac{1}{2\pi}\int_{-\infty}^{\infty}dt\>e^{i\omega
t}\tilde{\rho}_{21}^{\dagger}(t)\tilde{\rho}_{21}(0)$ (S11) $\displaystyle=$
$\displaystyle\frac{1}{2\pi}\int_{-\infty}^{\infty}dt\>e^{i(\omega-\omega_{s_{1}}+\omega_{s_{2}})t}\rho_{21}^{\dagger}(t)\rho_{21}(0).$

When the driven atoms reach the steady-state at a long time, $\rho_{21}(t)$
becomes time-independent. Then, we can replace $\rho_{21}(t)$ and
$\rho_{21}(0)$ by their steady-state value $\rho_{21}$ to find:

$\displaystyle P(\omega)$ $\displaystyle=$
$\displaystyle\frac{1}{2\pi}\int_{-\infty}^{\infty}dt\>e^{i(\omega-\omega_{s_{1}}+\omega_{s_{2}})t}|\rho_{21}|^{2}$
(S12) $\displaystyle=$
$\displaystyle\delta(\omega+\omega_{s_{2}}-\omega_{s_{1}})|\rho_{21}|^{2}.$

In the main text, we provide a relatively simple formula for $\rho_{21}$ by
choosing $\Omega_{13}=\Omega_{23}=\Omega$ and
$\Delta_{23}=0,\Delta_{13}=\Delta$, and that formula is employed for the
driven thermal vapors. Here, we give a more general formula when
$\Delta_{23}\neq 0$, which is our experimental condition for the driven
rubidium cold atoms.

$\displaystyle\rho_{21}=\frac{\gamma\Omega^{2}(\tilde{\Delta}(2i\gamma+\tilde{\Delta})-4\Omega^{2}+(-2\gamma+i\tilde{\Delta})\gamma^{\prime})}{\gamma(\tilde{\Delta}^{2}(2\gamma^{2}+\Delta_{23}^{2}+\Delta_{13}^{2})+2\Omega^{2}\tilde{\Delta}^{2}+8\Omega^{4})+\Omega^{2}(8\gamma^{2}+(\Delta_{23}+\Delta_{13})^{2}+12\Omega^{2})\gamma^{\prime}+\gamma\gamma^{\prime
2}(2\gamma^{2}+\Delta_{23}^{2}+\Delta_{13}^{2}+6\Omega^{2})},$ (S13)

where $\tilde{\Delta}=\Delta_{23}-\Delta_{13}$. The power spectrum in Eq. S12
gives a delta peak (broadened in our experiment by relative frequency jitter
of the two Raman fields derived from two independent AOMs) at
$\omega=\omega_{s_{1}}-\omega_{s_{2}}$, whose strength is determined by
$|\rho_{21}|^{2}$ given in Eq. S13. The strength of the peak is maximum at
$\Delta_{13}=0$ when $\Delta_{23}=0$, and the peak height falls with
increasing $\Delta_{13}$. For $\Delta_{23}=0$, the envelope of the sharp delta
peaks with changing $\Delta_{13}$ has a Lorentzian shape when $\Omega<\gamma$
but it changes to Gaussian form for larger $\Omega$. For $\Delta_{23}\neq 0$,
the peak height is maximum at $\Delta_{13}=\Delta_{23}$. The envelope of
$|\rho_{21}|^{2}$ is asymmetric with $\Delta_{13}$ when $\Delta_{23}\neq 0$.

##  .3 Simulation and data analysis

For a homogeneous magnetic field as in the thermal vapor measurements, the
Faraday rotation fluctuations signal (both the intrinsic and the Raman driven)
is centered around a single Larmor frequency $\nu_{L}$ ($=g_{F}\mu_{B}B/h$,
where $g_{F}$ is the Land$\acute{e}$ g-factor of the hyperfine F-levels,
$\mu_{B}$ is the Bohr magneton and h is the Planck’s constant), determined by
the magnetic field strength $B$. However, for the cold atoms inside an MOT,
the Larmor frequency $\nu_{L}(x)$ varies over space along $\hat{x}$.
Therefore, the Zeeman splittings of the ground hyperfine levels are determined
by the magnitude of the local magnetic field and, the Raman resonance
condition is also position-dependent, i.e., $\delta_{12}(x)=\nu_{L}(x)$. In
our numerical modeling, we take this position-dependent local magnetic field
into account to calculate the strength of the Faraday rotation fluctuations
signal at different frequency.

Moreover, the orientation ($\theta_{B}(x)$) of the local magnetic field also
varies along $\hat{x}$ inside the MOT. The SN signal strength at $\nu_{L}(x)$
also gets modified by a factor sin${}^{2}\theta_{B}(x)$ Sinitsyn and Pershin
(2016). We have experimentally verified this correction factor by performing a
separate calibration measurement of the intrinsic SN spectrum in thermal
atoms.

The atom density distribution within the MOT detected by the probe laser is
not uniform, which is evident from the absorption image in Fig. S1. We
incorporate this density distribution in the modelling of the SN signal
strength ($\propto n(x)$, where $n(x)$ is the number density of atoms at
position $x$) from the MOT Crooker  _et al._ (2004); Swar  _et al._ (2019).
Note that $n(x)$ can be measured precisely using absorption imaging.

Another minor correction to the SN signal strength can be from the definition
of the quantization axis, which also varies along $x$-direction on the x-z
plane. Since the local magnetic field alters over space, the coupling of the
Raman fields with the atoms in the MOT also depends on $x$-position. Such
correction can be incorporated in our modelling by an $x$-dependent Rabi
frequency defined as $\Omega(x)=\Omega(1-\sin^{2}\theta_{B}(x)/2)$.

So far, we have discussed the Raman-driven spin coherence between the ground
states involving a single $\Lambda$ system formed by states $|i\rangle$, where
$i$ = 1,2, and 3, as described in the paper. Ideally, in cold atom
experiments, six $\Lambda$ systems are involved in giving rise to the driven
power spectrum generated from $F=$ 3 and $F^{\prime}=$ 4 hyperfine levels of
85Rb. However, the value of Land$\acute{e}$ g-factor is different for those
two hyperfine levels, resulting in dissimilar contribution in building the
signal strength from individual $\Lambda$ system through the optical detuning
of the Raman fields.

Incorporating the above factors and corrections, we get the strength
$|\rho_{21}(x)|^{2}$ of the Faraday rotation fluctuations signal from the
atoms at position $x$ inside the MOT,

$|\rho_{21}(x)|^{2}=|\rho_{21}(\Omega(x))|^{2}n(x)\sin^{2}\theta_{B}(x).$
(S14)

For a fixed position $x$ within the MOT, we consider the contributions of all
six $\Lambda$ systems and add them up to get the total driven spectrum
strength. The six $\Lambda$ systems contribute differently via the optical
detuning $\Delta_{23}(n)$ of the Raman fields from the excited state, i.e.,

$\Delta_{23}(n)=2\gamma+(3-n)(g_{F^{\prime}}-g_{F})\mu_{B}B(x)/h,$ (S15)

where $n$ runs from 1 to 6 and $g_{F^{\prime}}$ is the Land$\acute{e}$
g-factor of the excited state. We fix the detuning $\Delta_{23}(n)=2\gamma$
for $n=3$ in our experiment as described in the main text.

In Fig. 4(c,d,e) of the main paper, we present some plots for the Faraday
rotation fluctuations signals as a function of $\delta_{12}$ from a driven
cold rubidium cloud. We have fitted the experimental data using the Eq. S14
along with $\rho_{21}(\Omega(x))$ from Eq. S13, where $\Omega$ is replaced by
$\Omega(x)$ and we employ $\Delta_{23}$ from Eq. S15. The only free parameter
in this fitting is the relaxation rate $\gamma^{\prime}$ of the hyperfine
ground level Zeeman states, and all other parameters are measured in our
experiment.

##  .4 Dependence of driven power spectrum on Raman fields’ polarization

We have experimentally investigated the dependence of the Raman driven power
spectrum on the polarization state of the Raman fields in both thermal vapors
and cold atomic clouds.

In Fig. S3 (a), we show the driven power spectrum for various combinations of
the polarization state of the Raman fields in thermal vapors. The polarization
of the R1 field is linear ($(\pi_{1})_{x}$), and kept fixed. We have tuned the
polarization state of R2 field, and recorded the driven power spectrum as
shown in Fig. S3(a). The strength of the spectrum is maximum for
$(\pi_{1})_{x}-(\sigma_{2}^{+})_{x}$ polarization of R1 and R2 fields, which
corresponds to the angle $\theta=$ 45∘ or 225∘ between the optic axis of the
QWP and the input polarization (p-polarized) of the R2 field. The Raman fields
can not drive the atoms coherently between the states $|1\rangle$ and
$|2\rangle$ for polarization combination $(\pi_{1})_{x}-(\pi_{2})_{x}$, which
corresponds to $\theta=$ 90∘ or 180∘. This fact was experimentally confirmed
and is presented in Fig. S3(a), which shows that we indeed coherently drive
the entire atomic sample as opposed to incoherent driving. Note that we have
observed an additional maximum at $\theta=135^{\circ}$. The appearance of this
maximum can be explained in the above way by considering a $\Lambda$ system
with ground states
$|F=2,m_{F}=-1\rangle\equiv|1\rangle,|F=2,m_{F}=0\rangle\equiv|2\rangle$ and
excited state $|F^{\prime}=3,m_{F^{\prime}}=-1\rangle\equiv|3\rangle$. Such a
combination of states is allowed for the alkali atom 87Rb in our thermal vapor
experiments.

We here demonstrate the role of angular momentum conservation in coherent
coupling through our measurements. In Fig. S3(b), the normalized peak strength
of the driven power spectrum ($|\rho_{21}|_{\Delta=0}^{2}$) from thermal
vapors is shown for various angle $\theta$. The observation in Fig. S3(b)
shows the fidelity of the coherent coupling of atoms by the Raman fields’
polarization state. This can also be applied to control the atomic coherence
between ground levels. The manipulation of atomic level coherence may find
applications in quantum communications and quantum information processing
using neutral atoms.

Figure S3: (a) The Raman driven power spectrum with detuning $\delta_{12}$ for
various polarization states of the R2 field in thermal vapors. The R1 field is
p-polarized ($(\pi_{1})_{x}$). $\theta$ is the angle between the optic axis of
the quarter-wave plate (QWP) and the input polarization (p-polarized) of the
R2 field. The angular momentum conservation of light-matter interactions in
$\Lambda$ system is satisfied for $(\pi_{1})_{x}-(\sigma_{2}^{\pm})_{x}$
polarization combinations of the Raman fields. These correspond to $\theta=$
45∘, 135∘, 225∘, where a maximum in the spectrum is observed. The driven
spectrum vanishes for $(\pi_{1})_{x}-(\pi_{2})_{x}$ polarization combinations
($\theta=$ 90∘and 180∘), implying no coherent coupling between the states
$|1\rangle\leftrightarrow|2\rangle$ by the Raman fields. (b) The driven
spectrum strength as a function of $\theta$ at Raman resonance condition.

A similar study for the polarization dependence of the driven power spectrum
in cold rubidium atoms is shown in Fig. S4. We have fixed $\delta_{12}=$ 2.73
MHz to resonantly drive the atoms located at $x=1.9$ mm, and $z=900~{}\mu$m
where $\theta_{B}\approx 45^{\circ}$. The dependence of $P(\omega)$ on the
Raman fields’ polarization for cold atoms is similar to the previous case with
thermal vapors (shown in Fig. S3). While the peak signal strength of
$P(\omega)$ reduces exactly to zero at $90^{\circ}$ and $180^{\circ}$ as
expected, the magnitude at $\theta=135^{\circ}$ reaches only half of the
observed values at $\theta=45^{\circ}$ and $225^{\circ}$ in cold atoms inside
the MOT, which is very different from the thermal vapors in homogeneous
magnetic field.

Figure S4: The driven power spectrum strength at $\delta_{12}=2.73$ MHz for
various angle $\theta$ in cold atoms. The signal strength at
$\theta=135^{\circ}$ is about $1/2$ of that at $\theta=45^{\circ}$ and
$225^{\circ}$. The polarization sensitivity at each $\delta_{12}$ carries its
unique signature for atoms in the MOT, unlike atoms in a homogeneous magnetic
field.

##  .5 Raman coherence in the quantization basis

The experiments in vapor cell were performed in the presence of a homogeneous
magnetic field (defining the quantization axis) applied along $\hat{z}$, and
the Raman fields propagating along $\hat{x}$. The probe laser propagating
along $x$-direction detects the $x$-component of the atomic spins. Here we
neglect the slight angle between the probe and the Raman lasers, as
schematically shown in the Fig. S5.

Figure S5: The diagram depicts the direction of the applied uniform magnetic
field ($B\hat{z}$), the propagation direction of probe ($k_{probe}\hat{x}$)
and Raman ($k_{Raman}\hat{x}$) lasers.

We consider the polarization state of the Raman fields propagating along
$\hat{k}$ $||$ $\hat{x}$ being linear ($(\pi_{1})_{x}$) and circular
($(\sigma^{+}_{2})_{x}$). In the presence of the Raman fields, the electronic
spins align along the $x$-axis. However, due to the homogeneous magnetic field
along $\hat{z}$, the spins precess about $z$-axis on the x-y plane. Since the
Larmor precession rate (e.g., $\nu_{L}\sim$ 4.6 MHz) in our experiments is
typically much higher than the spin relaxation rate $(1/2\pi T_{2}\sim 0.15$
MHz), the spins lie on the x-y plane.

We further restrict our discussion for a system with ground hyperfine level
$F=1$ and excited hyperfine level $F^{\prime}=1$. In the following, we will
describe the dependence of Raman-driven power spectrum strength on various
polarization combinations of R1 and R2 fields presented in Fig. S3. Any spin
component of $F$ or $F^{\prime}$ on the x-y plane can be written as a linear
superposition of all possible spin components along $\hat{z}$, for an example
Tekin (2016),

$|m_{F}=1\rangle_{x}=\frac{1}{2}|m_{F}=-1\rangle_{z}+\frac{1}{\sqrt{2}}|m_{F}=0\rangle_{z}+\frac{1}{2}|m_{F}=1\rangle_{z}.$
(S16)

We can also decompose the polarization of the Raman fields in terms of their
electric fields in the following fashion Katz and Firstenberg (2019); Ben-Kish
and Romalis (2010):

$(\pi_{1})_{x}\equiv\hat{e}_{1y}=\frac{1}{\sqrt{2}}\left(\frac{\hat{e}_{1y}+i\hat{e}_{1x}}{\sqrt{2}}\right)+\frac{1}{\sqrt{2}}\left(\frac{\hat{e}_{1y}-i\hat{e}_{1x}}{\sqrt{2}}\right),$
(S17)

and,

$\displaystyle(\sigma_{2}^{+})_{x}\equiv\frac{\hat{e}_{2z}+i\hat{e}_{2y}}{\sqrt{2}}$
$\displaystyle=$
$\displaystyle\frac{\hat{e}_{2z}}{\sqrt{2}}+\frac{i}{2}\left(\frac{\hat{e}_{2y}+i\hat{e}_{2x}}{\sqrt{2}}\right)$
(S18) $\displaystyle+$
$\displaystyle\frac{i}{2}\left(\frac{\hat{e}_{2y}-i\hat{e}_{2x}}{\sqrt{2}}\right),$

where, $\hat{e}_{1i}$ or $\hat{e}_{2i}$ is the $i$th component of the
corresponding electric field with $i=x,y,z$.

In our experiment, we fix the frequency of R1 field (of $(\pi_{1})_{x}$
polarization) on-resonance to
$|m_{F}=-1\rangle_{z}\leftrightarrow|m_{F^{\prime}}=0\rangle_{z}$ transition,
and that of R2 field (of $(\sigma^{+}_{2})_{x}$ polarization) on-resonance to
$|m_{F}=0\rangle_{z}\leftrightarrow|m_{F^{\prime}}=0\rangle_{z}$ transition.
According to our decomposition in the Eq. S17 and Eq. S18, the allowed optical
transitions in the $\hat{z}$ basis can be shown in Fig. S6, where the field
$\hat{e}_{2z}$ couples
$|m_{F}=0\rangle_{z}\leftrightarrow|m_{F^{\prime}}=0\rangle_{z}$ transition
and the field $(\hat{e}_{1y}+i\hat{e}_{1x})/\sqrt{2}$ couples
$|m_{F}=-1\rangle_{z}\leftrightarrow|m_{F^{\prime}}=0\rangle_{z}$ transition.

Figure S6: Formation of a $\Lambda$ system in $\hat{z}$ basis within the
Zeeman states for $(\pi_{1})_{x}$ and $(\sigma^{+}_{2})_{x}$ combination of
polarization of the Raman fields. The generated $\Lambda$ system is shown by
black arrows.

The Fig. S6 shows that a $\Lambda$ type 3LS is formed (indicated by black
arrows) in $\hat{z}$ basis, and a coherence is built between the states
$|m_{F}=-1\rangle_{z}\leftrightarrow|m_{F}=0\rangle_{z}$. The coherence
between $|m_{F}=0\rangle_{z}$ and $|m_{F}=1\rangle_{z}$ can also be explained
in a similar fashion. This coherence in $\hat{z}$ basis in turn built a
coherence in $\hat{x}$ basis via Eq. S16, and detected by the off-resonant
probe laser. This case corresponds to $\theta=$ 45∘ and 225∘ in Fig. S3. The
other maxima at $\theta=$ 135∘ can be explained by considering the
$(\pi_{1})_{x}$ and $(\sigma^{-}_{2})_{x}$ combination of the polarization
states of the R1 and R2 field.

For $(\pi_{1})_{x}-(\pi_{2})_{x}$ combination of R1 and R2 field polarizations
($\theta=$ 90∘ and 180∘), no $\Lambda$ system is formed in $\hat{z}$ basis
within the ground hyperfine level Zeeman states. Therefore, no amplification
in the driven power spectrum has been observed.

However, it can be shown using Eq. S18 that when both the Raman fields are
$\sigma^{+}$ polarized, a double $\Lambda$ system is formed within the
consecutive Zeeman states in $F$-manifolds. In this case, the signal strength
is two times stronger than the case discussed in Fig. S6.

The Table 1 summarizes the Raman driven signal strength for various
combinations of the Raman fields’ polarizations.

Polarization of Raman fields | Comments on signal strength
---|---
$(\pi_{1})_{x}-(\pi_{2})_{x}$ and $(\sigma^{\pm}_{1})_{x}-(\sigma^{\mp}_{2})_{x}$ | No amplification, Intrinsic
$(\pi_{1})_{x}-(\sigma^{\pm}_{2})_{x}$ and $(\sigma^{\pm}_{1})_{x}-(\pi_{2})_{x}$ | $|\rho_{21}|^{2}$
$(\sigma^{\pm}_{1})_{x}-(\sigma^{\pm}_{2})_{x}$ | $2|\rho_{21}|^{2}$
Table 1: Dependence of Raman-driven power spectrum signal strength on the
Raman fields’ polarization combination.

We have experimentally verified the dependence of the on-resonance Faraday
rotation fluctuations signal strength on different polarization combinations
of the Raman fields. The experimental results are shown in Fig. S7 for
completeness.

Figure S7: The on-resonance signal strength of the Raman driven spectrum for
$(\sigma^{+}_{1})_{x}-(\sigma^{+}_{2})_{x}$ and
$(\pi_{1})_{x}-(\sigma^{+}_{2})_{x}$ combination of the Raman fields’
polarization with various driving intensities. The plots support the results
summarized in Table 1.

## References

  * James Simpson and Youngquist (2001) C. I. James Simpson, John Lane and R. Youngquist, Tech Rep: 1–3 (2001).
  * Sinitsyn and Pershin (2016) N. A. Sinitsyn and Y. V. Pershin, Rep. Prog. Phys. 79, 106501 (2016).
  * Crooker  _et al._ (2004) S. A. Crooker, D. G. Rickel, A. V. Balatsky, and D. L. Smith, Nature 431, 49 (2004).
  * Swar  _et al._ (2019) M. Swar, D. Roy, D. Dhanalakshmi, S. Chaudhuri, S. Roy, and H. Ramachandran,  _Measurements of Temporal Fluctuations of Magnetization in Alkali Vapor and Applications_ (Emerging Trends in Advanced Spectroscopy, ISBN: 9788770220828, 2019).
  * Tekin (2016) B. Tekin, Eur. J. Phys. 37, 035401 (2016).
  * Katz and Firstenberg (2019) O. Katz and O. Firstenberg, Commun. Phys. 2, 58 (2019).
  * Ben-Kish and Romalis (2010) A. Ben-Kish and M. V. Romalis, Phys. Rev. Lett. 105, 193601 (2010).


# Efficient Argument Classification with Compact Language Models and ChatGPT-4
Refinements

Marcin Pietron
Institute of Electronics
AGH University of Krakow
Kraków, Poland
<EMAIL_ADDRESS>
Rafał Olszowski
Center for Collective Intelligence
MIT Massachusetts Institute of Technology
Cambridge, MA, United States
Jakub Gomułka
Faculty of Humanities
AGH University of Krakow
Kraków, Poland

###### Abstract

Argument mining (AM) is defined as the task of automatically identifying and
extracting argumentative components (e.g. premises, claims, etc.) and
detecting the existing relations among them (i.e., support, attack, no
relations). Deep learning models enable us to analyze arguments more
efficiently than traditional methods and extract their semantics. This paper
presents comparative studies between a few deep learning-based models in
argument mining. The work concentrates on argument classification. The
research was done on a wide spectrum of datasets (Args.me, UKP, US2016). The
main novelty of this paper is the ensemble model which is based on BERT
architecture and ChatGPT-4 as fine tuning model. The presented results show
that BERT+ChatGPT-4 outperforms the rest of the models including other
Transformer-based and LSTM-based models. The observed improvement is, in most
cases, greater than 10The presented analysis can provide crucial insights into
how the models for argument classification should be further improved.
Additionally, it can help develop a prompt-based algorithm to eliminate
argument classification errors.

_Keywords_ Argument Mining $\cdot$ Deep learning $\cdot$ Large Language Models
$\cdot$ BERT

##  1 Introduction

Argument mining (AM) is a multidisciplinary research field encompassing
diverse areas such as logic and philosophy, language, rhetoric and law,
psychology, and computer science. The theory of argumentation and the use of
logical reasoning to justify claims and conclusions is an extensively studied
field, but the application of data science methods to automate these processes
is a relatively recent development. In nearly every field, the ability to
automatically extract arguments and their relationships from the input source
is of significant importance. Over the last decade, AM has become one of the
core studies within artificial intelligence [1, 2] due to its ability to
conjugate representational needs with user-related cognitive models and
computational models for automated reasoning [3]. As a subfield of Natural
Language Processing (NLP) and computational linguistics, AM focuses on
automatically identifying, extracting, and analyzing argumentative structures
within natural language texts, which includes recognizing core components of
arguments, such as claims and evidence [4]. Early AM research predominantly
focused on edited texts [5, 6, 7]. However, more recent investigations have
broadened their scope to encompass user-generated content found on various
platforms, including debate portals, as well as social media sites like
Facebook and Twitter [8]. This expansion demonstrates the growing
applicability and relevance of AM in understanding and analyzing diverse forms
of discourse across different online platforms.

In AM, the whole task can be decomposed into three main sub-tasks depending on
their argumentative complexity. First, the identification of argument
components consists in distinguishing argumentative propositions from non-
argumentative propositions. This allows segmentation of the input text into
arguments, making it possible to carry out the subsequent sub-tasks. Second,
the identification of clausal properties is the part of AM that focuses on
finding premises or conclusions among the argumentative propositions. Third,
the last sub-task is the identification of relational properties. Two
different argumentative propositions are considered at a time, and the main
objective is to identify which type of relation links both propositions. This
task can be solved by a classification model.

The Transformer-based model architecture and its subsequent pre-training
approaches have been a turning point in the NLP research area (e.g. BERT,
Google T5). Conversely, the emergence of Chat-GPT, a generative pre-trained
transformer language model developed by OpenAI, has presented new
opportunities for argument mining. Introduced in November 2022, Chat-GPT is
based on the GPT architecture and refined via comprehensive training on large-
scale datasets. This model can comprehend context, produce coherent and
contextually appropriate responses, and participate in meaningful dialogues
with users.

This research paper aims to explore the cutting-edge capabilities of the
Transformer-based models in the field of Argument mining. Given the rising
prominence of Large Language Models (LLMs) such as GPT-4, our research
introduces the concept of the ’compact language model’ (CLM) as a distinct
category. Contrary to LLMs like GPT-4, Microsoft Copilot, or Google’s PaLM,
CLMs have fewer parameters, which makes them faster to run and easier to
deploy in resource-constrained environments. Furthermore, CLMs can be
precisely fine-tuned for specialized tasks, exemplified by argument
classification in our study. A notable benefit of CLMs is their optimization
for minimized computational and energy demands. In the presented approach, the
LLM plays an ’expert’ role and is engaged when dealing with a subset of
arguments classified with low certainty. Our research demonstrates the extent
to which this combined model can enhance the outcomes of argument
classification.

We formulated the following Research Questions:

RQ1. What efficiency can the compact language model achieve across different
argument classification benchmarks?

RQ2. How Large Language Model can improve the compact model efficiency in
argument classification?

RQ3. What efficiency can be achieved by a hybrid model, which consists of the
compact language model as a base model and the Large Language model as its
fine-tuning model?

##  2 Related works

Argument classification (AC) is a specific subtask within AM. It focuses on
classifying the identified components of an argument into predefined
categories, such as distinguishing between claims and premises or identifying
the stance of an argumentative component (supporting or opposing). Argument
classification can also involve determining the type of argument or its
quality (e.g., strong vs. weak argument). This task is crucial for building
the structured representation of arguments, as it helps in understanding the
role of each component within the argument and how different components
interact [3].

Various data science techniques based on natural language processing have
shown efficacy in AC and AM. In the early works, argument structures were
mapped through trees or similar hierarchical structures, facilitating
computation with existing tree-parsing methods. However, such idealized
structures seldom match the complexity of arguments encountered in practice.
More recently, the focus shifted towards utilizing non-hierarchical
frameworks. Prior to the advent of BERT and Transformer-based methodologies,
the field of AM heavily relied on Support Vector Machines (SVMs) and neural
networks for their pattern recognition and classification strengths to
pinpoint and dissect argumentative text structures. Techniques like Recurrent
Neural Networks, Convolutional Neural Networks, and Long Short-Term Memory
units were crucial in incorporating contextual data into machine learning
decision processes [9]. The introduction of the first non-hierarchical model
for AM by Niculae et al. [10], using factor graphs alongside structured SVMs
and Bidirectional LSTM, marked a significant shift. SVMs, known for their
binary classification prowess, fit well with AM tasks such as discerning
argumentative elements (e.g., distinguishing claims from non-claims). Galassi
et al. [11] then applied LSTMs with residual connections to better predict
argument components’ interrelations. Subsequently, Morio et al. [12] proposed
using Task-Specific Parameterization and Proposition-Level Biaffine Attention
for improved prediction of complex argument structures.

While SVMs and neural networks significantly contributed to the development of
AM, they had limitations, such as the need for extensive feature engineering
and difficulties in capturing long-distance dependencies in text. The
emergence of BERT and other Transformer-based models revolutionized the
landscape of AM. Introduced by Vaswani et al. [13], the transformer model,
with its self-attention mechanism, addressed the computational and memory
efficiency issues faced by RNNs. The utility of transformers, particularly
BERT [14], in AM has spurred extensive research, with fine-tuning these models
showing significant improvements across diverse tasks. Recent research has
applied Transformer-based models to AM, with Reimers et al. [15] improving
argument classification and clustering using BERT and ELMo, and Chakrabarty et
al. [16] advancing argument component and relation identification in
persuasive texts. Further, Chen [17] applies BERT for mapping argument
structures as directed acyclic graphs, while Ruiz-Dolz et al. [18] evaluate
the performance of various Transformer-based models across domain-specific
datasets to identify the most versatile model.

This paper proposes a novel AM methodology that leverages the compact
transformer-based models (BERT) with ChatGPT architecture, for automatic
argument classification. Distinguishing our work from predecessors, we engage
with a wider array of datasets and tasks, and discuss methods to overcome
challenges and boost AM efficiency through BERT usage with ChatGPT-4 based
refinement.

##  3 Datasets

One of the primary contributions of our approach is the broad range of
argument datasets considered. The previous studies by Ajjour et al. [19], Bar-
Haim et al. [20], Boltuzic and Snajder [21], Cabrio and Villata [22], Carstens
and Toni [23], Park and Cardie [4], Stab et al. [7], Visser et al. [25] focus
on a limited number of datasets. In each of these studies, only one dataset
was examined.

In our research, we decided to conduct a comparative study using three
different datasets containing arguments corpora: the US2016 corpus, the Moral
Maze multi-domain (UKP) corpus, and the Args.me corpus. All these datasets
have gained a reputation in argument mining research undertaken in recent
years, but have not yet been studied comparatively and using the latest models
such as Google T5 and BERT, as well as GPT and OpenLlama technologies.

First of the processed datasets, the US2016 is an argument annotated corpus of
the electoral debate carried out in 2016 in the United States. It contains
both, transcriptions of the different rounds of TV debate, and discussions
from the Reddit forums as detailed in Visser et al. [25]. Four annotators,
trained in the use of Inference Anchoring Theory [26], annotated the US2016
corpus with Argumentative Discourse Units (ADUs). The ADUs used in this
annotation are relations occurring between propositions, specifically: (1) The
annotations of Inference (RA), reflecting that a proposition is meant to
supply a reason for accepting another proposition; (2) The annotations of
Conflict (CA), reflecting that a proposition is meant to be incompatible with
another proposition or relation; (3) The annotations of Rephrases (MA),
reflecting that a proposition is meant to be a reformulation of another
proposition; and (4) the annotations of No Relation (NO) mean that there is no
relationship between the propositions. Table 1 presents the class distribution
of the processed US2016 corpus. This dataset has a total of approximately 12.5
thousand arguments.

| US 2016 | train | test
---|---|---|---
RA | 2744 | 2470 | 274
CA | 888 | 800 | 88
MA | 705 | 635 | 70
NO | 8055 | 7250 | 805
Total | 12392 | 11243 | 1149
Table 1: US2016 database statistics

The second dataset used in our study was UKP [24]. It contains a corpus of
arguments from online comments on eight controversial issues: abortion,
cloning, death penalty, gun control, minimum wage, nuclear energy, school
uniforms, marijuana legalization. The corpus includes over 25,000 instances.
In Table 2 and 3 the statistics of UKP datasets are shown. The sentences were
annotated by 7 people working independently, recruited through the Amazon
Mechanical Turk (AMT) crowdsourcing platform. For each classification, it was
necessary to achieve agreement, as measured by Cohen’s kappa ($\kappa$), at
the level of 0.723; this exceeds the commonly used threshold of 0.7 for
assuming the results are reliable [27]. The labels used in the classification
were: (1) supporting argument; (2) opposing argument; and (3) non-argument.

| abortion | cloning | death penalty | marijuana legalization
---|---|---|---|---
NON | 2282 | 1472 | 1918 | 1160
PRO | 634 | 702 | 424 | 535
CON | 766 | 825 | 1036 | 574
Table 2: UKP database statistics part 1 | gun control | minimum wage | nuclear energy | school uniforms
---|---|---|---|---
NON | 1846 | 1323 | 2051 | 1734
PRO | 775 | 564 | 591 | 545
CON | 650 | 541 | 831 | 729
Table 3: UKP database statistics part 2

The last dataset used in our study was Args.me corpus (version 1.0, cleaned),
provided by Ajjour et al. [19]. The distribution of this corpus is presented
in Table 4. It consists of arguments crawled from four debate portals in the
middle of 2019. The debate portals are Debatewise, IDebate.org, Debatepedia,
and Debate.org. The arguments were extracted using heuristics that are
designed for each debate portal. Idebate.org, Debatepedia, and Debatewise are
the sub-datasets used in our simulations. These three sub-datasets consist of
48 798 arguments. The Debatepedia has the highest number of arguments. In
IDebate.org the disproportion between PRO and CON arguments is the lowest, and
the highest disproportion appears in Debatepedia. Args.me is the newest and
the most extensive dataset among all datasets utilized in our study. Table 4
presents the statistics of the Args.me dataset. The annotations used in this
database include conclusions and premises, which are divided into PRO premises
(arguments supporting the conclusion) and CON premises (arguments against the
conclusion).

| idebate.org | debatepedia | debatewise
---|---|---|---
conclusions | 5011 | 10314 | 5992
premises | 13248 | 21197 | 14353
PRO premises | 6701 | 15791 | 8514
CON premises | 6547 | 5406 | 5839
Table 4: Args.me database statistics

##  4 Method

###  4.1 Fine tuning the language model

The compact language models which were used in following experiments are
DistilBERT and BERT models. The classifiers were built by adding fully
connected layers at the output of these to transformer based models.

Each Transformer based language model consists of the list of Transformers
blocks [28]:

$T_{\Theta}(X)={t_{\theta_{L}}(t_{\theta_{L-1}}...(t_{\theta_{0}}(X)))}$ (1)

the $\Theta$ is a list:

$\Theta=\\{\theta_{0},\theta_{1},...,\theta_{L}\\}$ (2)

is a list of parameters $\theta_{i}$ for each Transformer block. Each tensor
$\theta_{i}$ is a set of Transformer block FC layers attention weights. The
$T_{\Theta}$ gives an output with the same number of tokens as in input $X$
(generates embedding for each token). The classifiers in the presented
experiments are defined as follows:

$C_{\Theta_{c}}(X)=f_{\theta_{2}}(f_{\theta_{1}}(T_{\Theta}(X)))$ (3)

The two fully connected layers are added at the top of the Transformer blocks.
The output size of the last fully connected layer is equal to the number of
classes to recognize.

##  5 ChatGPT-4 based refinement

The next stage of our research entailed the application of the ChatGPT model
developed by OpenAI. The primary objective was to ascertain the feasibility of
enhancing the efficacy of argument classification through the utilization of
the GPT-4 model, particularly in instances where the BERT model exhibited
limitations.

The GPT-4 refinement algorithm is presented in Algorithm 1. The algorithm
involves creating a classifier based on a pre-trained language model as
described in the previous chapter (line 1). Then the model created in this way
is fine-tuned on the dataset (lines from 2 to 6). In the next step, the
threshold is calculated based on the uncertainty of the model’s response
(lines from 8 to 11). The arguments with the least certain answer are then
routed to the large language model (lines 13 and 14). The remaining arguments
are processed by a small language model (line 16).

0: $T_{\Theta}$, $LLM$, $T$-train data, $V$-validation data,
$\gamma$-threshold

1: $C_{\Theta_{c}}$ $\leftarrow$ $f_{\theta_{2}}(f_{\theta_{1}}(T_{\Theta})$
//add fully connected layers

2: for $b$ in $T$ do

3: $L_{C}$ $\leftarrow$ Loss($C_{\Theta_{c}}(X_{b})$, $Y_{b}$)

4: $g$ $\leftarrow$ grad($L_{C}$)

5: update $\Theta_{c}$

6: end for

7: $\Lambda$ $\leftarrow$ $\emptyset$

8: for $b$ in $T$ do

9: $\Lambda\leftarrow\Lambda\cup|C_{\Theta_{c}}(X_{b})$-$Y_{b}|$

10: end for

11: $\gamma$ $\leftarrow$ $\Lambda[kth]$

12: for $b$ in $V$ do

13: if $|C_{\Theta_{c}}(X_{b})$-$Y_{b}|>\gamma$ then

14: $Y\leftarrow LLM(X_{b})$

15: else

16: $Y\leftarrow C_{\Theta_{c}}(X_{b})$

17: end if

18: end for

Algorithm 1 Small language model with ChatGPT-4 refinement

The subset of arguments from all three databases was evaluated with ChatGPT-4
when running Algorithm 1. In each case, the same simple input-output prompting
template was used (theses and arguments were inserted into blank spaces):

> This is a thesis “…”. And this is a claim “…”. Is the claim an argument for
> or contra the thesis? Write a one-sentence answer.

The responses provided by ChatGPT-4 were predominantly accurate, enhancing the
performance quality of the BERT model. Nonetheless, ChatGPT-4 was not exempt
from errors, the analysis of which may yield intriguing insights. The analysis
of erroneous outputs indicates that the model, in some cases, struggles with
processing logical connectives. In one such instance, when asked to elaborate
on how it arrived at an inaccurate answer, ChatGPT-4 responded as follows: "I
came to this conclusion because the thesis statement argues that Tunisia
should not rely on tourism for economic growth, while the claim argues that
tourism causes pollution, which suggests that relying on tourism may have
negative consequences, supporting the argument against the thesis".
Furthermore, ChatGPT-4 fails to recognize sarcasm. In one example, a negative
argument was presented with a tone of irony, which confused the model and led
to the incorrect classification of the argument as positive. There were also
cases in which the model openly contradicted itself when questioned after
providing an incorrect answer.

Another type of error was associated with a failure to understand arguments
that place the issue in a context different from the most commonly encountered
one. For example, in a debate about cloning where the assertion was ’Cloning
should be allowed,’ ChatGPT-4 did not grasp an argument concerning the
positive impact of cloning on “creating plants that offer better nutritional
value”. Consequently, given that – as it articulated – “debates on cloning
typically focus on the ethical, legal, or social implications of cloning
humans or animals”, ChatGPT-4 claimed that this statement did not contain any
argument.

A frequently recurring problem is also the incorrect classification of
arguments expressed in sentences containing multiple negations. For instance,
the sentence ’The fact that some states or countries which do not use the
death penalty have lower murder rates than jurisdictions which do is not
evidence of the failure of deterrence’ was classified as an argument against
the death penalty, despite the explicit clarification that it is ’not
evidence’. Moreover, there are instances where ChatGPT-4 contradicts itself
when justifying an incorrect response, claiming, for example, that ’the
statement implicitly argues against the proposition that "Gun access should
not be limited" (…) Therefore, it is an argument against the proposition that
gun access should be limited.

Finally, we observed that some datasets, in particular UKP, contain a
noticeable group (approximately 10%) of incorrect or problematic
classifications made by human evaluators, while ChatGPT’s evaluation of these
cases seems to be more consistent with reality.

##  6 Results and discussion

| idebate.org | debatepedia | debatewise | top1 | F1
---|---|---|---|---|---
DistilBERT | 60.63 | 72.64 | 60.02 | 65.1 | 68.9
BERT | 73.76 | 83.1 | 69.2 | 76.22 | 71.5
BERT + ChatGPT-4 | 84.82 | 91.41 | 76.12 | 85.87 | 87.2
Table 5: Top1 and F1 Accuracy for Args.me - PRO and CON arguments recognition without conclusion. | idebate.org | debatepedia | debatewise | top1 | F1
---|---|---|---|---|---
DistilBERT | 60.48 | 72.83 | 60.02 | 65.1 | 69.3
BERT | 83.1 | 85.48 | 84.3 | 84.22 | 85.3
BERT + ChatGPT-4 | 88.91 | 92.92 | 88.51 | 89.6 | 91.3
Akiki et al. [30] | - | - | - | 75.5 | -
Table 6: Top1 and F1 Accuracy for Argsme - PRO and CON arguments recognition with conclusion | US2016
---|---
LSTM [28] | 26.0
DistilBERT [28] | 55.0
BERT [28] | 62.0
BERT + ChatGPT-4 | 72.5
Table 7: F1 metric for US2016 (four labels) | abortion | cloning | death penalty | marijuana legalization
---|---|---|---|---
DistilBERT | 60.0 | 51.33 | 57.39 | 54.38
BERT | 73.51 | 70.66 | 65.68 | 61.40
BERT + ChatGPT-4 | 77.18 | 74.19 | 74.19 | 64.47
Table 8: Top1 Accuracy for UKP dataset, part 2 (three labels). | gun control | minimum wage | nuclear energy | school uniforms
---|---|---|---|---
DistilBERT | 60.36 | 59.01 | 63.79 | 52.98
BERT | 70.12 | 68.03 | 77.01 | 76.15
BERT + ChatGPT-4 | 75.02 | 72.79 | 80.86 | 79.95
Table 9: Top1 Accuracy for UKP dataset, part 2 (three labels). model | UKP
---|---
LSTM [24] | 42.85
DistilBERT | 54.2
BERT | 57.7
BERT + ChatGPT-4 | 68.5
Table 10: Mean F1 score for UKP (three labels)

Table 5 and Table 6 show the results of all conducted evaluations for the
Args.me dataset. Trainings for the DistilBERT and BERT models were carried out
on three subsets: debatepedia, idebate and debatewise. One experiment involved
providing only the argument text in training and evaluation as input. However,
in the second experiment, an argument and a thesis were given as input. As can
be observed, better results were achieved in the second case. For all three
subsets the improvement in both top1 and F1 metrics can be observed (the
highest improvements were achieved for idebate.org and debatewise). The best
results were obtained for the debatepedia dataset (91.41 and 92.92 top1. RQ3).
Among all models (BERT, DistilBERT and [30]) BERT+ChatGPT-4 is the best and
outperforms other models (by 16% in F1 for the first case and by 6% for the
second one, RQ2). The $\gamma$ parameter for Args.me dataset was set to value
that about 20% of arguments with the highest uncertainty were delegated to
ChatGPT-4.

Table 7 shows the results for the US21016 set. The model for this set
classifies the input data into four possible classes. The results for the BERT
model (the F1 metric is 62.0) are definitely better than for the DistilBERT
(55.0) and LSTM (26.0) models (RQ1). The reported results are the same as
presented in [28]. In the case of the BERT+ChatGPT-4 mode, the $\gamma$
parameter from Algorithm 1 was set so that 25% of the data with the lowest
confidence was delegated from the BERT model to the ChatGPT-4 model. The
BERT+ChatGPT-4 again achieves the best results (72.5%, RQ3). The F1 metric is
improved by about 10% (RQ2).

Table 8 and Table 9 present the results obtained for the UKP dataset. The top1
metric for each thematic subset (8 subsets) is presented. As can be seen, the
results are the best for the largest subsets (especially abortion and nuclear
energy). This is due to the fact that the model has a larger number of
training courses, which allows it to better perform the fine-tuning process.
As expected, BERT achieves better results than DistilBERT in all groups.
However, the BERT+ChatGPT-4 hybrid model outperforms both of them. It should
be mentioned that during the analysis of the model’s response, the cases were
noticed in which the labels of some arguments were incorrect or controversial.
This may also result in the lowest accuracy of UKP across all datasets. The
second factor is the amount of data in this dataset.

Finally, Table 10 shows the aggregated results of the UKP set for the F1
metric. The correlation of results between models is maintained. The
BERT+ChatGPT-4 outperforms all presented models (by at least 10%, RQ2) and
achieves 68.5%(RQ3). The BERT and DistilBERT outperform again LSTM-based model
[24] as in the case of US2016, RQ1.

One of the main goals of the work was to show the possibility of increasing
the results of a compact language model by adding a large language model as an
expert in selected cases (in parallel minimizing the usage of LLM). Thanks to
this, you can scale the inference time (model response) and reduce the use of
a large language model.

All our experiments were executed on a workstation equipped with Nvidia Tesla
V100-SXM2-32GB GPUs.

##  7 Conclusions and future works

The presented results show that compact language models such as BERT and
Google T5 do not achieve fully satisfactory results in argument
classification. Their advantage, however, is their speed of fine-tuning.
Limited access to large models and their computational complexity makes it
difficult to evaluate the collections quickly. Our study shows that the
results of small models can be easily and quickly improved by using large
models for a selected subset of arguments. Our research also shows that
argument-mining datasets have minor shortcomings in the form of incorrect
labeling.

Future work will focus on large language models. In addition to the ChatGPT-4
model, open-source models will be explored, including LLAMA-2. In order to
conduct further research in GPT-based refinement, it will be necessary to
utilize more advanced prompting techniques, such as the Tree of Thoughts,
which recent reports suggest can significantly enhance ChatGPT’s reasoning
capabilities [29].

## References

  * [1] Schaefer, R., Stede M. GerCCT: An Annotated Corpus for Mining Arguments in German Tweets on Climate Change. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, 6121–613020, 2022, Marseille, France. European Language Resources Association
  * [2] Lawrence, J., Reed, C. Argument Mining: A Survey. Computational Linguistics 45(4), 765–818, 2019.
  * [3] Lippi, M., Torroni, P. Argumentation Mining: State of the Art and Emerging Trends. ACM Trans. Internet Technol. 16(2), Article 10, 2016, 25 pages. https://doi.org/10.1145/2850417.
  * [4] Park, J., Cardie, C. A corpus of eRulemaking user comments for measuring evaluability of arguments. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation, LREC 2018.
  * [5] Moens, M. F., Boiy, E., Palau, R. M., Reed, C. Automatic detection of arguments in legal texts. In Proceedings of the 11th International Conference on Artificial Intelligence and Law, pages 225–230, Stanford, CA, 2007.
  * [6] Levy, R., Bilu, Y., Hershcovich, D., Aharoni, E., Slonim, N. Context dependent claim detection. In Proceedings of the 25th International Conference on Computational Linguistics, pages 1489–1500, Dublin, 2014.
  * [7] Stab, C., Gurevych, I. Annotating argument components and relations in persuasive essays. In Proceedings of the 25th International Conference on Computational Linguistics, pages 1501–1510, Dublin, 2014.
  * [8] Dusmanu, M., Cabrio, E., Villata, S. Argument mining on Twitter: Arguments, facts and sources. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2317–2322, Copenhagen, 2017.
  * [9] Srivastava, P., Bhatnagar, P., Goel, A. Argument Mining using BERT and Self-Attention based Embeddings. 2022 4th International Conference on Advances in Computing, Communication Control and Networking (ICAC3N) ,2022. https://doi.org/10.48550/arXiv.2302.13906.
  * [10] Niculae, V., Park, J., Cardie, C. Argument mining with structured svms and rnns. arXiv preprint arXiv:1704.0686920, 2017.
  * [11] Galassi, A., Lippi, M., Torroni, P. Argumentative link prediction using residual networks and multi-objective learning. In: Proceedings of the 5th Workshop on Argument Mining, pp. 1–10, 2018.
  * [12] Morio, G., Ozaki, H., Morishita, T., Koreeda, Y., Yanai, K. Towards better non-tree argument mining: Proposition-level biaffine parsing with task-specific parameterization. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 3259–3266 2020.
  * [13] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I. Attention is all you need. Advances in neural information processing systems 30, 2017.
  * [14] Devlin, J., Chang, M.-W., Lee, K., Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June, 2019.
  * [15] Reimers, N., Schiller, B., Beck, T., Daxenberger, J., Stab, C., Gurevych, I. Classification and clustering of arguments with contextualized word embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 567–578, Florence, Italy, July, 2019.
  * [16] Chakrabarty, T., Hidey, C., Muresan, S., McKeown, K., Hwang, A. AMPERSAND: Argument mining for PERSuAsive oNline discussions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2933–2943, Hong Kong, China, November, 2019.
  * [17] Chen, T. BERT Argues: How Attention Informs Argument Mining, 2021. Honors Theses. 1589. https://scholarship.richmond.edu/honors-theses/1589.
  * [18] Ruiz-Dolz, R., Heras, S., Alemany, J., García-Fornes, A. Transformer-Based Models for Automatic Identification of Argument Relations: A Cross-Domain Evaluation. arXiv. https://doi.org/10.48550/arXiv.2011.13187, 2020.
  * [19] Ajjour, Y., Wachsmuth, H., Kiesel, J., Potthast, M., Hagen, M., Stein, B. Data Acquisition for Argument Search: The args.me Corpus. In: Benzmüller, C., Stuckenschmidt, H. Advances in Artificial Intelligence. KI 2019. Lecture Notes in Computer Science, vol 11793. Springer, Cham. ,2019.
  * [20] Bar-Haim, R., Bhattacharya, I., Dinuzzo, F., Saha, A., Slonim, N. Stance classification of context-dependent claims. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 251–261, 2017.
  * [21] Boltuzic, F., Snajder, J. Back up your stance: Recognizing arguments in online discussions. In Proceedings of the First Workshop on Argumentation Mining, pages 49–58, 2014.
  * [22] Cabrio, E., Villata, S. Node. A benchmark of natural language arguments. In Computational Models of Argument - Proceedings of COMMA 2014, Atholl Palace Hotel, Scottish Highlands, UK, September 9-12, 2014, pages 449–450, 2014.
  * [23] Carstens, L., Toni, F. Using argumentation to improve classification in natural language problems. ACM Transactions on Internet Technology, 17(3):30:1–30:23, 2017.
  * [24] Stab, C., Miller, T., Schiller, B., Rai, P., Gurevych, I. Cross-topic argument mining from heterogeneous sources. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3664–3674, Brussels, Belgium: Association for Computational Linguistics ,2018.
  * [25] Visser, J., Lawrence, J., Wagemans, J., Reed, C. An annotated corpus of argument schemes in US election debates. In Proceedings of the 9th Conference of the International Society for the Study of Argumentation (ISSA), 3-6 July 2018, pp. 1101-1111.
  * [26] Budzynska, K., Reed, C. Speech acts of argumentation: inference anchors and peripheral cues in dialogue. In Proceedings of the 10th AAAI Conference on Computational Models of Natural Argument (AAAIWS’11-10), AAAI Press, pp. 3–10, 2011.
  * [27] Carletta, J.: Assessing agreement on classification tasks: The kappa statistic. Computational Linguistics 22(2), 249–254, 1996.
  * [28] Li, K. et al. An Empirical Study of Transformer-Based Neural Language Model Adaptation. In ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain, 2020, pp. 7934-7938, doi: 10.1109/ICASSP40776.2020.9053399.
  * [29] Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., Narasimhan, K. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. ArXiv 2023. https://arxiv.org/abs/2305.10601.
  * [30] Akiki, C., Potthast, M. Exploring Argument Retrieval with Transformers. CLEF 2020. https://ceur-ws.org/Vol-2696.