# Influence of device geometry and imperfections on the interpretation of
transverse magnetic focusing experiments

Yik K. Lee<EMAIL_ADDRESS>Theoretical, Computational, and
Quantum Physics, School of Science, RMIT University, Melbourne, Australia  ARC
Centre of Excellence in Future Low-Energy Electronics Technologies, RMIT
University, Melbourne, Australia  Jackson S. Smith  Theoretical,
Computational, and Quantum Physics, School of Science, RMIT University,
Melbourne, Australia  ARC Centre of Excellence in Future Low-Energy
Electronics Technologies, RMIT University, Melbourne, Australia  Jared H. Cole
<EMAIL_ADDRESS>Theoretical, Computational, and Quantum Physics, School
of Science, RMIT University, Melbourne, Australia  ARC Centre of Excellence in
Future Low-Energy Electronics Technologies, RMIT University, Melbourne,
Australia

###### Abstract

Spatially separating electrons of different spins and efficiently generating
spin currents are crucial steps towards building practical spintronics
devices. Transverse magnetic focusing is a potential technique to accomplish
both those tasks. In a material where there is significant Rashba spin–orbit
interaction, electrons of different spins will traverse different paths in the
presence of an external magnetic field. Experiments have demonstrated the
viability of this technique by measuring conductance spectra that indicate the
separation of spin–up and spin–down electrons. However the effect that the
geometry of the leads has on these measurements is not well understood. By
simulating an InGaAs-based transverse magnetic focusing device, we show that
the resolution of features in the conductance spectra is affected by the
shape, separation and width of the leads. Furthermore, the number of subbands
occupied by the electrons in the leads affects the ratio between the
amplitudes of the spin–split peaks in the spectra. We simulated devices with
random onsite potentials and observed that transverse magnetic focusing
devices are sensitive to disorder. Ultimately we show that careful choice and
characterisation of device geometry are crucial for correctly interpreting the
results of transverse magnetic focusing experiments.

##  I Introduction

Unlike conventional electronics, spintronics manipulates the spin degree of
freedom of an electron to transfer and store information Xia  _et al._ (2012).
One of the challenges in spintronics is to generate a spin current efficiently
Hirohata  _et al._ (2020), and another is to spatially separate spin currents
Lo  _et al._ (2017). Transverse magnetic focusing (TMF) offers a solution to
both these challenges by exploiting the spin–momentum coupling induced by the
Rashba spin–orbit interaction Bychkov and Rashba (1984). Electrons injected
into a two-dimensional electron gas (2DEG) with a significant Rashba
spin–orbit coupling are separated by their spins and travel in different
cyclotron paths as shown in Fig. 1a when a perpendicular magnetic field is
applied to the device. Measuring the conductance between the points of
electron injection and collection, while varying the strength of this magnetic
field, gives a spectrum like the one shown in Fig. 1b, with peaks occurring at
magnetic field strengths that focus an electron at the point of collection.

TMF in a metal was first demonstrated by Tsoi in 1974 using a Bi single
crystal Tsoi (1974). TMF in a 2DEG was shown by van Houten et al. in 1988
using a GaAs/AlGaAs heterostructure at temperatures ranging from 0.05 K to 7 K
van Houten  _et al._ (1989). In 1992, Heremans et al. observed ballistic hole
transport by performing a TMF experiment in a two-dimensional hole gas (2DHG)
using a GaAs/AlGaAs heterostructure Heremans  _et al._ (1992, 1994). More
recently, TMF has been shown to be a viable spin filter in a wide range of
experiments Potok  _et al._ (2002); Folk  _et al._ (2003); Rokhinson  _et al._
(2004); Dedigama  _et al._ (2006); Heremans  _et al._ (2007); Rendell  _et
al._ (2015); Lo  _et al._ (2017); Yan  _et al._ (2017, 2018); Tsoi  _et al._
(1999); Bozhko (2014). There are also indications that TMF need not be limited
to semiconductor heterostructures, as Taychatanapat et al. demonstrated it can
be performed in graphene devices up to a temperature of 300 K Taychatanapat
_et al._ (2013).

In a TMF experiment, spatial separation of the spin currents is indicated by
the presence of a pair of spin–split peaks in the conductance spectrum that
each correspond to a different spin, such as the peaks at $B_{{z}}=0.061$ T
and 0.089 T in Fig. 1b. In materials without Rashba spin–orbit coupling, the
spin currents would not spatially separate, resulting in a single higher peak,
similar to the peak at $B_{{z}}=0.151$ T in Fig. 1b. The spin–split peaks do
not have equal amplitudes, which has previously been attributed to spin
polarisation in the injected electrons Rokhinson  _et al._ (2004); Reynoso
_et al._ (2007). The wave-like nature of electrons suggests that there should
be wave interference, which can be observed as interference fringes in the
spectrum Usaj and Balseiro (2004); Bladwell and Sushkov (2017). Although
transport in these devices is considered ballistic, there are many factors
that could affect measurements in experiments, such as temperature Potok  _et
al._ (2002); Heremans  _et al._ (2007); Yan  _et al._ (2017) and disorder. The
effects of physical device parameters such as the shape and width of the leads
used remain unclear. Square-shaped Lo  _et al._ (2017), trumpet-shaped
Dedigama  _et al._ (2006) and funnel-shaped Potok  _et al._ (2002) leads have
all been used in experiments.

In this paper we focus on computing TMF measurements in an InGaAs-based 2DEG
with Rashba spin–orbit coupling with realistic device geometries. We simulate
TMF using wave function matching as implemented through the program KWANT
Groth  _et al._ (2014) using a finite difference method with a square grid
discretisation scheme. A brief introduction of TMF is given in Section II.1.
We then present our numerical approach in Section II.2. The effects that the
shape, separation distance and width of the leads have on the TMF response are
presented in Section III.1. The effects of varying the number of subbands
occupied by the injected electrons are presented in Section III.2. We simulate
the leads as quantum point contacts (QPCs) in Section III.3 and show that
effects seen in previous sections are linked in this more realistic model. We
investigate the influence that disorder in the surface potential of a 2DEG has
on the TMF results in Section III.4 and conclude in Section IV.

##  II Methods

###  II.1 Semi-classical model

A typical TMF experiment consists of a two-dimensional focusing area connected
to two leads which serve as injector and collector for electrons (see Fig.
1a). An out-of-plane magnetic field $B_{{z}}$ is applied such that an electron
injected into the system experiences a Lorentz force which causes it to move
in a cyclotron orbit with a Larmor radius $R_{\mathrm{L}}$ given by van Houten
_et al._ (1989):

$R_{\mathrm{L}}=\frac{\hbar k}{|e|B_{{z}}}$ (1)

where $\hbar$ is the reduced Planck constant, $k$ is the magnitude of the
wavevector of the electron, and $e$ is the charge of an electron.

Figure 1:  (a) Schematic of a typical transverse magnetic focusing device.
Electrons injected at lead $\mathrm{L}_{1}$ are focused and collected at lead
$\textrm{L}_{2}$ by applying an out-of-plane magnetic field $B_{{z}}$. If the
Rashba effect is present, spin–up (red) and spin–down (blue) electrons have
different trajectories due to spin–orbit interaction. Distance between the
leads ($D$) is defined from the middle of each lead. (b) Example conductance
spectrum for a transverse magnetic focusing device showing conductance as a
function of the out-of-plane magnetic field calculated using the method
described in Section II.2. The solid line (purple) shows the total
conductance, the dashed (red) and dotted (blue) lines show the spin projected
conductance for spin–up and spin–down in $x$ respectively.

At particular values of $B_{{z}}$, where the distance between the leads
coincides with integer multiples of $2R_{\mathrm{L}}$, the probability of
transmission for an electron between these two leads is nonzero as long as the
reflections from the wall are specular. This is observed as sharp peaks in the
conductance (as shown in Fig. 1b). Experiments usually measure current and
resistance, which are proportional to the conductance spectra we calculate in
this paper.

When a strong Rashba spin–orbit interaction is present in the system, spin–up
and spin–down electrons entering the 2DEG with the same energy have different
crystal momenta. Consequently, spin–up and spin–down electrons have orbits
with different Larmor radii, which results in the spatial separation of spin
currents as indicated in Fig. 1a and the presence of spin–split peaks in the
conductance spectrum as seen in Fig. 1b. The Rashba interaction is described
by the Hamiltonian Bychkov and Rashba (1984):

$H_{\mathrm{R}}=\alpha\left(\boldsymbol{\sigma}\times\boldsymbol{k}\right)\cdot\hat{z}$
(2)

where $\alpha$ is the Rashba coupling parameter, $\boldsymbol{\sigma}$ is the
Pauli vector, $\boldsymbol{k}$ is the wavevector of the electron, and
$\hat{z}$ is the unit vector normal to the plane of the 2DEG.

The dispersion relation of a 2DEG with Rashba interaction is then given by
Bychkov and Rashba (1984):

$\displaystyle E_{\pm}$ $\displaystyle=\frac{\left(\hbar
k\right)^{2}}{2m^{*}}\pm\alpha k$ $\displaystyle E_{\pm}$
$\displaystyle=\frac{1}{2m^{*}}\left[\left(\hbar
k\pm\frac{m^{*}\alpha}{\hbar}\right)^{2}-\left(\frac{m^{*}\alpha}{\hbar}\right)^{2}\right]$
(3)

where $E_{\pm}$ are the energies of the spin–up($+$)/down($-$) electrons,
$k=|\boldsymbol{k}|$, and $m^{*}$ is the effective mass of the electron.

The magnetic field strengths, $B^{+}_{{z}}$($B^{-}_{{z}}$), required to focus
spin–up(spin–down) electrons of the same Fermi energy, $E_{\mathrm{F}}$, into
orbits of the same Larmor radius will be Rokhinson  _et al._ (2004):

$\displaystyle B^{\pm}_{{z}}$
$\displaystyle=\frac{1}{|e|R_{\mathrm{L}}}\left(\sqrt{2m^{*}E_{\mathrm{F}}}\mp\frac{m^{*}\alpha}{\hbar}\right)$
(4)

The difference between these two magnetic field strengths is the separation
between the spin–up and spin–down peak pairs in the conductance spectrum. This
semi-classical approximation ignores quantum effects such as the Berry
curvature correction to the velocity of the electron Xiao  _et al._ (2010).
However, for the parameter regime we consider, such corrections are several
orders of magnitude smaller than the velocities. Thus we do not consider such
corrections in this paper.

It is also worth noting that classically the Larmor radius (Eq. 1) is written
in terms of the electron’s tangential velocity. In a semi-classical treatment,
one might therefore be tempted to substitute the group velocity
(${v_{\mathrm{g}}=\frac{1}{\hbar}\frac{\partial E}{\partial k}}$) for the
tangential velocity, instead of substituting the crystal momentum ($\hbar k$)
for its counterpart in the classical theory. There is a subtlety in the semi-
classical treatment, however, where both the Larmor radius and the cyclotron
frequency depend on the spin state of the electron via the Rashba coupling,
and these effects cancel such that the electron’s velocity is independent of
spin Reynoso  _et al._ (2004). In other words, the fact that the velocities
are the same for the two spin states does not mean their Larmor radii are
equal, as their cyclotron frequencies also differ. Our numerical approach,
which is fully quantum mechanical, explicitly solves for the wave propagation
of the electron via the system’s Hamiltonian and therefore does not depend on
the definition of the Larmor radius. Nevertheless we find our results
correspond well to the radii predicted by equations 1 and 4.

We can now discuss the interpretation of a typical TMF response using this
semi-classical model. The conductance between the leads varies with magnetic
field strength, as illustrated in Fig. 1b. The first pair of peaks located at
approximately 0.061 T and 0.089 T correspond to when the Larmor radii of the
spin–down and spin–up electrons coincide with the distance between the leads
respectively, resulting in almost pure spin currents. The effect can be seen
more clearly by plotting the probability density of conduction electrons
injected into a TMF device. In Fig. 2a, b we see unfiltered spin–down and
spin–up electrons spatially separate into distinct trajectories in the
focusing area.

The peak at $B_{{z}}\approx 0.151$ T in Fig. 1b shows where the spin currents
combine into a single current, giving a much higher total conductance. If we
continue increasing the magnetic field strength, we eventually get another
pair of peaks around 0.225 T that are less distinct compared to the first
pair. Our numerical results show good agreement with the semi-classical
approximation of the peak positions calculated using Eq. 4. More sophisticated
semi-classical approximations can also allow the finer features in the
conductance spectrum to be analytically calculated Bladwell and Sushkov
(2017).

Figure 2:  spin–projected probablity densities for an electron injected into a
TMF device with an out-of-plane magnetic field of (a) 0.061 T, (b) 0.089 T,
and (c) 0.151 T, corresponding to the first three peaks in Fig. 1(b). The
colour in the plots indicate the direction of spin as measured along the $x$
axis. The ratio of red to blue in the RGB colour values at each point is set
to be proportional to the expected spin states at that point. Pure red
indicates a pure spin–up state and pure blue indicates a pure spin–down state,
while purple indicates a mixed state. The transparency values are on a
normalised tanh scale, given by $\tanh(2d_{i}/d_{\mathrm{max}})/\tanh(2)$,
where $d_{i}$ is the density at each point and $d_{\mathrm{max}}$ is the
maximum density in each plot. Points with the maximum density in each plot
will be completely opaque, while points with zero density will be completely
transparent.  Figure 3:  (a) Slice of the Fermi surface of a 2DEG with Rashba
spin orbit coupling at a certain energy. The directions of the arrows indicate
the spin alignment of the electron in the $xy$ plane, and the colour of the
bands indicates the spin of the electron measured along the $x$ direction. (b)
Classical trajectories of electrons injected into the scattering area of a TMF
device with an out-of-plane magnetic field $B_{{z}}$ applied. The colour of
the lines indicates the $x$-component of the spin of the electron, blue
indicates spin–down, red indicates spin–up and grey indicates zero spin in
$x$. The solid line shows the path of electrons with spin–down in $x$ at the
injection point, and the dashed line shows the path of electrons with spin–up
in $x$ at the injection point. Circled numbers in both figures label
corresponding positions in real space and $k$-space.

The recombination of the spatially separated spin currents upon reflection
from a wall of the device is a known phenomenon Usaj and Balseiro (2004); Lo
_et al._ (2017), but deserves more discussion as the reason it happens may not
immediately be clear. The probability density plot in Fig. 2c corresponds to
the highest peak at approximately 0.151 T in Fig. 1b. We see that the
spatially separated trajectories of the spin–up and spin–down electrons
overlap at the collector after reflecting off the bottom edge of the device.
This suggests the crystal momenta of the electrons change after reflection
from a hard wall boundary as the Larmor radius of each trajectory has changed.

The picture of the spin–up and spin–down electrons moving in their own
trajectories as shown in Fig. 1a is greatly simplified and commonly used in
the TMF literature. In this picture, spin–up and down are defined to be
parallel or anti-parallel to the Rashba effective magnetic field, which is
always perpendicular to the direction of the instantaneous velocity of the
electron. In contrast, if a stationary frame is chosen for the definition of
spin–up and spin–down (e.g. Cartesian coordinates in the lab frame), it
becomes more obvious that the spin of the electron precesses as it travels
through the focusing area Reynoso  _et al._ (2004). Figure 3a illustrates
this; the spin of the electron is always aligned tangentially to the Fermi
surface in $k$-space, and at points where the electron is moving only in the
$x$ or $y$ direction, its spin is aligned along the $y$ or $x$ direction,
respectively.

If we follow the path of an electron that has spin–down in $x$ when injected
into the focusing area as shown by the solid line in Fig. 3b, we see how the
spin precession leads to the spin currents recombining after reflecting off a
hard wall boundary in the device. At the injection point, labelled
\raisebox{-.9pt}{1}⃝, the electron is moving in the positive $y$ direction,
which corresponds to point \raisebox{-.9pt}{1}⃝ on the outer Fermi surface in
$k$-space. It has a larger crystal momentum than an electron with spin–up in
$x$ (which starts on the inner Fermi surface at \raisebox{-.9pt}{4}⃝) and
therefore a larger Larmor radius (Eq. 1). Here we see that the group velocity
is proportional to $\frac{\partial E}{\partial k}$, and the gradients of the
inner and outer Fermi surfaces are equal in magnitude, so the spin–up and
spin–down electrons have equal group velocities. Although the the group
velocities have the same signs as the corresponding crystal momenta at this
particular energy, this is not necessarily always the case, e.g. when the
Fermi surfaces are no longer concentric near the bottom of the band.

As the electron moves through the focusing area, it flips to spin–up in $x$
after point \raisebox{-.9pt}{2}⃝. When it reaches the boundary of the device
at point \raisebox{-.9pt}{3}⃝, the electron is moving in the negative $y$
direction before it is reflected. After reflection, spin is conserved but
velocity changes sign, i.e. it switches from point \raisebox{-.9pt}{3}⃝ on the
outer Fermi surface to point \raisebox{-.9pt}{4}⃝ on the inner Fermi surface,
similar to the effect observed by Chen et al. Chen  _et al._ (2005). The
electron now has a lower crystal momentum and a shorter Larmor radius as it
continues moving through point \raisebox{-.9pt}{5}⃝, where it flips back to
spin–down in $S_{{x}}$, and exits the device at point \raisebox{-.9pt}{6}⃝. An
electron which is spin–up in $S_{{x}}$ at injection would go through a similar
process, but starts with a lower $k$ at injection and changes to a higher $k$
after reflection. Thus the separate spin currents recombine after reflecting
off the boundary of the system once (as seen in Fig. 2c), resulting in a sharp
peak in the conductance spectrum.

###  II.2 Numerical model

For all calculations in this paper, we simulated a TMF device with a geometry
as shown in Fig. 1a. The injection and collection leads of the device were
separated by 1500 nm as measured from the centre of each lead. The left, right
and top edges of the device are simulated as open boundaries by attaching
semi-infinite leads to them. The bottom edge is a hard wall boundary except
for the injection and collection leads. An additional 312.5 nm was added
between the side edges and the leads to serve as a buffer. The size of this
buffer was chosen to be large enough that it did not significantly affect the
results. The height of the device (length in $y$ direction) was chosen to be
the equal to the length of the separation between the contacts, 1500 nm, so
that the focusing area was large enough to accommodate the cyclotron
trajectory of an electron focused from the injection lead to the collection
lead.

The device was described as a 2DEG confined to the $xy$ plane, with an
external magnetic field $B_{{z}}$ applied along the $z$ axis and Rashba
spin–orbit coupling with the following Hamiltonian Usaj and Balseiro (2004):

$\begin{split}H=-\frac{\hbar^{2}}{2m^{*}}\left(\frac{\partial^{2}}{\partial
x^{2}}+\frac{\partial^{2}}{\partial
y^{2}}\right)-\frac{g}{2}\mu_{\mathrm{B}}\sigma_{{z}}B_{{z}}\\\
-i\alpha\left(\frac{\partial}{\partial y}\sigma_{{x}}-\frac{\partial}{\partial
x}\sigma_{{y}}\right)\end{split}$ (5)

where $g$ is the effective g-factor, $\mu_{B}$ is the Bohr magneton, and
$\sigma_{x,y,z}$ are the Pauli spin matrices. To approximate the effect of the
magnetic field on the electron we used the Peierls substitution Peierls
(1933):

$-i\frac{\partial}{\partial x}\Rightarrow-i\frac{\partial}{\partial
x}+\frac{e}{\hbar}A_{x}$ (6)

where $A_{x}$ is the $x$-component of the magnetic vector potential. The
$y$-components were similarly substituted. The Landau gauge in $\hat{y}$,
$A=-xB_{{z}}\hat{y}$, was used everywhere except in the semi-infinite leads on
the left and right edges and a thin strip of the focusing area next to those
leads, where the Landau gauge in $\hat{x}$, $A=-yB_{{z}}\hat{x}$, was used
instead. We used the effective mass $m^{*}=0.04m_{e}$ where $m_{e}$ is the
free electron mass, and an effective g-factor $g=9$ for InGaAs Chuang  _et
al._ (2015). For the Rashba coupling parameter, we used the value for an
InGaAs-based 2DEG, $\alpha=3.1\times 10^{-11}$ eV$\cdot$m Lo  _et al._ (2017).
We note that other spin–orbit interactions such as the Dresselhaus effect are
present in InGaAs Kawaguchi  _et al._ (2019), however they are not as strong
as the Rashba effect, therefore we only included the Rashba effect in this
paper for simplicity. The external magnetic field and Rashba spin–orbit
coupling were applied everywhere in the system, including within the leads.
Note that while we refer to the injection and collection points as leads, they
are still part of the 2DEG, and typically correspond to QPCs in experiments.

Figure 4:  Four types of lead geometries defined on a finite difference grid:
a) funnel, b) square, c) trumpet, d) point. Bottom two rows in each image
represent semi-infinite leads (indicated in red).

Our calculations used a finite difference method to discretise the Hamiltonian
given in Eq. 5, resulting in a real space grid as shown in Fig. 4. Each vertex
in the grid represents a discretisation site that has an associated on-site
potential, and the lines represent hoppings that couple neighbouring sites.
Neighbouring sites in the grid are separated by a length of 3.125 nm. This
grid spacing was chosen by balancing computational efficiency and numerical
accuracy, while ensuring that results converged. Energies in our calculations
are given in terms of a “hopping energy” $t$, defined as:

$\displaystyle t$ $\displaystyle=\frac{\hbar^{2}}{2m^{*}a^{2}}$ (7)

where $a$ is the grid spacing. The Peierls substitution modifies the hopping
terms by a Peierls phase Peierls (1933), e.g. if the hopping term between
sites $r_{1}$ and $r_{2}$ is $t$, then:

$t\Rightarrow
t\exp\left(\frac{e}{\hbar}\int^{r_{2}}_{r_{1}}\mathbf{A}(\mathbf{r})\cdot
d\mathbf{r}\right)$ (8)

We calculated our results using the software package KWANT, which uses a wave
function matching method that is mathematically equivalent to the non-
equilibrium Green’s function method Groth  _et al._ (2014); Boumrar  _et al._
(2020). This model assumes ballistic transport (the electrons have mean free
paths longer than the device length) and the zero temperature limit (no
thermal broadening). It is a fully quantum description that captures the wave-
like nature of electrons as can be seen in the density plots in Fig. 2 where
the trajectories are clearly not classical cyclotron trajectories. We did not
apply any bias between the leads in all of our calculations, so conductance
was calculated using the Landauer-Büttiker formalism with zero-bias voltage,
giving Datta (1995):

$\displaystyle G$ $\displaystyle=\frac{2e^{2}}{h}\int
T(E)\left[-\frac{\partial f_{0}(E)}{\partial E}\right]dE$ (9)

where $h$ is Planck’s constant, $T(E)$ is the transmission probability between
the injector and collector, and $f_{0}(E)$ is the Fermi-Dirac distribution
function. In the zero temperature limit, the derivative of the Fermi-Dirac
distribution function is a Dirac delta function, thus the integral in Eq. 9 is
only evaluated at the exact energy chosen. The conductance is then directly
proportional to the transmission probability, with a proportionality constant
given by the conductance quantum, $2e^{2}/{h}$.

The spin–projected probability densities in Fig. 2 are calculated by:

$\displaystyle\rho_{i,\pm}$
$\displaystyle=\left\langle\psi_{i}\middle|\chi_{\pm}\right\rangle\left\langle\chi_{\pm}\middle|\psi_{i}\right\rangle$
(10)

where for each site $i$, $\rho_{i,\pm}$ is the spatially-resolved probability
density for spin–up/down in $x$, $\psi_{i}$ is the wavefunction element
corresponding to that site, and $\chi_{\pm}$ are the eigenvectors for the $x$
Pauli matrix.

##  III Results and discussion

###  III.1 Influence of device geometry

An important consideration in TMF experiments is the influence of device
geometry on the resulting signal. Whilst the leads in actual devices are QPCs
formed by smoothly varying potentials, here we model them as being confined by
simple hard walls such that there are well-defined widths. We present a more
realistic model of the leads in section III.3. We define the width of a lead
as the width of its narrowest section, and its shape by the position of that
section relative to the contact point between the lead and the focusing area.

We find three different shapes used in TMF experiments in the literature.
First, there is the funnel-shaped lead (Fig. 4a) which has its narrowest
section at the contact point and becomes wider towards the electron source van
Houten  _et al._ (1989); Potok  _et al._ (2002); Yan  _et al._ (2018). Second,
the square-shaped lead (Fig. 4b) maintains the same width throughout its
entire length Lo  _et al._ (2017). Third, the trumpet-shaped lead (Fig. 4c) is
at its widest at the contact point and becomes narrower towards the electron
source Dedigama  _et al._ (2006). Figure 4a–c shows the grids we used to
simulate each of these shapes. Finally, we also show a grid in Fig. 4d that
represents an idealised point (one-dimensional) lead, the end of which is
connected to a number of sites in the focusing area, giving the lead an
effective width. For example, a 50 nm wide point lead refers to a 1D lead that
is connected to 17 sites in the channel, which for a grid spacing of 3.125 nm
is equivalent to a width of 50 nm (see Fig. 4d).

Figure 5:  Conductance as a function of the out-of-plane magnetic field for a
transverse magnetic focusing device with 50 nm wide point, trumpet, square,
and funnel shaped leads with varying separation distances between the leads,
calculated with the Fermi energy set to 7 meV. The conductance values are
normalised to the maximum conductance in each plot. Note that plots in the
same column share the same $x$-axis but different columns have differently
scaled $x$-axes.  Figure 6:  Conductance as a function of the out-of-plane
magnetic field for transverse magnetic focusing devices with various
combinations of injector and collector widths. All leads are trumpet shaped
and set 1500 nm apart, with the Fermi energy set to 7 meV. Conductance values
are normalised to the maximum value in each spectrum.

This idealised point lead will only have one transmission mode Usaj and
Balseiro (2004). It is a simple theoretical model that ignores the effects of
wave interference in the lead itself. Similar grids for the point lead Usaj
and Balseiro (2004) and square-shaped lead Kormányos (2010) have been used in
the literature for numerical simulations of TMF experiments.

The shape of the lead influences the impedance matching between the lead and
the focusing area, and thus we expect that there is an effect on the measured
conductance spectra. The separation distance between the leads has also been
shown to have an effect on the conductance spectrum Rendell  _et al._ (2015).
We calculated conductance as a function of the out-of-plane magnetic field for
TMF devices using injector and collector leads of each of the aforementioned
shapes with widths of 50 nm. The separation distance between the leads was set
to 500 nm, 700 nm, 1500 nm and 2500 nm. Here a Fermi energy of
$E_{\text{F}}=7$ meV was used. Figure 5 shows the resulting conductance
spectra with the conductance normalised to the maximum value in each data
series. Note that each column has a different scale for the $x$-axis since the
peak positions shift towards lower magnetic field strengths as the lead
separation is increased, which agrees with Eq. 1.

We see that for each separation distance, the spectra for all lead shapes have
features similar to the spectrum for the idealised lead (point).

The main peaks are easily resolvable in all the spectra, but finer structures
such as the interference fringes are only clearly observed at the larger
separations of 1500 nm and 2500 nm, when the peaks become sharper. The shapes
of the spectra change as the separation distance is changed, but stay
relatively similar when the lead shape is changed for the same separation.
Interestingly, at a separation distance of 750 nm, a splitting appears in the
second focusing peak, which should not have spin–splitting as discussed in
Sec. II.1. This splitting appears to be dependent on the separation distance
as it is not noticeable at other separation distances.

The spectra for the trumpet shaped lead appear to have less fluctuations
compared to the square and funnel shaped leads for all separation distances.
While the interference fringes before the first and second peaks are better
resolved in the funnel and square shaped leads, we note that the magnitude of
the fringes are comparable to the fluctuations seen in the peaks at higher
magnetic field strengths. Thus it would not be easy to differentiate between
path interference fringes Bladwell and Sushkov (2017) and fluctuations from
coincidental interference effects. We chose to use the trumpet-shaped lead for
subsequent calculations in this paper as it appeared to consistently give the
best combination of low noise and clear resolution of features in the
conductance spectrum, while still being an experimentally realistic geometry.

Another geometric aspect that affects the measured conductance spectra is lead
width. Intuitively we would expect that the wider the leads, the wider the
peaks would be in the spectra, since a wider injector would create a larger
wavefront while a wider collector would collect more of that wavefront at each
magnetic field strength. Figure 6 shows the transmission spectra around the
first peak pair in these spectra at $B_{{z}}\approx 0.061$ T and
$B_{{z}}\approx 0.089$ T for TMF devices with the injector and collector
widths set to combinations of 50 nm, 100 nm, 150 nm, 200 nm, and 250 nm. Here
a Fermi energy of 7 meV was used. We see that resolving the spin–split peaks
depends on both the injector and collector widths. The resolvability of the
peaks is reduced as we move across each row and down each column of Fig. 6,
with the clearest peaks observed when both leads are at their minimum widths
(here 50 nm).

Figure 7:  Conductance spectra calculated with the same parameters as in Fig.
6 using idealised point leads (solid lines) and trumpet leads (dashed lines)
with both injector and collector widths as labelled and set 1500 nm apart.
Conductance values are normalised to the maximum value in each spectrum and
offset for clarity.  Figure 8:  Conductance as a function of the out-of-plane
magnetic field for a transverse magnetic focusing device with 100 nm wide
trumpet shaped leads set 1500 nm apart, calculated with Fermi energies set
between 8 and 75 meV. The Fermi energy crosses one (1 meV and 2 meV), three (8
meV and 11 meV), five (21 meV and 26 meV), and seven (40 meV and 48 meV)
subbands as indicated, corresponding to the band structure shown in Fig. 9.
The shape of the peaks changes as more subbands are occupied, but stays the
same for energies within the same subband.

If we look at the conductance spectra of a TMF device using idealised leads of
the same widths, as shown in Fig. 7, we can see a similar degradation in the
resolvability of the peaks as the widths of the leads increase. The individual
peaks broaden as the lead widens, and we observe that the interference fringes
before the spin–split peaks are not observable at widths above 50 nm. In
general, the spin–split peaks broaden as the leads widen and the interference
fringes become less observable.

###  III.2 Subbands

Thus far we have not considered the number of subbands occupied by the
electrons when we choose the Fermi energy in our calculations. In experiments
this can be controlled by changing the gate voltage at the leads to have
conductance values that correspond to the number of subbands occupied Chuang
_et al._ (2015). Adjusting the gate voltage in experiments effectively shifts
the band structure within the leads, whilst the Fermi energy stays constant.

In our calculations, we chose to shift the Fermi energy instead for numerical
simplicity. While this will shift the positions of the peaks in the
conductance spectra for different Fermi energies, the effect this has on the
overall features of the spectra (e.g. peak shapes) is equivalent to shifting
the band structure. We will present a more realistic model of the leads as
QPCs controlled by gate voltages in Sec. III.3. We examine in Fig. 8 how the
features of the peaks at the first focusing magnetic field strength change as
the the Fermi energy shifts through the lowest four odd numbers of subbands.
The TMF device simulated has trumpet-shaped leads with widths of 100 nm
separated by 1500 nm. To interpret these results, Fig. 9 shows the 1D band
structure calculated for a nanowire with a width of 100 nm and the dotted
lines indicate the corresponding Fermi energies shown in Fig. 8.

In Fig. 8, we see that the amplitude of the peaks increases as the Fermi
energy increases, as more conducting modes are available when more subbands
are occupied. The ratio of the first peak amplitude to the second peak
amplitude changes from low:high to high:low as more subbands are occupied, but
the peak shapes and amplitudes do not change significantly when the Fermi
energy changes within the same subband. In the spectra that correspond to one
occupied subband (blue), the first spin–split peak is greatly suppressed. This
strong spin polarisation is explained by adiabatic transitions between anti-
crossed subbands as the lead widens Eto  _et al._ (2005). However, note that
we do not observe the suppression of the first spin–split peak in the spectra
along the first row and column of Fig. 6, even though there is only one
occupied subband at 7 meV for a 50 nm wide nanowire. For a 50 nm wide
nanowire, the anti-crossing between the first and second subbands occurs at an
energy close to the minimum of the third subband, a much higher energy
compared to the case for a 100 nm wide nanowire (see Fig. 9). This suggests
the 50 nm wide lead does not widen enough for the electron to transition
through the anti-crossing point, thus the first spin–split peak is not
suppressed.

Figure 9:  1D band structure for a 100 nm wide infinite wire with Rashba
spin–orbit interaction present. The colour bar indicates the $x$-component of
the electron spin, with spin–up shown as red and spin–down shown as blue. The
horizontal dotted lines indicate energies used for calculating the conductance
plots in Fig. 8.  Figure 10:  Calculated potential field in 2DEG from 40 mV
potential applied at surface gates labelled 1, 2, 3 and 4 (marked by orange
solid lines). The green dotted box denotes the area simulated in calculations.
The yellow dashed lines indicate where 300 nm wide semi-infinite leads were
attached. The overlaid spin–projected density plot shows the probability
density of an electron with a Fermi energy of 7 meV injected at L1 and
collected at L2 under an out-of-plane magnetic field of 0.0957 T. The density
plot has the same colour scheme as in Fig. 2.

The band structure significantly affects the features of the TMF spectra,
therefore it is vitally important to ensure that the number of subbands
occupied and the spin compositions of the subbands are known when interpreting
TMF spectra.

###  III.3 Simulation of realistic QPCs

In experiments, the injector and collector leads in Fig. 1a are QPCs formed by
applying voltage to surface gates, depleting electrons within the 2DEG under
the gates van Houten  _et al._ (1989). Changing the gate voltages effectively
changes the width of the QPC van Houten  _et al._ (1989) and shifts the band
structure within the QPC. We can approximate the potential field within the
2DEG applied by the surface gates using the method derived by Davies et al.
Davies  _et al._ (1995). Adding this potential field as a position dependent
potential term to the Hamiltonian in Eq. 5 allows us to model the TMF device
more realistically.

For our calculations we used the potential generated from surface gates with
shapes illustrated by the orange lines in Fig. 10. The injector QPC is formed
by gates 3 and 4, while the collector QPC is formed by gates 1 and 2. The
gates forming each QPC are separated by 300 nm. To avoid asymmetric biasing in
the QPCs, we apply equal voltages on gates 1 and 2, as well as gates 3 and 4.
We define the injector and collector gate voltages ($V_{\mathrm{g}}$) as the
voltage applied to the gates forming the respective QPCs. The semi-infinite
leads attached below each QPC have the same potential profile as the sites
adjacent to the semi-infinite leads.

We calculated the transmission spectra for a TMF device with a lead separation
of 1500 nm and a Fermi energy of 7 meV. The injector and collector gate
voltages were set to combinations of 10 mV, 20 mV, 30 mV, 40 mV, and 50 mV
(Fig. 11). The conductance values in Fig. 11 are normalised for better
comparison of the peak features. However, we note that the conductance values
for gate voltages of 50 mV (bottom row and right-most column in Fig. 11) are
actually several orders of magnitude smaller than the conductance values in
the other spectra. Also, the conductance values for gate voltages of 40 mV are
close to one conductance quanta. This suggests that the QPC formed by gate
voltages of 40 mV is only allowing transmission in the lowest subband, and at
50 mV, the only transmission occurring is from tunneling electrons. We see
that similar to Fig. 6, the peaks broaden as the effective width of the QPCs
increase (i.e. the voltage decreases). However, unlike Fig. 6, reducing the
effective width of the QPC does not allow the interference fringes to be
resolved even with the narrowest QPCs. This suggests that to observe the
interference fringes experimentally, the QPCs will need to have sharp hard
wall-like potential profiles.

We also see that the first peak is greatly suppressed at gate voltages of 40
mV, similar to the single subband spectra in Fig. 8. Again, this spin
polarisation can be explained by adiabatic transitions between anti-crossed
subbands as the QPC narrows and widens Eto  _et al._ (2005). Thus, we can see
that the effect of changing the gate voltage is a combination of changing the
width and Fermi energy in our previous simpler model. While the hard wall
model of leads is less realistic, it allows us to explore the effects of width
and Fermi energy independently.

###  III.4 Disorder

Devices used in experiments are usually not perfect, they exhibit defects,
impurities, surface roughness and other imperfections. This causes electrons
to scatter as they travel through the device. Here we simulate disorder using
an Anderson localisation model Anderson (1958), where we add a random onsite
potential with values chosen from a uniform distribution within the range
$[-U/2,U/2]$ to each site within the focusing area, but not within the semi-
infinite leads (the red sites in Fig. 4). $U$ is defined as the disorder
strength and can be related to the mean free path ($l_{\mathrm{mfp}}$) of the
electrons using Fermi’s golden rule Ihn (2009) (Derivation shown in
supplementary information):

$\displaystyle l_{\mathrm{mfp}}=48a\frac{\sqrt{E_{\mathrm{F}}/t}}{(U/t)^{2}}$
(11)

where $a$ is the length of the grid spacing, $E_{\mathrm{F}}$ is the Fermi
energy, and $t$ is the hopping energy as defined in Eq. 7.

Figure 11:  Conductance as a function of the out-of-plane magnetic field for
transverse magnetic focusing devices with various combinations of gate
voltages at the injector and collector. All leads are set 1500 nm apart, with
the Fermi energy set to 7 meV. Conductance values are normalised to the
maximum value in each spectrum.  Figure 12:  Mean free path length
($l_{\mathrm{mfp}}$) of an electron in a device with disorder strength $U$
calculated using Eq. 11. The lattice constant ($a$) and Fermi energy
($E_{\mathrm{F}}$) were set to be 3.125 nm and 7 meV respectively. The inset
plots show the probability density for devices at a magnetic field strength of
$B_{{z}}=0.089$ T, with particular realisations of disorder strengths 0.10$t$
and 0.25$t$. The horizontal black dashed line at $l_{\mathrm{mfp}}=1500$ nm
indicates the distance $D$ between the injector and collector leads in the
device used in Fig. 13. The vertical dashed lines correspond to the disorder
strengths used in Fig. 13.

For our calculations, we set $a$ to be 3.125 nm and $E_{\mathrm{F}}$ to be 7
meV. The mean free path lengths at varying disorder strengths calculated from
Eq. 11 is shown in Fig. 12. The two inset probability density plots show that
the spin–split trajectories become less well-defined as the mean free path
length becomes shorter than the distance between the injector and collector
leads.

Figure 13 shows the conductance spectra calculated for a TMF device for
various disorder strengths $U$. We performed conductance calculations for 201
different disorder realisations at each disorder strength. The mean of the
conductance values at each magnetic field strength and disorder strength are
shown as solid lines in Fig. 13, and the shaded areas show the variance of the
conductance values for each disorder strength. We observe that as disorder
increases, the amplitude of the conductance peaks decrease as expected since
scattering reduces the probability that the electron reaches the collector
lead. The spin–split peaks at $B_{{z}}\approx 0.061$ T and $B_{{z}}\approx
0.089$ T are greatly suppressed and indistinguishable from signal noise at
disorder strengths of $0.25t$ and above. This agrees with our calculation that
the mean free path of the electron at $U=0.25t$ is less than half the distance
between the injector and collector leads (Fig. 12), thus the electron is
likely to be scattered without reaching the collector.

We only simulated disorder within the weak localisation regime where the mean
free path length is much greater than the Fermi wavelength. However, even
within this regime, the characteristic features in the conductance spectra
rapidly became indistinguishable as the devices were increasingly disordered.
Therefore, this implies that to observe the finer structure in the spectra
such as the interference fringes, the device used has to be relatively free
from disorder.

Figure 13:  Conductance as a function of the out-of-plane magnetic field for a
transverse magnetic focusing device with injector and collector QPCs set 1500
nm apart, and both gate voltages set to (a) 30 mV and (b) 40 mV. Fermi
energies were set to 7 meV. Disorder was implemented in the systems by
applying onsite potentials chosen at random from a uniform distribution within
the range $[-U/2,U/2]$. Solid lines show the mean values for 201 realisations
of that particular disorder strength, and the corresponding shaded areas show
the peak-to-peak variance.

##  IV Conclusions

We have shown the geometry of the device used in a TMF experiment plays a
significant role in the measured characteristics of the conductance spectra.
The shape of the leads used does not significantly affect the separation of
the spin–split peaks, but does affect the level of noise or fluctuations in
the spectra. The separation distance between the injector and collector leads
can significantly alter the features of the spectra. The width of the leads
used significantly affects the resolution of the finer features in the TMF
spectra, e.g. the interference fringes. Furthermore, the spin composition of
the subbands and the number of subbands occupied both affect the ratio of the
spin–split peak amplitudes. Thus, accurate knowledge of the device geometry
and band structure is essential when interpreting TMF results.

As a result of these effects, the potentials forming QPCs in a TMF device need
to be very well-defined in order to observe finer structures in the TMF
spectra, e.g. the interference fringes. In addition, we show that the TMF
device is sensitive to disorder and features in the measured conductance
spectrum quickly become obfuscated as the device becomes disordered. The use
of real-space numerical simulation of TMF experiments allows us to disentangle
geometric effects and device imperfections from the spin–split structure and
the interference fringes. This in turn allows for unambiguous interpretation
of TMF experiments in the future.

## Abbreviations

TMF: Transverse magnetic focusing; 2DEG: Two-dimensional electron gas; 2DHG:
Two-dimensional hole gas; QPC: Quantum point contact

## Declarations

### Availability of data and materials

The datasets used and/or analyzed during the current study are available from
the corresponding author on reasonable request.

### Competing interests

The authors declare that they have no competing interests.

### Funding

This work was funded in part by the Australian Research Council Centre of
Excellence in Future Low-Energy Electronics Technologies (project number
CE170100039). Computational resources and services were provided by the
National Computational Infrastructure (NCI), which is supported by the
Australian Government.

### Authors’ contributions

YKL wrote the code, performed the calculations and drafted the manuscript. All
authors analysed and interpreted the results, and revised the manuscript. All
authors read and approved the final manuscript.

### Acknowledgements

We acknowledge useful conversations with S. Bladwell, O. Sushkov and D.
Culcer.

## References

  * Xia  _et al._ (2012) J. Xia, W. Ge, and K. Chang,  _Semiconductor spintronics_ (World Scientific, 2012).
  * Hirohata  _et al._ (2020) A. Hirohata, K. Yamada, Y. Nakatani, I. L. Prejbeanu, B. Diény, P. Pirro, and B. Hillebrands, Journal of Magnetism and Magnetic Materials 509, 166711 (2020).
  * Lo  _et al._ (2017) S. T. Lo, C. H. Chen, J. C. Fan, L. W. Smith, G. L. Creeth, C. W. Chang, M. Pepper, J. P. Griffiths, I. Farrer, H. E. Beere, G. A. C. Jones, D. A. Ritchie, and T. M. Chen, Nature Communications 8, 15997 (2017).
  * Bychkov and Rashba (1984) Y. A. Bychkov and E. I. Rashba, Journal of Experimental and Theoretical Physics Letters 39, 78 (1984).
  * Tsoi (1974) V. S. Tsoi, ZhETF Pisma Redaktsiiu 19, 114 (1974).
  * van Houten  _et al._ (1989) H. van Houten, C. W. J. Beenakker, J. G. Williamson, M. E. I. Broekaart, P. H. M. van Loosdrecht, B. J. van Wees, J. E. Mooij, C. T. Foxon, and J. J. Harris, Physical Review B 39, 8556 (1989).
  * Heremans  _et al._ (1992) J. J. Heremans, M. B. Santos, and M. Shayegan, Applied Physics Letters 61, 1652 (1992).
  * Heremans  _et al._ (1994) J. J. Heremans, M. B. Santos, and M. Shayegan, Surface Science 305, 348 (1994).
  * Potok  _et al._ (2002) R. M. Potok, J. A. Folk, C. M. Marcus, and V. Umansky, Physical Review Letters 89, 266602 (2002).
  * Folk  _et al._ (2003) J. A. Folk, R. M. Potok, C. M. Marcus, and V. Umansky, Science 299, 679 (2003).
  * Rokhinson  _et al._ (2004) L. P. Rokhinson, V. Larkina, Y. B. Lyanda-Geller, L. N. Pfeiffer, and K. W. West, Physical Review Letters 93, 146601 (2004).
  * Dedigama  _et al._ (2006) A. R. Dedigama, D. Deen, S. Q. Murphy, N. Goel, J. C. Keay, M. B. Santos, K. Suzuki, S. Miyashita, and Y. Hirayama, Physica E: Low-dimensional Systems and Nanostructures 34, 647 (2006).
  * Heremans  _et al._ (2007) J. J. Heremans, H. Chen, M. B. Santos, N. Goel, W. Van Roy, and G. Borghs, in  _AIP Conference Proceedings_, Vol. 893 (AIP, 2007) pp. 1287–1288.
  * Rendell  _et al._ (2015) M. Rendell, O. Klochan, A. Srinivasan, I. Farrer, D. A. Ritchie, and A. R. Hamilton, Semiconductor Science and Technology 30, 102001 (2015).
  * Yan  _et al._ (2017) C. Yan, S. Kumar, M. Pepper, P. See, I. Farrer, D. Ritchie, J. Griffiths, and G. Jones, Nanoscale Research Letters 12, 553 (2017).
  * Yan  _et al._ (2018) C. Yan, S. Kumar, M. Pepper, K. Thomas, P. See, I. Farrer, D. Ritchie, J. Griffiths, and G. Jones, Journal of Physics: Conference Series 964, 012002 (2018).
  * Tsoi  _et al._ (1999) V. S. Tsoi, J. Bass, and P. Wyder, Rev. Mod. Phys. 71, 1641 (1999).
  * Bozhko (2014) S. I. Bozhko, JETP Letters 99, 487 (2014).
  * Taychatanapat  _et al._ (2013) T. Taychatanapat, K. Watanabe, T. Taniguchi, and P. Jarillo-Herrero, Nature Physics 9, 225 (2013).
  * Reynoso  _et al._ (2007) A. Reynoso, G. Usaj, and C. A. Balseiro, Physical Review B 75, 085321 (2007).
  * Usaj and Balseiro (2004) G. Usaj and C. A. Balseiro, Physical Review B 70, 041301(R) (2004).
  * Bladwell and Sushkov (2017) S. Bladwell and O. P. Sushkov, Physical Review B 96, 035413 (2017).
  * Groth  _et al._ (2014) C. W. Groth, M. Wimmer, A. R. Akhmerov, and X. Waintal, New Journal of Physics 16, 063065 (2014).
  * Xiao  _et al._ (2010) D. Xiao, M. C. Chang, and Q. Niu, Reviews of Modern Physics 82, 1959 (2010).
  * Reynoso  _et al._ (2004) A. Reynoso, G. Usaj, M. J. Sánchez, and C. A. Balseiro, Physical Review B 70, 235344 (2004).
  * Chen  _et al._ (2005) H. Chen, J. J. Heremans, J. A. Peters, A. O. Govorov, N. Goel, S. J. Chung, and M. B. Santos, Applied Physics Letters 86, 032113 (2005).
  * Peierls (1933) R. Peierls, Zeitschrift für Physik 80, 763 (1933).
  * Chuang  _et al._ (2015) P. Chuang, S. C. Ho, L. W. Smith, F. Sfigakis, M. Pepper, C. H. Chen, J. C. Fan, J. P. Griffiths, I. Farrer, H. E. Beere, G. A. C. Jones, D. A. Ritchie, and T. M. Chen, Nature Nanotechnology 10, 35 (2015).
  * Kawaguchi  _et al._ (2019) K. Kawaguchi, T. Fukasawa, I. Takazawa, H. Shida, Y. Saito, D. Iizasa, T. Saito, T. Kitada, Y. Ishitani, M. Kohda, and K. Morita, Applied Physics Letters 115, 172406 (2019).
  * Boumrar  _et al._ (2020) H. Boumrar, M. Hamidi, H. Zenia, and S. Lounis, Journal of Physics: Condensed Matter 32, 355302 (2020).
  * Datta (1995) S. Datta,  _Electronic transport in mesoscopic systems_ , 1st ed., Cambridge studies in semiconductor physics and microelectronic engineering (Cambridge University Press, Cambridge, 1995) p. 108.
  * Kormányos (2010) A. Kormányos, Physical Review B 82, 155316 (2010).
  * Eto  _et al._ (2005) M. Eto, T. Hayashi, and Y. Kurotani, Journal of the Physical Society of Japan 74, 1934 (2005).
  * Davies  _et al._ (1995) J. H. Davies, I. A. Larkin, and E. V. Sukhorukov, Journal of Applied Physics 77, 4504 (1995).
  * Anderson (1958) P. W. Anderson, Physical Review 109, 1492 (1958).
  * Ihn (2009) T. Ihn,  _Semiconductor Nanostructures_ (Oxford University Press, 2009) p. 165.

##  V Supplementary Information: Derivation of mean free path formula (Eq. 11)

Here we derive Eq. 11. We begin with an equation for the scattering time in
the single subband case ($\tau_{0}$) given in pg. 168 of Ihn (2009):

$\displaystyle\frac{\hbar}{\tau_{0}}$
$\displaystyle=n_{i}\frac{m^{*}}{2\pi\hbar^{2}}\int^{2\pi}_{0}d\varphi\left\langle\left|v^{(i)}(\vec{q})\right|^{2}\right\rangle_{\mathrm{imp}}\left(1-\cos\varphi\right)$
(12)

where $n_{i}$ is the areal density of scatterers, $\varphi$ is the scattering
angle, and
$\left\langle\left|v^{(i)}(\vec{q})\right|^{2}\right\rangle_{\mathrm{imp}}$ is
the ensemble average of the squared scattering potential matrix elements.

The areal density of scatterers is defined as:

$\displaystyle n_{i}$ $\displaystyle:=\frac{N_{i}}{A}$ (13)

where $N_{i}$ is the number of scattering centers within the normalization
area $A$.

In our disorder simulations we applied an onsite potential to every
discretisation site in the device, which act as the scatterers. Since there is
always one scatterer within each unit cell in the discretisation grid, our
normalization area is the area of one unit cell, $a^{2}$, where $a$ is the
grid spacing length. Therefore our areal density of scatterers is one per unit
cell area:

$\displaystyle n_{i}$ $\displaystyle=\frac{1}{a^{2}}$ (14)

The ensemble average of the squared scattering potential matrix elements can
be related to the ensemble average of the squared scattering matrix elements
by Ihn (2009):

$\displaystyle\left\langle\left|v^{(i)}(\vec{q})\right|^{2}\right\rangle_{\mathrm{imp}}$
$\displaystyle=\frac{A^{2}}{N_{i}}\left\langle\left|\left\langle
m\mathbf{k}_{m}\left|V\right|n\mathbf{k}_{n}\right\rangle\right|^{2}\right\rangle_{\mathrm{imp}}$
(15)

where $V$ is the scattering potential. Since our scattering potential is a
uniform random distribution within the range
$\left[-\frac{U}{2},\frac{U}{2}\right]$, independent of $\mathbf{k}$,
$\left\langle m\mathbf{k}_{m}\left|V\right|n\mathbf{k}_{n}\right\rangle=V$.
The probability distribution of $V$, $f_{{x}}(V)$ is given by:

$\displaystyle f_{{x}}(V)$ $\displaystyle=\frac{1}{V_{max}-V_{min}}$
$\displaystyle=\left[\frac{U}{2}-\left(-\frac{U}{2}\right)\right]^{-1}$
$\displaystyle=\frac{1}{U}$ (16)

Applying the law of the unconcious statistician to Eq. 15, we obtain:

$\displaystyle\left\langle\left|v^{(i)}(\vec{q})\right|^{2}\right\rangle_{\mathrm{imp}}$
$\displaystyle=\frac{A^{2}}{N_{i}}\left\langle\left|V\right|^{2}\right\rangle_{\mathrm{imp}}$
$\displaystyle=\frac{A}{n_{i}}\int^{\frac{U}{2}}_{-\frac{U}{2}}V^{2}f_{{x}}(V)dV$
$\displaystyle=\frac{a^{2}}{n_{i}}\frac{U^{2}}{12}$ (17)

We can rewrite the scattering time in terms of the mean free path length
$l_{\mathrm{mfp}}$ and Fermi velocity $v_{\mathrm{F}}$:

$\displaystyle\tau_{0}$
$\displaystyle=\frac{l_{\mathrm{mfp}}}{v_{\mathrm{F}}}$
$\displaystyle=l_{\mathrm{mfp}}\sqrt{\frac{m^{*}}{2E_{\mathrm{F}}}}$ (18)

Substitute Eqs. 14, 17 and 18 into Eq. 12:

$\displaystyle\frac{\hbar}{\tau_{0}}$
$\displaystyle=n_{i}\frac{m^{*}}{2\pi\hbar^{2}}\int^{2\pi}_{0}d\varphi\left\langle\left|v^{(i)}(\vec{q})\right|^{2}\right\rangle_{\mathrm{imp}}\left(1-\cos\varphi\right)$
$\displaystyle\frac{\hbar}{l_{\mathrm{mfp}}}\sqrt{\frac{2E_{\mathrm{F}}}{m^{*}}}$
$\displaystyle=n_{i}\frac{m^{*}}{2\pi\hbar^{2}}\frac{a^{2}}{n_{i}}\frac{U^{2}}{12}{2\pi}$
$\displaystyle l_{\mathrm{mfp}}$
$\displaystyle=\sqrt{\frac{2}{m^{*}}}\frac{12\hbar^{3}}{m^{*}a^{2}}\frac{\sqrt{E_{\mathrm{F}}}}{U^{2}}$

We want to express the energy terms as fractions of the hopping energy $t$.

$\displaystyle l_{\mathrm{mfp}}$
$\displaystyle=\sqrt{\frac{2}{m^{*}}}\frac{12\hbar^{3}}{m^{*}a^{2}}\frac{\sqrt{E_{\mathrm{F}}/t}}{U^{2}/t^{2}}\frac{\sqrt{t}}{t^{2}}$
$\displaystyle=48a\frac{\sqrt{E_{\mathrm{F}}/t}}{U^{2}/t^{2}}$ (19)


# ChatGPT: Beginning of an End of Manual Linguistic Data Annotation?
Use Case of Automatic Genre Identification

Taja Kuzman  Igor Mozetič
Jožef Stefan Institute, Slovenia
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
Nikola Ljubešić
Jožef Stefan Institute, Slovenia
Center za jezikovne vire in tehnologije
Univerze v Ljubljani, Slovenia
<EMAIL_ADDRESS>

###### Abstract

ChatGPT has shown strong capabilities in natural language generation tasks,
which naturally leads researchers to explore where its abilities end. In this
paper, we examine whether ChatGPT can be used for zero-shot text
classification, more specifically, automatic genre identification. We compare
ChatGPT with a multilingual XLM-RoBERTa language model that was fine-tuned on
datasets, manually annotated with genres. The models are compared on test sets
in two languages: English and Slovenian. Results show that ChatGPT outperforms
the fine-tuned model when applied to the dataset which was not seen before by
either of the models. Even when applied on Slovenian language as an under-
resourced language, ChatGPT’s performance is no worse than when applied to
English. However, if the model is fully prompted in Slovenian, the performance
drops significantly, showing the current limitations of ChatGPT usage on
smaller languages. The presented results lead us to questioning whether this
is the beginning of an end of laborious manual annotation campaigns even for
smaller languages, such as Slovenian.

##  1 Introduction

ChatGPT has shown strong capabilities as a dialogue system, providing clearer
and more helpful answers than humans (Guo et al., 2023). It remains unclear
whether its performance on text categorization tasks, more specifically on
automatic genre identification, can be compared to the existing large language
models (LLMs), such as the XLM-RoBERTa model (Conneau et al., 2020), fine-
tuned to the task. Despite the fact that the ChatGPT model was made available
to the public only a few months ago, some studies, analyzing the potential of
the model for numerous natural language processing (NLP) tasks, have already
been published. Qin et al. (2023) analyzed its zero-shot performance on
reasoning tasks, natural language inference, dialogue, question answering,
summarization, named-entity recognition and sentiment analysis. The results
showed that ChatGPT was outperformed by the fine-tuned large language models
at most tasks. It achieved significantly worse results in symbolic reasoning
and named-entity recognition. Similarly, Zhong et al. (2023) compared ChatGPT
with fine-tuned language models BERT (Kenton and Toutanova, 2019) and RoBERTa
(Liu et al., 2019) on the GLUE benchmark (Wang et al., 2018), consisting of
sentiment analysis, linguistic acceptability, paraphrase, textual similarity,
natural language inference, and question answering. The overall results showed
that ChatGPT performed comparably to the BERT model, while it was outperformed
by the RoBERTa model. In contrast, when Zhang et al. (2022) analyzed ChatGPT’s
performance on stance detection, it achieved state-of-the-art (SOTA) results
on this task. The ChatGPT model was also shown to achieve competitive
performance in machine translation of high-resource languages, while it is
lacking behind the SOTA models on low-resource languages based on the WMT22
(Kocmi et al., 2022) dataset (Hendy et al., 2023).

In addition to comparing the model to fine-tuned LLM models, some studies
compared the model’s performance with manual annotation. When Huang et al.
(2023) examined the performance of ChatGPT on categorizing implicit hate
speech and providing natural language explanations for the implicit hate
speech detection, the results showed great potential of ChatGPT for this and
similar tasks. The authors report that the results are promising, as the model
correctly identified 80% of implicit hateful tweets. What is more, its
explanations were evaluated to be better than those, provided by humans. Based
on this, the authors see a “great potential of ChatGPT as a data annotation
tool”.

In this paper, we examine ChatGPT’s performance on automatic genre
identification, a text classification task, where manual annotation has been
repeatedly shown to be very hard for humans (Egbert et al., 2015; Zu Eissen
and Stein, 2004; Suchomel, 2020). In addition to providing first insights into
the performance of ChatGPT on this task, this is also one of the first studies
of its performance on a language, other than English. We compare ChatGPT’s
zero-shot performance with the X-GENRE classifier – a multilingual XLM-RoBERTa
Transformer-based language model, fine-tuned on manually-annotated genre
datasets. The models are compared on two test sets, English EN-GINCO dataset
and Slovenian GINCO dataset (Kuzman et al., 2022b), in three scenarios:
English prompt with English text, English prompt with Slovenian text and
Slovenian prompt with Slovenian text. The experiments show impressive
performance of ChatGPT on this task, outperforming the fine-tuned LLM on
English test set. In addition, despite Slovenian being an under-resourced
language, ChatGPT’s performance on this language is no worse than on English,
provided that the prompt is in English instead of Slovenian. So much time and
effort went into manual annotation of datasets in numerous languages to fine-
tune the models for genre identification, which does not seem to be necessary
anymore. Is this a point for the NLP community to stop and ask ourselves:
“Have large annotation campaigns became redundant? Can we use ChatGPT to
annotate data for research purposes instead?”

The paper is organized as follows. In Section 2, we introduce the task of
automatic genre identification. We present genre-annotated datasets in Section
3, on which we test the ChatGPT model and the fine-tuned LLM, presented in
Section 4. We discuss the predictions of ChatGPT in Section 5 and compare it
with the fine-tuned model in Section 6. Finally, in Section 7, we conclude the
paper with discussion of the main findings and suggestions for further work.

##  2 Automatic Genre Identification

Automatic genre identification is a text classification task which focuses on
categorizing texts into genre categories, such as News, Legal, Promotion. The
genre categories are traditionally defined based on the author’s purpose, and
the function and conventional form of the text (Orlikowski and Yates, 1994).
Ever since the emergence of the world wide web, researchers both from the
information retrieval as well as the corpora creation and curation field have
approached this task with the aim of creating systems for automatically
identifying genres of texts. This information would allow the users of
information retrieval tools to obtain more relevant hits to their queries. In
addition, the advent of the web allowed for a fast collection of corpora based
on automatic methods. Since a challenge with this approach is that the corpora
creators and users do not know what texts are included in the resulting web
corpus (Baroni et al., 2009), annotating these large collections with genre
information provides valuable insight into the content and quality of a web
corpus. Moreover, genre information was shown to be a beneficial signal for
numerous NLP tasks, such as language processing (Giesbrecht and Evert, 2009;
Müller-Eberstein et al., 2021), machine translation (Van der Wees et al.,
2018), and automatic summarization (Stewart and Callan, 2009).

In contrast to topic detection, genre categories cannot be categorized solely
based on lexical information, such as keywords. The models need to find a
higher pattern in texts, often based on textual or syntactic characteristics,
unrelated to the topic of the document. In addition, classification of genres
was shown to be a hard task because texts can be more or less prototypical
examples of their genre classes, can show signals of multiple classes or lack
signals of any genre (Sharoff, 2021; Zu Eissen and Stein, 2004). That is why
this text categorization task is very challenging for non-neural methods which
were shown to be too dataset-dependent and were not capable of generalizing to
unseen datasets (Sharoff et al., 2010).

Then a breakthrough happened with the advent of deep neural Transformer-based
language models. By using the BERT-like language models that are pre-trained
on massive amounts of texts, and by fine-tuning them on genre identification
task, recent works (Kuzman et al., 2022a; Rönnqvist et al., 2021; Repo et al.,
2021) showed that the models are capable of identifying genres even on unseen
datasets and languages. In addition, they achieve good results already when
trained on only 1,000 texts. However, the models still need manually-annotated
texts, which is a time-consuming and expensive task.

Labels | EN-GINCO | GINCO | X-GENRE training
---|---|---|---
Information/Explanation | 25% | 24% | 17%
Promotion | 22% | 17% | 16%
Opinion/Argumentation | 18% | 11% | 14%
News | 18% | 29% | 19%
Other | 6% | 7% | 4%
Forum | 6% | 5% | 8%
Instruction | 5% | 5% | 12%
Legal | 0% | 1% | 4%
Prose/Lyrical | 0% | 1% | 6%
Table 1: Label distribution in the English test set (EN-GINCO), the Slovenian
test set (GINCO) and the dataset, used for training of the X-GENRE classifier.

##  3 Genre-annotated datasets

To evaluate the models’ performance on English and Slovenian texts, we use
random samples from two manually-annotated datasets: EN-GINCO and GINCO. We
test the models on 100 instances from each dataset. As the X-GENRE classifier
was trained on the training split of GINCO, we sample the instances to test
the models from the test split. In contrast, the EN-GINCO dataset was not part
of the training data, used for fine-tuning the X-GENRE model.

The GINCO dataset (Kuzman et al., 2022b) consists of Slovenian web texts,
originating from two Slovenian web corpora, the slWaC 2.0 corpus (Erjavec and
Ljubešić, 2014) from 2014 and the MaCoCu-sl 1.0
corpus111http://hdl.handle.net/11356/1517 from 2021. The EN-GINCO dataset is a
sample of English texts from the English web corpus
enTenTen20222https://www.sketchengine.eu/ententen-english-corpus (Jakubíček et
al., 2013). The EN-GINCO dataset has not been published online yet, so it is
impossible that ChatGPT would have seen it before. The GINCO dataset was
published in 2021333The dataset is available at
http://hdl.handle.net/11356/1467.

The instances were manually annotated with 24 genre categories from the GINCO
schema by two annotators with linguistic background and following detailed
guidelines for genre annotation444https://tajakuzman.github.io/GINCO-Genre-
Annotation-Guidelines/ (see Kuzman et al. (2022b) for more details on the
annotation procedure). In case of disagreement, the annotators discussed the
instance on which they did not agree, and jointly decided on the final label.
The inter-annotator agreement in GINCO, calculated before the discussions of
the annotators, reached nominal Krippendorff’s alpha of 0.71, which is above
the acceptable threshold of 0.67, defined by Krippendorff (2018). However, the
alpha is well below 0.8 – which represents good reliability of annotated data
– which further confirms the difficulty of manual annotation for this task,
addressed in multiple studies (Suchomel, 2020; Egbert et al., 2015).

###  3.1 Genre schema

In our experiments, we use the X-GENRE genre schema, a generalization of
various schemata applied on different datasets, that is, the CORE (Egbert et
al., 2015), FTD (Sharoff, 2018) and GINCO (Kuzman et al., 2022b) dataset. The
motivation behind the schema is two-fold – it is more user-friendly than any
of the specific schemata in any of the available training datasets, and it
allows for merging training data from different datasets, resulting in a more
robust model. For this purpose, we mapped the original labels in both test
sets, which were annotated with a schema with higher granularity, to the
X-GENRE schema. The final schema consists of 9 labels:
Information/Explanation, Instruction, Legal, News, Opinion/Argumentation,
Promotion, Prose/Lyrical, Forum and Other (for more details, see the
definitions of the labels in Table 3 in the Appendix).

As shown in Table 1, each test set is unbalanced, with 4 more frequent
categories representing more than 80% of instances in each test set, and five
less common categories, representing 7% or less of the set each. Two
categories from the genre schema appear only in the Slovenian test data: Legal
and Prose/Lyrical. From the comparison of distributions, we can also see that
the Slovenian test set has less Promotion and Opinion/Argumentation, while it
has more News texts. As label distribution can impact the performance of the
classifier, we also add into the comparison the distribution of genre classes
in the dataset, on which the X-GENRE classifier was trained on. As shown in
Table 1, the classes that are less frequent in the test sets are more
represented in the training data, and some of the most frequent categories
from the test sets, such as Information/Explanation and News, represent a
smaller percentage of the dataset, compared to the test sets.

##  4 Models

###  4.1 ChatGPT

ChatGPT is a large language model, provided by the OpenAI, that was fine-tuned
on the GPT-3.5 model (OpenAI, 2023). The model was optimized for dialogue
based on the Reinforcement Learning with Human Feedback (Christiano et al.,
2017), that is, a method that uses human feedback to optimize the model’s
answers. Firstly, the language model is fine-tuned on a dataset of prompts and
human-generated answers. Secondly, the model is asked to produce multiple
answers to the prompts from the dataset. Then humans evaluate which answer is
the best, and their feedback is used to learn a reward function. More
specifically, the reward model is trained on the dataset with evaluated
answers to predict which output the labelers would prefer, and the trained
model is used as a reward function. Finally, the reward function is optimized
with reinforcement learning using the proximal policy optimization algorithm
(Ouyang et al., 2022). We use the ChatGPT Feb 13 version and perform the
experiments in the period from February 24th to March 2nd 2023.

###  4.2 X-GENRE

We compare the ChatGPT model with a massively multilingual base-sized XLM-
RoBERTa Transformer-based model (Conneau et al., 2020), fine-tuned on genre-
annotated datasets, hereinafter referred to as the X-GENRE classifier. The
classifier was fine-tuned on around 1,700 instances from three datasets,
manually annotated with genre labels: English CORE (Egbert et al., 2015),
English FTD (Sharoff, 2018) and Slovenian GINCO (Kuzman et al., 2022b)
dataset. Each dataset has its own set of categories, which were then mapped
into a joint schema – the X-GENRE schema. We opted for using multiple datasets
instead of just one in order to assure better generalization of the model to
unseen datasets and languages, and to avoid the model being too dataset-
dependent, which was shown to be an issue in the past (Sharoff et al., 2010).

When we compared the X-GENRE model with models, trained only on one of the
datasets, mentioned above, and each using their original schemata, the X-GENRE
model outperformed all single-dataset-based models, reaching micro and macro
F1 scores between 0.79 and 0.80 in in-dataset testing scenario. We also
compared the X-GENRE classifier with models, trained on a single genre
dataset, to which the categories from the joint (X-GENRE) schema were mapped,
to be able to compare the cross-dataset capabilities of models. The results
showed the X-GENRE model to outperform single-dataset-based models on the EN-
GINCO dataset, with the single-dataset-based models reaching 0.49 to 0.65 in
micro F1 and 0.42 to 0.54 in macro F1555The experiments and results are
documented at https://github.com/TajaKuzman/Genre-Datasets-
Comparison/tree/main/Creation-of-classifiers-and-cross-prediction..

The model is freely available at the Hugging Face
repository666https://huggingface.co/classla/xlm-roberta-base-multilingual-
text-genre-classifier.

##  5 ChatGPT prompt and genre predictions

To classify the texts according to genre categories with ChatGPT, we fed it
prompts using the OpenAI platform and manually extracted the categories and
explanations from its answers. In the prompt, we defined the main criteria for
defining genres and which categories can the model choose from. We also asked
the model to provide an explanation of its choice, and we provided the text to
be classified. The prompt was repeated with each text. We used the genre
classes that are used by the X-GENRE classifier to be able to compare the two
models.

Example of the English prompt: Please classify the following text according to
genre (defined by function of the text, author’s purpose and form of the text)
and explain your decision. You can choose from the following classes: News,
Legal, Promotion, Opinion/Argumentation, Instruction, Information/Explanation,
Prose/Lyrical, Forum, Other. The text to classify: Shower pods install in no
time… <p> 1\. Prepare the floor with the waste and the water supply pipes. <p>
2\. Attach shower equipment to the shower pod shell running flexible tails
(H&C or just C) down back. <p> 3\. Move unit into position connecting water
supplies on the way and the waste outlet trap. <p> 4\. Having secured the
shower pod shell to the building structure doors may now be fitted.

Example of the ChatGPT’s answer:

Genre: Instruction

Explanation: The text provides step-by-step instructions for installing shower
pods. It is structured in a sequential manner, detailing the process of
preparing the floor, attaching shower equipment, moving the unit into
position, and securing the shower pod shell. The language used is imperative,
giving clear and concise commands to the reader. There is no attempt to
persuade, promote, or argue for any particular point of view. Instead, the
purpose of the text is to instruct the reader on how to install shower pods.
Therefore, the text can be classified as an instruction manual or guide.

Test set | Prompt | Model | Micro F1 | Macro F1 | Accuracy
---|---|---|---|---|---
EN-GINCO | EN | ChatGPT | 0.74 | 0.66 | 0.72
X-GENRE | 0.67 | 0.61 | 0.67
GINCO | EN | ChatGPT | 0.75 | 0.64 | 0.75
X-GENRE | 0.91 | 0.91 | 0.91
GINCO | SL | ChatGPT | 0.68 | 0.56 | 0.68
X-GENRE | 0.91 | 0.91 | 0.91
Table 2: Comparison of ChatGPT and fine-tuned X-GENRE model on the two test
sets and with prompts in two languages.

In most cases, the answers followed the form of the example, consisting of the
label in one line, and the explanation in the next paragraph or multiple
paragraphs. However, in some cases, the label was a part of the explanation.

The Slovenian prompt was a translation of the English prompt, where also the
names of the classes were translated into Slovenian. When we used the
Slovenian prompt, ChatGPT’s response with the label and explanation was in
Slovenian as well. However, in one instance, the model started the answer in
Slovenian but finished the explanation in English. An instance of Slovenian
prompt and ChatGPT’s response in Slovenian is shown in Subsection A.1 in the
Appendix.

In a few cases, ChatGPT’s output consisted of two classes instead of one. If
one of the classes matched the true label, we used this class for evaluation.
In addition, in a few cases, ChatGPT answered with a novel label that was not
a part of the genre class set, such as Review and Condolence. We experimented
with asking it to correct its decision and choose a label from the closed set.
As can be seen from the dialogue in Subsection A.2 in the Appendix, the model
apologized and then predicted the correct label.

##  6 Comparison with the fine-tuned model

We compare the two models in three scenarios: on English test set (EN-GINCO)
with English prompt, on Slovenian test set (GINCO) with English prompt, and on
Slovenian test set (GINCO) with Slovenian prompt. In the latter two scenarios,
only the language of the prompt is different, while the text instances to
classify are the same. The results are shown in Table 2.

When both models are tested on the EN-GINCO test set, surprisingly, ChatGPT
outperforms the X-GENRE classifier by 5–7 points in micro F1, macro F1 and
accuracy, achieving micro F1 of 0.74, macro F1 of 0.66 and accuracy of 0.72.
Considering the fact that the X-GENRE classifier was fine-tuned on more than
1,700 manually-annotated texts, which required laborious annotation campaigns,
while ChatGPT was not explicitly trained for this task, these results are
astonishing.

In contrast, when the models are tested on the Slovenian test set, the fine-
tuned X-GENRE classifier significantly outperforms ChatGPT, reaching micro F1,
macro F1 and accuracy scores of 0.91. However, we should note that the X-GENRE
model was trained on the training portion of the GINCO dataset, while the test
split of the same dataset is used for testing in this scenario. Thus, while
the results of the X-GENRE model on the EN-GINCO test set reflect its cross-
dataset performance, the results on the GINCO test set show its in-dataset
performance, which explains its significantly higher results on this dataset.

A more interesting result is the comparison of ChatGPT’s performance on
English texts and Slovenian texts. We can see from Table 2 that ChatGPT’s
performance on Slovenian is comparable to English, despite the fact that
Slovenian is considerably less present in the training data, used for pre-
training and fine-tuning the ChatGPT model, since it is an under-resourced
language.

In contrast, the performance of ChatGPT drops once we use the Slovenian prompt
instead of the English prompt. As shown in Table 2, on Slovenian prompt,
ChatGPT achieves 7 to 8 points lower micro F1, macro F1 and accuracy scores
than on English prompt, while the text instances are the same. This shows that
while the language of the text to be predicted does not have a large impact,
the language of the prompt does have an effect on the performance of ChatGPT
on automatic genre identification.

Figure 1: Comparison of differences in correct and incorrect predictions
between ChatGPT and X-GENRE.

To analyze the differences between the two models further, we examined the
cases where the models did not agree in their predictions. We performed the
analysis on the results of the experiments on the EN-GINCO test set, where an
English prompt and English text were used. As we can see from Figure 1, both
models mostly correctly predicted genre categories, with ChatGPT being
accurate in 72% of instances and X-GENRE in 67% of instances. In 69% the
responses of the models match, which means that they were either both correct
or both incorrect. 18 instances were correctly predicted by ChatGPT, but
incorrectly predicted by X-GENRE, while the X-GENRE model correctly predicted
13 instances to which ChatGPT assigned wrong labels. Out of all instances
where ChatGPT is correct, while X-GENRE is incorrect, 39% are of
Information/Explanation genre and 33% belong to Promotion. In contrast,
instances where X-GENRE is correct and ChatGPT is incorrect mostly belong to
Opinion/Argumentation genre (38%) and Information/Explanation (23%). We can
conclude from this analysis that, as expected given the different nature of
the two models, the outputs of the two models are quite complementary, and
that a merge of the outputs could be considered in specific use cases.

##  7 Conclusions

Up to now, supervised machine learning was the method which achieves the
highest results in most NLP tasks. It consists of manually annotating
thousands of texts with labels, preferably by multiple annotators, on which
the machine learning systems are to be trained and tested. However, manual
annotation requires a lot of time, human effort and money. What is more,
despite great efforts to assure reliable annotation, the annotation campaigns
often result in poor inter-annotator agreement, and with that, poor
reliability of the annotated data. This reflects how difficult these tasks are
even for humans. Results from some previous studies (Huang et al., 2023) and
our experiments hint that ChatGPT could substitute large manual annotation
campaigns, which would significantly change the workflow of researchers in
NLP.

In this paper, we analyze ChatGPT’s zero-shot performance on the task of
automatic genre identification. We compare it with the X-GENRE classifier, a
Transformer-based language model that was fine-tuned on more than 1,700 texts,
manually-annotated with genres in three annotation campaigns and two
languages. Surprisingly, the results show that when the models are compared on
a dataset, on which neither of them was trained, ChatGPT outperforms the
X-GENRE classifier. Until now, the paradigm in text classification was that
the better and the more manually-annotated data, the better the model.
However, these results might hint at a new era for text categorization, where
only minimal manual annotation for test sets would be necessary.

Despite the very promising results on the English dataset, our expectation was
that the ChatGPT model would not be up to the task on texts in other
languages, especially in Slovenian as an under-resourced language. However,
when the model was applied both on English and Slovenian genre-annotated
dataset, the results show stable performance regardless of the language. In
contrast, once the model is also prompted in the under-resourced language, the
results start to deteriorate. While this result shows the obvious limitation
of the current ChatGPT model, it does not mean much for the data-annotation-
oriented usage of the model in the research community, as prompting the model
in English surely does not present any challenge for researchers.

However, the difference in results, impacted by the prompt, indicates a need
for more detailed research on prompts in this task. A possible direction for
further work would be to experiment with more advanced prompting techniques to
find the structure which enables us to get the most out of the ChatGPT model,
such as manual few-shot chain-of-thought prompting (Wei et al., 2022) which
was shown to significantly improve ChatGPT’s performance on classification
tasks (Zhong et al., 2023). In addition, as multiple genre schemata exist of
various granularity, it would be interesting to compare how well does ChatGPT
predict genres based on different schemata, e.g., the CORE schema (Egbert et
al., 2015), FTD schema (Sharoff, 2018), GINCO schema (Kuzman et al., 2022b)
and so on, compared to the X-GENRE schema, used in this research.

In addition, we plan to extend the comparison in multiple ways. Firstly, while
the present experiments were performed on small test sets of 100 instances,
recently, an official ChatGPT API has been made available, which will allow
for much quicker classification experiments. Thus, we plan to extend the
comparison to bigger test sets in more languages and from more genre-annotated
datasets. Secondly, like some of previous studies (Qin et al., 2023; Hendy et
al., 2023), we plan to extend the comparison to other GPT models, such as the
GPT-3.5 model, as well as more BERT-like models, fine-tuned on different
genre-annotated datasets.

Furthermore, as web texts often do not undergo any review, automatic genre
identification was shown to be a hard task also due to hybrid texts, that is,
texts having characteristics of multiple genres. In addition, each document
from web corpora consists of all text extracted from one web page.
Consequently, another challenge for genre identification are multi-part
documents, where a document consists of two or more separate texts of separate
genres, such as a document from a news portal that consists of a news article
and user comments. In our preliminary experiments, ChatGPT assigned multiple
classes to some texts, and was also shown to be able to recognize that a
document consists of different parts, and assign genre classes to each part of
the document. Thus, as further work, we plan to explore what are the
capabilities of ChatGPT for multi-label classification and for genre
identification on spans of documents.

Finally, our results have sparked questions whether extensive manual
annotation is still needed for such text classification tasks. In the future,
we plan to explore the impact of the sizes of manually-annotated datasets to
the performance of fine-tuned models, compared to the zero-shot performance of
ChatGPT. In addition, as using ChatGPT and similar models for annotation of
very large text collections is computationally expensive, we might experiment
with using large language models to annotate the training data as an
alternative to manual annotation, and then fine-tune base-sized models for
annotation of large quantities of data.

## Acknowledgements

This work has received funding from the European Union’s Connecting Europe
Facility 2014-2020 - CEF Telecom, under Grant Agreement No.
INEA/CEF/ICT/A2020/2278341. This communication reflects only the author’s
view. The Agency is not responsible for any use that may be made of the
information it contains. This work was also funded by the Slovenian Research
Agency within the Slovenian-Flemish bilateral basic research project
“Linguistic landscape of hate speech on social media” (N06-0099 and
FWO-G070619N, 2019–2023) and the research programme “Language resources and
technologies for Slovene” (P6-0411).

## References

  * Baroni et al. (2009) Marco Baroni, Silvia Bernardini, Adriano Ferraresi, and Eros Zanchetta. 2009.  The WaCky wide web: a collection of very large linguistically processed web-crawled corpora.  _Language resources and evaluation_ , 43(3):209–226.
  * Christiano et al. (2017) Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017.  Deep reinforcement learning from human preferences.  _Advances in neural information processing systems_ , 30.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised Cross-lingual Representation Learning at Scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451.
  * Egbert et al. (2015) Jesse Egbert, Douglas Biber, and Mark Davies. 2015.  Developing a bottom-up, user-based method of web register classification.  _Journal of the Association for Information Science and Technology_ , 66(9):1817–1831.
  * Erjavec and Ljubešić (2014) Tomaž Erjavec and Nikola Ljubešić. 2014.  The slWaC 2.0 corpus of the Slovene web.  _T. Erjavec, J. Žganec Gros (ur.) Jezikovne tehnologije: zbornik_ , 17:50–55.
  * Giesbrecht and Evert (2009) Eugenie Giesbrecht and Stefan Evert. 2009.  Is part-of-speech tagging a solved task? An evaluation of POS taggers for the German web as corpus.  In _Proceedings of the fifth Web as Corpus workshop_ , pages 27–35.
  * Guo et al. (2023) Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. 2023.  How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection.  _arXiv preprint arXiv:2301.07597_.
  * Hendy et al. (2023) Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023.  How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation.  _arXiv preprint arXiv:2302.09210_.
  * Huang et al. (2023) Fan Huang, Haewoon Kwak, and Jisun An. 2023.  Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech.  _arXiv preprint arXiv:2302.07736_.
  * Jakubíček et al. (2013) Miloš Jakubíček, Adam Kilgarriff, Vojtěch Kovář, Pavel Rychlỳ, and Vít Suchomel. 2013.  The TenTen corpus family.  In _7th international corpus linguistics conference CL_ , pages 125–127. Lancaster University.
  * Kenton and Toutanova (2019) Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  pages 4171–4186.
  * Kocmi et al. (2022) Tom Kocmi, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Thamme Gowda, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Rebecca Knowles, Philipp Koehn, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Michal Novák, Martin Popel, and Maja Popović. 2022.  Findings of the 2022 conference on machine translation (WMT22).  In _Proceedings of the Seventh Conference on Machine Translation (WMT)_ , pages 1–45, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  * Krippendorff (2018) Klaus Krippendorff. 2018.  _Content analysis: An introduction to its methodology_.  Sage publications.
  * Kuzman et al. (2022a) Taja Kuzman, Nikola Ljubešić, and Senja Pollak. 2022a.  Assessing Comparability of Genre Datasets via Cross-Lingual and Cross-Dataset Experiments.  In _Jezikovne tehnologije in digitalna humanistika: zbornik konference_ , Jezikovne tehnologije in digitalna humanistika: zbornik konference, page 100–107. Institute of Contemporary History.
  * Kuzman et al. (2022b) Taja Kuzman, Peter Rupnik, and Nikola Ljubešić. 2022b.  The GINCO Training Dataset for Web Genre Identification of Documents Out in the Wild.  In _Proceedings of the Language Resources and Evaluation Conference_ , pages 1584–1594, Marseille, France. European Language Resources Association.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Müller-Eberstein et al. (2021) Max Müller-Eberstein, Rob van der Goot, and Barbara Plank. 2021.  Genre as Weak Supervision for Cross-lingual Dependency Parsing.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 4786–4802.
  * OpenAI (2023) OpenAI. 2023.  ChatGPT General FAQ.  https://help.openai.com/en/articles/6783457-chatgpt-general-faq.  Accessed: March 3, 2023.
  * Orlikowski and Yates (1994) Wanda J Orlikowski and JoAnne Yates. 1994.  Genre repertoire: The structuring of communicative practices in organizations.  _Administrative science quarterly_ , pages 541–574.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022\.  Training language models to follow instructions with human feedback.  _arXiv preprint arXiv:2203.02155_.
  * Qin et al. (2023) Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023.  Is ChatGPT a General-Purpose Natural Language Processing Task Solver?  _arXiv preprint arXiv:2302.06476_.
  * Repo et al. (2021) Liina Repo, Valtteri Skantsi, Samuel Rönnqvist, Saara Hellström, Miika Oinonen, Anna Salmela, Douglas Biber, Jesse Egbert, Sampo Pyysalo, and Veronika Laippala. 2021.  Beyond the English web: Zero-shot cross-lingual and lightweight monolingual classification of registers.  In _16th Conference of the European Chapter of the Associationfor Computational Linguistics: Student Research Workshop, EACL 2021_ , pages 183–191. Association for Computational Linguistics (ACL).
  * Rönnqvist et al. (2021) Samuel Rönnqvist, Valtteri Skantsi, Miika Oinonen, and Veronika Laippala. 2021\.  Multilingual and Zero-Shot is Closing in on Monolingual Web Register Classification.  In _Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa)_ , pages 157–165.
  * Sharoff (2018) Serge Sharoff. 2018.  Functional text dimensions for the annotation of web corpora.  _Corpora_ , 13(1):65–95.
  * Sharoff (2021) Serge Sharoff. 2021.  Genre annotation for the web: text-external and text-internal perspectives.  _Register studies_.
  * Sharoff et al. (2010) Serge Sharoff, Zhili Wu, and Katja Markert. 2010.  The Web Library of Babel: evaluating genre collections.  In _LREC_. Citeseer.
  * Stewart and Callan (2009) Jade Goldstein Stewart and J Callan. 2009.  _Genre oriented summarization_.  Ph.D. thesis, Carnegie Mellon University, Language Technologies Institute, School of Computer Science.
  * Suchomel (2020) Vít Suchomel. 2020.  Genre Annotation of Web Corpora: Scheme and Issues.  In _Proceedings of the Future Technologies Conference_ , pages 738–754. Springer.
  * Van der Wees et al. (2018) Marlies Van der Wees, Arianna Bisazza, and Christof Monz. 2018.  Evaluation of machine translation performance across multiple genres and languages.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018.  GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 353–355.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022.  Chain of Thought Prompting Elicits Reasoning in Large Language Models.  In _Advances in Neural Information Processing Systems_.
  * Zhang et al. (2022) Bowen Zhang, Daijun Ding, and Liwen Jing. 2022.  How would Stance Detection Techniques Evolve after the Launch of ChatGPT?  _arXiv preprint arXiv:2212.14548_.
  * Zhong et al. (2023) Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. 2023.  Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERT.  _arXiv preprint arXiv:2302.10198_.
  * Zu Eissen and Stein (2004) Sven Meyer Zu Eissen and Benno Stein. 2004.  Genre classification of web pages.  In _Annual Conference on Artificial Intelligence_ , pages 256–269. Springer.

##  Appendix A Appendix

###  A.1 Slovenian prompt and answer

Slovenian prompt:

Prosimo, določi naslednjemu besedilu, kakšnega žanra je (ki ga definiramo na
podlagi funkcije besedila, avtorjevega namena in oblike besedila) in utemelji
svojo odločitev. Izbiraš lahko med naslednjimi kategorijami: Novica, Pravno
besedilo, Promocijsko besedilo, Mnenje/Argumentativno besedilo, Navodila,
Informativno/razlagalno besedilo, Proza/poezija, Forum, Drugo. Besedilo, ki mu
določi žanr: Opis: Onesnaževala iz cestnega prometa so eden izmed glavnih
virov, ki vplivajo na koncentracije delcev PM10 v mestih, ter negativno
vplivajo na človekovo zdravje. V diplomski nalogi obravnavamo korelacije med
kakovostjo zunanjega zraka z delci PM10 in prometnimi podatki iz avtomatskih
števnih mest v Mariboru. Po naših ugotovitvah korelacij med omenjenimi
parametri ni, zato smo se odločili za postavitev novega merilnega mesta v
središču Maribora. Z novim merilnim mestom bomo dobili bolj izpopolnjeno
merilno mrežo v MOM, boljše informacije o kakovosti zraka v središču mesta
hkrati pa možnost boljše prepoznave zveze med prometom in koncentracijami
delcev PM10. Najdeno v: ključnih besedah Povzetek najdenega:… Onesnaževala iz
cestnega prometa so eden izmed glavnih… Ključne besede: promet, Maribor,
onesnaževala, prašni delci PM10 Objavljeno: 27.09.2011 Ogledov: 719 Prenosov:
73 Polno besedilo (20,99 MB)

Chat GPT’s response:

Žanr tega besedila je Informativno/razlagalno besedilo. Besedilo je namenjeno
predstavitvi rezultatov diplomske naloge, ki raziskuje korelacije med
kakovostjo zunanjega zraka in prometom v Mariboru ter ponuja predlog za
postavitev novega merilnega mesta v središču mesta. Besedilo vsebuje
znanstvene podatke in razlage, ki so namenjene informiranju bralca o
problematiki onesnaževanja zraka v mestu. Besedilo ni namenjeno prepričevanju
bralca, ampak zgolj predstavljanju dejstev in ugotovitev raziskave.

###  A.2 ChatGPT correction

In the following example, ChatGPT chose the label outside of the class set,
and after an additional prompt, corrected itself, ruling out all classes and
correctly identifying the text as Other.

Prompt:

Please classify the following text according to genre (defined by function of
the text, author’s purpose and form of the text) and explain your decision.
You can choose from the following classes: News, Legal, Promotion,
Opinion/Argumentation, Instruction, Information/Explanation, Prose/Lyrical,
Forum, Other. The text to classify: Condolence: Dear George,Charlene,Michael
and family; I am so sorry to see you have lost your beloved Yolande. She was a
wonderful person to know and converse with…always with a warm friendly smile
and a twinkle in her eye(:>) Marion will sure miss playing dominos with her at
the Camp. May your fond memories give you strength and hope knowing you will
be reunited again. Jesus cares and is praying for you. May you sense His
presence and peace in your time of sorrow. Love in Christ Jan and Bill

ChatGPT’s answer:

The genre of the given text is "Condolence," which is a type of personal
communication expressing sympathy and sorrow upon someone’s loss or
bereavement. The purpose of the text is to offer condolences and comfort to
the family of the deceased. The text’s form is informal and personal, written
in the form of a letter to the bereaved family, expressing the author’s
emotions and feelings about the loss. Therefore, the text is not a news item,
legal document, promotional material, instructional guide, or forum post. It
is not a piece of prose or lyrical writing, nor is it an argumentative opinion
piece. Therefore, it does not fit any of the other genres listed.

Researcher’s response:

"Condolence" is not in the list of allowed classes. Please choose from the
following classes: News, Legal, Promotion, Opinion/Argumentation, Instruction,
Information/Explanation, Prose/Lyrical, Forum, Other, and explain your
decision.

ChatGPT’s response:

Apologies for the confusion. I would classify the given text under the "Other"
category. The text is a personal message expressing condolences and offering
comfort to the family of the deceased. It is not a formal document, news item,
legal statement, promotional material, instructional guide, or forum post. It
is not an opinion or argumentative piece nor is it purely informative or
explanatory. The text’s purpose is to convey sympathy, share memories, and
offer emotional support, making it a form of personal communication.
Therefore, the "Other" category seems to be the most appropriate
classification for this text.

###  A.3 Genre categories

Label |  Description |  Examples
---|---|---
Information/Explanation |  An objective text that describes or presents an event, a person, a thing, a concept etc. Its main purpose is to inform the reader about something. |  research article, encyclopedia article, product specification, course materials, biographical story/history
Instruction |  An objective text which instructs the readers on how to do something. |  how-to texts, recipes, technical support
Legal |  An objective formal text that contains legal terms and is clearly structured. |  small print, software license, terms and conditions, contracts, law, copyright notices
News |  An objective or subjective text which reports on an event recent at the time of writing or coming in the near future. |  news report, sports report, police report, announcement
Opinion/Argumentation |  A subjective text in which the authors convey their opinion or narrate their experience. It includes promotion of an ideology and other non-commercial causes. |  review, blog, editorial, letter to editor, persuasive article or essay, political propaganda
Promotion |  A subjective text intended to sell or promote an event, product, or service. It addresses the readers, often trying to convince them to participate in something or buy something. |  advertisement, e-shops, promotion of an accommodation, promotion of company’s services, invitation to an event
Prose/Lyrical |  A literary text that consists of paragraphs or verses. A literary text is deemed to have no other practical purpose than to give pleasure to the reader. Often the author pays attention to the aesthetic appearance of the text. It can be considered as art. |  lyrics, poem, prayer, joke, novel, short story
Forum |  A text in which people discuss a certain topic in form of comments. |  discussion forum, reader/viewer responses, QA forum
Other |  A text which does not fall under any other genre category. |
Table 3: Descriptions of genre labels, with examples.