# Critical behaviour of the extended-ballistic transition for pulled self-
avoiding walks

C. J. Bradly<EMAIL_ADDRESS>New Zealand Institute of Advanced Studies
and Centre for Theoretical Chemistry and Physics, Massey University, Auckland
0632, New Zealand  School of Mathematics and Statistics, University of
Melbourne, Victoria 3010, Australia  A. L. Owczarek<EMAIL_ADDRESS>School of Mathematics and Statistics, University of Melbourne, Victoria 3010,
Australia

###### Abstract

In order to study the competition of pulling a long chain polymer with its
other system properties models of lattice polymers accomodate an applied
pulling force to some part of a lattice polymer in addition to features such
as energetic interaction between the lattice polymer and a surface. However,
the critical behaviour of the pulling force alone is less well studied, such
as characterizing the nature of the phase transition and particularly the
values of the associated exponents. We investigate a simple model of lattice
polymers subject to forced extension, namely self-avoiding walks (SAWs) on the
square and simple cubic lattices with one endpoint attached to an impermeable
surface and a force applied to the other endpoint acting perpendicular to the
surface. In the thermodynamic limit the system undergoes a transition to a
ballistic phase as the force is varied and it is known that this transition
occurs whenever the magnitude of the force is positive, i.e.
$f>f_{\text{c}}=0$. Using well established scaling arguments we show that the
crossover exponent $\phi$ for the finite-size model is identical to the well-
known exponent $\nu_{d}$, which controls the scaling of the size of the
polymer in $d$-dimensions. With extensive Monte Carlo simulations we test this
conjecture and show that the value of $\phi$ is indeed consistent with the
known values of $\nu_{2}=3/4$ and $\nu_{3}=0.587597(7)$. Scaling arguments, in
turn, imply the specific heat exponent $\alpha$ is $2/3$ in two dimensions and
$0.29815(2)$ in three dimensions.

##  I Introduction

Lattice models of polymers have been extended to model a huge variety of
different physical situations. One well studied addition to lattice polymer
models is the application of an external force that acts to deform the polymer
chain, possibly with reference to some surface to which it is attached. These
models apply to real applications with atomic force microscopy [1, 2, 3, 4] or
optical tweezer experiments [5]. The standard for lattice polymer models is
the self-avoiding walk (SAW) and its variants (self-avoiding trails, self
avoiding polygons etc.) Typically, an applied force is included in a SAW model
to complement some other interaction or geometric constraint. A force can be
applied to a self-interacting SAW to transition from a collapsed to a
stretched phase [6], or force desorption from an interacting surface [7, 8,
9]. The force can be applied in different ways such as applying it at the
midpoint of the polymer [10] or an arbitrary interior point [11], or it can be
applied to move two parallel slabs that confine the polymer [12]. A pulling
force also appears in models of non-linear lattice polymers such as polygons
[13], stars [14, 15, 16], or more general lattice animals as well as block
copolymers [17].

The critical properties of this wide range of models have received a lot of
attention, but less work has been applied to the behaviour of lattice polymer
models that feature only the applied force [12, 18, 6], perhaps because it is
among the simplest additions to the bare SAW model. When a force is applied to
say the endpoint of a SAW, with the other end terminally attached to some
surface, then the SAW behaves ballistically in that its size from endpoint to
endpoint is proportional to the number of steps in the walk, i.e. $R\sim n$.
In the thermodynamic limit of long lengths this occurs for any magnitude of
force so there is a critical point at zero force, where the SAW does not
behave ballistically. This location of this critical point at $f=0$ has been
proved exactly [18, 6], but further properties of this transition are less
well understood. Some previous work [12] has included Monte Carlo simulation
of SAW models with a force applied in slightly different ways but provided
limited estimation of critical exponents.

In this paper we argue that standard (finite length) scaling arguments for a
continuous transition applied to this transition leads directly to a
conjecture for the crossover exponent $\phi$ and strength of the phase
transition (via the specific heat exponent $\alpha$) when a pulling force is
applied to the end of a polymer in two and three dimensions (in fact, in any
dimension $d\geq 2$). We support this conjecture with strong numerical
evidence from Monte Carlo simulations of SAWs on the square and cubic lattices
in half spaces where one end is attached to the surface (wall) and a force
perpendicular to the wall is applied to the other end. If the size (say radius
of gyration) scaling exponent of a free extended polymer in dilute solution is
$\nu_{d}$ in dimension $d$ then our conjecture is that

$\displaystyle\phi=\nu_{d},\quad\alpha=2-\frac{1}{\nu_{d}}.$ (1)

In particular, for two dimensions

$\displaystyle\phi=\nu_{2}=\frac{3}{4},\quad\alpha=\frac{2}{3}$ (2)

whilst for three dimensions

$\displaystyle\phi=\nu_{3}=0.587597(7),\quad\alpha=0.29815(2),$ (3)

based on the best estimate of $\nu_{3}$ [19]. It should be noted that
$\nu_{d}\geq 1/2$ for all dimensions $d\geq 2$. It is sometimes inferred [12]
that this crossover exponent is close to $1/2$ as it is for the adsorption
transition but our numerical evidence discounts this assumption.

##  II Model and scaling arguments

We consider single polymers in dilute solution modeled as self-avoiding walks
(SAWs) on a square or simple cubic lattice lattice. If the coordinates of each
vertex of the walk lie in the space of coordinated defined by
$(x_{1},\ldots,x_{d})$ with $x_{i}\in\mathbb{Z}$ for $i=1,\ldots,d$, giving a
$d$-dimensional hypercubic lattice $\mathbb{Z}_{d}$, then we consider walks
that are restricted to the half-space defined by $x_{d}\geq 0$. One endpoint
of the walk is tethered to the surface at the origin and the other endpoint is
pulled by a force $f$ perpendicular to the impermeable surface at $x_{d}=0$.
This situation is illustrated for two dimensions in Fig. 1. Let the
coordinates of the vertices be denoted as $(x_{1,j},\ldots,x_{d,j})$ for
$j=0,\ldots,n$. We denote by $h=x_{d,n}$ the height of the pulled endpoint and
by $a_{nh}$ the number of such walks on the lattice that are of length (number
of steps) $n$. The number of all walks of length $n$ confined to one half-
space is $\sum_{h}a_{nh}$ and it is known that this quantity has the same
large $n$ limit as unrestricted SAWs on a hypercubic lattice [20]. That is,
$\lim_{n\to\infty}\tfrac{1}{n}\log\sum_{h}a_{nh}=\log\mu_{d}$, where $\mu_{d}$
is the growth constant of the $d$-dimensional hypercubic lattice [21].

Figure 1:  A self-avoiding walk on the square lattice with one endpoint
attached to an impermeable surface and the other pulled away from that surface
by a force $f$. The height of the pulled endpoint above the surface is $h$.

If the height of the endpoint is due to a force $f$ then we associate a
Boltzmann factor $y^{h}$ where $y=\exp(f/k_{\text{B}}T)$. The canonical
partition function of such walks of length $n$ with endpoint at height $h$ is

$Z_{n}(f)=\sum_{h}a_{nh}\,y^{h}.$ (4)

The (reduced) finite-length free energy is

$\lambda_{n}(f)=-\frac{1}{n}\log Z_{n}(f).$ (5)

and we consider the average height per step

$h_{n}(f)=\frac{\langle
h\rangle(f)}{n}=\frac{1}{n}\frac{\sum_{h}h\,a_{nh}\,y^{h}}{Z_{n}(f)}=-\frac{d\lambda_{n}(f)}{df}$
(6)

and its variance (also the specific heat)

$c_{n}(f)=\frac{\operatorname{var}(h)}{n}=\frac{\langle h^{2}\rangle-\langle
h\rangle^{2}}{n}=\frac{dh_{n}(f)}{df}.$ (7)

###  II.1 Thermodynamic scaling

In the thermodynamic, or long-length, limit the free energy is
$\lambda(f)=\lim_{n\to\infty}\lambda_{n}(f)$ and it is known that $\lambda(f)$
is singular at exactly $f_{c}=0$, or equivalently $y_{c}=1$, which is the
location of the extended-ballistic transition [18, 22, 6]. This holds for all
dimensions $d\geq 2$. Considering the thermodynamic limits

$H(f)=\lim_{n\rightarrow\infty}h_{n}(f)$ (8)

and

$C(f)=\lim_{n\rightarrow\infty}c_{n}(f)$ (9)

then the transition is signified by the behaviour

$H(f)=C(f)=0\mbox{ for }f\leq 0$ (10)

and

$H(f)>0\mbox{ and }C(f)>0\mbox{ for }f>0$ (11)

the average height per step can be considered an order parameter for the
transition.

The standard scaling Ansätze for a continuous transition near a critical point
$f_{\text{c}}$ at large length $n$ are

$c_{n}(f)\sim
n^{\alpha\phi}{\mathcal{C}}\left(\left[f-f_{\text{c}}\right]n^{\phi}\right)\mbox{
as }n\to\infty\mbox{ as }f\to f_{\text{c}}$ (12)

where $\phi<1$ is the crossover exponent, and

$h_{n}(f)\sim
n^{\alpha\phi-\phi}{\mathcal{H}}\left(\left[f-f_{\text{c}}\right]n^{\phi}\right)\mbox{
as }n\rightarrow\infty\mbox{ as }f\rightarrow f_{\text{c}}.$ (13)

A continuous transition is signified by $\alpha<1$. Standard arguments give a
relation between $\phi$ and $\alpha$

$2-\alpha=\frac{1}{\phi},$ (14)

so that when $f=f_{\text{c}}=0$ we have

$h_{n}(0)\sim H\,n^{\phi-1}$ (15)

and

$c_{n}(0)\sim C\,n^{2\phi-1}.$ (16)

The constants $H$ and $C$ are the values of the scaling functions
$\mathcal{H}$ and $\mathcal{C}$ at $f=0$.

Now at $f=0$ given that the height of the polymer end will scale in proportion
to the size of the polymer generally given the polymer has a free end it would
also be expected that

$\langle h\rangle(0)\sim A\;n^{\nu_{d}},$ (17)

where $\nu_{d}$ is the isotropic free $d$-dimensional size scaling exponents:
in two dimensions it is predicted that $\nu_{2}=3/4$ and in three dimensions
the best estimate is $\nu_{3}=0.587597(7)$ [19]. Hence we can deduce a
particular scaling relation for this pulling-pushing transition

$\phi=\nu_{d}.$ (18)

Although this relation is simple, it will hold in all models with a force that
changes from pulling to pushing.

Away from the critical point where the endpoint is subject to a pulling or
pushing force, the value of $\nu$ can change and we should look at some other
size-related quantities. Let the end point of our walks be at $(x_{1,n},h)$ or
$(x_{1,n},x_{2,n},h)$ for two and three dimensions respectively, and let the
walk fit into a slab or slit of no less than
$s=\max\\{x_{d,j},j=0,\ldots,n\\}$ units high (that is, all sites of the walk
have height less and or equal to $s$. We note $s\geq h$, and

$r_{x}=\left[\sum_{i=1}^{d-1}x_{i,n}^{2}\right]^{1/2}$ (19)

is shorthand for the distance from the origin to the projection of the
endpoint onto the impermeable surface, generalised for all dimensions $d$.
Similar to the discussion above, we define size exponents $\nu^{(h)}$,
$\nu^{(x)}$ and $\nu^{(s)}$ at fixed $f$

$\displaystyle\langle h\rangle(f)$ $\displaystyle\sim H\,n^{\nu^{(h)}}\,,$
(20a) $\displaystyle\langle r_{x}\rangle(f)$ $\displaystyle\sim
X\,n^{\nu^{(x)}}\,,$ (20b) $\displaystyle\langle s\rangle(f)$
$\displaystyle\sim S\,n^{\nu^{(s)}}\,.$ (20c)

The expected values of these exponents in each phase and at the critical point
is summarised in Table 1. In the extended phase, where $f<0$, or equivalently
$y<1$, the endpoint is pushed into the surface, that is, $\langle h\rangle=0$
and hence $\nu^{(h)}=0$. However, the pushing force is local to the endpoint
and the size of the rest of the polymer is also characterised by the same
exponents $\nu_{d}$. As $f\to-\infty$, or equivalently $y\to 0$, the pushed
endpoint is effectively fixed in the surface, equivalent to a self-avoiding
loop [23]. The ballistic phase, $f>0$, is so named because it is characterised
by the behaviour $h_{n}\sim O(1)$ and thus it is known that $\nu^{(h)}=1$ for
SAWs [22]. Since $s$ is an alternate measure of the extension of the polymer
when pulled away from the surface, we expect $\nu^{(s)}=1$ as well. The
polymer is stretched in this phase so the component of the endpoint position
parallel to the surface is small and we have $\nu^{(x)}=1/2$.

Phase | $f$ | $\nu^{(h)}$ | $\nu^{(x)}$ | $\nu^{(s)}$
---|---|---|---|---
Extended (pushed) | $f<f_{c}$ | 0 | $\nu_{d}$ | $\nu_{d}$
Critical point | $f=f_{c}$ | $\nu_{d}$ | $\nu_{d}$ | $\nu_{d}$
Ballistic (pulled) | $f>f_{c}$ | 1 | $\frac{1}{2}$ | 1
Table 1: Size exponent values for the phases in our model. The critical point
is at $f_{\text{c}}=0$, or equivalently $y_{\text{c}}=1$.

###  II.2 Simulation details

Walks are simulated using the flatPERM algorithm [24], an extension of the
pruned and enriched Rosenbluth method (PERM) [25]. The simulation works by
growing a walk on a given lattice up to some maximum length $N_{\text{max}}$.
At each step the number of bulk interactions $m$ and straight segments $s$ are
calculated and the cumulative Rosenbluth & Rosenbluth weight [26] is compared
with the current estimate of the weights of all samples $W_{nh}$. If the
current state has relatively low weight the walk is ‘pruned’ back to an
earlier state. On the other hand, if the current state has relatively high
weight, then microcanonical quantities $h$ are measured and $W_{nh}$ is
updated. The state is then ‘enriched’ by branching the simulation into several
possible further paths (which are explored when the current path is eventually
pruned back). When all branches are pruned a new iteration is started from the
origin. FlatPERM enhances this method by altering the prune or enrich choice
such that the sample histogram is flat in the microcanonical parameters $n$
and $h$. Further improvements are made to account for the correlation between
branches that are grown from the same enrichment point, which provides an
estimate of the number of effectively independent samples. The main output of
the simulation are the weights $W_{nh}$, which are an approximation to the
athermal density of states $a_{n,h}$ in Eq. (4), for all $n\leq
N_{\text{max}}$.

Once the simulation is finished thermodynamic quantities are determined by
specifying the Boltzmann weight $y$ and using the weighted sum

$\langle Q\rangle_{n}=\frac{\sum_{h}Q_{nh}y^{h}W_{nh}}{\sum_{h}y^{h}W_{nh}}.$
(21)

In particular, we calculate $h_{n}$ and its variance $c_{n}$ according to Eqs.
6 and 7 as well as $\langle x\rangle$ and $\langle s\rangle$. Similarly we
also calculate the third derivative of the specific heat

$t_{n}(f)=\frac{\langle h^{3}\rangle-3\langle h\rangle\langle
h^{2}\rangle+2\langle h\rangle^{3}}{n}.$ (22)

We use the flatPERM algorithm to simulate walks up to length
$N_{\text{max}}=1024$. Because this model is simple we can afford to run very
long simulations to obtain good statistics. For both 2D and 3D models we have
$1.2\times 10^{7}$ iterations, obtaining $5.2\times 10^{12}$ samples for 2D
walks and $6.3\times 10^{12}$ samples for 3D walks. For each case or
simulation mentioned in this work the results are comprised of a composite of
ten independent simulations in order to obtain some measure of statistical
error. We ran separate smaller simulations ($N_{\text{max}}=512$) to obtain
data for $\langle s\rangle$ in order to verify scaling predications.

Figure 2:  Thermodynamic quantities for pulled SAWs in 2D (top) and 3D
(bottom), for several different lengths $n$. From left to right are the
average extension above the surface $h_{n}$, the specific heat $c_{n}$, and
the third derivative of the free energy $t_{n}$.

##  III Results

We first show in Fig. 2 several thermodynamic quantities for the 2D walks (top
row) and 3D walks (bottom row) as functions of the Boltzmann weight
$y=e^{f/k_{B}T}$. The left plots show the average extension above the
impermeable surface, $h_{n}$, the specific heat $c_{n}$ and the third
derivative of the free energy $t_{n}$. All plots show example curves for
$n=256,512,1024$. There is a clear signal of a transition near $y=1$ ($f=0$),
most notably as a peak in $c_{n}$ that grows with $n$. The behaviour of
$h_{n}$ is suggestive of a continuous transition.

lattice |  | $h_{n}$ | $c_{n}$ | $t_{n}$
---|---|---|---|---
squ | $\alpha$ | $0.6630\pm 0.0024$ | $0.6673\pm 0.0031$ | $0.638\pm 0.020$
| $\phi$ | $0.7479\pm 0.0009$ | $0.7504\pm 0.0011$ | $0.734\pm 0.007$
sc | $\alpha$ | $0.2997\pm 0.0029$ | $0.3038\pm 0.0033$ | $0.300\pm 0.009$
| $\phi$ | $0.5881\pm 0.0009$ | $0.5896\pm 0.0010$ | $0.5882\pm 0.0026$
Table 2: Critical exponents $\alpha$ and $\phi$ estimated from the scaling of
thermodynamic quantities at the critical point $f_{\text{c}}$ for pulled SAWs
on the square and simple cubic lattices.

In order to verify the type of transition we estimate the critical exponents
using a correction-to-scaling method. That is, for a quantity $Q_{n}$ that has
power-law leading order behaviour $n^{b}$ for some expected exponent $b$, we
fit the data to

$Q_{n}\sim A\,n^{b}\left(1+C\,n^{-\Delta}\right),$ (23)

where $A$ and $C$ are fitting parameters and $\Delta$ is a correction to
scaling parameter. There are a range of corrections to scaling exponents that
have been seen in the literature [19, 27, 28, 29, 30] where $n^{-\Delta}$ is
only the first correction term of an infinite series, but for our purposes we
have assumed that a single term with $\Delta=1/2$ for three dimensions and
$\Delta=1$ for two dimensions is sufficient to take account of the total
effect of correction to scaling terms. We do not have the necessary numerical
precision to estimate $\Delta$ independently for this model. All reported
results for exponents $\phi$ and $\alpha$ are derived from a model of the form
of Eq. (23), although in most cases the effect of adding the correction-to-
scaling term is weak, usually in the third decimal place.

Three estimates of the exponents have been made using correction-to-scaling
models with the following leading order exponents. First, we fit the average
extension at the critical point $h_{n}(f_{\text{c}})$ with leading order
exponent $\phi-1$, as per Eq. (15). Second, we the specific heat at the
critical point $c_{n}(f_{\text{c}})$ with leading order exponent $2\phi-1$, as
per Eq. (16). Third, we fit the critical point values of
$t_{n}(f_{\text{c}})$. By similar scaling arguments to the previous section
the peak value of $t_{n}$ is expected to scale with leading-order exponent
$(\alpha+1)\phi=3\phi-1$. In all cases the value of $\alpha$ is then derived
from the relation in Eq. (14).

The values of $\alpha$ and $\phi$ determined from these fits are reported in
Table 2. These estimates are consistent with $\alpha\approx 2/3$ and
$\phi=3/4$ for the square lattice, confirming that $\phi=\nu_{2}$ in this
case. For the simple cubic lattice the values of $\phi$ are close to the
expected $\phi=\nu_{3}\approx 0.587597(7)$, or equivalently by the
hyperscaling relation Eq. (14) $\alpha=0.29815(5)$. For both two and three
dimensions these exponents confirm that the extended-ballistic transition is
continuous.

Figure 3:  The peak values of $c_{n}$ as an estimate for the location of the
transition $y_{c}^{(n)}$ for (a) the square lattice and (b) the simple cubic
lattice.  Figure 4:  The size exponent $\nu$ from several quantities across
the ballistic transition for (a) 2D and (b) 3D. At the critical point
$y_{\text{c}}=1$ the exponents coincide with value $\nu_{d}$.

The location of the extended-ballistic transition is rigorously known so we
have the luxury of immediately looking at the scaling of thermodynamic
quantities at exactly $y_{\text{c}}=1$ ($f_{\text{c}}=0$). This saves a lot of
additional work to accurately locate the critical point, for which one needs
some easily identifiable signal of the transition. In general for a finite-
size system, the critical point deviates from the long chain limit as

$y_{c}^{(n)}-y_{\text{c}}\sim n^{-\phi},$ (24)

where the crossover exponent $\phi$ appears again. An obvious choice for
$y_{c}^{(n)}$ is the location of the peak value of the specific heat
$c_{n}^{(\text{peak})}$, which occurs near, but slightly above, the transition
point. In the thermodynamic limit the specific heat at the critical point
diverges and so $c_{n}(0)$ should also diverge with $n$ if the crossover
exponent $\phi>1/2$. This is the case for our model and one could even
approximate $c_{n}(0)\approx c_{n}^{(\text{peak})}$. By comparison, locating
the critical point is a significant issue for other systems, e.g. polymer
adsorption where $\phi\leq 1/2$ and so the peaks of the specific heat are not
a good indicator of the transition [31, 32]. Thus, while our earlier estimates
of the exponents are justified, it is nevertheless worth comparing the known
exact value of the transition point to estimates from the simulations using
the peak values of $c_{n}$. This also provides a further estimate of $\phi$.

In Fig. 3 we show a log-log plot of
$y_{c}^{(n)}-y_{\text{c}}=c_{n}^{(\text{peak})}-1$. First, it is clear that in
both cases the long-length limit $y_{\text{c}}=1$ is good. The dashed lines
are least-squares fits to a simple power law (without correction to scaling)
to the larger values of $n$. Particularly for the the square lattice, this is
the case where we see the largest finite-size effect at small $n$. Without the
correction to scaling we find $\phi=0.705\pm 0.008$ for the square lattice and
$0.557\pm 0.001$ for the simple cubic lattice. When a correction-to-scaling
model is used with expected leading order exponent $-\phi$ according to Eq.
(24), we find improved estimates of $\phi=0.76\pm 0.05$ for the square lattice
and $\phi=0.567\pm 0.003$ for the simple cubic lattice. These values closely
match the estimates from the scaling of thermodynamic quantities presented
above and thus are again in good agreement with the relation $\phi=\nu_{d}$.
Our results disagree with $\phi\approx 1/2$ found in [12], which also
estimated $\phi$ from the shift of the critical point at finite lengths.

Lastly, we look at the size scaling of the walks via the quantities in Eq.
(20). Additional simulations with smaller maximum length $n_{\max}=512$ were
run to obtain data for $\langle s\rangle$. Estimates for the scaling exponents
of these quantities were obtained by simple power-law fits (without finite-
size corrections to scaling) for a range of values of $y$ spanning the
extended-ballistic transition. In Fig. 4 we plot the exponents $\nu^{(x)}$,
$\nu^{(x)}$ and $\nu^{(x)}$ (bottom), for the square (top) and simple cubic
(bottom) lattices. Broadly, the values of these exponents match what is
outlined in Table 1. In particular, the values of the different exponents
coincide at the critical point $y_{\text{c}}=1$ where it is expected that all
exponents have value $\nu_{d}$. However, the estimates for the simple-cubic
lattice show a larger spread compared to the square lattice.

##  IV Conclusion

We have investigated the critical behaviour of a pulled SAW near the known
extended-ballistic transition at zero force $f_{\text{c}}=0$
($y_{\text{c}}=1$). With standard scaling analysis we find that the exponent
$\phi$ governing the crossover from the finite size behaviour to the
thermodynamic limit is the same as the size exponent $\nu$. The exponent
$\alpha$ governing the the scaling of the specific heat follows from standard
scaling relations. Although this relation is simple, it has not been
explicitly stated in the literature despite the widespread use of a pulling
force in lattice polymer models. Monte Carlo simulation showed that this
result is verified in the scaling of thermodynamic quantities at the critical
point. It is also verified in the drift of the finite-size critical point
$y_{c}^{(n)}$ from the thermodynamic limit $y_{\text{c}}=1$, as represented by
a typical signature, namely the location of the specific heat peak.
Furthermore, we looked at the size exponent in both the extended and ballistic
phases by way of several quantities that are related to the extension of the
polymer away from the surface. The scaling of these quantities shows
consistent estimations of the size exponent $\nu$.

###### Acknowledgements.

The authors thank Stu Whittington for suggesting this problem to us and for
helpful comments. This research was supported by The University of Melbourne’s
Research Computing Services and the Petascale Campus Initiative.

## References

  * Bemis  _et al._ [1999] J. E. Bemis, B. B. Akhremitchev, and G. C. Walker, Single polymer chain elongation by atomic force microscopy, Langmuir 15, 2799 (1999).
  * Hansma [1996] H. G. Hansma, Atomic force microscopy of biomolecules, Journal of Vacuum Science & Technology B: Microelectronics and Nanometer Structures Processing, Measurement, and Phenomena 14, 1390 (1996).
  * Haupt  _et al._ [1999] B. J. Haupt, J. Ennis, and E. M. Sevick, The detachment of a polymer chain from a weakly adsorbing surface using an AFM tip, Langmuir 15, 3886 (1999).
  * Zhang and Zhang [2003] W. Zhang and X. Zhang, Single molecule mechanochemistry of macromolecules, Prog Polym Sci 28, 1271 (2003).
  * Svoboda and Block [1994] K. Svoboda and S. M. Block, Biological applications of optical forces, Annual Review of Biophysics and Biomolecular Structure 23, 247 (1994), pMID: 7919782.
  * Ioffe and Velenik [2010] D. Ioffe and Y. Velenik, The statistical mechanics of stretched polymers, Braz. J. Probab. Stat. 24, 279 (2010).
  * Janse van Rensburg and Whittington [2013] E. J. Janse van Rensburg and S. G. Whittington, Adsorbed self-avoiding walks subject to a force, J. Phys. A: Math. Theor. 46, 435003 (2013).
  * Janse van Rensburg and Whittington [2016a] E. J. Janse van Rensburg and S. G. Whittington, Self-avoiding walks subject to a force, J. Phys. A: Math. Theor. 49, 11LT01 (2016a).
  * Janse van Rensburg and Whittington [2016b] E. J. Janse van Rensburg and S. G. Whittington, Self-avoiding walks adsorbed at a surface and subject to a force, J. Phys. A: Math. Theor. 49, 244001 (2016b).
  * Janse van Rensburg and Whittington [2017] E. J. Janse van Rensburg and S. G. Whittington, Self-avoiding walks adsorbed at a surface and pulled at their mid-point, J. Phys. A: Math. Theor. 50, 055001 (2017).
  * Bradly  _et al._ [2019a] C. J. Bradly, E. J. Janse van Rensburg, A. L. Owczarek, and S. G. Whittington, Adsorbed self-avoiding walks pulled at an interior vertex, J. Phys. A: Math. Theor. 52, 405001 (2019a).
  * Janse van Rensburg  _et al._ [2009] E. J. Janse van Rensburg, E. Orlandini, M. C. Tesi, and S. G. Whittington, Thermodynamics and entanglements of walks under stress, J. Stat. Mech. Theory Exp. 2009, P07014 (2009).
  * Guttmann  _et al._ [2018] A. J. Guttmann, E. J. Janse van Rensburg, I. Jensen, and S. G. Whittington, Polygons pulled from an adsorbing surface, J. Phys. A: Math. Theor. 51, 074001 (2018).
  * Janse van Rensburg and Whittington [2019] E. J. Janse van Rensburg and S. G. Whittington, Force-induced desorption of uniform branched polymers, J. Phys. A: Math. Theor. 52, 115001 (2019).
  * Bradly and Owczarek [2019] C. J. Bradly and A. L. Owczarek, Monte Carlo simulation of 3-star lattice polymers pulled from an adsorbing surface, J. Phys. A: Math. Theor. 52, 275001 (2019).
  * Bradly  _et al._ [2019b] C. J. Bradly, E. J. Janse van Rensburg, A. L. Owczarek, and S. G. Whittington, Force-induced desorption of 3-star polymers in two dimensions, J. Phys. A: Math. Theor. 52, 315002 (2019b).
  * Janse van Rensburg  _et al._ [2020] E. Janse van Rensburg, C. E. Soteros, and S. G. Whittington, Force-induced desorption of uniform block copolymers, J. Phys. A: Math. Theor. 10.1088/1751-8121/abc5d2 (2020).
  * Beaton [2015] N. R. Beaton, The critical pulling force for self-avoiding walks, J. Phys. A: Math. Theor. 48, 16FT03 (2015).
  * Clisby [2010] N. Clisby, Accurate estimate of the critical exponent $\nu$ for self-avoiding walks via a fast implementation of the pivot algorithm, Phys. Rev. Lett. 104, 055702 (2010).
  * Hammersley  _et al._ [1982] J. M. Hammersley, G. M. Torrie, and S. G. Whittington, Self-avoiding walks interacting with a surface, J. Phys. A: Math. Gen. 15, 539 (1982).
  * Whittington [1975] S. G. Whittington, Self‐avoiding walks terminally attached to an interface, The Journal of Chemical Physics 63, 779 (1975).
  * Ioffe and Velenik [2008] D. Ioffe and Y. Velenik, Ballistic phase of self-interacting random walks, Analysis and stochastics of growth processes and interface models , 55 (2008).
  * Janse van Rensburg [2015] E. J. Janse van Rensburg,  _The Statistical Mechanics of Interacting Walks, Polygons, Animals and Vesicles_ (Oxford University Press, 2015).
  * Prellberg and Krawczyk [2004] T. Prellberg and J. Krawczyk, Flat histogram version of the pruned and enriched Rosenbluth method, Phys. Rev. Lett. 92, 120602 (2004).
  * Grassberger [1997] P. Grassberger, Pruned-enriched rosenbluth method: Simulations of $\theta$ polymers of chain length up to 1 000 000, Phys. Rev. E 56, 3682 (1997).
  * Rosenbluth and Rosenbluth [1955] M. N. Rosenbluth and A. W. Rosenbluth, Monte Carlo calculation of the average extension of molecular chains, J. Chem. Phys. 23, 356 (1955).
  * Caracciolo  _et al._ [2005] S. Caracciolo, A. J. Guttmann, I. Jensen, A. Pelissetto, A. N. Rogers, and A. D. Sokal, Correction-to-scaling exponents for two-dimensional self-avoiding walks, J. Stat. Phys. 120, 1037 (2005).
  * Guttmann and Conway [2001] A. Guttmann and A. Conway, Square lattice self-avoiding walks and polygons, Ann. Comb. 5, 319 (2001).
  * Nienhuis [1982] B. Nienhuis, Exact critical point and critical exponents of $\mathrm{O}(n)$ models in two dimensions, Phys. Rev. Lett. 49, 1062 (1982).
  * Shannon  _et al._ [1996] S. R. Shannon, T. C. Choy, and R. J. Fleming, Corrections to scaling in two-dimensional polymer statistics, Phys. Rev. B 53, 2175 (1996).
  * Janse van Rensburg and Rechnitzer [2004] E. J. Janse van Rensburg and A. R. Rechnitzer, Multiple Markov chain Monte Carlo study of adsorbing self-avoiding walks in two and in three dimensions, J. Phys. A: Math. Gen. 37, 6875 (2004).
  * Bradly  _et al._ [2018] C. J. Bradly, A. L. Owczarek, and T. Prellberg, Universality of crossover scaling for the adsorption transition of lattice polymers, Phys. Rev. E 97, 022503 (2018).


# Simple yet Effective Code-Switching Language Identification
with Multitask Pre-Training and Transfer Learning

Shuyue Stella Li, Cihan Xiao, Tianjian Li, Bismarck Odoom
Center for Language and Speech Processing, Johns Hopkins University
sli136, cxiao7, tli104<EMAIL_ADDRESS>
###### Abstract

Code-switching, also called code-mixing, is the linguistics phenomenon where
in casual settings, multilingual speakers mix words from different languages
in one utterance. Due to its spontaneous nature, code-switching is extremely
low-resource, which makes it a challenging problem for language and speech
processing tasks. In such contexts, Code-Switching Language Identification
(CSLID) becomes a difficult but necessary task if we want to maximally
leverage existing monolingual tools for other tasks. In this work, we propose
two novel approaches toward improving language identification accuracy on an
English-Mandarin child-directed speech dataset. Our methods include a stacked
Residual CNN+GRU model and a multitask pre-training approach to use Automatic
Speech Recognition (ASR) as an auxiliary task for CSLID. Due to the low-
resource nature of code-switching, we also employ careful silver data creation
using monolingual corpora in both languages and up-sampling as data
augmentation. We focus on English-Mandarin code-switched data, but our method
works on any language pair. Our best model achieves a balanced accuracy of
0.781 on a real English-Mandarin code-switching child-directed speech corpus
and outperforms the previous baseline by 55.3%.

Index Terms: multilingual, code-switching, low-resource, language
identification

##  1 Introduction

With more than 6000 languages still alive today, there are more people
speaking more than one language (whether from birth or through late
acquisition) than monolingual speakers Marian and Shook (2012). When
multilingual speakers who share two or more of the same languages engage in a
conversation, they naturally tend to switch languages spontaneously. Code-
switching allows bilingual speakers to express their intentions more freely
and to be better understood Heredia and Altarriba (2001). With the development
of machine learning and neural networks, language and speech processing with
most high-resource monolingual languages are highly effective. However, it is
non-trivial to adapt the monolingual tools to multilingual and code-switching
tasks. Additionally, as demonstrated in our later experiments, even large
models trained on multilingual data such as Whisper Radford et al. (2022) and
XLSR Babu et al. (2021); Conneau et al. (2020) are limited in processing code-
switched data between two high-resource languages. Therefore, an effective
approach to leverage existing monolingual or multilingual pre-trained speech
and language models and other NLP tools is to identify the language in each
segment of a code-switched speech or text.

Figure 1: “Split-then-Process” Pipeline. The red dotted box is the main focus
of our study. Given a segmented utterance from a child-directed domain, the
language of each segment is identified by our system. This can potentially be
useful for a range of downstream processing tasks that leverages existing
monolingual or multilingual tools.

In this work, we focus on the language identification task of code-switched
English and Mandarin speech data collected from Singapore on a child-directed
activity. Singapore is such a language-dense region where four primary
languages are spoken by the people - English, Malay, Mandarin, and Tamil, and
almost all Singaporeans are bilingual or multilingual. The language diversity
in the region contributes to the wide dialectal variations of the code-
switched data and increases the difficulty of speech-processing tasks. The
child-directed characteristic of the data makes the problem unique in that
both the content domain and the speech style deviate from standard datasets
and models. Domain mismatch problems have been addressed by data augmentation
Sun et al. (2021) or unsupervised adversarial training Wang et al. (2018)
approaches in Automatic Speech Recognition (ASR) and gradual fine-tuning (GFT)
in text-based settings Xu et al. (2021). We adopt both data augmentation and
GFT to the speech CSLID task in this work to improve the robustness of our
system.

As illustrated in Figure 1, the main objective of our model is to identify the
language of a segment of speech, so that monolingual or multilingual models
can be more effectively used for downstream tasks. With English and Mandarin
being the languages with the largest number of speakers in Singapore and high-
resource languages in the world, Code-Switching Language Identification is a
crucial step in the “Split-then-Process” pipeline. Possible downstream tasks
that could benefit from a robust language identification system include speech
recognition, speech synthesis, or speech translation. We leverage monolingual
data such as AISHELL (Mandarin) Bu et al. (2017) and LibriSpeech (English)
Panayotov et al. (2015) to build a CSLID model that is robust to both domain
and dialectal variations. The main contributions of our work are summarized as
follows:

  * •

We propose two systems for code-switching language identification, a Residual
CNN with BiRNN network (CRNN) and an Attention-based Multitask Training Model
with combined ASR and CSLID loss. The systems can be easily extended to any
language pair.

  * •

We investigate the effect of pre-training with data augmentation from
monolingual sources and the effect of fine-tuning with out-of-domain code-
switched data, concluding that data balance is more crucial than domain
similarity.

  * •

We demonstrate that small and efficient architectures with effective data
augmentation can be extremely successful in the CSLID task, outperforming
massive multilingual pre-trained language models (PLM). Our system placed 2nd
in a challenge featuring an English-Mandarin code-switching child-directed
speech corpus [reference redacted for review], and we make our code publicly
available111We make the project open source at [link hidden for review]. for
further explorations in the field of code-switching speech processing.

##  2 Related work

Due to the increase of globalization and the growing population of bilingual
and multilingual speakers, there is an emerging need for better language
technologies for code-switching languages. Due to its spontaneous nature,
code-switching happens more in colloquial settings, making it difficult for
data collection. Code-switching is also a complex sociocultural linguistic
phenomenon that depends on a combination of factors including topic,
formality, and speaker intent Mabule (2015); Nilep (2006). Code-switching can
happen at different levels of the utterance (intersentential, intrasentential,
intra-word) Myers-Scotton (1989). All the above characteristics make code-
switching a fascinatingly diverse and challenging topic of study. In both text
and speech processing, CSLID is a crucial step for downstream tasks such as
text normalization for text-to-speech synthesis Manghat et al. (2022), part-
of-speech tagging Solorio and Liu (2008), speech translation Weller et al.
(2022), and speech recognition Zhang et al. (2021, 2022); Zhou et al. (2022);
Sreeram and Sinha (2020).

###  2.1 Multidialectal Code-Switching

Code-switching speech processing faces the issue of dialectal variations.In
Singapore, Mandarin, Hokkien, and Cantonese are the major Chinese dialects
with most speakers, along with Teochew, Hakka, and Hainanese Gupta and Yeok
(1995). Chowdhury et al. (2021) proposed an end-to-end attention-based
conformer architecture for multi-dialectal Arabic ASR. Rivera (2019) built an
acoustic model for code-switching detection among Arabic dialects. However,
there is a lack of sufficient research on code-mixing between non-standard
Mandarin and non-standard English, which is the focus of our study.

###  2.2 Code-Switching Language Identification

The use of Convolutional Neural Networks (CNN) in speech processing is widely
adopted due to the use of spectrogram or filter bank as the first feature
extraction step of speech signal processing in monolingual tasks Ganapathy et
al. (2014). Deep Neural Networks (DNN) Yılmaz et al. (2016) and phoneme units-
based Hidden Markov Model (HMM) with Support Vector Machine (SVM) classifier
Mabokela et al. (2014); Mabokela and Manamela (2013) have also been used for
CSLID. Additionally, CSLID is often integrated into ASR systems as an
auxiliary task to improve the ASR performance Lounnas et al. (2020); Shan et
al. (2019). However, these approaches have a different focus from our current
study, which aims to improve the CSLID performance for a range of speech-
processing tasks.

###  2.3 Data Augmentation & Multilingual PLMs

Various data augmentation techniques have been used for code-switching, but
mostly focused on text processing tasks Xu and Yvon (2021); Li and Murray
(2022). Some work uses text-based data augmentation for speech tasks Hussein
et al. (2023); Nakayama et al. (2019). Ali et al. (2021) uses monolingual
English and Arabic speech data for the code-switched ASR task. However, there
is little prior work to synthetically generate code-switched speech data from
monolingual sources. In our work, we segment monolingual speech data in the
sub-utterance level to simulate code-switched speech data augmentation.

Additionally, with the recent development of massively multilingual pre-
trained speech and language models such as mSLAM Bapna et al. (2022), Whisper
Radford et al. (2022) and XLS-R Babu et al. (2021); Conneau et al. (2020), it
is easier to leverage monolingual data for multilingual tasks. The use of
multilingual PLMs for code-switching tasks in the text domain has proven to be
successful Rathnayake et al. (2022), but it has not been widely used in the
speech setting due to limited data and costly training. In our work, we use
the multilingual PLMs as a zero-shot baseline with which we compare our
parameter-efficient models.

##  3 Methodology

In this section, we first propose three systems for the CSLID task and
describe the architectural design tailored to the different characteristics of
the data and model. Then, we introduce our data augmentation method leveraging
out-of-domain code-switching data with a GFT schedule to improve upon the pre-
train-fine-tune paradigm.

###  3.1 CRNN

Figure 2: CRNN Model

The CRNN model, inspired by Bartz et al. (2017), is a stack of Residual CNNs
and RNNs. We utilize the power of CNN to extract features directly from the
spectral domain and use an RNN to extract temporal dependencies. We use bi-GRU
Cho et al. (2014) layers for the RNN component of the model because it has
fewer parameters, making it faster to train and less prone to over-fitting. As
illustrated in Figure 2(a), our CRNN model is a simplified version of Bartz et
al. (2017) with 3 CNN layers to extract acoustic features and 5 GRU layers
with hidden dimension 512 to learn features for language identification. We
apply a linear classifier to the last hidden state of the RNN.

Stage | MERLion (M) | SEAME (S) | Total | Ratios
---|---|---|---|---
zh | en | total | zh/en | zh | en | total | zh/en | zh | en | total | S/M | zh/en
1 |  5.4 (1) | 21.6 | 27.0 | 0.2 | 17.9 | 8.9 | 26.8 | 2.0 | 23.2 | 30.6 | 53.8 | 1.0 | 0.76
2 | 10.7 (2) | 21.6 | 32.4 | 0.5 | 10.7 | 5.4 | 16.1 | 2.0 | 21.4 | 27.0 | 48.4 | 0.5 | 0.79
3 | 10.7 (2) | 21.6 | 32.4 | 0.5 | 4.5 | 2.2 | 6.7 | 2.0 | 15.2 | 23.9 | 39.1 | 0.2 | 0.64
4 | 16.1 (3) | 21.6 | 37.7 | 0.7 | 0.0 | 0.0 | 0.0 | - | 16.1 | 21.6 | 37.7 | 0.0 | 0.74

Table 1:  Grdual FT Schedule. Values inside parentheses are up-sampling ratios
for the MERLion zh utterances.

###  3.2 Multi-Task Learning (MTL)

Figure 3: Multi-task Learning Model

To enhance the model’s ability to extract acoustic features, we train a model
via multitask learning with a joint Connectionist Temporal Classification
(CTC) and LID loss, as illustrated in Figure 3(b). The architecture of the
model is based on a Conformer encoder, along with a linear layer for CTC
decoding and an LSTM + linear layer for LID decoding. Similar to the CRNN
model, we conduct phased training to first pre-train the Conformer model with
the joint loss on the monolingual corpora and fine-tune the model on the
MERLIon and SEAME datasets with only the LID loss. This approach aims to
better adapt the model to the target classification task.

###  3.3 Multilingual PLMs

Being pre-trained on multiple languages, massively multilingual PLMs are a
powerful tool for cross-lingual tasks. We want to understand the out-of-the-
box ability of PLMs to process code-switching sentences by comparing the zero-
shot CSLID performance of Whisper Radford et al. (2022) and XLSR Babu et al.
(2021); Conneau et al. (2020) against the more parameter-efficient models we
introduce in this work. For Whisper, we use the `detect_language()` method
from the model class, passing in CutSets with a max duration of 50. For XLSR,
we perform two-way zero-shot classification using wav2vec2-xls-r-300m with a
LID head. The LID head is a 2-layer Bidirectional GRU with a linear layer.

###  3.4 Data Augmentation

Child-directed English-Mandarin code-switching is an extremely low resource
problem. As such, we propose a data augmentation method that takes advantage
of any additional data in a similar distribution to improve the performance of
the model. The target in domain data - MERLion - is unbalanced such that the
ratio of English to Mandarin labels is about 4:1. In addition to up-sampling
the Mandarin utterances during training, our proposed data augmentation
approach combines the SEAME code-switching dataset (as described in detail in
Section 4.1) that has more Mandarin utterances than English ones. Lastly, we
propose a gradual fine-tuning schedule for smooth domain adaptation as
described in Table 1 below Xu et al. (2021). As we up-sample the Mandarin
utterances in the MERLion dataset and vary the ratio of Mandarin to English in
the sampled SEAME dataset to control for a smooth transition to the real
Mandarin-English ratios in the development set. The gradual FT terminates with
a stage of using only the MERLion dataset (with Mnadarin up-sampled) without
the out-of-domain SEAME data. All the experiments described in Table 1 are
fine-tuning the model checkpoint pre-trained on monolingual Mandarin and
English speech.

##  4 Experiments

###  4.1 Dataset & Metric

We use multiple monolingual English and Mandarin and code-switched English-
Mandarin datasets in our experiments, including LibriSpeech Panayotov et al.
(2015), National Speech Corpus of Singapore (NSC) Koh et al. (2019), AISHELL
Bu et al. (2017), SEAME Lyu et al. (2010), and MERLIon Chua et al. (2023).
Table 2 reports the language and size of each dataset. Note that not all
datasets are used for each experiment. The MERLIon dataset is split into
training and development sets, and we refer to the train split of the MERLIon
dataset as “MERLIon” in our system descriptions.

Dataset | Language | Length (hr)
---|---|---
LibriSpeech-clean | en (US) | 100
NSC | en (SG) | 100
AISHELL | zh | 200
SEAME | en-zh | 100
MERLIon | en-zh | 30
Table 2: Datasets used in our experiments.

##### Metric

The MERLIon dataset roughly contains 25 hours of English speech and 5 hours of
Mandarin speech. Due to this severe data imbalance issue, we use the Balance
Accuracy (BAC), which is the average of recall obtained for each label class,
rather than the Absolute Accuracy as the metric to evaluate our systems. In
the submission of the English-Mandarin code-switching task, the evaluation
also reports the Equal Error Rate (EER), which is defined to be the threshold
for an equal false acceptance rate and false rejection rate.

##### Baseline

The baseline over which we attempt to improve is the system developed by the
task organizers, which consists of an end-to-end conformer model trained on
the same available data Chua et al. (2023). This system has a BAC of 50.32%
and an EER of 22.13%.

###  4.2 Preprocessing

We preprocess the data using lhotse222https://github.com/lhotse-speech/lhotse,
a Python toolkit designed for speech and audio data preparation. We
standardize the sample rate of all audio recordings to 16kHz by downsampling
utterances in the development and test dataset with sample rates $>$ 16kHz.
Prior to training, we extract 80-dimensional filterbank (fbank) features from
the speech recordings and apply speed perturbation with factors of 0.9 and
1.1. During training, we use on-the-fly SpecAug Park et al. (2019)
augmentation on the extracted filter bank features with a time-warping factor
of 80.

To train the model jointly with an ASR CTC loss, we first tokenize and
romanize the bilingual transcripts with space-delimited word-level
tokenization for monolingual English transcripts (LibriSpeech and NSC) and
monolingual Mandarin transcripts in AISHELL, as these transcripts were pre-
tokenized and separated by spaces. For the occasionally code-switched Mandarin
words in NSC, we remove the special tags and kept only the content of the
Mandarin words. The SEAME dataset contains a portion of untokenized Mandarin
transcripts. Hence, we tokenize all Mandarin text sequences with length $>4$
using a Mandarin word segmentation tool
jieba333https://github.com/fxsjy/jieba. Additionally, to reduce the size of
the model, we adopt a pronunciation lexicon, utilizing the CMU dictionary for
English word-to-phoneme conversion and the python-pinyin-jyutping-sentence
tool for generating the pinyin for Mandarin
words444https://github.com/Language-Tools/python-pinyin-jyutping-sentence. To
enhance the model’s ability to capture the lexical information in the training
data, we add a suffix "_cn" for Mandarin phonemes.

###  4.3 Experimental Setup

We follow the pre-train-fine-tune paradigm for most experiments except for the
zero-shot PLM baseline and ablation experiments to investigate the effect of
pre-training. In the pre-training stage, we use the monolingual datasets
(LibriSpeech, AISHELL, and NSC), and in the fine-tuning state, we use the
code-switched datasets (SEAME and MERLIon).

# | System | PT Data | FT Data | FT Method
---|---|---|---|---
1 | CRNN |  | - | -
2 | LibriSpeech | MERLIon | 1-stage
3 | \+ AISHELL | MERLIon | combined
4 |  | \+ SEAME | gradual
5 | - | MERLIon | 1-stage
6 | MERLIon | combined
\+ SEAME
7 | MTL | LibriSpeech | MERLIon | combined
8 | \+ AISHELL + NSC | \+ SEAME | 2-stage
9 | Whisper | - | - | -
10 | XLSR | - | - | -

Table 3: Experimental Setup for Basic Experiments

Table 3 shows the experiments conducted in our study along with the pre-
training and fine-tuning datasets and fine-tuning method. For this set of
experiments, FT Methods: 1-stage FT means fine-tuning the model on the MERLIon
dataset only; combined FT is fine-tuning the model on a 1-1 proportion of
SEAME and MERLIon data; and gradual FT is fine-tuning the model on more SEAME
(out-of-domain) data than MERLIon (in-domain) data, then increasing the ratio
of MERLIon data gradually until the fine-tuning set contains only MERLIon
data.

Table 4 summarizes the second set of experiments involving the up-sampling
with schedule described in Section 3.4. Note that in Experiment #15, only the
1:1 mix of MERLion:SEAME is used as a control for the setting for the MTL
system.

# | System | epoch/stage | total epochs | LR
---|---|---|---|---
11 | CRNN | 3 | 12 | 0.001
12 | 0.00001
13 | 5 | 20 | 0.001
14 | 0.00001
# | System | epoch range | total epochs | LR
15 | MTL | 1-20 | 20 | 0.00001
16 | 1-5 | 5
17 | 5-10 | 10
18 | 10-15 | 15
19 | 15-20 | 20

Table 4: Up-Sampling Experiments.

###  4.4 Training

####  4.4.1 CRNN Training

We pre-train our CRNN model for 5 epochs on 100 hours of clean speech from
LibriSpeechPanayotov et al. (2015) and 200 hours of preselected partition from
AISHELLBu et al. (2017). Each batch contains a balanced amount of English and
Mandarin sub-utterance level speech utterances to simulate an artificial
speech code-switching dataset. We select the pre-trained model checkpoint with
the best performance on the entire MERLIon dataset. Then, the model is fine-
tuned on the MERLIon dataset (exp #2) or the MERLIon+SEAME dataset (exp #3&4)
for 10 epochs, leaving out 1 hour of MERLIon data (1749 English utterances and
100 Mandarin utterances) for evaluation. During training, we set the max
duration of each cut to 120ms; we use the Adam optimizer with a pre-training
learning rate of 1e-4 and a fine-tuning learning rate of 1e-5, with a dropout
of 0.1. In the experiment, ratios between the out-of-domain and in-domain data
are $[3,2,1,0.5,0]$ over 5 epochs.

####  4.4.2 Multitask Pre-Training

The conformer model is pre-trained with the joint CTC/LID loss for 5 epochs as
well on the monolingual data, including LibriSpeech, AISHELL, and NSC. To
balance the loss for each task, we interpolate the losses with a
hyperparameter $\lambda$. Formally, the overall loss $L$ is computed as below:

$L=(1-\lambda)L_{\text{CTC}}+\lambda L_{\text{LID}}\cdot{\alpha}$ (1)

where $L_{\text{CTC}}$ denotes the CTC loss, $L_{\text{LID}}$ denotes the LID
loss and $\alpha$ is the scaler for the LID loss. We set $\lambda=0.2$ and
$\alpha=100$. The model is then fine-tuned for 15 epochs on the mixed MERLIon
and SEAME datasets. We intentionally balance the total duration of samples
drawn from each dataset, which implicitly biases toward the development set as
it contains fewer utterances, and our sampler terminates when it finishes an
epoch on the smaller corpus.

##  5 Results and Analysis

# | experiment | English | Mandarin | Balanced
---|---|---|---|---
1 | PT | 0.649 | 0.650 | 0.650
2 | PT + FT (M) | 0.927 | 0.630 | 0.779
3 | PT + FT (M+S) | 0.965 | 0.370 | 0.667
4 | PT + FT (gradual) | 0.851 | 0.720 | 0.785
5 | FT (M) | 1.0 | 0.0 | 0.5
6 | FT (M+S) | 0.988 | 0.09 | 0.539
7 | MTL + combined FT | 0.960 | 0.610 | 0.785
8 | MTL + 2-stage FT | 0.957 | 0.46 | 0.708
9 | Whisper Zero-Shot | 0.821 | 0.502 | 0.662
10 | XLSR Zero-Shot | 0.198 | 0.0 | 0.099

Table 5: English, Mandarin and Balanced Accuracy of our CRNN model on the
held-out development set of MERLIon. Table keys: PT = only pre-training, FT =
fine-tuned on the MERLIon train split, w/ SEAME = fine-tuned with mixed
MERLIon train split and SEAME dataset, MTL = multitask learning model with
pre-training and fine-tuning. (All rows without PT indicate that the model
parameters are randomly initialized.)

###  5.1 CRNN Results

Table 5 shows the English, Mandarin, and BAC of our CRNN model on the held-out
part of the MERLIon development set. The best-performing model is the model
initialized from the best pre-train checkpoint and gradually fine-tuned on the
MERLIon and SEAME dataset (PT+FT). Without , it is more effective to only
fine-tune on the MERLIon in-domain dataset, implying that directly combining
out-of-domain sources (SEAME) causes additional complexity for the model. The
Mandarin accuracies for training on the MERLIon dataset from scratch with
(exp#6) or without (exp#5) the SEAME dataset are both poor - 0.0 for the model
fine-tuning only on MERLIon and 0.09 for the model fine-tuning on the MERLIon
and SEAME datasets.

###  5.2 Multitask Pre-Training

Two fine-tuning approaches were used for the MTL model. We find that after
fine-tuning on the combined MERLIon + SEAME dataset, a second stage fine-
tuning on only the MERLIon dataset in fact hurts the performance. This might
result from the imbalanced labeling effect, biasing the model toward the
English predictions. Therefore, introducing more Mandarin samples from the
SEAME corpus balances the labeling and yields better performance on the held-
out set.

###  5.3 Multilingual PLMs

As shown in Table 5, the zero-shot performance of Whisper is not great but
reasonable given the massive amount of data it was pre-trained on. However,
zero-shot XLSR is extremely ineffective in doing CSLID. These results suggest
that multilingual PLMs do not have the out-of-the-box capability to understand
the complex phenomenon of code-switching and thus require careful fine-tuning.
We report the performance of our CRNN model at task submission time on the
MERLIon test set (labels unavailable to participants) in Table 6.

System | Dev | Test
---|---|---
EER | BAC | EER | BAC
Baseline Chua et al. (2023) | - | - | 0.221 | 0.503
Whisper Zero-Shot | 0.228 | 0.662 | 0.230 | 0.649
CRNN PT+FT | 0.146 | 0.663 | 0.155 | 0.701

Table 6: Equal Error Rate (EER) and Balanced Accuracy (BAC) on the MERLIon
development and test sets for our submitted system and the previous baseline.

###  5.4 Ablation Studies

####  5.4.1 Effect of Pre-Training

As shown in exp #5 and #6, removing the pre-training stage results in
significant performance drops. The model trained with only the MERLIon dataset
classifies all utterances as English because the MERLIon dataset is heavily
unbalanced, which contains 40287 English utterances and only 9903 mandarin
utterances. This implies that the pre-training on monolingual data with
balanced labels makes the model robust under heavily unbalanced classes,
allowing the model to extract meaningful features for both languages even if
data for one language is scarce.

####  5.4.2 Effect of Code-Switched Fine-Tuning

Directly using the pre-trained model (exp #1) suffers from domain mismatch,
suggesting that fine-tuning on gold data is necessary. First, pre-training
data are originally monolingual, so dataset features such as recording quality
and volume can be learned instead of linguistic features. Second, the pre-
training datasets are from general domains, while the MERLIon dataset contains
children-directed speech, which might have a different set of vocabulary.
Nevertheless, with the class imbalance issue, fine-tuning results on MERLIon
(exp #2) improves the BAC but lowers the Mandarin accuracy from 0.650 to
0.630.

####  5.4.3 Effect of Gradual Fine-Tuning

Comparing Experiment #4 with Experiment #3, The model’s classification
accuracy on Mandarin labels improves significantly with a GFT on combined
MERLion and SEAME data. Despite the class imbalance issue, the approach (exp
#4) is shown to be successful, allowing the model to effectively extract
enough linguistic information from the higher resource but out-of-domain
dataset (SEAME) to avoid the short-cut learning from imbalanced in-domain
dataset.

Given the effectiveness of GFT, we further explore experimental designs with
the GFT setup combined with data up-sampling to solve the label imbalance
issue in the target MERLion dataset. We report the model performance of these
additional GFT experiments in Table 7. First, for the CRNN model, which has a
fairly simple residual convolutional neural network architecture, GFT proves
to be extremely helpful when fine-tuning on a model pre-trained only on
monolingual Mandarin and English data. With a well-design gradual fine-tuning
schedule, the classification accuracy on Mandarin improves steadily while the
accuracy on English labels is maintained at a reasonable level as shown in
Experiment #14, making this model achieve the best overall results out of all
CRNN model variations.

# | CRNN Exp. Desc | English | Mandarin | Balanced
---|---|---|---|---
11 | 3ep-GFT lr=1e-3 | 0.938 | 0.270 | 0.604
12 | 3ep-GFT lr=1e-5 | 0.823 | 0.410 | 0.616
13 | 5ep-GFT lr=1e-3 | 0.798 | 0.610 | 0.704
14 | 5ep-GFT lr=1e-5 | 0.932 | 0.680 | 0.806
# | MTL Exp. Desc | Balanced Accuracy
15 | non-GFT 20ep |  | 0.835 |
16 | GFT ep1-5 |  | 0.800 |
17 | GFT ep5-10 |  | 0.806 |
18 | GFT ep10-15 |  | 0.817 |
19 | GFT ep15-20 |  | 0.805 |

Table 7: Performance of the two systems when fine-tuned with up-sampling and
gradual fine-tuning.

On the other hand, GFT does not seem to be the contributing factor to the
success of the MTL system in predicting the LID of the code-switched
utterances. While keeping the MERLion:SEAME data ratio constant, Experiment
#15 achieves the best performance across all systems and designs. This could
be explained by the ASR portion of the loss function in the MTL framework,
which forces the model to extract higher-level linguistic representations.
This increases the robustness of the model against out-of-domain data (SEAME
in this case) and therefore the smooth domain adaptation provided by the
gradual fine-tuning schedule does not contribute as much in this system
design. Up-sampling proves to be extremely helpful in this situation, as
Experiment #15 outperforms Experiment #7 by 6.4%. The up-sampling provides the
model with more opportunities to learn from and accurately classify instances
of the underrepresented class, which leads to a high BAC.

##  6 Conclusion

In this work, we propose two simple and efficient systems for the spoken
English-Mandarin child-directed code-switching LID task. The CRNN approach
uses a simple stack of CNNs and RNNs to capture information from both the
spectral and temporal axes. The multitask learning approach utilizes ASR CTC
loss as an auxiliary task to learn higher-level linguistic features for CSLID.
Our models significantly outperform previous baselines as well as multilingual
PLMs, and we conduct extensive ablation studies to investigate factors that
might influence CSLID performance. Future works include upsampling the
minority label class and fine-tuning PLMs for larger-scale transfer learning
to benefit code-switching speech processing.

## Limitations

Some of the limitations of our work include the fact that we are not able to
use a large batch size when training the model due to compute limits, which
might contribute to slower convergence and noisy model performance.
Furthermore, we do not leverage cross-lingual transfer from other languages
outside of the two languages that are included in the code-switched data.
Incorporating code-switched data in other language pairs or monolingual data
in related languages might result in additional positive cross-lingual
interference.

## References

  * Ali et al. (2021) Ahmed Ali, Shammur Chowdhury, Amir Hussein, and Yasser Hifny. 2021.  Arabic code-switching speech recognition using monolingual data.  _arXiv preprint arXiv:2107.01573_.
  * Babu et al. (2021) Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, et al. 2021\.  Xls-r: Self-supervised cross-lingual speech representation learning at scale.  _arXiv preprint arXiv:2111.09296_.
  * Bapna et al. (2022) Ankur Bapna, Colin Cherry, Yu Zhang, Ye Jia, Melvin Johnson, Yong Cheng, Simran Khanuja, Jason Riesa, and Alexis Conneau. 2022.  mslam: Massively multilingual joint pre-training for speech and text.  _arXiv preprint arXiv:2202.01374_.
  * Bartz et al. (2017) Christian Bartz, Tom Herold, Haojin Yang, and Christoph Meinel. 2017.  Language identification using deep convolutional recurrent neural networks.  In _International Conference on Neural Information Processing_ , pages 880–889. Springer.
  * Bu et al. (2017) Hui Bu, Jiayu Du, Xingyu Na, Bengu Wu, and Hao Zheng. 2017.  Aishell-1: An open-source mandarin speech corpus and a speech recognition baseline.  In _2017 20th COCOSDA_ , pages 1–5. IEEE.
  * Cho et al. (2014) Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014.  Learning phrase representations using RNN encoder–decoder for statistical machine translation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1724–1734, Doha, Qatar. Association for Computational Linguistics.
  * Chowdhury et al. (2021) Shammur Absar Chowdhury, Amir Hussein, Ahmed Abdelali, and Ahmed Ali. 2021.  Towards one model to rule all: Multilingual strategy for dialectal code-switching arabic asr.  _arXiv preprint arXiv:2105.14779_.
  * Chua et al. (2023) Y. H. Victoria Chua, Leibny Paola Garcia Perera, Sanjeev Khudanpur, Andy W. H. Khong, Justin Dauwels, Fei Ting Woon, and Suzy J Styles. 2023.  Development and evaluation data for multilingual everyday recordings - language identification on code-switched child-directed speech (merlion ccs) challenge.  In _DR-NTU (Data)_.
  * Conneau et al. (2020) Alexis Conneau, Alexei Baevski, Ronan Collobert, Abdelrahman Mohamed, and Michael Auli. 2020.  Unsupervised cross-lingual representation learning for speech recognition.  _arXiv preprint arXiv:2006.13979_.
  * Ganapathy et al. (2014) Sriram Ganapathy, Kyu Han, Samuel Thomas, Mohamed Omar, Maarten Van Segbroeck, and Shrikanth S Narayanan. 2014.  Robust language identification using convolutional neural network features.  In _15th ISCA_.
  * Gupta and Yeok (1995) Anthea Fraser Gupta and Siew Pui Yeok. 1995.  Language shift in a singapore family.  _Journal of Multilingual & Multicultural Development_, 16(4):301–314.
  * Heredia and Altarriba (2001) Roberto R Heredia and Jeanette Altarriba. 2001.  Bilingual language mixing: Why do bilinguals code-switch?  _Current Directions in Psychological Science_ , 10(5):164–168.
  * Hussein et al. (2023) Amir Hussein, Shammur Absar Chowdhury, Ahmed Abdelali, Najim Dehak, Ahmed Ali, and Sanjeev Khudanpur. 2023.  Textual data augmentation for arabic-english code-switching speech recognition.  In _2022 IEEE Spoken Language Technology Workshop (SLT)_ , pages 777–784. IEEE.
  * Koh et al. (2019) Jia Xin Koh, Aqilah Mislan, Kevin Khoo, Brian Ang, Wilson Ang, Charmaine Ng, and YY Tan. 2019.  Building the singapore english national speech corpus.  _Malay_ , 20(25.0):19–3.
  * Li and Murray (2022) Shuyue Stella Li and Kenton Murray. 2022.  Language agnostic code-mixing data augmentation by predicting linguistic patterns.  _arXiv preprint arXiv:2211.07628_.
  * Lounnas et al. (2020) Khaled Lounnas, Hassan Satori, Mohamed Hamidi, Hocine Teffahi, Mourad Abbas, and Mohamed Lichouri. 2020.  Cliasr: a combined automatic speech recognition and language identification system.  In _2020 1st iIRASET_ , pages 1–5. IEEE.
  * Lyu et al. (2010) Dau-Cheng Lyu, Tien Ping Tan, Chng Eng Siong, and Haizhou Li. 2010.  Seame: a mandarin-english code-switching speech corpus in south-east asia.  In _Interspeech_.
  * Mabokela and Manamela (2013) Koena Ronny Mabokela and Madimetja Jonas Manamela. 2013.  An integrated language identification for code-switched speech using decoded-phonemes and support vector machine.  In _2013 7th Conference on Speech Technology and Human-Computer Dialogue (SpeD)_ , pages 1–6. IEEE.
  * Mabokela et al. (2014) Koena Ronny Mabokela, Madimetja Jonas Manamela, and Mabu Manaileng. 2014.  Modeling code-switching speech on under-resourced languages for language identification.  In _Spoken Language Technologies for Under-Resourced Languages_.
  * Mabule (2015) Dorah R Mabule. 2015.  What is this? is it code switching, code mixing or language alternating?  _Journal of Educational and Social Research_ , 5(1):339.
  * Manghat et al. (2022) Sreeram Manghat, Sreeja Manghat, and Tanja Schultz. 2022.  Normalization of code-switched text for speech synthesis.  _Proc. Interspeech 2022_ , pages 4297–4301.
  * Marian and Shook (2012) Viorica Marian and Anthony Shook. 2012.  The cognitive benefits of being bilingual.  In _Cerebrum: the Dana forum on brain science_ , volume 2012. Dana Foundation.
  * Myers-Scotton (1989) Carol Myers-Scotton. 1989.  Codeswitching with english: types of switching, types of communities.  _World Englishes_ , 8(3):333–346.
  * Nakayama et al. (2019) Sahoko Nakayama, Takatomo Kano, Andros Tjandra, Sakriani Sakti, and Satoshi Nakamura. 2019.  Recognition and translation of code-switching speech utterances.  In _2019 22nd O-COCOSDA_ , pages 1–6. IEEE.
  * Nilep (2006) Chad Nilep. 2006.  “code switching” in sociocultural linguistics.  _Colorado research in linguistics_.
  * Panayotov et al. (2015) Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015.  Librispeech: An asr corpus based on public domain audio books.  In _2015 IEEE ICASSP_ , pages 5206–5210.
  * Park et al. (2019) Daniel S. Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D. Cubuk, and Quoc V. Le. 2019.  SpecAugment: A simple data augmentation method for automatic speech recognition.  In _Interspeech 2019_. ISCA.
  * Radford et al. (2022) Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022.  Robust speech recognition via large-scale weak supervision.
  * Rathnayake et al. (2022) Himashi Rathnayake, Janani Sumanapala, Raveesha Rukshani, and Surangika Ranathunga. 2022.  Adapter-based fine-tuning of pre-trained multilingual language models for code-mixed and code-switched text classification.  _Knowledge and Information Systems_ , 64(7):1937–1966.
  * Rivera (2019) Gabrielle Cristina Rivera. 2019.  _Automatic detection of code-switching in Arabic dialects_.  Ph.D. thesis, Massachusetts Institute of Technology.
  * Shan et al. (2019) Changhao Shan, Chao Weng, Guangsen Wang, Dan Su, Min Luo, Dong Yu, and Lei Xie. 2019\.  Investigating end-to-end speech recognition for mandarin-english code-switching.  In _2019 IEEE ICASSP_ , pages 6056–6060. IEEE.
  * Solorio and Liu (2008) Thamar Solorio and Yang Liu. 2008.  Part-of-speech tagging for english-spanish code-switched text.  In _Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing_ , pages 1051–1060.
  * Sreeram and Sinha (2020) Ganji Sreeram and Rohit Sinha. 2020.  Exploration of end-to-end framework for code-switching speech recognition task: Challenges and enhancements.  _IEEE Access_ , 8:68146–68157.
  * Sun et al. (2021) Xiusong Sun, Qun Yang, and Shaohan Liu. 2021.  A domain-mismatch speech recognition system in radio communication based on improved spectrum augmentation.  In _2021 IJCNN_ , pages 1–8. IEEE.
  * Wang et al. (2018) Qing Wang, Wei Rao, Sining Sun, Leib Xie, Eng Siong Chng, and Haizhou Li. 2018.  Unsupervised domain adaptation via domain adversarial training for speaker recognition.  In _2018 IEEE ICASSP_ , pages 4889–4893. IEEE.
  * Weller et al. (2022) Orion Weller, Matthias Sperber, Telmo Pires, Hendra Setiawan, Christian Gollan, Dominic Telaar, and Matthias Paulik. 2022.  End-to-end speech translation for code switched speech.  _arXiv preprint arXiv:2204.05076_.
  * Xu et al. (2021) Haoran Xu, Seth Ebner, Mahsa Yarmohammadi, Aaron Steven White, Benjamin Van Durme, and Kenton Murray. 2021.  Gradual fine-tuning for low-resource domain adaptation.  _arXiv preprint arXiv:2103.02205_.
  * Xu and Yvon (2021) Jitao Xu and François Yvon. 2021.  Can you traducir this? machine translation for code-switched input.  _arXiv preprint arXiv:2105.04846_.
  * Yılmaz et al. (2016) Emre Yılmaz, Henk van den Heuvel, and David van Leeuwen. 2016.  Code-switching detection using multilingual dnns.  In _2016 IEEE Spoken Language Technology Workshop (SLT)_ , pages 610–616.
  * Zhang et al. (2022) Chao Zhang, Bo Li, Tara Sainath, Trevor Strohman, Sepand Mavandadi, Shuo-yiin Chang, and Parisa Haghani. 2022.  Streaming end-to-end multilingual speech recognition with joint language identification.  _arXiv preprint arXiv:2209.06058_.
  * Zhang et al. (2021) Shuai Zhang, Jiangyan Yi, Zhengkun Tian, Jianhua Tao, and Ye Bai. 2021.  Rnn-transducer with language bias for end-to-end mandarin-english code-switching speech recognition.  In _2021 12th ISCSLP_ , pages 1–5. IEEE.
  * Zhou et al. (2022) Long Zhou, Jinyu Li, Eric Sun, and Shujie Liu. 2022.  A configurable multilingual model is all you need to recognize all languages.  In _2022 IEEE ICASSP_ , pages 6422–6426. IEEE.