#  System size dependent topological zero modes in coupled topolectrical
chains

S M Rafi-Ul-Islam<EMAIL_ADDRESS>Department of Electrical and Computer
Engineering, National University of Singapore, Singapore  Zhuo Bin Siu
<EMAIL_ADDRESS>Department of Electrical and Computer Engineering, National
University of Singapore, Singapore  Haydar Sahin<EMAIL_ADDRESS>Department of Electrical and Computer Engineering, National University of
Singapore, Singapore  Institute of High Performance Computing, A*STAR,
Singapore  Ching Hua Lee<EMAIL_ADDRESS>Department of Physics, National
University of Singapore, Singapore  Mansoor B. A. Jalil<EMAIL_ADDRESS>Department of Electrical and Computer Engineering, National University of
Singapore, Singapore

###### Abstract

In this paper, we demonstrate the emergence and disappearance of topological
zero modes (TZMs) in a coupled topolectrical (TE) circuit lattice.
Specifically, we consider non-Hermitian TE chains in which TZMs do not occur
in the individual uncoupled chains, but emerge when these chains are coupled
by inter-chain capacitors. The coupled system hosts TZMs which show size-
dependent behaviours and vanish beyond a certain critical size. In addition,
the emergence or disappearance of the TZMs in the open boundary condition
(OBC) spectra for a given size of the coupled system can be controlled by
modulating the signs of its inverse decay length. Analytically, trivial and
non-trivial phases of the coupled system can be distinguished by the differing
ranks of their corresponding Laplacian matrix. The TE circuit framework
enables the physical detection of the TZMs via electrical impedance
measurements. Our work establishes the conditions for inducing TZMs and
modulating their behavior in coupled TE chains.

##  I Introduction

One of the most interesting phenomena in condensed matter system is the
discovery of novel topologically protected states such as edge states [1, 2,
3, 4, 5], zero-energy modes [6, 7, 8], corner modes [9, 10, 11, 12] and, hinge
modes [13, 14, 15] in various Hermitian systems. These topologically protected
states constitute a new basis for diverse physical phenomena [16, 1, 17, 18]
and various related applications [19, 20, 21] because of their many
interesting characteristics such as their robustness against system disorders
and perturbations. The topologically non-trivial phases in such Hermitian
systems are characterized by Bloch wave-vectors that respect the usual bulk-
boundary correspondence (BBC), and their band spectra under open boundary
conditions (OBC) in the limit of infinite system size are identical to that
under periodic boundary conditions (PBC). The introduction of non-Hermiticity,
for example in the form of non-reciprocal couplings [22, 23, 24], may result
in the emergence of drastic differences between the OBC and PBC spectra and
the breakdown of the usual BBC and of the Bloch theorem [25, 26, 27, 28, 29,
30, 31, 32, 33, 34, 35]. Several non-Hermitian systems have been realized in
various platforms ranging from topolectrical [36, 37, 38, 39, 40, 41, 42, 43,
40, 44, 38, 45, 22], photonics [46, 47, 48], and acoustic [49, 50] systems, as
well as superconductors [51, 52, 53] and metamaterials [54, 52, 55]. In such
systems, the wavefunctions are localized in the vicinity of the system
boundaries under OBC, a phenomenon known as the non-Hermitian skin effect
(NHSE) [38, 56, 57, 58, 59, 60, 38, 61, 62, 63].

Interestingly, the introduction of coupling between two non-Hermitian chains
with dissimilar degree of non-Hermiticity fundamentally alters the topological
character of the coupled system [64]. It was recently found that coupled non-
Hermitian chains exhibit scale-free exponential wavefunctions [65] and the
critical NHSE (CNHSE) [59, 66, 67, 68, 69, 67, 57, 67], in which the
wavefunctions and eigenvalues experience a discontinuous transition as the
system size is increased beyond some critical point. However, much remains to
be understood regarding the evolution of the topological edge modes in such
coupled non-Hermitian systems including their dependence on system size.
In the following, we demonstrate via numerical and analytical results the
emergence of topologically protected zero modes (TZM) in a coupled non-
Hermitian topolectrical (TE) circuit chain consisting of inductors,
capacitors, and op-amps. We found that, under some specific combination of
inter-chain hopping and non-Hermiticity parameters, TZMs can emerge in the
coupled system even when the individual chains do not host any TZMs.
Conversely, when the individual chains are tuned to reside in the non-trivial
regimes, a finite inter-chain coupling can result in size-dependent zero modes
that persist up to critical system size and then vanish abruptly upon further
increase in the system size. Analytically, the emergence and disappearance of
the TZMs via inter-chain coupling can be predicted by evaluating the rank of a
matrix constructed from the eigenvectors of the surrogate Hamiltonian of the
circuit. Furthermore, in the physical TE circuit realization, the presences of
TZMs can be detected by impedance spectral measurements. In summary, we showed
the emergence and modulation of TZMs in coupled TE systems by varying either
the inter-chain coupling strengths or the system size.

Figure 1:  Schematic of a pair of coupled non-Hermitian TE chains. a. The
coupled TE circuit comprises of two non-Hermitian SSH chains with opposite
signs of the non-Hermiticity capacitances $C_{\gamma}$. Within each chain, the
intra unit cell coupling is directionally asymmetric; in the top (bottom)
chain, the couplings in the two directions are given by $C_{1}\pm
C_{\gamma}(C_{1}\pm\eta C_{\gamma})$, respectively. The coupling asymmetry is
implemented by negative impedance converter with current inversion (INIC). The
two chains are coupled via a inter-chain capacitor $C_{c}$. b. Grounding
mechanism for the different types of nodes. The couplings to the ground are
chosen such that a single resonant frequency of
$f_{r}=(2\pi\sqrt{L_{r}(C_{1}+C_{2}+C_{c})})^{-1}$ is established via tuning
the common inductance $L_{g}$, so that the onsite potential is zero at every
node.

## Results

###  I.1 Construction of coupled TE chains that host TZMs

The pivotal step in inducing TZMs is to construct a TE circuit lattice that
emulates the electronic band structure of a coupled non-Hermitian system
having skin modes of dissimilar inverse decay lengths. (The inverse decay
length is the imaginary part of the complex wavevector of the OBC mode.) To do
so, we consider the TE circuit array shown in Fig. 1 formed by connecting two
non-Hermitian chains (top and bottom rows of Fig. 1a) by an inter-chain
coupling capacitance $C_{c}$. In the absence of the inter-chain coupling, each
of the two chains are analogous to the non-Hermitian SSH model with asymmetric
directional couplings between two neighboring nodes in their unit cell. These
are given by ($C_{1}\pm C_{\gamma}$) in the two directions for the top chain,
and by ($C_{1}\pm\eta C_{\gamma}$) for the bottom chain. For both chains,
there is a reciprocal coupling of $C_{2}$ between adjacent unit cells (i.e.,
the coupling coefficient in the direction from a site to its neighbour is the
same as that in the direction from the neighbour to the site.) The coupling
asymmetry in the intra-chain segment is realized in the practical circuit via
negative impedance converters with current inversion (INICs)[25]. The non-
Hermiticity parameter $C_{\gamma}$ can be modulated so that the eigenmodes of
the upper and lower chains would have different inverse decay lengths when
$\eta\neq 1$. When the parameter $\eta$ is set to $-1$, the two chains are in
an antisymmetric configuration where the sum of the inverse decay lengths of
the two chains is zero.

Figure 2:  The OBC spectra as a functions of the system size for the
individual uncoupled $A$ and $B$ chains for later comparison with the coupled
chain system. a), b) The OBC eigenenergy distributions for chains $A$ and $B$,
respectively, at various system sizes $N$. Other common parameters: $\eta=1$,
$C_{1}=0.83$, $C_{2}=0.6$, and $C_{\gamma}=0.8$. The topological zero modes
are evident at all system lengths, unlike the coupled system which we shall
describe later.

The corresponding Laplacian for the circuit in Fig. 1a (multiplied by
$1/(i\omega)$) can be expressed as

$(i\omega)^{-1}L(k,\omega_{res})=\begin{pmatrix}H_{a}(k,\omega_{res})&C_{c}\mathcal{I}_{2\times
2}\\\ C_{c}\mathcal{I}_{2\times 2}&H_{b}(k,\omega_{res})\end{pmatrix},$ (1)

where

$\displaystyle H_{a}(k,\omega_{res})=$ (2)
$\displaystyle=\begin{pmatrix}(C_{1}+C_{2}+C_{c}-\frac{1}{\omega^{2}L_{r}})&C_{1}+C_{\gamma}+C_{2}\exp(-ik)\\\
C_{1}-C_{\gamma}+C_{2}\exp(ik)&(C_{1}+C_{2}+C_{c}-\frac{1}{\omega^{2}L_{r}})\end{pmatrix}$

and

$\displaystyle H_{b}(k,\omega_{res})=$ (3)
$\displaystyle=\begin{pmatrix}(C_{1}+C_{2}+C_{c}-\frac{1}{\omega^{2}L_{r}})&C_{1}-\eta
C_{\gamma}+C_{2}\exp(-ik)\\\ C_{1}+\eta
C_{\gamma}+C_{2}\exp(ik)&(C_{1}+C_{2}+C_{c}-\frac{1}{\omega^{2}L_{r}})\end{pmatrix}.$

We introduce the corresponding surrogate Hamiltonian to the Laplacian and the
non-Bloch factor $\beta=e^{ik}$ as
$H(\beta)\equiv(i\omega)^{-1}L(k=-i\ln\beta,\omega)$ where $k$ can now take or
imaginary on complex values under OBC. We will show that Eq. 1 represents a
coupled TE system that exhibits size-dependent non-trivial boundary states.

###  I.2 System size-dependent topological zero modes

To highlight the features of the coupled chain system, we first study the OBC
of the individual uncoupled chains for comparison. The OBC admittance
distributions are plotted at varying system size $N$ in Fig. 2 for the
individual uncoupled chains $A$ and $B$ at $\eta=1$, which implies that the
coupling strength for hoppings from left to right in the first chain is equal
to the coupling strength for the hopping in the reversed in the second chain,
and vice-versa, i.e., “anti-symmetric coupled chains”. Following the
properties of generic non-Hermitian SSH chains [70, 71], both individual
chains host TZMs for the parameter range of
$-\sqrt{C_{2}^{2}+C_{\gamma}^{2}}<C_{1}<\sqrt{C_{2}^{2}+C_{\gamma}^{2}}$. In
the limit of large system size, the TZMs of the uncoupled chains are
degenerate at zero energy, and their behaviour are independent of the system
size (see Fig. 2a, b). However for small system size (i.e. of the order of
$N\approx 4$), the TZMs become non-degenerate and are no longer pinned at zero
energy (see Fig. 2a, b). (The observed behaviour of the TZMs of uncoupled
chains are discussed in more detail in the Appendix).

Figure 3: Anomalous disappearance of topological modes as system size is
varied. Demonstration of the effect of system size and the inter-chain
coupling $C_{c}$ on the eigenenergy distribution for coupled TE chains. a)-c)
The OBC dispersion relations with respect to the system size at $\eta=1$ for
$C_{c}=10^{-7},0.005,0.3$ respectively. Zero-modes exist at small $C_{c}$ and
small $N$. d)-f) Comparison of the impedance $Z$ as functions of $N$ for three
values of $C_{c}=10^{-7},0.005,0.3$ respectively at $\eta=1$. High and low
impedance readouts signify the presence and absence of the topological zero-
modes, respectively. Other parameters: $C_{1}=0.83$, $C_{2}=0.6$ and
$C_{\gamma}=0.8$. The zero modes vanish at strong inter-chain coupling and /
or long chains.

Next, we plot the variation of OBC eigenenergy spectra as a function of $N$
for various values of the inter-chain couplings strengths ($C_{c}$)
corresponding to the nearly decoupled ($C_{c}=10^{-7}$), weakly coupled
($C_{c}=0.005$), and strongly coupled ($C_{c}=0.3$) systems. We set $\eta=1$,
which implies that the coupling strength for hoppings from left to right in
the first chain is equal to the coupling strength for the hopping from right
to left in the second chain, and vice-versa, i.e., “anti-symmetric coupled
chains”.

In nearly decoupled systems with very small $C_{c}$ values (on the order of
$10^{-7}$), the coupled systems host TZMs as long as the OBC spectra lie on
the real axis. These real energy spectra with TZMs in the coupled systems
occur at relatively small system sizes (see Fig. 3a). In other words, the TZMs
survive for system sizes less than some critical $N=N_{\mathrm{critical}}$.
When $N>N_{\mathrm{critical}}$, the OBC eigen spectra expand into the complex
plane and do not exhibit any TZMs (the four-fold degenerate TZMs split and
move away from the real axis). Therefore, in the nearly decoupled TE chains,
the TZMs vanish beyond a critical system size [57], i.e. they exhibit the non-
Hermitian skin effect (CNHSE) (see Fig. 3a). The emergence and disappearance
of these peculiar size-dependent TZMs in the coupled TE chains can be
characterized by impedance measurements between the two leftmost nodes of
Chain A as shown in Fig. 3d. Here, the presence or absence of the TZMs is
distinguished by high and low impedance readouts, respectively. Furthermore, a
sharp transition between the high and low impedance states occurs marks the
critical system size corresponding to the CNHSE.

As the magnitude of $C_{c}$ increases, the hybridization between the two
chains becomes more prominent and the characteristics of the coupled chains
deviate drastically from that of the uncoupled chains. For instance, even in
the case of weakly coupled chains ($C_{c}=0.005$), the CNHSE transition occurs
at a much smaller system size ($N\approx 1$) for our choice of model
parameters (see Fig. 3b). No TZMs exist for $N>N_{\mathrm{critical}}$,
resulting in the low impedance readouts in Fig. 3e. When the inter-chain
coupling $C_{c}$ takes on a significant value ( i.e., $C_{c}=0.3$) which is on
the order of the other parameters, the strong hybridization between two chains
prevents the emergence of TZMs and the OBC energy spectra become almost
independent of system size (see Fig. 3c). The absence of the TZMs translates
into very small impedance measurements for all system size (see Fig. 3f).

Figure 4: Impedance and zero-mode evolution under OBC for various
hybridization scenarios between TE chains. a),b) Evolution of TZM states as a
function of system size ($N$) for constructive (i.e., $\eta=-2$) and
destructive (i.e., $\eta=2$) hybridization, respectively. c),d) The variation
of impedance $Z$ with respect to system size ($N$) for constructive and
destructive hybridization respectively. Other common parameters: $C_{1}=0.83$,
$C_{2}=0.6$, $C_{c}=0.2$, and $C_{\gamma}=0.8$. Note that for this set of
parameters, TZMs exist only under condition of constructive hybridization.

Next, we plot the OBC energy spectra and impedance readout with respect to $N$
for the general case of $\eta\neq 1$, in which the coupling in the two chains
are no longer exactly anti-symmetric. For the case where $\eta<0$, the
$\ln|\beta|$ values of the eigenstates localized in the two chains $A$ and $B$
(i.e., $\ln|\beta|_{A-chain}=\frac{C_{1}+C_{\gamma}}{C_{1}-C_{\gamma}}$ and
$\ln|\beta|_{B-chain}=\frac{C_{1}+\eta C_{\gamma}}{C_{1}-\eta C_{\gamma}}$)
have the same sign. In this configuration, which we describe as exhibiting
“constructive hybridization”, (see [57]), there are well-defined TZMs in the
OBC spectra (see Fig. 4a). Furthermore, the TZMs exist for all values of
system size $N$, and thus the system no longer exhibits CNHSE. However, for
the case where $\eta$ has a positive value ($\eta\neq 1$), i.e., the
$ln|\beta|$ has opposite signs for the states localized in the two chains, the
configuration corresponds to ”destructive hybridization” and the TZMs vanish
(see Fig. 4b). As before, the presence and absence of TZMs under constructive
and destructive hybridization can be distinguished by the relatively large and
small impedance measurements, respectively, for a given $N$ (see Fig. 4c-d).

Figure 5: a. Phase diagram of the coupled system with respect to $C_{c}$ and
$\eta C_{\gamma}$. The other parameter values are chosen to be the same as
those in Figs. 3 and 4 for which the isolated $A$ chain is topologically non-
trivial. The label above each circle denotes the OBC spectra in Figs. 3 and 4,
which the corresponding parameter coordinates $(C_{c},\eta C_{\gamma})$
correspond to. b. Phase diagram of the coupled system over the $C_{c}$-$\eta
C_{\gamma}$ parameter space for the values of the other parameters as in Fig.
5a, except that $C_{\gamma}$ is now set to 1.6, so that chain $A$ is now
topologically trivial. The circle labelled ”5c” denotes the parameter pair of
$(C_{c}=0.3,\eta C_{\gamma}=-0.55)$ for which the admittance spectra are
plotted in panels c and d, respectively. The rectangles with the dotted
borders indicate the range of $\eta C_{\gamma}$ for which chain B is
topologically non-trivial. c. The variation of the admittance spectra for
$C_{\gamma}=1.6,C_{c}=0.3,\eta C_{\gamma}=-0.55$ with the system size $N$. d.
The OBC admittance spectrum for the parameter set in c. in the thermodynamic
limit (i.e., $N\to\infty$) found from the criteria that the admittance values
with the middle magnitudes should be coincident. Note the absence of the TZMs
highlighted in yellow in panel c here. These results show that topologically
zero modes may emerge or vanish in coupled chain systems regardless of the
topological character of the two the constituent uncoupled chains, i.e. TZMs
can exist in the coupled system when neither or either or both of the
constituent chains are topologically non-trivial.

A condition for the existence of a TZM in the thermodynamic limit is that the
system of (four) linear equations governing the boundary conditions at the
boundary where the TZM is localized (to ensure that the wavefunction vanishes
at that boundary) is not of full rank [72]. This fact allows a quick
analytical method of determining whether the system is topologically non-
trivial and hosts TZMs from the number of linearly independent zero admittance
eigenvectors among the first eigenvectors with the four smallest magnitudes of
$\beta$. In other words, we evaluate the rank of the matrix constructed from
the column-wise concatenation of these zero-admittance eigenvectors. The
Hamiltonian of the system would be deemed non-trivial if the rank of the
matrix is less than four. Details of this analytical procedure and the
rationale for this criterion are presented in the Appendix.

Fig. 5a shows the phase diagram of the system as a function of coupling
capacitance $C_{c}$ and non-Hermitian and asymmetric coupling parameter $\eta
C_{\gamma}$ with the other parameters taking the same values as those in Fig.
3(a) to (c) and 4(a),(b). The ranges of $C_{c}$ and $\eta C_{\gamma}$ shown
encompass those considered in Fig. 3 and 4, for which the positions in the
phase diagram corresponding to the plots in these figures are denoted by
circles with the corresponding labels. In particular, Fig. 5 shows that the
parameter set in Fig. 3(a) corresponds to the non-trivial regime. The absence
of TZMs at large $N$ for the parameter set in Fig. 3 can now be ascribed to
the topologically trivial nature of the coupled chains even though the
uncoupled chains in the figure are themselves topologically non-trivial. In
the nearly decoupled case in Fig. 3a, the topologically non-trivial nature of
the uncoupled chains are retained in the coupled chains of short system
lengths as shown by the existence of the TZMs. However, when the system size
exceeds the critical length corresponding to CNHSE the topologically trivial
nature of the coupled chains becomes dominant.

In the cases considered so far, the isolated chain A is always non-trivial. We
now set $C_{\gamma}=1.6$ so that chain A is now trivial and plot the phase
diagram under varying $C_{c}$ and $\eta C_{\gamma}$, while retaining the same
values for the other parameters, as shown in Fig. 5b. In the region outside
the dotted rectangles in Fig. 5b , both of the uncoupled chains $A$ and $B$
are trivial. Interestingly, topologically non-trivial phases can emerge when
both of these topologically trivial chains are coupled together, as evidenced
by the parameter space in the phase diagram lying outside the dotted
rectangles that corresponds to the non-trivial state. As an illustration, we
consider the specific case of $C_{c}=0.3$ and $\eta C_{\gamma}=0.55$, which is
denoted by a circle labelled “5c” in Fig. 5b. The admittance spectra for
increasing finite lengths of the system are shown in Fig. 5c. Fig. 5d in turn
shows the GBZ admittance spectrum obtained as the set of admittance values
where the two middle $\beta$ values, arranged in order of increasing
magnitude, coincide. Comparing Fig. 5c with 5d, it can be seen that the TZMs
marked in orange in the former are absent in the latter. This indicates that
the states marked in orange are indeed topological states, which are not
captured in the GBZ, unlike the remaining states, which are bulk states.

In conclusion, we show that modulation of the inter-chain $C_{c}$ and non-
Hermitian intra-chain $\eta C_{\gamma}$ couplings can control the onset or
disappearance of topological zero modes (TZMs) in the system. The TZMs in the
coupled system exhibits a more varied set of behaviour compared to that of
single non-Hermitian chains that have been studied hitherto. Our theoretical
results reveal that depending on the specific values of $C_{c}$, $\eta
C_{\gamma}$ and other capacitive couplings, TZMs can be made to appear or
vanish in the coupled system for all possible topological character of the two
constituent chains. In other words, TZMs can be absent or present for all
scenarios, i.e., when neither, either, or both of the uncoupled chains are
topologically non-trivial. The emergence of TZMs in the coupled system when
neither of the two constituent chains are topologically non-trivial implies
the key role played by the inter-chain coupling in determining the topological
character of the coupled system. Additionally, in the converse case where both
the consituent uncoupled chains are topologically non-trivial while the
coupled system is topologically trivial, one observes a size-dependent effect
in which TZMs persist at small inter-chain coupling strengths and for small
system size, but disappear when the system size exceeds a critical limit, a
phenomenon known as the critical non-Hermitian skin effect (CNHSE). We devise
an analytical method to distinguish the trivial and non-trivial phases of the
coupled system by considering the rank of the matrix constructed from zero-
admittance eigenvectors of the surrogate Hamiltonian. Based on this analytical
method, we plot the modified phase diagram of the coupled system over the
$C_{c}$ and $\eta C_{\gamma}$ parameter space, and elucidate the role of the
inter-chain and non-Hermitian couplings in determining the topology of the
coupled system. In practice, the trivial and non-trivial phases of the coupled
TE chains and their evolution with system size can be distinguished by circuit
impedance measurements, with the TZMs being associated with higher impedance
readouts. More broadly, our results indicate a practical means to induce TZMs
and modulate the topological phase transitions in coupled systems based on the
TE circuit platforms.

##  II Appendix

###  II.1 Analytical determination of topological non-triviality

The eigenstates of a non-Hermitian system may be localized at its edges under
OBC because of the two distinct mechanisms of the non-Hermitian skin effect or
because they are topologically non-trivial edge states. The former requires
boundaries to be present at both ends of the system, and occurs only at
energies at which the two median values of $|\beta|$ match in the
thermodynamic limit. This arises from the requirement that the linear
superposition of the PBC eigenstates constituting the OBC eigenstate satisfy
the boundary conditions that the wavefunction of the OBC eigenstate vanishes
at both boundaries simultaneously. (See our earlier paper for the details
[57]).

Another mechanism for the emergence of localized states is when these states
are topologically non-trivial. A hallmark of such a topologically non-trivial
state is that the system of linear equations for the boundary conditions at
the boundary where the state is localized near is not of full rank [72]. Here,
we describe a quick approach to determine whether a non-Hermitian system with
only nearest-neighbor inter-unit cell coupling supports topologically non-
trivial states near zero energy. Consider the Hamiltonian for a single
uncoupled chain with an arbitrary number of nodes, $2n_{d}$, in each unit cell
represented by

$H(\beta)=\begin{pmatrix}\mathbf{0}&\mathbf{H}_{-}\beta^{-1}+\mathbf{t}_{-+}\\\
\mathbf{H}_{+}\beta+\mathbf{t}_{+-}&\mathbf{0}\end{pmatrix}.$ (4)

where the quantities in bold are $n_{d}$ by $n_{d}$ matrices. We write the
time-independent Schrödigner equation for Eq. (4) as

$H(\beta)|\Psi\rangle=|\Psi\rangle E$ (5)

where $|\Psi\rangle$ is the (right) eigenstate with energy $E$. The
characteristic polynomial for an eigenenergy of 0, $|H(\beta)|=0$, can be made
into a quadratic polynomial in $\beta$. This implies that there are, in
general, only two finite values of $\beta$ that will result in $H(\beta)$
having an eigenenergy of zero regardless of the value of $n_{d}$. When
$n_{d}>1$, the system also admits the $\beta$ values of 0 and $\infty$: At
$\beta=0$, ($\infty$) the term containing $\beta^{-1}$ ($\beta$) in Eq. (4)
dominates over the remaining terms in Eq. (5), including the $|\Psi\rangle E$
term on the right side of the equal sign and Eq. (5) effectively becomes
$H_{+}|\Psi\rangle=0$ ($H_{-}|\Psi\rangle=0$). There are $n_{d}-1$
eigenvectors for $\beta=0$, and another $n_{d}-1$ eigenvectors for
$\beta=\infty$. Denoting the $i$th solution of $H_{\pm}|\Psi\rangle=0$ as
$|\Psi_{\pm;0}^{i}\rangle$, the $(n_{d}-1)$ $|\Psi_{\pm;0}^{i}\rangle$s for
each of the two signs of $\pm$ in the subscript constitute eigenvectors with
the effective values of $\ln|\beta|=\pm\infty$ that are localized at the right
(left) edges of the system. For the ease of explanation, let us now consider
the simplest example of Eq. 4 where all the bolded quantities are scalars,
i.e., the single uncoupled SSH chain described by the Hamiltonian

$H=\begin{pmatrix}0&c_{12}+d_{12}/\beta\\\ c_{21}+d_{21}\beta&0\end{pmatrix}$
(6)

where $c_{12}$ and $c_{21}$ denote the intra-unit cell coupling, and $d_{12}$
and $d_{21}$ the inter-unit cell coupling ($c_{ij}$ and $d_{ij}$ are non-zero
parameters). It can be easily found that the eigenstates and $\beta$ values of
Eq. 6 at $E=0$ are

$\displaystyle(1,0)^{\mathrm{T}}$
$\displaystyle:\beta=0,\beta=-c_{21}/d_{21};$
$\displaystyle(0,1)^{\mathrm{T}}$
$\displaystyle:\beta=\infty,\beta=-d_{12}/c_{12}.$ (7)

We assume for the moment that $|c_{21}/d_{21}|<1$, and consider a semi-
infinite system that spans from $x=1,2,...,\infty$. In this case, the system
can host an edge state $\psi_{\mathrm{left}}(x)$ satisfying the boundary
condition that $\psi_{\mathrm{left}}(x=0)=0$ given by

$\psi_{\mathrm{left}}(x)=\begin{pmatrix}1\\\
0\end{pmatrix}(\delta_{x,0}-(-c_{21}/d_{21})^{x})$ (8)

where $x$ in an integer (because the Hamiltonian describes a lattice system),
and the $\delta_{x,0}$ is the Kronecker (not Dirac) delta due to the
localization of the $\beta=0$ state on the left edge of the system. Note that
Eq. (8) is not admissible as an eigenstate of a system with a finite length
that has both a left and a right edge, as can be seen from the following:
Denoting the length of the finite system as $N$ so that that $x=1,2,...,N$ we
see substituting $x\rightarrow(N+1)$ into Eq. (8) does not satisfy the
requirement that the wavefunction vanishes at $x=(N+1)$ because
$\psi_{\mathrm{left}}(N+1)=(1,0)^{\mathrm{T}}(-(c_{21}/d_{21}))^{N+1})$.
Moreover, this remaining bit cannot be canceled off by any linear combination
of the remaining two eigenvectors, which are proportional to
$(0,1)^{\mathrm{T}}$. The formation of edge states that will satisfy the
boundary conditions at both edges in finite-length systems therefore requires
the energy to be shifted slightly away from $E=0$ so that the $\beta$ values
and eigenvectors in Eq. (7) now become

$\displaystyle\begin{pmatrix}1\\\ 0\end{pmatrix},\beta=0;\begin{pmatrix}1\\\
\delta a_{1}\end{pmatrix},\beta_{1}=-c_{21}/d_{21}+\delta b_{1};$ (9)
$\displaystyle\begin{pmatrix}\delta a_{2}\\\
1\end{pmatrix},\beta_{2}=-d_{12}/c_{12}+\delta b_{2};\begin{pmatrix}0\\\
1\end{pmatrix},\beta=\infty$

where the $\delta a_{i}$s and $\delta b_{i}$s are small shifts due to the
shift in the energy. The presence of the $\delta a_{i}$s in the eigenspinors
of the terms with finite $\beta$s now allow the terms to cancel off one
another at both boundaries to satisfy the boundary conditions at both edges.
For an edge state localized at the left edge, the resultant (unnormalized)
wavefunction can be written as

$\displaystyle\psi_{\mathrm{finite}}(x)=$ $\displaystyle\begin{pmatrix}1\\\
0\end{pmatrix}\delta_{x,0}+c_{1}\begin{pmatrix}1\\\ \delta
a_{1}\end{pmatrix}(\beta_{1})^{x}$ (10)
$\displaystyle+d_{2}\begin{pmatrix}\delta a_{2}\\\
1\end{pmatrix}(\beta_{2})^{x}+d_{3}\begin{pmatrix}0\\\
1\end{pmatrix}\delta_{x,N+1}$

where
$c_{1}\rightarrow-1,\beta_{1}\rightarrow(-c_{21}/d_{21}),\beta_{2}\rightarrow(-d_{12}/c_{12})$
and $\delta a_{1},\delta a_{2},d_{2},d_{3}\rightarrow 0$ as
$N\rightarrow\infty$. The finite length can in this case be interpreted as a
perturbation to the “ideal case” of the semi-infinite system.

Note that for Eq. (10) to describe a state localized at the left edge, we
require $|\beta_{1}|<|\beta_{2}|$ so that at the left edge at $x=0$, the
$(\delta a_{2},1)^{\mathrm{T}}$ state has a miniscule weight compared to the
$(1,a_{1})^{\mathrm{T}}$ state and the latter is largely cancelled off by the
$(1,0)^{\mathrm{T}}$ state with $\beta=0$, which is confined to only the left
edge. A similar consideration for the edge state localized on the right edge
of the system would also lead to the requirement for $|\beta_{1}|<|\beta_{2}|$
in order for such an edge state to be formed.

There is another crucial difference between the semi-infinite length system,
which has only a single edge, and the finite length system, which has two
edges. In the latter, there are no restrictions on whether $|\beta_{1}|$ or
$|\beta_{2}|$ are required to be smaller or larger than 1 for edge states to
exist in the system. In the semi-infinite system extending from
$x=[0,\infty)$, it is strictly required that the finite $|\beta|<1$ so that
the wavefunction amplitude remains bounded as it extends to infinity. In
contrast, in a system of finite length, we saw in Eq. (10) that in general, a
wavefunction satisfying the OBC at both the left and right boundaries requires
the simultaneous combination of the $|\beta|$ values that may be bigger or
smaller than one. The finite length of the system ensures that the resulting
wavefunction remains bounded to finite values. The simple SSH model discussed
above suggests the following criteria for quickly establishing whether
topological edge states can exist near $E=0$ in more general systems,
including the 4-by-4 Hamiltonian considered in the Eq. (4) of the coupled SSH
chain system, without having to perform the far more intensive calculation of
the topological invariant of the system: First, solve for the $\beta$ values
and their corresponding eigenvectors of the $2n_{d}$ by $2n_{d}$ Hamiltonian
at the eigenvalue of $E=0$. Arrange the $2n_{d}$ $\beta$ values (including
$\beta=0$ and $\beta=\infty$) in ascending order of $|\beta|$. If the rank of
the square matrix formed by the column-wise concatenation of the first (last)
$n_{d}$ eigenvectors is less than $n_{d}$, the system hosts a topological edge
state on the left (right) edge. In contrast, if the rank is $n_{d}$, there can
be no edge states. This can be seen from the definition of the rank of a
matrix as the number of linearly independent vectors in the matrix that the
only combination of the eigenvectors that sums up to 0 at the boundary to
satisfy the boundary condition is the trivial combination that the weights of
all the eigenvectors are 0. For instance, returning to the example of Eq. (6)
and its eigenvectors Eq. (7), if $|c_{21}/d_{21}|<|d_{12}/c_{12}|$ then the
corresponding square matrix formed by concatenating the first two eigenvectors
is $\begin{pmatrix}1&1\\\ 0&0\end{pmatrix}$, which has a rank less than 2 and
a linear superposition of these two eigenvectors that sums to zero can be
found without the weights both equal to 0. In contrast, if the condition is
not satisfied, the resulting matrix would be $\mathbf{I}_{2}$, which has a
rank of 2. In this case, the system does not host nearly-zero energy edge
modes because there is no non-trivial linear combination of the eigenvectors
that sums to 0.

## References

  * Hafezi  _et al._ [2013] M. Hafezi, S. Mittal, J. Fan, A. Migdall, and J. Taylor, Nat. Photon. 7, 1001 (2013).
  * Obana  _et al._ [2019] D. Obana, F. Liu, and K. Wakabayashi, Phys. Rev. B 100, 075437 (2019).
  * St-Jean  _et al._ [2017] P. St-Jean, V. Goblot, E. Galopin, A. Lemaître, T. Ozawa, L. Le Gratiet, I. Sagnes, J. Bloch, and A. Amo, Nat. Photon. 11, 651 (2017).
  * Rafi-Ul-Islam  _et al._ [2021a] S. Rafi-Ul-Islam, Z. B. Siu, and M. B. Jalil, Phys. Rev. B 103, 035420 (2021a).
  * Gao  _et al._ [2020a] H. Gao, H. Xue, Q. Wang, Z. Gu, T. Liu, J. Zhu, and B. Zhang, Phys. Rev. B 101, 180303 (2020a).
  * Ganeshan  _et al._ [2013] S. Ganeshan, K. Sun, and S. D. Sarma, Phys. Rev. Lett. 110, 180403 (2013).
  * Kempkes  _et al._ [2019] S. Kempkes, M. Slot, J. van Den Broeke, P. Capiod, W. Benalcazar, D. Vanmaekelbergh, D. Bercioux, I. Swart, and C. M. Smith, Nat. Mater. 18, 1292 (2019).
  * Ryu and Hatsugai [2002] S. Ryu and Y. Hatsugai, Phys. Rev. Lett. 89, 077002 (2002).
  * Imhof  _et al._ [2018] S. Imhof, C. Berger, F. Bayer, J. Brehm, L. W. Molenkamp, T. Kiessling, F. Schindler, C. H. Lee, M. Greiter, T. Neupert, _et al._ , Nat. Phys. 14, 925 (2018).
  * Sahin  _et al._ [2022a] H. Sahin, S. Rafi-Ul-Islam, Z. B. Siu, C. H. Lee, and M. Jalil, Bull. Am. Phys. Soc. (2022a).
  * Banerjee  _et al._ [2020] R. Banerjee, S. Mandal, and T. Liew, Phys. Rev. Lett. 124, 063901 (2020).
  * Wu  _et al._ [2021] X. Wu, Y. Meng, Y. Hao, R.-Y. Zhang, J. Li, and X. Zhang, Phys. Rev. Lett. 126, 226802 (2021).
  * Kunst  _et al._ [2018a] F. K. Kunst, G. van Miert, and E. J. Bergholtz, Phys. Rev. B 97, 241405 (2018a).
  * Benalcazar  _et al._ [2017] W. A. Benalcazar, B. A. Bernevig, and T. L. Hughes, Phys. Rev. B 96, 245115 (2017).
  * Ghorashi  _et al._ [2019] S. A. A. Ghorashi, X. Hu, T. L. Hughes, and E. Rossi, Phys. Rev. B 100, 020509 (2019).
  * Jin  _et al._ [2018] Y. Jin, D. Torrent, and B. Djafari-Rouhani, Phys. Rev. B 98, 054307 (2018).
  * Fujita  _et al._ [2011] T. Fujita, M. Jalil, S. Tan, and S. Murakami, J. Appl. Phys. 110, 17 (2011).
  * Tan  _et al._ [2020] S. G. Tan, S.-H. Chen, C. S. Ho, C.-C. Huang, M. B. Jalil, C. R. Chang, and S. Murakami, Phys. Rep. 882, 1 (2020).
  * Wu  _et al._ [2017] Y. Wu, C. Li, X. Hu, Y. Ao, Y. Zhao, and Q. Gong, Adv. Opt. Mater. 5, 1700357 (2017).
  * Vobornik  _et al._ [2011] I. Vobornik, U. Manju, J. Fujii, F. Borgatti, P. Torelli, D. Krizmancic, Y. S. Hor, R. J. Cava, and G. Panaccione, Nano Lett. 11, 4079 (2011).
  * Kesselring  _et al._ [2018] M. S. Kesselring, F. Pastawski, J. Eisert, and B. J. Brown, Quantum 2, 101 (2018).
  * Helbig  _et al._ [2020] T. Helbig, T. Hofmann, S. Imhof, M. Abdelghany, T. Kiessling, L. Molenkamp, C. Lee, A. Szameit, M. Greiter, and R. Thomale, Nat. Phys. 16, 747 (2020).
  * Qiu  _et al._ [2019] D.-X. Qiu, R. Bai, C. Zhang, L.-F. Xin, X.-Y. Zou, Y. Q. Zhang, X. R. Jin, C. An, and S. Zhang, Quant. Inf. Process. 18, 1 (2019).
  * Santos and da Silva [2014] R. B. Santos and V. R. da Silva, Mod. Phys. Lett. B 28, 1450223 (2014).
  * Rafi-Ul-Islam  _et al._ [2021b] S. Rafi-Ul-Islam, Z. B. Siu, and M. B. Jalil, New J. Phys. 23, 033014 (2021b).
  * Li and Lee [2022] L. Li and C. H. Lee, Sci. Bull. (2022).
  * Yang  _et al._ [2022] R. Yang, J. W. Tan, T. Tai, J. M. Koh, L. Li, S. Longhi, and C. H. Lee, arXiv preprint arXiv:2201.04153 (2022).
  * Zeuner  _et al._ [2015] J. M. Zeuner, M. C. Rechtsman, Y. Plotnik, Y. Lumer, S. Nolte, M. S. Rudner, M. Segev, and A. Szameit, Phys. Rev. Lett. 115, 040402 (2015).
  * Tai and Lee [2022] T. Tai and C. H. Lee, arXiv preprint arXiv:2202.03462 (2022).
  * Li  _et al._ [2021a] L. Li, S. Mu, C. H. Lee, and J. Gong, Nat. Commun. 12, 1 (2021a).
  * Gu  _et al._ [2016] Y. Gu, C. H. Lee, X. Wen, G. Y. Cho, S. Ryu, and X.-L. Qi, Phys. Rev. B 94, 125107 (2016).
  * Leykam  _et al._ [2017] D. Leykam, K. Y. Bliokh, C. Huang, Y. D. Chong, and F. Nori, Phys. Rev. Lett. 118, 040401 (2017).
  * Gong  _et al._ [2018] Z. Gong, Y. Ashida, K. Kawabata, K. Takasan, S. Higashikawa, and M. Ueda, Phys. Rev. X 8, 031079 (2018).
  * El-Ganainy  _et al._ [2018] R. El-Ganainy, K. G. Makris, M. Khajavikhan, Z. H. Musslimani, S. Rotter, and D. N. Christodoulides, Nat. Phys. 14, 11 (2018).
  * Yokomizo and Murakami [2019] K. Yokomizo and S. Murakami, Phys. Rev. Lett. 123, 066404 (2019).
  * Rafi-Ul-Islam  _et al._ [2020a] S. Rafi-Ul-Islam, Z. B. Siu, and M. B. Jalil, Commun. Phys. 3, 1 (2020a).
  * Zhang  _et al._ [2022] X. Zhang, B. Zhang, H. Sahin, Z. B. Siu, S. Rafi-Ul-Islam, J. F. Kong, M. Jalil, R. Thomale, and C. H. Lee, arXiv preprint arXiv:2204.05329 (2022).
  * Rafi-Ul-Islam  _et al._ [2021c] S. Rafi-Ul-Islam, Z. B. Siu, H. Sahin, C. H. Lee, and M. Jalil, ArXiv Preprint ArXiv:2108.01366 (2021c).
  * Lenggenhager  _et al._ [2021] P. M. Lenggenhager, A. Stegmaier, L. K. Upreti, T. Hofmann, T. Helbig, A. Vollhardt, M. Greiter, C. H. Lee, S. Imhof, H. Brand, _et al._ , arXiv preprint arXiv:2109.01148 (2021).
  * Rafi-Ul-Islam  _et al._ [2020b] S. Rafi-Ul-Islam, Z. B. Siu, and M. B. Jalil, Appl. Phys. Lett. 116, 111904 (2020b).
  * Stegmaier  _et al._ [2021] A. Stegmaier, S. Imhof, T. Helbig, T. Hofmann, C. H. Lee, M. Kremer, A. Fritzsche, T. Feichtner, S. Klembt, S. Höfling, _et al._ , Phys. Rev. Lett. 126, 215302 (2021).
  * Rafi-Ul-Islam  _et al._ [2020c] S. Rafi-Ul-Islam, Z. B. Siu, C. Sun, and M. B. Jalil, New J. Phys. 22, 023025 (2020c).
  * Lee  _et al._ [2018] C. H. Lee, S. Imhof, C. Berger, F. Bayer, J. Brehm, L. W. Molenkamp, T. Kiessling, and R. Thomale, Commun. Phys. 1, 1 (2018).
  * Hofmann  _et al._ [2020] T. Hofmann, T. Helbig, F. Schindler, N. Salgo, M. Brzezińska, M. Greiter, T. Kiessling, D. Wolf, A. Vollhardt, A. Kabaši, _et al._ , Phys. Rev. Res. 2, 023265 (2020).
  * Kotwal  _et al._ [2019] T. Kotwal, H. Ronellenfitsch, F. Moseley, A. Stegmaier, R. Thomale, and J. Dunkel, ArXiv Preprint ArXiv:1903.10130 (2019).
  * Zhu  _et al._ [2020] X. Zhu, H. Wang, S. K. Gupta, H. Zhang, B. Xie, M. Lu, and Y. Chen, Phys. Rev. Res. 2, 013280 (2020).
  * Song  _et al._ [2020] Y. Song, W. Liu, L. Zheng, Y. Zhang, B. Wang, and P. Lu, Phys. Rev. Appl. 14, 064076 (2020).
  * Xiao  _et al._ [2020] L. Xiao, T. Deng, K. Wang, G. Zhu, Z. Wang, W. Yi, and P. Xue, Nat. Phys. , 1 (2020).
  * Zhang and Gong [2020] X. Zhang and J. Gong, Phys. Rev. B 101, 045415 (2020).
  * Gao  _et al._ [2020b] P. Gao, M. Willatzen, and J. Christensen, Phys. Rev. Lett. 125, 206402 (2020b).
  * Wang  _et al._ [2021] Z.-H. Wang, F. Xu, L. Li, D.-H. Xu, W.-Q. Chen, and B. Wang, Phys. Rev. B 103, 134507 (2021).
  * Zhou [2020] L. Zhou, Phys. Rev. B 101, 014306 (2020).
  * Cao  _et al._ [2021] Y. Cao, Y. Li, Y. Chen, and X. Yang, ArXiv Preprint ArXiv:2103.17157 (2021).
  * Schomerus [2020] H. Schomerus, Phys. Rev. Res. 2, 013058 (2020).
  * Ghatak  _et al._ [2020] A. Ghatak, M. Brandenbourger, J. van Wezel, and C. Coulais, Proc. Nat. Acad. Sci. 117, 29561 (2020).
  * Longhi [2019] S. Longhi, Phys. Rev. Res. 1, 023013 (2019).
  * Rafi-Ul-Islam  _et al._ [2021d] S. Rafi-Ul-Islam, Z. B. Siu, H. Sahin, C. H. Lee, and M. Jalil, ArXiv Preprint ArXiv:2108.02457 (2021d).
  * Okuma  _et al._ [2020] N. Okuma, K. Kawabata, K. Shiozaki, and M. Sato, Phys. Rev. Lett. 124, 086801 (2020).
  * Li  _et al._ [2020a] L. Li, C. H. Lee, S. Mu, and J. Gong, Nat. Commun. 11, 1 (2020a).
  * Song  _et al._ [2019] F. Song, S. Yao, and Z. Wang, Phys. Rev. Lett. 123, 170401 (2019).
  * Kawabata  _et al._ [2020] K. Kawabata, M. Sato, and K. Shiozaki, Phys. Rev. B 102, 205118 (2020).
  * Yao and Wang [2018] S. Yao and Z. Wang, Phys. Rev. Lett. 121, 086803 (2018).
  * Li  _et al._ [2020b] L. Li, C. H. Lee, and J. Gong, Phys. Rev. Lett. 124, 250402 (2020b).
  * Rafi-Ul-Islam  _et al._ [2022] S. Rafi-Ul-Islam, Z. B. Siu, H. Sahin, C. H. Lee, and M. B. Jalil, Phys. Rev. Res. 4, 013243 (2022).
  * Li  _et al._ [2021b] L. Li, C. H. Lee, and J. Gong, Commun. Phys. 4, 1 (2021b).
  * Zhang  _et al._ [2021] W. Zhang, D. Zou, Q. Pei, W. He, J. Bao, H. Sun, and X. Zhang, Phys. Rev. Lett. 126, 146802 (2021).
  * Liu  _et al._ [2020] C.-H. Liu, K. Zhang, Z. Yang, and S. Chen, ArXiv Preprint ArXiv:2005.02617 (2020).
  * Sahin  _et al._ [2022b] H. Sahin, S. Rafi-Ul-Islam, Z. B. Siu, C. H. Lee, and M. Jalil, Bull. Am. Phys. Soc. (2022b).
  * Yokomizo and Murakami [2021] K. Yokomizo and S. Murakami, Phys. Rev. B 104, 165117 (2021).
  * Kunst  _et al._ [2018b] F. K. Kunst, E. Edvardsson, J. C. Budich, and E. J. Bergholtz, Phys. Rev. Lett. 121, 026808 (2018b).
  * Lieu [2018] S. Lieu, Phys. Rev. B 97, 045106 (2018).
  * Lee and Thomale [2019] C. H. Lee and R. Thomale, Phys. Rev. B 99, 201103 (2019).


# WanJuan-CC: A Safe and High-Quality Open-sourced English Webtext Dataset

Jiantao Qiu Haijun Lv11footnotemark: 1 Zhenjiang Jin Rui Wang Wenchang Ning
Jia Yu Chaobin Zhang Pei Chu Yuan Qu Jin Shi Lindong Lu
Runyu Peng Zhiyuan Zeng Huanze Tang Zhikai Lei Jiawei Hong
Keyu Chen Zhaoye Fei Ruiliang Xu Wei Li
Zhongyin Tu Hang Yan Conghui He22footnotemark: 2
Shanghai Artifcial Intelligence Laboratory
Shanghai, 200232, China
<EMAIL_ADDRESS><EMAIL_ADDRESS>
Co-first authors.Co-corresponding author

###### Abstract

This paper presents WanJuan-CC, a safe and high-quality open-sourced English
webtext dataset derived from Common Crawl data. The study addresses the
challenges of constructing large-scale pre-training datasets for language
models, which require vast amounts of high-quality data. A comprehensive
process was designed to handle Common Crawl data, including extraction,
heuristic rule filtering, fuzzy deduplication, content safety filtering, and
data quality filtering. From approximately 68 billion original English
documents, we obtained 2.22T Tokens of safe data and selected 1.0T Tokens of
high-quality data as part of WanJuan-CC. We have open-sourced 100B Tokens from
this dataset. The paper also provides statistical information related to data
quality, enabling users to select appropriate data according to their needs.
To evaluate the quality and utility of the dataset, we trained 1B-parameter
and 3B-parameter models using WanJuan-CC and another dataset, RefinedWeb.
Results show that WanJuan-CC performs better on validation datasets and
downstream tasks.

##  1 Introduction

Recently, large-scale language models have achieved significant
accomplishments in numerous natural language processing tasks. Notable
research outcomes such as GPT3[1], BLOOM[2], LLAMA[3], Palm[4], Falcon[5],
GPT4[6], InternLM[7], ChatGLM[8], and Baichuan[9] have been reported
successively. These language models, pre-trained on large-scale data,
demonstrate excellent performance across various NLP tasks, even without the
need for fine-tuning with labeled data[1].

Existing research indicates that there exists a scaling relationship, known as
the Scaling Law, among optimal model parameters, training data volume, and
total computational overhead in the training process of large models[10]. This
implies that to train more powerful models, we need to simultaneously increase
the number of model parameters and the scale of the pre-training dataset.
Studies have shown that training a language model with 175B parameters
requires approximately 3.7T tokens of high-quality pre-training data. However,
traditional data schemes that collect and custom clean from specific data
sources can no longer meet this scale of data demand, posing new challenges
for the construction of pre-training datasets.

Webtext, text data scraped and extracted from the internet, is a key resource
for building pre-training datasets. Many large language model studies [1; 2;
3; 4] have claimed to incorporate webtext data as part of their pre-training
data, mixing it with other high-quality data sources for model training. Some
research even demonstrates that using appropriately processed web-scraped data
alone can achieve results similar to those obtained with mixed datasets of
high-quality data [5].

Common Crawl[11] maintains a freely accessible web-crawled database,
significantly reducing the cost of re-crawling data. It provides historical
web page data since 2008, making it an essential source for webtext data.
However, the quality of data in Common Crawl is inconsistent, with a
substantial amount of low-quality data such as format errors, duplicate
content, and advertising information, which are not beneficial for model
training. Removing these low-quality data can optimize the training effect of
the model. Moreover, Common Crawl contains a large amount of unsafe content,
such as toxic and pornographic materials, which could lead to unfriendly
outputs from the model. The presence of personally identifiable information
(PII) in the data could also infringe on user privacy. Therefore, to enhance
the safety of the model, it is necessary to filter or mask these data.

To address the aforementioned issues, we designed and implemented a process
for handling Common Crawl data, which includes data extraction, heuristic rule
filtering, fuzzy deduplication, content safety filtering, and data quality
filtering. From the Common Crawl, we selected and extracted approximately 68B
original English documents. After undergoing this series of processing steps,
we obtained 2.22T Tokens of safe data. Further quality filtering yielded 1T
Tokens of high-quality data, forming what we refer to as the WanJuan-CC
dataset. We have extracted 100B Tokens from WanJuan-CC as open-source data,
accompanied by some statistical information related to data quality, allowing
users to select suitable data based on their specific needs.

In summary, the main contributions of this study are as follows:

  1. 1.

We designed and implemented a process for handling Common Crawl data, which
includes steps such as data extraction, heuristic rule filtering, fuzzy
deduplication, content safety filtering, and data quality filtering.

  2. 2.

We extracted raw data from approximately 68 billion documents in Common Crawl.
After processing through the aforementioned steps, we obtained 2.22T Tokens
(9.71TB) of safe data, and selected 1.0T Tokens (4.45TB) of high-quality data
as part of WanJuan-CC. We have open-sourced 100B Tokens from this dataset.

  3. 3.

We included some statistical information related to data quality in the
dataset, allowing users to select appropriate data according to their needs.

  4. 4.

We used WanJuan-CC and RefinedWeb as training data to train 1B-parameter and
3B-parameter models, using perplexity on the validation dataset and accuracy
on downstream tasks as evaluation metrics. The results show that the data
quality of WanJuan-CC is superior to that of RefinedWeb.

##  2 Related Work

###  2.1 Webtext Datasets

By openly sharing meticulously processed datasets with the community, we can
propel advancements in the field of large language models and reduce the cost
of data processing for various research institutions. Currently, some publicly
available datasets directly utilize the Common Crawl database or adopt the
OSCAR dataset [12] based on Common Crawl. This primarily includes studies such
as RefinedWeb [5], Redpajama [13], and Dolma [14].

Dataset Total Size$\star$ CC Dumps Quality Cls$\dagger$ URL/Words Toxic/Porn
Cls$\dagger$ PII Mask Redpajama 1.2T 46 Yes No No No Dolma 3.1T 24 No No Toxic
Yes RefinedWeb 5.0T 94 No Yes No No WanJuan-CC 1.0T 90 Yes Yes Both Yes
$\star$ : Count the total size of the dataset claimed by the publisher.
$\dagger$ : Quality Cls and Toxic/Porn Cls mean use model-based classifier to
filter the data.

Table 1: Comparison of Pre-training Datasets Including Common Crawl Data

The primary attributes of datasets processed based on Common Crawl include
dataset size, the number of CC dumps, as well as the safety of the dataset,
personal privacy protection, and data quality filtering methods. These are
widely recognized features. We have conducted relevant attribute statistics
for the aforementioned datasets and the dataset in this study, and the results
are summarized in Table 1. According to the statistical data in the table,
among the existing public datasets, only RefinedWeb covers data from more than
90 CC dumps. Currently, no public dataset fully covers content safety measures
in terms of toxicity, pornography, and personally identifiable information
(PII). Moreover, only redpajama has adopted a model-based quality filtering
method, while other datasets merely use heuristic rules for quality filtering.
The open-sourced dataset in this study covers 90 CC dumps. In addition to
keyword and URL-based blocking, we also used a model-based approach to exclude
data containing toxic and pornographic content, and regular expressions were
utilized to mask PII information. Finally, we adopted a model-based quality
filtering method to screen out relatively high-quality data. Ultimately, our
open-sourced dataset includes 1TB of high-quality data.

###  2.2 Data Utility

While Common Crawl provides a vast and diverse set of data, it is also
interspersed with a considerable amount of low-quality information. This
content could negatively impact the performance of models and may lead to the
generation of unsafe outputs, thereby reducing the practicality of the models.
We collectively refer to these two issues at the data level as Data Safety and
Data Quality, which are overall referred to as Data Utility. Herein, we
summarize some methods for enhancing Data Utility identified in related
research.

####  2.2.1 Data Safety

Employing keywords and domain names to filter certain data is a fundamental
measure for data security. For instance, RefinedWeb[5] utilizes a blacklist
comprising 4.6 million URLs to filter potentially harmful web pages, and
scores each URL based on an unsafe word list, setting thresholds to screen the
data. Similarly, Roots[15] also adopts a filtering method based on the density
of sensitive words.

Several studies have employed model-based methods to filter out toxic and
pornographic data. Toxic content is generally defined as material that causes
users to leave their current reading or discussion, posing a significant
problem for the primary applications of large language models. Pornography, on
the other hand, is typically defined as data containing explicit sexual
content. This type of data constitutes a considerable proportion in Common
Crawl, not only reducing the ratio of useful data but also potentially leading
to unsafe outputs from the model. In Dolma[14], a toxic classifier trained on
the Jigsaw dataset[16] was used to filter the data.

The Common Crawl dataset contains a significant number of pages with
Personally Identifiable Information (PII), and training models on this data
can lead to privacy breaches. To address this issue, the current common
practice is to use regular expressions to mask such information, as
demonstrated in works like Dolma[14], Roots[15], and Wudao[17] among others.

In this work, we use model-based methods for filtering toxic and pornographic
content in addition to keyword and domain blocking, and we mask PII using
regular expressions; see 3.5 for details.

####  2.2.2 Data Quality

Existing pre-training dataset processing pipelines incorporate various methods
aimed at data quality. We categorize these into three main aspects: format
accuracy, data redundancy, and content quality.

Format correctness primarily focuses on whether the data meets the expected
format, such as whether it contains HTML tags, uncommon characters, or whether
punctuation is used correctly. In existing research, this is mainly achieved
through manual rules targeting the format or heuristic rules based on
statistics. For example, clearing HTML tags or JavaScript code, normalizing
special characters in the text, and counting the number and proportion of
specific character sets, etc. Currently, almost all related studies include
this step in their data processing workflow. After referring to the heuristic
rules of Gopher and C4, we added some rules based on a deeper observation of
the data. For detailed information, please refer to 3.3.

“Duplication” refers to the presence of repeated content in the data. Numerous
studies have demonstrated that duplicate data not only wastes computational
resources but can also negatively impact model performance. Current
deduplication methods mainly fall into three categories: URL-based matching,
string-based matching, and fuzzy matching.

URL-based matching methods primarily address duplicate data captured in Common
Crawl, such as those employed by RefinedWeb and Dolma. String-based matching
methods are used to detect duplicate text segments; for instance, RefinedWeb
uses a global suffix tree to find duplicate segments, while Redpajama detects
duplicates by calculating and comparing the hash values of each line in every
document.

Fuzzy matching methods are mainly used to detect duplication across entire
documents, with Locality-Sensitive Hashing (LSH) based methods being the most
widely used. For example, Gopher, RefinedWeb, Slimpajama, Wudao,
ChineseWebText, and Dolma all use Bloom Filters for deduplication operations.
In this study, we chose to use an LSH-based deduplication method, for more
details please refer to 3.4.

Content quality refers to whether the information in the dataset can
effectively enhance the performance of the model. Typically, we intuitively
measure its quality by training and evaluating a model of a certain scale on
the dataset. However, this method requires substantial computational resources
and is difficult to directly correspond with data filtering strategies.

In existing research, the primary approach is to use known high-quality
datasets as positive samples, then train language models (LM) or classifiers
for scoring and filtering. For instance, in Redpajama’s work, they used
Wikipedia and random Common Crawl data as positive and negative samples
respectively, trained a classifier, and used the classification probability of
each document as a measure of its quality. Additionally, the Roots project
trained a Ken-LM model using Wikipedia data, and used the perplexity of each
document as a measure of its quality.

While using high-quality datasets as positive samples reduces additional
investment, it lacks intuitive means for data quality assessment and is
difficult to adjust specifically. To address the two widespread issues in
Common Crawl: Advertising Content and Non-Fluent Content, we adopted a quality
filtering method based on classification models in this study. For more
details, please refer to 3.6.

##  3 Method

Figure 1: Data Processing Pipeline. “Warc” represents the WARC format data
from Common Crawl, “Eng” represents normal English data, “&*#” represents data
with formatting errors, “Dup” represents duplicate data, “Porn” and “Toxic”
represent pornographic and toxic data, respectively. “ADs” and “Bad” represent
advertisements and low-quality data, respectively, while “Good” represents
high-quality data. The red cross indicates data discarded at that stage.

###  3.1 Processing Pipeline

The data processing pipeline adopted in this study is shown in Figure 1.
Initially, we extract text from the WARC format data of Common Crawl, yielding
the Raw Data. Subsequently, heuristic rules are applied to filter the Raw
Data, generating the Clean Data. We then employ a deduplication method based
on Locality-Sensitive Hashing (LSH) to process the Clean Data, resulting in
the Dedup Data. Following this, we use filtering methods based on keywords and
domain lists, as well as harmful content classifiers and obscene content
classifiers based on Bert, to filter the Dedup Data, producing the Safe Data.
Finally, we apply a Bert-based advertisement classifier and fluency classifier
to further filter the Safe Data, obtaining the High-Quality Data.

We arrange the order of the pipeline based on computational cost and iteration
frequency. Steps with lower computational costs are prioritized, while those
with higher iteration frequencies are placed later. For instance, heuristic
rule filtering has a relatively low computational cost and can effectively
filter out a large amount of invalid text, so we place it as the first step
after text extraction. Despite the high computational load of global
deduplication, its low iteration frequency leads us to position it as the
second step. Both safety filtering and quality filtering have high
computational costs, but the threshold for quality filtering may vary
depending on the use case, while the threshold for safety filtering is
relatively fixed. Therefore, we decide to perform safety filtering first,
followed by quality filtering.

Next we will introduce each step’s specific implementation.

###  3.2 Data Extraction

Each Common Crawl dump contains the original content of the crawled web pages,
such as HTML, CSS, JS, and URLs of multimedia resources. To obtain relatively
clean and coherent training corpora, we need to extract text information from
these raw data. Additionally, at this stage, we can also perform language
classification. We employed the trafilatura method[18] for text extraction.
Subsequently, we utilized the pycld2 library to classify the language of the
extracted text. As this study is solely concerned with English data, we
retained only the data marked as “en”, discarding data in other languages.

###  3.3 Heuristic Rule Filtering

Given the prevalence of parsing errors, formatting issues, and non-natural
text in the extracted data, the typical solution is to design heuristic rules
to modify and filter these data. Building on the heuristic rules established
by Gopher[19] and C4[20], we have added some new rules based on our in-depth
observation of the data. Firstly, we implement those rules that would alter
the content of the data:

  1. 1.

Character Normalization: Replace “\xa0” and “\u3000” with standard spaces, and
remove other invisible characters.

  2. 2.

Effective Line Extraction: Find the first and last lines containing
punctuation, retain the content between these two lines, and discard the rest.
If the first line contains more than three words, it is retained.

  3. 3.

Line Content Modification: Remove certain patterns, such as: “cookie policy”,
“use of cookies”, etc.

  4. 4.

Line Content Removal: Delete lines with the following characteristics: only
contain spaces; contain “lorem ipsum”, “javascript”, “{”, etc.

After applying the above rules to truncate and modify the documents, we
execute the following rules for filtering:

  1. 1.

Drop the doc with the most common character not being a letter;

  2. 2.

Drop the doc with the ratio of letters to numbers falling below 0.46;

  3. 3.

Drop the doc with the proportion of the most frequent word in a single
document exceeding 7.5% of the total word count (for texts with a total word
count of 500 or less, this threshold is 30%);

  4. 4.

Drop the doc with the word count outside the range of 50 to 100000;

  5. 5.

Drop the doc with the number of words containing at least one letter
accounting for less than 80% of the total word count;

  6. 6.

Drop the doc with fewer than two stop words;

  7. 7.

Drop the doc with the average word length outside the range of 3 to 10
characters;

  8. 8.

Drop the doc with fewer than three lines, or where the third longest line
contains fewer than 20 characters;

  9. 9.

Apply the n-gram duplicate filter proposed by Gopher [19];

  10. 10.

Drop the doc with the ratio of the total character count to the total word
count exceeding 1.2;

  11. 11.

Drop the doc with only space separator (“\r”, “\n”, “\t”, whitespace etc.);

  12. 12.

Drop the doc with either a sequence of 500 consecutive spaces or 8 consecutive
newline characters;

  13. 13.

Drop the doc with the proportion of newline characters exceeding 25% of the
total character count;

  14. 14.

Drop the doc with special characters resulting from decoding errors;

  15. 15.

Drop the doc with any word exceeding 45 letters in length;

  16. 16.

Drop the doc with any sentence exceeding 56 words in length;

  17. 17.

Drop the doc if the last character is a colon;

  18. 18.

Drop the doc containing image URL or no content other than the URL.

After the document has been intercepted, modified, and filtered according to
the above rules, we obtained the Clean Data.

###  3.4 Deduplication with MinHash-LSH

Common Crawl contains a large amount of duplicate data, including reprinted
articles, templated content, and repeatedly crawled information. This type of
duplicate data can negatively impact the training effectiveness of models. To
eliminate these duplicates, we employed a deduplication algorithm based on
MinHash[21].

Specifically, we utilized the datasketch library[22] for computation, applying
128 hash functions on the 5-gram of the documents to construct signatures, and
setting 0.7 as the deduplication threshold. During the processing, we
prioritized retaining the most recent data, meaning that Common Crawl dumps
with larger numbers were considered first. After the deduplication process
using LSH (Locality-Sensitive Hashing), we obtained the Dedup data.

###  3.5 Safety Filtering

The web data contains a vast amount of unsafe content, including toxic,
pornographic, and personally identifiable information (PII). To prevent pre-
training models from learning undesirable behaviors from these contents, we
initially employed a block domain list, block word list, toxicity classifier,
and pornography classifier to tag and uniformly eliminate such data.
Subsequently, we used regular expressions to mask any potential PII present in
the data.

####  3.5.1 Block Domain/Word List

We created a blocklist of approximately 13 million unsafe domain names, along
with a blocklist of 36,289 unsafe words. We employed domain matching and text
matching methods to label all the data. Considering that unsafe words could
potentially lead to excessive data deletion, we strived to minimize words that
might cause such deletions. By using domain and word filters, we labeled a
total of 7.4% of the data.

To compensate for the limitations of the conservative block word list, we
employed Bert-based Toxicity and Pornography classifiers to filter the data.
The following sections will provide a detailed introduction to the preparation
of training data and the training process for these two models, as well as
report on their performance.

####  3.5.2 Toxicity Classification

To address the potential harmful content on the Internet, we utilized the
training and test sets provided by the Toxic Comment Classification Challenge
on the Kaggle platform[16] to fine-tune the Bert model for toxic data
filtering. This dataset includes labels for six categories: toxic,
severe_toxic, obscene, threat, insult, and identity_hate. Each piece of data
has six 0/1 attribute tags, where 1 indicates that the data possesses the
corresponding category attribute, and 0 indicates the absence thereof. We re-
labeled each piece of data according to the following rule: if any of the six
attributes is 1, it is marked as "toxic"; otherwise, it is marked as "non-
toxic". In this way, we obtained a binary classification dataset, with a ratio
of approximately 1:2.5 between toxic and non-toxic data. During this process,
we did not alter the original division of the training and test sets in the
dataset.

We fine-tuned the bert for sequence classification model on the re-annotated
training set, and ultimately, with a threshold set at 0.5, the model’s
precision reached 81.56%, recall was 87.95%, and the F1 score was 84.63%.
Utilizing this model, we conducted a comprehensive screening of the Dedup Data
dataset, marking all data exceeding a threshold of 0.2 as to-be-discarded.
Approximately 3.9% of the data was marked for discarding.

####  3.5.3 Pornography Classification

Firstly, we referred to the publicly available collection of pornographic
website domains on GitHub at https://github.com/Bon-Appetit/porn-
domains/tree/master, which contains a total of 228,254 domain names of
pornographic websites. From the Dedup Data, we extracted 100,000 data entries
based on this list as samples of pornographic website domains, and inversely
extracted another 100,000 data entries as samples of non-pornographic website
domains. These samples were drawn proportionally from 90 dumps according to
the volume of data in each dump.

In order to obtain a higher quality dataset, we further utilized the
pornography content detection interface of Perspective API to annotate these
data. Data with scores exceeding 0.1 were classified as pornographic content,
while those scoring below 0.1 were considered non-pornographic. Ultimately, we
selected 30,000 pieces of pornographic/non-pornographic content data as the
training set for fine-tuning the BERT model. From this, we extracted 4,000
pieces of pornographic/non-pornographic content data, which after manual
annotation, served as the test set.

In our manually annotated test set, by setting the threshold to 0.3, the
optimal model we trained exhibited a precision of 0.79, a recall of 0.991668,
and an accuracy of 0.87. We used this model to filter the entire Dedup Data,
marking all data exceeding a threshold of 0.2 as to be discarded.
Approximately 7.3% of the data was marked in this way.

####  3.5.4 Personal Identifiable Information Masking

To safeguard user privacy, we employed regular expressions to mask potential
Personally Identifiable Information (PII) present in the data. The types of
PII that we masked include: IP addresses, passport numbers, GPS location data,
personal email addresses, personal phone numbers, identification card numbers,
bank account numbers, and physical addresses. For each type of PII, we used
corresponding regular expressions for matching, recorded their positions in
the text, and then replaced them with “[[pii_type]]”.

In summary, we employed four modules for data safety filtering, including:
block domain list, block word list, toxicity classifier, and pornography
classifier. Additionally, we utilized regular expressions to mask any
potential personally identifiable information (PII). Ultimately, all data
marked for discard were filtered out, with the remaining data considered as
Safe Data.

###  3.6 Data Quality Filtering

After completing the aforementioned filtering steps, we found that the data
still contained a large amount of promotional information and content with
poor language fluency. The promotional information mainly manifested as
enticing titles and content, which contributed minimally to the training
effectiveness of the model. Content with poor language fluency included format
errors, machine-generated random text, aggregation of irrelevant content, and
content with extremely low information content. To eliminate these elements,
we employed a Bert-based classifier for promotional content and language
fluency for further data filtering.

####  3.6.1 Advertisement Classification

In dealing with advertisement attributes, we established a detailed manual
annotation guide, requiring annotators to categorize the data into: a) full-
text advertisements; b) partial advertisements; c) non-advertisement full
text; d) invalid data. Here, "invalid data" represents data that cannot be
effectively judged, and this portion of data will be excluded. For the
remaining data, we tagged "full-text advertisements" and "partial
advertisements" with advertisement content labels, and labeled "non-
advertisement full text" with non-advertisement content labels, thereby
constructing a binary classification dataset.

####  3.6.2 Fluency Classification

For the attribute of fluency, we constructed a more comprehensive manual
annotation questionnaire, requiring annotators to evaluate from the following
four data usability dimensions:

  1. 1.

Consistency, that is, checking whether the data revolves around the same theme
throughout, to avoid topic confusion caused by extraction errors.

  2. 2.

Noisy, that is, judging whether there are a large number of meaningless
characters in the data, such as garbled text, special symbols, excessive
spaces, etc.

  3. 3.

Information, that is, assessing whether the data contains valuable
information, for example, whether the information is incomplete due to the
lack of image or video data.

  4. 4.

Grammar, that is, checking whether there are grammatical errors in the data,
such as spelling mistakes, structural errors, improper use of punctuation
marks, etc.

We normalize the weighted sum of scores across all dimensions for each data
point. Data with a score greater than 0.5 is labeled as having good fluency,
while data with a score less than 0.5 is marked as having poor fluency,
thereby forming a binary classification dataset.

We fine-tuned the BERT for sequence classification model using both of the
aforementioned datasets. Subsequently, we utilized these two models to compute
scores across the entirety of the Safe Data, and from this, we selected 1T
tokens worth of data as High-Quality Data.

###  3.7 Data Quality Evaluation

After processing with the aforementioned data processing pipeline, we obtain a
High-Quality Data. The design of rules and selection of filtering thresholds
are manually evaluated and adjusted to achieve the desired effect. To ensure
data quality, we have implemented a comprehensive data quality evaluation and
feedback mechanism in our engineering practice. It is widely recognized that
assessing the quality of the Common Crawl dataset presents significant
challenges, primarily due to its large volume, diverse problem types, vague
quality definitions, high degree of subjectivity, and scattered features.

Informed by industry standards and best practices, we assess the dataset
employing seven primary quality evaluation metrics, each encompassing multiple
secondary metrics, as delineated below:

  * •

Quality Signal: Effectiveness

    * –

Ratio of data items of null data values in the dataset.

    * –

Ratio of data items containing garbled characters in the dataset.

  * •

Quality Signal: Completeness

    * –

Ratio of data items in the dataset that contain incomplete content (such as
ending with a colon).

    * –

Ratio of data items in the dataset where paired content does not appear in
pairs (such as parentheses).

  * •

Quality Signal: Understandability

    * –

Ratio of data items in the dataset that contain errors in formatting (such as
formulas, tables, markdown, etc.).

    * –

Ratio of data items in the dataset that contain special characters (such as
high proportion of spaces, line breaks, #, invisible characters).

  * •

Quality Signal: Similarity

    * –

Ratio of data items in the dataset that contain duplicate content.

    * –

Ratio of duplicate data items in the dataset.

  * •

Quality Signal: Fluency

    * –

Ratio of data items in the dataset that contain large blocks of text without
punctuation.

    * –

Ratio of data items in the dataset that contain anti-scraping content.

  * •

Quality Signal: Relevance

    * –

Ratio of data items in the dataset that contain leading or trailing off-
topic/unrelated content (such as advertisements, copyright notices, references
from web pages).

  * •

Quality Signal: Security

    * –

Ratio of data items in the dataset that contain pornography content.

    * –

Ratio of data items in the dataset that contain toxic content.

In order to conduct a comprehensive, accurate, and efficient data quality
evaluation, we conceptualize the process into three distinct stages: metric
identification and definition, computation methods for these metrics, and
determination of metric thresholds. To address these stages, we employ three
disparate quality assessment methodologies: 1) Human Evaluation Method; 2)
Quality Signal Evaluation Method; 3) Model Evaluation Method.

Human Evaluation Method: Initially, given the large volume of Common Crawl
data, ambiguous quality definitions, high subjectivity, and dispersed
features, we extract a certain amount of data for multiple evaluators to
identify ’bad samples’. Quality indicators are then abstracted from the
characteristics of these identified bad samples.

Subsequently, in order to enhance the likelihood of encountering bad samples
during sampling, we randomly select a specific quantity (1 million) of data
points to calculate the loss via LLMs. Based on the distribution of high,
medium, and low loss values, we designate a certain number of data points
(2000 each from high and low intervals, 1000 from the middle interval) for
further human evaluation.

Quality Signal Evaluation Method: To accurately and efficiently gauge the
extent of data quality issues within the entire Common Crawl dataset, we have
devised machine rules that are independent of the data cleaning stage. These
rules automatically evaluate the proportion of poor-quality samples in the
entire dataset based on the quality indicators identified through the Human
Evaluation Method. We define this proportion as $X=\frac{A}{B}$, where $A$
represents the number of poor-quality samples, and $B$ denotes the total
number of samples in the dataset.

However, not all quality indicators can yield an accurate proportion of the
entire dataset via machine rules, especially for complex issues such as texts
containing leading or trailing promotional information. For these types of
quality indicators, we estimate the proportion of poor-quality samples as
$\tilde{X}=\frac{A^{\prime}}{B^{\prime}}$, where $A^{\prime}$ is the number of
poor-quality samples identified by Human Evaluation, and $B^{\prime}$ is the
total number of sampled instances.

Model Evaluation Method: We have obtained the proportions of quality
indicators through both human and machine methods, which serve to measure the
severity of issues. However, determining what proportion is deemed serious or
not requires an independent threshold for each quality indicator. We initially
set these thresholds based on our experience, then update the signals and
thresholds in response to feedback from downstream Model Evaluation. Firstly,
we will use the 1B and 3B experimental models trained in Section 4.4 for
evaluation. In the Model Evaluation phase, we have designed a test dataset
comprising approximately 600 prompts, spanning over 20 types in both Chinese
and English. We then conduct a comprehensive assessment of the model-generated
content using both subjective evaluation and Quality Signal Evaluation. This
process allows us to explore the impact of data quality issues on model
performance, thereby facilitating updates to the quality indicators and
thresholds within our quality standards.

The current Data Quality Evaluation employs three distinct methods. Should all
evaluations satisfy the quality requirement threshold, the dataset is deemed
to be of high quality. Conversely, if this standard is not met, we proceed to
optimize the data process and re-evaluate until the data quality threshold
requirements are fulfilled. The comprehensive workflow of the Data Quality
Evaluation process is depicted in Figure 2.

Figure 2: Data Quality Evaluation Workflow

##  4 Result

In order to more comprehensively illustrate the production process and data
quality of WanJuan-CC, we have divided our work and results into four
sections: document retention rate, distribution of statistical indicators
across documents, data toxicity detection, and model evaluation.

Initially, we enumerated the number of documents at each stage within the data
processing pipeline, computed both the relative removal rate and absolute
retention rate, and depicted these in Figure 3. Subsequently, we calculated
the distribution of various metrics within the dataset and represented these
in Figure 5. Following this, we employed the Perspective API to perform a
safety assessment on the dataset, with AUC values presented in Table 2.
Lastly, we utilized an autoregressive decoder-only Transformer model to
conduct experiments on diverse validation sets, with the experimental outcomes
displayed in Table 3.

###  4.1 Removal Rate for Different Stages

We quantified the document count at each phase of our data processing
pipeline, encompassing the Common Crawl web pages, Raw data, Clean data,
Deduplicated data, Safe data, and High-Quality data. We computed both the
elimination rate for each phase in relation to its predecessor and the
retention rate with respect to the total number of Common Crawl web pages.
These metrics are graphically represented in Figure 3. It should be noted that
a logarithmic scale was employed for the axes.

Figure 3: Documents Retention Rate and Removal Rate for Different Stages

Initially, we extracted pertinent text from the HTML data embedded in the WARC
format files of Common Crawl, identifying the English content therein. This
process retained 52.6% of the total documents. Following this, a heuristic
rule-based data cleansing step was implemented, which filtered out
approximately 37.8% of the data. This step primarily served to eliminate data
that was clearly not part of standard text or exhibited severe formatting
errors.

In the deduplication phase, our elimination rate markedly diverged from that
of other studies. Our LSH deduplication approach expunged 90.2% of the data.
Conversely, RefinedWeb’s combined strategy of fuzzy deduplication and exact
deduplication discarded a total of 50% of the data. The presence of duplicates
in pre-training data can substantially impair model performance, hence our
adoption of a more rigorous deduplication plan.

In the process of safe content filtering, a combination of sensitive word
detection, domain blocking, pornographic classification, and toxicity
classification eliminated 14.3% of the data. Subsequently, during the quality
filtering phase, we discarded approximately 50% of the remaining data. The
final high-quality dataset constituted 1.48% of the original data volume.

We also calculated the proportion of high-quality data originating from
different years of Common Crawl dump, as depicted in Figure 4. As seen from
the figure, the number of documents, bytes, and tokens in high-quality data
have all shown an increasing trend over time. This could be attributed to two
main factors. On one hand, it is due to the continuous growth of the data
volume in Common Crawl. On the other hand, it is related to our retention
strategy implemented during the LSH deduplication step, which prioritizes the
preservation of the most recent data.

Figure 4: Proportion of High-Quality Data from Different Common Crawl Dump
Years

###  4.2 Dataset Statistics

In order to streamline the assessment of the dataset’s quality, we drew upon
several data quality metrics from Redpajama-v2 and applied them to WanJuan-CC.
We quantified various aspects of the dataset including document length, line
count, token length, percentage of non-alphabetic characters, proportion of
unique words, average word length, sentence count, stop-word ratio, and
symbol-to-word ratio. The distribution for each metric is visualized in Figure
5.

(a) Content Length

(b) Line Number

(c) Token Length

(d) Non-alpha Fraction

(e) Unique Words Fraction

(f) Mean Word Length

(g) Sentence Number

(h) Stop Word Fraction

(i) Symbol to Word Ratio

Figure 5: Percentage Statistics for Different Metrics on WanJuan-CC. To
illustrate the main distribution area, the statistical range of some metrics
has been truncated due to the presence of long-tail distributions.

By quantifying and illustrating the distribution of these metrics on WanJuan-
CC, we enable users to gain a comprehensive understanding of the data’s
various characteristics. This empowers users to refine the data further in
accordance with their specific requirements.

###  4.3 Data Safety Metrics

We independently drew a random sample of 100K data points from each of the
following datasets: our own, Redpajama, and RefinedWeb. We utilized the
Perspective API to compute scores across seven dimensions. Let’s denote the
test dataset of any given dataset as $D=\\{d_{j}|1\leq j\leq N\\}$, where for
a specific dimension $m$, the score of data point $d_{j}$ is represented by
$c_{m}(d_{j})$. Given a threshold $t_{i}$, the percentage of data exceeding
$t_{i}$ is calculated as
$p_{m}(t_{i})=100\times\frac{|\\{d_{j}|c_{m}(d_{j})>t_{i}\\}|}{N}$.

By generating the “$t$-$p_{m}(t)$” curves and computing the AUC values for
each curve using Equation 1, we can intuitively assess the level of unsafety
across different dimensions. Detailed results are presented in Table 2. The
table reveals that our dataset’s AUC values are lower than those of both
Redpajama and RefinedWeb across all dimensions, suggesting a higher degree of
safety in our dataset.

$AUC_{m}=\sum_{i=1}^{L-1}\frac{p_{m}(t_{i})+p_{m}(t_{i+1})}{2}\times(t_{i+1}-t_{i})$
(1)

Table 2: Area Under Curve of Toxicity Metrics by Perspective API for Different Datasets | WanJuan-CC | Redpajama | RefinedWeb
---|---|---|---
Toxicity | 3.45 | 4.02 | 4.48
Severe Toxicity | 0.21 | 0.31 | 0.46
Identity Attack | 0.83 | 1.35 | 1.14
Insult | 1.83 | 2.16 | 2.39
Profanity | 1.88 | 2.30 | 2.88
Threat | 1.02 | 1.31 | 1.19
Sexually Explicit | 1.28 | 2.19 | 2.42

###  4.4 Data Utility Metrics

To further ascertain the quality of the data, we utilized the identical
autoregressive decoder-only Transformer model for training from scratch with
both WanJuan-CC and RefinedWeb datasets. Experiments were conducted at 1B and
3B parameter levels, where the 1B model processed 100B tokens and the 3B model
processed 200B tokens.

We employed the average PPL on the validation dataset as the evaluation metric
for the 1B model. This approach is due to the difficulty in observing changes
in downstream task metrics with models of smaller parameter sizes, thus making
the use of PPL metric on the validation set a more effective measure of
training outcomes for smaller models. This perspective aligns with that
expressed in Paper D4 [23]. We incorporated three subsets from Pile [24] \-
“pile-books3”, “pile-openwebtext2”, “pile-wikipedia-en” - and one subset from
Tiny-stroys [25] \- “tiny-storys” - as our four validation sets. The
experimental results are depicted in Table 3. As can be discerned from the
table, the PPL of WanJuan-CC on pile-books3, pile-openwebtext2, and pile-
wikipedia-en is marginally lower than that of RefinedWeb, while its PPL on
tiny-storys is markedly lower. These findings suggest that the data quality of
WanJuan-CC slightly surpasses that of RefinedWeb, particularly on validation
sets like tiny-storys that demand higher language fluency.

Table 3: Perplexity of Different Models on Different Validation Sets Eval Datasets | WanJuan-CC | RefinedWeb
---|---|---
pile-books3 | 12.34 | 12.56
pile-openwebtext2 | 11.86 | 11.96
pile-wikipedia-en | 8.68 | 8.81
tiny-storys | 5.78 | 6.15

For models with a larger number of parameters, the accuracy of downstream
tasks can serve as a more effective indicator of data quality. We have chosen
to evaluate three major categories of downstream tasks: English text
completion (LAMBADA, StoryCloze), English general capability (SuperGLUE), and
English commonsense question answering (HellaSwag, PIQA, WinoGrande),
amounting to six downstream tasks in total. The results of these experiments
are displayed in Table 4.

Table 4: Accuracy of Different Models on Different Downstream Tasks Task | RefinedWeb | WanJuan-CC
---|---|---
LAMBADA | 58.65 | 62.22
StoryCloze | 70.44 | 71.83
SuperGLUE | 41.86 | 44.46
HellaSwag | 60.71 | 58.12
PIQA | 74.92 | 74.21
WinoGrande | 55.33 | 57.54

Experimental results demonstrate that WanJuan-CC significantly enhances
performance in English text completion and general English proficiency, albeit
with a decline in HellaSwag and a minor decrease on PIQA for English
commonsense question answering. Conversely, there is a substantial improvement
on WinoGrande. In summary, WanJuan-CC surpasses RefinedWeb in terms of
performance on downstream tasks.

##  5 Conclusion

In this paper, we presented WanJuan-CC, a safe and high-quality open-sourced
English webtext dataset derived from Common Crawl data. We designed and
implemented a comprehensive process for handling Common Crawl data, which
includes data extraction, heuristic rule filtering, fuzzy deduplication,
content safety filtering, and data quality filtering. This process allowed us
to extract approximately 68 billion original English documents from 90 dumps
since 2013, yielding 2.22T Tokens of safe data and 1.0T Tokens of high-quality
data.The additional steps of advertisement and fluency classification on
dataset were key to ensuring high data quality.

The safety of our dataset was validated through rigorous testing. We used the
Perspective API to assess the level of unsafety across different dimensions,
and our dataset’s AUC values were found to be lower than those of other
datasets across all dimensions, indicating a higher degree of safety in our
dataset. Furthermore, the utility of our dataset was demonstrated through
perplexity (PPL) on validation sets and accuracy on downstream tasks. The PPL
of WanJuan-CC on various validation sets was found to be competitive,
particularly on sets like tiny-storys that demand higher language fluency. For
larger models, the accuracy of downstream tasks served as an effective
indicator of data quality. Experimental results showed that WanJuan-CC
significantly enhances performance in English text completion and general
English proficiency tasks.

In conclusion, WanJuan-CC represents a significant contribution to the field
of large-scale language model training. It provides a safe, high-quality, and
open-source dataset for researchers and practitioners. Future work could focus
on further refining the data processing pipeline to improve data quality and
safety, and exploring the application of this dataset in more diverse NLP
tasks.

## References

  * [1] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
  * [2] Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilic, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong, Daniel van Strien, David Ifeoluwa Adelani, and et al.  BLOOM: A 176b-parameter open-access multilingual language model.  CoRR, abs/2211.05100, 2022.
  * [3] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.  Llama: Open and efficient foundation language models.  CoRR, abs/2302.13971, 2023.
  * [4] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel.  Palm: Scaling language modeling with pathways.  J. Mach. Learn. Res., 24:240:1–240:113, 2023.
  * [5] Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay.  The refinedweb dataset for falcon LLM: outperforming curated corpora with web data, and web data only.  CoRR, abs/2306.01116, 2023.
  * [6] OpenAI.  GPT-4 technical report.  CoRR, abs/2303.08774, 2023.
  * [7] InternLM Team.  Internlm: A multilingual language model with progressively enhanced capabilities.  https://github.com/InternLM/InternLM, 2023.
  * [8] Xiyuan Zhang, Xinyue Zhang, and Ying Yu.  Chatglm-6b fine-tuning for cultural and creative products advertising words.  In 2023 International Conference on Culture-Oriented Science and Technology (CoST), pages 291–295. IEEE, 2023.
  * [9] Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, Fan Yang, Fei Deng, Feng Wang, Feng Liu, Guangwei Ai, Guosheng Dong, Haizhou Zhao, Hang Xu, Haoze Sun, Hongda Zhang, Hui Liu, Jiaming Ji, Jian Xie, Juntao Dai, Kun Fang, Lei Su, Liang Song, Lifeng Liu, Liyun Ru, Luyao Ma, Mang Wang, Mickel Liu, MingAn Lin, Nuolan Nie, Peidong Guo, Ruiyang Sun, Tao Zhang, Tianpeng Li, Tianyu Li, Wei Cheng, Weipeng Chen, Xiangrong Zeng, Xiaochuan Wang, Xiaoxi Chen, Xin Men, Xin Yu, Xuehai Pan, Yanjun Shen, Yiding Wang, Yiyu Li, Youxin Jiang, Yuchen Gao, Yupeng Zhang, Zenan Zhou, and Zhiying Wu.  Baichuan 2: Open large-scale language models.  CoRR, abs/2309.10305, 2023.
  * [10] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre.  Training compute-optimal large language models.  CoRR, abs/2203.15556, 2022.
  * [11] Common crawl - open repository of web crawl data, 2023.
  * [12] Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary.  Asynchronous pipeline for processing huge corpora on medium to low resource infrastructures.  In 7th Workshop on the Challenges in the Management of Large Corpora (CMLC-7). Leibniz-Institut für Deutsche Sprache, 2019.
  * [13] Together Computer.  Redpajama: an open dataset for training large language models, October 2023.
  * [14] Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Khyathi Chandu, Jennifer Dumas, Li Lucy, Xinxi Lyu, et al.  Dolma: An open corpus of 3 trillion tokens for language model pretraining research.  Allen Institute for AI, Tech. Rep, 2023.
  * [15] Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, Jörg Frohberg, Mario Sasko, Quentin Lhoest, Angelina McMillan-Major, Gérard Dupont, Stella Biderman, Anna Rogers, Loubna Ben Allal, Francesco De Toni, Giada Pistilli, Olivier Nguyen, Somaieh Nikpoor, Maraim Masoud, Pierre Colombo, Javier de la Rosa, Paulo Villegas, Tristan Thrush, Shayne Longpre, Sebastian Nagel, Leon Weber, Manuel Muñoz, Jian Zhu, Daniel van Strien, Zaid Alyafeai, Khalid Almubarak, Minh Chien Vu, Itziar Gonzalez-Dios, Aitor Soroa, Kyle Lo, Manan Dey, Pedro Ortiz Suarez, Aaron Gokaslan, Shamik Bose, David Ifeoluwa Adelani, Long Phan, Hieu Tran, Ian Yu, Suhas Pai, Jenny Chim, Violette Lepercq, Suzana Ilic, Margaret Mitchell, Alexandra Sasha Luccioni, and Yacine Jernite.  The bigscience ROOTS corpus: A 1.6tb composite multilingual dataset.  In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022, 2022.
  * [16] cjadams, Sorensen Jeffrey, Elliott Julia, Dixon Lucas, McDonald Mark, nithum, and Cukierski Will.  Toxic comment classification challenge, 2017.
  * [17] Sha Yuan, Hanyu Zhao, Zhengxiao Du, Ming Ding, Xiao Liu, Yukuo Cen, Xu Zou, Zhilin Yang, and Jie Tang.  Wudaocorpora: A super large-scale chinese corpora for pre-training language models.  AI Open, 2:65–68, 2021.
  * [18] Adrien Barbaresi.  Trafilatura: A web scraping library and command-line tool for text discovery and extraction.  In Heng Ji, Jong C. Park, and Rui Xia, editors, Proceedings of the Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL 2021 - System Demonstrations, Online, August 1-6, 2021, pages 122–131. Association for Computational Linguistics, 2021.
  * [19] Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, H. Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant M. Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d’Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew J. Johnson, Blake A. Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Edward Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving.  Scaling language models: Methods, analysis & insights from training gopher.  CoRR, abs/2112.11446, 2021.
  * [20] Jesse Dodge, Maarten Sap, Ana Marasovic, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner.  Documenting large webtext corpora: A case study on the colossal clean crawled corpus.  In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 1286–1305. Association for Computational Linguistics, 2021.
  * [21] Andrei Z. Broder.  On the resemblance and containment of documents.  In Bruno Carpentieri, Alfredo De Santis, Ugo Vaccaro, and James A. Storer, editors, Compression and Complexity of SEQUENCES 1997, Positano, Amalfitan Coast, Salerno, Italy, June 11-13, 1997, Proceedings, pages 21–29. IEEE, 1997.
  * [22] Eric Zhu and Vadim Markovtsev.  ekzhu/datasketch: First stable release, feb 2017.
  * [23] Kushal Tirumala, Daniel Simig, Armen Aghajanyan, and Ari S. Morcos.  D4: improving LLM pretraining via document de-duplication and diversification.  CoRR, abs/2308.12284, 2023.
  * [24] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy.  The pile: An 800gb dataset of diverse text for language modeling.  CoRR, abs/2101.00027, 2021.
  * [25] Ronen Eldan and Yuanzhi Li.  Tinystories: How small can language models be and still speak coherent english?  CoRR, abs/2305.07759, 2023.