# A Hybrid System for Learning Classical Data in Quantum States

Samuel A Stein12, Ryan L’Abbate1, Wenrui Mu 1, Yue Liu1, Betis Baheri3,
Ying Mao1, Guan Qiang3, Ang Li2 and Bo Fang2
1 Department of Computer and Information Science, Fordham University,
Email: {sstein17, rlabbate, wmu2, yliu720, and<EMAIL_ADDRESS>2 Pacific
Northwest National Laboratory, 2 Email: {samuel.stein, ang.li,
<EMAIL_ADDRESS>3Department of Computer Science, Kent State University, 3
Email: {bbaheri<EMAIL_ADDRESS>
###### Abstract

Deep neural network powered artificial intelligence has rapidly changed our
daily life with various applications. However, as one of the essential steps
of deep neural networks, training a heavily-weighted network requires a
tremendous amount of computing resources. Especially in the post Moore’s Law
era, the limit of semiconductor fabrication technology has restricted the
development of learning algorithms to cope with the increasing high intensity
training data. Meanwhile, quantum computing has demonstrated its significant
potential in terms of speeding up the traditionally compute-intensive
workloads. For example, Google illustrated quantum supremacy by completing a
sampling calculation task in 200 seconds, which is otherwise impracticable on
the world’s largest supercomputers. To this end, quantum-based learning has
become an area of interest, with the potential of a quantum speedup. In this
paper, we propose GenQu, a hybrid and general-purpose quantum framework for
learning classical data through quantum states. We evaluate GenQu with real
datasets and conduct experiments on both simulations and real quantum computer
IBM-Q. Our evaluation demonstrates that, compared with classical solutions,
the proposed models running on GenQu framework achieve similar accuracy with a
much smaller number of qubits, while significantly reducing the parameter size
by up to 95.86% and converging speedup by 33.33% faster.

##  I Introduction

In the past decade, machine learning and artificial intelligence-powered
applications have dramatically changed our daily life. Many novel algorithms
and models achieve widespread practical successes in a variety of domains such
as autonomous cars, healthcare, manufacturing, etc. Despite the wide adoption
of ML models, training the machine learning models such as DNNs requires a
tremendous amount of computing resources to tune millions of hyper-parameters.
Especially in the post Moore’s Law era, the limit of semiconductor fabrication
technology cannot satisfy the the rapidly increasing data volume needed for
training, which restricts the development of this field [1].

Encouraged by the recent demonstration of quantum supremacy [2], researchers
are searching for a transition from the classical learning to the quantum
learning, with the promise of providing a quantum speedup over the classical
learning. The current state of quantum-based learning inspires alternative
architectures to classical learning‘s sub-fields, such as Deep Learning (DL)
or Support Vector Machine (SVM) [3, 4, 5, 6, 7], where the quantum algorithm
provides improvements over their classical counterparts. For example, there
are quite a number of adoptions of quantum learning algorithms in domains of
expectation maximization solving (QEM) [8] that speeds up the kernel methods
to sub-linear time [9], Quantum-SVM [10], and NLP [11]. The use of quantum
systems to train deep learning models is rather developed with a multitude of
approaches for creating and mimicking aspects of classical deep learning
systems [12, 4, 13, 8]. For example, Tensorflow Quantum [14] provides a
quantum ML/DL library for rapid prototyping of hybrid quantum-classical
models. However, the applications cannot be executed directly on near-term
quantum devices, such as IBM-Q platform. These quantum systems, in the
literature, face several challenges: (i) such systems are held back by the low
qubit count of current quantum computers. (ii) learning in a quantum computer
becomes even more difficult due to the lack of efficient classical-to-quantum
data encoding methodology [15, 16]. (iii) most of the existing studies are
based on purely theoretical analysis or simulations, lacking practical
usability on near-term quantum devices [17].

More importantly, the above challenges would persist even as the number of
qubits supported in quantum machines increases significantly: when the number
of qubits in the quantum system increases, the computational complexity grows
exponentially [18], which quickly leads to tasks that become completely
infeasible for simulation and near-term quantum computers. Therefore,
discovering the representative power of qubits in quantum-based learning
systems is extremely important, as not only does it allow near-term devices to
tackle more complex learning problems, but it also eases the complexity of the
quantum state exponentially. However, current research on the topic of low-
qubit counts of current quantum machines is rather sparse Within this domain,
the learning potential of qubits are under-investigated.

In this paper, we propose GenQu, a general-purpose quantum-classic hybrid
framework for learning classical data in quantum states. We demonstrate the
power of qubits in machine learning by approaching the encoding of data onto a
single qubit and accomplish tasks that are impossible for comparative data
streams on classical machines, which addressing the challenges (i) and (ii).
Enabled by GenQU, we develop a deep neural network architecture for
classification problems with only 2 qubits, and a quantum generative
architecture for learning distributions with only 1 qubit, and, additionally,
We evaluate GenQU with intensive experiments on both IBM-Q real quantum
computers and simulators (addressing the challenge (iii)). Our major
contributions include the following:

  * •

We propose GenQu, a hybrid and general-purpose quantum framework that works
with near-term quantum computers and has the potential to fit in various
learning models with a low-qubit count.

  * •

We propose three different quantum-based learning models based on GenQu to
demonstrate the potential of learning data in quantum state.

  * •

We show through experiments on both simulators and IBM-Q real quantum
computers that models in GenQu are able to reduce parameters by up to 95.86%
but still achieves similar accuracy in classification with Principal Component
Analysis (PCA)[19] MNIST dataset, and converge up to 33.33% faster than
traditional neural networks.

##  II Preliminaries

###  II-A The Quantum Bit (Qubit)

Quantum computers operate on a fundamentally different set of physics when
compared to classical computers. Classical computers operate on binary digits
(bits), represented by a 1 or a 0. Quantum computers however, operate on
quantum bits (qubits). Qubits can represent a 1 or a 0, or a probabilistic
mixture of both 1 and 0 simultaneously, namely superposition. When discussing
a quantum framework, we make use of the $\langle bra|$ and $|ket\rangle$
notation, where a $\langle bra|$ indicates a horizontal state vector ($1\times
n$) and $|ket\rangle$ indicates a vertical state vector ($n\times 1$). A qubit
represented as a probabilistic combination of both a $|1\rangle$ and
$|0\rangle$. and therefore is described as a linear combination between of
$|0\rangle$ and $|1\rangle$. This combination is described in Equation 1.

$|\Psi\rangle=\alpha|0\rangle+\beta|1\rangle\text{ ,
}|\Psi\rangle=\left[\begin{array}[]{l}\alpha\\\ \beta\end{array}\right]\text{
, }$ (1)

In Equation 1, the state is described as a combination of both
$|0\rangle=\left[\begin{smallmatrix}1\\\ 0\end{smallmatrix}\right]$ and
$|1\rangle=\left[\begin{smallmatrix}0\\\ 1\end{smallmatrix}\right]$. The
values of $\alpha$ and $\beta$ are the probability coefficients and what
encode information regarding this qubit’s state. Although qubits can exist in
both $|1\rangle$ and $|0\rangle$ at the same time, when they are measured for
a definite output, they collapse to one of two possible values, where in the
case above those values are $|0\rangle$ or $|1\rangle$. The coefficients,
$\alpha$ and $\beta$, indicate the square root of the probability that the
qubit measures as a $|1\rangle$ or a $|0\rangle$. The definite states we are
measuring the qubit against are based on how we measure the qubit, measuring
as one of two possible measurements. These two possible measurements are two
orthogonal eigen-vectors, and can be in any 3-Dimensional direction. This is
best visualized and understood by the Bloch Sphere representation of a qubit,
as illustrated in Figure 1.

Figure 1: Bloch Sphere

A qubit can be represented by the unit Bloch Sphere visualized in Figure 1. In
the case of $|0\rangle$ and $|1\rangle$, we are measuring across the Z axis.

###  II-B Quantum Data Manipulation

Qubits are manipulated through quantum gates, which in turn manipulates the
overall quantum state. These gates can allow for manipulation over the state
vector, which describes the state of any number of qubits. We introduce the
few gates that we make use of in this paper in Equations 3, 4, 5, and 6.

$R_{Y}(\theta)=\left[\begin{array}[]{cc}\cos\left(\frac{\theta}{2}\right)&-\sin\left(\frac{\theta}{2}\right)\\\
\sin\left(\frac{\theta}{2}\right)&\cos\left(\frac{\theta}{2}\right)\end{array}\right]$
(2)

$R_{Z}(\theta)=\left[\begin{array}[]{cc}e^{\frac{-i\theta}{2}}&0\\\
0&e^{\frac{-i\theta}{2}}\end{array}\right]$ (3)

$\centering CR_{Y}(\theta)=\left[\begin{array}[]{cccc}1&0&0&0\\\ 0&1&0&0\\\
0&0&\cos(\frac{\theta}{2})&-\sin(\frac{\theta}{2})\\\
0&0&\sin(\frac{\theta}{2})&\cos(\frac{\theta}{2})\end{array}\right]\@add@centering$
(4)

$\centering CR_{Z}(\theta)=\left[\begin{array}[]{cccc}1&0&0&0\\\ 0&1&0&0\\\
0&0&e^{\frac{i\theta}{2}}&0\\\
0&0&0&e^{\frac{i\theta}{2}}\end{array}\right]\@add@centering$ (5)

The gates above accomplish specific state manipulation. Equations 3 and 4
represent single qubit rotations around the Y axis and Z axis respectively.
These two gates allow for a single qubit to be manipulated to any position on
a Bloch sphere’s surface, from any starting point on aforementioned sphere.
Equations 5 and 6 entangle two qubits using controlled qubit rotations around
the Y axis and Z axis respectively. Controlled rotations allow for a qubit’s
state to be manipulated based on whether a control qubit measures as a
$|1\rangle$, therefore entangling the target and control qubits’ states. These
two styles of gates, single qubit rotations and controlled qubit rotations,
allow for complete manipulation of quantum states, be it 1 or more qubits.

Figure 2: Gate Deep Learning Gate Design

###  II-C Quantum Deep Learning

Quantum Deep Learning is a relatively new approach to Quantum Machine Learning
that takes quantum circuits and applies variational quantum algorithm
techniques and learning methods similar to those of classical neural networks
[13, 20, 4]. In traditional deep learning, layers are often used, where a
layer is some large transformation function that takes in a set of inputs, and
transforms them to a set of outputs. These functions are connected and
typically trained through the use of back propagation [21, 13]. Similar to how
classical deep learning works, the way this data flows through time is up to
the practitioner, who chooses and designs their network according to their
needs. Quantum deep learning is approached through the use of layering gates
sequentially. For this paper, our layers are comprised of the gates in
Equations 3 through 6. Similar to how deep learning is parameterized by
connection weights, these gates are parameterized through rotations
($\theta$). At the end of the quantum circuit, a loss function is described,
and the quantum networks parameters $\theta$ are updated iteratively such that
the circuit’s loss is optimized [4, 22].

##  III GenQu Framework and learning models

###  III-A GenQu Framework

Figure 3: GenQu: A Hybrid Framework

Our proposed GenQu framework is illustrated in Figure 3. Before any operation
of the framework is performed, the data must be transformed from classical to
quantum states. This is done by transforming classical data into applicable
quantum rotations, and is described under section Section III-B. Following
this transformation, the rotations are loaded onto a quantum computer. The
quantum circuit preparation section is where the circuit relating to a
specific machine learning algorithm is designed. For example, this is where a
deep neural network or a convolutional neural networks architecture would be
set up, initialized, and prepared. This circuit is loaded onto the quantum
computer after the quantum data loading section.

Once the circuit is set up, it can be induced. Inducing the quantum circuit
results in the quantum state transformation of the input data over the quantum
machine learning model. From here, if the output of the model was a quantum
state, one could end here and feed it to another quantum algorithm. However,
in the case of updating learn-able parameters, the relevant qubits need to be
measured. We feed the qubits measurements to a loss analysis section, where we
update our parameters accordingly. Once the parameters have been updated, we
repeat this process of circuit loading, circuit inducing, and measurement,
updating parameters until a desired loss of the network is attained or a
predefined number of epochs have achieved.

###  III-B Data Qubitization

Prior to discussing our methods of illustrating the learning power of qubits,
we introduce our approach to encoding classical data into quantum states. We
encode two dimensions of data per qubit, by the simple two step process
outlined in Equations 6 and 7

$x_{1}\xrightarrow[|\phi\rangle]{\text{Encoded
onto}}=RY(2sin^{-1}(\sqrt{x_{1}}))$ (6)
$x_{2}\xrightarrow[|\phi\rangle]{\text{Encoded
onto}}=RZ(2sin^{-1}(\sqrt{x_{2}}))$ (7)

The value of $x_{1}$ is encoded along the Z-axis, followed by $x_{2}$ being
encoded along the Y-axis. For these rotations to work, the data along each
dimension must be normalized to be in the range of (0,1). This reduction in
qubit count is pertinent for the case of quantum machine learning as the state
space vector of a quantum system is of tensor rank $2^{n}$ values, and
therefore halving the qubit count provides a $2^{\frac{n}{2}}$ reduction in
state space.

###  III-C Single Qubit Kernelized Classification

When tackling a classification task on a quantum system, we want to encode our
data such that the probability of measuring a $|1\rangle$ is comparative with
the probability of classifying a data point as $Class$=1. Therefore, in the
case of classical data sets, we can wrap 2 dimensions of data around a qubit
such that we maximize the ability of the qubit to classify the data. In the
case of the circles data set, a data set comprised of points non-linearly
separable, we can wrap the qubit with data points such that the rotation
around the Z axis is correlated to the distance from the circle center. This
is visualized in Figure 4(b). This encoding accomplishes on one qubit the
encapsulation of 2 dimensions of non-linearly separable data, whilst
accomplishing a separation task. For this to be done, two parameters per
qubits are used to transfer between Classical data to Quantum state. These
parameters are the rotations around the Y axis, proceeded by a rotation around
the Z axis.

(a) Bit representation)

(b) Qubit representation

Figure 4: Cicle dataset with bit and qubit representations

Translating the classification problem of the data outlined in Figure 4(b) to
GenQu framework, we follow the following approach. We begin by translating our
data into rotations according to the functions outlined in Section III-B,
however the values encoded are vector distances from the circle center. This
can be considered both the Quantum Circuit Preparation and Data Transformation
components, as the expected measurement under Qubits Measurement is equivalent
to the classification of a data point. There is no updating of parameters in
this case, therefore we do not iterate and update circuit parameters.

###  III-D Quantum Deep Learning Architecture

In this paper we make use of the data qubitization techniques outlined above,
along side current quantum machine learning techniques to enable highly
performent quantum deep learning. Through encoding 2 dimensions of data per
qubit, the number of neurons in our network input is half the dimensionality
of the data set. Quantum deep learning layers comprised of single qubit
operations are namely called single qubit unitary layers. In these layers each
qubit has a RY and RZ gate appended, thereby adding $2n$ parameters, where n
iWs the number of qubits. Another type of layer consists of operations acting
on two qubits per gate, where operations are control operations (CRY or CRZ).
These are namely Entanglement layers. Entanglement layers entangle all qubits
by some learnable amount, performing CRY and CRZ gates on qubits $i$ and $i+1$
until there are no qubits left to pair. Entanglement layers require $2(n-1)$
parameters. These gates are visualized in Figure 2. Through the use of the
entanglement layer, we can reduce and grow the number of qubits at any time
point across a circuit dynamically. In our case of illustrating binary
classification, we make use of the entanglement layer to pool down data from
the other qubits onto one qubit, which is measured and used as the
classification qubit. The probability of the qubit measuring $|1\rangle$ is
thought of as the probability of labelling the quantum data that was fed to
the circuit as $Class=1$ similar to how a single output neuron operates of
activation function Sigmoid operates in classical neural networks.

Fitting this Quantum Deep Learning model to our GenQu framework, we begin by
translating our data into rotations described under Section III-B. From here,
the practitioner can describe their full quantum deep learning architecture
and initialize the parameters. The data is loaded onto a quantum computer in
series, with the quantum data loading circuit being appended with the quantum
deep learning architecture. The quantum circuit is induced and the
classification qubit measured. We feed this result back to a classical
computer, calculate our loss and update our parameters accordingly. This is
repeated until convergence occurs or sufficient accuracy is attained.

###  III-E Quantum Generative Nature

Another powerful use of qubits is in the representation of data. Through using
trainable circuits as discussed above, we can measure two values from one
qubit. Therefore, a quantum deep neural network can be trained to mimic some
input data by defining some loss function such as the Mean Squared Error, and
generate new samples that are close to what the qubit was trained on. This is
similar to Generative Adversarial Networks [23], but does not take any noise
as an input, nor does it require two networks to be used. We do not claim that
ours is better, however it is one of the side effects of qubits being used to
represent data, and quantum deep learning models. Therefore, we illustrate
through the use of quantum deep learning how a quantum deep learning
architecture with a tuned loss function can generate data similar to that of
the data it was fed, and at a generative diversity significantly greater that
is unattainable using similar architectures within its classical counterparts.

Translating a quantum generative state to GenQu framework, we repeat the steps
outlined in the Quantum Deep Learning architecture above. However, the only
change would be the loss function such that the quantum state instead of a
loss function such as cross entropy, could be mean squared error or some other
applicable loss function. Furthermore, no data loading for input is necessary,
and instead are just loaded as qubits in the state of $|0\rangle$. When
generating data, the qubits are measured and sent to the Output Data stream.

##  IV Results

We implement GenQu with IBM Qiskit and Tensorflow Quantum. It is evaluated
with the above mentioned three applications, kernelized classification,
quantum deep learning, and quantum generative nature. We evaluate GenQu on
both simulators and IBM-Q quantum computers (mainly Rome). We compare our
results with Tensorflow Quantum (TFQ) and traditional convolutional neural
networks with different numbers of parameters. In the rest of the evaluation,
we denote CNN - xP to be classical neural networks with x parameters and QNN -
xP is quantum based neural networks with x parameters.

###  IV-A The Kernelized Classification

As a proof of a single qubit natural machine learning, we employ the encoding
of a circles dataset illustrated in Figure 4(b) onto one qubit through the
radial kernel method. A single qubit has data points encoded as the vector
distance from the center of the circle in Figure 4(b). The qubit is then
measured, and the $P(|\phi\rangle)=|0\rangle$ is equivalent to $P(Class=1)$.
Through doing so, we attain $100\%$ accuracy on separating the non-linearly
separable data set, whilst maintaining both dimensions of information.
Furthermore, when our experiment is run on IBM-Q’s Quantum Computer Rome 100%
accuracy is attained, thereby providing us with confidence that our model
architecture works both on simulators and real quantum computers. This
approach, although not novel, is done to illustrate that certain problems can
be tackled very efficiently with qubits and how the solution can be
successfully run on real quantum computers.

###  IV-B Quantum Deep Learning

Figure 5: Quantum Deep Learning Circuit

To evaluate the learning potential of quantum deep learning, and its ability
to use fewer data-channels than its classical counterpart, we make use of the
MNIST data set. The MNIST data set is an image data set comprised of gray
scale hand-drawn digits of resolution $28\text{ by }28$. It is infeasible to
represent these images on current near-term quantum devices, and hence we make
use of PCA [19] to reduce dimensionality from $784$ to $4$. In this case, we
only need to make use of 2 qubits to feed our data to our quantum deep neural
network. We provide the deep learning circuit visualized in Figure 5. As can
be seen in the circuit, there is a total of 8 parameters. This network is
comprised of one single qubit unitary layer (Parameters 0 through 3), one
entanglement layer which accomplishes data pooling onto one qubit (Parameters
4 and 5), and finally a single qubit unitary on the final output qubit
(Parameters 6 and 7). We compare our architecture to classical deep learning
architectures and compare parameter counts when using the same gradient
descent approach (Adam Optimizer), same epochs, and same data set. The quantum
network is trained to perform binary classification of two numbers from the
MNIST dataset, where the classification is measured by the Qubit $(0,1)$ in
Figure 5. As for the classical networks, we make use of a network comprised of
a middle layer of tensor size $2,8,16\text{ and }32$. The comparative training
results are visualized in Figure 6.

(a) 9 vs 6

(b) 3 vs 1

(c) IBM-Q vs Simulations

Figure 6: QNN (simulation), QNN (IBM-Q Rome), TFQ and CNN Results

In Figure 6(a), the numbers 9 and 6 are used to train the data set. The
quantum network outperforms all other comparative solutions, with close to
equivocal performance of a 193 parameter deep neural network, thereby
attaining a 95.86% parameter count reduction, and converging 33.33% faster
than said network. However, in the case of 9 and 6, there is a less
significant difference between parameter counts than what is observed in other
cases, such as 3 and 1. In Figure 6(b), we observe how there is substantial
learning ability to be gained from increasing the classical parameter count.
However, similarly in this case, the 8 parameter QNN’s performance is matched
by the 97 parameter CNN, a 91.76% reduction in parameters. We also compare our
model to the Tensorflow Quantum (TFQ) MNIST classification example on the same
number pairs[14]. We illustrate that our network outperforms Tensorflow
Quantums MNIST classification task in Figures 6 (a) and (b). This illustrates
the significant learning potential of quantum networks and specifically the
architecture used in this paper. These architectures are able to, in certain
cases, reduce parameter counts significantly with no sacrifice to performance.
Furthermore, in our case we have encoded two dimensions of data per qubit.
Feeding 4 dimensions of data to a deep neural network through 2 neurons is
impractical, and is a further example of how powerful qubits are in deep
learning.

We validate our results by running similar experiments on a real quantum
computer using the IBM-Q platform, comparing the accuracy’s attained on a
simulator to that of a on a quantum computer. These results are visualized in
Figure 6(c). As can be seen, for numbers 4-3 and 9-6, actual quantum computer
performance was extremely similar to that of the simulator, with a difference
of less than 5%, and 9-6 having a measured difference of 0.2%. However, in the
case 3-1 we observe more significant differences between actual Quantum
Computing implementation. The largest difference between simulators and actual
quantum computers was 7.25% on the 3-1 dataset, which is due to the noise on
the quantum computer that depends on the computer itself and the rotated
workloads on it (random factors). We further validate our results by comparing
them to the TFQ MNIST classification task, and show how in some cases our
architecture and network attaining significantly higher accuracy.
Specifically, in the case of 4-3 with a 20% improvement and 3-1 with a 22%
improvement. However, in certain cases our network attained the same accuracy
as TFQ, in the cases of 2-9 and 9-6.

###  IV-C Quantum Generative Nature

Another point of interest is how powerful qubits are in representing data
sets. This has significant implications in Generative Adversarial Networks
(GANs) and loading data sets. We illustrate this potential by minimizing the
distance between a single qubit‘s quantum state and the MNIST data set PCA’ed
to 2 dimensions and of class 0. We illustrate in Figure 7 how a single qubit,
visualized by the blue shading, using only 2 parameters (an RY and RZ gate in
series on one qubit), can completely mimic the data it was fed (2 dimensions).
If sampled, the qubit will generate all samples it was fed as well as generate
new unique samples similar to that of which it was fed. We make use of the
architecture of a Generative Adversarial Network [23] to compare this to a
classical neural network, and observe how poorly the classical counterpart
performs. When given 9800% more parameters (2 vs 196), as visualized in Figure
7(b), the network was still unable to mimic the data fed to the network. The
classical network was able to converge when provided with $2144$ parameters,
as visualized in Figure 7(c). Furthermore, we illustrate the generative
potential by sampling from the aforementioned quantum circuits with a shot
count of 15, performing reconstructive PCA on the data and plotting the
images. As seen in the images the qubit is able to learn from almost nothing
in 8(a) to reasonable results after 50 epochs in 8(c). This further
illustrates the significant machine learning potential and parameter reduction
potential of quantum machine learning.

(a) Epoch 25 - QNN - 2P

(b) Epoch 25 - CNN - 196P

(c) Epoch 25 - CNN - 2144P

Figure 7: Single qubit generative model to learn the distribution of PCA MNIST
digit 0

(a) Epoch 0

(b) Epoch 25

(c) Epoch 50

Figure 8: 2-Dimension Images Generated by GenQu

##  V Conclusion

This paper proposes GenQu, a hybrid and general-purpose quantum framework for
learning classical data. It extends the Tensorflow Quantum framework such that
the applications can be executed on public-available quantum computers (IBM-Q
Platform). GenQU demonstrates the significant expressibility of qubits, and
their extensive applications in machine and deep learning. Based on GenQu, we
propose three different learning models that make use of a low-qubit count in
near-term quantum computers. In the model for kernelized classification, GenQu
is able to to encode the circle dataset onto a single qubit achieve 100%
accuracy in the experiment on a real quantum computer. With quantum deep
learning model in GenQu, when encoding two dimensions of data per one qubit,
it is able to show reductions in parameters equivalent to 95.86%, whilst still
attaining a similar accuracy or better than that of classical deep learning
models. Finally with respect to qubits learning potential, a single qubit
generative model is proposed. It is able to completely learn to generate 2
dimensions of data PCA’ed from the MNIST’s 0 class. As the future work, we
plan to fully integrate GenQu with Tensorflow Quantum and add more build-in
features, such as amplitude encoding, customized evaluation functions.

## Acknowledgment

This material is based upon work supported by the U.S. Department of Energy,
Office of Science, National Quantum Information Science Research Centers, Co-
design Center for Quantum Advantage ($C^{2}QA$) under contract number DE-
SC0012704.

## References

  * [1] N. C. Thompson, K. Greenewald, K. Lee, and G. F. Manso, “The computational limits of deep learning,” _arXiv preprint arXiv:2007.05558_ , 2020.
  * [2] F. Arute, K. Arya, R. Babbush, D. Bacon, J. C. Bardin, R. Barends, R. Biswas, S. Boixo, F. G. Brandao, D. A. Buell _et al._ , “Quantum supremacy using a programmable superconducting processor,” _Nature_ , vol. 574, no. 7779, pp. 505–510, 2019.
  * [3] S. A. Stein, Y. Mao, B. Baheri, Q. Guan, A. Li, D. Chen, S. Xu, and C. Ding, “Quclassi: A hybrid deep neural network architecture based on quantum state fidelity,” _arXiv preprint arXiv:2103.11307_ , 2021.
  * [4] K. Beer, D. Bondarenko, T. Farrelly, T. J. Osborne, R. Salzmann, D. Scheiermann, and R. Wolf, “Training deep quantum neural networks,” _Nature communications_ , vol. 11, no. 1, pp. 1–6, 2020.
  * [5] T. E. Potok, C. Schuman, S. Young, R. Patton, F. Spedalieri, J. Liu, K.-T. Yao, G. Rose, and G. Chakma, “A study of complex deep learning networks on high-performance, neuromorphic, and quantum computers,” _ACM Journal on Emerging Technologies in Computing Systems (JETC)_ , vol. 14, no. 2, pp. 1–21, 2018.
  * [6] Y. Levine, O. Sharir, N. Cohen, and A. Shashua, “Quantum entanglement in deep learning architectures,” _Physical review letters_ , vol. 122, no. 6, p. 065301, 2019.
  * [7] S. A. Stein, B. Baheri, R. M. Tischio, Y. Mao, Q. Guan, A. Li, B. Fang, and S. Xu, “Qugan: A generative adversarial network through quantum states,” _arXiv preprint arXiv:2010.09036_ , 2020.
  * [8] I. Kerenidis, A. Luongo, and A. Prakash, “Quantum expectation-maximization for gaussian mixture models,” _ICML 2020: Thirty-seventh International Conference on Machine Learning_ , 2019.
  * [9] T. Li, S. Chakrabarti, and X. Wu, “Sublinear quantum algorithms for training linear and kernel-based classifiers,” _ICML 2019: Thirty-sixth International Conference on Machine Learning_ , 2019.
  * [10] C. Ding, T.-Y. Bao, and H.-L. Huang, “Quantum-inspired support vector machine,” _arXiv preprint arXiv:1906.08902_ , 2019.
  * [11] A. Panahi, S. Saeedi, and T. Arodz, “word2ket: Space-efficient word embeddings inspired by quantum entanglement,” _ICLR 2019 : International Conference On Learning Representations_ , 2019.
  * [12] G. Verdon, M. Broughton, J. R. McClean, K. J. Sung, R. Babbush, Z. Jiang, H. Neven, and M. Mohseni, “Learning to learn with quantum neural networks via classical neural networks,” _arXiv preprint arXiv:1907.05415_ , 2019\.
  * [13] S. Y.-C. Chen, C.-H. H. Yang, J. Qi, P.-Y. Chen, X. Ma, and H.-S. Goan, “Variational quantum circuits for deep reinforcement learning,” _IEEE Access_ , vol. 8, pp. 141 007–141 024, 2020.
  * [14] M. Broughton, G. Verdon, T. McCourt, A. J. Martinez, J. H. Yoo, S. V. Isakov, P. Massey, M. Y. Niu, R. Halavati, E. Peters _et al._ , “Tensorflow quantum: A software framework for quantum machine learning,” _arXiv preprint arXiv:2003.02989_ , 2020.
  * [15] C. Zoufal, A. Lucchi, and S. Woerner, “Quantum generative adversarial networks for learning and loading random distributions,” _npj Quantum Information_ , vol. 5, no. 1, pp. 1–9, 2019.
  * [16] J. Cortese and T. Braje, “System and technique for loading classical data into a quantum computer,” Jul. 11 2019, uS Patent App. 16/239,983.
  * [17] J. Preskill, “Quantum computing in the nisq era and beyond,” _Quantum_ , vol. 2, p. 79, 2018.
  * [18] P. Kaye, R. Laflamme, M. Mosca _et al._ , _An introduction to quantum computing_. Oxford university press, 2007\.
  * [19] H. Hoffmann, “Kernel pca for novelty detection,” _Pattern recognition_ , vol. 40, no. 3, pp. 863–874, 2007.
  * [20] S. Garg and G. Ramakrishnan, “Advances in quantum deep learning: An overview,” _arXiv preprint arXiv:2005.04316_ , 2020.
  * [21] I. Goodfellow, Y. Bengio, A. Courville, and Y. Bengio, _Deep learning_. MIT Press, 2016, vol. 1.
  * [22] G. E. Crooks, “Gradients of parameterized quantum gates using the parameter-shift rule and gate decomposition,” _arXiv preprint arXiv:1905.13311_ , 2019.
  * [23] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in _Advances in neural information processing systems_ , 2014, pp. 2672–2680.

##  Appendix A Appendix: Experiments on IBM-Q quantum computers

To justify our proposed model, we conducted experiments on multiple IBM-Q
sites including Vigo, Ourense, Rome, Bogota and Valencia. Figure 10 is the
setup for Kernelized classification on the circle dataset. We ran this circuit
with 40 shots (repetition) 20 times on different locations and achieved
accuracy of 100% .

Figure 9: Experiment setup for Kernerlized classification

Figure 10: Quantum circuit on MNIST dataset (T1)

Figure 11: Quantum circuit on MNIST dataset (T2)

Figure 12: Quantum circuit on MNIST dataset (T3)

In PCA MNIST dataset, we train the model on 913 samples of two specific digits
(9 and 6) on different IBM-Q locations to justify our founding. Figure 10, 12
and 12 present a sample of each iteration over actual IBM-Q machine which ran
for 913 times to get accuracy of 95% on average.


# EENLP: Cross-lingual Eastern European NLP Index

###### Abstract

Motivated by the sparsity of NLP resources for Eastern European languages, we
present a broad index of existing Eastern European language resources (90$+$
datasets and 45$+$ models) published as a github repository open for updates
from the community. Furthermore, to support the evaluation of commonsense
reasoning tasks, we provide hand-crafted cross-lingual datasets for five
different semantic tasks (namely news categorization, paraphrase detection,
Natural Language Inference (NLI) task, tweet sentiment detection, and news
sentiment detection) for some of the Eastern European languages. We perform
several experiments with the existing multilingual models on these datasets to
define the performance baselines and compare them to the existing results for
other languages.

Keywords: cross-lingual, less-resourced languages, NLP

languageresourceLanguage Resources

EENLP: Cross-lingual Eastern European NLP Index

Alexey Tikhonov*††thanks: * The corresponding author is Alexey Tikhonov, Alex
Malkhasov, Andrey Manoshin, George Dima,
---
Réka Cserháti, Md.Sadek Hossain Asif, Matt Sárdi
Independent researcher, Germany; Financial University of Russia; MEPhI,
Russia;
University Politehnica of Bucharest, Romania; University of Szeged, Hungary;
Notre Dame College, Dhaka, Bangladesh; Mozaik Education, Hungary
{altsoph, realex1902, sqidde, andreigeorgedima, cserhatir, asifsadek509,
<EMAIL_ADDRESS>

Abstract content

## 1\. Introduction

Recent multilingual Transformer-based language models – such as mBERT [Devlin
et al., 2018], XLM-RoBERTa [Sanh et al., 2019], multilingual DistilBERT
[Conneau et al., 2020], etc. – show impressive results on different text
analysis tasks and their cross-lingual reasoning capabilities are still
actively studied [Lauscher et al., 2020]. For example, the mBERT model trained
on 104 languages has shown high cross-lingual performance; however, such
evaluations mostly focused on cross-lingual transfer within high-resource
languages [Wu and Dredze, 2020].

Commonsense reasoning is one of the key problems in natural language
processing, but the relative scarcity of labeled data holds back the progress
for languages other than English: there are widely spoken languages that still
did not receive the focus of the research community [Joshi et al., 2020].
Cross-lingual transfer learning could be beneficial for such languages in
solving both theoretical and practical tasks. One can speculate that such
high-level reasoning tasks could be less affected by the language syntax’s
specifics and low-level properties, so it can be effective to use a cross-
lingual approach here. However, it was shown that the success of cross-lingual
transfer learning depends on different factors such as the amount of shared
vocabulary, explicit alignment of representations across languages, size of
pretraining corpora, etc [Doddapaneni et al., 2021]. To get a better
understanding of the importance of these factors, researchers need to leverage
diverse and detailed datasets.

There are a bunch of cross-lingual datasets already, such as XGLUE [Liang et
al., 2020], XCOPA [Ponti et al., 2020], XL-WiC [Raganato et al., 2020], XWINO
[Tikhonov and Ryabinin, 2021], Wino-X [Emelin and Sennrich, 2021], XNLI
[Conneau et al., 2018], XL-WSD [Pasini et al., 2021], XTREME-R [Ruder et al.,
2021], BSNLP [Piskorski et al., 2019], MOROCO [Butnaru and Ionescu, 2019],
etc., but most of them cover high-resource languages.

In this paper, we concentrate on Eastern European languages. These languages
are numerous and heterogeneous; they include languages from at least two
language families (Indo-European and Uralic), and the former family is
represented by very diverse branches. Although there are several dedicated
multi-task benchmarks for a few Eastern European languages – e.g., KLEJ [Rybak
et al., 2020] for Polish, LiRo [Dumitrescu et al., 2021] for Romanian,
RussianSuperGLUE [Shavrina et al., 2020] for Russian, or the translation of
SuperGLUE for Slovene, they usually concentrate on one or two languages. They
also use different tasks and various data formats; hence they can not be used
for cross-lingual benchmarking without careful manual pre-processing.

The main contributions of this paper are:

  * •

To build a comprehensive picture of the current NLP state for the Eastern
European languages, we present the wide index of existing Eastern European
languages resources (90+ datasets and 45+ models) published as a github
repository111https://github.com/altsoph/EENLP open for updates from the
community;

  * •

Next, we provide hand-crafted cross-lingual datasets for five different
semantic tasks (namely news categorization, paraphrase detection, Natural
Language Inference (NLI) task, tweet sentiment detection, and news sentiment
detection), compiled by processing data from various sources into the same
format, opening room for evaluation scenarios such as zero-shot cross-lingual
transfer. Since the source datasets are licensed under various licenses, we
publish automatic scripts for our datasets compilation on the same github
repository;

  * •

Finally, we perform several experiments with the existing multilingual models
on our datasets to define the performance baselines and compared them to the
existing similar results for other languages.

We made all our code and data publicly available. We also published the
detailed results of our experiments at our Weights and Biases
project222https://wandb.ai/eenlp [Biewald, 2020].

Figure 1: MLM Transformer-based models with Eastern European languages
coverage.

## 2\. Dataset index

Aiming to build the index of Eastern European language NLP resources useful to
the community, we have created a long list of available datasets we were able
to find. We focused on supervised datasets (with data labels) and preferred
semantic tasks to syntactic ones. Lastly, since some languages are already
well covered with benchmarks (for example, Polish, Romanian, Russian), we
tried to focus on less represented languages. Indeed, we do not claim this
list is exhaustive; however, we provide an easy way to add any missing
resources via the creation of a github issue. We encourage the community to
help with updating this index further.

For this moment, we’ve already collected more than 90 datasets for 20
different languages: Albanian, Armenian, Belarusian, Bosnian, Bulgarian,
Croatian, Czech, Estonian, Hungarian, Latvian, Lithuanian, Macedonian,
Moldavian, Polish, Romanian, Russian, Serbian, Slovak, Slovenian, and
Ukrainian.

These datasets cover various tasks, including text category prediction,
coreference resolution, fake news detection, lemmatization, morphosyntactic
tagging, NER, NLI, offensive comments detection, paraphrase detection, POS
tagging, question answering, sentiment analysis, word sense disambiguation,
and many more. The complete list of the discovered datasets is published on
our github
repository333https://github.com/altsoph/EENLP/blob/main/docs/datasets.md.

## 3\. Models index

In a similar way, we tried to enumerate available models useful for Eastern
European languages. We were aiming for the Transformer-based MLM models (which
are state of the art for text classification nowadays), but we also found some
models of other types, including Causal Language Models, Adapters, static
embeddings, etc; we listed them as well.

We started with the modern well-known multilingual models, namely mBERT
[Devlin et al., 2018], mDistilBERT [Sanh et al., 2019], XLM-Roberta [Conneau
et al., 2020], and LaBSE [Reimers and Gurevych, 2019]. Next, there are several
cross-lingual models, such as CroSloEngual [Ulcar and Robnik-Sikonja, 2020],
BERTić [Ljubesic and Lauc, 2021], SlavicBERT [Arkhipov et al., 2019], or
LitLat BERT. Finally, we found more than 20 language-specific BERT-like
models; most of them are listed in Figure 1.

Besides MLM models, we found 6 GPT-like models and several dozens of models
from the pre-Transformer era: ULMFiT, ELMo, and static word embeddings. All
our findings are available on our github
repository444https://github.com/altsoph/EENLP/blob/main/docs/models.md as
well.

## 4\. Benchmark tasks

Considering the coverage and sparsity of source datasets, we decided to
proceed with five commonsense reasoning tasks with different Eastern European
languages subsets. We checked these tasks for the languages coverage and
finally decided to focus on this short-list:

  * •

news categorisation,

  * •

paraphrase detection,

  * •

news sentiment detection,

  * •

tweet sentiment detection,

  * •

NLI.

For each of these tasks, we manually crafted a dedicated dataset with
maximized Eastern European languages coverage. Whenever it was possible we
also added English as a language most common for contemporary pre-trained
models. The properties of these datasets are given in the following sections.
Since the source datasets are licensed under various licenses, we published
automatic scripts for our datasets compilation on the same github
repository555https://github.com/altsoph/EENLP/tree/main/build_benchmarks.

For each of these tasks, we evaluated three modern transformer-based pre-
trained models: a multilanguage version of BERT, a model XLM-Roberta that is
two times larger than BERT, and a distilled version of multilanguage BERT (to
check how the distillation affects the cross-lingual transfer quality).
Whenever the English language was also available for the task we added a base
English BERT model to our evaluation. We fine-tuned them using AdamW optimizer
with the learning rate 1e-5, epsilon 1e-8, and the linear schedule with
warmup. We reported all the metrics for the epoch with the best source
language validation result and published our code for
evaluation666https://github.com/altsoph/EENLP/tree/main/eval_benchmarks.

We will now describe every task in detail.

### 4.1. News categorisation task

In this task, the model should detect the correct category of the news text.
This is a basic semantic task; however, it’s not always obvious how to
separate, for example, categories like ”lifestyle” and ”entertainment.” Our
dataset consists of 8 major categories and covers 7 Eastern European languages
+ English (however, the distribution of categories differs from one language
to another, consider Table 1).

The sources for this dataset are:

  * •

English: [Liang et al., 2020],

  * •

Armenian: [Avetisyan and Ghukasyan, 2019],

  * •

Estonian: [Purver et al., 2021],

  * •

Latvian: [Pollak et al., 2021],

  * •

Moldavian: [Butnaru and Ionescu, 2019],

  * •

Romanian: [Butnaru and Ionescu, 2019],

  * •

Russian: [Liang et al., 2020],

  * •

Slovak: [Hladek et al., 2014].

| en | hy | ee | lv | mo | ro | ru | sk
---|---|---|---|---|---|---|---|---
entertainment | 3498 | 996 | 275 | 227 | 1234 | 1058 | 1498 | 22
finance | 13405 | 2066 | 13 | 34 | 3429 | 5105 | 2514 | 1022
health | 5751 |  | 121 | 119 |  |  | 311 | 559
lifestyle | 6295 | 1630 | 107 | 37 |  |  | 671 | 481
news/accidents | 31490 | 1454 | 33526 | 3722 |  |  | 12125 | 1655
sports | 38598 | 2797 |  | 149 | 3443 | 2583 | 1731 | 1042
tech/auto | 3926 |  | 144 | 40 | 1266 | 3392 | 387 |
travel | 3076 |  | 49 | 55 |  |  | 53 |
Table 1: News categorisation dataset properties. For each language / category
pair we reported the number of the items (short news texts) in our dataset.

We use this task to measure the models’ cross-lingual transfer quality while
transferring from English to various Eastern European languages. (One can also
use Latvian or Russian as a source language since both of them cover all of
the categories). Following [Liang et al., 2020], we use multi-class
classification accuracy as the key metric for this task. Table 2 sums up the
results of four models on a multilanguage dataset. Further details across
languages and categories are available on
W&B777https://wandb.ai/eenlp/newscat_full.

| mDBERT | mBERT | XLMR | BERT
---|---|---|---|---
en | 91.11% |  92.50% | 92.36% | 92.65%
ee | 74.83% |  81.58% | 72.55% | 27.33%
hy | 55.57% | 66.10% | 67.64% | 21.41%
lv | 63.79% | 86.60% | 87.68% | 6.31%
mo | 65.79% |  66.11% | 62.36% | 16.12%
ro | 46.18% |  47.54% | 44.18% | 7.46%
ru | 77.49% |  77.34% | 77.10% | 43.31%
sk | 59.88% |  63.25% | 60.32% | 35.34%
Table 2: News categorisation task. For each language / model pair we reported
the accuracy value on the same epoch where the validation accuracy was the
best. We also evaluated a standard English BERT here to compare.

The main observations based on the results of this task are:

  * •

multilingual pre-training is the key factor for successful cross-lingual
transfer learning, English BERT model shows no ability to transfer its
knowledge from English to other languages;

  * •

mBERT and XLM-R models show a similar level of quality as in [Liang et al.,
2020] for the similar NC task (however, note, it’s unfair to compare them
directly since the list of categories is not the same);

  * •

distillation significantly affects the ability to cross-lingual transfer
learning; this effect is significant for almost all of the tested languages;

  * •

the results vary across the languages a lot, this is a direction of possible
deeper analysis (note that, for example, Romanian and Moldavian are close
languages; however, all the models are performing significantly better on
Moldavian);

  * •

mBERT generally performs slightly better than XLM-RoBERTa, this could be an
effect of relatively small datasets.

### 4.2. Paraphrase detection task

Language | Items | Classes
---|---|---
| English
---
373 263 | 2
| Armenian
---
4 233 | 2
| Polish
---
8 000 | 2
| Romanian
---
5 749 | 2
| Serbian
---
835 | 2
Table 3: Paraphrase detection dataset properties. Each item is a pair of
sentences.

In the paraphrase detection task, the model should decide whether a pair of
sentences have the same meaning. It is another standard task for contemporary
NLU benchmarks; however, it is significantly hard and can be problematic even
for the English language [Yamshchikov et al., 2021]. Our dataset consists of
four Eastern European languages + English. The sources for this dataset are:

  * •

English: [Wang et al., 2018],

  * •

Armenian: [Malajyan et al., 2020],

  * •

Polish: [Rybak et al., 2020],

  * •

Romanian: RO-STS dataset888https://github.com/dumitrescustefan/RO-STS,

  * •

Serbian: [Batanović et al., 2011].

We also tried to leverage the TaPaCo dataset [Scherrer, 2020] to build a
derivative dataset with positive and negative examples of paraphrases. We
treated pairs of sentences from the same cluster as positive paraphrase
examples. Then, we used several strategies for sampling hard-negative samples
(including sampling from similar sentences by Levenshtein distance and LabSE
[Feng et al., 2020] embedding distances). To assess the quality of such
dataset augmentation, we performed a two-folded cross-check: considering
languages where we have both TaPaCo and non-TaPaCo datasets, we checked the
cross-datasets transfer learning in both directions. The results of non-
TaPaCo-$>$TaPaCo transfer were much better than for the opposite direction
(for example, for Polish, it was 68% versus 51%), which implies that datasets
represent different tasks with non-equal complexity. Thus, we decided to
reject a TaPaCo-based augmentation until further investigations. However, we
consider it an interesting direction for future work.

We used the same setup as in the previous task – cross-lingual zero-shot
learning from English to other languages. Following [Yang et al., 2019], we
use classification accuracy as the key metric for this task. We reported
values in table 4 from the same epoch where the validation accuracy was the
best. More details are available
online999https://wandb.ai/eenlp/paraphrase_detection/artifacts/
paraphrase_detection-dataset/paraphrase_detection-dataset/
d144e76e6129b022dc5d.

| mDBERT | mBERT | XLMR | BERT
---|---|---|---|---
en | 81.84% | 82.20% |  83.80% | 82.20%
hy | 88.68% |  92.79% | 92.70% | 75.36%
pl | 87.86% |  87.29% | 82.14% | 84.33%
ro | 79.82% | 82.36% |  86.88% | 80.03%
sr | 72.10% |  77.01% | 71.74% | 55.09%
Table 4: Paraphrase detection task. For each language / model pair we reported
the accuracy value on the same epoch where the validation accuracy was the
best. We also evaluated a standard English BERT here to compare.

The main observations based on these results are:

  * •

again, multilingual pre-trained models show consistently better performance
than English BERT, which highlights the importance of the multilingual pre-
training phase;

  * •

mBERT results are competitive with the results reported in [Yang et al.,
2019];

  * •

once more, distillation affects the ability to cross-lingual transfer
learning.

### 4.3. News sentiment detection task

Language | Items | Classes
---|---|---
| Croatian
---
1 500 | 3
| Lithuanian
---
10 000 | 3
| Russian
---
8 200 | 3
| Slovenian
---
10 000 | 3
Table 5: News sentiment detection dataset properties. Each item is a short
news text.

In this task, we made an evaluation of sentiment detection (3 classes:
positive, neutral, negative) for relatively long texts (news articles). We
compiled a dataset, that covers 4 Eastern European languages (Croatian,
Lithuanian, Russian, and Slovenian). Table 5 describes volumes of data per
language. The sources for this dataset are:

  * •

Croatian: [Pelicon et al., 2020],

  * •

Lithuanian: TN2gramos 101010https://www.kaggle.com/rokastrimaitis/lithuanian-
financial-news-dataset-and-bigrams,

  * •

Russian: Sentiment Analysis in
Russian111111https://www.kaggle.com/c/sentiment-analysis-in-russian,

  * •

Slovenian: [Bučar et al., 2018].

We used classification accuracy as our main metric and evaluated all possible
directions of transfer learning for three multilingual models (mBERT, mDBERT,
and XLM-R), as in [Pelicon et al., 2020]. Check table 6 for the results, more
details are available on W&B121212https://wandb.ai/eenlp/news_sentiment.

|  | hr | lt | ru | sl
---|---|---|---|---|---
| mDBERT | 67% | 20% | 48% | 53%
hr | mBERT | 66% | 23% | 53% | 57%
| XLMR |  70% |  24% |  50% |  60%
| mDBERT | 45% | 70% | 50% | 51%
lt | mBERT | 49% | 71% | 57% | 49%
| XLMR |  51% |  78% |  58% |  63%
| mDBERT |  63% | 39% | 68% | 52%
ru | mBERT | 57% |  39% |  71% | 55%
| XLMR | 56% | 65% | 70% |  59%
| mDBERT |  55% | 41% | 57% | 64%
sl | mBERT | 54% | 52% | 58% | 67%
| XLMR | 51% |  59% |  60% |  69%
Table 6: News sentiment detection task. Validation accuracy.

This task results support several observations:

  * •

as in previous experiments, the distillation decreases the ability to cross-
lingual transfer learning;

  * •

the cross-lingual transfer quality seems to be correlated with the degree of
language affinity;

  * •

XLM-RoBERTa generally dominates over the rest of the models.

### 4.4. Twitter sentiment detection task

Language | Items | Classes
---|---|---
| Czech
---
10 000 | 3
| Latvian
---
1 160 | 3
| Russian
---
230 000 | 2
| Slovak
---
1 600 | 3
Table 7: Twitter sentiment detection dataset properties. Each item is a tweet
text.

In addition to the previous task, we considered the task of short texts
(tweets) sentiment detection (same 3 classes: positive, neutral, negative).
The dataset for this task also covers 4 Eastern European languages (Czech,
Latvian, Russian, and Slovak). Table 7 describes volumes of data per language.
The sources for this dataset are:

  * •

Czech: [Habernal et al., 2013],

  * •

Latvian: latvian-tweet-sentiment-corpus 131313https://github.com/FnTm/latvian-
tweet-sentiment-corpus,

  * •

Russian: mokoron141414http://study.mokoron.com/,

  * •

Slovak: Sentigrade151515https://sentigrade.fiit.stuba.sk/data.

As previously, we used classification accuracy as our main metric and
evaluated all possible directions of transfer learning for three multilingual
models (mBERT, mDBERT, and XLM-R). Check table 8 for the results, more details
are available on W&B161616https://wandb.ai/eenlp/twit_sentiment

|  | ru | sk | lv | cz
---|---|---|---|---|---
| mDBERT | 88% | 61% | 75% | 68%
ru | mBERT | 89% | 60% | 76% | 67%
| XLMR |  95% |  72% |  78% |  79%
| mDBERT | 72% | 51% | 30% | 35%
sk | mBERT |  76% | 59% | 32% | 38%
| XLMR | 72% |  70% |  38% |  42%
| mDBERT | 39% | 35% | 70% | 52%
lv | mBERT |  56% |  38% | 71% | 51%
| XLMR | 32% |  38% |  73% |  57%
| mDBERT | 36% | 47% | 60% | 74%
cz | mBERT |  44% | 50% | 60% | 76%
| XLMR |  44% |  62% |  71% |  79%
Table 8: Twitter sentiment detection task. Validation accuracy.

This task results support several observations:

  * •

again, the distilled model shows the worst ability to cross-lingual transfer
learning;

  * •

again, the similarity of the languages matters for the cross-lingual transfer
quality;

  * •

again, XLM-RoBERTa generally dominates over the rest of the models regardless
of the transfer pair.

### 4.5. NLI task

Language | Items | Classes
---|---|---
| English
---
30 000 | 3
| Bulgarian
---
30 000 | 3
| Polish
---
9 000 | 3
| Russian
---
30 000 | 3
| Slovenian
---
306 | 3
Table 9: NLI dataset properties. Each item is a pair of sentences.

Finally, we proposed the evaluation on the natural language inference (NLI)
task. NLI is the task of predicting whether a hypothesis sentence is true
(entailment), false (contradiction), or undetermined (neutral) given a premise
sentence. Our dataset for this task also covers 4 Eastern European languages
(Bulgarian, Polish, Russian, and Slovenian) + English. Table 9 describes
volumes of data per language. The sources for this dataset are:

  * •

English: [Liang et al., 2020],

  * •

Bulgarian: [Liang et al., 2020],

  * •

Polish: [Rybak et al., 2020],

  * •

Russian: [Liang et al., 2020],

  * •

Slovenian: [Žagar et al., 2020].

As in [Artetxe and Schwenk, 2019] and [Nooralahzadeh et al., 2020], we used
classification accuracy as the metric for the NLI task and evaluated all
possible directions of transfer learning for the same three multilingual
models (mBERT, mDBERT, and XLM-R). Table 10 provides the results, more details
are available on W&B171717https://wandb.ai/eenlp/nli

|  | en | pl | ru | bg | sl
---|---|---|---|---|---|---
| mDBERT | 65% |  62% | 56% | 55% | 34%
en | mBERT | 69% | 48% | 59% | 58% |  37%
| XLMR |  76% | 48% |  69% |  72% | 36%
| mDBERT | 40% | 92% |  39% |  40% |  26%
pl | mBERT | 42% | 92% | 38% | 38% | 19%
| XLMR |  46% |  94% | 38% |  40% | 14%
| mDBERT | 62% |  57% | 54% | 55% |  33%
ru | mBERT | 68% | 54% | 54% | 60% | 28%
| XLMR |  75% |  57% |  64% |  73% | 31%
| mDBERT | 62% |  62% | 56% | 54% | 31%
bg | mBERT | 67% | 38% | 61% | 52% |  38%
| XLMR |  74% | 56% |  68% |  64% | 27%
Table 10: NLI task. Validation accuracy. We omitted training on Slovenian
since that dataset consists only of 306 items.

This task results support several observations:

  * •

this task seems to be significantly harder than others;

  * •

there is no consistent winner among the models for this task.

## 5\. Conclusions and discussion

In this project, we have made and published a broad index of NLP resources for
Eastern European languages, which, we hope, will be helpful for the NLP
community. We have invested in the creation of new cross-lingual datasets
focused on Eastern European languages, hand-crafted the benchmarks for 5
common reasoning tasks, and provided the evaluations of several modern
multilingual models. We have published all our code to support future
research.

As we have highlighted in our analysis, the quality of cross-lingual transfer
learning depends on various factors, including the pre-training and
distillation details of the model, the similarity between the languages, the
size of the dataset for the downstream task, etc. These observations are
consistent with the conclusions of previous studies (check [Doddapaneni et
al., 2021] for more details). The general observations from these experiments
are:

  * •

the multilingual pre-training is the key factor for successful cross-lingual
transfer learning;

  * •

the distillation significantly decreases the quality of cross-lingual transfer
learning;

  * •

the cross-lingual transfer quality seems to be correlated with the degree of
language affinity;

  * •

XLM-RoBERTa generally dominates over the rest of the tested models.

We are considering two major directions for future work in the scope of better
understanding cross-lingual abilities of the modern models in the context of
less-resourced Eastern European languages:

  * •

leveraging the synthetic datasets and data augmentation methods for supporting
the less-resourced languages;

  * •

and evaluation of the less general models (for example, models like
CroSloEngual[Ulcar and Robnik-Sikonja, 2020], BERTić [Ljubesic and Lauc,
2021], SlavicBERT [Arkhipov et al., 2019], or LitLat BERT).

## 6\. Acknowledgements

This work was initially started as an EEML 2021 summer school project. We
would like to thank all the organizers, especially Viorica Patraucean and
Razvan Pascanu for their valuable support. We also would like to thank our
mentor Tomáš Mikolov for wise advice and recommendations.

## 7\. Bibliographical References

## References

  * Arkhipov et al., 2019 Arkhipov, M., Trofimova, M., Kuratov, Y., and Sorokin, A.  (2019).  Tuning Multilingual Transformers for Language-Specific Named Entity Recognition.  In Proceedings of the 7th Workshop on Balto-Slavic Natural Language Processing, pages 89–93, Florence, Italy, August. Association for Computational Linguistics.
  * Artetxe and Schwenk, 2019 Artetxe, M. and Schwenk, H.  (2019).  Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond.  Transactions of the Association for Computational Linguistics, 7:597–610, 09.
  * Avetisyan and Ghukasyan, 2019 Avetisyan, K. and Ghukasyan, T.  (2019).  Word embeddings for the Armenian language: Intrinsic and extrinsic evaluation.  CoRR, abs/1906.03134.
  * Batanović et al., 2011 Batanović, V., Furlan, B., and Nikolić, B.  (2011).  A software system for determining the semantic similarity of short texts in Serbian.  In 2011 19thTelecommunications Forum (TELFOR) Proceedings of Papers, pages 1249–1252.
  * Biewald, 2020 Biewald, L.  (2020).  Experiment tracking with Weights and Biases.  Software available from wandb.com.
  * Butnaru and Ionescu, 2019 Butnaru, A. M. and Ionescu, R. T.  (2019).  MOROCO: The Moldavian and Romanian Dialectal Corpus.  CoRR, abs/1901.06543.
  * Bučar et al., 2018 Bučar, J., Žnidaršič, M., and Povh, J.  (2018).  Annotated news corpora and a lexicon for sentiment analysis in Slovene.  Language Resources and Evaluation, 52, 02.
  * Conneau et al., 2018 Conneau, A., Rinott, R., Lample, G., Williams, A., Bowman, S. R., Schwenk, H., and Stoyanov, V.  (2018).  XNLI: Evaluating cross-lingual sentence representations.  In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
  * Conneau et al., 2020 Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, É., Ott, M., Zettlemoyer, L., and Stoyanov, V.  (2020).  Unsupervised cross-lingual representation learning at scale.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–8451.
  * Devlin et al., 2018 Devlin, J., Chang, M., Lee, K., and Toutanova, K.  (2018).  BERT: pre-training of deep bidirectional transformers for language understanding.  CoRR, abs/1810.04805.
  * Doddapaneni et al., 2021 Doddapaneni, S., Ramesh, G., Khapra, M. M., Kunchukuttan, A., and Kumar, P.  (2021).  A primer on pretrained multilingual language models.
  * Dumitrescu et al., 2021 Dumitrescu, S. D., Rebeja, P., Lorincz, B., Gaman, M., Avram, A., Ilie, M., Pruteanu, A., Stan, A., Rosia, L., Iacobescu, C., Morogan, L., Dima, G., Marchidan, G., Rebedea, T., Chitez, M., Yogatama, D., Ruder, S., Ionescu, R. T., Pascanu, R., and Patraucean, V.  (2021).  LiRo: Benchmark and leaderboard for Romanian language tasks.
  * Emelin and Sennrich, 2021 Emelin, D. and Sennrich, R.  (2021).  Wino-X: Multilingual Winograd schemas for commonsense reasoning and coreference resolution.  In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8517–8532, Online and Punta Cana, Dominican Republic, November. Association for Computational Linguistics.
  * Feng et al., 2020 Feng, F., Yang, Y., Cer, D., Arivazhagan, N., and Wang, W.  (2020).  Language-agnostic BERT sentence embedding.
  * Habernal et al., 2013 Habernal, I., Ptáček, T., and Steinberger, J.  (2013).  Sentiment analysis in Czech social media using supervised machine learning.  In Proceedings of the 4th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 65–74, Atlanta, Georgia, June. Association for Computational Linguistics.
  * Hladek et al., 2014 Hladek, D., Stas, J., and Juhar, J.  (2014).  The Slovak categorized news corpus.  In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 1705–1708, Reykjavik, Iceland, May. European Language Resources Association (ELRA).
  * Joshi et al., 2020 Joshi, P., Santy, S., Budhiraja, A., Bali, K., and Choudhury, M.  (2020).  The state and fate of linguistic diversity and inclusion in the NLP world.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282–6293, Online, July. Association for Computational Linguistics.
  * Lauscher et al., 2020 Lauscher, A., Ravishankar, V., Vulić, I., and Glavaš, G.  (2020).  From Zero to Hero: On the limitations of zero-shot language transfer with multilingual Transformers.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4483–4499, Online, November. Association for Computational Linguistics.
  * Liang et al., 2020 Liang, Y., Duan, N., Gong, Y., Wu, N., Guo, F., Qi, W., Gong, M., Shou, L., Jiang, D., Cao, G., Fan, X., Zhang, B., Agrawal, R., Cui, E., Wei, S., Bharti, T., Qiao, Y., Chen, J., Wu, W., Liu, S., Yang, F., Majumder, R., and Zhou, M.  (2020).  XGLUE: A new benchmark dataset for cross-lingual pre-training, understanding and generation.  CoRR, abs/2004.01401.
  * Ljubesic and Lauc, 2021 Ljubesic, N. and Lauc, D.  (2021).  Bertić - The Transformer Language Model for Bosnian, Croatian, Montenegrin and Serbian.  CoRR, abs/2104.09243.
  * Malajyan et al., 2020 Malajyan, A., Avetisyan, K., and Ghukasyan, T.  (2020).  ARPA: Armenian paraphrase detection corpus and models.  pages 35–39, 09.
  * Nooralahzadeh et al., 2020 Nooralahzadeh, F., Bekoulis, G., Bjerva, J., and Augenstein, I.  (2020).  Zero-shot cross-lingual transfer with meta learning.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4547–4562, Online, November. Association for Computational Linguistics.
  * Pasini et al., 2021 Pasini, T., Raganato, A., and Navigli, R.  (2021).  XL-WSD: An extra-large and cross-lingual evaluation framework for word sense disambiguation.  In Proc. of AAAI.
  * Pelicon et al., 2020 Pelicon, A., Pranjić, M., Miljković, D., Škrlj, B., and Pollak, S.  (2020).  Zero-shot learning for cross-lingual news sentiment classification.  Applied Sciences, 10(17).
  * Piskorski et al., 2019 Piskorski, J., Laskova, L., Marcińczuk, M., Pivovarova, L., Přibáň, P., Steinberger, J., and Yangarber, R.  (2019).  The second cross-lingual challenge on recognition, normalization, classification, and linking of named entities across Slavic languages.  In Proceedings of the 7th Workshop on Balto-Slavic Natural Language Processing, pages 63–74, Florence, Italy, August. Association for Computational Linguistics.
  * Pollak et al., 2021 Pollak, S., Purver, M., Shekhar, R., Freienthal, L., Kuulmets, H.-A., and Krustok, I.  (2021).  Latvian Delfi article archive (in Latvian and Russian) 1.0.  Slovenian language resource repository CLARIN.SI.
  * Ponti et al., 2020 Ponti, E. M., Glavas, G., Majewska, O., Liu, Q., Vulic, I., and Korhonen, A.  (2020).  XCOPA: A multilingual dataset for causal commonsense reasoning.  CoRR, abs/2005.00333.
  * Purver et al., 2021 Purver, M., Pollak, S., Freienthal, L., Kuulmets, H.-A., Krustok, I., and Shekhar, R.  (2021).  Ekspress news article archive (in Estonian and Russian) 1.0.  Slovenian language resource repository CLARIN.SI.
  * Raganato et al., 2020 Raganato, A., Pasini, T., Camacho-Collados, J., and Pilehvar, M. T.  (2020).  XL-WiC: A multilingual benchmark for evaluating semantic contextualization.  CoRR, abs/2010.06478.
  * Reimers and Gurevych, 2019 Reimers, N. and Gurevych, I.  (2019).  Sentence-BERT: Sentence embeddings using siamese BERT-networks.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11.
  * Ruder et al., 2021 Ruder, S., Constant, N., Botha, J., Siddhant, A., Firat, O., Fu, J., Liu, P., Hu, J., Garrette, D., Neubig, G., and Johnson, M.  (2021).  XTREME-R: Towards more challenging and nuanced multilingual evaluation.  In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10215–10245, Online and Punta Cana, Dominican Republic, November. Association for Computational Linguistics.
  * Rybak et al., 2020 Rybak, P., Mroczkowski, R., Tracz, J., and Gawlik, I.  (2020).  KLEJ: Comprehensive benchmark for Polish language understanding.  CoRR, abs/2005.00630.
  * Sanh et al., 2019 Sanh, V., Debut, L., Chaumond, J., and Wolf, T.  (2019).  DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.  ArXiv, abs/1910.01108.
  * Scherrer, 2020 Scherrer, Y.  (2020).  TaPaCo: A corpus of sentential paraphrases for 73 languages.  In Proceedings of the 12th Language Resources and Evaluation Conference, pages 6868–6873, Marseille, France, May. European Language Resources Association.
  * Shavrina et al., 2020 Shavrina, T., Fenogenova, A., Emelyanov, A. A., Shevelev, D., Artemova, E., Malykh, V., Mikhailov, V., Tikhonova, M., Chertok, A., and Evlampiev, A.  (2020).  RussianSuperGLUE: A russian language understanding evaluation benchmark.  CoRR, abs/2010.15925.
  * Tikhonov and Ryabinin, 2021 Tikhonov, A. and Ryabinin, M.  (2021).  It’s all in the heads: Using attention heads as a baseline for cross-lingual transfer in commonsense reasoning.  CoRR, abs/2106.12066.
  * Ulcar and Robnik-Sikonja, 2020 Ulcar, M. and Robnik-Sikonja, M.  (2020).  FinEst BERT and CroSloEngual BERT: less is more in multilingual models.  CoRR, abs/2006.07890.
  * Wang et al., 2018 Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S.  (2018).  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355, Brussels, Belgium, November. Association for Computational Linguistics.
  * Wu and Dredze, 2020 Wu, S. and Dredze, M.  (2020).  Are all languages created equal in multilingual BERT?  In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 120–130, Online, July. Association for Computational Linguistics.
  * Yamshchikov et al., 2021 Yamshchikov, I. P., Shibaev, V., Khlebnikov, N., and Tikhonov, A.  (2021).  Style-transfer and paraphrase: Looking for a sensible semantic similarity metric.  Proceedings of the AAAI Conference on Artificial Intelligence, 35(16):14213–14220, May.
  * Yang et al., 2019 Yang, Y., Zhang, Y., Tar, C., and Baldridge, J.  (2019).  PAWS-X: A cross-lingual adversarial dataset for paraphrase identification.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3687–3692, Hong Kong, China, November. Association for Computational Linguistics.
  * Žagar et al., 2020 Žagar, A., Robnik-Šikonja, M., Goli, T., and Arhar Holdt, Š.  (2020).  Slovene translation of SuperGLUE.  Slovenian language resource repository CLARIN.SI.

lrec2022-bib