# Learned Construction Grammars Converge Across Registers
Given Increased Exposure

Jonathan Dunn
University of Canterbury
Christchurch, NZ
<EMAIL_ADDRESS>
Harish Tayyar Madabushi
University of Sheffield
Sheffield, UK
<EMAIL_ADDRESS>

###### Abstract

This paper measures the impact of increased exposure on whether learned
construction grammars converge onto shared representations when trained on
data from different registers. Register influences the frequency of
constructions, with some structures common in formal but not informal usage.
We expect that a grammar induction algorithm exposed to different registers
will acquire different constructions. To what degree does increased exposure
lead to the convergence of register-specific grammars? The experiments in this
paper simulate language learning in 12 languages (half Germanic and half
Romance) with corpora representing three registers (Twitter, Wikipedia, Web).
These simulations are repeated with increasing amounts of exposure, from 100k
to 2 million words, to measure the impact of exposure on the convergence of
grammars. The results show that increased exposure does lead to converging
grammars across all languages. In addition, a shared core of register-
universal constructions remains constant across increasing amounts of
exposure.

##  1 Exposure and Convergence

The central question that this work aims to answer is whether register-
specific grammars converge onto shared representations when exposed to more
training data. Variation in the context of production, called register
variation, has a significant impact on the frequency of constructions. For
example, imperative and wh-question constructions are much more frequent in
informal or conversational speech, while declarative constructions are much
more frequent in formal written usage Fodor and Crowther (2002); Sampson
(2002).

At the same time, usage-based grammar views language as a complex adaptive
system that emerges given exposure to usage Bybee (2006); Beckner et al.
(2009). Thus, a language learner is expected to be strongly influenced by the
observed frequency of constructions. Given this wide variance in the frequency
of constructions across registers, it is conceivable that learners exposed to
different registers learn different constructions.

This paper simulates the language acquisition process from a usage-based
perspective by learning Construction Grammars, called CxGs Goldberg (1995,
2006); Langacker (2008). A constructional approach to language focuses on
symbolic form-meaning mappings that are potentially idiomatic. Previous work
on computational CxG has explored how to discover potential constructions
Wible and Tsao (2010); Forsberg et al. (2014); Dunn (2017), the process of
construction learning Barak and Goldberg (2017); Barak et al. (2017), and
whether constructional information is implicitly encoded in models like BERT
Tayyar Madabushi et al. (2020).

A commonly discussed example of a construction is the ditransitive in (a1).
CxGs use a constraint-based formalism in which each slot in the construction
is defined by a particular slot-constraint; in (a1) these are syntactic
constraints. One of the important ideas in CxG is that constructions
themselves carry a meaning. For example, the ditransitive construction carries
a meaning of transfer regardless of the meaning of the particular verb that is
used in the ditransitive. In some cases, this notion of transfer also follows
from the meaning of the verb, as in (a2) with sold. But, in other cases,
utterances like (a3) can take on a meaning of transfer that is not present in
the verb smile. Constructions can also have idiomatic children, such as (a4),
in which an item-specific slot-constraint defines a sub-class of the
construction, like (a5), whose meaning is not entirely predictable or
transparent given the parent construction.

(a1) [ Syn: np – Syn: vp – Syn: np – Syn: np ]

(a2) “He sold them a car."

(a3) “He smiled himself an upgrade."

(a4) “He gave them a hand."

(a5) [ Syn: np – Lex: give – Syn: np – Lex: hand ]

Language | Code | Family
---|---|---
Danish | dan | Germanic
Dutch | nld | Germanic
English | eng | Germanic
German | deu | Germanic
Norwegian | nor | Germanic
Swedish | swe | Germanic
Catalan | cat | Romance
French | fra | Romance
Italian | ita | Romance
Portuguese | por | Romance
Romanian | ron | Romance
Spanish | spa | Romance
Table 1: Sources of Language Data, with 2 million words each for the tw, wk,
and cc registers

Register is a distinct pattern of usage that is associated with the context of
production. A substantial body of research has shown that register is a major
source of linguistic variation (Biber, 2012). Recent work has shown that the
impact of register variation exceeds the impact of geographic variation in
many cases (Dunn, 2021). The result of register variation is that large
corpora often contain a number of distinct sub-corpora, each with their own
unique patterns of usage (Sardinha, 2018; Cvrček et al., 2020). In other
words, a gigaword web-crawled corpus is not simply a flat collection of many
written documents: there is, instead, a register-based grouping of sub-corpora
which often contain significantly different linguistic forms.

This paper simulates the acquisition of constructions by incrementally
increasing the amount of exposure: 100k words, 200k words, 300k words and so
on up to 2 million words Alishahi and Stevenson (2008); Matusevych et al.
(2013); Beekhuizen et al. (2015). This provides a series of grammars, each
representing a different state in the learning process. This experiment is
repeated across three registers, each with a unique set of constructional
frequencies: Wikipedia (formal), Twitter (informal), and Web (mixed). Each
register has a progression of 20 register-specific grammars, with each grammar
representing different levels of exposure.

Is there a level of exposure at which register-specific grammars reach a
stable shared representation of the language? On the one hand, it is possible
that register-specific grammars are maintained as unique sub-sets of
linguistic behaviour. In this case, grammars would not converge given
increased exposure. On the other had, it is possible that register-specific
constructions fade away as increased exposure leads to more generalized
grammars. In this case, grammars would converge, becoming more similar and
less register-specific as they are learned through exposure to more training
data.

To avoid language-specific generalizations, this experiment is repeated across
six Germanic languages (Danish, Dutch, English, German, Norwegian, Swedish)
and six Romance languages (Catalan, French, Italian, Portuguese, Romanian,
Spanish). Each grammar contains a set of constructions that have been learned
to best represent the training data. Thus, for each stage in the learning
process, we can measure the overlap between register-specific grammars
(Twitter to Web, Twitter to Wikipedia, and Wikipedia to Web). When register-
specific grammars have a higher overlap this means that they share more of
their constructional representations. In other words, higher overlap means
that the grammars are more similar.

We say that grammars converge when they have a higher degree of overlap or
similarity. This paper develops two measures of grammar similarity to capture
different aspects of convergence: a fuzzy Jaccard similarity that captures
convergence across even rare constructions and a frequency-weighted Jaccard
similarity that focuses on the core constructions. These two measures of
convergence allow us to model the degree to which construction grammars learn
register-specific representations.

| Construction (Type) |  | Construction (Type)
---|---|---|---
(b) | [ Syn: n \- Lex: of \- Syn: det \- Sem:<587> ] | (c) | [ Lex: while \- Sem:<113> \- Syn: adp ]
| Examples (Tokens) |  | Examples (Tokens)
(b1) | ‘spirit of the alchemist’ | (c1) | ‘while working out’
(b2) | ‘provinces of the empire’ | (c2) | ‘while sitting by’
(b3) | ‘raft of the medusa’ | (c3) | ‘while going through’
(b4) | ‘constellations of the zodiac’ | (c4) | ‘while carrying out’
(b5) | ‘myth of the anaconda’ | (c5) | ‘while sticking around’
Table 2: Examples of Constructions (Types) and Instances of Constructions
(Tokens) for English

There are three possible outcomes for this experimental framework: First, it
is possible that grammars converge as exposure increases. This convergence
would indicate that register variation becomes less important given more data:
the grammars contain the same constructions regardless of the input register.
In other words, this would indicate that more data overall is able to
compensate for register variation. Second, it is possible that grammars do not
converge as the amount of exposure increases. This outcome would indicate that
each register represents a unique sub-grammar, a distinct set of linguistic
behaviours. In this case, more data overall would never compensate for
register variation. These two competing hypotheses are tested in Experiment 1
(Section 5). This experiment finds that increased exposure does, in fact, lead
to converging grammars. This finding supports the first hypothesis that is
described above.

Third, another possible outcome is that not all languages pattern together. In
other words, it may be the case that some Germanic languages converge given
increased exposure but some Romance languages retain register-specific
constructions. Part of the experimental design is to repeat the same framework
across many languages to determine whether the outcome is specific to one or
another language. We will see, in Experiment 1, that there is variation across
languages in both (i) the rate of convergence and (ii) the upper limit when
grammars stop converging. However, it remains the case that all languages show
increased convergence given increased exposure.

Given that the first experiments show that grammars do converge given
increased exposure, we undertake an additional experiment: is there a core
construction grammar for each language? Construction grammars have a proto-
type structure, meaning that some representations are central (and thus very
frequent) while others are peripheral (and thus somewhat rare). We use the
frequency-weighted Jaccard similarity in Experiment 2 (Section 6) to determine
whether the overall rate of convergence changes when we focus on the core of
the grammar rather than the periphery. These experiments show that for most
languages the core grammar is acquired very early with little change in
convergence given increased exposure.

##  2 Experimental Design

The basic experimental approach is to learn grammars over increasing amounts
of exposure: from 100k words to 2 million words in increments of 100k words
(thus creating 20 grammars per condition). This series of grammars simulates
the accumulation of grammatical knowledge as the amount of exposure increases.
This approach is repeated across each of the three registers that represent
different contexts of production.

The register-specific data used to progressively learn grammars is collected
from three sets of corpora: tweets (tw), Wikipedia articles (wk), and web
pages (cc for Common Crawl). This dataset is summarized in Table 1. The corpus
contains the same amount of data per register per language Dunn (2020); Dunn
and Adams (2020).

The pairwise similarity relationships between grammars differ, in part,
because some sources of data are more similar to one another. For example, cc
and wk are more similar registers and thus their grammars have a baseline
similarity that is higher than wk and tw. In other words, the Wikipedia
grammars (CxG-wk) are more similar to the web grammars (CxG-cc) than to the
Twitter grammars (CxG-tw). What matters, then, is the degree to which the
relative similarity between grammars changes as the amount of exposure
increases. This approach controls for the underlying similarity between
registers.

##  3 Learning Constructions

The grammar induction algorithm used to learn constructions is taken from
previous work Dunn (2017). At its core, this model of CxGs has three main
components: First, a psychologically-plausible measure of association, the
$\Delta P$, is used to measure the entrenchment of particular representations
Ellis (2007); Gries (2013); Dunn (2018c). Second, an association-based beam
search is used to identify constructions of arbitrary length by finding the
most entrenched representation for each training sentence in reference to a
matrix of $\Delta P$ values Dunn (2019a). Third, a Minimum Description Length
measure is used as an optimization function, balancing the trade-off between
increased storage of item-specific constructions and increased computation of
generalized constructions Dunn (2018b). We briefly review each of these
components of the algorithm in this section.

Constructions are constraint-based syntactic representations in which
individual slots are limited to particular syntactic, semantic, or lexical
items Goldberg (1995). Construction Grammars are usage-based in the sense that
constructions range from very idiomatic phrases (like “give me a hand") to
very abstract sequences (like np -> det adj np). One of the many advantages of
CxGs is that they represent actual usage, which means that they are capable of
identifying syntactic variation across dialects Dunn (2018a, 2019c, 2019b) and
even across individuals Dunn and Nini (2021). But the disadvantage is that the
induction algorithm must learn both the units of representation (i.e.,
semantic categories) as well as these multi-dimensional slot-constraints. For
example, the constructions in (b) and (c) in Table 2 include all three types
of representation (lexical, syntactic, semantic) so that the algorithm must be
able to navigate across these three representations during the learning
process.

The grammar induction algorithm starts by defining the types of
representation. lexical constraints are based on word-forms, without
lemmatization. These are the simplest and most idiomatic types of constraints.
syntactic constraints are formulating using the universal part-of-speech
tagset Petrov et al. (2012) and implemented using the Ripple Down Rules
algorithm Nguyen et al. (2016). semantic constraints are based on
distributional semantics, with k-means clustering applied to discretize pre-
trained fastText embeddings Grave et al. (2019). The semantic constraints in
(b) and (c) are formulated using the index of the corresponding clusters. A
complete list of semantic domains used in this paper, along with a grammar for
English, are available in the supplementary material.

Each sentence in the input corpus is transformed into these three parallel
dimensions of representation (lexical, syntactic, semantic). In the first
stage of the algorithm, a co-occurrence matrix is produced that represents the
association between all pairs of representations using the $\Delta P$ measure,
shown below. What distinguishes the $\Delta P$ from more common measures like
PPMI Church and Hanks (1989); Dagan et al. (1993) is that it has direction-
specific variants that take ordering into account, thus helping to capture
syntactic patterns. The measure, calculated left-to-right, is the probability
that two units occur together ($X$ and $Y$) adjusted by the probability that
$X$ occurs alone. In this notation, $Y_{P}$ indicates that unit $Y$ is present
and $Y_{A}$ that unit $Y$ is absent.

$\Delta P_{LR}=p(X_{P}|Y_{P})-p(X_{P}|Y_{A})$ (1)

Given (i) a three-dimensional representation of each sentence and (ii) a co-
occurrence matrix with the directional association for each pair of
representations, a beam search algorithm is used to find the most entrenched
sequence of constraints for each sentence in the training corpus. The basic
idea behind this search is to traverse all possible paths of constraints,
ending each path when the cumulative $\Delta P$ falls below a threshold Dunn
(2019a). For each sentence, the sequence of slot-constraints with the highest
cumulative association is added to a provisional grammar. In CxG, some
representations are very entrenched (grammaticalized) and others are only
slightly entrenched Goldberg (2011, 2016). The optimum sub-set of
constructions is then selected from this provisional grammar.

The grammar induction model itself is based on the Minimum Description Length
paradigm Grünwald and Rissanen (2007); Goldsmith (2001, 2006). In this kind of
model, observed probabilities are used to calculate the encoding size of a
grammar ($L_{1}$) as well as the encoding size of a test corpus given that
grammar ($L_{2}$). Usage-based grammar posits a trade-off between memory and
computation; this is modelled by MDL’s combination of $L_{1}$ and $L_{2}$
encoding size Dunn (2018b).

The best grammar is the one which minimizes this metric on a test corpus. In
(2), $G$ refers to the grammar being evaluated and $D$ refers to the test
corpus. For example, this is used to choose the parameters of the beam search
algorithm described above. In practice, the use of MDL to evaluate grammars is
quite similar to the use of perplexity to evaluate language models; for
example, the MDL metric is specific to each test corpus. The advantage of the
MDL metric for usage-based grammar is that it distinguishes between the
complexity of the grammar ($L_{1}$) and the fit between the grammar and the
test corpus ($L_{2}$).

$MDL=\min\limits_{G}\\{{L_{1}(G)+L_{2}(D\mid G)}\\}$ (2)

This induction algorithm provides a grammar of constructions to describe the
training corpus, where the grammar is chosen to minimize the MDL metric. The
grammar is a set of constructions, each of which is a sequence of slot-
constraints. And each slot-constraint is formulated using the basic inventory
of lexical, syntactic, and semantic fillers.

In previous work, the induction algorithm used alternating training and
testing sets to refine grammars. A large background corpus was used to
estimate the $\Delta P$ matrix that guides the selection of slot-constraints.
The experiments here, however, depend on limiting the amount of training data
as a means of controlling for different levels of exposure. In each condition,
then, the same training data is used for each stage in the algorithm. In other
words, the 200k word exposure condition has access only to the 200k word
training corpus (with the implicit exception of the pre-trained embeddings).
Each model is trained on the same underlying corpus, so that the 500k word
condition is given the same data as the 400k word condition plus an additional
100k words of new data.

Figure 1: Overlap between pairs of register-specific grammars by amount of
exposure, using Fuzzy Jaccard Similarity for Germanic languages to measure
Constructional Overlap (Experiment 1). Each line represents the similarity
between two grammars learned from different registers. Figure 2: Overlap
between pairs of register-specific grammars by amount of exposure, using Fuzzy
Jaccard Similarity for Romance languages to measure Constructional Overlap
(Experiment 1). Each line represents the similarity between two grammars
learned from different registers.

##  4 Measuring Grammar Similarity

A grammar in this context is a set of constructions, where each construction
is a sequence of slot-constraints. Our central measure of overlap between
grammars, then, is based on the Jaccard similarity, where values close to 1
represent very similar grammars and values close to 0 represent very different
grammars.

$J(A,B)=\frac{|A\cap B|}{|A\cup B|}$ (3)

One of the challenges with CxGs is that two different representations could
use different slot-constraints to capture a similar set of utterances,
essentially providing two versions of the same construction. Consider the
construction in (d), a part of the English web-based grammar. The tokens in
(d1) through (d4) are tokens of this construction. An alternate constraint
that specifies the rather than det might be chosen, leading to a different
representation for the same underlying construction.

(d) [ Lex: how to – Syn: v – Syn: det – Syn: n ]

(d1) ‘how to get the job’

(d2) ‘how to track a phone’

(d3) ‘how to improve the system’

(d4) ‘how to start a blog’

The challenge for calculating convergence using the Jaccard similarity between
grammars is that similar constructions could capture the same set of tokens.
For example, the syntactic constraint (det) could be replaced with a lexical
constraint (the) in the construction in (d). The Jaccard similarity on its own
would not capture the similarity between these two alternate formulations of
what is ultimately the same underlying construction.

Our first measure is thus a fuzzy jaccard similarity, in which the definition
of set membership is extended to very similar constructions. In this measure,
a sub-sequence matching algorithm is used to find how many slot-constraints
are shared between two constructions, taking order into account. Any two
constructions above a threshold of 0.71 shared sub-sequences are considered a
match. This threshold is chosen because it allows one slot-constraint to
differ between most constructions while still considering them to be similar
representations. For example, two six-slot constructions must share five
constraints in order to count as a match at this threshold. This measure thus
provides a better approximation of construction similarity, focusing on
constructions with slightly different internal constraints or with an added
slot-constraint in one position.

Our second measure is a frequency-weighted jaccard similarity, in which the
importance of each construction is weighted relative to its frequency in an
independent corpus. For each language, a background corpus is created from a
mix of different registers: Open Subtitles and Global Voices (news articles)
Tiedemann (2012) and Bible translations Christodoulopoulos and Steedman
(2015). This background corpus represents usage external to the three main
registers used in the experiments.

The frequency of each construction is derived from 500k words of this
background corpus, so that very common constructions are given more weight in
the similarity measure. The basic idea is that some constructions are part of
the core grammar, thus being frequent in all registers. This weighted measure
captures convergence within this core grammar by focusing on those
constructions which are most frequent in an independent corpus.

These two measures based on Jaccard similarity provide three values for each
condition: cc-wk, cc-tw, and wk-tw. Each of the values represents a pairwise
similarity between two register-specific grammars. The higher these values,
the more the learner is converging onto a shared grammar.

Figure 3: Overlap between pairs of register-specific grammars by amount of
exposure, using Weighted Jaccard Similarity for Germanic languages to measure
Core Grammatical Overlap (Experiment 2). Each line represents the similarity
between two grammars learned from different registers. Figure 4: Overlap
between pairs of register-specific grammars by amount of exposure, using
Weighted Jaccard Similarity for Romance languages to measure Core Grammatical
Overlap (Experiment 2). Each line represents the similarity between two
grammars learned from different registers.

We have thus formulated two measures of grammatical overlap. The first, fuzzy
Jaccard, captures the overall similarity between grammars. The second,
frequency-weighted Jaccard, captures the similarity between grammars with a
focus on the core constructions that are frequent across registers (ignoring
the long tail of register-specific forms). The following sections apply these
measures of grammatical overlap to the CxGs exposed to increasing amounts of
usage. In each case, higher values represent increased convergence between
grammars.

##  5 Experiment 1: Constructional Overlap

To what degree do grammars converge onto shared representations as they are
exposed to increasing amounts of data from different registers? The first
experiment uses the fuzzy Jaccard similarity to model convergence. The results
are shown in Figure 1 and Figure 2. In both cases, the y axis represents the
similarity between register-specific grammars, with higher values representing
more convergent sets of constructions. Note that each line represents the
similarity between two register-specific grammars. And the x axis represents
the amount of exposure, moving from 100k words up to 2 million words.
Languages are labelled using their language code, as listed in Table 1.

We notice, first, that there is a baseline difference in the similarity
between registers. In every language, for example, wk and tw are the least
similar. And, in every language, cc and wk are the most similar. This pattern
is shared across all 12 languages because the underlying contexts of
production have this similarity, with Wikipedia the most formal and Twitter
the least formal. The distance between registers does not matter here. What
does matter is the relative change in distance as the algorithm is exposed to
more data (i.e., cc-tw at 200k words and cc-tw at 2 million words).

We notice, second, that in every language grammars converge with increased
exposure. The overall similarity between constructions increases as more
training data is observed. This is true for both Germanic and Romance
languages, showing this to be a robust generalization.

Languages do differ, however, in (i) the overall amount of similarity and (ii)
the rate of convergence. In the first case, within Germanic languages the
range of grammar similarity is generally comparable, starting at approximately
0.2 and ending at approximately 0.4. Some languages (like English) reach a
higher level of convergence. Other languages have register-specific patterns:
in Norwegian the similarity between cc-wk is quite high throughout, but in
Swedish the similarity between cc-wk is the same as the similarity between cc-
tw. In other words, the ordering of register similarity is constant across
languages but the distance is not. Romance languages have a generally higher
overall rate of similarity, although there is a wide gap between French (the
highest) and Romanian (the lowest). This means that there is some variation
across languages in terms of how similar the register-specific grammars
become.

There is also variation in the rate of convergence. Some languages (like
Swedish) have a somewhat flat rate of convergence while other languages (like
English) have a steeper curve. A flat curve represents a slow convergence
while a steep curve represents a rapid convergence as exposure increases. Some
languages have bursts of convergence at specific amounts of exposure. For
example, Norwegian has a steep increase until about 300k words, then remains
flat until about 1 million words before beginning to converge again. In other
words, if we think about the growth curve of similarity, the pattern of
convergence differs across languages. The cause of these differences is a
matter for future research. The basic conclusion here is that all languages
show increasing convergence with increasing exposure. In other words,
register-specific syntactic patterns generalize as the induction algorithm
encounters more training data.

##  6 Experiment 2: The Core Constructions

Although many constructions have varying frequency across different registers,
we would expect that the grammar of each language also has a core set of very
frequent constructions which are shared across all registers. In other words,
register variation itself should not be strong enough to erase the syntactic
generalizations provided by a grammar. The frequency-weighted Jaccard
similarity measure is used to find this core set of constructions: how much do
grammars change with increasing exposure when we focus on the most frequent
constructions? As explained above, the frequency weighting is derived from
independent corpora that represent a different set of registers.

The weighted similarity measures are shown in Figure 3 and Figure 4, again
with the y axis showing similarity (higher values mean more overlap between
grammars) and the x axis showing increasing amounts of exposure. Each line
represents the similarity between two register-specific grammars. The overall
level of similarity is significantly higher here. The Germanic languages range
from approximately 0.7 (Dutch) to approximately 0.8 (Swedish). The Romance
languages have the same range, with Spanish the lowest and Portuguese the
highest. This overall increase in similarity shows that, when focusing on the
core constructions, the register-specific grammars converge quickly.

We notice, second, that the growth curve for the frequency-weighted measure
shows very little change after a certain point. For many languages, like
Portuguese and Italian, there is a sharp increase after several hundred
thousand words of exposure. This indicates that the initial grammars (based on
low exposure) are not adequate. After more exposure, however, the stable core
of constructions is acquired. Once that initial burst of acquisition is
complete, there are no significant changes. This is not true for Norwegian
(which continues to show a continuous growth of convergence) or for Romanian
(which has an initial decline and a much later burst of similarity). But the
overall pattern across all languages is that the core set of constructions
remains stable after a small amount of exposure.

##  7 Conclusions

These experiments show that register-specific grammars converge onto shared
constructions as they are exposed to more training data. This is observed
across 12 languages and three registers. At the same time, each language has a
core set of frequent constructions which is not influenced by register
variation. This core CxG is acquired for each language given a limited amount
of exposure and does not change significantly as exposure increases. These
results are important for describing the interaction between syntactic
generalizations (the core grammars) and syntactic variation (the register-
specific grammars).

## References

  * Alishahi and Stevenson (2008) A. Alishahi and S. Stevenson. 2008.  A Computational Model of Early Argument Structure Acquisition.  _Cognitive Science_ , 32(5):789–834.
  * Barak and Goldberg (2017) L. Barak and A Goldberg. 2017.  Modeling the Partial Productivity of Constructions.  In _Proceedings of AAAI 2017 Spring Symposium on Computational Construction Grammar and Natural Language Understanding_ , pages 131–138. Association for the Advancement of Artificial Intelligence.
  * Barak et al. (2017) L. Barak, A. Goldberg, and S. Stevenson. 2017.  Comparing Computational Cognitive Models of Generalization in a Language Acquisition Task.  In _Proceedings of the Conference on Empirical Methods in NLP_ , pages 96–106. Association for Computational Linguistics.
  * Beckner et al. (2009) C. Beckner, N. Ellis, R. Blythe, J. Holland, J. Bybee, J. Ke, M. Christiansen, D. Larsen-Freeman, W. Croft, and T. Schoenemann. 2009.  Language Is a Complex Adaptive System: Position Paper.  _Language Learning_ , 59:1–26.
  * Beekhuizen et al. (2015) B. Beekhuizen, R. Bod, A. Fazly, S. Stevenson, and A. Verhagen. 2015.  A Usage-Based Model of Early Grammatical Development.  In _Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics_ , pages 46–54. Association for Computational Linguistics.
  * Biber (2012) D. Biber. 2012.  Register as a predictor of linguistic variation. _Corpus Linguistics and Linguistic Theory_ , 8(1):9–37.
  * Bybee (2006) J. Bybee. 2006.  From Usage to Grammar: The mind’s response to repetition.  _Language_ , 82(4):711–733.
  * Christodoulopoulos and Steedman (2015) C. Christodoulopoulos and M. Steedman. 2015.  A massively parallel corpus: The Bible in 100 languages.  _Language Resources and Evaluation_ , 49:375–395.
  * Church and Hanks (1989) K. Church and P. Hanks. 1989.  Word association norms, mutual information, and lexicography.  In _27th Annual Meeting of the Association for Computational Linguistics_ , pages 76–83. Association for Computational Linguistics.
  * Cvrček et al. (2020) V. Cvrček, Z. Komrsková, D. Lukeš, P. Poukarová, A. Řehořková, A. Zasina, and V. Benko. 2020.  Comparing web-crawled and traditional corpora.  _Language Resources and Evaluation_ , 54(3):713–745.
  * Dagan et al. (1993) I. Dagan, S. Marcus, and S. Markovitch. 1993.  Contextual word similarity and estimation from sparse data.  In _31st Annual Meeting of the Association for Computational Linguistics_ , pages 164–171. Association for Computational Linguistics.
  * Dunn (2017) J. Dunn. 2017.  Computational learning of construction grammars.  _Language & Cognition_, 9(2):254–292.
  * Dunn (2018a) J. Dunn. 2018a.  Finding Variants for Construction-Based Dialectometry: A Corpus-Based Approach to Regional CxGs.  _Cognitive Linguistics_ , 29(2):275–311.
  * Dunn (2018b) J. Dunn. 2018b.  Modeling the Complexity and Descriptive Adequacy of Construction Grammars.  In _Proceedings of the Society for Computation in Linguistics_ , pages 81–90. Association for Computational Linguistics.
  * Dunn (2018c) J. Dunn. 2018c.  Multi-unit association measures: Moving beyond pairs of words.  _International Journal of Corpus Linguistics_ , 23:183–215.
  * Dunn (2019a) J. Dunn. 2019a.  Frequency vs. Association for Constraint Selection in Usage-Based Construction Grammar.  In _Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics_. Association for Computational Linguistics.
  * Dunn (2019b) J. Dunn. 2019b.  Global Syntactic Variation in Seven Languages: Toward a Computational Dialectology.  _Frontiers in Artificial Intelligence_ , 2:15.
  * Dunn (2019c) J. Dunn. 2019c.  Modeling Global Syntactic Variation in English Using Dialect Classification.  In _Proceedings of Workshop on NLP for Similar Languages, Varieties and Dialects_ , pages 42–53. Association for Computational Linguistics.
  * Dunn (2020) J. Dunn. 2020.  Mapping languages: The Corpus of Global Language Use.  _Language Resources and Evaluation_ , 54:999–1018.
  * Dunn (2021) J. Dunn. 2021.  Representations of language varieties are reliable given corpus similarity measures.  In _Proceedings of the Eighth Workshop on NLP for Similar Languages, Varieties and Dialects_ , pages 28–38. Association for Computational Linguistics.
  * Dunn and Adams (2020) J. Dunn and B. Adams. 2020.  Geographically-balanced gigaword corpora for 50 language varieties.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 2528–2536. European Language Resources Association.
  * Dunn and Nini (2021) J. Dunn and A. Nini. 2021.  Production vs Perception: The Role of Individuality in Usage-Based Grammar Induction.  In _Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics_ , pages 149–159. Association for Computational Linguistics.
  * Ellis (2007) N. Ellis. 2007.  Language Acquisition as Rational Contingency Learning.  _Applied Linguistics_ , 27(1):1–24.
  * Fodor and Crowther (2002) J. Fodor and C. Crowther. 2002.  Understanding Stimulus Poverty Arguments.  _The Linguistic Review_ , 19(1-2):105–145.
  * Forsberg et al. (2014) M. Forsberg, R. Johansson, L. Bckstrm, L. Borin, B. Lyngfelt, J. Olofsson, and J. Prentice. 2014.  From Construction Candidates to Constructicon Entries: An experiment using semi-automatic methods for identifying constructions in corpora.  _Constructions and Frames_ , 6(1):114–135.
  * Goldberg (1995) A. Goldberg. 1995.  _Constructions: A Construction Grammar Approach to Argument Structure_.  Cognitive Theory of Language and Culture Series. University of Chicago Press.
  * Goldberg (2006) A. Goldberg. 2006.  _Constructions at work: The nature of generalization in language_.  Oxford University Press, Oxford.
  * Goldberg (2011) A. Goldberg. 2011.  Corpus evidence of the viability of statistical preemption.  _Cognitive Linguistics_ , 22(1):131–154.
  * Goldberg (2016) A. Goldberg. 2016.  Partial Productivity of Linguistic Constructions: Dynamic categorization and Statistical preemption.  _Language & Cognition_, 8(3):369–390.
  * Goldsmith (2001) J. Goldsmith. 2001.  Unsupervised Learning of the Morphology of a Natural Language.  _Computational Linguistics_ , 27(2):153–198.
  * Goldsmith (2006) J. Goldsmith. 2006.  An Algorithm for the Unsupervised Learning of Morphology.  _Natural Language Engineering_ , 12(4):353–371.
  * Grave et al. (2019) E. Grave, P. Bojanowski, P. Gupta, A. Joulin, and T. Mikolov. 2019.  Learning word vectors for 157 languages.  In _International Conference on Language Resources and Evaluation_ , pages 3483–3487. European Language Resources Association.
  * Gries (2013) St. Th Gries. 2013.  50-something years of work on collocations: What is or should be next.  _International Journal of Corpus Linguistics_ , 18(1):137–165.
  * Grünwald and Rissanen (2007) P. Grünwald and J. Rissanen. 2007.  _The Minimum Description Length Principle_.  MIT Press.
  * Langacker (2008) R. Langacker. 2008.  _Cognitive Grammar A basic introduction_.  Oxford University Press, Oxford.
  * Matusevych et al. (2013) Y. Matusevych, A. Alishahi, and A. Backus. 2013.  Computational Simulations of Second Language Construction Learning.  In _Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics_ , pages 47–56. Association for Computational Linguistics.
  * Nguyen et al. (2016) Dat Quoca Dai Quocb Nguyen, Dat Quoca Dai Quocb Nguyen, Dang Ducc Pham, and Son Baod Pham. 2016.  A robust transformation-based learning approach using ripple down rules for part-of-speech tagging.  _AI Communications_ , 29(3):409–422.
  * Petrov et al. (2012) S. Petrov, D. Das, and R. McDonald. 2012.  A universal part-of-speech tagset.  In _Proceedings of the Eighth Conference on Language Resources and Evaluation_ , pages 2089–2096. European Language Resources Association.
  * Sampson (2002) G. Sampson. 2002.  Exploring the richness of the stimulus.  _The Linguistic Review_ , 19(1-2):73–104.
  * Sardinha (2018) T. Sardinha. 2018.  Dimensions of variation across Internet registers.  _International Journal of Corpus Linguistics_ , 23(2):125–157.
  * Tayyar Madabushi et al. (2020) H. Tayyar Madabushi, L. Romain, D. Divjak, and P. Milin. 2020.  CxGBERT: BERT meets construction grammar.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 4020–4032. International Committee on Computational Linguistics.
  * Tiedemann (2012) J. Tiedemann. 2012.  Parallel Data, Tools and Interfaces in OPUS.  In _Proceedings of the International Conference on Language Resources and Evaluation_. European Language Resources Association.
  * Wible and Tsao (2010) D. Wible and N. Tsao. 2010.  StringNet as a Computational Resource for Discovering and Investigating Linguistic Constructions.  In _Proceedings of the Workshop on Extracting and Using Constructions in Computational Linguistics_ , pages 25–31. Association for Computational Linguistics.


# Vikhr: The Family of Open-Source Instruction-Tuned Large Language Models for
Russian

Aleksandr Nikolich
ITMO University
<EMAIL_ADDRESS>
&Konstantin Korolev
HSE University
<EMAIL_ADDRESS>
&Artem Shelmanov
MBZUAI
<EMAIL_ADDRESS>

###### Abstract

There has been a surge in the development of various Large Language Models
(LLMs). However, text generation for languages other than English often faces
significant challenges, including poor generation quality and the reduced
computational performance due to the disproportionate representation of tokens
in model’s vocabulary. In this work, we address these issues and introduce
Vikhr, a new state-of-the-art open-source instruction-tuned LLM designed
specifically for the Russian language. “Vikhr” refers to the name of the
Mistral LLM series and means “strong gust of wind.” Unlike previous efforts
for Russian that utilize computationally inexpensive LoRA adapters on top of
English-oriented models, Vikhr features an adapted tokenizer vocabulary and
undergoes the continued pre-training and instruction tuning of all weights.
This approach not only enhances the model’s performance but also significantly
improves its computational and contextual efficiency. The remarkable
performance of Vikhr across various Russian-language benchmarks can also be
attributed to our efforts in expanding instruction datasets and corpora for
continued pre-training. Vikhr not only sets the new state of the art among
open-source LLMs for Russian, but even outperforms some proprietary closed-
source models on certain benchmarks. The model weights, instruction sets, and
code are publicly available111https://huggingface.co/Vikhrmodels.

##  1 Introduction

Instruction tuning has unlocked in Large Language Models (LLMs) vast zero-shot
capabilities without the need of careful prompt engineering Ouyang et al.
(2022). The most rapid research and development efforts are currently devoted
to English LLMs. There has been a surge in English open-source models: Llama
series Touvron et al. (2023a, b), Mistral series Jiang et al. (2023), Vicuna
series Chiang et al. (2023), etc. This growth is driven by the abundance of
raw training data in English and dedicated efforts to create comprehensive
sets of instruction-output pairs. Despite the fact that LLMs oriented on
English have some multilingual capabilities Zhao et al. (2024) due to small
portions of texts in various languages leaked into their training datasets
Touvron et al. (2023a), their overall performance in these languages remains
relatively low. Although they can usually generate portions of coherent texts,
these models struggle with reasoning in non-English languages, lack culture-
specific knowledge, and are highly inefficient in terms of tokenization. This
inefficiency arises due to the way bite-pair tokenization algorithms work:
they split the infrequent words into multiple tokens. Since multilingual data
typically represents a small portion of the training dataset, non-English
words are often split in many pieces. This leads to more steps during prompt
processing and text generation, shorter effective context windows, and
ultimately lower quality Tikhomirov and Chernyshev (2023); Petrov et al.
(2024). This disparity places non-English languages at a disadvantage.

There is a research direction focused on developing multilingual LLMs that
work well for multiple popular languages: BLOOMz Muennighoff et al. (2023),
mGPT Shliazhko et al. (2022), Bactrian-X Li et al. (2023), PALO Maaz et al.
(2024), Aya101 from CohereAI Üstün et al. (2024), etc. These models are
typically trained on rich multilingual datasets and are less skewed towards
English. However, when aiming to perform well across multiple languages
simultaneously, these models must still share their vocabulary and parameters.
This often hinders their performance for each particular language in
isolation, especially for the popular smaller model sizes, such as 7B and 13B.

The goal of maximizing the LLM performance for a specific language within a
certain number of parameters has led researchers to develop bi-lingual LLMs.
For example, Jais Sengupta et al. (2023) focus only on English and Arabic. The
inclusion of English data in pre-training alongside Arabic data is motivated
by the significantly larger volume of English data available. This helps LLMs
substantially enhance skills such as logical and common sense reasoning, which
are also applied when generating text in Arabic.

Russian is one of the high-resource languages and is typically represented in
multilingual LLMs. Additionally, there are several proprietary closed-source
LLMs, such as MTS AI, GigaChat, and YandexGPT, that meet or even surpass their
English-oriented flagship competitors when it comes to text processing and
generation in Russian. However, controllable research often requires white-box
access to LLM logits and layer outputs, the ability to modify weights and a
model architecture, and consistent answers for reproducibility, which is often
impossible in closed-source LLMs due to their constant development and
retirement. There are only a few open-source LLMs designed for Russian (such
as Saiga Gusev (2023), ruGPT AI Forever (2022), ruadapt Tikhomirov and
Chernyshev (2023)). Of these, only Saiga and ruadapt are instruction-tuned.

This work aims to build an efficient and effective open-source instruction-
following LLM for Russian facilitating multilingual natural language
processing research. Building even a small LLM that targets a particular
language from scratch requires a lot of computational resources. Consequently,
many researchers simply fine-tune LoRA adapters Hu et al. (2021) for English-
oriented LLMs on some language-specific data. While this approach can improve
model generation quality, it does not address computational inefficiency
because the tokenizer and model vocabulary remain unchanged. In contrast, our
approach not only fine-tunes a base LLM on Russian language data but also
reconstructs its underlying tokenizer and vocabulary, alongside suggesting an
improved method for continued pre-training. Additionally, we have
significantly expanded the available Russian datasets for instruction tuning.
The developed LLM achieves state-of-the-art results for the Russian language
among other open-source counterparts across a wide range of benchmarks.

Contributions of the paper are the following:

  * •

We have constructed Vikhr – a state-of-the-art open-source instruction-
following LLM oriented on the Russian language. In addition to its high
generation quality, Vikhr features an efficient tokenizer that enables rapid
text generation and good context utilization.

  * •

We have developed a pipeline for adapting English-oriented LLMs to the Russian
language. The pipeline implements vocabulary adaptation, continued pre-
training with regularization to prevent “catastrophic forgetting”, and
instruction tuning.

  * •

We have expanded the datasets for continued pre-training of Russian language
models and previously available instruction datasets.

  * •

We conducted an extensive evaluation of several open-source LLMs on evaluation
benchmarks for Russian, demonstrating that Vikhr achieves new state-of-the-art
results.

##  2 Related Work

One of the first notable series of generative LLMs for Russian is ruGPT AI
Forever (2022); Zmitrovich et al. (2023). The authors created several models
trained for the vanilla language modelling task with the sizes of up to 13b.
The models were created from the scratch and trained on large Russian corpora.
They are able to handle the linguistic nuances of Russian more effectively
than multilingual models Muennighoff et al. (2022). Since the training data
was mostly in Russian, these models have efficient tokenization, but the lack
of multilingual data (e.g. in English) limits their performance. ruGPT models
are not instruction tuned.

Gusev (2023) suggests to leverage reasoning capabilities of existing English-
oriented LLMs and adapt them to the Russian language by training LoRA
adapters. They also create an Alpaca-like set of Russian instruction-output
pairs and performed instruction tuning. They have established the Saiga model
series, which has a competitive performance and used to be a reasonable choice
for off-the-shelf open-source Russian LLM for the past year. However, the
tokenizer in theses models is not adapted, so they experience issues with
context and computational efficiency.

Tikhomirov and Chernyshev (2023) address these issues in Saiga. In addition to
model tuning on Russian data, they also adapt the model tokenizer. They note
that improving tokenization helps to both improve the efficiency of the model
and its performance while reducing memory consumption. However, during
continued pre-training, the authors freeze the model weights except LM heads
and token embeddings, which probably results in the suboptimal performance.

In this work, we take advantage of pre-trained English-oriented LLMs, adapt
LLM tokenizer for better computational efficiency, leverage continued pre-
training on vast Russian-language corpora with regularization for preventing
“catastrophic forgetting”, construct a novel extended set of Russian
instruction-output pairs, and perform instruction tuning. The created LLM
adaptation pipeline along with the data for continued pre-training and
instruction tuning enables Vikhr to achieve new state-of-the-art results for
Russian, maintain high performance for English, and demonstrate high
computational efficiency.

##  3 LLM Construction Pipeline

The construction of Vikhr starts from one of English-oriented LLMs. In this
work, we discuss the Vikhr model based on Mistral 7B. The strong logical and
common reasoning capabilities, as well as the extensive world knowledge
present in these LLMs provide an excellent starting point for our model. These
features partially transfer to Vikhr, enhancing its performance in generating
text in Russian. The process of LLM adaptation to Russian starts with the
vocabulary adaptation. Then we perform continued pre-training of the LLM on
large Russian datasets to mitigate the vocabulary shift and introduce culture
specific knowledge. Finally, we perform fine-tuning of Vikhr on a set of
instruction-output pairs in Russian.

###  3.1 Vocabulary Adaptation

Content |  Length |  Tokenization Result
---|---|---
Original Sentence |  31 |  Машинное обучение изменяет мир
Mistral Tokenizer |  13 |  [‘Ма’, ‘шин’, ‘ное’, ‘об’, ‘у’, ‘чение’, ‘из’, ‘мен’, ‘я’, ‘ет’, ‘ми’, ‘р’ ]
Vikhr Tokenizer |  7 |  [‘Ма’, ‘шин’, ‘ное’, ‘обучение’, ‘изменяет’, ‘мир’]
Table 1: Tokenizer comparisons between the original Mistral model and Vikhr
Figure 1: The Vikhr tokenizer efficiency in comparison to tokenizers of other
models.

The big drawback of English-oriented LLMs is that each Russian word would be
split into multiple tokens: a common case is when symbols in the word become
an individual tokens (see example in Table 1). This slows down the generation
by multiple times, reduces the amount of information that could be stored in
the context, and drastically hurts the generation quality.

To mitigate this problem in Vikhr, we adopt the approach suggested in Cui et
al. (2023); Tikhomirov and Chernyshev (2023), where authors rebuild the
tokenizer using a language-specific corpus. In particular, we trained a
SentencePiece tokenizer Kudo and Richardson (2018) with a 40k vocabulary on
the RuLM dataset Gusev (2023). As can be seen from Figure 1, the resulting
tokenizer for Russian is much more efficient than the tokenizer of the
original English-oriented model.

Data Source |  | Approx. size
---
(GB)
| Tokens
---
(Billion)
Scientific papers | 20 | 2.5
News articles | 4 | 1
Wikipedia | 25 | 4
Habr | 6 | 1
Other sources | 20 | 2.5
Table 2: The statistics of the Russian-language datasets for continued pre-
training.

###  3.2 Continued Pre-training

The new vocabulary requires also new embedding matrices and LM heads. The
tokens that were present in the original vocabulary are initialized with the
old embeddings, the new tokens are initialized by averaging the embeddings of
their pieces in the original embedding matrix Hewitt (2021). The similar
approach is also applied to LM heads. Training model with these modifications
requires much more computational resources than the mainstream technique for
adaptation of LLMs to new languages based on LoRA adapters Hu et al. (2021),
as it requires to perform continued pre-training of the whole model and on
much more language-specific data to mitigate the shift in the vocabulary.

The dataset for continued pre-training is constructed from Russian Wikipedia,
news articles, scientific papers, top 100k up-voted posts on Habr, and some
other sources. The statistics of these datasets is presented in Table 2. The
total number of tokens used for this step is 11 billion.

We note that the continued pre-training of a LLM might partially eliminate the
reasoning capabilities present in the original English-oriented model. This
drastically affects the model performance. In our preliminary experiments,
continued pre-training may result even in worse performance on Russian
benchmarks compared to the original model. To alleviate the “catastrophic
forgetting”, we use the loss regularization with KL penalty between the
probability distribution of Vikhr and the reference English-oriented original
LLM:

$L_{\text{Vikhr}}=L_{\text{CE}}+KL\left(P_{\text{Vikhr}}\|P_{\text{Ref
}}\right).$ (1)

In practice, we implement this approach using the SLERP interpolation of model
losses Goddard et al. (2024).

Hyperparam. | Value
---|---
LR | $1\times 10^{-3}$
AdamW eps | $1\times 10^{-8}$
Num warmup steps | 10
AdamW betas |  $0.99$, $0.95$
Accumulation steps | $128$
Batch size | $3$
Epochs | $1$
Sequence length | $1024$
Table 3: The hyperparameters for continued pre-training.

To speed up the process of continued pre-training, we use an optimized Flash
attention
implementation222https://huggingface.co/docs/optimum/bettertransformer/tutorials/convert.
As an optimization algorithm, we leverage AdamW as it trades some memory
efficiency in favor of robustness to the hyperparameter choice. The
hyperparameters used for continued pre-training are presented in Table 3.

###  3.3 Instruction Tuning

Instruction tuning is an essential step in reaching high zero-shot performance
with LLMs. It also allows to obtain more natural communication with the model
without complex prompting. Further fine-tuning techniques such as RLHF Ouyang
et al. (2022), which require input from the assessors, are also crucial for
such tasks as multicriteria alignment. However, the most significant
performance gains are still achieved through instruction tuning Jha et al.
(2023).

Previously, Gusev (2023) constructed an open-source set of instruction-output
pairs for the Russian language (Saiga). The core Saiga dataset was created
similar to Alpaca by querying ChatGPT (gpt-3.5-turbo) Taori et al. (2023). In
this work, we extend this set by translating two English instruction datasets.
First, we translated instructions for the FLAN model Wei et al. (2021) and
generated answers in Russian using ChatGPT. Originally, FLAN instructions were
constructed automatically from annotated datasets using templates to
facilitate multitask and zero-shot capabilities of seq2seq models. Later, it
was shown that this data also helps to improve decoder-only chat-oriented
models as well. Second, we construct
Veles333https://huggingface.co/datasets/Vikhrmodels/Veles-2.5 by translating
the English OpenHermes Teknium (2023) instruction dataset. We also include
without translation Nectar444https://huggingface.co/datasets/berkeley-
nest/Nectar Zhu et al. (2023) – the English instruction dataset. It helps to
keep the performance of Vikhr high also for English. Since the majority of the
outputs were machine generated there are many low quality outputs. To mitigate
this problem, we filtered out low quality pairs using a reward model trained
on human data. The statistics of the Vikhr instruction datasets is presented
in Table 4.

Instruction Set | Language | # instances
---|---|---
Veles | Russian | 30k
Nectar | English | 50k
Saiga | Russian | 100k
ruFLAN | Russian | 500k
Table 4: The statistics of instruction datasets. Hyperparam. | Value
---|---
LR | $1\times 10^{-5}$
AdamW, eps | $1\times 10^{-8}$
Num warmup steps | 10
AdamW, betas |  $0.99$, $0.95$
Accumulation steps | $64$
Batch size | $3$
Num epochs | $3$
Sequence length | $1024$
Table 5: The hyperparameters for instruction tuning.

LLM |  | Pre-train on
---
Russian
| Training
---
Method
En-MMLU | Ru-MMLU | CheGeKa |  | Russian
---
SuperGLUE
MERA
MTS AI Chat 7B (closed-source) $\diamondsuit$ | false | sft+dpo | - | 0.689 | 0.083 | 0.56 | 0.479
GigaChat-7B (closed-source) $\diamondsuit$ | true | sft+dpo | - | 0.67 | 0.451* | 0.71* | 0.479
aya101 | false | pt+sft | 0.41 | 0.37 | 0.005 | 0.36 | 0.320
Mistral-7B-Instruct-v0.2 | false | none | 0.60 | 0.78 | 0.005 | 0.57 | 0.400
rccmsu/ruadapt-mistral-7b-v0.1 | false | pt+sft | 0.61 | 0.72 | 0.005 | 0.64 | 0.421
rugpt13b | true | none | 0.25 | 0.25 | 0.132 | 0.52 | 0.208
saiga-mistral-7b-lora | false | sft | 0.60 | 0.76 | 0.223 | 0.64 | 0.442
saiga-llama3-8b | false | sft | 0.59 | 0.78 | 0.225 | 0.66 | 0.476
Vikhr-7B-instruct_0.2 | true | pt+sft | 0.62 | 0.80 | 0.231 | 0.67 | 0.485

Table 6: Evaluation results for Russian and multilingual LLMs. Pre-train on
Russian means that the model underwent (continued) pre-training on Russian
data. The following abbreviations are used: sft – instruction tuning, pt –
(continued) pre-training; dpo – direct preference optimization.
$\diamondsuit$ The results for GigaChat and MTS AI are taken from the
leaderboards. The best result among open-source models is highlighted with
bold, the second best is underscored. The best result among closed-source
proprietary models is marked with *.

Contrary to Saiga, we do not use LoRA adapters and just as in the phase of
continued pre-training, we update all model parameters. The hyperparameters
for the instruction tuning phase are presented in Table 5.

###  3.4 Hardware

Vikhr was trained on eight NVIDIA A100 GPUs 80GB. We spend approximately 1,000
GPU hours for the continued pre-training phase and 60 hours for instruction
tuning.

##  4 Experiments

###  4.1 Experimental Setup

#### Benchmarks.

The evaluation was performed on MMLU Hendrycks et al. (2021), Ru-
MMLU555https://github.com/NLP-Core-Team/mmlu_ru, CheGeKa, Russian SuperGLUE
Shavrina et al. (2020), and MERA Fenogenova et al. (2024). MMLU (En-MMLU)
evaluates LLMs across 57 subjects with multiple-choice questions, assessing a
model’s broad knowledge and reasoning abilities. We use this benchmark to
verify that the model retains bi-lingual capabilities. In the results, we
report the accuracy@1 score. RuMMLU is a translation of MMLU with GPT-3.5 to
Russian. Just as for MMLU, we report the accuracy@1 score. CheGeKa is based on
questions from the game “What? Where? When?”. This benchmark contains
challenging open-ended questions, requiring logical reasoning and world
knowledge. It includes 29,376 training and 416 test instances. The reported
evaluation metric is the F1 score. Russian SuperGLUE is a benchmark similar to
well-known English SuperGLUE Wang et al. (2019). It tests LLMs on various
natural language understanding tasks like reading comprehension and textual
entailment. The metric reported in the results is accuracy@1. The MERA
benchmark encompasses 21 evaluation tasks for generative LLMs in 11 skill
domains. Note that among other tasks MERA also includes CheGeKa, RuMMLU, and
one of the subtasks of SuperGLUE (RWSD). The reported evaluation metric is the
total score, which is the average of scores across all non-diagnostic tasks.

#### Baselines.

We compare Vikhr to six open-source and two proprietary closed-source
competitors of the similar size. Open-source models: aya101 – a massively
multilingual LLM from CohereAI that follows instructions in 101
languages666https://huggingface.co/CohereForAI/aya-101, it shows state-of-the-
art results among massively multilingual LLMs; Mistral-7B-0.2-instruct – an
English-oriented LLM that was used as the base model for Vikhr;
rccmsu/ruadapt_mistral_saiga_7b_v0.1 – a Russian-oriented LLM that was
constructed from the Mistral model using similar adaptations of the tokenizer,
token embeddings, and the LM head Tikhomirov and Chernyshev (2023); saiga-
mistral-7b-lora and saiga-llama3-8b – two versions of the Saiga models based
on English-oriented LLMs and obtained by fine-tuning LoRA adapters on the
Saiga instruction dataset777https://huggingface.co/collections/IlyaGusev.
Closed-source proprietary models for Russian: MTS AI
Chat888https://huggingface.co/MTSAIR/multi_verse_model and GigaChat-7b. The
access to GigaChat weights is closed, so the reported results are taken from
the leaderboards999https://mera.a-ai.ru/ru/submits/10257. The results of MTS
AI Chat are also taken from the
leaderboard101010https://mera.a-ai.ru/ru/submits/10290.

###  4.2 Results

The evaluation results are presented in Table 6. As we can see, Vikhr
outperforms all open-source models, including the ones that were built
specifically for Russian. It also slightly outperforms its parent model
Mistral on the En-MMLU benchmark, which might be the result of longer pre-
training. The second place with close scores for all 4 Russian language
benchmarks is obtained by the Saiga model based on recently released Llama-3.
The high scores of this model probably are the result of the transfer of the
outstanding performance of Llama-3. Since Saiga based on Llama-3 outperforms
Saiga based on Mistral, we expect that applying our adaptation pipeline to
Llama-3 would also help further improving the state of the art.

We note that the original Mistral-7B-0.2-instruct, despite being an English-
oriented model, demonstrates competitive performance in 3 out of 4 Russian
benchmarks. This indicates demonstrates that such models could be viable
alternatives. The only dataset, where its performance is very low is CheGeKa,
which is related to open-ended question-answering. This may be due to the lack
of culture-specific knowledge, as the English-oriented model has not seen much
Russian texts. Note that the MTS AI Chat also shows very low results on
CheGeKa, which might also indicate the lack of culture-specific knowledge.

The proprietary model GigaChat substantially outperforms Vikhr on CheGeKa and
notably on Russian SuperGLUE. We assume this is due to the use of much larger
Russian datasets for pre-training. However, surprisingly, it falls behind
Vikhr on Ru-MMLU. On all benchmarks, Vikhr outperforms the the proprietary
competitor from MTS AI.

##  5 Conclusion

We have presented Vikhr – a new state-of-the-art open-source instruction-
following LLM oriented on the Russian language. To create Vikhr, we developed
a comprehensive pipeline for adapting English-oriented LLMs to Russian. The
pipeline includes the adaptation of the tokenizer vocabulary, continued pre-
training of the entire model, and instruction tuning. We have also constructed
a new dataset for instruction tuning by expanding the Saiga dataset with
automatically translated and cleaned English instruction datasets. Our
extensive work enabled Vikhr to outperform the known baselines, while
maintaining computational efficiency.

We hope that the published models will foster the research on LLMs and enhance
the diversity of languages incorporated into research agendas.

## Limitations

We do not introduce additional restrictions to the usage of our models.
However, the users must comply with the license of the base model and
instruction datasets.

We do not implement RLHF / DPO fine-tuning of Vikhr due to the lack of the
resources for human annotation. We expect further performance improvements
from these techniques.

We do not introduce additional instruction-output pairs to facilitate LLM
alignment. However, we note that the majority of the data for supervised fine-
tuning of Vikhr are obtained from the ChatGPT model series, so our model
partially inherits its alignment.

## References

  * AI Forever (2022) AI Forever. 2022.  ru-gpts: Generative pre-trained transformer models for russian.  https://github.com/ai-forever/ru-gpts.
  * Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.
  * Cui et al. (2023) Yiming Cui, Ziqing Yang, and Xin Yao. 2023.  Efficient and effective text encoding for chinese llama and alpaca.  _arXiv preprint arXiv:2304.08177_.
  * Fenogenova et al. (2024) Alena Fenogenova, Artem Chervyakov, Nikita Martynov, Anastasia Kozlova, Maria Tikhonova, Albina Akhmetgareeva, Anton Emelyanov, Denis Shevelev, Pavel Lebedev, Leonid Sinev, et al. 2024.  Mera: A comprehensive llm evaluation in russian.  _arXiv preprint arXiv:2401.04531_.
  * Goddard et al. (2024) Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vlad Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz. 2024.  Arcee’s mergekit: A toolkit for merging large language models.  _arXiv preprint arXiv:2403.13257_.
  * Gusev (2023) Ilya Gusev. 2023.  rulm: A toolkit for training neural language models.  https://github.com/IlyaGusev/rulm.
  * Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021.  Measuring massive multitask language understanding.  _Proceedings of the International Conference on Learning Representations (ICLR)_.
  * Hewitt (2021) John Hewitt. 2021.  Initializing new word embeddings for pretrained language models.
  * Hu et al. (2021) Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2021.  Lora: Low-rank adaptation of large language models.  In _International Conference on Learning Representations_.
  * Jha et al. (2023) Aditi Jha, Sam Havens, Jeremy Dohmann, Alex Trott, and Jacob Portes. 2023.  Limit: Less is more for instruction tuning across evaluation paradigms.  _arXiv preprint arXiv:2311.13133_.
  * Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023.  Mistral 7b.  _arXiv preprint arXiv:2310.06825_.
  * Kudo and Richardson (2018) Taku Kudo and John Richardson. 2018.  Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 66–71.
  * Li et al. (2023) Haonan Li, Fajri Koto, Minghao Wu, Alham Fikri Aji, and Timothy Baldwin. 2023.  Bactrian-x: A multilingual replicable instruction-following model with low-rank adaptation.  _arXiv preprint arXiv:2305.15011_.
  * Maaz et al. (2024) Muhammad Maaz, Hanoona Rasheed, Abdelrahman Shaker, Salman Khan, Hisham Cholakal, Rao M Anwer, Tim Baldwin, Michael Felsberg, and Fahad S Khan. 2024.  Palo: A polyglot large multimodal model for 5b people.  _arXiv preprint arXiv:2402.14818_.
  * Muennighoff et al. (2022) Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. 2022.  Crosslingual generalization through multitask finetuning.  _arXiv preprint arXiv:2211.01786_.
  * Muennighoff et al. (2023) Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, et al. 2023.  Crosslingual generalization through multitask finetuning.  In _The 61st Annual Meeting Of The Association For Computational Linguistics_.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022.  Training language models to follow instructions with human feedback.  _Advances in neural information processing systems_ , 35:27730–27744.
  * Petrov et al. (2024) Aleksandar Petrov, Emanuele La Malfa, Philip Torr, and Adel Bibi. 2024.  Language model tokenizers introduce unfairness between languages.  _Advances in Neural Information Processing Systems_ , 36.
  * Sengupta et al. (2023) Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, et al. 2023.  Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models.  _arXiv preprint arXiv:2308.16149_.
  * Shavrina et al. (2020) Tatiana Shavrina, Alena Fenogenova, Emelyanov Anton, Denis Shevelev, Ekaterina Artemova, Valentin Malykh, Vladislav Mikhailov, Maria Tikhonova, Andrey Chertok, and Andrey Evlampiev. 2020.  Russiansuperglue: A russian language understanding evaluation benchmark.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4717–4726.
  * Shliazhko et al. (2022) Oleh Shliazhko, Alena Fenogenova, Maria Tikhonova, Vladislav Mikhailov, Anastasia Kozlova, and Tatiana Shavrina. 2022.  mgpt: Few-shot learners go multilingual.  _arXiv preprint arXiv:2204.07580_.
  * Taori et al. (2023) Rohan Taori, Ishaan Shum, Pieter Abbeel, Carlos Guestrin, and Percy Liang. 2023.  Stanford alpaca: An instruction-following language model.  _GitHub_.
  * Teknium (2023) Teknium. 2023.  Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants.
  * Tikhomirov and Chernyshev (2023) Mikhail Tikhomirov and Daniil Chernyshev. 2023.  Impact of tokenization on llama russian adaptation.  _arXiv preprint arXiv:2312.02598_.
  * Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Wang et al. (2019) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019.  Superglue: A stickier benchmark for general-purpose language understanding systems.  _Advances in neural information processing systems_ , 32.
  * Wei et al. (2021) Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021.  Finetuned language models are zero-shot learners.  In _International Conference on Learning Representations_.
  * Zhao et al. (2024) Jun Zhao, Zhihao Zhang, Qi Zhang, Tao Gui, and Xuanjing Huang. 2024.  Llama beyond english: An empirical study on language capability transfer.  _arXiv preprint arXiv:2401.01055_.
  * Zhu et al. (2023) Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, and Jiantao Jiao. 2023.  Starling-7b: Improving llm helpfulness & harmlessness with rlaif.
  * Zmitrovich et al. (2023) Dmitry Zmitrovich, Alexander Abramov, Andrey Kalmykov, Maria Tikhonova, Ekaterina Taktasheva, Danil Astafurov, Mark Baushenko, Artem Snegirev, Tatiana Shavrina, Sergey Markov, et al. 2023.  A family of pretrained transformer language models for russian.  _arXiv preprint arXiv:2309.10931_.
  * Üstün et al. (2024) Ahmet Üstün, Viraat Aryabumi, Zheng-Xin Yong, Wei-Yin Ko, Daniel D’souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer, and Sara Hooker. 2024.  Aya model: An instruction finetuned open-access multilingual language model.  _arXiv preprint arXiv:2402.07827_.