#  NOTE: This is an old and now obsolete draft
See https://arxiv.org/abs/2109.14723 for the final paper
Enriching a Model’s Notion of Belief using a Persistent Memory

Nora Kassner1,2, Oyvind Tafjord1, Hinrich Schütze2, Peter Clark1
1Allen Institute for AI, Seattle, WA
2Center for Information and Language Processing, LMU Munich, Germany
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Although pretrained language models (PTLMs) have been shown to contain
significant amounts of world knowledge, they can still produce inconsistent
answers to questions when probed, even after using specialized training
techniques to reduce inconsistency. As a result, it can be hard to identify
what the model actually "believes" about the world. Our goal is to reduce this
problem, so systems are more globally consistent and accurate in their
answers. Our approach is to add a memory component - a BeliefBank - that
records a model’s answers, and two mechanisms that use it to improve
consistency among beliefs. First, a reasoning component - a weighted SAT
solver - improves consistency by flipping answers that significantly clash
with others. Second, a feedback component re-queries the model but using known
beliefs as context. We show that, in a controlled experimental setting, these
two mechanisms improve both accuracy and consistency. This is significant as
it is a first step towards endowing models with an evolving memory, allowing
them to construct a more coherent picture of the world.111Dataset is available
at https://allenai.org/data/beliefbank

##  1 Introduction

Figure 1:  The proposed architecture. The model’s raw answers are stored in a
persistent memory (BeliefBank), and two mechanisms attempt to improve them:
(a) A constraint solver flips beliefs that clash significantly with others (b)
A feedback mechanism re-queries the model about those beliefs, but uses other,
relevant beliefs as the query context. We find both consistency and accuracy
of the overall system improves.

NOTE: This is an old and now obsolete draft. See
https://arxiv.org/abs/2109.14723 (“BeliefBank: Adding Memory to a Pre-Trained
Language Model for a Systematic Notion of Belief”) for the final paper

How might we ascribe a notion of belief to a model? Prior work has shown that,
while pretrained language models (PTLMs) contain substantial world knowledge
Petroni et al. (2019); Roberts et al. (2020), their answers to probing
questions can be inconsistent Elazar et al. (2021); Ravichander et al. (2020);
Kassner and Schütze (2020), making it hard to pin down what a model actually
“believes” about a proposition. Our goal is to reduce this problem by having
systems provide more globally consistent answers to questions.

Prior work on reducing inconsistency has focused on retraining the model
itself to be more consistent in its answers, e.g., Ribeiro et al. (2019); Li
et al. (2019), but with imperfect results. We present an alternative approach
in which the model is unchanged, but an evolving, persistent memory of beliefs
- called the BeliefBank - is layered on top, and two mechanisms use it to
improve consistency among beliefs. First a reasoning component - a weighted
SAT (satisfiability) solver - flips beliefs that clash with others. Second, a
feedback component re-queries the model but using known beliefs as context,
aiming for more accurate and consistent answers. Thus, the overall system
attempts to build a more coherent representation of the world from the model’s
raw answers, by ruminating on the answers seen so far. This can also be viewed
as assembling a simple “mental model” of the world Johnson-Laird (1983) from
the noisy output of a raw PTLM.

We explore this in a controlled experimental setting where both candidate
facts and constraints are provided. Candidate facts are simple sentences that
may be true or false, e.g., "An eagle is a bird" (T), “An eagle is a mammal”
(F). Constraints are between (variabilized) facts, e.g., “X is a bird
$\rightarrow$ X has wings”. These allow us both to probe and measure
improvement in the system’s consistency and accuracy.

We make the following contributions:

  * 1.

We augment a PTLM with a global memory - the BeliefBank - and show how it can
be leveraged to produce more globally consistent answers. Specifically, we
compare two mechanisms for using the BeliefBank - a reasoning and a feedback
mechanism - and demonstrate that both can improve system accuracy and
consistency in a restricted setting.

  * 2.

We contribute a controlled dataset to measure a PTLM’s consistency against
given constraints

  * 3.

We provide an analysis of the failure modes and directions for future work

These are significant as they enrich a model’s notion of “belief”, helping
them construct a more coherent picture of the world.

##  2 Related work

PTLMs are known to contain extensive world knowledge Petroni et al. (2019);
Roberts et al. (2020), yet be inconsistent in their answers to probing
questions Ettinger (2020); Davison et al. (2019); Kassner and Schütze (2020);
Ravichander et al. (2020); Elazar et al. (2021). While there has been some
prior work on improving answer consistency, the primary approach has been
through modified model training. Ribeiro et al. (2019) improved consistency by
adding question paraphrases and question implications to the training data
(data augmentation). Others have trained models with (small) sets of examples
with known constraints between them, and included an additional loss term
reflecting inconsistency among set members during training Minervini and
Riedel (2018); Li et al. (2019); Asai and Hajishirzi (2020). However, the
constraints are unused at test time (beyond what the model may have
internalized), and inconsistent answers are still produced.

For problems requiring a structured answer, e.g., predicting a sequence of
state changes, domain-specific constraints have been used to downgrade/block
answers that violate them Tandon et al. (2018); Du et al. (2019). This
encourages consistency within a single answer structure, but not among
different answers, our goal here.

In the area of knowledge graph construction, Pujara et al. (2013) define
“knowledge graph identification” as the task of building a maximally
consistent knowledge graph given noisy facts and their extraction confidences,
and ontological constraints between them. They develop a solution use
probabilistic soft logic (PSL) Broecheler et al. (2010) as their constraint
reasoner. Similarly, Berant et al. (2010) learn the globally optimal set of
entailments between a large database of candidate entailment pairs (with
associated confidences), by applying a global transitivity constraint
(X$\vdash$Y & Y$\vdash$Z $\rightarrow$ X$\vdash$Z) using Integer Logic
Programming. In our case, we follow similar ideas but show how they can be
usefully applied to the noisy predictions of a PTLM. Specifically, we
formulate the task as a weighted SAT (satisfiability) problem, and use the SMT
solver Z3 De Moura and Bjørner (2008) to solve it (Section 4.2.1).

In the area of formal knowledge-bases (KBs), efficient algorithms have been
developed for detecting, measuring, and resolving inconsistency Hansen and
Jaumard (2000); Andersen and Pretolani (2001); Thimm (2009); Muiño (2011);
Thimm (2013). Our contribution is to leverage some of these methods for PTLMs,
adding a reasoning capability that the PTLMs alone lack.

An important part of our contribution is the use of a dynamic, persistent
memory. While there are neural architectures that include an associated
memory, e.g., Henaff et al. (2016); Sukhbaatar et al. (2015), these components
typically play the role of a short-term working memory to help computation. In
contrast, our BeliefBank memory layer is a persistent, long-term memory of
explicit beliefs.

Finally, our feedback mechanism uses old answers to help answer new questions.
This builds on prior work such as Self-Talk Shwartz et al. (2020), where a
model asks itself related questions to help with new answers. In our case,
feedback is selected from a global BeliefBank, rather than generated with
templated subqueries, potentially allowing more control over feedback
selection.

##  3 Task

###  3.1 Beliefs

What does it mean to believe a proposition, say p = eagles are birds? In
general, a system can be said to (appear to) believe something if it acts as
if it were true. In the specific context of a QA system, we would expect it to
produce answers consistent with p (and its other beliefs). Pragmatically, we
expect the system to (a) give a consistent answer to different paraphrases of
the question "p?" ("Are eagles birds?", "Is an eagle a type of bird?", …), and
(b) give correct answers about implications of p ("Eagles lay eggs", "Eagles
have feathers", …). Of course, a system may not perfectly answer such
implications as the implications may have exceptions, or the system may not be
a perfect reasoner.222Similarly, people do not always behave fully
consistently with their professed beliefs.. Thus, to the external observer,
there are degrees to which a system acts as if it believes something.

###  3.2 Task Definition

Our goal is to ascribe a stronger notion of “belief” to a system that includes
a model $M$, by improving the consistency and accuracy of its answers
(compared with $M$). To measure this we consider a true/false probing task,
where we are also given a set of constraints between answers:
[vskip=0mm,leftmargin=2mm] Given:

  * •

a set of sentences $S$

  * •

a set of constraints $C(S)$ between (the truth values of) sentences in $S$,
each annotated with a weight $w_{i}$ (A penalty $w_{i}$ is applied if
$c_{i}\in C(S)$ is violated)

  * •

a model $M$ that takes as input a True/False natural language (NL) question
$Q$ and optionally an (NL) context $X$, and predicts a True/False answer $A$
with confidence score $F$

Predict:

  * •

the True/False labels for $S$, so as to maximally improve accuracy (with
respect to gold labels) and consistency (minimize total penalties of
constraint violations) compared with model $M$’s raw answers

##  4 Approach

Our approach is to add a memory layer, called the BeliefBank, on top of the
model to globally track beliefs. Two mechanisms are then used to modify
BeliefBank beliefs, namely (a) constraint reasoning and (b) re-asking queries
augmented with feedback from the BeliefBank.

###  4.1 Definitions

Let

  * •

a belief $b_{i}$ be a triple ($s_{i}$,$l_{i}$,$w_{i}$), where

    * –

$s_{i}$ is a sentence $\in S$

    * –

label $l_{i}\in$ {T,F} denotes the system’s True/False belief about the truth
of $s_{i}$

    * –

weight $w_{i}$ is a number $\in[0,1]$ representing the system’s strength of
that belief

For example:

> ("a poodle is a dog", T, 0.9)

denotes the belief (strength 0.9) that "a poodle is a dog" is a true statement
(T).

  * •

a BeliefBank $B(S)$ = a set of beliefs over assertions $S$ = $s_{1},...,s_{n}$

  * •

a constraint $c_{i}$ = a 4-tuple of the form ($s_{i}\rightarrow
s_{j},l_{j},w_{i}$) where

    * –

$s_{i},s_{j}$ are sentences $\in S$,

    * –

$l_{j}\in$ {T,F} denotes the expected truth of $s_{j}$ if $s_{i}$ is true,

    * –

$w_{i}$ denotes the strength of that expectation (a penalty $w_{i}$ is applied
if it is violated).

For convenience, we allow a shared variable X to be used in $s_{i},s_{j}$,
allowing a set of grounded constraints to be expressed in a single statement,
e.g. [vskip=0mm,leftmargin=2mm] (“X is a dog” $\rightarrow$ “X has a tail”, T,
0.8)  expresses that if something is a dog, then it should (T) have a tail,
with a penalty of 0.8 applied if it does not. Mutual exclusivity is expressed
using two rules, e.g., that fish and birds are mutually exclusive is
expressed: [vskip=0mm,leftmargin=2mm] (“X is a bird” $\rightarrow$ “X is a
fish”, F, 1.0)
(“X is a fish” $\rightarrow$ “X is a bird”, F, 1.0)  where “F” indicates the
conclusion should be false if the condition is true.

  * •

A constraint graph $C(S)$ = a set of constraints $c_{i}$ over assertions $S$

Given a set of beliefs $B(S)$ about $S$ and a set of constraints $C(S)$, we
measure consistency using (the complement of) Li et al. (2019)’s conditional
constraint violation ($\tau$) metric, namely the fraction of constraints whose
condition $s_{i}$ is believed true, but whose conclusion (that $s_{j}$ has
truth value $l_{j}$, denoted $l_{j}.s_{j}$) is not. In other words, over all
constraints $c_{i}\in C(S)$, inconsistency $\tau$ is
[vskip=0mm,leftmargin=4mm] $\tau=|\\{~{}c_{i}~{}|~{}\neg(s_{i}\rightarrow
l_{j}.s_{j})~{}\\}|~{}~{}~{}/~{}~{}~{}|\\{~{}c_{i}~{}|~{}s_{i}~{}\\}|$ i.e.,
the size of the set of violated constraints ($s_{i}\rightarrow l_{j}.c_{j}$ is
false) divided by the size of the set of applicable constraints (i.e., those
where the condition $s_{i}$ is true). We then define:
[vskip=0mm,leftmargin=4mm]

consistency = 1 - $\tau$

Figure 2: Simplified illustration of iteratively improving the BeliefBank
(oval). +/- denote true/false predictions for 10 facts about swallows. The
model alone makes 4 prediction errors (M). Re-querying for those beliefs using
other selected beliefs as context (feedback) fixes 2 of those errors (F).
Running constraint solving on the updated BeliefBank fixes another error (C),
resulting in just 1 error in the final BeliefBank. Here, the sequence is Model
$\rightarrow$ Feedback $\rightarrow$ Constraints.

###  4.2 Methods

We evaluate two methods for improving the BeliefBank’s accuracy and
consistency:

  * Constraint solving:

Given a model $M$’s answers (with confidences) in the BeliefBank, a constraint
solver seeks to reduce constraint violations by potentially flipping answers
that maximally clash with other answers.

  * Feedback:

Beliefs are checked by re-querying the model, but additionally using relevant
other beliefs as context for those (re-)queries.

Figure 1 shows these components, and Figure 2 illustrates how they can
iteratively improve the BeliefBank. In Figure 2, there are 10 beliefs of
interest about swallows. When probed about these, the raw model gets 4 of
these wrong, including inconsistently believing that a swallow is both a bird
and a mammal (see (M) in Figure 2). Applying feedback, we re-ask the model the
10 questions but adding in the most relevant known beliefs as context for
those (re-)queries. The BeliefBank is updated with these new answers, fixing 2
of the errors (F). We then run the constraint solver over the BeliefBank,
resulting in one answer being flipped (question 9, from T to F), thus fixing
another error (see (C)). The final BeliefBank has just 1 error and greater
consistency. Note that updates can also introduce new errors (not shown in
Figure 2), so improvement is not guaranteed.

####  4.2.1 Constraint Solving

Given a set of beliefs and constraints, the constraint solver has two
competing objectives: (a) flip beliefs so as to minimize constraint violations
(b) don’t flip beliefs, so as to preserve the model’s raw beliefs, i.e.,
minimize conflict between the model and BeliefBank. To implement this
tradeoff, the model’s beliefs are themselves treated as just another
constraint, e.g., the belief "a poodle is a dog" (weight 0.9) is treated as a
constraint "a poodle is a dog", with penalty 0.9 if it is violated (i.e.,
labeled as false by the constraint solver). To balance the two objectives (a)
and (b), the belief weights are scaled by a learned hyper-parameter $\lambda$,
trained on a calibration part our dataset, disjoint from the data then used in
experiments (Section 5).

To implement constraint solving, we translate the task into a weighted SAT
(satisfiability) problem $P$, for which efficient algorithms with guarantees
exist. Each belief becomes a weighted assertion in $P$, e.g., the belief ("a
poodle is a dog", T, 0.9) is expressed in SAT syntax:
[vskip=0mm,leftmargin=4mm] 0.9  "a poodle is a dog"333In practice, strings are
replaced with numeric identifiers in SAT syntax, but for clarity we leave them
as strings here. while the constraint ("a poodle is a dog" $\rightarrow$ "a
poodle has a tail", T, 0.8) is expressed: [vskip=0mm,leftmargin=4mm] 0.8 "a
poodle has a tail" -"a poodle is a dog" (literally: "a poodle has a tail" OR
NOT ("-") "a poodle is a dog"). We then apply the solver Z3 De Moura and
Bjørner (2008) to $P$, which outputs a set of truth assignments for all
individual sentences in $P$ so as to minimize the weighted sum of violations.
If the truth assignment of any sentence has changed, the BeliefBank is
correspondingly updated.

####  4.2.2 Feedback

Feedback involves re-asking the model a question, but with the benefit of
knowing answers to related questions. To use these answers in the re-query,
selected beliefs are added to the query context before re-asking the model.
(Note that the selected beliefs are not guaranteed to be correct, of course).
Our conjecture is that if the model is explicitly reminded of relevant beliefs
when answering a new question, it will answer the question more accurately and
consistently. For example, in Figure 1, when asked "Do swallows have gills?",
our model $M$ incorrectly answers "yes". But if reminded that swallows are not
fish, by asking: "CONTEXT Swallows are not fish. QUERY Do swallows have
gills?" the model now correctly answers "no".

We evaluate two policies for choosing which beliefs to feed back to $M$ when
re-asking question $Q$:

  * 1.

randomly selected from the BeliefBank

  * 2.

most relevant, using the constraint graph to identify relevance. As the
constraint graph captures potential clashes that the answer to $Q$ could
cause, we use the graph to select those beliefs that would be most affected by
that answer. For example, if the current query is: "Is a poodle an animal?",
the constraint graph identifies potential clashes the would occur if the model
answered "yes", and also clashes if it answered "no". In this example, if the
model answered "no", the resulting belief ("a poodle is not an animal") would
strongly clash with other beliefs "A poodle is a dog.", "A poodle is a
mammal.", and "A poodle is a domesticated canine.", so all three are strong
candidates for the context. We select the three overall strongest clashing
beliefs found in this way, considering both a "yes" and a "no" answer to $Q$.

In both cases, three (if possible) beliefs are selected, this number
empirically found to be most effective. (For experiments with small amounts of
data, sometimes less than three relevant beliefs are found.)

##  5 Dataset

We create a dataset to test our approach in a controlled way, allowing us to
perform systematic experiments to evaluate behavior. The dataset contains two
parts, constraints and facts, defined over simple sentences such as “a swallow
is a bird.”.

###  5.1 Constraints

The dataset contains two kinds of constraints:

  * positive implications:

(conclusion truth value $\textrm{l}_{i}$ = True), e.g.,
[vskip=0mm,leftmargin=2mm] “X is a dog $\rightarrow$ [TRUE] X has a tail.”

  * mutual exclusivities:

expressed as a pair of constraints with $\textrm{l}_{i}$ = False, e.g.,
[vskip=0mm,leftmargin=2mm] “X is a dog $\rightarrow$ [l=FALSE] X is a bird.”
“X is a bird $\rightarrow$ [l=FALSE] X is a dog.” expresses that an entity
cannot be both a dog and a bird at the same time.

Positive implications were manually gathered from ConceptNet Speer et al.
(2017). First, we identified 176 general concepts of interest, e.g., “mammal”,
choosing concepts with high occurrence ($>$ 100 times) in ConceptNet, avoiding
significantly ambiguous terms (e.g., “bat”), and filtering out plurals and
obscure concepts. For these entities, we then collected all ConceptNet facts
involving 6 relations: IsA, HasA, MadeOf, PartOf, HasProperty, and CapableOf,
and re-expressed them as constraints. For example, the ConceptNet triple (dog,
HasA, tail) gives rise to the constraint "X is a dog" $\rightarrow$ "X has a
tail." (Triples are converted into English sentences using simple templates).
We then manually filter theses constraints for factual correctness. We also
add the constraint in the backwards direction, "X has a tail" $\rightarrow$ "X
is a dog", expecting these to have lower weight, i.e., be weaker. (These
backwards rules discourage the trivial solution that everything is false.)
Finally, weights are assigned to all the constraints using a combination of
crowdsourcing and calibration, described shortly (Section 5.2).

Mutual exclusivities were gathered from the “isa” taxonomies in ConceptNet and
WordNet Fellbaum (2005), using the approximation that (generally) siblings in
the noun hierarchy are mutually exclusive. Thus, for any pair of concepts in
our concepts of interest that reside in different taxonomic subtrees, we add a
mutual exclusivity constraint (using two constraint rules).

We collected 12,147 constraints in this fashion (1798 implications, 8599
mutual exclusivities). The set of constraints can be viewed as a constraint
graph of implications from one sentence to another.

###  5.2 Constraint Weights

We used a combination of crowdsourcing and calibration to assign a reasonable
constraint weight $w_{i}$ to each constraint. Workers were shown each
constraint and asked to judge if the implication held
always/usually/sometimes/rarely/never with raw scores 4/3/2/1/0. Three workers
independently scored each constraint and the scores averaged.

When used with a particular model $M$, the raw scores need to be recalibrated
to appropriately match the confidences output by that model. This is described
shortly in Section 6.2.

###  5.3 Facts

We also collect a set of truth-labeled facts about different entities,
relevant to the constraints. To do this, we select a new entity, e.g.,
"poodle", that is a member of one of our general concepts, e.g., "dog", then
instantiate the constraint graph with that entity (i.e., set X = "poodle"). We
then identify the leaf (source) nodes of that graph, just considering forward
implication rules, i.e., find facts not implied by other facts in the graph,
and manually annotate their True/False labels. We then use the implications to
infer other True/False labels for other sentences, i.e., propagate the
annotated labels through the graph. This provides “silver” labels for
sentences reachable in this way (a subset of all the sentences in the graph),
silver because the implications are soft, hence not guaranteed to hold for all
entities.

We repeat this for 27 entities (17 animals, 10 plants), resulting a final
dataset containing 4998 “silver” facts (sentences + True/False labels).

Results after seeing % of data $\rightarrow$ | 10% | 50% | 100%
---|---|---|---
Accuracy (F1), Consistency (1-$\tau$) $\rightarrow$ | F1 | Con | F1 | Con | F1 | Con
model (raw) | 75.07 | 87.06 | 74.29 | 77.11 | 73.25 | 74.68
model + constraint-solving | 81.23 | 100.0 | 89.80 | 99.93 | 91.90 | 99.83
model + feedback (random) | 85.33 | 93.46 | 84.59 | 87.59 | 84.00 | 88.02
model + feedback (using constraints) | 82.04 | 94.69 | 84.74 | 92.24 | 88.28 | 91.00
model + feedback (using constraints) + constraint-solving | 83.69 | 100.0 | 91.66 | 99.53 | 93.40 | 99.42
Table 1: Results after seeing different proportions of data. model (raw) are
scores for the model stand-alone. In model + constraint-solving, the
constraint solver is run over all the raw answers so far. In model + feedback,
questions are re-asked using context selected from the other raw answers so
far. In model + feedback + constraint-solving, the constraint solver is
additionally run on the answers with feedback.

##  6 Experiment Environment

###  6.1 Model

The fixed model $M$ that we use for our experiments is the multi-angle QA
system MAQAw Bhakthavatsalam et al. (2021). MAQAw is a derivative of
UnifiedQA, a state-of-the-art T5 model fine-tuned on $\approx$400k question-
answer pairs Khashabi et al. (2020). MAQAw was then further fine-tuned on
several thousand science questions and trained on different permutations of
inputs and outputs (e.g., query Q + context X $\rightarrow$ answer A; XA
$\rightarrow$ Q; etc.). To query the model’s beliefs we pose the query and let
the model chose between the two answer options "yes" and "no". MAQAw also
outputs an answer confidence, used as the belief weight. Note that we do not
retrain MAQAw for this work; rather, it is used as a black-box QA module in
the broader system (Figure 1).

###  6.2 Calibration

To appropriately mix MAQAw’s confidences and the constraint scores, a
calibration step is needed. To do this, we use the 1239 facts about seven
entities in our dataset as a calibration set, and then perform experiments
using (only) the remaining facts.

We calibrate the crowdsourced raw constraint scores using sigmoid-scaling (two
parameters). We also calibrate the relative weight $\lambda$ of the model’s
beliefs and the constraints, the relative weight of the backward implications
(compared with the forward), and the relative weight of mutual exclusivity
rules. To do this we perform a grid search over these parameters, finding the
values that result in the highest F1 (accuracy) after running the constraint
solver over the raw model’s beliefs about these facts.

##  7 Experiments

We evaluate accuracy (F1)444 We measure accuracy with F1 (on the True class)
rather than % correct because the True/False distribution in our dataset is
unbalanced, with significantly fewer True than False answers. F1 avoids scores
being dominated by negative answers. and consistency (1-$\tau$, Section 4.1)
for 3 different-sized, randomly selected subsets of our data: 10%, 50%, 100%.
We average results across entities. The results are shown in Table 1, showing
the model’s a priori accuracy and consistency (line 1), and the effects of
constraint-solving, feedback, or both (lines 2-5).

###  7.1 Results

Several conclusions can be drawn:

  1. 1.

The basic model is both imprecise and inconsistent, with F1 $\approx$ 73% and
consistency $\approx$ 75%. This reflects similar observations of model
inconsistency made for other PTLMs (Section 2), and illustrates the challenge
we wish to address. As we show shortly, the main source of error is in
precision, rather than recall, i.e., the large majority of errors are false
positives.

  2. 2.

Constraint-solving removes almost all inconsistency with respect to our
constraints, even with only a subset of the data. (Of course, with less data,
there are fewer applicable constraints that can be violated.) It also
significantly improves accuracy ($\approx$+10% F1), indicating that the
flipped truth values not only reduce clashes but better align with the true
(gold) labels. Note that improvement in accuracy is not guaranteed:
constraint-solving can (and sometimes does) flip truth values the wrong way,
in order to reduce constraint violations. The improvement suggests that the
model is getting enough answers correct, a priori, to steer the constraint-
solving in the right direction. Comparing accuracy after seeing 10%, 50% and
100% of the data, we note incremental gains the more beliefs and constraints
the constraint-solver is able to leverage.

  3. 3.

Feedback (both types) also improves both accuracy and consistency, although
not to the same extent as constraint-solving. It is perhaps surprising that
explicitly reminding the model of facts it already believes, when answering a
new question, can improve the results. The context may be behaving loosely
like an attention mechanism, encouraging the model to focus on the facts
provided in the context, even if they are already latently known, thus
influencing question-answering behavior. Similar observations have been
observed by Shwartz et al. (2020).

  4. 4.

Constraint-guided feedback is more effective than random feedback when the
BeliefBank is large (100% column), with F1 rising to 88% and consistency to
91%. Conversely, randomly selected feedback perform similarly across different
data sizes. We analyze this further in Section 7.2.

  5. 5.

Combining constraint-guided feedback with constraint solving further improves
accuracy by $\approx$5 percent points compared with only using feedback, and
by $\approx$2 percent points compared with the constraint-solver alone. Again
the addition of the constraint-solver results in almost perfect consistency of
99%. This setup reflects the incremental setup depicted in Figure 2, where the
constraint-solver can leverage the updated BeliefBank.

###  7.2 Analysis

Both constraint-solving and feedback can cause the overall system to flip
beliefs, with the goal that the overall system’s beliefs (in the BeliefBank)
are more accurate and consistent than those of the underlying model. However,
both mechanisms have the potential to flip beliefs in the wrong direction
also. We provide some analysis of both good and bad flips here.

##### The System Behaving as Desired:

As an illustration of desired behavior, the raw model incorrectly believes
that a pine is both a plant (correct) and a vertebrate (incorrect), when
queried. However, this violates a mutual exclusivity rule, so the constraint-
solver considers flipping one of these. Flipping “pine is a plant” from T to F
would result in numerous other violations, e.g., “pine is a tree” (which the
model also believes) would become violated. As a result, it prefers to
(correctly) disbelieve “pine is a vertebrate”, improving both the accuracy and
consistency of the BeliefBank.

##### Types of Error:

From an analysis of the data, we see that the majority of the raw model errors
are false positives - the MAQAw model generally answers (almost) all the
positive facts correctly (recall is $\approx$97%), but mistakenly thinks some
negative facts are also true (precision is $\approx$60%). These false
positives are typically rather unusual facts, e.g., “A poodle is a bathroom.”
(MAQAw’s answer: True). It is unsurprising that the model knows most of the
positive facts, as they are simple statements about common entities (“eagles
can fly”), likely seen in pre-training. However, the fact that the model makes
(what a person would view as) catastrophic errors when asked more unusual
questions, e.g., believing that “a poodle is made of fermented milk and
bacteria”, reveals that the PTLM’s grasp of the world is still incomplete and
problematic. The constraint mechanism proposed here essentially asks the model
to think about its answers and their consequences, so that it can spot
problems that the PTLM alone does not see, and repair them accordingly.

##### Sensitivity to Constraints and Weights:

The constraint reasoner also makes mistakes, sometimes flipping things the
wrong way so as to improve consistency, at the expense of accuracy. For
example, the raw model correctly believes that a “a rat is not a cat”.
However, the constraint solver then (incorrectly) flips this to "a rat is a
cat". It does this as there are multiple constraint rules weakly suggesting
rats are cats given the model’s other beliefs (”rats catch mice”, “rats have
tails”, “rats have fur”,…), which together add up. However, the model also
(correctly) believes “a rat is not a feline”, and there is a constraint that
“a cat is a feline”, so in principle this should prevent the belief “a rat is
a cat”. In practice, though, the constraint “a cat is a feline" does not have
infinite weight, so here the constraint mechanism allows it to be violated,
allowing the wrong conclusion (“a rat is a cat”) to be drawn.

Of course, one could increase the weight on the “a cat is a feline” constraint
to solve this particular problem, or add new constraint rules like “cats are
larger than squirrels”. In general, though, the effectiveness of constraint
reasoning is sensitive to both the number and weights on the constraints.
Although we have used automated techniques to re-calibrate the various weights
(Section 6.2), the system’s behavior remains sensitive to them. Finding more
effective ways to discover and appropriately tune constraints remains an open
problem.

##### Ambiguity:

Although we have avoided obvious cases of ambiguity, e.g., reasoning about
“bats”, there are smaller cases of ambiguity that may explain some of the raw
model’s errors. For example, although the model (incorrectly) believes "a
swallow is a fish", there is a fish called "a black swallower" that it may
have seen in pre-training, confusing it. Similarly, some unusual statements
such as “A penguin is a protein” (gold label is False, model believes is True)
are partially ambiguous, possibly contributing to the discrepancy.

##### Does the Model Know the Constraints?

Finally, we are checking consistency using externally defined constraints,
which the raw model may not itself be aware of (i.e., may not have acquired in
pre-training). For example, although the model may appear to be internally
inconsistent if it thinks a swallow is both a bird and a fish, this is only
true if it also knows these beliefs are mutually exclusive. To check for
internal inconsistency of a model’s behaviour, we would also need to check if
the model knew the violated constraints themselves, e.g., using additional
probes. Again this is an area for future exploration.

##### The influence of Feedback:

We currently can only speculate why feedback (i.e., providing relevant facts
to the model as context for question-answering) improves results. One
explanation is that feedback helps the model focus attention. For example,
reminding the model that "a swallow is not a fish" should help it realize that
"a swallow has gills" is False (Figure 1). However, we also observe
significant gains when feeding back random facts about the entity being
queried. Possibly these facts still encourage the model to focus on the
entity. Or, given the majority of beliefs are negative, the simple presence of
negation in the context may discourage false positive answers, the primary
cause of raw model errors.

##  8 Future Work

These findings were made using a restricted experimental setup where facts are
short sentences, constraints are provided, and both facts and constraints use
the same language so it is clear when constraints apply. To broaden the
approach, several developments are required:

  * •

A system would need to automatically gather constraints and queries.
Constraints might be mined from text, or even extracted from the model itself
by direct querying. Alternatively, domain-specific constraints (e.g., “X is a
bird $\rightarrow$ X has wings”) could be replaced with more generally
applicable constraint patterns (e.g., “X is a Y & Y has Z $\rightarrow$ X has
Z”), turning the domain-specific parts of constraints into facts that
themselves could reside in the BeliefBank.

  * •

Our queries have been simple facts. To fully utilize the BeliefBank for more
complex queries (e.g., “What color is the gas that plants produce?”), a
mechanism would be needed to decompose compound queries into their primitives
(“What gas do plants produce?”, “What color is that gas?”), and ensure they
were consistent with BeliefBank beliefs (e.g., “Oxygen is colorless”); in
other words, ensure complex answers were faithful to the BeliefBank
Subramanian et al. (2020).

  * •

In general, detecting when a constraint applies to a fact is non-trivial, and
requires appropriate machinery for identifying sentence/constraint alignment,
e.g., an entailment model Seo et al. (2016).

  * •

Although constraint/model weights are automatically calibrated (Section 6.2),
the system is still sensitive to those weights and can make mistakes (Section
7.2). Improved mechanisms for automatic calibration would significantly help.

  * •

Our system is passive, in that it answers questions and uses answers in the
order they are provided. An alternative design would be an active system that,
given a new question, actively identifies which auxiliary questions would be
most useful to also ask, to help answer the main question (either with
constraint solving or feedback).

Finally, scaling to a massive BeliefBank would require addressing several
engineering requirements, including bounding the constraint checking, and not
re-asking every prior question as new answers were discovered. For example, we
might only check constraints and their implications up to depth $D$ from a new
fact, rather than exhaustively. Similarly, the system may maintain a
BeliefBank where earlier answers were generated using only feedback available
at that point, rather than re-asking all earlier questions as new knowledge
becomes available.

##  9 Conclusions

PTLMs can be inconsistent in their answers to probing questions, and can still
give (what to a person appears as) naively wrong answers to unusual questions.
This work is a first step to alleviating these problems. By augmenting a PTLM
with a persistent, global memory - the Belief Bank - and using it for both
constraint-solving and feedback, we have shown that both consistency and
accuracy can be significantly improved. The additional memory layer can
loosely be seen as a simple “mental model”, constructed from the PTLM’s raw
answers.

Our experiments were conducted in a restricted, controlled setting, and
further development is needed to scale to larger and more complex tasks.
Nevertheless, the work here is significant as it is a first step towards
endowing models with a globally consistent notion of belief, allowing them to
construct a more coherent picture of the world. The dataset is available at
https://allenai.org/data/beliefbank.

## References

  * Andersen and Pretolani (2001) Kim Allan Andersen and Daniele Pretolani. 2001.  Easy cases of probabilistic satisfiability.  _Annals of Mathematics and Artificial Intelligence_ , 33(1):69–91.
  * Asai and Hajishirzi (2020) Akari Asai and Hannaneh Hajishirzi. 2020.  Logic-guided data augmentation and regularization for consistent question answering.  _ArXiv_ , abs/2004.10157.
  * Berant et al. (2010) Jonathan Berant, Ido Dagan, and Jacob Goldberger. 2010.  Global learning of focused entailment graphs.  In _ACL_ , pages 1220–1229.
  * Bhakthavatsalam et al. (2021) Sumithra Bhakthavatsalam, Daniel Khashabi, Tushar Khot, B. D. Mishra, Kyle Richardson, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord, and P. Clark. 2021.  Think you have solved direct-answer question answering? Try ARC-DA, the direct-answer AI2 reasoning challenge.  _ArXiv_ , abs/2102.03315.
  * Broecheler et al. (2010) Matthias Broecheler, Lilyana Mihalkova, and L. Getoor. 2010.  Probabilistic similarity logic.  In _UAI_.
  * Davison et al. (2019) Joe Davison, Joshua Feldman, and Alexander Rush. 2019.  Commonsense knowledge mining from pretrained models.  In _EMNLP_.
  * De Moura and Bjørner (2008) Leonardo De Moura and Nikolaj Bjørner. 2008.  Z3: An efficient SMT solver.  In _International conference on Tools and Algorithms for the Construction and Analysis of Systems_ , pages 337–340. Springer.
  * Du et al. (2019) X. Du, Bhavana Dalvi Mishra, Niket Tandon, Antoine Bosselut, Wen-tau Yih, P. Clark, and Claire Cardie. 2019.  Be consistent! Improving procedural text comprehension using label consistency.  In _NAACL_.
  * Elazar et al. (2021) Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, E. Hovy, H. Schutze, and Yoav Goldberg. 2021.  Measuring and improving consistency in pretrained language models.  _ArXiv_ , abs/2102.01017.
  * Ettinger (2020) Allyson Ettinger. 2020.  What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models.  _Transactions of the Association for Computational Linguistics_ , 8:34–48.
  * Fellbaum (2005) Christiane Fellbaum. 2005.  Wordnet and wordnets.  In Keith et al. Brown, editor, _Encyclopedia of Language and Linguistics, Second Edition_. Elsevier, Oxford.
  * Hansen and Jaumard (2000) Pierre Hansen and Brigitte Jaumard. 2000.  Probabilistic satisfiability.  In _Handbook of Defeasible Reasoning and Uncertainty Management Systems_ , pages 321–367. Springer.
  * Henaff et al. (2016) Mikael Henaff, J. Weston, Arthur D. Szlam, Antoine Bordes, and Y. LeCun. 2016.  Tracking the world state with recurrent entity networks.  In _ICLR_.
  * Johnson-Laird (1983) P. Johnson-Laird. 1983.  Mental models : Towards a cognitive science of language.
  * Kassner and Schütze (2020) Nora Kassner and H. Schütze. 2020.  Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly.  In _ACL_.
  * Khashabi et al. (2020) Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, P. Clark, and Hannaneh Hajishirzi. 2020.  UnifiedQA: Crossing format boundaries with a single QA system.  In _EMNLP_.
  * Li et al. (2019) Tao Li, Vivek Gupta, Maitrey Mehta, and Vivek Srikumar. 2019.  A logic-driven framework for consistency of neural models.  In _EMNLP_.
  * Minervini and Riedel (2018) Pasquale Minervini and S. Riedel. 2018.  Adversarially regularising neural NLI models to integrate logical background knowledge.  In _CoNLL_.
  * Muiño (2011) David Picado Muiño. 2011.  Measuring and repairing inconsistency in probabilistic knowledge bases.  _International Journal of Approximate Reasoning_ , 52(6):828–840.
  * Petroni et al. (2019) F. Petroni, Tim Rocktäschel, Patrick Lewis, A. Bakhtin, Yuxiang Wu, Alexander H. Miller, and S. Riedel. 2019.  Language models as knowledge bases?  In _EMNLP_.
  * Pujara et al. (2013) J. Pujara, H. Miao, L. Getoor, and William W. Cohen. 2013.  Knowledge graph identification.  In _International Semantic Web Conference_.
  * Ravichander et al. (2020) Abhilasha Ravichander, Eduard Hovy, Kaheer Suleman, Adam Trischler, and Jackie Chi Kit Cheung. 2020.  On the systematicity of probing contextualized word representations: The case of hypernymy in BERT.  In _STARSEM_.
  * Ribeiro et al. (2019) Marco Tulio Ribeiro, Carlos Guestrin, and Sameer Singh. 2019.  Are red roses red? Evaluating consistency of question-answering models.  In _ACL_.
  * Roberts et al. (2020) Adam Roberts, Colin Raffel, and Noam Shazeer. 2020.  How much knowledge can you pack into the parameters of a language model?  In _EMNLP_.
  * Seo et al. (2016) Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016.  Bidirectional attention flow for machine comprehension.  In _ICLR_.
  * Shwartz et al. (2020) Vered Shwartz, Peter West, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020\.  Unsupervised commonsense question answering with self-talk.  In _EMNLP_.
  * Speer et al. (2017) Robyn Speer, Joshua Chin, and Catherine Havasi. 2017.  Conceptnet 5.5: An open multilingual graph of general knowledge.  In _AAAI_.
  * Subramanian et al. (2020) Sanjay Subramanian, Ben Bogin, Nitish Gupta, Tomer Wolfson, Sameer Singh, Jonathan Berant, and Matt Gardner. 2020.  Obtaining faithful interpretations from compositional neural networks.  In _ACL_.
  * Sukhbaatar et al. (2015) Sainbayar Sukhbaatar, Arthur D. Szlam, J. Weston, and R. Fergus. 2015.  End-to-end memory networks.  In _NeurIPS_.
  * Tandon et al. (2018) Niket Tandon, Bhavana Dalvi, Joel Grus, Wen-tau Yih, Antoine Bosselut, and Peter Clark. 2018.  Reasoning about actions and state changes by injecting commonsense knowledge.  In _EMNLP_.
  * Thimm (2009) Matthias Thimm. 2009.  Measuring inconsistency in probabilistic knowledge bases.  In _UAI_.
  * Thimm (2013) Matthias Thimm. 2013.  Inconsistency measures for probabilistic logics.  _Artificial Intelligence_ , 197:1–24.


# Cabrita: closing the gap for foreign languages

Celio Larcher  Marcos Piau  Paulo Finardi  Pedro Gengo  Piero Esposito
Vinicius Caridá

(email: {celiolarcher, marcos.piau.vieira, pfinardi, pedro.gengo.lourenco,
piero.skywalker<EMAIL_ADDRESS>)

###### Abstract

The strategy of training the model from scratch in a specific language or
domain serves two essential purposes: i) enhancing performance in the
particular linguistic or domain context, and ii) ensuring effective
tokenization. The main limitation inherent to this approach lies in the
associated cost, which can reach six to seven-digit dollar values, depending
on the model size and the number of parameters involved.

The main solution to overcome the cost challenge is to rely on available pre-
trained models, which, despite recent advancements such as the LLaMA and
LLaMA-2 models, still demonstrate inefficiency for certain specific domain
problems or prove ineffective in scenarios involving conversational memory
resources, given the large number of tokens required to represent text.

To overcome this issue, we present a methodology named Cabrita, which, as our
research demonstrates, successfully addresses the performance and efficient
tokenization problem, all at an affordable cost. We believe that this
methodology can be applied to any transformer-like architecture model. To
validate the study, we conducted continuous pre-training exclusively using
Portuguese text on a 3-billion-parameter model known as OpenLLaMA, resulting
in a model named openCabrita 3B. The openCabrita 3B also features a new
tokenizer that results in a significant reduction in the number of tokens
required to represent the text. In our assessment, for few-shot learning
tasks, we achieved similar results with this 3B model compared to a
traditional continuous pre-training approach as well as to 7B models English
pre-trained models.

##  1 Introduction

While the Portuguese language does not suffer from a lack of data for training
transformer-based language models, a native speaker can readily perceive
limitations in text generation and performance of pre-trained models
predominantly based on English language data. Following the research conducted
in Sabiá [1], we share the perspective that the conventional practice of
simultaneously pre-training models in multiple languages fails to capture the
cultural richness and intrinsic knowledge of each language.

As demonstrated in the study [2], language-specific adapted tokenizers have
the potential to significantly enhance the monolingual performance of
multilingual models.

Considering that pre-training a model in a single language with a specific
tokenizer leads to notably superior performance, it’s natural to question why
there aren’t more language-specific models. However, the answer lies in a
crucial factor: cost. An example of this is the cost of the LLaMa 2 7B model,
which required a total of 184,320 hours of processing on GPU A-100 80-GB.
Assuming a conservative estimate that 1 hour of this GPU costs $1, the
resulting cost approaches $200,000 significantly. This underscores how the
substantial investments required to create and maintain language-specific
language models can pose a significant financial challenge.

This article investigates an alternative approach to zero-shot pre-training
based on three fundamental principles: low cost, performance optimization, and
efficient tokenization. This approach aims to enable the model to use fewer
tokens and minimize inference time compared to existing multilingual models.

##  2 Methodology

###  2.1 LLaMA Models

The LLaMA models are Large Language Models developed by the Meta AI’s team
[3]. Based on the well-established transformers architecture it incorporates
several novel training and inference mechanisms, notably the use of Pre-
normalization [4], the SwiGlu activation function [5] and the inclusion of
Rotary Embeddings [6].

The primary distinction of LLaMA models lies in the magnitude of pre-training
data utilized during their development, scaling from the standard 300 billion
tokens in the optimal Chinchila’s law pattern [7] to an extensive 1 trillion
tokens. Although this setup incurs increased resource costs during training,
it significantly improves the state-of-the-art (SOTA) performance for all
model sizes at the time of release. As a result, inference costs are reduced,
as smaller models can achieve comparable levels of performance, thereby
justifying the theoretical non-optimal behavior during pre-training.

After its introduction, several other Large Language Models have been trained
following a similar approach, facilitating further advancements in SOTA
standards, particularly for smaller models. For this study, we adopt the
OpenLLaMA implementation[8], an open-source Apache 2.0 alternative with the
same recipe as the open-science-only LLaMA.

###  2.2 openCabrita Models

Despite the proliferation of numerous open source Large Language Models in
recent years, the availability of generative models for non-English languages
remains a persistent problem.

While employing English models for other languages might seem feasible, their
performance typically falls short of optimal, even when utilizing multilingual
options or implementing Instruction Fine Tuning [9] in the target language
[10, 11, 1].

Taking this aspect into consideration, this work introduces the openCabrita
models, a collection of checkpoints derived from the OpenLLama models. These
models undergo additional pre-training using a Portuguese corpus, inspired by
the methodology proposed in [1], while also incorporating additional tokenizer
adaption. The training procedure will be elaborated on in the next
subsections.

####  2.2.1 Tokenizer adaptation

One of the main challenges of adapting a Large Language Model to a new
language that is underrepresented in its pre-training is the tokenizer
behavior. This is the case for the Portuguese language in the OpenLLaMA model,
where despite the 1 trillion tokens used, just a small part were non-English.
The default tokenizer, in such cases, tends to be overly verbose for non-
English examples, necessitating the division of the text into too small parts.

This behavior is undesirable for two primary reasons: i) it shatters the
amount of information contained within each sub-unit processed by the model,
potentially making it more challenging to understand long-term relations
within the context; and ii) it escalates the computational cost of both
training and inference, as it increases the number of tokens that need to be
processed for the same text sample.

However, the inclusion of a new tokenizer in a pre-trained model poses a
challenge, as there exists a direct mapping between the tokens used in pre-
training and the model’s embedding representation. Merely replacing the
tokenizer would break this mapping, likely resulting in odd model behavior and
potentially requiring a considerable amount of additional pre-training to
learn the new association pattern.

To address this issue, the approach taken in this work involves adapting the
tokenizer to the new language while preserving the knowledge of the original
language already embedded in the model. Inspired by the work of [11], this
process involved three steps:

  * •

Training a new tokenizer exclusively on Portuguese data sourced from
Wikipedia. SentencePiece [12] implementation was used for this purpose,
resulting in a tokenizer with 40,000 tokens.

  * •

Merging the original tokenizer with the newly created Portuguese tokenizer.
During this merging process, all tokens present in the original tokenizer were
retained, and the new Portuguese tokens that were not yet represented in the
vocabulary were appended until a total of 52,000 tokens were reached. The
precedence level of each merge – BPE Score in the SentencePiece implementation
– was maintained from both tokenizers, given that this led to a better token
compression level.

  * •

Resizing the model’s embedding and head matrix to accommodate the newly added
tokens. The new rows corresponding to the Portuguese tokens were appended to
the end of the original matrix to ensure that the existing token mapping
remained unaffected.

The result was a bilingual tokenizer, able to handle both English and
Portuguese content efficiently.

####  2.2.2 Continuous pre-training

We use the Portuguese subset of the mC4 dataset [13] (hereafter called mC4-pt)
as our pretraining corpus. Following the recipe in [1], we also apply quality
filters based on MassiveText [14] to ensure our model is trained using high-
quality documents.

First, we normalize each document independently using ftfy [15]. Then, we
apply the quality filters. Unlike MassiveText, which only considers the stop
words {the, be, to, of, and, that, have, with}, we use the Portuguese stop
words from nltk [16]; we also keep only documents with at least 200 unique
tokens, following the the procedure outlined in [1]. A document is kept in the
dataset if it does not fall into any of the quality filters. Table 7 shows the
contribution of each filter to the final filtered dataset.

To estimate the total number of training tokens without processing the entire
dataset, we assume that the remaining shards have similar sizes and behavior
with respect to the quality filters. Based on this estimation, the unfiltered
mC4-pt training dataset consists of approximately 169 million documents and
188 billion tokens, which reduces to 133 million documents and 170 billion
tokens after applying the low-quality filters. To convert these documents into
training examples, we add <|bos|> and <|eos|> tokens to each document, and
then pack these documents into sequences of 2048 tokens.

We perform the continuous pre-training step on TPUs using the EasyLM, which is
a framework built in JAX/Flax and is a one-stop solution for training LLMs on
TPU and GPU devices.

To perform continued pre-training in the Portuguese language, the original
script and hyper-parameters of OpenLLaMA training were employed. The model was
initialized with the OpenLLaMA weights and then trained on an additional 7
billion tokens extracted from the Portuguese dataset.

The learning objective was the standard causal language modeling loss with the
learning rate set to increase linearly from 0 to 3e-4 over 2,000 steps,
followed by a cosine decrease to 3e-5 for the remaining 248,000 steps. The
optimization algorithm used was AdamW, with a weight decay of 0.1, $\beta_{1}$
set to 0.9, $\beta_{2}$ set to 0.99, and a gradient clip value of 1.0.

For training, a TPU v3-8 was utilized, with batches of 16, each containing a
sequence of 2048 tokens. To achieve the target of 2048 samples in a batch, 128
accumulation steps were performed. This configuration resulted in a throughput
of 7,900 tokens/second for the 3B model.

##  3 Evaluation

###  3.1 Tokenizer efficiency

For evaluating how alterations to the tokenizer affect text representation, a
set of experiments were performed. These experiments involved the comparison
of various tokenizer alternatives coming from different models and trained on
distinct corpora. Figure 1 presents a comparison of the token count required
to represent 7400 words of the Constitutional law of the USA in both English
and Portuguese (translated) for each of these tokenizer variations.

Figure 1: Tokenizer efficiency: the X axis show the number of tokens required
to represent 7400 words of the Constitutional law of the USA in English, while
the Y axis shows the same, but using a Portuguese translated version. The size
of each sphere represents the vocabulary size of each tokenizer.

It can be noted that the standard OpenLLaMA implementation demonstrates highly
inefficient behavior when applied to Portuguese data. It necessitates over 50%
more tokens for representing the same text, in contrast to the multilingual
Bloom and Portuguese pre-trained models (e.g. BERTimbau [17], PTT5 [18],
Bertaú [19]). While certain other LLaMA-based implementations exhibit improved
token counts, such as MPT [20] and Falcon [21], some inefficiency remains.
This observation holds true even in comparison with chatGPT [22], which
possesses a larger vocabulary than these LLaMA implementations.

In contrast to these choices, the Cabrita tokenizer demonstrated the ability
to enhance the OpenLLaMA token requirements. It achieved a reduction of over
35% in the token requirements, bringing it in line with the token counts of
the Bloom and native Portuguese implementations.

Furthermore, upon examining the token count in English, the Cabrita tokenizer
even exhibited a slight improvement in the token count when compared with the
original OpenLLaMA tokenizer, with no indication of degradation in this
regard.

Ultimately, the Cabrita approach presents an excellent trade-off between
Portuguese and English considerations, especially when seeking models with
comparable vocabulary sizes (52.000 tokens for Cabrita versus 250.680 tokens
for Bloom).

###  3.2 Portuguese Benchmark results

To assess the efficacy of this proposal, 8 Portuguese evaluation datasets were
employed across a variety of tasks: ASSIN 2 RTE and STS [23], FaQuAD [24],
TweetSentBr [25] are inherently in Portuguese, while for AG News [26], IMDB
[27], SST2 [28] and BoolQ [29] translated versions provided by the authors of
the Sabiá series paper were utilized. Poeta’s benchmark [1] was not executed
in our study due to the unavailability of its implementation at the time of
our publication.

Table 1 provides an overview of the datasets along with the specific few-shot
configuration employed in each experiment.

Table 1: Number of few-show samples and main metric for each task

Task | Type | Metric |  Few-shot Samples
---|---|---|---
FaQuAD | Extractive QA | F1 | 4
ASSIN 2 RTE | Binary classification | F1-macro | 20
ASSIN 2 STS | Regression | Pearson | 15
TweetSentBr | Multiclass classification | F1-macro | 30
SST2 | Binary classification | Accuracy | 34
IMDB | Binary classification | Accuracy | 2
AGNews | Multiclass classification | Accuracy | 12
BoolQ | Binary classification | Accuracy | 4

A diverse array of tasks was carefully selected, spanning from binary and
multi-class classification to Extractive Question Answering (QA). However, the
absence of a comprehensive Portuguese benchmark prevented the utilization of a
more extensive task set.

The Table 2 exhibits the performance comparison of openCabrita3B with the base
openLLaMA3B and a subsequent pre-training version that employs the openLLaMA
tokenizer (referred to as openCabrita3BPTOnly).

Table 2: This Table shows the results obtained for each task and each model.
The configurations are the ones presented in the Table 1.

Task | open LLaMA3b | openCabrita3B PTOnly | open Cabrita3B
---|---|---|---
FaQuAD | 58.71 | 66.72 | 62.16
ASSIN 2 RTE | 43.38 | 38.83 | 67.36
ASSIN 2 STS | 18.22 | 12.38 | 12.24
TweetSentBr | 44.48 | 47.60 | 54.03
SST2 | 86.69 | 90.25 | 89.00
IMDB | 80.60 | 78.28 | 86.02
AGNews | 60.39 | 67.76 | 64.98
BoolQ | 61.92 | 64.09 | 63.11

The results are mixed but somewhat favoring openCabrita3B. When openCabrita3B
performs better, it is notably superior, as in ASSIN 2 RTE, TweetSentBr, and
IMDB tasks. When openCabrita3BPTOnly outperforms, openCabrita3B is usually
close, just a few scores behind. While ASSIN 2 STS is the only task where the
original openLLaMA3b shows a better performance, which is unexpected but
consistent with what was observed in other experiments [1].

Nevertheless, it is evident that the Cabrita approach, which involves adapting
the tokenizer, can offer at least a comparable performance level in contrast
to the conventional continued pre-training. This equivalent performance
improvement also comes with the added benefit of enhanced inference
efficiency.

Table 3 shows the same set of tasks, with a focus on comparing the performance
of openCabrita3B to the outcomes presented in the Sabiá paper. It’s important
to highlight that while the datasets remain the same, possible variations in
the execution scripts require a cautious approach when drawing comparisons.

Table 3: This Table shows the results obtained for each task and each model
for Portuguese datasets. The configurations are presented in Table 1. The *
symbol represents results that come from a different running script than ours,
so the results need to be compared carefully.

Model | open Cabrita3B | GPT-J* | Sabiá-J* | LLaMa-7B* | Sabiá-7B*
---|---|---|---|---|---
FaQuAD | 62.16 | 59.52 | 69.28 | 77.38 | 77.43
ASSIN 2 RTE | 67.36 | 54.88 | 35.49 | 56.82 | 64.87
ASSIN 2 STS | 12.24 | 17.86 | 22.97 | 7.39 | 13.63
TweetSentBr | 54.03 | 20.98 | 64.16 | 44.19 | 67.17
SST2 | 89.00 | 83.94 | 87.16 | 88.76 | 90.69
IMDB | 86.02 | 72.68 | 90.86 | 86.92 | 92.7
AGNews | 64.98 | 64.15 | 84.3 | 76.94 | 83.28
BoolQ | 63.11 | 48.75 | 51.53 | 57.37 | 64.07

The performance of openCabrita3B seems very satisfactory for a model of its
size. It consistently outperforms GPT-J, a model with 6B parameters, and
demonstrates competitiveness with LLaMA-7B. In the case of continued pre-
training versions of these models, openCabrita3B closely approaches the
performance of Sabiá-J, while slightly lagging behind Sabiá-7B.

These results do not cast any negative light on openCabrita3B, as it performs
admirably despite having only half the parameter size compared to the other
four options.

###  3.3 English Benchmark results

To assess the efficacy of this approach and demonstrate its bilingual
capabilities, we conducted a series of evaluations using diverse English
datasets, ensuring a comprehensive understanding of its performance. Table 4
presents the performance of openCabrita in comparison with models trained
mainly in English and without any fine-tuning for a specific language.

Despite a slight decrease in performance for English, the model’s capabilities
remained robust and competitive across languages. Furthermore, we investigated
the impact of different tokenizers—Open Llama and Cabrita—on the model’s
output. Encouragingly, both tokenizers yielded highly consistent results,
indicating that the introduction of new tokens had minimal impact on the
model’s overall performance. This observation highlights the model’s
resilience and suggests its versatility for diverse applications without
significant trade-offs in performance.

Table 4: This Table shows the results obtained for each task and each model
for English datasets. The * symbol represents results that come from a
different running script than ours, so the results need to be compared
carefully.

Task (metric) | open Cabrita3B | openCabrita3B PTOnly | openLLama 3B* | GPT-J* | LLaMa-7B*
---|---|---|---|---|---
anli_r1 (acc) | 35.0 | 33.2 | 33.0 | 32.0 | 35.0
anli_r2 (acc) | 34.0 | 33.5 | 32.0 | 34.0 | 34.0
anli_r3 (acc) | 36.0 | 33.4 | 35.0 | 35.0 | 37.0
arc_challenge (acc) | 28.0 | 28.1 | 34.0 | 34.0 | 39.0
arc_challenge (acc_norm) | 32.0 | 31.9 | 37.0 | 37.0 | 41.0
arc_easy (acc) | 59.0 | 58.0 | 69.0 | 67.0 | 62.0
arc_easy (acc_norm) | 54.0 | 53.6 | 65.0 | 62.0 | 52.0
boolq (acc) | 63.0 | 62.9 | 68.0 | 66.0 | 75.0
hellaswag (acc) | 41.0 | 41.1 | 49.0 | 50.0 | 56.0
hellaswag (acc_norm) | 53.0 | 54.0 | 67.0 | 66.0 | 73.0
openbookqa (acc) | 21.0 | 20.6 | 27.0 | 29.0 | 29.0
openbookqa (acc_norm) | 31.0 | 32.0 | 40.0 | 38.0 | 41.0
piqa (acc) | 69.0 | 69.4 | 75.0 | 75.0 | 78.0
piqa (acc_norm) | 69.0 | 69.5 | 76.0 | 76.0 | 78.0
record (em) | 83.0 | 83.0 | 88.0 | 88.0 | 91.0
record (f1) | 82.0 | 82.2 | 89.0 | 89.0 | 91.0
rte (acc) | 55.0 | 52.7 | 58.0 | 54.0 | 56.0
thuthfulqa_mc (mc1) | 25.0 | 23.1 | 22.0 | 20.0 | 21.0
thuthfulqa_mc (mc2) | 38.0 | 37.1 | 35.0 | 36.0 | 34.0
wic (acc) | 50.0 | 50.0 | 48.0 | 50.0 | 50.0
winogrande (acc) | 56.0 | 58.7 | 62.0 | 64.0 | 68.0

##  4 Conclusion and next steps

This work is currently in progress and requires further development in terms
of a more comprehensive set of models and experiments before arriving at any
definitive conclusions. However, the outcomes obtained from modifying the
tokenizer prior to performing continuous pre-training seem very promising.

Even in the absence of state-of-the-art performance enhancements, the
openCabrita3b reaches a comparable metrics level when compared to the standard
continuous pre-training approach while also improving inference time — an
imperative factor in the context of deploying models for such applications
that is commonly overlooked.

Looking ahead, the authors have outlined the following as their forthcoming
initiatives:

  * •

Test the same strategy in a more diverse set of base models

  * •

Establish some type of benchmark and reperform runs for other models in order
to have a comparison with the same running methodology

##  5 Limitations

Scaling beyond 3B and for other languages Despite our best efforts, certain
ranges of model sizes and experiments remain beyond our reach due to budget
constraints. Consequently, our experimentation was limited to the Portuguese
language and 3B size models.

But, since the good results obtained, we are firmly convinced that employing
larger-scale models could yield results at least as promising. This line of
thinking could similarly be applied to other foreign languages, particularly
considering the successful experiment with Chinese presented in [11].

Comparison to other base models and adapting strategies Due to the absence of
a structured benchmark like Harness [30] for the Portuguese language, our
capacity to compare with other models and strategies for the same tasks is
confined to the experiments we are able to perform. Consequently, the scope of
comparisons we can undertake in this paper is restricted, both in terms of
quantity and depth.

##  6 Acknowledgments

We thank Google Cloud for the TPU grant through the TRC program.

## References

  * [1] R. Pires, H. Abonizio, T. S. Almeida, and R. Nogueira, “Sabiá: Portuguese large language models,” 2023.
  * [2] P. Rust, J. Pfeiffer, I. Vulić, S. Ruder, and I. Gurevych, “How good is your tokenizer? on the monolingual performance of multilingual language models,” in _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. Online: Association for Computational Linguistics, Aug. 2021, pp. 3118–3135. [Online]. Available: https://aclanthology.org/2021.acl-long.243
  * [3] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample, “Llama: Open and efficient foundation language models,” 2023.
  * [4] B. Zhang and R. Sennrich, “Root mean square layer normalization,” 2019.
  * [5] N. Shazeer, “Glu variants improve transformer,” 2020.
  * [6] J. Su, Y. Lu, S. Pan, A. Murtadha, B. Wen, and Y. Liu, “Roformer: Enhanced transformer with rotary position embedding,” 2022.
  * [7] J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre, “Training compute-optimal large language models,” 2022.
  * [8] X. Geng and H. Liu, “Openllama: An open reproduction of llama,” May 2023. [Online]. Available: https://github.com/openlm-research/open_llama
  * [9] S. Longpre, L. Hou, T. Vu, A. Webson, H. W. Chung, Y. Tay, D. Zhou, Q. V. Le, B. Zoph, J. Wei, and A. Roberts, “The flan collection: Designing data and methods for effective instruction tuning,” 2023.
  * [10] N. Muennighoff, T. Wang, L. Sutawika, A. Roberts, S. Biderman, T. L. Scao, M. S. Bari, S. Shen, Z.-X. Yong, H. Schoelkopf, X. Tang, D. Radev, A. F. Aji, K. Almubarak, S. Albanie, Z. Alyafeai, A. Webson, E. Raff, and C. Raffel, “Crosslingual generalization through multitask finetuning,” 2023.
  * [11] Y. Cui, Z. Yang, and X. Yao, “Efficient and effective text encoding for chinese llama and alpaca,” 2023.
  * [12] T. Kudo and J. Richardson, “SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing,” in _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_. Brussels, Belgium: Association for Computational Linguistics, Nov. 2018, pp. 66–71. [Online]. Available: https://aclanthology.org/D18-2012
  * [13] L. Xue, N. Constant, A. Roberts, M. Kale, R. Al-Rfou, A. Siddhant, A. Barua, and C. Raffel, “mT5: A massively multilingual pre-trained text-to-text transformer,” in _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Online: Association for Computational Linguistics, Jun. 2021, pp. 483–498. [Online]. Available: https://aclanthology.org/2021.naacl-main.41
  * [14] J. W. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, F. Song, J. Aslanides, S. Henderson, R. Ring, S. Young, E. Rutherford, T. Hennigan, J. Menick, A. Cassirer, R. Powell, G. van den Driessche, L. A. Hendricks, M. Rauh, P.-S. Huang, A. Glaese, J. Welbl, S. Dathathri, S. Huang, J. Uesato, J. Mellor, I. Higgins, A. Creswell, N. McAleese, A. Wu, E. Elsen, S. Jayakumar, E. Buchatskaya, D. Budden, E. Sutherland, K. Simonyan, M. Paganini, L. Sifre, L. Martens, X. L. Li, A. Kuncoro, A. Nematzadeh, E. Gribovskaya, D. Donato, A. Lazaridou, A. Mensch, J.-B. Lespiau, M. Tsimpoukelli, N. Grigorev, D. Fritz, T. Sottiaux, M. Pajarskas, T. Pohlen, Z. Gong, D. Toyama, C. de Masson d’Autume, Y. Li, T. Terzi, V. Mikulik, I. Babuschkin, A. Clark, D. de Las Casas, A. Guy, C. Jones, J. Bradbury, M. Johnson, B. Hechtman, L. Weidinger, I. Gabriel, W. Isaac, E. Lockhart, S. Osindero, L. Rimell, C. Dyer, O. Vinyals, K. Ayoub, J. Stanway, L. Bennett, D. Hassabis, K. Kavukcuoglu, and G. Irving, “Scaling language models: Methods, analysis & insights from training gopher,” 2022.
  * [15] R. Speer, “ftfy,” Zenodo, 2019, version 5.5. [Online]. Available: https://doi.org/10.5281/zenodo.2591652
  * [16] S. Bird, E. Klein, and E. Loper, _Natural language processing with Python: analyzing text with the natural language toolkit_. " O’Reilly Media, Inc.", 2009.
  * [17] F. Souza, R. Nogueira, and R. Lotufo, “Bertimbau: Pretrained bert models for brazilian portuguese,” in _Intelligent Systems_ , R. Cerri and R. C. Prati, Eds. Cham: Springer International Publishing, 2020, pp. 403–417.
  * [18] D. Carmo, M. Piau, I. Campiotti, R. F. Nogueira, and R. de Alencar Lotufo, “PTT5: pretraining and validating the T5 model on brazilian portuguese data,” _CoRR_ , vol. abs/2008.09144, 2020. [Online]. Available: https://arxiv.org/abs/2008.09144
  * [19] P. Finardi, J. D. Viegas, G. T. Ferreira, A. F. Mansano, and V. F. Caridá, “Bertaú: Itaú BERT for digital customer service,” _CoRR_ , vol. abs/2101.12015, 2021. [Online]. Available: https://arxiv.org/abs/2101.12015
  * [20] M. Team. (2023) Introducing mpt-7b: A new standard for open-source, commercially usable llms. [Online]. Available: https://www.mosaicml.com/blog/mpt-7b
  * [21] G. Penedo, Q. Malartic, D. Hesslow, R. Cojocaru, A. Cappelli, H. Alobeidli, B. Pannier, E. Almazrouei, and J. Launay, “The refinedweb dataset for falcon llm: Outperforming curated corpora with web data, and web data only,” 2023.
  * [22] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe, “Training language models to follow instructions with human feedback,” 2022\.
  * [23] L. Real, E. Fonseca, and H. Gonçalo Oliveira, “The assin 2 shared task: A quick overview,” in _Computational Processing of the Portuguese Language_ , P. Quaresma, R. Vieira, S. Aluísio, H. Moniz, F. Batista, and T. Gonçalves, Eds. Cham: Springer International Publishing, 2020, pp. 406–412.
  * [24] H. F. Sayama, A. V. Araujo, and E. R. Fernandes, “Faquad: Reading comprehension dataset in the domain of brazilian higher education,” in _2019 8th Brazilian Conference on Intelligent Systems (BRACIS)_ , 2019, pp. 443–448.
  * [25] H. B. Brum and M. das Graças Volpe Nunes, “Building a sentiment corpus of tweets in brazilian portuguese,” _CoRR_ , vol. abs/1712.08917, 2017. [Online]. Available: http://arxiv.org/abs/1712.08917
  * [26] X. Zhang, J. J. Zhao, and Y. LeCun, “Character-level convolutional networks for text classification,” _CoRR_ , vol. abs/1509.01626, 2015. [Online]. Available: http://arxiv.org/abs/1509.01626
  * [27] A. L. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts, “Learning word vectors for sentiment analysis,” in _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_. Portland, Oregon, USA: Association for Computational Linguistics, Jun. 2011, pp. 142–150. [Online]. Available: https://aclanthology.org/P11-1015
  * [28] R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Ng, and C. Potts, “Recursive deep models for semantic compositionality over a sentiment treebank,” in _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_. Seattle, Washington, USA: Association for Computational Linguistics, Oct. 2013, pp. 1631–1642. [Online]. Available: https://aclanthology.org/D13-1170
  * [29] C. Clark, K. Lee, M. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova, “Boolq: Exploring the surprising difficulty of natural yes/no questions,” _CoRR_ , vol. abs/1905.10044, 2019. [Online]. Available: http://arxiv.org/abs/1905.10044
  * [30] L. Gao, J. Tow, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, K. McDonell, N. Muennighoff, J. Phang, L. Reynolds, E. Tang, A. Thite, B. Wang, K. Wang, and A. Zou, “A framework for few-shot language model evaluation,” Sep. 2021. [Online]. Available: https://doi.org/10.5281/zenodo.5371628

##  Appendix A Tokenizer Comparison

Table 5: Comparison between models tokenizers. The data is the first 7400
words of the Constitutional law of the United States which was translated for
the Portuguese language.

Model | Vocab Size | Tokens (Portuguese data) | Tokens (English data) |
---|---|---|---|---
GPT-2 | 50257 | 15036 | 8126 |
MPT | 50254 | 12470 | 8313 |
Falcon | 65024 | 11531 | 8240 |
OpenLLaMA | 32000 | 15648 | 10280 |
openCabrita 3B | 52000 | 9666 | 9017 |
BERTaú | 34100 | 10573 | 13054 |
BERTimbaú | 29794 | 10057 | 13725 |
PTT5 | 32100 | 9447 | 13720 |
MT5 | 250100 | 13010 | 10688 |
BLOOM | 250680 | 8582 | 8337 |
chatGPT | 100277 | 11351 | 8136 |

Table 6: Comparison between tokenization strategy. The data is the first 7400
words of the Constitutional law of the United States which was translated for
the Portuguese language.

Model | Vocab Size | Tokens (Portuguese data) | Tokens (English data) |
---|---|---|---|---
openCabrita 3B | 52000 | 9666 | 9017 |
Cabrita 0x | 52000 | 10169 | 9753 |
Cabrita 0.5x | 52000 | 9694 | 9021 |
Cabrita 2x | 52000 | 9632 | 9010 |

##  Appendix B mC4-pt cleaning filters

Table 7: Impact of each low-quality filter on the mC4-pt final clean training
dataset.

Description | # (10 shards) | % | # (1024 shards estimate) |
---|---|---|---|---
Total examples | 1,652,725 | 100 | 169,239,040 |
At least one "bad example filter" | 361,732 | 21.89 | 37,041,357 |
Less than 200 unique tokens | 269,893 | 16.33 | 27,637,043 |
Number of words outside range of 50 to 100,000 | 170,928 | 10.34 | 17,503,027 |
Less than 50 words | 170,928 | 10.34 | 17,503,027 |
Less than 80% of words containing a alphabetic character | 80,836 | 4.89 | 8,277,606 |
More than 30% of lines ending with an ellipsis sign ("…") | 28,289 | 1.71 | 2,896,794 |
Mean word length outside range of 3 to 10 characters | 22,280 | 1.35 | 2,281,472 |
Less than 2 nltk Portuguese stopwords | 18,616 | 1.13 | 1,906,278 |
Symbol ratio ("…" or "#") greater than 0.1 | 8,805 | 0.53 | 901,632 |
More than 90% of lines starting with a bullet sign ("*") | 16 | 0.00 | 1,638 |
More than 100,000 words | 0 | 0.00 | 0 |