[datatype=bibtex] [fieldsource=booktitle, match=Conference, replace=Conf.]
[fieldsource=booktitle, match=Symposium, replace=Symp.]
[fieldsource=booktitle, match=Proceedings of the, replace=Proc.]
[fieldsource=booktitle, match=Proceedings, replace=Proc.]
[fieldsource=booktitle, match=International, replace=Intl.]
[fieldsource=journal, match=International Journal, replace=Intl. J.]

# Two Elements of Pair Programming Skill

Franz Zieris  Institut für Informatik
Freie Universität Berlin
Berlin, Germany
<EMAIL_ADDRESS>Lutz Prechelt  Institut für Informatik
Freie Universität Berlin
Berlin, Germany
<EMAIL_ADDRESS>
###### Abstract

Background: Pair programming (PP) can have many benefits in industry.
Researchers and practitioners recognize that successful and productive PP
involves some _skill_ that might take time to learn and improve.

Question: What are the elements of pair programming skill?

Method: We perform qualitative analyses of industrial pair programming
sessions following the Grounded Theory Methodology. We look for patterns of
problematic behavior to conceptualize key elements of what ‘good’ and ‘bad’
pairs do differently.

Results: Here, we report two elements of pair programming skill: Good pairs
(1) manage to maintain their _Togetherness_ and (2) keep an eye on their
session’s _Expediency_. We identify three problematic behavioral patterns that
affect one or both of these elements: _Getting Lost in the Weeds_ , _Losing
the Partner_ , and _Drowning the Partner_.

Conclusion: Pair programming skill is separate from general software
development skill. Years of PP experience are neither a prerequisite nor
sufficient for successful pair programming.

##  I Introduction

_Pair programming_ (PP) is the practice of two software developers working
closely together on one machine. Kent Beck characterizes it as _“a dialog
between two people trying to […] program (and analyze and design and test)”_
which _“is a subtle skill, one that you can spend the rest of your life
getting good at”_ [2, p. 100]. Beck sees many benefits in this practice, such
as higher code quality in less time [2, pp. 66–67]. He does not, however,
elaborate on the aspects of the “skill” underlying these benefits; he merely
alludes to the importance of _communication_ and _coordination_ [2, p. 141].

Much of pair programming research appears to be built on the assumption that
PP does _not_ involve any particular skill beyond general software development
experience: A large experiment by Arisholm et al. [1], for instance, was set
to determine PP’s effect on code quality and effort for junior, intermediate,
and expert developers, but 93 of its 98 subject pairs had _no prior pairing
experience at all_. A meta-analysis of PP effectiveness [7] later found only
weak effects and high between-study variance, which indicates a number of not-
understood (and hence uncontrolled) moderating factors including the _“amount
of training in pair programming”_ [7, Sec. 4], which the researchers expect to
have a positive effect on pair performance.

To understand the differences between skillful and problematic pair
programming, we perform qualitative analyses of industrial PP sessions. Here,
we report two elements of _PP skill_ we identified. We discuss related work in
Section II, characterize our data and explain our research method in Section
III, sketch our findings in Section IV, and provide a discussion and an
outlook in Sections V and VI.

##  II Related Work

###  II-A On the Existence of Pair Programming Skill

In the context of her PP experiments with students, Williams [10] describes
the process of _pair jelling_ as the transition of two individuals from
_“considering themselves as a two-programmer team [to] considering themselves
as one coherent, intelligent organism working with one mind”_ [10, p. 53].
Such _jelling_ appears to have two aspects: Individuals get accustomed to not
working alone [11, p. 22], and a specific pair gets to know each other in
order for some _“bonding”_ to happen [10, p. 101]. Williams does not explicate
what this phase entails or how long it takes, but she claims the student pairs
were jelled after their first experiment assignment [10, pp. 63–64].

From an industry context, practitioner Belshee reports that _“[i]t often takes
days for a given pair to be comfortable with each other”_ , which he describes
as a precondition for a pair being able to reach a state of highly productive
_“Pair Flow”_ in which both members have a shared understanding of their task
[3, Sec. 1.2]. Belshee does not, however, mention _what_ or _how long_ it
takes to get good at pair programming.

Bryant et al. [4, 5] analyze the abstraction level of utterances in the dialog
of pair programmers in industry and note that there are differences between
the pair member sitting at the keyboard and her partner—but _only_ for pairs
with less than six months of PP experience: The frequencies and types of
utterances of _“expert pair programmers”_ do not depend on who controls the
keyboard [4, Sec. 5.2]. Bryant et al. conclude that in experienced pairs, both
partners maintain a _“clear mental model of [the pair’s] current state”_ [5,
Sec. 6.3].

###  II-B On the Elements of Pair Programming Skill

While there is some awareness in the literature that two developers are not
suddenly more productive just because they are placed next to each other,
there has been only little research into the elements that make pair
programming work. In their ethnographic study, Chong & Hurlbutt [6] report
that two partners with a similar level of task-relevant expertise can become
_“tightly coupled”_ such that they do not even need complete sentences for
effective communication. However, with a large-enough expertise difference
between them, the more knowledgable partner will be dominant.

For the aspect of transferring knowledge during pair programming, both Plonka
et al. [8] and Zieris & Prechelt [12, 13] summarize a number of ‘expertly’
pair programming behaviors, including: Making one’s thinking process visible
[8, 13], letting the learner work something out on her own [8], letting the
explainer finish her thoughts [12], and making sure to deal with complex
topics completely [12].

##  III Research Method

###  III-A Research Goal and Data Collection

The overall goal of our research is to understand how ‘good’ and ‘bad’ pair
programming sessions differ. Ultimately, we want to provide actionable advice
for practitioners. Here, we want to understand the _elements_ of the _skill_
which pair programmers exhibit in successful sessions and _how_ sessions
suffer from a lack thereof.

The industrial data used by Bryant et al. [4, 5] is limited to audio
recordings, which makes it difficult to understand what the developers are
referring to: For one out of every eight utterances, the researchers could not
reconstruct what the pairs referred to [5, Sec. 5.1]. We therefore analyze
industrial PP sessions comprising _audio_ , _webcam_ , and _screencast_ from
the _PP-ind_ repository [15, 16], which contains a variety of over 60 everyday
PP sessions from 13 companies along with pre- and post-session questionnaires
filled out by the developers. Sessions from the repository have IDs like ‘CA2’
(session 2, from the first team A, at the third company C); developers are
numbered similarly, e.g. ‘C2’.

###  III-B Qualitative Research Approach

We follow Strauss’ & Corbin’s Grounded Theory Methodology [9]. In particular,
we perform _theoretical sampling_ [9, Ch. 11] by choosing sessions from the
repository with pair members who have been pair-programing regularly for years
and those that are new to the practice, as well as involving experienced
software developers and novices (see Table I). Below, we report our findings
mostly from _open coding_ [9, Ch. 5], where relevant phenomena in the data are
identified, analyzed, and characterized through concepts, and some findings
from _axial coding_ [9, Ch. 7], which investigates when and how these
phenomena occur and how the pair deals with them. We did not yet perform
_selective coding_ [9, Ch. 8] to formulate a theory and also did not yet reach
_theoretical saturation_ [9, p. 188].

TABLE I: PP sessions analyzed from the _PP-ind_ repository [15] Session | Length | Member #1 | Member #2 | Session content
---|---|---|---|---
ID | h:mm | ID | Dev. | PP | ID | Dev. | PP |
CA1 | 1:18 | C1 | 4y | 2y | C2 | 9y | 6y |  Implementation of a new GUI form (C1 already started)
DA2 | 2:24 | D3 | 3m | 3m | D4 | 1y | 0 |  Feature implementation pivoted to refactoring (D4’s 1st PP ever)
JA1 | 1:07 | J1 | 6y |  $>$6m | J2 | 2.5y |  $>$6m |  Walkthrough of J2’s code, discussion of possible refactorings
PA3 | 1:31 | P1 | 5y | 2y | P3 | 5y | 2y |  Implement new API endpoint incl. testing (P3 already started)

###  III-C Our Notions of ‘Good’ and ‘Bad’

We only assess _exhibited_ PP skill, not the developers’ potential and not
behavior changes over a longer time. This is also a purely qualitative study.
We use a deficit-oriented perspective, i.e., we analyze episodes of pairs
running into ‘trouble’, e.g., (a) one or both pair members getting frustrated
because they do not understand what their partner says or does, or (b) the
pair doing things that help neither with their actual task nor with some
overarching goal such as getting familiar with the code base.

##  IV Results

We first describe two elements of pair programming skill (_Togetherness_ and
_Expediency_) and then characterize three patterns of problematic behavior in
pair programming sessions: (_Getting Lost in the Weeds_ , _Losing the Partner_
, and _Drowning the Partner_). We illustrate each pattern with excerpts from
actual sessions.

Section IV-E then gives examples of the alternate (high-skill) case, where
pairs manage to avert these problems.

###  IV-A Two Elements of Pair Programming Skill

Togetherness. Good pairs manage to stay together, that is, to establish and
maintain a shared mental model throughout their session. They detect and
address relevant discrepancies in each other’s understanding of their task,
work state, software system, and software development in general.

Expediency. Good pairs balance short-term goals, such as identifying a defect
or implementing a new feature, and long-term goals, e.g., addressing any
member’s knowledge gaps.

###  IV-B Anti-Pattern: Getting Lost in the Weeds

Two developers may come up with more ideas on what to look up and how to
proceed than a single developer. But pairs risk _Getting Lost in the Weeds_
when they jump on too many of them with too little consideration. Such pairs
may manage to _Stay Together_ in that they both think about all these new
ideas together, but they risk thinking too much about irrelevant details,
losing track of what is important, and thus reduce their _Expediency_.

Example 1: Session DA2 (09:00–19:00). It is developer D4’s first week at the
company, and he and D3 are tasked with implementing a new feature. D3 wants to
explain the target state by showing a similar, already existing function.
While D3 scrolls through the source code, D4 repeatedly interrupts him with
questions unrelated to their task, and D3 always tries his best to provide all
the information he can (highly compressed excerpt follows):

D3: “In principle, there should be a toolbar up here […] I’ll show you how it
looked in the old calendar. [starts navigating in the source code]”

D4: “[reading from screen] What are these navigation things for? Are these
_Actions_? Where are they displayed?”

D3: “[stops navigating] There is a—What’s it called again? [starts searching
through package tree …]”

D4: “[later: reading from screen, chuckling] LicenseKey?”

D3: “[stops searching] You’ll see that more often around here […] let’s see
where it’s used [starts fulltext search …]”

Since D4 is new at the company, providing him with information about the code
base _could_ be a good thing even if not pertinent to the current task.
However, _none_ of the side-topics actually led to D4 understanding something
he did not already know (not shown above), so we characterize this as a case
of a pair running into trouble (as defined in Section III-C). Instead, the
main topic (i.e., explaining the target state to D4) is interrupted by
twelve(!) abrupt topic changes (only two of which are shown above). Finishing
an exchange with a net time of 30 seconds takes the pair about ten minutes—and
it could have been even worse, as D3 was nearly lost after five minutes and
only found his way back because a stacktrace happened to be displayed on-
screen:

D3: “OK. Now, where were we? [looks around, sees stacktrace on lower display
corner] Ah, the exception, right.”

###  IV-C Anti-Pattern: Losing the Partner

Sometimes, one pair member is deeply engaged with the task at hand, trying to
understand the code or developing a design idea, but does not pay much
attention to her partner’s state of mind who then may or may not understand
what the colleague is doing. Such behavior may be _expedient_ in the short-
term if, say, a defect is found sooner than later, but it reduces the
_Togetherness_ of the pair which may result in (a) the partner being less
knowledgable later or (b) missing a learning opportunity (as discussed in [14,
Sec. 6.4.3]).

Example 2: Session CA1 (19:00–21:00). Developer C1 already started
implementing a new form when C2 joins. They want to make the form interactive
such that one checkbox deactivates multiple input fields (called “panels” and
“components” below). C2 appears to see problems with their approach but
neither explains them to C1 nor reacts to C1’s questions:

C2: “The problem is, it doesn’t fit with _getComponents_ [scrolls through
file]”

C1: “Why doesn’t it fit?”

C2: “I think so. I could be wrong. [continues scrolling]”

C1: “We only need to get the individual component from the panel, right? Is
that complicated?”

C2: “[ignoring C1] Ah, I just see it has a _getContent_.”

C1: “[reading from screen] A _PanelBuilder_. Can we possibly get the other
panels from there?”

C2: “[ignoring C1, continues scrolling] I’m not sure whether this all will
work.”

C1: “Can we deactivate a _JPanel_ on its own?”

C2: “[ignoring C1, continues scrolling] OK, I’d say—Shall we simply try to
implement the methods?”

C1: “Yeah, sure.”

Although the pair appears to have reached an agreement (“Shall we?”—“Yeah,
sure.”), C2 has been _Losing his Partner_ during the above two minutes; there
is no way in which C1 could have properly assessed the proposal he agreed to,
given that every question he asked was ignored. Similar behavior of C2 occurs
multiple times in session CA1.

###  IV-D Anti-Pattern: Drowning the Partner

The opposite behavior is also a problem. Just as one pair member may provide
too little explanation, she may also _Drown the Partner_ in too many
explanations that (a) go far beyond the task and are hence not _expedient_ and
(b) also threaten the pair’s _Togetherness_.

Example 3: Session PA3 (29:50–31:40). Developers P1 and P3 just extracted
multiple occurrences of the value 0.01 that is used in several percentage
calculations into a constant. P1 starts a long-winded explanation which his
partner does not understand because it is built on hypotheticals and does not
relate to their actual code changes. P3 gets increasingly frustrated over the
course of two minutes:

P1: “It’s important to make clear that the last two ‘ _0.01_ ’ have no
relationship.”

P3: “Which last two?”

P1: “The last two in lines 31 and 32, for example. Assuming the two numbers
would have no relation and someone who only sees the implementation with raw
numbers thinks ‘Oh, there is a relation, I’ll introduce a constant’ […].”

[… P1 goes on for 40 seconds …]

P3: “But applied to our case this has no relevance.”

P1: “Yes, it has. Because it is a Magic Number, and Magic Number means”—P3:
“But it is no longer ‘magic’. We just named it!”

P1: “I wanted to explain why we are doing this”—P3: “[annoyed] I got
that.”—P1: “I only want to clarify that it’s important to”—P3: “[annoyed,
staring at screen] Got it.”—P1: “make the relation with this renaming. […] Not
only to rename the variable.” P3: “[annoyed] It’s ok.” In general, these two
developers are getting along great, but here P1 was _Drowning his Partner_
with explanations the partner neither wanted nor needed. After the session,
the two developers talked about that incident. P3 criticized P1 for providing
such _“unwanted lectures”_ too often, continuing _“If I didn’t know you
better, I’d perceive this behavior as arrogant”_. P1 responded that some
issues just need pro-active explanations, because the partner would not even
know what to ask, or when. Both agreed on this and continued working
productively the next day.

###  IV-E Doing the Right Thing

Good pairs maintain _Togetherness_ and _Expediency_ ; they avoid the three
negative patterns described above.

For the case of _Getting Lost in the Weeds_ , both pair members are
responsible for restraining their impulses of following new ideas right away.
If good pairs get side-tracked, they notice it and work their way back
together, as in the next example.

Example 4: Session JA1 (04:00–06:40). Early in the session, a simple question
by J1 interrupts an explanation by J2, which leads to a misunderstanding that
takes almost two minutes to clear up. To avoid _Getting Lost in the Weeds_ ,
both partners explicitly switch back to the original topic (last two lines in
the excerpt):

J2: “There is the central [News] plugin and multiple processors which each
handle one wave. […] It checks how the file size is changing.”

J1: “In what time window are you looking?”

[… two minutes of misunderstanding …]

J1: “ _30 seconds_ , that’s what I wanted.”

J2: “That’s 30 seconds long, the time window. Now I got you. I can show it to
you [in the code] in a minute.”

J1: “Yes. And the NewsPlugin is doing what in all of this? Does it do exactly
this monitoring and the delegation to the individual wave plugins, or what?”

J2: “No. The NewsPlugin basically only does—it gets called periodically by the
cron server […]”

In contrast to _Getting Lost in the Weeds_ , which the partners do together,
_Losing_ or _Drowning the Partner_ is asymmetrical: One pair member is doing
something ‘wrong’ to her partner (i.e., explaining too little or too much) who
should then try to avert the problem. The next two examples show how good
pairs agree on which topics to address and how to limit the scope of an
explanation.

Example 5: Session DA2 (1:30:50–1:38:00). After their somewhat chaotic start
(see Example 1 in Section IV-B), the session of D3 and D4 proceeds more
orderly. Newly hired D4 explicitly asks his partner multiple times whether he
should provide an explanation on some matter, which D3 agrees to:

D4: “Do you know about OSGi class loading?”

D3: “Class-what? Not really, no.”

D4: “Should I tell you?”

D3: “Sure.”

Example 6: Session JA1 (13:15–13:45). J2 explains code he wrote earlier to J1,
who limits the explanation’s scope:

J2: “It comes from the function _‘getLastFile()’_. […] Shall we look into the
function or not?”

J1: “No, not now, please.”

J2: “Not now, ok.”

In both cases, the developers make sure to neither _Lose_ nor _Drown_ their
partner.

###  IV-F Further Elements of Pair Programming Skill

We found more elements of pair programming skill which we do not report here,
e.g., _Agreeing on a Common Plan_ , which includes picking up on cues that the
partner did not understand one’s ideas. Failure to do so threatens the pair’s
_Togetherness_ and their work’s _Expediency_.

##  V Discussion

Previous research and practitioner reports suggest that getting accustomed to
working in pairs and getting familiar with a particular partner takes time.
However, exhibited PP skill does not appear to directly depend on experience:
We saw developers with _no PP experience_ skillfully avoid _Losing_ or
_Drowning_ their partner, e.g., D4 probing his partner’s knowledge gaps in the
latter half of session DA2 (which was D4’s very first PP session ever), such
as in Example 5; and we saw problematic behavior in developers with long PP
histories, e.g., C2 (6 years of PP) _Losing his Partner_ in session CA1, or P1
(2 years of PP) _Drowning his Partner_ in session PA3, see Examples 2 and 3.

This leads to open questions: How do PP novices manage to have good PP
sessions? Which elements of PP skill can be acquired through what types of
experience? Which are specific to the context and the involved partners?

##  VI Summary and Further Work

Pair programming (PP) does not just ‘work’ because two software developers sit
next to each other. Rather, developers can be more or less _skilled_ at pair
programming. We characterize two elements of that skill that are independent
from software development skills: Maintaining _Togetherness_ and keeping an
eye on _Expediency_. There are possibly more elements to be found. So far, we
described three patterns of problematic behavior:

  * •

_Getting Lost in the Weeds_ , during which both partners stay together as a
pair, but lose sight of which topics are worth pursuing.

  * •

_Losing the Partner_ , in which one pair member focuses too much on the task
and explains too little.

  * •

_Drowning the Partner_ , in which one pair member explains too much, which may
harm the pair’s Togetherness _and_ Expediency.

Our current data is limited to snapshots of pair programmer behavior: Most
pairs in the _PP-ind_ repository [15] were recorded only once or twice during
a short stretch of their pair programmer career. Nevertheless, we have already
seen that prior PP experience alone does not explain beneficial and
problematic behavior. Longitudinal research with the same developers over
longer time frames will be needed to understand and disentangle the influence
of developers’ personal styles, their day-to-day form, and their experience
with pair programming in general or with a particular partner.

##  VII Data Availability

Unfortunately, we cannot share the full audio and video material due to non-
disclosure agreements with the involved companies and confidentiality
agreements with the recorded developers. We do, however, provide the full
original German transcripts as well as English translations for the referenced
session excerpts [16].

## References

  * [1] Erik Arisholm, Hans Gallis, Tore Dybå and Dag I.K. Sjøberg  “Evaluating Pair Programming with Respect to System Complexity and Programmer Expertise”  In _IEEE Transactions on Software Engineering_ 33.2, 2007, pp. 65–86  DOI: 10.1109/TSE.2007.17
  * [2] Kent Beck  “Extreme Programming Explained: Embrace Change”  Addison-Wesley Professional, 1999
  * [3] Arlo Belshee  “Promiscuous Pairing and Beginner’s Mind: Embrace Inexperience”  In _Agile Development Conf._ IEEE, 2005, pp. 125–131  DOI: 10.1109/ADC.2005.37
  * [4] Sallyann Bryant  “Double Trouble: Mixing Qualitative and Quantitative Methods in the Study of eXtreme Programmers”  In _IEEE Symp. on Visual Languages and Human Centric Computing_ , VL/HCC ’04  IEEE, 2004, pp. 55–61  DOI: 10.1109/VLHCC.2004.20
  * [5] Sallyann Bryant, Pablo Romero and Benedict Boulay  “Pair Programming and the Mysterious Role of the Navigator”  In _Intl. J. of Human-Computer Studies_ 66.7, 2008, pp. 519–529  DOI: 10.1016/j.ijhcs.2007.03.005
  * [6] Jan Chong and Tom Hurlbutt  “The Social Dynamics of Pair Programming”  In _Proc. 29th Intl. Conf. on Software Engineering_ , ICSE ’07  Washington, DC, USA: IEEE Computer Society, 2007, pp. 354–363  DOI: 10.1109/ICSE.2007.87
  * [7] Jo E. Hannay, Tore Dybå, Erik Arisholm and Dag I.K. Sjøberg  “The effectiveness of pair programming: A meta-analysis”  In _Information and Software Technology_ 51.7, 2009, pp. 1110–1122  DOI: 10.1016/j.infsof.2009.02.001
  * [8] Laura Plonka, Helen Sharp, Janet Linden and Yvonne Dittrich  “Knowledge transfer in pair programming: An in-depth analysis”  In _Intl. J. of Human-Computer Studies_ 73, 2015, pp. 66–78  DOI: 10.1016/j.ijhcs.2014.09.001
  * [9] Anselm L. Strauss and Juliet M. Corbin  “Basics of Qualitative Research: Grounded Theory Procedures and Techniques”  SAGE Publications, Inc, 1990
  * [10] Laurie Williams  “The Collaborative Software Process”, 2000
  * [11] Laurie Williams, Robert R. Kessler, Ward Cunningham and Ron Jeffries  “Strengthening the Case for Pair Programming”  In _IEEE Software_ 17.4 Los Alamitos, CA, USA: IEEE Computer Society Press, 2000, pp. 19–25  DOI: 10.1109/52.854064
  * [12] Franz Zieris and Lutz Prechelt  “On Knowledge Transfer Skill in Pair Programming”  In _Proc. 8th ACM/IEEE Intl. Symp. on Empirical Software Engineering and Measurement_ , ESEM ’14  ACM, 2014, pp. 11:1–11:10  DOI: 10.1145/2652524.2652529
  * [13] Franz Zieris and Lutz Prechelt  “Observations on Knowledge Transfer of Professional Software Developers During Pair Programming”  In _Proc. 38th Intl. Conf. on Software Engineering Companion_ , ICSE ’16 (SEIP)  ACM, 2016, pp. 242–250  DOI: 10.1145/2889160.2889249
  * [14] Franz Zieris and Lutz Prechelt  “Explaining Pair Programming Session Dynamics from Knowledge Gaps”  In _Proc. 42nd Intl. Conf. on Software Engineering_ , ICSE ’20  ACM, 2020, pp. 421–432  DOI: 10.1145/3377811.3380925
  * [15] Franz Zieris and Lutz Prechelt  “PP-ind: A Repository of Industrial Pair Programming Session Recordings” Tech report, 2020  arXiv:2002.03121
  * [16] Franz Zieris and Lutz Prechelt  “PP-ind: A Repository of Industrial Pair Programming Research Data”  Zenodo, 2021  DOI: 10.5281/zenodo.4529143


# ptt5-v2: A Closer Look at Continued Pretraining of T5 Models for the
Portuguese Language

Marcos Piau  School of Electrical and Computing Engineering,
State University of Campinas (UNICAMP)  Roberto Lotufo  School of Electrical
and Computing Engineering,
State University of Campinas (UNICAMP)  Rodrigo Nogueira  School of Electrical
and Computing Engineering,
State University of Campinas (UNICAMP)

###### Abstract

Despite advancements in Natural Language Processing (NLP) and the growing
availability of pretrained models, the English language remains the primary
focus of model development. Continued pretraining on language-specific corpora
provides a practical solution for adapting models to other languages. However,
the impact of different pretraining settings on downstream tasks remains
underexplored. This work introduces ptt5-v2, investigating the continued
pretraining of T5 models for Portuguese. We first develop a baseline set of
settings and pretrain models with sizes up to 3B parameters. Finetuning on
three Portuguese downstream tasks (assin2 STS, assin2 RTE, and TweetSentBR)
yields SOTA results on the latter two. We then explore the effects of
different pretraining configurations, including quality filters, optimization
strategies, and multi-epoch pretraining. Perhaps surprisingly, their impact
remains subtle compared to our baseline. We release the ptt5-v2 pretrained
checkpoints and and the finetuned MonoT5 rerankers on
HuggingFace.111https://huggingface.co/collections/unicamp-
dl/ptt5-v2-666538a650188ba00aa8d2d0
222https://huggingface.co/collections/unicamp-
dl/monoptt5-66653981877df3ea727f720d

##  1 Introduction

Transformer-based pretrained language models have established themselves as
the core paradigm in the field of Natural Language Processing (NLP). Starting
with the advent of BERT [11], which popularized the “pretrain, then fine-tune”
approach and the use of the transformer architecture ifself, these models
acquire a general-purpose language representation by unsupervised pretraining
on extensive corpora of unlabeled text. The dynamics of the pretraining
process had been studied in depth by many works, like Raffel et al. [32], that
introduced T5, and scaled up models to billions of parameters and set new
SOTAs across many tasks. The trend towards increasing model sizes and datasets
to improve performance motivated studies like Kaplan et al. [22] on scaling
laws and Hoffmann et al. [17], who demonstrated the importance of training
data size relative to model size for compute-optimal training regimes; more
recently work from Gadre et al. [13] specifically examined the influence of
extended pretraining on downstream task performance.

Despite the extensive study of pretraining dynamics, the focus has
predominantly been on English, leaving non-English languages less explored.
Continued pretraining presents a strategic approach to adapting these models
to additional languages and domains using significantly less data and
computational resources than training from scratch. This method involves
further pretraining on language-specific corpora, which has been shown to
substantially enhance model performance on downstream tasks in the target
language [30, 5, 24, 4, 40, 10]. However, there is a lack of detailed
investigations into how different settings during the continued pretraining
phase influence on downstream tasks performance, with most studies merely
aiming for benchmark-leading results without a thorough examination of the
underlying factors.

In this work, we study the continued pretraining of T5 models for the
Portuguese language, analyzing the impact of various settings on downstream
task performance. Rather than solely focusing on achieving state-of-the-art
results, our study also investigates how factors like model size, optimization
schedules, and the application of quality filters in the pretraining dataset
affect the performance. We continue the pretraning of Google’s T5 with up to 3
billion parameters on Portuguese texts. By experimenting with different
configurations in the pretraining stage, we observe nuanced effects on
downstream tasks, with some settings only marginally outperforming the
baselines. Our findings also suggest that while continued pretraining enhances
model capabilities, the increments in performance diminish as model size
increases.

T5 models [32] demonstrate adaptability across various natural language
processing (NLP) tasks due to their encoder-decoder architecture. This
structure enables them to process text for both understanding and generation,
providing an advantage over encoder-only models like BERT. While not the focus
of this work, T5’s adaptability to instruction-based fine-tuning, as seen in
FLAN-T5 [7], also enables effective zero-shot and few-shot applications. These
factors, combined with the scarcity of Portuguese pretrained encoder-decoder
models, motivate our choice to continue the investigation the T5 architecture
in this study.

##  2 Related Work

The T5 model [32] is an encoder-decoder transformer, and one of its main
innovations was to cast all tasks into a text-to-text format, allowing for a
unified approach; scaling up models to 11B parameters, they consolidated the
transfer learning approach, setting new SOTAs for GLUE [44], SuperGLUE [43],
CNN/Daily Mail [16] benchmarks. It was pretrained using the “span corruption”
objective over the C4 (“Colossal Clean Crawled Corpus”) dataset, where random
consecutive spans in the input are replaced by special mask tokens, and the
model is trained to predict these corrupted tokens (models with up to 11B
parameters). Building upon this foundation, mT5 [46] extended the T5 framework
to multilingual settings, having been pretrained on mC4, a multilingual
dataset covering 101 languages (models up to 13B parameters). PTT5 [5] further
adapted T5 for Portuguese by continuing the pretraining of T5 models on the
BrWac dataset [42]. This approach led to significant improvements in
downstream Portuguese language tasks, which were further enhanced by a
Portuguese tokenizer. For clarity, we’ll refer to the work of Carmo et al. as
ptt5-v1. Other notable international adaptations of T5/mT5 include it5 [38]
(Italian), AfriTeVa [21] (low-resource African languages), AraT5 [28]
(Arabic), and plT5 [6] (Polish).

Bertimbau [40], a popular adaptation of the BERT encoder model, remains
influential within Portuguese language modeling. Others exploring encoder
architectures include Albertina [35], DeBERTinha [4], the work of Gomes et al.
[15] (which pretrains a Roberta model), and de Morais et al. [27]. Reflecting
a broader trend, numerous recent Portuguese models prioritize decoder-only
architectures, such as Sabiá [30], Glória [26], Bode [14], Cabrita [24], and
Gervásio [37]. In the encoder-decoder space, the work of Carmo et al.
(ptt5-v1) explored adapting T5 models for Portuguese. Beyond generic
Portuguese models, several works specialize in custom domains: de Barros et
al. [2] and BERTabaporu [9] were designed for Portuguese social media data,
while Bertaú [12] focuses on financial language.

##  3 Methodology

This section describes the methodology for pretraining and evaluating our key
experiments, covering the pretraining dataset, language-specific vocabulary,
model architectures, optimization strategies, and finetuning and validation
processes for downstream tasks.

###  3.1 Unsupervised continued pretraining

As the pretraining data, we utilized the Portuguese segment of the mC4 dataset
(hereafter referred to as mC4-pt), comprising approximately 524 GB of
uncompressed text across 169 million documents. This dataset is significantly
larger than the one used for the pretraining of ptt5-v1 models, which
originated from the BrWac dataset [42] and consisted of around 15 GB of text
from 7.4 million documents after preprocessing.

We adopted the Portuguese language vocabulary from ptt5-v1. This SentencePiece
Unigram tokenizer [23], comprising 32,000 tokens, was trained over a corpus of
2 million documents from the Portuguese Wikipedia. This vocabulary shares the
same number of tokens and control tokens as T5, facilitating the direct use of
Google’s model checkpoints.

As the pretraining objective, the span corruption task was employed, utilizing
batches of 128 sequences of 512 tokens (65,536 tokens) - a methodology
consistent with the baseline experiment by Raffel et al. [32]. Adafactor
optimizer [39] with a constant learning rate of 0.001 and cross-entropy as
loss was used during the entire pretraining process. Using these experimental
settings, we started from Google’s original checkpoints with sizes from
t5-small (60M parameters) up to t5-3B (3B parameters), and performed a
complete epoch of continued pretraining over the mC4-pt dataset. Considering
these settings, a single epoch over the mC4-pt dataset comprises approximately
1,764,515 iterations. Additional pretraining experiments are detailed in
Section 5.1.

Both pretraining and finetuning experiments utilized TPUv2-8 and TPUv3-8
devices, leveraging t5 [32] and seqio [34] frameworks.

###  3.2 Supervised finetuning on downstream tasks

To assess the impact of our pretraining on the ASSIN2 RTE, ASSIN2 STS and
TweetSentBR tasks, we finetuned the pretrained models over 100 epochs with
batches of 128 sequences and a maximum length of 512 tokens, using Adafactor
with a constant learning rate of 0.001 as the optimizer. The model checkpoint
yielding the best performance on the validation set was selected for testing,
and greedy decoding was utilized as the decoding method. Because the
TweetSentBr lacks a validation set, we reserved 10% of the training data for
validation and used the rest for training.

All tasks were approached using a text-to-text format. Specifically for the
ASSIN2 STS task, which involves the prediction of continuous values in the
range between 1 to 5, we adopted the strategy from Raffel et al. [32], by
rounding the target scores to the nearest 0.2 increment and converting these
to strings, thus framing it as a multiclass classification problem compatible
with the text-to-text format. Further details about each dataset, along with
examples of input and target formats, are provided in Appendix A.

To compare the quality of the new checkpoints against existing alternatives,
we also use the same finetuning procedure on Google T5 and mT5 models.

###  3.3 MonoPTT5 Rerankers

To evaluate the adaptability of the ptt5-v2 models for information retrieval
tasks, we trained MonoT5 rerankers [29] using checkpoints generated as
described in Section 3.1. We named these models MonoPTT5. The training data
originated from the mMARCO dataset [3], a translated version of the MS MARCO
passage retrieval dataset [1], originally in English, to 13 languages,
including Portuguese. We created a bilingual Portuguese-English training
dataset by randomly assigning one of the two languages to each training
triplet (query, positive passage, negative passage), which were then split
into (query, passage) pairs. This approach, known as “translate-train” [8, 46,
19], leverages synthetic data augmentation by integrating machine translations
with original text data to substantially expand the available training
material in the target language. Prior research [3, 36, 8] has shown the
effectiveness of this bilingual training strategy, motivating our adoption of
this method.

We adapted the input and target format for the Portuguese language to the
structure "Pergunta: {query} Documento: {document} Relevante:", assigning the
tokens “Sim” for relevant passages and “Não” for non-relevant ones. The models
were trained for 100k steps with batch sizes of 128 sequences and a maximum
length of 512 tokens, utilizing Adafactor with a constant learning rate of
0.001 as the optimizer. Instances exceeding the maximum token length were
excluded from the training dataset; these constituted approximately 0.01% of
the training samples and were predominantly attributed to noisy translation
data.

The effectiveness of our rerankers was assessed in Portuguese datasets under
two conditions: in-domain and zero-shot. The in-domain condition utilized the
“small dev” set of queries from the mMARCO-pt dataset, comprising 6,980
queries. For the zero-shot condition, we employed the Portuguese subset of the
mRobust dataset, consisting of 249 annotated queries. Initially, we used BM25
to generate an initial set of relevant documents 333All applications of BM25
in this work use Pyserini’s implementation [25] with default parameters
$k_{1}=0.9$ and $b=0.4$. These documents were subsequently reranked using our
finetuned MonoPTT5 models. Due to the longer length of documents in the
mRobust dataset, direct reranking could result in truncation. To mitigate
this, documents retrieved by BM25 were segmented into sliding sentence windows
using a Spacy [18] sentencizer pipeline, with a maximum length of 8 and a
stride of 4 sentences. Given the significant computational resources required
for training these rerankers, we focused exclusively on models based on the
main ptt5-v2 checkpoints.

##  4 Main Results

Table 1 shows the results on the downstream tasks considered. In the ASSIN2
RTE task, our 3B sets a new SOTA, surpassing the current one by 0.61 F1-macro
points. For the TweetSentBR dataset, we achieved better performance than
current finetuned SOTAs with ptt5-v2-large and ptt5-v2-3B, by 0.52 and 1.54
F1-macro points, respectively, but our results are worse when comparing to
GPT-4. We highlight that our ptt5-v2 were trained exclusively on each task
training data using the text-to-text framework without any data augmentation
or adaptation to the model’s architecture, unlike the works of [2] and [36],
which held the SOTA for TweetSentBR and ASSIN2 RTE. In the ASSIN2 STS task,
our models did not surpass the current SOTA; regardless, ptt5-v2 still shows
better performance than mT5 and T5 models with approximate sizes, and this is
also the only task where a smaller ptt5-v2 model (ptt5-v2-large) has better
performance of a large one ptt5-v2-3B.

|  | Binary classification | Regression | Multiclass classification | $\leftarrow$ | Retrieval
---|---|---|---|---|---|---
|  | ASSIN2 RTE | ASSIN2 STS | TweetSentBR | $\leftarrow$ | mMARCO-pt | mRobust-pt
Model | Parameters | F1-macro | Pearson | F1-macro | NPM | RR@10 | nDCG@20
T5
t5-small | 60M | 83.66 | 0.738 | 62.14 | 61.71 | - | -
t5-base | 220M | 85.80 | 0.764 | 65.43 | 65.63 | - | -
t5-large | 770M | 88.91 | 0.790 | 68.22 | 69.95 | - | -
t5-3B | 3B | 90.78 | 0.827 | 72.58 | 74.56 | - | -
mT5
mt5-small | 300M | 75.36 | 0.688 | 61.81 | 54.35 | - | -
mt5-base | 580M | 79.39 | 0.749 | 70.76 | 63.46 | - | -
mt5-large | 1.2B | 88.25 | 0.753 | 61.11 | 64.76 | - | -
mt5-xl | 3.7B | 91.81 | 0.827 | 77.05 | 77.45 | - | -
ptt5-v2
ptt5-v2-small | 60M | 87.14 | 0.782 | 70.99 | 69.86 | 0.273 | 0.344
ptt5-v2-base | 220M | 88.36 | 0.814 | 73.20 | 72.82 | 0.311 | 0.384
ptt5-v2-large | 770M | 91.73 | 0.839 | 76.78 | 77.68 | 0.315 | 0.462
ptt5-v2-3B | 3B | 92.68 | 0.829 | 77.80 | 78.48 | 0.332 | 0.512
Supervised SOTA | - | 92.07 [36] |  0.868 [35] | 76.26 [2] | - | 0.306 [36] | 0.391 [20]
Few shot baselines from [30]
GPT-4 | - | 90.96 | 0.776 | 82.40 | - | - | -
GPT-3.5-turbo | - | 88.28 | 0.664 | 74.39 | - | - | -

Table 1: Main results after finetuning. For supervised SOTAs, model sizes are
as follows: [36] (335M), [35] (900M), [2] (110M), [36], and [20] (580M). NPM
results exclude retrieval tasks.

In addition to using individual metrics for each task, we also incorporate the
Normalized Preferred Metric (NPM) [41] to facilitate the evaluation of the
overall performance of a pretrained model across multiple tasks. The NPM
normalizes a task’s preferred metric (e.g., F1-macro for ASSIN2 RTE),
assigning a value of 0 to represent random performance and 100 to denote
maximum performance. Below is the equation used to calculating the NPM for a
given model and set $N$ of tasks:

$\textup{NPM}=\frac{100}{N}\sum_{i=1}^{N}\frac{\textup{[raw preferred
metric]}_{i}-\textup{[random score]}_{i}}{\textup{[maximum
score]}_{i}-\textup{[random score]}_{i}}$ (1)

Given that MonoPTT5 rerankers were exclusively trained starting from ptt5-v2
pretrained checkpoints, retrieval tasks were excluded from this evaluation.
Therefore, we only considered ASSIN2 RTE, ASSIN2 STS, and TweetSentBR tasks.
For each model, we calculate its aggregate performance by first determining
the NPM for each task and then computing the average of these values.

Our ptt5-v2 models have higher NPM values than T5 and mT5 models with
considerably more parameters: for example, ptt5-v2-base is only surpassed by
t5-3B ($\sim$13.6x larger) and t5-xl ($\sim$16.81x larger); this performance
gap, however, is more pronounced is most pronounced in smaller models,
narrowing as model size increases. A similar result was also observed by Xue
et al. [46], which analyzed the performance of T5 and mT5 models peformance on
the SQuAD benchmark [33], observing a peformance gap between t5-small and
t5-base vs mT5 models of equivalent sizes, which is diminished starting from
t5-large. This performance gap observed on smaller model sizes is advantageous
when we consider environments constrained by computational resources,
increasing the maximum attainable level of performance; additionally, a
language-specific tokenizer reduces text splitting into fewer tokens, leading
to lower latency and the potential to accommodate more text within the same
maximum token context window. Interestingly, mT5 models tend to show lower NPM
values, except in the 3 billion parameter range, where they slightly
outperform T5 models.

For the retrieval tasks, our MonoPTT5 rerankers were able to set new SOTAs for
both mMARCO-pt and mRobust-pt. For the mMARCO-pt dataset, models starting from
t5-base size were able to surpass the current SOTA; the 3 billion parameters
reranker obtained a gain of +0.02 points of MRR@10. In the mRobust-pt, our
large and 3B rerankers surpassed the current SOTA by +0.071 and +0.12 in terms
of nDGC@20, respectively. A more in-depth analysis of the retrieval tasks is
conducted on section 5.2.

##  5 Ablations

###  5.1 Additional pretraining experiments

This sections includes pretraining experiments additional to those described
in 3.1.

Comparison with ptt5-v1: Given the title of our work, a pertinent question
arises: How do ptt5-v2 models compare with our initial work in ptt5-v1? A few
key differences exist in the pretraining of ptt5-v1. Notably, it utilized
BrWac, a considerably smaller dataset, and a slightly different pretraining
objective (denoising, where some input tokens are masked and the model is
trained to predict the original text, rather than span corruption).
Additionally, ptt5-v1 employed models ranging from t5-small to t5-large with
an Adafactor optimizer using a learning rate of 0.003 (threefold larger than
our setting). ptt5-v1 also explored both T5’s original English vocabulary and
a Portuguese language-specific tokenizer. In contrast, ptt5-v2 exclusively
uses the latter.

To ensure a fair comparison, we finetuned the ptt5-v1 checkpoints following
the methodology outlined in section 3.2. Figure 1 presents the NPM values for
both ptt5-v1 and ptt5-v2, alongside comparisons to mT5 and T5 models. The data
corroborates the enhancements achievable through monolingual pretraining on
the target language, which is further augmented by employing a dedicated
tokenizer. In comparison between the two ptt5 iterations, a performance
disparity favoring ptt5-v2 is apparent for the small and large sizes, with the
largest gap observed for large; however, models of the base size exhibit
marginally superior performance in the ptt5-v1 variant. Surprisingly, mT5’s
performance lags behind all other models, including the monolingual English
T5, except in the 3B parameter range, where it closely matches ptt5-v2 and
surpasses T5.

Figure 1: NPM vs parameters for varying T5 configurations.

Quality filters: In our primary experiments, we utilized the entire mC4-pt
dataset in the pretraining phase. In this additional experiment, we consider
the MassiveText quality filters [31], to investigate the impact of this
filtering process on downstream tasks; applying these filters to mC4-pt
decreases the number of training tokens by approximately 30%. This experiment
was restricted to the models of t5-base size, keeping the same batch size and
optimization strategy used in the main experiments. Figure 2 shows the effect
of using MassiveText’s quality filters on the downstream tasks performance,
measuread in terms of NPM. The pretraining using the filtered shows a updward
that doesn’t saturate up the considered duration of 1 mC4-pt epoch. It’s
important to highlight that 1 epoch in the filtered dataset has less steps
than the full dataset, so at 1 full dataset epoch, data is repeated for the
filtered dataset. Despite the upward trend favoring the filtered dataset
pretraining, the difference in terms of NPM is small at the 1 epoch mark.

Figure 2: Effect of using quality filters on mC4-pt. Epochs are relative to
mC4-pt.

Pretraining optimization strategy: In our exploration of optimization
strategies, we initially employed Adafactor with a constant learning rate.
This ablation extends our investigation to the “inverse square root” learning
rate schedule, as utilized by [32] in their final pretraining experiments.
This learning rate schedule computes the rate as $\frac{1}{\sqrt{\max(n,k)}}$,
where $n$ represents the current step, and $k$ is the number of warm-up steps.
Raffel et al. [32] used $k=10,000$, which sets the learning rate of 0.01 for
the initial 10k steps, subsequently decreasing exponentially. The learning
rate at the end of the pretraining, which consisted of around 1 million steps,
was close to 0.001, the same one used during the finetuning. Figure 3
illustrates the difference between these optimization strategies.

Figure 3: Learning rate schedules: Constant vs. inverse square root scheduler
as a function of training steps. Figure 4: Effect of scheduler used during
pretraining. Epochs are relative to mC4-pt.

Attempting to closely mirror the T5 pretraining recipe, we applied this
identical schedule in our preliminary experiments. However, we observed a
rapid overshoot in the training losses for t5-large t5-3B within hours;
adjusting $n$ only delayed the overshoot without averting it. Transitioning to
a constant learning rate of 0.001 solved the overshoot issue, leading to
stable pretraining loss across all model sizes and simplifying our
experimental setup. Because overshoot was observed only in the larger models,
and also given the computational costs associated with pretraining the larger
models, we performed additional pretraining experiments with “inverse square
root" scheduler for t5-small and t5-base models only.

Number of pretraining epochs: In the primary set of experiments, the mC4-pt
dataset was fully utilized for pretraining over one epoch. To explore the
influence of the number of pretraining epochs on downstream task performance,
we conducted experiments over various epochs, including partial epochs (0.25,
0.5, and 0.75 of an epoch). Considering the significant time and computational
resources required for extended pretraining, especially with larger models, we
limited the pretraining of the t5-large model to 2 epochs and the t5-3B model
to a single epoch. The reduced epoch duration for the t5-small and t5-base
models enabled more extensive pretraining periods for these configurations.

In Figure 4, the NPM values for t5-base are shown with the constant and the
inverse square root scheduler, across a varying number of pretraining epochs.
It is observed that there is a difference between these two optimization
strategies: the inverse square root scheduler has the advantage for up to 2
epochs; afterwards, the constant learning rate takes the upper hand, and by
the last epoch considered, they reach the same value. An increasing trend in
the values of NPM is also noted for more epochs.

###  5.2 MonoPTT5 Rerankers

The information retrieval tasks reported in 1 represents the performance of
our MonoPTT5 experiments, developed with the methodology describe in 3.3; in
this section we also report the results for other approaches, including BM25,
and dense retrieval using multilingual-e5 [45] models. The dense models are
used as a single stage retrieval system without reranking; dense indexing and
retrieval is performed with A100 and V100 GPUs on Google Colab, leveraging the
Pyserini framework. For mRobust-pt, which contain longer documents, we
mitigate document truncation by using the same splitting strategy described in
the section 3.3, using the maximum score among the document segments as the
document score.

Figures 5 and 6 are used to illustrate the discussion presented in this
section. Results in Table 1 only shows the effectiveness in each retrieval
task of our MonoPTT5 models and the SOTA competitors. For the in-domain
retrieval task, the mMarco-pt dataset, we note that BM25 is easily surpassed
by all alternatives considered, and the effectiveness figures for MonoPTT5
rerankers and multilingual-e5 models are similar when consider the size range
in common, and MonoPTT5 rerankers effectiveness is above SOTA starting from
models of size t5-base. For mRobust-pt, representing a zero-shot setting, BM25
is only surpassed by the mT5 reranker from Jeronymo et al. [20], and MonoPTT5
models starting from t5-base size.

Figure 5: Retrieval results on mMarco-pt. mColbert and mT5 values are from
Bonifacio et al. [3]. Total size excludes embedding parameters. Figure 6:
Retrieval results on mRobust-pt. mColbert and mT5 values are from Jeronymo et
al. [20]. Total size excludes embedding parameters.

##  6 Conclusion

In this study, we introduced ptt5-v2, exploring the continued pretraining of
T5 models for the Portuguese language. We pretrained T5 models using a
Portuguese language tokenizer, over a Portuguese language corpus. The
finetuned models achieved SOTA on ASSIN2 RTE and TweetSentBr datasets, two of
the three downstream tasks considered. Additionally, we applied these
pretrained checkpoints to develop MonoT5 rerankers customized for the
Portuguese language, achieving top performance on the mMARCO-pt and mRobust-pt
datasets. Our main results supports the evidence of a performance gap favoring
monolingual models over English-focused and multilingual models, a gap that
narrows as model capacity increases. This underscores the importance of
language-specific pretraining, and our analysis of pretraining settings
suggests that while data filtering, optimization strategies, and pretraining
duration can offer incremental improvements, the overall effects were limited
in comparison to our baseline settings and the core pretraining recipe
remained robust.

## Acknowledgements

We thank Google for the TPU grant through the TRC program.

## References

  * [1] Payal Bajaj et al.  “MS MARCO: A Human Generated MAchine Reading COmprehension Dataset”, 2018  arXiv:1611.09268 [cs.CL]
  * [2] Tiago Martinho Barros, Helio Pedrini and Zanoni Dias  “Leveraging emoji to improve sentiment classification of tweets”  In _Proceedings of the 36th Annual ACM Symposium on Applied Computing_ , SAC ’21  Virtual Event, Republic of Korea: Association for Computing Machinery, 2021, pp. 845–852  DOI: 10.1145/3412841.3441960
  * [3] Luiz Bonifacio et al.  “mMARCO: A Multilingual Version of the MS MARCO Passage Ranking Dataset”, 2022  arXiv:2108.13897 [cs.CL]
  * [4] Israel Campiotti et al.  “DeBERTinha: A Multistep Approach to Adapt DebertaV3 XSmall for Brazilian Portuguese Natural Language Processing Task”, 2023  arXiv:2309.16844 [cs.CL]
  * [5] Diedre Carmo et al.  “PTT5: Pretraining and validating the T5 model on Brazilian Portuguese data”, 2020  arXiv:2008.09144 [cs.CL]
  * [6] Aleksandra Chrabrowa et al.  “Evaluation of Transfer Learning for Polish with a Text-to-Text Model”, 2022  arXiv:2205.08808 [cs.CL]
  * [7] Hyung Won Chung et al.  “Scaling Instruction-Finetuned Language Models”, 2022  arXiv:2210.11416 [cs.LG]
  * [8] Alexis Conneau et al.  “Unsupervised Cross-lingual Representation Learning at Scale”, 2020  arXiv:1911.02116 [cs.CL]
  * [9] Pablo Botton Costa et al.  “BERTabaporu: Assessing a Genre-Specific Language Model for Portuguese NLP”  In _Proceedings of the 14th International Conference on Recent Advances in Natural Language Processing_ Varna, Bulgaria: INCOMA Ltd., Shoumen, Bulgaria, 2023, pp. 217–223  URL: https://aclanthology.org/2023.ranlp-1.24
  * [10] Yiming Cui, Ziqing Yang and Xin Yao  “Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca”, 2024  arXiv:2304.08177 [cs.CL]
  * [11] Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova  “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”, 2019  arXiv:1810.04805 [cs.CL]
  * [12] Paulo Finardi et al.  “BERTaú: Itaú BERT for digital customer service”, 2021  arXiv:2101.12015 [cs.CL]
  * [13] Samir Yitzhak Gadre et al.  “Language models scale reliably with over-training and on downstream tasks”, 2024  arXiv:2403.08540 [cs.CL]
  * [14] Gabriel Lino Garcia et al.  “Introducing Bode: A Fine-Tuned Large Language Model for Portuguese Prompt-Based Task”, 2024  arXiv:2401.02909 [cs.CL]
  * [15] Juliana Resplande Santanna Gomes et al.  “Deep Learning Brasil at ABSAPT 2022: Portuguese Transformer Ensemble Approaches”, 2023  arXiv:2311.05051 [cs.CL]
  * [16] Karl Moritz Hermann et al.  “Teaching Machines to Read and Comprehend”, 2015  arXiv:1506.03340 [cs.CL]
  * [17] Jordan Hoffmann et al.  “Training Compute-Optimal Large Language Models”, 2022  arXiv:2203.15556 [cs.CL]
  * [18] Matthew Honnibal, Ines Montani, Sofie Van Landeghem and Adriane Boyd  “spaCy: Industrial-strength Natural Language Processing in Python”, 2020  DOI: 10.5281/zenodo.1212303
  * [19] Junjie Hu et al.  “XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization”, 2020  arXiv:2003.11080 [cs.CL]
  * [20] Vitor Jeronymo, Mauricio Nascimento, Roberto Lotufo and Rodrigo Nogueira  “mRobust04: A Multilingual Version of the TREC Robust 2004 Benchmark”, 2022  arXiv:2209.13738 [cs.CL]
  * [21] Odunayo Jude Ogundepo et al.  “AfriTeVA: Extending ?Small Data? Pretraining Approaches to Sequence-to-Sequence Models”  In _Proceedings of the Third Workshop on Deep Learning for Low-Resource Natural Language Processing_ Hybrid: Association for Computational Linguistics, 2022, pp. 126–135  DOI: 10.18653/v1/2022.deeplo-1.14
  * [22] Jared Kaplan et al.  “Scaling Laws for Neural Language Models”, 2020  arXiv:2001.08361 [cs.LG]
  * [23] Taku Kudo and John Richardson  “Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing”  In _arXiv preprint arXiv:1808.06226_ , 2018
  * [24] Celio Larcher et al.  “Cabrita: closing the gap for foreign languages”, 2023  arXiv:2308.11878 [cs.CL]
  * [25] Jimmy Lin et al.  “Pyserini: A Python Toolkit for Reproducible Information Retrieval Research with Sparse and Dense Representations”  In _Proceedings of the 44th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2021)_ , 2021, pp. 2356–2362
  * [26] Ricardo Lopes, João Magalhães and David Semedo  “GlórIA – A Generative and Open Large Language Model for Portuguese”, 2024  arXiv:2402.12969 [cs.CL]
  * [27] Leonardo P Morais et al.  “Sub-language Sentiment Analysis in WhatsApp Domain with Deep Learning Approaches”  In _Revista de Sistemas de Informaçao da FSMA_ , 2023, pp. 32–47
  * [28] El Moatez Billah Nagoudi, AbdelRahim Elmadany and Muhammad Abdul-Mageed  “AraT5: Text-to-Text Transformers for Arabic Language Generation”, 2022  arXiv:2109.12068 [cs.CL]
  * [29] Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep and Jimmy Lin  “Document Ranking with a Pretrained Sequence-to-Sequence Model”  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ Online: Association for Computational Linguistics, 2020, pp. 708–718  DOI: 10.18653/v1/2020.findings-emnlp.63
  * [30] Ramon Pires, Hugo Abonizio, Thales Sales Almeida and Rodrigo Nogueira  “Sabiá: Portuguese Large Language Models”  In _Lecture Notes in Computer Science_ Springer Nature Switzerland, 2023, pp. 226–240  DOI: 10.1007/978-3-031-45392-2_15
  * [31] Jack W. Rae et al.  “Scaling Language Models: Methods, Analysis & Insights from Training Gopher”, 2022  arXiv:2112.11446 [cs.CL]
  * [32] Colin Raffel et al.  “Exploring the limits of transfer learning with a unified text-to-text transformer”  In _arXiv preprint arXiv:1910.10683_ , 2019
  * [33] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev and Percy Liang  “SQuAD: 100,000+ Questions for Machine Comprehension of Text”, 2016  arXiv:1606.05250 [cs.CL]
  * [34] Adam Roberts et al.  “Scaling Up Models and Data with t5x and seqio”, 2022  arXiv:2203.17189 [cs.LG]
  * [35] João Rodrigues et al.  “Advancing Neural Encoding of Portuguese with Transformer Albertina PT-*”, 2023  arXiv:2305.06721 [cs.CL]
  * [36] Guilherme Moraes Rosa et al.  “A cost-benefit analysis of cross-lingual transfer methods”, 2021  arXiv:2105.06813 [cs.CL]
  * [37] Rodrigo Santos et al.  “Advancing Generative AI for Portuguese with Open Decoder Gervásio PT*”, 2024  arXiv:2402.18766 [cs.CL]
  * [38] Gabriele Sarti and Malvina Nissim  “IT5: Large-scale Text-to-text Pretraining for Italian Language Understanding and Generation”, 2022  arXiv:2203.03759 [cs.CL]
  * [39] Noam Shazeer and Mitchell Stern  “Adafactor: Adaptive Learning Rates with Sublinear Memory Cost”, 2018  arXiv:1804.04235 [cs.LG]
  * [40] Fábio Souza, Rodrigo Nogueira and Roberto Lotufo  “BERTimbau: Pretrained BERT Models for Brazilian Portuguese”  In _Intelligent Systems_ Cham: Springer International Publishing, 2020, pp. 403–417
  * [41] Aarohi Srivastava et al.  “Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models”, 2023  arXiv:2206.04615 [cs.CL]
  * [42] Jorge A Wagner Filho, Rodrigo Wilkens, Marco Idiart and Aline Villavicencio  “The brwac corpus: A new open resource for brazilian portuguese”  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , 2018
  * [43] Alex Wang et al.  “SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems”, 2020  arXiv:1905.00537 [cs.CL]
  * [44] Alex Wang et al.  “GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding”, 2019  arXiv:1804.07461 [cs.CL]
  * [45] Liang Wang et al.  “Multilingual E5 Text Embeddings: A Technical Report”, 2024  arXiv:2402.05672 [cs.CL]
  * [46] Linting Xue et al.  “mT5: A massively multilingual pre-trained text-to-text transformer”, 2021  arXiv:2010.11934 [cs.CL]

##  Appendix A Downstream tasks datasets

Dataset | Task | Preferred metric | Train/validation/test examples | Random score | Possible outputs
---|---|---|---|---|---
ASSIN2 RTE | Binary classification | F1-macro | 6,500/500/2,448 | 50 | {Entailment, None}
ASSIN2 STS | Regression | Pearson | 6,500/500/2,448 | 0 | $[1,5]$
TweetSentBR | Multiclass Classification (3) | F1-macro | 11,525/1,281/1,982 | 32.4 | {negativo, positivo, neutro}

Table 2: Downstream tasks datasets.

Dataset | Example inputs | Example targets
---|---|---
ASSIN2 RTE | assin2_rte sentence1: Uma pessoa está escovando um gato sentence2: O pelo de um gato está sendo penteado por uma pessoa | Entailment
ASSIN2 STS | assin2_stsb sentence1: Uma mulher está cortando vegetais sentence2: Uma mulher está cortando brócolis | 4.2
TweetSentBR | ttsbr_neg_pos_neu_sentiment_pt: adorando esse com dr dráuzio varela | positivo

Table 3: Downstream tasks inputs and targets


∎

11institutetext: Diego H. Useche 22institutetext: MindLab Research Group,
Universidad Nacional de Colombia, Bogotá, Colombia
22email<EMAIL_ADDRESS>33institutetext: Andres Giraldo-Carvajal
44institutetext: Facultad de Ingenierías, Universidad Tecnológica de Pereira,
Pereira, Colombia
44email<EMAIL_ADDRESS>55institutetext: Hernan M. Zuluaga-Bucheli
66institutetext: Facultad de Ingenierías, Universidad Tecnológica de Pereira,
Pereira, Colombia
66email<EMAIL_ADDRESS>77institutetext: Jose A. Jaramillo-Villegas
88institutetext: Facultad de Ingenierías, Universidad Tecnológica de Pereira,
Pereira, Colombia
Laboratory for Research in Complex Systems, Menlo Park, California, USA.
88email<EMAIL_ADDRESS>99institutetext: Fabio A. González 1010institutetext:
MindLab Research Group, Universidad Nacional de Colombia, Bogotá, Colombia
1010email<EMAIL_ADDRESS>
# Quantum Measurement Classification with Qudits

Diego H. Useche  Andres Giraldo-Carvajal  Hernan M. Zuluaga-Bucheli  Jose A.
Jaramillo-Villegas  Fabio A. González

###### Abstract

This paper presents a hybrid classical-quantum program for density estimation
and supervised classification. The program is implemented as a quantum circuit
in a high-dimensional quantum computer simulator. We show that the proposed
quantum protocols allow to estimate probability density functions and to make
predictions in a supervised learning manner. This model can be generalized to
find expected values of density matrices in high-dimensional quantum
computers. Experiments on various data sets are presented. Results show that
the proposed method is a viable strategy to implement supervised
classification and density estimation in a high-dimensional quantum computer.

###### Keywords:

Quantum Computing Qudit Quantum Machine Learning Quantum Measurement
Classification High-dimensional quantum computing

##  1 Introduction

Quantum computing has gained a lot of attention in recent years due to its
potential to solve complex problems which would take exponential time in
classical computers. Most of the research efforts have been focused on
constructing quantum computers based on qubits Arute et al. (2019). However,
there has been a growing interest in building quantum computers based on
qudits, i.e. machines that simulate and operate d-dimensional quantum states,
with $d>2$. Various physical implementations of high-dimensional quantum
states have been proposed, such as photonic states integrated in chips Schaeff
et al. (2015); Carolan et al. (2015), photonic modes encoded in the orbital
angular momentum (OAM) Sit et al. (2017), ion traps Klimov et al. (2003),
ququarts implemented on a quadrupolar nuclear magnetic resonance (NMR) Gedik
et al. (2015), and molecular quantum magnets Moreno-Pineda et al. (2018). Two
of the main advantages of high-dimensional quantum computers compared to their
qubit-based counterparts are their larger information storage Cozzolino et al.
(2019), and their higher resilience to noise Sheridan and Scarani (2010).

One closely related field of quantum computing is quantum machine learning
(QML). This field aims to develop novel quantum-inspired machine learning (ML)
methods that may run on classical or quantum computers and to implement the
existing ML algorithms on quantum computers. For instance, some classical
machine learning algorithms like support vector machines and restricted
Boltzmann machines can be implemented on qubit-based quantum computers
Rebentrost et al. (2014); Wiebe et al. (2014), and many of the ML methods have
been reformulated in the language of quantum physics like quantum decision
trees Lu and Braunstein (2014), quantum neural networks Ezhov and Ventura
(2000); Cong et al. (2019), and quantum generative adversarial networks
Dallaire-Demers and Killoran (2018). In contrast with QML methods built on
qubits, less research has been done on QML based on qudits, i.e. algorithms
that run in high-dimensional quantum computers. Some of these methods include
protocols with qudits for reinforcement learning Cárdenas-López et al. (2018),
and for training quantum neural networks Diep (2020); Ricks and Ventura
(2004); Beer et al. (2019).

In addition to the aforementioned methods, Gonzalez et. al. González et al.
(2021), proposed two quantum-inspired machine learning methods, the Density
Matrix Kernel Density Estimation (DMKDE), which is a non-parametric density
estimation method, and the Density Matrix Kernel Density Classification
(DMKDC), a supervised machine learning algorithm based on density matrices and
kernel density estimation. In this article, we propose two quantum protocols
to implement the prediction phase of the DMKDE and the DMKDC in a high-
dimensional quantum computer, the simulations were performed using the high-
dimensional quantum simulator QuantumSkynet Giraldo-Carvajal et al. (2021).

This article is organized as follows: In Section 2, we present the background
with the descriptions of the DMKDE, the DMKDC, and the quantum simulator
QuantumSkynet with some of its quantum gates, in Section 3, we describe the
proposed high-dimensional quantum circuits, in Section 4, we show some results
of the method on some toy data sets, and finally, we present the conclusions
of the work in Section 5.

##  2 Background

In this section, we present a review of the Density Matrix Kernel Density
Estimation (DMKDE), and the Density Matrix Kernel Density Classification
(DMKDC) methods proposed by Gonzalez et. al. González et al. (2021), which are
the basis of this article. In addition, we describe the high-dimensional
quantum computer simulator, QuantumSkynet, with some qudit-based quantum gates
which were applied in this work.

###  2.1 Density Matrix Kernel Density Estimation (DMKDE)

The Density Matrix Kernel Density Estimation González et al. (2021) method
starts by computing a quantum feature map $x_{i}\rightarrow\ket{\psi_{i}}$
based on random Fourier features (RFF) Rahimi and Recht (2009), over a
training data set $X=\\{x_{i}\\}_{i=1,\cdots,N}$, where $\ket{\psi_{i}}$ is a
normalized vector. Then a training density matrix $\rho$ is constructed as a
maximally mixed state of all the $N$ training samples,

$\rho=\frac{1}{N}\sum_{i=1}^{N}\ket{\psi_{i}}\bra{\psi_{i}}.$ (1)

To predict the density of a testing sample $x\rightarrow\ket{\psi}$, the
expected value of the sample with the training density matrix is computed,

$\bra{\psi}\rho\ket{\psi}.$ (2)

The DMKDE in conjunction with RFF works as a non-parametric density estimator,
which can approximate probability density functions.

###  2.2 Density Matrix Kernel Density Classification (DMKDC)

The DMKDE can be used for classification as in the Density Matrix Kernel
Density Classification method (DMKDC) González et al. (2021). This algorithm
creates a quantum feature map of the training and testing samples
$x_{i}\rightarrow\ket{\psi_{i}}$, some possible quantum feature maps are based
on RFF, and soft-max encoding, as presented in González et al. (2021). Then
for each class $j\in\\{0,\cdots,D-1\\}$, it computes a training density matrix
$\rho_{j}$. The relative frequency, also called prior, of the training samples
per class is computed, $\pi_{j}=N_{j}/N$, with $N$ the total number of
training data points, and $N_{j}$ the number of training samples of class $j$.
The probability $P_{j}$ of a testing sample $x\rightarrow\ket{\psi}$ to belong
to class $j$, would be given by,

$P_{j}=\frac{\pi_{j}\bra{\psi}\rho_{j}\ket{\psi}}{\sum_{k=0}^{D-1}\pi_{k}\bra{\psi}\rho_{k}\ket{\psi}}.$
(3)

These density matrices $\rho_{j}$ can be trained as an average mixed state of
the training samples of each class (see equation 1), or by stochastic gradient
descent, which looks for the optimal parameters of the spectral decomposition
of the density matrices, $\rho_{j}=U_{j}\Lambda_{j}U_{j}^{\dagger}$ with the
training data points, using a categorical cross-entropy loss function, see
more details in González et al. (2021).

###  2.3 QuantumSkynet and high-dimensional quantum gates

QuantumSkynet Giraldo-Carvajal et al. (2021) is a high dimensional quantum
computing simulator, that allows to implement high-dimensional quantum
algorithms in a cloud-based environment. To simulate the quantum circuits
related to this project, the following gates were simulated using
QuantumSkynet:

1\. The single-qudit gate $X^{m}$ (see Fig. 1a), which is a generalized
version of the qubit-based $X$ gate for $d$ dimensions and raised to any $m$
exponent. A particular case of this gate $X$ is when the exponent $m$ is equal
to $-1$ (see Fig. 1b).

(a)

(b)

Figure 1: a High-dimensional gate $X^{m}$. b High-dimensional gate $X^{-1}$.

In Fig. 1, $\oplus$ and $\ominus$ stand for summation and subtraction modulo
$d$, respectively.

The result of applying the $X^{m}$ to the canonical basis is
$X^{m}\ket{i}=\ket{i+m}$. In particular, $X^{-1}\ket{i}=\ket{i-1}$.

2\. The control gate $CU$ (See Fig. 2). This control gate applies an arbitrary
unitary matrix U only when the control qudit takes the value $\ket{1}$.

Figure 2: High-dimensional control gate $CU$.

Here, $P_{k}$ is considered as the projection operator equals to
$\ket{k}\bra{k}$. For an arbitrary state which results from the tensor product
of two qudits, the CU gate does the following transformation,

$CU\big{(}\sum_{i=0}^{d-1}a_{i}\ket{i}\otimes\sum_{j=0}^{d-1}b_{j}\ket{j}\big{)}=\sum_{\\{i:i\neq
1\\}}a_{i}\ket{i}\otimes\sum_{j=0}^{d-1}b_{j}\ket{j}+a_{1}\ket{1}\otimes
U(\sum_{j=0}^{d-1}b_{j}\ket{j}),$

with the first qudit as the control and the second qudit as the target.

3\. The generalized controlled gate $CU^{k}$ (See Fig. 3). This gate applies
the gate $U^{k}$ to the target qudit, when the control qudit is in state
$\ket{k}$, for each possible state of the canonical basis.

Figure 3: High-dimensional generalized control gate $CU^{k}$.

One case of interest is when U is equal to $X^{-1}$. In this case, the gate
$C(X^{-1})^{k}$ does the following transformation to an arbitrary two qudit
state,

$C(X^{-1})^{k}\big{(}\sum_{(i,j)}a_{ij}\ket{ij}\big{)}=\sum_{(i,j)}a_{ij}\ket{i(j-i)},$
(4)

where the control is the first qudit and the target is the second qudit.

The $CU^{k}$ gate can be obtained by a series of $CU$ and $X^{m}$ gates, by
the same way as the multiplexer gate Khan and Perkowski (2006).

##  3 Quantum Measurement Classification with Qudits

The implementation of the algorithms DMKDE and DMKDC requires three phases:
(i) Quantum State Preparation, (ii) Training Phase, and (iii) Prediction
Phase. The first two steps were computed in a classical computer, with the
Tensorflow implementations of the algorithms González et al. (2021), while the
prediction phase was simulated in the high dimensional quantum computer
simulator QuantumSkynet Giraldo-Carvajal et al. (2021).

The steps of the DMKDE and DMKDC implementations are:

  1. 1.

Quantum State Preparation: Apply a suitable quantum feature map to the train
and test data sets.

  2. 2.

Training Phase: Construct the matrices $\rho_{j}$, one for each class, as a
mixed state of the training quantum states, see equation 1, compute the priors
$\pi_{j}$ of each class, and, calculate the spectral decomposition of these
density matrices $\rho_{j}=U_{j}\Lambda_{j}U_{j}^{\dagger}$ (in the DMKDE
method there is only one class).

  3. 3.

Prediction Phase: Apply the proposed quantum circuit to make the prediction on
each quantum state of the test data set, see equations 2, 3.

Figure 4: Qudit-based implementation of DMKDE and DMKDC methods. The quantum
feature map and the training were performed in a classical computer, while the
prediction was done in a high-dimensional quantum computer simulator.

The main contribution of this article is to show a concrete implementation of
the prediction phase of DMKDE and DMKDC as quantum circuits that can be run in
a high-dimensional quantum computer. Next, we will present the details of
these quantum protocols.

###  3.1 Initial comments of the DMKDE and DMKDC quantum circuits

To implement the prediction phase of DMKDE in a quantum computer with qudits,
we should notice that it is equivalent to finding the expected value of a
quantum state $\ket{\psi}\in\mathbb{C}^{d}$ with a training hermitian matrix
$\rho$, see equation 2, therefore, we can apply a spectral decomposition of
$\rho$,

$\bra{\psi}\rho\ket{\psi}=\bra{\psi}U\Lambda U^{{\dagger}}\ket{\psi},$ (5)

with $\Lambda$ a diagonal matrix with $\text{Tr}(\Lambda)=1$, and $U$ a
unitary matrix. But, $\Lambda=\sum_{i=0}^{d-1}\lambda_{i}\ket{i}\bra{i}$,
then,

$\bra{\psi}\rho\ket{\psi}=\bra{\psi}U\Big{(}\sum_{i=0}^{d-1}\lambda_{i}\ket{i}\bra{i}\Big{)}U^{{\dagger}}\ket{\psi}=\sum_{i=0}^{d-1}\lambda_{i}\norm{\bra{i}U^{\dagger}\ket{\psi}}^{2}.$
(6)

This form of the DMKDE can be implemented in a high-dimensional quantum
computer.

In addition, it is worth mentioning that the DMKDE quantum circuit starts by a
assuming we have a suitable quantum feature map of the testing sample
$x\rightarrow\ket{\psi}$. The quantum feature map might be based on random
Fourier features or soft-max encoding, as presented in González et al. (2021).
Also, we should have a training density matrix $\rho$ and the resulting
matrices of its spectral decomposition $U$ and $\Lambda$. In addition, based
on the matrix of eigenvalues $\Lambda$, we require the unitary transformation
$U_{\lambda}$, which satisfies that $U_{\lambda}\ket{0}=\ket{\lambda}$, where,

$\ket{\lambda}=\sum_{i=0}^{d-1}\sqrt{\lambda_{i}}\ket{i}.$ (7)

That is the state $\ket{\lambda}$ is a quantum state which encodes the
eigenvalues of the spectral decomposition of $\rho$.

The same previous arguments can be extended to the DMKDC method, but instead
of having only one training density matrix $\rho$, in the DMKDC we should have
a density matrix $\rho_{j}$ for each class of the data set.

These previous steps, i.e., the quantum feature map, the calculation of the
training density matrices $\rho_{j}$, and their spectral decompositions, were
done in a classical computer, following the tensorflow implementation of the
method González et al. (2021).

###  3.2 The DMKDE quantum circuit

The diagram of the DMKDE quantum circuit is presented in figure 5. It requires
two qudits each of dimension $d$. From the classical computer, we obtain the
quantum feature map of the input sample $\ket{\psi}\in\mathbb{C}^{d}$, the
matrix $U$ of eigenvectors of the spectral decomposition of $\rho$, and the
unitary matrix $U_{\lambda}$, which satisfies that,
$U_{\lambda}\ket{0}=\ket{\lambda}$, see equation 7.

Figure 5: DMDKE high-dimensional quantum circuit.

The circuit is initialized with $\ket{\psi}$ in the first qudit, and
$\ket{0}$, in the second qudit,

$\ket{\psi}\otimes\ket{0},$ (8)

being the first qudit the leftmost in the equation 8.

Then, we apply the unitary transformation $U^{\dagger}$ to the first qudit and
$U_{\lambda}$ to the second qudit,

$U^{\dagger}\ket{\psi}\otimes
U_{\lambda}\ket{0}=U^{\dagger}\ket{\psi}\otimes\sum_{j=0}^{d-1}\sqrt{\lambda_{j}}\ket{j}.$
(9)

We can write $U^{\dagger}\ket{\psi}=\sum_{i=0}^{d-1}a_{i}\ket{i}$, where
$\norm{a_{i}}^{2}=\norm{\bra{i}U^{\dagger}\ket{\psi}}^{2}$. Hence, the
coefficients $\norm{a_{i}}^{2}$ are the probabilities to measure
$U^{\dagger}\ket{\psi}$ in the canonical basis. Then we can write,

$\displaystyle U^{\dagger}\ket{\psi}\otimes U_{\lambda}\ket{0}$
$\displaystyle=\sum_{i=0}^{d-1}a_{i}\ket{i}\otimes\sum_{j=0}^{d-1}\sqrt{\lambda_{j}}\ket{j}$
$\displaystyle=\sum_{i=0}^{d-1}a_{i}\sqrt{\lambda_{i}}\ket{ii}+\sum_{\\{(i,j):i\neq
j\\}}a_{i}\sqrt{\lambda_{j}}\ket{ij}.$

We can then apply the generalized control gate $C(X^{-1})^{k}$ with control
the second qudit and target the first qudit. Which based on equation 4 results
in,

$C(X^{-1})^{k}(U^{\dagger}\ket{\psi}\otimes
U_{\lambda}\ket{0})=\sum_{i=0}^{d-1}a_{i}\sqrt{\lambda_{i}}\ket{0i}+\sum_{\\{(i,j):i\neq
j\\}}a_{i}\sqrt{\lambda_{j}}\ket{(i-j)j}.$ (10)

Finally, by measuring the first qudit the probability of the state $\ket{0}$
is,

$P_{0}=\sum_{i=0}^{d-1}\norm{a_{i}}^{2}\lambda_{i}=\sum_{i=0}^{d-1}\lambda_{i}\norm{\bra{i}U^{\dagger}\ket{\psi}}^{2}=\bra{\psi}\rho\ket{\psi},$
(11)

see equation 6.

###  3.3 The DMKDC quantum circuit

The proposed high-dimensional quantum circuit of the DMKDC is presented in
figure 6. As mentioned in section 2.2, the DMKDC algorithm requires $D$
density matrices $\rho_{j}$, one for each class, whose spectral decompositions
are given by $\rho_{j}=U_{j}\Lambda_{j}U_{j}^{\dagger}$. These training
density matrices are computed by equation 1 in a classical computer.

Figure 6: DMKDC high-dimensional quantum circuit.

The method requires three qudits in $\mathbb{C}^{d}$, assuming $d\geq D$. The
first qudit encodes each of the $D$ classification classes, and the relative
frequencies of the training data per class $\pi_{j}$ (priors). The second
qudit serves for two purposes, the input sample to be classified
$x\rightarrow\ket{\psi}$ in $\mathbb{C}^{d}$, at which a suitable quantum
feature map has been applied in advance, and for the matrices of eigenvectors
$U_{j}$, each class has a unitary matrix of eigenvectors. In addition, the
third qudit is responsible for the eigenvalues of each of the trained density
matrices, this can be achieved by the rotation matrices $U_{\lambda j}$, which
have the property that
$U_{\lambda_{j}}\ket{0}=\ket{\lambda_{j}}=\sum_{i=0}^{d-1}\sqrt{\lambda_{ij}}\ket{i}$,
where $\lambda_{ij}$ is the $i^{\text{th}}$ eigenvalue of the density matrix
$\rho_{j}$.

The circuit is initialized by
$\ket{\pi}=\sum_{j=0}^{D-1}\sqrt{\pi_{j}}\ket{j}$ in the first qudit, by
$\ket{\psi}$ in the second qudit, and by $\ket{0}$ in the third qudit,

$\ket{\pi}\otimes\ket{\psi}\otimes\ket{0}=\Big{(}\sum_{j=0}^{D-1}\sqrt{\pi_{j}}\ket{j}\Big{)}\otimes\ket{\psi}\otimes\ket{0},$
(12)

Since the first qudit encodes the classes of the algorithm, the circuit works
by changing the control class of the D classes with $X^{n}$ gates and applying
control $CU$ gates for the eigenvalues and eigenvectors of each of the
classes. Hence, we first apply a $X$ gate to the first qudit, to change the
control class from class 1 to class 0, the result is,

$X(\ket{\pi})\otimes\ket{\psi}\otimes\ket{0}=\Big{(}\sum_{j=0}^{D-1}\sqrt{\pi_{j}}\ket{j+1}\Big{)}\otimes\ket{\psi}\otimes\ket{0}.$
(13)

We then apply the $CU^{\dagger}_{0}$ gate with control qudit the first qudit
and target the second qudit, and the $CU_{\lambda_{0}}$ with control qudit the
first qubit and target the third qudit, see figure 6,

$\sqrt{\pi_{0}}\ket{1}\otimes
U_{0}^{\dagger}\ket{\psi}\otimes\ket{\lambda_{0}}+\Big{(}\sum_{j=1}^{D-1}\sqrt{\pi_{j}}\ket{j+1}\Big{)}\otimes\ket{\psi}\otimes\ket{0}.$
(14)

To replicate the process for class 1, we change the control class from class 0
to class 1, by applying the gate $X^{-1}$ to the first qudit, the result is,

$\sqrt{\pi_{0}}\ket{0}\otimes
U_{0}^{\dagger}\ket{\psi}\otimes\ket{\lambda_{0}}+\Big{(}\sum_{j=1}^{D-1}\sqrt{\pi_{j}}\ket{j}\Big{)}\otimes\ket{\psi}\otimes\ket{0}.$
(15)

We apply the $CU^{\dagger}_{1}$ gate with control qudit the first qudit and
target the second qudit, and the $CU_{\lambda_{1}}$ with control qudit the
first qudit and target the third qudit,

$\sqrt{\pi_{0}}\ket{0}\otimes
U_{0}^{\dagger}\ket{\psi}\otimes\ket{\lambda_{0}}+\sqrt{\pi_{1}}\ket{1}\otimes
U_{1}^{\dagger}\ket{\psi}\otimes\ket{\lambda_{1}}+\Big{(}\sum_{j=2}^{D-1}\sqrt{\pi_{j}}\ket{j}\Big{)}\otimes\ket{\psi}\otimes\ket{0}.$
(16)

We then extend the same block of the class 1 to the restating $D-2$ classes,
leaving the class $D-1$ as the control class at the end. Hence, we would have
that,

$\sum_{j=0}^{D-1}\big{(}\sqrt{\pi_{j}}\ket{j-(D-2)}\otimes
U_{j}^{\dagger}\ket{\psi}\otimes\ket{\lambda_{j}}\big{)}.$ (17)

To restore the class j to the corresponding $\ket{j}$, we apply the gate
$X^{D-2}$ to the first qudit,

$\sum_{j=0}^{D-1}\big{(}\sqrt{\pi_{j}}\ket{j}\otimes
U_{j}^{\dagger}\ket{\psi}\otimes\ket{\lambda_{j}}\big{)}.$ (18)

As with the DMKDE, we can write,

$\displaystyle U_{j}^{\dagger}\ket{\psi}\otimes\ket{\lambda_{j}}$
$\displaystyle=\sum_{l=0}^{d-1}a_{lj}\ket{l}\otimes\sum_{m=0}^{d-1}\sqrt{\lambda_{mj}}\ket{m}$
$\displaystyle=\sum_{l=0}^{d-1}a_{lj}\sqrt{\lambda_{lj}}\ket{ll}+\sum_{\\{(l,m):l\neq
m\\}}a_{lj}\sqrt{\lambda_{mj}}\ket{lm},$ (19)

where $\norm{a_{lj}}^{2}=\norm{\bra{l}U_{j}^{\dagger}\ket{\psi}}^{2}$.

Finally, by the same argument of the DMKDE, we would have that by applying the
generalized $C(X^{-1})^{k}$ with control qudit the third qudit and target the
second qudit, the circuit leads,

$\sum_{j=0}^{D-1}\Big{(}\sqrt{\pi_{j}}\ket{j}\otimes\big{(}\sum_{l=0}^{d-1}a_{lj}\sqrt{\lambda_{lj}}\ket{0l}+\sum_{\\{(l,m):l\neq
m\\}}a_{lj}\sqrt{\lambda_{mj}}\ket{(l-m)m}\big{)}\Big{)}.$ (20)

The desired result is achieved by measuring the amplitudes of the first two
qudits. We would have that,

$P_{j0}=\pi_{j}\sum_{l=0}^{d-1}\lambda_{lj}\norm{\bra{l}U_{j}^{\dagger}\ket{\psi}}^{2}=\pi_{j}\bra{\psi}\rho_{j}\ket{\psi},$
(21)

see equation 6. The sample $\ket{\psi}$ will be classified based on,

$\max\limits_{j}(\pi_{j}\bra{\psi}\rho_{j}\ket{\psi}).$ (22)

##  4 Results

We applied the DMKDE and DMKDC circuits to two data sets. We found that the
results of the quantum circuits simulated in the high-dimensional quantum
simulator QuantumSkynet mimic the results obtained in the Tensorflow
implementation of the DMKDE and DMKDC by Gonzalez et. al. González et al.
(2021).

For the DMKDE method, we used a 1-D synthetic data set, The training data set
corresponded to 1000 points sampled from the linear combination of two
Gaussian functions, and there were 1000 equally spaced data points for testing
as in González et al. (2021). In this article, they show that DMKDE in
combination with random Fourier features (RFF) can approximate any probability
density function (pdf).

Figure 7: Predictions of the DMKDE quantum circuit with 18-dits of RFF. The
DMKDE can approximate probability density functions with high enough RFF.

For the implementation of the DMKDE in the high-dimensional quantum computer
simulator, we applied a quantum feature map based on RFF to the raw data with
18 components. Figure 7 shows the results of the predictions of the DMKDE
quantum circuit. Even though more RFF components would create a better
approximation of the pdf, we were restricted by the maximum number of qudit
components allowed by the quantum computer simulator.

Furthermore, we classified two two-dimensional binary data sets of moons and
circles to test the DMKDC quantum circuit, see Figure 8. There were 1340
samples for training and 660 for testing in each data set. A quantum feature
map based on the softmax encoding González et al. (2021) was applied to each
data set, resulting in quantum features of 9 dimensions. Therefore, the
quantum circuit was constructed with qudits of 9 components. In Figure 8, we
show the classification boundaries, and the regions with higher probabilities
to be classified as either class. We obtained an accuracy of 86.66% on the
test data set of moons, and of 83.63% on the test data set of circles. The
results of the predictions with the high-dimensional quantum circuit are
consistent with the Tensorflow implementation of the DMKDC.

(a) Moons

(b) Circles

Figure 8: Binary predictions of the DMKDC circuit with 9-dits on moons and
circles.

Higher accuracies in the DMKDC quantum circuit would have been obtained, if we
had used a quantum feature map based on RFF with a higher number of
components, and if we had implemented a quantum circuit learning, in which the
weights of the trained density matrices are learned by methods like back-
propagation.

##  5 Conclusions

In the present article, we showed how to implement in a high-dimensional
quantum computer the prediction phase of the quantum-inspired machine learning
methods Density Matrix Kernel Density Estimation (DMKDE), and Density Matrix
Kernel Density Classification (DMKDC) proposed by Gonzalez et. al. González et
al. (2021). The DMKDE and DMKDC quantum circuits were simulated in the qudit-
based quantum computer simulator QuantumSkynet Giraldo-Carvajal et al. (2021).
The DMKDE quantum protocol can be extended to compute the expected value of a
density matrix with qudits.

Much work is to be done on reducing the complexity of the quantum computer
simulator to apply the DMKDC method to more realistic machine learning
problems like MNIST, and to improve the capability of the DMKDE to approximate
probability density functions by increasing the number of random Fourier
features. Furthermore, this framework opens up the possibility to implement
the DMKDE and DMKDC algorithms with stochastic-gradient descent in which the
weights of the trained density matrices are learned by some optimization
procedure, improving the performance of the density estimation and
classification.

## References

  * Arute et al. [2019] Frank Arute, Kunal Arya, Ryan Babbush, Dave Bacon, Joseph C Bardin, Rami Barends, Rupak Biswas, Sergio Boixo, Fernando G S L Brandao, David A Buell, Brian Burkett, Yu Chen, Zijun Chen, Ben Chiaro, Roberto Collins, William Courtney, Andrew Dunsworth, Edward Farhi, Brooks Foxen, Austin Fowler, Craig Gidney, Marissa Giustina, Rob Graff, Keith Guerin, Steve Habegger, Matthew P Harrigan, Michael J Hartmann, Alan Ho, Markus Hoffmann, Trent Huang, Travis S Humble, Sergei V Isakov, Evan Jeffrey, Zhang Jiang, Dvir Kafri, Kostyantyn Kechedzhi, Julian Kelly, Paul V Klimov, Sergey Knysh, Alexander Korotkov, Fedor Kostritsa, David Landhuis, Mike Lindmark, Erik Lucero, Dmitry Lyakh, Salvatore Mandrà, Jarrod R Mcclean, Matthew Mcewen, Anthony Megrant, Xiao Mi, Kristel Michielsen, Masoud Mohseni, Josh Mutus, Ofer Naaman, Matthew Neeley, Charles Neill, Murphy Yuezhen Niu, Eric Ostby, Andre Petukhov, John C Platt, Chris Quintana, Eleanor G Rieffel, Pedram Roushan, Nicholas C Rubin, Daniel Sank, Kevin J Satzinger, Vadim Smelyanskiy, Kevin J Sung, Matthew D Trevithick, Amit Vainsencher, Benjamin Villalonga, Theodore White, Z Jamie Yao, Ping Yeh, Adam Zalcman, Hartmut Neven, and John M Martinis.  Quantum supremacy using a programmable superconducting processor.  _Nature_ , 574:505, 2019.  doi: 10.1038/s41586-019-1666-5.  URL https://doi.org/10.1038/s41586-019-1666-5.
  * Schaeff et al. [2015] Christoph Schaeff, Robert Polster, Marcus Huber, Sven Ramelow, and Anton Zeilinger.  Experimental access to higher-dimensional entangled quantum systems using integrated optics.  _Optica_ , 2(6):523, 6 2015.  ISSN 2334-2536.  doi: 10.1364/optica.2.000523.  URL http://dx.doi.org/10.1364/OPTICA.2.000523.
  * Carolan et al. [2015] Jacques Carolan, Christopher Harrold, Chris Sparrow, Enrique Martín-López, Nicholas J. Russell, Joshua W. Silverstone, Peter J. Shadbolt, Nobuyuki Matsuda, Manabu Oguma, Mikitaka Itoh, Graham D. Marshall, Mark G. Thompson, Jonathan C.F. Matthews, Toshikazu Hashimoto, Jeremy L. O’Brien, and Anthony Laing.  Universal linear optics.  _Science_ , 349(6249):711–716, 8 2015.  ISSN 10959203.  doi: 10.1126/science.aab3642.  URL https://pubmed.ncbi.nlm.nih.gov/26160375/.
  * Sit et al. [2017] Alicia Sit, Frédéric Bouchard, Robert Fickler, Jérémie Gagnon-Bischoff, Hugo Larocque, Khabat Heshami, Dominique Elser, Christian Peuntinger, Kevin Günthner, Bettina Heim, Christoph Marquardt, Gerd Leuchs, Robert W. Boyd, and Ebrahim Karimi.  High-dimensional intracity quantum cryptography with structured photons.  _Optica_ , 4(9):1006, 9 2017.  ISSN 2334-2536.  doi: 10.1364/optica.4.001006.  URL https://doi.org/10.1364/OPTICA.4.001006.
  * Klimov et al. [2003] A. B. Klimov, R. Guzmán, J. C. Retamal, and C. Saavedra.  Qutrit quantum computer with trapped ions.  _Physical Review A - Atomic, Molecular, and Optical Physics_ , 67(6):7, 6 2003.  ISSN 10941622.  doi: 10.1103/PhysRevA.67.062313.  URL https://journals.aps.org/pra/abstract/10.1103/PhysRevA.67.062313.
  * Gedik et al. [2015] Z. Gedik, I. A. Silva, B. Çakmak, G. Karpat, E. L.G. Vidoto, D. O. Soares-Pinto, E. R. DeAzevedo, and F. F. Fanchini.  Computational speed-up with a single qudit.  _Scientific Reports_ , 5(1):14671, 10 2015.  ISSN 20452322.  doi: 10.1038/srep14671.  URL www.nature.com/scientificreports/.
  * Moreno-Pineda et al. [2018] Eufemio Moreno-Pineda, Clément Godfrin, Franck Balestro, Wolfgang Wernsdorfer, and Mario Ruben.  Molecular spin qudits for quantum algorithms, 1 2018.  ISSN 14604744.  URL https://pubs.rsc.org/en/content/articlehtml/2018/cs/c5cs00933bhttps://pubs.rsc.org/en/content/articlelanding/2018/cs/c5cs00933b.
  * Cozzolino et al. [2019] Daniele Cozzolino, Beatrice Da Lio, Davide Bacco, and Leif Katsuo Oxenløwe.  High-dimensional quantum communication: benefits, progress and future challenges.  Technical report, 2019.
  * Sheridan and Scarani [2010] Lana Sheridan and Valerio Scarani.  Security proof for quantum key distribution using qudit systems.  _Physical Review A - Atomic, Molecular, and Optical Physics_ , 82(3):030301, 9 2010.  ISSN 10502947.  doi: 10.1103/PhysRevA.82.030301.  URL https://journals.aps.org/pra/abstract/10.1103/PhysRevA.82.030301.
  * Rebentrost et al. [2014] Patrick Rebentrost, Masoud Mohseni, and Seth Lloyd.  Quantum support vector machine for big data classification.  _Physical Review Letters_ , 113(3), 9 2014.  ISSN 10797114.  doi: 10.1103/PhysRevLett.113.130503.
  * Wiebe et al. [2014] Nathan Wiebe, Ashish Kapoor, and Krysta M. Svore.  Quantum Deep Learning.  12 2014.
  * Lu and Braunstein [2014] Songfeng Lu and Samuel L Braunstein.  Quantum decision tree classifier.  13:757–770, 2014.  doi: 10.1007/s11128-013-0687-5.
  * Ezhov and Ventura [2000] Alexandr A. Ezhov and Dan Ventura.  Quantum Neural Networks.  pages 213–235. Physica, Heidelberg, 2000.  doi: 10.1007/978-3-7908-1856-7–“˙˝11.  URL https://link.springer.com/chapter/10.1007/978-3-7908-1856-7_11.
  * Cong et al. [2019] Iris Cong, Soonwon Choi, and Mikhail D. Lukin.  Quantum convolutional neural networks.  _Nature Physics_ , 15(12):1273–1278, 12 2019\.  ISSN 17452481.  doi: 10.1038/s41567-019-0648-8.  URL https://www.nature.com/articles/s41567-019-0648-8.
  * Dallaire-Demers and Killoran [2018] Pierre-Luc Dallaire-Demers and Nathan Killoran.  Quantum generative adversarial networks.  Technical report, 2018.
  * Cárdenas-López et al. [2018] F. A. Cárdenas-López, L. Lamata, J. C. Retamal, and E. Solano.  Multiqubit and multilevel quantum reinforcement learning with quantum technologies.  _PLoS ONE_ , 13(7):e0200455, 7 2018.  ISSN 19326203.  doi: 10.1371/journal.pone.0200455.  URL https://doi.org/10.1371/journal.pone.0200455.
  * Diep [2020] Do Ngoc Diep.  Some Quantum Neural Networks.  _International Journal of Theoretical Physics_ , 59(4):1179–1187, 4 2020.  ISSN 15729575.  doi: 10.1007/s10773-020-04397-1.  URL https://link.springer.com/article/10.1007/s10773-020-04397-1.
  * Ricks and Ventura [2004] Bob Ricks and Dan Ventura.  Training a quantum neural network.  In S. Thrun, L. Saul, and B. Schölkopf, editors, _Advances in Neural Information Processing Systems_ , volume 16. MIT Press, 2004.  URL https://proceedings.neurips.cc/paper/2003/file/505259756244493872b7709a8a01b536-Paper.pdf.
  * Beer et al. [2019] Kerstin Beer, Dmytro Bondarenko, Terry Farrelly, Tobias J. Osborne, Robert Salzmann, and Ramona Wolf.  Efficient learning for deep quantum neural networks, 2 2019.  ISSN 23318422.  URL https://doi.org/10.1038/s41467-020-14454-2.
  * González et al. [2021] Fabio A. González, Alejandro Gallego, Santiago Toledo-Cortés, and Vladimir Vargas-Calderón.  Learning with Density Matrices and Random Features.  2021\.  URL http://arxiv.org/abs/2102.04394.
  * Giraldo-Carvajal et al. [2021] Andres Giraldo-Carvajal, Daniel A. Duque-Ramirez, and Jose A. Jaramillo-Villegas.  Quantumskynet: A high-dimensional quantum computing simulator, 2021.
  * Rahimi and Recht [2009] Ali Rahimi and Benjamin Recht.  Random features for large-scale kernel machines.  In _Advances in Neural Information Processing Systems 20 - Proceedings of the 2007 Conference_ , 2009.  ISBN 160560352X.
  * González et al. [2021] Fabio A. González, Vladimir Vargas-Calderón, and Herbert Vinck-Posada.  Classification with quantum measurements.  _Journal of the Physical Society of Japan_ , 90(4):044002, 2021.  doi: 10.7566/JPSJ.90.044002.  URL https://doi.org/10.7566/JPSJ.90.044002.
  * Khan and Perkowski [2006] Faisal Shah Khan and Marek Perkowski.  Synthesis of multi-qudit hybrid and d-valued quantum logic circuits by decomposition.  _Theoretical Computer Science_ , 367(3):336–346, 2006.